VIP tutorial on Version extension

This commit is contained in:
Mindaugas Vinkelis
2020-02-02 20:03:21 +02:00
parent d24dfe14f5
commit 541632fa9e
17 changed files with 148 additions and 113 deletions

View File

@@ -64,11 +64,9 @@ void serialize(S& s, MyStruct& o) {
s.container4b(o.fs, 10);
}
using namespace bitsery;
using Buffer = std::vector<uint8_t>;
using OutputAdapter = OutputBufferAdapter<Buffer>;
using InputAdapter = InputBufferAdapter<Buffer>;
using OutputAdapter = bitsery::OutputBufferAdapter<Buffer>;
using InputAdapter = bitsery::InputBufferAdapter<Buffer>;
int main() {
MyStruct data{8941, MyEnum::V2, {15.0f, -8.5f, 0.045f}};
@@ -76,10 +74,10 @@ int main() {
Buffer buffer;
auto writtenSize = quickSerialization<OutputAdapter>(buffer, data);
auto state = quickDeserialization<InputAdapter>({buffer.begin(), writtenSize}, res);
auto writtenSize = bitsery::quickSerialization<OutputAdapter>(buffer, data);
auto state = bitsery::quickDeserialization<InputAdapter>({buffer.begin(), writtenSize}, res);
assert(state.first == ReaderError::NoError && state.second);
assert(state.first == bitsery::ReaderError::NoError && state.second);
assert(data.fs == res.fs && data.i == res.i && data.e == res.e);
}
```

View File

@@ -1,8 +1,8 @@
The grand plan for this tutorial is to learn how to serialize/deserialize any object efficiently in time and space, so you could focus on other, more interesting things.
This tutorial will cover these main topics:
* `Hello World` write one control flow for both: serialization and deserialization.
* `Composer` efficiently compose complex serialization flows.
* [Getting started](hello_world.md) with bitsery, and serialize/deserialize your first object.
* [Extend to your needs](first_extension.md) by enabling serialization/deserialization depending on version number.
* `Squeeze Me!` compress your data when you know what it stores.
* `Anything is Possible` extend library for custom container, compress geometry and more.
* `Little or Big` change endianness if you want best performance on PowerPC.

View File

@@ -1,3 +0,0 @@
*document in progress*
* explain why *value* and *object* is fundamental functions.
* write about *Growable* extension

View File

@@ -0,0 +1,67 @@
... TODO explain step-by-step what we need and how to get there
Instead I immediately provide implementation for an extension.
```cpp
#include "../details/adapter_common.h"
#include "../traits/core/traits.h"
namespace bitsery {
namespace ext {
template<size_t VERSION>
class Version {
public:
template<typename Ser, typename T, typename Fnc>
void serialize(Ser &ser, const T &v, Fnc &&fnc) const {
details::writeSize(ser.adapter(), VERSION);
fnc(ser, const_cast<T&>(v), VERSION);
}
template<typename Des, typename T, typename Fnc>
void deserialize(Des &des, T &v, Fnc &&fnc) const {
size_t version{};
details::readSize(des.adapter(), version, 0u, std::false_type{});
fnc(des, v, version);
}
};
}
namespace traits {
template<typename T, size_t V>
struct ExtensionTraits<ext::Version<V>, T> {
using TValue = T;
static constexpr bool SupportValueOverload = false;
static constexpr bool SupportObjectOverload = false;
static constexpr bool SupportLambdaOverload = true;
};
}
}
```
Adding such extension to the bitsery itself is impractical because it is very easy to implement, but at the same time it has a lot of customization options that actual user might require e.g.:
* how do you want to handle reading/writing version number? (in this case use compact representation as size, but do not check for errors if version number is too large)
* maybe you want to be able to set ReaderError, when version is larger than deserialization implementation handles. (we simply ignore this case and later we'll probably get reading error later anyway)
* or maybe you want to wrap object in `Growable` extension? so that you could ignore unknown fields in newer version of object.
Example of how to use this provided implementation:
```cpp
struct TypeV2 {
uint16_t x{};
uint16_t y{};
};
template <typename S>
void serialize(S& ser, TypeV2& obj) {
ser.ext(obj, bitsery::ext::Version<2u>{}, [](S& s, TypeV2&o, size_t version) {
s.value2b(o.x);
if (version == 2u) {
s.value2b(o.y);
}
});
}
```

View File

@@ -18,11 +18,9 @@ There's nothing to build or make - **bitsery** is header only.
#include <bitsery/traits/vector.h>
#include <bitsery/traits/string.h>
using namespace bitsery;
using Buffer = std::vector<uint8_t>;
using OutputAdapter = OutputBufferAdapter<Buffer>;
using InputAdapter = InputBufferAdapter<Buffer>;
using OutputAdapter = bitsery::OutputBufferAdapter<Buffer>;
using InputAdapter = bitsery::InputBufferAdapter<Buffer>;
```
@@ -62,7 +60,7 @@ void serialize(S& s, MyStruct& o) {
This example we choosed probably unfamiliar verbose syntax, so lets explain core functionality that you'll use all the time:
* **s.value4b(o.i);** serialize fundamental types (ints, floats, enums) value**4b** means, that data type is 4 bytes. If you use same code on different machines, if it compiles it means it is compatible.
* **s.text1b(o.str);** serialize text (null-terminated) of char type, if you use *wchar* then you would write *text2b*.
* **s.text1b(o.str);** serialize text (null-terminated) of char type, if you use *wchar* then you would write *text2b* or *text4b* depending on the OS platform.
* **s.container4b(o.fs, 100);** serializes any container of fundamental types of size 4bytes, **100** is max size of container.
**Bitsery** is designed to be save with untrusted (malicious) data from network, so for dynamic containers you always need to provide max possible size available, to avoid buffer-overflow attacks.
**text** didn't had this max size specified, because it was serializing fixed size container.
@@ -76,8 +74,8 @@ Create buffer and use helper functions for serialization and deserialization.
```cpp
Buffer buffer;
auto writtenSize = quickSerialization(OutputAdapter{buffer}, data);
auto state = quickDeserialization(InputAdapter{buffer.begin(), writtenSize}, res);
auto writtenSize = bitsery::quickSerialization(OutputAdapter{buffer}, data);
auto state = bitsery::quickDeserialization(InputAdapter{buffer.begin(), writtenSize}, res);
```
These helper functions use default configuration *bitsery::DefaultConfig*
@@ -93,11 +91,9 @@ deserialization state has two properties, error code and bool that indicates if
#include <bitsery/traits/vector.h>
#include <bitsery/traits/string.h>
using namespace bitsery;
using Buffer = std::vector<uint8_t>;
using OutputAdapter = OutputBufferAdapter<Buffer>;
using InputAdapter = InputBufferAdapter<Buffer>;
using OutputAdapter = bitsery::OutputBufferAdapter<Buffer>;
using InputAdapter = bitsery::InputBufferAdapter<Buffer>;
struct MyStruct {
uint32_t i;
@@ -110,17 +106,17 @@ void serialize(S& s, MyStruct& o) {
s.value4b(o.i);
s.text1b(o.str);
s.container4b(o.fs, 100);
};
}
int main() {
MyStruct data{8941, "hello", {15.0f, -8.5f, 0.045f}};
MyStruct res{};
Buffer buffer;
auto writtenSize = quickSerialization(OutputAdapter{buffer}, data);
auto state = quickDeserialization(InputAdapter{buffer.begin(), writtenSize}, res);
auto writtenSize = bitsery::quickSerialization(OutputAdapter{buffer}, data);
auto state = bitsery::quickDeserialization(InputAdapter{buffer.begin(), writtenSize}, res);
assert(state.first == ReaderError::NoError && state.second);
assert(state.first == bitsery::ReaderError::NoError && state.second);
assert(data.fs == res.fs && data.i == res.i && std::strcmp(data.str, res.str) == 0);
}
```

View File

@@ -22,12 +22,10 @@ void serialize(S& s, MyStruct& o) {
s.container4b(o.fs, 10);//resizable containers also requires maxSize, to make it safe from buffer-overflow attacks
}
using namespace bitsery;
//some helper types
using Buffer = std::vector<uint8_t>;
using OutputAdapter = OutputBufferAdapter<Buffer>;
using InputAdapter = InputBufferAdapter<Buffer>;
using OutputAdapter = bitsery::OutputBufferAdapter<Buffer>;
using InputAdapter = bitsery::InputBufferAdapter<Buffer>;
int main() {
//set some random data
@@ -39,12 +37,12 @@ int main() {
//use quick serialization function,
//it will use default configuration to setup all the nesessary steps
//and serialize data to container
auto writtenSize = quickSerialization<OutputAdapter>(buffer, data);
auto writtenSize = bitsery::quickSerialization<OutputAdapter>(buffer, data);
//same as serialization, but returns deserialization state as a pair
//first = error code, second = is buffer was successfully read from begin to the end.
auto state = quickDeserialization<InputAdapter>({buffer.begin(), writtenSize}, res);
auto state = bitsery::quickDeserialization<InputAdapter>({buffer.begin(), writtenSize}, res);
assert(state.first == ReaderError::NoError && state.second);
assert(state.first == bitsery::ReaderError::NoError && state.second);
assert(data.fs == res.fs && data.i == res.i && data.e == res.e);
}

View File

@@ -41,12 +41,10 @@ namespace MyTypes {
}
}
using namespace bitsery;
//use fixed-size buffer
using Buffer = std::array<uint8_t, 10000>;
using OutputAdapter = OutputBufferAdapter<Buffer>;
using InputAdapter = InputBufferAdapter<Buffer>;
using OutputAdapter = bitsery::OutputBufferAdapter<Buffer>;
using InputAdapter = bitsery::InputBufferAdapter<Buffer>;
int main() {
//set some random data
@@ -55,10 +53,10 @@ int main() {
//create buffer to store data to
Buffer buffer{};
auto writtenSize = quickSerialization<OutputAdapter>(buffer, data);
auto writtenSize = bitsery::quickSerialization<OutputAdapter>(buffer, data);
MyTypes::Monster res{};
auto state = quickDeserialization<InputAdapter>({buffer.begin(), writtenSize}, res);
auto state = bitsery::quickDeserialization<InputAdapter>({buffer.begin(), writtenSize}, res);
assert(state.first == ReaderError::NoError && state.second);
assert(state.first == bitsery::ReaderError::NoError && state.second);
}

View File

@@ -23,12 +23,10 @@ struct MyStruct {
};
using namespace bitsery;
//some helper types
using Buffer = std::vector<uint8_t>;
using OutputAdapter = OutputBufferAdapter<Buffer>;
using InputAdapter = InputBufferAdapter<Buffer>;
using OutputAdapter = bitsery::OutputBufferAdapter<Buffer>;
using InputAdapter = bitsery::InputBufferAdapter<Buffer>;
int main() {
//set some random data
@@ -37,10 +35,10 @@ int main() {
//serialization, deserialization flow is unchanged as in basic usage
Buffer buffer;
auto writtenSize = quickSerialization<OutputAdapter>(buffer, data);
auto writtenSize = bitsery::quickSerialization<OutputAdapter>(buffer, data);
auto state = quickDeserialization<InputAdapter>({buffer.begin(), writtenSize}, res);
auto state = bitsery::quickDeserialization<InputAdapter>({buffer.begin(), writtenSize}, res);
assert(state.first == ReaderError::NoError && state.second);
assert(state.first == bitsery::ReaderError::NoError && state.second);
assert(data.fs == res.fs && data.i == res.i && data.e == res.e);
}

View File

@@ -29,27 +29,24 @@ void serialize(S& s, MyStruct& o) {
using MyTuple = std::tuple<float, MyStruct>;
using MyVariant = std::variant<int64_t, MyTuple, MyStruct>;
// for convenience
using namespace bitsery;
// define default serialize function for MyVariant, so that we could use quickSerialization/Deserialization functions
template<typename S>
void serialize(S& s, MyVariant& o) {
// in order to serialize a variant, it needs to know how to do it for all types
// we can do this simply by providing any callable object, that accepts serializer and type as arguments
s.ext(o, ext::StdVariant{
s.ext(o, bitsery::ext::StdVariant{
// specify how to serialize tuple by creating a lambda
[](S& s, MyTuple& o) {
// StdTuple is used exactly the same as StdVariant
s.ext(o, ext::StdTuple{
s.ext(o, bitsery::ext::StdTuple{
// this is convenient callable object to specify integral value size
// it is different equivalent to lambda [](auto& s, float&o) { s.value4b(o);}
ext::OverloadValue<float, 4>{},
bitsery::ext::OverloadValue<float, 4>{},
// it is not required to provide MyStruct overload, because it we have defined 'serialize' function for it
});
},
// this might also be useful if you want to overload using extension
ext::OverloadExtValue<int64_t, 8, ext::CompactValue>{},
bitsery::ext::OverloadExtValue<int64_t, 8, bitsery::ext::CompactValue>{},
// you can even go further and instead of writing lambda for MyTuple you can as well compose the same functionality
// with OverloadExtObject, like this:
// (comment out MyTuple lambda, and uncomment this)
@@ -59,7 +56,7 @@ void serialize(S& s, MyVariant& o) {
[](S& s, MyStruct& o) {
s.value4b(o.f);
s.container(o.v, 1000, [](S& s, int32_t& v) {
s.ext4b(v, ext::CompactValue{});
s.ext4b(v, bitsery::ext::CompactValue{});
});
},
// NOTE.
@@ -78,8 +75,8 @@ void serialize(S& s, MyVariant& o) {
//some helper types
using Buffer = std::vector<uint8_t>;
using OutputAdapter = OutputBufferAdapter<Buffer>;
using InputAdapter = InputBufferAdapter<Buffer>;
using OutputAdapter = bitsery::OutputBufferAdapter<Buffer>;
using InputAdapter = bitsery::InputBufferAdapter<Buffer>;
int main() {
@@ -93,13 +90,13 @@ int main() {
//use quick serialization function,
//it will use default configuration to setup all the nesessary steps
//and serialize data to container
auto writtenSize = quickSerialization<OutputAdapter>(buffer, data);
auto writtenSize = bitsery::quickSerialization<OutputAdapter>(buffer, data);
//same as serialization, but returns deserialization state as a pair
//first = error code, second = is buffer was successfully read from begin to the end.
auto state = quickDeserialization<InputAdapter>({buffer.begin(), writtenSize}, res);
auto state = bitsery::quickDeserialization<InputAdapter>({buffer.begin(), writtenSize}, res);
assert(state.first == ReaderError::NoError && state.second);
assert(state.first == bitsery::ReaderError::NoError && state.second);
assert(data == res);
}
#else

View File

@@ -67,10 +67,9 @@ using Context = std::tuple<int, std::pair<uint32_t, uint32_t>>;
//use fixed-size buffer
using Buffer = std::vector<uint8_t>;
using namespace bitsery;
// define adapter types,
using OutputAdapter = OutputBufferAdapter<Buffer>;
using InputAdapter = InputBufferAdapter<Buffer>;
using OutputAdapter = bitsery::OutputBufferAdapter<Buffer>;
using InputAdapter = bitsery::InputBufferAdapter<Buffer>;
int main() {
@@ -90,10 +89,10 @@ int main() {
//create buffer to store data to
Buffer buffer{};
auto writtenSize = quickSerialization(ctx, OutputAdapter{buffer}, data);
auto writtenSize = bitsery::quickSerialization(ctx, OutputAdapter{buffer}, data);
MyTypes::GameState res{};
auto state = quickDeserialization(ctx, InputAdapter{buffer.begin(), writtenSize}, res);
auto state = bitsery::quickDeserialization(ctx, InputAdapter{buffer.begin(), writtenSize}, res);
assert(state.first == ReaderError::NoError && state.second);
assert(state.first == bitsery::ReaderError::NoError && state.second);
}

View File

@@ -19,8 +19,6 @@ void serialize(S& s, MyStruct& o) {
s.value8b(o.f);
}
using namespace bitsery;
int main() {
//set some random data
MyStruct data{8941, MyEnum::V2, 0.045};
@@ -35,7 +33,7 @@ int main() {
}
//we cannot use quick serialization function, because streams cannot use writtenBytesCount method
Serializer<OutputBufferedStreamAdapter> ser{s};
bitsery::Serializer<bitsery::OutputBufferedStreamAdapter> ser{s};
ser.object(data);
//flush to writer
ser.adapter().flush();
@@ -50,8 +48,8 @@ int main() {
//same as serialization, but returns deserialization state as a pair
//first = error code, second = is buffer was successfully read from begin to the end.
auto state = quickDeserialization<InputStreamAdapter>(s, res);
auto state = bitsery::quickDeserialization<bitsery::InputStreamAdapter>(s, res);
assert(state.first == ReaderError::NoError && state.second);
assert(state.first == bitsery::ReaderError::NoError && state.second);
assert(data.f == res.f && data.i == res.i && data.e == res.e);
}

View File

@@ -68,12 +68,10 @@ namespace MyTypes {
}
}
using namespace bitsery;
//use fixed-size buffer
using Buffer = std::array<uint8_t, 10000>;
using OutputAdapter = OutputBufferAdapter<Buffer>;
using InputAdapter = InputBufferAdapter<Buffer>;
using OutputAdapter = bitsery::OutputBufferAdapter<Buffer>;
using InputAdapter = bitsery::InputBufferAdapter<Buffer>;
int main() {
//set some random data
@@ -84,11 +82,11 @@ int main() {
//create buffer to store data to
Buffer buffer{};
//since we're using different configuration, we cannot use quickSerialization function.
auto writtenSize = quickSerialization<OutputAdapter>(buffer, data);
auto writtenSize = bitsery::quickSerialization<OutputAdapter>(buffer, data);
MyTypes::Monster res{};
//deserialize
auto state = quickDeserialization<InputAdapter>({buffer.begin(), writtenSize}, res);
auto state = bitsery::quickDeserialization<InputAdapter>({buffer.begin(), writtenSize}, res);
assert(state.first == ReaderError::NoError && state.second);
assert(state.first == bitsery::ReaderError::NoError && state.second);
}

View File

@@ -78,13 +78,10 @@ namespace bitsery {
struct SelectSerializeFnc<MultipleInheritance>:UseNonMemberFnc {};
}
using namespace bitsery;
//some helper types
using Buffer = std::vector<uint8_t>;
using Writer = OutputBufferAdapter<Buffer>;
using Reader = InputBufferAdapter<Buffer>;
using Writer = bitsery::OutputBufferAdapter<Buffer>;
using Reader = bitsery::InputBufferAdapter<Buffer>;
int main() {
@@ -95,13 +92,13 @@ int main() {
Buffer buf{};
ext::InheritanceContext ctx1;
auto writtenSize = quickSerialization(ctx1, Writer{buf}, data);
bitsery::ext::InheritanceContext ctx1;
auto writtenSize = bitsery::quickSerialization(ctx1, Writer{buf}, data);
assert(writtenSize == 4);//base is serialized once, because it is inherited virtually
MultipleInheritance res{0};
ext::InheritanceContext ctx2;
auto state = quickDeserialization(ctx2, Reader{buf.begin(), writtenSize}, res);
assert(state.first == ReaderError::NoError && state.second);
bitsery::ext::InheritanceContext ctx2;
auto state = bitsery::quickDeserialization(ctx2, Reader{buf.begin(), writtenSize}, res);
assert(state.first == bitsery::ReaderError::NoError && state.second);
assert(data.x == res.x && data.y1 == res.y1 && data.getY2() == res.getY2() && data.z == res.z);
}

View File

@@ -30,12 +30,10 @@ public:
}
};
using namespace bitsery;
//some helper types
using Buffer = std::vector<uint8_t>;
using Writer = OutputBufferAdapter<Buffer>;
using Reader = InputBufferAdapter<Buffer>;
using Writer = bitsery::OutputBufferAdapter<Buffer>;
using Reader = bitsery::InputBufferAdapter<Buffer>;
int main() {
@@ -49,16 +47,16 @@ int main() {
//we cant use quick (de)serialization helper methods, because we ant to serialize container directly
//create writer and serialize container
Serializer<Writer> ser{buffer};
bitsery::Serializer<Writer> ser{buffer};
ser.container(data, 10);
ser.adapter().flush();
//create reader and deserialize container
Deserializer<Reader> des{buffer.begin(), ser.adapter().writtenBytesCount()};
bitsery::Deserializer<Reader> des{buffer.begin(), ser.adapter().writtenBytesCount()};
des.container(res, 10);
//check if everything went ok
assert(des.adapter().error() == ReaderError::NoError && des.adapter().isCompletedSuccessfully());
assert(des.adapter().error() == bitsery::ReaderError::NoError && des.adapter().isCompletedSuccessfully());
assert(res == data);
}

View File

@@ -78,12 +78,10 @@ private:
}
};
using namespace bitsery;
//some helper types
using Buffer = std::vector<uint8_t>;
using Writer = OutputBufferAdapter<Buffer>;
using Reader = InputBufferAdapter<Buffer>;
using Writer = bitsery::OutputBufferAdapter<Buffer>;
using Reader = bitsery::InputBufferAdapter<Buffer>;
//we will need PointerLinkingContext to work with pointers
//if we would require additional context for our own custom flow, we can define it as tuple like this:
@@ -114,7 +112,7 @@ int main() {
size_t writtenSize{};
//in order to use pointers, we need to pass pointer linking context serializer/deserializer
{
ext::PointerLinkingContext ctx{};
bitsery::ext::PointerLinkingContext ctx{};
writtenSize = quickSerialization(ctx, Writer{buffer}, data);
//make sure that pointer linking context is valid
@@ -125,10 +123,10 @@ int main() {
Test1Data res{};
{
ext::PointerLinkingContext ctx{};
bitsery::ext::PointerLinkingContext ctx{};
auto state = quickDeserialization(ctx, Reader{buffer.begin(), writtenSize}, res);
//check if everything went find
assert(state.first == ReaderError::NoError && state.second);
assert(state.first == bitsery::ReaderError::NoError && state.second);
//also check for dangling pointers, after deserialization
assert(ctx.isValid());
}

View File

@@ -181,23 +181,21 @@ namespace bitsery {
// also it automatically ensures, that classes is registered in the same order for serialization and deserialization
using MyPolymorphicClassesForRegistering = bitsery::ext::PolymorphicClassesList<Shape>;
//use bitsery namespace for convenience
using namespace bitsery;
//some helper types
using Buffer = std::vector<uint8_t>;
using Writer = OutputBufferAdapter<Buffer>;
using Reader = InputBufferAdapter<Buffer>;
using Writer = bitsery::OutputBufferAdapter<Buffer>;
using Reader = bitsery::InputBufferAdapter<Buffer>;
//we need to define few things in order to work with polymorphism
//1) we need pointer linking context to work with pointers
//2) we need polymorphic context to be able to work with polymorphic types
using TContext = std::tuple<ext::PointerLinkingContext, ext::PolymorphicContext<ext::StandardRTTI>>;
using TContext = std::tuple<
bitsery::ext::PointerLinkingContext,
bitsery::ext::PolymorphicContext<bitsery::ext::StandardRTTI>>;
//NOTE:
// RTTI can be customizable, if you can't use dynamic_cast and typeid, and have 'custom' solution
using MySerializer = Serializer<Writer, TContext>;
using MyDeserializer = Deserializer<Reader, TContext>;
using MySerializer = bitsery::Serializer<Writer, TContext>;
using MyDeserializer = bitsery::Deserializer<Reader, TContext>;
//checks if deserialized data is equal
void assertSameShapes(const SomeShapes &data, const SomeShapes &res) {
@@ -257,7 +255,7 @@ int main() {
//deserialize our data
MyDeserializer des{ctx, buffer.begin(), writtenSize};
des.object(res);
assert(des.adapter().error() == ReaderError::NoError && des.adapter().isCompletedSuccessfully());
assert(des.adapter().error() == bitsery::ReaderError::NoError && des.adapter().isCompletedSuccessfully());
//also check for dangling pointers, after deserialization
assert(std::get<0>(ctx).isValid());
// clear shared state from pointer linking context,