Niels Lohmann
dc768a7d5d
* Write and read the inner length of BSON binary subtype 0x02 The BSON specification defines the payload of the "binary (old)" subtype 0x02 as an int32 followed by that many bytes. to_bson wrote the plain payload instead, so decoders such as pymongo and libbson-based readers rejected the bytes, and from_bson returned the inner length as part of the payload (documented as lenient behavior in gh-5333). The writer now emits the outer length (payload + 4), the subtype, the inner int32 holding the payload size, and the payload. The reader requires an outer length of at least 4 and an inner length equal to the outer length minus 4 (the cases the bson-corpus lists as decode errors), reports parse error 112 otherwise, and returns only the payload. Both directions change together so that the round trip keeps working. The single-reservation fast path for scalar entries declines subtype 0x02 because it has no room for the inner length. This is a deliberate exception to the opt-in rule for behavior changes: the previous output was invalid BSON. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Check the declared BSON document size before reading its content The BSON reader compared a document's declared size with the number of bytes read only when it reached the terminator. The declared size never bounded the reads, so a SAX handler saw the keys and values of a document whose size was too small before parse error 112 was reported. The container frame now stores the position where the document ends according to its declared size. open_bson_document rejects a size below 5 (size prefix plus terminator) and an embedded document that does not end before the terminator of its enclosing document; the element loop rejects an element whose type byte leaves no room for the name terminator and the document terminator. The terminator check stays and still catches a declared size that is too large. No input that was accepted before is rejected now; documents with a wrong size are reported earlier, with fewer SAX events, a different byte position and, in three cases, a different message. The error id (parse_error.112) is unchanged and strict=false still does not skip the check. The tests that asserted the old messages and positions are updated, with the outer sizes of two inputs corrected so that they still exercise the embedded document. New tests cover the SAX events, a size below 5, and an embedded document that fits exactly. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Document BSON subtype 0x02, document size checks, and lenient reads Describe the inner length of binary subtype 0x02 that to_bson now writes and from_bson checks and removes. Add the document size checks (parse_error.112, not affected by strict=false) to bson.md and from_bson.md, and extend the list of lenient reads with the UUID and MD5 payload lengths (not checked), duplicate keys (last value wins), embedded NUL bytes in strings, and the ignored trailing bytes with strict=false. State that the members of an object are written in iteration order. Correct the statement that element names are never validated on read: they are checked and sanitized like string values by the error handler. Add the BSON message to the examples of out_of_range.415 and type_error.321, the subtype limit of 255 to the binary values page, CBOR to the formats that use byte_container_with_subtype, and fix a typo in the incomplete mapping box. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Regenerate single_include and apply astyle Result of make amalgamate (including make pretty) for the preceding changes. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Address review of the BSON documentation Describe exactly when a too small document size is detected and note that subtype 2 data written by earlier versions is not readable. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * docs: use the Oxford comma, drop discarded-value remarks Add the serial comma in a list added by this PR and remove the BSON example of type_error.321 (with the type_error.317 remark) so the docs do not mention discarded values for BSON; restore the original singular heading of that section. No tables were changed by this PR. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix -Weffc++ in the BSON recorder test and a clang-tidy use-auto hit Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix clang-tidy readability-redundant-member-init in BSON test Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me>
14 KiB
Binary Values
The library implements several binary formats that encode JSON in an efficient way. Most of these formats support binary values; that is, values that have semantics defined outside the library and only define a sequence of bytes to be stored.
JSON itself does not have a binary value. As such, binary values are an extension that this library implements to store values received by a binary format. Binary values are never created by the JSON parser and are only part of a serialized JSON text if they have been created manually or via a binary format.
API for binary values
classDiagram
class binary_t ["json::binary_t"] {
+void set_subtype(std::uint64_t subtype)
+void clear_subtype()
+std::uint64_t subtype() const
+bool has_subtype() const
}
class vector ["std::vector<uint8_t>"]
vector <|-- binary_t
By default, binary values are stored as std::vector<std::uint8_t>. This type can be changed by providing a template
parameter to the basic_json type. To store binary subtypes, the storage type is extended and exposed as
json::binary_t:
auto binary = json::binary_t({0xCA, 0xFE, 0xBA, 0xBE});
auto binary_with_subtype = json::binary_t({0xCA, 0xFE, 0xBA, 0xBE}, 42);
There are several convenience functions to check and set the subtype:
binary.has_subtype(); // returns false
binary_with_subtype.has_subtype(); // returns true
binary_with_subtype.clear_subtype();
binary_with_subtype.has_subtype(); // returns false
binary_with_subtype.set_subtype(42);
binary.set_subtype(23);
binary.subtype(); // returns 23
As json::binary_t is subclassing std::vector<std::uint8_t>, all member functions are available:
binary.size(); // returns 4
binary[1]; // returns 0xFE
JSON values can be constructed from json::binary_t:
json j = binary;
Binary values are primitive values just like numbers or strings, as reflected by
is_binary() and is_primitive():
j.is_binary(); // returns true
j.is_primitive(); // returns true
Given a binary JSON value, the binary_t can be accessed by reference via
get_binary():
j.get_binary().has_subtype(); // returns true
j.get_binary().size(); // returns 4
For convenience, binary JSON values can be constructed via json::binary:
auto j2 = json::binary({0xCA, 0xFE, 0xBA, 0xBE}, 23);
auto j3 = json::binary({0xCA, 0xFE, 0xBA, 0xBE});
j2 == j; // returns true
j3.get_binary().has_subtype(); // returns false
j3.get_binary().subtype(); // returns std::uint64_t(-1) as j3 has no subtype
Serialization
Binary values are serialized differently according to the formats.
JSON
JSON does not have a binary type, and this library does not introduce a new type as this would break conformance.
Instead, binary values are serialized as an object with two keys: bytes holds an array of integers, and subtype
is an integer or null.
??? example "Example: serialize a binary value to JSON"
Code:
```cpp
// create a binary value of subtype 42
json j;
j["binary"] = json::binary({0xCA, 0xFE, 0xBA, 0xBE}, 42);
// serialize to standard output
std::cout << j.dump(2) << std::endl;
```
Output:
```json
{
"binary": {
"bytes": [202, 254, 186, 190],
"subtype": 42
}
}
```
!!! warning "No roundtrip for binary values"
The JSON parser will not parse the objects generated by binary values back to binary values. This is by design to
remain standards compliant. Serializing binary values to JSON is only implemented for debugging purposes.
BJData
BJData neither supports binary values nor subtypes and proposes to serialize binary values as an array of uint8 values. The library implements this translation.
??? example "Example: serialize a binary value to BJData"
Code:
```cpp
// create a binary value of subtype 42 (will be ignored in BJData)
json j;
j["binary"] = json::binary({0xCA, 0xFE, 0xBA, 0xBE}, 42);
// convert to BJData
auto v = json::to_bjdata(j);
```
`v` is a `std::vector<std::uint8_t>` with the following 20 elements:
```c
0x7B // '{'
0x69 0x06 // i 6 (length of the key)
0x62 0x69 0x6E 0x61 0x72 0x79 // "binary"
0x5B // '['
0x55 0xCA 0x55 0xFE 0x55 0xBA 0x55 0xBE // content (each byte prefixed with 'U')
0x5D // ']'
0x7D // '}'
```
The following code uses the type and size optimization for BJData:
```cpp
// convert to BJData using the size and type optimization
auto v = json::to_bjdata(j, true, true);
```
The resulting vector has 22 elements; the optimization is not effective for examples with few values:
```c
0x7B // '{'
0x23 0x69 0x01 // '#' 'i' type of the array elements: unsigned integers
0x69 0x06 // i 6 (length of the key)
0x62 0x69 0x6E 0x61 0x72 0x79 // "binary"
0x5B // '[' array
0x24 0x55 // '$' 'U' type of the array elements: unsigned integers
0x23 0x69 0x04 // '#' i 4 number of array elements
0xCA 0xFE 0xBA 0xBE // content
```
Note that subtype (42) is **not** serialized and that BJData has **no binary type**, and deserializing `v` would
yield the following value:
```json
{
"binary": [202, 254, 186, 190]
}
```
BON8
BON8 neither supports binary values nor subtypes. The library serializes binary values as an array of integers.
??? example "Example: serialize a binary value to BON8"
Code:
```cpp
// create a binary value of subtype 42 (will be ignored in BON8)
json j;
j["binary"] = json::binary({0xCA, 0xFE, 0xBA, 0xBE}, 42);
// convert to BON8
auto v = json::to_bon8(j);
```
`v` is a `std::vector<std::uint8_t>` with the following 16 elements:
```c
0x87 // object with 1 member
0x62 0x69 0x6E 0x61 0x72 0x79 // "binary"
0x84 // array with 4 elements
0xC3 0x22 0xC3 0x56 0xC3 0x12 0xC3 0x16 // content (each byte as a 2-byte integer)
```
Note that the subtype is lost, and deserializing `v` would yield the following value:
```json
{
"binary": [202, 254, 186, 190]
}
```
BSON
BSON supports binary values and subtypes. If a subtype is given, it is used and added as an
unsigned 8-bit integer; a subtype above 255 throws
out_of_range.415. If no subtype is given, the generic binary
subtype 0x00 is used. The payload of the "old binary" subtype 0x02 is preceded by a second length, which the library
writes and removes again when reading; it is not part of the payload.
??? example "Example: serialize a binary value to BSON"
Code:
```cpp
// create a binary value of subtype 42
json j;
j["binary"] = json::binary({0xCA, 0xFE, 0xBA, 0xBE}, 42);
// convert to BSON
auto v = json::to_bson(j);
```
`v` is a `std::vector<std::uint8_t>` with the following 22 elements:
```c
0x16 0x00 0x00 0x00 // number of bytes in the document
0x05 // binary value
0x62 0x69 0x6E 0x61 0x72 0x79 0x00 // key "binary" + null byte
0x04 0x00 0x00 0x00 // number of bytes
0x2a // subtype
0xCA 0xFE 0xBA 0xBE // content
0x00 // end of the document
```
Note that the serialization preserves the subtype, and deserializing `v` would yield the following value:
```json
{
"binary": {
"bytes": [202, 254, 186, 190],
"subtype": 42
}
}
```
CBOR
CBOR supports binary values and subtypes. Any binary value will be serialized as a byte
string; a subtype is serialized as a tag preceding it. When parsing, tags are only read back as subtypes with
cbor_tag_handler_t::store (see from_cbor). The library will choose the smallest
representation using the length of the byte array.
??? example "Example: serialize a binary value to CBOR"
Code:
```cpp
// create a binary value of subtype 42
json j;
j["binary"] = json::binary({0xCA, 0xFE, 0xBA, 0xBE}, 42);
// convert to CBOR
auto v = json::to_cbor(j);
```
`v` is a `std::vector<std::uint8_t>` with the following 15 elements:
```c
0xA1 // map(1)
0x66 // text(6)
0x62 0x69 0x6E 0x61 0x72 0x79 // "binary"
0xD8 0x2A // tag(42)
0x44 // bytes(4)
0xCA 0xFE 0xBA 0xBE // content
```
Note that the subtype is serialized as tag. However, parsing tagged values yield a parse error unless
`json::cbor_tag_handler_t::ignore` or `json::cbor_tag_handler_t::store` is passed to
[`json::from_cbor`](../api/basic_json/from_cbor.md) (see
[`cbor_tag_handler_t`](../api/basic_json/cbor_tag_handler_t.md)).
With `ignore`, the subtype is dropped (`"subtype": null`); with `store`, the tag is kept (`"subtype": 42`):
```json
{
"binary": {
"bytes": [202, 254, 186, 190],
"subtype": null
}
}
```
```json
{
"binary": {
"bytes": [202, 254, 186, 190],
"subtype": 42
}
}
```
MessagePack
MessagePack supports binary values and subtypes. If a subtype is given, the ext family is used. The library will choose the smallest representation among fixext1, fixext2, fixext4, fixext8, fixext16, ext8, ext16, and ext32. The subtype is then written as the signed 8-bit ext type: subtypes 128 to 255 map to the negative (reserved) ext types, and on reading the ext type is exposed as an unsigned subtype from 0 to 255.
If no subtype is given, the bin family (bin8, bin16, bin32) is used.
??? example "Example: serialize a binary value to MessagePack"
Code:
```cpp
// create a binary value of subtype 42
json j;
j["binary"] = json::binary({0xCA, 0xFE, 0xBA, 0xBE}, 42);
// convert to MessagePack
auto v = json::to_msgpack(j);
```
`v` is a `std::vector<std::uint8_t>` with the following 14 elements:
```c
0x81 // fixmap1
0xA6 // fixstr6
0x62 0x69 0x6E 0x61 0x72 0x79 // "binary"
0xD6 // fixext4
0x2A // subtype
0xCA 0xFE 0xBA 0xBE // content
```
Note that the serialization preserves the subtype, and deserializing `v` would yield the following value:
```json
{
"binary": {
"bytes": [202, 254, 186, 190],
"subtype": 42
}
}
```
UBJSON
UBJSON neither supports binary values nor subtypes and proposes to serialize binary values as an array of uint8 values. The library implements this translation.
??? example "Example: serialize a binary value to UBJSON"
Code:
```cpp
// create a binary value of subtype 42 (will be ignored in UBJSON)
json j;
j["binary"] = json::binary({0xCA, 0xFE, 0xBA, 0xBE}, 42);
// convert to UBJSON
auto v = json::to_ubjson(j);
```
`v` is a `std::vector<std::uint8_t>` with the following 20 elements:
```c
0x7B // '{'
0x69 0x06 // i 6 (length of the key)
0x62 0x69 0x6E 0x61 0x72 0x79 // "binary"
0x5B // '['
0x55 0xCA 0x55 0xFE 0x55 0xBA 0x55 0xBE // content (each byte prefixed with 'U')
0x5D // ']'
0x7D // '}'
```
The following code uses the type and size optimization for UBJSON:
```cpp
// convert to UBJSON using the size and type optimization
auto v = json::to_ubjson(j, true, true);
```
The resulting vector has 23 elements; the optimization is not effective for examples with few values:
```c
0x7B // '{'
0x24 // '$' type of the object elements
0x5B // '[' array
0x23 0x69 0x01 // '#' i 1 number of object elements
0x69 0x06 // i 6 (length of the key)
0x62 0x69 0x6E 0x61 0x72 0x79 // "binary"
0x24 0x55 // '$' 'U' type of the array elements: unsigned integers
0x23 0x69 0x04 // '#' i 4 number of array elements
0xCA 0xFE 0xBA 0xBE // content
```
Note that subtype (42) is **not** serialized and that UBJSON has **no binary type**, and deserializing `v` would
yield the following value:
```json
{
"binary": [202, 254, 186, 190]
}
```