Scarfski/json
1
0
Fork 0
mirror of https://github.com/nlohmann/json.git synced 2026-09-29 13:35:45 +00:00
Code Issues Packages Projects Releases Wiki Activity

Fix BSON conformance issue (#5185)

Browse Source 
* 🐛 fix BSON conformance issue

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* 🐛 fix BSON conformance issue

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* 🐛 reject ill-formed UTF-8 in CBOR/MessagePack/BSON text strings at decode time (#5531)

from_cbor()/from_msgpack()/from_bson() copied the raw bytes of a decoded
text string into the resulting json value without any UTF-8 validation,
even though RFC 8949 §3.1 (CBOR) and the MessagePack/BSON specifications
all require text strings to be valid UTF-8. Malformed input only failed
later, if the value was dump()'d, with a type_error.316 - so the
allow_exceptions=false pattern used specifically to get a discarded
sentinel instead of an exception did not discard this category of
malformed input, unlike every other kind of malformed binary input this
library rejects at decode time (see #5529).

Fix this at the single choke point shared by BSON/CBOR/MessagePack/UBJSON
string reads, binary_reader::get_string(): validate the bytes with the
UTF-8 DFA right after they are read, and report failures the same way as
every other binary_reader error (parse_error.113), so allow_exceptions
and strict discarding behave consistently. get_binary()/binary blob reads
are untouched and still accept arbitrary bytes, since only text strings
are required to be UTF-8.

There were two independent implementations of a UTF-8 validator: the
lexer's streaming scanner, and the serializer's Hoehrmann DFA used by
dump_escaped_impl(). Rather than write a third, the serializer's decode()
function, its utf8d table and the UTF8_ACCEPT/UTF8_REJECT constants are
extracted into detail/string_utils.hpp (a low-level header already
included before both detail/input/ and detail/output/), alongside a new
is_valid_utf8() helper built on the same decode() step. serializer.hpp's
dump_escaped_impl() now calls the shared decode(), so there is exactly
one UTF-8 validator in the codebase; dump()'s exact type_error.316
messages and byte-index reporting are unchanged (see the added
regression-guard test in unit-serialization.cpp).

Claude-Session: https://claude.ai/code/session_01N4RQ1Ahan5YAGbnAQGjZTY

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

* ⚡ validate only newly read bytes of binary-format strings

get_string() validated the whole result after each call, but get_bytes()
appends to it and CBOR indefinite-length strings collect all chunks in
the same result, so every chunk re-validated everything read before it.
An input of many small chunks took quadratic time (80000 one-byte chunks,
160 KB of input, took about 7 seconds). Only the newly read bytes are
validated now, which also matches RFC 8949's requirement that every
chunk is valid UTF-8 on its own.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

---------

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Niels Lohmann
2026-09-25 20:45:28 +02:00
committed by GitHub
parent 632a5812a8
commit d19f7f5dce
12 changed files with 470 additions and 125 deletions
Download Patch File Download Diff File Expand all files Collapse all files

73
tests/src/unit-bson.cpp
View File

@@ -1697,3 +1697,76 @@ TEST_CASE("BSON roundtrips" * doctest::skip())
}
}
}
TEST_CASE("Invalid document size handling")
{
SECTION("document size must be at least 5")
{
std::vector<std::uint8_t> const v = {0x04, 0x00, 0x00, 0x00, 0x00};
json _;
CHECK_THROWS_WITH_AS(_ = json::from_bson(v), "[json.exception.parse_error.112] parse error at byte 5: syntax error while parsing BSON document: document size 4 does not match the number of bytes read (5)", json::parse_error&);
CHECK(json::from_bson(v, true, false).is_discarded());
}
SECTION("declared document size must match consumed bytes (extra trailing element)")
{
// Declares 5-byte empty document but appends an int32 element after the declared end.
std::vector<std::uint8_t> const v =
{
0x05, 0x00, 0x00, 0x00,
0x10, 'a', 'd', 'm', 'i', 'n', 0x00,
0x01, 0x00, 0x00, 0x00,
0x00
};
json _;
CHECK_THROWS_WITH_AS(_ = json::from_bson(v), "[json.exception.parse_error.112] parse error at byte 16: syntax error while parsing BSON document: document size 5 does not match the number of bytes read (16)", json::parse_error&);
CHECK(json::from_bson(v, true, false).is_discarded());
}
SECTION("declared document size must match consumed bytes (premature terminator)")
{
// Declares 32-byte document but only contains the size field followed by an immediate terminator.
std::vector<std::uint8_t> const v =
{
0x20, 0x00, 0x00, 0x00,
0x00
};
json _;
CHECK_THROWS_WITH_AS(_ = json::from_bson(v), "[json.exception.parse_error.112] parse error at byte 5: syntax error while parsing BSON document: document size 32 does not match the number of bytes read (5)", json::parse_error&);
CHECK(json::from_bson(v, true, false).is_discarded());
}
SECTION("array declared size must match consumed bytes")
{
// Outer object contains an array "a" that declares 5 bytes (empty) but
// actually contains an int32 element before its terminator.
std::vector<std::uint8_t> const v =
{
0x14, 0x00, 0x00, 0x00, // object size = 20
0x04, 'a', 0x00, // key "a", array type
0x05, 0x00, 0x00, 0x00, // array declared size = 5 (empty)
0x10, '0', 0x00, 0x01, 0x00, 0x00, 0x00, // extra int32 element "0" = 1
0x00, // array terminator
0x00 // object terminator
};
json _;
CHECK_THROWS_WITH_AS(_ = json::from_bson(v), "[json.exception.parse_error.112] parse error at byte 19: syntax error while parsing BSON document: document size 5 does not match the number of bytes read (12)", json::parse_error&);
CHECK(json::from_bson(v, true, false).is_discarded());
}
SECTION("BSON string must end with 0x00")
{
// Length-prefixed string whose terminator byte is 'X' (0x58), not 0x00.
std::vector<std::uint8_t> const v =
{
0x0F, 0x00, 0x00, 0x00,
0x02, 's', 0x00,
0x02, 0x00, 0x00, 0x00,
'A', 'X',
0x00
};
json _;
CHECK_THROWS_WITH_AS(_ = json::from_bson(v), "[json.exception.parse_error.112] parse error at byte 13: syntax error while parsing BSON string: BSON string is not null-terminated", json::parse_error&);
CHECK(json::from_bson(v, true, false).is_discarded());
}
}