mirror of
https://github.com/nlohmann/json.git
synced 2026-09-30 22:15:19 +00:00
Add BON8 support (#2998)
* Add BON8 support Add to_bon8/from_bon8 and input_format_t::bon8 for BON8, a binary format that uses the byte values that cannot begin a UTF-8 character as type markers, so strings need no length prefix. It is the most compact of the supported binary formats on the benchmark files. The reader is non-recursive like the other binary readers. A string ends at the first byte that cannot continue it, so the reader hands the one or two bytes it reads past a string back to the value that follows. The writer produces the canonical representation of the specification, except for NFC normalization; its output is identical to that of the reference implementation (HikoGUI) on all files of the test data. The round-trip tests need the .bon8 files of json_test_data 3.2.0. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Address review comments - Reuse detail::validate_one_utf8 to check strings in to_bon8; the error now names the first byte of the invalid sequence. - Document that to_bon8 leaves bytes in the output adapter on an exception, and that string_open is only an output of write_bon8_marker. - Explain why the pushback buffer of the BON8 reader cannot overflow. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Select the BON8 float prefix by type get_bon8_float_prefix only depends on the type of its argument, so make the type a template parameter instead of passing an unused value. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Rename a test variable that Flawfinder mistakes for read() Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix the BON8 CI failures - compare the float in write_bon8_float with number_float_t constants, so GCC does not warn about a float-to-double conversion - mark check_bon8_utf8's context as used when exceptions are disabled - choose the compact float prefix in a helper rather than with nested conditional operators (clang-tidy) - use auto for the cast in the BON8 integer reader (clang-tidy) - write the int32 minimum test values as long long literals (MSVC C4146) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Amalgamate Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Read BON8 strings in bulk from contiguous input - copy the valid UTF-8 of a string in one step when the input is contiguous (twitter.json is read in 1.68 instead of 2.52 ms, jeopardy.json in 196 instead of 297 ms, close to CBOR and MessagePack) - share the new valid_utf8_prefix() with the writer's UTF-8 check, which now skips ASCII 8 bytes at a time - let the fuzzer check that contiguous and stream input give the same value or error, and test both paths in the unit tests - clarify that a second 0xFF after a string is an empty string Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Link the BON8 functions from the other binary format pages Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Name the bulk scan flag after the input, not BON8 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Read BSON keys in bulk from contiguous input BSON keys (and array indices) are C-style strings, which were read byte by byte. For contiguous input they are now read up to their \x00-byte in one step, using the same bulk_scan flag as BON8 strings: twitter.json is read in 1.46 instead of 2.01 ms, citm_catalog.json in 2.93 instead of 3.33 ms, jeopardy.json in 182 instead of 207 ms. canada.json, whose keys are almost all one-digit array indices, takes 2 % longer. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix the BON8 CI failures of the bulk-read tests - skip the contiguous-versus-stream tests of BON8 strings and BSON keys when exceptions are disabled: they catch the parse errors of invalid input, and without exceptions the library aborts instead - use static_cast for the int64 test value (google-readability-casting) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Move the explicit basic_json instantiation into its own test file Linking test-regression3_cpp20 with clang and MinGW failed with "relocation truncated to fit: IMAGE_REL_AMD64_REL32 against `.rdata'", as test-regression2 did before #5511. The explicit instantiation of basic_json<> for #4825 compiles every member function, including the BON8 reader and writer, into that object, and it was already close to the limit (2,226,104 bytes on develop, 2,234,960 with BON8; clang -O1, C++20). Give the instantiation a file of its own: unit-regression3 is now 1,594,736 bytes and unit-explicit_instantiation 1,095,064. The new file mentions JSON_HAS_CPP_17 and JSON_HAS_CPP_20 so it keeps being built for the C++17 standard the regression was about. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Convert the bytes of the BON8 test strings explicitly The str() helper constructed a std::string from a byte range, which converts each unsigned char implicitly; -fsanitize=integer reports that for bytes of 0x80 and above (ci_test_clang_sanitizer). Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me>
This commit is contained in:
Niels Lohmann
GitHub
@@ -25,6 +25,7 @@ TEST_CASE("Binary Formats" * doctest::skip())
|
||||
const auto bjdata_1_size = json::to_bjdata(j).size();
|
||||
const auto bjdata_2_size = json::to_bjdata(j, true).size();
|
||||
const auto bjdata_3_size = json::to_bjdata(j, true, true).size();
|
||||
const auto bon8_size = json::to_bon8(j).size();
|
||||
const auto bson_size = json::to_bson(j).size();
|
||||
const auto cbor_size = json::to_cbor(j).size();
|
||||
const auto msgpack_size = json::to_msgpack(j).size();
|
||||
@@ -36,6 +37,7 @@ TEST_CASE("Binary Formats" * doctest::skip())
|
||||
CHECK(bjdata_1_size == 1112030);
|
||||
CHECK(bjdata_2_size == 1224148);
|
||||
CHECK(bjdata_3_size == 1224148);
|
||||
CHECK(bon8_size == 1055792);
|
||||
CHECK(bson_size == 1794522);
|
||||
CHECK(cbor_size == 1055552);
|
||||
CHECK(msgpack_size == 1056145);
|
||||
@@ -47,6 +49,7 @@ TEST_CASE("Binary Formats" * doctest::skip())
|
||||
CHECK((100.0 * double(bjdata_1_size) / double(json_size)) == Approx(53.199));
|
||||
CHECK((100.0 * double(bjdata_2_size) / double(json_size)) == Approx(58.563));
|
||||
CHECK((100.0 * double(bjdata_3_size) / double(json_size)) == Approx(58.563));
|
||||
CHECK((100.0 * double(bon8_size) / double(json_size)) == Approx(50.509));
|
||||
CHECK((100.0 * double(bson_size) / double(json_size)) == Approx(85.849));
|
||||
CHECK((100.0 * double(cbor_size) / double(json_size)) == Approx(50.497));
|
||||
CHECK((100.0 * double(msgpack_size) / double(json_size)) == Approx(50.526));
|
||||
@@ -64,6 +67,7 @@ TEST_CASE("Binary Formats" * doctest::skip())
|
||||
const auto bjdata_1_size = json::to_bjdata(j).size();
|
||||
const auto bjdata_2_size = json::to_bjdata(j, true).size();
|
||||
const auto bjdata_3_size = json::to_bjdata(j, true, true).size();
|
||||
const auto bon8_size = json::to_bon8(j).size();
|
||||
const auto bson_size = json::to_bson(j).size();
|
||||
const auto cbor_size = json::to_cbor(j).size();
|
||||
const auto msgpack_size = json::to_msgpack(j).size();
|
||||
@@ -75,6 +79,7 @@ TEST_CASE("Binary Formats" * doctest::skip())
|
||||
CHECK(bjdata_1_size == 425342);
|
||||
CHECK(bjdata_2_size == 429970);
|
||||
CHECK(bjdata_3_size == 429970);
|
||||
CHECK(bon8_size == 391396);
|
||||
CHECK(bson_size == 444568);
|
||||
CHECK(cbor_size == 402814);
|
||||
CHECK(msgpack_size == 401510);
|
||||
@@ -86,6 +91,7 @@ TEST_CASE("Binary Formats" * doctest::skip())
|
||||
CHECK((100.0 * double(bjdata_1_size) / double(json_size)) == Approx(91.097));
|
||||
CHECK((100.0 * double(bjdata_2_size) / double(json_size)) == Approx(92.089));
|
||||
CHECK((100.0 * double(bjdata_3_size) / double(json_size)) == Approx(92.089));
|
||||
CHECK((100.0 * double(bon8_size) / double(json_size)) == Approx(83.828));
|
||||
CHECK((100.0 * double(bson_size) / double(json_size)) == Approx(95.215));
|
||||
CHECK((100.0 * double(cbor_size) / double(json_size)) == Approx(86.273));
|
||||
CHECK((100.0 * double(msgpack_size) / double(json_size)) == Approx(85.993));
|
||||
@@ -103,6 +109,7 @@ TEST_CASE("Binary Formats" * doctest::skip())
|
||||
const auto bjdata_1_size = json::to_bjdata(j).size();
|
||||
const auto bjdata_2_size = json::to_bjdata(j, true).size();
|
||||
const auto bjdata_3_size = json::to_bjdata(j, true, true).size();
|
||||
const auto bon8_size = json::to_bon8(j).size();
|
||||
const auto bson_size = json::to_bson(j).size();
|
||||
const auto cbor_size = json::to_cbor(j).size();
|
||||
const auto msgpack_size = json::to_msgpack(j).size();
|
||||
@@ -114,6 +121,7 @@ TEST_CASE("Binary Formats" * doctest::skip())
|
||||
CHECK(bjdata_1_size == 390781);
|
||||
CHECK(bjdata_2_size == 433557);
|
||||
CHECK(bjdata_3_size == 432964);
|
||||
CHECK(bon8_size == 317879);
|
||||
CHECK(bson_size == 479430);
|
||||
CHECK(cbor_size == 342373);
|
||||
CHECK(msgpack_size == 342473);
|
||||
@@ -125,6 +133,7 @@ TEST_CASE("Binary Formats" * doctest::skip())
|
||||
CHECK((100.0 * double(bjdata_1_size) / double(json_size)) == Approx(78.109));
|
||||
CHECK((100.0 * double(bjdata_2_size) / double(json_size)) == Approx(86.659));
|
||||
CHECK((100.0 * double(bjdata_3_size) / double(json_size)) == Approx(86.541));
|
||||
CHECK((100.0 * double(bon8_size) / double(json_size)) == Approx(63.538));
|
||||
CHECK((100.0 * double(bson_size) / double(json_size)) == Approx(95.828));
|
||||
CHECK((100.0 * double(cbor_size) / double(json_size)) == Approx(68.433));
|
||||
CHECK((100.0 * double(msgpack_size) / double(json_size)) == Approx(68.453));
|
||||
@@ -142,6 +151,7 @@ TEST_CASE("Binary Formats" * doctest::skip())
|
||||
const auto bjdata_1_size = json::to_bjdata(j).size();
|
||||
const auto bjdata_2_size = json::to_bjdata(j, true).size();
|
||||
const auto bjdata_3_size = json::to_bjdata(j, true, true).size();
|
||||
const auto bon8_size = json::to_bon8(j).size();
|
||||
const auto bson_size = json::to_bson({{"", j}}).size(); // wrap array in object for BSON
|
||||
const auto cbor_size = json::to_cbor(j).size();
|
||||
const auto msgpack_size = json::to_msgpack(j).size();
|
||||
@@ -153,6 +163,7 @@ TEST_CASE("Binary Formats" * doctest::skip())
|
||||
CHECK(bjdata_1_size == 50710965);
|
||||
CHECK(bjdata_2_size == 51144830);
|
||||
CHECK(bjdata_3_size == 51144830);
|
||||
CHECK(bon8_size == 45942080);
|
||||
CHECK(bson_size == 56008520);
|
||||
CHECK(cbor_size == 46187320);
|
||||
CHECK(msgpack_size == 46158575);
|
||||
@@ -164,6 +175,7 @@ TEST_CASE("Binary Formats" * doctest::skip())
|
||||
CHECK((100.0 * double(bjdata_1_size) / double(json_size)) == Approx(96.576));
|
||||
CHECK((100.0 * double(bjdata_2_size) / double(json_size)) == Approx(97.402));
|
||||
CHECK((100.0 * double(bjdata_3_size) / double(json_size)) == Approx(97.402));
|
||||
CHECK((100.0 * double(bon8_size) / double(json_size)) == Approx(87.494));
|
||||
CHECK((100.0 * double(bson_size) / double(json_size)) == Approx(106.665));
|
||||
CHECK((100.0 * double(cbor_size) / double(json_size)) == Approx(87.961));
|
||||
CHECK((100.0 * double(msgpack_size) / double(json_size)) == Approx(87.906));
|
||||
@@ -181,6 +193,7 @@ TEST_CASE("Binary Formats" * doctest::skip())
|
||||
const auto bjdata_1_size = json::to_bjdata(j).size();
|
||||
const auto bjdata_2_size = json::to_bjdata(j, true).size();
|
||||
const auto bjdata_3_size = json::to_bjdata(j, true, true).size();
|
||||
const auto bon8_size = json::to_bon8(j).size();
|
||||
// BSON cannot process the file as it contains code point U+0000
|
||||
const auto cbor_size = json::to_cbor(j).size();
|
||||
const auto msgpack_size = json::to_msgpack(j).size();
|
||||
@@ -192,6 +205,7 @@ TEST_CASE("Binary Formats" * doctest::skip())
|
||||
CHECK(bjdata_1_size == 148695);
|
||||
CHECK(bjdata_2_size == 150569);
|
||||
CHECK(bjdata_3_size == 150569);
|
||||
CHECK(bon8_size == 144477);
|
||||
CHECK(cbor_size == 147095);
|
||||
CHECK(msgpack_size == 147017);
|
||||
CHECK(ubjson_1_size == 148695);
|
||||
@@ -202,6 +216,7 @@ TEST_CASE("Binary Formats" * doctest::skip())
|
||||
CHECK((100.0 * double(bjdata_1_size) / double(json_size)) == Approx(88.153));
|
||||
CHECK((100.0 * double(bjdata_2_size) / double(json_size)) == Approx(89.264));
|
||||
CHECK((100.0 * double(bjdata_3_size) / double(json_size)) == Approx(89.264));
|
||||
CHECK((100.0 * double(bon8_size) / double(json_size)) == Approx(85.653));
|
||||
CHECK((100.0 * double(cbor_size) / double(json_size)) == Approx(87.205));
|
||||
CHECK((100.0 * double(msgpack_size) / double(json_size)) == Approx(87.158));
|
||||
CHECK((100.0 * double(ubjson_1_size) / double(json_size)) == Approx(88.153));
|
||||
|
||||
Reference in New Issue
Block a user