mirror of
https://github.com/nlohmann/json.git
synced 2026-11-15 20:16:07 +00:00
* Bound the total number of valueless UBJSON elements per input An optimized UBJSON array of type Z, T or F has no payload, so its declared count is not bounded by the input. The limit of 1048576 applied to each array on its own, so 9 bytes (`[$Z#l` and a count) produced 1048576 values and every further 9 bytes another 1048576: 362 bytes allocated 661 MB in a DOM, and a SAX handler received 1M events per 9 bytes. The reader now adds the counts of all such arrays of one input and rejects the input with out_of_range.408 when the sum exceeds 1048576 plus 4 for every byte read so far. The base is the per-array limit, so an array that was accepted on its own before is still accepted at the start of an input. These arrays add about 80 bytes of memory per input byte (measured 77; one `Z` per byte needs 32), plus the base; the memory of a document as a whole is not bounded by this. The check runs once per valueless array, before its start event, with plain size_t arithmetic that cannot overflow for a 32-bit size_t. Containers with a payload type and objects (every member needs a key) are bounded by the input already and are unchanged. BJData rejects these types. to_ubjson counts the elements of the typed arrays it writes in one call and writes an array in the unoptimized form (`[#n Z Z ...`) once it would bring the total above 1048576, which a reader accepts at any input size. The check is made only for a typed array of a valueless type, so BJData and other arrays do not pay for it. Everything to_ubjson writes in version 3.13.0 can therefore be read back. Two consequences: - to_ubjson output changes for values with more than 1048576 elements in arrays of type Z, T or F: such an array is written at one byte per element, e.g. 270000 rows of 100 nulls take 26.8 MB instead of 1.35 MB. - Output of version 3.12.0 and earlier (typed arrays without such a total) is rejected by from_ubjson when the arrays hold more than 1048576 elements plus 4 per byte of input. 3.12.0 writes a row of up to 255 nulls in 5 bytes (6 above 255), so a table is rejected from 1075 rows of 1000 nulls or 209723 rows of 25 (accepted: 1000 rows of 1000, 209000 rows of 25); rows of 20 or fewer are never rejected. Measured on the test data: no file of json_test_data (converted with to_ubjson in all three modes, 846 round trips) holds an array of this kind with more than 0.17 values per byte. All of them are read back. Reproducer (K arrays of 1048576 nulls), Apple clang -O2: K=40 (362 bytes): before 41943040 values, 664 MB; after rejected, 33 MB K=2 (20 bytes): before 2097152 values, 53 MB; after rejected, 33 MB The new tests count events with a SAX handler where they can: building and comparing million-element values made the first version take 176 s under Valgrind, the current one about 22 s. Behaviour change: an input that holds more than 1048576 elements in arrays of type Z, T or F at more than 4 per byte is now rejected. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Report an empty array for UBJSON `[$N#<n>` instead of n elements A no-op (N) is not a value, so an optimized array of type N holds no element whatever count it declares. The reader skipped the elements but still passed the declared count to the SAX parser: a handler was told start_array(n) and no element followed, and the DOM parser reserved min(n, 16384) values (256 KB) for an array that stays empty. Nine bytes (`[$N#l 0x7FFFFFFF`) therefore committed 256 KB: 2000 of them in one input, 18 KB, committed 512 MB of heap (capacity 163,840,000 values for the 10000 arrays of the original probe: 2.6 GB for 70 KB of input). Untouched pages keep the resident size small, but the commit charge and a 32-bit address space are consumed. The reader now announces 0 elements for an optimized array of type N. The loop that skips the elements is unchanged (it never ran). A count that does not fit into std::size_t is still rejected by the count reader; a count that is too large for the array type of the DOM (before: out_of_range.408 "excessive array size" from the DOM handler) is accepted now, because the array is empty. Behaviour change: sax_parse reports start_array(0) for `[$N#n` (before: n); from_ubjson no longer throws out_of_range.408 for an N-typed array whose count exceeds the maximum size of the array type. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Bound what the DOM parsers reserve for arrays that announce a size Cause: #5476 made the DOM parsers reserve storage (up to 16384 elements) for a counted array of CBOR, MessagePack, UBJSON and BJData when it opens, before any input backs the count. The arrays that are open at the same time keep their storage until the parse ends, so N nested headers that announce 16384 elements and are never completed hold N * 16384 * sizeof(json): 100,000 nested CBOR headers `99 40 00` (300,000 bytes) peaked at 1.65 GiB, 3.12.0 at 12 MiB; the same for MessagePack `DC 40 00`, UBJSON `[#I 16384` and BJData. Fix: json_sax_dom_error_state, the base of the three DOM parsers, keeps a credit of elements that may still be reserved: 65,536 at the start plus 2 for every byte the reader has read. An array that announces no more than the credit left is reserved as before (one comparison and one subtraction per counted array); a larger one is reserved what is left and grows as its elements arrive. binary_reader tells a DOM parser (a handler whose pointer converts to json_sax_dom_error_state*; a handler that derives from one privately or protectedly, or any other SAX handler, is not told anything and reserves as in the base of this change) where to find its count of characters read for the duration of sax_parse(). An RAII guard puts back the previous binding on every exit, also by exception, so the parser never keeps a pointer to a reader that is gone (unbound, only the per-array cap of 16384 applies); the public SAX interface is unchanged, and start_array() still reports the announced number. A bound per array by the remaining input would not help, since the same bytes would back every open level; the bound has to be over all arrays. The elements of an array take at least one byte of input each, so the credit is more than honest input needs: none of the 19 inputs measured (canada, citm_catalog, twitter, mesh, update-center, random, numbers, animation, ik_rig, instruments, geo_response, jeopardy and 7 synthetic ones, each as CBOR, MessagePack, UBJSON and BJData, with and without size/type optimization) was given less than it announced. A value that nests five or more arrays of 16384 elements, each with an array as its first element, is the exception: the later ones grow as before. Measurements (clang -O2, macOS arm64, peak memory footprint of `/usr/bin/time -l`, reproducer: n nested headers): n = 20,000 n = 100,000 develop fix 3.12.0 develop fix 3.12.0 CBOR 338.7 5.9 8.2 1687.7 23.3 11.8 MiB MessagePack 338.9 6.0 7.3 1687.9 23.8 11.9 MiB UBJSON 339.0 7.6 8.0 1687.9 30.1 11.6 MiB BJData 339.0 7.3 7.9 1688.0 30.8 11.6 MiB That is at most (65,536 + 2 per input byte) values: 1 MiB plus 32 bytes per input byte with 16-byte values (a value is 24 bytes with JSON_DIAGNOSTICS and up to 40 with JSON_DIAGNOSTIC_POSITIONS), as documented in untrusted_input.md. Lower fixed caps do not get there: a cap of 64 elements still peaks at 113 MiB (n = 100,000) and costs 4.7 % on arrays of 300 elements (mat300.cbor) and 5.0 % on arrays of 10,000 (arrays_of_big.cbor). Reader instructions retired (min of 7, 108 rows: 18 inputs in the 6 forms CBOR, MessagePack, UBJSON and BJData plain and optimized; jeopardy, 46 MB, is within 0.1 %), compared with the base of this change: -0.19 % to +0.37 % in the formats that announce sizes (canada.cbor 81.9 M -> 82.1 M, many_small.cbor 333.5 M -> 334.2 M); the benefit of #5476 is kept (without any reserve canada.cbor takes 116.8 M, +42.6 %, many_small.cbor +71.5 %, mat300.cbor +81.6 %). Rows of plain UBJSON and BJData, whose arrays announce no size, move by -2.7 % to +2.6 % without any code of theirs changing: the base alternates between two instruction counts per process (452 M and 463 M for many_pairs.ubjson, equal in cycles), and a struct with three unused members added to the same class reproduces the 474 M mode. Tests (CBOR, MessagePack, UBJSON, BJData): 200 nested headers that announce 16384 elements peak at 52,433,600 bytes of DOM allocations on the base and below 128 Ki values with the fix (scaled by sizeof(value), so that it holds with the diagnostic macros) (counting allocator, no process memory measured); an array that completes is still reserved in one allocation; CBOR also checks that a DOM parser is not bound to a finished parse (ASan: stack-use-after-scope without the guard), that wrappers derived publicly (also with a parse inside the parse), privately and protectedly compile and are bound or not as described, that an array that got less than it announced decodes and that a SAX handler is told the announced size. The new tests take 4.6 s under Valgrind at -O0 (gcc:12, all four files). Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Clarify the reservation note for announced array counts Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Report an empty array for UBJSON [$N#<n>] instead of n elements (#6079) * Report an empty array for UBJSON `[$N#<n>` instead of n elements A no-op (N) is not a value, so an optimized array of type N holds no element whatever count it declares. The reader skipped the elements but still passed the declared count to the SAX parser: a handler was told start_array(n) and no element followed, and the DOM parser reserved min(n, 16384) values (256 KB) for an array that stays empty. Nine bytes (`[$N#l 0x7FFFFFFF`) therefore committed 256 KB: 2000 of them in one input, 18 KB, committed 512 MB of heap (capacity 163,840,000 values for the 10000 arrays of the original probe: 2.6 GB for 70 KB of input). Untouched pages keep the resident size small, but the commit charge and a 32-bit address space are consumed. The reader now announces 0 elements for an optimized array of type N. The loop that skips the elements is unchanged (it never ran). A count that does not fit into std::size_t is still rejected by the count reader; a count that is too large for the array type of the DOM (before: out_of_range.408 "excessive array size" from the DOM handler) is accepted now, because the array is empty. Behaviour change: sax_parse reports start_array(0) for `[$N#n` (before: n); from_ubjson no longer throws out_of_range.408 for an N-typed array whose count exceeds the maximum size of the array type. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Clarify the reservation note for announced array counts Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Add JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE to set the UBJSON whole-input budget (#6083) * Add JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE to set the UBJSON whole-input budget The whole-input budget for the elements of optimized UBJSON arrays of type Z, T and F (1048576 plus 4 per byte read) rejects files written by 3.12.0 that have long arrays of these types, such as 1100 rows of 1000 nulls with a `$Z` annotation, and there was no way to read them. The per-byte rate is now the macro JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE. Semantics: - default 4, so the default build is unchanged (the generated code of from_ubjson is identical at -O2 up to source line numbers); - the value must be a positive integer: 0, negative, non-integral, bool, character and wider-than-uintmax_t values fail to compile with a static_assert that names the macro; - a value that does not fit into std::size_t is the largest size_t, so one spelling works for 32- and 64-bit targets; - the limit base + rate * bytes_read saturates at the largest size_t instead of wrapping around, for any rate and input size; a rate of 131072 (2^20 / 8) or more cannot be exceeded by any input (the shortest header of an array of more than 32767 elements is 8 bytes), which switches the whole-input budget off; - the limit of one array (1048576) and the base (1048576) stay fixed constants, so the writer fallback still follows the base only and a reader with any rate reads everything the writer produces; - the macro is not part of the ABI tag and must have the same value in every translation unit; it is documented that way. The memory the arrays can take in a DOM is about 20 bytes per rate and byte of input (measured, macOS, containers of 9 * rate elements: 25 bytes per input byte at rate 1, 81 at 4, 312 at 16, 1226 at 64, 5493 at 256) plus about 33 MB for the base. A 3.12.0 row of N values has a header of b bytes (5 up to 255, 6 up to 32767, 8 above), so a table is read completely if N <= rate * b, and otherwise up to (1048576 + h * rate, h = outer header: 6, 7 or 9 bytes) / (N - rate * b) rows; 1100 rows of 1000 nulls need a rate of 8. The new page of the macro documents the formulas, the measurements, and two examples that are run by the documentation checks. The tests of the budget move to tests/src/unit-ubjson_valueless.cpp, which derives every boundary from the macro and is built with the rates 1, 1000000 and 0xFFFFFFFFFFFFFFFF in addition to the default (the last two check that nothing is refused and that the limit does not wrap around); they fail with the unchanged header for rate 1, 1000000 and the maximum. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Refuse char8_t for JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE and clarify its documentation Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Use fseek instead of rewind in the UBJSON FILE* test clang-tidy (bugprone-unsafe-functions) rejects rewind() because it has no error detection. Check the result of fseek() instead. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Charge an announced array only what is reserved for it An array that announced more than reserve_cap() elements but no more than the credit left was charged the announced count, while reserve_array() reserved at most reserve_cap() of them. An outer array of 20000 elements therefore left a third nested array of 16384 with 12796 instead of the full reservation. reserve_count() now caps the count first and charges that, and the DOM parsers reserve it through reserve_array_capped(), which does not cap again, so the fast path costs the same as before (instructions retired within +-0.06 % on canada, citm_catalog and twitter in CBOR, MessagePack and UBJSON). Also re-wrap the comment of valueless_elements_base. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me>
3717 lines
165 KiB
C++
3717 lines
165 KiB
C++
// __ _____ _____ _____
|
|
// __| | __| | | | JSON for Modern C++ (supporting code)
|
|
// | | |__ | | | | | | version 3.12.0
|
|
// |_____|_____|_____|_|___| https://github.com/nlohmann/json
|
|
//
|
|
// SPDX-FileCopyrightText: 2013-2026 Niels Lohmann <https://nlohmann.me>
|
|
// SPDX-License-Identifier: MIT
|
|
|
|
#include "doctest_compatibility.h"
|
|
|
|
// capture whether JSON_DELETE_DEPRECATED_FUNCTIONS was enabled on the command
|
|
// line *before* including json.hpp, since the library #undefs it once the header
|
|
// has been fully processed (see include/nlohmann/detail/macro_unscope.hpp); the
|
|
// tests of deprecated functions are skipped if these functions are deleted
|
|
#if defined(JSON_DELETE_DEPRECATED_FUNCTIONS) && (JSON_DELETE_DEPRECATED_FUNCTIONS == 1)
|
|
#define JSON_TEST_DEPRECATED_FUNCTIONS_DELETED
|
|
#endif
|
|
|
|
#include <nlohmann/json.hpp>
|
|
using nlohmann::json;
|
|
|
|
#include <iostream>
|
|
#include <fstream>
|
|
#include <list>
|
|
#include <set>
|
|
#include "make_test_data_available.hpp"
|
|
#include "round_trip_corpus.hpp"
|
|
#include "test_utils.hpp"
|
|
#include "sax_countdown.hpp"
|
|
#include "allocation_counter.hpp"
|
|
#include "sax_event_loggers.hpp"
|
|
#include "ubjson_valueless.hpp"
|
|
using utils::SaxCountdown;
|
|
using utils::SaxEventLogger;
|
|
|
|
|
|
TEST_CASE("UBJSON")
|
|
{
|
|
SECTION("individual values")
|
|
{
|
|
SECTION("discarded")
|
|
{
|
|
// a discarded value cannot be serialized to UBJSON
|
|
json const j = json::value_t::discarded;
|
|
CHECK_THROWS_WITH_AS(json::to_ubjson(j), "[json.exception.type_error.321] cannot serialize discarded value to UBJSON", json::type_error&);
|
|
}
|
|
|
|
SECTION("discarded values nested in a container")
|
|
{
|
|
json const discarded = json::value_t::discarded;
|
|
|
|
SECTION("in an array")
|
|
{
|
|
json const j = {1, discarded, 2};
|
|
#if JSON_DIAGNOSTICS
|
|
CHECK_THROWS_WITH_AS(json::to_ubjson(j), "[json.exception.type_error.321] (/1) cannot serialize discarded value to UBJSON", json::type_error&);
|
|
#else
|
|
CHECK_THROWS_WITH_AS(json::to_ubjson(j), "[json.exception.type_error.321] cannot serialize discarded value to UBJSON", json::type_error&);
|
|
#endif
|
|
}
|
|
|
|
SECTION("as an object value")
|
|
{
|
|
json j;
|
|
j["a"] = 1;
|
|
j["b"] = discarded;
|
|
#if JSON_DIAGNOSTICS
|
|
CHECK_THROWS_WITH_AS(json::to_ubjson(j), "[json.exception.type_error.321] (/b) cannot serialize discarded value to UBJSON", json::type_error&);
|
|
#else
|
|
CHECK_THROWS_WITH_AS(json::to_ubjson(j), "[json.exception.type_error.321] cannot serialize discarded value to UBJSON", json::type_error&);
|
|
#endif
|
|
}
|
|
|
|
SECTION("nested deeper (array in object in array)")
|
|
{
|
|
json inner_array = {1, discarded};
|
|
json middle_object;
|
|
middle_object["x"] = inner_array;
|
|
json const j = {middle_object};
|
|
#if JSON_DIAGNOSTICS
|
|
CHECK_THROWS_WITH_AS(json::to_ubjson(j), "[json.exception.type_error.321] (/0/x/1) cannot serialize discarded value to UBJSON", json::type_error&);
|
|
#else
|
|
CHECK_THROWS_WITH_AS(json::to_ubjson(j), "[json.exception.type_error.321] cannot serialize discarded value to UBJSON", json::type_error&);
|
|
#endif
|
|
}
|
|
|
|
SECTION("optimized array of all-discarded elements")
|
|
{
|
|
json const j = {discarded, discarded};
|
|
#if JSON_DIAGNOSTICS
|
|
CHECK_THROWS_WITH_AS(json::to_ubjson(j, true, true), "[json.exception.type_error.321] (/0) cannot serialize discarded value to UBJSON", json::type_error&);
|
|
#else
|
|
CHECK_THROWS_WITH_AS(json::to_ubjson(j, true, true), "[json.exception.type_error.321] cannot serialize discarded value to UBJSON", json::type_error&);
|
|
#endif
|
|
}
|
|
}
|
|
|
|
SECTION("null")
|
|
{
|
|
json const j = nullptr;
|
|
std::vector<uint8_t> expected = {'Z'};
|
|
const auto result = json::to_ubjson(j);
|
|
CHECK(result == expected);
|
|
|
|
// roundtrip
|
|
CHECK(json::from_ubjson(result) == j);
|
|
CHECK(json::from_ubjson(result, true, false) == j);
|
|
}
|
|
|
|
SECTION("boolean")
|
|
{
|
|
SECTION("true")
|
|
{
|
|
json const j = true;
|
|
std::vector<uint8_t> const expected = {'T'};
|
|
const auto result = json::to_ubjson(j);
|
|
CHECK(result == expected);
|
|
|
|
// roundtrip
|
|
CHECK(json::from_ubjson(result) == j);
|
|
CHECK(json::from_ubjson(result, true, false) == j);
|
|
}
|
|
|
|
SECTION("false")
|
|
{
|
|
json const j = false;
|
|
std::vector<uint8_t> const expected = {'F'};
|
|
const auto result = json::to_ubjson(j);
|
|
CHECK(result == expected);
|
|
|
|
// roundtrip
|
|
CHECK(json::from_ubjson(result) == j);
|
|
CHECK(json::from_ubjson(result, true, false) == j);
|
|
}
|
|
}
|
|
|
|
SECTION("number")
|
|
{
|
|
SECTION("signed")
|
|
{
|
|
SECTION("-9223372036854775808..-2147483649 (int64)")
|
|
{
|
|
std::vector<int64_t> const numbers
|
|
{
|
|
(std::numeric_limits<int64_t>::min)(),
|
|
-1000000000000000000LL,
|
|
-100000000000000000LL,
|
|
-10000000000000000LL,
|
|
-1000000000000000LL,
|
|
-100000000000000LL,
|
|
-10000000000000LL,
|
|
-1000000000000LL,
|
|
-100000000000LL,
|
|
-10000000000LL,
|
|
-2147483649LL,
|
|
};
|
|
for (auto i : numbers)
|
|
{
|
|
CAPTURE(i)
|
|
|
|
// create JSON value with integer number
|
|
json const j = i;
|
|
|
|
// check type
|
|
CHECK(j.is_number_integer());
|
|
|
|
// create expected byte vector
|
|
std::vector<uint8_t> const expected
|
|
{
|
|
static_cast<uint8_t>('L'),
|
|
static_cast<uint8_t>((i >> 56) & 0xff),
|
|
static_cast<uint8_t>((i >> 48) & 0xff),
|
|
static_cast<uint8_t>((i >> 40) & 0xff),
|
|
static_cast<uint8_t>((i >> 32) & 0xff),
|
|
static_cast<uint8_t>((i >> 24) & 0xff),
|
|
static_cast<uint8_t>((i >> 16) & 0xff),
|
|
static_cast<uint8_t>((i >> 8) & 0xff),
|
|
static_cast<uint8_t>(i & 0xff),
|
|
};
|
|
|
|
// compare result + size
|
|
const auto result = json::to_ubjson(j);
|
|
CHECK(result == expected);
|
|
CHECK(result.size() == 9);
|
|
|
|
// check individual bytes
|
|
CHECK(result[0] == 'L');
|
|
int64_t const restored = (static_cast<int64_t>(result[1]) << 070) +
|
|
(static_cast<int64_t>(result[2]) << 060) +
|
|
(static_cast<int64_t>(result[3]) << 050) +
|
|
(static_cast<int64_t>(result[4]) << 040) +
|
|
(static_cast<int64_t>(result[5]) << 030) +
|
|
(static_cast<int64_t>(result[6]) << 020) +
|
|
(static_cast<int64_t>(result[7]) << 010) +
|
|
static_cast<int64_t>(result[8]);
|
|
CHECK(restored == i);
|
|
|
|
// roundtrip
|
|
CHECK(json::from_ubjson(result) == j);
|
|
CHECK(json::from_ubjson(result, true, false) == j);
|
|
}
|
|
}
|
|
|
|
SECTION("-2147483648..-32769 (int32)")
|
|
{
|
|
std::vector<int32_t> numbers;
|
|
numbers.push_back(-32769);
|
|
numbers.push_back(-100000);
|
|
numbers.push_back(-1000000);
|
|
numbers.push_back(-10000000);
|
|
numbers.push_back(-100000000);
|
|
numbers.push_back(-1000000000);
|
|
numbers.push_back(-2147483647 - 1); // https://stackoverflow.com/a/29356002/266378
|
|
for (auto i : numbers)
|
|
{
|
|
CAPTURE(i)
|
|
|
|
// create JSON value with integer number
|
|
json const j = i;
|
|
|
|
// check type
|
|
CHECK(j.is_number_integer());
|
|
|
|
// create expected byte vector
|
|
std::vector<uint8_t> const expected
|
|
{
|
|
static_cast<uint8_t>('l'),
|
|
static_cast<uint8_t>((i >> 24) & 0xff),
|
|
static_cast<uint8_t>((i >> 16) & 0xff),
|
|
static_cast<uint8_t>((i >> 8) & 0xff),
|
|
static_cast<uint8_t>(i & 0xff),
|
|
};
|
|
|
|
// compare result + size
|
|
const auto result = json::to_ubjson(j);
|
|
CHECK(result == expected);
|
|
CHECK(result.size() == 5);
|
|
|
|
// check individual bytes
|
|
CHECK(result[0] == 'l');
|
|
int32_t const restored = (static_cast<int32_t>(result[1]) << 030) +
|
|
(static_cast<int32_t>(result[2]) << 020) +
|
|
(static_cast<int32_t>(result[3]) << 010) +
|
|
static_cast<int32_t>(result[4]);
|
|
CHECK(restored == i);
|
|
|
|
// roundtrip
|
|
CHECK(json::from_ubjson(result) == j);
|
|
CHECK(json::from_ubjson(result, true, false) == j);
|
|
}
|
|
}
|
|
|
|
SECTION("-32768..-129 (int16)")
|
|
{
|
|
for (int32_t i = -32768; i <= -129; i = utils::next_integer_sample(i, -129, 7))
|
|
{
|
|
CAPTURE(i)
|
|
|
|
// create JSON value with integer number
|
|
json const j = i;
|
|
|
|
// check type
|
|
CHECK(j.is_number_integer());
|
|
|
|
// create expected byte vector
|
|
std::vector<uint8_t> const expected
|
|
{
|
|
static_cast<uint8_t>('I'),
|
|
static_cast<uint8_t>((i >> 8) & 0xff),
|
|
static_cast<uint8_t>(i & 0xff),
|
|
};
|
|
|
|
// compare result + size
|
|
const auto result = json::to_ubjson(j);
|
|
CHECK(result == expected);
|
|
CHECK(result.size() == 3);
|
|
|
|
// check individual bytes
|
|
CHECK(result[0] == 'I');
|
|
auto const restored = static_cast<int16_t>(((result[1] << 8) + result[2]));
|
|
CHECK(restored == i);
|
|
|
|
// roundtrip
|
|
CHECK(json::from_ubjson(result) == j);
|
|
CHECK(json::from_ubjson(result, true, false) == j);
|
|
}
|
|
}
|
|
|
|
SECTION("-9263 (int16)")
|
|
{
|
|
json const j = -9263;
|
|
std::vector<uint8_t> expected = {'I', 0xdb, 0xd1};
|
|
|
|
// compare result + size
|
|
const auto result = json::to_ubjson(j);
|
|
CHECK(result == expected);
|
|
CHECK(result.size() == 3);
|
|
|
|
// check individual bytes
|
|
CHECK(result[0] == 'I');
|
|
auto const restored = static_cast<int16_t>(((result[1] << 8) + result[2]));
|
|
CHECK(restored == -9263);
|
|
|
|
// roundtrip
|
|
CHECK(json::from_ubjson(result) == j);
|
|
CHECK(json::from_ubjson(result, true, false) == j);
|
|
}
|
|
|
|
SECTION("-128..-1 (int8)")
|
|
{
|
|
for (auto i = -128; i <= -1; ++i)
|
|
{
|
|
CAPTURE(i)
|
|
|
|
// create JSON value with integer number
|
|
json const j = i;
|
|
|
|
// check type
|
|
CHECK(j.is_number_integer());
|
|
|
|
// create expected byte vector
|
|
std::vector<uint8_t> const expected
|
|
{
|
|
'i',
|
|
static_cast<uint8_t>(i),
|
|
};
|
|
|
|
// compare result + size
|
|
const auto result = json::to_ubjson(j);
|
|
CHECK(result == expected);
|
|
CHECK(result.size() == 2);
|
|
|
|
// check individual bytes
|
|
CHECK(result[0] == 'i');
|
|
CHECK(static_cast<int8_t>(result[1]) == i);
|
|
|
|
// roundtrip
|
|
CHECK(json::from_ubjson(result) == j);
|
|
CHECK(json::from_ubjson(result, true, false) == j);
|
|
}
|
|
}
|
|
|
|
SECTION("0..127 (int8)")
|
|
{
|
|
for (size_t i = 0; i <= 127; ++i)
|
|
{
|
|
CAPTURE(i)
|
|
|
|
// create JSON value with integer number
|
|
json j = -1;
|
|
j.get_ref<json::number_integer_t&>() = static_cast<json::number_integer_t>(i);
|
|
|
|
// check type
|
|
CHECK(j.is_number_integer());
|
|
|
|
// create expected byte vector
|
|
std::vector<uint8_t> const expected
|
|
{
|
|
static_cast<uint8_t>('i'),
|
|
static_cast<uint8_t>(i),
|
|
};
|
|
|
|
// compare result + size
|
|
const auto result = json::to_ubjson(j);
|
|
CHECK(result == expected);
|
|
CHECK(result.size() == 2);
|
|
|
|
// check individual bytes
|
|
CHECK(result[0] == 'i');
|
|
CHECK(result[1] == i);
|
|
|
|
// roundtrip
|
|
CHECK(json::from_ubjson(result) == j);
|
|
CHECK(json::from_ubjson(result, true, false) == j);
|
|
}
|
|
}
|
|
|
|
SECTION("128..255 (uint8)")
|
|
{
|
|
for (size_t i = 128; i <= 255; ++i)
|
|
{
|
|
CAPTURE(i)
|
|
|
|
// create JSON value with integer number
|
|
json j = -1;
|
|
j.get_ref<json::number_integer_t&>() = static_cast<json::number_integer_t>(i);
|
|
|
|
// check type
|
|
CHECK(j.is_number_integer());
|
|
|
|
// create expected byte vector
|
|
std::vector<uint8_t> const expected
|
|
{
|
|
static_cast<uint8_t>('U'),
|
|
static_cast<uint8_t>(i),
|
|
};
|
|
|
|
// compare result + size
|
|
const auto result = json::to_ubjson(j);
|
|
CHECK(result == expected);
|
|
CHECK(result.size() == 2);
|
|
|
|
// check individual bytes
|
|
CHECK(result[0] == 'U');
|
|
CHECK(result[1] == i);
|
|
|
|
// roundtrip
|
|
CHECK(json::from_ubjson(result) == j);
|
|
CHECK(json::from_ubjson(result, true, false) == j);
|
|
}
|
|
}
|
|
|
|
SECTION("256..32767 (int16)")
|
|
{
|
|
for (size_t i = 256; i <= 32767; i = utils::next_integer_sample(i, static_cast<size_t>(32767), static_cast<size_t>(7)))
|
|
{
|
|
CAPTURE(i)
|
|
|
|
// create JSON value with integer number
|
|
json j = -1;
|
|
j.get_ref<json::number_integer_t&>() = static_cast<json::number_integer_t>(i);
|
|
|
|
// check type
|
|
CHECK(j.is_number_integer());
|
|
|
|
// create expected byte vector
|
|
std::vector<uint8_t> const expected
|
|
{
|
|
static_cast<uint8_t>('I'),
|
|
static_cast<uint8_t>((i >> 8) & 0xff),
|
|
static_cast<uint8_t>(i & 0xff),
|
|
};
|
|
|
|
// compare result + size
|
|
const auto result = json::to_ubjson(j);
|
|
CHECK(result == expected);
|
|
CHECK(result.size() == 3);
|
|
|
|
// check individual bytes
|
|
CHECK(result[0] == 'I');
|
|
auto const restored = static_cast<uint16_t>((static_cast<uint8_t>(result[1]) * 256) + static_cast<uint8_t>(result[2]));
|
|
CHECK(restored == i);
|
|
|
|
// roundtrip
|
|
CHECK(json::from_ubjson(result) == j);
|
|
CHECK(json::from_ubjson(result, true, false) == j);
|
|
}
|
|
}
|
|
|
|
SECTION("65536..2147483647 (int32)")
|
|
{
|
|
for (uint32_t i :
|
|
{
|
|
65536u, 77777u, 1048576u
|
|
})
|
|
{
|
|
CAPTURE(i)
|
|
|
|
// create JSON value with integer number
|
|
json j = -1;
|
|
j.get_ref<json::number_integer_t&>() = static_cast<json::number_integer_t>(i);
|
|
|
|
// check type
|
|
CHECK(j.is_number_integer());
|
|
|
|
// create expected byte vector
|
|
std::vector<uint8_t> const expected
|
|
{
|
|
'l',
|
|
static_cast<uint8_t>((i >> 24) & 0xff),
|
|
static_cast<uint8_t>((i >> 16) & 0xff),
|
|
static_cast<uint8_t>((i >> 8) & 0xff),
|
|
static_cast<uint8_t>(i & 0xff),
|
|
};
|
|
|
|
// compare result + size
|
|
const auto result = json::to_ubjson(j);
|
|
CHECK(result == expected);
|
|
CHECK(result.size() == 5);
|
|
|
|
// check individual bytes
|
|
CHECK(result[0] == 'l');
|
|
uint32_t const restored = (static_cast<uint32_t>(result[1]) << 030) +
|
|
(static_cast<uint32_t>(result[2]) << 020) +
|
|
(static_cast<uint32_t>(result[3]) << 010) +
|
|
static_cast<uint32_t>(result[4]);
|
|
CHECK(restored == i);
|
|
|
|
// roundtrip
|
|
CHECK(json::from_ubjson(result) == j);
|
|
CHECK(json::from_ubjson(result, true, false) == j);
|
|
}
|
|
}
|
|
|
|
SECTION("2147483648..9223372036854775807 (int64)")
|
|
{
|
|
std::vector<uint64_t> const v = {2147483648ul, 9223372036854775807ul};
|
|
for (uint64_t i : v)
|
|
{
|
|
CAPTURE(i)
|
|
|
|
// create JSON value with integer number
|
|
json j = -1;
|
|
j.get_ref<json::number_integer_t&>() = static_cast<json::number_integer_t>(i);
|
|
|
|
// check type
|
|
CHECK(j.is_number_integer());
|
|
|
|
// create expected byte vector
|
|
std::vector<uint8_t> const expected
|
|
{
|
|
'L',
|
|
static_cast<uint8_t>((i >> 070) & 0xff),
|
|
static_cast<uint8_t>((i >> 060) & 0xff),
|
|
static_cast<uint8_t>((i >> 050) & 0xff),
|
|
static_cast<uint8_t>((i >> 040) & 0xff),
|
|
static_cast<uint8_t>((i >> 030) & 0xff),
|
|
static_cast<uint8_t>((i >> 020) & 0xff),
|
|
static_cast<uint8_t>((i >> 010) & 0xff),
|
|
static_cast<uint8_t>(i & 0xff),
|
|
};
|
|
|
|
// compare result + size
|
|
const auto result = json::to_ubjson(j);
|
|
CHECK(result == expected);
|
|
CHECK(result.size() == 9);
|
|
|
|
// check individual bytes
|
|
CHECK(result[0] == 'L');
|
|
uint64_t const restored = (static_cast<uint64_t>(result[1]) << 070) +
|
|
(static_cast<uint64_t>(result[2]) << 060) +
|
|
(static_cast<uint64_t>(result[3]) << 050) +
|
|
(static_cast<uint64_t>(result[4]) << 040) +
|
|
(static_cast<uint64_t>(result[5]) << 030) +
|
|
(static_cast<uint64_t>(result[6]) << 020) +
|
|
(static_cast<uint64_t>(result[7]) << 010) +
|
|
static_cast<uint64_t>(result[8]);
|
|
CHECK(restored == i);
|
|
|
|
// roundtrip
|
|
CHECK(json::from_ubjson(result) == j);
|
|
CHECK(json::from_ubjson(result, true, false) == j);
|
|
}
|
|
}
|
|
}
|
|
|
|
SECTION("unsigned")
|
|
{
|
|
SECTION("0..127 (int8)")
|
|
{
|
|
for (size_t i = 0; i <= 127; ++i)
|
|
{
|
|
CAPTURE(i)
|
|
|
|
// create JSON value with unsigned integer number
|
|
json const j = i;
|
|
|
|
// check type
|
|
CHECK(j.is_number_unsigned());
|
|
|
|
// create expected byte vector
|
|
std::vector<uint8_t> const expected
|
|
{
|
|
'i',
|
|
static_cast<uint8_t>(i),
|
|
};
|
|
|
|
// compare result + size
|
|
const auto result = json::to_ubjson(j);
|
|
CHECK(result == expected);
|
|
CHECK(result.size() == 2);
|
|
|
|
// check individual bytes
|
|
CHECK(result[0] == 'i');
|
|
auto const restored = static_cast<uint8_t>(result[1]);
|
|
CHECK(restored == i);
|
|
|
|
// roundtrip
|
|
CHECK(json::from_ubjson(result) == j);
|
|
CHECK(json::from_ubjson(result, true, false) == j);
|
|
}
|
|
}
|
|
|
|
SECTION("128..255 (uint8)")
|
|
{
|
|
for (size_t i = 128; i <= 255; ++i)
|
|
{
|
|
CAPTURE(i)
|
|
|
|
// create JSON value with unsigned integer number
|
|
json const j = i;
|
|
|
|
// check type
|
|
CHECK(j.is_number_unsigned());
|
|
|
|
// create expected byte vector
|
|
std::vector<uint8_t> const expected
|
|
{
|
|
'U',
|
|
static_cast<uint8_t>(i),
|
|
};
|
|
|
|
// compare result + size
|
|
const auto result = json::to_ubjson(j);
|
|
CHECK(result == expected);
|
|
CHECK(result.size() == 2);
|
|
|
|
// check individual bytes
|
|
CHECK(result[0] == 'U');
|
|
auto const restored = static_cast<uint8_t>(result[1]);
|
|
CHECK(restored == i);
|
|
|
|
// roundtrip
|
|
CHECK(json::from_ubjson(result) == j);
|
|
CHECK(json::from_ubjson(result, true, false) == j);
|
|
}
|
|
}
|
|
|
|
SECTION("256..32767 (int16)")
|
|
{
|
|
for (size_t i = 256; i <= 32767; i = utils::next_integer_sample(i, static_cast<size_t>(32767), static_cast<size_t>(7)))
|
|
{
|
|
CAPTURE(i)
|
|
|
|
// create JSON value with unsigned integer number
|
|
json const j = i;
|
|
|
|
// check type
|
|
CHECK(j.is_number_unsigned());
|
|
|
|
// create expected byte vector
|
|
std::vector<uint8_t> const expected
|
|
{
|
|
'I',
|
|
static_cast<uint8_t>((i >> 8) & 0xff),
|
|
static_cast<uint8_t>(i & 0xff),
|
|
};
|
|
|
|
// compare result + size
|
|
const auto result = json::to_ubjson(j);
|
|
CHECK(result == expected);
|
|
CHECK(result.size() == 3);
|
|
|
|
// check individual bytes
|
|
CHECK(result[0] == 'I');
|
|
auto const restored = static_cast<uint16_t>((static_cast<uint8_t>(result[1]) * 256) + static_cast<uint8_t>(result[2]));
|
|
CHECK(restored == i);
|
|
|
|
// roundtrip
|
|
CHECK(json::from_ubjson(result) == j);
|
|
CHECK(json::from_ubjson(result, true, false) == j);
|
|
}
|
|
}
|
|
|
|
SECTION("65536..2147483647 (int32)")
|
|
{
|
|
for (uint32_t i :
|
|
{
|
|
65536u, 77777u, 1048576u
|
|
})
|
|
{
|
|
CAPTURE(i)
|
|
|
|
// create JSON value with unsigned integer number
|
|
json const j = i;
|
|
|
|
// check type
|
|
CHECK(j.is_number_unsigned());
|
|
|
|
// create expected byte vector
|
|
std::vector<uint8_t> const expected
|
|
{
|
|
'l',
|
|
static_cast<uint8_t>((i >> 24) & 0xff),
|
|
static_cast<uint8_t>((i >> 16) & 0xff),
|
|
static_cast<uint8_t>((i >> 8) & 0xff),
|
|
static_cast<uint8_t>(i & 0xff),
|
|
};
|
|
|
|
// compare result + size
|
|
const auto result = json::to_ubjson(j);
|
|
CHECK(result == expected);
|
|
CHECK(result.size() == 5);
|
|
|
|
// check individual bytes
|
|
CHECK(result[0] == 'l');
|
|
uint32_t const restored = (static_cast<uint32_t>(result[1]) << 030) +
|
|
(static_cast<uint32_t>(result[2]) << 020) +
|
|
(static_cast<uint32_t>(result[3]) << 010) +
|
|
static_cast<uint32_t>(result[4]);
|
|
CHECK(restored == i);
|
|
|
|
// roundtrip
|
|
CHECK(json::from_ubjson(result) == j);
|
|
CHECK(json::from_ubjson(result, true, false) == j);
|
|
}
|
|
}
|
|
|
|
SECTION("2147483648..9223372036854775807 (int64)")
|
|
{
|
|
std::vector<uint64_t> const v = {2147483648ul, 9223372036854775807ul};
|
|
for (uint64_t i : v)
|
|
{
|
|
CAPTURE(i)
|
|
|
|
// create JSON value with integer number
|
|
json const j = i;
|
|
|
|
// check type
|
|
CHECK(j.is_number_unsigned());
|
|
|
|
// create expected byte vector
|
|
std::vector<uint8_t> const expected
|
|
{
|
|
'L',
|
|
static_cast<uint8_t>((i >> 070) & 0xff),
|
|
static_cast<uint8_t>((i >> 060) & 0xff),
|
|
static_cast<uint8_t>((i >> 050) & 0xff),
|
|
static_cast<uint8_t>((i >> 040) & 0xff),
|
|
static_cast<uint8_t>((i >> 030) & 0xff),
|
|
static_cast<uint8_t>((i >> 020) & 0xff),
|
|
static_cast<uint8_t>((i >> 010) & 0xff),
|
|
static_cast<uint8_t>(i & 0xff),
|
|
};
|
|
|
|
// compare result + size
|
|
const auto result = json::to_ubjson(j);
|
|
CHECK(result == expected);
|
|
CHECK(result.size() == 9);
|
|
|
|
// check individual bytes
|
|
CHECK(result[0] == 'L');
|
|
uint64_t const restored = (static_cast<uint64_t>(result[1]) << 070) +
|
|
(static_cast<uint64_t>(result[2]) << 060) +
|
|
(static_cast<uint64_t>(result[3]) << 050) +
|
|
(static_cast<uint64_t>(result[4]) << 040) +
|
|
(static_cast<uint64_t>(result[5]) << 030) +
|
|
(static_cast<uint64_t>(result[6]) << 020) +
|
|
(static_cast<uint64_t>(result[7]) << 010) +
|
|
static_cast<uint64_t>(result[8]);
|
|
CHECK(restored == i);
|
|
|
|
// roundtrip
|
|
CHECK(json::from_ubjson(result) == j);
|
|
CHECK(json::from_ubjson(result, true, false) == j);
|
|
}
|
|
}
|
|
}
|
|
|
|
SECTION("float64")
|
|
{
|
|
SECTION("3.1415925")
|
|
{
|
|
double v = 3.1415925;
|
|
json const j = v;
|
|
std::vector<uint8_t> expected =
|
|
{
|
|
'D', 0x40, 0x09, 0x21, 0xfb, 0x3f, 0xa6, 0xde, 0xfc
|
|
};
|
|
const auto result = json::to_ubjson(j);
|
|
CHECK(result == expected);
|
|
|
|
// roundtrip
|
|
CHECK(json::from_ubjson(result) == j);
|
|
CHECK(json::from_ubjson(result) == v);
|
|
CHECK(json::from_ubjson(result, true, false) == j);
|
|
}
|
|
}
|
|
|
|
SECTION("high-precision number")
|
|
{
|
|
SECTION("unsigned integer number")
|
|
{
|
|
std::vector<uint8_t> const vec = {'H', 'i', 0x14, '1', '2', '3', '4', '5', '6', '7', '8', '9', '0', '1', '2', '3', '4', '5', '6', '7', '8', '9', '0'};
|
|
const auto j = json::from_ubjson(vec);
|
|
CHECK(j.is_number_unsigned());
|
|
CHECK(j.dump() == "12345678901234567890");
|
|
}
|
|
|
|
SECTION("signed integer number")
|
|
{
|
|
std::vector<uint8_t> const vec = {'H', 'i', 0x13, '-', '1', '2', '3', '4', '5', '6', '7', '8', '9', '0', '1', '2', '3', '4', '5', '6', '7', '8'};
|
|
const auto j = json::from_ubjson(vec);
|
|
CHECK(j.is_number_integer());
|
|
CHECK(j.dump() == "-123456789012345678");
|
|
}
|
|
|
|
SECTION("floating-point number")
|
|
{
|
|
std::vector<uint8_t> const vec = {'H', 'i', 0x16, '3', '.', '1', '4', '1', '5', '9', '2', '6', '5', '3', '5', '8', '9', '7', '9', '3', '2', '3', '8', '4', '6'};
|
|
const auto j = json::from_ubjson(vec);
|
|
CHECK(j.is_number_float());
|
|
CHECK(j.dump() == "3.141592653589793");
|
|
}
|
|
|
|
SECTION("errors")
|
|
{
|
|
// error while parsing length
|
|
std::vector<uint8_t> const vec0 = {'H', 'i'};
|
|
CHECK(json::from_ubjson(vec0, true, false).is_discarded());
|
|
// error while parsing string
|
|
std::vector<uint8_t> const vec1 = {'H', 'i', '1'};
|
|
CHECK(json::from_ubjson(vec1, true, false).is_discarded());
|
|
|
|
json _;
|
|
std::vector<uint8_t> const vec2 = {'H', 'i', 2, '1', 'A', '3'};
|
|
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(vec2), "[json.exception.parse_error.115] parse error at byte 5: syntax error while parsing UBJSON high-precision number: invalid number text: 1A", json::parse_error);
|
|
std::vector<uint8_t> const vec3 = {'H', 'i', 2, '1', '.'};
|
|
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(vec3), "[json.exception.parse_error.115] parse error at byte 5: syntax error while parsing UBJSON high-precision number: invalid number text: 1.", json::parse_error);
|
|
// Reject NULs where they are read, including trailing NULs and payloads cut off after one.
|
|
SECTION("NUL in high-precision number (issue #5753)")
|
|
{
|
|
for (const auto& vec : std::vector<std::vector<uint8_t>>
|
|
{
|
|
{'H', 'i', 3, '1', 0, 'x'},
|
|
{'H', 'i', 2, '1', 0},
|
|
{'H', 'i', 3, '1', 0}
|
|
})
|
|
{
|
|
CAPTURE(vec)
|
|
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(vec), "[json.exception.parse_error.115] parse error at byte 5: syntax error while parsing UBJSON high-precision number: invalid number text; last byte: 0x00", json::parse_error);
|
|
CHECK(json::from_ubjson(vec, true, false).is_discarded());
|
|
CHECK(json::from_ubjson(vec, false, false).is_discarded());
|
|
}
|
|
|
|
std::vector<uint8_t> const nested = {'[', 'H', 'i', 3, '1', 0, 'x', ']'};
|
|
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(nested), "[json.exception.parse_error.115] parse error at byte 6: syntax error while parsing UBJSON high-precision number: invalid number text; last byte: 0x00", json::parse_error);
|
|
CHECK(json::from_ubjson(nested, true, false).is_discarded());
|
|
|
|
std::vector<uint8_t> const valid = {'H', 'i', 1, '1'};
|
|
const auto j = json::from_ubjson(valid);
|
|
CHECK(j.is_number_unsigned());
|
|
CHECK(j == json(1));
|
|
}
|
|
|
|
std::vector<uint8_t> const vec_overflow = {'H', 'i', 5, '1', 'e', '4', '0', '0'};
|
|
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(vec_overflow), "[json.exception.out_of_range.406] number overflow parsing '1e400'", json::out_of_range&);
|
|
std::vector<uint8_t> const vec4 = {'H', 2, '1', '0'};
|
|
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(vec4), "[json.exception.parse_error.113] parse error at byte 2: syntax error while parsing UBJSON size: expected length type specification (U, i, I, l, L) after '#'; last byte: 0x02", json::parse_error);
|
|
}
|
|
|
|
SECTION("serialization")
|
|
{
|
|
// number that does not fit int64
|
|
json const j = 11111111111111111111ULL;
|
|
CHECK(j.is_number_unsigned());
|
|
|
|
// number will be serialized to high-precision number
|
|
const auto vec = json::to_ubjson(j);
|
|
std::vector<uint8_t> expected = {'H', 'i', 0x14, '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1'};
|
|
CHECK(vec == expected);
|
|
|
|
// roundtrip
|
|
CHECK(json::from_ubjson(vec) == j);
|
|
}
|
|
}
|
|
}
|
|
|
|
SECTION("string")
|
|
{
|
|
SECTION("N = 0..127")
|
|
{
|
|
for (size_t N = 0; N <= 127; ++N)
|
|
{
|
|
CAPTURE(N)
|
|
|
|
// create JSON value with string containing of N * 'x'
|
|
const auto s = std::string(N, 'x');
|
|
json const j = s;
|
|
|
|
// create expected byte vector
|
|
std::vector<uint8_t> expected;
|
|
expected.push_back('S');
|
|
expected.push_back('i');
|
|
expected.push_back(static_cast<uint8_t>(N));
|
|
for (size_t i = 0; i < N; ++i)
|
|
{
|
|
expected.push_back('x');
|
|
}
|
|
|
|
// compare result + size
|
|
const auto result = json::to_ubjson(j);
|
|
CHECK(result == expected);
|
|
CHECK(result.size() == N + 3);
|
|
// check that no null byte is appended
|
|
if (N > 0)
|
|
{
|
|
CHECK(result.back() != '\x00');
|
|
}
|
|
|
|
// roundtrip
|
|
CHECK(json::from_ubjson(result) == j);
|
|
CHECK(json::from_ubjson(result, true, false) == j);
|
|
}
|
|
}
|
|
|
|
SECTION("N = 128..255")
|
|
{
|
|
for (size_t N = 128; N <= 255; ++N)
|
|
{
|
|
CAPTURE(N)
|
|
|
|
// create JSON value with string containing of N * 'x'
|
|
const auto s = std::string(N, 'x');
|
|
json const j = s;
|
|
|
|
// create expected byte vector
|
|
std::vector<uint8_t> expected;
|
|
expected.push_back('S');
|
|
expected.push_back('U');
|
|
expected.push_back(static_cast<uint8_t>(N));
|
|
for (size_t i = 0; i < N; ++i)
|
|
{
|
|
expected.push_back('x');
|
|
}
|
|
|
|
// compare result + size
|
|
const auto result = json::to_ubjson(j);
|
|
CHECK(result == expected);
|
|
CHECK(result.size() == N + 3);
|
|
// check that no null byte is appended
|
|
CHECK(result.back() != '\x00');
|
|
|
|
// roundtrip
|
|
CHECK(json::from_ubjson(result) == j);
|
|
CHECK(json::from_ubjson(result, true, false) == j);
|
|
}
|
|
}
|
|
|
|
SECTION("N = 256..32767")
|
|
{
|
|
for (size_t N :
|
|
{
|
|
256u, 999u, 1025u, 3333u, 2048u, 32767u
|
|
})
|
|
{
|
|
CAPTURE(N)
|
|
|
|
// create JSON value with string containing of N * 'x'
|
|
const auto s = std::string(N, 'x');
|
|
json const j = s;
|
|
|
|
// create expected byte vector (hack: create string first)
|
|
std::vector<uint8_t> expected(N, 'x');
|
|
// reverse order of commands, because we insert at begin()
|
|
expected.insert(expected.begin(), static_cast<uint8_t>(N & 0xff));
|
|
expected.insert(expected.begin(), static_cast<uint8_t>((N >> 8) & 0xff));
|
|
expected.insert(expected.begin(), 'I');
|
|
expected.insert(expected.begin(), 'S');
|
|
|
|
// compare result + size
|
|
const auto result = json::to_ubjson(j);
|
|
CHECK(result == expected);
|
|
CHECK(result.size() == N + 4);
|
|
// check that no null byte is appended
|
|
CHECK(result.back() != '\x00');
|
|
|
|
// roundtrip
|
|
CHECK(json::from_ubjson(result) == j);
|
|
CHECK(json::from_ubjson(result, true, false) == j);
|
|
}
|
|
}
|
|
|
|
SECTION("N = 65536..2147483647")
|
|
{
|
|
for (size_t N :
|
|
{
|
|
65536u, 77777u, 1048576u
|
|
})
|
|
{
|
|
CAPTURE(N)
|
|
|
|
// create JSON value with string containing of N * 'x'
|
|
const auto s = std::string(N, 'x');
|
|
json const j = s;
|
|
|
|
// create expected byte vector (hack: create string first)
|
|
std::vector<uint8_t> expected(N, 'x');
|
|
// reverse order of commands, because we insert at begin()
|
|
expected.insert(expected.begin(), static_cast<uint8_t>(N & 0xff));
|
|
expected.insert(expected.begin(), static_cast<uint8_t>((N >> 8) & 0xff));
|
|
expected.insert(expected.begin(), static_cast<uint8_t>((N >> 16) & 0xff));
|
|
expected.insert(expected.begin(), static_cast<uint8_t>((N >> 24) & 0xff));
|
|
expected.insert(expected.begin(), 'l');
|
|
expected.insert(expected.begin(), 'S');
|
|
|
|
// compare result + size
|
|
const auto result = json::to_ubjson(j);
|
|
CHECK(result == expected);
|
|
CHECK(result.size() == N + 6);
|
|
// check that no null byte is appended
|
|
CHECK(result.back() != '\x00');
|
|
|
|
// roundtrip
|
|
CHECK(json::from_ubjson(result) == j);
|
|
CHECK(json::from_ubjson(result, true, false) == j);
|
|
}
|
|
}
|
|
}
|
|
|
|
SECTION("binary")
|
|
{
|
|
SECTION("N = 0..127")
|
|
{
|
|
for (std::size_t N = 0; N <= 127; ++N)
|
|
{
|
|
CAPTURE(N)
|
|
|
|
// create JSON value with byte array containing of N * 'x'
|
|
const auto s = std::vector<std::uint8_t>(N, 'x');
|
|
json const j = json::binary(s);
|
|
|
|
// create expected byte vector
|
|
std::vector<std::uint8_t> expected;
|
|
expected.push_back(static_cast<std::uint8_t>('['));
|
|
if (N != 0)
|
|
{
|
|
expected.push_back(static_cast<std::uint8_t>('$'));
|
|
expected.push_back(static_cast<std::uint8_t>('U'));
|
|
}
|
|
expected.push_back(static_cast<std::uint8_t>('#'));
|
|
expected.push_back(static_cast<std::uint8_t>('i'));
|
|
expected.push_back(static_cast<std::uint8_t>(N));
|
|
for (size_t i = 0; i < N; ++i)
|
|
{
|
|
expected.push_back(0x78);
|
|
}
|
|
|
|
// compare result + size
|
|
const auto result = json::to_ubjson(j, true, true);
|
|
CHECK(result == expected);
|
|
if (N == 0)
|
|
{
|
|
CHECK(result.size() == N + 4);
|
|
}
|
|
else
|
|
{
|
|
CHECK(result.size() == N + 6);
|
|
}
|
|
|
|
// check that no null byte is appended
|
|
if (N > 0)
|
|
{
|
|
CHECK(result.back() != '\x00');
|
|
}
|
|
|
|
// roundtrip only works to an array of numbers
|
|
json j_out = s;
|
|
CHECK(json::from_ubjson(result) == j_out);
|
|
CHECK(json::from_ubjson(result, true, false) == j_out);
|
|
}
|
|
}
|
|
|
|
SECTION("N = 128..255")
|
|
{
|
|
for (std::size_t N = 128; N <= 255; ++N)
|
|
{
|
|
CAPTURE(N)
|
|
|
|
// create JSON value with byte array containing of N * 'x'
|
|
const auto s = std::vector<std::uint8_t>(N, 'x');
|
|
json const j = json::binary(s);
|
|
|
|
// create expected byte vector
|
|
std::vector<uint8_t> expected;
|
|
expected.push_back(static_cast<std::uint8_t>('['));
|
|
expected.push_back(static_cast<std::uint8_t>('$'));
|
|
expected.push_back(static_cast<std::uint8_t>('U'));
|
|
expected.push_back(static_cast<std::uint8_t>('#'));
|
|
expected.push_back(static_cast<std::uint8_t>('U'));
|
|
expected.push_back(static_cast<std::uint8_t>(N));
|
|
for (size_t i = 0; i < N; ++i)
|
|
{
|
|
expected.push_back(0x78);
|
|
}
|
|
|
|
// compare result + size
|
|
const auto result = json::to_ubjson(j, true, true);
|
|
CHECK(result == expected);
|
|
CHECK(result.size() == N + 6);
|
|
// check that no null byte is appended
|
|
CHECK(result.back() != '\x00');
|
|
|
|
// roundtrip only works to an array of numbers
|
|
json j_out = s;
|
|
CHECK(json::from_ubjson(result) == j_out);
|
|
CHECK(json::from_ubjson(result, true, false) == j_out);
|
|
}
|
|
}
|
|
|
|
SECTION("N = 256..32767")
|
|
{
|
|
for (std::size_t N :
|
|
{
|
|
256u, 999u, 1025u, 3333u, 2048u, 32767u
|
|
})
|
|
{
|
|
CAPTURE(N)
|
|
|
|
// create JSON value with byte array containing of N * 'x'
|
|
const auto s = std::vector<std::uint8_t>(N, 'x');
|
|
json const j = json::binary(s);
|
|
|
|
// create expected byte vector
|
|
std::vector<std::uint8_t> expected(N + 7, 'x');
|
|
expected[0] = '[';
|
|
expected[1] = '$';
|
|
expected[2] = 'U';
|
|
expected[3] = '#';
|
|
expected[4] = 'I';
|
|
expected[5] = static_cast<std::uint8_t>((N >> 8) & 0xFF);
|
|
expected[6] = static_cast<std::uint8_t>(N & 0xFF);
|
|
|
|
// compare result + size
|
|
const auto result = json::to_ubjson(j, true, true);
|
|
CHECK(result == expected);
|
|
CHECK(result.size() == N + 7);
|
|
// check that no null byte is appended
|
|
CHECK(result.back() != '\x00');
|
|
|
|
// roundtrip only works to an array of numbers
|
|
json j_out = s;
|
|
CHECK(json::from_ubjson(result) == j_out);
|
|
CHECK(json::from_ubjson(result, true, false) == j_out);
|
|
}
|
|
}
|
|
|
|
SECTION("N = 32768..2147483647")
|
|
{
|
|
for (std::size_t N :
|
|
{
|
|
32768u, 77777u, 1048576u
|
|
})
|
|
{
|
|
CAPTURE(N)
|
|
|
|
// create JSON value with byte array containing of N * 'x'
|
|
const auto s = std::vector<std::uint8_t>(N, 'x');
|
|
json const j = json::binary(s);
|
|
|
|
// create expected byte vector
|
|
std::vector<std::uint8_t> expected(N + 9, 'x');
|
|
expected[0] = '[';
|
|
expected[1] = '$';
|
|
expected[2] = 'U';
|
|
expected[3] = '#';
|
|
expected[4] = 'l';
|
|
expected[5] = static_cast<std::uint8_t>((N >> 24) & 0xFF);
|
|
expected[6] = static_cast<std::uint8_t>((N >> 16) & 0xFF);
|
|
expected[7] = static_cast<std::uint8_t>((N >> 8) & 0xFF);
|
|
expected[8] = static_cast<std::uint8_t>(N & 0xFF);
|
|
|
|
// compare result + size
|
|
const auto result = json::to_ubjson(j, true, true);
|
|
CHECK(result == expected);
|
|
CHECK(result.size() == N + 9);
|
|
// check that no null byte is appended
|
|
CHECK(result.back() != '\x00');
|
|
|
|
// roundtrip only works to an array of numbers
|
|
json j_out = s;
|
|
CHECK(json::from_ubjson(result) == j_out);
|
|
CHECK(json::from_ubjson(result, true, false) == j_out);
|
|
}
|
|
}
|
|
|
|
SECTION("Other Serializations")
|
|
{
|
|
const std::size_t N = 10;
|
|
const auto s = std::vector<std::uint8_t>(N, 'x');
|
|
json const j = json::binary(s);
|
|
|
|
SECTION("No Count No Type")
|
|
{
|
|
std::vector<uint8_t> expected;
|
|
expected.push_back(static_cast<std::uint8_t>('['));
|
|
for (std::size_t i = 0; i < N; ++i)
|
|
{
|
|
expected.push_back(static_cast<std::uint8_t>('U'));
|
|
expected.push_back(static_cast<std::uint8_t>(0x78));
|
|
}
|
|
expected.push_back(static_cast<std::uint8_t>(']'));
|
|
|
|
// compare result + size
|
|
const auto result = json::to_ubjson(j, false, false);
|
|
CHECK(result == expected);
|
|
CHECK(result.size() == N + 12);
|
|
// check that no null byte is appended
|
|
CHECK(result.back() != '\x00');
|
|
|
|
// roundtrip only works to an array of numbers
|
|
json j_out = s;
|
|
CHECK(json::from_ubjson(result) == j_out);
|
|
CHECK(json::from_ubjson(result, true, false) == j_out);
|
|
}
|
|
|
|
SECTION("Yes Count No Type")
|
|
{
|
|
std::vector<std::uint8_t> expected;
|
|
expected.push_back(static_cast<std::uint8_t>('['));
|
|
expected.push_back(static_cast<std::uint8_t>('#'));
|
|
expected.push_back(static_cast<std::uint8_t>('i'));
|
|
expected.push_back(static_cast<std::uint8_t>(N));
|
|
|
|
for (size_t i = 0; i < N; ++i)
|
|
{
|
|
expected.push_back(static_cast<std::uint8_t>('U'));
|
|
expected.push_back(static_cast<std::uint8_t>(0x78));
|
|
}
|
|
|
|
// compare result + size
|
|
const auto result = json::to_ubjson(j, true, false);
|
|
CHECK(result == expected);
|
|
CHECK(result.size() == N + 14);
|
|
// check that no null byte is appended
|
|
CHECK(result.back() != '\x00');
|
|
|
|
// roundtrip only works to an array of numbers
|
|
json j_out = s;
|
|
CHECK(json::from_ubjson(result) == j_out);
|
|
CHECK(json::from_ubjson(result, true, false) == j_out);
|
|
}
|
|
}
|
|
}
|
|
|
|
SECTION("array")
|
|
{
|
|
SECTION("empty")
|
|
{
|
|
SECTION("size=false type=false")
|
|
{
|
|
json const j = json::array();
|
|
std::vector<uint8_t> expected = {'[', ']'};
|
|
const auto result = json::to_ubjson(j);
|
|
CHECK(result == expected);
|
|
|
|
// roundtrip
|
|
CHECK(json::from_ubjson(result) == j);
|
|
CHECK(json::from_ubjson(result, true, false) == j);
|
|
}
|
|
|
|
SECTION("size=true type=false")
|
|
{
|
|
json const j = json::array();
|
|
std::vector<uint8_t> expected = {'[', '#', 'i', 0};
|
|
const auto result = json::to_ubjson(j, true);
|
|
CHECK(result == expected);
|
|
|
|
// roundtrip
|
|
CHECK(json::from_ubjson(result) == j);
|
|
CHECK(json::from_ubjson(result, true, false) == j);
|
|
}
|
|
|
|
SECTION("size=true type=true")
|
|
{
|
|
json const j = json::array();
|
|
std::vector<uint8_t> expected = {'[', '#', 'i', 0};
|
|
const auto result = json::to_ubjson(j, true, true);
|
|
CHECK(result == expected);
|
|
|
|
// roundtrip
|
|
CHECK(json::from_ubjson(result) == j);
|
|
CHECK(json::from_ubjson(result, true, false) == j);
|
|
}
|
|
}
|
|
|
|
SECTION("[null]")
|
|
{
|
|
SECTION("size=false type=false")
|
|
{
|
|
json const j = {nullptr};
|
|
std::vector<uint8_t> expected = {'[', 'Z', ']'};
|
|
const auto result = json::to_ubjson(j);
|
|
CHECK(result == expected);
|
|
|
|
// roundtrip
|
|
CHECK(json::from_ubjson(result) == j);
|
|
CHECK(json::from_ubjson(result, true, false) == j);
|
|
}
|
|
|
|
SECTION("size=true type=false")
|
|
{
|
|
json const j = {nullptr};
|
|
std::vector<uint8_t> expected = {'[', '#', 'i', 1, 'Z'};
|
|
const auto result = json::to_ubjson(j, true);
|
|
CHECK(result == expected);
|
|
|
|
// roundtrip
|
|
CHECK(json::from_ubjson(result) == j);
|
|
CHECK(json::from_ubjson(result, true, false) == j);
|
|
}
|
|
|
|
SECTION("size=true type=true")
|
|
{
|
|
json const j = {nullptr};
|
|
std::vector<uint8_t> expected = {'[', '$', 'Z', '#', 'i', 1};
|
|
const auto result = json::to_ubjson(j, true, true);
|
|
CHECK(result == expected);
|
|
|
|
// roundtrip
|
|
CHECK(json::from_ubjson(result) == j);
|
|
CHECK(json::from_ubjson(result, true, false) == j);
|
|
}
|
|
}
|
|
|
|
SECTION("[1,2,3,4,5]")
|
|
{
|
|
SECTION("size=false type=false")
|
|
{
|
|
json const j = json::parse("[1,2,3,4,5]");
|
|
std::vector<uint8_t> expected = {'[', 'i', 1, 'i', 2, 'i', 3, 'i', 4, 'i', 5, ']'};
|
|
const auto result = json::to_ubjson(j);
|
|
CHECK(result == expected);
|
|
|
|
// roundtrip
|
|
CHECK(json::from_ubjson(result) == j);
|
|
CHECK(json::from_ubjson(result, true, false) == j);
|
|
}
|
|
|
|
SECTION("size=true type=false")
|
|
{
|
|
json const j = json::parse("[1,2,3,4,5]");
|
|
std::vector<uint8_t> expected = {'[', '#', 'i', 5, 'i', 1, 'i', 2, 'i', 3, 'i', 4, 'i', 5};
|
|
const auto result = json::to_ubjson(j, true);
|
|
CHECK(result == expected);
|
|
|
|
// roundtrip
|
|
CHECK(json::from_ubjson(result) == j);
|
|
CHECK(json::from_ubjson(result, true, false) == j);
|
|
}
|
|
|
|
SECTION("size=true type=true")
|
|
{
|
|
json const j = json::parse("[1,2,3,4,5]");
|
|
std::vector<uint8_t> expected = {'[', '$', 'i', '#', 'i', 5, 1, 2, 3, 4, 5};
|
|
const auto result = json::to_ubjson(j, true, true);
|
|
CHECK(result == expected);
|
|
|
|
// roundtrip
|
|
CHECK(json::from_ubjson(result) == j);
|
|
CHECK(json::from_ubjson(result, true, false) == j);
|
|
}
|
|
}
|
|
|
|
SECTION("[[[[]]]]")
|
|
{
|
|
SECTION("size=false type=false")
|
|
{
|
|
json const j = json::parse("[[[[]]]]");
|
|
std::vector<uint8_t> expected = {'[', '[', '[', '[', ']', ']', ']', ']'};
|
|
const auto result = json::to_ubjson(j);
|
|
CHECK(result == expected);
|
|
|
|
// roundtrip
|
|
CHECK(json::from_ubjson(result) == j);
|
|
CHECK(json::from_ubjson(result, true, false) == j);
|
|
}
|
|
|
|
SECTION("size=true type=false")
|
|
{
|
|
json const j = json::parse("[[[[]]]]");
|
|
std::vector<uint8_t> expected = {'[', '#', 'i', 1, '[', '#', 'i', 1, '[', '#', 'i', 1, '[', '#', 'i', 0};
|
|
const auto result = json::to_ubjson(j, true);
|
|
CHECK(result == expected);
|
|
|
|
// roundtrip
|
|
CHECK(json::from_ubjson(result) == j);
|
|
CHECK(json::from_ubjson(result, true, false) == j);
|
|
}
|
|
|
|
SECTION("size=true type=true")
|
|
{
|
|
json const j = json::parse("[[[[]]]]");
|
|
std::vector<uint8_t> expected = {'[', '$', '[', '#', 'i', 1, '$', '[', '#', 'i', 1, '$', '[', '#', 'i', 1, '#', 'i', 0};
|
|
const auto result = json::to_ubjson(j, true, true);
|
|
CHECK(result == expected);
|
|
|
|
// roundtrip
|
|
CHECK(json::from_ubjson(result) == j);
|
|
CHECK(json::from_ubjson(result, true, false) == j);
|
|
}
|
|
}
|
|
|
|
SECTION("array with uint16_t elements")
|
|
{
|
|
SECTION("size=false type=false")
|
|
{
|
|
json j(257, nullptr);
|
|
std::vector<uint8_t> expected(j.size() + 2, 'Z'); // all null
|
|
expected[0] = '['; // opening array
|
|
expected[258] = ']'; // closing array
|
|
const auto result = json::to_ubjson(j);
|
|
CHECK(result == expected);
|
|
|
|
// roundtrip
|
|
CHECK(json::from_ubjson(result) == j);
|
|
CHECK(json::from_ubjson(result, true, false) == j);
|
|
}
|
|
|
|
SECTION("size=true type=false")
|
|
{
|
|
json j(257, nullptr);
|
|
std::vector<uint8_t> expected(j.size() + 5, 'Z'); // all null
|
|
expected[0] = '['; // opening array
|
|
expected[1] = '#'; // array size
|
|
expected[2] = 'I'; // int16
|
|
expected[3] = 0x01; // 0x0101, first byte
|
|
expected[4] = 0x01; // 0x0101, second byte
|
|
const auto result = json::to_ubjson(j, true);
|
|
CHECK(result == expected);
|
|
|
|
// roundtrip
|
|
CHECK(json::from_ubjson(result) == j);
|
|
CHECK(json::from_ubjson(result, true, false) == j);
|
|
}
|
|
|
|
SECTION("size=true type=true")
|
|
{
|
|
json j(257, nullptr);
|
|
std::vector<uint8_t> expected = {'[', '$', 'Z', '#', 'I', 0x01, 0x01};
|
|
const auto result = json::to_ubjson(j, true, true);
|
|
CHECK(result == expected);
|
|
|
|
// roundtrip
|
|
CHECK(json::from_ubjson(result) == j);
|
|
CHECK(json::from_ubjson(result, true, false) == j);
|
|
}
|
|
}
|
|
|
|
SECTION("array with uint32_t elements")
|
|
{
|
|
SECTION("size=false type=false")
|
|
{
|
|
json j(65793, nullptr);
|
|
std::vector<uint8_t> expected(j.size() + 2, 'Z'); // all null
|
|
expected[0] = '['; // opening array
|
|
expected[65794] = ']'; // closing array
|
|
const auto result = json::to_ubjson(j);
|
|
CHECK(result == expected);
|
|
|
|
// roundtrip
|
|
CHECK(json::from_ubjson(result) == j);
|
|
CHECK(json::from_ubjson(result, true, false) == j);
|
|
}
|
|
|
|
SECTION("size=true type=false")
|
|
{
|
|
json j(65793, nullptr);
|
|
std::vector<uint8_t> expected(j.size() + 7, 'Z'); // all null
|
|
expected[0] = '['; // opening array
|
|
expected[1] = '#'; // array size
|
|
expected[2] = 'l'; // int32
|
|
expected[3] = 0x00; // 0x00010101, first byte
|
|
expected[4] = 0x01; // 0x00010101, second byte
|
|
expected[5] = 0x01; // 0x00010101, third byte
|
|
expected[6] = 0x01; // 0x00010101, fourth byte
|
|
const auto result = json::to_ubjson(j, true);
|
|
CHECK(result == expected);
|
|
|
|
// roundtrip
|
|
CHECK(json::from_ubjson(result) == j);
|
|
CHECK(json::from_ubjson(result, true, false) == j);
|
|
}
|
|
|
|
SECTION("size=true type=true")
|
|
{
|
|
json j(65793, nullptr);
|
|
std::vector<uint8_t> expected = {'[', '$', 'Z', '#', 'l', 0x00, 0x01, 0x01, 0x01};
|
|
const auto result = json::to_ubjson(j, true, true);
|
|
CHECK(result == expected);
|
|
|
|
// roundtrip
|
|
CHECK(json::from_ubjson(result) == j);
|
|
CHECK(json::from_ubjson(result, true, false) == j);
|
|
}
|
|
}
|
|
}
|
|
|
|
SECTION("object")
|
|
{
|
|
SECTION("empty")
|
|
{
|
|
SECTION("size=false type=false")
|
|
{
|
|
json const j = json::object();
|
|
std::vector<uint8_t> expected = {'{', '}'};
|
|
const auto result = json::to_ubjson(j);
|
|
CHECK(result == expected);
|
|
|
|
// roundtrip
|
|
CHECK(json::from_ubjson(result) == j);
|
|
CHECK(json::from_ubjson(result, true, false) == j);
|
|
}
|
|
|
|
SECTION("size=true type=false")
|
|
{
|
|
json const j = json::object();
|
|
std::vector<uint8_t> expected = {'{', '#', 'i', 0};
|
|
const auto result = json::to_ubjson(j, true);
|
|
CHECK(result == expected);
|
|
|
|
// roundtrip
|
|
CHECK(json::from_ubjson(result) == j);
|
|
CHECK(json::from_ubjson(result, true, false) == j);
|
|
}
|
|
|
|
SECTION("size=true type=true")
|
|
{
|
|
json const j = json::object();
|
|
std::vector<uint8_t> expected = {'{', '#', 'i', 0};
|
|
const auto result = json::to_ubjson(j, true, true);
|
|
CHECK(result == expected);
|
|
|
|
// roundtrip
|
|
CHECK(json::from_ubjson(result) == j);
|
|
CHECK(json::from_ubjson(result, true, false) == j);
|
|
}
|
|
}
|
|
|
|
SECTION("{\"\":null}")
|
|
{
|
|
SECTION("size=false type=false")
|
|
{
|
|
json const j = {{"", nullptr}};
|
|
std::vector<uint8_t> expected = {'{', 'i', 0, 'Z', '}'};
|
|
const auto result = json::to_ubjson(j);
|
|
CHECK(result == expected);
|
|
|
|
// roundtrip
|
|
CHECK(json::from_ubjson(result) == j);
|
|
CHECK(json::from_ubjson(result, true, false) == j);
|
|
}
|
|
|
|
SECTION("size=true type=false")
|
|
{
|
|
json const j = {{"", nullptr}};
|
|
std::vector<uint8_t> expected = {'{', '#', 'i', 1, 'i', 0, 'Z'};
|
|
const auto result = json::to_ubjson(j, true);
|
|
CHECK(result == expected);
|
|
|
|
// roundtrip
|
|
CHECK(json::from_ubjson(result) == j);
|
|
CHECK(json::from_ubjson(result, true, false) == j);
|
|
}
|
|
|
|
SECTION("size=true type=true")
|
|
{
|
|
json const j = {{"", nullptr}};
|
|
std::vector<uint8_t> expected = {'{', '$', 'Z', '#', 'i', 1, 'i', 0};
|
|
const auto result = json::to_ubjson(j, true, true);
|
|
CHECK(result == expected);
|
|
|
|
// roundtrip
|
|
CHECK(json::from_ubjson(result) == j);
|
|
CHECK(json::from_ubjson(result, true, false) == j);
|
|
}
|
|
}
|
|
|
|
SECTION("{\"a\": {\"b\": {\"c\": {}}}}")
|
|
{
|
|
SECTION("size=false type=false")
|
|
{
|
|
json const j = json::parse(R"({"a": {"b": {"c": {}}}})");
|
|
std::vector<uint8_t> expected =
|
|
{
|
|
'{', 'i', 1, 'a', '{', 'i', 1, 'b', '{', 'i', 1, 'c', '{', '}', '}', '}', '}'
|
|
};
|
|
const auto result = json::to_ubjson(j);
|
|
CHECK(result == expected);
|
|
|
|
// roundtrip
|
|
CHECK(json::from_ubjson(result) == j);
|
|
CHECK(json::from_ubjson(result, true, false) == j);
|
|
}
|
|
|
|
SECTION("size=true type=false")
|
|
{
|
|
json const j = json::parse(R"({"a": {"b": {"c": {}}}})");
|
|
std::vector<uint8_t> expected =
|
|
{
|
|
'{', '#', 'i', 1, 'i', 1, 'a', '{', '#', 'i', 1, 'i', 1, 'b', '{', '#', 'i', 1, 'i', 1, 'c', '{', '#', 'i', 0
|
|
};
|
|
const auto result = json::to_ubjson(j, true);
|
|
CHECK(result == expected);
|
|
|
|
// roundtrip
|
|
CHECK(json::from_ubjson(result) == j);
|
|
CHECK(json::from_ubjson(result, true, false) == j);
|
|
}
|
|
|
|
SECTION("size=true type=true")
|
|
{
|
|
json const j = json::parse(R"({"a": {"b": {"c": {}}}})");
|
|
std::vector<uint8_t> expected =
|
|
{
|
|
'{', '$', '{', '#', 'i', 1, 'i', 1, 'a', '$', '{', '#', 'i', 1, 'i', 1, 'b', '$', '{', '#', 'i', 1, 'i', 1, 'c', '#', 'i', 0
|
|
};
|
|
const auto result = json::to_ubjson(j, true, true);
|
|
CHECK(result == expected);
|
|
|
|
// roundtrip
|
|
CHECK(json::from_ubjson(result) == j);
|
|
CHECK(json::from_ubjson(result, true, false) == j);
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
SECTION("errors")
|
|
{
|
|
SECTION("strict mode")
|
|
{
|
|
std::vector<uint8_t> const vec = {'Z', 'Z'};
|
|
SECTION("non-strict mode")
|
|
{
|
|
const auto result = json::from_ubjson(vec, false);
|
|
CHECK(result == json());
|
|
}
|
|
|
|
SECTION("strict mode")
|
|
{
|
|
json _;
|
|
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(vec), "[json.exception.parse_error.110] parse error at byte 2: syntax error while parsing UBJSON value: expected end of input; last byte: 0x5A", json::parse_error&);
|
|
}
|
|
}
|
|
|
|
SECTION("excessive size")
|
|
{
|
|
SECTION("array")
|
|
{
|
|
std::vector<uint8_t> const v_ubjson = {'[', '$', 'Z', '#', 'L', 0x78, 0x28, 0x00, 0x68, 0x28, 0x69, 0x69, 0x17};
|
|
json _;
|
|
CHECK_THROWS_AS(_ = json::from_ubjson(v_ubjson), json::out_of_range&);
|
|
|
|
json j;
|
|
nlohmann::detail::json_sax_dom_callback_parser<json, decltype(nlohmann::detail::input_adapter(v_ubjson))> scp(j, [](int /*unused*/, json::parse_event_t /*unused*/, const json& /*unused*/) noexcept
|
|
{
|
|
return true;
|
|
});
|
|
CHECK_THROWS_AS(_ = json::sax_parse(v_ubjson, &scp, json::input_format_t::ubjson), json::out_of_range&);
|
|
}
|
|
|
|
SECTION("object")
|
|
{
|
|
std::vector<uint8_t> const v_ubjson = {'{', '$', 'Z', '#', 'L', 0x78, 0x28, 0x00, 0x68, 0x28, 0x69, 0x69, 0x17};
|
|
json _;
|
|
CHECK_THROWS_AS(_ = json::from_ubjson(v_ubjson), json::out_of_range&);
|
|
|
|
json j;
|
|
nlohmann::detail::json_sax_dom_callback_parser<json, decltype(nlohmann::detail::input_adapter(v_ubjson))> scp(j, [](int /*unused*/, json::parse_event_t /*unused*/, const json& /*unused*/) noexcept
|
|
{
|
|
return true;
|
|
});
|
|
CHECK_THROWS_AS(_ = json::sax_parse(v_ubjson, &scp, json::input_format_t::ubjson), json::out_of_range&);
|
|
}
|
|
|
|
SECTION("array with a known size, read with a callback")
|
|
{
|
|
// a sized array announces its length to start_array()
|
|
std::vector<uint8_t> const v_ubjson = {'[', '#', 'i', 2, 'i', 1, 'i', 2};
|
|
json j;
|
|
nlohmann::detail::json_sax_dom_callback_parser<json, decltype(nlohmann::detail::input_adapter(v_ubjson))> scp(j, [](int /*unused*/, json::parse_event_t /*unused*/, const json& /*unused*/) noexcept
|
|
{
|
|
return true;
|
|
});
|
|
CHECK(json::sax_parse(v_ubjson, &scp, json::input_format_t::ubjson));
|
|
CHECK(j == json({1, 2}));
|
|
|
|
// the readers reject a size this large before they announce
|
|
// it, so it can only reach start_array() directly (the largest
|
|
// value stands for an unknown size and is never checked)
|
|
json k;
|
|
nlohmann::detail::json_sax_dom_callback_parser<json, decltype(nlohmann::detail::input_adapter(v_ubjson))> scp2(k, [](int /*unused*/, json::parse_event_t /*unused*/, const json& /*unused*/) noexcept
|
|
{
|
|
return true;
|
|
});
|
|
CHECK_THROWS_AS(scp2.start_array((std::numeric_limits<std::size_t>::max)() - 1), json::out_of_range&);
|
|
|
|
// without exceptions, the error is reported to the caller
|
|
json l;
|
|
nlohmann::detail::json_sax_dom_callback_parser<json, decltype(nlohmann::detail::input_adapter(v_ubjson))> scp3(l, [](int /*unused*/, json::parse_event_t /*unused*/, const json& /*unused*/) noexcept
|
|
{
|
|
return true;
|
|
}, false);
|
|
CHECK_FALSE(scp3.start_array((std::numeric_limits<std::size_t>::max)() - 1));
|
|
CHECK(scp3.is_errored());
|
|
}
|
|
|
|
SECTION("array with a known size, read without exceptions")
|
|
{
|
|
std::vector<uint8_t> const v_ubjson = {'[', '#', 'i', 2, 'i', 1, 'i', 2};
|
|
json j;
|
|
nlohmann::detail::json_sax_dom_parser<json, decltype(nlohmann::detail::input_adapter(v_ubjson))> sax(j, false);
|
|
CHECK(json::sax_parse(v_ubjson, &sax, json::input_format_t::ubjson));
|
|
CHECK(j == json({1, 2}));
|
|
|
|
// the readers reject a size this large before they announce
|
|
// it, so it can only reach start_array() directly
|
|
json k;
|
|
nlohmann::detail::json_sax_dom_parser<json, decltype(nlohmann::detail::input_adapter(v_ubjson))> sax2(k, false);
|
|
CHECK_FALSE(sax2.start_array((std::numeric_limits<std::size_t>::max)() - 1));
|
|
CHECK(sax2.is_errored());
|
|
}
|
|
}
|
|
}
|
|
|
|
SECTION("SAX aborts")
|
|
{
|
|
SECTION("start_array()")
|
|
{
|
|
std::vector<uint8_t> const v = {'[', 'T', 'F', ']'};
|
|
SaxCountdown scp(0);
|
|
CHECK(!json::sax_parse(v, &scp, json::input_format_t::ubjson));
|
|
}
|
|
|
|
SECTION("start_object()")
|
|
{
|
|
std::vector<uint8_t> const v = {'{', 'i', 3, 'f', 'o', 'o', 'F', '}'};
|
|
SaxCountdown scp(0);
|
|
CHECK(!json::sax_parse(v, &scp, json::input_format_t::ubjson));
|
|
}
|
|
|
|
SECTION("key() in object")
|
|
{
|
|
std::vector<uint8_t> const v = {'{', 'i', 3, 'f', 'o', 'o', 'F', '}'};
|
|
SaxCountdown scp(1);
|
|
CHECK(!json::sax_parse(v, &scp, json::input_format_t::ubjson));
|
|
}
|
|
|
|
SECTION("start_array(len)")
|
|
{
|
|
std::vector<uint8_t> const v = {'[', '#', 'i', '2', 'T', 'F'};
|
|
SaxCountdown scp(0);
|
|
CHECK(!json::sax_parse(v, &scp, json::input_format_t::ubjson));
|
|
}
|
|
|
|
SECTION("start_object(len)")
|
|
{
|
|
std::vector<uint8_t> const v = {'{', '#', 'i', '1', 3, 'f', 'o', 'o', 'F'};
|
|
SaxCountdown scp(0);
|
|
CHECK(!json::sax_parse(v, &scp, json::input_format_t::ubjson));
|
|
}
|
|
|
|
SECTION("key() in object with length")
|
|
{
|
|
std::vector<uint8_t> const v = {'{', 'i', 3, 'f', 'o', 'o', 'F', '}'};
|
|
SaxCountdown scp(1);
|
|
CHECK(!json::sax_parse(v, &scp, json::input_format_t::ubjson));
|
|
}
|
|
}
|
|
|
|
SECTION("parsing values")
|
|
{
|
|
SECTION("strings")
|
|
{
|
|
// create a single-character string for all number types
|
|
std::vector<uint8_t> s_i = {'S', 'i', 1, 'a'};
|
|
std::vector<uint8_t> const s_U = {'S', 'U', 1, 'a'};
|
|
std::vector<uint8_t> const s_I = {'S', 'I', 0, 1, 'a'};
|
|
std::vector<uint8_t> const s_l = {'S', 'l', 0, 0, 0, 1, 'a'};
|
|
std::vector<uint8_t> const s_L = {'S', 'L', 0, 0, 0, 0, 0, 0, 0, 1, 'a'};
|
|
|
|
// check if string is parsed correctly to "a"
|
|
CHECK(json::from_ubjson(s_i) == "a");
|
|
CHECK(json::from_ubjson(s_U) == "a");
|
|
CHECK(json::from_ubjson(s_I) == "a");
|
|
CHECK(json::from_ubjson(s_l) == "a");
|
|
CHECK(json::from_ubjson(s_L) == "a");
|
|
|
|
// roundtrip: output should be optimized
|
|
CHECK(json::to_ubjson(json::from_ubjson(s_i)) == s_i);
|
|
CHECK(json::to_ubjson(json::from_ubjson(s_U)) == s_i);
|
|
CHECK(json::to_ubjson(json::from_ubjson(s_I)) == s_i);
|
|
CHECK(json::to_ubjson(json::from_ubjson(s_l)) == s_i);
|
|
CHECK(json::to_ubjson(json::from_ubjson(s_L)) == s_i);
|
|
}
|
|
|
|
SECTION("no-op markers")
|
|
{
|
|
// A no-op ('N') is valid wherever a value may start; it is consumed
|
|
// by get_ignore_noop() before the value is read. It is not valid
|
|
// where a string length type specification is expected.
|
|
|
|
SECTION("accepted where a value may start")
|
|
{
|
|
// at top level, also repeated
|
|
CHECK(json::from_ubjson(std::vector<uint8_t>({'N', 'i', 1})) == json(1));
|
|
CHECK(json::from_ubjson(std::vector<uint8_t>({'N', 'N', 'N', 'i', 1})) == json(1));
|
|
|
|
// inside an array of unknown size, before and after an element
|
|
CHECK(json::from_ubjson(std::vector<uint8_t>({'[', 'N', 'i', 1, ']'})) == json({1}));
|
|
CHECK(json::from_ubjson(std::vector<uint8_t>({'[', 'i', 1, 'N', ']'})) == json({1}));
|
|
|
|
// inside an object of unknown size: before a key, between key
|
|
// and value, and before the closing '}'
|
|
CHECK(json::from_ubjson(std::vector<uint8_t>({'{', 'N', 'U', 1, 'a', 'i', 1, '}'})) == json({{"a", 1}}));
|
|
CHECK(json::from_ubjson(std::vector<uint8_t>({'{', 'U', 1, 'a', 'N', 'i', 1, '}'})) == json({{"a", 1}}));
|
|
CHECK(json::from_ubjson(std::vector<uint8_t>({'{', 'U', 1, 'a', 'i', 1, 'N', '}'})) == json({{"a", 1}}));
|
|
}
|
|
|
|
SECTION("rejected where a length type specification is expected")
|
|
{
|
|
json _;
|
|
|
|
// after the 'S' marker of a string value
|
|
std::vector<uint8_t> const v_S = {'S', 'N', 'U', 1, 'a'};
|
|
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(v_S), "[json.exception.parse_error.113] parse error at byte 2: syntax error while parsing UBJSON string: expected length type specification (U, i, I, l, L); last byte: 0x4E", json::parse_error&);
|
|
|
|
// as the key length of an object with a known size, where
|
|
// no-ops are not permitted in the first place
|
|
std::vector<uint8_t> const v_key = {'{', '#', 'i', 1, 'N', 'U', 1, 'a', 'i', 1};
|
|
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(v_key), "[json.exception.parse_error.113] parse error at byte 5: syntax error while parsing UBJSON string: expected length type specification (U, i, I, l, L); last byte: 0x4E", json::parse_error&);
|
|
}
|
|
}
|
|
|
|
SECTION("number")
|
|
{
|
|
SECTION("float")
|
|
{
|
|
// float32
|
|
std::vector<uint8_t> const v_d = {'d', 0x40, 0x49, 0x0f, 0xd0};
|
|
CHECK(json::from_ubjson(v_d) == 3.14159f);
|
|
|
|
// float64
|
|
std::vector<uint8_t> const v_D = {'D', 0x40, 0x09, 0x21, 0xf9, 0xf0, 0x1b, 0x86, 0x6e};
|
|
CHECK(json::from_ubjson(v_D) == 3.14159);
|
|
|
|
// float32 is serialized as float64 as the library does not support float32
|
|
CHECK(json::to_ubjson(json::from_ubjson(v_d)) == json::to_ubjson(3.14159f));
|
|
}
|
|
}
|
|
|
|
SECTION("array")
|
|
{
|
|
SECTION("optimized version (length only)")
|
|
{
|
|
// create vector with two elements of the same type
|
|
std::vector<uint8_t> const v_TU = {'[', '#', 'U', 2, 'T', 'T'};
|
|
std::vector<uint8_t> const v_T = {'[', '#', 'i', 2, 'T', 'T'};
|
|
std::vector<uint8_t> const v_F = {'[', '#', 'i', 2, 'F', 'F'};
|
|
std::vector<uint8_t> const v_Z = {'[', '#', 'i', 2, 'Z', 'Z'};
|
|
std::vector<uint8_t> const v_i = {'[', '#', 'i', 2, 'i', 0x7F, 'i', 0x7F};
|
|
std::vector<uint8_t> const v_U = {'[', '#', 'i', 2, 'U', 0xFF, 'U', 0xFF};
|
|
std::vector<uint8_t> const v_I = {'[', '#', 'i', 2, 'I', 0x7F, 0xFF, 'I', 0x7F, 0xFF};
|
|
std::vector<uint8_t> const v_l = {'[', '#', 'i', 2, 'l', 0x7F, 0xFF, 0xFF, 0xFF, 'l', 0x7F, 0xFF, 0xFF, 0xFF};
|
|
std::vector<uint8_t> const v_L = {'[', '#', 'i', 2, 'L', 0x7F, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 'L', 0x7F, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF};
|
|
std::vector<uint8_t> const v_D = {'[', '#', 'i', 2, 'D', 0x40, 0x09, 0x21, 0xfb, 0x4d, 0x12, 0xd8, 0x4a, 'D', 0x40, 0x09, 0x21, 0xfb, 0x4d, 0x12, 0xd8, 0x4a};
|
|
std::vector<uint8_t> const v_S = {'[', '#', 'i', 2, 'S', 'i', 1, 'a', 'S', 'i', 1, 'a'};
|
|
std::vector<uint8_t> const v_C = {'[', '#', 'i', 2, 'C', 'a', 'C', 'a'};
|
|
|
|
// check if vector is parsed correctly
|
|
CHECK(json::from_ubjson(v_TU) == json({true, true}));
|
|
CHECK(json::from_ubjson(v_T) == json({true, true}));
|
|
CHECK(json::from_ubjson(v_F) == json({false, false}));
|
|
CHECK(json::from_ubjson(v_Z) == json({nullptr, nullptr}));
|
|
CHECK(json::from_ubjson(v_i) == json({127, 127}));
|
|
CHECK(json::from_ubjson(v_U) == json({255, 255}));
|
|
CHECK(json::from_ubjson(v_I) == json({32767, 32767}));
|
|
CHECK(json::from_ubjson(v_l) == json({2147483647, 2147483647}));
|
|
CHECK(json::from_ubjson(v_L) == json({9223372036854775807, 9223372036854775807}));
|
|
CHECK(json::from_ubjson(v_D) == json({3.1415926, 3.1415926}));
|
|
CHECK(json::from_ubjson(v_S) == json({"a", "a"}));
|
|
CHECK(json::from_ubjson(v_C) == json({"a", "a"}));
|
|
|
|
// roundtrip: output should be optimized
|
|
CHECK(json::to_ubjson(json::from_ubjson(v_T), true) == v_T);
|
|
CHECK(json::to_ubjson(json::from_ubjson(v_F), true) == v_F);
|
|
CHECK(json::to_ubjson(json::from_ubjson(v_Z), true) == v_Z);
|
|
CHECK(json::to_ubjson(json::from_ubjson(v_i), true) == v_i);
|
|
CHECK(json::to_ubjson(json::from_ubjson(v_U), true) == v_U);
|
|
CHECK(json::to_ubjson(json::from_ubjson(v_I), true) == v_I);
|
|
CHECK(json::to_ubjson(json::from_ubjson(v_l), true) == v_l);
|
|
CHECK(json::to_ubjson(json::from_ubjson(v_L), true) == v_L);
|
|
CHECK(json::to_ubjson(json::from_ubjson(v_D), true) == v_D);
|
|
CHECK(json::to_ubjson(json::from_ubjson(v_S), true) == v_S);
|
|
CHECK(json::to_ubjson(json::from_ubjson(v_C), true) == v_S); // char is serialized to string
|
|
}
|
|
|
|
SECTION("optimized version (type and length)")
|
|
{
|
|
// create vector with two elements of the same type
|
|
std::vector<uint8_t> const v_N = {'[', '$', 'N', '#', 'i', 2};
|
|
std::vector<uint8_t> const v_T = {'[', '$', 'T', '#', 'i', 2};
|
|
std::vector<uint8_t> const v_F = {'[', '$', 'F', '#', 'i', 2};
|
|
std::vector<uint8_t> const v_Z = {'[', '$', 'Z', '#', 'i', 2};
|
|
std::vector<uint8_t> const v_i = {'[', '$', 'i', '#', 'i', 2, 0x7F, 0x7F};
|
|
std::vector<uint8_t> const v_U = {'[', '$', 'U', '#', 'i', 2, 0xFF, 0xFF};
|
|
std::vector<uint8_t> const v_I = {'[', '$', 'I', '#', 'i', 2, 0x7F, 0xFF, 0x7F, 0xFF};
|
|
std::vector<uint8_t> const v_l = {'[', '$', 'l', '#', 'i', 2, 0x7F, 0xFF, 0xFF, 0xFF, 0x7F, 0xFF, 0xFF, 0xFF};
|
|
std::vector<uint8_t> const v_L = {'[', '$', 'L', '#', 'i', 2, 0x7F, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0x7F, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF};
|
|
std::vector<uint8_t> const v_D = {'[', '$', 'D', '#', 'i', 2, 0x40, 0x09, 0x21, 0xfb, 0x4d, 0x12, 0xd8, 0x4a, 0x40, 0x09, 0x21, 0xfb, 0x4d, 0x12, 0xd8, 0x4a};
|
|
std::vector<uint8_t> const v_S = {'[', '$', 'S', '#', 'i', 2, 'i', 1, 'a', 'i', 1, 'a'};
|
|
std::vector<uint8_t> const v_C = {'[', '$', 'C', '#', 'i', 2, 'a', 'a'};
|
|
|
|
// check if vector is parsed correctly
|
|
CHECK(json::from_ubjson(v_N) == json::array());
|
|
CHECK(json::from_ubjson(v_T) == json({true, true}));
|
|
CHECK(json::from_ubjson(v_F) == json({false, false}));
|
|
CHECK(json::from_ubjson(v_Z) == json({nullptr, nullptr}));
|
|
CHECK(json::from_ubjson(v_i) == json({127, 127}));
|
|
CHECK(json::from_ubjson(v_U) == json({255, 255}));
|
|
CHECK(json::from_ubjson(v_I) == json({32767, 32767}));
|
|
CHECK(json::from_ubjson(v_l) == json({2147483647, 2147483647}));
|
|
CHECK(json::from_ubjson(v_L) == json({9223372036854775807, 9223372036854775807}));
|
|
CHECK(json::from_ubjson(v_D) == json({3.1415926, 3.1415926}));
|
|
CHECK(json::from_ubjson(v_S) == json({"a", "a"}));
|
|
CHECK(json::from_ubjson(v_C) == json({"a", "a"}));
|
|
|
|
// roundtrip: output should be optimized
|
|
std::vector<uint8_t> const v_empty = {'[', '#', 'i', 0};
|
|
CHECK(json::to_ubjson(json::from_ubjson(v_N), true, true) == v_empty);
|
|
CHECK(json::to_ubjson(json::from_ubjson(v_T), true, true) == v_T);
|
|
CHECK(json::to_ubjson(json::from_ubjson(v_F), true, true) == v_F);
|
|
CHECK(json::to_ubjson(json::from_ubjson(v_Z), true, true) == v_Z);
|
|
CHECK(json::to_ubjson(json::from_ubjson(v_i), true, true) == v_i);
|
|
CHECK(json::to_ubjson(json::from_ubjson(v_U), true, true) == v_U);
|
|
CHECK(json::to_ubjson(json::from_ubjson(v_I), true, true) == v_I);
|
|
CHECK(json::to_ubjson(json::from_ubjson(v_l), true, true) == v_l);
|
|
CHECK(json::to_ubjson(json::from_ubjson(v_L), true, true) == v_L);
|
|
CHECK(json::to_ubjson(json::from_ubjson(v_D), true, true) == v_D);
|
|
CHECK(json::to_ubjson(json::from_ubjson(v_S), true, true) == v_S);
|
|
CHECK(json::to_ubjson(json::from_ubjson(v_C), true, true) == v_S); // char is serialized to string
|
|
}
|
|
}
|
|
}
|
|
|
|
SECTION("parse errors")
|
|
{
|
|
SECTION("empty byte vector")
|
|
{
|
|
json _;
|
|
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(std::vector<uint8_t>()), "[json.exception.parse_error.110] parse error at byte 1: syntax error while parsing UBJSON value: unexpected end of input", json::parse_error&);
|
|
}
|
|
|
|
SECTION("char")
|
|
{
|
|
SECTION("eof after C byte")
|
|
{
|
|
std::vector<uint8_t> const v = {'C'};
|
|
json _;
|
|
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(v), "[json.exception.parse_error.110] parse error at byte 2: syntax error while parsing UBJSON char: unexpected end of input", json::parse_error&);
|
|
}
|
|
|
|
SECTION("byte out of range")
|
|
{
|
|
std::vector<uint8_t> const v = {'C', 130};
|
|
json _;
|
|
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(v), "[json.exception.parse_error.113] parse error at byte 2: syntax error while parsing UBJSON char: byte after 'C' must be in range 0x00..0x7F; last byte: 0x82", json::parse_error&);
|
|
}
|
|
}
|
|
|
|
SECTION("strings")
|
|
{
|
|
SECTION("eof after S byte")
|
|
{
|
|
std::vector<uint8_t> const v = {'S'};
|
|
json _;
|
|
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(v), "[json.exception.parse_error.110] parse error at byte 2: syntax error while parsing UBJSON value: unexpected end of input", json::parse_error&);
|
|
}
|
|
|
|
SECTION("invalid byte")
|
|
{
|
|
std::vector<uint8_t> const v = {'S', '1', 'a'};
|
|
json _;
|
|
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(v), "[json.exception.parse_error.113] parse error at byte 2: syntax error while parsing UBJSON string: expected length type specification (U, i, I, l, L); last byte: 0x31", json::parse_error&);
|
|
}
|
|
|
|
SECTION("negative length")
|
|
{
|
|
json _;
|
|
|
|
std::vector<uint8_t> const vi = {'S', 'i', 0xFF};
|
|
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(vi), "[json.exception.parse_error.113] parse error at byte 3: syntax error while parsing UBJSON string: string length must not be negative", json::parse_error&);
|
|
CHECK(json::from_ubjson(vi, true, false).is_discarded());
|
|
|
|
std::vector<uint8_t> const vI = {'S', 'I', 0xFF, 0xFF};
|
|
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(vI), "[json.exception.parse_error.113] parse error at byte 4: syntax error while parsing UBJSON string: string length must not be negative", json::parse_error&);
|
|
CHECK(json::from_ubjson(vI, true, false).is_discarded());
|
|
|
|
std::vector<uint8_t> const vl = {'S', 'l', 0xFF, 0xFF, 0xFF, 0xFF};
|
|
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(vl), "[json.exception.parse_error.113] parse error at byte 6: syntax error while parsing UBJSON string: string length must not be negative", json::parse_error&);
|
|
CHECK(json::from_ubjson(vl, true, false).is_discarded());
|
|
|
|
std::vector<uint8_t> const vL = {'S', 'L', 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF};
|
|
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(vL), "[json.exception.parse_error.113] parse error at byte 10: syntax error while parsing UBJSON string: string length must not be negative", json::parse_error&);
|
|
CHECK(json::from_ubjson(vL, true, false).is_discarded());
|
|
|
|
// a length of zero remains valid and yields an empty string
|
|
std::vector<uint8_t> const v0 = {'S', 'i', 0};
|
|
CHECK(json::from_ubjson(v0) == json(""));
|
|
}
|
|
}
|
|
|
|
SECTION("array")
|
|
{
|
|
SECTION("optimized array: no size following type")
|
|
{
|
|
std::vector<uint8_t> const v = {'[', '$', 'i', 2};
|
|
json _;
|
|
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(v), "[json.exception.parse_error.112] parse error at byte 4: syntax error while parsing UBJSON size: expected '#' after type information; last byte: 0x02", json::parse_error&);
|
|
}
|
|
}
|
|
|
|
SECTION("strings")
|
|
{
|
|
std::vector<uint8_t> const vS = {'S'};
|
|
json _;
|
|
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(vS), "[json.exception.parse_error.110] parse error at byte 2: syntax error while parsing UBJSON value: unexpected end of input", json::parse_error&);
|
|
CHECK(json::from_ubjson(vS, true, false).is_discarded());
|
|
|
|
std::vector<uint8_t> const v = {'S', 'i', '2', 'a'};
|
|
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(v), "[json.exception.parse_error.110] parse error at byte 5: syntax error while parsing UBJSON string: unexpected end of input", json::parse_error&);
|
|
CHECK(json::from_ubjson(v, true, false).is_discarded());
|
|
|
|
std::vector<uint8_t> const vC = {'C'};
|
|
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(vC), "[json.exception.parse_error.110] parse error at byte 2: syntax error while parsing UBJSON char: unexpected end of input", json::parse_error&);
|
|
CHECK(json::from_ubjson(vC, true, false).is_discarded());
|
|
}
|
|
|
|
SECTION("sizes")
|
|
{
|
|
std::vector<uint8_t> const vU = {'[', '#', 'U'};
|
|
json _;
|
|
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(vU), "[json.exception.parse_error.110] parse error at byte 4: syntax error while parsing UBJSON number: unexpected end of input", json::parse_error&);
|
|
CHECK(json::from_ubjson(vU, true, false).is_discarded());
|
|
|
|
std::vector<uint8_t> const vi = {'[', '#', 'i'};
|
|
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(vi), "[json.exception.parse_error.110] parse error at byte 4: syntax error while parsing UBJSON number: unexpected end of input", json::parse_error&);
|
|
CHECK(json::from_ubjson(vi, true, false).is_discarded());
|
|
|
|
std::vector<uint8_t> const vI = {'[', '#', 'I'};
|
|
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(vI), "[json.exception.parse_error.110] parse error at byte 4: syntax error while parsing UBJSON number: unexpected end of input", json::parse_error&);
|
|
CHECK(json::from_ubjson(vI, true, false).is_discarded());
|
|
|
|
std::vector<uint8_t> const vl = {'[', '#', 'l'};
|
|
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(vl), "[json.exception.parse_error.110] parse error at byte 4: syntax error while parsing UBJSON number: unexpected end of input", json::parse_error&);
|
|
CHECK(json::from_ubjson(vl, true, false).is_discarded());
|
|
|
|
std::vector<uint8_t> const vL = {'[', '#', 'L'};
|
|
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(vL), "[json.exception.parse_error.110] parse error at byte 4: syntax error while parsing UBJSON number: unexpected end of input", json::parse_error&);
|
|
CHECK(json::from_ubjson(vL, true, false).is_discarded());
|
|
|
|
std::vector<uint8_t> const v0 = {'[', '#', 'T', ']'};
|
|
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(v0), "[json.exception.parse_error.113] parse error at byte 3: syntax error while parsing UBJSON size: expected length type specification (U, i, I, l, L) after '#'; last byte: 0x54", json::parse_error&);
|
|
CHECK(json::from_ubjson(v0, true, false).is_discarded());
|
|
}
|
|
|
|
SECTION("types")
|
|
{
|
|
std::vector<uint8_t> const v0 = {'[', '$'};
|
|
json _;
|
|
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(v0), "[json.exception.parse_error.110] parse error at byte 3: syntax error while parsing UBJSON type: unexpected end of input", json::parse_error&);
|
|
CHECK(json::from_ubjson(v0, true, false).is_discarded());
|
|
|
|
std::vector<uint8_t> const vi = {'[', '$', '#'};
|
|
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(vi), "[json.exception.parse_error.112] parse error at byte 3: syntax error while parsing UBJSON type: marker 0x23 is not a permitted optimized array type", json::parse_error&);
|
|
CHECK(json::from_ubjson(vi, true, false).is_discarded());
|
|
|
|
std::vector<uint8_t> const vT = {'[', '$', 'T'};
|
|
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(vT), "[json.exception.parse_error.110] parse error at byte 4: syntax error while parsing UBJSON value: unexpected end of input", json::parse_error&);
|
|
CHECK(json::from_ubjson(vT, true, false).is_discarded());
|
|
}
|
|
|
|
SECTION("arrays")
|
|
{
|
|
std::vector<uint8_t> const vST = {'[', '$', 'i', '#', 'i', 2, 1};
|
|
json _;
|
|
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(vST), "[json.exception.parse_error.110] parse error at byte 8: syntax error while parsing UBJSON number: unexpected end of input", json::parse_error&);
|
|
CHECK(json::from_ubjson(vST, true, false).is_discarded());
|
|
|
|
std::vector<uint8_t> const vS = {'[', '#', 'i', 2, 'i', 1};
|
|
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(vS), "[json.exception.parse_error.110] parse error at byte 7: syntax error while parsing UBJSON value: unexpected end of input", json::parse_error&);
|
|
CHECK(json::from_ubjson(vS, true, false).is_discarded());
|
|
|
|
std::vector<uint8_t> const v = {'[', 'i', 2, 'i', 1};
|
|
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(v), "[json.exception.parse_error.110] parse error at byte 6: syntax error while parsing UBJSON value: unexpected end of input", json::parse_error&);
|
|
CHECK(json::from_ubjson(v, true, false).is_discarded());
|
|
}
|
|
|
|
SECTION("objects")
|
|
{
|
|
std::vector<uint8_t> const vST = {'{', '$', 'i', '#', 'i', 2, 'i', 1, 'a', 1};
|
|
json _;
|
|
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(vST), "[json.exception.parse_error.110] parse error at byte 11: syntax error while parsing UBJSON value: unexpected end of input", json::parse_error&);
|
|
CHECK(json::from_ubjson(vST, true, false).is_discarded());
|
|
|
|
std::vector<uint8_t> const vT = {'{', '$', 'i', 'i', 1, 'a', 1};
|
|
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(vT), "[json.exception.parse_error.112] parse error at byte 4: syntax error while parsing UBJSON size: expected '#' after type information; last byte: 0x69", json::parse_error&);
|
|
CHECK(json::from_ubjson(vT, true, false).is_discarded());
|
|
|
|
std::vector<uint8_t> const vS = {'{', '#', 'i', 2, 'i', 1, 'a', 'i', 1};
|
|
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(vS), "[json.exception.parse_error.110] parse error at byte 10: syntax error while parsing UBJSON value: unexpected end of input", json::parse_error&);
|
|
CHECK(json::from_ubjson(vS, true, false).is_discarded());
|
|
|
|
std::vector<uint8_t> const v = {'{', 'i', 1, 'a', 'i', 1};
|
|
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(v), "[json.exception.parse_error.110] parse error at byte 7: syntax error while parsing UBJSON value: unexpected end of input", json::parse_error&);
|
|
CHECK(json::from_ubjson(v, true, false).is_discarded());
|
|
|
|
std::vector<uint8_t> const v2 = {'{', 'i', 1, 'a', 'i', 1, 'i'};
|
|
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(v2), "[json.exception.parse_error.110] parse error at byte 8: syntax error while parsing UBJSON number: unexpected end of input", json::parse_error&);
|
|
CHECK(json::from_ubjson(v2, true, false).is_discarded());
|
|
|
|
std::vector<uint8_t> const v3 = {'{', 'i', 1, 'a'};
|
|
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(v3), "[json.exception.parse_error.110] parse error at byte 5: syntax error while parsing UBJSON value: unexpected end of input", json::parse_error&);
|
|
CHECK(json::from_ubjson(v3, true, false).is_discarded());
|
|
|
|
std::vector<uint8_t> const vST1 = {'{', '$', 'd', '#', 'i', 2, 'i', 1, 'a'};
|
|
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(vST1), "[json.exception.parse_error.110] parse error at byte 10: syntax error while parsing UBJSON number: unexpected end of input", json::parse_error&);
|
|
CHECK(json::from_ubjson(vST1, true, false).is_discarded());
|
|
|
|
std::vector<uint8_t> const vST2 = {'{', '#', 'i', 2, 'i', 1, 'a'};
|
|
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(vST2), "[json.exception.parse_error.110] parse error at byte 8: syntax error while parsing UBJSON value: unexpected end of input", json::parse_error&);
|
|
CHECK(json::from_ubjson(vST2, true, false).is_discarded());
|
|
}
|
|
}
|
|
|
|
SECTION("writing optimized values")
|
|
{
|
|
SECTION("integer")
|
|
{
|
|
SECTION("array of i")
|
|
{
|
|
json const j = {1, -1};
|
|
std::vector<uint8_t> expected = {'[', '$', 'i', '#', 'i', 2, 1, 0xff};
|
|
CHECK(json::to_ubjson(j, true, true) == expected);
|
|
}
|
|
|
|
SECTION("array of U")
|
|
{
|
|
json const j = {200, 201};
|
|
std::vector<uint8_t> expected = {'[', '$', 'U', '#', 'i', 2, 0xC8, 0xC9};
|
|
CHECK(json::to_ubjson(j, true, true) == expected);
|
|
}
|
|
|
|
SECTION("array of I")
|
|
{
|
|
json const j = {30000, -30000};
|
|
std::vector<uint8_t> expected = {'[', '$', 'I', '#', 'i', 2, 0x75, 0x30, 0x8a, 0xd0};
|
|
CHECK(json::to_ubjson(j, true, true) == expected);
|
|
}
|
|
|
|
SECTION("array of l")
|
|
{
|
|
json const j = {70000, -70000};
|
|
std::vector<uint8_t> expected = {'[', '$', 'l', '#', 'i', 2, 0x00, 0x01, 0x11, 0x70, 0xFF, 0xFE, 0xEE, 0x90};
|
|
CHECK(json::to_ubjson(j, true, true) == expected);
|
|
}
|
|
|
|
SECTION("array of L")
|
|
{
|
|
json const j = {5000000000, -5000000000};
|
|
std::vector<uint8_t> expected = {'[', '$', 'L', '#', 'i', 2, 0x00, 0x00, 0x00, 0x01, 0x2A, 0x05, 0xF2, 0x00, 0xFF, 0xFF, 0xFF, 0xFE, 0xD5, 0xFA, 0x0E, 0x00};
|
|
CHECK(json::to_ubjson(j, true, true) == expected);
|
|
}
|
|
}
|
|
|
|
SECTION("unsigned integer")
|
|
{
|
|
SECTION("array of i")
|
|
{
|
|
json const j = {1u, 2u};
|
|
std::vector<uint8_t> expected = {'[', '$', 'i', '#', 'i', 2, 1, 2};
|
|
std::vector<uint8_t> expected_size = {'[', '#', 'i', 2, 'i', 1, 'i', 2};
|
|
CHECK(json::to_ubjson(j, true, true) == expected);
|
|
CHECK(json::to_ubjson(j, true) == expected_size);
|
|
}
|
|
|
|
SECTION("array of U")
|
|
{
|
|
json const j = {200u, 201u};
|
|
std::vector<uint8_t> expected = {'[', '$', 'U', '#', 'i', 2, 0xC8, 0xC9};
|
|
std::vector<uint8_t> expected_size = {'[', '#', 'i', 2, 'U', 0xC8, 'U', 0xC9};
|
|
CHECK(json::to_ubjson(j, true, true) == expected);
|
|
CHECK(json::to_ubjson(j, true) == expected_size);
|
|
}
|
|
|
|
SECTION("array of I")
|
|
{
|
|
json const j = {30000u, 30001u};
|
|
std::vector<uint8_t> expected = {'[', '$', 'I', '#', 'i', 2, 0x75, 0x30, 0x75, 0x31};
|
|
std::vector<uint8_t> expected_size = {'[', '#', 'i', 2, 'I', 0x75, 0x30, 'I', 0x75, 0x31};
|
|
CHECK(json::to_ubjson(j, true, true) == expected);
|
|
CHECK(json::to_ubjson(j, true) == expected_size);
|
|
}
|
|
|
|
SECTION("array of l")
|
|
{
|
|
json const j = {70000u, 70001u};
|
|
std::vector<uint8_t> expected = {'[', '$', 'l', '#', 'i', 2, 0x00, 0x01, 0x11, 0x70, 0x00, 0x01, 0x11, 0x71};
|
|
std::vector<uint8_t> expected_size = {'[', '#', 'i', 2, 'l', 0x00, 0x01, 0x11, 0x70, 'l', 0x00, 0x01, 0x11, 0x71};
|
|
CHECK(json::to_ubjson(j, true, true) == expected);
|
|
CHECK(json::to_ubjson(j, true) == expected_size);
|
|
}
|
|
|
|
SECTION("array of L")
|
|
{
|
|
json const j = {5000000000u, 5000000001u};
|
|
std::vector<uint8_t> expected = {'[', '$', 'L', '#', 'i', 2, 0x00, 0x00, 0x00, 0x01, 0x2A, 0x05, 0xF2, 0x00, 0x00, 0x00, 0x00, 0x01, 0x2A, 0x05, 0xF2, 0x01};
|
|
std::vector<uint8_t> expected_size = {'[', '#', 'i', 2, 'L', 0x00, 0x00, 0x00, 0x01, 0x2A, 0x05, 0xF2, 0x00, 'L', 0x00, 0x00, 0x00, 0x01, 0x2A, 0x05, 0xF2, 0x01};
|
|
CHECK(json::to_ubjson(j, true, true) == expected);
|
|
CHECK(json::to_ubjson(j, true) == expected_size);
|
|
}
|
|
}
|
|
|
|
SECTION("discarded")
|
|
{
|
|
// a discarded value cannot be serialized to UBJSON, even as part
|
|
// of an optimized array of a single (here: valueless) type
|
|
json const j = {json::value_t::discarded, json::value_t::discarded};
|
|
#if JSON_DIAGNOSTICS
|
|
CHECK_THROWS_WITH_AS(json::to_ubjson(j, true, true), "[json.exception.type_error.321] (/0) cannot serialize discarded value to UBJSON", json::type_error&);
|
|
#else
|
|
CHECK_THROWS_WITH_AS(json::to_ubjson(j, true, true), "[json.exception.type_error.321] cannot serialize discarded value to UBJSON", json::type_error&);
|
|
#endif
|
|
}
|
|
}
|
|
}
|
|
|
|
TEST_CASE("UBJSON nesting does not consume the call stack")
|
|
{
|
|
// Containers used to be read by calling back into the value reader once
|
|
// per element, so the native call stack grew with the nesting depth of the
|
|
// input. '[' alone opens a container, so a payload of repeated '[' crashed
|
|
// the process (#5104), as did the optimized forms, which reach the same
|
|
// path through a type or size annotation. The containers are kept on a
|
|
// heap stack now.
|
|
//
|
|
// Deeply nested values must not be compared, copied or dumped here: those
|
|
// operations are still recursive and would reintroduce the crash.
|
|
json _;
|
|
|
|
SECTION("containers that end at a marker")
|
|
{
|
|
const std::vector<uint8_t> input(500000, '[');
|
|
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(input), "[json.exception.parse_error.110] parse error at byte 500001: syntax error while parsing UBJSON value: unexpected end of input", json::parse_error&);
|
|
CHECK(json::from_ubjson(input, true, false).is_discarded());
|
|
}
|
|
|
|
SECTION("containers with a size")
|
|
{
|
|
std::vector<uint8_t> input;
|
|
for (std::size_t i = 0; i < 100000; ++i)
|
|
{
|
|
input.push_back('[');
|
|
input.push_back('#');
|
|
input.push_back('i');
|
|
input.push_back(1);
|
|
}
|
|
CHECK_THROWS_AS(_ = json::from_ubjson(input), json::parse_error&);
|
|
CHECK(json::from_ubjson(input, true, false).is_discarded());
|
|
}
|
|
|
|
SECTION("containers with a type and a size")
|
|
{
|
|
// '[' is a permitted optimized type in UBJSON, so each element of such
|
|
// a container is itself a container, read without a marker of its own
|
|
std::vector<uint8_t> input;
|
|
for (std::size_t i = 0; i < 100000; ++i)
|
|
{
|
|
const std::vector<uint8_t> level = {'[', '$', '[', '#', 'i', 1};
|
|
input.insert(input.end(), level.begin(), level.end());
|
|
}
|
|
CHECK_THROWS_AS(_ = json::from_ubjson(input), json::parse_error&);
|
|
CHECK(json::from_ubjson(input, true, false).is_discarded());
|
|
}
|
|
|
|
SECTION("a well-formed deep value is read through the SAX interface")
|
|
{
|
|
std::vector<uint8_t> input(100000, '[');
|
|
input.insert(input.end(), 100000, ']');
|
|
|
|
SaxCountdown accept_all(1000000);
|
|
CHECK(json::sax_parse(input, &accept_all, json::input_format_t::ubjson));
|
|
}
|
|
|
|
SECTION("a well-formed deep value is read into a value")
|
|
{
|
|
const std::size_t depth = 10000;
|
|
std::vector<uint8_t> input(depth, '[');
|
|
input.insert(input.end(), depth, ']');
|
|
|
|
json j = json::from_ubjson(input);
|
|
|
|
std::size_t measured = 0;
|
|
const json* p = &j;
|
|
while (p->is_array() && !p->empty())
|
|
{
|
|
p = &p->front();
|
|
++measured;
|
|
}
|
|
// the innermost array is empty, so the descent stops one level short
|
|
CHECK(measured == depth - 1);
|
|
}
|
|
|
|
SECTION("containers are still read the same way")
|
|
{
|
|
CHECK(json::from_ubjson(std::vector<uint8_t>({'[', ']'})) == json::array());
|
|
CHECK(json::from_ubjson(std::vector<uint8_t>({'{', '}'})) == json::object());
|
|
CHECK(json::from_ubjson(std::vector<uint8_t>({'[', '#', 'i', 0})) == json::array());
|
|
CHECK(json::from_ubjson(std::vector<uint8_t>({'{', '#', 'i', 0})) == json::object());
|
|
CHECK(json::from_ubjson(std::vector<uint8_t>({'[', '$', 'i', '#', 'i', 2, 1, 2})) == json({1, 2}));
|
|
CHECK(json::from_ubjson(std::vector<uint8_t>({'[', '#', 'i', 2, 'i', 1, 'i', 2})) == json({1, 2}));
|
|
CHECK(json::from_ubjson(std::vector<uint8_t>({'{', '$', 'i', '#', 'i', 1, 'i', 1, 'a', 1})) == json({{"a", 1}}));
|
|
// a no-op is not a value, so a container of them holds none
|
|
CHECK(json::from_ubjson(std::vector<uint8_t>({'[', '$', 'N', '#', 'i', 2})) == json::array());
|
|
// sized and unsized forms nested inside one another
|
|
CHECK(json::from_ubjson(std::vector<uint8_t>({'[', '[', '#', 'i', 2, 'i', 1, 'i', 2, ']'})) == json({{1, 2}}));
|
|
CHECK(json::from_ubjson(std::vector<uint8_t>({'[', '#', 'i', 1, '[', 'i', 1, ']'})) == json({{1}}));
|
|
// an optimized container of containers
|
|
CHECK(json::from_ubjson(std::vector<uint8_t>({'[', '$', '[', '#', 'i', 2, 'i', 1, ']', 'i', 2, ']'})) == json({{1}, {2}}));
|
|
}
|
|
|
|
SECTION("BJData containers are still read the same way")
|
|
{
|
|
// the ND-array wrapper and the binary shortcut are complete values,
|
|
// not containers the reader descends into
|
|
CHECK(json::from_bjdata(std::vector<uint8_t>({'[', '$', 'U', '#', '[', '$', 'i', '#', 'i', 2, 2, 3, 1, 2, 3, 4, 5, 6})) ==
|
|
json({{"_ArrayType_", "uint8"}, {"_ArraySize_", {2, 3}}, {"_ArrayData_", {1, 2, 3, 4, 5, 6}}}));
|
|
CHECK(json::from_bjdata(std::vector<uint8_t>({'[', '$', 'i', '#', 'i', 2, 1, 2})) == json({1, 2}));
|
|
CHECK(json::from_bjdata(std::vector<uint8_t>({'[', '[', 'i', 1, ']', ']'})) == json({{1}}));
|
|
}
|
|
}
|
|
|
|
TEST_CASE("UBJSON input that cannot be read is discarded by every overload")
|
|
{
|
|
std::vector<std::uint8_t> input = json::to_ubjson(json({{"a", {1, 2}}}));
|
|
input.pop_back();
|
|
|
|
json _;
|
|
CHECK_THROWS_AS(_ = json::from_ubjson(input.begin(), input.end()), json::parse_error&);
|
|
CHECK(json::from_ubjson(input, true, false).is_discarded());
|
|
CHECK(json::from_ubjson(input.begin(), input.end(), true, false).is_discarded());
|
|
#ifndef JSON_TEST_DEPRECATED_FUNCTIONS_DELETED
|
|
CHECK(json::from_ubjson(input.data(), input.size(), true, false).is_discarded());
|
|
CHECK(json::from_ubjson({input.data(), input.size()}, true, false).is_discarded());
|
|
#endif
|
|
}
|
|
|
|
TEST_CASE("UBJSON SAX parsing stops at every event")
|
|
{
|
|
// Containers are opened and closed by the loop that reads them; a SAX
|
|
// handler that rejects any event - including the end of a nested
|
|
// container - must stop the parse right there.
|
|
const auto count_events = [](const std::vector<std::uint8_t>& input)
|
|
{
|
|
int events = 0;
|
|
while (true)
|
|
{
|
|
SaxCountdown scp(events);
|
|
if (json::sax_parse(input, &scp, json::input_format_t::ubjson))
|
|
{
|
|
return events;
|
|
}
|
|
++events;
|
|
REQUIRE(events < 1000);
|
|
}
|
|
};
|
|
|
|
// 20 events: every container kind closes inside another one
|
|
const json j = json::parse(R"({"a": [1, {"b": []}], "c": {"d": [[2]]}})");
|
|
CHECK(count_events(json::to_ubjson(j)) == 20);
|
|
CHECK(count_events(json::to_ubjson(j, true)) == 20);
|
|
CHECK(count_events(json::to_ubjson(j, true, true)) == 20);
|
|
}
|
|
|
|
TEST_CASE("UBJSON optimized arrays of a valueless type are bounded")
|
|
{
|
|
// An element of type 'Z', 'T' or 'F' is encoded by its marker alone, so an
|
|
// optimized array of one of those has no payload and the declared count is
|
|
// the only thing deciding how much is allocated. Ten bytes used to produce
|
|
// billions of values (#2793); every other type costs at least one byte per
|
|
// element and is bounded by the end of the input.
|
|
json _;
|
|
|
|
SECTION("an excessive count is rejected")
|
|
{
|
|
// 'l' is a big-endian int32: 0x7FFFFFFF elements, about 34 GB of value;
|
|
// OSS-Fuzz reported this shape as a parse_ubjson_fuzzer timeout
|
|
// (testcase 6347769435193344, no issue filed)
|
|
for (const auto marker :
|
|
{'Z', 'T', 'F'
|
|
})
|
|
{
|
|
const std::vector<uint8_t> input = {'[', '$', static_cast<uint8_t>(marker), '#', 'l', 0x7F, 0xFF, 0xFF, 0xFF};
|
|
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(input), "[json.exception.out_of_range.408] syntax error while parsing UBJSON size: excessive array size", json::out_of_range&);
|
|
CHECK(json::from_ubjson(input, true, false).is_discarded());
|
|
}
|
|
}
|
|
|
|
SECTION("a count equal to SIZE_MAX is rejected")
|
|
{
|
|
// the reader marks a container without a count by SIZE_MAX, so a
|
|
// count of that value must not be read as "no count"; UBJSON's
|
|
// largest count type (int64) can only reach it where size_t is 32 bits
|
|
#if SIZE_MAX == 0xffffffff
|
|
const std::vector<uint8_t> input = {'[', '#', 'L', 0x00, 0x00, 0x00, 0x00, 0xFF, 0xFF, 0xFF, 0xFF, 'i', 0x01, ']'};
|
|
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(input), "[json.exception.out_of_range.408] syntax error while parsing UBJSON size: integer value overflow", json::out_of_range&);
|
|
CHECK(json::from_ubjson(input, true, false).is_discarded());
|
|
#endif
|
|
}
|
|
|
|
SECTION("ordinary counts are unaffected")
|
|
{
|
|
CHECK(json::from_ubjson(std::vector<uint8_t>({'[', '$', 'Z', '#', 'i', 3})) == json({nullptr, nullptr, nullptr}));
|
|
CHECK(json::from_ubjson(std::vector<uint8_t>({'[', '$', 'T', '#', 'i', 2})) == json({true, true}));
|
|
CHECK(json::from_ubjson(std::vector<uint8_t>({'[', '$', 'F', '#', 'i', 2})) == json({false, false}));
|
|
// 'N' is a no-op rather than a value, and still yields an empty array
|
|
CHECK(json::from_ubjson(std::vector<uint8_t>({'[', '$', 'N', '#', 'i', 2})) == json::array());
|
|
}
|
|
|
|
SECTION("a type with a payload is unaffected")
|
|
{
|
|
// A count past the limit is not rejected for 'U', which costs a byte
|
|
// per element and is bounded by the end of the input instead. The
|
|
// count is kept just past the limit rather than made huge, because a
|
|
// count that also exceeds the array's max_size() is reported as
|
|
// out_of_range before the input runs out, and max_size() depends on
|
|
// the width of std::size_t.
|
|
const std::vector<uint8_t> input = {'[', '$', 'U', '#', 'l', 0x00, 0x10, 0x00, 0x01};
|
|
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(input), "[json.exception.parse_error.110] parse error at byte 10: syntax error while parsing UBJSON number: unexpected end of input", json::parse_error&);
|
|
CHECK(json::from_ubjson(input, true, false).is_discarded());
|
|
}
|
|
|
|
SECTION("the writer stays within what the reader accepts")
|
|
{
|
|
// below the limit the optimized form is used and is tiny; above it the
|
|
// writer falls back so that the result can still be read back
|
|
json const at_limit(1048576, nullptr);
|
|
const auto v_at_limit = json::to_ubjson(at_limit, true, true);
|
|
CHECK(v_at_limit.size() == 9);
|
|
CHECK(v_at_limit.at(1) == '$');
|
|
CHECK(json::from_ubjson(v_at_limit) == at_limit);
|
|
|
|
json const above_limit(1048577, nullptr);
|
|
const auto v_above_limit = json::to_ubjson(above_limit, true, true);
|
|
CHECK(v_above_limit.at(1) != '$');
|
|
CHECK(json::from_ubjson(v_above_limit) == above_limit);
|
|
}
|
|
}
|
|
|
|
TEST_CASE("UBJSON the writer keeps the valueless elements of an input within the base")
|
|
{
|
|
// The budget of the reader is 1048576 plus a rate per byte read, see
|
|
// unit-ubjson_valueless.cpp, which tests it for several values of the rate.
|
|
// The writer does not use the rate and stays within the base
|
|
json _;
|
|
|
|
SECTION("to_ubjson writes the typed form up to the base only")
|
|
{
|
|
// 1048 rows of 1000 values fit under the base of 1048576, the 1049th
|
|
// would not: its elements are written one by one. Written all typed,
|
|
// 1100 rows of 6 bytes would be refused by a reader with the default
|
|
// rate of the macro JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE (1074 rows
|
|
// are the most it accepts); the writer does not follow the macro, since
|
|
// the limit of a reader is never below the base. The rows hold null,
|
|
// true or false, and the fallback does not depend on which
|
|
std::vector<std::uint8_t> encoded;
|
|
{
|
|
json table = json::array();
|
|
for (int row = 0; row < 1100; ++row)
|
|
{
|
|
const json cell = row % 3 == 0 ? json(nullptr) : json(row % 3 == 1);
|
|
table.push_back(json(1000, cell));
|
|
}
|
|
encoded = json::to_ubjson(table, true, true);
|
|
}
|
|
// outer array: `[$[#I` and the count (7 bytes), then 1048 typed rows of 6
|
|
// bytes (`$Z#I` and two bytes) and 52 rows of `#I`, two bytes, and 1000 `Z`
|
|
CHECK(encoded.size() == 7 + (1048 * 6) + (52 * (4 + 1000)));
|
|
|
|
// the output is read back
|
|
utils::ValuelessEventCounter sax;
|
|
CHECK(json::sax_parse(encoded, &sax, json::input_format_t::ubjson));
|
|
CHECK(sax.error_id == 0);
|
|
CHECK(sax.values == 1100u * 1000u);
|
|
}
|
|
|
|
SECTION("BJData is not affected")
|
|
{
|
|
// BJData has no optimized arrays of these types, and the writer does
|
|
// not charge them
|
|
json table = json::array();
|
|
for (int row = 0; row < 3; ++row)
|
|
{
|
|
table.push_back(json(10, nullptr));
|
|
}
|
|
const auto encoded = json::to_bjdata(table, true, true);
|
|
CHECK(json::from_bjdata(encoded) == table);
|
|
|
|
const std::vector<std::uint8_t> input = {'[', '$', 'Z', '#', 'l', 0x00, 0x10, 0x00, 0x00};
|
|
CHECK_THROWS_AS(_ = json::from_bjdata(input), json::parse_error&);
|
|
}
|
|
|
|
SECTION("an object of a valueless type needs a key for every value")
|
|
{
|
|
// an object member costs a key, so the count is bounded by the input
|
|
// and is not charged to the budget
|
|
const std::vector<std::uint8_t> input = {'{', '$', 'Z', '#', 'l', 0x00, 0x10, 0x00, 0x00};
|
|
utils::ValuelessEventCounter sax;
|
|
CHECK(!json::sax_parse(input, &sax, json::input_format_t::ubjson));
|
|
CHECK(sax.error_id == 110);
|
|
CHECK(sax.values == 0);
|
|
}
|
|
}
|
|
|
|
TEST_CASE("UBJSON an optimized array of no-ops is empty whatever its count")
|
|
{
|
|
// A no-op ('N') is not a value, so `[$N#<n>` is an array without elements
|
|
// for every n. The count used to be passed on to the SAX parser, which
|
|
// told it to expect n elements, and the DOM parser reserved storage for
|
|
// them: 16384 values (256 KB) for a count of 2^31 in 9 bytes of input.
|
|
SECTION("the SAX parser is told that the array is empty")
|
|
{
|
|
for (const std::vector<std::uint8_t>& input :
|
|
{
|
|
std::vector<std::uint8_t> {'[', '$', 'N', '#', 'i', 2},
|
|
std::vector<std::uint8_t> {'[', '$', 'N', '#', 'I', 0x7F, 0xFF},
|
|
std::vector<std::uint8_t> {'[', '$', 'N', '#', 'l', 0x7F, 0xFF, 0xFF, 0xFF},
|
|
std::vector<std::uint8_t> {'[', '$', 'N', '#', 'i', 0}
|
|
})
|
|
{
|
|
SaxEventLogger logger;
|
|
CHECK(json::sax_parse(input, &logger, json::input_format_t::ubjson));
|
|
CHECK(logger.events == std::vector<std::string>({"start_array(0)", "end_array()"}));
|
|
}
|
|
|
|
// between other values, the next value is read as usual
|
|
const std::vector<std::uint8_t> nested = {'[', '[', '$', 'N', '#', 'l', 0x7F, 0xFF, 0xFF, 0xFF, 'T', '[', '$', 'N', '#', 'i', 3, ']'};
|
|
SaxEventLogger logger;
|
|
CHECK(json::sax_parse(nested, &logger, json::input_format_t::ubjson));
|
|
CHECK(logger.events == std::vector<std::string>({"start_array()", "start_array(0)", "end_array()", "boolean(true)", "start_array(0)", "end_array()", "end_array()"}));
|
|
}
|
|
|
|
SECTION("the DOM parser reserves nothing")
|
|
{
|
|
// 2000 arrays of 9 bytes that announce 2^31 elements each: with a
|
|
// reservation of 16384 values per array this took about 500 MB
|
|
std::vector<std::uint8_t> input = {'['};
|
|
for (int i = 0; i < 2000; ++i)
|
|
{
|
|
input.insert(input.end(), {'[', '$', 'N', '#', 'l', 0x7F, 0xFF, 0xFF, 0xFF});
|
|
}
|
|
input.push_back(']');
|
|
|
|
const json j = json::from_ubjson(input);
|
|
REQUIRE(j.size() == 2000);
|
|
std::size_t capacity = 0;
|
|
for (const auto& element : j)
|
|
{
|
|
CHECK(element.is_array());
|
|
CHECK(element.empty());
|
|
capacity += element.get_ref<const json::array_t&>().capacity();
|
|
}
|
|
CHECK(capacity == 0);
|
|
|
|
// the 7-byte form inside an optimized array of arrays
|
|
std::vector<std::uint8_t> typed = {'[', '$', '[', '#', 'I', 0x07, 0xD0};
|
|
for (int i = 0; i < 2000; ++i)
|
|
{
|
|
typed.insert(typed.end(), {'$', 'N', '#', 'I', 0x7F, 0xFF});
|
|
}
|
|
const json k = json::from_ubjson(typed);
|
|
REQUIRE(k.size() == 2000);
|
|
capacity = 0;
|
|
for (const auto& element : k)
|
|
{
|
|
CHECK(element.empty());
|
|
capacity += element.get_ref<const json::array_t&>().capacity();
|
|
}
|
|
CHECK(capacity == 0);
|
|
}
|
|
|
|
SECTION("a count that no container could hold is no longer an error")
|
|
{
|
|
// the array is empty, so nothing has to fit: 0x7FFFFFFF used to be
|
|
// reported as out_of_range.408 where it exceeds the maximum size of
|
|
// an array, and is accepted now
|
|
const std::vector<std::uint8_t> input = {'[', '$', 'N', '#', 'l', 0x7F, 0xFF, 0xFF, 0xFF};
|
|
CHECK(json::from_ubjson(input) == json::array());
|
|
|
|
// a count that does not fit into std::size_t still is
|
|
#if SIZE_MAX == 0xffffffff
|
|
const std::vector<std::uint8_t> too_large = {'[', '$', 'N', '#', 'L', 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x00};
|
|
json _;
|
|
CHECK_THROWS_AS(_ = json::from_ubjson(too_large), json::out_of_range&);
|
|
#endif
|
|
}
|
|
}
|
|
|
|
TEST_CASE("issue #5405 - array reserve for definite-length UBJSON arrays")
|
|
{
|
|
#if !defined(JSON_NOEXCEPTION)
|
|
// this SECTION relies on catching a thrown exception to distinguish
|
|
// which of two acceptable, bounded rejections a hostile header took;
|
|
// under JSON_NOEXCEPTION, JSON_THROW never produces a catchable C++
|
|
// exception (it aborts instead), so this cannot be tested that way here
|
|
SECTION("a huge claimed length with no element data must not over-allocate")
|
|
{
|
|
// optimized form [$type#count: type 'i' (int8), count as a four-byte
|
|
// 'l' (int32) of 0x7FFFFFFF (2147483647), but no element data at all.
|
|
// max_size() for a std::vector is far larger than this count, so it
|
|
// does not reject the header outright; the (capped) reservation must
|
|
// not attempt to allocate space for billions of elements before the
|
|
// missing data is detected.
|
|
json _;
|
|
const std::vector<uint8_t> input = {'[', '$', 'i', '#', 'l', 0x7F, 0xFF, 0xFF, 0xFF};
|
|
// On a platform where std::vector<json>::max_size() is smaller than
|
|
// the claimed count (e.g. 32-bit, where max_size() is bounded by a
|
|
// 32-bit SIZE_MAX divided by sizeof(json)), the SAX consumer's own
|
|
// check rejects the header outright (out_of_range.408, with the
|
|
// claimed count in the message) instead of accepting it and only
|
|
// finding it short of data once the (capped) reservation looks for
|
|
// element bytes that were never provided (parse_error.110). Either
|
|
// is an acceptable, bounded rejection of the hostile header -- the
|
|
// property under test is that no path attempts to allocate space
|
|
// for billions of elements.
|
|
bool threw = false;
|
|
try
|
|
{
|
|
_ = json::from_ubjson(input);
|
|
}
|
|
catch (const json::parse_error& e)
|
|
{
|
|
threw = true;
|
|
CHECK(e.id == 110);
|
|
CHECK(std::string(e.what()) == "[json.exception.parse_error.110] parse error at byte 10: syntax error while parsing UBJSON number: unexpected end of input");
|
|
}
|
|
catch (const json::out_of_range& e)
|
|
{
|
|
threw = true;
|
|
CHECK(e.id == 408);
|
|
CHECK(std::string(e.what()).find("excessive array size") != std::string::npos);
|
|
}
|
|
CHECK(threw);
|
|
|
|
// json_sax_dom_parser::start_array()'s max_size() check (unlike the
|
|
// scanner's own parse_error path) throws unconditionally via
|
|
// JSON_THROW rather than going through sax->parse_error(), so it is
|
|
// not gated by allow_exceptions=false on a platform where this
|
|
// header hits that check (e.g. 32-bit, see above) -- allow either
|
|
// a discarded result or the same out_of_range it throws with
|
|
// exceptions enabled.
|
|
try
|
|
{
|
|
CHECK(json::from_ubjson(input, true, false).is_discarded());
|
|
}
|
|
catch (const json::out_of_range& e)
|
|
{
|
|
CHECK(e.id == 408);
|
|
}
|
|
}
|
|
#endif
|
|
|
|
SECTION("arrays of various sizes decode to the same value as before the reserve optimization")
|
|
{
|
|
for (const auto size :
|
|
{
|
|
std::size_t{0}, std::size_t{1}, std::size_t{5}, // small
|
|
std::size_t{16384}, // exactly at the reserve cap
|
|
std::size_t{20000} // above the reserve cap
|
|
})
|
|
{
|
|
CAPTURE(size)
|
|
json j = json::array();
|
|
for (std::size_t i = 0; i < size; ++i)
|
|
{
|
|
j.push_back(static_cast<int>(i % 1000));
|
|
}
|
|
|
|
// exercise both the plain and the optimized [$type#count encoding
|
|
const auto packed_plain = json::to_ubjson(j);
|
|
CHECK(json::from_ubjson(packed_plain) == j);
|
|
|
|
const auto packed_optimized = json::to_ubjson(j, true, true);
|
|
CHECK(json::from_ubjson(packed_optimized) == j);
|
|
}
|
|
}
|
|
|
|
SECTION("a user-defined SAX consumer is unaffected by the internal DOM reserve optimization")
|
|
{
|
|
// the reserve() call is local to json_sax_dom_parser / json_sax_dom_callback_parser;
|
|
// a custom SAX consumer that does not touch a DOM array sees identical events
|
|
json j = json::array();
|
|
for (int i = 0; i < 100; ++i)
|
|
{
|
|
j.push_back(i);
|
|
}
|
|
const auto packed = json::to_ubjson(j, true, true);
|
|
|
|
SaxCountdown scp(1000000); // large enough to never trigger an abort
|
|
CHECK(json::sax_parse(packed, &scp, json::input_format_t::ubjson));
|
|
}
|
|
}
|
|
|
|
TEST_CASE("UBJSON - open arrays reserve only what the input has backed")
|
|
{
|
|
// an array that announces its size gets storage for (up to 16384 of) its
|
|
// elements when it opens; the arrays that are open at the same time must
|
|
// not hold more than the input read so far can back
|
|
SECTION("nested headers that announce 16384 elements and are never completed")
|
|
{
|
|
// 200 levels would hold 200 * 16384 * sizeof(json),
|
|
// if each of them reserved what it announces
|
|
const auto input = utils::repeated({'[', '#', 'I', 0x40, 0x00}, 200);
|
|
|
|
utils::counts() = utils::allocation_counts{};
|
|
const auto j = utils::counted_json::from_ubjson(input, true, false);
|
|
CHECK(j.is_discarded());
|
|
CAPTURE(utils::counts().peak)
|
|
// the credit is 65,536 values; the base peaks at more than 3 million
|
|
CHECK(utils::counts().peak < sizeof(utils::counted_json) * 128 * 1024);
|
|
}
|
|
|
|
SECTION("a completed array is still reserved in one allocation")
|
|
{
|
|
json array = json::array();
|
|
for (int i = 0; i < 10000; ++i)
|
|
{
|
|
array.push_back(i % 100);
|
|
}
|
|
|
|
// an array without a size is not reserved, so only these two
|
|
{
|
|
const auto packed = json::to_ubjson(array, true);
|
|
|
|
utils::counts() = utils::allocation_counts{};
|
|
const auto j = utils::counted_json::from_ubjson(packed);
|
|
CHECK(j.size() == 10000);
|
|
// the array and its storage; growing the storage as the elements
|
|
// arrive would take more than a dozen allocations
|
|
CAPTURE(utils::counts().allocations)
|
|
CHECK(utils::counts().allocations <= 4);
|
|
}
|
|
|
|
{
|
|
const auto packed = json::to_ubjson(array, true, true);
|
|
|
|
utils::counts() = utils::allocation_counts{};
|
|
const auto j = utils::counted_json::from_ubjson(packed);
|
|
CHECK(j.size() == 10000);
|
|
// the array and its storage; growing the storage as the elements
|
|
// arrive would take more than a dozen allocations
|
|
CAPTURE(utils::counts().allocations)
|
|
CHECK(utils::counts().allocations <= 4);
|
|
}
|
|
}
|
|
}
|
|
|
|
|
|
TEST_CASE("Universal Binary JSON Specification Examples 1")
|
|
{
|
|
SECTION("Null Value")
|
|
{
|
|
json const j = {{"passcode", nullptr}};
|
|
std::vector<uint8_t> const v = {'{', 'i', 8, 'p', 'a', 's', 's', 'c', 'o', 'd', 'e', 'Z', '}'};
|
|
CHECK(json::to_ubjson(j) == v);
|
|
CHECK(json::from_ubjson(v) == j);
|
|
}
|
|
|
|
SECTION("No-Op Value")
|
|
{
|
|
json const j = {"foo", "bar", "baz"};
|
|
std::vector<uint8_t> const v = {'[', 'S', 'i', 3, 'f', 'o', 'o',
|
|
'S', 'i', 3, 'b', 'a', 'r',
|
|
'S', 'i', 3, 'b', 'a', 'z', ']'
|
|
};
|
|
std::vector<uint8_t> const v2 = {'[', 'S', 'i', 3, 'f', 'o', 'o', 'N',
|
|
'S', 'i', 3, 'b', 'a', 'r', 'N', 'N', 'N',
|
|
'S', 'i', 3, 'b', 'a', 'z', 'N', 'N', ']'
|
|
};
|
|
CHECK(json::to_ubjson(j) == v);
|
|
CHECK(json::from_ubjson(v) == j);
|
|
CHECK(json::from_ubjson(v2) == j);
|
|
}
|
|
|
|
SECTION("Boolean Types")
|
|
{
|
|
json const j = {{"authorized", true}, {"verified", false}};
|
|
std::vector<uint8_t> const v = {'{', 'i', 10, 'a', 'u', 't', 'h', 'o', 'r', 'i', 'z', 'e', 'd', 'T',
|
|
'i', 8, 'v', 'e', 'r', 'i', 'f', 'i', 'e', 'd', 'F', '}'
|
|
};
|
|
CHECK(json::to_ubjson(j) == v);
|
|
CHECK(json::from_ubjson(v) == j);
|
|
}
|
|
|
|
SECTION("Numeric Types")
|
|
{
|
|
json const j =
|
|
{
|
|
{"int8", 16},
|
|
{"uint8", 255},
|
|
{"int16", 32767},
|
|
{"int32", 2147483647},
|
|
{"int64", 9223372036854775807},
|
|
{"float64", 113243.7863123}
|
|
};
|
|
std::vector<uint8_t> const v = {'{',
|
|
'i', 7, 'f', 'l', 'o', 'a', 't', '6', '4', 'D', 0x40, 0xfb, 0xa5, 0xbc, 0x94, 0xbc, 0x34, 0xcf,
|
|
'i', 5, 'i', 'n', 't', '1', '6', 'I', 0x7f, 0xff,
|
|
'i', 5, 'i', 'n', 't', '3', '2', 'l', 0x7f, 0xff, 0xff, 0xff,
|
|
'i', 5, 'i', 'n', 't', '6', '4', 'L', 0x7f, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff,
|
|
'i', 4, 'i', 'n', 't', '8', 'i', 16,
|
|
'i', 5, 'u', 'i', 'n', 't', '8', 'U', 0xff,
|
|
'}'
|
|
};
|
|
CHECK(json::to_ubjson(j) == v);
|
|
CHECK(json::from_ubjson(v) == j);
|
|
}
|
|
|
|
SECTION("Char Type")
|
|
{
|
|
json const j = {{"rolecode", "a"}, {"delim", ";"}};
|
|
std::vector<uint8_t> const v = {'{', 'i', 5, 'd', 'e', 'l', 'i', 'm', 'C', ';', 'i', 8, 'r', 'o', 'l', 'e', 'c', 'o', 'd', 'e', 'C', 'a', '}'};
|
|
//CHECK(json::to_ubjson(j) == v);
|
|
CHECK(json::from_ubjson(v) == j);
|
|
}
|
|
|
|
SECTION("String Type")
|
|
{
|
|
SECTION("English")
|
|
{
|
|
json const j = "hello";
|
|
std::vector<uint8_t> const v = {'S', 'i', 5, 'h', 'e', 'l', 'l', 'o'};
|
|
CHECK(json::to_ubjson(j) == v);
|
|
CHECK(json::from_ubjson(v) == j);
|
|
}
|
|
|
|
SECTION("Russian")
|
|
{
|
|
json const j = "привет";
|
|
std::vector<uint8_t> const v = {'S', 'i', 12, 0xD0, 0xBF, 0xD1, 0x80, 0xD0, 0xB8, 0xD0, 0xB2, 0xD0, 0xB5, 0xD1, 0x82};
|
|
CHECK(json::to_ubjson(j) == v);
|
|
CHECK(json::from_ubjson(v) == j);
|
|
}
|
|
|
|
SECTION("Russian")
|
|
{
|
|
json const j = "مرحبا";
|
|
std::vector<uint8_t> const v = {'S', 'i', 10, 0xD9, 0x85, 0xD8, 0xB1, 0xD8, 0xAD, 0xD8, 0xA8, 0xD8, 0xA7};
|
|
CHECK(json::to_ubjson(j) == v);
|
|
CHECK(json::from_ubjson(v) == j);
|
|
}
|
|
|
|
SECTION("ill-formed UTF-8 (see #5529, #5651)")
|
|
{
|
|
// none of the binary format specs requires a decoder to reject
|
|
// ill-formed UTF-8 in a text string, so a value whose bytes are
|
|
// not valid UTF-8 (0xC0 0xAE is an overlong encoding of '.')
|
|
// round-trips byte for byte as a string value; to_ubjson() writes
|
|
// the bytes unchanged, as before 3.13.0, unless
|
|
// JSON_STRICT_BINARY_UTF8 is enabled (see
|
|
// unit-binary_utf8_strict.cpp)
|
|
const std::vector<uint8_t> v = {'S', 'i', 2, 0xc0, 0xae};
|
|
json j;
|
|
CHECK_NOTHROW(j = json::from_ubjson(v));
|
|
REQUIRE(j.is_string());
|
|
CHECK(j.get_ref<const json::string_t&>() == std::string("\xc0\xae"));
|
|
CHECK_THROWS_AS(utils::ignore_return_value(j.dump()), json::type_error&);
|
|
CHECK(json::from_ubjson(json::to_ubjson(j)) == j);
|
|
|
|
// the same bytes as an object key round-trip as well
|
|
const std::vector<uint8_t> v_key = {'{', 'i', 2, 0xc0, 0xae, 'i', 1, '}'};
|
|
json j_key;
|
|
CHECK_NOTHROW(j_key = json::from_ubjson(v_key));
|
|
REQUIRE(j_key.is_object());
|
|
CHECK(j_key.contains(std::string("\xc0\xae")));
|
|
CHECK(json::from_ubjson(json::to_ubjson(j_key)) == j_key);
|
|
|
|
CHECK(json::from_ubjson(json::to_ubjson(json("\xFF"))) == json("\xFF"));
|
|
// a truncated multi-byte sequence
|
|
CHECK(json::from_ubjson(json::to_ubjson(json("\xC3"))) == json("\xC3"));
|
|
// an encoded surrogate half (U+D800)
|
|
CHECK(json::from_ubjson(json::to_ubjson(json("\xED\xA0\x80"))) == json("\xED\xA0\x80"));
|
|
// an overlong encoding of '.'
|
|
CHECK(json::from_ubjson(json::to_ubjson(json("\xC0\xAF"))) == json("\xC0\xAF"));
|
|
|
|
// an object key with ill-formed UTF-8 is kept the same way
|
|
CHECK(json::from_ubjson(json::to_ubjson(json{{"\xFF", 1}})) == json{{"\xFF", 1}});
|
|
}
|
|
}
|
|
|
|
SECTION("Array Type")
|
|
{
|
|
SECTION("size=false type=false")
|
|
{
|
|
// note the float has been replaced by a double
|
|
json const j = {nullptr, true, false, 4782345193, 153.132, "ham"};
|
|
std::vector<uint8_t> const v = {'[', 'Z', 'T', 'F', 'L', 0x00, 0x00, 0x00, 0x01, 0x1D, 0x0C, 0xCB, 0xE9, 'D', 0x40, 0x63, 0x24, 0x39, 0x58, 0x10, 0x62, 0x4e, 'S', 'i', 3, 'h', 'a', 'm', ']'};
|
|
CHECK(json::to_ubjson(j) == v);
|
|
CHECK(json::from_ubjson(v) == j);
|
|
}
|
|
|
|
SECTION("size=true type=false")
|
|
{
|
|
// note the float has been replaced by a double
|
|
json const j = {nullptr, true, false, 4782345193, 153.132, "ham"};
|
|
std::vector<uint8_t> const v = {'[', '#', 'i', 6, 'Z', 'T', 'F', 'L', 0x00, 0x00, 0x00, 0x01, 0x1D, 0x0C, 0xCB, 0xE9, 'D', 0x40, 0x63, 0x24, 0x39, 0x58, 0x10, 0x62, 0x4e, 'S', 'i', 3, 'h', 'a', 'm'};
|
|
CHECK(json::to_ubjson(j, true) == v);
|
|
CHECK(json::from_ubjson(v) == j);
|
|
}
|
|
|
|
SECTION("size=true type=true")
|
|
{
|
|
// note the float has been replaced by a double
|
|
json const j = {nullptr, true, false, 4782345193, 153.132, "ham"};
|
|
std::vector<uint8_t> const v = {'[', '#', 'i', 6, 'Z', 'T', 'F', 'L', 0x00, 0x00, 0x00, 0x01, 0x1D, 0x0C, 0xCB, 0xE9, 'D', 0x40, 0x63, 0x24, 0x39, 0x58, 0x10, 0x62, 0x4e, 'S', 'i', 3, 'h', 'a', 'm'};
|
|
CHECK(json::to_ubjson(j, true, true) == v);
|
|
CHECK(json::from_ubjson(v) == j);
|
|
}
|
|
}
|
|
|
|
SECTION("Object Type")
|
|
{
|
|
SECTION("size=false type=false")
|
|
{
|
|
json const j =
|
|
{
|
|
{
|
|
"post", {
|
|
{"id", 1137},
|
|
{"author", "rkalla"},
|
|
{"timestamp", 1364482090592},
|
|
{"body", "I totally agree!"}
|
|
}
|
|
}
|
|
};
|
|
std::vector<uint8_t> const v = {'{', 'i', 4, 'p', 'o', 's', 't', '{',
|
|
'i', 6, 'a', 'u', 't', 'h', 'o', 'r', 'S', 'i', 6, 'r', 'k', 'a', 'l', 'l', 'a',
|
|
'i', 4, 'b', 'o', 'd', 'y', 'S', 'i', 16, 'I', ' ', 't', 'o', 't', 'a', 'l', 'l', 'y', ' ', 'a', 'g', 'r', 'e', 'e', '!',
|
|
'i', 2, 'i', 'd', 'I', 0x04, 0x71,
|
|
'i', 9, 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', 'L', 0x00, 0x00, 0x01, 0x3D, 0xB1, 0x78, 0x66, 0x60,
|
|
'}', '}'
|
|
};
|
|
CHECK(json::to_ubjson(j) == v);
|
|
CHECK(json::from_ubjson(v) == j);
|
|
}
|
|
|
|
SECTION("size=true type=false")
|
|
{
|
|
json const j =
|
|
{
|
|
{
|
|
"post", {
|
|
{"id", 1137},
|
|
{"author", "rkalla"},
|
|
{"timestamp", 1364482090592},
|
|
{"body", "I totally agree!"}
|
|
}
|
|
}
|
|
};
|
|
std::vector<uint8_t> const v = {'{', '#', 'i', 1, 'i', 4, 'p', 'o', 's', 't', '{', '#', 'i', 4,
|
|
'i', 6, 'a', 'u', 't', 'h', 'o', 'r', 'S', 'i', 6, 'r', 'k', 'a', 'l', 'l', 'a',
|
|
'i', 4, 'b', 'o', 'd', 'y', 'S', 'i', 16, 'I', ' ', 't', 'o', 't', 'a', 'l', 'l', 'y', ' ', 'a', 'g', 'r', 'e', 'e', '!',
|
|
'i', 2, 'i', 'd', 'I', 0x04, 0x71,
|
|
'i', 9, 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', 'L', 0x00, 0x00, 0x01, 0x3D, 0xB1, 0x78, 0x66, 0x60
|
|
};
|
|
CHECK(json::to_ubjson(j, true) == v);
|
|
CHECK(json::from_ubjson(v) == j);
|
|
}
|
|
|
|
SECTION("size=true type=true")
|
|
{
|
|
json const j =
|
|
{
|
|
{
|
|
"post", {
|
|
{"id", 1137},
|
|
{"author", "rkalla"},
|
|
{"timestamp", 1364482090592},
|
|
{"body", "I totally agree!"}
|
|
}
|
|
}
|
|
};
|
|
std::vector<uint8_t> const v = {'{', '$', '{', '#', 'i', 1, 'i', 4, 'p', 'o', 's', 't', '#', 'i', 4,
|
|
'i', 6, 'a', 'u', 't', 'h', 'o', 'r', 'S', 'i', 6, 'r', 'k', 'a', 'l', 'l', 'a',
|
|
'i', 4, 'b', 'o', 'd', 'y', 'S', 'i', 16, 'I', ' ', 't', 'o', 't', 'a', 'l', 'l', 'y', ' ', 'a', 'g', 'r', 'e', 'e', '!',
|
|
'i', 2, 'i', 'd', 'I', 0x04, 0x71,
|
|
'i', 9, 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', 'L', 0x00, 0x00, 0x01, 0x3D, 0xB1, 0x78, 0x66, 0x60
|
|
};
|
|
CHECK(json::to_ubjson(j, true, true) == v);
|
|
CHECK(json::from_ubjson(v) == j);
|
|
}
|
|
}
|
|
|
|
SECTION("Optimized Format")
|
|
{
|
|
SECTION("Array Example")
|
|
{
|
|
SECTION("No Optimization")
|
|
{
|
|
// note the floats have been replaced by doubles
|
|
json const j = {29.97, 31.13, 67.0, 2.113, 23.888};
|
|
std::vector<uint8_t> const v = {'[',
|
|
'D', 0x40, 0x3d, 0xf8, 0x51, 0xeb, 0x85, 0x1e, 0xb8,
|
|
'D', 0x40, 0x3f, 0x21, 0x47, 0xae, 0x14, 0x7a, 0xe1,
|
|
'D', 0x40, 0x50, 0xc0, 0x00, 0x00, 0x00, 0x00, 0x00,
|
|
'D', 0x40, 0x00, 0xe7, 0x6c, 0x8b, 0x43, 0x95, 0x81,
|
|
'D', 0x40, 0x37, 0xe3, 0x53, 0xf7, 0xce, 0xd9, 0x17,
|
|
']'
|
|
};
|
|
CHECK(json::to_ubjson(j) == v);
|
|
CHECK(json::from_ubjson(v) == j);
|
|
}
|
|
|
|
SECTION("Optimized with count")
|
|
{
|
|
// note the floats have been replaced by doubles
|
|
json const j = {29.97, 31.13, 67.0, 2.113, 23.888};
|
|
std::vector<uint8_t> const v = {'[', '#', 'i', 5,
|
|
'D', 0x40, 0x3d, 0xf8, 0x51, 0xeb, 0x85, 0x1e, 0xb8,
|
|
'D', 0x40, 0x3f, 0x21, 0x47, 0xae, 0x14, 0x7a, 0xe1,
|
|
'D', 0x40, 0x50, 0xc0, 0x00, 0x00, 0x00, 0x00, 0x00,
|
|
'D', 0x40, 0x00, 0xe7, 0x6c, 0x8b, 0x43, 0x95, 0x81,
|
|
'D', 0x40, 0x37, 0xe3, 0x53, 0xf7, 0xce, 0xd9, 0x17
|
|
};
|
|
CHECK(json::to_ubjson(j, true) == v);
|
|
CHECK(json::from_ubjson(v) == j);
|
|
}
|
|
|
|
SECTION("Optimized with type & count")
|
|
{
|
|
// note the floats have been replaced by doubles
|
|
json const j = {29.97, 31.13, 67.0, 2.113, 23.888};
|
|
std::vector<uint8_t> const v = {'[', '$', 'D', '#', 'i', 5,
|
|
0x40, 0x3d, 0xf8, 0x51, 0xeb, 0x85, 0x1e, 0xb8,
|
|
0x40, 0x3f, 0x21, 0x47, 0xae, 0x14, 0x7a, 0xe1,
|
|
0x40, 0x50, 0xc0, 0x00, 0x00, 0x00, 0x00, 0x00,
|
|
0x40, 0x00, 0xe7, 0x6c, 0x8b, 0x43, 0x95, 0x81,
|
|
0x40, 0x37, 0xe3, 0x53, 0xf7, 0xce, 0xd9, 0x17
|
|
};
|
|
CHECK(json::to_ubjson(j, true, true) == v);
|
|
CHECK(json::from_ubjson(v) == j);
|
|
}
|
|
}
|
|
|
|
SECTION("Object Example")
|
|
{
|
|
SECTION("No Optimization")
|
|
{
|
|
// note the floats have been replaced by doubles
|
|
json const j = { {"lat", 29.976}, {"long", 31.131}, {"alt", 67.0} };
|
|
std::vector<uint8_t> const v = {'{',
|
|
'i', 3, 'a', 'l', 't', 'D', 0x40, 0x50, 0xc0, 0x00, 0x00, 0x00, 0x00, 0x00,
|
|
'i', 3, 'l', 'a', 't', 'D', 0x40, 0x3d, 0xf9, 0xdb, 0x22, 0xd0, 0xe5, 0x60,
|
|
'i', 4, 'l', 'o', 'n', 'g', 'D', 0x40, 0x3f, 0x21, 0x89, 0x37, 0x4b, 0xc6, 0xa8,
|
|
'}'
|
|
};
|
|
CHECK(json::to_ubjson(j) == v);
|
|
CHECK(json::from_ubjson(v) == j);
|
|
}
|
|
|
|
SECTION("Optimized with count")
|
|
{
|
|
// note the floats have been replaced by doubles
|
|
json const j = { {"lat", 29.976}, {"long", 31.131}, {"alt", 67.0} };
|
|
std::vector<uint8_t> const v = {'{', '#', 'i', 3,
|
|
'i', 3, 'a', 'l', 't', 'D', 0x40, 0x50, 0xc0, 0x00, 0x00, 0x00, 0x00, 0x00,
|
|
'i', 3, 'l', 'a', 't', 'D', 0x40, 0x3d, 0xf9, 0xdb, 0x22, 0xd0, 0xe5, 0x60,
|
|
'i', 4, 'l', 'o', 'n', 'g', 'D', 0x40, 0x3f, 0x21, 0x89, 0x37, 0x4b, 0xc6, 0xa8
|
|
};
|
|
CHECK(json::to_ubjson(j, true) == v);
|
|
CHECK(json::from_ubjson(v) == j);
|
|
}
|
|
|
|
SECTION("Optimized with type & count")
|
|
{
|
|
// note the floats have been replaced by doubles
|
|
json const j = { {"lat", 29.976}, {"long", 31.131}, {"alt", 67.0} };
|
|
std::vector<uint8_t> const v = {'{', '$', 'D', '#', 'i', 3,
|
|
'i', 3, 'a', 'l', 't', 0x40, 0x50, 0xc0, 0x00, 0x00, 0x00, 0x00, 0x00,
|
|
'i', 3, 'l', 'a', 't', 0x40, 0x3d, 0xf9, 0xdb, 0x22, 0xd0, 0xe5, 0x60,
|
|
'i', 4, 'l', 'o', 'n', 'g', 0x40, 0x3f, 0x21, 0x89, 0x37, 0x4b, 0xc6, 0xa8
|
|
};
|
|
CHECK(json::to_ubjson(j, true, true) == v);
|
|
CHECK(json::from_ubjson(v) == j);
|
|
}
|
|
}
|
|
|
|
SECTION("Special Cases (Null, No-Op and Boolean)")
|
|
{
|
|
SECTION("Array")
|
|
{
|
|
std::vector<uint8_t> const v = {'[', '$', 'N', '#', 'I', 0x02, 0x00};
|
|
CHECK(json::from_ubjson(v) == json::array());
|
|
}
|
|
|
|
SECTION("Object")
|
|
{
|
|
std::vector<uint8_t> const v = {'{', '$', 'Z', '#', 'i', 3, 'i', 4, 'n', 'a', 'm', 'e', 'i', 8, 'p', 'a', 's', 's', 'w', 'o', 'r', 'd', 'i', 5, 'e', 'm', 'a', 'i', 'l'};
|
|
CHECK(json::from_ubjson(v) == json({ {"name", nullptr}, {"password", nullptr}, {"email", nullptr} }));
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
TEST_CASE("Parse UBJSON directly from a file using iterator and sentinel")
|
|
{
|
|
std::string const filename = TEST_DATA_DIRECTORY "/json_testsuite/sample.json.ubjson";
|
|
std::ifstream file(filename, std::ios::binary);
|
|
const std::istreambuf_iterator<char> first(file);
|
|
const json parsed = json::from_ubjson(first, utils::istreambuf_sentinel{});
|
|
CHECK((parsed.is_object() || parsed.is_array()));
|
|
}
|
|
|
|
#if !defined(JSON_NOEXCEPTION)
|
|
TEST_CASE("all UBJSON first bytes")
|
|
{
|
|
// these bytes will fail immediately with exception parse_error.112
|
|
std::set<uint8_t> supported =
|
|
{
|
|
'T', 'F', 'Z', 'U', 'i', 'I', 'l', 'L', 'd', 'D', 'C', 'S', '[', '{', 'N', 'H'
|
|
};
|
|
|
|
for (auto i = 0; i < 256; ++i)
|
|
{
|
|
const auto byte = static_cast<uint8_t>(i);
|
|
CAPTURE(byte)
|
|
|
|
try
|
|
{
|
|
auto res = json::from_ubjson(std::vector<uint8_t>(1, byte));
|
|
}
|
|
catch (const json::parse_error& e)
|
|
{
|
|
// check that parse_error.112 is only thrown if the
|
|
// first byte is not in the supported set
|
|
INFO_WITH_TEMP(e.what());
|
|
if (supported.find(byte) == supported.end())
|
|
{
|
|
CHECK(e.id == 112);
|
|
}
|
|
else
|
|
{
|
|
CHECK(e.id != 112);
|
|
}
|
|
}
|
|
}
|
|
}
|
|
#endif
|
|
|
|
TEST_CASE("UBJSON use_type requires use_size")
|
|
{
|
|
SECTION("non-empty array throws other_error.502")
|
|
{
|
|
const json j = {1, 2, 3};
|
|
CHECK_THROWS_WITH_AS(json::to_ubjson(j, false, true),
|
|
"[json.exception.other_error.502] use_type requires use_size = true",
|
|
json::other_error&);
|
|
}
|
|
|
|
SECTION("non-empty object throws other_error.502")
|
|
{
|
|
const json j = {{"a", 1}, {"b", 2}};
|
|
CHECK_THROWS_WITH_AS(json::to_ubjson(j, false, true),
|
|
"[json.exception.other_error.502] use_type requires use_size = true",
|
|
json::other_error&);
|
|
}
|
|
|
|
SECTION("scalars do not throw with use_type=true, use_count=false")
|
|
{
|
|
CHECK_NOTHROW(json::to_ubjson(42, false, true));
|
|
CHECK_NOTHROW(json::to_ubjson(3.14, false, true));
|
|
CHECK_NOTHROW(json::to_ubjson("hello", false, true));
|
|
CHECK_NOTHROW(json::to_ubjson(true, false, true));
|
|
CHECK_NOTHROW(json::to_ubjson(nullptr, false, true));
|
|
}
|
|
|
|
SECTION("empty containers do not throw with use_type=true, use_count=false")
|
|
{
|
|
CHECK_NOTHROW(json::to_ubjson(json::array(), false, true));
|
|
CHECK_NOTHROW(json::to_ubjson(json::object(), false, true));
|
|
}
|
|
|
|
SECTION("valid combinations on non-empty containers")
|
|
{
|
|
const json j = {1, 2, 3};
|
|
CHECK_NOTHROW(json::to_ubjson(j, false, false));
|
|
CHECK_NOTHROW(json::to_ubjson(j, true, false));
|
|
CHECK_NOTHROW(json::to_ubjson(j, true, true));
|
|
}
|
|
}
|
|
|
|
TEST_CASE("UBJSON round-trip invariants")
|
|
{
|
|
// This checks what the parse_ubjson_fuzzer driver checks (see
|
|
// tests/src/fuzzer-parse_ubjson.cpp), so that a regression shows up in CI
|
|
// rather than as an OSS-Fuzz report: every value from_ubjson() returns
|
|
// (j1) can be serialized with any combination of options, the result can
|
|
// be parsed back (j2), and serializing j2 again with the same options
|
|
// reproduces the exact bytes. Beyond the driver, this also checks that j2
|
|
// equals j1. Values are compared with dump() rather than operator==,
|
|
// because a NaN never compares equal to itself.
|
|
struct options
|
|
{
|
|
bool use_size;
|
|
bool use_type;
|
|
};
|
|
const std::vector<options> all_options =
|
|
{
|
|
{false, false},
|
|
{true, false},
|
|
{true, true},
|
|
};
|
|
|
|
for (const auto& j0 : utils::round_trip_corpus::values())
|
|
{
|
|
// turn the corpus value into a value as from_ubjson() returns it; this
|
|
// has no binary values, as UBJSON writes them as arrays of integers
|
|
for (const auto& initial : all_options)
|
|
{
|
|
const json j1 = json::from_ubjson(json::to_ubjson(j0, initial.use_size, initial.use_type));
|
|
|
|
for (const auto& o : all_options)
|
|
{
|
|
INFO("j1 = " << j1.dump() << ", use_size = " << o.use_size << ", use_type = " << o.use_type);
|
|
|
|
const std::vector<std::uint8_t> vec = json::to_ubjson(j1, o.use_size, o.use_type);
|
|
json j2;
|
|
// anything the library writes must be parsable by the library
|
|
REQUIRE_NOTHROW(j2 = json::from_ubjson(vec));
|
|
CHECK(j2.dump() == j1.dump());
|
|
CHECK(json::to_ubjson(j2, o.use_size, o.use_type) == vec);
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
TEST_CASE("UBJSON roundtrips" * doctest::skip())
|
|
{
|
|
SECTION("input from self-generated UBJSON files")
|
|
{
|
|
for (std::string filename :
|
|
{
|
|
TEST_DATA_DIRECTORY "/json_nlohmann_tests/all_unicode.json",
|
|
TEST_DATA_DIRECTORY "/json.org/1.json",
|
|
TEST_DATA_DIRECTORY "/json.org/2.json",
|
|
TEST_DATA_DIRECTORY "/json.org/3.json",
|
|
TEST_DATA_DIRECTORY "/json.org/4.json",
|
|
TEST_DATA_DIRECTORY "/json.org/5.json",
|
|
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip01.json",
|
|
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip02.json",
|
|
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip03.json",
|
|
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip04.json",
|
|
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip05.json",
|
|
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip06.json",
|
|
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip07.json",
|
|
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip08.json",
|
|
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip09.json",
|
|
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip10.json",
|
|
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip11.json",
|
|
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip12.json",
|
|
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip13.json",
|
|
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip14.json",
|
|
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip15.json",
|
|
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip16.json",
|
|
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip17.json",
|
|
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip18.json",
|
|
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip19.json",
|
|
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip20.json",
|
|
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip21.json",
|
|
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip22.json",
|
|
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip23.json",
|
|
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip24.json",
|
|
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip25.json",
|
|
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip26.json",
|
|
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip27.json",
|
|
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip28.json",
|
|
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip29.json",
|
|
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip30.json",
|
|
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip31.json",
|
|
TEST_DATA_DIRECTORY "/json_roundtrip/roundtrip32.json",
|
|
TEST_DATA_DIRECTORY "/json_testsuite/sample.json",
|
|
TEST_DATA_DIRECTORY "/json_tests/pass1.json",
|
|
TEST_DATA_DIRECTORY "/json_tests/pass2.json",
|
|
TEST_DATA_DIRECTORY "/json_tests/pass3.json"
|
|
})
|
|
{
|
|
CAPTURE(filename)
|
|
|
|
std::ifstream f_json(filename);
|
|
json const j1 = json::parse(f_json);
|
|
auto const packed = utils::read_binary_file(filename + ".ubjson");
|
|
|
|
{
|
|
INFO_WITH_TEMP(filename + ": std::vector<uint8_t>");
|
|
json j2;
|
|
CHECK_NOTHROW(j2 = json::from_ubjson(packed));
|
|
CHECK(j1 == j2);
|
|
}
|
|
|
|
{
|
|
INFO_WITH_TEMP(filename + ": std::ifstream");
|
|
std::ifstream f_ubjson(filename + ".ubjson", std::ios::binary);
|
|
json j2;
|
|
CHECK_NOTHROW(j2 = json::from_ubjson(f_ubjson));
|
|
CHECK(j1 == j2);
|
|
}
|
|
|
|
#ifndef JSON_TEST_DEPRECATED_FUNCTIONS_DELETED
|
|
{
|
|
INFO_WITH_TEMP(filename + ": uint8_t* and size");
|
|
json j2;
|
|
CHECK_NOTHROW(j2 = json::from_ubjson({packed.data(), packed.size()}));
|
|
CHECK(j1 == j2);
|
|
}
|
|
#endif
|
|
|
|
{
|
|
INFO_WITH_TEMP(filename + ": output to output adapters");
|
|
{
|
|
INFO_WITH_TEMP(filename + ": output adapters: std::vector<uint8_t>");
|
|
std::vector<uint8_t> vec;
|
|
json::to_ubjson(j1, vec);
|
|
CHECK(vec == packed);
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
TEST_CASE("UBJSON optimized array of unsigned integers beyond int64")
|
|
{
|
|
// UBJSON has no unsigned 64-bit type, so such values are written as
|
|
// high-precision numbers - also as the type of an optimized container
|
|
const json j = {18446744073709551615ULL, 9223372036854775808ULL};
|
|
const std::vector<std::uint8_t> expected =
|
|
{
|
|
'[', '$', 'H', '#', 'i', 2,
|
|
'i', 20, '1', '8', '4', '4', '6', '7', '4', '4', '0', '7', '3', '7', '0', '9', '5', '5', '1', '6', '1', '5',
|
|
'i', 19, '9', '2', '2', '3', '3', '7', '2', '0', '3', '6', '8', '5', '4', '7', '7', '5', '8', '0', '8'
|
|
};
|
|
CHECK(json::to_ubjson(j, true, true) == expected);
|
|
CHECK(json::from_ubjson(expected) == j);
|
|
}
|
|
|
|
namespace
|
|
{
|
|
// the bytes that follow the marker of an integer: the value in the width of
|
|
// the marker (big endian for UBJSON, little endian for BJData), or, for a
|
|
// high-precision number, the length and the decimal digits
|
|
std::vector<std::uint8_t> integer_payload(const char marker, const json& value, const bool little_endian)
|
|
{
|
|
std::size_t width = 0;
|
|
switch (marker)
|
|
{
|
|
case 'i':
|
|
case 'U':
|
|
width = 1;
|
|
break;
|
|
case 'I':
|
|
case 'u':
|
|
width = 2;
|
|
break;
|
|
case 'l':
|
|
case 'm':
|
|
width = 4;
|
|
break;
|
|
case 'L':
|
|
case 'M':
|
|
width = 8;
|
|
break;
|
|
default:
|
|
{
|
|
const std::string digits = value.dump();
|
|
std::vector<std::uint8_t> result = {'i', static_cast<std::uint8_t>(digits.size())};
|
|
for (const char c : digits)
|
|
{
|
|
result.push_back(static_cast<std::uint8_t>(c));
|
|
}
|
|
return result;
|
|
}
|
|
}
|
|
|
|
const std::uint64_t bits = value.is_number_unsigned()
|
|
? value.get<std::uint64_t>()
|
|
: static_cast<std::uint64_t>(value.get<std::int64_t>());
|
|
std::vector<std::uint8_t> result(width);
|
|
for (std::size_t i = 0; i < width; ++i)
|
|
{
|
|
result[little_endian ? i : width - 1 - i] = static_cast<std::uint8_t>(bits >> (8 * i));
|
|
}
|
|
return result;
|
|
}
|
|
|
|
json i64(const std::int64_t v)
|
|
{
|
|
return v;
|
|
}
|
|
|
|
json u64(const std::uint64_t v)
|
|
{
|
|
return v;
|
|
}
|
|
} // namespace
|
|
|
|
TEST_CASE("UBJSON and BJData integer markers at every range edge")
|
|
{
|
|
// An optimized container announces the marker of its values after `$` and
|
|
// then writes every value without a marker, so the marker the writer
|
|
// announces and the width it writes must match for every value. This
|
|
// checks both for the values around each edge of the integer types, as
|
|
// scalars and as the values of optimized arrays and objects.
|
|
struct integer_case
|
|
{
|
|
json value;
|
|
char ubjson; // expected UBJSON marker
|
|
char bjdata; // expected BJData marker
|
|
};
|
|
|
|
const std::int64_t int64_min = (std::numeric_limits<std::int64_t>::min)();
|
|
const std::int64_t int64_max = (std::numeric_limits<std::int64_t>::max)();
|
|
const std::uint64_t uint64_max = (std::numeric_limits<std::uint64_t>::max)();
|
|
|
|
const std::vector<integer_case> cases =
|
|
{
|
|
// int8
|
|
{i64(-129), 'I', 'I'},
|
|
{i64(-128), 'i', 'i'},
|
|
{i64(-127), 'i', 'i'},
|
|
{i64(-1), 'i', 'i'},
|
|
{i64(0), 'i', 'i'},
|
|
{u64(0), 'i', 'i'},
|
|
{i64(126), 'i', 'i'},
|
|
{i64(127), 'i', 'i'},
|
|
{u64(127), 'i', 'i'},
|
|
{i64(128), 'U', 'U'},
|
|
{u64(128), 'U', 'U'},
|
|
// uint8
|
|
{i64(254), 'U', 'U'},
|
|
{i64(255), 'U', 'U'},
|
|
{u64(255), 'U', 'U'},
|
|
{i64(256), 'I', 'I'},
|
|
{u64(256), 'I', 'I'},
|
|
// int16
|
|
{i64(-32769), 'l', 'l'},
|
|
{i64(-32768), 'I', 'I'},
|
|
{i64(-32767), 'I', 'I'},
|
|
{i64(32766), 'I', 'I'},
|
|
{i64(32767), 'I', 'I'},
|
|
{u64(32767), 'I', 'I'},
|
|
{i64(32768), 'l', 'u'},
|
|
{u64(32768), 'l', 'u'},
|
|
// uint16 (BJData only)
|
|
{i64(65534), 'l', 'u'},
|
|
{i64(65535), 'l', 'u'},
|
|
{u64(65535), 'l', 'u'},
|
|
{i64(65536), 'l', 'l'},
|
|
{u64(65536), 'l', 'l'},
|
|
// int32
|
|
{i64(-2147483649LL), 'L', 'L'},
|
|
{i64(-2147483648LL), 'l', 'l'},
|
|
{i64(-2147483647LL), 'l', 'l'},
|
|
{i64(2147483646LL), 'l', 'l'},
|
|
{i64(2147483647LL), 'l', 'l'},
|
|
{u64(2147483647ULL), 'l', 'l'},
|
|
{i64(2147483648LL), 'L', 'm'},
|
|
{u64(2147483648ULL), 'L', 'm'},
|
|
// uint32 (BJData only)
|
|
{i64(4294967294LL), 'L', 'm'},
|
|
{i64(4294967295LL), 'L', 'm'},
|
|
{u64(4294967295ULL), 'L', 'm'},
|
|
{i64(4294967296LL), 'L', 'L'},
|
|
{u64(4294967296ULL), 'L', 'L'},
|
|
// int64
|
|
{i64(int64_min), 'L', 'L'},
|
|
{i64(int64_min + 1), 'L', 'L'},
|
|
{i64(int64_max - 1), 'L', 'L'},
|
|
{i64(int64_max), 'L', 'L'},
|
|
{u64(static_cast<std::uint64_t>(int64_max)), 'L', 'L'},
|
|
// uint64 (BJData only; UBJSON writes a high-precision number)
|
|
{u64(static_cast<std::uint64_t>(int64_max) + 1), 'H', 'M'},
|
|
{u64(uint64_max - 1), 'H', 'M'},
|
|
{u64(uint64_max), 'H', 'M'},
|
|
};
|
|
|
|
for (const auto& c : cases)
|
|
{
|
|
for (const bool bjdata :
|
|
{
|
|
false, true
|
|
})
|
|
{
|
|
const char marker = bjdata ? c.bjdata : c.ubjson;
|
|
const std::vector<std::uint8_t> payload = integer_payload(marker, c.value, bjdata);
|
|
const auto to_binary = [bjdata](const json & j, const bool use_size, const bool use_type)
|
|
{
|
|
return bjdata ? json::to_bjdata(j, use_size, use_type) : json::to_ubjson(j, use_size, use_type);
|
|
};
|
|
const auto from_binary = [bjdata](const std::vector<std::uint8_t>& v)
|
|
{
|
|
return bjdata ? json::from_bjdata(v) : json::from_ubjson(v);
|
|
};
|
|
INFO("value = " << c.value.dump() << (c.value.is_number_unsigned() ? " (unsigned)" : "") << ", format = " << (bjdata ? "BJData" : "UBJSON"));
|
|
|
|
// scalar
|
|
std::vector<std::uint8_t> expected = {static_cast<std::uint8_t>(marker)};
|
|
expected.insert(expected.end(), payload.begin(), payload.end());
|
|
for (const bool use_size :
|
|
{
|
|
false, true
|
|
})
|
|
{
|
|
CHECK(to_binary(c.value, use_size, false) == expected);
|
|
}
|
|
CHECK(from_binary(expected) == c.value);
|
|
|
|
const json arr = {c.value, c.value, c.value};
|
|
|
|
// array without count or type: every value has its marker
|
|
expected = {'['};
|
|
for (int i = 0; i < 3; ++i)
|
|
{
|
|
expected.push_back(static_cast<std::uint8_t>(marker));
|
|
expected.insert(expected.end(), payload.begin(), payload.end());
|
|
}
|
|
expected.push_back(']');
|
|
CHECK(to_binary(arr, false, false) == expected);
|
|
CHECK(from_binary(expected) == arr);
|
|
|
|
// array with count: every value has its marker
|
|
expected = {'[', '#', 'i', 3};
|
|
for (int i = 0; i < 3; ++i)
|
|
{
|
|
expected.push_back(static_cast<std::uint8_t>(marker));
|
|
expected.insert(expected.end(), payload.begin(), payload.end());
|
|
}
|
|
CHECK(to_binary(arr, true, false) == expected);
|
|
CHECK(from_binary(expected) == arr);
|
|
|
|
// array with type and count: the marker once, then the payloads
|
|
expected = {'[', '$', static_cast<std::uint8_t>(marker), '#', 'i', 3};
|
|
for (int i = 0; i < 3; ++i)
|
|
{
|
|
expected.insert(expected.end(), payload.begin(), payload.end());
|
|
}
|
|
CHECK(to_binary(arr, true, true) == expected);
|
|
CHECK(from_binary(expected) == arr);
|
|
|
|
// object with type and count: the marker once, then key and payload
|
|
const json obj = {{"a", c.value}, {"b", c.value}};
|
|
expected = {'{', '$', static_cast<std::uint8_t>(marker), '#', 'i', 2};
|
|
for (const char key :
|
|
{'a', 'b'
|
|
})
|
|
{
|
|
expected.push_back('i');
|
|
expected.push_back(1);
|
|
expected.push_back(static_cast<std::uint8_t>(key));
|
|
expected.insert(expected.end(), payload.begin(), payload.end());
|
|
}
|
|
CHECK(to_binary(obj, true, true) == expected);
|
|
CHECK(from_binary(expected) == obj);
|
|
}
|
|
}
|
|
}
|
|
|
|
TEST_CASE("UBJSON large strings (chunked reader)")
|
|
{
|
|
// get_ubjson_string() reads through get_string(), which reads in bounded
|
|
// chunks (binary_reader.hpp, chunk_size == 4096); make sure roundtripping
|
|
// is correct for lengths around and beyond that chunk size, for both
|
|
// vector (iterator) and pointer inputs.
|
|
//
|
|
// A binary value is not included here: plain UBJSON (unlike BJData, see
|
|
// the "BJData large strings and binaries" test) has no reader-side binary
|
|
// type, so even the optimized uint8_t-array encoding of a binary value is
|
|
// read back element-by-element as a JSON array of numbers rather than
|
|
// through get_binary() - it never reaches the chunked path this test is
|
|
// about (see the "roundtrip only works to an array of numbers" case
|
|
// above).
|
|
for (const std::size_t len :
|
|
{
|
|
std::size_t{0}, std::size_t{1}, std::size_t{4095}, std::size_t{4096},
|
|
std::size_t{4097}, std::size_t{8192}, std::size_t{100000}
|
|
})
|
|
{
|
|
CAPTURE(len)
|
|
|
|
const json j_string = std::string(len, 'x');
|
|
const std::vector<std::uint8_t> v_string = json::to_ubjson(j_string);
|
|
CHECK(json::from_ubjson(v_string) == j_string);
|
|
// pointer input exercises the std::memcpy fast path
|
|
CHECK(json::from_ubjson(reinterpret_cast<const char*>(v_string.data()),
|
|
reinterpret_cast<const char*>(v_string.data()) + v_string.size()) == j_string);
|
|
|
|
// a truncated payload must still be reported as an error
|
|
if (len > 16)
|
|
{
|
|
std::vector<std::uint8_t> truncated = v_string;
|
|
truncated.resize(truncated.size() - 8);
|
|
json _;
|
|
CHECK_THROWS_AS(_ = json::from_ubjson(truncated), json::parse_error);
|
|
}
|
|
}
|
|
}
|
|
|
|
TEST_CASE("UBJSON optimized container type marker is validated for count 0")
|
|
{
|
|
// the '$' type marker must be a legal marker even if no element follows
|
|
SECTION("invalid marker")
|
|
{
|
|
for (const char marker :
|
|
{'x', 'E', 'B', 'u', 'm', 'M', 'h'
|
|
})
|
|
{
|
|
for (const char open :
|
|
{'[', '{'
|
|
})
|
|
{
|
|
for (const int count :
|
|
{
|
|
0, 1
|
|
})
|
|
{
|
|
CAPTURE(marker)
|
|
CAPTURE(open)
|
|
CAPTURE(count)
|
|
const std::vector<std::uint8_t> v = {static_cast<std::uint8_t>(open), '$', static_cast<std::uint8_t>(marker), '#', 'i', static_cast<std::uint8_t>(count)};
|
|
json _;
|
|
CHECK_THROWS_AS(_ = json::from_ubjson(v), json::parse_error&);
|
|
CHECK(json::from_ubjson(v, true, false).is_discarded());
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
SECTION("error message")
|
|
{
|
|
const std::vector<std::uint8_t> v = {'[', '$', 'x', '#', 'i', 0};
|
|
json _;
|
|
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(v), "[json.exception.parse_error.112] parse error at byte 3: syntax error while parsing UBJSON type: marker 0x78 is not a permitted optimized array type", json::parse_error&);
|
|
}
|
|
|
|
SECTION("valid marker")
|
|
{
|
|
for (const char marker :
|
|
{'Z', 'N', 'T', 'F', 'i', 'U', 'I', 'l', 'L', 'd', 'D', 'H', 'C', 'S', '[', '{'
|
|
})
|
|
{
|
|
CAPTURE(marker)
|
|
const std::vector<std::uint8_t> va = {'[', '$', static_cast<std::uint8_t>(marker), '#', 'i', 0};
|
|
const std::vector<std::uint8_t> vo = {'{', '$', static_cast<std::uint8_t>(marker), '#', 'i', 0};
|
|
CHECK(json::from_ubjson(va) == json::array());
|
|
CHECK(json::from_ubjson(vo) == json::object());
|
|
}
|
|
}
|
|
}
|
|
|
|
TEST_CASE("UBJSON optimized arrays from contiguous and non-contiguous input")
|
|
{
|
|
// contiguous input reads the numbers of an array with a type marker in
|
|
// one loop, other input number by number; both must give the same events,
|
|
// value, or error for the complete document and for every truncation of
|
|
// it, and stop at every event a SAX handler rejects
|
|
json document =
|
|
{
|
|
{"u8", {0, 200, 255}},
|
|
{"i8", {-5, 100, -128}},
|
|
{"i16", {-30000, 1000, 30000}},
|
|
{"i32", {-2000000000, 5, 70000}},
|
|
{"i64", {-9000000000000000000LL, 1, 5000000000LL}},
|
|
{"f64", {1.5, -0.1, 2.5}},
|
|
{"pairs", {{1.25, -3.75}, {1.25, -3.75}, json::array()}},
|
|
{"mixed", {1, "a", 2.5, nullptr}},
|
|
};
|
|
const std::vector<std::uint8_t> encoded = json::to_ubjson(document, true, true);
|
|
CHECK(json::from_ubjson(encoded) == document);
|
|
|
|
for (std::size_t length = 0; length <= encoded.size(); ++length)
|
|
{
|
|
CAPTURE(length)
|
|
const std::vector<std::uint8_t> contiguous(encoded.begin(), encoded.begin() + static_cast<std::ptrdiff_t>(length));
|
|
const std::list<std::uint8_t> bytewise(contiguous.begin(), contiguous.end());
|
|
|
|
CHECK(json::from_ubjson(contiguous, true, false).dump() == json::from_ubjson(bytewise.begin(), bytewise.end(), true, false).dump());
|
|
#if !defined(JSON_NOEXCEPTION)
|
|
std::string contiguous_error;
|
|
std::string bytewise_error;
|
|
try
|
|
{
|
|
json _;
|
|
_ = json::from_ubjson(contiguous);
|
|
}
|
|
catch (const json::exception& e)
|
|
{
|
|
contiguous_error = e.what();
|
|
}
|
|
try
|
|
{
|
|
json _;
|
|
_ = json::from_ubjson(bytewise.begin(), bytewise.end());
|
|
}
|
|
catch (const json::exception& e)
|
|
{
|
|
bytewise_error = e.what();
|
|
}
|
|
CHECK(contiguous_error == bytewise_error);
|
|
#endif
|
|
|
|
SaxEventLogger from_contiguous;
|
|
SaxEventLogger from_bytewise;
|
|
CHECK(json::sax_parse(contiguous, &from_contiguous, json::input_format_t::ubjson)
|
|
== json::sax_parse(bytewise.begin(), bytewise.end(), &from_bytewise, json::input_format_t::ubjson));
|
|
CHECK(from_contiguous.events == from_bytewise.events);
|
|
}
|
|
|
|
for (int events = 0; events <= 80; ++events)
|
|
{
|
|
CAPTURE(events)
|
|
const std::list<std::uint8_t> bytewise(encoded.begin(), encoded.end());
|
|
SaxCountdown from_contiguous(events);
|
|
SaxCountdown from_bytewise(events);
|
|
CHECK(json::sax_parse(encoded, &from_contiguous, json::input_format_t::ubjson)
|
|
== json::sax_parse(bytewise.begin(), bytewise.end(), &from_bytewise, json::input_format_t::ubjson));
|
|
}
|
|
}
|