mirror of
https://github.com/nlohmann/json.git
synced 2026-09-30 22:15:19 +00:00
* Add BON8 support Add to_bon8/from_bon8 and input_format_t::bon8 for BON8, a binary format that uses the byte values that cannot begin a UTF-8 character as type markers, so strings need no length prefix. It is the most compact of the supported binary formats on the benchmark files. The reader is non-recursive like the other binary readers. A string ends at the first byte that cannot continue it, so the reader hands the one or two bytes it reads past a string back to the value that follows. The writer produces the canonical representation of the specification, except for NFC normalization; its output is identical to that of the reference implementation (HikoGUI) on all files of the test data. The round-trip tests need the .bon8 files of json_test_data 3.2.0. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Address review comments - Reuse detail::validate_one_utf8 to check strings in to_bon8; the error now names the first byte of the invalid sequence. - Document that to_bon8 leaves bytes in the output adapter on an exception, and that string_open is only an output of write_bon8_marker. - Explain why the pushback buffer of the BON8 reader cannot overflow. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Select the BON8 float prefix by type get_bon8_float_prefix only depends on the type of its argument, so make the type a template parameter instead of passing an unused value. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Rename a test variable that Flawfinder mistakes for read() Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix the BON8 CI failures - compare the float in write_bon8_float with number_float_t constants, so GCC does not warn about a float-to-double conversion - mark check_bon8_utf8's context as used when exceptions are disabled - choose the compact float prefix in a helper rather than with nested conditional operators (clang-tidy) - use auto for the cast in the BON8 integer reader (clang-tidy) - write the int32 minimum test values as long long literals (MSVC C4146) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Amalgamate Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Read BON8 strings in bulk from contiguous input - copy the valid UTF-8 of a string in one step when the input is contiguous (twitter.json is read in 1.68 instead of 2.52 ms, jeopardy.json in 196 instead of 297 ms, close to CBOR and MessagePack) - share the new valid_utf8_prefix() with the writer's UTF-8 check, which now skips ASCII 8 bytes at a time - let the fuzzer check that contiguous and stream input give the same value or error, and test both paths in the unit tests - clarify that a second 0xFF after a string is an empty string Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Link the BON8 functions from the other binary format pages Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Name the bulk scan flag after the input, not BON8 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Read BSON keys in bulk from contiguous input BSON keys (and array indices) are C-style strings, which were read byte by byte. For contiguous input they are now read up to their \x00-byte in one step, using the same bulk_scan flag as BON8 strings: twitter.json is read in 1.46 instead of 2.01 ms, citm_catalog.json in 2.93 instead of 3.33 ms, jeopardy.json in 182 instead of 207 ms. canada.json, whose keys are almost all one-digit array indices, takes 2 % longer. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix the BON8 CI failures of the bulk-read tests - skip the contiguous-versus-stream tests of BON8 strings and BSON keys when exceptions are disabled: they catch the parse errors of invalid input, and without exceptions the library aborts instead - use static_cast for the int64 test value (google-readability-casting) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Move the explicit basic_json instantiation into its own test file Linking test-regression3_cpp20 with clang and MinGW failed with "relocation truncated to fit: IMAGE_REL_AMD64_REL32 against `.rdata'", as test-regression2 did before #5511. The explicit instantiation of basic_json<> for #4825 compiles every member function, including the BON8 reader and writer, into that object, and it was already close to the limit (2,226,104 bytes on develop, 2,234,960 with BON8; clang -O1, C++20). Give the instantiation a file of its own: unit-regression3 is now 1,594,736 bytes and unit-explicit_instantiation 1,095,064. The new file mentions JSON_HAS_CPP_17 and JSON_HAS_CPP_20 so it keeps being built for the C++17 standard the regression was about. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Convert the bytes of the BON8 test strings explicitly The str() helper constructed a std::string from a byte range, which converts each unsigned char implicitly; -fsanitize=integer reports that for bytes of 0x80 and above (ci_test_clang_sanitizer). Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me>
325 lines
9.3 KiB
C++
325 lines
9.3 KiB
C++
// __ _____ _____ _____
|
|
// __| | __| | | | JSON for Modern C++ (supporting code)
|
|
// | | |__ | | | | | | version 3.12.0
|
|
// |_____|_____|_____|_|___| https://github.com/nlohmann/json
|
|
//
|
|
// SPDX-FileCopyrightText: 2013-2026 Niels Lohmann <https://nlohmann.me>
|
|
// SPDX-License-Identifier: MIT
|
|
|
|
#include "doctest_compatibility.h"
|
|
|
|
#include <nlohmann/json.hpp>
|
|
|
|
#include <cstdint>
|
|
#include <map>
|
|
#include <string>
|
|
#include <type_traits>
|
|
#include <utility>
|
|
#include <vector>
|
|
|
|
|
|
namespace
|
|
{
|
|
|
|
// An ObjectType that does *not* define a key_compare member type, which is
|
|
// what every hash map looks like to the library.
|
|
//
|
|
// A hash map is deliberately not used here: object_t is probed for
|
|
// key_compare inside the definition of basic_json, that is, while basic_json
|
|
// is still an incomplete type, and whether a hash map can be instantiated
|
|
// with an incomplete mapped type depends on the standard library (libstdc++ 9
|
|
// needs the size of the mapped type for its node type and rejects it). So the
|
|
// object type wraps a std::map instead of inheriting from it: an earlier
|
|
// version derived from std::map and shadowed the inherited key_compare type
|
|
// with a same-named member function, relying on ordinary member hiding to
|
|
// make key_compare unreachable as a type. MSVC 2017 (AppVeyor, /std:c++17)
|
|
// does not honor that hiding for a typename-qualified lookup performed from
|
|
// outside the class and still resolves key_compare to the base's comparator
|
|
// type, so the library's probe incorrectly found one. Composition sidesteps
|
|
// the question entirely: with no base class, there is no key_compare to find
|
|
// under any lookup rule.
|
|
template<class Key, class T, class Compare, class Allocator>
|
|
class no_key_compare_map
|
|
{
|
|
using map_t = std::map<Key, T, Compare, Allocator>;
|
|
map_t data;
|
|
|
|
public:
|
|
using key_type = typename map_t::key_type;
|
|
using mapped_type = typename map_t::mapped_type;
|
|
using value_type = typename map_t::value_type;
|
|
using size_type = typename map_t::size_type;
|
|
using allocator_type = typename map_t::allocator_type;
|
|
using iterator = typename map_t::iterator;
|
|
using const_iterator = typename map_t::const_iterator;
|
|
|
|
// -Weffc++ asks for the member to be initialized in the member
|
|
// initialization list, which a defaulted constructor does not do; the
|
|
// exception specification a defaulted one would have carried has to be
|
|
// written out as well, or -Wnoexcept objects where the standard library
|
|
// takes noexcept(construct(...))
|
|
no_key_compare_map() noexcept(std::is_nothrow_default_constructible<map_t>::value) : data() {}
|
|
|
|
// converting between two basic_json types builds the object from a range
|
|
template<class InputIt>
|
|
no_key_compare_map(InputIt first, InputIt last) : data(first, last) {}
|
|
|
|
iterator begin() noexcept
|
|
{
|
|
return data.begin();
|
|
}
|
|
iterator end() noexcept
|
|
{
|
|
return data.end();
|
|
}
|
|
const_iterator begin() const noexcept
|
|
{
|
|
return data.begin();
|
|
}
|
|
const_iterator end() const noexcept
|
|
{
|
|
return data.end();
|
|
}
|
|
const_iterator cbegin() const noexcept
|
|
{
|
|
return data.cbegin();
|
|
}
|
|
const_iterator cend() const noexcept
|
|
{
|
|
return data.cend();
|
|
}
|
|
|
|
bool empty() const noexcept
|
|
{
|
|
return data.empty();
|
|
}
|
|
size_type size() const noexcept
|
|
{
|
|
return data.size();
|
|
}
|
|
size_type max_size() const noexcept
|
|
{
|
|
return data.max_size();
|
|
}
|
|
void clear() noexcept
|
|
{
|
|
data.clear();
|
|
}
|
|
|
|
iterator find(const key_type& key)
|
|
{
|
|
return data.find(key);
|
|
}
|
|
const_iterator find(const key_type& key) const
|
|
{
|
|
return data.find(key);
|
|
}
|
|
size_type count(const key_type& key) const
|
|
{
|
|
return data.count(key);
|
|
}
|
|
|
|
std::pair<iterator, bool> emplace(const key_type& key, const mapped_type& value)
|
|
{
|
|
return data.emplace(key, value);
|
|
}
|
|
|
|
std::pair<iterator, bool> insert(const value_type& value)
|
|
{
|
|
return data.insert(value);
|
|
}
|
|
|
|
template<class InputIt>
|
|
void insert(InputIt first, InputIt last)
|
|
{
|
|
data.insert(first, last);
|
|
}
|
|
|
|
mapped_type& operator[](const key_type& key)
|
|
{
|
|
return data[key];
|
|
}
|
|
|
|
mapped_type& at(const key_type& key)
|
|
{
|
|
return data.at(key);
|
|
}
|
|
const mapped_type& at(const key_type& key) const
|
|
{
|
|
return data.at(key);
|
|
}
|
|
|
|
iterator erase(iterator pos)
|
|
{
|
|
return data.erase(pos);
|
|
}
|
|
iterator erase(iterator first, iterator last)
|
|
{
|
|
return data.erase(first, last);
|
|
}
|
|
size_type erase(const key_type& key)
|
|
{
|
|
return data.erase(key);
|
|
}
|
|
|
|
void swap(no_key_compare_map& other) noexcept(noexcept(data.swap(other.data)))
|
|
{
|
|
data.swap(other.data);
|
|
}
|
|
|
|
friend bool operator==(const no_key_compare_map& lhs, const no_key_compare_map& rhs)
|
|
{
|
|
return lhs.data == rhs.data;
|
|
}
|
|
friend bool operator<(const no_key_compare_map& lhs, const no_key_compare_map& rhs)
|
|
{
|
|
return lhs.data < rhs.data;
|
|
}
|
|
};
|
|
|
|
using no_key_compare_json = nlohmann::basic_json<no_key_compare_map>;
|
|
|
|
// An ObjectType whose erase(iterator) returns void rather than the following
|
|
// iterator, as for instance Abseil's hash maps do
|
|
template<class Key, class T, class Compare, class Allocator>
|
|
struct void_erase_map : std::map<Key, T, Compare, Allocator>
|
|
{
|
|
using base_t = std::map<Key, T, Compare, Allocator>;
|
|
using iterator = typename base_t::iterator;
|
|
using base_t::erase;
|
|
|
|
void erase(iterator pos)
|
|
{
|
|
base_t::erase(pos);
|
|
}
|
|
};
|
|
|
|
using void_erase_json = nlohmann::basic_json<void_erase_map>;
|
|
|
|
} // namespace
|
|
|
|
TEST_CASE("object type whose erase() returns void")
|
|
{
|
|
SECTION("erasing every element through the returned iterator")
|
|
{
|
|
void_erase_json j;
|
|
for (int i = 0; i < 8; ++i)
|
|
{
|
|
j["k" + std::to_string(i)] = i;
|
|
}
|
|
|
|
std::size_t erased = 0;
|
|
for (auto it = j.begin(); it != j.end(); ++erased)
|
|
{
|
|
it = j.erase(it);
|
|
}
|
|
CHECK(erased == 8);
|
|
CHECK(j.empty());
|
|
}
|
|
|
|
SECTION("erasing in the middle returns the following element")
|
|
{
|
|
void_erase_json j;
|
|
for (int i = 0; i < 4; ++i)
|
|
{
|
|
j["k" + std::to_string(i)] = i;
|
|
}
|
|
|
|
auto it = j.begin();
|
|
++it;
|
|
const auto after = j.erase(it);
|
|
CHECK(j.size() == 3);
|
|
CHECK(after.key() == "k2");
|
|
CHECK(after.value() == 2);
|
|
CHECK(!j.contains("k1"));
|
|
}
|
|
|
|
SECTION("the other erase overloads are unaffected")
|
|
{
|
|
void_erase_json j;
|
|
j["a"] = 1;
|
|
j["b"] = 2;
|
|
j["c"] = 3;
|
|
|
|
CHECK(j.erase("a") == 1);
|
|
CHECK(j.erase("nope") == 0);
|
|
j.erase(j.begin(), j.end());
|
|
CHECK(j.empty());
|
|
}
|
|
}
|
|
|
|
TEST_CASE("object type without key_compare")
|
|
{
|
|
SECTION("object_comparator_t falls back to default_object_comparator_t")
|
|
{
|
|
CHECK(std::is_same < no_key_compare_json::object_comparator_t,
|
|
no_key_compare_json::default_object_comparator_t >::value);
|
|
}
|
|
|
|
SECTION("object types defining key_compare are unaffected")
|
|
{
|
|
CHECK(std::is_same<nlohmann::json::object_comparator_t,
|
|
nlohmann::json::object_t::key_compare>::value);
|
|
CHECK(std::is_same<nlohmann::ordered_json::object_comparator_t,
|
|
nlohmann::ordered_json::object_t::key_compare>::value);
|
|
}
|
|
|
|
SECTION("creating and accessing values")
|
|
{
|
|
no_key_compare_json j;
|
|
j["one"] = 1;
|
|
j["two"] = "zwei";
|
|
j["three"]["nested"] = true;
|
|
|
|
CHECK(j.size() == 3);
|
|
CHECK(j.at("one") == 1);
|
|
CHECK(j["two"] == "zwei");
|
|
CHECK(j["three"]["nested"] == true);
|
|
CHECK(j.contains("one"));
|
|
CHECK(!j.contains("four"));
|
|
CHECK(j.find("one") != j.end());
|
|
CHECK(j.count("one") == 1);
|
|
CHECK(j.erase("one") == 1);
|
|
CHECK(j.size() == 2);
|
|
}
|
|
|
|
SECTION("serialization and deserialization")
|
|
{
|
|
const auto j = no_key_compare_json::parse(R"({"a":[1,2,3],"b":{"c":null}})");
|
|
CHECK(j["a"].size() == 3);
|
|
CHECK(j["a"][2] == 3);
|
|
CHECK(j["b"]["c"].is_null());
|
|
CHECK(no_key_compare_json::parse(j.dump()) == j);
|
|
}
|
|
|
|
SECTION("binary formats")
|
|
{
|
|
const auto j = no_key_compare_json::parse(R"({"a":[1,2,3],"b":"x"})");
|
|
CHECK(no_key_compare_json::from_cbor(no_key_compare_json::to_cbor(j)) == j);
|
|
CHECK(no_key_compare_json::from_msgpack(no_key_compare_json::to_msgpack(j)) == j);
|
|
CHECK(no_key_compare_json::from_bon8(no_key_compare_json::to_bon8(j)) == j);
|
|
}
|
|
|
|
SECTION("flatten and unflatten")
|
|
{
|
|
// "o" has a key that looks like an array index, so unflatten() must
|
|
// not turn it into an array
|
|
const auto j = no_key_compare_json::parse(
|
|
R"({"c":[1,2,3],"d":{"e":"s"},"n":[[0,1],[2]],"o":{"2":"x"}})");
|
|
CHECK(j.flatten().unflatten() == j);
|
|
}
|
|
|
|
SECTION("conversion to and from nlohmann::json")
|
|
{
|
|
const auto j = no_key_compare_json::parse(R"({"a":1,"b":[true,null]})");
|
|
const nlohmann::json converted(j);
|
|
|
|
CHECK(converted.is_object());
|
|
CHECK(converted["a"] == 1);
|
|
CHECK(converted["b"][0] == true);
|
|
CHECK(converted["b"][1].is_null());
|
|
CHECK(no_key_compare_json(converted) == j);
|
|
}
|
|
}
|
|
|