- ci_test_gcc (Linux x86-64): std::mt19937::result_type is
std::uint_fast32_t (unsigned long there), so returning it as
std::uint32_t failed -Werror=conversion; convert explicitly.
static_cast<std::uint64_t>(18446744073709551615u) and
static_cast<std::int64_t>(-9223372036854775807 - 1) were useless casts
there; use std::numeric_limits instead.
- ci_test_noexceptions: exception_of_call() catches outside a
CHECK_THROWS, so the invalid UTF-8 checks aborted with JSON_NOEXCEPTION;
compile them only with exceptions.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
The node index section on the architecture page says how edits are kept
(the edit buffer, moved elements, links, and new nodes), and the feature
page links to it.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Test edits of an empty document and assignments through a view of a
value that is no longer part of the document; copy the entries of a
block through deref() (one path for links and values); mark the
4 GiB limit and the returns of find_parent() that no document reaches.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
- API pages for set and push_back of basic_json_document and for the
editable aliases; the class overviews name the Editable parameter
- type_error.319 on the exceptions page
- the feature page explains editing, and the examples show when it
helps: a configuration file changed without reformatting it
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Pick the overloads of encode() with a first_true trait instead of
nested conditionals, name the pointer type in the copies of links,
mark the owning pointers of the edit storage, and compare doubles
by their bits in the tests.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
basic_json_document<BasicJsonType, true> (json_editable_document,
ordered_json_editable_document) can be edited:
- set(view, value): replace a value
- set(object, key, value): assign a member, or add it (a null becomes an
object); with duplicate keys, the first is assigned and the others go
- set(array, index, value): assign an element
- set(json_pointer, value): the member, element, or ("-", or the size of
the array) the end of an array a pointer names
- push_back(array, value): append (a null becomes an array)
Values are views (of any document, copied), BasicJsonType values, and
everything BasicJsonType can be constructed from. The source text is never
written, and the parsed index never moves: new values and element
sequences go to storage owned by the document (edit_storage.hpp), so views
stay valid, and a view keeps referring to its value (after an assignment,
it sees the new one). Read-only documents are unchanged; editing one does
not compile.
Errors are those of basic_json where the operation corresponds
(type_error.305/308, out_of_range.401/403/405, parse_error.106/109); a view
of another document is invalid_iterator.202. Strings are checked for UTF-8
when they enter the document, with the type_error.316 that
basic_json::dump() throws for the same string, so that a document only
holds valid UTF-8. Binary values cannot be stored (the new
type_error.319), and edits of 4 GiB or more end with out_of_range.416.
Views of editable and read-only documents compare with each other.
Tests (unit-json_view_edit.cpp): random assignments, member and element
changes, copies within and between documents, and pushes, applied to an
ordered_json_editable_document and to the ordered_json value; after every
edit both must serialize (also indented and with ensure_ascii),
materialize, compare, and read back the same. Further: the errors, strings
that stay valid while the edit arena grows, numbers (NaN, infinities,
extremes; number_format::source), nulls that become containers, the root
replaced, duplicate keys, values of other documents, large objects, and
documents reused with read().
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Preparation for editable documents, without a change in behavior: views and
documents get a template parameter Editable (false by default), and every
walk over the index (iterators, lookups, dump(), materialize()) goes
through detail::view::navigation<Editable>. For read-only documents it is
the plain node array, as before, so they compile without any of the edit
handling. For editable documents it also follows the representation of
edits, which this commit defines:
- node flags `edited` (a string or number token in the edit arena),
`moved` (the elements of an array/object live in a separate sequence),
and `is_new` (no source position), and link nodes (kind_link) that
stand for a value stored elsewhere
- document_data::edit_state: the moved sequences, the storage of new
values, and the edit arena
materialize() now keeps a frame per open container instead of returning
to the end of a closed one, as the serializer does, so that it can
follow moved sequences. Floats whose token lives in the edit arena (also
"nan", "inf", "-inf") are converted out of line. dump() copies only
strings of the source without escaping, and shrink_to_fit() leaves the
node array in place once there are edits, as they link into it.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
With the nested struct object_index, clang 4 (and, by the same bug, the
clang 3.x of ci_test_compilers_clang) rejects the explicitly noexcept
defaulted constructor: "default member initializer for 'indexes' needed
within definition of enclosing class 'document_data' outside of member
functions". Nothing depends on the constructor being noexcept, so let it
take the implicit exception specification.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
GCC -Werror=useless-cast on Linux x86-64 rejects
static_cast<std::size_t>(key_hash(...)): the call returns a
std::uint64_t prvalue, the same type as std::size_t there, while the cast
is needed where std::size_t is 32 bits wide. Store the hash in a variable
and cast that, which GCC does not report.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Lookups in objects are linear, as for ordered_json. Objects with 128
members or more now get a hash table after parsing (open addressing; the
first of duplicate keys is kept, as for the linear search), so that
operator[], at(), find(), contains(), count(), value(), and JSON pointers
take constant time on average in them; the idea of switching to a hash
table for large objects is Boost.JSON's. The parser notes such objects when
it closes them (out of line, so that the parse loop only has a call for
it), and the object node keeps the number of its table.
Looking up each key of an object with 10,000 members: 59.8 ms -> 0.16 ms.
Parsing (json_document::parse, best of 7, separate processes): most files
within 1%; canada +5%, mesh.pretty +3%, citm +3%.
Tests: objects with 127, 128, 129, and 10,000 members (escaped, empty,
and duplicate keys, missing keys, comparisons), nested large objects, and
documents reused with read().
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Hold the UTF-8 lookup tables in std::array, compute the length of a
sequence without nested conditionals, and use std::array in the tests.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Long runs of string bytes are scanned 16 at a time with NEON (AArch64, with
GCC and Clang) and SSE2 (x86-64): both belong to the baseline instruction
sets. A signed compare with 0x20 finds control characters and non-ASCII
bytes at once. Keys keep 16 table checks before the vector loop (their
lengths repeat from record to record, so the branches predict well);
string values have 8, as their lengths vary more.
Non-ASCII text is validated 16 bytes at a time with the "lookup4" check of
simdjson (J. Keiser and D. Lemire, "Validating UTF-8 In Less Than One
Instruction Per Byte", 2021): with NEON, and on x86-64 with SSSE3 if
JSON_VIEW_USE_SSSE3 is defined (SSSE3 is not part of x86-64, and the code
must not depend on the flags of a translation unit). JSON_VIEW_NO_SIMD
selects the portable code. The vector code sits in
detail/view/simd.hpp; the same input is accepted either way.
json_document::parse, best of 7 runs in separate processes (M1 Max):
poet.json (CJK text) -72%, random.json -25%, twitter.json -22%,
gsoc-2018.json -20%, semanticscholar -19%, github_events -11%,
apache_builds -9.5%, canada/citm -5/-6%; lottie +4%, tree-pretty +2.5%.
Tests: every two-byte sequence and three- and four-byte sequences with
continuation bytes at the edges of their ranges, at every offset around
the vector blocks of keys and values, cut short, and long runs of text
with a damaged byte, against json::accept and json::parse. CMake builds
the parser tests again with JSON_VIEW_NO_SIMD, and on x86-64 with
JSON_VIEW_USE_SSSE3 and -mssse3; the macros are documented.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
A workflow runs compare.py with pinned downloads on GitHub-hosted
Ubuntu runners and shows the results as the job summary and as an
artifact: started by hand (workflow_dispatch: x86-64 or AArch64, GCC or
Clang), or when a pull request gets the label "benchmark" (both
architectures, GCC). The label trigger gives numbers before the
workflow is on the default branch, which workflow_dispatch needs.
Shared runners are noisy, so the numbers show where json_view stands on
another architecture; published numbers still need a quiet machine.
compare.py takes the CPU name from lscpu where /proc/cpuinfo has none
(AArch64 Linux), and falls back to the architecture. Checked in Linux
containers (AArch64, Clang 15 and GCC 9, offline with the pinned
archives).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
compare.py --download now checks the SHA-256 of the yyjson 0.13.0,
simdjson 4.6.11, and Boost 1.92.0 archives. It unpacks each archive
once (Boost's directory is boost_1_92_0) and, where Python supports it,
with the 'data' filter.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
tests/benchmarks/json_view/ holds the comparison with other libraries,
which is not built by CMake or run by CI:
- bench_view.cpp: parse, traverse, select, and dump of twitter,
citm_catalog, canada, jeopardy, a single tweet, and a JSON-RPC request,
with json_view, yyjson, simdjson (DOM and On-Demand), Boost.JSON, and
json::parse; all engines must agree on every document before anything
is timed, and run interleaved in every round
- bench_corpus.cpp: parse, traverse, and dump of any list of files
- compare.py: builds both against include/ with the libraries of the
system (or pinned downloads), runs them, and writes the results with
what is needed to reproduce them (date, commit, CPU, OS, compiler,
flags, library versions) to results/<date>-<host>.md and .csv; only the
Python 3 standard library is used
- README.md: how to run it, what is measured, and which features the
engines have, so the numbers can be read correctly
Boost.JSON is optional (JSON_VIEW_BENCH_BOOST).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
The parser records where the integer digits, the fraction digits, and the
exponent of a float token are. For floats and doubles with at most 19
digits, the value is now read from that layout: the digits eight at a time,
without scanning the token, and rounded by the library's conversion core
(detail::decimal_to_float(): Clinger's fast path where both operands are
exact, else the Eisel-Lemire algorithm, which needs no fallback for up to 19
digits). It rounds correctly, so the values are those of parse(); other
tokens and types keep the library's conversion of the whole token.
get<double>(), materialize(), dump(), and comparisons use it. Traversing
canada.json (111,000 floats, every number converted): 0.95 -> 1.29 GB/s.
Tests add tokens around the limits (19 and 20 digits, 2^53, 10^22, and
those of float) to the bit-for-bit comparison with parse().
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Separate the comparison of discarded values from the other types, so
that the conditional chain has no repeated branch bodies, and mark
the deliberate comparisons of views with empty containers in the
tests.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
- API pages for operator== and operator!= of basic_json_view, linked
both ways with the basic_json pages
- the feature page and the class overview list the comparisons
- the examples show when the view helps: detecting a changed document
without building json values
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
basic_json_view gains operator== and operator!= with other views and with
basic_json values. Two views are equal if the values parse() would
produce for them are equal by basic_json's operator==: numbers compare by
value across their types, and objects by their members, with duplicate
keys resolved as parse() resolves them (the last value, at the position of
the first key). Objects are compared in member order if the object type
keeps an order (ordered_json), by key otherwise, as basic_json does.
Discarded views compare as discarded basic_json values do, which follows
JSON_USE_LEGACY_DISCARDED_VALUE_COMPARISON. Nothing is materialized except
single numbers, and the walk is iterative.
Tests compare the results for pairs of 1,200 generated documents (also
written differently: sorted keys, canonical numbers) with those of
basic_json, for json and ordered_json, plus numbers, duplicate keys,
member order, discarded values, and 100,000 levels of nesting.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
The output buffer initializes its members in the initializer list, and the
escaping has no nested conditional operators; the test marks a fixed seed.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
- API pages for dump, number_format, and operator<< of basic_json_view,
linked both ways with the basic_json pages
- the feature page describes document order and number_format::source
- the examples show when the view helps: forwarding part of a message
and writing numbers exactly as they were read
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
basic_json_view::dump(indent, indent_char, ensure_ascii, number_format)
writes the text of a value as ordered_json::parse(text).dump() writes it
for the same arguments: members in document order (all of them, should a
key occur more than once), strings escaped by the same rules and with the
library's scanning kernels, floats with the library's conversion, and
integers copied from the source, where they are canonical except "-0".
With number_format::source, numbers are copied as they appear in the
source ("1.50", "1E2", "-0", all digits of long integers). operator<<
takes the indentation from the stream width, as for basic_json.
The writer (detail/view/serializer.hpp) writes through a raw pointer into
a string sized from the source extent of the value, and walks the index
iteratively, so the nesting depth is limited by memory only.
Tests compare the output of 2,000 generated documents with
ordered_json::dump() for several indentations and ensure_ascii, strings
with every kind of escape, numbers (5,000 random doubles, float as
number_float_t), duplicate keys, 100,000 levels of nesting, and streams.
ViewDump joins the benchmarks.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
GCC -Werror=useless-cast on Linux x86-64 rejected
static_cast<std::uint64_t>((std::numeric_limits<std::size_t>::max)()),
as both are the same type there. Compare without the cast: std::size_t
converts to std::uint64_t implicitly on every platform.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
- ci_test_noexceptions: exception_of() and without_path() exist only
with exceptions (they catch outside a CHECK_THROWS, which aborts with
JSON_NOEXCEPTION); compile the comparisons of the conversion, value(),
and JSON pointer errors only with exceptions as well.
- ci_test_gcc: -Werror=unused-result for static_cast<void>(j.contains(p))
(GCC's warn_unused_result ignores a cast to void); store the result.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
get<T>() of arithmetic types is inlined down to the conversion, so that
its checks of the node kind merge with those of the caller, and reading
an integer needs no call. Traversing every value: citm_catalog -6%,
marine_ik -5%, numbers and twitter -3%, mesh -2.5%, canada -1% (and
more above the float conversion from the digit layout: citm_catalog
-14%, marine_ik -11%).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
get_string() and number_token() return braced lists; the test compares
floats by their bit patterns instead of with memcmp, uses std::any_of, and
marks a fixed seed, a default member initializer (needed by GCC's
-Weffc++), and a string search.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
- API pages for get, get_to, get_string, number_token, and value of
basic_json_view; JSON pointer overloads of operator[], at, and
contains; links both ways with the basic_json pages
- the feature page describes which conversions copy nothing
- the examples show when the view helps: strings without copies, numbers
exactly as written, and paths into a large text
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
basic_json_view gains get<T>(), get_to(), value() with keys and JSON
pointers, and operator[], at(), and contains() with JSON pointers, plus
two functions basic_json has no counterpart for:
- get_string(): the string without a copy (a string_view into the source,
or into the decoded strings for strings with escapes)
- number_token(): the text of a number as it appears in the source
get<T>() converts arithmetic types, strings (also string_view_t),
std::nullptr_t, std::vector, maps with string keys, and views directly;
floats are converted from the digit layout recorded by the parser with the
library's conversion chain, so the values are bit-identical to parse().
Other types, including user types with from_json(), go through
materialize().
The exceptions are those of basic_json, message included. Where const
basic_json has undefined behavior (a missing key or an index out of range
with operator[] and a JSON pointer), the result is a discarded view;
value() returns the default wherever basic_json catches out_of_range, and
contains() never throws. Array indices of JSON pointers follow
json_pointer's rules (parse_error.106/109, out_of_range.404/410).
Tests compare the conversions of 2,000 generated documents, 20,000 float
tokens (double and float, bit for bit), and every JSON pointer of 1,000
documents with basic_json, and the exceptions for malformed pointers.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
detail::json_pointer_access returns the reference tokens of a pointer, so
that code resolving pointers without a basic_json value (such as the
zero-copy view) does not have to parse to_string() again. No change in
behavior.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
- ci_test_noexceptions: the element access tests compare the exceptions
of json_view and basic_json through exception_of(), which catches them
outside a CHECK_THROWS; with JSON_NOEXCEPTION the first one aborted the
test. Compile those comparisons only with exceptions.
- clang 3.6: value-initialize a const json_view, as in the tests of
json-view/10-view-document.
- Format three new documentation examples with the pinned astyle, which
the "check" job runs once it gets past the amalgamation step.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Marks the default initializer of the item's index string (needed by GCC's
-Weffc++) and, in the test, an escaped literal and a comparison of find()
with end(), which is what the test is about.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
- API pages for operator[], at, front, back, find, contains, count,
begin, end, cbegin, cend, items, and type_name of basic_json_view,
linked both ways with the basic_json pages
- the feature page and size() describe document order and duplicate
keys
- the examples show when the view helps: reading a few fields of a large
text, probing optional members, and members in source order
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
basic_json_view gains the read-only access functions of basic_json:
operator[] and at() with keys and indices, front(), back(), find(),
contains(), count(), begin()/end(), items() (with structured bindings from
C++17 on), and type_name(). They throw the exceptions (ids and messages)
that the const functions of basic_json throw; where basic_json has
undefined behavior (operator[] with a missing key or an index out of
range, front()/back() of an empty container), the view returns a
discarded view or throws invalid_iterator.214.
Objects are iterated in document order, and all members are visited. With
duplicate keys, lookups find the first member, so that a lookup can stop
at the first match; parse() keeps the last value. Keys of up to 16 bytes
are compared with two overlapping loads instead of memcmp, and most keys
are rejected by their length alone, from the index.
The iterators and items live in detail/view/iterator.hpp, the lookups in
detail/view/lookup.hpp. Tests compare every element and member of 2,000
generated documents with ordered_json, keys of every length around the
load sizes, the exceptions against const basic_json, and the iterators.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
basic_json::type_name() now calls detail::value_type_name(value_t), so
that code which reports types without a basic_json value at hand, such as
the zero-copy view, uses the same names in its exception messages. No
change in behavior.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
The "check" job (Check amalgamation) runs develop's amalgamate.py and read
all configurations from the develop checkout, where config_json_view.json
does not exist until this stack lands, so it failed with
FileNotFoundError. Read that configuration from the pull request's
checkout; the tool itself stays develop's.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
- ci_test_noexceptions: the helpers that compare the exceptions of
json_document::parse() and json::parse() catch them outside a
CHECK_THROWS, so with JSON_NOEXCEPTION the first parse error aborted
the test. Compile those comparisons only with exceptions, as
unit-class_parser.cpp does.
- ci_test_gcc: -Werror=unused-result for CHECK_THROWS_AS(json_document::
parse(...)); assign the result to a dummy document.
- ci_test_compilers_clang (3.6): `const json_view invalid;` needs a
user-provided default constructor there (CWG 253); value-initialize it.
- ci_test_single_header: json_view.hpp now exists as a single header and
contains the internal view headers, so unit-json_view_builder.cpp
includes it instead of the detail headers in that mode, and the test
is built again with the single header.
- Regenerate single_include/nlohmann/json_view.hpp for the builder change
merged from json-view/08-view-builder.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
A new section describes the 16-byte node: its fields, how integers,
floats, and object members are stored, how views navigate without
pointers, and a worked example. The feature page and the pages of
basic_json_document and node_count link to it where they mention the
16 bytes.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
The 4 GiB limit and the fallback for an input that parse() accepts but
the view rejects (a bug) are excluded from the coverage; shrink_to_fit()
of an empty document is tested.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
- the input dispatch takes byte ranges by const reference and reads the
size once (which also settles a finding of the static analyzer); input
adapters are taken by value
- the classification of inputs keeps its nested conditional operators, a
constant expression of C++11 (NOLINT)
- the test's C arrays, fixed seed, and escaped literals are marked, as in
the other tests
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
benchmarks_view.cpp adds ViewParse, ViewRead (a reused document),
ViewParseIndented, ViewAccept, and ViewMaterialize on the files of
ParseString, so that each row can be read against the json::parse row of
the same file; benchmarks.cpp gains Accept (json::accept) as the
counterpart of ViewAccept. The view benchmarks are built only if the
header directory has json_view.hpp, so that older versions can still be
benchmarked.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
- API pages for basic_json_document and basic_json_view, one per member,
and for the four aliases, each with an example
- features/json_view.md: the problem the view solves, ownership and
lifetime, what matches basic_json::parse() and what differs, and when
to choose json, ordered_json, SAX, or the view
- the examples show why one would use the view, not only how: borrowed
vs. owned input, reading a few fields and materializing one subtree,
reusing a document across many messages
- registered in the mkdocs navigation, llms.txt, the docset, the
exceptions page (out_of_range.416), architecture.md, the integration
page, and the README; the yyjson credit is added to the README and
license.md
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
