Infer 1.3.0 reports NULLPTR_DEREFERENCE because the node pointer can come from navigation<true>::value(), which follows links. A link always has a target in a valid index, so suppress it on that line, as develop does for its own Infer false positives (#5750).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Conflicts in See also lists (docs/mkdocs/docs/api/basic_json/operator_ne.md), where develop (#5638) and this branch both edited: kept develop's entries and added this branch's basic_json_view links.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
The tests added here instantiate basic_json::get() with a type for which Infer 1.3.0 reports STACK_VARIABLE_ADDRESS_ESCAPE on "return ret;", although ret is returned by value. Suppress it on that line, as develop does for its own Infer false positives (#5750).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Conflicts in See also lists (docs/mkdocs/docs/api/basic_json/begin.md,docs/mkdocs/docs/api/basic_json/cbegin.md,docs/mkdocs/docs/api/basic_json/cend.md,docs/mkdocs/docs/api/basic_json/end.md,docs/mkdocs/docs/api/basic_json/type_name.md), where develop (#5638) and this branch both edited: kept develop's entries and added this branch's basic_json_view links.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Conflicts in the See also lists of nine basic_json pages, is_discarded.md, and features/index.md, where develop (#5638) and this branch both added entries: kept both. Ran make amalgamate.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Conflicts:
- number_parse.hpp: kept this branch's float parser, which replaces the
Eisel-Lemire code that develop's side changed (#5750 made its digit
counter unsigned; this parser has no such counter, and it compiles
cleanly with GCC's -Wstrict-overflow=5).
- number_handling.md, template_parameters.md: kept this branch's
description of the conversion and added develop's "Before version
3.13.0" sentence.
Ran make amalgamate.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Review and extend the documentation, and check it in CI
A review of all documentation pages found factual errors, dead links,
missing cross-references, and gaps in examples. This fixes them and adds
checks so the same problems are caught automatically.
Fixes:
- wrong signatures and version histories (operator!= C++20 member,
binary() subtype type, get<PointerType>(), JSON_NO_THREAD_LOCAL, ...)
- stale descriptions (number parsing since #5283, UBJSON table, SAX
example that no longer compiled, tsl::ordered_map advice)
- dead internal and external links; repology.org badges (the domain is
suspended) replaced by badges that query the registries directly
- deprecation notes link the migration guide; the guide itself fixed
Additions:
- "See also" sections, cross-references, 25 runnable examples, 12
Mermaid diagrams, new API pages for json_pointer::operator<=> and
byte_container_with_subtype::operator==/!=
- landing page, guides for untrusted input and performance
- "unreleased" badge after versions newer than the latest release
Checks:
- strict documentation build (broken links/anchors fail it); CI and
the publish workflow fetch the full history the build needs
- weekly external link check, Mermaid syntax check in CI
- check_structure.py: example titles, heading levels, alt texts,
header links, docset index coverage; its unused-example check works
again
- all examples produce the same output on every platform
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Keep the customer links that could not be fixed
A dead link on the customers page is still the evidence of where the
use of the library was documented. Keep the original URLs of the entries
without a working replacement (Marne, Cisco Webex Desk Camera, Philips
Hue, CyberArk) and exclude exactly these URLs from the link check.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Correct the duplicate-key recipe's claim about SAX positions
The SAX interface's key() receives no position either; only parse_error()
does. Also note that the recipe does not report the path to the repeated
key (see discussion #5085).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Say the library is available as a single header and mention json_fwd.hpp
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Correct documentation errors found while hunting for bugs
- patch/patch_inplace: list the JSON pointer errors parse_error.106-109
and out_of_range.402/404, and quote the actual parse_error.105 message.
- unflatten: list parse_error.106/107/108 and out_of_range.404.
- to_bson: list out_of_range.415 (binary subtype above 255) and note
that 412 and 415 are new in 3.13.0.
- to_string: state that string_t must be convertible to std::string, also
in the StringType requirements table.
- JSON Lines: a `while (input >> j)` loop also throws after the last value
for concatenated JSON values; show a loop that works for both.
- BON8: a string gets 0xFF only if nothing follows it in the message; a
string at the end of an array or object is ended by 0xFE.
- custom_string_type.hpp: add operator+=(char), which the "Always
required" list asks for (json_pointer::to_string, flatten, unflatten,
and diff did not compile), and an ADL int_to_string for diff and items.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Cache the release headers with functools.lru_cache
Codacy (Pylint) flagged the mutable default argument that header() used
as its cache. functools.lru_cache keeps the same memoization without it.
The script's output is unchanged.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix CI configuration broken by recent merges and tool updates
- gcc_flags.cmake: drop -Wexperimental-fmv-target, which GCC 16 accepts
only on aarch64; amd64 rejects it, so every GCC job failed while
checking the compiler.
- ci_get_cmake: add VERBATIM so the checksum pipeline is passed to the
shell intact (the unescaped `$'` broke the generated Makefile and
build.ninja, failing ci_cmake_flags and ci_module_cpp20); match the
SHA-256 entry case-insensitively, as CMake 3.5.0 lists the archive as
"Linux-x86_64"; and unpack with --strip-components, as that archive's
top-level directory is spelled "Linux" too.
- ci_single_binaries: compile json.hpp's TU without IWYU's --error, as
the comment above the gate already intends.
- tests: restore -Wno-deprecated-declarations for all non-MSVC compilers
(#5737 kept it for GCC only), as several tests call deprecated
functions on purpose; include thirdparty/fifo_map as SYSTEM.
- .clang-tidy: set misc-use-internal-linkage.AnalyzeTypes to false;
clang-tidy 22.1 extended the check to classes and enums and flagged
100 test helper types.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix library warnings and a JSON_DIAGNOSTICS parent bug
- binary_reader: pass integers to sax->number_integer() through
conditional_static_cast<number_integer_t>, making the existing
narrowing for a narrow number_integer_t explicit (MSVC C4244 and GCC
-Wconversion/-Warith-conversion with the int16_t test from #5694);
mark two Infer DEAD_STORE false positives with @infer-ignore.
- to_json: set the parents of an array built from a C++20 range view
after all elements are in place; a reallocating push_back moved the
earlier elements and left their parent pointers stale, failing the
JSON_DIAGNOSTICS invariant assertion.
- json.hpp: suppress MSVC C4127 for the new is_ordered_map check in
diff(), like the three existing ones; spell out std::formatter::parse's
return and iterator types for clang-tidy 22.1.
- number_parse: make the Eisel-Lemire digit counter unsigned
(GCC -Wstrict-overflow).
- string_utils: take encode_utf8's callable by const reference
(cppcoreguidelines-missing-std-forward) and drop a \u from its doc
comment (-Wdocumentation-unknown-command).
- ordered_map: include <memory> for std::allocator (cpplint).
Ran make amalgamate.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix tests failing in CI on develop
- unit-allocator: skip the #5640 test under MSVC STL iterator debugging,
where containers allocate a debug proxy in noexcept move constructors
and a failing allocation terminates; move a decrement out of an if
condition (bugprone-inc-dec-in-conditions).
- unit-conversions: expect the "(/0)" path with JSON_DIAGNOSTICS;
compare strict enums via get<>() rather than through the noexcept
operator==(ScalarType, json), which bugprone-exception-escape flags.
- unit-alt-string: suppress -Wexit-time-destructors for the strict enum
macro and misc-use-internal-linkage for its enum.
- unit-bjdata: call the static lookup functions through the type and
pass unsigned char (-Wsign-conversion on amd64).
- Mark Infer false positives with @infer-ignore in unit-diagnostics,
unit-pointer_access, unit-udt, and unit-conversions.
- Smaller clang-tidy 22.1 findings in unit-class_parser,
unit-constructor2, unit-custom-base-class, unit-locale-cpp, and
unit-noexcept.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
A lead byte (0xC2..0xF7) inside a string begins either another character
(if a continuation byte follows) or an integer (otherwise). When the input
ended right after the lead byte, the reader took the missing byte as "not
a continuation byte", ended the string before the lead byte, and treated
the lead byte as the start of the next value. With strict=false, a message
cut off there was therefore read as a shorter value: the 11 bytes of
"😀😀é" cut after 9 bytes gave "😀😀", and ["aé"] cut after 3 of its 5
bytes gave ["a"]. With strict=true, the input was rejected with a
misleading message ("expected end of input"), or, for a key, with
parse_error.112 instead of 110.
Either reading of the lead byte leaves the message incomplete: a string at
the end of a message must be terminated by 0xFF, so the lead byte cannot
belong to a following message. Report parse_error.110 (unexpected end of
input) for strings and keys, as the comment on get_bon8_string() already
requires and as the reference decoder (HikoGUI) does.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
json_document::read is a member function, not POSIX read(), and the C
string overload requires null-terminated input like json::parse.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Conflicted only in tests/src/unit-class_lexer.cpp, where develop's #5737
lint fix (CAPTURE(x); -> CAPTURE(x)) collided with this PR's rewrite of
the Eisel-Lemire float tests; kept the PR's new tests and applied the
lint-fixed CAPTURE style. single_include regenerated via make amalgamate.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Re-amalgamate single_include
#5737 changed 13 headers under include/ but merged without the matching
single_include/nlohmann/json.hpp update, so the amalgamated header still
had, among others, the GCC C++20 -Wignored-attributes pragma block and
the clang -Wdocumentation push/pop that #5737 removed, the forwarding
from_json tuple/array helpers it replaced with const references, and
lacked the output_adapter char_traits changes it added.
Regenerated with `make amalgamate` (astyle 3.4.13). The diff is exactly
`git diff b54ed188ee5a89d671 -- include/` (164+/95-); json_fwd.hpp and
json_literals.hpp were already up to date.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Run the amalgamation check on pushes to develop
#5737 was merged 39 seconds after its last push, while its own
check_amalgamation run was still queued (earlier runs had been
cancelled by the concurrency group), so the stale single_include
reached develop without any failing check.
Also run the check on pushes to develop, without cancelling in-progress
develop runs. The "save" job (PR number/author for the comment
workflow) only runs for pull requests, the checkout falls back to
github.sha, and comment_check_amalgamation.yml only comments for
PR-triggered runs, since push runs have no PR and no "pr" artifact.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Share the diagnostic-position setter of the DOM SAX parsers
json_sax_dom_parser and json_sax_dom_callback_parser each had a private
copy of handle_diagnostic_positions_for_json_value(), identical except
for comments. Move the body into one static member function,
detail::diagnostic_positions::set_from_lexer(value, lexer), which both
classes call with their lexer pointer. basic_json befriends the new
struct (only when JSON_DIAGNOSTIC_POSITIONS is enabled), as the position
members are private.
The discarded case is reached through the callback parser, so the
LCOV_EXCL markers that only the dom parser's copy had are gone. The
NOLINT on the unreachable default case loses the stray
"-warnings-as-errors", which is not a check name.
The start-position setup in start_object()/start_array() is left alone,
as #5706 is editing the callback parser's versions.
Behavior, the public API and the ABI are unchanged.
Part of #5712
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Correct the parser comments on recursion and skip_to_state_evaluation
The class documentation called the parser a recursive descent parser,
but sax_parse_internal() is a loop that keeps the open containers on an
explicit stack. The comment at the end of an array and of an object
said the flag is set to false while the code below it sets it to true.
Describe what the code does instead.
Comments only; behavior, the public API and the ABI are unchanged.
Part of #5712
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Update the discard_number_values comments to the current number path
The comments explaining the accept() shortcut in convert_number() and
the member documentation still argued in terms of strtoull()/strtoll()
and errno, which #5283 replaced with convert_integer(), and pointed at
scan_number() instead of convert_number(). They also did not say that
scan_number_bulk_contiguous() converts integers itself, so the shortcut
is only reached for input without bulk access, with
JSON_DIAGNOSTIC_POSITIONS, or when the bulk scanner falls back.
Rewrite both comments to describe the digit-count check in front of
convert_integer(), keeping the 18-digit bound and the json_sax_acceptor
argument. The stale <cstdlib> comment is left for after #5616, which
edits that include block.
Comments only; behavior, the public API and the ABI are unchanged.
Part of #5712
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* List the UTF-8 validators instead of calling the DFA the only one
The documentation of decode() called the Hoehrmann DFA the single
source of truth for UTF-8 validation. It is used only by the serializer
and by is_valid_utf8() (CBOR/MessagePack/BSON/UBJSON/BJData text
strings). The lexer's scan_string() switch, validate_one_utf8() /
valid_utf8_prefix() (bulk string scan, BON8 bulk path and BON8 writer)
and the BON8 byte path in get_bon8_string() check the RFC 3629 ranges
on their own.
Replace the sentence with a list of the four validators, what each is
used for, and a note that they must accept the same sequences. Sharing
code between them was considered and dropped: it would save a few lines
in a validator that is entangled with BON8 pushback, and #5677 is
editing the BON8 byte path.
Comments only; behavior, the public API and the ABI are unchanged.
Part of #5712
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix stale doc comments and include lists in the input headers
input_adapters.hpp included <memory> and <numeric> for the removed
shared_ptr-based adapter design but used neither; it called
(std::min) without including <algorithm>. json_sax.hpp used
std::numeric_limits without including <limits>. Also corrected
comments that no longer matched the code: input_stream_adapter does
not skip the input's BOM (the lexer's skip_bom() does), the
span_input_adapter comment named the no-longer-existing
input_buffer_adapter type, lexer::get_string() does not reset the
token, binary_reader's get_number() doc opened with /* instead of
/*! (so Doxygen skipped it) and omitted BON8 from its endianness
note, and the UBJSON-binary-types note did not mention that BJData
'B' arrays are read as binary.
Left out: the lgtm suppression on lexer.hpp's scan_number() (in
#5616's hunk) and the "-1 if unknown" wording in json_sax.hpp's
start_object/start_array docs (in draft #5267's hunk), per the
verdict's conflict list.
Part of #5712
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Deduplicate the strict-EOF/release_lookahead/error block in parser::parse()
json_sax_dom_callback_parser and json_sax_dom_parser branches of
parser::parse() ran the same ~25 lines after sax_parse_internal():
the strict-mode EOF check (raising parse_error.101 through the SAX
parser), release_lookahead() in non-strict mode, and mapping an
errored SAX parser to a discarded result. The two copies had already
drifted apart in formatting and in the second copy's "see above"
comment.
Add a private parse_dom(DomSax&, strict) member that runs this shared
sequence once and returns whether the SAX parser did not error; both
branches of parse() now only construct their DOM SAX parser, call
parse_dom(), and (for the callback parser) map a discarded top-level
value to null. sax_parse() is left untouched, since it only runs the
EOF check and release_lookahead() when sax_parse_internal() succeeded,
unlike parse(), which runs them unconditionally.
Behavior-preserving: same operations in the same order for both SAX
parser kinds. Verified with unit-class_parser (strict/non-strict,
callback and non-callback), unit-deserialization and
unit-disabled_exceptions (JSON_NOEXCEPTION), plus a clean
make amalgamate / make check-amalgamation diff.
Overlaps #5601, which touches the same lines.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
#5712 item 2
* Share the code point to UTF-8 encoding between the wide-string helpers and the lexer
The 1/2/3/4-byte UTF-8 encoding ladder was written out by hand three
times: in wide_string_input_helper<..., 4>::fill_buffer() for a UTF-32
code point, in the UTF-16 helper for both a BMP code unit and a valid
surrogate pair, and in the lexer's \uXXXX/\uXXXX\uYYYY handling. The
copies had drifted: the UTF-32 helper masked the leading bits of each
byte (& 0x1Fu, & 0x0Fu, & 0x07u) where the others relied on the shift
alone, even though both give the same result for a code point that is
already known to be in range.
Add detail::encode_utf8(cp, out) in string_utils.hpp, a single encoder
that invokes a callable once per output byte, most significant byte
first. Use it in the three valid-code-point branches (UTF-32 code
points up to U+10FFFF, UTF-16 code units outside the surrogate range,
and valid UTF-16 surrogate pairs) and in the lexer's \u handling, where
out forwards to add(). The UTF-16 helper's deliberate pass-through of
malformed surrogate units and the UTF-32 helper's 0xFF sentinel for
code points above U+10FFFF are untouched, since neither reaches the new
helper.
Behavior-preserving: same bytes in the same order for every valid code
point, verified with unit-class_lexer, unit-class_parser,
unit-deserialization, unit-wstring and the non-test-data parts of
unit-unicode1..5 (ASan/UBSan, C++11/17/20), and an escape-heavy parse
microbenchmark that shows no change (about 73 ms either way, median of
3, 1M escape sequences). single_include/ regenerated with make
amalgamate; make check-amalgamation leaves a clean tree.
Overlaps #5704, which rewrites the wide_string_input_helper
specializations touched here.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
#5712 item 6
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Share scalar serialization between dump_internal and dump_value
dump_value()'s cases for string, binary, boolean, number_integer,
number_unsigned, number_float, discarded and null were a byte-for-byte
copy of dump_internal()'s (added together in #5285 for the iterative
fallback path). Any future change to scalar output had to be made in
both places, or the recursive and depth-limited paths would silently
start producing different bytes.
Extract the shared cases into a private dump_scalar() and have both
dump_internal() and dump_value() call it. Output is unchanged: dump(),
dump(4), dump(-1,' ',true) and the replace/ignore error_handler_t
variants are byte-identical over the json_test_data corpus before and
after, and dump() throughput on a scalar-heavy document is unaffected.
Part of #5709
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Drop serializer.hpp's dependency on binary_writer.hpp
The only use of binary_writer in serializer.hpp was
binary_writer<BasicJsonType, char>::to_char_type() to write the
U+FFFD replacement character's three bytes. With CharType=char this
is an identity conversion, so the include of binary_writer.hpp (and
transitively binary_reader.hpp) pulled in a large, unrelated header
for a no-op call.
Write the three bytes directly instead. serializer.hpp compiles
standalone with -Wall -Wextra -Werror, with and without
-funsigned-char, and unit-serialization's error_handler_t::replace
cases (with and without ensure_ascii) still pass. Moving
binary_writer's to_char_type/to_msgpack_length to its private section
is left as an optional follow-up.
Part of #5709
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix stale #include lines in the output headers
output_adapters.hpp included <algorithm> and <iterator> for std::copy
and std::back_inserter, which have not been used there since #3569
(2022). serializer.hpp included <algorithm> for std::reverse (also
unused), <cmath> for labs/isnan/signbit (only std::isfinite is used)
and <utility> for std::move (nothing from <utility> is used there),
while using std::next without including <iterator> at all, relying on
getting it transitively through output_adapters.hpp's own stale
<iterator>.
Drop the unused includes, add <iterator> for std::next, and correct
the remaining include comments. Both headers still compile standalone
with -Wall -Wextra -Werror.
Part of #5709
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Remove JSON_HEDLEY_NON_NULL(2) from write_characters() overrides
output_vector_adapter, output_stream_adapter and output_string_adapter
declared their write_characters(const CharType*, std::size_t) override
JSON_HEDLEY_NON_NULL(2), but binary_writer legitimately calls it with
a null pointer and length 0 for an empty string or binary value; the
type-erased call path only stayed silent under UBSan because the
static callee at those call sites is the unattributed virtual base.
A nonnull attribute on a definition lets GCC and Clang assume the
parameter is non-null inside the function body even when the call is
virtual, so this was latent undefined behavior, not just style.
Drop the attribute from the three overrides and document the
(nullptr, 0) contract on output_adapter_protocol::write_characters.
unit-cbor, unit-msgpack, unit-bson and unit-bon8 (which all exercise
empty binary/string payloads through the stream and vector/string
adapters) pass under -fsanitize=address,undefined,nonnull-attribute.
Part of #5709
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Update stale serializer doc comments to match the current implementation
dump_internal()'s doc block still described the pre-#5285/#5449
implementation: an escape_string() function that does not exist
(the function is dump_escaped), integer conversion "implicitly via
operator<<" (dump_integer actually uses a digit-pair lookup table),
and floating-point conversion via "%g" (IEEE-754 types go through
to_chars, others through snprintf). dump_value()'s comment said
elements are pushed for dump_internal to walk, but it is
dump_iteratively() that walks the stack. dump_escaped(), dump_integer()
and dump_float() each said they write "to output stream @a o", which
has not been true since the writer moved to write_buffer.
Doc-only change; no behavior, API or ABI impact.
Part of #5709
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Merge duplicate byte-to-hex helper and drop stale '| 0' promotions
serializer::hex_bytes() and binary_writer::hex_byte() had identical
bodies. Keep one, detail::hex_byte() in string_utils.hpp, and use it
from both. Also drop the `| 0` at the two serializer call sites
(hex_bytes(byte | 0) and hex_bytes(s.back() | 0)): #3088 (7440786b8)
added it so that `ss << std::hex << (byte | 0)` printed a number
rather than a char with the old stringstream writer; the int result
just narrows back to uint8_t now, so it was a no-op.
Behavior is unchanged: unit-serialization, unit-bon8 (whose
type_error.316 messages exercise this code) and unit-diagnostics
pass, and both headers still compile standalone with -Werror.
Part of #5709
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Trim two stale lint suppressions in serializer.hpp
dump_integer()'s `auto buffer_ptr = number_buffer.begin();` carried
NOLINT entries for cppcoreguidelines-pro-type-vararg and hicpp-vararg,
left over from the snprintf-based implementation (#3088); there is no
variadic call on that line, so keep only the qualified-auto
suppressions it actually needs. remove_sign()'s assert checked
`x < 0 && x < (std::numeric_limits<number_integer_t>::max)()) `with a
NOLINT(misc-redundant-expression) to hide it; the second conjunct is
always true once x < 0, and has been since 6ce2f35ba (2019), so
reduce the assert to `x < 0` and drop the suppression instead of
masking it.
Both are documentation-only changes to assertions/suppressions, not
behavior. The to_chars.hpp `#if 0` branch this item also flagged is
left alone, next to draft PR #5634's pending hunk.
Part of #5709
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Move the Hoehrmann SPDX copyright line to string_utils.hpp
serializer.hpp carried the SPDX-FileCopyrightText line for Björn
Hoehrmann's UTF-8 decoder, but the decoder (decode() and the utf8d
table) has lived in string_utils.hpp since #5185 (d19f7f5dc);
serializer.hpp now only calls decode(). Move the copyright line to
where the code it covers actually is.
Part of #5709
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Stop calling std::localeconv() on every dump()
The serializer constructor snapshotted std::localeconv() into a
locale_chars member on every dump(), even though the only reader is
dump_float(number_float_t, std::false_type)'s snprintf path, taken
only for a number_float_t that is neither IEEE single nor double.
localeconv() is not required to be thread-safe with setlocale(), so
every dump() paid for and raced on a lookup that almost never mattered.
Remove locale_chars and the locale member. Right before the
thousands-separator/decimal-point fixups in the snprintf path, read
std::localeconv() into local thousands_sep/decimal_point variables
(null-checked, first byte only, as before) - the same way
lexer::get_decimal_point() already does since #5597. Output is
unchanged unless the locale changes during a single dump(); in that
case the fixups now match what snprintf just produced, instead of a
value snapshotted before the call.
Overlaps draft PR #5608, which touches the same constructor and
dump_float() lines to move this code into a new
dump_float_snprintf(); this lands the lookup change now as #5709 asks,
and #5608 can do the lookup inside dump_float_snprintf() when it
rebases.
Verification: the full json_test_data corpus (742 files, dump(),
dump(4) and dump(-1,' ',true)) is byte-identical to before the change
under the C locale. Added a test pinning the new per-conversion
lookup: it switches LC_NUMERIC mid-dump() (via a streambuf that
switches on its first write, after the serializer's write buffer has
been flushed once but before a later float is converted) and checks
the decimal point is still normalized using the locale active at
conversion time. On a platform where long double is IEEE-754 double
(e.g. 64-bit Arm), dump_float() takes the locale-independent
to_chars() path and the test is a no-op there; it is meaningful on a
platform where long double is extended precision (most x86 targets).
Part of #5709 item 3
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Diff deeply nested values without recursing per nesting level
diff() descended into both values once per nesting level, and compared
them with operator== on every level on the way, which recurses as well.
Values nested deeply enough - 25,000 levels on an 8 MiB stack - exhausted
the call stack and terminated the process, although parse() accepts
them without complaint. On such a chain the per-level comparisons and
path strings also made diff() quadratic in time and memory.
Both the recursion and operator== only descend as far as the source is
nested. So diff() first checks, recursing at most diff_depth_limit()
(128) levels, whether the source is nested more deeply than that. If not
- all but a vanishing minority of values - the recursive algorithm
diffs it exactly as before, now as diff_recursively(). Otherwise
diff_iteratively() walks the two values on an explicit stack, emitting
the same operations in the same order. It does not compare arrays and
objects with operator== up front (equal ones yield no operations
anyway), keeps the path in one buffer instead of a new string per
level, and hands every subtree that is not nested too deeply back to
diff_recursively(), so equal parts are still skipped quickly.
The check costs one pass over the source. On a 3,000-object document
that is about 30% of diffing two equal values (which is just an
operator== call), about 10% of diffing values that differ in a few
places, and noise when arrays change length. Once operator== no longer
recurses (#5390), the check can go.
Tests check that the patch reproduces the target at every depth up to
300, for json and ordered_json, including reordered members. They also
check the exact operation for a difference deep inside, and diff values
nested 100,000 levels deep.
Fixes#5393 for diff().
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Make diff_frame a member struct that declares its special members
GCC's -Weffc++ (an error in CI) asks a class with pointer members, a
user constructor and a non-trivial destructor to declare its copy
constructor and copy assignment; diff_frame's vector and basic_json
members make its destructor non-trivial. Declare all five as defaulted,
which also satisfies clang-tidy's special-member-functions check. Leave
their exception specifications implicit: GCC 4.8 rejects an explicit
one that differs from the implicit one, as it does for flatten_task in
#5517.
The converting constructor cannot throw, and is now declared noexcept
for GCC's -Wnoexcept, which flags the emplace_back() under C++26
otherwise. The struct also moves from diff_iteratively() into the class,
like dump_frame in the serializer.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Use the shared recursion limit in diff()
diff_depth_limit() is gone in favor of detail::recursion_depth_limit().
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Diff fewer nesting depths so the test does not time out under Valgrind
Checking every depth up to 300 made test-json_patch exceed the 1500 s ctest
timeout in ci_test_valgrind. Check the depths up to 16, those around the
recursion limit of 128, and 300 instead.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Mark the diff frame's value-initialized members for clang-tidy
The braces are kept for GCC's -Weffc++, as in json_sax.hpp.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Bound diff()'s descent with a depth count instead of scanning the source
Now that operator== no longer recurses (#5390), diff() can keep its per-level
equality shortcut all the way down. It diffs recursively for the first
detail::recursion_depth_limit() levels, as merge_patch() does, and hands
anything deeper to diff_iteratively(). The nesting_exceeds() scan, which
cost about 30% on equal documents, is gone, and diff() is on par with
develop again.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Note that the diff frame reference is invalidated by pop_back() too
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Keep diff()'s recursive levels small and its result elided
diff_recursively built every patch operation in place from initializer
lists. Unoptimized builds give each of those temporaries its own stack
slot, so every level of the bounded descent cost kilobytes of stack
(about 6 KB with clang -O0), and the 128 recursive levels overflowed the
1 MB stack of MSVC Debug in the "deeply nested values" test. The
operations and the key comparison of two objects are now built by
separate functions, which diff_iteratively shares, and both diff
functions append to one result instead of returning a patch per level
that the caller copies. With clang -O0, diffing values nested 300 levels
deep now peaks at about 190 KB of stack instead of 880 KB.
Since diff() now owns the only returned value, clang's -Wnrvo no longer
reports the returns of diff_recursively, which alternated between the
local patch and diff_iteratively's result.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Copy the diff frame's members instead of holding a reference to it
The loop in diff_iteratively held a reference to the top frame, which
enter() invalidates when it pushes and the end of the loop invalidates
when it pops. Nothing used it afterwards, but a later change could. As in
the other iterative walks, the members the loop reads are now copied out
as constants and the ones it advances are changed through stack.back().
The frame as a whole is not copied: it holds the common keys and the
"add" operations of an object.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix the ci_cmake_flags wiring so every option is checked
The CMake 3.31.6 flag list referred to itself before it was defined,
so only JSON_BuildTests was checked with that version. The targets for
the CMake running the build ("_2") were created but never added to
ci_cmake_flags, and the three versions shared one build directory.
JSON_StrictNulHandling was not in the list at all.
Use the 3.5.0 list for 3.31.6, add JSON_StrictNulHandling, and create
one ci_cmake_flag_<flag> target per option for the running CMake with
its own build directory. Also use the function parameter in the
COMMENT, refresh the stale version comment, and let ci_clean remove
the downloaded cmake-<version> directories instead of the long-gone
cmake-3.5.0-Darwin64.
Part of #5715
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix ci_test_clang_libcxx_cxx* jobs silently building without warnings
CMake only seeds CMAKE_CXX_FLAGS from the CXXFLAGS environment variable
when the cache entry is unset, so the explicit -DCMAKE_CXX_FLAGS="-stdlib=libc++"
argument made it ignore CXXFLAGS="${CLANG_CXXFLAGS}" entirely. The six
ci_test_standards_clang (..., libcxx) jobs therefore compiled without
-Weverything/-Werror while their libstdc++ siblings did use them.
Pass -stdlib=libc++ through the same CXXFLAGS value instead of a separate
-D argument, and give the target its own build directory
(build_clang_libcxx_cxx${CXX_STANDARD}) so it no longer shares a CMake
cache with the libstdc++ variant. Suppress the resulting
-Wthread-safety-negative finding from libc++'s std::mutex annotations,
which fires on doctest's reporters in this translation unit only.
Part of #5715
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Remove the no-op AppVeyor with_win_header job
The with_win_header matrix entry patched Windows.h into
single_include/nlohmann/json.hpp before building, but JSON_MultipleHeaders
has defaulted to ON since #3532 (2022-06), so CMakeLists.txt points the
tests at include/ and the patched single header is never compiled. The
job has been a no-op VS2015 build since then.
Windows.h coverage already exists through tests/src/unit-windows_h.cpp
(#3631), which runs in every MSVC job. Delete the dead matrix entry and
its before_build steps, and cite unit-windows_h.cpp from the QA page.
Part of #5715
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Remove unused ci_oclint and ci_pvs_studio targets
No workflow invokes ci_oclint, ci_pvs_studio, or their tool discovery.
ci_oclint also had a side effect on every JSON_CI configure: it copied
the single header into src_single/all.cpp and added an add_executable()
for it without EXCLUDE_FROM_ALL, so a plain build compiled a 1.2 MB
translation unit that only that unused target consumed. ci_pvs_studio
duplicates the Makefile's pvs_studio target, which is kept.
Also drop the duplicate --check-level=exhaustive flag passed twice to
the same ci_cppcheck invocation.
Part of #5715
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Stop Dependabot from proposing astyle bumps
astyle is deliberately pinned at 3.4.13 because newer versions reformat
unrelated lines and this version defines the formatting that
check_amalgamation.yml enforces. Without an ignore rule, Dependabot
keeps opening PRs for every new astyle release (most recently #4580,
#4942, #5445, #5448), each of which fails the amalgamation check and
gets closed unmerged.
Part of #5715
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Move Linux arm64 CI from dead Cirrus CI to ubuntu-24.04-arm
Cirrus CI stopped reporting check runs on develop sometime after
d10879bca (2026-05-26); every commit since has only github-actions
check runs, so .cirrus.yml silently lost its only consumer while
README.md, FILES.md and the QA page kept advertising the coverage.
Add a ci_test_arm64 job to ubuntu.yml using the same pinned
actions/checkout and lukka/get-cmake actions as the other jobs, on the
native ubuntu-24.04-arm runner, with a step that confirms uname -m
reports aarch64. Delete .cirrus.yml and its README badge and FILES.md
section, and update the QA page's arm64 row.
Part of #5715
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Make scan-build fail on findings and drop irrelevant checkers (#5715 item 4a)
ci_clang_analyze ran scan-build without --status-bugs, so the job
passed whenever the ninja build succeeded, no matter what the
analyzer found ("No bugs found" in a green run gave no signal either
way). It is also missing --use-analyzer=${CLANG_TOOL}, so scan-build
picks whichever clang happens to be first on PATH inside the
silkeh/clang:dev container instead of the one this file already
selected and versioned.
Add --status-bugs and --use-analyzer=${CLANG_TOOL} to the scan-build
invocation. While here, drop the osx.*, webkit.*, fuchsia.*, and
optin.mpi.* checkers from CLANG_ANALYZER_CHECKS: none of them apply
to this portable C++ library, and leaving them enabled only adds
noise once the job can actually fail on a finding.
The job is currently clean (0 bugs), so this alone does not surface
any new finding; it only makes the existing "no bugs found" result
authoritative. This is 4a of 3 independent steps in #5715 item 4;
4b (Infer) and 4c (IWYU) still need their existing findings triaged
before --fail-on-issue/-Xiwyu --error can be added, and are handled
in separate commits.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Deduplicate the amalgamation/format check's file set and add BUILD.bazel (#5715 item 5)
The amalgamation/format check existed three times with three different
file sets: the Makefile's pretty/check-amalgamation, the pull_request-only
check_amalgamation.yml workflow, and the ci_test_amalgamation CMake target
that also runs on direct pushes to develop/master/release/*. The CMake
target's glob was a strict subset of the workflow's (missing the
docs/mkdocs/docs/examples/*.hpp headers, tests/abi/, tests/cmake_*/project/,
tests/cuda_example/, tests/fmt_formatter/, and tests/module_cpp20/), and it
never checked BUILD.bazel at all, so a misformatted file in any of those
paths, or a stale BUILD.bazel, could reach develop through a direct push
even though the PR-only workflow would have caught it.
Make ci_test_amalgamation glob the same roots (docs/mkdocs/docs/examples,
include, tests) and extensions (*.hpp, *.cpp, *.cu) as check_amalgamation.yml,
excluding tests/thirdparty/ and tests/abi/include/nlohmann/ the same way, and
regenerate and diff BUILD.bazel next to json.hpp/json_fwd.hpp. Also add
docs/mkdocs/docs/examples/*.hpp to the Makefile's pretty/pretty_format
targets, which were missing the four custom_*_type.hpp example headers, and
drop the stale "called by Travis" comment on check-amalgamation (Travis is
gone; nothing currently calls that Makefile target from CI).
Leaves the workflow itself untouched: it deliberately runs amalgamate.py
from a fresh develop checkout so a PR cannot change the tool that checks it.
Overlaps #5610 and #5621, which each add a new amalgamated header and touch
the same INDENT_FILES/ci_test_amalgamation/check_amalgamation.yml hunks.
Verified with `make check-amalgamation` on this branch: clean, no diff.
#5715 item 5.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Download prebuilt CMake binaries on Linux x86_64 instead of building from source (#5715 item 6)
ci_get_cmake() downloaded the source tarball of CMake 3.5.0, 3.31.6, and
4.0.0 and compiled each one completely (including CMake's own test
helpers) with -DCMAKE_POLICY_VERSION_MINIMUM=3.5 as a workaround for
building old CMake with a newer one. On CI this made the
ci_cmake_options (ci_cmake_flags) job take about 11 minutes, most of it
spent building CMake itself, even though Kitware has published
ready-to-run Linux x86_64 archives for all three of these releases
since 3.20 (lowercase platform name).
On Linux x86_64, download and unpack the prebuilt
cmake-<version>-linux-x86_64.tar.gz archive instead and point the
existing ${var} output at its bin/cmake, skipping the configure/build
steps and CMAKE_POLICY_VERSION_MINIMUM entirely. Keep the previous
source build as a fallback for any other platform (macOS, Linux
aarch64), since Kitware does not publish binaries for every
CMake/platform combination this project might build on.
Verify the downloaded archive against Kitware's own published checksum
before unpacking it: download cmake-<version>-SHA-256.txt alongside the
archive and run `sha256sum -c` on the matching line. A CI job that wgets
and untars a binary from a release page with no integrity check is a
supply-chain gap; Kitware has published this file for every release
since 3.20, so checking it costs one extra download and one grep.
As a separate, mechanical change: the ci_cmake_options job's container
only needed to stay on ubuntu:focal for the source build's
libssl-dev dependency and its own aging toolchain; now that the
Linux/x86_64 path never compiles CMake, drop libssl-dev from its apt
install line and move the job to ubuntu:24.04 (Ubuntu 20.04 left
standard support in May 2025). ci_clean already removes the
cmake-3.5.0/cmake-3.31.6/cmake-4.0.0 directories from the #5715 item 3
fix, and the prebuilt path reuses those same directory names, so no
further cleanup changes are needed.
Overlaps #5598, which edits the same ci_cmake_options matrix line in
ubuntu.yml; a rebase may be needed once that lands.
Verified locally: `cmake -S . -B build -DJSON_CI=On` configures cleanly
on macOS/arm64 (source-build fallback branch) and on Linux/x86_64 in an
ubuntu:24.04 Docker container (47 `ci_cmake_flag_*` targets generated,
one built and run successfully); `.github/workflows/ubuntu.yml` still
parses as valid YAML; downloaded the real v3.31.6 Linux x86_64 archive
and SHA-256 file from Kitware and confirmed the `grep | sha256sum -c`
pipeline both accepts the genuine file and is anchored to the exact
filename (not a prefix match).
CI must confirm: the prebuilt-binary path actually runs on the
ubuntu-latest/ubuntu:24.04 x86_64 runner, all `ci_cmake_options`
entries still pass with the new container's GCC, and the job's
runtime drops from roughly 11 minutes.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Make Infer fail on findings, with a type-level baseline for the ~174 pre-existing ones (#5715 item 4b)
ci_infer ran `infer run` without --fail-on-issue, so the job passed
regardless of what Pulse found; the last recorded run (35829411620,
commit 1054b2097) logged "Found 174 issues" and still went green.
report.txt was also never uploaded, so the full finding list was only
ever visible in the truncated 5-issue console excerpt.
Add a repository-root .inferconfig (auto-discovered by Infer; passing
--project-root on the `infer run` invocation makes sure it is found
even though the analysis runs from build/build_infer) that sets
fail-on-issue and disables the six PULSE issue types that made up all
174 findings in that run: PULSE_UNNECESSARY_COPY_ASSIGNMENT (129),
PULSE_UNNECESSARY_COPY (22), PULSE_UNNECESSARY_COPY_INTERMEDIATE (15),
PULSE_RESOURCE_LEAK (5), PULSE_CONST_REFABLE (2), and
PULSE_UNNECESSARY_COPY_OPTIONAL (1).
This is a deliberate, narrower fix than "triage and fix everything in
this PR": the visible sample is entirely doctest-macro copies in test
code (for example tests/src/unit-algorithms.cpp:141 and
tests/src/unit-bjdata.cpp:3706), but 169 of the 174 findings were never
uploaded anywhere and this PR cannot respectably claim to have fixed
issues it never saw, including the resource-leak and const-refable
ones that are the most likely to be genuine bugs. Disabling by issue
type is a coarser baseline than a per-finding one (Infer has no
built-in per-finding baseline short of the two-run `infer reportdiff`
workflow, which this repository does not have the CI infrastructure
for), but it has the same effect today: the job goes from always green
to green-only-when-clean-of-everything-else, so CI now fails the
moment a *new* issue type appears, and report.txt is uploaded as a
workflow artifact on every run (including failures) so the six
disabled types can be triaged and re-enabled incrementally in follow-up
PRs.
#5715 item 4b. 4a (scan-build) and 4c (IWYU) are handled in separate
commits.
Verified: .inferconfig parses as JSON, ubuntu.yml still parses as
YAML. Infer itself is not available in this environment (v1.3.0 tar.xz
requires a Linux x86_64 runner), so CI must confirm that `infer run
--project-root ... -- make` picks up .inferconfig, that fail-on-issue
takes effect, and that the six disabled types actually suppress the
existing findings without also hiding an unrelated new one.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix IWYU findings for json.hpp/json_fwd.hpp/ordered_map.hpp and make CI fail on new ones (#5715 item 4c)
ci_single_binaries ran IWYU via CMake's CXX_INCLUDE_WHAT_YOU_USE launcher
property, which only printed "Warning: include-what-you-use reported
diagnostics" without failing the build: CMake's own __run_co_compile
wrapper does not propagate the launched tool's exit code, so even
`-Xiwyu --error` could never fail `cmake --build` this way. Verified
this empirically by injecting a deliberately-unused #include and
confirming the build still exited 0.
Fix the findings from the last recorded run (issue #5715 item 4, log
35829411620):
- ordered_map.hpp: add <new> (placement new) and
nlohmann/detail/abi_macros.hpp; drop <memory> (std::allocator is
still visible transitively via <vector>, confirmed by full local and
containerized test suite runs).
- json_fwd.hpp: drop <memory> (same reasoning). Keep every forward
declaration IWYU wanted removed (adl_serializer, basic_json,
json_pointer, ordered_map): this file's only job is to forward-declare
them for downstream users, so "nothing in this TU uses them" is
expected, not a real finding. Mark each with `// IWYU pragma: keep`.
- json.hpp: add <cmath>, <cstdint>, <set>, <type_traits>,
<unordered_map>, and the detail/abi_macros.hpp, detail/input/json_sax.hpp,
detail/meta/detected.hpp, thirdparty/hedley/hedley.hpp includes IWYU
says it needs. Do NOT remove adl_serializer.hpp,
detail/conversions/from_json.hpp, detail/conversions/to_json.hpp,
detail/macro_unscope.hpp, or ordered_map.hpp as IWYU suggests: nothing
else in include/nlohmann includes adl_serializer.hpp or
ordered_map.hpp, so basic_json<>'s own default template arguments
(JSONSerializer = adl_serializer, and ordered_json = basic_json<ordered_map>)
would lose their complete type; detail/macro_unscope.hpp is what
undoes the JSON_* macros detail/macro_scope.hpp defines earlier in
this same file, and removing it leaks those macros into every
translation unit that includes <nlohmann/json.hpp>. Verified by
actually removing them in a scratch test: the header still "compiles"
stand-alone but ordered_json and every macro-using translation unit
break. Marked each `// IWYU pragma: keep`.
Enforce it with `iwyu_tool` (ships with IWYU, e.g. as /usr/bin/iwyu_tool
on Debian/Ubuntu) instead of relying on the launcher property: it reads
compile_commands.json (now exported project-wide under JSON_CI) and
does return a real exit code for its own analysis, independent of
CMake's wrapper. ci_single_binaries now runs it over every
src_single/*.cpp with `-Xiwyu --error`, so a *new* finding fails CI.
json.hpp itself is excluded from that hard gate: even after every fix
above, IWYU's suggestion for one remaining symbol (a container
`swap, operator!=` used somewhere via a templated comparator) is not
deterministic — repeated, otherwise-identical containerized runs
reported <set>, then <unordered_map>, then <map> as "the" header to
add/remove for the exact same source. Gating a whole CI job on a
nondeterministic suggestion would make ci_single_binaries flaky rather
than informative, so json.hpp keeps the existing informational warning
(still shown during its normal compile) without failing the build on
it. Every other one of the ~50 single-header checks is included in the
hard gate.
#5715 item 4c. 4a (scan-build) and 4b (Infer) are separate commits.
Verified: full local ctest suite (129/129) and the ci_single_binaries
target itself both green in a containerized silkeh/clang:dev run
(matching the actual CI job) after this fix; a deliberately-reintroduced
unused #include in ordered_map.hpp was confirmed to fail
`cmake --build ... --target ci_single_binaries` (exit 2) with this
change, and to pass without it, on the same container/IWYU version CI
uses. `make check-amalgamation` is clean. Compiled with Clang and GCC
at -std=c++11/14/17/20 locally with no new warnings.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix stale and missing comments in binary_writer
The doc block of write_number() ended up above the byte_swap() helpers
added in #5286, about 80 lines from the function. It was also a plain
comment that Doxygen skips, said "write a number to output input", and
left BON8 out of the big-endian formats. Move it back onto
write_number() as a /*! block and fix the text.
write_bson() documented "@pre j.type() == value_t::object", but it
throws type_error.317 for every other type, and to_bson() relies on
that. Document the exception instead.
Explain why the CBOR binary subtype is always written with a 0xD8..0xDB
head and never in the one-byte tag form: binary_reader with
cbor_tag_handler_t::store only keeps those heads as a subtype, so
switching to write_cbor_head() would break round trips for subtypes
0..23.
Also fix the grammar of the to_char_type comment. Comments only; no
change in behavior, API or ABI.
Part of #5710
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Merge the duplicated UBJSON/BJData integer marker ladders
write_number_with_ubjson_prefix() (unsigned and signed overloads) and
ubjson_prefix() (number_integer and number_unsigned cases) each picked
the UBJSON/BJData integer marker (i, U, I, u, l, m, L, M, H) with their
own independent if/else ladder, and the values beyond 64 bits were
handled by a second, tag-dispatched pair of ladders. An optimized
container announces the marker of its first element via ubjson_prefix()
and then writes every element through write_number_with_ubjson_prefix(),
so the two had to be kept in lockstep by hand across four call sites.
Replace all of that with one ubjson_integer_prefix() built on
value_in_range_of<T>, and one write_ubjson_integer_payload() that
writes the value (or, for 'H', the decimal digits) for a given marker.
write_number_with_ubjson_prefix() and ubjson_prefix() keep their
signatures and now just call these two helpers.
Behavior, the public API and the ABI are unchanged. Verified with a
new regression test covering scalars and $-optimized arrays/objects at
every int8/uint8/int16/uint16/int32/uint32/int64/uint64 boundary for
to_ubjson/to_bjdata (both use_size/use_type settings), and by diffing
to_ubjson/to_bjdata output before and after over the json_test_data
corpus (bit-identical).
Part of #5710
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Remove dead get_char parameters in binary_reader
The non-recursive rewrite of the binary readers (#5505, #5506, #5507)
left parse_cbor_internal()'s and parse_ubjson_internal()'s get_char
parameters dead: parse_cbor_internal() has one caller and it always
passes true, and parse_ubjson_internal() has one caller and it always
uses the true default. Both parameters, and the @param docs describing
the "reuse the last character" mode they used to select, no longer
correspond to anything.
Drop both parameters, initialise fetch/prefix unconditionally, and
update the two call sites in sax_parse(). parse_cbor_value()'s and
get_ubjson_string()'s own get_char parameters are unrelated and are
left alone; both still have a false caller.
Also delete a stray `@return whether a valid MessagePack value was
passed to the SAX parser` doxygen block that sits directly above
parse_msgpack_value()'s real doc comment, a leftover of the same
rewrite.
Behavior, the public API and the ABI are unchanged; these are private
members of detail::binary_reader. Verified by compiling with
-Wunused-parameter and running unit-cbor, unit-ubjson, unit-bjdata and
unit-msgpack (offline, against the stubbed test_data.hpp).
Part of #5711
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Share the IEEE half-precision decoder between CBOR and BJData
binary_reader had two ~45-line copies of the IEEE 754 half-precision
decoder: CBOR's case 0xF9 and BJData's case 'h'. Once formatting is
normalised, the two blocks were identical except for the byte order
used to assemble the 16-bit half (CBOR is big endian, BJData is little
endian). Any future change to half-float decoding had to be made and
kept in sync in both places.
Add one get_half_float(format, little_endian) helper that does the two
get()/unexpect_eof() reads, assembles the half in the requested byte
order, decodes it per RFC 8949 Appendix D, and calls sax->number_float.
Both cases now just call it with their byte order; the BJData case
keeps its bjdata-only guard.
Behavior, the public API and the ABI are unchanged. Verified with a
scratch probe comparing the old and new decoders bit-for-bit (NaN by
isnan()) over all 65536 wire byte pairs, in both formats, and by
running unit-cbor and unit-bjdata (offline, against the stubbed
test_data.hpp).
Part of #5711
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Deduplicate the MessagePack unsigned-integer writer ladder
The number_integer (non-negative branch) and number_unsigned cases in
write_msgpack() each held their own copy of the fixint/uint8/16/32/64
ladder, kept in lockstep only by a comment ("we used the code from the
value_t::number_unsigned case here"). Both copies mixed union members:
the signed copy compared number_unsigned but wrote number_integer, and
vice versa.
Extract write_msgpack_unsigned(std::uint64_t), mirroring how
write_cbor_head() already avoids the same duplication for CBOR, and
call it from both cases. Each case now reads only its own active
union member. Output bytes are unchanged for the default 64-bit
number types.
#5710 item 3
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Unify float marker selection and fix the long double compile error
Four formats picked between a float32 and float64 marker through four
different helper styles: dummy-argument overloads for CBOR and
MessagePack, an std::is_same template for BON8, and a runtime if-chain
on input_format_t for write_compact_float(). With number_float_t set
to long double, to_cbor, to_msgpack and to_ubjson failed inside the
library with "call to 'get_cbor_float_prefix' is ambiguous", while
to_bson kept working because write_bson_double() takes a plain double.
Change write_compact_float() to take the two marker bytes directly
(each of its three callers already knows them at compile time) instead
of an input_format_t it only forwarded, and delete the now-unused
get_cbor_float_prefix(), get_msgpack_float_prefix(),
get_bon8_float_prefix() and get_compact_float_prefix() helpers. Turn
the two get_ubjson_float_prefix() overloads into one template. Both
write_compact_float() and get_ubjson_float_prefix() now report an
unsupported number_float_t with a static_assert naming the requirement,
rather than an ambiguous-overload error; the assert lives in the
function body, not the class scope, so to_bson with long double is
unaffected.
Verified with a probe basic_json<..., long double>: to_bson still
compiles and round-trips, while to_cbor/to_msgpack/to_ubjson now fail
to compile with the new static_assert message.
This changes the text of an existing compile error for users with an
unsupported number_float_t (documented as a public-API-visible change
in #5710).
#5710 item 1
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Deduplicate the BJData ndarray writer's dtype dispatch and drop <map>
write_bjdata_ndarray() built a 12-entry std::map<string_t, CharType> on
every call just to translate the _ArrayType_ name to a dtype marker
(the only reason binary_writer.hpp included <map>), then mapped dtype
to C++ type twice more: once as a switch for the range-check pass and
once as a separate if/else chain for the write pass, with nothing
checking that the two agreed. The caller also ran three at() lookups,
and the callee called value.at(key) about ten more times for the same
three members.
Replace the map with bjdata_ndarray_type_marker(), a plain string
comparison chain (a C++11 constexpr function cannot contain a switch,
so this mirrors binary_reader's own static table style). Replace the
switch/if-chain pair with one write_bjdata_ndarray_elements() that
switches on dtype once and calls a per-type helper -
write_bjdata_ndarray_element<T>() for the eight integer dtypes and
write_bjdata_ndarray_float_element() for 'd' - with a dry_run flag
selecting the range check or the actual write, so the two passes can
no longer disagree on the type. _ArrayType_, _ArraySize_ and
_ArrayData_ are now looked up once into references, and the four
header marker bytes ('[', '$', '#') are written through to_char_type()
like the rest of the UBJSON/BJData writer.
The 'd' (single-precision) rule is left exactly as before, since #5707
is expected to change it separately.
Verified byte-for-byte identical output before/after for every dtype
(including the Draft 2/Draft 3 'byte' fallback and the use_count/
use_type combinations) via a standalone probe, plus round-tripping
through from_bjdata().
Overlaps #5707, which is expected to touch the 'd' dtype case, and
#5518, which is expected to move the write_bjdata_ndarray() call site.
#5710 item 4
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Assert that write_bson_document() consumes every calc_bson_sizes() entry
calc_bson_sizes() and write_bson_document() are a hand-synchronized
pair of passes over the same object/array tree, introduced by #5553:
the size pass appends to nested_sizes in visiting order, and the write
pass consumes the table by position with nested_sizes[next_size++].
Nothing checked that the write pass consumed the whole table. If a
future change touched only one of the two passes - for example to skip
or reject an entry - every later size prefix in the document would be
silently wrong.
Add JSON_ASSERT(next_size == nested_sizes.size()) where
write_bson_document() returns, so such a future drift between the two
passes is caught immediately (JSON_ASSERT expands to nothing in
release builds using assert(), and the fuzzers/tests already build
with it enabled). The two passes agree today, so this changes nothing
observable; it only guards against the risk described in #5710 item 5.
Extracting a shared stepper for the two passes (the second half of the
proposed change) is left for a follow-up: it only saves ~30 lines and
the issue asks for it only if the result reads clearly, which needs
more room to get right than a mechanical cleanup pass allows.
#5710 item 5
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Make the BJData lookup tables static functions instead of members
binary_reader held bjd_optimized_type_markers and bjd_types_map as
non-static const members (12 string_t objects for the type-name table),
built and destroyed on every from_cbor/from_msgpack/from_bson/
from_ubjson/from_bon8/from_bjdata call even though only from_bjdata
ever reads them. They also needed the #define/decltype/#undef
workaround from #3637 and two NOLINTNEXTLINE suppressions, and
binary_writer already carries the same two lists in another form
(is_bjdata_excluded_type_marker() and a local std::map in
write_bjdata_ndarray(), the latter removed by the item-4 commit), so
the excluded-marker lists could drift apart.
Replace bjd_optimized_type_markers with static constexpr
is_bjd_excluded_optimized_type(char_int_type), using the same ||-chain
as binary_writer's is_bjdata_excluded_type_marker(). Replace
bjd_types_map with a non-constexpr static bjd_type_name(char_int_type)
switch returning nullptr for an unknown marker (a C++11 constexpr
function cannot contain a switch). Delete both
JSON_BINARY_READER_MAKE_* macros, the bjd_type pair alias, the
NOLINTNEXTLINE suppressions, detail::make_array() (no longer used
anywhere), and the now-unused <algorithm> and <array> includes.
Update the two call sites (the ND-array excluded-type check and the
_ArrayType_ lookup) accordingly, and replace unit-bjdata.cpp's
"LUT arrays are sorted" section, which only checked the two tables'
internal ordering, with a check of all 12 type names and all 8
excluded markers against both new functions.
#5711 item 1
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Read CBOR's 1/2/4/8-byte argument through one helper
parse_cbor_internal() hand-wrote the same "read a 1/2/4/8-byte
big-endian unsigned integer" ladder four times over:
- twice for tag numbers 0xD8-0xDB, once in the tag_handler::ignore
branch and once, nearly identically, in the ::store branch (~90
lines to read one integer);
- twice more for container lengths, once for array heads 0x98-0x9B and
once for map heads 0xB8-0xBB, where the 1/2-byte forms called
enter_array()/enter_object() directly and the 4/8-byte forms
additionally went through get_cbor_container_size().
Add get_cbor_argument(std::uint64_t&), reading the width selected by
current & 0x1F via the same get_number() calls as before (so EOF is
reported exactly as before), and route all four sites through it:
- 0xD8-0xDB now read the argument once per branch instead of switching
on `current` a second time; behavior split cleanly from embedded tags
0xC0-0xD7 (tag value in the head, no argument to read), which is now
its own case block that no longer has to fall into the ::store
switch's "default" case to reach the same tag_pending = true; return
true; outcome.
- 0x98-0x9B and 0xB8-0xBB collapse into one case block each, always
going through get_cbor_container_size() (harmless for 1/2-byte
lengths, which already always fit).
Verified byte-for-byte identical behavior before/after with a
standalone probe covering embedded and multi-byte tags under all three
tag_handler_t settings, a tag over a byte string (subtype path),
truncated tag/length arguments of every width, and array/map lengths
of every width, including the out_of_range.408 "excessive size" case:
same exceptions, same messages, same chars_read, same successful
results.
Left the string/byte-string length ladders in get_cbor_string()/
get_cbor_binary() untouched, as noted in #5711 item 2, since #5325 is
expected to touch them separately.
Overlaps #5601 (adds a branch right above the embedded-tag case) and
#5607 (touches the integer cases 0x18-0x1B, which share this ladder's
shape in separate hunks).
#5711 item 2
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Add leave_container() to match enter_container()
Every container is opened through enter_container(), whose docs
promise that a check placed there runs before every start event. The
close side had no equivalent: the same
"container_stack.pop_back(); dispatch to end_object() or end_array()"
sequence was written out separately in BSON, CBOR, MessagePack,
UBJSON/BJData and BON8, each copying the pattern of keeping an
is_object flag around the pop_back() that would otherwise invalidate
a reference to it. A check needed on close would have had to be added
in five places, and a sixth copy could go unnoticed.
Add leave_container() next to enter_container(), doing the same
pop-then-dispatch, and replace the five sites with it. Each site keeps
its own surrounding logic (BSON's check_bson_document_size() call
before popping, MessagePack's is_object copy used again below,
UBJSON/BJData's remaining-container handling after popping, BON8's
top used again below); only the repeated pop/dispatch line pair is
now shared.
Verified all six binary-format unit suites and unit-regression2's
deep-nesting tests (dependent count/reuse count and the bjdata ndarray
depth cases) still pass, compiled with -Wall -Wextra and ASan/UBSan.
Overlaps #5601, which is expected to add a sixth close site in its own
skip loop; that site can route through leave_container() too once it
lands.
#5711 item 4
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Stop passing the input format to sax_parse() when the reader already has it
binary_reader's constructor stores the format in the input_format
member, and sax_parse(format, sax_, strict, tag_handler) took the same
value again purely to dispatch on it. Every in-tree caller passed the
same value both times (all 16 from_cbor/from_msgpack/from_ubjson/
from_bjdata/from_bon8/from_bson call sites in json.hpp, and the three
public basic_json::sax_parse() overloads), so nothing was broken
today, but a caller of the detail class directly (only reachable via
JSON_PRIVATE_UNLESS_TESTED, as unit-bjdata.cpp already does) could
pass a mismatched pair - say bjdata to the constructor and ubjson to
sax_parse - and dispatch on one format while applying the other
format's rules; the default-constructed input_format_t::json reader
would additionally hit JSON_ASSERT(false) in exception_message() on
its first error.
Add sax_parse(json_sax_t*, bool, cbor_tag_handler_t) forwarding to the
existing overload with the stored input_format, and switch every
caller to it: the 16 from_*() sites (keeping their
`// cppcheck-suppress[accessMoved]` comments) and the three
basic_json::sax_parse() overloads, all of which already had the format
available from their own `format` parameter. The four-argument overload
is kept for anyone still calling it, now with
JSON_ASSERT(format == input_format) so a mismatch fails immediately
in a debug build (assert-enabled binaries, including the fuzzers and
test suite) instead of misbehaving; verified with a probe that
constructs a reader for one format and calls the explicit overload
with another, which aborts on that assertion as expected.
Removing or asserting against the constructor's input_format_t::json
default, which would affect direct detail users, is left as a separate
decision per #5711 item 5.
Overlaps #5601, which is expected to add an AllowRecovery template
parameter to sax_parse() and touch these same call sites in json.hpp.
#5711 item 5
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Deduplicate UBJSON/BJData signed-count handling, drop dead ndarray checks
get_ubjson_size_value()'s 'i'/'I'/'l'/'L' cases each read a differently
sized signed integer and then repeated the same "reject negative with
error 113" check; only 'L' additionally checked value_in_range_of for
the out_of_range.408 case. Any change to that error path had to be
made four times.
Add get_ubjson_signed_count<SignedType>(std::size_t&), doing the read,
the negative check and the range check once, and route all four
markers through it. The range check is a no-op for 'i'/'I'/'l' (their
values always fit std::size_t) and only live for 'L' on a 32-bit
std::size_t target, matching today's behavior exactly.
In the ndarray dimension-product loop, the preceding loop already
returns early on any zero dimension and result starts at 1, so `i > 0`
in the pre-multiplication overflow check was always true, and
`result == 0` in the post-multiplication check could not be reached
either: two positive factors whose product does not overflow (as the
pre-check already guarantees) cannot be zero. Drop the dead `i > 0 &&`
and narrow the post-check to `result == npos`, the one case the
pre-check cannot rule out (an exact, non-overflowing match with the
sentinel reserved for unknown-size containers), with a comment
explaining why.
Verified byte-for-byte identical behavior before/after with a
standalone probe covering negative counts for every marker, a matching
positive count, and ndarray inputs, plus the full unit-ubjson and
unit-bjdata suites (same assertion counts as before this change).
Overlaps #5601 (rewrites the four parse_error calls and the overflow
checks touched here) and #5607/#5707 (touch neighboring lines in the
same functions).
#5711 item 6
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Drop redundant format parameter and dummy float argument (review)
binary_reader::sax_parse(format, ...) only ever had to equal the format
given to the constructor, which it asserted. With every caller already
on the format-less overload, remove the four-argument overload and
dispatch on the stored input_format directly. binary_reader is a
detail class, so this is not a public API change.
get_ubjson_float_prefix() took a value only to deduce its type; make
the type an explicit template argument instead.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
