mirror of
https://github.com/nlohmann/json.git
synced 2026-10-02 15:05:16 +00:00
dddb2d6e43a384d93138c5b70c4266998da610a9
38 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Niels Lohmann
|
e5a89d671f |
Fix lint debt: enum-macro NOLINTs, doctest as SYSTEM, no-op analyzer (#5737)
* Drop stale LCOV_EXCL_LINE from the json_pointer out_of_range.410 throw The comment said the size_type overflow check in array_index() is only triggered on special platforms like 32-bit, and the throw was excluded from coverage. On 64-bit platforms the check is true for SIZE_MAX itself, and unit-json_pointer.cpp has asserted that case four times since #5395, so the line is executed in the coverage job. Reword the comment and remove the exclusion marker so the coverage report notices if the tests stop reaching it. Part of #5725 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Name all three C-array check aliases in the enum-macro NOLINTs NLOHMANN_JSON_SERIALIZE_ENUM(_STRICT) suppressed the c-array warning under modernize-avoid-c-arrays only, but clang-tidy emits the same diagnostic under the aliases cppcoreguidelines-avoid-c-arrays and hicpp-avoid-c-arrays too. Any user running those checks got a false positive at every macro expansion, and our own tests needed a local NOLINT at each call site to work around it. Name all three aliases in the four macro comments instead, and drop the now-redundant c-array names from the five test call-site NOLINTs. Comment-only change; behavior, the public API, and the ABI do not change. Ran make amalgamate. Part of #5725 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Include doctest as a SYSTEM directory instead of disabling warnings for all tests test_main added -Wno-deprecated and -Wno-float-equal as PUBLIC compile options for every non-MSVC compiler, so they were applied to every translation unit, library headers included, and silenced the CI warnings meant to check the library's own -Wfloat-equal pragmas. The only code that actually needed the suppression was the vendored doctest.h, which was included as a normal (non-SYSTEM) directory. Include thirdparty/doctest as SYSTEM for test_main, matching what tests/abi/CMakeLists.txt already does, and drop the two suppressions from both targets. Verified locally that unit-comparison, unit-conversions and unit-constructor1 compile clean with -Werror -Weverything and doctest as -isystem, and that CMake still configures with JSON_BuildTests=ON. Part of #5725 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Remove the no-op ci_clang_analyze target ci_clang_analyze configured the build with the real compiler and only then wrapped ninja with scan-build. scan-build intercepts compiles by overriding CC/CXX, but build.ninja already had the compiler path baked in from the configure step, so every run bypassed the analyzer: CI logs show "No bugs found" after a normal build, never an analysis. The job also used Debian's frozen clang-tools-14 rather than the image's own clang, and CLANG_ANALYZER_CHECKS still named three valist.* checkers that current clang merged into security.VAList. ci_clang_tidy already runs every clang-analyzer-* check (via .clang-tidy's "Checks: '*'") with warnings as errors, so nothing is lost by removing the dead job. Delete ci_clang_analyze, CLANG_ANALYZER_CHECKS and the SCAN_BUILD_TOOL lookup from cmake/ci.cmake, drop it from the ubuntu.yml ci_static_analysis_clang matrix, and drop the now-unused clang-tools apt package (iwyu stays for ci_single_binaries). Reword quality_assurance.md and assurance_case.md, which described the dead job as a working control, to say the Clang Static Analyzer checks run through clang-tidy. Verified that `cmake -DJSON_CI=ON` still configures cleanly and that ci_clang_analyze no longer appears in the generated build or in any CMake/workflow file. Part of #5725 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Re-enable portability-template-virtual-member-function; remove redundant forwards .clang-tidy disabled three checks "to get the CI going" (#4489, 2024-11-13): portability-template-virtual-member-function, bugprone-use-after-move and its alias hicpp-invalid-access-moved. portability-template-virtual-member-function only flagged output_stream_adapter::write_character/write_characters; annotate both with NOLINT and re-enable the check. bugprone-use-after-move flagged several double forwards that have no effect at runtime: - from_json.hpp calls std::forward<BasicJsonType>(j).at(Idx) inside pack expansions; at() has no ref-qualified overloads and always returns an lvalue reference, so the forward is a no-op. Replace with plain j.at(Idx) in all four places. - the move constructor forwards the whole object to its base class and then reads other's members. That is item 9 of #5724 (together with its cppcheck suppressions) and is left to that change. - input_adapters.hpp forwards the container twice on purpose, so the begin/end iterator types match adapter_type; annotate with NOLINT and a comment instead of changing behavior. The check still flags the move constructor (see above) and two sites in at(KeyType&&) (both overloads, json.hpp, in the throw's string_t(std::forward<KeyType>(key)) after find(std::forward<KeyType>(key))). Open PR #5689 rewrites that hunk, so bugprone-use-after-move (and hicpp-invalid-access-moved) stay disabled for now, with a comment explaining why; re-enable them once #5689 and the #5724 move-constructor change have landed. Also resolve the portability-avoid-pragma-once TODO: single_include never has #pragma once (amalgamate.py strips it) and every supported compiler accepts it in include/, so keep it disabled with an explanatory comment instead of a TODO. Fix the stale "json.hpp, around line 1265" comment in unit-class_parser.cpp, which now points at the move constructor's actual line. Behavior, the public API and the ABI do not change. Verified with clang-tidy 22.1.8 that portability-template-virtual-member-function now reports nothing, that bugprone-use-after-move/ hicpp-invalid-access-moved report only the known at(KeyType&&) and move-constructor sites, and that unit-custom-base-class, unit-constructor1, unit-conversions, unit-element_access2, unit-class_parser and unit-diagnostic-positions (JSON_DIAGNOSTIC_POSITIONS=1) compile under ASan/UBSan and pass with the same assertion counts as before. Ran make amalgamate. Part of #5725 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix stale and malformed NOLINT comments json_sax.hpp named "-warnings-as-errors" in the NOLINT list on the two JSON_ASSERT(false) lines; that is the suffix clang-tidy appends to a diagnostic tag under WarningsAsErrors, not a check name, and every other JSON_ASSERT(false) omits it. unit-capacity.cpp carried 30 "// NOLINT(misc-const-correctness)" comments on "json j = ...;" declarations that are all used with non-const members afterwards, so the check has nothing to report there. unit-constructor2.cpp used a blanket "// NOLINT: access after move is OK here" on a use-after-move that hides every check on the line; naming bugprone-use-after-move and hicpp-invalid-access-moved keeps the intent once those checks are re-enabled (#5724). Signed-off-by: Niels Lohmann <mail@nlohmann.me> #5725 item 10 * Remove stale .clang-tidy entries -google-runtime-references disabled a check that neither clang-tidy 22.1.8 nor 23.1.2 lists under --list-checks -checks='*'; it was removed upstream. The commented-out HeaderFilterRegex line has been unused since the active HeaderFilterRegex was introduced in #2561 (2021). Signed-off-by: Niels Lohmann <mail@nlohmann.me> #5725 item 11 * Remove the GCC C++20 -Wignored-attributes pragma in json.hpp The pragma (added in #5164) claimed to work around the C++ modules redefinition errors of #5103, but #5103 is about hard errors (e.g. "redefinition of std::__is_constant_evaluated()", conflicting std::integral_constant) that ignoring a warning cannot suppress; they are traced to GCC PR 124430 and reproduce with <map> or <string> instead of json.hpp too. A GCC 16.2 -std=gnu++20 -fmodules build following #5103's repro steps still fails with the pragma in place, and a build of all test TUs with GCC_CXXFLAGS (which enable -Wignored-attributes) and the pragma removed produces no such warning. The block only hid a warning class from GCC C++20 users while suggesting #5103 was handled. Overlaps #5610, whose hunks touch the closing half of this pragma to insert the json_literals.hpp include. Signed-off-by: Niels Lohmann <mail@nlohmann.me> #5725 item 9 * Fix stale doxygen comments hidden by the -Wdocumentation pragma macro_scope.hpp ignores -Wdocumentation and -Wdocumentation-unknown-command for the whole library, which also hides genuine documentation mistakes: - detail::unescape() documented "@return unescaped string" but returns void and unescapes its argument in place; reworded to "@param[in,out] s string to unescape in place" and dropped the bogus @return. - basic_json::get()'s copy-conversion overload wrote "converted to @tparam ValueType" inside @return, which Doxygen and Clang parse as a second, malformed @tparam; changed to "@a ValueType", matching the two other get() overloads a few lines above that already use it. This narrows the gap the -Wdocumentation pragma needs to cover; fully replacing the Doxygen-only commands it also hides (item 2c) is left for after #5267. Signed-off-by: Niels Lohmann <mail@nlohmann.me> #5725 item 2 * Fix -Wextra-semi-stmt at its actual source, not assert() clang_flags.cmake blamed the global -Wno-extra-semi-stmt on assert(), but assert() expands to an expression under glibc and libc++ and does not trigger this warning. unit-assert_macro.cpp overrides JSON_ASSERT with "{if (!(x)) ++assert_counter; }", a bare block followed by a semicolon at every JSON_ASSERT(...) call site in the library; that was the actual source of 151 of the 208 -Wextra-semi-stmt sites found in a Clang 22 -Weverything sweep of the test suite with the flag removed. Switched to the standard do/while(false) macro idiom, which does not expand to a statement-plus-semicolon, and corrected the comment to name the remaining source instead: vendored Doctest's CAPTURE(x) shim, which already ends in a semicolon. Verified with clang++ -Wextra-semi-stmt (plus the file's other CI ignores) that unit-assert_macro.cpp now compiles without any -Wextra-semi-stmt diagnostic. Signed-off-by: Niels Lohmann <mail@nlohmann.me> #5725 item 8 (step 1 of 2; step 2 covers the CAPTURE() call sites) * Drop the redundant semicolon from CAPTURE() call sites; remove -Wno-extra-semi-stmt doctest_compatibility.h defines CAPTURE(x) as DOCTEST_CAPTURE(x); (with a trailing semicolon baked into the macro), specifically so call sites do not need to add one themselves; most of the ~267 call sites already follow that convention. The remaining 64 call sites across 20 files wrote "CAPTURE(x);" anyway, turning into a statement plus an empty statement and triggering -Wextra-semi-stmt. Dropped the redundant semicolon at each of those sites. With item 6 having already made vendored Doctest a SYSTEM include, and this the last known source of -Wextra-semi-stmt findings, removed the flag from clang_flags.cmake entirely. Verified with clang++ -Wextra-semi-stmt (plus the file's other CI ignores) that all 20 touched files, plus a file with no CAPTURE() use (unit-json_pointer.cpp), compile without any -Wextra-semi-stmt diagnostic. Signed-off-by: Niels Lohmann <mail@nlohmann.me> #5725 item 8 (step 2 of 2) * Switch ci_static_analysis_clang off the frozen LLVM 22 dev image ubuntu.yml pinned the clang-tidy/clang-tidy-sanitizer/single-binaries job to silkeh/clang:dev, a tag last pushed 2026-02-18 that reports "clang version 22.0.0 (...+20251015...)", a pre-release snapshot from before the LLVM 22 release; the maintainer now updates dev-unstable, 22, and latest instead. Switched to silkeh/clang:22, matching the other clang jobs on :latest. Verified with clang-tidy 22.1.8 (the image's actual version) against this repository's .clang-tidy and library headers what the release image newly reports compared to :dev: - readability-redundant-typename fires at ~250 sites across the _cpp20-relevant conversion/to_chars headers; the library targets C++11 and keeps the typenames, so the check is disabled in .clang-tidy, matching how the file already handles checks that don't fit a C++11 codebase. - misc-anonymous-namespace-in-header fires on the two anonymous namespaces in from_json.hpp and to_json.hpp; added the alias to their existing NOLINT (cert-dcl59-cpp, fuchsia-header-anon-namespaces, google-build-namespaces). - bugprone-std-namespace-modification fires on every addition to namespace std: the std::hash, std::formatter and std::swap overloads in json.hpp, and the std::tuple_size/std::tuple_element specializations in iteration_proxy.hpp (this last file is not named in #5725's item 5, found by actually running clang-tidy 22.1.8 against the current tree). All six are legal, deliberate additions to namespace std (explicit/partial specializations of std types, or the pre-C++20 std::swap overload); annotated each with the check name next to its existing cert-dcl58-cpp NOLINT. - modernize-avoid-c-style-cast reported nothing new. Also added clang++-22/21, clang-tidy-22/21, g++-16 and gcov-16 to the find_program search lists in ci.cmake so a local "maximal warnings" configure prefers the current toolchain version over an older one on PATH. #5725 item 5 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Regenerate cmake/gcc_flags.cmake for GCC 16.2.0 GCC_CXXFLAGS was generated for GCC 15.1.0, but ci_test_gcc and ci_test_gcc_cxx{11..26} now run in gcc:latest, currently GCC 16.2.0, so the "maximal warnings" job was missing warnings introduced since 15.1.0 while carrying entries GCC 16 treats as duplicates or no-ops. Regenerated with https://github.com/nlohmann/gcc_flags (patched locally to not crash on an option whose "-x c++ <opt> -" probe fails before it reads stdin, e.g. -Wabi=; the tool otherwise raises BrokenPipeError instead of recording the option as an error) run against g++ 16.2.0 in the official gcc:16 Docker image, keeping the documented -Wno-* exclusions and the same alphabetical placement scheme as before. Also added three GCC 16 warnings the generator cannot discover on its own because it only probes value ranges/lists it finds in the -Q option name itself, not in the enum choices --help=warnings documents separately: - -Wbidi-chars=any, -Wleading-whitespace=spaces: manually verified these compile cleanly with g++ 16.2.0. - -Wstrict-flex-arrays: deliberately NOT added, unlike the other two. Without -fstrict-flex-arrays (which the library does not enable, as it would change codegen for flexible array members), GCC prints "'-Wstrict-flex-arrays' is ignored when '-fstrict-flex-arrays' is not present" on every translation unit, and under our -Werror that note itself aborts the build. This differs from the harmless no-op warnings already kept in the file (-Whsa, -Wsynth, -Wunreachable-code, -Wunsafe-loop-optimizations), which emit nothing; #5725 item 7 named -Wstrict-flex-arrays as one of the flags GCC 16 adds, but did not anticipate this failure mode. Verified: compiled the library header and a representative set of test translation units (including ones touched by items 1, 3, 8, 9, 10 of this issue) with the regenerated GCC_CXXFLAGS plus -Werror under g++ 16.2.0 at -std=c++11 through -std=c++26, with zero warnings; ran the full local test suite (129/129 passing, unrelated to this compiler) as a regression check. CI must still confirm the actual ci_test_gcc / ci_test_standards_gcc targets end to end, since this was verified with direct g++ invocations rather than through the CMake/ CXXFLAGS environment-variable plumbing in ci.cmake. #5725 item 7 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Avoid std::basic_string<CharType> for non-character output_adapter CharType output_adapter<CharType, StringType> defaulted StringType to std::basic_string<CharType>, and (with JSON_NO_IO undefined) always declared a std::basic_ostream<CharType>&-taking constructor. For CharType with no non-deprecated std::char_traits specialization (only std::uint8_t is ever used this way, by the binary writers), simply naming either type - as an unused default template argument, or as an unused, never-called constructor's parameter type - instantiates std::char_traits<CharType> merely to name it, which some standard libraries mark deprecated: with the library-wide -Wdocumentation pragma (item 2's other half, left for a later commit) temporarily removed, an Apple clang 21 / libc++ TU calling json::to_cbor(j, vec) with std::vector<std::uint8_t>& got one -Wdeprecated-declarations warning per binary writer at the old output_adapters.hpp:193. Replaced the eager std::basic_string<CharType> / std::basic_ostream <CharType> defaults with a bool-tagged partial specialization (not std::conditional, which requires naming both branches' types up front regardless of which is selected, reproducing the same warning) that only ever names std::basic_string<CharType> / std::basic_ostream <CharType> when CharType is actually one of char, wchar_t, char16_t, char32_t, or (with __cpp_lib_char8_t) char8_t. For any other CharType, output_adapter's StringType and ostream-constructor parameter fall back to two distinct empty placeholder types, kept distinct so the two constructor overloads do not collide into a single redeclaration. Public API / behavior: passing a std::basic_string<std::uint8_t>& or std::basic_ostream<std::uint8_t>& directly to a binary writer's output_adapter now fails to compile instead of compiling with a deprecation warning; this was neither documented nor tested. All documented uses (std::vector<CharType>, std::basic_ostream<CharType> and StringType for character CharType) are unaffected. Verified with Apple clang 21 / libc++, with the two -Wdocumentation* "ignored" pragma lines in macro_scope.hpp temporarily removed and -std=c++11/c++20 plus the project's -Weverything flag set: calling to_cbor/to_msgpack/to_ubjson/to_bjdata/to_bson/to_bon8 on a std::vector<std::uint8_t> now produces no char_traits<unsigned char> (or any other) deprecation warning, while the char-based string- and ostream-adapter paths, and a to_cbor/from_cbor round trip, still compile and run correctly; also verified with GCC 16.2.0. Ran the full local test suite, including the binary-format unit tests (unit-cbor, unit-msgpack, unit-ubjson, unit-bjdata, unit-bson, unit-bon8, unit-binary_writer_sinks, unit-binary_formats, unit-custom-binary-type): 129/129 passing. #5725 item 2 (step a) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Remove the library-wide -Wdocumentation pragma; fix what it hid macro_scope.hpp / macro_unscope.hpp pushed and popped a Clang diagnostic region over the entire library that ignored -Wdocumentation and -Wdocumentation-unknown-command. Removed both pragmas and fixed every finding a full -Wdocumentation (which implies -Wdocumentation-unknown-command and -Wdocumentation-deprecated-sync) build reports, so the library now compiles clean under Clang's documentation checks without a blanket suppression. Overlaps #5267, which is still open and edits a nearby doc block (json.hpp's get()/get_impl() @return, already fixed in the item 2 step (b) commit of this branch); this commit does not touch that block again. Unknown Doxygen alias commands (Doxyfile removed in #3071, so these were never rendered by anything) rewritten as plain prose, keeping the same information: - @requirement REQ-JSON-01 / REQ-JSON-02 (iter_impl.hpp, json_reverse_iterator.hpp): now "This class satisfies the following concept requirements (REQ-JSON-0N):". - @liveexample{prose,example-id} (three sites in json.hpp): kept the prose, dropped the command wrapper and the trailing example-id (docs/mkdocs/docs/examples/*.cpp still exist and are used directly by the rendered docs, not through this in-header alias) and unescaped the "\," commas that were only needed for the old alias's comma-separated argument syntax. - @complexity X (json.hpp x4, json_pointer.hpp x2, serializer.hpp x1): now "Complexity: X". Backslash sequences Clang's comment lexer tried to parse as commands, escaped to render as literal backslashes: - lexer.hpp get_codepoint(): two `\u` occurrences. - binary_reader.hpp get_bson_cstr() / get_bson_cstr_bulk(): two `\x00` occurrences. - serializer.hpp: three `\uXXXX` occurrences (constructor @param, append_codepoint_to_string_buffer() @brief, and the ensure_ascii member comment). One finding remained after all of the above: Clang reports "declaration is marked with '@deprecated' command but does not have a deprecation attribute" on the deprecated sax_parse(span_input_adapter&&, ...) overload, even though JSON_HEDLEY_DEPRECATED_FOR does expand to __attribute__((deprecated(...))) for Clang. Several isolated reproductions of this exact declaration shape - doc comment, template<>, two stacked __attribute__ macros, an overload set sharing the name - did not reproduce the warning, so this looks like a Clang comment/declaration-association quirk specific to this overload inside the much larger basic_json class template, not an actual documentation defect. Rather than keep the pragma library-wide for one Clang false positive, added a tightly scoped -Wdocumentation-deprecated-sync push/pop around just that overload. Verified with Apple clang 21 and the project's actual -Weverything flag set (cmake/clang_flags.cmake) on the full header at -std=c++11 and -std=c++20: zero -Wdocumentation* diagnostics. Also compiled clean with GCC 16.2.0 (the pragmas are already __clang__-gated, so this only confirms no unrelated breakage). Ran make check-amalgamation and the full local test suite: 129/129 passing. #5725 item 2 (step c) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Take the JSON value by const reference in the array and tuple from_json paths Review feedback on #5737 (gregmarr): once the no-op std::forward calls are gone, the forwarding references have no purpose. from_json_fn passes the value as const BasicJsonType&, so these functions were only ever instantiated with a const lvalue anyway. The std::array, std::pair and std::tuple overloads of from_json and their helpers now take const BasicJsonType& and pass j on unchanged. Because the deduced BasicJsonType is now the plain type, tuple_type and the static_assert name const BasicJsonType& explicitly, so the reference checks are unchanged: get<std::tuple<const std::string&>>() still works, and get<std::tuple<std::string&>>() still fails the same static_assert. from_json_tuple_get_impl keeps its forwarding reference, since tuple_type calls it through std::declval. Behavior, the public API and the ABI do not change. unit-conversions, unit-constructor1, unit-udt, unit-udt_macro, unit-regression1/2/3, unit-deserialization, unit-noexcept, unit-items, unit-allocator, unit-custom-object-type, unit-ordered_json2 and unit-brace-init-copy-semantics pass at C++11, C++17 and C++20 with unchanged assertion counts. Ran make amalgamate. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
b54ed188e6 |
Remove test debt: dead guards, discarded results, and unreferenced files (#5732)
* Run the README test case in JSON_FastTests jobs The "README" test case was marked doctest::skip() when the tests moved from Catch to doctest in 2019, where it replaced Catch's hidden tag. It is not slow (17 assertions, about 0.00 s), but cmake/test.cmake only passes --no-skip when JSON_FastTests is off, so the per-compiler ci_test_*_cxxNN matrix, macOS, Windows Release/ARM, icpc, icpx and nvhpc compiled the README examples without running them. Drop the skip decorator so every job runs the case. Test-only change. Part of #5713 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Remove stale clang ranges guards in unit-iterators2.cpp The "algorithms" and "views" sections were guarded by clang/libstdc++ checks written for a clang 15 (04/2022) bug. The first guard's condition contradicts its own comment: it skips clang+libc++ and keeps clang+libstdc++. Both sections already sit inside `#if JSON_HAS_RANGES`, which macro_scope.hpp excludes for the toolchains these guards targeted, so the inner guards never let the sections run on the platforms they meant to protect and are redundant on the rest. Verified locally with Apple clang 21/libc++ and clang 16.0.6/libstdc++ 12 (Docker): both pass all 1355 assertions with the guards removed. Part of #5713 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix copy-pasted CBOR half-float checks; enable stale encode checks In the RFC 8949 Appendix A test case, the decode checks for 5.960464477539063e-8 (0xf9 0x00 0x01) and 0.00006103515625 (0xf9 0x04 0x00) were copy-pasted from the neighboring -4.0 example, so those two half-float byte sequences were never actually decoded and checked, and -4.0 was checked three times instead. The two float32 encode checks for 100000.0 and 3.4028234663852886e+38 were commented out before the writer supported emitting float32 and are now verified to match byte for byte, so they are enabled. The remaining commented-out half-precision to_cbor checks are collapsed into a single explanatory comment, since the writer never emits half-precision floats. Part of #5713 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Assert on the result of STL container conversions in tests The "object-like STL containers" and "array-like STL containers" sections converted json values into std::map, unordered_map, multimap, unordered_multimap, list, forward_list, array, valarray, vector, deque, set and unordered_set and discarded the result, so these ~60 conversions only proved that the code compiles and does not throw; a conversion that dropped or reordered elements would still pass. Bind each result and compare it against the expected container. Also fix a copy-paste slip in the deque section (`j2.get<std::deque<double>>()` instead of j3, so j3's doubles were never converted to a deque), and remove the dead `// CHECK(m5["one"] == "eins")` comments that referred to a variable that did not exist by asserting the equivalent through the bound result. Part of #5713 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Deduplicate SaxCountdown and other test helpers across formats SaxCountdown was copied byte-for-byte into six binary-format test files (unit-cbor.cpp, unit-msgpack.cpp, unit-ubjson.cpp, unit-bjdata.cpp, unit-bon8.cpp, unit-bson.cpp), about 370 redundant lines. Move it into tests/src/sax_countdown.hpp (namespace utils, alongside test_utils.hpp and round_trip_corpus.hpp) and include it from all six. trait_test_arg and the "value_in_range_of trait" TEST_CASE_TEMPLATE_DEFINE were duplicated between unit-32bit.cpp and unit-bjdata.cpp; the trait is a detail/meta trait, not specific to either file. Move it into tests/src/value_in_range_of_test.hpp; unit-32bit.cpp keeps its own include, since JSON_32bitTest=ONLY builds only that file. Each file keeps its own TEST_CASE_TEMPLATE_INVOKE list. sax_no_exception and the "issue #2824" section were duplicated in unit-regression2.cpp and unit-disabled_exceptions.cpp. Drop the copy from unit-regression2.cpp; unit-disabled_exceptions.cpp already covers the no-exceptions case that #2824 was about, and ci_test_noexceptions reruns it. No behavior change. Verified by building and running unit-cbor, unit-msgpack, unit-ubjson, unit-bjdata, unit-bon8, unit-bson, unit-32bit, unit-regression2 and unit-disabled_exceptions against include/ (clang++ -std=c++11, ASan/UBSan where applicable); assertion counts are unchanged from before the refactor. Part of #5714 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Remove the unreferenced vendored libFuzzer tests/thirdparty/Fuzzer (155 files, ~776 KB of vendored Apache-2.0 LLVM code from the 2016 OSS-Fuzz import) is not referenced by any CMakeLists, Makefile or workflow: the fuzz drivers link against -fsanitize=fuzzer or the repo's own tests/src/fuzzer-driver_afl.cpp. Its vendored README only points at llvm.org's own libFuzzer docs. Being dead code, it also adds noise to the flawfinder code-scanning workflow, which scans the whole tree. Remove the directory and its .reuse/dep5 entry. Part of #5714 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Remove unreferenced 2016 benchmark and fuzz reports tests/reports (1.6 MB) holds AFL status pages and plots from 2016-08-29 and 2016-10-02, and a nativejson-benchmark snapshot from 2016 with links to rawgit.com, which shut down in 2019. Nothing references this directory: no doc, README section, script or workflow points at it, and it describes a ten-years-old, pre-2.0 snapshot of the library. Part of #5714 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Run the CBOR, MessagePack, BSON and BON8 round-trip invariants in CI tests/src/round_trip_corpus.hpp exists so that the byte-stability invariant the fuzzer drivers check also runs on a fixed corpus in CI, instead of only at OSS-Fuzz. So far only the UBJSON and BJData drivers had a matching unit test; the CBOR, MessagePack, BSON and BON8 drivers assert the same invariant (assert(to_X(j2) == vec)) but nothing ran it outside OSS-Fuzz. Add "<FORMAT> round-trip invariants" test cases to unit-cbor.cpp, unit-msgpack.cpp, unit-bson.cpp and unit-bon8.cpp, modeled on the UBJSON case: seed j1 from the corpus (skipping values that do not survive the format's own round trip, as the fuzzer drivers only ever see values from_X() actually produced), then require from_X(to_X(j1)) not to throw and check to_X(j2) == to_X(j1). BSON only serializes objects, so non-object corpus values are skipped. Update the comments in round_trip_corpus.hpp and tests/fuzzing.md to name all six formats. The stream-versus-contiguous check in the BON8 driver is left out, as #5601 reworks it. A local probe confirms no violations on the current corpus (CBOR 3849 checked, MessagePack 3909, BSON 2958, BON8 3841 - matching the counts already recorded for this probe in the issue). Closes #5714 item 1. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix fuzzer driver step lists to match the checks the code performs The header comment of six of the seven binary-format fuzzer drivers listed an invariant the code does not check: CBOR, MessagePack, BSON and BON8 said "assert(j1 == j2)", but the code checks byte stability, assert(to_X(j2) == vec). UBJSON and BJData still described the old "assert(j1 == j2/j3/j4)" byte-exact check from before PR #5494 replaced it with a use_size/use_type-aware round trip (UBJSON) and a value-stability check (BJData); BJData's added paragraph already explained the new check, but the step list above it did not. Also remove a dead branch in fuzzer-parse_bson.cpp: from_bson() is called with allow_exceptions = true, so it throws instead of returning a discarded value, and the "if (j1.is_discarded()) return 0;" guard could never trigger. Drop the unused <iostream> include from all seven drivers and <sstream> from all but fuzzer-parse_bon8.cpp, which is the only one that uses std::istringstream. Overlaps #5601, which edits all seven drivers in the same hunks. Closes #5714 item 4. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Silence the CMP0169 deprecation in cmake_fetch_content, fix stale guards tests/cmake_fetch_content/project calls the single-argument FetchContent_Populate(json) after FetchContent_Declare(), which CMake 3.30 deprecated as CMP0169. Since the project declares cmake_minimum_required(VERSION 3.11...3.14), the policy stays unset, so every configure with a current CMake prints the deprecation warning. The test is kept on purpose: it is the only coverage of the FetchContent_Populate + add_subdirectory pattern for CMake 3.11-3.13 users, which the docs still describe as supported. Explicitly set CMP0169 to OLD, with a comment explaining why. Also fix two stale version guards: - tests/cmake_fetch_content/CMakeLists.txt guarded the test with VERSION_GREATER "3.11.0", which is dead now that tests/CMakeLists.txt requires CMake 3.13. - tests/cmake_fetch_content2/CMakeLists.txt guarded with VERSION_GREATER "3.14.0", which skips exactly 3.14.0, the first version with FetchContent_MakeAvailable. Change it to VERSION_GREATER_EQUAL "3.14". Verified locally: `ctest -R cmake_fetch_content` passes with CMake 4.1, and the CMP0169 deprecation warning that appeared before this change is gone. Closes #5714 item 5. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Make the CMake integration-test wrappers consistent The six tests/cmake_* integration-test wrappers had drifted: - Only cmake_import and cmake_import_minver forwarded -A "${CMAKE_GENERATOR_PLATFORM}" to the inner configure, and none forwarded -T "${CMAKE_GENERATOR_TOOLSET}". The Windows workflow configures the outer build with -A Win32 -T ClangCL, so without forwarding, the inner projects of cmake_add_subdirectory, cmake_fetch_content, cmake_fetch_content2 and cmake_target_include_directories built with the generator defaults instead of matching the outer build's platform and toolset. Forward both consistently from all six wrappers. - cmake_fetch_content and cmake_fetch_content2 passed -Dnlohmann_json_source to their inner projects, which never read it (CMake warns "manually-specified variables were not used"); the inner projects fetch their own copy of the library instead. Drop it. - tests/CMakeLists.txt set JSON_FORCED_GLOBAL_COMPILE_OPTIONS from the matching environment variable but never read the cache variable again; the lines right below it read $ENV{JSON_FORCED_GLOBAL_COMPILE_OPTIONS} directly, like the LINK_OPTIONS counterpart already does. Remove the dead set(). This changes which platform and toolset the Win32 and ClangCL CI jobs build the four newly-forwarding wrappers' inner projects with, which may surface new failures there; CI has to confirm those jobs. Verified locally with Ninja (empty -A ""/-T "" is accepted): all 12 cmake_* tests still pass, and the inner fetch_content configures no longer warn about the unused variable. Closes #5714 item 6. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Turn the #972 fifo_map regression test into a real test The #972 regression test in unit-regression1.cpp only built a my_json array from a string literal (the original crash) and had no CHECK, so the fifo_map object type it exists to demonstrate was never exercised. Meanwhile the docs recommend fifo_map for keeping object keys in insertion order (object_order.md, template_parameters.md), and nothing tested that recommendation. Extend the section: after the original array assignment, parse an object with my_json::parse() (not via the "..."_json UDL, which returns a plain nlohmann::json and would exercise the cross-basic_json conversion constructor instead of the parser's own key insertion - and, as tried locally, does not keep fifo order for this stateful comparator) and check that dump() keeps insertion order, and that it survives erase() and inserting a new key. Also narrow thirdparty/fifo_map off the include path of every other test-* target: it was a PUBLIC include directory of test_main, even though unit-regression1.cpp is its only user. Add a small fifo_map_include INTERFACE library with that include directory and attach it to test-regression1 only via json_test_set_test_options(). Verified locally (test-regression1_cpp11, default build and -fsanitize=address,undefined): the new checks pass; `git grep fifo_map tests` still only finds unit-regression1.cpp and the vendored header. Closes #5714 item 7. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Document the vendored doctest.h patch; fix stale doctest_compatibility.h comments tests/thirdparty/doctest/doctest.h is doctest 2.4.12, imported in #4771. Two weeks later, #4801 hand-edited translateActiveException() to declare "String res;" inside the translator loop instead of before it, so a translator that does not match does not leave a previous translator's result in "res" for the next iteration to see. Nothing recorded this, so re-vendoring doctest.h from upstream would silently drop the fix. Add a comment at the patched site naming the version, the PR and the reason, so a future re-vendor knows to re-apply it. Also fix two stale comments in doctest_compatibility.h: - The DOCTEST_THREAD_LOCAL comment referenced Xcode 6/7, which is no longer supported; reword it to explain why the define must stay regardless (it keeps doctest's own thread_local usage out of the way of the same Clang/MinGW crash that JSON_NO_THREAD_LOCAL works around in the library, see ci_test_no_thread_local). - The <iosfwd> include's comment justified it with tests that define "private" as "public"; no test under tests/src does that any more (removed by #2352). Reword the comment instead of dropping the include, since confirming it is safe to drop needs the full CI matrix including MSVC 2015+. Verified locally that tests/src/unit-readme.cpp still builds and passes 17/17 with these headers. Closes #5714 item 9. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Stop compiling unit-wstring.cpp out entirely on classic ICC tests/src/unit-wstring.cpp wrapped the whole file in #ifndef __INTEL_COMPILER, with the comment "ICPC errors out on multibyte character sequences in source files". The ci_icpc job (intel/oneapi-hpckit:2023.2.1) still exists, so that job ran none of the wstring/u16string/u32string input adapter tests, including the malformed-input checks #5704 (open) extends. Only 9 lines contained non-ASCII bytes: the three *_is_utf16()/ *_is_utf32() probe functions, and three std::wstring/u16string/ u32string literals plus their narrow-string dump() expectations. Rewrite all of them with \u/\U escapes in the wide/u16/u32 literals and \x escapes (split into separate string-literal tokens so a following byte is never read as part of the same hex escape, e.g. "\xE1\x83\x85" "a") in the narrow ones. Remove the #ifndef __INTEL_COMPILER/#endif guard along with it. The *_is_utf16()/*_is_utf32() probes compared a raw multibyte literal against an escape-based one to detect a compiler that misreads the source file's encoding; with no raw literals left to misread, the comparison is now tautological, so drop the probes and the "if" guards around each SECTION's body instead of leaving them in as dead checks. The same non-ASCII-in-source-and-in-a-narrow-comparison pattern existed once more in unit-deserialization.cpp's "Using _json with char8_t literals #4945" test: a raw emoji character in a u8R"(...)" literal, guarded by a check_utf8() that returned false for ICC (same reason) and for Windows without the active UTF-8 code page. Rewrite the literal with a \U escape and compare it against a \x-escaped expectation instead of a second raw literal, and drop check_utf8() and the now-unused <windows.h> include along with the guard. Verified locally (clang, -std=c++11 and -std=c++20, -fsanitize=address,undefined, and a plain build): test-wstring keeps 18/18 assertions and unit-deserialization keeps 466/466 (c++11) and 477/477 (c++20) assertions, matching this branch before the change exactly - no coverage was gained or lost, only the source-encoding dependency was removed. ci_icpc has to confirm classic ICC actually builds and passes test-wstring now; if it does not, that is a real finding, not a reason to restore the guard. Overlaps #5704 (open), which edits unit-wstring.cpp inside the previously-guarded region (an include near the top, checks in the invalid-string sections, and a new section at the end). Closes #5713 item 5. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
9e1a09eec0 |
Name the key type when rejecting non-string CBOR/MessagePack map keys (#5594)
* Name the key type when rejecting non-string CBOR/MessagePack map keys CBOR and MessagePack allow map keys of any type, but JSON object keys are always strings, so such maps are rejected. The error so far was the one for a malformed string (e.g. "expected length specification (0xA0-0xBF, 0xD9-0xDB); last byte: 0xC0" for a nil key), which does not tell the user what went wrong. Report the type of the key instead: syntax error while parsing MessagePack object key: only string keys are supported, but found nil; last byte: 0xC0 The exception id (parse_error.113) and type are unchanged. Malformed string keys and a missing key keep their previous messages. Document the restriction on the CBOR and MessagePack pages. Refs #2766, #3381 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Point the MessagePack key note to the spec's profile section The note linked to "Serialization: type to format conversion", which says nothing about key types. Restricting map keys to strings is only mentioned in the "Profile" section (under "Future discussion") as an example of a JSON-compatible profile, so link there and describe it as such instead of as a permission. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
Kartikey Negi
|
6bd106893a |
Fix CBOR tag handling in cbor_tag_handler_t::store for non-binary items (#5559)
When using cbor_tag_handler_t::store, tags 0xD8-0xDB previously assumed that the tagged item was a byte string, unconditionally attempting to parse binary data and failing on valid CBOR documents containing tags applied to integers, strings, arrays, or objects (such as self-describe tag 55799). Check whether the tagged data item is a byte string (0x40-0x5B or 0x5F). If it is a byte string, store the subtype on the binary value as before. Otherwise, iteratively process the tagged value in the driver loop using item_read so that chained tags do not consume native stack space. Part of #5316. Signed-off-by: ReturnKartikey <kartikeynegi2000.work@gmail.com> |
||
bucketbase26
|
98e00d22e5 |
Cut test suite runtime in binary roundtrips and integer sweeps (#5519)
* Cut test suite runtime in binary roundtrips and integer sweeps The Linux CI jobs pass --no-skip, so skip() does not help there. Parse each corpus file once in the binary roundtrip loops instead of four times. Sample the 16-bit integer ranges with stride 7 (still hits every low byte) and always keep the endpoints. Also drop the 5M-node parse test to 500k, which still covers the non-recursive destructor, and move jeopardy.json into its own skipped test so the cheaper binary-format size checks actually run. See #5418. Signed-off-by: ayush-singh-0601 <singhayush062006@gmail.com> * Drop useless int32_t casts in the sampled integer loops ci_test_gcc compiles with -Werror=useless-cast. On that compiler int32_t is int, so static_cast<int32_t> of the loop bound is an error. The bounds are already int, and the sampled values do not change. Signed-off-by: ayush-singh-0601 <singhayush062006@gmail.com> * Revert unit-binary_formats.cpp to develop and fix comment Revert tests/src/unit-binary_formats.cpp to its develop state. The test-case split made valgrind jobs slower instead of faster, because the cheaper corpus files (canada/twitter/citm/sample) now ran under valgrind where they never did before. Fix the next_integer_sample comment: the function has no 'first' parameter, so describe what the function actually does. Signed-off-by: ayush-singh-0601 <singhayush062006@gmail.com> --------- Signed-off-by: ayush-singh-0601 <singhayush062006@gmail.com> |
||
|
Niels Lohmann
|
85f8b21e1c |
Add tests for uncovered code paths (#5581)
* Add tests for uncovered code paths Cover code the test suite did not reach, found from the Coveralls report of develop and a local coverage run of HEAD: - dump() of every kind of value below the bound of the recursive descent (pretty-printed objects, binary values, discarded values, scalars), and flushes of the escape and write buffers mid-string and mid-binary - the iterative comparison: objects with different keys, containers that are a prefix of each other, and elements that cannot be ordered, each both at the top level and below the nesting bound - SAX handlers that stop at any event, including the end of a nested container, in the BSON, CBOR, MessagePack, UBJSON and BJData readers - from_bson/cbor/msgpack/ubjson/bjdata returning a discarded value through the iterator and pointer overloads - JSON Patch, diff, merge_patch and update(..., true) on ordered_json - smaller gaps: get_allocator(), to_ubjson/to_bjdata into a string, value() with an unresolvable JSON pointer, integer/float comparison below the integer range and with negative fractions, conversion to a custom binary type, std::formatter::parse on a spec without '}', unescape() of a lone '~', and the callback parser's start_array() Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Cover more paths that were thought unreachable - parse_float_fast() declining malformed or inexact input, called directly since the lexer only passes well-formed numbers to it - a UTF-16 high surrogate followed by a unit above the low surrogates - self-assignment of a const_iterator - a truncated CBOR string read through non-contiguous iterators - serializing a long double under the de_DE locale, which undoes the locale's decimal point and thousands separator - values read from a binary format carrying no diagnostic positions, with and without a parser callback Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix the CI failures of the new coverage tests - declare the self-assignment reference const (misc-const-correctness) - expect the (/path) prefix that JSON_DIAGNOSTICS adds to the messages of the failing ordered_json patch operations Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Expect the byte range JSON_DIAGNOSTIC_POSITIONS adds to the patch errors Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Build the expected dump of the nested-object test with += clang-tidy (performance-inefficient-string-concatenation) reported the chain of operator+ calls that assembled the expected indented output. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Compare the BJData and UBJSON test outputs byte by byte Building a std::string from the byte vector converts each byte implicitly, which -fsanitize=integer reports for bytes of 0x80 and above (ci_test_clang_sanitizer). Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
d19f7f5dce |
Fix BSON conformance issue (#5185)
* 🐛 fix BSON conformance issue Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 🐛 fix BSON conformance issue Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 🐛 reject ill-formed UTF-8 in CBOR/MessagePack/BSON text strings at decode time (#5531) from_cbor()/from_msgpack()/from_bson() copied the raw bytes of a decoded text string into the resulting json value without any UTF-8 validation, even though RFC 8949 §3.1 (CBOR) and the MessagePack/BSON specifications all require text strings to be valid UTF-8. Malformed input only failed later, if the value was dump()'d, with a type_error.316 - so the allow_exceptions=false pattern used specifically to get a discarded sentinel instead of an exception did not discard this category of malformed input, unlike every other kind of malformed binary input this library rejects at decode time (see #5529). Fix this at the single choke point shared by BSON/CBOR/MessagePack/UBJSON string reads, binary_reader::get_string(): validate the bytes with the UTF-8 DFA right after they are read, and report failures the same way as every other binary_reader error (parse_error.113), so allow_exceptions and strict discarding behave consistently. get_binary()/binary blob reads are untouched and still accept arbitrary bytes, since only text strings are required to be UTF-8. There were two independent implementations of a UTF-8 validator: the lexer's streaming scanner, and the serializer's Hoehrmann DFA used by dump_escaped_impl(). Rather than write a third, the serializer's decode() function, its utf8d table and the UTF8_ACCEPT/UTF8_REJECT constants are extracted into detail/string_utils.hpp (a low-level header already included before both detail/input/ and detail/output/), alongside a new is_valid_utf8() helper built on the same decode() step. serializer.hpp's dump_escaped_impl() now calls the shared decode(), so there is exactly one UTF-8 validator in the codebase; dump()'s exact type_error.316 messages and byte-index reporting are unchanged (see the added regression-guard test in unit-serialization.cpp). Claude-Session: https://claude.ai/code/session_01N4RQ1Ahan5YAGbnAQGjZTY Signed-off-by: Niels Lohmann <mail@nlohmann.me> Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> * ⚡ validate only newly read bytes of binary-format strings get_string() validated the whole result after each call, but get_bytes() appends to it and CBOR indefinite-length strings collect all chunks in the same result, so every chunk re-validated everything read before it. An input of many small chunks took quadratic time (80000 one-byte chunks, 160 KB of input, took about 7 seconds). Only the newly read bytes are validated now, which also matches RFC 8949's requirement that every chunk is valid UTF-8 on its own. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
Niels Lohmann
|
a50c2537eb |
Reserve capped array capacity in json_sax_dom_parser::start_array() for definite-length binary arrays (#5476)
* Reserve capped array capacity for definite-length binary arrays CBOR, MessagePack, and the optimized [$type#count UBJSON/BJData form all pass an exact element count to sax->start_array(len), but json_sax_dom_parser::start_array() (and the callback variant) only used len for an overflow check against max_size() and never reserved the underlying vector, so each element triggered a reallocation cascade via emplace_back(). Reserve upfront, but cap the reservation at 16384 elements: max_size() for a std::vector is far larger than any realistic input, so an unbounded reserve(len) would let a crafted/truncated header (e.g. CBOR 0x9A + a huge uint32 count with no data) trigger a multi-gigabyte allocation attempt instead of the normal graceful parse_error. With the cap, a hostile length still fails fast with the existing parse_error, while realistic arrays get a single up-front allocation. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Make the huge-claimed-length DoS regression tests portable across size_t widths On a platform where size_t is narrower than 64 bits (e.g. 32-bit mingw/msvc x86), the previously-hardcoded huge test lengths either collide with that platform's unknown_size() sentinel (CBOR/MessagePack, both using exactly SIZE_MAX) or exceed the platform's smaller vector<json>::max_size() (UBJSON/BJData's 0x7FFFFFFF), so the header is now rejected outright (out_of_range.408) instead of being accepted and only found short of data (parse_error.110). Both are safe, bounded rejections of the hostile input; the property under test -- no attempt to allocate space for billions of elements -- holds either way. Accept both outcomes instead of pinning the 64-bit-only exact result. Also fixed an unrelated clang-tidy finding (google-readability-casting) on the functional-style std::size_t(...) casts in the neighboring "arrays of various sizes" section. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix remaining CI failures in the huge-claimed-length DoS regression tests - Apply the same google-readability-casting fix (std::size_t{N} instead of std::size_t(N)) to the "arrays of various sizes" section in unit-msgpack.cpp, unit-ubjson.cpp, and unit-bjdata.cpp; only unit-cbor.cpp had been fixed previously, since clang-tidy's build didn't get far enough to report the other three in the same pass. - json_sax_dom_parser::start_array()'s max_size() check calls JSON_THROW directly rather than going through sax->parse_error(), so unlike the scanner's own "not enough data" parse_error it is not gated by allow_exceptions=false. On a platform where a header's claimed count exceeds max_size() (e.g. 32-bit, for UBJSON/BJData's 0x7FFFFFFF test value), from_ubjson/from_bjdata(input, true, false) can therefore still throw instead of returning a discarded value. Make that assertion tolerant of either outcome, same as the main exception-catching check above it. - Guard all four "a huge claimed length..." SECTIONs with #if !defined(JSON_NOEXCEPTION), matching this test suite's existing convention for exception-dependent tests: under JSON_NOEXCEPTION, JSON_THROW never produces a catchable C++ exception at all (it aborts the process), so a section that relies on try/catch to distinguish between two acceptable outcomes cannot be expressed under that build configuration regardless of platform. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Use (std::min)(len, reserve_cap) instead of a ternary in start_array() Addresses review feedback from @gregmarr on PR #5476. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
91ab3e81f5 |
Read CBOR containers and tags without recursing per nesting level (#5506)
* Read CBOR containers and tags without recursing per nesting level get_cbor_array() and get_cbor_object() read their elements by calling back into the value reader, which called them again for a nested container, and a tag was handled by reading the tagged value the same way. All three cost native stack, and all three cost a single byte to encode: 0x9F opens an indefinite-length array, 0x81 a one-element array, and 0xC2 is a tag. Half a million of any of them crashes the process before the input runs out (#5104). Apply the shape the MessagePack reader already uses: the open containers live on the heap stack, parse_cbor_value() reads a single value and only opens a container rather than reading it to its end, and parse_cbor_internal() loops, resuming the innermost container after each element. Two things are specific to CBOR. An indefinite-length container ends at a break marker rather than at a count, and testing for that marker consumes a byte which is the first byte of the next element when it is not one; the frame's count is npos for those, and the driver tracks whether the next value starts at a fresh byte. And a tag is not a value of its own: instead of reading the tagged value by recursing, the value reader reports that a tag was read and the driver reads on, so a chain of tags costs no stack at all. The switch that decodes a value is unchanged apart from the twelve container cases and the two tag sites. Verified against the previous commit over definite and indefinite arrays and maps, all four counted forms, empty containers, nesting of the forms inside each other, truncated inputs, and all three tag handlers: identical values, error codes, messages and byte offsets. 500,000 levels of each of the three vectors now report parse_error.110 instead of crashing, and a well-formed 200,000-level value is read to completion. On performance: the driver does per element what a counted loop used to do per container, and CBOR pays for it more than MessagePack because the value reader also has to be told whether to fetch a byte. Parsing 60,000 small objects and one array of a million integers is 3 to 4 % slower than the recursive reader, measured over five alternating runs. Against develop the same two inputs are about 44 % faster, because the entry point no longer copies the value it parsed; the earlier commit in this series is what pays for that. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Make parse_cbor_internal's top a copy so it survives pop_back() top aliased container_stack.back(), and was still read (top.is_object) right after container_stack.pop_back() destroyed the element it aliased. Nothing currently reorders those two lines, but the comment claiming the reference's lifetime was already fine only accounted for reallocation from a push, not this. A trivially-copyable container_frame makes top a copy instead, so reads of it stay valid regardless of what happens to the stack; the one place that mutates the live entry now does so through container_stack.back() directly rather than through top. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
dd50f0eb16 |
Stop CBOR indefinite-length strings from recursing per chunk (#5502)
get_cbor_string() and get_cbor_binary() handled the indefinite-length forms (0x7F and 0x5F) by calling themselves once per chunk. Each chunk therefore cost a native stack frame, and since a chunk may itself be an indefinite- length string, an input of repeated 0x7F bytes reached one frame per input byte: 200,000 of them crash the process with SIGSEGV before a single byte is rejected. This is the same defect as #5104, in a path the container-level work does not touch. Count the open levels instead of recursing through them. That is enough here because every chunk is appended to the same result -- get_bytes() writes at result.size() -- so there is no per-level state to keep. The temporary chunk string and its copy into the result go away with the recursion. The definite-length cases move to get_cbor_string_chunk() and get_cbor_binary_chunk() unchanged, including their error messages, which still name 0x7F and 0x5F because those are handled one level up. Behaviour is unchanged. Comparing against develop over the interesting byte sequences -- empty, single-chunk, nested, over-closed and truncated forms, both strings and byte arrays, and an indefinite-length map key -- produces identical values, error codes, messages and byte offsets. The 200,000-level input now reports parse_error.110 at byte 200001 instead of crashing. Note that nesting these is not valid CBOR: RFC 8949, Section 3.2.3 forbids it. This does not change that either way -- it has always been accepted, and rejecting it is a separate decision (#5317, #5325). Should it be rejected later, that is now one condition on the level counter rather than a change to the control flow. Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
Sahil_Kamate
|
b5378e8deb |
Fix CBOR tag handlers not recognizing tags 0-5 and 21-23 (#5331)
* Fix CBOR tag handlers not recognizing tags 0-5 and 21-23 The tagged-item switch in binary_reader::parse_cbor_internal() only handled head bytes 0xC6-0xD4 and 0xD8-0xDB. Bytes 0xC0-0xC5 (tags 0-5: date/time, epoch, bignum, decimal, bigfloat) and 0xD5-0xD7 (tags 21-23: base64url, base64, base16 conversion hints) fell through to the default case and were reported as invalid bytes, even under cbor_tag_handler_t::ignore and ::store, despite being valid CBOR major-type-6 tags per RFC 8949. Add the missing case labels so the full 0xC0-0xDB range is handled uniformly. Extend the "Tagged values" test in unit-cbor.cpp to cover 0xC0-0xD7, and update the CBOR docs to state the corrected tag range. Fixes #5315 Signed-off-by: sahilkamate03 <45514385+sahilkamate03@users.noreply.github.com> * Fix stale CBOR tag docs and add store-mode binary-payload test The "Incomplete mapping" warning still listed tags 0-5 (date/time, bignum, decimal fraction, bigfloat) and 21-23 (expected conversions) as unsupported, even though they now parse correctly under cbor_tag_handler_t::ignore/store, same as 0xC6..0xD4/0xD8..0xDB. Remove those five bullets and cross-reference the "Tagged items" warning below, matching the equivalent docs fix landed independently in PR #5367. Also add a cbor_tag_handler_t::store test that wraps a binary payload (not just a string) for every byte in 0xC0..0xD7, confirming these tags are unwrapped the same way as 0xC6..0xD4 rather than mistaken for the 0xD8..0xDB binary-subtype marker syntax, per review feedback on #5331. Signed-off-by: sahilkamate03 <45514385+sahilkamate03@users.noreply.github.com> --------- Signed-off-by: sahilkamate03 <45514385+sahilkamate03@users.noreply.github.com> |
||
Angadi56
|
d94cbd99dc |
reject CBOR array/map length equal to the indefinite-length marker (#5274)
* reject CBOR array/map length equal to the indefinite-length marker Signed-off-by: Angadi Yashaswini <angadi@digiscrypt.com> * reject CBOR lengths that do not fit in std::size_t via value_in_range_of Signed-off-by: Angadi Yashaswini <angadi@digiscrypt.com> --------- Signed-off-by: Angadi Yashaswini <angadi@digiscrypt.com> |
||
Patrick10199
|
868506dcc0 |
Fix CBOR half-float assertion bounds (#5335)
Signed-off-by: Patrick Armstrong <patrick@erpassistant.ai> |
||
|
Niels Lohmann
|
ca76c37650 |
Add iterator+sentinel tests and docs for binary deserializers (#5265)
* Add iterator+sentinel tests and docs for binary deserializers This commit extends the C++20 ranges support (iterator+sentinel pairs) to the binary format deserializers from_cbor, from_msgpack, from_ubjson, from_bjdata, and from_bson, matching what was already done for parse(), accept(), and sax_parse(). Changes: - Add istreambuf_sentinel helper to test_utils.hpp for EOF detection in tests - Add 5 new test cases that read binary files directly via std::istreambuf_iterator<char> + sentinel, without pre-buffering - Update documentation for all 5 from_* functions to document overload (3) with SentinelType parameter - All tests pass; verified against existing test suite data - Fix potential buffer over-read warning in heterogeneous iterator test Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Merge iterator+sentinel overloads and fix ambiguity/CI issues Address PR review feedback and CI failures: - Merge the separate same-type and sentinel-type iterator overloads of parse(), accept(), sax_parse(), and the five from_* binary deserializers into a single overload with SentinelType defaulted to IteratorType, as suggested in review. Applied the same simplification to the detail::input_adapter() free functions. - Fix a latent ambiguity: some compilers (e.g. GCC 4.8) unreliably SFINAE the operator!= detection for std::nullptr_t against container/string types, making calls like parse(s, nullptr, ...) ambiguous with the compatible-input overload. can_compare_ne now explicitly excludes std::nullptr_t as a SentinelType. - Use a named enable_if_t template parameter instead of an unnamed function parameter for the SFINAE guard, fixing a clang-tidy hicpp-named-parameter/readability-named-parameter failure. - Update parse.md, accept.md, sax_parse.md, and the five from_*.md pages to document the merged overload instead of separate (2)/(3) overloads, also fixing an over-160-char line that broke the documentation style_check CI job. - Rework the BSON iterator+sentinel test to parse a BSON file already present in the test suite instead of writing/deleting a temp file. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix -Wunneeded-internal-declaration for CustomSentinel in test CustomSentinel lives in an anonymous namespace (internal linkage), and the library's parse loop only ever evaluates the iterator-first direction (it != last), so the reversed-order friend operator!= was never referenced. Clang's -Weverything flags such unused internal declarations as an error. Drop the unused overload; the used direction is enough to satisfy can_compare_ne's either-order detection. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix clang-tidy hicpp-named-parameter and misc-const-correctness - Drop the unused reversed-order operator!= overload from utils::istreambuf_sentinel (only iterator != sentinel is ever evaluated) and name the remaining friend's sentinel parameter, fixing hicpp-named-parameter/readability-named-parameter. - Mark the istreambuf_iterator first/last helper variable const in the five binary-format sentinel tests, fixing misc-const-correctness. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix clang-tidy misc-const-correctness in heterogeneous sentinel test json_str is only read via .data()/.size() and never reassigned, so clang-tidy correctly flags it as const-able. Verified against the exact CI job (silkeh/clang:dev, ci_clang_tidy target) by running clang-tidy directly on this file plus the five binary-format sentinel tests touched by prior commits; all are now clean. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
83c87cb9e0 | Read binary strings/blobs in bulk chunks with a memcpy fast path (#5233) | ||
Ville Vesilehto
|
457bc283ff |
fix(cbor): reject overflowing negative integers (#5039)
* fix(cbor): reject negative ints overflowing int64 CBOR encodes negative integers as "-1 - n" where n is uint64_t. When n > INT64_MAX, casting to int64_t caused undefined behavior and silent data corruption. Large negative values were incorrectly parsed as positive integers (e.g., -9223372036854775809 became 9223372036854775807). Add bounds check for to reject values that exceed int64_t representable range, returning parse_error instead of silently corrupting data. Added regression test cases to verify. Signed-off-by: Ville Vesilehto <ville@vesilehto.fi> * chore: clarify tests Add test for "n=0" case (result=-1) to cover the smallest magnitude boundary. Update comments to explain CBOR 0x3B encoding and why "result=0" is not possible. Clarify that n is an unsigned integer in the formula "result = -1 - n" to help understanding the tests. Signed-off-by: Ville Vesilehto <ville@vesilehto.fi> * fix(cbor): extend overflow checks for other types Extend negative integer overflow detection to all CBOR negative integer cases (0x38, 0x39, 0x3A) for consistency with the existing 0x3B check. Signed-off-by: Ville Vesilehto <ville@vesilehto.fi> --------- Signed-off-by: Ville Vesilehto <ville@vesilehto.fi> |
||
|
Niels Lohmann
|
515d994acb |
📄 adjust year (#5044)
Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
54be9b04f0 | 📄 update REUSE (#4960) | ||
|
Niels Lohmann
|
e02de2f971 |
Update to Doctest 2.4.12 (#4771)
* ⬆️ Doctest 2.4.12 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 🚨 fix warnings Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 🚨 fix warnings Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 🚨 fix warnings Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 🚨 fix warnings Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 🚨 fix warnings Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 🚨 fix warnings Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 🚨 fix warnings Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 🚨 fix warnings Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 🚨 fix warnings Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 🚨 fix warnings Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 🚨 fix warnings Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 🚨 fix warnings Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 🚨 fix warnings Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 🚨 fix warnings Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 🚨 fix warnings Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
1705bfe914 |
🔖 set version to 3.12.0 (#4727)
Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
f06604fce0 |
Bump the copyright years (#4606)
* 📄 bump the copyright years Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 📄 bump the copyright years Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 📄 bump the copyright years Signed-off-by: Niels Lohmann <niels.lohmann@gmail.com> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> Signed-off-by: Niels Lohmann <niels.lohmann@gmail.com> |
||
|
Niels Lohmann
|
6057b31df7 |
Overwork astyle call (#4573)
* 🔧 overwork astyle call * 🔧 overwork astyle call * 🔧 overwork astyle call * 🔧 overwork astyle call * 🔧 overwork astyle call * 🔧 overwork astyle call * 🔧 overwork astyle call * 🔧 overwork astyle call * 🔧 overwork astyle call * 🔧 overwork astyle call * 🔧 overwork astyle call * Use ubuntu-latest image to run Valgrind (#4575) * 🔧 use Clang image to run valgrind * 🔧 use Clang image to run valgrind * 🔧 use Clang image to run valgrind * 🔧 use Ubuntu image to run valgrind * Use Clang image to run iwyu (#4574) * 🔧 use Clang image to run iwyu * 🔧 use Clang image to run iwyu * 🔧 overwork astyle call * 🔧 overwork astyle call * 🔧 overwork astyle call * 🔧 overwork astyle call * 🔧 overwork astyle call * 🔧 overwork astyle call * 🔧 overwork astyle call * 🔧 overwork astyle call * 🔧 overwork astyle call * 🔧 overwork astyle call * 🔧 overwork astyle call * 🎨 format code * 🔨 clean up |
||
Sushrut Shringarputale
|
58f5f25968 |
json start/end position implementation (#4517)
* Add implementation to retrieve start and end positions of json during parse * Add more unit tests and add start/stop parsing for arrays * Add raw value for all types * Add more tests and fix compiler warning * Amalgamate * Fix CLang GCC warnings * Fix error in build * Style using astyle 3.1 * Fix whitespace changes * revert * more whitespace reverts * Address PR comments * Fix failing issues * More whitespace reverts * Address remaining PR comments * Address comments * Switch to using custom base class instead of default basic_json * Adding a basic using for a json using the new base class. Also address PR comments and fix CI failures * Address decltype comments * Diagnostic positions macro (#4) Co-authored-by: Sush Shringarputale <sushring@linux.microsoft.com> * Fix missed include deletion * Add docs and address other PR comments (#5) * Add docs and address other PR comments --------- Co-authored-by: Sush Shringarputale <sushring@linux.microsoft.com> * Address new PR comments and fix CI tests for documentation * Update documentation based on feedback (#6) --------- Co-authored-by: Sush Shringarputale <sushring@linux.microsoft.com> * Address std::size_t and other comments * Fix new CI issues * Fix lcov * Improve lcov case with update to handle_diagnostic_positions call for discarded values * Fix indentation of LCOV_EXCL_STOP comments * fix amalgamation astyle issue --------- Co-authored-by: Sush Shringarputale <sushring@linux.microsoft.com> |
||
|
Niels Lohmann
|
1b9a9d1f21 |
Update licenses (#4521)
* 📄 update licenses * 📄 update licenses |
||
|
Niels Lohmann
|
e3ac5a64bf |
Add test for libstdc++ (#4495)
* 👷 add test for libstdc++ * 👷 add test for libstdc++ * 💚 add fixes from #4490 * 💚 add fixes from #4490 |
||
|
Niels Lohmann
|
1825117e63 |
Another desperate try to fix the CI (#4489)
* 🚨 fix warning * 💚 update actions * 🚨 fix warning * 🚨 fix warning * 🚨 fix warning * 💚 update actions * 💚 update actions * 🚨 fix warning * 🚨 fix warning * 💚 update actions * 🚨 fix warning * 💚 update actions * 💚 update actions * 💚 update actions * 🚨 fix warning * 🚨 fix warning * 🚨 fix warning * 🚨 fix warning * 💚 update actions * 💚 update actions * 🚨 fix warning * 💚 update actions * 💚 update actions * 💚 update actions * 💚 update actions * 💚 update actions |
||
|
Niels Lohmann
|
9cca280a4d | JSON for Modern C++ 3.11.3 (#4222) | ||
|
Niels Lohmann
|
f56c6e2e30 | Update documentation for the next release (#4216) | ||
|
Niels Lohmann
|
1ce29fa22f | Fix CI (#4160) | ||
|
Niels Lohmann
|
836b7beca4 | Fix CI, again (#4083) | ||
Florian Albrechtskirchinger
|
3d1252bbff | Replace limit macros with std::numeric_limits (#3723) | ||
|
Niels Lohmann
|
58bd97e2b1 |
Add clang-tools to required tools for ci_static_analysis_clang (#3724)
* 💚 add clang-tools to required tools for ci_static_analysis_clang * 🚨 update Clang-Tidy warning selection * 🚨 fix Clang-Tidy warnings * 🚨 fix Clang-Tidy warnings * 🚨 fix Clang-Tidy warnings * 🚨 fix Clang-Tidy warnings * 🚨 fix Clang-Tidy warnings * 🚨 fix Clang-Tidy warnings * 🚨 fix Clang-Tidy warnings * 🚨 fix Clang-Tidy warnings * 🚨 fix Clang-Tidy warnings * 🚨 fix Clang-Tidy warnings * 🚨 fix Clang-Tidy warnings (#3738) * ⏪ revert fix * ⏪ revert fix * 🚨 fix Clang-Tidy warnings (#3739) Co-authored-by: Florian Albrechtskirchinger <falbrechtskirchinger@gmail.com> |
||
|
Niels Lohmann
|
9d69186291 | 🔖 set version to 3.11.2 | ||
|
Niels Lohmann
|
f2020da0dd | 🔖 set version to 3.11.1 | ||
|
Niels Lohmann
|
ce0e13ccea | 🔖 set version to 3.11.0 | ||
|
Niels Lohmann
|
feef0eb595 |
Add error message if test suite cannot be found (#3585)
* 🚸 add error message if test suite cannot be found Fixes #3584 |
||
|
Niels Lohmann
|
527da54dcb |
Use REUSE framework (#3546)
* 📄 add licenses * 👷 add REUSE compliance check * 📝 add badge for REUSE Co-authored-by: Florian Albrechtskirchinger <falbrechtskirchinger@gmail.com> |
||
|
Niels Lohmann
|
b21c345179 |
Reorganize directories (#3462)
* 🚚 move files * 🚚 rename doc folder to docs * 🚚 rename test folder to tests |