cc472af13f
* Check the fuzzers' UBJSON/BJData round-trip invariants in the unit tests The strongest correctness checks for the UBJSON and BJData writers lived only in the OSS-Fuzz drivers: anything from_ubjson()/from_bjdata() returns must serialize with every option combination, parse back, and re-serialize stably. Those checks only run at OSS-Fuzz, so regressions surfaced days later as external reports - the same BJData assert pair was reported five times over three years, and #5494's harness change was followed by OSS-Fuzz 563659413 within a day. Add "UBJSON round-trip invariants" and "BJData round-trip invariants" test cases that run the drivers' checks on a fixed, deterministic corpus (tests/src/round_trip_corpus.hpp): integer and float boundaries, non-finite numbers, strings, binary values, optimized containers, deep nesting, the JData annotated-array matrix, and seeded random containers. They also check two properties the drivers do not: the first round trip preserves the value, and re-serializing reproduces the exact bytes. For BJData both exclude values containing a binary value, which is read back as an array of integers unless it was written as a Draft 3 optimized binary array; this carve-out is now documented in bjdata.md. Run against the headers before #5542, the BJData test fails, including on the shape from OSS-Fuzz 563659413. Also document how OSS-Fuzz reports are handled (reference them as "OSS-Fuzz: <id>", turn the reproducer into a unit test, keep drivers and unit tests in sync) in tests/fuzzing.md, and link it from the PR template and the quality assurance page. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Add the OSS-Fuzz reproducers for 474400817 and 474480402 as unit tests Following the convention added to tests/fuzzing.md, the reproducers of the two BJData fuzzer asserts tracked since January are now unit tests: - 474400817 (assert(false)): an empty object _ArraySize_ was written as the ND-array header length, which from_bjdata() could not read back. Fixed by #5455. - 474480402 (to_bjdata(j2, false, false) == vec2): a one-byte Draft 3 binary array is written in Draft 2 mode as a uint8 array and then re-serialized with the int8 marker. This is the documented exception to byte stability, not a library bug; OSS-Fuzz closed it after #5494 relaxed the harness to value stability. The test pins the exact bytes so the exception stays deliberate. The 563659413 reproducer is already a unit test (#5542). A comment also ties the existing UBJSON excessive-count test to the timeout OSS-Fuzz reported for that shape (testcase 6347769435193344). OSS-Fuzz: 474400817 OSS-Fuzz: 474480402 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix GCC -Weffc++ and -Wuseless-cast warnings in the round-trip corpus Initialize the atoms in the member initialization list, and drop the cast of the generator's result, which already is std::size_t on 64-bit Linux. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me>
4.9 KiB
Fuzz testing
Each parser of the library (JSON, BJData, BSON, CBOR, MessagePack, and UBJSON) can be fuzz tested. Currently, libFuzzer and afl++ are supported.
Corpus creation
For most effective fuzzing, a corpus should be provided. A corpus is a directory with some simple input files that cover several features of the parser and is hence a good starting point for mutations.
TEST_DATA_VERSION=3.1.0
wget https://github.com/nlohmann/json_test_data/archive/refs/tags/v$TEST_DATA_VERSION.zip
unzip v$TEST_DATA_VERSION.zip
rm v$TEST_DATA_VERSION.zip
for FORMAT in json bjdata bson cbor msgpack ubjson
do
rm -fr corpus_$FORMAT
mkdir corpus_$FORMAT
find json_test_data-$TEST_DATA_VERSION -size -5k -name "*.$FORMAT" -exec cp "{}" "corpus_$FORMAT" \;
done
rm -fr json_test_data-$TEST_DATA_VERSION
The generated corpus can be used with both libFuzzer and afl++. The remainder of this documentation assumes the corpus
directories have been created in the tests directory.
libFuzzer
To use libFuzzer, you need to pass -fsanitize=fuzzer as FUZZER_ENGINE. In the tests directory, call
make fuzzers FUZZER_ENGINE="-fsanitize=fuzzer"
This creates a fuzz tester binary for each parser that supports these command line options.
In case your default compiler is not a Clang compiler that includes libFuzzer (Clang 6.0 or later), you need to set the
CXX variable accordingly. Note the compiler provided by Xcode (AppleClang) does not contain libFuzzer. Please install
Clang via Homebrew calling brew install llvm and add CXX=$(brew --prefix llvm)/bin/clang to the make call:
make fuzzers FUZZER_ENGINE="-fsanitize=fuzzer" CXX=$(brew --prefix llvm)/bin/clang
Then pass the corpus directory as command-line argument (assuming it is located in tests):
./parse_cbor_fuzzer corpus_cbor
The fuzzer should be able to run indefinitely without crashing. In case of a crash, the tested input is dumped into
a file starting with crash-.
afl++
To use afl++, you need to pass -fsanitize=fuzzer as FUZZER_ENGINE. It will be replaced by a libAFLDriver.a to
re-use the same code written for libFuzzer with afl++. Furthermore, set afl-clang-fast++ as compiler.
CXX=afl-clang-fast++ make fuzzers FUZZER_ENGINE="-fsanitize=fuzzer"
Then the fuzzer is called like this in the tests directory:
afl-fuzz -i corpus_cbor -o out -- ./parse_cbor_fuzzer
The fuzzer should be able to run indefinitely without crashing. In case of a crash, the tested input is written to the
directory out.
OSS-Fuzz
The library is further fuzz-tested 24/7 by Google's OSS-Fuzz project. It uses
the same fuzzers target as above and also relies on the FUZZER_ENGINE variable. See the used
build script for more information.
In case the build at OSS-Fuzz fails, an issue will be created automatically.
Handling OSS-Fuzz reports
OSS-Fuzz files the crashes it finds in its own issue tracker, not on GitHub. So that each report can be traced to the change that fixed it, and each fix to the report it answers, fixes follow these conventions:
- Reference the OSS-Fuzz issue in the pull request, next to any GitHub issue it closes, as
OSS-Fuzz: <id>(for example,OSS-Fuzz: 563659413), and in the commit message. The ID alone does not disclose the crash. If the report was triaged into a GitHub issue, link the OSS-Fuzz issue there too. - Turn the reproducer into a unit test. Download the testcase from the OSS-Fuzz report, reduce it if possible, and
add it as a regression test to the unit test of the affected format (e.g.,
tests/src/unit-bjdata.cpp), with a comment naming the OSS-Fuzz issue. This way the input is checked by every CI run rather than only by OSS-Fuzz, and it stays covered even if OSS-Fuzz later closes the report as not reproducible. - Keep the fuzzer drivers and the unit tests in sync. The round-trip checks of the UBJSON and BJData drivers are
also run on a fixed corpus in the unit tests (see
tests/src/round_trip_corpus.hppand the "round-trip invariants" test cases), so a regression shows up in CI first. When a driver's checks change, change the unit tests with them. - Record in the report whether the bug shipped. OSS-Fuzz asks whether a crash was a short-lived regression or affects a released version; answer it when the fix is merged, as it decides whether the fix needs a release note or a security advisory (see the security policy).
After the fix is merged, OSS-Fuzz re-runs the reproducer on its next build and marks the report as verified and closed. If it does not, the fix is incomplete.