diff --git a/docs/mkdocs/docs/api/basic_json/cbor_tag_handler_t.md b/docs/mkdocs/docs/api/basic_json/cbor_tag_handler_t.md index 55a7954d5..528fb199e 100644 --- a/docs/mkdocs/docs/api/basic_json/cbor_tag_handler_t.md +++ b/docs/mkdocs/docs/api/basic_json/cbor_tag_handler_t.md @@ -15,11 +15,13 @@ error : report a parse error in case of a tag (the `from_cbor` overloads throw a `parse_error` exception by default) ignore -: ignore tags +: ignore tags (also in front of a text string map key) store : store tagged byte strings (for bytes 0xd8..0xdb) as binary values with the tag as subtype; other tagged values are - read as if the tag were ignored. If several tags precede a byte string, only the innermost one is stored. + read as if the tag were ignored. If several tags precede a byte string, only the tag directly in front of it is + stored, and only when it has the form 0xd8..0xdb (a tag 0..23 written as 0xc0..0xd7 is skipped). Tags in front of a + text string map key are skipped. ## Examples diff --git a/docs/mkdocs/docs/api/basic_json/from_cbor.md b/docs/mkdocs/docs/api/basic_json/from_cbor.md index 4fcd7fe98..1c14d550d 100644 --- a/docs/mkdocs/docs/api/basic_json/from_cbor.md +++ b/docs/mkdocs/docs/api/basic_json/from_cbor.md @@ -32,8 +32,10 @@ The exact mapping and its limitations are described on a [dedicated page](../../ - an `std::istream` object - a `FILE` pointer - - a C-style array of characters - - a pointer to a null-terminated string of single byte characters + - a C-style array of characters (its terminating NUL is read as trailing data: `parse_error.110` unless `strict` is + `false`) + - a pointer to a null-terminated string of single byte characters (read up to the first 0x00 byte, which is a valid + CBOR byte) - a container `obj` for which `begin(obj)` and `end(obj)` produce a valid pair of iterators (as found via ADL or member functions, with semantics compatible to `std::begin` and `std::end`) @@ -89,8 +91,14 @@ Strong guarantee: if an exception is thrown, there are no changes in the JSON va - Throws [parse_error.112](../../home/exceptions.md#jsonexceptionparse_error112) if unsupported features from CBOR were used in the given input or if the input is not valid CBOR - Throws [parse_error.113](../../home/exceptions.md#jsonexceptionparse_error113) if a map key is not a string (keys of - other types are not supported, as JSON object keys are always strings), or if a string value or object key is not - valid UTF-8 and `error_handler` is `strict` + other types are not supported, as JSON object keys are always strings), if the head of a string or byte string is + malformed (including a chunk of the wrong type or an indefinite-length string nested inside an indefinite-length + string), or if a string value or object key (or one chunk of an indefinite-length string) is not valid UTF-8 and + `error_handler` is `strict` +- Throws [out_of_range.408](../../home/exceptions.md#jsonexceptionout_of_range408) if an array or map announces + a length that `std::size_t` cannot hold, or that equals `std::size_t(-1)` (for example `9B FF FF FF FF FF FF FF FF`) +- Throws [out_of_range.406](../../home/exceptions.md#jsonexceptionout_of_range406) if a floating-point number does not + fit into `number_float_t` ## Complexity diff --git a/docs/mkdocs/docs/api/basic_json/to_cbor.md b/docs/mkdocs/docs/api/basic_json/to_cbor.md index 6508864e1..f0be593f0 100644 --- a/docs/mkdocs/docs/api/basic_json/to_cbor.md +++ b/docs/mkdocs/docs/api/basic_json/to_cbor.md @@ -42,7 +42,8 @@ The exact mapping and its limitations are described on a [dedicated page](../../ ## Exception safety -Strong guarantee: if an exception is thrown, there are no changes in the JSON value. +Strong guarantee: if an exception is thrown, there are no changes in the JSON value. Bytes that were already written to +the output adapter are not taken back. ## Exceptions diff --git a/docs/mkdocs/docs/features/binary_formats/cbor.md b/docs/mkdocs/docs/features/binary_formats/cbor.md index 8933630da..35d72d6a1 100644 --- a/docs/mkdocs/docs/features/binary_formats/cbor.md +++ b/docs/mkdocs/docs/features/binary_formats/cbor.md @@ -11,7 +11,7 @@ extremely small code sizes, fairly small message size, and extensibility without ## Byte layout -CBOR encodes every data item as a *head* followed by optional content. The first byte of the head holds the *major type* in its upper three bits (unsigned integer, negative integer, byte string, text string, array, map, tag, or simple/float) and an *additional information* value in its lower five bits. Additional information below 24 is the argument itself (for example, a small integer or a short length); the values 24 to 27 announce that the argument follows in 1, 2, 4, or 8 big-endian bytes. Strings, arrays, and maps are prefixed with their length (or count of elements/pairs), and content follows directly, so a reader always knows how many items to expect without a terminator. +CBOR encodes every data item as a *head* followed by optional content. The first byte of the head holds the *major type* in its upper three bits (unsigned integer, negative integer, byte string, text string, array, map, tag, or simple/float) and an *additional information* value in its lower five bits. Additional information below 24 is the argument itself (for example, a small integer or a short length); the values 24 to 27 announce that the argument follows in 1, 2, 4, or 8 big-endian bytes. Strings, arrays, and maps are prefixed with their length (or count of elements/pairs), and content follows directly. A reader of definite-length items knows how many items to expect without a terminator; indefinite-length items (additional information 31) are instead closed by the "break" byte 0xFF, which `from_cbor` reads but `to_cbor` never writes. Consider the following JSON value (shown formatted for readability): @@ -52,16 +52,16 @@ The library uses the following mapping from JSON values types to CBOR types acco | null | `null` | Null | 0xF6 | | boolean | `true` | True | 0xF5 | | boolean | `false` | False | 0xF4 | -| number_integer | -9223372036854775808..-2147483649 | Negative integer (8 bytes follow) | 0x3B | -| number_integer | -2147483648..-32769 | Negative integer (4 bytes follow) | 0x3A | -| number_integer | -32768..-129 | Negative integer (2 bytes follow) | 0x39 | -| number_integer | -128..-25 | Negative integer (1 byte follow) | 0x38 | +| number_integer | -9223372036854775808..-4294967297 | Negative integer (8 bytes follow) | 0x3B | +| number_integer | -4294967296..-65537 | Negative integer (4 bytes follow) | 0x3A | +| number_integer | -65536..-257 | Negative integer (2 bytes follow) | 0x39 | +| number_integer | -256..-25 | Negative integer (1 byte follow) | 0x38 | | number_integer | -24..-1 | Negative integer | 0x20..0x37 | | number_integer | 0..23 | Integer | 0x00..0x17 | | number_integer | 24..255 | Unsigned integer (1 byte follow) | 0x18 | | number_integer | 256..65535 | Unsigned integer (2 bytes follow) | 0x19 | | number_integer | 65536..4294967295 | Unsigned integer (4 bytes follow) | 0x1A | -| number_integer | 4294967296..18446744073709551615 | Unsigned integer (8 bytes follow) | 0x1B | +| number_integer | 4294967296..9223372036854775807 | Unsigned integer (8 bytes follow) | 0x1B | | number_unsigned | 0..23 | Integer | 0x00..0x17 | | number_unsigned | 24..255 | Unsigned integer (1 byte follow) | 0x18 | | number_unsigned | 256..65535 | Unsigned integer (2 bytes follow) | 0x19 | @@ -95,7 +95,24 @@ see "binary" cells in the table above. !!! success "Complete mapping" - The mapping is **complete** in the sense that any JSON value type can be converted to a CBOR value. + The mapping is **complete** in the sense that any JSON value type can be converted to a CBOR value, with one + exception: an integer outside the range of CBOR integers (only possible with a number type wider than 64 bits) + throws [`out_of_range.407`](../../home/exceptions.md#jsonexceptionout_of_range407). + +!!! info "Not a deterministic encoding" + + `to_cbor` does not produce the deterministic encoding of + [RFC 8949, Section 4.2.1](https://www.rfc-editor.org/rfc/rfc8949.html#section-4.2.1), and not always the preferred + serialization of Section 4.1: + + - floating-point numbers are written as single precision (0xFA) when that is exact and as double precision (0xFB) + otherwise; half precision (0xF9) is used only for `NaN` and `Infinity`/`-Infinity` (`1.0` is `fa 3f 80 00 00`); + - a binary subtype 0..23 is written as `d8 xx` instead of the one-byte tag 0xC0..0xD7 (`d8 01 41 01`); + - object members are written in the order of `object_t` (for the default `std::map`: sorted by the key string, not + by the encoded key bytes). + + Integers and lengths use the shortest form, and indefinite-length items are never written. Decoders must accept all + of this; the RFC does not require the preferred serialization. !!! info "NaN/infinity handling" @@ -105,9 +122,8 @@ see "binary" cells in the table above. !!! note - Prior to version 3.13.0, NaN and Infinity were instead serialized as a CBOR double-precision float (type 0xFB, - 9 bytes total), because the check used to select a smaller encoding compared magnitudes with NaN, which is - always `false` and caused the intended half-precision path to be skipped. + Prior to version 3.8.0, NaN and Infinity were instead serialized as a CBOR double-precision float (type 0xFB, + 9 bytes total). !!! info "Unused CBOR types" @@ -124,7 +140,6 @@ see "binary" cells in the table above. - expected conversions (0xD5..0xD7) - simple values (0xE0..0xF3, 0xF8) - undefined (0xF7) - - half-precision floats (0xF9) - break (0xFF) !!! info "Tagged items" @@ -226,10 +241,20 @@ string of the same major type, as required by [RFC 8949, Section 3.2.3](https:// ([RFC 8392](https://www.rfc-editor.org/rfc/rfc8392.html)), cannot be read with this library and need a general-purpose CBOR library instead. + A tag in front of a text string key is handled like a tag in front of a value: it is skipped with + `cbor_tag_handler_t::ignore` or `cbor_tag_handler_t::store` and rejected with the default `error`. A tag does not + turn a key of another type into a text string. + + A map with duplicate keys is well-formed, but invalid under + [RFC 8949, Section 5.3.1](https://www.rfc-editor.org/rfc/rfc8949.html#section-5.3.1). `from_cbor` accepts it and the + last value wins; the [SAX interface](../parsing/sax_interface.md) reports every occurrence. + !!! warning "Ill-formed UTF-8 in text strings" [RFC 8949, Section 3.1](https://www.rfc-editor.org/rfc/rfc8949.html#section-3.1) requires CBOR text strings (major - type 3) to be valid UTF-8, but leaves it up to the decoder whether to enforce this, so checking is opt-in: with the + type 3) to be valid UTF-8 (a string with ill-formed UTF-8 is "well-formed but invalid"), and + [Section 5.3.1](https://www.rfc-editor.org/rfc/rfc8949.html#section-5.3.1) leaves it up to the decoder whether to + verify this, so checking is opt-in: with the [`error_handler`](../../api/basic_json/from_cbor.md) parameter left at `keep` (the default), `from_cbor()` accepts a text string (object keys included) whose bytes are not valid UTF-8 and hands them back unchanged. Passing `error_handler_t::strict` makes `from_cbor()` check and throw @@ -242,9 +267,14 @@ string of the same major type, as required by [RFC 8949, Section 3.2.3](https:// [`JSON_STRICT_BINARY_UTF8`](../../api/macros/json_strict_binary_utf8.md) is enabled), it throws the same exception instead. Byte strings (major type 2) are unaffected, since they are not required to hold text. + The chunks of an indefinite-length text string are checked one by one, as required by + [RFC 8949, Section 3.2.3](https://www.rfc-editor.org/rfc/rfc8949.html#section-3.2.3): a code point cannot be spread + over two chunks. With `error_handler_t::strict` such a string throws `parse_error.113`; `replace` and `ignore` + sanitize each chunk on its own. + !!! warning "Tagged items" - Tagged items (0xC0..0xDB) will throw a parse error by default. They can be ignored by passing `cbor_tag_handler_t::ignore` to function `from_cbor`, in which case the tag is skipped and the enclosed data item is parsed on its own. Passing `cbor_tag_handler_t::store` to function `from_cbor` stores tagged byte strings (for bytes 0xd8..0xdb) as binary values with the tag as subtype; other tagged values are read as if the tag were ignored. If several tags precede a byte string, only the innermost one is stored. Note that no tag is ever interpreted: for instance, a text string tagged with tag 0 (date/time) stays a string. + Tagged items (0xC0..0xDB) will throw a parse error by default. They can be ignored by passing `cbor_tag_handler_t::ignore` to function `from_cbor`, in which case the tag is skipped and the enclosed data item is parsed on its own. Passing `cbor_tag_handler_t::store` to function `from_cbor` stores tagged byte strings (for bytes 0xd8..0xdb) as binary values with the tag as subtype; other tagged values are read as if the tag were ignored. If several tags precede a byte string, only the tag directly in front of it is stored, and only when it has the form 0xD8..0xDB (a tag 0..23 written as 0xC0..0xD7 is skipped); for example, `D8 05 C1 41 AA` yields subtype `null`, whereas `C1 D8 05 41 AA` yields 5. Tags in front of a text string map key are skipped as well. Note that no tag is ever interpreted: for instance, a text string tagged with tag 0 (date/time) stays a string. ??? example "Example: deserialize a JSON value from CBOR" diff --git a/docs/mkdocs/docs/features/binary_values.md b/docs/mkdocs/docs/features/binary_values.md index b4c3bab6d..dd683f469 100644 --- a/docs/mkdocs/docs/features/binary_values.md +++ b/docs/mkdocs/docs/features/binary_values.md @@ -301,6 +301,8 @@ representation using the length of the byte array. [`json::from_cbor`](../api/basic_json/from_cbor.md) (see [`cbor_tag_handler_t`](../api/basic_json/cbor_tag_handler_t.md)). + With `ignore`, the subtype is dropped (`"subtype": null`); with `store`, the tag is kept (`"subtype": 42`): + ```json { "binary": { @@ -310,6 +312,15 @@ representation using the length of the byte array. } ``` + ```json + { + "binary": { + "bytes": [202, 254, 186, 190], + "subtype": 42 + } + } + ``` + ### MessagePack [MessagePack](binary_formats/messagepack.md) supports binary values and subtypes. If a subtype is given, the ext family diff --git a/docs/mkdocs/docs/features/types/template_parameters.md b/docs/mkdocs/docs/features/types/template_parameters.md index 29329fea8..cc079cd72 100644 --- a/docs/mkdocs/docs/features/types/template_parameters.md +++ b/docs/mkdocs/docs/features/types/template_parameters.md @@ -368,8 +368,7 @@ using array_t = ArrayType>; `#!cpp std::from_chars` is not available or declines the token, and `std::strtold` reads up to the null character. A type whose `data()` is not null-terminated does not fail to compile; with such a `NumberFloatType` it can silently misparse floating-point numbers. -- `append(const char*, size_type)`, used by [`dump`](../../api/basic_json/dump.md), and `append(const StringType&)`, - used by the CBOR reader for indefinite-length strings. The library's internal string concatenation additionally has +- `append(const char*, size_type)`, used by [`dump`](../../api/basic_json/dump.md) and by the binary readers. The library's internal string concatenation additionally has to append a `#!cpp char` and a `#!cpp const char*`; for each it selects between `append(arg)`, `#!cpp operator+=`, `append(first, last)`, and `append(data, size)`. - The comparison operator `==` against another `StringType`, and `<` for use as a key of the chosen @@ -679,8 +678,7 @@ using binary_t = nlohmann::byte_container_with_subtype; [`StringType`](#stringtype), whose non-`#!cpp const` `data()` is C++17 and later only. - Default-constructible, copy-constructible, and move-constructible. - Member functions `size()`, `empty()`, `data()`, `resize()`, `operator[]`, `back()`, `begin()`, `end()`, `cbegin()`, - and `cend()` with random-access iterators, and `insert(pos, first, last)`, which the CBOR reader uses to join the - chunks of an indefinite-length byte string. `push_back()` is **not** required. + and `cend()` with random-access iterators. `push_back()` is **not** required. - Comparison operators: `==` is used by [`byte_container_with_subtype`](../../api/byte_container_with_subtype/index.md), the relational operators by [`basic_json`'s comparison operators](../../api/basic_json/operator_le.md). diff --git a/include/nlohmann/detail/input/binary_reader.hpp b/include/nlohmann/detail/input/binary_reader.hpp index d3c3d5bdc..929ac7573 100644 --- a/include/nlohmann/detail/input/binary_reader.hpp +++ b/include/nlohmann/detail/input/binary_reader.hpp @@ -1218,6 +1218,10 @@ class binary_reader // level as required by RFC 8949, Section 3.2.3 bool indefinite = false; + // the chunk currently being validated; only used for the chunks of an + // indefinite-length string when the UTF-8 check can do something + string_t chunk; + while (true) { if (JSON_HEDLEY_UNLIKELY(!unexpect_eof("string"))) @@ -1237,26 +1241,82 @@ class binary_reader } // a break marker closes the indefinite-length string; outside - // of one it falls through to the error below + // of one it falls through to the error below. Every chunk has + // been checked on its own, so there is nothing left to check. if (indefinite && current == 0xFF) { - return check_string_utf8(result, context); - } - - if (JSON_HEDLEY_UNLIKELY(!get_cbor_string_chunk(result, indefinite))) - { - return false; + return true; } if (!indefinite) { - return check_string_utf8(result, context); + return get_cbor_string_chunk(result, false) && check_string_utf8(result, context); + } + + if (error_handler == error_handler_t::keep) + { + // nothing to check: read the chunk directly into the result + if (JSON_HEDLEY_UNLIKELY(!get_cbor_string_chunk(result, true))) + { + return false; + } + } + else + { + // RFC 8949, Section 3.2.3: each chunk is a text string of its + // own and must be valid UTF-8 by itself, i.e., a code point + // cannot be spread over two chunks + chunk.clear(); + if (JSON_HEDLEY_UNLIKELY(!get_cbor_string_chunk(chunk, true) || !check_string_utf8(chunk, context))) + { + return false; + } + result.append(chunk.data(), chunk.size()); } get(); } } + /*! + @brief skips the tags in front of a CBOR object key + + A tag may wrap any data item, so it may also wrap a key. Like in front of a + value, the tags are skipped for the handlers ignore and store; with the + handler error they are left to @ref get_cbor_object_key, which rejects the + key. The tag loop is iterative, so the number of tags is not limited by the + recursion depth. + + @param[in] tag_handler how CBOR tags should be treated + + @return whether the tags could be skipped (false on EOF in a tag argument) + + @pre @a current is the first byte of the key + */ + bool skip_cbor_key_tags(const cbor_tag_handler_t tag_handler) + { + if (tag_handler == cbor_tag_handler_t::error) + { + return true; + } + + // 0xC0..0xDB: tag with the tag number in the head (0xC0..0xD7) or in + // the 1, 2, 4, or 8 bytes that follow (0xD8..0xDB) + while (current >= 0xC0 && current <= 0xDB) + { + if (current >= 0xD8) + { + std::uint64_t tag_to_ignore{}; + if (JSON_HEDLEY_UNLIKELY(!get_cbor_argument(tag_to_ignore))) + { + return false; + } + } + get(); + } + return true; + } + /*! @brief reads a CBOR object key @@ -1623,6 +1683,10 @@ class binary_reader if (top.is_object) { + if (JSON_HEDLEY_UNLIKELY(!skip_cbor_key_tags(tag_handler))) + { + return false; + } if (JSON_HEDLEY_UNLIKELY(!read_object_key<&binary_reader::get_cbor_object_key>(key))) { return false; @@ -4130,9 +4194,9 @@ class binary_reader // sequence), UBJSON, BJData, or BSON requires a decoder to reject // ill-formed UTF-8. Checking (and, with @ref error_handler_t::strict, // rejecting, or with `replace`/`ignore`, sanitizing) is opt-in via - // @ref error_handler, applied once the whole string (all chunks of - // an indefinite-length CBOR string included) has been assembled, by - // @ref check_string_utf8 at the call site. + // @ref error_handler, applied by @ref check_string_utf8 at the call + // site to the whole string or, for an indefinite-length CBOR string, + // to each chunk on its own (RFC 8949, Section 3.2.3). return get_bytes(len, "string", result); } diff --git a/single_include/nlohmann/json.hpp b/single_include/nlohmann/json.hpp index f0b788c9e..5db064c63 100644 --- a/single_include/nlohmann/json.hpp +++ b/single_include/nlohmann/json.hpp @@ -18685,6 +18685,10 @@ class binary_reader // level as required by RFC 8949, Section 3.2.3 bool indefinite = false; + // the chunk currently being validated; only used for the chunks of an + // indefinite-length string when the UTF-8 check can do something + string_t chunk; + while (true) { if (JSON_HEDLEY_UNLIKELY(!unexpect_eof("string"))) @@ -18704,26 +18708,82 @@ class binary_reader } // a break marker closes the indefinite-length string; outside - // of one it falls through to the error below + // of one it falls through to the error below. Every chunk has + // been checked on its own, so there is nothing left to check. if (indefinite && current == 0xFF) { - return check_string_utf8(result, context); - } - - if (JSON_HEDLEY_UNLIKELY(!get_cbor_string_chunk(result, indefinite))) - { - return false; + return true; } if (!indefinite) { - return check_string_utf8(result, context); + return get_cbor_string_chunk(result, false) && check_string_utf8(result, context); + } + + if (error_handler == error_handler_t::keep) + { + // nothing to check: read the chunk directly into the result + if (JSON_HEDLEY_UNLIKELY(!get_cbor_string_chunk(result, true))) + { + return false; + } + } + else + { + // RFC 8949, Section 3.2.3: each chunk is a text string of its + // own and must be valid UTF-8 by itself, i.e., a code point + // cannot be spread over two chunks + chunk.clear(); + if (JSON_HEDLEY_UNLIKELY(!get_cbor_string_chunk(chunk, true) || !check_string_utf8(chunk, context))) + { + return false; + } + result.append(chunk.data(), chunk.size()); } get(); } } + /*! + @brief skips the tags in front of a CBOR object key + + A tag may wrap any data item, so it may also wrap a key. Like in front of a + value, the tags are skipped for the handlers ignore and store; with the + handler error they are left to @ref get_cbor_object_key, which rejects the + key. The tag loop is iterative, so the number of tags is not limited by the + recursion depth. + + @param[in] tag_handler how CBOR tags should be treated + + @return whether the tags could be skipped (false on EOF in a tag argument) + + @pre @a current is the first byte of the key + */ + bool skip_cbor_key_tags(const cbor_tag_handler_t tag_handler) + { + if (tag_handler == cbor_tag_handler_t::error) + { + return true; + } + + // 0xC0..0xDB: tag with the tag number in the head (0xC0..0xD7) or in + // the 1, 2, 4, or 8 bytes that follow (0xD8..0xDB) + while (current >= 0xC0 && current <= 0xDB) + { + if (current >= 0xD8) + { + std::uint64_t tag_to_ignore{}; + if (JSON_HEDLEY_UNLIKELY(!get_cbor_argument(tag_to_ignore))) + { + return false; + } + } + get(); + } + return true; + } + /*! @brief reads a CBOR object key @@ -19090,6 +19150,10 @@ class binary_reader if (top.is_object) { + if (JSON_HEDLEY_UNLIKELY(!skip_cbor_key_tags(tag_handler))) + { + return false; + } if (JSON_HEDLEY_UNLIKELY(!read_object_key<&binary_reader::get_cbor_object_key>(key))) { return false; @@ -21597,9 +21661,9 @@ class binary_reader // sequence), UBJSON, BJData, or BSON requires a decoder to reject // ill-formed UTF-8. Checking (and, with @ref error_handler_t::strict, // rejecting, or with `replace`/`ignore`, sanitizing) is opt-in via - // @ref error_handler, applied once the whole string (all chunks of - // an indefinite-length CBOR string included) has been assembled, by - // @ref check_string_utf8 at the call site. + // @ref error_handler, applied by @ref check_string_utf8 at the call + // site to the whole string or, for an indefinite-length CBOR string, + // to each chunk on its own (RFC 8949, Section 3.2.3). return get_bytes(len, "string", result); } diff --git a/tests/src/unit-cbor.cpp b/tests/src/unit-cbor.cpp index 9ee371495..44cfa7723 100644 --- a/tests/src/unit-cbor.cpp +++ b/tests/src/unit-cbor.cpp @@ -1807,6 +1807,93 @@ TEST_CASE("CBOR") CHECK(json::from_cbor(std::vector({0xa1, 0xff, 0x01}), true, false).is_discarded()); } + SECTION("tag in front of a key") + { + const auto ignore = json::cbor_tag_handler_t::ignore; + const auto store = json::cbor_tag_handler_t::store; + const json expected = {{"a", 10}}; + + // the tags are skipped in front of a text string key, like in + // front of a value + const std::vector> tagged = + { + {0xA1, 0xC0, 0x61, 0x61, 0x0A}, // tag in the head + {0xA1, 0xD8, 0x05, 0x61, 0x61, 0x0A}, // 1-byte tag number + {0xA1, 0xD9, 0x01, 0x2C, 0x61, 0x61, 0x0A}, // 2-byte tag number + {0xA1, 0xDA, 0x00, 0x01, 0x00, 0x00, 0x61, 0x61, 0x0A}, // 4-byte tag number + {0xA1, 0xDB, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x00, 0x61, 0x61, 0x0A}, // 8-byte tag number + {0xA1, 0xC0, 0xD8, 0x05, 0xC1, 0x61, 0x61, 0x0A}, // chained tags + {0xBF, 0xC0, 0x61, 0x61, 0x0A, 0xFF}, // indefinite-length map + {0xBF, 0x61, 0x62, 0x01, 0xC0, 0x61, 0x61, 0x0A, 0xFF}, // second key tagged + {0xA1, 0xC0, 0x7F, 0x61, 0x61, 0xFF, 0x0A}, // indefinite-length text key + {0xA1, 0xC0, 0x61, 0x61, 0xC0, 0x0A}, // tagged key and value + }; + + for (const auto& input : tagged) + { + CAPTURE(input) + json const j_ignore = json::from_cbor(input, true, true, ignore); + json const j_store = json::from_cbor(input, true, true, store); + CHECK(j_ignore.at("a") == 10); + CHECK(j_store.at("a") == 10); + CHECK(j_ignore == j_store); + } + CHECK(json::from_cbor(tagged[0], true, true, ignore) == expected); + CHECK(json::from_cbor(tagged[1], true, true, store) == expected); + + // nested maps + CHECK(json::from_cbor(std::vector({0xA1, 0xC0, 0x61, 0x61, 0xA1, 0xD8, 0x05, 0x61, 0x62, 0x01}), true, true, ignore) == json{{"a", {{"b", 1}}}}); + + // the tag handler error still rejects every tag + for (const auto& input : tagged) + { + CAPTURE(input) + json _; + CHECK_THROWS_AS(_ = json::from_cbor(input), json::parse_error&); + } + { + json _; + CHECK_THROWS_WITH_AS(_ = json::from_cbor(std::vector({0xA1, 0xD8, 0x05, 0x61, 0x61, 0x0A})), "[json.exception.parse_error.113] parse error at byte 2: syntax error while parsing CBOR object key: only string keys are supported, but found a tag; last byte: 0xD8", json::parse_error&); + } + + // a tag does not turn a key of another type into a text string + { + json _; + CHECK_THROWS_WITH_AS(_ = json::from_cbor(std::vector({0xA1, 0xC0, 0x01, 0x01}), true, true, ignore), "[json.exception.parse_error.113] parse error at byte 3: syntax error while parsing CBOR object key: only string keys are supported, but found an unsigned integer; last byte: 0x01", json::parse_error&); + CHECK_THROWS_WITH_AS(_ = json::from_cbor(std::vector({0xA1, 0xD8, 0x05, 0x41, 0x61, 0x01}), true, true, store), "[json.exception.parse_error.113] parse error at byte 4: syntax error while parsing CBOR object key: only string keys are supported, but found a byte string; last byte: 0x41", json::parse_error&); + CHECK(json::from_cbor(std::vector({0xA1, 0xC0, 0x01, 0x01}), true, false, ignore).is_discarded()); + } + + // end of input after the tag, or inside the tag number + { + json _; + CHECK_THROWS_WITH_AS(_ = json::from_cbor(std::vector({0xA1, 0xC0}), true, true, ignore), "[json.exception.parse_error.110] parse error at byte 3: syntax error while parsing CBOR string: unexpected end of input", json::parse_error&); + CHECK_THROWS_AS(_ = json::from_cbor(std::vector({0xA1, 0xD8}), true, true, ignore), json::parse_error&); + CHECK_THROWS_AS(_ = json::from_cbor(std::vector({0xA1, 0xDB, 0x00, 0x00}), true, true, store), json::parse_error&); + CHECK_THROWS_AS(_ = json::from_cbor(std::vector({0xBF, 0xC0}), true, true, store), json::parse_error&); + } + + // SAX: the key event is reported without the tag (start_object, + // key, number_unsigned, end_object are four events) + { + const std::vector input = {0xA1, 0xC0, 0x61, 0x61, 0x0A}; + SaxCountdown four(4); + CHECK(json::sax_parse(input, &four, json::input_format_t::cbor, true, false, false, ignore)); + SaxCountdown three(3); + CHECK(!json::sax_parse(input, &three, json::input_format_t::cbor, true, false, false, ignore)); + SaxCountdown error_handler(4); + CHECK(!json::sax_parse(input, &error_handler, json::input_format_t::cbor)); + } + + // a long chain of tags needs no recursion + { + std::vector input = {0xA1}; + input.insert(input.end(), 100000, 0xC0); + input.insert(input.end(), {0x61, 0x61, 0x0A}); + CHECK(json::from_cbor(input, true, true, ignore) == expected); + } + } + SECTION("non-string key (see #2766 and #3381)") { // only text strings map to JSON object keys; any other key is @@ -1915,9 +2002,10 @@ TEST_CASE("CBOR") { json _; - // the chunks are concatenated as is, without checking that each - // chunk is valid UTF-8 on its own (RFC 8949, Section 3.2.3), so - // a code point split across two chunks yields a valid string + // with the default error handler keep, the chunks are concatenated + // as is, without checking that each chunk is valid UTF-8 on its + // own (RFC 8949, Section 3.2.3), so a code point split across two + // chunks yields a valid string CHECK_NOTHROW(_ = json::from_cbor(std::vector({0x7f, 0x61, 0xc3, 0x61, 0xa9, 0xff}))); CHECK(_ == "\xc3\xa9"); CHECK(_.dump() == "\"\xc3\xa9\""); @@ -1937,6 +2025,63 @@ TEST_CASE("CBOR") CHECK(json::from_cbor(std::vector({0x7f, 0x62, 0xc3, 0xa9, 0x62, 0xc3, 0xb6, 0xff})) == "\xc3\xa9\xc3\xb6"); } + SECTION("ill-formed UTF-8 in a chunk of an indefinite-length string (RFC 8949, Section 3.2.3)") + { + const auto strict = json::error_handler_t::strict; + const auto replace = json::error_handler_t::replace; + const auto ignore = json::error_handler_t::ignore; + const auto keep = json::error_handler_t::keep; + const auto tags = json::cbor_tag_handler_t::error; + + // a code point split over two chunks + const std::vector split = {0x7f, 0x61, 0xc3, 0x61, 0xa9, 0xff}; + // a 4-byte code point split over two chunks + const std::vector split_emoji = {0x7f, 0x62, 0xf0, 0x9f, 0x62, 0x98, 0x80, 0xff}; + // the same in an object key + const std::vector split_key = {0xa1, 0x7f, 0x61, 0xc3, 0x61, 0xa9, 0xff, 0x01}; + // a valid chunk followed by one with a truncated code point + const std::vector truncated = {0x7f, 0x62, 0x41, 0xc3, 0x61, 0xa9, 0xff}; + + // strict: the string is invalid + for (const auto& input : + { + split, split_emoji, split_key, truncated + }) + { + CAPTURE(input) + json _; + CHECK_THROWS_AS(_ = json::from_cbor(input, true, true, tags, strict), json::parse_error&); + CHECK(json::from_cbor(input, true, false, tags, strict).is_discarded()); + } + { + json _; + CHECK_THROWS_WITH_AS(_ = json::from_cbor(split, true, true, tags, strict), "[json.exception.parse_error.113] parse error at byte 3: syntax error while parsing CBOR string: invalid string: ill-formed UTF-8 byte", json::parse_error&); + CHECK_THROWS_WITH_AS(_ = json::from_cbor(split_key, true, true, tags, strict), "[json.exception.parse_error.113] parse error at byte 4: syntax error while parsing CBOR key: invalid string: ill-formed UTF-8 byte", json::parse_error&); + } + + // replace and ignore: every chunk is sanitized on its own + CHECK(json::from_cbor(split, true, true, tags, replace) == "\xEF\xBF\xBD\xEF\xBF\xBD"); + CHECK(json::from_cbor(split, true, true, tags, ignore) == ""); + CHECK(json::from_cbor(truncated, true, true, tags, replace) == "A\xEF\xBF\xBD\xEF\xBF\xBD"); + CHECK(json::from_cbor(truncated, true, true, tags, ignore) == "A"); + CHECK(json::from_cbor(split_key, true, true, tags, replace) == json{{"\xEF\xBF\xBD\xEF\xBF\xBD", 1}}); + + // keep: unchanged, the chunks are concatenated as is + CHECK(json::from_cbor(split, true, true, tags, keep) == "\xc3\xa9"); + CHECK(json::from_cbor(split_emoji, true, true, tags, keep) == "\xf0\x9f\x98\x80"); + + // conforming input is unaffected by the error handler + const std::vector valid = {0x7f, 0x62, 0xc3, 0xa9, 0x64, 0xf0, 0x9f, 0x98, 0x80, 0x60, 0xff}; + for (const auto handler : + { + strict, replace, ignore, keep + }) + { + CHECK(json::from_cbor(valid, true, true, tags, handler) == "\xc3\xa9\xf0\x9f\x98\x80"); + CHECK(json::from_cbor(std::vector({0x7f, 0xff}), true, true, tags, handler) == ""); + } + } + SECTION("many chunks in indefinite-length string") { constexpr std::size_t chunks = 100000;