Follow each binary format's UTF-8 rule: strict writers (CBOR/UBJSON/BJData/BSON), lenient readers (#5741)

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

docs/mkdocs/docs/api/basic_json/to_bjdata.md

@@ -56,6 +56,9 @@ Strong guarantee: if an exception is thrown, there are no changes in the JSON va
 
 - Throws [`other_error.502`](../../home/exceptions.md#jsonexceptionother_error502) if `use_type` is true and `use_size`
   is false, and `j` contains a non-empty array, object, or binary value.
+- Throws [type_error.316](../../home/exceptions.md#jsonexceptiontype_error316) if a string or object key in `j` is not
+  valid UTF-8 and [`JSON_STRICT_BINARY_UTF8`](../macros/json_strict_binary_utf8.md) is enabled; otherwise, the bytes are
+  written unchanged
 
 ## Complexity
 
@@ -104,4 +107,6 @@ Linear in the size of the JSON value `j`.
 ## Version history
 
 - Added in version 3.11.0.
-- BJData version parameter (for draft3 binary encoding) added in version 3.12.0.
+- BJData version parameter (for draft3 binary encoding) added in version 3.12.0.
+- Throwing `type_error.316` for a string or object key that is not valid UTF-8 if
+  [`JSON_STRICT_BINARY_UTF8`](../macros/json_strict_binary_utf8.md) is enabled added in version 3.13.0.

docs/mkdocs/docs/api/basic_json/to_bson.md

@@ -46,6 +46,9 @@ Strong guarantee: if an exception is thrown, there are no changes in the JSON va
 - Throws [`out_of_range.415`](../../home/exceptions.md#jsonexceptionout_of_range415) if the subtype of a binary value
   exceeds 255, the maximum of the BSON binary subtype; example:
   `"subtype 70000 is too large for the BSON binary subtype (max 255)"`
+- Throws [type_error.316](../../home/exceptions.md#jsonexceptiontype_error316) if a string or object key is not valid
+  UTF-8 and [`JSON_STRICT_BINARY_UTF8`](../macros/json_strict_binary_utf8.md) is enabled; otherwise, the bytes are
+  written unchanged
 
 ## Complexity
 
@@ -98,3 +101,6 @@ pass before anything is written.
 - Throws `out_of_range.412` and `out_of_range.415` since version 3.13.0.
 - Linear in the size of `j`, and no longer limited by the call stack for deeply nested values, since version 3.13.0.
 - `out_of_range.415` is now detected before anything is written, like the other exceptions above, since version 3.13.0.
+- Throwing `type_error.316` for a string value or object key that is not valid UTF-8 if
+  [`JSON_STRICT_BINARY_UTF8`](../macros/json_strict_binary_utf8.md) is enabled, detected before anything is written,
+  added in version 3.13.0.

docs/mkdocs/docs/api/basic_json/to_cbor.md

@@ -35,6 +35,12 @@ The exact mapping and its limitations are described on a [dedicated page](../../
 
 Strong guarantee: if an exception is thrown, there are no changes in the JSON value.
 
+## Exceptions
+
+- Throws [type_error.316](../../home/exceptions.md#jsonexceptiontype_error316) if a string or object key in `j` is not
+  valid UTF-8 and [`JSON_STRICT_BINARY_UTF8`](../macros/json_strict_binary_utf8.md) is enabled; otherwise, the bytes are
+  written unchanged
+
 ## Complexity
 
 Linear in the size of the JSON value `j`.
@@ -68,3 +74,5 @@ Linear in the size of the JSON value `j`.
 
 - Added in version 2.0.9.
 - Compact representation of floating-point numbers added in version 3.8.0.
+- Throwing `type_error.316` for a string or object key that is not valid UTF-8 if
+  [`JSON_STRICT_BINARY_UTF8`](../macros/json_strict_binary_utf8.md) is enabled added in version 3.13.0.

docs/mkdocs/docs/api/basic_json/to_ubjson.md

@@ -49,6 +49,9 @@ Strong guarantee: if an exception is thrown, there are no changes in the JSON va
 
 - Throws [`other_error.502`](../../home/exceptions.md#jsonexceptionother_error502) if `use_type` is true and `use_size`
   is false, and `j` contains a non-empty array, object, or binary value.
+- Throws [type_error.316](../../home/exceptions.md#jsonexceptiontype_error316) if a string or object key in `j` is not
+  valid UTF-8 and [`JSON_STRICT_BINARY_UTF8`](../macros/json_strict_binary_utf8.md) is enabled; otherwise, the bytes are
+  written unchanged
 
 ## Complexity
 
@@ -97,3 +100,5 @@ Linear in the size of the JSON value `j`.
 ## Version history
 
 - Added in version 3.1.0.
+- Throwing `type_error.316` for a string or object key that is not valid UTF-8 if
+  [`JSON_STRICT_BINARY_UTF8`](../macros/json_strict_binary_utf8.md) is enabled added in version 3.13.0.

docs/mkdocs/docs/api/macros/index.md

@@ -18,6 +18,8 @@ header. See also the [macro overview page](../../features/macros.md).
 
 - [**JSON_PRECISE_STREAM_POSITION**](json_precise_stream_position.md) - opt in to leaving an input stream positioned
   right after a parsed number
+- [**JSON_STRICT_BINARY_UTF8**](json_strict_binary_utf8.md) - opt in to checking strings for valid UTF-8 in the CBOR,
+  UBJSON, BJData, and BSON writers
 - [**JSON_STRICT_NUL_HANDLING**](json_strict_nul_handling.md) - opt in to rejecting a NUL byte in the input instead of
   treating it as end of input
 

docs/mkdocs/docs/api/macros/json_strict_binary_utf8.md

@@ -0,0 +1,97 @@
+# JSON_STRICT_BINARY_UTF8
+
+```cpp
+#define JSON_STRICT_BINARY_UTF8 /* value */
+```
+
+When defined to `1`, the binary writers [`to_cbor`](../basic_json/to_cbor.md), [`to_ubjson`](../basic_json/to_ubjson.md),
+[`to_bjdata`](../basic_json/to_bjdata.md), and [`to_bson`](../basic_json/to_bson.md) check every string value and
+object key for valid UTF-8 and throw [`type_error.316`](../../home/exceptions.md#jsonexceptiontype_error316) for
+ill-formed UTF-8, like [`dump`](../basic_json/dump.md) does. Without it, they write the bytes unchanged.
+
+The macro does not affect:
+
+- [`to_msgpack`](../basic_json/to_msgpack.md): the MessagePack specification allows a `str` value to contain bytes that
+  are not valid UTF-8, so it always writes them unchanged.
+- [`to_bon8`](../basic_json/to_bon8.md): BON8 always checks, because the UTF-8 lead bytes mark where a string ends.
+- The binary readers ([`from_cbor`](../basic_json/from_cbor.md), [`from_msgpack`](../basic_json/from_msgpack.md),
+  [`from_ubjson`](../basic_json/from_ubjson.md), [`from_bjdata`](../basic_json/from_bjdata.md),
+  [`from_bson`](../basic_json/from_bson.md)): none of these formats requires a decoder to reject ill-formed UTF-8, so
+  they always return the bytes unchanged.
+
+## Default definition
+
+The default value is `0` (disabled, the behavior of version 3.12.0 and earlier is preserved).
+
+```cpp
+#define JSON_STRICT_BINARY_UTF8 0
+```
+
+## Notes
+
+!!! note "Background"
+
+    CBOR, UBJSON, BJData, and BSON all require strings to be UTF-8. Up to version 3.12.0, the writers did not check
+    this, so they could produce output that other decoders reject. Checking by default would break code that stores
+    other encodings (for instance ISO 8859-1) in a string and only ever writes it to a binary format, so this macro
+    offers the check as an opt-in ahead of version 4.0.0, where it is planned to become the default (see
+    [#5529](https://github.com/nlohmann/json/issues/5529) and [#5651](https://github.com/nlohmann/json/issues/5651)).
+
+!!! warning "Opt-in only"
+
+    This macro must be defined **before** including `<nlohmann/json.hpp>`. Defining it after the include has no
+    effect.
+
+!!! note "ABI compatibility"
+
+    The value of this macro is encoded in the [namespace](../../features/namespace.md) (tag `_sbu8`), resulting in
+    distinct symbol names. Translation units compiled with and without it can therefore be linked into the same program
+    without One Definition Rule (ODR) violations, but they cannot exchange instances of library types.
+
+## Examples
+
+??? example "Default behavior (macro not defined)"
+
+    Without the macro, the bytes are written unchanged:
+
+    ```cpp
+    #include <nlohmann/json.hpp>
+
+    using json = nlohmann::json;
+
+    int main()
+    {
+        auto v = json::to_cbor(json("\xFF"));
+        // v is {0x61, 0xFF}
+    }
+    ```
+
+??? example "Opt-in check (macro defined to 1)"
+
+    With the macro, ill-formed UTF-8 is rejected:
+
+    ```cpp
+    #define JSON_STRICT_BINARY_UTF8 1
+    #include <nlohmann/json.hpp>
+
+    using json = nlohmann::json;
+
+    int main()
+    {
+        auto v = json::to_cbor(json("\xFF"));
+        // throws type_error.316: invalid UTF-8 byte at index 0: 0xFF
+    }
+    ```
+
+## See also
+
+- [**to_cbor**](../basic_json/to_cbor.md) - create a CBOR serialization of a JSON value
+- [**to_ubjson**](../basic_json/to_ubjson.md) - create a UBJSON serialization of a JSON value
+- [**to_bjdata**](../basic_json/to_bjdata.md) - create a BJData serialization of a JSON value
+- [**to_bson**](../basic_json/to_bson.md) - create a BSON serialization of a JSON value
+- [**error_handler_t**](../basic_json/error_handler_t.md) - how [`dump`](../basic_json/dump.md) treats ill-formed UTF-8
+
+## Version history
+
+- Added in version 3.13.0.
+- Planned to become the default (with the macro removed) in version 4.0.0.