From f7972970a4d620a9807bc63217cde94db8ff0da9 Mon Sep 17 00:00:00 2001 From: Niels Lohmann Date: Sun, 27 Sep 2026 14:28:16 +0200 Subject: [PATCH] Throw instead of writing MessagePack lengths beyond UINT32_MAX (#5584) * Throw instead of writing MessagePack lengths beyond UINT32_MAX MessagePack stores the length of a string, binary value, array, or object in at most 32 bits. For a larger value, to_msgpack wrote no length at all, so the output could not be read back. It now throws out_of_range.412, which BSON already uses for its 32-bit length fields. The check lives in one function, so each length is written by an if/else chain that ends in a plain else, without a condition that can never be false. It is tested with string and binary types that report a size beyond UINT32_MAX without allocating it, like the BSON tests do. Signed-off-by: Niels Lohmann * Fix the CI failures of the MessagePack length check - mark to_msgpack_length's value as used when exceptions are disabled (-Wunused-parameter, misc-unused-parameters) - put "Exception safety" before "Exceptions" in to_msgpack.md, as the documentation style check requires - create the test's string value from its type: constructing it from a beyond_uint32_string_t considers the std::filesystem::path conversion, which libstdc++ 10 reports as ambiguous for a class derived from std::string (clang 13) Signed-off-by: Niels Lohmann * Skip the MessagePack string length test for clang with libstdc++ 10 C++17 builds consider the std::filesystem::path conversion for the string type, and with clang and libstdc++ 10 that conversion is ambiguous for a class derived from std::string. Creating the value from its type did not avoid it, since any basic_json with that string type instantiates the check. The binary and ext cases are still tested there. Signed-off-by: Niels Lohmann * Keep the MessagePack string test type and its alias in one block astyle indented the alias oddly when it had an #ifdef of its own after the binary alias; declare it right after the string type, in the same block. Signed-off-by: Niels Lohmann --------- Signed-off-by: Niels Lohmann --- docs/mkdocs/docs/api/basic_json/to_msgpack.md | 10 +++ .../features/binary_formats/messagepack.md | 2 + docs/mkdocs/docs/home/exceptions.md | 14 +++- .../nlohmann/detail/output/binary_writer.hpp | 53 ++++++------ single_include/nlohmann/json.hpp | 53 ++++++------ tests/src/unit-msgpack.cpp | 84 ++++++++++++++++++- 6 files changed, 152 insertions(+), 64 deletions(-) diff --git a/docs/mkdocs/docs/api/basic_json/to_msgpack.md b/docs/mkdocs/docs/api/basic_json/to_msgpack.md index 66b104f52..b3bcaab7f 100644 --- a/docs/mkdocs/docs/api/basic_json/to_msgpack.md +++ b/docs/mkdocs/docs/api/basic_json/to_msgpack.md @@ -34,6 +34,15 @@ The exact mapping and its limitations are described on a [dedicated page](../../ Strong guarantee: if an exception is thrown, there are no changes in the JSON value. +## Exceptions + +- Throws [`out_of_range.412`](../../home/exceptions.md#jsonexceptionout_of_range412) if the length of a string, binary + value, array, or object exceeds 4294967295, the maximum MessagePack can store; example: + `"MessagePack length 4294967296 exceeds maximum of 4294967295"` +- Throws [`out_of_range.415`](../../home/exceptions.md#jsonexceptionout_of_range415) if the subtype of a binary value + exceeds 255, the maximum of the MessagePack ext type; example: + `"subtype 70000 is too large for the MessagePack ext type (max 255)"` + ## Complexity Linear in the size of the JSON value `j`. @@ -65,3 +74,4 @@ Linear in the size of the JSON value `j`. ## Version history - Added in version 2.0.9. +- Throws `out_of_range.412` and `out_of_range.415` since version 3.13.0. diff --git a/docs/mkdocs/docs/features/binary_formats/messagepack.md b/docs/mkdocs/docs/features/binary_formats/messagepack.md index a434909c4..0ca82c145 100644 --- a/docs/mkdocs/docs/features/binary_formats/messagepack.md +++ b/docs/mkdocs/docs/features/binary_formats/messagepack.md @@ -65,6 +65,8 @@ specification: - arrays with more than 4294967295 elements - objects with more than 4294967295 elements + Serializing such a value throws [`out_of_range.412`](../../home/exceptions.md#jsonexceptionout_of_range412). + !!! info "NaN/infinity handling" `NaN`, `Infinity`, and `-Infinity` are serialized as a MessagePack float 32 (type 0xCA, 5 bytes total), diff --git a/docs/mkdocs/docs/home/exceptions.md b/docs/mkdocs/docs/home/exceptions.md index ee76596f6..bf18baab1 100644 --- a/docs/mkdocs/docs/home/exceptions.md +++ b/docs/mkdocs/docs/home/exceptions.md @@ -932,19 +932,25 @@ A JSON Patch `add` operation cannot be applied because the target location's par ### json.exception.out_of_range.412 -BSON stores the length of documents, arrays, strings, and binary values in a signed 32-bit integer. This exception is thrown when a value is too large to be described by such a length field. +BSON stores the length of documents, arrays, strings, and binary values in a signed 32-bit integer, and MessagePack +stores the length of strings, binary values, arrays, and objects in at most an unsigned 32-bit integer. This exception +is thrown when a value is too large to be described by such a length field. -!!! failure "Example message" +!!! failure "Example messages" ``` BSON length 2147483661 exceeds maximum of 2147483647 ``` + ``` + MessagePack length 4294967296 exceeds maximum of 4294967295 + ``` !!! note - This exception was added in version 3.13.0. Before that, the length was silently truncated, and + This exception was added in version 3.13.0. Before that, the BSON length was silently truncated, and [`to_bson`](../api/basic_json/to_bson.md) produced documents with negative length prefixes that - [`from_bson`](../api/basic_json/from_bson.md) rejected. + [`from_bson`](../api/basic_json/from_bson.md) rejected; [`to_msgpack`](../api/basic_json/to_msgpack.md) wrote such + a value without any length, producing output that could not be read back. ### json.exception.out_of_range.413 diff --git a/include/nlohmann/detail/output/binary_writer.hpp b/include/nlohmann/detail/output/binary_writer.hpp index 9c41e2962..5b11a3b2f 100644 --- a/include/nlohmann/detail/output/binary_writer.hpp +++ b/include/nlohmann/detail/output/binary_writer.hpp @@ -291,6 +291,23 @@ class binary_writer } } + /*! + @brief check that @a length fits into the 32 bits that MessagePack stores + the length of a string, binary value, array, or object in + @return the length as an unsigned 32-bit integer + @throw out_of_range.412 if @a length exceeds the range of std::uint32_t + */ + static std::uint32_t to_msgpack_length(const std::size_t length, const BasicJsonType& j) + { + if (JSON_HEDLEY_UNLIKELY(!value_in_range_of(length))) + { + JSON_THROW(out_of_range::create(412, concat("MessagePack length ", std::to_string(length), " exceeds maximum of ", std::to_string((std::numeric_limits::max)())), &j)); + } + + static_cast(j); + return static_cast(length); + } + /*! @param[in] j JSON value to serialize */ @@ -430,7 +447,7 @@ class binary_writer case value_t::string: { // step 1: write control byte and the string length - const auto N = j.m_data.m_value.string->size(); + const auto N = to_msgpack_length(j.m_data.m_value.string->size(), j); if (N <= 31) { // fixstr @@ -448,17 +465,12 @@ class binary_writer oa.write_character(to_char_type(0xDA)); write_number(static_cast(N)); } - else if (N <= (std::numeric_limits::max)()) + else { // str 32 oa.write_character(to_char_type(0xDB)); write_number(static_cast(N)); } - else - { - JSON_THROW(out_of_range::create(412, concat("MessagePack size ", std::to_string(N), " exceeds maximum of ", - std::to_string((std::numeric_limits::max)())), &j)); - } // step 2: write the string oa.write_characters( @@ -470,7 +482,7 @@ class binary_writer case value_t::array: { // step 1: write control byte and the array size - const auto N = j.m_data.m_value.array->size(); + const auto N = to_msgpack_length(j.m_data.m_value.array->size(), j); if (N <= 15) { // fixarray @@ -482,17 +494,12 @@ class binary_writer oa.write_character(to_char_type(0xDC)); write_number(static_cast(N)); } - else if (N <= (std::numeric_limits::max)()) + else { // array 32 oa.write_character(to_char_type(0xDD)); write_number(static_cast(N)); } - else - { - JSON_THROW(out_of_range::create(412, concat("MessagePack size ", std::to_string(N), " exceeds maximum of ", - std::to_string((std::numeric_limits::max)())), &j)); - } // step 2: write each element for (const auto& el : *j.m_data.m_value.array) @@ -509,7 +516,7 @@ class binary_writer const bool use_ext = j.m_data.m_value.binary->has_subtype(); // step 1: write control byte and the byte string length - const auto N = j.m_data.m_value.binary->size(); + const auto N = to_msgpack_length(j.m_data.m_value.binary->size(), j); if (N <= (std::numeric_limits::max)()) { std::uint8_t output_type{}; @@ -561,7 +568,7 @@ class binary_writer oa.write_character(to_char_type(output_type)); write_number(static_cast(N)); } - else if (N <= (std::numeric_limits::max)()) + else { const std::uint8_t output_type = use_ext ? 0xC9 // ext 32 @@ -570,11 +577,6 @@ class binary_writer oa.write_character(to_char_type(output_type)); write_number(static_cast(N)); } - else - { - JSON_THROW(out_of_range::create(412, concat("MessagePack size ", std::to_string(N), " exceeds maximum of ", - std::to_string((std::numeric_limits::max)())), &j)); - } // step 1.5: if this is an ext type, write the subtype if (use_ext) @@ -598,7 +600,7 @@ class binary_writer case value_t::object: { // step 1: write control byte and the object size - const auto N = j.m_data.m_value.object->size(); + const auto N = to_msgpack_length(j.m_data.m_value.object->size(), j); if (N <= 15) { // fixmap @@ -610,17 +612,12 @@ class binary_writer oa.write_character(to_char_type(0xDE)); write_number(static_cast(N)); } - else if (N <= (std::numeric_limits::max)()) + else { // map 32 oa.write_character(to_char_type(0xDF)); write_number(static_cast(N)); } - else - { - JSON_THROW(out_of_range::create(412, concat("MessagePack size ", std::to_string(N), " exceeds maximum of ", - std::to_string((std::numeric_limits::max)())), &j)); - } // step 2: write each element for (const auto& el : *j.m_data.m_value.object) diff --git a/single_include/nlohmann/json.hpp b/single_include/nlohmann/json.hpp index ba41273de..bf1aee332 100644 --- a/single_include/nlohmann/json.hpp +++ b/single_include/nlohmann/json.hpp @@ -19756,6 +19756,23 @@ class binary_writer } } + /*! + @brief check that @a length fits into the 32 bits that MessagePack stores + the length of a string, binary value, array, or object in + @return the length as an unsigned 32-bit integer + @throw out_of_range.412 if @a length exceeds the range of std::uint32_t + */ + static std::uint32_t to_msgpack_length(const std::size_t length, const BasicJsonType& j) + { + if (JSON_HEDLEY_UNLIKELY(!value_in_range_of(length))) + { + JSON_THROW(out_of_range::create(412, concat("MessagePack length ", std::to_string(length), " exceeds maximum of ", std::to_string((std::numeric_limits::max)())), &j)); + } + + static_cast(j); + return static_cast(length); + } + /*! @param[in] j JSON value to serialize */ @@ -19895,7 +19912,7 @@ class binary_writer case value_t::string: { // step 1: write control byte and the string length - const auto N = j.m_data.m_value.string->size(); + const auto N = to_msgpack_length(j.m_data.m_value.string->size(), j); if (N <= 31) { // fixstr @@ -19913,17 +19930,12 @@ class binary_writer oa.write_character(to_char_type(0xDA)); write_number(static_cast(N)); } - else if (N <= (std::numeric_limits::max)()) + else { // str 32 oa.write_character(to_char_type(0xDB)); write_number(static_cast(N)); } - else - { - JSON_THROW(out_of_range::create(412, concat("MessagePack size ", std::to_string(N), " exceeds maximum of ", - std::to_string((std::numeric_limits::max)())), &j)); - } // step 2: write the string oa.write_characters( @@ -19935,7 +19947,7 @@ class binary_writer case value_t::array: { // step 1: write control byte and the array size - const auto N = j.m_data.m_value.array->size(); + const auto N = to_msgpack_length(j.m_data.m_value.array->size(), j); if (N <= 15) { // fixarray @@ -19947,17 +19959,12 @@ class binary_writer oa.write_character(to_char_type(0xDC)); write_number(static_cast(N)); } - else if (N <= (std::numeric_limits::max)()) + else { // array 32 oa.write_character(to_char_type(0xDD)); write_number(static_cast(N)); } - else - { - JSON_THROW(out_of_range::create(412, concat("MessagePack size ", std::to_string(N), " exceeds maximum of ", - std::to_string((std::numeric_limits::max)())), &j)); - } // step 2: write each element for (const auto& el : *j.m_data.m_value.array) @@ -19974,7 +19981,7 @@ class binary_writer const bool use_ext = j.m_data.m_value.binary->has_subtype(); // step 1: write control byte and the byte string length - const auto N = j.m_data.m_value.binary->size(); + const auto N = to_msgpack_length(j.m_data.m_value.binary->size(), j); if (N <= (std::numeric_limits::max)()) { std::uint8_t output_type{}; @@ -20026,7 +20033,7 @@ class binary_writer oa.write_character(to_char_type(output_type)); write_number(static_cast(N)); } - else if (N <= (std::numeric_limits::max)()) + else { const std::uint8_t output_type = use_ext ? 0xC9 // ext 32 @@ -20035,11 +20042,6 @@ class binary_writer oa.write_character(to_char_type(output_type)); write_number(static_cast(N)); } - else - { - JSON_THROW(out_of_range::create(412, concat("MessagePack size ", std::to_string(N), " exceeds maximum of ", - std::to_string((std::numeric_limits::max)())), &j)); - } // step 1.5: if this is an ext type, write the subtype if (use_ext) @@ -20063,7 +20065,7 @@ class binary_writer case value_t::object: { // step 1: write control byte and the object size - const auto N = j.m_data.m_value.object->size(); + const auto N = to_msgpack_length(j.m_data.m_value.object->size(), j); if (N <= 15) { // fixmap @@ -20075,17 +20077,12 @@ class binary_writer oa.write_character(to_char_type(0xDE)); write_number(static_cast(N)); } - else if (N <= (std::numeric_limits::max)()) + else { // map 32 oa.write_character(to_char_type(0xDF)); write_number(static_cast(N)); } - else - { - JSON_THROW(out_of_range::create(412, concat("MessagePack size ", std::to_string(N), " exceeds maximum of ", - std::to_string((std::numeric_limits::max)())), &j)); - } // step 2: write each element for (const auto& el : *j.m_data.m_value.object) diff --git a/tests/src/unit-msgpack.cpp b/tests/src/unit-msgpack.cpp index 0b8c6ca63..d86e6f068 100644 --- a/tests/src/unit-msgpack.cpp +++ b/tests/src/unit-msgpack.cpp @@ -14,6 +14,7 @@ using nlohmann::json; using namespace nlohmann::literals; // NOLINT(google-build-using-namespace) #endif +#include // SIZE_MAX, UINT32_MAX #include #include #include @@ -2226,7 +2227,7 @@ TEST_CASE("MessagePack Size above uint32 for array") CHECK_THROWS_WITH_AS( huge_array_json::to_msgpack(j), - "[json.exception.out_of_range.412] MessagePack size 4294967296 exceeds maximum of 4294967295", + "[json.exception.out_of_range.412] MessagePack length 4294967296 exceeds maximum of 4294967295", json::out_of_range&); array.fake_size = false; @@ -2279,7 +2280,7 @@ TEST_CASE("MessagePack Size above uint32 for object") CHECK_THROWS_WITH_AS( huge_object_json::to_msgpack(j), - "[json.exception.out_of_range.412] MessagePack size 4294967296 exceeds maximum of 4294967295", + "[json.exception.out_of_range.412] MessagePack length 4294967296 exceeds maximum of 4294967295", json::out_of_range&); object.fake_size = false; @@ -2315,7 +2316,7 @@ TEST_CASE("MessagePack Size above uint32 for string") CHECK_THROWS_WITH_AS( huge_string_json::to_msgpack(j), - "[json.exception.out_of_range.412] MessagePack size 4294967296 exceeds maximum of 4294967295", + "[json.exception.out_of_range.412] MessagePack length 4294967296 exceeds maximum of 4294967295", json::out_of_range&); } @@ -2352,7 +2353,82 @@ TEST_CASE("MessagePack Size above uint32 for binary") CHECK_THROWS_WITH_AS( huge_binary_json::to_msgpack(j), - "[json.exception.out_of_range.412] MessagePack size 4294967296 exceeds maximum of 4294967295", + "[json.exception.out_of_range.412] MessagePack length 4294967296 exceeds maximum of 4294967295", json::out_of_range&); } +namespace +{ +// types that report a size beyond UINT32_MAX without allocating that much +// memory, so the MessagePack length limit can be tested cheaply; see the +// similar types in unit-bson.cpp +std::size_t beyond_uint32_size() +{ + return static_cast((std::numeric_limits::max)()) + 1; +} + +class beyond_uint32_binary_t : public std::vector +{ + public: + using std::vector::vector; + + size_type size() const noexcept // NOLINT(readability-convert-member-functions-to-static) + { + return beyond_uint32_size(); + } +}; + +// with clang and libstdc++ 10, the std::filesystem::path conversion that +// C++17 builds consider for every string type is ambiguous for a class +// derived from std::string, so the string case is not tested there +#if !(defined(__clang__) && defined(_GLIBCXX_RELEASE) && _GLIBCXX_RELEASE < 11) + #define JSON_TEST_BEYOND_UINT32_STRING 1 +#endif + +#ifdef JSON_TEST_BEYOND_UINT32_STRING +class beyond_uint32_string_t : public std::string +{ + public: + using std::string::string; + + size_type size() const noexcept // NOLINT(readability-convert-member-functions-to-static) + { + return beyond_uint32_size(); + } +}; + +using beyond_uint32_string_json = nlohmann::basic_json < + std::map, std::vector, beyond_uint32_string_t, bool, std::int64_t, std::uint64_t, + double, std::allocator, nlohmann::adl_serializer, std::vector, void >; +#endif + +using beyond_uint32_binary_json = nlohmann::basic_json < + std::map, std::vector, std::string, bool, std::int64_t, std::uint64_t, + double, std::allocator, nlohmann::adl_serializer, beyond_uint32_binary_t, void >; +} // namespace + +TEST_CASE("MessagePack lengths beyond UINT32_MAX cannot be serialized") +{ + // MessagePack stores the length of a string, binary value, array, or + // object in at most 32 bits; a larger one used to be written without any + // length at all +#if SIZE_MAX > UINT32_MAX + { + const char* const expected = "[json.exception.out_of_range.412] MessagePack length 4294967296 exceeds maximum of 4294967295"; + + const beyond_uint32_binary_json binary = beyond_uint32_binary_json::binary(beyond_uint32_binary_t{}); + CHECK_THROWS_WITH_AS(beyond_uint32_binary_json::to_msgpack(binary), expected, beyond_uint32_binary_json::out_of_range&); + + const beyond_uint32_binary_json ext = beyond_uint32_binary_json::binary(beyond_uint32_binary_t{}, 42); + CHECK_THROWS_WITH_AS(beyond_uint32_binary_json::to_msgpack(ext), expected, beyond_uint32_binary_json::out_of_range&); + +#ifdef JSON_TEST_BEYOND_UINT32_STRING + // created from its type rather than from a beyond_uint32_string_t: + // that would consider the std::filesystem::path conversion, which + // libstdc++ 10 cannot decide for a class derived from std::string + const beyond_uint32_string_json string(beyond_uint32_string_json::value_t::string); + CHECK_THROWS_WITH_AS(beyond_uint32_string_json::to_msgpack(string), expected, beyond_uint32_string_json::out_of_range&); +#endif + } +#endif +}