Scarfski/json
1
0
Fork 0
mirror of https://github.com/nlohmann/json.git synced 2026-09-28 13:05:46 +00:00
Code Issues Packages Projects Releases Wiki Activity

Throw instead of writing MessagePack lengths beyond UINT32_MAX (#5584)

Browse Source 
* Throw instead of writing MessagePack lengths beyond UINT32_MAX

MessagePack stores the length of a string, binary value, array, or
object in at most 32 bits. For a larger value, to_msgpack wrote no length
at all, so the output could not be read back. It now throws
out_of_range.412, which BSON already uses for its 32-bit length fields.

The check lives in one function, so each length is written by an
if/else chain that ends in a plain else, without a condition that can
never be false. It is tested with string and binary types that report a
size beyond UINT32_MAX without allocating it, like the BSON tests do.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Fix the CI failures of the MessagePack length check

- mark to_msgpack_length's value as used when exceptions are disabled
  (-Wunused-parameter, misc-unused-parameters)
- put "Exception safety" before "Exceptions" in to_msgpack.md, as the
  documentation style check requires
- create the test's string value from its type: constructing it from a
  beyond_uint32_string_t considers the std::filesystem::path conversion,
  which libstdc++ 10 reports as ambiguous for a class derived from
  std::string (clang 13)

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Skip the MessagePack string length test for clang with libstdc++ 10

C++17 builds consider the std::filesystem::path conversion for the
string type, and with clang and libstdc++ 10 that conversion is
ambiguous for a class derived from std::string. Creating the value from
its type did not avoid it, since any basic_json with that string type
instantiates the check. The binary and ext cases are still tested there.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Keep the MessagePack string test type and its alias in one block

astyle indented the alias oddly when it had an #ifdef of its own after
the binary alias; declare it right after the string type, in the same
block.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

---------

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
This commit is contained in:
Niels Lohmann
2026-09-27 14:28:16 +02:00
committed by GitHub
parent 6178982b8d
commit f7972970a4
6 changed files with 152 additions and 64 deletions
Download Patch File Download Diff File Expand all files Collapse all files

10
docs/mkdocs/docs/api/basic_json/to_msgpack.md
View File

@@ -34,6 +34,15 @@ The exact mapping and its limitations are described on a [dedicated page](../../
Strong guarantee: if an exception is thrown, there are no changes in the JSON value.
## Exceptions
- Throws [`out_of_range.412`](../../home/exceptions.md#jsonexceptionout_of_range412) if the length of a string, binary
value, array, or object exceeds 4294967295, the maximum MessagePack can store; example:
`"MessagePack length 4294967296 exceeds maximum of 4294967295"`
- Throws [`out_of_range.415`](../../home/exceptions.md#jsonexceptionout_of_range415) if the subtype of a binary value
exceeds 255, the maximum of the MessagePack ext type; example:
`"subtype 70000 is too large for the MessagePack ext type (max 255)"`
## Complexity
Linear in the size of the JSON value `j`.
@@ -65,3 +74,4 @@ Linear in the size of the JSON value `j`.
## Version history
- Added in version 2.0.9.
- Throws `out_of_range.412` and `out_of_range.415` since version 3.13.0.

2
docs/mkdocs/docs/features/binary_formats/messagepack.md
View File

@@ -65,6 +65,8 @@ specification:
- arrays with more than 4294967295 elements
- objects with more than 4294967295 elements
Serializing such a value throws [`out_of_range.412`](../../home/exceptions.md#jsonexceptionout_of_range412).
!!! info "NaN/infinity handling"
`NaN`, `Infinity`, and `-Infinity` are serialized as a MessagePack float 32 (type 0xCA, 5 bytes total),

14
docs/mkdocs/docs/home/exceptions.md
View File

@@ -932,19 +932,25 @@ A JSON Patch `add` operation cannot be applied because the target location's par
### json.exception.out_of_range.412
BSON stores the length of documents, arrays, strings, and binary values in a signed 32-bit integer. This exception is thrown when a value is too large to be described by such a length field.
BSON stores the length of documents, arrays, strings, and binary values in a signed 32-bit integer, and MessagePack
stores the length of strings, binary values, arrays, and objects in at most an unsigned 32-bit integer. This exception
is thrown when a value is too large to be described by such a length field.
!!! failure "Example message"
!!! failure "Example messages"
```
BSON length 2147483661 exceeds maximum of 2147483647
```
```
MessagePack length 4294967296 exceeds maximum of 4294967295
```
!!! note
This exception was added in version 3.13.0. Before that, the length was silently truncated, and
This exception was added in version 3.13.0. Before that, the BSON length was silently truncated, and
[`to_bson`](../api/basic_json/to_bson.md) produced documents with negative length prefixes that
[`from_bson`](../api/basic_json/from_bson.md) rejected.
[`from_bson`](../api/basic_json/from_bson.md) rejected; [`to_msgpack`](../api/basic_json/to_msgpack.md) wrote such
a value without any length, producing output that could not be read back.
### json.exception.out_of_range.413

53
include/nlohmann/detail/output/binary_writer.hpp
View File

@@ -291,6 +291,23 @@ class binary_writer
}
}
/*!
@brief check that @a length fits into the 32 bits that MessagePack stores
the length of a string, binary value, array, or object in
@return the length as an unsigned 32-bit integer
@throw out_of_range.412 if @a length exceeds the range of std::uint32_t
*/
static std::uint32_t to_msgpack_length(const std::size_t length, const BasicJsonType& j)
{
if (JSON_HEDLEY_UNLIKELY(!value_in_range_of<std::uint32_t>(length)))
{
JSON_THROW(out_of_range::create(412, concat("MessagePack length ", std::to_string(length), " exceeds maximum of ", std::to_string((std::numeric_limits<std::uint32_t>::max)())), &j));
}
static_cast<void>(j);
return static_cast<std::uint32_t>(length);
}
/*!
@param[in] j JSON value to serialize
*/
@@ -430,7 +447,7 @@ class binary_writer
case value_t::string:
{
// step 1: write control byte and the string length
const auto N = j.m_data.m_value.string->size();
const auto N = to_msgpack_length(j.m_data.m_value.string->size(), j);
if (N <= 31)
{
// fixstr
@@ -448,17 +465,12 @@ class binary_writer
oa.write_character(to_char_type(0xDA));
write_number(static_cast<std::uint16_t>(N));
}
else if (N <= (std::numeric_limits<std::uint32_t>::max)())
else
{
// str 32
oa.write_character(to_char_type(0xDB));
write_number(static_cast<std::uint32_t>(N));
}
else
{
JSON_THROW(out_of_range::create(412, concat("MessagePack size ", std::to_string(N), " exceeds maximum of ",
std::to_string((std::numeric_limits<std::uint32_t>::max)())), &j));
}
// step 2: write the string
oa.write_characters(
@@ -470,7 +482,7 @@ class binary_writer
case value_t::array:
{
// step 1: write control byte and the array size
const auto N = j.m_data.m_value.array->size();
const auto N = to_msgpack_length(j.m_data.m_value.array->size(), j);
if (N <= 15)
{
// fixarray
@@ -482,17 +494,12 @@ class binary_writer
oa.write_character(to_char_type(0xDC));
write_number(static_cast<std::uint16_t>(N));
}
else if (N <= (std::numeric_limits<std::uint32_t>::max)())
else
{
// array 32
oa.write_character(to_char_type(0xDD));
write_number(static_cast<std::uint32_t>(N));
}
else
{
JSON_THROW(out_of_range::create(412, concat("MessagePack size ", std::to_string(N), " exceeds maximum of ",
std::to_string((std::numeric_limits<std::uint32_t>::max)())), &j));
}
// step 2: write each element
for (const auto& el : *j.m_data.m_value.array)
@@ -509,7 +516,7 @@ class binary_writer
const bool use_ext = j.m_data.m_value.binary->has_subtype();
// step 1: write control byte and the byte string length
const auto N = j.m_data.m_value.binary->size();
const auto N = to_msgpack_length(j.m_data.m_value.binary->size(), j);
if (N <= (std::numeric_limits<std::uint8_t>::max)())
{
std::uint8_t output_type{};
@@ -561,7 +568,7 @@ class binary_writer
oa.write_character(to_char_type(output_type));
write_number(static_cast<std::uint16_t>(N));
}
else if (N <= (std::numeric_limits<std::uint32_t>::max)())
else
{
const std::uint8_t output_type = use_ext
? 0xC9 // ext 32
@@ -570,11 +577,6 @@ class binary_writer
oa.write_character(to_char_type(output_type));
write_number(static_cast<std::uint32_t>(N));
}
else
{
JSON_THROW(out_of_range::create(412, concat("MessagePack size ", std::to_string(N), " exceeds maximum of ",
std::to_string((std::numeric_limits<std::uint32_t>::max)())), &j));
}
// step 1.5: if this is an ext type, write the subtype
if (use_ext)
@@ -598,7 +600,7 @@ class binary_writer
case value_t::object:
{
// step 1: write control byte and the object size
const auto N = j.m_data.m_value.object->size();
const auto N = to_msgpack_length(j.m_data.m_value.object->size(), j);
if (N <= 15)
{
// fixmap
@@ -610,17 +612,12 @@ class binary_writer
oa.write_character(to_char_type(0xDE));
write_number(static_cast<std::uint16_t>(N));
}
else if (N <= (std::numeric_limits<std::uint32_t>::max)())
else
{
// map 32
oa.write_character(to_char_type(0xDF));
write_number(static_cast<std::uint32_t>(N));
}
else
{
JSON_THROW(out_of_range::create(412, concat("MessagePack size ", std::to_string(N), " exceeds maximum of ",
std::to_string((std::numeric_limits<std::uint32_t>::max)())), &j));
}
// step 2: write each element
for (const auto& el : *j.m_data.m_value.object)

53
single_include/nlohmann/json.hpp
View File

@@ -19756,6 +19756,23 @@ class binary_writer
}
}
/*!
@brief check that @a length fits into the 32 bits that MessagePack stores
the length of a string, binary value, array, or object in
@return the length as an unsigned 32-bit integer
@throw out_of_range.412 if @a length exceeds the range of std::uint32_t
*/
static std::uint32_t to_msgpack_length(const std::size_t length, const BasicJsonType& j)
{
if (JSON_HEDLEY_UNLIKELY(!value_in_range_of<std::uint32_t>(length)))
{
JSON_THROW(out_of_range::create(412, concat("MessagePack length ", std::to_string(length), " exceeds maximum of ", std::to_string((std::numeric_limits<std::uint32_t>::max)())), &j));
}
static_cast<void>(j);
return static_cast<std::uint32_t>(length);
}
/*!
@param[in] j JSON value to serialize
*/
@@ -19895,7 +19912,7 @@ class binary_writer
case value_t::string:
{
// step 1: write control byte and the string length
const auto N = j.m_data.m_value.string->size();
const auto N = to_msgpack_length(j.m_data.m_value.string->size(), j);
if (N <= 31)
{
// fixstr
@@ -19913,17 +19930,12 @@ class binary_writer
oa.write_character(to_char_type(0xDA));
write_number(static_cast<std::uint16_t>(N));
}
else if (N <= (std::numeric_limits<std::uint32_t>::max)())
else
{
// str 32
oa.write_character(to_char_type(0xDB));
write_number(static_cast<std::uint32_t>(N));
}
else
{
JSON_THROW(out_of_range::create(412, concat("MessagePack size ", std::to_string(N), " exceeds maximum of ",
std::to_string((std::numeric_limits<std::uint32_t>::max)())), &j));
}
// step 2: write the string
oa.write_characters(
@@ -19935,7 +19947,7 @@ class binary_writer
case value_t::array:
{
// step 1: write control byte and the array size
const auto N = j.m_data.m_value.array->size();
const auto N = to_msgpack_length(j.m_data.m_value.array->size(), j);
if (N <= 15)
{
// fixarray
@@ -19947,17 +19959,12 @@ class binary_writer
oa.write_character(to_char_type(0xDC));
write_number(static_cast<std::uint16_t>(N));
}
else if (N <= (std::numeric_limits<std::uint32_t>::max)())
else
{
// array 32
oa.write_character(to_char_type(0xDD));
write_number(static_cast<std::uint32_t>(N));
}
else
{
JSON_THROW(out_of_range::create(412, concat("MessagePack size ", std::to_string(N), " exceeds maximum of ",
std::to_string((std::numeric_limits<std::uint32_t>::max)())), &j));
}
// step 2: write each element
for (const auto& el : *j.m_data.m_value.array)
@@ -19974,7 +19981,7 @@ class binary_writer
const bool use_ext = j.m_data.m_value.binary->has_subtype();
// step 1: write control byte and the byte string length
const auto N = j.m_data.m_value.binary->size();
const auto N = to_msgpack_length(j.m_data.m_value.binary->size(), j);
if (N <= (std::numeric_limits<std::uint8_t>::max)())
{
std::uint8_t output_type{};
@@ -20026,7 +20033,7 @@ class binary_writer
oa.write_character(to_char_type(output_type));
write_number(static_cast<std::uint16_t>(N));
}
else if (N <= (std::numeric_limits<std::uint32_t>::max)())
else
{
const std::uint8_t output_type = use_ext
? 0xC9 // ext 32
@@ -20035,11 +20042,6 @@ class binary_writer
oa.write_character(to_char_type(output_type));
write_number(static_cast<std::uint32_t>(N));
}
else
{
JSON_THROW(out_of_range::create(412, concat("MessagePack size ", std::to_string(N), " exceeds maximum of ",
std::to_string((std::numeric_limits<std::uint32_t>::max)())), &j));
}
// step 1.5: if this is an ext type, write the subtype
if (use_ext)
@@ -20063,7 +20065,7 @@ class binary_writer
case value_t::object:
{
// step 1: write control byte and the object size
const auto N = j.m_data.m_value.object->size();
const auto N = to_msgpack_length(j.m_data.m_value.object->size(), j);
if (N <= 15)
{
// fixmap
@@ -20075,17 +20077,12 @@ class binary_writer
oa.write_character(to_char_type(0xDE));
write_number(static_cast<std::uint16_t>(N));
}
else if (N <= (std::numeric_limits<std::uint32_t>::max)())
else
{
// map 32
oa.write_character(to_char_type(0xDF));
write_number(static_cast<std::uint32_t>(N));
}
else
{
JSON_THROW(out_of_range::create(412, concat("MessagePack size ", std::to_string(N), " exceeds maximum of ",
std::to_string((std::numeric_limits<std::uint32_t>::max)())), &j));
}
// step 2: write each element
for (const auto& el : *j.m_data.m_value.object)

84
tests/src/unit-msgpack.cpp
View File

@@ -14,6 +14,7 @@ using nlohmann::json;
using namespace nlohmann::literals; // NOLINT(google-build-using-namespace)
#endif
#include <cstdint> // SIZE_MAX, UINT32_MAX
#include <fstream>
#include <sstream>
#include <iomanip>
@@ -2226,7 +2227,7 @@ TEST_CASE("MessagePack Size above uint32 for array")
CHECK_THROWS_WITH_AS(
huge_array_json::to_msgpack(j),
"[json.exception.out_of_range.412] MessagePack size 4294967296 exceeds maximum of 4294967295",
"[json.exception.out_of_range.412] MessagePack length 4294967296 exceeds maximum of 4294967295",
json::out_of_range&);
array.fake_size = false;
@@ -2279,7 +2280,7 @@ TEST_CASE("MessagePack Size above uint32 for object")
CHECK_THROWS_WITH_AS(
huge_object_json::to_msgpack(j),
"[json.exception.out_of_range.412] MessagePack size 4294967296 exceeds maximum of 4294967295",
"[json.exception.out_of_range.412] MessagePack length 4294967296 exceeds maximum of 4294967295",
json::out_of_range&);
object.fake_size = false;
@@ -2315,7 +2316,7 @@ TEST_CASE("MessagePack Size above uint32 for string")
CHECK_THROWS_WITH_AS(
huge_string_json::to_msgpack(j),
"[json.exception.out_of_range.412] MessagePack size 4294967296 exceeds maximum of 4294967295",
"[json.exception.out_of_range.412] MessagePack length 4294967296 exceeds maximum of 4294967295",
json::out_of_range&);
}
@@ -2352,7 +2353,82 @@ TEST_CASE("MessagePack Size above uint32 for binary")
CHECK_THROWS_WITH_AS(
huge_binary_json::to_msgpack(j),
"[json.exception.out_of_range.412] MessagePack size 4294967296 exceeds maximum of 4294967295",
"[json.exception.out_of_range.412] MessagePack length 4294967296 exceeds maximum of 4294967295",
json::out_of_range&);
}
namespace
{
// types that report a size beyond UINT32_MAX without allocating that much
// memory, so the MessagePack length limit can be tested cheaply; see the
// similar types in unit-bson.cpp
std::size_t beyond_uint32_size()
{
return static_cast<std::size_t>((std::numeric_limits<std::uint32_t>::max)()) + 1;
}
class beyond_uint32_binary_t : public std::vector<std::uint8_t>
{
public:
using std::vector<std::uint8_t>::vector;
size_type size() const noexcept // NOLINT(readability-convert-member-functions-to-static)
{
return beyond_uint32_size();
}
};
// with clang and libstdc++ 10, the std::filesystem::path conversion that
// C++17 builds consider for every string type is ambiguous for a class
// derived from std::string, so the string case is not tested there
#if !(defined(__clang__) && defined(_GLIBCXX_RELEASE) && _GLIBCXX_RELEASE < 11)
#define JSON_TEST_BEYOND_UINT32_STRING 1
#endif
#ifdef JSON_TEST_BEYOND_UINT32_STRING
class beyond_uint32_string_t : public std::string
{
public:
using std::string::string;
size_type size() const noexcept // NOLINT(readability-convert-member-functions-to-static)
{
return beyond_uint32_size();
}
};
using beyond_uint32_string_json = nlohmann::basic_json <
std::map, std::vector, beyond_uint32_string_t, bool, std::int64_t, std::uint64_t,
double, std::allocator, nlohmann::adl_serializer, std::vector<std::uint8_t>, void >;
#endif
using beyond_uint32_binary_json = nlohmann::basic_json <
std::map, std::vector, std::string, bool, std::int64_t, std::uint64_t,
double, std::allocator, nlohmann::adl_serializer, beyond_uint32_binary_t, void >;
} // namespace
TEST_CASE("MessagePack lengths beyond UINT32_MAX cannot be serialized")
{
// MessagePack stores the length of a string, binary value, array, or
// object in at most 32 bits; a larger one used to be written without any
// length at all
#if SIZE_MAX > UINT32_MAX
{
const char* const expected = "[json.exception.out_of_range.412] MessagePack length 4294967296 exceeds maximum of 4294967295";
const beyond_uint32_binary_json binary = beyond_uint32_binary_json::binary(beyond_uint32_binary_t{});
CHECK_THROWS_WITH_AS(beyond_uint32_binary_json::to_msgpack(binary), expected, beyond_uint32_binary_json::out_of_range&);
const beyond_uint32_binary_json ext = beyond_uint32_binary_json::binary(beyond_uint32_binary_t{}, 42);
CHECK_THROWS_WITH_AS(beyond_uint32_binary_json::to_msgpack(ext), expected, beyond_uint32_binary_json::out_of_range&);
#ifdef JSON_TEST_BEYOND_UINT32_STRING
// created from its type rather than from a beyond_uint32_string_t:
// that would consider the std::filesystem::path conversion, which
// libstdc++ 10 cannot decide for a class derived from std::string
const beyond_uint32_string_json string(beyond_uint32_string_json::value_t::string);
CHECK_THROWS_WITH_AS(beyond_uint32_string_json::to_msgpack(string), expected, beyond_uint32_string_json::out_of_range&);
#endif
}
#endif
}