Merge deeply nested objects without recursing per nesting level (#5547)

* Merge deeply nested objects without recursing per nesting level

merge_patch() and update(j, true) merged a nested object by calling
themselves on it, once per nesting level. A value nested deeply enough -
50,000 levels of objects on an 8 MiB stack - exhausted the call stack
and terminated the process, although parse() accepts such values without
complaint.

Bound the descent the same way dump() does. The recursion now carries
the nesting level, and once merge_depth_limit() (128) levels have been
entered, update_members_iteratively() and merge_patch_iteratively()
finish the merge on an explicit stack. They still merge a nested object
completely before the next member, and in the same order, so the results,
including the parents JSON_DIAGNOSTICS reports paths from, are unchanged.
Values nested less deeply than the bound run the same code as before, so
the common case does not pay for the stack: merging only on it cost
10-14% in a first version.

The public signatures are unchanged. The recursive worker behind
merge_patch() has its own name rather than being a private overload, so
that &basic_json::merge_patch stays unambiguous.

Tests check every depth up to 300 against recursive reference
implementations of both operations, check the diagnostic paths past the
bound, and merge objects nested 100,000 levels deep.

Fixes #5545 for update(j, true), and #5393 for merge_patch().

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Use the shared recursion limit in update() and merge_patch()

merge_depth_limit() is gone in favor of detail::recursion_depth_limit().
The two identical function-local frame structs become one member struct,
merge_frame, with a constructor, so both loops emplace_back() their
frames. merge_patch_iteratively() copies the frame it works on out of the
stack and changes it only through stack.back().

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Build the update()/merge_patch() diagnostics test values instead of parsing them

Parsed values carry byte positions under JSON_DIAGNOSTIC_POSITIONS, which
the expected messages do not include.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

---------

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

tests/src/unit-merge_patch.cpp

@@ -14,6 +14,60 @@ using nlohmann::json;
     using namespace nlohmann::literals; // NOLINT(google-build-using-namespace)
 #endif
 
+#include <string>
+
+namespace
+{
+// RFC 7396's MergePatch, written recursively as in the RFC; only usable on
+// values nested a few hundred levels deep
+void reference_merge_patch(json& target, const json& patch)
+{
+    if (!patch.is_object())
+    {
+        target = patch;
+        return;
+    }
+    if (!target.is_object())
+    {
+        target = json::object();
+    }
+    for (auto it = patch.begin(); it != patch.end(); ++it)
+    {
+        if (it.value().is_null())
+        {
+            target.erase(it.key());
+        }
+        else
+        {
+            reference_merge_patch(target[it.key()], it.value());
+        }
+    }
+}
+
+// objects nested `depth` levels deep under the key "a", with members that
+// differ by `variant` on the way down
+std::string nested_objects(const std::size_t depth, const int variant)
+{
+    std::string text;
+    for (std::size_t i = 0; i < depth; ++i)
+    {
+        text += "{";
+        if ((i + static_cast<std::size_t>(variant)) % 3 == 0)
+        {
+            text += "\"s" + std::to_string(variant) + "\":" + std::to_string(i) + ",";
+        }
+        if (variant == 2 && i % 5 == 0)
+        {
+            text += "\"s0\":null,";
+        }
+        text += "\"a\":";
+    }
+    text += variant == 1 ? "{\"x\":1,\"y\":null}" : "{\"y\":2}";
+    text.append(depth, '}');
+    return text;
+}
+} // namespace
+
 TEST_CASE("JSON Merge Patch")
 {
     SECTION("examples from RFC 7396")
@@ -242,3 +296,52 @@ TEST_CASE("JSON Merge Patch")
         }
     }
 }
+
+TEST_CASE("JSON Merge Patch on deeply nested values")
+{
+    SECTION("patching past the descent bound gives the same result")
+    {
+        // every depth on either side of where the iterative version takes
+        // over (detail::recursion_depth_limit(), 128)
+        for (std::size_t depth = 0; depth <= 300; ++depth)
+        {
+            CAPTURE(depth);
+            for (int variant = 0; variant < 3; ++variant)
+            {
+                CAPTURE(variant);
+                const json patch = json::parse(nested_objects(depth, variant));
+
+                json result = json::parse(nested_objects(depth, (variant + 1) % 3));
+                json expected = result;
+                result.merge_patch(patch);
+                reference_merge_patch(expected, patch);
+                CHECK(result == expected);
+
+                // a target that is not an object, and an empty one
+                json from_null;
+                from_null.merge_patch(patch);
+                json expected_from_null;
+                reference_merge_patch(expected_from_null, patch);
+                CHECK(from_null == expected_from_null);
+            }
+        }
+    }
+
+    SECTION("patches nested too deeply for the call stack (#5393)")
+    {
+        // applying a patch used to recurse once per nesting level. The result
+        // is only walked, never copied or compared, since those recurse too.
+        const std::size_t depth = 100000;
+        json target = json::parse(nested_objects(depth, 0));
+        target.merge_patch(json::parse(nested_objects(depth, 1)));
+
+        const json* p = &target;
+        for (std::size_t i = 0; i < depth; ++i)
+        {
+            p = &p->at("a");
+        }
+        // {"y":2} patched with {"x":1,"y":null}
+        CHECK(p->size() == 1);
+        CHECK(p->at("x") == 1);
+    }
+}