An assert checking invariants would sometime trigger, the problem
was a logic error that would be exposed by a race when running out
of space in the list.
The root of the problem is that in one place we were not remapping the
-1 offset to nullptr, storing a pointer violating our invariants.
Also added more asserts!!!
cf5606e940