Automatic eviction only ever covered disconnected live instances
(_evict_disconnected_idle explicitly skipped anything with a
path), so a file-loaded capture -- potentially many GB -- stayed
resident forever until the TRACY_MCP_MAX_INSTANCES cap forced an
LRU eviction to make room. list_instances already documented
"unload_capture instead of waiting for automatic eviction" as the
alternative, but no automatic path actually existed for this case.
Generalizes the sweep into _evict_idle: file-loaded instances now
get their own idle-since-last-use TTL (TRACY_MCP_FILE_IDLE_TTL_S,
default 1800s matching the disconnected-live TTL). Safe to evict
on a timer since they're already durably on disk -- load_capture
brings them back. Connected live instances are untouched, same as
before.
Addresses the "server just vanishes" pattern from
alandtse/tracy#2, where Unable to connect gives no way to tell a
segfault apart from a hang and required manually killing an
unresponsive process before restart.
- faulthandler.enable() at startup writes a thread-state
traceback to tracy_mcp.crash.log on a genuine fatal crash
(works on Windows via SetUnhandledExceptionFilter).
- The periodic sweep loop now also logs a heartbeat (uptime,
instance/task counts, evictions) so a hung event loop is
distinguishable from a dead process by the last timestamp on
disk.
- _is_our_server_running() now backs its os.kill(pid, 0) check
with an HTTP self-ping. A deadlocked-but-alive process passed
the old PID-only check, silently blocking restart; a
non-responsive server is now reported by PID and a fresh
instance starts on a new port instead.
There is a legitimate, if rare, way to get nonReentrantCount == 0 with
total != 0 — an outer zone that never terminated before the capture ended,
whose reentrant inner zone did terminate. The outer never contributes to
slz (no end event), but it did increment the stack count, so the inner one
is correctly recorded as a reentry.
The child calls display normalized hotness colors by the sum of two
independently tracked maxima, which can occur on different addresses.
The denominator could therefore exceed any real combined cost, so
nothing reached full heat and the color scale changed meaning when the
child calls display was toggled. Track the maximum of the per-address
sums instead.
Rows in the sampling statistics were only clickable when the symbol had
exclusive samples. In the "with children" accumulation mode this made
symbols which never directly executed, such as dispatchers or wrappers
with all cost in their callees, visible but completely inert, even
though the entry stacks window can now display how such symbols were
reached. Base the interactivity on the count shown in the current
accumulation mode. The popup menu items already disable themselves when
their data is not available.
The inline function expansion follows the same rule, so expanding such
a symbol now lists its inline functions with their inclusive counts
instead of showing an empty tree.
The find zone samples list shares the drawing code with the sampling
statistics window, which computed the percentage denominator from its
own state: the whole-trace sample count, or the statistics range
filter, if one was active. The find zone counts are scoped to the
matched zones, so the percentages mixed two meanings in one table: the
time column was relative to the zone selection while the count column
was relative to the whole trace, and changing the range filter in the
statistics window silently rescaled it.
Pass the denominator from the caller. Find zone sums its zone-scoped
counts, so both columns are now relative to the selection, and the
statistics window computes the same denominator as before.
The symbol disassembly tool scoped its cost data to the statistics
range filter. This is invisible UI state which the model cannot see, so
range-limited numbers were indistinguishable from whole-trace figures
and could silently change between tool calls. The other sampling tools
always report whole-trace data; do the same here.
The heuristic reconstruction picks the call stack from the samples
within the zone's time span. Context switch samples are always parked
at the scheduler, so for zones which spent their time blocked they
dominated the root selection and the reconstructed stack showed the
scheduler path instead of the zone's real call stack. A zone covering
only context switch samples now reconstructs nothing instead.
Context switch samples are excluded from the sampling statistics, but
the flame graph built from thread samples included them. Threads which
spend time blocked accumulated large scheduler towers which none of the
other sampling views show, and the flame graph totals did not
correspond to the statistics for the same trace. Filter the samples the
same way the trace load jobs do.
The per-symbol exclusive counts reported by the tool come from the
sampling statistics, which exclude context switch samples, but the
total time reference was computed from the full sample count. This
inflated the total and made every symbol look proportionally cheaper
to the model. Use the same denominator as the statistics window.
The job which builds the symbol samples and child samples maps called
CompressThread, which updates the lookup cache and can insert into the
compression map, while the timeline processing job concurrently reads
the map, deliberately using the raw lookup to avoid this exact hazard.
All threads are already present in the compression data loaded from the
trace, so use the raw lookup as well.
When displaying the hottest inline function's name in place of the base
symbol name, the symbol map lookup was dereferenced without checking
for a missing entry, crashing in release builds. Keep the base symbol
name when the inline symbol has no symbol data.
Several code paths read the per-symbol sample lists, symbol statistics
or child sample data without checking the readiness flags. These
structures are populated by background jobs during trace load, and the
flags are the only synchronization mechanism, so reading early races
the jobs and trips the readiness asserts in debug builds. The find zone
samples list checked a different flag than the data it reads requires,
and two symbol view conditions called the accessor before the readiness
check made elsewhere in the same function. The trace information window
performed no check at all, while it is typically open during loading.
The denominator includes samples with unresolved call stacks and
samples belonging to filtered-out rows, so describing it as the number
of samples attributable to the displayed symbols was overstating what
the code does.
The button was gated on the entry stack maps being non-empty, but the
sample entry stacks window also needs the symbol data to display
anything. Without it, clicking the button silently closed the window
through its null guard. Match the gate used by the statistics window
menu items.
Since the inclusive count of an aggregated symbol entry is taken from
the base symbol's own statistics, a row whose base symbol could not be
resolved can display an inclusive count of zero while its inline
functions have nonzero counts. With the relative inline display active,
the per-inline percentages divided by the base count, printing infinity
in such cases. The time percentage variant divided by the same count
scaled by the sampling period, which cancels out, so both variants can
share the guarded reciprocal.
SortedVector considered an appended element equal to the last one to
break the ordering, marking the vector as unsorted. A non-decreasing
sequence is sorted, so only a strictly smaller element has to trigger
the marker. Equal keys are common: child sample vectors receive
identical timestamps whenever a recursive call stack contains the same
call site twice, which flipped the vectors to unsorted on virtually
every recursive workload and caused the lazy sort in GetChildSamples to
run over and over again while holding the data lock.
The lazy sort machinery handles duplicate keys correctly, as both the
prefix and tail merge windows are computed with lower bounds.
The denominator subtracts context switch sample counts from total
sample counts, both for the whole trace and within the range filter.
The subtraction could underflow: on traces with inconsistent sample
ordering the binary searches over the unsorted sample vectors can
return too few samples, and a sample which arrives with a duplicated
timestamp is merged into the existing entry while its context switch
classification is still recorded, so the context switch samples are not
a strict subset of the samples. An underflow made every percentage in
the table display as zero.
Clamp the subtraction, per thread in range mode, so that one thread
with bad data does not affect the counts of the others.
Context switch samples were appended in arrival order. Samples which
were postponed due to missing context switch data are replayed after
newer samples were already classified, so the vector could become
unordered. Everything that reads it assumes time order: the wait stacks
range filter, the sampling statistics percentage denominator, and the
context switch sample filters in the trace load jobs. In the load jobs
an unordered vector could silently disable the filtering for the rest
of a thread, reintroducing the context switch samples into the symbol
and child sample maps.
Use a SortedVector and restore the ordering at the points where it can
break: after the postponed sample replay and when saving a trace. The
save file version is bumped, so that the sort order check on load is
only performed for traces saved by previous versions.
The tool only reported "was executing" entry stacks, while the sample
entry stacks window now defaults to showing the stacks through which a
symbol was reached. The model was told there is no data for symbols
which never executed directly, even when the window right next to it
displayed their entry stacks.
Add an optional "mode" parameter with the "reached" (non-reentrant),
"reached_recursive" and "executing" values. The default matches the UI
default. Inline symbols are now accepted, as the reached maps cover
them directly, and the executing mode can use their exact statistics.
The reported mode is included in the tool output, and the chat view
shows it in the tool call label.
The tool description keeps the mode summaries short. The detailed
guidance on choosing a mode is in the optimization skill, so it only
occupies context when the skill is loaded.
LLM tool calls execute on the LLM worker thread, while the render
thread holds the worker data lock for the duration of each frame and
the network thread mutates worker data under the same lock during live
capture. The tools accessed worker data with no synchronization at all.
Some of the accessors also lazily modify worker state, such as the
postponed symbol list sorting or the lazy sorting of child sample
vectors, so this raced even in a fully loaded trace.
Take the data lock in the five tools which access worker state. No LLM
locks are held while tools execute, so no lock ordering cycle with the
render thread is possible. Tools which perform network requests do not
touch worker data and remain lockless, as holding the lock across a
network transfer would stall the profiler.
The background jobs which compute sampling statistics during trace load
do not hold the data lock. The readiness flags are the synchronization
mechanism there, so gate the sampling tools on them. This also matches
the readiness asserts in the worker accessors these tools call.
Alan Tse
Bartosz Taudul
Ivan Stepanov
David McCloskey