There is a legitimate, if rare, way to get nonReentrantCount == 0 with
total != 0 — an outer zone that never terminated before the capture ended,
whose reentrant inner zone did terminate. The outer never contributes to
slz (no end event), but it did increment the stack count, so the inner one
is correctly recorded as a reentry.
The child calls display normalized hotness colors by the sum of two
independently tracked maxima, which can occur on different addresses.
The denominator could therefore exceed any real combined cost, so
nothing reached full heat and the color scale changed meaning when the
child calls display was toggled. Track the maximum of the per-address
sums instead.
Rows in the sampling statistics were only clickable when the symbol had
exclusive samples. In the "with children" accumulation mode this made
symbols which never directly executed, such as dispatchers or wrappers
with all cost in their callees, visible but completely inert, even
though the entry stacks window can now display how such symbols were
reached. Base the interactivity on the count shown in the current
accumulation mode. The popup menu items already disable themselves when
their data is not available.
The inline function expansion follows the same rule, so expanding such
a symbol now lists its inline functions with their inclusive counts
instead of showing an empty tree.
The find zone samples list shares the drawing code with the sampling
statistics window, which computed the percentage denominator from its
own state: the whole-trace sample count, or the statistics range
filter, if one was active. The find zone counts are scoped to the
matched zones, so the percentages mixed two meanings in one table: the
time column was relative to the zone selection while the count column
was relative to the whole trace, and changing the range filter in the
statistics window silently rescaled it.
Pass the denominator from the caller. Find zone sums its zone-scoped
counts, so both columns are now relative to the selection, and the
statistics window computes the same denominator as before.
The symbol disassembly tool scoped its cost data to the statistics
range filter. This is invisible UI state which the model cannot see, so
range-limited numbers were indistinguishable from whole-trace figures
and could silently change between tool calls. The other sampling tools
always report whole-trace data; do the same here.
The heuristic reconstruction picks the call stack from the samples
within the zone's time span. Context switch samples are always parked
at the scheduler, so for zones which spent their time blocked they
dominated the root selection and the reconstructed stack showed the
scheduler path instead of the zone's real call stack. A zone covering
only context switch samples now reconstructs nothing instead.
Context switch samples are excluded from the sampling statistics, but
the flame graph built from thread samples included them. Threads which
spend time blocked accumulated large scheduler towers which none of the
other sampling views show, and the flame graph totals did not
correspond to the statistics for the same trace. Filter the samples the
same way the trace load jobs do.
The per-symbol exclusive counts reported by the tool come from the
sampling statistics, which exclude context switch samples, but the
total time reference was computed from the full sample count. This
inflated the total and made every symbol look proportionally cheaper
to the model. Use the same denominator as the statistics window.
The job which builds the symbol samples and child samples maps called
CompressThread, which updates the lookup cache and can insert into the
compression map, while the timeline processing job concurrently reads
the map, deliberately using the raw lookup to avoid this exact hazard.
All threads are already present in the compression data loaded from the
trace, so use the raw lookup as well.
When displaying the hottest inline function's name in place of the base
symbol name, the symbol map lookup was dereferenced without checking
for a missing entry, crashing in release builds. Keep the base symbol
name when the inline symbol has no symbol data.
Several code paths read the per-symbol sample lists, symbol statistics
or child sample data without checking the readiness flags. These
structures are populated by background jobs during trace load, and the
flags are the only synchronization mechanism, so reading early races
the jobs and trips the readiness asserts in debug builds. The find zone
samples list checked a different flag than the data it reads requires,
and two symbol view conditions called the accessor before the readiness
check made elsewhere in the same function. The trace information window
performed no check at all, while it is typically open during loading.
The denominator includes samples with unresolved call stacks and
samples belonging to filtered-out rows, so describing it as the number
of samples attributable to the displayed symbols was overstating what
the code does.
The button was gated on the entry stack maps being non-empty, but the
sample entry stacks window also needs the symbol data to display
anything. Without it, clicking the button silently closed the window
through its null guard. Match the gate used by the statistics window
menu items.
Since the inclusive count of an aggregated symbol entry is taken from
the base symbol's own statistics, a row whose base symbol could not be
resolved can display an inclusive count of zero while its inline
functions have nonzero counts. With the relative inline display active,
the per-inline percentages divided by the base count, printing infinity
in such cases. The time percentage variant divided by the same count
scaled by the sampling period, which cancels out, so both variants can
share the guarded reciprocal.
SortedVector considered an appended element equal to the last one to
break the ordering, marking the vector as unsorted. A non-decreasing
sequence is sorted, so only a strictly smaller element has to trigger
the marker. Equal keys are common: child sample vectors receive
identical timestamps whenever a recursive call stack contains the same
call site twice, which flipped the vectors to unsorted on virtually
every recursive workload and caused the lazy sort in GetChildSamples to
run over and over again while holding the data lock.
The lazy sort machinery handles duplicate keys correctly, as both the
prefix and tail merge windows are computed with lower bounds.
The denominator subtracts context switch sample counts from total
sample counts, both for the whole trace and within the range filter.
The subtraction could underflow: on traces with inconsistent sample
ordering the binary searches over the unsorted sample vectors can
return too few samples, and a sample which arrives with a duplicated
timestamp is merged into the existing entry while its context switch
classification is still recorded, so the context switch samples are not
a strict subset of the samples. An underflow made every percentage in
the table display as zero.
Clamp the subtraction, per thread in range mode, so that one thread
with bad data does not affect the counts of the others.
Context switch samples were appended in arrival order. Samples which
were postponed due to missing context switch data are replayed after
newer samples were already classified, so the vector could become
unordered. Everything that reads it assumes time order: the wait stacks
range filter, the sampling statistics percentage denominator, and the
context switch sample filters in the trace load jobs. In the load jobs
an unordered vector could silently disable the filtering for the rest
of a thread, reintroducing the context switch samples into the symbol
and child sample maps.
Use a SortedVector and restore the ordering at the points where it can
break: after the postponed sample replay and when saving a trace. The
save file version is bumped, so that the sort order check on load is
only performed for traces saved by previous versions.
The tool only reported "was executing" entry stacks, while the sample
entry stacks window now defaults to showing the stacks through which a
symbol was reached. The model was told there is no data for symbols
which never executed directly, even when the window right next to it
displayed their entry stacks.
Add an optional "mode" parameter with the "reached" (non-reentrant),
"reached_recursive" and "executing" values. The default matches the UI
default. Inline symbols are now accepted, as the reached maps cover
them directly, and the executing mode can use their exact statistics.
The reported mode is included in the tool output, and the chat view
shows it in the tool call label.
The tool description keeps the mode summaries short. The detailed
guidance on choosing a mode is in the optimization skill, so it only
occupies context when the skill is loaded.
LLM tool calls execute on the LLM worker thread, while the render
thread holds the worker data lock for the duration of each frame and
the network thread mutates worker data under the same lock during live
capture. The tools accessed worker data with no synchronization at all.
Some of the accessors also lazily modify worker state, such as the
postponed symbol list sorting or the lazy sorting of child sample
vectors, so this raced even in a fully loaded trace.
Take the data lock in the five tools which access worker state. No LLM
locks are held while tools execute, so no lock ordering cycle with the
render thread is possible. Tools which perform network requests do not
touch worker data and remain lockless, as holding the lock across a
network transfer would stall the profiler.
The background jobs which compute sampling statistics during trace load
do not hold the data lock. The readiness flags are the synchronization
mechanism there, so gate the sampling tools on them. This also matches
the readiness asserts in the worker accessors these tools call.
With an active range filter, the percentage denominator was a
theoretical capacity estimate: the number of samples a single thread
would produce if it ran continuously through the range at the nominal
sampling rate. Actual sample volume scales with total CPU occupancy
across all threads, so the estimate was off in either direction: on
multi-core workloads percentages were inflated several times over and
could exceed 100%, while on mostly idle workloads they were deflated.
Toggling the range filter also silently changed what the percentages
meant, as the whole-trace mode divides by the collected sample count.
Count the samples actually present in the range instead, and exclude
context switch samples from both denominators, as they cannot be
attributed to any displayed symbol. Percentages now mean the same thing
with and without a range filter: the share of attributable samples.
Time percentages are intentionally unchanged. They are normalized by
wall clock time in both modes, where exceeding 100% legitimately means
more than one core was busy.
During live capture, child samples were appended to their per-address
vectors in arrival order. Samples postponed due to missing context
switch data are replayed after newer samples were already processed, so
the vectors could become unordered. All range-limited queries binary
search these vectors by time and would silently return wrong results.
Inserting in sorted order at collection time would require a mid-vector
insertion for every stack frame of every replayed sample, so instead
the vectors are now SortedVector and are sorted lazily when accessed,
following what the inline symbol list already does. This also restores
proper query results for traces with inconsistent sample order.
Context switch samples are excluded from sampling statistics, as they
are not produced by the statistical profiling timer and their stacks
are always parked at the scheduler. During live capture this exclusion
is structural, as such samples never enter the statistics processing
path. On trace load, however, only the job which computes symbol
statistics and the instruction pointer map filtered them out. The job
which builds the per-symbol sample lists and the child sample map did
not, so on a loaded trace these two structures included tens of
thousands of context switch samples that a live session would not
count.
This made range-limited statistics counts exceed the whole-trace
counts, inflated child sample costs in the symbol view, and caused the
same trace to show different numbers live and after a save and reload.
On the test trace, __schedule reported 56 exclusive samples but 32747
entries in its sample list.
Apply the same context switch sample filter when building the symbol
samples and child samples maps.
When inline functions are aggregated into their base symbols in the
sampling statistics, the inclusive counts were summed up, the same as
the exclusive counts. This is wrong. Exclusive counts partition the
samples, as each sample is attributed to exactly one symbol. Inclusive
counts overlap: an inline function can only ever be on the stack within
a frame group whose base is its parent symbol, so every sample counted
for an inline function is also counted for the base symbol. Summing
therefore counted the same sample multiple times, inflating the counts,
the sort order and the displayed percentages, in the worst observed
case by a factor of 7.
The base symbol's own inclusive count is exactly the correct value for
the aggregated entry, so use it directly.
