SortedVector considered an appended element equal to the last one to
break the ordering, marking the vector as unsorted. A non-decreasing
sequence is sorted, so only a strictly smaller element has to trigger
the marker. Equal keys are common: child sample vectors receive
identical timestamps whenever a recursive call stack contains the same
call site twice, which flipped the vectors to unsorted on virtually
every recursive workload and caused the lazy sort in GetChildSamples to
run over and over again while holding the data lock.
The lazy sort machinery handles duplicate keys correctly, as both the
prefix and tail merge windows are computed with lower bounds.
The denominator subtracts context switch sample counts from total
sample counts, both for the whole trace and within the range filter.
The subtraction could underflow: on traces with inconsistent sample
ordering the binary searches over the unsorted sample vectors can
return too few samples, and a sample which arrives with a duplicated
timestamp is merged into the existing entry while its context switch
classification is still recorded, so the context switch samples are not
a strict subset of the samples. An underflow made every percentage in
the table display as zero.
Clamp the subtraction, per thread in range mode, so that one thread
with bad data does not affect the counts of the others.
Context switch samples were appended in arrival order. Samples which
were postponed due to missing context switch data are replayed after
newer samples were already classified, so the vector could become
unordered. Everything that reads it assumes time order: the wait stacks
range filter, the sampling statistics percentage denominator, and the
context switch sample filters in the trace load jobs. In the load jobs
an unordered vector could silently disable the filtering for the rest
of a thread, reintroducing the context switch samples into the symbol
and child sample maps.
Use a SortedVector and restore the ordering at the points where it can
break: after the postponed sample replay and when saving a trace. The
save file version is bumped, so that the sort order check on load is
only performed for traces saved by previous versions.
The tool only reported "was executing" entry stacks, while the sample
entry stacks window now defaults to showing the stacks through which a
symbol was reached. The model was told there is no data for symbols
which never executed directly, even when the window right next to it
displayed their entry stacks.
Add an optional "mode" parameter with the "reached" (non-reentrant),
"reached_recursive" and "executing" values. The default matches the UI
default. Inline symbols are now accepted, as the reached maps cover
them directly, and the executing mode can use their exact statistics.
The reported mode is included in the tool output, and the chat view
shows it in the tool call label.
The tool description keeps the mode summaries short. The detailed
guidance on choosing a mode is in the optimization skill, so it only
occupies context when the skill is loaded.
LLM tool calls execute on the LLM worker thread, while the render
thread holds the worker data lock for the duration of each frame and
the network thread mutates worker data under the same lock during live
capture. The tools accessed worker data with no synchronization at all.
Some of the accessors also lazily modify worker state, such as the
postponed symbol list sorting or the lazy sorting of child sample
vectors, so this raced even in a fully loaded trace.
Take the data lock in the five tools which access worker state. No LLM
locks are held while tools execute, so no lock ordering cycle with the
render thread is possible. Tools which perform network requests do not
touch worker data and remain lockless, as holding the lock across a
network transfer would stall the profiler.
The background jobs which compute sampling statistics during trace load
do not hold the data lock. The readiness flags are the synchronization
mechanism there, so gate the sampling tools on them. This also matches
the readiness asserts in the worker accessors these tools call.
With an active range filter, the percentage denominator was a
theoretical capacity estimate: the number of samples a single thread
would produce if it ran continuously through the range at the nominal
sampling rate. Actual sample volume scales with total CPU occupancy
across all threads, so the estimate was off in either direction: on
multi-core workloads percentages were inflated several times over and
could exceed 100%, while on mostly idle workloads they were deflated.
Toggling the range filter also silently changed what the percentages
meant, as the whole-trace mode divides by the collected sample count.
Count the samples actually present in the range instead, and exclude
context switch samples from both denominators, as they cannot be
attributed to any displayed symbol. Percentages now mean the same thing
with and without a range filter: the share of attributable samples.
Time percentages are intentionally unchanged. They are normalized by
wall clock time in both modes, where exceeding 100% legitimately means
more than one core was busy.
During live capture, child samples were appended to their per-address
vectors in arrival order. Samples postponed due to missing context
switch data are replayed after newer samples were already processed, so
the vectors could become unordered. All range-limited queries binary
search these vectors by time and would silently return wrong results.
Inserting in sorted order at collection time would require a mid-vector
insertion for every stack frame of every replayed sample, so instead
the vectors are now SortedVector and are sorted lazily when accessed,
following what the inline symbol list already does. This also restores
proper query results for traces with inconsistent sample order.
Context switch samples are excluded from sampling statistics, as they
are not produced by the statistical profiling timer and their stacks
are always parked at the scheduler. During live capture this exclusion
is structural, as such samples never enter the statistics processing
path. On trace load, however, only the job which computes symbol
statistics and the instruction pointer map filtered them out. The job
which builds the per-symbol sample lists and the child sample map did
not, so on a loaded trace these two structures included tens of
thousands of context switch samples that a live session would not
count.
This made range-limited statistics counts exceed the whole-trace
counts, inflated child sample costs in the symbol view, and caused the
same trace to show different numbers live and after a save and reload.
On the test trace, __schedule reported 56 exclusive samples but 32747
entries in its sample list.
Apply the same context switch sample filter when building the symbol
samples and child samples maps.
When inline functions are aggregated into their base symbols in the
sampling statistics, the inclusive counts were summed up, the same as
the exclusive counts. This is wrong. Exclusive counts partition the
samples, as each sample is attributed to exactly one symbol. Inclusive
counts overlap: an inline function can only ever be on the stack within
a frame group whose base is its parent symbol, so every sample counted
for an inline function is also counted for the base symbol. Summing
therefore counted the same sample multiple times, inflating the counts,
the sort order and the displayed percentages, in the worst observed
case by a factor of 7.
The base symbol's own inclusive count is exactly the correct value for
the aggregated entry, so use it directly.
The statistics window can display rows for which the sample entry
stacks window cannot show anything: symbols with no symbol data,
symbols that only have range-mode sample counts without callstack
statistics, or any row while the statistics are still being computed
in the background. Grey out the menu item in such cases, following
what the "View symbol" item already does.
The button was displayed unconditionally, allowing the sample entry
stacks window to be opened for symbols without any sample data. Gate it
on the availability of callstack sample statistics and a non-empty
reached map, following what the call stack window does. Checking for
statistics readiness also prevents asserting when the button is used
while a trace is still being loaded in the background.
The window can be opened for a symbol which has no symbol stats or no
symbol data. For example, the statistics window can produce clickable
rows for symbols that are not present in the symbol map. Close the
window instead of dereferencing null pointers.
The instruction tooltip advertised the number of "was executing" entry
call stacks, but middle-clicking opens the sample entry stacks window,
which now defaults to the "was reached" mode. Count the non-reentrant
reached stacks instead, so the tooltip matches what the window shows.
Gating on the reached map also makes entry stacks accessible for
symbols which only ever appear deeper in the call stack. This can
happen when inspecting instructions whose cost consists purely of
child samples.
SymbolStats now has two additional entry stack maps, which record the
call stacks below every occurrence of a symbol in a sample, not only
when the symbol was at the top of the stack:
- wasReached counts each occurrence separately. If a symbol re-enters
itself through recursion, every re-entry adds an entry stack, which
will itself contain the symbol somewhere below.
- wasReachedNonReentrant counts only the outermost occurrence, so each
sample contributes exactly once and recursion is ignored. The sum of
counts in this map equals the symbol's inclusive sample count.
To achieve this, the call stack is walked bottom-up, which makes the
first encountered occurrence of a symbol the outermost one. Symbols at
inline positions get the remaining inline frames of their frame group
as a synthetic frame, mirroring what was already done for the top of
the stack.
The wasExecuting and wasExecutingBase maps are now filled by the same
walk, as its topmost-frame special case. The keys they receive are
identical to what the previous code produced.
Note that there is no "base" variant of the reached maps. A base symbol
is present as the last frame of every frame group that contains its
inline functions, so its wasReached map already covers the whole symbol
at base granularity. Merging in the inline symbols' maps, as done for
wasExecutingBase, would only multi-count the same samples.
A sample with a single-frame call stack already produces a zero-length parent
stack, and GetParentsCallstackFrameTree{BottomUp,TopDown} call cs.back() /
cs.front() on it, resulting in UB.
patchelf the stripped binary to /../lib, collapsing the RUNPATH
to the single intended entry. The wayland .pc files baked an absolute
build-host path and a trailing empty component into RUNPATH; the former
leaks the build environment and could shadow bundled libs on the target,
the latter can cause cwd library searches on older glibc.
Add patchelf to the appimage workflow's apt install list.
This update introduces a new function, GetThreadLockWaitTime, which computes the total wait time for locks held by a specific thread. The function aggregates wait times across all locks and updates the tooltip to display lock wait time and count, enhancing the profiling capabilities of the timeline item.
