Replace the mutex-guarded empty check in OnBufferCompleted with an
std::atomic<bool> dirty flag. The mutex is now only acquired when
there is actual retirement work to do. Also update stale comment
on cudaGraphCurrentLaunch that said "let them leak".
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
operator[] on ConcurrentHashMap returns a reference after releasing the
read lock — the subsequent assignment happens with no lock held. This is
a latent data race if the map is ever accessed from multiple threads.
insert_or_assign performs the lookup and assignment atomically under a
single write lock, which is the correct pattern for a ConcurrentHashMap.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add `using GraphID = uint32_t` typedef and use it throughout for
graphId-typed variables (PersistentState, matchGraphActivityToAPICall,
getGraphIdFromRecord, retirement set, buffer loop).
- Move matchError from matchGraphActivityToAPICall to caller sites
(KERNEL, MEMCPY, MEMSET handlers). Keeping the error at the caller
provides more debugging context about which activity kind failed.
Remove the now-unnecessary `kind` parameter from the function.
- Replace insert_or_assign with operator[] assignment in
matchGraphActivityToAPICall. Access to graphLaunchCache is
single-threaded (CUPTI worker), so the simpler syntax is sufficient.
Remove the insert_or_assign method from ConcurrentHashMap entirely.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
cudaMalloc/cudaFree (and driver equivalents) were tracked in
cbidRuntimeTrackers/cbidDriverTrackers, creating a cudaCallSiteInfo
entry on each API call. But the MEMORY2 handler never calls
matchActivityToAPICall (and never calls EmitGpuZone) — it only needs
the address, size, and timestamp from the activity record itself. Since
no activity handler consumes these entries, they leaked indefinitely.
Remove the 6 memory API CBIDs from both tracker maps so no entry is
created. This eliminates the leak with no change in visible behavior:
the MEMORY2 handler already operates independently of cudaCallSiteInfo.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Without retirement, the cache grows by one entry per unique exec handle
ever launched and never shrinks. While bounded by the number of distinct
execs in the application, long-running programs creating and destroying
many exec handles accumulate stale entries indefinitely.
Retirement mechanism:
- At cudaGraphExecDestroy (ENTER, while handle is still valid): call
cuptiGetGraphExecId to translate exec handle → graphId and add to a
pending-retirement set. Works for both runtime (cudaGraphExecDestroy)
and driver (cuGraphExecDestroy) APIs. No new subscription needed —
the existing cuptiEnableDomain already routes all API callbacks here.
- Deferral in OnBufferCompleted: erasure is not done immediately because
cudaGraphExecDestroy does not wait for GPU completion. CUPTI may still
have undelivered activity records for the last launch in its internal
buffers. We defer the erase until a full buffer arrives that contains
no records bearing the retired graphId, indicating all in-flight
records have been delivered.
- getGraphIdFromRecord: new helper that extracts the graphId field from
CONCURRENT_KERNEL / MEMCPY / MEMSET activity records (the three kinds
that carry a graphId) for use in the per-buffer tracking.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Tests whether CUPTI recycles graphId values after cudaGraphExecDestroy,
which would be the only scenario where the graphLaunchCache in TracyCUDA
could serve stale entries for a non-matching exec handle.
Result (H100, CUDA 12, CUPTI): graphId is a monotonically increasing
counter that is never recycled. 22 create/instantiate/launch/destroy
cycles produced unique IDs ranging from 2 to 65 (incrementing by 3 per
cycle — one unit per node created during graph construction).
This confirms that the stale-cache concern raised in code review is not
a real risk in practice: two distinct exec handles always have distinct
graphIds.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Tests two questions:
1. Does relaunching the same cudaGraphExec produce a new correlationId
each time, or is it reused?
2. Do two different cudaGraphExec handles from the same cudaGraph share
a graphId?
Results on H100, CUDA 13.1:
- Each launch of the same exec handle gets a strictly unique, monotonically
increasing correlationId. CPU callback corrId == GPU activity corrId.
This is formally documented in cupti_activity.h:
"Each graph launch is assigned a unique correlation ID that is
identical to the correlation ID in the driver API activity record
that launched the graph."
- graphId identifies the exec handle (instantiation), not the graph
definition. Two cudaGraphInstantiate calls on the same graph produce
different graphIds.
These findings confirm that the cudaGraphCurrentLaunch cache in
matchGraphActivityToAPICall is always refreshed by the first activity
of each new launch before the graphId fallback path is ever used.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Calling matchActivityToAPICall in the MEMORY2 handler was consuming
the cudaCallSiteInfo entry for the graph launch correlationId. If a
graph mixes alloc nodes with kernel/memcpy nodes, all activities share
the same correlationId — consuming it here would cause
matchGraphActivityToAPICall to fail for the kernel/memcpy records that
follow, silently dropping their GPU zones.
Since apiCall is never used by the MEMORY2 handler (only address, size,
and timestamp from the activity record are needed), remove the call
entirely and leave the entry for the kernel/memcpy to consume and cache.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
CUpti_ActivityMemory3 has no graphId field, so graph-launched alloc
nodes and pre-profiling allocations can't be correlated to an API call.
The handler only needs address, size, and timestamp from the activity
record — apiCall is never used. Remove the early return so memory
tracking works in all cases.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
CUpti_ActivityMemory3 has no graphId field, so matchGraphActivityToAPICall
cannot be applied. Graph-launched cudaGraphAddMemAllocNode emits multiple
MEMORY2 records sharing the launch correlationId; only the first is
tracked, subsequent ones fire a spurious matchError.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Fix wrong comments on graph launch tracker entries: they claimed
correlation works "via CUPTI_ACTIVITY_KIND_GRAPH_TRACE", but that
approach was rejected (GRAPH_TRACE suppresses per-kernel records).
The actual mechanism is the shared correlationId across all nodes
in one graph launch.
- Fix ConcurrentHashMap::fetch() missing its read lock — a pre-existing
data race now exercised by the new graph correlation hot path.
- Cache PersistentState::Get().cudaGraphCurrentLaunch in a local ref
inside matchGraphActivityToAPICall instead of calling Get() twice.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The cudaGraphCurrentLaunch cache update was acquiring the write lock
twice (once for erase, once for emplace). Wrapping
std::unordered_map::insert_or_assign under a single write lock lets
the caller do it in one operation.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
NVCC 13.1 defaults to a PTX version incompatible with the installed
driver (580.105.08), causing kernels to silently fail with "provided
PTX was compiled with an unsupported toolchain". Use -arch=native so
NVCC auto-detects the target GPU (H100, sm_90) at build time.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The kernel, memcpy, and memset cases all had identical logic for
handling graph-launched activities. Extract it into a single helper
next to matchActivityToAPICall.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
CUPTI discovery: all kernels launched by one cuGraphLaunch share the
same correlationId as the launch call itself. GRAPH_TRACE was the
wrong approach — enabling it suppresses per-kernel CONCURRENT_KERNEL
records entirely, replacing them with graph-level summaries.
New approach:
- Drop CUPTI_ACTIVITY_KIND_GRAPH_TRACE (it conflicts with CONCURRENT_KERNEL)
- Drop two-pass buffer processing (no longer needed)
- On the first kernel/memcpy/memset from a graph launch, matchActivityToAPICall
succeeds (consuming the cuGraphLaunch entry) and the result is cached in
cudaGraphCurrentLaunch[graphId]
- Subsequent operations from the same launch find the cached entry via graphId
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The repro uses cudaGraphLaunch (runtime API) not cuGraphLaunch (driver
API). Add cudaGraphLaunch_v10000 and its _ptsz variant to
cbidRuntimeTrackers so that graphs launched via the runtime API also
get their CPU call site captured for GPU zone correlation.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace the synthetic APICallInfo hack with proper correlation via
CUPTI_ACTIVITY_KIND_GRAPH_TRACE. When cuGraphLaunch fires an API
callback, its correlationId is stored in cudaCallSiteInfo. The
GRAPH_TRACE activity record carries the same correlationId plus the
graphId, which lets us build a graphId→APICallInfo map. Kernel/memcpy/
memset activities then look up this map via their graphId field.
Key changes:
- Add cuGraphLaunch/cuGraphLaunch_ptsz to cbidDriverTrackers so the
API callback machinery captures the CPU call site
- Enable CUPTI_ACTIVITY_KIND_GRAPH_TRACE and handle it in
DoProcessDeviceEvent to populate cudaGraphCurrentLaunch[graphId]
- Add cudaGraphCurrentLaunch map to PersistentState
- Two-pass buffer processing in OnBufferCompleted so GRAPH_TRACE
records (which complete last on GPU) are processed before the
kernel/memcpy/memset records that depend on them
- Replace graphId=0 fallback in kernel/memcpy/memset with proper
cudaGraphCurrentLaunch lookup; fall through to matchError if
the graphId is not found
- Update repro to include TracyCUDA headers and properly test
GPU zone correlation
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Minimal reproducer showing that CUDA Graph-launched kernels produce
0 GPU zones in Tracy. The repro creates a simple graph (2 kernels +
1 memcpy), launches it 10 times, and expects ~30 GPU zones. Without
the fallback patch, all activity records are dropped by matchError().
Tested on NVIDIA H100, CUDA 13.1.
When kernels are launched via CUDA Graphs (cuGraphLaunch), CUPTI delivers
CONCURRENT_KERNEL, MEMCPY, and MEMSET activity records but no
corresponding API callback fires for the individual operations. This
means matchActivityToAPICall() always fails, and every GPU activity
record is silently dropped by matchError().
Fix this by falling back to a synthetic APICallInfo using the GPU
timestamps from the activity record when no API correlation exists.
This produces correct GPU zones with kernel names and timing — just
without the CPU-to-GPU launch correlation arrow.
Tested on NVIDIA H100 with CUDA 13.1: before this fix, 0 GPU zones
appeared for CUDA Graph workloads; after, all kernel and memcpy zones
are visible in the Tracy timeline.
