OverviewHistoryStatsSecurity
npx skills add ...
Documentation
SKILL.md
npx skills add nvidia/tensorrt-llm --skill perf-torch-sync-free
Identify and eliminate host-device synchronizations in PyTorch code. Detects sync points (.item(), .cpu(), boolean indexing, torch.tensor on CUDA), classifies false vs true dependencies, provides sync-free alternatives. Triggers: sync-free, synchronization, .item(), .cpu(), host-device sync, eliminate syncs, CPU stall, non_blocking, set_sync_debug_mode, cudaStreamSynchronize, cudaEventSynchronize, remove syncs, async GPU.
npx skills add nvidia/tensorrt-llm --skill perf-torch-sync-free
Sync-free code means the CPU continuously queues work to the GPU without waiting for GPU operations to complete. When host-device synchronizations are eliminated, the GPU works continuously without idle stalls.
Every host-device synchronization ultimately calls one of three CUDA driver APIs that block the CPU thread:
cuEventSynchronize -- CPU waits until a specific GPU event completescuStreamSynchronize -- CPU waits until all work on a stream finishescuCtxSynchronize -- CPU waits until all work across all streams finishesReach for this skill when you encounter:
.item() or
.cpu() calls that block the CPU, or understand why specific PyTorch
operations cause synchronizationcudaStreamSynchronize in nsys profiles,
warnings from torch.cuda.set_sync_debug_mode, training throughput
limited by CPU-GPU round-trips, .item() or .cpu() calls in hot loopsDo NOT use this skill for:
perf-torch-cuda-graphs instead)perf-nsight-compute-analysis or perf-nsight-systems)kernel-triton-writing)torch.compile| Dependency | Version | Notes |
|---|---|---|
| PyTorch | >=2.0 | With CUDA support |
| NVIDIA GPU | Any | CUDA-capable |
| Nsight Systems | Optional | For comprehensive sync detection via nsys |
Use one or both methods to find sync points in the code.
Quick detection -- PyTorch sync debug mode prints a warning with stack trace on every synchronization:
This mode only detects syncs going through PyTorch's wrapped
cuStreamSynchronize. Third-party libraries calling CUDA sync APIs
directly are not detected.
Comprehensive detection -- Nsight Systems captures all sync calls including those from extensions and libraries:
In the Nsight Systems GUI, check the CUDA API timeline row and search
for cudaStreamSynchronize, cudaEventSynchronize, or
cudaDeviceSynchronize. The call stack panel shows which Python line
triggered each sync.
After detecting syncs, classify each one before deciding how to fix it.
False dependencies (avoidable) -- CPU does not actually need the GPU result. These can be eliminated without changing program logic:
print(loss.item())).item() calls for logging that could be deferred.cuda() instead of .to('cuda', non_blocking=True).type(torch.LongTensor) instead of .type(torch.long)True dependencies (require restructuring) -- CPU genuinely needs the GPU value to proceed:
if loss.item() > threshold: -- CPU
branches on a GPU-computed valueoutput = x[mask] -- output size depends
on GPU computationTrue dependencies require restructuring: move logic to GPU
(torch.where()), delay to end of iteration, or accept that those parts
stay outside any CUDA Graph capture region.
Apply fixes in order of increasing difficulty. Start with easy wins.
1. Remove redundancy -- Delete operations that do not need to happen:
.item() calls2. Use non_blocking=True -- Make transfers async where CPU does not
immediately use the result:
Only use non_blocking=True for GPU-to-CPU when the CPU does not
immediately read the result. Otherwise the CPU may operate on incomplete
data.
3. Switch to sync-free API alternatives -- See the Quick Reference Table below for a condensed mapping of common patterns.
4. Delay synchronization to end of iteration -- Move logging and validation to after the optimizer step rather than mid-forward/backward:
5. Coalesce multiple syncs into one -- If you need several GPU values on CPU, gather them and transfer once:
6. Offload logic to GPU -- Replace CPU-side logic with GPU-native ops:
7. Exclude unavoidable syncs from capture range (last resort) -- If a sync cannot be eliminated, keep it outside the CUDA Graph capture region and graph only the sync-free sections. Partial graphing is better than no graphing.
Re-run detection to confirm syncs are eliminated:
Or re-profile with Nsight Systems and confirm no cudaStreamSynchronize /
cudaEventSynchronize / cudaDeviceSynchronize calls appear in the
target region.
| Sync-Inducing Pattern | Sync-Free Alternative |
|---|---|
| Device Transfers | |
.cpu() or .to('cpu') | .to('cpu', non_blocking=True) (fire-and-forget only) |
.cuda() or .to('cuda') | .to('cuda', non_blocking=True) |
.type(torch.LongTensor) | .type(torch.long) (dtype conversion, stays on GPU) |
| Tensor Creation | |
torch.tensor(obj, device='cuda') | Create on CPU, then .to('cuda', non_blocking=True) |
torch.tensor(0, device='cuda') | torch.zeros(1, device='cuda', dtype=...).squeeze() |
torch.as_tensor(arr, device='cuda') | Create on CPU, then .to('cuda', non_blocking=True) |
torch.cuda.BoolTensor(list) | torch.tensor(list, device='cpu').to('cuda', non_blocking=True) |
| Control Flow | |
.item() in conditionals | torch.where() or move outside critical region |
if gpu_tensor: | Keep logic on GPU with torch.where() |
Python max(a, b) on GPU tensors | torch.max(a, b) |
torch.is_nonzero(t) | Avoid; use GPU-side comparisons |
| Indexing | |
x_gpu[idx_cpu] or x_gpu[idx_list] | x_gpu[idx_gpu] (keep indices on same device) |
x_gpu[idx] = 0 (scalar assignment) | x_gpu[idx] = zero_gpu (GPU tensor value) |
x[i:j] with CUDA tensor bounds | x[:, s] with s = torch.arange(i, j, device='cuda') |
| Dynamic Shapes | |
x_gpu[mask_gpu] (masked selection) | torch.where(mask_gpu, x_gpu, 0) (fixed shape) |
torch.nonzero(mask) | torch.where() or move outside critical region |
torch.masked_select(x, mask) | torch.where(mask, x, 0) |
torch.unique(x) | Avoid in hot path; precompute if possible |
torch.repeat_interleave(x, r) | Specify output_size=N if known |
references/sync-patterns.md): Comprehensive pattern catalog
with 9 categories, full code examples showing sync-inducing and sync-free
versions, and the specific CUDA driver API triggered by each patternnsys profile --capture-range=cudaProfilerApi \
--python-sampling=true \
--backtrace=dwarf \
python your_script.py# Before (syncs)
x_gpu = x_cpu.cuda()
x_cpu = x_gpu.cpu()
# After (async, no sync)
x_gpu = x_cpu.to('cuda', non_blocking=True)
x_cpu = x_gpu.to('cpu', non_blocking=True) # only if CPU does not use x_cpu immediately# Before: sync mid-iteration
loss = model(batch)
print(f"Loss: {loss.item()}") # cuStreamSynchronize
loss.backward()
# After: delay to end of iteration
loss = model(batch)
loss.backward()
optimizer.step()
print(f"Loss: {loss.item()}") # sync is outside the hot path# Before: 3 separate syncs
loss_val = loss.item() # cuStreamSynchronize
acc_val = accuracy.item() # cuStreamSynchronize
gnorm_val = grad_norm.item() # cuStreamSynchronize
# After: 1 sync
metrics = torch.stack([loss, accuracy, grad_norm])
vals = metrics.cpu() # single cuStreamSynchronize
loss_val, acc_val, gnorm_val = vals.tolist()# Before: CPU control flow (syncs)
if loss.item() > threshold:
result = a
else:
result = b
# After: GPU-side selection (no sync)
result = torch.where(loss > threshold, a, b)
# Before: Python max (syncs)
val = max(x_gpu[0, 0], x_gpu[0, 1])
# After: torch.max (no sync)
val = torch.max(x_gpu[0, 0], x_gpu[0, 1])torch.cuda.set_sync_debug_mode('error') # will raise if any sync remains
train_step(model, batch)
torch.cuda.set_sync_debug_mode(0)