install-profilers

OverviewHistoryStatsSecurity

npx skills add nvidia/omniperf --skill install-profilers

Documentation

SKILL.md

npx skills add ...

Documentation

SKILL.md

Install profiling tools for Isaac Sim / Isaac Lab / Kit-based applications. Covers Nsight Systems (`nsys` CLI), `sqlite3`, Tracy `csvexport`, canonical Tracy `capture`/`capture-release`, and `update` for memory strip tests, with optional `tracy-capture`/`tracy-update` aliases. Use when setting up a profiling environment, when nsys/sqlite3/csvexport/capture/update tools are missing, or before running profiling, nsys-analyze, or tracy-memory.

npx skills add nvidia/omniperf --skill install-profilers

Documentation

SKILL.mdView source

Install Profilers for Omniverse / Kit Apps

Quick Check — What's Already Installed?

nsys --version 2>/dev/null && echo "nsys: OK" || echo "nsys: MISSING"
csvexport --help 2>/dev/null && echo "csvexport: OK" || echo "csvexport: MISSING"
CAPTURE_BIN=$(command -v capture || command -v capture-release || command -v tracy-capture)
[ -n "$CAPTURE_BIN" ] && echo "capture: OK ($CAPTURE_BIN)" || echo "capture: MISSING"
UPDATE_BIN=$(command -v update || command -v tracy-update)
[ -n "$UPDATE_BIN" ] && echo "update: OK ($UPDATE_BIN)" || echo "update: MISSING (optional; needed for memory strip tests)"
sqlite3 --version 2>/dev/null && echo "sqlite3: OK" || echo "sqlite3: MISSING"

Install only what's missing.

1. Nsight Systems (nsys CLI)

The nsys CLI captures GPU/CPU traces and exports .nsys-rep → SQLite.

Option A: Standalone installer from NVIDIA (recommended)

The profiling guide recommends the latest standalone Nsight Systems package because CUDA Toolkit packages may lag behind.

# Check the latest version at https://developer.nvidia.com/nsight-systems
wget https://developer.nvidia.com/downloads/assets/tools/secure/nsight-systems/2026_2/nsight-systems-2026.2.1_2026.2.1.210-1_amd64.deb
sudo dpkg -i nsight-systems-*.deb
nsys --version

For non-Debian Linux, use the standalone .run installer from the same download page:

chmod +x NsightSystems-linux-public-*.run
sudo ./NsightSystems-linux-public-*.run --accept

Option B: From NVIDIA CUDA apt repo (fallback)

If the CUDA apt repo is already configured (check ls /etc/apt/sources.list.d/*cuda*):

# List available versions
apt-cache search nsight-systems-20 | sort -V
 
# Install the newest available package from the repo
sudo apt-get update
NSYS_PKG="nsight-systems-<version-from-apt-cache>"
sudo apt-get install -y "$NSYS_PKG"
 
# Verify
nsys --version

The package installs to /opt/nvidia/nsight-systems/<version>/bin/nsys. It typically creates a symlink at /usr/local/bin/nsys, but if not:

sudo ln -sf /opt/nvidia/nsight-systems/*/bin/nsys /usr/local/bin/nsys

Option C: Add CUDA repo first (if not present)

# Ubuntu 22.04 x86_64
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.1-1_all.deb
sudo dpkg -i cuda-keyring_1.1-1_all.deb
sudo apt-get update
 
# Then install the newest available package as in Option B
apt-cache search nsight-systems-20 | sort -V
NSYS_PKG="nsight-systems-<version-from-apt-cache>"
sudo apt-get install -y "$NSYS_PKG"

For other distros, replace ubuntu2204/x86_64 with your platform. See: https://developer.nvidia.com/cuda-downloads (select "deb (network)").

Option D: CLI-only (headless servers)

sudo apt-get install -y nsight-systems-cli
# Smaller package, no GUI — just the nsys command

Post-install: perf_event_paranoid

nsys needs sampling access. Check and fix:

cat /proc/sys/kernel/perf_event_paranoid
# If > 2:
sudo sh -c 'echo 2 > /proc/sys/kernel/perf_event_paranoid'
# Persistent (survives reboot):
sudo sh -c 'echo kernel.perf_event_paranoid=2 > /etc/sysctl.d/99-nsys.conf'

Container note: perf_event_paranoid may be read-only in containers. nsys can still trace CUDA/NVTX but won't collect CPU IP samples.

Container note: --gpu-metrics-devices may fail with ERR_NVGPUCTRPERM ("Insufficient privilege"). Drop this flag in containers — NVTX/CUDA tracing still works fine without GPU hardware counters.

2. sqlite3 (for nsys SQLite exports)

nsys export --type=sqlite creates a .sqlite file from .nsys-rep traces. Use sqlite3 to query NVTX events, CUDA kernels, and GPU metrics.

sudo apt-get install -y sqlite3
sqlite3 --version

Export smoke test

# Export .nsys-rep to SQLite
nsys export --type=sqlite -o profile.sqlite profile.nsys-rep
 
# Confirm SQLite can read the export
sqlite3 profile.sqlite ".tables"

SQLite schema reference (key tables)

Table	Contents
`NVTX_EVENTS`	NVTX ranges/markers — use `text` or join `textId→StringIds.id` for names
`CUPTI_ACTIVITY_KIND_KERNEL`	CUDA kernel launches (empty for Kit/RTX apps — normal)
`CUPTI_ACTIVITY_KIND_MEMCPY`	CUDA memcpy operations
`TARGET_INFO_GPU`	GPU hardware info
`TARGET_INFO_SYSTEM_ENV`	System environment
`StringIds`	String lookup table for `textId` foreign keys

Gotcha: NVTX_EVENTS has NO name column — use text (inline string) or join on textId.

3. Tracy Profiler Tools (`csvexport`, `capture`, `update`)

Tracy is used by Kit/Isaac Sim when --/app/profilerBackend=tracy is set. You need two tools for the standard flow: capture (record traces) and csvexport (export to CSV for analysis). For memory profiling strip tests, also expose Tracy's update tool. tracy-capture and tracy-update are acceptable local aliases, but capture / capture-release / update are the source-guide names.

Option A: Use the bundled Kit binary (best protocol match)

Building Isaac Sim from source, or another Kit-based app, downloads omni.kit.profiler.tracy, which ships a matching Tracy capture binary. Look under:

exts/omni.kit.profiler.tracy/
extscore/omni.kit.profiler.tracy/
extscache/omni.kit.profiler.tracy/

Using the bundled binary guarantees version compatibility with the Tracy protocol embedded in Kit.

Option B: Build Tracy 0.11.1 from source (recommended fallback)

Before building, check Kit's all-deps.packman.xml for the carb_sdk_plugins version so the capture protocol matches the profiled app:

`carb_sdk_plugins` version	Tracy version
`< 178`	`0.9.1` legacy protocol
`>= 178`	`0.11.1+nv1` current protocol

The commands below build Tracy v0.11.1, which matches current Kit builds using carb_sdk_plugins >= 178. For older Kit builds, check out the matching legacy Tracy tag instead.

sudo apt-get install -y build-essential cmake git libcapstone-dev
 
git clone https://github.com/wolfpld/tracy.git
cd tracy
git checkout v0.11.1
 
# Build csvexport (headless, no GUI deps)
cmake -B csvexport/build -S csvexport -DCMAKE_BUILD_TYPE=Release -DNO_ISA_EXTENSIONS=ON
cmake --build csvexport/build --parallel
sudo cp csvexport/build/csvexport /usr/local/bin/csvexport
 
# Build capture using the source-guide path
make -C capture/build/unix release
sudo cp capture/build/unix/capture-release /usr/local/bin/capture
sudo ln -sf /usr/local/bin/capture /usr/local/bin/tracy-capture  # optional compatibility alias
 
# Build update (optional; required by the tracy-memory strip test)
cmake -B update/build -S update -DCMAKE_BUILD_TYPE=Release
cmake --build update/build --parallel
sudo cp update/build/update /usr/local/bin/update
sudo ln -sf /usr/local/bin/update /usr/local/bin/tracy-update  # optional compatibility alias
 
# Verify
csvexport --help
capture --help
update --help

Note: The GUI profiler (tracy-profiler) requires libglfw3-dev libdbus-1-dev libfreetype-dev and a display. Skip on headless servers — csvexport + sqlite3 queries are sufficient for automated analysis.

Option C: Ubuntu 25.04+ packages (if available)

# Available in Ubuntu 25.04+ universe repo
sudo apt-get install -y tracy-csvexport tracy-capture

Not available on Ubuntu 22.04/24.04 from default repos.

For Tracy capture, CSV export usage, shutdown handling, and analysis handoff, use the profiling skill. This skill only installs and verifies the capture/export binaries.

Verification Checklist

After installation, verify the full toolchain:

echo "=== Profiling Toolchain ==="
nsys --version 2>/dev/null        || echo "MISSING: nsys"
sqlite3 --version 2>/dev/null     || echo "MISSING: sqlite3"
csvexport --help 2>&1 | head -1   || echo "MISSING: csvexport (Tracy)"
CAPTURE_BIN=$(command -v capture || command -v capture-release || command -v tracy-capture)
[ -n "$CAPTURE_BIN" ] && "$CAPTURE_BIN" --help 2>&1 | head -1 || echo "MISSING: capture"
UPDATE_BIN=$(command -v update || command -v tracy-update)
[ -n "$UPDATE_BIN" ] && "$UPDATE_BIN" --help 2>&1 | head -1 || echo "MISSING: update (optional; needed for tracy-memory)"
echo "perf_event_paranoid: $(cat /proc/sys/kernel/perf_event_paranoid 2>/dev/null || echo 'N/A')"

These tools are needed for the full profiling workflow:

nsys — capture Nsight Systems traces
sqlite3 — query nsys SQLite exports
csvexport — export Tracy .tracy files to CSV
capture / capture-release — record Tracy traces from Kit apps
update — strip/transform Tracy captures for memory-capture verification

npx skills add nvidia/omniperf --skill install-profilers

Documentation

SKILL.md

npx skills add nvidia/omniperf --skill install-profilers

Documentation

SKILL.mdView source

Install Profilers for Omniverse / Kit Apps

Quick Check — What's Already Installed?

nsys --version 2>/dev/null && echo "nsys: OK" || echo "nsys: MISSING"
csvexport --help 2>/dev/null && echo "csvexport: OK" || echo "csvexport: MISSING"
CAPTURE_BIN=$(command -v capture || command -v capture-release || command -v tracy-capture)
[ -n "$CAPTURE_BIN" ] && echo "capture: OK ($CAPTURE_BIN)" || echo "capture: MISSING"
UPDATE_BIN=$(command -v update || command -v tracy-update)
[ -n "$UPDATE_BIN" ] && echo "update: OK ($UPDATE_BIN)" || echo "update: MISSING (optional; needed for memory strip tests)"
sqlite3 --version 2>/dev/null && echo "sqlite3: OK" || echo "sqlite3: MISSING"

Install only what's missing.

1. Nsight Systems (nsys CLI)

The nsys CLI captures GPU/CPU traces and exports .nsys-rep → SQLite.

Option A: Standalone installer from NVIDIA (recommended)

The profiling guide recommends the latest standalone Nsight Systems package because CUDA Toolkit packages may lag behind.

# Check the latest version at https://developer.nvidia.com/nsight-systems
wget https://developer.nvidia.com/downloads/assets/tools/secure/nsight-systems/2026_2/nsight-systems-2026.2.1_2026.2.1.210-1_amd64.deb
sudo dpkg -i nsight-systems-*.deb
nsys --version

For non-Debian Linux, use the standalone .run installer from the same download page:

chmod +x NsightSystems-linux-public-*.run
sudo ./NsightSystems-linux-public-*.run --accept

Option B: From NVIDIA CUDA apt repo (fallback)

If the CUDA apt repo is already configured (check ls /etc/apt/sources.list.d/*cuda*):

# List available versions
apt-cache search nsight-systems-20 | sort -V
 
# Install the newest available package from the repo
sudo apt-get update
NSYS_PKG="nsight-systems-<version-from-apt-cache>"
sudo apt-get install -y "$NSYS_PKG"
 
# Verify
nsys --version

The package installs to /opt/nvidia/nsight-systems/<version>/bin/nsys. It typically creates a symlink at /usr/local/bin/nsys, but if not:

sudo ln -sf /opt/nvidia/nsight-systems/*/bin/nsys /usr/local/bin/nsys

Option C: Add CUDA repo first (if not present)

# Ubuntu 22.04 x86_64
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.1-1_all.deb
sudo dpkg -i cuda-keyring_1.1-1_all.deb
sudo apt-get update
 
# Then install the newest available package as in Option B
apt-cache search nsight-systems-20 | sort -V
NSYS_PKG="nsight-systems-<version-from-apt-cache>"
sudo apt-get install -y "$NSYS_PKG"

For other distros, replace ubuntu2204/x86_64 with your platform. See: https://developer.nvidia.com/cuda-downloads (select "deb (network)").

Option D: CLI-only (headless servers)

sudo apt-get install -y nsight-systems-cli
# Smaller package, no GUI — just the nsys command

Post-install: perf_event_paranoid

nsys needs sampling access. Check and fix:

cat /proc/sys/kernel/perf_event_paranoid
# If > 2:
sudo sh -c 'echo 2 > /proc/sys/kernel/perf_event_paranoid'
# Persistent (survives reboot):
sudo sh -c 'echo kernel.perf_event_paranoid=2 > /etc/sysctl.d/99-nsys.conf'

Container note: perf_event_paranoid may be read-only in containers. nsys can still trace CUDA/NVTX but won't collect CPU IP samples.

Container note: --gpu-metrics-devices may fail with ERR_NVGPUCTRPERM ("Insufficient privilege"). Drop this flag in containers — NVTX/CUDA tracing still works fine without GPU hardware counters.

2. sqlite3 (for nsys SQLite exports)

nsys export --type=sqlite creates a .sqlite file from .nsys-rep traces. Use sqlite3 to query NVTX events, CUDA kernels, and GPU metrics.

sudo apt-get install -y sqlite3
sqlite3 --version

Export smoke test

# Export .nsys-rep to SQLite
nsys export --type=sqlite -o profile.sqlite profile.nsys-rep
 
# Confirm SQLite can read the export
sqlite3 profile.sqlite ".tables"

SQLite schema reference (key tables)

Table	Contents
`NVTX_EVENTS`	NVTX ranges/markers — use `text` or join `textId→StringIds.id` for names
`CUPTI_ACTIVITY_KIND_KERNEL`	CUDA kernel launches (empty for Kit/RTX apps — normal)
`CUPTI_ACTIVITY_KIND_MEMCPY`	CUDA memcpy operations
`TARGET_INFO_GPU`	GPU hardware info
`TARGET_INFO_SYSTEM_ENV`	System environment
`StringIds`	String lookup table for `textId` foreign keys

Gotcha: NVTX_EVENTS has NO name column — use text (inline string) or join on textId.

3. Tracy Profiler Tools (`csvexport`, `capture`, `update`)

Option A: Use the bundled Kit binary (best protocol match)

Building Isaac Sim from source, or another Kit-based app, downloads omni.kit.profiler.tracy, which ships a matching Tracy capture binary. Look under:

exts/omni.kit.profiler.tracy/
extscore/omni.kit.profiler.tracy/
extscache/omni.kit.profiler.tracy/

Using the bundled binary guarantees version compatibility with the Tracy protocol embedded in Kit.

Option B: Build Tracy 0.11.1 from source (recommended fallback)

Before building, check Kit's all-deps.packman.xml for the carb_sdk_plugins version so the capture protocol matches the profiled app:

`carb_sdk_plugins` version	Tracy version
`< 178`	`0.9.1` legacy protocol
`>= 178`	`0.11.1+nv1` current protocol

The commands below build Tracy v0.11.1, which matches current Kit builds using carb_sdk_plugins >= 178. For older Kit builds, check out the matching legacy Tracy tag instead.

sudo apt-get install -y build-essential cmake git libcapstone-dev
 
git clone https://github.com/wolfpld/tracy.git
cd tracy
git checkout v0.11.1
 
# Build csvexport (headless, no GUI deps)
cmake -B csvexport/build -S csvexport -DCMAKE_BUILD_TYPE=Release -DNO_ISA_EXTENSIONS=ON
cmake --build csvexport/build --parallel
sudo cp csvexport/build/csvexport /usr/local/bin/csvexport
 
# Build capture using the source-guide path
make -C capture/build/unix release
sudo cp capture/build/unix/capture-release /usr/local/bin/capture
sudo ln -sf /usr/local/bin/capture /usr/local/bin/tracy-capture  # optional compatibility alias
 
# Build update (optional; required by the tracy-memory strip test)
cmake -B update/build -S update -DCMAKE_BUILD_TYPE=Release
cmake --build update/build --parallel
sudo cp update/build/update /usr/local/bin/update
sudo ln -sf /usr/local/bin/update /usr/local/bin/tracy-update  # optional compatibility alias
 
# Verify
csvexport --help
capture --help
update --help

Note: The GUI profiler (tracy-profiler) requires libglfw3-dev libdbus-1-dev libfreetype-dev and a display. Skip on headless servers — csvexport + sqlite3 queries are sufficient for automated analysis.

Option C: Ubuntu 25.04+ packages (if available)

# Available in Ubuntu 25.04+ universe repo
sudo apt-get install -y tracy-csvexport tracy-capture

Not available on Ubuntu 22.04/24.04 from default repos.

For Tracy capture, CSV export usage, shutdown handling, and analysis handoff, use the profiling skill. This skill only installs and verifies the capture/export binaries.

Verification Checklist

After installation, verify the full toolchain:

echo "=== Profiling Toolchain ==="
nsys --version 2>/dev/null        || echo "MISSING: nsys"
sqlite3 --version 2>/dev/null     || echo "MISSING: sqlite3"
csvexport --help 2>&1 | head -1   || echo "MISSING: csvexport (Tracy)"
CAPTURE_BIN=$(command -v capture || command -v capture-release || command -v tracy-capture)
[ -n "$CAPTURE_BIN" ] && "$CAPTURE_BIN" --help 2>&1 | head -1 || echo "MISSING: capture"
UPDATE_BIN=$(command -v update || command -v tracy-update)
[ -n "$UPDATE_BIN" ] && "$UPDATE_BIN" --help 2>&1 | head -1 || echo "MISSING: update (optional; needed for tracy-memory)"
echo "perf_event_paranoid: $(cat /proc/sys/kernel/perf_event_paranoid 2>/dev/null || echo 'N/A')"

These tools are needed for the full profiling workflow:

nsys — capture Nsight Systems traces
sqlite3 — query nsys SQLite exports
csvexport — export Tracy .tracy files to CSV
capture / capture-release — record Tracy traces from Kit apps
update — strip/transform Tracy captures for memory-capture verification

install-profilers

Documentation

Documentation

install-profilers

Documentation

Install Profilers for Omniverse / Kit Apps

Quick Check — What's Already Installed?

1. Nsight Systems (nsys CLI)

Option A: Standalone installer from NVIDIA (recommended)

Option B: From NVIDIA CUDA apt repo (fallback)

Option C: Add CUDA repo first (if not present)

Option D: CLI-only (headless servers)

Post-install: perf_event_paranoid

2. sqlite3 (for nsys SQLite exports)

Export smoke test

SQLite schema reference (key tables)

3. Tracy Profiler Tools (csvexport, capture, update)

Option A: Use the bundled Kit binary (best protocol match)

Option B: Build Tracy 0.11.1 from source (recommended fallback)

Option C: Ubuntu 25.04+ packages (if available)

Verification Checklist

Documentation

Documentation

Install Profilers for Omniverse / Kit Apps

Quick Check — What's Already Installed?

1. Nsight Systems (nsys CLI)

Option A: Standalone installer from NVIDIA (recommended)

Option B: From NVIDIA CUDA apt repo (fallback)

Option C: Add CUDA repo first (if not present)

Option D: CLI-only (headless servers)

Post-install: perf_event_paranoid

2. sqlite3 (for nsys SQLite exports)

Export smoke test

SQLite schema reference (key tables)

3. Tracy Profiler Tools (csvexport, capture, update)

Option A: Use the bundled Kit binary (best protocol match)

Option B: Build Tracy 0.11.1 from source (recommended fallback)

Option C: Ubuntu 25.04+ packages (if available)

Verification Checklist

3. Tracy Profiler Tools (`csvexport`, `capture`, `update`)

3. Tracy Profiler Tools (`csvexport`, `capture`, `update`)