OverviewHistoryStatsSecurity
npx skills add ...
Documentation
SKILL.md
npx skills add nvidia/skills --skill foundationpose-setup
Install or repair the FoundationPose perception pipeline and build its FoundationStereo TensorRT engines. Use for SAM3/TAO dependency conflicts, CUDA library failures, and depth-engine shape or precision decisions.
npx skills add nvidia/skills --skill foundationpose-setup
Prepare the FoundationPose perception pipeline
for depth, segmentation, and pose inference. Environment installation and engine construction
belong here; dataset adaptation, inference, and pose evaluation belong to
foundationpose-pipeline when that skill is installed.
Work from the product checkout, not the installed skill directory. Find the user's checkout by
checking for pyproject.toml (project foundationpose-perception-pipeline),
tools/build_tao_engine.py, and config/defaults.yaml. If absent and setup was requested, clone
the product URL above into the user's workspace and enter it. For advice-only requests, use the
supplied diagnostics without cloning or installing anything. Commands below use paths relative
to the product root; references/ links are relative to this skill.
Read the checkout's README.md Requirements and Install sections for the matching revision.
The supported stack requires Linux x86_64, glibc >= 2.38, GLIBCXX_3.4.31, NVIDIA driver >= 580,
a CUDA toolkit >= 12.8 with nvcc, Python 3.12, uv, Git, Docker with GPU access, wget, and unzip.
Start GPU sizing at 24 GB and measure the densest scene; 32 GB was tested. Budget depth-cache
disk as roughly width * height * 4 * 3 bytes per scene, plus predictions and models.
Keep sibling directories for sam3/, foundation-pose-inference-library/, and models/ beside
the product checkout. models/ contains the deployable ONNX and engine, not FoundationStereo source.
Preflight before installing. Check glibc, GLIBCXX, driver, nvcc, tools, and Docker GPU
access. An Ubuntu 22.04 host with glibc 2.35 cannot load the shipped FoundationPose library;
report the unsupported runtime and stop setup there. Do not replace system libc or try to
solve this with LD_LIBRARY_PATH. See installation.
Install into the product's Python 3.12 venv. Follow
installation for uv, SAM3, the FoundationPose
build, and TAO Deploy. On a fresh venv use uv sync --extra foundationpose; on an existing
venv use uv sync --inexact --extra foundationpose to preserve out-of-band packages.
Verify checkpoint access. SAM3 is gated at Hugging Face; an existing authorized token or usable cached checkpoint is sufficient. Request user action only if access is missing. FoundationPose and the documented FoundationStereo export are public Hugging Face downloads; credential hunting is not the first response to a network failure.
Prepare the depth engine. Read engine construction. Use the
user's ONNX location or the sibling models/ directory. Adapt the dataset before measuring
the engine shape; use tools/bop_adapt/adapt.py --config <profile> --src <source> as described
in the checkout's README Dataset adaptation section. Only registered adapters are supported.
Build with --shape-from-scene on an adapted scene and FP32 unless the user requests a
precision experiment. Set overrides.depth.engine in config/<profile>.yaml.
Set runtime paths and verify. From the product root:
The first three verify components; the last also needs an adapted dataset. Expect
backend=tao, normalization=imagenet, the intended fixed shape, and no cropping N rows
warning. An unloaded or unavailable engine is an incomplete verification, not a pass.
| Symptom | Action |
|---|---|
GLIBC_2.38 not found | Use a supported OS/runtime; a venv or library search path cannot upgrade host libc. |
libcudart.so.13 missing | Check the product venv runtime wheels and absolute library paths before retrying pose. |
| SAM3 breaks after sync | Use --inexact; confirm numpy 1.26.x and reinstall the sibling SAM3 package if pruned. |
pycuda build cannot find cuda.h | Check the CUDA toolkit, nvcc on PATH, or CUDA_ROOT. |
| TAO import or dependency conflict | Use TAO Deploy 7.1.0 with --no-deps; sync declared dependencies with --inexact. |
| Engine sidecar mismatch or cropping | Rebuild for this GPU, TensorRT version, precision, and adapted scene shape. |
Engines are machine-specific and must not be committed. With no dataset, download the ONNX and report shape-dependent engine construction and scene validation as pending; do not invent a rig shape. The pipeline's Apache license does not cover separately downloaded model weights; retain their upstream terms and SAM3's access requirements.
Report which preflight, install, checkpoint, engine, and verification steps actually passed, the checkout and engine paths, versions used, and remaining blockers. Do not equate installation or a smoke check with measured pose accuracy.