physical-ai-datasets

NVIDIA Physical AI Datasets (Hugging Face)

Purpose

Help the agent find, evaluate, and download an NVIDIA PhysicalAI-* dataset on Hugging Face that fits the user's downstream task — autonomous-vehicle reconstruction, robotics manipulation, GR00T post-training, spatial-intelligence research, grasping, or sim-ready content — and then hand off to the sibling skill (ncore, nre, asset-harvester, nurec-fixer) or upstream NVIDIA tool that actually consumes it.

Use this skill when: the user asks "is there an NVIDIA dataset for X?", "where do I get NCore / NuRec / GR00T sample data?", or is shopping the Hugging Face NVIDIA org for PhysicalAI-* collections.

Do NOT use this skill when:

• The user already knows the dataset and just wants to run a pipeline — jump straight to the consuming skill.
• The user needs a non-NVIDIA dataset (Waymo, nuScenes, KITTI, …) — this catalog is NVIDIA-only.
• The user wants to train Cosmos / GR00T / Isaac Sim itself — that's the upstream tool's job, not this catalog's.

Overview

Catalog of NVIDIA's open Physical AI dataset family on Hugging Face. Pick by task (Section 2 § lookup table) or family (Sections 3–10). Every entry lists: dataset path, size, format, license, gating, and the downstream skill in this repo that consumes it.

Source of truth: https://huggingface.co/nvidia (filter PhysicalAI-) and the curated Physical AI collection. When upstream cards drift, re-check the HF page; this skill mirrors the cards as of Apr 2026.

Prerequisites

• HuggingFace account with the dataset card opened in a browser at least once, and the gating checkbox accepted on every dataset you intend to download.
• HuggingFace user access token exported as HF_TOKEN (create at https://huggingface.co/settings/tokens).
• git, git-lfs, and huggingface_hub[cli] on PATH.
• Storage room sized to the dataset you're pulling (see the per-row size column; some are < 1 GB, the AV dataset is 133 TB — always pre-filter with --include or physical_ai_av).

Verifying secrets safely

Always check token presence with hf auth whoami or a length-only shell test; never write ad-hoc bash that interpolates HF_TOKEN values. The common one-liner

# BAD — leaks the secret to the terminal when the variable is set
echo "HF_TOKEN: ${HF_TOKEN:+yes}${HF_TOKEN:-no}"

prints yes<token-value> whenever HF_TOKEN is set, because ${VAR:-no} only falls back to "no" when VAR is empty — when set it expands to $VAR. Use one of these instead:

Rotate any token you suspect was echoed at https://huggingface.co/settings/tokens.

Table of Contents

1. Common download recipe — HF auth, gating, CLI.
2. Filtered AV download recipe — default hyperion_8.1 filter for PhysicalAI-Autonomous-Vehicles raw pulls.
3. Pick a dataset by task — fast lookup table.
4. Autonomous Vehicles — 5 datasets.
5. Robotics — Manipulation — 6 datasets.
6. Robotics — GR00T — 7 datasets.
7. Robotics — mindmap — 4 datasets.
8. Robotics — NuRec / Sim-Ready scenes — 2 datasets.
9. Robotics — Healthcare — 1 dataset.
10. Robotics — Grasping — 1 dataset.
11. Robotics — Physical / material properties — 2 datasets.
12. Spatial Intelligence + SimReady scenes — 5 datasets.
13. Community / sample — 1 dataset.
14. License decision tree — what you can do with each.
15. Cross-skill usage map — which skill consumes which dataset.

Common download recipe

All NVIDIA Physical AI datasets live on huggingface.co/datasets/nvidia/... and use the same access shape:

The token must:

1. Have a HF user account that's logged in to the dataset page in a browser AT LEAST ONCE.
2. Have accepted any license / terms-of-use checkbox the dataset shows (re-accept if the dataset has been re-gated — common for AV).

Three download patterns:

Special-case downloaders:

• PhysicalAI-Autonomous-Vehicles (133 TB) — use the official Python toolkit pip install physical_ai_av (NVlabs/physical_ai_av) to filter by sensor / country / split before downloading; otherwise you will pull TBs you don't need.

  Default policy: pre-filter to platform_class == 'hyperion_8.1' before any bulk pull. The dataset mixes two sensor rigs (hyperion_8 and hyperion_8.1) and the entire downstream NuRec / NCore / Asset-Harvester chain in this repo is only validated for hyperion_8.1. Roughly half of the 306k clips fall outside that platform, so the filter typically cuts the working set (and download size) in half.

  Apply this filter by default in any download script you generate. Only skip the filter when the user explicitly asks for it (e.g. "download all platforms", "include hyperion_8", "don't filter by platform") or when they hand you a specific clip UUID — in that single-clip case, download_clip_features(clip_id=...) is already scoped and no platform filter is needed.

  Canonical filtered recipe (see § Filtered AV download recipe below for an end-to-end example):

  from physical_ai_av import PhysicalAIAVDatasetInterface
   
  dataset = PhysicalAIAVDatasetInterface()
  dataset.download_metadata()
  dc = dataset.metadata['data_collection']
  hyperion_81_clip_ids = dc[dc['platform_class'] == 'hyperion_8.1'].index.tolist()

• PhysicalAI-Autonomous-Vehicle-Cosmos-Drive-Dreams (3 TB) — use the upstream download.py (nv-tlabs/Cosmos-Drive-Dreams) with --file_types {hdmap,lidar,synthetic} to select layers.

• PhysicalAI-SpatialIntelligence-Lyra-SDG (25 TB) — hf download --local-dir lyra_dataset/tar; untar each tar yourself.

• Spatial-Intelligence-Warehouse — chunked TAR-GZs need a manual loop after download (script provided in the upstream card).

For dataset filtering / preview: NVIDIA's Cosmos Dataset Search (CDS) lets you query a 41K subset of the AV dataset semantically before downloading.

Filtered AV download recipe

Always use this recipe (or a derivative of it) when the user asks to download raw clips from PhysicalAI-Autonomous-Vehicles. The hyperion_8.1 filter is the default; it matches every downstream skill in this repo (ncore, nre, asset-harvester, nurec-fixer).

When to skip the platform filter:

1. The user explicitly opts out — e.g. "download all platforms", "include hyperion_8 too", "ignore platform_class", or asks for a dataset-wide statistic. In that case, drop the platform mask and warn them that NuRec / NCore tooling will not work on the hyperion_8 clips.
2. The user gave you a specific clip UUID. A single-clip download_clip_features(clip_id=...) is already scoped, and filtering by platform_class for one clip is pointless.

Recipe (defaults: filter on; user may layer extra masks like country or sensor presence on top):

from physical_ai_av import PhysicalAIAVDatasetInterface
 
dataset = PhysicalAIAVDatasetInterface()
 
dataset.download_metadata()
dc = dataset.metadata['data_collection']
 
clip_mask = dc['platform_class'] == 'hyperion_8.1'
 
# Optional extra masks (only add when the user asked for them):
# clip_mask &= dc['country'] == 'US'
# sp = dataset.metadata['feature_presence']   # 26.03+; was 'sensor_presence' in 25.10
# clip_mask &= sp['lidar_top_360fov']
 
clip_ids = dc[clip_mask].index.tolist()
print(f"Downloading {len(clip_ids)} hyperion_8.1 clips")
 
dataset.download_clip_features(
    clip_id=clip_ids,
    features=["camera_front_wide_120fov", "lidar_top_360fov", "egomotion"],
    max_workers=8,
)

Single-clip fast path (no platform filter — the UUID is already specific):

dataset.download_clip_features(
    clip_id="<paste-clip-uuid>",
    features=["camera_front_wide_120fov", "lidar_top_360fov", "egomotion"],
)

Explicit opt-out (only when the user asked for it):

clip_ids = dc.index.tolist()   # NO platform filter — all 306k clips
# WARNING: downstream NuRec / NCore / Asset-Harvester only handle
# the hyperion_8.1 subset.

Pick a dataset by task

Autonomous Vehicles

PhysicalAI-Autonomous-Vehicles

The flagship real-world AV dataset.

PhysicalAI-Autonomous-Vehicles-NCore

Curated NCore V4 subset of the above.

PhysicalAI-Autonomous-Vehicles-NuRec

Pre-built NuRec dynamic neural reconstructions ready for IsaacSim / CARLA.

PhysicalAI-Autonomous-Vehicle-Cosmos-Drive-Dreams

Cosmos-Transfer-style synthetic + HD-map labels for diverse weather.

PhysicalAI-Autonomous-Vehicle-Cosmos-Synthetic

Robotics — Manipulation

All in LeRobot v2.x format unless noted, generated in IsaacSim with task-and-motion planning + scene_synthesizer procedural scenes + CuRobo motion generation.

PhysicalAI-Robotics-Manipulation-Kitchen

Bimanual Kinova Gen3 in procedurally-generated kitchens.

PhysicalAI-Robotics-Manipulation-Objects

Same kitchen environment, bimanual Kinova; pick / place bench / place cabinet.

PhysicalAI-Robotics-Manipulation-SingleArm

Franka Panda tabletop, procedurally generated.

PhysicalAI-Robotics-Manipulation-Augmented

Mimic-generated Franka cube-stacking, plus Cosmos-Transfer1 visual augmentation.

PhysicalAI-Robotics-Manipulation-Kitchen-Demos

Massive human-teleop dataset on Franka + Omron mobile base.

PhysicalAI-Robotics-Manipulation-Objects-Kitchen-MJCF

The MuJoCo XML assets that back the Kitchen-Demos environment.

Robotics — GR00T

GR00T = NVIDIA's generalist humanoid foundation model line. Most data is sim-generated for post-training; eval / real-robot supplements are small.

PhysicalAI-Robotics-GR00T-X-Embodiment-Sim

Largest GR00T post-training corpus.

PhysicalAI-GR00T-Tuned-Tasks

Two industrial post-training task families.

PhysicalAI-Robotics-GR00T-Teleop-G1

Real-robot Unitree G1 fruit pick-and-place.

PhysicalAI-Robotics-GR00T-Teleop-Sim

Simulated GR1 tabletop teleop.

PhysicalAI-Robotics-GR00T-Teleop-GR1

DreamDojo pretraining corpus — large-scale human egocentric video.

PhysicalAI-Robotics-GR00T-GR1

Lab-recorded GR1-T2 third-person video.

PhysicalAI-Robotics-GR00T-Eval

GR00T eval initial-state frames.

Robotics — mindmap

Spatial-memory benchmark from nvidia-isaac/nvblox_mindmap. Each dataset is one task with the same multimodal layout (RGB-D + camera intrinsics/poses + nvblox vertex features in .zst + robot state). All four are CC-BY-NC-4.0 (research only). Models trained: nvidia/PhysicalAI-Robotics-mindmap-Checkpoints.

Provided datasets ship 10 mindmap-formatted demos for storage reasons; regenerate the full 200/1000/250 with the upstream data-generation docs.

Paper: https://arxiv.org/abs/2509.20297. Codebase: https://github.com/nvidia-isaac/nvblox_mindmap.

Robotics — NuRec / Sim-Ready scenes

PhysicalAI-Robotics-NuRec

Indoor 3DGUT scenes for IsaacSim AMR simulation.

PhysicalAI-NuRec-PPISP

Photometric-variation benchmark for radiance-field methods.

Robotics — Healthcare

PhysicalAI-Robotics-Open-H-Embodiment

Surgical / ultrasound robotics multi-embodiment corpus.

Robotics — Grasping

PhysicalAI-Robotics-GraspGen

Sim2Real grasping at scale.

Robotics — Physical / material properties

VoMP = Volumetric Mechanical Properties.

PhysicalAI-Robotics-PhysicalAssets-VoMP

PhysicalAI-Robotics-PhysicalAssets-VoMP-Eval

Spatial Intelligence + SimReady scenes

PhysicalAI-SimReady-Warehouse-01

OpenUSD warehouse scene + asset library.

PhysicalAI-DigitalCousin-Assets

Companion-asset library for the GR1 tabletop sim environments.

PhysicalAI-SmartSpaces

Multi-camera tracking + 3D box benchmark (AI City Challenge).

PhysicalAI-Spatial-Intelligence-Warehouse

VLM-style spatial QA in warehouses.

PhysicalAI-SpatialIntelligence-Lyra-SDG

GEN3C-derived multi-view 3D + 4D training data for nv-tlabs/lyra.

(See also) PhysicalAI-NuRec-PPISP

Listed in § 7 but is also a spatial-intelligence radiance-field benchmark.

License decision tree

For internal NVIDIA use, the auto-derivable rule of thumb:

1. If Robotics-Manipulation-* and not mindmap / DigitalCousin-Assets → CC-BY-4.0 commercial OK.
2. If mindmap-* → research only (NC).
3. If Autonomous-Vehicles* → AV License only, gated.
4. Everything else → check the card.

Cross-skill usage map

Sibling skills in this hub are linked by relative path; upstream projects without an in-repo skill are linked by URL.

Verify

After downloading any dataset:

ls <local-dir>
du -sh <local-dir>          # confirm size matches the table above (within 10 %)
 
# For LeRobot v2.x:
python -c "from lerobot.common.datasets.lerobot_dataset import LeRobotDataset; \
           d = LeRobotDataset('<local-dir>'); print(d.meta.info)"
 
# For NCore V4 (.zarr.itar):
ncore_vis <local-dir>/clips/<uuid>/pai_<uuid>.json   # via ../ncore/SKILL.md
 
# For USDZ scenes:
# load in Isaac Sim 5.1 — File → Open → <scene>.usdz

GREEN when:

• File counts and total size match the dataset card to within ~10 %.
• For gated datasets, the download did not silently terminate at the license-agreement page (re-hf auth login if the first chunk is HTML).
• For LeRobot datasets, meta/info.json parses and the episode count matches the card.

Troubleshooting

• Repo gated. Cannot access … 401 — open the dataset URL in a browser, click Agree on the license / contact-info form, then retry with the same token. Tokens don't get auto-refreshed when a new agreement appears (re-accept after major version bumps).
• First chunk is 5 KB of HTML — same cause as above.
• Download hangs indefinitely on AV (133 TB) — you almost certainly don't want the whole thing. Use physical_ai_av to filter platform_class == 'hyperion_8.1' (default — see Filtered AV download recipe), plus any sensor / country / split mask, BEFORE pulling.
• AV clip downloaded but downstream NCore / NuRec / Asset-Harvester rejects it — double-check data_collection['platform_class'] for that clip; only hyperion_8.1 is validated. If it's hyperion_8, either swap to a hyperion_8.1 clip or accept that the downstream reconstruction tooling will not work.
• Cosmos-Drive-Dreams 3 TB on a small disk — pass --file_types synthetic (700 GB) or --file_types hdmap (small) to the official download.py.
• Lyra-SDG 25 TB out of disk — hf download --include "tar/static_*" --exclude "tar/dynamic_*" to take just the 3D half (or vice-versa).
• Cosmos-Synthetic looks empty (2.59 kB) — it's a pointer page redirecting to Cosmos-Drive-Dreams. Use that.
• mindmap dataset only has 10 demos when the card says 200 / 1000 — expected; regenerate the rest from the HDF5 via the upstream mindmap data-generation docs.
• AV license expired (12 months) — re-accept on the HF page; this is by-design per the License Agreement section 7.

Limitations

• Catalogue-only. This skill does not actually run any pipeline — it routes the agent to a sibling skill once the right dataset is chosen.
• NVIDIA-only. Non-NVIDIA datasets (Waymo, nuScenes, KITTI, PandaSet, …) are out of scope; the ncore skill covers how to convert those into NCore V4 manually.
• Gating is enforced by Hugging Face, not by this skill. Several datasets (e.g. PhysicalAI-Autonomous-Vehicles*, nvidia/Fixer-adjacent assets, mindmap large variants) require both license acceptance on the HF page and a valid HF_TOKEN. Acceptances expire (AV is annual) and must be re-clicked.
• Footprint can be enormous. PhysicalAI-Autonomous-Vehicles is ~133 TB; Lyra-SDG ~25 TB; Cosmos-Drive-Dreams ~3 TB. Always check free disk and use --include / --exclude filters before kicking off a full pull.
• The catalogue drifts. NVIDIA ships new PhysicalAI-* datasets regularly. If a dataset isn't in this skill, browse https://huggingface.co/nvidia?search_models=PhysicalAI- and consider opening a PR to update Section 2 / the relevant family.
• No GPU / runtime checks here. Storage and HF auth are the only prerequisites this skill validates; per-dataset compute needs belong to the consuming sibling skill.

Source links

• HF org (filter PhysicalAI-): https://huggingface.co/nvidia
• Curated collection (29 items): https://huggingface.co/collections/nvidia/physical-ai
• Cosmos Dataset Search (semantic preview of AV): https://build.nvidia.com/nvidia/cosmos-dataset-search
• AV Python toolkit: https://github.com/NVlabs/physical_ai_av
• NCore: https://github.com/NVIDIA/ncore
• Cosmos-Drive-Dreams downloader: https://github.com/nv-tlabs/Cosmos-Drive-Dreams/blob/main/scripts/download.py
• Lyra repo: https://github.com/nv-tlabs/lyra
• mindmap repo: https://github.com/nvidia-isaac/nvblox_mindmap
• Isaac-GR00T repo: https://github.com/NVIDIA/Isaac-GR00T
• DreamDojo project: https://dreamdojo-world.github.io/
• AI City Challenge eval: https://eval.aicitychallenge.org/**