nurec-fixer

NVIDIA DiffusionHarmonizer (NuRec post-processing)

Purpose

Run NVIDIA DiffusionHarmonizer on rendered images from neural reconstructions. DiffusionHarmonizer is a single-step, temporally-aware image diffusion enhancer for NeRF / 3DGS / NuRec-style renderings. It improves realism, reduces reconstruction artifacts, and harmonizes inserted dynamic objects with the surrounding scene.

When to Use / When NOT to Use

Use this skill when the user has rendered frames from NRE, NuRec, 3DGS, NeRF, or a similar reconstruction pipeline and wants to enhance, harmonize, evaluate, or optionally fine-tune the DiffusionHarmonizer model.

Do NOT use this skill when:

• The user wants to train or render the 3D reconstruction itself (use nre).
• The user wants to convert raw sensor data to NCore V4 (use ncore).
• The user wants a generic photo enhancer. DiffusionHarmonizer is tuned for neural-reconstruction artifacts and object-insertion failures.
• The user only wants NRE inline rendering with --enable-difix. That remains an NRE runtime feature; use the nre skill for the complete serve-grpc / render-grpc command shape.

What changed from the older Fixer skill

This skill follows the public NVIDIA/harmonizer release, not the older NGC JIT .pt artifact recipe. Use these public release artifacts:

• Code: https://github.com/NVIDIA/harmonizer
• Model: nvidia/Harmonizer on Hugging Face (the paper checkpoint models/diffusion_harmonizer.pkl), plus the base nvidia/Cosmos-Predict2-0.6B-Text2Image model that inference also requires.
• Checkpoint download: ./download_checkpoints.sh from the repo root. It fetches the Harmonizer checkpoints into models/ (diffusion_harmonizer.pkl, harmonizer_nontemporal.pt) and the base Cosmos DiT + tokenizer into src/checkpoints/nvidia/Cosmos-Predict2-0.6B-Text2Image/.
• Runtime: the harmonizer-cosmos-env image built from Dockerfile.cosmos (base nvcr.io/nvidia/pytorch:25.10-py3).
• Inference entry: src/inference_pix2pix_turbo_harmonizer.py, run from inside /work/src so it can import its sibling modules.
• Evaluation entry: src/evaluate_test_dataset.py
• Training entry: src/train_pix2pix_turbo_harmonizer.py

Do not use the obsolete standalone recipe that downloads nvidia/nre/nurec-fixer:cosmos_3dgut_fixer_harmonizer, mounts harmonizer_temporal.pt, or runs inference_jit_harmonizer.py inside nvcr.io/nvidia/pytorch:24.10-py3 unless the user explicitly asks for that older beta artifact. Do not run inference_pretrained_model.py; the current README documents inference_pix2pix_turbo_harmonizer.py as the inference entry point.

Background

DiffusionHarmonizer is described in DiffusionHarmonizer: Bridging Neural Reconstruction and Photorealistic Simulation with Online Diffusion Enhancer (arXiv 2602.24096, CVPR 2026). It distills a pretrained multi-step diffusion model into a single-step enhancer designed for online simulation and offline data cleanup.

Two operating modes:

• Offline: clean pseudo-training views rendered from a reconstruction, then distill the improved views back into the 3D representation.
• Online: enhance frames during simulation/inference by harmonizing color and lighting, reconstructing missing or inconsistent shadows for inserted actors, and reducing residual reconstruction artifacts.

The public model card describes DiffusionHarmonizer-cosmos-0.6B, a Cosmos Predict2 Diffusion Transformer post-trained at 576x1024 input and output resolution.

Inputs

• code_dir — checkout of https://github.com/NVIDIA/harmonizer.
• model_dir — checkpoints fetched by ./download_checkpoints.sh from the repo root. It places the paper checkpoint at models/diffusion_harmonizer.pkl and the required base Cosmos model under src/checkpoints/nvidia/Cosmos-Predict2-0.6B-Text2Image/.
• input_dir — directory of rendered RGB frames (.png, .jpg, .jpeg) to enhance.
• output_dir — inference_pix2pix_turbo_harmonizer.py does not take an output flag; it writes to a sibling folder named <input_dir>_<model_identifier> next to the input directory.
• HF_TOKEN — Hugging Face token with access to nvidia/Harmonizer, nvidia/Cosmos-Predict2-0.6B-Text2Image, and (if used) nvidia/Harmonizer-Dataset. Accept the model/dataset license terms first.
• NGC_API_KEY — often needed to authenticate docker pull from nvcr.io. Use only for container pulls, not model download.

Instructions

1. Validate the host. Have the agent execute scripts/validate_setup.py via its standard script runner — e.g. run_script("scripts/validate_setup.py") or python scripts/validate_setup.py. It checks Docker, the NVIDIA Container Toolkit, GPU architecture, git, the Hugging Face CLI, token presence, and free disk space — and exits non-zero on any missing prerequisite.

2. Clone the code and build (or pull) the runtime image. Full commands and the Blackwell patch caveat live in references/inference.md.

   git clone https://github.com/NVIDIA/harmonizer.git
   cd harmonizer
   docker build -t harmonizer-cosmos-env -f Dockerfile.cosmos .

3. Download the checkpoints. From the repo root run the helper, which fetches both the Harmonizer checkpoints and the base Cosmos model into the paths the code expects.

   export HF_TOKEN=<your-hugging-face-token>
   hf auth login --token "$HF_TOKEN"
   ./download_checkpoints.sh

   Verify models/diffusion_harmonizer.pkl and src/checkpoints/nvidia/Cosmos-Predict2-0.6B-Text2Image/ exist.

4. Confirm input_dir exists and filenames sort into frame order. The temporal inference script sorts frames with natural sort and uses previous outputs as references; prefer zero-padded names such as frame_000001.png.

5. Run inference inside the container with the repo mounted at /work, then cd /work/src and run inference_pix2pix_turbo_harmonizer.py. Pin -u $(id -u):$(id -g) so outputs are owned by the host user. Output frames land in <input_dir>_<model_identifier>. Full docker run recipe and flag matrix in references/inference.md.

6. Validate that the output frame count matches the input frame count, spot-check frames, and (if ground truth is available) run paired evaluation with PSNR/LPIPS — see references/evaluation.md.

7. (Optional) Train or fine-tune. Download the dataset (or prepare JSON manifests in the documented format), then run src/train_pix2pix_turbo_harmonizer.py with the recommended hyperparameters. For fine-tuning, initialize from the released checkpoint with --pretrained_path /path/to/diffusion_harmonizer.pkl. Full recipe + NuRec data-pair recipes in references/training.md.

8. (Optional) Teardown. Follow references/teardown.md to remove images, code clones, model weights, datasets, and outputs.

Examples

Example 1 — Enhance a folder of rendered frames

Run scripts/validate_setup.py, build the image once (see references/wrapper-image.md), then invoke inference_pix2pix_turbo_harmonizer.py inside the harmonizer-cosmos-env container with the repo checkout mounted at /work. From /work/src, point --input_image at the rendered frames, --model_path at /work/models/diffusion_harmonizer.pkl, set --model_identifier, and pass typical flags --timestep 250 --resolution 1024 --use_sched. Enhanced frames are written to <input_dir>_<model_identifier>. The canonical docker run command and the full flag matrix live in references/inference.md.

Example 2 — Quantitative PSNR / LPIPS evaluation

Prepare the paired test_dataset/{scene}/render + test_dataset/{scene}/gt layout, then run src/evaluate_test_dataset.py inside the container. See references/evaluation.md for the exact directory shape and the docker run command.

Example 3 — Fine-tune from the public checkpoint

Download nvidia/Harmonizer-Dataset, prepare the training JSON, then run src/train_pix2pix_turbo_harmonizer.py with the multi-GPU accelerate launch recipe in references/training.md. For fine-tuning add --pretrained_path /path/to/diffusion_harmonizer.pkl and use --fixing_data_weight 3 on the released dataset.

Example 4 — Non-temporal (frame-by-frame) enhancement

inference_pix2pix_turbo_harmonizer.py is temporal by default and uses previous enhanced frames as references (--offset_list -1 -2 -3 -4). When the user wants each frame enhanced independently (e.g. unordered images), add --nontemporal to disable temporal conditioning. Full command + --offset_list defaults in references/inference.md.

Consuming the inline Fixer via NRE

nre --enable-difix is still the right answer when the user wants NRE to enhance frames as part of rendering without a separate harmonizer checkout. That path is owned by the sibling nre skill. Do not mix the standalone DiffusionHarmonizer HF/Cosmos workflow with NRE's internal cache flags unless the NRE documentation for the user's tag explicitly says they share weights.

Use this standalone skill when the user wants the public DiffusionHarmonizer code, model card, training/evaluation scripts, or post-processing of frames that already exist on disk.

Prerequisites

• OS: Linux host.
• GPU / driver: NVIDIA GPU Ampere or newer (compute capability >= 8.0; A100, A10, L40, H100, RTX 30/40/PRO, B200, GB200).
• Container runtime: Docker with the NVIDIA Container Toolkit.
• Tools: git, python3, Hugging Face CLI (hf or huggingface-cli).
• Secrets:
  • HF_TOKEN with the nvidia/Harmonizer and nvidia/Cosmos-Predict2-0.6B-Text2Image licenses accepted (required to download model weights and the optional dataset).
  • NGC_API_KEY (often required for docker login nvcr.io before pulling nvcr.io/nvidia/pytorch:25.10-py3).
• Disk: at least ~120 GB free for the runtime image, build cache, model weights, optional dataset, and outputs combined.
• Source / model / dataset:
  • Code: https://github.com/NVIDIA/harmonizer.
  • Model: https://huggingface.co/nvidia/Harmonizer.
  • Base model: https://huggingface.co/nvidia/Cosmos-Predict2-0.6B-Text2Image.
  • Optional dataset: https://huggingface.co/datasets/nvidia/Harmonizer-Dataset.

The fail-fast check that enforces all of the above is scripts/validate_setup.py.

Scripts

References

• references/inference.md — container build, raw-base fallback, Blackwell patches, checkpoint download, inference_pix2pix_turbo_harmonizer.py flag matrix, non-temporal mode.
• references/evaluation.md — paired test_dataset/ layout and evaluate_test_dataset.py command.
• references/training.md — dataset download, training JSON format, multi-GPU accelerate launch recipe, fine-tuning flags, NuRec data-pair recipes.
• references/wrapper-image.md — build and run the project image for repeat inference.
• references/troubleshooting.md — extended diagnostic notes.
• references/teardown.md — cleanup inventory for images, code, Hugging Face caches, datasets, and outputs.
• Public code: https://github.com/NVIDIA/harmonizer
• Model card: https://huggingface.co/nvidia/Harmonizer
• Dataset: https://huggingface.co/datasets/nvidia/Harmonizer-Dataset
• Paper: https://arxiv.org/abs/2602.24096
• Project page: https://research.nvidia.com/labs/sil/projects/diffusion-harmonizer/

Limitations

• Rendered inputs only. The model is tuned for neural-reconstruction renderings and object-insertion artifacts, not arbitrary real photos.
• Primary model resolution is 576x1024. The inference script maps resolution key 1024 to 1024x576. Only 1024, 960, and 1360 are supported keys; 1024 matches the model-card operating point.
• Temporal references and filename order matter. The inference script enhances frames in natural-sorted order and feeds previous outputs back as temporal references. Use zero-padded frame numbers, or pass --nontemporal for unordered images.
• Container builds can be large. The runtime image, build cache, model weights, and optional dataset can exceed 100 GB.
• Training is multi-GPU by default. The README command assumes 8 GPUs with bf16 mixed precision.
• Public code evolves. The current README documents inference_pix2pix_turbo_harmonizer.py as the inference entry point; if a future checkout renames it, prefer the script and flags present in that checkout.

Troubleshooting (top 5)

Full matrix in references/troubleshooting.md.

Teardown

A full workflow can leave large artifacts on disk: the Cosmos image, project image, build cache, harmonizer code checkout, Hugging Face model weights, optional dataset, evaluation outputs, and enhanced frames. Reclaim them with the inventory in references/teardown.md. Do not revoke HF_TOKEN or NGC_API_KEY as normal cleanup. Rotate a token only if you suspect it was leaked.