npx skills add ...
npx skills add nvidia/nvidia-resiliency-ext --skill nvrx-attr
Orchestration layer over nvidia_resiliency_ext attribution modules. Provides log-analysis, fr-analysis, and a Megatron-LM-oriented fault-injection feedback loop for benchmarking attribution quality on SLURM workloads.
npx skills add nvidia/nvidia-resiliency-ext --skill nvrx-attr
High-level orchestration layer over the nvidia_resiliency_ext.attribution modules.
Each subdirectory is a self-contained skill with its own SKILL.md and helper scripts.
| Directory | Purpose | Entry point |
|---|---|---|
log-analysis/ | Analyze SLURM job logs for failure root-cause and restart decisions | RestartAgent (python -m nvidia_resiliency_ext.attribution.restart_agent.cli) |
fr-analysis/ | Analyze NCCL flight-recorder dumps for collective-hang root-cause | CollectiveAnalyzer (fr_attribution.py) |
fault-injection-loop/ | Run a batched SLURM fault-injection feedback loop and score attribution accuracy | prepare_node_alloc.sh / watch_and_analyze.sh |
Use the packaged restart_agent and trace_analyzer entry points for current
log and FR analysis. Legacy LogSage, SPLITLOG, and combined LogSage+FR tools live
under attribution/legacy_logsage/ for source-checkout workflows only.
LLM_API_KEY environment variable, LLM_API_KEY_FILE, or ~/.llm_api_keyBefore using fault-injection-loop/, create the local config file from the tracked
template and fill in your site-specific values:
The feedback-loop scripts require src/nvidia_resiliency_ext/skills/nvrx-attr/scripts/user.env
to exist at runtime. Keep user.env local and untracked.