fault-injection-loop

Skill: fault-injection-loop

Iterative closed-loop skill that runs a prioritized fault-injection experiment pool 2 jobs at a time, analyzes every artifact, scores attribution accuracy, aggregates gaps across the matrix, and proposes targeted improvements to attribution modules.

Overview

┌───────────────────────────────────────────────────────────────────────┐
│  0. POOL     → build ordered pool of (fault_type, rank, iter, nodes)  │
│               GPU faults first, then crash, Python-hang, signal       │
│                                                                        │
│  repeat until pool exhausted:                                          │

Step 0 — Fault Pool Design

The pool is defined as an ordered list of FAULT_TYPE:RANK:ITER:NODES entries inside scripts/prepare_node_alloc.sh. Default pool (34 experiments, 17 batches):

Rank coverage per node count (4 GPUs/node):

To run a custom subset, override POOL before calling the script:

Local User Config

Start from the tracked template:

Then edit scripts/user.env with cluster-specific settings. This file is sourced by run_session.sh, prepare_node_alloc.sh, watch_and_analyze.sh, l4_gb200_reduced.sh, and n3_super_gb200_fi.sh. It is required for this skill to run and is intended to stay local and untracked.

Recommended contents:

Use user.env for stable site defaults such as partition, container image, and host paths, plus local LLM credentials and endpoint settings for log-analysis and the judge. Use per-run environment overrides for experiment-specific controls such as POOL, WORKLOAD, BATCH_SIZE, FAULT_TYPE, FAULT_AT_ITER, or FAULT_DELAY.

If you use local Triton/Inductor cache staging, set the cache variables in scripts/user.env. See scripts/user.env.example for the supported ENABLE_NFS_CACHE_STAGING, NFS_TRITON_CACHE, and NFS_INDUCTOR_CACHE entries and workload-specific path examples.

Environment variables:

Registered workloads (scripts/workloads.conf)

Workload note:

• llama4_scout requires at least 2 nodes.
• n3_super requires at least 8 nodes. Its default registered pool contains only 8-node experiments.

Step 1 & 2 — Batched Submission + Wait (automated)

The script loops: submit 2 jobs → poll squeue every 30 s until both finish → submit next 2. Progress is printed inline:

A session directory and TSV tracking file are created at launch time:

Tracking file columns: JOB_ID FAULT_TYPE RANK ITER NODES EXPERIMENT_DIR

Step 3 — Analyze All Experiments

Run the watcher/analyzer — it reads the tracking file and processes each experiment as its job state leaves RUNNING/PENDING (works whether jobs are still running or already done):

The watcher:

1. Reads each row from the tracking TSV
2. Calls restart-agent as python -m nvidia_resiliency_ext.attribution.restart_agent.cli and parses the JSON decision
3. Calls FR analysis as python -m nvidia_resiliency_ext.attribution.trace_analyzer.fr_attribution --fr-path "${EXPERIMENT_DIR}/checkpoints" -p "_dump_*" and passes the raw table output to the judge
4. Scores 7 dimensions (restart correctness, rank primary, rank any, category, type, FR rank)
5. Appends a scored row to <session>_report.md as a markdown table row
6. Repeats until all experiments are analyzed

To also run the sub-skills interactively for a single experiment:

Step 4 — Score Each Experiment

Scoring is performed by scripts/score_attribution.py, an LLM judge that receives the ground truth, the filtered raw log, the restart-agent attribution output, and the FR analysis output, then returns structured JSON scores with a reasoning note.

The judge is given:

1. Ground truth: fault_type, rank, iter, nodes
2. Expected restart decision + rationale (derived from score_attribution.py:_RESTART_TABLE)
3. Filtered raw log (last 400 lines, with fault-injection markers removed)
4. Raw restart-agent JSON output
5. Raw FR analysis table output from fr_attribution.py --fr-path ... -p "_dump_*"
6. GPUS_PER_NODE and FR_SEGMENT_SIZE to map the injected rank to exact node-sized and segment-sized scopes for FR scoring

Default judge model: qwen/qwen3.5-397b-a17b. Override with --model in score_attribution.py. Default segment size for FR scope scoring: 32 ranks. Override with FR_SEGMENT_SIZE.

Step 5 — Aggregate Results

The canonical output of the loop is the markdown table in <session>_report.md. When summarizing results for users, prefer linking to that file and reproducing the same table shape rather than flattening the results into plain prose.

The report markdown table from watch_and_analyze.sh gives a matrix view. Look for patterns across rows:

Common failure mode patterns and their meaning:

Step 6 — Identify and Apply Improvements

FR analysis

Deterministic graph algorithm — do not modify automatically. Note misidentifications and escalate to the team.

Log analysis (safe to modify)

Editable area: attribution/restart_agent/

After each patch, re-run the same pool subset that previously failed:

Step 7 — Loop

Increment experiment counter. Suggested sweep order across code-change iterations:

1. Iteration 1: full default pool (34 experiments)
2. Iteration 2: targeted re-run of all failing cells from iteration 1
3. Iteration 3: expand iter dimension (FAULT_AT_ITER=2 and 10) for remaining gaps
4. Iteration 4: add SEGFAULT and LOCK_GIL 4-node/8-node coverage

Stop condition: all cells pass all four scoring dimensions for two consecutive code-change iterations.

Adapting A SLURM Script For The Feedback Loop

The feedback loop is not tied to l4_gb200_reduced.sh, but your sbatch script must match a small contract so the loop can submit, analyze, and score each run.

Required changes for a custom workload script:

1. Accept these exported variables from prepare_node_alloc.sh: FAULT_TYPE, FAULT_RANK, FAULT_AT_ITER, EXPERIMENT_DIR, BASE_EXPERIMENTS_DIR, and GPUS_PER_NODE.
2. Write the main training log to: ${EXPERIMENT_DIR}/logs/slurm/${SLURM_JOB_ID}.*.1.main_workload.log so watch_and_analyze.sh can find it.
3. Write NCCL flight-recorder dumps under ${EXPERIMENT_DIR}/checkpoints/.
4. Emit a fault-injection marker when the fault is injected. watch_and_analyze.sh uses this to decide whether the run reached the injection point.
5. Preserve the per-experiment directory layout: logs/slurm/, checkpoints/, and tensorboard/.

This has only been validated with Megatron-LM because the current run-valid check and fault markers depend on Megatron's debug_fault_injection.py behavior. If you adapt the loop to another framework, update both the sbatch script and watch_and_analyze.sh.

Appendix A: SBATCH_SCRIPT fault parameters

The example SBATCH_SCRIPT reads these env vars from prepare_node_alloc.sh via --export:

Valid FAULT_TYPE values: GPU_ERROR, GPU_SLEEP, WORKLOAD_EXC, ASYNC_EXC, SIGNAL_EXC, OS_ABORT, LOCK_GIL, SEGFAULT, SIGINT, SIGKILL, SIGTERM, SIGSTOP

Appendix B: Single-experiment manual run

Optional site-specific cleanup: