log-analysis

Skill: log_analysis

Analyze a SLURM job log file for failure root-cause attribution and restart decisions using RestartAgent.

Command: python -m nvidia_resiliency_ext.attribution.restart_agent.cli

Reads the log file (UTF-8, falls back to latin-1).
Builds deterministic L0 evidence from training progress, failure lines, checkpoint state, and known restart-policy signals.
Optionally runs L1 model enrichment to explain the primary failure and related evidence.
Applies the restart-agent retry/recovery policy and returns a structured STOP or RESTART decision.

python -m nvidia_resiliency_ext.attribution.restart_agent.cli \
    /path/to/job.log \
    [--job-id JOB_ID] \
    [--cycle-id CYCLE_ID] \

Flag	Default	Description
`log_path`	required	Path to the job log file
`--job-id`	unset	Optional job id for request metadata and retry history
`--cycle-id`	unset	Optional cycle id for retry history
`--config`	unset	Versioned restart-agent config for configured model routes and retry policy
`--llm-model`	environment/default config	L1 enrichment model when `--config` is not used
`--llm-base-url`	environment/default config	OpenAI-compatible base URL when `--config` is not used
`--llm-api-key-file`	environment/default config	File containing the L1 API key
`--disable-l1`	off	Run deterministic-only analysis
`--summary`	off	Emit a compact human-readable summary to stderr in addition to JSON stdout

The CLI prints one JSON object to stdout. Important fields include:

Field	Meaning
`decision`	`RESTART` or `STOP`
`decision_basis`	Policy reason for the decision
`retry_policy`	Retry budget and selected rule details
`primary_failure`	Primary failure evidence, when identified
`secondary_failures`	Related or cascading failure evidence
`evidence_coverage`	Which evidence sources were checked/found
`justification`	Human-readable explanation for the final decision

LLM_API_KEY set (env var, LLM_API_KEY_FILE, or ~/.llm_api_key) for L1 enrichment
Use --disable-l1 when you want deterministic-only analysis without model access

Analyze a SLURM job log file for failure root-cause attribution and restart decisions using RestartAgent.

Command: python -m nvidia_resiliency_ext.attribution.restart_agent.cli

Reads the log file (UTF-8, falls back to latin-1).
Builds deterministic L0 evidence from training progress, failure lines, checkpoint state, and known restart-policy signals.
Optionally runs L1 model enrichment to explain the primary failure and related evidence.
Applies the restart-agent retry/recovery policy and returns a structured STOP or RESTART decision.

python -m nvidia_resiliency_ext.attribution.restart_agent.cli \
    /path/to/job.log \
    [--job-id JOB_ID] \
    [--cycle-id CYCLE_ID] \

Flag	Default	Description
`log_path`	required	Path to the job log file
`--job-id`	unset	Optional job id for request metadata and retry history
`--cycle-id`	unset	Optional cycle id for retry history
`--config`	unset	Versioned restart-agent config for configured model routes and retry policy
`--llm-model`	environment/default config	L1 enrichment model when `--config` is not used
`--llm-base-url`	environment/default config	OpenAI-compatible base URL when `--config` is not used
`--llm-api-key-file`	environment/default config	File containing the L1 API key
`--disable-l1`	off	Run deterministic-only analysis
`--summary`	off	Emit a compact human-readable summary to stderr in addition to JSON stdout

The CLI prints one JSON object to stdout. Important fields include:

Field	Meaning
`decision`	`RESTART` or `STOP`
`decision_basis`	Policy reason for the decision
`retry_policy`	Retry budget and selected rule details
`primary_failure`	Primary failure evidence, when identified
`secondary_failures`	Related or cascading failure evidence
`evidence_coverage`	Which evidence sources were checked/found
`justification`	Human-readable explanation for the final decision

LLM_API_KEY set (env var, LLM_API_KEY_FILE, or ~/.llm_api_key) for L1 enrichment
Use --disable-l1 when you want deterministic-only analysis without model access