launching-evals

NeMo Evaluator Skill

Quick Reference

nemo-evaluator-launcher CLI

# Run evaluation

Workflow

The complete evaluation workflow is divided into the following steps you should follow IN ORDER.

1. Create or modify a config using the nel-assistant skill. If the user provides a past run, use its config.yml artifact as a starting point.
2. Run the evaluation. See references/run-evaluation.md when executing this step.
3. Monitor progress (MANDATORY after every nel run): poll status repeatedly until SUCCESS/FAILED. See references/check-progress.md.
4. Post-run actions (when terminal state reached):
   1. When the evaluation status is SUCCESS, analyze the results. See references/analyze-results.md when executing this step.
   2. When the evaluation status is FAILED, debug the failed run. See references/debug-failed-runs.md when executing this step.

Key Facts

• Benchmark-specific info learned during launching/analyzing evals should be added to references/benchmarks/
• PPP = Slurm account / project portfolio code (the account field in cluster_config.yaml). When the user says "change PPP to X", update the account value (e.g., <old_account> → <new_account>).
• Slurm job pairs: NEL (nemo-evaluator-launcher) submits paired Slurm jobs — a RUNNING job + a PENDING restart job (for when the 4h walltime expires). Never cancel the pending restart jobs — they are expected and necessary.
• HF cache requirement: For configs with HF_HUB_OFFLINE=1, models must be pre-downloaded to the HF cache on each cluster before launching. Before running a model on a new cluster, always ask the user if the model is already cached there. If not, on the cluster login node: python3 -m venv hf_cli && source hf_cli/bin/activate && pip install huggingface_hub then HF_HOME=<your_hf_cache_path> hf download <model> (on lustre-style HPC clusters this is typically under /lustre/.../<group>/users/<username>/cache/huggingface). Without this, vLLM will fail with LocalEntryNotFoundError.
• data_parallel_size is per node: dp_size=1 with num_nodes=8 means 8 model instances total (one per node), load-balanced by haproxy. Do NOT interpret dp_size as the global replica count.
• payload_modifier interceptor: The params_to_remove list (e.g. [max_tokens, max_completion_tokens]) strips those fields from the outgoing payload, intentionally lifting output length limits so reasoning models can think as long as they need.
• Auto-export git workaround: The export container (python:3.12-slim) lacks git. When installing the launcher from a git URL, set auto_export.launcher_install_cmd to install git first (e.g., apt-get update -qq && apt-get install -qq -y git && pip install "nemo-evaluator-launcher[all] @ git+...#subdirectory=packages/nemo-evaluator-launcher").
• Do NOT use nemo-evaluator-launcher export --dest local — it only writes a summary JSON (processed_results.json), it does NOT copy actual logs or artifacts despite accepting --copy_logs and --copy-artifacts flags. nel info --copy-artifacts works but copies everything (very slow for large benchmarks). Preferred approach: use nel info to discover paths — if local, read directly; if remote, SSH to explore and rsync only what you need. Note that nel info prints standard artifacts but benchmarks produce additional artifacts in subdirs — explore to find them.