kermt-infer

kermt-infer

Run predictions with a finetuned KERMT checkpoint on a SMILES-only CSV. The skill is the workflow orchestrator: validate ckpt, validate CSV, prepare data, launch the runner blocking, return the predictions CSV.

Hardware requirements

• GPUs: 1 (single-GPU). Multi-GPU inference is not currently supported.
• VRAM: ≥ 4 GB for the default batch_size 32.
• Disk: a few hundred MB per run (cleaned CSV + features + predictions).
• Driver / CUDA: any host supporting CUDA 12.6 (the kermt image base).

Inputs

Required:

• --ckpt <path> — finetuned checkpoint (must have task FFN heads). The validator refuses pretrain ckpts with a redirect to kermt-finetune.
• --csv <path> — SMILES-only CSV. First column is smiles; other columns are ignored.

Optional:

• --batch-size N — override the configured default (32).
• --seed N — random seed for inference (deterministic featurization paths).
• --gpus 0 — single GPU id (default 0). Multi-GPU rejected.
• --from-prepare <dir> — skip the prepare step and reuse an existing prepare_data.json in <dir>.

Workflow

Let $KERMT_REPO be the path to your kermt repo checkout, and assume kermt-setup has built kermt:latest.

1. Pre-flight: ensure container + system probe.

   $KERMT_REPO/agent/scripts/kermt_container.sh check_system

   Refuse to proceed on ok: false.

2. Compute run directory.

   RUN_DIR=$KERMT_REPO/runs/infer_$(date -u +%Y-%m-%dT%H-%M-%SZ)

3. Validate the checkpoint.

   $KERMT_REPO/agent/scripts/kermt_container.sh run --ckpt <user-ckpt> -- \
       "python agent/scripts/check_checkpoint.py --mode inference --ckpt /ckpt"

   Parse the JSON. Abort on ok: false. The validator rejects pretrain ckpts (has_task_ffn: false) with a redirect to kermt-finetune.

4. Validate the data.

   $KERMT_REPO/agent/scripts/kermt_container.sh run --data <user-csv> -- \
       "python agent/scripts/check_data.py --mode inference --csv /data/<basename>"

   Abort on ok: false.

5. Prepare the data.

   $KERMT_REPO/agent/scripts/kermt_container.sh run --data <user-csv> --run-dir $RUN_DIR -- \
       "python agent/scripts/prepare_data.py --mode inference \\
            --csv /data/<basename> --out /runs/data"

   Outputs land at $RUN_DIR/data/prepare_data.json with clean_csv + clean_npz paths (rdkit_2d_normalized features).

6. Launch the runner (blocking).

   $KERMT_REPO/agent/scripts/kermt_container.sh run \\
       --ckpt <user-ckpt> --run-dir $RUN_DIR -- \\
       "python agent/scripts/run_inference.py \\
            --ckpt /ckpt \\
            --prepare-manifest /runs/data/prepare_data.json \\
            --out /runs \\
            [--gpus 0 --batch-size N --seed N]"

   Returns the predictions CSV path on success.

7. Report to the user. Output a short summary:

   • Predictions: $RUN_DIR/out/predictions.csv (smiles + per-target columns)
   • Manifest: $RUN_DIR/run.json (cmd_replay + image digest + applied args)
   • Log: $RUN_DIR/logs/inference.log
   • Row count: molecules predicted across targets

Hard rules

• Never modify the user's ckpt. The runner symlinks the ckpt into a unique <out>/ckpt_link/ subdir so main.py predict --checkpoint_dir picks it up; the source file stays untouched.
• Arch comes from the ckpt, never from CLI/defaults. The runner records the validator's arch block in run.json but does not pass arch flags into main.py predict — predict reads them from the loaded ckpt's saved_args.
• Single-GPU only. Multi-GPU inference is not currently supported.
• Echo applied defaults. The args_applied field of run.json records every flag's value + source (user / default-config). Surface a short summary of any default-filled flag.

Common errors

• inference requires a finetuned ckpt with task FFN heads → ckpt is a pretrain ckpt; use kermt-finetune first.
• prepare_data manifest reports ok=False → check the manifest errors for the failed step (typically clean_smiles or save_features).
• could not convert string to float: '<value>' from save_features or main.py predict → input CSV has a non-numeric passthrough column (e.g. a 'split' label). The prep step now strips the CSV to SMILES-only at inference; if this error still surfaces, the CSV is being read by a runner that bypassed prepare_data. Re-run via the skill, not main.py directly.
• --gpus '0,1' is single-GPU only → pass a single id.

Replayability

The run.json cmd_replay field is a single-line command that re-runs the inference with the same inputs. To replay inside the kermt container:

$(jq -r .cmd_replay $RUN_DIR/run.json)

If ok_to_replay: false (dirty kermt repo worktree at launch time), pin the commit via repo.commit and git checkout it first.