OverviewHistoryStatsSecurity
npx skills add ...
Documentation
SKILL.md
npx skills add nvidia/megatron-lm --skill mcore-migrate-gpt-to-hybrid
Migration guide for moving Megatron Core GPTModel checkpoints, model providers, training commands, and layer mappings to HybridModel, including the mechanical steps for transferring an existing pretrain_gpt.py launch script.
npx skills add nvidia/megatron-lm --skill mcore-migrate-gpt-to-hybrid
docs/user-guide/hybrid-model-migration.md.The canonical document specifies what the migrated command must contain. This section covers how to edit a working script into it without silent breakage. Apply the edits in order.
1. Entrypoint. pretrain_gpt.py → pretrain_hybrid.py. When the
entrypoint comes from a shell variable or a wrapper, follow it to the real
invocation.
2. Generate the pattern instead of typing it. A 96-layer model needs a 192-character pattern; hand-typing invites a silent off-by-one.
With pipeline segments (seg must divide n, and the segment count must be
divisible by --pipeline-model-parallel-size):
3. Replace --num-layers N with --hybrid-layer-pattern. Deleting
--num-layers matters: leaving a stale value is only a warning, so it looks
healthy while being silently overridden by the pattern-derived count.
4. Add the stack spec, replacing any GPT --spec rather than adding a
second one.
5. Delete the pipeline-layout arguments the parser rejects, and repoint
--save at a new directory. See the canonical document for both lists.
Most scripts under examples/ collect arguments in arrays and expand them
unquoted:
Unquoted ${ARR[@]} re-runs word-splitting and pathname expansion on every
element, so the pattern is globbed against the launch directory at expansion
time — single-quoting it where the array is defined does not protect it:
An unmatched glob survives intact, so this passes by luck in most working directories and fails only when some file happens to match. Store the pattern in a variable and quote that array's expansion:
Both checks are cheap and catch the common slips:
Then diff the migrated script against the original: it should contain the edits above and nothing else.
A *- or *E transfer changes the layer indexing, not the model. On a
measured 8-block dense run (2 GPUs, bf16, seq 4096, 100 iterations, identical
seed and data), pretrain_gpt.py --num-layers 8 and pretrain_hybrid.py --hybrid-layer-pattern '*-*-*-*-*-*-*-*-' produced:
HybridModel: ... layers='*-*-*-*-*-*-*-*-' (16 layers) from the allocator;Treat a systematic loss offset, a parameter-count difference, or a throughput gap beyond noise as a migration bug, not as expected behavior. Note that per-iteration wall clock early in a run is dominated by dataset-cache warmup, so compare steady-state iterations only.
If the implementation and migration guide disagree:
n=32; seg=4; per=$((n/seg))
b=$(printf "$blk%.0s" $(seq $per)); pat=$b
for ((i=1;i<seg;i++)); do pat="$pat|$b"; donetorchrun ${DISTRIBUTED_ARGS[@]} pretrain_gpt.py ${MODEL_ARGS[@]}touch 'a-b-'; ARGS=(--hybrid-layer-pattern '*-*-')
printf '[%s]\n' ${ARGS[@]} # -> [--hybrid-layer-pattern] [a-b-] silently corrupted
printf '[%s]\n' "${ARGS[@]}" # -> [--hybrid-layer-pattern] [*-*-] correctHYBRID_PATTERN=$(printf '*-%.0s' $(seq $NUM_LAYERS))
MODEL_ARGS=( ... --hybrid-layer-pattern "$HYBRID_PATTERN" ... )
torchrun "${DISTRIBUTED_ARGS[@]}" pretrain_hybrid.py "${MODEL_ARGS[@]}" ...# 1. No rejected or stale arguments survived -- must print nothing.
grep -nE -- '--(num-layers|num-layers-per-virtual-pipeline-stage|num-virtual-stages-per-pipeline-rank|pipeline-model-parallel-layout|account-for-embedding-in-pipeline-split|account-for-loss-in-pipeline-split|hybrid-override-pattern|fim-data)\b' train_hybrid.sh
# 2. Pattern shape -- attn and mlp must each equal the source GPT layer count.
p='*-*-|*-*-'
main=${p%%/*}; main=${main//|/}
attn=${main//[^\*]/}; mlp=${main//[^-E]/}; segs=${p%%/*}; segs=${segs//[^|]/}
echo "layers=${#main} attn=${#attn} mlp=${#mlp} segments=$(( ${#segs} + 1 ))"