k8s-launch-kit-generate

l8k: Manifest Generation

PREREQUISITE: Read ../k8s-launch-kit-shared/SKILL.md for install paths, global flags, and output modes.

This workflow uses the default host target; --target host is equivalent. The CLI snapshots generation arguments and runs the concrete Host generation operation through the target registry; command syntax and artifacts are unchanged.

Generate Kubernetes YAML manifests for NVIDIA networking from a cluster config and profile selection.

Usage

l8k generate --user-config <CONFIG> \
  --save-deployment-files <OUTPUT_DIR>

Configs produced by l8k discover already contain the resolved profile. Profile flags remain available as generation-time overrides. When generation uses a file-backed config, resolved defaults and CLI overrides are written back to that source file; embedded --for generation does not write a config.

Profile Selection Flags

*Not required when --spectrum-x is used.

Examples

# SR-IOV Ethernet RDMA (most common for GPU clusters)
l8k generate --user-config cluster-config.yaml \
  --fabric ethernet --deployment-type sriov \
  --save-deployment-files ./output
 
# Spectrum-X with hardware plane load balancing
l8k generate --user-config cluster-config.yaml \
  --spectrum-x RA2.2 --multiplane-mode hwplb --number-of-planes 4 \
  --topology-scheme 2-tier --ip-version ipv6 \
  --topology-file ./topology.json \
  --save-deployment-files ./output
 
# Host device RDMA
l8k generate --user-config cluster-config.yaml \
  --fabric ethernet --deployment-type host_device \
  --save-deployment-files ./output
 
# IPoIB RDMA shared (InfiniBand)
l8k generate --user-config cluster-config.yaml \
  --fabric infiniband --deployment-type rdma_shared \
  --save-deployment-files ./output
 
# Agent mode
l8k generate --user-config cluster-config.yaml \
  --fabric ethernet --deployment-type sriov \
  --save-deployment-files ./output \
  --output json 2>/dev/null
 
# Generate from a known server SKU (no cluster discovery required)
l8k preset list   # see available presets
l8k generate --user-config cluster-config.yaml \
  --for ThinkSystem-SR680a-V3 \
  --node-selector "nvidia.com/gpu.product=NVIDIA-H200" \
  --fabric ethernet --deployment-type sriov \
  --save-deployment-files ./output

Choosing l8k discover vs --for

• l8k discover then l8k generate — default flow. Discovery learns the hardware and persists the resolved profile, so generation needs only the resulting cluster-config.yaml unless an override is desired.
• l8k generate --for <preset> — skip discovery entirely when the SKU is already known and there is a preset for it. Useful for ahead-of-time generation (CI scaffolding, lab runbooks, demos), or when you don't have kubectl access yet. Requires --node-selector to identify the target nodes at apply time.

A preset used with --for must declare capabilities.nodes.{sriov,rdma,ib} in its topology.yaml. All bundled presets do.

Netplan Interface-Name Conflict

Before emitting a NicInterfaceNameTemplate, generation checks clusterConfig[].netplanManaged for the selected source groups. If an affected group would receive the CR, l8k generate returns a validation error because the host netplan set-name rule can conflict with NCO's udev naming rule for the same NIC. Clean up the affected set-name stanzas in the host netplan configuration, re-run l8k discover, and then retry l8k generate.

The check follows the actual render plan: a group excluded with --groups or --gpu-type does not block generation, and neither does a group for which no NicInterfaceNameTemplate is rendered. Do not clear netplanManaged by hand; re-run discovery so the config reflects the host state.

Profile Quick Reference

For detailed profile selection guidance (NIC constraints, multiplane modes, when to use each), read references/profile-decision-tree.md.

Output

Generated YAMLs are written to the output directory under network-operator/. Each profile normally emits a values.yaml (Helm values for the nvidia/network-operator chart) alongside the CR manifests:

When validation.gpuDirect.enabled is true, every generated example DaemonSet requests validation.gpuDirect.gpuResourceType only on its primary DOCA container. The request exposes the highest GPU<N> referenced by PF topology, the image comes from the selected release's validation.image, and networkOperator.imagePullSecrets is copied to the Pod spec. The primary container also sets LD_LIBRARY_PATH to prefer the host-injected NVIDIA driver libraries and exclude the CUDA compatibility-library directory. The environment override is omitted when GPUDirect is disabled. Do not inject these fields later at validation runtime.

output/
└── network-operator/
    ├── values.yaml                       # Phase 0 helm-install input for `l8k deploy`
    ├── 10-nicclusterpolicy.yaml
    ├── 11-nicnodepolicy-<group>.yaml
    ├── 20-ippool-<group>.yaml
    ├── 40-sriovnetworknodepolicy-<group>.yaml
    └── 50-sriovnetwork-<group>.yaml

values.yaml is rendered from the profile's 00-values.yaml template. --network-operator-release <MAJOR.MINOR> populates the chart repository URL and image tag from the embedded catalog. For Spectrum-X profiles, the same catalog entry supplies the independently versioned xPlane repository and tag rather than reusing the generic Network Operator component coordinates. To install or upgrade the chart alongside the CRs, pass --deploy (and --overwrite-existing when the release already exists with different values).

For non-Spectrum-X profiles, docaDriver.env entries are forwarded into the rendered ofedDriver.env list in either NicClusterPolicy (legacy and host-device paths) or NicNodePolicy. This is an advanced escape hatch: custom values override all generated driver environment values with the same name, and duplicate custom names use the last value.

Set networkOperator.skipHelmChart: true or pass --skip-network-operator-helm when another system owns the Helm release. The output directory is cleaned and values.yaml is omitted; all Network Operator custom-resource manifests are still rendered.

For Network Operator 26.1 and newer, the rendered values enable Maintenance Operator requestor mode. Profiles with DOCA/OFED enable operator.maintenanceOperator.useRequestor. Profiles that deploy the SR-IOV Operator enable both the Network Operator drain requestor and sriov-network-operator.operator.externalDrainer; the two SR-IOV switches are a coordinated handoff and must not be separated. The generated MaintenanceOperatorConfig gets the global limits from the config's maintenance section.

Generated values do not add network.nvidia.com/operator.nic-configuration.wait to the SR-IOV config daemon node selector. The daemon keeps the Network Operator chart's default scheduling selectors; drain sequencing remains controlled by the release-appropriate SR-IOV drain path described below.

Before release 26.1, OFED uses maintenance.maxParallelUpgrades and the SR-IOV internal drainer uses maintenance.maxUnavailable through SriovNetworkPoolConfig. Starting with 26.1, the global Maintenance Operator limits are effective for both flows; the legacy OFED and SR-IOV pool limits do not control requestor-mode concurrency.

Reusing the Discovered Profile

l8k discover writes the final profile.fabric, profile.deployment, profile.multirail, and any enabled Spectrum-X settings to the config. Run generate without profile flags to reuse them; explicit generate flags still win when a one-off override is needed.

Common Mistakes

• There is no --profile flag. Profiles are selected via --fabric + --deployment-type (or --spectrum-x). Do NOT invent flags.
• The multiplane flag is --multiplane-mode, not --spcx-multiplane or --multiplane.

Tips

• Default to SR-IOV Ethernet for new GPU cluster deployments unless told otherwise.
• For Spectrum-X, GPU platform and NIC type determine the safe defaults, but B300/GB300 platform type does not distinguish SWPLB from HWPLB. l8k defaults to SWPLB; use an explicit HWPLB override when the site topology requires it. Read references/spectrum-x-modes.md.
• Spectrum-X renders one NicConfigurationTemplate per source group and derives spec.nicSelector.nicType plus pciAddresses from that group's east-west PFs. The intersection prevents same-device-ID north-south DPUs from matching. Do not ask users to configure spectrumX.nicType; missing or mixed east-west device IDs and missing east-west PCI addresses are generation errors.
• NVIDIA AIR topology support requires the documented one-based node/interface naming contract (su<S>, h<H>, leaf-p<P>, r<R>, rail<R>p<P>, and pod<D> for 3-tier). See docs/user/spectrum-x.md in the l8k repository.
• Spectrum-X CIDRPool allocation matches clusterConfig.workerNodes to topology host endpoint node values exactly and case-sensitively. A zero-match error usually means the wrong topology file, a case difference, or a short-name/FQDN difference. Partial-pool errors report the worker's available rail/plane coverage; check host attributes.rail and, for swplb, leaf attributes.plane.
• RA2.2 and RA2.3 v1alpha2 SpectrumXRailPoolConfig output intentionally omits the removed spec.withBCM field; current CRDs reject it during strict decoding.
• Group identifiers produced by discovery omit complete NVIDIA segments and shorten common machine segments (ThinkSystem → ts, PowerEdge → pe). They are bounded to 30 bytes with balanced machine/GPU prefixes and a 6-character deterministic hash suffix. The machine label uses the same value; the GPU label still retains its discovered value such as NVIDIA-H200. Use the exact persisted identifier from cluster-config.yaml with --groups; do not reconstruct it from long machineType and gpuType strings.
• Use --groups <a,b,...> (case-sensitive identifier list) or --gpu-type <X> (case-insensitive) to scope a generate to a subset of source groups in heterogeneous clusters. Mutually exclusive. Empty match is a validation error. Strict-subset filters split per-source rendering: NodePolicies emit one CR per source (each with its own machine-label nodeSelector but a shared bucket-level resourceName); IPPool/example DaemonSet emit one CR per bucket with an In list of source machine labels.
• Generated IPPools use nvIpam.perNodeBlockSize (default 10). Generation warns when it is lower than sriov.numVfs, since the per-node block may not have an address for every VF.

[!CAUTION] Generation does not apply anything to the cluster. Use --deploy or k8s-launch-kit-deploy to apply.

See Also

• k8s-launch-kit-shared — Global flags and output modes
• k8s-launch-kit-discover — Produce the cluster config needed for generation
• k8s-launch-kit-deploy — Apply generated manifests
• k8s-launch-kit-dryrun — Preview before applying*