k8s-launch-kit-discover

l8k: Cluster Discovery

PREREQUISITE: Read ../k8s-launch-kit-shared/SKILL.md for install paths, global flags, and output modes.

This workflow uses the default host target; --target host is equivalent. The CLI snapshots discovery arguments and runs the concrete Host discovery operation through the target registry; command syntax and artifacts are unchanged.

Discover cluster hardware and produce a cluster-config.yaml describing NICs, GPUs, rails, and node groups.

Three things to know about the saved file:

• Fresh discovery persists a resolved profile. Without --user-config, discovery fills profile fields from hardware/defaults and applies explicit CLI overrides last. With --user-config, it replaces only clusterConfig, preserves every other section, and then applies explicit CLI overrides.
• North-south-only groups are dropped. A node group whose NICs are all north-south (BlueField DPUs / OOB NICs) produces no manifests and is omitted from cluster-config.yaml (it would otherwise consume an NV-IPAM subnet slice and shift the per-rail pools). North-south NICs alongside east-west NICs in a mixed group are still listed.
• Documentation comments are preserved. The field-reference comments from the source config (l8k-config.yaml or --user-config) are carried over into the saved cluster-config.yaml, with a Generated by l8k discover banner at the top.

l8k discover is self-contained. It does NOT require a pre-installed Network Operator. On every run it bootstraps the NIC Configuration Daemon (and the 5 nic-configuration-operator CRDs, only if missing) into a private namespace nvidia-k8s-launch-kit, reads NicDevice CRs published by the daemon, then tears the namespace down on exit (unless --keep-namespace is set). The daemon image is pulled from <networkOperator.repository>/ nic-configuration-operator-daemon:<networkOperator.componentVersion> (both fields come from l8k-config.yaml).

Usage (from AI agent)

l8k discover \
  --kubeconfig ~/.kube/config \
  --save-cluster-config ./cluster-config.yaml \
  --output json 2>/dev/null | jq .

Usage (human-interactive)

l8k discover --save-cluster-config <OUTPUT> [--kubeconfig <PATH>]

Flags

Examples

# Basic discovery
l8k discover \
  --kubeconfig ~/.kube/config \
  --save-cluster-config ./cluster-config.yaml
 
# Using $KUBECONFIG env var (no --kubeconfig needed)
l8k discover --save-cluster-config ./cluster-config.yaml
 
# Keep the bootstrap namespace for debugging
l8k discover \
  --kubeconfig ~/.kube/config \
  --keep-namespace \
  --save-cluster-config ./cluster-config.yaml
 
# Refresh clusterConfig in an existing config
l8k discover --user-config my-config.yaml \
  --kubeconfig ~/.kube/config \
  --save-cluster-config ./cluster-config.yaml
 
# Persist explicit profile overrides
l8k discover \
  --kubeconfig ~/.kube/config \
  --fabric infiniband --deployment-type rdma_shared \
  --multirail=false \
  --save-cluster-config ./cluster-config.yaml
 
# Agent mode (JSON output)
l8k discover \
  --kubeconfig ~/.kube/config \
  --save-cluster-config ./cluster-config.yaml \
  --output json 2>/dev/null

Output Format

The generated cluster-config.yaml contains the final profile plus a clusterConfig[] hardware-group array:

profile:
  fabric: ethernet
  deployment: sriov
  multirail: true
clusterConfig:
  - identifier: "dgx-b200-h100-nvl"
    machineType: DGX-B200
    gpuType: NVIDIA-H100-NVL
    netplanManaged: true
    capabilities:
      nodes:
        sriov: true
        rdma: true
    pfs:
      - deviceID: "101e"
        networkInterface: "eth0"
        rail: 0
    workerNodes: [node-01, node-02]
    nodeSelector:
      nvidia.kubernetes-launch-kit.machine: "dgx-b200-h100-nvl"
    thirdPartyRDMAModules: [nv_peer_mem]

Discovery patches every node in the group with two labels:

• nvidia.kubernetes-launch-kit.machine: <identifier> — per-source-group identity, identical to the persisted group identifier and used as its nodeSelector.
• nvidia.kubernetes-launch-kit.gpu: <gpuType> — used as the merged-group nodeSelector when l8k generate auto-merges source groups sharing a GPU type.

Group identifiers and machine-label values are lowercase and omit complete NVIDIA segments. The same pass maps ThinkSystem to ts and PowerEdge to pe. They are bounded to 30 bytes; long values retain balanced machine/GPU prefixes plus a 6-character deterministic hash, and unused prefix space moves to the longer component. GPU-only labels retain their discovered case and vendor segment, use Kubernetes' 63-byte limit, and keep the existing 8-character shortening hash. When machineType or gpuType couldn't be resolved (GPU operator labels absent and hardware probe failed), a fallback group-N identifier is used and the machine label is not written; the GPU label is still written when gpuType alone is resolved.

Prerequisites

• Node Feature Discovery (NFD) is not required. Before bootstrapping the daemon, discovery lists Nodes and renders node-name affinity for nodes that are Ready=True and not spec.unschedulable. The DaemonSet still has no NFD nodeSelector and tolerates taints, so eligible control-plane/tainted nodes are included. NIC-bearing nodes are detected by a sysfs probe for PCI vendor 0x15b3 rather than the NFD feature.node.kubernetes.io/pci-15b3.present label. BlueFields for which mlxprivhost reports level: restricted are excluded because the daemon will not publish NicDevice resources for them. Mixed nodes remain eligible when they have at least one non-restricted NVIDIA NIC.
• Image-pull access from every worker node to <networkOperator.repository>/nic-configuration-operator-daemon:<networkOperator.componentVersion>. Use --image-pull-secrets <name> (or networkOperator.imagePullSecrets in the config file) for private registries.
• Network Operator does NOT need to be pre-installed.

Fresh discovery writes validation.gpuDirect.enabled. It is true only when every worker in every discovered group can satisfy its render bucket's topology-derived validation.gpuDirect.gpuResourceType request; otherwise it is false. With --user-config, the supplied validation section is preserved. Do not add GPU counts or resource maps under clusterConfig.

Tips

• Fresh profile precedence is hardware/default < explicit CLI. On a rerun with --user-config, no profile fields are recalculated; explicit CLI flags still override their corresponding values.

• A fresh discovery ignores the example profile in the installed reference config. Supplying that file explicitly through --user-config makes its profile values user input and therefore preserves them.

• If fabric probes are mixed or unconfirmed, discovery still writes the file with an empty profile.fabric; set --fabric on discovery or generation.

• clusterConfig[].netplanManaged is true when any worker has an NVIDIA PF selected by a host netplan match.macaddress plus set-name rule. Discovery checks every worker and uses PF MACs only transiently; they are not saved in the shared group config. If generation would emit a NicInterfaceNameTemplate for an affected group, clean up the host's set-name stanzas and re-run discovery before retrying generation; do not clear the field manually.

• The bootstrap namespace is always nvidia-k8s-launch-kit — not configurable. The daemon's SA / ClusterRole / ClusterRoleBinding are renamed to k8s-launch-kit-nic-config-daemon so they don't collide with the cluster-scoped names a coexisting Network Operator install would create.

• CRDs (nicdevices, nicconfigurationtemplates, etc.) are applied only when missing. If they already exist (because Network Operator or a prior l8k discover run created them), they're left alone — discovery never overwrites a different version.

• After discovery finishes the namespace is torn down (cascade-delete handles SA / RoleBinding / DaemonSet / pods / NicDevice CRs); the CRDs intentionally persist so any external NicDevice consumers survive. Cluster-scoped ClusterRole / ClusterRoleBinding are deleted explicitly.

• Pass --keep-namespace to leave the namespace in place — useful when debugging daemon pod start-up failures (kubectl describe pod -n nvidia-k8s-launch-kit).

• Before bootstrapping, discovery pre-cleans the nvidia-k8s-launch-kit namespace (deletes any leftover DaemonSet/pods/NicDevice CRs from a crashed prior run and waits up to 2 min for it to clear) so a fresh daemon is never layered on stale state.

• Discovery waits up to 5 minutes for daemon pods to be Ready, but tolerates stuck pods: if some pods are wedged (e.g. ImagePullBackOff/CrashLoopBackOff on an unrelated node) it proceeds with the Ready ones rather than blocking the whole window, and only aborts if no pod ever becomes Ready. Common causes of a total failure: image tag missing in your registry, pull-secret missing.

• If discovery reports "no nodes with an NVIDIA NIC (PCI vendor 15b3) were found", the daemon ran but no node's sysfs exposed a 0x15b3 device (no Mellanox/NVIDIA NICs, or /sys not mounted/readable in the pod).

• If discovery reports "no discoverable NVIDIA NICs were found", every detected BlueField is in zero-trust (restricted) mode. Those devices cannot be configured from the host and intentionally do not publish NicDevice resources.

• After determining each group's (machineType, gpuType), discovery looks up a topology preset under presets/ using exact-match lookup on that pair. A matching preset overrides heuristic-derived topology fields (traffic class, rail, NUMA, GPU affinity). There is no any-GPU fallback — a preset with empty gpuType: is rejected at load time. If no preset matches, discovery proceeds with heuristic classification.

• If you already know the SKU and want to skip cluster discovery entirely, use l8k generate --for <preset> (see k8s-launch-kit-generate).

Rail collapsing (one rail per NIC)

By default (--collapse-nic-rails, default true) discovery advertises one rail per NIC, not one per PF. A single NIC can expose several east-west PFs that are planes of one physical port (Spectrum-X multi-plane ConnectX-8/9). For those, only the master PF (lowest PCI function) is written to cluster-config.yaml, so the rail count reflects physical NICs — e.g. an 8-PF ConnectX-8 node yields 4 rails, not 8. The per-PF planes are reconstructed at generate time from the multiplane mode + numberOfPlanes.

The exception is a genuinely dual-port NIC: when the NIC's VPD model name (read from NicDevice.Status.modelName, surfaced as the model field on each PF) contains a port-count keyword like 2-port/Dual-port, each port is kept as its own rail. Single-port (1P) and multi-plane models (no port-count keyword) collapse. An empty/unreadable model collapses by default.

Pass --collapse-nic-rails=false to restore the legacy one-rail-per-PF behaviour (handy on dev setups). Note: a collapsed group no longer has the same PF count as a full-PF topology preset, so such presets won't apply by exact match and the live (collapsed) classification is kept (a preset-deviation warning is emitted).

See Also

• k8s-launch-kit-shared — Global flags and output modes
• k8s-launch-kit-config — Understand the config file discovery produces
• k8s-launch-kit-generate — Generate manifests from cluster config