helm-dev-environment

Helm Dev Environment

Set up, run, and tear down the local Kubernetes development environment for OpenShell. The stack is: k3d (Docker-backed k3s) for the cluster, Skaffold for image builds and Helm deploys, and the OpenShell Helm chart (deploy/helm/openshell/).

Prerequisites

• Docker Desktop (macOS) or Docker Engine (Linux) running
• mise install completed (provides k3d, kubectl, skaffold, helm)

Startup

1. Create the cluster

mise run helm:k3s:create

Creates a k3d cluster and merges its kubeconfig into the worktree-local kubeconfig file. When the named cluster already exists, the task starts any stopped containers and refreshes same-named kubeconfig entries so a recreated load balancer's current API port takes effect. Also applies the upstream agent-sandbox CRDs/controller (pinned via AGENT_SANDBOX_VERSION in tasks/scripts/helm-k3s-local.sh, fetched from github.com/kubernetes-sigs/agent-sandbox releases), enables its OTLP tracing on v0.5 and later, installs an OTLP trace collector and UI in the observability namespace, and preloads the default community sandbox image into k3d so the first sandbox create does not wait on a large registry pull. Traefik is disabled at cluster creation time.

Multi-worktree support: the cluster name is derived from the last component of the current git branch (e.g. branch kube-support/local-dev/tmutch → cluster openshell-dev-tmutch). Each worktree therefore gets its own isolated cluster and its own kubeconfig file. Override with HELM_K3S_CLUSTER_NAME to force a specific name or share one cluster across worktrees.

Port mappings created at cluster time (cannot be changed without recreating):

Override with env vars before running helm:k3s:create:

• HELM_K3S_LB_HOST_PORT (default: 8080)
• HELM_K3S_PRELOAD_SANDBOX_IMAGE (default: ghcr.io/nvidia/openshell-community/sandboxes/base:latest; set to an empty value to skip)
• HELM_K3S_COLLECTOR_IMAGE (default: mcr.microsoft.com/dotnet/aspire-dashboard:latest)
• HELM_K3S_COLLECTOR_HEALTH_TIMEOUT (default: 120 seconds)

2. Deploy OpenShell

Iterative dev (rebuilds on file changes, recommended during active development):

mise run helm:skaffold:dev

One-shot deploy (build once and leave running):

mise run helm:skaffold:run

The Skaffold flow builds distinct gateway, sandbox, and supervisor images and deploys the OpenShell Helm chart. The Kubernetes driver creates a capability-free workload Pod and a directly managed capability-free supervisor Pod. One namespace-wide NetworkPolicy denies direct egress from every OpenShell workload Pod. The pkiInitJob hook (a pre-install Job that runs openshell-gateway generate-certs) generates mTLS secrets on first install. The default Skaffold values export gateway and Kubernetes-driver traces to the collector service installed by helm:k3s:create. Envoy Gateway is opt-in; see the Optional Add-ons section.

The gateway Service uses ClusterIP. Access is via Envoy Gateway (port 8080) or the unified local forwarding task:

mise run helm:k3s:forward

The task forwards OTLP/gRPC to http://127.0.0.1:4317 and the trace UI to http://127.0.0.1:18888. When Skaffold has deployed a Kubernetes gateway, it also forwards the gateway to http://127.0.0.1:8090; otherwise it continues with the collector ports only. A successful plaintext helm:skaffold:run registers the gateway under the worktree-specific k3d cluster name and selects it as the active gateway. Keep the forwarding task running while using those endpoints.

Viewing local traces

The gateway exports OTLP/gRPC to http://openshell-collector.observability.svc.cluster.local:4317 through the default Skaffold values. Forward OTLP/gRPC and the trace UI to the host:

mise run helm:k3s:forward

Open http://127.0.0.1:18888 and exercise the gateway to inspect gateway and Kubernetes compute-driver spans under their distinct service names, along with Agent Sandbox controller reconciliation spans linked through the Sandbox trace-context annotation. The same command exposes OTLP/gRPC on http://127.0.0.1:4317 and, when deployed, the Kubernetes gateway on http://127.0.0.1:8090. The local gateway:docker, gateway:podman, and gateway:vm tasks detect the collector listener at startup and enable trace export only while it is reachable.

HA test deploy (two gateway replicas + external PostgreSQL Secret): uncomment #- ci/values-high-availability.yaml in deploy/helm/openshell/skaffold.yaml, create the Secret named openshell-ha-pg with a uri key, then run mise run helm:skaffold:run or mise run helm:skaffold:dev.

TLS behaviour

ci/values-skaffold.yaml sets server.disableTls: true, so Skaffold-based deploys run plaintext by default. Override server.disableTls=false to exercise TLS/mTLS.

Connecting through the forwarding task

Port 8080 is already bound by the k3d load balancer when Envoy Gateway is active, so the forwarding task uses local port 8090 for the gateway. In a second terminal, confirm that the gateway is registered and active:

openshell gateway list

Plaintext (default Skaffold deploy):

openshell sandbox list

With mTLS enabled — extract the client cert the PKI hook wrote to the cluster, then place it where the CLI expects it. Run once after each fresh install:

mkdir -p ~/.config/openshell/gateways/openshell/mtls
KUBECONFIG=kubeconfig kubectl get secret openshell-client-tls -n openshell \
  -o jsonpath='{.data.ca\.crt}'  | base64 -d > ~/.config/openshell/gateways/openshell/mtls/ca.crt
KUBECONFIG=kubeconfig kubectl get secret openshell-client-tls -n openshell \
  -o jsonpath='{.data.tls\.crt}' | base64 -d > ~/.config/openshell/gateways/openshell/mtls/tls.crt
KUBECONFIG=kubeconfig kubectl get secret openshell-client-tls -n openshell \
  -o jsonpath='{.data.tls\.key}' | base64 -d > ~/.config/openshell/gateways/openshell/mtls/tls.key

The server cert SANs include localhost and 127.0.0.1, so hostname verification passes over a port-forward without any extra flags:

openshell sandbox list --gateway-endpoint https://localhost:8090

Teardown

Remove the Helm releases (keep cluster)

mise run helm:skaffold:delete

Delete the cluster entirely

mise run helm:k3s:delete

This removes the k3d cluster and all resources. Kubeconfig context is left behind but will point to a deleted cluster — safe to ignore or clean up manually.

Optional Add-ons

Each add-on requires uncommenting the corresponding valuesFiles entry in deploy/helm/openshell/skaffold.yaml before running helm:skaffold:dev or helm:skaffold:run.

Envoy Gateway (Gateway API / GRPCRoute)

Envoy Gateway is already installed by Skaffold (the envoy-gateway Helm release in skaffold.yaml). To activate routing:

1. Uncomment #- values-gateway.yaml in skaffold.yaml
2. Redeploy: mise run helm:skaffold:run
3. Apply the GatewayClass: mise run helm:gateway:apply
4. Access: http://127.0.0.1:8080

values-gateway.yaml creates a Gateway (listener on port 80, class eg) and a GRPCRoute in the openshell namespace. Envoy Gateway provisions a LoadBalancer service for the proxy; klipper-lb binds it to hostPort 80, reachable via the 8080:80 load balancer port mapping.

BackendTLSPolicy (end-to-end TLS)

To enable end-to-end TLS between the Gateway proxy and the gateway pod, add BackendTLSPolicy values to the Helm install:

helm upgrade --install openshell deploy/helm/openshell \
  --set grpcRoute.enabled=true \
  --set grpcRoute.backendTLSPolicy.enabled=true \
  --set server.tls.enableMtls=false \
  ...

This requires server.tls.enableMtls=false because ingress proxies cannot present client certificates to the backend. The certgen hook creates a backend CA ConfigMap from the server Secret's ca.crt key. With cert-manager, a separate post-install Job polls for the cert-manager-issued certificate (up to pkiInitJob.timeoutSeconds); with built-in PKI the ConfigMap is created in the same pre-install hook. The ConfigMap is reconciled on every upgrade so CA rotations propagate automatically.

Key Helm values:

• grpcRoute.backendTLSPolicy.enabled: create the BackendTLSPolicy resource
• grpcRoute.backendTLSPolicy.caCertificateConfigMapName: override ConfigMap name
• grpcRoute.backendTLSPolicy.hostname: override backend validation hostname
• server.tls.enableMtls: must be false for BackendTLSPolicy
• pkiInitJob.timeoutSeconds: polling duration for cert-manager mode
• pkiInitJob.failOnTimeout: fail install if cert-manager times out

Keycloak OIDC

Initial setup — rerun it whenever you want to rotate the development CA:

mise run keycloak:k8s:setup

This deploys Keycloak (quay.io/keycloak/keycloak:24.0) into the keycloak namespace, imports the openshell realm from scripts/keycloak-realm.json, generates a short-lived development TLS certificate, and publishes its trust anchor as the openshell-keycloak-ca ConfigMap in the OpenShell namespace. The command prints a port-forward command for acquiring tokens from the CLI. Rerunning setup rotates the development certificate and trust anchor; redeploy the gateway afterward so it reloads the mounted CA bundle.

Then activate OIDC in the OpenShell Helm chart:

1. Uncomment #- ci/values-keycloak.yaml in skaffold.yaml
2. Redeploy: mise run helm:skaffold:run

To remove Keycloak:

mise run keycloak:k8s:teardown

SPIRE / SPIFFE Provider Token Grants

Skaffold can install SPIRE with the SPIFFE hardened Helm charts. To activate SPIFFE JWT-SVIDs for dynamic provider token grants:

1. Uncomment the spire-crds and spire releases in deploy/helm/openshell/skaffold.yaml
2. Uncomment #- ci/values-spire.yaml in the OpenShell release values files
3. Redeploy: mise run helm:skaffold:run

ci/values-spire-stack.yaml configures the local SPIRE trust domain as openshell.local and adds a ClusterSPIFFEID that maps sandbox pod annotations to spiffe://openshell.local/openshell/sandbox/<sandbox-id>. OpenShell mounts the SPIFFE CSI Workload API socket at /spiffe-workload-api/spire-agent.sock only into supervisor Pods for provider token grants. Supervisor-to-gateway authentication remains on the Kubernetes ServiceAccount bootstrap and gateway-minted sandbox JWT path; the selected Kubernetes compute driver validates the projected token before the gateway mints its JWT.

Vault Credential Driver

The credential-driver-vault Skaffold profile applies ci/values-credential-driver-vault.yaml. Its external OpenBao/Vault backend must expose HTTPS at the configured service DNS name and publish the issuing CA certificate as the ca.crt key in the openbao-ca ConfigMap. Local e2e uses OpenBao dev TLS and an openbao-0 DNS alias matching its generated certificate. The Helm value server.credentialDrivers.vault.caConfigMapName mounts that key into the gateway and renders the driver's ca_bundle setting. Non-loopback HTTP addresses fail gateway startup, and hostname verification requires the service DNS name in the server certificate SANs.

cd deploy/helm/openshell
skaffold run -p credential-driver-vault
kubectl -n openshell logs statefulset/openshell -c openshell-gateway --tail=200

Cluster Lifecycle (stop/start)

Stop the cluster without losing state (faster than delete/recreate):

mise run helm:k3s:stop
mise run helm:k3s:start

Check cluster status:

mise run helm:k3s:status

Helm Chart Checks

Run the chart lint task before changing Helm templates, values overlays, or Skaffold inputs:

mise run helm:lint

If Helm reports missing chart dependencies, remove the specific stale subchart archive or directory named by the error from deploy/helm/openshell/charts/, then rerun the lint task.

For example, when lint reports chart metadata is missing these dependencies: postgresql, remove stale PostgreSQL chart artifacts:

rm -f deploy/helm/openshell/charts/postgresql-*.tgz
rm -rf deploy/helm/openshell/charts/postgresql
mise run helm:lint

The charts/ directory is ignored and regenerated by helm dependency build for dependencies still declared in Chart.yaml.

Key Files