OverviewHistoryStatsSecurity
npx skills add ...
Documentation
SKILL.md
npx skills add google/skills --skill google-cloud-solution-guided-gke-ai-migration
Guides the migration of existing AI workloads (Cloud Run, Gemini API, Gemini Enterprise Agent Platform) to self-hosted GKE inference using gcloud and kubectl. Use when the user has an existing AI inference workload (on Cloud Run, the Gemini API, Gemini Enterprise Agent Platform, or a custom VM) and wants to move it to self-hosted inference on GKE, or asks follow-up questions during such a migration (hardware sizing, model staging, manifest generation, validation, traffic cutover). DO NOT use for brand new GKE inference deployments with no existing workload to migrate (use gke-inference instead). DO NOT use if the user intends to automate the migration via the Gemini Cloud Assist MCP server.
npx skills add google/skills --skill google-cloud-solution-guided-gke-ai-migration
This skill guides agents through the end-to-end process of migrating existing AI inference workloads (e.g., from Cloud Run, Gemini API, Gemini Enterprise Agent Platform) to self-hosted inference on Google Kubernetes Engine (GKE). The agent will act as an interactive architect, using a structured 4-phase workflow to discover requirements, design a Google Cloud-native solution, execute the implementation using gcloud and kubectl, and validate the deployment.
This skill covers manual, architect-guided migration only. Automated migration is the job of the Gemini Cloud Assist MCP server. Route between them as follows:
When stopping for an MCP request, your response MUST include these 4 points:
google-cloud-solution-guided-gke-ai-migration is strictly intended for manual, architect-guided migration using native CLIs (gcloud and kubectl), and that this manual skill workflow is being stopped.gemini_cloud_assist:ask_cloud_assist) or direct Google Cloud resource mutation (gemini_cloud_assist:invoke_operation).This skill is specifically intended for migrating existing AI workloads (from Cloud Run, Gemini API, Agent Platform, or other platforms) to GKE.
If the user wants to deploy a new AI model server from scratch on GKE (and does NOT have an existing deployment to migrate), STOP and recommend using the gke-inference skill instead. Explain that google-cloud-solution-guided-gke-ai-migration focuses on migration workflows (discovering existing Cloud Run/Agent Platform configurations, traffic cutover, etc.), while gke-inference is optimized for fresh GKE AI model server deployments using AI Profiles and golden path manifests.
When designing the solution, always default to the latest GKE AI best practices:
gcloud inspection commands (list, describe) directly and summarize the results.gcloud and kubectl commands for the user to run. Do not execute mutating commands (apply, create, delete, cluster or IAM changes) unless the user explicitly asks you to run them, in which case execute them and report each command's actual output.gcloud for infrastructure, kubectl for workloads), and opinionated templates.kubectl.vllm/vllm-openai) as the standard LLM serving engine. If migrating from Vertex AI, the user may opt to retain the Vertex AI Model Garden image (e.g., pytorch-vllm-serve), which is permissible.command: ["python", "-m", "vllm.entrypoints.openai.api_server"] to bypass potentially problematic entrypoint scripts (like gcs_download_launcher.sh in Vertex AI images) that crash when passed standard vLLM arguments.:latest. Resolve the current stable release at design time (check the vLLM releases page, or take the tag from gcloud container ai profiles manifests create output) and record it in migration-state.md; do not reuse a tag remembered from a previous migration or from documentation examples.gatewayClassName: gke-l7-rilb) with an HTTPRoute that sends /v1 requests to the vLLM ClusterIP service ({workload_name}-vllm-svc) on port 8000, based on assets/gke-inference-gateway.yaml.tmpl.InferencePool resource as the HTTPRoute backend instead of a Service, and it is only supported on the gke-l7-rilb and gke-l7-regional-external-managed GatewayClasses. Fetch About GKE Inference Gateway before generating InferencePool manifests; do not improvise them from memory.HF_TOKEN):
env.valueFrom.secretKeyRef (e.g., pointing to hf-secret).Secret manifest to disk, and do NOT include it in templates. Instead, explicitly instruct the user to create the Secret directly via CLI before applying any other manifests: kubectl create secret generic hf-secret --namespace={namespace} --from-literal=hf_api_token=<YOUR_HF_TOKEN> with the user substituting the real value themselves.<YOUR_HF_TOKEN> placeholder in every command and manifest you produce, and advise the user to revoke and reissue the token at https://huggingface.co/settings/tokens once the migration is complete.gke-l7-rilb). The vLLM OpenAI-compatible endpoint has no built-in authentication; if the user requires external exposure, warn them explicitly that an unauthenticated external listener is an open inference API on their GPU bill, and require an explicit decision plus a fronting control (IAP, an authenticating API gateway, or strict client allowlisting) before generating an externally-exposed Gateway manifest.model-staging-job.yaml and instruct the user to run kubectl apply -f model-staging-job.yaml.gcloud storage cp).gke-gcsfuse/volumes: "true" (this injects the FUSE sidecar) and MUST run as the Kubernetes ServiceAccount bound to a Google service account with roles/storage.objectUser on the model bucket via Workload Identity. A pod missing either one will fail to mount or fail to read; check both before troubleshooting anything else storage-related.vllm:num_requests_waiting or batch size).The solution design and implementation workflow consists of the following 4 phases:
gcloud and gather model/traffic requirements.kubectl.At the start of every architectural response, print a simple visual progress indicator line to keep both the user and model aligned on the current phase:
(Update ● to mark the current active phase, e.g., [● Solution Design] during Phase 2).
Determine the active phase based on the user's prompt context:
If migration-state.md exists in the current directory, read it before anything else and resume from the recorded phase with the recorded values; only re-ask a discovery question if its value is missing from the file or contradicted by the user's prompt.
**Migration Progress:** [○ Discovery] ➔ [○ Solution Design] ➔ [● Implementation] ➔ [○ Validation].The goal of Phase 1 is to discover all workload specifications necessary to design and build the target GKE inference infrastructure.
Phase 1 Response Requirements: Every response during Phase 1 MUST begin with the visual progress indicator: **Migration Progress:** [● Discovery] ➔ [○ Solution Design] ➔ [○ Implementation] ➔ [○ Validation].
The agent must discover or confirm the following 6 core attribute categories:
Source Platform & Service Configuration:
Model Specifications:
HF_TOKEN access is required).config.json for custom configuration requirements. For example, determine if the architecture requires --trust-remote-code (like Qwen models), specific rope scaling arguments, or other custom flags.Target GKE & Hardware Infrastructure:
Model Storage & Staging:
Traffic Profile & Load Balancing:
Model Equivalence (Gemini API / Agent Platform sources only):
gcloud run services list or gcloud run services describe) before executing any discovery commands.gcloud CLI commands, inspect environment variables, and review local workspace files to populate checklist items automatically:
gcloud run services list --format="table(metadata.name,status.url,status.latestReadyRevisionName)" to enumerate services, then gcloud run services describe {service_name} --format="yaml(spec.template.spec.containers,spec.template.metadata.annotations,spec.template.spec.serviceAccountName,spec.template.spec.containerConcurrency)" to extract only the container image, env vars, resource limits, concurrency, and secret bindings. Prefer --format filters on all discovery commands; never pull a full unfiltered resource description into the conversation.gcloud container clusters list and gcloud container clusters describe {cluster_name} to inspect active cluster config, Workload Identity setup, and available accelerator pools.gcloud ai endpoints list or gcloud storage buckets list to locate model artifacts and storage buckets.migration-state.md in the current directory: one section per checklist category with the confirmed values, plus a final line Current phase: <phase name>. Update the Current phase: line every time the workflow advances a phase.Based on the discovery phase, design the architecture and manifests needed to accomplish the migration.
Phase 2 Response Requirements: Every response during Phase 2 MUST begin with the visual progress indicator: **Migration Progress:** [○ Discovery] ➔ [● Solution Design] ➔ [○ Implementation] ➔ [○ Validation].
Just-in-Time Context Loading: When evaluating specific architectural choices below (e.g., storage options, load balancing, or autoscaling), fetch and read the relevant reference documentation link from the Supporting links section as needed.
assets/ are GPU-only. If the user selects Cloud TPU, state this explicitly, and base the serving manifests on the GKE TPU serving documentation instead of the templates in this skill.)nodeSelector: cloud.google.com/compute-class: {compute_class_name} so the workload actually schedules through the ComputeClass. If the user declines CCC and wants on-demand nodes only, replace this selector with cloud.google.com/gke-accelerator: {accelerator_type} and skip ccc-profile.yaml entirely; do not apply a ComputeClass that no workload references.assets/ (assets/vllm-deployment.yaml.tmpl, assets/ccc-profile.yaml.tmpl, assets/gke-inference-gateway.yaml.tmpl, assets/storage-config.yaml.tmpl, and assets/model-staging-job.yaml.tmpl), substitute the parameters discovered in Phase 1, and save the resulting YAML manifests (vllm-deployment.yaml, ccc-profile.yaml, gke-inference-gateway.yaml, storage-config.yaml, model-staging-job.yaml) to disk in the current directory. When creating vllm-deployment.yaml, explicitly inject any required model-specific architecture flags discovered in Phase 1 (e.g., --trust-remote-code) into the container args array. When creating gke-inference-gateway.yaml, base it on assets/gke-inference-gateway.yaml.tmpl: use gatewayClassName: gke-l7-rilb for the Gateway resource unless the user has explicitly chosen external exposure or the InferencePool-based GKE Inference Gateway, and route /v1 traffic to the vLLM ClusterIP service ({workload_name}-vllm-svc on port 8000) in the HTTPRoute resource. If using a storage class other than gcsfuse-csi (e.g., lustre-csi), remove the gcsfuse.cloud.google.com annotations from storage-config.yaml.To accurately calculate VRAM requirements for model serving/inference, use the following deterministic formula:
Hardware & Sizing Recommendation Requirements: When calculating VRAM sizing or recommending hardware:
$$VRAM_{\text{total}} = \left( \frac{\text{Parameters} \times 2}{\text{Quantization}} + KV_Cache_Overhead \right) \times 1.2$$
Where:
8 for 8B, 70 for 70B).1 for 16-bit (FP16 / BF16, 2 bytes/param)2 for 8-bit (FP8 / INT8, 1 byte/param)4 for 4-bit (INT4 / AWQ / GPTQ, 0.5 bytes/param)1.2 Multiplier: 20% safety margin for CUDA context, activation memory, and serving engine overhead.When mapping the result to an accelerator, compare $VRAM_{\text{total}}$ against the card's full memory (e.g., 24 GB for an L4), not against memory discounted by --gpu-memory-utilization. The 1.2 multiplier and vLLM's utilization cap reserve headroom for the same overheads; applying both double-counts the margin and pushes sizing one accelerator tier too high.
Carry out the approved design per the Execution policy: generate the commands below and either hand them to the user or, if the user has asked you to run them, execute them and report the output.
gcloud commands to provision storage and cluster prerequisites (such as enabling the Cloud Storage FUSE CSI driver) and configure Workload Identity IAM bindings. Gateway API Pre-flight Check: Explicitly instruct the user to verify the Gateway API is enabled on their cluster. Recommend running gcloud container clusters update <CLUSTER_NAME> --gateway-api=standard before they attempt to apply the routing manifests to prevent CRD-not-found errors.kubectl create secret generic hf-secret ... command locally before applying any jobs or deployments (see Gated Model Secret Security rules). Once the secret is created, instruct the user to run kubectl apply -f model-staging-job.yaml.kubectl apply -f model-staging-job.yaml (which should be configured to use gcloud storage cp or similar).kubectl wait --for=condition=complete job/{workload_name}-model-staging --timeout=90m (scale the timeout to the model size). If the Job fails, inspect it with kubectl logs job/{workload_name}-model-staging before retrying. Explain that staging through a cluster Job avoids downloading heavy weights to the user's workstation and avoids re-downloading on every container restart.kubectl apply commands to deploy the storage config (kubectl apply -f storage-config.yaml), ComputeClass (kubectl apply -f ccc-profile.yaml), vLLM deployment (kubectl apply -f vllm-deployment.yaml), and Inference Gateway (kubectl apply -f gke-inference-gateway.yaml). Ensure the vLLM deployment spec mounts the staged model weights from the PVC into /models. Ensure that if a Secret was created for a gated model, it is referenced correctly in the Deployment manifest. Set vLLM's --model flag to the staged local path (/models/{model_name}), never to the Hugging Face repo ID; a repo ID makes vLLM re-download the full weights on every pod start and silently defeats the staging step. Preserve the model's public name for API clients with --served-model-name={model_id}.kubectl rollout status deployment/{workload_name}-vllm --timeout=30m and kubectl get pods -l app={workload_name}-vllm -o wide. If the rollout fails or times out, go directly to Troubleshooting Guidance with the observed error; do not ask the user whether the deployment succeeded when the command output already answers it. Ask the user only about outcomes the cluster cannot verify (for example, whether response quality matches the source system).Verify that the deployed infrastructure meets the workload's requirements and provide instructions for traffic migration.
Phase 4 Response Requirements: Every response during Phase 4 MUST begin with the visual progress indicator: **Migration Progress:** [○ Discovery] ➔ [○ Solution Design] ➔ [○ Implementation] ➔ [● Validation].
kubectl get pods, kubectl get nodes, and kubectl get gateway).kubectl port-forward command (kubectl port-forward svc/{workload_name}-vllm-svc 8000:8000) and a sample curl request to /v1/chat/completions to test endpoint inference. Compare responses with the previous infrastructure if applicable.When users report issues where pods are created but the inference endpoint is not responding (or request troubleshooting help):
kubectl logs {pod_name} (to check container startup logs) and kubectl describe pod {pod_name} (to inspect pod initialization state).curl returns Connection refused but the Pod is Running, the serving engine (e.g., vLLM) may still be executing JIT compilation (such as Triton PTX or Torch Inductor) or capturing CUDA graphs. This can take several minutes after model weights are loaded. Advise the user to check kubectl logs {pod_name} and explicitly wait for the Uvicorn running on http://0.0.0.0:8000 (or equivalent) log message before assuming there is a networking issue.Use these references as needed to ground your design choices, answer user questions, and generate implementation manifests: