OverviewHistoryStatsSecurity
npx skills add ...
Documentation
SKILL.md
npx skills add grafana/skills --skill tempo
Stand up Grafana Tempo as a cost-efficient distributed-tracing backend that only needs object storage, and write TraceQL queries against it. Covers OTLP + Jaeger + Zipkin ingestion, the distributor → live-store → block-builder → object-storage write path, metrics-generator for RED spanmetrics + service graphs, Helm `tempo-distributed` deployment, multi-tenant `X-Scope-OrgID`, TraceQL span / resource / event scopes, structural operators (`>>`, `<<`), `rate()` + `quantile_over_time` metrics, and the traces-to-logs / metrics / profiles datasource links. Use when deploying Tempo, writing a TraceQL query for slow / errored requests, debugging "no traces showing in Explore", sizing queriers / compactors, configuring S3 / GCS / Azure block storage, or wiring trace ↔ log ↔ profile correlation — even when the user says "tracing backend", "find slow requests", "show me the service graph", "store traces in S3", "Jaeger compatible store", or "what called this span" without naming Tempo.
npx skills add grafana/skills --skill tempo
Cost-efficient distributed tracing. Accepts OTLP / Jaeger / Zipkin / OpenCensus / Kafka. Stores Parquet blocks in S3/GCS/Azure.
tempo-cli for synthetic trafficFull operator + scope cheat sheet, intrinsics list, metric functions: references/traceql.md.
Full architecture, ports, performance tuning, metrics-generator config, multi-tenant client snippets, traces-to-logs/metrics/profiles datasource: references/architecture-and-operations.md.
/ready → 503 → ingester still joining; check tempo_ingester_* metrics + logsmax_outstanding_per_tenant or per-tenant ingest limitsX-Scope-OrgID matches between writer and Grafana datasourcestart/end, add a service.name filter, enable dedicated Parquet columns for hot attributesreferences/traceql.md — full TraceQL cheat sheetreferences/architecture-and-operations.md — components, ports, Helm, tuning, datasource links// alloy.river
otelcol.receiver.otlp "default" {
grpc { endpoint = "0.0.0.0:4317" }
http { endpoint = "0.0.0.0:4318" }
output { traces = [otelcol.exporter.otlp.tempo.input] }
}
otelcol.exporter.otlp "tempo" {
client {
endpoint = "tempo:4317"
tls { insecure = true }
}
}# Verify Alloy forwarded successfully
curl -s http://localhost:12345/metrics | grep otelcol_exporter_sent_spans
# Then: same Grafana → Explore → Tempo check.# Slow requests from a service
{ resource.service.name = "frontend" && duration > 1s }
# Server span that has a downstream error (structural)
{ kind = server } >> { status = error }
# Error rate per service (metrics)
{ status = error } | rate() by (resource.service.name)# Via the API
curl -sG --data-urlencode 'q={resource.service.name="frontend" && duration > 1s}' \
--data-urlencode "start=$(date -d '1h ago' +%s)" --data-urlencode "end=$(date +%s)" \
http://localhost:3200/api/search | jq '.traces | length'helm repo add grafana https://grafana.github.io/helm-charts
helm install tempo grafana/tempo-distributed --version 1.61.3 \
--set storage.trace.backend=s3 \
--set storage.trace.s3.bucket=my-tempo-bucket \
--set storage.trace.s3.region=us-east-1
# Verify every pod is Ready (distributor, ingester, querier, query-frontend, compactor)
kubectl get pods -n default -l app.kubernetes.io/instance=tempo
kubectl port-forward svc/tempo-query-frontend 3200:3200 &
curl -sf http://localhost:3200/readymultitenancy_enabled: true
# All requests must include header: X-Scope-OrgID: <tenant-id>