npx skills add ...
npx skills add victoriametrics/skills --skill victoriatraces-query
Query VictoriaTraces via curl using the Jaeger-compatible API. Use when discovering traced services and operations, searching traces by service/operation/duration/tags, retrieving traces by ID, or mapping service dependencies. Triggers on: trace queries, span search, trace ID lookup, service discovery, operation discovery, service dependencies, distributed tracing, Jaeger API.
npx skills add victoriametrics/skills --skill victoriatraces-query
Query VictoriaTraces Jaeger-compatible HTTP API directly via curl. Covers service/operation discovery, trace search, trace retrieval by ID, and dependency mapping.
IMPORTANT: The Jaeger API lives under /select/jaeger/api/..., NOT root /api/.... The $VM_TRACES_URL env var already includes the /select/jaeger prefix, so all endpoints below use $VM_TRACES_URL/api/....
All curl commands load auth from a curl config file:
When VM_CURL_CONFIG is unset, curl reads /dev/null and sends no auth header. When set, it must point to a mode-0600 curl config file containing header = "Authorization: Bearer <token>". Never print its contents.
start/end use Unix MICROSECONDS (16 digits, e.g., 1709769600000000)endTs/lookback use Unix MILLISECONDS (13 digits, e.g., 1709769600000)service is required for trace search — you must know the service name firstminDuration/maxDuration are string durations (e.g., "1s", "500ms")errors array, not an HTTP 404No parameters. Returns all traced service names. Always start here to discover what's available.
service is a PATH parameter. Returns operation names for the given service.
Parameters: service (required), operation, start (Unix µs), end (Unix µs), limit, minDuration (e.g., 1s), maxDuration, tags (URL-encoded JSON object or VictoriaTraces key=value format)
Returns full trace with all spans. If trace not found, response contains "errors": [{"code": 404, "msg": "trace not found"}].
Parameters: endTs (Unix ms, required), lookback (duration in ms, required). Returns edges between services showing call relationships.
Trace search and dependencies use DIFFERENT timestamp units:
| Endpoint | Parameter | Unit | Digits | Example |
|---|---|---|---|---|
/api/traces | start, end | Microseconds | 16 | 1709769600000000 |
/api/dependencies | endTs | Milliseconds | 13 | 1709769600000 |
/api/dependencies | lookback | Milliseconds | 13 | 3600000 (1 hour) |
/select/jaeger is included in $VM_TRACES_URL — do NOT add it againtags parameter supports both Jaeger JSON format ({"key":"value"}) and VictoriaTraces extended format (key=value resource_attr:key=value scope_attr:key=value)processID in spans, mapped in the trace's processes objectreferences/api-reference.mdcurl -q --config "${VM_CURL_CONFIG:-/dev/null}" -s \
"$VM_TRACES_URL/api/services" | jq '.data[]'curl -q --config "${VM_CURL_CONFIG:-/dev/null}" -s \
"$VM_TRACES_URL/api/services/my-service/operations" | jq '.data[]'# Basic search (last hour, limit 20) — timestamps in microseconds (16 digits)
curl -q --config "${VM_CURL_CONFIG:-/dev/null}" -s \
"$VM_TRACES_URL/api/traces?service=my-service&start=$(($(date +%s%6N) - 3600000000))&end=$(date +%s%6N)&limit=20" | jq .
# With operation and duration filter
curl -q --config "${VM_CURL_CONFIG:-/dev/null}" -s \
"$VM_TRACES_URL/api/traces?service=my-service&operation=GET+/api/health&start=$(($(date +%s%6N) - 3600000000))&end=$(date +%s%6N)&minDuration=100ms&limit=20" | jq .
# With tag filter (Jaeger JSON format)
curl -q --config "${VM_CURL_CONFIG:-/dev/null}" -s \
"$VM_TRACES_URL/api/traces?service=my-service&start=$(($(date +%s%6N) - 3600000000))&end=$(date +%s%6N)&tags=%7B%22http.status_code%22%3A%22500%22%7D&limit=20" | jq .
# With tag filter (VictoriaTraces extended format — key=value pairs, space-separated)
curl -q --config "${VM_CURL_CONFIG:-/dev/null}" -s -G \
--data-urlencode 'tags=http.status_code=500 resource_attr:service.namespace=production' \
"$VM_TRACES_URL/api/traces?service=my-service&start=$(($(date +%s%6N) - 3600000000))&end=$(date +%s%6N)&limit=20" | jq .curl -q --config "${VM_CURL_CONFIG:-/dev/null}" -s \
"$VM_TRACES_URL/api/traces/abc123def456789" | jq .# Dependencies for the last hour
curl -q --config "${VM_CURL_CONFIG:-/dev/null}" -s \
"$VM_TRACES_URL/api/dependencies?endTs=$(date +%s%3N)&lookback=3600000" | jq '.data[]'# Current time in microseconds (for trace search)
date +%s%6N
# 1 hour ago in microseconds
echo $(($(date +%s%6N) - 3600000000))
# Specific time to microseconds (GNU date)
date -d "2026-03-07T09:00:00Z" +%s%6N
# Current time in milliseconds (for dependencies)
date +%s%3N
# Duration values for minDuration/maxDuration are strings: "1s", "500ms", "100us"
# Duration value for lookback is ms number: 3600000 = 1 hour# List service names
... | jq '.data[]'
# Extract trace IDs from search results
... | jq '.data[] | .traceID'
# Get span count per trace
... | jq '.data[] | {traceID: .traceID, spans: (.spans | length)}'
# Extract spans with duration > 1s (duration is in microseconds)
... | jq '.data[].spans[] | select(.duration > 1000000) | {operation: .operationName, duration_ms: (.duration / 1000)}'
# Get root span of each trace
... | jq '.data[] | .spans[] | select(.references == null or (.references | length) == 0) | {traceID: .traceID, operation: .operationName, service: .processID}'
# Extract service-to-service calls from dependencies
... | jq '.data[] | "\(.parent) -> \(.child) (\(.callCount) calls)"'
# Get all tags from a span
... | jq '.data[].spans[] | {operation: .operationName, tags: [.tags[] | {(.key): .value}] | add}'
# Find error spans
... | jq '.data[].spans[] | select(.tags[] | select(.key == "error" and .value == true)) | {traceID: .traceID, operation: .operationName}'# Full discovery workflow: services -> operations -> traces
curl -q --config "${VM_CURL_CONFIG:-/dev/null}" -s "$VM_TRACES_URL/api/services" | jq '.data[]'
curl -q --config "${VM_CURL_CONFIG:-/dev/null}" -s "$VM_TRACES_URL/api/services/my-service/operations" | jq '.data[]'
curl -q --config "${VM_CURL_CONFIG:-/dev/null}" -s "$VM_TRACES_URL/api/traces?service=my-service&start=$(($(date +%s%6N) - 3600000000))&end=$(date +%s%6N)&limit=10" | jq '.data[] | .traceID'
# Find slow traces (> 5 seconds)
curl -q --config "${VM_CURL_CONFIG:-/dev/null}" -s \
"$VM_TRACES_URL/api/traces?service=my-service&start=$(($(date +%s%6N) - 3600000000))&end=$(date +%s%6N)&minDuration=5s&limit=20" | jq '.data[] | {traceID: .traceID, spans: (.spans | length)}'
# Look up a trace ID found in logs
curl -q --config "${VM_CURL_CONFIG:-/dev/null}" -s \
"$VM_TRACES_URL/api/traces/abc123def456789" | jq '.data[].spans[] | {operation: .operationName, duration_ms: (.duration / 1000), service: .processID}'
# Map service dependencies (last 24 hours) — dependencies use milliseconds
curl -q --config "${VM_CURL_CONFIG:-/dev/null}" -s \
"$VM_TRACES_URL/api/dependencies?endTs=$(date +%s%3N)&lookback=86400000" | jq '.data[]'# Check current environment
echo "VM_TRACES_URL: $VM_TRACES_URL"
echo "VM_CURL_CONFIG: ${VM_CURL_CONFIG:-(unset - no auth)}"