app-observability

OverviewHistoryStatsSecurity

npx skills add grafana/skills --skill app-observability

Documentation

SKILL.md

npx skills add ...

Documentation

SKILL.md

Get RED metrics + service maps + frontend RUM + AI/LLM monitoring out of Grafana Cloud — Application Observability (`traces_spanmetrics_*` from OTel traces, p50/p95/p99 latency, exemplar-to-trace, traces-to-logs / profiles), Frontend Observability with the Faro Web SDK (Core Web Vitals, session replay, `pushError`, React + router integration, `TracingInstrumentation` for browser → backend trace correlation), and AI Observability via OpenLIT (token / cost / latency, GPU, hallucination + toxicity evals). Use when standing up APM for a service, wiring an Alloy OTLP receiver + forwarding to Cloud, instrumenting a React frontend for RUM, debugging why service-map edges are missing, monitoring LLM cost drift, or correlating a frontend error to its backend trace — even when the user says "set up APM", "show service map", "monitor browser perf", "session replay", "RUM SDK", or "watch our OpenAI bill" without naming App / Frontend / AI Observability.

npx skills add grafana/skills --skill app-observability

Documentation

SKILL.mdView source

Grafana Cloud Application Observability

Docs: https://grafana.com/docs/grafana-cloud/monitor-applications/

Three products that share the same OTLP + Mimir / Loki / Tempo / Pyroscope plumbing:

Application Observability — APM from OTel spanmetrics
Frontend Observability — Faro Web SDK, RUM + session replay
AI Observability — LLM / vector-DB monitoring via OpenLIT

Prerequisites

Grafana Cloud stack + OTLP endpoint + numeric instance ID + API key with MetricsPublisher + LogsPublisher + TracesPublisher
For APM: app instrumented with OTel SDK; for Frontend: a web app + Faro app key; for AI: Python ≥ 3.10
Grafana Alloy as the local OTLP receiver (recommended)

Common Workflows

1. Stand up APM — Alloy receiver → Grafana Cloud + verify

# 1. Set Cloud creds + start Alloy with config from references/apm.md
export GRAFANA_CLOUD_OTLP_ENDPOINT=https://otlp-gateway-prod-us-east-0.grafana.net/otlp
export GRAFANA_CLOUD_INSTANCE_ID=123456
export GRAFANA_CLOUD_API_KEY=glc_eyJ...
alloy fmt /etc/alloy/config.alloy   # syntax check
alloy run /etc/alloy/config.alloy
 
# 2. Verify Alloy is receiving + forwarding
curl -s http://localhost:12345/api/v0/web/components \
  | jq '.[] | select(.id|test("otelcol\\.exporter\\.otlphttp"))
        | {id, health:.health.state}'
# Expect health.state == "healthy"
curl -s http://localhost:12345/metrics \
  | grep -E 'otelcol_(receiver_accepted_spans|exporter_sent_spans)'
 
# 3. Point your app at Alloy (with required attributes!)
export OTEL_SERVICE_NAME="my-api"
export OTEL_RESOURCE_ATTRIBUTES="service.namespace=myteam,deployment.environment=production"
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
export OTEL_EXPORTER_OTLP_PROTOCOL=grpc
 
# 4. Verify spans landed in Tempo + spanmetrics generated
#    Tempo (TraceQL):  { resource.service.name = "my-api" }
#    Mimir (PromQL):   sum by (job) (rate(traces_spanmetrics_calls_total{service_name="my-api"}[5m]))
#    Expect > 0 within ~1 minute.
 
# 5. Verify it's wired to App Observability
#    Grafana → Application → Service Inventory: "my-api" should appear with RED metrics
#    Click into it → Service Map edges visible (requires span.kind on outbound calls)

Full Alloy block + required resource attributes + spanmetric names + correlation links: references/apm.md.

2. Instrument a React frontend with Faro

# 1. Install
npm install @grafana/faro-react @grafana/faro-web-tracing

// 2. initializeFaro with TracingInstrumentation + ReactIntegration (see references/faro.md)
//    Push a smoketest event so we have a known signal:
faro.api.pushEvent('faro_smoketest', { ts: Date.now().toString() });

# 3. Verify in DevTools Network — POST to /collect returns 202
#    (401 → wrong app key; 404 → wrong url region)
 
# 4. Verify in Grafana Cloud
#    - Frontend Observability → your app → Sessions: your session appears
#    - LogQL on Loki: {kind="event"} |= "faro_smoketest"
#    - With TracingInstrumentation: open the session → the trace ID links to Tempo

Full React example, CDN setup, session config: references/faro.md.

3. Add AI / LLM observability

pip install openlit==1.42.0

# At app startup
import openlit
openlit.init(application_name="my-ai-app", environment="production")
# Your existing OpenAI / Anthropic / Cohere calls now emit OTel spans + metrics.

# Env (same OTLP endpoint as APM)
export OTEL_SERVICE_NAME="my-ai-app"
export OTEL_EXPORTER_OTLP_ENDPOINT="https://otlp-gateway-<region>.grafana.net/otlp"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic $(echo -n $ID:$KEY | base64)"
 
# Verify after a few LLM calls:
#   PromQL: sum by (gen_ai_request_model) (rate(gen_ai_usage_input_tokens_total[5m]))
#   Dashboard: Grafana → AI Observability → "GenAI Observability" auto-populates

Full OpenLIT install, evals/guards, GenAI metric list, dashboard names: references/ai-observability.md.

Full-stack correlation cheat sheet

Signal	Product	Storage	Query
RED metrics	App Observability	Mimir	PromQL
Traces	Tempo	Tempo	TraceQL
Logs	Loki	Loki	LogQL
Profiles	Pyroscope	Pyroscope	ProfileQL
Browser RUM	Frontend Observability	Loki + Tempo	LogQL / TraceQL
LLM metrics	AI Observability	Mimir	PromQL

Correlation keys: service.name joins all signals; trace exemplars embed trace IDs in metric points; traceID in logs and traceparent injected by Faro for FE → BE linking.

Troubleshooting

Service missing from Service Inventory → missing service.namespace (job label) or deployment.environment resource attribute
Service Map edges missing → span.kind not set on outbound calls (must be CLIENT) or inbound (SERVER)
Faro /collect returns 401 → wrong app key; 404 → region in URL doesn't match the Faro app
No GenAI metrics → confirm OpenLIT version matches OTel semantic-conv version expected by Cloud; verify auth with curl as in workflow #3

References

references/apm.md — APM essentials: how RED metrics are generated, required OTel resource attributes, Alloy config, correlation links
references/apm-setup.md — deep dive: full per-language OTel SDK setup (Node / Python / Java / Go), span-metrics options, complete Alloy config
references/faro.md — Faro essentials: SDK init, instrumentations, session replay
references/frontend-observability.md — deep dive: full Faro SDK reference, React/Vue/Angular integration, custom events, source maps
references/ai-observability.md — OpenLIT auto-instrumentation for OpenAI / Anthropic / Bedrock / Vertex AI

Resources

npx skills add grafana/skills --skill app-observability

Documentation

SKILL.md

npx skills add grafana/skills --skill app-observability

Documentation

SKILL.mdView source

Grafana Cloud Application Observability

Docs: https://grafana.com/docs/grafana-cloud/monitor-applications/

Three products that share the same OTLP + Mimir / Loki / Tempo / Pyroscope plumbing:

Application Observability — APM from OTel spanmetrics
Frontend Observability — Faro Web SDK, RUM + session replay
AI Observability — LLM / vector-DB monitoring via OpenLIT

Prerequisites

Grafana Cloud stack + OTLP endpoint + numeric instance ID + API key with MetricsPublisher + LogsPublisher + TracesPublisher
For APM: app instrumented with OTel SDK; for Frontend: a web app + Faro app key; for AI: Python ≥ 3.10
Grafana Alloy as the local OTLP receiver (recommended)

Common Workflows

1. Stand up APM — Alloy receiver → Grafana Cloud + verify

# 1. Set Cloud creds + start Alloy with config from references/apm.md
export GRAFANA_CLOUD_OTLP_ENDPOINT=https://otlp-gateway-prod-us-east-0.grafana.net/otlp
export GRAFANA_CLOUD_INSTANCE_ID=123456
export GRAFANA_CLOUD_API_KEY=glc_eyJ...
alloy fmt /etc/alloy/config.alloy   # syntax check
alloy run /etc/alloy/config.alloy
 
# 2. Verify Alloy is receiving + forwarding
curl -s http://localhost:12345/api/v0/web/components \
  | jq '.[] | select(.id|test("otelcol\\.exporter\\.otlphttp"))
        | {id, health:.health.state}'
# Expect health.state == "healthy"
curl -s http://localhost:12345/metrics \
  | grep -E 'otelcol_(receiver_accepted_spans|exporter_sent_spans)'
 
# 3. Point your app at Alloy (with required attributes!)
export OTEL_SERVICE_NAME="my-api"
export OTEL_RESOURCE_ATTRIBUTES="service.namespace=myteam,deployment.environment=production"
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
export OTEL_EXPORTER_OTLP_PROTOCOL=grpc
 
# 4. Verify spans landed in Tempo + spanmetrics generated
#    Tempo (TraceQL):  { resource.service.name = "my-api" }
#    Mimir (PromQL):   sum by (job) (rate(traces_spanmetrics_calls_total{service_name="my-api"}[5m]))
#    Expect > 0 within ~1 minute.
 
# 5. Verify it's wired to App Observability
#    Grafana → Application → Service Inventory: "my-api" should appear with RED metrics
#    Click into it → Service Map edges visible (requires span.kind on outbound calls)

Full Alloy block + required resource attributes + spanmetric names + correlation links: references/apm.md.

2. Instrument a React frontend with Faro

# 1. Install
npm install @grafana/faro-react @grafana/faro-web-tracing

// 2. initializeFaro with TracingInstrumentation + ReactIntegration (see references/faro.md)
//    Push a smoketest event so we have a known signal:
faro.api.pushEvent('faro_smoketest', { ts: Date.now().toString() });

# 3. Verify in DevTools Network — POST to /collect returns 202
#    (401 → wrong app key; 404 → wrong url region)
 
# 4. Verify in Grafana Cloud
#    - Frontend Observability → your app → Sessions: your session appears
#    - LogQL on Loki: {kind="event"} |= "faro_smoketest"
#    - With TracingInstrumentation: open the session → the trace ID links to Tempo

Full React example, CDN setup, session config: references/faro.md.

3. Add AI / LLM observability

pip install openlit==1.42.0

# At app startup
import openlit
openlit.init(application_name="my-ai-app", environment="production")
# Your existing OpenAI / Anthropic / Cohere calls now emit OTel spans + metrics.

# Env (same OTLP endpoint as APM)
export OTEL_SERVICE_NAME="my-ai-app"
export OTEL_EXPORTER_OTLP_ENDPOINT="https://otlp-gateway-<region>.grafana.net/otlp"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic $(echo -n $ID:$KEY | base64)"
 
# Verify after a few LLM calls:
#   PromQL: sum by (gen_ai_request_model) (rate(gen_ai_usage_input_tokens_total[5m]))
#   Dashboard: Grafana → AI Observability → "GenAI Observability" auto-populates

Full OpenLIT install, evals/guards, GenAI metric list, dashboard names: references/ai-observability.md.

Full-stack correlation cheat sheet

Signal	Product	Storage	Query
RED metrics	App Observability	Mimir	PromQL
Traces	Tempo	Tempo	TraceQL
Logs	Loki	Loki	LogQL
Profiles	Pyroscope	Pyroscope	ProfileQL
Browser RUM	Frontend Observability	Loki + Tempo	LogQL / TraceQL
LLM metrics	AI Observability	Mimir	PromQL

Correlation keys: service.name joins all signals; trace exemplars embed trace IDs in metric points; traceID in logs and traceparent injected by Faro for FE → BE linking.

Troubleshooting

Service missing from Service Inventory → missing service.namespace (job label) or deployment.environment resource attribute
Service Map edges missing → span.kind not set on outbound calls (must be CLIENT) or inbound (SERVER)
Faro /collect returns 401 → wrong app key; 404 → region in URL doesn't match the Faro app
No GenAI metrics → confirm OpenLIT version matches OTel semantic-conv version expected by Cloud; verify auth with curl as in workflow #3

References

references/apm.md — APM essentials: how RED metrics are generated, required OTel resource attributes, Alloy config, correlation links
references/apm-setup.md — deep dive: full per-language OTel SDK setup (Node / Python / Java / Go), span-metrics options, complete Alloy config
references/faro.md — Faro essentials: SDK init, instrumentations, session replay
references/frontend-observability.md — deep dive: full Faro SDK reference, React/Vue/Angular integration, custom events, source maps
references/ai-observability.md — OpenLIT auto-instrumentation for OpenAI / Anthropic / Bedrock / Vertex AI