PromQL Query Generator
Overview
This skill provides a comprehensive, interactive workflow for generating production-ready PromQL queries with best practices built-in. Generate queries for monitoring dashboards, alerting rules, and ad-hoc analysis with an emphasis on user collaboration and planning before code generation.
When to Use This Skill
Invoke this skill when:
- Creating new PromQL queries from scratch
- Building monitoring dashboards (Grafana, Prometheus UI, etc.)
- Implementing alerting rules for Prometheus Alertmanager
- Analyzing metrics for troubleshooting or capacity planning
- Converting monitoring requirements into PromQL expressions
- Learning PromQL or teaching others
- The user asks to "create", "generate", "build", or "write" PromQL queries
- Working with Prometheus metrics (counters, gauges, histograms, summaries)
- Implementing RED (Rate, Errors, Duration) or USE (Utilization, Saturation, Errors) metrics
Interactive Query Planning Workflow
CRITICAL: This skill emphasizes interactive planning before query generation. Always engage the user in a collaborative planning process to ensure the generated query matches their exact intentions.
Follow this workflow when generating PromQL queries:
Stage 1: Understand the Monitoring Goal
Start by understanding what the user wants to monitor or measure. Ask clarifying questions to gather requirements:
-
Primary Goal: What are you trying to monitor or measure?
- Request rate (requests per second)
- Error rate (percentage of failed requests)
- Latency/duration (response times, percentiles)
- Resource usage (CPU, memory, disk, network)
- Availability/uptime
- Queue depth, saturation, throughput
- Custom business metrics
-
Use Case: What will this query be used for?
- Dashboard visualization (Grafana, Prometheus UI)
- Alerting rule (firing when threshold exceeded)
- Ad-hoc troubleshooting/analysis
- Recording rule (pre-computed aggregation)
- Capacity planning or SLO tracking
-
Context: Any additional context?
- Service/application name
- Team or project
- Priority level
- Existing metrics or naming conventions
Use the AskUserQuestion tool to gather this information if not provided.
When to Ask vs. Infer: If the user's initial request already clearly specifies the goal, use case, and context (e.g., "Create an alert for P95 latency > 500ms for payment-service"), you may acknowledge these details in your response instead of re-asking. Only ask clarifying questions for information that is missing or ambiguous.
Stage 2: Identify Available Metrics
Determine which metrics are available and relevant:
-
Metric Discovery: What metrics are available?
- Ask the user for metric names
- If uncertain, suggest common naming patterns
- Check for metric type indicators in the name:
_total suffix → Counter_bucket, _sum, _count suffix → Histogram- No suffix → Likely Gauge
_created suffix → Counter creation timestamp
-
Metric Type Identification: Confirm the metric type(s)
- Counter: Cumulative metric that only increases (or resets to zero)
- Examples:
http_requests_total, errors_total, bytes_sent_total
- Use with:
rate(), irate(), increase()
- Gauge: Point-in-time value that can go up or down
- Examples:
memory_usage_bytes, cpu_temperature_celsius, queue_length
- Use with:
avg_over_time(), min_over_time(), max_over_time(), or directly
- Histogram: Buckets of observations with cumulative counts
- Examples:
http_request_duration_seconds_bucket, response_size_bytes_bucket
- Use with:
histogram_quantile(), rate()
- Summary: Pre-calculated quantiles with count and sum
- Examples:
rpc_duration_seconds{quantile="0.95"}
- Use
_sum and _count for averages; don't average quantiles
-
Label Discovery: What labels are available on these metrics?
- Common labels:
job, instance, environment, service, endpoint, status_code, method
- Ask which labels are important for filtering or grouping
Use the AskUserQuestion tool to confirm metric names, types, and available labels.
Stage 3: Determine Query Parameters
Gather specific requirements for the query.
Pre-confirmation for User-Provided Parameters
IMPORTANT: When the user has already specified parameters in their initial request (e.g., "5-minute window", "500ms threshold", "> 5% error rate"), you MUST:
- Acknowledge the provided values explicitly in your response
- Present them as pre-filled defaults in AskUserQuestion with the first option being "Use specified values"
- Allow quick confirmation rather than re-asking for information already given
Example: If user says "alert when P95 latency exceeds 500ms", use:
This respects the user's input and speeds up the workflow while still allowing modifications.
-
Time Range: What time window should the query cover?
- Instant value (current)
- Rate over time (
[5m], [1h], [1d])
- For rate calculations: typically
[1m] to [5m] for real-time, [1h] to [1d] for trends
- Rule of thumb: Rate range should be at least 4x the scrape interval
-
Label Filtering: Which labels should filter the data?
- Exact matches:
job="api-server", status_code="200"
- Negative matches:
status_code!="200"
- Regex matches:
instance=~"prod-.*"
- Multiple conditions:
{job="api", environment="production"}
-
Aggregation: Should the data be aggregated?
- No aggregation: Return all time series as-is
- Aggregate by labels:
sum by (job, endpoint), avg by (instance)
- Aggregate without labels:
sum without (instance, pod), avg without (job)
- Common aggregations:
sum, avg, max, min, count, topk, bottomk
-
Thresholds or Conditions: Are there specific conditions?
- For alerting: threshold values (e.g., error rate > 5%)
- For filtering: only show series above/below a value
- For comparison: compare against historical data (offset)
Use the AskUserQuestion tool to gather or confirm these parameters. When the user has already provided values (e.g., "5-minute window", "> 5%"), present them as the default option for confirmation.
Stage 4: Present the Query Plan
BEFORE GENERATING ANY CODE, present a plain-English query plan and ask for user confirmation:
Use the AskUserQuestion tool to confirm the plan with options:
- "Yes, generate this query"
- "Modify [specific aspect]"
- "Show me alternative approaches"
When the user chooses:
- "Modify [specific aspect]": ask one focused follow-up question about what to change (metric, labels, function, time range, threshold, or output shape), then present an updated plan before generating.
- "Show me alternative approaches": provide at least two valid query plans with trade-offs (accuracy, cost, cardinality, readability), then ask the user to choose one before generating.
Stage 5: Generate the PromQL Query
Once the user confirms the plan, generate the actual PromQL query following best practices.
IMPORTANT: Consult Reference Files Before Generating
Before writing any query code, you MUST:
-
Identify the query category first (histogram, RED, USE, function-specific, optimization, etc.).
-
Read only the relevant reference section(s) using the Read tool:
- For histogram queries → Read
references/metric_types.md (Histogram section)
- For error/latency patterns → Read
references/promql_patterns.md (RED method section)
- For resource monitoring → Read
references/promql_patterns.md (USE method section)
- For optimization questions → Read
references/best_practices.md
- For specific functions → Read
references/promql_functions.md
- Re-read a section only if requirements changed or you have not consulted it yet in the current thread.
-
If a needed reference cannot be read, state the issue and continue with best-effort generation using the most applicable documented pattern you already have.
-
Cite the applicable pattern or best practice in your response:
-
Reference example files when generating similar queries:
This keeps generated queries aligned with documented patterns while avoiding unnecessary full-file rereads on iterative follow-ups.
Best Practices for Query Generation
-
Always Use Label Filters
-
Use Appropriate Functions for Metric Types
-
Apply Aggregations with by() or without()
-
Use Exact Matches Over Regex When Possible
-
Calculate Ratios Properly
-
Use Recording Rules for Complex Queries
- If a query is used frequently or is computationally expensive
- Pre-aggregate data to reduce query load
- Follow naming convention:
level:metric:operations
-
Format for Readability
Common Query Patterns
Pattern 1: Request Rate
Pattern 2: Error Rate
Pattern 3: Latency Percentile (Histogram)
Pattern 4: Resource Usage
Pattern 5: Availability
Pattern 6: Saturation/Queue Depth
Stage 6: Validate the Generated Query
ALWAYS attempt to validate the generated query first using the devops-skills:promql-validator skill:
Validation checklist:
- Syntax is correct (balanced brackets, valid operators)
- Metric type matches function usage
- Label filters are specific enough
- Aggregation is appropriate
- Time ranges are reasonable
- No known anti-patterns
- Query is optimized for performance
If validation fails, fix issues and re-validate until all checks pass.
If the validator skill is unavailable, fails to run, or cannot complete after two fix/re-validate cycles:
- Report the validator failure briefly (tool unavailable, timeout, parsing error, etc.).
- Run a manual fallback check (syntax shape, metric/function compatibility, label filtering, aggregation, time range sanity).
- Mark any unchecked areas as UNVERIFIED and ask the user whether to proceed with best-effort output or provide more context for another validation attempt.
IMPORTANT: Display Validation Results to User
After running validation, you MUST display the structured results to the user in this format:
This transparency helps users understand the validation process and any recommendations.
Stage 7: Provide Usage Instructions
After generation and validation (or manual fallback validation), provide the user with:
-
The Final Query:
-
Query Explanation:
- What the query measures
- How to interpret the results
- Expected value range
- Labels in the output
-
How to Use It:
- For Dashboards: Copy into Grafana/Prometheus UI panel query
- For Alerts: Integrate into Alertmanager rule with threshold
- For Recording Rules: Add to Prometheus recording rule config
- For Ad-hoc: Run directly in Prometheus expression browser
-
Customization Notes:
- Time ranges that might need adjustment
- Labels to modify for different environments
- Threshold values to tune
- Alternative functions if requirements change
-
Related Queries:
- Suggest complementary queries
- Mention recording rule opportunities
- Recommend dashboard panels
Native Histograms (Prometheus 3.x+)
Native histograms are now stable in Prometheus 3.0+ (released November 2024). They offer significant advantages over classic histograms:
- Sparse bucket representation with near-zero cost for empty buckets
- No configuration of bucket boundaries during instrumentation
- Coverage of the full float64 range
- Efficient mergeability across histograms
- Simpler query syntax
Important: Starting with Prometheus v3.8.0, native histograms are fully stable. However, scraping native histograms still requires explicit activation via the scrape_native_histograms configuration setting. Starting with v3.9, no feature flag is needed but scrape_native_histograms must be set explicitly.
Native vs Classic Histogram Syntax
Native Histogram Functions
Detecting Native vs Classic Histograms
Native histograms are identified by:
- No
_bucket suffix on the metric name
- No
le label in the time series
- The metric stores histogram data directly (not separate bucket counters)
When querying, check if your Prometheus instance has native histograms enabled:
Custom Bucket Native Histograms (NHCB)
Prometheus 3.4+ supports custom bucket native histograms (schema -53), allowing classic histogram to native histogram conversion. This is a key migration path for users with existing classic histograms.
Benefits of NHCB:
- Keep existing instrumentation (no code changes needed)
- Store classic histograms as native histograms for lower costs
- Query with native histogram syntax
- Improved reliability and compression
Configuration (Prometheus 3.4+):
Querying NHCB:
Note: Schema -53 indicates custom bucket boundaries. These histograms with different custom bucket boundaries are generally not mergeable with each other.
SLO, Error Budget, and Burn Rate Patterns
Service Level Objectives (SLOs) are critical for modern SRE practices. These patterns help implement SLO-based monitoring and alerting.
Error Budget Calculation
Burn Rate Calculation
Burn rate measures how fast you're consuming error budget. A burn rate of 1 means you'll exhaust the budget exactly at the end of the SLO window.
Multi-Window, Multi-Burn-Rate Alerts (Google SRE Standard)
The recommended approach for SLO alerting uses multiple windows to balance detection speed and precision:
SLO Recording Rules
Pre-compute SLO metrics for efficient alerting:
Latency SLO Queries
Burn Rate Reference Table
| Burn Rate | Budget Consumed | Time to Exhaust 30-day Budget | Alert Severity |
|---|
| 1 | 100% over 30d | 30 days | None |
| 2 | 100% over 15d | 15 days | Low |
| 6 | 5% in 6h | 5 days | Ticket |
| 14.4 | 2% in 1h | ~2 days | Page |
| 36 | 5% in 1h | ~20 hours | Page (urgent) |
Advanced Query Techniques
Using Subqueries
Subqueries enable complex time-based calculations:
Syntax: <query>[<range>:<resolution>]
<range>: Time window to evaluate over<resolution>: Step size between evaluations
Using Offset Modifier
Compare current data with historical data:
Using @ Modifier
Query metrics at specific timestamps:
Binary Operators and Vector Matching
Combine metrics with operators and control label matching:
Logical Operators
Filter time series based on conditions:
Documentation Lookup
If the user asks about specific Prometheus features, operators, or custom metrics:
-
Try context7 MCP first (preferred):
-
Fallback to WebSearch:
Common Monitoring Scenarios
RED Method (for Request-Driven Services)
-
Rate: Request throughput
-
Errors: Error rate
-
Duration: Latency percentiles
USE Method (for Resources)
-
Utilization: Resource usage percentage
-
Saturation: Queue depth or resource contention
-
Errors: Error counters
Alerting Rules
When generating queries for alerting:
-
Include the Threshold: Make the condition explicit
-
Use Boolean Operators: Return 1 (fire) or 0 (no alert)
-
Consider for Duration: Alerts typically use for clause
Recording Rules
When generating queries for recording rules:
-
Follow Naming Convention: level:metric:operations
-
Pre-aggregate Expensive Queries:
-
Use Recorded Metrics in Dashboards:
Error Handling
Common Issues and Solutions
-
Empty Results:
- Check if metrics exist:
up{job="your-job"}
- Verify label filters are correct
- Check time range is appropriate
- Confirm metric is being scraped
-
Too Many Series (High Cardinality):
- Add more specific label filters
- Use aggregation to reduce series count
- Consider using recording rules
- Check for label explosion (dynamic labels)
-
Incorrect Values:
- Verify metric type (counter vs gauge)
- Check function usage (rate on counters, not gauges)
- Verify time range is appropriate
- Check for counter resets
-
Performance Issues:
- Reduce time range for range vectors
- Add label filters to reduce cardinality
- Use recording rules for complex queries
- Avoid expensive regex patterns
- Consider query timeout settings
Communication Guidelines
When generating queries:
- Explain the Plan: Always present a plain-English plan before generating
- Ask Questions: Use AskUserQuestion tool to gather requirements
- Confirm Intent: Verify the query matches user goals before finalizing
- Educate: Explain why certain functions or patterns are used
- Provide Context: Show how to interpret results
- Suggest Improvements: Offer optimizations or alternative approaches
- Validate Proactively: Always validate and fix issues
- Follow Up: Ask if adjustments are needed
Fallback When AskUserQuestion Is Unavailable
If a structured question tool is unavailable, continue with an explicit inline questionnaire in plain text:
- Ask for goal, metric names/types, labels, time range, aggregation, and use case in one compact prompt.
- If the user provides partial answers, proceed with conservative defaults and clearly mark assumptions.
- If core inputs are still ambiguous, offer 2-3 concrete query-plan options and ask the user to pick one.
- Do not block generation indefinitely waiting for perfect context; generate a best-effort query with assumption notes.
Relevant Reference Criteria and Trivial-Case Skip Rules
Use references deterministically, but avoid unnecessary reads for trivial requests.
Read references when ANY of the following is true:
- Histogram or summary quantiles are requested
- Query uses joins/vector matching, subqueries, offsets, or recording/alerting rules
- Query is for SLO/burn-rate/error-budget workflows
- Query includes optimization or cardinality concerns
- Metric type is unknown or contested
Skip reference reads only when ALL of the following are true:
- Single-metric, single-function query (
rate, increase, sum, avg, max, min)
- No joins, no recording/alert rules, no advanced functions
- Metric type and labels are clearly provided by the user
When skipping, explicitly state: Reference read skipped (trivial case) and keep validation mandatory.
Integration with devops-skills:promql-validator
After generating any PromQL query, automatically invoke the devops-skills:promql-validator skill to ensure quality:
This ensures all generated queries follow best practices and are production-ready.
Resources
IMPORTANT: Explicit Reference Consultation
When generating queries, you SHOULD explicitly read the relevant reference files using the Read tool and cite applicable best practices. This ensures generated queries follow documented patterns and helps users understand why certain approaches are recommended.
references/
promql_functions.md
- Comprehensive reference of all PromQL functions
- Grouped by category (aggregation, math, time, histogram, etc.)
- Usage examples for each function
- Read this file when: implementing specific function requirements or when user asks about function behavior
promql_patterns.md
- Common query patterns for typical monitoring scenarios
- RED method patterns (Rate, Errors, Duration)
- USE method patterns (Utilization, Saturation, Errors)
- Alerting and recording rule patterns
- Read this file when: implementing standard monitoring patterns like error rates, latency, or resource usage
best_practices.md
- PromQL best practices and anti-patterns
- Performance optimization guidelines
- Cardinality management
- Query structure recommendations
- Read this file when: optimizing queries, reviewing for anti-patterns, or when cardinality concerns arise
metric_types.md
- Detailed guide to Prometheus metric types
- Counter, Gauge, Histogram, Summary
- When to use each type
- Appropriate functions for each type
- Read this file when: clarifying metric type questions or determining appropriate functions for a metric
examples/
common_queries.promql
- Collection of commonly-used PromQL queries
- Request rate, error rate, latency queries
- Resource usage queries
- Availability and uptime queries
- Can be copied and customized
red_method.promql
- Complete RED method implementation
- Request rate queries
- Error rate queries
- Duration/latency queries
use_method.promql
- Complete USE method implementation
- Utilization queries
- Saturation queries
- Error queries
alerting_rules.yaml
- Example Prometheus alerting rules
- Various threshold-based alerts
- Best practices for alert expressions
recording_rules.yaml
- Example Prometheus recording rules
- Pre-aggregated metrics
- Naming conventions
slo_patterns.promql
- SLO, error budget, and burn rate queries
- Multi-window, multi-burn-rate alerting patterns
- Latency SLO compliance queries
kubernetes_patterns.promql
- Kubernetes monitoring patterns
- kube-state-metrics queries (pods, deployments, nodes)
- cAdvisor container metrics (CPU, memory)
- Vector matching and joins for Kubernetes
Important Notes
- Always Plan Interactively: Never generate a query without confirming the plan with the user
- Use AskUserQuestion: Leverage the tool to gather requirements and confirm plans
- Validate Everything: Always invoke devops-skills:promql-validator after generation
- Educate Users: Explain what the query does and why it's structured that way
- Consider Use Case: Tailor the query based on whether it's for dashboards, alerts, or analysis
- Think About Performance: Always include label filters and consider cardinality
- Follow Metric Types: Use appropriate functions for counters, gauges, and histograms
- Format for Readability: Use multi-line formatting for complex queries
Success Criteria
A successful query generation session should meet these measurable checkpoints:
- Requirement capture completed: goal/use-case/metric/time-range/aggregation recorded.
- Plan confirmation completed: user approved plan OR explicit assumption set documented.
- Reference decision recorded:
consulted with file names OR skipped (trivial case) with reason.
- Query validity completed: syntax passes validator or manual fallback check.
- Semantic sanity completed: function choice matches metric type (counter/gauge/histogram/summary).
- Cardinality guard completed: query includes explicit filters or aggregation rationale.
- Delivery completed: final query + interpretation + next-step customization guidance provided.
Remember
The goal is to collaboratively plan and generate PromQL queries that exactly match user intentions. Always prioritize clarity, correctness, and performance. The interactive planning phase is the most important part of this skill—never skip it!