dt-obs-problems

Problem Analysis Skill

Analyze Dynatrace AI-detected problems including root cause identification, impact assessment, and correlation with logs and metrics.

Use Cases

1. Active Problem Triage

• Goal: List and prioritize currently active problems
• Trigger: "active problems", "what problems are open", "current issues", "availability issues"
• Done: Prioritized list of active problems with category, user impact, and display IDs

2. Root Cause Investigation

• Goal: Identify the root cause entity for a specific problem
• Trigger: "root cause of P-12345", "what caused this problem", "which entity is the root cause"
• Done: Root cause entity identified with affected entity list and blast radius

3. Problem Trending

• Goal: Analyze problem patterns over time to identify recurring issues
• Trigger: "recurring problems", "problem history", "problem trends last 30 days"
• Done: Trend data showing problem frequency, recurring root causes, and resolution times

Overview

Dynatrace automatically detects anomalies, performance degradations, and failures across your environment, creating problems that aggregate related alert, warning and info-level events and provide root cause and impact insights.

What are Problems?

Problems are automatically detected, software and infrastructure health and resilience issues that:

• Automatically correlate related alert, warning, and info-level events across services, infrastructure, frontend applications, and user sessions
• Identify root causes using causal analysis of Smartscape dependencies
• Assess business impact by tracking affected users and services
• Reduce alert noise by grouping related symptoms into single problems that share the same root cause and impact
• Track problem lifecycle from early detection through resolution

Event Kinds

The event.kind field (stable, permission) identifies the high-level event type:

event.provider (stable, permission) identifies the event source.

Problem Categories

Common event.category values:

Problem Lifecycle

Detection → ACTIVE → Under Investigation → CLOSED

• ACTIVE: Currently occurring issues requiring attention
• CLOSED: Resolved issues used for historical analysis

Essential Fields

Common Field Name Mistakes

Correct Status Values

// ✅ CORRECT: Use these status values
fetch dt.davis.problems
| filter event.status == "ACTIVE"   // Currently occurring problems
//     or event.status == "CLOSED"  // Resolved problems
// ❌ INCORRECT: event.status == "OPEN" does not exist!

Key Fields Reference

Standard Query Pattern

Always start problem queries with this foundation:

Key components:

• fetch dt.davis.problems - The problems data source
• not(dt.davis.is_duplicate) - Filter out duplicate detections
• event.status == "ACTIVE" - Show only active problems
• Time range - Always specify a reasonable window

Common Query Patterns

Active Problems by Category

High-Impact Active Problems (affecting many users)

High-Impact Active Problems (affecting many smartscape entities)

Specific Problem Details

Service-Specific Problem History

Root Cause Analysis Patterns

Basic Root Cause Query

Root Cause by Entity Type

Identify which entity types most frequently cause problems:

Affected entity is an AWS resource

Infrastructure Root Cause with Service Impact

Problem Blast Radius

Calculate entity impact per root cause:

Recurring Root Causes

Identify entities repeatedly causing problems:

Cause Category vs. Root Cause Entity

These are different questions — pick the right approach:

• "What causes problems?" / "most common cause" → Summarize by event.category (SLOWDOWN, ERROR, RESOURCE, AVAILABILITY, CUSTOM). Explain what triggers each category.
• "Which entity causes problems?" / "root cause entity" → Group by root_cause_entity_name. Lists specific services, hosts, or apps.

Cause category breakdown (use when asked about common causes, patterns, or types):

Then for each category, explain what triggers it using the Problem Categories table and cite specific entities from the tenant data as examples.

Problem Trending and Pattern Analysis

Track problem trends over time, identify recurring issues, and analyze resolution performance.

Primary Files:

• references/problem-trending.md - Timeseries analysis and pattern detection

Common Use Cases:

• Active problems over time with makeTimeseries
• Problem creation rate by category
• Recurring problem detection by schedule
• Resolution time trends and P95 duration analysis

Key Techniques:

• makeTimeseries vs bin(): Choose the right approach for lifecycle spans vs discrete events
• NULL handling: Use coalesce(event.end, now()) for active problems
• Peak hours analysis: Identify when problems occur most frequently
• Impact trending: Track user impact changes over time

See references/problem-trending.md for complete query patterns and best practices.

Cross-Domain Problem Queries

Problems Associated with Kubernetes Clusters

Use affected_entity_ids or dt.smartscape_source.id to find problems related to Kubernetes:

Alternative: expand affected entities and filter for K8s entity types:

Simple Problem Listing

List all problems from the last 24 hours (common request):

Response Construction

Problem Cause Summaries

When summarizing problem causes, categories, or patterns, provide a comprehensive breakdown across all standard categories present in the data: AVAILABILITY, ERROR, SLOWDOWN, RESOURCE, and CUSTOM. For each category found:

1. Category name and count of problems
2. What triggers it — brief explanation (e.g., RESOURCE = CPU/memory/disk threshold exceeded; AVAILABILITY = service or entity became unreachable)
3. Specific examples from the tenant's data (affected entity names, problem IDs)

Do not stop after the first two categories — users expect the full picture. Reference the Problem Categories table above for trigger descriptions.

Analysis Results

When presenting query results:

• Include entity names (not just IDs) — but choose the efficient method:
  • Few entities (< 5): get-entity-name calls are fine
  • Many entities: Use query-problems tool which returns names directly, or include root_cause_entity_name / entityName() in the DQL query to resolve names inline. Avoid calling get-entity-name in a loop for 10+ entities — this can exhaust the tool call limit and return no answer at all.
• Provide actionable recommendations aligned to the identified causes
• Organize by frequency or impact for easy prioritization

Best Practices

Essential Rules

1. Always filter duplicates: Use not(dt.davis.is_duplicate) to avoid counting the same problem multiple times
2. Use correct status values: "ACTIVE" or "CLOSED", never "OPEN"
3. Specify time ranges: Always include time bounds to optimize performance
4. Include display_id: Essential for problem identification and linking
5. Test incrementally: Add one filter or field at a time when building queries
6. Filter early: Apply not(dt.davis.is_duplicate) immediately after fetch

Query Development

• Start simple: Begin with basic filtering, then add complexity
• Test fields first: Run with | limit 1 to verify field names exist
• Use meaningful time ranges: Too broad wastes resources, too narrow misses data
• Document problem IDs: Always capture and store display_id for reference

Root Cause Verification

• Always filter isNotNull(root_cause_entity_id) when required
• Cross-reference events using dt.davis.event_ids
• Consider time delays: root cause may appear in logs minutes before problem

Time Range Guidelines

Absolute Timeframes Require Double Quotes

When using absolute ISO 8601 timestamps for from and to in DQL queries, always wrap them in double quotes. Unquoted timestamps are a syntax error.

Troubleshooting

When to Load References

Load problem-trending.md when:

• Analyzing problem frequency over time
• Detecting recurring problems on a schedule
• Calculating resolution time trends and P95 durations
• Comparing problem creation rates by category

Load problem-correlation.md when:

• Correlating problems with logs or other telemetry
• Investigating events that preceded a problem
• Linking problems to deployment or config changes

Load impact-analysis.md when:

• Assessing business impact (affected users, services)
• Calculating blast radius for a root cause entity
• Prioritizing problems by technical and user impact

References

• problem-trending.md — Problem trending and timeseries analysis patterns
• problem-correlation.md — Correlating problems with logs and other telemetry
• impact-analysis.md — Business and technical impact assessment
• problem-merging.md — When and why DAVIS merges events into problems

Related Skills

• dt-dql-essentials - Core DQL syntax and query structure for problem queries
• dt-obs-logs - Correlate problems with application and infrastructure logs
• dt-obs-tracing - Investigate problems through distributed trace analysis