dd-audit-ai-activity

OverviewHistoryStatsSecurity

npx skills add datadog-labs/agent-skills --skill dd-audit-ai-activity

Documentation

SKILL.md

npx skills add ...

Documentation

SKILL.md

Audit what the Bits AI assistant (MCP server) has done in your Datadog org — tool calls by user, resources accessed, and anomaly flags for AI governance.

npx skills add datadog-labs/agent-skills --skill dd-audit-ai-activity

Documentation

SKILL.mdView source

Audit Trail: AI Activity Audit

Every Datadog MCP tool call is recorded in Audit Trail under the Bits AI SRE category. This skill surfaces what the AI assistant has done in your org — which users invoked it, which tools were called, and which resources were affected.

Prerequisites

pup auth login   # OAuth2 (recommended)
# or set DD_API_KEY + DD_APP_KEY with audit_logs_read scope

Queries

All MCP tool activity in a time window

pup audit-logs search --query "@evt.name:\"MCP Server\"" --from 7d --limit 500 -o json \
  | jq '[.data[] | {
      timestamp: .attributes.timestamp,
      user: .attributes.attributes.usr.email,
      actor_type: .attributes.attributes.evt.actor.type,
      action: .attributes.attributes.action,
      resource_type: .attributes.attributes.asset.type,
      resource_id: .attributes.attributes.asset.id,
      ip: .attributes.attributes.network.client.ip,
      country: .attributes.attributes.network.client.geoip.country.name
    }]'

Activity by user (who is using the AI assistant most?)

pup audit-logs search --query "@evt.name:\"MCP Server\"" --from 30d --limit 1000 -o json \
  | jq '[.data[] | .attributes.attributes.usr.email]
    | group_by(.)
    | map({user: .[0], tool_calls: length})
    | sort_by(-.tool_calls)'

Resources modified by AI tool calls

pup audit-logs search \
  --query "@evt.name:\"MCP Server\" @action:(created OR modified OR deleted)" \
  --from 7d --limit 500 -o json \
  | jq '[.data[] | {
      timestamp: .attributes.timestamp,
      user: .attributes.attributes.usr.email,
      action: .attributes.attributes.action,
      resource_type: .attributes.attributes.asset.type,
      resource_id: .attributes.attributes.asset.id
    }]'

AI activity for a specific user

pup audit-logs search \
  --query "@evt.name:\"MCP Server\" @usr.email:user@example.com" \
  --from 30d --limit 500 -o json \
  | jq '[.data[] | {
      timestamp: .attributes.timestamp,
      action: .attributes.attributes.action,
      resource_type: .attributes.attributes.asset.type,
      resource_id: .attributes.attributes.asset.id
    }]'

Weekly summary report

pup audit-logs search --query "@evt.name:\"MCP Server\"" --from 7d --limit 1000 -o json \
  | jq '{
      total_tool_calls: (.data | length),
      unique_users: ([.data[] | .attributes.attributes.usr.email] | unique | length),
      top_users: (
        [.data[] | .attributes.attributes.usr.email]
        | group_by(.)
        | map({user: .[0], calls: length})
        | sort_by(-.calls)
        | .[:5]
      ),
      actions_breakdown: (
        [.data[] | .attributes.attributes.action]
        | group_by(.)
        | map({action: .[0], count: length})
        | sort_by(-.count)
      ),
      resource_types: (
        [.data[] | .attributes.attributes.asset.type]
        | group_by(.)
        | map({type: .[0], count: length})
        | sort_by(-.count)
      )
    }'

Anomaly Flags

Signal	Governance concern
AI performing `deleted` actions on monitors or dashboards	Review whether destructive AI operations are expected
AI acting as `SUPPORT_USER`	Datadog support using AI on behalf of org
First-time user invoking AI tools	New user accessing AI assistant
High volume of tool calls in short window	Automated/batch AI usage
AI accessing resources outside user's normal scope	Potential over-permissioned AI session

Output Format

AI Activity Audit — [Org] — [Date Range]

Total MCP tool calls: [N]
Unique users: [N]

Top users:
  [user@example.com]: [N] calls

Actions breakdown:
  accessed: [N]
  modified: [N]
  created: [N]
  deleted: [N]

Resource types affected:
  dashboard: [N]
  monitor: [N]

Anomalies:
  [List any flagged events with timestamp, user, action, resource]

Context

This skill is most useful for:

Security reviews: Verifying AI actions were authorized and within expected scope
Compliance audits: Demonstrating AI activity is logged and attributable to specific users
Governance reports: Understanding adoption and risk surface of the AI assistant across the org

No other observability vendor audits their AI assistant's actions at this level of detail.

References

npx skills add datadog-labs/agent-skills --skill dd-audit-ai-activity

Documentation

SKILL.md

Audit what the Bits AI assistant (MCP server) has done in your Datadog org — tool calls by user, resources accessed, and anomaly flags for AI governance.

npx skills add datadog-labs/agent-skills --skill dd-audit-ai-activity

Documentation

SKILL.mdView source

Audit Trail: AI Activity Audit

Prerequisites

pup auth login   # OAuth2 (recommended)
# or set DD_API_KEY + DD_APP_KEY with audit_logs_read scope

Queries

All MCP tool activity in a time window

pup audit-logs search --query "@evt.name:\"MCP Server\"" --from 7d --limit 500 -o json \
  | jq '[.data[] | {
      timestamp: .attributes.timestamp,
      user: .attributes.attributes.usr.email,
      actor_type: .attributes.attributes.evt.actor.type,
      action: .attributes.attributes.action,
      resource_type: .attributes.attributes.asset.type,
      resource_id: .attributes.attributes.asset.id,
      ip: .attributes.attributes.network.client.ip,
      country: .attributes.attributes.network.client.geoip.country.name
    }]'

Activity by user (who is using the AI assistant most?)

pup audit-logs search --query "@evt.name:\"MCP Server\"" --from 30d --limit 1000 -o json \
  | jq '[.data[] | .attributes.attributes.usr.email]
    | group_by(.)
    | map({user: .[0], tool_calls: length})
    | sort_by(-.tool_calls)'

Resources modified by AI tool calls

pup audit-logs search \
  --query "@evt.name:\"MCP Server\" @action:(created OR modified OR deleted)" \
  --from 7d --limit 500 -o json \
  | jq '[.data[] | {
      timestamp: .attributes.timestamp,
      user: .attributes.attributes.usr.email,
      action: .attributes.attributes.action,
      resource_type: .attributes.attributes.asset.type,
      resource_id: .attributes.attributes.asset.id
    }]'

AI activity for a specific user

pup audit-logs search \
  --query "@evt.name:\"MCP Server\" @usr.email:user@example.com" \
  --from 30d --limit 500 -o json \
  | jq '[.data[] | {
      timestamp: .attributes.timestamp,
      action: .attributes.attributes.action,
      resource_type: .attributes.attributes.asset.type,
      resource_id: .attributes.attributes.asset.id
    }]'

Weekly summary report

pup audit-logs search --query "@evt.name:\"MCP Server\"" --from 7d --limit 1000 -o json \
  | jq '{
      total_tool_calls: (.data | length),
      unique_users: ([.data[] | .attributes.attributes.usr.email] | unique | length),
      top_users: (
        [.data[] | .attributes.attributes.usr.email]
        | group_by(.)
        | map({user: .[0], calls: length})
        | sort_by(-.calls)
        | .[:5]
      ),
      actions_breakdown: (
        [.data[] | .attributes.attributes.action]
        | group_by(.)
        | map({action: .[0], count: length})
        | sort_by(-.count)
      ),
      resource_types: (
        [.data[] | .attributes.attributes.asset.type]
        | group_by(.)
        | map({type: .[0], count: length})
        | sort_by(-.count)
      )
    }'

Anomaly Flags

Signal	Governance concern
AI performing `deleted` actions on monitors or dashboards	Review whether destructive AI operations are expected
AI acting as `SUPPORT_USER`	Datadog support using AI on behalf of org
First-time user invoking AI tools	New user accessing AI assistant
High volume of tool calls in short window	Automated/batch AI usage
AI accessing resources outside user's normal scope	Potential over-permissioned AI session

Output Format

AI Activity Audit — [Org] — [Date Range]

Total MCP tool calls: [N]
Unique users: [N]

Top users:
  [user@example.com]: [N] calls

Actions breakdown:
  accessed: [N]
  modified: [N]
  created: [N]
  deleted: [N]

Resource types affected:
  dashboard: [N]
  monitor: [N]

Anomalies:
  [List any flagged events with timestamp, user, action, resource]

Context

This skill is most useful for:

Security reviews: Verifying AI actions were authorized and within expected scope
Compliance audits: Demonstrating AI activity is logged and attributable to specific users
Governance reports: Understanding adoption and risk surface of the AI assistant across the org

No other observability vendor audits their AI assistant's actions at this level of detail.