OverviewHistoryStatsSecurity
npx skills add ...
Documentation
SKILL.md
npx skills add langchain-ai/skills-benchmarks --skill langsmith-trace
INVOKE THIS SKILL when working with LangSmith tracing OR querying traces. Covers adding tracing to applications and querying/exporting trace data. Uses the langsmith CLI tool.
npx skills add langchain-ai/skills-benchmarks --skill langsmith-trace
Authentication is REQUIRED: either set the LANGSMITH_API_KEY environment variable, or pass the --api-key flag to CLI commands (preferred):
IMPORTANT: Always check the environment variables or .env file for LANGSMITH_PROJECT before querying or interacting with LangSmith. This tells you which project contains the relevant traces and data. If the LangSmith project is not available, use your best judgement to identify the right one.
CLI Tool
<trace_langchain_oss> For LangChain/LangGraph apps, tracing is automatic. Just set environment variables:
Optional variables:
LANGSMITH_PROJECT - specify project name (defaults to "default")LANGCHAIN_CALLBACKS_BACKGROUND=false - use for serverless to ensure traces complete before function exit (Python)
</trace_langchain_oss><trace_other_frameworks>
For anything other than LangChain/LangGraph, read the matching reference file in references/ before writing tracing code. Each reference covers install, env vars, setup snippet, and gotchas specific to that framework. The setup is rarely identical across frameworks ā picking the wrong pattern (e.g. using @traceable when the framework has native OTel) creates duplicate/missing spans.
Decision order:
references/otel.mdreferences/traceable.mdreferences/api.md (last resort)Routing table:
| If you're tracingā¦ | Read |
|---|---|
| OpenAI / Azure OpenAI / Anthropic / any plain LLM client | references/traceable.md |
| AutoGen | references/autogen.md |
| CrewAI | references/crewai.md |
| Google ADK | references/google-adk.md |
Google Gemini (google-genai SDK directly) | references/google-gemini.md |
| Instructor (structured outputs) | references/instructor.md |
| LiveKit Agents (voice AI) | references/livekit.md |
| Mastra (TypeScript) | references/mastra.md |
| Microsoft Agent Framework | references/microsoft-agent-framework.md |
| Mistral | references/mistral.md |
| n8n (self-hosted) | references/n8n.md |
| OpenAI Agents SDK | references/openai-agents-sdk.md |
| OpenCode | references/opencode.md |
| OpenAI Codex CLI | references/codex.md |
| Pipecat (voice AI) | references/pipecat.md |
| PydanticAI | references/pydantic-ai.md |
| Semantic Kernel | references/semantic-kernel.md |
| Strands Agents | references/strands-agents.md |
| Temporal workflows (Go/Python/TS) | references/temporal.md |
| Vercel AI SDK | references/vercel-ai-sdk.md |
| Any other framework with native OTel | references/otel.md |
| Multi-backend OTel fan-out | references/otel.md (Collector section) |
| Raw REST (no SDK available) | references/api.md |
If the framework you need isn't listed here, check references/ ā new integrations are added there, not inline.
</trace_other_frameworks>
<traces_vs_runs>
Use the langsmith CLI to query trace data.
Understanding the difference is critical:
Generally, query traces first ā they provide complete context and preserve hierarchy needed for trajectory analysis and dataset generation. </traces_vs_runs>
<command_structure> Two command groups with consistent behavior:
Key differences:
traces * | runs * | |
|---|---|---|
| Filters apply to | Root run only | Any matching run |
--run-type | Not available | Available |
| Returns | Full hierarchy | Flat list |
| Export output | Directory (one file/trace) | Single file |
| </command_structure> |
<querying_traces>
Query traces using the langsmith CLI. Commands are language-agnostic.
</querying_traces>
All commands support these filters (all AND together):Basic filters:
--trace-ids abc,def - Filter to specific traces--limit N - Max results--project NAME - Project name--last-n-minutes N - Time filter--since TIMESTAMP - Time filter (ISO format)--error / --no-error - Error status--name PATTERN - Name contains (case-insensitive)Performance filters:
--min-latency SECONDS - Minimum latency (e.g., 5 for >= 5s)--max-latency SECONDS - Maximum latency--min-tokens N - Minimum total tokens--tags tag1,tag2 - Has any of these tagsAdvanced filter:
--filter QUERY - Raw LangSmith filter query for complex cases (feedback, metadata, etc.)<export_format>
Export creates .jsonl files (one run per line) with these fields:
Use --include-io or --full to include inputs/outputs (required for dataset generation).
</export_format>
langsmith trace list --project my-project --api-key $LANGSMITH_API_KEYcurl -sSL https://raw.githubusercontent.com/langchain-ai/langsmith-cli/main/scripts/install.sh | shexport LANGSMITH_TRACING=true
export LANGSMITH_API_KEY=<your-api-key>
export OPENAI_API_KEY=<your-openai-api-key> # or your LLM provider's keylangsmith
āāā trace (operations on trace trees - USE THIS FIRST)
ā āāā list - List traces (filters apply to root run)
ā āāā get - Get single trace with full hierarchy
ā āāā export - Export traces to JSONL files (one file per trace)
ā
āāā run (operations on individual runs - for specific analysis)
ā āāā list - List runs (flat, filters apply to any run)
ā āāā get - Get single run
ā āāā export - Export runs to single JSONL file (flat)
ā
āāā dataset (dataset operations)
ā āāā list - List datasets
ā āāā get - Get dataset details
ā āāā create - Create empty dataset
ā āāā delete - Delete dataset
ā āāā export - Export dataset to file
ā āāā upload - Upload local JSON as dataset
ā
āāā example (example operations)
ā āāā list - List examples in a dataset
ā āāā create - Add example to a dataset
ā āāā delete - Delete an example
ā
āāā evaluator (evaluator operations)
ā āāā list - List evaluators
ā āāā upload - Upload evaluator
ā āāā delete - Delete evaluator
ā
āāā experiment (experiment operations)
ā āāā list - List experiments
ā āāā get - Get experiment results
ā
āāā thread (thread operations)
ā āāā list - List conversation threads
ā āāā get - Get thread details
ā
āāā project (project operations)
āāā list - List tracing projects# List recent traces (most common operation)
langsmith trace list --limit 10 --project my-project --api-key $LANGSMITH_API_KEY
# List traces with metadata (timing, tokens, costs)
langsmith trace list --limit 10 --include-metadata --api-key $LANGSMITH_API_KEY
# Filter traces by time
langsmith trace list --last-n-minutes 60 --api-key $LANGSMITH_API_KEY
langsmith trace list --since 2025-01-20T10:00:00Z --api-key $LANGSMITH_API_KEY
# Get specific trace with full hierarchy
langsmith trace get <trace-id> --api-key $LANGSMITH_API_KEY
# List traces and show hierarchy inline
langsmith trace list --limit 5 --show-hierarchy --api-key $LANGSMITH_API_KEY
# Export traces to JSONL (one file per trace, includes all runs)
langsmith trace export ./traces --limit 20 --full --api-key $LANGSMITH_API_KEY
# Filter traces by performance
langsmith trace list --min-latency 5.0 --limit 10 --api-key $LANGSMITH_API_KEY # Slow traces (>= 5s)
langsmith trace list --error --last-n-minutes 60 --api-key $LANGSMITH_API_KEY # Failed traces
# List specific run types (flat list)
langsmith run list --run-type llm --limit 20 --api-key $LANGSMITH_API_KEY# Filter traces by feedback score using raw LangSmith query
langsmith trace list --filter 'and(eq(feedback_key, "correctness"), gte(feedback_score, 0.8))' --api-key $LANGSMITH_API_KEY{"run_id": "...", "trace_id": "...", "name": "...", "run_type": "...", "parent_run_id": "...", "inputs": {...}, "outputs": {...}}