langsmith-trace

LangSmith Trace

Add tracing to your agent and query traces for debugging. Supports Python and TypeScript.

[!IMPORTANT] This skill is tuned for AgentSeek template backends (LangGraph + middleware stacks). For general LangSmith concepts, see the upstream langsmith-skills repo.

[!CAUTION] Never pass --api-key as a CLI flag or expose API keys in shell commands / tool calls. The CLI reads LANGSMITH_API_KEY from the environment automatically. Using --api-key <value> leaks secrets into shell history, process listings, and agent tool-call logs. Always rely on the environment variable set in your shell profile or .env file.

Installation & Setup

1. Install the CLI

curl -sSL https://raw.githubusercontent.com/langchain-ai/langsmith-cli/main/scripts/install.sh | sh

The binary installs to ~/.local/bin/langsmith. If langsmith is not found after install, add to your shell profile:

echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.zshrc
source ~/.zshrc

2. Set your API key (environment variable only)

Add to ~/.zshrc (or the template's .env file) — the user must fill in their own key:

Key format must start with lsv2_pt_. Get yours at https://smith.langchain.com/settings.

The CLI reads this env var automatically — never use --api-key flags. Do not echo, print, or pass the key value in any shell command or tool call argument.

3. Verify

If you see a JSON array of projects, you're set. Common failures:

• command not found — ~/.local/bin not in PATH (see step 1)
• 401 Unauthorized — key is wrong or expired; regenerate at LangSmith settings
• Empty array [] — valid auth but no projects yet; create one in the UI or run a traced app

Adding Tracing

LangGraph / LangChain apps (automatic)

Just set environment variables — no code changes needed:

For serverless (Python): also set LANGCHAIN_CALLBACKS_BACKGROUND=false to ensure traces flush before function exit.

Non-LangChain apps

Use the @traceable decorator (Python) or traceable() wrapper (TypeScript) and wrap your LLM client:

Debugging a Trace (Step-by-Step)

This is the recommended workflow for investigating agent behavior:

Step 1: Find the right project

Look at last_run_start_time to find which project has recent activity. Don't assume — LangGraph apps default to the "default" project, not a named one.

Step 2: List recent traces

Step 3: Get the trace hierarchy

This returns the full run tree — use it to understand the agent's execution flow.

Step 4: Get IO for all runs in a trace

This gives you inputs and outputs for every run (LLM calls, tool calls, middleware).

Step 5: Drill into a specific run

AgentSeek Trace Structure

Our templates produce traces with this typical hierarchy:

Key points:

• The actual LLM call is always the innermost ChatOpenAI run
• Tool results are in the <tool_name> run (e.g., generate_cover, execute)
• Middleware wrappers are transparent — they add latency but the IO you care about is at the leaf nodes

Gotchas

--full vs --include-io on individual runs

Problem: langsmith run get <id> --full can return null for inputs/outputs, even though the data exists. Despite --full being documented as equivalent to --include-metadata --include-io --include-feedback, the underlying API behavior differs for individual run fetches in some CLI versions.

Solution: Always use --include-io explicitly when inspecting specific runs:

--full works reliably on trace export and run list, but has inconsistent behavior on individual run get calls. If this is fixed in a future CLI version, --include-io still works correctly — it's always safe.

Null IO even with --include-io

If inputs/outputs come back null even with --include-io, the project has IO logging disabled. Check for these environment variables in the template's .env:

Remove or set to false to enable IO capture for debugging.

Project confusion

LangGraph apps trace to the "default" project unless LANGSMITH_PROJECT is explicitly set. Always run langsmith project list first and check last_run_start_time to find where your traces actually landed.

Tips

• Add --format pretty for human-readable output during interactive debugging
• Use LANGSMITH_ENDPOINT env var if connecting to a self-hosted LangSmith instance
• The middleware ordering in the trace tree is configurable — your template may differ slightly from the diagram above

CLI Reference

Full command reference: reference/cli-commands.md