mantis-structural-index

Structural Code Index Builder

This is an optional first-class stage in the Pass Lifecycle Contract. It runs immediately after the snapshot is pinned (Block D), before the first code-reading analysis stage (summarize/architecture). It only needs CODE_ROOT + SNAPSHOT_ID and must not depend on architecture/KB.

System Goal

Structural Code Index Builder. Builds a content-addressed semantic-unit index from source code using capability-based per-partition backend selection, degrading gracefully to grep. Provides find_callers(symbol), get_function_boundary(file, line), and call-site awareness to improve LLM reasoning quality during discovery — supplementing (never replacing) grep-based call-site discovery with structural data. The index is composed from independent semantic compilation units, persisted as a manifest + SQLite catalog, and exposed through a bounded query helper.

Command Definition

• Command: /mantis-structural-index
• Description: Build a content-addressed semantic-unit index from source code under CODE_ROOT.
• Arguments (optional; supplied by the orchestrator, consumed by Block A): --snapshot_root/--snapshot_id/--state_root. All absent → MODE-OFF/legacy mode (reads source from the current directory, writes index to ./workspace/kb/structural_index/).

Input/Output Contract

• Reads:
  • workspace/.mantis_state.json (to read active_snapshot for provenance checking and snapshot-aware rebuild logic).
  • workspace/kb/structural_index/manifest.json (to check snapshot_id for reuse-on-match idempotency — primary check).
  • workspace/kb/structural_index.jsonl (backward-compat provenance check if manifest absent).
  • workspace/kb/structural_index/units/ (content-addressed cache for incremental unit reuse).
  • workspace/kb/structural_index/native/ and sidecar provenance.json files (prebuilt index attachments and metadata manifests).
  • CODE_ROOT source files (via generated helper script — the script parses all source files under CODE_ROOT).
• Writes:
  • workspace/kb/structural_index/manifest.json (STATE-RELATIVE — atomic commit point, written LAST).
  • workspace/kb/structural_index/catalog.sqlite (STATE-RELATIVE — query-optimized serving store).
  • workspace/kb/structural_index/units/ (STATE-RELATIVE — content-addressed immutable unit outputs).
  • workspace/kb/structural_index/shards/ (STATE-RELATIVE — partitioned serving data for large corpora).
  • workspace/kb/structural_index/native/ (STATE-RELATIVE — prebuilt index attachments: SCIP, Kythe, LSIF).
  • workspace/kb/structural_index/tmp/ (STATE-RELATIVE — temporary objects during build).
  • workspace/kb/structural_index.jsonl (STATE-RELATIVE — compatibility pointer; full export below threshold).
  • workspace/helpers/build_structural_index.py (STATE-RELATIVE — the builder helper script).
  • workspace/helpers/query_structural_index.py (STATE-RELATIVE — the query helper script).
• Preconditions:
  • Source files must exist under CODE_ROOT. If CODE_ROOT is not resolved (MODE-OFF and no readable active_snapshot), build against the current directory with snapshot_id set to "unknown". Do NOT skip — this is the standalone-efficiency case.
• Inert until wired: This skill returns an empty index until a caller invokes it (the harness, mantis-plan, or mantis-researcher). It never fails — it simply returns an empty index if tools are unavailable or source cannot be parsed.
• Idempotency Guarantee:
  • Read-only on CODE_ROOT. Writes only to STATE-RELATIVE paths. Re-running with the same CODE_ROOT and SNAPSHOT_ID reuses the existing index (manifest snapshot_id match) rather than rebuilding — except in MODE-OFF, where it always rebuilds. Individual semantic units are reused across snapshots when their content-addressed cache keys match (incremental rebuild).

Instructions

Step 0: Locator Resolution (run first)

This is a CODE-READING stage — it reads target source files under CODE_ROOT via the helper script. Block A step 0's findings-only skip does NOT apply.

Step 1: Idempotency / Freshness Check (MANDATORY FIRST)

MANDATORY FIRST STEP: Before writing any build scripts, probing backends, or extracting symbols, check for MODE-OFF or an existing manifest:

1. MODE-OFF check (FIRST): In MODE-OFF (SNAPSHOT_ID is "unknown" or absent) → always rebuild. Do NOT reuse a previous "unknown" index, because the live tree is mutable and "unknown" is a constant (not a freshness signal). Skip directly to Step 2 (individual units whose content has not changed may still hit the content-addressed cache).
2. If workspace/kb/structural_index/manifest.json exists, read its snapshot_id field.
3. If snapshot_id matches the current SNAPSHOT_ID (and SNAPSHOT_ID is NOT "unknown") → reuse the index immediately and STOP. Do NOT invoke Step 2, probe backends, or write build scripts. This bounds cost across retries/crash-resume.
4. If manifest.json is absent but workspace/kb/structural_index.jsonl exists (backward compat), read its provenance header (_provenance, snapshot_id keys). If snapshot_id matches (and is NOT "unknown") → reuse and STOP. Otherwise proceed to rebuild.
5. If SNAPSHOT_ID differs → proceed to rebuild (Steps 2–5). For incremental reuse: before rebuilding a semantic unit, check units/ for a content-addressed cache hit (see Content-Addressed Cache Key below). A cache hit reuses the unit output without re-extraction.

Step 2: Select Backend per Partition

Partition the codebase into semantic units (see Per-Language Semantic Units below). For each unit, select the most precise backend available in this environment using a capability-based, per-partition decision — not a single global ladder. The following are examples, not an exhaustive enum. The helper probes each tier per partition and selects the highest available:

1. Snapshot-matched SCIP / LSIF / Kythe / clangd-static / remote index (precision: semantic) — if a pre-built index matching the current snapshot_id or root_fingerprint is available. Most precise: full type-aware cross-reference, call hierarchy, and hover/signature data.
2. Compiler / typechecker-backed extractor (precision: typecheck) — if compile_commands.json, build context, or typechecker is available. Type-accurate symbol resolution and call edges.
3. Language-aware AST extraction (precision: ast) — if tree-sitter, ast-grep, or a language-specific parser is available. Full AST parsing: function boundaries, call expressions, signatures.
4. Symbol-only extraction (precision: symbol-only) — if ctags or equivalent is on PATH. Symbol table only (function definitions, locations — no call graph). Call-site extraction uses a lightweight regex pass within known function boundaries.
5. Heuristic fallback (precision: heuristic) — Python stdlib regex pass over source files. Identifies function definitions and call patterns using language-agnostic heuristics. Less precise but zero-dependency.
6. Coverage-only manifest + lexical fallback (precision: coverage-only) — grep; no structural index is written. Manifest records status: "empty". Consumers fall back to grep-based discovery (today's behavior byte-for-byte).

Native Index Probing & Resolution Rules:

• Probe instruction: Before evaluating per-partition backends, probe workspace/kb/structural_index/native/ and subdirectories native/{scip,lsif,kythe}/ for prebuilt index files.
• Snapshot-declaration convention: Because native formats (SCIP, LSIF, Kythe) do not embed snapshot identity directly in their binary payload, prebuilt indexes MUST declare their target snapshot using a sidecar provenance.json manifest located at workspace/kb/structural_index/native/provenance.json or native/<kind>/provenance.json. The manifest contains an array of attachments: [{"kind": "scip|lsif|kythe", "path": "...", "snapshot_id": "...", "root_fingerprint": "...", "language": "...", "indexer": "...", "precision": "semantic", "files": [...]}]. A native index is matched if its declared snapshot_id equals SNAPSHOT_ID (when SNAPSHOT_ID != "unknown") or its root_fingerprint matches the workspace's calculated root fingerprint. If the files array is absent or empty, treat the native index as covering no individual files directly (record in manifest.native_indexes but do not update coverage rows; fall through to lower tiers for all files).
• Record-and-Defer Ingestion Rule: Parsing raw binary native indexes (e.g. SCIP protobuf) in pure Python without dependencies is costly and complex. Option A builder scripts MUST detect matching prebuilt native indexes, record their entries in manifest.native_indexes, and set the coverage table backend (e.g., "scip" or "scip-clangd") for all files listed in the provenance manifest. If catalog.sqlite is NOT populated with symbols from the native index (raw binary deferred to harness/MCP readers), set coverage.status = "deferred" and precision = "deferred" (with indexed_files = 0). This prevents the query helper from claiming an un-ingested partition is "authoritative empty" at semantic precision, ensuring consumers run the mandatory grep fallback. When catalog.sqlite IS populated (e.g., via Option B pre-ingestion or scip-to-sqlite), set precision = "semantic" and status = "indexed".

LSP is NOT equivalent to SCIP / LSIF. LSP is an interactive protocol whose workspace state may be partial or mutable. Use it only when the server can demonstrate snapshot identity AND complete workspace coverage. A running language server does not automatically qualify as a semantic backend.

SCIP merging. SCIP explicitly permits merging complementary information from indexers with different precision levels. Its format records indexer version and per-document language metadata. The builder MAY merge results from multiple indexers (e.g., a SCIP index for Go + tree-sitter for Python) within a single catalog.

The determinism lives in a runtime-generated versioned helper (build_structural_index.py, # MANTIS_HELPER_VERSION = 5, grep-and-regenerate on reuse) that probes and selects backends per partition. No shipped binaries; air-gapped-safe.

Deterministic partial coverage. On very large source trees, a full rebuild can dominate stage-0 wall-clock. Replace discovery-order truncation with a deterministic priority queue:

1. Explicit target files and symbols (from plan.json, if available).
2. Changed units and known reverse dependencies (from Block E diff + reverse-dependency edges in the existing catalog).
3. Containing packages/modules and direct imports of tier 1+2 units.
4. Remaining units in normalized path order.

Apply deterministic max_units (default: 10000) or max_source_bytes (default: 500MB) bounds to that queue. Persist deferred units in the manifest so another invocation can resume. Statuses: complete, partial, empty, failed.

A consumer should only interpret "no callers" strongly when the relevant partition is complete, snapshot-matched, covered, and semantically precise. Otherwise the answer is "no indexed callers", followed by the existing lexical fallback. The index is HINT-only, so partial coverage is safe (grep remains authoritative).

Step 3: Write and Run Helper Scripts

Two runtime-generated helpers are used. Both follow the grep-and-regenerate pattern: before reuse, grep the first line for the version marker; if absent or a different integer, REGENERATE.

Builder: build_structural_index.py

1. Write the builder to workspace/helpers/build_structural_index.py. The FIRST LINE MUST be exactly # MANTIS_HELPER_VERSION = 5. Before reusing an existing helper, grep its first lines for MANTIS_HELPER_VERSION = 5; if that marker is absent or a different integer, REGENERATE the helper.
2. The builder partitions the codebase into semantic units (see Per-Language Semantic Units), computes content-addressed cache keys, checks units/ for reuse, selects a backend per partition (Step 2), extracts symbols + edges, and writes results to catalog.sqlite.
3. The builder must extract:
   • Symbols: symbol_id, name, qualified_name, namespace, language, file_path, start_line, end_line, kind, signature, backend, precision.
   • Call edges: caller_id, callee_id, callee_name, file_path, line, edge_kind.
   • Function boundaries: symbol_id, file_path, start_line, end_line, signature, language.
   • Coverage: file_path, indexed, backend, precision, unit_cache_key, status.
4. The builder writes the manifest LAST (Step 4) and the compatibility pointer.
5. The builder MUST use ABSOLUTE paths and set its own working directory (Block A step 6). It MUST NOT write anything under CODE_ROOT when snapshot_pinned is true (Block A step 4). Backends that produce sidecar files (ctags tags, cscope cscope.out, clangd cache) MUST be redirected to STATE-RELATIVE paths: ctags -f <state>/helpers/tags, cscope -f <state>/helpers/cscope.out, CLANGD_INDEX_STORAGE=<state>/helpers/. Read-only LSP/SCIP queries to a running server need no redirect.

Query helper: query_structural_index.py

1. Write the query helper to workspace/helpers/query_structural_index.py. The FIRST LINE MUST be exactly # MANTIS_HELPER_VERSION = 5. Before reusing an existing helper, grep its first lines for MANTIS_HELPER_VERSION = 5; if absent or a different integer, REGENERATE.
2. The query helper provides bounded, paginated operations against catalog.sqlite (or a remote endpoint — identical API). It IS the consumption contract (see Query Interface below).
3. If catalog.sqlite is absent but structural_index.jsonl exists, the query helper falls back to linear scanning of the JSONL file (slower but functional). If both are absent, it returns empty results with coverage.partition_status = "empty".

Step 4: Write Manifest and Commit

1. Write the manifest to workspace/kb/structural_index/tmp/manifest.json first. Then atomically rename it to workspace/kb/structural_index/manifest.json. This is the atomic commit point — the manifest is written LAST, after all units, catalog, and coverage data are written.
2. Write a compatibility pointer to workspace/kb/structural_index.jsonl (STATE-RELATIVE — NEVER under CODE_ROOT). Below a configurable threshold (default: 10K records), emit a complete JSONL export (provenance header + all records). Above the threshold, emit only a provenance header with compat_pointer.full_export = false and compat_pointer.symbol_count set. Large consumers MUST use the query interface.
3. The manifest provider.backend_versions field records which backend was used per language (e.g., {"go": {"backend_name": "scip-clangd", "precision": "semantic"}, "python": {"backend_name": "tree-sitter", "precision": "ast"}}), so consumers know the precision level.
4. Interrupted builds leave unreferenced temp objects in tmp/ without corrupting the last published index. On resume, check manifest.json status and snapshot_id (Step 1).

Step 5: Return Results / Notify Caller

1. Return the path to manifest.json, catalog.sqlite, and the query helper. Include a summary (unit count, symbol count, call-edge count, backends used, coverage status).
2. If the index is empty (no tools available or no source files found), notify the caller: "Structural index is empty — structural context unavailable."
3. Do not notify the user directly — this skill is invoked as a sub-agent by the harness, planner, or researcher.

Index Schema and On-Disk Contract

On-Disk Layout

Manifest Schema (manifest.json)

Atomic commit: The manifest is written LAST (atomic rename from tmp/). Interrupted builds leave unreferenced temp objects without corrupting the last published index.

SQLite Catalog Schema (catalog.sqlite)

Partitioning: When symbols exceed 500K or edges exceed 2M, partition into shard SQLite files under shards/ by lang:{language}:bucket:{NN} (stable hash). Each shard has the same schema. The query helper routes queries to the correct shard(s) based on the manifest.

Content-Addressed Cache Key

Critical: snapshot_id is NOT in the cache key. It goes in provenance only. This permits reuse across commits, branches, and snapshots with identical units.

Canonical Symbol IDs

Native IDs (when semantic backend is available):

• SCIP: scip:{symbol}
• Kythe: kythe:{uri}
• clangd: clangd:{usr}

Fallback IDs (when no semantic backend):

This distinguishes namespaces, overloads, methods, local functions, and duplicate names across languages or repositories.

Edge Kinds

Edge kinds are kept explicitly separate — never silently merged:

• direct — statically resolved call to a known symbol
• indirect — function pointer, closure, callback
• virtual — virtual method dispatch (runtime-resolved)
• macro — call introduced by macro expansion
• unresolved — callee name found but no symbol_id resolved

Per-Language Semantic Units

Fallback Granularity Rule: For the fallback tier (regex, AST parser, or lightweight crawler without a language build system), each source file MUST be its own independent semantic unit. Never bundle multiple source files into a single fallback unit. Per-file unit isolation is essential for content-addressed cache efficiency — when one file changes in a future snapshot, only that single file's unit invalidates while all unchanged files hit the cache (reused = N-1).

Baseline + Delta Overlay

• Static baseline: Built once in CI for the complete snapshot.
• Incremental cache: Immutable unit outputs reused across snapshots.
• Delta overlay: Changed/target files indexed locally, layered over baseline.
• Remote serving: Optional when global index is too costly locally.

At query time, the query helper merges baseline + overlay results. Overlay symbols supersede baseline symbols for the same file. The manifest records provider.kind = "baseline+overlay" with baseline manifest reference and overlay unit count.

Compatibility JSONL Export

The old structural_index.jsonl format is retained as a compatibility export only — it is NOT the query contract. Below the threshold (default 10K records), a complete JSONL export is emitted. Above the threshold, only a provenance pointer is written. Large consumers MUST use the query interface.

Line 1 — provenance header:

Lines 2+ — structural records (one per line, _type discriminator):

Provenance header fields:

Record types:

Snapshot Safety

1. Build from CODE_ROOT, not live tree (when pinned). When the snapshot is pinned, the structural index is built from the pinned CODE_ROOT, ensuring it reflects the exact bytes the pipeline is analyzing.
2. Reuse-on-match. If manifest.json already carries the current SNAPSHOT_ID (and SNAPSHOT_ID is NOT "unknown" — in MODE-OFF, always rebuild), reuse it — do not rebuild. Rebuild when SNAPSHOT_ID differs or is "unknown". Individual units may still be reused from the content-addressed cache. This bounds cost across retries/crash-resume.
3. Manifest atomicity. The manifest is written LAST via atomic rename from tmp/. Interrupted builds leave unreferenced temp objects without corrupting the last published index. On resume, check manifest.json status and snapshot_id.
4. STALE flag in HALT mode. When snapshot_pinned is false (HALT), the index may be built from the unpinned CODE_ROOT but is marked as potentially stale. Consumers treat structural hints as advisory.
5. MODE-OFF: build, do not skip. When active_snapshot is absent (MODE-OFF), build against the current directory (cwd) with provenance snapshot_id set to "unknown". This is the standalone-efficiency case — the index is still useful for the researcher/planner even without snapshot pinning. Do NOT return an empty index merely because the snapshot is absent.

Consumption Contract

These are runtime instructions for callers — they define how consumers use the structural index. The structural index is a HINT-only enhancement; skills that do not use it behave exactly as they do today.

Query Interface

The query helper IS the contract. All consumers use query_structural_index.py (or a compatible remote endpoint). The JSONL file is NOT the query contract — it is a compatibility export only.

Operations:

1. resolve_symbol(name, language?, file?, namespace?) → {results, total, ambiguous, coverage}

   • Returns ALL matches — no silent selection among ambiguous symbols.
   • Caller MUST disambiguate before calling find_callers / find_callees.

2. find_callers(symbol_id, limit=100, offset=0) → {results, total, has_more, coverage}

   • Bounded, paginated caller lookup.
   • Each result carries precision, backend, edge_kind.
   • Empty results carry coverage.partition_status.

3. find_callees(symbol_id, limit=100, offset=0) → {results, total, has_more, coverage}

   • Same shape as find_callers.

4. get_function_boundary(file, line) → {symbol_id, start_line, end_line, signature, precision, backend}

5. get_coverage(file?) → {total_files, indexed_files, failed, deferred, partition_status, backends_used}

Key properties:

• Bounded results and pagination (prevents loading entire index into memory).
• Explicit name resolution before graph traversal (no silent selection among ambiguous symbols).
• Precision and backend attached to every result.
• Coverage attached to empty results.
• Identical operations for local and remote providers (remote via MANTIS_STRUCTURAL_INDEX_URL env var or manifest provider.kind = "remote").

Coverage on empty results:

mantis-plan

• Use the structural index query helper for function-level dependency fan-out. When planning investigations, call resolve_symbol() then find_callers() to identify all functions that call into a target — this broadens the audit set beyond single-file analysis.
• The structural index decides ORDER of investigations (which functions to audit first based on call-graph centrality), never MEMBERSHIP. It may broaden the audit set (safe over-reporting), but must never REMOVE or drop a file; the planner's existing logic remains the membership floor.

mantis-researcher

• Wave 1 (Rapid Triage): Run a repo-wide grep for the function name to build the exhaustive set of candidate call-sites — this is the mandatory floor. Then use the structural index query helper (resolve_symbol then find_callers) to RANK and prioritize which call-sites to audit first (the index distinguishes actual calls from comments/strings/variable names). Audit the union of both result sets — the structural index may miss macro-based calls, function pointers, and dynamic dispatch, so grep remains the floor.
• Wave 2 (Deep Audit): Use get_function_boundary(file, line) to start with the enclosing function, expanding to callers/callees/file as needed for cross-function context — this saves context while preserving coverage.
• If the structural index is absent or empty, fall back to grep-based discovery (today's behavior). The structural index is a coverage HINT only — it improves audit quality but is never required.

Safety

Non-negotiable invariants:

1. Agnostic. Nothing is ever required; no-tool / parse-fail / not-invoked → empty index → grep fallback = today's behavior byte-for-byte. Optional in the Pass Lifecycle Contract; a non-conformant harness simply skips it.
2. HINT-only / union / never MEMBERSHIP. Consumers audit the union with grep; a symbol the index misses must still be reachable by the exhaustive sweep. The structural index decides ORDER, never MEMBERSHIP. The query contract enforces this: find_callers returns HINTs, never authoritative membership — consumers MUST union with grep.
3. No verdicts, touches no findings. Cannot violate INV-1 and cannot itself drop a finding.
4. Narrow scope: source cross-reference only. Binary/build-derived reachability ("is it compiled into production") is explicitly out of scope — a dev customization, not part of this skill, because absence-from-a-build can hide a real finding (INV-2). Do not fold build-derived reachability into this skill.

A reference blueprint is available at mantis-pipeline-adapter/references/mantis-structural-index.md. It is a stub that points to this SKILL.md as the single source of truth — do not duplicate spec content there.