mantis-dedupe

Deduplicator (/mantis-dedupe)

System Goal

Duplicate Finding Merger. Evaluates lists of raw findings to cluster and consolidate identical or highly overlapping issues into singular, descriptive records.

Command Definition

• Command: /mantis-dedupe
• Description: Consolidates raw security findings to eliminate redundant reports.
• Arguments (optional; supplied by the orchestrator, consumed by Block A): --snapshot_root/--snapshot_id/--state_root. All absent -> MODE-OFF/legacy mode (behaves as today; snapshot gating disabled).

Input/Output Contract

• Reads:
  • workspace/findings/ (raw finding JSON files, ignoring .trash/).
  • workspace/archive/findings_pass_*/*.json and workspace/archive/loop*_findings/*.json (to skip findings already evaluated and triaged in previous passes).
  • workspace/.mantis_state.json (to track current loop pass).
• Writes:
  • Moves duplicate findings to workspace/findings/.trash/ after setting "status": "DUPLICATE" and "duplicate_of".
  • Sets "possible_duplicate_of" (soft, non-terminal) on NOT_MATCHED matches and stamps "discovery_commit" on current findings that lack it. Reads active_snapshot/snapshot_pinned from .mantis_state.json.
  • Appends transaction logs to workspace/.tx_log.jsonl.
  • Generates/executes merging script workspace/helpers/merge_findings.py.
  • Updates primary finding workspace/findings/<primary_id>.json (merges fields and history).
• Preconditions:
  • workspace/findings/ must exist and contain finding files.
• Idempotency Guarantee:
  • Cross-references against archived findings in workspace/archive/ to filter out any findings already processed in previous passes of this run. Snapshot-gated: a resolved archived finding on a differing snapshot is flagged as POSSIBLE REGRESSION (never silently filtered). Logs transactions to workspace/.tx_log.jsonl to support tracking and potential rollbacks. Deterministic merging rules implemented in merge_findings.py.

Instructions

Step 0: Locator Resolution (run first)

[!NOTE] CURRENT-PASS CHECK (defensive; the binding guarantee is on the harness per mantis-pipeline-adapter Scenario 2): if active_snapshot is present AND active_snapshot.pass != state.pass_number, treat the snapshot as STALE for this pass — STOP "stale active_snapshot: pass mismatch" or degrade as HALT (snapshot_pinned effectively false: no authoritative verdicts, Block B NOT_MATCHED, reproduce not_attempted). This catches a custom harness that preserved active_snapshot across the Stage 15 pass increment without re-pinning. The reference meta-agent re-pins every pass, so this check never fires there. Block B itself cannot detect this (it is snapshot_id-only, not pass-aware).

Notes: workspace/findings/, workspace/archive/, workspace/.tx_log.jsonl, workspace/helpers/ and .mantis_state.json are STATE-RELATIVE (under --state_root). Any code snippet you inspect for a finding is SNAPSHOT-RELATIVE (under CODE_ROOT). Never write under CODE_ROOT.

Review a list of security findings and merge duplicate findings that refer to the exact same security flaw or adjacent code paths.

Execute your task as follows:

1. Load Raw Findings & Archived Findings Queue:

   • List the contents of the directory and read the files in workspace/findings/. If the directory is empty or does not exist, notify the user and exit.
   • Important: Ignore hidden files and directories (such as the .trash/ subdirectory) when listing or processing findings.
   • Locate and load all archived finding JSON files from previous loop passes, if they exist, under workspace/archive/findings_pass_*/*.json and workspace/archive/loop*_findings/*.json. These files represent vulnerabilities that have already been fully evaluated, triaged, and potentially patched in previous passes.
   • Important: Do NOT read or deduplicate against workspace/historical_learnings.jsonl (VCS history), as we want to catch regressions if old bugs were reintroduced.

2. Filter Loop Duplicates (snapshot-gated). First, stamp discovery_commit on any CURRENT finding that lacks it, using active_snapshot.snapshot_id from .mantis_state.json (skip when unpinned). Then, for each current finding that matches an archived finding (by code_paths+title similarity), run:

   Signature-based candidate matching (Phase 3) — TIGHTENS, never replaces: signature may only PROMOTE a pair to "candidate for the pairwise snapshot check"; it may NEVER by itself cause a hard DUPLICATE/trash. A pair is a candidate for the Pairwise Snapshot Match Check below ONLY if it satisfies BOTH:

   • it matches under today's code_paths + title similarity, comparing code_paths entries line-inclusively (WITH their trailing :line); AND
   • (when both findings have a signature) their signature fields are equal. A signature match WITHOUT the code_paths + title agreement is NOT a duplicate — at most a soft possible_duplicate_of (keep the finding ACTIVE), never a trash. Rationale: signature strips the line number and all-but-first code_paths entry, so two DISTINCT bugs in the same file (e.g. parser.c:100 vs parser.c:900) with the same title+CWE share one signature; trashing on signature alone would silently delete a real finding. If EITHER lacks signature, use today's code_paths + title similarity matching unchanged.

   PAIRWISE SNAPSHOT MATCH CHECK (decides MATCHED vs NOT_MATCHED) — compares the CURRENT finding's discovery_commit against the ARCHIVED finding's discovery_commit for this pair (NOT against SNAPSHOT_ID):

   1. If snapshot_pinned is false AND there is NO active_snapshot in state (MODE-OFF) -> NOT_MATCHED. Stop. (In HALT — active_snapshot present but snapshot_pinned=false — do NOT short-circuit here; fall through to the pairwise comparison below, which will be NOT_MATCHED because the current finding's discovery_commit is a live: id that will not equal the archived one.)
   2. Read the CURRENT finding's discovery_commit and the ARCHIVED finding's discovery_commit:
      • If EITHER is missing, empty, or the literal "MIXED" -> NOT_MATCHED.
      • If they are NOT byte-for-byte equal to each other -> NOT_MATCHED.
      • If they ARE byte-for-byte equal to each other (both present, non-MIXED) -> MATCHED. There is no other route to MATCHED; never fuzzy-compare. The global "default the field and proceed" backward-compat rule does NOT apply to discovery_commit: absent = NOT_MATCHED. (There is NO separate "dirty" gate: a dirty tree's SNAPSHOT_ID already embeds the working-tree content hash, so within-pass findings MATCH and cross-pass bare-commit findings do not.) Note: this is a PAIRWISE check (current vs archived), NOT a check against the global SNAPSHOT_ID — dedupe stamps the current finding's discovery_commit to SNAPSHOT_ID in Step 2 above, so a check against SNAPSHOT_ID would always be MATCHED and would trash reintroduced/regression bugs as DUPLICATE.

   Then, using the idempotency rule (Input/Output Contract → Idempotency Guarantee) to avoid double-writes, decide mechanically:

   • MATCHED (both present and equal): soft-delete the current finding as a loop-duplicate exactly as before — set "status": "DUPLICATE" and "duplicate_of": "<archived_uuid>", clear possible_duplicate_of if present, ensure mkdir -p workspace/findings/.trash/, move it there, and log a loop_filter transaction in workspace/.tx_log.jsonl. If the current finding lacks lineage_id but the archived finding has one, inherit the archived finding's lineage_id onto the current finding before moving it (so the lineage chain is preserved across the merge).
   • NOT_MATCHED (differ, or either absent): do NOT set DUPLICATE and do NOT move to trash. Keep the current finding ACTIVE and set "possible_duplicate_of": "<archived_uuid>" (a soft, non-terminal hint). If the current finding lacks lineage_id but the archived finding has one, inherit the archived finding's lineage_id onto the current finding (so the lineage chain is preserved for report folding even when the findings are on different snapshots).

   [!IMPORTANT] STATUS & DUPLICATE INVARIANTS:

   • A finding MUST NOT carry both duplicate_of and possible_duplicate_of pointing to the same target UUID.
   • status = "DUPLICATE" MUST NOT coexist with possible_duplicate_of pointing to the same target UUID.
   • Under NOT_MATCHED (outside the MODE-OFF fallback exception), the finding's status MUST remain active (e.g. VALID, PROVISIONALLY_VALID, NEEDS_RESEARCH), duplicate_of MUST NOT be set, and the finding MUST NOT be moved to .trash/. Setting possible_duplicate_of is a non-terminal hint only.

   • POSSIBLE REGRESSION: if the archived match has a RESOLVED status (patch_status in {VERIFIED_SECURE,MITIGATION_PROPOSED} OR status==FALSE_POSITIVE OR production_viability==NON_VIABLE) AND the pair is NOT MATCHED, keep the current finding ACTIVE, add a history note "POSSIBLE REGRESSION vs <archived_uuid>", and do NOT filter it. (A reverted fix re-discovered on new code must never be trashed.)
   • EXACT-UUID retry exception (unchanged): if the current finding has the EXACT SAME UUID as the archived one, it was intentionally copied back for a retry — do NOT filter it (keep as-is).
   • Permanently-unpinned exception (MODE-OFF only — no active_snapshot): if there is NO active_snapshot in state (MODE-OFF = today's default; a target with no snapshot boundary, e.g. a live endpoint), snapshot_pinned is false at the PASS level (active_snapshot.snapshot_pinned — it is NOT a per-finding field) and Block B is uninformative — fall back to today's dedup by signature if present, else stable_key = normalized_title + first code_paths entry including its trailing :line (line-inclusive, same as Step 2). Rationale: stripping :line would collapse two DISTINCT bugs in the same file (e.g. parser.c:100 vs parser.c:900) with the same title+CWE into one — silently deleting a real finding. Note: signature itself strips :line by design (it is a coarse identity for cross-pass lineage, not a dedup key); this fallback therefore prefers signature only when stable_key's line-inclusive match ALSO agrees, never on signature alone. This preserves dedup for targets that can never MATCH. When active_snapshot IS present but unpinned (HALT mode), this exception does NOT fire: keep the snapshot-gated behavior above (NOT_MATCHED → keep ACTIVE + possible_duplicate_of, never DUPLICATE). POSSIBLE REGRESSION takes precedence over this fallback: a resolved-archived finding paired with a NOT_MATCHED current finding is ALWAYS kept active (never trashed), regardless of mode.

3. 3-Tier Deduplication Ladder (Deterministic & Semantic Matching): When resolving finding lineages and evaluating deduplication candidates, the deduplicator uses a hierarchical 3-tier ladder designed for sub-millisecond fast-path execution with semantic RAG fallback:

   • Tier 1 (Fast-Path Exact Heuristic Anchors — < 1ms, 0 tokens):

     1. Exact content identity signature match: hashlib.sha256(canonical_fp | canonical_cwe | target_symbol) (invariant to line shifts, backticks, and title paraphrasing).
     2. Exact canonical_filepath + normalized CWE + target_symbol match.
     3. Strict line proximity window ($\le 3$ lines) on exact canonical filepath when target symbol is empty.
     • Result: If matched, immediately inherit ancestor lineage_id without LLM or embedding overhead.

   • Tier 2 (RCA Normalization): If Tier 1 heuristic matching does not produce an exact anchor, synthesize a standardized 5-line Root Cause Analysis (RCA) summary:

     • Component: Normalized canonical filepath and symbol.
     • Vulnerability Class: Canonical CWE taxonomy identifier and name.
     • Root Cause Mechanism: Underlying programming or logic defect.
     • Failure Condition: Specific input, state, or boundary condition.
     • Taint Dataflow: Source-to-sink dataflow trajectory.

   • Tier 3 (Vector Embedding & Cosine Similarity Scan):

     • Project the standardized RCA summary into dense vector embeddings using the configured multi-provider embedding engine (default: vertex_ai/gemini-embedding-001).
     • Perform nearest-neighbor scan over historical lineage_vectors in the database using bounded cosine similarity.
     • CWE Family & Class Structural Guard: When comparing query finding vectors against candidate lineage records, if both findings have explicit CWE classifications (e.g. CWE-89 vs CWE-78), normalize them. If they belong to distinct, incompatible CWE IDs, skip candidate vector comparison entirely to structurally prevent false-merging distinct vulnerability classes regardless of cosine score.
     • Positive threshold ($\ge 0.90$): Semantically equivalent findings with differing phrasing, scanner labels, or line shifts merge into the same ancestor lineage_id (default: 0.90, configurable via EMBEDDING_SIMILARITY_THRESHOLD).
     • Negative threshold ($< 0.70$): Distinct vulnerability classes (e.g., SQLi vs Command Injection, Stored vs Reflected XSS) maintain low similarity and fail closed, minting a fresh unique lineage_id.

4. Filter Duplicate Findings in Current Batch: Check the current findings against each other to find duplicates. Two findings are duplicates ONLY if they share the same code_paths entry line-inclusively (WITH trailing :line) AND have the same or highly similar title. If multiple findings refer to the exact same flaw at the same location, they must be merged. Findings at different lines in the same file are DISTINCT — never merge them.

5. Map/Reduce Chunking Strategy (For Scale): If there are many finding files (e.g., > 20 items), use a Map/Reduce approach to group them by target file or component before checking for overlaps to avoid context window limits.

6. Token-Optimized Consolidation and Merging: To minimize LLM output tokens and prevent data loss, do not manually rewrite or output the merged JSON files in your response. Instead, follow this pattern:

   1. Identify Duplicates: Internally map which findings are duplicates of a primary finding.
   2. Reusable Deterministic Scripting (versioned): Write a reusable helper script (e.g. workspace/helpers/merge_findings.py) whose FIRST LINE is exactly # MANTIS_HELPER_VERSION = 2. Before reusing an existing helper, grep its first lines for MANTIS_HELPER_VERSION = 2; if that marker is absent or a different integer (a helper left by an older pipeline version), REGENERATE the helper. Only reuse it when the marker matches. The script must follow these deterministic rules:
      • Title: Pick the most comprehensive and descriptive title.
      • ID: Preserve the unique "id" of the primary finding being kept.
      • Severity: Pick the highest severity level specified among the merged items.
      • Privileges Required: Inherit the most severe privilege requirement (priority: NONE > LOW > HIGH).
      • Attacker Position: Inherit the most critical position requirement (priority: EXTERNAL > INTERNAL_NETWORK > IN_CLUSTER > LOCAL > HOST_SYSTEM > SUPPLY_CHAIN > PHYSICAL_TEMPORARY > PHYSICAL_LONG_TERM).
      • User Interaction: Inherit the most severe user interaction requirement (priority: NONE > REQUIRED).
      • Code Paths: Collect and deduplicate all file paths and line numbers into a single unique array.
      • Description, Mitigation, & Impact: Concatenate cleanly.
      • History: Concatenate and preserve all "history" entries from the merged findings. Append a new entry to the "history" array for this merge action conforming to the schema (containing "stage": "dedupe", "action": "merge", "details": "Merged duplicate findings: [comma-separated-ids]", "pass_number": <current_pass_number>, and "timestamp": "<current_iso8601_timestamp>").
      • Unknown keys & provenance (MANDATORY): The script MUST copy through EVERY key it does not explicitly handle (including discovery_commit, repro_snapshot_id, patch_base_snapshot, possible_duplicate_of, signature, lineage_id, cwe) from the primary finding onto the merged object — never drop unknown fields. It MUST REFUSE to merge two findings whose discovery_commit values differ (they describe different code versions); leave them separate and log the refusal. When merging findings that all share one discovery_commit, preserve it unchanged.
   3. Execute the Script: Run your script to update the primary finding's file (workspace/findings/<primary_id>.json) on disk.

7. Transactional Staged Clean Up: Do not permanently delete redundant files. Ensure the trash directory exists (e.g., mkdir -p workspace/findings/.trash/). Before moving, the script must update the duplicate finding files, setting "status": "DUPLICATE" and "duplicate_of": "<primary_uuid>". Move the merged duplicate .json files to the trash staging directory (workspace/findings/.trash/). For every file moved, append a transaction record to workspace/.tx_log.jsonl.

   Transaction Log Schema Format (workspace/.tx_log.jsonl)

   Each line must be a self-contained JSON object documenting the transaction:

   {"timestamp": "2026-07-14T15:13:00Z", "action": "loop_filter | dedupe_merge", "primary_uuid": "[UUID] (or null for loop_filter)", "moved_uuid": "[UUID]"}

   This cleans up the directory for downstream stages while preserving rollback capability.

When complete, notify the user.