OverviewHistoryStatsSecurity
npx skills add ...
Documentation
SKILL.md
npx skills add trailofbits/skills --skill vector-forge
Mutation-driven test vector generation. Finds implementations of a cryptographic algorithm or protocol, runs mutation testing to identify escaped mutants, then generates new test vectors that deliberately exercise the uncovered code paths. Compares before/after mutation kill rates to prove vector effectiveness. Use when generating cryptographic test vectors, measuring Wycheproof coverage gaps, finding escaped mutants via mutation testing, creating cross-implementation test suites, or improving test vector coverage for crypto primitives.
npx skills add trailofbits/skills --skill vector-forge
Uses mutation testing to systematically identify gaps in test vector coverage, then generates new test vectors that close those gaps. Measures effectiveness by comparing mutation kill rates before and after.
uv run trailmark fails, run:
Phase 1: Discovery → Find implementations to test ↓ Phase 2: Harness → Write/adapt test vector harness for each impl ↓ Phase 3: Baseline → Run mutation testing with existing vectors ↓ Phase 4: Escape Analysis → Classify escaped mutants by code path ↓ Phase 5: Vector Gen → Create test vectors targeting escapes ↓ Phase 6: Validation → Re-run mutation testing, compare before/after ↓ Output: Coverage Report + New Test Vectors
If the implementation already has test vectors:
Run mutation testing with existing test vectors only.
See references/mutation-frameworks.md for language-specific setup.
| Language | Framework | Command |
|---|---|---|
| Go | gremlins | gremlins unleash ./path/to/package |
| Rust | cargo-mutants | cargo mutants -j N --timeout T |
| Python | mutmut | mutmut run --paths-to-mutate src/ |
| C/C++ | Mull | mull-runner -test-framework=GoogleTest binary |
Always use parallel execution for large codebases:
cargo mutants -j 8 (Rust, 8 parallel workers)gremlins unleash --timeout-coefficient 3 (Go, increase timeouts)mutmut run --runner "pytest -x -q" (Python, fail-fast)Capture these metrics per implementation:
| Metric | Description |
|---|---|
| Total mutants | Number of mutations generated |
| Killed | Mutants caught by tests |
| Survived/Lived | Mutants NOT caught (these are the targets) |
| Not covered | Code paths no test reaches at all |
| Timed out | Ambiguous — resolve before comparing |
| Efficacy % | Killed / (Killed + Survived) |
| Coverage % | (Total - Not covered) / Total |
Save the full mutation log for Phase 4 analysis.
Classify each escaped (survived + not covered) mutant using the Trailmark call graph for reachability and blast radius analysis.
This phase MUST use the genotoxic skill's triage methodology. The call graph transforms mutation results from a flat list of survived mutants into an actionable, prioritized set of vector targets.
Build a Trailmark code graph for each implementation before triaging mutations:
The graph provides:
Mutation frameworks test the entire package. Filter results to only the files/functions that test vectors should exercise:
For each escaped mutant, map it to its containing function in the call graph and apply the genotoxic triage criteria:
| Graph Signal | Classification | Action |
|---|---|---|
| No callers in graph | False Positive | Dead code, skip |
| Only test callers | False Positive | Test infrastructure |
| Logging/display/formatting | False Positive | Cosmetic |
| Cross-package callers but NOT COVERED | Cross-Package Gap | See below |
| Reachable from public API, low CC | Missing Vector | Design targeted vector |
| Reachable from public API, high CC (>10) | Fuzzing Target | Both vector + fuzz harness |
| Validation/error-handling path | Negative Vector | Craft invalid input that triggers path |
| Optimization path (GLV, SIMD, batch) | Edge-Case Vector | Input that triggers optimization threshold |
|→^ after left shift (e.g. (t<<1) | carry) | Equivalent Mutant | Skip — bit 0 always 0, OR=XOR |
ct_eq &→| on Montgomery limbs | API-Unreachable | Needs library-internal tests, not vectors |
| Equivalent mutation (behavior unchanged) | False Positive | Skip |
Critical pitfall: Mutation frameworks often only run tests within the same package as the mutation. For Go (gremlins) and Rust (cargo-mutants), this means:
hash_to_curve/g2.go only runs tests in the
hash_to_curve package, NOT tests in the parent bls12381
package that imports itTo resolve cross-package gaps:
--test-pkg ./... (if supported)Using the call graph, rank surviving mutants by impact:
| Priority | Criteria | Example |
|---|---|---|
| P0 — Critical | Mutant weakens validation/equality/authentication | ct_eq: & → | makes equality permissive |
| P1 — High | Mutant in deserialization flag parsing | from_compressed: & → | accepts invalid flags |
| P2 — Medium | Mutant in field arithmetic internals | Fp::square: | → ^ corrupts computation |
| P3 — Low | Mutant in optimization path | phi endomorphism: only affects performance path |
| Skip | Formatting, display, equivalent mutation | Debug::fmt return value replacement |
Group escaped mutants by the code path they represent and the type of test vector needed:
Each group becomes a target for new test vectors in Phase 5.
For each escaped code path group, design test vectors that force execution through that path.
| Code Path Type | Vector Strategy |
|---|---|
| Point deserialization | Malformed points: wrong length, invalid field elements, off-curve, wrong subgroup, identity point |
| Signature verification | Valid sig + all single-bit corruptions of sig, pk, msg |
| Hash-to-curve | Known answer tests (KATs) with edge-case inputs: empty, single byte, max length |
| Aggregate operations | 1 signer, many signers, duplicate signers, mixed valid/invalid |
| Error handling | Every error path should have a vector that triggers it |
| Arithmetic edge cases | Zero, one, field modulus - 1, points at infinity |
| Serialization flags | Every valid flag combination + every invalid flag combination |
| Roundtrip integrity | For every valid deser vector, assert serialize(deserialize(b)) == b |
| Carry/reduction faults | Reimplement at reduced limb widths, inject faults, extract distinguishing inputs |
Each negative vector should have exactly one defect with everything else valid — this isolates which validation check is being tested. See references/vector-patterns.md for per-flag construction examples.
When mutation testing only applies local operator swaps, deeper architectural bugs (carry propagation, reduction overflow) go untested. To close this gap, reimplement the target algorithm at reduced limb widths (8, 16, 25, 32 bits) and deliberately inject faults — then generate vectors that catch them.
See references/fault-simulation.md for the full methodology: limb-width selection, fault injection catalog, vector extraction, and validation workflow.
Every new test vector MUST be verified against at least two independent implementations before being added to the suite:
Use Wycheproof JSON format (algorithm, testGroups[].tests[]
with tcId, comment, result, flags). See
references/vector-patterns.md
for the full schema.
Wycheproof contributions: Use Wycheproof's vectorgen tool rather
than formatting vector files directly. Supply the generated changes as an
envelope. The vectorgen tool can add, update, or replace vectors while
handling tcId assignment, test counts, canonical formatting, and schema
validation. Go-based generators can avoid the vectorgen CLI tool and instead
call the programmatic github.com/c2sp/wycheproof/vectorgen API.
See references/lessons-learned.md §14 and the upstream vectorgen guide for the current workflow and commands.
Re-run mutation testing with the new test vectors included.
Tip: Use per-file mutation testing for fast iteration during vector development (see references/lessons-learned.md §12). Only run full-crate tests for the final comparison.
| Metric | Baseline | With New Vectors | Delta |
|---|---|---|---|
| Killed | X | Y | Y - X |
| Survived | A | B | A - B (should decrease) |
| Not Covered | C | D | C - D (should decrease) |
| Efficacy % | E% | F% | F - E |
Vectors have both retroactive value (killing mutants in existing code) and proactive value (catching bugs in future implementations). Generate both kinds — boundary-condition vectors may not improve kill rates in mature libraries but will catch bugs in new implementations. See references/lessons-learned.md §13.
Retroactive (measurable): previously survived/uncovered mutants become killed, no regressions.
If kill rates don't change: the implementation's own tests likely already cover those paths. The vectors still add cross-implementation verification value. Document which case applies.
Write VECTOR_FORGE_REPORT.md covering: target algorithm,
implementations tested, baseline results, escape analysis,
new vectors generated, after results, before/after delta, and
conclusions. See
references/report-template.md
for the full template.
Before delivering:
VECTOR_FORGE_REPORT.md| Skill | Relationship |
|---|---|
| genotoxic (required for Phase 4) | Provides graph-informed triage — call graph cuts actionable mutants by 30-50% |
| mutation-testing (mewt/muton) | Use for Solidity; Vector Forge is language-agnostic |
| property-based-testing | Better than hand-crafted vectors for bitwise mutations in field arithmetic |
| testing-handbook-skills (fuzzing) | Functions with CC > 10 and surviving mutants need both vectors and fuzz harnesses |