test-gap-analysis

Test Gap Analysis

Answer one question: which caller-visible production behaviors could change without an existing test failing? Mutation reasoning is a probe, not the goal. Inventory public outcomes first, then verify only credible gaps.

Decision flow

1. Set scope

Discover production and test files from manifests and file types. After a narrow search misses, inspect the current directory broadly before asking for paths.

When the request names a risk, turn it into a one-line public-outcome allowlist before reading code. An outcome is not in scope merely because the same method writes it. For money math, allow computed or returned amounts, rates, tier/boundary choice, percentage base/order, floors/caps, and rounding; exclude non-monetary state predicates (including derived booleans), identity, and formatting. Private code is in scope only to trace an allowed outcome.

Do not expand a focused request into a repository audit, plan artifact, or dashboard. Use source and tests directly for familiar frameworks. Invoke test-analysis-extensions only when discovery or assertion semantics are unclear.

2. Establish one baseline

Run the narrowest existing test command once. Choose it from the project manifest; Microsoft.Testing.Platform executables may require dotnet run. Confirm tests executed: exit 0 with build-only output is not green. If that one attempt cannot run the suite, do not troubleshoot the runner or try alternate commands for an advisory review; continue statically and label all candidates unverified. Do not infer a project-configuration cause from missing output; name a cause only when the command reports it.

Missing runner output limits only claims of empirical mutation survival. It does not make source-proven facts tentative: a public outcome with no reaching test is still No coverage, and an exact expected value derived from the unmodified implementation is still actionable. State the baseline limitation once, then give the static source/assertion conclusion directly instead of hedging every row.

For an advisory review such as "would tests catch this?", stop execution after that baseline. Source-to-assertion mapping is sufficient evidence for No coverage and Candidate survivor (unverified). Trace or run the unmodified code once only when an original value is unclear. Apply mutations only for explicit verification, an exhaustive audit, or closing gaps with tests.

Any focused mutation budget limits execution, not discovery. Keep every distinct unasserted public outcome in the inventory.

3. Inventory public outcomes

For each public entry point, map:

• input partitions: classifier arms, compound conditions, invalid and nearest-valid guard boundaries, and default cases;
• each independent observation: returned field/variant, exception type, invalid-input acceptance, public state transition, or external side effect;
• private-helper composition, constants/rates, rounding, retries, cancellation, and error propagation as observed through the public caller.

Use public input/sequence -> expected outcome -> existing assertion -> gap. One asserted return field does not cover another. One allowed result does not cover its denial.

Money math: inventory the no-op path, every rate/tier and exact boundary, operation order, percentage base or composition, floor/cap, and rounding. Trace private helpers through the public result. A test asserting only a broad range does not pin any exact amount. For each actionable money row, derive one witness input and its exact original result through the complete call chain; do not recommend a generic "assert the exact amount" without supplying that amount.

Ordered guards and retries: inventory invalid below minimum | first valid | last allowed or retryable | first blocked | later blocked. For an upper guard such as value >= limit, use limit - 1, limit, and limit + 1; the last witness exposes narrowing to value == limit. Inventory every accepted and rejected error class. When type matching is polymorphic, include a representative derived accepted type that would expose exact-runtime-type narrowing. A test at the first blocked value does not protect the last allowed or later blocked value.

Authorization: enumerate each relevant identity/role, resource class, and action from the caller's view. Untested false, forbidden, and unchanged-role outcomes are first-class security gaps. Do not analyze variants of an allowed path while a denial outcome remains uninventoried. Check each public surface:

• permission-returning APIs: every distinct role/resource class and every returned capability independently;
• action-dispatch APIs: each read/write/delete-style action branch, especially paths that must return denial;
• role/state transitions: accepted, rejected, invalid, null, and empty inputs, including outcomes that must leave state unchanged.

Reserve execution for wholly untested public branches before another variant of a partially covered helper. If more than five high-risk behaviors are unasserted, execute the top 3-5 and keep the rest visible as No coverage or Candidate survivor (unverified).

Execution never replaces the ledger. Before mutating or answering, classify every required outcome, including each invalid input, guard boundary, classifier arm, action, and denial.

Completeness checkpoint: before selecting findings, explicitly account for every independent mode/flag, both zero and negative for a <= 0 guard, every accepted exception class, and a representative derived accepted exception when matching is polymorphic. For a removed guard, trace the fallthrough: if it still produces the same public exception type, it is equivalent unless finer exception metadata is an established contract.

4. Admit only observable candidates

First replay each exact mutation against every existing asserted input or sequence with all arguments fixed. Any changed return, exception, state, or side effect is Likely killed; a dedicated single-purpose test is unnecessary. Never compare the mutant on one input with the original on another.

For survivors, choose a witness before execution or reporting and state witness -> original observation -> mutant observation. Reuse it in the smallest test. Admit it only when the last two differ publicly after tracing the full call chain; otherwise choose a distinguishing witness or drop it.

Exclude:

• edits that require inserting or reordering statements rather than changing or removing an existing expression, condition, constant, return, or side effect;
• edits that do not compile, including removal of a declaration whose value is still referenced;
• overflow behavior, exception message/ParamName metadata, or other semantics not established by the current contract, source intent, or tests;
• a removed guard or short-circuit that falls through to the same result, exception, state, and side effects;
• private representation changes that every public input sequence observes identically, even if the suite stays green;
• a mutation whose proposed test passes against both original and mutant;
• boundary edits that return the same value on the distinguishing input; for example, changing result < floor ? floor : result to <= is equivalent at equality because both branches return floor;
• a standalone auto-property or trivial one-line wrapper/predicate with no meaningful branch, calculation, or side effect, unless the user names it;
• hypothetical future impact, generated code, logging/formatting-only changes, impossible values, and duplicate syntax variants.

Missing direct assertions do not prove No coverage: first trace existing assertions through public callers and shared branches. Missing assertions make an observable candidate a survivor; they do not make an inert mutation meaningful.

5. Rank and classify

Rank: (1) security denials, financial outcomes, errors, and state changes; (2) wholly unasserted public outcomes; (3) boundaries or exact values reached by weak assertions; (4) alternate variants of already-asserted behavior.

Finish the inventory before selecting mutations or a verdict. One killed attempt, exception type, or switch arm does not clear its siblings.

Choose the verdict from the completed inventory:

• Strong when core branches and primary boundaries are protected and only a few validation or default-case variants remain;
• Mixed when meaningful coverage exists but at least one important outcome partition is unprotected;
• Weak when important outcomes are broadly unprotected.

A handful of validation gaps does not make an otherwise broad suite Mixed unless validation is the named risk or the gaps threaten security, data, or other contract-critical behavior.

When the inventory meets the Strong criteria above, lead with Strong and name the protected boundaries and dual assertions before listing minor gaps. Do not open with Mixed, "only core paths", or a risk-heavy dashboard.

Stop when existing assertions kill the remaining candidates or no credible public survivor remains. Do not mutate every operator merely to fill a report or calculate a score.

Outside explicit verification, an exhaustive audit, or a requested test addition, execute no mutations. Do not mutate to confirm obvious no coverage. For explicit verification, execute one representative candidate per distinct high-risk outcome in scope; do not stop after the first one or two while another guard, action branch, error class, or denial remains unclassified. Omit equivalent syntax variants.

6. Verify without creating false positives

Enter this phase only for explicit verification, an exhaustive audit, or a requested test addition.

1. Apply one candidate and confirm the diff changes exactly one intended expression.
2. Run the narrowest covering test: green means Survived, red means Killed, for that edit only.
3. Revert immediately and confirm the clean source/test baseline.
4. After a green run, re-check the public counterfactual. Execution proves the suite missed the edit, not that the edit changes behavior; drop inert or unobservable mutants.

Never leave mutations in the workspace. Before reporting, reconcile every unasserted high-risk outcome as Survived, Candidate survivor (unverified), No coverage, or omitted Equivalent. Stop when no credible public gap remains; do not fill a report with internal details or calculate a score unless the user requested an exhaustive audit.

7. Close gaps only when requested

1. Add focused tests only for executed Survived mutations or demonstrated No coverage behavior.
2. Cover every distinct gap in the requested scope before adding tests for alternate variants of an already-covered behavior.
3. Before editing, create a survivor-to-test checklist. Before stopping, map every verified survivor to an added test and every added test back to a verified survivor; a passing final suite alone does not prove completeness.
4. Preserve production code and existing tests when requested.
5. Prefer one behavior-focused test that kills related mutations over one test per syntax change.
6. Re-apply the original mutation and prove the new test kills it, then restore the source and run the narrow suite cleanly.
7. If the fixture or repository supplies a canonical mutation verifier, run that exact command after the tests are added and cite its successful result. Hand-created substitute mutations, a broad green suite, or a test-count increase do not replace the supplied oracle. Once every requested survivor maps to a focused test and the canonical verifier passes, stop; extra tests are not an advantage.
8. When the request requires existing source or test files to remain unchanged, compare each protected file byte-for-byte with its pre-edit snapshot and report that evidence. Before adding a test, prove its witness differs from every existing case on the relevant branch, boundary, or rounded result so a nominally new test does not duplicate existing coverage.

Output contract

Scale the response to the request.

For focused or small analysis, return:

1. A one-line verdict: Strong, Mixed, or Weak, with the reason.

2. For a Strong suite, one short strengths sentence naming the concrete protected boundaries, guards, or paired observations that justify the verdict.

3. One compact row per actionable Survived, Candidate survivor (unverified), or No coverage outcome. Before adding a row, apply the outcome allowlist when the request names a risk, then apply the observable-candidate rules; omit any candidate that fails either filter. Include every high-risk outcome, use one row per distinct public outcome, and consolidate only related low-risk variants:

   Risk	Public outcome	Change	Result/evidence	Smallest test

   Every gap needs a distinguishing witness and a concrete smallest test. An error-path gap must name an invalid input and the expected error/result.

4. For a Mixed or Weak suite, one short strengths sentence naming important killed behavior.

5. When the request names exclusions, one short scope sentence naming the generated, trivial, or unrelated code intentionally skipped.

Do not repeat the table in prose or report discarded mutants, tool chronology, or in-flight reasoning.

For an exhaustive audit, add counts for Killed / Survived / No coverage / Equivalent and group findings by risk. Count only executed or definitively classified candidates.

For test additions, name the tests added, the verified mutations they kill, and the successful final command.

Reliability rules

• A passing test that does not assert the changed outcome does not kill a mutation.
• Coverage is per behavior partition. One switch/ternary arm or compound input does not prove siblings: allow does not prove deny; read does not prove write; null does not prove empty or whitespace when those inputs have different caller-visible outcomes. A kill clears only the edit and path that ran.
• Private helpers reached through a public method remain in scope.
• Error semantics are language-specific: in Rust, ? propagation versus panic is observable behavior; in C#, exception type and whether an input guard accepts or rejects a value are observable behavior.
• Cross-check every exact amount or boundary result against the unmodified implementation or an existing exact assertion. If it cannot be checked, state the behavioral relation without inventing a number.
• Do not label a finding high-risk merely because a mutation survived.
• Never recommend a redundant test for behavior the existing suite already protects.

Validation

• Scope stayed proportional to the request
• The original suite passed, or static-only limits are explicit
• Every high-risk public outcome in scope was inventoried
• Original and mutant have different caller-visible observations
• Every outcome labeled Survived was executed; unexecuted candidates use Candidate survivor (unverified)
• Every temporary mutation was reverted
• Findings exclude trivial, generated, and equivalent changes
• Recommendations target only demonstrated gaps
• Every public entry-point branch and each accepted exception type in scope is explicitly accounted for
• A supplied canonical mutation verifier was run and reported, not replaced with an ad-hoc proxy