differential-fuzzer

Differential Fuzzer

Always load Debugging skill for reference

The differential fuzzer compares Turso results against SQLite for generated SQL statements to find correctness bugs.

Location

testing/differential-oracle/fuzzer/

Running the Fuzzer

Single Run

# Basic run (100 statements, random seed)
cargo run --bin differential_fuzzer
 
# With specific seed for reproducibility
cargo run --bin differential_fuzzer -- --seed 12345
 
# More statements with verbose output
cargo run --bin differential_fuzzer -- -n 1000 --verbose
 
# Keep database files after run (for debugging)
cargo run --bin differential_fuzzer -- --seed 12345 --keep-files
 
# All options
cargo run --bin differential_fuzzer -- \
  --seed <SEED>           # Deterministic seed
  -n <NUM>                # Number of statements (default: 100)
  -t <NUM>                # Number of tables (default: 2)
  -c <NUM>                # Columns per table (default: 5)
  --verbose               # Print each SQL statement
  --keep-files            # Persist .db files to disk

Continuous Fuzzing (Loop Mode)

# Run forever with random seeds
cargo run --bin differential_fuzzer -- loop
 
# Run 50 iterations
cargo run --bin differential_fuzzer -- loop 50

Docker Runner (CI/Production)

# Build and run from repo root
docker build -f testing/differential-oracle/fuzzer/docker-runner/Dockerfile -t fuzzer .
docker run -e GITHUB_TOKEN=xxx -e SLACK_WEBHOOK_URL=xxx fuzzer

Environment variables for docker-runner:

• TIME_LIMIT_MINUTES - Total runtime (default: 1440 = 24h)
• PER_RUN_TIMEOUT_SECONDS - Per-run timeout (default: 1200 = 20min)
• NUM_STATEMENTS - Statements per run (default: 1000)
• LOG_TO_STDOUT - Print fuzzer output (default: false)
• GITHUB_TOKEN - For auto-filing issues
• SLACK_WEBHOOK_URL - For notifications

Output Files

All output goes to simulator-output/ directory:

Reproducing Errors

Always follow these steps

1. Find the seed and profile in the error output:

   INFO: Starting differential_fuzzer with config: SimConfig { seed: 12345, ..., weight_profile: Writes }

2. Re-run with that seed and profile (a seed only replays under the same profile):

   cargo run --bin differential_fuzzer -- --seed 12345 --profile writes --verbose --keep-files

3. Read the minimized reproduction first. On an oracle failure the fuzzer writes these files to simulator-output/:

   • minimized.sql - a shrunken state script plus the shrunken failing statement, produced automatically. Start here.
   • turso-state.sql / sqlite-state.sql - each engine's full state as a replayable script, when you need more than the minimized version kept.
   • test.sql - every executed statement (the failing one is marked -- FAILED:). The minimizer falls back to replaying this history when the failure depends on how the state was built, not just its contents.
   • schema.json - table structure at failure time.

4. Probe the reproduction with differential_probe. It runs a statement-per-line script on Turso and SQLite side by side, prints both outcomes for every statement, marks divergences, and compares the final table contents. Exit code 1 means something diverged.

   cargo run -q -p differential-fuzzer --bin differential_probe -- \
       simulator-output/minimized.sql

   Use it instead of piping SQL into the two shells: the tursodb shell cannot ATTACH ':memory:' AS aux, so fuzzer reproductions with an aux schema only run correctly through the probe. Reading from stdin also works: echo "SELECT ~X'96';" | cargo run -q -p differential-fuzzer --bin differential_probe.

5. Bisect by editing the script. Copy minimized.sql, simplify one thing at a time (replace an expression with a constant, drop a column, drop a state line), and re-run the probe after each edit. The divergence marker tells you immediately whether the edit kept the bug. This loop usually ends at a one-line kernel you can hand to EXPLAIN on both engines.

6. Create a regression test in .sqltest (preferred) or .rs from the kernel. Always load the Debugging skill for reference.

Understanding Failures

Oracle Failure Types

1. Row set mismatch - Turso returned different rows than SQLite
2. Turso errored but SQLite succeeded - Turso rejected valid SQL
3. SQLite errored but Turso succeeded - Turso accepted invalid SQL
4. Schema mismatch - Tables/columns differ after DDL

Warning (non-fatal)

• Unordered LIMIT mismatch - LIMIT without ORDER BY may return different valid rows

Key Source Files

Tracing

Set RUST_LOG for more detailed output:

RUST_LOG=debug cargo run --bin differential_fuzzer -- --seed 12345