OverviewHistoryStatsSecurity
npx skills add ...
Documentation
SKILL.md
npx skills add dagster-io/fake-driven-testing --skill fdt-refactor-mock-to-fake
Refactor tests that use unittest.mock.patch or MagicMock into erk's gateway-based fake pattern. Use when tests import unittest.mock, use @patch decorators, or directly call mock.patch() as context managers. Essential when test_*.py files patch module-level attributes like subprocess.run, shutil.which, os.environ, or other system calls. Covers both making source code injectable AND rewriting tests.
npx skills add dagster-io/fake-driven-testing --skill fdt-refactor-mock-to-fake
Remove unittest.mock.patch from tests by making source code inject gateway
dependencies, then configuring pre-canned fakes in tests.
Use this skill when: A test imports from unittest.mock import patch or uses
@patch(...) decorators.
Key principle: Don't stop at the lowest-level matching gateway. Look for a higher-level abstraction that covers ALL the things being mocked together.
Read the test file. For each patch(...) call, record:
| Mock target (fully qualified) | System boundary (tool) | What it simulates | Return value configured |
|---|---|---|---|
erk.core.fast_llm.shutil.which | Claude CLI | CLI availability check | None or "/usr/bin/claude" |
erk.core.fast_llm.subprocess.run | Claude CLI | CLI execution result | CompletedProcess(returncode=0, stdout="...") |
The system boundary column identifies which gateway should own this mock. Multiple mocks with the same system boundary should be covered by a single gateway.
Group mocks by test: A single test patching 2-3 things together suggests those things form a unit that should be covered by one injectable gateway.
For each mock group, find the right gateway. Do not stop at the first match.
Rule:
subprocess.runis never the right gateway boundary. The gateway should be named after the tool being invoked (e.g.,GhCli,CmuxGateway,GitGateway), not after the underlying mechanism (subprocess,shell). A gateway that just wrapssubprocess.runis no better than mockingsubprocess.rundirectly — it skips the meaningful abstraction layer.
Ask: what is the test actually testing? Not "what function is being patched" but "what behavior is under test?" Think in terms of the tool or service being interacted with, not the Python function being called.
Examples:
shutil.which("claude") -> "is the Claude CLI installed?" (tool: Claude CLI)subprocess.run(["claude", "--print", ...]) -> "run a prompt via Claude CLI" (tool: Claude CLI)PromptExecutor (Claude-specific),
not Shell (subprocess-generic)Before broad exploration, do a quick targeted search using the tool names from Phase 1:
If zero hits for a tool → no gateway exists for it. You'll need to create one (see Phase 5). If hits → read the matching gateway to assess coverage.
This takes seconds and immediately tells you whether you're in "reuse" or "create" territory.
Before creating anything new, check if existing gateways already cover some of your
mock targets. A test mocking subprocess.run(["gh", "pr", "view", ...]) is already
covered by LocalGitHub.get_pr() — no new gateway needed for that mock.
For each system boundary from Phase 1, ask:
This often eliminates half the mocks immediately, reducing the scope of new work.
Search from highest to lowest. A higher-level gateway is preferable because it covers multiple low-level calls as a unit.
Priority order when multiple gateways match:
PromptExecutor.execute_prompt
rather than Shell.get_installed_tool_path)Erk gateway locations:
packages/erk-shared/src/erk_shared/gateway/*/abc.py -- gateway ABCspackages/erk-shared/src/erk_shared/gateway/*/fake.py -- matching fakespackages/erk-shared/src/erk_shared/core/fakes.py -- fakes for service ABCs
(FakePromptExecutor, FakeLlmCaller, FakeScriptWriter, etc.)tests/fakes/ -- erk-specific fakesRead the fake's __init__ signature. Check:
calls, prompt_calls, etc.)?is_available() return what you need?If no fake exists at the right level, you'll need to create one (see Phase 5).
After Phase 2, you're in one of two paths:
Path A: Existing gateway covers all mocks
Path B: No gateway exists for one or more system boundaries
gateway-abc-implementation doc (docs/learned/architecture/gateway-abc-implementation.md)Identify what source code needs to change.
Read the source file being patched (e.g., erk.core.fast_llm -> src/erk/core/fast_llm.py).
Find the class or function that directly calls the mocked thing.
Find where the class is instantiated in production (usually src/erk/core/context.py).
Plan what real implementation to wire in:
FallbackPromptExecutor -> ClaudeCliPromptExecutor(console=None)Shell -> RealShell()Add the gateway as a constructor parameter. Follow erk's conventions:
def __init__(self, *, gateway: GatewayABC))self._gatewayReplace direct system calls with gateway method calls:
Map gateway return types to the source function's return types. If the gateway
returns PromptResult(success, output, error) but the function returns
LlmResponse | LlmCallFailed, add the mapping:
Update production wiring in context.py:
For Click commands (especially exec scripts), use Click's context system instead of constructor injection:
Replace direct system calls with gateway method calls obtained from context.
See src/erk/cli/commands/exec/scripts/AGENTS.md for the full pattern.
Load docs/learned/architecture/gateway-abc-implementation.md for full patterns.
Most new gateways for mock-to-fake refactoring use the 3-file pattern.
packages/erk-shared/src/erk_shared/gateway/<tool_name>/__init__.py — emptyabc.py — ABC with abstract methods named after tool operations (not subprocess)real.py — Production implementation using subprocess calls to the toolfake.py — Constructor-injected test data, NamedTuple call tracking, read-only properties<tool>: <ToolABC> field to ErkContext dataclass
(packages/erk-shared/src/erk_shared/context/context.py)<tool> parameter to ErkContext.for_test() with default Fake<Tool>(...)Real<Tool>() in production context factory
(src/erk/core/context.py, near other Real* instantiations)RuntimeErrorFor each test that used patch:
from unittest.mock import patch (and any from subprocess import CompletedProcess)mock_run.assert_called_once()) with fake property checks
(assert len(executor.prompt_calls) == 1)Pattern:
Note: monkeypatch.delenv is a pytest fixture, not mock.patch -- it's fine to keep.
Run the affected test file:
Then lint and type-check the modified source files.
Pitfall 1: Matching the wrong gateway level
If shutil.which is mocked, the obvious match is Shell.get_installed_tool_path().
But if subprocess.run is also mocked in the same test, the real abstraction is
something that covers BOTH -- often PromptExecutor or a similar higher-level gateway.
Pitfall 2: monkeypatch.delenv vs mock.patch
monkeypatch.delenv("ANTHROPIC_API_KEY") is a pytest builtin, not mock.patch.
Keep it -- it's acceptable and doesn't need replacement.
Pitfall 3: Forgetting to update production wiring
After adding a constructor parameter, always update context.py (or wherever the
class is instantiated). The type checker will catch this but only if you run it.
Pitfall 4: One test, multiple patch contexts
Multiple patch() calls in one test is a red flag that something needs to be at a
higher abstraction level. A single fake should replace all of them.
Pitfall 5: Creating a subprocess-level gateway
If you find yourself designing a gateway called ShellRunner, SubprocessGateway, or
CommandRunner, stop. That's still mocking at the wrong level. The gateway must be
specific to the tool being called:
subprocess.run(["gh", ...]) → LocalGitHub or a GhCli gatewaysubprocess.run(["cmux", ...]) → CmuxGatewaysubprocess.run(["claude", ...]) → PromptExecutorsubprocess.run(["git", ...]) → Git gatewayName the gateway after what it represents, not how it executes._
# Before:
if shutil.which("claude") is None: ...
result = subprocess.run(["claude", "--print", ...])
# After:
if not self._prompt_executor.is_available(): ...
result = self._prompt_executor.execute_prompt(prompt, model=..., ...)if not result.success:
return LlmCallFailed(message=f"CLI failed: {result.error}")
return LlmResponse(text=result.output)from erk.core.prompt_executor import ClaudeCliPromptExecutor
MyClass(prompt_executor=ClaudeCliPromptExecutor(console=None))@click.command(name="my-command")
@click.pass_context
def my_command(ctx: click.Context, ...) -> None:
github = require_github(ctx) # existing gateway from context
cmux = require_context(ctx).cmux # new gateway from context# Before:
def test_falls_back_to_cli(monkeypatch):
monkeypatch.delenv("ANTHROPIC_API_KEY", raising=False)
fake_result = CompletedProcess(args=[], returncode=0, stdout="my-slug\n", stderr="")
with (
patch("erk.core.fast_llm.shutil.which", return_value="/usr/bin/claude"),
patch("erk.core.fast_llm.subprocess.run", return_value=fake_result) as mock_run,
):
result = AnthropicLlmCaller().call("test prompt", system_prompt="sys", max_tokens=50)
assert isinstance(result, LlmResponse)
mock_run.assert_called_once()
# After:
def test_falls_back_to_cli(monkeypatch):
monkeypatch.delenv("ANTHROPIC_API_KEY", raising=False)
executor = FakePromptExecutor(
prompt_results=[PromptResult(success=True, output="my-slug", error=None)]
)
caller = AnthropicLlmCaller(prompt_executor=executor)
result = caller.call("test prompt", system_prompt="sys", max_tokens=50)
assert isinstance(result, LlmResponse)
assert result.text == "my-slug"
assert len(executor.prompt_calls) == 1
assert executor.prompt_calls[0].prompt == "test prompt"uv run pytest <test_file> -v