npx skills add ...
npx skills add github/awesome-copilot --skill automate-this
npx skills add github/awesome-copilot --skill automate-this
Analyze a screen recording of a manual process and produce targeted, working automation scripts. Extracts frames and audio narration from video files, reconstructs the step-by-step workflow, and proposes automation at multiple complexity levels using tools already installed on the user machine.
Analyze a screen recording of a manual process and build working automation for it.
The user records themselves doing something repetitive or tedious, hands you the video file, and you figure out what they're doing, why, and how to script it away.
Before analyzing any recording, verify the required tools are available. Run these checks silently and only surface problems:
brew install ffmpeg (macOS) or the equivalent for their OS.pip install openai-whisper or brew install whisper-cpp. If the user declines, proceed with visual analysis only.Given a video file path (typically on ~/Desktop/), extract both visual frames and audio:
Extract frames at one frame every 2 seconds. This balances coverage with context window limits.
Use $WORK_DIR for all subsequent temp file paths in the session. The per-run directory with mode 0700 ensures extracted frames are only readable by the current user.
If the recording is longer than 5 minutes (more than 150 frames), increase the interval to one frame every 4 seconds to stay within context limits. Tell the user you're sampling less frequently for longer recordings.
Check if the video has an audio track:
If audio exists:
If neither whisper binary is available and the recording has audio, inform the user they're missing narration context and ask if they want to install Whisper (pip install openai-whisper or brew install whisper-cpp) or proceed with visual-only analysis.
Analyze the extracted frames (and transcript, if available) to build a structured understanding of what the user did. Work through the frames sequentially and identify:
Present this reconstruction to the user as a numbered step list and ask them to confirm it's accurate before proposing automation. This is critical — a wrong understanding leads to useless automation.
Format:
Do NOT proceed to Phase 3 until the user confirms the reconstruction is accurate.
Before proposing automation, understand what the user actually has to work with. Run these checks:
Use this to constrain proposals to tools the user already has. Never propose automation that requires installing five new things unless the simpler path genuinely doesn't work.
Based on the reconstructed process and the user's environment, propose automation at up to three tiers. Not every process needs three tiers — use judgment.
Tier 1 — Quick Win (under 5 minutes to set up) The smallest useful automation. A shell alias, a one-liner, a keyboard shortcut, an AppleScript snippet. Automates the single most painful step, not the whole process.
Tier 2 — Script (under 30 minutes to set up) A standalone script (bash, Python, or Node — whichever the user has) that automates the full process end-to-end. Handles common errors. Can be run manually when needed.
Tier 3 — Full Automation (under 2 hours to set up) The script from Tier 2, plus: scheduled execution (cron, launchd, or GitHub Actions), logging, error notifications, and any necessary integration scaffolding (API keys, auth tokens, etc.).
For each tier, provide:
Use these strategies based on which applications appear in the recording:
Browser-based workflows:
curl or wget for simple HTTP requests with known endpoints.Spreadsheet and data workflows:
csvkit for quick command-line CSV manipulation without writing code.Email workflows:
osascript can control Mail.app to send emails with attachments.smtplib for sending, imaplib for reading.File management workflows:
find + xargs for batch operations.fswatch or watchman for triggered-on-change automation.Terminal/CLI workflows:
macOS-specific workflows:
automator for file-based workflows.launchd plist files for scheduled tasks (prefer over cron on macOS).Cross-application workflows (data moves between apps):
Apply these principles to every proposal:
Automate the bottleneck first. The narration and timing in the recording reveal which step is actually painful. A 30-second automation of the worst step beats a 2-hour automation of the whole process.
Match the user's skill level. If the recording shows someone comfortable in a terminal, propose shell scripts. If it shows someone navigating GUIs, propose something with a simple trigger (double-click a script, run a Shortcut, or type one command).
Estimate real time savings. Count the recording duration and multiply by how often they do it. "This recording is 4 minutes. You said you do this daily. That's 17 hours per year. Tier 1 cuts it to 30 seconds each time — you get 16 hours back."
Handle the 80% case. The first version of the automation should cover the common path perfectly. Edge cases can be handled in Tier 3 or flagged for manual intervention.
Preserve human checkpoints. If the recording shows the user reviewing or approving something mid-process, keep that as a manual step. Don't automate judgment calls.
Propose dry runs. Every script should have a mode where it shows what it would do without doing it. --dry-run flags, preview output, or confirmation prompts before destructive actions.
Account for auth and secrets. If the process involves logging in or using credentials, never hardcode them. Use environment variables, keychain access (macOS security command), or prompt for them at runtime.
Consider failure modes. What happens if the website is down? If the file doesn't exist? If the format changes? Good proposals mention this and handle it.
When the user picks a tier:
~/Desktop/ otherwise).After analysis is complete (regardless of outcome), clean up extracted frames and audio:
Tell the user you're cleaning up temporary files so they know nothing is left behind.
WORK_DIR=$(mktemp -d "${TMPDIR:-/tmp}/automate-this-XXXXXX")
chmod 700 "$WORK_DIR"
mkdir -p "$WORK_DIR/frames"
ffmpeg -y -i "<VIDEO_PATH>" -vf "fps=0.5" -q:v 2 -loglevel warning "$WORK_DIR/frames/frame_%04d.jpg"
ls "$WORK_DIR/frames/" | wc -lffprobe -i "<VIDEO_PATH>" -show_streams -select_streams a -loglevel error | head -5ffmpeg -y -i "<VIDEO_PATH>" -ac 1 -ar 16000 -loglevel warning "$WORK_DIR/audio.wav"
# Use whichever whisper binary is available
if command -v whisper >/dev/null 2>&1; then
whisper "$WORK_DIR/audio.wav" --model small --language en --output_format txt --output_dir "$WORK_DIR/"
cat "$WORK_DIR/audio.txt"
elif command -v whisper-cpp >/dev/null 2>&1; then
whisper-cpp -m "$(brew --prefix 2>/dev/null)/share/whisper-cpp/models/ggml-small.bin" -l en -f "$WORK_DIR/audio.wav" -otxt -of "$WORK_DIR/audio"
cat "$WORK_DIR/audio.txt"
else
echo "NO_WHISPER"
fiHere's what I see you doing in this recording:
1. Open Chrome and navigate to [specific URL]
2. Log in with credentials
3. Click through to the reporting dashboard
4. Download a CSV export
5. Open the CSV in Excel
6. Filter rows where column B is "pending"
7. Copy those rows into a new spreadsheet
8. Email the new spreadsheet to [recipient]
You repeated steps 3-8 three times for different report types.
[If narration was present]: You mentioned that the export step is the slowest
part and that you do this every Monday morning.
Does this match what you were doing? Anything I got wrong or missed?echo "=== OS ===" && uname -a
echo "=== Shell ===" && echo $SHELL
echo "=== Python ===" && { command -v python3 && python3 --version 2>&1; } || echo "not installed"
echo "=== Node ===" && { command -v node && node --version 2>&1; } || echo "not installed"
echo "=== Homebrew ===" && { command -v brew && echo "installed"; } || echo "not installed"
echo "=== Common Tools ===" && for cmd in curl jq playwright selenium osascript automator crontab; do command -v $cmd >/dev/null 2>&1 && echo "$cmd: yes" || echo "$cmd## Tier [N]: [Name]
**What it automates:** [Which steps from the reconstruction]
**What stays manual:** [Which steps still need a human]
**Time savings:** [Estimated time saved per run, based on the recording length and repetition count]
**Prerequisites:** [Anything needed that isn't already installed — ideally nothing]
**How it works:**
[2-3 sentence plain-English explanation]
**The code:**
[Complete, working, commented code — not pseudocode]
**How to test it:**
[Exact steps to verify it works, starting with a dry run if possible]
**How to undo:**
[How to reverse any changes if something goes wrong]rm -rf "$WORK_DIR"