npx skills add ...
npx skills add openai/skills --skill playwright-interactive
npx skills add openai/skills --skill playwright-interactive
Persistent browser and Electron interaction through `js_repl` for fast iterative UI debugging.
Use a persistent js_repl Playwright session to debug local web or Electron apps, keep the same handles alive across iterations, and run functional plus visual QA without restarting the whole toolchain unless the process ownership changed.
js_repl must be enabled for this skill.js_repl is missing, enable it in ~/.codex/config.toml:--enable js_repl (equivalent to -c features.js_repl=true).js_repl, start a new Codex session so the tool list refreshes.--sandbox danger-full-access (or the equivalent config for sandbox_mode=danger-full-access). This is a temporary requirement while js_repl + Playwright support inside the sandbox is still being completed.js_repl_reset as a recovery tool, not routine cleanup. Resetting the kernel destroys your Playwright handles.If you switch to a different workspace later, repeat setup there.
Binding rules:
var for the shared top-level Playwright handles because later js_repl cells reuse them.undefined and rerun the cell instead of adding recovery logic everywhere.page, mobilePage, appWindow) over repeatedly rediscovering pages from the context.Shared web helpers:
For web apps, use an explicit viewport by default and treat native-window mode as a separate validation pass.
deviceScaleFactor rather than switching straight to native-window mode.viewport: null) for a separate headed pass when you need to validate launched window size, OS-level DPI behavior, browser chrome interactions, or bugs that may depend on the host display configuration.noDefaultViewport, so treat it like a real desktop window and check the as-launched size and layout before resizing anything.context for a native-window pass or vice versa; close the old page and context, then create a new one for the new mode.Desktop and mobile web sessions share the same browser, helpers, and QA flow. The main difference is which context and page pair you create.
Set TARGET_URL to the app you are debugging. For local servers, prefer 127.0.0.1 over localhost.
If context or page is stale, set context = page = undefined and rerun the cell.
Reuse TARGET_URL when it already exists; otherwise set a mobile target directly.
If mobileContext or mobilePage is stale, set mobileContext = mobilePage = undefined and rerun the cell.
Set ELECTRON_ENTRY to . when the current workspace is the Electron app and package.json points main to the right entry file. If you need to target a specific main-process file directly, use a path such as ./main.js instead.
If js_repl is not already running from the Electron app workspace, pass cwd explicitly when launching.
If the app process looks stale, set electronApp = appWindow = undefined and rerun the cell.
If you already have an Electron session but need a fresh process after a main-process, preload, or startup change, use the restart cell in the next section instead of rerunning this one.
Keep the same session alive whenever you can.
Web renderer reload:
Electron renderer-only reload:
Electron restart after main-process, preload, or startup changes:
If your launch requires an explicit cwd, include the same cwd here.
Default posture:
js_repl cell short and focused on one interaction burst.browser, context, page, electronApp, appWindow) instead of redeclaring them.electronApp.evaluate(...) only for main-process inspection or purpose-built diagnostics.js_repl once, then keep the same Playwright handles alive across iterations.page.evaluate(...) and electronApp.evaluate(...) may inspect or stage state, but they do not count as signoff input.If you plan to emit a screenshot through codex.emitImage(...), use the CSS-normalized paths in the next section by default. Those are the canonical examples for screenshots that will be interpreted by the model or used for coordinate-based follow-up actions. Keep raw captures as an exception for fidelity-sensitive debugging only; the raw exception examples appear after the normalization guidance.
If you will emit a screenshot with codex.emitImage(...) for model interpretation, normalize it to CSS pixels for the exact region you captured before emitting. This keeps returned coordinates aligned with Playwright CSS pixels if the reply is later used for clicking, and it also reduces image payload size and model token cost.
Do not emit raw native-window screenshots by default. Skip normalization only when you explicitly need device-pixel fidelity, such as Retina or DPI artifact debugging, pixel-accurate rendering inspection, or another fidelity-sensitive case where raw pixels matter more than payload size. For local-only inspection that will not be emitted to the model, raw capture is fine.
Do not assume page.screenshot({ scale: "css" }) is enough in native-window mode (viewport: null). In Chromium on macOS Retina displays, headed native-window screenshots can still come back at device-pixel size even when scale: "css" is requested. The same caveat applies to Electron windows launched through Playwright because Electron runs with noDefaultViewport, and appWindow.screenshot({ scale: "css" }) may still return device-pixel output.
Use separate normalization paths for web pages and Electron windows:
page.screenshot({ scale: "css" }) directly. If native-window Chromium still returns device-pixel output, resize inside the current page with canvas; no scratch page is required.appWindow.context().newPage() or electronApp.context().newPage() as a scratch page. Electron contexts do not support that path reliably. Capture in the main process with BrowserWindow.capturePage(...), resize with nativeImage.resize(...), and emit those bytes directly.Shared helpers and conventions:
page or mobilePage for web, or appWindow for Electron, as the surface.clip as CSS pixels from getBoundingClientRect() in the renderer.quality: 85 unless lossless fidelity is specifically required.{ x, y } directly.Preferred web path for explicit-viewport contexts, and often for web in general:
Mobile web uses the same path; substitute mobilePage for page:
If the model returns { x, y }, click it directly:
Mobile web click path:
For web clip screenshots or element screenshots in this normal path, scale: "css" usually works directly. Add the region origin back when clicking.
await emitWebJpeg(page, { clip })await emitWebJpeg(mobilePage, { clip })await clickCssPoint({ surface: page, clip, x, y })await tapCssPoint({ page: mobilePage, clip, x, y })await clickCssPoint({ surface: page, clip: box, x, y }) after const box = await locator.boundingBox()Web native-window fallback when scale: "css" still comes back at device-pixel size:
For a full viewport fallback capture, treat returned { x, y } as direct CSS coordinates:
For a clipped fallback capture, add the clip origin back:
For Electron, normalize in the main process instead of opening a scratch Playwright page. The helper below returns CSS-scaled bytes for the full content area or for a clipped CSS-pixel region. Treat clip as content-area CSS pixels, for example values taken from getBoundingClientRect() in the renderer.
Full Electron window:
Clipped Electron region using CSS pixels from the renderer:
Use these only when raw pixels matter more than CSS-coordinate alignment, such as Retina or DPI artifact debugging, pixel-accurate rendering inspection, or other fidelity-sensitive review.
Web desktop raw emit:
Electron raw emit:
Mobile raw emit after the mobile web context is already running:
Do not assume a screenshot is acceptable just because the main widget is visible. Before signoff, explicitly verify that the intended initial view matches the product requirement, using both screenshot review and numeric checks.
Web or renderer check:
Electron check:
Augment the numeric check with getBoundingClientRect() checks for the required visible regions in your specific UI when clipping is a realistic failure mode; document-level metrics alone are not sufficient for fixed shells.
For local web debugging, keep the app running in a persistent TTY session. Do not rely on one-shot background commands from a short-lived shell.
Use the project's normal start command, for example:
Before page.goto(...), verify the chosen port is listening and the app responds.
For Electron debugging, launch the app from js_repl through _electron.launch(...) so the same session owns the process. If the Electron renderer depends on a separate dev server (for example Vite or Next), keep that server running in a persistent TTY session and then relaunch or reload the Electron app from js_repl.
Only run cleanup when the task is actually finished:
js_repl session does not implicitly run electronApp.close(), context.close(), or browser.close().If you plan to exit Codex immediately after debugging, run the cleanup cell first and wait for the "Playwright session closed" log before quitting.
Cannot find module 'playwright': run the one-time setup in the current workspace and verify the import before using js_repl.npx playwright install chromium.page.goto: net::ERR_CONNECTION_REFUSED: make sure the dev server is still running in a persistent TTY session, recheck the port, and prefer http://127.0.0.1:<port>.electron.launch hangs, times out, or exits immediately: verify the local electron dependency, confirm the args target, and make sure any renderer dev server is already running before launch.Identifier has already been declared: reuse the existing top-level bindings, choose a new name, or wrap the code in { ... }. Use js_repl_reset only when the kernel is genuinely stuck.browserContext.newPage: Protocol error (Target.createTarget): Not supported while working with Electron: do not use appWindow.context().newPage() or electronApp.context().newPage() as a scratch page; use the Electron-specific screenshot normalization flow in the model-bound screenshots section.js_repl timed out or reset: rerun the bootstrap cell and recreate the session with shorter, more focused cells.--sandbox danger-full-access and restart that way if needed.var chromium;
var electronLauncher;
var browser;
var context;
var page;
var mobileContext;
var mobilePage;
var electronApp;
var appWindow;
try {
({ chromium, _electron: electronLauncher } = await import("playwright"));
console.log("Playwright loaded");
} catch (error) {
throw new Error(
`Could not load playwright from the current js_repl cwd. Run the setup commands from this workspace first. Original error: ${error}`
);
}var resetWebHandles = function () {
context = undefined;
page = undefined;
mobileContext = undefined;
mobilePage = undefined;
};
var ensureWebBrowser = async function () {
if (browser && !browser.isConnected()) {
browser = undefined;
resetWebHandles();
}
browser ??= await chromium.launch({ headless: false });
return browser;
};
var reloadWebContexts = async function () {
for (const currentContext of [context, mobileContext]) {
if (!currentContext) continue;
for (const p of currentContext.pages()) {
await p.reload({ waitUntil: "domcontentloaded" });
}
}
console.log("Reloaded existing web tabs");
};var TARGET_URL = "http://127.0.0.1:3000";
if (page?.isClosed()) page = undefined;
await ensureWebBrowser();
context ??= await browser.newContext({
viewport: { width: 1600, height: 900 },
});
page ??= await context.newPage();
await page.goto(TARGET_URL, { waitUntil: "domcontentloaded" });
console.log("Loaded:", await page.title());var MOBILE_TARGET_URL = typeof TARGET_URL === "string"
? TARGET_URL
: "http://127.0.0.1:3000";
if (mobilePage?.isClosed()) mobilePage = undefined;
await ensureWebBrowser();
mobileContext ??= await browser.newContext({
viewport: { width: 390, height: 844 },
isMobile: true,
hasTouch: true,
});
mobilePage ??= await mobileContext.newPage();
await mobilePage.goto(MOBILE_TARGET_URL, { waitUntil: "domcontentloaded" });
console.log("Loaded mobile:", await mobilePage.title());var TARGET_URL = "http://127.0.0.1:3000";
await ensureWebBrowser();
await page?.close().catch(() => {});
await context?.close().catch(() => {});
page = undefined;
context = undefined;
browser ??= await chromium.launch({ headless: false });
context = await browser.newContext({ viewport: null });
page = await context.newPage();
await page.goto(TARGET_URL, { waitUntil: "domcontentloaded" });
console.log("Loaded native window:", await page.title());var ELECTRON_ENTRY = ".";
if (appWindow?.isClosed()) appWindow = undefined;
if (!appWindow && electronApp) {
await electronApp.close().catch(() => {});
electronApp = undefined;
}
electronApp ??= await electronLauncher.launch({
args: [ELECTRON_ENTRY],
});
appWindow ??= await electronApp.firstWindow();
console.log("Loaded Electron window:", await appWindow.title());await reloadWebContexts();await appWindow.reload({ waitUntil: "domcontentloaded" });
console.log("Reloaded Electron window");await electronApp.close().catch(() => {});
electronApp = undefined;
appWindow = undefined;
electronApp = await electronLauncher.launch({
args: [ELECTRON_ENTRY],
});
appWindow = await electronApp.firstWindow();
console.log("Relaunched Electron window:", await appWindow.title());var emitJpeg = async function (bytes) {
await codex.emitImage({
bytes,
mimeType: "image/jpeg",
detail: "original",
});
};
var emitWebJpeg = async function (surface, options = {}) {
await emitJpeg(await surface.screenshot({
type: "jpeg",
quality: 85,
scale: "css",
...options,
}));
};
var clickCssPoint = async function ({ surface, x, y, clip }) {
await surface.mouse.click(
clip ? clip.x + x : x,
clip ? clip.y + y : y
);
};
var tapCssPoint = async function ({ page, x, y, clip }) {
await page.touchscreen.tap(
clip ? clip.x + x : x,
clip ? clip.y + y : y
);
};await emitWebJpeg(page);await emitWebJpeg(mobilePage);await clickCssPoint({ surface: page, x, y });await tapCssPoint({ page: mobilePage, x, y });var emitWebScreenshotCssScaled = async function ({ page, clip, quality = 0.85 } = {}) {
var NodeBuffer = (await import("node:buffer")).Buffer;
const target = clip
? { width: clip.width, height: clip.height }
: await page.evaluate(() => ({
width: window.innerWidth,
height: window.innerHeight,
}));
const screenshotBuffer = await page.screenshot({
type: "png",
...(clip ? { clip } : {}),
});
const bytes = await page.evaluate(
async ({ imageBase64, targetWidth, targetHeight, quality }) => {
const image = new Image();
image.src = `data:image/png;base64,${imageBase64}`;
await image.decode();
const canvas = document.createElement("canvas");
canvas.width = targetWidth;
canvas.height = targetHeight;
const ctx = canvas.getContext("2d");
ctx.imageSmoothingEnabled = true;
ctx.drawImage(image, 0, 0, targetWidth, targetHeight);
const blob = await new Promise((resolve) =>
canvas.toBlob(resolve, "image/jpeg", quality)
);
return new Uint8Array(await blob.arrayBuffer());
},
{
imageBase64: NodeBuffer.from(screenshotBuffer).toString("base64"),
targetWidth: target.width,
targetHeight: target.height,
quality,
}
);
await emitJpeg(bytes);
};await emitWebScreenshotCssScaled({ page });
await clickCssPoint({ surface: page, x, y });await emitWebScreenshotCssScaled({ page, clip });
await clickCssPoint({ surface: page, clip, x, y });var emitElectronScreenshotCssScaled = async function ({ electronApp, clip, quality = 85 } = {}) {
const bytes = await electronApp.evaluate(async ({ BrowserWindow }, { clip, quality }) => {
const win = BrowserWindow.getAllWindows()[0];
const image = clip ? await win.capturePage(clip) : await win.capturePage();
const target = clip
? { width: clip.width, height: clip.height }
: (() => {
const [width, height] = win.getContentSize();
return { width, height };
})();
const resized = image.resize({
width: target.width,
height: target.height,
quality: "best",
});
return resized.toJPEG(quality);
}, { clip, quality });
await emitJpeg(bytes);
};await emitElectronScreenshotCssScaled({ electronApp });
await clickCssPoint({ surface: appWindow, x, y });var clip = await appWindow.evaluate(() => {
const rect = document.getElementById("board").getBoundingClientRect();
return {
x: Math.round(rect.x),
y: Math.round(rect.y),
width: Math.round(rect.width),
height: Math.round(rect.height),
};
});
await emitElectronScreenshotCssScaled({ electronApp, clip });
await clickCssPoint({ surface: appWindow, clip, x, y });await codex.emitImage({
bytes: await page.screenshot({ type: "jpeg", quality: 85 }),
mimeType: "image/jpeg",
detail: "original",
});await codex.emitImage({
bytes: await appWindow.screenshot({ type: "jpeg", quality: 85 }),
mimeType: "image/jpeg",
detail: "original",
});await codex.emitImage({
bytes: await mobilePage.screenshot({ type: "jpeg", quality: 85 }),
mimeType: "image/jpeg",
detail: "original",
});console.log(await page.evaluate(() => ({
innerWidth: window.innerWidth,
innerHeight: window.innerHeight,
clientWidth: document.documentElement.clientWidth,
clientHeight: document.documentElement.clientHeight,
scrollWidth: document.documentElement.scrollWidth,
scrollHeight: document.documentElement.scrollHeight,
canScrollX: document.documentElement.scrollWidth > document.documentElement.clientWidth,
canScrollY: document.documentElement.scrollHeight > document.documentElement.clientHeight,
})));console.log(await appWindow.evaluate(() => ({
innerWidth: window.innerWidth,
innerHeight: window.innerHeight,
clientWidth: document.documentElement.clientWidth,
clientHeight: document.documentElement.clientHeight,
scrollWidth: document.documentElement.scrollWidth,
scrollHeight: document.documentElement.scrollHeight,
canScrollX: document.documentElement.scrollWidth > document.documentElement.clientWidth,
canScrollY: document.documentElement.scrollHeight > document.documentElement.clientHeight,
})));npm startif (electronApp) {
await electronApp.close().catch(() => {});
}
if (mobileContext) {
await mobileContext.close().catch(() => {});
}
if (context) {
await context.close().catch(() => {});
}
if (browser) {
await browser.close().catch(() => {});
}
browser = undefined;
context = undefined;
page = undefined;
mobileContext = undefined;
mobilePage = undefined;
electronApp = undefined;
appWindow = undefined;
console.log("Playwright session closed");