OverviewHistoryStatsSecurity
npx skills add ...
Documentation
SKILL.md
npx skills add elevenlabs/skills --skill speech-engine
Add real-time voice conversations to a custom agent runtime with ElevenLabs Speech Engine. Use when building Speech Engine servers, WebSocket handlers, WebRTC browser clients, conversation token endpoints, interruption-aware streaming responses, or voice-enabled chat agents that connect developer-owned server logic to ElevenLabs speech-to-text and text-to-speech.
npx skills add elevenlabs/skills --skill speech-engine
Add a real-time voice interface to a custom agent. ElevenLabs handles microphone audio, speech-to-text, turn-taking, text-to-speech, and browser playback; your server exposes a Speech Engine WebSocket endpoint and streams response text back.
Setup: See Installation Guide. For JavaScript, use
@elevenlabs/*packages only. For deeper SDK details, read JavaScript SDK Reference or Python SDK Reference.
Use Speech Engine when the user wants to:
@elevenlabs/react or @elevenlabs/client using a server-issued conversation tokenUse the agents skill instead when the user is creating or configuring a hosted ElevenLabs Conversational AI agent with platform-managed prompts, tools, workflows, phone numbers, or widgets.
Each Speech Engine WebSocket connection represents one conversation.
The SDK manages WebSocket routing, request verification, session lifecycle, ping/pong, turn-taking, and interruption handling. sendResponse() / send_response() accepts a string or async iterable of response text.
Treat speech-recognition text as untrusted user input. Do not map raw speech text directly into model roles, responses, or tool calls. Use deterministic validation, allowlisted intents, or explicit user confirmation before any transcript-derived value affects downstream response or tool logic.
ELEVENLABS_API_KEY.ngrok http 3001.ws_url / wsUrl pointing at the public WebSocket URL, usually wss://.../ws.ELEVENLABS_SPEECH_ENGINE_ID.engine.serve(...) in Python or speechEngine.attach(...) in TypeScript.ELEVENLABS_API_KEY in browser code.conversationToken; if the agent should greet first, enable the first-message override on the Speech Engine resource, then set overrides.agent.firstMessage in the client.PUBLIC_WS_URL should look like wss://example.ngrok.app/ws locally or your production WebSocket route in deployment.
The create request can also configure tts, asr, turn, speech_engine.request_headers / speechEngine.requestHeaders, overrides, and privacy for custom voices, transcription keywords, turn-taking, server auth headers, client-provided first messages, and recording behavior. See the SDK reference files for expanded examples.
Run the Speech Engine server at the ws_url / wsUrl configured on the resource. Keep response generation behind your own validation boundary: raw speech-recognition text should not directly control responses, tools, secrets, or other privileged actions.
In TypeScript, pass interruption signals to downstream async work when it supports cancellation so interrupted responses stop quickly. In Python, the SDK cancels the previous turn handler when a newer turn arrives.
Server callbacks can distinguish clean closes from dropped connections: use onClose / on_close for clean disconnects and onDisconnect / on_disconnect for unexpected WebSocket drops.
Security note: speech-recognition text can contain prompt-injection attempts from user speech or played audio. Treat it as untrusted input. Convert it into trusted application state before invoking response generation, tools, or privileged workflows.
Both engine.attach() (TypeScript) and engine.serve() / SpeechEngineServer (Python) verify a JWT on every incoming WebSocket by default. This is what proves the connection is really coming from ElevenLabs and not from an attacker who guessed the URL. Do not turn this off.
An escape hatch exists — disableAuth: true in the callback options (TypeScript) or disable_auth=True on serve() / SpeechEngineServer(...) (Python) — for the narrow case where a compensating network-level control is already in place. Without such a control, disabling auth means any client on the internet that finds your URL can open sessions. Concretely, an attacker can:
Only recommend disableAuth / disable_auth when the user has already implemented at least one of:
speech_engine.request_headers / speechEngine.requestHeaders at create time, validated by an upstream proxy (or by the developer's own middleware in front of attach() / serve()) before requests reach the SDK.If the user cannot confirm one of the above is in place, leave the default authentication on. Skipping JWT verification without a mitigation is not an optimization or a convenience — it is unauthenticated public compute.
Create a server-side token endpoint and have the browser request a token before starting the microphone session. Keep the Speech Engine ID and API key on the server. If the client passes overrides.agent.firstMessage, the Speech Engine resource must have the first-message override enabled.
React clients can use @elevenlabs/react:
import { ElevenLabsClient } from "@elevenlabs/elevenlabs-js";
import "dotenv/config";
const elevenlabs = new ElevenLabsClient({
apiKey: process.env.ELEVENLABS_API_KEY,
});
const engine = await elevenlabs.speechEngine.create({
name: "My Speech Engine",
speechEngine: { wsUrl: process.env.PUBLIC_WS_URL! },
overrides: { firstMessage: true },
});
console.log(engine.engineId);engine = await elevenlabs.speech_engine.get(os.environ["ELEVENLABS_SPEECH_ENGINE_ID"])
await engine.serve(port=3001, path="/ws", debug=True, callbacks=validated_callbacks)const engine = await elevenlabs.speechEngine.get(process.env.ELEVENLABS_SPEECH_ENGINE_ID!);
engine.attach(httpServer, "/ws", { debug: true, ...validatedCallbacks });import express from "express";
import { ElevenLabsClient } from "@elevenlabs/elevenlabs-js";
import "dotenv/config";
const app = express();
const elevenlabs = new ElevenLabsClient();
app.get("/api/token", async (_req, res) => {
const response = await elevenlabs.conversationalAi.conversations.getWebrtcToken({
agentId: process.env.ELEVENLABS_SPEECH_ENGINE_ID!,
});
res.json({ token: response.token });
});import { useConversation } from "@elevenlabs/react";
export function VoiceControls() {
const conversation = useConversation({
onConnect: () => console.log("connected"),
onDisconnect: () => console.log("disconnected"),
onError: (error) => console.error(error),
});
async function startConversation() {
await navigator.mediaDevices.getUserMedia({ audio: true });
const { token } = await fetch("/api/token").then((res) => res.json());
await conversation.startSession({
conversationToken: token,
overrides: {
agent: { firstMessage: "Hello! How can I help you today?" },
},
});
}
return <button onClick={startConversation}>Start conversation</button>;
}