Speech Engine

What it does

When your agent uses it

• Building Speech Engine servers
• WebSocket handlers
• WebRTC browser clients
• Conversation token endpoints

Example prompts

• “/speech-engine”

Requirements

• Python 3
• A credential in ELEVENLABS_API_KEY
• Compatibility (from SKILL.md): Requires internet access and an ElevenLabs API key (ELEVENLABS_API_KEY).

Workflow steps

5 steps, taken from the first numbered list in SKILL.md.

1. 1The browser sends user audio to ElevenLabs.
2. 2ElevenLabs sends speech-recognition events to your server.
3. 3Your server derives trusted application state without letting raw speech text control tools or privileged actions.
4. 4Your server streams text back through the SDK.
5. 5ElevenLabs converts the response to speech and plays it in the browser.

What it can do on your machine

Read from SKILL.md and the folder at commit 1d08a4a. It shows what the files ask for, not the result of running them.

• Tool permissions

  Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.

  From allowed-tools in the SKILL.md frontmatter.

• Runs code

  Shell commands in SKILL.md call:

  • ngrok

  From the folder's file list and the shell code blocks in SKILL.md.

• Network

  Links to these hosts (documentation or services it may open):

  • elevenlabs.io

  From URLs in SKILL.md, links to its own repository left out.

• Credentials

  Names these keys or tokens, usually read from environment variables:

  • ELEVENLABS_API_KEY

  From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.

• Compatibility

  Requires internet access and an ElevenLabs API key (ELEVENLABS_API_KEY).

  From compatibility in the SKILL.md frontmatter.

Context cost

Speech Engine loads about 2.5k tokens when it runs, and up to ~5.4k if it reads all its reference files. Until then it costs about 95 tokens; SKILL.md has 897 words of instructions outside code blocks.

Always · name and description, kept in context so the agent knows when to use it

~95

When it runs · the whole SKILL.md, loaded when a task matches

~2.5k

With references · SKILL.md plus every file in references/, read only if the agent opens them

~5.4k

Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.

Safety

• WarningMentions a paste, webhook or tunnelling service often used to send data outSKILL.md:97

  WS_URL` should look like `wss://example.ngrok.app/ws` locally or your production WebSocket route in deployment.

Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.

SKILL.md

The full file from elevenlabs/skills at commit 1d08a4a, republished under its MIT licence (© elevenlabs). 897 words, ~2,507 tokens.

name

speech-engine

description

Add real-time voice conversations to a custom agent runtime with ElevenLabs Speech Engine. Use when building Speech Engine servers, WebSocket handlers, WebRTC browser clients, conversation token endpoints, interruption-aware streaming responses, or voice-enabled chat agents that connect developer-owned server logic to ElevenLabs speech-to-text and text-to-speech.

compatibility

Requires internet access and an ElevenLabs API key (ELEVENLABS_API_KEY).

license

MIT

import asyncio
import os

from dotenv import load_dotenv
from elevenlabs import AsyncElevenLabs

load_dotenv()

elevenlabs = AsyncElevenLabs(api_key=os.getenv("ELEVENLABS_API_KEY"))

async def main():
    engine = await elevenlabs.speech_engine.create(
        name="My Speech Engine",
        speech_engine={"ws_url": os.environ["PUBLIC_WS_URL"]},
        overrides={"first_message": True},
    )
    print(engine.engine_id)

asyncio.run(main())

import { ElevenLabsClient } from "@elevenlabs/elevenlabs-js";
import "dotenv/config";

const elevenlabs = new ElevenLabsClient({
  apiKey: process.env.ELEVENLABS_API_KEY,
});

const engine = await elevenlabs.speechEngine.create({
  name: "My Speech Engine",
  speechEngine: { wsUrl: process.env.PUBLIC_WS_URL! },
  overrides: { firstMessage: true },
});

console.log(engine.engineId);

engine = await elevenlabs.speech_engine.get(os.environ["ELEVENLABS_SPEECH_ENGINE_ID"])
await engine.serve(port=3001, path="/ws", debug=True, callbacks=validated_callbacks)

const engine = await elevenlabs.speechEngine.get(process.env.ELEVENLABS_SPEECH_ENGINE_ID!);
engine.attach(httpServer, "/ws", { debug: true, ...validatedCallbacks });

import express from "express";
import { ElevenLabsClient } from "@elevenlabs/elevenlabs-js";
import "dotenv/config";

const app = express();
const elevenlabs = new ElevenLabsClient();

app.get("/api/token", async (_req, res) => {
  const response = await elevenlabs.conversationalAi.conversations.getWebrtcToken({
    agentId: process.env.ELEVENLABS_SPEECH_ENGINE_ID!,
  });
  res.json({ token: response.token });
});

import { useConversation } from "@elevenlabs/react";

export function VoiceControls() {
  const conversation = useConversation({
    onConnect: () => console.log("connected"),
    onDisconnect: () => console.log("disconnected"),
    onError: (error) => console.error(error),
  });

  async function startConversation() {
    await navigator.mediaDevices.getUserMedia({ audio: true });
    const { token } = await fetch("/api/token").then((res) => res.json());

    await conversation.startSession({
      conversationToken: token,
      overrides: {
        agent: { firstMessage: "Hello! How can I help you today?" },
      },
    });
  }

  return <button onClick={startConversation}>Start conversation</button>;
}

© elevenlabs, MIT. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file

Files

SKILL.md and 3 other files (references) in speech-engine of elevenlabs/skills.

• SKILL.md
• references/installation.md
• references/javascript-sdk-reference.md
• references/python-sdk-reference.md

Open the folder on GitHubat commit 1d08a4a