LiveKit Agents

Overview

LiveKit Agents builds voice agents out of three swappable legs — STT, LLM, TTS — plus VAD and end-of-turn detection. Svara plugs into the TTS leg through the official Python SDK: from svara.livekit import TTS. By default it rides the input-streaming WebSocket, so the agent starts speaking while the LLM is still writing its sentence.

mic ─▶ VAD ─▶ STT ─▶ LLM ─┬▶  Svara TTS (eager WS)  ─▶ speaker
                          └── token stream, not sentences

The plugin emits 24 kHz, 16-bit, mono PCM straight into LiveKit’s AudioEmitter — no decode, no resample on the hot path — and pins the platform-certified sampling defaults so the agent sounds exactly like the API it fronts.

The TTS plugin

Two synthesis paths, chosen by mode:

• mode="eager" (default) — forwards the LLM token stream into the input-streaming WebSocket; speech starts a few words in and prosody stays continuous across the whole turn. Measured roughly 2× faster to first audio than the sentence-buffered path in the same pipeline.
• mode="http" — buffers whole sentences and synthesizes each over streaming HTTP. Used automatically for fixed-text say(), and available as the conservative fallback for full turns.

from svara.livekit import TTS

tts = TTS(voice="sv_enhdbrj5")            # Aanya; eager WS mode by default
# tts = TTS(voice="sv_enhdbrj5", mode="http")   # sentence-buffered fallback
# tts = TTS(voice="sv_enhdbrj5", language="hi", pronunciation_dictionary_id="pd_...")

Wiring the agent

Pass the plugin as tts= on the AgentSession. Every other leg is a stock plugin, and each is independently swappable — nothing below is Svara-specific.

from livekit import agents
from livekit.agents import Agent, AgentServer, AgentSession, JobContext, JobProcess
from livekit.plugins import openai, silero

from svara.livekit import TTS

server = AgentServer(setup_fnc=lambda proc: proc.userdata.__setitem__(
    "vad", silero.VAD.load()))


@server.rtc_session(agent_name="svara-agent")
async def entrypoint(ctx: JobContext) -> None:
    await ctx.connect()

    session = AgentSession(
        vad=ctx.proc.userdata["vad"],
        stt=openai.STT(model="gpt-4o-mini-transcribe"),
        llm=openai.LLM(model="gpt-4.1-mini"),
        tts=TTS(voice="sv_enhdbrj5"),      # any id from GET /v1/voices
    )
    await session.start(
        Agent(instructions="You are a helpful voice assistant. "
                           "Reply in one or two short sentences."),
        room=ctx.room,
    )
    await session.generate_reply(instructions="Greet the caller in one sentence.")


if __name__ == "__main__":
    agents.cli.run_app(server)

Run python agent.py download-files once (it fetches the VAD and turn-detector models), then python agent.py dev.

Streaming from the LLM

With mode="eager" the plugin declares streaming=True and LiveKit hands it the raw token stream; the plugin forwards each delta over one WebSocket and pushes the binary PCM frames that come back. Nothing to implement — the protocol details in Input streaming are what the plugin speaks for you.

Prefer sentence-at-a-time synthesis for a specific agent? Construct the plugin with mode="http", or flip it at runtime with tts.update_options(mode="http").

Gotchas

• Use the gateway host. https://api.kenpathlabs.com serves the API; the console host does not.
• Don’t resample. The plugin declares sample_rate=24000and LiveKit’s emitter takes the raw PCM; LiveKit itself resamples for an 8 kHz telephony leg. Anything you add on top only brings latency and artifacts.
• 429s are already mapped. The plugin converts Svara’s ElevenLabs-shaped errors into LiveKit’s APIStatusError with the right retryability, so the agent backs off on rate_limit_exceeded and too_many_concurrent_requests instead of dropping the turn. insufficient_quota is terminal until your quota resets.
• Use sv_ voice ids in code. Ids like sv_enhdbrj5 come from GET /v1/voices and are stable across renames.
• Leave the sampling knobs alone. The plugin pins the platform-certified sampling; overriding it per-agent is how long utterances start to ramble.