Voice Mode Plugin

The Voice Mode plugin adds Hands-free listening to the web UI's voice panel. You talk; an intent mediator waits until you have finished a request, then starts a normal agent turn; the reply is spoken back while captions stay on screen.

Every voice feature is reached through the one mic button on the chat input (design: doc/core-dev/design/voice-panel.md):

Hold the mic to dictate a single message — push to talk. This needs only the speech: service, not this plugin.
Tap the mic to open the Voice panel. There you choose who controls the microphone (Push to talk or Hands-free) and how replies come back (On screen, Read aloud via the Read Aloud plugin, or Live voice). The panel can shrink to a small card so the chat stays visible.
Telegram voice notes — voice in/out inside Telegram, not the web UI.

All of those share the top-level speech: service (STT/TTS). Voice Mode is what makes Hands-free selectable.

Hands-free can run on two engines, chosen with the panel's Replies switch:

Read aloud (pipeline, the default) — speech recognition, then the intent mediator, then speech synthesis, all through the speech: service. You hear the agent's answer word for word.
Live voice (live) — a realtime speech-to-speech model (OpenAI gpt-realtime-2.1) listens and talks directly. It answers in well under a second, follows natural turn-taking, and you can interrupt it mid-sentence. It still passes every real request to the agent — it never answers about your code itself.

Only engines the server has ready are offered; the choice is remembered per browser.

Using it

The mic button appears on the chat input when speech is configured. Hands-free is selectable in the panel when this plugin is enabled. It needs an open conversation.

Tap the mic. The Voice panel opens in Push to talk; switch the header to Hands-free. The browser will ask for microphone access the first time.
Status cycles through Listening…, Thinking…, and Speaking…. The hint is "Just start talking — I'm listening."
Speak a request in your own language. Short pauses are fine; a longer pause (or a clearly finished sentence) is what sends it. Filler ("mhm", "okay") is ignored.
While the assistant is working you can: - Steer — add or refine the same task ("also cover the async path"). Work so far is kept. - Redirect — change the goal. The running turn is cancelled and replaced. - Interrupt — say stop / wait / forget it. Work is cancelled and nothing new starts.
If a tool needs approval, the panel shows Approve / Deny (and pops back to full screen if it was shrunk). Spoken yes/no is off unless the operator enables voice_approvals (see below).
Click End to leave. The conversation transcript still has the turns that were dispatched.

The first utterances of a session may wait a little longer so the speech service can detect the language. After that, short phrases ("stop", "igen") inherit it.

Keep the tab visible: the panel holds a screen wake lock, and the mic loop pauses in a background tab. Use HTTPS in production — browsers only expose the microphone on a secure origin (localhost is allowed).

With Live voice the steps are the same, but there is no language warm-up, and the voice you hear is the realtime model's. It tells you about the agent's finished answer rather than reading it out; the full answer is on screen. A live call ends after live.max_session_s (default 30 minutes) — click Try again to start a new one.

If the mic is missing, speech is not configured. If Hands-free is greyed out, the plugin is disabled; if the panel says voice conversation isn't available, the same is true at runtime (for example the API key no longer resolves).

Configuration

Enable the plugin and the shared speech service. The speech api_key must resolve through an environment variable (the same constraint as Read Aloud): a per-user ${secret:…} cannot resolve for these out-of-turn web calls.

speech:
  enabled: true
  base_url: "https://api.openai.com/v1"
  api_key: "${OPENAI_API_KEY}"
  tts_model: "tts-1"
  tts_voice: "alloy"

plugins:
  - module: codumentor.plugins.voice_mode
    class: VoiceModePlugin
    args:
      enabled: true
      mediator_model_role: voice_mediator
      speak_progress: brief          # off | brief | full
      voice_approvals: false

If speech.enabled is false, or base_url / api_key do not resolve, the plugin loads but the Read-aloud engine is unavailable (and Hands-free stays greyed out unless the Live engine is ready).

Live engine

plugins:
  - module: codumentor.plugins.voice_mode
    class: VoiceModePlugin
    args:
      default_engine: pipeline       # pipeline | live — what Hands-free opens with
      live:
        enabled: true
        model: gpt-realtime-2.1
        voice: marin

The live engine needs an OpenAI key with Realtime access. live.api_key / live.base_url default to the speech: block's values, so a deployment that already has speech: pointed at OpenAI only needs live.enabled: true — the live engine does not require speech.enabled. The key never reaches the browser: the server sets up each call (it relays the browser's WebRTC offer) and hangs it up when the panel closes, the session is swept, or max_session_s is reached. Audio then flows directly between the browser and OpenAI.

Parameter	Type	Default	Description
`default_engine`	string	`pipeline`	Engine Hands-free opens with (`pipeline` or `live`). Falls back to whichever is ready.
`live.enabled`	bool	`false`	Turn the live engine on.
`live.model`	string	`gpt-realtime-2.1`	Realtime model.
`live.voice`	string	`marin`	One of `alloy`, `ash`, `ballad`, `coral`, `echo`, `sage`, `shimmer`, `verse`, `marin`, `cedar`.
`live.reasoning_effort`	string	`low`	`minimal` … `xhigh`. Higher adds latency to every spoken turn.
`live.transcription_model`	string	`gpt-4o-mini-transcribe`	Captions of your speech in the panel (the model hears the audio itself). `""` turns them off.
`live.turn_detection`	string	`semantic_vad`	`semantic_vad` (end-of-turn by meaning) or `server_vad` (by silence).
`live.vad_eagerness`	string	`auto`	`semantic_vad` only: `low` waits longer before answering, `high` answers sooner.
`live.noise_reduction`	string	`near_field`	`near_field` (headset/laptop), `far_field` (speakerphone/room), or `off`.
`live.max_session_s`	int	`1800`	Longest single call; the server hangs up after it. 60–3600.
`live.api_key` / `live.base_url`	string	`""`	Override the `speech:` credentials for the live engine only.

speak_progress, answer_style and voice_approvals apply to both engines. Live calls are billed per audio token by OpenAI and cost noticeably more than the Read-aloud engine; call setup is limited to 6 per minute per user.

Parameters

Parameter	Type	Default	Description
`enabled`	bool	`true`	Master on/off.
`mediator_model_role`	string	`voice_mediator`	Model-profile role for the intent mediator. Use a fast model; falls back to the default chat model when the role is unset. See Model Profiles.
`utterance_silence_ms`	int	`800`	Continuous silence (ms) that closes an utterance. Minimum 200.
`utterance_warmup_silence_ms`	int	`2200`	Longer pause used only until the session language is known (or detection has given up). `0` turns the warm-up off.
`language_detect_min_speech_ms`	int	`2000`	Speech (ms) needed before a provider language detection is trusted, and before a segment counts as "enough content".
`pause_reeval_ms`	int	`2500`	After a "keep listening" decision, re-evaluate once the user has been silent this long. Minimum 500.
`speak_progress`	string	`brief`	How much of the in-progress answer is spoken. `off` — stay quiet until the turn finishes, then speak the final answer. `brief` — a short spoken budget per segment, then the full final answer. `full` — speak everything as it streams.
`echo_filter`	bool	`true`	Ignore transcripts that look like the assistant's own last spoken words picked up by the mic.
`phantom_filter`	bool	`true`	Drop transcripts that are the speech recogniser captioning noise ("Thanks for watching!", "Jó étvágyat!", subtitle credits). Plausible short phrases ("thank you") are dropped only over playback or when very short.
`max_tts_chars_per_turn`	int	`6000`	Cap on characters synthesised per turn. `0` disables the cap. When hit, the rest stays on screen and a short "the rest is on screen" line is spoken.
`session_ttl_s`	int	`300`	Idle hands-free sessions are swept after this many seconds. Minimum 30.
`voice_approvals`	bool	`false`	Map spoken yes/no onto a pending permission prompt. Opt-in: a "yes" heard over a speaker would otherwise approve a tool. `always_allow` is never reachable by voice; the panel still shows on-screen Approve / Deny either way.

Timing knobs the browser loop needs (utterance_silence_ms, the warm-up pair, pause_reeval_ms, speak_progress, echo_filter, max_tts_chars_per_turn, voice_approvals) are sent to the client when a session starts, so mic segmentation and the server mediator agree.

Intent mediator

A small, fast model watches the rolling transcript and chooses one action: wait, dispatch a turn, ignore filler/echo, steer, cancel-and-redirect, interrupt, or (when voice_approvals is on) approve/deny. It does not answer the user and it does not do the work — it only turns rambling speech into one clean instruction in the user's language.

Give voice_mediator a cheap/fast profile in Model Profiles so this loop stays snappy. If the role is missing, the default chat model is used (correct, but slower and more expensive).

Notes

Dispatched turns are normal conversation turns. They appear in the transcript; live captions and mediator state do not.
The voice panel is web-only. The TUI has no voice conversation view.
Utterance uploads and TTS calls are rate-limited (120 per minute per user) so a stuck client cannot hammer the speech provider.
Changes to the speech: block affect Voice Mode, Read Aloud, and Telegram voice together.