Voice Mode Plugin

The Voice Mode plugin adds a full-screen, hands-free voice conversation to the web UI. You talk; an intent mediator waits until you have finished a request, then starts a normal agent turn; the reply is spoken back while captions stay on screen.

It is not the same as:

Push-to-talk — hold the mic button in the chat input to dictate a single message.
Speak replies — a toggle that reads the current typed-chat answer aloud.
Read Aloud — a per-message "read this" action.
Telegram voice notes — voice in/out inside Telegram, not the web overlay.

All of those share the top-level speech: service (STT/TTS). Voice Mode is the conversational overlay on top of that service.

Using it

The Start voice conversation button appears on the chat input when the plugin is enabled and speech is configured. It needs an open conversation.

Click the button. The browser will ask for microphone access the first time.
A full-screen overlay opens. Status cycles through Listening…, Thinking…, and Speaking…. The hint is "Just start talking — I'm listening."
Speak a request in your own language. Short pauses are fine; a longer pause (or a clearly finished sentence) is what sends it. Filler ("mhm", "okay") is ignored.
While the assistant is working you can: - Steer — add or refine the same task ("also cover the async path"). Work so far is kept. - Redirect — change the goal. The running turn is cancelled and replaced. - Interrupt — say stop / wait / forget it. Work is cancelled and nothing new starts.
If a tool needs approval, the overlay shows Approve / Deny. Spoken yes/no is off unless the operator enables voice_approvals (see below).
Click End to leave the overlay. The conversation transcript still has the turns that were dispatched.

The first utterances of a session may wait a little longer so the speech service can detect the language. After that, short phrases ("stop", "igen") inherit it.

Keep the tab visible: the overlay holds a screen wake lock, and the mic loop pauses in a background tab. Use HTTPS in production — browsers only expose the microphone on a secure origin (localhost is allowed).

If the button is missing, speech is not configured or the plugin is disabled. If the overlay says voice conversation isn't available, the same is true at runtime (for example the API key no longer resolves).

Configuration

Enable the plugin and the shared speech service. The speech api_key must resolve through an environment variable (the same constraint as Read Aloud): a per-user ${secret:…} cannot resolve for these out-of-turn web calls.

speech:
  enabled: true
  base_url: "https://api.openai.com/v1"
  api_key: "${OPENAI_API_KEY}"
  tts_model: "tts-1"
  tts_voice: "alloy"

plugins:
  - module: codumentor.plugins.voice_mode
    class: VoiceModePlugin
    args:
      enabled: true
      mediator_model_role: voice_mediator
      speak_progress: brief          # off | brief | full
      voice_approvals: false

If speech.enabled is false, or base_url / api_key do not resolve, the plugin loads but the button stays hidden.

Parameters

Parameter	Type	Default	Description
`enabled`	bool	`true`	Master on/off.
`mediator_model_role`	string	`voice_mediator`	Model-profile role for the intent mediator. Use a fast model; falls back to the default chat model when the role is unset. See Model Profiles.
`utterance_silence_ms`	int	`800`	Continuous silence (ms) that closes an utterance. Minimum 200.
`utterance_warmup_silence_ms`	int	`2200`	Longer pause used only until the session language is known (or detection has given up). `0` turns the warm-up off.
`language_detect_min_speech_ms`	int	`2000`	Speech (ms) needed before a provider language detection is trusted, and before a segment counts as "enough content".
`pause_reeval_ms`	int	`2500`	After a "keep listening" decision, re-evaluate once the user has been silent this long. Minimum 500.
`speak_progress`	string	`brief`	How much of the in-progress answer is spoken. `off` — stay quiet until the turn finishes, then speak the final answer. `brief` — a short spoken budget per segment, then the full final answer. `full` — speak everything as it streams.
`echo_filter`	bool	`true`	Ignore transcripts that look like the assistant's own last spoken words picked up by the mic.
`max_tts_chars_per_turn`	int	`6000`	Cap on characters synthesised per turn. `0` disables the cap. When hit, the rest stays on screen and a short "the rest is on screen" line is spoken.
`session_ttl_s`	int	`300`	Idle overlay sessions are swept after this many seconds. Minimum 30.
`voice_approvals`	bool	`false`	Map spoken yes/no onto a pending permission prompt. Opt-in: a "yes" heard over a speaker would otherwise approve a tool. `always_allow` is never reachable by voice; the overlay still shows on-screen Approve / Deny either way.

Timing knobs the browser loop needs (utterance_silence_ms, the warm-up pair, pause_reeval_ms, speak_progress, echo_filter, max_tts_chars_per_turn, voice_approvals) are sent to the client when a session starts, so mic segmentation and the server mediator agree.

Intent mediator

A small, fast model watches the rolling transcript and chooses one action: wait, dispatch a turn, ignore filler/echo, steer, cancel-and-redirect, interrupt, or (when voice_approvals is on) approve/deny. It does not answer the user and it does not do the work — it only turns rambling speech into one clean instruction in the user's language.

Give voice_mediator a cheap/fast profile in Model Profiles so this loop stays snappy. If the role is missing, the default chat model is used (correct, but slower and more expensive).

Notes