Inference Metrics Plugin

Measures how the model is generating: time to first token (TTFT) and output speed (tokens/sec). Averages are kept per prompt-size tier over a rolling window. When those averages look slow, the plugin can pause background LLM work so interactive chat stays responsive.

Enabled in the shipped default config. Pairs with Telemetry (overall timing) and Console Report (session summary).

Configuration

plugins:
  - module: codumentor.plugins.inference_metrics
    class: InferenceMetricsPlugin
    args:
      rolling_window_seconds: 180
      ttft_threshold_ms: 6000
      tokens_per_sec_threshold: 40
      min_congestion_samples: 3
      staleness_seconds: 30

All args are optional.

Arg	Type	Default	Description
rolling_window_seconds	float	180	How far back rolling averages look (seconds). At most 100 samples are kept per tier.
ttft_threshold_ms	float	6000	Average TTFT at or above this is treated as congested.
tokens_per_sec_threshold	float	40	Average tokens/sec at or below this is treated as congested.
min_congestion_samples	int	3	Minimum samples in the window before congestion can be declared.
staleness_seconds	float	30	If the newest sample is older than this and no LLM call is in flight, the system is treated as idle (not congested).

Prompt-size tiers (by prompt tokens): short below 2K, medium 2K–10K, long 10K and above.

Usage

After each turn, a card appears in the transcript (web UI and TUI): TTFT 245ms | 42.5 tok/s. A slow request also shows slow. When the rolling averages are congested, a small hourglass appears on the compact card.

Expand the card for this turn (TTFT, tokens/sec, prompt size, output tokens, and LLM call count when there was more than one) plus a short / medium / long table of recent averages.

At INFO log level each LLM call logs TTFT, tokens/sec, tier, and token counts. See Logging.

With session_report: true, an Inference Performance section is added to the end-of-session report (printed by Console Report). Congestion is labeled Status: CONGESTED when thresholds are exceeded.

Notes

• TTFT on the card is the first LLM call of the turn (what the user waited for). Tokens/sec is the average of the calls in that turn.
• Tokens/sec is measured after the first token, so it does not include TTFT.
• When job queues are enabled, congestion closes the background_llm queue: work such as memory storage waits until averages recover. Interactive requests are not gated.
• There is no YAML switch to hide the transcript cards.