Codumentor logo Codumentor

Inference Metrics Plugin

The Inference Metrics Plugin tracks LLM generation performance, specifically Time to First Token (TTFT) and generation speed (tokens/sec). It provides rolling global averages tiered by prompt size.

Overview

The plugin measures:

Configuration

plugins:
  - module: "codumentor.plugins.inference_metrics"
    class: "InferenceMetricsPlugin"
    args:
      rolling_window_seconds: 180      # Time window for rolling averages (default: 3 min)
      ttft_threshold_ms: 6000          # TTFT >= this triggers congestion warning (default: 6s)
      tokens_per_sec_threshold: 40     # Tokens/sec <= this triggers congestion warning
      min_congestion_samples: 3        # Min samples required to declare congestion (default: 3)
      staleness_seconds: 30            # Samples older than this are considered stale (default: 30s)

All parameters are optional and have sensible defaults.

Output

Log Output

After each LLM call, the plugin logs inference metrics:

INFO - Inference metrics: TTFT=245.3ms, tokens/sec=42.5, tier=medium, prompt=1250, completion=450

Artifacts

The plugin stores metrics in ctx.kv.artifacts["inference_metrics"]:

{
  "request": {
    "ttft_ms": 245.3,
    "tokens_per_sec": 42.5,
    "prompt_tokens": 1250,
    "completion_tokens": 450,
    "prompt_tier": "medium"
  },
  "global": {
    "short": {"avg_ttft_ms": 150.5, "avg_tokens_per_sec": 45.2, "sample_count": 12},
    "medium": {"avg_ttft_ms": 280.3, "avg_tokens_per_sec": 42.1, "sample_count": 8},
    "long": {"avg_ttft_ms": 520.7, "avg_tokens_per_sec": 38.5, "sample_count": 3},
    "overall": {"avg_ttft_ms": 317.2, "avg_tokens_per_sec": 41.9, "total_samples": 23, "rolling_window_seconds": 180},
    "congested": false,
    "last_updated": 1707123456.789
  },
  "congested": false
}

The congested field is available at the top level for easy access by other plugins that may want to buffer non-live requests during high-load periods.

UI Card

When enabled in settings, displays a compact card after each response:

Compact view:

[lightning] TTFT 245ms | 42.5 tok/s

When congested, shows a subtle hourglass indicator and "slow" text:

[lightning] TTFT 1245ms | 15.2 tok/s | slow [hourglass]

Expanded view shows:

Implementation Details

Hook Registration

async def register(self, bus):
    bus.on("onLLMCallStart", self._on_llm_call_start, priority=100)
    bus.on("onLLMStream", self._on_llm_stream, priority=100)
    bus.on("onLLMCallEnd", self._on_llm_call_end, priority=100)
    bus.on("onUsageTracked", self._on_usage_tracked, priority=100)
    bus.on("onResponsePersist", self._on_response_persist, priority=100)
    bus.on("onSessionReport", self._on_session_report, priority=100)

Data Flow

HookPurpose
onLLMCallStartRecord start time
onLLMStreamDetect first non-empty content token, record first token time
onLLMCallEndRecord end time, calculate TTFT
onUsageTrackedCapture token counts from iteration_usage, calculate tokens/sec, add sample to global buffer
onResponsePersistPublish to artifacts, emit UI event

First Token Detection

The plugin detects the first meaningful content token via onLLMStream:

async def _on_llm_stream(self, agent, ctx, payload):
    kv = ctx.get("kv", {})
    delta_text = payload.get("delta_text", "")

    # Skip if already started or empty/whitespace chunk
    if kv.get("_inference_content_started"):
        return None
    if delta_text and delta_text.strip():
        kv["_inference_first_token_time"] = time.time()
        kv["_inference_content_started"] = True
    return None

Generation Speed Calculation

Tokens/sec is calculated in onUsageTracked using stored timing and token counts from iteration_usage:

async def _on_usage_tracked(self, agent, ctx, payload):
    iteration_usage = payload.get("iteration_usage", {})
    completion_tokens = iteration_usage.get("completion_tokens", 0)

    first_token_time = kv.get("_inference_first_token_time")
    end_time = kv.get("_inference_call_end")

    if first_token_time and end_time and completion_tokens > 0:
        generation_time = end_time - first_token_time
        tokens_per_sec = completion_tokens / generation_time

This measures actual generation throughput, excluding TTFT latency.

Prompt Size Tiers

TIER_THRESHOLDS = {
    "short": (0, 2000),        # < 2K tokens
    "medium": (2000, 10000),   # 2K-10K tokens
    "long": (10000, float('inf')),  # > 10K tokens
}

Rolling Average Storage

Thread-safe global storage with bounded memory:

# Rolling buffers per tier (deque with maxlen=100)
_INFERENCE_SAMPLES: Dict[str, Deque[InferenceSample]] = {
    "short": deque(maxlen=100),
    "medium": deque(maxlen=100),
    "long": deque(maxlen=100),
}
_SAMPLES_LOCK = threading.Lock()
ROLLING_WINDOW_SECONDS = 180

Metrics Explained

MetricDescription
TTFTTime from onLLMCallStart to first non-empty content chunk in onLLMStream
Tokens/secCompletion tokens divided by time from first token to onLLMCallEnd
Prompt TierClassification based on prompt token count
Rolling AveragesPer-tier averages computed over samples from the last 3 minutes

Why Tier by Prompt Size?

TTFT typically increases with prompt size due to:

Tiering allows meaningful comparisons within similar prompt sizes.

Use Cases

  1. Performance Monitoring: Track LLM inference latency and throughput trends
  2. Model Comparison: Compare TTFT and generation speed across different models
  3. Debugging: Identify slow responses or degraded performance
  4. Capacity Planning: Understand generation throughput for load estimation

Settings Schema

The plugin exposes this user setting via the UI:

settings_schema:
  properties:
    show_inference_metrics:
      type: boolean
      title: "Show Inference Metrics"
      description: "Display TTFT and generation speed after each response"
      default: false

Rolling window, TTFT threshold, and tokens/sec threshold are system-level parameters configured via the YAML config (see Configuration above), not per-user settings.

Congestion Detection

The plugin monitors overall inference performance and flags congestion when:

The staleness check ensures the gate opens quickly when the system goes idle (no active LLM calls), rather than waiting for samples to expire from the full rolling window.

Congestion state is:

Other plugins can check this flag to decide whether to buffer non-urgent requests during high-load periods.

Session Report Contribution

On session exit (when the session report is enabled), the plugin contributes an "Inference Performance" section to the end-of-session report via onSessionReport. This section includes:

This data is rendered by the Console Report Plugin.

Job Queue Gate

When the job queue system is enabled, the Inference Metrics Plugin automatically registers as a gate for the background_llm queue. This pauses background LLM work (like memory storage) when the system is congested:

class InferenceMetricsGate:
    def is_open(self) -> bool:
        """Return False when congested (pause processing)."""
        return not is_congested

This prevents background work from competing with interactive requests during high-load periods. When congestion clears, queued jobs resume automatically.

See Job Queue System for details on how gates control job processing.

Dependencies

Source Code

See src/codumentor/plugins/inference_metrics/inference_metrics_plugin.py for the complete implementation.

See Also