Context Summarization Plugin

The Context Summarization Plugin manages context window limits by intelligently truncating conversations while preserving important context like system prompts and recent messages.

Overview

This plugin implements Phase 1 of the context window management system, providing:

• Basic context detection: Automatically detects when conversations exceed model context limits
• Simple truncation: Preserves system prompts and recent messages while removing middle content
• Configurable limits: Supports different context limits for different models
• Safety margins: Reserves tokens for response generation

Configuration

Basic Configuration

plugins:
  - module: "codumentor.plugins.context_summarization"
    class: "ContextSummarizationPlugin"
    priority: 10
    args:
      enabled: true
      max_context_tokens: 128000
      safety_margin: 1000
      min_recent_messages: 5
      split_ratio: 0.7

Configuration Options

Option	Type	Default	Description
enabled	boolean	true	Enable/disable the plugin
max_context_tokens	integer	128000	Default context limit for unknown models
safety_margin	integer	1000	Tokens to reserve for response generation
min_recent_messages	integer	5	Minimum recent messages to preserve
split_ratio	float	0.7	Portion of conversation to summarize (0.0-1.0)
model_context_limits	object	{}	Model-specific context limits

Model-Specific Limits

You can override the default context limit for specific models:

args:
  model_context_limits:
    "Qwen3-235B-A22B-Instruct-2507": 262144

How It Works

1. Context Detection

The plugin intercepts LLM calls via the onLLMCallStart hook and:

1. Estimates the total token count for all messages
2. Compares against the model's context limit (minus safety margin)
3. Determines if truncation is needed

2. Truncation Strategy

When truncation is needed, the plugin:

1. Preserves system messages: All system prompts are kept intact
2. Preserves recent messages: Keeps the last N messages (configurable)
3. Removes middle content: Eliminates older messages from the middle of the conversation
4. Maintains order: System messages first, then preserved conversation content

3. Token Estimation

The plugin uses the existing tokenizer infrastructure to accurately estimate token counts:

• Uses model-specific tokenizers when available
• Falls back to character-based estimation if tokenizers fail
• Accounts for message formatting overhead

Examples

Example 1: Basic Configuration

plugins:
  - module: "codumentor.plugins.context_summarization"
    class: "ContextSummarizationPlugin"
    args:
      enabled: true
      max_context_tokens: 128000
      safety_margin: 1000
      min_recent_messages: 5

Example 2: Aggressive Truncation for Small Models

plugins:
  - module: "codumentor.plugins.context_summarization"
    class: "ContextSummarizationPlugin"
    args:
      enabled: true
      max_context_tokens: 16385  # GPT-3.5 limit
      safety_margin: 500
      min_recent_messages: 3  # Preserve fewer messages
      split_ratio: 0.8  # Summarize more content

Example 3: Disabled Plugin

plugins:
  - module: "codumentor.plugins.context_summarization"
    class: "ContextSummarizationPlugin"
    args:
      enabled: false  # Plugin is disabled

Logging

The plugin provides detailed logging for debugging:

• Info level: Reports when truncation occurs and the impact
• Debug level: Detailed token counts and truncation decisions
• Warning level: Token estimation failures and fallbacks
• Error level: Critical errors during context management

Limitations (Phase 1)

This is Phase 1 implementation with the following limitations:

• Simple truncation only: No intelligent summarization of removed content
• No caching: Truncation decisions are made fresh each time
• Basic preservation: Only preserves system messages and recent messages
• No fallback models: No automatic switching to models with larger context windows

Future Phases

• Phase 2: Summarization agent integration
• Phase 3: Advanced configuration and optimization
• Phase 4: Caching and performance improvements

Testing

The plugin includes comprehensive unit tests covering:

• Configuration parsing and validation
• Token estimation accuracy
• Context limit detection
• Truncation logic and message preservation
• Error handling and fallbacks
• Plugin registration and hook integration

Run tests with:

python -m pytest tests/test_plugins/test_context_summarization_plugin.py -v