Codumentor logo Codumentor

Web Retrieve Plugin

The Web Retrieve Plugin provides a web_retrieve tool that allows agents to fetch and analyze content from web URLs, with optional agentic processing for complex information extraction tasks.

Overview

The Web Retrieve plugin:

  1. Registers a web_retrieve tool for fetching web content
  2. Operates in two modes: simple (direct fetch) and agentic (subagent processing)
  3. Pre-fetches content before spawning subagents to minimize tool call overhead
  4. Supports HTML (converted to text) and plain text content types
  5. Provides context management with configurable limits

Architecture

The plugin uses a layered architecture:

Main Agent
    │
    ▼
┌─────────────────────────────────────────┐
│ web_retrieve(urls, prompt?)             │
│ - Simple mode: returns content directly │
│ - Agentic mode: pre-fetches, spawns     │
│   subagent with content + fetch_urls    │
└────────────────────┬────────────────────┘
                     │ (if prompt provided)
                     ▼
            ┌────────────────────┐
            │ Subagent           │
            │ - Pre-fetched data │
            │ - fetch_urls tool  │
            │   (with seeking)   │
            └────────────────────┘

Key design decisions:

Configuration

Basic Configuration

plugins:
  - module: codumentor.plugins.webretrieve
    class: WebRetrievePlugin
    args:
      enabled: true
      timeout: 30               # Request timeout in seconds
      max_urls: 5               # Maximum URLs per request
      max_chars_per_url: 30000  # Maximum characters per URL
      max_content_length: 500000  # Maximum raw content in bytes
      target_agents: "main"     # Only main agent gets this tool

Configuration with JavaScript Rendering

For sites that load content dynamically with JavaScript, configure js_rendering_mode:

plugins:
  - module: codumentor.plugins.webretrieve
    class: WebRetrievePlugin
    args:
      enabled: true
      js_rendering_mode: "default"  # JS rendering ON by default, agent can disable
      timeout: 60                   # Longer timeout for JS rendering
      target_agents: "main"

JavaScript Rendering Modes

ModeDescriptionTool Schema
defaultJS rendering ON by default, agent can disable per-callShows render_js parameter with default: true
optionalJS rendering OFF by default, agent can enable per-callShows render_js parameter with default: false
disabledNever use JavaScript renderingNo render_js parameter shown

When to use each mode:

Note: JavaScript rendering requires Playwright to be installed:

# Using pip:
pip install playwright
playwright install chromium
playwright install-deps chromium  # Install system dependencies (may require sudo)

# Using uv:
uv pip install playwright
uv run playwright install chromium
uv run playwright install-deps chromium  # Install system dependencies (may require sudo)

If js_rendering_mode is not disabled but Playwright is not installed, the plugin will:

  1. Log an error-level message with installation instructions
  2. Fall back to js_rendering_mode="disabled"
  3. The render_js parameter will not appear in the tool schema

Legacy support: The deprecated enable_js_rendering parameter is still supported:

User Settings

Users can override the plugin's default js_rendering_mode via the Settings dialog in the UI (under "Extensions"). The user setting takes precedence over the plugin configuration.

Settings UI:

How it works:

  1. Plugin configuration sets the default js_rendering_mode
  2. User's personal setting (if set) overrides the plugin default
  3. If neither is set and Playwright is available, defaults to "default"

This allows administrators to set a sensible default while giving users control over their own experience.

Configuration Parameters

ParameterTypeDefaultDescription
enabledbooleantrueEnable/disable the plugin
timeoutinteger30Request timeout in seconds
max_urlsinteger5Maximum number of URLs per request
max_chars_per_urlinteger30000Maximum characters returned per URL
max_content_lengthinteger500000Maximum raw content per URL in bytes
target_agentsstring"main"Which agents get the tool: "main", "subagent", or "all"
js_rendering_modestring"default"JavaScript rendering mode: "default", "optional", or "disabled"
enable_js_renderingboolean-Deprecated. Use js_rendering_mode instead

Tool Usage

Main Tool: web_retrieve

The web_retrieve tool provides these parameters:

Subagent Tool: fetch_urls

The subagent has access to fetch_urls with additional parameters:

Modes of Operation

Simple Mode (no prompt)

When called without a prompt, the tool:

  1. Fetches the specified URLs concurrently
  2. Extracts text from HTML using html2text
  3. Returns content with truncation metadata
  4. Adds a note suggesting agentic mode if content was truncated
web_retrieve(urls="https://example.com/docs")

Output example:

## https://example.com/docs
*Showing first 30000 of 85000 characters*

[Content here...]

---

*Note: Some content was truncated. To extract specific information from
these pages, call web_retrieve again with a `prompt` parameter describing
what you need to find.*

Agentic Mode (with prompt)

When called with a prompt, the tool:

  1. Pre-fetches all URLs before spawning the subagent
  2. Creates a subagent with the pre-fetched content in its initial message
  3. Provides the subagent with fetch_urls tool for follow-up fetches
  4. The subagent analyzes content and completes the requested task
web_retrieve(
    urls="https://example.com/pricing",
    prompt="Find the pricing tiers and summarize the features of each tier"
)

The subagent can:

Examples

Example 1: Quick Content Preview

Fetch a page to see what's there:

web_retrieve(urls="https://docs.python.org/3/library/asyncio.html")

Example 2: Extract Specific Information

When you need specific information from a page:

web_retrieve(
    urls="https://docs.python.org/3/library/asyncio.html",
    prompt="List all the main asyncio functions for running coroutines and their brief descriptions"
)

Example 3: Multi-Page Research

Research across multiple pages:

web_retrieve(
    urls=["https://example.com/features", "https://example.com/pricing"],
    prompt="Compare the features available in each pricing tier"
)

When you need the subagent to follow links:

web_retrieve(
    urls="https://example.com/documentation",
    prompt="Find the API authentication documentation and explain the OAuth2 flow"
)

The subagent can use fetch_urls to retrieve linked pages discovered in the initial content.

How It Works

Content Extraction

  1. HTML pages: Converted to markdown/text using html2text - Links are preserved - Images are ignored - Script and style tags are removed
  1. Plain text: Returned as-is
  1. Other content types: Rejected with error message

Context Management

The plugin implements several limits to manage context size:

  1. URL count limit (max_urls): Limits concurrent fetches
  2. Character limit (max_chars_per_url): Truncates long content
  3. Byte limit (max_content_length): Limits raw HTTP response size

Pre-fetching Strategy

In agentic mode, URLs are pre-fetched before creating the subagent:

  1. All URLs are fetched concurrently
  2. Content is truncated per configured limits
  3. Truncation metadata is included
  4. Content is embedded in the subagent's initial prompt
  5. Subagent can immediately analyze without making tool calls

This reduces latency and token usage compared to having the subagent fetch content via tool calls.

Content Quality Detection

The plugin includes intelligent detection of potentially incomplete content, which is common with JavaScript-heavy websites:

  1. JavaScript Framework Detection: The raw HTML is analyzed for common framework indicators: - Next.js (__NEXT_DATA__) - React (data-reactroot, _reactRootContainer) - Vue (data-v- attributes) - Angular (ng-version, ng-app) - Nuxt (__NUXT__) - Svelte (data-svelte) - Astro (data-astro)
  1. Content Quality Heuristics: Content is flagged as potentially incomplete when: - Extracted text is very short (< 200 characters) - Extraction ratio is very low (< 5% of raw HTML converted to text) - JavaScript frameworks are detected and extracted text is short (< 500 characters)
  1. Automatic Hints (only when js_rendering_mode="optional"): When content may be incomplete, the tool adds a note to the output: `` Note: Content may be incomplete. JavaScript frameworks detected: Next.js, React. Consider using render_js=true for JavaScript-rendered pages. ``

Note: Hints only appear when js_rendering_mode="optional" because:

This helps agents understand when they should retry with JavaScript rendering enabled.

Error Handling

The plugin handles various error scenarios:

Security

Caching

The plugin includes an automatic URL cache to improve performance and consistency:

The cache is automatically managed and requires no configuration. To force a fresh fetch (bypassing cache), use the programmatic API with use_cache=False.

Troubleshooting

Content Not Extracted Properly

  1. Check content type: Only text/html and text/* are supported
  2. Try html2text: Install it for better HTML conversion: pip install html2text
  3. Check encoding: Site may use unusual encoding

Timeout Errors

  1. Increase timeout: Set timeout higher in plugin config
  2. Check network: Verify URL is accessible
  3. Try fewer URLs: Reduce concurrent URL count

Truncated Content

  1. Use agentic mode: Add a prompt to enable subagent with seeking
  2. Increase limits: Adjust max_chars_per_url if needed
  3. Be specific: Describe exactly what you need in the prompt

Subagent Not Finding Information

  1. Check pre-fetched content: The content might be on a linked page
  2. Guide the subagent: Be specific about what to look for
  3. Provide context: Mention where on the page the information might be

Testing

Run the plugin tests with:

python -m unittest tests.test_web_retrieve_plugin -v

See Also