Web Search Plugin

The Web Search Plugin provides a web_search tool that allows agents to search the web and gather information, with optional agentic research mode for comprehensive multi-page analysis.

Overview

The Web Search plugin:

1. Registers a web_search tool for searching the web
2. Operates in two modes: simple (direct search) and research (subagent processing)
3. Supports multiple search providers (DuckDuckGo, Brave, Tavily, Serper, Google CSE, SearXNG)
4. Provides a unified interface regardless of the underlying search provider

Architecture

The plugin uses a layered architecture:

Main Agent
    │
    ▼
┌─────────────────────────────────────────────┐
│ web_search(query, agentic_research?)        │
│ - Simple mode: returns search results       │
│ - Research mode: spawns subagent with       │
│   search + fetch_urls tools                 │
└────────────────────┬────────────────────────┘
                     │ (if agentic_research=true)
                     ▼
            ┌────────────────────┐
            │ Subagent           │
            │ - search tool      │
            │ - fetch_urls tool  │
            │   (with seeking)   │
            └────────────────────┘

Key design decisions:

• The main agent sees only web_search (single tool interface)
• Simple mode returns search results (titles, URLs, snippets) directly
• Research mode spawns a subagent with search and fetch_urls tools
• The subagent can search multiple times and fetch page content
• Multiple search providers supported with consistent interface

Configuration

Basic Configuration (DuckDuckGo - Free, No API Key)

plugins:
  - module: codumentor.plugins.websearch
    class: WebSearchPlugin
    args:
      enabled: true
      provider: "duckduckgo"  # Free, unlimited, no API key or extra packages needed
      max_results: 10
      target_agents: "main"

Brave Search (Free Tier: 2000 queries/month)

plugins:
  - module: codumentor.plugins.websearch
    class: WebSearchPlugin
    args:
      enabled: true
      provider: "brave"
      provider_config:
        api_key: "${BRAVE_API_KEY}"  # Environment variable
        country: "us"
        safesearch: "moderate"

Tavily (AI-Optimized Search)

plugins:
  - module: codumentor.plugins.websearch
    class: WebSearchPlugin
    args:
      enabled: true
      provider: "tavily"
      provider_config:
        api_key: "${TAVILY_API_KEY}"
        search_depth: "basic"  # or "advanced"

SearXNG (Self-Hosted)

plugins:
  - module: codumentor.plugins.websearch
    class: WebSearchPlugin
    args:
      enabled: true
      provider: "searxng"
      provider_config:
        base_url: "https://your-searxng-instance.com"

Configuration Parameters

Parameter	Type	Default	Description
enabled	boolean	true	Enable/disable the plugin
provider	string	"duckduckgo"	Search provider to use
provider_config	object	{}	Provider-specific configuration
max_results	integer	10	Maximum results per search
timeout	integer	30	Request timeout in seconds
research_max_iterations	integer	15	Maximum iterations for research subagent
max_urls	integer	5	Maximum URLs per fetch request (research mode)
max_chars_per_url	integer	30000	Maximum characters per URL (research mode)
target_agents	string	"main"	Which agents get the tool: "main", "subagent", or "all"

Supported Providers

Provider	Auth Required	Free Tier	Best For
duckduckgo	No	Unlimited	General use, no setup needed
brave	API Key	2000/month	Privacy-focused search
searxng	No (self-hosted)	Unlimited	Privacy, self-hosted
tavily	API Key	Free tier	AI-optimized results
serper	API Key	2500 free	Google results
google_cse	API Key + CSE ID	100/day	Custom search scope

Provider Configuration Details

DuckDuckGo

provider_config:
  region: "wt-wt"      # Search region (wt-wt = worldwide)
  safesearch: "moderate"  # off, moderate, strict
  timelimit: null      # d (day), w (week), m (month), y (year)

Brave

provider_config:
  api_key: "${BRAVE_API_KEY}"
  country: "us"
  search_lang: "en"
  safesearch: "moderate"

SearXNG

provider_config:
  base_url: "https://searx.example.com"  # Required
  categories: "general"
  language: "en"
  safesearch: 1  # 0=off, 1=moderate, 2=strict

Tavily

provider_config:
  api_key: "${TAVILY_API_KEY}"
  search_depth: "basic"  # or "advanced"
  include_answer: false
  include_domains: []
  exclude_domains: []

Serper

provider_config:
  api_key: "${SERPER_API_KEY}"
  gl: "us"  # Country
  hl: "en"  # Language

Google CSE

provider_config:
  api_key: "${GOOGLE_API_KEY}"
  cx: "${GOOGLE_CSE_ID}"  # Custom Search Engine ID
  gl: "us"
  safe: "medium"

Tool Usage

Main Tool: web_search

The web_search tool provides these parameters:

• query (required): The search query or research task description
• In simple mode: a search query
• In research mode: a detailed task description

• agentic_research (optional, default: false): Enable research mode
• false: Execute search, return results directly
• true: Spawn subagent for comprehensive research

• num_results (optional, default: 10): Number of results (simple mode only)

Modes of Operation

Simple Mode (agentic_research=false)

When called without research mode, the tool:

1. Executes the search query
2. Returns formatted results with titles, URLs, and snippets
3. Fast and efficient for finding relevant pages

web_search(query="Python asyncio tutorial")

Output example:

## Search Results for: Python asyncio tutorial

Found 10 results:

### 1. Asyncio — Python 3.12 Documentation
**URL:** https://docs.python.org/3/library/asyncio.html
This module provides infrastructure for writing single-threaded concurrent code...

### 2. Python Asyncio Tutorial
**URL:** https://realpython.com/async-io-python/
A comprehensive guide to async IO in Python, covering coroutines, tasks...

---

*To get detailed content from any of these pages, use web_retrieve
or call web_search again with agentic_research=true for in-depth research.*

Research Mode (agentic_research=true)

When called with research mode enabled, the tool:

1. Creates a subagent with search and fetch_urls tools
2. The subagent searches for relevant information
3. Fetches and analyzes page content from promising results
4. Synthesizes findings into a comprehensive response

web_search(
    query="What are the performance characteristics of different Python async frameworks? Compare asyncio, Trio, and AnyIO.",
    agentic_research=true
)

Important: When using research mode, provide a detailed task description rather than a simple search query. The subagent will formulate its own search queries based on your task.

Examples

Example 1: Quick Lookup

Find resources on a topic:

web_search(query="FastAPI authentication best practices")

Example 2: Research a Topic

Deep dive into a technical topic:

web_search(
    query="Investigate the current state of Python type checking tools. Compare mypy, pyright, and pyre in terms of features, performance, and IDE integration.",
    agentic_research=true
)

Example 3: Find Documentation

Locate official documentation:

web_search(query="site:docs.python.org asyncio.gather")

Example 4: Compare Technologies

Research and compare technologies:

web_search(
    query="Compare Redis and Memcached for Python web application caching. Consider performance, features, ease of use, and Python library support.",
    agentic_research=true
)

How It Works

Search Execution

1. Simple mode: Query is sent directly to the configured provider
2. Research mode: Subagent formulates search queries based on the task

Result Processing

1. Results are normalized to a common format (title, URL, snippet, position)
2. Results are formatted for easy reading
3. URLs are preserved for follow-up fetching

Subagent Capabilities (Research Mode)

The research subagent can:

• Execute multiple search queries
• Fetch full page content from promising results
• Navigate through long pages using offset/limit
• Synthesize information from multiple sources
• Cite sources with URLs

Error Handling

The plugin handles various error scenarios:

• Missing API key: Clear error message with setup instructions
• Network errors: Reported with specific error message
• Rate limits: Provider-specific error messages
• Invalid queries: Validation before sending request
• Provider errors: Detailed error from provider API

Security

• API key handling: Supports environment variable substitution (${VAR})
• No arbitrary code execution: Search results are text only
• HTTPS required: For paid providers
• Configurable safe search: Control result filtering

Troubleshooting

No Results Returned

1. Check query: Try different search terms
2. Check provider: Ensure API key is valid (if required)
3. Try different provider: DuckDuckGo is a good fallback

API Key Errors

1. Check environment variable: Ensure ${VAR} is set
2. Check key format: Some providers require specific formats
3. Check quota: Free tiers have limits

Research Mode Not Finding Information

1. Be more specific: Describe exactly what you need
2. Provide context: Include relevant background
3. Check simple mode first: Verify search returns relevant results

Slow Responses

1. Use simple mode: For quick lookups
2. Reduce max_results: Fewer results = faster
3. Check provider: Some providers are faster than others

Testing

Run the plugin tests with:

python -m unittest tests.test_web_search_plugin -v

See Also

• Web Retrieve Plugin - Fetch and analyze specific URLs
• Plugin Development Guide - Create custom plugins
• MCP Plugin - Alternative for integrating external tools