Web Retrieve

Gives the agent a web_retrieve tool to fetch web pages (and some documents) and, optionally, analyze them.

• Simple mode (no prompt): returns extracted text.
• Agentic mode (with prompt): a subagent reads the pages, can follow links, and can page through long content.

HTML is converted to text; PDF, DOCX, and XLSX are extracted when possible. Only http/https URLs are allowed.

Configuration

plugins:
  - module: codumentor.plugins.webretrieve
    class: WebRetrievePlugin
    args:
      enabled: true
      timeout: 30
      max_urls: 5
      max_chars_per_url: 30000
      max_content_length: 500000
      target_agents: "main"
      js_rendering_mode: "default"

Parameter	Default	Description
enabled	true	Enable the plugin.
timeout	30	Request timeout in seconds.
max_urls	5	Maximum URLs per call.
max_chars_per_url	30000	Maximum characters returned per URL.
max_content_length	500000	Maximum raw response size per URL, in bytes.
target_agents	"main"	"main", "subagent", or "all".
js_rendering_mode	"default"	How JavaScript-rendered pages are handled (see below).
enable_js_rendering	—	Deprecated. true → optional, false → disabled.

JavaScript rendering

Needs a Chromium-family browser on the host (Chrome, Edge, or Chromium). If the mode is not disabled and no browser is found, the plugin logs install instructions and falls back to disabled. Set CODUMENTOR_CHROME_PATH if the browser is not on the usual path.

Mode	Behavior
default	Render JS unless the agent sets render_js=false.
optional	Static fetch unless the agent sets render_js=true. Incomplete-looking pages get a hint to retry with JS.
disabled	Never render JS; no render_js argument.

Users can override the mode (and turn the plugin off) in Settings → Extensions → Web Retrieve. A personal setting wins over YAML.

Tool: web_retrieve

Argument	Required	Description
urls	yes	One URL or a list (capped by max_urls).
prompt	no	If set, run agentic analysis of the pages.
render_js	no	Present when mode is default or optional.

Truncated simple-mode results suggest calling again with a prompt. Successful fetches are cached for 5 minutes so repeated hits on the same URL stay consistent.

Notes

• Increase timeout for slow or JS-heavy sites; use fewer URLs if calls time out.
• For a specific fact on a long page, use agentic mode with a precise prompt.
• Local file:// URLs are rejected; redirects are followed up to 5 times.