Monitoring

This guide covers logging configuration and real-time log monitoring for Codumentor deployments.

For reading and analyzing logs an instance has already written — format eras, parsing pitfalls, and scan tooling — see Reading Instance Logs.

Logging Configuration

Codumentor's logging is configured through the logging section of codumentor.yaml.

Basic Configuration

logging:
  level: INFO
  file: ./logs/codumentor.log
  logger_levels:
    codumentor.agent: DEBUG
    codumentor.tools: WARNING
    codumentor.api: INFO
    codumentor.ingest: DEBUG

Setting	Default	Description
`level`	`INFO`	Base log level. Options: `DEBUG`, `INFO`, `WARNING`, `ERROR`, `CRITICAL`.
`file`	`./logs/codumentor.log`	Path to the log file.
`logger_levels`	`{}`	Per-logger level overrides. Keys are dotted logger names (e.g., `codumentor.agent`).

The logger_levels setting lets you increase verbosity for specific components without flooding the log with debug output from the entire application. For example, setting codumentor.agent: DEBUG while keeping the base level at INFO gives detailed agent traces without noisy tool output.

Structured JSON Logging

For production deployments, enable structured JSON logging to integrate with log aggregation systems like ELK, Splunk, or Datadog:

logging:
  level: INFO
  file: ./logs/codumentor.log
  json_logging: true
  json_log_file: ./logs/codumentor.json.log
  json_include_context: true
  json_include_stack_info: false
  json_include_process_info: false

Setting	Default	Description
`json_logging`	`false`	Enable structured JSON logging.
`json_log_file`	`null`	Path to the JSON log file. If not set, uses the standard log file path with a `.json` extension.
`json_include_context`	`true`	Include contextual metadata (session IDs, request data) in log entries.
`json_include_stack_info`	`false`	Include stack traces. Useful for debugging, noisy in production.
`json_include_process_info`	`false`	Include process and thread identifiers.
`json_date_format`	ISO 8601 with ms	Custom date format string. Default: `%Y-%m-%dT%H:%M:%S.%f`.

JSON Log Format

Each log entry is a single JSON line:

{
  "timestamp": "2026-02-16T10:30:00.123",
  "level": "INFO",
  "logger": "codumentor.api",
  "message": "Request completed",
  "module": "api",
  "function": "handle_request",
  "line": 87,
  "context": {
    "session_id": "sess_abc123",
    "user_id": "jdoe",
    "request_id": "req_xyz789"
  }
}

The context field contains metadata about the current operation, including session identifiers, user information, and request-specific data. This is valuable for correlating log entries across a single user session or request.

Processing Logs with jq

JSON logs can be queried and filtered with jq:

# Show only ERROR-level entries
jq 'select(.level == "ERROR")' logs/codumentor.json.log

# Filter by logger name
jq 'select(.logger == "codumentor.agent")' logs/codumentor.json.log

# Extract timestamps and messages
jq '{timestamp: .timestamp, message: .message}' logs/codumentor.json.log

# Find entries related to a specific session
jq 'select(.context.session_id == "sess_abc123")' logs/codumentor.json.log

# Count entries by log level
jq -s 'group_by(.level) | map({level: .[0].level, count: length})' logs/codumentor.json.log

# Search for a keyword in messages
jq 'select(.message | contains("ingestion complete"))' logs/codumentor.json.log

Per-Logger Levels

Customize log verbosity per component to focus on what matters:

logging:
  level: WARNING
  logger_levels:
    codumentor.agent: DEBUG      # Detailed agent reasoning traces
    codumentor.tools: WARNING    # Only tool warnings and errors
    codumentor.api: INFO         # API request/response logging
    codumentor.ingest: INFO      # Ingestion progress
    codumentor.plugins: DEBUG    # Plugin lifecycle debugging

Common logger names:

Logger	Content
`codumentor.agent`	Agent turns, tool calls, reasoning
`codumentor.tools`	Tool execution, shell commands, file operations
`codumentor.api`	HTTP requests, SSE connections, authentication
`codumentor.ingest`	Repository cloning, chunking, embedding
`codumentor.plugins`	Plugin loading, hook execution, plugin errors

Log Monitoring

Codumentor includes a real-time log monitoring feature with a web interface for viewing live log updates.

Usage

# Real-time monitoring (default)
python scripts/logs_viewer/__main__.py

# Static view (disable real-time monitoring)
python scripts/logs_viewer/__main__.py --static

# Custom log file and port
python scripts/logs_viewer/__main__.py --log-file logs/codumentor.json.log --port 8080

The log viewer is a standalone script (scripts/logs_viewer/) and is not included in the built codumentor binary.

Web Interface Features

Live updates -- Log entries appear in real time without page refresh.
Connection status -- Visual indicator showing whether monitoring is active.
New session highlighting -- Newly detected sessions are highlighted with animation.
Auto-reconnection -- The WebSocket connection automatically reconnects if dropped.
Syntax highlighting -- Log entries are color-coded by level and formatted for readability.

Session Metadata

The monitoring interface tracks metadata for each session:

Field	Description
Log count	Number of log entries in the session
Timestamps	First seen and last seen times
Loggers	Which loggers produced entries in this session
Log levels	Which severity levels appeared
User messages	First user input message extracted from the logs

Architecture

The log monitoring system uses a file watcher with incremental parsing:

LogFileHandler monitors the log file for changes using the watchdog library.
Only new entries are parsed -- the system tracks its file position and processes appended content.
LogFileMonitor manages the monitoring lifecycle and forwards updates.
A WebSocket connection pushes updates to the web interface in real time.

This design keeps CPU and memory usage low even for large log files.

Best Practices

Use JSON logging in production. Structured logs enable filtering, searching, and alerting through standard tools.
Set appropriate log levels per component. Use DEBUG only for the components you are actively investigating. Leave everything else at INFO or WARNING.
Configure log rotation. Use logrotate or a similar tool to prevent log files from filling the disk. Example logrotate configuration:

``/opt/codumentor/logs/*.log { daily rotate 14 compress delaycompress missingok notifempty copytruncate }``

Monitor for errors. Set up alerts on ERROR and CRITICAL entries. With JSON logging, this is straightforward to integrate with any alerting system.
Protect log files. Logs may contain conversation content, session identifiers, and user information. Restrict filesystem permissions to the service user.