Codumentor logo Codumentor

Reading Instance Logs

A field guide to reading and analyzing the logs that Codumentor instances have
already written — formats, eras, pitfalls, and scan tooling. Written primarily
for agents (and humans) doing log forensics on a production host. For
configuring logging, see Monitoring. For a worked example of
a full-fleet analysis, see the
2026-07-07 log audit.

Concrete paths below describe the ardinsys production host (user codumentor);
adapt the pattern for other deployments.

1. Where logs live

Each instance is a directory ~/codumentor-<name>/ containing logs/,
its YAML config, audit.db, and a codumentor symlink to the shared checkout.
Each runs as a systemd user service codumentor-api-<name>.service.

ls -d ~/codumentor-*/logs                          # instances on disk
systemctl --user list-units 'codumentor-api-*'     # instances actually running

Always compare those two lists. They can disagree in both directions:
a directory with no service (dead instance), or a running service whose
directory was deleted (ghost instance — it still serves its port and writes
logs to deleted inodes; check ls -l /proc/<MainPID>/cwd and
lsof -p <pid> | grep deleted).

Inside logs/:

FileWhat it is
codumentor.logcurrent log (text format)
codumentor.log.YYYY-MM-DDdaily rotation, kept ~30 files
codumentor.json, codumentor.json.YYYY-MM-DDlegacy JSON era (ended 2026-05-15); orphaned by rotation, never auto-deleted — still valuable history

Sizes range from ~100 KB to 139 MB per day per instance. Never read a whole
file blindly; see §5.

2. The three format eras

The same instance's history spans up to three formats. Any longitudinal
analysis must handle all three.

Era 1 — JSON lines (until 2026-05-15, codumentor.json*):

{"timestamp": "2026-05-15T09:24:35.093", "level": "INFO", "logger": "codumentor.api",
 "message": "...", "module": "main", "function": "lifespan", "line": 447,
 "exception": "Traceback ...", "context": {"session_id": "c_937b96..."}}

Richest format: module/function/line, structured exception, full
session id. Parse with json.loads per line (skip non-{ lines).

Era 2 — plain text (2026-05 onward):

2026-07-06 10:40:19,067  - api - INFO - Shutting down Codumentor API server...

Era 3 — text with context prefix (mixed into era-2 files, session-scoped
records):

2026-07-07 08:23:39,735 > session_id=c_a27b741af82c4ae... tool=subagent - telemetry - INFO - Tool read_file ms=10.1

Caveats: context field order is not stable (session_id=... tool=... and
tool=... session_id=... both occur); the session id is **truncated with a
literal ...** — treat it as a prefix, match with grep -F 'c_a27b741', and
resolve full ids via audit.db if needed.

One regex parses eras 2 and 3 (battle-tested against the full corpus):

TEXT_RE = re.compile(
    r"^(\d{4}-\d{2}-\d{2}) (\d{2}:\d{2}:\d{2}),\d{3}\s+(?:>\s+(.*?)\s+)?-\s+([\w.]+)\s+-\s+"
    r"(DEBUG|INFO|WARNING|ERROR|CRITICAL)\s+-\s+(.*)$"
)   # groups: date, time, context?, logger, level, message

Continuation lines: anything not matching the regex belongs to the previous
record — tracebacks, multi-line payloads, raw subprocess stderr. Up to ~30 % of
all lines in agent-heavy instances. A line-oriented grep therefore misses most
traceback content, and a "records per level" count is not a "lines" count.

3. Timestamps and rotation

4. Logger names drift across eras

The same module has logged under three namespaces over time:
codumentor.plugins.branch_switching → plugins.branch_switching →
src.codumentor.plugins.branch_switching (also
codumentor.src.codumentor.plugins...). Grep by suffix, not full name:

grep -E ' - [a-z_.]*branch_switching - ' codumentor.log*

5. Volume safety rules (agents: read this twice)

  1. Never cat/Read a whole log file — files reach 139 MB. Use grep -m, head/tail, byte-capped head -c, or the bundled scanner.
  2. Log lines can be tens of KB: the models, prompt, tools, and workspace_isolation.strategies loggers embed full LLM payloads, prompts, and tool results (entire source files) in single records. Pipe grep output through cut -c1-300 before eyeballing.
  3. Payloads poison naive greps. The logs contain source code and documents, which match almost any keyword. Anchor patterns to record structure — the level token with its separators — not bare words:

``bash
grep -c ' - ERROR - ' file # good: counts ERROR records
grep -c ERROR file # bad: also matches payload content
``

The same trap applies to journalctl output ("Started" inside a payload is
not a service start; filter with --identifier systemd or -o cat +
^Started ).

  1. Rough era-2/3 fleet baseline (audit of 2026-07): DEBUG ≈ 85 % of records; models alone ≈ 43 % of bytes. ERROR records are rare (≈ 0.04 %) — scanning only errors/warnings is cheap even fleet-wide (~15 s for 2.7 GB).

6. Standard analysis method

For anything beyond a single lookup, do a signature-clustering pass first,
then sample clusters with context. This turns millions of lines into a few
hundred signatures.

The scanner used for the 2026-07-07 audit is bundled as a project skill:

python3 .claude/skills/reading-instance-logs/scripts/scan_logs.py \
    --base /home/codumentor --out /tmp/scan_result.json

It handles all three eras, counts levels per instance, clusters
ERROR/WARNING/CRITICAL by normalized signature (uuids/hex/paths/emails/IPs/
numbers/long-quoted-strings → placeholders), extracts traceback exception
types, builds a per-day error timeline, and attributes bytes per logger.
Read the printed summary, then drill into scan_result.json.

Then pull context per cluster:

grep -n -m3 -B5 -A25 'distinctive message fragment' logs/codumentor.log.2026-07-02 | cut -c1-300

Quick recipes:

# ERROR records today, by logger + message head
grep ' - ERROR - ' logs/codumentor.log | grep "^$(date +%F)" \
  | sed -E 's/^[^-]+- ([a-z_.]+) - ERROR - (.{0,60}).*/\1 :: \2/' | sort | uniq -c | sort -rn

# everything for one session (id may be truncated in the log — use the prefix)
grep -F 'session_id=c_a27b741' logs/codumentor.log | cut -c1-300

# true service starts/stops (not payload noise)
journalctl --user -u codumentor-api-<name>.service -o cat --identifier systemd \
  | grep -E '^(Started|Stopped) '

# a JSON-era error with its structured exception
grep -m1 '"level": "ERROR".*fragment' logs/codumentor.json.2026-05-13 | python3 -m json.tool

7. Where else to look (the log is not the whole story)

SourceWhat's there
journalctl --user -u codumentor-api-<name>duplicate of app stdout (full DEBUG firehose) + systemd lifecycle events; survives log deletion
~/.codumentor/setup-uppers/<repo>/<hash>/logworkspace-isolation project-setup output; the main log only records Setup failed (rc=N); see log at <path> — quarantined attempts get suffix .failed-<epoch>
~/codumentor-<name>/audit.dbSQLite audit trail (full session ids, user actions)
~/codumentor-<name>/logs/ after CLI runsingest.sh runs the CLI against the same config → a second process writes the same codumentor.log; interleaved records from two pids are possible, and rotation can race
/var/log/auth.logssh activity incl. the LLM tunnel

8. Known interpretation gotchas

9. Fleet-wide error sweep, start to finish

# 1. inventory (and ghost check)
ls -d ~/codumentor-*/logs
systemctl --user list-units 'codumentor-api-*'

# 2. cluster everything (~15 s per 2.7 GB)
python3 .claude/skills/reading-instance-logs/scripts/scan_logs.py --base ~ --out /tmp/scan.json

# 3. read: per-instance level counts, top signatures, incident days
#    (script prints these; /tmp/scan.json has the full data)

# 4. sample each significant cluster with grep -B/-A (±25 lines, cut -c1-300)

# 5. cross-check systemd/journal for restarts, disk (df -h), journal size
#    (journalctl --disk-usage), and OOM (journalctl -k | grep -i oom)

# 6. write findings with: signature, count, first/last seen, instances,
#    severity, sample line, root cause if determined