Codumentor logo Codumentor

Reading Instance Logs

A field guide to reading and analyzing the logs that Codumentor instances have
already written — formats, eras, pitfalls, and scan tooling. Written primarily
for agents (and humans) doing log forensics on a production host. For
configuring logging, see Monitoring. For a worked example of
a full-fleet analysis, see the
2026-07-07 log audit.

Concrete paths below describe the ardinsys production host (user codumentor);
adapt the pattern for other deployments.

1. Where logs live

Each instance is a directory ~/codumentor-<name>/ containing logs/,
its YAML config, audit.db, and a codumentor symlink to the shared checkout.
Each runs as a systemd user service codumentor-api-<name>.service.

ls -d ~/codumentor-*/logs                          # instances on disk
systemctl --user list-units 'codumentor-api-*'     # instances actually running

Always compare those two lists. They can disagree in both directions:
a directory with no service (dead instance), or a running service whose
directory was deleted (ghost instance — it still serves its port and writes
logs to deleted inodes; check ls -l /proc/<MainPID>/cwd and
lsof -p <pid> | grep deleted).

Inside logs/:

FileWhat it is
codumentor.logcurrent log (text format)
codumentor.log.YYYY-MM-DDdaily rotation, kept ~30 files
codumentor.json, codumentor.json.YYYY-MM-DDlegacy JSON era (ended 2026-05-15); orphaned by rotation, never auto-deleted — still valuable history

Sizes range from ~100 KB to 139 MB per day per instance. Never read a whole
file blindly; see §5.

2. The three format eras

The same instance's history spans up to three formats. Any longitudinal
analysis must handle all three.

Era 1 — JSON lines (until 2026-05-15, codumentor.json*):

{"timestamp": "2026-05-15T09:24:35.093", "level": "INFO", "logger": "codumentor.api",
 "message": "...", "module": "main", "function": "lifespan", "line": 447,
 "exception": "Traceback ...", "context": {"session_id": "c_937b96..."}}

Richest format: module/function/line, structured exception, full
session id. Parse with json.loads per line (skip non-{ lines).

Era 2 — plain text (2026-05 onward):

2026-07-06 10:40:19,067  - api - INFO - Shutting down Codumentor API server...

Era 3 — text with context prefix (mixed into era-2 files, session-scoped
records):

2026-07-07 08:23:39,735 > session_id=c_a27b741af82c4ae... tool=subagent - telemetry - INFO - Tool read_file ms=10.1

Caveats: context field order is not stable (session_id=... tool=... and
tool=... session_id=... both occur); the session id is **truncated with a
literal ...** — treat it as a prefix, match with grep -F 'c_a27b741', and
resolve full ids via audit.db if needed.

One regex parses eras 2 and 3 (battle-tested against the full corpus):

TEXT_RE = re.compile(
    r"^(\d{4}-\d{2}-\d{2}) (\d{2}:\d{2}:\d{2}),\d{3}\s+(?:>\s+(.*?)\s+)?-\s+([\w.]+)\s+-\s+"
    r"(DEBUG|INFO|WARNING|ERROR|CRITICAL)\s+-\s+(.*)$"
)   # groups: date, time, context?, logger, level, message

Continuation lines: anything not matching the regex belongs to the previous
record — tracebacks, multi-line payloads, raw subprocess stderr. Up to ~30 % of
all lines in agent-heavy instances. A line-oriented grep therefore misses most
traceback content, and a "records per level" count is not a "lines" count.

3. Timestamps and rotation

4. Logger names drift across eras

The same module has logged under three namespaces over time:
codumentor.plugins.branch_switching → plugins.branch_switching →
src.codumentor.plugins.branch_switching (also
codumentor.src.codumentor.plugins...). Grep by suffix, not full name:

grep -E ' - [a-z_.]*branch_switching - ' codumentor.log*

5. Safety rules (agents: read this twice)

  1. Never cat/Read a whole log file — files reach 139 MB. Use grep -m, head/tail, byte-capped head -c, or the bundled scanner.
  2. Log lines can be tens of KB: the models, prompt, tools, and workspace_isolation.strategies loggers embed full LLM payloads, prompts, and tool results (entire source files) in single records. Pipe grep output through cut -c1-300 before eyeballing.
  3. Payloads poison naive greps. The logs contain source code and documents, which match almost any keyword. Anchor patterns to record structure — the level token with its separators — not bare words:

``bash
grep -c ' - ERROR - ' file # good: counts ERROR records
grep -c ERROR file # bad: also matches payload content
``

The same trap applies to journalctl output ("Started" inside a payload is
not a service start; filter with --identifier systemd or -o cat +
^Started ).

  1. Never open a live instance's SQLite databases — not even read-only. The .db/.sqlite3 files under ~/codumentor-<name>/data/ are WAL mode, and ?mode=ro is not read-only at the filesystem level: SQLite creates the -wal and -shm sidecars when they are absent, because read-only WAL access still needs the shared-memory index. Opened from your own account they are created with your uid and umask, and the service loses write access to its own database (§8). Copy first, query the copy:

``bash
cp ~codumentor/codumentor-<name>/data/agent_storage/chat_history.db /tmp/x.db
sqlite3 /tmp/x.db 'select ...'
``

The log usually answers the question anyway — reach for the DB only when it
genuinely cannot.

  1. Rough era-2/3 fleet baseline (audit of 2026-07): DEBUG ≈ 85 % of records; models alone ≈ 43 % of bytes. ERROR records are rare (≈ 0.04 %) — scanning only errors/warnings is cheap even fleet-wide (~15 s for 2.7 GB).

6. Standard analysis method

For anything beyond a single lookup, do a signature-clustering pass first,
then sample clusters with context. This turns millions of lines into a few
hundred signatures.

The scanner used for the 2026-07-07 audit is bundled as a project skill:

python3 .claude/skills/reading-instance-logs/scripts/scan_logs.py \
    --base /home/codumentor --out /tmp/scan_result.json

It handles all three eras, counts levels per instance, clusters
ERROR/WARNING/CRITICAL by normalized signature (uuids/hex/paths/emails/IPs/
numbers/long-quoted-strings → placeholders), extracts traceback exception
types, builds a per-day error timeline, and attributes bytes per logger.
Read the printed summary, then drill into scan_result.json.

Then pull context per cluster:

grep -n -m3 -B5 -A25 'distinctive message fragment' logs/codumentor.log.2026-07-02 | cut -c1-300

Quick recipes:

# ERROR records today, by logger + message head
grep ' - ERROR - ' logs/codumentor.log | grep "^$(date +%F)" \
  | sed -E 's/^[^-]+- ([a-z_.]+) - ERROR - (.{0,60}).*/\1 :: \2/' | sort | uniq -c | sort -rn

# everything for one session (id may be truncated in the log — use the prefix)
grep -F 'session_id=c_a27b741' logs/codumentor.log | cut -c1-300

# true service starts/stops (not payload noise)
journalctl --user -u codumentor-api-<name>.service -o cat --identifier systemd \
  | grep -E '^(Started|Stopped) '

# a JSON-era error with its structured exception
grep -m1 '"level": "ERROR".*fragment' logs/codumentor.json.2026-05-13 | python3 -m json.tool

7. Where else to look (the log is not the whole story)

SourceWhat's there
journalctl --user -u codumentor-api-<name>duplicate of app stdout (full DEBUG firehose) + systemd lifecycle events; survives log deletion
~/.codumentor/setup-uppers/<repo>/<hash>/logworkspace-isolation project-setup output; the main log only records Setup failed (rc=N); see log at <path> — quarantined attempts get suffix .failed-<epoch>
~/codumentor-<name>/audit.dbSQLite audit trail (full session ids, user actions)
~/codumentor-<name>/data/instance state as SQLite: agent_storage/chat_history.db (conversations + messages), ui_events.sqlite3, jobs.sqlite3, per-plugin DBs. Read only via a copy — §5 rule 4
~/codumentor-<name>/logs/ after CLI runsingest.sh runs the CLI against the same config → a second process writes the same codumentor.log; interleaved records from two pids are possible, and rotation can race
/var/log/auth.logssh activity incl. the LLM tunnel

8. Known interpretation gotchas

``bash
find ~codumentor/codumentor-*/ ! -user codumentor -printf '%M %u:%g %p\n'
``

A -wal/-shm owned by anyone but the service user — usually left behind by
an admin's read-only probe (§5 rule 4) — drops the service into the file's
"other" class, and every write fails. Fix needs no root, since the admin
account is in group codumentor:

``bash
chgrp codumentor <db>-wal <db>-shm && chmod 664 <db>-wal <db>-shm
``

No restart required, and ownership self-heals: the next clean close deletes
the sidecars and the following open recreates them as the service user.
Never rm the sidecars of a live database — unlinking a mapped -shm
lets a second opener create a fresh one while the first still uses the old
mapping, which is a genuine corruption route. Symptom in the UI is "The agent
stopped due to a fatal error"; in the log it is a storage - ERROR pair *per
message* plus agent - ERROR - Error in streaming agent loop, so record
counts wildly overstate root causes (2026-09-05 on selfdev-stage: 85 records,
3 conversations, one mis-owned file).

9. Fleet-wide error sweep, start to finish

# 1. inventory (and ghost check)
ls -d ~/codumentor-*/logs
systemctl --user list-units 'codumentor-api-*'

# 2. cluster everything (~15 s per 2.7 GB)
python3 .claude/skills/reading-instance-logs/scripts/scan_logs.py --base ~ --out /tmp/scan.json

# 3. read: per-instance level counts, top signatures, incident days
#    (script prints these; /tmp/scan.json has the full data)

# 4. sample each significant cluster with grep -B/-A (±25 lines, cut -c1-300)

# 5. cross-check systemd/journal for restarts, disk (df -h), journal size
#    (journalctl --disk-usage), and OOM (journalctl -k | grep -i oom)
#    plus stray file ownership under the instances (see 8):
find ~/codumentor-*/ ! -user "$(id -un)" -printf '%M %u:%g %p\n'

# 6. write findings with: signature, count, first/last seen, instances,
#    severity, sample line, root cause if determined