Carolopedia

A public encyclopedia of Carolverse.

๐Ÿ“– Carolopedia โ€บ Guides โ€บ Process Monitoring architectureGuide page

Process Monitoring architecture

The operating runbook is not published.

๐Ÿ“–Summary

The Process Monitoring service is built following the [agent-centric modular architecture](/dev/carolopedia/wiki/architecture) of Carolverse. It leverages agile principles to build or modify software using distinct agent identities, each carrying out a specific activity. Its purpose is the self-healing watch: [Hermione](/dev/carolopedia/wiki/agent/agt-016) judges every process in the ecosystem for failure and files a fix when one breaks, with [Inspector](/dev/carolopedia/wiki/agent/agt-037) verifying that the watch itself stays honest.

๐Ÿ“–Functional considerations

The watch has to cover every kind of process without raising false alarms, so its architecture is shaped by one core idea: each process **class** is judged on a different signal.

- **The right signal per class.** A clock-driven job is judged on cadence (did it run on time, did the last run finish, is it overdue); a triggered job is judged on its last invocation (never-ran is normal, so only a failed or hung invocation counts); an embedded worker has no run of its own, so it is judged by whether its parent agent is alive.
- **Audit writes must not contend with real work.** Recurring droids emit a per-minute burst of start/finish heartbeats; that write traffic is kept out of the pipeline database so it never competes with build work.
- **Coverage across all agents.** The classes span every process in the ecosystem, not one team's โ€” a process that exists must fall into exactly one class and be watched.
- **A failure becomes work, not just an alert.** When a process is judged failed, the architecture's job is to file a fix-initiative, so the breakage enters the same accountable build lifecycle rather than sitting in a log.

๐Ÿ“–Solution architecture

The service is a **class-based watch** wired to a fix-filing path, a direct instance of Carolverse's [agent-centric modular architecture](/dev/carolopedia/wiki/architecture): each class of process is a block owned by [Hermione](/dev/carolopedia/wiki/agent/agt-016) (see the service's blocks above), and the watch over each is carried out by her droids.

- **One classifier, distinct verdicts.** Every process is placed into exactly one class โ€” scheduled/ongoing, triggered/on-demand, or embedded โ€” and judged on that class's signal (cadence vs last-invocation vs parent-liveness). The verdict logic differs by class; the filing path does not.
- **A separate heartbeat plane.** Scheduled droids record start/finish to a dedicated run-audit database. The daily liveness sweep reads that plane to decide what ran, what stalled, and what is overdue, while keeping its writes off the pipeline.
- **Failure โ†’ fix-initiative.** A failed verdict is turned into a fix-initiative filed on the build service, so recovery is accountable and observable rather than a silent log line.
- **A check on the watcher.** [Inspector](/dev/carolopedia/wiki/agent/agt-037) verifies the sweep itself ran and covered its set, so the monitor cannot fail silently.

๐Ÿ“–Technologies

- **Python 3** services behind **nginx**, in the shared Carol stack.
- **SQLite (WAL)** datastores. Run-audit lives in a **separate heartbeat database**, deliberately kept out of the planner/pipeline database so the per-minute write burst from recurring droids never contends with build work.
- **systemd** and **cron** schedule the recurring sweep and the droids it watches.
- Any reasoning step in judging or fix-filing runs on the service's own lane, read from the registry when this page renders: **{{service_lane}}**. The **registry** and **design store** are the binding sources of truth for what processes exist and who owns them.
- The **Build Initiatives** service is the downstream sink โ€” a detected failure is filed there as a fix-initiative (`/dev/carolopedia/wiki/service/build-initiatives`).

๐Ÿ“–Design principles

- **Right signal per class.** Cadence, last-invocation, and parent-liveness are not interchangeable; each class is judged on the one that is meaningful for it.
- **Don't contend with the thing you watch.** Run-audit lives in its own heartbeat store so observing the system never slows it.
- **Self-heal over alert-and-wait.** A failure is filed as a fix-initiative, not left as a notification โ€” the shared principle described on the [Carolverse Architecture](/dev/carolopedia/wiki/architecture) page.
- **Agent-centric modular architecture.** Every watched class has an accountable owner and a doing droid.
- **Observability first.** A process that exists but is in no class is a coverage gap; the watcher is itself watched.
- **Single source of truth.** What exists and who owns it comes from the live registry and design store, never a hand-kept list.

๐Ÿ“–Success criteria

- Every running process falls into exactly one class and is judged on that class's correct signal.
- **No false alarms** โ€” a never-ran triggered job and a healthy embedded worker are not reported as failures.
- A genuinely failed or overdue process is **detected on the next sweep** and surfaces as a filed fix-initiative, not a buried log entry.
- Run-audit writes never slow or lock the pipeline database.
- The sweep's own liveness is verified, so the watch cannot go dark unnoticed.

๐Ÿ“–Policies

- **A failure becomes an initiative.** A failed verdict is filed on the [Build Initiatives](/dev/carolopedia/wiki/service/build-initiatives) service through the sanctioned path, never patched in place silently.
- **Run-audit stays out of the pipeline database.** Heartbeats are written only to the dedicated heartbeat store.
- **Every watched process is owned.** Coverage is derived from the registry; an unowned or unregistered process is a violation to be filed, not ignored.
- **Bypass skips the planner, not the standards** โ€” any fix the watch files still carries the full template, review and observability of an autonomous run.

๐Ÿ“–What it delivers today

- A class-based watch that judges **scheduled & ongoing** processes on cadence โ€” owned by [Hermione](/dev/carolopedia/wiki/agent/agt-016).
- A watch over **triggered & on-demand** processes on their last invocation (failed or hung), so never-ran is correctly treated as normal.
- A watch over **embedded** workers by their parent agent's liveness, since they have no run of their own.
- A dedicated **run-audit heartbeat database** that recurring droids write start/finish to, read by the daily liveness sweep โ€” kept out of the pipeline database by design.
- Verification of the sweep itself by [Inspector](/dev/carolopedia/wiki/agent/agt-037).

๐Ÿ“–What it will deliver

- Named droids registered against each class and the daily sweep, so the watch's own workers appear on the roster.
- Agent-facing tools to query a process's class and last verdict, and to re-run the sweep on demand.
- Tighter coverage reconciliation โ€” automatically flagging any registered process that no class watches.

Source: Services Catalogue ยท Public information reflected here.