Carolopedia

A friendly guide to Carol, her ecosystem, and the agents who built her.

📖 CarolopediaServicesBuild InitiativesAll activitiesINI-999901536Guide page
📋

CAROL-INI-2894-00: Pipeline uptime RSI measure: % of time the pipeline is RUNNING vs stalled, target 100%

Initiative
Open in Initiatives →

📖About

Ninad ruling 2026-07-16: the pipeline gets a SECOND RSI measure — UPTIME. It measures the share of time the build pipeline was RUNNING vs STALLED; the target is 100% (the pipeline should always be running). Definition of a STALLED sample: the dispatch breaker/switch is paused, OR dispatched work is waiting while nothing is executing. A RUNNING sample: something is executing, or nothing is waiting (an idle-but-available pipeline is healthy). Implementation: a 5-minute sampler droid under Elrond (el-uptime-01, cron via the audit wrapper so every tick is run-audited) records each sample with its reason and maintains a daily score (running samples / total samples) in Prometheus's quality store; the measure is added to the ONE metric catalogue (definition, target 100%, comparator gte, owner Elrond, home service initiatives) so the RSI Dashboard and the Quality Scorecard master hub pick it up per the CAROL-INI-2749 governance. Droid registered in the registry so Hermione's process sweep monitors the sampler itself.

⚖️Decisions

  • Elrond's bypass methodology checklist (a reminder, not a gate -- you've got this): 0. File it requested_mode='bypass' (planner-vs-bypass is a deliberate choice). bypass_start REFUSES a non-bypass initiative (CAROL-INI-1846), and the dispatcher only skips the bypass lane when the mode says bypass -- a 'planner' mistag lets Merlin's pipeline grab the placeholder step and block your finished work. 1. Filed as planned status -- let the bypass claim/activate it; never file active. 2. Open the bypass (bypass_start) with your droid id + the remediation answer (remediates_initiative_id=NNN, or remediates_nothing=True). 3. Work the blocks for your work-type: template -> design -> code -> test -> review. Do the real work; record decisions on the initiative as you make them. 4. Reality is recorded for you at close -- code (files changed), each decision, and the twin-review verdict become real activities tied to this initiative and show in the Activity Tracker like a planner run (CAROL-INI-1840). No dummy rows. 5. Keep the initiative status moving; it parks in 'reviewing' and is tagged uat-pending for you at close (CAROL-INI-1836), so the stuck-watchdog leaves it alone until UAT. 6. Close runs the gates (design/architecture compliance + caller-audit). If a gate flags something pre-existing or unrelated to your change, waive it with a clear written rationale -- audit, don't skip. 7. Bypass skips the planner's auto-orchestration, NOT the standards. Same template checklist, same review, same observability as a planner run. (elrond)
  • [status-router] planned -> executing | event=bypass_executing | bypass transition (or-bx-01)
  • [status-router] executing -> reviewing | event=bypass_reviewing | bypass transition (or-bx-01)
  • UAT rework (Ninad, 2026-07-16 round 2): opening the Pipeline uptime measure crashed the dashboard (trend endpoint returned null — no drilldown branches existed). Added the trend line (Running %, target 100), the weekly drilldown (running vs stalled samples per day) and the day drilldown (per-hour running %) to the hub + metric module; apps bounced via Radagast; all three verified over HTTP. The measure is already showing truth: only 2 of 9 samples so far today found the pipeline running. (orion)
  • UAT rework (Ninad, 2026-07-16 round 3): chart follows the house convention — line = uptime score out of 10 (10 = running 100% of the time, target 10), bars = initiatives that executed that week; tile unit switched to /10 to match. Verified live over HTTP after the Radagast bounce. (orion)
  • UAT rework (Ninad, 2026-07-16 round 4): trend spans the full fixed 10-week window like every other measure; weeks with no samples assume score 10 (no evidence of a stall). Verified live: 10 weeks 2026-05-11..2026-07-13, line 10.0 x9 then 1.8 (current week, measured), bars = initiatives executed per week across the whole span. (orion)
  • UAT rework (Ninad, 2026-07-16 round 5): (1) bars render now — the template needs [ok, volume] count PAIRS (it reads c[1] for bar height); plain ints drew nothing. (2) Weekly score = average of PER-DAY scores with elapsed no-data days scoring 10 and future days excluded — this week reads 7.9 (Mon+Tue assumed 10, today measured low). Todays low reading is REAL: since midnight the samples found 8 dispatched initiatives waiting with nothing executing. (orion)
  • UAT rework (Ninad, 2026-07-16 round 6): dashboard refresh was ~2-5s because the catalogue endpoint recomputed every measure live per page load (process_success alone scores ~280 processes, ~3.5s — pre-existing, surfaced by the uptime UAT reloads). Added a 60s catalogue cache; first load computes, repeats serve in milliseconds. (orion)
  • [status-router] reviewing -> closed | event=operator_signoff | Auto-accepted (CAROL-INI-1859): Orion-initiated, >2 days in reviewing with no objection. (el-srac-01)

Success criteria

  • An operator opening the RSI Dashboard or Quality Scorecard sees a Pipeline uptime measure with target 100% and a live value computed from real samples (must_have)
  • Every 5 minutes a sample records whether the pipeline was running or stalled, and each stalled sample names its cause (breaker paused / work waiting with nothing executing) (must_have)
  • The sampler is a registered, run-audited process visible to the process-health sweep, so a dead sampler is itself detected (must_have)