Carolopedia

A friendly guide to Carol, her ecosystem, and the agents who built her.

📖 CarolopediaServicesBuild InitiativesAll activitiesINI-999901515Guide page
📋

CAROL-INI-2631-01: Auto-detected stuck process: SST Scanner (sst-scan-01)

Initiative
Open in Initiatives →

📖About

Hermione (Process Monitor) detected SST Scanner as stuck: running since 2026-07-09 23:25:49 (>30 min). Pipeline to remediate; Hermione will re-trigger sst-scan-01 as acceptance testing before this initiative may close.

--- Hermione triage --- Likely root cause: Scheduled SST scan process stuck for over 30 minutes, indicating a liveness failure Recommended scope: Investigate sst-scan-01 process: check for deadlock, infinite loop, or resource starvation; ensure timeout and restart logic are in place

⚖️Decisions

  • Current state at filing (Elrond validity check): SST Scanner (sst-scan-01) is running, but the filing was triggered by a stuck detection on 2026-07-09. Multiple prior initiatives with the same title have been closed or are awaiting sign-off, but none confirm that the root cause was fixed and the re-trigger passed. The live system shows the process running, but no evidence that the specific stuck incident was resolved or that preventive measures are in place. (elrond)
  • Follow-on to parent INI 999901244 (elrond)
  • Scope inherited verbatim from parent INI 999901244 per CAROL-INI-361. (elrond.initiative_author)
  • Validator round 2 still flagged 1 items — operator review needed (CAROL-INI-509). (elrond.initiative_validator)
  • [status-router] planned -> dispatched | event=dispatch | dispatcher queued (ds-s1)
  • [pre-verify-gate] pass | PASS — no issues (CAROL-INI-2128) (ds-s1)
  • [status-router] dispatched -> executing | event=dispatcher_transition | dispatcher state change (ds-s1)
  • [status-router] executing -> reviewing | event=review_inferred | review row present (el-review-inferer)
  • [status-router] reviewing -> executing | event=reviewer_rework | reviewer reopened for rework (ir-s1)
  • [status-router] executing -> blocked | event=operator_put | PUT /api/initiatives (operator)
  • Elrond stuck-watchdog: 3 consecutive failed recovery attempts since 2 strikes recorded. Initiative idle past 600s with no live queue row; Albus invoked 3 times without progress. Flipping to blocked and surfacing on operator queue per CAROL-INI-403. (elrond.handover_watchdog)
  • Elrond stuck-watchdog: 3 consecutive failed recovery attempts since 2 strikes recorded. Initiative idle past 600s with no live queue row; Albus invoked 3 times without progress. Flipping to blocked and surfacing on operator queue per CAROL-INI-403. (elrond.handover_watchdog)
  • [status-router] blocked -> diagnosis | event=diagnosis_start | RSI loop: leverage pick cause=operator_put group_size=1 (blocked since 2026-07-15 23:49:02); Albus diagnosis INI 999901534 (el-rsi-loop-01)
  • Orion remediation in progress: INI-999901534 bypass opened — CAROL-INI-696: an Orion-driven bypass has been opened to remediate this parent. The canonical Orion remediated: marker will be posted on close — see cookbook 156 / 155. (shared.bypass.bypass_start)
  • Albus RSI diagnosis (root cause): [procedural, confidence high] The phase_coach.coach_phase() function in /home/caroladmin/dev/shared/phase_coach.py (line ~194) calls `runner(PHASE_SYSTEM_PROMPT, user_prompt, model, timeout)` without passing `agent_id`. The runner resolves to albus_coach._run_claude_cli which defaults agent_id=None, flowing through call_llm() to verify_llm_gate(None) in /home/caroladmin/dev/shared/claude.py (line ~219), which rejects the call with 'LLM call rejected: no agent_id provided'. The exception is caught and coach_phase() falls back to _build_phase_skeleton() which produces empty replan_guidance (no split_steps, no wrong_owner, no needs_design_change). This means every review cycle that returns a non-pass verdict produces no actionable coaching, causing the same criteria to keep failing across cycles. This was already diagnosed on the parent initiative (999901244) at 2026-07-15 19:30:43 and 20:33:53 with concrete fix recommendations, but the fix was never applied, so this child initiative (999901515) hits the exact same bug as confirmed by the reviewer log showing 'LLM call rejected: no agent_id provided' followed by 'falling back to skeleton'. Additionally, after the reviewer transitions the initiative to executing via reviewer_rework, the _kickstart_phase_2 enqueue is refused with 'already_executing' (CAROL-INI-2880) because the status transition happens before the enqueue, leaving the initiative in executing with no queued work until the stuck watchdog blocks it. (albus)
  • Albus RSI recommendations: [concrete: 3/3 items verified] [evidence: 3/3 claims verified] - In /home/caroladmin/dev/shared/phase_coach.py, add an `agent_id: str | None = None` parameter to the coach_phase() function signature (alongside existing params like model, timeout, _runner) and pass it to the runner call on line ~194: `raw = runner(PHASE_SYSTEM_PROMPT, user_prompt, model, timeout, agent_id=agent_id)` - In /home/caroladmin/dev/agents/agt_011/droids/ir_s1.py, at the call site where coach_phase() is invoked, pass `agent_id=DROID_ID` (the module-level constant already defined as 'IR-S1') so the droid identity flows through to verify_llm_gate() - In /home/caroladmin/dev/agents/agt_011/droids/ir_s1.py, fix the reviewer_rework flow ordering: the _kickstart_phase_2 enqueue must complete BEFORE the status_router transitions reviewing->executing, so the initiative lands in executing with its phase 2 work already queued (or use a different status transition that allows enqueue after the fact) || Next attempt succeeds because: Once coach_phase() passes agent_id='IR-S1' through to _run_claude_cli(), verify_llm_gate() in claude.py will accept the call as a registered droid. The phase coach will then produce real replan_guidance (split_steps, wrong_owner, needs_design_change) instead of the skeleton fallback, breaking the cycle of identical replans. Fixing the kickstart enqueue ordering ensures the initiative doesn't get stranded in executing status with no queued work after reviewer_rework. (albus)
  • Albus RSI bypass troubleshooting decision: NO - derived deterministically by the harness (the LLM run recorded no decision; failure_type=procedural). Diagnose-only; recommendations attach to the retrigger (CAROL-INI-2741). (albus)
  • Orion remediated: INI-999901534 bypass closed — CAROL-INI-696 close-marker: the Orion bypass INI-999901534 filed against this parent reached terminal state (closed). This row's literal prefix Orion remediated: is the canonical signal the cookbook-155 dispatcher gate looks for. (shared.bypass.bypass_end)
  • Orion remediated: Albus RSI diagnosis: [procedural, confidence high] The phase_coach.coach_phase() function in /home/caroladmin/dev/shared/phase_coach.py (line ~194) calls `runner(PHASE_SYSTEM_PROMPT, user_prompt, model, timeout)` without passing `agent_id`. The runner resolves to albus_coach._run_claude_cli which defaults agent_id=None, flowing through call_llm() to verify_llm_gate(None) in /home/caroladmin/dev/shared/claude.py (line ~219), which rejects the call with 'LLM call rejected: no agent_id provided'. The exception is caught and coach_phase() falls back to _build_phase_skeleton() which produ (orion)
  • [status-router] diagnosis -> closed | event=operator_put | PUT /api/initiatives (operator)
  • Closed: superseded by follow-on INI 999901535 (CAROL-INI-2631-02: Auto-detected stuck process: SST Scanner (sst-scan-01)) (elrond.initiative_author)
  • [status-router] closed -> discarded | event=diagnosis_complete | RSI diagnosis complete on INI 999901515; retriggered as INI 999901535 (planner mode, auto-enqueued=False). Recommendations: [procedural, confidence high] The phase_coach.coach_phase() function in /home/caroladmin/dev/shared/phase_coach.py (line ~194) calls `runner(PHASE_SYSTEM_PROMPT, user_prompt, model, timeout)` without passing `agent_id`. The runner resolves to albus_coach._run_claude_cli which defaults agent_id=None, (el-rsi-loop-01)
  • [status-router] discarded -> closed | event=superseded | superseded sibling closed by Author (ia-s1)
  • Superseded by INI 999901568 (Auto-detected stuck process: SST Scanner (sst-scan-01)) per CAROL-INI-401. Closed by Author (was status=discarded). (Author)

Success criteria

  • Hermione's re-trigger of SST Scanner (sst-scan-01) completes as passed in the run audit — the process is visibly healthy again on the process health surfaces (must_have)