Fabrik Issue State Machine

Every issue in Fabrik follows a defined lifecycle: from intake through a series of AI-driven stages (Specify → Research → Plan → Implement → Review → Validate → Done), with automated gates at key transitions. The diagram below shows the happy path at a glance.

Fabrik issue lifecycle: linear pipeline from Specify through Done with review, CI, and merge-conflict gates
Fabrik issue lifecycle — linear pipeline with gate annotations. Review Gate holds advancement until all PR reviewers submit; CI Gate holds until checks pass; Merge Gate holds until rebase conflicts are resolved.

Not an engineer? The diagram and the Pipeline Overview table are the fastest way to understand Fabrik’s workflow.

Engine contributor or debugger? The dense reference below covers every reachable state, every label mutation, every guard condition, and the visual state diagrams in §10. Use the State Enumeration section as the authoritative source when diagnosing unexpected engine behavior.


This document is the formal specification of Fabrik’s issue-level state machine: how an issue moves between states across multiple invocations of the engine. It covers every reachable state, every event that triggers a transition, every label mutation, and every guard condition.

Companion document: stage-lifecycle.md describes the per-invocation lifecycle (what happens before, during, and after a single Claude invocation). This document describes the cross-invocation state machine (how an issue progresses through the pipeline over time). They are complementary.

As-built specification: This document describes what the code actually does, not what it ideally should do. Discrepancies between intended and actual behavior are flagged with > **Bug?:** callout blocks.

Source of truth for: state enumeration, transition tables, label semantics, and guard conditions. Supersedes partial label references in CLAUDE.md.


Pipeline Overview

Issues traverse a linear pipeline of stages, each corresponding to a column on the GitHub Project board:

Specify → Research → Plan → Implement → Review → Validate → [Queued†] → Done
Stage Order Read-Only PostToPR CreateDraftPR MarkPRReady WaitForReviews CleanupWorktree
Specify 0 Yes No No No No No
Research 1 Yes No No No No No
Plan 2 Yes No No No No No
Implement 3 No Yes Yes Yes No No
Review 4 No Yes No Yes Yes* No
Validate 5 No Yes No No Yes* No
Queued† 6 — — — — — —
Done 99 N/A No No No No Yes

Specify additionally ships persist_spec: true (§7.16): it stays read-only, but the engine commits the issue-body spec to specs/<issue>-<slug>/spec.md.

* All flags in this table reflect the default stage configuration shipped in .fabrik/stages/. Each flag is opt-in per stage YAML and may differ in custom configurations. wait_for_reviews is enabled for Review and Validate in the defaults.

† Queued is a holding stage (holding_stage: true) active only when merge_train: on. Items enter Queued when attemptMergeOnValidate routes to advanceToQueued instead of GitHub auto-merge. Per-item dispatch is suppressed (itemMayNeedWork / itemNeedsWork return false for holding stages); the batch handler handleMergeTrainBatch snapshots the Queued items once per poll cycle via groupQueuedByRepoAndBase (engine/poll.go, #1648), which excludes closed and fabrik:paused members (see the poison-well guard under Merge-Train Red-Batch Bisection) and further partitions the remaining items by resolved base branch (baseBranchForItem) — one queuedRepoGroup per distinct (repo, base) pair actually present, not one group per repo. routeQueuedGroup runs once per partition and dispatches runMergeTrainWorker for each when its own batch is ready, so a repo with members on both the default branch and a base:<branch> label runs two fully independent trains simultaneously — each with its own pinned base SHA, trial branch, draft CI PR, and integration PR targeting its own base (ADR-1648, superseding ADR-1647’s exclusion of such members from batching entirely). On a green trial-branch CI result, landMergeTrainBatch opens a landing integration PR, merges it, advances each member from Queued → Done via advanceToNextStage, and closes their individual PRs and issues (ADR-059 D3). The Queued column must exist on the GitHub Project board when merge_train: on; it is validated at startup.


1. State Enumeration

A state is defined by the tuple (BoardColumn, ControllingLabelSet). Not every label combination is a valid state — only reachable combinations are enumerated.

1.1 Controlling Labels

These labels define distinct states (their presence changes what the engine does with an item):

Label Type Defines State?
fabrik:locked:<user> Lock Yes — gates processing by other instances. <user> is cfg.User in PAT mode; under GitHub App auth it is the instance’s App-derived lock identity <slug>-<hash> (§7.4)
fabrik:editing Mutex Yes — prevents stage dispatch during comment processing
fabrik:paused Pause Yes — blocks processing on active stages unless a human comment arrives (cleanup stages ignore comments entirely while paused; a Validate-stage item with an externally merged PR can still advance via runValidatePRTerminalAdvance regardless of this label). Also applied by the comment-processing circuit breaker (§4.6) after N non-advancing comment-processing invocations within a rolling window, or by the success-agnostic no-op comment cycle breaker (§4.7) after N consecutive no-progress comment-processing invocations for the same stage, regardless of window. Also suppresses deep-fetch admission (not just dispatch) for items with no new activity this poll — see Appendix B’s “Paused deep-fetch skip” note (#1379).
fabrik:awaiting-input Sub-pause Yes (with fabrik:paused) — blocked-on-input variant; also both comment-processing circuit breakers’ trip pairing (§4.6, §4.7)
fabrik:awaiting-review Gate Yes — review gate is active
fabrik:awaiting-ci Gate Yes — CI gate is active; waiting for CI checks to pass (checks may be running or have failed)
fabrik:rebase-needed Gate Yes — merge-conflict gate is active; PR is not mergeable against its base
fabrik:awaiting-done Gate Yes — a FABRIK_NO_WORK_NEEDED decision has been made; the Done board move and/or issue close is still outstanding and is retried every poll, independent of item.Status
fabrik:awaiting-placement Gate Yes — a spawned child’s initial project-board Status placement is still outstanding; retried every poll, independent of item.Status/stages.FindStage resolving a stage for the child’s current column
fabrik:awaiting-close Gate Yes — closeIssueIfNonDefaultBase’s explicit CloseIssue call (ADR-1096) failed — or, on the default base, GitHub’s Closes #N auto-close had not closed the issue by the landing (#1962); the close is still outstanding and is retried every poll, independent of item.Status
fabrik:awaiting-pr-ready Gate Yes, but does not suppress dispatch — markPRReady’s client.MarkPRReady call (§5.2, ADR-1582) failed non-transiently or exhausted its in-process retry; the mark-ready is still outstanding and is retried every poll, independent of item.Status. Unlike most of this table’s other Gate rows, the item continues advancing through later stages normally while this label is present (§6.22)
fabrik:nondefault-base-pr-noted Informational No — does not gate dispatch or suppress anything; marks that the one-time non-default-base linkage notice (§6.21, ADR-1649) has already been posted (or attempted) for this item
fabrik:awaiting-advance Gate Yes — a terminal advance (advanceToNextStage, called from advanceValidateTerminalItem’s merged-PR path or advanceConvergedPRToDone) failed to move the project-board Status forward — most commonly a missing target Status option; the advance is still outstanding and is retried every poll, independent of item.Status
fabrik:awaiting-runaway-alert Gate Yes — the merge-train runaway guard (ADR-059 D8) already paused this member (fabrik:paused + fabrik:awaiting-input applied), but its AddComment alert call failed; the alert is still outstanding and is retried every poll, independent of item.Status (ADR-1533)
fabrik:blocked Dependency Yes — blocked by open dependency issues
stage:<X>:in_progress Progress Yes — a stage invocation is active
stage:<X>:complete Completion Yes — stage finished successfully
stage:<X>:failed Failure Yes — stage exhausted retry limit
fabrik:revalidate Operator trigger Transient — consumed by the revalidate-scan loop; triggers Validate re-entry then removes itself

1.2 Modifier Labels (Guard Conditions)

These labels do not define distinct states but influence transition behavior:

Label Effect
fabrik:yolo Forces auto-advance; triggers GitHub native auto-merge at Validate; overrides auto_advance: false
fabrik:cruise Forces auto-advance without auto-merge; stops at Validate completion; cruise wins over yolo for both end-of-Validate decisions — the PR is not auto-merged and the issue does not advance to Done, even when yolo is also present
fabrik:auto-merge-enabled Engine-internal; marks that GitHub’s native auto-merge has been enabled on the linked PR; anchors the convergence budget start time; bypasses legacy merge/CI gates
fabrik:unrestricted Passes --dangerously-skip-permissions to Claude Code
fabrik:extend-turns Pre-grants a 2× turn budget for the first stage invocation and first comment processing invocation while present, with max_wall_time scaled by the same 2× factor for that invocation (§7.7, ADR-1206); persists across stages; removed only at the Done cleanup stage or manually; no-op when max_turns == 0 (stage) or always applies for comments since commentMaxTurns is never 0
model:<name> Selects a specific model for this issue (e.g., model:opus)
effort:<level> Overrides stage effort level (low, medium, high, max); highest wins
base:<branch> Overrides worktree base branch; falls back to default if not on remote (checked via an authoritative git ls-remote probe, not just the local clone — see §1.3); updates PR base if PR exists
review-authority:<mode> Overrides the stage’s configured review_authority (advisory/authoritative) for this issue only; only meaningful alongside wait_for_reviews: true. No label → stage config governs. One recognized label → it wins. Both present → resolves to authoritative (more restrictive), with a logged warning. Malformed/unknown suffix → ignored, with a logged warning, falls back to stage config. See §6.1.1 and ADR-1261
expected-reviewers:<mode> Overrides the stage’s configured expected_reviewers (none/declared) for this issue only; only meaningful alongside wait_for_reviews: true. No label → stage config governs (nil stays nil). none → resolves to &[]string{} (enables the FR-2 fast-advance path). declared → resolves to a fixed synthetic reviewer identity. Both present → resolves to declared (more restrictive — imposes waiting, unlike none’s immediate advance), with a logged warning. Malformed/unknown suffix → ignored, with a logged warning, falls back to stage config. See §6.1 and #1304
fabrik:sub-issue Informational; marks issue as created by Fabrik’s sub-issue spawn mechanism
fabrik:credited-pr:<N> Engine-internal; applied only by the two merge-train landing paths (landMergeTrainBatch, landSingleton) alongside fabrik:awaiting-landing-verification, recording the integration/singleton PR credited for the Done transition — never applied by the ordinary auto-merge path, whose credited PR is always rediscoverable via FetchLinkedPR. See §6.19

1.3 Reachable States by Board Column

For each board column, the reachable sub-states are listed. States are written as Column + {labels}. An issue in a column with no controlling labels is in the Idle sub-state for that column.

Specify / Research / Plan / Implement / Review / Validate (Active Stages)

Each active stage column has the same set of reachable sub-states:

Sub-State Labels Present Description
Idle (none of the controlling labels) Ready for the engine to pick up
Locked + In Progress fabrik:locked:<user>, stage:<X>:in_progress Stage invocation is active
Editing fabrik:editing Comment processing is active (Claude invoked for comment review)
Paused fabrik:paused Manually paused or engine-escalated pause; no work until unpause or a human comment (bot-authored comments do not resume). Exception: a Validate-stage item whose PR was externally merged advances via runValidatePRTerminalAdvance regardless of this label.
Paused + Failed fabrik:paused, stage:<X>:failed Engine paused after MaxRetries exhausted
Awaiting Input fabrik:paused, fabrik:awaiting-input Claude signaled FABRIK_BLOCKED_ON_INPUT; waiting for user comment. Engine posts a dedicated 🏭 **Fabrik** — @<user>: notification comment so GitHub delivers a mobile push to the configured operator (cfg.User). Under GitHub App auth there is no operator: mentionTargets mentions the item’s human assignees, else its human author, else nobody (bot logins — including the App itself, in either its REST <slug>[bot] or GraphQL <slug> form — are never mentioned; #1893). If Claude’s output included a FABRIK_SUMMARY block, the question is embedded as a blockquote.
Awaiting Review fabrik:awaiting-review, stage:<X>:complete Review gate active; waiting for PR reviewers (only on stages with wait_for_reviews: true)
Awaiting CI fabrik:awaiting-ci CI gate active; waiting for CI checks to pass (pending or failed); stage:<X>:complete is withheld until CI clears (only on stages with wait_for_ci: true)
Rebase Needed fabrik:rebase-needed (+ fabrik:awaiting-ci when wait_for_ci: true) Merge-conflict detected; PR is not mergeable against its base; engine dispatching a rebase re-invocation. Applies to both the conjunctive gate path (wait_for_ci: true stages, via checkMergeabilityGate) and the legacy auto-merge path (yolo+Validate without wait_for_ci, via attemptMergeOnValidate). Distinct from a MergePR CI refusal (gh.ErrNotMergeableCI — a required check is blocked/pending, not a conflict): this label is applied only from settle.Status == PRMergeConflicting, never from MergePR’s own return value, so a CI refusal never enters this state or consumes a rebase cycle (ADR-933)
Awaiting Done fabrik:awaiting-done A FABRIK_NO_WORK_NEEDED decision has been made; the Done board move and/or issue close is still outstanding. Dispatch is suppressed for every non-cleanup stage regardless of which column the item is (still) sitting in — the settle scan (§6.8) retries every poll until it succeeds or escalates
Awaiting Advance fabrik:awaiting-advance A terminal advance (advanceToNextStage) failed to move the project-board Status forward — most commonly a missing target Status option; the item’s own stage is already complete (this is not a dispatch-suppression state — nothing further would dispatch here regardless). The settle scan (§6.17) retries every poll until the board is fixed or escalates
Blocked fabrik:blocked Dependency gate active; waiting for blocking issues to close
Complete stage:<X>:complete Stage finished; waiting for advancement (manual or auto)
Locked by Other fabrik:locked:<other_user> Another Fabrik instance owns this issue
Cooldown (no label; in-memory LastAttemptAt[stageName] in itemstate.StageState) Stage attempted but didn’t complete; waiting for dispatch cooldown to expire

Note: The Cooldown sub-state is purely in-memory — there is no label for it. The engine uses LastAttemptAt[stageName] from itemstate.StageState (written by StageAttempted mutation) to enforce dispatch cooldown. On restart, cooldown state is lost and the item is retried immediately.

Distinct from Cooldown is Deferred Dispatch: an item whose dispatch was skipped in the current poll cycle solely because a worker from a prior cycle is still running. Deferred-Dispatch items still receive the CooldownAt("periodic-re-eval") stamp at end-of-poll — the cooldown avoids repeated deep-fetch evaluation (and the fallback GraphQL fetch when the cache is invalidated or disabled) for an item the dispatch guard (snap.Worker() != nil) would block anyway. Prompt re-dispatch after the prior worker exits is guaranteed by WorkerExited → WorkerLifecycleChanged, which is in wakeChFlags and wakes the poll loop immediately (#544). Note: WorkerLifecycleChanged is excluded from cycleSetFlags (Fix B, issue #576) so it does not bypass the cooldown gate via mayNeedWork. For successful workers, StatusChanged from advanceToNextStage adds the item to cycleSet (bypassing the cooldown) so the next stage dispatches promptly. For workers that ended without completing — whether preempted at the turn cap or genuinely errored — the exit records its own retry-backoff cooldown (§7.1, #2232), whose expiry admits the item on the first poll after the backoff. A second kind of deferral is slot-deferred dispatch (#2114): the candidate passed every guard but every Engine.sem slot was busy. It is deferred with nothing recorded against it and, unlike the worker-guard case above, is not stamped with periodic-re-eval; it is re-queued into mayNeedWork so the next poll admits it through cycleSet. See §9.9.1. See §3.2, §9.2, and §9.9.

Not listed above: fabrik:awaiting-placement is a related but distinct marker applied to a spawned child issue whose initial board placement failed — by construction it is sitting outside any of these active-stage columns (typically Backlog, which is declared unmanaged: true — see below — and therefore never dispatched to regardless), so it never appears as a sub-state of an active-stage column. See §6.9.

Backlog (Unmanaged Stage)

Sub-State Labels Present Description
Parked (any / none) Item sits in the column; engine never inspects it beyond the stage-membership check

itemMayNeedWork and itemNeedsWork both return false for any stage with unmanaged: true, so these items are never dispatched to a per-item worker and never auto-advanced by the catch-up loop (same mechanism as holding_stage, see below). Unlike a holding stage, no batch handler ever acts on an unmanaged stage’s items either — it is a pure no-op declaration. stages/examples/backlog.yaml (name: Backlog, unmanaged: true) is the shipped default; the legacy "Backlog" compat net for installs that predate this file is one predicate, stages.IsParkingColumn (a configured stage decides by its own unmanaged flag; a column with no stage is a parking column only when it is exactly Backlog), shared by checkStageColumnAlignment (engine/startup.go), resolveSpecifyOptionID (engine/spawn.go) and the fabrik_promote guard (parkingRefusal, §7.15). A configured Backlog stage that is not unmanaged keeps its configured meaning. The attention classifier is the one site that does not use it: a legacy Backlog item still reports idle with code no-stage — see USER_GUIDE.md §Startup Board Validation.

unmanaged: true is not rejected at load time when combined with prompt/skill/cleanup_worktree/holding_stage (mirroring the codebase’s existing lack of mutual-exclusion validation between cleanup_worktree and holding_stage themselves), but the resulting configuration is safe, not just permissively invalid: every order-sensitive resolver that could otherwise pick an Unmanaged stage as “the” target for something, or hand it bookkeeping it never earned, skips it. cleanupStage/holdingStage (engine/stages.go) never return a stage with Unmanaged: true, even if it has the lowest Order or is HoldingStage: true — falling back to nil (same as “no such stage configured”) if no eligible non-unmanaged stage remains. NextStage (stages/stages.go) walks past any Unmanaged stage when computing the next stage in the pipeline, so an operator who accidentally gives a parking column (e.g. On Hold) an Order between two dispatched stages does not strand items there — advancement continues to the next real stage instead. The gate-checked completion-label fill loop in runValidatePRTerminalAdvance (engine/pr_terminal_advance.go) stops at the first CleanupWorktree stage that is not also Unmanaged, so a misconfigured unmanaged + cleanup_worktree stage at a low Order cannot short-circuit the fill before any real gate-checked stage is considered. And settleNoWorkNeeded’s skip-labeling loop (engine/no_work_needed_settle.go) both excludes Unmanaged stages from its doneOrder boundary computation and never adds a skip label or “skipped” comment for an Unmanaged stage it steps over — a parking column an operator placed mid-pipeline gets none of the bookkeeping a real intermediate stage would.

Done (Cleanup Stage)

Sub-State Labels Present Description
Pending Cleanup (none) Worktree exists; engine will remove it
Complete stage:Done:complete Worktree removed; terminal state
Paused fabrik:paused Manually paused; cleanup skipped
Awaiting Landing Verification fabrik:awaiting-landing-verification (+ fabrik:credited-pr:<N> for a merge-train landing) Just reached Done via a merge-attributable path; the post-Done backstop is confirming the credited PR actually merged. See §6.19.

Queued (Holding Stage — merge_train: on only)

This stage is a no-op for per-item dispatch. It is the universal landing rendezvous: a yolo (non-cruise) Validate completion under merge_train: on advances here via the dedicated attemptMergeOnValidate → advanceToQueued enqueue, and the Queued handler then picks the landing engine per repo (ADR-059 D6 — see Engine Selection below).

Reachable only via dedicated code (issue #1072). stages.NextStage skips any HoldingStage exactly as it already skips Unmanaged — a holding stage is never returned as “the next stage” by generic positional advancement. The only call site that intentionally lands an item here is advanceToQueued, gated on yoloActive && !cruise && merge_train: on (engine/stages.go). Every other terminal-advance path — runValidatePRTerminalAdvance (cruise, human-merge, or any other gate-label convergence) and advanceConvergedPRToDone (non-train yolo convergence) — resolves NextStage from Validate and now lands directly on Done, never on Queued, regardless of merge_train state. Before this fix, NextStage’s purely positional walk landed every such path in Queued whenever the column was configured, stranding items with no active drain (ADR-1072).

Sub-State Labels Present Description
Waiting stage:Validate:complete Item moved to Queued column; advanceToQueued added stage:Validate:complete atomically. Awaiting handleMergeTrainBatch to form a landing batch.
Landing stage:Validate:complete Batch owned by a persisted merge-train run (§”Merge-Train Run State”, #2051): trial branch being assembled, a trial waiting on CI (no goroutine held — evaluated per poll), red-batch bisection, one-at-a-time fallback, or the landing integration PR open/merge in progress.
Done (none — advanced by advanceToNextStage) Integration PR merged; member advanced to Done column via advanceToNextStage; member PR closed with batch reference comment.

itemMayNeedWork and itemNeedsWork both return false for any stage with holding_stage: true, so these items are never dispatched to a per-item worker. processItem also short-circuits on stage.HoldingStage as a defense-in-depth guard. The catch-up loop skips holding stages for the same reason. A closed item that nonetheless ends up stranded in Queued (e.g. a human gh pr merge outside the train) is rescued by the settle-scan backstop described in §6.11.

The same holding-stage exclusion that keeps dispatch off Queued items also blacks out PR feedback processing while they sit there — review-reinvoke and comment processing never run on a Queued item either, since both live behind the identical HoldingStage admission gate. §6.16 describes the dedicated settle scan (settleQueuedReviewFindings, ADR-1208) that detects an unresolved review-thread finding (and, since #1863/ADR-1863, an unprocessed human comment) arriving on a Queued member and ejects it back onto a stage the ordinary reinvoke/comment path can reach, rather than widening this section’s dispatch exclusion itself.

Engine Selection: 058 Native Queue vs 059 Internal Train (ADR-059 D6)

Queued is one board column served by two landing engines. The single convergence owner handleMergeTrainBatch (ADR-056 — there is no parallel scanner) picks the engine per repo each poll, because isMergeQueueEnabled is a per-PR/per-repo signal and the Queued column may hold items from several repos at once.

Selection algorithm. handleMergeTrainBatch groups the current Queued snapshot by owner/repo and resolved base branch (groupQueuedByRepoAndBase, #1648/ADR-1648 — superseding the repo-only groupQueuedByRepo this function used before), preserving first-seen repo/partition order for the groups themselves. Within each (repo, base) partition, member order is a deterministic function of the Queued set, not of whatever order the board-state source handed items back in (#1833/ADR-1833): items are sorted ascending by (StatusEnteredAt, Number) — the member that has waited longest to enter Queued goes first, issue number breaking any tie — via sortQueuedGroupsByEntry, applied after partitioning so ordering never crosses a partition boundary. StatusEnteredAt is a record-on-write timestamp (itemstate.ItemState.StatusEnteredAt, sibling of LabelAppliedAt/ADR-1314) stamped the instant a Status mutation is applied, threaded into gh.ProjectItem only by the in-memory-cache path (boardcache.CacheImpl’s snapshotToProjectItem); the direct-GraphQL GitHubAdapter path never populates it, since no cheap durable “entered this Status” signal exists via REST/GraphQL for Project v2 items — every item there ties on the zero value and the sort degenerates to ascending Number order, still fully deterministic, just not wait-time-ordered. Before this fix, the in-memory cache’s underlying map iteration (itemstate.Store.All()) was genuinely randomized per poll, so a Queued set exceeding max_batch_size could select a different arbitrary subset every cycle, abandoning the previous cycle’s in-flight trial CI each time. baseBranchForItem is called only for items that actually carry a base: label — an item with no label buckets under the defaultPartitionBase sentinel without ever touching git, so partitioning costs nothing for the overwhelming common case. Resolving a label needs the repo’s in-memory WorktreeManager, which only ensureRepoReady registers (on a stage dispatch); after a restart a repo whose only open items are Queued would have none, and its base:-labelled members were skipped every poll for ever. So handleMergeTrainBatch first calls ensureQueuedBaseReposReady (#2250), which runs ensureRepoReady — clone, or a clone-failure pause, exactly as on a stage dispatch — once per poll for each repo with a base:-labelled Queued member and no manager; a repo with only unlabelled members is never touched. The “WorktreeManager not yet registered” exclusion remains as the fail-closed fallback for a failed or unfinished clone. A third fail-closed guard runs before this, ahead of the base-label check (#1772/ADR-1772): an item whose board-cache entry has never been deep-fetched (CacheImpl.IsItemDeepFetched, checked per (repo, number) — never a global “cache is bootstrapped” flag) carries an empty Labels slice indistinguishable from a genuinely label-free item; rather than risk silently pinning it to defaultPartitionBase, it is excluded from batching for that poll (logged under the merge-train tag naming the item and reason) and left in Queued, picked up warm on a later poll once hydrated. This check fails open when no CacheImpl is wired, since a live-read engine has no unhydrated state to guard against. routeQueuedGroup then runs once per (repo, base) partition, routing its items by the poll-native LinkedPRIsMergeQueueEnabled signal (populated by the GraphQL board query — never a REST pulls fetch, which always reports false for the queue flag, and never a webhook payload):

Batch-snapshot logging is deduped by composition (#1151), per partition. routeQueuedGroup runs once per poll cycle for as long as any Queued items exist for a partition — including the full CI-wait duration of an in-flight train, which can span many poll cycles. Before dispatching, it computes an order-independent signature of the internal-train subset (mergeTrainBatchSignature: sorted item numbers, comma-joined) and compares it against Engine.mergeTrainBatchSnapshotSeen (a sync.Map keyed on the same composite mergeTrainKey). The batch snapshot for <trainKey>: N item(s) — ... line is only logged, and the stored signature updated, when the signature differs from the last-logged one for that partition — a membership or count change re-logs; a pure reordering of the same members (GraphQL item order is not guaranteed stable poll-to-poll) does not. Dispatch itself (dispatchMergeTrainWorker) always runs regardless of whether the log fired.

Both engines drain the same Queued column and advance their members to Done on land; only who batches differs (GitHub’s queue vs Fabrik’s trial branch). max_batch_size caps the internal-train subset per (repo, base) partition, not the repo’s combined total across partitions — so a large repo A never starves repo B (the pre-D6 batch[0]-anchored latent multi-repo bug, hardened by grouping), and one of repo A’s own bases never starves a sibling base in the same repo either.

Precedence (FR-3). For any item reaching a landing decision:

  1. Native merge queue present on the repo (MergeQueue != "off" && LinkedPRIsMergeQueueEnabled) → 058 enqueue path, regardless of merge_train. The queue always wins: a direct or trial-branch merge on a queue-required branch returns HTTP 405. Under merge_train: off this fires inline in attemptMergeOnValidate; under merge_train: on it fires from the Queued handler.
  2. Else merge_train: on → the internal train (from the Queued handler).
  3. Else → legacy per-PR serial auto-merge (attemptMergeOnValidate’s native-auto-merge / direct-merge fallback), unchanged from pre-merge-train behavior.
Merge-Train Landing Lifecycle (ADR-059 D3)

When handleMergeTrainBatch has a non-empty batch of Queued items, it calls dispatchMergeTrainWorker, which starts a short goroutine running runMergeTrainWorker. Since #2051 (ADR 2051) that goroutine no longer holds the episode: it prepares the batch, assembles and opens the trial, records the open trial as persisted run state and exits. Every later step — evaluating the trial’s CI, bisecting, the one-at-a-time fallback, landing — is a step of the run state machine (§”Merge-Train Run State, Per-Poll Evaluation and Restart Resume” below), run by a short step goroutine only when there is something to do. The steps own the assembly-to-landing sequence:

Two-PR model: The goroutine creates two PRs: (1) a draft CI PR (chore(merge-train): trial integration for #N…) that triggers checks on the trial branch immediately, and (2) a landing integration PR ([merge-train] batch: #N1, #N2, …) opened only after trial CI passes. The draft CI PR is closed implicitly when the trial branch is deleted at cleanup.

Trial assembly — merge conflicts (issue #1235): assembleTrialBranch merges each batch member’s head SHA into the trial worktree in sequence (git merge --no-ff --no-edit). A clean merge just continues to the next member. On conflict, it calls resolveTrainConflict, which first lists the conflicted paths (unmergedPaths, parsing git status --porcelain for the UU/AA/DD/AU/UD/UA/DU codes, returning each path paired with its two-letter status code as a conflictedPath) and classifies them against a single declared generated-path → regeneration-command mapping (generatedFiles in engine/generated_files.go; today’s only entry is docs/llms-full.txt → bash scripts/generate-llms-full.sh). If unmergedPaths itself errors (e.g. a transient git status failure), resolveTrainConflict cannot classify anything and falls back to dispatching Claude for the full, unscoped conflict exactly as it would pre-#1235 — a deliberate, narrow tradeoff: a git-level failure says nothing about whether the conflict is confined to a generated path, so failing open to the safe pre-existing behavior is preferable to guessing (TestResolveTrainConflict_UnmergedPathsErrorFallsBackToPlainClaude). Otherwise, three outcomes:

classifyConflictedPaths treats a declared generated path as eligible for regeneration only when its conflict status carries no deletion intent. A status of DD (both sides deleted), UD (deleted by them), or DU (deleted by us) — deletionInvolvingStatus — routes the path into the non-generated bucket instead, so it reaches Claude like any other conflict rather than being silently regenerated: a contributor who deleted a declared generated file meant to remove it, and blindly rerunning the regeneration command would recreate content nobody wants (TestMergeTrainWorker_DeletionConflictOnGeneratedPathRoutesToClaude). AA/UU/AU/UA carry no deletion intent and remain eligible for regeneration as before. classifyConflictedPaths also returns this deletion-excluded subset separately (deletionExcluded, alongside matched and nonGenerated) precisely so the mixed-case call into regenerateAndCommit can pass it through as protectedPaths.

protectedPaths closes a further interaction between the two mechanisms above: if a matched path and a deletion-excluded sibling declare the same regeneration command, executing that command for the matched path also regenerates the sibling on disk as a side effect — even though the sibling isn’t in specs and Claude (not regeneration) owns its resolution. regenerateAndCommit never stages such a sibling; instead it discards the command’s side effect on that one path — restoring the working-tree file from the index via git checkout-index -f when Claude staged content for it, or removing the file entirely when Claude staged its deletion (git rm) — before staging and committing the rest as usual (TestMergeTrainWorker_SharedCommandDoesNotOverwriteDeletionExcludedSibling).

A regeneration failure (non-zero exit, inability to stage, conflict markers still present afterward, or — in the mixed case — Claude having committed prematurely despite being instructed not to) ejects the member via the existing ejectMember path with a diagnosable reason describing the specific failure — it never falls back to Claude. The premature-commit case is detected structurally, not by content: regenerateAndCommit requires MERGE_HEAD to still be present at entry (it is only ever called with the merge either still fully in conflict, or with Claude having resolved-and-staged but not committed the non-generated part) and fails closed immediately if it’s already gone, before running any regeneration command. This is deliberately not a git diff --cached content comparison after the fact — that would miss the case where Claude’s premature commit happens to write byte-identical content to what regeneration would have produced. This mirrors the existing unresolvable-conflict eject path (assembleTrialBranch’s abort-and-eject branch) and reuses the same three-way (resolved, reason, err) contract style as the pre-existing usage-limit sentinel handling described under ADR-1120 below: a non-nil err (the ADR-1120 usage-limit sentinel) is still a fatal assembly abort, never an ejection, in every one of these three outcomes.

resolveConflictWithClaude also guards against a false “resolved” verdict: buildTrainConflictComment’s fallback instructions tell Claude to run git merge --abort when it judges a conflict genuinely unresolvable, and an abort clears every conflict marker (and MERGE_HEAD) exactly as a real resolution would — unmergedPaths alone can’t tell the two apart. assembleTrialBranch captures the trial worktree’s HEAD SHA immediately before attempting each member’s merge (preMergeHEAD) and threads it through resolveTrainConflict into resolveConflictWithClaude; after confirming no non-generated conflict markers remain, it additionally requires either MERGE_HEAD still present (merge legitimately in progress) or HEAD having moved past preMergeHEAD (a real commit happened). A MERGE_HEAD-less worktree still sitting on preMergeHEAD is treated as unresolved — an abort, not a resolution — so the member is ejected instead of silently landing with its whole contribution missing. This applies to both the plain and mixed conflict paths.

A turn-limit exit is not automatically “unresolvable” (#1841). buildTrainConflictComment’s instructions no longer tell Claude to run the target project’s own build/test/lint/install commands — that unconditional, Go-specific step (go build ./..., go vet ./..., hardcoded regardless of the target repo’s actual language/toolchain) was identified as the dominant turn sink for small conflicts (a live reproduction: the same single-file conflict took 10 turns with the instruction present, including exploratory detours — checking for go.mod, substituting the project’s real npm scripts, reviewing unrelated git history — versus 5 turns once it was replaced with an explicit “do not verify, do not explore outside the conflicted path(s)” instruction; see adrs/1841-*.md for the full evidence). Verification is left entirely to the trial’s own CI, which runs unconditionally after assembleTrialBranch regardless of how the conflict was resolved (assembleAndValidateInner → pollTrainCI) — removing the instruction does not remove the safety net, only a redundant, expensive one. Even with a shrunk turn sink, a conflict-resolution invocation can still exhaust its configured turn budget (commentMaxTurns(holdingStg), default 50, doubled by fabrik:extend-turns); before #1841, resolveConflictWithClaude treated any non-usage-limit error — including this turn-cap exit (claudeTurnLimitError, CLI subtype error_max_turns) — as an immediate, unconditional “unresolvable,” returning before the worktree-inspection logic above ever ran. Since a turn-cap exit still means the dispatch reached and ran Claude (the same “Claude is healthy” signal a clean success carries — the per-invocation analogue of the account-wide ADR-1120 principle below), resolveConflictWithClaude now runs the identical inspection (extracted into finalizeConflictResolution, shared by both the clean-exit and turn-limited-exit paths) before concluding anything: if the worktree shows no remaining conflict markers, the resolution is committed and the member stays in the batch exactly as a clean success would, rather than discarding real, completed work purely because the invocation ran out of turns on unrelated work after finishing. The account-wide Claude-usage-limit hold-clearing call also fires on this path, unchanged from the clean-exit case.

Ejection comments now say why, not just that (#1841). When a conflict genuinely remains unresolved — whether from a clean-but-incomplete exit or a turn-limited one — resolveConflictWithClaude/finalizeConflictResolution return a conflictEjectionDiagnostic (TurnLimited, NumTurns, RemainingPaths, Reason) threaded as a plain return value — never shared/mutable state, mirroring ADR-1420’s diagnostic-threading precedent below, but as its own small type rather than layered onto the CI-specific trainCIDiagnostic — through resolveTrainConflict to assembleTrialBranch’s ejection call site. buildConflictEjectionReason renders it into ejectMember’s plain-string reason: naming the still-conflicted file(s) (file-level only, from unmergedPaths’ git status --porcelain granularity — not hunk-level, a documented non-goal) and distinguishing “the invocation ran out of turns rather than judging the conflict unresolvable” from “conflict judged unresolvable” wording, so an operator reading the ejection comment knows immediately whether the invocation ran out of budget or a genuine judgment call. This also improves the pre-existing, non-turn-limited ejection path, which previously always reported a generic “unresolvable conflict” message with no file names at all. A nil diagnostic (the unmergedPaths-error fallback path inside resolveTrainConflict, which has no original conflicted-path list to report) still falls back to that same pre-#1841 generic message.

A clean unmergedPaths result doesn’t prove marker-free content (#1841 review hardening). git status --porcelain clears a path’s UU status the moment it’s git added, regardless of whether the staged content still contains literal conflict-marker text — so by the time finalizeConflictResolution’s plain-case git diff --check (no --cached) runs, every originally conflicted path has necessarily already been staged, making the working tree and the index identical and the check a structural no-op. pathsStillContainConflictMarkers closes this by reading each of originalNonGeneratedPaths’ current on-disk content directly and checking for conflict-marker text — immune to whether the content has since been staged or even committed (the plain case’s own prompt instructs Claude to commit itself as its last step, which git diff --cached --check alone would also miss). Checked once, right after the abort-disambiguation block, covering both the plain and mixed conflict paths ahead of the mixed case’s early return. conflictMarkerBlockRegex went through two rounds of narrowing after review: first, from matching any line starting with 7+ of <, =, >, or | down to just the two boundary forms <<<<<<</>>>>>>> (a bare =======/||||||| line is common legitimate content — a Markdown setext heading underline, an RST section underline, an ASCII banner — and matching it alone would eject a correctly resolved file); then, since a lone boundary line is itself plausible legitimate content in a repo whose purpose involves documenting or testing conflict-marker syntax (a doc example, a test fixture, quoted diff/patch text), the regex now requires the full three-part structure in order — a <<<<<<< line, then later a bare ======= line, then later still a >>>>>>> line — matching git’s own conflict-marker shape exactly. This still doesn’t rule out every false positive (a file documenting a complete worked conflict example would still match), but real content essentially never happens to contain all three markers in the correct order, so detection coverage of a genuine unresolved conflict is unaffected. A fourth pass fixed a false negative: the separator span was anchored as ^={7}\n, an exact zero-width transition from the seventh = to a line feed, which never matches a CRLF-terminated =======\r\n line (common in Windows-origin repos or any repo with .gitattributes eol=crlf) — a genuinely unresolved, CRLF-terminated conflict block went undetected and would have been committed and passed to the trial’s own CI instead of being caught here. The anchor is now ^={7}\r?\n; the boundary spans didn’t need the same fix, since [^\n]* and . already absorb a preceding \r as ordinary line content.

assembleTrialBranch’s own eject-on-unresolvable cleanup reuses that same captured preMergeHEAD: since resolution can now fail after a commit already landed on wtDir (e.g. regenerateAndCommit’s premature-commit guard tripping), git merge --abort alone is not sufficient cleanup — with MERGE_HEAD already gone, it is a silent no-op, and the bad commit would remain as wtDir’s HEAD, contaminating every subsequent member’s merge attempt and the pushed trial branch. The eject path therefore unconditionally follows the best-effort abort with git reset --hard <preMergeHEAD>, which is a no-op when the abort already fully reverted things and the authoritative cleanup when it didn’t. git reset --hard only rewinds tracked content, so the eject path also unconditionally runs git clean -fd afterward to remove any untracked files a failed resolution attempt (Claude, or a regeneration command) may have left behind — without it, a stray untracked file would make the next member’s git merge fail with git’s own “untracked working tree file would be overwritten by merge” error, which resolveTrainConflict would misclassify as a plain conflict (no MERGE_HEAD, no unmerged paths) and dispatch Claude against a worktree with nothing to resolve.

Landing sequence (FR-1 through FR-5):

Pre-FR-1 — base sanity check (#1773). Immediately before FR-1’s if integrationPR != nil { reuse } else { create } split — covering both the open and reuse branches uniformly — landMergeTrainBatch (and, at its own equivalent CreatePR call, landSingleton) calls refuseIfBaseContradictsMembers. On the default-base partition only, it reads each member’s base:<branch> label live (never the cached snapshot) and refuses to open or reuse the PR if any member’s live label declares a different base, or if the live read itself errors — leaving members in Queued rather than landing against a base that contradicts what they declare. See §6.24 for the full mechanism; this is a no-op on any non-default-base partition and on a healthy default-base batch.

  1. FR-1 — Open integration PR: landMergeTrainBatch calls CreatePR (non-draft) with title [merge-train] batch: #N1, #N2, … and a body containing the idempotency marker <!-- fabrik-merge-train-batch -->. The body lists all batch members (issue number + title) and a Closes #N line per member. The member PRs are closed-not-merged (the change lands via this integration PR), so a member PR’s own Closes #N never fires — the integration PR’s Closes #N is what restores issue↔landing-PR connectivity and auto-closes each member issue when the integration PR merges into the default branch. (For a non-default base, or on auto-close lag, FR-3’s explicit CloseIssue is the fallback.) The same Closes #N lines are carried on the trial draft CI PR body from assembly, since that draft PR IS the landing PR when reused (FR-5).

  2. FR-2 — Poll and merge: pollForMergeable polls the integration PR with the same 30s interval as pollTrainCI, using FetchPRDetails (mergeable_state and head SHA in one call) and classifyLandingCI (ADR-1441, #1441) — the merge-train landing counterpart to pollTrainCI’s own check-run-aware classification, built from the same shared primitives (gh.ClassifyCheckRuns, e.classifyRequiredContexts, describeCheckRuns, gh.MergeableStateAccepted). mergeable_state == "dirty" rejects immediately with no check-run fetch; otherwise check runs on the head SHA are fetched and classified: a confirmed failure or required-context failure rejects; a check still pending keeps polling; an all-clear (plus required contexts satisfied) lands. Only with zero check runs at all does an accepted mergeable_state (clean/unstable, gh.MergeableStateAccepted) become load-bearing by itself — mirroring pollTrainCI’s own zero-check-runs fallback. Before ADR-1441, this step accepted mergeable_state ∈ {"clean", "unstable"} outright with no check-run awareness at all — the merge-train landing counterpart to the single-PR advance gate’s pre-ADR-1441 shortcut, left unfixed by ADR-1153 and explicitly flagged there as a “candidate fast-follow.” Once classifyLandingCI judges the PR landable, MergePR is called (no admin bypass). MergePR itself independently re-checks mergeable_state against the clean/unstable allowlist before merging (see “MergePR’s own CI precondition (ADR-933)” in §5, after §5.3) — normally a no-op here since pollForMergeable already judged the PR acceptable, but it closes a narrow TOCTOU window if the state flips between the two checks; this merge-side check is deliberately unchanged by ADR-1441 (its R3 decision: the merge path continues to defer to branch protection alone, per ADR-072’s operator note). On timeout, a warning comment is posted on the first batch member and the function returns without advancing members (they remain in Queued for the next train cycle). On merge API failure — including gh.ErrNotMergeableCI from that TOCTOU window — an error comment is posted and landing aborts, leaving members in Queued to retry on the next train cycle; this is not escalated to a pause and does not touch fabrik:rebase-needed.

  3. FR-3 — Member lifecycle: For each surviving batch member: (a) advanceToNextStage moves the item from Queued → Done on the project board; (b) addLandedCommentWithRetry posts 🏭 **Fabrik merge-train** — Landed via batch PR #<N>. on the member PR — this is the sole cross-landing-path, member-scoped record of which PR actually landed the change (issue #1275), so a bare AddComment failure would silently and permanently lose it; the shared helper retries up to 3 attempts total with exponential backoff (landedCommentRetryDelay, base 200ms, doubling), gated on isTransientError (a non-transient error like a 404/422 short-circuits to the warn immediately, without wasting the backoff window) — on exhaustion it falls back to the pre-existing warn-and-continue behavior unchanged, never blocking or delaying landing; the identical helper is used by landSingleton’s equivalent comment post (🏭 **Fabrik merge-train** — Landed one-at-a-time via singleton PR #<N>.); (c) CloseIssue closes the member PR explicitly; (d) CloseIssue closes the member issue explicitly. Step (d) is the fallback that guarantees issue closure independent of the integration PR’s Closes #N (which only auto-closes on merge into the default branch) — without it a member landed via a non-default base, or before auto-close propagates, is left landed-but-open. It is idempotent (a no-op on an already-closed issue). Members whose Status == "Done" are silently skipped (restart safety). The same close-issue fallback runs in the one-at-a-time singleton landing path (landSingleton) — but only in landSingleton is a failure of step (d) retried: on failure, markMergeTrainMemberCloseOutstanding writes fabrik:awaiting-member-close, and settleMergeTrainMemberCloses retries it every poll thereafter, escalating to fabrik:paused after MaxRetries (§6.10, ADR-061). landMergeTrainBatch’s own step (d) has no such retry — a deliberate, narrowly-scoped-fix decision (ADR-061 §Sibling Audit); a follow-up issue is expected to extend the same machinery to this path.

  4. FR-4 — Cleanup: CleanupTrainWorktree(name, deleteBranch: true) runs in a defer, removing the trial worktree directory and deleting both the local and remote trial branch (git push origin --delete). This is the single call site responsible for local worktree removal per trial lifecycle — assembleAndValidate no longer removes the worktree itself after opening the draft CI PR (pre-#1151, it did, making every downstream cleanup call a guaranteed no-op on the success path); the trial worktree now stays checked out for the full CI-poll window and is reclaimed exactly once, here (or in the equivalent failure-path cleanupTrialArtifacts call), regardless of outcome. All cleanup steps are best-effort (failures are logged, not propagated) — but only genuine failures are logged: a git worktree remove failing because the path is already not a working tree, or a git push --delete failing because the remote ref already doesn’t exist (e.g. removed by GitHub’s own “delete branch on merge” behavior once the integration PR — which reuses the trial branch — merges), is classified as a no-op success and does not emit a warn: line (#1151). The remote-branch-delete push also runs with an SSH LogLevel=ERROR override scoped to this one invocation, suppressing the “Permanently added … to the list of known hosts” notice without weakening StrictHostKeyChecking.

  5. FR-5 — Restart idempotency: findIntegrationPR scans recent PRs via ListPRs (state=all) for this trial’s own PR — HeadRefName == trialBranch (fabrik/merge-train/<trialName>) is the sole, mandatory identity gate (#1615, R1). The shared idempotency marker <!-- fabrik-merge-train-batch --> is checked too, but only as non-fatal corroboration: a branch match is returned regardless of whether the marker is present (a warning is logged if it’s missing), and a marker match on a different branch is never returned, no matter how recently that PR was updated or how many repo-wide PRs ListPRs surfaces. Before #1615, the marker alone was sufficient — since every merge-train PR in the repo carries it, in any state, a later trial’s search could return an unrelated, already-merged PR from a completely different batch, and landMergeTrainBatch would trust that PR’s Merged field (alreadyMerged) to skip its own merge step and proceed straight to member advancement, closing the member PR and issue as though the batch had actually landed (reported in #1614). Once branch identity gates the match, the returned PR’s State/Merged are inspected to decide what happens next: open — FR-1 is skipped, FR-2 proceeds as normal (poll, then merge); merged — FR-2 is also skipped (alreadyMerged) and FR-3 proceeds directly, now provably safe since identity is confirmed; closed and unmerged — a failed trial, not a reusable integration PR (R2) — routes to the escalation path below instead of FR-3, never treated as “already landed” (R5). Member advancement is additionally guarded, per member, by two checks: Status == "Done" (restart safety, unchanged) and, new in #1615, whether the member’s issue number actually appears in the integration PR’s body (parseTrainMembers) — a batch that dropped a member during assembly must not be able to claim it (R4). A member failing either the closed-unmerged trial check or the per-member membership check is escalated (below) rather than silently skipped or, worse, advanced.

Landing-failure escalation (#1615, R4/R5). When findIntegrationPR returns this trial’s own PR closed and unmerged, escalateClosedUnmergedTrial escalates every survivor; when the matched (merged) integration PR’s body doesn’t list a specific member, that one member alone is escalated in place inside the FR-3 loop, leaving the rest of the batch to land normally. Both call the same escalateStrandedTrainMember, mirroring ejectRedSingleton’s reroute+comment+pause convention (§ “Merge-Train Red-Batch Bisection” below) rather than ejectMember’s counter/stay-in-Queued semantics — this is an infra-level landing failure (the integration PR itself, or the batch’s own composition), not a retryable member-level defect: rerouteQueuedMemberOffHolding moves the member off Queued to stageBeforeHolding first (a failed reroute leaves the member untouched for the next poll to retry, exactly as ejectRedSingleton’s identical failure path); on success, an explanatory 🏭 **Fabrik merge-train — landing failed** comment is posted (naming the closed PR, or the merged PR that dropped the member) and pauseMergeTrainMember applies fabrik:paused + fabrik:awaiting-input. The reentry guidance (fabrik:revalidate when the reroute target is literally named "Validate", or naming the real blocking stage:<target>:complete label otherwise) is the same reentryInstruction helper ejectRedSingleton uses, so both escalation paths give identical, correct instructions for the same reroute target. No “landed via” comment is ever posted for an escalated member (R5/AC4), and R6 — that comment may only be emitted after the batch PR is observed MERGED — holds structurally once R1–R4 do: the FR-3 loop only ever reaches the comment for a member that passed both the branch-identity and membership checks on a PR whose Merged field is true.

pollForMergeable’s own polling loop never inspects pr.State — if this trial’s PR is open when findIntegrationPR returns it but closes (unmerged) mid-poll, it is a narrower, deliberately out-of-scope variant of the same defect: the loop simply times out (unchanged pre-#1615 behavior, members remain in Queued for a fresh retry) rather than routing to the escalation path above, since it can’t misattribute a landing the way the fixed bug could.

State threading: board.ProjectID is threaded from handleMergeTrainBatch → dispatchMergeTrainWorker(ctx, batch, projectID) → mergeTrainWorkerState.projectID (immutable after dispatch, no mutex needed). Inside landMergeTrainBatch, a minimal &gh.ProjectBoard{ProjectID: projectID} is constructed for the advanceToNextStage call.

mergeTrainInFlight lifecycle: set at dispatchMergeTrainWorker entry (LoadOrStore), keyed on the composite mergeTrainKey(owner/repo, base) — not bare owner/repo — since #1648, so two partitions of the same repo each get their own independent claim/clear lifecycle. Cleared by finishTrain — the single ADR-067-mandated clear point, called from prepareTrainWorker’s own-failure defer and from the run’s finish() when the episode ends (landed, dissolved, ejected, abandoned). Since #2051 the claim lives as long as the RUN, not as long as a goroutine (ADR 2051 amends ADR-067): while a trial waits on CI no goroutine exists but the claim persists, so the duplicate-launch guard, ADR-1208’s pending-eject routing (mergeTrainBatchMembers) and the invalidateConflictingQueued gate behave exactly as they did with a live worker. The Store liveness registry (Store.ExitRepoWorker, same composite key, §9.2, issue #1222) now means “a step goroutine is executing” — assembly, a Claude conflict resolution, a landing: it is set when a step is launched and cleared by leaveStep when the step returns without finishing the run (the claim stays). So an open trial that is merely waiting on CI does not block the auto-upgrade idle guard (a restart resumes it), while a step in progress still does. mergeTrainWorkerActiveForRepo (closed_item_advance_settle.go) answers “live” for either a Store marker or an open run. mergeTrainInFlight is the atomic duplicate-launch claim plus the assembling/bisecting/CIResult/prNum/trialName sub-state; the Store registry is the liveness answer the auto-upgrade idle guard reads (repo-wide, via Store.RepoWorkerActiveForAnyBase’s prefix scan).

Merge-Train Red-Batch Bisection (ADR-059 D4)

A batch is usually green: every member already passed Validate individually, so a batch fails only on a genuine cross-PR semantic conflict. runMergeTrainWorker is therefore structured as a re-form loop around a single combined validation per (re-formed) batch:

Bisection on red (FR-1): when the combined Validate is red, the run enters its bisection state (bisectState, engine/train_bisect.go, #2051) with a per-episode cost budget (Used = 1 for the initial red validation, Cap = effectiveBisectCap() = MaxBisectValidations, default 2·⌈log₂(max_batch_size)⌉ + 1 ≈ 7, read once when the episode starts and persisted). The recursion of the old bisect was a tail call, so its entire state is five values — Origin (the batch the episode started from), Red (the set currently known red), NextHalf (0, 1, or 2 = both halves green), Used, Cap — plus the red set’s diagnostic. next() reproduces the halving exactly: a single member in Red is the poisoner; otherwise mid := len(Red)/2, half A is Red[:mid] then half B Red[mid:], the cost cap is checked before each half, a red half with survivors becomes the new Red (survivors, not the half — members ejected during assembly are not in it) and restarts at its first half, and both halves green is the non-isolable-interaction fallback. Used is spent after each half trial and before the cancel / assembly-error / runaway / infra checks. A property test pins the state machine to a verbatim copy of the old recursion. Every trial (initial and every sub-trial) is assembled by assembleAndValidate off the same base SHA pinned once at batch start (D-b, via EnsureTrainWorktreeAt), so redness is attributable to member composition, not a moving base branch (main-moved is D5, out of scope). Each sub-trial’s branch + worktree + draft CI PR is reaped immediately by cleanupTrialArtifacts — but since #1835, assembleTrialBranch’s assembly step underlying every one of these assembleAndValidate calls first checks whether the trial’s member list starts with a previously-recorded prefix from earlier in this same worker invocation, forking from that prefix’s already-built commit instead of the pinned base SHA whenever one matches — bisect’s first half is one such prefix by construction, needing zero merge work. See §6.27.

Red singleton short-circuit (#1440, reroute added #1545, auto-repair added #2045): (Since #2045 the reroute-and-pause described below is the fallback — cap reached, max_train_auto_repair_attempts: 0, or a reroute target not literally named Validate; by default ejectRedSingleton first runs the bounded auto-repair of §6.31.) a red batch of exactly one member is never handed to handleRedBatch at all — runMergeTrainWorker’s TrainCIRed branch checks len(survivors) == 1 first and, when true, calls ejectRedSingleton directly. Bisection exists to isolate a poisoner among two or more members; a batch of one has nothing to isolate, so routing it through bisect (whose own len(red) == 1 base case is a pure pass-through) would only reach the same poisoner-isolation ejection wording — “isolated by halving bisection,” “retried in a future train with a different composition” — that is actively misleading for a batch that never had more than one member to begin with. ejectRedSingleton:

handleRedBatch and bisect are otherwise unchanged; every batch with two or more members bisects exactly as described in the rest of this section, and the arity guard above means bisection itself never sees a true singleton. landOneAtATime’s own red-singleton-ejection branch (below, FR-5) validates its member completely alone — the same true-singleton scenario, reached via a different path (bisection degrading a genuinely multi-member batch rather than the top-level arity guard) — so it also calls ejectRedSingleton rather than ejectMember, getting the identical wording/no-counter/immediate-pause disposition instead of leaving the pre-#1440 misleading wording live in a second reachable path.

Eject + re-form + re-validate (FR-2/FR-3): the isolated poisoner is ejected via ejectMember (D-a — the same shared MaxMergeTrainEjections counter that assembly-conflict ejections increment; D-c — the ejection comment is the attention marker; cap→pause reuse: after MaxMergeTrainEjections ejections the member gets fabrik:paused + fabrik:awaiting-input). The poisoner remains in the Queued column — ejectMember never moves it off the board — so a still-unpaused poisoner is retried in a future differently-composed train. Poison-well guard: once a member is fabrik:paused (or its issue is closed), groupQueuedByRepoAndBase excludes it from every subsequent batch snapshot. Without this, a poisoner that reds the combined Validate even in isolation — paused but left in Queued — would be re-snapshotted into every future batch, reding and bisecting the train indefinitely and starving clean members from ever landing. A human removing fabrik:paused after resolving the underlying conflict re-admits the member. handleRedBatch returns the surviving members; the main loop re-forms and re-validates them (a survivor batch is not assumed green — a second poisoner or interaction may remain) — since #1835 this re-form reuses the prefix through the position just before the ejected poisoner, merging only the members after it (see §6.27). Terminal states are clean: an empty survivor set completes the train with nothing to land (the existing zero-survivor path); each episode ejects ≥1 member or falls back, so the loop runs ≤ N episodes.

Ejection comment carries the combined-Validate diagnostic (#1420). Because a merge-train failure is, by construction, a failure that does not exist on any member’s own branch — it arises only from combining with a base that moved since the branch was cut — the combined-Validate run that observed it is the only place the failure output ever exists. pollTrainCI builds a trainCIDiagnostic (failing check names, output text/summary, and details/run URL for ordinary check-run failures; context names only for classic-commit-status RequiredContextsFailed failures, which carry no check-run output; a free-text Note for a dirty mergeable_state with no per-check signal at all) at the point of failure and threads it as a plain return value — never shared or mutable state — through assembleAndValidate → bisect/handleRedBatch/landOneAtATime → ejectMember/ejectRedSingleton (#1440: landOneAtATime’s red-singleton branch and the top-level arity-guard short-circuit above both terminate in ejectRedSingleton instead, for the true-singleton case), so the run that isolates the poisoner is always the diagnostic’s origin, and bisection’s later, unrelated re-validation of the reformed survivor batch cannot overwrite it. ejectMember/ejectRedSingleton render the diagnostic (bounded per-check inline excerpt, degrading to a head/tail truncation plus a details/run link when oversized, capped again at a whole-block hard limit well under GitHub’s comment size limit) into every ejection comment for that failure — the first as fully as the last, so an operator doesn’t pay for three uninformative cycles before learning there’s something to look at — alongside a sentence naming the other members riding in that train attempt (or stating explicitly that none were present, for a single-member train), so the operator knows before investigating that the fault does not exist on their own branch. The pause-after-MaxMergeTrainEjections comment names the same failing check(s)/context(s) and links the permalink of the ejection comment that carries the full diagnostic, rather than repeating “resolve the underlying conflict” with nothing to act on. Two of the three ejection call sites outside this issue’s original scope (unfetchable PR, no head SHA) still pass a nil trainCIDiagnostic, leaving their ejection comments exactly as before. The third — unresolvable merge conflict — gained its own, differently-shaped diagnostic under #1841 (conflictEjectionDiagnostic, described above), folded directly into ejectMember’s plain-string reason rather than passed as this function’s diag parameter; ejectMember’s own trainCIDiagnostic parameter is still nil for every conflict-resolution ejection call. See ADR-1420 and ADR-1841.

One-at-a-time fallback (FR-5): when bisection cannot isolate a single culprit within the cost budget — either the budget is exhausted, or both halves validate green (a non-isolable cross-PR interaction: each half green alone, the union red — D-e, no bespoke interaction-detection in v1) — handleRedBatch logs the degrade clearly (never silent) and calls landOneAtATime, which validates and lands each remaining member as its own singleton batch. A green singleton lands via landSingleton; a red singleton (fails even in isolation) is disposed of via ejectRedSingleton (own-validation-failed wording, immediate pause, no counter — see above); a pending singleton stays in Queued. This dissolves any interaction by construction (no two members co-reside). Two non-obvious correctness points:

Dispatch-guard logging: state.bisecting (set for the duration of handleRedBatch) makes dispatchMergeTrainWorker log “bisecting red batch” for a re-dispatch attempt during bisection, rather than the misleading “CI red — needs attention”.

Runaway guard (ADR-059 D8): a composition-agnostic rate cap that fires when a (repo, base) partition creates ≥ MaxTrainTrialsPerWindow trial branches (default 20) with zero successful landings within a rolling TrainTrialWindowDuration window (default 60 min). The guard is distinct from the poison-well guard (which terminates the loop) and the bisect cost cap (which degrades to one-at-a-time within one goroutine run): it bounds the cross-poll-cycle burst that occurs when infra is broken and every trial fails. Since #1648, this fires per (repo, base) partition, not per repo — a maint/1.x partition’s runaway trip pauses only that partition’s own Queued members and never touches a healthy sibling main partition in the same repo.

Counter: assembleAndValidate is a thin wrapper around assembleAndValidateInner (the single site where all trial branches are created) that calls recordTrial(trainKey) only when the trial’s result is not TrainCIGreen (#1528) and not TrainCIInfra (#2052 — a trial abandoned because CI never started says nothing about the members, so it is not a trial for the guard’s purposes; see §6.29). A green result is, by construction, either the landing attempt itself or a bisection sub-trial that just proved a sub-batch clean — never a “zero successful lands” event, which is the guard’s entire premise. Before this fix, every trial counted unconditionally, including the green survivor-validation trial that confirms a bisection isolated its poisoner and is about to land — so a successful bisection could trip the guard on its own green trial, pausing the clean survivors it had just vindicated one second before they would have landed (deterministically for any 3-member poisoned batch under the default MaxTrainTrialsPerWindow=6 test configuration, since bisecting a 3-member batch down to a single poisoner takes exactly 6 raw trials). The exclusion is by-outcome, not by-origin: a red bisection sub-trial still counts, exactly as before — it represents genuine “no progress,” the same as a red top-level trial. TrainCIPending results and assembly errors also still count unconditionally, unchanged. The counter itself is a rolling window (slice of time.Time) keyed on the composite trainKey (mergeTrainKey(owner/repo, base) — #1648, was bare owner/repo), protected by mergeTrainTrialsMu. resetTrialCounter(trainKey) deletes the entry; it is called from both landMergeTrainBatch and landSingleton after a successful merge, so any train where survivors do land never accumulates toward the cap — and a landing on one partition never resets a sibling partition’s still-accumulating counter, since each has its own trainKey.

Guard fires: when isRunawayTripped(trainKey) returns true (count ≥ N within M minutes with zero resets), fireRunawayGuard is called. It:

Atomicity and per-episode idempotency (ADR-1533): fireRunawayGuard is called from three independent sites — twice inside runMergeTrainWorker (Hook 1, the worker goroutine) and once from routeQueuedGroup (Hook 2, the poll goroutine) — and nothing prevents Hook 1 and Hook 2 from running concurrently for the same trainKey once the shared trial counter trips: the poll loop does not check whether a worker is mid-firing. Each call site constructs its own, possibly-overlapping member slice from whatever local state it holds. Before #1533, this meant a member appearing in two racing calls’ slices could receive a duplicate alert, and — independently — a member whose AddComment call happened to fail was left fabrik:paused with no comment and no retry path, since a paused member permanently drops out of groupQueuedByRepoAndBase’s Queued snapshot on every later poll.

The fix makes the whole pause+alert sequence for one fireRunawayGuard call a single critical section, serialized by mergeTrainRunawayMu (a single, unsharded, engine-wide mutex — since #1648 this also serializes concurrent per-base workers within the same repo, not just across repos; same accepted rare-event trade-off, just a larger set of legitimate concurrent callers), so two concurrent calls can never interleave their loops. Within that section, mergeTrainRunawayAlerted (an in-memory map[string]int keyed "trainKey#N" — #1648, was "owner/repo#N" — recording the trial count at which the member was last alerted) records which members have already been alerted this episode; a call re-encountering an already-alerted member at the same or a lower count skips it entirely (no duplicate comment, no redundant label calls) — a strictly higher count means genuinely new trials ran since the last alert (only possible after an operator manually resumes a paused member without the episode having ended via a successful land), and falls through to a fresh alert rather than being skipped. resetTrialCounter — the guard’s only existing “this trip is over” signal, called on a successful land — also clears mergeTrainRunawayAlerted for the same trainKey, so the next trip starts a fresh episode without touching a sibling partition’s own alert history. The runaway-alert settle scan (runaway_alert_settle.go), working from a bare already-paused board item with no partition context, reconstructs the identical trainKey via mergeTrainKeyForItem — the sentinel for a no-label item (matching groupQueuedByRepoAndBase’s own zero-cost bucketing exactly), or a re-resolution via baseBranchForItem for a labeled one — so it can never disagree with the key fireRunawayGuard originally alerted under.

A member whose AddComment call fails is not marked alerted. Instead it is left with a new durable marker label, fabrik:awaiting-runaway-alert (the fabrik:paused application is unaffected — pausing never depends on the comment succeeding). settleRunawayGuardAlertScan (engine/runaway_alert_settle.go), a per-poll settle scan sourced directly from board.Items (not groupQueuedByRepoAndBase, which the member’s own fabrik:paused label already excludes it from), retries the alert every poll independent of any fireRunawayGuard call ever reaching that member again. After MaxRetries failed retries, escalateRunawayAlertFailure posts a fallback comment carrying the same explanation — but the marker is removed, and the member marked alerted, only if that fallback comment itself is confirmed posted; on a persistent AddComment outage that outlasts the fallback attempt too, the marker stays and the settle scan keeps retrying indefinitely, so the one remaining diagnostic signal is never erased out from under a member that still has zero delivered explanation (#1533 review, finding 1). See §6.18 and the Label Semantics Reference (§1.4) for fabrik:awaiting-runaway-alert’s full added-by/removed-by contract.

Two-hook dispatch: the guard is checked at two complementary points to cover both the active batch and any beyond-cap Queued members:

  1. Hook 1 (run-side, advance): after the initial re-form trial and after a bisection half-trial or a one-at-a-time singleton trial finds the guard tripped. Pauses the active batch members (membersToItems(survivors) — the survivors of the initial trial assembly, which are all members when the guard trips during bisect before any ejection).
  2. Hook 2 (poll-side, routeQueuedGroup): before dispatchMergeTrainWorker is called. If the partition’s counter is already tripped (from a prior worker run), pauses the current partition’s Queued snapshot (the uncapped partition items — every Queued member in that (repo, base) partition, not just the batch-capped subset that would have been dispatched) and skips dispatch for that partition only. This handles beyond-cap members that Hook 1 could not reach. Since #1648 partitions are formed by groupQueuedByRepoAndBase before this check ever runs, there is no longer a separate “exclude the excluded” step here (ADR-1647’s exception, which existed only because non-default-base members used to be excluded from batching rather than routed to their own partition, no longer applies — see ADR-1648). Queue-enabled members and batch-cap overflow are unaffected: both are still paused as before, scoped to their own partition.

Hook 1 and Hook 2 can run concurrently for the same (repo, base) partition — the “beyond-cap members cannot form their own batch while the worker goroutine is still active” argument bounds only whether a second train can form for that same partition, not whether Hook 2’s own guard check can race Hook 1’s. Correctness for the alerting side comes from mergeTrainRunawayMu and mergeTrainRunawayAlerted (above), not from an assumption that the two hooks never overlap. A sibling partition’s Hook 1/Hook 2 pair is entirely independent — its own trainKey, own counter, own critical section contention only through the shared mergeTrainRunawayMu.

Operator resume: once the runaway guard fires for a partition, all affected members have fabrik:paused and fabrik:awaiting-input applied and are therefore excluded from groupQueuedByRepoAndBase on every subsequent poll. No re-formation occurs for that partition until a human manually removes both fabrik:paused and fabrik:awaiting-input from each affected member — a sibling base partition in the same repo is entirely unaffected and continues normally. Before resuming, the operator should investigate the infra root cause (GitHub Actions billing, required-check configuration, base-branch health). The in-memory trial counter resets to zero on engine restart, but fabrik:paused labels persist on GitHub — so a restart alone does not re-enable the train; the labels must be cleared explicitly.

fireRunawayGuard is deliberately exempted from the #1545 reroute fix (R5 audit, below): unlike ejectRedSingleton’s pre-#1545 defect, a Queued member paused by the runaway guard is not stranded — groupQueuedByRepoAndBase/routeQueuedGroup re-admit an unpaused Queued member directly from board.Items on the very next poll, entirely independent of itemMayNeedWork/processItem’s HoldingStage exclusion (that exclusion only blocks per-item stage dispatch, not the batch handler’s own board read). Rerouting every paused member to Validate instead would force each one through a full, redundant Validate re-run for a cause (infra-wide: billing, a broken base branch, all required checks erroring) no code stage can fix, and would make each member re-earn its way back into Queued via a fresh stage completion instead of simply resuming the batch it had already validated into.

Pause-in-Holding-Column Audit (#1545)

Any site that applies fabrik:paused to an item while its board Status is still a HoldingStage column (Queued) has the same structural defect ejectRedSingleton had before this issue: itemMayNeedWork excludes HoldingStage items from dispatch, processItem (the comment-unpause path) is never reached, and settleQueuedReviewFindings applies the identical closed/fabrik:paused exclusion — so nothing can ever act on the pause without a human manually moving the board card. Every such site was audited:

Site Disposition Reason
ejectRedSingleton (standalone-validation-failure pause) Fixed; mostly superseded by auto-repair (#2045) Rerouted off Queued via rerouteQueuedMemberOffHolding before pausing (R1/R2, above). Since #2045 the pause is the fallback only (cap reached, cap 0, non-Validate target); the default path re-enters Validate without pausing (§6.31).
fireRunawayGuard (member-pausing path) Exempted Its recovery path is already reachable: groupQueuedByRepoAndBase/routeQueuedGroup re-admit an unpaused Queued member directly from board.Items, bypassing the HoldingStage dispatch exclusion entirely (see the Runaway guard section above). Rerouting the whole paused column to Validate would force every member through a redundant re-run for a cause (persistent infra failure) no code stage can address.
ejectMember’s cap-reached pause (MaxMergeTrainEjections) Exempted — out of scope Governs the counter-driven bisection/conflict ejection ladder, explicitly excluded from this issue’s Scope. stayInQueue=true for every cause it serves, so the member — paused or not — was always meant to stay in Queued for a future differently-composed train; that design is unchanged by this issue.
checkAutoMergeConvergence’s fabrik:paused sites (engine/merge_gate.go) Exempted — structurally unreachable Its only call site, handleAutoMergeConvergence, runs from the Phase 1 catch-up loop, which unconditionally excludes HoldingStage items before it is ever reached — so it cannot fire while an item’s board Status is still Queued. An ADR-058-enqueued item that stays at Status == "Queued" is drained by settleClosedItemsToDone once closed (regardless of column), or simply sits, unpaused, in Queued until GitHub’s native merge queue merges and closes it.

References: ADR-1545 (this issue’s R3 design record), ADR-1208: Queued Review-Finding Ejection (the reroute-before-side-effects precedent), ADR-1420: Merge-Train Ejection Diagnostics, issue #1545.

Merge-Train Run State, Per-Poll Evaluation and Restart Resume (#2051, ADR 2051)

Supersedes the worker model of ADR-059 D3/D5; amends ADR-067, ADR-1208 (claim lifetime), ADR-1222, ADR-1835, ADR-2050. Files: engine/train_run.go (record + store), engine/train_bisect.go, engine/train_step.go (state machine), engine/train_run_poll.go (per-poll evaluation, adoption, phase source), engine/train_ci_eval.go (evalTrialCI).

The run. One run is one merge-train episode of one (repo, base) partition. Its state is a trainRunRecord: train key, owner/repo, partition base, resolved base branch, pinned BaseSHA, project ID, the trial-name sequence (BaseTrialName, TrialSeq), the step (below) and when its phase began, the dispatched batch, the members still in play (number, PR, head SHA — the board item is re-attached from the live board, never stored), the main loop’s Current set, the one open trial (kind main/half/single, name, head SHA, draft CI PR, member numbers, OpenedAt, absolute Deadline, and the persisted CI memory — the #2052 startup-watch and failed-job re-run state), the bisectState, the one-at-a-time cursor (Members, Index) and a write-ahead Poisoner mark.

Steps. form (top of the re-form loop) → trial-ci (main trial open) → bisect (between half-trials) ⇄ bisect-trial (a half open) → one-at-a-time (between singletons) ⇄ one-at-a-time-trial (a singleton open) → landing. The phase exposed to readers is the spec vocabulary — trial-ci, bisect, one-at-a-time, landing — derived from the step.

One state machine, two drivers. advance(run, verdict) holds all the logic that used to sit between the blocking CI waits of runMergeTrainWorker: the singleton fast path and catch-up, runaway Hook 1, pending-eject Hook 2, green → landGreenBatch, red singleton → ejectRedSingleton, red batch → bisection, pending/infra → abandon, and the one-at-a-time loop. It runs from the current step to the next point where it must wait for a trial’s CI, opens that trial (assemble + draft PR — openTrial), persists, and returns awaiting. Two drivers run it:

Per-poll evaluation (R2). For each live run that no goroutine is stepping, settleTrainRuns evaluates the open trial with evalTrialCI — the loop body of pollTrainCI, extracted so both share every decision (CI classification, ciSuiteHold, required contexts, the #2052 retrigger and the single failed-job re-run with its stale-failure guard). API reads only: no goroutine, no Engine.sem slot. The evaluator’s memory (trialCIState) is mirrored into the record whenever it changes, so a retrigger or re-run budget already spent survives the next poll and a restart. Once a verdict exists the scan launches a short step goroutine for advance(verdict) (liveness marker set, the run’s stepping flag held). A run positioned between trials (adopted after a restart) gets its next step the same way. pollTrainCI remains for the synchronous driver and the landing-internal waits.

Deadline (R2/FR-010). The trial’s Deadline is e.now() + ciBackstopTimeout() at open, persisted. The poll compares it with e.now() (so the sim clock drives it); the trial’s CI is always read first (a trial that finished while the daemon was down is never abandoned for age alone); on expiry with no verdict it logs trial … has no CI verdict … — treating it as timed out and feeds TrainCIPending to the existing pending handling (clean up artifacts, members stay Queued, episode abandoned, record removed) — detected within one poll of the deadline. There is no per-trial liveness signal (a trial PR’s check runs are never attributed to a board item, so LastCIProgressAt does not exist for it).

No double advance (FR-011). A run has a stepping compare-and-swap flag: a poll evaluation, a step goroutine and a concurrent poll are mutually exclusive owners of the run. The creating worker holds stepping before the run is registered, so a concurrent poll can never start a second step over a run its worker still owns. A step that ends because shutdown cancelled its context keeps the persisted record (the in-memory run is released, the file is not), so the next daemon adopts the bisection position and the spent cost budget; every other end of a run removes it.

Write-ahead. advance persists the post-verdict state before the side effect it licenses: the trial is recorded before anyone waits on it; the bisection progress (Used, NextHalf, Red) and “no open trial” before the next half is opened; the Poisoner mark before ejectMember. A verdict is a pure function of GitHub’s CI state and so is recomputed after a crash. A record with no open trial means “open the next trial”. A restart during landing drops the record: the durable reconstructTrainState routes finish the landing idempotently (ADR-1871).

Scope of “per poll”. Only trial CI waits are per-poll: the batch’s main trial, bisection half-trials and one-at-a-time singleton trials. Waits inside a landing path stay in the (short) step goroutine that runs it: landGreenBatch’s rebase revalidation, pollForMergeable in landMergeTrainBatch/landSingleton, and the ADR-2044 catch-up waitMemberCI (which needs the pinned base). Landing paths and CI classification are otherwise unchanged.

Restart adoption (R3, FR-008/FR-009). On start New() creates the store over .fabrik/state/merge-train/ (one <hash(trainKey)>.json per partition, written atomically through a temp file + rename). The first settleTrainRuns loads the directory — a file that does not parse or has an unknown version is renamed to <file>.corrupt and treated as absent — and adopts each record against the board snapshot (adoption needs it, so it is not done in Run()). All of the following must hold, else the record is discarded (logged with the reason, its trial PR closed and its branch deleted so reconstructTrainState Route 2 cannot find and land a stale trial that still carries a departed member’s commits, the file removed; a trial that was landing or is already merged is left alone for Routes 1/2 to finish) and the partition forms fresh through the unchanged reconstructTrainState routes: the step is not landing (the main trial’s green landing and each green one-at-a-time singleton set it, write-ahead, before landSingleton/landGreenBatch run); every live member is still on the board, in the holding stage, open and not fabrik:paused, with an unchanged linked-PR head SHA and a live Status (liveLandingState, ADR-1871) that is not “moved”; the trial PR (FetchPRDetails) is open at the recorded head SHA (a merged trial PR is deliberately left to Route 1); the pinned BaseSHA still resolves in the bare clone. The base having moved is not a failure — landGreenBatch’s trialBehind / rebase cycle handles it at landing. Members that assembly ejected from a trial (Ejected in the record — they remain in the pre-assembly Current/Origin sets) are exempt from the Queued requirement and are dropped from Current, so an assembly-time ejection before the restart does not cost the run. Transient failures (repo not ready, PR read error, live status read error) defer adoption to the next poll, and while a record awaits adoption dispatchMergeTrainWorker forms no train for the partition. An adopted run re-registers the claim, rebuilds the TUI episode (phase and phase start restored — noteTrainPhaseAt), and carries on from its step: a restart mid-trial evaluates the same trial PR and lands it; a restart mid-bisection opens the next pending half in the same order, with the trials already spent still counting against the cap. A poisoner marked but already off Queued is dropped from the record (its ejection completed before the restart); one still Queued is ejected once.

Phase source (R5, FR-012). Engine.TrainRunPhases() returns, per open run, {TrainKey, Phase, Step, Position, Of, Since, TrialPR, Members} read from the record — no goroutine involved. (Position/Of is the validation about to run out of the cost cap for a bisection, and the member index out of the fallback list for one-at-a-time.) The TUI row and board status line keep going through the single noteTrainPhase hook; syncRunPhase feeds it from the record on adoption.

Prefix cache (ADR-1835). The trial-assembly prefix cache now belongs to the run, not to a goroutine, so a bisection that spans several step goroutines still reuses its prefixes; a restart loses it (git replays only).

What stays in memory (and so resets on restart, unchanged): the runaway guard’s trial timestamps, ejection counts, auto-repair and catch-up caps, the overlap cache, the policy and infra-cooldown memos. After a restart the TUI episode’s outcome facts cover only the post-restart part of the run.

Not a regression to note: a pending verdict on a bisection half-trial is not red — it falls through to the next half and finally the fallback — exactly as before; the parity oracle (engine/train_bisect_parity_test.go) pins it.

Merge-Train Serialization + Main-Moved Recovery (ADR-059 D5)

The batch validates a combined batch against main once; its correctness rests on the main the batch validated against being the main it lands on. Serializing the train (one batch in flight per repo) keeps main from moving under a validating batch; the only remaining mover is an external direct push by a human, which is detected and recovered. D5 adds two things without weakening the atomic LoadOrStore guard on mergeTrainInFlight.

Durable, restart-safe in-flight reconstruction (FR-1/FR-4). dispatchMergeTrainWorker keeps LoadOrStore as the sole race-critical, in-memory check (dispatch runs only from the single poll goroutine). Because that map is lost on restart, prepareTrainWorker calls reconstructTrainState inside the already-guarded goroutine, before pinning the base. (Since #2051 this is the fallback: a persisted run record, when valid, is adopted first — see §”Merge-Train Run State” — and only an absent, unusable or invalid record reaches these routes.), probing only durable artifacts — merge-train PRs via ListPRs and origin trial branches via WorktreeManager.ListTrainBranchesOnOrigin() (git ls-remote --heads origin refs/heads/fabrik/merge-train/*). A PR is recognized as a train PR by isTrainPR — structurally, by its fabrik/merge-train/* head branch (HeadRefName) alone (R7, #1615): every genuine trial PR, landing PR or unpromoted draft CI PR alike, is Fabrik-created on that branch. The batch marker (<!-- fabrik-merge-train-batch -->, carried by a landing PR, never by a draft CI PR) is corroboration only — reconstructTrainState logs when it’s absent on a branch-matched PR, but never treats its presence or absence as identity. Before #1615’s R7, isTrainPR accepted the marker or the branch — a PR whose body merely quoted the marker literal in prose (with nothing to do with any trial) was misidentified as a train PR; this is what let the sweep close a live, unrelated PR during reconstruction for a different batch, an incident that hit this very fix’s own PR (see R8 below).

Relevance filtering — the critical guard. ListPRs returns state=all, so it also surfaces merged/closed integration PRs from prior completed batches; those still carry the batch marker but have no members left in today’s Queued snapshot (members only leave Queued on a successful land). Reconstruction therefore selects the first train PR whose parsed members still intersect the current Queued batch (filterBatchByNumbers(batch, parseTrainMembers(body)) non-empty). A historical PR that fails this test is skipped — routing on it would wrongly abort today’s fresh batch (complete-deferred would find no still-Queued members and exit), permanently stalling the train after the first landing. A stale open train PR that fails the test is closed (and its branch cleaned) so it cannot later hijack findIntegrationPR during a fresh batch’s landing — but only after a second, deliberately redundant structural check immediately before the CloseIssue call (R8, #1615): closing a PR is destructive and irreversible, so the identity that authorizes it is re-confirmed at the point of the action itself, not only inherited from the isTrainPR filter above it in the loop. A PR that reaches this point without a train-branch head ref is skipped and logged instead of closed — ambiguity fails closed, never toward a destructive action. From the selected relevant PR (or none), it routes to:

Reconstruction reads only durable state (never the map) and never launches a goroutine, so a restart with an empty map resumes/completes/dissolves an existing train rather than starting a duplicate. The ls-remote branch probe is gated under the trainValidateFn test seam (membership-keyed tests drive routing through listPRsFn).

Main-moved (behind) recovery at the landing gate (FR-2/FR-6). The green case calls landGreenBatch, which — before merging — checks whether the validated-green trial branch has fallen behind its base via trialBehind (FetchCommitsBehind(base, trialBranch) > 0; a probe error is treated as up-to-date, fail-safe). MergeableStateAccepted stays narrow — behind is not widened into it; it gets this dedicated branch. If up-to-date, landGreenBatch delegates to the unchanged landMergeTrainBatch. If behind (an external push advanced main), it re-pins the base to the current origin/<base> and re-assembles the survivors off the new base via assembleAndValidate (which reuses resolveConflictWithClaude for FR-6 rebase conflicts), re-runs the combined Validate, and loops back to the landing gate — bounded by MaxTrainRebaseCycles (default 3; effectiveMaxTrainRebaseCycles). On a green re-validation it loops back to the gate; on exhaustion, a non-green re-validation, or an assembly wipeout it dissolves the batch.

Dissolve semantics (FR-5). dissolveBatch closes the integration/CI PR if open (via CloseIssue — PRs are issues; there is no ClosePR), deletes the trial branch locally and on origin (cleanupTrialArtifacts), clears the mergeTrainInFlight entry (and, via finishTrain, the itemstate.Store repo-worker liveness marker), and posts an explanatory comment on each member so the outcome is observable. Members are left untouched in the Queued column — they only advance to Done on a successful land, so no status rollback is needed. It is idempotent (safe to re-run after a crash mid-dissolve: CloseIssue on an already-closed PR and cleanup on an already-deleted branch are best-effort no-ops; the comment may double-post — acceptable and observable). The next poll re-snapshots Queued and forms a fresh train.

Compose-not-duplicate with bisection. Main-moved rebase recovery lives only in landGreenBatch (the green landing path); a red re-validation after a rebase dissolves rather than entering bisection. The rebase-cycle budget (MaxTrainRebaseCycles) and the bisection cost cap (MaxBisectValidations) are therefore disjoint and never double-count — the next poll re-forms a fresh train that bisects cleanly.

Per-repo isolation (FR-3). Serialization is keyed strictly owner/repo, so trains for different repos run concurrently; the per-repo guard never cross-blocks distinct repos.

Worker-slot scope (#2046, ADR 2046). A train holds no Engine.sem slot for its lifecycle. prepareTrainWorker and runMergeTrainWorker acquire none; assembly git work, PR operations, the per-poll trial evaluation (evalTrialCI), bisection bookkeeping and landing all run slot-free, so the train is not counted against MaxConcurrent while a trial waits on CI — since #2051 it holds not even a goroutine then. The only slot use is resolveConflictWithClaude, which brackets exactly the InvokeForComments call via acquireTrainSlot (non-blocking try first; a real wait is announced once through logfRepo as waiting for a free worker slot for conflict resolution on #N, so the repo’s job row shows it, ADR-1661) and releases it as soon as the call returns — before finalizeConflictResolution. This one site covers every path that reaches Claude: assembleTrialBranch (the main re-form step, bisection half-trials, the one-at-a-time fallback, landGreenBatch’s rebase loop) and the singleton catch-up merge (runCatchUpGit, §6.30). Rerere replay and the regenerate-only branches never take a slot. The usage-limit hold is checked before the wait and again after it. The wait is bounded only by the context, not by the invocation’s wall-time cap.

Reserved train slot (#2115, ADR 2115). With merge_train: on, New() also creates Engine.trainSem (capacity 1), so the host can run MaxConcurrent + 1 Claude processes; with it off the channel is nil and nothing changes. acquireTrainSlotEp is its only user: it tries trainSem (non-blocking), then a free Engine.sem slot, and only when both are busy logs the wait, enters the waiting for slot phase and blocks on either (or the context). Release returns the pool actually taken and stays sync.Once-idempotent. Stage dispatch, the reinvoke dispatchers and comment processing never reference trainSem, so the reserved slot is invisible to them. The #2045 auto-repair Validate is an ordinary dispatch on the general pool. Local API workers carries train_in_use/train_max; the TUI title shows slots a/b + train c/d from a sampled WorkerSlotsEvent.

Cancellation is not a verdict (#2046). A context cancellation while waiting for the slot (or killing an in-flight conflict-resolution invocation) returns the non-nil “not attempted” error (the same channel as a usage-limit exit): the member is not ejected, nothing is paused, the runaway trial counter is not charged (assembleAndValidate skips recordTrial when trainCancelled), bisect aborts the episode instead of degrading to one-at-a-time, landOneAtATime stops, landGreenBatch’s rebase loop returns without dissolveBatch, and the catch-up returns catchUpDefer. Members stay Queued. The in-flight marker is still cleared only by the two finishTrain defers (ADR-067).

References: ADR-1615: Structural Identity for Destructive Merge-Train Actions.

1.4 Label Semantics Reference

Label Added By When Added Removed By When Removed Gates
fabrik:locked:<user> processItem Before stage invocation (lock-then-verify protocol) releaseLock On stage completion, permanent failure, blocked-on-input, or lock conflict loss Prevents other instances from processing the item
fabrik:editing processComments Step 2 of comment processing processComments Step 9 of comment processing (also on error paths). Removal uses bounded retry (≤3 attempts, 500ms/1s/2s backoff) for transient network errors; ErrNotFound is a silent no-op. Stale labels with no active Worker are cleaned up at startup by runStartupCleanup(). Pre-dispatch gate in itemNeedsWork (prevents goroutine launch); defense-in-depth check retained in processItem for the race window. Symmetric with fabrik:locked:<other-user>. Note: JobStartedEvent is emitted at processComments entry (step 0) — before fabrik:editing is added at step 2. The pre-dispatch gate only blocks new dispatches; the active session’s JobStartedEvent fires before the label exists.
fabrik:reworking:<Stage> beginStageRework, called immediately after the Step 2 fabrik:editing add succeeds Only when the re-entered stage’s own stage:<Stage>:complete is already present on item.Labels — the common mid-flight-rework case (nothing to clear yet) is a no-op. Added before stage:<Stage>:complete is removed (mark-first-then-clear) so the marker’s presence is always a safe, idempotent restore trigger regardless of which of the two mutations actually landed before a crash (#1802) endStageRework, called at every exit path (the two setup-failure early returns, the err != nil && !completed block, and finalizeComments) On a non-completing exit: removed immediately, right after stage:<Stage>:complete is restored directly. On a completing exit: removal is deferred until after finalizeComments’s call to handleStageComplete returns (see the stage:<X>:complete row below) — not alongside it — so a crash anywhere during handleStageComplete’s own completion write still leaves the marker present as a durable “restore owed” signal. Stale copies with no active Worker are repaired at startup by runStartupCleanup()’s third pass, which reads the reworked stage’s name directly out of the fabrik:reworking:<Stage> label suffix — never from the item’s current board Status (a Pruefer review finding, #1802: a completing rework that also auto-advances moves Status to the next stage before the marker is removed, so a Status-keyed restore could stamp stage:<NextStage>:complete onto a stage that never ran). If stage:<Stage>:complete or fabrik:awaiting-ci is already present for that named stage (the rework settled correctly before the crash), only the marker is removed; otherwise stage:<Stage>:complete is restored first Strictly nested inside the fabrik:editing bracket — itemNeedsWork’s fabrik:editing gate already makes the transient stage:<Stage>:complete clear unobservable to any dispatch decision, so this label changes no gate (R4); it exists purely so the board’s label set reflects an in-progress rework instead of a stale completion claim. See ADR-1802
fabrik:paused escalateFailedStage, blockOnInput, pauseForReviewTimeout, pauseForReviewCycleLimit, pauseForCITimeout, pauseForCIFixCycleLimit, pauseForRebaseCycleLimit, attemptMergeOnValidate (on ErrNotMergeable rebase cycle limit reached, or CI wait timeout), handleStopRequest (TUI manual stop) and runShutdownPause (daemon clean-stop, ADR-1393) — both via the shared pauseInterruptedIssue primitive, tripCommentBreaker (comment-processing circuit breaker, #1089, §4.6) After MaxRetries, FABRIK_BLOCKED_ON_INPUT, review/CI/rebase timeout or cycle limit, TUI s-key stop, a daemon-wide clean stop (SIGINT/SIGTERM, ADR-1393), or N non-advancing comment-processing invocations within the configured window (§4.6) User (manual removal), or processItem (on new human comment that triggers unpause) When user removes it manually, or a human comments on a paused issue Blocks processing on active stages; a human comment is an implicit resume — bot-authored comments (github.IsBotLogin) and Fabrik’s own output do not resume (resumeAuthorised, #1083), nor does a human comment that predates the pause (#1813). Cleanup stages ignore comments entirely while paused. A Validate-stage item with an externally merged PR can still advance via runValidatePRTerminalAdvance regardless of this label. tripCommentBreaker applies this label through the same pauseIssue helper as every other pause source, so the human-only resume guarantee applies uniformly (§4.6). Unchanged semantics for the daemon clean-stop writer (ADR-1393 R6) — as of #1379 also suppresses deep-fetch admission.
fabrik:awaiting-input blockOnInput, pauseForReviewTimeout, pauseForReviewCycleLimit, pauseForCITimeout, pauseForCIFixCycleLimit, handleStopRequest (TUI manual stop) and runShutdownPause (daemon clean-stop, ADR-1393), tripCommentBreaker (comment-processing circuit breaker, #1089, §4.6) After FABRIK_BLOCKED_ON_INPUT, review/CI timeout/cycle limit, TUI s-key stop, a daemon-wide clean stop, or a comment-processing circuit-breaker trip (§4.6) unblockAwaitingInput; handleStageComplete (on FABRIK_STAGE_COMPLETE, to clear any orphaned label); handleNoWorkNeeded (on FABRIK_NO_WORK_NEEDED); cleanupClosedIssueTransientLabels (defensive sweep) When a human comment arrives (bot comments do not resume); when a stage completes (removes any orphaned label that survived a manual fabrik:paused removal); or when issue is closed (defensive sweep — label has no meaning on a closed issue) Combined with fabrik:paused, identifies the “awaiting user input” pause variant
fabrik:awaiting-review handleStageComplete (Path 1 — only when wait_for_ci: false), checkReviewGate (Path 2), reviewGateBlocksLanding via attemptMergeOnValidate (Path 3 — the landing-decision gate, §6.6.6) Path 1: optimistically after stage completion when wait_for_reviews: true and wait_for_ci: false (does not check reviewer state — data is stale; omitted for wait_for_ci: true stages because Path 2 handles the gate after CI clears). Path 2: when LinkedPRReviewRequests is non-empty OR when len(outstanding)==0 && !hasReviews (the bot self-submission case — covers Copilot/Gemini-style reviewers that don’t appear in the formal requested-reviewer list but still need to submit a review). Path 3: immediately before any landing action (auto-merge enable, enqueue, direct merge, advance-to-Queued) when a live FetchPRReviews/FetchPRReviewRequests read shows the same clearing condition unmet — including when any fetch errors (either review fetch, or the FetchLinkedPR PR-number fallback), which blocks conservatively rather than clearing on unknown state. All three paths are idempotent and non-conflicting; Path 3 blocks and labels only, and never runs escalation or the timeout (those stay with checkReviewGate) checkReviewGate (both natural clear and timeout paths); cleanupClosedIssueTransientLabels (defensive sweep, non-gate-checked stages only — see below) When all reviewers submit, or when timeout elapses (removed by checkReviewGate before pauseForReviewTimeout is called); or when issue is closed at a non-gate-checked stage (defensive sweep). Excluded from the sweep at a gate-checked stage (Validate, ADR-1387, R6) — there, the settle-owner pair (runValidatePRTerminalAdvance/settleClosedValidateAdvance, §6.15) clears it atomically as part of its own transition instead Phase 1 / Phase 2 reprompt timers in checkReviewGate fire on label-applied-at age (not on updatedAt movement). A non-responsive bot reviewer produces no comment / no review / no PR activity, so updatedAt never moves — without periodic re-evaluation the timers would never get a chance to fire. The catch-up loop’s blocked-path records CooldownAt("review-blocked") (via CooldownRecorded{Reason: "review-blocked"} mutation) so itemMayNeedWork’s cooldown retry path re-admits the item every 10 × PollSeconds (same pattern as fabrik:blocked); a per-poll cache bypass is intentionally avoided because long-lived review-waiting items would otherwise become a permanent GraphQL hot path. Blocks auto-advance until review gate clears
fabrik:awaiting-ci handleStageComplete (on FABRIK_STAGE_COMPLETE for wait_for_ci: true stages; idempotent); checkCIGate (on confirmed CI failure; idempotent) handleStageComplete: immediately on FABRIK_STAGE_COMPLETE — replaces premature stage:X:complete and keeps the item in the CI-await window (ADR 032). checkCIGate: when CI check runs for the PR head SHA have conclusion: failure/timed_out/action_required. checkCIGate (when mergeable_state == "clean", when the check-run/required-context classification chain reports all-green — including for an unstable PR whose observed checks all passed, ADR-1441 — or when gate times out); attemptMergeOnValidate (when the mergeable_state == "clean" shortcut fires, ADR-1441); cleanupClosedIssueTransientLabels (defensive sweep) When GitHub’s mergeable_state is "clean" (v0.0.52 shortcut, narrowed by ADR-1441/#1441 to clean only — unstable no longer shortcuts here, see below); when all CI checks pass (green) under the per-check classification fallback (this is now the only path unstable can clear through); when timeout elapses (removed before pauseForCITimeout is called); or when issue is closed (defensive sweep) Signals CI gate is active (pending or failed); triggers itemMayNeedWork updatedAt cache bypass; suppresses dispatcher re-invocation (itemNeedsWork returns false); blocks auto-advance until CI gate clears. stage:X:complete is absent while this label is present — it is added by checkCIGate when CI clears (R5) or when the mergeable_state == "clean" shortcut clears the gate (v0.0.52/ADR-1441). A confirmed check-run failure on an unstable PR (non-required check red) now applies this label and blocks, instead of the pre-ADR-1441 shortcut clearing it outright — see §2.10 and ADR-1441.
fabrik:rebase-needed checkMergeabilityGate (catch-up loop, wait_for_ci: true stages); checkAutoMergeConvergence (convergence flow, yolo+Validate with fabrik:auto-merge-enabled) When GitHub reports mergeable == false on the linked PR — a confirmed base-branch conflict. Applied idempotently in both paths. NOT added when mergeable == null (GitHub still computing). checkMergeabilityGate (when mergeable flips back to true); checkAutoMergeConvergence (when PR merges or closes); handleValidateSHAInvalidation (SHA-invalidation scan, §2.16 — the conflict determination was tied to a completion SHA that no longer exists); cleanupClosedIssueTransientLabels (defensive sweep) When GitHub reports mergeable == true (after Claude’s rebase push lands), or when PR merges/closes; when the linked PR’s HEAD SHA changes after stage:Validate:complete was recorded, invalidating the prior mergeability determination (#1225); or when issue is closed (defensive sweep) Signals confirmed merge conflict; triggers itemMayNeedWork updatedAt cache bypass (base-branch advances don’t bump the item’s updatedAt); blocks CI gate and auto-advance until rebase resolves the conflict. Never applied for a MergePR CI refusal (gh.ErrNotMergeableCI) — none of MergePR’s four call sites read its returned error to drive this label or increment a rebase cycle; see “MergePR’s own CI precondition (ADR-933)” in §5
fabrik:claude-limit handleUsageLimitExit On a Claude invocation that exits because the account’s usage limit was hit (claudeUsageLimitError, detected structurally in interpretClaudeResult/classifyUsageLimitExit from resp.TerminalReason == "blocking_limit", or resp.TerminalReason == "api_error" with api_error_status == 429 (§7.3a) — never from output prose), gated on the label’s own absence (idempotent — comment posted only on the absent→present transition) The non-limit path in finalizeStageOutcome, unconditionally, immediately after the usage-limit branch — fires for success, blocked-on-input, no-work-needed, genuine failure/retry, and PR-creation failure alike; settleClaudeLimitLabelSweep (account-wide, once the usage-limit hold lifts); cleanupClosedIssueTransientLabels (defensive sweep, via transientLifecycleLabels) On the next invocation that is not itself classified as a usage-limit exit; account-wide, every poll, once claudeUsageLimitedUntilTime reports no active usage-limit hold; or when issue is closed (defensive sweep) Distinct from GitHub’s own rate-limit terminology (engine/backoff.go/engine/terminal.go) — this is the Claude account usage limit, not the GitHub GraphQL budget. StageAttempted is recorded (so the normal dispatch cooldown applies, preventing a tight retry loop against the limit) but StageRetryIncremented is deliberately never called — the stage never ran, so this does not count against max_retries. Neither stage:<name>:failed nor fabrik:paused is applied. See §7.3 and ADR-1119, ADR-1183.
fabrik:clear-claude-limit operator Applied manually to any open board item to clear an active account-wide Claude usage-limit hold without an engine restart settleClaudeLimitClearRequests, the same poll cycle it was observed on Consumed immediately — the scan clears the usage-limit hold once and removes the label from every carrying item in the same pass One-shot command label, mirroring fabrik:revalidate. Not scoped to items also carrying fabrik:claude-limit — the usage-limit hold it clears is account-wide, not per-issue. See §7.3 and ADR-1183.
fabrik:api-key-helper-detected handleAPIKeyHelperDetected On a Claude invocation skipped because the worktree’s own .claude/settings.json or settings.local.json sets apiKeyHelper (apiKeyHelperDetectedError, returned by runInvocationWithExtension before Claude is ever invoked — a repo-resident setting Fabrik cannot see until the worktree exists, distinct from the startup-time checkAPIKeyHelper preflight covering the managed-policy/user/fabrikDir-project layers), gated on the label’s own absence (idempotent — comment posted only on the absent→present transition) The non-detection path in finalizeStageOutcome, unconditionally, alongside the fabrik:claude-limit clear — fires for success, blocked-on-input, no-work-needed, genuine failure/retry, and PR-creation failure alike; cleanupClosedIssueTransientLabels (defensive sweep, via transientLifecycleLabels) On the next invocation that is not itself classified as a usage-limit exit or an apiKeyHelper detection; or when issue is closed (defensive sweep) Mirrors fabrik:claude-limit’s “stage never ran” shape exactly: StageAttempted is recorded (normal dispatch cooldown applies) but StageRetryIncremented is deliberately never called — does not count against max_retries. Neither stage:<name>:failed nor fabrik:paused is applied. Unlike fabrik:claude-limit, there is no account-wide settle sweep — the condition is inherently per-worktree, self-resolving once a human removes apiKeyHelper from the repo. See ADR-1346, R13.
fabrik:skill-unresolved handleSkillUnresolved (engine/skill_unresolved.go), from finalizeStageOutcome (stage path, via skillUnresolvedError returned by runInvocationWithExtension) and processCommentsClassified (comment/CI-fix/rebase/review-reinvoke path) When a stage’s skill reference resolves to no loaded plugin (§7.18), gated on the label’s own absence (idempotent — comment posted only on the absent→present transition) The non-denial path in finalizeStageOutcome (any invocation that reached Claude), and clearSkillUnresolved clears the panel entry when the reference resolves again; cleanupClosedIssueTransientLabels (defensive sweep, via transientLifecycleLabels) On the next invocation that reaches Claude; or when issue is closed (defensive sweep) Self-healing did-not-run, mirroring fabrik:api-key-helper-detected: StageAttempted recorded on the stage path, StageRetryIncremented never; neither stage:<name>:failed nor fabrik:paused. On the comment path the result is didNotRunSkillUnresolved (reinvoke cycle refunded, no breaker charged). Surfaces in the attention classifier as needs-human. See §7.18, ADR 2137.
fabrik:tools-denied Inline in finalizeStageOutcome’s final else branch (the toolsDenied case) — the stage-dispatch path; and, identically (#1704), a mirroring errors.As(err, &toolsDeniedErr) branch in processComments’s error handling — the shared funnel for ordinary comment review and all three reinvoke dispatchers (dispatchCIFixReinvoke, dispatchReviewReinvoke, dispatchRebaseReinvoke). Both call the same recordToolsDeniedDetection helper, so the label/comment/counter bookkeeping cannot drift between the two consumers of interpretClaudeResult’s shared classification On a Claude invocation whose tool call(s) were denied by the CLI’s own permission layer (claudeToolsDeniedError, detected structurally in interpretClaudeResult’s clean-exit path from a non-empty resp.PermissionDenials array, gated on !completed — never from output prose), gated on the label’s own absence (idempotent — comment posted only on the absent→present transition) The non-denial path in finalizeStageOutcome, gated on !toolsDenied (unlike fabrik:claude-limit/fabrik:api-key-helper-detected, this invocation itself may BE the condition — the classification does not short-circuit, so the clear must not fire on the very detection it would erase); cleanupClosedIssueTransientLabels (defensive sweep, via transientLifecycleLabels) On the next invocation that is not itself classified as a tools-denied exit; or when issue is closed (defensive sweep) Structurally unlike its two siblings: the CLI exits cleanly (is_error: false, terminal_reason: "completed") and real work may have happened before the denial, so this does NOT short-circuit finalizeStageOutcome — commitWIP, the branch push, and markCommentsSeenByStage all still run, mirroring claudeTurnLimitError/claudeResumeFailureError’s continue-processing shape rather than the did-not-run family’s early return. StageAttempted is recorded (normal dispatch cooldown applies) but StageRetryIncremented is deliberately never called — does not count against max_retries. Bounded instead by its own ToolsDeniedRetries/MaxToolsDeniedRetries counter (default 3) — shared across both consumers, so a denial on ci-fix-reinvoke and a denial on the stage path accumulate against the same per-stage counter; at the bound, pauseForToolsDeniedLimit applies fabrik:paused + fabrik:awaiting-input (never stage:<name>:failed — the condition is never treated as a stage failure, even at the bound), reused unforked from either call site. Because classification is expressed as a non-nil error from interpretClaudeResult, blockedOnInput := err == nil && ... is structurally unreachable for a detected denial — the outcome is identical whether or not the worker also emits FABRIK_BLOCKED_ON_INPUT. On the comment/reinvoke path, the new branch sits ahead of checkNoOpCommentCycle/checkCommentBreaker and returns before either runs, so a mode denial is bounded exclusively by MaxToolsDeniedRetries, never by the comment breaker’s frequency-based thresholds (which stay unchanged). See §7.3b, §7.3c, and ADR-1523/#1704.
fabrik:toolchain-stale checkToolchainDrift (engine/toolchain.go) — called from both runInvocationWithExtension (stage dispatch, immediately after the apiKeyHelper check) and processComments (comment review, right after context files are written) against the same shared implementation, so the two invocation paths cannot drift (#1786) On a comparable declared-vs-resolved toolchain mismatch (.nvmrc, package.json engines.node, go.mod toolchain/go directive, or .tool-versions, compared against the binary version actually resolved on the daemon’s own inherited PATH via node --version/go version), gated on the label’s own absence (idempotent — comment posted only on the absent→present transition). Detection never returns an error and is never routed through finalizeStageOutcome — unlike fabrik:claude-limit/fabrik:api-key-helper-detected, it never skips the invocation; the stage/comment-review call proceeds exactly as it would have regardless of what it finds checkToolchainDrift itself, the next time it runs for that issue and finds no comparable mismatch (declaration file fixed, or the daemon restarted with a corrected PATH); cleanupClosedIssueTransientLabels (defensive sweep, via transientLifecycleLabels) On the next invocation (stage dispatch or comment review) that observes no comparable mismatch; or when issue is closed (defensive sweep) Warn-and-continue, not a gate: StageAttempted/StageRetryIncremented/max_retries bookkeeping is entirely untouched, and neither stage:<name>:failed nor fabrik:paused is applied. Comparison is deliberately narrow and fails closed on ambiguity (an .nvmrc alias, a complex engines range, an unparseable .tool-versions line, or a binary missing from PATH are all “not comparable” — silence, never a false-positive warning); .nvmrc/.tool-versions/engines.node compare at major-version granularity, go.mod’s toolchain/go directive at major.minor using minimum-version semantics. A successful resolution is memoized process-wide per tool for the daemon’s own lifetime (resolveToolVersion) — sound specifically because the bug this label exists to catch is that this PATH cannot change without a restart; a resolution failure (timeout, transient exec error) is deliberately never cached, since a failure carries no such invariance guarantee — caching it would permanently and silently disable detection for that tool after a single bad moment. No dedicated settle scan: this is a fire-once informational label, structurally identical to fabrik:api-key-helper-detected. See ADR-1786.
fabrik:awaiting-done handleNoWorkNeeded As the very first mutation, the instant processItem decides completed && noWorkNeeded — before the fabrik:awaiting-input clear, before the emitting stage’s completion label, before anything else (idempotent: a no-op if already present) clearNoWorkNeededMarker (after a fully successful settleNoWorkNeeded pass — status moved to Done and issue closed); escalateNoWorkNeededFailure (after MaxRetries failed settle passes) When the Done move and issue close have both succeeded (or were already true), or when escalated (fabrik:paused takes over dispatch suppression instead) Suppresses dispatch of every non-cleanup stage in itemMayNeedWork/itemNeedsWork, independent of item.Status — the outstanding board move is exactly what may be failing, so the item can be observed sitting at any column while this label is present. Retried every poll by the no-work-needed settle scan, settleNoWorkNeededScan (engine/poll_settle.go; called from poll() in engine/poll.go, immediately after runValidatePRTerminalAdvance), which resolves the current stage via stages.FindStage(e.cfg.Stages, item.Status) and calls settleNoWorkNeeded. Deliberately excluded from cleanupClosedIssueTransientLabels’s closed-issue defensive sweep — unlike other gate labels, stripping it before the settle scan has finished would silently resurrect the bug this label exists to prevent (§6.8, ADR-060)
fabrik:awaiting-member-close markMergeTrainMemberCloseOutstanding (landSingleton) Only in the failure branch of landSingleton’s member-issue CloseIssue call — after the PR merge, Done-move, and member-PR close have already run (idempotent: a no-op if already present) clearMergeTrainMemberCloseMarker (after a fully successful settleMergeTrainMemberClose pass — issue confirmed closed); escalateMergeTrainMemberCloseFailure (after MaxRetries failed settle passes) When the member issue is confirmed closed (by us or by GitHub’s own Closes #N auto-close), or when escalated (fabrik:paused takes over instead) Not wired into itemMayNeedWork/itemNeedsWork or transientLifecycleLabels — by the time this label can be written, the item has already reached its terminal singleton-landing outcome, so there is no per-stage redispatch risk to guard against (§6.10, ADR-061). Retried every poll by settleMergeTrainMemberCloses (engine/merge_train_member_close_settle.go; called from poll() in engine/poll.go, immediately after handleMergeTrainBatch), which scans raw board.Items directly — independent of merge_train: on/off, deepFetchCandidates, and the terminal-skip optimization (#689)
fabrik:awaiting-close markNonDefaultBaseCloseOutstanding (closeIssueIfNonDefaultBase) In the failure branch of closeIssueIfNonDefaultBase’s explicit CloseIssue call, or by guardDefaultBaseAutoClose when a default-base issue is still open (or unreadable) at the landing (#1962) — after the caller’s Done-advance has already run (idempotent: a no-op if already present) clearNonDefaultBaseCloseMarker (after a fully successful settleNonDefaultBaseClose pass — issue confirmed closed); escalateNonDefaultBaseCloseFailure (after MaxRetries failed settle passes) When the issue is confirmed closed (by us or by any other actor), or when escalated (fabrik:paused takes over instead) Not wired into itemMayNeedWork/itemNeedsWork or transientLifecycleLabels — structurally identical to fabrik:awaiting-member-close: by the time this label can be written, the item has already reached Done, so there is no per-stage redispatch risk to guard against (§6.13, ADR-1097). Retried every poll by settleNonDefaultBaseCloses (engine/close_nondefault_base_settle.go; called from poll() in engine/poll.go, immediately after settleMergeTrainMemberCloses), which scans raw board.Items directly
fabrik:awaiting-advance markAdvanceFailureOutstanding (recordAdvanceOutcome) Only in the failure branch of recordAdvanceOutcome’s advanceToNextStage call — advanceToNextStage is the last board mutation both call sites (advanceValidateTerminalItem’s merged-PR path, advanceConvergedPRToDone) make before it, so every other side effect that precedes it (gate-label clearing, completion-label filling, fabrik:auto-merge-enabled removal) has already run (idempotent: a no-op if already present — gated so a repeat failure in the same episode never posts a second comment, R5). A one-time explanatory comment naming the failing stage and the underlying error (react=false) is posted in the same branch. Note: advanceToNextStage is not the final mutation either call site makes — both unconditionally call closeIssueIfNonDefaultBase immediately afterward regardless of the advance’s outcome (Pruefer, PR #1469 review, third round); see §6.17’s “Scope” paragraph. clearAwaitingAdvanceMarker only — after a fully successful recordAdvanceOutcome retry pass. Unlike every other label in this table, escalateAwaitingAdvanceFailure does not remove this marker (adds fabrik:paused alongside it instead) — see §6.17 for why. When the terminal advance finally succeeds (board Status option now exists) Not wired into itemMayNeedWork/itemNeedsWork or transientLifecycleLabels — structurally identical to fabrik:awaiting-close/fabrik:awaiting-member-close: by the time this label can be written, the item’s own stage is already gate-complete, so there is no per-stage redispatch risk to guard against (§6.17, ADR-1422). Retried every poll by settleAwaitingAdvanceScan (engine/advance_settle.go; called from poll() in engine/poll.go, immediately after settleClosedValidateAdvance), which scans raw board.Items directly and shares the poll’s advancedItems dedup map with runValidatePRTerminalAdvance/settleClosedValidateAdvance. Can coexist with fabrik:paused after an escalation — the settle scan’s own guard (hasLabel(fabrik:paused)) is what suppresses further retries while paused, not the marker’s absence
fabrik:awaiting-placement spawnChildren (via recordChildPlacementFailure) On any of the three UpdateProjectItemStatus failure branches at spawn time (call error, nil statusField, or no suitable status option) — applied to the child issue, not the parent clearChildPlacementMarker (after a successful settleChildPlacement pass, or the closed-child short-circuit in the child board-placement settle scan); escalateChildPlacementFailure (after MaxRetries failed settle passes) When placement succeeds, the child is observed closed, or escalation fires (fabrik:paused takes over) Unlike fabrik:awaiting-done, this label does not suppress dispatch via itemMayNeedWork/itemNeedsWork — a child stranded in an unmatched column (typically Backlog) never resolves to a stage there in the first place, so there is no dispatch to suppress. Retried every poll by the child board-placement settle scan, settleChildPlacements (engine/poll_settle.go; called from poll() in engine/poll.go, immediately after cleanupClosedIssueTransientLabels), which is sourced from board.Items directly (not deepFetchCandidates — the child never passes itemMayNeedWork’s stage == nil guard to get there) and calls settleChildPlacement. Deliberately excluded from cleanupClosedIssueTransientLabels’s closed-issue defensive sweep, for the same reason as fabrik:awaiting-done — see §6.9, ADR-062
fabrik:awaiting-runaway-alert markRunawayAlertOutstanding (fireRunawayGuard) Only in the failure branch of fireRunawayGuard’s per-member AddComment call — fabrik:paused/fabrik:awaiting-input are applied unconditionally regardless of this branch (idempotent: a no-op if already present) clearRunawayAlertMarker (after a fully successful settleRunawayGuardAlert pass, or a racing fireRunawayGuard call for the same member succeeding first); escalateRunawayAlertFailure, but only if its own fallback comment succeeds (after MaxRetries failed settle passes) When the alert comment is confirmed posted (by us or by a racing call), or when escalated and the fallback comment itself is confirmed posted — unlike every other marker in this family, escalateRunawayAlertFailure does not delegate to the shared escalateSettle helper, precisely so a persistently-failing AddComment (not just a transient one) cannot silently erase the marker while leaving the member with zero delivered explanation (#1533 review, finding 1); if the fallback comment also fails, the marker stays and settleRunawayGuardAlertScan keeps retrying indefinitely Deliberately not gated on fabrik:paused’s absence in its own settle scan — unlike every sibling in this family (fabrik:awaiting-member-close, fabrik:awaiting-close, fabrik:awaiting-advance), fabrik:paused is present from the very first application of this marker (the pause never depends on the comment succeeding), so a paused-item guard here would make the scan a permanent no-op. Retried every poll by settleRunawayGuardAlertScan (engine/runaway_alert_settle.go; called from poll() in engine/poll.go, immediately after settleMergeTrainMemberCloses), which scans raw board.Items directly. See §6.18, ADR-1533
fabrik:awaiting-landing-verification landMergeTrainBatch/landSingleton (engine/merge_train.go), advanceConvergedPRToDone (engine/merge_gate.go), advanceValidateTerminalItem’s merged-PR branch (engine/pr_terminal_advance.go) Immediately after each call site’s own advanceToNextStage/recordAdvanceOutcome call succeeds — the item has actually reached Done, not merely attempted the advance (idempotent: a no-op if already present) clearLandingVerificationMarkers (after a fully successful settleLandingVerification pass — credited PR confirmed merged); failLandingVerification (on a confirmed non-merge — R2’s immediate reopen path, not gated by the retry counter); escalateLandingVerificationFailure (after MaxRetries failed inconclusive passes) When the credited PR is confirmed merged, when a confirmed non-merge is found (fires on the same pass, R2), or when escalated (fabrik:paused takes over instead) Not wired into itemMayNeedWork/itemNeedsWork or transientLifecycleLabels — by the time this label can be written, the item has already reached Done, so there is no per-stage redispatch risk to guard against (§6.19). Retried every poll by settleLandingVerification (engine/landing_verification_settle.go; called from poll() in engine/poll.go, immediately after settleNonDefaultBaseCloses), which scans raw board.Items directly. Two regimes never conflated: a confirmed failure fires immediately (AC1, “within one poll cycle”); an inconclusive result (API error, or no crediting reference discoverable) goes through the bounded retry-then-escalate regime and never reopens the issue on ambiguity (AC4). See §6.19, ADR-1616
fabrik:landing-verification-failed failLandingVerification On a confirmed non-merge of the credited PR — alongside reopening the issue, moving it back to Validate, and removing fabrik:awaiting-landing-verification/fabrik:credited-pr:<N> User (manual, after investigating and re-landing) Manual only — this is a human-gated distinguishing label, not self-clearing Lets an operator tell a landing-verification failure apart from an ordinary stage failure at a glance. See §6.19, ADR-1616
fabrik:auto-merge-enabled attemptMergeOnValidate (yolo, non-cruise, Validate completion; held while unprocessed feedback is pending — §6.6.6 “Feedback gate”) After successful enablePullRequestAutoMerge GraphQL call — signals GitHub is handling the merge atomically. Also serves as the idempotency guard (prevents re-calling the mutation) and the budget-start anchor (timestamp read by FetchLabelAppliedAt to compute convergence budget elapsed time). checkAutoMergeConvergence (when PR merges, PR closes, user disables auto-merge, or budget exhausts); cleanupClosedIssueTransientLabels (defensive sweep) When GitHub merges the PR; when PR is closed without merging; when user disables auto-merge in GitHub UI; or when convergence budget exhausts (→ pauseForConvergenceFailed); or when issue is closed (defensive sweep) Engine-internal label. Bypasses legacy merge/CI gate dispatch — checkAutoMergeConvergence is the sole Phase 1 handler while present. Triggers itemMayNeedWork updatedAt cache bypass (convergence state changes independently of issue updatedAt). stage:Validate:complete remains in place while this label is present; checkAutoMergeConvergence removes it when the PR merges and the item advances to Done. NOT applied when fabrik:cruise is also present (cruise wins).
fabrik:blocked checkDependencies When open blocking issues exist (first transition only — idempotent) PushUnblockObserver (primary — fires immediately on blocker close, any column); checkDependencies (defense-in-depth — fires via dep-blocked cooldown-retry for items in stage columns) When all blocking issues close Pre-dispatch gate in itemNeedsWork prevents re-dispatch after the label is set (parallel to fabrik:editing post-#550 and fabrik:locked:<other> always); exception: when the dep-blocked cooldown has expired (or no store entry exists), itemNeedsWork admits the item for one dependency re-check per 10 × PollSeconds — checkDependencies either re-stamps the cooldown (still blocked) or removes the label (resolved). Initial detection (first dispatch, label not yet set) still reaches checkDependencies normally. Blocks stage start. Push path removes the label even for items in non-stage columns (Backlog, Done, custom columns) that would otherwise never be re-evaluated by processItem.
stage:<X>:in_progress processItem After lock acquired and verified releaseLock Same as fabrik:locked:<user> Informational — shows which stage is active on GitHub
stage:<X>:complete handleStageComplete (for non-wait_for_ci stages), checkCIGate (for wait_for_ci: true stages — added only after CI passes), handleNoWorkNeeded (emitting stage + all skipped stages), cleanup stage handler, endStageRework (restore on a non-completing comment-re-entry exit, #1802) handleStageComplete: after Claude signals FABRIK_STAGE_COMPLETE on stages without wait_for_ci: true. checkCIGate: when all CI checks pass (R5) — this is the conjunctive gate (ADR 032): stage:X:complete is deferred until the CI gate actually clears, not applied on FABRIK_STAGE_COMPLETE. After FABRIK_NO_WORK_NEEDED (emitting stage + all subsequent non-cleanup stages) or worktree cleanup. For stage:Validate:complete specifically, handleStageComplete and checkCIGate also apply ValidateCompletedAtSHA carrying the current HEAD SHA so the SHA-invalidation scan (§2.16) can detect future SHA changes. endStageRework: immediately, on a comment re-entry’s non-completing exit that had cleared it via beginStageRework — a direct re-add, not routed through handleStageComplete. Never removed in the ordinary case, with two exceptions: stage:Validate:complete is removed by the SHA-invalidation scan (§2.16) when the linked PR’s HEAD SHA changes after completion; and any stage’s :complete is cleared by beginStageRework for the duration of a comment re-entry that reworks it (#1802) — restored directly by endStageRework on a non-completing exit, or left to handleStageComplete’s own re-derivation (including its wait_for_ci deferral) when the re-entry itself re-signals FABRIK_STAGE_COMPLETE Permanent, with the same two transient exceptions: stage:Validate:complete when the linked PR SHA changes after completion (§2.16); and any stage during a comment-re-entry rework window, bracketed by fabrik:reworking:<Stage> (#1802, ADR-1802) — never observable by any dispatch-admission gate, since the window is strictly nested inside fabrik:editing’s own gate Prevents re-invocation of the stage; triggers catch-up advancement
stage:<X>:failed escalateFailedStage After MaxRetries exhausted (includes retries exhausted due to repeated degenerate-output detections, §2.6/§7.2) clearFailedStage When user removes fabrik:paused (manual unpause) Indicates permanent failure; paired with fabrik:paused
fabrik:yolo User (manual) Any time User (manual) Any time Forces auto-advance; triggers auto-merge at Validate; overrides auto_advance: false per stage
fabrik:cruise User (manual) Any time User (manual) Any time Forces auto-advance without merge; stops at Validate; cruise wins over yolo for both end-of-Validate decisions — auto-merge suppressed and the issue held at Validate — even when yolo is also present; both guards test the raw cruise label, not cruiseActive
fabrik:unrestricted User (manual) Any time User (manual) Any time Passes --dangerously-skip-permissions instead of --permission-mode dontAsk
fabrik:extend-turns User (manual) Any time processItem cleanup branch or User (manual) At Done cleanup stage completion; or manual removal Pre-grants 2× stage.MaxTurns budget for the first stage invocation while present, with max_wall_time scaled 2× to match for that invocation; no-op for stage path when max_turns == 0 (unlimited); also pre-grants 2× commentMaxTurns(stage) budget (with matching 2× max_wall_time scaling) for the first comment processing invocation (comment budget is never 0); subsequent extensions beyond 2× reset to the base budget and unscaled deadline, require progress detection, for both paths; persists across all intermediate stages
model:<name> User (manual) Any time User (manual) Any time Selects Claude model; first label wins if multiple present
effort:<level> User (manual) Any time User (manual) Any time Overrides stage effort level; highest-ranked wins if multiple present
base:<branch> User (manual) Before Research (recommended) User (manual) Any time Overrides worktree base branch; falls back to default if branch not found on remote; if a PR exists, its base branch is updated to match on each stage invocation. Resolution (baseBranchForItem/resolveBaseLabelBranch) first checks the local bare clone (branchExists, fast path); on a local miss it probes origin directly via git ls-remote --heads and, if confirmed there, fetches the branch into the local clone before honoring it — so a branch that exists on the remote but hasn’t yet been fetched locally is not mistaken for genuinely absent. Only falls back to default when the ls-remote probe itself confirms absence (or errors, fail-safe)
review-authority:<mode> User (manual) Any time User (manual) Any time Overrides the wait_for_reviews gate’s review_authority for this issue only (advisory/authoritative); no label → stage config governs; one recognized label → it wins; both present → resolves to authoritative (more restrictive), warning logged; malformed/unknown suffix → ignored, warning logged, falls back to stage config. Resolved by effectiveReviewAuthority, consulted identically by checkReviewGate, reviewGateBlocksLanding, and pauseForReviewTimeout’s message. Only meaningful alongside wait_for_reviews: true — see §6.1.1, ADR-1261
expected-reviewers:<mode> User (manual) Any time User (manual) Any time Overrides the wait_for_reviews gate’s expected_reviewers for this issue only (none/declared); no label → stage config governs (nil stays nil); one recognized label → it wins; both present → resolves to declared (more restrictive — imposes waiting), warning logged; malformed/unknown suffix → ignored, warning logged, falls back to stage config. Resolved by effectiveExpectedReviewers, consulted identically by checkReviewGate, reviewGateBlocksLanding, and pauseForReviewTimeout’s message. Only meaningful alongside wait_for_reviews: true — see §6.1, #1304
fabrik:sub-issue spawnChildren (engine-side, shared by preImplement and the Review/Validate mid-flight hook, §6.7.2 — ADR-1419) Applied to each spawned child issue, from Plan (via pre-Implement) or a mid-flight Review/Validate declaration alike N/A N/A Informational — marks issues created by Fabrik’s sub-issue spawn mechanism; no engine-side gate semantics
fabrik:children-spawned spawnChildren (engine-side, shared by preImplement and the Review/Validate mid-flight hook, §6.7.2 — ADR-1419) After all FABRIK_SPAWN_CHILD_* children in one batch are successfully created and linked as blockedBy of parent, whichever stage originated the spawn User (manual, to re-trigger spawn) If user wants a fresh spawn (children created under the prior spawn must be closed manually first — fabrik:spawned-child:<i>:<n> markers, ADR-1583, are only consulted while this label is absent) Idempotency guard against preImplement re-spawning the same Plan-declared batch on retry — while present, preImplement is a no-op. The Review/Validate mid-flight hook needs no equivalent guard (its own idempotency comes from parsing each dispatch’s fresh output exactly once, §6.7.2), but still sets this label on a successful spawn — it remains true that this parent has spawned children at least once.
fabrik:spawned-child:<blockIndex>:<childNumber> spawnChildren (ADR-1583) Immediately after each spawn block’s child issue is successfully created, on the parent spawnChildren, once the full batch (including sibling DEPENDS_ON wiring) succeeds N/A (best-effort cleanup on success) Durable, restart-surviving marker recording that this Plan-declared block’s child already exists — see §6.7’s “Resume-safe retry” note. Transient only: present during an interrupted/in-progress spawn, absent once fabrik:children-spawned is set. Removing one manually forces re-creation of that specific block on the next retry (e.g. if the recorded child was genuinely deleted).
fabrik:revalidate User (manual) Applied to any issue to request Validate re-entry Revalidate-scan loop (handleRevalidateLabel), plus cleanupClosedIssueTransientLabels (defensive sweep) On Validate-stage items: immediately after removing all gate/completion labels; on non-Validate items: after logging a warning; on closed issues: defensive sweep Triggers removal of stage:Validate:complete, stage:Validate:failed, fabrik:paused, fabrik:awaiting-input, fabrik:awaiting-ci, fabrik:auto-merge-enabled, then itself; resets PausedByEngine, StageRetryCount, LastAttemptAt, EngineCycles for Validate; Validate dispatches on the next poll cycle. Applied to non-Validate issues: only the trigger label is removed, no other action. Applied while a Validate worker is in-flight: held in place until the worker exits (FR-4). Bypasses the updatedAt cooldown cache via hasAwaitingLabel so stuck-Validate items with settled updatedAt are still deep-fetched.

2. Event Enumeration

Thirteen distinct event types drive state transitions (§2.1–2.11, §2.13, §2.14), plus one TUI display event (§2.12) that does not drive transitions:

2.1 Poll Tick

Trigger: The engine’s poll loop fires on a configurable interval (PollSeconds).

Code path: poll() → itemMayNeedWork() (shallow filter) → FetchItemDetails() (deep fetch) → itemNeedsWork() (full filter) → catch-up loop → dispatch loop → processItem()

Effect: The primary driver of all state transitions. Each poll cycle evaluates every item on the board through the filter chain and dispatches work for qualifying items.

2.2 New User Comment

Trigger: A user posts a comment on an issue or its linked PR. Detected by findNewComments() — filters out Fabrik-generated comments (prefix 🏭 **Fabrik) and already-processed comments (ROCKET reaction or CommentProcessed entry in itemstate.Store).

Code path: itemNeedsWork() detects new comments → processItem() routes to processComments() or triggers unpause/unblock

Landing hold (#1862, #1953): unprocessed feedback — a comment (exactly findNewComments), an unresolved current-head review thread, or an unaddressed review body — on a Validate-complete item also blocks the landing decision itself — see §6.6.6 “Feedback gate”. The feedback is processed first; landing is re-evaluated afterwards. A bot review of a pure merge-train catch-up push (§6.30) is the one non-actionable exception.

Effect: Can trigger three distinct behaviors:

  1. Unpause: On a paused issue, a human comment (resumeAuthorised() — excludes bot logins via github.IsBotLogin; Fabrik’s own output is separately excluded upstream by findNewComments’ 🏭 **Fabrik body-prefix check) created at or after the latest fabrik:paused labeled event removes fabrik:paused (and clears failed state if present) and falls through to comment processing. A bot-authored comment does not unpause (#1083), and neither does a human comment that strictly predates the pause (#1813, §4.4).
  2. Unblock awaiting-input: On an awaiting-input issue, a human comment removes both fabrik:paused and fabrik:awaiting-input, then routes to processComments(). A bot-authored comment, or a human comment that strictly predates the pause, does not unblock.
  3. Comment processing: On an active (non-paused) issue, any new comment (human or bot) routes directly to processComments() — unaffected by the human-only resume restriction above
  4. Post-merge guard (#1862, ADR-1862): Before any of the above reaches a worker, processCommentsClassified checks whether the item’s work has already landed (itemPRAlreadyLanded: a merged linked PR — confirmed with FetchPRMerged when the list endpoint shows closed-not-merged — or a merge-train member’s fabrik:credited-pr:<N> / fabrik:awaiting-landing-verification label). If so it invokes no worker, makes no commit or push, and does not add 👀 or fabrik:editing. For each human-authored comment (with a DatabaseID) not already answered, it posts one reply on the issue and the linked PR saying the work already landed, the change was not applied, and it needs a new issue. The triggering comment gets no 🚀 — it must not look processed — so it stays “new”; the reply carries an HTML marker <!-- fabrik:post-merge-comment:<commentID> --> and the guard skips comment IDs already marked in item.Comments, so it answers each comment once, durably across restarts. The guard runs before recordCommentBreakerInvocation, so it never feeds the circuit breakers, and returns the didNotRunPostMerge kind so reinvoke cycle counters are refunded. This deliberately breaks “processed ⇒ 🚀” for this one class of comment. It acts only on positive evidence: a read error, no PR, an open PR, or a human-closed (closed-not-merged, no landing label) PR falls through to normal processing.
  5. Circuit breaker trip: Every processComments() invocation (from any of the three paths above, or from a review/CI-fix/rebase reinvoke, §6.2/§6.5/§6.6) records a comment-processing circuit-breaker invocation; if the issue accumulates N invocations within a rolling window with no forward progress, the engine applies fabrik:paused + fabrik:awaiting-input itself and suppresses further dispatch — see §4.6.
  6. Overseer audit comments (#1969, ADR-1966-c): The 🏭 **Fabrik — overseer action: <action>** comment an fabrik mcp action tool posts is Fabrik’s own output: the same 🏭 **Fabrik prefix check drops it from findNewComments, so it can neither resume a paused item nor start a comment-review cycle even when, under PAT auth, its author is the operator’s own non-bot login; see §7.15.

2.3 PR Review State Change

Trigger: A pull_request_review webhook event with action ∈ {submitted, edited, dismissed} arrives. All three actions carry the full review object (author, state, body, DatabaseID) and are routed through the same review-upsert path in applyPullRequestReviewDelta. The webhook action itself is not stored — only the review state (from the payload’s review.state field, normalised to uppercase) is recorded in item.LinkedPRReviews (upserted by DatabaseID).

Code path: Delta applied by applyPullRequestReviewDelta → itemstate.PRReviewSubmitted upsert → catch-up loop in poll() re-evaluates checkReviewGate().

Effect: Can clear the review gate when len(outstanding) == 0 and at least one non-DISMISSED review exists. A DISMISSED review does not satisfy the hasReviews condition. Does not directly trigger a stage invocation.

The self-review escape hatch. hasReviews does not check the review’s author or state beyond “not DISMISSED” — a COMMENTED review submitted by the PR author itself satisfies it, even though GitHub forbids a PR author from approving their own PR. This is intentional, not an oversight: the advisory gate’s real question is “has a human looked at this?”, not “has a reviewer approved this?” (see §6.1.1 for the authoritative-mode distinction, which layers an approval requirement on top). On a repo with no reviewer ever requested — no CODEOWNERS, no review bot installed — a self-authored COMMENTED review is the supported manual way to clear an otherwise-unsatisfiable gate; wait_for_reviews: false is the supported stage-config setting for such a repo going forward. See docs/USER_GUIDE.md’s wait_for_reviews section for the operator-facing version of this note.

2.4 PR Review Threads with Feedback

Trigger: Reviewers leave inline code comments on the linked PR in unresolved review threads. These are real GitHub comments with DatabaseIDs.

Code path: Detected by buildReviewThreadComments() in the catch-up loop → dispatchReviewReinvoke() → async processComments() with synthetic comments

Effect: Triggers a review reinvocation cycle — the stage agent is re-invoked via processComments() with the review thread comments as input, allowing it to address reviewer feedback. This is a distinct event type from regular comment processing (see §6.2).

2.5 Blocking Issue Closed

Trigger: An issue listed in item.BlockedBy transitions to the CLOSED state, OR a dependent item’s BlockedBy slice is populated for the first time via deep-fetch while all listed blockers are already closed.

Primary code path (push-based): PushUnblockObserver

PushUnblockObserver is registered on engine.store and fires on two distinct events. The dependent unblocks within seconds regardless of which event arrives first.

Trigger 1 — StateChanged (blocker closes): When a blocker Y closes:

  1. The observer calls store.All() to scan all known items.
  2. For each item X that carries fabrik:blocked and has Y in its BlockedBy list, it checks every remaining blocker via store.Get() (fresher than dep.State from the last board fetch; falls back to dep.State == "CLOSED" if the blocker is not in the store).
  3. If all of X’s blockers are now closed, the observer dispatches removeBlockedIfResolved on a goroutine, which removes fabrik:blocked via RemoveLabelFromIssue (3 attempts, exponential backoff, idempotent) and applies the cache write-through.

Trigger 2 — BlockedByChanged (dependent’s BlockedBy first populated via deep-fetch): BlockedBy is a deep-fetch-only field — after bootstrap every item has BlockedBy = nil until its first ItemDeepFetched mutation. If Trigger 1 fires before the dependent’s first deep-fetch, the dependent’s BlockedBy is empty and the scan silently skips it. When the dependent IS later deep-fetched, the Store emits BlockedByChanged, and the observer reacts:

  1. It reads the post-mutation snapshot of the dependent item X directly (no store.All() scan).
  2. If BlockedBy is empty after the mutation, it returns immediately (no-op — BlockedByChanged only fires when the slice actually changes, so this guards against a deep-fetch that clears or results in an empty dependency list).
  3. If X carries fabrik:blocked and all listed blockers are closed in the store, it dispatches removeBlockedIfResolved on a goroutine.

Both triggers are idempotent — double-removal races are handled correctly (ErrNotFound is treated as success by removeBlockedIfResolved).

This path bypasses processItem and itemMayNeedWork entirely, so it works for items in any column — including Backlog, Done, or custom non-stage columns where itemMayNeedWork would return false. No comment is posted; the push path is a label-removal-only operation. Observer decisions (skipped, unblocked, or still-blocked reasons) are emitted under the [push-unblock] log tag in fabrik.log.

Defense-in-depth path (pull-based): itemNeedsWork exception + processItem() → checkDependencies()

processItem() applies CooldownRecorded{Reason: "dep-blocked"} each time checkDependencies() returns true (blocked). While the cooldown is active the pre-filter skips deep-fetch entirely (no goroutine, no GraphQL burn — #576 fix preserved). When the cooldown expires, itemNeedsWork detects the expiry via snap.CooldownAt("dep-blocked") and admits the item — bypassing the #576 pre-dispatch gate for this one re-check — so processItem → checkDependencies can re-evaluate: if still blocked it re-stamps the cooldown (resetting the 150s window); if resolved it removes fabrik:blocked. A missing store entry (cold-start or restart) also admits the item, since no active cooldown exists. This path fires once per 10 × PollSeconds for items in configured stage columns and serves as defense-in-depth for missed webhook events.

Note: within checkDependencies(), for each blocker the engine first consults store.Get(blocker).IsClosed() (the cache’s view) and only falls back to dep.State != "CLOSED" from the GraphQL deep-fetch if the blocker is not present in the store. This preference avoids false “still blocked” conclusions caused by GitHub indexer lag, which can delay dep.State updates by several seconds after a blocker actually closes.

Resume-from-blocked live re-read (ADR-1419, requirement 6). On the recheck path (alreadyBlocked — the item already carries fabrik:blocked), checkDependencies bypasses the cache entirely with a raw, uncached FetchItemDetails call before computing openDeps, rather than trusting the item snapshot’s own (possibly stale) BlockedBy list — mirroring recoverMissingPlanComment (engine/spawn.go) and verifyAndHealLinkage (engine/prcreate.go). This fires unconditionally on alreadyBlocked alone — not additionally gated on the cached item.BlockedBy being non-empty. A stale-empty cache is exactly the shape a bypassed engine spawn produces: a stage agent calling gh issue create directly instead of emitting FABRIK_SPAWN_CHILD never creates a blockedBy edge, so the cached snapshot has always shown zero blockers even while fabrik:blocked is present from some other source (or becomes stale after a genuine edge is added out-of-band). Gating the live read on a non-empty cached list — the pre-#1419 behavior — would skip it entirely in that shape and trust the wrong empty list, silently clearing the label on nothing. This is the check that closes the observed regression (repo-b#102): a parent resuming from fabrik:blocked must re-verify its dependency set is actually satisfied, not merely that the cached count says so.

Concurrency note: Neither StateChanged nor BlockedByChanged is in wakeChFlags or cycleSetFlags, so PushUnblockObserver fires do not wake the poll loop or populate mayNeedWork. Label removal is a direct side effect dispatched on a goroutine to avoid blocking the store notification call path. Double-removal races (two blockers closing within milliseconds, or both triggers firing for the same item) are handled correctly — ErrNotFound is treated as success by removeBlockedIfResolved.

2.6 Claude Output Markers

Trigger: Claude’s output contains one of the Fabrik markers. Checked after each stage invocation.

Markers and priority order (enforced by the if/else if dispatch chain in processItem()):

  1. FABRIK_STAGE_COMPLETE + FABRIK_NO_WORK_NEEDED (both present) — highest priority among completed == true paths; completed && noWorkNeeded branch fires before the plain completed branch
  2. FABRIK_STAGE_COMPLETE (alone) — next; handled by the plain completed branch
  3. FABRIK_BLOCKED_ON_INPUT — checked last; only honored if completed is false and err == nil

FABRIK_NO_WORK_NEEDED is ignored unless FABRIK_STAGE_COMPLETE also appears in the same output — the no-work path requires the emitting stage to explicitly declare itself complete. The timeout/kill recovery path that scans buffered output for FABRIK_STAGE_COMPLETE does not trigger the no-work path (only FABRIK_STAGE_COMPLETE is scanned in that recovery path, not FABRIK_NO_WORK_NEEDED).

Code path: processItem() → outcome dispatch based on which marker is present

Effect:

FABRIK_SPAWN_CHILD_BEGIN/END blocks are not processed inline — they are structured data emitted by the Plan stage and preserved in the Plan stage comment. The engine’s preImplement step reads them at Implement dispatch time to create child issues. See §6.7.

FABRIK_PR_CREATE_BEGIN/END blocks are processed inline — the engine parses the block during processItem(), creates the draft PR with Closes #N prepended as the first line, and records the PR number before posting output. See §5.6.

Invocation-level kill paths: The max_wall_time and inactivity timeout mechanisms (see §7.7) can terminate the Claude process before it writes a clean {"type":"result"} line. After such a kill, runClaude() retroactively scans the already-buffered output for FABRIK_STAGE_COMPLETE in intermediate {"type":"assistant"} NDJSON lines via extractTextFromAssistantTurns(). If found, completed=true is returned and the invocation is treated identically to a live FABRIK_STAGE_COMPLETE. If not found, completed=false is returned and the invocation routes to the cooldown/retry path. These kills are distinguished from engine-shutdown cancellation by the wasTimedOut flag, so they do not trigger the hard-error path.

Assistant-turn artifact harvest and no-artifact completion guard (#1782): Earlier still — inside interpretClaudeResult(), before finalizeStageOutcome() or processComments() ever see the output — the engine addresses a related but distinct defect: the CLI’s terminal result field reflects only the agent’s very last turn, which can be a near-empty wrap-up if one more tool call followed the turn that produced the real artifact. artifactMissingOnComplete() detects this (FABRIK_STAGE_COMPLETE present, FABRIK_NO_WORK_NEEDED absent, and the text carries nothing beyond Fabrik’s own bare control-marker lines — hasArtifactContent()). When it fires, extractLastSubstantialAssistantTurn() re-scans the raw NDJSON for the last assistant turn with real content (not necessarily the marker-bearing turn — the reported case’s marker turn contained only the marker) and prepends it to the result text. If nothing is found, or the harvest recovers only more control-marker-only text, artifactMissingOnComplete() still holds against the (now harvested) text, and both places interpretClaudeResult() would otherwise return completed=true are gated on its negation — the invocation returns completed=false instead, indistinguishable at this level from an ordinary incomplete run. FABRIK_NO_WORK_NEEDED co-occurring with FABRIK_STAGE_COMPLETE is excluded from both the scan and the guard (a legitimate artifact-free completion, not this defect).

Because this runs upstream of both invocation paths, completed=false arrives at finalizeStageOutcome()/processComments() exactly as it would for any other incomplete run — the comment-review path needs no dedicated handling beyond its existing no-progress machinery. The stage-dispatch path additionally recomputes artifactMissingOnComplete() against the pre-strip output (engine/item.go) purely to drive distinct messaging: a one-time first-detection comment (“no artifact harvested”) on first occurrence, and — like the degenerate-output guard below — counts against MaxRetries (the invocation did real work) rather than being exempted the way a usage-limit or tools-denied exit is, escalating via escalateFailedStage() at the limit with a cause note naming this defect distinctly from the unrelated bare-file-reference one. See ADR-1782.

Degenerate-output guard: Before any of the marker-dispatch branches above run, finalizeStageOutcome() checks the stripped/trimmed output (postOutput) against isDegenerateOutput() — a conservative check for a single-line body that is nothing but a bare @file reference or an absolute filesystem path (e.g. @/tmp/plan.md, /tmp/plan.md). This catches a model writing its real stage output to a file and returning a dangling reference instead of emitting it inline (issue #1065). If it trips, regardless of which marker was present: the output is not posted as a comment, and completed is forced to false before the branches above evaluate — so FABRIK_STAGE_COMPLETE (with or without FABRIK_NO_WORK_NEEDED) cannot advance the stage on a degenerate body. The invocation instead falls through to the “None of the above” cooldown-retry path and, on the first detection, an explanatory comment is posted immediately (rather than staying silent until MaxRetries); at the retry limit it escalates via the normal escalateFailedStage() path (see §7.2), whose pause comment names the offending reference. The detector requires an unambiguous signal (@-prefix or a leading absolute path with ≥2 segments) and only ever matches single-line output — ordinary short prose (N/A, TBD, a one-sentence completion note, or a sentence that merely mentions a path) is not flagged.

2.7 Manual Label Change

Trigger: A human adds or removes a label on the issue via the GitHub UI.

Code path: Detected on the next poll cycle when labels are fetched

Effect varies by label:

2.8 Issue Closed

Trigger: The issue is closed on GitHub (e.g., by PR merge with Closes #N).

Code path: itemMayNeedWork() and itemNeedsWork() check item.IsClosed

Effect (ADR-1387): A closed item is never dispatched to a Claude stage invocation. The closed-issue guard admits an item only if one of the following holds:

  1. The current stage is a cleanup stage (CleanupWorktree: true) — cleanup can remove the worktree
  2. The current stage has a stage:<X>:complete label — the catch-up loop can advance to the next stage (e.g., a PR merge closes an issue sitting in Validate with stage:Validate:complete; it needs to move to Done)

Everything else a closed item needs — advancing to Done, clearing stale labels, reaping a worktree, retrying a failed close — is board/label reconciliation performed by a board.Items-sourced settle scan, never by widening this admission gate. In particular, a closed item at a gate-checked stage (Validate) lacking stage:Validate:complete — carrying any gate label (fabrik:awaiting-review, fabrik:paused, fabrik:awaiting-ci) or none at all — is healed exclusively by settleClosedValidateAdvance (§6.15), independent of this admission gate. Before ADR-1387, this guard additionally admitted such items directly (a !stageIsGateChecked(stage) disjunct, plus explicit fabrik:awaiting-ci/fabrik:auto-merge-enabled admissions) on the theory that admission was the only way for the settle-owner to observe them — see §6.15 for why that coupling produced an unbounded post-close dispatch loop, and why removing it here is safe now that the settle-owner has its own independent feed.

Comment-triggered dispatch is closed on the same terms (R1, ADR-1387). Clause 2 above admits a closed, stage-complete item, so itemNeedsWork must also refuse every path that could turn a new comment on such an item into a Claude invocation. There are three: the fabrik:awaiting-input resume, the fabrik:paused unpause resume, and the plain new-comment fast path. A single item.IsClosed && !stage.CleanupWorktree guard sits ahead of all three, so a closed item carrying stage:<X>:complete together with fabrik:paused / fabrik:awaiting-input — reachable, since the pause paths apply those labels without touching the completion label — is not dispatched when a human comments. processItem carries the same guard ahead of its two resume branches as a redundant-but-explicit ownership boundary. The cleanup-stage exclusion is R1’s stated exception and covers worktree reaping only; the plain new-comment path additionally keeps its own !item.IsClosed condition so a closed item at a cleanup stage is still never routed into comment processing.

The stage:<X>:complete admission clause triggers a real FetchItemDetails deep fetch, pre-existing and unchanged by this ADR (raised on review, PR #1388). A closed item admitted solely because stage:<X>:complete is present enters deepFetchCandidates like any other admitted item — itemNeedsWork’s later “already completed this stage” check is what actually withholds dispatch, not itemMayNeedWork’s shallow filter. In the common case this is transient (the settle-owner or catch-up loop moves the item off its current column the same poll it heals it), but if the board move itself keeps failing (advanceToNextStage’s aerr/fillFailed paths in advanceValidateTerminalItem, or the equivalent in settleClosedItemsToDone), the item can sit indefinitely with stage:complete set and the board column unchanged, paying one FetchItemDetails call per poll. This clause is not new: it was present in the closed-issue guard before commit 7311a14e and is unmodified by R3’s simplification (R3 only removed the !stageIsGateChecked(stage), fabrik:awaiting-ci, and fabrik:auto-merge-enabled disjuncts) — so this ADR neither introduces nor changes this cost. A repeatedly-failing board move is itself an existing, unrelated failure mode (e.g. a missing status option, or a transient GitHub error) with no escalation path today; that gap is out of scope here, same as the “unresolved-PR polling cost” gap noted in §6.15.

Outbound case — engine-initiated close (ADR-1096): GitHub only honours Closes #N auto-close when the merged PR’s base branch is the repository’s default branch; on a base:<branch> repo where the PR targets a non-default base, the keyword is inert and nothing closes the issue, which stalls native dependency edges that unblock on close (not merge). Both terminal merge-advance paths — runValidatePRTerminalAdvance (cruise) and advanceConvergedPRToDone (non-train yolo) — call the shared helper closeIssueIfNonDefaultBase(item, prNumber) immediately after their board advance to Done, on a confirmed merge only (never for a PR closed without merging). The helper resolves the item’s base via the base:<branch> label (baseBranchForItem) and the repo’s actual default via wm.DefaultBaseBranch(); it calls CloseIssue only when they differ — when base equals default, GitHub’s own auto-close already handles it, and skipping here is what prevents a double-close. The close is best-effort and idempotent (already-closed / ErrNotFound treated as success, failures logged and never block the advance); a failed close is durably retried via the fabrik:awaiting-close settle scan — see §6.13, ADR-1097. The merge-train path (engine/merge_train.go) and the FABRIK_NO_WORK_NEEDED short-circuit (no_work_needed_settle.go) already close explicitly and are unaffected. GitHub’s inertness on a non-default base goes further than this close half alone, though: it creates no Development-panel issue↔PR link at all for such a PR, which is a purely human-visible gap the engine itself is unaffected by (FetchLinkedPR resolves by branch-name convention, not GitHub’s linkage graph) — see §6.20, ADR-1649 for the one-time notice that closes that gap.

Default-base backstop (#1962, ADR-1962): GitHub’s own auto-close on the default base is not trusted either. On 2026-09-30 GitHub silently stopped creating closing-keyword links for new PRs and dropped the auto-close even for already-linked ones, with no declared incident. When base equals default, closeIssueIfNonDefaultBase calls guardDefaultBaseAutoClose. It does not close immediately, because a landing path such as the singleton fast path reaches it within the auto-close’s own 1–2 s latency and closing then would race GitHub. Instead a live FetchIssue read (never the board snapshot, which predates the merge) decides:

settleNonDefaultBaseCloses then works through the same retry and escalation path. It live-reads the issue before acting, on every base, because the board snapshot can lag a late auto-close by a poll and CloseIssue is a bare PATCH that succeeds on an already-closed issue. If the live read shows the issue closed, the marker clears silently. If the read fails, the check waits for the next poll, and no “did not fire” comment is posted. That failed pass still counts toward the MaxRetries budget, so an issue that stays unreadable escalates instead of stalling. Otherwise the settle scan closes the issue with state_reason: completed and posts 🏭 **Fabrik — closed after merge**, naming the merged PR. It resolves the PR through FetchLinkedPR (by branch), since GitHub’s closing-keyword link is exactly what may be missing. Its escalation comment is worded for the default base (“post-merge explicit close failed”).

2.9 Review Reinvoke

Trigger: The catch-up loop Phase 1 detects unresolved PR review feedback — inline thread comments and/or an unaddressed review body from any review whose state is not DISMISSED or PENDING (Finding 4, #1375; widened from CHANGES_REQUESTED-only to any non-DISMISSED/PENDING state by #1045, since a bot reviewer that submits its findings exclusively as COMMENTED — e.g. Pruefer — was otherwise silently unaddressed; see §6.2) — on any stage:<X>:complete item (or fabrik:awaiting-ci item on a wait_for_ci: true stage) — regardless of whether the item has fabrik:yolo, fabrik:cruise, or any auto_advance config, and regardless of whether checkReviewGate currently reports the gate blocked (R1, #1375, amending ADR-1250 — see §6.1.1). Phase 1 runs unconditionally; only Phase 2 (stage advancement) is gated on those labels.

Code path: poll() catch-up loop Phase 1 → buildReviewFeedbackComments() (buildReviewThreadComments() + buildReviewBodyComments()) → cycle limit check → dispatchReviewReinvoke() → async goroutine → processComments() with synthetic comments

Distinct from regular comment processing because:

Yolo-merge guards against unresolved current-head review threads (#1207). Review Reinvoke’s own dispatch loop protects an item only while it is parked at a completed stage. With fabrik:yolo, Validate completing does not park anything — attemptMergeOnValidate (§2.10’s runValidatePRTerminalAdvance discussion and engine/stages.go) advances out of Validate or enables GitHub auto-merge in the same flow, and native auto-merge then leaves the PR pending for however long CI takes, with GitHub poised to merge the instant checks go green. A review finding arriving inside that window is invisible to both the Review Reinvoke trigger above (nothing is parked to reinvoke) and to §2.16’s SHA-invalidation scan (a review comment does not change the head SHA). Two guards close this, both built on currentHeadReviewThreadComments (which excludes threads GitHub marks isOutdated — superseded by a later push — so a stale-SHA thread never blocks indefinitely) rather than a parallel detection path — note the scope split: guard 1 (attemptMergeOnValidate) is scoped to inline thread comments only, unchanged by #1375; guard 2 (inside handleReviewGate) additionally covers unaddressed review bodies via currentHeadReviewFeedbackComments (currentHeadReviewThreadComments + all review-body comments, R8 — see §6.1.1/§6.2), since that guard sits directly alongside the wider Finding-4 reinvoke check it was already updated for:

Both guards are an explicit stopgap for fabrik:yolo specifically (see #1071, which is expected to supersede this mechanism with a decidable “hold a valid signature from every required signer for this SHA” check); fabrik:cruise is unaffected because it already stops at Validate for human merge, well before either guard point.

2.10 CI Check Completed

Trigger: CI check runs on the PR head SHA transition from pending to a terminal state (success, failure, etc.). Fabrik detects this by polling on each catch-up loop iteration — there are no webhooks.

Pending-over-failed precedence (#958): Both settlePRMergeState() and checkCIGate() classify check runs via the shared github.ClassifyCheckRuns() helper, which (1) reduces check runs to the latest run per check name (highest ID wins — a stale completed/failed run left behind by a rerun under a new ID is discarded) and (2) treats any pending (queued/in_progress) run as taking global precedence over any failed run, regardless of whether the failed run is a different check name or a superseded run of the same name. A single failed check-run coexisting with a pending check-run on the same head therefore classifies as WAIT (PRMergeUnsettled / ciBlocked=true, ciFailure=false), never FAIL — the engine waits for the pending check to reach a terminal state instead of dispatching a CI-fix reinvoke. Per ADR-1410, this wait is bounded by liveness (no observed check-run progress for CIWaitTimeout, tracked via LinkedPRState.LastCIProgressAt), not elapsed time — a suite that keeps transitioning check-run state waits indefinitely, up to the separate, much larger CIBackstopTimeout absolute cap. Both call sites use the same helper so they cannot drift out of agreement. See §6.4 rule 19 and §6.5.1 for the full classification rules.

Required-context awareness (ADR-933, #933): ClassifyCheckRuns() alone has no concept of “required” — skipped/neutral/absent check runs fall through to CheckRunsReady, which is correct for a non-required job but wrong for a required one that simply never ran (e.g. a repo whose required signal is a classic commit status posted out-of-band, with GitHub Actions disabled). settlePRMergeState() layers a required-context pre-filter (engine.classifyRequiredContexts() → github.ClassifyRequiredContexts()) in front of every point where check-run classification alone would otherwise resolve to PRMergeReady: the zero-check-runs “no CI configured” fallback (§6.4 rules 13 & 17) and the check-runs-present “all green” fallback (§6.4 rule 19). The pre-filter checks each name in the repo’s configured required_status_contexts (.fabrik/config.yaml, keyed by "owner/repo"; unconfigured repos are always a no-op — zero behavior change) against the union of check-run names and classic commit-status contexts (FetchCombinedStatus, fetched only when a required name isn’t already resolvable from check runs) observed on the exact head SHA. Only a confirmed success counts; a confirmed failure resolves to PRMergeBlocked, and anything else not-yet-confirmed (missing, pending, skipped, neutral) resolves to PRMergeUnsettled — never silently to PRMergeReady. checkCIGate() mirrors this with a matching pre-check, classifyCIFromRequiredContexts(), run ahead of classifyCIFromCheckRuns()/classifyCIFromMergeableState() — necessary because a required-context failure sourced solely from a classic commit status has no check-run footprint for those two functions’ checkRuns-only view to react to. The merge-train poller (pollTrainCI(), engine/merge_train.go — see §1.3’s FR-2 mention of its 30s poll interval) classifies check runs via the same shared github.ClassifyCheckRuns() helper (replacing a previously-separate inline duplicate) and applies the identical required-context pre-filter to its trial-branch SHA. See ADR-933 for the full required-context design.

pollTrainCI’s own mergeable_state shortcut — touched by ADR-1153, #1153. pollTrainCI no longer treats an accepted mergeable_state (clean/unstable) as sufficient for TrainCIGreen by itself. mergeable_state is computed by GitHub from required checks only, so it can read as accepted while a non-required check (e.g. the actual test suite, if left unmarked-required) is still queued/in_progress on the trial SHA — exactly what happened on integration PR #1150, where the train merged 20 seconds after the sole required check went green, 14 seconds before the non-required test job itself reported. The two signals now compose instead of racing:

Every terminal pollTrainCI decision now logs the specific check runs (name, status/conclusion) it was based on. See ADR-1153 for the full design, the Strict non-required-failure policy rationale, and its relationship to ADR-072 (the single-PR MergePR self-gate, which still accepts unstable and is unaffected) and ADR-933 (whose required-context classifier is reused unmodified).

Suite-aware green (#1822). pollTrainCI’s two green returns (all-green runs + required contexts satisfied; zero runs + accepted mergeable_state) and classifyLandingCI’s two green returns are each gated on ciSuiteHold(e.client, …): an outstanding suite keeps pollTrainCI polling (bounded by CIBackstopTimeout) and makes classifyLandingCI return TrainCIPending; a confirmed red still returns TrainCIRed immediately. See §6.4 Suite-aware completeness, ADR-1822.

CI infrastructure failures (#2052, ADR 2052). pollTrainCI also tells a CI provider failure from a failure of the code under test: on the zero-check-run path it reads the trial SHA’s workflow runs before the zero-check green shortcut above, and a startup_failure (or zero-job failure) run is retriggered by closing and reopening the trial PR (at most maxCIRetriggers = 2, counting only runs absent when the previous retrigger was issued) and then abandoned as TrainCIInfra — never red, never bisected, never charged; the members stay Queued. On a confirmed check-run failure it re-runs the failed jobs once (rerun-failed-jobs) before returning TrainCIRed; only a second failure is red. Full mechanics, the cooldown and the stage-gate counterpart are in §6.29.

pollForMergeable’s identical shortcut — fixed by ADR-1441, #1441 (previously the unfixed sibling ADR-1153 flagged and left open). pollForMergeable (engine/merge_train.go, the merge-train landing-step poller — distinct from pollTrainCI’s CI-poll step above) used to treat an accepted mergeable_state as an unconditional green light for landing, with no check-run fallback at all — not even the partial classification settlePRMergeState had for other values. It now fetches FetchPRDetails (for HeadSHA alongside MergeableState) and classifies via a new classifyLandingCI helper that mirrors pollTrainCI’s composition exactly (same shared primitives — gh.ClassifyCheckRuns, e.classifyRequiredContexts, describeCheckRuns, gh.MergeableStateAccepted — same ordering, same zero-check-runs fallback), reusing TrainCIResult as its verdict type: mergeable_state == "dirty" → immediate reject, no check-run fetch; a confirmed check-run or required-context failure → reject; a check still pending → keep polling (bounded by CIBackstopTimeout, exactly as before); zero check runs → the mergeableAccepted && RequiredContextsSatisfied fallback. classifyLandingCI is a new function, not literally shared code with pollTrainCI (out of scope for #1441, already fixed) — see ADR-1441 for why “one shared mechanism” is satisfied at the primitive level rather than by refactoring pollTrainCI itself.

Code path: poll() catch-up loop Phase 1 → settlePRMergeState() (one combined REST pass for mergeable, mergeable_state, and check runs) → checkCIGate() interprets the PRSettleResult:

Settling primitive consolidation (v0.0.53): settlePRMergeState() reads mergeable, mergeable_state, and check runs in one pass before both the merge-conflict gate and the CI gate. This eliminates the split-brain where two separate REST calls within one poll cycle could observe different GitHub state. See §6.4 for the full specification. MergeableState is intentionally omitted from PRSettleResult in the hadChecks, post-push dwell, and HeadSHA-empty cases so that checkCIGate’s R3 path (MergeableState == "blocked" + empty CheckRuns) cannot misfire on those cases. The mergeable_state field is null on the list endpoint used by FetchLinkedPR, so FetchPRMergeableFields() (a separate single-PR REST call) provides both mergeable and mergeable_state in one request.

(false, false, false) return semantics: checkCIGate returns (false, false, false) for all of the following terminal conditions, which it handles internally (the poll.go call site never sees ciTimedOut=true for these): R1 merged PR, R2 closed-not-merged PR, R3 required-never-running check, no linked PR, all-green checks. The caller only calls pauseForCITimeout when timedOut=true — since ADR-1410, this fires only for a genuine liveness-dwell-exceeded condition (check-run progress stalled, or the R3/mergeable-state elapsed dwell), never for a confirmed CI failure (which returns ciFailure=true instead, unconditionally).

Distinct from Review Reinvoke because:

Single-owner Validate advance, split into an open-item and closed-item pair (runValidatePRTerminalAdvance / settleClosedValidateAdvance, R4 — ADR-056 D2, ADR-1387): Both functions delegate to the same shared per-item logic, advanceValidateTerminalItem, and differ only in what they iterate and where they’re called from. For each candidate item, e.client.FetchLinkedPR is called directly (not e.readClient — boardcache may have stale Merged/State), and the logic runs regardless of which gate label (fabrik:awaiting-ci, fabrik:awaiting-review, fabrik:rebase-needed, or any future label) is present — no disjointness maintained by label negation anywhere. Items already in advancedItems are skipped (no double-advance), and items carrying fabrik:auto-merge-enabled are excluded (that convergence flow is owned exclusively by checkAutoMergeConvergence, Phase 1). When pr.Merged == true: the shared logic iterates all pipeline stages in ascending Order from the stage after the highest already-complete stage up to (but not including) the cleanup-terminal stage, adding stage:<N>:complete for every WaitForCI or WaitForReviews gate-checked stage whose label is absent (with cache write-through; fail-fast on error for idempotent retry). After all completion labels are added, removeAwaitingCILabel, removeAwaitingReviewLabel, and removeRebaseNeededLabel are called as applicable, fabrik:paused + fabrik:awaiting-input are removed, and advanceToNextStage() is called. Immediately after the Done advance, closeIssueIfNonDefaultBase(item, pr.Number) runs (ADR-1096) and explicitly closes the issue when its resolved base:<branch> differs from the repo default — see §2.8. When pr.State == "closed" and !pr.Merged (closed without merging): pauseForPRClosedNotMerged() is called unless the item is already paused (to avoid duplicate comments). Neither function may ever dispatch workers or acquire e.sem (carries ADR-053 constraints; superseded by ADR-057).

Note: Non-Validate-stage items with fabrik:paused and a merged PR are not handled by either function; they require manual label intervention to clear fabrik:paused, after which the normal catch-up loop advances them.

2.11 Base Branch Advanced

Trigger: The PR’s base branch moves forward (a different PR merges) while this branch is sitting in the post-stage:Validate:complete catch-up window. GitHub recomputes mergeable on the linked PR; if the new base conflicts with this branch, mergeable transitions from true (or null) to false.

Code path: poll() catch-up loop Phase 1 → settlePRMergeState() (combined REST pass: FetchLinkedPR + FetchPRMergeableFields for mergeable and mergeable_state) → checkMergeabilityGate() interprets PRSettleResult → evaluates the flag → optionally dispatches dispatchRebaseReinvoke() → async goroutine → processComments() with a synthetic rebase-required comment

Distinct from CI Check Completed because:

2.12 TurnProgressEvent (TUI Display Event)

Trigger: A {"type":"user"} NDJSON line is written to Claude’s stdout pipe during a Claude invocation. Each logical turn (one user→assistant cycle) begins with exactly one such line (either the initial prompt or a tool-result response), so this fires once per logical turn.

Code path: runClaude() stdout pipe → turnCountingWriter.Write() → detects type == "user" line → increments per-invocation counter → calls claudeTurnProgress(issueNumber, turnsUsed, maxTurns) → Engine.emit(TurnProgressEvent{...}) → TUI channel

Effect: Purely additive display — does not trigger any state transitions, label mutations, or issue processing. The TUI consumes TurnProgressEvent to update the live turn counter shown in:

TurnProgressEvent is only emitted in TUI mode (claudeTurnProgress is nil in plain-text mode and tests). It uses the non-blocking emit path (drop-if-full), so turn-progress updates are best-effort and may be dropped under backpressure. This does not affect engine behavior because the event is display-only; at most one event is produced per logical turn.

MaxTurns in the event carries the effective budget for the current invocation — effectiveBudget as computed in InvokeClaude() (which already accounts for opts.MaxTurnsOverride from the extension loop). This means:

2.13 Manual Assignee Change

Webhook event: issues.assigned / issues.unassigned

Detection: applyIssuesDelta (boardcache/delta.go:382) applies IssueAssigneesUpdated, which emits the AssigneesChanged flag. The mayNeedWorkObserver and wakeChObserver (engine/observers.go) include AssigneesChanged in wakeChFlags, so the assignment fires both a wake signal and a mayNeedWork cycleSet entry.

Effect: Dispatcher re-evaluates the item on the next poll cycle. The engine does not currently filter dispatch on assignee — assignee changes do not change what work happens, only that the item is re-considered. Future assignee-as-dispatch-filter work (planned, not yet filed) will give this event additional dispatch semantics.

Why: Assignment is a strong “please look at this” signal from the user, and is the mechanical underpinning of multi-user shared boards (each fabrik instance picking up only items assigned to its cfg.User).

2.14 Worker Lifecycle

Source: Engine-internal mutation, not a webhook.

Detection: The dispatch loop (and each reinvoke dispatcher — dispatchReviewReinvoke, dispatchCIFixReinvoke, dispatchRebaseReinvoke) applies WorkerEntered{Repo, Number, StageName, StartedAt} synchronously before the goroutine is launched. WorkerExited{Repo, Number} is deferred at the top of each goroutine so it fires on any exit path (context cancel, ensureRepoReady failure, normal return). Both mutations emit WorkerChanged | WorkerLifecycleChanged; WorkerLifecycleChanged (not the broader WorkerChanged) is the flag in wakeChFlags. WorkerHeartbeat and WorkerPIDSet emit only WorkerChanged and do not wake the poll loop — this prevents deep-fetch churn for active workers (30s heartbeat × N workers would otherwise trigger repeated deep-fetches for items that can’t be dispatched anyway).

Effect: WorkerExited adds the item to mayNeedWork and, when wakeCh != nil, fires the wakeChObserver so the dispatcher re-evaluates within milliseconds. In headless runs (--notui or any configuration without a wake channel), the mutation populates mayNeedWork for the next ticker-driven poll (within PollSeconds); there is no immediate wake. Either way, re-evaluation does not depend on cooldown expiry or external events. This eliminates the previous race where self-advance to the next stage would wait up to 150s (PollSeconds × 10) if the departing worker had not finished cleanup before the post-advance dispatch loop ran.

Dispatch guard: The dispatch loop uses snap.Worker() != nil (Store-backed) instead of the former e.inFlight.Load(iKey) (sync.Map). Because WorkerEntered is applied before wg.Add(1) and before the goroutine starts, snap.Worker() != nil is true from the instant the goroutine is scheduled — there is no window where a new dispatch cycle could race in and double-dispatch the item.

Why: Worker lifecycle is engine state the dispatcher must react to. Pre-Fix B (issue #544), it lived in e.inFlight (sync.Map) outside the Store — a known bypass that violated ADR 036’s single-owner reactive cache invariant. WorkerEntered/WorkerExited complete the migration begun by the Phase 5 F3 store unification.

2.15 Revalidate Label (fabrik:revalidate)

Source: Operator-applied GitHub label.

Detection: A dedicated revalidate-scan loop, settleRevalidateScan (engine/poll_settle.go; called from poll() in engine/poll.go), runs after the paused-item recovery loop and before the dispatch loop. It iterates all deepFetchCandidates unconditionally (paused items included — FR-5). Items that don’t carry fabrik:revalidate are skipped immediately. The loop is reached by stuck-Validate items because fabrik:revalidate is included in the hasAwaitingLabel pre-filter bypass at poll.go:1294, ensuring items with settled updatedAt are still deep-fetched on every poll.

Effect (Validate-stage item, no in-flight worker):

  1. Removes stage:Validate:complete, stage:Validate:failed, fabrik:paused, fabrik:awaiting-input, fabrik:awaiting-ci, fabrik:auto-merge-enabled from GitHub (best-effort sequential REST calls; 404 responses are treated as already-absent and are not errors).
  2. Removes fabrik:revalidate itself from GitHub.
  3. For each successful removal: cacheImpl.ApplyLabelRemoved + webhookMgr.RegisterEcho (cache write-through and echo registration, same pattern as all other label mutations in the engine).
  4. Applies four store resets: StageRetryCleared, EngineUnpaused, StageLastAttemptCleared, EngineCyclesCleared — all for StageName: "Validate". This is the same full-reset sequence used by clearFailedStage.
  5. Validate dispatches on the next poll cycle: the revalidate scan does not update deepFetchCandidates in place. After label removal, the item re-enters the board on the next poll with no blocking labels and no LastAttemptAt cooldown, so itemNeedsWork returns true and the dispatch loop invokes processItem.

Effect (non-Validate item): Logs a warning, removes only fabrik:revalidate, takes no other action.

In-flight worker guard (FR-4): If snap.Worker() != nil (a Validate invocation is still active), the label is left in place and the revalidate scan defers to the next poll cycle. When the worker exits, WorkerExited populates mayNeedWork and the next poll processes the label normally.

Partial label-removal failure: Each REST call is independent. If a removal fails (non-404 error), the engine logs a warning and continues. On the next poll, the item is still deep-fetched (if fabrik:revalidate was not yet removed), and the revalidate scan retries the remaining removals idempotently.

Closed-issue cleanup: fabrik:revalidate is in transientLifecycleLabels, so cleanupClosedIssueTransientLabels sweeps it from closed issues defensively (FR-7).

Why: Recovery from a stuck-Validate state previously required knowing and clearing 4–5 individual labels. fabrik:revalidate collapses recovery to one action. It stays useful even after structural CI-gate fixes (#855) for operator-initiated re-runs when CI infrastructure has recovered out-of-band, a flaky test was resolved manually, or other non-engine causes.

2.16 SHA-Invalidation Scan

Source: Automatic — triggered by PRHeadSHAUpdated events from any source (webhook, reconcile, deep-fetch probe).

Detection: A SHA-invalidation scan, settleSHAInvalidationScan (engine/poll_settle.go; called from poll() in engine/poll.go), runs immediately after the revalidate scan (§2.15) and before the dispatch loop. It iterates all deepFetchCandidates. Items that don’t carry stage:Validate:complete are skipped immediately. For qualifying items it reads snap.ValidateCompletedSHA() (the HEAD SHA recorded when Validate last completed) and snap.LinkedPR().HeadSHA (the current PR HEAD). A mismatch triggers invalidation unless one of the two exemptions below applies (#2158). Items are admitted to deepFetchCandidates via the standard PRHeadSHAUpdated → LinkedPRChanged → mayNeedWorkObserver → cycleSet observer pipeline — no bypass label is required.

Effect (SHA mismatch, no in-flight worker):

  1. Removes fabrik:rebase-needed, fabrik:auto-merge-enabled, fabrik:awaiting-ci, fabrik:awaiting-review, stage:Validate:complete from GitHub (best-effort sequential REST calls; 404 responses are treated as already-absent and are not errors). fabrik:rebase-needed is included because a conflict determination tied to the invalidated completion SHA no longer describes reality (#1225).
  2. For each successful removal: cacheImpl.ApplyLabelRemoved + webhookMgr.RegisterEcho (cache write-through and echo registration).
  3. Applies ValidateCompletedAtSHACleared to zero LinkedPR.ValidateCompletedSHA in the Store.
  4. Applies four store resets: StageRetryCleared, EngineUnpaused, StageLastAttemptCleared, EngineCyclesCleared — all for StageName: "Validate".
  5. Validate dispatches on the next poll cycle: after label removal, the item re-enters the board with no blocking labels and no LastAttemptAt cooldown, so itemNeedsWork returns true and the dispatch loop invokes processItem.

Guards:

Loop prevention: The SHA is recorded AFTER Claude’s final commit and AFTER the worktree HEAD is finalized (git rev-parse HEAD in the worktree). The subsequent PRHeadSHAUpdated event for Validate’s own commit arrives with SHA == ValidateCompletedSHA, so the mismatch check returns false and the scan is a no-op. The loop is architecturally closed. The two exemptions above do not weaken this: a skipped landed item never reaches the label removal, and an accepted catch-up head is re-recorded as the completion SHA, so the next evaluation sees HeadSHA == ValidateCompletedSHA.

Relationship to the comment-rework clear/restore (#1802, ADR-1802): this scan is the direct precedent for #1802’s fabrik:reworking:<Stage>/beginStageRework/endStageRework mechanism — both clear a stage’s stage:<Stage>:complete when it has gone stale and let the normal completion flow re-populate it. The two are independent and never coordinate directly: this scan only ever targets stage:Validate:complete, triggered by a SHA change, outside any comment re-entry; #1802’s mechanism targets whichever stage is being re-entered, triggered by processComments, bracketed by fabrik:editing. Both removal and (re-)addition of stage:Validate:complete are idempotent GitHub label mutations, so the two mechanisms running in the same window — e.g. a SHA change arriving while a comment re-entry is reworking Validate — cannot double-apply or conflict; each just mutates the label set to match its own view of truth.

Partial label-removal failure: If any REST removal fails (non-404 error), hasError is set and the function returns without applying Store mutations. stage:Validate:complete remains; the next poll re-runs the scan and retries idempotently.

Why: stage:Validate:complete validates a SHA, not an issue. A force-push, external commit, or operator rebase after Validate completes produces a new artifact that Fabrik has not validated. Without this scan, attemptMergeOnValidate’s auto-merge enablement would target the new SHA against a gate Fabrik never ran, risking auto-merge of unvalidated code. The scan closes this gap automatically for any SHA change, including ones the operator did not intend as a re-Validate signal.


2.17 Consecutive Resume Failure (Session Abandonment)

Source: Automatic — evaluated inline at the end of every Claude invocation, on both invocation paths: InvokeClaude (the primary stage run) and InvokeClaudeForComments (comment review), and transitively merge_train.go’s conflict-resolution invocation (which also goes through InvokeForComments). Unlike every other event in this section, it is not detected by a settle scan reading board/label state — it is classified structurally, in-process, from the CLI’s own result object.

Detection: interpretClaudeResult (engine/claude.go) is the single funnel all three invocation paths converge on. When an invocation fails for a reason none of the more specific classifiers already claim (stale session, turn-cap error_max_turns, usage-limit blocking_limit, transient api_error), and the failing invocation itself passed a non-empty --resume session ID, classifyResumeFailure reads a plain-text sidecar counter file colocated with the session pointer (<Stage>.session.resumefails, alongside <Stage>.session) and increments it. The counter is durable across an engine restart — deliberately a plain file, not itemstate.Store (confirmed fully in-memory, resets on restart), since every sibling counter’s resting place (SliceRetries, ReviewCycles, etc.) does not need to survive a restart but this one must. The counter is keyed purely to the session-pointer file path, which both InvokeClaude and InvokeClaudeForComments compute identically for a given (issue, stage) — so a failure from either path increments the same counter, and an alternating stage/comment-review failure sequence still reaches the threshold.

Effect (count reaches MaxResumeFailures, default 2):

  1. Logs the abandonment (claudeLog, tag resume): session id, stage, consecutive-failure count, threshold, and the last error — the sole record of this event; see Guards below for why no comment or label is used.
  2. Removes the session pointer file — the exact same os.Remove call the pre-existing stale-session path (§ “the errors[] substring No conversation found with session ID”) already uses.
  3. Resets the sidecar counter to 0.
  4. The invocation’s error is wrapped as *claudeResumeFailureError (Cause, SessionID, ConsecutiveFailures, Threshold, Abandoned: true) and returned to the caller — for every occurrence of this branch, not only the one that crosses the threshold, so a below-threshold failure is also visible to finalizeStageOutcome’s MaxRetries exemption (below).
  5. No explicit “force cold start” signal is threaded anywhere: resolveResumeSessionID already treats an absent session file as a cold start identically for both InvokeClaude and InvokeClaudeForComments (logs a warning, returns ""). The file’s absence at step 2 is the cold-start signal for whichever invocation path runs next — guaranteed sequential, never concurrent, by the existing per-issue single-worker lock.

Effect (below threshold — count incremented but not yet at MaxResumeFailures): Same wrapped *claudeResumeFailureError is returned (with Abandoned: false), but the session pointer is left in place — only the sidecar counter is written. The next invocation still resumes normally; only reaching the threshold triggers abandonment.

Effect (finalizeStageOutcome, engine/item.go, stage path only): resumeFailed := errors.As(err, &resumeFailErr) is computed alongside the pre-existing turnLimited check. Unlike the usage-limit/api_error “did-not-run” family (which early-return before commitWIP/push), a resume failure does not short-circuit — a resumed session can break after real work happened, so commitWIP, the branch push, markCommentsSeenByStage, and InvocationRecorded all still run exactly as for any other incomplete run. StageAttempted is recorded unconditionally for any invocation that ran, satisfying the dispatch-cooldown requirement with no new call. In the final escalation block, resumeFailed is a third branch alongside turnLimited: it skips StageRetryIncremented entirely — for every consecutive resume failure up to MaxResumeFailures, not only the one that triggers abandonment, mirroring the fabrik:claude-limit precedent (ADR-1119). No label, no comment is applied from this branch.

Effect (engine/comments.go, comment-review path): No control-flow change. A resume failure is not excluded from checkCommentBreaker’s “no forward progress” counting the way a usage-limit hit is — it counts toward the comment-processing circuit breaker exactly like *claudeAPIErrorExit already does (ADR-1458), since the breaker is this path’s only bound and a resume failure, unlike an account-wide usage limit, is specific to this issue’s own session.

Guards:

Why: A --resume invocation resumes the same session every time, including when the session’s own transcript (grown too large, or otherwise structurally broken) is the cause of the failure — every retry re-triggers the identical condition. Because InvokeClaude and InvokeClaudeForComments compute the identical session-pointer path for a given (issue, stage), a transcript poisoned by a long stage run equally poisons the next comment-review invocation, and vice versa; the counter and the abandonment must therefore be shared across both paths rather than kept separately, or an alternating failure sequence between the two paths would never reach the threshold. See #1414, ADR-1414.


3. Transition Tables

3.1 Happy Path — Linear Stage Progression

This table shows the normal flow when an issue progresses through the pipeline without interruption.

Current State Event Guard Resulting State Labels Added Labels Removed Side Effects
Column <X>, Idle Poll tick Stage exists, not paused/locked/editing/blocked, cooldown expired Column <X>, Locked + In Progress fabrik:locked:<user>, stage:<X>:in_progress   Lock-then-verify protocol (2s delay), worktree ensured, Claude invoked
Column <X>, Locked + In Progress FABRIK_STAGE_COMPLETE shouldAdvance=false (see below) Column <X>, Complete stage:<X>:complete fabrik:locked:<user>, stage:<X>:in_progress, stage:<X>:failed (if present) Output posted; draft PR created (if create_draft_pr); PR marked ready (if mark_pr_ready_on_complete); lock released
Column <X>, Complete Human moves to next column — Column <Y>, Idle     Manual board column move
Column <X>, Locked + In Progress FABRIK_STAGE_COMPLETE shouldAdvance=true (see below) Column <Y>, Idle stage:<X>:complete fabrik:locked:<user>, stage:<X>:in_progress, stage:<X>:failed (if present) Output posted; draft PR / mark ready (if configured); board column updated to next stage; lock released
Column <X>, Complete Poll tick (catch-up) yolo or cruise active, stage:<X>:complete present, no pending comments Column <Y>, Idle     Board column updated to next stage (Path 2 advancement)

shouldAdvance resolution (Path 1, handleStageComplete):

  1. yoloActive = cfg.Yolo || hasYoloLabel(item) — re-fetches labels first to pick up mid-run changes
  2. cruiseActive = !yoloActive && hasCruiseLabel(item) — false when yolo is active. This is not a precedence rule: it only avoids double-counting in step 3, where yoloActive already contributes the same true. Step 5 keys off the raw label, so cruise still wins at Validate.
  3. shouldAdvance = yoloActive || cruiseActive
  4. If stage.AutoAdvance != nil AND neither fabrik:yolo nor fabrik:cruise label is present: shouldAdvance = *stage.AutoAdvance — this means auto_advance: false in YAML overrides cfg.Yolo (the --yolo flag), but explicit yolo/cruise labels override auto_advance: false
  5. If hasCruiseLabel(item) && stage.Name == "Validate": shouldAdvance = false — cruise stops at Validate. Note the raw label, not cruiseActive: this fires even when yolo is also present, which is what makes cruise win the stop-at-Validate decision.

Catch-up loop shouldAdvance resolution (Path 2): runCatchUpPhase2 calls refreshAutonomyLabels(&item) first (#1769, §6.23) — mirroring Path 1’s step 1 — so this decision also reads live label state, not whatever the item snapshot happened to carry into the catch-up loop. It then checks cfg.Yolo || hasYoloLabel(item) || hasCruiseLabel(item) — items without any of these are skipped entirely. Then: if neither yolo nor cruise LABEL is present and stage.AutoAdvance is explicitly false, the item is skipped. This produces the same behavior as Path 1.

Self-advance wake guarantee (Fix B, #544): When advanceToNextStage runs, two independent wake events fire: the new column status (LocalStatusUpdated → StatusChanged) and the worker exit (WorkerExited → WorkerLifecycleChanged). Both flags are in wakeChFlags. When wakeCh != nil (TUI mode or any wake-channel-enabled setup), the wakeChObserver fires and the dispatcher re-evaluates within milliseconds, without waiting for cooldown expiry. In headless runs (--notui or any configuration without a wake channel), re-evaluation occurs within PollSeconds via the next ticker poll. This eliminates the previous race where the departing worker was still alive when the post-advance dispatch loop ran, causing the item to receive a 150s CooldownAt("periodic-re-eval") stamp and wait the full cooldown window before the next stage was dispatched.

The same guarantee applies to the Review → Validate catch-up advance: when a review reinvoke worker exits after clearing the review gate, WorkerExited → WorkerLifecycleChanged wakes the poll loop and adds the item to mayNeedWork. The next poll’s catch-up loop sees stage:Review:complete and advances to Validate (typically within 15s). Pre-Fix B, this transition depended on external event noise — e.g., an unrelated check_run webhook for a different PR opportunistically waking the dispatcher — because CooldownAt("review-blocked") was still active from the gate-waiting period. The CooldownAt("review-blocked") retry timer (10 × PollSeconds) remains valid for non-responsive bot reviewers where no event fires, but it is no longer the primary re-admission path after the gate clears.

Validate → Done special cases:

Current State Event Guard Resulting State Labels Added Labels Removed Side Effects
Validate, Locked + In Progress FABRIK_STAGE_COMPLETE yolo active Validate, Complete + Awaiting Merge stage:Validate:complete, fabrik:auto-merge-enabled fabrik:locked:<user>, stage:Validate:in_progress GitHub auto-merge enabled via enablePullRequestAutoMerge; board advanced to Done only after GitHub merges the PR (deferred convergence monitor, §5.5). For wait_for_ci: true stages, fabrik:awaiting-ci is added instead and stage:Validate:complete is deferred to checkCIGate (ADR 032).
Validate, Complete Poll tick (catch-up) yolo active Done, Pending Cleanup     Convergence monitor (checkAutoMergeConvergence) detects GitHub has merged the PR; board column updated to Done (§5.5)
Validate, Locked + In Progress FABRIK_STAGE_COMPLETE cruise active (no yolo) Validate, Complete stage:Validate:complete fabrik:locked:<user>, stage:Validate:in_progress Cruise stops here — no merge, no advancement to Done
Validate, Complete Poll tick (catch-up) cruise active (no yolo) Validate, Complete     Cruise catch-up skips Validate — no merge, no advancement
Done, Pending Cleanup Poll tick Worktree exists on disk Done, Complete stage:Done:complete   Worktree removed from disk

3.2 Off-Path Transitions

Pause / Unpause

Current State Event Guard Resulting State Labels Added Labels Removed Side Effects
Any active column, Idle Human adds fabrik:paused — Same column, Paused     Engine skips item on next poll
Same column, Paused Human removes fabrik:paused — Same column, Idle     Engine processes item on next poll
Same column, Paused New human comment Not a bot login (github.IsBotLogin) and created at/after the pause (resumeAuthorised(), #1813) Same column, Idle → comment processing   fabrik:paused Unpause; clearFailedStage() also called (clears any failed label + resets retries); falls through to processComments(). A bot-authored comment does not match this guard and leaves the item Paused (#1083).

This table covers processItem’s own unpause gate — reachable only for a pause that fires before the stage is marked stage:<X>:complete (escalateFailedStage, escalatePRCreationFailure, handleBoundaryViolation, pauseForSliceLimit). A pause that fires after the stage is already complete (the four cycle-limit pause sites: review, CI-fix, rebase, enqueue — see §7.2 “Resumable Engine Pauses”) is resumed by a different mechanism, handleEngineUnpause in the catch-up loop’s Phase 1 handler chain (§6.2, Handler 0) — processItem is never dispatched for a stage-complete item with no new comment, so its gate above is structurally unreachable for those four sites. See ADR-1460.

Lock Conflict (Multi-Instance)

Current State Event Guard Resulting State Labels Added Labels Removed Side Effects
Any column, Idle Poll tick (two instances) Both acquire lock Depends on tie-break fabrik:locked:<user> (both) Loser’s lock removed 2s verify delay; lexicographic tie-break: lower username wins, higher username yields
Any column, Locked by Other Poll tick fabrik:locked:<other> present Same (skipped)     itemNeedsWork returns false; processItem also checks and skips

Dependency Blocking

Current State Event Guard Resulting State Labels Added Labels Removed Side Effects
Any column, Idle Poll tick Open blockers in item.BlockedBy Same column, Blocked fabrik:blocked   Comment posted listing blockers (first time only); TUI event emitted
Same column, Blocked Blocker closes (StateChanged) OR dependent’s BlockedBy first populated via deep-fetch (BlockedByChanged) All of X’s blockers now CLOSED (store-side view) Same column, Idle   fabrik:blocked Push path: PushUnblockObserver fires immediately; StateChanged scans all items (O(n)); BlockedByChanged checks only the dependent item (O(k blockers)); works for any column including Backlog/Done; no comment posted
Stage column, Blocked Poll tick (dep-blocked cooldown retry) All blockers now CLOSED (dep.State view) Same column, Idle   fabrik:blocked Pull path (defense-in-depth): processItem → checkDependencies; only for items in stage columns

Awaiting Input (FABRIK_BLOCKED_ON_INPUT)

Current State Event Guard Resulting State Labels Added Labels Removed Side Effects
Column <X>, Locked + In Progress FABRIK_BLOCKED_ON_INPUT completed false, no error Same column, Awaiting Input fabrik:paused, fabrik:awaiting-input fabrik:locked:<user>, stage:<X>:in_progress Lock released
Same column, Awaiting Input New human comment Not a bot login (github.IsBotLogin) and created at/after the pause (resumeAuthorised(), #1813) Same column → comment processing   fabrik:paused, fabrik:awaiting-input unblockAwaitingInput() clears LastAttemptAt/retry state for the stage, applies EngineCyclesCleared (resets ReviewCycles/ReviewBlockedCycles/CIFixCycles/RebaseCycles/EnqueueCycles/NoOpCommentCycles, §4.7) and resets §4.6’s CommentBreaker — the same full reset clearFailedStage applies for its own pause shape (#1555) — then routes to processComments(). A bot-authored comment does not match this guard and leaves the item Awaiting Input (#1083).

TUI Manual Stop (s-key)

Current State Event Guard Resulting State Labels Added Labels Removed Side Effects
Column <X>, Locked + In Progress User presses s in Active pane, confirms with y Job is selected in TUI Active pane Same column, Awaiting Input fabrik:paused, fabrik:awaiting-input fabrik:locked:<user>, stage:<X>:in_progress TUI sends StopRequest on stopCh; handleStopRequest stores "user_stop" in killReasonHolder and calls cancel() on the per-issue context; worker exits via SIGTERM→SIGKILL escalation. stage:<X>:in_progress is now cleared directly by handleStopRequest itself (removeInProgressLabel), not by the worker goroutine’s own release() path (ADR-1393 R2 — closes a pre-existing gap where the label’s removal raced the worker’s exit). Labels + comment are applied through the shared pauseInterruptedIssue primitive (ADR-1393 R3, engine/mutate.go) — the same one the daemon-wide clean stop below uses — which skips re-posting the comment if the store already shows fabrik:paused for this issue (idempotency guard). Resume: user removes fabrik:paused only — fabrik:awaiting-input is auto-cleared on next stage entry by unblockAwaitingInput(). If the worker has already exited when the stop arrives (no entry in issueCtxs), labels and comment are still applied. Any uncommitted worktree changes are committed (chore: partial <Stage> stage progress (incomplete)) and pushed by the worker’s own cancellation path before it exits (ADR-1393 R8) — asynchronous to, and not ordered against, this comment/label write.

Daemon Clean-Stop Shutdown (SIGINT/SIGTERM)

Generalizes TUI Manual Stop to every issue with a live worker at once, on receipt of the first SIGINT/SIGTERM (bare-CLI Ctrl-C, or the TUI’s q/Ctrl-C quit path, which self-raises SIGTERM on its own PID — both routes converge on the same signal handler). See ADR-1393 for the full design record.

Current State Event Guard Resulting State Labels Added Labels Removed Side Effects
Any column, Locked + In Progress (any issue with an active Worker() in the store) First SIGINT/SIGTERM — Same column, Awaiting Input fabrik:paused, fabrik:awaiting-input stage:<X>:in_progress The signal handler stores "daemon_shutdown" in every live issueCtxs holder, enumerates every store item with Worker() != nil into a snapshot (inFlightSnapshot()), then cancels the root context and launches runShutdownPause(inFlight) (engine/shutdown.go) as one more unit of work on e.wg — in that order. The enumeration happens strictly before cancel(), not inside runShutdownPause’s own goroutine: a worker cancelled before/while starting its Claude subprocess can reach its own cancellation branch and exit fast enough to beat a post-cancel scan, which would otherwise silently drop it from the pause set (review finding). One goroutine per issue in the captured snapshot then, in parallel, clears stage:<Name>:in_progress directly and pauses via the same pauseInterruptedIssue primitive TUI Manual Stop uses (ADR-1393 R2/R3), posting a "paused by a daemon clean stop" audit comment naming the interrupted stage. Idempotent under a retried/duplicate call: if the store already shows fabrik:paused, only the labels are reapplied — no second comment. fabrik:locked:<user> is removed separately by the pre-existing cleanupLockedIssues(), unchanged by this issue.
— (idle queue: no issue has an active Worker()) First SIGINT/SIGTERM No in-flight workers — — — runShutdownPause() returns immediately having written nothing — a stop with an idle queue is unchanged in speed and noise from before ADR-1393 (R9).
Any Second SIGINT/SIGTERM during drain — Process exits (os.Exit(1)) — — Force-quit: unconditional, unaffected by the pause-write phase’s progress. cleanupHook runs (releases the terminal), then os.Exit(1) — no label cleanup, no wait. This is the operator’s escape hatch when a clean stop is itself wedged (R5).
Any Drain deadline elapses with no second signal e.wg (workers + pause-write phase) has not completed within DrainDeadline (default 30s, --drain-deadline/FABRIK_DRAIN_DEADLINE) Process exits (Run() returns) — — waitGroupTimeout (engine/shutdown.go) bounds the same drain that used to be an unbounded e.wg.Wait(); on timeout a warning is logged and drainAndExit proceeds to exit anyway — any still-running pause-write goroutines are abandoned (accepted, not fixed — R5 requires this path to always terminate promptly).

Any cancellation reaching finalizeStageOutcome’s early-out (ctx.Err() != nil — daemon shutdown, TUI stop, or any future cancellation reason) now commits and best-effort pushes any uncommitted worktree changes before releasing the lock (ADR-1393 R8) — see the TUI Manual Stop row above for what a resumed issue’s worktree looks like. This is a shared code path, not something specific to the daemon-shutdown case.

Startup recovery (ADR-1393 R7): a stage:<Name>:in_progress label that survives with neither a fabrik:locked:<any-user> label nor a stage:<Name>:complete/failed sibling — the shape a shutdown pause write’s own failure, a crash, or a force-quit can still produce despite the direct clear above — is healed by a third startup pass, runStartupBareInProgressScan (§9.7), alongside the two pre-existing conditional passes.

Awaiting Review (wait_for_reviews gate)

Current State Event Guard Resulting State Labels Added Labels Removed Side Effects
Column <X>, Locked + In Progress FABRIK_STAGE_COMPLETE wait_for_reviews: true, shouldAdvance Same column, Awaiting Review stage:<X>:complete, fabrik:awaiting-review fabrik:locked:<user>, stage:<X>:in_progress Path 1: optimistic label application; lock released; returns without advancing
Same column, Awaiting Review + Complete Poll tick (catch-up) Outstanding reviewers remain, not timed out Same (blocked) fabrik:awaiting-review (idempotent)   checkReviewGate logs pending reviewers
Same column, Awaiting Review + Complete PR review submitted All reviewers submitted Same column, Complete → advance   fabrik:awaiting-review Gate cleared; falls through to advance or review reinvoke
Same column, Awaiting Review + Complete Poll tick (catch-up) Timeout elapsed Same column, Awaiting Input fabrik:paused, fabrik:awaiting-input fabrik:awaiting-review pauseForReviewTimeout() posts explanatory comment
Validate column, Awaiting Review + Complete, not fabrik:auto-merge-enabled Poll tick Linked PR is merged (pr.Merged == true) Done stage:<X>:complete (for each gate-checked stage missing the label) fabrik:awaiting-review, fabrik:paused, fabrik:awaiting-input (if present) R4 (single owner — ADR-056 D2): runValidatePRTerminalAdvance — gate-label agnostic; applies symmetrically to fabrik:awaiting-review items just as to fabrik:awaiting-ci items. See Awaiting CI R4 row below.

Awaiting CI (wait_for_ci gate)

In the conjunctive gate design (ADR 032), stage:X:complete is withheld until the CI gate actually clears. handleStageComplete adds fabrik:awaiting-ci as the durable in-flight marker; checkCIGate adds stage:X:complete once CI passes.

Current State Event Guard Resulting State Labels Added Labels Removed Side Effects
Column <X>, Locked + In Progress FABRIK_STAGE_COMPLETE wait_for_ci: true Same column, Awaiting CI fabrik:awaiting-ci fabrik:locked:<user>, stage:<X>:in_progress Conjunctive gate: stage:<X>:complete NOT added here — deferred to checkCIGate when CI passes (ADR 032). fabrik:awaiting-review is NOT seeded here (even when wait_for_reviews: true) — Path 2 (checkReviewGate in catch-up loop) fires after CI clears and stage:X:complete is present, and Path 3 (reviewGateBlocksLanding, §6.6.6) gates the landing decision on the pass CI clears. Dispatcher will not re-invoke while fabrik:awaiting-ci is present (itemNeedsWork returns false for R3).
Same column, Awaiting CI Poll tick (catch-up) Linked PR is merged (pr.Merged == true) Same column, Complete → advance stage:<X>:complete fabrik:awaiting-ci R1: checkCIGate detects merged PR before fetching check runs. addCompleteLabelAndRemoveCI runs; falls through to advance. No manual intervention required.
Same column, Awaiting CI Poll tick (catch-up) Linked PR is closed without merging (pr.State == "closed" and pr.Merged == false) Same column, Awaiting Input fabrik:paused, fabrik:awaiting-input fabrik:awaiting-ci, fabrik:awaiting-review, fabrik:rebase-needed (whichever present — ADR-1387 R6) R2: pauseForPRClosedNotMerged() posts comment naming the PR number. Human must reopen or create a new PR and remove fabrik:paused to resume. stage:<X>:complete is NOT added. checkCIGate returns terminated=true for this branch (ADR-1223), so handleMergeAndCIGates claims the item — Phase 2 does not run for it in the same poll pass.
Same column, Awaiting CI Poll tick (catch-up) CI checks still pending, or a failed check-run coexists with a pending check-run on the same head (#958 — pending always wins via github.ClassifyCheckRuns) Same (blocked) (none — fabrik:awaiting-ci already present)   checkCIGate logs pending checks; re-evaluates next poll; no CI-fix dispatch, no CIFixCycles increment. Per ADR-1410, this state’s own liveness-stall dwell (row below) is anchored on LinkedPRState.LastCIProgressAt, stamped every time this row’s own re-observation actually changes check-run content.
Same column, Awaiting CI Poll tick (catch-up) CI check(s) failed and nothing is pending on the current head Same column, Awaiting CI (failure confirmed) fabrik:awaiting-ci (idempotent)   CI failure detected; dispatch CI-fix reinvoke or pause on cycle limit (unless the current head already has a recorded CI-fix no-op — see §6.5.2). Per ADR-1410 (R3), this is a verdict and fires unconditionally, regardless of how long fabrik:awaiting-ci has been applied — never a timeout.
Same column, Awaiting CI Poll tick (catch-up) All CI checks green (or no CI configured — R5: mergeableState ∈ {"", "unknown"} + no check runs + hadChecks == false + dwell elapsed or zero LastHeadSHAUpdate) Same column, Complete → advance stage:<X>:complete fabrik:awaiting-ci Gate cleared; checkCIGate adds stage:<X>:complete and removes fabrik:awaiting-ci; falls through to advance (or merge for Validate+yolo)
Same column, Awaiting CI Poll tick (catch-up) mergeableState ∈ {"", "unknown"} + no check runs + hadChecks == false + LastHeadSHAUpdate non-zero + time.Since(LastHeadSHAUpdate) < PostPushDwell Same (blocked, no label)     Post-push dwell guard: GitHub has not yet computed mergeability or started CI for the newly-pushed SHA. Returns (true, false, false); re-evaluates on next poll. Zero LastHeadSHAUpdate (cold start / post-restart) falls through to R5 immediately (EC-1 preserved).
Same column, Awaiting CI Poll tick (catch-up) mergeable_state == "clean" (v0.0.52 shortcut, narrowed to clean-only by ADR-1441/#1441) Same column, Complete → advance stage:<X>:complete fabrik:awaiting-ci Gate cleared via mergeable_state shortcut before the per-check classification runs — GitHub has confirmed every check, required or not, passed. addCompleteLabelAndRemoveCI runs; falls through to advance. mergeable_state == "unstable" no longer takes this row (ADR-1441): it falls through to the check-run/required-context classification rows above (740–744) instead — a confirmed failure on a non-required check now blocks via row 743, not clears via this row.
Same column, Awaiting CI Poll tick (catch-up) mergeable_state == "blocked" + no check runs + hadChecks == false + fabrik:awaiting-ci applied < CIWaitTimeout ago Same (blocked, no label)     R3 dwell guard: first-push window; checks may not have registered yet. Re-evaluated on next poll.
Same column, Awaiting CI Poll tick (catch-up) mergeable_state == "blocked" + no check runs + hadChecks == false + fabrik:awaiting-ci applied ≥ CIWaitTimeout ago Same column, Awaiting Input fabrik:paused, fabrik:awaiting-input fabrik:awaiting-ci R3: pauseForRequiredNeverRunningCheck() posts distinct comment explaining the PR is blocked but no check runs were ever observed — required check likely converted to workflow_dispatch but still required by branch protection. Human must run the check manually or remove from branch protection. classifyCIFromMergeableState returns terminated=true for this branch (ADR-1223), so handleMergeAndCIGates claims the item — Phase 2 does not run for it in the same poll pass.
Same column, Awaiting CI Poll tick (catch-up) mergeableState ∉ {"", "unknown"} (e.g. behind, dirty, draft, has_hooks) + no check runs + hadChecks == false + fabrik:awaiting-ci applied < CIWaitTimeout ago Same (blocked, no label)     New guard (dwell not elapsed): mergeable_state actively blocking but no check_runs visible; returns (true, false, false); re-evaluates on next poll.
Same column, Awaiting CI Poll tick (catch-up) mergeableState ∉ {"", "unknown"} (e.g. behind, dirty, draft, has_hooks) + no check runs + hadChecks == false + fabrik:awaiting-ci applied ≥ CIWaitTimeout ago Same column, Awaiting Input fabrik:paused, fabrik:awaiting-input fabrik:awaiting-ci New guard (timeout elapsed): removes fabrik:awaiting-ci, returns (false, false, true); caller calls pauseForCITimeout() which posts explanatory comment and adds fabrik:paused + fabrik:awaiting-input.
Same column, Awaiting CI Poll tick (catch-up) Checks still pending, and LinkedPRState.LastCIProgressAt is non-zero (progress observed at least once this process’s lifetime) with time.Since(LastCIProgressAt) ≥ CIWaitTimeout (ADR-1410) Same column, Awaiting Input fabrik:paused, fabrik:awaiting-input fabrik:awaiting-ci classifyCIFromCheckRuns returns (false, false, true); caller calls pauseForCITimeout(), which posts explanatory comment. Liveness-stall dwell, not elapsed time — a LastCIProgressAt of zero (cold start / no progress observed yet) never escalates, only re-observes; see §6.14.5.
Same column, Awaiting CI (any classification) Poll tick (settleAwaitingCIScan’s unconditional backstop) fabrik:awaiting-ci applied ≥ CIBackstopTimeout ago (default 4h) and the item not resumed from a pause within that window, not on its first evaluation since daemon start, and with no worker in flight (#2059, §6.14.6), independent of what any classifier would otherwise decide (ADR-1410, R5) Same column, Awaiting Input fabrik:paused, fabrik:awaiting-input fabrik:awaiting-ci pauseForCITimeout() posts explanatory comment; bounds per-poll cost regardless of CI duration — see §6.14.5. Unlike the row above, elapsed-time-based via FetchLabelAppliedAt, and fires even when a silent claim earlier in the handler chain would otherwise make checkCIGate unreachable (§6.14.1).
Validate column, any gate label (or none), paused or not + not fabrik:auto-merge-enabled Poll tick Linked PR is merged (pr.Merged == true) Done stage:<X>:complete (for each gate-checked stage missing the label) gate labels (fabrik:awaiting-ci, fabrik:awaiting-review, fabrik:rebase-needed), fabrik:paused, fabrik:awaiting-input R4 (single owner — ADR-056 D2): runValidatePRTerminalAdvance detects merged PR via direct e.client.FetchLinkedPR (not e.readClient). Iterates all pipeline stages in ascending Order from the highest already-complete stage onward, adding stage:<N>:complete for every WaitForCI or WaitForReviews gate-checked stage whose label is absent. Clears all gate labels + removes pause labels + advances. Gate-label agnostic: runs regardless of which gate label (fabrik:awaiting-ci, fabrik:awaiting-review, fabrik:rebase-needed, or any future label) is present. See ADR-057.

R4b — Convergence-paused recovery: Items paused by pauseForConvergenceFailed (convergence budget exhausted) carry fabrik:paused + stage:Validate:complete but hold no gate label (fabrik:awaiting-ci or fabrik:awaiting-review). The R4 row above admits these items via its “any gate label (or none), paused or not” guard. When their linked PR merges externally after the convergence pause, runValidatePRTerminalAdvance detects the merge, removes fabrik:paused + fabrik:awaiting-input, and advances to Done — no manual unpause required. This is distinct from the active convergence monitor path (checkAutoMergeConvergence), which is excluded from R4 by the fabrik:auto-merge-enabled guard: pauseForConvergenceFailed removes fabrik:auto-merge-enabled before pausing, so R4 correctly picks up the item on the next poll after the external merge.

Merge-conflict gate (wait_for_ci: true only; runs before the CI gate):

Current State Event Guard Resulting State Labels Added Labels Removed Side Effects
Same column, Awaiting CI Poll tick (catch-up) mergeable == false on linked PR Same column, Rebase Needed fabrik:rebase-needed   Dispatch rebase reinvoke or pause on cycle limit
Same column, Rebase Needed (Awaiting CI + rebase-needed) Poll tick (catch-up) mergeable == true on linked PR (Claude’s rebase push landed) Same column, Awaiting CI → (CI gate evaluates next)   fabrik:rebase-needed Gate cleared; catch-up falls through to the CI gate on the same poll
Same column, Awaiting CI Poll tick (catch-up) mergeable == null (GitHub still computing) Same (blocked, no label)     Re-evaluated on next poll; no label churn for transient unknown state
Same column, Rebase Needed Poll tick (catch-up) snap.RebaseCycles(stageName) ≥ MaxRebaseCycles Same column, Awaiting Input fabrik:paused, fabrik:awaiting-input   pauseForRebaseCycleLimit() posts explanatory comment; fabrik:rebase-needed is left in place so the human can see why Fabrik stopped

Cooldown Retry and Failed Stage Escalation

Current State Event Guard Resulting State Labels Added Labels Removed Side Effects
Column <X>, Locked + In Progress max_wall_time exceeded SIGTERM→10s→SIGKILL; FABRIK_STAGE_COMPLETE found in buffered assistant stream Same column, Complete stage:<X>:complete fabrik:locked:<user>, stage:<X>:in_progress extractTextFromAssistantTurns() recovers marker; same completion flow as live FABRIK_STAGE_COMPLETE
Column <X>, Locked + In Progress max_wall_time exceeded SIGTERM→10s→SIGKILL; no FABRIK_STAGE_COMPLETE in buffered stream Same column, Cooldown     wasTimedOut=true; routes to cooldown/retry (not a hard error); lock NOT released
Column <X>, Locked + In Progress Inactivity timeout (15m) No streamed output for 15 consecutive minutes; FABRIK_STAGE_COMPLETE found in buffered stream Same column, Complete stage:<X>:complete fabrik:locked:<user>, stage:<X>:in_progress Same completion flow
Column <X>, Locked + In Progress Inactivity timeout (15m) No streamed output for 15 consecutive minutes; no FABRIK_STAGE_COMPLETE in buffered stream Same column, Cooldown     wasTimedOut=true; routes to cooldown/retry; lock NOT released
Column <X>, Locked + In Progress No marker in output claudeRan is true (includes both error-free runs and runs that errored mid-execution; excludes only start failures like binary-not-found) Same column, Cooldown     CooldownAt("periodic-re-eval") recorded (via CooldownRecorded); cooldown = PollSeconds * 10; lock NOT released (stays locked through retries)
Column <X>, Locked + In Progress FABRIK_STAGE_COMPLETE present, but stripped output is degenerate (bare @file/absolute-path reference, §2.6) isDegenerateOutput(postOutput) is true Same column, Cooldown     Output not posted; completed forced to false before the marker-dispatch branches run; treated identically to “no marker in output” for retry purposes; first detection posts an immediate explanatory comment
Column <X>, Locked + In Progress FABRIK_STAGE_COMPLETE present, but no artifact harvestable anywhere (§2.6, #1782) artifactMissingOnComplete(text) is true inside interpretClaudeResult() (upstream of this row, before finalizeStageOutcome() runs) Same column, Cooldown     completed already false on arrival; treated identically to “no marker in output” for retry purposes; first detection posts an immediate “no artifact harvested” comment; excluded when FABRIK_NO_WORK_NEEDED co-occurs
Same column, Cooldown Poll tick Cooldown expired Same column, Locked + In Progress (retry)   stage:<X>:failed (if present from prior escalation) Claude re-invoked with resume=true
Same column, Cooldown Retry count ≥ MaxRetries claudeRan && !turnLimited && MaxRetries > 0 (genuine error, or a clean run that never emitted FABRIK_STAGE_COMPLETE — see §7.12 for why the latter counts here rather than as a slice) Same column, Paused + Failed fabrik:paused, stage:<X>:failed fabrik:locked:<user>, stage:<X>:in_progress escalateFailedStage() posts comment; lock released; Attempts incremented via StageRetryIncremented (#1199 — turnLimited outcomes never reach this row; see the row below)
Same column, Cooldown Slice count ≥ MaxSliceRetries claudeRan && turnLimited && MaxSliceRetries > 0 Same column, Paused + Awaiting Input fabrik:paused, fabrik:awaiting-input fabrik:locked:<user>, stage:<X>:in_progress pauseForSliceLimit() posts a slice-budget comment (never “failed”); SliceRetries incremented via SliceRetryIncremented; no stage:<X>:failed — the stage has not failed (#1199, §7.12)
Same column, Paused + Failed Human removes fabrik:paused stage:<X>:failed present OR snap.PausedByEngine(stageName) Same column, Idle   stage:<X>:failed clearFailedStage() applies StageRetryCleared (also zeroes SliceRetries — the two counters share one reset point), EngineUnpaused, StageLastAttemptCleared, EngineCyclesCleared

In-flight items and cooldown (#544): CooldownAt("periodic-re-eval") is stamped for in-flight items (those where a prior worker is still running at end-of-poll). Stamping is intentional: without it, once a prior expired cooldown ages out, the item would be re-admitted to the deep-fetch path on every poll cycle until the worker exits — causing repeated unnecessary deep-fetch evaluation work (and the fallback GraphQL fetch when the cache is invalidated or disabled) for items that can’t be dispatched anyway (snap.Worker() != nil blocks them). The prompt re-dispatch after the worker finishes is guaranteed by WorkerExited → WorkerLifecycleChanged, which is in wakeChFlags and wakes the poll loop immediately. Note: WorkerLifecycleChanged is excluded from cycleSetFlags (Fix B, issue #576), so it does not bypass the cooldown gate via mayNeedWork — but the cooldown is already expired (or was already bypassed by an entry in cycleSet from earlier in the worker’s lifecycle). This note concerns items blocked by the snap.Worker() != nil guard; an item deferred because every worker slot was busy is handled differently (no stamp, re-queued — §9.9.1). See §2.14, §9.2, and §9.9.

Turn Limit Extension

The turn budget is a preemption mechanism, not a failure threshold. It exists for two reasons, neither of which is “the work went wrong”:

  1. Runaway guard — bound an unbounded Claude loop.
  2. Time-slicer — chop a large job so a single issue cannot starve every other issue of workers.

When the budget is exhausted, the engine commits and pushes partial work, and the next dispatch resumes the same Claude session (--resume) against the same worktree. Work continues from where it stopped; nothing is discarded and nothing restarts. A capped invocation is a slice boundary, and a job needing several slices is normal — cost typically rises across slices as each resumed session carries more accumulated context.

Treat “turn limit hit” throughout this section as this slice ended, never as this stage failed. The distinction is load-bearing: the CLI reports it structurally as subtype: "error_max_turns" / terminal_reason: "max_turns" (captured since #1178), separate from a genuine fault, and the TUI renders it as ↻ (turn limit) rather than ✗ (error).

Retry accounting matches this model (#1199). A turn-capped slice calls SliceRetryIncremented, not StageRetryIncremented — it is bounded by its own counter, SliceRetries/MaxSliceRetries (default 10), never by MaxRetries (default 3, unchanged). A job needing more slices than max_retries completes without pausing, as long as it stays within MaxSliceRetries. See §7.12 for the full detection/handling/recovery treatment, and #1191 for the operator-facing report this resolves.

When Claude exits a stage invocation due to max_turns (i.e., the per-invocation turn usage satisfies invUsage.TurnsUsed >= currentBudget and !completed && err == nil), the engine evaluates whether to extend before entering the cooldown/retry path.

Extension trigger condition: !completed && err == nil && stage.MaxTurns > 0 && invUsage.TurnsUsed >= currentBudget

Hard cap: 3× stage.MaxTurns total across all invocations. When totalMultiple >= 3, no further extension is attempted.

Per-stage progress signals:

Stage Progress Signal API Cost
Implement New git commit (HEAD SHA changed) OR (baseline was clean AND working tree is now dirty — uncommitted file edits by Claude) Zero — local git only
Review New git commit OR LinkedPRResolvedThreadCount increased One FetchItemDetails GraphQL call (only if no new commit)
Validate Total comment count on issue/PR increased One FetchItemDetails GraphQL call
All others (Research, Specify, Plan, custom) No signal — always fail on turn-limit None

The “baseline clean AND working tree dirty” guard for Implement prevents a pre-existing dirty worktree (e.g. from a prior interrupted session) from counting as progress. Only new uncommitted changes made during the invocation trigger extension.

Extension loop behavior (within a single processItem call — no poll-cycle gap):

  1. At invocation start, a progressBaseline is snapshotted: git HEAD SHA (Implement, Review), working-tree dirty state (Implement), comment count (Validate), and resolved thread count (Review).
  2. Claude is invoked with the current budget.
  3. If the turn limit is hit AND totalMultiple < 3: call detectProgress. If progress → totalMultiple++, re-invoke with --resume. If no progress or progress check fails → proceed to cooldown/retry as today.
  4. Output is accumulated across all invocations before posting as a single stage comment.
  5. WIP commit and push are deferred to after the loop.

fabrik:extend-turns label: When present at invocation start, the first invocation receives 2 × stage.MaxTurns as its budget (pre-granted extension, no progress check required for the first turn-limit hit). That same first invocation also gets max_wall_time scaled by the identical 2× factor (scaledWallTime in engine/claude.go, §7.7, ADR-1206), so the extra turn budget has proportionate wall-clock headroom instead of being killed on a deadline sized for the un-extended case. Subsequent extensions beyond 2× still require the progress check, reset to the un-multiplied stage.MaxTurns budget, and — since each such extension re-invokes with --resume and thus gets its own fresh runClaude call — run under the unscaled max_wall_time (their own correctly-sized window starting at that call’s own spawn time, not a further-inflated one). The label persists across all intermediate stages — it is not removed on stage completion. It is removed only in the cleanup (Done) stage branch of processItem, after the stage:Done:complete label is added. ErrNotFound on removal is treated as success (user removed it manually). The label is a no-op when stage.MaxTurns == 0 (which also means no wall-time scaling occurs — scaledWallTime cannot scale against a zero baseline).

Log tag: [#N extend-turns] — emitted on every detectProgress call (pass or fail), reporting the evaluated signals and has_progress=true/false. When extension is granted, an additional line logs the new budget multiple and cumulative turns used.

Current State Event Guard Resulting State Labels Added Labels Removed Side Effects
Column <X>, Locked + In Progress Turn limit hit totalMultiple < 3; progress detected Same column, Locked + In Progress (extension)     totalMultiple++; resume=true; output accumulated; no WIP commit or push between extensions
Column <X>, Locked + In Progress Turn limit hit totalMultiple >= 3 (hard cap) Same column, Cooldown     Hard cap reached; slice ends (a preemption, not a failure — the session resumes on the next dispatch). Cooldown routes this to the slice counter (SliceRetries/MaxSliceRetries), never MaxRetries — see the “Slice count ≥ MaxSliceRetries” row above and §7.12 (#1199). CooldownAt("periodic-re-eval") recorded; WIP commit + push (skipped for read_only stages; a persist_spec Specify stage has already committed its spec, §7.16)
Column <X>, Locked + In Progress Turn limit hit No progress detected or progress check failed Same column, Cooldown     No extension; slice ends (a preemption, not a failure). Same as the row above: Cooldown routes this to the slice counter, never MaxRetries (#1199, §7.12). CooldownAt("periodic-re-eval") recorded; WIP commit + push (skipped for read_only stages; a persist_spec Specify stage has already committed its spec, §7.16)
Column <X>, Locked + In Progress FABRIK_STAGE_COMPLETE (any extension) completed = true Same column, Complete stage:<X>:complete fabrik:locked:<user>, stage:<X>:in_progress Normal completion flow; extend-turns label persists to next stage

Cleanup Stage

Current State Event Guard Resulting State Labels Added Labels Removed Side Effects
Done, Pending Cleanup Poll tick Worktree exists, not paused, not already complete Done, Complete stage:Done:complete fabrik:extend-turns (if present) Worktree removed; no lock/Claude/comment processing
Done, Complete Poll tick Already complete Done, Complete (no-op)     Skipped by both itemMayNeedWork and processItem

Review Reinvoke

Current State Event Guard Resulting State Labels Added Labels Removed Side Effects
Column <X>, Awaiting Review + Complete Review gate clears + unresolved thread comments snap.Worker() == nil, cycle count < MaxReviewCycles Same column (comment processing via async goroutine) fabrik:editing (during processing)   dispatchReviewReinvoke() spawns goroutine; ReviewCycleIncremented applied; WorkerEntered applied; semaphore acquired
Column <X>, Awaiting Review + Complete Review gate clears + unresolved thread comments Cycle count ≥ MaxReviewCycles Same column, Awaiting Input fabrik:paused, fabrik:awaiting-input   pauseForReviewCycleLimit() posts comment
Column <X>, Awaiting Review + Complete Review gate clears + unresolved thread comments snap.Worker() != nil Same (skipped)     Previous reinvoke goroutine still running; skipped entirely (no cycle-limit check)

CI Fix Reinvoke

Current State Event Guard Resulting State Labels Added Labels Removed Side Effects
Column <X>, Awaiting CI Poll tick (catch-up) CI failed; snap.Worker() == nil; snap.LastCIFixNoOpSHA() != settle.PR.HeadSHA; snap.CIFixCycles(stageName) < MaxCiFixCycles Same column (CI-fix goroutine running) fabrik:editing (during processing)   dispatchCIFixReinvoke() spawns goroutine; CIFixCycleIncremented applied; WorkerEntered applied; semaphore acquired; synthetic CI-fix comment passed to processComments()
Column <X>, Awaiting CI Poll tick (catch-up) CI failed; snap.LastCIFixNoOpSHA() == settle.PR.HeadSHA (#958 leg 2) Same (skipped)     A prior CI-fix reinvoke for this exact head SHA already observed no new commit pushed; dispatching again would repeat the same no-op. No dispatch, no CIFixCycles increment. CIBackstopTimeout remains the backstop if CI never resolves on this SHA (ADR-1410 — CI is a confirmed failure here, so the liveness-stall dwell does not apply; only the unconditional per-poll-cost backstop bounds this); the guard is implicitly cleared once the head SHA advances.
Column <X>, Awaiting CI Poll tick (catch-up) CI failed; snap.CIFixCycles(stageName) ≥ MaxCiFixCycles Same column, Awaiting Input fabrik:paused, fabrik:awaiting-input   pauseForCIFixCycleLimit() posts explanatory comment
Column <X>, Awaiting CI Poll tick (catch-up) CI failed; snap.Worker() != nil Same (skipped)     Previous CI-fix goroutine still running; skipped entirely

Rebase Reinvoke

Current State Event Guard Resulting State Labels Added Labels Removed Side Effects
Column <X>, Rebase Needed + Complete Poll tick (catch-up) mergeable == false; snap.Worker() == nil; snap.RebaseCycles(stageName) < MaxRebaseCycles Same column (rebase goroutine running) fabrik:editing (during processing)   dispatchRebaseReinvoke() spawns goroutine; RebaseCycleIncremented applied; WorkerEntered applied; semaphore acquired; synthetic rebase-required comment passed to processComments()
Column <X>, Rebase Needed + Complete Poll tick (catch-up) mergeable == false; snap.RebaseCycles(stageName) ≥ MaxRebaseCycles Same column, Awaiting Input fabrik:paused, fabrik:awaiting-input   pauseForRebaseCycleLimit() posts explanatory comment (usually signals a semantic conflict needing human judgment)
Column <X>, Rebase Needed + Complete Poll tick (catch-up) mergeable == false; snap.Worker() != nil Same (skipped)     Previous rebase goroutine still running; skipped entirely

3.3 Comment Processing During Non-Pause Gate States

When a new user comment arrives while a non-pause gate label (fabrik:awaiting-ci, fabrik:awaiting-review, fabrik:rebase-needed, or fabrik:auto-merge-enabled) is active, the engine dispatches comment processing before evaluating the gate. This behavior is intentional: item.go checks for new comments (len(newComments) > 0 → processComments()) before reaching any gate check, and itemNeedsWork guard 7 admits any item with new comments regardless of gate labels.

Current State Event Race Resolution Resulting State Notes
Column <X>, Awaiting CI New user comment arrives snap.Worker() != nil guard in catch-up loop reinvoke dispatch: if a comment worker is in-flight when the catch-up loop runs, the reinvoke (CI-fix, rebase, etc.) is skipped for that poll cycle Comment processing runs; catch-up loop reinvoke skipped until worker exits Intended: user may need to override CI behavior mid-gate. The next poll after the comment worker exits will re-evaluate the CI gate.
Column <X>, Awaiting Review New user comment arrives Same worker guard: review reinvoke skipped while comment worker is in-flight Comment processing runs; review gate re-evaluated next poll Intended: user may need to provide direction to Claude while awaiting review.
Column <X>, Rebase Needed New user comment arrives Same worker guard: rebase reinvoke skipped while comment worker is in-flight Comment processing runs; rebase gate re-evaluated next poll Intended: user may provide rebase guidance while gate is active.
Column <X>, Auto-Merge Enabled New user comment arrives Same worker guard Comment processing runs; convergence monitor re-evaluated next poll Intended: user may need to intervene while convergence is in progress.

Known limitation: If both the catch-up loop and a comment arrive at the same poll cycle before any worker is recorded, both dispatches can fire in the same cycle. The advancedItems map prevents double-advancement within a single poll cycle; the fabrik:editing label added by the comment goroutine at entry prevents a second goroutine from launching immediately. This is not structurally exclusive — it is a best-effort guard.

3.4 Operator-Level and System-Level Transitions

These transitions are driven by operator actions or background engine scans rather than stage-lifecycle events. They do not fit the “stage-running → stage-complete” grammar of §3.1/§3.2.

Event Source Event Current State Effect Labels Added Labels Removed Notes
User (manual) Assignee change Any column Re-evaluated on next poll; AssigneesChanged fires wakeChObserver (none) (none) No label mutation. Assignee is read by logf for display only; it does not gate dispatch. See §2.13.
User (manual) fabrik:revalidate applied Validate-column, any state Removes gate/completion labels; re-dispatches Validate on next poll (none) stage:Validate:complete, stage:Validate:failed, fabrik:paused, fabrik:awaiting-input, fabrik:awaiting-ci, fabrik:auto-merge-enabled, fabrik:revalidate (trigger removed last) Back-edge into Validate. Resets PausedByEngine, StageRetryCount, LastAttemptAt, EngineCycles for Validate. See §2.15.
User (manual) fabrik:revalidate applied Non-Validate column Only the trigger label is removed; no other action (none) fabrik:revalidate Warning logged. See §2.15.
Engine scan SHA-invalidation detected Validate, Complete Removes Validate completion; re-dispatches Validate on next poll (none) stage:Validate:complete, fabrik:awaiting-ci, fabrik:awaiting-review, fabrik:rebase-needed, fabrik:auto-merge-enabled Back-edge into Validate. Triggered when snap.ValidateCompletedSHA() ≠ LinkedPR.HeadSHA — Claude pushed a new commit after Validate ran. Not taken for a landed/terminal item, nor when the new head is the engine’s own pure catch-up merge (#2158). See §2.16.
Engine (convergence monitor) PR merged by GitHub auto-merge Validate, Complete + fabrik:auto-merge-enabled Removes convergence label; advances to Done (none) fabrik:auto-merge-enabled checkAutoMergeConvergence detects pr.Merged == true; calls advanceToNextStage. The stage:Validate:complete label remains. See §5.5.
Engine (Implement dispatch) FABRIK_PR_CREATE_BEGIN/END marker Implement, Locked + In Progress Draft PR created on GitHub; no stage advance (none) (none) Engine reads marker from Claude output, calls gh pr create --draft; posts PR URL as a comment on the issue. Stage advance requires a separate FABRIK_STAGE_COMPLETE. See §5.6.
Engine / Claude FABRIK_STAGE_COMPLETE + FABRIK_NO_WORK_NEEDED Any stage, Locked + In Progress All subsequent non-cleanup stages marked complete; issue moved to Done once the settle scan’s Done-move/close succeeds (retried every poll if it fails; escalated to fabrik:paused after MaxRetries) fabrik:awaiting-done, stage:<X>:complete, stage:<Y>:complete … (all subsequent non-cleanup stages) fabrik:locked:<user>, stage:<X>:in_progress, fabrik:awaiting-done (once Done is reached or on escalation) No PR created. Validate re-entry back-edge from fabrik:revalidate and SHA-invalidation are visible in §10.4. See §6.8.

4. Comment Processing Lifecycle

When new comments are detected on an issue (or synthetic review comments on a PR), the engine processes them through processComments(). This is an 11-step flow.

4.1 Comment Detection

findNewComments() filters item.Comments to find unprocessed comments using four independent dedup signals:

  1. In-memory CommentProcessed in itemstate.Store (session-scoped) — skip comments whose ID is recorded via snap.CommentProcessed(c.ID) (written by CommentProcessed mutation). Fast but lost on restart.
  2. 🏭 **Fabrik body prefix (engine-authored output convention) — skip comments whose body starts with this header. Durable but requires the header to be present.
  3. 🚀 ROCKET reaction (durable, cross-restart) — skip comments that already have a rocket reaction. Applied to user comments by processComments step 10 after processing; also applied by the engine to every comment it posts immediately after AddComment succeeds. Review bodies (no REST reactions endpoint, no DatabaseID) get the same 👀/🚀 through GraphQL addReaction on the review’s node ID (#1953 R8); that reaction is human-visible only and is not read back for dedup.
  4. Bot service-notice pattern (content-based, isBotServiceNotice) — skip comments whose author matches github.IsBotLogin and whose body matches a known service-notice pattern from botServiceNoticePatterns (engine/comments.go), a flat, vendor-grouped list of literal case-insensitive substrings. Covers two vendors as of #1122, plus a third CodeRabbit marker added by #1141: Gemini’s quota/rate-limit/sunset/unsupported-file-type notices (e.g. “daily quota limit”, “you have reached your rate limit”, “the consumer version of gemini code assist on github has been sunset”), CodeRabbit’s rate-limit notice, matched primarily via its HTML comment marker (rate limited by coderabbit.ai — a structural, non-user-facing signal that won’t drift when CodeRabbit rewrites its banner copy) with prose fallbacks (“## review limit reached”, “you’ve reached your pr review limit, so we couldn’t start this review”) in case a future CodeRabbit variant omits the marker — the fallbacks are deliberately full-phrase rather than bare fragments, since a bare "review limit reached" substring would itself match a genuine bot review comment that quotes these very patterns (e.g. a diff review critiquing this pattern list) — and CodeRabbit’s auto-generated acknowledgement-reply marker (auto-generated reply by coderabbit, matching the HTML comment <!-- This is an auto-generated reply by CodeRabbit -->), distinct from the rate-limit marker above: it covers a content-free “acknowledged, no action taken” reply rather than a quota-exhaustion notice, and — like the rate-limit marker — is matched structurally so it holds regardless of which acknowledgement phrasing CodeRabbit uses. Unlike signals 1–3, this is a content classification rather than a watermark: it must hold on every poll’s raw fetch, so a matching comment never reaches processComments in the first place — no 👀 reaction, no worktree/Claude invocation, no reply. Because this signal excludes the comment before signal 1’s watermark would ever be recorded by the normal processComments flow, settleBotServiceNotices — an unconditional per-poll scan over the raw board snapshot, independent of dispatch state — separately applies the 🚀 reaction and CommentProcessed watermark so the comment doesn’t need to re-qualify via this pattern check on every future poll. Classification is per-comment, not per-author or per-length: a bot can post both a rate-limit notice and a genuine review on the same PR, and each comment is classified independently on its own body. Introduced to stop a runaway reply loop where a bot’s quota-exhaustion auto-reply triggered a Fabrik “not actionable” reply, which in turn re-triggered the bot (#1083/#1088); extended in #1122 once Gemini was usage limited and CodeRabbit became the active review bot with its own unmatched rate-limit phrasing; extended again in #1141 to cover CodeRabbit’s acknowledgement-reply loop (see the outbound mention-neutralization invariant below — this marker was one half of a two-part fix, the other half being that the acknowledgement was also being re-triggered by Fabrik’s own reply naming the bot). Enforcement is now split across two points (#1221): this raw-fetch exclusion inside findNewComments still governs whether a worker is dispatched at all for a bot-notice-only item.Comments backlog (a notice excluded here never causes itemNeedsWork to see “new work”), while a second chokepoint inside processComments itself (§6.2) applies the same isBotServiceNotice classifier to the fully-assembled working slice just before Step 1 — covering comments merged in from item.LinkedPRReviewThreadComments and comments supplied by any of the three reinvoke dispatchers (dispatchReviewReinvoke, dispatchCIFixReinvoke, dispatchRebaseReinvoke), none of which route through this findNewComments exclusion.

Any single signal catching the comment is sufficient to skip it. The four signals are orthogonal — any combination can fail independently without triggering the self-review loop.

Dedup coverage by comment type:

Invariant: every engine-emitted AddComment call must start with 🏭 **Fabrik — <context>**. This is an engine-wide convention enforced by TestAddCommentCompliance in engine/compliance_test.go, not just a detection heuristic.

Invariant (#1141): generated stage/comment output never renders a live GitHub mention of a bot login. neutralizeBotMentions (engine/mentions.go) rewrites any @<login> where github.IsBotLogin(login) is true into an inline code span — @coderabbitai becomes `@coderabbitai` — so GitHub does not notify the bot, regardless of what the surrounding prose claims about whether a reply was posted. Detection is code-span-aware (text already inside backticks is skipped), which makes the rewrite idempotent. Applied inside the four points Claude-derived freeform text is embedded into posted or persisted content: formatOutputComment, formatPRSummaryComment, formatReviewFeedbackComment (all engine/pr.go), updatePRVerification’s PR-body ## Verification rewrite (engine/pr.go — a UpdateIssueBody call outside the AddComment compliance funnel above, but equally capable of triggering a live mention), and buildAwaitingInputComment (engine/item.go). github.IsBotLogin gained a bare-literal "coderabbitai" match for this: CodeRabbit’s GitHub API login carries a [bot] suffix (already matched), but its @-mention surface is the bare form, which the suffix rule alone does not catch. Fixes the #933 incident, where Fabrik’s own “no action taken” comment-processing summary re-mentioned @coderabbitai, notifying the bot into another acknowledgement reply that Fabrik then processed again — see ADR-073.

4.2 The 11-Step Flow

Step Action Code Side Effects
0 Emit JobStartedEvent + defer JobCompletedEvent{Skipped:true} e.emitStructural(tui.JobStartedEvent{...}) TUI active-pane entry created; deferred JobCompletedEvent{Skipped:true} fires unconditionally on function return (cleanup guard). HistoryPaneComponent filters it out — only InvocationObserver’s Skipped:false event reaches history. This is the TUI work-boundary — fires at processComments entry before any external I/O.
1 React with 👀 to all new comments AddCommentReaction("eyes") / AddPRReviewCommentReaction("eyes") Signals acknowledgment to the user
2 Add fabrik:editing label AddLabelToIssue("fabrik:editing") Pre-dispatch gate in itemNeedsWork (prevents goroutine launch); defense-in-depth check retained in processItem for the race window. Symmetric with fabrik:locked:<other-user>. JobStartedEvent already fired at step 0 — the active session registers as in-progress in the TUI before this label is added.
2b Clear stale stage:<Stage>:complete if present (#1802) beginStageRework() No-op unless the re-entered stage’s own stage:<Stage>:complete is already on the item. When present: adds fabrik:reworking:<Stage>, then removes stage:<Stage>:complete — mark-first-then-clear, strictly inside the step-2 fabrik:editing bracket so no dispatch gate ever observes the transient clear (R4). See ADR-1802.
3 Ensure worktree exists EnsureWorktree() Creates or updates worktree; writes context files
4 Invoke Claude with comment review prompt InvokeForComments() Uses comment_prompt / comment_skill and comment_max_turns
5 Check for FABRIK_STAGE_COMPLETE in output checkCompletion() Determines if comment processing resolved the stage
6 Extract and apply FABRIK_ISSUE_UPDATE if present extractUpdatedBody() Applied unconditionally when markers are present; stripped from output regardless
7 Strip all Fabrik markers from output stripLine() calls Removes FABRIK_STAGE_COMPLETE, FABRIK_BLOCKED_ON_INPUT, FABRIK_NO_WORK_NEEDED, FABRIK_SUMMARY_BEGIN/END
8 Post or update stage comment AddComment() / UpdateComment() For post_to_pr stages: always posts new comment on issue (labeled as “comment review”); for other stages: rewrites existing stage comment or creates new one. Review-reinvoke branch (Step 8b): when the input batch is all-ReviewThreadID comments (isReviewReinvoke == true) and output != "", also posts a Fabrik-marked "<StageName> (review feedback addressed)" comment on the linked PR (via FindPRForIssue); includes per-thread footer with path:line for each addressed thread; skipped if no linked PR is found (logs warning). The issue comment is always posted first; the PR comment is additive. Suppressed on FABRIK_NO_WORK_NEEDED (#1088): when the invocation output contains FABRIK_NO_WORK_NEEDED (checked before markers are stripped), none of the three reply paths above post — no stage comment, no post_to_pr comment, no review-reinvoke PR summary. Posting any reply is what re-triggers a subscribed bot into a loop, so silence replaces the old “not actionable” message. Steps 6 (issue-body update), 9 (editing-label removal), 10 (🚀 reactions), and 11 (stage-complete handling) are unaffected — only the reply post itself is gated.
8b Restore or re-derive stage:<Stage>:complete (#1802) endStageRework() No-op unless step 2b actually cleared the label. On a non-completing exit (this step, or any of the earlier setup-failure/invocation-failure early returns): restores stage:<Stage>:complete directly, then removes fabrik:reworking:<Stage> immediately — nothing later in the flow will restore it. On a completing exit: this step is deliberately skipped here — stage:<Stage>:complete is left to step 11’s handleStageComplete() to re-derive from scratch (including its wait_for_ci deferral), and fabrik:reworking:<Stage> itself is not removed until after step 11 completes (not here at step 8b), so a crash anywhere between this point and step 11’s completion-label write still leaves the marker as a durable signal that a restore is owed. See ADR-1802.
9 Remove fabrik:editing label RemoveLabelFromIssue("fabrik:editing") Releases the editing mutex
10 React with 🚀 to all processed comments + resolve review threads AddCommentReaction("rocket") / AddPRReviewCommentReaction("rocket") / AddReviewReaction("rocket") (review bodies, GraphQL, #1953 R8) + ResolveReviewThread() Marks comments as processed (durable); resolves addressed review threads
11 If FABRIK_STAGE_COMPLETE was detected: handle completion handleStageComplete() Same completion flow as a normal stage invocation (advance, PR ops, etc.). On a completing exit that began a rework (step 2b), fabrik:reworking:<Stage> is removed immediately after this step returns — not before — per step 8b’s note (#1802, ADR-1802).

4.3 Turn Limit Extension

When Claude exits a comment processing invocation due to comment_max_turns (i.e., invUsage.TurnsUsed >= currentBudget and !invCompleted && err == nil), the engine evaluates whether to extend before returning partial output.

Extension trigger condition: !invCompleted && err == nil && currentBudget > 0 && invUsage.TurnsUsed >= currentBudget

Note: currentBudget > 0 is only satisfied when fabrik:extend-turns is present (label absent → currentBudget = 0 → no extension possible). This differs from stage invocations (§3), where the progress-based extension loop fires whenever stage.MaxTurns > 0 is hit — the label only pre-grants the 2× first budget there. Comment processing is intentionally label-gated: extending comment-review turns is a new opt-in capability, and changing no-label behavior would silently extend comment budgets for all existing issues.

Hard cap: 3× commentMaxTurns(stage) total across all invocations. When totalMultiple >= 3, no further extension is attempted.

commentMaxTurns(stage): Returns CommentMaxTurns if set, else MaxTurns, else 50. This value is always > 0 (unlike stage.MaxTurns which can be 0 for unlimited).

Per-stage progress signals: Same signals as the stage invocation path — see §3 Turn Limit Extension table. For no-signal stages (Research, Specify, Plan, Done), detectProgress returns false immediately, so fabrik:extend-turns grants the 2× pre-budget for the first invocation but no further extension.

Extension loop behavior (within a single processComments call):

  1. hadExtendTurnsLabel snapshotted before the loop. If present: currentBudget = 2 × commentMaxTurns(stage), totalMultiple = 2. If absent: currentBudget = 0, totalMultiple = 1.
  2. snapshotBaseline called before the loop (same function as stage path).
  3. InvokeForComments called with opts.MaxTurnsOverride = currentBudget. Session resume is handled internally by InvokeClaudeForComments — no loop-level session management needed.
  4. If limit hit AND totalMultiple < 3: call detectProgress. If progress → totalMultiple++, currentBudget = commentMaxTurns(stage), re-invoke. If no progress or error → return partial output.
  5. Output accumulated across all invocations before posting as a single comment.
  6. Usage totals (tokens, turns) and InvocationRecorded store event applied once after loop completes.

fabrik:extend-turns label: When present, the first comment processing invocation receives 2 × commentMaxTurns(stage) as its budget (pre-granted, no progress check required for the first turn-limit hit). That same first invocation also gets max_wall_time scaled by the identical 2× factor (scaledWallTime in engine/claude.go, §7.7, ADR-1206) — the same mechanism as the stage-invocation path (§3.2). Subsequent extensions reset to the un-multiplied commentMaxTurns(stage) budget and, via their own fresh InvokeClaudeForComments/runClaude call, run under the unscaled deadline.

Log tag: [#N extend-turns] — same tag as stage path. Emitted on each detectProgress call. When extension is granted, an additional line logs the new budget multiple and cumulative turns used. [#N stats] emitted after loop with final accumulated usage.

Current State Event Guard Resulting State Labels Added Labels Removed Side Effects
Comment Processing, In Progress Turn limit hit totalMultiple < 3; progress detected Comment Processing, In Progress (extension)     totalMultiple++; currentBudget = commentMaxTurns(stage); output accumulated; InvocationRecorded deferred
Comment Processing, In Progress Turn limit hit totalMultiple >= 3 (hard cap) Comment Processing, Complete (partial)     Hard cap reached; partial output posted; InvocationRecorded applied
Comment Processing, In Progress Turn limit hit No progress detected or progress check failed Comment Processing, Complete (partial)     No extension; partial output posted
Comment Processing, In Progress FABRIK_STAGE_COMPLETE (any extension) invCompleted = true Comment Processing, Complete     Normal comment completion flow; output posted

4.4 Comment Processing Entry Points

Comments can trigger processing through three paths in processItem():

  1. Awaiting-input unblock: isAwaitingInput(item) is true + new human comments created at/after the pause (resumeAuthorised()) → unblockAwaitingInput() → processComments()
  2. Paused unpause: fabrik:paused present + new human comments created at/after the pause (resumeAuthorised()) → remove fabrik:paused, clearFailedStage() → fall through → processComments()
  3. Normal comment processing: Item is not paused → findNewComments() finds comments (human or bot) → processComments()

Resume anchor (#1813, ADR-1813). resumeAuthorised() is the single predicate behind all four resume gates (itemNeedsWork’s awaiting-input and paused branches, processItem’s awaiting-input block and paused loop), so admission and resume cannot disagree. It authorises a resume only from an event — a human comment whose CreatedAt is at or after the latest labeled event for fabrik:paused (read live via FetchLabelAppliedAt, never the record-on-write cache, so a label removed in the UI cannot leave a stale early anchor; isAwaitingInput implies paused, so this one label covers both) — not from the state “an unprocessed human comment exists”. A comment that can never be processed therefore lifts a pause at most once: after the resume any re-pause writes a newer labeled event, and the comment predates it. Consequently the breaker counters (CommentBreakerReset, EngineCyclesCleared, clearFailedStage) are only zeroed by a genuine resume. Tie-break (R6): every indeterminate input resumes — anchor fetch error, no labeled event found, a zero CreatedAt, or a comment timestamp equal to the anchor (one-second resolution); only “every human comment strictly predates the pause” refuses. Trade-off: a human comment posted while an invocation runs, which then pauses (blockOnInput, a breaker trip), predates that pause and does not resume it — the human comments again.

In paths 1 and 2, resumeAuthorised() only gates whether to resume — once a human comment authorizes it, the raw (unfiltered) comment set from findNewComments() is what’s actually handed to processComments(), in the same processItem invocation. Any bot comments that accumulated while paused/awaiting-input are processed together with the resuming human comment, not deferred to a later poll (#1083) — except bot service-notice comments (§4.1 signal 4), which findNewComments() excludes from that raw set entirely regardless of pause state, since they are non-actionable either way.

All three paths ultimately source their candidate comments from findNewComments() (paths 1–2 via resumeAuthorised()’s wrapping, path 3 directly). But processItem()’s three paths are not the only route into processComments() — the three catch-up-loop reinvoke dispatchers (dispatchReviewReinvoke, dispatchCIFixReinvoke, dispatchRebaseReinvoke, §6.2) call it directly with comments built by buildReviewThreadComments() or a synthetic single-comment builder, neither of which goes through findNewComments(). A quota/rate-limit bot notice can therefore still reach processComments() as an entry in its working comments slice via one of these non-findNewComments() paths — the claim that it “never reaches processComments() via any path” was false, and as of #1221 that gap is closed differently: processComments() itself (§6.2) applies isBotServiceNotice as a single chokepoint to its fully-assembled working comments slice, immediately after the slice is assembled — covering the findNewComments()-sourced set (idempotently) as well as the LinkedPRReviewThreadComments merge and all three reinvoke dispatchers’ input — before any reaction, label, worktree, or invocation side effect. findNewComments() is the enforcement point for whether a worker is dispatched at all (§4.1 signal 4); processComments() is the enforcement point for whether a comment — however it arrived — is ever handed to Claude or otherwise acted upon.

All six of these entry points — the three processItem() paths above and the three reinvoke dispatchers — funnel through the same processComments() function body, so all six inherit the fabrik:reworking:<Stage>/stage:<Stage>:complete clear-and-restore behavior (§4.2 steps 2b/8b, ADR-1802) uniformly, with no per-site special-casing. There is exactly one place that decides whether a re-entered stage’s completion claim needs clearing (beginStageRework, called once, immediately after step 2’s fabrik:editing add succeeds) and exactly one place that restores or defers to the normal completion flow (endStageRework, called from every exit path of processComments()/finalizeComments()) — regardless of which of the six triggers brought the item here.

Post-invocation restore of the engine’s own rebase (#2150, ADR-2150). Step 3 of processComments() (EnsureWorktree) rebases the issue branch onto origin/<base> (unless the PR is queued or the repo is queue-enabled, ADR-058). When the pass then commits nothing, nothing is pushed and the local branch would be left a rebased copy of its remote — diverged, with nothing on either side the other lacks, and a head nobody has CI for (which also defeats the ci-status.md HEAD == head_sha skip rule). processComments() therefore captures the head before EnsureWorktree and, immediately after the extension loop returns and headChanged is computed — before either fabrik:editing removal and before finalizeComments’ possible markPRReady push — calls WorktreeManager.RestoreUnpushedRebase, which resets to the pre-rebase head only when the worktree is on the issue branch with HEAD still exactly the post-rebase head (no commit, reset or amend), the tree is clean with no merge/rebase/cherry-pick in progress, a fresh fetch of the issue branch succeeds and shows the rebased head is not already on the remote, and the rebased head carries no patch (patch-id, base history excluded) or merge commit that was not on the pre-rebase head. Any failure or doubt leaves the branch alone and logs the reason (no commit this pass, but the engine rebase was left in place: …); a pass that commits is unchanged. Side effect: dispatchReviewReinvoke/dispatchCIFixReinvoke capture headBefore pre-rebase, so a restored no-commit pass now compares equal and the no-op refund (ReviewCycleDecremented, #1045) and CIFixNoOpRecorded (#958) fire where a moved HEAD previously hid them. Catch-up covers worktrees already in the rebased state (§6.30). SetCommentRebaseRestoreDisabledForTest is the neutralisation seam.

4.5 markCommentsSeenByStage

After a stage invocation (not comment processing), markCommentsSeenByStage() adds ROCKET reactions to all pre-existing comments that were included in the prompt as context. This prevents those comments from triggering the awaiting-input unblock logic on subsequent polls.

4.6 Comment-Processing Circuit Breaker (#1089)

Fixes 1/4 (#1087, ADR-069) and 2/4 (#1088, ADR-070) close the two known pumps behind the #1083 incident: a bot comment can no longer silently lift an operator’s pause, and a known bot service-notice no longer spawns a worker or draws a reply. Neither fix, however, counts damage from a loop it doesn’t recognize — a new bot-notice variant, a webhook that re-triggers on Fabrik’s own reply some other way, or any other self-sustaining pattern not yet anticipated. The circuit breaker is a defense-in-depth backstop of last resort, not a replacement for fixes 1/4 or 2/4: if the same issue undergoes N comment-processing invocations within a rolling window T with no forward progress, the engine pauses the issue, posts an explanatory comment, emits a structural TUI event, and suppresses further dispatch. It should rarely if ever fire in practice.

Counter. Tracked per issue (not per stage — the #1083 incident stayed on one stage throughout, but a legitimate stage transition mid-window must not itself reset the count to a lower, easier-to-hit number) in itemstate.Store as ItemState.CommentBreaker: InvocationsAt []time.Time plus LastAuthor string (the author of the comment that triggered the most recent recorded invocation, surfaced in the trip comment). Two mutations drive it: CommentBreakerInvocationRecorded{Repo, Number, At, Author, Cutoff} (append, pruning any InvocationsAt entries older than Cutoff first when Cutoff is non-zero) and CommentBreakerReset{Repo, Number} (clear both fields; a no-op — no Change emitted — when already empty). The Store holds only raw timestamps; threshold/window business logic lives in the engine (engine/comment_breaker.go), mirroring the existing mergeTrainTrials runaway-guard precedent (ADR-059 D8). recordCommentBreakerInvocation() computes Cutoff from the current window and passes it on every append, so InvocationsAt is pruned at write time and doesn’t grow without bound for an issue that receives invocations sparser than the window; commentBreakerCount() additionally prunes-and-counts on every read against time.Now().Add(-window) as a cheap belt-and-suspenders pass that also stays correct if the configured window shrinks between writes.

Recording point. recordCommentBreakerInvocation() is called inside processComments (engine/comments.go) immediately after the working comment slice is finalized (once the item’s bot-service-notice filtering has run), before any setup side effect — the step 0 JobStartedEvent emission, the step 1 👀 reaction, the step 2 fabrik:editing label add, and step 3 worktree setup (§4.2) — and well before step 4’s InvokeForComments() call. Recording this early (#1413) means a failure in any of the three setup steps counts as a cycle, not just an invocation-time failure: each of the three setup-failure early-return paths also calls checkCommentBreaker() (see “Trip action” below) with a reason describing the failing step, so a persistently failing setup step — e.g. a fabrik:editing label add that fails on every poll under GraphQL budget pressure — trips the breaker within the configured window instead of looping unbounded (the #1382/#1386 failure mode this closes). Before #1413, the record call sat immediately before the Claude invocation, so all three setup steps could fail indefinitely without ever incrementing the counter; ADR-1089’s original rationale for that placement (“an early return before Claude runs… does not count as a wasted cycle”) is superseded by ADR-1413.

Threshold and window. Default N = 10 invocations within T = 30 minutes, both zero-means-default (mirroring MaxTrainTrialsPerWindow/TrainTrialWindowDuration): Config.MaxCommentCyclesPerWindow / Config.CommentCycleWindow, overridable via --max-comment-cycles-per-window / --comment-cycle-window flags, FABRIK_MAX_COMMENT_CYCLES_PER_WINDOW / FABRIK_COMMENT_CYCLE_WINDOW env vars, or max_comment_cycles_per_window / comment_cycle_window in config.yaml (flag > env > config.yaml > default 10 / 30m — same precedence order as every other zero-means-default engine setting).

Reset triggers (any one resets InvocationsAt to empty before the threshold is reached):

Trigger Hook point Rationale
stage:*:complete transition handleStageComplete() (engine/stages.go), reset called before any completion side effect The one function reached on stage completion regardless of whether it originated from a plain stage run or from finalizeComments’s completed branch — a single choke point.
New commit on the branch Inside processComments, comparing gitHeadSHA(workDir) immediately before and after the Claude invocation Reuses the existing gitHeadSHA helper (already used by detectProgress, §3/§4.3); detects the commit locally without depending on a webhook-lagged PR head-SHA update.
PR state change CommentBreakerObserver (engine/observers.go), subscribed to itemstate.Store, reacting to the PRStateChanged flag PRStateChanged is a narrower sub-flag of LinkedPRChanged, set only by PRDetailsUpdated (a genuine PR-level transition: merged, closed, or draft↔ready). It is deliberately not keyed off the broader LinkedPRChanged — that flag also fires on PRReviewSubmitted/PRReviewCommentCreated/ReviewThreadCommentAdded (a new PR review or inline review comment), which is the same event that supplies the comment processComments is about to process for a review-reinvoke. Resetting on that broader flag would zero the counter on the same cycle that records its own invocation, capping the observed count at 1 forever and defeating the breaker for its primary PR-review-comment-loop use case (caught during PR review of #1089, fixed before merge).
Issue body edited (FABRIK_ISSUE_UPDATE) Inside publishCommentOutput, in the existing extractUpdatedBody(output) != "" branch (§4.2 step 6) Pre-PR stages (Specify/Research/Plan) iterate purely via issue-body edits — no commit, no PR yet, and no stage completion until the human is satisfied. Without this trigger, legitimate iterative Q&A in those stages could accumulate invocations toward the threshold with no other reset available.
Manual human unpause clearFailedStage() (engine/item.go), alongside the existing ReviewCycles/CIFixCycles/RebaseCycles/EnqueueCycles clears A human who investigates and manually removes fabrik:paused is explicitly giving the issue another shot; without this reset they would be re-tripped after a single subsequent invocation that hadn’t yet produced one of the four automatic signals above.

Trip action. checkCommentBreaker(item, reason string) is called from five points in processComments: each of the three setup-failure early-return paths (editing-label add, base-branch resolution, worktree setup — reason names the failing step and its error), the non-completing invocation-failure early-return path (reason == ""), and, when the invocation proceeds normally, after publishCommentOutput/finalizeComments (reason == "", so any reset those steps applied takes effect first). A sixth, distinct early-return sits just above the non-completing-failure point (mirroring the pre-existing usage-limit exclusion there): a tools-denied classification (*claudeToolsDeniedError, #1704, §7.3c) returns before either checkNoOpCommentCycle or checkCommentBreaker is ever called, so a mode denial is bounded exclusively by its own MaxToolsDeniedRetries counter, never by this breaker’s frequency-based thresholds. When commentBreakerCount() >= N, tripCommentBreaker(item, count, window, reason):

  1. Applies fabrik:paused + fabrik:awaiting-input via the shared pauseIssue()/pauseOpts helper (ADR-065) — the same helper and same labels pauseForReviewCycleLimit and the merge-train runaway guard already use, so ADR-069’s honorable-pause guarantee (a subsequent bot comment cannot silently lift the pause, §1.4 fabrik:paused row) applies automatically. No new label is introduced.
  2. Posts an explanatory 🏭 **Fabrik — comment-processing circuit breaker tripped** comment naming the invocation count, the window, the last comment author, and how to resume (remove fabrik:paused). When reason is non-empty (the tripping cycle was a setup failure, #1413), the comment gains an additional sentence naming the specific setup step and error that caused it — e.g. “The cycle that tripped this breaker failed during setup: the fabrik:editing label add failed: ." — so the pause is diagnosable rather than a bare "no forward progress" notice. `reason` is passed as a call parameter, not persisted in `itemstate` — it always describes the cycle that is tripping the breaker synchronously, in the same call, so no durable storage is needed (mirrors `LastAuthor`'s "only the triggering cycle's context matters" precedent).
  3. Emits tui.CommentBreakerTrippedEvent{IssueNumber, Repo, Title, StageName, InvocationCount, Window, LastCommentAuthor} via emitStructural — the TUI’s active pane renders it as a blocked entry, mirroring IssueBlockedEvent (§9.8).
  4. Suppresses further dispatch: once fabrik:paused + fabrik:awaiting-input are present, the item is gated by the same paused/awaiting-input machinery as every other pause (§1.4, §4.4) — only a human comment created after the pause (resumeAuthorised()) can resume it, and per §4.4 a resuming human comment is processed together with any bot chatter that accumulated during the pause, which itself records a fresh invocation and is subject to the breaker again.

Scope note. The breaker’s suppression is a downstream consequence of the standard pause/awaiting-input labels, not a separate dispatch gate — it adds no new entry to itemNeedsWork/itemMayNeedWork (§8.1, Appendix C). This is unlike fabrik:awaiting-done or fabrik:blocked, which are engine-internal gates with their own dispatch-suppression wiring; the circuit breaker deliberately reuses the existing human-only pause semantics instead of introducing a fifth.

4.7 Success-Agnostic No-Op Comment Cycle Breaker (#1555)

handarbeit/fabrik#1254’s Validate stage redelivered the same stale, already-processed bot review body seven consecutive times — each a clean (err == nil) cycle: no commit, no issue-body update, no FABRIK_STAGE_COMPLETE. §4.6’s breaker records every invocation regardless of success, but its counter is a rolling 30-minute window, sized for #1083’s burst-loop shape (~995 invocations in rapid succession). A loop whose invocations are spaced sparser than 30 minutes — as this one was, driven by Fabrik’s own self-upgrade restart cadence (the redelivery’s actual root cause, fixed directly by §6.2’s “Durable review-ids-addressed marker”) — never accumulates enough entries within any single window to trip §4.6’s breaker, no matter how many total invocations occur over the issue’s lifetime. This section’s counter is the safety net for that redelivery, not its fix; it is a second, independent breaker that closes the windowing gap regardless of what causes a cycle to be a no-op.

Counter. StageState.NoOpCommentCycles map[string]int (internal/itemstate/itemstate.go) — per-stage, unlike §4.6’s item-scoped CommentBreaker, and never time-pruned, mirroring ReviewCycles/ReviewBlockedCycles’s shape (§6.2, ADR-1518) rather than CommentBreaker’s rolling-timestamp shape. Two mutations drive it: NoOpCommentCycleIncremented{Repo, Number, StageName} and NoOpCommentCycleReset{Repo, Number, StageName} (a no-op — no Change emitted — when already zero).

Recording point. checkNoOpCommentCycle(item, stage, progressed bool, lastAuthor string) (engine/comment_noop_breaker.go) is called from the same five mutually-exclusive exit points in processComments that call checkCommentBreaker (§4.6’s “Trip action”) — exactly one of the five executes per cycle. It is evaluated first at each site; only when it does not trip does checkCommentBreaker also run for that cycle:

if !e.checkNoOpCommentCycle(item, stage, progressed, lastCommentAuthor(comments)) {
    e.checkCommentBreaker(item, reason)
}

Both counters still record every invocation independently — NoOpCommentCycles increments even on a cycle where §4.6’s breaker is the one that ultimately trips. What’s mutually exclusive is the pause escalation: at most one fabrik:paused application and one trip comment per cycle.

progressed — R2’s three signals, computed once per cycle. progressed := headChanged || extractUpdatedBody(output) != "" || completed:

The three setup-failure sites (editing-label add, base-branch resolution, worktree setup) pass progressed=false directly — no invocation ran, so none of the three signals can be true by construction.

No refund/decrement counterpart, unlike ReviewCycles. progressed=true unconditionally resets the counter to 0; there is no partial-credit decrement. A cycle that would qualify for #1045’s ReviewCycleDecremented refund in the review-gate path (a reinvoke dispatched while the gate was already clear, landing no commit) is, from this counter’s perspective, indistinguishable from any other no-progress cycle — it correctly increments here. This counter and §6.2/ADR-1045’s refund answer different questions: ADR-1045 asks “did this reinvoke change the review-gate verdict”; this counter asks “did this comment-processing cycle change anything observable on the issue.”

Threshold. Default 10, zero-means-default: Config.MaxNoOpCommentCycles, overridable via --max-no-op-comment-cycles flag, FABRIK_MAX_NO_OP_COMMENT_CYCLES env var, or max_no_op_comment_cycles in config.yaml — the same flag > env > config.yaml > default precedence as §4.6’s own threshold.

Deliberately not equal to MaxReviewCycles’s default (5). Both counters observe the same processComments funnel (see below), but only ReviewCycles is ever refunded — a shared default would make NoOpCommentCycles the one that always trips first, silently nullifying ADR-1518’s tolerate-forever guarantee at whatever count the shared value happened to be. An early version of this fix did ship both at 5 and isolated the collision out of the review-reinvoke tests with an inflated per-test override (eng.cfg.MaxNoOpCommentCycles = 100) rather than fixing the default — that hid, rather than resolved, the fact that TestHandleReviewGate_FiveNoOpReinvokes_DoNotExhaustBudget_SixthGenuineFindingAddressed’s own guarantee was false under the shipped default config. Live evidence on this repo during review (a healthy, mergeable PR accumulating 4 consecutive no-op comment-processing cycles from ordinary duplicate bot delivery) showed 5 was too tight regardless of the test-isolation question — normal operation was one round from tripping it. 10 was chosen as comfortably clear of that observed ordinary-churn ceiling while still being a hard bound. See ADR-1555’s Consequences.

Interaction with §6.2/ADR-1045’s no-op refund on the review-reinvoke path. Because checkNoOpCommentCycle runs at the same processComments funnel every dispatch source shares (§4.6), it also counts cycles dispatched via dispatchReviewReinvoke. ReviewCycles’ refund (ReviewCycleDecremented) was deliberately built to forgive an unbounded run of no-op reinvokes as long as the review gate itself keeps returning a clear verdict (ADR-1518); this counter does not forgive the same cycles, so at the default threshold, ten consecutive no-op review-reinvoke rounds now pause the issue even where ReviewCycles alone would have tolerated more. This narrows ADR-1518’s “forgive forever” guarantee to “forgive up to MaxNoOpCommentCycles consecutive rounds” in practice — deliberately, with enough headroom above both MaxReviewCycles’s own default and observed ordinary-operation churn that it should not be reached by a review bot behaving normally. An operator whose review bots need more than 10 consecutive no-op rounds before converging should raise --max-no-op-comment-cycles.

Reset triggers are narrower than §4.6’s five — no PRStateChanged reset, no new itemstate.Store observer. Only R2’s three signals above reset the counter, plus a manual unpause (below). If a future incident shows this is too narrow, that is a follow-up, not a blocking gap — this counter is a safety net, not the primary defense (the durable-marker fix in §6.2 is).

Manual unpause — from either pause route. EngineCyclesCleared’s store.go apply case deletes StageState.NoOpCommentCycles[stageName] alongside its existing deletes for ReviewCycles/ReviewBlockedCycles/CIFixCycles/RebaseCycles/EnqueueCycles. This breaker’s own trip action (below) pauses via fabrik:paused + fabrik:awaiting-input — the resume path §1.4’s table routes through unblockAwaitingInput (isAwaitingInput(item) true + a human comment), not clearFailedStage (which handles the separate stage:<name>:failed + fabrik:paused shape). Both clearFailedStage and unblockAwaitingInput apply EngineCyclesCleared and reset §4.6’s CommentBreaker, so the counter is reset regardless of which of the two pause shapes an item is resumed from. (unblockAwaitingInput gained this call as part of #1555 — before that fix it cleared retry/pause state but not the cycle counters, so NoOpCommentCycles survived an awaiting-input resume at its tripped value and could re-trip on the very next cycle; see ADR-1555.)

Trip action. tripNoOpCommentCycleBreaker reuses pauseIssue/pauseOpts exactly like §4.6’s breaker — fabrik:paused + fabrik:awaiting-input, ADR-069’s honorable-pause guarantee. The trip comment (🏭 **Fabrik — no-op comment-processing circuit breaker tripped** — a distinct header from §4.6’s 🏭 **Fabrik — comment-processing circuit breaker tripped**, so an operator always knows which mechanism fired) names the stage, the consecutive no-progress count, and the last comment’s author.

See ADR-1555, which also covers the durable review-ids-addressed marker fix (§6.2) for the redelivery this counter is a safety net for.


5. PR Lifecycle Integration

5.1 Draft PR Creation

When: After a stage signals FABRIK_STAGE_COMPLETE, if the stage has create_draft_pr: true. An early-guard path also runs before output is posted when both create_draft_pr: true and post_to_pr: true are set, so the PR exists before postOutputToPR runs.

Code path: processItem() → ensureDraftPR()

Flow (idempotent, up to 3 attempts with exponential backoff):

  1. Check for an existing open PR via FetchLinkedPR() — if found open and not merged, ensure body contains Closes #N and return its number. Closed or merged PRs are ignored; a new PR will be created.
  2. Push the issue branch via PushBranch()
  3. Build a seed body from .fabrik-context/ files (issue summary, plan approach, verification placeholder). The Summary is ## Summary, else the Spec Kit template’s **Input** text, else the issue’s first paragraph; the Problem is ## Problem, else ## Background, else the first paragraph. The template’s # Feature Specification: title and header-field lines are never used as either section
  4. Create draft PR via CreateDraftPR() with title from issue, targeting baseBranch, body ending with Closes #N

Transient errors (network errors, 5xx) are retried with backoff (base delay 500ms, doubled each attempt). Non-transient errors (4xx including 422) return immediately without retry. Returns (prNumber, nil) on success, (0, error) on failure.

Success logging: [#N pr] created draft PR #<num> (branch: fabrik/issue-N)

Failure logging: [#N pr] failed to create draft PR for branch fabrik/issue-N: <error>

5.5 PR Creation Failure Path

Trigger: Stage with create_draft_pr: true emits FABRIK_STAGE_COMPLETE, but ensureDraftPR returns (0, error) after exhausting its in-process retry loop.

Critical invariant: handleStageComplete is NOT called when PR creation fails. The stage does not advance. stage:<X>:complete is not added.

Retry counting: Each PR creation failure increments the same per-stage Attempts counter used by StageRetryIncremented. The StageRetryCleared mutation (which resets the counter) fires only on PR creation success, not on failure. This means PR creation failures count against MaxRetries just like Claude failures.

Code path (completion block in processItem):

FABRIK_STAGE_COMPLETE received
  → if stage.CreateDraftPR and prNumber == 0:
      prErr = ensureDraftPR(item, baseBranch)
      if prErr != nil:
          if MaxRetries > 0:
              StageRetryIncremented
              if Attempts >= MaxRetries:
                  escalatePRCreationFailure()  → fabrik:paused, stage:<X>:failed, comment
                  releaseLock()
                  return
          PRCreationFailedRecorded            → sets in-memory flag for R5
          releaseLock()
          return                               → NO handleStageComplete
      else:
          updatePRVerification(prNumber, summary)
  releaseLock()
  StageRetryCleared, EngineUnpaused
  handleStageComplete()                        → stage:X:complete, optional advance

Escalation comment (posted by escalatePRCreationFailure): Names PR creation as the failure cause (not Claude), includes the manual workaround command using the actual base branch: `gh pr create --head fabrik/issue-N --base <baseBranch> --body "Closes #N"`, and instructs the user to remove fabrik:paused to resume.

PRCreationFailed in-memory flag (R5): StageState.PRCreationFailed map[string]bool records that Claude completed a stage but the draft PR could not be created. This flag does not survive engine restarts. It is cleared by StageRetryCleared (on PR creation success). On restart, the item re-runs Claude, which is safe and conservative — Claude’s commits are idempotent.

R5 skip-Claude retry path: On the next poll cycle for an item with PRCreationFailed[stageName] == true, the engine checks this flag early in runStage (before the Claude invocation). If ensureDraftPR now succeeds, the engine calls handleStageComplete directly — no Claude re-invocation needed, since the worktree already has the commits from the prior run. If ensureDraftPR still fails, the retry counter is incremented and the escalation path applies when MaxRetries is reached. Exception: for a fabrik:children-spawned Implement item with zero commits ahead of base (an empty coordinator, §6.7.1), this branch instead routes directly to handleNoWorkNeeded before ever attempting ensureDraftPR — see §6.7.1 for why this case self-heals to Done rather than retrying a doomed PR creation.

State Trigger Action Outcome
Locked + In Progress FABRIK_STAGE_COMPLETE + create_draft_pr + PR creation fails PRCreationFailedRecorded; lock released; NO handleStageComplete Item waits for next poll
Locked + In Progress FABRIK_STAGE_COMPLETE + create_draft_pr + PR creation fails + Attempts >= MaxRetries escalatePRCreationFailure: fabrik:paused, stage:<X>:failed, comment Issue paused for human intervention
Locked + In Progress PRCreationFailed flag set; ensureDraftPR succeeds on retry (R5) StageRetryCleared, handleStageComplete — no Claude run Stage advances normally

5.2 Mark PR Ready

When: After a stage signals FABRIK_STAGE_COMPLETE, if the stage has mark_pr_ready_on_complete: true.

Code path: processItem() → markPRReady()

Flow:

  1. Push the issue branch
  2. Find PR number (uses knownPR from ensureDraftPR if available, else FindPRForIssue())
  3. MarkPRReady() transitions draft → ready-for-review; retries up to 3 times on transient 5xx errors with exponential backoff (500ms / 1s / 2s); non-transient errors (4xx, including 429) are logged immediately without retry

Durable retry beyond this in-process budget (ADR-1582): if the MarkPRReady() call is still failing once this 3-attempt budget is exhausted — or fails non-transiently on the first attempt — markPRReady durably records the outstanding call via fabrik:awaiting-pr-ready, retried by a dedicated per-poll settle scan (settlePRReadyScan) independent of this stage ever being re-dispatched. See §6.22 for the full mechanism.

Note: This triggers external review bots and populates LinkedPRReviewRequests, which is why the review gate in handleStageComplete() (Path 1) is always optimistic — reviewer data is stale at that point.

5.3 Linked PR Discovery

Fabrik discovers PR comments through the closedByPullRequestsReferences GraphQL field, which traverses issue → linked PRs → PR comments. The Closes #N keyword in the PR body creates this linkage.

MergePR’s own CI precondition (ADR-933). Before this addition, github.Client.MergePR gated only on GitHub’s mergeable field (merge-conflict status), which stays true while required status checks are still pending or failing — a gap that branch protection (enforce_admins) normally masked, but which was live and unmitigated on handarbeit/fabrik (enforce_admins: false, engine token is an org admin). MergePR now self-gates instead of relying on branch protection: after the existing mergeable check passes, it calls FetchPRMergeableFields a second time and refuses to merge — without ever calling the merge endpoint — unless the returned mergeable_state satisfies gh.MergeableStateAccepted ({clean, unstable}). A refusal returns the sentinel gh.ErrNotMergeableCI, deliberately distinct from gh.ErrNotMergeable (the conflict/dirty case) so errors.Is lets callers tell “CI not green” apart from “merge conflict” — a CI refusal must never apply fabrik:rebase-needed or consume a rebase cycle. This self-gate is deliberately unchanged by ADR-1441 (#1441, R3): the merge path continues to defer to mergeable_state alone, per ADR-072’s operator note that a check that should refuse an engine merge belongs in branch protection’s required-checks list, not in MergePR. Only the CI advance gate (checkCIGate, below) tightened — checkCIGate no longer shares this {clean, unstable} allowlist; its own fast path narrowed to mergeable_state == "clean" only (see §2.10, §6.4 rule 9).

This precondition runs identically inside MergePR regardless of caller, so all four call sites inherit it:

None of the four call sites currently branch on ErrNotMergeableCI to escalate — each already logs the failure and lets the item retry on its existing cadence (a full Validate re-invocation for attemptMergeOnValidate, since that call site has no lighter-weight poll-only retry; the next poll’s convergence pass for the rebase-reinvoke fallback; the next merge-train cycle, with members left in Queued, for the landing paths). This is unchanged behavior, not new machinery — fabrik:rebase-needed is applied earlier and independently by checkMergeabilityGate from settle.Status == PRMergeConflicting, never from MergePR’s own return value, so a CI refusal was already incapable of reaching the rebase-cycle path before this change.

5.4 Auto-Merge on Validate (yolo issues)

When: Validate stage completes and the issue has fabrik:yolo (or global cfg.Yolo) and does NOT have fabrik:cruise.

Code path: handleStageComplete() → attemptMergeOnValidate() (Path 1); or catch-up loop Phase 2 → attemptMergeOnValidate() (Path 2 — when fabrik:auto-merge-enabled is absent)

Flow:

  1. If fabrik:auto-merge-enabled is already present on the issue: return nil (idempotent — auto-merge was already enabled in a previous invocation).
  2. Fetch linked PR via FetchLinkedPR().
  3. Call EnablePullRequestAutoMerge(owner, repo, pr.Number, AutoMergeStrategy). The strategy defaults to MERGE and is configurable via FABRIK_AUTO_MERGE_STRATEGY. On failure, two branches apply:
    • ErrAutoMergeNotEnabled (repo-level setting disabled): log a guidance message pointing to Settings → General → Allow auto-merge; return a retriable error (next poll retries). No direct-merge attempt.
    • Any other error (PR in a terminal GitHub state — e.g. CLEAN, UNSTABLE, or future variants where GitHub refuses to queue auto-merge): log the specific error, then attempt a direct MergePR call as a fallback. This is the only call site with no independent mergeable_state gate upstream of MergePR (reached whenever wait_for_ci: false), so MergePR’s own CI precondition (“MergePR’s own CI precondition (ADR-933)” above) does real work here. If MergePR succeeds: apply fabrik:auto-merge-enabled and return (true, nil). If MergePR fails with ErrNotMergeable (DIRTY PR — a conflict): return (false, err) so existing rebase/CI-fix gates can act on it. If MergePR fails with ErrNotMergeableCI (a required check is blocked, or GitHub is still computing): return (false, err) as well, but this is not a conflict — fabrik:rebase-needed is never applied and no rebase cycle is consumed. Since handleStageComplete does not add stage:Validate:complete on either error, the whole Validate stage simply re-dispatches on the next poll.
  4. On EnablePullRequestAutoMerge success: apply fabrik:auto-merge-enabled label. GitHub now owns the merge decision atomically — no Fabrik-side MergePR call. Done advancement is deferred to checkAutoMergeConvergence in Phase 1 of the catch-up loop.

cruise > yolo precedence: If fabrik:cruise is present on the issue, attemptMergeOnValidate returns immediately without calling EnablePullRequestAutoMerge. Cruise items keep the current stage:Validate:complete behavior: the branch is maintained (rebase reinvoke, CI-fix reinvoke) but merging is left to the user.

See section 5.5 for how Fabrik monitors the convergence flow after auto-merge is enabled.

Non-yolo issues: Items without fabrik:yolo and without fabrik:cruise follow the same behavior as cruise for purposes of merge tracking — checkMergeabilityGate and checkCIGate continue to operate using MaxRebaseCycles / MaxCiFixCycles per-gate cycle limits. EnablePullRequestAutoMerge is never called for these issues. This is unchanged from pre-5.4 behavior.


5.5 Post-Validate Convergence Monitor (yolo issues)

After fabrik:auto-merge-enabled is applied (section 5.4), every Phase 1 iteration of the catch-up loop calls checkAutoMergeConvergence() instead of the legacy checkMergeabilityGate / checkCIGate path. All merge decisions for the issue are delegated to GitHub’s server-side auto-merge logic (legacy native auto-merge) or to GitHub’s merge queue (ADR-058). handleAutoMergeConvergence calls settlePRMergeState() before forwarding to checkAutoMergeConvergence(); the settle result is consumed for merge/CI state interpretation (eliminating the split-brain), though the merge/CI gates remain bypassed — see §6.4. The function is the single mutation point for the convergence/ejection lifecycle (ADR-056 — no parallel scanner is added); the merge-queue ejection classifier (ADR-058 D4) is composed into the same branch ladder.

Convergence budget:

checkAutoMergeConvergence() decision tree (ordered; first match wins):

# Observed state Action
① Terminal-first (settle.Status == PRMergeTerminal \|\| pr.Merged \|\| pr.State == "closed") advanceConvergedPRToDone: remove fabrik:auto-merge-enabled + fabrik:rebase-needed, advance to Done. Checked FIRST so a queue merge (which also dequeues the PR) is never misread as an ejection failure (the #913 trap). settle.Status == PRMergeTerminal already re-confirmed the merge via the authoritative single-PR endpoint, catching the window where the REST list endpoint still reports the PR open. On a confirmed merge (never for closed-without-merging), also calls closeIssueIfNonDefaultBase(item, prNumber) (ADR-1096) to explicitly close the issue when its resolved base differs from the repo default — see §2.8.
② In-queue hand-off (settle.Status == PRMergeQueued) Record LastEnqueuedSHA (the SHA GitHub holds in the queue), log, and wait — no label churn. The queue owns the PR. Replaces #935’s inline settle.PR.IsInMergeQueue read (ADR-056 D1 consolidation). Then falls to step ②′.
②′ Merge-group stall (settle.Status == PRMergeQueued persisting past CIWaitTimeout, ADR-058 D5) pauseForMergeGroupStall(): post instructional comment, apply fabrik:paused + fabrik:awaiting-input, remove fabrik:auto-merge-enabled. Guards on cfg.CIWaitTimeout > 0 (default 30 min). Dwell anchor: FetchLabelAppliedAt("fabrik:auto-merge-enabled") — set at first enqueue, survives restarts. Detection signal: settle.Status == PRMergeQueued persisting past the dwell means no merge-group CI ever reported (merge-group runs are observable as a status change away from QUEUED/AWAITING_CHECKS). Idempotency: pauseForMergeGroupStall removes fabrik:auto-merge-enabled, so handleAutoMergeConvergence returns false on the next poll (missing label guard, line 126), and the catch-up loop also skips fabrik:paused items unconditionally. Resume: operator fixes CI (adds on: merge_group), re-queues the PR manually, removes fabrik:paused. On the next Validate re-invoke, attemptMergeOnValidate re-applies fabrik:auto-merge-enabled with a fresh timestamp. ADR-1410 note: this dwell is structurally identical to the CI gate’s own R3/mergeable-state liveness dwells (§6.14.5) — no merge-group CI has ever reported, so there is nothing to observe progress on — which is exactly why it needed no code change when CIWaitTimeout was repurposed for that same class of dwell; it correctly keeps reading CIWaitTimeout directly rather than the new, much larger CIBackstopTimeout.
③ Auto-merge disabled (!pr.AutoMergeEnabled && !mergeQueueEnabled) Apply fabrik:paused + fabrik:awaiting-input; remove fabrik:auto-merge-enabled; post comment with resume options. Guarded on !mergeQueueEnabled: every enqueue-path PR has AutoMergeEnabled == false as its normal state (EnqueuePullRequest never sets auto_merge), so on a merge-queue repo a false flag is NOT a user action — misreading it would pause every ejected PR (a #913-adjacent trap). mergeQueueEnabled = item.LinkedPRIsMergeQueueEnabled \|\| settle.PR.IsMergeQueueEnabled (dual-source).
④ Budget exhausted Call pauseForConvergenceFailed() (see below). Applies to both queue and non-queue repos — the ultimate strand safety-net.
⑤ settle.Status == PRMergeUnsettled Wait — GitHub still computing after a push/dequeue; re-evaluate next poll.
⑥ settle.Status == PRMergeConflicting Worker guard (snap.Worker() != nil) → skip if in-flight; cycle-limit check (snap.RebaseCycles(stage) >= MaxRebaseCycles) → pauseForRebaseCycleLimit(); otherwise increment RebaseCycles and dispatch a Claude rebase reinvoke. Shared by queue and non-queue repos; past step ②, a conflicting PR is guaranteed not-in-queue, so #935’s inline in-queue guard here is removed (dead). On a queue repo this is the ejection→resolve path; the clean re-enqueue (⑦) fires later once the resolved PR re-derives clean.

Ejection-recovery ladder (steps ⑦–⑧) — merge-queue repos only (mergeQueueEnabled == true):

# Observed state Action
⑦ settle.Status == PRMergeBlocked (ejected, re-derived CI failure) Worker guard → skip if in-flight; cycle-limit (snap.CIFixCycles(stage) >= MaxCiFixCycles) → pauseForCIFixCycleLimit(); otherwise increment CIFixCycles and dispatch a Claude ci-fix reinvoke. Re-enqueue (⑧) fires later once CI is fixed.
⑧ settle.Status == PRMergeReady (ejected, re-derived clean) and (leftQueue \|\| (LastEnqueuedSHA != "" && pr.HeadSHA != LastEnqueuedSHA)) reEnqueueOrPause — re-enqueue fresh (see below). leftQueue (= priorInQueue) fires the immediate same-SHA ejection re-enqueue; the SHA-change clause fires the post-resolution tail re-enqueue. When neither holds (post-enqueue consistency window: priorInQueue == false and the SHA still equals LastEnqueuedSHA), wait — the PR is already enqueued and GitHub just hasn’t reflected isInMergeQueue == true yet.
⑨ Non-queue fall-through (any other status) Wait — GitHub’s native auto-merge is handling it.

Poll-native ejection detection (the “left the queue” edge): leftQueue = priorInQueue (settle.Status != PRMergeQueued is guaranteed past step ②). priorInQueue is the item’s previous-poll LinkedPRState.IsInMergeQueue, captured in poll.go from the pre-fetch store snapshot before ItemDeepFetched overwrites the store with the current value, then threaded through phase1Ctx → handleAutoMergeConvergence → checkAutoMergeConvergence (ADR-058 D4 OQ-3). Reading “prior” from e.store inside the classifier would yield the already-overwritten current value, silently losing the edge. No webhook is required; dequeued.reason is never consulted for correctness.

reEnqueueOrPause — bounded fresh re-enqueue (FR-3):

  1. Merged-first re-confirmation at the mutation point (the #913 trap): FetchPRMerged (authoritative single-PR endpoint) — if merged → advanceConvergedPRToDone; if the call errors → wait (never re-enqueue on an unconfirmed state). The REST list endpoint reports merged == false for several seconds after a queue merge, so this guard runs right before the dangerous mutation.
  2. Worker-in-flight guard (snap.Worker() != nil) → skip.
  3. snap.EnqueueCycles(stage) >= MaxEnqueueCycles → pauseForEnqueueCycleLimit() (queue-thrash exhaustion).
  4. Otherwise EnqueuePullRequest(owner, repo, prNum, pr.HeadSHA) at the fresh head SHA (optimistic concurrency fails safe on a stale SHA; skipped if HeadSHA is empty), then apply EnqueueCycleIncremented + PREnqueueRecorded. Re-enqueue is always fresh after off-queue resolution, never re-enqueue-in-place, so conflict-heavy PRs do not starve the queue. An enqueue failure does not increment the cycle.

Rebase reinvoke + auto-merge re-enable: GitHub disables auto-merge on every push. After dispatchRebaseReinvoke() completes successfully (Claude resolved conflicts, pushed), EnablePullRequestAutoMerge is called again to re-arm auto-merge — but only on non-merge-queue repos (!item.LinkedPRIsMergeQueueEnabled, ADR-058 D4). On a merge-queue repo the recovery path is re-enqueue, not native auto-merge, so the convergence monitor re-enqueues the resolved PR once it re-derives clean (step ⑧); re-enabling native auto-merge there would fight the queue model. This is the only scenario where auto-merge is re-enabled without going through attemptMergeOnValidate.

pauseForConvergenceFailed() — convergence budget exhausted: Uses settle.PR (from the pre-fetched PRSettleResult) for PR diagnostic fields, FetchCommitsBehind for the commits-behind count, settle.CheckRuns for the CI summary, and the store for the current rebase cycle count — no independent FetchLinkedPR or FetchCheckRuns calls. Posts a structured pause comment containing:

Then: applies fabrik:paused + fabrik:awaiting-input; removes fabrik:auto-merge-enabled. Does NOT add fabrik:awaiting-ci (CI may be healthy; the convergence failure is about the merge window, not CI).

RebaseCycles vs. budget: Under the convergence flow, RebaseCycles is incremented on every rebase dispatch. MaxRebaseCycles IS consulted as a gate — the same three-step guard used by handleMergeAndCIGates (in-flight guard → cycle-limit check → dispatch or pauseForRebaseCycleLimit) applies here. When FABRIK_CONVERGENCE_BUDGET=0, the time-based budget check is skipped, but MaxRebaseCycles still bounds rebase reinvokes — so an unresolvable conflict never produces an unbounded rebase loop.

Cap composition (ADR-058 D4 FR-3): Four bounded paths pause independently, first-to-trip wins, no double-pause (each pause returns immediately after applying fabrik:paused, and a paused item is skipped on the next poll):

Cap Counter Increment site Pause Catches
MaxEnqueueCycles (default 5; --max-enqueue-cycles / FABRIK_MAX_ENQUEUE_CYCLES) EnqueueCycles each fresh re-enqueue trip (step ⑧) pauseForEnqueueCycleLimit queue-thrash loop (enqueue → eject → re-enqueue → eject) that no single sub-cap would catch
MaxRebaseCycles (default 3) RebaseCycles each rebase dispatch (step ⑥) pauseForRebaseCycleLimit unresolvable conflict
MaxCiFixCycles (default 5) CIFixCycles each ci-fix dispatch (step ⑦) pauseForCIFixCycleLimit persistent CI failure
ConvergenceBudget (default 30m; 0 disables) wall-clock since fabrik:auto-merge-enabled applied, or since the latest fabrik:paused removal if later (#2064) n/a (time-based) pauseForConvergenceFailed the ultimate strand safety-net

EnqueueCycles is dedicated and independent of RebaseCycles/CIFixCycles: the sub-paths increment their own counter when they fire, while EnqueueCycles counts trips through the queue. All four counters are cleared by EngineCyclesCleared (applied by clearFailedStage on unpause/success), so a resumed issue starts fresh. LastEnqueuedSHA (on LinkedPRState) records the head SHA at the last enqueue (recorded both at the in-queue hand-off ② and at each re-enqueue ⑧), making the step-⑧ post-enqueue-window suppression SHA-precise.

State transitions for yolo Validate convergence:

State Trigger New state Labels added Labels removed  
Validate, Complete Poll tick (Phase 2) Validate, Convergence fabrik:auto-merge-enabled    
Validate, Convergence PR merged (GitHub, incl. via merge queue) Done, Pending Cleanup   fabrik:auto-merge-enabled, fabrik:rebase-needed  
Validate, Convergence settle.Status=PRMergeQueued (enqueued / in queue) Validate, Convergence (hand-off)     (no churn; queue owns the PR)
Validate, Convergence settle.Status=PRMergeConflicting (conflict, below MaxRebaseCycles) Validate, Convergence + Rebase in-flight fabrik:rebase-needed    
Validate, Convergence settle.Status=PRMergeConflicting (conflict, at MaxRebaseCycles limit) Validate, Paused fabrik:paused, fabrik:awaiting-input    
Validate, Convergence + Rebase in-flight Rebase push succeeds Validate, Convergence      
Validate, Convergence Ejected + re-derived clean (leftQueue/SHA-changed), EnqueueCycles < Max Validate, Convergence (re-enqueued)     (merge-queue repos; EnqueueCycles++, LastEnqueuedSHA recorded)
Validate, Convergence Ejected + re-derived clean, EnqueueCycles ≥ MaxEnqueueCycles Validate, Paused fabrik:paused, fabrik:awaiting-input   (merge-queue repos; pauseForEnqueueCycleLimit)
Validate, Convergence Ejected + re-derived CI failure (PRMergeBlocked), CIFixCycles < Max Validate, Convergence + CI-fix in-flight     (merge-queue repos; CIFixCycles++)
Validate, Convergence Budget exhausted Validate, Paused fabrik:paused, fabrik:awaiting-input fabrik:auto-merge-enabled  
Validate, Convergence User disables auto-merge in GitHub UI (non-queue repo) Validate, Paused fabrik:paused, fabrik:awaiting-input fabrik:auto-merge-enabled  
Validate, Convergence settle.Status=PRMergeQueued persisting past CIWaitTimeout (merge-group stall, ADR-058 D5) Validate, Paused fabrik:paused, fabrik:awaiting-input fabrik:auto-merge-enabled (merge-queue repos; pauseForMergeGroupStall; operator must add on: merge_group CI + re-queue)

5.6 Engine-Mechanized PR Creation (FABRIK_PR_CREATE Marker)

Motivation: If the Implement skill creates the draft PR directly via gh pr create, it can silently omit the Closes #N closing keyword. Without that keyword, GitHub’s closingIssuesReferences / closedByPullRequestsReferences fields return empty, breaking every downstream gate (review gate, CI gate, auto-merge). To eliminate this failure class, PR creation is mechanized: the skill emits a structured marker block, the engine creates the PR with Closes #N guaranteed as the first line.

Marker format:

FABRIK_PR_CREATE_BEGIN [owner/repo]
TITLE: <single-line PR title>

<PR body content — no closing keyword>
FABRIK_PR_CREATE_END

Code path: processItem() → ParsePRCreateBlock() → processPRCreateMarker()

Processing (runs when stage.CreateDraftPR and a FABRIK_PR_CREATE_BEGIN/END block is found in output, before output is posted):

  1. Cross-repo guard: If the BEGIN line specifies a different owner/repo, the engine pauses the issue with fabrik:paused and a “cross-repo PR creation not supported in v1” comment. Does not create a PR.
  2. Idempotency: Call FetchLinkedPR() — if an open PR already exists on fabrik/issue-N, use it (record in Store, skip create).
  3. Push branch: Call PushBranch() (non-fatal warning if push fails — mirrors ensureDraftPR).
  4. Compose body: finalBody = "Closes #N\n\n" + block.Body. This is the mechanized guarantee.
  5. Create PR: Call CreateDraftPR() with the composed body and verbatim title. Up to 3 attempts with exponential backoff for transient errors.
  6. On success: Cache write-through (RecordPRLinkage), RegisterEcho("pull_request", "opened"), post acknowledgement comment "🏭 **Fabrik** — opened PR #M" on the issue.
  7. On failure: Apply fabrik:paused, post error comment, return error. handleStageComplete is NOT called — the stage does not advance.

After the marker block is processed, the FABRIK_PR_CREATE_BEGIN/END content is stripped from the output string before posting the stage comment.

Malformed marker (missing TITLE, empty body, missing END, invalid owner/repo format): ParsePRCreateBlock returns an error. Each error quotes the line the parser actually found — the first non-empty line, truncated and sanitised for inline rendering by quoteFoundLine (newlines and control characters collapsed, backticks replaced, @ defused with a zero-width space, truncated via sanitizeToolsDeniedCommand) — or says the block was empty: block is empty, missing TITLE line … found <line>, TITLE line has no value, found <line>, PR body is empty after TITLE line <line>. Work is preserved first (ADR 2091): before posting the error, finalizeStageOutcome calls preserveWorkOnMarkerFailure — commitWIP (a no-op on the clean tree a completed Implement normally leaves) then pushBranchOnCancel, which never pauses, so the zero-ahead guard (§”Zero-ahead push guard”) cannot add a second pause. A commit or push failure (including a zero-ahead refusal) is logged only and never suppresses the comment or the pause. Then the engine pauses the issue with fabrik:paused and a comment naming the malformation. Does not create a PR. Un-pausing re-runs Implement on the already-pushed branch rather than from scratch (the stage was never marked complete, and StageAttempted is not recorded on this early-return path, as before).

Fallback to ensureDraftPR: If no FABRIK_PR_CREATE marker is found and stage.CreateDraftPR is true, the engine falls back to the legacy ensureDraftPR path (§5.1). The post-Implement linkage backstop (§5.7) runs in either case.

PRCreationFailed in-memory flag and retry path: When the marker path fails to create a PR, the same PRCreationFailed and R5 retry path as §5.5 applies.

State Trigger Action Outcome
Locked + In Progress FABRIK_PR_CREATE block found + valid processPRCreateMarker → CreateDraftPR + acknowledgement comment PR created with Closes #N as first line
Locked + In Progress FABRIK_PR_CREATE block malformed commitWIP + push, then fabrik:paused + comment quoting the found line Work on the remote branch; issue paused; no PR created
Locked + In Progress FABRIK_PR_CREATE block → cross-repo target fabrik:paused + “cross-repo not supported” comment Issue paused; no PR created
Locked + In Progress FABRIK_PR_CREATE block → CreateDraftPR fails fabrik:paused + error comment Issue paused; no PR created

5.7 Post-Implement Linkage Backstop

Motivation: Belt-and-braces for the case where a PR exists but lacks the Closes #N keyword — either because a legacy skill called gh pr create directly, because a gh pr edit overwrote the body, or because the engine-mechanized path (§5.6) failed and a fallback PR was opened manually.

Trigger: After the Implement stage signals FABRIK_STAGE_COMPLETE (via either the marker path or the ensureDraftPR fallback), before handleStageComplete() is called. Only runs when stage.CreateDraftPR is true.

Code path: processItem() → verifyAndHealLinkage()

Flow:

  1. If prNumber == 0 (no PR exists): skip, return true (no linkage to verify).
  2. Fast path (default-branch items only, !itemHasBaseLabel(item)): call FetchItemDetails(&item) (GraphQL closedByPullRequestsReferences) to get fresh item.LinkedPRNumber.
    • If item.LinkedPRNumber != 0: linkage confirmed — return true. No further calls.
  3. Otherwise — item.LinkedPRNumber == 0, or the item carries a base:<branch> label and skipped the fast path entirely — delegate unconditionally to verifyAndHealLinkageByBody (see below). item.LinkedPRNumber == 0 is never itself treated as “linkage missing” — it is read only as “the fast path didn’t confirm; fall through to the authoritative body check” (#1598, R1/R3). closedByPullRequestsReferences is populated asynchronously, so a read taken within seconds of the write that should have set it (PR creation, or a heal that just edited the body) can legitimately still be empty; the old flow trusted that emptiness as definitive and paused a correctly-linked issue.
  4. verifyAndHealLinkageByBody: call FetchLinkedPR() to find the PR by branch name (fabrik/issue-N).
    • If no PR found: user has diverged from the fabrik/issue-N naming convention; log warning and return true (skip heal — do not pause).
  5. Call FetchPRClosingIssues() (a direct REST fetch + regex parse of the PR body for Closes/Fixes/Resolves #N — synchronous, no propagation lag).
    • If the parsed set already contains the issue number: linkage confirmed — return true. No body edit.
  6. Otherwise call attemptLinkageHeal:
    • R2 guard: fetch the current body (GetIssueBody) and re-check it with gh.ParseClosingIssues before writing anything. If the keyword is already present for this issue — the caller’s own presence check (step 5, or the fast path’s item.LinkedPRNumber) raced a lagging read — log and return success without writing the body and without touching the idempotency guard. This is what stops the historical duplicate-Closes #N write (#1598, R2).
    • Body-length guard (FR-015): if len(currentBody) + len("Closes #N\n\n") > 65,300, pause with fabrik:paused + “body too long for auto-heal” comment. Return false.
    • Idempotency guard: check Snapshot.LinkageHealAttempted(stage.Name, prSHA). If true (a genuine heal — one that actually wrote the body — was already attempted once for this PR head SHA): pause with fabrik:paused + copy-paste gh pr edit recovery command. Return false.
    • Record LinkageHealAttempted{Repo, Number, StageName, PRSHA} mutation in Store (in-memory only), then heal: balanceFences(currentBody), prepend "Closes #N\n\n", call UpdateIssueBody().
  7. Re-verify (only when a write happened in step 6): call FetchPRClosingIssues() again (never FetchItemDetails — re-reading the async derived field a second time is exactly the race this fix removes).
    • If the parsed set now contains the issue number: post "🏭 **Fabrik** — PR body auto-corrected: Closes #N prepended" comment, return true.
    • If still absent: call pauseForBrokenLinkage() — fabrik:paused + copy-paste recovery command, with a reason string that is now unambiguous ("auto-heal completed but PR body still lacks the closing keyword") because the body was actually checked, not inferred from a timeout (R4). This condition can now only mean the write itself didn’t take (e.g. a further concurrent body edit) — the derived-field-lag case is structurally excluded by never consulting that field here.
  8. If heal returns false: handleStageComplete is NOT called — stage:Implement:complete is NOT applied.

LinkageHealAttempted in-memory flag: StageState.LinkageHealAttempted map[string]string maps stage name to PR head SHA. In-memory only — does not survive engine restart. On restart, one more heal attempt is permitted (conservative / safe). Keyed by SHA so a force-push clears the guard naturally. Not recorded when step 6’s R2 guard finds the keyword already present and skips the write — no attempt occurred, so nothing should count against the guard.

pauseForBrokenLinkage comment: Names PR number, issue number, failure reason, and provides a copy-paste gh pr edit ... --body-file ... command the user can run to manually add the closing keyword. After running the command, the user removes fabrik:paused to resume.

State Trigger Action Outcome
Locked + FABRIK_STAGE_COMPLETE Fast path (FetchItemDetails) confirms linkage Return true handleStageComplete proceeds normally
Locked + FABRIK_STAGE_COMPLETE Fast path empty/skipped, but PR body already carries the keyword (FetchPRClosingIssues) Return true — no write handleStageComplete proceeds normally; no duplicate keyword
Locked + FABRIK_STAGE_COMPLETE No linkage in body; PR found on branch; body short; no prior heal Auto-heal: prepend Closes #N, re-verify via FetchPRClosingIssues If verified: comment + advance; if still missing: pause
Locked + FABRIK_STAGE_COMPLETE No linkage in body; body too long (> 65,300 chars) fabrik:paused + “body too long” comment Issue paused for manual recovery
Locked + FABRIK_STAGE_COMPLETE No linkage in body; heal already attempted for this SHA fabrik:paused + copy-paste recovery comment Issue paused for manual recovery
Locked + FABRIK_STAGE_COMPLETE No PR found by branch name Log warning; return true (user-diverged) Stage advances normally

Non-default-base branch (base:<branch>) exception — now the shared path, not a separate implementation: GitHub only populates closingIssuesReferences / closedByPullRequestsReferences for a PR targeting the repository’s default branch — the field is structurally empty for a PR targeting any other branch, regardless of body content (#1046). Before #1598, the default-branch flow (steps 2–9 above) and the base:<branch> flow were two parallel implementations of the same heal/re-verify shape, one trusting the derived field throughout and one trusting the PR body throughout. #1598 collapsed them: verifyAndHealLinkage now only ever owns the cheap item.LinkedPRNumber fast path (step 2, skipped entirely when itemHasBaseLabel(item) is true), and both a base:<branch> item and a default-branch item whose fast path didn’t confirm linkage fall through to the same verifyAndHealLinkageByBody — the body-based confirm/heal/re-verify logic described in steps 4–7. attemptLinkageHeal (step 6) is shared unchanged by both callers. The only remaining default-branch-specific behavior is the fast path itself; everything past it is now identical for both kinds of item. A default-branch item’s log lines read verifyAndHealLinkageByBody: ... whenever the fast path doesn’t confirm — a cosmetic consequence of the shared implementation, not a functional difference.

Review-gate data feed (closed — #1050): This section’s fix addresses linkage verification only. The broader review-gate data feed (LinkedPRReviewRequests, LinkedPRReviews — see §6.1) is nested inside the same closedByPullRequestsReferences GraphQL field and remains structurally empty for base:<branch> repos. That gap is closed by a separate, additive REST-fetch branch inside checkReviewGate (§6.1) — handleBrokenReviewLinkage above now also returns the PR number it resolves, which checkReviewGate reuses to fetch reviews/requested-reviewers directly via FetchPRReviews/FetchPRReviewRequests. LinkedPRReviewThreadComments (the inline per-line comments consumed by review reinvoke) is unaffected — that field remains structurally empty for base:<branch> repos and is out of scope here.


5.8 Merge-Queue Awareness (ADR-058 D3)

When the linked PR’s repository has GitHub’s merge queue enabled, Fabrik’s preemptive rebasing and branch mutations fight the queue: the queue already enforces “up-to-date at merge time,” and any push, rebase, or base-branch change to a PR currently in the queue ejects it. Two guards make every engine-initiated git/PR mutation queue-aware, across all paths (yolo, cruise, manual). Both guards source their signal exclusively from the GraphQL-populated ProjectItem fields — never from e.client.FetchLinkedPR (the REST pulls endpoint does not carry these flags, so it always reports false). The convergence-owner conflict-dispatch site no longer reads settle.PR.IsInMergeQueue inline: D4 (§5.5) consolidates queue membership into the PRMergeQueued settle status, which intercepts an in-queue PR before the conflict branch is reached.

Signal fields:

Guard 1 — per-PR in-queue skip (FR-1). When LinkedPRIsInMergeQueue is true, every branch/PR mutation is skipped. Fires on the in-queue signal alone, regardless of the merge_queue config kill-switch — a PR physically in the queue is queued no matter the config (it may have been queued before the operator flipped the switch). Helper: prInMergeQueue(item).

Guard 2 — per-repo stop-preemptive-rebase (FR-2). When LinkedPRIsMergeQueueEnabled is true and cfg.MergeQueue != "off", the preemptive rebase (behind-but-clean) in updateWorktreeFromMain is suppressed — the queue enforces up-to-date at merge time, so it is redundant. The merge_queue: off kill-switch restores legacy preemptive rebasing (mirrors the D2 enqueue kill-switch). Helper: (*Engine).suppressPreemptiveRebase(item).

FR-2 boundary — preemptive rebase vs. conflict resolution. Preemptive rebasing (behind-but-clean) lives exclusively in updateWorktreeFromMain, which git rebase --aborts on any conflict and never resolves it. Genuine conflict resolution lives exclusively in dispatchRebaseReinvoke, gated on settle.Status == PRMergeConflicting (⟺ mergeable_state == "dirty"). A behind-but-clean PR is "behind", never PRMergeConflicting, so suppressing the former (Guard 2) never suppresses the latter. Conflict resolution is therefore preserved and still runs when a PR is actually conflicting and not in the queue; it is skipped (via Guard 1) only when the PR is in the queue.

Guarded mutation sites (completeness — FR-1):

Site Mutation Guard FR
engine/worktree.go updateWorktreeFromMain (via EnsureWorktree skipUpdate) per-stage git rebase origin/<base> item.go / comments.go OR prInMergeQueue \|\| suppressPreemptiveRebase into skipUpdate FR-1, FR-2
engine/pr.go ensureDraftPR git push --force-with-lease pushBranchForNewPR (in-queue skip only; no zero-ahead guard — no PR exists yet) FR-1
engine/pr.go markPRReady git push --force-with-lease pushBranchUnlessQueued (in-queue skip, then the zero-ahead guard, #2089) FR-1
engine/prcreate.go processPRCreateMarker git push --force-with-lease pushBranchForNewPR (in-queue skip only; runs after the existing-open-PR early return) FR-1
engine/item.go post-stage push; engine/comments.go spec push (#2034) git push --force-with-lease pushBranchUnlessQueued (in-queue skip, then the zero-ahead guard, #2089) FR-1
engine/item.go cancellation WIP push (R8, #1393) git push --force-with-lease pushBranchOnCancel (in-queue skip, then the zero-ahead refusal without the pause, #2089) FR-1
engine/pr.go syncPRBase UpdatePRBase (PR base change) early-return if prInMergeQueue(item) FR-1
engine/catch_up_handlers.go handleMergeAndCIGates dispatchRebaseReinvoke (synthetic rebase + force-push) for an in-queue PR settlePRMergeState returns PRMergeQueued, so checkMergeabilityGate clears mergeConflict and the rebase dispatch is never reached; the inline prInMergeQueue \|\| settle.PR.IsInMergeQueue guard at the dispatch site remains as a defensive backstop FR-1
engine/merge_gate.go checkAutoMergeConvergence dispatchRebaseReinvoke (synthetic rebase + force-push) D4 (landed): the in-queue PR is intercepted by the PRMergeQueued hand-off (step ②) before the conflict branch — a PRMergeConflicting settle is therefore guaranteed not-in-queue, so the former inline conflict-branch guard is removed (dead). After the PR leaves the queue, the conflict branch dispatches the rebase (the ejection→resolve composition); see §5.5 FR-1
engine/stages.go attemptMergeOnValidate MergePR fallback direct merge no new guard — the D2 enqueue early-return (cfg.MergeQueue != "off" && pr.IsMergeQueueEnabled) returns before MergePR is reached on a queue repo; a queued PR is never directly merged FR-1 (audit)

Non-mutating (no guard needed): engine/pr_terminal_advance.go rebase-needed handling is removeRebaseNeededLabel only (label cleanup, no git/PR mutation) and runs only after a PR is already merged/closed.

Skill-side complement (ADR-1364): the guards above cover only engine-initiated mutations. The validate skill’s own Pre-Completion Gate rebase is skill-initiated — it runs inside the worker subprocess via gh/git, with no access to the engine’s ProjectItem snapshot — so none of the guards in the table above reach it. ADR-1364 extends the same queue-awareness principle to that site with an in-session gh api graphql check (plus an up-to-date check and a CI-freshness check comparing the base branch’s commit time against the earliest start time among the PR’s required-check runs, requiring every one of them to be currently successful — start time, not completion time, since a long-running check can finish after the base has moved again, and earliest rather than most recent, since one fresh required check does not vouch for a sibling required check that hasn’t rerun), closing a gap where CI-fix re-invocation or fabrik:revalidate could otherwise push an already-enqueued PR out of its queue.

Scope boundary. These guards cover engine-initiated mutations. Claude-driven git push/rebase inside a stage (the default tool allowlist includes Bash(git:*)) is outside engine control. Stages rarely run while a PR is in the queue (queue entry happens at Validate completion), so the realistic exposure is the rebase-reinvoke synthetic comment — which is guarded at dispatch.

WIP-preservation trade-off. Skipping the post-stage push (item.go) when the PR is in the queue means in-flight WIP is not pushed for that cycle. This is acceptable: queue entry happens at Validate completion, so a stage rarely runs while the PR is queued.

Backward-compat (FR-3). Both signal fields are false-by-default, so on a non-queue repo (LinkedPRIsMergeQueueEnabled == false, LinkedPRIsInMergeQueue == false) every guard is a no-op and all rebase/mutation behavior is byte-for-byte unchanged (the ADR-058 D1 guarantee).

Zero-ahead push guard (#2089, ADR 2089)

An existing worktree’s local fabrik/issue-N branch is never synced from origin/fabrik/issue-N: updateWorktreeFromMain fetches and rebases onto the base branch only. If someone pushes to the issue branch out of band, the next engine push uses a bare --force-with-lease, which compares against the shared bare clone’s refs/remotes/origin/fabrik/issue-N — a ref many other paths refresh — so the lease no longer protects the remote. When the stale local branch carries no commits of its own, its head is the base tip, and the push makes the remote issue branch identical to base; GitHub then closes the PR as having no diff (report #2058).

pushBranchUnlessQueued (engine/merge_queue.go) therefore applies a second guard after the in-queue skip, zeroAheadPushRefused (engine/push_guard.go). It is a safety net, not a resync: it never fetches the issue branch, fast-forwards or resets the local branch, or sets an explicit lease SHA.

Trigger. The push is refused only when all of these hold, checked cheapest first:

  1. the base branch resolves (baseBranchForItem, so a base:<branch> label is honoured);
  2. the worktree’s HEAD is literally zero commits ahead of origin/<base> (WorktreeManager.CommitsAheadOfRef, git rev-list --count, under wm.mu). This is not commitsAheadOfBase: spec-only commits (#921) count as ahead here, because a refusal must be conservative; and
  3. a live FetchLinkedPR read (only when the count is zero) reports an open, unmerged PR.

Fail-open. An unresolvable base, a git error, or a PR read error all let the push proceed (logged as [#N push-guard] warn: …), as do no PR and a closed or merged PR. A branch that is only behind base but carries its own commits is not zero-ahead and pushes.

Refusal. The guard logs [#N push-guard] REFUSING to push …, leaves the remote ref untouched, pauses the item with fabrik:paused + fabrik:awaiting-input through pauseIssue, and posts one 🏭 **Fabrik — push refused: local branch has no commits** comment saying the local branch carries no commits ahead of origin/<base>, that the remote was left untouched, and naming the PR. It does not assert a cause (a rebase that dropped already-absorbed commits also yields zero ahead). Comment dedup follows #1408: if the item already carries that comment (hasPauseComment) the labels are re-applied (reapplyPauseLabels) without a second comment. pushBranchUnlessQueued returns ErrPushRefusedZeroAhead. A human comment after the pause resumes the item (ADR-1813); if the branch is unchanged the guard fires again.

Callers.

Caller Treatment
item.go post-stage push (every stage that ran Claude) Guarded. On refusal processItem releases the lock and returns before the completion/blocked/retry chain, so handleStageComplete cannot strip fabrik:awaiting-input or advance/auto-merge a paused item; the stage stays incomplete.
item.go cancellation WIP push Guarded through pushBranchOnCancel: the push is refused (remote untouched) and logged, but the item is not paused or commented from here — the daemon-stop / TUI-stop interruption that cancelled the worker already pauses it under pauseIssueMu, and a second pause would race it. The lock is released as before.
comments.go publishCommentOutput spec push (#2034) Guarded; a refusal is logged.
pr.go markPRReady Guarded; on refusal it returns before MarkPRReady (a stale PR is not flipped to ready-for-review) and returns true so its callers can act on it. All three callers (item.go post-stage completion, item.go R5 PR-creation retry, comments.go comment-path completion) skip handleStageComplete on true, so the paused item is neither advanced nor auto-merged; the comment path also restores the rework marker state via endStageRework(…, false). TestPushGuard_MarkPRReadyCallersActOnRefusal pins that every call site uses the result.
pr.go ensureDraftPR, prcreate.go processPRCreateMarker Exempt (pushBranchForNewPR, in-queue skip only). They push because no open PR exists yet, so a zero-ahead push (for example a coordinator parent) is legitimate and there is no PR for the overwrite to close.

Known gap. The guard catches only the zero-ahead shape. A stale local branch that carries commits of its own can still overwrite out-of-band work. Syncing the issue branch from origin before work and pushing with an explicit lease on the synced SHA (the logic PrepareCatchUp already has for the train path) is the follow-up, and is expected to subsume this guard.

Test seam. SetPushZeroAheadGuardDisabledForTest neutralises the guard so the refusal tests can show they bite (engine/push_guard_test.go, tests/sim/push_guard_test.go).


6. Review Gate and Review Reinvoke

6.1 Two-Phase Review Gate

The review gate has three paths that handle different timing scenarios:

Path 1: handleStageComplete() (inside worker goroutine)

Path 2: Catch-up loop in poll() (in poll goroutine)

Path 3: reviewGateBlocksLanding() inside attemptMergeOnValidate (landing decision)

Declared unrequested reviewers (expected_reviewers, #1283). Every real-world review bot — Pruefer, Gemini, CodeRabbit, Copilot — is unrequested and asynchronous: it polls or receives a webhook and posts a review without ever being formally requested, so it never appears in ReviewRequest/outstanding. Before #1283, reviewGateAllBots(reviewRequests, outstanding) computed allBots := len(outstanding) > 0, which is unconditionally false whenever nothing was formally requested — the escalation ladder below was therefore unreachable dead code for every self-submitting bot, the exact case it exists to handle; fabrik:bot-reprompted had been applied to zero issues, ever, in production. GitHub has no representation for “a reviewer will turn up here unrequested” (requesting these bots as reviewers is not possible; CODEOWNERS requires collaborator permissions a GitHub App does not hold; inferring from history fails open on a cold-started repo) — the information exists only in the operator’s head, so it must be declared.

Bot-aware escalation ladder:

The ladder engages whenever reviewGateAllBots is true and item.LinkedPRNumber > 0 — either every outstanding requested reviewer is a bot (detected via ReviewRequest.IsBot, unchanged from before #1283), or nothing was requested and at least one declared expected_reviewers identity has not yet responded (declaredOutstanding, new in #1283). checkReviewGate applies a two-phase escalation instead of immediately pausing:

A declared bot never preempts an authoritative human escalation (Finding 2, #1375). At the checkReviewGate call site, the ladder’s allBots is computed as reviewGateAllBots(reviewRequests, outstanding, declaredOutstanding) && authorityReason == "" — not reviewGateAllBots(...) alone. authorityReason (§6.1.1) is only ever non-empty once a requested human has already responded but with a verdict authoritative mode does not accept (outstanding is empty by the time it’s set), so this cannot change behavior for the “human hasn’t responded yet” case — outstanding non-empty already forces allBots=false via the per-reviewer loop below. It only closes the gap where a human has responded with a blocking verdict and a stage-declared bot is separately still outstanding: before this fix, reviewGateAllBots’s declared-reviewer branch (len(outstanding)==0) saw only the bot and returned true, routing the item into the bot re-prompt ladder instead of the human-escalation timeout path — deferring the human’s CHANGES_REQUESTED verdict behind a full ReviewWaitTimeout bot-reprompt cycle that cannot possibly resolve it. reviewGateAllBots’s own signature and pure-function contract are unchanged; the gate is applied at the call site, preserving ADR-1283’s “shared function, minimal call-site divergence” discipline.

Mixed bot+human outstanding: Phase 1 does NOT fire. The gate falls through to the existing pauseForReviewTimeout() at 1× ReviewWaitTimeout from fabrik:awaiting-review, unchanged. Re-prompting humans is not appropriate (they have inbox notifications; webhooks don’t apply).

pauseForReviewTimeout() messaging (FR-4): In all timeout paths (Phase 2, mixed, pure-human), the pause comment names all requested outstanding reviewers and tags each as (bot) or (human) for easy triage, as before #1283. When the stage declares expected_reviewers (non-empty), the message additionally appends a per-declared-reviewer status line — e.g. Expected reviewers: `handarbeit-pruefer` reviewed; `gemini-code-assist` did not respond — built from live review data (resolvedReviewData, the same REST-fallback helper the authoritative-mode verdict messaging uses) via declaredReviewersOutstanding, so a partial response across multiple declared reviewers is diagnosable. This line is unconditional on which timeout branch fired (Phase 2 / mixed / generic), and replaces the old, pre-#1283 message’s generic “no reviewers were requested” framing — which was true but told the operator nothing about why nobody was requested. In Phase 2, the comment additionally notes that a re-prompt was already attempted and provides four recovery options.

ReviewWaitTimeout semantics (depends on outstanding-reviewer mix):

Outstanding reviewers Meaning of ReviewWaitTimeout
Pure bot(s) — requested, declared, or both How long before Phase 1 fires (bot re-prompt); Phase 2 pause triggers at 2× (1× from fabrik:awaiting-review + 1× from fabrik:bot-reprompted)
Mixed bot(s) + human(s) How long before the engine pauses for human input (existing behavior, Phase 1 does not fire)
Pure human(s) How long before the engine pauses for human input (existing behavior, unchanged)
Nothing requested, nothing declared/expected (undeclared stage) Same as pure bot(s) above — FR-5’s unchanged default
Nothing requested, expected_reviewers: [] declared N/A — the FR-2 fast path advances immediately instead

Label lifecycle for bot escalation:

Startup notice for under-specified stages (stages.WarnUndeclaredReviewers, FR-7). On every startup, alongside the existing stage-drift check, Fabrik emits a one-line [startup] notice: for each wait_for_reviews: true stage with ExpectedReviewers == nil (undeclared). This is deliberately not part of drift detection (WarnStageDrift/FilterNoOpKeys), which stays silent by design here — an omitted declaration is a behaviorally-identical no-op under FR-5, so drift correctly has nothing to say about it. The notice answers a different question (“is this config under-specified?”, not “is this config outdated?”) and uses the same warnings.Record/warnings.Clear self-limiting idiom, keyed "undeclared_reviewers:" + stage.Name: informational only, never blocking, and it disappears the moment a declaration — including an explicit expected_reviewers: [] — is added for that stage.

6.1.1 review_authority: Verdict-Aware Clearing (Authoritative Mode)

Everything in §6.1 describes advisory mode — the default, and the only mode that existed before ADR-1250. Advisory clears the gate on existence: len(outstanding) == 0 && hasReviews, regardless of what any review actually says. A CHANGES_REQUESTED review does not, by itself, block advancement; a required reviewer only needs to submit, not approve. This is why advisory mode’s self-review escape hatch (§2.3) works at all: a COMMENTED review from the PR author is existence, not approval, and GitHub permits self-COMMENT while forbidding self-approval. Authoritative mode (below) layers an approval requirement on top of existence — on a repo with no reviewer and no branch-protection review requirement, a self-review still clears authoritative mode’s own-repo fallback verdict check (the “Fabrik’s own fallback” bullet below), since that check only fails on an active CHANGES_REQUESTED, not on the reviewer’s identity.

review_authority: authoritative is a per-stage YAML opt-in (stages.Stage.ReviewAuthority, {"", "advisory", "authoritative"}, default ""/advisory) that makes the same clearing branch additionally verdict-aware. It is orthogonal to Fabrik’s autonomy controls (yolo/cruise/manual) — autonomy answers “does Fabrik proceed without a human clicking go?”; authority answers “what must be true before proceeding is allowed?” See ADR-1250.

Where the check lives. reviewGateAuthorityVerdict(reviewDecision string, reviews []gh.PRReview) (satisfied bool, reason string) (engine/reviews.go) is a pure function called from inside the existing len(outstanding) == 0 && hasReviews branch of both checkReviewGate (Path 2, the advance gate) and reviewGateBlocksLanding (Path 3, the landing gate) — never as a parallel or replacement check. This is strictly additive: a PR with zero reviews stays blocked by the outer advisory condition regardless of mode, and advisory mode’s existing behavior is completely untouched (the new check is gated on effectiveReviewAuthority(item, stage) == "authoritative" and never runs otherwise — zero extra API calls, zero behavior change for advisory/default repos).

Per-issue override — the review-authority:<mode> label (#1261, extending ADR-1250). review_authority is a stage-YAML-only setting by default, applying repo-wide to every issue on that stage. review-authority:advisory/review-authority:authoritative lets a single issue opt into (or out of) authoritative mode without editing stage config, mirroring the established model:<name>/effort:<level>/base:<branch> per-issue override idiom. extractReviewAuthorityOverride(issueNumber int, labels []string) string (engine/reviews.go) scans item.Labels for the review-authority: prefix and validates the suffix against the same {"advisory", "authoritative"} value set stages.go’s validReviewAuthorities enforces at YAML-load time. Precedence follows extractEffortOverride’s “pick deterministically, don’t arbitrate” convention (rank array, prefer the higher-ranked/more-restrictive value), not extractModelOverride’s “first wins” convention: no label → stage config governs; exactly one recognized label → it overrides the stage config for this issue only; both review-authority:advisory and review-authority:authoritative present → resolves to authoritative, with a logged warning; a label whose suffix is neither exactly advisory nor authoritative (typo, casing, unknown value) → ignored with a logged warning, falls back to stage config — never a hard failure, never a silent escalation to authoritative. effectiveReviewAuthority(item gh.ProjectItem, stage *stages.Stage) string composes the extractor with the stage fallback (override, else stage.ReviewAuthority) and is the single resolution point checkReviewGate, reviewGateBlocksLanding, and pauseForReviewTimeout’s authoritative-mode pause message all consult — none of the three reads stage.ReviewAuthority directly anymore, so the advance gate, the landing gate, and the message describing why a pause happened can never disagree about which mode applies to a given issue. The override changes nothing about whether the gate applies (wait_for_reviews: true on the stage is still required for either gate to engage at all) — only the verdict-strictness once the gate is active. See ADR-1261.

Verdict source, preferred then fallback:

  1. GitHub’s reviewDecision, when it reflects a real branch-protection review requirement — one of APPROVED, CHANGES_REQUESTED, REVIEW_REQUIRED (computed server-side, including CODEOWNERS-required approvals and required-approval-count rules — Fabrik does no CODEOWNERS parsing of its own). Satisfied only on APPROVED. REVIEW_REQUIRED covers both “zero reviews yet” and “some, but not enough, approvals” (e.g. a required-approval-count of 2 with only 1 approval so far) — GitHub reports the same value for both, and reviewGateAuthorityVerdict is only reached once hasReviews is already true, so in practice this branch represents the not-enough-approvals case.
  2. Fabrik’s own fallback, when reviewDecision is "" (no branch-protection review requirement configured on the repo) — satisfied unless any review in the caller’s reviews slice has State == "CHANGES_REQUESTED". This exists so authoritative mode is never a silent no-op on a repo without branch protection; without it, opting in on such a repo would be indistinguishable from advisory. On the base:<branch> REST fallback path, reviews comes from FetchPRReviews (github/prs.go), which collapses each author’s full review history down to one entry — critically, a COMMENTED follow-up submission never overwrites a prior formal verdict (APPROVED/CHANGES_REQUESTED/DISMISSED) from the same author during that collapse, so a reviewer who requests changes and later leaves a comment-only follow-up still reads as an active CHANGES_REQUESTED here, matching GitHub’s own review-decision semantics (COMMENTED is informational, not a state transition).
  3. Any other non-empty reviewDecision (an undocumented future GitHub enum value) blocks conservatively rather than falling through to the fallback in (2) — GitHub did report a real verdict, just not one Fabrik recognizes yet, so treating it as “no requirement configured” could wrongly satisfy the gate.

Fetching reviewDecision. FetchPRReviewDecision(owner, repo string, prNumber int) (string, error) (github/prs.go) is a GraphQL-by-PR-number query (repository(owner,name){ pullRequest(number){ reviewDecision } }), mirroring prNodeID’s shape rather than nesting inside closedByPullRequestsReferences — REST has no equivalent field, and this shape works identically for default-branch and base:<branch> items with no ProjectItem schema change, no webhook-delta freshness plumbing, and no caching (boardcache.CacheImpl.FetchPRReviewDecision always delegates, mirroring FetchPRReviews/FetchPRReviewRequests’ “review state is highly volatile” rationale). The call is made only when stage.ReviewAuthority == "authoritative" and the gate has already reached the “would otherwise clear” branch, so the extra GraphQL round-trip is paid only by stages that opt in, and only on the poll where it would matter.

Fetch failure blocks conservatively. A FetchPRReviewDecision error at either gate site does not clear the gate and does not silently fall back to the no-branch-protection computation — it is treated as unknown state, mirroring the existing FetchPRReviews/FetchPRReviewRequests failure handling in both functions. At checkReviewGate this falls through into the existing still-waiting/escalation logic with the reason "review verdict unreadable (fetch failed), blocking conservatively"; at reviewGateBlocksLanding it returns from holdLandingForReview with an equivalent message.

No PR number resolved. If checkReviewGate’s prNumber <= 0 at the point authoritative mode would consult it (linkage not yet resolved), the gate blocks with reason "review verdict unreadable — no PR number resolved" rather than skipping the check.

Non-clear → reinvoke first, pause only at MaxReviewCycles (R1, #1375, amending ADR-1250). review_authority governs merging, never working. A verdict that does not clear (most commonly an unresolved CHANGES_REQUESTED) does not make Fabrik sit idle waiting for a human — handleReviewGate (§6.2) computes buildReviewFeedbackComments() (unresolved inline thread comments plus unaddressed review bodies) before ever consuming checkReviewGate’s blocked/timedOut return, and dispatches a bounded reinvoke whenever that set is non-empty, regardless of blocked state. Per GitHub’s REST contract, a CHANGES_REQUESTED review always carries a non-empty body, so there is always something actionable to reinvoke on — the reinvoke, not a pause, is the primary response to a change request. The reinvoke loop is bounded by the pre-existing MaxReviewCycles/dispatchWithCycleLimit/pauseForReviewCycleLimit machinery (§6.2), plus a non-convergence terminal check independent of any single poll’s dedup outcome (R1/R2, ADR-1518): the cycle-limit pause fires whenever the gate is still blocked/timedOut and max(ReviewCycles, ReviewBlockedCycles) has reached MaxReviewCycles — including on a poll where the currently-outstanding feedback happens to have already been deduped as processed (snap.CommentProcessed), which is exactly the steady state immediately after the MaxReviewCycles-th reinvoke and was previously unreachable (#1518). pauseForReviewTimeout remains reserved for the genuinely different case: nothing actionable to reinvoke on at all — a plain “nobody has responded yet” block, which can only have a structurally-zero cycle count (a reinvoke can only ever fire in response to actual review content), so the two pause reasons cannot collide. See ADR-1375, which amends this ADR-1250 consequence, and ADR-1518 for the terminal-check/ReviewBlockedCycles mechanism.

Messaging distinguishes “no reviews yet” from “reviews exist but the verdict blocks.” checkAwaitingReviewTimeout takes an authorityReason parameter (non-empty only when the authoritative check blocked) and uses it verbatim as both the wait-log reason and the timeout-pause reason, instead of the generic "no reviews submitted yet" / "pending reviewers" text — which would otherwise misleadingly suggest nobody has reviewed at all. pauseForReviewTimeout’s standard (non-Phase-2) message additionally live-fetches the verdict and reuses reviewGateAuthorityVerdict itself when authoritative mode is on, appending its reason string verbatim (e.g. "no branch-protection review requirement; bob requested changes" — the reviewer’s login comes from reviewGateAuthorityVerdict’s own fallback-branch message, not a separate helper) — so this line can never disagree with why the gate is actually blocking, and covers REVIEW_REQUIRED/fetch-failure cases too, not just an active CHANGES_REQUESTED review. item.LinkedPRNumber/item.LinkedPRReviews are structurally empty for base:<branch> items (same gap as the existing pendingLine), so this resolves the PR and its reviews via the same FetchLinkedPR/FetchPRReviews REST fallback checkReviewGate/reviewGateBlocksLanding use, rather than omitting the line there.

yolo/cruise never bypass the gate — no new branching needed. Because the check lives inside the shared clearing predicate rather than at attemptMergeOnValidate/handleStageComplete’s yolo/cruise decision points, authoritative + yolo “waits” for free: attemptMergeOnValidate simply never reaches its merge/enqueue logic while reviewGateBlocksLanding returns true, exactly like an outstanding-reviewer block today. Once the verdict clears (a fix pushed → stale review dismissed on push → re-review → APPROVED, or a required approval lands), yolo merges immediately with no human click required — the timing is still automatic, only the gate has tightened. cruise composes identically at both layers: hasCruiseLabel is checked before reviewGateBlocksLanding is ever consulted (§1’s cruise-wins-over-yolo precedent — cruise is checked on the raw label, not cruiseActive — applies unchanged here), so a cruise item never even reaches the authoritative check — cruise’s “leave the PR for human merge” behavior is unaffected by review_authority.

Unit tests: TestReviewGateAuthorityVerdict (engine/reviews_test.go) is the pure-function table (APPROVED/CHANGES_REQUESTED/REVIEW_REQUIRED/empty-with-clean-reviews/empty-with-CHANGES_REQUESTED/empty-with-DISMISSED-CHANGES_REQUESTED). TestCheckReviewGate_Authoritative_* and TestCheckReviewGate_NonDefaultBase_Authoritative_* cover the advance gate across reviewDecision × branch-protection-present/absent × default-branch/base:<branch> × fetch-error. TestAttemptMergeOnValidate_Authoritative_* covers the identical matrix at the landing gate, plus TestAttemptMergeOnValidate_Authoritative_CruiseSkipsAutoMerge (cruise composition) — attemptMergeOnValidate’s own yolo-vs-advisory tests (TestAttemptMergeOnValidate_YoloEnablesAutoMerge etc.) already exercise the yolo path unconditionally, since the function has no yolo-specific branching to begin with. TestCheckReviewGate_Advisory_ChangesRequestedReview_StillClears and TestAttemptMergeOnValidate_Advisory_ChangesRequestedReview_StillMerges are explicit non-regression controls: advisory mode clears/merges regardless of a CHANGES_REQUESTED verdict, and FetchPRReviewDecision is asserted never called. For the per-issue label override (#1261): TestExtractReviewAuthorityOverride and TestEffectiveReviewAuthority (engine/reviews_test.go) cover the extractor and resolution-helper precedence matrix directly (no label, single label, both labels, malformed suffix). TestCheckReviewGate_LabelOverride_* and TestAttemptMergeOnValidate_LabelOverride_* (engine/reviews_test.go/engine/stages_test.go) re-run the label-authoritative-on-advisory-stage and label-advisory-on-authoritative-stage cases at both gate sites, plus TestAttemptMergeOnValidate_LabelOverride_Authoritative_CruiseSkipsAutoMerge/_YoloWaitsForGate confirming yolo/cruise compose with a label-resolved gate exactly as they do with a stage-YAML-resolved one.

6.2 Review Reinvoke Mechanics

The catch-up loop in poll() is split into two phases for every non-paused non-cleanup item that has either stage:<X>:complete OR fabrik:awaiting-ci (on a wait_for_ci: true stage):

Phase 1 — unconditional, ordered handler list (catchUpPhase1Handlers):

Phase 1 is implemented as an explicit ordered slice of named handler methods (catchUpPhase1Handlers in engine/catch_up_handlers.go). Each handler returns true to claim the item — Phase 2 is skipped for this item this cycle — or false to pass through to the next handler. Ordering is structurally enforced by slice position (ADR-056 D3); a future reorder requires a deliberate code change and triggers a test failure, rather than occurring as an accidental side effect of a continue insertion.

Handler 0: handleEngineUnpause (#1460 — numbered 0, not renumbering Handlers 1-4 below, to avoid an invasive cross-reference cascade) — runs first, unconditionally, ahead of every other Phase 1 handler. It is the actual resume trigger for the four cycle-limit pause sites (pauseForReviewCycleLimit, pauseForCIFixCycleLimit, pauseForRebaseCycleLimit, pauseForEnqueueCycleLimit; see §7.2 “Resumable Engine Pauses”): checks snap.PausedByEngine(stage.Name); if true — meaning the item reached this handler chain (so fabrik:paused is currently absent, per both the main catch-up loop’s and settleAwaitingCIScan’s admission filters) while still carrying a stale itemstate.EnginePaused record from before the label was removed — it calls clearFailedStage(), the same reset processItem’s own unpause gate already uses (zeroing ReviewCycles/CIFixCycles/RebaseCycles/EnqueueCycles, clearing Attempts, resetting LastAttemptAt, and resetting the comment circuit breaker). Always returns false — it only resets state, never claims — so the rest of the chain evaluates the just-reset counters in the same pass, letting a genuine resume converge in one poll rather than two. See ADR-1460 for why this handler exists at all (processItem’s pre-existing unpause gate is structurally unreachable for these four sites — they only ever fire post-stage-completion, where itemNeedsWork never re-dispatches processItem).

Handler 1: handleDependencies — thin wrapper around checkDependencies(). If the item has unresolved blocking dependencies, claims the item (returns true); otherwise passes through. checkDependencies’s own bool return is the Phase 1 claim signal directly (no translation layer, unlike the review/CI gates below) — so its cycle-detection branch (a bounded BFS finds the item is itself a transitive blocker of one of its own blockers) also returns true when it pauses the issue via pauseIssue (ADR-1223), rather than reusing the false value returned for “no open dependencies.” Before this fix the cycle-detection pause fell through unclaimed, the same overloaded-return bug present in the review/CI gates.

Handler 2: handleReviewGate — its gate half (checkReviewGate: the fabrik:awaiting-review write and the reviewer timeout) is only active when stage:<X>:complete is present (pctx.hasComplete == true; skipped during the CI-await window when fabrik:awaiting-ci is present but stage:X:complete is absent — prevents spurious fabrik:awaiting-review re-application from stale board data; #617). Since #1953 (R1) its feedback-detection and review-reinvoke dispatch half runs in every state: with !hasComplete it resolves reviews itself (resolveReviewsForFeedback), passes blocked = timedOut = false, writes no label, and returns false when nothing was dispatched so the CI gate still runs; and a completed stage without wait_for_reviews now also resolves reviews, so its review bodies are dispatched (before, checkReviewGate’s early return handed back no reviews and only threads were). Its review data comes from the item the poll admitted, i.e. the board cache, while the feedback gate reads live; a gate hold on feedback this handler cannot see is repaired by invalidating the cache (§6.6.6 “Dispatcher input agreement”, #2160). Because feedback now holds an advance or landing (§6.6.6), a reinvoke dispatched while feedback stands between the item and its advance (feedbackHoldsProgress: stage incomplete, or yolo/cruise/auto_advance would advance it) is also charged to the never-refunded ReviewBlockedCycles counter, so a reviewer producing fresh non-actionable bodies forever still ends in pauseForReviewCycleLimit (ADR-1953-a):

Handler 3: handleAutoMergeConvergence — if the item does not have fabrik:auto-merge-enabled, passes through. If it does: calls settlePRMergeState() to build a PRSettleResult, then forwards to checkAutoMergeConvergence() which consumes the settle result for PRMergeUnsettled/PRMergeConflicting branch decisions. Claims the item (returns true). The merge/CI gates remain bypassed — GitHub owns the merge decision for these items — but checkAutoMergeConvergence() no longer independently interprets mergeable_state or calls FetchPRMergeableFields/FetchCheckRuns.

Handler 4: handleMergeAndCIGates — calls settlePRMergeState() once (the settle call shared by both gates), then the merge-conflict gate, then the CI gate. Merge runs before CI per ADR-028: a PR made unmergeable by a base-branch advance must be rebased before the engine spins on CI-await polls. See §6.4 for the full settling primitive specification.

After Phase 1 + Phase 2 — single-owner Validate advance (runValidatePRTerminalAdvance, R4 — ADR-056 D2, ADR-057): A single authoritative scan iterates deepFetchCandidates for all Validate-stage items not in the fabrik:auto-merge-enabled convergence flow, regardless of which gate label (fabrik:awaiting-ci, fabrik:awaiting-review, fabrik:rebase-needed, or any future label) is present. For each, e.client.FetchLinkedPR is called directly (not e.readClient — boardcache may have stale state). When pr.Merged == true: iterates all pipeline stages in ascending Order from the stage after the highest already-complete stage up to (but not including) the cleanup-terminal stage, adding stage:<N>:complete for every WaitForCI or WaitForReviews gate-checked stage whose label is absent (with cache write-through; fail-fast on error for idempotent retry). After all labels are added, removeAwaitingCILabel, removeAwaitingReviewLabel, and removeRebaseNeededLabel are called as applicable, fabrik:paused + fabrik:awaiting-input are removed, and advanceToNextStage() is called. Items already in advancedItems are skipped (prevents double-advance). This self-heals Validate-stage items merged externally regardless of which gate was active, without requiring a manual unpause. The function never dispatches workers or acquires e.sem — label-mutation-only path (see ADR-053 constraints; superseded in structure by ADR-057).

Phase 2 — gated (yolo/cruise/auto_advance only):

processComments widening: processComments() itself also merges any unresolved LinkedPRReviewThreadComments at entry, before Step 1. This closes the race where a user nudge arrives before the catch-up loop Phase 1 fires — the review thread comments are addressed in the same invocation as the nudge comment.

Bot service-notice chokepoint (#1221): immediately after that merge, and before Step 1, processComments() filters the fully-assembled comments slice through isBotServiceNotice (filterBotServiceNotices, §4.1 signal 4) — the single point every comment reaching processComments() passes through regardless of source (findNewComments(), the LinkedPRReviewThreadComments merge above, or a reinvoke dispatcher’s build() output). If the filtered result is empty, processComments() returns immediately, before any reaction, label, worktree setup, or Claude invocation — mirroring the claudeUsageLimitedUntilTime early-return earlier in the function. This closes a review-thread bypass: bot reviewers (Gemini, CodeRabbit) post non-actionable service notices via inline review comments, not only via issue comments, and buildReviewThreadComments() (used by both dispatchReviewReinvoke’s precheck and the merge above) filters only ROCKET reaction and CommentProcessed — it does not itself classify bot notices. Left unfixed by design: dispatchReviewReinvoke’s precheck and the catch-up loop’s ReviewCycles increment (§6.2 Handler 2) still use the unfiltered buildReviewThreadComments() count, so an un-watermarked, notice-only review thread still dispatches a worker each poll (immediately no-op’d by this chokepoint) and still consumes a review cycle — self-limiting via MaxReviewCycles/pauseForReviewCycleLimit rather than silently free. Watermarking bot notices within LinkedPRReviewThreadComments (extending settleBotServiceNoticesForItem, currently scoped to item.Comments only) is a deliberate follow-up, not required to close the processing risk this chokepoint addresses.

Why #1045’s no-op cycle exemption (§6.2 Handler 2) does not subsume this case. That exemption compares gitHeadSHA(workDir) before and after the reinvoke, and this chokepoint returns from processComments() at the point quoted above — before wm.EnsureWorktree ever runs. There is therefore no worktree to read a HEAD from at either snapshot point; gitHeadSHA fails identically before and after, and dispatchReviewReinvoke’s headBefore != "" guard means no decrement is ever applied. This is a structural consequence of the two mechanisms’ relative ordering, not a deliberate carve-out re-litigating this paragraph’s own trade-off — it happens to leave this paragraph’s documented cost exactly as it was.

Review thread resolution: Step 10 of processComments() resolves addressed review threads via ResolveReviewThread() after adding ROCKET reactions.

PR summary comment: Step 8b of processComments() posts a Fabrik-marked "<StageName> (review feedback addressed)" comment on the linked PR when the invocation is a review-reinvoke (all-ReviewThreadID batch) and output != "". The comment includes Claude’s cleaned output plus a machine-generated per-thread footer listing path:line — resolved for each unique ReviewThreadID in the input batch (deduped; line resolved from Comment.Line with OriginalLine fallback). This gives reviewers a visible record in the PR timeline that their inline feedback was addressed.

6.3 Review Reinvoke vs Regular Comment Processing

Aspect Regular Comments Review Reinvoke
Source item.Comments (issue comments) item.LinkedPRReviewThreadComments (PR inline comments)
Detection findNewComments() buildReviewThreadComments()
Dispatch Synchronous in processItem() Async goroutine via dispatchReviewReinvoke()
Cycle limits None MaxReviewCycles (default 5)
Timeout None Integrated with ReviewWaitTimeout
Thread resolution Yes — processComments() merges unresolved LinkedPRReviewThreadComments at entry, so a user nudge resolves threads in the same invocation Yes — resolves review threads after processing
PR summary posting None Posts "<StageName> (review feedback addressed)" on the linked PR with per-thread footer (one path:line — resolved bullet per unique ReviewThreadID); skipped when output == "" or no linked PR
Worker guard Uses dispatch loop’s snap.Worker() != nil Has its own snap.Worker() != nil check in catch-up loop

6.4 PR Merge State Settling Primitive

settlePRMergeState() is called once per catch-up loop Phase 1 iteration to read mergeable, mergeable_state, and check runs in a single pass. It is called from two handlers:

In both cases, this eliminates the split-brain where two separate REST calls within one poll cycle could observe different GitHub state.

Return type: PRSettleResult carries:

Seven status constants:

Constant Meaning
PRMergeNoPR No linked PR on this item; both gates are no-ops
PRMergeUnsettled GitHub has not yet computed a definitive merge state; gates block without label churn
PRMergeReady PR is ready to merge (all checks green under rules 13–19 below, or mergeable_state == "clean" via the rule 9 fast path — ADR-1441 narrowed this from {clean, unstable})
PRMergeConflicting mergeable == false — confirmed base-branch conflict
PRMergeBlocked CI checks have failed; checkCIGate dispatches dispatchCIFixReinvoke()
PRMergeTerminal PR is merged or closed without merging; both gates handle terminal state
PRMergeQueued PR is in GitHub’s native merge queue (ADR-058 D4) — a transient hand-off: the queue owns the PR and will merge or eject it. Both gates block like PRMergeUnsettled (no conflict, no label churn); the convergence monitor waits. The single canonical queue-membership signal (ADR-056 D1), evaluated strictly after the terminal check (the #913 merged-first invariant: a PR that merged via the queue is terminal, never “queued”).

Settling rules (applied in order):

  1. FetchLinkedPR() — if no linked PR: return PRMergeNoPR.
  2. PR merged (pr.Merged == true): return PRMergeTerminal.
  3. PR closed, not merged (pr.State == "closed"): return PRMergeTerminal (re-confirmed via the authoritative single-PR FetchPRMerged endpoint, since the list endpoint reports merged == false for several seconds after a queue merge — the #913 trap).
  4. PR in merge queue (prInMergeQueue(item) || pr.IsInMergeQueue || pr.MergeQueueEntry.State ∈ {QUEUED, AWAITING_CHECKS}): return PRMergeQueued. Evaluated after the terminal check (rules 2–3, the merged-first invariant) and before mergeable/CI classification — a queued PR is a transient hand-off, so the gates never interpret its mergeable_state. Dual-sourced from the GraphQL-authoritative ProjectItem field (reliable on every poll) and settle.PR/MergeQueueEntry, so it holds on a boardcache miss (the REST FetchLinkedPR fallback reports the flag as false).
  5. FetchPRMergeableFields() — fetches mergeable (bool ptr) and mergeable_state (string) in one REST call (single-PR endpoint; the list endpoint used by FetchLinkedPR does not return mergeable). API error → return PRMergeUnsettled.
  6. mergeable == nil (GitHub still computing): return PRMergeUnsettled (MergeableState set).
  7. mergeable_state == "unknown" (transient): return PRMergeUnsettled (MergeableState set).
  8. mergeable == false: return PRMergeConflicting (MergeableState set).
  9. mergeable_state == "clean" (ADR-033, narrowed to clean-only by ADR-1441/#1441 — previously mergeable_state ∈ {clean, unstable} per the MergeableStateAccepted allowlist): return PRMergeReady — GitHub reports no required check failing or pending. This is not proof every job ran (#1822): a job still queued for a runner has no check run, so GitHub reports clean for a green prefix. FetchCheckRuns is not called, but since #1822 the shortcut first consults the suite hold (see Suite-aware completeness below); a held suite returns PRMergeUnsettled instead. An empty HeadSHA skips the shortcut and falls to rule 10. mergeable_state == "unstable" no longer matches this rule — it falls through to rules 10–19 below like any other non-clean value (e.g. "blocked"), so a confirmed failure on a non-required check is caught by rule 19 instead of being silently accepted here.
  10. HeadSHA empty: return PRMergeUnsettled (MergeableState intentionally omitted — see invariant).
  11. FetchCheckRuns() — API error → return PRMergeUnsettled.
  12. Check runs non-empty: apply PRChecksObserved store event (sets HasHadChecks).
  13. Check runs empty + a configured required context confirmed-failed (ADR-933): classifyRequiredContexts() is consulted here, unconditionally ahead of rules 14–16 below, and returns RequiredContextsFailed → return PRMergeBlocked (RequiredContextsStatus/RequiredFailed set). This ordering is deliberate: a confirmed required-context failure (e.g. a classic commit status with no check-run footprint at all — the local-CI-takeover case #933 was filed for) is a definitive, timing-independent signal, and rules 14–16 exist only to wait out transient GitHub propagation delays for check-run-based CI. Placing the failure check after them (as an early draft of this fix did) let the R3 mergeable_state branch (rule 15, which fires for almost any non-empty mergeable_state, e.g. "blocked" — exactly the local-CI-takeover case) mask the failure as generic PRMergeUnsettled before it was ever reached. A repo with no required_status_contexts configured always resolves Satisfied here (no-op, zero behavior change).
  14. Check runs empty + hadChecks (store: LinkedPRState.HasHadChecks): return PRMergeUnsettled (MergeableState intentionally omitted — see invariant).
  15. Check runs empty + post-push dwell active (LastHeadSHAUpdate non-zero and time.Since < PostPushDwell, default 90 s): return PRMergeUnsettled (MergeableState intentionally omitted — see invariant). Zero LastHeadSHAUpdate (cold start / post-restart) falls through.
  16. Check runs empty + mergeable_state ∉ {"", "unknown"}: return PRMergeUnsettled (MergeableState set — R3/branch-protection signal for checkCIGate).
  17. Check runs empty + a configured required context not yet confirmed satisfied (ADR-933): classifyRequiredContexts() is consulted again; a RequiredContextsPending result (missing entirely, or reported skipped/neutral/still-pending) → return PRMergeUnsettled (RequiredContextsStatus/RequiredMissing/RequiredPending set) instead of falling through to rule 18’s “no CI configured” — nothing has regressed (so no CI-fix reinvoke fires from this alone; see classifyCIFromRequiredContexts in engine/ci.go), it simply hasn’t reported yet.
  18. Check runs empty + mergeable_state ∈ {"", "unknown"} + required contexts satisfied (or unconfigured): return PRMergeReady (no CI configured) — but only if the suite hold (below) is clear: a suite reporting runs, or a run-less suite younger than the post-push dwell, contradicts “no CI” and returns PRMergeUnsettled instead (#1822).
  19. Check runs non-empty: classify via github.ClassifyCheckRuns(checkRuns) (#958) — reduces to the latest run per check name (highest ID wins), then: any pending (queued/in_progress) at any name → return PRMergeUnsettled (CheckRuns set) regardless of any failed run, whether that failed run is a different check name or a stale (lower-ID) entry for the same name superseded by the pending rerun; else any failed (failure/timed_out/action_required) → return PRMergeBlocked (CheckRuns set); else all green: before returning PRMergeReady (ADR-933), classifyRequiredContexts() is consulted a third time (fetching classic commit statuses via FetchCombinedStatus only when at least one required name isn’t already resolvable from the present check runs) — a non-Satisfied result returns PRMergeBlocked/PRMergeUnsettled instead (same Failed→Blocked, Pending→Unsettled mapping as rules 13/17), closing the exact hole #933 was filed for: an all-skipped/neutral check-run set on the head must never resolve to PRMergeReady when a required context hasn’t confirmed success. Only once required contexts are satisfied (or unconfigured) and the suite hold (below) is clear (#1822) does this rule return PRMergeReady (CheckRuns set); a held suite returns PRMergeUnsettled with CheckRuns set and MergeableState omitted. Pending-over-failed (within ClassifyCheckRuns itself) is still evaluated first, reversing the pre-#958 order where a failed run short-circuited to PRMergeBlocked before pending was even checked.

MergeableState omission invariant: checkCIGate uses settle.MergeableState to detect R3 (OPEN+BLOCKED+no-check-runs+fabrik:awaiting-ci elapsed → pauseForRequiredNeverRunningCheck). The primitive omits MergeableState from the PRSettleResult in the hadChecks, post-push dwell, and HeadSHA-empty cases. This prevents R3 from misfiring on cases where hadChecks == true (checks have been observed) or where GitHub simply hasn’t computed mergeability yet (post-push window). Only a non-empty MergeableState in the settle result is genuinely relevant for R3/branch-protection timeout checking.

Startup failure (#2052). settlePRMergeStateForCIGate (the form handleMergeAndCIGates calls; settlePRMergeState itself never retriggers) additionally treats a PR head with zero check runs and a startup_failure workflow run as CI infrastructure for a wait_for_ci stage — inside the clean shortcut (read the cached check runs first) and ahead of the hadChecks/dwell/R3 returns. Retrigger/wait return PRMergeUnsettled with MergeableState omitted; abandon sets PRSettleResult.StartupFailure. See §6.29.

Suite-aware completeness (#1822, ADR-1822). A check run exists only once its job has been scheduled onto a runner, so a needs:-dependent job (or one queued for a busy runner) is invisible to FetchCheckRuns, and an all-green prefix of a workflow reads as a complete pass — while GitHub itself reports mergeable_state == "clean" whenever no required check is outstanding. The check suite is created at queue time and stays non-completed until every job has run, so it is the signal missing from the run set. settlePRMergeState therefore consults ciSuiteHold (engine/ci_suites.go) at every point that would otherwise return PRMergeReady: rule 9, rule 18 and rule 19. It is deliberately consulted last and only on a would-be-green path, so a confirmed failure (PRMergeBlocked), a pending run, or an unsatisfied required context (ADR-933) is never masked, and a red or pending SHA does not pay for the extra call.

FetchCheckRuns consolidation: Two callers consume settle.CheckRuns from the PRSettleResult instead of making independent FetchCheckRuns calls:

The base-branch FetchCheckRuns(baseSHA) inside buildCIFixComment() is a non-gate diagnostic read — it fetches check runs for the base branch’s HEAD SHA for regression-vs-pre-existing classification only, and is structurally independent (different SHA; result is not used by either gate).

Stale-cache self-healing on webhook-less deployments (#958 leg 3): settlePRMergeState()’s step 11 FetchCheckRuns() call routes through e.readClient, which on a live deployment is boardcache.CacheImpl.FetchCheckRuns(). Two additive fixes close the “cached failure never refreshes” gap:

Together these mean a genuinely-stuck failure still reaches PRMergeBlocked (the live refetch confirms it), but a stale failure whose live status has since gone green or pending self-heals on the next poll without operator intervention.

6.5 CI Gate and CI-Fix Reinvoke

Channel events (#1968): pauseForCITimeout (a fresh CI-gate timeout pause) emits ci-timeout; the CI-fix cycle counter reaching one short of its limit emits cycle-limit-near; see §7.14.

6.5.1 Two-Phase CI Gate

The CI gate has two paths that handle different timing scenarios:

Path 1: attemptMergeOnValidate() (Merge Guard)

Path 2: settleAwaitingCIScan() (checkCIGate())

Clearing-owner invariant for wait_for_ci: true stages: Exactly two code paths may add stage:X:complete for these stages; they are mutually exclusive by PR state at the time of evaluation:

  1. Normal path (addCompleteLabelAndRemoveCI, called from checkCIGate): PR is still open; CI checks clear (or R5 — no CI configured). Runs inside handleMergeAndCIGates, invoked from either the main catch-up loop’s Phase 1 (for items with stage:X:complete already present — a narrow re-evaluation window, not the common case) or settleAwaitingCIScan (Path 2 above; the common case, since open awaiting-ci items are routed there exclusively as of #1270). Both call sites reuse the same handleMergeAndCIGates/checkCIGate code — settleAwaitingCIScan is a second call site, not a second owner: it never mutates stage:X:complete or fabrik:awaiting-ci directly, only through this shared function.
  2. PR-merged recovery path (runValidatePRTerminalAdvance, ADR-056 D2): PR is already merged. Adds stage:X:complete for every gate-checked stage (WaitForCI or WaitForReviews) missing its completion label. Runs only for Validate-stage items without fabrik:auto-merge-enabled. See §3.2 (“Awaiting CI” table, R4 row).

Both paths call EnsureLabel, which is idempotent. The advancedItems map (keyed by issue number, populated by advanceToNextStage) prevents double-advancement within a single poll cycle even if both paths fire in the same poll. This supersedes the original ADR 032 single-owner claim; the two-path structure is captured by ADR-056 D2.

Two different timeout strategies:

6.5.2 CI Fix Reinvoke Mechanics

The catch-up loop Phase 1 calls checkCIGate() after the review gate check. When CI has failed:

  1. Worker guard (snap.Worker() != nil): If a CI-fix goroutine from a previous poll is still running for this item, skip dispatch entirely (no further checks)
  2. No-op debounce guard (snap.LastCIFixNoOpSHA() == settle.PR.HeadSHA, #958 leg 2): If the most recent CI-fix reinvoke for this exact head SHA already observed no new commit pushed, skip dispatch entirely (no cycle-limit check, no increment) — dispatching again would just repeat the same no-op and burn cycle budget for nothing. The guard is keyed to the current head SHA, so it clears implicitly the moment a genuine fix advances HeadSHA (before it applies, ciFlakeRerun re-runs the failed jobs once for that head — #2072 R2, §6.29); CIBackstopTimeout remains the backstop if CI never resolves on the stuck SHA (ADR-1410).
  3. Cycle limit check: snap.CIFixCycles(stage.Name) is compared against MaxCiFixCycles (default 5)
    • If exceeded: pauseForCIFixCycleLimit() adds fabrik:paused + fabrik:awaiting-input and posts comment
    • If not exceeded: increment count, dispatch reinvoke via dispatchCIFixReinvoke():
      • Applies WorkerEntered (prevents double-dispatch)
      • Acquires semaphore slot (respects MaxConcurrent)
      • Snapshots gitHeadSHA(workDir) before reinvoking (headBefore) — the same git rev-parse HEAD helper used by the Implement/Review turn-extension progress check (§ detectProgress)
      • Calls buildCIFixComment() to construct a synthetic gh.Comment (DatabaseID: 0) with a structured CI failure report — uses settle.CheckRuns for the PR-head check runs (no separate FetchCheckRuns call); classifies each failed check as NEW REGRESSION (not failing on base branch) or pre-existing (also failing on base branch) by calling FetchCheckRuns(baseSHA) for the base-branch HEAD SHA (a non-gate diagnostic read — different SHA, independent of the settle result)
      • Failure context (#1991, ADR-1991): for each NEW REGRESSION check, ciFailureDetail() appends the check’s failure detail beneath its line — up to 5 non-empty annotations (live GET /check-runs/{id}/annotations?per_page=50, first page only, via the optional checkRunAnnotationFetcher interface asserted on e.client — never the board cache; Actions’ generic “Process completed with exit code N.” annotation is shown only when nothing more specific exists), then any output.summary/output.text the check set — each capped at ~500 chars, ~2000 chars per check (truncateMiddle). Annotations are the only channel that carries an Actions job’s ::error:: text to the API ($GITHUB_STEP_SUMMARY never populates output.summary, #916). Every failure here is soft: a fetch error is logged ([#N ci-fix-reinvoke] warn: …) and the prompt degrades to the check name and conclusion; a client without the capability (e.g. simgh) yields no annotations. Pre-existing failures get no detail (the prompt tells the agent not to fix them).
      • Prompt log line (#1991): the synthetic comment is never posted to GitHub, so the build hook logs it — [#N ci-fix-reinvoke] prompt (<n> bytes): "<%q-rendered body>" — bounded to 4 KB (truncateMiddle) and one line. The live TestCIFixReinvoke asserts the line carries the failing check’s name and the run-ID nonce. A log line is not a charged cycle (ADR-1812 refunds a did-not-run reinvoke), so tests must not count lines as cycles.
      • Calls processComments() with the synthetic comment and the ci_fix_skill (falls back to comment_skill if unset)
      • After processComments() returns, snapshots gitHeadSHA(workDir) again (headAfter). If headBefore == headAfter (and both are non-empty — no git-command failure), applies CIFixNoOpRecorded{SHA: headAfter}, populating LinkedPRState.LastCIFixNoOpSHA for the guard above to consult on the next poll
      • On exit: releases semaphore, applies WorkerExited

DatabaseID: 0 guard: Synthetic CI-fix comments have DatabaseID: 0 and no ReactionNodeID, which skips the 👀 and 🚀 reaction steps in processComments() (reactions require a real GitHub comment ID or, for review bodies only, a review node ID — #1953 R8).

CI-fix cycle counter reset: StageState.CIFixCycles[stageName] is reset to 0 by clearFailedStage() (via EngineCyclesCleared mutation) when the user removes fabrik:paused from a paused-failed item, allowing fresh CI-fix attempts after human intervention. StageState.RebaseCycles[stageName] is reset in the same call for the same reason. LinkedPRState.LastCIFixNoOpSHA is not cleared by this reset — it is scoped to a specific SHA and self-obsoletes once the head SHA changes, so a stale no-op record from before the pause cannot block a fresh CI-fix attempt on a since-advanced head.

6.5.2a Startup Failure on the Gate (#2052)

For a wait_for_ci stage whose PR head has zero check runs, settlePRMergeStateForCIGate also reads the workflow runs; a startup_failure run is retriggered by close/reopen (at most twice) and then escalated through pauseForCIStartupFailure — a pause naming the startup-failed run — instead of waiting for the CI backstop, clearing the gate with CI never run (mergeable_state: clean), or ending in the R3 “required check never runs” pause. Full description in §6.29.

6.5.3 CI Fix Reinvoke vs Review Reinvoke

Aspect Review Reinvoke CI Fix Reinvoke
Trigger Unresolved PR review thread comments CI check runs with failure/timed_out/action_required conclusion
Source data item.LinkedPRReviewThreadComments settle.CheckRuns from settlePRMergeState() (no separate REST call for PR-head checks)
Label on waiting fabrik:awaiting-review (always applied) fabrik:awaiting-ci (applied by handleStageComplete on FABRIK_STAGE_COMPLETE; present for both pending and failed checks — covers the full CI-await window)
Timeout tracking In-memory ReviewWaitTimeout timer FetchLabelAppliedAt on fabrik:awaiting-ci (durable across restarts; label is present from FABRIK_STAGE_COMPLETE onwards)
Cycle counter snap.ReviewCycles(stageName) / ReviewCycleIncremented snap.CIFixCycles(stageName) / CIFixCycleIncremented
Max cycles MaxReviewCycles (default 5) MaxCiFixCycles (default 5)
Skill comment_skill ci_fix_skill (falls back to comment_skill)
Synthetic comment PR review thread text Structured CI failure report with NEW REGRESSION classification
Worker guard Yes — snap.Worker() != nil Yes — snap.Worker() != nil
Thread resolution Yes — ResolveReviewThread() after processing No
PR summary comment Yes — "<StageName> (review feedback addressed)" on linked PR No
Stage gate config wait_for_reviews: true wait_for_ci: true

6.6 Merge-Conflict Gate and Rebase Reinvoke

The merge-conflict gate is a third prong of the catch-up loop Phase 1, sitting between review reinvoke and the CI gate. It is the direct response to the failure mode in which a base-branch advance during the CI-await window leaves a PR unmergeable — the CI gate alone will happily keep polling check runs on the branch head while the real blocker is a conflict.

Note: attemptMergeOnValidate() (§5.4, the legacy auto-merge path for stages without wait_for_ci: true) now uses the same snap.RebaseCycles(stage.Name) + RebaseCycleIncremented, dispatchRebaseReinvoke(), and pauseForRebaseCycleLimit() pattern as the conjunctive gate described in this section. Both paths share the same per-item store field; MaxRebaseCycles applies to both.

Independence from MergePR’s CI precondition (ADR-933): the rebase-cycle machinery in this section is driven exclusively by settle.Status == PRMergeConflicting (a confirmed base-branch conflict, mergeable == false). MergePR’s own mergeable_state self-gate (§5, “MergePR’s own CI precondition”) is a separate, later check inside MergePR itself — a refusal there (gh.ErrNotMergeableCI, e.g. a required check still blocked) is not a conflict and is never observed by checkMergeabilityGate or checkAutoMergeConvergence. No call site reads MergePR’s return value to apply fabrik:rebase-needed or increment RebaseCycles, so a CI refusal cannot consume a rebase cycle or trigger pauseForRebaseCycleLimit.

6.6.1 Gate Mechanics

checkMergeabilityGate() runs only when stage.WaitForCI is true (the same opt-in that admits items to the catch-up window via fabrik:awaiting-ci). It returns (blocked, conflict):

The gate’s inputs come from settlePRMergeState() (§6.4), called once per Phase 1 iteration before both gates. FetchPRMergeableFields() (single-PR endpoint, not the list endpoint used by FetchLinkedPR) provides both mergeable and mergeable_state in one request — the list endpoint does not return mergeable.

Clearing-owner invariant for fabrik:rebase-needed: Four code paths may remove this label; all call RemoveLabelFromIssue, which is idempotent:

  1. Primary clearing owner (checkMergeabilityGate): when settle.Status == PRMergeReady or PRMergeBlocked (i.e., mergeable == true — Claude’s rebase push landed and GitHub confirms no conflict). The gate falls through to the CI gate on the same poll.
  2. PR-merged recovery path (runValidatePRTerminalAdvance): when the PR is already merged; removes all gate labels including fabrik:rebase-needed as part of terminal advance. Runs only for Validate-stage items without fabrik:auto-merge-enabled.
  3. Convergence success path (checkAutoMergeConvergence): when the PR merges under GitHub’s auto-merge after a conflict was resolved; removes fabrik:rebase-needed as part of convergence cleanup.
  4. SHA-invalidation scan (handleValidateSHAInvalidation, §2.16): when the linked PR’s HEAD SHA changes after stage:Validate:complete was recorded. Distinct in kind from the first three, which each re-check live mergeability before clearing — this path clears without re-checking mergeability (apart from the landed/terminal and pure-catch-up exemptions of #2158, which leave the label alone), because a new push invalidates the entire prior Validate-completion episode the label was scoped to, regardless of whether the conflict determination would still hold (#1225).

Additionally, cleanupClosedIssueTransientLabels performs a defensive sweep when the issue is closed — except when the item resolves specifically to the Validate stage, where fabrik:rebase-needed is deliberately left for the settle-owner pair (runValidatePRTerminalAdvance/settleClosedValidateAdvance, §6.15) to clear atomically as part of its own transition, rather than being stripped independently by the generic sweep (R6, ADR-1387; see §6.15’s own discussion of why the sweep previously raced this label, and why the exclusion is scoped to stage.Name == "Validate" rather than stageIsGateChecked generally).

6.6.2 Ordering Against the CI Gate

The merge-conflict gate runs before the CI gate so that a confirmed conflict preempts CI-await polling. The rationale: a PR that cannot merge has no reason to wait for CI, and Claude on CI-fix reinvoke cannot productively act on a branch that must first be rebased. When the merge gate emits conflict, Phase 1 continues without reaching the CI gate.

When the merge gate clears (mergeable == true, CI failed, or no conflict), Phase 1 falls through to the CI gate on the same poll. When the merge gate is blocked (PRMergeUnsettled — mergeable == null or a transient API error), Phase 1 skips to the next item — the next poll re-evaluates once GitHub has a definite answer or the API recovers.

6.6.3 Rebase Reinvoke Mechanics

When checkMergeabilityGate returns conflict=true:

  1. Worker guard (snap.Worker() != nil): if a rebase goroutine from a previous poll is still running for this item, skip dispatch entirely (no cycle-limit check).
  2. Cycle limit check: snap.RebaseCycles(stage.Name) is compared against MaxRebaseCycles (default 3 — lower than review/CI because rebase either works in one shot or needs human judgment):
    • If exceeded: pauseForRebaseCycleLimit() pauses the issue with fabrik:paused + fabrik:awaiting-input; fabrik:rebase-needed is intentionally left in place so the reason is visible.
    • If not exceeded: increment count, dispatch dispatchRebaseReinvoke():
      • Applies WorkerEntered (prevents double-dispatch)
      • Acquires semaphore slot (respects MaxConcurrent)
      • Calls buildRebaseComment() to construct a synthetic gh.Comment (DatabaseID: 0) instructing Claude to git fetch origin <base> && git rebase origin/<base>, resolve conflicts conservatively (never dropping code from base), watch for semantic collisions (duplicated ADR numbers, migration slots), run the project’s build + tests, and force-push with --force-with-lease.
      • Calls processComments() with the synthetic comment and the rebase_skill (falls back to comment_skill if unset)
      • On exit: releases semaphore, applies WorkerExited

DatabaseID: 0 guard: like the CI-fix and review synthetic comments, the rebase synthetic comment uses DatabaseID: 0 (and no ReactionNodeID) so processComments() skips the 👀 and 🚀 reaction steps (no real GitHub comment exists to react to).

6.6.4 Why Claude Rebases (Not the Engine)

The engine could in principle run git fetch && git rebase directly from the worker, but does not. Automatic rebase is right most of the time and catastrophically wrong sometimes: two PRs independently pick adr-054.md, both PRs pick migration slot 0042, both PRs add a new line at the same point in a shared config file. A mechanical rebase drops one side silently; a Claude-driven rebase can rename, renumber, and keep both contributions. The synthetic comment explicitly flags this — “watch for semantic collisions” — so Claude’s judgment is applied where it matters.

The cost is a re-invocation rather than an inline exec.Cmd. This is why MaxRebaseCycles defaults to 3 rather than 5: if Claude cannot rebase cleanly in three attempts the conflict almost certainly needs a human.

6.6.5 Rebase Reinvoke vs CI Fix Reinvoke

Aspect CI Fix Reinvoke Rebase Reinvoke
Trigger CI check runs in failure state mergeable == false on linked PR
Source data settle.CheckRuns from settlePRMergeState() (no separate REST call for PR-head checks) settle.Status == PRMergeConflicting from settlePRMergeState() (FetchPRMergeableFields provides mergeable)
Label on waiting fabrik:awaiting-ci (only on confirmed failure) fabrik:rebase-needed (only on confirmed mergeable == false)
Order in Phase 1 After merge-conflict gate Before CI gate
Cycle counter snap.CIFixCycles(stageName) / CIFixCycleIncremented snap.RebaseCycles(stageName) / RebaseCycleIncremented
Max cycles MaxCiFixCycles (default 5) MaxRebaseCycles (default 3)
Skill ci_fix_skill (falls back to comment_skill) rebase_skill (falls back to comment_skill)
Synthetic comment Structured CI failure report with NEW REGRESSION classification Rebase instructions with explicit semantic-collision guidance
Thread resolution No No
PR summary comment No No
Stage gate config wait_for_ci: true wait_for_ci: true (same opt-in — these are the stages that enter the catch-up window)
Label left on pause fabrik:awaiting-ci removed before pause fabrik:rebase-needed retained on pause so the human sees the reason

References: ADR-028: Merge-Conflict Gate and Rebase Reinvoke

6.6.6 CI ∧ Review Gate Joint-Clearing Sequence

Channel events (#1968): The moment both gates have cleared and the engine reaches runCatchUpPhase2 (or, for yolo, the landing decision in attemptMergeOnValidate) is where validate-settled is emitted; pauseForReviewTimeout emits review-timeout; see §7.14.

When a stage has both wait_for_ci: true and wait_for_reviews: true, the two gates are enforced by two distinct owners: handleReviewGate (which owns escalation and the timeout) and reviewGateBlocksLanding inside attemptMergeOnValidate (which owns the landing decision). The second owner exists because the Phase 1 handler chain alone cannot gate the poll pass in which CI clears.

Within a single poll pass, CI clearing and landing are not separated by a poll boundary. pctx.hasComplete is computed once, before the Phase 1 handler chain runs, and catchUpPhase1Handlers orders reviewGate ahead of mergeAndCIGates. As of #1270, a not-yet-complete fabrik:awaiting-ci item on a wait_for_ci stage reaches this chain via settleAwaitingCIScan (§6.5.1 Path 2), not the main catch-up loop — the main loop’s admission gate is hasComplete-only, so it never sees these items while CI is still pending. settleAwaitingCIScan runs runCatchUpPhase2 (the same gated stage-advancement/landing step the main loop’s Phase 2 performs) immediately after the handler chain if no handler claimed the item, so the same-poll handoff below is unchanged in outcome, only in which iteration performs it. So on the pass where CI turns green:

  1. handleReviewGate runs first. stage:X:complete is still absent (the frozen snapshot was taken before the CI gate ran), so its gate half (checkReviewGate) is skipped — by design, per #617 — while its feedback detection still runs and dispatches any unprocessed feedback (#1953 R1).
  2. handleMergeAndCIGates runs later in the same chain. CI clears; checkCIGate calls addCompleteLabelAndRemoveCI, adding stage:X:complete and removing fabrik:awaiting-ci. It returns false (unclaimed) because the gate is now clear.
  3. No handler claimed the item → settleAwaitingCIScan calls runCatchUpPhase2 immediately, in the same scan pass. For a yoloActive Validate item without fabrik:auto-merge-enabled, this calls attemptMergeOnValidate directly.

There is therefore no poll pass in which the item sits at Validate with hasComplete == true and handleReviewGate has not already skipped itself. Relying on the handler chain alone (with no immediate Phase-2-equivalent step) would leave the gate permanently unarmed whenever wait_for_ci defers completion — i.e. on every real PR, since CI is essentially never already green at the instant FABRIK_STAGE_COMPLETE is emitted (#1216).

Landing-decision gate (reviewGateBlocksLanding, engine/reviews.go). attemptMergeOnValidate is the single landing-decision owner for both merge_train modes (ADR-058/ADR-059 “invoke, don’t relocate”). The gate check sits after the cruise and fabrik:auto-merge-enabled early-returns and before the merge_train fork, so it covers auto-merge enable, merge-queue enqueue, direct merge, and advance-to-Queued identically:

#617 non-regression. The landing gate is provably unreachable while CI is genuinely still blocking: handleMergeAndCIGates claims the item in that case, so Phase 2 never runs and attemptMergeOnValidate is never called. No extra guard is needed, and catchUpPhase1Handlers ordering is unchanged.

wait_for_ci is not required for the gate to matter. handleStageComplete calls attemptMergeOnValidate whenever yoloActive && !waitForCI, which is before its own fabrik:awaiting-review seeding block runs. The landing gate covers that path too — which is why the fix lives in attemptMergeOnValidate rather than in the handler chain.

Feedback gate (feedbackGateBlocks, engine/feedback_gate.go, #1862 → #1953, ADR-1862, ADR-1953-a). The same choke point gates on unprocessed feedback — issue comments (#1862), and since #1953 also unresolved current-head review threads and unaddressed review bodies (a COMMENTED bot review used to clear every landing gate: reviewers outstanding and the authority verdict only). One predicate, unprocessedFeedback, answers “is anything unprocessed on this item right now”, read live and fail-closed (R2/R5), and is consulted at attemptMergeOnValidate (placed after reviewGateBlocksLanding, and again before the direct-merge fallback), at handleStageComplete’s advance and at runCatchUpPhase2’s non-Validate advance — never inside advanceToNextStage, which is also the terminal Done advance after a merge, and advanceConvergedPRToDone runs only after a confirmed merge so there is nothing to hold there. A hold is logged [#N feedback-gate] holding <where> — 1 unprocessed review body (5356334497) (R6). Review findings go through dropCatchUpFeedback first (#2044, §6.30): a bot review or thread comment made against the live head of a pure merge-train catch-up (self-authored marker comment, exact commit match) is non-actionable for the landing; everything else — humans, conflict-edited catch-ups, older heads, unverifiable markers — is unchanged. Feedback that arrives while a stage invocation is running is not injected into it: the run finishes, then the gate holds the completion from being final and the ordinary reinvoke path processes it (R3, complete-then-reinvoke; the reinvoke’s push restarts any CI wait, an accepted cost). Original #1862 text follows, and still describes the comment half. Without it, Phase 2 merged a Validate-complete item from a snapshot that already contained a fresh steering comment (typically posted just before a cruise → yolo swap), and dispatchCandidates — which runs after the catch-up loop in the same poll — then processed that comment against the merged branch, orphaning the rework.

Reviewer review submitted during CI-await: a pull_request_review webhook applies PRReviewSubmitted to the board cache store. handleReviewGate’s gate half is still not evaluated during the CI-await window (!hasComplete guard, #617), but its feedback detection is (#1953 R1), so a review body or thread is dispatched as a review-reinvoke during the wait, and the landing gate re-reads review state live on the pass where CI clears, so the reviewer’s decision is honoured at the landing decision rather than one poll later.

Unit tests:

review_authority: authoritative composes unchanged (ADR-1250). The verdict check described in §6.1.1 lives inside reviewGateBlocksLanding’s own len(outstanding) == 0 && hasReviews branch, so it participates in the joint-clearing sequence exactly as the advisory clearing condition does — no separate ordering, no new poll-gap risk. A CHANGES_REQUESTED verdict on the pass where CI turns green blocks the landing exactly as an outstanding reviewer would; the item still gets stage:Validate:complete (from handleMergeAndCIGates) with the landing withheld, and checkReviewGate (Path 2) owns escalation from the next poll onward, same as always.

References: ADR-1216: Review gate checked at the landing decision, ADR-1250: Review authority — advisory vs. authoritative, an axis orthogonal to autonomy

6.7 Pre-Implement Spawn Path

Channel events (#1968): spawnChildren, once every child is created and wired and fabrik:children-spawned is applied, emits children-spawned (scope grew); see §7.14.

Trigger: The Implement stage dispatcher calls preImplement() before the Claude invocation on every Implement dispatch. preImplement is a no-op unless the Plan stage comment contains FABRIK_SPAWN_CHILD_BEGIN/END blocks AND the parent issue does not yet have fabrik:children-spawned.

FABRIK_SPAWN_CHILD_* marker convention: Plan emits structured blocks in its output to declare sub-issues to spawn:

FABRIK_SPAWN_CHILD_BEGIN owner/repo
TITLE: <single-line title for the new issue>
DEPENDS_ON: <n>                          # optional — see below

<full scoped spec body — multiple paragraphs OK>
FABRIK_SPAWN_CHILD_END

These blocks persist in the Plan stage comment — they are data, not consumed-and-stripped signals. The preImplement engine step reads them from the most-recent Plan comment body at Implement dispatch time.

DEPENDS_ON: (ADR-1337, sibling ordering): an optional header line expressing a forward-only dependency on an earlier block in the same Plan output — the mechanism for the common case of sequentially dependent, same-repo decomposition slices, where without it Plan could only encode the ordering as prose (“Depends on: Slice N”) that checkDependencies cannot see. When present, DEPENDS_ON: must immediately follow TITLE: with no blank line between them; a DEPENDS_ON:-looking line separated from TITLE: by a blank line is parsed as ordinary body content, not a header. The value is a 1-based index into this Plan output’s own block list — DEPENDS_ON: 2 on block 3 means “block 3 depends on block 2” — never an issue number, since children don’t have issue numbers yet when Plan runs. References must be strictly forward (1 <= DEPENDS_ON < ownIndex); this makes sibling dependency cycles structurally impossible without any graph walk. An absent header leaves the block’s dependency behavior exactly as before this feature — the parallel-star graph. DEPENDS_ON is not restricted to same-repo blocks; the underlying AddBlockedByIssue primitive is already repo-agnostic.

On-thread spawn receipt (#1338): at the moment the Plan comment is posted — well before preImplement ever runs — the engine appends a deterministic note naming the count of well-formed blocks it parsed (via the same ParseSpawnBlocks preImplement itself uses) and stating that no sub-issues exist yet, that they will be created when the parent advances to Implement. This makes the declarative nature of the blocks visible on the thread itself, rather than only in docs/USER_GUIDE.md or the engine log; see docs/stage-lifecycle.md’s “Spawn Receipt Note” section for the posting-path details. A block that fails to parse (#2091) is reported instead: the note becomes a ⚠️ N spawn block(s) could not be parsed: block 2 — … found `` warning (block index counts every BEGIN/END pair; the quoted line is sanitised), stating that Implement will pause until the Plan is corrected and that no sub-issues will be created. Prose that merely mentions the marker without forming a BEGIN/END pair (e.g. #1263) still yields zero blocks and no note. A BEGIN with no matching END remains silent apart from the `logUnparsedSpawnMarkers` log line. The note is gated to the Plan stage specifically, matching `preImplement`'s own hardcoded `findStageComment(item.Comments, "Plan")` lookup above — a note appearing on any other stage's comment would promise a spawn that lookup will never act on.

Malformed blocks are held and paused (#2091, ADR 2091). ParseSpawnBlocksDetailed returns, besides the well-formed blocks, one SpawnBlockError{Index, Repo, Found, Reason} per BEGIN/END pair that failed to parse (ParseSpawnBlocks is a wrapper returning only the blocks). Header keys (TITLE, DEPENDS_ON) match case-insensitively and whitespace-tolerantly via matchHeader; DEPENDS_ON must still be on the line immediately after TITLE. preImplement and recoverMissingPlanComment call pauseForMalformedSpawn when any error exists: one [#N spawn] log line per block, a 🏭 **Fabrik — malformed FABRIK_SPAWN_CHILD marker** comment listing each block’s 1-based index (position among all pairs — what a human counts in the Plan), repo and quoted offending line, and fabrik:paused. The whole declaration is held — no child is created, including the well-formed siblings: spawning the siblings would shift the DEPENDS_ON indexes and the ADR-1583 fabrik:spawned-child:<index>:<n> markers and leave the parent’s blockedBy set half-populated. The stored Plan comment is immutable, so a bare un-pause re-pauses until the Plan is edited (or a comment gets it revised); the comment says so.

Own-line requirement (enforced by ParseSpawnBlocks): Both markers must stand on their own line, the same convention FABRIK_STAGE_COMPLETE follows. On the BEGIN line the target repo must be the only trailing token and a well-formed owner/repo; the END line must carry nothing else. Leading whitespace is tolerated, so a block nested under a list item still parses — trailing prose after the repo is not. A marker occurring inside prose (for example a backticked mention in the Plan’s own task checklist) is therefore ignored, and cannot consume, truncate, or suppress a genuine block later in the same comment. Until this was enforced, such a mention silently destroyed the real block and the spawn simply did not happen (#1263). When a BEGIN-looking line is present but no well-formed block parses, preImplement now logs that fact rather than returning the same silent “nothing to spawn” result as a Plan that declared no children.

Code path: processItem() → Implement stage dispatch → preImplement() (in engine/spawn.go)

Idempotency guard: If the parent issue has fabrik:children-spawned, preImplement returns immediately (no-op). Manual removal of this label (and closing of children created under the completed spawn) is required to trigger a fresh spawn. This guard runs before the recovery path below and must remain first — recovery must never bypass it, or repeated deferred-retry cycles risk double-spawning.

Missing-Plan-comment recovery (#982): findStageComment(item.Comments, "Plan") returning nil is ambiguous on its own — it means either “Plan genuinely declared nothing to spawn” or “the item snapshot’s Comments is stale” (Comments is a deep field whose cache freshness is updatedAt-keyed; see #957). preImplement disambiguates using the stage:Plan:complete label, which is populated through a different path and can be visible even when Comments lags:

spawnChildren is shared by three callers (ADR-1419). preImplement’s direct path, recoverMissingPlanComment’s recovery path, and — since #1419 — the Review/Validate mid-flight hook (§6.7.2) all funnel through the single spawnChildren function documented below. This is what makes a Review/Validate spawn “not a second-class path” relative to a Plan-spawned one: board-servability, assignment, blocked_by linking, and default labeling are identical regardless of which stage originated the spawn, because there is exactly one implementation to drift out of sync.

Flow (when spawn blocks are present and not yet spawned):

  1. Validate all DEPENDS_ON headers upfront, before any GitHub mutation: validateSpawnDependsOn checks that every declared index is a strictly-forward reference to an earlier block (1 <= DEPENDS_ON < ownIndex) — purely structural, no created-issue data needed. Any invalid value (out-of-range, non-forward/self-referencing, or syntactically malformed — non-numeric, empty, zero, negative) is a hard failure: post an error comment (Created so far: none), add fabrik:paused, and stop — no children are created. A silent drop here would reproduce the exact bug DEPENDS_ON exists to fix.
  2. Board-servability scope-check (ADR-1419, requirement 3). For each unique target repo in the spawn blocks, spawnTargetServedByThisInstance(childOwner, childRepo) asks: does this instance’s own, already-known config cover this repo? A multi-repo instance (cfg.Repo == "") already legitimately serves any repo the org grants board access to — proceeds unchanged, exactly as before #1419 (this is why “Instance A processes multiple repos on board 5 without issue” in the originating report). A repo:-scoped instance whose repo does not match the target has no basis to claim that repo is servable here — per the deliberately out-of-scope “no cross-instance board discovery” boundary, the fix cannot locate the correct board, only refuse to guess wrong. It posts an error comment naming the unservable target and this instance’s own owner/repo scope, adds fabrik:paused, and stops — no children are created for the whole batch (a routing failure aborts the entire spawn, not just the offending block, mirroring DEPENDS_ON validation’s all-or-nothing semantics). This is the fix for failure mode 1 (a Plan-spawned cross-repo child silently registered onto the parent’s own board and left unassigned) — it does not relocate the child to the correct board (no data exists to do so), it makes the failure visible instead of silent.
  3. Validate all target repos are clonable: call ensureSpawnTargetReady for each unique target repo that passed the scope-check in step 2. If a repo cannot be bare-cloned, post an error comment, add fabrik:paused, and stop — no children are created.
  4. For each FABRIK_SPAWN_CHILD_BEGIN/END block in document order (creation pass — unchanged by DEPENDS_ON, except that the block-index → child node-ID mapping is now retained for the sibling-wiring pass below):
    • Resume-safe retry (ADR-1583). Before the loop, parseSpawnChildLabels(item.Labels) recovers which 1-based block indices already have a fabrik:spawned-child:<blockIndex>:<childNumber> marker from a previous, interrupted attempt at this same batch; if any exist, item.BlockedBy is refreshed live first (refreshForSpawnResume, mirroring the missing-Plan-comment recovery’s cooldown-gated live-read pattern below — a live-read failure or active cooldown returns the same errPreImplementDeferred outcome, not a pause). For a block with an existing marker: CreateIssue is skipped — the child’s node ID is instead resolved via FetchProjectItem(childOwner, childRepo, childNumber), pausing loud (naming the specific marker label to remove) if that recorded child can no longer be found. For a block with no marker, the sequence below runs exactly as it always has.
    • CreateIssue(owner, repo, title, body+footer, assignees) via REST — returns (number, nodeID). assignees is [cfg.User] in PAT mode (ADR-1419, requirement 4) — under GitHub App auth (#1893) it is the parent’s human assignees instead (spawnAssignees; possibly none), and an HTTP 422 on them is retried once with no assignees rather than aborting the batch. In PAT mode every spawned child, same-repo or cross-repo, is assigned to the user of the instance meant to process it; folded into the same POST as creation rather than a separate mutation, so a bad/misconfigured user: value fails loud through this single already-fail-loud call rather than opening a second, silent failure point. Immediately after a successful CreateIssue, the durable fabrik:spawned-child:<blockIndex>:<childNumber> marker is written on the parent (addLabelChecked, best-effort — a write failure is logged, not fatal, and only regresses that one child to pre-ADR-1583 behavior if a later step in the same attempt also fails).
    • AddProjectV2ItemById(board.ProjectID, nodeID) — adds child to the same project board; returns childItemID. When resuming a marked block, this is skipped in favor of a LookupIssueProjectItem check first — if the child is already on the board, its existing item ID (and current Status) is reused instead of calling AddProjectV2ItemById again (unconfirmed against real GitHub whether a repeat call is safe — ADR-1583 checks rather than assumes). A fresh (unmarked) block always takes the direct-call path, unchanged.
    • AddBlockedByIssue(parent.NodeID, nodeID) — links child as a blockedBy dependency of the parent. When resuming, the (freshly refreshed) item.BlockedBy is checked first; if the edge already exists, this call is skipped for the same unconfirmed-idempotency reason as the board-add above.
    • AddLabelToIssue for fabrik:sub-issue on the child (informational; idempotent add, always attempted)
    • Inherited labels (#2090, ADR 2090) — applied before the Status placement below. inheritChildLabels copies, in order: the parent’s base:<branch> label (same owner/repo only, compared case-insensitively; a cross-repo child gets none because the branch may not exist there, and keeps today’s missing-branch fallback-with-comment behavior), then fabrik:yolo if the parent has it, then fabrik:cruise if the parent has it (any repo, unchanged). When the parent carries several base: labels the child gets only the one the parent itself resolves to — baseLabelFor, the same first-non-empty-wins rule baseBranchForItem uses; a parent with none gives the child none. The label is the parent’s label as written, not an ls-remote-verified branch: if the branch is missing from the remote, the child falls back to the default with the usual one-time comment at its own worktree creation. The ordering matters: the Status placement is what makes a child dispatchable, and its first dispatch’s EnsureWorktree reads base:, so a label added afterwards could lose that race and fork from the default branch. The base: write is fail-closed: if it fails, inheritChildLabels returns the error immediately and spawnChildren pauses the parent (fabrik:paused, the same path as the other per-child failures) and returns an error before the Status placement, so the child is never dispatchable against the wrong base — a child placed without base: would fork from and target the default branch, and under yolo land on it. The child already carries its fabrik:spawned-child:<i>:<n> marker (resumable origins), so the ADR-1583 retry resumes it and the resume path’s missing-label check writes base: then; no new retry machinery. A non-resumable origin (Review/Validate mid-flight) writes no marker, so its retry re-spawns as before — which means a failed base: write there leaves the already-created child (on the board, linked as blockedBy of the parent, labelled fabrik:sub-issue, but with no Status and therefore never dispatched) as an orphan that the retry does not reuse: the retry creates a duplicate, and the orphan must be closed by hand, or the parent stays blocked by it. This is the same pre-existing duplicate-on-retry exposure that a failed board-add or blockedBy link has on these origins (ADR-1583), not a new failure class. The fabrik:yolo/fabrik:cruise writes stay best-effort (failure logs a warning, spawn continues), since a missed autonomy label only reduces automation. When resuming, the child’s labels from the FetchProjectItem result already in hand are checked and only the missing ones are written — a second retry makes no label writes — and this runs even for a child with a non-empty Status (its Status is not touched). A label cannot repair a child that already created its worktree from the default base. Because the same code path serves every origin, Plan, Review and Validate spawns inherit identically. Children of a non-default-base parent therefore take the ADR-1096/1097 explicit-close path and join the parent’s base merge-train partition (ADR-1648).
    • UpdateProjectItemStatus(board.ProjectID, childItemID, sf.FieldID, specifyOptionID) — moves child to the Specify column (or first non-Backlog, non-terminal column as fallback). Non-fatal: if the call fails, e.statusField is nil (startup fetch failed), or no viable column exists, the child lands in Backlog, a warning is logged, spawn continues — and recordChildPlacementFailure writes the durable fabrik:awaiting-placement marker on the child so the board-placement settle scan retries it (§6.9, ADR-062). When resuming a block whose LookupIssueProjectItem check (above) found a non-empty Status, this call is skipped entirely rather than re-run — the child may have already progressed under its own pipeline while the parent sat paused, and blindly re-placing it into Specify would clobber that progress (ADR-1583).
    • On any failure in the fatal steps (CreateIssue, AddProjectV2ItemById, AddBlockedByIssue, the inherited base: label write, or resolving a marked-but-now-missing child): post error comment naming completed and failed children, add fabrik:paused to parent, stop without adding fabrik:children-spawned. The comment no longer instructs closing orphaned children (ADR-1583) — it states that already-created children are tracked and will be resumed, not recreated, on the next retry.
  5. Sibling-wiring pass (second pass, after all children in step 4 exist): for each block that declared DEPENDS_ON, call AddBlockedByIssue(childNodeIDs[ownIndex], childNodeIDs[dependsOnIndex]) — the same primitive used for the parent edge in step 4, now linking two children. Immediately on success, apply BlockedByEdgeAdded to the Store too — keyed on the dependent child’s own (repo, number), not the parent’s, mirroring step 4’s parent-edge write (#1783 follow-up) so a dependent child is not exposed to the same stale-BlockedBy race this ADR closes for the parent. A block may depend on any earlier sibling regardless of where in step 4 its own creation happened to land. On failure: post an error comment (all children created so far are listed), add fabrik:paused to parent, stop without adding fabrik:children-spawned — the same partial-creation-on-failure shape step 4 already uses, not a new failure mode.
  6. After all children are created, linked to the parent, and all declared sibling edges are wired: best-effort remove every fabrik:spawned-child:<blockIndex>:<childNumber> marker this call knows about (ADR-1583 — steady state carries none of them, they exist only transiently during an interrupted/in-progress spawn), then add fabrik:children-spawned to the parent. The idempotency guard is applied for the full two-phase operation (steps 4+5) — after both, not immediately-after-step-4 — specifically so a sibling-wiring failure is retried on the next attempt rather than silently accepted as “done.”

After spawn: spawnChildren returns (spawned, true, nil), where spawned lists each child as owner/repo#N. From preImplement’s call site, processItem returns without invoking Claude — decomposition is a terminal outcome for that dispatch (R1, #1783).

The dispatch gate is a synchronous Store write, not a race against deep-fetch/cycleSet timing (#1783, ADR-1783). Before #1783, the only thing that could stop a later dispatch attempt from running Claude on the full parent was fabrik:blocked, and fabrik:blocked depended on checkDependencies seeing a freshly deep-fetched BlockedBy — which required a LabelsChanged change (from fabrik:children-spawned) to populate cycleSet, which required a subsequent poll’s selectDeepFetchCandidates to admit the item for a real FetchItemDetails call. Under GraphQL rate-limit pressure or a transient deep-fetch failure, that chain could take several poll cycles to complete — or, if the item was admitted to dispatch by some other route in the interim (e.g. a comment) while BlockedBy was still stale-empty, checkDependencies would report “not blocked” and Implement would build the whole parent as a monolith minutes after the children were filed (the #1659 report). Immediately after each child is linked via AddBlockedByIssue, spawnChildren now also applies a BlockedByEdgeAdded mutation (internal/itemstate) directly to the engine’s own Store, appending that one dependency edge with no other deep field touched. Because boardcache.CacheImpl.FetchProjectBoard reconstructs every item’s BlockedBy straight from the Store on every call (snapshotToProjectItem), the very next board fetch — this poll’s own continuation or the next poll, whichever comes first — already carries the correct edges, with zero dependency on cycleSet wake timing, deep-fetch admission, or a live re-fetch succeeding. checkDependencies then sees the new blockedBy edges — both parent edges and any sibling edges — on that board and adds fabrik:blocked, gating the parent until all children close. checkDependencies and PushUnblockObserver require no changes for sibling edges: both already operate generically over any item.BlockedBy entry regardless of whether it originated from the parent-wiring step or the sibling-wiring step. The BlockedByEdgeAdded write is unconditional per child — it fires whether the edge was freshly linked or found already-linked on an ADR-1583 resume — so a partial (paused) spawn’s already-linked children are reflected in the Store even though the batch as a whole did not complete (R5 non-regression).

Partial spawn failure: fabrik:children-spawned is NOT applied if any step fails — including a DEPENDS_ON validation failure (step 1, before any child exists), a board-servability refusal (step 2, before any child exists), or a sibling-wiring failure (step 5, after all children exist but some sibling edges may be unwired). On retry (after user removes fabrik:paused), the caller re-runs spawnChildren from the beginning — but resume-safe as of ADR-1583: any block already recorded via a fabrik:spawned-child:<blockIndex>:<childNumber> marker (step 4, above) is recognized and resumed rather than recreated. Users no longer need to manually close orphaned children before re-advancing for the three per-child steps this covers (CreateIssue, AddProjectV2ItemById, AddBlockedByIssue); the sibling DEPENDS_ON wiring pass (step 5) itself is unchanged — a raw re-call on every retry, deliberately out of this ADR’s scope (its own AddBlockedByIssue calls are expected, but not confirmed, to no-op on a repeat).

Recursive decomposition: A child issue created by preImplement runs the full Fabrik pipeline. If the child’s own Plan emits FABRIK_SPAWN_CHILD_* blocks, the child’s Implement dispatch triggers another preImplement — grandchildren are created identically. There is no depth limit.

References: ADR-048: Engine-Side Pre-Implement Spawn, ADR-1583: Spawn-Children Resume-Safe Retry via Durable Per-Child Markers, ADR-1783: Deterministic Post-Spawn Dependency Gate

6.7.1 Empty-Coordinator Completion

Trigger: A parent issue carries fabrik:children-spawned (§6.7) and its Implement stage completes (FABRIK_STAGE_COMPLETE, err == nil) with zero commits in the worktree ahead of origin/<baseBranch>. This is the case of a fully-delegated coordinator: all of the parent’s actual work was decomposed into spawned children, which have since merged to main, leaving the parent’s own worktree with nothing to commit.

Why this needs a dedicated check. Before this, a completed Implement invocation with create_draft_pr: true unconditionally attempted ensureDraftPR (§5.5). For a pure coordinator, GitHub’s CreateDraftPR call fails with a 422 (“No commits between main and fabrik/issue-N”) because there is genuinely nothing to diff. The failure repeats every retry (the worktree state doesn’t change), so the issue always reaches MaxRetries and lands in escalatePRCreationFailure — paused with fabrik:paused, requiring a manual close even though the work is substantively complete. ADR-048 anticipated a coordinator-only parent emitting FABRIK_NO_WORK_NEEDED itself, but the Implement skill prompt was never taught to detect this case, so in practice Claude just completes normally and falls into the PR-creation attempt.

Detection is commit-count-based, not label-based, and only runs after a completed Claude invocation. Checking fabrik:children-spawned alone, before Claude even runs, would be unsafe: specs/sub-issue-decomposition/spec.md explicitly supports a hybrid parent — one whose Plan both spawns children and describes its own implementation work, which runs normally in Implement after all children close. On the first post-unblock Implement dispatch, “zero commits so far” is true for both a pure coordinator and a hybrid parent that hasn’t started yet — the two are only distinguishable after Claude has had the chance to produce its own commits. Gating on completed (Claude ran to completion, no error) makes the check safe for both: a hybrid parent that does its own work ends up with commits ahead of base and is unaffected; a pure coordinator ends up with zero and is caught.

Implementation — commitsAheadOfBase(workDir, baseBranch string, issueNumber int) (int, error) (engine/item.go): lists git log --name-only origin/<baseBranch>..HEAD in the worktree and counts every commit except those whose only changes are the item’s own persisted spec (specs/<issueNumber>-*/spec.md, isSpecOnlyCommit — ADR 2034, §7.16), so the Specify stage’s engine-written spec commit never makes a fully-delegated coordinator look like it has work of its own. Empty and merge commits (no touched files) still count. Fails safe — any git or parse error returns a non-nil error, never an assumed zero, so callers must treat an error as “unknown” and fall through to the normal PR-creation path rather than short-circuiting to Done.

Two call sites, both folding the result into the existing noWorkNeeded boolean rather than introducing a separate completion path — this lets the fully-built, ADR-060-hardened handleNoWorkNeeded/settleNoWorkNeeded machinery (§6.8) handle everything else (durable marker, idempotent settle, Done move, issue close, retry/escalation) with no new code:

  1. Primary path — finalizeStageOutcome (called after every Claude invocation completes): after computing noWorkNeeded := err == nil && CheckNoWorkNeeded(output), if !noWorkNeeded && completed && stage.Name == "Implement" && hasLabel(item.Labels, "fabrik:children-spawned"), calls commitsAheadOfBase(workDir, baseBranch, item.Number). On success with ahead == 0, sets noWorkNeeded = true — routing the rest of finalizeStageOutcome through the same completed && noWorkNeeded branch a Claude-emitted FABRIK_NO_WORK_NEEDED would take (§6.8), instead of the plain completed branch that calls ensureDraftPR. This computation runs before finalizeStageOutcome’s own separate eager ensureDraftPR call — the one gated on completed && stage.CreateDraftPR && stage.PostToPR that exists purely to guarantee a PR exists before posting output to it — so that eager call is itself gated on !noWorkNeeded and never fires for the empty-coordinator case either. Without this ordering, an empty coordinator would still trigger a real CreateDraftPR 422 from that eager call even though the later completed && noWorkNeeded branch correctly skips escalation — the real Implement stage config has both create_draft_pr: true and post_to_pr: true, so this eager call is always reachable in production.
  2. R5 self-heal — the skip-Claude retry precheck (processItem, §5.5): when an item already has PRCreationFailed[stage.Name] == true (e.g. from a pre-fix retry cycle, or from before this check existed), the same guard runs unconditionally — reaching this branch already proves Claude completed the stage at least once and still produced zero commits, so there is no hybrid-parent ambiguity to protect against here. On a match, the lock is released, StageRetryCleared/EngineUnpaused are applied, and handleNoWorkNeeded is called directly — before ensureDraftPR is attempted again. This lets an issue already stuck mid-retry-cycle when this fix ships (or a pre-fix issue with fabrik:paused manually removed) recover to Done on its next poll instead of continuing to hammer ensureDraftPR toward escalation.

Comment text is distinct from the marker-emitted case. noWorkNeededSkipComment (engine/no_work_needed_settle.go) composes the “skipped: no work needed” comment body: for an Implement stage with fabrik:children-spawned (this path), it posts _Skipped: no work needed — all work was delegated to spawned children and this issue has no commits of its own._ rather than the generic _Skipped: no work needed (FABRIK_NO_WORK_NEEDED emitted by Implement)._ text used everywhere else. Earlier revisions of this fix reused the generic text, but it asserts something false — Claude never emitted FABRIK_NO_WORK_NEEDED here, the engine inferred completion from zero commits — and a durable GitHub comment that misdescribes its own mechanism erodes trust in the record later. No separate completion path was introduced to fix this: hasSkippedComment’s idempotency check derives the same way, so retries remain safe.

A parent with own commits is unaffected. If the coordinator’s Implement invocation produces its own commits (hybrid parent, or a coordinator whose “own work” happens to be a no-op that still commits something), commitsAheadOfBase returns nonzero, noWorkNeeded stays false, and ensureDraftPR runs normally — the guard checks actual commit state, not just the label.

References: ADR-048: Engine-Side Pre-Implement Spawn, handarbeit/fabrik#921

6.7.2 Mid-flight Spawn Recognition (Review/Validate)

Trigger (ADR-1419). FABRIK_SPAWN_CHILD_BEGIN/END recognition is not exclusive to Plan. finalizeStageOutcome (engine/item.go) also scans for well-formed blocks when stage.Name is "Review" or "Validate", immediately after the FABRIK_PR_CREATE marker handling and before output is stripped/posted. This closes failure mode 2 from the originating report: a Review or Validate agent that discovers a blocker mid-flight previously had no sanctioned route to declare it and would call gh issue create directly — the engine never observed that spawn, so none of its wiring ran (no board registration, no assignee, no blocked_by edge), and the parent resumed and could report green on a tree that was, in fact, still missing its blocking dependency.

Why raw output, not a stored comment. Plan’s mechanism (§6.7) is “stage N’s output, read back later as a stored comment, drives stage N+1’s pre-dispatch step” — preImplement reads the most-recent comment literally named "Plan" via findStageComment. Review and Validate are post_to_pr: true stages, so nothing later re-reads their comment for this purpose; instead, ParseSpawnBlocks runs directly against the in-memory output string produced by this dispatch, before postOutputToPR/stripMarkers runs — structurally identical to the existing FABRIK_PR_CREATE_BEGIN/END handling immediately above it in the same function. This sidesteps the post_to_pr complication entirely: it does not matter whether the eventual comment lands on the issue or the PR, because parsing already happened on the pre-posting string.

No new idempotency label needed. output is this dispatch’s own fresh content, never replayed across dispatches — a block is processed exactly once by construction, the same guarantee FABRIK_STAGE_COMPLETE detection already relies on. This is unlike Plan’s fabrik:children-spawned guard, which exists because preImplement re-reads the same stored comment on every Implement dispatch.

Same shared spawnChildren, same wiring — except ADR-1583’s resume marker. The block list parses via the same ParseSpawnBlocks/spawnBeginRepo/isSpawnEndLine own-line-marker discipline documented in §6.7 (the #1263 hardening applies identically — a Review/Validate output that merely mentions the marker in prose cannot misfire) and is passed to e.spawnChildren(ctx, board, item, owner, repo, blocks, false) — the exact function §6.7’s Flow documents, including the board-servability scope-check, mandatory assignment, and blocked_by linking. Nothing about board registration, assignment, or dependency-edge creation is reimplemented for this call site. The final argument (resumable) is false here, unlike preImplement’s and recoverMissingPlanComment’s true: ADR-1583’s fabrik:spawned-child:<blockIndex>:<childNumber> marker assumes blocks reparses identically on every retry, which holds for Plan’s immutable, already-posted comment but not for this call site’s output — a fresh Claude generation on every dispatch, including a redispatch after a failed mid-flight spawn’s fabrik:paused is removed. With resumable=false, this origin never reads, writes, or leaves behind that marker; a retried mid-flight spawn always takes the fresh-CreateIssue path for every block, exactly as it did before ADR-1583 — carrying forward that ADR’s original duplicate-on-retry exposure for this origin specifically, rather than risking the marker resuming an unrelated, already-created child under a same-numbered but different block on a non-deterministic redispatch (caught in review on PR #1708).

On success: formatMidflightSpawnReceiptNote(spawned) — a present-tense sibling of Plan’s formatSpawnReceiptNote (§6.7’s on-thread spawn receipt) — is prepended to output, and stripSpawnBlocks(output) removes the raw FABRIK_SPAWN_CHILD_BEGIN/TITLE:/FABRIK_SPAWN_CHILD_END block(s) that were actually spawned before posting — mirroring the stripMarkers call already made for FABRIK_PR_CREATE/FABRIK_ISSUE_UPDATE immediately above in the same function. This differs from Plan’s own comment, which deliberately leaves its raw block visible (formatSpawnReceiptNote’s “declared above” wording depends on it staying there); the mid-flight note is self-contained and already names what was spawned, so leaving the internal marker syntax in a human-facing PR/issue comment would only duplicate that information verbatim (a pruefer review finding on the originating PR). stripSpawnBlocks re-uses ParseSpawnBlocks’s own line-scanning (spawnBeginRepo/isSpawnEndLine/parseTitleAndBody), so it removes exactly the blocks that were parsed and spawned. A malformed block never reaches it (#2091): finalizeStageOutcome parses with ParseSpawnBlocksDetailed first, and on any SpawnBlockError runs preserveWorkOnMarkerFailure (commit + push, as for FABRIK_PR_CREATE), pauseForMalformedSpawn (comment naming the block index and quoted line, fabrik:paused), releases the lock and returns — no spawnChildren, no stage completion; a retry re-runs the stage and regenerates the declaration, so there is no re-pause loop (unlike Plan). The leave-visible branch of stripSpawnBlocks is defensive only. It correctly handles a stage output declaring more than one blocker in the same dispatch. The stage’s own narrative output (and, when present, FABRIK_STAGE_COMPLETE) is otherwise untouched and posted normally — a mid-flight spawn does not, by itself, suppress or force the stage’s completion decision.

On failure: spawnChildren has already posted its own fail-loud pause comment and added fabrik:paused (the same path §6.7’s Flow documents for every fatal step). finalizeStageOutcome releases the dispatch lock and returns immediately — this dispatch’s own stage-complete comment/label is suppressed entirely, mirroring the FABRIK_PR_CREATE failure precedent immediately above it in the same function.

Validate dependency guard: live re-check before any landing decision. attemptMergeOnValidate (engine/stages.go) is reached from handleStageComplete synchronously, in the same dispatch that just ran finalizeStageOutcome’s mid-flight spawn hook above — before checkDependencies is ever called for this dispatch through the normal per-item admission path (checkDependencies only gates the shouldAdvance branch, which a successful auto-merge/direct-merge short-circuits to false, skipping it entirely). Left unguarded, a Validate output that both declares a genuine blocker and emits FABRIK_STAGE_COMPLETE in the same turn could have its PR merged before the freshly-created blocked_by edge was ever consulted — the exact same-turn race the mid-flight hook would otherwise introduce. attemptMergeOnValidate closes this itself: immediately after the fabrik:auto-merge-enabled idempotency check and before guard 1’s review-thread check, it re-fetches the item live (FetchItemDetails, the same pattern already used by the direct-merge fallback’s own live re-check further down this function) and calls checkDependencies against the fresh snapshot. A dependency created moments earlier in this exact dispatch is caught: checkDependencies applies fabrik:blocked and posts the usual blocked comment, and attemptMergeOnValidate returns (false, false, nil) without ever calling EnablePullRequestAutoMerge/MergePR/EnqueuePullRequest. Both callers of attemptMergeOnValidate (handleStageComplete and poll.go’s catch-up loop) already gate on yolo and return early once fabrik:auto-merge-enabled is present, so this extra live fetch is bounded to the same narrow per-Validate-completion window guard 1 already re-fetches live data in — it is not a new per-poll cost for items already converging toward merge. This is a genuine engine-side hard stop, not prompt-level mitigation alone; plugin/fabrik-workflows/skills/validate/SKILL.md’s “If You Discover a Blocking Issue” section still instructs the agent not to emit FABRIK_STAGE_COMPLETE alongside a genuine spawn (consistent with Validate’s own “Decision: Complete or Block” criteria), but the engine no longer depends on the agent following that instruction to avoid a premature merge. Review carries no equivalent hazard in the first place: checkDependencies runs (and correctly gates) at the next stage’s dispatch off a non-Validate completion, so completing Review while also spawning a blocker was always safe.

References: ADR-048: Engine-Side Pre-Implement Spawn, ADR-1419: Cross-repo spawn board-servability and mid-flight spawn recognition

6.8 No Work Needed Path

Trigger: Claude outputs both FABRIK_STAGE_COMPLETE and FABRIK_NO_WORK_NEEDED in the same invocation output. Expected most commonly from the Plan stage when Research findings show no code or documentation changes are needed.

Key invariant: FABRIK_NO_WORK_NEEDED alone has no effect. Both markers must co-occur. This keeps Claude honest — the emitting stage must declare itself complete before the bypass fires.

Marker priority: This path fires in the completed && noWorkNeeded branch, which precedes the plain completed branch in processItem()’s dispatch chain. This ensures PR creation (ensureDraftPR) and markPRReady — both in the plain completed branch — are not called.

Mutual exclusivity: FABRIK_NO_WORK_NEEDED is mutually exclusive with FABRIK_BLOCKED_ON_INPUT (structurally: blockedOnInput is only reached when completed is false).

Durable marker (fabrik:awaiting-done), written before anything else. The instant processItem decides completed && noWorkNeeded, handleNoWorkNeeded writes fabrik:awaiting-done as its very first mutation — before clearing any orphaned fabrik:awaiting-input, before the emitting stage’s stage:<X>:complete label, before anything else (idempotent — a no-op if already present, e.g. on a retried invocation). This closes a real failure window: handleNoWorkNeeded’s original form made upward of ten sequential GraphQL calls, any of which could be the one that first hits an exhausted rate limit (the observed real-world trigger — #981). Writing the marker unconditionally first is the only placement that survives every plausible failure point, including a full engine restart — see ADR-060 for why this must be a durable GitHub label rather than an itemstate.Store mutation (which does not persist across restarts).

Dispatch suppression, independent of board column. While fabrik:awaiting-done is present, itemMayNeedWork and itemNeedsWork (engine/item.go) return false for every non-cleanup stage — checked immediately after each function’s stage == nil guard, ahead of the HoldingStage check — regardless of item.Status. The item may still be sitting in whichever column the emitting stage ran in, because the board move to Done is exactly the step that may be outstanding. Cleanup stages (CleanupWorktree: true) are exempt so Done’s own worktree cleanup can still run once the marker is eventually cleared.

Code path: processItem() → handleNoWorkNeeded() (writes the marker, then delegates) → settleNoWorkNeeded() (idempotent; does the rest of the work) — and, on retry, the no-work-needed settle scan (settleNoWorkNeededScan, engine/poll_settle.go) calls settleNoWorkNeeded() directly.

handleNoWorkNeeded flow:

  1. If fabrik:awaiting-done is not already present, add it.
  2. Delegate to settleNoWorkNeeded.

settleNoWorkNeeded flow — idempotent; safe to call repeatedly, every sub-step checks current state before mutating:

  1. Only if item.Status != "Done" (see below for why this guards the whole block): clear any orphaned fabrik:awaiting-input label (same rationale as handleStageComplete).
  2. Add stage:<emitting>:complete if not already present (hasLabel check) — same item.Status != "Done" guard.
  3. Find doneOrder: iterate e.cfg.Stages for the stage with CleanupWorktree == true; fall back to math.MaxInt if none exists.
  4. For each stage with Order > emitting.Order && Order < doneOrder:
    • Add stage:<name>:complete if not already present (checked independently per stage).
    • Post the one-line “skipped” comment (_Skipped: no work needed (FABRIK_NO_WORK_NEEDED emitted by <emitting stage>)._, no rocket reaction) — but only once per decision, not once per stage. Every skip comment for a given decision carries identical text (it names the emitting stage, not the individual skipped stage — unchanged from the original behavior), so hasSkippedComment(item, emittingStage.Name) is checked once before the loop: if any skip comment already exists, the full comment set is assumed already posted.
  5. If any label/comment sub-step failed in this pass, stop here — do not attempt the status move or the close. Record a retry (below) and return; the next settle pass re-attempts only whatever is still missing.
  6. Move the issue to Done via UpdateProjectItemStatus, unless item.Status is already "Done" (a prior pass already succeeded). On failure, record a retry and return — no PR is created, and CloseIssue is not attempted, preserving the pre-existing invariant (TestHandleNoWorkNeeded_CloseIssueNotCalledOnStatusFailure) that the issue is never closed before the status move succeeds.
  7. Close the issue via CloseIssue, unless item.IsClosed is already true. On failure, record a retry and return.
  8. Once both the status move and close have succeeded (or were already true), clear the durable marker: remove fabrik:awaiting-done and reset the retry counter.

Why steps 1–4 are skipped once item.Status == "Done": the Done move (step 6) is only ever attempted after every label/comment sub-step in steps 1–4 has already succeeded in that same pass (step 5 gates it), so item.Status == "Done" is itself proof that steps 1–4 already landed — there is nothing left for them to do. This isn’t just an optimization: skipping them also sidesteps a stage-resolution hazard that would otherwise corrupt state on a retry. The settle scan (below) re-derives stage from item.Status, and once the board has moved to "Done", that resolves to the cleanup stage itself, not the original emitting stage. Running steps 1–4 with that wrong stage would add a spurious stage:Done:complete label — before the real Done-stage worktree cleanup has ever run for the item — which itemNeedsWork’s CleanupWorktree branch (§1.1) reads as “cleanup already complete,” permanently skipping normal cleanup dispatch and leaving the worktree janitor (§8) as the only remaining path to reap it.

Retry-owner: the no-work-needed settle scan, settleNoWorkNeededScan (engine/poll_settle.go; called from poll() in engine/poll.go). Runs unconditionally once per poll, immediately after runValidatePRTerminalAdvance (ADR-057) — the closest existing structural analog, since it too must act independent of item.Status/current column and idempotently fill in whatever a prior partial pass left missing. Sourced directly from board.Items, not deepFetchCandidates (see #1220): itemMayNeedWork returns false for exactly the items this scan exists to retry — a non-cleanup-stage item carrying fabrik:awaiting-done — so a candidate list built through that filter would drop the item before the scan ever saw it. This is the same precedent already established by settleMergeTrainMemberCloses, settleNonDefaultBaseCloses, and settleChildPlacements (§6.9): retry eligibility and dispatch eligibility are different questions, and itemMayNeedWork/itemNeedsWork answer only the latter. For every item in board.Items carrying fabrik:awaiting-done and not fabrik:paused (an already-escalated or independently-paused item is left alone — see below): resolve stage := stages.FindStage(e.cfg.Stages, item.Status). If no stage matches the current column (e.g. the board column was renamed away), record a retry without calling settleNoWorkNeeded — there is no safe stage to derive the skip-list from. Otherwise, unless item.Status == "Done", perform a targeted inline FetchItemDetails call and write the result through via itemstate.ItemDeepFetched — this item was never deep-fetched by the normal per-poll pipeline (that’s precisely why it’s stranded), and settleNoWorkNeeded’s hasSkippedComment idempotency check (below) needs item.Comments populated to avoid re-posting the “skipped” comment set on every retry pass; a failed fetch records a retry (via itemstate.DeepFetchFailed) and skips this item for the pass without calling settleNoWorkNeeded. The fetch is skipped once item.Status == "Done", since hasSkippedComment is never consulted in that case (only the close-issue retry remains) and the extra GraphQL call would be wasted on the already-working “Done move succeeded, only CloseIssue failed” sub-case. Finally call settleNoWorkNeeded(board, item, stage) — note that once item.Status == "Done", the resolved stage here is the cleanup stage, not the emitting stage, but per the previous paragraph settleNoWorkNeeded no longer uses it for anything in that case.

Retry counting and escalation. Every failed settle pass calls recordNoWorkNeededRetry, which increments the existing itemstate.StageRetryIncremented/Attempts counter — but keyed by the dedicated constant "__no_work_needed__", not the emitting stage’s real name (the emitting stage’s own counter is cleared, not reused, immediately before handleNoWorkNeeded is called, so reusing it would be safe but conflates two different failure semantics; the constant is also unrepresentable as a real YAML stage name:, so it can never collide). Once Attempts("__no_work_needed__") >= e.cfg.MaxRetries, escalateNoWorkNeededFailure fires — mirroring escalatePRCreationFailure/escalateFailedStage (§5.5, §7.2): adds fabrik:paused, removes fabrik:awaiting-done (dispatch is already suppressed by the pre-existing paused-item gate once fabrik:paused is present, so the marker is no longer needed, and removing it prevents the settle scan from fighting an operator investigating the issue), posts an explanatory comment naming the failure and giving manual recovery steps (gh issue close <N> --repo <owner>/<repo> plus a note that the board column must be moved to Done by hand — there is no simple gh one-liner for Projects-v2 status fields, unlike the gh pr create workaround in §5.5), and applies itemstate.EnginePaused.

Marker clearing. fabrik:awaiting-done is removed in exactly two places: clearNoWorkNeededMarker (a fully successful settle pass — the normal, expected path) and escalateNoWorkNeededFailure (giving up after MaxRetries — the issue is paused for human intervention instead). It is never removed by cleanupClosedIssueTransientLabels’s closed-issue defensive sweep (§1.4), unlike most other gate labels: if a no-work-needed issue is closed out-of-band (e.g. manually) while retries are still below MaxRetries, stripping the marker prematurely would silently resurrect the exact bug this mechanism exists to prevent. The settle scan is itself closed-issue-aware (its item.IsClosed check in step 7 above skips the now-redundant CloseIssue call and still completes the status move), so it is the sole self-healing path for this label — not the sweep.

State transitions:

Before Trigger After Labels Added Labels Removed
Column <X>, Locked + In Progress FABRIK_STAGE_COMPLETE + FABRIK_NO_WORK_NEEDED Column <X>, Awaiting Done (marker present; board move outstanding) fabrik:awaiting-done, stage:<X>:complete, stage:<Y>:complete … (all subsequent non-cleanup stages) fabrik:locked:<user>, stage:<X>:in_progress
Column <X>, Awaiting Done Settle pass succeeds (status move + close) Done, Pending Cleanup — fabrik:awaiting-done
Column <X>, Awaiting Done Settle pass fails Attempts >= MaxRetries times Column <X>, Paused fabrik:paused fabrik:awaiting-done

References: ADR-045: No Work Needed Marker, ADR-060: Durable No-Work-Needed Marker

6.9 Child Board-Placement Retry

Trigger: spawnChildren (§6.7) creates a child issue, adds it to the project board, and links it as blockedBy of the parent — all of which must succeed, or the parent itself is paused and the whole spawn aborts (§6.7’s fatal branches). Only the final per-child step, moving the new board item’s Status to Specify (or the first non-Backlog, non-terminal column) via UpdateProjectItemStatus, can fail without aborting the spawn: the child, its board item, and its blockedBy link already exist by the time this call is attempted, so a failure here is recoverable in place rather than a reason to tear down what has already been created.

How this compares to §6.8’s settle scan. fabrik:awaiting-done (§6.8) is written on an item that is still sitting in the column of the real, matched stage that emitted the decision — stages.FindStage always resolves a non-nil stage for it. A child whose placement failed is different in kind: by construction it is sitting in whichever column GitHub defaulted it to when placement failed — typically Backlog, a column with no configured stage at all, so stages.FindStage returns nil for it. Both cases nonetheless fail itemMayNeedWork/itemNeedsWork and are absent from deepFetchCandidates — one at the stage == nil check (before either function even inspects labels, engine/item.go), the other at the explicit fabrik:awaiting-done label check a few lines later — so neither settle scan can be sourced from deepFetchCandidates. Both are, accordingly, sourced directly from board.Items (§6.8 was fixed to match this scan’s shape in #1220, having originally and incorrectly assumed it was already deepFetchCandidates-reachable). The one remaining difference: this scan needs no deep fetch at all — the shallow board query already includes everything an ordinary retry pass needs (ItemID, Status, Repo, Number, Labels) — whereas §6.8’s scan performs a conditional, per-item deep fetch to populate item.Comments for its comment-based idempotency check, which this scan has no equivalent of. itemMayNeedWork/itemNeedsWork are not modified by either fix — this marker’s retry work is independent of stage dispatch, so routing it through the dispatch pre-filter would solve an already-solved problem the wrong way.

Durable marker (fabrik:awaiting-placement), written on the child. recordChildPlacementFailure adds this label to the child issue (not the parent) at the UpdateProjectItemStatus call site in spawnChildren, covering all three failure branches: the call itself erroring, e.statusField being nil (status-field metadata never populated), and no suitable status option existing on the board (no exact "Specify" match and no non-Backlog, non-terminal fallback column). As with fabrik:awaiting-done (ADR-060), this must be a durable GitHub label rather than an in-memory itemstate.Store mutation: there is no idempotent artifact to safely redo on restart — a lost marker would silently resurrect the stall, exactly the failure mode this mechanism exists to prevent.

Code path: preImplement() → spawnChildren() (writes the marker on failure) — and, independently, the child board-placement settle scan (settleChildPlacements, engine/poll_settle.go) calls settleChildPlacement() every poll for any item carrying the marker.

Settle-scan flow (settleChildPlacements, engine/poll_settle.go; called from poll() in engine/poll.go, immediately after cleanupClosedIssueTransientLabels):

  1. Iterate board.Items directly. Skip any item without fabrik:awaiting-placement, and skip any item also carrying fabrik:paused (either already escalated — marker already removed, so this is defense-in-depth — or an operator is independently investigating; the scan must not fight them).
  2. Closed-child short-circuit: if item.IsClosed, call clearChildPlacementMarker directly and move on — no placement attempt, no escalation, no comment. A closed child needs no further board dispatch; the only purpose of correct placement was to let the pipeline process it, which no longer applies once the child is closed (manually, or resolved out-of-band). This is a departure from §6.8’s precedent: settleNoWorkNeeded treats issue-closed as a step to still complete (the Done move remains required work even on a closed issue), but this marker’s precondition (the child needs dispatching) is simply moot once closed.
  3. Otherwise call settleChildPlacement(board, item).

settleChildPlacement flow:

  1. Snapshot e.statusField under e.mu (mirrors spawnChildren’s own snapshot pattern) and call resolveSpecifyOptionID.
  2. If no option resolves (nil statusField, or no suitable column), record a retry and return — no API call attempted. e.statusField self-heals with no special-casing: it is re-populated whenever nil on any poll (existing poll.go startup/refresh behavior), so this failure branch requires no extra logic beyond “try again next pass.”
  3. Otherwise call UpdateProjectItemStatus(board.ProjectID, item.ItemID, sf.FieldID, optionID). On failure, record a retry and return.
  4. On success, clear the durable marker and the retry counter.

No idempotency short-circuit (e.g., checking item.Status before calling UpdateProjectItemStatus) is performed — the call is idempotent, and once the marker is cleared the scan simply stops revisiting the item, so a pre-check would be an optimization without a correctness benefit.

Retry counting and escalation. Every failed settle pass calls recordChildPlacementRetry, which increments the existing itemstate.StageRetryIncremented/Attempts counter — keyed by the dedicated constant "__child_placement__" (mirroring "__no_work_needed__"’s double-underscore convention: unrepresentable as a real YAML stage name:, so it can never collide with a configured stage’s own counter, nor with noWorkNeededRetryStage). Once Attempts("__child_placement__") >= e.cfg.MaxRetries, escalateChildPlacementFailure fires:

  1. Adds fabrik:paused to the child.
  2. Removes fabrik:awaiting-placement from the child.
  3. Posts an explanatory comment on the child naming the failure and manual recovery steps (move the board column to Specify/first-processing by hand, then remove fabrik:paused; notes that a board-configuration problem — no suitable column — may be the underlying cause).
  4. Applies itemstate.EnginePaused keyed by "__child_placement__".
  5. Calls notifyParentOfStalledChild — a best-effort attempt to also comment on the parent issue (see below). This step runs last and its outcome never affects steps 1–4, which have already completed.

Parent notification and its link-recovery mechanism. Unlike fabrik:awaiting-done (which escalates on the same issue it’s already operating on), this marker’s escalation must also reach a different issue — the parent — since the parent has no other visibility into why it remains blocked (via blockedBy) on a child that will never close. There is no structured child→parent link: the only edge is the free-text back-reference childFooter writes into the child’s Body at spawn time ("...Spawned by Fabrik from parent issue owner/repo#N..."). notifyParentOfStalledChild therefore:

  1. Lazily deep-fetches the child’s Body via e.readClient.FetchItemDetails — this is the only place in the child board-placement retry path that needs a deep fetch; every ordinary settle pass above uses only shallow board.Items fields.
  2. Regex-parses the parent’s owner/repo#number out of the footer text via parseParentFromChildBody.
  3. If the deep-fetch fails, or the footer cannot be found (e.g. a human edited the child’s body and removed it), logs a warning and returns — silently, without posting anything.
  4. Otherwise posts a best-effort comment on the parent naming the stalled child and pointing at the child’s own escalation comment for recovery steps. A failure to post this comment is logged and swallowed.

This is a deliberately fragile-by-design, best-effort mechanism: a parameterized label carrying the parent link verbatim (set alongside fabrik:sub-issue at spawn time) would be more robust against body edits and cheaper (no deep-fetch needed at escalation), but the spec frames parent notification as best-effort only, and adding a new label kind for a once-per-child, escalation-only lookup was judged disproportionate (ADR-062).

Marker clearing. fabrik:awaiting-placement is removed in exactly two places: clearChildPlacementMarker (a successful settle pass, or the closed-child short-circuit) and escalateChildPlacementFailure (giving up after MaxRetries — the child is paused for human intervention instead). It is never removed by cleanupClosedIssueTransientLabels’s closed-issue defensive sweep (§1.4) — for the same reason as fabrik:awaiting-done: if the sweep stripped the marker before the settle scan’s own closed-child short-circuit ran, a closed child mid-retry (below MaxRetries) would silently lose the marker without ever going through the deliberate, observable clear path. In practice this is a narrow window (the settle scan’s closed-child branch runs every poll, same as the sweep), but the exclusion keeps the marker’s removal paths exhaustively enumerable, matching §6.8’s own rationale.

No change to spawnChildren’s own idempotency guard from this section’s fix. fabrik:children-spawned continues to be added to the parent unconditionally once all children are created, added to the board, and linked — regardless of each individual child’s board-placement outcome. This section only makes the child’s own placement durable/recoverable; it does not make the parent’s guard conditional on it (per the issue’s explicit scope). (A separate mechanism, ADR-1583/§6.7, does change what a retried spawn does before this guard is ever reached — see “Resume-safe retry” in §6.7’s Flow step 4 — but that fix is orthogonal to board-placement recoverability, which this section covers.)

State transitions:

Before Trigger After Labels Added Labels Removed
(child created, board item added, blockedBy linked) UpdateProjectItemStatus fails (any of 3 branches) Column Backlog (or wherever GitHub defaulted it), Awaiting Placement fabrik:awaiting-placement —
Column <X>, Awaiting Placement Settle pass succeeds Column Specify (or first processing stage) — fabrik:awaiting-placement
Column <X>, Awaiting Placement Child observed closed Column <X> (unchanged), closed — fabrik:awaiting-placement
Column <X>, Awaiting Placement Settle pass fails Attempts >= MaxRetries times Column <X>, Paused fabrik:paused fabrik:awaiting-placement

References: ADR-060: Durable No-Work-Needed Marker, ADR-062: Child Board-Placement Retry Marker


6.10 Merge-Train Singleton Member-Issue Close Retry

Trigger: landSingleton (the merge-train one-at-a-time landing fallback, §1.3) calls e.client.CloseIssue for the member issue as its last action, after the PR merge, the Queued→Done board move, and the member-PR close have already succeeded. This call fails (e.g. rate limit, transient API error).

Key difference from §6.8’s fabrik:awaiting-done: that marker is written before a chain of up to ten sequential calls, because any of them could be the one that first hits a rate limit. Here there is exactly one at-risk call, positioned after every other side effect of landSingleton has already landed — so the marker is written only in the failure branch, not unconditionally beforehand (ADR-061).

Durable marker (fabrik:awaiting-member-close), written only on failure. markMergeTrainMemberCloseOutstanding adds the label (idempotently — a no-op if already present) only inside the if closeErr != nil branch of the member-issue close. landSingleton’s two calls immediately following the close — resetEjectionCount and resetTrialCounter — run unconditionally regardless of the close’s outcome, exactly as before this fix; the retry mechanism lives entirely outside landSingleton’s call frame and never re-enters it.

No dispatch-suppression wiring — deliberately. Unlike fabrik:awaiting-done, this marker is not checked by itemMayNeedWork/itemNeedsWork, and is not added to transientLifecycleLabels. By construction, the item has already reached its terminal singleton-landing outcome (Status == "Done" in the common case; Status == "<HoldingStage>"/Queued in the rarer case where the Done-move itself also failed) by the time this marker can exist — there is no per-stage redispatch this marker needs to prevent. HoldingStage items are never individually dispatched by itemMayNeedWork/itemNeedsWork regardless (“batch-scoped, not per-item”), and Done-column items are gated by the ordinary stage:Done:complete check in itemNeedsWork’s CleanupWorktree branch, untouched by this marker. This sidesteps the terminal-skip/transientLifecycleLabels interaction that a naive verbatim port of §6.8’s shape would have hit (see ADR-061 Context): once stage:Done:complete lands — which happens independently of this marker, on essentially the same poll — isTerminalPredicate (#689) would otherwise permanently skip the item from all further polling unless the marker were added to transientLifecycleLabels. Not depending on deepFetchCandidates/itemMayNeedWork at all avoids that dependency entirely.

Code path: landOneAtATime → landSingleton (writes the marker on close failure) — and, on retry, settleMergeTrainMemberCloses → settleMergeTrainMemberClose (both engine/merge_train_member_close_settle.go; settleMergeTrainMemberCloses is called from poll() in poll.go) directly.

Retry-owner: settleMergeTrainMemberCloses (engine/merge_train_member_close_settle.go; called from poll() in poll.go). Runs unconditionally once per poll, immediately after handleMergeTrainBatch — independent of merge_train: on/off, so a marker written while merge-train was enabled keeps draining even if the setting is later turned off. It iterates the raw board.Items (not deepFetchCandidates — this scan has no dependency on the deep-fetch/terminal-skip machinery at all). For every item carrying fabrik:awaiting-member-close and not fabrik:paused (mirroring the no-work-needed settle scan’s own paused-item guard — an operator investigating a paused item must not be fought), it calls settleMergeTrainMemberClose.

settleMergeTrainMemberClose flow — idempotent; safe to call repeatedly:

  1. If item.IsClosed is already true (the member PR’s own Closes #N, the landing PR’s Closes #N, or a prior settle pass already closed it — or GitHub’s own auto-close finally landed), skip the redundant CloseIssue call and go straight to clearing the marker.
  2. Otherwise call CloseIssue. On success, clear the marker. On failure, record a retry and stop — the next settle pass re-attempts.

Retry counting and escalation. Every failed settle pass calls recordMergeTrainMemberCloseRetry, which increments the existing itemstate.StageRetryIncremented/Attempts counter — keyed by the dedicated constant "__merge_train_member_close__" (same double-underscore-wrapped, YAML-unrepresentable shape as "__no_work_needed__", so it can never collide with a configured stage’s own counter). MaxRetries <= 0 means unlimited retries, never escalate (same guard as recordNoWorkNeededRetry). Once Attempts("__merge_train_member_close__") >= e.cfg.MaxRetries, escalateMergeTrainMemberCloseFailure fires — mirroring escalateNoWorkNeededFailure: adds fabrik:paused, removes fabrik:awaiting-member-close, posts an explanatory comment with the manual recovery step (gh issue close <N> --repo <owner>/<repo>), and applies itemstate.EnginePaused.

Marker clearing. fabrik:awaiting-member-close is removed in exactly two places: clearMergeTrainMemberCloseMarker (a fully successful settle pass, or an already-closed issue found on a later pass — the normal, expected path) and escalateMergeTrainMemberCloseFailure (giving up after MaxRetries). It is not referenced by cleanupClosedIssueTransientLabels’s defensive sweep at all — unlike fabrik:awaiting-done, there is no “silently resurrect the bug” risk from an early strip, since the settle scan’s own item.IsClosed check already treats a closed issue as fully settled; the label is simply informational once the issue is closed, and clearMergeTrainMemberCloseMarker removes it on the very next poll regardless.

Scope: this mechanism covers only landSingleton’s member-issue close. landMergeTrainBatch’s structurally identical member-issue close (§1.3, FR-3 step d) is a deliberately deferred follow-up (ADR-061 §Sibling Audit) — the settle/escalate helpers are written generic over gh.ProjectItem so that follow-up can reuse mergeTrainAwaitingMemberCloseLabel/settleMergeTrainMemberCloses directly rather than duplicating them.

State transitions:

Before Trigger After Labels Added Labels Removed
Done (or Queued, if the Done-move itself also failed) landSingleton’s member-issue CloseIssue fails Same column, awaiting member close (marker present) fabrik:awaiting-member-close —
Same column, awaiting member close Settle pass succeeds (issue closed, or already closed) Same column, settled — fabrik:awaiting-member-close
Same column, awaiting member close Settle pass fails Attempts >= MaxRetries times Same column, Paused fabrik:paused fabrik:awaiting-member-close

References: ADR-060: Durable No-Work-Needed Marker, ADR-061: Merge-Train Singleton Member-Issue Close Retry


6.11 Closed-Item-At-Any-Stage Advance To Done

Trigger: An issue is closed (manually, or via GitHub’s own Closes #N auto-close on PR merge) while its board item sits at any column other than Done — Specify, Plan, Implement, Review, Backlog, a Holding stage (e.g. Queued), any other non-terminal configured column, or a column with no matching stage config at all (Backlog, or any custom/extra column). itemMayNeedWork/itemNeedsWork (engine/item.go:118/:195) admit a closed item only if the current stage is a cleanup stage or carries stage:<stage>:complete (§2.8, ADR-1387) — a closed item at Validate lacking stage:Validate:complete is, like any other incomplete non-cleanup stage, deliberately not admitted, but unlike every other such stage (which §6.11 below handles), Validate is the domain of §6.15’s settleClosedValidateAdvance instead — excluded from this scan (stage.Name == "Validate", step 3 of the Flow below — narrowed from the broader stageIsGateChecked(stage) in an ADR-1387 follow-up, see below) specifically so the two settle-owners never race. A closed item at any other stage matches neither admission condition and is dropped by the admission guard — it is never dispatched again, so its worktree is never reaped and it never gets archived. This includes items closed while sitting in Queued: the merge train’s own landing paths (landSingleton/landMergeTrainBatch) advance a batch member to Done as part of landing, but an item can reach Queued and then be closed some other way (a human gh pr merge outside the train, or — before issue #1072’s NextStage fix — the generic terminal-advance leak) with no train worker ever picking it up.

Why this can’t reuse §6.10’s (runValidatePRTerminalAdvance) shape verbatim. runValidatePRTerminalAdvance (ADR-057) already handles this exact transition, but is deliberately scoped to stage.Name == "Validate" only (fills gate-checked completion labels in ascending Order, clears gate labels, calls advanceToNextStage). A closed item at Specify/Plan/Implement/Review never reaches that function’s per-item loop at all — it iterates deepFetchCandidates, and a stranded closed item at a non-admitted stage never reaches deepFetchCandidates in the first place, for the identical reason §6.9’s child-placement item never does. Widening runValidatePRTerminalAdvance itself to non-Validate stages was rejected (ADR-064): its completion-label-filling logic exists specifically because Validate is gate-checked and downstream tooling depends on the gate-checked stages’ stage:X:complete labels being present; a closed item at an ordinary, non-gate-checked stage has no such labels to fill and no such downstream dependency — it only needs to reach Done so the existing cleanup machinery there takes over.

Why no durable marker is needed here — unlike every sibling in §6.7–6.10. Every other settle-owner in this section keys on a GitHub label because the failure it recovers from happens mid-sequence (a spawn step, a close call, a Done-move) and the marker records “this specific side effect is still outstanding” so a restart doesn’t lose the decision. This scan has almost no such sequence to protect: item.IsClosed && !(stage != nil && (stage.CleanupWorktree || stage.Name == "Validate")) — plus one additional in-memory (not durable) check for Holding stages, below — is otherwise a pure function of durable, re-derivable board state (the issue’s closed status and its current column), so nothing is lost on an engine restart; the predicate re-evaluates identically from board.Items on every poll. The predicate itself doubles as the idempotency check: once the item reaches the cleanup column, its resolved stage has CleanupWorktree true and the scan naturally stops touching it — no separate item.Status != "Done" comparison, and no marker to write or clear. Note the guard is intentionally stage != nil && (...), not stage == nil || (...): a column with no matching stage config (Backlog, or any custom/extra column) has nothing to protect it from advancing — it has no worktree and no stage bookkeeping — so it is treated the same as any other eligible column, not skipped.

Holding stages are a conditional exception, not a blanket skip (issue #1072). A closed item resolved to a HoldingStage (e.g. Queued) is skipped only while mergeTrainWorkerActive(repoKey) reports a merge-train worker currently in flight for that item’s owner/repo. mergeTrainWorkerActive reads Store.RepoWorkerActive(repoKey) (issue #1222) — the same repo-scoped in-memory liveness registry inside itemstate.Store that the auto-upgrade idle guard reads via Store.HasInFlightWorker() (§9.2), populated by dispatchMergeTrainWorker’s Store.EnterRepoWorker call and cleared by finishTrain’s Store.ExitRepoWorker call on worker exit (ADR-067) — in lockstep with the older Engine.mergeTrainInFlight sync.Map, which dispatchMergeTrainWorker/finishTrain still populate/clear for the atomic duplicate-launch claim, but which no consumer reads for a liveness answer anymore. This is the one real race a Holding-stage item poses that no other stage does: it can be closed-without-merging while still a live batch member mid-assembly or mid-bisection, and yanking it to Done out from under the worker would corrupt that in-flight batch. The check is deliberately in-memory rather than a durable label — the Store’s repo-worker registry only ever reflects “is a goroutine for this repo running right now,” which by construction cannot survive (or need to survive) a restart: a restarted engine has no in-flight worker for any repo, so every closed Holding-stage item is immediately eligible again, which is exactly correct. Once no worker owns the repo, item.IsClosed alone is sufficient — the same accepted predicate as any other stage (ADR-064) — with no PR-merge re-confirmation, since a closed issue at a non-terminal column is itself the durable, sufficient signal regardless of column. See ADR-1072 and ADR-1222.

Deliberately label-state-agnostic. Unlike every other settle-owner, this scan is not conditioned on fabrik:paused, fabrik:awaiting-input, fabrik:blocked, or any other in-flight label. A closed issue at a non-terminal column is itself a terminal state that supersedes any gate/lock label — no further pipeline work can occur on a closed issue sitting outside Done regardless of what else is set on it, so there is nothing for those labels to protect against here (ADR-064).

Code path: the settle scan, settleClosedItemsToDone (engine/closed_item_advance_settle.go), called unconditionally every poll from poll.go, immediately after the child board-placement settle scan (§6.9). Auto-archiving of Done items is covered separately by §6.12.

Flow:

  1. Resolve the cleanup (Done) stage via cleanupStage(e.cfg) — the lowest-Order stage with CleanupWorktree: true, mirroring holdingStage’s shape but by explicit min-Order scan (the same convention settleNoWorkNeeded uses) rather than first-encountered, since e.cfg.Stages could theoretically contain more than one cleanup-flagged stage. If none is configured, the scan is a no-op for the whole poll — no panic.
  2. Iterate board.Items directly — not deepFetchCandidates — for the same sourcing reason as §6.9/§6.10: a stranded closed item at a non-admitted stage never reaches deepFetchCandidates. All fields the predicate needs (IsClosed, Status, Labels) are present on the shallow board query already; no deep fetch is required.
  3. For each item: resolve its current stage via stages.FindStage(e.cfg.Stages, item.Status). Skip if !item.IsClosed, or if a resolved stage is stage.CleanupWorktree or stage.Name == "Validate" (excludes Validate specifically — left to runValidatePRTerminalAdvance/settleClosedValidateAdvance, preventing any double-advance/race between the two settle-owners; not stageIsGateChecked(stage), a broader category that also matches any other stage configured with wait_for_reviews: true, such as the shipped default Review stage — see the ADR-1387 follow-up note below). If the resolved stage is stage.HoldingStage, skip only when mergeTrainWorkerActive(repoKey) is true for the item’s owner/repo — otherwise fall through and advance it like any other eligible stage. A stage == nil column (Backlog, or any custom/extra column) is not skipped — it is treated as eligible, since it has no worktree and no stage bookkeeping to protect.
  4. Otherwise call advanceClosedItemToDone, which mirrors advanceToQueued’s shape: look up the cleanup stage’s status option, call UpdateProjectItemStatus, write through boardcache.CacheImpl.UpdateItemStatus, and register the webhook echo. No completion label is added — unlike advanceToQueued’s stage:Validate:complete, this scan has no per-stage bookkeeping to do; landing the item at the cleanup column is sufficient for the existing CleanupWorktree dispatch branch (engine/item.go) to take over on a later poll (worktree reaping when a worktree exists, stage:Done:complete labeling regardless — see the correction below).
  5. On any failure (e.statusField == nil, no matching status option, or the API call itself failing), log a warning and move on — no retry counter, no escalation. The next poll re-derives the same predicate from board.Items and retries automatically; there is nothing to lose between polls, so a bare retry-forever loop is sufficient (ADR-064; contrast with §6.7–6.10’s MaxRetries/escalation machinery, which exists to protect an at-risk sequence of calls or a durable marker this scan does not have).

Downstream: the cleanup-stage admission gates explicitly admit worktree-less closed items (#1224). Once an item lands at the cleanup column via this scan, it must still pass itemMayNeedWork/itemNeedsWork before handleCleanupStage (which adds stage:Done:complete) ever runs. Before #1224, both gates’ CleanupWorktree branches keyed solely on worktreeExistsForItem — correct for an item that reached Done via the normal pipeline (Implement always creates a worktree), but this scan’s whole reason for existing is to land items that may never have had one (a closed issue that never reached Implement, or whose worktree was already reaped). A worktree-less item therefore failed worktreeExistsForItem, was dropped by both gates, and was permanently stranded: un-labelled (no stage:Done:complete) and consequently invisible to §6.12’s archival scan, which requires that exact label. This was a real, load-bearing bug in this scan’s own “sufficient to take over” assumption below, not a hypothetical.

itemMayNeedWork and itemNeedsWork’s CleanupWorktree branches now read, in order: (1) if the completion label (stage:<Done>:complete) is already present, refuse — the item is done, and refusing here (rather than only in itemNeedsWork) is what keeps a labelled worktree-less item cheap to skip on every subsequent poll, matching a worktree-having item’s cost once its worktree is reaped; (2) if a worktree exists on disk, admit — the pre-#1224 behavior, unchanged; (3) otherwise admit only if item.IsClosed && !item.IsPR — the case this scan produces. PR board items are deliberately excluded from step (3) (handleCleanupStage already special-cases !item.IsPR for worktree removal, but widening admission to worktree-less PR items was left as a separate, deferred concern — see #1224’s Scope).

With that fix, this scan’s original design intent now holds: landing an item at the cleanup column — worktree or not — is sufficient for the existing CleanupWorktree dispatch branch to take over on a later poll and apply stage:Done:complete exactly as it does for every other route into Done (§6.10, the normal pipeline advance, settleNoWorkNeeded). This scan closes the reachability gap (worktree leak, permanent dispatch stall); the archival gap — removing the item from the board afterward — is closed by §6.12’s settle scan, which requires no additional wiring here: once this scan lands an item at the cleanup column and handleCleanupStage applies stage:Done:complete, §6.12 picks it up on a later poll like any other Done item, worktree-agnostically (confirmed, not merely assumed — see §6.12’s own note).

Out of scope for #1224 (left for a follow-up): the cold-start/restart path. engine/terminal.go’s isProbeOnlyTerminal/seedTerminalFromProbeItems is a second, independent mechanism that can seed TerminalFlagSet{Terminal: true} for a closed, CleanupWorktree-stage, worktree-absent item using only probe-only data — which does not include labels — bypassing itemMayNeedWork/itemNeedsWork entirely (selectDeepFetchCandidates skips any item already flagged terminal before either gate runs). If an engine restart observes a closed, worktree-less item this scan already landed at Done before handleCleanupStage had a chance to label it, that item can be seeded terminal without ever being labelled — stranding it exactly as before #1224’s fix, via a different trigger (restart racing an unlabelled item, rather than every poll unconditionally). This is a distinct, narrower failure mode from the one #1224 fixes and was deliberately left out of this issue’s scope (see #1224’s Plan-stage Key Decisions); this note exists so the gap is documented rather than silently implied to be closed.

State transitions:

Before Trigger After Labels Added Labels Removed
Column <X> (non-Done, non-cleanup, non-gate-checked), open Issue closed Column <X>, closed — —
Column <X>, closed (including Holding, if no train worker is active for its repo) Settle pass succeeds Column <Done>, closed, worktree present or absent — (stage:Done:complete added separately by the CleanupWorktree dispatch branch on a later poll, once itemMayNeedWork/itemNeedsWork admit the item — see the correction above; admission no longer depends on a worktree existing) —
Column <Done> or Validate, closed Settle pass evaluated No change (out of scope for this scan; Validate is settleClosedValidateAdvance/runValidatePRTerminalAdvance’s exclusive domain, §6.15) — —
Holding column, closed, merge-train worker active for the item’s repo Settle pass evaluated No change (left to the in-flight worker; retried next poll) — —
Column <X>, closed Settle pass fails (transient API error, missing status field/option) Column <X> (unchanged) — —

ADR-1387 follow-up: exclusion narrowed from stageIsGateChecked(stage) to stage.Name == "Validate". Originally this scan excluded any stageIsGateChecked stage, on the theory that gate-checked stages were categorically deferred to the Validate-specific settle-owner pair. But that pair only ever processes stage.Name == "Validate" — and both shipped default stage templates (stages/examples/review.yaml, stages/examples/validate.yaml) set wait_for_reviews: true, making Review gate-checked too. Under the broader exclusion, a closed item stranded at Review with no stage:Review:complete had zero remaining owners: dispatch admission refused it (R1, §2.8), no settle-owner processed anything but Validate, and this scan skipped it as well — a permanent, silent strand, caught in review on PR #1388. The fix is safe because advanceClosedItemToDone (what this scan calls) never inspects the linked PR at all, unlike advanceValidateTerminalItem (§6.15) — it only moves the board column, exactly as it already does, unconditionally, for a closed item at Specify/Plan/Implement. Extending that same behavior to Review needs no new PR-state reasoning, since Fabrik’s own PR merge action (attemptMergeOnValidate) only ever runs as part of Validate completing.

References: ADR-057: Single-Owner Validate PR Terminal Advance, ADR-064: Closed-Item-At-Any-Stage Advance To Done, ADR-1072: Holding Stages Are Reachable Only Via Dedicated Code, Not Positional Advance, ADR-1387: Closed Items Are Never Dispatched (the exclusion-narrowing follow-up described above)


6.12 Done-Item Archival

Trigger: A board item’s current Status resolves to the Done (cleanup) stage and it carries that stage’s completion label (stage:<Done>:complete), and at least ArchiveAfter (default 168h/1 week; --archive-after/FABRIK_ARCHIVE_AFTER) has elapsed since that label was applied. Disabled entirely via --archive-done off / FABRIK_ARCHIVE_DONE=off.

Why this exists. Un-archived Done and closed items accumulate on the project board indefinitely; every one of them is fetched on each poll cycle, inflating per-poll GraphQL cost. A prior implementation, archiveDoneCompleteItems, archived Done items immediately on first observation — with no grace period, completed work appeared to vanish from the board the instant it finished, so its call site was deliberately commented out and the dead function later removed (#1025). This settle scan re-implements the capability with a grace period anchored to a restart-safe, GitHub-side timestamp instead (ADR-068).

Why FetchLabelAppliedAt, not a local timestamp. No GraphQL field exposes “when did Status become Done” directly; ProjectV2Item.updatedAt is a coalesced max of issue/item/linked-PR activity, bumped by unrelated edits, and cannot serve as a “time since Done” measurement. GitHubClient.FetchLabelAppliedAt (github/labels.go) — the same GitHub-side, restart-safe REST lookup already backing the CI-wait gate (§6.5), the review-wait timeout (§6.1’s ReviewWaitTimeout), and the post-Validate convergence budget (§5.5) — reads the actual stage:<Done>:complete label-applied timestamp. A separate, abandoned prior attempt (#687/#688) used a local itemstate field set once at label-observation time, falling back to item.UpdatedAt after a restart; that fallback reintroduces the exact premature-archival risk the grace period exists to prevent, since updatedAt has no contractual link to the Done transition. FetchLabelAppliedAt avoids the fallback problem entirely by having no local state to lose.

Bounding the REST cost. FetchLabelAppliedAt pages through the full issue-events history — not cheap to call every poll for every Done item sitting in its grace window. The computed eligible-at time (appliedAt + ArchiveAfter) is cached in itemstate.CooldownAt under the reason key "archive-eligible-at" after the first successful lookup; every subsequent poll is a pure time.Now() comparison against the cached value, with no REST call. The cache is a cost-bounding optimization only, not a correctness-load-bearing marker: losing it (e.g., on an engine restart, since itemstate.Store is in-memory) costs exactly one extra FetchLabelAppliedAt call, never an incorrect archival.

Bounding the restart burst. The CooldownAt cache above bounds the cost per item, but on the first poll after a restart the cache is empty for every still-Done item at once — without a further cap, a bloated board (this feature’s own target scenario) would fire that many synchronous FetchLabelAppliedAt calls within a single poll. settleArchiveDoneItems tracks a per-poll fetch budget, maxArchiveLabelFetchesPerPoll (default 20, a package-level var so tests can lower it): once exhausted, remaining cache-misses are left uncached and simply retried on the next poll — the same safe fallback as the “not yet known” path below, so this only spreads the burst across polls, never causes early, stuck, or duplicate archival.

Fail-open on unknown timestamp. FetchLabelAppliedAt returns the zero time with a nil error when the label-applied event isn’t found (a deliberate fail-open contract). This scan treats a zero result as “not yet known” — no eligible-at time is cached, and the item is retried on the next poll — so an unknown timestamp can never trigger early archival, only a delayed one.

Code path: the settle scan, settleArchiveDoneItems (engine/archive_done_settle.go), called unconditionally every poll from poll.go, immediately after §6.11’s settleClosedItemsToDone — so an item this same poll advances into Done becomes visible to this scan on the next poll.

Flow:

  1. If ArchiveDone == "off", the entire scan is a no-op for the poll.
  2. Resolve the cleanup (Done) stage via cleanupStage(e.cfg). If none is configured, no-op.
  3. Iterate board.Items directly — not deepFetchCandidates — for the same sourcing reason as §6.9–§6.11: a Done item with its worktree already reaped never passes itemMayNeedWork’s admission guard again, so it is never deep-fetched. Status and Labels are present on the shallow board query already.
  4. For each item where item.Status == cleanup.Name && hasLabel(item.Labels, "stage:<cleanup.Name>:complete"), call maybeArchiveDoneItem, threading a per-poll fetchBudget shared across all items this scan evaluates (see “Bounding the restart burst” above).
  5. maybeArchiveDoneItem first checks itemstate.CooldownAt("archive-eligible-at") for a cached eligible-at time. On a cache miss, it consults fetchBudget: if exhausted, the item is left uncached and retried next poll; otherwise it calls FetchLabelAppliedAt once and decrements the budget. A zero result skips (see above); otherwise it computes appliedAt + ArchiveAfter, caches it via itemstate.CooldownRecorded, and uses it as the eligible-at time for this poll too.
  6. If time.Now() is before the eligible-at time, the item is left alone (still within its grace period).
  7. Otherwise: call ArchiveProjectItem(board.ProjectID, item.ItemID). On success, write through the cache via boardcache.CacheImpl.RemoveItem(item.ItemID) (wrapping store.RemoveByItemID, mirroring the webhook delta path’s "deleted"/"archived" case), and logf the archived issue for operator visibility. On failure, log a warning and retry on the next poll — no retry counter, no escalation. No webhook echo is registered — boardcache/delta.go’s "deleted"/"archived" case never calls matchEchoFn (only "edited" does), so a registered echo here would always expire unmatched and could spuriously trip WebhookStreamUnhealthy on burst-archival; the RemoveItem write-through already gives immediate cache coherence, mirroring the identical no-echo rationale in engine/no_work_needed_settle.go for issue-close.

Idempotency and restart-safety. Archival is terminal: an archived item stops appearing in default board queries, so there is nothing left to re-observe — no marker is needed to prevent double-archival. On an engine restart, the CooldownAt cache is empty, so the very next poll re-fetches FetchLabelAppliedAt once per still-Done item, up to the per-poll fetch budget; if the label was already applied long enough ago, the item archives on that same poll (one extra REST call, not one extra full grace-period wait). A restart during the grace-period wait therefore costs at most one extra FetchLabelAppliedAt call (spread across polls if the budget is exceeded) — it can never cause early archival, a stuck item, or a double-archive.

Accepted residual risk: shallow-label truncation. The board query’s labels(first: 30) may not surface the completion label for an item with 30+ labels, silently skipping its archival indefinitely. ADR-021 already accepted this trade-off for the original implementation; mitigating it would require a deep fetch, which would reintroduce the per-poll GraphQL cost this scan exists to eliminate.

Accepted residual risk: manual Done→away→Done round trip. stage:<Done>:complete is never removed by any engine code path (consistent with every other stage:<Name>:complete label, which are permanent provenance markers). If an operator manually drags an item’s Status away from Done and back — there is no engine-driven path that does this — FetchLabelAppliedAt still returns the original label-applied timestamp (no new labeled event exists to reflect it), so the item re-archives on the stale, already-elapsed eligibleAt instead of waiting a fresh grace period. Not mitigated: the trigger is a human manually undoing/redoing a board move on an already-terminal item, not a state the engine itself produces. See ADR-068 for why clearing the CooldownAt cache alone would not fix this either.

Accepted assumption: single cleanup stage. cleanupStage(e.cfg) resolves only the lowest-Order CleanupWorktree: true stage — a board configured with a second cleanup-marked column would have items there never evaluated for archival. Shared with §6.11’s settleClosedItemsToDone; Fabrik’s stage-config convention is a single terminal Done/cleanup stage.

Worktree-less items (#1224): no change needed here. This scan’s eligibility check (item.Status == cleanup.Name && hasLabel(item.Labels, completeLabel)) was already worktree-agnostic by construction — it never inspects the filesystem. §6.11’s admission-gate fix means a worktree-less closed item now reaches this exact same eligible state (Done column + completion label) as any worktree-having item; this scan picks it up identically, with no code change of its own. Confirmed by the end-to-end test covering the full closed-item → Done → labelled → archived chain (engine/worktreeless_done_e2e_test.go), not merely by inspection.

State transitions:

Before Trigger After Labels Added Labels Removed
Done, stage:Done:complete, no cached eligible-at Settle pass: cache miss, FetchLabelAppliedAt returns non-zero Done (unchanged); eligible-at cached in CooldownAt — —
Done, stage:Done:complete, no cached eligible-at Settle pass: FetchLabelAppliedAt returns zero (not found) Done (unchanged); nothing cached — —
Done, stage:Done:complete, eligible-at cached, not yet elapsed Settle pass evaluated Done (unchanged) — —
Done, stage:Done:complete, eligible-at cached, elapsed Settle pass succeeds Archived (removed from board) — —
Done, stage:Done:complete, eligible-at elapsed ArchiveProjectItem call fails Done (unchanged); retried next poll — —
Any non-Done column, or Done without stage:Done:complete Settle pass evaluated No change (out of scope for this scan) — —

References: ADR-021: Housekeeping Mutations on Shallow Data, ADR-068: Done-Item Archive Timing


6.13 Non-Default-Base Explicit Close Retry

Trigger: closeIssueIfNonDefaultBase (§2.8, ADR-1096) calls e.client.CloseIssue as the last step of the outbound explicit-close path, after the caller’s (runValidatePRTerminalAdvance’s cruise path, or advanceConvergedPRToDone’s non-train yolo path) board advance to Done has already succeeded. This call fails (e.g. rate limit, transient API error).

Key difference from §6.8’s fabrik:awaiting-done: that marker is written before a chain of up to ten sequential calls, because any of them could be the one that first hits a rate limit. Here there is exactly one at-risk call, positioned after every other side effect of the caller has already landed — so the marker is written only in the failure branch, not unconditionally beforehand (ADR-1097, mirroring ADR-061).

Durable marker (fabrik:awaiting-close), written only on failure. markNonDefaultBaseCloseOutstanding adds the label (idempotently — a no-op if already present) only inside the if err != nil branch of the explicit close, after the gh.ErrNotFound short-circuit (already-deleted issue is treated as success, never reaches this branch) and after the item.IsClosed short-circuit (already-closed issue never attempts the call at all).

No dispatch-suppression wiring — deliberately. Unlike fabrik:awaiting-done, this marker is not checked by itemMayNeedWork/itemNeedsWork, and is not added to transientLifecycleLabels. By construction, the item has already reached Done (both callers advance the board before calling closeIssueIfNonDefaultBase) by the time this marker can exist — there is no per-stage redispatch this marker needs to prevent. This sidesteps the terminal-skip/transientLifecycleLabels interaction a naive verbatim port of §6.8’s shape would have hit, identical to §6.10’s reasoning (ADR-061 Context).

Code path: runValidatePRTerminalAdvance / advanceConvergedPRToDone → closeIssueIfNonDefaultBase (writes the marker on close failure) — and, on retry, settleNonDefaultBaseCloses → settleNonDefaultBaseClose (both engine/close_nondefault_base_settle.go; settleNonDefaultBaseCloses is called from poll() in poll.go) directly.

Retry-owner: settleNonDefaultBaseCloses (engine/close_nondefault_base_settle.go; called from poll() in poll.go). Runs unconditionally once per poll, immediately after settleMergeTrainMemberCloses. It iterates the raw board.Items (not deepFetchCandidates — this scan has no dependency on the deep-fetch/terminal-skip machinery at all). For every item carrying fabrik:awaiting-close and not fabrik:paused (mirroring settleMergeTrainMemberCloses’s own paused-item guard — an operator investigating a paused item must not be fought), it calls settleNonDefaultBaseClose.

settleNonDefaultBaseClose flow — idempotent; safe to call repeatedly:

  1. If item.IsClosed is already true (a prior settle pass already closed it, or a human closed it manually), skip the redundant CloseIssue call and go straight to clearing the marker.
  2. Otherwise call CloseIssue. On success, clear the marker (and write through the boardcache via ApplyIssueClosed, matching closeIssueIfNonDefaultBase’s own success path). On failure, record a retry and stop — the next settle pass re-attempts.

Retry counting and escalation. Every failed settle pass calls recordNonDefaultBaseCloseRetry, which increments the existing itemstate.StageRetryIncremented/Attempts counter — keyed by the dedicated constant "__non_default_base_close__" (same double-underscore-wrapped, YAML-unrepresentable shape as "__merge_train_member_close__"/"__no_work_needed__", so it can never collide with a configured stage’s own counter). MaxRetries <= 0 means unlimited retries, never escalate (same guard as recordMergeTrainMemberCloseRetry). Once Attempts("__non_default_base_close__") >= e.cfg.MaxRetries, escalateNonDefaultBaseCloseFailure fires — mirroring escalateMergeTrainMemberCloseFailure: adds fabrik:paused, removes fabrik:awaiting-close, posts an explanatory comment naming the merged PR (item.LinkedPRNumber, when non-zero — already populated on the board item, so no new storage is needed to thread it from the original closeIssueIfNonDefaultBase(item, prNumber) call) with the manual recovery step (gh issue close <N> --repo <owner>/<repo>), and applies itemstate.EnginePaused.

Marker clearing. fabrik:awaiting-close is removed in exactly two places: clearNonDefaultBaseCloseMarker (a fully successful settle pass, or an already-closed issue found on a later pass — the normal, expected path) and escalateNonDefaultBaseCloseFailure (giving up after MaxRetries). It is not referenced by cleanupClosedIssueTransientLabels’s defensive sweep at all — unlike fabrik:awaiting-done, there is no “silently resurrect the bug” risk from an early strip, since the settle scan’s own item.IsClosed check already treats a closed issue as fully settled.

Scope: this mechanism covers only the two terminal merge-advance sites’ calls into closeIssueIfNonDefaultBase (the merge-train path and the FABRIK_NO_WORK_NEEDED short-circuit already close explicitly via their own paths and are unaffected — see §2.8). The generic recordSettleRetry/escalateSettle/clearSettleMarker helpers (engine/settle.go) this mechanism reuses are the same ones backing fabrik:awaiting-done (§6.8) and fabrik:awaiting-member-close (§6.10).

State transitions:

Before Trigger After Labels Added Labels Removed
Done closeIssueIfNonDefaultBase’s explicit CloseIssue fails Done, awaiting close (marker present) fabrik:awaiting-close —
Done, awaiting close Settle pass succeeds (issue closed, or already closed) Done, settled — fabrik:awaiting-close
Done, awaiting close Settle pass fails Attempts >= MaxRetries times Done, Paused fabrik:paused fabrik:awaiting-close

References: ADR-1096: Explicit Close on Non-Default-Base Merge, ADR-061: Merge-Train Singleton Member-Issue Close Retry, ADR-1097: Non-Default-Base Explicit Close Retry

6.14 Awaiting-CI Settle Scan

Trigger: Field evidence (issue #1270) showed an item can strand indefinitely with fabrik:awaiting-ci present and a confirmed CI failure: the shared, three-layer admission pipeline that fed the CI gate — itemMayNeedWork (engine/item.go), selectDeepFetchCandidates’s cooldown pre-filter, and the main catch-up loop’s own per-item admission gate — could silently exclude the item from ever reaching checkCIGate, with no log line marking the exclusion. dispatchCIFixReinvoke was never called, MaxCiFixCycles never tripped, and no timeout fired.

Same failure class as four prior fixes. fabrik:awaiting-ci was the one durable “awaiting-X” marker in the codebase still relying exclusively on the shared, admission-gated catch-up path rather than a dedicated board.Items-sourced settle scan — the same shape already fixed for fabrik:awaiting-done (§6.8, ADR-060), spawned-child board placement (§6.9, ADR-062), merge-train singleton member-issue close (§6.10, ADR-061), and non-default-base explicit close (§6.13, ADR-1097). settleAwaitingCIScan is the fifth instance of the pattern.

Code path: settleAwaitingCIScan (engine/ci_settle.go; called from poll() in poll.go, immediately after runValidatePRTerminalAdvance and before settleNoWorkNeededScan).

Not a durable-marker-write-on-failure mechanism, unlike §6.10/§6.13. fabrik:awaiting-ci is applied by handleStageComplete on FABRIK_STAGE_COMPLETE (§6.5.1 Path 1/2) — this scan does not write the marker, it is the marker’s sole evaluator for open items. See §6.5.1 Path 2 for the full evaluation mechanics (admission model, orphan-column handling, and the runCatchUpPhase2 same-poll landing handoff shared with §6.6.6).

Admission independence is the entire fix. The scan iterates the raw board.Items, filtering only on fabrik:awaiting-ci present, fabrik:paused absent, and item.IsClosed false (closed-issue recovery is runValidatePRTerminalAdvance’s job, ADR-056 D2 — duplicating it here would be a second owner for no benefit). It never calls itemMayNeedWork, is never filtered by selectDeepFetchCandidates’s cooldown logic, and does not depend on the item having appeared in deepFetchCandidates this poll at all — so whatever excluded #3915 from that pipeline in the field cannot exclude it from this scan.

No double-dispatch. The main catch-up loop’s per-item admission gate was narrowed to hasComplete-only as part of this fix (§6.5.1). Since fabrik:awaiting-ci and stage:X:complete are mutually exclusive in steady state, an item is always routed to exactly one of the two paths per poll — never both — so CIFixCycles/worker dispatch can never double-fire for the same item in the same poll.

Diagnosability. Every pass logs under the awaiting-ci-settle tag: a cache-hit/deep-fetch pre-check line per item (mirroring selectDeepFetchCandidates’s poll.go pattern); a stray-column skip names the column and the reason it can’t host the CI gate. A summary line reports two counts when non-zero — items examined (every fabrik:awaiting-ci item the scan looked at, including ones stuck in the orphan-column/deep-fetch-failure retry branches) and items that reached the CI gate — kept separate so a poll where every item is retrying without reaching the gate doesn’t read as “0 items, nothing happening.” This directly addresses the issue’s own root-cause complaint — an 80-minute silence with no log line explaining why the item wasn’t progressing. Corrected by #1303 (see §6.14.1): “reached the CI gate” now means exactly that — phase1Ctx.reachedCIGate is set only immediately before handleMergeAndCIGates calls checkCIGate, not merely “made it through catchUpPhase1Handlers” (an earlier handler in the chain, e.g. checkMergeabilityGate’s claim, can still consume an iteration without checkCIGate ever running).

State transitions:

Before Trigger After Labels Added Labels Removed
wait_for_ci stage, awaiting CI Settle pass: CI still pending Unchanged — —
wait_for_ci stage, awaiting CI Settle pass: CI fails Unchanged (or paused at MaxCiFixCycles) fabrik:paused (cycle limit only) —
wait_for_ci stage, awaiting CI Settle pass: CI clears stage:X:complete present; landing decision reached via runCatchUpPhase2 stage:X:complete fabrik:awaiting-ci
Non-wait_for_ci/HoldingStage/Unmanaged/CleanupWorktree column, awaiting CI Settle pass: orphan column detected Unchanged — —
wait_for_ci stage, awaiting CI Settle pass: FetchItemDetails fails with a transient/rate-limit error (isTransientAPIError) Unchanged — deferred, no counter change (issue #1313, see §6.14.2) — —
wait_for_ci stage, awaiting CI Settle pass: FetchItemDetails fails with a non-transient (structural) error Unchanged — counts toward escalation — —
Orphan column or repeated non-transient FetchItemDetails failure, awaiting CI Attempts("__awaiting_ci_orphan__") >= MaxRetries (the two causes share one counter — see below) Paused fabrik:paused fabrik:awaiting-ci

A repeated non-transient FetchItemDetails failure is the other “gate genuinely cannot be evaluated” case (in practice this is permissions, a deleted issue node, or another persistent, per-item error) — the scan can never reach checkCIGate for such an item either, even though it resolves to a perfectly valid wait_for_ci stage. It shares the same __awaiting_ci_orphan__ retry counter and escalateAwaitingCIOrphanFailure escalation path as the orphan-column case (mirroring escalateNoWorkNeededFailure’s precedent of one counter covering multiple failure causes with a generic message), so orphan-column retries and non-transient deep-fetch-failure retries accumulate toward the same MaxRetries threshold regardless of which cause fires on which pass. escalateAwaitingCIOrphanFailure re-resolves the item’s current stage at escalation time so the posted comment names whichever cause is actually current — a stray column, or a persistent non-transient fetch failure — rather than always claiming a stray column. A transient/rate-limit FetchItemDetails failure (issue #1313) never reaches this counter at all — see §6.14.2.

References: ADR-1270: Awaiting-CI Settle Scan, ADR-056 (convergence/settling consolidation), ADR-060 (fabrik:awaiting-done settle scan), ADR-061 (merge-train member-close retry), ADR-1097 (non-default-base close retry), ADR-1216 (§6.6.6, joint-clearing handoff), ADR-1223 (gate-pause-terminates-not-clears), #1135 (orphaned durable labels — same failure class)

6.14.1 Awaiting-CI Handler-Chain Stall (issue #1303)

Trigger: #1270 fixed the admission layer for fabrik:awaiting-ci (§6.14 above) — the scan reliably reaches catchUpPhase1Handlers every poll. Field evidence (fabrik-test-alpha#3930, PR #3932) showed a second, downstream stall: an item with a mergeable PR and a confirmed check-run failure could still sit indefinitely once inside the handler chain, with zero settle/ci-gate log output and a stale fabrik:rebase-needed label never cleared, for over 80 minutes with no CI-fix reinvoke, no pause, and no timeout.

Confirmed root cause: an unbounded cached CheckRunsPending classification. boardcache.CacheImpl.FetchCheckRuns() only forces a live GitHub refetch when the cached classification would read as CheckRunsFailed (the #958 leg 3 guard, §6.4 above) — a cached CheckRunsPending snapshot is served from cache indefinitely. On a webhook-less deployment nothing else ever supersedes it: no check_run event arrives to update the store, so a poll that happens to cache a PENDING snapshot moments before the real checks resolve latches the item permanently. This is a latent race, not a regression — the same code path is unchanged since the last known-good release; it only fires when a poll’s timing catches CI mid-flight, which explains why the field incident was intermittent (1 of 4 runs) rather than universal.

The resulting silent chain: settlePRMergeState()’s CheckRunsPending case returns PRMergeUnsettled → checkMergeabilityGate claims (blocked=true) → handleMergeAndCIGates’s if mergeBlocked { return true } claims the item → checkCIGate (and the 30-minute CIWaitTimeout escalation logic inside it) is never reached. Three of these four steps produced no log line at all before this fix, which is why the incident took over an hour to diagnose from logs alone.

Fix 1 — CacheImpl.RefreshCheckRunsLive(owner, repo, sha) (boardcache/boardcache.go). Superseded by #2180 (ADR 2180): FetchCheckRuns now confirms any cached PENDING or FAILED set live (a PENDING set confirmed within 15 s is served), which makes this per-poll prime redundant; RefreshCheckRunsLive and the settleAwaitingCIScan call are removed. The text below is the as-decided history. An unconditional live check-run fetch that bypasses FetchCheckRuns’s cache-trust check entirely and applies the result into the Store exactly as FetchCheckRuns’s own miss path does. settleAwaitingCIScan calls it once per fabrik:awaiting-ci item per poll — after FetchItemDetails, before the handler chain — guarded on item.LinkedPRHeadSHA/item.LinkedPRNumber being populated. Scoped narrowly to this one caller rather than inverting FetchCheckRuns’s general cache-trust contract: the general contract (still denylist — refetch only on a would-be-FAILED classification) is unchanged for its ~35 other call sites, and TestFetchCheckRuns_CachedPending_ServedFromCache continues to document that contract for everyone else. This closes the confirmed root cause: a stale cached Pending is now primed with genuinely fresh data before checkMergeabilityGate ever evaluates it. Correction (issue #1325, see §6.14.3): this guard’s item.LinkedPRHeadSHA does not always come from “that poll’s own fresh GraphQL deep-fetch” as originally stated here — on the far more common cache-hit path, FetchItemDetails never performs a GraphQL fetch at all, and until #1325 the guard’s SHA field was left unpopulated on that path, making this fix inert almost everywhere it needed to run. See §6.14.3 for the full story.

Fix 2 — an unconditional CIWaitTimeout backstop, ahead of the handler chain (engine/ci_settle.go). Fix 1 makes checkCIGate reachable for the confirmed cause, but does not bound the class: any other future silent claim earlier in the chain would reproduce the same unbounded stall, because the CI-wait timeout logic lives inside checkCIGate, which such a claim would make unreachable. settleAwaitingCIScan now checks fabrik:awaiting-ci’s FetchLabelAppliedAt age against ciWaitTimeout() before building phase1Ctx and running the handler chain at all; once exceeded, it calls pauseForCITimeout directly and escalates regardless of what any gate would otherwise classify or claim. Inert on the happy path — an item that reaches checkCIGate normally hits the identical timeout at the identical threshold; the backstop only fires when the item never gets that far. Correction (issue #1408, see §6.14.4): “idempotent… no-ops” understated what happened when hasCIGatePauseComment already matched — pauseForCITimeout silently did nothing at all (no labels, no return signal), and the backstop continued past the handler chain regardless, permanently stranding any item resumed after this fired. §6.14.4 describes the fix.

Fix 3 — every previously-silent Phase 1 claim path now logs. checkMergeabilityGate’s PRMergeUnsettled and PRMergeQueued cases (engine/merge_gate.go) and handleMergeAndCIGates’s if mergeBlocked { return true } branch (engine/catch_up_handlers.go) each log under their respective merge-gate/ci-gate tags, naming the branch and settle.Reason. handleDependencies and handleReviewGate were audited and already logged on every claim path (checkDependencies’s “waiting for…“/cycle-detection lines; checkReviewGate’s “waiting for reviewers…” and handleBrokenReviewLinkage’s broken-linkage lines) — no changes were needed there. These are logging-only additions; no claim/no-claim return value changed for any existing caller.

Fix 4 — gateReached now measures what its log line claims (engine/ci_settle.go). Before this fix, gateReached incremented unconditionally at the end of every loop iteration in settleAwaitingCIScan, regardless of which handler in catchUpPhase1Handlers claimed the item (or whether any did) — so the summary line (“examined N item(s), N reached the CI gate”) could report an item as having reached the CI gate when checkCIGate never actually ran. This misled triage of the very incident this section documents: an operator initially read “1 reached the CI gate” as proof the gate was being evaluated. phase1Ctx now carries a reachedCIGate bool field, set true by handleMergeAndCIGates immediately before its checkCIGate call — the one call site that matters — and settleAwaitingCIScan reads it after the handler loop instead of incrementing unconditionally.

Fix 5 — the in-flight worker marker can no longer outlive a pre-PID hang (engine/worker_liveness.go). Investigated as a candidate root cause and ruled out for this incident (the field log shows exactly one “already in-flight” skip, not a sustained one), but a real, narrower gap was found and closed: runWorkerDetectorScan previously skipped every WorkerHandle with PID <= 0 unconditionally — correct for the common case (a worker still on its way to onPIDReady), but a dispatch goroutine that hangs before onPIDReady fires (e.g. stuck in ensureRepoReady) never records a PID, so isWorkerStale’s signal-0 liveness check can never confirm it dead, and the marker (plus the fabrik:locked:<user>/stage:<name>:in_progress labels it gates) could outlive the hung goroutine indefinitely, permanently suppressing dispatch for the item. The detector now applies workerStaleTimeout() against w.StartedAt (not LastSignAt, which never starts ticking for a worker that never reached PID assignment) for the PID <= 0 case, and clears via the existing cleanupStaleWorker path on timeout — logged distinctly as “could not verify liveness” (a timeout-based clear, not a signal-0-confirmed-dead clear, since this path can never confirm death directly).

References: ADR-1270: Awaiting-CI Settle Scan (the admission-layer predecessor fix), #958 leg 3 (the pre-existing FAILED-refetch precedent this fix extends the same reasoning to for PENDING, scoped narrowly rather than generally), ADR-1223 (ciTerminated claim-ordering convention, unaffected by this change), issue #1303

6.14.2 Awaiting-CI Settle Scan: Transient vs. Structural FetchItemDetails Failures (issue #1313)

Trigger: Field evidence (fabrik-test-alpha#3989) showed settleAwaitingCIScan’s FetchItemDetails error path (§6.14) routed every error — transient or structural — through the same __awaiting_ci_orphan__ escalation counter. A GraphQL rate-limit exhaustion episode ("API rate limit already exceeded for user ID ...") affects every item on the board simultaneously, not just the one being processed, and resolves itself automatically at the next hourly reset. Three consecutive rate-limited settle passes exhausted MaxRetries and paused an otherwise-healthy issue — final labels included fabrik:cruise, fabrik:paused, stage:Specify:complete … stage:Review:complete — requiring manual fabrik:paused removal to recover. This is distinct from #1270’s original stall (an unevaluatable gate that never surfaced): here the gate was evaluatable, but a global, self-healing API condition was mistaken for a per-item, persistent one.

Fix — classify the error before calling recordAwaitingCIOrphanRetry. isTransientAPIError(err error) bool (engine/item.go, immediately after isTransientError) layers rate-limit/quota-exhaustion detection on top of the existing isTransientError predicate: it returns true for everything isTransientError already recognizes (network-layer errors, unexpected EOF, GitHub 5xx, connection reset, i/o timeout) plus GraphQL/REST rate-limit and secondary-rate-limit/abuse-detection error text, matched case-insensitively via rateLimitErrorPatterns ("api rate limit exceeded", "api rate limit already exceeded", "secondary rate limit", "abuse detection" — specific multi-word phrases, deliberately avoiding a bare "rate limit" or "api rate limit" substring per the precedent in botServiceNoticePatterns, engine/comments.go), plus a direct HTTP-429 status-code check ("github api returned 429") since 429 is used exclusively for rate-limiting by GitHub’s API and would otherwise be missed by a response body that doesn’t happen to match any known phrase. Both narrowings (splitting the bare "api rate limit" phrase, and adding the 429 status check) were added in PR review (pruefer) after the bare phrase was found to also match unrelated prose that merely mentions the concept without reporting an actual exhaustion. Anything not confidently matched falls through to false (structural) — including the plain, unwrapped "node not found" error FetchItemDetails returns for a deleted issue node, which has no underlying error to match against and correctly stays structural.

settleAwaitingCIScan’s FetchItemDetails error branch now checks isTransientAPIError(err) before deciding whether to call recordAwaitingCIOrphanRetry: a transient/global error logs distinctly and continues without calling recordAwaitingCIOrphanRetry at all — no counter touch, however many consecutive polls are affected, so a rate-limit episode of any length can never mass-pause the item. A non-transient error is handled exactly as before (§6.14): logged, recordAwaitingCIOrphanRetry called, escalating to fabrik:paused + comment at MaxRetries. The itemstate.DeepFetchFailed store mutation (bookkeeping only — records LastDeepFetchFailureAt) stays unconditional for both branches, since it has no bearing on the escalation counter.

Escalation now names the failure class. Because a transient/global error never reaches recordAwaitingCIOrphanRetry, escalateAwaitingCIOrphanFailure (§6.14) can only ever fire for the orphan-column case or a genuinely non-transient fetch failure — never a rate-limit episode. Its fetch-failure problem string was reworded to say the failure was non-transient (“rate-limiting and other transient conditions defer automatically and never reach this point”), so an operator reading the pause comment can immediately rule out a rate-limit episode as the cause.

Scope. isTransientAPIError is a sibling predicate, not an extension of isTransientError itself — isTransientError’s four other call sites (removeEditingLabel, dependencies.go, pr.go ×3, prcreate.go) are untouched and cannot be affected by this change. recordSettleRetry/escalateSettle (engine/settle.go, shared by four other settle scans per ADR-060/061/062/1097) are unchanged — this fix acts entirely at the ci_settle.go call site, deciding whether to call recordAwaitingCIOrphanRetry, never changing what it or its downstream helpers do. The orphan-column retry path (§6.14) is unaffected — it doesn’t call FetchItemDetails and still shares the same counter unconditionally, exactly as before this fix.

References: ADR-1270: Awaiting-CI Settle Scan (amended in place for this issue rather than superseded — see its “Why does a FetchItemDetails failure retry-then-escalate through the same path as an orphan column?” subsection), engine/backoff.go (shouldPauseForRESTRateLimit/isRateLimitNearZero, the pre-existing numeric rate-limit-aware poll-interval backoff — a structurally different mechanism from this issue’s per-error-string classification), engine/terminal.go (GraphQL-budget-aware polling pause via RateLimitStats()), issue #1313

6.14.3 Awaiting-CI Cache-Hit Guard Never Fired (issue #1325)

Trigger: #1303’s primary fix (§6.14.1, Fix 1) had never executed a single time in a multi-hour, multi-slice validation run — awk '/refreshing check runs live/' fabrik.log | wc -l returned 0 across the entire test-bed log, despite the cache being demonstrably active. A validation slice on main reproduced the original #1303 symptom exactly: fabrik-test-alpha#4157/PR #4158 reported CI checks pending for 89 minutes while the live PR head had already resolved to a definitive ci-fix-sentinel FAILURE within minutes. Only the CIWaitTimeout backstop (§6.14.1, Fix 2) stopped it — the primary fix contributed nothing, and the backstop’s sensible pause-with-comment behavior masked the wedge from the outside.

Confirmed root cause: copyDeepFieldsFromState never copied LinkedPRHeadSHA. The RefreshCheckRunsLive call site’s guard (engine/ci_settle.go) is item.LinkedPRHeadSHA != "" && item.LinkedPRNumber != 0, evaluated against the item value returned by settleAwaitingCIScan’s FetchItemDetails call. That call goes through boardcache.CacheImpl.FetchItemDetails, which on a cache hit calls copyDeepFieldsFromState to overlay deep fields from the store’s cached ItemState onto the passed *gh.ProjectItem — and, until this fix, that function copied LinkedPRNumber from s.LinkedPR but never assigned dst.LinkedPRHeadSHA, leaving it at its zero value ("") on every hit. On a cache miss, the live GraphQL path (applyLinkedPRs, github/project.go) populates both fields together from headRefOid, so the guard passed — but only in the one case (a fresh live fetch just happened) where RefreshCheckRunsLive’s rescue wasn’t needed. Net effect: the live refresh could only run when the cache was not serving a stale read, making it inert exactly when the #1303 wedge was in play.

Fix — copyDeepFieldsFromState (boardcache/boardcache.go) now also copies dst.LinkedPRHeadSHA = s.LinkedPR.HeadSHA, symmetric with the existing LinkedPRNumber copy from the same s.LinkedPR struct. This benefits every FetchItemDetails caller uniformly rather than adding a second, settleAwaitingCIScan-specific SHA-sourcing mechanism at the call site. ItemState.LinkedPR.HeadSHA is safe to trust on a cache hit: it’s written by three independent, pre-existing freshness mechanisms — every deep fetch (CacheImpl.FetchItemDetails’s own PRHeadSHAUpdated write on a miss), pull_request webhook deltas, and check_run webhook deltas (boardcache/delta.go) — the identical set of writers that already keep every other LinkedPR field (reviews, mergeability, queue state) current. No new, independently-stale cache is introduced. FetchCheckRuns’s general cache-trust contract (§6.14.1, Fix 1’s denylist-only-on-FAILED behavior, ~35 other call sites) is untouched — this fix only supplies the missing input to a guard whose own logic was already correct, and does not change the guard itself, the CIWaitTimeout backstop, or FetchCheckRuns’s scope.

Test-gap note. #1303’s own regression test (TestSettleAwaitingCIScan_StaleCachedPendingCheckRuns_EndToEnd) sets item.LinkedPRHeadSHA directly in its mock fetchItemDetailsFn, so its single FetchItemDetails call is always a cache miss — it never exercised copyDeepFieldsFromState and so never caught this defect. The new regression test (TestSettleAwaitingCIScan_CacheHitPath_RefreshCheckRunsLiveReached) primes the real boardcache.CacheImpl with a genuine prior deep fetch, then calls settleAwaitingCIScan with a fresh/zeroed board item and an unchanged UpdatedAt so FetchItemDetails takes the cache-hit branch — confirmed failing against the pre-fix copyDeepFieldsFromState and passing after.

References: §6.14.1 (issue #1303 — the fix this issue makes reachable), §6.14.2 (issue #1313), issue #1320/#1323 (unrelated e2e scenario defects that had independently prevented TestCIFixReinvokeCycleLimit from reaching cycle 2 and exposing this)

6.14.4 CI-Gate Pause Comment Reuse on Resume (issue #1408)

Trigger: Field evidence (verveguy/liminis-context-graph#342, 2026-08-04) showed an item that once hit the CIWaitTimeout backstop (§6.14.1, Fix 2) could never be resumed by the remedy its own pause comment instructs. The linked PR was CLEAN/APPROVED/MERGEABLE with every check green since 13:25:44Z; the issue sat stranded from 20:00:57Z onward carrying fabrik:awaiting-ci with no stage:Validate:complete and no fabrik:paused — the timeout comment had been posted once, at 13:21:56Z, and a human had since removed fabrik:paused to resume, exactly as instructed.

Confirmed root cause: hasCIGatePauseComment’s unscoped match, combined with a never-reset fabrik:awaiting-ci appliedAt. fabrik:awaiting-ci’s applied-at timestamp is set once, at handleStageComplete time (ADR-1314), and is never reset for the label’s lifetime — neither pauseForCITimeout nor pauseForCIFixCycleLimit remove or reapply fabrik:awaiting-ci itself, only checkCIGate’s classify helpers do, on a confirmed gate-clear or confirmed timeout. So once an item’s fabrik:awaiting-ci ages past CIWaitTimeout once, every subsequent appliedAt-based check still reads it as timed out, regardless of live CI state, until the label is genuinely removed or reapplied. hasCIGatePauseComment (engine/ci.go) was written to identify a single timeout/cycle-limit episode by scanning the issue’s entire comment history for the stable prose fragment — deliberately unscoped by time, to catch the same-poll two-call label-swap race described in §6.14 — but before this fix, pauseForCITimeout/pauseForCIFixCycleLimit treated any match as “nothing more to do” and returned without touching labels or reporting anything. The backstop’s caller (settleAwaitingCIScan) then continued past the handler chain unconditionally, on the assumption that calling pauseForCITimeout always means “this item was just escalated.” A human resuming the item by removing only fabrik:paused (per the pause comment’s own instructions) never touches the comment or fabrik:awaiting-ci, so the very next settle pass re-derived “timed out” from the same stale appliedAt, matched the old comment, and looped: no label churn, no comment, no re-evaluation — forever.

Why the no-op itself was not the bug. Suppressing a duplicate pause comment within the same episode (the two-call race) is correct and remains unchanged (R3). The defect was conflating that suppression with “escalated” at the call site: the backstop’s continue treated both outcomes identically, so a suppressed escalation was never followed up by anything that could re-evaluate the item.

Fix — split by whether a live CI read has already happened this call. The fix has two coordinated pieces; neither alone satisfies both the “green CI advances” and “still-failing CI re-escalates” requirements (R1/R4) — see the two rejected single-piece designs recorded in ADR-1408:

  1. The backstop (engine/ci_settle.go) checks hasCIGatePauseComment itself, before ever calling pauseForCITimeout. When no comment exists yet (a genuinely fresh timeout), it pauses and continues exactly as before. When a comment already exists for this episode, it does not call pauseForCITimeout and does not continue — it logs that the item is deferred to the live-data-informed handler chain and falls through to RefreshCheckRunsLive and catchUpPhase1Handlers, unchanged otherwise. This is what lets a resumed item with green CI clear the gate cleanly (checkCIGate’s PRMergeReady/PRMergeTerminal/PRMergeNoPR cases short-circuit to addCompleteLabelAndRemoveCI before any timeout check runs) with zero pause-label churn.
  2. pauseForCITimeout/pauseForCIFixCycleLimit (engine/ci.go) stop no-opping when hasCIGatePauseComment matches. Every caller reaching this branch has, by construction, already done a live CI read this poll (either the backstop’s own fresh-timeout branch — irrelevant, since that branch never has a match — or handleMergeAndCIGates, reached only after checkCIGate/classifyCIFrom* re-derives ciTimedOut/ciFailure from the same stale appliedAt for a still-blocked item). So it is now safe to reapply fabrik:paused + fabrik:awaiting-input (idempotent GitHub label writes, via a small reapplyCIGatePauseLabels helper) without posting a new comment — the existing episode’s comment is reused. Both functions now return escalated bool: true for a fresh post, false for a reused/relabeled-only pass (R2).

handleMergeAndCIGates (engine/catch_up_handlers.go) needed no changes — its existing unconditional calls into the two pause functions already do the right thing once those functions stopped no-opping, since every path into them already ran a live settle read this poll.

Why reuse the comment rather than repost (R4). hasCIGatePauseComment’s unscoped, full-history match is exactly what makes comment reuse possible without a new time-scoping mechanism (a per-poll dedup set does not survive the two-call race test’s two separate top-level settleAwaitingCIScan calls; comment-recency matching would require realistic CreatedAt stamping in test mocks that don’t currently provide it). The fix repurposes the existing check rather than adding one, and hasCIGatePauseComment’s matching logic itself is unchanged (R6 below covers only its doc comment and the two call sites’ log wording).

R4 — a still-blocked resumed item is genuinely re-escalatable, not silently unpausable. Because the reused-comment path still reapplies fabrik:paused + fabrik:awaiting-input every time it’s reached, a resumed item whose CI is genuinely still stalled (classifyCIFromCheckRuns’s own independent liveness check re-derives timedOut for a still-pending, no-progress case) is paused again on the very next settle pass, not left stranded a second time. Updated by ADR-1410: this description originally also covered “still failing” — at the time this fix landed, classifyCIFromCheckRuns re-derived timedOut identically for a confirmed failure past the deadline as for a still-pending one (the R3 defect ADR-1410 later fixed). Since ADR-1410, a confirmed failure instead re-derives ciFailure=true unconditionally and dispatches CI-fix reinvocation rather than re-pausing — see the table row below and TestSettleAwaitingCIScan_ResumedAfterTimeout_StillFailing_DispatchesCIFix (engine/ci_settle_test.go), which replaced this section’s original still-failing pause-based test for the identical fixture.

R5 — pauseForCIFixCycleLimit gets the identical fix, and the strand is reachable independent of the CIWaitTimeout backstop. pauseForCIFixCycleLimit has two call sites — handleMergeAndCIGates’s cycle-limit callback (engine/catch_up_handlers.go) and checkAutoMergeConvergence’s queue-repo ejection-recovery path on PRMergeBlocked (engine/merge_gate.go) — both always reached after a live CI classification this poll (ciFailure and settle.Status == PRMergeBlocked respectively; autoMergeConvergence and mergeAndCIGates are mutually-exclusive handlers in catchUpPhase1Handlers, so they can’t double-fire for the same item), so neither needs caller-side gating — only the same reapply-without-repost behavior internally. The strand is independently reachable via resume: clearFailedStage() only resets CIFixCycles when stage:<X>:failed is present or snap.PausedByEngine(stageName) is true, and a CI-fix-cycle-limit pause sets neither (pauseIssue never applies itemstate.EnginePaused) — so CIFixCycles survives a resume unreset, and a human resuming an item paused at the cycle limit before CIWaitTimeout has separately elapsed hits this exact path, entirely independent of the backstop.

R6 — corrected wording. hasCIGatePauseComment’s doc comment and both pause functions’ log lines previously described the match as “already posted this poll” — inaccurate, since the match is unscoped by time and identifies an entire episode, not a single poll. Both now describe the unscoped, full-episode match and the reapply-without-repost behavior it enables.

R7 — no behavior change for an item that has never posted a CI-gate pause comment. hasCIGatePauseComment returns false for such an item exactly as before; the backstop’s fresh-timeout branch (pause and continue) and both pause functions’ fresh-post branch (post via pauseIssue, return true) are byte-identical to pre-#1408 behavior.

Correction (issue #1937): the match is anchored to the pause-comment shape. The shared primitive behind hasCIGatePauseComment and the review/rebase/enqueue wrappers is hasPauseComment (engine/mutate.go). It still scans the whole comment history, so it still identifies an episode. But a comment now counts only if it has the engine’s own pause shape: a 🏭 **Fabrik — …** header line, a blank line, then the fragment at the start of the body text. Every pause message is built that way.

The previous unanchored substring match also hit a fragment merely quoted elsewhere. In the live case (release gate, TestCIFixReinvokeCycleLimit), a Research output recounted an earlier run’s “The stage Validate has been re-invoked to fix CI failures 2 time(s)…”. When the real limit was hit, pauseForCIFixCycleLimit took the reapply branch and paused with labels only: no explanatory comment was ever posted. A human quoting a pause message had the same effect.

State transitions (new rows; existing §6.14 rows unaffected):

Before Trigger After Labels Added Labels Removed
wait_for_ci stage, awaiting CI, existing pause comment, no fabrik:paused (resumed) Settle pass: live CI green stage:X:complete present; landing decision reached via runCatchUpPhase2 stage:X:complete fabrik:awaiting-ci
wait_for_ci stage, awaiting CI, existing pause comment, no fabrik:paused (resumed) Settle pass: live CI still genuinely stalled (no progress past the liveness dwell, or the R3/mergeable-state-blocked case) Paused (re-escalated; no new comment) fabrik:paused, fabrik:awaiting-input —
wait_for_ci stage, awaiting CI, existing pause comment (from the old, pre-ADR-1410 timeout-labeled-as-failure defect), no fabrik:paused (resumed) Settle pass: live CI confirmed still failing Not re-paused — ciFailure=true dispatches CI-fix reinvocation instead (ADR-1410, R3) fabrik:awaiting-ci (idempotent; already present) —
wait_for_ci stage, awaiting CI, existing cycle-limit comment, no fabrik:paused (resumed), CIFixCycles at MaxCiFixCycles Settle pass: live CI still failing Paused (re-escalated via pauseForCIFixCycleLimit’s reuse path; no new comment) fabrik:paused, fabrik:awaiting-input —

References: §6.14 (Fix 2, corrected above), §6.14.1 (issue #1303, the backstop this fix modifies), ADR-1270: Awaiting-CI Settle Scan (the two-call label-swap race hasCIGatePauseComment was originally written for — its regression test, TestSettleAwaitingCIScan_RaceWithMainLoop_CycleLimitPause_NoDuplicateComment, is unaffected by this fix), ADR-1387: Closed Items Are Never Dispatched (structural precedent for a “zero remaining owners” defect), ADR-1408: CI-Gate Pause Comment Reuse on Resume, issue #1408

6.14.5 CI Gate: Liveness, Not Elapsed Time (issue #1410)

Trigger: CIWaitTimeout was a single wall clock applied uniformly to four structurally different CI states — check runs pending (alive and reporting), check runs failed (a verdict, not a wait), R3’s never-checked case, and a blocking mergeable_state with no check runs (both genuine liveness problems — no event is coming without a human). Only the last two are legitimately clock-worthy. Field evidence (verveguy/liminis-context-graph#342, 2026-08-04) showed the first misapplication in practice: fabrik:awaiting-ci anchored at 12:51:45Z, a rebase reinvocation consuming 15m46s of the 30-minute budget before pushing a fresh head at 13:07:31Z, an 18m04s CI suite starting at 13:07:40Z with only 14m05s left — Fabrik paused at 13:21:57Z, CI finished green at 13:25:44Z on a CLEAN/APPROVED/MERGEABLE PR. The failure row was its own defect: classifyCIFromCheckRuns’s CIWaitTimeout guard fired ahead of pending/failed classification, so a genuine failure occurring after the deadline was reported as a timeout — bypassing MaxCiFixCycles and the CI-fix reinvocation path, and posting a message claiming Fabrik “timed out waiting for checks to pass” when checks had in fact reported a verdict.

Fix — split the single clock into a verdict/liveness pair. See ADR-1410 for the full design and rationale; summary of the as-built mechanism:

  1. Verdicts never wait (R3). classifyCIFromCheckRuns classifies via gh.ClassifyCheckRuns first; on CheckRunsFailed it returns ciFailure=true unconditionally, no clock. classifyCIFromRequiredContexts drops its elapsed-time guard entirely and always returns ciFailure=true on RequiredContextsFailed. Both route to the existing MaxCiFixCycles path exactly as any fresh failure always has — see §6.4 rules 13/17/19 and §6.5.2, both unaffected by this change.
  2. The pending branch gets a liveness dwell, not an elapsed one. A new ciProgressStalledSince() helper (engine/ci.go) reads LinkedPRState.LastCIProgressAt from the itemstate store; classifyCIFromCheckRuns escalates ((false, false, true)) only once that timestamp is both non-zero and CIWaitTimeout or more in the past. LastCIProgressAt is stamped in internal/itemstate/store.go: applyCheckRunCompleted, whenever upsertCheckRunByID reports actual content change (a new check-run ID, or an existing one’s Status/Conclusion transitioning — not on an identical duplicate observation); and PRHeadSHAUpdated’s existing SHA-change branch and pending-buffer drain (a fresh push resets CI entirely, which is progress in its own right — this is what removes #342’s dependence on when the wait started). All three check-run producers (RefreshCheckRunsLive, FetchCheckRuns’s cache-miss path, check_run webhook deltas) already funnel through applyCheckRunCompleted, so one edit point covers all of them.
  3. classifyCIFromMergeableState is unchanged. Its two cases (R3 never-checked, mergeable-state-blocked-with-no-checks) have no check-run signal to observe progress on at all, so they keep the original labelAppliedAt-anchored elapsed dwell — the one classifier for which a plain clock remains the right instrument.
  4. A new, separate CIBackstopTimeout (240-minute default) bounds settleAwaitingCIScan’s unconditional backstop (R5) — see §6.14.1 Fix 2, now repointed at this setting instead of CIWaitTimeout. It stays elapsed-time-based (FetchLabelAppliedAt, unchanged mechanism) since its purpose — bounding per-poll cost regardless of CI duration — is an elapsed-time question, deliberately decoupled from the liveness dwell’s own justification.
  5. Merge-train’s blocking polls (pollForMergeable, pollTrainCI, engine/merge_train.go) keep an elapsed bound, repointed at CIBackstopTimeout (R6). They are synchronous, single-goroutine blocking loops, not re-entrant poll-driven state — adopting liveness semantics would hold a worker goroutine open for a suite’s full duration. They already degrade gracefully on timeout (retry next merge-train cycle, no pause) rather than reproducing #342’s destructive pause; repointing at the much larger backstop removes the real remaining cost (a wasted trial-branch rebuild roughly every 30 minutes for a healthy-but-slow suite under the old, now-repurposed CIWaitTimeout). 5a. Suite-only holds are bounded the same way (#1822). A hold caused by an outstanding check suite (§6.4 Suite-aware completeness) is PRMergeUnsettled, claimed by checkMergeabilityGate before checkCIGate, so it is bounded by CIBackstopTimeout, not the liveness dwell (no check run is pending, so LastCIProgressAt never advances for it). pauseForCITimeout names the outstanding suite and app. The train polls’ suite holds are likewise bounded by their existing CIBackstopTimeout deadline.
  6. pauseForMergeGroupStall/checkAutoMergeConvergence (§5.5 ②′, engine/merge_gate.go, ADR-058 D5) needed no code change — already liveness-shaped (fires only when merge-group CI has never reported), already anchored on labelAppliedAt rather than the CI-await window’s own clock. It correctly keeps reading the repurposed CIWaitTimeout directly.
  7. No new pause function or comment text. The liveness-stall escalation reuses pauseForCITimeout() verbatim (§6.14.4) — every caller of that function now represents a genuine stall, so its existing “timed out waiting for checks to pass” message stays truthful, and #1408’s episode-matching recoverability composes for free.

Architectural discovery — the pending-branch dwell is unreachable via the real async pipeline today. checkMergeabilityGate (§6.6, engine/merge_gate.go) unconditionally claims any PRMergeUnsettled classification, and settlePRMergeState (§6.4 rule 19) always returns PRMergeUnsettled for CheckRunsPending — there is no other path to that state. Since handleMergeAndCIGates returns immediately when the merge gate claims the item (§6.2 Handler 4), checkCIGate — and therefore classifyCIFromCheckRuns’s pending branch, item 2 above — is never reached for a merely-pending CI state via settleAwaitingCIScan/the catch-up loop. This was equally true of the old CIWaitTimeout guard before this issue (the pending-branch clock was already dead code from the pipeline’s perspective, reachable only via a direct checkCIGate call, as the TestCheckCIGate_Pending_* unit tests in engine/ci_test.go do). settleAwaitingCIScan’s backstop (§6.14.1 Fix 2) — not classifyCIFromCheckRuns’s inner guard — is and was the mechanism that actually escalated #342 in the field; TestSettleAwaitingCIScan_342Repro_SlowButHealthyCI_DoesNotPause (engine/ci_settle_test.go) reproduces the incident end-to-end through the backstop and confirms this, including a non-vacuous sub-test showing a 30-minute-equivalent CIBackstopTimeout does fire at #342’s own 34-elapsed-minute timeline. Item 2’s liveness dwell remains correct behavior for a direct checkCIGate caller and is defensive against a future change to the merge-gate claim priority, but item 4’s backstop resize is what actually resolves #342 for real traffic.

Consequence for §6.14.4 (R4). A resumed item (#1408) whose original pause was a genuine liveness stall, and which is still stalled on resume with no confirmed verdict, is not automatically re-escalated by the async gate for the same reason: the handler chain cannot re-derive “still stalled” for a pending-CI item the way it can for a confirmed-failing one. It falls back to being silently re-blocked by the merge gate, re-evaluated every poll, until CI produces a verdict or the backstop (independently anchored on labelAppliedAt, unaffected by the resume) fires again. This is a pre-existing property of the claim-priority architecture, not a regression introduced by this issue — the old elapsed-time guard had the identical unreachability — and closing it is out of this issue’s scope (Non-goals: no change to the conjunctive CI/review gate’s semantics, #895).

R8 — configuration compatibility. FABRIK_CI_WAIT_TIMEOUT/--ci-wait-timeout keep their name, flag, and 30-minute default (TestCiWaitTimeout_DefaultUnchanged pins the value; TestCheckCIGate_Failed_NeverTimesOut_RegardlessOfElapsedTime and its required-context sibling pin the behavior change). Only the meaning changes — cap on total CI wait → liveness-stall dwell — stated in this ADR, in docs/USER_GUIDE.md, and via a one-time, unconditional startup log line (logCIWaitTimeoutSemantics, engine/poll.go) so no operator discovers the change silently.

References: ADR-1410: CI Gate — Liveness, Not Elapsed Time, §6.14 (Fix 2 / backstop, repointed), §6.14.1 (issue #1303, the backstop’s origin), §6.14.4 (issue #1408, the recoverability model this reuses), §6.6.2 (merge-group stall precedent, ADR-058 D5), verveguy/liminis-context-graph#342 / PR #344 (field evidence)

6.14.6 CIBackstopTimeout Backstop: Anchor and Guards (issue #2059)

Trigger: On #2052 / PR #2053 (2026-10-05/06) a daemon clean stop paused an item mid-Validate while it carried fabrik:awaiting-ci. Eleven hours later a human’s steering comment resumed it (fabrik:paused + fabrik:awaiting-input removed). Three seconds later settleAwaitingCIScan’s backstop logged “exceeded CIBackstopTimeout … escalating” and re-paused it, in the same poll in which the resume had dispatched a comment worker; the worker then fixed the outstanding items under a pause that would not lift on its own. The #1408 “existing pause comment → defer” rule did not apply: the earlier pause was a clean-stop pause, so no CI-timeout pause comment existed for the episode.

Two defects. (1) The clock measured from fabrik:awaiting-ci’s applied time, which pausing and resuming never reset (ADR-1314), so it kept running through paused time and daemon downtime: an item paused overnight was already past the timeout on its first poll after the resume. (2) The scan had no in-flight check, so it could escalate an item whose worker had just been dispatched.

As-built. The backstop block (engine/ci_settle.go) now evaluates, cheapest first, and only escalates when every guard passes. The timeout value, the §6.14.4 deferral, the escalation message and the bound on per-poll cost are unchanged.

  1. Label anchor (unchanged). time.Since(appliedAt) < ciBackstopTimeout() → nothing to do. An unreadable or zero appliedAt is also a no-op, as before.
  2. Pause comment present (unchanged, §6.14.4). Defers to the live handler chain.
  3. R3 — worker in flight. If the store holds a worker for the item (snap.Worker() != nil — set synchronously at dispatch, so it covers a worker started this poll by an earlier catch-up phase as well as one from a previous poll), the backstop is skipped for this poll and logged with the worker’s stage. Only the backstop is deferred; the handler chain still runs. The next poll re-evaluates, so the bound holds once the worker exits.
  4. R2 — first evaluation since daemon start. The first time the scan evaluates an item in this process, the backstop does not escalate: the live-data chain (the live-confirmed check-run read, then checkCIGate et al.) runs first, and the item is marked evaluated (Engine.ciBackstopLiveEvaluated) only once the chain has actually run — the early continues (orphan column, deep-fetch failure, label changed after the fetch, escalation) do not count. If a gate silently claims the item on that pass, the result is “skip once, then enforce”. The marker is in memory by design: a restart clears it, so each awaiting-ci item gets one live evaluation per daemon start, and a genuinely stuck item is escalated one poll later than before.
  5. R1 — the clock does not count paused time. The effective anchor is max(appliedAt, latest fabrik:paused removal). The removal time comes from FetchLabelRemovedAt (github/labels.go: the newest unlabeled event for the label in GitHub’s issue event log, same paging and restMaxPages bound as FetchLabelAppliedAt), read live and never from the record-on-write label cache, so it survives restarts and sees removals made in the UI (the reasoning of resumeAuthorised, ADR-1813). A failed read or an absent event falls back to appliedAt and still escalates — the opposite fail direction to resumeAuthorised; the backstop must never fail open into “never escalate”, or its per-poll cost bound is lost. A resume still inside the timeout skips the backstop (logged).
  6. Resume-read memo. A recently resumed item would otherwise cost one events page-through per poll for up to the timeout. Engine.ciBackstopResumeSeen memoises the newest removal time read; a memoised resume still inside the timeout skips without a read. The memo is skip-only: it can only be older than the truth, so it can only produce a skip, and an escalation always follows a live read.

Sim fidelity. simgh.LabelEvent carries At (the injected clock at the mutation) and Sim.FetchLabelRemovedAt answers from that log (tests/sim/simgh/FIDELITY.md, “Label events”). tests/sim/ci_backstop_resume_test.go replays the #2052 sequence: awaiting-ci → clean-stop pause → restart → resume → comment worker → no re-pause → gate clears on green.

Sibling timeouts (#2064, ADR 2059). The other timeouts anchored on never-reset label timestamps use the same effectiveAnchor helper (engine/resume_anchor.go): the CIWaitTimeout classifiers classifyCIFromMergeableState (R3 never-checked and mergeable-state-blocked dwells, engine/ci.go, anchored on fabrik:awaiting-ci), the merge-queue stall dwell and ConvergenceBudget (engine/merge_gate.go, anchored on fabrik:auto-merge-enabled). Each measures from the later of its label anchor and the latest fabrik:paused removal, so paused time never counts. Each first tests the cheap label anchor and calls the helper (one FetchLabelRemovedAt read, memoised in the shared resume memo, tested against the caller’s own timeout) only once already over its timeout; an unreadable removal time falls back to the label anchor and still escalates. Operators, comparison operators (>=; > for ConvergenceBudget), disable-on-zero guards, timeout values and escalation messages are unchanged, and pauseForConvergenceFailed reports the elapsed time from the effective anchor. The classifyCIFromCheckRuns liveness stall uses in-memory LastCIProgressAt and is unaffected.

References: §6.14.1 (the backstop), §6.14.4 (the §6.14.4 deferral this keeps), §6.14.5 (CIBackstopTimeout), ADR-2059, ADR-1813, ADR-1314, #2052.

6.15 Closed-Item Validate Terminal Advance Settle Scan (ADR-1387)

Trigger: Field evidence (handarbeit/fabrik-test-alpha#4246) showed an issue closed out-of-band while parked at Validate — wait_for_ci: true, so handleStageComplete withholds stage:Validate:complete and applies fabrik:awaiting-ci instead (the conjunctive gate, §6.5) — was re-dispatched as a full Claude stage invocation every poll cycle, 87 times over ~14 hours. The loop composed from four individually-reasonable mechanisms: (1) the conjunctive gate defers stage:Validate:complete and applies fabrik:awaiting-ci as the interim suppressor; (2) cleanupClosedIssueTransientLabels (see R6 below) swept fabrik:awaiting-ci off the closed issue every poll, as ordinary transient-label hygiene; (3) the item was then left with neither the completion label (deferred) nor the suppressing label (swept) — still sitting at Validate; (4) the pre-ADR-1387 closed-issue admission guard (§2.8) admitted it anyway via a !stageIsGateChecked(stage) disjunct, so it was dispatched — returning to step 1. The loop only terminated by chance, when the model happened to emit FABRIK_NO_WORK_NEEDED instead of FABRIK_STAGE_COMPLETE.

Root cause: the settle-owner’s only feed was dispatch-admission-gated. Before ADR-1387, runValidatePRTerminalAdvance (§2.10, ADR-056 D2) — the settle-owner responsible for healing exactly this class of closed, paused, or awaiting-review Validate-stage item (the #874 class, commit 7311a14e) — was fed exclusively from deepFetchCandidates, which is itself filtered by itemMayNeedWork. The only way to make a closed item observable to the settle-owner was therefore to admit it to dispatch — conflating “let the settle-owner see this item” with “let this item receive a real Claude invocation.” 7311a14e widened admission to reach the settle-owner and, in doing so, also widened dispatch eligibility, since both consumers shared one gate.

Same failure class as §6.9/§6.10/§6.13/§6.14: a durable-marker/stranding class recovered by the shared, admission-gated catch-up path instead of a dedicated board.Items-sourced settle scan. settleClosedValidateAdvance gives the settle-owner a feed that is entirely independent of dispatch admission, mirroring settleAwaitingCIScan’s (§6.14, ADR-1270) “admission independence is the entire fix” shape — the same idiom applied to a different stranding class.

Code path: settleClosedValidateAdvance (engine/pr_terminal_advance.go; called from poll() in poll.go, immediately after runValidatePRTerminalAdvance).

Shared logic, split ownership. Both runValidatePRTerminalAdvance (§2.10, open items) and settleClosedValidateAdvance (closed items) delegate to the same extracted per-item function, advanceValidateTerminalItem(board, item, advancedItems) — the exact pre-ADR-1387 logic of runValidatePRTerminalAdvance, unchanged. This keeps ADR-057’s “single authoritative owner” contract intact in spirit: one union of healing logic, split only by feed (deepFetchCandidates vs. board.Items) and therefore by cost, not two independently-reasoning implementations that could drift apart. runValidatePRTerminalAdvance’s loop skips any item.IsClosed item (a redundant-but-explicit ownership boundary — admission no longer lets a closed item reach deepFetchCandidates at Validate in the first place, so this is unit-testable clarity, not load-bearing correctness) and settleClosedValidateAdvance’s loop skips any open item, so the two functions partition board.Items/deepFetchCandidates disjointly and can never both process the same item in the same poll.

No double-advance, by construction (R5). advancedItems[iKey] is checked before the live FetchLinkedPR call inside the shared per-item logic, and poll() is single-threaded/sequential, so even without the IsClosed partition above, whichever of the two callers ran second would find the item already marked and no-op. settleClosedItemsToDone (§6.11, ADR-064) excludes stage.Name == "Validate" — it defers closed Validate items to whichever function owns them, which only needed to actually be reachable independent of admission, which is exactly what this scan supplies. (An initial implementation of this fix left settleClosedItemsToDone’s exclusion unchanged at the broader stageIsGateChecked(stage), which turned out to need its own follow-up correction — see the note below.)

Cost: O(closed items currently sitting at Validate) — bounded by WIP at one late-pipeline stage, not board size, and structurally identical in kind to settleClosedItemsToDone’s and settleAwaitingCIScan’s existing unconditional per-poll board.Items scans. Calls only the lightweight FetchLinkedPR per candidate (never the expensive FetchItemDetails GraphQL deep-fetch), and only for items already known-closed from the shallow board snapshot — materially cheaper than what deepFetchCandidates was originally introduced to bound. No rate-limit concern identified.

R6 — cleanupClosedIssueTransientLabels no longer races the settle-owner. Before this fix, the per-poll transient-label sweep (engine/poll.go, #617) stripped fabrik:awaiting-ci (and fabrik:awaiting-review, fabrik:rebase-needed) off every closed issue unconditionally — including one sitting at Validate with its completion still deferred, which is exactly step 2 of the loop described above. The sweep now resolves each closed item’s stage and, when it is specifically stage.Name == "Validate" — not stageIsGateChecked(stage) generally — skips exactly fabrik:awaiting-ci, fabrik:awaiting-review, and fabrik:rebase-needed — the three labels the settle-owner clears itself, atomically, as part of a real merge/pause transition — while every other transient label (including fabrik:auto-merge-enabled, which per attemptMergeOnValidate’s design always coexists with stage:Validate:complete and so was never part of the stranding mechanism) continues to sweep unconditionally. Every closed item at any other column, gate-checked or not (e.g. a Review stage independently configured with wait_for_reviews: true, as in the shipped default stages/examples/review.yaml), is entirely unaffected — the sweep runs exactly as before there. This scoping is deliberate, not an oversight: the settle-owner pair only ever processes stage.Name == "Validate" (matching runValidatePRTerminalAdvance’s own pre-existing, unchanged hardcoding — see the settleClosedItemsToDone exclusion-narrowing note below); excluding a label at any other gate-checked stage from the sweep would strand it with no owner left to ever clear it. A first implementation of this fix used stageIsGateChecked(stage) here and was caught and corrected during Implement precisely because it would have created that stranding for a closed, incomplete Review-stage item.

fabrik:auto-merge-enabled-without-stage:Validate:complete is healed inline, not left to a delayed self-heal (caught in review on PR #1388, with the initial fix’s rationale itself corrected on a follow-up review pass). advanceValidateTerminalItem defers entirely to checkAutoMergeConvergence (Phase 1) whenever fabrik:auto-merge-enabled is present, on the assumption that attemptMergeOnValidate never applies the label before the gate has already cleared — true for both callers today. An initial implementation deferred unconditionally on the label alone and, when that assumption was violated (a labeling race, or a future caller of the label), only logged a warning before returning — claiming this “may permanently strand the item,” the same framing correctly used for the Review-stage strand documented above. That framing was itself wrong: fabrik:auto-merge-enabled is not among the labels cleanupClosedIssueTransientLabels withholds from its closed-item sweep (gateSettleOwnedTransientLabels, R6 above), so for a closed item the same poll’s later sweep step removes the label unconditionally, and the next poll’s call into this function falls through normally and heals it — a one-poll-cycle delay, not a permanent strand. (An open item has no such fallback, since the sweep only runs on closed issues — but this state is believed rare for either.) The fix itself is unaffected by that correction: it narrows the defer-to-Phase-1 skip to the invariant-respecting case only (fabrik:auto-merge-enabled and stage:Validate:complete both present) and, when the label is present without the completion label, logs the same warning but falls through into the ordinary merge-vs-pause flow below instead of returning — healing the item inline via the identical logic used for every other Validate terminal advance, rather than depending on sweep ordering and a wasted poll cycle (and closing the gap entirely for the open-item case, which has no sweep-based fallback at all).

advanceValidateTerminalItem intentionally hardcodes stage.Name == "Validate" rather than stageIsGateChecked(stage) — Fabrik’s own PR merge action (attemptMergeOnValidate) only ever runs as part of Validate completing, so the PR-fetch-and-decide logic (merged → fill labels/advance; closed-unmerged → pause) this function implements has no equivalent nuance to add for any other stage. This is unaffected by, and distinct from, settleClosedItemsToDone’s own exclusion below.

settleClosedItemsToDone (§6.11) excludes stage.Name == "Validate" specifically, not stageIsGateChecked(stage) generally (caught in review on PR #1388, after an initial implementation shipped the broader exclusion as a documented “known limitation”). Both shipped default stage templates, stages/examples/review.yaml and stages/examples/validate.yaml, set wait_for_reviews: true — so a deployment using the defaults genuinely has a second gate-checked stage (Review), not a hypothetical one. Under the broader stageIsGateChecked exclusion, a closed item stranded at Review with no stage:Review:complete was never dispatched (R1 held) but also never healed by any settle-owner — settleClosedItemsToDone skipped it (gate-checked), and the settle-owner pair skipped it too (not Validate). Zero remaining owners, a permanent silent strand — worse than the pre-ADR-1387 bug it replaced, which was at least loud and moving. Closing an abandoned or superseded issue mid-pipeline is ordinary operator behavior, not a rare edge case — it is exactly what produced the field evidence this section is built on. The fix needs no new reasoning about PR state: advanceClosedItemToDone (what this scan calls) never inspects the linked PR at all, it only moves the board column — exactly what it already does, unconditionally, for a closed item at Specify/Plan/Implement. Narrowing the exclusion to stage.Name == "Validate" simply stops incorrectly withholding that existing, already-correct behavior from Review. The per-poll “no settle-owner” warning this once required has been removed along with the condition that produced it.

Unresolved-PR polling cost, pre-existing and unchanged by this fix. For a closed item at Validate whose linked PR is neither merged nor closed (a human closed the issue without touching the PR — an unusual, out-of-band action, the same trigger class as the loop this fix targets), advanceValidateTerminalItem returns immediately with no state change, and settleClosedValidateAdvance re-evaluates that item every poll — one FetchLinkedPR call, indefinitely, until the PR resolves — with no timeout/escalation comparable to settleAwaitingCIScan’s CIWaitTimeout backstop (§6.14, ADR-1270). This exact no-escalation behavior already existed in the pre-ADR-1387 runValidatePRTerminalAdvance (verified against the pre-fix code); ADR-1387 does not introduce it, and only lowers its cost — pre-ADR-1387 the same item was also being fully re-dispatched to Claude every poll (the bug this fix targets), so the post-fix steady-state cost (one lightweight, read-only API call per poll) is strictly cheaper, not a new failure mode. A stuck-PR escalation path is unrelated to this issue’s R1–R7 and left out of scope.

State transitions:

Before Trigger After Labels Added Labels Removed
Validate, closed, no stage:Validate:complete, any gate label or none PR confirmed merged Next stage (Done — stages.NextStage skips Holding stages like Queued per issue #1072, §1.3), closed stage:<gate-checked>:complete for each missing gate-checked stage up to and including Validate fabrik:awaiting-ci, fabrik:awaiting-review, fabrik:rebase-needed, fabrik:paused, fabrik:awaiting-input (whichever present)
Validate, closed, PR closed without merging, not already paused Settle pass observes closed-unmerged PR Unchanged column fabrik:paused, fabrik:awaiting-input fabrik:awaiting-ci, fabrik:awaiting-review, fabrik:rebase-needed (whichever present — pauseForPRClosedNotMerged clears all three, not just fabrik:awaiting-ci; this is the only remaining chance to clear them once paused, since the sweep withholds them at Validate)
Validate, closed, PR closed without merging, already paused Settle pass evaluated No change (avoids duplicate comment) — —
Validate, closed, fabrik:auto-merge-enabled present with stage:Validate:complete Settle pass evaluated No change (owned exclusively by checkAutoMergeConvergence, Phase 1) — —
Validate, closed, fabrik:auto-merge-enabled present without stage:Validate:complete (invariant violation) PR confirmed merged or closed-unmerged Healed via the normal merge-vs-pause flow above (same rows as the no-auto-merge-enabled cases) — —
Validate, closed, already in advancedItems this poll Settle pass evaluated No change (dedup, R5) — —
Validate, open (any state) Settle pass evaluated No change (owned exclusively by runValidatePRTerminalAdvance) — —

R1 follow-up: comment-driven dispatch on a closed, stage-complete item (caught by Pruefer review, ADR-1387). The closed-issue admission guard’s retained exception — a closed item carrying stage:<X>:complete stays admitted so the catch-up loop can still act on it — interacted with a separate, older fast path: itemNeedsWork’s “new comments are always worth processing (even on completed stages)” check, and its mirror in processItem, had no item.IsClosed guard. A closed, stage-complete item that received a fresh comment was therefore still routed to processComments, a real Claude invocation — a narrower, pre-existing instance of the same class of bug this section fixes, reachable independently of the CI-gate/fabrik:awaiting-ci mechanism and predating commit 7311a14e. Both fast paths now skip when item.IsClosed: in itemNeedsWork, simply not short-circuiting is sufficient, since the “already completed this stage” check immediately below it already rejects a closed+stage:complete item once the fast path stops pre-empting it; in processItem the added guard is redundant-but-explicit, since itemNeedsWork’s gate already prevents processItem from being invoked at all for this case. A human comment on a closed, completed issue is no longer actionable by Fabrik — consistent with “a closed issue has no computable work left” applying to comment processing exactly as it applies to stage re-invocation.

References: ADR-056: Consolidate Convergence Gate Recovery (D2, the settle-owner’s original single-authoritative-owner contract), ADR-057: Single-Owner Validate PR Terminal Advance, ADR-064: Closed-Item-At-Any-Stage Advance To Done (§6.11, the sibling scan this one defers to/from), ADR-1270: Awaiting-CI Settle Scan (§6.14, the direct template for “board-sourced, admission-independent settle scan for a specific stranding class”), ADR-1387: Closed Items Are Never Dispatched, commit 7311a14e (the regression point whose intent — #874-class healing — this fix preserves, only changing its mechanism), issue #617 (the transient-label sweep whose interaction with the conjunctive gate is closed by R6 above)

6.16 Queued Review-Finding Ejection (Settle Scan)

Channel events (#1968): ejectMember and ejectQueuedMemberForComments emit merge-train-ejected; see §7.14.

Trigger: Issue #1208. While a merge-train member sits in the Queued holding stage awaiting a batch, Fabrik processes no PR feedback at all — the HoldingStage exclusion shared by itemMayNeedWork (§6.14’s admission-independence framing applies equally here: deepFetchCandidates never even deep-fetches a Queued item, so reviewThreads/LinkedPRReviewThreadComments are never populated) and the main catch-up loop’s own per-item admission gate blacks out review-reinvoke and comment processing for as long as the item remains Queued. This window is not brief: batch formation, trial-branch assembly, CI, and possibly inline conflict resolution and bisection can span tens of minutes, and a live merge-train worker can block synchronously inside pollTrainCI for up to CIWaitTimeout (default 30 min) — precisely the window in which a self-submitting review bot (Pruefer) is most likely to post a fresh finding, since it re-reviews on every new head SHA and train activity produces new SHAs. #1207 (§ADR-1207, the sibling Validate-side race) narrows this issue’s practical exposure to the mid-flight case — a finding that lands after the member is already in Queued — since a member with unresolved threads at the moment Validate would hand off to Queued is now guarded there, but the code path this section fixes has no such guard of its own.

Same failure class as §6.9/§6.10/§6.13/§6.14/§6.15: a durable condition invisible to the shared, admission-gated catch-up path, recovered by a dedicated board.Items-sourced settle scan. settleQueuedReviewFindings is the sixth instance of the ADR-1270 pattern. Unlike those five siblings, this scan does not merely make an already-evaluable gate reachable — it introduces a new ejection trigger on top of the pre-existing ejectMember mechanism (ADR-059), because a Queued member cannot address a review finding in place: pushing a fix would change the member’s head SHA mid-batch, invalidating a trial branch the train may have already assembled, CI’d, or bisected.

Code path: settleQueuedReviewFindings (engine/queued_review_settle.go; called from poll() in poll.go, immediately after handleMergeTrainBatch, gated on merge_train: on). Review bodies (#1953 R2): detection (queuedReviewFindings) is current-head thread comments plus unaddressed review bodies (a body-only COMMENTED review, the #616 shape), routed through the same direct/pending eject. A body-read error skips bodies for that poll only. Arrival before admission needs no admission-time gate (admitTrainMembers stays fail-open): a member flagged here is ejected directly when no worker holds it, or consumed by the worker at its first applyPendingReviewEjects checkpoint ahead of any trial or landing.

Catch-up recognition (#2044, §6.30). queuedReviewFindings passes the thread comments and reviews through dropCatchUpFeedback before building findings, so a bot review of a pure merge-train catch-up head neither flags a pending eject nor ejects the member directly — the same predicate unprocessedFeedback applies at the landing gate, so the two cannot disagree. A review of a conflict-edited catch-up is actionable as usual.

Interaction with the own-PR CI admission gate (§6.25, #1821). A pending eject signal for a member that the fresh-batch admission gate then defers for confirmed-red own-PR CI is consumed (takePendingReviewEject) and logged by the deferral, so the member is never double-routed or double-commented and the stale signal cannot later fire on a re-queued member. If the deferral’s reroute fails, the signal is left intact.

Detection. The scan groups the current Queued snapshot by owner/repo via groupQueuedByRepoAndBase — the identical holding-stage-column filter and closed/fabrik:paused exclusion routeQueuedGroup (§”Engine Selection” above) already applies, so a poisoner already excluded from batch formation is equally excluded here. For each remaining member it skips native-merge-queue items (MergeQueue != "off" && LinkedPRIsMergeQueueEnabled, or fabrik:auto-merge-enabled already present) — ejectMember/MaxMergeTrainEjections have no meaning for a member GitHub’s own queue is landing, mirroring routeQueuedGroup’s FR-3 precedence exactly — then calls FetchItemDetails itself (the deep-fetch deepFetchCandidates never performs for a Queued item) and checks currentHeadReviewThreadComments — the ADR-1207-canonical, current-head-scoped primitive (not raw buildReviewThreadComments), so a thread anchored to a commit the PR has since moved past never triggers a spurious ejection.

Direct-eject vs. pending-signal split (the concurrency design). The three pre-existing ejectMember call sites all run from inside the merge-train worker goroutine that owns the batch state, so there is no concurrent-mutation hazard: the same goroutine that decides to eject also owns the slice being mutated. This scan’s trigger is different in kind — it fires from the poll loop, which can run concurrently with a worker blocked for up to CIWaitTimeout inside pollTrainCI. Reaching into that goroutine’s own in-memory batch slice from outside it would race the worker’s own assemble/validate/land sequence. The scan therefore branches on batch membership, not repo-level activity: mergeTrainBatchMembers(repoKey) returns the in-flight worker’s batchNumbers — the issue-number set it was dispatched with, snapshotted once in dispatchMergeTrainWorker and immutable afterward (worker membership only ever shrinks via ejection/landing, never grows).

Why batch membership, not mergeTrainWorkerActive, decides the branch. A worker is always dispatched with its batch already capped to effectiveMaxBatchSize (default 5, capBatch/FR-4) — a Queued member beyond that cap is never part of any worker’s in-memory batch for as long as that worker runs. An earlier version of this scan branched on mergeTrainWorkerActive(repoKey) alone (any worker active for the repo → always the pending-signal path); found in review (issue #1208) to leave such an overflow member’s pending-eject signal permanently unconsumed, since none of the three checkpoints above ever iterate an issue number outside the dispatched batch — silently reproducing the very multi-poll-cycle blackout this settle scan exists to close, just for members outside the front of the queue. Checking batchNumbers membership directly closes this: an overflow member is ejected immediately, exactly as safely as the no-worker-at-all case, since a worker never looks at, fetches, or mutates an item outside the batch it was dispatched with.

This is checkpoint-based, not continuous preemption — a pending eject flagged mid-bisection is only applied at the next outer-loop checkpoint, the same granularity the runaway guard already has.

Reroute-then-eject ordering. ejectQueuedMemberForReviewFindings calls rerouteQueuedMemberOffHolding before posting the ejection comment or incrementing the MaxMergeTrainEjections counter. rerouteQueuedMemberOffHolding moves the board Status to stageBeforeHolding — the non-Unmanaged stage with the highest Order strictly less than the holding stage’s (derived structurally, not hardcoded to "Validate", so a custom stage config whose pre-Queued stage isn’t literally named Validate is still handled) — via a plain UpdateProjectItemStatus call, with the same cache write-through/SelfWriteObserved/webhook-echo bookkeeping advanceToQueued already uses for the reverse move. If the reroute fails (status-field metadata unavailable, or the mutation itself errors), ejectQueuedMemberForReviewFindings returns immediately — no comment is posted, no counter is incremented — so a transient board-mutation failure can never produce a duplicate ejection comment or double-count toward MaxMergeTrainEjections; the settle scan simply re-detects the same still-unresolved thread on a member still sitting in Queued and retries the whole operation next poll.

Distinct ejection wording (stayInQueue). ejectMember gained a stayInQueue bool parameter rather than a new sibling function, so the existing comment-posting/counter/pause-at-cap logic is reused unconditionally, exactly as the issue’s Requirements ask. All five pre-existing call sites (unfetchable linked PR, missing head SHA, unresolvable conflict, bisection-isolated poisoner, singleton-fails-in-isolation) pass stayInQueue=true and keep their unchanged closing sentence (“This issue remains in the Queued column and will be retried in a future train with a different composition.”). This new cause passes stayInQueue=false, producing the opposite, textually distinguishable sentence: “This issue has left the Queued column so the unresolved review-thread finding above can be addressed via the normal review pipeline. Once addressed and Validate completes again, it will re-queue and join a later batch.” — so an operator can tell from the ejection comment alone which of the four causes fired.

MaxReviewCycles composes across the eject/re-queue cycle “for free.” The reroute is a plain status move only — it never touches stage:Validate:complete (already present from the original Validate completion, and never removed by advanceToQueued/advanceToNextStage on the Queued visit) and never calls EngineCyclesCleared (the only mutation that zeroes ReviewCycles and, as of #1518/ADR-1518, ReviewBlockedCycles — otherwise reachable only via the manual-intervention paths in §7). Because both are left untouched, the very next poll admits the rerouted item to Phase 1 with hasComplete=true; handleReviewGate — completely unmodified by this issue, and, per #1518’s own Research pass, requiring no merge-train-specific change either — finds the same still-unresolved thread comment (dedup is keyed on the 🚀 reaction / ProcessedComments, not on ejection state) and dispatches through the existing dispatchWithCycleLimit/max(ReviewCycles, ReviewBlockedCycles) gate (§6.2 Handler 2), keyed by stage name exactly as ordinary CHANGES_REQUESTED reinvocation is. A member that keeps attracting findings therefore eventually escalates via pauseForReviewCycleLimit, the same terminal fallback any other Validate-stage item reaches — rather than oscillating Queued ↔ Validate unbounded. MaxMergeTrainEjections (a separate, pre-existing bound reused via ejectMember) fires independently of ReviewCycles/ReviewBlockedCycles from the same repeated-ejection counter every other ejectMember cause already increments — a member could in principle hit the train-ejection pause cap purely from repeated review findings before MaxReviewCycles ever fires; the two bounds are not substitutes for each other.

Scope: mid-flight ejection only, no admission-time filtering. A member with an unresolved thread is still admitted into a forming batch; it is ejected once this scan (or a worker checkpoint) observes the finding. groupQueuedByRepoAndBase/routeQueuedGroup’s batch-composition logic is untouched.

Comment cause: an unprocessed human comment ejects the member too (#1863, ADR-1863). Review-thread findings were the only feedback this scan noticed; an ordinary issue or PR comment posted while the member sat Queued was ignored until the member landed (where the #1862 post-merge guard declines it). The same scan now also detects an unprocessed human comment and ejects the member so the ordinary comment path can act on it.

State transitions:

Before Trigger After Labels/State Added Labels/State Removed
Queued, no unresolved review-thread findings Settle pass evaluated Unchanged — —
Queued, unresolved finding(s), member’s issue number NOT in any live worker’s batchNumbers (no worker for the repo, or a worker is active but this member is beyond the batch cap) Settle pass detects finding(s) Rerouted to stageBeforeHolding (e.g. Validate); ejection counter incremented Ejection comment (stayInQueue=false wording) — (stage:Validate:complete and ReviewCycles deliberately untouched)
Queued, unresolved finding(s), member’s issue number IS in the live worker’s batchNumbers Settle pass detects finding(s) Unchanged this poll — pending-eject signal recorded — —
Run holds a pending-eject signal for a member Checkpoint reached (the run state machine’s verdict step for the main trial, the one-at-a-time per-singleton step, or landGreenBatch’s main-moved rebase loop) Trial discarded regardless of CI result; member rerouted + ejected exactly as the direct-eject row above; the re-form loop and landOneAtATime continue with the reduced membership, while landGreenBatch discards the whole trial and returns (non-flagged survivors re-form on a future poll) Ejection comment (stayInQueue=false wording) —
Queued, no review findings, unprocessed human comment(s), member NOT in any live worker’s batchNumbers (#1863) Settle pass detects filterHuman(findNewComments) non-empty Rerouted to stageBeforeHolding; no ejection count, no pause Comment-eject comment (ejected (unprocessed comment)) — (stage:Validate:complete and ReviewCycles untouched)
Queued, no review findings, unprocessed human comment(s), member IS in the live worker’s batchNumbers (#1863) Settle pass detects them Unchanged this poll — comment pending-eject signal recorded — —
Worker holds a pending comment-eject signal for a member (#1863) Same checkpoints as the review-finding row above Trial discarded; member rerouted exactly as the direct row (no count, no pause); the rest of the batch continues Comment-eject comment —
Queued, only bot/engine/🚀’d/watermarked comments (#1863) Settle pass evaluated Unchanged — never ejected — —
Comment-ejected member, comment processed, Validate completes again (#1863) handleStageComplete / runCatchUpPhase2 Back in Queued, eligible for a later batch — —
Rerouted member reaches handleReviewGate on the next poll, max(ReviewCycles, ReviewBlockedCycles) < MaxReviewCycles Still-unresolved thread comment found Review reinvoke dispatched (unmodified path) — —
Rerouted member reaches handleReviewGate, max(ReviewCycles, ReviewBlockedCycles) >= MaxReviewCycles Still-unresolved thread comment found Paused (pauseForReviewCycleLimit, unmodified path) fabrik:paused, fabrik:awaiting-input —
Member ejected for this cause MaxMergeTrainEjections consecutive times (independent of ReviewCycles) Ejection counter reaches cap Paused fabrik:paused, fabrik:awaiting-input —
Member addresses the finding, Validate completes again advanceToQueued Back in Queued, eligible for a later batch stage:Validate:complete (already present, unchanged) —

References: ADR-1208: Queued Review-Finding Ejection, ADR-059: Internal Merge Train (ejectMember’s original three causes, the runaway guard’s two-hook precedent this issue’s pending-eject signal mirrors), ADR-1420: Merge-Train Ejection Diagnostics (diag/otherMembers contract — this new cause passes nil, nil, consistent with the three pre-1420 call sites), ADR-1270: Awaiting-CI Settle Scan (the direct template for this settle scan’s admission-independence shape), ADR-1207: Yolo-Merge Review-Thread Guards (the sibling Validate-side race; establishes currentHeadReviewThreadComments as the canonical detection primitive this issue reuses), ADR-067 (finishTrain as the single clear point for mergeTrainInFlight/Store.ExitRepoWorker; the worker-checkpoint early-return paths added by this issue still route through it), issue #1208; ADR-1863: Queued Comment Ejection (the comment cause), issue #1863

6.17 Terminal-Advance Escalation and Settle Scan (ADR-1422)

Trigger: Issue #1422 (reported by @bdueck in #1082). A terminal advance (advanceToNextStage, called from advanceValidateTerminalItem’s merged-PR path and advanceConvergedPRToDone) that cannot resolve its target Status option — most commonly because the intermediate or Done column named by stages.NextStage has no matching option on the project board — previously only logged a warning and returned. The issue stayed open with every stage complete and a merged, green PR: nothing red, nothing paused, nothing requesting attention. Because native dependency edges clear on close, not on merge, every downstream issue that listed the stranded issue as a blocker kept holding fabrik:blocked, waiting on work that had already shipped. In the reported incident, three merged issues stranded five downstream slices, and the stall was found only by reading fabrik.log directly, after it had been in effect for some time — the most expensive shape of stall Fabrik can produce, because the natural operator response to “quietly working” is to wait longer.

Same failure class as §6.9/§6.10/§6.13/§6.14/§6.15/§6.16: a durable stranding condition invisible to the operator and only fragile-at-best self-healing via the shared, admission-gated catch-up path. settleAwaitingAdvanceScan is the seventh instance of the ADR-1270 pattern. Research (see the R6 audit below) found this call site was not, in fact, never retried — advanceValidateTerminalItem’s closed-item half is already board.Items-sourced via settleClosedValidateAdvance (§6.15, ADR-1387), and advanceConvergedPRToDone’s admission-gated path mostly kept retrying as long as stage:Validate:complete stayed present. The two missing pieces were visibility (R1 — nothing signaled the stall to the operator) and boundedness (R3 — an admission-gated retry has no escalation path when something else in the pipeline starves it on a given poll).

Durable marker (fabrik:awaiting-advance), written only on failure — mirrors fabrik:awaiting-close (§6.13), not fabrik:awaiting-done (§6.8). advanceToNextStage is the last mutation at both call sites — by the time it can fail, every other side effect (gate-label clearing, completion-label filling, fabrik:auto-merge-enabled removal) has already landed. There is nothing left to redo except the status move itself, so a generic retry of the bare advanceToNextStage call is both correct and sufficient — no re-derivation of “which caller failed” is needed, since stages.FindStage(e.cfg.Stages, item.Status) always resolves back to the same stage the original caller had (the move is exactly what didn’t happen).

One shared label/counter/scan for both call sites, not two. Both failures are the identical “final board-Status move failed” shape with identical recovery, so a single recordAdvanceOutcome wrapper — routed through by both advanceValidateTerminalItem and advanceConvergedPRToDone in place of calling advanceToNextStage directly — avoids duplicating the label/comment/escalation machinery for no behavioral difference.

First-failure comment, gated on the label’s own absence (R1, R5). Unlike fabrik:awaiting-close (which only comments at escalation), markAdvanceFailureOutstanding posts a one-time explanatory comment on the very first failure — naming the failing stage and embedding the underlying error verbatim (react=false, mirroring handleUsageLimitExit’s fabrik:claude-limit shape, §7.3). For the “no status option” case, advanceToNextStage’s own error text already names the missing option and the options that do exist (no status option %q found on project board (available: %v)), so no new typed-error plumbing is needed to satisfy R1. Every failing pass — first or repeat — still counts toward the retry budget via recordSettleRetry, but only the absent→present transition ever posts a comment, so repeated failures in the same episode never produce more than one (R5).

No dispatch-suppression wiring, no transientLifecycleLabels entry — mirrors fabrik:awaiting-close’s reasoning exactly (§6.13). By the time advanceToNextStage can fail at either call site, stage:<X>:complete (or the gate-checked completion labels advanceValidateTerminalItem fills first) is already present, so ordinary admission already treats the item as stage-complete-and-idle. Adding fabrik:awaiting-advance to itemMayNeedWork/itemNeedsWork would be inert (nothing left to suppress); adding it to transientLifecycleLabels would let the closed-issue sweep race the settle scan, reproducing the exact class of bug ADR-1387 R6 (§6.15) closed for fabrik:awaiting-ci/fabrik:awaiting-review/fabrik:rebase-needed at Validate.

Code path: advanceValidateTerminalItem / advanceConvergedPRToDone → recordAdvanceOutcome → markAdvanceFailureOutstanding (writes the marker + first-failure comment on failure) — and, on retry, settleAwaitingAdvanceScan → recordAdvanceOutcome (all in engine/advance_settle.go; settleAwaitingAdvanceScan is called from poll() in poll.go) directly.

Retry-owner: settleAwaitingAdvanceScan (engine/advance_settle.go; called from poll() in poll.go, immediately after settleClosedValidateAdvance). Runs unconditionally once per poll, iterating the raw board.Items — not deepFetchCandidates — for the same reason as every prior settle scan in this family: a stranded item’s stage is already complete, and admission does not reliably see it (this is exactly the fragility Research identified for advanceConvergedPRToDone’s pre-fix path). For every item carrying fabrik:awaiting-advance and not fabrik:paused, not already in this poll’s advancedItems dedup set, and resolving to a real configured stage via stages.FindStage(e.cfg.Stages, item.Status), it calls recordAdvanceOutcome again.

Shares the poll’s advancedItems dedup map with runValidatePRTerminalAdvance/settleClosedValidateAdvance (§6.15). For a closed item stuck at Validate, that pair already retries advanceToNextStage unconditionally every poll and, running earlier in the same poll() pass, will already have set advancedItems[iKey] = true before this scan reaches it — so this scan is a harmless, correctly-skipped no-op there. It is the exclusive retry-owner only for the two genuine gaps: an open item admission-gated out of runValidatePRTerminalAdvance’s deepFetchCandidates source, and advanceConvergedPRToDone’s path, reachable only via the Phase 1 catch-up handler chain and otherwise unowned on any poll where that chain doesn’t reach it.

Retry counting and escalation (R3). Every failing pass — inside markAdvanceFailureOutstanding, called from both the original call sites and the settle scan’s retries — calls recordSettleRetry, keyed by the dedicated constant "__awaiting_advance__" (same double-underscore-wrapped, YAML-unrepresentable shape as the rest of this family). MaxRetries <= 0 means unlimited retries, never escalate (same guard every settle scan in this family shares). Once Attempts("__awaiting_advance__") >= e.cfg.MaxRetries, escalateAwaitingAdvanceFailure fires: adds fabrik:paused and posts an explanatory comment naming the attempt count with the manual-fix instruction (check every stage name has a matching board Status option, then remove fabrik:paused) — but, unlike every other escalation in this family, does not remove fabrik:awaiting-advance (see below).

Marker persists through escalation — the one deliberate deviation from this family’s shared shape (Pruefer, PR #1469 review, second round). Every other settle scan in this family (fabrik:awaiting-done, fabrik:awaiting-member-close, fabrik:awaiting-close, etc.) removes its marker label on escalation via the shared escalateSettle helper, whose own doc comment justifies this with “dispatch/retry suppression is no longer needed once fabrik:paused takes over.” That holds for their direct call sites only because each one unconditionally re-creates the marker on its very next relevant poll regardless of pause state (e.g. closeIssueIfNonDefaultBase, §6.13, has no fabrik:paused guard at all). advanceConvergedPRToDone (merge_gate.go) does not have this property: removing fabrik:auto-merge-enabled up front structurally prevents checkAutoMergeConvergence from ever re-entering it, so it fires at most once per episode. For an item whose only driver is settleAwaitingAdvanceScan (i.e. this call site’s items specifically), removing the marker at escalation would permanently strand it — nothing would ever re-create it, even after an operator fixes the board and removes fabrik:paused alone, which the escalation comment’s own recovery instruction promises is sufficient. escalateAwaitingAdvanceFailure therefore does not call the shared escalateSettle helper; it inlines the same pause/comment/EnginePaused steps without the marker removal. This is safe: settleAwaitingAdvanceScan’s own admission guard already skips any item carrying fabrik:paused, so the marker sitting alongside fabrik:paused is inert until the pause is lifted.

Counter reset on unpause, in both retry owners. Because the marker (and hence Attempts("__awaiting_advance__"), already at or above MaxRetries) survives escalation, both awaitingAdvanceStuckOrReset (called by advanceValidateTerminalItem before it touches anything, so it can skip the whole branch while still paused) and awaitingAdvanceResetIfUnpaused (called by settleAwaitingAdvanceScan directly, whose own pre-filter already excludes paused items) reset the counter via StageRetryCleared the moment they observe the item is no longer paused — mirroring clearFailedStage’s reset, scoped to this synthetic retry key. Without this, a retry issued right after a manual unpause would immediately re-escalate on its very first subsequent failure instead of getting a fresh MaxRetries budget (Pruefer’s first-round second finding, which turned out to apply to both retry owners, not just the direct call sites).

Marker clearing. fabrik:awaiting-advance is removed in exactly one place: clearAwaitingAdvanceMarker, on a fully successful recordAdvanceOutcome retry pass. Adding the missing board option is sufficient to unstick the item on the very next poll — no engine restart, no manual re-dispatch, and (per the two points above) no manual re-labeling either, even for an item that was previously escalated and paused (R2, Acceptance #2).

R6 audit of the sibling warn-and-return call sites named in #1422’s Problem section:

Three other advanceToNextStage call sites exist in the codebase (poll.go, stages.go, merge_train.go) but are not named in #1422’s Scope section and are left untouched — a candidate follow-up, not fixed here.

Scope: only the bare status-move failure, nothing else. recordAdvanceOutcome does not duplicate closeIssueIfNonDefaultBase (§6.13) — that call already runs once, unconditionally, at both original call sites regardless of advanceToNextStage’s outcome, and has its own independent retry owner (fabrik:awaiting-close) if it also fails. “Closing the issue,” part of R2/Acceptance #2’s phrasing, needs no new code here: GitHub’s own Closes #N auto-close fires on merge for the default branch regardless of Fabrik’s board-Status bookkeeping, and closeIssueIfNonDefaultBase already owns the non-default-base explicit close independently. This section’s sole responsibility is the stuck board-Status move.

Corollary (Pruefer, PR #1469 review, third round): because closeIssueIfNonDefaultBase runs regardless of recordAdvanceOutcome’s outcome, a base:<branch> issue whose terminal advance just failed can still end up closed on the very same pass — fabrik:awaiting-advance and a stranded board Status coexisting with a closed issue. This is pre-existing ordering (unchanged by this section’s introduction), and harmless in practice: settleAwaitingAdvanceScan does not filter on IsClosed, so the retry (and, on exhaustion, the escalation pause/comment) still fire correctly — they just land on an issue that is already closed, which can read oddly to an operator glancing at the thread. No code change follows from this; it is recorded here so the ordering is discoverable without re-deriving it from the two call sites.

R4 — no silent fallback to Done. #1082 suggested falling back to Done when the intermediate option is absent. Rejected as the primary mechanism: it would paper over a misconfigured board and hide this exact bug. No fallback of any kind is implemented — a failed advance stays failed, visibly, until the board is fixed or MaxRetries escalates it.

State transitions:

Before Trigger After Labels Added Labels Removed
Any active stage, stage-complete recordAdvanceOutcome’s advanceToNextStage call fails (first failure) Unchanged column, stranded fabrik:awaiting-advance —
Stranded (marker present) Settle pass: advanceToNextStage still fails Unchanged (no second comment) — —
Stranded (marker present) Settle pass: advanceToNextStage succeeds (board fixed) Next stage — fabrik:awaiting-advance
Stranded (marker present) Attempts("__awaiting_advance__") >= MaxRetries Paused, still stranded fabrik:paused — (marker deliberately retained — see above)
Paused, still stranded (both labels present) Operator removes fabrik:paused only (board not yet fixed) Stranded, fresh retry budget — — (Attempts reset to 0 via awaitingAdvanceStuckOrReset/awaitingAdvanceResetIfUnpaused)
Stranded (marker present, post-reset) advanceToNextStage succeeds (board now fixed) Next stage — fabrik:awaiting-advance

References: ADR-1422: Terminal-Advance Escalation and Settle Scan, ADR-1097: Non-Default-Base Explicit Close Retry (§6.13, the closest structural precedent — marker-on-failure-only, single at-risk call after other side effects already landed), ADR-1387: Closed Items Are Never Dispatched (§6.15, defines the current closed-item admission rule that already retries advanceValidateTerminalItem’s closed half), ADR-1270: Awaiting-CI Settle Scan (the “admission-gated retry is fragile even when it mostly fires” rationale, and the direct template for this pattern), ADR-064: Closed-Item-At-Any-Stage Advance To Done (rationale for closed_item_advance_settle.go’s deliberately unbounded retry, audited above and left unchanged), ADR-060/061/062/1208 (the other instances of the dedicated-settle-scan pattern this section follows), issue #1422, issue #1082

6.18 Runaway Guard Alert Retry (ADR-1533)

Channel events (#1968): fireRunawayGuard pausing a member emits merge-train-failed (cause runaway-guard); see §7.14.

Trigger: Issue #1533. fireRunawayGuard (see the “Runaway guard (ADR-059 D8)” section above for its full pause+alert contract) pauses every member it’s given, but its alert AddComment call and its fabrik:paused/fabrik:awaiting-input label calls were independent, unconditional GitHub API calls — a comment failure was logged and the loop continued straight into the label calls anyway. Because groupQueuedByRepoAndBase (Hook 2’s snapshot source) excludes any fabrik:paused member from every subsequent poll’s Queued snapshot, and Hook 1 (the worker goroutine) only ever knows the members it started with, a member whose comment failed had no path back to a retry — it stayed fabrik:paused with no explanation, permanently, unless an operator noticed and investigated by other means (e.g. a sibling member’s comment in the same batch). This is the same “stranded with no signal” shape ADR-060/#1422/#1408 already treat as a defect.

Same failure class as §6.8/§6.9/§6.10/§6.13/§6.14/§6.15/§6.16/§6.17 — the eighth instance of the ADR-1270 dedicated-settle-scan pattern. The durable marker (fabrik:awaiting-runaway-alert), retry-owner settle scan (settleRunawayGuardAlertScan), and MaxRetries-bounded escalation all reuse the shared recordSettleRetry/clearSettleMarker/escalateSettle helpers (engine/settle.go) this whole family shares.

The one structural difference from every sibling in this family: fabrik:paused is not a “later” state, it’s a “day one” state. For fabrik:awaiting-member-close/fabrik:awaiting-close/fabrik:awaiting-advance, the marker is written when some terminal step fails after the item has already reached its natural resting state — the item isn’t fabrik:paused yet, and each settle scan explicitly skips items that are (an operator investigating a pause must not be fought, and a paused item there means “already escalated, stop retrying”). Here, fabrik:paused is applied by fireRunawayGuard unconditionally, in the same loop iteration as the comment attempt, regardless of whether the comment succeeded — so a member carrying fabrik:awaiting-runaway-alert always also carries fabrik:paused, from the very first poll the marker exists. settleRunawayGuardAlertScan therefore does not gate on fabrik:paused’s absence (unlike settleMergeTrainMemberCloses/settleNonDefaultBaseCloses/settleAwaitingAdvanceScan) — doing so would make it a permanent no-op. The marker’s own presence/absence is the sole retry-eligibility signal.

Atomicity/idempotency is the primary fix; the settle scan is the residual-failure backstop. Most of #1533’s defect (a member appearing in two racing fireRunawayGuard calls’ item slices) is closed by mergeTrainRunawayMu/mergeTrainRunawayAlerted — see the “Atomicity and per-episode idempotency” paragraph in the Runaway guard section above. The settle scan exists for the residual case that atomicity alone cannot fix: a single call’s AddComment failing outright (network error, rate limit) for a member no future call will ever revisit, because that member is now fabrik:paused and therefore excluded from groupQueuedByRepoAndBase forever.

Code path: fireRunawayGuard → markRunawayAlertOutstanding (writes the marker on comment failure only; engine/runaway_alert_settle.go) — and, on retry, settleRunawayGuardAlertScan → settleRunawayGuardAlert (same file; called from poll() in poll.go, immediately after settleMergeTrainMemberCloses) directly.

Retry-owner: settleRunawayGuardAlertScan. Runs unconditionally once per poll — independent of merge_train: on/off, so a marker written while it was enabled keeps draining even if the setting is later turned off (mirroring settleMergeTrainMemberCloses) — iterating the raw board.Items, not groupQueuedByRepoAndBase or deepFetchCandidates: the member has already reached its terminal “paused, awaiting operator” state by the time the marker is written, and groupQueuedByRepoAndBase’s own fabrik:paused exclusion (load-bearing for the poison-well guard, unrelated to this scan) would otherwise hide exactly the items this scan exists to find. For every item carrying fabrik:awaiting-runaway-alert, it calls settleRunawayGuardAlert, which acquires mergeTrainRunawayMu for its entire body — the same critical section fireRunawayGuard uses — before re-reading the count/window live via isRunawayTripped/effectiveTrialWindow (no storage persists the original firing’s exact count — a live re-read is accurate enough for the retry comment’s wording) and retrying the postComment call. Holding the mutex across the retry, not just the map update afterward, matters: a member carrying this marker can still appear in a later Hook 1 call’s own current/survivors (the worker’s own in-flight member list, not a fresh board read that would exclude an already-fabrik:paused item — unlike Hook 2’s groupQueuedByRepoAndBase), so without the lock spanning the comment post itself, a racing fireRunawayGuard call and this settle retry could both observe “not yet alerted” and post the comment concurrently. On success, the member is recorded in mergeTrainRunawayAlerted (so a still-in-flight or later fireRunawayGuard call for the same member, within the same episode, skips it rather than double-posting) and the marker is cleared.

Retry counting and escalation. Every failed retry calls recordRunawayAlertRetry, keyed by the dedicated constant "__awaiting_runaway_alert__" (same double-underscore-wrapped, YAML-unrepresentable shape as the rest of this family). MaxRetries <= 0 means unlimited retries, never escalate. Once Attempts("__awaiting_runaway_alert__") >= e.cfg.MaxRetries, escalateRunawayAlertFailure fires.

Escalation removes the marker only if its own fallback comment succeeds — a deliberate departure from the rest of the family. fabrik:paused is already present (re-applying it here is a harmless no-op) and is not the thing being escalated into — the escalation’s job is purely to guarantee the member ends up with some explanation. escalateRunawayAlertFailure posts a fallback comment ("🏭 **Fabrik merge-train — runaway guard alert delivery failed**") embedding the same runawayGuardAlertMessage body the original alert would have carried. Unlike every other escalation in this family, it does not delegate to the shared escalateSettle helper: escalateSettle unconditionally removes the marker and discards its comment closure’s error (via postItemComment), which is fine when the escalation comment is purely informational — but here the fallback comment is the last remaining delivery of the explanation R1 requires. escalateRunawayAlertFailure therefore checks the fallback’s own result: on success, the marker is cleared and the member is recorded alerted (mergeTrainRunawayAlerted); on failure — a persistent AddComment outage outlasting MaxRetries, not just a transient one — the marker stays, so the next settleRunawayGuardAlertScan pass keeps retrying the primary alert, then the fallback, every poll, indefinitely, until a comment actually lands (#1533 review, finding 1: the original shape removed the marker regardless of the fallback’s outcome, which could erase the only diagnostic signal that the alert never landed).

Marker clearing. fabrik:awaiting-runaway-alert is removed in exactly two places: clearRunawayAlertMarker (a fully successful settleRunawayGuardAlert pass, or a racing fireRunawayGuard call for the same member succeeding first — both go through the identical helper) and escalateRunawayAlertFailure (giving up after MaxRetries, as described above).

State transitions:

Before Trigger After Labels Added Labels Removed
Queued (or any column), not runaway-flagged fireRunawayGuard: AddComment succeeds Same column, paused + alerted fabrik:paused, fabrik:awaiting-input —
Queued (or any column), not runaway-flagged fireRunawayGuard: AddComment fails Same column, paused + alert outstanding fabrik:paused, fabrik:awaiting-input, fabrik:awaiting-runaway-alert —
Paused, alert outstanding (marker present) Settle pass: AddComment succeeds Same column, paused + alerted — fabrik:awaiting-runaway-alert
Paused, alert outstanding (marker present) Settle pass: AddComment still fails Unchanged — —
Paused, alert outstanding (marker present) Attempts("__awaiting_runaway_alert__") >= MaxRetries, fallback comment succeeds Paused, fallback comment posted — fabrik:awaiting-runaway-alert
Paused, alert outstanding (marker present) Attempts("__awaiting_runaway_alert__") >= MaxRetries, fallback comment also fails Unchanged — still retried every poll — —

References: ADR-1533: Runaway Guard Atomic Pause and Alert, ADR-059: Internal Merge Train §D8 (original runaway guard design — see ADR-1533’s correction note), ADR-061: Merge-Train Singleton Member-Issue Close Retry (§6.10, the closest structural precedent among this family), issue #1533

6.19 Post-Done Landing Verification (ADR-1616)

Channel events (#1968): The fabrik:awaiting-landing-verification label (applied right after a merge-attributable Done transition) is what emits merged; failLandingVerification applying fabrik:landing-verification-failed emits landing-verification-failed; see §7.14.

Trigger: Issue #1614. A merge-train batch attribution bug closed two issues as COMPLETED whose work was never actually merged — the Done transition is driven entirely by inferred success (a merge call returned, or a PR was believed merged), never by observing the credited work actually landing. This is silent and self-certifying: the issue reads COMPLETED, the PR reads landed, and no dispatch path ever revisits a Done item, so there is no signal for anything to notice later. Issue #1615 fixes the specific root cause; this is the cause-agnostic backstop that would have caught it — and would catch the next one, whatever its cause.

Mechanism: merge-state of the crediting PR — not branch-tip ancestry. The issue’s Prior Discussion thread recorded a live false-positive audit: a literal git merge-base --is-ancestor-style check, hand-run against 25 real closed issues in this repo, flagged three (#1573, #1523, #1520) as “not landed” that had, in fact, landed — because their branches were rebased onto a later base after landing, so the same content exists on base under different commit SHAs while the branch tip is no longer an ancestor. A merge-train member is especially prone to this: the train merges the trial branch, and the member’s own branch then drifts independently. Separately, under --auto-merge-strategy SQUASH/REBASE, GitHub creates a new commit object on the base branch, so a merged PR’s original branch-tip commit is never a literal ancestor — branch-tip ancestry would false-positive on every correctly-shipped issue in a non-MERGE-strategy repo, not just drifted ones. Both findings point the same way: the property this issue needs cannot be expressed as a fact about the item’s own branch tip at all. The chosen mechanism instead asserts that the PR credited for the Done transition reached MERGED (FetchPRMerged) — inherently strategy-agnostic (GitHub’s REST merged boolean means the same thing under MERGE/SQUASH/REBASE) and immune by construction to both false-positive classes, since it never inspects the item’s own branch. R3 (a branch legitimately deleted after a real merge is not a failure) is satisfied the same way — the branch is simply never examined.

Discovery: a durable label recorded at the Done transition itself, not comment-parsing. Three landing paths write to Done, and none of them carries PR/merge identity forward through advanceToNextStage’s bare Status-field mutation. What differs is which PR is credited and how it’s later rediscoverable:

Path Credited PR How it’s found later
Ordinary auto-merge (advanceConvergedPRToDone, advanceValidateTerminalItem’s merged branch) The issue’s own linked/closing PR FetchLinkedPR(owner, repo, issueNumber) — durably rediscoverable, no new storage needed
Merge-train batch (landMergeTrainBatch) The integration PR — distinct from the member’s own PR, which stays closed-not-merged fabrik:credited-pr:<N>, applied at the Done transition
Merge-train singleton (landSingleton) A dedicated singleton landing PR — again distinct from the member’s own PR fabrik:credited-pr:<N>, applied at the Done transition
Merge-train singleton fast path (trySingletonFastPath, §6.20, ADR-1644) The member’s own PR — no separate PR is ever minted FetchLinkedPR(owner, repo, issueNumber) — durably rediscoverable, same as the ordinary auto-merge path, no fabrik:credited-pr:<N> needed

An earlier design considered parsing the existing "🏭 **Fabrik merge-train** — Landed via batch PR #%d." comment landMergeTrainBatch/landSingleton already post — rejected because addLandedCommentWithRetry is best-effort (three attempts, then a silent log warning) and the pattern is coupled to exact prose with no test tying the two together; a verifier that goes quiet exactly when the landing path is already misbehaving is not a backstop (the same failure shape as #1614 itself — a mechanism that appears to protect while silently doing nothing). Recording fabrik:credited-pr:<N> as a label at the transition — cheap (the number is already in hand), durable (survives a restart, unlike an itemstate field — itemstate has no cross-restart persistence and GitHub exposes no change-history equivalent to backfill it), and matching this repo’s “labels are state” convention — removes that dependency entirely. Not applied by the ordinary auto-merge path, whose credited PR needs no such capture.

Marker-driven scope, not backlog-driven. fabrik:awaiting-landing-verification is applied at the Done transition itself by all three call sites, immediately after their own advanceToNextStage/recordAdvanceOutcome call succeeds — not unconditionally the way closeIssueIfNonDefaultBase runs regardless of the advance’s outcome (§6.13), since this marker’s entire purpose is to verify a Done transition that must have actually happened. settleLandingVerification owns items carrying the marker exclusively, sourced from raw board.Items (ADR-1270 precedent), never deepFetchCandidates. Old Done items — anything that landed before this feature shipped — never had the marker applied, so they are naturally out of scope: no backfill pass, no risk of misclassifying a pre-feature merge-train landing (whose member PR is legitimately closed-not-merged) as a failure, and AC1’s “within one poll cycle” is trivially satisfied since the marker exists from the same poll the transition happens in.

R4 (base:<branch>) is satisfied by the mechanism itself, not by an active check. Verifying “did the credited PR merge” needs no base-branch resolution at all — the credited PR’s own base is whatever it was opened against, already base:<branch>-correct at PR-creation time (#1554). baseBranchForItem/WorktreeManager resolution (resolveLandingVerificationBranchInfo) is used only to name the base in the failure comment — mirroring closeIssueIfNonDefaultBase’s existing best-effort, degrade-gracefully pattern (§6.13) — never as an input to the merge/no-merge decision. A repo with no registered WorktreeManager degrades to an unnamed base in the comment, never a blocked scan.

Two failure regimes, never conflated:

  1. Confirmed failure — the credited PR resolves and FetchPRMerged definitively reports false. failLandingVerification fires immediately, on the same pass that reaches this determination — never gated by the retry counter, so AC1’s “within one poll cycle” holds regardless of MaxRetries.
  2. Inconclusive — a FetchPRMerged/FetchLinkedPR API error, or no crediting reference discoverable at all (no fabrik:credited-pr:<N> label and no linked PR). Absence of evidence is not evidence of a false COMPLETED, so this goes through the same recordSettleRetry/escalateSettle idiom (engine/settle.go) every settle-scan sibling in this family shares, bounded by MaxRetries, escalating to fabrik:paused — never reopens the issue on ambiguity (R5, AC4).

Conflating these would either reopen a correctly-closed issue on a single transient hiccup, or silently retry a confirmed false-COMPLETED forever.

Code path: landMergeTrainBatch/landSingleton/finishSingletonFastPathLanding (engine/merge_train.go), advanceConvergedPRToDone (engine/merge_gate.go), advanceValidateTerminalItem’s merged-PR branch (engine/pr_terminal_advance.go) — each applies fabrik:awaiting-landing-verification (and, for landMergeTrainBatch/landSingleton only, fabrik:credited-pr:<N>) immediately after its own successful Done transition. finishSingletonFastPathLanding (§6.20, ADR-1644) follows advanceConvergedPRToDone’s labeling exactly — see the table above. Retry-owner: settleLandingVerification → settleLandingVerificationItem (engine/landing_verification_settle.go; settleLandingVerification called from poll() in engine/poll.go, immediately after settleNonDefaultBaseCloses).

settleLandingVerificationItem flow:

  1. Resolve the credited PR: fabrik:credited-pr:<N> label if present → N; else FetchLinkedPR → its number. Neither found → inconclusive (recordLandingVerificationRetry).
  2. FetchPRMerged(owner, repo, creditedPR).
    • Error → inconclusive (recordLandingVerificationRetry).
    • merged == true → clearLandingVerificationMarkers removes fabrik:awaiting-landing-verification and any fabrik:credited-pr:<N>, clears the retry counter. No further action (AC2).
    • merged == false → failLandingVerification: reopens the issue (ReopenIssue, if closed), moves the item’s board Status back to the fixed target Validate (moveItemToValidate — a direct UpdateProjectItemStatus call with a looked-up optionID, since advanceToNextStage only ever moves forward to the next configured stage and has no “move backward to a caller-chosen target” mode), applies fabrik:landing-verification-failed, removes fabrik:awaiting-landing-verification/fabrik:credited-pr:<N>, and posts a comment naming the branch (fabrik/issue-<N>), the resolved base (best-effort — blank if no WorktreeManager is registered), and the credited PR checked and found not merged. Every step is independently logged and best-effort — a ReopenIssue failure must not prevent the board move, label, or comment from still happening.

The remediation is human-gated, and the comment says so. failLandingVerification moves the board Status back to Validate but deliberately leaves stage:Validate:complete in place, so the move alone does not re-dispatch the stage — matching R2’s “park it for inspection” posture rather than silently re-running a landing that just failed verification. Because a board move that doesn’t re-dispatch is exactly the hazard §6.16/#1545 R4 documents elsewhere, the failure comment states this explicitly and names fabrik:revalidate as the label that does re-run Validate, so an operator is never left waiting on a re-dispatch that will not come.

Restart safety at the merge-train landing sites. advanceToNextStage and the marker writes are separate, non-atomic API calls, so a run can advance a member to Done and die before markCreditedLanding executes. Both merge-train landing paths have a pre-existing “member already in Done column” fast path for exactly that restart shape — and both now call markCreditedLanding on that path too, rather than skipping straight past it. Without the backfill the item would be permanently unverified: nothing ever revisits a Done item, so the marker would never be applied and the backstop would be silently absent for precisely the merge-train restart scenario #1614 came from. Both label writes are idempotent, so re-marking an item whose verification already completed costs at most one redundant FetchPRMerged that confirms merged and clears the markers again.

“Move it off Done” target: Validate, uniform across all three paths. Validate already knows how to attempt/re-attempt landing (auto-merge, or re-entry into the merge-train’s holding stage on its next cycle) — a human-gated posture (parked for inspection via the reopen + distinguishing label + comment), not a silently self-healing one. "Validate" is a literal, hardcoded stage name here, the same convention used throughout this package (pr_terminal_advance.go, poll_settle.go).

Retry counting and escalation. Every inconclusive pass calls recordLandingVerificationRetry, keyed by the dedicated constant "__landing_verification__" (same double-underscore-wrapped, YAML-unrepresentable shape as the rest of this family, governing only the inconclusive regime — never the confirmed-failure regime). MaxRetries <= 0 means unlimited retries, never escalate. Once Attempts("__landing_verification__") >= e.cfg.MaxRetries, escalateLandingVerificationFailure fires: pauses the issue, removes the marker, and posts a comment deliberately worded differently from failLandingVerification’s confirmed-failure comment — explicitly not a confirmed failure, so an operator reading it does not conclude work was actually lost.

A known, narrow blind spot: a merge-train label write that fails outright. The two merge-train landing paths write their labels through markCreditedLanding, which applies fabrik:credited-pr:<N> first, with its error checked, and applies fabrik:awaiting-landing-verification only if that credited-PR record actually landed on GitHub. This ordering and check are load-bearing, not defensive habit: the two labels are separate GitHub calls, so the credited-PR write can fail on its own (rate limit, momentary 5xx). Were the marker applied regardless, the scan would find no fabrik:credited-pr:<N> label and fall back to FetchLinkedPR — which for a merge-train member resolves to the member’s own PR, closed and deliberately never merged. FetchPRMerged would report false and the scan would fire its immediate confirmed-failure path against a correctly-landed issue, precisely the false positive AC2 forbids.

So a lost credited-PR write leaves the item unmarked — not verified (the pre-#1616 status quo), logged loudly at the landing call site — rather than marked-and-unresolvable. Not verifying is a coverage gap; falsely reversing a good landing is a regression, and between the two the gap is strictly safer. A board snapshot taken between the two writes can therefore only ever show the credited-PR label without the marker, which the scan ignores — never the reverse.

Scope: covers only the three merge-attributable Done-transition call sites. No change to merge-train assembly, bisection, or batching — every write this feature makes is a label applied after advanceToNextStage has already succeeded, never a change to the landing logic itself. Issue #1615’s trial-identity root-cause fix is independent and out of scope here — this backstop detects and reverses a false COMPLETED within a poll cycle; it does not prevent one.

State transitions:

Before Trigger After Labels Added Labels Removed
(any Done-transition call site) Own advanceToNextStage/recordAdvanceOutcome call succeeds Done, awaiting landing verification fabrik:awaiting-landing-verification (+ fabrik:credited-pr:<N> for merge-train paths) —
Done, awaiting landing verification Settle pass: credited PR confirmed merged Done, verified — fabrik:awaiting-landing-verification, fabrik:credited-pr:<N>
Done, awaiting landing verification Settle pass: credited PR confirmed not merged Validate, reopened, failure-labeled fabrik:landing-verification-failed fabrik:awaiting-landing-verification, fabrik:credited-pr:<N>
Done, awaiting landing verification Settle pass: inconclusive (API error, or no crediting reference found) Unchanged — —
Done, awaiting landing verification Attempts("__landing_verification__") >= MaxRetries (inconclusive regime exhausted) Done, Paused fabrik:paused fabrik:awaiting-landing-verification

References: ADR-1616: Post-Done Landing Verification, ADR-1270: Awaiting-CI Settle Scan (the settle-scan pattern this is the ninth instance of), ADR-1096: Explicit Close on Non-Default-Base Merge / §6.13 (closest structural precedent — merge-adjacent, best-effort, settle-scan-backed), issue #1614 (the motivating incident), issue #1615 (its independent root-cause fix)


6.20 Merge-Train Singleton Fast Path (ADR-1644)

Trigger: Issue #1644. A merge-train batch containing exactly one Queued member pays the full trial cycle — a disposable trial worktree, a pushed branch, a draft CI PR, a complete CI run, a landing integration PR — even when the member’s own PR has already been validated against the exact tree the trial would produce. Fabrik’s own Validate-stage convention (rebase onto base before signalling completion) means a member arriving fresh at Queued is typically already up to date with base, making the trial’s re-validation redundant by construction on the common path. This is not a bug — ADR-1153’s check-completeness gate and pollTrainCI’s backstop timeout both behave correctly throughout — it is avoidable, correctly-performed, duplicate work.

Mechanism: skip the trial only on positive evidence, never on absence of evidence. trySingletonFastPath (engine/merge_train.go), called from runMergeTrainWorker’s re-form loop on every iteration where len(current) == 1, immediately before a trial would otherwise be built. singletonFastPathEligible requires all four of the following to hold; any single one failing — including any API error — falls through to the unmodified trial path:

  1. No re-push since batch formation (TOCTOU guard). The member’s live PR head SHA (fetched fresh via FetchPRDetails) still equals the trainMember.headSHA snapshot captured once by fetchTrainMembers at worker start. A member re-pushed between batch formation and this check must never be landed on stale evidence.
  2. The pinned base SHA is an ancestor of the member’s head. FetchCommitsBehind(owner, repo, p.baseSHA, m.headSHA) == 0 — checked against trialParams.baseSHA, the base pinned once at batch start (ADR-059 D-b), never a live origin/<base> read (R3), so a base that moves mid-flight cannot make a stale decision look current. behind_by == 0 means the pinned base is a fast-forward ancestor of the member’s head, so the trial’s tree (pinned_base + member) would be byte-identical to the member’s own PR head tree.
  3. The PR is mergeable. gh.MergeableStateAccepted(pr.MergeableState) (ADR-072) — the same {clean, unstable} acceptance test MergePR’s own self-gate uses; a dirty state means an actual conflict.
  4. The member’s own CI is green and complete. FetchCheckRuns must return at least one check run — zero check runs is rejected outright, before classifyLandingCI is even consulted, regardless of mergeable_state (see below for why this is stricter than classifyLandingCI’s own zero-check-runs handling). Given at least one check run, classifyLandingCI (ADR-1153/ADR-1441, reused unmodified — the same check-run-based classifier pollForMergeable uses for the ordinary landing path) must return TrainCIGreen. Since #1822 classifyLandingCI additionally returns TrainCIPending while any check suite has outstanding work or the suite read fails (§6.4 Suite-aware completeness), so a member whose own all-green runs are only a prefix of a still-running workflow is not eligible and falls through to the trial path.

Every branch that cannot positively confirm a condition returns (false, reason) — there is no code path that returns eligible except by every check having been positively satisfied. This is the deliberate inverse of #1614’s failure mode: there the train wrongly skipped a merge; here a wrong skip would mean skipping validation, so only positive evidence may authorize it (R2).

Why zero check runs is rejected outright, not delegated to classifyLandingCI’s own zero-check-runs fallback. classifyLandingCI’s zero-check-runs branch falls back to mergeable_state alone (ADR-933’s “Actions disabled” case) — correct for its usual caller, pollForMergeable, which judges a landing PR whose tree a just-polled trial CI run already validated, so zero check runs there is a benign GitHub quirk, not an absence of validation. This fast path never builds a trial, so that precondition doesn’t hold: zero check runs on the member’s own PR head means nothing has actually validated that tree, not “green by absence.” singletonFastPathEligible therefore rejects on zero check runs before classifyLandingCI is ever consulted — a fast-path-specific precondition layered in front of the reused classifier, not a fork of it.

Why this is a loop-level guard, not a change to assembleAndValidate. assembleAndValidate has four call sites: the main re-form loop (the only genuinely-fresh Queued-batch decision), bisect’s poisoner-isolation sub-trials (a member’s own-PR CI passing says nothing about whether it’s the poisoner in an already-red combination — fast-pathing here would be a correctness bug, not just scope creep), landOneAtATime’s one-at-a-time fallback (explicitly out of scope — landSingleton’s existing fallback role is unchanged), and landGreenBatch’s main-moved rebase loop (D5 recovery of an already-green trial after base moved — a different problem from “should a trial be built at all”). Placing the guard inside assembleAndValidate itself would silently reach all four; placing it in the loop, immediately before the call, reaches only the one call site this feature targets. Applied on every loop iteration, not just the first, so a batch that bisects down to a single clean survivor and re-forms is also eligible.

A member that is merely behind is caught up, not tried (#2044). When check 2 fails only because the pinned base is ahead of the member’s head, the worker does not go straight to a trial: trySingletonCatchUp (§6.30) first merges the pinned base into the member’s own branch, waits for its own CI and re-enters this same eligibility check, unmodified, on the new head. Every other ineligibility, and singleton_catch_up: off, still builds the trial exactly as before.

Pending review-finding ejects (#1208) are applied before the fast path is even considered. applyPendingReviewEjects (§6.16) existed at three checkpoints before this feature, all downstream of a call to assembleAndValidate — the re-form loop’s own Hook 2, landOneAtATime’s per-singleton loop, landGreenBatch’s main-moved rebase loop. The fast path never calls assembleAndValidate, so without a fourth checkpoint specific to it, a member flagged mid-trial for unresolved review-thread findings could be landed directly, never reaching any of the three existing checkpoints. The guard therefore applies applyPendingReviewEjects to the length-1 current slice before calling trySingletonFastPath: an ejected member leaves current empty, and the loop continues into the top-of-loop zero-survivors return; an unflagged member falls through to the fast path exactly as before. (Found in review — the original implementation had no such checkpoint.)

Landing shape: advanceConvergedPRToDone, not landSingleton. landSingleton (the bisection fallback’s per-member landing helper) is built entirely around minting a dedicated landing PR off a validated trial branch — a shape this feature has no trial branch to produce. Instead, finishSingletonFastPathLanding follows the ordinary auto-merge path’s own Done-transition template: MergePRAtHeadSHA(owner, repo, m.prNum, m.headSHA) merges the member’s own PR directly, pinned to the exact head SHA singletonFastPathEligible validated (see below), recordAdvanceOutcome advances Queued → Done (with the same durable failure-retry/escalation every terminal advance gets, §6.17), closeIssueIfNonDefaultBase runs unconditionally on the confirmed merge (see below), and — only once the advance itself succeeded — awaitingLandingVerificationLabel is applied with no fabrik:credited-pr:<N> label, since the merged PR is the item’s own linked PR and is durably rediscoverable via the ordinary FetchLinkedPR fallback (§6.19’s table, row 1) — unlike merge-train’s other two landing paths, whose credited PR is never the member’s own. resetEjectionCount/resetTrialCounter still run, exactly as every other successful land performs.

The landed comment, and why its shape differs from the other two paths. Every merge-train landing path posts a 🏭 **Fabrik merge-train** — Landed … comment on the member’s own PR — the sole cross-landing-path, member-scoped record of which PR actually landed the change (#1275). finishSingletonFastPathLanding posts Landed via singleton fast path PR #<n>., and here <n> is the member’s own PR number, because that PR is the merge vehicle — there is no distinct integration or singleton PR to name, by construction. Under landMergeTrainBatch (Landed via batch PR #<n>.) and landSingleton (Landed one-at-a-time via singleton PR #<n>.) the cited PR is always distinct from the member’s, and a comment citing the member’s own PR there would indicate a direct merge that bypassed the train. Anything reading these comments must therefore key off the path named in the text, not off distinctness alone (tests/e2e/mergetrain_helpers.go’s landedPRPattern/waitForLandingPRDetail do exactly this).

Unlike the other two paths, this one is guarded, rather than posted unconditionally. landMergeTrainBatch/landSingleton close the member’s PR immediately after commenting, so they can never be re-entered for the same member; this path leaves the PR to GitHub’s own merge-close and can legitimately be re-entered while the member is still Queued (the pr.Merged restart branch above, and a retry after a failed recordAdvanceOutcome). The guard is two-part, and both halves are load-bearing. The comment posts only when this call actually recorded a completed landing — the same condition that applies fabrik:awaiting-landing-verification, i.e. a successful recordAdvanceOutcome — and only when that label is absent from the item’s own snapshot, meaning no earlier call already did. The label’s absence alone is insufficient: it is applied only after the advance succeeds, so on a failed advance (a missing board Status option, say — the case fabrik:awaiting-advance exists to retry, §6.17) the member stays Queued with its PR already merged and re-enters every poll with the guard still open, posting one duplicate per poll until MaxRetries escalates to fabrik:paused. Tying the comment to the successful-label condition closes that window without suppressing it permanently: no advance, no comment, and the retry that eventually advances posts exactly one. The landed half alone is equally insufficient, since the already-Done restart branch re-applies the label idempotently on every re-entry. As with the other paths, the comment gates nothing: addLandedCommentWithRetry retries transient failures and then warns and continues — a landing is never blocked or delayed on it.

Why MergePRAtHeadSHA, not the plain MergePR every other merge-train landing path uses. MergePR’s own self-gate re-reads mergeable/mergeable_state live immediately before merging, but never confirms the head is still the exact commit some earlier, separate check validated — immaterial for landSingleton/landMergeTrainBatch/the ordinary auto-merge path, since all three merge a Fabrik-owned trial, integration, or landing branch nobody else pushes to. This fast path is the first merge-train landing path to merge the member’s own, externally-writable PR branch directly after a separate, earlier CI/ancestry classification — so a commit landing on that branch between the classification and the merge call would otherwise be merged having never been validated at all. MergePRAtHeadSHA closes this by passing the expected head SHA through to GitHub’s merge endpoint’s own sha field: a request whose live head no longer matches is refused (gh.ErrConflict, from GitHub’s 409) rather than silently merging whatever is actually there. A refusal is handled identically to any other merge failure at this call site (see the state-transition table below) — the member stays in Queued for a fresh, correctly re-evaluated attempt next poll.

closeIssueIfNonDefaultBase is required here for the first time in merge-train. Both existing merge-train landing paths (landMergeTrainBatch, landSingleton) always call CloseIssue explicitly regardless of base, so neither ever needed this guard. This fast path relies on the merged PR’s own Closes #N (R5’s stated rationale for landing the member’s own PR directly) — but GitHub only auto-honours that keyword into the repository’s default branch (#1096). A base:<branch>-labeled Queued member would otherwise land but never close; closeIssueIfNonDefaultBase is the same best-effort, fabrik:awaiting-close-backed guard (§6.13) the ordinary auto-merge path already uses.

Restart safety is close to free. MergePRAtHeadSHA shares MergePR’s idempotent-on-already-merged behavior (returns immediately once the PR reports merged). trySingletonFastPath checks pr.Merged before running eligibility at all: if a prior run merged the member’s own PR but died before the Done transition, the next poll’s fast-path check resumes straight to finishSingletonFastPathLanding without re-evaluating eligibility (which could plausibly now read differently against an already-merged PR). finishSingletonFastPathLanding itself mirrors landMergeTrainBatch’s “already Done” restart branch: when m.item.Status == "Done", it still re-applies closeIssueIfNonDefaultBase and the landing-verification label, both idempotent.

No interaction with the runaway guard or trial counter. e.recordTrial (the wrapper around assembleAndValidate that feeds isRunawayTripped) only fires for non-green results and is never invoked by the fast path, since the fast path never calls assembleAndValidate at all. A landed fast path is unambiguous progress, identical in kind to any other green land.

Contrast with batch admission (§6.25). The fast path is fail-closed: it skips validation, so only positive evidence of green authorises it. §6.25’s admission gate at fresh batch formation is the deliberate inverse, fail-open: it removes a member from a batch, so only positive evidence of red (classifyLandingCI → TrainCIRed, reused unmodified) acts, and every ambiguity admits.

Logging (R4). Every disposition — eligible-and-landed, or the specific unmet condition that forced a trial — is logged from trySingletonFastPath/singletonFastPathEligible, so “why didn’t it fast-path?” is answerable from logs without inference. On a completed landing finishSingletonFastPathLanding additionally emits the repo-scoped landing complete for <trainKey> (singleton fast path, PR #<n>, 1 member) line that landMergeTrainBatch emits for a batch — the train’s terminal progress signal on its TUI job row, and the only log-visible statement that a train actually reached a landing. It is repo-scoped (logfRepo), not per-issue: a per-issue line would not reach the train’s job row. Unlike the landed comment above it is emitted unconditionally rather than guarded, since a repeat on re-entry is accurate and is itself the signal that a re-entry happened.

Code path: trySingletonFastPath, singletonFastPathEligible, finishSingletonFastPathLanding (engine/merge_train.go), called from runMergeTrainWorker’s re-form loop. See §6.19’s table for how this path’s credited PR is discovered post-Done.

State transitions:

Before Trigger After Labels Added Labels Removed
Queued, single member, eligibility unresolved trySingletonFastPath: any condition not positively confirmed (TOCTOU mismatch, ancestry not confirmed, not mergeable, CI not green/complete, or an API error) Queued (unchanged) — falls through to the ordinary trial path — —
Queued, single member, pinned base ahead of the head (behind) trySingletonFastPath declines; trySingletonCatchUp (§6.30) Catch-up merge pushed, then — once own CI is green — the fast path re-runs on the new head; otherwise deferred or the trial (§6.30) — —
Queued, single member, all conditions positively confirmed MergePRAtHeadSHA succeeds; recordAdvanceOutcome succeeds Done, awaiting landing verification fabrik:awaiting-landing-verification —
Queued, single member, all conditions positively confirmed MergePRAtHeadSHA succeeds; recordAdvanceOutcome fails Queued (unchanged); advance failure durably retried (§6.17) — —
Queued, single member, all conditions positively confirmed MergePRAtHeadSHA fails (transient, or the pinned head SHA no longer matches — gh.ErrConflict) Queued (unchanged); retried next poll — —
Queued, single member, PR already merged (restart) trySingletonFastPath observes pr.Merged == true and the member’s live Status is still the holding stage (§6.28) Resumes straight to the Done transition, as above (as above) —
Snapshot-Queued single member, PR already merged, live Status already Done pr.Merged == true, liveLandingState → moved (§6.28, #1871) Disposition decided; nothing re-landed, no trial built — —
Queued, single member, pending review-finding eject flagged (#1208) or pending unprocessed-comment eject flagged (#1863) applyPendingReviewEjects, applied before the fast path is considered Rerouted off Queued (§6.16) — never reaches trySingletonFastPath — —

References: ADR-1644: Merge-Train Singleton Fast Path, ADR-1153: Train CI Completeness over Mergeable State, ADR-1441: Unstable Requires Check-Run Classification, ADR-072: MergePR Self-Gates on Mergeable State, ADR-1208: Queued Review-Finding Ejection, §6.13 (closeIssueIfNonDefaultBase), §6.16 (applyPendingReviewEjects), §6.19 (post-Done landing verification), issue #1644


6.21 Non-Default-Base Linkage Notice (#1649)

Trigger: GitHub’s Closes #N inertness on a non-default base branch (§2.8, ADR-1096) goes further than suppressing auto-close: per GitHub’s own docs, closing keywords “are interpreted only when the pull request targets the repository’s default branch… otherwise these keywords are ignored, no links are created” — so a base:<branch> item’s issue shows no PR at all in its Development panel. Fabrik itself is unaffected (FetchLinkedPR resolves by the fabrik/issue-N head-branch convention, never GitHub’s linkage graph), but the operator looking at the board has no visual cue and has to go hunting. Reported as the “adjacent, lower priority” half of #1646; ADR-1096/1097 already cover the closing half — this covers the purely human-visible half.

Durable marker (fabrik:nondefault-base-pr-noted), written unconditionally after the post attempt. Unlike fabrik:awaiting-close (§6.13), this label is not written only in a failure branch — it’s added right after postItemComment is called, whether or not that call actually succeeded. There is no durable outstanding action to distinguish a “written on failure, retry later” state from: a missed comment (transient AddComment error) is simply gone, not queued. This mirrors fabrik:api-key-helper-detected’s accepted trade-off (CLAUDE.md), not ADR-1097’s retry/escalation shape.

Why no retry/escalation machinery, deliberately. fabrik:awaiting-close’s retry loop exists because a stalled explicit close has a real downstream cost — dependent issues that unblock on close (not merge) stay blocked indefinitely. A missed one-time informational comment has no such cost: nothing gates on its delivery, no dependent issue waits on it, and the underlying fact it reports (this PR has no GitHub linkage) remains discoverable by other means (the issue’s base:<branch> label itself, or a cross-referenced timeline event GitHub still creates). Adding recordSettleRetry/escalateSettle here would be retry machinery in search of a problem it doesn’t have. R2’s “must survive restarts and repeated polls without duplicating” is satisfied entirely by the label’s own presence — it says nothing about guaranteeing eventual delivery.

Code path: noteNonDefaultBasePRLinkage (engine/nondefault_base_linkage_notice.go), reusing itemHasBaseLabel/baseBranchForItem (§6.13’s closeIssueIfNonDefaultBase precedent) for the base-vs-default comparison. Retry-owner (detection only, not delivery): settleNonDefaultBaseLinkageNotice, called from poll() in engine/poll.go, immediately after settleNonDefaultBaseCloses.

settleNonDefaultBaseLinkageNotice flow, run unconditionally every poll over raw board.Items (ADR-1270 family, ninth-or-later instance — ordinal not load-bearing), independent of itemMayNeedWork/itemNeedsWork dispatch and of the item’s board column (eligibility is “has a base: label and a discoverable PR,” not “reached a particular stage or Done”):

  1. item.IsPR → skip.
  2. !itemHasBaseLabel(item) → skip (R3’s fast path: per baseBranchForItem’s documented contract, an item with no base: label always resolves to exactly the default, so the overwhelming common case never touches a WorktreeManager or does a git shell-out).
  3. item.LinkedPRNumber == 0 → skip (R1: no PR discovered yet — LinkedPRNumber is already populated on the board snapshot via the deep-fetch cache’s FetchLinkedPR, so this costs no new API call, R4).
  4. hasLabel(item.Labels, "fabrik:nondefault-base-pr-noted") → skip (R2 idempotency).
  5. Non-panicking WorktreeManager lookup (e.mu.Lock(); wm, ok := e.worktreeManagers[key]; e.mu.Unlock()), mirroring closeIssueIfNonDefaultBase — skip-and-log if unregistered (self-heals next poll once a dispatch registers it; same restart-window caveat as ADR-1096).
  6. base, err := e.baseBranchForItem(item, wm) → skip-and-log on error.
  7. def, err := wm.DefaultBaseBranch() → skip-and-log on error.
  8. base == def → skip (R3: covers both a genuinely default-base item and the case where baseBranchForItem’s own remote-fallback already reduced the effective base to default — e.g. the labeled branch doesn’t exist on the remote, in which case GitHub’s ordinary linkage applies and no notice is warranted).
  9. Otherwise: post the one-time comment naming the PR (item.LinkedPRNumber), the resolved base, and the repository default; then unconditionally add fabrik:nondefault-base-pr-noted.

A distinct comment from baseBranchForItem’s own fallback notice. Step 6 can itself post a different one-time comment (“base branch not found on remote, falling back to default”), gated by an in-memory-only sync.Map (not restart-durable, unlike this feature’s own label-gated marker). When that fallback fires, step 8’s base == def comparison correctly no-ops this feature’s notice — but the two comments are independent and must not be conflated when reading an issue’s thread or writing tests against it.

Closed issues are not excluded. Unlike closeIssueIfNonDefaultBase (which treats an already-closed issue as nothing-to-do, since closing an already-closed issue is a no-op action), this scan posts regardless of item.IsClosed — the information (“this PR was never GitHub-linked”) remains true and useful for an issue closed by any means, and posting an informational comment is not the same kind of action as an idempotent-by-necessity state mutation.

State transitions:

Before Trigger After Labels Added Labels Removed
base:<branch> item, PR discovered, base ≠ default Settle pass reaches step 9 Notice posted fabrik:nondefault-base-pr-noted —
base:<branch> item, no PR yet, or base == default (incl. remote-fallback) Settle pass reaches step 3 or 8 Unchanged — —
Default-base item (no base: label) Settle pass reaches step 2 Unchanged — —

References: ADR-1649: Non-Default-Base Linkage Notice, ADR-1096: Explicit Close on Non-Default-Base Merge / ADR-1097: Non-Default-Base Close Retry / §6.13 (the closing half of this same gap, and the structural precedent this feature’s base-resolution logic reuses verbatim), issue #1646 (the original community report), issue #1649 (this feature).

6.22 Mark-PR-Ready Durable Retry (ADR-1582)

Trigger: markPRReady (§5.2, engine/pr.go) calls e.client.MarkPRReady as the final step of transitioning a stage’s draft PR to ready-for-review, after its own in-process 3-attempt retry (#599). This call fails non-transiently, or exhausts that retry budget. handleStageComplete proceeds regardless — stage:<name>:complete is granted whether or not the PR ever became ready — and, before this feature, nothing ever retried the call again: the PR could remain draft forever, which attemptMergeOnValidate’s direct-merge fallback treats as unconditionally not-CI-clean (mergeable_state == "draft", checked ahead of any check-run/dirty logic), permanently blocking the issue from reaching Done via that path.

Key difference from §6.13’s fabrik:awaiting-close: structurally the same single-at-risk-call shape, but this marker is not terminal-only. mark_pr_ready_on_complete: true can be configured on a non-terminal stage (e.g. Implement, with Review and Validate still ahead), so the marker can be written while the item is still mid-pipeline — and it is deliberately left free to keep advancing through subsequent stages while the marker is outstanding, since nothing about a later stage’s dispatch depends on the PR being ready.

Durable marker (fabrik:awaiting-pr-ready), written only on failure, inline inside markPRReady. markPRReadyOutstanding adds the label (idempotently — a no-op if already present) in exactly the two places markPRReady previously just logged and returned: the non-transient-error branch, and the retry-exhausted fallthrough at the end of its loop. No call-site changes were needed at any of markPRReady’s three callers for this marker (the later bool return, #2089, is unrelated to it — see §”Zero-ahead push guard”) (engine/item.go ×2, engine/comments.go) — every value the marker write needs (item, owner, repo, the resolved prNumber) was already in scope at both failure points. markPRReady’s success path also calls clearPRReadyMarker when the marker happens to already be present, so an issue previously escalated and un-paused self-heals the moment its own next markPRReady call succeeds directly, without waiting for the settle scan below to notice.

No dispatch-suppression wiring — deliberately, and unlike most of this family. This marker is not checked by itemMayNeedWork/itemNeedsWork, and is not added to transientLifecycleLabels. Unlike fabrik:awaiting-close (§6.13) or fabrik:awaiting-member-close (§6.10), which only ever exist after the item has already reached Done, this marker can be present on an item still actively progressing through later stages — gating stage completion on MarkPRReady succeeding was explicitly out of scope for #1582 (a bigger change to the conjunctive-gate model than the bug called for).

Code path: markPRReady (engine/pr.go, writes the marker on failure, clears it on a direct success) — and, on retry, settlePRReadyScan → settlePRReady (both engine/pr_ready_settle.go; settlePRReadyScan is called from poll() in poll.go) directly.

Retry-owner: settlePRReadyScan (engine/pr_ready_settle.go; called from poll() in poll.go). Runs unconditionally once per poll. It iterates the raw board.Items (not deepFetchCandidates — this scan has no dependency on the deep-fetch/terminal-skip machinery at all). For every item carrying fabrik:awaiting-pr-ready and not fabrik:paused (mirroring every other settle scan’s own paused-item guard), it calls settlePRReady.

settlePRReady flow — idempotent; safe to call repeatedly. Re-resolves the PR live every pass (via FetchLinkedPR) rather than trusting a stashed PR number — a GitHub label carries no payload, and this scan runs on a completely independent poll cadence from when the marker was written:

  1. No linked PR found at all → clear the marker (nothing left to mark ready — the PR was deleted, or the marker survived a scenario that no longer applies).
  2. PR closed or merged → clear the marker (the issue was abandoned, or it landed via a path that never called markPRReady).
  3. PR found and already not-draft → clear the marker without calling MarkPRReady again (self-healed by a later stage’s own markPRReady call, or a human clicking “Ready for review” manually).
  4. Otherwise (genuinely still draft and open): call e.client.MarkPRReady again — safe regardless of how many times this fires, since GitHub’s markPullRequestReadyForReview mutation is a documented no-op success on an already-ready PR. On success, clear the marker. On failure, record a retry and stop — the next settle pass re-attempts.
  5. A FetchLinkedPR API error (distinct from “no PR found”) retries without clearing the marker — the scan cannot tell whether the PR is still draft, so it must not assume either outcome.

Retry counting and escalation. Every failed settle pass calls recordPRReadyRetry, which increments the existing itemstate.StageRetryIncremented/Attempts counter — keyed by the dedicated constant "__awaiting_pr_ready__" (same double-underscore-wrapped, YAML-unrepresentable shape as its siblings, so it can never collide with a configured stage’s own counter). MaxRetries <= 0 means unlimited retries, never escalate (same guard as every sibling counter). Once Attempts("__awaiting_pr_ready__") >= e.cfg.MaxRetries, escalatePRReadyFailure fires — mirroring escalateNonDefaultBaseCloseFailure: adds fabrik:paused, removes fabrik:awaiting-pr-ready, posts an explanatory comment naming the draft PR (item.LinkedPRNumber, when non-zero) with the manual recovery step (gh pr ready <N>), and applies itemstate.EnginePaused.

Marker clearing. fabrik:awaiting-pr-ready is removed in three places: clearPRReadyMarker via a fully successful settle pass (PR found not-draft, closed, merged, or not found at all — the normal, expected path), markPRReady’s own success path (a direct self-heal, without waiting for the settle scan), and escalatePRReadyFailure (giving up after MaxRetries).

Scope: this mechanism covers only markPRReady’s own client.MarkPRReady call (engine/pr.go, called from engine/item.go and engine/comments.go for any stage configured with mark_pr_ready_on_complete: true). The merge-train’s own inline MarkPRReady call (engine/merge_train.go, promoting a reused draft integration PR to ready before landing) is out of scope — a failure there already leaves the affected members in Queued and is retried naturally on the worker’s own next iteration. The generic recordSettleRetry/escalateSettle/clearSettleMarker helpers (engine/settle.go) this mechanism reuses are the same ones backing fabrik:awaiting-done (§6.8), fabrik:awaiting-member-close (§6.10), and fabrik:awaiting-close (§6.13).

Interaction with attemptMergeOnValidate: no change needed. That function already retries indefinitely on every poll while un-gated. Once settlePRReadyScan flips the PR out of draft, the very next poll’s existing merge attempt sees a non-draft mergeable_state and proceeds normally — fixing retryability at the source is sufficient to unblock the direct-merge fallback described in this section’s Trigger.

State transitions:

Before Trigger After Labels Added Labels Removed
Any stage, mark_pr_ready_on_complete: true markPRReady’s client.MarkPRReady call fails (non-transient, or retry-exhausted) Same stage, awaiting pr-ready (marker present); item continues advancing normally fabrik:awaiting-pr-ready —
Awaiting pr-ready Settle pass succeeds (PR marked ready, or found already not-draft/closed/merged/missing) Settled — fabrik:awaiting-pr-ready
Awaiting pr-ready A later markPRReady call succeeds directly (self-heal) Settled — fabrik:awaiting-pr-ready
Awaiting pr-ready Settle pass fails Attempts >= MaxRetries times Paused fabrik:paused fabrik:awaiting-pr-ready

References: ADR-1582: Durable Mark-PR-Ready Retry, ADR-1097: Non-Default-Base Explicit Close Retry (§6.13, the closest structural precedent), ADR-061: Merge-Train Singleton Member-Issue Close Retry, issue #599 (the in-process retry this feature’s durable follow-up builds on), issue #1582 (this feature).

6.23 Autonomy-Label Live Re-Read at Both Advance Decisions (#1769)

Trigger: An operator adds or removes fabrik:yolo/fabrik:cruise on an issue while a stage is in flight, intending to change the advance/auto-merge decision the engine is about to make for that issue.

The problem this section fixes. Two independent advance paths read hasYoloLabel(item)/hasCruiseLabel(item) off item.Labels at the moment they decide whether to advance the stage (and, at Validate, whether to enable auto-merge):

Before this fix, Path 1 attempted a mid-run label re-fetch but targeted e.cfg.Owner/e.cfg.Repo — empty in multi-repo mode (a 404, silently swallowed by an err == nil && guard) and, on a single-repo instance whose config repo differs from the item’s own, the wrong issue’s labels entirely. Path 2 performed no re-fetch at all: item.Labels is not part of the deep-fetch cache contract (copyDeepFieldsFromState excludes Labels), so even an item that went through a “fresh” deep fetch earlier in the same poll pass could still carry stale autonomy labels at the point runCatchUpPhase2 reads them. Both defects meant a label removal made mid-run was never observed by either path — the stage would complete and auto-advance (or auto-merge) under authority the operator had already withdrawn, only picking up the change on the next full board refresh.

The fix: one shared helper, consulted by both paths. refreshAutonomyLabels(item *gh.ProjectItem) (engine/stages.go) performs a live FetchLabels call scoped to the item’s own resolved owner, repo (itemOwnerRepo, never e.cfg.Owner/e.cfg.Repo) via e.client (the raw, uncached GitHubClient — never e.readClient, since Labels is not part of that cache’s deep-field contract regardless), and mutates item.Labels in place. Both handleStageComplete and runCatchUpPhase2 call it as the very first step of their autonomy-decision logic, before any hasYoloLabel/hasCruiseLabel read — mirroring effectiveReviewAuthority/effectiveExpectedReviewers’s “one shared helper, every gate” precedent (ADR-1250, ADR-1283, engine/reviews.go), except — unlike those two, which only resolve a label already present on the snapshot — this helper performs the fetch itself.

On a fetch error, item.Labels is left untouched and a warning naming the item and the error is logged — silent fallback is exactly what hid this defect, and does not survive the fix. On a successful fetch, item.Labels is replaced unconditionally, including when the fresh result is an empty label set: a real fetch returning zero labels is data, not a failure, and must not be discarded the way the old len(freshLabels) > 0 guard discarded it.

One fetch covers every downstream autonomy-label read. attemptMergeOnValidate (the single landing-decision function both paths call — ADR-1216) receives item by value and performs its own hasCruiseLabel(item) check. Because both callers invoke refreshAutonomyLabels upstream of every attemptMergeOnValidate call site, that check is fresh too, without a second REST call from this fix — bounding the autonomy-label fetch cost to exactly one live FetchLabels per handleStageComplete/runCatchUpPhase2 invocation (i.e. at most once per item per poll, since the two paths are mutually exclusive admission for the same item in the same pass). This is narrower than “one fetch total,” though: attemptMergeOnValidate already performs its own live re-read for an unrelated purpose — the pre-existing ADR-1419 FetchItemDetails dependency guard (for BlockedBy) — and that call’s GraphQL query resets item.Labels as a side effect, overwriting the REST-fresh snapshot refreshAutonomyLabels just produced. On the Validate/yolo merge path, item.Labels is genuinely fetched twice per invocation, not once. This is harmless today (the dependency guard runs after both the hasCruiseLabel check and the fabrik:auto-merge-enabled idempotency check, and nothing later in the function reads hasYoloLabel/hasCruiseLabel), but a future label check added after the dependency guard must not assume it is reading the REST-fresh snapshot.

Code path: refreshAutonomyLabels (engine/stages.go); called from handleStageComplete (engine/stages.go) and runCatchUpPhase2 (engine/poll.go).

Precedent: this is the same class of fix as checkDependencies’s live BlockedBy re-read (ADR-1419) and reviewGateBlocksLanding’s live review-state re-read (ADR-1216) — a decision with real consequences (advance, or at Validate, merge) must not be made on a cached snapshot that a concurrent operator action can invalidate.

References: ADR-1769: Live Re-Read of Autonomy Labels at Both Advance Decision Points, ADR-1419: Cross-Repo Spawn Servability and Mid-flight Recognition, ADR-1216: Review Gate at Landing Decision, ADR-1250: Review Authority Orthogonal to Autonomy, ADR-1283: Declared Unrequested Reviewers, issue #1768 (originating report), issue #1769 (this fix).


6.24 Merge-Train Base Sanity Check (#1773)

Trigger: In #1688, the merge train opened an integration PR against a protected main while every member of the batch carried base:develop, and it self-merged nine minutes later with no human in the loop — a production incident. #1772 fixes the specific cause (an unhydrated member silently partitioning to the default base at grouping time). This is a deliberately independent second line of defence: a check on the outcome the train is about to act on, immediately before it acts, rather than another cause-specific fix — this is already the second time this class of defect (ADR-1647/ADR-1648 fixed the first, #1637/#1646) has reached production, so a check that catches the outcome regardless of cause is warranted.

Mechanism: fires only on the default partition, reads live, refuses rather than repairs. refuseIfBaseContradictsMembers(owner, repo, baseBranch, trainKey string, members []trainMember) bool (engine/merge_train.go) is called immediately before either of the two landing paths that can mint a new integration/landing PR:

  1. isDefaultPartitionKey(owner, repo, trainKey) — trainKey == owner+"/"+repo — decides whether the pinned base is the repository default. This is exactly mergeTrainKey’s own default-sentinel condition (§ above, ADR-1648), not an approximation of it, so it needs no additional git call and costs nothing on a base:<branch>-partitioned train (R5): the check returns immediately, false, without touching e.client at all.
  2. Only when triggered, each member’s labels are read live via e.client.FetchLabels — never e.readClient (boardcache.CacheImpl.FetchLabels serves the cached snapshot whenever the cache isn’t paused) and never the member’s own cached trainMember.item.Labels. This is the entire point of the check (R2): the #1688 incident shape is exactly a cached/stale label disagreeing with a live one, so a check reusing either cached path would have passed cleanly in that incident.
  3. nonDefaultBaseLabelValue(labels []string, pinnedBase string) string scans for a base:<branch> label whose value is non-empty and differs from pinnedBase. It deliberately does not resolve remote branch existence or fall back to a default the way baseBranchForItem (§6.13/§6.21) does — R3 forbids repairing a contradiction found here, only refusing.
  4. Any contradiction, or any per-member FetchLabels error (treated identically — fail-closed, matching #1772’s own posture), causes the caller to refuse: no PR is opened or reused, the member(s) stay in Queued, and e.logfRepo names the pinned base and every contradicting member with its declared base (R4). A healthy train — including the ordinary all-default-base case — produces no new log output at all.

Both CreatePR sites that can mint a new PR are covered; the singleton fast path is exempt by construction. landMergeTrainBatch calls the check immediately after the closed-unmerged-trial escalation and before its if integrationPR != nil { reuse } else { create } split — covering both the PR-open and PR-reuse branches uniformly, since a reused PR (a promoted draft CI PR) is just as capable of carrying a stale pinned base as a freshly-created one. The check is skipped, however, when the found integrationPR is already merged (integrationPR != nil && integrationPR.Merged — the FR-2 restart-safety state, e.g. an engine restart between merging the integration PR and advancing its members to Done): nothing is being opened or reused in that state, so refusing there would only strand an already-landed batch in Queued forever on a transient FetchLabels error or a label edited after the merge, blocking FR-3’s advance-to-Done/close-PR logic for no protective benefit — found during PR review and fixed in the same PR (TestLandMergeTrainBatch_AlreadyMergedPR_SkipsBaseCheck). landSingleton (the FR-5 one-at-a-time fallback, not a rare corner case — any bisection failure routes through it) calls the check immediately before its own CreatePR call; it never searches for or reuses an existing PR at all (by design — see the ADR-059 D4 data-loss note in its own doc comment), so it has no equivalent already-merged state to guard against. The singleton fast path (trySingletonFastPath/finishSingletonFastPathLanding, §6.20) never calls CreatePR at all — it merges the member’s own existing PR via MergePRAtHeadSHA, targeting whatever base that PR was already opened against — so it structurally cannot reproduce this contradiction and needs no equivalent check.

Self-healing, not sticky. The check carries no state of its own across polls — a refusal simply returns without opening or reusing a PR, leaving the batch’s members in Queued exactly as they were. The next poll re-forms the batch and re-runs the check from scratch; once the underlying label disagreement resolves (by any means — a human fixing a stale label, a cache entry expiring, or #1772 preventing the bad partition in the first place), the very next attempt proceeds normally.

No new label, no escalation machinery. Unlike most of this file’s other guard mechanisms, a refusal here is not durably marked on the board — an operator distinguishes it from ordinary merge-train activity by the merge-train-tagged log line alone. This is a deliberate, narrower scope than the pattern most fabrik:awaiting-* siblings follow (§6.9–§6.22): the acceptance criteria for this feature are fully satisfiable with logging and self-healing alone, and building a durable repeat-counter + fabrik:paused escalation now would be speculative state for a scope not yet shown to need it (see ADR-1773’s Rejected Alternatives).

Independent of #1772, deliberately. #1772 fixes the partition-time cause of a member ending up in the wrong batch; this check fires at landing time, downstream of and structurally unrelated to that fix, sharing no code. With #1772 also applied, a member it excludes at partition time never reaches this check at all (there is nothing left to contradict); this check still fires for any other cause of the same pinned-base-vs-declared-base disagreement, including causes not yet discovered — which is the entire reason this is a second, independent line of defence rather than a refactor of the first.

Code path: refuseIfBaseContradictsMembers, isDefaultPartitionKey, nonDefaultBaseLabelValue (engine/merge_train.go), called from landMergeTrainBatch and landSingleton.

State transitions:

Before Trigger After Labels Added Labels Removed
Queued batch, pinned base == repo default, all members agree (or genuinely default-base) refuseIfBaseContradictsMembers returns false Landing proceeds normally (open or reuse the integration/landing PR) — —
Queued batch, pinned base == a non-default partition base isDefaultPartitionKey is false — check never fires Landing proceeds normally, no FetchLabels calls made — —
Queued batch, pinned base == repo default, ≥1 member’s live label declares a different base refuseIfBaseContradictsMembers returns true No PR opened or reused; members remain in Queued; loud log line naming base + members — —
Queued batch, pinned base == repo default, a member’s live FetchLabels call errors refuseIfBaseContradictsMembers returns true (fail-closed) No PR opened or reused; members remain in Queued; loud log line naming base + members — —
landMergeTrainBatch found an existing integration PR that is already merged (FR-2 restart-safety state) integrationPR != nil && integrationPR.Merged — the check is skipped entirely, regardless of member labels FR-3 (advance to Done, close member PR/issue) proceeds normally; no FetchLabels calls made, no refusal possible — —

References: ADR-1773: Merge-Train Base Sanity Check at Landing Time, ADR-1647: Merge-Train Non-Default-Base Exclusion, ADR-1648: Merge-Train Batches Partitioned Per (Repo, Base), §6.13 (baseBranchForItem’s remote-resolution semantics, deliberately not reused here), §6.20 (the singleton fast path, confirmed exempt), issue #1688 (the community report that motivated this issue), issue #1773 (this feature).


6.25 Merge-Train Own-PR CI Admission Gate (ADR-1821)

Trigger: Issue #1821. fetchTrainMembers validated exactly two things per Queued member — a linked PR exists and has a head SHA — and never consulted the member’s CI. A member whose own PR checks were already red was therefore admitted, merged into a trial, and only discovered by that trial’s CI (or by bisection): on verveguy/concept-maps two such members cost roughly 2 hours each (an ~54-minute assembly, a 24-minute CI cycle, then re-bisection) to rediscover a fact that was readable off the members’ own check-runs in one API call. classifyLandingCI already answers the question and is already trusted for a landing decision (§6.20 step 4), but was only applied on the singleton fast path, never at admission.

Mechanism: a fail-open admission gate at fresh batch formation. admitTrainMembers (engine/merge_train_admission.go) is called from prepareTrainWorker immediately after fetchTrainMembers. For each member it reads e.client.FetchCheckRuns(owner, repo, m.headSHA) — the live REST client, at the head SHA fetchTrainMembers snapshotted, so the evidence corresponds to the exact commit that would enter the trial — and classifies it with classifyLandingCI, unmodified. The gate has no notion of red of its own: “confirmed red” is exactly TrainCIRed, which includes a required-context failure.

Polarity is fail-open — the inverse of §6.20. The singleton fast path skips validation, so it authorises only on positive evidence and every ambiguity blocks it. The gate removes a member from a batch, so it acts only on positive evidence of failure and every ambiguity admits:

classifyLandingCI / read outcome Gate action
TrainCIRed (failed check run, or required-context failure) Defer (below)
TrainCIPending (incl. a rerun in progress masking an older failure) Admit
TrainCIGreen Admit
Zero check runs, no required-context failure Admit (the fast path’s fail-closed zero-run rejection is deliberately not copied)
FetchCheckRuns error Admit, logged as fetch-error
Context cancelled mid-gate Admit the remainder unchecked

Admitting is byte-identical to pre-#1821 behavior. Deferring on ambiguity would silently drain the queue and strand members that have done nothing wrong; the cost of the fail-open choice is that a red member with a pending run, a zero-run head, or an unreadable read is discovered by the trial exactly as it always was.

Scope: fresh formation only (R7). The gate lives in prepareTrainWorker, after reconstructTrainState has returned true for both restart routes (completeDeferredLanding of an already-merged marker PR, and resumeTrain of an existing in-flight trial), and it is not inside fetchTrainMembers itself. Those routes re-resolve an already-decided batch and must not re-litigate admission. Bisection sub-trials, landOneAtATime, and landGreenBatch never pass through it either. The gate needs state.projectID for the reroute, which fetchTrainMembers does not have.

Precondition: the reroute target must have wait_for_ci (R9). A deferred member is rerouted to stageBeforeHolding (normally Validate) with its stage:<X>:complete label intact. What re-picks it up is Phase 1’s handleMergeAndCIGates: settlePRMergeState classifies red checks as PRMergeBlocked, checkCIGate returns ciFailure, fabrik:awaiting-ci is applied and dispatchCIFixReinvoke runs (bounded by MaxCiFixCycles, with the no-op-SHA debounce). Phase 2 — the only place attemptMergeOnValidate → advanceToQueued runs — is never reached for a claimed item, so a red member is not advanced straight back to Queued, even under fabrik:yolo. But checkCIGate is a no-op unless the stage has wait_for_ci: true; without it nothing would claim the member (under yolo it would bounce back to Queued and be re-deferred every poll; under cruise it would strand). The gate therefore checks the target stage first and, if it lacks wait_for_ci, admits every member without reading anything, logging once per formation. The shipped and live validate.yaml both set wait_for_ci: true; a custom config that omits it simply gets no gate.

Deferral (deferRedMember). Modelled on ejectRedSingleton (#1545) minus the pause:

  1. The member is dropped from the batch unconditionally — it is confirmed red.
  2. rerouteQueuedMemberOffHolding runs first (the ADR-1208 reroute-before-side-effects primitive). On failure nothing is posted, consumed, or recorded; the next poll’s fresh formation retries the whole operation. The member stays out of the current batch regardless.
  3. On success, any pending review-finding eject signal for the member (takePendingReviewEject, §6.16) — and any pending unprocessed-comment eject signal (takePendingCommentEject, #1863) — is consumed and logged, so a member flagged both ways is never double-routed or double-commented and a stale signal cannot later fire on a re-queued member that already addressed the finding.
  4. One comment is posted: header 🏭 **Fabrik merge-train — deferred (own CI failing)**, the failing check(s) via renderFailedChecks (falling back to the classifier’s detail for a required-context red, which has no failing runs to render), a statement that the failure is the PR’s own and not a train interaction, and that the member has not been paused — the CI gate will dispatch a fix from the target stage and the issue re-queues once CI is green and that stage completes again. It is composed locally: ejectMember(stayInQueue=false) hard-codes review-finding wording, and renderDiagnosticBlock / renderBatchContext / reentryInstruction carry trial/integration-PR wording and a pause-assuming recovery instruction that are wrong for this cause.

Never paused, never counted (R5). A deferral does not touch mergeTrainEjectionCounts and never calls pauseMergeTrainMember, so it cannot accumulate toward MaxMergeTrainEjections. This is the opposite of ejectRedSingleton’s reasoning: there the failure was only ever observed on a synthetic trial branch, so nothing external would re-detect it and a pause was the only human gate. Here the failure is observable on the member’s own PR check-runs, so the ordinary CI gate has an external signal to act on.

Ping-pong backstop (R9). mergeTrainCIDeferred (owner/repo#N → head SHA, in memory, mergeTrainCIDeferredMu) records the SHA each member was last deferred at. A repeat deferral of the same SHA — reachable only if the live read shows red while Phase 1’s cache-backed read does not — still reroutes but posts no second comment. The entry is cleared when the member is next admitted non-red. It resets on restart, costing at most one duplicate comment.

Cost. One FetchCheckRuns per member per fresh batch formation, sequentially, bounded by MaxBatchSize. A pending or timed-out trial re-forms the batch on the next poll, so the read repeats per member per poll until a landing or eject. classifyLandingCI additionally calls FetchCombinedStatus (cache-backed) only when required_status_contexts is configured and a required name is absent from the check runs. A one-member batch reads check runs again in singletonFastPathEligible. The small per-poll cost is set against ~2 hours avoided per red member. Every disposition — defer, or admit with its reason (pending, green, fetch-error, no-wait-for-ci) — is logged with the classifier’s detail.

A new sustained label state. A deferred member carries both stage:<X>:complete and fabrik:awaiting-ci for the whole CI-fix cycle (classifyCIFromCheckRuns adds the label but never removes the completion label). Both the main Phase 1 loop and settleAwaitingCIScan (ADR-1270) run the same handler chain on it; duplicate dispatch is prevented by the snap.Worker() != nil guard and duplicate pause comments by hasCIGatePauseComment. MaxCiFixCycles and the CIBackstopTimeout can still pause a slow fix — an ordinary Validate pause, not a deferral pause.

Crash window. The reroute and the comment are two calls; a crash between them leaves a rerouted member with no comment, matching ADR-1208 and ADR-1545.

Code path: admitTrainMembers, deferRedMember, composeCIDeferralBody (engine/merge_train_admission.go), called from prepareTrainWorker (engine/merge_train.go).

State transitions:

Before Trigger After Labels Added Labels Removed
Queued member, fresh batch formation, reroute target has wait_for_ci classifyLandingCI → TrainCIRed on the member’s own head; reroute succeeds Rerouted to the stage before Queued; excluded from the batch; one comment; ordinary CI gate takes over — —
(as above) TrainCIRed; reroute fails Queued (unchanged), excluded from this batch; no comment, no dedupe record; whole operation retried next poll — —
(as above) TrainCIRed again at the same head SHA (e.g. stale-cache bounce) Rerouted again; no repeat comment — —
(as above) Pending, green, zero check runs, or FetchCheckRuns error Admitted to the batch, unchanged from pre-#1821 — —
Queued member, reroute target lacks wait_for_ci Gate skipped before any read Admitted, unchanged — —
Queued member, restart reconstruction (Route 1 / resumeTrain), bisection, landOneAtATime, landGreenBatch Never reaches the gate Unchanged — —
Deferred member at the reroute target, red checks Phase 1 handleMergeAndCIGates → ciFailure CI-fix reinvoke dispatched; not advanced (Phase 2 unreached) fabrik:awaiting-ci —

References: ADR-1821: Merge-Train Own-PR CI Admission Gate, ADR-1644: Merge-Train Singleton Fast Path (the inverse fail-closed polarity), ADR-1208: Queued Review-Finding Ejection (reroute primitive), ADR-1545: Red-Singleton Reroute Before Pause, ADR-1420: Merge-Train Ejection Diagnostics, §6.16 (applyPendingReviewEjects), §6.20 (the singleton fast path), issue #1821


6.26 Merge-Train Conflict Resolution Replay via git rerere (ADR-1834)

Trigger: Issue #1834. A merge-train conflict resolution lives only on its trial branch — cleanupTrialArtifacts deletes the trial worktree and branch after every trial, including bisection sub-trials, win or lose. When a trial is abandoned (bisected, re-formed after an eject, rebuilt after the base moves), the resolution is deleted with it, and the next assembly pays a fresh Claude invocation for the byte-identical conflict. On one operator’s repo, the same file’s conflict was resolved twice in 80 minutes across two trials for the same members.

Mechanism: git rerere enabled repo-wide, replay detected via the merge’s own output. enableRerere (engine/worktree.go) sets rerere.enabled/rerere.autoupdate on the bare clone, called from ensureBareClone alongside the existing setCommitterIdentity — every worktree derived from that bare clone (regular per-issue worktrees and merge-train trial worktrees alike) shares the enablement and the rr-cache it writes to. For a bare repo, bareDir is $GIT_DIR, so <bareDir>/rr-cache is already scoped correctly and survives CleanupTrainWorktree’s git worktree remove/git branch -D untouched — no new cache-location plumbing was needed.

Scoping is repo-wide, not worktree-scoped — a correction made during implementation. The originally planned narrower approach — enabling rerere only for merge-train trial worktrees via Git’s per-worktree config extension (extensions.worktreeConfig + git config --worktree) — was implemented, then reverted after direct reproduction showed it is unsafe: turning on extensions.worktreeConfig for a bare clone makes core.bare (true, a bare repo’s own top-level value) the extension’s shared/common config value, and any linked worktree that doesn’t explicitly override it inherits that value — breaking every ordinary git operation in that worktree with fatal: this operation must be run in a work tree. Since Fabrik’s every worktree for a given repo — every open issue’s worktree, not just merge-train trials — shares one bare clone (NewWorktreeManagerForRepo), this would have broken every currently-open worktree for the repo the instant the extension was first turned on, not merely the newly-created trial worktree. Repo-wide enablement carries no such risk: a rerere lookup with no recorded resolution is a pure no-op, so a regular per-issue worktree’s Validate-stage rebase conflict is, at worst, itself replayed for free if it happens to exactly match a merge-train-recorded resolution — a strict improvement, never a regression.

git merge always exits non-zero on a conflict, even when rerere fully replays it. Git’s merge machinery invokes rerere synchronously inside git merge itself: it hashes each conflict hunk and, given a recorded resolution, rewrites the file in place. With rerere.autoupdate=true, a file whose every hunk was replayed is also automatically staged — but git merge still reports a conflict and a non-zero exit regardless of how much rerere already resolved, so mergeErr != nil in assembleTrialBranch is uninformative on its own about whether anything is actually left to do.

Three-way dispatch in resolveTrainConflict, keyed on the post-merge unmerged-path set:

unmergedPaths after the failed merge Dispatch
Empty, and no declared generated path was named in the merge’s own CONFLICT (...) output commitRerereReplayedMerge finalizes directly (git add -A + git commit) — no Claude invocation
Empty, but a declared generated path was named in the merge’s CONFLICT (...) output regenerateAndCommit runs the declared regeneration command fresh — no Claude invocation, and the replayed content is never trusted (see the ADR-1235 interaction below)
Non-empty Unchanged from before this issue: classifyConflictedPaths runs against whatever remains, dispatching Claude for the non-generated remainder (if any) and/or regeneration for generated paths, exactly as ADR-1235 already specified. Claude’s own git status naturally shows only what rerere left behind — buildTrainConflictComment was never handed a precomputed path list for the plain case, so no change was needed here at all

The empty-unmergedPaths cases above are the ones this issue adds: previously, an empty result fell through to the same “no generated paths in the conflict” branch as an ordinary clean-Claude-dispatch case, invoking Claude even when there was nothing left to resolve.

The ADR-1235 interaction: rerere has no awareness that a path is a declared generated file. If a conflict confined to a declared generated path (e.g. docs/llms-full.txt) was already resolved via regenerateAndCommit once, and the identical conflict recurs, rerere.autoupdate would — left unchecked — silently replay and re-stage that old committed content before resolveTrainConflict’s unmergedPaths-based classification ever ran, so matched would come back empty and the regeneration command would never re-run, landing stale content instead of ADR-1235’s guaranteed fresh regeneration of the trial’s actual current sources. The fix is conflictedGeneratedSpecsFromMergeOutput (engine/generated_files.go): git always prints a CONFLICT (...) line naming every originally-conflicted path, independent of whether rerere then silently resolved it, so a substring check of each declared generated path against those lines recovers the conflict’s original membership regardless of what unmergedPaths shows now. resolveTrainConflict unions that recovered set with whatever classifyConflictedPaths finds in the current (post-rerere) unmerged set via unionGeneratedSpecsByPath, before any dispatch decision — so a declared generated path is always force-regenerated, whether it’s still genuinely unmerged or was already silently replayed.

Trust boundary is unchanged (Requirement 4). A rerere replay is committed and pushed exactly like a fresh Claude resolution, then validated by the trial’s own combined CI — the same gate, with no weakening. Nothing about being “already resolved before” skips or shortens downstream validation.

Red-trial resolution hygiene: forgetPoisonerResolutions (Requirement 5). Bisection sub-trial worktrees are destroyed (cleanupTrialArtifacts) immediately after each sub-trial’s own CI result is known — before the poisoner is identified, since bisection only learns the poisoner once every sub-trial has run. This means the porcelain git rerere forget <path> command (which re-derives the current conflict’s hash from a live conflicted working tree — it cannot be pointed at an arbitrary historical hash from outside a live conflict) can’t be run against the sub-trial that actually observed the poisoner’s conflict; that worktree is already gone. Once handleRedBatch isolates the poisoner, forgetPoisonerResolutions reconstructs a fresh live conflict instead — matching the episode’s original merge order, not merely poisoner-vs-base: assembleTrialBranch merges members strictly in order into one accumulating branch, so the conflict that actually produced a poisoner’s recorded resolution is usually poisoner vs. the trial-so-far (base plus every earlier-in-order red member already merged in), not poisoner vs. base alone — a solo poisoner-vs-base reconstruction can silently miss the resolution entirely (a review caught this in the initial implementation; see ADR-1834’s Decision 4). forgetPoisonerResolutions takes red (the full episode’s red batch, in its original order — bisect only ever slices it, never reorders it) and poisoner’s position idx within it: it replays red[:idx] into a disposable worktree forked off the same pinned p.baseSHA every trial in the episode used (each earlier member merged with --no-ff --no-edit, relying on rerere.autoupdate to silently replay that member’s own already-recorded resolution — never invoking Claude; an earlier member that doesn’t fully replay aborts the reconstruction and skips forgetting, fail-safe), then merges poisoner.headSHA last. If that final merge conflicts, git rerere forget runs for every path named in that merge’s own output (conflictedPathsFromMergeOutput, the same mergeOut-derived technique as the ADR-1235 guard above, not unmergedPaths: once rerere’s autoupdate has replayed and staged a path, git status no longer reports it as unmerged even though there is still a recorded resolution worth forgetting there). If it doesn’t conflict at all, there is nothing to forget for this episode’s primary trial order — silently, with no warning logged. This is a narrower guarantee than “every resolution poisoner was ever party to”: a bisection sub-trial that combined poisoner with a different partial subset of red can record a distinct resolution this reconstruction doesn’t replay and so can’t forget — accepted, not solved, since the unchanged CI trust boundary still gates any such stale replay before it lands. Entirely best-effort throughout: every git failure here is logged and none of them affect the eject that already happened. Skipped entirely under the trainValidateFn test seam, which never runs assembleTrialBranch’s real git assembly in the first place, so there is no real rr-cache entry from that episode to forget.

Pruning: periodic git rerere gc. Nothing in the codebase invoked git rerere gc before this issue, so gc.rerereResolved/gc.rerereUnresolved’s default expiries (60 days / 15 days) were moot and rr-cache would grow unbounded. runMergeTrainWorker now runs it once per worker invocation, immediately after prepareTrainWorker succeeds (mirroring that function’s own base-SHA-pinning gate: skipped under the trainValidateFn test seam), best-effort. The defaults themselves are accepted as-is — rr-cache entries are small text blobs and the merge-train’s own observed reuse window (minutes to weeks, per the motivating incident) comfortably fits inside 60 days.

Accepted, not mitigated: concurrent rr-cache writers. Two merge-train workers for the same repo but different (repo, base) partitions (ADR-1648) can run truly concurrently, each in its own trial worktree off the same shared bare clone, both writing into the same rr-cache with no mutex serializing the git merge/git commit sequence between them. Git’s rr-cache writes are per-hash-directory, so two different conflicts recorded concurrently don’t collide; the only real race is two workers resolving the identical conflict hash at the same moment — narrow, low-probability, and low-severity (worst case, one valid resolution’s postimage is overwritten by another, still gated by the unchanged CI trust boundary above). No code mitigates this; it is a conscious trade-off, not an oversight.

No configuration toggle. rerere is unconditionally enabled whenever a bare clone is created or repaired; there is no merge_train_rerere-style opt-out, matching the issue’s own requirements (which never mention one).

Interaction with #1841’s turn-limit/prompt changes. #1841 changed resolveConflictWithClaude’s invocation shape (dropped the build/test verification instruction, added turn-limited-exit handling and InvokeOptions.NoResume) but touches none of the three-way dispatch table above: rerere’s replay-or-not decision (resolveTrainConflict’s unmergedPaths/conflictedGeneratedSpecsFromMergeOutput classification) runs entirely before resolveConflictWithClaude is ever called for the plain and mixed cases, so it is unaffected by what Claude is told or how a turn-limited exit is subsequently classified. A conflict rerere fully replays still short-circuits to commitRerereReplayedMerge/regenerateAndCommit with no Claude invocation at all, exactly as before.

Code path: enableRerere (engine/worktree.go); conflictedGeneratedSpecsFromMergeOutput, unionGeneratedSpecsByPath (engine/generated_files.go); commitRerereReplayedMerge, resolveTrainConflict, conflictedPathsFromMergeOutput, forgetPoisonerResolutions (engine/merge_train.go, called from handleRedBatch), the git rerere gc call in runMergeTrainWorker.

References: ADR-1834: Merge-Train Conflict Resolution Replay via git rerere, ADR-1235: Generated-File Regeneration in Merge Train (the interaction this issue must preserve), ADR-059: Internal Merge Train (D3 conflict resolution, D4 bisection, D-e non-isolable interaction), ADR-1648: Merge-Train Per-Base Partitioning (the accepted concurrent-writer surface), ADR-1833: Deterministic Queued Batch Ordering (the prerequisite for rerere’s hit rate), issue #1834

6.27 Merge-Train Trial Prefix Reuse (ADR-1835)

Trigger: Issue #1835. A merge-train trial is a linear chain of merge commits — pinned base → +m1 → +m2 → … → +mn (assembleTrialBranch). Whenever a trial is abandoned — a bisection sub-trial, a re-form after ejecting a poisoner, or any other re-assembly — the whole chain was discarded and the next assembly started again from the pinned base, redoing merge work (including Claude conflict resolution) for members whose result was still valid: everything before the first changed input is completely unaffected by that change. The motivating incident (verveguy/concept-maps, 2026-09-20) shows bisection’s first probe landing on exactly the first four merges of the red trial’s own eight-member chain — a prefix by construction — yet the train rebuilt it from base anyway, re-paying 4.5 minutes of Claude conflict resolution for a commit that had existed minutes earlier.

Mechanism: a content-addressed hash chain, scoped to one worker invocation. trainPrefixCache (engine/merge_train_prefix.go) records h0 = chainHashSeed(trainKey, baseSHA) — computed per lookup from the baseSHA passed to lookup(p.baseSHA, members), never fixed at construction, because landOneAtATime’s per-singleton re-pin and the main-moved rebuild loop both change p.baseSHA mid-invocation while sharing the cache — and h_i = chainHashStep(h_{i-1}, member[i].Number, member[i].headSHA) for each member that merges successfully; only successful merges advance the chain — an ejected member (either assembleTrialBranch’s own mid-assembly unresolvable-conflict ejection, or bisection’s post-CI poisoner ejection) contributes no step, so the next successful member’s hash is computed against whatever hash preceded the ejected one. Each h_i maps to the resulting merge commit’s SHA. assembleTrialBranch looks up the new member list’s longest matching prefix by walking the identical recurrence forward from h0 and stopping at the first miss; on any match it forks the trial worktree (EnsureTrainWorktreeAt) from that prefix’s commit instead of the pinned base SHA, pre-seeds survivors with the matched members, and merges only what’s left. This single mechanism, with no separate invalidation logic, covers every re-assembly path named in the issue:

Scope is one merge-train RUN (#2051: the cache belongs to the run, so it survives the worker goroutine exiting between bisection steps) — never persisted across restarts. The cache is constructed right after prepareTrainWorker succeeds (same spot as the existing git rerere gc call, skipped identically under the trainValidateFn test seam, which never runs real git assembly at all) and threaded through trialParams.prefixCache to every trial the invocation assembles — the main-loop re-form and every bisection sub-trial share the same pointer. p.baseSHA and every member’s headSHA are pinned/fetched exactly once per worker dispatch and held constant for its whole lifetime, so a cache that only lives as long as those values are guaranteed stable needs no cross-invocation staleness tracking (“did a member land or leave Queued since this was recorded?”) at all. A restart, or even the very next poll’s fresh worker dispatch, starts from a nil-equivalent cache — by construction this can only ever reproduce full re-assembly, never a wrongly-trusted reuse, since there is no persisted state to misapply.

Reachability: non-pushed refs, gc-protection only. Each successful merge is recorded into the in-memory entries map (the only thing lookup ever consults for correctness) and, in the same step, as a git ref under refs/fabrik/merge-train-prefix/<sha256(trainKey)>/<h_i> in the shared bare clone — never pushed to origin — so git gc cannot prune a commit the cache might still reuse. The ref’s existence is never itself evidence of validity; it exists purely so the commit survives external garbage collection while the in-memory entry that governs reuse is still live. At the start of a worker invocation, sweepStaleRefs deletes everything under this trainKey’s own ref directory before any lookup or record — mopping up refs a prior invocation left behind if it crashed before reaching its own cleanup — and at the end of every invocation (any exit path, via defer), the same sweep removes everything the invocation created, unconditionally: nothing will ever look up this invocation’s chain again once it’s gone. The ref directory is keyed by a hash of trainKey alone, independent of the pinned base SHA — this makes the crash-recovery sweep provably safe against a concurrently-running sibling (repo, base) partition sharing the same bare clone (ADR-1648): the in-flight guard already guarantees no other goroutine is using this exact trainKey right now, and a different trainKey hashes to a different directory.

Trust boundary is unchanged (Requirement 7). A reused prefix commit is pushed as a new trial branch and validated by the trial’s own combined CI exactly like any freshly-assembled one — this saves merge work, not CI runs, and a wrong prefix match would at worst waste a CI run on a wrongly-composed trial, never silently land anything incorrect.

forgetPoisonerResolutions (ADR-1834) is deliberately not rewired to consume this cache — it’s a best-effort hygiene helper outside this issue’s stated scope, not one of the re-assembly paths above, and it runs after the poisoner has already been ejected and the batch’s trial artifacts are already gone, a different lifecycle than the one this cache covers.

Test-only observation seams. trainPrefixLookupHookFn (matched length, total length, and member numbers at the start of every prefix lookup) and trainMergeAttemptHookFn (fires once per member an assembly actually attempts to merge, whatever the outcome) are both nil in production. The second exists because ADR-1834’s rerere replay can resolve a repeated identical conflict without ever invoking Claude — so the Claude-invocation counter alone can’t distinguish “the merge never ran at all” (this issue’s reuse) from “the merge ran but its conflict resolved for free” (rerere, unrelated); the merge-attempt hook is the literal, rerere-immune “zero merges” signal. mergeTrainPrefixReuseDisabledForTest (mirroring mergeTrainQueueSortDisabledForTest, ADR-1833) lets a test disable reuse entirely and observe the previously-suppressed merges return, proving the reuse tests are not vacuously true of a scenario that never needed any merge work to begin with.

Code path: trainPrefixCache, newTrainPrefixCache, chainHashSeed, chainHashStep (engine/merge_train_prefix.go); assembleTrialBranch’s prefix lookup/fork/record, trialParams.prefixCache, the cache construction/sweep/defer-cleanup in runMergeTrainWorker (engine/merge_train.go).

References: ADR-1835: Merge-Train Trial Prefix Reuse, ADR-1834: Merge-Train Conflict Resolution Replay via git rerere (the direct dependency and complementary mechanism), ADR-1648: Merge-Train Per-Base Partitioning (the shared-bare-clone concurrency this issue’s ref namespacing accounts for), ADR-1833: Deterministic Queued Batch Ordering (the prerequisite for stable member-list comparison), ADR-059: Internal Merge Train (D-b base-SHA pinning, D4 bisection), issue #1835

6.28 Merge-Train Live-Status Resume Guard (ADR-1871)

Trigger: Issue #1871. Right after the train lands a member, the next batch formation could still see it in Queued and run its landing a second time — a second Done move, close, Landed via … comment and landing-verification write (3 of 25 landings on verveguy/concept-maps, on both landing paths, 5–6 s apart). Cause: poll() reads the board once (engine/poll.go) and reaches handleMergeTrainBatch seconds later with that same object; meanwhile the previous worker moved the member to Done and finishTrain released the in-flight marker, so LoadOrStore succeeds and a fresh worker starts from the pre-landing snapshot. Both “resume an interrupted landing” branches then read Queued off that snapshot as “interrupted”. Their own Status == "Done" / fabrik:awaiting-landing-verification checks read snapshot fields too, so could not catch it.

Mechanism: a live Status read at the two resume decision points. liveLandingState (engine/merge_train.go) reads each member’s Status via e.client.FetchProjectItemStatus(item.ItemID) (falling back to LookupIssueProjectItem when the snapshot has no ItemID) — never e.readClient (boardcache-backed), item.Status or item.Labels; the same live-read discipline as §6.24. It is called from completeDeferredLanding (Route 1 of restart reconstruction, a per-member filter applied before fetchTrainMembers), from the pr.Merged branch of trySingletonFastPath (§6.20), and — via dropLiveLandedMembers — at fresh batch formation in prepareTrainWorker, right after fetchTrainMembers and before admitTrainMembers (§6.25).

Live read Outcome Action
Status equals the holding stage liveHolding Resume as before — a genuine interrupted landing (crash between merge and Done) or the fabrik:awaiting-advance retry (§6.17), where the member stays Queued
Status non-empty, not the holding stage liveMoved The landing already completed (or a human moved the item): skip — no Done move, close, comment or label write; logged
Status empty (no error) liveUnknown Resume (no positive evidence of completion)
Read error liveReadFailed Defer this poll, logged as deferring resume of #N: live status read failed

Polarity. Skip only on positive evidence (a confirmed non-empty, non-holding Status); an ambiguous empty Status resumes. A read error fails closed: both resume branches re-run every poll (each poll re-dispatches, reconstructTrainState re-probes, the fast path re-fetches FetchPRDetails), so deferral delays but never strands a member, whereas failing open would keep the duplicate landing exactly when reads are flaky.

Integration-PR path. Members that are moved or read-failed drop out of this poll’s landing; the rest still land (a merged batch PR can have some members Done live and others still Queued). When none remain the existing “no still-Queued members” early return applies and reconstructTrainState still reports handled, so no fresh batch forms that poll. Singleton fast path. liveMoved/liveReadFailed return true (“disposition decided”) — no trial is built and no landing is re-run.

Fresh batch formation. A stale snapshot can also put just-landed members into a fresh batch: when no merged marker PR matches (any landing that carries no batch marker, e.g. a series of singleton fast-path landings), reconstructTrainState returns false and prepareTrainWorker falls through to formation. fetchTrainMembers only checks that a linked PR exists with a head SHA (a merged PR passes) and admitTrainMembers only classifies the member’s own CI (green), so the trial — whose landed members’ heads are already in base, making the merges no-ops — would pass and landMergeTrainBatch would re-land each one. dropLiveLandedMembers therefore drops liveMoved members and excludes liveReadFailed members from this batch only (the next poll retries them); liveHolding/liveUnknown members are kept, so a genuinely Queued member is never dropped. If nothing remains the worker logs “no survivors” and returns. The resume-branch guards stay: they cover the restart case formation never reaches. Cost: one live read per member per fresh formation (including polls where the batch is merely pending).

Consequence for the landing functions’ own Done branches. landMergeTrainBatch’s Status == "Done" branch and finishSingletonFastPathLanding’s Done-status branch (both commented as restart safety) are now reachable only if the Status changes between the guard’s read and their own snapshot check.

Known gap (out of scope, ADR-1871). landSingleton’s dedicated landing PR carries no batch marker, so Route 1 never matches it; a stale-Queued member from that path goes to fresh formation and was not traced. Bisection, landOneAtATime and landGreenBatch share the snapshot source but are not entered from either resume branch. Follow-up to be filed separately.

Test seams. Engine.SetMergeTrainLandingGuardDisabledForTest makes the guard report liveHolding (pre-#1871 behaviour) so the sim scenarios can show the duplicate with the guard off. simgh.Sim.LagBoardStatus makes the bulk board reads keep reporting a held Status/labels while the single-node reads stay truthful.

Code path: landingLiveState, liveLandingState, dropLiveLandedMembers, prepareTrainWorker, completeDeferredLanding, trySingletonFastPath (engine/merge_train.go).

State transitions:

Before Trigger After Labels Added Labels Removed
Snapshot Queued, live Done (or any other non-holding column), merged PR found Resume branch runs liveLandingState → liveMoved Unchanged; nothing written — —
Snapshot Queued, live Queued, merged PR found liveHolding Landing completed once (as §6.20 / Route 1) (as the landing path) —
Snapshot Queued, live read errors liveReadFailed Unchanged; retried next poll — —
Fresh batch lists a member that is live Done (or any non-holding column) dropLiveLandedMembers → liveMoved Member dropped from the batch; nothing written — —
Fresh batch lists a member whose live read errors dropLiveLandedMembers → liveReadFailed Member excluded from this batch; retried next poll — —

References: ADR-1871: Merge-Train Live-Status Resume Guard, ADR-059: Internal Merge Train (FR-2 restart safety, unchanged), ADR-1644: Merge-Train Singleton Fast Path, ADR-1773: Merge-Train Base Sanity Check (live-read discipline), §6.20, §6.24, issue #1871

6.29 CI Infrastructure Failures: Startup Failure and Flake Re-run (#2052, ADR 2052)

Trigger: Two failures of the CI provider were read as the members’ fault or as “still pending”. A workflow run that fails before creating any job (startup_failure) leaves zero check runs, which every gate reads as “CI has not reported yet” — pollTrainCI polled it to CIBackstopTimeout (4h) and then threw the trial away, and with no required checks (mergeable_state: clean) the zero-check shortcut even returned green and landed a batch whose CI never ran. A single flaky test made a trial red and started a bisection of full CI cycles that could end by blaming an unrelated member.

Detection (engine/ci_startup.go). isStartupFailure(run): the workflow run is completed and either concluded startup_failure or has zero jobs and concluded failure. Any other zero-job completion is not one: cancelled (a concurrency-group supersession, #2033), action_required (a run held for a human’s approval), skipped, stale, success. Runs come from FetchWorkflowRuns (GET /actions/runs?head_sha=… plus one jobs?per_page=1 read per completed run for the job count), called only on the zero-check-run path. With no startup failure in view, a workflow run that is queued or in_progress holds the zero-check green shortcut (CI is on its way, so the absence of check runs is not completeness); a run in waiting or pending does not, so such a trial behaves as before #2052. startupWatch.observe considers only runs absent from the baseline snapshot taken when the last retrigger was issued (R3a: the dead run stays on the head SHA and would otherwise burn the budget in one loop) and returns none, retrigger, wait or abandon.

Retrigger. retriggerPR calls CloseIssue then ReopenIssue back to back (the reopen is retried reopenAttempts times; a PR that cannot be reopened is reported as an error and the caller abandons). A pull_request reopened event is what starts a fresh run; GitHub refuses to re-run a run that never started (403). The bound is maxCIRetriggers (2); after a retrigger, if no new run appears within ciRetriggerNewRunDwell (5m — e.g. a workflow whose types: omits reopened) the trial is abandoned. These are minutes, never the 4h backstop.

TrainCIInfra. The abandon outcome. It is never red and never pending, so no call site can fall into a charging or green-reading default::

Site Behaviour
assembleAndValidate wrapper does not recordTrial (runaway guard stays uncharged)
runMergeTrainWorker cleanup, cooldown, return — no bisection, ejection, pause, comment or board move
bisect / handleRedBatch a sub-trial returning infra aborts the whole episode (5th/4th return value); nothing is ejected because ejection follows bisection
landOneAtATime cleanup, cooldown, stop (the remaining singletons would hit the same failure)
landGreenBatch rebase loop cleanup, cooldown, return — no dissolveBatch comment
resumeTrain keeps today’s dissolve-and-re-form, plus the cooldown

After an abandon dispatchMergeTrainWorker refuses the (repo, base) partition for trainInfraAbandonCooldown (5m): nothing is charged, so the runaway guard cannot bound a permanently broken workflow (a member whose PR breaks the workflow YAML); the cooldown limits it to one trial per cooldown. Known residual: such a batch re-forms indefinitely at that cadence; a cross-trial pause is a possible follow-up.

Ownership of the closed window. The close/reopen runs only inside the owning worker’s pollTrainCI. reconstructTrainState’s stale-PR sweep is reached only through prepareTrainWorker, after dispatchMergeTrainWorker’s LoadOrStore on mergeTrainInFlight; a second dispatch for an in-flight partition returns before it. escalateClosedUnmergedTrial is reachable only from landMergeTrainBatch, after green — never from pollTrainCI. If the daemon crashes inside the window, the trial PR is left closed with no worker: the next dispatch’s sweep treats a closed-unmerged trial PR as neither open nor merged, so it does not close, reopen, land or escalate it, orphaned trial branches are cleaned silently, and a fresh train forms. Pinned by TestMergeTrainInfra_*.

Flake re-run (R5). On the first CheckRunsFailed the engine takes the workflow-run ID from each failed check’s DetailsURL (…/actions/runs/<id>/job/<n>) and calls RerunFailedJobs once per distinct run, then keeps polling. A failed check with no Actions run behind it (a third-party status) or an erroring re-run request stays red at once, exactly as before. The second failure counts only if a failing latest-per-name check run has an ID absent from the first failure (the stale failure is still visible for a poll or two after the re-run is accepted); if only stale IDs are visible, polling continues until trainRerunSettleDwell (2m) elapses; the stale failure is then red unless one of the workflow runs that was re-run is still queued or in_progress (rerunInFlight, one FetchWorkflowRuns read, matched by run ID — an unrelated workflow, or a run held in waiting/pending, never postpones the verdict). The in-flight read is made before the check-run read of the same poll, never after: read the other way round, a re-run finishing between the two reads shows as a stale failure plus “nothing in flight” and the stale failure is judged the second one. A re-run waiting for a runner is waited for at most trainRerunMaxWait (30m) past the dwell — its own bound, far below CIBackstopTimeout. A read error or refused actions permission falls back to red. The state is local to one pollTrainCI call, so every bisection sub-trial gets its own single re-run. RequiredContextsFailed and dirty stay immediately red. ciSuiteHold keeps the gate unsettled while the re-run is in flight. This softens ADR-1153’s strict policy by one re-run: a genuinely red trial takes one extra CI cycle to detect.

Permissions and degradation (R7). Both endpoints need the actions permission (read for the list, write for the re-run). actions: write is in RequiredGitHubAppPermissions (#2105, ADR 2105), so the fail-hard startup grant check and fabrik init --github-app refuse an installation without it (with PermissionShortfallRemedy’s two manual steps), and a new App’s manifest requests it. The runtime degrade below covers a permission revoked after startup. A refused call (gh.ErrForbidden for a non-rate-limit 403, or ErrNotFound) logs once per process and the engine behaves exactly as before.

Stage wait_for_ci gate (R6). settlePRMergeStateForCIGate (used by handleMergeAndCIGates; the auto-merge convergence path keeps the plain settlePRMergeState) has the same blind spot in two forms: clean with zero check runs cleared the gate with CI never run (rule 9), and blocked waited to the backstop and could end in a misleading R3 “required check never runs” pause. For a stage with wait_for_ci: true and zero check runs (inside the clean shortcut, which now reads the cached check runs, and ahead of the hadChecks/dwell/R3 returns) it applies the same startupWatch per PR head SHA (Engine.startupWatches, in memory — a restart may grant one extra retrigger; a push resets it). A retrigger or wait returns PRMergeUnsettled with MergeableState omitted; an abandon (or a reopen that failed) sets PRSettleResult.StartupFailure, which handleMergeAndCIGates routes ahead of every other gate to pauseForCIStartupFailure — a pause naming the startup-failed run. The pause also drops the PR’s retrigger state (resetStartupState): the fix a human makes (a workflow repair on the base branch, or the end of an outage) does not move the PR’s head SHA, so a kept state — spent budget, old baseline — would make the resumed item abandon and re-pause at once without ever retriggering. A resume therefore gets a fresh budget of maxCIRetriggers. Unlike the train, an item’s own PR has no in-flight marker, so the closed window cannot be made exclusive; it is mitigated by closing and reopening back to back, and a PR reading closed within ciRetriggerGrace (2m) of an engine retrigger is PRMergeUnsettled, not PRMergeTerminal (which would pause the item as “closed without merging”).

Stage wait_for_ci gate: flake re-run (#2072, ADR 2072). The gate’s counterpart of the train’s R5 re-run. In handleMergeAndCIGates’ ciFailure branch, ahead of the ci-fix-reinvoke dispatch, ciFlakeRerun (engine/ci_flake_rerun.go) acts only when gh.ClassifyCheckRuns(settle.CheckRuns) is CheckRunsFailed and no worker is in flight (a required-context- or commit-status-only failure has no check run to re-run and dispatches as before). Per PR head (Engine.flakeReruns, keyed owner/repo#PR, reset when the head SHA moves, pruned past startupStateTTL, in memory only — a restart may grant one extra re-run) it grants at most one re-run before the first CI-fix dispatch (R1) and one after a recorded no-op fix (R2: LastCIFixNoOpSHA() == head); the budget is marked under flakeRerunMu before RerunFailedJobs is called, so the poll loop and settleAwaitingCIScan can never both re-run a head. A re-run calls the train’s rerunFailedWorkflowRuns (run IDs from each failed check’s DetailsURL) and claims the item instead of dispatching; no issue comment is posted — a [#N ci-gate] flake re-run (R1|R2) log line names the PR, head and workflow run IDs. CIFixCycles is never charged. Stale-failure guard (across polls): the failed check-run IDs at re-run time are recorded; while only those IDs are failing the gate waits for rerunSettleDwell (engine clock, e.now()), then for as long as a re-run workflow run is queued/in_progress (rerunInFlight) up to rerunMaxWait; after that the stale failure is the verdict. A failing check run with a new ID counts immediately. After the R1 re-run a failure therefore dispatches the CI-fix worker, and after both re-runs a failure follows the no-op skip and then CIBackstopTimeout. Degrade (R4): a check with no Actions run ID (external CI), ErrForbidden/ErrNotFound (no actions: write) or any other re-run error marks the head degraded — logged once, never retried for that head, never a pause or failure — and the gate proceeds to today’s dispatch. The auto-merge convergence dispatch (merge_gate.go) and the train are out of scope. SetCIFlakeRerunDisabledForTest is the test-only neutralisation seam the R1/R2 tests’ twins use.

Tests: engine/ci_flake_rerun_test.go, tests/sim/ci_flake_rerun_test.go, engine/ci_startup_test.go, engine/merge_train_ci_infra_test.go, engine/merge_train_infra_test.go, engine/pr_settle_infra_test.go, github/workflowruns_test.go, tests/sim/mergetrain_ci_infra_test.go. There is no live e2e twin: a startup_failure cannot be induced on demand against the bed (recorded in tests/sim/simgh/FIDELITY.md).


6.30 Merge-Train Singleton Catch-Up (#2044, ADR-2044)

Trigger: Issue #2044 (reported in #2041; proposal 1 of #1826). A singleton whose own branch is behind the pinned base never takes the §6.20 fast path (its ancestry check requires FetchCommitsBehind == 0), so it built a trial branch, opened a draft integration PR and ran a full CI run — then discarded the trial and left the member’s branch as stale as before. On a busy board that is most landings.

Mechanism. trySingletonCatchUp (engine/merge_train_catchup.go), called from runMergeTrainWorker’s re-form loop for a length-1 batch after trySingletonFastPath declines and before the trial is built (never inside assembleAndValidate, so bisection sub-trials and landOneAtATime are out of reach; batches of two or more keep trial branches, and landOneAtATime keeps trials by decision — ADR-2044). Gated on singleton_catch_up (merge default, off restores the trial path exactly), a known pinned base, FetchCommitsBehind(p.baseSHA, m.headSHA) > 0 (an unreadable ancestry builds the trial) and the per-member attempt cap (effectiveMaxTrainRebaseCycles(), in memory, reset on landing/eject; past it the trial takes over). Then:

  1. Ownership (R4). The member is deferred — left Queued, nothing charged, no trial this poll — when a stage session is in flight (itemstate worker), a live FetchLabels read shows fabrik:editing (or cannot be read), the issue worktree is dirty or mid-merge, or the remote branch is no longer the batch-snapshot head (WorktreeManager.PrepareCatchUp: ErrCatchUpBusy). The checks run again just before the push. A local fabrik/issue-N that is ahead of or diverged from the remote (ErrCatchUpLocalDiverged) carries unpushed work: it is never touched and the member takes the trial — unless it is a pure rebase of the remote (#2150, ADR-2150): the same patches on a newer base, with no local-only patch (compared by patch-id, git rev-list --right-only --cherry-pick --no-merges <remote>...HEAD ^origin/<base>; plain git cherry would count the new base commits as local) and no local-only merge commit. That is the footprint of an engine pre-invocation rebase that was never pushed (§4.4); it holds no work the remote lacks, so PrepareCatchUp logs catch-up: local branch is a pure rebase of the remote — resetting to the remote head, runs git reset --hard <expected head> and proceeds from the remote head — the head CI validated. The local rebase is never pushed. Any local-only patch or merge commit, a conflict-edited rebase (its patch-id changed), or any git error or failed base fetch keeps the refusal. SetCatchUpPureRebaseDisabledForTest is the neutralisation seam.
  2. Merge (R1). In the member’s own issue worktree (so the local branch stays equal to the remote), git merge --no-ff <p.baseSHA> — the pinned SHA, never a live origin/<base>; a merge commit, no rebase. A conflict runs the trial’s path (resolveTrainConflict: rerere → generated-file regeneration → Claude, NoResume, p.maxTurnsOverride; prompt re-worded via InvokeOptions.CatchUpBaseSHA). The result must be a two-parent merge of the member head and the pinned base (anything else restores the worktree and takes the trial). A clean merge is pure; any conflict resolution makes it not pure. An unresolvable conflict restores the worktree and ejects the member through ejectMember (stay in queue), as a trial would; the account-wide Claude usage-limit sentinel defers without ejecting (ADR-1120).
  3. Push. A Fabrik-Train-Catch-Up: <base sha> trailer is stamped (audit only), then PushCatchUp pushes with --force-with-lease=<ref>:<expected head> — a compare-and-swap, never a real force — and advances the remote-tracking ref so the next stage’s PushBranch lease starts from the catch-up commit. A rejection (branch protection, signed commits, the member moved) resets the worktree to the member’s head and falls back to the trial, uncharged: no recordTrial, no ejection count, no pause. Policy memo (#2065, ADR-2065). PushCatchUp returns a *CatchUpPushError carrying git’s raw output; classifyCatchUpPushRejection (engine/catchup_policy.go) treats it as repo policy only on a positive GH006 (protected branch) or GH013 (repository rule) marker, vetoed by any lease (stale info, fetch first, non-fast-forward) or transport/permission marker and by a GH013 that mentions secrets/push protection (content-dependent). Free prose never matches. On a policy match runCatchUpGit returns the fallback with policyReason set, and trySingletonCatchUp records an in-memory memo keyed by owner/repo (not the (repo, base) partition) for catchUpPolicyMemoTTL (24h, engine clock) and logs one line naming the repo, GitHub’s sanitised one-line reason (≤300 runes) and singleton_catch_up: off. While the memo is unexpired, trySingletonCatchUp returns to the trial path immediately after the baseSHA check — before FetchCommitsBehind, so no GitHub call, worktree, merge or push — and logs nothing. A rejection arriving during an active memo neither extends it nor logs. Expiry and restart re-probe; nothing is persisted. Other fallbacks (refused PrepareCatchUp, bad merge shape, CI never starting) are not memoised.
  4. Marker. One comment on the member’s PR carrying <!-- fabrik:train-catch-up head=<sha> base=<sha> pure=<bool> --> (see below).
  5. CI wait (R2). waitMemberCI blocks in the worker (only the worker keeps the pinned base meaningful; across polls a busy base would catch up again and again) on the member PR’s own CI at the new head: classifyLandingCI/ciSuiteHold unmodified, bounded by CIBackstopTimeout. It applies pending review/comment ejects each iteration, treats zero check runs as pending until the no-CI dwell (default 10m → fall back to the trial; the fast path rejects zero runs anyway), treats a head that is not the pushed commit (after a short sync dwell), a closed/merged PR or dirty as “moved” (member stays Queued), and never closes and reopens the member’s PR (a trial-PR technique). The first confirmed failure re-runs the failed jobs once (rerunFailedWorkflowRuns, §6.29); a second is red.
  6. Land (R2/R5). Green → trySingletonFastPath on the caught-up member: its four eligibility checks (§6.20 — live head == the caught-up commit is R5’s TOCTOU check) are the landing gate, unmodified; MergePRAtHeadSHA pinned to the caught-up head. The landed comment becomes “Landed via singleton fast path PR #N after a catch-up…”. Red → ejectRedCatchUpSingleton (reroute off Queued, comment naming the catch-up commit and the diagnostic, fabrik:paused) — the red-singleton disposition (the “Red singleton short-circuit” in the merge-train overview, #1440/#1545). Backstop timeout, moved head, applied eject → member stays Queued/rerouted, nothing charged.

Review churn (R3). Pruefer reviews every head, and an unaddressed review body holds landing (§6.6.6) while the Queued settle scan ejects a member with review findings (§6.16) — which would eject the member the catch-up meant to land. A review or review-thread comment is non-actionable for the landing only if all hold (engine/catchup_feedback.go, catchUpSuppresses): the PR carries a marker comment authored by selfLogin() whose head equals the PR’s live head; pure=true; the finding’s commit (the review’s commit.oid — now selected by the board query, so PRReview.CommitID is populated on the GraphQL path — or the thread comment’s originalCommit.oid, Comment.CommitOID) equals that head exactly; and the author is a bot (gh.IsBotLogin or a declared expected_reviewers identity). Everything else stays actionable — a conflict-edited catch-up, a human, an older head, no attribution, no marker, a marker from another author, an unreadable marker. Human comments (findNewComments) are never filtered. It is applied in the two shared places — queuedReviewFindings (the settle scan and the worker’s checkpoints) and unprocessedFeedback (the landing/advance gate) — via dropCatchUpFeedback, which reads the marker (one FetchIssueComments) only when a bot finding was made against the live head. That lookup is cached per (PR, head) (catchUpMarkerCached): a found marker for good, “no marker” for catchUpNoMarkerTTL (2 min), a read error never — so a bot-reviewed head that was never caught up costs at most one extra read per TTL window, not one per poll. The worker drops the entry when it posts the head’s marker (forgetCatchUpMarker). The commit trailer is not trusted — the branch is writable by anyone with push access, so a forged trailer must not mute feedback. Member CI on the new head is the safety net for a genuine finding on a merge-only head.

Validate completion survives the catch-up (#2158, ADR-2158). The catch-up moves the member’s head past the SHA recorded at Validate completion. The SHA-invalidation scan (§2.16) must not read that as an external push: it skips an item that has landed (the catch-up lands the member’s own PR, so the Done item would otherwise lose stage:Validate:complete a poll later) and, for an open item, accepts a head named by this section’s self-authored pure=true marker — re-recording it as the completion SHA — while a pure=false catch-up, a foreign marker or a bare trailer still invalidates.

Restart safety. Nothing but the pushed commit and the marker is durable. A crash between the push and the marker leaves reviews actionable for one cycle (fail closed). The next worker pins a (possibly newer) base and re-evaluates; the in-memory attempt counter bounds the loop.

Code path: trySingletonCatchUp, runCatchUpGit, waitMemberCI, ejectRedCatchUpSingleton, postCatchUpMarker (engine/merge_train_catchup.go); PrepareCatchUp/PushCatchUp/localIsPureRebase (engine/worktree_catchup.go); RestoreUnpushedRebase (engine/worktree_restore.go); dropCatchUpFeedback/catchUpSuppresses (engine/catchup_feedback.go); trainMember.caughtUpFrom; tests engine/merge_train_catchup*_test.go, engine/catchup_feedback_test.go, engine/worktree_catchup_test.go, tests/sim/mergetrain_singleton_catchup_test.go, engine/worktree_restore_test.go, engine/comments_restore_test.go, tests/sim/mergetrain_singleton_catchup_rebase_test.go.

State transitions:

Before Trigger After Labels Added Labels Removed
Queued singleton, behind the pinned base, owned catch-up pushed; member CI green; fast path eligible; MergePRAtHeadSHA Done, awaiting landing verification (as §6.20) fabrik:awaiting-landing-verification —
Queued singleton, behind stage session / fabrik:editing / dirty tree / remote moved / usage-limit hold Queued (unchanged) — deferred, no trial, nothing charged — —
Queued singleton, behind, local branch is a pure rebase of the remote (#2150) PrepareCatchUp resets the local branch to the remote head catch-up proceeds from the remote head (rows above) — —
Queued singleton, behind push rejected (lease or ambiguous), local branch carries a local-only patch or merge commit, bad merge shape, CI never starts, attempt cap, singleton_catch_up: off Queued → ordinary trial path, uncharged — —
Queued singleton, behind push rejected by repo policy (GH006/GH013) — first time Queued → trial path, uncharged; per-repo memo set (24h), one operator log line — —
Queued singleton, behind per-repo policy memo active Queued → trial path, no catch-up attempted (no merge, no push), uncharged — —
Queued singleton, behind unresolvable conflict stays in Queue via ejectMember (counted, as a trial ejection) — —
Queued singleton, caught up own CI red (after one failed-job re-run) Rerouted off Queued, paused fabrik:paused, fabrik:awaiting-input —
Queued singleton, caught up CI wait timeout, head moved, PR closed/merged, dirty, applied review/comment eject Queued (or rerouted by the eject) — nothing charged — —

References: ADR-2044: Merge-Train Singleton Catch-Up, ADR-1644, ADR-1953-a, ADR-1208/1863 (§6.16), ADR-2052 (§6.29), ADR-2150.

6.31 Merge-Train Red-Singleton Auto-Repair (#2045, ADR-2045)

Trigger: Issue #2045 (reported in #2039). ADR-1545 paused a red singleton for a human, who then applied fabrik:revalidate; on a busy board that step was the main reason finished work did not land. The engine now performs that step itself, bounded by an attempt cap that stands in for the external re-detection signal ADR-1545 found missing. Amends ADR-1545’s “pause for a human”; the pause remains the fallback.

Entry. ejectRedSingleton(projectID, owner, repo, m, p, diag) (both call sites: the true-singleton red trial and a member that fails alone in landOneAtATime; p is the trial’s trialParams, whose baseSHA is the pinned base — landOneAtATime re-pins it per member). Auto-repair is considered only when Config.MaxTrainAutoRepairAttempts > 0 (flag/env/yaml max_train_auto_repair_attempts, default 1 applied in cmd, 0 disables everything below), p.baseSHA is known and stageBeforeHolding is literally named Validate. Otherwise the unchanged ADR-1545 path (pauseRedSingleton) runs.

  1. R3 — already fixed (memberFixedAgainstLiveBase). Re-fetch the PR, resolve the live origin/<base> (liveBaseSHA; trainLiveBaseFn is the test seam), rebuild a trainMember from the live head and call singletonFastPathEligible unmodified (live head contains the live base, mergeable, own check runs present and classifyLandingCI green incl. ciSuiteHold). Confirmed → return with no reroute, comment, label or pause; the member stays Queued. Fail-closed: any error is “not confirmed”. Allowed once per (member, pinned base SHA) (takeRequeue), tracked apart from the repair budget.
  2. Cap check. autoRepairAttemptsAt(member, p.baseSHA) counts recorded attempts for that base SHA only. At or past the cap → pauseRedSingleton, whose comment gains an “Auto-repair attempts made (N)” list (base SHA, member head SHA at failure, failing checks, time).
  3. Repair (startAutoRepair).
    • Reroute to Validate (rerouteQueuedMemberOffHolding). A failure returns with nothing posted, counted or labelled (ADR-1208 ordering); the next poll retries.
    • Register the in-memory pending context (autoRepairState.pending) before re-entry so the next dispatch cannot miss it.
    • reenterValidate (extracted from handleRevalidateLabel, which still delegates to it): clears stage:Validate:complete/:failed, fabrik:paused, fabrik:awaiting-input, fabrik:awaiting-ci, fabrik:auto-merge-enabled and resets the store’s Validate retry/pause/last-attempt/cycle state. If it defers (a label removal failed), fabrik:revalidate is applied so settleRevalidateScan retries each poll. If that also fails the pending context is dropped and the member is paused as before (already rerouted, nothing counted).
    • Only once re-entry is secured: record the attempt, post 🏭 **Fabrik merge-train — auto-repair started** (failing checks, attempt N of cap) and emit the event.
  4. Dispatch. Validate dispatches in the poll goroutine like any stage run (R5: the train worker starts no Claude and holds no slot). writeContextFiles calls writeMergeTrainRepair, which for a non-comment Validate invocation writes .fabrik-context/merge-train-repair.md from the pending context without consuming it (a retry of the same Validate — incomplete run, turn-limit slice, tools-denied — is given the file again; handleStageComplete drops the record when Validate completes); on every other invocation (or with nothing pending — e.g. after a restart) a leftover file is removed. The file holds the failing checks (renderDiagnosticBlock), the trial head, pinned base and member head SHAs and the instruction to rebase/merge onto the moved base and reconcile with what landed, never reverting or working around it. The same instruction is in the embedded validate skill and one static buildPrompt line. A pending context older than 2 hours (repairContextTTL) is discarded instead of written, and so is one recorded for a PR head other than the item’s current LinkedPRHeadSHA — but that head check applies only before the context has first been written (repairContext.delivered): once delivered, the repair run itself rebases/merges and pushes, so a moved head is its own work and a retry of that run still gets the file, so a record that outlived its repair flow (item paused, blocked or usage limited) cannot leak into a later, unrelated Validate run. A missing context never blocks the dispatch.
  5. Rejoining. The repaired Validate rejoins Queued through the path any Validate completion takes; the repair adds no enqueue path.

State. autoRepairState on Engine (mutex-guarded; the train worker writes, the poll goroutine reads, handleStageComplete drops on Validate completion): attempts per member (history capped at 20), spent R3 requeues, pending contexts. In memory only: a restart resets the counts and drops pending contexts. resetEjectionCount (every landing path) also clears a member’s auto-repair state.

Observers. emitTrainEvent emits merge-train-failed with cause red-singleton-auto-repair, auto_repair=true, attempt, cap and a reason saying no human action is needed; cause red-singleton is emitted only when the member is paused. The pause comment keeps the 🏭 **Fabrik prefix, so neither comment counts as steering.

Unchanged (R4). ejectMember (bisection-isolated poisoners, MaxMergeTrainEjections), the runaway guard, #1615 escalations and ejectRedCatchUpSingleton (§6.30) keep their human gate.

Code path: engine/merge_train_autorepair.go (startAutoRepair, memberFixedAgainstLiveBase, writeMergeTrainRepair, autoRepairState), ejectRedSingleton/pauseRedSingleton (engine/merge_train.go), reenterValidate (engine/item.go), writeContextFiles (engine/context.go).

State transitions:

Before Trigger After Labels Added Labels Removed
Queued singleton, red trial cap > 0, target Validate, attempts at this base < cap, not already fixed Validate (rerouted), Validate re-entered, repair context pending — stage:Validate:complete, stage:Validate:failed, fabrik:paused, fabrik:awaiting-input, fabrik:awaiting-ci, fabrik:auto-merge-enabled
Queued singleton, red trial direct re-entry deferred Validate (rerouted); settle scan retries fabrik:revalidate —
Queued singleton, red trial live head contains live base and own CI green (once per base) Queued (unchanged) — —
Queued singleton, red trial cap reached, cap 0, non-Validate target, or re-entry could not be secured Validate (rerouted), paused (ADR-1545) fabrik:paused, fabrik:awaiting-input —
Queued singleton, red trial reroute fails Queued (unchanged); retried next poll, nothing posted or counted — —

References: ADR-2045, ADR-1545, ADR-1208, ADR-1644, ADR-2044.


6.32 Merge-Train Overlap-Aware Batch Composition and Post-Landing Invalidation (#2047, ADR-2047)

Trigger: Issue #2047 (reported in #2040; proposal 2 of #1826). Batches were formed in entry order and conflicts were found only by spending trials; after a landing every Queued member that now conflicts with the new base cost a trial (and often a pause) to discover. Two mechanisms, with opposite failure polarities on purpose.

A. Fresh-batch composition (admitByOverlap, engine/merge_train_overlap.go). The third fresh-formation filter in prepareTrainWorker, after dropLiveLandedMembers (§6.28) and admitTrainMembers (§6.25). Fresh formation only — both restart routes return from reconstructTrainState first; bisection, landOneAtATime and landGreenBatch never call it. It runs on the already-capped batch (routeQueuedGroup caps first), so a deferral can leave the batch smaller than the cap.

  1. Order. Members that were deferred for overlap overlapStarvationThreshold (3) consecutive formations of their partition are considered first (FR-003); otherwise today’s (StatusEnteredAt, Number) order (§6.16/ADR-1833). Survivors keep their original relative order.
  2. Changed files. FetchPRFiles on the live client, read once per (repo, PR, head SHA) and cached in memory (a new head SHA overwrites). A read error, an empty/404 list or a list of 3000 or more (GitHub’s silent cap) is unreadable: the member is admitted unchecked, contributes nothing to the union, and an error is never cached.
  3. Walk. Paths matching merge_train_overlap_ignore (internal/pathglob: * never crosses /, ** spans segments) are dropped from both sides. A member is kept if its remaining paths are disjoint from those already kept, else deferred. The first member considered is always admitted.
  4. Deferral. The member simply stays Queued — no reroute, comment, ejection count or pause. One log line: deferred #B: overlaps #A on <smallest shared path>. The skip count increments; it resets when the member is admitted or is absent from a formation of its partition (counts are per trainKey).

B. Post-landing invalidation (invalidateConflictingQueued, engine/merge_train_invalidate.go). Every landing path (landSingleton, finishSingletonFastPathLanding, both branches of landMergeTrainBatch) records the landed member with noteTrainLanded. routeQueuedGroup consumes the record for a partition only while no worker is in flight for its trainKey (the ADR-1208 ownership rule: with no worker, the poll goroutine owns the Queued members), before the cap and before dispatch, and skips it under the trainValidateFn seam.

  1. Skipped (members untouched, logged) unless the reroute target stage has wait_for_ci (ADR-1821 R9), a WorktreeManager is registered, the base resolves, and git fetch origin succeeds.
  2. For each candidate that is not a landed member and carries neither fabrik:paused nor fabrik:editing: resolve refs/remotes/origin/fabrik/issue-N and run git merge-tree --write-tree --name-only --no-messages refs/remotes/origin/<base> <head> in the bare clone — no CI, no GitHub API reads. A conflict is accepted only when both refs resolve and stdout starts with a result-tree OID (git merge-tree also exits 1 for an unresolvable ref).
  3. Conflict → rerouteQueuedMemberOffHolding first (reroute-before-side-effects, ADR-1208); on success any pending review/comment eject signal is consumed, fabrik:rebase-needed is applied best-effort (Validate’s mergeability gate re-derives it from live mergeability, so a wrong application self-heals, and applying it closes the yolo/cruise bounce window), one invalidated #N: conflicts with landed #M (merge-tree) line is logged (M = landed members whose cached file lists intersect the conflicted paths, else all landed this pass) and one 🏭 **Fabrik merge-train — rerouted (conflicts with new base)** comment is posted, deduplicated per (member, head SHA). It never calls ejectMember, never touches mergeTrainEjectionCounts and never pauses (FR-009); the rebase cycle is charged normally by dispatchRebaseReinvoke.
  4. Clean merge (even when the member overlaps the landed files) → left alone. Any error (git < 2.38, missing ref/object, failed fetch) → left Queued and logged (fail-safe). A failed reroute leaves the member Queued for a trial.

State. Engine.overlap (file cache, skip counts) and Engine.invalidate (landed record, comment dedupe), each behind its own mutex; in memory only, so a restart delays the starvation guard and drops a pending scan (cost: one trial, no worse than before). The signal is consumed once — no retry loop. Test seams: SetMergeTrainOverlapDisabledForTest, SetMergeTrainInvalidationDisabledForTest.

Out of scope. Bisection, restart reconstruction, one-at-a-time and green-batch landing paths; rerouting on file overlap alone; persisting counts or the cache.

Code path: engine/merge_train_overlap.go, engine/merge_train_invalidate.go, prepareTrainWorker and the landing sites (engine/merge_train.go), routeQueuedGroup (engine/poll.go), internal/pathglob, FetchPRFiles (github/prs.go, engine/interfaces.go).

State transitions:

Before Trigger After Labels Added Labels Removed
Queued candidate fresh formation, files overlap an already-admitted member Queued (unchanged; skip count +1) — —
Queued candidate fresh formation, files unreadable / not known complete admitted as before — —
Queued candidate skipped 3 consecutive formations considered first in the next formation — —
Queued member post-landing scan, merge-tree conflict Validate (rerouted), not paused, no ejection counted fabrik:rebase-needed —
Queued member post-landing scan, clean merge (overlap or not) Queued (unchanged) — —
Queued member post-landing scan, git error / unresolvable ref / fetch failed / no wait_for_ci target Queued (unchanged), logged — —

References: ADR-2047, ADR-1821, ADR-1208, ADR-1833, ADR-1648, ADR-2044, ADR-2045.


7. Edge Case States

7.1 Cooldown Retry

When Claude runs but does not output any completion marker, the engine enters a cooldown retry loop. This applies both when Claude exits cleanly without a marker and when it exits with an error (e.g., timeout, crash). Only start failures (binary not found, exec.Error, os.PathError) skip the cooldown — the item is retried on the next poll instead.

7.2 Failed Stage / Pause on Retry Limit

Channel events (#1968): A pause or escalation is reported as paused, awaiting-input or escalated — the transition into the same internal/attention state the read API shows (§7.13) — and a counter one short of its limit as cycle-limit-near; see §7.14.

When a stage fails MaxRetries times (default: configurable, 0 disables):

  1. escalateFailedStage() adds fabrik:paused + stage:<X>:failed
  2. Posts an explanatory comment
  3. Sets PausedByEngine(stageName) via itemstate.EnginePaused mutation
  4. Releases the lock

Degenerate output as a failure trigger: A stage whose stripped output trips the degenerate-output guard (§2.6 — a bare @file reference or absolute path, e.g. @/tmp/plan.md) counts as a non-completion for retry purposes, identical to “no marker present.” escalateFailedStage() takes an optional pre-formatted causeNote string, built by the caller; when the failure was caused by degenerate output, causeNote carries the offending reference and is appended to the pause comment as a **Cause:** paragraph naming it. On the first detection (before MaxRetries is reached), a one-time explanatory comment is also posted immediately so the operator isn’t left with silence until the final escalation.

No-artifact completion as a failure trigger (#1782): A stage whose invocation arrives with completed=false because artifactMissingOnComplete() fired upstream (§2.6) is a distinct cause from degenerate output — the CLI’s terminal result field (and every assistant turn) carried nothing beyond FABRIK_STAGE_COMPLETE itself — and counts against MaxRetries the same way. finalizeStageOutcome() recomputes artifactMissingOnComplete() against the pre-strip output specifically to build a distinct causeNote for escalateFailedStage() (never reusing the degenerate-output guard’s bare-file-reference wording, which would misdescribe this cause) and to post its own one-time first-detection comment (“no artifact harvested”) on first occurrence, mirroring the degenerate-output guard’s visibility. escalateFailedStage()’s causeNote parameter is exactly what makes both causes representable without either misdescribing the other — message construction lives at each call site, not inside escalateFailedStage() itself.

Recovery: User investigates, makes fixes, then removes fabrik:paused. On next poll, processItem() detects the failed label (or snap.PausedByEngine(stageName) from the store) and calls clearFailedStage(), which:

7.2.1 Resumable Engine Pauses (ADR-1460)

The invariant: every pause comment Fabrik posts ends with “remove fabrik:paused to resume” — this must actually be true for every pause site, not just stage-failure escalation. Before #1460, four pause sites violated it: pauseForReviewCycleLimit, pauseForCIFixCycleLimit, pauseForRebaseCycleLimit, pauseForEnqueueCycleLimit never applied itemstate.EnginePaused, so removing fabrik:paused was a no-op — the triggering cycle counter (ReviewCycles/CIFixCycles/RebaseCycles/EnqueueCycles) stayed pinned at its limit and the item re-paused on its very next evaluation (reproduced live on #1208: five seconds from unpause to re-pause, byte-identical repost comment, no comment involved in the resume attempt at all).

Two resume gates, chosen by when the pause fires relative to stage completion:

  1. Pre-completion pauses (stage not yet stage:<X>:complete) — escalateFailedStage, escalatePRCreationFailure, handleBoundaryViolation (§7.2 above), pauseForSliceLimit (§7.12) — resume via processItem()’s own wasPaused || hasFailedLabel gate, described above. Unchanged by #1460.
  2. Post-completion pauses (stage already stage:<X>:complete, or in the fabrik:awaiting-ci window) — the four cycle-limit sites — resume via handleEngineUnpause, a Phase 1 catch-up-loop handler (§6.2, Handler 0) added by #1460. processItem() is never dispatched for a stage-complete item with no new comment (itemNeedsWork returns false), so gate 1 above is structurally unreachable for these four sites — this was the flaw in the issue’s own originally-suggested fix. handleEngineUnpause runs first, unconditionally, in every Phase 1 pass: if snap.PausedByEngine(stage.Name) is true while the item is here at all (meaning fabrik:paused is currently absent, per the catch-up loop’s own admission filter), it calls the identical clearFailedStage() reset gate 1 uses, then lets the rest of the chain re-evaluate the just-reset counter in the same pass.

All four now-fixed sites apply itemstate.EnginePaused in both their fresh-pause branch and their reapply-existing-comment branch (see below) — the reapply branch must re-arm it too, or a second cycle-limit episode after a successful first resume would itself become a fresh one-way door.

No-repost on re-pause (R4): reusing the pattern from ADR-1408 (pauseForCITimeout/pauseForCIFixCycleLimit), each of the four sites checks hasPauseComment(item, <its own stable message fragment>) before posting — a mutate.go-shared helper generalized from ADR-1408’s CI-specific hasCIGatePauseComment. If a matching comment already exists for the episode, the labels are reapplied (reapplyPauseLabels) without a duplicate post.

Classification of every pause/counter site (full detail in ADR-1460):

Category Sites
Already correct (applies EnginePaused, resumes via gate 1) escalateFailedStage, escalatePRCreationFailure, handleBoundaryViolation, escalateSettle (synthetic stage key, out of scope), pauseForSliceLimit (§7.12 — mid-stage, gate 1 reachable)
Fixed by #1460 (applies EnginePaused, resumes via gate 2 / handleEngineUnpause) pauseForReviewCycleLimit, pauseForCIFixCycleLimit, pauseForRebaseCycleLimit, pauseForEnqueueCycleLimit
Independently resets (no fix needed) pauseForReviewTimeout (strips its own fabrik:awaiting-review anchor before pausing), pauseForConvergenceFailed / pauseForMergeGroupStall / checkAutoMergeConvergence’s “auto-merge disabled” branch (all strip fabrik:auto-merge-enabled, their shared anchor), tripCommentBreaker (self-expiring window), checkDependencies (live-derived)
Condition-driven, no counter (no fix needed) pauseForCITimeout (anchored to external fabrik:awaiting-ci state, not an attempt budget — re-pausing while CI is genuinely still unresolved is correct), pauseForPRClosedNotMerged, pauseForRequiredNeverRunningCheck, handleBrokenReviewLinkage, pauseForBrokenLinkage, ensureRepoReady/postSpawnCloneError/spawnChildren

Accepted side effect: clearFailedStage()’s reset (reused unmodified by handleEngineUnpause) also zeroes Attempts(stage.Name) — the unrelated MaxRetries failure counter — via StageRetryCleared. A manual unpause of a cycle-limit-paused item therefore also gives that stage’s failure budget a clean slate. This is deliberate (“a human unpause is ‘try again’”), consistent with the same reset already applying to escalateFailedStage/escalatePRCreationFailure, and is a net-new intentional behavior for the four newly-fixed sites (they had no working “unchanged” baseline to preserve — they were broken before #1460).

Restart durability: handleEngineUnpause reads PausedByEngine, which is in-memory-only (does not survive an engine restart — internal/itemstate/snapshot.go). This is not a gap: ReviewCycles/CIFixCycles/RebaseCycles/EnqueueCycles live in the same in-memory StageState, constructed empty by NewStore(nil) on every process start with no persistence layer for any StageState field — so a restart clears the pause flag and the triggering counter together, never one without the other. A restart before a human unpauses an item is equivalent to handleEngineUnpause having already fired for free (both read back as zero); there is no interleaving that reproduces this section’s defect via a restart. See ADR-1460’s “Restart Durability” section.

7.3 Claude Usage-Limit Exemption

Channel events (#1968): activateClaudeUsageLimit / clearClaudeUsageLimit, and the usage-limit hold’s deadline simply passing, emit the account-wide claude-usage-limited / claude-usage-limit-lifted; see §7.14.

When a Claude invocation exits because the account’s usage limit was hit (e.g. You've hit your session limit · resets 10:20pm (America/Edmonton)), it is a distinct condition from both §7.2 (genuine stage failure) and a process start failure (claudeRan == false, §1.4’s claudeRan discussion) — the process started and ran, but produced no usable stage outcome, and retrying immediately would only hammer the same limit. See ADR-1119 for why this reuses the same StageAttempted-without-StageRetryIncremented split handleBoundaryViolation established for an unrelated condition (cross-repo worktree boundary violations).

Detection: interpretClaudeResult (engine/claude.go) classifies a genuine usage-limit exit structurally, from the CLI’s own parsed result object only — never from anything the assistant wrote. classifyUsageLimitExit(resp claudeResponse, usage TokenUsage) checks resp.TerminalReason == "blocking_limit" (the usageLimitTerminalReason constant), the same terminal_reason field the sibling turn-cap check (resp.Subtype == "error_max_turns") already consults. A second structural shape is also a usage-limit exit (#1811, ADR-1811): the CLI reports some session-limit hits as terminal_reason == "api_error" with api_error_status == 429 (the HTTP status behind the failure), which classifyUsageLimitExit claims ahead of §7.3a’s api_error classifier. api_error_status decodes tolerantly (apiErrorStatus): only a JSON integer counts; absent, null, zero, or wrong-typed values read as “not 429” and never drop the rest of the result object. The result text is never consulted. Both shapes sit behind the same turns/cost exclusion gate, and detection only runs when a result object actually parsed (ok == true). When Claude exits non-zero without a FABRIK_STAGE_COMPLETE marker, no turn cap, and TerminalReason matches, interpretClaudeResult returns a *claudeUsageLimitError{Message, ResetAt, ResetFallbackReason} sentinel instead of the generic error wrapper; ResetAt/ResetFallbackReason come from the separate structured rate_limit_event scan (see “Structured reset instant” below, ADR-1815) — never from the result text, which for a 429 api_error names a reset time that is deliberately not parsed. An unparseable-JSON invocation (ok == false — process killed mid-stream, truncated output) has no structured payload to trust and is never classified as a usage-limit exit by any means; it falls through to ordinary failure/timeout handling.

The turn-count/cost shape from #1184 is kept as a belt-and-suspenders exclusion, carried over from the original prose-matching guard: classifyUsageLimitExit returns detected = false when usage.TurnsUsed > 0 && usage.CostUSD > 0, because a real usage-limit exit terminates the invocation immediately, before any work is billed, so an invocation that consumed turns and incurred cost cannot have hit one regardless of what TerminalReason says. The exclusion is conjunctive (turns and cost) so a partially-recorded usage struct cannot suppress a genuine detection.

This replaces the original usageLimitHitRE/usageLimitResetRE/detectUsageLimitExit prose match over raw NDJSON stdout bytes, which matched Anthropic’s usage-limit exit message wherever it appeared in output — including text the assistant itself had written. Without a structural check, any stage whose output merely discusses usage limits self-triggers an account-wide usage-limit hold — on 2026-07-27 a turn-capped run on #1178 that quoted #1084’s example message held dispatch for ~11 hours (#1183). A diagnostic-only log line (claudeLog(..., "error exit with unmatched terminal_reason=%q ...")) records any other non-empty TerminalReason seen on an error exit (e.g. the SDK’s documented rapid_refill_breaker value, a plausibly short-lived burst throttle deliberately not treated as a usage limit) so a real-world sighting is auditable without affecting classification. See ADR-1183.

Handling: finalizeStageOutcome() (engine/item.go) detects the sentinel via errors.As, immediately after the existing engine-shutdown guard, and routes to handleUsageLimitExit(), which:

  1. Applies itemstate.StageAttempted — the normal dispatch cooldown (retry_backoff, §7.1) applies, so the item does not retry on the very next poll and hammer the limit in a tight loop. Deliberately does not call StageRetryIncremented — the stage never ran, so this does not count against MaxRetries.
  2. If fabrik:claude-limit is absent, posts an explanatory comment naming the condition and applies the label — gated on the label’s own absence, the same once-per-episode idiom as fabrik:awaiting-ci/fabrik:bot-reprompted. If already present (a repeated hit within the same episode), neither the comment nor a duplicate label-add fires. The comment names the reset time only when it was sourced from the structured resetsAt (see “Structured reset instant” below); it is omitted on the fixed fallback.
  3. Skips commitWIP, push, and markCommentsSeenByStage — nothing was produced.
  4. Releases the lock and returns — no stage:<name>:failed, no fabrik:paused.

Auto-clear (per-issue): Immediately after the branch above, any invocation that reaches that point (i.e. was not itself classified as a usage-limit exit — success, blocked-on-input, no-work-needed, genuine failure/retry, or PR-creation failure alike) unconditionally removes fabrik:claude-limit if present. This is broader than “only clears on success”: a subsequent genuine failure right after a cleared usage-limit episode is still tracked fully independently via stage:<name>:failed/MaxRetries, so clearing on any non-limit invocation correctly signals “not currently limited” without masking a real failure.

This per-issue clear only fires when the issue is dispatched again. An issue that is labelled and then paused, blocked, or simply not redispatched keeps fabrik:claude-limit indefinitely after the account-wide usage-limit hold has already ended — see “Account-wide label sweep” below for the settle scan that closes this gap.

Regression guard: a usage-limit hit followed by a genuine failure leaves the full MaxRetries budget available for the genuine failure — the limit hit consumes zero attempts.

Account-wide usage-limit hold and backoff-to-reset (ADR-1120): the per-issue handling above is layered under an account-wide gate, because the usage limit applies to the account, not the issue — without it, every concurrently-dispatched item independently rediscovers the same limit and each waits out its own per-issue cooldown. Engine carries a single usage-limit hold deadline (claudeUsageLimitedUntil time.Time, guarded by a dedicated claudeUsageLimitMu, distinct from the general-purpose e.mu) that is checked lazily — a cheap read — at each of the three places a Claude invocation can actually be attempted: the stage-invocation loop (runInvocationWithExtension, engine/item.go), the comment-review loop (processComments/runCommentExtensionLoop, engine/comments.go), and merge-train inline conflict resolution (resolveConflictWithClaude, engine/merge_train.go).

Out of scope (still deferred): a dedicated escalation/safety-net path for persistent or repeated misdetection. Without MaxRetries counting against a usage-limit exit, a misdetected or unusually long-lived condition still retries indefinitely (now spaced out by the account-wide usage-limit hold window rather than the 5-minute cooldown) instead of ever escalating for human review — an accepted, explicit limitation per ADR-1119 and ADR-1120. Cross-process coordination between multiple Fabrik instances sharing one account is also out of scope: the usage-limit hold is per-process, so a fleet of instances will each discover the limit once.

7.3a Transient api_error Exemption

Scope (#1811, ADR-1811): this section covers api_error exits whose api_error_status is not

  1. An api_error carrying api_error_status == 429 is a session/usage limit — account-wide, not per-invocation — and is claimed by classifyUsageLimitExit first, taking §7.3’s path (label, comment, account-wide usage-limit hold). Every live api_error exit measured on 2026-09-19 (532 across three daemons) was such a 429; the eight-exit sample below almost certainly was too. Non-429, absent, or unparseable statuses keep the handling described here unchanged.

A Claude invocation that exits with resp.TerminalReason == "api_error" (the apiErrorTerminalReason constant) is, like §7.3’s usage-limit exit, a condition where the stage never ran — but unlike a usage limit, it is per-invocation and self-resolving, not account-wide. Observed live on 2026-08-08 (#1458): eight such exits across two issues, each at 1 turn and $0.0000, with three consecutive hits on #1408 incorrectly consuming its entire MaxRetries budget and pausing a PR that was otherwise mergeable and green.

Detection: classifyAPIErrorExit(resp claudeResponse, usage TokenUsage) (engine/claude.go) mirrors classifyUsageLimitExit exactly — structural-only (resp.TerminalReason == "api_error", never output prose), with the same conjunctive usage.TurnsUsed > 0 && usage.CostUSD > 0 exclusion carried over unchanged (a real api_error sample’s CostUSD could not be empirically captured during Research, so the guard is reused rather than dropped or re-derived — see ADR-1458). interpretClaudeResult checks this immediately after the classifyUsageLimitExit branch and before the diagnostic-only unmatched-terminal_reason log line, returning a *claudeAPIErrorExit{TerminalReason, NumTurns, CostUSD} sentinel in place of the generic error wrapper.

Handling — deliberately a distinct type, not a reuse of claudeUsageLimitError: claudeUsageLimitError is the trigger, by Go type via errors.As, for two behaviors beyond handleUsageLimitExit itself: activateClaudeUsageLimit (account-wide, in both runInvocationWithExtension and runCommentExtensionLoop) and the comment-processing circuit breaker bypass (processComments). Both are correct only for a genuine account-wide usage limit — an api_error says nothing about the account as a whole, so claudeAPIErrorExit is a separate sentinel type that simply never matches either errors.As(&claudeUsageLimitError{}) check, avoiding both behaviors by construction rather than by an added conditional. finalizeStageOutcome() detects it via its own errors.As branch (alongside the usage-limit and apiKeyHelper branches) and routes to handleAPIErrorExit(), which:

  1. Applies itemstate.StageAttempted — the normal dispatch cooldown (retry_backoff, §7.1) applies on the stage-dispatch path, exactly as for a usage-limit exit. Deliberately does not call StageRetryIncremented — the stage never ran, so this does not count against MaxRetries.
  2. Logs only. Unlike handleUsageLimitExit/handleAPIKeyHelperDetected, posts no comment and applies no label — a fifth, more minimal “did not run” shape. The condition is per-invocation and self-resolving on the next attempt; a durable label here would be indistinguishable from the orphaned-durable-state leak ADR-1183’s sweep exists to clean up, and a comment for a self-healing event is noise (see #1414’s “comment when a human must act, log when the engine self-heals”).
  3. Releases the lock and returns — no stage:<name>:failed, no fabrik:paused, no account-wide usage-limit hold.

The comment-triggered dispatch path (engine/comments.go) is deliberately unmodified. That path has no LastAttemptAt-based cooldown at all (§7.3’s cooldown only exists in itemNeedsWork’s stage-dispatch fallthrough); its only bound is the comment-processing circuit breaker (checkCommentBreaker, #1089, default 10 invocations/30 min). Because claudeAPIErrorExit is a distinct type from claudeUsageLimitError, it automatically falls through to the same checkCommentBreaker(item, "") call every other unclassified error already reaches — unlike a genuine usage-limit hit, which processComments deliberately exempts from the breaker. Counting an api_error toward the breaker is correct here: it says nothing account-wide that would justify exempting an issue’s comment thread from “no forward progress” accounting, and the breaker is the only mechanism that bounds this path (the 5-invocations-in-31-seconds burst observed on #1208 came from comment-triggered redispatch, not the 5-minute-cadence stage-dispatch cooldown).

Reinvoke cycle counters (#1812, ADR-1812). The exemption extends to the reinvoke path’s cycle budgets: an api_error (or usage-limit, usage-limit-gated, or apiKeyHelper) exit on a review, CI-fix or rebase reinvoke is refunded from ReviewCycles/CIFixCycles/RebaseCycles (and ReviewBlockedCycles when it was charged) rather than being permanently charged — see §6.2’s “Did-not-run cycle refund”. The comment circuit breaker (above) still counts the exit; only the cycle counters are compensated.

Regression guard: an api_error exit followed by a genuine failure leaves the full MaxRetries budget available for the genuine failure, exactly as §7.3 describes for a usage-limit exit. A genuine stage failure with no matching TerminalReason is entirely unaffected by this section — it still counts against MaxRetries and still escalates via escalateFailedStage() (§7.2) at the limit.

See ADR-1458 and #1458.

7.3b Tool-Permission Denial Exemption

On 2026-08-09, a Claude Code profile misconfiguration made every mutating tool (Edit, Write, any write-capable Bash) return a permission denial to Fabrik’s stage workers, while read-only operations kept working. The engine responded to the same condition two incompatible ways depending entirely on whether the worker happened to also emit FABRIK_BLOCKED_ON_INPUT: two issues paused cleanly via the marker path, and two others (#1456, #1462) retried three times, burned their full MaxRetries budget, and marked themselves stage:<name>:failed — a durable claim that the work itself was wrong, when the machine had simply been unable to write files. See #1523.

Structurally unlike every sibling in this family (§7.3, §7.3a, fabrik:api-key-helper-detected): a tool-permission denial does not abort the invocation — the CLI exits cleanly (is_error: false, subtype: "success", terminal_reason: "completed"), and real, committable work may have happened before the denial (e.g. several successful edits before one write is blocked). There is also no dedicated terminal_reason enum value to key off, unlike "blocking_limit"/"api_error" — the only structural signal is the CLI’s own permission_denials array on the terminal result line, confirmed empirically against the installed CLI (2.1.227) using Fabrik’s own invocation flags (--output-format stream-json --verbose --permission-mode dontAsk, no --dangerously-skip-permissions): a PreToolUse-hook-denied tool call populates this array with one entry per denial (tool_name, tool_use_id, tool_input), on an otherwise ordinary clean exit.

Detection: classifyToolsDenied(resp claudeResponse) (engine/claude.go) returns the deduplicated, first-seen-order list of denied tool names, alongside a per-denial (not deduplicated) []toolDenial carrying each denial’s tool name plus — for Bash only, best-effort decoded from the CLI’s tool_input — the specific command that was denied (#1775; see §7.3d). interpretClaudeResult consults it only in the clean-exit path (runErr == nil), gated on !completed — a denial the model worked around and still completed the stage is ordinary success, with no exemption and no label, exactly matching every incident report and this section’s own empirical reproduction. When the gate matches, interpretClaudeResult returns a *claudeToolsDeniedError{ToolNames} sentinel in place of the unconditional nil the clean-exit path previously always returned. Detection is scoped to the clean-exit path only, per the evidence above — a diagnostic-only log line records a permission_denials array seen on a non-clean exit (evidence-gathering for a shape not yet observed, never a classification), mirroring §7.3’s unmatched-terminal_reason diagnostic.

Handling — continue-processing, not short-circuit: because real work can precede the denial, *claudeToolsDeniedError follows §7.3a’s sibling claudeTurnLimitError/claudeResumeFailureError shape, not §7.3/§7.3a’s did-not-run early return. toolsDenied := errors.As(err, &toolsDeniedErr) is computed in finalizeStageOutcome (engine/item.go) alongside the pre-existing turnLimited/ resumeFailed checks; commitWIP, the branch push, markCommentsSeenByStage, and InvocationRecorded (with Errored excluding toolsDenied, alongside turnLimited) all still run exactly as for any other incomplete run — a late-invocation denial never discards earlier valid edits. In the final escalation block, toolsDenied is a fourth branch alongside turnLimited/ resumeFailed:

  1. itemstate.StageAttempted was already recorded unconditionally above (the normal dispatch cooldown applies). itemstate.ToolsDeniedRetryIncremented increments a distinct counter — deliberately not StageRetryIncremented (R2) — so a tool-permission denial never counts against MaxRetries.
  2. If fabrik:tools-denied is absent, posts an explanatory comment naming the denied tool(s) (toolsDeniedErr.ToolNames, joined) — plus the first denied command when the CLI’s tool_input made one decodable (firstToolsDeniedCommand/sanitizeToolsDeniedCommand, #1775) — stating that the denial is scoped to that one command, not the tool for the rest of the session, then applies the label — gated on the label’s own absence, the same once-per-episode idiom as fabrik:claude-limit/fabrik:awaiting-ci. A repeated detection within the same episode posts neither a duplicate comment nor a duplicate label-add.
  3. Compares the running count against MaxToolsDeniedRetries (default 3 — see “The MaxToolsDeniedRetries bound” below); at the bound, pauseForToolsDeniedLimit applies fabrik:paused + fabrik:awaiting-input (via the shared pauseIssue/EnginePaused primitives, mirroring pauseForSliceLimit’s shape exactly) — never stage:<name>:failed. The condition is never treated as a stage failure, even at the bound.

Label clearing (R3, per-worktree self-resolution): gated on !toolsDenied — unlike fabrik:claude-limit/fabrik:api-key-helper-detected, this invocation itself may be the condition being classified (the detection does not short-circuit), so an unconditional clear at the same site those two labels use would immediately erase the label the branch above is about to apply. On the next invocation that is genuinely not classified as tools-denied, the label clears — mirroring fabrik:api-key-helper-detected’s per-worktree self-resolution (a human fixes the permission configuration directly) rather than fabrik:claude-limit’s account-wide settle sweep, since the cause here is always local to one worktree’s environment, never account-wide. fabrik:tools-denied is included in transientLifecycleLabels for the closed-issue defensive sweep (§7.5).

Marker independence (R6): because classification is expressed as a non-nil error returned from interpretClaudeResult rather than a separately-threaded boolean, blockedOnInput := err == nil && CheckBlockedOnInput(output) (engine/item.go) is structurally unreachable once a denial is detected — no explicit marker-suppression code exists anywhere, and none is needed. The outcome (label, comment, counter, escalation) is identical whether or not the worker’s output also contains FABRIK_BLOCKED_ON_INPUT: that marker is a worker-side courtesy, never an engine guarantee, and the engine’s own structural classification is what governs the outcome — the asymmetry that let #1453 and #1498 recover cleanly while #1456 and #1462 burned their retry budget on the identical condition cannot recur.

The MaxToolsDeniedRetries bound (R5, ADR-1523): defaults to 3 (--max-tools-denied-retries / FABRIK_MAX_TOOLS_DENIED_RETRIES), lower than MaxSliceRetries (10 — a turn-cap preemption is routine and self-resolving by construction). A denial is command-scoped and often self-resolves once the worker reshapes the offending command (#1775, §7.3d) — unlike a genuine permission misconfiguration, which would recur identically — but a handful of cycles is still a reasonable place to draw the line before asking a human to look, higher than MaxResumeFailures (2) since the explanatory comment already reaches the operator on the very first detection (R4); the extra cycles before escalating guard only against a single spurious/flaky denial, never against expecting a retry to fix the underlying cause. An exempt condition that never escalates would be its own failure mode — an issue silently retrying an unwatched environment problem forever — which is exactly what this bound exists to prevent, the same rationale as §7.12’s slice budget.

Regression guard: a tools-denied detection followed by a genuine failure leaves the full MaxRetries budget available for the genuine failure, exactly as §7.3/§7.3a describe for their own conditions — ToolsDeniedRetries is tracked entirely independently of Attempts/MaxRetries.

See ADR-1523 and #1523.

7.3c Tool-Permission Denial Coverage on Reinvoke Paths

§7.3b’s bound (ToolsDeniedRetries/MaxToolsDeniedRetries, pauseForToolsDeniedLimit) originally existed only on the stage-dispatch path (finalizeStageOutcome, engine/item.go) — every reference lived in that one file. But interpretClaudeResult (engine/claude.go) is the single seam both invocation paths share: Invoke (stage dispatch) and InvokeForComments (comment processing) each receive an identical *claudeToolsDeniedError under the identical detection rule. processComments (engine/comments.go) is the funnel every comment-driven invocation runs through — ordinary comment review and all three reinvoke dispatchers (dispatchCIFixReinvoke, dispatchReviewReinvoke, dispatchRebaseReinvoke) via the shared dispatchReinvoke scaffold — so before #1704 a mode denial detected on any of those four invocation shapes fell into processComments’s generic warn log and the comment circuit breaker (§4.6), which is calibrated for a different failure shape (a runaway comment loop triggered by frequency) than a slow, deterministic denial recurring once per retry cooldown. At a poll: 120 cadence with a 20-minute stage-retry cooldown, a denial recurs roughly 1.5 times per 30-minute window — structurally unreachable against the breaker’s default threshold of 10 per window — so the loop never converged: no terminal state, ~$0.30–0.40 burned per attempt, indefinitely. The ci-fix-reinvoke case was the sharpest instance of this: CI had failed, so ci-fix-reinvoke was the designed repair path, and it was the one disabled by the fault it exists to recover from (#1657, #1704).

Fix: a new errors.As(err, &toolsDeniedErr) branch in processComments’s error handling (alongside the pre-existing usage-limit exclusion, in the if err != nil && !completed block) — covering all four invocation shapes in the single place they already share, rather than duplicating the check inside each of the three reinvokeOpts.after callbacks. It calls the same recordToolsDeniedDetection helper §7.3b’s stage-dispatch branch calls (extracted from the previously inline block so the two consumers cannot drift), and, once the shared counter reaches MaxToolsDeniedRetries, the same unforked pauseForToolsDeniedLimit. The branch returns before either checkNoOpCommentCycle or checkCommentBreaker runs, so a tools-denied cycle is bypassed from the comment breaker entirely, exactly mirroring how the usage-limit exclusion immediately above it is bypassed — the breaker’s own thresholds are untouched.

Consequences of sharing the counter: ToolsDeniedRetries is keyed by (repo, issue, stageName) only, not by which code path incremented it — so a denial on the stage path followed by a denial on ci-fix-reinvoke for the same stage accumulate against the same counter and the same MaxToolsDeniedRetries bound, and the outcome (pause at the bound, exemption from max_retries) is identical regardless of which invocation type detected it (R3). No change was needed in reinvoke.go itself — dispatchReinvoke’s error handling only logs whatever processComments returns; the classification, accounting, and escalation are already complete by the time control returns there.

Remedy naming (R4, corrected by #1775 — see §7.3d): pauseForToolsDeniedLimit’s escalation comment states the per-command scope, then offers remediation in order: try re-running the step as separate, simpler commands, or add a matching allowed_tools rule for the command that keeps getting denied, first; fabrik:unrestricted is named only as a last resort — a headless worker has no interactive prompt to grant a denied tool, so it is the fallback once the command-scoped fixes have been tried — with its trade-off (it removes all tool restrictions, not just the denied one) stated alongside it.

See ADR-1704 and #1704, #1657, #1523.

7.3d Command-Scoped Messaging and Measured Progress Wording (#1775, #1743)

Two corrections to the messaging §7.3b/§7.3c describe, both landing in #1775 since #1743 targets the same log line #1775 already touches:

Denials are command-scoped, not tool- or session-scoped. ADR-1523’s original reasoning — that a tool missing from --allowedTools “self-reports it unavailable, with no permission_denials entry,” so a real denial “most plausibly involved a hook” — was explicitly flagged in that ADR’s own text as “by strong inference,” never observed. A community report (#1741) directly contradicts it: within one session, an ordinary git status && … Bash call ran, and a later, differently-shaped base_branch=$(gh pr view …) Bash call was denied — impossible if a hook or mode had disabled Bash session-wide. The reporter’s own 745-denial census attributes denials to command shape (/tmp redirects, unallowed binaries, env-prefix/variable-assignment forms, cd-chains, conditionals), not to session or tool. ADR-1775 supersedes this specific piece of ADR-1523’s reasoning (and ADR-1704’s repetition of the related decision_reason_type: "mode" framing) without rewriting either — both remain as accurate historical records of the decisions as made at the time.

Consequently: classifyToolsDenied now also returns per-denial []toolDenial (tool name plus, for Bash only, a best-effort tool_input.command decode — decodeToolCommand, silently degrading to "" for absent/non-Bash/malformed input, never a panic). sanitizeToolsDeniedCommand renders a denied command safely for inline display: embedded newlines collapse to spaces, backticks become straight quotes, and the result is truncated via the existing truncateMiddle convention (engine/merge_train.go) with short constants sized for a single command line rather than truncateMiddle’s CI-log-sized defaults. Both recordToolsDeniedDetection’s initial comment and pauseForToolsDeniedLimit’s escalation comment name the first available command this way, state the per-command scope explicitly, and no longer contain “no retry can fix this on its own” — the phrase this section’s earlier text described, now removed from both the comment templates and claudeerr.ToolsDeniedError’s doc comment. ToolsDeniedError gained an additive Denials []ToolDenial field alongside the pre-existing ToolNames; every construction site that only ever set ToolNames keeps compiling and behaving unchanged.

The tools-denied log line no longer asserts unmeasured progress (#1743). Independently of the command-scoping fix, interpretClaudeResult’s tools-denied branch previously logged “stage did not make progress” — a claim derived purely from !completed (the absence of FABRIK_STAGE_COMPLETE), never from checking the worktree. This directly contradicted this document’s own §7.3b text (“real, committable work may have happened before the denial”) and plugin/fabrik-workflows/LABELS.md’s shipped fabrik:tools-denied entry (which already correctly says commits and pushes still happen). A reported instance: an Implement comment-review invocation made three commits (the entire fix, pushed), spent 80 turns, and was logged as having made no progress — the next run then treated the review comment as already handled and did nothing. The fix: runClaude captures the worktree’s HEAD before starting the Claude process and again immediately after cmd.Wait() returns (best-effort, gitHeadSHA, mirroring the existing headBefore/headAfter pattern dispatchReviewReinvoke uses for review-reinvoke, #1045), computes a commit count between them (gitCommitCountBetween, git rev-list --count) when both SHAs are non-empty and differ, and passes that count into interpretClaudeResult as a new parameter. The tools-denied branch reports “N commit(s) pushed, stage did not signal completion” when a positive count was measured, or “stage did not signal completion” — exactly what !completed actually establishes — when it wasn’t; it never asserts an unmeasured claim, and never prints “0 commit(s)”. The string “did not make progress” no longer appears anywhere in the engine.

See adrs/1775-command-scoped-tools-denied-messaging.md, #1775, #1741, #1743.

7.3e Toolchain Declaration Drift Detection

A long-lived daemon inherits PATH once, from whatever shell hosted it at process start. A worktree can later declare a toolchain version (.nvmrc, package.json engines.node, go.mod’s toolchain/go directive, or .tool-versions) newer than what that inherited PATH resolves, and nothing notices — workers keep running the stale toolchain silently, indefinitely; restarting fabrik itself does not help, since it re-inherits the same PATH from the same parent shell. Reported in #1658: a daemon whose hosting shell started 2026-08-11 kept resolving Node 20.20.2 after the repo pinned Node 24 via .nvmrc on 2026-08-23, producing 36 Unsupported engine warnings in a single issue’s worker logs before the mismatch was noticed.

Detection is structurally unlike every other exemption in this section (§7.3–§7.3c): it never skips the invocation. checkToolchainDrift (engine/toolchain.go) is a direct, best-effort side-effecting call at the top of both invocation paths — runInvocationWithExtension (stage dispatch, immediately after the apiKeyHelper check) and processComments (comment review, right after context files are written) — never routed through finalizeStageOutcome, never returning an error, never touching StageAttempted/StageRetryIncremented/max_retries. There is nothing for Fabrik to do about a detected mismatch except tell someone: re-resolving PATH or driving a version manager (nvm use, asdf install, …) is explicitly out of scope (#1786 R3).

  1. detectToolchainDrift reads whichever declaration files exist in the worktree and, for each comparable declaration, resolves the corresponding binary (node --version/go version) via exec.LookPath against the daemon’s own inherited PATH. A binary entirely absent from PATH is “cannot compare” — a distinct, pre-existing condition, never a mismatch. A successful resolution is memoized process-wide per tool for the daemon’s lifetime (resolveToolVersion): since the bug’s own premise is that this PATH cannot change without a restart, a second check for the same tool, on any issue, is a cache hit rather than a second shell-out. A resolution failure (subprocess timeout, transient exec error) is deliberately never cached — a failure carries no such invariance guarantee, so caching it would permanently and silently disable detection for that tool after a single bad moment; every invocation with a matching declaration retries an uncached failure instead.
  2. Comparison is deliberately narrow and fails closed on ambiguity: .nvmrc, package.json engines.node, and .tool-versions compare at major-version granularity only (an .nvmrc alias like lts/*, or an engines range using ||/x/multiple clauses, is “not comparable” — silence, never a false-positive warning); go.mod’s toolchain/go directive compares at major.minor granularity using minimum-version (>=) semantics, since a go.mod directive states a floor, not a pin — and GOTOOLCHAIN=auto (the Go 1.21+ default) already self-corrects in the common case, so this format mainly catches the GOTOOLCHAIN=local/auto-download-disabled edge case.
  3. On any comparable mismatch, gated on fabrik:toolchain-stale’s own absence: an explanatory comment naming every mismatched declaration (source, declared version, resolved version) is posted and the label applied — once per episode (per issue), not once per invocation (#1786 R5).
  4. If no comparable mismatch is found and the label is present, it is cleared — self-resolving once a worktree’s declaration file is fixed, or the daemon is restarted with a corrected PATH.

A daemon/worktree whose resolved toolchain already satisfies every declaration sees no new behavior and no new output at all (#1786 R4) — no comment, no label, no added latency worth noticing beyond a few small file reads and, at most once per tool per daemon lifetime, a version-check subprocess.

See §1.4’s fabrik:toolchain-stale row, docs/stage-lifecycle.md’s “Toolchain Declaration Drift Detection (Worktree)”, and ADR-1786.

7.4 Multi-Instance Lock Protocol

Per ADR-007:

  1. Instance acquires its own lock label, fabrik:locked:<identity> — e.lockLabel(), where the identity (e.lockIdentity()) is cfg.User in PAT mode and the App-derived ID under GitHub App auth (see “Lock identity under App auth” below)
  2. Waits lockVerifyDelay (2 seconds) for competing instances to place their locks
  3. Re-fetches labels via FetchLabels()
  4. If another fabrik:locked:* label is present: lexicographic tie-break — lower username wins (proceeds), higher username loses (releases lock and skips)
  5. Winner proceeds with stage invocation; loser returns nil

Edge cases:

Lock identity under App auth (#1893, ADR-1893). With no operator user:, the lock identity is <slug>-<hash>: the App slug (no [bot]) plus six hex characters of sha256(hostname NUL abs(fabrikDir)). Two local instances sharing one App installation therefore hold different labels, which keeps every check above meaningful — the label != lockLabel “held by someone else” test, the lexicographic tie-break, and the own-label-only cleanups. One shared fabrik:locked:<slug> label would make each instance read the other’s lock as its own, let a starting instance strip a live lock in runStartupCleanup, and reduce the tie-break to “both win”. fabrik:locked: (14 chars) leaves 36 of GitHub’s 50-character label limit, so the slug is truncated to 29 characters (deterministically; the hash keeps truncated slugs distinct). The label is stable across restarts on the same host and directory; moving the directory or renaming the host changes it and orphans a crashed instance’s old label until removed by hand. FABRIK_INSTANCE_ID, when set, replaces the hostname in the hash (container/pod deployments, where the hostname changes on every restart); startup logs the derived label. cfg.User, if still configured under App auth, is never used as the identity (R8); it is read only by the one-shot legacy sweep below. itemstate.LocalLockAcquired{User} and SeedLabels(…, lockedUser) carry the same identity.

Legacy lock sweep. A deployment that switches from PAT to App mode may have crashed holding fabrik:locked:<user>; under the new identity that label reads as “locked by another instance” and would strand the item. runStartupCleanup therefore additionally removes exactly fabrik:locked:<cfg.User> (only when App auth is active and user: is still set) from items with no live local Worker(). It does not sweep other fabrik:locked:* labels, which may belong to a live instance. Because nothing ties the <cfg.User> label to this instance (a PAT-mode instance still running as that user legitimately holds it), the sweep also skips a label applied less than legacyLockMinAge (2h) ago, or whose age cannot be read (FetchLabelAppliedAt, live), logging that the label was left for a human to remove.

Age-window limitation. The 2h window is a heuristic, not a liveness proof: no process can see whether another process is alive. It only matters when a PAT-mode instance and an App-mode instance share a username and a board. In that case a PAT-mode stage that runs longer than 2h (possible with no max_wall_time) could have its live fabrik:locked:<user> lock swept when the App-mode instance starts. Deployments without that mix are unaffected.

Host and directory instance locks (#2097, ADR-2097). The label protocol above arbitrates between engines that are already running; the flocks below keep a second engine from running at all. Run() takes them before anything else touches disk (acquireInstanceLocks, engine/instance_lock.go), in this order: (1) checkNotInWorktree(fabrikDir) refuses a directory with a .fabrik/worktrees path-component pair (symlinks resolved); (2) the directory lock, an exclusive non-blocking flock on <fabrikDir>/.fabrik/fabrik.lock whose content stays exactly "<pid>\n" (the live e2e gate parses it as a bare pid); (3) the board lock, a flock on <lockdir>/<slug>-<8 hex>.lock where <lockdir> is $FABRIK_LOCK_DIR or os.UserCacheDir()/fabrik/locks (mode 0700; no resolvable directory is a startup error, never an unlocked start) and the key is lower(host)/lower(owner)/projectNumber (empty ghes_host = github.com; OwnerType excluded). The board file holds one JSON line {"pid","dir","started"}, written in a single write only after the flock is owned. A refused engine reads that record through a separate open, retrying briefly, and names the holder’s pid and directory (or says the owner could not be identified); it releases the directory lock it had taken. All of this precedes log rotation, so a refused engine never rotates the winner’s log. Both fds are close-on-exec and never given to a child.

Per-poll verification (R3). Run()’s doPollCycle closure calls instanceLocks.Verify() before PollWithBackoff: for each lock, fstat of the held fd against stat of the path (dev+inode), then the recorded pid read from the held fd must equal os.Getpid(). It logs nothing on success. Failures are classified: a definitive loss — the path no longer exists (ENOENT), a dev/inode mismatch, a record naming another pid, or an empty/unparseable record — is fatal at once; any other fstat/stat/read error (a momentary EIO/ESTALE/EACCES/EINTR, e.g. an NFS home or a flaky cache mount) is inconclusive: it logs a warning and the lock is treated as still held, and only maxInconclusiveVerifies (3) consecutive inconclusive verifications of one lock count as a loss (a clean verification resets the count). acquireDirLock returns a failed pid write (e.g. disk full) at startup rather than leaving it to surface at the first poll as a lost lock. On a loss it prints lost the instance lock; another Fabrik may own this board (<which lock>: <reason>) to stderr and fabrik.log, sets Engine.lockLost, cancels the run context and returns errInstanceLockLost; drainAndExit then skips cleanupLockedIssues (in PAT mode another engine may legitimately hold fabrik:locked:<user>), never takes the SIGHUP branch, waits the bounded drain and returns the error, so the process exits non-zero. Verification is not in poll()/PollOnce (the sim seam deliberately skips the lock preamble, ADR-1449), and cycles triggered through the test-only poll seam bypass it.

Re-exec. performSighupRestart releases both locks immediately before syscall.Exec; a self-upgrade re-exec relies on close-on-exec. The pid is unchanged across exec, and Run() re-acquires both locks in the new image. If a competitor wins the gap it keeps the lock and the re-exec’d image exits. Older binaries and a second host are not covered.

7.5 Closed-Issue Catch-Up

Closed issues are normally skipped by itemMayNeedWork() and itemNeedsWork(). Exceptions:

  1. Cleanup stages: A closed issue in Done with a worktree still needs cleanup
  2. Complete-labeled items: A closed issue with stage:<X>:complete can be advanced by the catch-up loop (e.g., PR merge closes the issue while it’s in Validate with the complete label — it needs to move to Done)

Stale lock cleanup: cleanupClosedIssueLocks() runs every poll cycle and removes fabrik:locked:<user> labels from any closed issues on the board. This handles stale locks left when an issue was closed while a stage was in-flight.

Transient label sweep: cleanupClosedIssueTransientLabels() runs every poll cycle (immediately after cleanupClosedIssueLocks) and removes the labels in transientLifecycleLabels (fabrik:awaiting-review, fabrik:awaiting-ci, fabrik:auto-merge-enabled, fabrik:awaiting-input, fabrik:rebase-needed, fabrik:bot-reprompted, fabrik:revalidate, fabrik:claude-limit, fabrik:clear-claude-limit, fabrik:api-key-helper-detected, fabrik:skill-unresolved, fabrik:tools-denied, fabrik:toolchain-stale) from any closed issues. These labels have no meaning once an issue is closed and leaving them behind can cause confusion. The sweep is idempotent (treats ErrNotFound as success) and non-fatal (API errors are logged as warnings and processing continues). Operates on cache data from e.readClient.FetchProjectBoard(...) — labels come from the most recent deep-fetch (FetchItemDetails, up to 20 labels). Items never deep-fetched carry no labels in the store (probe-based bootstrap does not populate labels). In practice, closed terminal items are seeded by seedTerminalFromProbeItems (called after BootstrapFromProbe) and skipped by runProbeAndDeepFetch, so they will not be deep-fetched; the sweep sees empty label sets for these items and skips them (no matching transient labels to remove). Active items receive labels on the first deep-fetch cycle. A one-shot startup scan (runStartupTransientLabelScan) runs after the first successful poll to handle stale labels on closed issues that may have been missed during a prior crash — it scans the Store directly, no extra GitHub call needed.

7.6 In-Memory vs Durable State

State In-Memory Durable (Label/Reaction) Behavior on Restart
Stage invocation timestamp itemstate.Store → StageState.LastAttemptAt[stageName] None Lost — item retried immediately
Periodic re-eval cooldown itemstate.Store → ItemState.CooldownAt["periodic-re-eval"] None Lost — item retried immediately
Retry count itemstate.Store → StageState.Attempts[stageName] None Lost — retries restart from 0
Paused-due-to-retries itemstate.Store → StageState.PausedByEngine[stageName] fabrik:paused + stage:<X>:failed Labels survive; in-memory flag lost but processItem() detects the failed label directly
Review cycle count itemstate.Store → StageState.ReviewCycles[stageName] None Lost — cycle count restarts from 0
CI-fix cycle count itemstate.Store → StageState.CIFixCycles[stageName] None Lost — cycle count restarts from 0
Rebase cycle count itemstate.Store → StageState.RebaseCycles[stageName] None Lost — cycle count restarts from 0
CI merge pending since itemstate.Store → LinkedPRState.CIMergePendingSince None Lost — merge guard re-evaluates CI fresh on next poll
Comment processed itemstate.Store → StageState.ProcessedComments[commentID] ROCKET (🚀) reaction (review bodies: the durable review-ids-addressed marker; the 🚀 is GraphQL and human-visible only, #1953 R8) Reaction survives restart; in-memory dedup is defense-in-depth
Lock tracking itemstate.Store → ItemState.Lock fabrik:locked:<user> label Label may survive if process crashes; cleanupLockedIssues() runs on graceful shutdown. Lock is set only by stage dispatch; a reinvoke registers a Worker only (#2179, ADR 2179)
Change-feed set Engine.mayNeedWork[iKey] (Engine.mayNeedWorkMu) None Lost — all items re-evaluated on first poll
Deep-fetch failure itemstate.Store → ItemState.LastDeepFetchFailureAt None Lost — failed items retried immediately
Channel subscriptions, held events, dedup keys, settle-episode counters (#1968) channelevents.Hub (mirrors of the files) .fabrik/state/channel/subscriptions.json, held-<hash>.json, dedup.json (atomic writes, corrupt files quarantined) Survive — loaded before the socket binds; held events are delivered in order on re-attach
Channel deriver’s per-item memory (last pause-family state, counter arming, last comment link) engine.channelEvents (consumer goroutine only) None Lost — re-seeded silently from the store, so a restart announces nothing about the existing board

7.7 Invocation-Level Kill Mechanisms

Two proactive kill mechanisms cap how long a single Claude invocation can run. Both are implemented in runClaude() in engine/claude.go and operate independently of the engine-level context cancellation.

max_wall_time (per-stage YAML field)

Inactivity timeout (hardcoded 15 minutes)

Shared post-kill behavior:

  1. extractTextFromAssistantTurns(rawOutput) scans the already-buffered output for FABRIK_STAGE_COMPLETE appearing in the text content of any {"type":"assistant"} NDJSON line.
  2. If found: returns completed=true — the invocation is treated identically to a live FABRIK_STAGE_COMPLETE.
  3. If not found: returns completed=false — routes to cooldown/retry (see §3.2 and §7.1), not a hard error.

wasTimedOut flag: inactivityFired.Load() || (stageCtx.Err() == context.DeadlineExceeded && ctx.Err() == nil) — distinguishes our kills from engine-shutdown context cancellation. When wasTimedOut=true, the no-marker path follows the same cooldown/retry flow as a clean exit without markers. When ctx.Err() != nil && !wasTimedOut (engine shutdown), runClaude returns immediately with zero output.

Kill sequence: killProcGroupGraceful(pid, issueNumber, label, reason, sigintGrace, sigtermGrace) runs the session-wide SIGINT → grace → SIGTERM → grace → SIGKILL escalation of internal/sessionreap over every process in the worker’s session and its sampled command sessions (#1989; see docs/stage-lifecycle.md “Kill escalation”), not a single SIGTERM to -pid. This terminates grandchild processes (e.g., background sleep spawned by Monitor tool) that would otherwise hold the stdout pipe open past cmd.WaitDelay. The post-exit killProcGroup group signal and the escalation’s listing-failure fallback are ownership-checked before they signal (#1957, “Stored-PID group kills” in the same doc).

No-op on Windows: killProcGroupGraceful is a no-op on Windows (process groups work differently). Both timeout mechanisms still fire and set their flags, but the kill is a best-effort cmd.Cancel.

7.8 Poll Loop Idle Backoff and Webhook Health State

The poll loop’s effective interval grows when there is nothing to do (idle backoff). The cap on that backoff depends on the webhook stream health state.

Idle backoff algorithm (computeEffectiveInterval in engine/backoff.go):

REST/core exhaustion is a hard pause, not a slowdown (shouldPauseForRESTRateLimit in engine/backoff.go): the interval adjustment above conserves the GraphQL budget (spent by the poll read), but the REST/core budget is spent by per-item mutations (reactions, labels, comments, merges) and janitor fetches, which interval-stretching does not throttle. So when REST remaining drops to near zero (≤1% of limit, isRateLimitNearZero), doPollCycle skips the entire work phase — fetch, dispatch, and mutations — until GitHub’s hourly reset (reset + rateLimitResetBuffer), emitting RateLimitAlertEvent{Bucket: tui.RateLimitBucketREST, Exhausted: true}. The independent worktree-janitor goroutine (runWorktreeJanitor) applies the same gate on entry, so it does not keep spending REST while the poll loop is paused. REST stats are refreshed from the headers of every REST response (including 403s), so the reset timestamp is authoritative even at zero budget. Introduced to stop a 403 retry storm after the REST budget was drained (#1118/#1121). RateLimitAlertEvent.Bucket (tui.RateLimitBucketREST tui.RateLimitBucketGraphQL) identifies which of the two independent budgets an emit/clear event concerns, so AlertBannerComponent (tui/alert.go) tracks each bucket’s active/reset state separately and never lets one bucket’s Exhausted: false clear a banner the other bucket raised — see #1482. The GraphQL-only banner string reads “polling backed off,” not “polling usage limited” (#1716) — GraphQL exhaustion only slows polls via the interval/gate machinery above, it does not stop them the way REST exhaustion’s hard gate does; the REST-only and combined banner strings are unchanged and remain accurate.

Rate-limited probe logging is a per-attempt statement, not a usage-limit hold claim (R2, #1716). runProbeAndDeepFetch (engine/terminal.go) runs inside poll(), before PollWithBackoff’s GraphQL hysteresis block (above) computes for that cycle — so when its own ProbeProjectBoard call fails on a low/exhausted GraphQL budget, it has no visibility into whether the real backoff this section describes is active, and no usage-limit hold mechanism of its own. Its log line describes only what happened at this call site (one probe attempt was rejected for rate limit) rather than claiming a stand-down it does not control; the RateLimitAlertEvent{Bucket: GraphQL, Exhausted: true} emission is unchanged (it remains the sole source of the GraphQL-exhausted TUI signal). Log-rate protection (R4, #1716): this log line is routed through Engine.logfThrottled (engine/logthrottle.go) — a shared dedup helper that suppresses a repeated identical message under the same key until either the message changes or a fixed interval elapses, appropriate here since this line’s content (a minute-granular reset time, a mostly-stable error string) is genuinely static for the duration of one rate-limit episode. The three unconditional per-poll rate-limit stats/warning lines inside poll() (REST remaining, GraphQL remaining, GraphQL-low warning) instead use Engine.logfThrottledByInterval — a variant that throttles purely by elapsed time, ignoring message content entirely: their interpolated remaining-count changes on nearly every poll in ordinary (non-rate-limited) operation, so logfThrottled’s message-change-forces-log rule would have defeated the throttle for exactly the routine case it needed to cover (caught in review). Before this, a sustained rate-limited condition reproduced the same handful of lines every poll cycle; combined with the pre-#1716 wake-path bug this produced 39 log lines/second sustained in production (14 MB/hour). Both are thin wrappers over the same underlying dedup map — see engine/logthrottle.go’s doc comments for the exact semantics of each.

Idle cap selection (effectiveIdleCap in engine/backoff.go):

Webhook stream state Idle cap
WebhookStreamHealthy 60 minutes (webhookIdleCap)
WebhookStreamStartingUp 60 minutes (webhookIdleCap)
WebhookStreamUnhealthy 5 minutes (maxIdleBackoff)
Webhook mode disabled 5 minutes (maxIdleBackoff)

When the webhook stream is healthy or starting up, steady-state polling is suppressed to a 60-minute safety-net interval. Events that arrive on the webhook stream signal the wakeCh channel, which triggers an immediate poll regardless of the current interval. The backoff multiplier is preserved unless that poll finds active work (see “Activity reset” above).

Webhook stream health states (managed by webhookManager in engine/webhook.go):

State Meaning Idle cap used TUI indicator
WebhookStreamStartingUp Subprocess launched; no reconcile tick has completed yet. State persists until the first reconcileTicker tick confirms no drift. 60 min Blue ○
WebhookStreamHealthy Most recent reconcileTicker tick found no drift between in-memory cache and GitHub 60 min Green ●
WebhookStreamUnhealthy Cache is known to be out of sync (most recent reconcileTicker tick found drift, Pause/Reconcile/Resume in progress), OR subprocess failed to start / was permanently disabled by the 422 circuit-breaker 5 min Yellow ◌

State transitions (reconcile-driven — webhooks no longer drive health state):

Webhooks continue to apply deltas to the cache via ApplyDelta but no longer drive health state. Event silence does not produce health transitions — only LightReconcile drift detection does. LightReconcile discounts engine writes the listing cannot have seen (D.4, #2159), so an item whose only difference from the listing is a recent engine write neither marks the stream unhealthy nor triggers a reconcile.

The reconcile ticker runs independent of webhooks (poll-only correctness backstop). reconcileLoop (engine/reconcile.go) is launched unconditionally whenever the cache is active — it is not nested in the webhook-manager start path. This is required because webhooks are an optimization, not a correctness requirement: on a webhook-less deployment (or when the gh webhook forward subprocess fails to start), the reconcile ticker is the only mechanism that re-syncs the cache with GitHub, and it must still run. When the webhook manager is present it also receives the health-state transitions above; when it is absent (wm == nil) the drift detection and Pause → Reconcile → Resume repair still run, only the health-state signaling is skipped. LightReconcile drift detection compares status, updatedAt, and the fabrik-managed label set (fabrik:* / stage:* labels only — the board query truncates labels at 30, so the full set is not compared; the gate-label subset is small and never truncated). Comparing the gate-label subset closes the case where a store label set diverges from GitHub with a matching updatedAt (e.g. updatedAt advanced by a deep-fetch that syncs updatedAt but not labels), which would otherwise strand an item at a gate (e.g. fabrik:awaiting-ci missing from the store) indefinitely on a webhook-less deployment.

Webhook mode is always non-fatal. If the gh webhook forward subprocess fails to start, the stream state stays Unhealthy and the 5-minute idle cap applies. The poll loop (and the reconcile ticker) continues normally.

Idle backoff never sleeps past a due retry (#2232, ADR-2232). After computeEffectiveInterval, PollWithBackoff caps the interval at the earliest unexpired short engine-scheduled cooldown (retry-backoff, sentinel-live, merge-unsettled) found by one in-memory Store.Scan (earliestShortCooldown; cleanup-stage and closed items and expired entries are ignored), floored at 1 s (capIntervalToDue), and logs next poll capped to <d>. Long cooldowns (periodic-re-eval, dep-blocked, review-blocked, feedback-pending, spawn-*) deliberately do not cap, or the webhook-healthy 60-minute idle cap would shrink to 10 × poll. The cap only ever shortens the interval, so setNextPollDue/setPollPeriod stay correct, and it is skipped while GraphQL rate-limit backoff is active. The webhook wake path is unchanged.

References: ADR-032: Webhook-Driven Event Delivery

7.8a Test-Only Poll Hold/Trigger Seam (#1978)

Test-only — not a production feature. Enabled solely by the hidden environment variable FABRIK_TEST_POLL_CONTROL (Config.PollControlFile), which the live e2e gate runner and harness set on the bed engine and nothing else does. It has no flag, no YAML key and no entry in docs/USER_GUIDE.md. Unset (the production state) Engine.pollSeam is nil, every hook below is a nil check, and the engine starts no goroutine, touches no file and logs nothing extra — the poll loop is behaviourally identical to one without the seam. Distinct from the sim’s Engine.PollOnce seam (ADR-1449), which bypasses Run(); this one gates the real Run() loop. See ADR-1978.

Some live scenarios need the engine to take one decision in one specific poll — their subject is a state window (TestMergeTrainQueuedDeeperThanBatchCap: “the engine sees exactly 7 Queued members at once”). With the seam enabled the harness can hold automatic polls, build the window, then trigger exactly one complete poll and read back when it finished.

What “held” covers. Every path that starts poll() or poll-equivalent work goes through one of two hooks, and a held cycle is dropped, never queued, so the window cannot move:

Trigger Hook
Startup poll, ticker, wakeCh wake (including PollWithBackoff’s own GraphQL-recovery self-wake, which only ever sends on wakeCh) doPollCycle → pollSeam.ordinary; no select in Run() is restructured
reconcileLoop tick (LightReconcile, cache reconcile) pollSeam.runUnlessHeld around the extracted reconcileTick

No dispatch, catch-up phase, settle scan or light reconcile runs while held. Workers already dispatched by an earlier poll (item workers, a merge-train worker goroutine) keep running: hold gates new cycles only. A hold never blocks ctx.Done()/the drain — the held path returns immediately.

A trigger runs one complete poll: Run()’s own pollCycle closure (PollWithBackoff — minimum-poll-interval floor, REST hard gate, poll(), idle/GraphQL backoff bookkeeping — then the ticker reset), so backoff state is real, not reduced. PollBackoffResult.Ran is true only when poll() executed. A call that hit the 500 ms floor is retried a bounded number of times; anything else that ran no poll is reported blocked, never ran. A poll error is reported error.

Protocol (internal/pollctl; two single-writer JSON files, both replaced by atomic rename). The harness writes <bed>/.fabrik/poll-control.json (hold, hold_until, hold_gen, trigger_seq); a watcher goroutine — started only when the seam is enabled, polling every 200 ms — applies it and writes poll-control.ack.json (held, applied hold_gen, done_seq, outcome ∈ ran/error/blocked, detail). poll_seam.go’s pollMu serializes poll execution: an ordinary poll, a triggered poll and the application of a hold all take it, so two polls never overlap, a trigger during a running poll waits for it, and a hold is acknowledged only once no poll is running and none will start (reconMu does the same for reconcile ticks).

Lifecycle. The engine deletes any stale request at startup and writes a fresh released ack — its existence is the harness’s “seam is live on this bed” signal — so a hold never survives a restart (both bed launch sites, tests/gate/bed.go and tests/e2e/lifecycle.go, append pollctl.Env, so a restarted bed keeps the seam; enabled-but-released it is a normal free-running bed). A hold carries hold_until (the harness stamps now + pollctl.MaxHold, 10 min); past it the engine releases on its own, so a hard-killed harness cannot leave a bed held.

Who may use it. Holding polls starves the whole bed, so the harness helpers (HoldPolls/TriggerPoll/ReleasePolls, tests/e2e/poll_control.go over tests/e2e/pollhold) fail any test not marked exclusive in tests/e2e/registry/registry.json, and the registry check (plain go test ./...) fails any test that reaches them without being exclusive. HoldPolls registers the release with t.Cleanup. The only user is TestMergeTrainQueuedDeeperThanBatchCap; pipeline, convergence and gate tests stay free-running.

7.9 Webhook Wake Semantics: Burst Coalescence and Self-Feedback

Burst coalescence. wakeCh is a buffered channel with capacity 1. When multiple webhook events arrive in rapid succession, at most one wake is queued. The wakeChObserver uses a non-blocking send (select { case wakeCh <- struct{}{}: default: }), so additional fires while the channel is full are dropped. A burst of N simultaneous events produces at most 1 extra poll cycle. Test: TestHandleWebhookBurstCoalescence in engine/webhook_test.go.

Observer-based signaling (Phase 3-H). wakeCh is no longer signaled directly by the webhook handler. Instead, newWakeChObserver is registered on the shared store. The observer fires a non-blocking send whenever a Change includes any wakeChFlag (StatusChanged | LabelsChanged | CommentsChanged | LockChanged | LinkedPRChanged | AssigneesChanged | WorkerLifecycleChanged). The webhook handler calls deltaFn(eventType, body), which applies typed mutations to the cache store via boardcache delta functions; those Apply calls synchronously invoke all registered observers, including wakeChObserver. Changes that don’t affect dispatch eligibility (e.g., InvocationChanged, StageStateChanged, WorkerChanged from heartbeats/PID-sets) are filtered out and do not wake the poll loop. AssigneesChanged was added in issue #543; WorkerLifecycleChanged was added in Fix B (issue #544). WorkerLifecycleChanged is in wakeChFlags (for the wake channel) but excluded from cycleSetFlags (for mayNeedWorkObserver) to prevent early-return goroutine exits from bypassing the cooldown gate — see §9.2 and §9.9 (Fix B, issue #576).

Self-feedback loop (known gap). Fabrik runs as the human operator’s own GitHub account. Every API action Fabrik takes — label mutations, comment posts, status field updates, PR opens — generates webhook events from that account. These events arrive at the webhook handler and signal wakeCh (via the observer), triggering one extra poll cycle per burst of activity. The burst-coalescence guarantee bounds the damage: a stage advance producing N API actions generates at most 1 extra poll.

Sender-filter approach — considered and rejected. Suppressing wakeCh signals when sender.login matches cfg.User would eliminate these spurious wakes, but cfg.User is the human operator’s login. Filtering by it would also suppress every event the user generates (comments, label changes, PR reviews) — the most important input class. Sender filtering is only viable when Fabrik runs as a dedicated bot account separate from any human user. That is a future change; no sender filtering is currently implemented.

Backoff impact. Before the fix for issue #490, the case <-e.wakeCh: branch unconditionally reset e.idleStart and prevMultiplier = 1 before calling doPollCycle. Self-feedback events would therefore destroy the idle-backoff state on every stage advance, defeating the GraphQL budget savings webhooks provide. After the fix, those unconditional resets are removed. A webhook wake triggers an immediate poll, but backoff is preserved unless result.Active == true (see 7.7).

Wake-path rate-limit gate (#1716, ADR-1716). The idle-backoff interaction above assumes the wake reaches a poll cycle at all. It doesn’t, while rate-limit backoff is active: Run()’s case <-e.wakeCh: branch checks Engine.wakeBlockedByRateLimitBackoff() (§7.8) before calling doPollCycle(), and drops the wake entirely when it returns true — no doPollCycle() call, no ticker drain (an already-pending tick is left to fire on the next loop iteration rather than being silently discarded along with the dropped wake). This closes the production incident this section’s self-feedback loop made possible: under --webhooks, every self-feedback wake this section describes arrives while GraphQL or REST is rate-limited exactly when Fabrik’s own mutations (label changes, comments) are what triggered the exhaustion in the first place, and pre-#1716 each one walked straight through the backed-off ticker interval to trigger an immediate, immediately-rejected poll.

7.10 Stall Detection and Corrective Re-Invocation

A stalled attempt — one where the worker backgrounded a long-running command (a dev server, build, or test run) and then idled waiting for a completion notification that never arrives in a headless environment — is distinguishable from a genuinely failed one by its turn-usage trend across retries within §7.1’s cooldown loop: a stage that did real work and failed burns turns roughly consistently across attempts; a stalled one shows a clean incomplete attempt followed by a retry that completes fewer turns, still without completing. A genuinely-progressing retry does not shrink like that. This holds whether the first attempt burned its full turn budget before giving up, or — the cleanest and most common real-world shape — the worker correctly recognized it was waiting on a backgrounded command and ended its turn early, deliberately, well under budget: the latter is textually the #1142 production incident (three clean, uncapped, declining Validate attempts — 45/100, 20/100, 7/100 — none of which armed the hint before the original turn-capped-predecessor precondition was widened). See issue #1146 and ADR-1146 for the original incident (#816) that motivated this, and #1767/ADR-1146’s amended Consequences for the #1142 gap and the widened signature below.

Detection (detectAndArmStallHint, engine/item.go). Runs inside §7.1’s incomplete-outcome branch, after StageRetryIncremented and the count >= MaxRetries escalation decision (§7.2), whenever claudeRan is true and this attempt is not the one triggering escalation (see “Escalating attempt” below) — unconditionally, regardless of whether this attempt stopped cleanly or errored out. A clean flag (err == nil, or err classifies as *claudeTurnLimitError per §7.2’s turnLimited) is passed in and gates only the second half (arming), not the first (recording) — see step 4:

  1. Reads the previous attempt’s TurnsUsed/capped/clean status for this stage from the store (LastTurnsUsed, LastTurnsCapped, LastTurnsClean — zero-valued if this is the first attempt), and whether a hint has already armed this episode (StallEpisodeArmed).
  2. Computes this attempt’s own capped status: clean && usage.MaxTurns > 0 && usage.TurnsUsed >= usage.MaxTurns. Forcing capped to false whenever !clean matters independently of arming (step 4) — see the recording note below. Capped-ness is evaluated against this attempt’s own MaxTurns, so it is unaffected by fabrik:extend-turns widening the budget between attempts. LastTurnsCapped is still recorded and read (step 4), but only to phrase the stall comment correctly — it dropped out of the arming precondition itself in #1767.
  3. Records this attempt’s TurnsUsed/capped/clean status via itemstate.StageTurnUsageRecorded, overwriting the previous values, on every call — clean or not. Recording a non-clean attempt with Clean=false (regardless of its actual turn count) keeps it from going invisible to the trend: it immediately invalidates the “previous clean” precondition (step 4) for the next comparison, so a generic error sitting between two clean attempts breaks the chain at the point of the error rather than being skipped over. An earlier version of this fix gated the whole function call (recording included) behind clean, which left LastTurnsUsed/LastTurnsCapped describing the last clean attempt across an intervening error — a later clean attempt would then compare against that non-consecutive predecessor and could still misfire. See ADR-1146’s Rationale for the incident this reproduced and the regression test that guards it.
  4. If clean is true, the previous attempt was itself clean (LastTurnsClean — capped or not; #1767 dropped the “previous attempt was turn-capped” requirement, since it made the cleanest form of the stall — a worker that deliberately ends its turn early, well under budget, having recognized it’s waiting on a background job — invisible to the detector, exactly #1142’s shape), this attempt is not itself turn-capped, this attempt used more than zero turns but strictly fewer than the previous attempt, and no hint has already armed this episode (!StallEpisodeArmed), the pattern is a detected stall: applies itemstate.StallHintArmed (which also sets StallEpisodeArmed) for the stage and posts a one-time informational comment citing both turn counts, phrased according to whether the predecessor was actually turn-capped (LastTurnsCapped, from step 2) or stopped short on its own — never asserting a turn-limit hit that didn’t happen. The “not itself capped” precondition guards against a false positive from the progress-based turn-extension loop (runInvocationWithExtension, see “Turn Limit Extension” above): that loop can widen the effective budget on one dispatch (e.g. to 2×/3× stage.MaxTurns) without the widening persisting to the next, separate dispatch, so two consecutive turn-capped attempts could otherwise show a smaller absolute TurnsUsed on the second purely because its own budget reset lower, not because it is a declining, stalled retry. The clean precondition guards against a different false positive: claudeRan is true for most invocation errors (network failure, git-push error, malformed CLI output), not only clean incomplete stops, so without it a generic error whose partial usage.TurnsUsed happened to be smaller than a prior clean attempt’s would be misread as a declining, stalled retry and arm a specific, confident — but wrong — diagnosis. err == nil alone is not sufficient to compute clean, since a turn-cap exit is itself reported as a non-nil *claudeTurnLimitError (ADR-1178); turnLimited is reused from the classification already computed earlier in finalizeStageOutcome for the InvocationRecorded write, so both consult the same definition of “clean.” See ADR-1146 (as amended by #1767) for the fuller rationale.
  5. One-shot per episode via an explicit guard, not an emergent property. Before #1767, the attempt that triggered arming was (by construction, per the then-precondition) not itself capped, so step 3’s overwrite cleared the “previous capped” precondition for arming again immediately — no separate one-shot guard was needed. Widening the precondition to “previous clean” removed that emergent property: the arming attempt is clean too (arming requires clean for the current attempt), so it no longer clears its own precondition by being uncapped. StallEpisodeArmed is the explicit replacement: set by StallHintArmed the first time a hint arms in a retry episode, checked before arming again, and cleared only by StageRetryCleared (see “Cross-stage isolation and cleanup” below) — so a continued multi-attempt decline (e.g. #1142’s real 45→20→7 sequence) still arms exactly once, not on every subsequent comparison.

A detected stall still applies StageRetryIncremented and counts against MaxRetries exactly like any other non-completion — unlike §7.3’s usage-limit exemption, the stage genuinely ran and spent real tokens; the fix targets spending the existing retry budget more effectively, not exempting stalls from it. Detection itself runs whenever claudeRan is true, independent of MaxRetries: max_retries: 0 (“unlimited retries”, §7.2) is a first-class configuration, not an edge case, and is exactly the setting where a stalled stage would otherwise grind identical retries forever with no mitigation — only the surrounding retry-count/escalation bookkeeping is gated on MaxRetries > 0.

Escalating attempt: arming is skipped. detectAndArmStallHint’s comment tells the operator “the next invocation will receive a corrective hint.” If the capped-then-declining pattern is detected on the same attempt whose incremented count also reaches MaxRetries, §7.2’s escalateFailedStage() pauses the issue immediately afterward, and a human’s later clearFailedStage() applies StageRetryCleared — which wipes StallHintPending before any further invocation happens. The promised hint would never be delivered, and the comment would be actively misleading about what already occurred. finalizeStageOutcome therefore computes the escalation decision (count >= MaxRetries) first and skips the detectAndArmStallHint call entirely when this attempt is the one being escalated; no stall comment is posted and nothing is armed in that case.

Injection (consumeStallHint, engine/item.go; buildPrompt, engine/claude.go). When building InvokeOptions for the next invocation of a stage (runInvocationWithExtension), the engine checks StallHintPending for that stage. If armed, it applies itemstate.StallHintConsumed (clearing the flag) and sets InvokeOptions.CorrectiveHint to a fixed, stage-agnostic, hedged callout describing the suspected cause and suggesting the worker run any long-running command in the foreground with an explicit timeout instead of backgrounding it — the same guidance already deployed worker-side per #1077. buildPrompt prepends this text as a callout when non-empty; it is otherwise absent from the prompt. Because retries --resume the same underlying Claude session (§1’s resume semantics; see also the “Retry after a turn-cap kill” section of docs/stage-lifecycle.md for the retry-trustworthiness fix, #1081, this must not regress), the hint reads as a new turn in a conversation that already has full context of its own prior attempt — it does not need to re-explain the task, only redirect the strategy.

runInvocationWithExtension checks §7.3’s account-wide usage-limit hold gate before building InvokeOptions (and therefore before calling consumeStallHint). This ordering matters: consumeStallHint destructively clears StallHintPending the moment it runs, so if the check happened after, a dispatch that lands while usage limited would consume an armed hint without ever reaching Claude — silently discarding the one-shot corrective re-invocation for that stall episode. Gating first means a usage limited dispatch leaves the hint pending for the next dispatch that actually reaches Claude.

Cross-stage isolation and cleanup. StageRetryCleared (applied by clearFailedStage() on unpause, and by a stage’s completion) deletes LastTurnsUsed, LastTurnsCapped, LastTurnsClean, StallHintPending, and StallEpisodeArmed for that stage alongside the existing Attempts reset — so a later, unrelated incomplete run of the same stage (e.g. after fabrik:revalidate) never inherits a stale armed hint (or a stale one-shot guard) from a long-past episode.

State: In-memory only (StageState.LastTurnsUsed/LastTurnsCapped/LastTurnsClean/StallHintPending/StallEpisodeArmed, all map[string]bool/map[string]int keyed by stage name), mirroring every other per-stage StageState field — same restart-gap tradeoff already accepted by ADR-030’s turn-extension baseline.

Scope note: this implements only the “cheaper to detect” trend signal from issue #1146 — a clean incomplete attempt (capped or not, #1767) followed by a declining, incomplete one. It does not add live NDJSON tool-use parsing to detect backgrounding-then-silence within a single invocation (the issue’s other suggested signal); that remains a possible follow-up if the trend signal proves insufficient in practice.

7.11 Stale Warning Sweep

.fabrik/warnings.json (warnings/warnings.go, ADR-052) backs the TUI’s Warnings panel. Several engine-side detectors record an entry when a condition is true and clear it via warnings.Clear when the same re-checked subject’s condition resolves — but a subject that stops being checked at all (a repo leaves the board, a stage is renamed or its YAML deleted) never revisits the Clear branch, so its entry is immortal. This is the same durable-state-leak shape as the orphaned stage:*:in_progress labels (#1135) and the fabrik:claude-limit account-wide label before §7.3’s sweep (ADR-1183). Observed in production on the verveguy project (#1348): four allow_auto_merge warnings for repos that had left the board persisted indefinitely, pushing live warnings past the panel’s row cap.

warnings.ClearMissing(warningType string, present map[string]bool) ([]string, error) is the shared bulk-predicate primitive both sweeps below use. warnings.Clear(key) always calls save() even when key was never present — a caller-side loop over N possibly-stale keys would therefore write the file on every poll unless every caller independently pre-filters correctly. ClearMissing removes that footgun structurally: it loads the file once, removes every entry whose Type == warningType and whose Key subject (the portion after "<warningType>:") is absent from present, and only calls save() when at least one entry was actually removed. Entries of other Types are never touched regardless of Key, and a present-subject entry is preserved whether or not it is Dismissed — the sweep is about absent subjects, not about un-dismissing anything still around. Returns the full Key of each cleared entry for the caller to log.

allow_auto_merge (repo-keyed, per-poll). sweepStaleAllowAutoMergeWarnings (engine/allow_auto_merge_settle.go) is called from poll() immediately after seenRepos — the board repo set poll() already computes for the label-seeding path (poll.go ~line 925) — is built, before the label-seeding loop runs. It calls warnings.ClearMissing("allow_auto_merge", present) with present = seenRepos unioned with e.defaultRepo() (single-repo mode only). The union matters: checkAllowAutoMerge(e.cfg.Owner, e.cfg.Repo) fires unconditionally at Run() startup regardless of whether the board currently has any open items for that repo, but seenRepos is built purely from board.Items — without the union, a transient zero-open-items poll for the operator’s own configured repo (e.g. everything currently in Done) would durably clear a legitimate warning for it, and because checkedAutoMergeRepos never re-fires for that repo after the first startup call, the warning would not reappear until a process restart. Multi-repo mode (e.cfg.Repo == "") has no equivalent always-present repo, so defaultRepo() returning "" is a correct no-op there. Runs every poll, because seenRepos itself is rebuilt every poll (a repo leaving/rejoining the board is visible immediately). No new API calls: everything consumed is already computed this poll cycle. Each clear logs one line (tag "poll", matching the cadence of the other per-poll log lines it sits next to — not "startup", which the codebase reserves for once-per-process/once-per-repo events) naming the full key and the reason (“repo no longer on the board”).

stage_drift / undeclared_reviewers (stage-name-keyed, startup-only). stages.SweepStaleWarnings (stages/drift.go) is called once from Run(), immediately after the existing WarnStageDrift/ WarnUndeclaredReviewers calls (both the e.logFile != nil and the plain-stderr branch), with the same unfiltered e.cfg.Stages those two functions themselves consume — using the Unmanaged-excluding stageNames list built later in poll.go for SeedLabels would wrongly sweep a still-valid Unmanaged stage’s warning. It builds the current stage-name set and calls warnings.ClearMissing twice, once per Type. Startup-only (not per-poll) is correct here, unlike the repo sweep: e.cfg.Stages is fixed for the life of the process and only changes across a restart (including the in-place SIGHUP restart, which re-execs and re-runs Run() from scratch), so re-running the sweep every poll would be correct but pointless. Each clear logs one line naming the full key and the reason (“stage no longer configured”), written to the same io.Writer (os.Stderr, tee’d to fabrik.log when open) as the sibling drift/ undeclared-reviewers warnings.

version_skew: no sweep needed, by construction. checkVersionSkew’s (engine/upgrade.go) key subject is the resolved on-disk executable path, re-derived fresh via filepath.EvalSymlinks on every idle-upgrade check (poll.go), not looked up against a shrinking discovered set the way a board repo or a configured stage name is. Its Clear branch (diskVersion == running) is therefore reachable on every single evaluation, so the warning can never outlive the condition that produced it — see the doc comment on checkVersionSkew for the same reasoning in code.

source_staleness: no sweep needed, by construction — same reasoning as version_skew. checkSourceStaleness (engine/staleness.go, #1464) uses a single fixed key ("source_staleness") rather than a per-subject key, because there is exactly one source checkout per Fabrik process — unlike allow_auto_merge/stage_drift (one entry per board repo/configured stage, a set that shrinks as repos leave the board or stages are removed). Every throttled evaluation (§9.2 below) re-derives the comparison from scratch via selfupgrade.CompareDevBuild and either records or clears the same key, so its Clear branch is reachable on every single evaluation the same way checkVersionSkew’s is — the warning cannot outlive the condition that produced it, and there is no “subject went away” case to sweep for.

What belongs in the panel (#2135, ADR-2135, FR-001). An operator-level condition: a configuration, environment or engine fault that blocks or degrades work and needs a person to act. Per-item states (paused, awaiting input, escalated), transient runtime states (poll errors and rate limits — the header’s backoff status shows them) and the output of interactive one-shot subcommands do not belong. Borderline lines are dispositioned individually, with a reason, in the audit table below.

The shared path (FR-005). warnings/report.go:

Call Use
warnings.Report(w, Condition{Type, Subject, Title, Detail, FixAction?, FixParams?, Fingerprint?, LogLine?, Level?}) log line to w (nil = none) and record the entry, in one call. Safe every poll.
warnings.Resolve(key) clear one entry when its re-checked condition no longer holds.
warnings.Reconcile(w, type, []Condition) report a complete set, then ClearMissing(type, present). Only when the check ran to completion: an empty set clears the whole type, so a check that errored or returned early must not call it.

Rules: a new operator-level warning is a Report/Reconcile call, never a bare log line followed (or not) by Record; the writer is a parameter so the log-file-open and log-file-failed branches of Run() share one startupW; FixAction == "" is valid (the panel shows title and detail; the TUI fix key is then a no-op); keys are <type>:<subject> (the ClearMissing shape — Condition.KeyOverride exists only for the legacy source_staleness key). warnings/scan_test.go fails on any [startup] warn…, [warn], [upgrade] warning or WARNING: literal in engine/, cmd/, config/, stages/ that is not on its audited exception list (each entry carries a reason). How a sibling adds a condition (skill resolution, plugin loading, bare-name deprecation, fabrik upgrade migration — #2137/#2139/#2140): build a warnings.Condition with a stable Type, a Subject, a Title, a Detail ending in Fix: …, and call Report where the condition is detected and Resolve/Reconcile where the check passes (including every early-return branch that means “checked, nothing wrong”); add one test that records, repeats (one entry) and clears.

Dismissal and re-appearance. Entry.Fingerprint is the stable identity of the condition. A dismissed entry stays dismissed while its fingerprint is unchanged and reappears when it changes; an empty new fingerprint keeps the stored one (so the TUI’s [Fix failed] re-record never un-dismisses). Report defaults it to Detail. Volatile recorders pin an explicit one: source_staleness constant (commit counts), version_skew the version pair (PID), stage_drift the sorted missing-key set, undeclared_reviewers constant. Record skips the write when nothing material changed and LastSeen is under 10 minutes old (SC-003); Clear of an absent key does not write; the temp file is unique per write.

Audit (FR-004 / SC-002). Every operator-level warning in the daemon’s source, with its disposition:

Source Disposition Type / clear rule
source staleness, repo access, allow_auto_merge, version skew, stage drift, undeclared reviewers Panel (migrated onto the shared path) unchanged keys; Resolve/ClearMissing as before
unrecognised .fabrik/config.yaml key Panel (supersedes ADR-1544’s stderr-only scope) config_key:<key>; Reconcile each start
~60 [warn] … invalid; using default lines (cmd/root.go) Panel, one aggregated entry; per-setting log lines kept config_value:settings; Reconcile each start
fine-grained PAT (github_pat_) Panel config_token:fine_grained_pat; cleared when a classic token is used
.fabrik/config.yaml gitignored Panel config_ignored:config.yaml; Reconcile each start
plugin state check failed / skill refresh failed / customised plugin skips auto-refresh Panel (also still in the header) plugin_state:{check_failed,refresh_failed,customized,migration_pending} (migration_pending = fabrik upgrade has a workflow migration to apply, §7.19); Reconcile each start, except refresh_failed is swept only on a start that attempted the refresh (plain-text mode with stale skills, or nothing stale) — a TUI-mode or customized start leaves an earlier failure in place
no git credential helper (HTTPS) Panel git_credentials:https; Resolve when a helper exists or SSH/App auth applies
Claude settings file unreadable/unparseable (apiKeyHelper refusal may be bypassed) Panel api_key_helper_check:settings; Reconcile each start
FABRIK_WORKER_ENV_PASSTHROUGH entries ignored; Hookdeck env names not scrubbed Panel worker_env:{ignored,hookdeck_dropped}; Reconcile each start
CIBackstopTimeout / DrainDeadline ordering Panel timeout_ordering:{ci_backstop,drain_deadline}; Resolve when valid
log rotation failed / log file unopenable Panel log_file:{rotation,open}; Reconcile each start
App installation does not cover N managed repos Panel (aggregated) hookdeck_coverage:repos; Reconcile each check
repos with no forwarding hook (poll-only) Panel (aggregated) webhook_coverage:repos; Reconcile each check
board / status-field fetch failed at startup; GHES /meta fetch or parse failed Log only: transient, retried at the next start; the floor check is advisory –
board columns with no matching stage; holding stage without a column while merge_train is off Log only: neither blocks nor degrades work –
board drift on a terminal item Log only: per-item (FR-001) –
could not determine repo access / list the installation’s repos Log only: fail-open guess; the confirmed result is the repo_access entry –
could not resolve the bot user ID Log only: cosmetic, fail-soft –
display-field (Fabrik, Last activity, Last run) write failed Log only: per-item, retried at the next transition –
display field absent from the board / of the wrong type / lookup failed (#2218) Panel (display_field:missing, display_field:mismatch:<label>, display_field_lookup:<label>) — see §7.17 fabrik repair-board --apply for missing (none on a user-owned board or for a mismatch)
local API fails to start (#2223) Panel local_api:unavailable; names the socket path and the error and says fabrik mcp/PM plugin tools will report the daemon as not running; no FixAction (the cause is the path’s directory, its permissions or a non-socket file, which is never replaced); fingerprint pinned to the path; Resolve on every successful start, so a restart, SIGHUP re-exec or self-upgrade clears it; the existing local API unavailable log line is kept
webhook runtime failures (subprocess crash, 422s, unsupported event, orphan cleanup) Log only: runtime health is the header’s webhook ingestion status –
hookdeck coverage read failed Log only: transient; the prior entry is deliberately left –
webhook hook lookup errored for a repo (hook-coverage check failed) Log only: transient; the repo is unknown, not missing, so it stays out of webhook_coverage:repos and, when nothing else is confirmed missing, the prior entry is deliberately left –
the check for unrecognised keys itself failed Log only: nothing is known, so reconciling would wrongly clear entries –
re-exec path plugin messages Log only: Execute re-evaluates and reports plugin_state right after –
the panel store itself failing (record/clear/sweep errors) Log only –
fabrik init / upgrade / watch / repair-board output Out of scope: interactive one-shot subcommands –
per-item logf(item, "warn", …) lines; poll errors and rate limits Out of scope: per-item or transient runtime (FR-001) –

Non-goals. No TUI layout change (the panel stays flat — no severity), and this is not a general warning-expiry/TTL mechanism — only subjects that have provably gone away (absent from a known-good set already computed elsewhere) are ever cleared. See ADR-1348, ADR-2135.

7.12 Slice Budget / Turn-Cap Preemption Limit

max_retries (§7.2) is the failure counter: genuine errors, degenerate output, a completion marker with no harvestable artifact (#1782), PR-creation failures, and a clean run that never emits FABRIK_STAGE_COMPLETE. It was previously overloaded to also bound turn-cap preemptions — a large job resuming across several slices looked identical to a stage that kept genuinely failing, so a job needing more slices than max_retries was paused with stage:<X>:failed while progressing normally (#816, #1114, #1183; reported independently as the undocumented max_retries × max_turns ceiling in #1191). §3.1’s Turn Limit Extension subsection describes what a turn cap means — a resumable preemption, never a failure, per ADR-1178. This section describes how the retry accounting treats it (#1199).

The two counters:

Counter Field Config Default Counts
Failure counter StageState.Attempts MaxRetries / --max-retries / FABRIK_MAX_RETRIES 3 (0 = unlimited) Genuine errors, degenerate output, no harvestable artifact (#1782), PR-creation failures, clean run with no completion marker
Slice counter StageState.SliceRetries MaxSliceRetries / --max-slice-retries / FABRIK_MAX_SLICE_RETRIES 10 Turn-cap preemptions only (turnLimited == true, CLI subtype: "error_max_turns")

The slice counter’s default is intentionally higher than the failure counter’s: a job legitimately needing several slices is routine, while a stage that keeps genuinely failing is not — the two want different bounds, which was exactly #1191’s complaint about the single overloaded counter. MaxRetries == 0 still means “unlimited failures”; there is no equivalent “unlimited slices” setting for MaxSliceRetries — it is resolved to its default (10) whenever configured as 0 or unset, the same tier MaxRebaseCycles/MaxEnqueueCycles use (flag + env only, no config.yaml field).

Detection: finalizeStageOutcome() (engine/item.go) already computes turnLimited := errors.As(err, &turnLimitErr) for the InvocationRecorded write (§3.1/ADR-1178) — the same boolean is reused here, so no new detection work exists. This is a structural signal (the CLI’s own subtype/terminal_reason, captured since #1178), not an inference: distinguishing a preemption from a failure by a heuristic over output text would risk misclassifying either direction.

Handling: Inside the existing non-completed/non-blocked else branch — after StageAttempted, commitWIP, the branch push, and markCommentsSeenByStage have already run unconditionally on claudeRan exactly as before (ADR-1178 requires this machinery keep running for a turn-cap exit; unlike §7.3’s usage-limit exemption, which is a stage-never-ran early return, a turn-capped invocation did run and made progress) — the routing is a single branch on turnLimited:

The no-FABRIK_STAGE_COMPLETE decision (deferred by the issue to Plan): a clean run (err == nil) that never emits FABRIK_STAGE_COMPLETE counts against the failure counter, not the slice counter. turnLimited is the only structurally verified signal available; a clean run with no marker has no equivalent structural signal distinguishing it from “stopped for an unknown reason,” so it is treated the same as a genuine error rather than inferred to be a slice. This preserves today’s (already narrow) protection against a silently-non-terminating stage.

Recovery: StageRetryCleared — applied on normal stage completion and by clearFailedStage() on manual unpause — zeroes SliceRetries(stageName) alongside Attempts(stageName); the two counters share one reset point. pauseForSliceLimit() applies itemstate.EnginePaused (never stage:<X>:failed), and — unlike the four cycle-limit sites #1460 fixes (§7.2.1) — resumes via processItem()’s own gate directly, not via handleEngineUnpause: a slice-limit pause fires mid-stage, before stage:<X>:complete is ever applied, so the item is not yet stage-complete when paused and processItem()’s gate remains reachable. Without EnginePaused, processItem()’s unpause guard (wasPaused || hasFailedLabel, both false for a pure slice-limit pause) would never fire clearFailedStage(), SliceRetries would never reset, and removing fabrik:paused would be a no-op: the very next dispatch takes exactly one more slice, re-checks a counter already at MaxSliceRetries, and re-pauses immediately — the documented recovery path would not work. With EnginePaused applied, snap.PausedByEngine(stageName) is true on the next pass, so removing fabrik:paused genuinely triggers clearFailedStage() → StageRetryCleared, and the stage resumes with a fresh slice budget. clearFailedStage()’s stage:<X>:failed label removal is a harmless no-op on this path, since that label was never applied. (This section previously contrasted with pauseForRebaseCycleLimit, which at the time did not apply EnginePaused either — #1460 fixed that site too, via the separate handleEngineUnpause resume path described in §7.2.1, since a rebase-cycle-limit pause fires post-completion, unlike this section’s mid-stage slice-limit pause.)

Regression guard: a job that has already taken more turn-cap slices than MaxRetries (the old, accidental ceiling per #1191) is never paused as long as it stays within MaxSliceRetries — Attempts is untouched by any number of turn-cap preemptions.

7.13 Overseer Attention Classification and the Local Read API (#1967)

ADR-1966-a. The running engine serves a read-only, zero-GitHub-cost view of its in-memory state on a per-daemon Unix socket; fabrik mcp proxies three tools to it (see USER_GUIDE §9, “Overseer Tools: fabrik mcp”). This section is the as-built definition of the attention classification those tools report, in internal/attention (pure — no engine, store or GitHub dependency — so the push-event work reuses it). It is a product definition, pinned by internal/attention/attention_test.go.

States. needs-human (the engine is paused and only a human reply resumes it), suspended (nothing is wrong and nothing will move until a human decides — a completed stage that does not auto-advance, or work blocked only by a parked dependency; #2120, below), escalated (the engine stopped on its own account), waiting (the engine is correctly waiting on something it owns), working (a worker holds a slot and progress is recent), stalled (nothing is correctly waited on, and no progress for longer than the threshold), idle.

Precedence — first match wins:

# Condition State / code
1 Closed; or Terminal; or the cleanup stage; or an Unmanaged column; or a column with no configured stage idle (closed, done, unmanaged-column, no-stage)
2 fabrik:landing-verification-failed, or fabrik:paused with an escalation signal: StageState.PausedByEngine for the item’s stage, any cycle counter at its known positive limit (limit:<counter>), or fabrik:awaiting-runaway-alert escalated
3 fabrik:paused without an escalation signal needs-human (fabrik:awaiting-input when that label is present, else fabrik:paused)
4 Status == Validate, stage:Validate:complete, fabrik:cruise, an open unmerged linked PR, no worker needs-human, code awaiting-merge-decision (cruise never merges; ranked last in the attention view)
4b fabrik:api-key-helper-detected (the engine skips the item until a human removes apiKeyHelper from the worktree settings; no retry or deadline clears it) needs-human (fabrik:api-key-helper-detected, ranked with the paused items)
4c Suspended (#2120): a completed stage that does not auto-advance, or an item blocked only by work parked in an unmanaged column — see “Suspended” below suspended (decision-pending, parked-blocker), ranked with needs-human; never stalled
5 A worker, or a merge-train conflict resolution, is queued or running (#2121) — above every wait label and cooldown (R4) holding no slot yet: waiting, queued-for-slot, cause slot-starved, unranked; holding a slot: working, worker-in-flight, naming the kind, its trigger and (known) turns; stalled (worker-no-progress) only when progress age exceeds both the threshold and the stage’s wall-clock budget (max_wall_time, or the 15-minute Claude inactivity timeout when unset) plus a 5-minute margin — a healthy 30–45 minute run is working (its summary is flagged long-running past the threshold). While a worker is present no cooldown:* appears as a reason or a deadline
6 Any state-bearing waiting label (fabrik:awaiting-* except awaiting-input, fabrik:blocked, fabrik:rebase-needed, fabrik:bot-reprompted, fabrik:claude-limit, fabrik:tools-denied) waiting — unless a known deadline for it passed more than the stall threshold ago, then stalled (overdue); or, when it has no known deadline and no progress for longer than the stall threshold, stalled (no-progress) — except the open-ended waits fabrik:blocked, fabrik:claude-limit
6a No worker, no wait label, and a retry the engine scheduled after a killed or incomplete invocation (#2121) waiting, retry-pending / retry-waiting-for-slot, cause wall-time-kill or max-turns when that is the kill; stalled (overdue) only when the retry was due more than the threshold ago and a slot is free
6b No worker, no wait label, and a gate holding the item’s advance (#2121) waiting, gate-hold, cause gate-hold:<gate>
7 Waiting for dispatch (#2121): the item has work the engine will dispatch and no worker waiting, waiting-for-dispatch, unranked; never idle — see below
8 An unexpired cooldown, or the holding stage waiting (same deadline/stall handling as row 6; an unexpired cooldown and the holding stage are open-ended)
9 Otherwise stalled (no-progress, or stage-complete-not-advanced when the stage’s complete label is present) past the threshold; else idle

fabrik:toolchain-stale, fabrik:children-spawned and fabrik:nondefault-base-pr-noted are listed as reasons but never change the state.

Poll-stall attribution (post-pass, #2124, ADR 2124). Classify ends with attributePollStall, applied only when Input.PollStall.Active (the daemon’s poll loop is overdue, below). It never changes which row above matched; it changes what an engine-owned verdict says. A stalled result with code overdue, no-progress or stage-complete-not-advanced (rows 6, 6a and 9) becomes waiting, code waiting-for-poll, unranked, with summary and next_action waiting for the poll: blocked for N s, in step <step> (the step is omitted when unknown). A waiting result, or idle with code idle, keeps its state and code and has the same text prepended. Rows 1-4b (closed, escalated, needs-human, api-key-helper) and row 5 (a worker is running, including worker-no-progress, and a worker queued for a slot, queued-for-slot) are untouched: a poll does not unblock the first group, and a queued worker’s goroutine is already blocked on the slot semaphore, so blaming the poll for its slot wait would be wrong (#2121). With PollStall zero every classification is byte-identical to before. pauseFamilyKey maps only stalled to the stalled event, so an item whose only obstacle is the poll produces no item-level push: the cause is carried once, by the account-wide pipeline-stalled event (§7.14).

Cooldown exemption and item-wedged (#2170, ADR 2170). An active cooldown classifies as waiting, so a gate hold that re-stamps its cooldown on every poll (feedback-pending, review-blocked, merge-unsettled) never becomes stalled and produces no stalled push however long it lasts. Since #2121 a fresh recorded gate hold is additionally explained as gate-hold (row 6b) and is likewise never stalled; the cooldown handling itself is unchanged, and item-wedged (§7.14) covers that gap with its own detector, which does not consult cooldowns: a gate hold the engine keeps confirming, with no worker dispatched or queued since it began, is reported once it outlives its threshold.

Recovery grace. Input.PollRecoveredAt (the end of the last overdue episode) floors a known progress anchor, so the items the stall held back get a fresh stall window instead of flipping to stalled (and pushing) the moment the poll returns. An unknown anchor stays unknown.

Attention-view rank (lower is more urgent; items with no rank are omitted): needs-human and suspended 0 (a pause that needs an answer sorts ahead of a decision that waits), escalated 1, stalled 2, settled-at-Validate awaiting-merge-decision 3, an unaddressed red CI on the linked PR 4 (RankCIRed, #2122).

Red CI overlay (post-pass, #2122, ADR 2122). Classify ends with overlayCI, after attributePollStall. It is additional information about an item the table above has already placed, not a table row: it never changes State or Code, so every row’s precedence is unchanged. Input.CI (attention.CIFact, built by attentionCIFact from the Store alone) is non-nil only for a verdict whose HeadSHA equals the PR’s current head and whose PR is open and not merged. The overlay applies only when the verdict is red and not stale (attentionCIFact measures staleness against one poll interval times Engine.prFactsRotation, the number of polls settlePRFacts needs to rotate through all candidates at its per-poll check budget, so a red is not dropped between re-reads when more PRs can spend the budget than it covers (merged/closed PRs and greens read live within the recheck window do not count); the status tool’s own ci_stale still uses one interval), the item is in a managed stage and open, it does not carry fabrik:awaiting-ci (a CI-gated wait is untouched), and the table gave waiting, working, idle or stalled — needs-human, escalated and suspended keep their classification. It then (1) adds the reason ci:red, (2) prefixes the summary with CI is red on <failing checks>; ((a required status context failed) when no check run failed), (3) sets Result.CIFailing, and (4) either — with a worker or a merge-train conflict resolution — appends a CI-fix reinvoke is addressing it (kind ci-fix-reinvoke) or <kind> is running (every other kind, which is not claimed to address CI) and leaves the rank alone, or — with none, and the PR not a draft — sets the rank to RankCIRed (4) when the result had none or a lower urgency. A draft PR gets the text marked (draft PR) and no rank (Implement opens drafts early, so red there is routine). Green, pending, none, unknown and (beyond one rotation) stale verdicts add nothing. Reporting a red verdict does not make the engine dispatch anything.

Progress anchor. The latest of StatusEnteredAt, EligibilityChangedAt, every stage’s LastAttemptAt, every LabelAppliedAt entry, LinkedPR.LastCIProgressAt and Worker.StartedAt. EligibilityChangedAt (#2120) is stamped by the store whenever a before/after label diff of an item it already held touches fabrik:cruise, fabrik:yolo, fabrik:blocked, fabrik:paused or a stage:*:complete label (a population or baseline never stamps; unknown after a restart), so an autonomy change, a resolved decision or a cleared blocker restarts the stall clock: stalled measures time since the item could last have moved on its own, never time spent waiting for a human. It deliberately excludes Worker.LastSignAt (a ticker heartbeat from the engine’s own goroutine, fresh even while Claude is hung) and UpdatedAt (bumped by Fabrik’s own writes). An anchor of zero is unknown and never stalls an item. Stall threshold: --stall-threshold / FABRIK_STALL_THRESHOLD (minutes, default 30); fabrik_board accepts a per-call override; the effective value is echoed in every response. Zero disables stalling.

Restart caveat. StatusEnteredAt is reset to now on every store reset (ADR-1833), so right after a restart every item’s anchor is “just now”: nothing classifies stalled until one threshold has elapsed since the daemon started, and “time in column” means “since this daemon first saw it” (responses carry status_entered_basis and daemon uptime). This is a deliberate fail-safe for the stall decision. It no longer applies to a pending decision (#2120): that is anchored on a GitHub time, listed from the first served board, and never stalled (see “Suspended”).

Next action and deadlines. The result lists every deadline the engine will act on: unexpired cooldowns (cooldown:<reason>; only an entry still in the future is a deadline; an expired one is either history or an expiry poll admission has not yet acted on — the engine deletes it once it has, #2096 — and is neither a deadline nor a reason); the CI liveness stall (LinkedPR.LastCIProgressAt + CIWaitTimeout); the CI backstop (LabelAppliedAt[awaiting-ci] + CIBackstopTimeout); the review wait (LabelAppliedAt[awaiting-review] + ReviewWaitTimeout); the bot re-prompt escalation (LabelAppliedAt[bot-reprompted] + ReviewWaitTimeout); the end of the account-wide Claude usage-limit hold. next.at is the earliest known deadline. A deadline anchored on a value the cache does not hold — LabelAppliedAt is record-on-write and empty after a restart, and the API may not use the live FetchLabelAppliedAt fallback — is listed with an unknown time, and next.at is unknown when none is known; it is never “now”. Next-action honesty (#2120 R5): a state only a human or an external event moves — suspended, needs-human, escalated, awaiting-merge-decision — reports no deadline and no next.at, and a fabrik:blocked wait drops its cooldown deadlines, because a dispatch cooldown’s expiry does not change what happens to such an item (for #1342 the cooldown had been presented as its next action). The periodic-re-eval safety-net cooldown is never a deadline anywhere: it can only re-check, so it stays a reason and never reads “re-dispatched at …”.

Suspended (#2120, ADR 2120). A derived state — never a label — for the most common human decision point, which the pre-#2120 classifier called stalled after daemon uptime, or waiting. It is suspended when the item is a managed, non-holding stage and nothing the engine owns is going on: no worker, no fabrik:paused, no in-flight marker (fabrik:editing, fabrik:locked:*, stage:*:in_progress), and no state-bearing wait label. Cooldowns never mask it (they can only mean “re-check later”). Two cases:

Honest age. pending.since is the time the deciding fact became true and pending.since_basis says how far to trust it. For a completed stage it is the stage:<X>:complete label’s GitHub time (github-label); until the poll-side recorder has that, the cached Fabrik stage comment’s CreatedAt stands in (stage-comment, an estimate: the comment is posted before the label and a rework or step back can leave an older completion’s comment as the latest), else the daemon’s first sight (first-seen). A parked-blocker wait has two facts, the item’s fabrik:blocked label and the blocker being parked, and starts at the later of them: blocker-parked is reported only when the label’s GitHub time is known and the daemon watched every direct blocker move into its parking column (ItemState.StatusMovedAt, stamped only for a move between two known columns of an item the store already held; zero after a restart). A 30-day-old label says nothing about a blocker parked minutes ago, so otherwise — an unwatched move, or a transitive chain whose intermediate edges nobody observes — the basis is first-seen. A later EligibilityChangedAt wins over all of these (eligibility-change: the age starts at the change). github-label, eligibility-change and blocker-parked are exact; stage-comment and first-seen are not: then age_seconds is unknown, at_least_age_seconds carries the value and the summary reads “waiting at least since …” (first-seen) or “waiting since about …” (stage-comment); a daemon-observed age is never presented as the wait. The store stamps (EligibilityChangedAt, StatusMovedAt) come from the store clock (Store.SetClock, set to the engine clock by Engine.SetClock), so a sim clock drives them and the classifier together. The label time is read once per item per restart by the poll-side recorder (engine/decision_anchor.go, settleDecisionAnchors: at most 8 labelAppliedAt reads per poll, only for items the classifier would call suspended, a miss or error retried after 10 minutes) and written into the store’s LabelAppliedAt map; engine/localapi*.go and engine/channel_*.go still make no GitHub call.

Options. A decision-bearing result carries options — an ordered list of {action, tool}, tool empty when none performs it (merge PR #N is the human’s own merge). Decision pending: fabrik_advance, fabrik_set_autonomy cruise, fabrik_set_autonomy yolo. Parked blocker: one fabrik_promote per blocker, then “remove the blocking dependency”. Settled at Validate under cruise: merge PR #N, fabrik_set_autonomy yolo. Paused/needs-human/escalated: the resume method the engine’s own pause comment gives (R-options-2). resumeMethodOf walks back from the newest Fabrik comment to the newest state-bearing one and skips comments that say nothing about a pause, so an overseer audit comment (🏭 **Fabrik — overseer action:) or another informational Fabrik comment posted after the pause never shadows it: a comment that tells the operator to remove fabrik:paused (a daemon clean stop, a CI/spawn/merge-gate pause) gives “remove the fabrik:paused label”; one that asks for a reply or comment “to resume” gives “a human comment”; a later stage comment (an awaiting-input question) supersedes an older pause comment and, with no instruction of its own, leaves the method unknown — a plain awaiting-input question then reads “a human comment”. When the cache cannot say, the text points at the pause comment and claims neither.

Worker kinds, queued and in use (#2121, ADR 2121). The worker record (itemstate.WorkerHandle) carries a Kind — stage-run, comment-review, review-reinvoke, ci-fix-reinvoke, rebase-reinvoke or merge-train-conflict — a Trigger (the comment or review-body ids the worker is addressing, comma separated), a Queued flag with QueuedSince, and the live TurnsUsed/TurnsKnown. Kind is set where the worker is created (dispatchCandidates, dispatchReinvoke via reinvokeOpts.kind) or, for a comment review, by processCommentsClassified, which marks the stage placeholder comment-review for the pass and restores the prior kind afterwards; it never overrides a reinvoke kind. The trigger of a reinvoke is applied after its build step, once the worktree is ready and the ids are known. A kind the engine cannot determine is absent and the item is still working. A merge-train conflict resolution is not a Worker (the stale-worker detector, the shutdown pause set, the janitor and the overseer refusals all treat Worker != nil as a stage worker): it is a separate ItemState.ConflictResolution{Queued, Since} on the member, set around acquireTrainSlotEp and InvokeForComments in resolveConflictWithClaude and cleared in a defer; the classifier treats it like a worker.

Queued is not running. A handle stays non-nil while the worker waits for a slot — about a dozen Worker != nil guards (double-dispatch, shutdown pause, janitor, overseer refusals) mean “claimed” — with Queued set. dispatchCandidates takes its slot before WorkerEntered (ADR 2114); dispatchReinvoke used to apply it on the poll goroutine before its goroutine blocked on e.sem (F16) and now applies a queued WorkerEntered, then WorkerSlotAcquired immediately after the slot is taken: that clears Queued and sets StartedAt and LastSignAt to the acquisition time (WorkerChanged only, never WorkerLifecycleChanged, so it wakes nothing). TestWorkerEnteredSitesAreAudited pins the audited set of WorkerEntered sites, so a new site must be reviewed for this ordering. The stale-worker detector skips a queued handle (it holds no process or lock to be stale about; the goroutine’s deferred WorkerExited ends it). Status reports worker.queued, worker.queued_since and worker.in_flight: false until the slot is held; worker.started_at is the slot-acquired time. fabrik_health.workers.in_use is the number of item workers that hold a slot, plus any merge-train conflict resolution holding a general-pool slot (ConflictResolution.GeneralSlot, recorded by acquireTrainSlotIn), clamped to max_concurrent; queued is the number waiting. A resolution on the reserved merge-train slot is reported separately (train_in_use), not in in_use.

Live turns. InvokeOptions.OnTurn(turns, budget) is called by turnCountingWriter once with zero turns when an invocation starts and after every logical turn; liveTurnRecorder applies a WorkerTurnsRecorded for each call (a single in-memory field update that wakes nothing). The zero-turn call forgets the previous invocation’s count (several invocations can run under one worker handle); a final OnTurn(0, 0) when runClaude returns forgets the count and budget again, because the handle outlives the Claude process (result handling, pushes, later work) and must not report a finished invocation’s turns as live and records that invocation’s effective budget — the stage’s max_turns, a comment review’s own limit, or a fabrik:extend-turns/extension budget — as WorkerHandle.TurnsMax. worker.turns_used is unknown until the first turn of the current invocation is recorded, never zero, worker.max_turns is that invocation’s budget (unknown before it starts), and the working summary reads (turn n of max) when known.

Waiting for dispatch (#2121 R3). Input.Dispatchable (engine/attention_workers.go, cache-only, fails closed) is true when the item is an open, non-PR card in a managed stage, its stage:<Status>:complete is absent, nothing gates it (pause, fabrik:blocked with an open blocker, any fabrik:awaiting-*, fabrik:editing/reworking/lock/in-progress markers, api-key-helper-detected, skill-unresolved, claude-limit, an engine-paused stage, a held Claude usage limit, a cycle counter at its limit, a gate cooldown) and it has no worker, conflict resolution or gate hold. It is deliberately not conditioned on autonomy: a stage is dispatched whatever the autonomy label, which only governs advancing. The reason is exactly one of: all N of N worker slots are busy (cause slot-starved; the count is len/cap of e.sem), a cooldown that will really re-dispatch it (retry-backoff, sentinel-live — stated with its expiry), or the next poll. Gate cooldowns (feedback-pending, review-blocked, merge-unsettled), dep-blocked and periodic-re-eval are never offered as the reason. Only an unexplained gap (slots free, no real cooldown, past the stall threshold) falls through to stalled. A sim test pins the predicate against the real dispatcher.

Retry and gate-hold reasons (#2121). The facts the channel anomaly detectors held in memory only when a hub ran now also reach the Store, written from the same single hook points before the hub nil check: ItemState.PendingRetry{Stage, Cause, Due} from emitRetryScheduled (only on the paths where the retry will really happen, so an exhausted or escalated retry is never one; cleared on dispatch, completion, pause and close) and ItemState.GateHold{Gate, Reason, IDs, Since, ConfirmedAt} from noteGateHold/noteGateClear. Each gate ends its own hold the moment it stops holding — the feedback gate when nothing is unprocessed, the review gate when it neither blocks nor times out, the merge gate once mergeability settles, the CI gate once CI is no longer pending — so status stops naming a hold that has ended; a hold a gate site stopped confirming without clearing (a path that never reaches those points) is still ignored after wedgeLapseFor. A killed item is waiting with a summary naming the kill and the retry; the leftover stage:*:in_progress and fabrik:locked:* labels get the reason “left over from the killed run until the retry runs; not a live worker”. A held item names the gate and the pending feedback kind and id(s) (“+N more” past three). Neither is stalled.

Cause vocabulary (#2121). Result.Cause (attention.cause on the wire, omitted when empty) says why the item is in the reported state, from a fixed vocabulary, one cause per result, absent rather than guessed. Each is established by an explicit recorded fact:

Cause Establishing input
wall-time-kill, max-turns PendingRetry.Cause (the retry rows). inactivity-kill, api-error and the other engine retry causes appear in the summary but yield no cause
slot-starved a queued worker or conflict resolution, or waiting for dispatch with every slot busy
gate-hold:<gate> a fresh GateHold; <gate> is feedback, review, ci or merge
daemon-stop the item is paused and the newest state-bearing Fabrik comment carries the <!-- fabrik:pause-cause=daemon-stop --> marker that runShutdownPause adds, and the store has not watched an eligibility change (EligibilityChangedAt) more than two minutes after that comment — a resume followed by a bare re-pause that wrote no Fabrik comment would otherwise inherit the old marker. An unknown (zero) stamp proves nothing and keeps the cause. A bare stale in_progress label, a crash or a later pause comment yields no cause
rebase-conflict-after-landing fabrik:rebase-needed is present and ItemState.RebaseCause records after-landing, written only by invalidateConflictingQueued (a merge-tree conflict found after a train landing) and cleared with the label. The ErrNotMergeable path writes none

Label meanings (#2121 R5). Every state-bearing fabrik:* label has a one-line meaning in the reasons (attention.MeaningOf), including fabrik:editing, fabrik:reworking:<Stage>, fabrik:locked:<id>, fabrik:credited-pr:<N>, fabrik:spawned-child:<i>:<n>, the autonomy and operator-command labels. Families with a variable part resolve by prefix. internal/attention/labels_scan_test.go scans the engine, github, cmd and internal sources for fabrik:* label literals and fails for any with no meaning that is not on its explicit exemption list (each entry carries a reason); a non-vacuity test proves it fails for a fake label.

Zero GitHub cost. engine/localapi*.go reads the Store only through Peek (no FallbackFetcher) and Scan, and never the engine’s GitHub clients, Store.Get or labelAppliedAt — enforced by TestLocalAPISourceDoesNotReachGitHub and a nil-client test. The poll loop’s single-goroutine fields reach the API only through the mutex-guarded daemonHealth mirror.

PR facts and CI verdict (#2122, ADR 2122). The status tool’s pr block is rendered from two poll-side records on LinkedPRState, separate from the fields the gates read (State, Draft, Merged, Mergeable, MergeableState, CheckRuns), so writing them cannot change a gate. Facts (PRFacts: state, draft, merged, mergeable, mergeable state, the head they were read for, ObservedAt) is written by PRFactsObserved from two places: the per-poll board probe (runProbeAndDeepFetch applies the probe query’s linked-PR scalars state isDraft merged mergeable mergeStateStatus, which add no connection to the query, as FromProbe), and — for an item whose PR the probe cannot see (a base:<branch> PR, which closedByPullRequestsReferences omits) or whose facts the probe did not stamp this poll — a bounded REST FetchPRDetails in settlePRFacts (prFactsRESTPerPoll = 5 per poll). CIVerdict (CIVerdictRecord: verdict, the head it was computed for, failing check names, a fingerprint of the latest-per-name runs, ObservedAt, LiveReadAt) is written by CIVerdictObserved from settlePRFacts (engine/pr_facts.go, poll goroutine, after settleDecisionAnchors): for each managed open item with a linked open PR, least recently attempted first (in-memory attempt stamp; a PR known merged/closed is not re-read by REST inside the recheck window), it reads the head’s runs live through e.client.FetchCheckRuns — not the cache, whose READY set is trusted for ever and is never refreshed on a poll-only deployment; nothing is written to the Store from that read, so the gates see what they saw before — and classifies them with ciStatusVerdictFor, i.e. classifyLandingCI unmodified, so the overseer view and the gates cannot disagree about a head. A green verdict is restamped without a read only for prFactsGreenRecheck (5 minutes) after the last live read (CIVerdictRecord.LiveReadAt) — whether or not the Store’s cached runs are populated (they are empty on a poll-only deployment; when a webhook or gate filled them they must still be a READY set matching the verdict’s fingerprint) — then re-read, so a check rerun that fails on the same head surfaces; at most prFactsCheckBudgetPerPoll (8) evaluations that can cost a GitHub read run per poll, and an item past the budget is not refreshed and ages into stale. A failed read keeps the previous record unstamped. Both mutations return PRFactsChanged, which is not in wakeChFlags/cycleSetFlags, never call SelfWriteObserved, never move LastSeenPRUpdatedAt/LinkedPRUpdatedAt and never touch LastCIProgressAt; a verdict whose head is not the PR’s current head is dropped on write (and when the engine-recorded head and the head the facts were read for are both known and disagree there is no agreed head: AgreedHead() is empty, nothing is read or evaluated, any verdict is dropped and the API reports unknown until the two agree), PRHeadSHAUpdated clears it on a genuine head change (next to the CheckRuns prune), a merged or closed PR clears it, and the renderer shows a verdict only when CIHeadSHA equals head_sha. prInfo reports each group with facts_as_of/ci_as_of and facts_stale/ci_stale (older than one poll interval — the larger of the configured interval and the poll ticker’s effective period from the health mirror, so a back-off is not stale; unknown when the interval is not known). A fact never observed is unknown, not stale. The old ciVerdict(runs) summariser, which disagreed with the gates over reruns and cancelled/stale conclusions, was deleted. Seam: SetPRFactsRefreshDisabledForTest; sim twin tests/sim/pr_facts_test.go.

Poll liveness in fabrik_health (#2124, ADR 2124). poll() — the one function both PollWithBackoff (production) and PollOnce (the sim seam) run — records its lifecycle in daemonHealth: beginPoll at entry and, in a defer so an error or early return cannot leave it stuck, endPoll (in_progress, current_started_at, last_completed_at — completed means returned, with or without error; last_success_at keeps the success meaning), plus setPollStep at about ten phase boundaries (layer 2 status sweep, probe and deep fetch, fetching the project board, selecting deep-fetch candidates, settle scans, dispatch, merge train, post-dispatch settle scans, cleanup). A step is a phase name only; #2114 made worker-slot acquisition non-blocking, so there is no waiting for a worker slot step and nothing is invented. endPoll publishes next_due_at = end + max(configured interval, the period the poll ticker was last set to) as a fallback — an errored poll does not reset the ticker, so the next poll is due after the engine’s last chosen back-off period (and backoff_reason keeps that period’s reason), not after the configured interval, and one transient error in a deeply idle daemon is not reported as a missed poll; PollWithBackoff then overrides it on success with the engine’s own value — the effective interval after idle, GraphQL and webhook-cap back-off, or the REST reset on the hard-gate path — with backoff_reason idle, graphql or rest.

Overdue verdict (computePollVerdict, pure, engine/pipeline_liveness.go), with K = pipelineOverdueFactor = 2 (one constant, not an operator setting) and the configured interval: a poll in progress is overdue when now - started > K × interval (poll-in-progress); with no poll in progress it is overdue when now > next_due_at + K × interval (no-poll-started), where next_due_at falls back to the daemon start before the first poll. Measuring from the engine’s own next_due_at, not the configured interval, is what keeps an idle daemon (up to 60 min between polls with healthy webhooks) and a REST pause (up to ~1 h) from being reported as stalled. An unknown interval (poll_seconds unset, as in test engines) or baseline makes the verdict unknown, never healthy and never overdue. SetClock re-stamps the daemon start on the injected clock, and the infra-abandon cooldown reads e.now(), so tests control both.

Gauge ages. graphql_backoff.as_of is set only by setGraphQLBackoff, and rest_paused_as_of by setRestPaused, which the poll now calls on every attempt that passes the floor (a refresh, not only a transition). Neither moves with time alone; both are "unknown" until first refreshed.

Merge-train partitions. partitionStates joins the Store’s Queued rows (trainRows), the durable TrainRunPhases(), the worker markers and a non-mutating infraCooldownPeek (infraCooldownRemaining deletes an elapsed entry on read; the read path must not). A partition has an active run when it has an open run or an in-flight worker (the async driver lets the worker goroutine exit with a run still open, so worker_in_flight alone under-reports). With members and no active run computePartitionLiveness (pure; health and the deriver share it) gives idle_reason: poll-overdue, then infra-abandon-cooldown, else unknown — other holds (overlap deferral, the runaway guard, a base contradiction) are deliberately not duplicated in the read path. The partition stall condition is: merge_train: on, members, no active run, not in the infra cooldown, not inside a deliberate poll back-off, a known oldest_queued_since, and now - oldest_queued_since > K × interval. oldest_queued_since is the daemon-observed StatusEnteredAt, so it restarts with the daemon and is unknown without the board cache.

Read-surface polish (#2127, ADR 2127). Additive wire fields over state the daemon already holds; nothing reads GitHub and an input the cache lacks is unknown, never false or 0. fabrik_board: every result carries matched (items that passed the filter); the flow view’s excluded.closed counts matched closed items it leaves out, so sum(flow.count) + excluded.closed == matched; an attention view always serialises attention (BoardResult.MarshalJSON, [] when none), the flow view never does. fabrik_status: place.other_labels (labels that are neither stage: nor fabrik:), url derived from the repo and number (Engine.webURL, honouring ghes_host) when the cache holds none, no <column>:<kind> limit rows for an unmanaged or holding current column (attentionStageKind), last_invocation.figure_basis (duration and turns per-invocation, tokens and cost session-cumulative, as the CLI reports them; InvocationRecorded accounting is untouched) and pr.url / pr.reviewers. pr.reviewers and the validate-settled reviewers meta come from one pure builder, reviewerVerdicts (engine/review_verdict.go), evaluated at LinkedPRState.AgreedHead() (#2122): per reviewer’s latest non-dismissed review, at_head (review commit_id equals the head; unknown when either is missing - the pull_request_review webhook now carries commit_id), inline_at_head (unresolved thread comments by the reviewer with CommitOID == head; a thread comment has no review id, so author plus commit is the link), outstanding (unresolved non-outdated threads plus the latest review body when actionable and not in the durable review-ids-addressed marker, regardless of the processed/rocket mark), clean_pass (true only when at_head and outstanding == 0; false for an older head and for a CHANGES_REQUESTED review; unknown when at_head is unknown) and unaddressed (the feedback gate’s own cores, filterReviewThreadComments + dropOutdatedThreadComments + reviewBodyCandidates, per author). outstanding and unaddressed differ: a non-relaxed validate-settled is only announced when nothing is unaddressed, so outstanding is what distinguishes a clean pass from findings already worked through but never resolved. The builder lives outside the scanned globs and is itself pinned GitHub-free by TestReadPolishVerdictBuilderReachesNoGitHub. SetReadPolishDisabledForTest("R1".."R8") neutralises each requirement for the non-vacuity twins in engine/read_polish_test.go.

The situation view and fabrik_situation (#2171, ADR 2171). A fourth read method, situation, beside status / board / health. It is served through an optional interface (localapi.SituationBackend, set on Server.Situation, advertised by hello only when set; an older daemon answers unknown_method, which the shim reports as “this daemon predates fabrik_situation”), the same pattern as Streamer and Actor, so the three-method Backend is unchanged. Params {subscriber?, push?, force?}; result Envelope (as_of) plus view, plus pushed / push_note when a push was asked for. engine/localapi_situation.go holds the one builder, buildSituation(scope). It is a view over the existing classification, not a second classifier: each open, labelled item is run through attention.Classify (via attentionInput) and placed by its (State, Code) - Suspended → decisions; code awaiting-merge-decision → merge_decisions (review verdict reviewSummary, CI verdict cachedCIVerdict, the cores validate-settled uses); NeedsHuman / Escalated → human_input (with ResumeVia); Stalled → stalled; Waiting / Idle with a gate → gates. A gate is an unannounced anomalyState.holds entry (gate, reason, firstHeld), else the fabrik:awaiting-ci / fabrik:awaiting-review label wait with the label’s applied time when the daemon holds one; the hold start is "unknown", never “now”, when it began before this run. Because placement is a function of the one classification, an item is in at most one item section; a hold already announced as item-wedged is reported under anomalies (not again under gates). Items whose labels are not yet read are counted in unclassified_items, never placed, and stop nothing_needs_you. The account sections come from Health’s own sources (Health() is called for poll liveness, worker slots, the Claude hold and partitionStates); anomalies are the poll / partition stall verdicts, openAnomalyFacts (the open retry-starved / item-wedged episodes anomalyCatchUpEvents reports, now sharing that accessor) and cache-divergence - a point event, so anomalyState.recent keeps a bounded (50) memory of divergences inside cacheDivergenceWindow for it. Ordering is attention rank, then repo and number (gates: longest held first, unknown last); every section is cut at 10 before rendering with an exact omitted. The builder reads the Store (Scan, never nested with Peek), the health mirror and anom.mu only: localapi_situation.go and channel_situation.go match the localapi*.go / channel_*.go scan globs, so TestLocalAPISourceDoesNotReachGitHub and TestChannelEventsSourceDoesNotReachGitHub cover them, and TestSituationSourcesAreInsideTheGitHubFreeScans pins that. The tool is unscoped (account-wide); a scope narrows the item sections and item-scoped anomalies for a digest.

Local API lifecycle. Run() binds .fabrik/state/fabrik.sock (mode 0600) after the instance lock (§7.4) is held and after the webhook manager is assigned, before the first poll. A socket file already there is stale — replaced, but only if it is a socket. The socket is closed and unlinked when Run() returns and before the SIGHUP re-exec; the self-upgrade exec, a crash and an exec failure leave a file that the next start replaces. A bind failure only disables the API: it is logged (local API unavailable (continuing without it)) and recorded as the local_api:unavailable panel entry (§7.11), which the next successful start resolves.

7.14 Channel Events: Proactive Push to fabrik mcp Sessions (#1968)

ADR-1966-b. Where §7.13’s read API answers a question, channel events tell a session that a question is due. The daemon derives events from transitions it already observes, routes them to named subscribers through a persisted hub, and fabrik mcp writes them to the session as Claude Code Channels notifications/claude/channel messages (a research preview — see USER_GUIDE §9, “Proactive push”). Events observe; they never cause or alter an engine decision (R10), and they cost no GitHub call (R9).

Sources. (1) The itemstate observer: every non-no-op Store.Apply delivers a Change whose LabelDeltas list each label the mutation actually added or removed (a before/after diff, computed only for an item whose labels the store already knew (ItemState.LabelsKnown) — the first label-bearing population of an item, whichever fetch carries it, is a baseline, not a change; see “Baseline, arrivals, column moves and closes” below), plus its Origin (LocalLabel* write-through = the engine; webhook; anything else = a poll/reconcile) and, for webhooks, the sender.login and whether the engine’s echo registry matched. (2) A small number of nil-safe, void hooks at engine transition points that have no store signal of their own. (3) A 30 s ticker for the time-driven events. The observer only appends to a queue; one consumer goroutine owns all per-item memory and is the only publisher, which is what serialises ordering and dedup.

One event per actual change. The engine’s LocalLabelAdded write-through changes the store; the webhook echo that follows is an idempotent no-op that never reaches an observer. A reconcile that re-observes an unchanged label is a no-op diff. A reconcile that merely declines to revert a recent engine write (D.4, #2159) changes nothing in the store and so emits no event — previously it reverted the label and emitted a false unknown-actor event. Residual race: a webhook delivered between the REST call returning and the write-through is applied first, so its event is attributed by the webhook (actor = fabrik when the echo registry matched, else from the sender) — never duplicated.

Actor of a label event (engine/channel_events.go labelAttribute, #2125, ADR 2125), resolved in this order: overseer (the write was made by an overseer action — the requester’s sanitised subscriber name rides the write-through mutation and the echo-registry entry; the event also carries via=<subscriber>, and this takes precedence over the App-sender check), fabrik (the engine’s own autonomous write-through, a webhook that matched the engine’s echo registry with no requester, or — under GitHub App auth — a sender equal to the App’s bot login), bot (another [bot] sender), human (any other sender, including the operator’s own login under PAT auth, which cannot be told apart from Fabrik’s), unknown (no webhook sender: first observed by a poll, reconcile, deep fetch or probe, so nobody told us who did it). human and bot events carry the sender’s login as actor_login; overseer events carry via; fabrik and unknown carry neither. A requester is believed only from the engine’s own write or an echo-registry match (RegisterEchoVia/MatchEchoVia), never from an unmatched webhook, and a failed write registers nothing. Which unknown paths resolve: only the webhook sender (already on the delta). The events-API actor behind FetchLabelAppliedAt/FetchLabelRemovedAt is not used (it arrives after the event is built, and the labels those reads cover are almost all engine-applied), and the board/deep-fetch queries carry no timeline actors, so a human’s change first seen by polling stays unknown; no GitHub read is added for attribution.

Where each event comes from:

Event Produced by Notes
validate-settled runCatchUpPhase2 (the settle point — reached only when no Phase 1 handler claimed the item, i.e. dependencies, review gate, auto-merge convergence and the CI/merge gates are clear; §6.4, §6.6.6) via noteValidateSettled, called before the autonomy gate so cruise is covered; and attemptMergeOnValidate after its dependency, review-thread, review and feedback gates passed (via noteValidateLanding) — the settle point for yolo, whose landing happens inside that call and never reaches Phase 2. The yolo event is captured before the landing call but announced only if the merge, enqueue or move to Queued succeeded; noteValidateSettled at the top of Phase 2 announces only items that wait for a human, so a landing that fails (a conflict, an enqueue error) has announced nothing Never at FABRIK_STAGE_COMPLETE while fabrik:awaiting-ci is present (the complete label is deferred and the predicate requires it). A cache-only predicate confirms: Validate, open linked PR, stage:Validate:complete, none of awaiting-ci/awaiting-review/bot-reprompted/paused/blocked/rebase-needed/revalidate, no open blocker, no worker, and no unprocessed feedback (the feedback gate’s own pure cores, §2.2, read from the store; TestCachedFeedbackMatchesLiveGate pins the agreement). The event is captured on the calling goroutine, because a yolo item can be merged and moved to Done in the same pass. Episodes: one event per episode (meta.options — merge PR #N, fabrik_set_autonomy yolo — accompanies next=waiting-for-human); an episode ends when the complete label is removed (fabrik:revalidate), fabrik:rebase-needed is applied (the settle after the rebase is announced afresh), the item is closed or leaves Validate, or the PR head moves after Validate completed, or (non-yolo episodes) a worker is dispatched or unprocessed feedback arrives — each of those that the engine caused is announced with one settlement-withdrawn; the episode counter and dedup key persist, so a restart neither replays an announced episode nor loses an unannounced one. meta.next: waiting-for-human (cruise — the raw cruise label wins over yolo — or no autonomy), auto-merge (yolo, merge train off, or fabrik:auto-merge-enabled), merge-train (yolo, merge_train: on). meta.ci is read from cached check runs and is unknown when the cache holds none (the cache fills from check_run webhooks, so absence is not “no checks”); meta.ci_gate says whether the engine’s own CI gate was configured and cleared. meta.reviewers is a compact JSON array of per-reviewer verdicts at the PR head (§7.13 “Read-surface polish”); meta.ci_rerun (#2127 R7) names the failed checks the stage wait_for_ci gate re-ran (#2072 R1/R2) on this same head, and is present only when meta.ci is green - it reads Engine.flakeRerunChecks, the in-memory prFlakeState.rerunChecks (recorded only for a re-run actually made, never for a refused one), so it is absent after a daemon restart or startupStateTTL without a gate touch and for the merge train’s own (#2052) re-runs
decision-pending engine/channel_decision.go: the consumer goroutine, at first sight, on a debounced re-evaluation and on the ticker, when the classifier (§7.13) reports an item suspended and no episode is open (#2120) one event per episode. Episode counters dp:<ref>/dpclosed:<ref> and dedup key dp:<ref>:<n> persist in the hub like validate-settled’s, so a restart neither replays an announced episode nor loses an unannounced one. Unlike the pause family it is announced at first sight (that is what makes a pending decision visible after a restart). meta: column, completed_stage, next_stage (decision) or blockers (parked), decision (advance/unblock), options (a JSON list of {action, tool}), since, since_basis, autonomy; the latest Fabrik comment link is always attached. An item whose GitHub time is not yet known waits decisionAnchorGrace (3 min) for the recorder before it is announced with since_basis=first-seen. The episode ends when the item stops being suspended
settlement-withdrawn the same consumer, when a validate-settled or decision-pending episode ends because the engine took the item back (#2120 R-unsettle) meta.withdrew (the event type whose announcement no longer holds) and meta.reason, one of rebase-needed (fabrik:rebase-needed is present, or was applied since the last evaluation), ci-on-new-head (the PR head moved after Validate completed), worker-dispatched (a worker or fabrik:reworking:* marker) and new-feedback (unprocessed feedback, read from the cache); deduplicated per episode (swvs:<ref>:<n>, swdp:<ref>:<n>). A human acting — autonomy set, the item advanced or merged, closed, fabrik:revalidate, a blocker cleared, paused — is silent. Once the item resettles, a fresh episode announces itself afresh (R-unsettle-3). Immediate (never digested)
label-applied, label-removed the observer’s LabelDeltas label, actor, stage, and unverified=true when the change is one a cache-divergence was raised for (#2170; the same marker is set on every other event built from such a change). Label patterns filter only these
item-added the observer: a Change.Created mutation after the board was populated (Engine.boardPopulated, set when the first poll succeeds) registers a pending arrival; the consumer publishes it from evaluateItem once the snapshot has LabelsKnown, a board Status in a column Fabrik knows (a configured stage of any kind, or the legacy Backlog parking column per stages.IsParkingColumn), is open and is not a PR card (announceArrival, engine/channel_lifecycle.go) title, column, labels, autonomy (cruise wins over yolo; none), milestone (title, none, or absent when unknown), author, url. Event.Labels carries the arrival labels: a subscription’s labels include patterns match when any of them matches, and exclude_labels does not apply. Dedup key item-added:<ref>. Items present at startup, closed items, PR cards and items in a column with no configured stage never produce it; an unmanaged column (Backlog, Triage, …) does announce, and the later move into a managed column is a column-changed with no second item-added; an item first seen in an unconfigured column and later moved into a configured one gets column-changed only; a pending arrival whose labels or column never become known is dropped (with a log line) after the one-hour prune interval
column-changed the observer, on a StatusChanged Change whose PrevStatus is known and differs from the new Status (a Created change or an unknown previous Status is a baseline and invents nothing) from, to, actor (+ via/actor_login, the §7.14 label vocabulary: fabrik for the engine’s LocalStatusUpdated write-through or an echo that matched, overseer with via=<subscriber> for a fabrik_promote move — the issue’s overseer:<subscriber> shorthand — human/bot when a projects_v2_item webhook named the sender, unknown for a move first seen by the Layer 2 sweep, probe, deep fetch or reconcile), status_at (the webhook’s updated_at; absent, never “now”, when unknown). One move is one event: the write-through changes the store first and the echo, sweep or probe re-observing the same Status is a no-op
closed the observer, on a Change.Closed (open → closed on an item the store held) column (the board Status), actor (fabrik only when emitLanded announced the landing and the issue has not since been reopened or had its landing verification fail; a landing inferred from the snapshot alone keeps the webhook-derived actor), landing (true when emitLanded announced a landing for the item, or the snapshot shows fabrik:awaiting-landing-verification, a close-tracking label, or a merged linked PR). IssueClosed carries the webhook sender (no echo match: the engine registers no issues/closed echo); the engine’s ApplyIssueClosed write-through is OriginEngine
paused, awaiting-input, escalated, stalled the same internal/attention classification the read API uses (§7.13), re-evaluated after a short debounce when an item’s labels, stage state, status or invocation change, and on the ticker; one event per transition into the state paused and awaiting-input are the classifier’s needs-human (fabrik:awaiting-input present → awaiting-input); escalated is its escalated (PausedByEngine — set by escalateSettle and every pauseFor* that arms it — a counter at its limit, or fabrik:awaiting-runaway-alert; the §6.x settle-scan exhaustion paths and §7.2 pause-on-retry-limit all land here); stalled is its stalled with the one --stall-threshold, so a push is never looser than fabrik_board and a subscription cannot override it. The debounce means a pause’s labels and EnginePaused are classified together, not as two events. A cruise item settled at Validate (awaiting-merge-decision) is validate-settled’s business, a suspended item is decision-pending’s, and a failed landing verification is its own event, so none is reported here. A pause-family event that waits on a human also carries meta.options and meta.resume (remove-label, human-comment or see-pause-comment): the resume method the engine’s own pause comment gives (§7.13 “Options”), never a generic one that contradicts it. A stalled event is never published for a pending decision. The event carries meta.cause (§7.13 “Cause vocabulary”) when the same classification gives one, e.g. daemon-stop on a paused event, and omits it otherwise (#2121)
awaiting-input-stale the observer, when an invocation is recorded while fabrik:awaiting-input is present, that run did not emit FABRIK_BLOCKED_ON_INPUT and the engine did not pause the item itself once per awaiting-input episode; after a restart the invocation fields are empty, so no event (unknown, not guessed)
cycle-limit-near the observer, on a stage-state change a cycle/retry counter (not attempts) reaches limit - 1 (needs a known limit ≥ 2; zero config is unknown, never “limit 0”); re-arms when the counter drops below limit - 1, so a no-op refund (§2.9) followed by a new approach fires again
merge-train-ejected ejectMember (§6.16: trial/conflict/bisection causes stay queued, review-findings leaves it) and ejectQueuedMemberForComments (§6.16, #1863) cause, reason, failing_checks (from the trainCIDiagnostic the engine already holds), ejections, max_ejections, stays_queued
merge-train-failed ejectRedSingleton (own validation failing; “Merge-Train Red-Batch Bisection”; cause red-singleton-auto-repair when §6.31 started a repair instead of pausing), escalateStrandedTrainMember (landing failed, also the closed-unmerged-trial path) and fireRunawayGuard (§6.18) cause, reason, failing_checks when held
ci-timeout pauseForCITimeout, only on a fresh pause (a reapplied pause on an episode that already announced itself is silent) timeout, ci
review-timeout pauseForReviewTimeout reviews_before_wait names reviewers whose review predates the engine applying fabrik:awaiting-review — unknown after a restart, when that record-on-write timestamp is gone — and pending_reviewers
merged emitLanded (engine/channel_lifecycle.go, #2123), called by the landing paths at the merge itself (attemptMergeOnValidate’s direct merge, the merge-gate rebase fallback, landSingleton, the singleton fast path, the batch loop) and as a backstop at the Done advance and in markCreditedLanding; fallback: the fabrik:awaiting-landing-verification label delta (§6.19) pr is the credited PR (fabrik:credited-pr:<N>, applied first, for merge-train members whose own PR is closed, not merged) else the linked PR; both producers share the dedup key merged:<ref>:<pr>, so exactly one is published. The early emit is what makes merged precede closed and the close-tracking labels (fabrik:awaiting-close, awaiting-member-close, credited-pr:*), which then carry landing=true in meta
blocker-cleared the fabrik:blocked removal delta (the push-unblock path and the checkDependencies defense-in-depth path, see the fabrik:blocked label row) per dependent issue
children-spawned spawnChildren, after every child is created, wired and fabrik:children-spawned applied (§6.7) children, count
landing-verification-failed the fabrik:landing-verification-failed label delta (failLandingVerification, §6.19)  
claude-usage-limited, claude-usage-limit-lifted the edge of claudeUsageLimitedUntilTime (§7.3), checked on activateClaudeUsageLimit/clearClaudeUsageLimit and on the ticker, so a usage-limit hold that simply expires is also reported account-wide: no issue or repo; they ignore repo/issue/milestone scopes and reach every subscriber whose event filter admits them
pipeline-stalled, pipeline-resumed engine/channel_pipeline.go (#2124, ADR 2124), checkPipeline from the deriver’s ticker, on the consumer goroutine — independent of the poll loop, so a blocked poll is still seen (FR-012). Poll scope (scope=poll): the §7.13 overdue verdict turns true (pipeline-stalled, cause poll-in-progress or no-poll-started, interval_seconds, blocked_seconds, overdue_after_seconds, step when a poll is running and the step is known) and ends (pipeline-resumed) when the verdict clears or the health mirror’s stall_epoch — bumped by beginPoll/endPoll whenever they close an overdue interval — moved, which catches an episode that ended and restarted between two ticks. Merge-train scope (scope=merge-train, partition, members, oldest_queued_seconds, idle_reason): a partition meets the §7.13 stall condition; resolved when a run starts (active run), the members leave Queued, or the partition disappears. A partition stall is published even while the poll stall is open; the poll event carries the poll cause, the partition event its own account-wide and immediate: no issue, they ignore repo/issue/milestone scopes. Dedup keys pipeline-stalled:<scope>:<anchor RFC3339Nano>:<seq> (and pipeline-resumed:… with the same suffix) are unique per episode — the anchor is the poll’s start (or missed due time) or the partition’s oldest queued time, seq a per-process episode counter — because the hub persists dedup keys and a static key would swallow every later episode. Episode memory is in memory only: a restarted daemon that is still stalled re-raises. Detection latency is at most K × interval plus the 30 s tick
retry-starved, retry-resumed engine/channel_anomaly.go (#2170, ADR 2170), checkAnomalies from the deriver’s ticker and drain. finalizeStageOutcome’s retry branch (and the usage-limit short-circuit) calls the emitRetryScheduled hook with the cause and a due time stamped from e.now() (e.now() + stageRetryBackoff(), or the hold end), because LastAttemptAt is wall-clock and CooldownAt is consumed on admission (ADR 2096). dispatchCandidates calls emitDispatchDeferred where it logs deferred dispatch. An open fact becomes an episode when now - max(due, last usage-limit lift) >= threshold with no worker; reason is no worker slot (a deferral recorded since due, or the general pool full; slots_in_use/slots_max), else poll overdue (§7.13 verdict), else unknown. A fact that was never announced is dropped silently when the item is paused, blocked, awaiting input or done, left the stage or closed; an announced episode always closes with retry-resumed, whose how meta is dispatched when a worker enters and otherwise closed, advanced, held or gone. A deferral explains no worker slot only while it keeps recurring (recorded no earlier than the due time and within max(3 × poll, 2 min) of the check) item-scoped, immediate; dedup keys retry-starved:<ref>:<due RFC3339Nano>:<seq> and retry-resumed:… are unique per episode. In memory only. Reinvoke and comment-review dispatches and cooldown holds (dep-blocked, feedback-pending, review-blocked) are not stage retries and are not reported here
item-wedged, item-unwedged engine/channel_anomaly.go (#2170). Gate sites call noteGateHold(gate, reason, ids) after their own hold (feedbackGateBlocks with the describePendingFeedback text it logs; the review, merge and CI holds in catch_up_handlers.go / ci.go) and noteGateClear when the feedback gate stops holding. A hold keeps firstHeld and lastConfirmed and lapses when not re-confirmed for max(3 × PollSeconds, 2m) (plus githubRecheckInterval(), 10 × PollSeconds, for the feedback, review and CI gates, which are re-evaluated that rarely once nothing bumps the item). It becomes an episode when held for at least the gate’s threshold (--item-wedged-threshold for feedback; max(threshold, backstop + 2 × PollSeconds) for review, CI and merge), the item is not paused, no worker was dispatched or deferred since firstHeld, and the poll is not overdue. Ends with item-unwedged on a worker entering, a column change, a clear or a lapse item-scoped, immediate; dedup keys item-wedged:<ref>:<gate>:<firstHeld>:<seq>. internal/attention treats an active cooldown as waiting, so a hold that re-stamps a cooldown every poll never reaches stalled; this event covers that gap without changing the classifier. meta.cause is gate-hold:<gate>, the status vocabulary’s name for the hold (#2121). In memory only
cache-divergence engine/channel_events.go (#2170). onChange keeps a mutex-guarded ledger of recent engine writes (OriginEngine label deltas and EngineStatusWrite column changes, stamped with e.now()). A change whose Change.Path is reconcile (ShallowBoardItemUpdated, the batch board apply), deep-fetch (ItemDeepFetched) or probe (FromProbe) that contradicts an in-window entry (an opposite label delta, or a status change away from the written column) publishes one cache-divergence and consumes the entry; a later webhook or engine change to the same key clears it. Every event built from that change (label-applied/label-removed, the label-derived merged/blocker-cleared/landing-verification-failed, column-changed, closed) carries unverified=true (channelevents.MetaUnverified, defined once so #2169’s provenance reuses it) item-scoped, ordinary; no recovery event. A column revert may be a legitimate external move that only the reconcile observed
daemon-unreachable, daemon-reachable the fabrik mcp shim, when the held connection stays down past a 10 s grace and when it re-attaches live only: a daemon that is down cannot hold them
events-dropped, digest the hub (synthetic) count; dropped_types on events-dropped (e.g. label-applied x4), plus expired when entries aged out
stream-superseded the fabrik mcp shim (synthetic, immediate, not subscribable), when RunStream reports Superseded — a newer attach under the same name took the stream subscriber; live only
situation (#2171) engine/channel_situation.go: renderSituation(buildSituation(scope)), published by Hub.PublishSituation. Triggers: the catch-up (forced), a request (fabrik_situation push) and the cadence (situation_seconds, from onTick → publishSituations) account-scoped, immediate; a per-subscriber fingerprint dedup (below) instead of a DedupKey
catch-up (validate-settled, decision-pending, paused, awaiting-input, escalated, and an open pipeline-stalled, with catch_up=true) engine/channel_catchup.go: a snapshot of items currently settled at Validate and waiting on a human, currently suspended, or currently needs-human / escalated, plus (pipelineCatchUpEvents) any pipeline stall open now and (anomalyCatchUpEvents, #2170) any open retry-starved / item-wedged episode — current account state, unlike an item’s time-driven stalled, queued on a new subscription, on the first attach of a session (attach {catch_up: true}, never on a plain reconnect) and, since #2169, to every subscriber at daemon readiness after a start (see “Restart delivery” below); carries source=catch_up not a transition: no dedup key, no episode counter or per-item memory touched, and an event already held for the same (type, repo, issue) is skipped; Peek/Scan only

Anomaly events (#2170, ADR 2170). retry-starved/retry-resumed, item-wedged/item-unwedged and cache-divergence report per-item conditions the engine already knew about and previously only logged. They are observation only (R5/R10): engine/channel_anomaly.go reads the Store through Peek, the poll verdict through the health mirror and the slot counts through len/cap of the semaphore; facts the Store lacks arrive through void, nil-safe, panic-recovered hooks in channel_hooks.go (emitRetryScheduled, emitDispatchDeferred, noteGateHold, noteGateClear) called from the engine files that know them. Episode state is in memory only (a restarted daemon that is still in the condition raises a fresh episode) and every dedup key carries the episode’s anchor time and a per-process sequence number.

Baseline, arrivals, column moves and closes (#2123, ADR 2123). The line between “present at startup” and “added later” is drawn once. (1) Label baseline: gh.ProjectItem.LabelsAbsent marks a producer that did not observe labels (the probe bootstrap BootstrapFromProbe and the minimal item runProbeAndDeepFetch seeds); applyProjectItem/applyShallowItem then leave Labels alone and do not set ItemState.LabelsKnown. A mutation produces LabelDeltas only when before.LabelsKnown (explicit label writes — LocalLabel*, IssueLabeled/Unlabeled — always do), so a probe-seeded record followed by a deep fetch, a shallow reconcile or a board reconcile is a silent baseline for open, Done, Backlog and closed items alike; ItemDeepFetched always clears LabelsAbsent (a deep fetch carries labels). The flag is negative on purpose: a producer that forgets it stays noisy rather than silently swallowing real changes. (2) State-derived events: derive, seed and the ticker skip an item until LabelsKnown, so the first label-bearing evaluation takes the existing silent first-sight branch and a restart re-announces no pause, awaiting-input or escalation; subscribers wanting the current picture use catch-up. (3) Arrivals: Change.Created is set only for a record a running store learns about (never by Reset), and only after Engine.boardPopulated; items created between hub start and the end of the first successful poll are silent. The arrival rule (#2187, amending #2123 FR-004): item-added is published for an open, non-PR issue arriving after startup in any column that has a configured stage — managed or unmanaged, keyed on the flag and never the name — plus the legacy Backlog column with no stage (stages.IsParkingColumn); an item in a board column with no configured stage never produces it, whichever path first sees it. Parking-column items are seeded into the store by the probe loop (minimal IssueOpened, LabelsAbsent, no deep fetch) so a later hand move out of the column is a real column-changed; their item-added waits for the first label-bearing population (a webhook, or the next light reconcile). (4) Catch-up never replays item-added, column-changed or closed — they are transitions, not current state (fabrik_board is the pull alternative; TestCatchUpNeverReplaysLifecycleEvents). (5) Noise option: progress_via_column on a subscription drops label-applied/label-removed for stage:*:complete and stage:*:in_progress when the subscription takes column-changed (events empty or naming it; Validate rejects the option with an events list that omits it); it overrides labels and exclude_labels: [] for those two patterns, and the default exclusions are unchanged.

The situation digest and the catch-up (#2171, ADR 2171). situation is a subscribable event (Immediate, AccountWide: Matches ignores the repo / issue / milestone scopes, and the builder narrows by the subscriber’s own scopes instead - Subscription.ScopeMatches, factored out of Matches, over the union of the scopes of the subscriber’s subscriptions that take it). Subscription.TakesSituation() is true for an explicit events list naming it and for the semantic default (EventsDefault); a grandfathered subscription (empty events, no marker) and an explicit list without it do not. Subscription.SituationSeconds (situation_seconds, fabrik_subscribe’s situation_seconds; default 0 = off; 60 s - 24 h, channelevents.MinSituation / MaxSituation; refused unless the subscription takes situation) is the cadence, deliberately not digest_seconds, which batches ordinary events into the synthetic digest. Three triggers, one builder: fabrik_situation (push: true, force), the cadence and the catch-up all call Engine.pushSituation(name, trigger, force) → buildSituation + renderSituation, so their Content is identical for the same state (trigger meta and the catch-up tags catch_up=true, source=catch_up differ). Cadence: channelEvents.publishSituations, on the consumer goroutine from onTick (the 30 s tick, so granularity is the tick), once the daemon is ready, for each attached subscriber whose shortest cadence has elapsed on e.now() since its last evaluation (sitEval, in memory); a detached subscriber is caught up on attach instead. Dedup: Hub.PublishSituation(name, ev, fingerprint, force) sends unless !force and the fingerprint equals the one last sent to name; the memory is per subscriber and persisted in dedup.json (stateFile.Situation, forgotten when the last subscription goes), and the hub’s DedupKey mechanism is not used (it is global and permanent, and could not resend “A, then B, then A again”). situationFingerprint is an explicit allowlist projection of the view - identity, title, link, state, code, rank, options, gate name / reason / start, verdicts, resume method, pending kind and blockers, partition members / phase / idle reason, slots, train queued, overdue verdict, anomaly kind / subject / start, truncation counts - and excludes everything that moves with the clock or the poll cycle alone (the as_of envelope, every age or *_seconds, whether a poll is running, the step, the classifier’s prose reason). Held queue: a send first drops any situation entry already queued for that subscriber (queueFile.dropType), so a subscriber that is away holds at most one, always the newest. Catch-up: Engine.catchUp, catchUpAll and catchUpLate push a forced situation digest (trigger=catch_up) for the selected subscriptions that take it - which also records its fingerprint, so the first scheduled digest does not repeat it, and restarts the reminder clock of subscriptions with remind_after via Hub.RestartReminders - and give the per-item catchUpEvents snapshot, through Hub.CatchUpWhere(..., notTakingSituation), only to the subscriptions that do not take it. Timing, the readiness gate and who receives a catch-up (§ “Restart delivery”) are unchanged; an empty system still gets an explicit “nothing needs you” digest. Seams: SetSituationDedupDisabledForTest (every digest is sent - the sim’s non-vacuity check), SetSituationMinForTest.

Restart delivery and downtime deltas (#2169, ADR 2169). Supersedes ADR-1966-b’s “a reconnect never gets a catch-up” and “a restart is silent”. After a daemon start (also a self-upgrade or SIGHUP re-exec) a subscriber receives, in queue order: (1) daemon-reachable, written by the shim at the attach ack, before the hub drains; (2) events already held for it (real, previously published); (3) one catch-up snapshot; (4) the downtime deltas. Nothing replays unchanged state, and a reconnect that is not preceded by a daemon start gets no catch-up (unchanged).

Title on every item-scoped event (#2127 R8). baseEvent, the one constructor of item-scoped events, sets meta.title from the item’s title (pushTitle: line breaks and backticks become spaces, whitespace collapses, cut at 200 runes). Account-wide events, and the two fallbacks that build an event for an item that has left the Store (retryResumedEvent, itemUnwedgedEvent), carry none; an event held before an upgrade lacks it and a held event keeps its creation-time title.

Hub, subscriptions and held delivery. Subscribers are keyed by a stable name (the operator’s topic or session name — never a PID; supplied by fabrik mcp --subscriber / FABRIK_SUBSCRIBER). A subscription ANDs its set scopes (repos, issues, milestone, label patterns for label events, event types) and ORs the entries within a list; an item whose milestone is unknown never matches a milestone scope. One event reaches a subscriber at most once however many of its subscriptions match. The hub appends to a per-subscriber persisted queue before any delivery; the attached session’s drain loop sends in order and removes an entry only after a successful write, so held events (no session attached) go out first, in order and individually, then live ones. The queue is bounded (--channel-held-max, default 200): overflow evicts the oldest non-immediate entry first and drops an immediate one (validate-settled, escalated, paused) only when nothing else is evictable, and the next delivery starts with one events-dropped notice naming the dropped event types (an entry that aged out is counted by type too, and the notice says expired rather than overflow); entries older than 7 days and subscribers not connected for 30 days are pruned (the clock is the last time a session was connected: it is refreshed while attached, on detach and at shutdown, so a long-lived session is never mistaken for an abandoned one). A second attach under the same name supersedes the first (it receives a superseded frame, the shim tells its session with one stream-superseded notice, and it stops reconnecting). Digest mode batches ordinary events per subscription (30 s–1 h); validate-settled, escalated, paused and daemon-unreachable bypass it, and a digest entry queued while the subscriber had no attached session is delivered individually, in order, on its next attach (not re-digested). A digest collected while a session was attached keeps batching across a brief reconnect (heartbeat reset, daemon re-exec) and is sent when due. Delivery means “written to the shim”: Claude Code never acknowledges a channel message and drops it silently when the session did not enable the channel, which the daemon cannot detect.

Subscribe under version skew (#2168, ADR 2168). The daemon self-upgrades while a fabrik mcp shim lives as long as its Claude session, so a shim routinely names event types its daemon no longer (or does not yet) know, and no daemon-side change reaches a schema baked into a running shim. channelStreamer.Subscribe therefore runs channelevents.NormalizeEvents on the request before the hub sees it: the two pre-#2129 names (claude-limit-suspended, claude-limit-lifted) are mapped to claude-usage-limited / claude-usage-limit-lifted, every other unknown (or synthetic) name is rejected, and the rest of the subscription is applied. The result reports both: mapped ([{from, to}]) and rejected ([]string), each omitted when empty, so a request of current names alone is unchanged. The kept list is de-duplicated, so ["claude-limit-suspended", "claude-usage-limited"] is one entry, and because mapping happens before the hub’s idempotency check, re-subscribing with an old name is the same subscription, not a second one. An empty event list means every event, so a request that named events and keeps none is refused (bad_request, naming each rejected type), stores nothing and leaves the subscriber’s existing subscriptions untouched. Hub and Subscription.Validate stay strict; the persisted-state migration in Open is unchanged. A new shim against an old daemon is not covered: the old daemon rejects the request as before.

Version skew is reported. hello carries daemon_version (omitted when the daemon has none), and fabrik_health carries the daemon’s version. The shim compares the daemon’s version with its own as opaque strings and, the first time they differ, appends one text block to a tool result naming both and advising a session restart; it is never repeated. fabrik_health also gets a “Versions — daemon X, MCP server Y” block. Every other tool checks with a hello until the notice has been shown, but a check that found no skew (equal builds, or a daemon without daemon_version) is trusted for five minutes (skewRecheckInterval), so a daemon upgrade mid-session is noticed within that time without a hello on every call, and the hello is made outside the lock so concurrent tool calls do not queue behind it; a failed hello is retried by the next call, a daemon without daemon_version or a shim without a version yields no notice and no error. Nothing on the daemon side depends on what the shim reports.

Unsubscribe is a hard stop (daemon side). Hub.Unsubscribe removes the subscription (all of the subscriber’s when no id is given), bumps the subscriber’s epoch, discards every queued entry that none of the remaining subscriptions matches (all entries and the drop counters when none remains; removing one of several subscriptions therefore also discards entries only it matched), and then waits for the drain loop’s current cycle: the drain loop holds sendMu across a peek→deliver→remove cycle, so an in-flight Deliver has finished before Unsubscribe returns. The drain loop for its part rechecks the epoch and its attachment immediately before every Deliver (a batch taken before the unsubscribe is abandoned unwritten) and, at the top of every cycle, re-matches the queue against a fresh snapshot of the subscriber’s subscriptions, which catches a Publish or CatchUp that resolved its targets before the unsubscribe and enqueued after it. The guarantee is: no Deliver for a removed subscription completes after Unsubscribe returns; the wait is bounded by the session’s 5 s write deadline. The attachment stays bound after the last removal (closing it would make the shim reconnect and re-attach), so a later subscribe under the same name resumes delivery on it. Lock order: h.mu is never held while taking s.mu or sendMu, and sendMu is never taken under h.mu. What the daemon cannot recall: events already accepted by the socket, the shim’s stdout pipe or Claude Code’s own queue when Unsubscribe returns may still surface in the session.

Durable vs in-memory. Subscriptions, held queues, dedup keys and settle-episode counters live under .fabrik/state/channel/ (subscriptions.json, held-<hash>.json, dedup.json), written atomically and quarantined (.corrupt-<unix>) when unreadable, like workers.json. Subscriptions and held queues are written on every change; dedup keys and counters are coalesced into one dedup.json write at most every 250 ms (Publish runs on the engine’s poll and tick paths and never does a per-event rewrite) and flushed synchronously by Hub.Close, which the SIGHUP path calls before the exec — so only a crash inside that window can lose the newest dedup keys and republish an event once. They are loaded before the socket binds, so a reconnect during startup finds them. The per-item memory the deriver keeps (last pause-family state, counter arming) is in-memory only and re-seeded silently from the store at startup, so a restart announces nothing about the existing board. Events that occur while the daemon is down, or during a self-upgrade exec, are not captured.

Reminders (#2120 R6a). A subscription may set remind_after (seconds, 600 – 2 592 000; default off, fabrik_subscribe’s remind_after). A still-open decision-pending episode is then re-sent to attached sessions of that subscription once per interval — meta.reminder=true, meta.reminder_count, a (reminder) content prefix, and the original event’s attributes and comment link (the reminder is built from the event as announced, so it cannot differ). The interval runs from the later of the hub’s persisted announcement time (its dedup entry) and the previous reminder (persisted per subscriber in dedup.json’s reminders), so a restart neither fires nor postpones one and no reminder is ever driven by daemon uptime; with no announcement on record a first look only plants the anchor. Reminders are never held for a detached subscriber (its catch-up on attach already carries the pending decision), have no dedup key, and end with the episode. A catch-up of an open decision-pending episode restarts that subscriber’s reminder clock (the catch-up event carries the episode key to Hub.CatchUp, which clears it before queueing), so a reminder already overdue at attach time does not fire right behind the catch-up. A stalled pending decision is never re-published as stalled.

Label events are not a readiness signal. A label such as stage:Validate:complete can be applied and removed again while the PR still needs a rebase, CI or a fix (it was re-applied and pushed while a PR was DIRTY). Only validate-settled (and decision-pending) say an item awaits a human, and settlement-withdrawn says that stopped being true before the human acted.

Connection lifetime (protocol v2). hello reports protocol_version 2 (and daemon_version) and lists subscribe/unsubscribe/attach only when the channel hub is running; a client that does not find attach degrades to read tools. After attach the server writes a heartbeat frame every 30 s (Server.HeartbeatInterval); every successful write moves the connection’s read deadline out by the idle timeout (Server.IdleTimeout, default 10 min), so a push-only subscriber is never dropped, and a write that fails (the 5 s write deadline) reaps a dead peer within one heartbeat. The client treats 3 missed heartbeats as a half-open connection and reconnects with jittered exponential backoff (250 ms → 30 s), re-attaching under the same name every time; a busy refusal at the 32-connection cap is retried. Each attached shim holds one of the 32 slots.

Exec paths. The SIGHUP restart closes the hub (closeChannelEvents) before the exec. The self-upgrade exec (checkAndUpgrade → selfupgrade’s PreExecHook → flushChannelEvents) only flushes the hub’s lazily written dedup and counter state immediately before syscall.Exec and tears nothing down, because a failed exec leaves the process running; so a settle announced within the 250 ms flush window of an upgrade is not announced again by the new process.

Known limits. Delivery is at-most-once to the session (no acknowledgement; a half-open connection can lose up to about one heartbeat window of events). fabrik_unsubscribe is a hard stop only for what the daemon has not yet written: pushes already in the kernel socket buffer, the shim’s stdout pipe or Claude Code’s queue (on the order of 200 notifications) cannot be recalled, so a few can still appear after it returns. Label events can be reordered in a narrow race between two concurrent applies of the same label. A restart or self-upgrade replays nothing: no re-published pause-family events and no label events for unchanged items; what it does deliver is the catch-up snapshot and the downtime deltas (#2169, “Restart delivery” above). Arrivals that happen while the daemon is down are not captured, and items created between hub start and the end of the first successful poll are not announced. A column move is attributed human/bot only when a projects_v2_item webhook named the sender; Status often reaches the daemon only through the Layer 2 sweep (§9), so unknown is common and status_at is then absent. merged precedes the close events of a landing only when the engine itself performs the merge: for a landing GitHub completes on its own (native auto-merge, merge queue) the issues closed webhook can reach the store before the engine observes the merge, and that closed is then published first with landing=false. A sweep that observes an engine move before the write-through lands attributes it unknown, and the write-through is then a no-op (still exactly one event). The label actor is unknown when only a poll saw the change (no cheap read names the actor; ADR 2125), via is a self-asserted advisory name, an echo that arrives before the engine registers it keeps today’s attribution, and under PAT auth a worker’s gh label write whose echo was missed reads as human. One incident can produce several related pushes (paused + ci-timeout + label-applied).

Zero GitHub cost and observation only. engine/channel_*.go and internal/channelevents read item state only through Peek/Scan and the guarded engine state, and never the engine’s GitHub clients, Store.Get, the live label-timestamp fallback or the live feedback/review gates — enforced by TestChannelEventsSourceDoesNotReachGitHub; TestChannelEventsAreObservationOnly (tests/sim) shows a run with a hub and a subscriber makes exactly the same GitHub mutations in the same order as a run without one.

7.15 Overseer Actions: the Mutating Local-API Method Set (#1969)

ADR-1966-c. §7.13’s read API and §7.14’s push stream never write to GitHub. The action method set — promote, advance, move_parked, set_autonomy, revalidate, clear_claude_limit, surfaced as the fabrik_promote / fabrik_advance / fabrik_move_parked / fabrik_set_autonomy / fabrik_revalidate / fabrik_clear_claude_limit MCP tools — is the first part of the local API that does. It adds no engine transition: each action performs one the engine already has (a board Status move out of a parking column or between two parking columns, the one-stage advance of a completed stage, the fabrik:cruise/fabrik:yolo labels, the one-shot fabrik:revalidate and fabrik:clear-claude-limit command labels) and the engine’s existing settle scans consume the command labels exactly as if an operator had applied them in the UI.

Separation. localapi.Actor is a second optional interface beside the three-method read Backend (the Streamer precedent). Server.handle routes an action method to Server.handleAction, the only caller of the Actor; a server with no Actor answers every action method unknown_method and hello does not list them. The engine implements Actor in engine/overseer_actions.go (advance in engine/overseer_advance.go, move_parked in engine/overseer_park.go) — deliberately not in localapi*.go or channel_*.go, whose static scans forbid the GitHub-client identifiers an action needs, so the read-path guarantee of §7.13 and §7.14 stays pinned. startLocalAPI wires it with one line. Refusals are CodeRefused with the item’s current state in Error.Data.

Per-action sequence (serialised with each other by one mutex; no per-item in-flight guard is taken, so an action races the poll loop exactly as a human applying a label or dragging a card does): (1) resolve the issue through the same resolver as fabrik_status, from Store.Peek only — a miss is not_found, a closed item is refused; (2) check the precondition on the cached snapshot, a missing fact counting as unknown and refusing; (3) for promote, advance and revalidate, one ID-keyed live FetchProjectItemStatus read confirms the column (the #1871 liveLandingState precedent; a read error refuses — fail closed — and the live column, not the cache, decides); (4) write through the engine’s own idioms — addLabelChecked / removeLabelChecked (error-returning, with cache write-through, SelfWriteObserved and echo registration) and moveItemToStatus (the status-move idiom of advanceToNextStage/moveItemToValidate, taking the project ID from the board cache, the item ID from the store and the field and option IDs from the loaded statusField, so no FetchStatusField, FetchProjectBoard or LookupIssueProjectItem is made); (5) post the audit comment through postComment (via the single postAudit helper). If set_autonomy fails part-way after at least one label write landed, the comment is posted anyway, worded as a partial application, and the error is still returned; if no write landed, no comment is posted.

Action Precondition (all on top of: one managed, open, non-PR item) Writes
promote to is a configured, non-unmanaged, non-holding, non-cleanup stage with a board option; the cached and live column is a parking column (stages.IsParkingColumn: an Unmanaged stage, or the legacy Backlog when no Backlog stage is configured); project ID, item ID and status field are known one UpdateProjectItemStatus
advance (#2126, ADR 2126) cached and live column is a configured pipeline stage whose following stage (skipping parking columns) is not a holding or cleanup stage — so Validate, which leads to Queued/Done, is refused with “use the PR merge”; the next stage has a board option; no worker (ItemState.Worker, live fabrik:locked:* / fabrik:editing / stage:<X>:in_progress); live stage:<X>:complete present; no fabrik:paused / fabrik:awaiting-input / fabrik:blocked and none of fabrik:awaiting-ci, fabrik:awaiting-review, fabrik:rebase-needed, fabrik:auto-merge-enabled, fabrik:awaiting-done (labels read live, failing closed); then the read-only gates: no open dependency (openDependencies on a live FetchItemDetails copy), feedbackGateBlocks clear, and reviewGateHold clear on a wait_for_reviews stage one advanceToNextStage — the transition auto-advance uses, including clearStatusLine; no label write
move_parked (#2188, ADR 2188) cached and live column is a parking column (stages.IsParkingColumn, as for promote); to is also a parking column — a configured Unmanaged stage, or the legacy Backlog when no Backlog stage is configured — that exists as a board option, and differs from the current (cached and live) column; project ID, item ID and status field are known. Pipeline, holding and cleanup columns are neither a source nor a destination. one UpdateProjectItemStatus via moveItemToStatus; no label write (a paused, blocked, cruise or yolo item moves like any other and keeps its labels) and no auto-advance note, since nothing is dispatched for an unmanaged column
set_autonomy mode ∈ {cruise,yolo,none}; the labels are read live (FetchLabels; a failed read refuses) and if they already equal the mode, nothing is written add the wanted label first, then remove the other (none: yolo then cruise), so a partial failure leaves the conservative state
revalidate cached and live column is Validate; no fabrik:revalidate; no fabrik:paused / fabrik:awaiting-input, checked on the cache and again on a live FetchLabels read that fails closed (handleRevalidateLabel clears both, which would be a label-based pause lift) add fabrik:revalidate (an in-flight Validate worker defers it to a later poll; the result notes this)
clear_claude_limit claudeUsageLimitedUntilTime(now) reports an active usage-limit hold; no clear request already pending add fabrik:clear-claude-limit to the chosen item: open, non-PR, on the board; one carrying fabrik:claude-limit first, then the lowest number

Promote refusal shape (#2113). Promote checks the source column (cached, then live) before it checks to. A refusal because the item is not in a parking column carries the current state but no valid_targets — the move is impossible, so a target list would contradict it. valid_targets is set only while the item is established as parked (a missing or invalid to); the stale-cache (confirmed live) refusal omits it too. The pipeline-column refusal also names fabrik_advance as the way to move a completed stage forward (#2126); the unknown-column and unparked refusals do not.

Move parked (#2188, ADR 2188). MoveParked is Promote with the destination restricted to parking columns, as its own method so that its name does not suggest work starts. The refusal order is: board metadata; the source column (cached; a pipeline, holding, cleanup or unconfigured column is refused, with no valid_targets); to (missing, not a parking destination, a parking stage with no board option, or no other parking column at all); to equal to the current column; the live column (a read error refuses; the live column is judged by the same predicate and the same-column check is re-run on it). valid_targets lists the parking destinations other than the current column and is set only while the item is established as parked. parkingDestinations decides membership with stages.IsParkingColumn plus the board-option check, so the legacy-Backlog rule is not redefined. A refusal writes nothing. tools/list is static, as for the other actions; a daemon without an Actor does not advertise move_parked in hello, and the shim reports an older daemon as not supporting overseer actions.

Advance (#2126, ADR 2126). Advance is the overseer’s version of the move a stage with auto-advance makes when it completes; it exists for stages that wait for a human (Specify, typically) and moves the item one stage without changing its autonomy. The refusal order is fixed so each case has one reason: board metadata; the column (parking column → fabrik_promote, unknown column, cleanup, holding, and “the move out of this stage is the engine’s landing” for Validate); the next stage’s board column; a worker (in memory, so it is checked before any live read); the live column and labels (the live column decides which stage is current, and a stale cache that already shows the next stage refuses as “not complete”, so two requests can never move the item two stages); completion; pause; block; engine-owned wait labels; dependencies; feedback; the review wait. Every gate is a read-only core shared with the engine — openDependencies (extracted from checkDependencies), feedbackGateBlocks, and reviewGateHold (extracted from reviewGateBlocksLanding, which now wraps it and applies fabrik:awaiting-review; it shares the pure verdict helpers with checkReviewGate but not its timeout, bot re-prompt ladder or pause — those stay with the engine’s catch-up gate, so a stuck review wait ends in the engine’s own pause and the advance only ever fails closed) — because stage:<X>:complete alone does not cover a review wait (for a wait_for_reviews stage it is added before the reviewers are recorded as outstanding). A refused advance writes nothing to GitHub; the only side effect of the refusal path is feedbackGateBlocks’ in-memory feedback-pending cooldown, the same one the engine’s own held advance records. advanceToNextStage reads e.statusField without the lock (as its other callers do); the project ID comes from the board cache, or the last polled board (Engine.boardProjectID) when none is wired. An item carrying fabrik:cruise or fabrik:yolo is not refused: the result notes it and the label is untouched. A “queued” item here means a worker handle (ItemState.Worker is set before the dispatch goroutine starts); retry cooldowns are not refusal conditions.

Attribution. The requester is the --subscriber name the shim was launched with; the shim refuses an action without it and the daemon re-validates it. It is reduced to [A-Za-z0-9._:/-] and truncated to 64 characters before it is rendered. The socket is unauthenticated (mode 0600), so the name is self-asserted: an audit aid, not authorisation. Since #2123 (ADR 2123) promote’s board move carries the requester too (moveItemToStatus → CacheImpl.UpdateItemStatusBy, and the projects_v2_item echo entry via RegisterEchoIfSubscribedVia), so the resulting column-changed event reads actor=overseer, via=<subscriber> (§7.14); move_parked’s board move does the same (one column-changed, actor overseer).

Audit comment and the comment-processing lifecycle (§2.2). The comment is 🏭 **Fabrik — overseer action: <action>**\n\n<what happened>; requested by subscriber ` via MCP.`, built in one function, `overseerAuditBody`. It must start with the exact `🏭 **Fabrik` prefix, which `filterNewComments` (hence `findNewComments`, `resumeAuthorised`, `unprocessedFeedback`/`feedbackGateBlocks`, `settleQueuedCommentCause` and the channel `cachedPendingFeedback`) uses to drop the engine's own output. Under PAT auth the comment's author is the operator's own non-bot login, so `filterHuman` does not exclude it and the prefix is the only protection; the comment can therefore neither lift a pause, nor start a comment-review cycle, nor hold a stage advance or landing. This is pinned against the real predicates (`engine/overseer_actions_test.go`) and the real poll loop (`tests/sim/overseer_actions_test.go`, including a replay of the posted body under an operator login). The `TestAddCommentCompliance` funnel scan treats `overseerAuditBody` as a compliant body source.

No pause lift, no free-form comment (R2). No method removes fabrik:paused or fabrik:awaiting-input and none posts a comment other than the audit comment; a test pins both the method set and the source of overseer_actions.go, overseer_advance.go and overseer_park.go (the last also writes no label). A pause is lifted only by resumeAuthorised (§4.4, ADR-1813).

Ordering and failure. Write first, then comment. A failed write returns the error and posts no comment (a multi-label set_autonomy names the writes that did land). A failed comment leaves the action in place and reports audit_comment: failed. The action’s own label write reaches the channel observer like any engine write-through and surfaces as a label-applied / label-removed event with actor overseer and via set to the requester’s subscriber name (#2125); the engine’s later removal of a one-shot command label (fabrik:revalidate, fabrik:clear-claude-limit) is its own write and reads fabrik.

7.16 Specify Spec Persistence (persist_spec, ADR 2034)

A stage with persist_spec: true (the default Specify config) has the engine project its canonical issue-body update to specs/<issue>-<slug>/spec.md in the issue’s worktree and commit that one file. The stage stays read_only: true — Claude writes nothing and gains no tool permission — so the stash/restore, the commitWIP skip and the worktree-boundary layers are unchanged. docs/stage-lifecycle.md “Read-Only Stage Stashing” covers the per-invocation view.

Where it runs. persistSpec (engine/spec_persist.go) is called immediately after a successful or failed UpdateIssueBody from the same extractUpdatedBody string, on both entry points:

Path Site Push
Stage dispatch finalizeStageOutcome, after the issue-body update (post stash-pop) the existing post-run pushBranchUnlessQueued (runs after any invocation where Claude ran)
Comment review publishCommentOutput, after the issue-body update pushBranchUnlessQueued, only when persistSpec created a commit — processComments has no push of its own

Because it keys on the body update rather than on completed, a round ending in FABRIK_BLOCKED_ON_INPUT persists the spec too. A round with no FABRIK_ISSUE_UPDATE block, or one that emits FABRIK_NO_WORK_NEEDED, writes nothing.

Rules.

Upgrade behavior. persist_spec is opt-in per stage YAML. A specify.yaml predating it silently writes no file (no denial, no pause); startup drift detection reports the missing key (it is not a no-op default) and fabrik refresh-stages --apply adds it.

7.17 Display-Only Status-Line Field (#2048, ADR 2048)

The engine projects a one-line description of what an item is doing or waiting on into an optional ProjectV2 text field (project_fields.status_line, default name Fabrik; off disables). The field is display-only and is not state. No handler, gate, settle scan or cache reads it back — not from GitHub, not from the Store, not from the writer’s own record of what it wrote. Labels and the item store remain the only state; an operator editing the field by hand changes nothing the engine does.

Absent, mismatched and failed lookups (#2218, ADR 2218). fetchProjectFieldID (github/textfield.go) treats GitHub’s GraphQL error Could not resolve to a Unions::ProjectV2FieldConfiguration with the name … (on the field path, via the typed *github.GraphQLError) as absent, so a missing field reaches the existing one-line unavailable notice with no retry; any other error stays an error. A field of the name with another type returns *github.FieldTypeMismatchError. Each displayFieldState records its outcome (found / absent / mismatch / off), and resolveDisplayFields (called at startup and every poll) brings the warnings panel in line: one display_field:missing entry listing the enabled fields the board lacks, with FixAction fabrik_command (repair-board --apply) — omitted on a user-owned board, which repair-board refuses; one display_field:mismatch:<label> entry per wrong-type field, never with a FixAction; and display_field_lookup:<label> for a genuine lookup failure, reported once per outage and resolved by a later success or by off. warnings.Reconcile clears stale display_field entries only when every enabled field has an answer; an unresolved field never clears or is named as missing. An absent field is final for the process, so the missing entry clears at the next start after the fields are created, or when the missing ones are set to off.

Writer. engine/status_line.go: setStatusLine(item, line) / clearStatusLine(item) are the only entry points (setMembersStatusLine for a train’s members, withStatusLine for a bracketed activity such as comment review). The wording lives in engine/status_line_text.go.

Transitions that write.

Line Hook
queued · batch of N end of prepareTrainWorker (fresh formation)
queued advanceToQueued; a trial abandoned (pending, infra, cancelled, assemble error); a member ejected but staying Queued; a dissolved batch; a catch-up decided for this poll
deferred: overlaps #M admitByOverlap
trial · resolving conflicts assembleTrialBranch, when a member’s merge conflicts (no trial PR exists yet, so no number)
trial #<PR> · CI running assembleAndValidateInner, after the draft CI PR is created (PR number = the draft CI PR)
bisecting · step i of n bisect, before each sub-trial (i = the validation about to run, n = the cost cap; sub-trials write no trial line of their own)
catch-up · CI on PR #N trySingletonCatchUp, while waitMemberCI waits on the member’s own PR
landing landMergeTrainBatch, landSingleton, the singleton fast path
<Stage> · running acquireLockAndVerify, once the in-progress label is set (not for the cleanup stage)
<Stage> · comment review processCommentsClassified, restored to the previous line afterwards unless something newer was written
blocked by #N checkDependencies, on the first-time block
paused: <reason> pauseIssue (reason = the pause comment’s header), blockOnInput, the merge-train pause paths, the settle escalations
claude-limit until HH:MM handleUsageLimitExit, for the item whose invocation hit the limit only (daemon local time)
awaiting CI on PR #N finalizeStageOutcome, when fabrik:awaiting-ci is first applied
awaiting review the review gate (engine/reviews.go), when fabrik:awaiting-review is first applied

Cleared by advanceToNextStage (merge-train landings, ordinary merge, and every stage-to-stage advance), by the two Done moves that bypass it (the no-work-needed settle and the closed-item advance), by rerouteQueuedMemberOffHolding, when fabrik:blocked is lifted, and by the stage worker’s release when its <Stage> · running line is still showing (clearStatusLineIfShowing: the stage ended into a state with no line of its own, so the card must not keep claiming to be running). Lines can still go stale where no transition fires (a crash mid-phase); they are overwritten by the item’s next transition or Done and are not repaired by a read-back.

No webhook echo is registered for a status-line write. RegisterEchoIfSubscribed("projects_v2_item", "edited", <ItemID>) has no field discriminator, so a registration would collide with Status-move registrations, and a delivery that never arrives would inflate the ADR-042 miss counter.

Cache and drift (R5). Three places must treat a display-field write as a no-op:

  1. Webhook. applyProjectsV2ItemDelta (boardcache/delta.go) still runs the echo match first, then returns for any edit whose field_type is not single_select, before the Store is touched: no Store change (so no wake), no localDeltaAt stamp (so no invalidation), no drift. changes.field_value.to is decoded as json.RawMessage because its type depends on the field type; a text edit’s value never causes an unmarshal error.
  2. updatedAt. A field write is assumed to bump the project item’s updatedAt (unverified for text fields). The board and probe queries therefore select the display field’s own updatedAt (statusLine: fieldValueByName(...) @include(if: $withStatusLine), set through github.Client.SetStatusLineField), and the parsers drop the project-item contribution to UpdatedAt / EffectiveUpdatedAt when it is not after that value (2 s tolerance). The probe, LightReconcile and the store all read that one parsed value, so none count a display write as drift or a reason to deep-fetch. Status is compared directly, so a real Status move is still seen even if a display write shortly follows it. With the feature off the selection is not sent and parsing is unchanged.
  3. Project-level gate. project.updatedAt may still advance, so the layer-2 status batch can run once per write burst; it wakes nothing.

Neutralisation seams: CacheImpl.SetTextEditSuppressionDisabledForTest (boardcache) and leaving SetStatusLineField unset (the discount); the tests that pin the behaviour are shown to fail without them.

7.17.1 Last activity and Last run fields (#2049, ADR 2049)

Two more display-only fields share the writer: Last activity (ProjectV2 date, project_fields.last_activity, default Last activity) and Last run (text, project_fields.last_run, default Last run); off disables each independently. They follow every rule above — display-only and never read back (not from GitHub, the Store, or the writer’s own record), write-on-change only, a missing / wrong-typed / off field is a silent no-op with one [startup] last-activity|last-run field unavailable … line, a failed write is logged and swallowed, a successful write applies no SelfWriteObserved (the parsers discount its updatedAt bump, #2180), and no webhook echo is registered. No new state is introduced: no ItemState field, no persistence, no counter; the only memory is the per-field skip-unchanged record (displayFieldState.last), forgotten on restart.

Writer. engine/status_line.go is generalised to a displayFieldState per field (the status line, Engine.lastActivity, Engine.lastRun), each with its own lookup, startup line, lookup-retry back-off and per-item write lock, and a displayFieldSpec carrying the field kind (text or date). engine/run_fields.go holds the two entry points and engine/run_fields_text.go the wording (lastActivityDate, lastRunLine, formatRunDuration).

Hooks — call sites, not a subscription. The values come from fields the TUI events also carry, but the writer is called at the sites that hold them rather than from the InvocationObserver / event channel: the observer would also see the Done-cleanup InvocationRecorded (not a run) and reads the board column at observation time rather than the stage that ran.

Write Hook
Last activity ← UTC date of the start instant processItem, after acquireLockAndVerify succeeds (beside the JobStartedEvent); processCommentsClassified, beside its JobStartedEvent
Last activity ← UTC date of the completion instant, Last run ← lastRunLine finalizeStageOutcome, beside InvocationRecorded; processCommentsClassified, beside its InvocationRecorded

Never hooked: merge-train batch job events (repo-level, IssueNumber 0), the synthetic Skipped completions, TurnProgressEvent (per turn), and the Done-cleanup record. Nothing clears either field at Done.

Value. Last activity is the UTC YYYY-MM-DD of StartedAt / the completion time (both time.Now, the same source the events use). Last run is <Stage>[ · comment review] · <outcome>[ · <used>/<budget> turns][ · <duration>]; outcome precedence is turn-limited, blocked on input, completed, failed (errored, not completed), else incomplete; turns appear for completed and turn-limited runs, duration for completed runs only; capped at 60 characters like the status line.

Cache and drift. The same three layers as the status line, extended to all three fields: the webhook guard already ignores any non-single_select edit (a date edit has field_type: "date"; pinned by TestDeltaProjectsV2ItemDateEditIsNoOp); the board and probe queries additionally select lastActivity (ProjectV2ItemFieldDateValue { updatedAt }) and lastRun (ProjectV2ItemFieldTextValue { updatedAt }) by alias, and projectItemUpdatedAt discounts the project item’s updatedAt against the latest present display-field value updatedAt (2 s tolerance, nil values ignored). These fields are never cleared, so every bump of theirs has a value node to compare against (the status-line clear at Done remains the known uncovered bump). Whether GitHub bumps the item updatedAt for a date write is unverified; the sim models the conservative case.

7.17.2 Train phase hook, live row and episode outcomes (#2050, ADR 2050)

The merge-train TUI row and the board status line are two projections of one train, driven by one hook so they cannot disagree. Presentation only: no train decision, ordering or timing changes, and the board’s lines, dedupe and count are byte-identical to §7.17.

Hook. e.noteTrainPhase(ep, repoKey, members, phase) (engine/train_phase.go) is the single “train phase changed” call. Every merge-train phase site — prepareTrainWorker (admitted), the loop’s assembly, assembleTrialBranch (conflict), assembleAndValidateInner (trial CI), bisect, the three landing sites, landOneAtATime, trySingletonCatchUp, acquireTrainSlotEp (slot wait) and the worker-exit releases — calls it exactly once per transition, and it (1) writes the board line through the unchanged setMembersStatusLine builders (skipped for phases with no board text: assembling, waiting for slot, one-at-a-time), (2) records the phase and its start time on the episode, then (3) emits one tui.TrainRowEvent through emitStructural (never the droppable emit, never under the episode lock). TestTrainPhaseSites_GoThroughTheHook forbids the phase builders (statusLineTrial/Bisect/Landing/CatchUp/Queued) in engine/merge_train*.go outside comments, and TestMergeTrainTUI_BoardAndRowSeeSamePhaseSequence shows the TUI rows equal the hook records one to one and the board writes equal their board projection. Sub-trials under bisection stay quiet (quietTrialLines) on both projections.

TUI phase Board line
assembling queued · batch of N (once, at batch formation); none on later re-forms
waiting for slot none; the previous phase is restored once the slot is obtained
resolving conflicts on #N trial · resolving conflicts
trial CI #NNNN trial #NNNN · CI running
bisecting (step n) bisecting · step n of cap
landing landing
one-at-a-time none (each singleton’s own trial writes its lines)
catching up #N catch-up · CI on PR #N
released queued

Episode. trainEpisode (engine/train_episode.go) is created per merge-train run (one (repo, base) partition; since #2051 a run outlives its worker goroutine, and an adopted run rebuilds its episode — phase and phase start restored from the persisted record — so the row and the single History entry span the whole run), reachable as state.episode and trialParams.episode, nil-safe, with its own lock. It holds the original batch size, the held members, ejected and deferred members with short reasons, the current phase and its start, and the outcome facts. ejectMember is untouched (it also runs on the poll goroutine); ejections are recorded at worker-side call sites (noteTrainEjected / noteTrainDeferred), which refresh the row title and keep the phase clock. The live title is [base] N of M: #a #b (ejected #x) (deferred #y), lists capped at three plus +k.

Outcomes. Sites only record facts; the single tui.JobCompletedEvent (Skipped: false, Completed: true, Outcome, Detail, Success) is emitted when the run finishes (trainRun.finish, or directly when prepareTrainWorker fails — every exit) — exactly one History entry per episode, never zero, never two. resolve() applies a fixed precedence: abandoned (nothing landed and a fault/abandon cause was recorded; Success=false) > one-at-a-time > red → bisected > landed > ejected at assembly / ejected > dissolved (the loop ran out of members; nothing to land) > catch-all ended — nothing landed (Success=false). Success is true for everything except abandoned and the catch-all. Ejections and deferrals are always appended to the detail. “Landed” members are those the integration PR claims (and members already Done on replay), not survivors; the engine’s own main-moved dissolveBatch is reported as abandoned (cause: base moved) so dissolved keeps its “nothing to land” meaning. A restart mid-train produces no entry for the interrupted run.

Neutralisation. SetTrainOutcomeNeutralisedForTest restores the blanket Skipped: true completion; TestMergeTrainTUI_OutcomeTestFailsWhenNeutralised shows the per-outcome checks fail without the real emission.

Sibling (repo, base) partitions still share the repo-keyed row (ADR 1661 caveat, display-only); each partition’s episode emits its own History entry.


7.18 Stage Skill References (#2137, ADR 2137; runtime load failure #2138, ADR 2138)

Stage YAML names a skill as <plugin>:<skill> (split on the first colon, plugin.Split); the engine resolves each reference against the plugins actually loaded for the worker and the prompt carries the fully-qualified name.

Names. plugin.NamespaceSep (:) and plugin.BasePluginName (fabrik-stages since #2140, ADR 2140; it was fabrik) are each defined once (plugin/names.go), beside plugin.LegacyBasePluginName (fabrik), plugin.PMPluginID (fabrik@fabrik) and the single pre-rename table plugin.LegacyRef (fabrik:fabrik-X and bare fabrik-X → fabrik-stages:X, the twelve known names only). stages.loadOne rejects a malformed reference (empty plugin or skill half, whitespace) at load; whether the plugin exists is decided by the engine.

Catalogue. engine/skill_catalogue.go builds an immutable snapshot from the same directory list that becomes --plugin-dir (claudePluginDirs, §7 ADR 2136): per plugin, the manifest name (else the directory basename) and its skills, found as Claude Code finds them — skills/<dir>/SKILL.md, directories added by the manifest skills key (string or array, "." allowed, paths escaping the plugin root ignored), and a root SKILL.md as a single skill when there is no skills/ and no skills key. The skill identity is the directory name; commands/ entries are not skills. The snapshot lives on Engine.skillResolver (built in New() after resolvePluginDirs succeeds; nil on NewWithDeps engines) and is rebuilt once, under a mutex, on an unresolved lookup so a plugin edited or upgraded while the daemon runs is picked up.

Resolution (engine/skill_resolve.go, resolveStageSkills). A qualified reference resolves when its plugin is loaded and has the skill. A bare reference resolves to the loaded plugin that provides it — the plugin named plugin.BasePluginName if it does, otherwise the first in load order — and is flagged deprecated, naming any other providers. Anything else is unresolved. Legacy fallback (#2140, FR-004): the as-is lookup always runs first, so a customised, not yet migrated base plugin still named fabrik keeps its own fabrik-review; only on a miss is plugin.LegacyRef applied and the mapped name resolved, flagged deprecated (the resolver and fabrik upgrade share the one table, so they cannot disagree). An empty catalogue (no loaded plugin directory) passes every reference through unchanged. resolveStageSkills(stage, keys...) returns a copy of the stage carrying the resolved names (the configured stage is shared by concurrent workers and is never mutated), so buildPrompt and buildCommentReviewPrompt emit exactly the registered name.

Gates. Two choke points, each before any worker is spawned or reinvoke cycle spent:

Path Where Key resolved
Stage dispatch runInvocationWithExtension, after the suspension and apiKeyHelper gates skill
Comment review, CI-fix, rebase, review-reinvoke processCommentsClassified, right after the suspension gate (stageVariant has already swapped ci_fix_skill/rebase_skill into comment_skill) comment_skill

The merge-train conflict resolver’s stage has no skill keys and skips resolution.

Unresolved reference — a self-healing did-not-run. handleSkillUnresolved (engine/skill_unresolved.go): on the stage path StageAttempted is recorded (the dispatch cooldown applies) but StageRetryIncremented never is, and neither stage:<name>:failed nor fabrik:paused is applied; on the comment path the call returns didNotRunSkillUnresolved (so the reinvoke cycle charge is refunded, #1812) with a nil error (no comment breaker is fed). One comment naming the reference, the plugins searched and their paths is posted and fabrik:skill-unresolved applied, both gated on the label’s absence. The label is cleared by finalizeStageOutcome on the next invocation that reaches Claude, and clearSkillUnresolved removes the panel entry when a reference resolves again. The next cooldown-bounded dispatch re-resolves, so fixing the YAML or the plugin needs no human comment (the reason it is not a pause).

Runtime load failure — a pause, never a completion (#2138, ADR 2138). Pre-flight resolution cannot see a skill that is on disk but that Claude Code fails to register or load in the worker; a worker without its skill has no instructions, yet still prints FABRIK_STAGE_COMPLETE. runClaude therefore takes the invocation’s own configured skill (stage.Skill for stage dispatch, stage.CommentSkill for the comment/CI-fix/rebase paths) and, after interpretClaudeResult, calls detectSkillLoadFailure (engine/skill_load_detect.go) over the raw stream-json. The signal is structural only: an assistant tool_use named Skill whose input.skill matches the own skill (base names equal; when both sides are plugin-qualified the plugins must be equal; a bare side matches any plugin), and a tool_result for its tool_use_id with is_error: true. Content text is never used to decide, only quoted. It is a failure only when an own-skill call failed and none succeeded; no call, a call without a result, unparseable lines, sub-agent traffic (parent_tool_use_id set) and unrelated failing Skill calls or tools never fire, and a stage without a skill is inert. skillLoadOverride applies it to an unclassified outcome and to the marker-wins non-zero exit, so the marker no longer wins; usage-limit, api-error, tools-denied, turn-limit and resume-failure classifications keep precedence. The result is a claudeerr.SkillLoadError, handled by handleSkillLoadFailed (engine/skill_load_failed.go): one comment through pauseIssue (fabrik:paused + fabrik:awaiting-input) naming the stage, skill, the name the worker called, skillResolver.describe() (plugins loaded) and the sanitised error; a panel entry skill_load:<stage>.<key>; and EnginePaused. Unlike the pre-flight case this does not self-heal (resolution already passed), so it is a pause rather than a cooldown. It records no StageRetryIncremented and no stage:<name>:failed, and skips commitWIP and the push. On the stage path finalizeStageOutcome handles it beside skillUnresolvedError; on the comment path processCommentsClassified returns didNotRunSkillLoadFailed (cycle refunded via classifyDidNotRun, #1812; no comment breaker fed). The human lifts the pause by removing fabrik:paused and fabrik:awaiting-input; the panel entry clears on the next invocation of that stage and key that reaches Claude without the failure (clearSkillLoadFailed).

Operator surfaces (engine/skill_report.go, ADR 2135). reportStageSkills, called from Run() beside WarnUndeclaredReviewers, checks every skill-valued key of every stage and then Reconciles two panel types: skill_reference_deprecated:deprecated (LevelNotice, #2140; formerly subject bare; one entry and one log line listing every bare or pre-rename fabrik:fabrik-* reference as stage.key → qualified name, pointing at fabrik upgrade) and skill_reference:<stage>.<key> (LevelWarning; reference, reason, plugins searched with their paths). A dispatch-time failure reports the same entry.

Drift. stages/drift.go compares top-level key presence only, so a bare-vs-qualified difference is never reported as drift (pinned by TestWarnStageDrift_BareVsQualifiedSkillIsNotDrift). The embedded stage defaults are written qualified (fabrik-stages:<stage>, #2140).

7.19 Workflow Migration and local-workflows (#2139, ADR 2139)

fabrik upgrade — never the daemon — migrates a deployment to fully-qualified skill references and moves operator edits of the deployed plugin out of Fabrik’s own plugin name. It decides from what is on disk and a per-file table of shipped versions; there are no version-number checks, so a deployment that skipped releases reaches the same end state, and a second run changes nothing.

Per-file shipped table. plugin/shipped_files.json (embedded; plugin/shipped_files.go, IsShippedFile) maps each relative plugin file to every sha256 any release shipped; the current embed always counts as shipped at runtime. KnownEmbeddedVersions holds only aggregate fingerprints and cannot answer “does this one file differ from every shipped version”. The table is generated from the v* tags by tools/gen-shipped-files; scripts/cut-release.sh runs -add-current and -check beside the KnownEmbeddedVersions step and commits the file. Between releases the table need not cover the current embed (it always counts as shipped at runtime), so a PR that edits a plugin file does not touch the table.

Classification (plugin/classify.go). ClassifyDir hashes every file under a plugin tree (.installed-version excluded) and lists the unknown ones; a skill is edited when any file under skills/<name>/ is unknown (a skill only in the tree is edited). A tree is moved only when CheckPluginState reports customWorkflow and a file is unknown: a mixture of shipped versions is refreshed in place, and a dev-build tree (ADR 1297) is never moved.

Plan, then apply (cmd/upgrade_migrate.go). planMigration reads the disk and writes nothing; Pending() is true when it would change something. Skipped outright (reported, nothing moved) when FABRIK_PLUGIN_DIR is set or .fabrik/plugin is a symlink; blocked (nothing changed, the old refusal follows) when a move is needed but .fabrik/plugins/local-workflows exists — it is never overwritten, or when the manifest’s top-level name cannot be renamed safely (checked at plan time by checkManifestRenamable, before anything is moved). applyMigration runs in an order whose every intermediate state the next plan recognises: (1) rename .fabrik/plugin → .fabrik/plugins/local-workflows; (2) drop its .installed-version and change only the manifest name to plugin.LocalWorkflowsName (textual substitution of the top-level name token only — a nested author.name is never touched — verified by decoding both versions); (3) restore the base with RefreshPluginTo + WriteInstalledVersion; (4) rewrite the stage files. On a resume the edited skills are read from the existing local-workflows tree, and that directory is touched (manifest name, .installed-version) only when it is recognisably a moved base copy — its manifest still names plugin.BasePluginName, or is missing; a manifest naming any other plugin is the operator’s own, left byte-for-byte alone with a report note, and no stage is pointed at it. The base name comes only from plugin.BasePluginName.

Stage files (stages/skillrefs.go, RewriteSkillRefs). Only column-0 skill/comment_skill/ci_fix_skill/rebase_skill lines are edited, in place, so comments, quoting and CRLF survive and block-scalar text can never match; the original and result are both decoded and the only permitted difference is the rewritten values. A bare edited skill → local-workflows:<s>; a bare shipped skill → <BasePluginName>:<s>; a bare name a project plugin provides → that plugin (below); a bare name nothing provides stays bare and is reported (and does not keep the migration pending); a current qualified reference is never rewritten (if it names an edited base skill, the report names the local-workflows copy). Pre-rename forms (#2140, ADR 2140): fabrik:fabrik-X and bare fabrik-X go through plugin.LegacyRef — an unedited skill becomes fabrik-stages:X; an edited one becomes local-workflows:fabrik-X (the moved copy keeps its old skill names) when a usable local copy exists, otherwise it is left alone with a note, never silently pointed at the shipped copy. local-workflows:* and other plugins’ references are untouched. Project plugins (#2203, ADR 2140 amendment): a bare fabrik-X (and any bare non-shipped name) is first looked up in the project-plugin set (cmd/upgrade_projectplugins.go), the plugins the daemon would load besides the base: the operator’s own local-workflows (manifest naming something other than local-workflows/fabrik/fabrik-stages), then the plugin_dirs extras in order — FABRIK_PLUGIN_DIRS if set, else plugin_dirs in <dir>/.fabrik/config.yaml (.env is loaded first, as the daemon does, an already-set variable winning; there is no --plugin-dirs flag). Name and skill set come from engine.DescribePlugin (the resolver’s own discoverPlugin), the first provider in load order wins (as resolveAsIs’s providers[0] once the base is excluded) and a report note names it and any others. The base under either name, an extra named local-workflows, and the moved/about-to-be-moved copy of the base are never project plugins, so edited skills still go to local-workflows:… and unedited ones to fabrik-stages:X unless a project plugin provides them. An unreadable extra, a missing manifest or a duplicate name is skipped with a note, never an error. The edited-skill check runs first: a base skill the operator edited goes to local-workflows:… even when a project plugin also provides the name (the edit lives in the moved copy, which loads ahead of the extras; pinned by TestProjectPlugin_EditedBaseSkillOutranksExtra). The qualified pre-rename form fabrik:fabrik-X is not looked up. projectPluginLookupDisabledForTest restores the always-base rewrite. The “is this a moved base copy” manifest checks accept both LegacyBasePluginName and BasePluginName. Pruning: RefreshPluginTo (used by the base restore, the re-exec auto-refresh and fabrik init) also deletes files that are byte-identical to some shipped version (IsShippedFile) but absent from the current embed, then empty directories, so the rename leaves no duplicate skills/fabrik-*; edited files and .installed-version are never removed. Worker exclusion: buildClaudeArgs adds --settings {"enabledPlugins":{"fabrik@fabrik":false}} to every worker so the PM plugin never loads (docs/stage-lifecycle.md).

Engine. resolvePluginDirs loads <fabrikDir>/.fabrik/plugins/local-workflows right after the base and before plugin_dirs, through the same validation (readable manifest, unique name, path de-duplication); a copy whose manifest still says fabrik is refused loudly. The dev-build PostBuildHook runs fabrik upgrade --no-migrate so unattended upgrades never move files or edit tracked YAML; --no-migrate keeps the pre-#2139 refusal exactly.

Startup. After the plugin state check, cmd/root.go computes the plan (skipped under --plugin-dir) and reportPluginState records plugin_state:migration_pending through warnings.Reconcile; it clears on the first start after the migration ran. The daemon changes nothing.

8. Invalid / Unexpected States

The engine handles unexpected label combinations (from manual human manipulation) through its guard chain. The behavior is defined by the order of checks in itemMayNeedWork(), itemNeedsWork(), and processItem().

8.1 Guard Chain Order in processItem()

Guards are checked in this order. The first matching guard determines behavior:

Priority Guard Check Behavior
1 No matching stage FindStage(stages, item.Status) == nil Skip (return nil)
2 Repo not ready ensureRepoReady() fails Skip (ErrSkipItem) or return error
3 Locked by other user fabrik:locked:<other> present Skip with log
4 Editing fabrik:editing present Skip with log (defense-in-depth; primary gate is in itemNeedsWork)
5 Awaiting input + comment isAwaitingInput() + new human comments created at/after the pause (resumeAuthorised()) Unblock → comment processing
6 Awaiting input, no comment isAwaitingInput() Skip with log
7 Paused + comment fabrik:paused + new human comments created at/after the pause (resumeAuthorised()) Unpause → fall through
8 Paused, no comment fabrik:paused Skip with log
9 Dependencies blocked checkDependencies() returns true Skip (label + comment handled by checkDependencies)
10 Cleanup stage stage.CleanupWorktree Remove worktree, add complete label
11 Failed label + unpause detection stage:<X>:failed present or snap.PausedByEngine(stage.Name) clearFailedStage() then continue
12 New comments findNewComments() non-empty processComments()
13 PR item item.IsPR Skip (PRs only support comments)
14 Stage complete stage:<X>:complete present Skip
15 Cooldown active snap.LastAttemptAt(stage.Name) within cooldown window Skip
16 (all guards pass) — Acquire lock → invoke Claude

8.2 Notable Unexpected Scenarios

fabrik:editing without active comment processing: If fabrik:editing is left orphaned by a prior crash (no active processComments goroutine), the engine skips the item — first at the itemNeedsWork pre-dispatch gate (preventing goroutine launch), then at guard 4 in processItem (defense-in-depth). On restart, runStartupCleanup() automatically removes stale fabrik:editing labels from items with no active Worker — the same startup self-healing mechanism that handles fabrik:locked:<user>. Both labels are cleaned up in parallel on restart, so a crashed Fabrik instance leaves no permanent stuck state. If a human manually applies fabrik:editing, the label must be manually removed for processing to resume (startup cleanup only runs once per restart and only for items with no active Worker).

fabrik:reworking:<Stage> without active comment processing (#1802): If fabrik:reworking:<Stage> is left orphaned by a prior crash mid-rework, the item’s completion claim may be either still cleared or already durably resolved, depending on exactly where the crash landed — finalizeComments only removes this marker after handleStageComplete has made its own completion decision on a completing exit, not before (§4.2 step 8b/11). On restart, runStartupCleanup()’s third pass (mirroring the fabrik:editing pass immediately above it) reads the reworked stage’s name directly out of the marker label’s own suffix — <Stage> in fabrik:reworking:<Stage> — never from the item’s current board Status. This is a deliberate correction (a Pruefer review finding, #1802): a completing rework that also auto-advances the board (yolo/cruise/auto_advance: true) moves Status to the next stage synchronously inside handleStageComplete, before finalizeComments’s deferred removal of this marker ever runs. A crash in that window would leave Status pointing at a stage that never actually ran; resolving the reworked stage from Status there would falsely stamp stage:<NextStage>:complete onto it, which itemNeedsWork then reads as “already done,” silently and permanently skipping that stage’s real dispatch. Reading the stage name from the label itself avoids this entirely, regardless of how far Status has since moved. Using that resolved stage name, the pass first checks whether stage:<Stage>:complete or fabrik:awaiting-ci is already present: if either is found, the rework already settled correctly before the crash and only the stale marker is removed; otherwise it restores stage:<Stage>:complete directly (the crash landed before any completion decision was recorded — indistinguishable from, and treated the same as, a non-completing exit) and then removes fabrik:reworking:<Stage>. Either way this only runs for an item with no active Worker. If the marker is malformed (no stage name encoded in the suffix), it is left in place (logged loudly) rather than silently dropping the only remaining signal that a restore is owed — see ADR-1802.

stage:<X>:complete without board column advancement: The item is skipped by guard 14 in processItem(). The catch-up loop will attempt to advance it if yolo/cruise/autoAdvance is active. Without auto-advance, it waits for a human to move the board column.

fabrik:paused on a complete item: The catch-up loop checks fabrik:paused and skips paused items. The item will not be advanced until unpaused.

fabrik:awaiting-review without stage:<X>:complete: The catch-up loop only processes items with the complete label, so fabrik:awaiting-review alone has no effect in the catch-up path. In processItem(), the label is not checked — it’s only relevant in the catch-up loop.

stage:<X>:failed without fabrik:paused: processItem() guard 11 detects this as an “unpause” scenario — the user has already removed fabrik:paused, so clearFailedStage() resets retry state and processing continues.

Multiple stage:<X>:in_progress labels: No special handling. Each is independent. The engine only checks the in_progress label for the current stage’s column.


9. Concurrency Model

9.1 Semaphore

Engine.sem is a buffered channel of size MaxConcurrent (default 5). With merge_train: on there is also Engine.trainSem, a separate one-slot channel used only by the merge train (#2115, Reserved train slot); stage work never touches it. The dispatch loop, dispatchReviewReinvoke(), and dispatchCIFixReinvoke() all acquire slots from this semaphore before invoking Claude. The dispatch loop’s acquire is non-blocking (#2114, §9.9.1) — a full semaphore defers the candidate and the poll carries on; the reinvoke dispatchers acquire inside their own goroutine. The merge train takes a slot only around each of its own Claude conflict-resolution invocations (acquireTrainSlot, #2046) — never for its lifecycle (see “Merge-Train Serialization + Main-Moved Recovery”, Worker-slot scope).

9.2 Worker In-Flight Guard (formerly inFlight Map)

Engine.inFlight (sync.Map) has been removed (Fix B, issue #544). The dispatch guard is now entirely Store-backed.

The dispatch loop uses snap.Worker() != nil to detect whether a goroutine is already running for an item. WorkerEntered{Repo, Number, StageName, StartedAt} is applied synchronously before wg.Add(1) and before the goroutine is launched, so the store guard is effective from the instant the goroutine is scheduled. The reinvoke dispatchers (dispatchReviewReinvoke, dispatchCIFixReinvoke, dispatchRebaseReinvoke) follow the same pattern. WorkerExited is deferred at the top of each goroutine in all four dispatchers — main (engine/poll.go) and the three reinvoke dispatchers — so it fires on every exit path, including processItem’s many early-return guards (paused, blocked, awaiting-input, locked-by-other, stage-complete, etc.).

Version-skew watchdog (#1074): Piggybacked on the same idle+no-in-flight gate described above, Engine.checkVersionSkew() (engine/upgrade.go) runs immediately before checkAndUpgrade() whenever AutoUpgrade is enabled and idleUpgradeThreshold is reached — no new counter or threshold. It resolves the on-disk executable path (versionSkewExecutableFn → os.Executable + filepath.EvalSymlinks — engine’s own private seam, distinct from internal/selfupgrade’s executableFn used by PerformReleaseUpgrade/CheckAndRebuildDev since ADR-1196 moved the self-upgrade trigger logic to its own package), spawns <exe> --version with a 5s timeout, and compares the trimmed output to e.cfg.Version. A mismatch is a general “the code on disk has moved on from what’s running” signal — it catches a SemverGreater bug, a syscall.Exec failure after a successful binary replace, or an externally-replaced binary (e.g. a fleet sharing ~/go/bin/fabrik) uniformly, without needing a dedicated warning per cause. The mismatch is recorded as a persistent warnings.Entry (Type: "version_skew", keyed by the resolved executable path, FixAction: "shell_command" suggesting kill -HUP <pid>) via the same warnings.Record/warnings.Clear pattern as checkAllowAutoMerge (ADR-052), so it survives process restarts and surfaces in the TUI Warnings panel. A matching version clears any existing entry for that key. All failures resolving the path or running the subprocess are logged and non-fatal — the check is simply skipped for that poll.

Source-checkout staleness reporting (#1464): deliberately not piggybacked on the idle gate. Unlike the version-skew watchdog above, Engine.checkSourceStaleness() (engine/staleness.go) does not run inside the idle+no-in-flight branch — it is called unconditionally from poll(), before the dispatched == 0 check, throttled internally by its own poll-count gate (maybeCheckSourceStaleness, every stalenessCheckPollInterval = 30 polls, ≈15 minutes at the default 30s --poll interval, firing on the very first poll too). This is the fix for the gap the version-skew watchdog and checkAndUpgrade both share: both require idleUpgradeThreshold consecutive fully-idle polls to ever run, so a board that dispatches work (or has a merge-train worker in flight) on every single poll never reaches either one — the daemon can run arbitrarily stale code indefinitely with no signal. Since checkSourceStaleness only reports (via selfupgrade.CompareDevBuild, which never calls git pull, go build, or execFn — see its doc comment), it has no in-flight-worker precondition to honor: there’s nothing here that would kill a live Claude invocation, unlike checkAndUpgrade’s syscall.Exec. CompareDevBuild is the same comparison selfupgrade.CheckAndRebuildDev uses to decide whether to actually rebuild, so the staleness warning and the real upgrade decision are structurally unable to disagree. Applicability mirrors checkAndUpgrade’s own fork (e.cfg.Version must start with "dev", then CompareDevBuild’s own IsSourceCheckout gate) — a release build never calls into selfupgrade for this check at all, so it never runs a git fetch. Recorded via the same warnings.Record/warnings.Clear pattern as version_skew, under the single fixed key "source_staleness" (§7.11 above); the FixAction differs depending on e.cfg.AutoUpgrade, since kill -HUP only self-heals staleness when the startup-time checkAndUpgrade() call that a SIGHUP re-exec re-triggers is itself enabled.

Safe-point-gating race (investigated for #1074, none found): the concern was whether a webhook-triggered concurrent poll could dispatch a new worker while a synchronous checkReleaseUpgrade() download+exec is in flight on another poll cycle. It cannot: Engine.Run()’s single for { select {...} } loop is the only caller of doPollCycle(); webhook wakes are delivered through e.wakeCh and consumed serially by that same loop, so checkAndUpgrade() (and its syscall.Exec) always runs synchronously on the main goroutine between poll cycles — never concurrently with worker dispatch from a different cycle. The in-flight check at the top of this section (Store.HasInFlightWorker(), covering both snap.Worker() != nil over e.store.All() and the repo-scoped merge-train registry) already detects workers dispatched by any prior poll cycle regardless of which cycle started them, so no additional guard was needed. This also covers the same-cycle case: dispatchMergeTrainWorker calls Store.EnterRepoWorker synchronously before returning, so the very poll cycle that just launched a merge-train worker (which dispatchCandidates does not count in its dispatched return value) no longer misreports itself as idle.

Both mutations emit WorkerChanged | WorkerLifecycleChanged (§2.14). Only WorkerLifecycleChanged is in wakeChFlags, so worker entry and exit wake the poll loop via newWakeChObserver. WorkerHeartbeat and WorkerPIDSet emit only WorkerChanged and do not feed the wake pipeline.

WorkerLifecycleChanged remains in wakeChFlags (wake channel still fires on goroutine entry/exit) but is excluded from cycleSetFlags (used by newMayNeedWorkObserver). This prevents early-return goroutine exits — e.g. from a dep-blocked item — from bypassing the cooldown gate for items that did no useful work. Non-blocked items are still re-evaluated promptly: the wake channel fires, the poll loop runs, and any item that passes itemNeedsWork is dispatched. See §9.9 (pre-dispatch label gates and wake-loop avoidance) and ADR-039.

Wake channel scope: the wakeChObserver is registered only when a wake channel is configured (TUI mode or any other configuration that calls SetWakeCh). In headless / --notui runs, WorkerLifecycleChanged does not populate mayNeedWork (it is excluded from cycleSetFlags — see Fix B, issue #576) and does not trigger an immediate wake (no wake channel). Re-evaluation in headless runs is driven by StatusChanged/LabelsChanged events (which ARE in cycleSetFlags) or the next ticker-driven poll (within PollSeconds). The dispatch re-evaluation guarantee is therefore “by the next scheduled poll” (≤ PollSeconds) for worker-exit events in headless mode, and “within milliseconds” in TUI mode (wake channel fires on WorkerLifecycleChanged via wakeChFlags).

9.3 Worktree Mutex

Git operations that write .git/config are not concurrent-safe. WorktreeManager.mu serializes worktree creation and updates within a single repo.

9.4 Catch-Up vs Dispatch Ordering

Within a single poll() call:

  1. Catch-up loop runs first — processes items with stage:<X>:complete labels for yolo/cruise advancement, review gate evaluation, and review reinvoke dispatch
  2. Dispatch loop runs second — processes items that need stage invocations or comment processing

The advancedItems map tracks items that the catch-up loop advanced during this poll cycle. Items in advancedItems are excluded from the deferred CooldownAt["periodic-re-eval"] refresh at the end of poll(), so they appear in the next poll cycle’s cycleSet naturally (via the observer that fires when their status changes) rather than being suppressed by cooldown; their expired cooldown (if any) is consumed at admission (#2096), so the status change — not a leftover expiry — is what admits them (#544). Items dispatched by dispatchReviewReinvoke() in the catch-up loop are guarded from double-dispatch by the dispatch loop via snap.Worker() != nil (see §9.2).

9.5 Engine.mu Mutex

Engine.mu (sync.Mutex) protects in-memory state that is not covered by its own synchronization primitive: totalTokens, lastReportedCost. Critical sections are kept small — typically a single map read/write. Engine.mayNeedWork is protected by its own Engine.mayNeedWorkMu (a separate mutex) so that observer callbacks writing to mayNeedWork from any goroutine don’t contend with Engine.mu-held code paths.

Per-item engine state previously stored in Engine.mu-protected maps has been migrated to itemstate.Store (Phase 3-E and 3-F; see ADR-036). Those fields — Lock, LastTokenUsage, LastInvocationCompleted, LastInvocationBlocked, LastDeepFetchFailureAt, LinkedPR.HasHadChecks, LinkedPR.CIMergePendingSince, plus the Phase 3-F fields StageState.LastAttemptAt, StageState.Attempts, StageState.PausedByEngine, StageState.ReviewCycles, StageState.CIFixCycles, StageState.RebaseCycles, StageState.ProcessedComments, and ItemState.CooldownAt — are now read via e.store.Get(repo, n) (returning an immutable Snapshot) and written via e.store.Apply(Mutation). The Store has its own internal mutex; no Engine.mu guard is needed for these fields.

9.6 Engine Internal State (itemstate.Store)

Per-item state lives in a single shared *itemstate.Store instance. The Engine creates it (sharedStore := itemstate.NewStore(nil) in New()), assigns it to eng.store, and passes it to boardcache.NewCacheImpl. All mutations — engine-side (locks, invocations, stage state) and webhook/reconcile-side (status, labels, comments, linked-PR fields) — flow through sharedStore.Apply. NewWithDeps (test factory) constructs its own independent store because it never creates a CacheImpl.

Phase 5 F2 (issue #562): The previously separate linkedPRs map[string]*gh.PRDetails and prNumToKey map[string]string CacheImpl maps were migrated into itemstate.Store: PR detail fields (Title, State, Merged, Draft) are now stored in LinkedPRState via PRDetailsUpdated mutations, and the PR→issue reverse-index is store.prToKey (populated by PRHeadSHAUpdated / PRDetailsUpdated; queried via store.GetByPRKey).

Phase 5 F4 (issue #563): The previously separate checkRuns map[string][]gh.CheckRun CacheImpl map (check-run lists keyed by commit SHA) was migrated into itemstate.Store as pendingCheckRuns map[string][]gh.CheckRun. This pre-linkage buffer holds check runs for SHAs not yet linked to any item. When PRHeadSHAUpdated fires (establishing the SHA→item mapping), the buffer is drained into LinkedPR.CheckRuns via upsert-by-ID. FetchCheckRuns now reads from store.CheckRunsBySHA, which returns the union of pendingCheckRuns[sha] and the linked item’s LinkedPR.CheckRuns; on total miss it falls back to GitHub and populates the Store via CheckRunCompleted mutations. See §9.8 for CheckRunChanged flag semantics.

Per-SHA pruning (#958 leg 3): gh.CheckRun carries no SHA field, so LinkedPR.CheckRuns is a flat, per-item (not per-SHA) slice — CheckRunsBySHA(sha) only used sha to resolve which item to read from shaToKey, then returned that item’s entire CheckRuns accumulation regardless of which SHA was actually requested. This let a run from a prior push shadow the current SHA’s classification indefinitely on a webhook-less deployment (no check_run event ever prunes it). PRHeadSHAUpdated now clears LinkedPR.CheckRuns whenever the head SHA genuinely changes (v.SHA != item.LinkedPR.HeadSHA && item.LinkedPR.HeadSHA != "" — excludes first linkage), restoring the informal “this only reflects the current SHA” invariant without a full map[SHA][]CheckRun storage rewrite (deferred to #957 if ever needed).

Field ownership is by mutation type, not by store identity: a reader calling store.Get("owner/repo", 1) receives a Snapshot with all field groups populated regardless of which code path applied each mutation. This is the single-Store design originally proposed in ADR-036 and completed in Phase 5 F3 (issue #537).

The following fields are stored in ItemState / LinkedPRState and accessed exclusively through the store:

ItemState field Mutation (write) Snapshot accessor (read) Former Engine map
Lock (LockState) LocalLockAcquired, LocalLockReleased snap.Lock() e.lockedIssues[iKey]
LastTokenUsage InvocationRecorded{Usage: ...} snap.State().LastTokenUsage e.lastUsage[iKey]
LastInvocationCompleted InvocationRecorded{Completed: ...} snap.State().LastInvocationCompleted e.lastCompleted[iKey]
LastInvocationBlocked InvocationRecorded{Blocked: ...} snap.State().LastInvocationBlocked e.lastBlocked[iKey]
LastInvocationTurnLimited InvocationRecorded{TurnLimited: ...} snap.State().LastInvocationTurnLimited N/A (new in #1178)
LastDeepFetchFailureAt DeepFetchFailed{At: ...} (set); ItemDeepFetched (clears) snap.State().LastDeepFetchFailureAt e.deepFetchFailureTime[iKey]
LinkedPR.CheckRuns CheckRunCompleted (when SHA is already in shaToKey); PRHeadSHAUpdated drain (flushes pendingCheckRuns[sha]); PRHeadSHAUpdated prune (clears the slice when the head SHA genuinely changes, not on first linkage — #958 leg 3); fallback populate in FetchCheckRuns snap.LinkedPR().CheckRuns CacheImpl: c.checkRuns[sha] (Phase 5 F4: migrated to Store in #563)
LinkedPR.LastCIFixNoOpSHA CIFixNoOpRecorded{SHA} (#958 leg 2) snap.LastCIFixNoOpSHA() N/A (new in #958)
LinkedPR.HasHadChecks PRChecksObserved (REST path); CheckRunCompleted and PRHeadSHAUpdated drain (webhook path) snap.LinkedPR().HasHadChecks e.prHasHadChecks[iKey]
LinkedPR.CIMergePendingSince CIMergePendingStarted{At: ...} (set); CIMergePendingCleared (clear) snap.LinkedPR().CIMergePendingSince e.ciMergePendingSince[iKey]
LinkedPR.Number, LinkedPR.HeadSHA PRHeadSHAUpdated{LinkedPRNum, SHA} snap.LinkedPR().Number, snap.LinkedPR().HeadSHA CacheImpl: c.prNumToKey[pk] (routing), c.linkedPRs[pk].HeadSHA (Phase 5 F2: migrated in #562)
LinkedPR.ValidateCompletedSHA ValidateCompletedAtSHA{Repo, Number, SHA} (set); ValidateCompletedAtSHACleared{Repo, Number} (clear) snap.ValidateCompletedSHA() N/A (engine-internal; not persisted across restarts)
LinkedPR.Title, LinkedPR.State, LinkedPR.Merged, LinkedPR.Draft PRDetailsUpdated{Title, State, Merged, Draft} snap.LinkedPR().Title, etc. CacheImpl: c.linkedPRs[pk].* (Phase 5 F2: migrated in #562)
LinkedPR.Facts, LinkedPR.CIVerdict (#2122) PRFactsObserved{PRNumber, Facts} (the probe apply and settlePRFacts’ REST fallback); CIVerdictObserved{Verdict} (settlePRFacts); CIVerdict is cleared by PRHeadSHAUpdated on a genuine head change and by PRFactsObserved for a merged or closed PR. Flag PRFactsChanged, not a wake flag snap.LinkedPR().Facts, .CIVerdict (local API and attention only; no gate reads them) N/A (new in #2122)
StageState.LastAttemptAt StageAttempted{StageName, At} (set); StageLastAttemptCleared (clear) snap.LastAttemptAt(stageName) e.processedSet[stageKey] (invocation timestamp)
ItemState.CooldownAt CooldownRecorded{Reason, Until} snap.CooldownAt(reason) e.processedSet[stageKey] (cooldown timestamp; same key, different semantic — now split)
StageState.Attempts StageRetryIncremented (increment); StageRetryCleared (reset) snap.Attempts(stageName) e.retryCount[stageKey]
StageState.PausedByEngine EnginePaused (set); EngineUnpaused (clear) snap.PausedByEngine(stageName) e.pausedDueToRetries[stageKey]
StageState.ReviewCycles ReviewCycleIncremented (increment); EngineCyclesCleared (reset) snap.ReviewCycles(stageName) e.reviewCycleCount[stageKey]
StageState.CIFixCycles CIFixCycleIncremented (increment); EngineCyclesCleared (reset) snap.CIFixCycles(stageName) e.ciFixCycleCount[stageKey]
StageState.RebaseCycles RebaseCycleIncremented (increment); EngineCyclesCleared (reset) snap.RebaseCycles(stageName) e.rebaseCycleCount[stageKey]
StageState.DidNotRunReinvokes DidNotRunReinvokeRecorded (increment); DidNotRunReinvokesReset (reset on a charged reinvoke); EngineCyclesCleared (reset) snap.DidNotRunReinvokes(stageName) —
StageState.ProcessedComments CommentProcessed{CommentID, At} snap.CommentProcessed(commentID) e.processedSet["…comment-ID"]
Worker (*WorkerHandle) WorkerEntered{StageName, StartedAt, Kind, Trigger, Queued, QueuedSince} (placeholder, before goroutine launch; Queued for a reinvoke that holds no slot yet — #2121); LocalLockAcquired{Worker: &WorkerHandle{...}} (full details, after lock acquired — stage dispatch); WorkerSlotAcquired{At} (a queued reinvoke now holds a slot: clears Queued, re-stamps StartedAt/LastSignAt, no lock claimed — #2179/#2121); WorkerKindSet/WorkerTriggerSet/WorkerTurnsRecorded (kind, trigger, live turns — #2121); WorkerPIDSet{PID} (update PID); WorkerHeartbeat{At} (update LastSignAt); WorkerExited (clear) snap.Worker() N/A (new in Phase 3-G; WorkerEntered added in Fix B, #544)

mayNeedWork (Phase 3-H): The map e.mayNeedWork map[string]bool (protected by e.mayNeedWorkMu) is the Phase 3-H replacement for the removed e.seenUpdatedAt map. It is populated by the mayNeedWorkObserver registered once on the shared store whenever a Change includes any wakeChFlag. All mutation types — engine-side and webhook/reconcile-side — now fire through the same observer registration. Each poll cycle drains the map into a local cycleSet — only items in the set (or with bypass conditions) proceed to deep-fetch evaluation. See section 9.8 for the full observer pattern description.

See also: ADR-036 (adrs/036-reactive-cache-single-owner.md) for the full rationale and the Phase 5 F3 addendum documenting the unification. ADR-038 (adrs/038-dual-store-observer-wiring.md) documents the historical dual-store registration design (superseded by the unification).

9.7 Worker Liveness (Heartbeat and Stale-Lock Recovery)

Phase 3-G adds a heartbeat-based liveness system that allows the engine to detect and recover from stale fabrik:locked:<user> labels left by crashed worker processes.

WorkerHandle Struct

type WorkerHandle struct {
    PID        int       // Claude subprocess PID (0 until cmd.Start() returns)
    StageName  string    // name of the stage being invoked
    StartedAt  time.Time // time LocalLockAcquired was applied
    LastSignAt time.Time // time of the most recent WorkerHeartbeat mutation
}

ItemState.Worker is non-nil while a worker goroutine is in flight; nil when no worker is active. snap.Worker() returns a deep copy (nil-safe).

Heartbeat Protocol

Every Claude-spawning dispatch path starts a heartbeat goroutine at dispatch time:

Dispatch site File
processItem() — main stage invocation engine/item.go
dispatchReviewReinvoke() — review comment processing engine/reviews.go
dispatchCIFixReinvoke() — CI failure re-processing engine/ci.go
dispatchRebaseReinvoke() — rebase conflict re-processing engine/merge_gate.go

Lifecycle per dispatch path:

This is the engine-state lifecycle (goroutine boundary). A parallel TUI-presentation lifecycle (JobStartedEvent/JobCompletedEvent) fires at the work boundary — inside processItem/processComments past all early-return guards — not at goroutine launch. See Appendix E for the full comparison.

  1. Before goroutine launch: store.Apply(WorkerEntered{StageName, StartedAt}) — placeholder Worker set so snap.Worker() != nil is true from the instant the goroutine is scheduled. WorkerEntered emits WorkerLifecycleChanged, which wakes the poll loop.

  2. Inside goroutine (after lock acquired): store.Apply(LocalLockAcquired{Worker: &WorkerHandle{StageName, StartedAt, PID: 0}}) — full Worker details replace the placeholder (PID=0 until subprocess starts). processItem also emits JobStartedEvent here (at the lock-acquired boundary) and defers JobCompletedEvent{Skipped:true}.
  3. A done := make(chan struct{}) is created; defer close(done) is set on the dispatch goroutine.
  4. go startHeartbeat(ctx, repo, number, done) — heartbeat goroutine starts. It applies WorkerHeartbeat{At: time.Now()} every 30 seconds until done is closed or ctx is cancelled. WorkerHeartbeat emits only WorkerChanged (not WorkerLifecycleChanged) so it does not wake the poll loop.
  5. Claude is invoked. After cmd.Start(), opts.OnPIDReady(pid) applies WorkerPIDSet{PID: pid} — the Claude subprocess PID is recorded in Worker. Emits only WorkerChanged.
  6. When the dispatch goroutine exits (defer): close(done) stops the heartbeat goroutine; store.Apply(WorkerExited{}) sets Worker = nil. WorkerExited emits WorkerLifecycleChanged, deterministically waking the poll loop.

Reinvoke registration (#2179, ADR 2179). A review/CI-fix/rebase reinvoke (dispatchReinvoke) applies WorkerEntered before the goroutine, as a queued worker, then WorkerSlotAcquired the moment it holds a semaphore slot (clearing Queued and re-stamping StartedAt/LastSignAt so the PID<=0 detector below measures from then; the detector skips a queued handle, #2121), and WorkerExited in the goroutine’s defer on every exit path. It never applies LocalLockAcquired/LocalLockReleased and never adds a fabrik:locked:<id> label: Lock mirrors that label and is therefore set only by stage dispatch (the reinvoke’s cross-instance marker is fabrik:editing; in-process exclusion is Worker() != nil). cleanupLockedIssues consequently sees only real stage locks.

No-op semantics:

Stale-Worker Detector (Runtime Crash Case)

startWorkerDetector(ctx) launches a background goroutine in Run() that scans for stale workers every 60 seconds.

Detection requires BOTH conditions (implemented by isWorkerStale(w, threshold)):

  1. time.Since(Worker.LastSignAt) > WorkerStaleTimeout (heartbeat is stale), AND
  2. Signal-0 confirms the process is dead (kill -0 PID returns ESRCH)

The “both conditions” requirement prevents spurious clearing of live workers whose heartbeat goroutine is delayed under system load. If the heartbeat is stale but the PID is alive, the engine logs a warning and re-checks on the next scan cycle.

WorkerStaleTimeout (default 5 minutes) is configurable via --worker-stale-timeout <N> (minutes) or FABRIK_WORKER_STALE_TIMEOUT=N. Must be longer than heartbeatInterval × 2 (currently > 60 s).

PID<=0 timeout-based clear (issue #1303): Workers with PID <= 0 (PID not yet set, or invalid) are in the narrow window between LocalLockAcquired and cmd.Start() — signal-0 has no PID to target, so the ordinary confirmed-dead check cannot run for them at all. An unconditional skip here would let a dispatch goroutine that hangs before onPIDReady fires (e.g. stuck in ensureRepoReady, before the child process is even started) outlive its own WorkerEntered marker indefinitely — and the fabrik:locked:<user> / stage:<name>:in_progress labels it gates — permanently suppressing dispatch for the item. Instead, the detector applies WorkerStaleTimeout against Worker.StartedAt (not LastSignAt, which may never have started ticking for a worker that hung before acquireLockAndVerify started the heartbeat goroutine): once a PID<=0 worker has been running longer than WorkerStaleTimeout, it is a candidate for a timeout-based clear — not a confirmed-dead clear, since nothing here has verified the process is actually gone.

Sentinel-probe verification tier (issue #1779): Before a PID<=0 worker past the threshold is actually cleared, the detector probes for a live process carrying that worker’s deterministic sessionNameSentinel (fabrik:<owner>/<repo>#<issue>:<stage> — see “Worker Session Naming” in docs/USER_GUIDE.md/docs/stage-lifecycle.md), already passed as --name on every worker invocation. This closes the gap #1303 left open: a real claude subprocess can be alive and progressing (the reported incident: cleared after 5 minutes, ran a further 1h36m, authored five files) even though the engine’s own Worker.PID field was never updated — the two facts are unrelated once the subprocess has actually started. handleUnverifiablePIDWorker (engine/worker_liveness.go) implements this as a three-way branch, gated on claudeNameFlagSupported (a one-time, process-lifetime capability probe — if the installed claude binary predates --name support, no worker in this process could ever carry a sentinel, so the probe is skipped entirely and behavior falls straight through to the plain #1303 clear below):

Probe outcome Action
Sentinel found live Mirror the PID-recorded path’s posture: log [#N worker-liveness] sentinel "…" found live … — adopting PID, signal-0 liveness now governs (or “waiting for natural exit” if no PID was recovered) and do not clear. When the probe also yields the OS PID, WorkerPIDSet{PID} adopts it into the handle and a fresh WorkerHeartbeat{At: now} is applied, so the item’s Worker.PID > 0 on the very next scan cycle and ordinary signal-0 liveness governs from then on.
Sentinel affirmatively not found Clear exactly as the plain #1303 timeout path: store.Apply(WorkerExited{}) + label removal + log [#N worker-liveness] worker never reached PID assignment (started T ago) — clearing, could not verify liveness (sentinel "…" not found).
Probe itself failed (unsupported platform, ps error, non-zero exit, timeout) Neither clear nor defer forever: a per-(repo, issue) consecutive-failure counter (Engine.sentinelProbeFailures, in-memory, scan-goroutine-local — not itemstate.Store state) is incremented and compared against sentinelProbeUnverifiableCycleLimit (currently 3, ~3 scan cycles ≈ 3 minutes on top of WorkerStaleTimeout). Below the bound: log sentinel probe unverifiable (…) — deferring clear and leave the worker alone. At the bound: clear, logged as clearing UNVERIFIED after N consecutive sentinel-probe failures: <err>. The counter is cleared whenever the worker leaves this state — sentinel found live, sentinel confirmed not found, cleared at the bound, or a real PID reaches the store via the ordinary onPIDReady callback.
claudeNameFlagSupported == false Probe skipped entirely (never invoked); clears exactly as the plain #1303 timeout path, logged without the “(sentinel … not found)” suffix.

The probe (probeSentinelLive, unix-only; a Windows build-tag stub always reports itself unsupported, routing Windows through the bounded-unverifiable row above — the same posture isProcessAlive’s Windows stub takes for signal-0) shells out to ps -eo pid=,args= -w -w (no shell, short timeout) and matches the sentinel as a whole argv token — never a substring — so a sentinel for issue #2966 cannot match a decoy carrying #29660.

Dispatch-time backstop (R5, issue #1779): R1/R2 above are meant to make “two workers on one worktree” unreachable, but dispatchCandidates (engine/poll.go) asserts it independently rather than relying on that alone: immediately after the existing snap.Worker() != nil in-flight guard (which only catches an item this process still thinks is in-flight), it re-checks the same sentinel for the item’s resolved stage. A Live result refuses dispatch (logged refusing dispatch: sentinel "…" still live for a cleared worker) without ever applying WorkerEntered; “not found” or a probe error both fall through to ordinary dispatch — the guard fails open on a probe error, since treating an unverifiable probe as a hard block would let a broken ps wedge dispatch for every item on every poll, a strictly worse failure mode than the rare duplicate-writer it exists to prevent. Scoped to the main stage-dispatch path only; the structurally identical snap.Worker() != nil guards in the reinvoke dispatchers (reinvoke.go, reviews.go, ci.go, merge_gate.go) do not get this check.

Unlike the scan’s one-worker-at-a-time probeSentinelLive call, dispatchCandidates fetches the live process table at most once per call (listProcessArgvFn, lazily on first need) and matches every dispatch-eligible item’s sentinel against that single snapshot in-process (matchSentinelInArgvList) — not once per item. A poll with many simultaneously dispatch-eligible items (a startup burst, or a wide cooldown-expiry window) would otherwise serialize one ps subprocess spawn per item ahead of the first dispatch in that pass.

Signal-0 liveness check (syscall.Kill(pid, 0), for workers with PID > 0 — whether from the ordinary onPIDReady callback or adopted via the sentinel probe above):

Outcome Action
Signal-0 fails with ESRCH (PID dead) AND heartbeat stale store.Apply(WorkerExited{}) + RemoveLabel(fabrik:locked:<user>) + RemoveLabel(stage:<StageName>:in_progress) + log [#N worker-liveness] stale worker handle (pid=P last_sign=T) — clearing
Signal-0 succeeds (PID alive) with stale heartbeat Log [#N worker-detector] stale heartbeat for PID P — process alive, waiting for natural exit; do not remove labels or kill the process; re-check on next scan
Heartbeat fresh (within WorkerStaleTimeout) Skip regardless of PID state

StageName for label construction is taken from Worker.StageName, which was set at dispatch time.

Windows note: isProcessAlive always returns true on Windows (signal-0 is unsupported). The detector never clears stale workers on Windows via the signal-0 path; a PID<=0 worker on Windows always falls through the sentinel probe’s unsupported stub to the bounded-unverifiable row above. The startup cleanup pass handles the restart case on any platform.

Janitor Integration (Stale-Worker Awareness)

The worktree janitor’s in-flight-worker check (reaping criterion FR-4(d)) uses the same isWorkerStale() helper as the detector. A worker that the detector would clear is not permitted to block the janitor.

Effective guard: snap.Worker() != nil && !isWorkerStale(snap.Worker(), e.workerStaleTimeout())

This prevents the multi-hour stall scenario where a Claude process dies without the engine’s context-cancel path firing, leaving a permanent WorkerEntered block that the janitor would otherwise honour indefinitely.

Startup Cleanup Pass (Restart Case)

When a Fabrik process crashes, deferred cleanup never runs. On the next startup, stale fabrik:locked:<user> and stage:*:in_progress labels may remain on issues. The startup cleanup pass removes them.

Trigger: Immediately after the first doPollCycle() completes (not a wall-clock timer). This guarantees the store is populated before the scan in both webhook and non-webhook modes.

Grace period: Any lock acquired during the first poll cycle already has Worker != nil in the store (set by LocalLockAcquired). The scan skips these items — no artificial sleep is needed.

Detection: Scan store.All() for items where:

Cleanup per item:

  1. RemoveLabel(fabrik:locked:<user>) from GitHub (with write-through to cache).
  2. For each label in snap.Labels() matching strings.HasPrefix("stage:") && strings.HasSuffix(":in_progress"): RemoveLabel(label) from GitHub (with write-through). StageName is unavailable; labels are identified by pattern.
  3. store.Apply(WorkerExited{}) — no-op since Worker is already nil; applied for idempotency.
  4. Log the cleanup at "startup" tag.

Startup Orphaned in_progress Scan (Broader Sweep)

The lock-gated pass above only reaches items that still hold a stale lock label. An in_progress label orphaned after its paired lock was already cleanly released — the normal outcome when release() (engine/item.go) succeeds at removing the lock but the paired removeInProgressLabel call fails transiently, since that call has no retry — is invisible to it and survives every restart. runStartupOrphanedInProgressScan() closes this gap with a second, broader sweep.

Trigger: Same call site as the lock-gated pass — runs immediately after runStartupCleanup(), inside the same if firstPollErr == nil block in poll.go.

Detection: Scan store.All() — every managed item in the store, not filtered by lock label. For each item, build the set of stage names carrying stage:<Name>:complete or stage:<Name>:failed. Then, for each stage:<Name>:in_progress label on the same item whose <Name> is in that set, the label is stale by definition — a stage cannot be both finished and in progress — and is removed.

No Worker() or lock-label filter. This is a deliberate divergence from the lock-gated pass, not an oversight: dispatch ordering already guarantees complete/failed and in_progress never legitimately co-occur for the same stage — removeFailedLabel runs before in_progress is re-added on retry, and complete blocks further dispatch of that stage entirely. Any occurrence of the co-occurrence is therefore stale regardless of whether a worker or lock is present, making the predicate self-justifying and safe to run unconditionally at startup.

Cleanup per item: for each stale in_progress label found, RemoveLabel(label) from GitHub (with write-through to cache on success), logged as [startup] removed stale label %q — the same format as the lock-gated pass. Errors are logged as warnings and not retried, matching the lock-gated pass’s behavior.

Additive, not a replacement: this pass runs alongside, not instead of, the lock-gated sweep — the two predicates (stale lock label vs. complete/failed co-occurrence) are independent and both run every startup.

Scope: startup-only. A periodic (mid-run) version of this sweep would additionally need to exclude items with a live in-process worker for that stage — the startup case is safe without that check only because no workers exist yet in a freshly-started process. Running this sweep periodically is not implemented (see issue #1135).

Startup Bare in_progress Scan (ADR-1393 R7)

The two passes above are each conditional: the lock-gated pass requires a stale fabrik:locked:<user> label; the orphaned-scan pass requires a stage:<Name>:complete/failed sibling. An item with stage:<Name>:in_progress, no lock label, and no terminal sibling matches neither — the shape produced when the lock is cleanly removed (e.g. by cleanupLockedIssues() on a daemon clean-stop) but in_progress itself is never cleared, because the shutdown-pause write that should have cleared it directly (ADR-1393 R2) either never ran (force-quit, crash) or itself failed. runStartupBareInProgressScan() closes this third, disjoint gap.

Trigger: Same call site as the other two passes — runs immediately after runStartupOrphanedInProgressScan(), inside the same if firstPollErr == nil block in poll.go.

Detection: Scan store.All(), skipping any item with an active Worker() (defensive — always nil this early in startup, mirroring the lock-gated pass’s own guard). For each remaining item, build the set of stage names carrying stage:<Name>:complete/failed, and check whether any fabrik:locked: label (any user) is present. Skip the item entirely if a lock label is present (owned by the lock-gated pass) or if the item carries no in_progress label outside the terminal-sibling set (owned by the orphaned-scan pass). What remains — stage:<Name>:in_progress with neither a lock label nor a terminal sibling for <Name> — is removed.

Deliberately a third, narrow pass, not a broadened existing one: the three passes’ predicates are disjoint by construction (lock-gated; terminal-sibling-gated; neither), so each stays individually self-justifying and none silently absorbs another’s responsibility.

Cleanup per item: for each bare in_progress label found, RemoveLabel(label) from GitHub (with write-through to cache on success), logged as [startup] removed bare in_progress label %q (no lock, no terminal sibling). Errors are logged as warnings and not retried, matching the other two passes.

Scope: startup-only, for the same reason as the orphaned-scan pass (§ above).

Updated fabrik:locked:<user> Label Lifecycle

The lock label is now removed by four paths (previously two):

Path Trigger Function
Normal completion Stage completes (any outcome) releaseLock()
Graceful shutdown Engine receives SIGINT/SIGTERM cleanupLockedIssues()
Stale-worker detector Worker PID dead + heartbeat stale > WorkerStaleTimeout (default 5 min) cleanupStaleWorker()
Startup cleanup Engine restart; Worker nil + lock label present runStartupCleanup()

stage:<Name>:in_progress is additionally cleared directly by the daemon clean-stop pause (runShutdownPause) and by handleStopRequest (ADR-1393 R2) — not just by releaseLock() — and, on next startup, by runStartupBareInProgressScan() above for whatever survives despite that direct clear.

9.8 Change-feed / Observer Pattern (Phase 3-H)

Phase 3-H wires itemstate.Store.Subscribe into the engine to replace polling-based “has this item changed?” detection with a reactive change-feed. The key concept: after every non-no-op Store.Apply call, registered observers are called synchronously (outside the store’s write-lock) with a Change value indicating which fields changed and for which item.

Single-Store Architecture (Phase 5 F3)

There is exactly one *itemstate.Store instance. Engine.New() creates it and passes it to boardcache.NewCacheImpl. Both eng.store and cacheImpl.store reference the same pointer. Field ownership is by mutation type:

Mutation category Mutation types ChangeFlags produced
Engine-side LocalLockAcquired, LocalLockReleased, InvocationRecorded, StageAttempted, CooldownRecorded, WorkerEntered, WorkerSlotAcquired, WorkerPIDSet, WorkerHeartbeat, WorkerExited, PRChecksObserved, … LockChanged, InvocationChanged, StageStateChanged, CooldownChanged, WorkerChanged, WorkerLifecycleChanged (only WorkerEntered/WorkerExited), LinkedPRChanged (partial)
Webhook/reconcile-side IssueLabeled, IssueUnlabeled, IssueCommentCreated, PRReviewSubmitted, CheckRunCompleted, … LabelsChanged, CommentsChanged, LinkedPRChanged, AssigneesChanged, …
Both LocalStatusUpdated (reconcile/webhook delta path and engine write-through via cacheImpl.UpdateItemStatus) StatusChanged

Because both categories write to the same store, any store.Get(...) returns a Snapshot with all field groups populated. CacheImpl.Subscribe is a thin wrapper over c.store.Subscribe; since c.store is the shared store, engine code should call e.store.Subscribe directly rather than going through cacheImpl.Subscribe to avoid double-registration.

Boundary note: After Phase 5 F4 (issue #563), all per-item and pre-linkage state lives in itemstate.Store. CacheImpl retains only paused, recentMissCache, projectID/Title/OwnerType, and localDeltaAt — none of which are per-item state. checkRuns was migrated to Store.pendingCheckRuns (Phase 5 F4, #563); linkedPRs was migrated to LinkedPRState via PRDetailsUpdated (Phase 5 F2, #562). FetchLinkedPR and FetchCheckRuns now reconstruct their results from Store snapshots.

ChangeFlags and Their Trigger Mutations

All ChangeFlags are produced by the single shared store. Field ownership is by mutation type (see table above).

ChangeFlag Mutation category Trigger mutations
StatusChanged Both LocalStatusUpdated (reconcile, board fetch, projects_v2_item webhook delta); also engine-side via cacheImpl.UpdateItemStatus called from advanceToNextStage/handleStageComplete (stages.go)
LabelsChanged Webhook/reconcile IssueLabeled, IssueUnlabeled, LocalLabelAdded, LocalLabelRemoved
LockChanged Engine-side LocalLockAcquired, LocalLockReleased
LinkedPRChanged Both PR review, check runs, head SHA updates (webhook); PRChecksObserved, CIMergePendingStarted/Cleared (engine)
PRStateChanged Webhook/reconcile PRDetailsUpdated only — a narrower sub-flag of LinkedPRChanged for genuine PR-level state transitions (merged, closed, draft↔ready). Emitted alongside LinkedPRChanged, never alone. Not set by PRReviewSubmitted/PRReviewCommentCreated/ReviewThreadCommentAdded — see CommentBreakerObserver (§4.6) for why that distinction is load-bearing.
CommentsChanged Webhook/reconcile IssueCommentCreated, LocalCommentAdded
AssigneesChanged Webhook/reconcile IssueAssigneesUpdated (via applyIssuesDelta on issues.assigned/issues.unassigned events)
InvocationChanged Engine-side InvocationRecorded (token usage, completed, blocked, turn-limited, IsComment)
StageStateChanged Engine-side StageAttempted, StageRetryIncremented, ReviewCycleIncremented, etc.
WorkerChanged Engine-side WorkerEntered, WorkerSlotAcquired, WorkerKindSet, WorkerTriggerSet, WorkerTurnsRecorded, LocalLockAcquired (with Worker), WorkerPIDSet, WorkerHeartbeat, WorkerExited — all worker-handle mutations
WorkerLifecycleChanged Engine-side WorkerEntered, WorkerExited only — lifecycle transitions that change dispatch eligibility; emitted alongside WorkerChanged; this is the wake-relevant sub-flag
CooldownChanged Engine-side CooldownRecorded, ExpiredCooldownsConsumed (only when it deleted an entry)
DeepFetchChanged Engine-side DeepFetchFailed, ItemDeepFetched
ItemRemoved Reset Store.Reset for items present in the old map but absent from the new items slice
CheckRunChanged Webhook/reconcile CheckRunCompleted when SHA is not yet in shaToKey — run buffered into Store.pendingCheckRuns; also emitted (combined with LinkedPRChanged) by PRHeadSHAUpdated when the buffer is drained. Not in wakeChFlags: the CI gate is catch-up-driven (the catch-up loop polls on every cycle), not wake-driven. The flag exists for observability (future TUI consumers) without forcing a poll wake.
PRFactsChanged Poll-side PRFactsObserved, CIVerdictObserved, and PRHeadSHAUpdated when it clears the CI verdict on a head change (#2122). Informational; not in wakeChFlags/cycleSetFlags, so a poll-side write never wakes the poll

wakeChFlags: Which Changes Wake the Poll Loop

The wakeChObserver is registered once on the shared store. It fires a non-blocking send on wakeCh when Change.Fields & wakeChFlags != 0:

wakeChFlags = StatusChanged | LabelsChanged | CommentsChanged | LockChanged | LinkedPRChanged | AssigneesChanged | WorkerLifecycleChanged

AssigneesChanged was added in issue #543 so that assignment webhooks wake the dispatcher. WorkerLifecycleChanged was added in Fix B (issue #544) so that WorkerEntered and WorkerExited mutations deterministically wake the poll loop. The broader WorkerChanged flag (which also covers WorkerHeartbeat and WorkerPIDSet) is intentionally not in wakeChFlags — heartbeats fire every 30s per active worker and would otherwise cause repeated deep-fetch cycles for items that cannot be dispatched. See §2.14.

cycleSetFlags (issue #576): newMayNeedWorkObserver uses a narrower constant — cycleSetFlags = wakeChFlags &^ WorkerLifecycleChanged — rather than wakeChFlags. WorkerLifecycleChanged is excluded so that WorkerExited from an early-return goroutine (e.g. a dep-blocked item) does not bypass the cooldown gate by adding the item to mayNeedWork. The wake channel (newWakeChObserver) still uses the full wakeChFlags, so non-blocked items are re-evaluated promptly on worker exit. See §9.2 and §9.9.

Probe-origin writes never wake (#2080 R3). The probe loop (runProbeAndDeepFetch) runs at the start of the same poll() that dispatches, so everything it writes is already seen by that poll; waking Run() for those writes only schedules a redundant poll ~minPollInterval later, and a probe/cache disagreement that kept re-invalidating turned that into a forced poll every ~1.5 s. The probe loop therefore wraps its Store writes in itemstate.FromProbe (ProbeBoardItemUpdated, DeepFetchInvalidated, PRDetailsUpdated, PRHeadSHAUpdated, TerminalFlagSet, and — through CacheImpl.FetchItemDetailsFromProbe — the stale-item deep fetch’s ItemDeepFetched/PRHeadSHAUpdated), which tags the resulting Change.Origin as OriginProbe; newWakeChObserver ignores that origin. mayNeedWorkObserver is unaffected. Webhook, worker and REST writes are untagged and wake as before, as does the new-item path (IssueOpened). Any new probe-loop write must go through FromProbe or it will wake the poll again. The probe apply also honours the engine-write protection for Status and closed state (D.4, #2178).

Changes that do NOT fire wakeCh: InvocationChanged, StageStateChanged, CooldownChanged, WorkerChanged (from heartbeats/PID-sets/lock-with-worker), TitleBodyChanged, StateChanged, BlockedByChanged, DeepFetchChanged, BaseBranchChanged, ItemRemoved, CheckRunChanged (CI gate is catch-up-driven; see §9.8 ChangeFlag table).

The unconditional wakeCh <- struct{}{} send that previously lived in the webhook handler has been removed. The observer path is the sole mechanism for immediate wake — but wakeChObserver is only registered when wakeCh != nil. In headless runs (--notui or any configuration without a wake channel), mayNeedWorkObserver (always registered) alone populates the pre-filter set; wakeCh is never signalled and there is no immediate dispatcher wake. Because all mutation types flow through the single shared store, the single wakeChObserver registration (when present) is sufficient — no per-source registration is needed.

Registered Observers and Where They Live

Observer Registered on Fires when Emits
wakeChObserver shared store (once) Change.Fields & wakeChFlags != 0 wakeCh <- struct{}{} (non-blocking)
mayNeedWorkObserver shared store (once) Change.Fields & cycleSetFlags != 0 (excludes WorkerLifecycleChanged — see §9.9) adds repo#number to e.mayNeedWork
InvocationObserver shared store Change.Fields & InvocationChanged != 0 tui.JobCompletedEvent (Success/Errored are false/true respectively only for a genuine fault as of #1178 — a turn-cap exit (CLI subtype: "error_max_turns") sets TurnLimited: true instead and no longer counts as an error for rendering purposes, even though the underlying Claude process still exited non-zero and the existing retry/cooldown mechanics are unaffected; Completed remains a separate axis meaning “no FABRIK_STAGE_COMPLETE marker was seen”)
StageChangeObserver shared store Change.Fields & StatusChanged != 0 tui.StageChangedEvent
Pause observer (closure) CacheImpl.SubscribePause Pause() / Resume() tui.WebhookStatusEvent

All observers are registered in Engine.Run() after extracting cacheImpl. Their unsubscribe funcs are deferred so observers are cleaned up when Run() returns.

mayNeedWork Set: Poll Cycle Pre-Filter

Engine.mayNeedWork map[string]bool (protected by Engine.mayNeedWorkMu) is populated by mayNeedWorkObserver on every change that includes wakeChFlags. At the start of each poll cycle, poll() drains it to a local cycleSet:

cycleSet := func() map[string]bool {
    e.mayNeedWorkMu.Lock()
    defer e.mayNeedWorkMu.Unlock()
    s := e.mayNeedWork
    e.mayNeedWork = make(map[string]bool)
    return s
}()

Each item in board.Items passes the pre-filter if any of these bypass conditions apply (authoritative source: poll.go:1291–1310, hasAwaitingLabel variable in selectDeepFetchCandidates):

Bypass condition Label / Condition Rationale
Item is in cycleSet (observer-driven) Observer saw a relevant change since last poll
Stage has CleanupWorktree: true (stage config) Cleanup triggers on local filesystem state, not board changes
Item has fabrik:awaiting-ci label fabrik:awaiting-ci CI check-run completions don’t bump the issue’s updatedAt; CI gate must be evaluated every poll
Item has fabrik:rebase-needed label fabrik:rebase-needed Base-branch advances don’t bump updatedAt; rebase gate must be evaluated every poll
Item has fabrik:awaiting-review label fabrik:awaiting-review Review-submission webhooks don’t bump updatedAt; gate clearance and Phase 1/Phase 2 reprompt timers require per-poll evaluation (issue #616)
Item has fabrik:auto-merge-enabled label fabrik:auto-merge-enabled GitHub auto-merge state changes (e.g., PR merges) don’t bump the issue’s updatedAt; convergence monitor must be evaluated every poll
Item has fabrik:revalidate label fabrik:revalidate Stuck-Validate items may have settled updatedAt long ago; the revalidate scan must still be reached on every poll
snap.HasExpiredCooldown(now) is true (cooldown timer) Periodic re-evaluation window has passed

Items not meeting any bypass condition are skipped for that poll cycle (no deep-fetch, no dispatch). Items with an active CooldownAt (not yet expired) are also skipped — the CooldownAt[“periodic-re-eval”] entry gates time-based re-evaluation.

This replaces the removed engine.seenUpdatedAt map[string]time.Time, which performed an equivalent “has this item changed?” gate via timestamp comparison.

CacheImpl.SubscribePause

In addition to Store.Subscribe, CacheImpl exposes SubscribePause(fn func(bool)) func() for components that need to react to pause/resume transitions (e.g., the TUI WebhookStatusEvent). Pause observers are called outside c.mu — Pause() and Resume() snapshot the list before releasing c.mu, then call observers on the snapshot. This mirrors Store’s snapshot-then-call pattern. Observers registered via SubscribePause MUST NOT call Pause() or Resume() re-entrantly. Because c.mu is released before observers are called, calling other CacheImpl methods from an observer is deadlock-free; however, calling Pause/Resume from within a pause observer produces semantic re-entrancy (double-fire, inconsistent paused state).

Invariants

References: ADR-036: Reactive Cache Single-Owner, ADR-038: Observer Wiring (historical dual-store design).

9.9 Pre-Dispatch Label Gates and Wake-Loop Avoidance

Problem: When processItem returns early — without invoking Claude — WorkerExited fires with WorkerLifecycleChanged. Before Fix B (issue #576), WorkerLifecycleChanged was in wakeChFlags and was used by newMayNeedWorkObserver to populate mayNeedWork (the cycleSet). This caused the blocked item to bypass the cooldown gate on the next poll cycle, be deep-fetched, pass through itemNeedsWork, dispatch another goroutine, hit the same early-return — creating a tight wake-loop at 2-3 Hz that burned GraphQL budget.

Pattern: Labels that cause processItem to return early without invoking Claude must be checked in itemNeedsWork before goroutine launch. This is the pre-dispatch gate pattern. The gate prevents WorkerEntered/WorkerExited from firing at all — no goroutine means no WorkerLifecycleChanged, no wake signal, no cycleSet entry.

Currently gated labels in itemNeedsWork:

Label Gate Added Notes
fabrik:locked:<other> Always (original) Items owned by another instance
fabrik:editing Issue #550 Items in active comment processing
fabrik:blocked Issue #576 Items waiting for blocking issues to close

Defense-in-depth (Fix B, issue #576): cycleSetFlags excludes WorkerLifecycleChanged. Even if a pre-dispatch gate is missing for a given early-return label, WorkerExited no longer bypasses the cooldown gate via mayNeedWork. The item is still re-evaluated on cooldown expiry (CooldownAt["dep-blocked"]) or when a genuinely-new event fires LabelsChanged/StatusChanged. Fix A (pre-dispatch gate) eliminates goroutine launch; Fix B ensures the loop cannot self-perpetuate even if Fix A is absent for some future early-return path.

Other early-return paths: fabrik:awaiting-input and fabrik:paused are conditionally gated in itemNeedsWork — they return false when no new comment is present, preventing goroutine launch and the associated wake-loop. fabrik:awaiting-review is handled by the catch-up loop rather than by an early-return inside processItem, so it does not apply this pattern. Future additions to processItem that introduce early-return paths should add a corresponding pre-dispatch gate in itemNeedsWork; Fix B’s cycleSetFlags provides defense-in-depth while a gate is absent.

Cross-references: §1.4 (label semantics), §9.2 (cycleSetFlags split), ADR-039 (decision record for the wakeChFlags/cycleSetFlags split).

9.9.1 Non-Blocking Slot Acquire and Slot-Deferred Dispatch (#2114, ADR 2114)

dispatchCandidates acquires an Engine.sem slot with a non-blocking select (default branch). It never waits on a worker: everything poll() runs after dispatch (settleTrainRuns/handleMergeTrainBatch and the settle scans) and the next poll would otherwise stall for as long as the shortest-remaining worker, and a quiet worker extends that indefinitely. ctx.Err() is checked at the top of each loop iteration, since the default branch no longer observes cancellation.

When no slot is free the candidate is deferred: no WorkerEntered, issue context, StageAttempted, retry count, cooldown or label is applied, and the rest of the candidate loop still runs (once the semaphore is full every remaining admitted candidate is deferred; guard-skipped ones cost nothing extra). The snap.Worker() != nil guard, the R5 sentinel check (#1779), WorkerEntered-before-goroutine and the release-then-WorkerExited defer order are unchanged.

Logging. One poll-level line per poll with a deferral: [poll] deferred dispatch of N candidate(s) #A, #B: all worker slots busy (in_use/max in use), where in_use/max is len(e.sem)/cap(e.sem) — true slot occupancy including reinvoke and merge-train conflict-resolution holders (not the Worker-entry count the local API reports).

Re-admission. The WorkerExited wake alone does not re-admit a slot-deferred item: its cycleSet entry was consumed by the poll that deferred it, an end-of-poll periodic-re-eval stamp would leave it behind a cooldown of up to PollSeconds*10, and cycleSetFlags excludes WorkerLifecycleChanged (§9.2). So dispatchCandidates returns the deferred issueKeys; poll()’s end-of-poll stamp skips them, and requeueDeferredDispatch writes them into e.mayNeedWork (directly, under mayNeedWorkMu, so it causes no wake). The next poll — woken by the freed slot or by the ticker — admits them through cycleSet, bypassing the cooldown gate, without changing ADR-039’s flag split.

Re-admission survives a failed deep-fetch. The poll that re-admits a slot-deferred item does so through cycleSet alone, so no expired cooldown is consumed. If that poll then drops the item before it joins deepFetchCandidates — FetchItemDetails fails, or itemMayNeedWork refuses it inside the deep-fetch failure backoff — selectDeepFetchCandidates re-arms periodic-re-eval (rearmPeriodicReEval) for any cycleSet-admitted item, not only for one whose expiry was consumed. Otherwise the item would have no cooldown and no cycleSet entry and would sit stranded until an unrelated change. The re-arm interval equals the failure backoff (githubRecheckInterval), so a persistent failure does not become a per-poll API call.

Reinvokes can take a freed slot first. A review- or CI-fix reinvoke goroutine blocked on e.sem (engine/reinvoke.go) is already waiting when a slot frees, so it can take that slot before the poll’s non-blocking attempt. A deferred dispatch can therefore be delayed by reinvokes, but never stranded: it stays queued in mayNeedWork and is re-attempted on every following poll.

Known limits. A wake dropped during GraphQL/REST backoff (#1716) leaves the item to the backed-off ticker poll, which still re-admits it. A merge-train conflict-resolution slot release (acquireTrainSlot, ADR-2046) fires no WorkerExited, so it does not wake the poll; the next ticker poll covers it. Under sustained saturation each poll re-deep-fetches each deferred item (cache-served when the board cache is fresh). Sim coverage: tests/sim/slot_saturation_test.go.

9.9.3 Dispatch Order (#2232, ADR-2232)

dispatchCandidates hands free worker slots out by time without a worker, longest first (orderDispatchCandidates), not in board order: the oldest LastAttemptAt for the item’s current stage first, items that have never run before all others, ties by issue number so the order is deterministic. This matters because a time slice (§7.1) is re-dispatched with no backoff: the sliced item has just had a worker, so it sorts after every longer-waiting candidate and takes its own slot back only if one is still free after they have had their turn. A slot-deferred candidate (§9.9.1) is re-admitted by the next poll and sorted by the same rule, so a deferral never moves it to the back. Holding stages, settle scans and merge-train dispatch do not pass through it. LastAttemptAt is stamped when an invocation ends, so it approximates the instant the item last held a worker.

9.9.2 Post-Run Feedback Admission (#2232, ADR-2232)

Feedback that arrived while a worker ran is read into the cache during the run, so its arrival raises no change signal once the run ends: a poll woken by the comment drained its cycleSet entry and dropped the item because a worker was in flight (§9.9, snap.Worker() != nil), and the bare WorkerExited wake is excluded from cycleSetFlags. The item then fell through to periodic-re-eval. So the goroutines that run a worker (dispatchCandidates and the shared reinvoke scaffold dispatchReinvoke) defer postRunFeedbackAdmission, declared so it runs first on exit — before the slot release and the WorkerExited wake. It reads the item from the Store (Peek, no GitHub call) and applies cachedFeedbackArrivedSince — cachedPendingFeedbackOf (the cache-only twin of unprocessedFeedback, pinned by TestCachedFeedbackMatchesLiveGate) restricted to feedback created at or after the worker’s start (e.now() captured when the goroutine began; a comment, thread comment or review body with an unknown timestamp does not count). Feedback that was already pending when the run began is what the run was for: if it is still pending afterwards the pass failed, and admitting it again would bypass every cooldown for a pass that ended without clearing it and without pausing, so it takes the ordinary retry path instead. When anything that arrived during the run is pending (comments, current-head review threads, unaddressed review bodies) it sets e.mayNeedWork[key] (as requeueDeferredDispatch does), so the next poll admits the item through cycleSet, bypassing cooldowns; itemNeedsWork already lets a new comment past the retry backoff. The entry is drained once. It is conditional on known pending feedback — admitting every exit would undo ADR-039 — and is skipped on shutdown, for cleanup-stage, closed, PR and paused items (a pause lifts only on a human comment created after it, so cached feedback cannot resume the item). Admission is not a dispatch: the comment pass or review reinvoke it leads to stays bounded by the comment circuit breakers and the review cycle limit, and the stage’s own retry still waits out retry_backoff (LastAttemptAt). Feedback GitHub has but the cache lacks is still caught by feedbackGateBlocks at every advance and landing.


10. State Diagrams

10.1 Happy Path — Linear Stage Progression

stateDiagram-v2
    direction TB

    [*] --> Specify_Idle : Issue added to board
    Specify_Idle --> Specify_Running : Poll tick
    Specify_Running --> Specify_Complete : FABRIK_STAGE_COMPLETE

    Specify_Complete --> Research_Idle : Auto-advance or human move
    Research_Idle --> Research_Running : Poll tick
    Research_Running --> Research_Complete : FABRIK_STAGE_COMPLETE

    Research_Complete --> Plan_Idle : Auto-advance or human move
    Plan_Idle --> Plan_Running : Poll tick
    Plan_Running --> Plan_Complete : FABRIK_STAGE_COMPLETE

    Plan_Complete --> Implement_Idle : Auto-advance or human move
    Implement_Idle --> Implement_Running : Poll tick
    Implement_Running --> Implement_Complete : FABRIK_STAGE_COMPLETE
    note right of Implement_Running
        Draft PR created
        PR marked ready
    end note

    Implement_Complete --> Review_Idle : Auto-advance or human move
    Review_Idle --> Review_Running : Poll tick
    Review_Running --> Review_Complete : FABRIK_STAGE_COMPLETE

    Review_Complete --> Validate_Idle : Auto-advance or human move
    Validate_Idle --> Validate_Running : Poll tick
    Validate_Running --> Validate_Complete : FABRIK_STAGE_COMPLETE
    Validate_Complete --> Done_Pending : Yolo: merge + advance

    Done_Pending --> Done_Complete : Worktree cleaned up
    Done_Complete --> [*]

10.2 Off-Path Flows

stateDiagram-v2
    state "Active Stage" as Active {
        Idle --> Running : Poll tick
        Running --> Complete : FABRIK_STAGE_COMPLETE
        Running --> AwaitingInput : FABRIK_BLOCKED_ON_INPUT
        Running --> Cooldown : No marker (incomplete)

        Cooldown --> Running : Cooldown expired (retry)
        Cooldown --> PausedFailed : MaxRetries exceeded

        AwaitingInput --> CommentProcessing : Human comment
        PausedFailed --> Idle : User removes fabrik:paused

        Idle --> Paused : Human adds fabrik:paused
        Paused --> CommentProcessing : Human comment (implicit unpause)
        Paused --> Idle : Human removes fabrik:paused

        Idle --> Blocked : Open dependencies
        Blocked --> Idle : All dependencies closed

        Complete --> AwaitingReview : wait_for_reviews gate
        AwaitingReview --> ReviewReinvoke : Gate clears + unresolved threads
        AwaitingReview --> AwaitingInput : Review timeout
        AwaitingReview --> NextStage : Gate clears, no threads
        ReviewReinvoke --> AwaitingReview : Reinvoke completes, new reviews arrive
        ReviewReinvoke --> AwaitingInput : Cycle limit exceeded

        Complete --> AwaitingCI : wait_for_ci gate (CI failure detected)
        AwaitingCI --> CIFixReinvoke : CI failed, cycle ok
        AwaitingCI --> NextStage : CI checks all pass
        AwaitingCI --> AwaitingInput : CI timeout or cycle limit
        CIFixReinvoke --> AwaitingCI : Reinvoke completes, CI re-evaluated next poll
        CIFixReinvoke --> AwaitingInput : Cycle limit exceeded

        Complete --> NextStage : Auto-advance or human move
        Paused --> NextStage : Validate stage + PR merged externally (runValidatePRTerminalAdvance — any gate label, ADR-056 D2)
        CommentProcessing --> Complete : FABRIK_STAGE_COMPLETE in comment output
        CommentProcessing --> Idle : Comment processed (no completion)
    }

    state "Next Stage" as NextStage
    NextStage --> Active : Board column updated

10.3 Review Reinvoke Cycle

stateDiagram-v2
    direction TB

    StageComplete --> CheckDependencies : Catch-up loop Phase 1 (unconditional — all items)
    CheckDependencies --> Blocked : Has open blockers
    CheckDependencies --> CheckReviewGate : No blockers

    Blocked --> CheckDependencies : Next poll tick

    CheckReviewGate --> WaitingForReviewers : Outstanding reviewers
    CheckReviewGate --> TimedOut : Timeout elapsed
    CheckReviewGate --> GateCleared : All reviewers submitted

    WaitingForReviewers --> CheckReviewGate : Next poll tick

    TimedOut --> PausedForTimeout : pauseForReviewTimeout()
    note right of PausedForTimeout
        fabrik:paused
        fabrik:awaiting-input
    end note

    GateCleared --> CheckThreads : buildReviewThreadComments()
    CheckThreads --> Phase2 : No unresolved threads → Phase 2
    CheckThreads --> CheckInFlight : Unresolved threads exist

    CheckInFlight --> SkipReinvoke : Already in-flight
    CheckInFlight --> CheckCycleLimit : Not in-flight

    CheckCycleLimit --> PausedForCycles : cycleCount >= MaxReviewCycles
    note right of PausedForCycles
        fabrik:paused
        fabrik:awaiting-input
    end note

    CheckCycleLimit --> DispatchReinvoke : cycleCount < MaxReviewCycles
    DispatchReinvoke --> ProcessComments : Async goroutine
    ProcessComments --> CheckReviewGate : Next poll (if new reviews arrive)
    ProcessComments --> Phase2 : Stage complete after addressing feedback

    SkipReinvoke --> CheckReviewGate : Next poll tick

    Phase2 --> Advance : yolo/cruise/auto_advance gate passes
    Phase2 --> Idle : Gate not met — no advancement
    note right of Idle
        Item stays in stage:X:complete
        until user advances manually
        or adds yolo/cruise label
    end note

10.4 Additional Flows

These flows are absent from 10.1–10.3 but represent reachable states documented in §2, §3.4, §5, and §6.8.

stateDiagram-v2
    direction TB

    %% Validate re-entry back-edges
    ValidateComplete --> ValidateIdle : fabrik:revalidate applied (§2.15)\nRemoves stage:Validate:complete, gate labels, paused labels;\nresets retry counters; dispatches Validate on next poll

    ValidateComplete --> ValidateIdle : SHA-invalidation scan (§2.16)\nPR HEAD SHA ≠ ValidateCompletedSHA\n(not landed, not a pure engine catch-up head);\nremoves stage:Validate:complete + gate labels

    note right of ValidateIdle
        ValidateIdle = Validate column,
        Idle (no completion label)
    end note

    %% Convergence monitor deferred Done advance
    ValidateAwaitingMerge --> Done : GitHub merges PR (§5.5)\ncheckAutoMergeConvergence detects pr.Merged == true;\nremoves fabrik:auto-merge-enabled; advanceToNextStage

    note right of ValidateAwaitingMerge
        ValidateAwaitingMerge =
        stage:Validate:complete present
        + fabrik:auto-merge-enabled present
        (yolo flow after handleStageComplete)
    end note

    %% FABRIK_PR_CREATE marker (Implement)
    ImplementInProgress --> ImplementInProgress : FABRIK_PR_CREATE_BEGIN/END (§5.6)\nEngine creates draft PR; no stage advance\nPR URL posted as issue comment

    %% FABRIK_NO_WORK_NEEDED durable-marker path (§6.8, ADR-060)
    AnyStageInProgress --> AwaitingDone : FABRIK_STAGE_COMPLETE + FABRIK_NO_WORK_NEEDED\nfabrik:awaiting-done written FIRST (durable, survives restart);\nAll subsequent non-cleanup stages marked complete;\nno PR created; dispatch suppressed regardless of column
    AwaitingDone --> AwaitingDone : Settle pass fails (status move or close)\nRetried every poll by the settle scan;\nfabrik:awaiting-done remains
    AwaitingDone --> Done : Settle pass succeeds\nfabrik:awaiting-done removed
    AwaitingDone --> Paused : Settle pass fails Attempts >= MaxRetries times\nescalateNoWorkNeededFailure: fabrik:paused added,\nfabrik:awaiting-done removed, comment posted

    note right of AwaitingDone
        AwaitingDone = fabrik:awaiting-done present
        Board column may still be the
        emitting stage's own column —
        the Done move itself may be
        what's failing
    end note

    note right of Paused
        Paused (here) = fabrik:paused present,
        fabrik:awaiting-done removed.
        Escalation after MaxRetries settle
        failures — human intervention required
    end note

    note right of Done
        Done = Pending Cleanup
        (worktree removal by janitor)
    end note

    %% Spawned child board-placement retry (§6.9, ADR-062)
    ChildCreated --> AwaitingPlacement : UpdateProjectItemStatus fails\n(call error, nil statusField, or no suitable option)\nfabrik:awaiting-placement written on the CHILD;\nchild, board item, blockedBy link already exist
    AwaitingPlacement --> AwaitingPlacement : Settle pass fails\nRetried every poll, sourced from board.Items directly\n(NOT deepFetchCandidates — child's column matches no stage);\nfabrik:awaiting-placement remains
    AwaitingPlacement --> ChildPlaced : Settle pass succeeds\nfabrik:awaiting-placement removed
    AwaitingPlacement --> ChildClosed : Child observed closed\nclearChildPlacementMarker: fabrik:awaiting-placement removed;\nno placement attempt, no pause, no comment
    AwaitingPlacement --> ChildPaused : Settle pass fails Attempts >= MaxRetries times\nescalateChildPlacementFailure: fabrik:paused added to child,\nfabrik:awaiting-placement removed, comment posted on child,\nbest-effort comment posted on parent

    note right of AwaitingPlacement
        AwaitingPlacement = fabrik:awaiting-placement present
        Board column is typically Backlog —
        declared unmanaged (or, on installs
        without backlog.yaml, still unrecognized),
        so ordinary dispatch never revisits it
    end note

    note right of ChildPaused
        ChildPaused = fabrik:paused present on the child,
        fabrik:awaiting-placement removed.
        Parent remains blocked (blockedBy) with
        no other visibility — the best-effort
        parent comment is the only signal
    end note

    note right of ChildClosed
        ChildClosed = child issue closed;
        no further board dispatch is needed,
        so the marker is simply cleared
    end note

Assignee transitions (§2.13): Assignee changes fire AssigneesChanged → wakeChObserver, re-evaluating the item on the next poll. No label is mutated. No separate diagram node is warranted.


11. Worktree Janitor

11.1 Purpose

The worktree janitor reaps orphaned git worktrees left behind when issues exit the managed lifecycle without triggering the normal cleanup_worktree: true Done-stage path. Three scenarios produce stranded worktrees:

  1. Off-board items — issues removed from the project board (archived manually by an operator, or deleted) are no longer iterated by the poll loop; their worktrees persist indefinitely.
  2. Narrow restart window — Fabrik dies after attemptMergeOnValidate advances the board to Done but before the next poll dispatches the Done stage.
  3. Stage YAML drift — an operator who renames or removes the Done stage from their config orphans worktrees that will never again be visited by the dispatch loop.

11.2 Schedule

11.3 Reaping Gate (FR-4)

A worktree is reaped only when all four conditions hold:

Condition Rationale
(a) Issue is closed Open issues are never safe to reap — the issue may still be in progress, even if no worker is currently dispatched.
(b) Off-board OR cleanup-complete If the issue is still on the board at an active (non-cleanup) stage, the dispatch loop may still visit it. A worktree at a cleanup_worktree: true stage with stage:<Name>:complete is safe to reap: CleanupWorktree runs before the label is applied, so the directory can only persist if CleanupWorktree errored.
(c) Clean worktree git status --porcelain returns no non-engine-managed paths. Engine-managed files (.fabrik-context/, .fabrik/issue.md) are excluded by isEngineManagedPath. A dirty worktree may contain uncommitted work from an interrupted Claude session.
(d) No in-flight worker snap.Worker() == nil from the Store. Concurrent reads are safe — Store uses RWMutex.

All four conditions are evaluated in order (cheapest first). A worktree that fails any condition is left in place and counted toward the skipped total.

Closed-status resolution (FR-5):

  1. Primary: Store.Get(repo, number) → snap.IsClosed() (free; no API call).
  2. Fallback (item not in Store): GitHubClient.FetchIssue(owner, repo, number) — one REST call per stranded worktree per cycle; result cached in-memory for the cycle duration.
  3. Error: skip with warning; treat as not-closed to avoid false positives.

Cleanup path (FR-6):
WorktreeManager.CleanupWorktree(number, false) — the same path used by the Done stage. If the bare repo is absent (manually deleted), falls back to os.RemoveAll with a warning. Both paths now run the cwd-rooted process reaper immediately before removing the directory (including the os.RemoveAll fallback) — see stage-lifecycle.md § Worktree Teardown Process Reaping for detail.

11.4 Owner/Repo Resolution

The on-disk directory is named owner-repo (dash-separated), which is ambiguous for hyphenated owner or repo names.

11.5 Configuration

# .fabrik/config.yaml
janitor_interval_hours: 1   # default; set to 0 to disable the janitor entirely

Also settable via flag (--janitor-interval) or environment variable (FABRIK_JANITOR_INTERVAL). EC-6: changing this value at runtime has no effect; Fabrik must be restarted for a new cadence to take effect.

11.6 Summary Log (FR-7)

At the end of each cycle the janitor emits an INFO-level log line:

[janitor] cycle complete: scanned N worktrees, reaped K, skipped M (reasons: ...)

The reasons field tallies distinct skip reasons (e.g. issue open ×3, dirty worktree, in-flight worker). Each reaped worktree is also logged individually at reap time.

11.7 Process-Sweep Janitor (#1798 R3)

A fourth periodic janitor, runProcessSweepJanitor, runs alongside the worktree/log/session janitors above — same schedule (§11.2: startup scan, then hourly), same JanitorIntervalHours != 0 gate, no separate config knob. Unlike the other three, it does not scan .fabrik/worktrees/ at all: it sweeps the durable session-scoped descendant registry (.fabrik/state/descendants.json) for orphans that escaped both the PGID-scoped grandchild cleanup and the per-invocation session-scoped reap — see stage-lifecycle.md § Session-Scoped Descendant Reaping for the full mechanism, and ADR 1798 for the design rationale.

Sweep condition: a registry entry is only ever acted on when its recorded worker PID is confirmed dead — a live signal-0 probe and an identity re-check against the worker’s own comm/lstart fingerprint (recorded alongside the entry at discovery time). A live PID whose fingerprint no longer matches means the original worker exited and the OS has since reused its PID for an unrelated process; treating that PID as “still alive” without the identity re-check would leave the entry unswept forever, not merely delayed. An entry whose worker is still alive and still the same worker is left untouched — that invocation is still in flight, and the per-invocation reap will handle it at its own invocation end. This means the sweep only ever reaps genuine orphans: descendants of a worker that crashed before reaching its own invocation-end reap, descendants left behind by a previous engine run entirely (the case this janitor exists for, since the registry is a JSON file on disk and survives a restart while the engine’s own in-memory dispatch state does not), or descendants whose worker PID was reused.

Identity re-verification (R5): immediately before killing any entry, its recorded comm+lstart identity fingerprint is re-checked against the live process at that PID. A mismatch — the process already exited, or the PID was reused for something unrelated since it was recorded — means the entry is dropped from the registry and never signalled.

Summary log:

[proc-janitor] cycle complete: scanned N registry entries, reaped K, skipped M

Each reaped PID is also logged individually, via the existing [#N kill] convention, at sweep time.

Registry-independent session sweep (#1814): runProcessSweepJanitor additionally calls sweepOrphanedWorkerSessions, because the registry sweep above can only act on a descendant that was recorded — one spawned in its worker’s final seconds, or during a transient ps failure, has no entry and is invisible to it. The worker record is written at spawn to .fabrik/state/workers.json (a JSON array; one record per invocation, keyed by unique ID, never by PID; an unparseable file is quarantined to workers.json.corrupt with a logged warning and treated as empty, so a corrupt file cannot disable recording). The sweep takes one process-table scan (the shared cached one) and one Getsid pass, and reaps every process whose SID equals the PID of a recorded worker classified as dead. The invocation-end variant runs the same logic with a fresh scan (stage-lifecycle.md § Session-Scoped Descendant Reaping).

Worker PID state Result
not alive dead — sweep members
alive, fingerprint lookup errors inconclusive — skip, keep record
alive, record fingerprint empty skip (never signalled on liveness alone)
alive, lstart equal to record live worker — skip
alive, lstart differs recycled PID — sweep only members that started at or after the recorded worker and strictly before the current holder; equal/unparseable timestamps skip

Ownership is by session lineage only (no comm/args allow-list), and only for a recorded worker: a dead session leader with no record is never touched. Before each SIGKILL the candidate’s fingerprint is re-read and Getsid re-checked; a lookup failure skips it. A record is pruned only when the worker is dead, a successful scan found zero live members and nothing was skipped, on two consecutive scans (EmptyScans >= 2, persisted across janitor passes); a scan error or any skip resets the count and keeps the record. A scan error changes nothing.

Owner-marker reap (#2128, ADR-2128). A worker record also carries Owner: the path of the owner file named by the FABRIK_WORKER_OWNER the worker was started with (.fabrik/state/worker-owners/<nonce>.json, holding the worker’s PID and start token, written right after cmd.Start()). For a record classified dead or recycled — never live or inconclusive (R4) — both sweeps call sessionreap.ReapMarked before the session pass: every live process whose initial environment carries exactly that marker gets SIGTERM then SIGKILL, per exact PID, after re-checking its start token and marker and the CheckSID/CheckPGID refusals. A process with no marker is never signalled on shape (ppid 1, empty group) alone. The prune rule additionally requires the marked set to be empty (two consecutive scans), and pruning removes the owner file; the janitor also deletes owner files that no record references and that are older than 24h. The unreadable-environment cases (SIP-protected macOS binaries, other users, hidepid mounts) read as unmarked.

Summary log (a second line; the line above is unchanged):

[proc-janitor] session sweep complete: worker records N, session members reaped K, skipped M, records pruned P

Each reap is logged individually as [#N kill] sending SIGKILL to PID <pid> (<comm>) — orphaned session member of dead worker PID <W> (reason=session_sweep_periodic). See ADR 1814.


Appendix A: Two Paths to Stage Advancement

Stage advancement can occur through two code paths:

Aspect Path 1: handleStageComplete() Path 2: Catch-up loop in poll()
Runs in Worker goroutine Poll goroutine
Triggered by Claude outputs FABRIK_STAGE_COMPLETE Poll cycle finds stage:<X>:complete label
Review data Stale (just ran MarkPRReady) Fresh (from FetchItemDetails)
Review gate Optimistic: applies fabrik:awaiting-review, returns Real: calls checkReviewGate(), evaluates timeout
Label freshness Re-fetched via refreshAutonomyLabels() Re-fetched via refreshAutonomyLabels()
Merge at Validate attemptMergeOnValidate() called directly attemptMergeOnValidate() called from catch-up (yolo only)
Advancement advanceToNextStage() if should advance and no gate advanceToNextStage() after Phase 2 gate (yolo/cruise/auto_advance)

Autonomy-label re-fetch, both paths (#1769, §6.23). handleStageComplete() and runCatchUpPhase2() each call the shared refreshAutonomyLabels() helper (engine/stages.go) as the first step of their autonomy-decision logic, before either reads hasYoloLabel/hasCruiseLabel. This row previously read “Already fresh from deep fetch” for Path 2 — that was false: Labels is not part of the deep-fetch cache contract (copyDeepFieldsFromState excludes it), so an item that went through a “fresh” deep fetch earlier in the same poll pass could still carry a stale fabrik:yolo/fabrik:cruise from before an operator removed it. Both paths now perform the same live, item-scoped FetchLabels() call (never e.cfg.Owner/e.cfg.Repo, which is empty in multi-repo mode) immediately before the autonomy decision, so a mid-run add or remove of either label is observed by whichever path makes the next advance decision — see §6.23 for the full rationale and the D1/D2 defects this replaced.

Path 2 is split into two phases:

Phase Gate What it does
Phase 1 Unconditional (all stage:<X>:complete non-paused non-cleanup items; also items with fabrik:awaiting-ci on wait_for_ci: true stages — they have no stage:<X>:complete until CI clears) checkDependencies() → checkReviewGate() → buildReviewThreadComments() / dispatchReviewReinvoke() → checkMergeabilityGate() → checkCIGate() / dispatchCIFixReinvoke() (review reinvoke and CI-fix reinvoke are mutually exclusive per poll cycle)
Phase 2 fabrik:yolo (cfg or label) OR fabrik:cruise label OR stage auto_advance: true attemptMergeOnValidate() (yolo only) → findNewComments() deferral → advanceToNextStage()

Phase 1 ensures inline PR review thread comments (from Copilot, Gemini, or human reviewers) are addressed and CI failures are fixed on all issues, not just yolo/cruise ones. Phase 2 keeps stage advancement gated as before. Items that dispatch a reinvoke in Phase 1 (review reinvoke or CI-fix reinvoke) skip Phase 2 on that poll cycle and are re-evaluated on the next poll.

Advancement invariant: Every code path that adds stage:X:complete must call advanceToNextStage() (or a well-defined deferral mechanism) in the same logical transaction. The board column is a derived view of label state; a column that diverges from a stage:X:complete label is a stuck issue. Authorized deferral mechanisms are:

Any new code path that adds stage:X:complete without one of these deferrals is a bug that will leave the board column stuck until a manual intervention.

Appendix B: Shallow Pre-Filter (Engine.poll() pre-filter + itemMayNeedWork())

Shallow pre-filtering is a two-pass process that avoids the expensive FetchItemDetails() GraphQL call for items that clearly don’t need work:

Pass 1 — Engine.poll() pre-filter (runs first, on every item in the board snapshot):

Check Passes If
cycleSet / cooldown pre-filter Item is in cycleSet (a Store observer fired for a relevant change), OR has a bypass label (fabrik:awaiting-ci, fabrik:awaiting-review, fabrik:rebase-needed, fabrik:auto-merge-enabled, fabrik:revalidate — need per-poll evaluation because their unblocking events don’t bump updatedAt; see §9.8 for per-label rationale), OR cleanup stage (checks local filesystem), OR CooldownAt has an unconsumed expiry (periodic re-evaluation window — an expiry admits the item once, then is deleted; see “Cooldown Cache-Key Strategy”), OR item is not yet in the engine store. Items with an active CooldownAt and no other signal are suppressed. See “Cooldown Cache-Key Strategy” section in Appendix B below.

Paused deep-fetch skip (runs after Pass 1 admits the item and after the cleanup-stage/FetchItemDetails split, inside selectDeepFetchCandidates — #1379, ADR-1379): if the item carries fabrik:paused and has no cycleSet entry (no observer-driven change this poll), the FetchItemDetails call is skipped — the item is still appended to deepFetchCandidates unfetched, so list-membership-only consumers (runValidatePRTerminalAdvance’s merged-while-paused self-heal, ADR-056 D2; settleRevalidateScan’s FR-5 guarantee) are unaffected. A paused item that also carries a bypass label (fabrik:awaiting-review, fabrik:rebase-needed, fabrik:auto-merge-enabled, fabrik:revalidate) is still skipped — pause wins over the bypass for the fetch decision, though the bypass label is still what got the item past Pass 1 in the first place. A paused item with a cycleSet entry (a real comment or label change since last poll) still gets fetched normally, so itemNeedsWork’s human-comment-resume check always sees fresh item.Comments. A paused item not yet recorded in the engine store gets a one-time baseline fetch regardless (mirrors Pass 1’s own not-in-store bypass). Unpausing re-admits within one poll, no restart required: board.Items[i].Labels is refreshed every poll by the shallow board-wide read (FetchProjectBoard) independent of this gate, so the moment fabrik:paused is removed on GitHub, this check reads false on the very next poll. Periodic re-evaluation reads through the cache (#1944, #2180, ADR 2180): the one exception to the skip is a paused item that is not in cycleSet and was admitted by an expired periodic-re-eval cooldown. It calls FetchItemDetails through the board cache. The cache’s freshness rule (§7.9 “Self-write staleness baseline”) sees a human reply that a self-write baseline would have masked, because GitHub’s issue comment total then differs from the cached comments, so the read costs nothing while the cache is genuinely fresh and refetches once when it is not. A long-parked item with a fresh cache is therefore still never fetched (#1379’s cost intent). Without a board cache every read is live and the skip is kept. This replaces the #1944 backstop (recentBaseline, a hand-applied DeepFetchInvalidated, ItemState.PausedBackstopBaseline, PausedBackstopServed, ADR 2096 item 5), which existed only because the baseline could hide a reply within about a second of the pause comment (0.0.83 release gate, TestBlockedOnInput). The same masking class affects review feedback on non-paused, stage-complete items (#2160): repairDispatcherBlindness (§6.6.6 “Dispatcher input agreement”) is kept as the gate-side guard; the PR-layer baseline above removes the updatedAt cause, but not the latestReviews cap or per-author collapse.

Pass 2 — itemMayNeedWork() (runs after the pre-filter admits the item, on shallow board data):

Check Passes If
Stage exists FindStage(stages, item.Status) != nil
Closed issue Not closed, OR cleanup stage, OR has stage:<X>:complete label
Repo write access Repo not yet resolved by resolveRepoAccess (fail-open on “unknown”), OR resolved with CanPush: true. A repo cached with CanPush: false is never admitted, regardless of stage or status — see ADR-1347. Purely in-memory (no API call); the probe itself always runs earlier in the same poll cycle, in poll()’s seeding block. resolveRepoAccess’s probe branches on auth mode (#1750, ADR-1750): PAT mode is FetchRepoAccess’s permissions.push (unchanged, ADR-1347). App mode (e.ghAppAuth != nil) instead consults resolveAppRepoAccess against Engine.appAccessibleRepos — the pinned installation’s own GET /installation/repositories list, fetched once at startup (New()), since permissions.push is structurally meaningless for an installation token (GitHub always returns it false, regardless of actual grants). Ambiguous App-mode answers (the startup fetch never succeeded, or the list was truncated and the repo wasn’t found in it) fail open through the same if err != nil branch PAT mode’s probe errors already use — never a silent CanPush: false. A confirmed exclusion (App or PAT mode) additionally gets a persistent warnings.Record entry (Type: "repo_access"), not just the one-line startup log. A confirmed zero-repo installation is a hard startup refusal from New() instead of a per-repo determination.
Cleanup stage Worktree exists on disk (local filesystem check only)
Deep-fetch failure cooldown No recent FetchItemDetails failure, OR failure cooldown expired

Note: itemMayNeedWork() intentionally does NOT check updatedAt, the cycleSet/cooldown gate, lock, editing, pause, or dependency labels — those are either handled by the poll.go pre-filter (Pass 1) or require the full label set from deep fetch and are checked in itemNeedsWork().

Cooldown Cache-Key Strategy

The engine uses two distinct in-memory stores (both in itemstate.Store) for cooldown tracking, split to prevent the #504 regression where invocation timestamps and periodic-re-eval cooldowns shared the same key:

What writes CooldownAt("periodic-re-eval"):

Expired cooldowns are consumed on admission (#2096, ADR-2096). An expired entry admits the item once: when the pre-filter admits a non-cleanup, in-store item that has an expired entry (on the strength of the expiry or of any other reason, including cycleSet), selectDeepFetchCandidates applies ExpiredCooldownsConsumed, which deletes every non-zero expired entry except archive-eligible-at. HasExpiredCooldown therefore reports only expiries the engine has not yet acted on (and ignores archive-eligible-at), which ends the old behaviour where one expired reason made the item admitted on every poll for ever (#2062). Every reader of CooldownAt outside admission tests only now.Before(t), so none depends on an expired entry persisting. Because the entry used to double as the periodic safety net, the net is re-armed explicitly: an item admitted and fetched is re-stamped by the poll() defer; an item admitted but dropped before joining deepFetchCandidates (itemMayNeedWork false, or a failed deep fetch) gets periodic-re-eval = now + githubRecheckInterval() at the drop site (rearmPeriodicReEval); an item advanced during the poll (advancedItems, #544) is skipped by the defer on purpose and is admitted next poll by its status change (StatusChanged puts it in cycleSet), not by a cooldown — the same path that always admitted an advanced item that had no cooldown entry. A failed non-Validate catch-up advance changes no status, so runCatchUpPhase2 leaves the item out of advancedItems; the defer then re-stamps periodic-re-eval and the advance is retried at the periodic interval.

What writes other CooldownAt reason keys:

What bypasses the cooldown gate (returns true regardless of cooldown):

What suppresses the cooldown gate (returns false despite expired cooldown):

Root-cause fix (#488): Terminal items (cruise+Validate complete, paused+complete, closed-with-stage-complete) triggered a perpetual deep-fetch loop: CooldownAt("periodic-re-eval") was only written by processItem() when work actually ran, so after cooldown expiry, itemMayNeedWork() returned true on every poll cycle indefinitely — each producing a no-op deep-fetch that did not update the cooldown. The fix has two parts: (1) primary — check stage:X:complete in shallow labels before returning true from the cooldown-expired branch, with an exemption for fabrik:awaiting-review items (which also carry stage:X:complete but need periodic re-evaluation for Phase 1/Phase 2 timers); (2) belt-and-suspenders — the deferred block in Engine.poll() sets CooldownAt("periodic-re-eval") for non-advanced deepFetchCandidates after each full cycle, capping deep-fetch frequency to once per cooldown period for all items (not just terminal ones). Phase 3-F structural fix (#504): The original Part 2 was scoped to terminal items only (stage:X:complete present in the full label set) because refreshing incomplete-stage entries reset the cooldown timer and prevented retries from firing. Phase 3-F removes this constraint: LastAttemptAt[stageName] (written only by StageAttempted, never by observation) now carries dispatch suppression independently of CooldownAt, so refreshing CooldownAt["periodic-re-eval"] for ALL non-cleanup items is safe — incomplete stages still retry after their LastAttemptAt cooldown expires, regardless of the CooldownAt["periodic-re-eval"] entry.

Appendix C: Guard Evaluation in itemNeedsWork() (Full Filter)

itemNeedsWork() runs after FetchItemDetails() has populated comments and the full label set.

Priority Check Passes If
1 Stage exists FindStage returns non-nil
2 Closed issue Not closed, OR cleanup stage, OR has stage:<X>:complete
3 Cleanup stage Not paused, not complete, worktree exists
4 Locked by other No fabrik:locked:<other> label
5 Awaiting input isAwaitingInput() true AND new human comments created at/after the pause exist (resumeAuthorised())
6 Paused Not paused, OR paused with new human comments created at/after the pause (resumeAuthorised())
7 New comments Any unprocessed comments → true
8 PR item Not a PR (PRs only support comments, checked after comment check)
9 Stage complete No stage:<X>:complete label
10 Cooldown Not attempted, OR cooldown expired

Appendix D: In-Memory Board Cache Lifecycle

The engine always maintains an in-memory cache of board state in boardcache.CacheImpl, wired as e.readClient for production runs (tests using engine.NewWithDeps bypass the cache and use boardcache.GitHubAdapter directly). The cache is the unified source of truth for downstream engine logic and reactive observers (PushUnblockObserver, StageChangeObserver, etc.) regardless of whether webhooks are configured. This appendix describes the cache lifecycle, delta semantics, reconciliation, and stream-health failover.

D.1 Bootstrap

The cache is bootstrapped via ProbeProjectBoard → BootstrapFromProbe. Both startup paths use the same probe-based cold-start:

  1. Webhook startup path — when --webhooks is enabled, before wm.Start() is called (before the webhook listener binds), the engine calls e.client.ProbeProjectBoard(...) and feeds the results to cacheImpl.BootstrapFromProbe(probeItems, projectID). After BootstrapFromProbe returns, e.seedTerminalFromProbeItems(probeItems) seeds Terminal=true for qualifying items (see below). This pre-bootstrap closes the gap between webhook subprocess start and the first poll cycle, so deltas don’t land in an empty cache. If the probe returns zero items (transient indexer hiccup), the cache is left virgin and the first poll retries.

  2. First-poll path — when the cache is not yet bootstrapped (webhooks disabled, or webhook-startup probe failed), the first poll() cycle calls ProbeProjectBoard and passes the results to BootstrapFromProbe, then calls seedTerminalFromProbeItems. On subsequent polls (cache already bootstrapped), the probe-and-refresh path (runProbeAndDeepFetch) replaces the bootstrap; see D.4.

BootstrapFromProbe calls Store.Reset with synthetic items constructed from probe results. After Store.Reset, the engine calls seedTerminalFromProbeItems, which applies TerminalFlagSet{Terminal: true} for each item that is closed, in a cleanup stage (CleanupWorktree: true), and has no on-disk worktree. The worktree-presence check prevents stranding cleanup work: if the worktree still exists (the Done stage’s cleanup_worktree action hasn’t run yet), the item must not be treated as terminal — it must proceed through processItem so cleanup can execute. Seeding the terminal flag for worktree-absent items prevents runProbeAndDeepFetch from calling FetchItemDetails on already-cleaned Done items. Deep fields (comments, linked PR data) are not populated at bootstrap; they are fetched lazily on the first active-item probe cycle.

Store.Reset fires observer notifications for every item (one Change per item, with non-zero Fields). mayNeedWorkObserver is always registered in Engine.Run() before bootstrap and is the mechanism that makes bootstrapped items visible to the dispatch loop via the next ticker poll — no external webhook event is needed to unblock them. wakeChObserver is also registered before bootstrap, but only when wakeCh != nil; in headless runs there is no wake channel, so visibility of bootstrapped items relies solely on mayNeedWorkObserver and the next ticker-driven poll (within PollSeconds).

If the probe fails (transient network error), the cache stays unbootstrapped and the next poll cycle retries. No data is lost. Labels are not available from probe results; they are populated by FetchItemDetails on the first active-item deep-fetch cycle.

D.2 Delta Application

Every verified webhook payload is passed to cacheImpl.ApplyDelta(eventType, payload []byte) before the poll loop is woken. ApplyDelta is a no-op when IsPaused() returns true. Otherwise it dispatches to a typed handler.

D.2.1 Unknown-item fallback

When a delta handler looks up an item that is not yet in the cache (e.g., issues.labeled arrives before issues.opened), the handler calls ensureIssueInStore(owner, fullRepo, issueNumber). This helper:

  1. Fast path: if the item is already in the Store, returns immediately.
  2. Miss path: calls fallback.FetchProjectItem(owner, repo, issueNumber) via REST GET /repos/{owner}/{repo}/issues/{number}, applies IssueOpened{Item: *pi} to the Store, then continues with the original delta.

This resolves the “fabrik went deaf” bug class where webhooks for new issues were silently dropped because the issue had not yet been seen in a board reconcile.

Exception: issues.opened itself creates the item from the webhook payload directly (no REST call needed — the payload contains full issue data). issues.transferred and issues.deleted remove the item rather than ensuring it.

D.2.2 Webhook event coverage table

All handlers hold the write lock for their mutation only. No lock is held during network calls (the ensureIssueInStore fallback fetch and resolvePRLinkage auto-heal are performed without holding c.mu — ADR 037 lock-ordering invariant).

PR→issue auto-heal and the authoritative mapping path: When a pull_request, pull_request_review, pull_request_review_comment, or check_run webhook arrives for a PR whose linked issue is not yet indexed in store.prToKey, the handler calls resolvePRLinkage to determine which issue the PR closes. resolvePRLinkage first checks store.GetByPRKey for an authoritative engine-recorded mapping. Only when no authoritative mapping is present does it fall back to fetching the PR body via REST and scanning it with the reClosingKeyword regex. The authoritative mapping is written by CacheImpl.RecordPRLinkage, which the engine calls immediately after CreateDraftPR succeeds (and also when an existing PR is found on restart), so the index is populated before any webhooks arrive for the new PR. This eliminates mismaps from prose issue references in PR bodies (e.g., “before fixes #598 landed, Closes #602” would have matched #598 first without the authoritative path).

Event type Action Handler decision Cache mutation Engine reaction
issues opened Create item from payload IssueOpened (builds item from full webhook payload; no API call) Item appears in next FetchProjectBoard
issues closed Fallback if missing; update state IssueClosed IsClosed=true; itemMayNeedWork skips unless cleanup stage
issues reopened Fallback if missing; update state IssueReopened IsClosed=false; re-enters dispatch
issues transferred Remove item store.Remove (cleans prToKey index automatically) Item gone from next board fetch
issues deleted Remove item store.Remove (cleans prToKey index automatically) Item gone from next board fetch
issues edited Fallback if missing; update title+body IssueEdited Title/Body updated; next FetchItemDetails serves updated body
issues assigned Fallback if missing; update assignees IssueAssigneesUpdated (full list replace) Assignees updated
issues unassigned Fallback if missing; update assignees IssueAssigneesUpdated (full list replace) Assignees updated
issues labeled Fallback if missing; add label IssueLabeled Labels updated; poll woken for next dispatch
issues unlabeled Fallback if missing; remove label IssueUnlabeled Labels updated; poll woken
issues milestoned, demilestoned, locked, unlocked, pinned, unpinned No-op — No engine state depends on these fields
issue_comment created Guard: item must be in Store; append comment IssueCommentCreated Comment appears in next FetchItemDetails; poll woken for comment processing
issue_comment edited, deleted No-op — Reaction-based state machine reads reactions not bodies; next deep-fetch heals
pull_request opened, closed, reopened, synchronize Look up issue via store.GetByPRKey (authoritative mapping from RecordPRLinkage); auto-heal via resolvePRLinkage (REST + regex) if not found; update SHA and PR details PRHeadSHAUpdated + PRDetailsUpdated; DeepFetchInvalidated on auto-heal shaToKey + prToKey updated; CI gate can evaluate this SHA; LinkedPR.Title/State/Merged/Draft populated
pull_request ready_for_review Look up linked issue via store.GetByPRKey; read current LinkedPR snapshot; apply Draft=false PRDetailsUpdated{Draft: false} (preserves Title/State/Merged from snapshot) LinkedPR.Draft=false; PR no longer draft; review bots can see it
pull_request converted_to_draft Look up linked issue via store.GetByPRKey; read current LinkedPR snapshot; apply Draft=true PRDetailsUpdated{Draft: true} (preserves Title/State/Merged from snapshot) LinkedPR.Draft=true; PR back to draft
pull_request review_requested Look up linked issue via store.GetByPRKey; replace reviewer list PRReviewRequested (full list replace) LinkedPRReviewRequests updated; review gate re-evaluated
pull_request review_request_removed Look up linked issue via store.GetByPRKey; remove one reviewer PRReviewRequestRemoved (remove by login) LinkedPRReviewRequests updated
pull_request labeled, unlabeled, assigned, unassigned, edited No-op — PR-level labels/assignees/metadata not tracked; Closes #N linkage healed by next Reconcile
pull_request_review submitted, edited, dismissed Route via store.GetByPRKey; auto-heal if missing; all three actions carry the full review object and are processed identically (upsert by DatabaseID) PRReviewSubmitted + PRReviewRequestRemoved (side effect); PRHeadSHAUpdated + DeepFetchInvalidated on auto-heal LinkedPRReviews updated (state stored as-is, uppercased); LinkedPRReviewRequests also updated (reviewer removed) — covers cases where review_request_removed webhook is not fired (e.g. Copilot edited reviews); review gate re-evaluated; DISMISSED state excludes the review from hasReviews in checkReviewGate
pull_request_review_comment created Route via store.GetByPRKey; auto-heal if missing ReviewThreadCommentAdded; DeepFetchInvalidated Thread comment appears; review reinvoke path can see it
pull_request_review_comment edited, deleted No-op — Thread comment content not decision-relevant; next deep-fetch heals
check_run completed Upsert in checkRuns[sha]; index via shaToKey; auto-heal if SHA unknown CheckRunCompleted; PRHeadSHAUpdated + DeepFetchInvalidated on auto-heal CI gate evaluates conclusion; LinkedPRCheckRuns updated
check_run created, rerequested, requested_action No-op — Only terminal state (completed) matters for CI gate
check_suite completed, requested, rerequested No-op — Check suites are coarse aggregates; individual runs tracked via check_run.completed. The store holds no suite state, so the suite-aware CI gate (#1822, §6.4) reads suites live via FetchCheckSuites instead of relying on this webhook
projects_v2_item created Call FetchItemDetails by content_node_id; apply as new item IssueOpened Item appears in board; triggers dispatch on next poll
projects_v2_item edited Update Status via itemIDToKey reverse lookup ProjectV2ItemEdited Status updated; stage selection uses new column
projects_v2_item deleted, archived Remove item store.RemoveByItemID (cleans prToKey index automatically) Item gone from next board fetch
projects_v2_item restored No-op — Next Reconcile re-adds the item from the full board snapshot
projects_v2_item reordered No-op — Column order not modeled in Fabrik’s cache or engine

D.3 Cache Read Semantics

CacheImpl implements boardcache.ReadClient. When the poll loop calls a read method:

Method Cache behavior
FetchProjectBoard Returns reconstructed board from items map; falls back to GitHub when cache is empty
FetchItemDetails Serves deep fields from cache when deepFetched[key] is true; falls back to GitHub and populates on miss; logs [cache] miss for #N — fetching from GitHub
FetchCheckRuns Serves from checkRuns[sha]; falls back and caches on miss
FetchLinkedPR Cache hit when snap.LinkedPR().Title != "" (set by PRDetailsUpdated); reconstructs *gh.PRDetails from Store snapshot; falls back to GitHub REST on miss and applies PRDetailsUpdated write-back
FetchPRMergeable Always delegates to fallback (mergeability changes without webhook events)
FetchPRMergeableState Always delegates to fallback
FetchLabels Serves from item.Labels when item is cached; falls back on miss
FetchStatusField Always delegates to fallback (static field metadata)
RateLimitStats Always delegates to fallback

D.4 Reconciliation

Reconciliation operates at two levels:

Periodic status-only sweep (Layer 2)

At the top of every poll() cycle (~15 s), when the in-memory cache is bootstrapped (cacheImpl != nil and cacheImpl.ProjectID() != ""), the engine runs a two-step gate:

  1. Gate step — e.client.FetchProjectUpdatedAt(projectID): fetches the single updatedAt field for the project node (node(id:$id){ ...on ProjectV2 { updatedAt } }). Cost: ~1 GraphQL point per poll cycle. If the returned timestamp equals e.lastProjectUpdatedAt (no project-level change since last cycle), the batch is skipped entirely.

  2. Batch step — fired only when the timestamp has advanced: e.client.FetchProjectItemStatusBatch(projectID) returns itemNodeID → statusName for every board item (paginated in pages of 100, no nested fields). The result is passed to cacheImpl.ApplyStatusBatch(updates).

ApplyStatusBatch holds the write lock for one pass:

After a successful batch step, e.lastProjectUpdatedAt is updated to the new timestamp so the next poll cycle can compare correctly.

First poll cycle: e.lastProjectUpdatedAt starts at the zero value (time.Time{}). The first real updatedAt returned by GitHub is always later than zero, so the batch fires on the first cycle. This catches any Status drift that occurred before engine start.

GitHub API limitation: GitHub’s ProjectV2.items does not support a since filter or orderBy: UPDATED_AT. The full batch (FetchProjectItemStatusBatch) is the only way to identify which specific items changed once the gate fires. The Store’s existing diff in ApplyStatusBatch de-dupes unchanged-Status items so no spurious observer notifications are emitted.

Gate placement: The gate runs before e.readClient.FetchProjectBoard(...) in poll(), so the cache holds the latest Status values when the board is read and items are dispatched in the same cycle.

Gate inactive when: e.readClient is not a *boardcache.CacheImpl (non-cache mode), or cacheImpl.ProjectID() is empty (bootstrap not yet complete).

Per-poll probe (runProbeAndDeepFetch)

After the Layer 2 status gate above, every poll() cycle on a bootstrapped cache calls runProbeAndDeepFetch(cacheImpl). This replaces the previous unconditional full-shallow FetchProjectBoard + Reconcile call, reducing per-poll GraphQL cost ~5–10× on idle boards by eliminating the labels(first:30) and closedByPullRequestsReferences(first:5) nested connections.

The probe loop:

  1. Calls e.client.ProbeProjectBoard(...) — fetches only scalar fields per item plus closedByPullRequestsReferences(first:1) for linked-PR drift detection. No labels connection at any nesting level.

  2. For each probe item, computes effectiveUpdatedAt = max(issue.updatedAt, projectItem.updatedAt, linkedPR.updatedAt).

  3. Stage-membership guard: checks stages.FindStage(e.cfg.Stages, pi.Status).
    • New item (not yet in store): if the column has no matching Fabrik stage, the item is skipped entirely — no store.Apply, no FetchItemDetails. It is still recorded in the newKeys tracking set before the guard fires, so it is not evicted by the post-loop tombstoning pass.
    • Existing item (already in store): ProbeBoardItemUpdated is applied so Status, IsClosed, and related fields stay current (important when items drift into or out of unconfigured columns), but FetchItemDetails is skipped.
  4. New item (not in store): seeds minimal state via IssueOpened, then calls FetchItemDetails unconditionally to populate labels and deep fields.

  5. Linkage drift (probe.linkedPRNumber ≠ cached LinkedPR.Number) — on a warm cache (LastDeepFetchAt set), applies DeepFetchInvalidated to reset LastSeenSourceUpdatedAt, then falls through to the staleness check. The check is subject to three rules (#2080, ADR 2080):

    • Terminal items are exempt (R2). An item that is flagged Terminal in the same cleanup stage, or is closed in a cleanup stage with no worktree on disk (the quiet probeOnlyTerminal predicate — the same test as isProbeOnlyTerminal, without its log lines), skips the drift check entirely: nothing about its linkage changes Fabrik’s work. This runs ahead of the check, not after it.
    • A (cached, probe) pair invalidates once (R1). The engine-local probeDriftLedger (engine/probe_drift.go) records the pair last invalidated per item. The probe’s 0 is ambiguous — it means “no PR” or “a PR that closedByPullRequestsReferences omits” (a closed, unmerged PR, which neither the probe nor the deep fetch returns; a REST lookup finds it and writes it to the cache) — so a deep fetch can legitimately leave the cached number in place. A repeat of the same pair is therefore not invalidated again and is logged once (linkage disagreement persists after deep fetch); a different pair (a genuine PR A → PR B, or 0 → N) invalidates once; when cached and probe agree the pair is forgotten, so returning to an earlier pair invalidates again. The cached LinkedPR is deliberately left untouched — a deep-fetch 0 is not authoritative (it would break base: items and PRs found through REST). The ledger is in memory, pruned when the item leaves the board or turns terminal; a restart costs one extra invalidation per affected item.
    • A loop is reported (R4). If one item causes more than probeDriftLoopThreshold (10) invalidations within probeDriftLoopWindow (1 h, on e.now()), one probe: linkage drift loop warning names the item and both PR numbers. It re-arms once the count falls back to the threshold.

    base:<branch> suppression: closedByPullRequestsReferences is only populated by GitHub for PRs targeting the repository’s default branch, so the shallow probe structurally always reports LinkedPRNumber == 0 for an item whose linked PR targets a non-default base:<branch>, even while the warm deep cache correctly holds the real PR number (resolved via the base-independent linkage path). To avoid perpetual false-positive invalidation on every poll, the drift check is suppressed — left as a true no-op, no DeepFetchInvalidated and no PRDetailsUpdated — when all three hold: the probe reports LinkedPRNumber == 0, the cached LinkedPR.Number is non-zero, and the cached item (read from the Store’s ItemState.Labels, never from the probe item — probe items carry no Labels field) satisfies itemHasBaseLabel. A probe reporting a different non-zero PR number on a base-labeled item is still treated as genuine drift and invalidates as usual. Default-branch behavior (no base: label) is unaffected — the suppression can never fire since itemHasBaseLabel is false.

  6. Applies ProbeBoardItemUpdated (updates IsClosed, State, IsPR, Status, UpdatedAt; explicitly skips Labels to preserve the cached label set populated by prior deep-fetches or webhook deltas).

  7. Staleness check: calls cacheImpl.IsItemCacheFresh(repo, number, boardcache.EvidenceFromProbe(pi)). The cache is fresh only if the issue/project-item updatedAt is not after LastSeenSourceUpdatedAt, the linked PR’s updatedAt is not after LastSeenPRUpdatedAt, and GitHub’s issue comment total equals the cached issue comment count (the three-part rule under “Self-write staleness baseline” below, #2180). If fresh, continues without GitHub traffic. If stale, calls FetchItemDetails to refresh.

  8. After the item loop, removes from the store any items no longer present in the probe result (they left the board). A guard skips removal when the probe returns 0 items and the cache is non-empty (transient indexer hiccup).

FetchItemDetails writes ItemDeepFetched to the Store, updating LastSeenSourceUpdatedAt to the new effectiveUpdatedAt and LastSeenPRUpdatedAt to the linked PR’s own updatedAt. The candidates loop that runs later in the same poll cycle sees these items as fresh (via IsItemCacheFresh) and skips duplicate fetches.

Self-write staleness baseline (SelfWriteObserved)

The issue-layer part of the staleness check in step 7 compares the issue/project-item updatedAt against LastSeenSourceUpdatedAt, which is otherwise written only by a full deep-fetch (ItemDeepFetched). Without a second writer, every one of Fabrik’s own board mutations — which bump the real GitHub updatedAt just like an external edit — would make the very next probe cycle see the item as stale and force an immediate FetchItemDetails, even though nothing actionable changed (the #1083 incident: a self-posted reply forced its own re-fetch on the next poll, running the ping-pong loop as fast as polling allowed instead of backing off).

itemstate.SelfWriteObserved{Repo, Number} closes this gap: applied via e.store.Apply(...) directly (bypassing boardcache.CacheImpl, so it needs no cache-wired guard and is not gated on e.webhookMgr), it advances LastSeenSourceUpdatedAt to the local wall-clock time at the moment of the self-write — no deep-fetch, no other field touched. The advance is monotonic (store.go’s applyToItem only updates when time.Now() is after the current baseline), so it never regresses an earlier SelfWriteObserved, and a DeepFetchInvalidated reset (zero value) always “wins” against a stale SelfWriteObserved racing behind it. Only SelfWriteObserved is monotonic: ItemDeepFetched assigns the fetched value unconditionally, which is how a real refetch lowers a clock-anchored baseline.

It is wired into exactly five call-site groups, mirroring the existing webhook RegisterEcho/RegisterEchoIfSubscribed success-gating at each:

A failed mutation at any of these sites never advances the baseline, so the next probe still correctly treats the item as stale. And because the mechanism only ever advances the baseline to the current wall clock, a genuine external change that lands with a later effectiveUpdatedAt (a human/bot comment, label, or status change arriving after our self-write) still compares as stale and triggers a real deep-fetch — the suppression only ever applies to the self-write’s own updatedAt bump, never to activity after it.

Display-field writes (status line, Last activity, Last run) are deliberately not call sites (#2180): the board and probe parsers already discount a project-item bump explained by a display field’s own updatedAt, so those writes move no updatedAt the cache compares and anchoring the baseline would only hide an external change.

Known scope boundary: this covers only the call sites above. PR-body self-writes (engine/pr.go’s updatePRVerification/ensurePRLinksIssue, engine/prcreate.go’s linkage-heal edit) and a second issue-body-edit site (engine/item.go’s stage-output publishing path) bump a component of effectiveUpdatedAt the same way but are not wired to SelfWriteObserved — a deliberate, documented gap (see ADR-044 Addendum 4), not an oversight, left as a candidate follow-up if it proves to matter in practice.

The freshness rule: self-write anchoring may absorb only what the engine can account for (#2180, ADR 2180)

One scalar compared with one scalar cannot tell “my write at T” from “an external change at T−ε plus my write at T”, so SelfWriteObserved alone masked external changes that predate it (the #2125 shape). boardcache.cacheIsStale(Evidence, ItemState) therefore serves a cached deep copy only when all three hold:

  1. Issue layer. Evidence.IssueLayer (the later of the issue’s and the project item’s updatedAt, display-field bumps discounted, linked PR excluded) is not after LastSeenSourceUpdatedAt. This is the baseline SelfWriteObserved advances and the only layer the engine writes.
  2. PR layer. The linked PR’s own updatedAt is not after ItemState.LastSeenPRUpdatedAt, set only from GitHub values by ItemDeepFetched and zeroed by DeepFetchInvalidated. SelfWriteObserved never touches it, so an issue-side self-write cannot hide a review, thread comment or PR comment.
  3. Comment evidence. GitHub’s comments { totalCount } for the issue equals the number of issue comments in the cached copy (FromPR == 0, no path), unless the cached copy already holds a first page (100). The engine’s own comments are written through the cache (LocalCommentAdded, idempotent by DatabaseID), so a self-write keeps the two equal; a human comment hidden by the issue-layer baseline makes them differ.

A linked PR on the board that the cache has no record of (no PR baseline and no cached PR) means a PR appeared after the fetch and reads as stale. When a producer did not capture the layers (zero IssueLayer, or no PR baseline for a cached PR) the predicate compares the overall updatedAt with LastSeenSourceUpdatedAt as before; a zero overall updatedAt is fresh (no signal, no hot-loop). The evidence rides ProjectItem/BoardProbeItem (IssueLayerUpdatedAt, LinkedPRUpdatedAt, IssueCommentTotal/IssueCommentTotalKnown; the board and probe queries select comments { totalCount } on the Issue fragment) and is mirrored on ItemState. A webhook stamp (localDeltaAt) raises the issue layer too. Every refetch re-anchors all baselines and rebuilds the comment list, so a mismatch cannot persist across polls: engine self-writes on the issue layer cost no refetch (the #1090 guarantee), and an engine write on the PR side (pr.go, prcreate.go, a push) costs one deep fetch at the next probe.

CI is outside updatedAt. A check completion does not change any updatedAt the board reports, so no baseline rule can reveal it. boardcache.FetchCheckRuns serves a cached set on trust only when it classifies READY; PENDING and FAILED sets are confirmed live and written back, and a PENDING set confirmed within 15 s (checkRunsConfirmTTL) is served, so several readers in one poll cost one REST call. This subsumes the #1303 per-poll prime that settleAwaitingCIScan used to run (RefreshCheckRunsLive, removed).

Cost. The added comments { totalCount } selection measured 0 extra points on the dev board’s listing queries (probe 1 point per page of 100, board listing 2 points per page, with and without it).

Backstops. The #1944 paused-item backstop is removed (§7.9 paused deep-fetch skip); the ADR 2160 gate repair (§6.6.6) is kept because it also guards causes this rule cannot (the latestReviews cap, per-author collapse).

Terminal-item skip

An item is terminal once Fabrik has confirmed it has nothing left to do: its status is a cleanup stage (e.g. Done), its labels include stage:<StageName>:complete, and no transient lifecycle label (transientLifecycleLabels — see “Transient label sweep” above) or lock label (fabrik:locked:*) is present.

Terminal state is stored as a Terminal bool field in ItemState and set/cleared via the TerminalFlagSet mutation.

When the flag is set: after a successful FetchItemDetails call in either runProbeAndDeepFetch or the admit pass, if isTerminalPredicate(labels, status, stages) returns true. Logged once (first-time only) at [poll] terminal flag set.

When the flag is cleared:

Probe-loop skip: the terminal gate in runProbeAndDeepFetch runs before ProbeBoardItemUpdated is applied (ordering constraint: the cached Terminal flag must be read before applyProbeItem would clear it on a status change). If Terminal == true and the probe’s status is still a cleanup stage, the item is skipped without any deep-fetch. If Terminal == true but the probe shows a non-cleanup status, the flag is cleared and the item falls through to normal probe processing. The linkage-drift check (step 5 above) now consults the same terminal condition first (R2, #2080), so a terminal item is never invalidated by drift on its way to being skipped.

Admit-pass skip: the same gate in the per-poll admit pass (before the cycleSet check) unconditionally skips items that are terminal and still in the cleanup stage. This guard is redundant for the normal probe path (where runProbeAndDeepFetch already handled the item) but covers non-cache and paused-cache modes.

Cold-start (startup scan): after runStartupTransientLabelScan removes stale transient labels, runStartupTerminalScan iterates all items in the Store and applies TerminalFlagSet{Terminal: true} to any that pass the terminal predicate. This uses bootstrap label data already present in the Store — no GitHub API call is needed. On a 362-item board with 340 terminal Done items, this eliminates ~340 deep-fetches on the first poll cycle.

Reconcile is drift-recovery only: cacheImpl.Reconcile(board) is called only during drift-recovery, independent of webhook mode (see below). It is no longer called on every poll cycle, and it is not called during cold-start bootstrap — that path uses BootstrapFromProbe instead.

Full reconcile — drift-recovery (runs independent of webhook mode)

LightReconcile returns the fresh board snapshot when drift is detected, which the reconcileTicker goroutine passes directly to cacheImpl.Reconcile(board) — avoiding a second API call. The older full-fetch-and-reconcile helper (reconcileCache) has been removed; reconcileLoop’s LightReconcile path is the sole reconciliation mechanism.

Reconcile performs a deep partial update:

Engine-write protection (#2159, ADR 2159). A listing is authoritative about changes made outside the engine, not about the engine’s own writes made after the listing was taken — and GitHub’s board listing lags writes by seconds (the #1974 probe measures it). Without protection a reconcile that ran 1–2 s after an engine label write received the pre-write listing, counted the difference as drift and put the old value back (observed on 2026-10-11: fabrik:awaiting-ci back on #2091, fabrik:awaiting-landing-verification missing from #2097, stage:Review:complete missing from #2116), leaving every gate that reads the cache wrong until the next tick (~3 min) and emitting a false unknown-actor label event.

D.5 Stream-Health Failover

Health transitions are driven by the reconcileTicker goroutine (engine/reconcile.go), which calls cacheImpl.LightReconcile(...) on each tick (default: every 3 minutes, configurable via --reconcile-interval / FABRIK_RECONCILE_INTERVAL):

LightReconcile result Action
No drift (a difference that is only a recent engine write is not drift — see D.4, #2159) wm.transitionHealthState(Healthy, "") — no-op if already healthy; logs if recovering from unhealthy
Drift detected wm.transitionHealthState(Unhealthy, reason) → cacheImpl.Pause() → cacheImpl.Reconcile(freshBoard) → cacheImpl.Resume() → wm.transitionHealthState(Healthy, "drift reconciled")
Network error Log warning; no state change (treat as “unable to determine”, not drift)

When IsPaused() is true:

On drift recovery, LightReconcile returns the fresh board it already fetched; this is passed directly to cacheImpl.Reconcile(freshBoard) before Resume(), so the cache is coherent immediately when IsPaused() returns false.

API cost: Each LightReconcile tick costs ~1 GraphQL call (FetchProjectBoard). At the default 3-minute cadence: ≤ 20 calls/hour — negligible against the 5000-point/hour budget.

healthChangeFn removed (issue #641): The callback-based healthChangeFn pattern (ADR-034) has been replaced. The reconcileTicker goroutine owns the full Pause/Reconcile/Resume sequence directly; no indirection through a closure is required.

D.6 Cache Mode

The --board-cache flag has been removed. CacheImpl is now wired unconditionally for production runs. The cache is freshened by:

  1. Per-poll probe (runProbeAndDeepFetch) (always active on bootstrapped cache) — see D.4
  2. Layer 2 status sweep (always active) — see D.4 and D.7
  3. Webhook deltas (when --webhooks is enabled) — see D.2
  4. Light-reconcile drift recovery (always active, independent of --webhooks) — see D.5

Tests that need to bypass the cache use engine.NewWithDeps, which wires boardcache.GitHubAdapter directly. This bypasses the Store-mutation pipeline and silently disables observers, so it is intended only for unit tests that don’t exercise Store-driven behavior.

D.7 Status field reconciliation in user mode

In user mode (PAT-based, repo-level webhooks via gh webhook forward), GitHub does not deliver projects_v2_item events. Board-column changes — including the Status field that drives stage selection — are invisible to the webhook stream. Fabrik uses a four-layer strategy to keep the cached Status current despite this gap:

Layer Mechanism Latency Cost
0 Write-through After any successful mutation call in the engine (status, labels, comments), the corresponding CacheImpl method is called immediately Zero Zero
1 Per-event refresh After ApplyDelta for an issues or issue_comment event, fetches the current Status: fast path (FetchProjectItemStatus) when the item’s itemID is cached; fallback (LookupIssueProjectItem) when it isn’t yet (e.g., brand-new issues arriving via issues.opened before projects_v2_item.created) Seconds (best-effort) ~1–5 pts/event
2 Periodic status sweep In-poll updatedAt gate: FetchProjectUpdatedAt (~1 pt) every poll cycle (~15 s); fires FetchProjectItemStatusBatch + ApplyStatusBatch only when project timestamp advances Up to 15 s ~1 pt/cycle idle; O(⌈N/100⌉) pts when gate fires
Startup bootstrap ProbeProjectBoard + BootstrapFromProbe on cold start (both webhook and polling-only modes) Startup only ~250 nodes
Drift-recovery Full FetchProjectBoard + Reconcile when reconcileTicker detects drift Minutes Full board cost

Residual latency: For external column moves (user moves issue on the board), the upper bound on detection latency is the main poll cadence (~15 s) — Layer 2 runs at the top of every poll() cycle. If the column move happens to coincide with any other issue activity (a comment, label change, etc.), Layer 1 may catch it sooner. Layer 0 applies only to Fabrik’s own mutations.

Layer 0 write-through convention: Every engine call site that mutates dispatch-relevant cache state must call the corresponding CacheImpl write-through method immediately after the API call succeeds. The safe type-assertion pattern used at every site:

if cacheImpl, ok := e.readClient.(*boardcache.CacheImpl); ok {
    cacheImpl.<Method>(boardcache.ItemKey(item.Repo, item.Number), ...)
}

This is a no-op when e.readClient is a GitHubAdapter (cache-disabled mode), so the guard is always safe to include.

Covered mutation types and their CacheImpl methods:

Mutation CacheImpl method Dispatch relevance
UpdateProjectItemStatus UpdateItemStatus Stage selection reads Status
AddLabelToIssue ApplyLabelAdded Label guards (fabrik:locked, stage:*:complete, etc.)
RemoveLabelFromIssue ApplyLabelRemoved Same
AddComment (issue-targeted) ApplyCommentAdded Comment dispatch reads Comments for rocket-reaction checks

Excluded mutation types (annotated with // no write-through: excluded — at each call site):

Mutation Reason excluded
AddCommentReaction Reactions are not read from cache for dispatch decisions
AddPRReviewCommentReaction Same
AddComment targeting a PR number Posts to PR comment thread, not issue cache
UpdateIssueBody Issue body is not read from cache for dispatch decisions
CreateDraftPR PR existence is resolved live via FindPRForIssue; not cached
MarkPRReady Draft state is not read from cache for dispatch decisions
MergePR Merge state is not read from cache for dispatch decisions

CacheImpl existence guard: All three new write-through methods (ApplyLabelAdded, ApplyLabelRemoved, ApplyCommentAdded) call c.store.Get(repo, number) before applying the mutation. If the key is not present in the Store (i.e., was never bootstrapped), the method returns without creating a phantom Store entry.

Layer 1 scope: Only issues and issue_comment event types trigger a per-event status fetch. pull_request, pull_request_review, check_run, and other event types are excluded from Layer 1 (finding the linked issue for a PR event requires an O(N) cache scan; check_run does not carry an item ID). Layer 2’s per-poll-cycle gate covers these cases within ~15 s. PR conversation comments are skipped (#2093): an issue_comment whose issue object carries a pull_request key returns before any cache or GraphQL access and logs nothing, because repository.issue(number:) cannot resolve a PR number.

For new issues whose itemID is not yet in the cache (e.g., added via issues.opened before a projects_v2_item.created event is received), Layer 1 falls back to LookupIssueProjectItem to populate both itemID and current Status in one GraphQL query (repository.issue.projectItems). If cache.ProjectID() is empty (Bootstrap not yet complete), the fallback is skipped. After a successful fallback, subsequent Layer 1 calls for the same issue use the cheaper FetchProjectItemStatus fast path.

Negative cache (#2093, engine/layer1_offboard.go): a fallback lookup that succeeds with no item (the issue is not on the board) is remembered in memory for 10 minutes per (project, repo, number), so further events for that issue make no call. Errors are never cached, so a transient failure is retried on the next event. The fast path (GetItemID) is consulted before the entry, so any path that gives the cache an itemID bypasses it, and every projects_v2_item created event clears all entries (that payload carries no repo/number to target one). A restart clears the cache.

Appendix E: Goroutine-Boundary vs. Work-Boundary Lifecycle

Two parallel lifecycle mechanisms track active work in Fabrik. They exist at different abstraction layers and must not be conflated.

E.1 Engine-State Lifecycle (Goroutine Boundary)

WorkerEntered / WorkerExited mark the goroutine boundary — the interval during which a dispatch goroutine is allocated and running. Introduced in Fix B (issue #544, PR #568).

Event Applied Where
WorkerEntered{StageName, StartedAt} Synchronously before wg.Add(1) and goroutine launch engine/poll.go (main dispatch), engine/reviews.go, engine/ci.go, engine/merge_gate.go
WorkerExited{} Deferred at the top of each goroutine — fires on every exit path Same four files

Purpose: Prevent double-dispatch. snap.Worker() != nil is the guard. Because WorkerEntered is applied before the goroutine starts, the guard is effective from the instant the goroutine is scheduled. WorkerExited wakes the poll loop via WorkerLifecycleChanged so the next stage dispatches promptly after the goroutine exits.

Coverage: All exit paths including processItem early-returns, context cancel, ensureRepoReady failure, and normal completion.

E.2 TUI-Presentation Lifecycle (Work Boundary)

JobStartedEvent / JobCompletedEvent mark the work boundary — the interval during which Claude is actually running. Introduced in issue #578.

Event Emitted Where
JobStartedEvent{...} Past all early-return guards, at the “committed to real work” point engine/item.go (after lock acquired + WorkerExited deferred); engine/comments.go (at function entry, after log line)
JobCompletedEvent{Skipped:true} (cleanup guard, unconditional) Deferred immediately after JobStartedEvent; fires on every function return including success Same two sites — HistoryPaneComponent filters these out; only InvocationObserver’s Skipped:false event reaches history
JobCompletedEvent{Skipped:false} (authoritative success path) By InvocationObserver when InvocationRecorded is applied to the store engine/observers.go

Purpose: Drive the TUI active pane. “In Progress (N)” reflects items where Claude is running, not items whose goroutines have merely launched.

Why the distinction matters: A goroutine that launches and immediately hits a fabrik:blocked early-return allocates a worker slot for ~1ms but does not invoke Claude. The goroutine-boundary lifecycle must track it (to prevent double-dispatch). The TUI-presentation lifecycle must not track it (to avoid ghost “In Progress” entries in the active pane that never clear).

Constraint for contributors: Do NOT move JobStartedEvent emission back to the goroutine-launch site (dispatch goroutines in poll.go or reinvoke dispatchers). Ghost entries from early-return paths have no matching JobCompletedEvent and persist in the active pane indefinitely. Issue #578 was filed for exactly this bug. The constraint is documented in the ActivePaneComponent struct comment in tui/active.go.

E.3 Summary of Differences

Dimension Engine-State Lifecycle TUI-Presentation Lifecycle
Events WorkerEntered / WorkerExited JobStartedEvent / JobCompletedEvent
Boundary Goroutine lifetime Claude invocation interval
Applied at Goroutine launch / goroutine exit Past all early-return guards / invocation return
Coverage Every goroutine including early-returns Only goroutines that invoke Claude
Consumer itemstate.Store observers (dispatch guard, wake channel) tui.ActivePaneComponent (active pane display)
Idempotent on double-fire? Yes (WorkerExited is a no-op when Worker is nil) Yes (delete(a.active, key) is a no-op on missing key)
Introduced PR #568 (Fix B, issue #544) Issue #578