Fabrik Issue State Machine
Every issue in Fabrik follows a defined lifecycle: from intake through a series of AI-driven stages (Specify → Research → Plan → Implement → Review → Validate → Done), with automated gates at key transitions. The diagram below shows the happy path at a glance.
Not an engineer? The diagram and the Pipeline Overview table are the fastest way to understand Fabrik’s workflow.
Engine contributor or debugger? The dense reference below covers every reachable state, every label mutation, every guard condition, and the visual state diagrams in §10. Use the State Enumeration section as the authoritative source when diagnosing unexpected engine behavior.
This document is the formal specification of Fabrik’s issue-level state machine: how an issue moves between states across multiple invocations of the engine. It covers every reachable state, every event that triggers a transition, every label mutation, and every guard condition.
Companion document: stage-lifecycle.md describes the per-invocation lifecycle (what happens before, during, and after a single Claude invocation). This document describes the cross-invocation state machine (how an issue progresses through the pipeline over time). They are complementary.
As-built specification: This document describes what the code actually does, not what it ideally should do. Discrepancies between intended and actual behavior are flagged with > **Bug?:** callout blocks.
Source of truth for: state enumeration, transition tables, label semantics, and guard conditions. Supersedes partial label references in CLAUDE.md.
Pipeline Overview
Issues traverse a linear pipeline of stages, each corresponding to a column on the GitHub Project board:
Specify → Research → Plan → Implement → Review → Validate → [Queued†] → Done
| Stage | Order | Read-Only | PostToPR | CreateDraftPR | MarkPRReady | WaitForReviews | CleanupWorktree |
|---|---|---|---|---|---|---|---|
| Specify | 0 | Yes | No | No | No | No | No |
| Research | 1 | Yes | No | No | No | No | No |
| Plan | 2 | Yes | No | No | No | No | No |
| Implement | 3 | No | Yes | Yes | Yes | No | No |
| Review | 4 | No | Yes | No | Yes | Yes* | No |
| Validate | 5 | No | Yes | No | No | Yes* | No |
| Queued† | 6 | — | — | — | — | — | — |
| Done | 99 | N/A | No | No | No | No | Yes |
Specify additionally ships persist_spec: true (§7.16): it stays read-only, but the engine commits the issue-body spec to specs/<issue>-<slug>/spec.md.
* All flags in this table reflect the default stage configuration shipped in .fabrik/stages/. Each flag is opt-in per stage YAML and may differ in custom configurations. wait_for_reviews is enabled for Review and Validate in the defaults.
† Queued is a holding stage (holding_stage: true) active only when merge_train: on. Items enter Queued when attemptMergeOnValidate routes to advanceToQueued instead of GitHub auto-merge. Per-item dispatch is suppressed (itemMayNeedWork / itemNeedsWork return false for holding stages); the batch handler handleMergeTrainBatch snapshots the Queued items once per poll cycle via groupQueuedByRepoAndBase (engine/poll.go, #1648), which excludes closed and fabrik:paused members (see the poison-well guard under Merge-Train Red-Batch Bisection) and further partitions the remaining items by resolved base branch (baseBranchForItem) — one queuedRepoGroup per distinct (repo, base) pair actually present, not one group per repo. routeQueuedGroup runs once per partition and dispatches runMergeTrainWorker for each when its own batch is ready, so a repo with members on both the default branch and a base:<branch> label runs two fully independent trains simultaneously — each with its own pinned base SHA, trial branch, draft CI PR, and integration PR targeting its own base (ADR-1648, superseding ADR-1647’s exclusion of such members from batching entirely). On a green trial-branch CI result, landMergeTrainBatch opens a landing integration PR, merges it, advances each member from Queued → Done via advanceToNextStage, and closes their individual PRs and issues (ADR-059 D3). The Queued column must exist on the GitHub Project board when merge_train: on; it is validated at startup.
1. State Enumeration
A state is defined by the tuple (BoardColumn, ControllingLabelSet). Not every label combination is a valid state — only reachable combinations are enumerated.
1.1 Controlling Labels
These labels define distinct states (their presence changes what the engine does with an item):
| Label | Type | Defines State? |
|---|---|---|
fabrik:locked:<user> |
Lock | Yes — gates processing by other instances. <user> is cfg.User in PAT mode; under GitHub App auth it is the instance’s App-derived lock identity <slug>-<hash> (§7.4) |
fabrik:editing |
Mutex | Yes — prevents stage dispatch during comment processing |
fabrik:paused |
Pause | Yes — blocks processing on active stages unless a human comment arrives (cleanup stages ignore comments entirely while paused; a Validate-stage item with an externally merged PR can still advance via runValidatePRTerminalAdvance regardless of this label). Also applied by the comment-processing circuit breaker (§4.6) after N non-advancing comment-processing invocations within a rolling window, or by the success-agnostic no-op comment cycle breaker (§4.7) after N consecutive no-progress comment-processing invocations for the same stage, regardless of window. Also suppresses deep-fetch admission (not just dispatch) for items with no new activity this poll — see Appendix B’s “Paused deep-fetch skip” note (#1379). |
fabrik:awaiting-input |
Sub-pause | Yes (with fabrik:paused) — blocked-on-input variant; also both comment-processing circuit breakers’ trip pairing (§4.6, §4.7) |
fabrik:awaiting-review |
Gate | Yes — review gate is active |
fabrik:awaiting-ci |
Gate | Yes — CI gate is active; waiting for CI checks to pass (checks may be running or have failed) |
fabrik:rebase-needed |
Gate | Yes — merge-conflict gate is active; PR is not mergeable against its base |
fabrik:awaiting-done |
Gate | Yes — a FABRIK_NO_WORK_NEEDED decision has been made; the Done board move and/or issue close is still outstanding and is retried every poll, independent of item.Status |
fabrik:awaiting-placement |
Gate | Yes — a spawned child’s initial project-board Status placement is still outstanding; retried every poll, independent of item.Status/stages.FindStage resolving a stage for the child’s current column |
fabrik:awaiting-close |
Gate | Yes — closeIssueIfNonDefaultBase’s explicit CloseIssue call (ADR-1096) failed — or, on the default base, GitHub’s Closes #N auto-close had not closed the issue by the landing (#1962); the close is still outstanding and is retried every poll, independent of item.Status |
fabrik:awaiting-pr-ready |
Gate | Yes, but does not suppress dispatch — markPRReady’s client.MarkPRReady call (§5.2, ADR-1582) failed non-transiently or exhausted its in-process retry; the mark-ready is still outstanding and is retried every poll, independent of item.Status. Unlike most of this table’s other Gate rows, the item continues advancing through later stages normally while this label is present (§6.22) |
fabrik:nondefault-base-pr-noted |
Informational | No — does not gate dispatch or suppress anything; marks that the one-time non-default-base linkage notice (§6.21, ADR-1649) has already been posted (or attempted) for this item |
fabrik:awaiting-advance |
Gate | Yes — a terminal advance (advanceToNextStage, called from advanceValidateTerminalItem’s merged-PR path or advanceConvergedPRToDone) failed to move the project-board Status forward — most commonly a missing target Status option; the advance is still outstanding and is retried every poll, independent of item.Status |
fabrik:awaiting-runaway-alert |
Gate | Yes — the merge-train runaway guard (ADR-059 D8) already paused this member (fabrik:paused + fabrik:awaiting-input applied), but its AddComment alert call failed; the alert is still outstanding and is retried every poll, independent of item.Status (ADR-1533) |
fabrik:blocked |
Dependency | Yes — blocked by open dependency issues |
stage:<X>:in_progress |
Progress | Yes — a stage invocation is active |
stage:<X>:complete |
Completion | Yes — stage finished successfully |
stage:<X>:failed |
Failure | Yes — stage exhausted retry limit |
fabrik:revalidate |
Operator trigger | Transient — consumed by the revalidate-scan loop; triggers Validate re-entry then removes itself |
1.2 Modifier Labels (Guard Conditions)
These labels do not define distinct states but influence transition behavior:
| Label | Effect |
|---|---|
fabrik:yolo |
Forces auto-advance; triggers GitHub native auto-merge at Validate; overrides auto_advance: false |
fabrik:cruise |
Forces auto-advance without auto-merge; stops at Validate completion; cruise wins over yolo for both end-of-Validate decisions — the PR is not auto-merged and the issue does not advance to Done, even when yolo is also present |
fabrik:auto-merge-enabled |
Engine-internal; marks that GitHub’s native auto-merge has been enabled on the linked PR; anchors the convergence budget start time; bypasses legacy merge/CI gates |
fabrik:unrestricted |
Passes --dangerously-skip-permissions to Claude Code |
fabrik:extend-turns |
Pre-grants a 2× turn budget for the first stage invocation and first comment processing invocation while present, with max_wall_time scaled by the same 2× factor for that invocation (§7.7, ADR-1206); persists across stages; removed only at the Done cleanup stage or manually; no-op when max_turns == 0 (stage) or always applies for comments since commentMaxTurns is never 0 |
model:<name> |
Selects a specific model for this issue (e.g., model:opus) |
effort:<level> |
Overrides stage effort level (low, medium, high, max); highest wins |
base:<branch> |
Overrides worktree base branch; falls back to default if not on remote (checked via an authoritative git ls-remote probe, not just the local clone — see §1.3); updates PR base if PR exists |
review-authority:<mode> |
Overrides the stage’s configured review_authority (advisory/authoritative) for this issue only; only meaningful alongside wait_for_reviews: true. No label → stage config governs. One recognized label → it wins. Both present → resolves to authoritative (more restrictive), with a logged warning. Malformed/unknown suffix → ignored, with a logged warning, falls back to stage config. See §6.1.1 and ADR-1261 |
expected-reviewers:<mode> |
Overrides the stage’s configured expected_reviewers (none/declared) for this issue only; only meaningful alongside wait_for_reviews: true. No label → stage config governs (nil stays nil). none → resolves to &[]string{} (enables the FR-2 fast-advance path). declared → resolves to a fixed synthetic reviewer identity. Both present → resolves to declared (more restrictive — imposes waiting, unlike none’s immediate advance), with a logged warning. Malformed/unknown suffix → ignored, with a logged warning, falls back to stage config. See §6.1 and #1304 |
fabrik:sub-issue |
Informational; marks issue as created by Fabrik’s sub-issue spawn mechanism |
fabrik:credited-pr:<N> |
Engine-internal; applied only by the two merge-train landing paths (landMergeTrainBatch, landSingleton) alongside fabrik:awaiting-landing-verification, recording the integration/singleton PR credited for the Done transition — never applied by the ordinary auto-merge path, whose credited PR is always rediscoverable via FetchLinkedPR. See §6.19 |
1.3 Reachable States by Board Column
For each board column, the reachable sub-states are listed. States are written as Column + {labels}. An issue in a column with no controlling labels is in the Idle sub-state for that column.
Specify / Research / Plan / Implement / Review / Validate (Active Stages)
Each active stage column has the same set of reachable sub-states:
| Sub-State | Labels Present | Description |
|---|---|---|
| Idle | (none of the controlling labels) | Ready for the engine to pick up |
| Locked + In Progress | fabrik:locked:<user>, stage:<X>:in_progress |
Stage invocation is active |
| Editing | fabrik:editing |
Comment processing is active (Claude invoked for comment review) |
| Paused | fabrik:paused |
Manually paused or engine-escalated pause; no work until unpause or a human comment (bot-authored comments do not resume). Exception: a Validate-stage item whose PR was externally merged advances via runValidatePRTerminalAdvance regardless of this label. |
| Paused + Failed | fabrik:paused, stage:<X>:failed |
Engine paused after MaxRetries exhausted |
| Awaiting Input | fabrik:paused, fabrik:awaiting-input |
Claude signaled FABRIK_BLOCKED_ON_INPUT; waiting for user comment. Engine posts a dedicated 🏭 **Fabrik** — @<user>: notification comment so GitHub delivers a mobile push to the configured operator (cfg.User). Under GitHub App auth there is no operator: mentionTargets mentions the item’s human assignees, else its human author, else nobody (bot logins — including the App itself, in either its REST <slug>[bot] or GraphQL <slug> form — are never mentioned; #1893). If Claude’s output included a FABRIK_SUMMARY block, the question is embedded as a blockquote. |
| Awaiting Review | fabrik:awaiting-review, stage:<X>:complete |
Review gate active; waiting for PR reviewers (only on stages with wait_for_reviews: true) |
| Awaiting CI | fabrik:awaiting-ci |
CI gate active; waiting for CI checks to pass (pending or failed); stage:<X>:complete is withheld until CI clears (only on stages with wait_for_ci: true) |
| Rebase Needed | fabrik:rebase-needed (+ fabrik:awaiting-ci when wait_for_ci: true) |
Merge-conflict detected; PR is not mergeable against its base; engine dispatching a rebase re-invocation. Applies to both the conjunctive gate path (wait_for_ci: true stages, via checkMergeabilityGate) and the legacy auto-merge path (yolo+Validate without wait_for_ci, via attemptMergeOnValidate). Distinct from a MergePR CI refusal (gh.ErrNotMergeableCI — a required check is blocked/pending, not a conflict): this label is applied only from settle.Status == PRMergeConflicting, never from MergePR’s own return value, so a CI refusal never enters this state or consumes a rebase cycle (ADR-933) |
| Awaiting Done | fabrik:awaiting-done |
A FABRIK_NO_WORK_NEEDED decision has been made; the Done board move and/or issue close is still outstanding. Dispatch is suppressed for every non-cleanup stage regardless of which column the item is (still) sitting in — the settle scan (§6.8) retries every poll until it succeeds or escalates |
| Awaiting Advance | fabrik:awaiting-advance |
A terminal advance (advanceToNextStage) failed to move the project-board Status forward — most commonly a missing target Status option; the item’s own stage is already complete (this is not a dispatch-suppression state — nothing further would dispatch here regardless). The settle scan (§6.17) retries every poll until the board is fixed or escalates |
| Blocked | fabrik:blocked |
Dependency gate active; waiting for blocking issues to close |
| Complete | stage:<X>:complete |
Stage finished; waiting for advancement (manual or auto) |
| Locked by Other | fabrik:locked:<other_user> |
Another Fabrik instance owns this issue |
| Cooldown | (no label; in-memory LastAttemptAt[stageName] in itemstate.StageState) |
Stage attempted but didn’t complete; waiting for dispatch cooldown to expire |
Note: The Cooldown sub-state is purely in-memory — there is no label for it. The engine uses
LastAttemptAt[stageName]fromitemstate.StageState(written byStageAttemptedmutation) to enforce dispatch cooldown. On restart, cooldown state is lost and the item is retried immediately.Distinct from Cooldown is Deferred Dispatch: an item whose dispatch was skipped in the current poll cycle solely because a worker from a prior cycle is still running. Deferred-Dispatch items still receive the
CooldownAt("periodic-re-eval")stamp at end-of-poll — the cooldown avoids repeated deep-fetch evaluation (and the fallback GraphQL fetch when the cache is invalidated or disabled) for an item the dispatch guard (snap.Worker() != nil) would block anyway. Prompt re-dispatch after the prior worker exits is guaranteed byWorkerExited → WorkerLifecycleChanged, which is inwakeChFlagsand wakes the poll loop immediately (#544). Note:WorkerLifecycleChangedis excluded fromcycleSetFlags(Fix B, issue #576) so it does not bypass the cooldown gate viamayNeedWork. For successful workers,StatusChangedfromadvanceToNextStageadds the item to cycleSet (bypassing the cooldown) so the next stage dispatches promptly. For workers that ended without completing — whether preempted at the turn cap or genuinely errored — the exit records its ownretry-backoffcooldown (§7.1, #2232), whose expiry admits the item on the first poll after the backoff. A second kind of deferral is slot-deferred dispatch (#2114): the candidate passed every guard but everyEngine.semslot was busy. It is deferred with nothing recorded against it and, unlike the worker-guard case above, is not stamped withperiodic-re-eval; it is re-queued intomayNeedWorkso the next poll admits it throughcycleSet. See §9.9.1. See §3.2, §9.2, and §9.9.Not listed above:
fabrik:awaiting-placementis a related but distinct marker applied to a spawned child issue whose initial board placement failed — by construction it is sitting outside any of these active-stage columns (typicallyBacklog, which is declaredunmanaged: true— see below — and therefore never dispatched to regardless), so it never appears as a sub-state of an active-stage column. See §6.9.
Backlog (Unmanaged Stage)
| Sub-State | Labels Present | Description |
|---|---|---|
| Parked | (any / none) | Item sits in the column; engine never inspects it beyond the stage-membership check |
itemMayNeedWork and itemNeedsWork both return false for any stage with unmanaged: true, so these items are never dispatched to a per-item worker and never auto-advanced by the catch-up loop (same mechanism as holding_stage, see below). Unlike a holding stage, no batch handler ever acts on an unmanaged stage’s items either — it is a pure no-op declaration. stages/examples/backlog.yaml (name: Backlog, unmanaged: true) is the shipped default; the legacy "Backlog" compat net for installs that predate this file is one predicate, stages.IsParkingColumn (a configured stage decides by its own unmanaged flag; a column with no stage is a parking column only when it is exactly Backlog), shared by checkStageColumnAlignment (engine/startup.go), resolveSpecifyOptionID (engine/spawn.go) and the fabrik_promote guard (parkingRefusal, §7.15). A configured Backlog stage that is not unmanaged keeps its configured meaning. The attention classifier is the one site that does not use it: a legacy Backlog item still reports idle with code no-stage — see USER_GUIDE.md §Startup Board Validation.
unmanaged: true is not rejected at load time when combined with prompt/skill/cleanup_worktree/holding_stage (mirroring the codebase’s existing lack of mutual-exclusion validation between cleanup_worktree and holding_stage themselves), but the resulting configuration is safe, not just permissively invalid: every order-sensitive resolver that could otherwise pick an Unmanaged stage as “the” target for something, or hand it bookkeeping it never earned, skips it. cleanupStage/holdingStage (engine/stages.go) never return a stage with Unmanaged: true, even if it has the lowest Order or is HoldingStage: true — falling back to nil (same as “no such stage configured”) if no eligible non-unmanaged stage remains. NextStage (stages/stages.go) walks past any Unmanaged stage when computing the next stage in the pipeline, so an operator who accidentally gives a parking column (e.g. On Hold) an Order between two dispatched stages does not strand items there — advancement continues to the next real stage instead. The gate-checked completion-label fill loop in runValidatePRTerminalAdvance (engine/pr_terminal_advance.go) stops at the first CleanupWorktree stage that is not also Unmanaged, so a misconfigured unmanaged + cleanup_worktree stage at a low Order cannot short-circuit the fill before any real gate-checked stage is considered. And settleNoWorkNeeded’s skip-labeling loop (engine/no_work_needed_settle.go) both excludes Unmanaged stages from its doneOrder boundary computation and never adds a skip label or “skipped” comment for an Unmanaged stage it steps over — a parking column an operator placed mid-pipeline gets none of the bookkeeping a real intermediate stage would.
Done (Cleanup Stage)
| Sub-State | Labels Present | Description |
|---|---|---|
| Pending Cleanup | (none) | Worktree exists; engine will remove it |
| Complete | stage:Done:complete |
Worktree removed; terminal state |
| Paused | fabrik:paused |
Manually paused; cleanup skipped |
| Awaiting Landing Verification | fabrik:awaiting-landing-verification (+ fabrik:credited-pr:<N> for a merge-train landing) |
Just reached Done via a merge-attributable path; the post-Done backstop is confirming the credited PR actually merged. See §6.19. |
Queued (Holding Stage — merge_train: on only)
This stage is a no-op for per-item dispatch. It is the universal landing rendezvous: a yolo (non-cruise) Validate completion under
merge_train: onadvances here via the dedicatedattemptMergeOnValidate→advanceToQueuedenqueue, and theQueuedhandler then picks the landing engine per repo (ADR-059 D6 — see Engine Selection below).
Reachable only via dedicated code (issue #1072).
stages.NextStageskips anyHoldingStageexactly as it already skipsUnmanaged— a holding stage is never returned as “the next stage” by generic positional advancement. The only call site that intentionally lands an item here isadvanceToQueued, gated onyoloActive && !cruise && merge_train: on(engine/stages.go). Every other terminal-advance path —runValidatePRTerminalAdvance(cruise, human-merge, or any other gate-label convergence) andadvanceConvergedPRToDone(non-train yolo convergence) — resolvesNextStagefrom Validate and now lands directly on Done, never on Queued, regardless ofmerge_trainstate. Before this fix,NextStage’s purely positional walk landed every such path in Queued whenever the column was configured, stranding items with no active drain (ADR-1072).
| Sub-State | Labels Present | Description |
|---|---|---|
| Waiting | stage:Validate:complete |
Item moved to Queued column; advanceToQueued added stage:Validate:complete atomically. Awaiting handleMergeTrainBatch to form a landing batch. |
| Landing | stage:Validate:complete |
Batch owned by a persisted merge-train run (§”Merge-Train Run State”, #2051): trial branch being assembled, a trial waiting on CI (no goroutine held — evaluated per poll), red-batch bisection, one-at-a-time fallback, or the landing integration PR open/merge in progress. |
| Done | (none — advanced by advanceToNextStage) |
Integration PR merged; member advanced to Done column via advanceToNextStage; member PR closed with batch reference comment. |
itemMayNeedWork and itemNeedsWork both return false for any stage with holding_stage: true, so these items are never dispatched to a per-item worker. processItem also short-circuits on stage.HoldingStage as a defense-in-depth guard. The catch-up loop skips holding stages for the same reason. A closed item that nonetheless ends up stranded in Queued (e.g. a human gh pr merge outside the train) is rescued by the settle-scan backstop described in §6.11.
The same holding-stage exclusion that keeps dispatch off Queued items also blacks out PR feedback processing while they sit there — review-reinvoke and comment processing never run on a Queued item either, since both live behind the identical HoldingStage admission gate. §6.16 describes the dedicated settle scan (settleQueuedReviewFindings, ADR-1208) that detects an unresolved review-thread finding (and, since #1863/ADR-1863, an unprocessed human comment) arriving on a Queued member and ejects it back onto a stage the ordinary reinvoke/comment path can reach, rather than widening this section’s dispatch exclusion itself.
Engine Selection: 058 Native Queue vs 059 Internal Train (ADR-059 D6)
Queued is one board column served by two landing engines. The single convergence owner handleMergeTrainBatch (ADR-056 — there is no parallel scanner) picks the engine per repo each poll, because isMergeQueueEnabled is a per-PR/per-repo signal and the Queued column may hold items from several repos at once.
Selection algorithm. handleMergeTrainBatch groups the current Queued snapshot by owner/repo and resolved base branch (groupQueuedByRepoAndBase, #1648/ADR-1648 — superseding the repo-only groupQueuedByRepo this function used before), preserving first-seen repo/partition order for the groups themselves. Within each (repo, base) partition, member order is a deterministic function of the Queued set, not of whatever order the board-state source handed items back in (#1833/ADR-1833): items are sorted ascending by (StatusEnteredAt, Number) — the member that has waited longest to enter Queued goes first, issue number breaking any tie — via sortQueuedGroupsByEntry, applied after partitioning so ordering never crosses a partition boundary. StatusEnteredAt is a record-on-write timestamp (itemstate.ItemState.StatusEnteredAt, sibling of LabelAppliedAt/ADR-1314) stamped the instant a Status mutation is applied, threaded into gh.ProjectItem only by the in-memory-cache path (boardcache.CacheImpl’s snapshotToProjectItem); the direct-GraphQL GitHubAdapter path never populates it, since no cheap durable “entered this Status” signal exists via REST/GraphQL for Project v2 items — every item there ties on the zero value and the sort degenerates to ascending Number order, still fully deterministic, just not wait-time-ordered. Before this fix, the in-memory cache’s underlying map iteration (itemstate.Store.All()) was genuinely randomized per poll, so a Queued set exceeding max_batch_size could select a different arbitrary subset every cycle, abandoning the previous cycle’s in-flight trial CI each time. baseBranchForItem is called only for items that actually carry a base: label — an item with no label buckets under the defaultPartitionBase sentinel without ever touching git, so partitioning costs nothing for the overwhelming common case. Resolving a label needs the repo’s in-memory WorktreeManager, which only ensureRepoReady registers (on a stage dispatch); after a restart a repo whose only open items are Queued would have none, and its base:-labelled members were skipped every poll for ever. So handleMergeTrainBatch first calls ensureQueuedBaseReposReady (#2250), which runs ensureRepoReady — clone, or a clone-failure pause, exactly as on a stage dispatch — once per poll for each repo with a base:-labelled Queued member and no manager; a repo with only unlabelled members is never touched. The “WorktreeManager not yet registered” exclusion remains as the fail-closed fallback for a failed or unfinished clone. A third fail-closed guard runs before this, ahead of the base-label check (#1772/ADR-1772): an item whose board-cache entry has never been deep-fetched (CacheImpl.IsItemDeepFetched, checked per (repo, number) — never a global “cache is bootstrapped” flag) carries an empty Labels slice indistinguishable from a genuinely label-free item; rather than risk silently pinning it to defaultPartitionBase, it is excluded from batching for that poll (logged under the merge-train tag naming the item and reason) and left in Queued, picked up warm on a later poll once hydrated. This check fails open when no CacheImpl is wired, since a live-read engine has no unhydrated state to guard against. routeQueuedGroup then runs once per (repo, base) partition, routing its items by the poll-native LinkedPRIsMergeQueueEnabled signal (populated by the GraphQL board query — never a REST pulls fetch, which always reports false for the queue flag, and never a webhook payload):
- Queue-enabled repo (
MergeQueue != "off" && LinkedPRIsMergeQueueEnabled) → the ADR-058 enqueue path, invoked per item viaenqueueForQueue(the same helperattemptMergeOnValidateuses on themerge_train: offpath). It callsEnqueuePullRequestwith the poll-nativeLinkedPRHeadSHAas the expected-OID guard and appliesfabrik:auto-merge-enabled(idempotency + convergence anchor).checkAutoMergeConvergencethen drains each enqueued itemQueued → Done(it claims anyfabrik:auto-merge-enableditem regardless of its column). Two guards: an item already carryingfabrik:auto-merge-enabledis mid-convergence and is not re-enqueued; an item whoseLinkedPRNumber/LinkedPRHeadSHAis empty (cache miss) is skipped this poll and retried next (no REST fetch). - Non-queue repo → the ADR-059 internal merge train: the partition’s remaining items form one batch, capped independently per partition (
max_batch_size). The surviving items are dispatched to a singledispatchMergeTrainWorker, keyed on the compositemergeTrainKey(owner/repo, base)— not bareowner/repo— viamergeTrainInFlight, so a repo with members on two different bases runs two fully independent trains simultaneously, neither blocking, cancelling, or alerting on the other’s guard state (ADR-1648, superseding ADR-1647’s exclusion ofbase:<branch>members from batching entirely). Each partition’s worker lands its own membersQueued → DonevialandMergeTrainBatch, forking from and targeting its own resolved base — abase:maint/1.xbatch lands ontomaint/1.x, never touchingmain.
Batch-snapshot logging is deduped by composition (#1151), per partition. routeQueuedGroup runs once per poll cycle for as long as any Queued items exist for a partition — including the full CI-wait duration of an in-flight train, which can span many poll cycles. Before dispatching, it computes an order-independent signature of the internal-train subset (mergeTrainBatchSignature: sorted item numbers, comma-joined) and compares it against Engine.mergeTrainBatchSnapshotSeen (a sync.Map keyed on the same composite mergeTrainKey). The batch snapshot for <trainKey>: N item(s) — ... line is only logged, and the stored signature updated, when the signature differs from the last-logged one for that partition — a membership or count change re-logs; a pure reordering of the same members (GraphQL item order is not guaranteed stable poll-to-poll) does not. Dispatch itself (dispatchMergeTrainWorker) always runs regardless of whether the log fired.
Both engines drain the same Queued column and advance their members to Done on land; only who batches differs (GitHub’s queue vs Fabrik’s trial branch). max_batch_size caps the internal-train subset per (repo, base) partition, not the repo’s combined total across partitions — so a large repo A never starves repo B (the pre-D6 batch[0]-anchored latent multi-repo bug, hardened by grouping), and one of repo A’s own bases never starves a sibling base in the same repo either.
Precedence (FR-3). For any item reaching a landing decision:
- Native merge queue present on the repo (
MergeQueue != "off" && LinkedPRIsMergeQueueEnabled) → 058 enqueue path, regardless ofmerge_train. The queue always wins: a direct or trial-branch merge on a queue-required branch returns HTTP 405. Undermerge_train: offthis fires inline inattemptMergeOnValidate; undermerge_train: onit fires from theQueuedhandler. - Else
merge_train: on→ the internal train (from theQueuedhandler). - Else → legacy per-PR serial auto-merge (
attemptMergeOnValidate’s native-auto-merge / direct-merge fallback), unchanged from pre-merge-train behavior.
Merge-Train Landing Lifecycle (ADR-059 D3)
When handleMergeTrainBatch has a non-empty batch of Queued items, it calls dispatchMergeTrainWorker, which starts a short goroutine running runMergeTrainWorker. Since #2051 (ADR 2051) that goroutine no longer holds the episode: it prepares the batch, assembles and opens the trial, records the open trial as persisted run state and exits. Every later step — evaluating the trial’s CI, bisecting, the one-at-a-time fallback, landing — is a step of the run state machine (§”Merge-Train Run State, Per-Poll Evaluation and Restart Resume” below), run by a short step goroutine only when there is something to do. The steps own the assembly-to-landing sequence:
Two-PR model: The goroutine creates two PRs: (1) a draft CI PR (chore(merge-train): trial integration for #N…) that triggers checks on the trial branch immediately, and (2) a landing integration PR ([merge-train] batch: #N1, #N2, …) opened only after trial CI passes. The draft CI PR is closed implicitly when the trial branch is deleted at cleanup.
Trial assembly — merge conflicts (issue #1235): assembleTrialBranch merges each batch member’s head SHA into the trial worktree in sequence (git merge --no-ff --no-edit). A clean merge just continues to the next member. On conflict, it calls resolveTrainConflict, which first lists the conflicted paths (unmergedPaths, parsing git status --porcelain for the UU/AA/DD/AU/UD/UA/DU codes, returning each path paired with its two-letter status code as a conflictedPath) and classifies them against a single declared generated-path → regeneration-command mapping (generatedFiles in engine/generated_files.go; today’s only entry is docs/llms-full.txt → bash scripts/generate-llms-full.sh). If unmergedPaths itself errors (e.g. a transient git status failure), resolveTrainConflict cannot classify anything and falls back to dispatching Claude for the full, unscoped conflict exactly as it would pre-#1235 — a deliberate, narrow tradeoff: a git-level failure says nothing about whether the conflict is confined to a generated path, so failing open to the safe pre-existing behavior is preferable to guessing (TestResolveTrainConflict_UnmergedPathsErrorFallsBackToPlainClaude). Otherwise, three outcomes:
- No generated paths involved: unchanged pre-#1235 behavior —
resolveConflictWithClaudedispatches Claude with a synthetic comment instructing it to resolve every conflict marker and commit. - Confined entirely to declared generated path(s): Claude is never invoked.
regenerateAndCommitruns each distinct regeneration command once (deduplicated by command, not by path, so two paths sharing one command don’t re-run it), bounded by a fixedregenerationCommandTimeout(5 minutes) viaexec.CommandContext— unlike Claude dispatch, which isctx-aware throughout, a hung regeneration command has no other circuit breaker, and a timed-out command fails the member closed (ejected with a diagnosable reason) rather than blocking the worker indefinitely. Staging follows the same scope as execution, not just the conflicted subset: for every command it runs, it stages every declared path (from the fullgeneratedFilesmapping, not only the conflicted ones passed in) tied to that command, since running a shared command once regenerates all of its declared outputs as a side effect. A non-conflicted sibling path left unstaged here would surface as a dirty working-tree change in the next member’s merge. It then verifies the tree is fully resolved before committing. - Mixed (a declared generated path conflicts alongside a normal path in the same member): Claude is dispatched for the non-generated part only — the synthetic comment names the generated path(s) as out of scope and instructs Claude to stage only what it resolved and not commit.
regenerateAndCommitalways runs after Claude returns, never before: a co-conflicted non-generated file can itself be one of the generator’s own source inputs (e.g.docs/state-machine.md, one ofgenerate-llms-full.sh’s fourORDEREDpages), so regenerating first would read stale/conflicted content.regenerateAndCommitperforms the single commit that finalizes both parts once its own tree-widegit diff --checkand unmerged-path checks pass.
classifyConflictedPaths treats a declared generated path as eligible for regeneration only when its conflict status carries no deletion intent. A status of DD (both sides deleted), UD (deleted by them), or DU (deleted by us) — deletionInvolvingStatus — routes the path into the non-generated bucket instead, so it reaches Claude like any other conflict rather than being silently regenerated: a contributor who deleted a declared generated file meant to remove it, and blindly rerunning the regeneration command would recreate content nobody wants (TestMergeTrainWorker_DeletionConflictOnGeneratedPathRoutesToClaude). AA/UU/AU/UA carry no deletion intent and remain eligible for regeneration as before. classifyConflictedPaths also returns this deletion-excluded subset separately (deletionExcluded, alongside matched and nonGenerated) precisely so the mixed-case call into regenerateAndCommit can pass it through as protectedPaths.
protectedPaths closes a further interaction between the two mechanisms above: if a matched path and a deletion-excluded sibling declare the same regeneration command, executing that command for the matched path also regenerates the sibling on disk as a side effect — even though the sibling isn’t in specs and Claude (not regeneration) owns its resolution. regenerateAndCommit never stages such a sibling; instead it discards the command’s side effect on that one path — restoring the working-tree file from the index via git checkout-index -f when Claude staged content for it, or removing the file entirely when Claude staged its deletion (git rm) — before staging and committing the rest as usual (TestMergeTrainWorker_SharedCommandDoesNotOverwriteDeletionExcludedSibling).
A regeneration failure (non-zero exit, inability to stage, conflict markers still present afterward, or — in the mixed case — Claude having committed prematurely despite being instructed not to) ejects the member via the existing ejectMember path with a diagnosable reason describing the specific failure — it never falls back to Claude. The premature-commit case is detected structurally, not by content: regenerateAndCommit requires MERGE_HEAD to still be present at entry (it is only ever called with the merge either still fully in conflict, or with Claude having resolved-and-staged but not committed the non-generated part) and fails closed immediately if it’s already gone, before running any regeneration command. This is deliberately not a git diff --cached content comparison after the fact — that would miss the case where Claude’s premature commit happens to write byte-identical content to what regeneration would have produced. This mirrors the existing unresolvable-conflict eject path (assembleTrialBranch’s abort-and-eject branch) and reuses the same three-way (resolved, reason, err) contract style as the pre-existing usage-limit sentinel handling described under ADR-1120 below: a non-nil err (the ADR-1120 usage-limit sentinel) is still a fatal assembly abort, never an ejection, in every one of these three outcomes.
resolveConflictWithClaude also guards against a false “resolved” verdict: buildTrainConflictComment’s fallback instructions tell Claude to run git merge --abort when it judges a conflict genuinely unresolvable, and an abort clears every conflict marker (and MERGE_HEAD) exactly as a real resolution would — unmergedPaths alone can’t tell the two apart. assembleTrialBranch captures the trial worktree’s HEAD SHA immediately before attempting each member’s merge (preMergeHEAD) and threads it through resolveTrainConflict into resolveConflictWithClaude; after confirming no non-generated conflict markers remain, it additionally requires either MERGE_HEAD still present (merge legitimately in progress) or HEAD having moved past preMergeHEAD (a real commit happened). A MERGE_HEAD-less worktree still sitting on preMergeHEAD is treated as unresolved — an abort, not a resolution — so the member is ejected instead of silently landing with its whole contribution missing. This applies to both the plain and mixed conflict paths.
A turn-limit exit is not automatically “unresolvable” (#1841). buildTrainConflictComment’s instructions no longer tell Claude to run the target project’s own build/test/lint/install commands — that unconditional, Go-specific step (go build ./..., go vet ./..., hardcoded regardless of the target repo’s actual language/toolchain) was identified as the dominant turn sink for small conflicts (a live reproduction: the same single-file conflict took 10 turns with the instruction present, including exploratory detours — checking for go.mod, substituting the project’s real npm scripts, reviewing unrelated git history — versus 5 turns once it was replaced with an explicit “do not verify, do not explore outside the conflicted path(s)” instruction; see adrs/1841-*.md for the full evidence). Verification is left entirely to the trial’s own CI, which runs unconditionally after assembleTrialBranch regardless of how the conflict was resolved (assembleAndValidateInner → pollTrainCI) — removing the instruction does not remove the safety net, only a redundant, expensive one. Even with a shrunk turn sink, a conflict-resolution invocation can still exhaust its configured turn budget (commentMaxTurns(holdingStg), default 50, doubled by fabrik:extend-turns); before #1841, resolveConflictWithClaude treated any non-usage-limit error — including this turn-cap exit (claudeTurnLimitError, CLI subtype error_max_turns) — as an immediate, unconditional “unresolvable,” returning before the worktree-inspection logic above ever ran. Since a turn-cap exit still means the dispatch reached and ran Claude (the same “Claude is healthy” signal a clean success carries — the per-invocation analogue of the account-wide ADR-1120 principle below), resolveConflictWithClaude now runs the identical inspection (extracted into finalizeConflictResolution, shared by both the clean-exit and turn-limited-exit paths) before concluding anything: if the worktree shows no remaining conflict markers, the resolution is committed and the member stays in the batch exactly as a clean success would, rather than discarding real, completed work purely because the invocation ran out of turns on unrelated work after finishing. The account-wide Claude-usage-limit hold-clearing call also fires on this path, unchanged from the clean-exit case.
Ejection comments now say why, not just that (#1841). When a conflict genuinely remains unresolved — whether from a clean-but-incomplete exit or a turn-limited one — resolveConflictWithClaude/finalizeConflictResolution return a conflictEjectionDiagnostic (TurnLimited, NumTurns, RemainingPaths, Reason) threaded as a plain return value — never shared/mutable state, mirroring ADR-1420’s diagnostic-threading precedent below, but as its own small type rather than layered onto the CI-specific trainCIDiagnostic — through resolveTrainConflict to assembleTrialBranch’s ejection call site. buildConflictEjectionReason renders it into ejectMember’s plain-string reason: naming the still-conflicted file(s) (file-level only, from unmergedPaths’ git status --porcelain granularity — not hunk-level, a documented non-goal) and distinguishing “the invocation ran out of turns rather than judging the conflict unresolvable” from “conflict judged unresolvable” wording, so an operator reading the ejection comment knows immediately whether the invocation ran out of budget or a genuine judgment call. This also improves the pre-existing, non-turn-limited ejection path, which previously always reported a generic “unresolvable conflict” message with no file names at all. A nil diagnostic (the unmergedPaths-error fallback path inside resolveTrainConflict, which has no original conflicted-path list to report) still falls back to that same pre-#1841 generic message.
A clean unmergedPaths result doesn’t prove marker-free content (#1841 review hardening). git status --porcelain clears a path’s UU status the moment it’s git added, regardless of whether the staged content still contains literal conflict-marker text — so by the time finalizeConflictResolution’s plain-case git diff --check (no --cached) runs, every originally conflicted path has necessarily already been staged, making the working tree and the index identical and the check a structural no-op. pathsStillContainConflictMarkers closes this by reading each of originalNonGeneratedPaths’ current on-disk content directly and checking for conflict-marker text — immune to whether the content has since been staged or even committed (the plain case’s own prompt instructs Claude to commit itself as its last step, which git diff --cached --check alone would also miss). Checked once, right after the abort-disambiguation block, covering both the plain and mixed conflict paths ahead of the mixed case’s early return. conflictMarkerBlockRegex went through two rounds of narrowing after review: first, from matching any line starting with 7+ of <, =, >, or | down to just the two boundary forms <<<<<<</>>>>>>> (a bare =======/||||||| line is common legitimate content — a Markdown setext heading underline, an RST section underline, an ASCII banner — and matching it alone would eject a correctly resolved file); then, since a lone boundary line is itself plausible legitimate content in a repo whose purpose involves documenting or testing conflict-marker syntax (a doc example, a test fixture, quoted diff/patch text), the regex now requires the full three-part structure in order — a <<<<<<< line, then later a bare ======= line, then later still a >>>>>>> line — matching git’s own conflict-marker shape exactly. This still doesn’t rule out every false positive (a file documenting a complete worked conflict example would still match), but real content essentially never happens to contain all three markers in the correct order, so detection coverage of a genuine unresolved conflict is unaffected. A fourth pass fixed a false negative: the separator span was anchored as ^={7}\n, an exact zero-width transition from the seventh = to a line feed, which never matches a CRLF-terminated =======\r\n line (common in Windows-origin repos or any repo with .gitattributes eol=crlf) — a genuinely unresolved, CRLF-terminated conflict block went undetected and would have been committed and passed to the trial’s own CI instead of being caught here. The anchor is now ^={7}\r?\n; the boundary spans didn’t need the same fix, since [^\n]* and . already absorb a preceding \r as ordinary line content.
assembleTrialBranch’s own eject-on-unresolvable cleanup reuses that same captured preMergeHEAD: since resolution can now fail after a commit already landed on wtDir (e.g. regenerateAndCommit’s premature-commit guard tripping), git merge --abort alone is not sufficient cleanup — with MERGE_HEAD already gone, it is a silent no-op, and the bad commit would remain as wtDir’s HEAD, contaminating every subsequent member’s merge attempt and the pushed trial branch. The eject path therefore unconditionally follows the best-effort abort with git reset --hard <preMergeHEAD>, which is a no-op when the abort already fully reverted things and the authoritative cleanup when it didn’t. git reset --hard only rewinds tracked content, so the eject path also unconditionally runs git clean -fd afterward to remove any untracked files a failed resolution attempt (Claude, or a regeneration command) may have left behind — without it, a stray untracked file would make the next member’s git merge fail with git’s own “untracked working tree file would be overwritten by merge” error, which resolveTrainConflict would misclassify as a plain conflict (no MERGE_HEAD, no unmerged paths) and dispatch Claude against a worktree with nothing to resolve.
Landing sequence (FR-1 through FR-5):
Pre-FR-1 — base sanity check (#1773). Immediately before FR-1’s if integrationPR != nil {
reuse } else { create } split — covering both the open and reuse branches uniformly —
landMergeTrainBatch (and, at its own equivalent CreatePR call, landSingleton) calls
refuseIfBaseContradictsMembers. On the default-base partition only, it reads each member’s
base:<branch> label live (never the cached snapshot) and refuses to open or reuse the PR if
any member’s live label declares a different base, or if the live read itself errors — leaving
members in Queued rather than landing against a base that contradicts what they declare. See §6.24
for the full mechanism; this is a no-op on any non-default-base partition and on a healthy
default-base batch.
-
FR-1 — Open integration PR:
landMergeTrainBatchcallsCreatePR(non-draft) with title[merge-train] batch: #N1, #N2, …and a body containing the idempotency marker<!-- fabrik-merge-train-batch -->. The body lists all batch members (issue number + title) and aCloses #Nline per member. The member PRs are closed-not-merged (the change lands via this integration PR), so a member PR’s ownCloses #Nnever fires — the integration PR’sCloses #Nis what restores issue↔landing-PR connectivity and auto-closes each member issue when the integration PR merges into the default branch. (For a non-default base, or on auto-close lag, FR-3’s explicitCloseIssueis the fallback.) The sameCloses #Nlines are carried on the trial draft CI PR body from assembly, since that draft PR IS the landing PR when reused (FR-5). -
FR-2 — Poll and merge:
pollForMergeablepolls the integration PR with the same 30s interval aspollTrainCI, usingFetchPRDetails(mergeable_state and head SHA in one call) andclassifyLandingCI(ADR-1441, #1441) — the merge-train landing counterpart topollTrainCI’s own check-run-aware classification, built from the same shared primitives (gh.ClassifyCheckRuns,e.classifyRequiredContexts,describeCheckRuns,gh.MergeableStateAccepted).mergeable_state == "dirty"rejects immediately with no check-run fetch; otherwise check runs on the head SHA are fetched and classified: a confirmed failure or required-context failure rejects; a check still pending keeps polling; an all-clear (plus required contexts satisfied) lands. Only with zero check runs at all does an acceptedmergeable_state(clean/unstable,gh.MergeableStateAccepted) become load-bearing by itself — mirroringpollTrainCI’s own zero-check-runs fallback. Before ADR-1441, this step acceptedmergeable_state ∈ {"clean", "unstable"}outright with no check-run awareness at all — the merge-train landing counterpart to the single-PR advance gate’s pre-ADR-1441 shortcut, left unfixed by ADR-1153 and explicitly flagged there as a “candidate fast-follow.” OnceclassifyLandingCIjudges the PR landable,MergePRis called (no admin bypass).MergePRitself independently re-checksmergeable_stateagainst theclean/unstableallowlist before merging (see “MergePR’s own CI precondition (ADR-933)” in §5, after §5.3) — normally a no-op here sincepollForMergeablealready judged the PR acceptable, but it closes a narrow TOCTOU window if the state flips between the two checks; this merge-side check is deliberately unchanged by ADR-1441 (its R3 decision: the merge path continues to defer to branch protection alone, per ADR-072’s operator note). On timeout, a warning comment is posted on the first batch member and the function returns without advancing members (they remain in Queued for the next train cycle). On merge API failure — includinggh.ErrNotMergeableCIfrom that TOCTOU window — an error comment is posted and landing aborts, leaving members in Queued to retry on the next train cycle; this is not escalated to a pause and does not touchfabrik:rebase-needed. -
FR-3 — Member lifecycle: For each surviving batch member: (a)
advanceToNextStagemoves the item from Queued → Done on the project board; (b)addLandedCommentWithRetryposts🏭 **Fabrik merge-train** — Landed via batch PR #<N>.on the member PR — this is the sole cross-landing-path, member-scoped record of which PR actually landed the change (issue #1275), so a bareAddCommentfailure would silently and permanently lose it; the shared helper retries up to 3 attempts total with exponential backoff (landedCommentRetryDelay, base 200ms, doubling), gated onisTransientError(a non-transient error like a 404/422 short-circuits to the warn immediately, without wasting the backoff window) — on exhaustion it falls back to the pre-existing warn-and-continue behavior unchanged, never blocking or delaying landing; the identical helper is used bylandSingleton’s equivalent comment post (🏭 **Fabrik merge-train** — Landed one-at-a-time via singleton PR #<N>.); (c)CloseIssuecloses the member PR explicitly; (d)CloseIssuecloses the member issue explicitly. Step (d) is the fallback that guarantees issue closure independent of the integration PR’sCloses #N(which only auto-closes on merge into the default branch) — without it a member landed via a non-default base, or before auto-close propagates, is left landed-but-open. It is idempotent (a no-op on an already-closed issue). Members whoseStatus == "Done"are silently skipped (restart safety). The same close-issue fallback runs in the one-at-a-time singleton landing path (landSingleton) — but only inlandSingletonis a failure of step (d) retried: on failure,markMergeTrainMemberCloseOutstandingwritesfabrik:awaiting-member-close, andsettleMergeTrainMemberClosesretries it every poll thereafter, escalating tofabrik:pausedafterMaxRetries(§6.10, ADR-061).landMergeTrainBatch’s own step (d) has no such retry — a deliberate, narrowly-scoped-fix decision (ADR-061 §Sibling Audit); a follow-up issue is expected to extend the same machinery to this path. -
FR-4 — Cleanup:
CleanupTrainWorktree(name, deleteBranch: true)runs in adefer, removing the trial worktree directory and deleting both the local and remote trial branch (git push origin --delete). This is the single call site responsible for local worktree removal per trial lifecycle —assembleAndValidateno longer removes the worktree itself after opening the draft CI PR (pre-#1151, it did, making every downstream cleanup call a guaranteed no-op on the success path); the trial worktree now stays checked out for the full CI-poll window and is reclaimed exactly once, here (or in the equivalent failure-pathcleanupTrialArtifactscall), regardless of outcome. All cleanup steps are best-effort (failures are logged, not propagated) — but only genuine failures are logged: agit worktree removefailing because the path is already not a working tree, or agit push --deletefailing because the remote ref already doesn’t exist (e.g. removed by GitHub’s own “delete branch on merge” behavior once the integration PR — which reuses the trial branch — merges), is classified as a no-op success and does not emit awarn:line (#1151). The remote-branch-delete push also runs with an SSHLogLevel=ERRORoverride scoped to this one invocation, suppressing the “Permanently added … to the list of known hosts” notice without weakeningStrictHostKeyChecking. -
FR-5 — Restart idempotency:
findIntegrationPRscans recent PRs viaListPRs(state=all) for this trial’s own PR —HeadRefName == trialBranch(fabrik/merge-train/<trialName>) is the sole, mandatory identity gate (#1615, R1). The shared idempotency marker<!-- fabrik-merge-train-batch -->is checked too, but only as non-fatal corroboration: a branch match is returned regardless of whether the marker is present (a warning is logged if it’s missing), and a marker match on a different branch is never returned, no matter how recently that PR was updated or how many repo-wide PRsListPRssurfaces. Before #1615, the marker alone was sufficient — since every merge-train PR in the repo carries it, in any state, a later trial’s search could return an unrelated, already-merged PR from a completely different batch, andlandMergeTrainBatchwould trust that PR’sMergedfield (alreadyMerged) to skip its own merge step and proceed straight to member advancement, closing the member PR and issue as though the batch had actually landed (reported in #1614). Once branch identity gates the match, the returned PR’sState/Mergedare inspected to decide what happens next: open — FR-1 is skipped, FR-2 proceeds as normal (poll, then merge); merged — FR-2 is also skipped (alreadyMerged) and FR-3 proceeds directly, now provably safe since identity is confirmed; closed and unmerged — a failed trial, not a reusable integration PR (R2) — routes to the escalation path below instead of FR-3, never treated as “already landed” (R5). Member advancement is additionally guarded, per member, by two checks:Status == "Done"(restart safety, unchanged) and, new in #1615, whether the member’s issue number actually appears in the integration PR’s body (parseTrainMembers) — a batch that dropped a member during assembly must not be able to claim it (R4). A member failing either the closed-unmerged trial check or the per-member membership check is escalated (below) rather than silently skipped or, worse, advanced.
Landing-failure escalation (#1615, R4/R5). When findIntegrationPR returns this trial’s own PR closed and unmerged, escalateClosedUnmergedTrial escalates every survivor; when the matched (merged) integration PR’s body doesn’t list a specific member, that one member alone is escalated in place inside the FR-3 loop, leaving the rest of the batch to land normally. Both call the same escalateStrandedTrainMember, mirroring ejectRedSingleton’s reroute+comment+pause convention (§ “Merge-Train Red-Batch Bisection” below) rather than ejectMember’s counter/stay-in-Queued semantics — this is an infra-level landing failure (the integration PR itself, or the batch’s own composition), not a retryable member-level defect: rerouteQueuedMemberOffHolding moves the member off Queued to stageBeforeHolding first (a failed reroute leaves the member untouched for the next poll to retry, exactly as ejectRedSingleton’s identical failure path); on success, an explanatory 🏭 **Fabrik merge-train — landing failed** comment is posted (naming the closed PR, or the merged PR that dropped the member) and pauseMergeTrainMember applies fabrik:paused + fabrik:awaiting-input. The reentry guidance (fabrik:revalidate when the reroute target is literally named "Validate", or naming the real blocking stage:<target>:complete label otherwise) is the same reentryInstruction helper ejectRedSingleton uses, so both escalation paths give identical, correct instructions for the same reroute target. No “landed via” comment is ever posted for an escalated member (R5/AC4), and R6 — that comment may only be emitted after the batch PR is observed MERGED — holds structurally once R1–R4 do: the FR-3 loop only ever reaches the comment for a member that passed both the branch-identity and membership checks on a PR whose Merged field is true.
pollForMergeable’s own polling loop never inspects pr.State — if this trial’s PR is open when findIntegrationPR returns it but closes (unmerged) mid-poll, it is a narrower, deliberately out-of-scope variant of the same defect: the loop simply times out (unchanged pre-#1615 behavior, members remain in Queued for a fresh retry) rather than routing to the escalation path above, since it can’t misattribute a landing the way the fixed bug could.
State threading: board.ProjectID is threaded from handleMergeTrainBatch → dispatchMergeTrainWorker(ctx, batch, projectID) → mergeTrainWorkerState.projectID (immutable after dispatch, no mutex needed). Inside landMergeTrainBatch, a minimal &gh.ProjectBoard{ProjectID: projectID} is constructed for the advanceToNextStage call.
mergeTrainInFlight lifecycle: set at dispatchMergeTrainWorker entry (LoadOrStore), keyed on the composite mergeTrainKey(owner/repo, base) — not bare owner/repo — since #1648, so two partitions of the same repo each get their own independent claim/clear lifecycle. Cleared by finishTrain — the single ADR-067-mandated clear point, called from prepareTrainWorker’s own-failure defer and from the run’s finish() when the episode ends (landed, dissolved, ejected, abandoned). Since #2051 the claim lives as long as the RUN, not as long as a goroutine (ADR 2051 amends ADR-067): while a trial waits on CI no goroutine exists but the claim persists, so the duplicate-launch guard, ADR-1208’s pending-eject routing (mergeTrainBatchMembers) and the invalidateConflictingQueued gate behave exactly as they did with a live worker. The Store liveness registry (Store.ExitRepoWorker, same composite key, §9.2, issue #1222) now means “a step goroutine is executing” — assembly, a Claude conflict resolution, a landing: it is set when a step is launched and cleared by leaveStep when the step returns without finishing the run (the claim stays). So an open trial that is merely waiting on CI does not block the auto-upgrade idle guard (a restart resumes it), while a step in progress still does. mergeTrainWorkerActiveForRepo (closed_item_advance_settle.go) answers “live” for either a Store marker or an open run. mergeTrainInFlight is the atomic duplicate-launch claim plus the assembling/bisecting/CIResult/prNum/trialName sub-state; the Store registry is the liveness answer the auto-upgrade idle guard reads (repo-wide, via Store.RepoWorkerActiveForAnyBase’s prefix scan).
Merge-Train Red-Batch Bisection (ADR-059 D4)
A batch is usually green: every member already passed Validate individually, so a batch fails only on a genuine cross-PR semantic conflict. runMergeTrainWorker is therefore structured as a re-form loop around a single combined validation per (re-formed) batch:
- Green path is a hard invariant (D-d): a green combined Validate costs exactly one validation and performs zero bisection — it lands immediately via
landMergeTrainBatch. Bisection is strictly gated behind a red result. max_batch_sizecap (FR-4):handleMergeTrainBatchcaps the Queued snapshot to the firstMaxBatchSizeitems (default 5) viacapBatch, logging any truncation.capBatchitself performs no ordering — it truncates whatever order it receives, which since #1833/ADR-1833 is alwaysgroupQueuedByRepoAndBase’s deterministic(StatusEnteredAt, Number)sort (see the “Selection algorithm” paragraph above), so “firstMaxBatchSize” means the members that have waited longest to enter Queued. This bounds the worst-case bisection depth.
Bisection on red (FR-1): when the combined Validate is red, the run enters its bisection state (bisectState, engine/train_bisect.go, #2051) with a per-episode cost budget (Used = 1 for the initial red validation, Cap = effectiveBisectCap() = MaxBisectValidations, default 2·⌈log₂(max_batch_size)⌉ + 1 ≈ 7, read once when the episode starts and persisted). The recursion of the old bisect was a tail call, so its entire state is five values — Origin (the batch the episode started from), Red (the set currently known red), NextHalf (0, 1, or 2 = both halves green), Used, Cap — plus the red set’s diagnostic. next() reproduces the halving exactly: a single member in Red is the poisoner; otherwise mid := len(Red)/2, half A is Red[:mid] then half B Red[mid:], the cost cap is checked before each half, a red half with survivors becomes the new Red (survivors, not the half — members ejected during assembly are not in it) and restarts at its first half, and both halves green is the non-isolable-interaction fallback. Used is spent after each half trial and before the cancel / assembly-error / runaway / infra checks. A property test pins the state machine to a verbatim copy of the old recursion. Every trial (initial and every sub-trial) is assembled by assembleAndValidate off the same base SHA pinned once at batch start (D-b, via EnsureTrainWorktreeAt), so redness is attributable to member composition, not a moving base branch (main-moved is D5, out of scope). Each sub-trial’s branch + worktree + draft CI PR is reaped immediately by cleanupTrialArtifacts — but since #1835, assembleTrialBranch’s assembly step underlying every one of these assembleAndValidate calls first checks whether the trial’s member list starts with a previously-recorded prefix from earlier in this same worker invocation, forking from that prefix’s already-built commit instead of the pinned base SHA whenever one matches — bisect’s first half is one such prefix by construction, needing zero merge work. See §6.27.
Red singleton short-circuit (#1440, reroute added #1545, auto-repair added #2045): (Since #2045 the reroute-and-pause described below is the fallback — cap reached, max_train_auto_repair_attempts: 0, or a reroute target not literally named Validate; by default ejectRedSingleton first runs the bounded auto-repair of §6.31.) a red batch of exactly one member is never handed to handleRedBatch at all — runMergeTrainWorker’s TrainCIRed branch checks len(survivors) == 1 first and, when true, calls ejectRedSingleton directly. Bisection exists to isolate a poisoner among two or more members; a batch of one has nothing to isolate, so routing it through bisect (whose own len(red) == 1 base case is a pure pass-through) would only reach the same poisoner-isolation ejection wording — “isolated by halving bisection,” “retried in a future train with a different composition” — that is actively misleading for a batch that never had more than one member to begin with. ejectRedSingleton:
- Reroutes off Queued before doing anything else (#1545 R1/R2): calls
rerouteQueuedMemberOffHolding— the same primitive and reroute-before-side-effects ordering §6.16 established for the structurally identical review-findings cause — to move the member’s board Status from Queued tostageBeforeHolding(normally Validate) before posting a comment or pausing. Before this fix the member was paused in place inside Queued, aHoldingStagecolumnitemMayNeedWork/processItem/settleQueuedReviewFindingsall structurally refuse to touch — the pause was permanently unreachable by any stage, and the “fix it, then removefabrik:paused” instruction had an unstated precondition that only a human moving the board card by hand could satisfy. If the reroute fails, nothing is posted and the member is not paused — it looks like nothing happened, and the very next poll’s train re-forms the same singleton and re-hits this same disposition. - Posts one comment stating plainly that the PR’s own combined Validate is failing (never “ejected… isolated by halving bisection,” never a promise of a differently-composed future train, never a conflict framing), still naming the failing check(s) via the shared
renderDiagnosticBlock(#1420). The comment now names the reroute target stage and, only when that target is literally named"Validate", instructs applyingfabrik:revalidate— not a barefabrik:pausedremoval (#1545 R4):stage:Validate:completeis already set from this item’s original completion, so removing onlyfabrik:pausedwould silently no-op againstitemNeedsWork’s completed-stage check.fabrik:revalidate’s existing handler (handleRevalidateLabel) clearsstage:Validate:completealongsidefabrik:paused/fabrik:awaiting-input/etc. together, which is what actually makes Validate re-run.handleRevalidateLabelis itself hardcoded to that literal stage name, not generic overstageBeforeHolding’s (Order-derived) result — a Pruefer review finding on #1550 — so when the reroute target isn’t literally"Validate"(a config-only edge case the merge-train unit test fixture exercises via an"Implement"stage; production.fabrik/stages/*.yamlalways has Validate precede Queued, but nothing enforces it), the comment instead names the item’s real blocking labels (stage:<target>:complete,fabrik:paused) directly and explains whyfabrik:revalidatewould not help there. - Applies
fabrik:paused+fabrik:awaiting-inputimmediately after the reroute succeeds, via the sharedpauseMergeTrainMemberhelper, rather than accumulating towardejectMember’smergeTrainEjectionCounts3-strike counter — every red-singleton disposition for the same member carries identical information, so counting to 3 would only measure retries of a deterministic outcome. R3 design choice — reroute + pause, not reroute-without-pause: unlike the review-findings cause (§6.16), a standalone combined-Validate failure has no external, persistent signal for the ordinary pipeline to re-detect once rerouted — the failure was only ever observed on the synthetic combined trial branch, never on the member’s own already-green PR, and the ejection comment itself is filtered out of “new comment” detection byfindNewComments’🏭 **Fabrikprefix skip. Rerouting without pausing would therefore land the member inertly on Validate with nothing to pick it back up — a quieter stranding than the pre-#1545 bug (nofabrik:pausedsignal at all). Reroute + pause keeps the human gate the original design already relied on, now on a column where clearing it viafabrik:revalidateactually works. See ADR-1545. - Since the member leaves Queued entirely, the poison-well guard below no longer needs to do any work for it — the reroute alone removes it from every future Queued batch snapshot, whether or not the pause is later manually cleared incorrectly (a member re-admitted to Queued must complete Validate again first).
handleRedBatch and bisect are otherwise unchanged; every batch with two or more members bisects exactly as described in the rest of this section, and the arity guard above means bisection itself never sees a true singleton. landOneAtATime’s own red-singleton-ejection branch (below, FR-5) validates its member completely alone — the same true-singleton scenario, reached via a different path (bisection degrading a genuinely multi-member batch rather than the top-level arity guard) — so it also calls ejectRedSingleton rather than ejectMember, getting the identical wording/no-counter/immediate-pause disposition instead of leaving the pre-#1440 misleading wording live in a second reachable path.
Eject + re-form + re-validate (FR-2/FR-3): the isolated poisoner is ejected via ejectMember (D-a — the same shared MaxMergeTrainEjections counter that assembly-conflict ejections increment; D-c — the ejection comment is the attention marker; cap→pause reuse: after MaxMergeTrainEjections ejections the member gets fabrik:paused + fabrik:awaiting-input). The poisoner remains in the Queued column — ejectMember never moves it off the board — so a still-unpaused poisoner is retried in a future differently-composed train. Poison-well guard: once a member is fabrik:paused (or its issue is closed), groupQueuedByRepoAndBase excludes it from every subsequent batch snapshot. Without this, a poisoner that reds the combined Validate even in isolation — paused but left in Queued — would be re-snapshotted into every future batch, reding and bisecting the train indefinitely and starving clean members from ever landing. A human removing fabrik:paused after resolving the underlying conflict re-admits the member. handleRedBatch returns the surviving members; the main loop re-forms and re-validates them (a survivor batch is not assumed green — a second poisoner or interaction may remain) — since #1835 this re-form reuses the prefix through the position just before the ejected poisoner, merging only the members after it (see §6.27). Terminal states are clean: an empty survivor set completes the train with nothing to land (the existing zero-survivor path); each episode ejects ≥1 member or falls back, so the loop runs ≤ N episodes.
Ejection comment carries the combined-Validate diagnostic (#1420). Because a merge-train failure is, by construction, a failure that does not exist on any member’s own branch — it arises only from combining with a base that moved since the branch was cut — the combined-Validate run that observed it is the only place the failure output ever exists. pollTrainCI builds a trainCIDiagnostic (failing check names, output text/summary, and details/run URL for ordinary check-run failures; context names only for classic-commit-status RequiredContextsFailed failures, which carry no check-run output; a free-text Note for a dirty mergeable_state with no per-check signal at all) at the point of failure and threads it as a plain return value — never shared or mutable state — through assembleAndValidate → bisect/handleRedBatch/landOneAtATime → ejectMember/ejectRedSingleton (#1440: landOneAtATime’s red-singleton branch and the top-level arity-guard short-circuit above both terminate in ejectRedSingleton instead, for the true-singleton case), so the run that isolates the poisoner is always the diagnostic’s origin, and bisection’s later, unrelated re-validation of the reformed survivor batch cannot overwrite it. ejectMember/ejectRedSingleton render the diagnostic (bounded per-check inline excerpt, degrading to a head/tail truncation plus a details/run link when oversized, capped again at a whole-block hard limit well under GitHub’s comment size limit) into every ejection comment for that failure — the first as fully as the last, so an operator doesn’t pay for three uninformative cycles before learning there’s something to look at — alongside a sentence naming the other members riding in that train attempt (or stating explicitly that none were present, for a single-member train), so the operator knows before investigating that the fault does not exist on their own branch. The pause-after-MaxMergeTrainEjections comment names the same failing check(s)/context(s) and links the permalink of the ejection comment that carries the full diagnostic, rather than repeating “resolve the underlying conflict” with nothing to act on. Two of the three ejection call sites outside this issue’s original scope (unfetchable PR, no head SHA) still pass a nil trainCIDiagnostic, leaving their ejection comments exactly as before. The third — unresolvable merge conflict — gained its own, differently-shaped diagnostic under #1841 (conflictEjectionDiagnostic, described above), folded directly into ejectMember’s plain-string reason rather than passed as this function’s diag parameter; ejectMember’s own trainCIDiagnostic parameter is still nil for every conflict-resolution ejection call. See ADR-1420 and ADR-1841.
One-at-a-time fallback (FR-5): when bisection cannot isolate a single culprit within the cost budget — either the budget is exhausted, or both halves validate green (a non-isolable cross-PR interaction: each half green alone, the union red — D-e, no bespoke interaction-detection in v1) — handleRedBatch logs the degrade clearly (never silent) and calls landOneAtATime, which validates and lands each remaining member as its own singleton batch. A green singleton lands via landSingleton; a red singleton (fails even in isolation) is disposed of via ejectRedSingleton (own-validation-failed wording, immediate pause, no counter — see above); a pending singleton stays in Queued. This dissolves any interaction by construction (no two members co-reside). Two non-obvious correctness points:
- Per-singleton base re-pin: unlike bisection (which pins the base once, D-b),
landOneAtATimere-pins the base to the currentorigin/<base>before each singleton so a prior singleton’s land is visible to the next member’s validation — this is what actually dissolves a genuine{A,B}interaction (landingA, then validatingBagainstmain-with-A, which now goes red and ejectsBrather than letting both land and poisonmain). This is a deliberate sequential base advance, distinct from the D5 concurrent main-moved race. Under the membership-keyed test seam this git step is skipped (the stub is stateless). landSingleton, notlandMergeTrainBatch:landSingletoncreates a marker-free integration PR. ReusinglandMergeTrainBatchacross sequential singletons would make a later singleton’sfindIntegrationPRmatch an earlier singleton’s already-merged integration PR by the shared<!-- fabrik-merge-train-batch -->marker, skip the later singleton’s own merge, and advance it to Done without landing its code — a data-loss bug. This was true when written; since #1615,findIntegrationPRgates on trial-branch identity first (FR-5 above), so a marker match on a different branch — including an earlier singleton’s — is no longer possible onfindIntegrationPR’s own side either. The reasoning above remains valid belt-and-suspenders documentation (andlandSingleton’s marker-free design is unchanged), but the hazard it describes is no longer live even if a future change routed singletons throughlandMergeTrainBatch. Landing PR reuse: in the real path the singleton’s trial was validated through a draft CI PR on its own trial branch (assembleAndValidateInner), and GitHub allows one open PR per head branch, solandSingletonfirst callsfindIntegrationPR(trialBranch)and reuses that PR — marked ready if draft, merged; already merged → the landing completes without a second merge; closed-unmerged → the member stays in Queued. It callsCreatePRonly when no PR exists on the trial branch (thetrainValidateFntest seam opens none). Before this, it always calledCreatePR, which 422’d (“a pull request already exists”) on every green one-at-a-time trial, so the fallback could never land anything with real CI.
Dispatch-guard logging: state.bisecting (set for the duration of handleRedBatch) makes dispatchMergeTrainWorker log “bisecting red batch” for a re-dispatch attempt during bisection, rather than the misleading “CI red — needs attention”.
Runaway guard (ADR-059 D8): a composition-agnostic rate cap that fires when a (repo, base) partition creates ≥ MaxTrainTrialsPerWindow trial branches (default 20) with zero successful landings within a rolling TrainTrialWindowDuration window (default 60 min). The guard is distinct from the poison-well guard (which terminates the loop) and the bisect cost cap (which degrades to one-at-a-time within one goroutine run): it bounds the cross-poll-cycle burst that occurs when infra is broken and every trial fails. Since #1648, this fires per (repo, base) partition, not per repo — a maint/1.x partition’s runaway trip pauses only that partition’s own Queued members and never touches a healthy sibling main partition in the same repo.
Counter: assembleAndValidate is a thin wrapper around assembleAndValidateInner (the single site where all trial branches are created) that calls recordTrial(trainKey) only when the trial’s result is not TrainCIGreen (#1528) and not TrainCIInfra (#2052 — a trial abandoned because CI never started says nothing about the members, so it is not a trial for the guard’s purposes; see §6.29). A green result is, by construction, either the landing attempt itself or a bisection sub-trial that just proved a sub-batch clean — never a “zero successful lands” event, which is the guard’s entire premise. Before this fix, every trial counted unconditionally, including the green survivor-validation trial that confirms a bisection isolated its poisoner and is about to land — so a successful bisection could trip the guard on its own green trial, pausing the clean survivors it had just vindicated one second before they would have landed (deterministically for any 3-member poisoned batch under the default MaxTrainTrialsPerWindow=6 test configuration, since bisecting a 3-member batch down to a single poisoner takes exactly 6 raw trials). The exclusion is by-outcome, not by-origin: a red bisection sub-trial still counts, exactly as before — it represents genuine “no progress,” the same as a red top-level trial. TrainCIPending results and assembly errors also still count unconditionally, unchanged. The counter itself is a rolling window (slice of time.Time) keyed on the composite trainKey (mergeTrainKey(owner/repo, base) — #1648, was bare owner/repo), protected by mergeTrainTrialsMu. resetTrialCounter(trainKey) deletes the entry; it is called from both landMergeTrainBatch and landSingleton after a successful merge, so any train where survivors do land never accumulates toward the cap — and a landing on one partition never resets a sibling partition’s still-accumulating counter, since each has its own trainKey.
Guard fires: when isRunawayTripped(trainKey) returns true (count ≥ N within M minutes with zero resets), fireRunawayGuard is called. It:
- Logs the event at the
merge-traintag (never silent): the (repo, base) train key, trial count, window duration, and member count. - Applies
fabrik:paused+fabrik:awaiting-inputto every member passed to it, and posts an alert comment explaining the trial count seen, the window, and operator instructions — atomically per member, and idempotently per episode (ADR-1533, #1533).
Atomicity and per-episode idempotency (ADR-1533): fireRunawayGuard is called from three independent sites — twice inside runMergeTrainWorker (Hook 1, the worker goroutine) and once from routeQueuedGroup (Hook 2, the poll goroutine) — and nothing prevents Hook 1 and Hook 2 from running concurrently for the same trainKey once the shared trial counter trips: the poll loop does not check whether a worker is mid-firing. Each call site constructs its own, possibly-overlapping member slice from whatever local state it holds. Before #1533, this meant a member appearing in two racing calls’ slices could receive a duplicate alert, and — independently — a member whose AddComment call happened to fail was left fabrik:paused with no comment and no retry path, since a paused member permanently drops out of groupQueuedByRepoAndBase’s Queued snapshot on every later poll.
The fix makes the whole pause+alert sequence for one fireRunawayGuard call a single critical section, serialized by mergeTrainRunawayMu (a single, unsharded, engine-wide mutex — since #1648 this also serializes concurrent per-base workers within the same repo, not just across repos; same accepted rare-event trade-off, just a larger set of legitimate concurrent callers), so two concurrent calls can never interleave their loops. Within that section, mergeTrainRunawayAlerted (an in-memory map[string]int keyed "trainKey#N" — #1648, was "owner/repo#N" — recording the trial count at which the member was last alerted) records which members have already been alerted this episode; a call re-encountering an already-alerted member at the same or a lower count skips it entirely (no duplicate comment, no redundant label calls) — a strictly higher count means genuinely new trials ran since the last alert (only possible after an operator manually resumes a paused member without the episode having ended via a successful land), and falls through to a fresh alert rather than being skipped. resetTrialCounter — the guard’s only existing “this trip is over” signal, called on a successful land — also clears mergeTrainRunawayAlerted for the same trainKey, so the next trip starts a fresh episode without touching a sibling partition’s own alert history. The runaway-alert settle scan (runaway_alert_settle.go), working from a bare already-paused board item with no partition context, reconstructs the identical trainKey via mergeTrainKeyForItem — the sentinel for a no-label item (matching groupQueuedByRepoAndBase’s own zero-cost bucketing exactly), or a re-resolution via baseBranchForItem for a labeled one — so it can never disagree with the key fireRunawayGuard originally alerted under.
A member whose AddComment call fails is not marked alerted. Instead it is left with a new durable marker label, fabrik:awaiting-runaway-alert (the fabrik:paused application is unaffected — pausing never depends on the comment succeeding). settleRunawayGuardAlertScan (engine/runaway_alert_settle.go), a per-poll settle scan sourced directly from board.Items (not groupQueuedByRepoAndBase, which the member’s own fabrik:paused label already excludes it from), retries the alert every poll independent of any fireRunawayGuard call ever reaching that member again. After MaxRetries failed retries, escalateRunawayAlertFailure posts a fallback comment carrying the same explanation — but the marker is removed, and the member marked alerted, only if that fallback comment itself is confirmed posted; on a persistent AddComment outage that outlasts the fallback attempt too, the marker stays and the settle scan keeps retrying indefinitely, so the one remaining diagnostic signal is never erased out from under a member that still has zero delivered explanation (#1533 review, finding 1). See §6.18 and the Label Semantics Reference (§1.4) for fabrik:awaiting-runaway-alert’s full added-by/removed-by contract.
Two-hook dispatch: the guard is checked at two complementary points to cover both the active batch and any beyond-cap Queued members:
- Hook 1 (run-side,
advance): after the initial re-form trial and after a bisection half-trial or a one-at-a-time singleton trial finds the guard tripped. Pauses the active batch members (membersToItems(survivors)— the survivors of the initial trial assembly, which are all members when the guard trips during bisect before any ejection). - Hook 2 (poll-side,
routeQueuedGroup): beforedispatchMergeTrainWorkeris called. If the partition’s counter is already tripped (from a prior worker run), pauses the current partition’s Queued snapshot (the uncapped partitionitems— every Queued member in that (repo, base) partition, not just the batch-capped subset that would have been dispatched) and skips dispatch for that partition only. This handles beyond-cap members that Hook 1 could not reach. Since #1648 partitions are formed bygroupQueuedByRepoAndBasebefore this check ever runs, there is no longer a separate “exclude the excluded” step here (ADR-1647’s exception, which existed only because non-default-base members used to be excluded from batching rather than routed to their own partition, no longer applies — see ADR-1648). Queue-enabled members and batch-cap overflow are unaffected: both are still paused as before, scoped to their own partition.
Hook 1 and Hook 2 can run concurrently for the same (repo, base) partition — the “beyond-cap members cannot form their own batch while the worker goroutine is still active” argument bounds only whether a second train can form for that same partition, not whether Hook 2’s own guard check can race Hook 1’s. Correctness for the alerting side comes from mergeTrainRunawayMu and mergeTrainRunawayAlerted (above), not from an assumption that the two hooks never overlap. A sibling partition’s Hook 1/Hook 2 pair is entirely independent — its own trainKey, own counter, own critical section contention only through the shared mergeTrainRunawayMu.
Operator resume: once the runaway guard fires for a partition, all affected members have fabrik:paused and fabrik:awaiting-input applied and are therefore excluded from groupQueuedByRepoAndBase on every subsequent poll. No re-formation occurs for that partition until a human manually removes both fabrik:paused and fabrik:awaiting-input from each affected member — a sibling base partition in the same repo is entirely unaffected and continues normally. Before resuming, the operator should investigate the infra root cause (GitHub Actions billing, required-check configuration, base-branch health). The in-memory trial counter resets to zero on engine restart, but fabrik:paused labels persist on GitHub — so a restart alone does not re-enable the train; the labels must be cleared explicitly.
fireRunawayGuard is deliberately exempted from the #1545 reroute fix (R5 audit, below): unlike ejectRedSingleton’s pre-#1545 defect, a Queued member paused by the runaway guard is not stranded — groupQueuedByRepoAndBase/routeQueuedGroup re-admit an unpaused Queued member directly from board.Items on the very next poll, entirely independent of itemMayNeedWork/processItem’s HoldingStage exclusion (that exclusion only blocks per-item stage dispatch, not the batch handler’s own board read). Rerouting every paused member to Validate instead would force each one through a full, redundant Validate re-run for a cause (infra-wide: billing, a broken base branch, all required checks erroring) no code stage can fix, and would make each member re-earn its way back into Queued via a fresh stage completion instead of simply resuming the batch it had already validated into.
Pause-in-Holding-Column Audit (#1545)
Any site that applies fabrik:paused to an item while its board Status is still a HoldingStage column (Queued) has the same structural defect ejectRedSingleton had before this issue: itemMayNeedWork excludes HoldingStage items from dispatch, processItem (the comment-unpause path) is never reached, and settleQueuedReviewFindings applies the identical closed/fabrik:paused exclusion — so nothing can ever act on the pause without a human manually moving the board card. Every such site was audited:
| Site | Disposition | Reason |
|---|---|---|
ejectRedSingleton (standalone-validation-failure pause) |
Fixed; mostly superseded by auto-repair (#2045) | Rerouted off Queued via rerouteQueuedMemberOffHolding before pausing (R1/R2, above). Since #2045 the pause is the fallback only (cap reached, cap 0, non-Validate target); the default path re-enters Validate without pausing (§6.31). |
fireRunawayGuard (member-pausing path) |
Exempted | Its recovery path is already reachable: groupQueuedByRepoAndBase/routeQueuedGroup re-admit an unpaused Queued member directly from board.Items, bypassing the HoldingStage dispatch exclusion entirely (see the Runaway guard section above). Rerouting the whole paused column to Validate would force every member through a redundant re-run for a cause (persistent infra failure) no code stage can address. |
ejectMember’s cap-reached pause (MaxMergeTrainEjections) |
Exempted — out of scope | Governs the counter-driven bisection/conflict ejection ladder, explicitly excluded from this issue’s Scope. stayInQueue=true for every cause it serves, so the member — paused or not — was always meant to stay in Queued for a future differently-composed train; that design is unchanged by this issue. |
checkAutoMergeConvergence’s fabrik:paused sites (engine/merge_gate.go) |
Exempted — structurally unreachable | Its only call site, handleAutoMergeConvergence, runs from the Phase 1 catch-up loop, which unconditionally excludes HoldingStage items before it is ever reached — so it cannot fire while an item’s board Status is still Queued. An ADR-058-enqueued item that stays at Status == "Queued" is drained by settleClosedItemsToDone once closed (regardless of column), or simply sits, unpaused, in Queued until GitHub’s native merge queue merges and closes it. |
References: ADR-1545 (this issue’s R3 design record), ADR-1208: Queued Review-Finding Ejection (the reroute-before-side-effects precedent), ADR-1420: Merge-Train Ejection Diagnostics, issue #1545.
Merge-Train Run State, Per-Poll Evaluation and Restart Resume (#2051, ADR 2051)
Supersedes the worker model of ADR-059 D3/D5; amends ADR-067, ADR-1208 (claim lifetime), ADR-1222, ADR-1835, ADR-2050. Files: engine/train_run.go (record + store), engine/train_bisect.go, engine/train_step.go (state machine), engine/train_run_poll.go (per-poll evaluation, adoption, phase source), engine/train_ci_eval.go (evalTrialCI).
The run. One run is one merge-train episode of one (repo, base) partition. Its state is a trainRunRecord: train key, owner/repo, partition base, resolved base branch, pinned BaseSHA, project ID, the trial-name sequence (BaseTrialName, TrialSeq), the step (below) and when its phase began, the dispatched batch, the members still in play (number, PR, head SHA — the board item is re-attached from the live board, never stored), the main loop’s Current set, the one open trial (kind main/half/single, name, head SHA, draft CI PR, member numbers, OpenedAt, absolute Deadline, and the persisted CI memory — the #2052 startup-watch and failed-job re-run state), the bisectState, the one-at-a-time cursor (Members, Index) and a write-ahead Poisoner mark.
Steps. form (top of the re-form loop) → trial-ci (main trial open) → bisect (between half-trials) ⇄ bisect-trial (a half open) → one-at-a-time (between singletons) ⇄ one-at-a-time-trial (a singleton open) → landing. The phase exposed to readers is the spec vocabulary — trial-ci, bisect, one-at-a-time, landing — derived from the step.
One state machine, two drivers. advance(run, verdict) holds all the logic that used to sit between the blocking CI waits of runMergeTrainWorker: the singleton fast path and catch-up, runaway Hook 1, pending-eject Hook 2, green → landGreenBatch, red singleton → ejectRedSingleton, red batch → bisection, pending/infra → abandon, and the one-at-a-time loop. It runs from the current step to the next point where it must wait for a trial’s CI, opens that trial (assemble + draft PR — openTrial), persists, and returns awaiting. Two drivers run it:
- Synchronous driver —
runMergeTrainWorkerwhen the engine has no run store (everyNewWithDepsengine): loopsadvanceand obtains each verdict by blocking (trainValidateFnunder the seam, otherwisepollTrainCI). Behaviour, log lines and the ordering of every side effect are those of the pre-#2051 worker, so the existing merge-train tests run unmodified against the state machine. - Asynchronous driver (production —
New()creates the store) —runMergeTrainWorkerrunsprepareTrainWorkerandadvanceup to the first open trial, then its goroutine exits.settleTrainRuns(a per-poll settle scan, ADR-1270 pattern, run before Queued routing) takes over.
Per-poll evaluation (R2). For each live run that no goroutine is stepping, settleTrainRuns evaluates the open trial with evalTrialCI — the loop body of pollTrainCI, extracted so both share every decision (CI classification, ciSuiteHold, required contexts, the #2052 retrigger and the single failed-job re-run with its stale-failure guard). API reads only: no goroutine, no Engine.sem slot. The evaluator’s memory (trialCIState) is mirrored into the record whenever it changes, so a retrigger or re-run budget already spent survives the next poll and a restart. Once a verdict exists the scan launches a short step goroutine for advance(verdict) (liveness marker set, the run’s stepping flag held). A run positioned between trials (adopted after a restart) gets its next step the same way. pollTrainCI remains for the synchronous driver and the landing-internal waits.
Deadline (R2/FR-010). The trial’s Deadline is e.now() + ciBackstopTimeout() at open, persisted. The poll compares it with e.now() (so the sim clock drives it); the trial’s CI is always read first (a trial that finished while the daemon was down is never abandoned for age alone); on expiry with no verdict it logs trial … has no CI verdict … — treating it as timed out and feeds TrainCIPending to the existing pending handling (clean up artifacts, members stay Queued, episode abandoned, record removed) — detected within one poll of the deadline. There is no per-trial liveness signal (a trial PR’s check runs are never attributed to a board item, so LastCIProgressAt does not exist for it).
No double advance (FR-011). A run has a stepping compare-and-swap flag: a poll evaluation, a step goroutine and a concurrent poll are mutually exclusive owners of the run. The creating worker holds stepping before the run is registered, so a concurrent poll can never start a second step over a run its worker still owns. A step that ends because shutdown cancelled its context keeps the persisted record (the in-memory run is released, the file is not), so the next daemon adopts the bisection position and the spent cost budget; every other end of a run removes it.
Write-ahead. advance persists the post-verdict state before the side effect it licenses: the trial is recorded before anyone waits on it; the bisection progress (Used, NextHalf, Red) and “no open trial” before the next half is opened; the Poisoner mark before ejectMember. A verdict is a pure function of GitHub’s CI state and so is recomputed after a crash. A record with no open trial means “open the next trial”. A restart during landing drops the record: the durable reconstructTrainState routes finish the landing idempotently (ADR-1871).
Scope of “per poll”. Only trial CI waits are per-poll: the batch’s main trial, bisection half-trials and one-at-a-time singleton trials. Waits inside a landing path stay in the (short) step goroutine that runs it: landGreenBatch’s rebase revalidation, pollForMergeable in landMergeTrainBatch/landSingleton, and the ADR-2044 catch-up waitMemberCI (which needs the pinned base). Landing paths and CI classification are otherwise unchanged.
Restart adoption (R3, FR-008/FR-009). On start New() creates the store over .fabrik/state/merge-train/ (one <hash(trainKey)>.json per partition, written atomically through a temp file + rename). The first settleTrainRuns loads the directory — a file that does not parse or has an unknown version is renamed to <file>.corrupt and treated as absent — and adopts each record against the board snapshot (adoption needs it, so it is not done in Run()). All of the following must hold, else the record is discarded (logged with the reason, its trial PR closed and its branch deleted so reconstructTrainState Route 2 cannot find and land a stale trial that still carries a departed member’s commits, the file removed; a trial that was landing or is already merged is left alone for Routes 1/2 to finish) and the partition forms fresh through the unchanged reconstructTrainState routes: the step is not landing (the main trial’s green landing and each green one-at-a-time singleton set it, write-ahead, before landSingleton/landGreenBatch run); every live member is still on the board, in the holding stage, open and not fabrik:paused, with an unchanged linked-PR head SHA and a live Status (liveLandingState, ADR-1871) that is not “moved”; the trial PR (FetchPRDetails) is open at the recorded head SHA (a merged trial PR is deliberately left to Route 1); the pinned BaseSHA still resolves in the bare clone. The base having moved is not a failure — landGreenBatch’s trialBehind / rebase cycle handles it at landing. Members that assembly ejected from a trial (Ejected in the record — they remain in the pre-assembly Current/Origin sets) are exempt from the Queued requirement and are dropped from Current, so an assembly-time ejection before the restart does not cost the run. Transient failures (repo not ready, PR read error, live status read error) defer adoption to the next poll, and while a record awaits adoption dispatchMergeTrainWorker forms no train for the partition. An adopted run re-registers the claim, rebuilds the TUI episode (phase and phase start restored — noteTrainPhaseAt), and carries on from its step: a restart mid-trial evaluates the same trial PR and lands it; a restart mid-bisection opens the next pending half in the same order, with the trials already spent still counting against the cap. A poisoner marked but already off Queued is dropped from the record (its ejection completed before the restart); one still Queued is ejected once.
Phase source (R5, FR-012). Engine.TrainRunPhases() returns, per open run, {TrainKey, Phase, Step, Position, Of, Since, TrialPR, Members} read from the record — no goroutine involved. (Position/Of is the validation about to run out of the cost cap for a bisection, and the member index out of the fallback list for one-at-a-time.) The TUI row and board status line keep going through the single noteTrainPhase hook; syncRunPhase feeds it from the record on adoption.
Prefix cache (ADR-1835). The trial-assembly prefix cache now belongs to the run, not to a goroutine, so a bisection that spans several step goroutines still reuses its prefixes; a restart loses it (git replays only).
What stays in memory (and so resets on restart, unchanged): the runaway guard’s trial timestamps, ejection counts, auto-repair and catch-up caps, the overlap cache, the policy and infra-cooldown memos. After a restart the TUI episode’s outcome facts cover only the post-restart part of the run.
Not a regression to note: a pending verdict on a bisection half-trial is not red — it falls through to the next half and finally the fallback — exactly as before; the parity oracle (engine/train_bisect_parity_test.go) pins it.
Merge-Train Serialization + Main-Moved Recovery (ADR-059 D5)
The batch validates a combined batch against main once; its correctness rests on the main the batch validated against being the main it lands on. Serializing the train (one batch in flight per repo) keeps main from moving under a validating batch; the only remaining mover is an external direct push by a human, which is detected and recovered. D5 adds two things without weakening the atomic LoadOrStore guard on mergeTrainInFlight.
Durable, restart-safe in-flight reconstruction (FR-1/FR-4). dispatchMergeTrainWorker keeps LoadOrStore as the sole race-critical, in-memory check (dispatch runs only from the single poll goroutine). Because that map is lost on restart, prepareTrainWorker calls reconstructTrainState inside the already-guarded goroutine, before pinning the base. (Since #2051 this is the fallback: a persisted run record, when valid, is adopted first — see §”Merge-Train Run State” — and only an absent, unusable or invalid record reaches these routes.), probing only durable artifacts — merge-train PRs via ListPRs and origin trial branches via WorktreeManager.ListTrainBranchesOnOrigin() (git ls-remote --heads origin refs/heads/fabrik/merge-train/*). A PR is recognized as a train PR by isTrainPR — structurally, by its fabrik/merge-train/* head branch (HeadRefName) alone (R7, #1615): every genuine trial PR, landing PR or unpromoted draft CI PR alike, is Fabrik-created on that branch. The batch marker (<!-- fabrik-merge-train-batch -->, carried by a landing PR, never by a draft CI PR) is corroboration only — reconstructTrainState logs when it’s absent on a branch-matched PR, but never treats its presence or absence as identity. Before #1615’s R7, isTrainPR accepted the marker or the branch — a PR whose body merely quoted the marker literal in prose (with nothing to do with any trial) was misidentified as a train PR; this is what let the sweep close a live, unrelated PR during reconstruction for a different batch, an incident that hit this very fix’s own PR (see R8 below).
Relevance filtering — the critical guard. ListPRs returns state=all, so it also surfaces merged/closed integration PRs from prior completed batches; those still carry the batch marker but have no members left in today’s Queued snapshot (members only leave Queued on a successful land). Reconstruction therefore selects the first train PR whose parsed members still intersect the current Queued batch (filterBatchByNumbers(batch, parseTrainMembers(body)) non-empty). A historical PR that fails this test is skipped — routing on it would wrongly abort today’s fresh batch (complete-deferred would find no still-Queued members and exit), permanently stalling the train after the first landing. A stale open train PR that fails the test is closed (and its branch cleaned) so it cannot later hijack findIntegrationPR during a fresh batch’s landing — but only after a second, deliberately redundant structural check immediately before the CloseIssue call (R8, #1615): closing a PR is destructive and irreversible, so the identity that authorizes it is re-confirmed at the point of the action itself, not only inherited from the isTrainPR filter above it in the loop. A PR that reaches this point without a train-branch head ref is skipped and logged instead of closed — ambiguity fails closed, never toward a destructive action. From the selected relevant PR (or none), it routes to:
- complete-deferred — a merged marker PR whose members are still Queued (checked first, so already-landed work is never misclassified as an orphan):
completeDeferredLandingparses the member#Ns from the PR body (parseTrainMembers), intersects them with the still-Queued snapshot, and first re-checks each member’s Status live (liveLandingState, §6.28 — a member the stale snapshot shows as Queued but that is already Done is skipped, never re-landed), then runs the idempotentlandMergeTrainBatch(itsalreadyMergedshort-circuit skips the merge) to finish the member lifecycle; clears the in-flight marker; - resume — an open train PR (with still-Queued members) backed by an origin trial branch:
resumeTrainre-resolves the still-Queued members, re-polls CI on the existing trial head, and on green lands vialandGreenBatch(with main-moved recovery); any non-green outcome (or no resolvable members) dissolves, so the next poll re-forms cleanly rather than re-entering bisection on resume; clears the in-flight marker; - dissolve — an open train PR (with still-Queued members) without a backing trial branch:
dissolveBatch(FR-5), commenting only on that PR’s own members; clears the in-flight marker; - fresh — nothing relevant, or only orphaned remnants: any orphaned
fabrik/merge-train/*origin branch without a relevant PR is a crash remnant and is cleaned up silently (neverdissolveBatchd with today’s members — that would post confusing “batch dissolved” comments on unrelated fresh Queued issues), thenreconstructTrainStatereturnsfalseand the worker forms a new batch this poll (a fresh trial gets a new, unique branch name — no clash).
Reconstruction reads only durable state (never the map) and never launches a goroutine, so a restart with an empty map resumes/completes/dissolves an existing train rather than starting a duplicate. The ls-remote branch probe is gated under the trainValidateFn test seam (membership-keyed tests drive routing through listPRsFn).
Main-moved (behind) recovery at the landing gate (FR-2/FR-6). The green case calls landGreenBatch, which — before merging — checks whether the validated-green trial branch has fallen behind its base via trialBehind (FetchCommitsBehind(base, trialBranch) > 0; a probe error is treated as up-to-date, fail-safe). MergeableStateAccepted stays narrow — behind is not widened into it; it gets this dedicated branch. If up-to-date, landGreenBatch delegates to the unchanged landMergeTrainBatch. If behind (an external push advanced main), it re-pins the base to the current origin/<base> and re-assembles the survivors off the new base via assembleAndValidate (which reuses resolveConflictWithClaude for FR-6 rebase conflicts), re-runs the combined Validate, and loops back to the landing gate — bounded by MaxTrainRebaseCycles (default 3; effectiveMaxTrainRebaseCycles). On a green re-validation it loops back to the gate; on exhaustion, a non-green re-validation, or an assembly wipeout it dissolves the batch.
Dissolve semantics (FR-5). dissolveBatch closes the integration/CI PR if open (via CloseIssue — PRs are issues; there is no ClosePR), deletes the trial branch locally and on origin (cleanupTrialArtifacts), clears the mergeTrainInFlight entry (and, via finishTrain, the itemstate.Store repo-worker liveness marker), and posts an explanatory comment on each member so the outcome is observable. Members are left untouched in the Queued column — they only advance to Done on a successful land, so no status rollback is needed. It is idempotent (safe to re-run after a crash mid-dissolve: CloseIssue on an already-closed PR and cleanup on an already-deleted branch are best-effort no-ops; the comment may double-post — acceptable and observable). The next poll re-snapshots Queued and forms a fresh train.
Compose-not-duplicate with bisection. Main-moved rebase recovery lives only in landGreenBatch (the green landing path); a red re-validation after a rebase dissolves rather than entering bisection. The rebase-cycle budget (MaxTrainRebaseCycles) and the bisection cost cap (MaxBisectValidations) are therefore disjoint and never double-count — the next poll re-forms a fresh train that bisects cleanly.
Per-repo isolation (FR-3). Serialization is keyed strictly owner/repo, so trains for different repos run concurrently; the per-repo guard never cross-blocks distinct repos.
Worker-slot scope (#2046, ADR 2046). A train holds no Engine.sem slot for its lifecycle. prepareTrainWorker and runMergeTrainWorker acquire none; assembly git work, PR operations, the per-poll trial evaluation (evalTrialCI), bisection bookkeeping and landing all run slot-free, so the train is not counted against MaxConcurrent while a trial waits on CI — since #2051 it holds not even a goroutine then. The only slot use is resolveConflictWithClaude, which brackets exactly the InvokeForComments call via acquireTrainSlot (non-blocking try first; a real wait is announced once through logfRepo as waiting for a free worker slot for conflict resolution on #N, so the repo’s job row shows it, ADR-1661) and releases it as soon as the call returns — before finalizeConflictResolution. This one site covers every path that reaches Claude: assembleTrialBranch (the main re-form step, bisection half-trials, the one-at-a-time fallback, landGreenBatch’s rebase loop) and the singleton catch-up merge (runCatchUpGit, §6.30). Rerere replay and the regenerate-only branches never take a slot. The usage-limit hold is checked before the wait and again after it. The wait is bounded only by the context, not by the invocation’s wall-time cap.
Reserved train slot (#2115, ADR 2115). With merge_train: on, New() also creates Engine.trainSem (capacity 1), so the host can run MaxConcurrent + 1 Claude processes; with it off the channel is nil and nothing changes. acquireTrainSlotEp is its only user: it tries trainSem (non-blocking), then a free Engine.sem slot, and only when both are busy logs the wait, enters the waiting for slot phase and blocks on either (or the context). Release returns the pool actually taken and stays sync.Once-idempotent. Stage dispatch, the reinvoke dispatchers and comment processing never reference trainSem, so the reserved slot is invisible to them. The #2045 auto-repair Validate is an ordinary dispatch on the general pool. Local API workers carries train_in_use/train_max; the TUI title shows slots a/b + train c/d from a sampled WorkerSlotsEvent.
Cancellation is not a verdict (#2046). A context cancellation while waiting for the slot (or killing an in-flight conflict-resolution invocation) returns the non-nil “not attempted” error (the same channel as a usage-limit exit): the member is not ejected, nothing is paused, the runaway trial counter is not charged (assembleAndValidate skips recordTrial when trainCancelled), bisect aborts the episode instead of degrading to one-at-a-time, landOneAtATime stops, landGreenBatch’s rebase loop returns without dissolveBatch, and the catch-up returns catchUpDefer. Members stay Queued. The in-flight marker is still cleared only by the two finishTrain defers (ADR-067).
References: ADR-1615: Structural Identity for Destructive Merge-Train Actions.
1.4 Label Semantics Reference
| Label | Added By | When Added | Removed By | When Removed | Gates |
|---|---|---|---|---|---|
fabrik:locked:<user> |
processItem |
Before stage invocation (lock-then-verify protocol) | releaseLock |
On stage completion, permanent failure, blocked-on-input, or lock conflict loss | Prevents other instances from processing the item |
fabrik:editing |
processComments |
Step 2 of comment processing | processComments |
Step 9 of comment processing (also on error paths). Removal uses bounded retry (≤3 attempts, 500ms/1s/2s backoff) for transient network errors; ErrNotFound is a silent no-op. Stale labels with no active Worker are cleaned up at startup by runStartupCleanup(). |
Pre-dispatch gate in itemNeedsWork (prevents goroutine launch); defense-in-depth check retained in processItem for the race window. Symmetric with fabrik:locked:<other-user>. Note: JobStartedEvent is emitted at processComments entry (step 0) — before fabrik:editing is added at step 2. The pre-dispatch gate only blocks new dispatches; the active session’s JobStartedEvent fires before the label exists. |
fabrik:reworking:<Stage> |
beginStageRework, called immediately after the Step 2 fabrik:editing add succeeds |
Only when the re-entered stage’s own stage:<Stage>:complete is already present on item.Labels — the common mid-flight-rework case (nothing to clear yet) is a no-op. Added before stage:<Stage>:complete is removed (mark-first-then-clear) so the marker’s presence is always a safe, idempotent restore trigger regardless of which of the two mutations actually landed before a crash (#1802) |
endStageRework, called at every exit path (the two setup-failure early returns, the err != nil && !completed block, and finalizeComments) |
On a non-completing exit: removed immediately, right after stage:<Stage>:complete is restored directly. On a completing exit: removal is deferred until after finalizeComments’s call to handleStageComplete returns (see the stage:<X>:complete row below) — not alongside it — so a crash anywhere during handleStageComplete’s own completion write still leaves the marker present as a durable “restore owed” signal. Stale copies with no active Worker are repaired at startup by runStartupCleanup()’s third pass, which reads the reworked stage’s name directly out of the fabrik:reworking:<Stage> label suffix — never from the item’s current board Status (a Pruefer review finding, #1802: a completing rework that also auto-advances moves Status to the next stage before the marker is removed, so a Status-keyed restore could stamp stage:<NextStage>:complete onto a stage that never ran). If stage:<Stage>:complete or fabrik:awaiting-ci is already present for that named stage (the rework settled correctly before the crash), only the marker is removed; otherwise stage:<Stage>:complete is restored first |
Strictly nested inside the fabrik:editing bracket — itemNeedsWork’s fabrik:editing gate already makes the transient stage:<Stage>:complete clear unobservable to any dispatch decision, so this label changes no gate (R4); it exists purely so the board’s label set reflects an in-progress rework instead of a stale completion claim. See ADR-1802 |
fabrik:paused |
escalateFailedStage, blockOnInput, pauseForReviewTimeout, pauseForReviewCycleLimit, pauseForCITimeout, pauseForCIFixCycleLimit, pauseForRebaseCycleLimit, attemptMergeOnValidate (on ErrNotMergeable rebase cycle limit reached, or CI wait timeout), handleStopRequest (TUI manual stop) and runShutdownPause (daemon clean-stop, ADR-1393) — both via the shared pauseInterruptedIssue primitive, tripCommentBreaker (comment-processing circuit breaker, #1089, §4.6) |
After MaxRetries, FABRIK_BLOCKED_ON_INPUT, review/CI/rebase timeout or cycle limit, TUI s-key stop, a daemon-wide clean stop (SIGINT/SIGTERM, ADR-1393), or N non-advancing comment-processing invocations within the configured window (§4.6) |
User (manual removal), or processItem (on new human comment that triggers unpause) |
When user removes it manually, or a human comments on a paused issue | Blocks processing on active stages; a human comment is an implicit resume — bot-authored comments (github.IsBotLogin) and Fabrik’s own output do not resume (resumeAuthorised, #1083), nor does a human comment that predates the pause (#1813). Cleanup stages ignore comments entirely while paused. A Validate-stage item with an externally merged PR can still advance via runValidatePRTerminalAdvance regardless of this label. tripCommentBreaker applies this label through the same pauseIssue helper as every other pause source, so the human-only resume guarantee applies uniformly (§4.6). Unchanged semantics for the daemon clean-stop writer (ADR-1393 R6) — as of #1379 also suppresses deep-fetch admission. |
fabrik:awaiting-input |
blockOnInput, pauseForReviewTimeout, pauseForReviewCycleLimit, pauseForCITimeout, pauseForCIFixCycleLimit, handleStopRequest (TUI manual stop) and runShutdownPause (daemon clean-stop, ADR-1393), tripCommentBreaker (comment-processing circuit breaker, #1089, §4.6) |
After FABRIK_BLOCKED_ON_INPUT, review/CI timeout/cycle limit, TUI s-key stop, a daemon-wide clean stop, or a comment-processing circuit-breaker trip (§4.6) |
unblockAwaitingInput; handleStageComplete (on FABRIK_STAGE_COMPLETE, to clear any orphaned label); handleNoWorkNeeded (on FABRIK_NO_WORK_NEEDED); cleanupClosedIssueTransientLabels (defensive sweep) |
When a human comment arrives (bot comments do not resume); when a stage completes (removes any orphaned label that survived a manual fabrik:paused removal); or when issue is closed (defensive sweep — label has no meaning on a closed issue) |
Combined with fabrik:paused, identifies the “awaiting user input” pause variant |
fabrik:awaiting-review |
handleStageComplete (Path 1 — only when wait_for_ci: false), checkReviewGate (Path 2), reviewGateBlocksLanding via attemptMergeOnValidate (Path 3 — the landing-decision gate, §6.6.6) |
Path 1: optimistically after stage completion when wait_for_reviews: true and wait_for_ci: false (does not check reviewer state — data is stale; omitted for wait_for_ci: true stages because Path 2 handles the gate after CI clears). Path 2: when LinkedPRReviewRequests is non-empty OR when len(outstanding)==0 && !hasReviews (the bot self-submission case — covers Copilot/Gemini-style reviewers that don’t appear in the formal requested-reviewer list but still need to submit a review). Path 3: immediately before any landing action (auto-merge enable, enqueue, direct merge, advance-to-Queued) when a live FetchPRReviews/FetchPRReviewRequests read shows the same clearing condition unmet — including when any fetch errors (either review fetch, or the FetchLinkedPR PR-number fallback), which blocks conservatively rather than clearing on unknown state. All three paths are idempotent and non-conflicting; Path 3 blocks and labels only, and never runs escalation or the timeout (those stay with checkReviewGate) |
checkReviewGate (both natural clear and timeout paths); cleanupClosedIssueTransientLabels (defensive sweep, non-gate-checked stages only — see below) |
When all reviewers submit, or when timeout elapses (removed by checkReviewGate before pauseForReviewTimeout is called); or when issue is closed at a non-gate-checked stage (defensive sweep). Excluded from the sweep at a gate-checked stage (Validate, ADR-1387, R6) — there, the settle-owner pair (runValidatePRTerminalAdvance/settleClosedValidateAdvance, §6.15) clears it atomically as part of its own transition instead |
Phase 1 / Phase 2 reprompt timers in checkReviewGate fire on label-applied-at age (not on updatedAt movement). A non-responsive bot reviewer produces no comment / no review / no PR activity, so updatedAt never moves — without periodic re-evaluation the timers would never get a chance to fire. The catch-up loop’s blocked-path records CooldownAt("review-blocked") (via CooldownRecorded{Reason: "review-blocked"} mutation) so itemMayNeedWork’s cooldown retry path re-admits the item every 10 × PollSeconds (same pattern as fabrik:blocked); a per-poll cache bypass is intentionally avoided because long-lived review-waiting items would otherwise become a permanent GraphQL hot path. Blocks auto-advance until review gate clears |
fabrik:awaiting-ci |
handleStageComplete (on FABRIK_STAGE_COMPLETE for wait_for_ci: true stages; idempotent); checkCIGate (on confirmed CI failure; idempotent) |
handleStageComplete: immediately on FABRIK_STAGE_COMPLETE — replaces premature stage:X:complete and keeps the item in the CI-await window (ADR 032). checkCIGate: when CI check runs for the PR head SHA have conclusion: failure/timed_out/action_required. |
checkCIGate (when mergeable_state == "clean", when the check-run/required-context classification chain reports all-green — including for an unstable PR whose observed checks all passed, ADR-1441 — or when gate times out); attemptMergeOnValidate (when the mergeable_state == "clean" shortcut fires, ADR-1441); cleanupClosedIssueTransientLabels (defensive sweep) |
When GitHub’s mergeable_state is "clean" (v0.0.52 shortcut, narrowed by ADR-1441/#1441 to clean only — unstable no longer shortcuts here, see below); when all CI checks pass (green) under the per-check classification fallback (this is now the only path unstable can clear through); when timeout elapses (removed before pauseForCITimeout is called); or when issue is closed (defensive sweep) |
Signals CI gate is active (pending or failed); triggers itemMayNeedWork updatedAt cache bypass; suppresses dispatcher re-invocation (itemNeedsWork returns false); blocks auto-advance until CI gate clears. stage:X:complete is absent while this label is present — it is added by checkCIGate when CI clears (R5) or when the mergeable_state == "clean" shortcut clears the gate (v0.0.52/ADR-1441). A confirmed check-run failure on an unstable PR (non-required check red) now applies this label and blocks, instead of the pre-ADR-1441 shortcut clearing it outright — see §2.10 and ADR-1441. |
fabrik:rebase-needed |
checkMergeabilityGate (catch-up loop, wait_for_ci: true stages); checkAutoMergeConvergence (convergence flow, yolo+Validate with fabrik:auto-merge-enabled) |
When GitHub reports mergeable == false on the linked PR — a confirmed base-branch conflict. Applied idempotently in both paths. NOT added when mergeable == null (GitHub still computing). |
checkMergeabilityGate (when mergeable flips back to true); checkAutoMergeConvergence (when PR merges or closes); handleValidateSHAInvalidation (SHA-invalidation scan, §2.16 — the conflict determination was tied to a completion SHA that no longer exists); cleanupClosedIssueTransientLabels (defensive sweep) |
When GitHub reports mergeable == true (after Claude’s rebase push lands), or when PR merges/closes; when the linked PR’s HEAD SHA changes after stage:Validate:complete was recorded, invalidating the prior mergeability determination (#1225); or when issue is closed (defensive sweep) |
Signals confirmed merge conflict; triggers itemMayNeedWork updatedAt cache bypass (base-branch advances don’t bump the item’s updatedAt); blocks CI gate and auto-advance until rebase resolves the conflict. Never applied for a MergePR CI refusal (gh.ErrNotMergeableCI) — none of MergePR’s four call sites read its returned error to drive this label or increment a rebase cycle; see “MergePR’s own CI precondition (ADR-933)” in §5 |
fabrik:claude-limit |
handleUsageLimitExit |
On a Claude invocation that exits because the account’s usage limit was hit (claudeUsageLimitError, detected structurally in interpretClaudeResult/classifyUsageLimitExit from resp.TerminalReason == "blocking_limit", or resp.TerminalReason == "api_error" with api_error_status == 429 (§7.3a) — never from output prose), gated on the label’s own absence (idempotent — comment posted only on the absent→present transition) |
The non-limit path in finalizeStageOutcome, unconditionally, immediately after the usage-limit branch — fires for success, blocked-on-input, no-work-needed, genuine failure/retry, and PR-creation failure alike; settleClaudeLimitLabelSweep (account-wide, once the usage-limit hold lifts); cleanupClosedIssueTransientLabels (defensive sweep, via transientLifecycleLabels) |
On the next invocation that is not itself classified as a usage-limit exit; account-wide, every poll, once claudeUsageLimitedUntilTime reports no active usage-limit hold; or when issue is closed (defensive sweep) |
Distinct from GitHub’s own rate-limit terminology (engine/backoff.go/engine/terminal.go) — this is the Claude account usage limit, not the GitHub GraphQL budget. StageAttempted is recorded (so the normal dispatch cooldown applies, preventing a tight retry loop against the limit) but StageRetryIncremented is deliberately never called — the stage never ran, so this does not count against max_retries. Neither stage:<name>:failed nor fabrik:paused is applied. See §7.3 and ADR-1119, ADR-1183. |
fabrik:clear-claude-limit |
operator | Applied manually to any open board item to clear an active account-wide Claude usage-limit hold without an engine restart | settleClaudeLimitClearRequests, the same poll cycle it was observed on |
Consumed immediately — the scan clears the usage-limit hold once and removes the label from every carrying item in the same pass | One-shot command label, mirroring fabrik:revalidate. Not scoped to items also carrying fabrik:claude-limit — the usage-limit hold it clears is account-wide, not per-issue. See §7.3 and ADR-1183. |
fabrik:api-key-helper-detected |
handleAPIKeyHelperDetected |
On a Claude invocation skipped because the worktree’s own .claude/settings.json or settings.local.json sets apiKeyHelper (apiKeyHelperDetectedError, returned by runInvocationWithExtension before Claude is ever invoked — a repo-resident setting Fabrik cannot see until the worktree exists, distinct from the startup-time checkAPIKeyHelper preflight covering the managed-policy/user/fabrikDir-project layers), gated on the label’s own absence (idempotent — comment posted only on the absent→present transition) |
The non-detection path in finalizeStageOutcome, unconditionally, alongside the fabrik:claude-limit clear — fires for success, blocked-on-input, no-work-needed, genuine failure/retry, and PR-creation failure alike; cleanupClosedIssueTransientLabels (defensive sweep, via transientLifecycleLabels) |
On the next invocation that is not itself classified as a usage-limit exit or an apiKeyHelper detection; or when issue is closed (defensive sweep) |
Mirrors fabrik:claude-limit’s “stage never ran” shape exactly: StageAttempted is recorded (normal dispatch cooldown applies) but StageRetryIncremented is deliberately never called — does not count against max_retries. Neither stage:<name>:failed nor fabrik:paused is applied. Unlike fabrik:claude-limit, there is no account-wide settle sweep — the condition is inherently per-worktree, self-resolving once a human removes apiKeyHelper from the repo. See ADR-1346, R13. |
fabrik:skill-unresolved |
handleSkillUnresolved (engine/skill_unresolved.go), from finalizeStageOutcome (stage path, via skillUnresolvedError returned by runInvocationWithExtension) and processCommentsClassified (comment/CI-fix/rebase/review-reinvoke path) |
When a stage’s skill reference resolves to no loaded plugin (§7.18), gated on the label’s own absence (idempotent — comment posted only on the absent→present transition) | The non-denial path in finalizeStageOutcome (any invocation that reached Claude), and clearSkillUnresolved clears the panel entry when the reference resolves again; cleanupClosedIssueTransientLabels (defensive sweep, via transientLifecycleLabels) |
On the next invocation that reaches Claude; or when issue is closed (defensive sweep) | Self-healing did-not-run, mirroring fabrik:api-key-helper-detected: StageAttempted recorded on the stage path, StageRetryIncremented never; neither stage:<name>:failed nor fabrik:paused. On the comment path the result is didNotRunSkillUnresolved (reinvoke cycle refunded, no breaker charged). Surfaces in the attention classifier as needs-human. See §7.18, ADR 2137. |
fabrik:tools-denied |
Inline in finalizeStageOutcome’s final else branch (the toolsDenied case) — the stage-dispatch path; and, identically (#1704), a mirroring errors.As(err, &toolsDeniedErr) branch in processComments’s error handling — the shared funnel for ordinary comment review and all three reinvoke dispatchers (dispatchCIFixReinvoke, dispatchReviewReinvoke, dispatchRebaseReinvoke). Both call the same recordToolsDeniedDetection helper, so the label/comment/counter bookkeeping cannot drift between the two consumers of interpretClaudeResult’s shared classification |
On a Claude invocation whose tool call(s) were denied by the CLI’s own permission layer (claudeToolsDeniedError, detected structurally in interpretClaudeResult’s clean-exit path from a non-empty resp.PermissionDenials array, gated on !completed — never from output prose), gated on the label’s own absence (idempotent — comment posted only on the absent→present transition) |
The non-denial path in finalizeStageOutcome, gated on !toolsDenied (unlike fabrik:claude-limit/fabrik:api-key-helper-detected, this invocation itself may BE the condition — the classification does not short-circuit, so the clear must not fire on the very detection it would erase); cleanupClosedIssueTransientLabels (defensive sweep, via transientLifecycleLabels) |
On the next invocation that is not itself classified as a tools-denied exit; or when issue is closed (defensive sweep) | Structurally unlike its two siblings: the CLI exits cleanly (is_error: false, terminal_reason: "completed") and real work may have happened before the denial, so this does NOT short-circuit finalizeStageOutcome — commitWIP, the branch push, and markCommentsSeenByStage all still run, mirroring claudeTurnLimitError/claudeResumeFailureError’s continue-processing shape rather than the did-not-run family’s early return. StageAttempted is recorded (normal dispatch cooldown applies) but StageRetryIncremented is deliberately never called — does not count against max_retries. Bounded instead by its own ToolsDeniedRetries/MaxToolsDeniedRetries counter (default 3) — shared across both consumers, so a denial on ci-fix-reinvoke and a denial on the stage path accumulate against the same per-stage counter; at the bound, pauseForToolsDeniedLimit applies fabrik:paused + fabrik:awaiting-input (never stage:<name>:failed — the condition is never treated as a stage failure, even at the bound), reused unforked from either call site. Because classification is expressed as a non-nil error from interpretClaudeResult, blockedOnInput := err == nil && ... is structurally unreachable for a detected denial — the outcome is identical whether or not the worker also emits FABRIK_BLOCKED_ON_INPUT. On the comment/reinvoke path, the new branch sits ahead of checkNoOpCommentCycle/checkCommentBreaker and returns before either runs, so a mode denial is bounded exclusively by MaxToolsDeniedRetries, never by the comment breaker’s frequency-based thresholds (which stay unchanged). See §7.3b, §7.3c, and ADR-1523/#1704. |
fabrik:toolchain-stale |
checkToolchainDrift (engine/toolchain.go) — called from both runInvocationWithExtension (stage dispatch, immediately after the apiKeyHelper check) and processComments (comment review, right after context files are written) against the same shared implementation, so the two invocation paths cannot drift (#1786) |
On a comparable declared-vs-resolved toolchain mismatch (.nvmrc, package.json engines.node, go.mod toolchain/go directive, or .tool-versions, compared against the binary version actually resolved on the daemon’s own inherited PATH via node --version/go version), gated on the label’s own absence (idempotent — comment posted only on the absent→present transition). Detection never returns an error and is never routed through finalizeStageOutcome — unlike fabrik:claude-limit/fabrik:api-key-helper-detected, it never skips the invocation; the stage/comment-review call proceeds exactly as it would have regardless of what it finds |
checkToolchainDrift itself, the next time it runs for that issue and finds no comparable mismatch (declaration file fixed, or the daemon restarted with a corrected PATH); cleanupClosedIssueTransientLabels (defensive sweep, via transientLifecycleLabels) |
On the next invocation (stage dispatch or comment review) that observes no comparable mismatch; or when issue is closed (defensive sweep) | Warn-and-continue, not a gate: StageAttempted/StageRetryIncremented/max_retries bookkeeping is entirely untouched, and neither stage:<name>:failed nor fabrik:paused is applied. Comparison is deliberately narrow and fails closed on ambiguity (an .nvmrc alias, a complex engines range, an unparseable .tool-versions line, or a binary missing from PATH are all “not comparable” — silence, never a false-positive warning); .nvmrc/.tool-versions/engines.node compare at major-version granularity, go.mod’s toolchain/go directive at major.minor using minimum-version semantics. A successful resolution is memoized process-wide per tool for the daemon’s own lifetime (resolveToolVersion) — sound specifically because the bug this label exists to catch is that this PATH cannot change without a restart; a resolution failure (timeout, transient exec error) is deliberately never cached, since a failure carries no such invariance guarantee — caching it would permanently and silently disable detection for that tool after a single bad moment. No dedicated settle scan: this is a fire-once informational label, structurally identical to fabrik:api-key-helper-detected. See ADR-1786. |
fabrik:awaiting-done |
handleNoWorkNeeded |
As the very first mutation, the instant processItem decides completed && noWorkNeeded — before the fabrik:awaiting-input clear, before the emitting stage’s completion label, before anything else (idempotent: a no-op if already present) |
clearNoWorkNeededMarker (after a fully successful settleNoWorkNeeded pass — status moved to Done and issue closed); escalateNoWorkNeededFailure (after MaxRetries failed settle passes) |
When the Done move and issue close have both succeeded (or were already true), or when escalated (fabrik:paused takes over dispatch suppression instead) |
Suppresses dispatch of every non-cleanup stage in itemMayNeedWork/itemNeedsWork, independent of item.Status — the outstanding board move is exactly what may be failing, so the item can be observed sitting at any column while this label is present. Retried every poll by the no-work-needed settle scan, settleNoWorkNeededScan (engine/poll_settle.go; called from poll() in engine/poll.go, immediately after runValidatePRTerminalAdvance), which resolves the current stage via stages.FindStage(e.cfg.Stages, item.Status) and calls settleNoWorkNeeded. Deliberately excluded from cleanupClosedIssueTransientLabels’s closed-issue defensive sweep — unlike other gate labels, stripping it before the settle scan has finished would silently resurrect the bug this label exists to prevent (§6.8, ADR-060) |
fabrik:awaiting-member-close |
markMergeTrainMemberCloseOutstanding (landSingleton) |
Only in the failure branch of landSingleton’s member-issue CloseIssue call — after the PR merge, Done-move, and member-PR close have already run (idempotent: a no-op if already present) |
clearMergeTrainMemberCloseMarker (after a fully successful settleMergeTrainMemberClose pass — issue confirmed closed); escalateMergeTrainMemberCloseFailure (after MaxRetries failed settle passes) |
When the member issue is confirmed closed (by us or by GitHub’s own Closes #N auto-close), or when escalated (fabrik:paused takes over instead) |
Not wired into itemMayNeedWork/itemNeedsWork or transientLifecycleLabels — by the time this label can be written, the item has already reached its terminal singleton-landing outcome, so there is no per-stage redispatch risk to guard against (§6.10, ADR-061). Retried every poll by settleMergeTrainMemberCloses (engine/merge_train_member_close_settle.go; called from poll() in engine/poll.go, immediately after handleMergeTrainBatch), which scans raw board.Items directly — independent of merge_train: on/off, deepFetchCandidates, and the terminal-skip optimization (#689) |
fabrik:awaiting-close |
markNonDefaultBaseCloseOutstanding (closeIssueIfNonDefaultBase) |
In the failure branch of closeIssueIfNonDefaultBase’s explicit CloseIssue call, or by guardDefaultBaseAutoClose when a default-base issue is still open (or unreadable) at the landing (#1962) — after the caller’s Done-advance has already run (idempotent: a no-op if already present) |
clearNonDefaultBaseCloseMarker (after a fully successful settleNonDefaultBaseClose pass — issue confirmed closed); escalateNonDefaultBaseCloseFailure (after MaxRetries failed settle passes) |
When the issue is confirmed closed (by us or by any other actor), or when escalated (fabrik:paused takes over instead) |
Not wired into itemMayNeedWork/itemNeedsWork or transientLifecycleLabels — structurally identical to fabrik:awaiting-member-close: by the time this label can be written, the item has already reached Done, so there is no per-stage redispatch risk to guard against (§6.13, ADR-1097). Retried every poll by settleNonDefaultBaseCloses (engine/close_nondefault_base_settle.go; called from poll() in engine/poll.go, immediately after settleMergeTrainMemberCloses), which scans raw board.Items directly |
fabrik:awaiting-advance |
markAdvanceFailureOutstanding (recordAdvanceOutcome) |
Only in the failure branch of recordAdvanceOutcome’s advanceToNextStage call — advanceToNextStage is the last board mutation both call sites (advanceValidateTerminalItem’s merged-PR path, advanceConvergedPRToDone) make before it, so every other side effect that precedes it (gate-label clearing, completion-label filling, fabrik:auto-merge-enabled removal) has already run (idempotent: a no-op if already present — gated so a repeat failure in the same episode never posts a second comment, R5). A one-time explanatory comment naming the failing stage and the underlying error (react=false) is posted in the same branch. Note: advanceToNextStage is not the final mutation either call site makes — both unconditionally call closeIssueIfNonDefaultBase immediately afterward regardless of the advance’s outcome (Pruefer, PR #1469 review, third round); see §6.17’s “Scope” paragraph. |
clearAwaitingAdvanceMarker only — after a fully successful recordAdvanceOutcome retry pass. Unlike every other label in this table, escalateAwaitingAdvanceFailure does not remove this marker (adds fabrik:paused alongside it instead) — see §6.17 for why. |
When the terminal advance finally succeeds (board Status option now exists) | Not wired into itemMayNeedWork/itemNeedsWork or transientLifecycleLabels — structurally identical to fabrik:awaiting-close/fabrik:awaiting-member-close: by the time this label can be written, the item’s own stage is already gate-complete, so there is no per-stage redispatch risk to guard against (§6.17, ADR-1422). Retried every poll by settleAwaitingAdvanceScan (engine/advance_settle.go; called from poll() in engine/poll.go, immediately after settleClosedValidateAdvance), which scans raw board.Items directly and shares the poll’s advancedItems dedup map with runValidatePRTerminalAdvance/settleClosedValidateAdvance. Can coexist with fabrik:paused after an escalation — the settle scan’s own guard (hasLabel(fabrik:paused)) is what suppresses further retries while paused, not the marker’s absence |
fabrik:awaiting-placement |
spawnChildren (via recordChildPlacementFailure) |
On any of the three UpdateProjectItemStatus failure branches at spawn time (call error, nil statusField, or no suitable status option) — applied to the child issue, not the parent |
clearChildPlacementMarker (after a successful settleChildPlacement pass, or the closed-child short-circuit in the child board-placement settle scan); escalateChildPlacementFailure (after MaxRetries failed settle passes) |
When placement succeeds, the child is observed closed, or escalation fires (fabrik:paused takes over) |
Unlike fabrik:awaiting-done, this label does not suppress dispatch via itemMayNeedWork/itemNeedsWork — a child stranded in an unmatched column (typically Backlog) never resolves to a stage there in the first place, so there is no dispatch to suppress. Retried every poll by the child board-placement settle scan, settleChildPlacements (engine/poll_settle.go; called from poll() in engine/poll.go, immediately after cleanupClosedIssueTransientLabels), which is sourced from board.Items directly (not deepFetchCandidates — the child never passes itemMayNeedWork’s stage == nil guard to get there) and calls settleChildPlacement. Deliberately excluded from cleanupClosedIssueTransientLabels’s closed-issue defensive sweep, for the same reason as fabrik:awaiting-done — see §6.9, ADR-062 |
fabrik:awaiting-runaway-alert |
markRunawayAlertOutstanding (fireRunawayGuard) |
Only in the failure branch of fireRunawayGuard’s per-member AddComment call — fabrik:paused/fabrik:awaiting-input are applied unconditionally regardless of this branch (idempotent: a no-op if already present) |
clearRunawayAlertMarker (after a fully successful settleRunawayGuardAlert pass, or a racing fireRunawayGuard call for the same member succeeding first); escalateRunawayAlertFailure, but only if its own fallback comment succeeds (after MaxRetries failed settle passes) |
When the alert comment is confirmed posted (by us or by a racing call), or when escalated and the fallback comment itself is confirmed posted — unlike every other marker in this family, escalateRunawayAlertFailure does not delegate to the shared escalateSettle helper, precisely so a persistently-failing AddComment (not just a transient one) cannot silently erase the marker while leaving the member with zero delivered explanation (#1533 review, finding 1); if the fallback comment also fails, the marker stays and settleRunawayGuardAlertScan keeps retrying indefinitely |
Deliberately not gated on fabrik:paused’s absence in its own settle scan — unlike every sibling in this family (fabrik:awaiting-member-close, fabrik:awaiting-close, fabrik:awaiting-advance), fabrik:paused is present from the very first application of this marker (the pause never depends on the comment succeeding), so a paused-item guard here would make the scan a permanent no-op. Retried every poll by settleRunawayGuardAlertScan (engine/runaway_alert_settle.go; called from poll() in engine/poll.go, immediately after settleMergeTrainMemberCloses), which scans raw board.Items directly. See §6.18, ADR-1533 |
fabrik:awaiting-landing-verification |
landMergeTrainBatch/landSingleton (engine/merge_train.go), advanceConvergedPRToDone (engine/merge_gate.go), advanceValidateTerminalItem’s merged-PR branch (engine/pr_terminal_advance.go) |
Immediately after each call site’s own advanceToNextStage/recordAdvanceOutcome call succeeds — the item has actually reached Done, not merely attempted the advance (idempotent: a no-op if already present) |
clearLandingVerificationMarkers (after a fully successful settleLandingVerification pass — credited PR confirmed merged); failLandingVerification (on a confirmed non-merge — R2’s immediate reopen path, not gated by the retry counter); escalateLandingVerificationFailure (after MaxRetries failed inconclusive passes) |
When the credited PR is confirmed merged, when a confirmed non-merge is found (fires on the same pass, R2), or when escalated (fabrik:paused takes over instead) |
Not wired into itemMayNeedWork/itemNeedsWork or transientLifecycleLabels — by the time this label can be written, the item has already reached Done, so there is no per-stage redispatch risk to guard against (§6.19). Retried every poll by settleLandingVerification (engine/landing_verification_settle.go; called from poll() in engine/poll.go, immediately after settleNonDefaultBaseCloses), which scans raw board.Items directly. Two regimes never conflated: a confirmed failure fires immediately (AC1, “within one poll cycle”); an inconclusive result (API error, or no crediting reference discoverable) goes through the bounded retry-then-escalate regime and never reopens the issue on ambiguity (AC4). See §6.19, ADR-1616 |
fabrik:landing-verification-failed |
failLandingVerification |
On a confirmed non-merge of the credited PR — alongside reopening the issue, moving it back to Validate, and removing fabrik:awaiting-landing-verification/fabrik:credited-pr:<N> |
User (manual, after investigating and re-landing) | Manual only — this is a human-gated distinguishing label, not self-clearing | Lets an operator tell a landing-verification failure apart from an ordinary stage failure at a glance. See §6.19, ADR-1616 |
fabrik:auto-merge-enabled |
attemptMergeOnValidate (yolo, non-cruise, Validate completion; held while unprocessed feedback is pending — §6.6.6 “Feedback gate”) |
After successful enablePullRequestAutoMerge GraphQL call — signals GitHub is handling the merge atomically. Also serves as the idempotency guard (prevents re-calling the mutation) and the budget-start anchor (timestamp read by FetchLabelAppliedAt to compute convergence budget elapsed time). |
checkAutoMergeConvergence (when PR merges, PR closes, user disables auto-merge, or budget exhausts); cleanupClosedIssueTransientLabels (defensive sweep) |
When GitHub merges the PR; when PR is closed without merging; when user disables auto-merge in GitHub UI; or when convergence budget exhausts (→ pauseForConvergenceFailed); or when issue is closed (defensive sweep) |
Engine-internal label. Bypasses legacy merge/CI gate dispatch — checkAutoMergeConvergence is the sole Phase 1 handler while present. Triggers itemMayNeedWork updatedAt cache bypass (convergence state changes independently of issue updatedAt). stage:Validate:complete remains in place while this label is present; checkAutoMergeConvergence removes it when the PR merges and the item advances to Done. NOT applied when fabrik:cruise is also present (cruise wins). |
fabrik:blocked |
checkDependencies |
When open blocking issues exist (first transition only — idempotent) | PushUnblockObserver (primary — fires immediately on blocker close, any column); checkDependencies (defense-in-depth — fires via dep-blocked cooldown-retry for items in stage columns) |
When all blocking issues close | Pre-dispatch gate in itemNeedsWork prevents re-dispatch after the label is set (parallel to fabrik:editing post-#550 and fabrik:locked:<other> always); exception: when the dep-blocked cooldown has expired (or no store entry exists), itemNeedsWork admits the item for one dependency re-check per 10 × PollSeconds — checkDependencies either re-stamps the cooldown (still blocked) or removes the label (resolved). Initial detection (first dispatch, label not yet set) still reaches checkDependencies normally. Blocks stage start. Push path removes the label even for items in non-stage columns (Backlog, Done, custom columns) that would otherwise never be re-evaluated by processItem. |
stage:<X>:in_progress |
processItem |
After lock acquired and verified | releaseLock |
Same as fabrik:locked:<user> |
Informational — shows which stage is active on GitHub |
stage:<X>:complete |
handleStageComplete (for non-wait_for_ci stages), checkCIGate (for wait_for_ci: true stages — added only after CI passes), handleNoWorkNeeded (emitting stage + all skipped stages), cleanup stage handler, endStageRework (restore on a non-completing comment-re-entry exit, #1802) |
handleStageComplete: after Claude signals FABRIK_STAGE_COMPLETE on stages without wait_for_ci: true. checkCIGate: when all CI checks pass (R5) — this is the conjunctive gate (ADR 032): stage:X:complete is deferred until the CI gate actually clears, not applied on FABRIK_STAGE_COMPLETE. After FABRIK_NO_WORK_NEEDED (emitting stage + all subsequent non-cleanup stages) or worktree cleanup. For stage:Validate:complete specifically, handleStageComplete and checkCIGate also apply ValidateCompletedAtSHA carrying the current HEAD SHA so the SHA-invalidation scan (§2.16) can detect future SHA changes. endStageRework: immediately, on a comment re-entry’s non-completing exit that had cleared it via beginStageRework — a direct re-add, not routed through handleStageComplete. |
Never removed in the ordinary case, with two exceptions: stage:Validate:complete is removed by the SHA-invalidation scan (§2.16) when the linked PR’s HEAD SHA changes after completion; and any stage’s :complete is cleared by beginStageRework for the duration of a comment re-entry that reworks it (#1802) — restored directly by endStageRework on a non-completing exit, or left to handleStageComplete’s own re-derivation (including its wait_for_ci deferral) when the re-entry itself re-signals FABRIK_STAGE_COMPLETE |
Permanent, with the same two transient exceptions: stage:Validate:complete when the linked PR SHA changes after completion (§2.16); and any stage during a comment-re-entry rework window, bracketed by fabrik:reworking:<Stage> (#1802, ADR-1802) — never observable by any dispatch-admission gate, since the window is strictly nested inside fabrik:editing’s own gate |
Prevents re-invocation of the stage; triggers catch-up advancement |
stage:<X>:failed |
escalateFailedStage |
After MaxRetries exhausted (includes retries exhausted due to repeated degenerate-output detections, §2.6/§7.2) | clearFailedStage |
When user removes fabrik:paused (manual unpause) |
Indicates permanent failure; paired with fabrik:paused |
fabrik:yolo |
User (manual) | Any time | User (manual) | Any time | Forces auto-advance; triggers auto-merge at Validate; overrides auto_advance: false per stage |
fabrik:cruise |
User (manual) | Any time | User (manual) | Any time | Forces auto-advance without merge; stops at Validate; cruise wins over yolo for both end-of-Validate decisions — auto-merge suppressed and the issue held at Validate — even when yolo is also present; both guards test the raw cruise label, not cruiseActive |
fabrik:unrestricted |
User (manual) | Any time | User (manual) | Any time | Passes --dangerously-skip-permissions instead of --permission-mode dontAsk |
fabrik:extend-turns |
User (manual) | Any time | processItem cleanup branch or User (manual) |
At Done cleanup stage completion; or manual removal | Pre-grants 2× stage.MaxTurns budget for the first stage invocation while present, with max_wall_time scaled 2× to match for that invocation; no-op for stage path when max_turns == 0 (unlimited); also pre-grants 2× commentMaxTurns(stage) budget (with matching 2× max_wall_time scaling) for the first comment processing invocation (comment budget is never 0); subsequent extensions beyond 2× reset to the base budget and unscaled deadline, require progress detection, for both paths; persists across all intermediate stages |
model:<name> |
User (manual) | Any time | User (manual) | Any time | Selects Claude model; first label wins if multiple present |
effort:<level> |
User (manual) | Any time | User (manual) | Any time | Overrides stage effort level; highest-ranked wins if multiple present |
base:<branch> |
User (manual) | Before Research (recommended) | User (manual) | Any time | Overrides worktree base branch; falls back to default if branch not found on remote; if a PR exists, its base branch is updated to match on each stage invocation. Resolution (baseBranchForItem/resolveBaseLabelBranch) first checks the local bare clone (branchExists, fast path); on a local miss it probes origin directly via git ls-remote --heads and, if confirmed there, fetches the branch into the local clone before honoring it — so a branch that exists on the remote but hasn’t yet been fetched locally is not mistaken for genuinely absent. Only falls back to default when the ls-remote probe itself confirms absence (or errors, fail-safe) |
review-authority:<mode> |
User (manual) | Any time | User (manual) | Any time | Overrides the wait_for_reviews gate’s review_authority for this issue only (advisory/authoritative); no label → stage config governs; one recognized label → it wins; both present → resolves to authoritative (more restrictive), warning logged; malformed/unknown suffix → ignored, warning logged, falls back to stage config. Resolved by effectiveReviewAuthority, consulted identically by checkReviewGate, reviewGateBlocksLanding, and pauseForReviewTimeout’s message. Only meaningful alongside wait_for_reviews: true — see §6.1.1, ADR-1261 |
expected-reviewers:<mode> |
User (manual) | Any time | User (manual) | Any time | Overrides the wait_for_reviews gate’s expected_reviewers for this issue only (none/declared); no label → stage config governs (nil stays nil); one recognized label → it wins; both present → resolves to declared (more restrictive — imposes waiting), warning logged; malformed/unknown suffix → ignored, warning logged, falls back to stage config. Resolved by effectiveExpectedReviewers, consulted identically by checkReviewGate, reviewGateBlocksLanding, and pauseForReviewTimeout’s message. Only meaningful alongside wait_for_reviews: true — see §6.1, #1304 |
fabrik:sub-issue |
spawnChildren (engine-side, shared by preImplement and the Review/Validate mid-flight hook, §6.7.2 — ADR-1419) |
Applied to each spawned child issue, from Plan (via pre-Implement) or a mid-flight Review/Validate declaration alike | N/A | N/A | Informational — marks issues created by Fabrik’s sub-issue spawn mechanism; no engine-side gate semantics |
fabrik:children-spawned |
spawnChildren (engine-side, shared by preImplement and the Review/Validate mid-flight hook, §6.7.2 — ADR-1419) |
After all FABRIK_SPAWN_CHILD_* children in one batch are successfully created and linked as blockedBy of parent, whichever stage originated the spawn |
User (manual, to re-trigger spawn) | If user wants a fresh spawn (children created under the prior spawn must be closed manually first — fabrik:spawned-child:<i>:<n> markers, ADR-1583, are only consulted while this label is absent) |
Idempotency guard against preImplement re-spawning the same Plan-declared batch on retry — while present, preImplement is a no-op. The Review/Validate mid-flight hook needs no equivalent guard (its own idempotency comes from parsing each dispatch’s fresh output exactly once, §6.7.2), but still sets this label on a successful spawn — it remains true that this parent has spawned children at least once. |
fabrik:spawned-child:<blockIndex>:<childNumber> |
spawnChildren (ADR-1583) |
Immediately after each spawn block’s child issue is successfully created, on the parent | spawnChildren, once the full batch (including sibling DEPENDS_ON wiring) succeeds |
N/A (best-effort cleanup on success) | Durable, restart-surviving marker recording that this Plan-declared block’s child already exists — see §6.7’s “Resume-safe retry” note. Transient only: present during an interrupted/in-progress spawn, absent once fabrik:children-spawned is set. Removing one manually forces re-creation of that specific block on the next retry (e.g. if the recorded child was genuinely deleted). |
fabrik:revalidate |
User (manual) | Applied to any issue to request Validate re-entry | Revalidate-scan loop (handleRevalidateLabel), plus cleanupClosedIssueTransientLabels (defensive sweep) |
On Validate-stage items: immediately after removing all gate/completion labels; on non-Validate items: after logging a warning; on closed issues: defensive sweep | Triggers removal of stage:Validate:complete, stage:Validate:failed, fabrik:paused, fabrik:awaiting-input, fabrik:awaiting-ci, fabrik:auto-merge-enabled, then itself; resets PausedByEngine, StageRetryCount, LastAttemptAt, EngineCycles for Validate; Validate dispatches on the next poll cycle. Applied to non-Validate issues: only the trigger label is removed, no other action. Applied while a Validate worker is in-flight: held in place until the worker exits (FR-4). Bypasses the updatedAt cooldown cache via hasAwaitingLabel so stuck-Validate items with settled updatedAt are still deep-fetched. |
2. Event Enumeration
Thirteen distinct event types drive state transitions (§2.1–2.11, §2.13, §2.14), plus one TUI display event (§2.12) that does not drive transitions:
2.1 Poll Tick
Trigger: The engine’s poll loop fires on a configurable interval (PollSeconds).
Code path: poll() → itemMayNeedWork() (shallow filter) → FetchItemDetails() (deep fetch) → itemNeedsWork() (full filter) → catch-up loop → dispatch loop → processItem()
Effect: The primary driver of all state transitions. Each poll cycle evaluates every item on the board through the filter chain and dispatches work for qualifying items.
2.2 New User Comment
Trigger: A user posts a comment on an issue or its linked PR. Detected by findNewComments() — filters out Fabrik-generated comments (prefix 🏭 **Fabrik) and already-processed comments (ROCKET reaction or CommentProcessed entry in itemstate.Store).
Code path: itemNeedsWork() detects new comments → processItem() routes to processComments() or triggers unpause/unblock
Landing hold (#1862, #1953): unprocessed feedback — a comment (exactly findNewComments), an unresolved current-head review thread, or an unaddressed review body — on a Validate-complete item also blocks the landing decision itself — see §6.6.6 “Feedback gate”. The feedback is processed first; landing is re-evaluated afterwards. A bot review of a pure merge-train catch-up push (§6.30) is the one non-actionable exception.
Effect: Can trigger three distinct behaviors:
- Unpause: On a paused issue, a human comment (
resumeAuthorised()— excludes bot logins viagithub.IsBotLogin; Fabrik’s own output is separately excluded upstream byfindNewComments’🏭 **Fabrikbody-prefix check) created at or after the latestfabrik:pausedlabeledevent removesfabrik:paused(and clears failed state if present) and falls through to comment processing. A bot-authored comment does not unpause (#1083), and neither does a human comment that strictly predates the pause (#1813, §4.4). - Unblock awaiting-input: On an awaiting-input issue, a human comment removes both
fabrik:pausedandfabrik:awaiting-input, then routes toprocessComments(). A bot-authored comment, or a human comment that strictly predates the pause, does not unblock. - Comment processing: On an active (non-paused) issue, any new comment (human or bot) routes directly to
processComments()— unaffected by the human-only resume restriction above - Post-merge guard (#1862, ADR-1862): Before any of the above reaches a worker,
processCommentsClassifiedchecks whether the item’s work has already landed (itemPRAlreadyLanded: a merged linked PR — confirmed withFetchPRMergedwhen the list endpoint shows closed-not-merged — or a merge-train member’sfabrik:credited-pr:<N>/fabrik:awaiting-landing-verificationlabel). If so it invokes no worker, makes no commit or push, and does not add 👀 orfabrik:editing. For each human-authored comment (with aDatabaseID) not already answered, it posts one reply on the issue and the linked PR saying the work already landed, the change was not applied, and it needs a new issue. The triggering comment gets no 🚀 — it must not look processed — so it stays “new”; the reply carries an HTML marker<!-- fabrik:post-merge-comment:<commentID> -->and the guard skips comment IDs already marked initem.Comments, so it answers each comment once, durably across restarts. The guard runs beforerecordCommentBreakerInvocation, so it never feeds the circuit breakers, and returns thedidNotRunPostMergekind so reinvoke cycle counters are refunded. This deliberately breaks “processed ⇒ 🚀” for this one class of comment. It acts only on positive evidence: a read error, no PR, an open PR, or a human-closed (closed-not-merged, no landing label) PR falls through to normal processing. - Circuit breaker trip: Every
processComments()invocation (from any of the three paths above, or from a review/CI-fix/rebase reinvoke, §6.2/§6.5/§6.6) records a comment-processing circuit-breaker invocation; if the issue accumulates N invocations within a rolling window with no forward progress, the engine appliesfabrik:paused+fabrik:awaiting-inputitself and suppresses further dispatch — see §4.6. - Overseer audit comments (#1969, ADR-1966-c): The
🏭 **Fabrik — overseer action: <action>**comment anfabrik mcpaction tool posts is Fabrik’s own output: the same🏭 **Fabrikprefix check drops it fromfindNewComments, so it can neither resume a paused item nor start a comment-review cycle even when, under PAT auth, its author is the operator’s own non-bot login; see §7.15.
2.3 PR Review State Change
Trigger: A pull_request_review webhook event with action ∈ {submitted, edited, dismissed} arrives. All three actions carry the full review object (author, state, body, DatabaseID) and are routed through the same review-upsert path in applyPullRequestReviewDelta. The webhook action itself is not stored — only the review state (from the payload’s review.state field, normalised to uppercase) is recorded in item.LinkedPRReviews (upserted by DatabaseID).
submitted— reviewer submitted a new review (APPROVED, CHANGES_REQUESTED, or COMMENTED).edited— reviewer edited an in-progress review (the only action some bot reviewers, e.g. GitHub Copilot, ever send).dismissed— a prior APPROVED or CHANGES_REQUESTED review was dismissed; the stored state becomesDISMISSED.
Code path: Delta applied by applyPullRequestReviewDelta → itemstate.PRReviewSubmitted upsert → catch-up loop in poll() re-evaluates checkReviewGate().
Effect: Can clear the review gate when len(outstanding) == 0 and at least one non-DISMISSED review exists. A DISMISSED review does not satisfy the hasReviews condition. Does not directly trigger a stage invocation.
The self-review escape hatch. hasReviews does not check the review’s author or state beyond “not DISMISSED” — a COMMENTED review submitted by the PR author itself satisfies it, even though GitHub forbids a PR author from approving their own PR. This is intentional, not an oversight: the advisory gate’s real question is “has a human looked at this?”, not “has a reviewer approved this?” (see §6.1.1 for the authoritative-mode distinction, which layers an approval requirement on top). On a repo with no reviewer ever requested — no CODEOWNERS, no review bot installed — a self-authored COMMENTED review is the supported manual way to clear an otherwise-unsatisfiable gate; wait_for_reviews: false is the supported stage-config setting for such a repo going forward. See docs/USER_GUIDE.md’s wait_for_reviews section for the operator-facing version of this note.
2.4 PR Review Threads with Feedback
Trigger: Reviewers leave inline code comments on the linked PR in unresolved review threads. These are real GitHub comments with DatabaseIDs.
Code path: Detected by buildReviewThreadComments() in the catch-up loop → dispatchReviewReinvoke() → async processComments() with synthetic comments
Effect: Triggers a review reinvocation cycle — the stage agent is re-invoked via processComments() with the review thread comments as input, allowing it to address reviewer feedback. This is a distinct event type from regular comment processing (see §6.2).
2.5 Blocking Issue Closed
Trigger: An issue listed in item.BlockedBy transitions to the CLOSED state, OR a dependent item’s BlockedBy slice is populated for the first time via deep-fetch while all listed blockers are already closed.
Primary code path (push-based): PushUnblockObserver
PushUnblockObserver is registered on engine.store and fires on two distinct events. The dependent unblocks within seconds regardless of which event arrives first.
Trigger 1 — StateChanged (blocker closes): When a blocker Y closes:
- The observer calls
store.All()to scan all known items. - For each item X that carries
fabrik:blockedand has Y in itsBlockedBylist, it checks every remaining blocker viastore.Get()(fresher thandep.Statefrom the last board fetch; falls back todep.State == "CLOSED"if the blocker is not in the store). - If all of X’s blockers are now closed, the observer dispatches
removeBlockedIfResolvedon a goroutine, which removesfabrik:blockedviaRemoveLabelFromIssue(3 attempts, exponential backoff, idempotent) and applies the cache write-through.
Trigger 2 — BlockedByChanged (dependent’s BlockedBy first populated via deep-fetch): BlockedBy is a deep-fetch-only field — after bootstrap every item has BlockedBy = nil until its first ItemDeepFetched mutation. If Trigger 1 fires before the dependent’s first deep-fetch, the dependent’s BlockedBy is empty and the scan silently skips it. When the dependent IS later deep-fetched, the Store emits BlockedByChanged, and the observer reacts:
- It reads the post-mutation snapshot of the dependent item X directly (no
store.All()scan). - If
BlockedByis empty after the mutation, it returns immediately (no-op —BlockedByChangedonly fires when the slice actually changes, so this guards against a deep-fetch that clears or results in an empty dependency list). - If X carries
fabrik:blockedand all listed blockers are closed in the store, it dispatchesremoveBlockedIfResolvedon a goroutine.
Both triggers are idempotent — double-removal races are handled correctly (ErrNotFound is treated as success by removeBlockedIfResolved).
This path bypasses processItem and itemMayNeedWork entirely, so it works for items in any column — including Backlog, Done, or custom non-stage columns where itemMayNeedWork would return false. No comment is posted; the push path is a label-removal-only operation. Observer decisions (skipped, unblocked, or still-blocked reasons) are emitted under the [push-unblock] log tag in fabrik.log.
Defense-in-depth path (pull-based): itemNeedsWork exception + processItem() → checkDependencies()
processItem() applies CooldownRecorded{Reason: "dep-blocked"} each time checkDependencies() returns true (blocked). While the cooldown is active the pre-filter skips deep-fetch entirely (no goroutine, no GraphQL burn — #576 fix preserved). When the cooldown expires, itemNeedsWork detects the expiry via snap.CooldownAt("dep-blocked") and admits the item — bypassing the #576 pre-dispatch gate for this one re-check — so processItem → checkDependencies can re-evaluate: if still blocked it re-stamps the cooldown (resetting the 150s window); if resolved it removes fabrik:blocked. A missing store entry (cold-start or restart) also admits the item, since no active cooldown exists. This path fires once per 10 × PollSeconds for items in configured stage columns and serves as defense-in-depth for missed webhook events.
Note: within checkDependencies(), for each blocker the engine first consults store.Get(blocker).IsClosed() (the cache’s view) and only falls back to dep.State != "CLOSED" from the GraphQL deep-fetch if the blocker is not present in the store. This preference avoids false “still blocked” conclusions caused by GitHub indexer lag, which can delay dep.State updates by several seconds after a blocker actually closes.
Resume-from-blocked live re-read (ADR-1419, requirement 6). On the recheck path (alreadyBlocked — the item already carries fabrik:blocked), checkDependencies bypasses the cache entirely with a raw, uncached FetchItemDetails call before computing openDeps, rather than trusting the item snapshot’s own (possibly stale) BlockedBy list — mirroring recoverMissingPlanComment (engine/spawn.go) and verifyAndHealLinkage (engine/prcreate.go). This fires unconditionally on alreadyBlocked alone — not additionally gated on the cached item.BlockedBy being non-empty. A stale-empty cache is exactly the shape a bypassed engine spawn produces: a stage agent calling gh issue create directly instead of emitting FABRIK_SPAWN_CHILD never creates a blockedBy edge, so the cached snapshot has always shown zero blockers even while fabrik:blocked is present from some other source (or becomes stale after a genuine edge is added out-of-band). Gating the live read on a non-empty cached list — the pre-#1419 behavior — would skip it entirely in that shape and trust the wrong empty list, silently clearing the label on nothing. This is the check that closes the observed regression (repo-b#102): a parent resuming from fabrik:blocked must re-verify its dependency set is actually satisfied, not merely that the cached count says so.
Concurrency note: Neither StateChanged nor BlockedByChanged is in wakeChFlags or cycleSetFlags, so PushUnblockObserver fires do not wake the poll loop or populate mayNeedWork. Label removal is a direct side effect dispatched on a goroutine to avoid blocking the store notification call path. Double-removal races (two blockers closing within milliseconds, or both triggers firing for the same item) are handled correctly — ErrNotFound is treated as success by removeBlockedIfResolved.
2.6 Claude Output Markers
Trigger: Claude’s output contains one of the Fabrik markers. Checked after each stage invocation.
Markers and priority order (enforced by the if/else if dispatch chain in processItem()):
FABRIK_STAGE_COMPLETE+FABRIK_NO_WORK_NEEDED(both present) — highest priority amongcompleted == truepaths;completed && noWorkNeededbranch fires before the plaincompletedbranchFABRIK_STAGE_COMPLETE(alone) — next; handled by the plaincompletedbranchFABRIK_BLOCKED_ON_INPUT— checked last; only honored ifcompletedis false anderr == nil
FABRIK_NO_WORK_NEEDED is ignored unless FABRIK_STAGE_COMPLETE also appears in the same output — the no-work path requires the emitting stage to explicitly declare itself complete. The timeout/kill recovery path that scans buffered output for FABRIK_STAGE_COMPLETE does not trigger the no-work path (only FABRIK_STAGE_COMPLETE is scanned in that recovery path, not FABRIK_NO_WORK_NEEDED).
Code path: processItem() → outcome dispatch based on which marker is present
Effect:
- FABRIK_STAGE_COMPLETE + FABRIK_NO_WORK_NEEDED:
handleNoWorkNeeded()— adds completion label for emitting stage; adds dummystage:<name>:completelabels and one-line “skipped” comments for each subsequent non-cleanup stage; moves issue directly to Done; no PR created (see §6.8) - FABRIK_STAGE_COMPLETE:
handleStageComplete()— adds completion label, potentially advances to next stage - FABRIK_BLOCKED_ON_INPUT:
blockOnInput()— addsfabrik:paused+fabrik:awaiting-input - None of the above: cooldown retry path; eventually
escalateFailedStage()if MaxRetries exceeded
FABRIK_SPAWN_CHILD_BEGIN/END blocks are not processed inline — they are structured data emitted by the Plan stage and preserved in the Plan stage comment. The engine’s preImplement step reads them at Implement dispatch time to create child issues. See §6.7.
FABRIK_PR_CREATE_BEGIN/END blocks are processed inline — the engine parses the block during processItem(), creates the draft PR with Closes #N prepended as the first line, and records the PR number before posting output. See §5.6.
Invocation-level kill paths: The max_wall_time and inactivity timeout mechanisms (see §7.7) can terminate the Claude process before it writes a clean {"type":"result"} line. After such a kill, runClaude() retroactively scans the already-buffered output for FABRIK_STAGE_COMPLETE in intermediate {"type":"assistant"} NDJSON lines via extractTextFromAssistantTurns(). If found, completed=true is returned and the invocation is treated identically to a live FABRIK_STAGE_COMPLETE. If not found, completed=false is returned and the invocation routes to the cooldown/retry path. These kills are distinguished from engine-shutdown cancellation by the wasTimedOut flag, so they do not trigger the hard-error path.
Assistant-turn artifact harvest and no-artifact completion guard (#1782): Earlier still — inside interpretClaudeResult(), before finalizeStageOutcome() or processComments() ever see the output — the engine addresses a related but distinct defect: the CLI’s terminal result field reflects only the agent’s very last turn, which can be a near-empty wrap-up if one more tool call followed the turn that produced the real artifact. artifactMissingOnComplete() detects this (FABRIK_STAGE_COMPLETE present, FABRIK_NO_WORK_NEEDED absent, and the text carries nothing beyond Fabrik’s own bare control-marker lines — hasArtifactContent()). When it fires, extractLastSubstantialAssistantTurn() re-scans the raw NDJSON for the last assistant turn with real content (not necessarily the marker-bearing turn — the reported case’s marker turn contained only the marker) and prepends it to the result text. If nothing is found, or the harvest recovers only more control-marker-only text, artifactMissingOnComplete() still holds against the (now harvested) text, and both places interpretClaudeResult() would otherwise return completed=true are gated on its negation — the invocation returns completed=false instead, indistinguishable at this level from an ordinary incomplete run. FABRIK_NO_WORK_NEEDED co-occurring with FABRIK_STAGE_COMPLETE is excluded from both the scan and the guard (a legitimate artifact-free completion, not this defect).
Because this runs upstream of both invocation paths, completed=false arrives at finalizeStageOutcome()/processComments() exactly as it would for any other incomplete run — the comment-review path needs no dedicated handling beyond its existing no-progress machinery. The stage-dispatch path additionally recomputes artifactMissingOnComplete() against the pre-strip output (engine/item.go) purely to drive distinct messaging: a one-time first-detection comment (“no artifact harvested”) on first occurrence, and — like the degenerate-output guard below — counts against MaxRetries (the invocation did real work) rather than being exempted the way a usage-limit or tools-denied exit is, escalating via escalateFailedStage() at the limit with a cause note naming this defect distinctly from the unrelated bare-file-reference one. See ADR-1782.
Degenerate-output guard: Before any of the marker-dispatch branches above run, finalizeStageOutcome() checks the stripped/trimmed output (postOutput) against isDegenerateOutput() — a conservative check for a single-line body that is nothing but a bare @file reference or an absolute filesystem path (e.g. @/tmp/plan.md, /tmp/plan.md). This catches a model writing its real stage output to a file and returning a dangling reference instead of emitting it inline (issue #1065). If it trips, regardless of which marker was present: the output is not posted as a comment, and completed is forced to false before the branches above evaluate — so FABRIK_STAGE_COMPLETE (with or without FABRIK_NO_WORK_NEEDED) cannot advance the stage on a degenerate body. The invocation instead falls through to the “None of the above” cooldown-retry path and, on the first detection, an explanatory comment is posted immediately (rather than staying silent until MaxRetries); at the retry limit it escalates via the normal escalateFailedStage() path (see §7.2), whose pause comment names the offending reference. The detector requires an unambiguous signal (@-prefix or a leading absolute path with ≥2 segments) and only ever matches single-line output — ordinary short prose (N/A, TBD, a one-sentence completion note, or a sentence that merely mentions a path) is not flagged.
2.7 Manual Label Change
Trigger: A human adds or removes a label on the issue via the GitHub UI.
Code path: Detected on the next poll cycle when labels are fetched
Effect varies by label:
- Adding
fabrik:paused→ engine skips the item (unless a human comment arrives) - Removing
fabrik:pausedfrom a failed issue →clearFailedStage()resets retry state - Adding or removing
fabrik:yoloorfabrik:cruise→ takes effect at the next advance decision, even mid-run, on both advance paths (handleStageComplete()and the catch-up loop’srunCatchUpPhase2()) — both call the sharedrefreshAutonomyLabels()helper, which performs a live, item-scopedFetchLabels()immediately before either readshasYoloLabel/hasCruiseLabel(#1769; see §6.23 / Appendix A for the full rationale, including why removal previously went unobserved on both paths) - Adding
model:<name>oreffort:<level>→ takes effect on next Claude invocation
2.8 Issue Closed
Trigger: The issue is closed on GitHub (e.g., by PR merge with Closes #N).
Code path: itemMayNeedWork() and itemNeedsWork() check item.IsClosed
Effect (ADR-1387): A closed item is never dispatched to a Claude stage invocation. The closed-issue guard admits an item only if one of the following holds:
- The current stage is a cleanup stage (
CleanupWorktree: true) — cleanup can remove the worktree - The current stage has a
stage:<X>:completelabel — the catch-up loop can advance to the next stage (e.g., a PR merge closes an issue sitting in Validate withstage:Validate:complete; it needs to move to Done)
Everything else a closed item needs — advancing to Done, clearing stale labels, reaping a worktree, retrying a failed close — is board/label reconciliation performed by a board.Items-sourced settle scan, never by widening this admission gate. In particular, a closed item at a gate-checked stage (Validate) lacking stage:Validate:complete — carrying any gate label (fabrik:awaiting-review, fabrik:paused, fabrik:awaiting-ci) or none at all — is healed exclusively by settleClosedValidateAdvance (§6.15), independent of this admission gate. Before ADR-1387, this guard additionally admitted such items directly (a !stageIsGateChecked(stage) disjunct, plus explicit fabrik:awaiting-ci/fabrik:auto-merge-enabled admissions) on the theory that admission was the only way for the settle-owner to observe them — see §6.15 for why that coupling produced an unbounded post-close dispatch loop, and why removing it here is safe now that the settle-owner has its own independent feed.
Comment-triggered dispatch is closed on the same terms (R1, ADR-1387). Clause 2 above admits a closed, stage-complete item, so itemNeedsWork must also refuse every path that could turn a new comment on such an item into a Claude invocation. There are three: the fabrik:awaiting-input resume, the fabrik:paused unpause resume, and the plain new-comment fast path. A single item.IsClosed && !stage.CleanupWorktree guard sits ahead of all three, so a closed item carrying stage:<X>:complete together with fabrik:paused / fabrik:awaiting-input — reachable, since the pause paths apply those labels without touching the completion label — is not dispatched when a human comments. processItem carries the same guard ahead of its two resume branches as a redundant-but-explicit ownership boundary. The cleanup-stage exclusion is R1’s stated exception and covers worktree reaping only; the plain new-comment path additionally keeps its own !item.IsClosed condition so a closed item at a cleanup stage is still never routed into comment processing.
The stage:<X>:complete admission clause triggers a real FetchItemDetails deep fetch, pre-existing and unchanged by this ADR (raised on review, PR #1388). A closed item admitted solely because stage:<X>:complete is present enters deepFetchCandidates like any other admitted item — itemNeedsWork’s later “already completed this stage” check is what actually withholds dispatch, not itemMayNeedWork’s shallow filter. In the common case this is transient (the settle-owner or catch-up loop moves the item off its current column the same poll it heals it), but if the board move itself keeps failing (advanceToNextStage’s aerr/fillFailed paths in advanceValidateTerminalItem, or the equivalent in settleClosedItemsToDone), the item can sit indefinitely with stage:complete set and the board column unchanged, paying one FetchItemDetails call per poll. This clause is not new: it was present in the closed-issue guard before commit 7311a14e and is unmodified by R3’s simplification (R3 only removed the !stageIsGateChecked(stage), fabrik:awaiting-ci, and fabrik:auto-merge-enabled disjuncts) — so this ADR neither introduces nor changes this cost. A repeatedly-failing board move is itself an existing, unrelated failure mode (e.g. a missing status option, or a transient GitHub error) with no escalation path today; that gap is out of scope here, same as the “unresolved-PR polling cost” gap noted in §6.15.
Outbound case — engine-initiated close (ADR-1096): GitHub only honours Closes #N auto-close when the merged PR’s base branch is the repository’s default branch; on a base:<branch> repo where the PR targets a non-default base, the keyword is inert and nothing closes the issue, which stalls native dependency edges that unblock on close (not merge). Both terminal merge-advance paths — runValidatePRTerminalAdvance (cruise) and advanceConvergedPRToDone (non-train yolo) — call the shared helper closeIssueIfNonDefaultBase(item, prNumber) immediately after their board advance to Done, on a confirmed merge only (never for a PR closed without merging). The helper resolves the item’s base via the base:<branch> label (baseBranchForItem) and the repo’s actual default via wm.DefaultBaseBranch(); it calls CloseIssue only when they differ — when base equals default, GitHub’s own auto-close already handles it, and skipping here is what prevents a double-close. The close is best-effort and idempotent (already-closed / ErrNotFound treated as success, failures logged and never block the advance); a failed close is durably retried via the fabrik:awaiting-close settle scan — see §6.13, ADR-1097. The merge-train path (engine/merge_train.go) and the FABRIK_NO_WORK_NEEDED short-circuit (no_work_needed_settle.go) already close explicitly and are unaffected. GitHub’s inertness on a non-default base goes further than this close half alone, though: it creates no Development-panel issue↔PR link at all for such a PR, which is a purely human-visible gap the engine itself is unaffected by (FetchLinkedPR resolves by branch-name convention, not GitHub’s linkage graph) — see §6.20, ADR-1649 for the one-time notice that closes that gap.
Default-base backstop (#1962, ADR-1962): GitHub’s own auto-close on the default base is not trusted either. On 2026-09-30 GitHub silently stopped creating closing-keyword links for new PRs and dropped the auto-close even for already-linked ones, with no declared incident. When base equals default, closeIssueIfNonDefaultBase calls guardDefaultBaseAutoClose. It does not close immediately, because a landing path such as the singleton fast path reaches it within the auto-close’s own 1–2 s latency and closing then would race GitHub. Instead a live FetchIssue read (never the board snapshot, which predates the merge) decides:
- Already closed: GitHub did its job, and nothing happens.
- Still open, or the read failed: it applies
fabrik:awaiting-close.
settleNonDefaultBaseCloses then works through the same retry and escalation path. It live-reads the issue before acting, on every base, because the board snapshot can lag a late auto-close by a poll and CloseIssue is a bare PATCH that succeeds on an already-closed issue. If the live read shows the issue closed, the marker clears silently. If the read fails, the check waits for the next poll, and no “did not fire” comment is posted. That failed pass still counts toward the MaxRetries budget, so an issue that stays unreadable escalates instead of stalling. Otherwise the settle scan closes the issue with state_reason: completed and posts 🏭 **Fabrik — closed after merge**, naming the merged PR. It resolves the PR through FetchLinkedPR (by branch), since GitHub’s closing-keyword link is exactly what may be missing. Its escalation comment is worded for the default base (“post-merge explicit close failed”).
2.9 Review Reinvoke
Trigger: The catch-up loop Phase 1 detects unresolved PR review feedback — inline thread comments and/or an unaddressed review body from any review whose state is not DISMISSED or PENDING (Finding 4, #1375; widened from CHANGES_REQUESTED-only to any non-DISMISSED/PENDING state by #1045, since a bot reviewer that submits its findings exclusively as COMMENTED — e.g. Pruefer — was otherwise silently unaddressed; see §6.2) — on any stage:<X>:complete item (or fabrik:awaiting-ci item on a wait_for_ci: true stage) — regardless of whether the item has fabrik:yolo, fabrik:cruise, or any auto_advance config, and regardless of whether checkReviewGate currently reports the gate blocked (R1, #1375, amending ADR-1250 — see §6.1.1). Phase 1 runs unconditionally; only Phase 2 (stage advancement) is gated on those labels.
Code path: poll() catch-up loop Phase 1 → buildReviewFeedbackComments() (buildReviewThreadComments() + buildReviewBodyComments()) → cycle limit check → dispatchReviewReinvoke() → async goroutine → processComments() with synthetic comments
Distinct from regular comment processing because:
- Uses synthetic comments derived from PR review threads (
LinkedPRReviewThreadComments) and review bodies (LinkedPRReviews[].Body, keyed on a synthetic"review-body:" + PRReview.DatabaseIDID — REST has no reaction endpoint for a top-level review, so these are reacted to through GraphQLaddReaction(#1953 R8), but that reaction is only the human-visible signal —snap.CommentProcessedplus the durablereview-ids-addressedmarker remain the dedup mechanism, R7/#1555), not issue comments - Has cycle limits (
MaxReviewCycles, default 5) — exceeding pauses the issue viapauseForReviewCycleLimit, the terminal fallback for a verdict that never converges (R5), reached whether the gate cleared naturally or (underreview_authority: authoritative) is still blocked - Has timeout integration (review wait timeout can also trigger pause, but only when there is genuinely nothing actionable to reinvoke on)
- Dispatches asynchronously via goroutine with semaphore slot; the goroutine itself blocks on the slot, never the poll (the main dispatch loop’s acquire is non-blocking — §9.9.1)
- The worker guard (
snap.Worker() != nil) prevents double-dispatch across poll cycles - Resolves review threads (marks them resolved on GitHub) after processing; a review-body comment has no thread to resolve, so only its
CommentProcessedrecord is what prevents reprocessing
Yolo-merge guards against unresolved current-head review threads (#1207). Review Reinvoke’s own dispatch loop protects an item only while it is parked at a completed stage. With fabrik:yolo, Validate completing does not park anything — attemptMergeOnValidate (§2.10’s runValidatePRTerminalAdvance discussion and engine/stages.go) advances out of Validate or enables GitHub auto-merge in the same flow, and native auto-merge then leaves the PR pending for however long CI takes, with GitHub poised to merge the instant checks go green. A review finding arriving inside that window is invisible to both the Review Reinvoke trigger above (nothing is parked to reinvoke) and to §2.16’s SHA-invalidation scan (a review comment does not change the head SHA). Two guards close this, both built on currentHeadReviewThreadComments (which excludes threads GitHub marks isOutdated — superseded by a later push — so a stale-SHA thread never blocks indefinitely) rather than a parallel detection path — note the scope split: guard 1 (attemptMergeOnValidate) is scoped to inline thread comments only, unchanged by #1375; guard 2 (inside handleReviewGate) additionally covers unaddressed review bodies via currentHeadReviewFeedbackComments (currentHeadReviewThreadComments + all review-body comments, R8 — see §6.1.1/§6.2), since that guard sits directly alongside the wider Finding-4 reinvoke check it was already updated for:
-
Guard 1 —
attemptMergeOnValidate(engine/stages.go): before the merge-train/enqueue/enable logic, the function checkscurrentHeadReviewThreadComments(item). If non-empty, it logs"not advancing: N unresolved review thread(s) on <sha>"and returns(false, true, nil)— the new third return value,deferred, distinguishes this from the two pre-existing benign(false, nil)cases (cruise label, no linked PR).handleStageCompletefoldsdeferredintoshouldAdvanceexactly likeautoMergeEnabled. No new retry plumbing is needed: poll.go’s Phase 2 Validate branch already callsattemptMergeOnValidateon every poll whilefabrik:auto-merge-enabledis absent, so once the review-reinvoke loop above resolves the thread, the very next poll proceeds normally.Freshness caveat at the primary call site.
handleStageComplete’s own “Path 1” (§2.6/§2.9 above) is synchronous with the Claude invocation that just completed Validate, and reuses the pre-invocation item snapshot — the same staleness already documented for itswait_for_reviewshandling a few lines below inengine/stages.go, and can be tens of minutes old on a long Validate run, not just a few seconds. A thread posted during that run is invisible to guard 1 at this call site, so Fabrik can proceed to enable auto-merge in this narrow window. This is deliberately not closed by re-fetching insideattemptMergeOnValidate— that would reintroduce the per-completion GraphQL round-trip Path 1 exists to avoid. It is closed instead by guard 2 on the very next poll:fabrik:auto-merge-enabledis now set,poll()’s deep-fetch (selectDeepFetchCandidates) has refreshed the item with currentLinkedPRReviewThreadComments, andhandleReviewGatedisables auto-merge before GitHub can act on the miss. The resulting exposure is the samePollSeconds-bounded residual race already accepted for guard 2’s convergence-window disable (see below), not a new unbounded gap. poll.go’s own Phase 2 Validate retry — the other caller ofattemptMergeOnValidate— always operates on deep-fetched, fresh data and is unaffected by this caveat.Exception: the direct-merge fallback has no convergence window, so it gets a live re-check instead of trusting the caveat above. The freshness caveat’s reasoning (“a miss here is closed by guard 2 on the next poll”) depends on there being a next poll to close it on — true for the native-auto-merge and merge-queue-enqueue paths, which leave the PR pending in a convergence window guard 2 monitors. It does not hold for the direct-merge fallback (taken when
EnablePullRequestAutoMergefails for any reason other thanErrAutoMergeNotEnabled— e.g. the PR is already in CLEAN or UNSTABLE status): that branch callsMergePRsynchronously, in the same invocation, with no window and no later poll for guard 2 to act on. A thread that appeared during the staleness gap above would otherwise merge underneath both guards in one step — identified in review (Pruefer, PR #1211). Immediately before theMergePRcall,attemptMergeOnValidatenow does a raw, uncachedFetchItemDetailsre-read (the same “re-read immediately before an irreversible decision” idiom used bycheckDependencies,recoverMissingPlanComment, andverifyAndHealLinkage) and re-checkscurrentHeadReviewThreadCommentsagainst the fresh data. A blocking thread defers ((false, true, nil), same as guard 1 above); a fetch failure surfaces as an error so the existing retry-next-dispatch path handles it, rather than proceeding to merge on stale data. -
Guard 2 — inside
handleReviewGate, notcheckAutoMergeConvergence(engine/catch_up_handlers.go): the naive placement — insidecheckAutoMergeConvergence(engine/merge_gate.go), the function that polls repeatedly during the auto-merge-enabled window — does not work.catchUpPhase1Handlers(§6.2) isdependencies → reviewGate → autoMergeConvergence → mergeAndCIGates, and each handler claims and stops the chain ontrue. Whenever fresh unresolved feedback (a thread comment or a review body, Finding 4) appears on an item that already carriesfabrik:auto-merge-enabled,handleReviewGate(Handler 2) finds it viabuildReviewFeedbackCommentsand dispatches a reinvoke on that same poll — claiming the item and preventinghandleAutoMergeConvergence(Handler 3) from ever running. A check placed only incheckAutoMergeConvergencewould therefore never fire in exactly the scenario it exists to catch. Guard 2 instead lives insidehandleReviewGateitself: immediately before thedispatchWithCycleLimitcall, if the item carriesfabrik:auto-merge-enabledandcurrentHeadReviewFeedbackComments(item)is non-empty,disableAutoMergeForReviewThreadsfires — logging"disabling auto-merge: N unresolved review thread(s) on <sha>", then callingDequeuePullRequest(merge-queue-enabled PRs) or the newDisablePullRequestAutoMerge(native auto-merge PRs, mirroringreenableAutoMergeAfterRebase’s existing queue-aware branch), and only removingfabrik:auto-merge-enabledon mutation success. Mutation-before-label-removal is deliberate: on partial failure (mutation succeeds, label removal fails) the label stays, so the disable is retried next poll — the alternative order risks Fabrik going label-blind to a PR GitHub might still merge. Removing the label on success stopshandleAutoMergeConvergencefrom running for this item at all (so it cannot fight the ejection-recovery ladder or misread the dequeue as a genuine ejection), and hands re-enablement to guard 1’s existing retry loop once the review-reinvoke dispatch (fired on the same poll, immediately after) resolves the thread — the same remove-then-reapply idiompauseForMergeGroupStallalready uses for the convergence-budget anchor.MaxReviewCyclesis unaffected — the disable is additive to the existing dispatch/cycle-limit/pause path inhandleReviewGate, so the cycle limit still bounds and escalates regardless of which guard triggered the poll.
Both guards are an explicit stopgap for fabrik:yolo specifically (see #1071, which is expected to supersede this mechanism with a decidable “hold a valid signature from every required signer for this SHA” check); fabrik:cruise is unaffected because it already stops at Validate for human merge, well before either guard point.
2.10 CI Check Completed
Trigger: CI check runs on the PR head SHA transition from pending to a terminal state (success, failure, etc.). Fabrik detects this by polling on each catch-up loop iteration — there are no webhooks.
Pending-over-failed precedence (#958): Both settlePRMergeState() and checkCIGate() classify check runs via the shared github.ClassifyCheckRuns() helper, which (1) reduces check runs to the latest run per check name (highest ID wins — a stale completed/failed run left behind by a rerun under a new ID is discarded) and (2) treats any pending (queued/in_progress) run as taking global precedence over any failed run, regardless of whether the failed run is a different check name or a superseded run of the same name. A single failed check-run coexisting with a pending check-run on the same head therefore classifies as WAIT (PRMergeUnsettled / ciBlocked=true, ciFailure=false), never FAIL — the engine waits for the pending check to reach a terminal state instead of dispatching a CI-fix reinvoke. Per ADR-1410, this wait is bounded by liveness (no observed check-run progress for CIWaitTimeout, tracked via LinkedPRState.LastCIProgressAt), not elapsed time — a suite that keeps transitioning check-run state waits indefinitely, up to the separate, much larger CIBackstopTimeout absolute cap. Both call sites use the same helper so they cannot drift out of agreement. See §6.4 rule 19 and §6.5.1 for the full classification rules.
Required-context awareness (ADR-933, #933): ClassifyCheckRuns() alone has no concept of “required” — skipped/neutral/absent check runs fall through to CheckRunsReady, which is correct for a non-required job but wrong for a required one that simply never ran (e.g. a repo whose required signal is a classic commit status posted out-of-band, with GitHub Actions disabled). settlePRMergeState() layers a required-context pre-filter (engine.classifyRequiredContexts() → github.ClassifyRequiredContexts()) in front of every point where check-run classification alone would otherwise resolve to PRMergeReady: the zero-check-runs “no CI configured” fallback (§6.4 rules 13 & 17) and the check-runs-present “all green” fallback (§6.4 rule 19). The pre-filter checks each name in the repo’s configured required_status_contexts (.fabrik/config.yaml, keyed by "owner/repo"; unconfigured repos are always a no-op — zero behavior change) against the union of check-run names and classic commit-status contexts (FetchCombinedStatus, fetched only when a required name isn’t already resolvable from check runs) observed on the exact head SHA. Only a confirmed success counts; a confirmed failure resolves to PRMergeBlocked, and anything else not-yet-confirmed (missing, pending, skipped, neutral) resolves to PRMergeUnsettled — never silently to PRMergeReady. checkCIGate() mirrors this with a matching pre-check, classifyCIFromRequiredContexts(), run ahead of classifyCIFromCheckRuns()/classifyCIFromMergeableState() — necessary because a required-context failure sourced solely from a classic commit status has no check-run footprint for those two functions’ checkRuns-only view to react to. The merge-train poller (pollTrainCI(), engine/merge_train.go — see §1.3’s FR-2 mention of its 30s poll interval) classifies check runs via the same shared github.ClassifyCheckRuns() helper (replacing a previously-separate inline duplicate) and applies the identical required-context pre-filter to its trial-branch SHA. See ADR-933 for the full required-context design.
pollTrainCI’s own mergeable_state shortcut — touched by ADR-1153, #1153. pollTrainCI no longer treats an accepted mergeable_state (clean/unstable) as sufficient for TrainCIGreen by itself. mergeable_state is computed by GitHub from required checks only, so it can read as accepted while a non-required check (e.g. the actual test suite, if left unmarked-required) is still queued/in_progress on the trial SHA — exactly what happened on integration PR #1150, where the train merged 20 seconds after the sole required check went green, 14 seconds before the non-required test job itself reported. The two signals now compose instead of racing:
mergeable_state == "dirty"still returnsTrainCIRedimmediately — unambiguous, unchanged.- An accepted
mergeable_stateis recorded but no longer returns green by itself; it is necessary but not sufficient. - The check-run pass (
gh.ClassifyCheckRuns+ the required-context pre-filter above) is the actual green determinant on the common path: any run stillqueued/in_progresskeeps polling; a confirmed failing run — required or not (Strict policy, ADR-1153) — returnsTrainCIRed; all-clear plus required contexts satisfied returnsTrainCIGreen. - Zero check runs at all (e.g. GitHub Actions disabled — the #933 case): there is no per-check completeness signal to consult, so an accepted
mergeable_statecombined withRequiredContextsSatisfiedis the one remaining basis forTrainCIGreen. A confirmed required-context failure still returnsTrainCIRedimmediately, exactly as before.
Every terminal pollTrainCI decision now logs the specific check runs (name, status/conclusion) it was based on. See ADR-1153 for the full design, the Strict non-required-failure policy rationale, and its relationship to ADR-072 (the single-PR MergePR self-gate, which still accepts unstable and is unaffected) and ADR-933 (whose required-context classifier is reused unmodified).
Suite-aware green (#1822). pollTrainCI’s two green returns (all-green runs + required contexts satisfied; zero runs + accepted mergeable_state) and classifyLandingCI’s two green returns are each gated on ciSuiteHold(e.client, …): an outstanding suite keeps pollTrainCI polling (bounded by CIBackstopTimeout) and makes classifyLandingCI return TrainCIPending; a confirmed red still returns TrainCIRed immediately. See §6.4 Suite-aware completeness, ADR-1822.
CI infrastructure failures (#2052, ADR 2052). pollTrainCI also tells a CI provider failure from a failure of the code under test: on the zero-check-run path it reads the trial SHA’s workflow runs before the zero-check green shortcut above, and a startup_failure (or zero-job failure) run is retriggered by closing and reopening the trial PR (at most maxCIRetriggers = 2, counting only runs absent when the previous retrigger was issued) and then abandoned as TrainCIInfra — never red, never bisected, never charged; the members stay Queued. On a confirmed check-run failure it re-runs the failed jobs once (rerun-failed-jobs) before returning TrainCIRed; only a second failure is red. Full mechanics, the cooldown and the stage-gate counterpart are in §6.29.
pollForMergeable’s identical shortcut — fixed by ADR-1441, #1441 (previously the unfixed sibling ADR-1153 flagged and left open). pollForMergeable (engine/merge_train.go, the merge-train landing-step poller — distinct from pollTrainCI’s CI-poll step above) used to treat an accepted mergeable_state as an unconditional green light for landing, with no check-run fallback at all — not even the partial classification settlePRMergeState had for other values. It now fetches FetchPRDetails (for HeadSHA alongside MergeableState) and classifies via a new classifyLandingCI helper that mirrors pollTrainCI’s composition exactly (same shared primitives — gh.ClassifyCheckRuns, e.classifyRequiredContexts, describeCheckRuns, gh.MergeableStateAccepted — same ordering, same zero-check-runs fallback), reusing TrainCIResult as its verdict type: mergeable_state == "dirty" → immediate reject, no check-run fetch; a confirmed check-run or required-context failure → reject; a check still pending → keep polling (bounded by CIBackstopTimeout, exactly as before); zero check runs → the mergeableAccepted && RequiredContextsSatisfied fallback. classifyLandingCI is a new function, not literally shared code with pollTrainCI (out of scope for #1441, already fixed) — see ADR-1441 for why “one shared mechanism” is satisfied at the primitive level rather than by refactoring pollTrainCI itself.
Code path: poll() catch-up loop Phase 1 → settlePRMergeState() (one combined REST pass for mergeable, mergeable_state, and check runs) → checkCIGate() interprets the PRSettleResult:
PRMergeTerminal(pr.Merged == true):addCompleteLabelAndRemoveCI→ gate clears, advance to Done.PRMergeTerminal(closed, not merged):pauseForPRClosedNotMerged()posts comment naming the PR, addsfabrik:paused+fabrik:awaiting-input, removesfabrik:awaiting-ci,fabrik:awaiting-review, andfabrik:rebase-needed(all three gate labels, not justfabrik:awaiting-ci— ADR-1387 R6:cleanupClosedIssueTransientLabelsno longer sweeps these independently for a closed item at Validate, so this is the only remaining place that clears them on the closed-unmerged/pause path). Returns(false, false, false).PRMergeReadyfrom themergeable_state == "clean"fast path (ADR-033/ADR-1441): gate clears immediately (addCompleteLabelAndRemoveCI); per-check classification is skipped — GitHub has confirmed every check, required or not, passed.mergeable_state == "unstable"is no longer a shortcut (ADR-1441, #1441): it falls through, unmodified, into the same check-run/required-context classification chain described below that"blocked"and every other non-accepted value already use — a confirmed failure on a non-required check now blocks (PRMergeBlocked) instead of clearing the gate.PRMergeReadycan still result forunstablePRs, but only when that classification chain itself finds nothing blocking (e.g. every observed check passed, or only skipped/neutral/cancelled conclusions are present).- A confirmed required-context failure (ADR-933):
classifyCIFromRequiredContexts()returns(true, true, false)unconditionally — a verdict, never a timeout, regardless of how longfabrik:awaiting-cihas been applied (ADR-1410, R3) — evaluated before the check-run/mergeable_state classification below, since the failure may be visible only viasettle.RequiredContextsStatus/RequiredFailed, notsettle.CheckRuns. PRMergeUnsettledorPRMergeBlockedwithCheckRunspopulated:classifyCIFromCheckRuns()classifies viagh.ClassifyCheckRunsfirst (ADR-1410):- Confirmed failure (
CheckRunsFailed): returns(true, true, false)unconditionally — a verdict, never a timeout, regardless of elapsed time (R3, fixing the pre-ADR-1410 defect where a failure past the deadline was misreported as a timeout). Optionally dispatchesdispatchCIFixReinvoke()→ async goroutine →processComments()with synthetic CI failure comment. - Still pending: liveness-stall dwell (R1/R2) — a new
ciProgressStalledSince()helper readsLinkedPRState.LastCIProgressAtfrom theitemstatestore (set whenever a check run’s content actually changes, or the head SHA advances — see §6.14.5); if that timestamp is zero (no progress observed yet this process’s lifetime — cold start) ortime.Since(...) < CIWaitTimeout, returns(true, false, false)(still blocked, re-evaluate next poll); once non-zero and elapsed ≥CIWaitTimeout, removesfabrik:awaiting-ciand returns(false, false, true)(caller callspauseForCITimeout). - R3 (empty
CheckRunsarm,MergeableState == "blocked"):hadChecks == false; checkFetchLabelAppliedAtdwell — unchanged by ADR-1410, since this case has no check-run signal to observe progress on at all; if < CIWaitTimeout → return(true, false, false)(dwell guard, not yet paused); if ≥ CIWaitTimeout →pauseForRequiredNeverRunningCheck()posts distinct comment, addsfabrik:paused+fabrik:awaiting-input, removesfabrik:awaiting-ci - New guard (empty
CheckRunsarm,MergeableState ∉ {"", "unknown"}):hadChecks == false; checksFetchLabelAppliedAtinline — unchanged by ADR-1410, same “no signal to observe” reasoning — iffabrik:awaiting-cihas been present ≥ CIWaitTimeout, removesfabrik:awaiting-ciand returns(false, false, true)(caller callspauseForCITimeout); otherwise returns(true, false, false)—mergeable_stateactively blocking via a signal Fabrik cannot see via check_runs (e.g. a Commit Status / legacy Statuses API); re-evaluate on next poll. - Post-push dwell guard (empty
CheckRunsarm,MergeableState ∈ {"", "unknown"}): whenhadChecks == falseANDlpr.LastHeadSHAUpdateis non-zero ANDtime.Since(lpr.LastHeadSHAUpdate) < PostPushDwell(default 90 s): return(true, false, false)— GitHub has not yet computed mergeability or started CI for the newly-pushed SHA; re-evaluate after the dwell window elapses. ZeroLastHeadSHAUpdate(cold start / post-restart) falls through to R5 immediately (EC-1 preserved).LastHeadSHAUpdateis stamped in the Store’sPRHeadSHAUpdatedhandler only when the SHA actually changes. Configured viaPostPushDwellinengine.Config; set from--post-push-dwellflag (seconds) orFABRIK_POST_PUSH_DWELLenv var; default 90 s. - R5 (fallthrough):
hadChecks == false+MergeableState ∈ {"", "unknown"}+ dwell elapsed (orLastHeadSHAUpdatezero) + required contexts satisfied or unconfigured (ADR-933) →addCompleteLabelAndRemoveCI→ gate clears as “no CI configured”;stage:<X>:completeadded
- Confirmed failure (
Settling primitive consolidation (v0.0.53): settlePRMergeState() reads mergeable, mergeable_state, and check runs in one pass before both the merge-conflict gate and the CI gate. This eliminates the split-brain where two separate REST calls within one poll cycle could observe different GitHub state. See §6.4 for the full specification. MergeableState is intentionally omitted from PRSettleResult in the hadChecks, post-push dwell, and HeadSHA-empty cases so that checkCIGate’s R3 path (MergeableState == "blocked" + empty CheckRuns) cannot misfire on those cases. The mergeable_state field is null on the list endpoint used by FetchLinkedPR, so FetchPRMergeableFields() (a separate single-PR REST call) provides both mergeable and mergeable_state in one request.
(false, false, false) return semantics: checkCIGate returns (false, false, false) for all of the following terminal conditions, which it handles internally (the poll.go call site never sees ciTimedOut=true for these): R1 merged PR, R2 closed-not-merged PR, R3 required-never-running check, no linked PR, all-green checks. The caller only calls pauseForCITimeout when timedOut=true — since ADR-1410, this fires only for a genuine liveness-dwell-exceeded condition (check-run progress stalled, or the R3/mergeable-state elapsed dwell), never for a confirmed CI failure (which returns ciFailure=true instead, unconditionally).
Distinct from Review Reinvoke because:
- Triggered by check run status changes, not reviewer submissions
- Uses
fabrik:awaiting-cilabel (notfabrik:awaiting-review) - Only active on stages with
wait_for_ci: true fabrik:awaiting-ciis applied byhandleStageCompleteon FABRIK_STAGE_COMPLETE (the in-flight CI-await marker, present for both pending and failed checks);stage:X:completeis withheld untilcheckCIGateconfirms CI is green (ADR 032)- Timeout tracked two different ways depending on which dwell applies (ADR-1410): the R3/mergeable-state-blocked dwells and the
settleAwaitingCIScanbackstop use the later ofFetchLabelAppliedAtonfabrik:awaiting-ciand the item’s latestfabrik:pausedremoval (FetchLabelRemovedAt, #2059/#2064 — see §6.14.6), both durable across restarts (see §6.14.5); the check-run-pending liveness-stall dwell uses in-memoryLinkedPRState.LastCIProgressAtinstead (not durable — a restart resets it to “no progress observed yet,” the safe cold-start default, never a blind escalation) - CI-fix cycle counter is
StageState.CIFixCycles[stageName]initemstate.Store(written byCIFixCycleIncrementedmutation; read viasnap.CIFixCycles(stageName))
Single-owner Validate advance, split into an open-item and closed-item pair (runValidatePRTerminalAdvance / settleClosedValidateAdvance, R4 — ADR-056 D2, ADR-1387): Both functions delegate to the same shared per-item logic, advanceValidateTerminalItem, and differ only in what they iterate and where they’re called from. For each candidate item, e.client.FetchLinkedPR is called directly (not e.readClient — boardcache may have stale Merged/State), and the logic runs regardless of which gate label (fabrik:awaiting-ci, fabrik:awaiting-review, fabrik:rebase-needed, or any future label) is present — no disjointness maintained by label negation anywhere. Items already in advancedItems are skipped (no double-advance), and items carrying fabrik:auto-merge-enabled are excluded (that convergence flow is owned exclusively by checkAutoMergeConvergence, Phase 1). When pr.Merged == true: the shared logic iterates all pipeline stages in ascending Order from the stage after the highest already-complete stage up to (but not including) the cleanup-terminal stage, adding stage:<N>:complete for every WaitForCI or WaitForReviews gate-checked stage whose label is absent (with cache write-through; fail-fast on error for idempotent retry). After all completion labels are added, removeAwaitingCILabel, removeAwaitingReviewLabel, and removeRebaseNeededLabel are called as applicable, fabrik:paused + fabrik:awaiting-input are removed, and advanceToNextStage() is called. Immediately after the Done advance, closeIssueIfNonDefaultBase(item, pr.Number) runs (ADR-1096) and explicitly closes the issue when its resolved base:<branch> differs from the repo default — see §2.8. When pr.State == "closed" and !pr.Merged (closed without merging): pauseForPRClosedNotMerged() is called unless the item is already paused (to avoid duplicate comments). Neither function may ever dispatch workers or acquire e.sem (carries ADR-053 constraints; superseded by ADR-057).
runValidatePRTerminalAdvanceis the open-item owner: it iteratesdeepFetchCandidates(unchanged sourcing) and skips anyitem.IsCloseditem — a redundant-but-explicit ownership boundary, since admission (§2.8) no longer lets a closed item reachdeepFetchCandidatesat a gate-checked stage in the first place.settleClosedValidateAdvance(§6.15) is the closed-item owner: it iteratesboard.Itemsdirectly, independent of dispatch admission entirely, so healing a closed item at Validate never depends on that item having been admitted to dispatch.
Note: Non-Validate-stage items with fabrik:paused and a merged PR are not handled by either function; they require manual label intervention to clear fabrik:paused, after which the normal catch-up loop advances them.
2.11 Base Branch Advanced
Trigger: The PR’s base branch moves forward (a different PR merges) while this branch is sitting in the post-stage:Validate:complete catch-up window. GitHub recomputes mergeable on the linked PR; if the new base conflicts with this branch, mergeable transitions from true (or null) to false.
Code path: poll() catch-up loop Phase 1 → settlePRMergeState() (combined REST pass: FetchLinkedPR + FetchPRMergeableFields for mergeable and mergeable_state) → checkMergeabilityGate() interprets PRSettleResult → evaluates the flag → optionally dispatches dispatchRebaseReinvoke() → async goroutine → processComments() with a synthetic rebase-required comment
Distinct from CI Check Completed because:
- Triggered by base-branch movement, not check run status changes
- Uses
fabrik:rebase-neededlabel (notfabrik:awaiting-ci) - Runs before the CI gate in catch-up Phase 1 — a PR that cannot merge has no reason to spin on CI-await
- Only active on stages with
wait_for_ci: true(same opt-in as the CI gate — these are the stages admitted to the catch-up window viafabrik:awaiting-ci) fabrik:rebase-neededis only applied on confirmed conflict (mergeable == false), not onmergeable == null(GitHub still computing)- Rebase cycle counter is
StageState.RebaseCycles[stageName]initemstate.Store(written byRebaseCycleIncrementedmutation; read viasnap.RebaseCycles(stageName)) - Resolution relies on Claude rebasing in the worktree (to handle semantic collisions like duplicated ADR numbers) rather than an engine-side
git rebase
2.12 TurnProgressEvent (TUI Display Event)
Trigger: A {"type":"user"} NDJSON line is written to Claude’s stdout pipe during a Claude invocation. Each logical turn (one user→assistant cycle) begins with exactly one such line (either the initial prompt or a tool-result response), so this fires once per logical turn.
Code path: runClaude() stdout pipe → turnCountingWriter.Write() → detects type == "user" line → increments per-invocation counter → calls claudeTurnProgress(issueNumber, turnsUsed, maxTurns) → Engine.emit(TurnProgressEvent{...}) → TUI channel
Effect: Purely additive display — does not trigger any state transitions, label mutations, or issue processing. The TUI consumes TurnProgressEvent to update the live turn counter shown in:
- The In Progress pane row for the active issue (width-adaptive badge
[N/M turns]/[N/M]/ omitted) - The detail panel for the selected active item (
Turns: N/M)
TurnProgressEvent is only emitted in TUI mode (claudeTurnProgress is nil in plain-text mode and tests). It uses the non-blocking emit path (drop-if-full), so turn-progress updates are best-effort and may be dropped under backpressure. This does not affect engine behavior because the event is display-only; at most one event is produced per logical turn.
MaxTurns in the event carries the effective budget for the current invocation — effectiveBudget as computed in InvokeClaude() (which already accounts for opts.MaxTurnsOverride from the extension loop). This means:
- First invocation without
fabrik:extend-turns:stage.MaxTurns - First invocation with
fabrik:extend-turns:2 × stage.MaxTurns - Extension loop second iteration:
stage.MaxTurns(per-invocation limit, not cumulative)
2.13 Manual Assignee Change
Webhook event: issues.assigned / issues.unassigned
Detection: applyIssuesDelta (boardcache/delta.go:382) applies IssueAssigneesUpdated, which emits the AssigneesChanged flag. The mayNeedWorkObserver and wakeChObserver (engine/observers.go) include AssigneesChanged in wakeChFlags, so the assignment fires both a wake signal and a mayNeedWork cycleSet entry.
Effect: Dispatcher re-evaluates the item on the next poll cycle. The engine does not currently filter dispatch on assignee — assignee changes do not change what work happens, only that the item is re-considered. Future assignee-as-dispatch-filter work (planned, not yet filed) will give this event additional dispatch semantics.
Why: Assignment is a strong “please look at this” signal from the user, and is the mechanical underpinning of multi-user shared boards (each fabrik instance picking up only items assigned to its cfg.User).
2.14 Worker Lifecycle
Source: Engine-internal mutation, not a webhook.
Detection: The dispatch loop (and each reinvoke dispatcher — dispatchReviewReinvoke, dispatchCIFixReinvoke, dispatchRebaseReinvoke) applies WorkerEntered{Repo, Number, StageName, StartedAt} synchronously before the goroutine is launched. WorkerExited{Repo, Number} is deferred at the top of each goroutine so it fires on any exit path (context cancel, ensureRepoReady failure, normal return). Both mutations emit WorkerChanged | WorkerLifecycleChanged; WorkerLifecycleChanged (not the broader WorkerChanged) is the flag in wakeChFlags. WorkerHeartbeat and WorkerPIDSet emit only WorkerChanged and do not wake the poll loop — this prevents deep-fetch churn for active workers (30s heartbeat × N workers would otherwise trigger repeated deep-fetches for items that can’t be dispatched anyway).
Effect: WorkerExited adds the item to mayNeedWork and, when wakeCh != nil, fires the wakeChObserver so the dispatcher re-evaluates within milliseconds. In headless runs (--notui or any configuration without a wake channel), the mutation populates mayNeedWork for the next ticker-driven poll (within PollSeconds); there is no immediate wake. Either way, re-evaluation does not depend on cooldown expiry or external events. This eliminates the previous race where self-advance to the next stage would wait up to 150s (PollSeconds × 10) if the departing worker had not finished cleanup before the post-advance dispatch loop ran.
Dispatch guard: The dispatch loop uses snap.Worker() != nil (Store-backed) instead of the former e.inFlight.Load(iKey) (sync.Map). Because WorkerEntered is applied before wg.Add(1) and before the goroutine starts, snap.Worker() != nil is true from the instant the goroutine is scheduled — there is no window where a new dispatch cycle could race in and double-dispatch the item.
Why: Worker lifecycle is engine state the dispatcher must react to. Pre-Fix B (issue #544), it lived in e.inFlight (sync.Map) outside the Store — a known bypass that violated ADR 036’s single-owner reactive cache invariant. WorkerEntered/WorkerExited complete the migration begun by the Phase 5 F3 store unification.
2.15 Revalidate Label (fabrik:revalidate)
Source: Operator-applied GitHub label.
Detection: A dedicated revalidate-scan loop, settleRevalidateScan (engine/poll_settle.go; called from poll() in engine/poll.go), runs after the paused-item recovery loop and before the dispatch loop. It iterates all deepFetchCandidates unconditionally (paused items included — FR-5). Items that don’t carry fabrik:revalidate are skipped immediately. The loop is reached by stuck-Validate items because fabrik:revalidate is included in the hasAwaitingLabel pre-filter bypass at poll.go:1294, ensuring items with settled updatedAt are still deep-fetched on every poll.
Effect (Validate-stage item, no in-flight worker):
- Removes
stage:Validate:complete,stage:Validate:failed,fabrik:paused,fabrik:awaiting-input,fabrik:awaiting-ci,fabrik:auto-merge-enabledfrom GitHub (best-effort sequential REST calls; 404 responses are treated as already-absent and are not errors). - Removes
fabrik:revalidateitself from GitHub. - For each successful removal:
cacheImpl.ApplyLabelRemoved+webhookMgr.RegisterEcho(cache write-through and echo registration, same pattern as all other label mutations in the engine). - Applies four store resets:
StageRetryCleared,EngineUnpaused,StageLastAttemptCleared,EngineCyclesCleared— all forStageName: "Validate". This is the same full-reset sequence used byclearFailedStage. - Validate dispatches on the next poll cycle: the revalidate scan does not update
deepFetchCandidatesin place. After label removal, the item re-enters the board on the next poll with no blocking labels and noLastAttemptAtcooldown, soitemNeedsWorkreturns true and the dispatch loop invokesprocessItem.
Effect (non-Validate item): Logs a warning, removes only fabrik:revalidate, takes no other action.
In-flight worker guard (FR-4): If snap.Worker() != nil (a Validate invocation is still active), the label is left in place and the revalidate scan defers to the next poll cycle. When the worker exits, WorkerExited populates mayNeedWork and the next poll processes the label normally.
Partial label-removal failure: Each REST call is independent. If a removal fails (non-404 error), the engine logs a warning and continues. On the next poll, the item is still deep-fetched (if fabrik:revalidate was not yet removed), and the revalidate scan retries the remaining removals idempotently.
Closed-issue cleanup: fabrik:revalidate is in transientLifecycleLabels, so cleanupClosedIssueTransientLabels sweeps it from closed issues defensively (FR-7).
Why: Recovery from a stuck-Validate state previously required knowing and clearing 4–5 individual labels. fabrik:revalidate collapses recovery to one action. It stays useful even after structural CI-gate fixes (#855) for operator-initiated re-runs when CI infrastructure has recovered out-of-band, a flaky test was resolved manually, or other non-engine causes.
2.16 SHA-Invalidation Scan
Source: Automatic — triggered by PRHeadSHAUpdated events from any source (webhook, reconcile, deep-fetch probe).
Detection: A SHA-invalidation scan, settleSHAInvalidationScan (engine/poll_settle.go; called from poll() in engine/poll.go), runs immediately after the revalidate scan (§2.15) and before the dispatch loop. It iterates all deepFetchCandidates. Items that don’t carry stage:Validate:complete are skipped immediately. For qualifying items it reads snap.ValidateCompletedSHA() (the HEAD SHA recorded when Validate last completed) and snap.LinkedPR().HeadSHA (the current PR HEAD). A mismatch triggers invalidation unless one of the two exemptions below applies (#2158). Items are admitted to deepFetchCandidates via the standard PRHeadSHAUpdated → LinkedPRChanged → mayNeedWorkObserver → cycleSet observer pipeline — no bypass label is required.
Effect (SHA mismatch, no in-flight worker):
- Removes
fabrik:rebase-needed,fabrik:auto-merge-enabled,fabrik:awaiting-ci,fabrik:awaiting-review,stage:Validate:completefrom GitHub (best-effort sequential REST calls; 404 responses are treated as already-absent and are not errors).fabrik:rebase-neededis included because a conflict determination tied to the invalidated completion SHA no longer describes reality (#1225). - For each successful removal:
cacheImpl.ApplyLabelRemoved+webhookMgr.RegisterEcho(cache write-through and echo registration). - Applies
ValidateCompletedAtSHAClearedto zeroLinkedPR.ValidateCompletedSHAin the Store. - Applies four store resets:
StageRetryCleared,EngineUnpaused,StageLastAttemptCleared,EngineCyclesCleared— all forStageName: "Validate". - Validate dispatches on the next poll cycle: after label removal, the item re-enters the board with no blocking labels and no
LastAttemptAtcooldown, soitemNeedsWorkreturns true and the dispatch loop invokesprocessItem.
Guards:
- FR-5 (empty completion SHA): If
snap.ValidateCompletedSHA() == ""— pre-feature legacy item, or worktree HEAD was unavailable at completion time — the scan skips the item unconditionally. Operators usefabrik:revalidate(§2.15) for legacy items. - FR-6 (in-flight worker): If
snap.Worker() != nil(a Validate invocation is active), the scan defers to the next poll. When the worker exits,WorkerExitedre-populatesmayNeedWorkand the SHA-invalidation scan re-evaluates on the next poll. - Landed or terminal items are exempt (#2158, ADR-2158). After the FR-5/head/FR-6 checks,
validateItemLanded(engine/validate_sha_guard.go) skips the item when any cached fact says it has landed: the issue is closed; its boardStatus(shallow item or store snapshot) names acleanup_worktreestage (Done); its linked PR is merged; or it carriesfabrik:awaiting-landing-verificationor anyfabrik:credited-pr:<N>label. Both the shallow board item and the store snapshot are consulted and ORed, so a disagreement between them resolves to “landed” — the safe polarity for a closed/Done/merged item — and the check needs no live read (the merge train’s singleton catch-up, §6.30, moves the head past the recorded completion SHA and then lands the member’s own PR, so before this exemption the scan strippedstage:Validate:completefrom the just-landed Done item roughly one poll later; a member caught up earlier but landed through a trial PR was stripped the same way (#2114), so the trigger is any catch-up commit on the member’s branch after Validate completion, whichever path lands it, and the exemption is path-agnostic). The skip logs onevalidate-shaline naming the signal, never the “clearing stale completion labels” message, and changes no label or store state. - The engine’s own pure catch-up head is not external (#2158, ADR-2158). For an open, non-landed item whose head differs,
headIsPureCatchUplooks up the catch-up marker for the live head throughcatchUpMarkerCached— the same lookup and trust rules asdropCatchUpFeedback(§6.30): only a comment authored by the engine’s own login counts, an empty login or head fails closed, and theFabrik-Train-Catch-Upcommit trailer is never trusted. The head is exempt only when the marker names exactly the live head and sayspure=true(a conflict-resolved catch-up contains content Validate has not seen, so it is invalidated as usual). On a match the scan removes no label and instead re-records the live head withValidateCompletedAtSHA, so a later real push moves the head again and is still detected (R3); the record is in memory only, like the original SHA, and after a restart FR-5 skips the item exactly as it does today. The push-then-marker window is covered too:trySingletonCatchUpmarks the PR’s catch-up in flight (beginCatchUpPush, in memory) from beforerunCatchUpGitpushes untilpostCatchUpMarkerreturns, and the scan defers the item whilecatchUpPushInFlightis true, so a poll landing between the head move and the marker post cannot read the unmarked head as external (an external push inside that window is detected one poll late). If the marker post itself fails, the head stays unmarked and the next poll after the window invalidates it. A marker read error defers the item to the next poll (logged) rather than invalidating — wrongly stripping completion is the bug being fixed, and a genuine external push is detected at most one poll late. Known limitation: the marker names the head but not its predecessor, so an external push followed by an engine catch-up on the same item is indistinguishable from a catch-up alone and goes undetected. - FR-4 (idempotency): After the first mismatch triggers clearance,
stage:Validate:completeis removed. SubsequentPRHeadSHAUpdatedevents for the same SHA find the label absent — the scan’s first guard (!hasLabel(item.Labels, "stage:Validate:complete")) causes an immediate skip.
Loop prevention: The SHA is recorded AFTER Claude’s final commit and AFTER the worktree HEAD is finalized (git rev-parse HEAD in the worktree). The subsequent PRHeadSHAUpdated event for Validate’s own commit arrives with SHA == ValidateCompletedSHA, so the mismatch check returns false and the scan is a no-op. The loop is architecturally closed. The two exemptions above do not weaken this: a skipped landed item never reaches the label removal, and an accepted catch-up head is re-recorded as the completion SHA, so the next evaluation sees HeadSHA == ValidateCompletedSHA.
Relationship to the comment-rework clear/restore (#1802, ADR-1802): this scan is the direct precedent for #1802’s fabrik:reworking:<Stage>/beginStageRework/endStageRework mechanism — both clear a stage’s stage:<Stage>:complete when it has gone stale and let the normal completion flow re-populate it. The two are independent and never coordinate directly: this scan only ever targets stage:Validate:complete, triggered by a SHA change, outside any comment re-entry; #1802’s mechanism targets whichever stage is being re-entered, triggered by processComments, bracketed by fabrik:editing. Both removal and (re-)addition of stage:Validate:complete are idempotent GitHub label mutations, so the two mechanisms running in the same window — e.g. a SHA change arriving while a comment re-entry is reworking Validate — cannot double-apply or conflict; each just mutates the label set to match its own view of truth.
Partial label-removal failure: If any REST removal fails (non-404 error), hasError is set and the function returns without applying Store mutations. stage:Validate:complete remains; the next poll re-runs the scan and retries idempotently.
Why: stage:Validate:complete validates a SHA, not an issue. A force-push, external commit, or operator rebase after Validate completes produces a new artifact that Fabrik has not validated. Without this scan, attemptMergeOnValidate’s auto-merge enablement would target the new SHA against a gate Fabrik never ran, risking auto-merge of unvalidated code. The scan closes this gap automatically for any SHA change, including ones the operator did not intend as a re-Validate signal.
2.17 Consecutive Resume Failure (Session Abandonment)
Source: Automatic — evaluated inline at the end of every Claude invocation, on both invocation paths: InvokeClaude (the primary stage run) and InvokeClaudeForComments (comment review), and transitively merge_train.go’s conflict-resolution invocation (which also goes through InvokeForComments). Unlike every other event in this section, it is not detected by a settle scan reading board/label state — it is classified structurally, in-process, from the CLI’s own result object.
Detection: interpretClaudeResult (engine/claude.go) is the single funnel all three invocation paths converge on. When an invocation fails for a reason none of the more specific classifiers already claim (stale session, turn-cap error_max_turns, usage-limit blocking_limit, transient api_error), and the failing invocation itself passed a non-empty --resume session ID, classifyResumeFailure reads a plain-text sidecar counter file colocated with the session pointer (<Stage>.session.resumefails, alongside <Stage>.session) and increments it. The counter is durable across an engine restart — deliberately a plain file, not itemstate.Store (confirmed fully in-memory, resets on restart), since every sibling counter’s resting place (SliceRetries, ReviewCycles, etc.) does not need to survive a restart but this one must. The counter is keyed purely to the session-pointer file path, which both InvokeClaude and InvokeClaudeForComments compute identically for a given (issue, stage) — so a failure from either path increments the same counter, and an alternating stage/comment-review failure sequence still reaches the threshold.
Effect (count reaches MaxResumeFailures, default 2):
- Logs the abandonment (
claudeLog, tagresume): session id, stage, consecutive-failure count, threshold, and the last error — the sole record of this event; see Guards below for why no comment or label is used. - Removes the session pointer file — the exact same
os.Removecall the pre-existing stale-session path (§ “the errors[] substringNo conversation found with session ID”) already uses. - Resets the sidecar counter to 0.
- The invocation’s error is wrapped as
*claudeResumeFailureError(Cause,SessionID,ConsecutiveFailures,Threshold,Abandoned: true) and returned to the caller — for every occurrence of this branch, not only the one that crosses the threshold, so a below-threshold failure is also visible tofinalizeStageOutcome’sMaxRetriesexemption (below). - No explicit “force cold start” signal is threaded anywhere:
resolveResumeSessionIDalready treats an absent session file as a cold start identically for bothInvokeClaudeandInvokeClaudeForComments(logs a warning, returns""). The file’s absence at step 2 is the cold-start signal for whichever invocation path runs next — guaranteed sequential, never concurrent, by the existing per-issue single-worker lock.
Effect (below threshold — count incremented but not yet at MaxResumeFailures): Same wrapped *claudeResumeFailureError is returned (with Abandoned: false), but the session pointer is left in place — only the sidecar counter is written. The next invocation still resumes normally; only reaching the threshold triggers abandonment.
Effect (finalizeStageOutcome, engine/item.go, stage path only): resumeFailed := errors.As(err, &resumeFailErr) is computed alongside the pre-existing turnLimited check. Unlike the usage-limit/api_error “did-not-run” family (which early-return before commitWIP/push), a resume failure does not short-circuit — a resumed session can break after real work happened, so commitWIP, the branch push, markCommentsSeenByStage, and InvocationRecorded all still run exactly as for any other incomplete run. StageAttempted is recorded unconditionally for any invocation that ran, satisfying the dispatch-cooldown requirement with no new call. In the final escalation block, resumeFailed is a third branch alongside turnLimited: it skips StageRetryIncremented entirely — for every consecutive resume failure up to MaxResumeFailures, not only the one that triggers abandonment, mirroring the fabrik:claude-limit precedent (ADR-1119). No label, no comment is applied from this branch.
Effect (engine/comments.go, comment-review path): No control-flow change. A resume failure is not excluded from checkCommentBreaker’s “no forward progress” counting the way a usage-limit hit is — it counts toward the comment-processing circuit breaker exactly like *claudeAPIErrorExit already does (ADR-1458), since the breaker is this path’s only bound and a resume failure, unlike an account-wide usage limit, is specific to this issue’s own session.
Guards:
- Reset on evidence of a healthy session: the sidecar counter resets to 0 on (a) a clean process exit (
runErr == nil), (b)FABRIK_STAGE_COMPLETEfound despite a trailing error, and (c) a turn-cap exit (*claudeTurnLimitError) — by construction a turn-cap exit consumed real turns and real cost, the strongest possible evidence the session is not the poisoned-session symptom this mechanism targets. - Untouched on a did-not-run exit: a usage-limit (
blocking_limit) orapi_errorexit neither increments nor resets the counter — the stage never ran, so the outcome says nothing about session health either way. - Untouched on a cold start’s own failure: if the failing invocation’s own
--resumesession ID was empty (this call was itself a cold start), the failure is not attributable to a resumed session — the counter resets to 0 (clearing any stale leftover from a since-abandoned or since-pruned prior session lineage) and the plain, unwrapped error is returned. NormalMaxRetriesaccounting applies to this outcome exactly as before this mechanism existed. - No label, no issue comment: unlike
fabrik:claude-limit(which comments once per episode because the operator can act — wait, or clear the usage-limit hold), this condition is self-healing by construction: its entire purpose is that the next attempt succeeds without intervention. A comment or label here would be indistinguishable from the orphaned-durable-state leak §”fabrik:claude-limit” and ADR-1183 were written to clean up. If the guaranteed cold-start attempt also fails, that outcome has no--resumesession ID attached (see the guard above) and is therefore a genuine, unexempted failure — it flows into the normalMaxRetries→stage:<name>:failed→fabrik:pausedpath, which already comments. - Independent of
MaxRetries, by design: every consecutive resume failure up toMaxResumeFailuresis exempted fromStageRetryIncremented, not only the abandoning one. If only the abandoning failure were exempted, the earlier ones would still burnMaxRetriesbudget and could pause the issue before the cold-start attempt this mechanism exists to produce is ever reached — reproducing, one layer up, the exact indefinite-retry bug this mechanism fixes.
Why: A --resume invocation resumes the same session every time, including when the session’s own transcript (grown too large, or otherwise structurally broken) is the cause of the failure — every retry re-triggers the identical condition. Because InvokeClaude and InvokeClaudeForComments compute the identical session-pointer path for a given (issue, stage), a transcript poisoned by a long stage run equally poisons the next comment-review invocation, and vice versa; the counter and the abandonment must therefore be shared across both paths rather than kept separately, or an alternating failure sequence between the two paths would never reach the threshold. See #1414, ADR-1414.
3. Transition Tables
3.1 Happy Path — Linear Stage Progression
This table shows the normal flow when an issue progresses through the pipeline without interruption.
| Current State | Event | Guard | Resulting State | Labels Added | Labels Removed | Side Effects |
|---|---|---|---|---|---|---|
Column <X>, Idle |
Poll tick | Stage exists, not paused/locked/editing/blocked, cooldown expired | Column <X>, Locked + In Progress |
fabrik:locked:<user>, stage:<X>:in_progress |
Lock-then-verify protocol (2s delay), worktree ensured, Claude invoked | |
Column <X>, Locked + In Progress |
FABRIK_STAGE_COMPLETE | shouldAdvance=false (see below) | Column <X>, Complete |
stage:<X>:complete |
fabrik:locked:<user>, stage:<X>:in_progress, stage:<X>:failed (if present) |
Output posted; draft PR created (if create_draft_pr); PR marked ready (if mark_pr_ready_on_complete); lock released |
Column <X>, Complete |
Human moves to next column | — | Column <Y>, Idle |
Manual board column move | ||
Column <X>, Locked + In Progress |
FABRIK_STAGE_COMPLETE | shouldAdvance=true (see below) | Column <Y>, Idle |
stage:<X>:complete |
fabrik:locked:<user>, stage:<X>:in_progress, stage:<X>:failed (if present) |
Output posted; draft PR / mark ready (if configured); board column updated to next stage; lock released |
Column <X>, Complete |
Poll tick (catch-up) | yolo or cruise active, stage:<X>:complete present, no pending comments |
Column <Y>, Idle |
Board column updated to next stage (Path 2 advancement) |
shouldAdvance resolution (Path 1, handleStageComplete):
yoloActive = cfg.Yolo || hasYoloLabel(item)— re-fetches labels first to pick up mid-run changescruiseActive = !yoloActive && hasCruiseLabel(item)— false when yolo is active. This is not a precedence rule: it only avoids double-counting in step 3, whereyoloActivealready contributes the sametrue. Step 5 keys off the raw label, so cruise still wins at Validate.shouldAdvance = yoloActive || cruiseActive- If
stage.AutoAdvance != nilAND neitherfabrik:yolonorfabrik:cruiselabel is present:shouldAdvance = *stage.AutoAdvance— this meansauto_advance: falsein YAML overridescfg.Yolo(the--yoloflag), but explicit yolo/cruise labels overrideauto_advance: false - If
hasCruiseLabel(item) && stage.Name == "Validate":shouldAdvance = false— cruise stops at Validate. Note the raw label, notcruiseActive: this fires even when yolo is also present, which is what makes cruise win the stop-at-Validate decision.
Catch-up loop shouldAdvance resolution (Path 2): runCatchUpPhase2 calls refreshAutonomyLabels(&item) first (#1769, §6.23) — mirroring Path 1’s step 1 — so this decision also reads live label state, not whatever the item snapshot happened to carry into the catch-up loop. It then checks cfg.Yolo || hasYoloLabel(item) || hasCruiseLabel(item) — items without any of these are skipped entirely. Then: if neither yolo nor cruise LABEL is present and stage.AutoAdvance is explicitly false, the item is skipped. This produces the same behavior as Path 1.
Self-advance wake guarantee (Fix B, #544): When
advanceToNextStageruns, two independent wake events fire: the new column status (LocalStatusUpdated → StatusChanged) and the worker exit (WorkerExited → WorkerLifecycleChanged). Both flags are inwakeChFlags. WhenwakeCh != nil(TUI mode or any wake-channel-enabled setup), thewakeChObserverfires and the dispatcher re-evaluates within milliseconds, without waiting for cooldown expiry. In headless runs (--notuior any configuration without a wake channel), re-evaluation occurs withinPollSecondsvia the next ticker poll. This eliminates the previous race where the departing worker was still alive when the post-advance dispatch loop ran, causing the item to receive a 150sCooldownAt("periodic-re-eval")stamp and wait the full cooldown window before the next stage was dispatched.The same guarantee applies to the Review → Validate catch-up advance: when a review reinvoke worker exits after clearing the review gate,
WorkerExited → WorkerLifecycleChangedwakes the poll loop and adds the item tomayNeedWork. The next poll’s catch-up loop seesstage:Review:completeand advances to Validate (typically within 15s). Pre-Fix B, this transition depended on external event noise — e.g., an unrelatedcheck_runwebhook for a different PR opportunistically waking the dispatcher — becauseCooldownAt("review-blocked")was still active from the gate-waiting period. TheCooldownAt("review-blocked")retry timer (10 × PollSeconds) remains valid for non-responsive bot reviewers where no event fires, but it is no longer the primary re-admission path after the gate clears.
Validate → Done special cases:
| Current State | Event | Guard | Resulting State | Labels Added | Labels Removed | Side Effects |
|---|---|---|---|---|---|---|
| Validate, Locked + In Progress | FABRIK_STAGE_COMPLETE | yolo active | Validate, Complete + Awaiting Merge | stage:Validate:complete, fabrik:auto-merge-enabled |
fabrik:locked:<user>, stage:Validate:in_progress |
GitHub auto-merge enabled via enablePullRequestAutoMerge; board advanced to Done only after GitHub merges the PR (deferred convergence monitor, §5.5). For wait_for_ci: true stages, fabrik:awaiting-ci is added instead and stage:Validate:complete is deferred to checkCIGate (ADR 032). |
| Validate, Complete | Poll tick (catch-up) | yolo active | Done, Pending Cleanup | Convergence monitor (checkAutoMergeConvergence) detects GitHub has merged the PR; board column updated to Done (§5.5) |
||
| Validate, Locked + In Progress | FABRIK_STAGE_COMPLETE | cruise active (no yolo) | Validate, Complete | stage:Validate:complete |
fabrik:locked:<user>, stage:Validate:in_progress |
Cruise stops here — no merge, no advancement to Done |
| Validate, Complete | Poll tick (catch-up) | cruise active (no yolo) | Validate, Complete | Cruise catch-up skips Validate — no merge, no advancement | ||
| Done, Pending Cleanup | Poll tick | Worktree exists on disk | Done, Complete | stage:Done:complete |
Worktree removed from disk |
3.2 Off-Path Transitions
Pause / Unpause
| Current State | Event | Guard | Resulting State | Labels Added | Labels Removed | Side Effects |
|---|---|---|---|---|---|---|
| Any active column, Idle | Human adds fabrik:paused |
— | Same column, Paused | Engine skips item on next poll | ||
| Same column, Paused | Human removes fabrik:paused |
— | Same column, Idle | Engine processes item on next poll | ||
| Same column, Paused | New human comment | Not a bot login (github.IsBotLogin) and created at/after the pause (resumeAuthorised(), #1813) |
Same column, Idle → comment processing | fabrik:paused |
Unpause; clearFailedStage() also called (clears any failed label + resets retries); falls through to processComments(). A bot-authored comment does not match this guard and leaves the item Paused (#1083). |
This table covers
processItem’s own unpause gate — reachable only for a pause that fires before the stage is markedstage:<X>:complete(escalateFailedStage,escalatePRCreationFailure,handleBoundaryViolation,pauseForSliceLimit). A pause that fires after the stage is already complete (the four cycle-limit pause sites: review, CI-fix, rebase, enqueue — see §7.2 “Resumable Engine Pauses”) is resumed by a different mechanism,handleEngineUnpausein the catch-up loop’s Phase 1 handler chain (§6.2, Handler 0) —processItemis never dispatched for a stage-complete item with no new comment, so its gate above is structurally unreachable for those four sites. See ADR-1460.
Lock Conflict (Multi-Instance)
| Current State | Event | Guard | Resulting State | Labels Added | Labels Removed | Side Effects |
|---|---|---|---|---|---|---|
| Any column, Idle | Poll tick (two instances) | Both acquire lock | Depends on tie-break | fabrik:locked:<user> (both) |
Loser’s lock removed | 2s verify delay; lexicographic tie-break: lower username wins, higher username yields |
| Any column, Locked by Other | Poll tick | fabrik:locked:<other> present |
Same (skipped) | itemNeedsWork returns false; processItem also checks and skips |
Dependency Blocking
| Current State | Event | Guard | Resulting State | Labels Added | Labels Removed | Side Effects |
|---|---|---|---|---|---|---|
| Any column, Idle | Poll tick | Open blockers in item.BlockedBy |
Same column, Blocked | fabrik:blocked |
Comment posted listing blockers (first time only); TUI event emitted | |
| Same column, Blocked | Blocker closes (StateChanged) OR dependent’s BlockedBy first populated via deep-fetch (BlockedByChanged) |
All of X’s blockers now CLOSED (store-side view) | Same column, Idle | fabrik:blocked |
Push path: PushUnblockObserver fires immediately; StateChanged scans all items (O(n)); BlockedByChanged checks only the dependent item (O(k blockers)); works for any column including Backlog/Done; no comment posted |
|
| Stage column, Blocked | Poll tick (dep-blocked cooldown retry) | All blockers now CLOSED (dep.State view) |
Same column, Idle | fabrik:blocked |
Pull path (defense-in-depth): processItem → checkDependencies; only for items in stage columns |
Awaiting Input (FABRIK_BLOCKED_ON_INPUT)
| Current State | Event | Guard | Resulting State | Labels Added | Labels Removed | Side Effects |
|---|---|---|---|---|---|---|
Column <X>, Locked + In Progress |
FABRIK_BLOCKED_ON_INPUT | completed false, no error |
Same column, Awaiting Input | fabrik:paused, fabrik:awaiting-input |
fabrik:locked:<user>, stage:<X>:in_progress |
Lock released |
| Same column, Awaiting Input | New human comment | Not a bot login (github.IsBotLogin) and created at/after the pause (resumeAuthorised(), #1813) |
Same column → comment processing | fabrik:paused, fabrik:awaiting-input |
unblockAwaitingInput() clears LastAttemptAt/retry state for the stage, applies EngineCyclesCleared (resets ReviewCycles/ReviewBlockedCycles/CIFixCycles/RebaseCycles/EnqueueCycles/NoOpCommentCycles, §4.7) and resets §4.6’s CommentBreaker — the same full reset clearFailedStage applies for its own pause shape (#1555) — then routes to processComments(). A bot-authored comment does not match this guard and leaves the item Awaiting Input (#1083). |
TUI Manual Stop (s-key)
| Current State | Event | Guard | Resulting State | Labels Added | Labels Removed | Side Effects |
|---|---|---|---|---|---|---|
Column <X>, Locked + In Progress |
User presses s in Active pane, confirms with y |
Job is selected in TUI Active pane | Same column, Awaiting Input | fabrik:paused, fabrik:awaiting-input |
fabrik:locked:<user>, stage:<X>:in_progress |
TUI sends StopRequest on stopCh; handleStopRequest stores "user_stop" in killReasonHolder and calls cancel() on the per-issue context; worker exits via SIGTERM→SIGKILL escalation. stage:<X>:in_progress is now cleared directly by handleStopRequest itself (removeInProgressLabel), not by the worker goroutine’s own release() path (ADR-1393 R2 — closes a pre-existing gap where the label’s removal raced the worker’s exit). Labels + comment are applied through the shared pauseInterruptedIssue primitive (ADR-1393 R3, engine/mutate.go) — the same one the daemon-wide clean stop below uses — which skips re-posting the comment if the store already shows fabrik:paused for this issue (idempotency guard). Resume: user removes fabrik:paused only — fabrik:awaiting-input is auto-cleared on next stage entry by unblockAwaitingInput(). If the worker has already exited when the stop arrives (no entry in issueCtxs), labels and comment are still applied. Any uncommitted worktree changes are committed (chore: partial <Stage> stage progress (incomplete)) and pushed by the worker’s own cancellation path before it exits (ADR-1393 R8) — asynchronous to, and not ordered against, this comment/label write. |
Daemon Clean-Stop Shutdown (SIGINT/SIGTERM)
Generalizes TUI Manual Stop to every issue with a live worker at once, on receipt of the first
SIGINT/SIGTERM (bare-CLI Ctrl-C, or the TUI’s q/Ctrl-C quit path, which self-raises SIGTERM on
its own PID — both routes converge on the same signal handler). See ADR-1393 for the full design
record.
| Current State | Event | Guard | Resulting State | Labels Added | Labels Removed | Side Effects |
|---|---|---|---|---|---|---|
Any column, Locked + In Progress (any issue with an active Worker() in the store) |
First SIGINT/SIGTERM | — | Same column, Awaiting Input | fabrik:paused, fabrik:awaiting-input |
stage:<X>:in_progress |
The signal handler stores "daemon_shutdown" in every live issueCtxs holder, enumerates every store item with Worker() != nil into a snapshot (inFlightSnapshot()), then cancels the root context and launches runShutdownPause(inFlight) (engine/shutdown.go) as one more unit of work on e.wg — in that order. The enumeration happens strictly before cancel(), not inside runShutdownPause’s own goroutine: a worker cancelled before/while starting its Claude subprocess can reach its own cancellation branch and exit fast enough to beat a post-cancel scan, which would otherwise silently drop it from the pause set (review finding). One goroutine per issue in the captured snapshot then, in parallel, clears stage:<Name>:in_progress directly and pauses via the same pauseInterruptedIssue primitive TUI Manual Stop uses (ADR-1393 R2/R3), posting a "paused by a daemon clean stop" audit comment naming the interrupted stage. Idempotent under a retried/duplicate call: if the store already shows fabrik:paused, only the labels are reapplied — no second comment. fabrik:locked:<user> is removed separately by the pre-existing cleanupLockedIssues(), unchanged by this issue. |
— (idle queue: no issue has an active Worker()) |
First SIGINT/SIGTERM | No in-flight workers | — | — | — | runShutdownPause() returns immediately having written nothing — a stop with an idle queue is unchanged in speed and noise from before ADR-1393 (R9). |
| Any | Second SIGINT/SIGTERM during drain | — | Process exits (os.Exit(1)) |
— | — | Force-quit: unconditional, unaffected by the pause-write phase’s progress. cleanupHook runs (releases the terminal), then os.Exit(1) — no label cleanup, no wait. This is the operator’s escape hatch when a clean stop is itself wedged (R5). |
| Any | Drain deadline elapses with no second signal | e.wg (workers + pause-write phase) has not completed within DrainDeadline (default 30s, --drain-deadline/FABRIK_DRAIN_DEADLINE) |
Process exits (Run() returns) |
— | — | waitGroupTimeout (engine/shutdown.go) bounds the same drain that used to be an unbounded e.wg.Wait(); on timeout a warning is logged and drainAndExit proceeds to exit anyway — any still-running pause-write goroutines are abandoned (accepted, not fixed — R5 requires this path to always terminate promptly). |
Any cancellation reaching finalizeStageOutcome’s early-out (ctx.Err() != nil — daemon shutdown,
TUI stop, or any future cancellation reason) now commits and best-effort pushes any uncommitted
worktree changes before releasing the lock (ADR-1393 R8) — see the TUI Manual Stop row above for
what a resumed issue’s worktree looks like. This is a shared code path, not something specific to the
daemon-shutdown case.
Startup recovery (ADR-1393 R7): a stage:<Name>:in_progress label that survives with neither a
fabrik:locked:<any-user> label nor a stage:<Name>:complete/failed sibling — the shape a shutdown
pause write’s own failure, a crash, or a force-quit can still produce despite the direct clear above —
is healed by a third startup pass, runStartupBareInProgressScan (§9.7), alongside the two
pre-existing conditional passes.
Awaiting Review (wait_for_reviews gate)
| Current State | Event | Guard | Resulting State | Labels Added | Labels Removed | Side Effects |
|---|---|---|---|---|---|---|
Column <X>, Locked + In Progress |
FABRIK_STAGE_COMPLETE | wait_for_reviews: true, shouldAdvance |
Same column, Awaiting Review | stage:<X>:complete, fabrik:awaiting-review |
fabrik:locked:<user>, stage:<X>:in_progress |
Path 1: optimistic label application; lock released; returns without advancing |
| Same column, Awaiting Review + Complete | Poll tick (catch-up) | Outstanding reviewers remain, not timed out | Same (blocked) | fabrik:awaiting-review (idempotent) |
checkReviewGate logs pending reviewers | |
| Same column, Awaiting Review + Complete | PR review submitted | All reviewers submitted | Same column, Complete → advance | fabrik:awaiting-review |
Gate cleared; falls through to advance or review reinvoke | |
| Same column, Awaiting Review + Complete | Poll tick (catch-up) | Timeout elapsed | Same column, Awaiting Input | fabrik:paused, fabrik:awaiting-input |
fabrik:awaiting-review |
pauseForReviewTimeout() posts explanatory comment |
Validate column, Awaiting Review + Complete, not fabrik:auto-merge-enabled |
Poll tick | Linked PR is merged (pr.Merged == true) |
Done | stage:<X>:complete (for each gate-checked stage missing the label) |
fabrik:awaiting-review, fabrik:paused, fabrik:awaiting-input (if present) |
R4 (single owner — ADR-056 D2): runValidatePRTerminalAdvance — gate-label agnostic; applies symmetrically to fabrik:awaiting-review items just as to fabrik:awaiting-ci items. See Awaiting CI R4 row below. |
Awaiting CI (wait_for_ci gate)
In the conjunctive gate design (ADR 032), stage:X:complete is withheld until the CI gate actually clears. handleStageComplete adds fabrik:awaiting-ci as the durable in-flight marker; checkCIGate adds stage:X:complete once CI passes.
| Current State | Event | Guard | Resulting State | Labels Added | Labels Removed | Side Effects |
|---|---|---|---|---|---|---|
Column <X>, Locked + In Progress |
FABRIK_STAGE_COMPLETE | wait_for_ci: true |
Same column, Awaiting CI | fabrik:awaiting-ci |
fabrik:locked:<user>, stage:<X>:in_progress |
Conjunctive gate: stage:<X>:complete NOT added here — deferred to checkCIGate when CI passes (ADR 032). fabrik:awaiting-review is NOT seeded here (even when wait_for_reviews: true) — Path 2 (checkReviewGate in catch-up loop) fires after CI clears and stage:X:complete is present, and Path 3 (reviewGateBlocksLanding, §6.6.6) gates the landing decision on the pass CI clears. Dispatcher will not re-invoke while fabrik:awaiting-ci is present (itemNeedsWork returns false for R3). |
| Same column, Awaiting CI | Poll tick (catch-up) | Linked PR is merged (pr.Merged == true) |
Same column, Complete → advance | stage:<X>:complete |
fabrik:awaiting-ci |
R1: checkCIGate detects merged PR before fetching check runs. addCompleteLabelAndRemoveCI runs; falls through to advance. No manual intervention required. |
| Same column, Awaiting CI | Poll tick (catch-up) | Linked PR is closed without merging (pr.State == "closed" and pr.Merged == false) |
Same column, Awaiting Input | fabrik:paused, fabrik:awaiting-input |
fabrik:awaiting-ci, fabrik:awaiting-review, fabrik:rebase-needed (whichever present — ADR-1387 R6) |
R2: pauseForPRClosedNotMerged() posts comment naming the PR number. Human must reopen or create a new PR and remove fabrik:paused to resume. stage:<X>:complete is NOT added. checkCIGate returns terminated=true for this branch (ADR-1223), so handleMergeAndCIGates claims the item — Phase 2 does not run for it in the same poll pass. |
| Same column, Awaiting CI | Poll tick (catch-up) | CI checks still pending, or a failed check-run coexists with a pending check-run on the same head (#958 — pending always wins via github.ClassifyCheckRuns) |
Same (blocked) | (none — fabrik:awaiting-ci already present) |
checkCIGate logs pending checks; re-evaluates next poll; no CI-fix dispatch, no CIFixCycles increment. Per ADR-1410, this state’s own liveness-stall dwell (row below) is anchored on LinkedPRState.LastCIProgressAt, stamped every time this row’s own re-observation actually changes check-run content. |
|
| Same column, Awaiting CI | Poll tick (catch-up) | CI check(s) failed and nothing is pending on the current head | Same column, Awaiting CI (failure confirmed) | fabrik:awaiting-ci (idempotent) |
CI failure detected; dispatch CI-fix reinvoke or pause on cycle limit (unless the current head already has a recorded CI-fix no-op — see §6.5.2). Per ADR-1410 (R3), this is a verdict and fires unconditionally, regardless of how long fabrik:awaiting-ci has been applied — never a timeout. |
|
| Same column, Awaiting CI | Poll tick (catch-up) | All CI checks green (or no CI configured — R5: mergeableState ∈ {"", "unknown"} + no check runs + hadChecks == false + dwell elapsed or zero LastHeadSHAUpdate) |
Same column, Complete → advance | stage:<X>:complete |
fabrik:awaiting-ci |
Gate cleared; checkCIGate adds stage:<X>:complete and removes fabrik:awaiting-ci; falls through to advance (or merge for Validate+yolo) |
| Same column, Awaiting CI | Poll tick (catch-up) | mergeableState ∈ {"", "unknown"} + no check runs + hadChecks == false + LastHeadSHAUpdate non-zero + time.Since(LastHeadSHAUpdate) < PostPushDwell |
Same (blocked, no label) | Post-push dwell guard: GitHub has not yet computed mergeability or started CI for the newly-pushed SHA. Returns (true, false, false); re-evaluates on next poll. Zero LastHeadSHAUpdate (cold start / post-restart) falls through to R5 immediately (EC-1 preserved). |
||
| Same column, Awaiting CI | Poll tick (catch-up) | mergeable_state == "clean" (v0.0.52 shortcut, narrowed to clean-only by ADR-1441/#1441) |
Same column, Complete → advance | stage:<X>:complete |
fabrik:awaiting-ci |
Gate cleared via mergeable_state shortcut before the per-check classification runs — GitHub has confirmed every check, required or not, passed. addCompleteLabelAndRemoveCI runs; falls through to advance. mergeable_state == "unstable" no longer takes this row (ADR-1441): it falls through to the check-run/required-context classification rows above (740–744) instead — a confirmed failure on a non-required check now blocks via row 743, not clears via this row. |
| Same column, Awaiting CI | Poll tick (catch-up) | mergeable_state == "blocked" + no check runs + hadChecks == false + fabrik:awaiting-ci applied < CIWaitTimeout ago |
Same (blocked, no label) | R3 dwell guard: first-push window; checks may not have registered yet. Re-evaluated on next poll. | ||
| Same column, Awaiting CI | Poll tick (catch-up) | mergeable_state == "blocked" + no check runs + hadChecks == false + fabrik:awaiting-ci applied ≥ CIWaitTimeout ago |
Same column, Awaiting Input | fabrik:paused, fabrik:awaiting-input |
fabrik:awaiting-ci |
R3: pauseForRequiredNeverRunningCheck() posts distinct comment explaining the PR is blocked but no check runs were ever observed — required check likely converted to workflow_dispatch but still required by branch protection. Human must run the check manually or remove from branch protection. classifyCIFromMergeableState returns terminated=true for this branch (ADR-1223), so handleMergeAndCIGates claims the item — Phase 2 does not run for it in the same poll pass. |
| Same column, Awaiting CI | Poll tick (catch-up) | mergeableState ∉ {"", "unknown"} (e.g. behind, dirty, draft, has_hooks) + no check runs + hadChecks == false + fabrik:awaiting-ci applied < CIWaitTimeout ago |
Same (blocked, no label) | New guard (dwell not elapsed): mergeable_state actively blocking but no check_runs visible; returns (true, false, false); re-evaluates on next poll. |
||
| Same column, Awaiting CI | Poll tick (catch-up) | mergeableState ∉ {"", "unknown"} (e.g. behind, dirty, draft, has_hooks) + no check runs + hadChecks == false + fabrik:awaiting-ci applied ≥ CIWaitTimeout ago |
Same column, Awaiting Input | fabrik:paused, fabrik:awaiting-input |
fabrik:awaiting-ci |
New guard (timeout elapsed): removes fabrik:awaiting-ci, returns (false, false, true); caller calls pauseForCITimeout() which posts explanatory comment and adds fabrik:paused + fabrik:awaiting-input. |
| Same column, Awaiting CI | Poll tick (catch-up) | Checks still pending, and LinkedPRState.LastCIProgressAt is non-zero (progress observed at least once this process’s lifetime) with time.Since(LastCIProgressAt) ≥ CIWaitTimeout (ADR-1410) |
Same column, Awaiting Input | fabrik:paused, fabrik:awaiting-input |
fabrik:awaiting-ci |
classifyCIFromCheckRuns returns (false, false, true); caller calls pauseForCITimeout(), which posts explanatory comment. Liveness-stall dwell, not elapsed time — a LastCIProgressAt of zero (cold start / no progress observed yet) never escalates, only re-observes; see §6.14.5. |
| Same column, Awaiting CI (any classification) | Poll tick (settleAwaitingCIScan’s unconditional backstop) |
fabrik:awaiting-ci applied ≥ CIBackstopTimeout ago (default 4h) and the item not resumed from a pause within that window, not on its first evaluation since daemon start, and with no worker in flight (#2059, §6.14.6), independent of what any classifier would otherwise decide (ADR-1410, R5) |
Same column, Awaiting Input | fabrik:paused, fabrik:awaiting-input |
fabrik:awaiting-ci |
pauseForCITimeout() posts explanatory comment; bounds per-poll cost regardless of CI duration — see §6.14.5. Unlike the row above, elapsed-time-based via FetchLabelAppliedAt, and fires even when a silent claim earlier in the handler chain would otherwise make checkCIGate unreachable (§6.14.1). |
Validate column, any gate label (or none), paused or not + not fabrik:auto-merge-enabled |
Poll tick | Linked PR is merged (pr.Merged == true) |
Done | stage:<X>:complete (for each gate-checked stage missing the label) |
gate labels (fabrik:awaiting-ci, fabrik:awaiting-review, fabrik:rebase-needed), fabrik:paused, fabrik:awaiting-input |
R4 (single owner — ADR-056 D2): runValidatePRTerminalAdvance detects merged PR via direct e.client.FetchLinkedPR (not e.readClient). Iterates all pipeline stages in ascending Order from the highest already-complete stage onward, adding stage:<N>:complete for every WaitForCI or WaitForReviews gate-checked stage whose label is absent. Clears all gate labels + removes pause labels + advances. Gate-label agnostic: runs regardless of which gate label (fabrik:awaiting-ci, fabrik:awaiting-review, fabrik:rebase-needed, or any future label) is present. See ADR-057. |
R4b — Convergence-paused recovery: Items paused by
pauseForConvergenceFailed(convergence budget exhausted) carryfabrik:paused + stage:Validate:completebut hold no gate label (fabrik:awaiting-ciorfabrik:awaiting-review). The R4 row above admits these items via its “any gate label (or none), paused or not” guard. When their linked PR merges externally after the convergence pause,runValidatePRTerminalAdvancedetects the merge, removesfabrik:paused+fabrik:awaiting-input, and advances to Done — no manual unpause required. This is distinct from the active convergence monitor path (checkAutoMergeConvergence), which is excluded from R4 by thefabrik:auto-merge-enabledguard:pauseForConvergenceFailedremovesfabrik:auto-merge-enabledbefore pausing, so R4 correctly picks up the item on the next poll after the external merge.
Merge-conflict gate (wait_for_ci: true only; runs before the CI gate):
| Current State | Event | Guard | Resulting State | Labels Added | Labels Removed | Side Effects |
|---|---|---|---|---|---|---|
| Same column, Awaiting CI | Poll tick (catch-up) | mergeable == false on linked PR |
Same column, Rebase Needed | fabrik:rebase-needed |
Dispatch rebase reinvoke or pause on cycle limit | |
| Same column, Rebase Needed (Awaiting CI + rebase-needed) | Poll tick (catch-up) | mergeable == true on linked PR (Claude’s rebase push landed) |
Same column, Awaiting CI → (CI gate evaluates next) | fabrik:rebase-needed |
Gate cleared; catch-up falls through to the CI gate on the same poll | |
| Same column, Awaiting CI | Poll tick (catch-up) | mergeable == null (GitHub still computing) |
Same (blocked, no label) | Re-evaluated on next poll; no label churn for transient unknown state | ||
| Same column, Rebase Needed | Poll tick (catch-up) | snap.RebaseCycles(stageName) ≥ MaxRebaseCycles |
Same column, Awaiting Input | fabrik:paused, fabrik:awaiting-input |
pauseForRebaseCycleLimit() posts explanatory comment; fabrik:rebase-needed is left in place so the human can see why Fabrik stopped |
Cooldown Retry and Failed Stage Escalation
| Current State | Event | Guard | Resulting State | Labels Added | Labels Removed | Side Effects |
|---|---|---|---|---|---|---|
Column <X>, Locked + In Progress |
max_wall_time exceeded |
SIGTERM→10s→SIGKILL; FABRIK_STAGE_COMPLETE found in buffered assistant stream |
Same column, Complete | stage:<X>:complete |
fabrik:locked:<user>, stage:<X>:in_progress |
extractTextFromAssistantTurns() recovers marker; same completion flow as live FABRIK_STAGE_COMPLETE |
Column <X>, Locked + In Progress |
max_wall_time exceeded |
SIGTERM→10s→SIGKILL; no FABRIK_STAGE_COMPLETE in buffered stream |
Same column, Cooldown | wasTimedOut=true; routes to cooldown/retry (not a hard error); lock NOT released |
||
Column <X>, Locked + In Progress |
Inactivity timeout (15m) | No streamed output for 15 consecutive minutes; FABRIK_STAGE_COMPLETE found in buffered stream |
Same column, Complete | stage:<X>:complete |
fabrik:locked:<user>, stage:<X>:in_progress |
Same completion flow |
Column <X>, Locked + In Progress |
Inactivity timeout (15m) | No streamed output for 15 consecutive minutes; no FABRIK_STAGE_COMPLETE in buffered stream |
Same column, Cooldown | wasTimedOut=true; routes to cooldown/retry; lock NOT released |
||
Column <X>, Locked + In Progress |
No marker in output | claudeRan is true (includes both error-free runs and runs that errored mid-execution; excludes only start failures like binary-not-found) |
Same column, Cooldown | CooldownAt("periodic-re-eval") recorded (via CooldownRecorded); cooldown = PollSeconds * 10; lock NOT released (stays locked through retries) |
||
Column <X>, Locked + In Progress |
FABRIK_STAGE_COMPLETE present, but stripped output is degenerate (bare @file/absolute-path reference, §2.6) |
isDegenerateOutput(postOutput) is true |
Same column, Cooldown | Output not posted; completed forced to false before the marker-dispatch branches run; treated identically to “no marker in output” for retry purposes; first detection posts an immediate explanatory comment |
||
Column <X>, Locked + In Progress |
FABRIK_STAGE_COMPLETE present, but no artifact harvestable anywhere (§2.6, #1782) |
artifactMissingOnComplete(text) is true inside interpretClaudeResult() (upstream of this row, before finalizeStageOutcome() runs) |
Same column, Cooldown | completed already false on arrival; treated identically to “no marker in output” for retry purposes; first detection posts an immediate “no artifact harvested” comment; excluded when FABRIK_NO_WORK_NEEDED co-occurs |
||
| Same column, Cooldown | Poll tick | Cooldown expired | Same column, Locked + In Progress (retry) | stage:<X>:failed (if present from prior escalation) |
Claude re-invoked with resume=true |
|
| Same column, Cooldown | Retry count ≥ MaxRetries | claudeRan && !turnLimited && MaxRetries > 0 (genuine error, or a clean run that never emitted FABRIK_STAGE_COMPLETE — see §7.12 for why the latter counts here rather than as a slice) |
Same column, Paused + Failed | fabrik:paused, stage:<X>:failed |
fabrik:locked:<user>, stage:<X>:in_progress |
escalateFailedStage() posts comment; lock released; Attempts incremented via StageRetryIncremented (#1199 — turnLimited outcomes never reach this row; see the row below) |
| Same column, Cooldown | Slice count ≥ MaxSliceRetries | claudeRan && turnLimited && MaxSliceRetries > 0 |
Same column, Paused + Awaiting Input | fabrik:paused, fabrik:awaiting-input |
fabrik:locked:<user>, stage:<X>:in_progress |
pauseForSliceLimit() posts a slice-budget comment (never “failed”); SliceRetries incremented via SliceRetryIncremented; no stage:<X>:failed — the stage has not failed (#1199, §7.12) |
| Same column, Paused + Failed | Human removes fabrik:paused |
stage:<X>:failed present OR snap.PausedByEngine(stageName) |
Same column, Idle | stage:<X>:failed |
clearFailedStage() applies StageRetryCleared (also zeroes SliceRetries — the two counters share one reset point), EngineUnpaused, StageLastAttemptCleared, EngineCyclesCleared |
In-flight items and cooldown (#544):
CooldownAt("periodic-re-eval")is stamped for in-flight items (those where a prior worker is still running at end-of-poll). Stamping is intentional: without it, once a prior expired cooldown ages out, the item would be re-admitted to the deep-fetch path on every poll cycle until the worker exits — causing repeated unnecessary deep-fetch evaluation work (and the fallback GraphQL fetch when the cache is invalidated or disabled) for items that can’t be dispatched anyway (snap.Worker() != nilblocks them). The prompt re-dispatch after the worker finishes is guaranteed byWorkerExited → WorkerLifecycleChanged, which is inwakeChFlagsand wakes the poll loop immediately. Note:WorkerLifecycleChangedis excluded fromcycleSetFlags(Fix B, issue #576), so it does not bypass the cooldown gate viamayNeedWork— but the cooldown is already expired (or was already bypassed by an entry in cycleSet from earlier in the worker’s lifecycle). This note concerns items blocked by thesnap.Worker() != nilguard; an item deferred because every worker slot was busy is handled differently (no stamp, re-queued — §9.9.1). See §2.14, §9.2, and §9.9.
Turn Limit Extension
The turn budget is a preemption mechanism, not a failure threshold. It exists for two reasons, neither of which is “the work went wrong”:
- Runaway guard — bound an unbounded Claude loop.
- Time-slicer — chop a large job so a single issue cannot starve every other issue of workers.
When the budget is exhausted, the engine commits and pushes partial work, and the next dispatch resumes the same Claude session (
--resume) against the same worktree. Work continues from where it stopped; nothing is discarded and nothing restarts. A capped invocation is a slice boundary, and a job needing several slices is normal — cost typically rises across slices as each resumed session carries more accumulated context.Treat “turn limit hit” throughout this section as this slice ended, never as this stage failed. The distinction is load-bearing: the CLI reports it structurally as
subtype: "error_max_turns"/terminal_reason: "max_turns"(captured since #1178), separate from a genuine fault, and the TUI renders it as↻ (turn limit)rather than✗ (error).Retry accounting matches this model (#1199). A turn-capped slice calls
SliceRetryIncremented, notStageRetryIncremented— it is bounded by its own counter,SliceRetries/MaxSliceRetries(default 10), never byMaxRetries(default 3, unchanged). A job needing more slices thanmax_retriescompletes without pausing, as long as it stays withinMaxSliceRetries. See §7.12 for the full detection/handling/recovery treatment, and #1191 for the operator-facing report this resolves.
When Claude exits a stage invocation due to max_turns (i.e., the per-invocation turn usage satisfies invUsage.TurnsUsed >= currentBudget and !completed && err == nil), the engine evaluates whether to extend before entering the cooldown/retry path.
Extension trigger condition: !completed && err == nil && stage.MaxTurns > 0 && invUsage.TurnsUsed >= currentBudget
Hard cap: 3× stage.MaxTurns total across all invocations. When totalMultiple >= 3, no further extension is attempted.
Per-stage progress signals:
| Stage | Progress Signal | API Cost |
|---|---|---|
| Implement | New git commit (HEAD SHA changed) OR (baseline was clean AND working tree is now dirty — uncommitted file edits by Claude) | Zero — local git only |
| Review | New git commit OR LinkedPRResolvedThreadCount increased |
One FetchItemDetails GraphQL call (only if no new commit) |
| Validate | Total comment count on issue/PR increased | One FetchItemDetails GraphQL call |
| All others (Research, Specify, Plan, custom) | No signal — always fail on turn-limit | None |
The “baseline clean AND working tree dirty” guard for Implement prevents a pre-existing dirty worktree (e.g. from a prior interrupted session) from counting as progress. Only new uncommitted changes made during the invocation trigger extension.
Extension loop behavior (within a single processItem call — no poll-cycle gap):
- At invocation start, a
progressBaselineis snapshotted: git HEAD SHA (Implement, Review), working-tree dirty state (Implement), comment count (Validate), and resolved thread count (Review). - Claude is invoked with the current budget.
- If the turn limit is hit AND
totalMultiple < 3: calldetectProgress. If progress →totalMultiple++, re-invoke with--resume. If no progress or progress check fails → proceed to cooldown/retry as today. - Output is accumulated across all invocations before posting as a single stage comment.
- WIP commit and push are deferred to after the loop.
fabrik:extend-turns label: When present at invocation start, the first invocation receives 2 × stage.MaxTurns as its budget (pre-granted extension, no progress check required for the first turn-limit hit). That same first invocation also gets max_wall_time scaled by the identical 2× factor (scaledWallTime in engine/claude.go, §7.7, ADR-1206), so the extra turn budget has proportionate wall-clock headroom instead of being killed on a deadline sized for the un-extended case. Subsequent extensions beyond 2× still require the progress check, reset to the un-multiplied stage.MaxTurns budget, and — since each such extension re-invokes with --resume and thus gets its own fresh runClaude call — run under the unscaled max_wall_time (their own correctly-sized window starting at that call’s own spawn time, not a further-inflated one). The label persists across all intermediate stages — it is not removed on stage completion. It is removed only in the cleanup (Done) stage branch of processItem, after the stage:Done:complete label is added. ErrNotFound on removal is treated as success (user removed it manually). The label is a no-op when stage.MaxTurns == 0 (which also means no wall-time scaling occurs — scaledWallTime cannot scale against a zero baseline).
Log tag: [#N extend-turns] — emitted on every detectProgress call (pass or fail), reporting the evaluated signals and has_progress=true/false. When extension is granted, an additional line logs the new budget multiple and cumulative turns used.
| Current State | Event | Guard | Resulting State | Labels Added | Labels Removed | Side Effects |
|---|---|---|---|---|---|---|
Column <X>, Locked + In Progress |
Turn limit hit | totalMultiple < 3; progress detected |
Same column, Locked + In Progress (extension) | totalMultiple++; resume=true; output accumulated; no WIP commit or push between extensions |
||
Column <X>, Locked + In Progress |
Turn limit hit | totalMultiple >= 3 (hard cap) |
Same column, Cooldown | Hard cap reached; slice ends (a preemption, not a failure — the session resumes on the next dispatch). Cooldown routes this to the slice counter (SliceRetries/MaxSliceRetries), never MaxRetries — see the “Slice count ≥ MaxSliceRetries” row above and §7.12 (#1199). CooldownAt("periodic-re-eval") recorded; WIP commit + push (skipped for read_only stages; a persist_spec Specify stage has already committed its spec, §7.16) |
||
Column <X>, Locked + In Progress |
Turn limit hit | No progress detected or progress check failed | Same column, Cooldown | No extension; slice ends (a preemption, not a failure). Same as the row above: Cooldown routes this to the slice counter, never MaxRetries (#1199, §7.12). CooldownAt("periodic-re-eval") recorded; WIP commit + push (skipped for read_only stages; a persist_spec Specify stage has already committed its spec, §7.16) |
||
Column <X>, Locked + In Progress |
FABRIK_STAGE_COMPLETE (any extension) | completed = true |
Same column, Complete | stage:<X>:complete |
fabrik:locked:<user>, stage:<X>:in_progress |
Normal completion flow; extend-turns label persists to next stage |
Cleanup Stage
| Current State | Event | Guard | Resulting State | Labels Added | Labels Removed | Side Effects |
|---|---|---|---|---|---|---|
| Done, Pending Cleanup | Poll tick | Worktree exists, not paused, not already complete | Done, Complete | stage:Done:complete |
fabrik:extend-turns (if present) |
Worktree removed; no lock/Claude/comment processing |
| Done, Complete | Poll tick | Already complete | Done, Complete (no-op) | Skipped by both itemMayNeedWork and processItem |
Review Reinvoke
| Current State | Event | Guard | Resulting State | Labels Added | Labels Removed | Side Effects |
|---|---|---|---|---|---|---|
Column <X>, Awaiting Review + Complete |
Review gate clears + unresolved thread comments | snap.Worker() == nil, cycle count < MaxReviewCycles |
Same column (comment processing via async goroutine) | fabrik:editing (during processing) |
dispatchReviewReinvoke() spawns goroutine; ReviewCycleIncremented applied; WorkerEntered applied; semaphore acquired |
|
Column <X>, Awaiting Review + Complete |
Review gate clears + unresolved thread comments | Cycle count ≥ MaxReviewCycles | Same column, Awaiting Input | fabrik:paused, fabrik:awaiting-input |
pauseForReviewCycleLimit() posts comment |
|
Column <X>, Awaiting Review + Complete |
Review gate clears + unresolved thread comments | snap.Worker() != nil |
Same (skipped) | Previous reinvoke goroutine still running; skipped entirely (no cycle-limit check) |
CI Fix Reinvoke
| Current State | Event | Guard | Resulting State | Labels Added | Labels Removed | Side Effects |
|---|---|---|---|---|---|---|
Column <X>, Awaiting CI |
Poll tick (catch-up) | CI failed; snap.Worker() == nil; snap.LastCIFixNoOpSHA() != settle.PR.HeadSHA; snap.CIFixCycles(stageName) < MaxCiFixCycles |
Same column (CI-fix goroutine running) | fabrik:editing (during processing) |
dispatchCIFixReinvoke() spawns goroutine; CIFixCycleIncremented applied; WorkerEntered applied; semaphore acquired; synthetic CI-fix comment passed to processComments() |
|
Column <X>, Awaiting CI |
Poll tick (catch-up) | CI failed; snap.LastCIFixNoOpSHA() == settle.PR.HeadSHA (#958 leg 2) |
Same (skipped) | A prior CI-fix reinvoke for this exact head SHA already observed no new commit pushed; dispatching again would repeat the same no-op. No dispatch, no CIFixCycles increment. CIBackstopTimeout remains the backstop if CI never resolves on this SHA (ADR-1410 — CI is a confirmed failure here, so the liveness-stall dwell does not apply; only the unconditional per-poll-cost backstop bounds this); the guard is implicitly cleared once the head SHA advances. |
||
Column <X>, Awaiting CI |
Poll tick (catch-up) | CI failed; snap.CIFixCycles(stageName) ≥ MaxCiFixCycles |
Same column, Awaiting Input | fabrik:paused, fabrik:awaiting-input |
pauseForCIFixCycleLimit() posts explanatory comment |
|
Column <X>, Awaiting CI |
Poll tick (catch-up) | CI failed; snap.Worker() != nil |
Same (skipped) | Previous CI-fix goroutine still running; skipped entirely |
Rebase Reinvoke
| Current State | Event | Guard | Resulting State | Labels Added | Labels Removed | Side Effects |
|---|---|---|---|---|---|---|
Column <X>, Rebase Needed + Complete |
Poll tick (catch-up) | mergeable == false; snap.Worker() == nil; snap.RebaseCycles(stageName) < MaxRebaseCycles |
Same column (rebase goroutine running) | fabrik:editing (during processing) |
dispatchRebaseReinvoke() spawns goroutine; RebaseCycleIncremented applied; WorkerEntered applied; semaphore acquired; synthetic rebase-required comment passed to processComments() |
|
Column <X>, Rebase Needed + Complete |
Poll tick (catch-up) | mergeable == false; snap.RebaseCycles(stageName) ≥ MaxRebaseCycles |
Same column, Awaiting Input | fabrik:paused, fabrik:awaiting-input |
pauseForRebaseCycleLimit() posts explanatory comment (usually signals a semantic conflict needing human judgment) |
|
Column <X>, Rebase Needed + Complete |
Poll tick (catch-up) | mergeable == false; snap.Worker() != nil |
Same (skipped) | Previous rebase goroutine still running; skipped entirely |
3.3 Comment Processing During Non-Pause Gate States
When a new user comment arrives while a non-pause gate label (fabrik:awaiting-ci, fabrik:awaiting-review, fabrik:rebase-needed, or fabrik:auto-merge-enabled) is active, the engine dispatches comment processing before evaluating the gate. This behavior is intentional: item.go checks for new comments (len(newComments) > 0 → processComments()) before reaching any gate check, and itemNeedsWork guard 7 admits any item with new comments regardless of gate labels.
| Current State | Event | Race Resolution | Resulting State | Notes |
|---|---|---|---|---|
Column <X>, Awaiting CI |
New user comment arrives | snap.Worker() != nil guard in catch-up loop reinvoke dispatch: if a comment worker is in-flight when the catch-up loop runs, the reinvoke (CI-fix, rebase, etc.) is skipped for that poll cycle |
Comment processing runs; catch-up loop reinvoke skipped until worker exits | Intended: user may need to override CI behavior mid-gate. The next poll after the comment worker exits will re-evaluate the CI gate. |
Column <X>, Awaiting Review |
New user comment arrives | Same worker guard: review reinvoke skipped while comment worker is in-flight | Comment processing runs; review gate re-evaluated next poll | Intended: user may need to provide direction to Claude while awaiting review. |
Column <X>, Rebase Needed |
New user comment arrives | Same worker guard: rebase reinvoke skipped while comment worker is in-flight | Comment processing runs; rebase gate re-evaluated next poll | Intended: user may provide rebase guidance while gate is active. |
Column <X>, Auto-Merge Enabled |
New user comment arrives | Same worker guard | Comment processing runs; convergence monitor re-evaluated next poll | Intended: user may need to intervene while convergence is in progress. |
Known limitation: If both the catch-up loop and a comment arrive at the same poll cycle before any worker is recorded, both dispatches can fire in the same cycle. The advancedItems map prevents double-advancement within a single poll cycle; the fabrik:editing label added by the comment goroutine at entry prevents a second goroutine from launching immediately. This is not structurally exclusive — it is a best-effort guard.
3.4 Operator-Level and System-Level Transitions
These transitions are driven by operator actions or background engine scans rather than stage-lifecycle events. They do not fit the “stage-running → stage-complete” grammar of §3.1/§3.2.
| Event Source | Event | Current State | Effect | Labels Added | Labels Removed | Notes |
|---|---|---|---|---|---|---|
| User (manual) | Assignee change | Any column | Re-evaluated on next poll; AssigneesChanged fires wakeChObserver |
(none) | (none) | No label mutation. Assignee is read by logf for display only; it does not gate dispatch. See §2.13. |
| User (manual) | fabrik:revalidate applied |
Validate-column, any state | Removes gate/completion labels; re-dispatches Validate on next poll | (none) | stage:Validate:complete, stage:Validate:failed, fabrik:paused, fabrik:awaiting-input, fabrik:awaiting-ci, fabrik:auto-merge-enabled, fabrik:revalidate (trigger removed last) |
Back-edge into Validate. Resets PausedByEngine, StageRetryCount, LastAttemptAt, EngineCycles for Validate. See §2.15. |
| User (manual) | fabrik:revalidate applied |
Non-Validate column | Only the trigger label is removed; no other action | (none) | fabrik:revalidate |
Warning logged. See §2.15. |
| Engine scan | SHA-invalidation detected | Validate, Complete | Removes Validate completion; re-dispatches Validate on next poll | (none) | stage:Validate:complete, fabrik:awaiting-ci, fabrik:awaiting-review, fabrik:rebase-needed, fabrik:auto-merge-enabled |
Back-edge into Validate. Triggered when snap.ValidateCompletedSHA() ≠ LinkedPR.HeadSHA — Claude pushed a new commit after Validate ran. Not taken for a landed/terminal item, nor when the new head is the engine’s own pure catch-up merge (#2158). See §2.16. |
| Engine (convergence monitor) | PR merged by GitHub auto-merge | Validate, Complete + fabrik:auto-merge-enabled |
Removes convergence label; advances to Done | (none) | fabrik:auto-merge-enabled |
checkAutoMergeConvergence detects pr.Merged == true; calls advanceToNextStage. The stage:Validate:complete label remains. See §5.5. |
| Engine (Implement dispatch) | FABRIK_PR_CREATE_BEGIN/END marker |
Implement, Locked + In Progress | Draft PR created on GitHub; no stage advance | (none) | (none) | Engine reads marker from Claude output, calls gh pr create --draft; posts PR URL as a comment on the issue. Stage advance requires a separate FABRIK_STAGE_COMPLETE. See §5.6. |
| Engine / Claude | FABRIK_STAGE_COMPLETE + FABRIK_NO_WORK_NEEDED |
Any stage, Locked + In Progress | All subsequent non-cleanup stages marked complete; issue moved to Done once the settle scan’s Done-move/close succeeds (retried every poll if it fails; escalated to fabrik:paused after MaxRetries) |
fabrik:awaiting-done, stage:<X>:complete, stage:<Y>:complete … (all subsequent non-cleanup stages) |
fabrik:locked:<user>, stage:<X>:in_progress, fabrik:awaiting-done (once Done is reached or on escalation) |
No PR created. Validate re-entry back-edge from fabrik:revalidate and SHA-invalidation are visible in §10.4. See §6.8. |
4. Comment Processing Lifecycle
When new comments are detected on an issue (or synthetic review comments on a PR), the engine processes them through processComments(). This is an 11-step flow.
4.1 Comment Detection
findNewComments() filters item.Comments to find unprocessed comments using four independent dedup signals:
- In-memory
CommentProcessedinitemstate.Store(session-scoped) — skip comments whose ID is recorded viasnap.CommentProcessed(c.ID)(written byCommentProcessedmutation). Fast but lost on restart. 🏭 **Fabrikbody prefix (engine-authored output convention) — skip comments whose body starts with this header. Durable but requires the header to be present.- 🚀 ROCKET reaction (durable, cross-restart) — skip comments that already have a rocket reaction. Applied to user comments by
processCommentsstep 10 after processing; also applied by the engine to every comment it posts immediately afterAddCommentsucceeds. Review bodies (no REST reactions endpoint, noDatabaseID) get the same 👀/🚀 through GraphQLaddReactionon the review’s node ID (#1953 R8); that reaction is human-visible only and is not read back for dedup. - Bot service-notice pattern (content-based,
isBotServiceNotice) — skip comments whose author matchesgithub.IsBotLoginand whose body matches a known service-notice pattern frombotServiceNoticePatterns(engine/comments.go), a flat, vendor-grouped list of literal case-insensitive substrings. Covers two vendors as of #1122, plus a third CodeRabbit marker added by #1141: Gemini’s quota/rate-limit/sunset/unsupported-file-type notices (e.g. “daily quota limit”, “you have reached your rate limit”, “the consumer version of gemini code assist on github has been sunset”), CodeRabbit’s rate-limit notice, matched primarily via its HTML comment marker (rate limited by coderabbit.ai— a structural, non-user-facing signal that won’t drift when CodeRabbit rewrites its banner copy) with prose fallbacks (“## review limit reached”, “you’ve reached your pr review limit, so we couldn’t start this review”) in case a future CodeRabbit variant omits the marker — the fallbacks are deliberately full-phrase rather than bare fragments, since a bare"review limit reached"substring would itself match a genuine bot review comment that quotes these very patterns (e.g. a diff review critiquing this pattern list) — and CodeRabbit’s auto-generated acknowledgement-reply marker (auto-generated reply by coderabbit, matching the HTML comment<!-- This is an auto-generated reply by CodeRabbit -->), distinct from the rate-limit marker above: it covers a content-free “acknowledged, no action taken” reply rather than a quota-exhaustion notice, and — like the rate-limit marker — is matched structurally so it holds regardless of which acknowledgement phrasing CodeRabbit uses. Unlike signals 1–3, this is a content classification rather than a watermark: it must hold on every poll’s raw fetch, so a matching comment never reachesprocessCommentsin the first place — no 👀 reaction, no worktree/Claude invocation, no reply. Because this signal excludes the comment before signal 1’s watermark would ever be recorded by the normalprocessCommentsflow,settleBotServiceNotices— an unconditional per-poll scan over the raw board snapshot, independent of dispatch state — separately applies the 🚀 reaction andCommentProcessedwatermark so the comment doesn’t need to re-qualify via this pattern check on every future poll. Classification is per-comment, not per-author or per-length: a bot can post both a rate-limit notice and a genuine review on the same PR, and each comment is classified independently on its own body. Introduced to stop a runaway reply loop where a bot’s quota-exhaustion auto-reply triggered a Fabrik “not actionable” reply, which in turn re-triggered the bot (#1083/#1088); extended in #1122 once Gemini was usage limited and CodeRabbit became the active review bot with its own unmatched rate-limit phrasing; extended again in #1141 to cover CodeRabbit’s acknowledgement-reply loop (see the outbound mention-neutralization invariant below — this marker was one half of a two-part fix, the other half being that the acknowledgement was also being re-triggered by Fabrik’s own reply naming the bot). Enforcement is now split across two points (#1221): this raw-fetch exclusion insidefindNewCommentsstill governs whether a worker is dispatched at all for a bot-notice-onlyitem.Commentsbacklog (a notice excluded here never causesitemNeedsWorkto see “new work”), while a second chokepoint insideprocessCommentsitself (§6.2) applies the sameisBotServiceNoticeclassifier to the fully-assembled working slice just before Step 1 — covering comments merged in fromitem.LinkedPRReviewThreadCommentsand comments supplied by any of the three reinvoke dispatchers (dispatchReviewReinvoke,dispatchCIFixReinvoke,dispatchRebaseReinvoke), none of which route through thisfindNewCommentsexclusion.
Any single signal catching the comment is sufficient to skip it. The four signals are orthogonal — any combination can fail independently without triggering the self-review loop.
Dedup coverage by comment type:
- Engine-authored comments: carry signals (2) and (3) — the
🏭 **Fabrikprefix (when formatted viaformatOutputComment) and a 🚀 reaction added by the engine at post time. - User comments: carry signals (1) and (3) after processing — the
CommentProcessedentry added toitemstate.StoreduringprocessComments, and the 🚀 reaction added at step 10. - Bot service-notice comments: carry signal (4) immediately, then signals (1) and (3) once
settleBotServiceNoticeswatermarks them on a later poll.
Invariant: every engine-emitted
AddCommentcall must start with🏭 **Fabrik — <context>**. This is an engine-wide convention enforced byTestAddCommentComplianceinengine/compliance_test.go, not just a detection heuristic.
Invariant (#1141): generated stage/comment output never renders a live GitHub mention of a bot login.
neutralizeBotMentions(engine/mentions.go) rewrites any@<login>wheregithub.IsBotLogin(login)is true into an inline code span —@coderabbitaibecomes`@coderabbitai`— so GitHub does not notify the bot, regardless of what the surrounding prose claims about whether a reply was posted. Detection is code-span-aware (text already inside backticks is skipped), which makes the rewrite idempotent. Applied inside the four points Claude-derived freeform text is embedded into posted or persisted content:formatOutputComment,formatPRSummaryComment,formatReviewFeedbackComment(allengine/pr.go),updatePRVerification’s PR-body## Verificationrewrite (engine/pr.go— aUpdateIssueBodycall outside theAddCommentcompliance funnel above, but equally capable of triggering a live mention), andbuildAwaitingInputComment(engine/item.go).github.IsBotLogingained a bare-literal"coderabbitai"match for this: CodeRabbit’s GitHub API login carries a[bot]suffix (already matched), but its@-mention surface is the bare form, which the suffix rule alone does not catch. Fixes the #933 incident, where Fabrik’s own “no action taken” comment-processing summary re-mentioned@coderabbitai, notifying the bot into another acknowledgement reply that Fabrik then processed again — see ADR-073.
4.2 The 11-Step Flow
| Step | Action | Code | Side Effects |
|---|---|---|---|
| 0 | Emit JobStartedEvent + defer JobCompletedEvent{Skipped:true} |
e.emitStructural(tui.JobStartedEvent{...}) |
TUI active-pane entry created; deferred JobCompletedEvent{Skipped:true} fires unconditionally on function return (cleanup guard). HistoryPaneComponent filters it out — only InvocationObserver’s Skipped:false event reaches history. This is the TUI work-boundary — fires at processComments entry before any external I/O. |
| 1 | React with 👀 to all new comments | AddCommentReaction("eyes") / AddPRReviewCommentReaction("eyes") |
Signals acknowledgment to the user |
| 2 | Add fabrik:editing label |
AddLabelToIssue("fabrik:editing") |
Pre-dispatch gate in itemNeedsWork (prevents goroutine launch); defense-in-depth check retained in processItem for the race window. Symmetric with fabrik:locked:<other-user>. JobStartedEvent already fired at step 0 — the active session registers as in-progress in the TUI before this label is added. |
| 2b | Clear stale stage:<Stage>:complete if present (#1802) |
beginStageRework() |
No-op unless the re-entered stage’s own stage:<Stage>:complete is already on the item. When present: adds fabrik:reworking:<Stage>, then removes stage:<Stage>:complete — mark-first-then-clear, strictly inside the step-2 fabrik:editing bracket so no dispatch gate ever observes the transient clear (R4). See ADR-1802. |
| 3 | Ensure worktree exists | EnsureWorktree() |
Creates or updates worktree; writes context files |
| 4 | Invoke Claude with comment review prompt | InvokeForComments() |
Uses comment_prompt / comment_skill and comment_max_turns |
| 5 | Check for FABRIK_STAGE_COMPLETE in output | checkCompletion() |
Determines if comment processing resolved the stage |
| 6 | Extract and apply FABRIK_ISSUE_UPDATE if present | extractUpdatedBody() |
Applied unconditionally when markers are present; stripped from output regardless |
| 7 | Strip all Fabrik markers from output | stripLine() calls |
Removes FABRIK_STAGE_COMPLETE, FABRIK_BLOCKED_ON_INPUT, FABRIK_NO_WORK_NEEDED, FABRIK_SUMMARY_BEGIN/END |
| 8 | Post or update stage comment | AddComment() / UpdateComment() |
For post_to_pr stages: always posts new comment on issue (labeled as “comment review”); for other stages: rewrites existing stage comment or creates new one. Review-reinvoke branch (Step 8b): when the input batch is all-ReviewThreadID comments (isReviewReinvoke == true) and output != "", also posts a Fabrik-marked "<StageName> (review feedback addressed)" comment on the linked PR (via FindPRForIssue); includes per-thread footer with path:line for each addressed thread; skipped if no linked PR is found (logs warning). The issue comment is always posted first; the PR comment is additive. Suppressed on FABRIK_NO_WORK_NEEDED (#1088): when the invocation output contains FABRIK_NO_WORK_NEEDED (checked before markers are stripped), none of the three reply paths above post — no stage comment, no post_to_pr comment, no review-reinvoke PR summary. Posting any reply is what re-triggers a subscribed bot into a loop, so silence replaces the old “not actionable” message. Steps 6 (issue-body update), 9 (editing-label removal), 10 (🚀 reactions), and 11 (stage-complete handling) are unaffected — only the reply post itself is gated. |
| 8b | Restore or re-derive stage:<Stage>:complete (#1802) |
endStageRework() |
No-op unless step 2b actually cleared the label. On a non-completing exit (this step, or any of the earlier setup-failure/invocation-failure early returns): restores stage:<Stage>:complete directly, then removes fabrik:reworking:<Stage> immediately — nothing later in the flow will restore it. On a completing exit: this step is deliberately skipped here — stage:<Stage>:complete is left to step 11’s handleStageComplete() to re-derive from scratch (including its wait_for_ci deferral), and fabrik:reworking:<Stage> itself is not removed until after step 11 completes (not here at step 8b), so a crash anywhere between this point and step 11’s completion-label write still leaves the marker as a durable signal that a restore is owed. See ADR-1802. |
| 9 | Remove fabrik:editing label |
RemoveLabelFromIssue("fabrik:editing") |
Releases the editing mutex |
| 10 | React with 🚀 to all processed comments + resolve review threads | AddCommentReaction("rocket") / AddPRReviewCommentReaction("rocket") / AddReviewReaction("rocket") (review bodies, GraphQL, #1953 R8) + ResolveReviewThread() |
Marks comments as processed (durable); resolves addressed review threads |
| 11 | If FABRIK_STAGE_COMPLETE was detected: handle completion | handleStageComplete() |
Same completion flow as a normal stage invocation (advance, PR ops, etc.). On a completing exit that began a rework (step 2b), fabrik:reworking:<Stage> is removed immediately after this step returns — not before — per step 8b’s note (#1802, ADR-1802). |
4.3 Turn Limit Extension
When Claude exits a comment processing invocation due to comment_max_turns (i.e., invUsage.TurnsUsed >= currentBudget and !invCompleted && err == nil), the engine evaluates whether to extend before returning partial output.
Extension trigger condition: !invCompleted && err == nil && currentBudget > 0 && invUsage.TurnsUsed >= currentBudget
Note: currentBudget > 0 is only satisfied when fabrik:extend-turns is present (label absent → currentBudget = 0 → no extension possible). This differs from stage invocations (§3), where the progress-based extension loop fires whenever stage.MaxTurns > 0 is hit — the label only pre-grants the 2× first budget there. Comment processing is intentionally label-gated: extending comment-review turns is a new opt-in capability, and changing no-label behavior would silently extend comment budgets for all existing issues.
Hard cap: 3× commentMaxTurns(stage) total across all invocations. When totalMultiple >= 3, no further extension is attempted.
commentMaxTurns(stage): Returns CommentMaxTurns if set, else MaxTurns, else 50. This value is always > 0 (unlike stage.MaxTurns which can be 0 for unlimited).
Per-stage progress signals: Same signals as the stage invocation path — see §3 Turn Limit Extension table. For no-signal stages (Research, Specify, Plan, Done), detectProgress returns false immediately, so fabrik:extend-turns grants the 2× pre-budget for the first invocation but no further extension.
Extension loop behavior (within a single processComments call):
hadExtendTurnsLabelsnapshotted before the loop. If present:currentBudget = 2 × commentMaxTurns(stage),totalMultiple = 2. If absent:currentBudget = 0,totalMultiple = 1.snapshotBaselinecalled before the loop (same function as stage path).InvokeForCommentscalled withopts.MaxTurnsOverride = currentBudget. Session resume is handled internally byInvokeClaudeForComments— no loop-level session management needed.- If limit hit AND
totalMultiple < 3: calldetectProgress. If progress →totalMultiple++,currentBudget = commentMaxTurns(stage), re-invoke. If no progress or error → return partial output. - Output accumulated across all invocations before posting as a single comment.
- Usage totals (tokens, turns) and
InvocationRecordedstore event applied once after loop completes.
fabrik:extend-turns label: When present, the first comment processing invocation receives 2 × commentMaxTurns(stage) as its budget (pre-granted, no progress check required for the first turn-limit hit). That same first invocation also gets max_wall_time scaled by the identical 2× factor (scaledWallTime in engine/claude.go, §7.7, ADR-1206) — the same mechanism as the stage-invocation path (§3.2). Subsequent extensions reset to the un-multiplied commentMaxTurns(stage) budget and, via their own fresh InvokeClaudeForComments/runClaude call, run under the unscaled deadline.
Log tag: [#N extend-turns] — same tag as stage path. Emitted on each detectProgress call. When extension is granted, an additional line logs the new budget multiple and cumulative turns used. [#N stats] emitted after loop with final accumulated usage.
| Current State | Event | Guard | Resulting State | Labels Added | Labels Removed | Side Effects |
|---|---|---|---|---|---|---|
| Comment Processing, In Progress | Turn limit hit | totalMultiple < 3; progress detected |
Comment Processing, In Progress (extension) | totalMultiple++; currentBudget = commentMaxTurns(stage); output accumulated; InvocationRecorded deferred |
||
| Comment Processing, In Progress | Turn limit hit | totalMultiple >= 3 (hard cap) |
Comment Processing, Complete (partial) | Hard cap reached; partial output posted; InvocationRecorded applied |
||
| Comment Processing, In Progress | Turn limit hit | No progress detected or progress check failed | Comment Processing, Complete (partial) | No extension; partial output posted | ||
| Comment Processing, In Progress | FABRIK_STAGE_COMPLETE (any extension) | invCompleted = true |
Comment Processing, Complete | Normal comment completion flow; output posted |
4.4 Comment Processing Entry Points
Comments can trigger processing through three paths in processItem():
- Awaiting-input unblock:
isAwaitingInput(item)is true + new human comments created at/after the pause (resumeAuthorised()) →unblockAwaitingInput()→processComments() - Paused unpause:
fabrik:pausedpresent + new human comments created at/after the pause (resumeAuthorised()) → removefabrik:paused,clearFailedStage()→ fall through →processComments() - Normal comment processing: Item is not paused →
findNewComments()finds comments (human or bot) →processComments()
Resume anchor (#1813, ADR-1813). resumeAuthorised() is the single predicate behind all four resume gates (itemNeedsWork’s awaiting-input and paused branches, processItem’s awaiting-input block and paused loop), so admission and resume cannot disagree. It authorises a resume only from an event — a human comment whose CreatedAt is at or after the latest labeled event for fabrik:paused (read live via FetchLabelAppliedAt, never the record-on-write cache, so a label removed in the UI cannot leave a stale early anchor; isAwaitingInput implies paused, so this one label covers both) — not from the state “an unprocessed human comment exists”. A comment that can never be processed therefore lifts a pause at most once: after the resume any re-pause writes a newer labeled event, and the comment predates it. Consequently the breaker counters (CommentBreakerReset, EngineCyclesCleared, clearFailedStage) are only zeroed by a genuine resume. Tie-break (R6): every indeterminate input resumes — anchor fetch error, no labeled event found, a zero CreatedAt, or a comment timestamp equal to the anchor (one-second resolution); only “every human comment strictly predates the pause” refuses. Trade-off: a human comment posted while an invocation runs, which then pauses (blockOnInput, a breaker trip), predates that pause and does not resume it — the human comments again.
In paths 1 and 2, resumeAuthorised() only gates whether to resume — once a human comment authorizes it, the raw (unfiltered) comment set from findNewComments() is what’s actually handed to processComments(), in the same processItem invocation. Any bot comments that accumulated while paused/awaiting-input are processed together with the resuming human comment, not deferred to a later poll (#1083) — except bot service-notice comments (§4.1 signal 4), which findNewComments() excludes from that raw set entirely regardless of pause state, since they are non-actionable either way.
All three paths ultimately source their candidate comments from findNewComments() (paths 1–2 via resumeAuthorised()’s wrapping, path 3 directly). But processItem()’s three paths are not the only route into processComments() — the three catch-up-loop reinvoke dispatchers (dispatchReviewReinvoke, dispatchCIFixReinvoke, dispatchRebaseReinvoke, §6.2) call it directly with comments built by buildReviewThreadComments() or a synthetic single-comment builder, neither of which goes through findNewComments(). A quota/rate-limit bot notice can therefore still reach processComments() as an entry in its working comments slice via one of these non-findNewComments() paths — the claim that it “never reaches processComments() via any path” was false, and as of #1221 that gap is closed differently: processComments() itself (§6.2) applies isBotServiceNotice as a single chokepoint to its fully-assembled working comments slice, immediately after the slice is assembled — covering the findNewComments()-sourced set (idempotently) as well as the LinkedPRReviewThreadComments merge and all three reinvoke dispatchers’ input — before any reaction, label, worktree, or invocation side effect. findNewComments() is the enforcement point for whether a worker is dispatched at all (§4.1 signal 4); processComments() is the enforcement point for whether a comment — however it arrived — is ever handed to Claude or otherwise acted upon.
All six of these entry points — the three processItem() paths above and the three reinvoke dispatchers — funnel through the same processComments() function body, so all six inherit the fabrik:reworking:<Stage>/stage:<Stage>:complete clear-and-restore behavior (§4.2 steps 2b/8b, ADR-1802) uniformly, with no per-site special-casing. There is exactly one place that decides whether a re-entered stage’s completion claim needs clearing (beginStageRework, called once, immediately after step 2’s fabrik:editing add succeeds) and exactly one place that restores or defers to the normal completion flow (endStageRework, called from every exit path of processComments()/finalizeComments()) — regardless of which of the six triggers brought the item here.
Post-invocation restore of the engine’s own rebase (#2150, ADR-2150). Step 3 of processComments() (EnsureWorktree) rebases the issue branch onto origin/<base> (unless the PR is queued or the repo is queue-enabled, ADR-058). When the pass then commits nothing, nothing is pushed and the local branch would be left a rebased copy of its remote — diverged, with nothing on either side the other lacks, and a head nobody has CI for (which also defeats the ci-status.md HEAD == head_sha skip rule). processComments() therefore captures the head before EnsureWorktree and, immediately after the extension loop returns and headChanged is computed — before either fabrik:editing removal and before finalizeComments’ possible markPRReady push — calls WorktreeManager.RestoreUnpushedRebase, which resets to the pre-rebase head only when the worktree is on the issue branch with HEAD still exactly the post-rebase head (no commit, reset or amend), the tree is clean with no merge/rebase/cherry-pick in progress, a fresh fetch of the issue branch succeeds and shows the rebased head is not already on the remote, and the rebased head carries no patch (patch-id, base history excluded) or merge commit that was not on the pre-rebase head. Any failure or doubt leaves the branch alone and logs the reason (no commit this pass, but the engine rebase was left in place: …); a pass that commits is unchanged. Side effect: dispatchReviewReinvoke/dispatchCIFixReinvoke capture headBefore pre-rebase, so a restored no-commit pass now compares equal and the no-op refund (ReviewCycleDecremented, #1045) and CIFixNoOpRecorded (#958) fire where a moved HEAD previously hid them. Catch-up covers worktrees already in the rebased state (§6.30). SetCommentRebaseRestoreDisabledForTest is the neutralisation seam.
4.5 markCommentsSeenByStage
After a stage invocation (not comment processing), markCommentsSeenByStage() adds ROCKET reactions to all pre-existing comments that were included in the prompt as context. This prevents those comments from triggering the awaiting-input unblock logic on subsequent polls.
4.6 Comment-Processing Circuit Breaker (#1089)
Fixes 1/4 (#1087, ADR-069) and 2/4 (#1088, ADR-070) close the two known pumps behind the #1083 incident: a bot comment can no longer silently lift an operator’s pause, and a known bot service-notice no longer spawns a worker or draws a reply. Neither fix, however, counts damage from a loop it doesn’t recognize — a new bot-notice variant, a webhook that re-triggers on Fabrik’s own reply some other way, or any other self-sustaining pattern not yet anticipated. The circuit breaker is a defense-in-depth backstop of last resort, not a replacement for fixes 1/4 or 2/4: if the same issue undergoes N comment-processing invocations within a rolling window T with no forward progress, the engine pauses the issue, posts an explanatory comment, emits a structural TUI event, and suppresses further dispatch. It should rarely if ever fire in practice.
Counter. Tracked per issue (not per stage — the #1083 incident stayed on one stage throughout, but a legitimate stage transition mid-window must not itself reset the count to a lower, easier-to-hit number) in itemstate.Store as ItemState.CommentBreaker: InvocationsAt []time.Time plus LastAuthor string (the author of the comment that triggered the most recent recorded invocation, surfaced in the trip comment). Two mutations drive it: CommentBreakerInvocationRecorded{Repo, Number, At, Author, Cutoff} (append, pruning any InvocationsAt entries older than Cutoff first when Cutoff is non-zero) and CommentBreakerReset{Repo, Number} (clear both fields; a no-op — no Change emitted — when already empty). The Store holds only raw timestamps; threshold/window business logic lives in the engine (engine/comment_breaker.go), mirroring the existing mergeTrainTrials runaway-guard precedent (ADR-059 D8). recordCommentBreakerInvocation() computes Cutoff from the current window and passes it on every append, so InvocationsAt is pruned at write time and doesn’t grow without bound for an issue that receives invocations sparser than the window; commentBreakerCount() additionally prunes-and-counts on every read against time.Now().Add(-window) as a cheap belt-and-suspenders pass that also stays correct if the configured window shrinks between writes.
Recording point. recordCommentBreakerInvocation() is called inside processComments (engine/comments.go) immediately after the working comment slice is finalized (once the item’s bot-service-notice filtering has run), before any setup side effect — the step 0 JobStartedEvent emission, the step 1 👀 reaction, the step 2 fabrik:editing label add, and step 3 worktree setup (§4.2) — and well before step 4’s InvokeForComments() call. Recording this early (#1413) means a failure in any of the three setup steps counts as a cycle, not just an invocation-time failure: each of the three setup-failure early-return paths also calls checkCommentBreaker() (see “Trip action” below) with a reason describing the failing step, so a persistently failing setup step — e.g. a fabrik:editing label add that fails on every poll under GraphQL budget pressure — trips the breaker within the configured window instead of looping unbounded (the #1382/#1386 failure mode this closes). Before #1413, the record call sat immediately before the Claude invocation, so all three setup steps could fail indefinitely without ever incrementing the counter; ADR-1089’s original rationale for that placement (“an early return before Claude runs… does not count as a wasted cycle”) is superseded by ADR-1413.
Threshold and window. Default N = 10 invocations within T = 30 minutes, both zero-means-default (mirroring MaxTrainTrialsPerWindow/TrainTrialWindowDuration): Config.MaxCommentCyclesPerWindow / Config.CommentCycleWindow, overridable via --max-comment-cycles-per-window / --comment-cycle-window flags, FABRIK_MAX_COMMENT_CYCLES_PER_WINDOW / FABRIK_COMMENT_CYCLE_WINDOW env vars, or max_comment_cycles_per_window / comment_cycle_window in config.yaml (flag > env > config.yaml > default 10 / 30m — same precedence order as every other zero-means-default engine setting).
Reset triggers (any one resets InvocationsAt to empty before the threshold is reached):
| Trigger | Hook point | Rationale |
|---|---|---|
stage:*:complete transition |
handleStageComplete() (engine/stages.go), reset called before any completion side effect |
The one function reached on stage completion regardless of whether it originated from a plain stage run or from finalizeComments’s completed branch — a single choke point. |
| New commit on the branch | Inside processComments, comparing gitHeadSHA(workDir) immediately before and after the Claude invocation |
Reuses the existing gitHeadSHA helper (already used by detectProgress, §3/§4.3); detects the commit locally without depending on a webhook-lagged PR head-SHA update. |
| PR state change | CommentBreakerObserver (engine/observers.go), subscribed to itemstate.Store, reacting to the PRStateChanged flag |
PRStateChanged is a narrower sub-flag of LinkedPRChanged, set only by PRDetailsUpdated (a genuine PR-level transition: merged, closed, or draft↔ready). It is deliberately not keyed off the broader LinkedPRChanged — that flag also fires on PRReviewSubmitted/PRReviewCommentCreated/ReviewThreadCommentAdded (a new PR review or inline review comment), which is the same event that supplies the comment processComments is about to process for a review-reinvoke. Resetting on that broader flag would zero the counter on the same cycle that records its own invocation, capping the observed count at 1 forever and defeating the breaker for its primary PR-review-comment-loop use case (caught during PR review of #1089, fixed before merge). |
Issue body edited (FABRIK_ISSUE_UPDATE) |
Inside publishCommentOutput, in the existing extractUpdatedBody(output) != "" branch (§4.2 step 6) |
Pre-PR stages (Specify/Research/Plan) iterate purely via issue-body edits — no commit, no PR yet, and no stage completion until the human is satisfied. Without this trigger, legitimate iterative Q&A in those stages could accumulate invocations toward the threshold with no other reset available. |
| Manual human unpause | clearFailedStage() (engine/item.go), alongside the existing ReviewCycles/CIFixCycles/RebaseCycles/EnqueueCycles clears |
A human who investigates and manually removes fabrik:paused is explicitly giving the issue another shot; without this reset they would be re-tripped after a single subsequent invocation that hadn’t yet produced one of the four automatic signals above. |
Trip action. checkCommentBreaker(item, reason string) is called from five points in processComments: each of the three setup-failure early-return paths (editing-label add, base-branch resolution, worktree setup — reason names the failing step and its error), the non-completing invocation-failure early-return path (reason == ""), and, when the invocation proceeds normally, after publishCommentOutput/finalizeComments (reason == "", so any reset those steps applied takes effect first). A sixth, distinct early-return sits just above the non-completing-failure point (mirroring the pre-existing usage-limit exclusion there): a tools-denied classification (*claudeToolsDeniedError, #1704, §7.3c) returns before either checkNoOpCommentCycle or checkCommentBreaker is ever called, so a mode denial is bounded exclusively by its own MaxToolsDeniedRetries counter, never by this breaker’s frequency-based thresholds. When commentBreakerCount() >= N, tripCommentBreaker(item, count, window, reason):
- Applies
fabrik:paused+fabrik:awaiting-inputvia the sharedpauseIssue()/pauseOptshelper (ADR-065) — the same helper and same labelspauseForReviewCycleLimitand the merge-train runaway guard already use, so ADR-069’s honorable-pause guarantee (a subsequent bot comment cannot silently lift the pause, §1.4fabrik:pausedrow) applies automatically. No new label is introduced. - Posts an explanatory
🏭 **Fabrik — comment-processing circuit breaker tripped**comment naming the invocation count, the window, the last comment author, and how to resume (removefabrik:paused). Whenreasonis non-empty (the tripping cycle was a setup failure, #1413), the comment gains an additional sentence naming the specific setup step and error that caused it — e.g. “The cycle that tripped this breaker failed during setup: the fabrik:editing label add failed:." — so the pause is diagnosable rather than a bare "no forward progress" notice. `reason` is passed as a call parameter, not persisted in `itemstate` — it always describes the cycle that is tripping the breaker synchronously, in the same call, so no durable storage is needed (mirrors `LastAuthor`'s "only the triggering cycle's context matters" precedent). - Emits
tui.CommentBreakerTrippedEvent{IssueNumber, Repo, Title, StageName, InvocationCount, Window, LastCommentAuthor}viaemitStructural— the TUI’s active pane renders it as a blocked entry, mirroringIssueBlockedEvent(§9.8). - Suppresses further dispatch: once
fabrik:paused+fabrik:awaiting-inputare present, the item is gated by the same paused/awaiting-input machinery as every other pause (§1.4, §4.4) — only a human comment created after the pause (resumeAuthorised()) can resume it, and per §4.4 a resuming human comment is processed together with any bot chatter that accumulated during the pause, which itself records a fresh invocation and is subject to the breaker again.
Scope note. The breaker’s suppression is a downstream consequence of the standard pause/awaiting-input labels, not a separate dispatch gate — it adds no new entry to itemNeedsWork/itemMayNeedWork (§8.1, Appendix C). This is unlike fabrik:awaiting-done or fabrik:blocked, which are engine-internal gates with their own dispatch-suppression wiring; the circuit breaker deliberately reuses the existing human-only pause semantics instead of introducing a fifth.
4.7 Success-Agnostic No-Op Comment Cycle Breaker (#1555)
handarbeit/fabrik#1254’s Validate stage redelivered the same stale, already-processed bot review body seven consecutive times — each a clean (err == nil) cycle: no commit, no issue-body update, no FABRIK_STAGE_COMPLETE. §4.6’s breaker records every invocation regardless of success, but its counter is a rolling 30-minute window, sized for #1083’s burst-loop shape (~995 invocations in rapid succession). A loop whose invocations are spaced sparser than 30 minutes — as this one was, driven by Fabrik’s own self-upgrade restart cadence (the redelivery’s actual root cause, fixed directly by §6.2’s “Durable review-ids-addressed marker”) — never accumulates enough entries within any single window to trip §4.6’s breaker, no matter how many total invocations occur over the issue’s lifetime. This section’s counter is the safety net for that redelivery, not its fix; it is a second, independent breaker that closes the windowing gap regardless of what causes a cycle to be a no-op.
Counter. StageState.NoOpCommentCycles map[string]int (internal/itemstate/itemstate.go) — per-stage, unlike §4.6’s item-scoped CommentBreaker, and never time-pruned, mirroring ReviewCycles/ReviewBlockedCycles’s shape (§6.2, ADR-1518) rather than CommentBreaker’s rolling-timestamp shape. Two mutations drive it: NoOpCommentCycleIncremented{Repo, Number, StageName} and NoOpCommentCycleReset{Repo, Number, StageName} (a no-op — no Change emitted — when already zero).
Recording point. checkNoOpCommentCycle(item, stage, progressed bool, lastAuthor string) (engine/comment_noop_breaker.go) is called from the same five mutually-exclusive exit points in processComments that call checkCommentBreaker (§4.6’s “Trip action”) — exactly one of the five executes per cycle. It is evaluated first at each site; only when it does not trip does checkCommentBreaker also run for that cycle:
if !e.checkNoOpCommentCycle(item, stage, progressed, lastCommentAuthor(comments)) {
e.checkCommentBreaker(item, reason)
}
Both counters still record every invocation independently — NoOpCommentCycles increments even on a cycle where §4.6’s breaker is the one that ultimately trips. What’s mutually exclusive is the pause escalation: at most one fabrik:paused application and one trip comment per cycle.
progressed — R2’s three signals, computed once per cycle. progressed := headChanged || extractUpdatedBody(output) != "" || completed:
headChanged—gitHeadSHA(workDir)compared immediately before/afterrunCommentExtensionLoopreturns, the same comparison §4.6’s own commit-reset trigger already uses.extractUpdatedBody(output) != ""— the same issue-body-update signal §4.6 uses, evaluated against the caller’s own copy ofoutput(publishCommentOutputtakesoutputby value, so its internal marker-stripping never mutates the caller’s copy).completed— the sameFABRIK_STAGE_COMPLETE-derived bool threaded through the rest ofprocessComments.
The three setup-failure sites (editing-label add, base-branch resolution, worktree setup) pass progressed=false directly — no invocation ran, so none of the three signals can be true by construction.
No refund/decrement counterpart, unlike ReviewCycles. progressed=true unconditionally resets the counter to 0; there is no partial-credit decrement. A cycle that would qualify for #1045’s ReviewCycleDecremented refund in the review-gate path (a reinvoke dispatched while the gate was already clear, landing no commit) is, from this counter’s perspective, indistinguishable from any other no-progress cycle — it correctly increments here. This counter and §6.2/ADR-1045’s refund answer different questions: ADR-1045 asks “did this reinvoke change the review-gate verdict”; this counter asks “did this comment-processing cycle change anything observable on the issue.”
Threshold. Default 10, zero-means-default: Config.MaxNoOpCommentCycles, overridable via --max-no-op-comment-cycles flag, FABRIK_MAX_NO_OP_COMMENT_CYCLES env var, or max_no_op_comment_cycles in config.yaml — the same flag > env > config.yaml > default precedence as §4.6’s own threshold.
Deliberately not equal to MaxReviewCycles’s default (5). Both counters observe the same processComments funnel (see below), but only ReviewCycles is ever refunded — a shared default would make NoOpCommentCycles the one that always trips first, silently nullifying ADR-1518’s tolerate-forever guarantee at whatever count the shared value happened to be. An early version of this fix did ship both at 5 and isolated the collision out of the review-reinvoke tests with an inflated per-test override (eng.cfg.MaxNoOpCommentCycles = 100) rather than fixing the default — that hid, rather than resolved, the fact that TestHandleReviewGate_FiveNoOpReinvokes_DoNotExhaustBudget_SixthGenuineFindingAddressed’s own guarantee was false under the shipped default config. Live evidence on this repo during review (a healthy, mergeable PR accumulating 4 consecutive no-op comment-processing cycles from ordinary duplicate bot delivery) showed 5 was too tight regardless of the test-isolation question — normal operation was one round from tripping it. 10 was chosen as comfortably clear of that observed ordinary-churn ceiling while still being a hard bound. See ADR-1555’s Consequences.
Interaction with §6.2/ADR-1045’s no-op refund on the review-reinvoke path. Because checkNoOpCommentCycle runs at the same processComments funnel every dispatch source shares (§4.6), it also counts cycles dispatched via dispatchReviewReinvoke. ReviewCycles’ refund (ReviewCycleDecremented) was deliberately built to forgive an unbounded run of no-op reinvokes as long as the review gate itself keeps returning a clear verdict (ADR-1518); this counter does not forgive the same cycles, so at the default threshold, ten consecutive no-op review-reinvoke rounds now pause the issue even where ReviewCycles alone would have tolerated more. This narrows ADR-1518’s “forgive forever” guarantee to “forgive up to MaxNoOpCommentCycles consecutive rounds” in practice — deliberately, with enough headroom above both MaxReviewCycles’s own default and observed ordinary-operation churn that it should not be reached by a review bot behaving normally. An operator whose review bots need more than 10 consecutive no-op rounds before converging should raise --max-no-op-comment-cycles.
Reset triggers are narrower than §4.6’s five — no PRStateChanged reset, no new itemstate.Store observer. Only R2’s three signals above reset the counter, plus a manual unpause (below). If a future incident shows this is too narrow, that is a follow-up, not a blocking gap — this counter is a safety net, not the primary defense (the durable-marker fix in §6.2 is).
Manual unpause — from either pause route. EngineCyclesCleared’s store.go apply case deletes StageState.NoOpCommentCycles[stageName] alongside its existing deletes for ReviewCycles/ReviewBlockedCycles/CIFixCycles/RebaseCycles/EnqueueCycles. This breaker’s own trip action (below) pauses via fabrik:paused + fabrik:awaiting-input — the resume path §1.4’s table routes through unblockAwaitingInput (isAwaitingInput(item) true + a human comment), not clearFailedStage (which handles the separate stage:<name>:failed + fabrik:paused shape). Both clearFailedStage and unblockAwaitingInput apply EngineCyclesCleared and reset §4.6’s CommentBreaker, so the counter is reset regardless of which of the two pause shapes an item is resumed from. (unblockAwaitingInput gained this call as part of #1555 — before that fix it cleared retry/pause state but not the cycle counters, so NoOpCommentCycles survived an awaiting-input resume at its tripped value and could re-trip on the very next cycle; see ADR-1555.)
Trip action. tripNoOpCommentCycleBreaker reuses pauseIssue/pauseOpts exactly like §4.6’s breaker — fabrik:paused + fabrik:awaiting-input, ADR-069’s honorable-pause guarantee. The trip comment (🏭 **Fabrik — no-op comment-processing circuit breaker tripped** — a distinct header from §4.6’s 🏭 **Fabrik — comment-processing circuit breaker tripped**, so an operator always knows which mechanism fired) names the stage, the consecutive no-progress count, and the last comment’s author.
See ADR-1555, which also covers the durable review-ids-addressed marker fix (§6.2) for the redelivery this counter is a safety net for.
5. PR Lifecycle Integration
5.1 Draft PR Creation
When: After a stage signals FABRIK_STAGE_COMPLETE, if the stage has create_draft_pr: true. An early-guard path also runs before output is posted when both create_draft_pr: true and post_to_pr: true are set, so the PR exists before postOutputToPR runs.
Code path: processItem() → ensureDraftPR()
Flow (idempotent, up to 3 attempts with exponential backoff):
- Check for an existing open PR via
FetchLinkedPR()— if found open and not merged, ensure body containsCloses #Nand return its number. Closed or merged PRs are ignored; a new PR will be created. - Push the issue branch via
PushBranch() - Build a seed body from
.fabrik-context/files (issue summary, plan approach, verification placeholder). The Summary is## Summary, else the Spec Kit template’s**Input**text, else the issue’s first paragraph; the Problem is## Problem, else## Background, else the first paragraph. The template’s# Feature Specification:title and header-field lines are never used as either section - Create draft PR via
CreateDraftPR()with title from issue, targetingbaseBranch, body ending withCloses #N
Transient errors (network errors, 5xx) are retried with backoff (base delay 500ms, doubled each attempt). Non-transient errors (4xx including 422) return immediately without retry. Returns (prNumber, nil) on success, (0, error) on failure.
Success logging: [#N pr] created draft PR #<num> (branch: fabrik/issue-N)
Failure logging: [#N pr] failed to create draft PR for branch fabrik/issue-N: <error>
5.5 PR Creation Failure Path
Trigger: Stage with create_draft_pr: true emits FABRIK_STAGE_COMPLETE, but ensureDraftPR returns (0, error) after exhausting its in-process retry loop.
Critical invariant: handleStageComplete is NOT called when PR creation fails. The stage does not advance. stage:<X>:complete is not added.
Retry counting: Each PR creation failure increments the same per-stage Attempts counter used by StageRetryIncremented. The StageRetryCleared mutation (which resets the counter) fires only on PR creation success, not on failure. This means PR creation failures count against MaxRetries just like Claude failures.
Code path (completion block in processItem):
FABRIK_STAGE_COMPLETE received
→ if stage.CreateDraftPR and prNumber == 0:
prErr = ensureDraftPR(item, baseBranch)
if prErr != nil:
if MaxRetries > 0:
StageRetryIncremented
if Attempts >= MaxRetries:
escalatePRCreationFailure() → fabrik:paused, stage:<X>:failed, comment
releaseLock()
return
PRCreationFailedRecorded → sets in-memory flag for R5
releaseLock()
return → NO handleStageComplete
else:
updatePRVerification(prNumber, summary)
releaseLock()
StageRetryCleared, EngineUnpaused
handleStageComplete() → stage:X:complete, optional advance
Escalation comment (posted by escalatePRCreationFailure): Names PR creation as the failure cause (not Claude), includes the manual workaround command using the actual base branch: `gh pr create --head fabrik/issue-N --base <baseBranch> --body "Closes #N"`, and instructs the user to remove fabrik:paused to resume.
PRCreationFailed in-memory flag (R5): StageState.PRCreationFailed map[string]bool records that Claude completed a stage but the draft PR could not be created. This flag does not survive engine restarts. It is cleared by StageRetryCleared (on PR creation success). On restart, the item re-runs Claude, which is safe and conservative — Claude’s commits are idempotent.
R5 skip-Claude retry path: On the next poll cycle for an item with PRCreationFailed[stageName] == true, the engine checks this flag early in runStage (before the Claude invocation). If ensureDraftPR now succeeds, the engine calls handleStageComplete directly — no Claude re-invocation needed, since the worktree already has the commits from the prior run. If ensureDraftPR still fails, the retry counter is incremented and the escalation path applies when MaxRetries is reached. Exception: for a fabrik:children-spawned Implement item with zero commits ahead of base (an empty coordinator, §6.7.1), this branch instead routes directly to handleNoWorkNeeded before ever attempting ensureDraftPR — see §6.7.1 for why this case self-heals to Done rather than retrying a doomed PR creation.
| State | Trigger | Action | Outcome |
|---|---|---|---|
| Locked + In Progress | FABRIK_STAGE_COMPLETE + create_draft_pr + PR creation fails |
PRCreationFailedRecorded; lock released; NO handleStageComplete |
Item waits for next poll |
| Locked + In Progress | FABRIK_STAGE_COMPLETE + create_draft_pr + PR creation fails + Attempts >= MaxRetries |
escalatePRCreationFailure: fabrik:paused, stage:<X>:failed, comment |
Issue paused for human intervention |
| Locked + In Progress | PRCreationFailed flag set; ensureDraftPR succeeds on retry (R5) |
StageRetryCleared, handleStageComplete — no Claude run |
Stage advances normally |
5.2 Mark PR Ready
When: After a stage signals FABRIK_STAGE_COMPLETE, if the stage has mark_pr_ready_on_complete: true.
Code path: processItem() → markPRReady()
Flow:
- Push the issue branch
- Find PR number (uses
knownPRfromensureDraftPRif available, elseFindPRForIssue()) MarkPRReady()transitions draft → ready-for-review; retries up to 3 times on transient 5xx errors with exponential backoff (500ms / 1s / 2s); non-transient errors (4xx, including 429) are logged immediately without retry
Durable retry beyond this in-process budget (ADR-1582): if the MarkPRReady() call is still failing once this 3-attempt budget is exhausted — or fails non-transiently on the first attempt — markPRReady durably records the outstanding call via fabrik:awaiting-pr-ready, retried by a dedicated per-poll settle scan (settlePRReadyScan) independent of this stage ever being re-dispatched. See §6.22 for the full mechanism.
Note: This triggers external review bots and populates LinkedPRReviewRequests, which is why the review gate in handleStageComplete() (Path 1) is always optimistic — reviewer data is stale at that point.
5.3 Linked PR Discovery
Fabrik discovers PR comments through the closedByPullRequestsReferences GraphQL field, which traverses issue → linked PRs → PR comments. The Closes #N keyword in the PR body creates this linkage.
MergePR’s own CI precondition (ADR-933). Before this addition, github.Client.MergePR gated only on GitHub’s mergeable field (merge-conflict status), which stays true while required status checks are still pending or failing — a gap that branch protection (enforce_admins) normally masked, but which was live and unmitigated on handarbeit/fabrik (enforce_admins: false, engine token is an org admin). MergePR now self-gates instead of relying on branch protection: after the existing mergeable check passes, it calls FetchPRMergeableFields a second time and refuses to merge — without ever calling the merge endpoint — unless the returned mergeable_state satisfies gh.MergeableStateAccepted ({clean, unstable}). A refusal returns the sentinel gh.ErrNotMergeableCI, deliberately distinct from gh.ErrNotMergeable (the conflict/dirty case) so errors.Is lets callers tell “CI not green” apart from “merge conflict” — a CI refusal must never apply fabrik:rebase-needed or consume a rebase cycle. This self-gate is deliberately unchanged by ADR-1441 (#1441, R3): the merge path continues to defer to mergeable_state alone, per ADR-072’s operator note that a check that should refuse an engine merge belongs in branch protection’s required-checks list, not in MergePR. Only the CI advance gate (checkCIGate, below) tightened — checkCIGate no longer shares this {clean, unstable} allowlist; its own fast path narrowed to mergeable_state == "clean" only (see §2.10, §6.4 rule 9).
This precondition runs identically inside MergePR regardless of caller, so all four call sites inherit it:
attemptMergeOnValidate’s direct-merge fallback (§5.4) — the only call site with no independentmergeable_stategate upstream ofMergePR(reached only whenwait_for_ci: false), so this is where the precondition closes a real, previously-live gap.dispatchRebaseReinvoke’s already-clean-after-rebase fallback (engine/merge_gate.go,reenableAutoMergeAfterRebase) — narrows a TOCTOU window betweenEnablePullRequestAutoMergereturningErrAutoMergeAlreadyCleanand the directMergePRcall.landSingletonandlandMergeTrainBatch(the merge-train landing sequence, §1.3 Queued, FR-2) — both already gated bypollForMergeable’sclassifyLandingCIclassification (ADR-1441), whosemergeable_statecomponent uses the identical{clean, unstable}allowlist for its dirty-check and zero-check-runs fallback, soMergePR’s own check is normally a no-op there; it only fires in a narrow TOCTOU window betweenpollForMergeable’s last poll and theMergePRcall.
None of the four call sites currently branch on ErrNotMergeableCI to escalate — each already logs the failure and lets the item retry on its existing cadence (a full Validate re-invocation for attemptMergeOnValidate, since that call site has no lighter-weight poll-only retry; the next poll’s convergence pass for the rebase-reinvoke fallback; the next merge-train cycle, with members left in Queued, for the landing paths). This is unchanged behavior, not new machinery — fabrik:rebase-needed is applied earlier and independently by checkMergeabilityGate from settle.Status == PRMergeConflicting, never from MergePR’s own return value, so a CI refusal was already incapable of reaching the rebase-cycle path before this change.
5.4 Auto-Merge on Validate (yolo issues)
When: Validate stage completes and the issue has fabrik:yolo (or global cfg.Yolo) and does NOT have fabrik:cruise.
Code path: handleStageComplete() → attemptMergeOnValidate() (Path 1); or catch-up loop Phase 2 → attemptMergeOnValidate() (Path 2 — when fabrik:auto-merge-enabled is absent)
Flow:
- If
fabrik:auto-merge-enabledis already present on the issue: return nil (idempotent — auto-merge was already enabled in a previous invocation). - Fetch linked PR via
FetchLinkedPR(). - Call
EnablePullRequestAutoMerge(owner, repo, pr.Number, AutoMergeStrategy). The strategy defaults toMERGEand is configurable viaFABRIK_AUTO_MERGE_STRATEGY. On failure, two branches apply:ErrAutoMergeNotEnabled(repo-level setting disabled): log a guidance message pointing to Settings → General → Allow auto-merge; return a retriable error (next poll retries). No direct-merge attempt.- Any other error (PR in a terminal GitHub state — e.g. CLEAN, UNSTABLE, or future variants where GitHub refuses to queue auto-merge): log the specific error, then attempt a direct
MergePRcall as a fallback. This is the only call site with no independentmergeable_stategate upstream ofMergePR(reached wheneverwait_for_ci: false), soMergePR’s own CI precondition (“MergePR’s own CI precondition (ADR-933)” above) does real work here. IfMergePRsucceeds: applyfabrik:auto-merge-enabledand return(true, nil). IfMergePRfails withErrNotMergeable(DIRTY PR — a conflict): return(false, err)so existing rebase/CI-fix gates can act on it. IfMergePRfails withErrNotMergeableCI(a required check isblocked, or GitHub is still computing): return(false, err)as well, but this is not a conflict —fabrik:rebase-neededis never applied and no rebase cycle is consumed. SincehandleStageCompletedoes not addstage:Validate:completeon either error, the whole Validate stage simply re-dispatches on the next poll.
- On
EnablePullRequestAutoMergesuccess: applyfabrik:auto-merge-enabledlabel. GitHub now owns the merge decision atomically — no Fabrik-sideMergePRcall. Done advancement is deferred tocheckAutoMergeConvergencein Phase 1 of the catch-up loop.
cruise > yolo precedence: If fabrik:cruise is present on the issue, attemptMergeOnValidate returns immediately without calling EnablePullRequestAutoMerge. Cruise items keep the current stage:Validate:complete behavior: the branch is maintained (rebase reinvoke, CI-fix reinvoke) but merging is left to the user.
See section 5.5 for how Fabrik monitors the convergence flow after auto-merge is enabled.
Non-yolo issues: Items without fabrik:yolo and without fabrik:cruise follow the same behavior as cruise for purposes of merge tracking — checkMergeabilityGate and checkCIGate continue to operate using MaxRebaseCycles / MaxCiFixCycles per-gate cycle limits. EnablePullRequestAutoMerge is never called for these issues. This is unchanged from pre-5.4 behavior.
5.5 Post-Validate Convergence Monitor (yolo issues)
After fabrik:auto-merge-enabled is applied (section 5.4), every Phase 1 iteration of the catch-up loop calls checkAutoMergeConvergence() instead of the legacy checkMergeabilityGate / checkCIGate path. All merge decisions for the issue are delegated to GitHub’s server-side auto-merge logic (legacy native auto-merge) or to GitHub’s merge queue (ADR-058). handleAutoMergeConvergence calls settlePRMergeState() before forwarding to checkAutoMergeConvergence(); the settle result is consumed for merge/CI state interpretation (eliminating the split-brain), though the merge/CI gates remain bypassed — see §6.4. The function is the single mutation point for the convergence/ejection lifecycle (ADR-056 — no parallel scanner is added); the merge-queue ejection classifier (ADR-058 D4) is composed into the same branch ladder.
Convergence budget:
- The budget starts when
fabrik:auto-merge-enabledis applied. The start timestamp is stored durably in GitHub’s issue event log (FetchLabelAppliedAt("fabrik:auto-merge-enabled")), so it survives engine restarts. - The budget is configured via
FABRIK_CONVERGENCE_BUDGET(Go duration syntax, e.g.,30m). Default: 30 minutes. Set to0to disable the budget — Fabrik waits indefinitely for auto-merge to complete (or for the user to disable auto-merge in the GitHub UI).MaxCiFixCyclesis not consulted whilefabrik:auto-merge-enabledis present.MaxRebaseCyclesIS consulted: the convergence path’s rebase dispatch is bounded by the sameMaxRebaseCyclesgate used byhandleMergeAndCIGates, preventing an unbounded rebase loop when a conflict is unresolvable (seeRebaseCycles vs. budgetbelow). - On each Phase 1 iteration:
elapsed = time.Since(budgetStart). Ifelapsed > budget,pauseForConvergenceFailed()fires.
checkAutoMergeConvergence() decision tree (ordered; first match wins):
| # | Observed state | Action |
|---|---|---|
| ① | Terminal-first (settle.Status == PRMergeTerminal \|\| pr.Merged \|\| pr.State == "closed") |
advanceConvergedPRToDone: remove fabrik:auto-merge-enabled + fabrik:rebase-needed, advance to Done. Checked FIRST so a queue merge (which also dequeues the PR) is never misread as an ejection failure (the #913 trap). settle.Status == PRMergeTerminal already re-confirmed the merge via the authoritative single-PR endpoint, catching the window where the REST list endpoint still reports the PR open. On a confirmed merge (never for closed-without-merging), also calls closeIssueIfNonDefaultBase(item, prNumber) (ADR-1096) to explicitly close the issue when its resolved base differs from the repo default — see §2.8. |
| ② | In-queue hand-off (settle.Status == PRMergeQueued) |
Record LastEnqueuedSHA (the SHA GitHub holds in the queue), log, and wait — no label churn. The queue owns the PR. Replaces #935’s inline settle.PR.IsInMergeQueue read (ADR-056 D1 consolidation). Then falls to step ②′. |
| ②′ | Merge-group stall (settle.Status == PRMergeQueued persisting past CIWaitTimeout, ADR-058 D5) |
pauseForMergeGroupStall(): post instructional comment, apply fabrik:paused + fabrik:awaiting-input, remove fabrik:auto-merge-enabled. Guards on cfg.CIWaitTimeout > 0 (default 30 min). Dwell anchor: FetchLabelAppliedAt("fabrik:auto-merge-enabled") — set at first enqueue, survives restarts. Detection signal: settle.Status == PRMergeQueued persisting past the dwell means no merge-group CI ever reported (merge-group runs are observable as a status change away from QUEUED/AWAITING_CHECKS). Idempotency: pauseForMergeGroupStall removes fabrik:auto-merge-enabled, so handleAutoMergeConvergence returns false on the next poll (missing label guard, line 126), and the catch-up loop also skips fabrik:paused items unconditionally. Resume: operator fixes CI (adds on: merge_group), re-queues the PR manually, removes fabrik:paused. On the next Validate re-invoke, attemptMergeOnValidate re-applies fabrik:auto-merge-enabled with a fresh timestamp. ADR-1410 note: this dwell is structurally identical to the CI gate’s own R3/mergeable-state liveness dwells (§6.14.5) — no merge-group CI has ever reported, so there is nothing to observe progress on — which is exactly why it needed no code change when CIWaitTimeout was repurposed for that same class of dwell; it correctly keeps reading CIWaitTimeout directly rather than the new, much larger CIBackstopTimeout. |
| ③ | Auto-merge disabled (!pr.AutoMergeEnabled && !mergeQueueEnabled) |
Apply fabrik:paused + fabrik:awaiting-input; remove fabrik:auto-merge-enabled; post comment with resume options. Guarded on !mergeQueueEnabled: every enqueue-path PR has AutoMergeEnabled == false as its normal state (EnqueuePullRequest never sets auto_merge), so on a merge-queue repo a false flag is NOT a user action — misreading it would pause every ejected PR (a #913-adjacent trap). mergeQueueEnabled = item.LinkedPRIsMergeQueueEnabled \|\| settle.PR.IsMergeQueueEnabled (dual-source). |
| ④ | Budget exhausted | Call pauseForConvergenceFailed() (see below). Applies to both queue and non-queue repos — the ultimate strand safety-net. |
| ⑤ | settle.Status == PRMergeUnsettled |
Wait — GitHub still computing after a push/dequeue; re-evaluate next poll. |
| ⑥ | settle.Status == PRMergeConflicting |
Worker guard (snap.Worker() != nil) → skip if in-flight; cycle-limit check (snap.RebaseCycles(stage) >= MaxRebaseCycles) → pauseForRebaseCycleLimit(); otherwise increment RebaseCycles and dispatch a Claude rebase reinvoke. Shared by queue and non-queue repos; past step ②, a conflicting PR is guaranteed not-in-queue, so #935’s inline in-queue guard here is removed (dead). On a queue repo this is the ejection→resolve path; the clean re-enqueue (⑦) fires later once the resolved PR re-derives clean. |
Ejection-recovery ladder (steps ⑦–⑧) — merge-queue repos only (mergeQueueEnabled == true):
| # | Observed state | Action |
|---|---|---|
| ⑦ | settle.Status == PRMergeBlocked (ejected, re-derived CI failure) |
Worker guard → skip if in-flight; cycle-limit (snap.CIFixCycles(stage) >= MaxCiFixCycles) → pauseForCIFixCycleLimit(); otherwise increment CIFixCycles and dispatch a Claude ci-fix reinvoke. Re-enqueue (⑧) fires later once CI is fixed. |
| ⑧ | settle.Status == PRMergeReady (ejected, re-derived clean) and (leftQueue \|\| (LastEnqueuedSHA != "" && pr.HeadSHA != LastEnqueuedSHA)) |
reEnqueueOrPause — re-enqueue fresh (see below). leftQueue (= priorInQueue) fires the immediate same-SHA ejection re-enqueue; the SHA-change clause fires the post-resolution tail re-enqueue. When neither holds (post-enqueue consistency window: priorInQueue == false and the SHA still equals LastEnqueuedSHA), wait — the PR is already enqueued and GitHub just hasn’t reflected isInMergeQueue == true yet. |
| ⑨ | Non-queue fall-through (any other status) | Wait — GitHub’s native auto-merge is handling it. |
Poll-native ejection detection (the “left the queue” edge): leftQueue = priorInQueue (settle.Status != PRMergeQueued is guaranteed past step ②). priorInQueue is the item’s previous-poll LinkedPRState.IsInMergeQueue, captured in poll.go from the pre-fetch store snapshot before ItemDeepFetched overwrites the store with the current value, then threaded through phase1Ctx → handleAutoMergeConvergence → checkAutoMergeConvergence (ADR-058 D4 OQ-3). Reading “prior” from e.store inside the classifier would yield the already-overwritten current value, silently losing the edge. No webhook is required; dequeued.reason is never consulted for correctness.
reEnqueueOrPause — bounded fresh re-enqueue (FR-3):
- Merged-first re-confirmation at the mutation point (the #913 trap):
FetchPRMerged(authoritative single-PR endpoint) — if merged →advanceConvergedPRToDone; if the call errors → wait (never re-enqueue on an unconfirmed state). The REST list endpoint reportsmerged == falsefor several seconds after a queue merge, so this guard runs right before the dangerous mutation. - Worker-in-flight guard (
snap.Worker() != nil) → skip. snap.EnqueueCycles(stage) >= MaxEnqueueCycles→pauseForEnqueueCycleLimit()(queue-thrash exhaustion).- Otherwise
EnqueuePullRequest(owner, repo, prNum, pr.HeadSHA)at the fresh head SHA (optimistic concurrency fails safe on a stale SHA; skipped if HeadSHA is empty), then applyEnqueueCycleIncremented+PREnqueueRecorded. Re-enqueue is always fresh after off-queue resolution, never re-enqueue-in-place, so conflict-heavy PRs do not starve the queue. An enqueue failure does not increment the cycle.
Rebase reinvoke + auto-merge re-enable:
GitHub disables auto-merge on every push. After dispatchRebaseReinvoke() completes successfully (Claude resolved conflicts, pushed), EnablePullRequestAutoMerge is called again to re-arm auto-merge — but only on non-merge-queue repos (!item.LinkedPRIsMergeQueueEnabled, ADR-058 D4). On a merge-queue repo the recovery path is re-enqueue, not native auto-merge, so the convergence monitor re-enqueues the resolved PR once it re-derives clean (step ⑧); re-enabling native auto-merge there would fight the queue model. This is the only scenario where auto-merge is re-enabled without going through attemptMergeOnValidate.
pauseForConvergenceFailed() — convergence budget exhausted:
Uses settle.PR (from the pre-fetched PRSettleResult) for PR diagnostic fields, FetchCommitsBehind for the commits-behind count, settle.CheckRuns for the CI summary, and the store for the current rebase cycle count — no independent FetchLinkedPR or FetchCheckRuns calls. Posts a structured pause comment containing:
- Total wall-clock elapsed time and configured budget
- Number of rebase reinvokes dispatched
- Commits-behind-base count
- Current
mergeable_state - Latest CI check run summary (passed / failed / pending)
- Three named user options: (1) manual rebase + re-yolo, (2) switch to cruise, (3) leave as-is
Then: applies fabrik:paused + fabrik:awaiting-input; removes fabrik:auto-merge-enabled. Does NOT add fabrik:awaiting-ci (CI may be healthy; the convergence failure is about the merge window, not CI).
RebaseCycles vs. budget: Under the convergence flow, RebaseCycles is incremented on every rebase dispatch. MaxRebaseCycles IS consulted as a gate — the same three-step guard used by handleMergeAndCIGates (in-flight guard → cycle-limit check → dispatch or pauseForRebaseCycleLimit) applies here. When FABRIK_CONVERGENCE_BUDGET=0, the time-based budget check is skipped, but MaxRebaseCycles still bounds rebase reinvokes — so an unresolvable conflict never produces an unbounded rebase loop.
Cap composition (ADR-058 D4 FR-3): Four bounded paths pause independently, first-to-trip wins, no double-pause (each pause returns immediately after applying fabrik:paused, and a paused item is skipped on the next poll):
| Cap | Counter | Increment site | Pause | Catches |
|---|---|---|---|---|
MaxEnqueueCycles (default 5; --max-enqueue-cycles / FABRIK_MAX_ENQUEUE_CYCLES) |
EnqueueCycles |
each fresh re-enqueue trip (step ⑧) | pauseForEnqueueCycleLimit |
queue-thrash loop (enqueue → eject → re-enqueue → eject) that no single sub-cap would catch |
MaxRebaseCycles (default 3) |
RebaseCycles |
each rebase dispatch (step ⑥) | pauseForRebaseCycleLimit |
unresolvable conflict |
MaxCiFixCycles (default 5) |
CIFixCycles |
each ci-fix dispatch (step ⑦) | pauseForCIFixCycleLimit |
persistent CI failure |
ConvergenceBudget (default 30m; 0 disables) |
wall-clock since fabrik:auto-merge-enabled applied, or since the latest fabrik:paused removal if later (#2064) |
n/a (time-based) | pauseForConvergenceFailed |
the ultimate strand safety-net |
EnqueueCycles is dedicated and independent of RebaseCycles/CIFixCycles: the sub-paths increment their own counter when they fire, while EnqueueCycles counts trips through the queue. All four counters are cleared by EngineCyclesCleared (applied by clearFailedStage on unpause/success), so a resumed issue starts fresh. LastEnqueuedSHA (on LinkedPRState) records the head SHA at the last enqueue (recorded both at the in-queue hand-off ② and at each re-enqueue ⑧), making the step-⑧ post-enqueue-window suppression SHA-precise.
State transitions for yolo Validate convergence:
| State | Trigger | New state | Labels added | Labels removed | |
|---|---|---|---|---|---|
| Validate, Complete | Poll tick (Phase 2) | Validate, Convergence | fabrik:auto-merge-enabled |
||
| Validate, Convergence | PR merged (GitHub, incl. via merge queue) | Done, Pending Cleanup | fabrik:auto-merge-enabled, fabrik:rebase-needed |
||
| Validate, Convergence | settle.Status=PRMergeQueued (enqueued / in queue) |
Validate, Convergence (hand-off) | (no churn; queue owns the PR) | ||
| Validate, Convergence | settle.Status=PRMergeConflicting (conflict, below MaxRebaseCycles) |
Validate, Convergence + Rebase in-flight | fabrik:rebase-needed |
||
| Validate, Convergence | settle.Status=PRMergeConflicting (conflict, at MaxRebaseCycles limit) |
Validate, Paused | fabrik:paused, fabrik:awaiting-input |
||
| Validate, Convergence + Rebase in-flight | Rebase push succeeds | Validate, Convergence | |||
| Validate, Convergence | Ejected + re-derived clean (leftQueue/SHA-changed), EnqueueCycles < Max |
Validate, Convergence (re-enqueued) | (merge-queue repos; EnqueueCycles++, LastEnqueuedSHA recorded) |
||
| Validate, Convergence | Ejected + re-derived clean, EnqueueCycles ≥ MaxEnqueueCycles |
Validate, Paused | fabrik:paused, fabrik:awaiting-input |
(merge-queue repos; pauseForEnqueueCycleLimit) |
|
| Validate, Convergence | Ejected + re-derived CI failure (PRMergeBlocked), CIFixCycles < Max |
Validate, Convergence + CI-fix in-flight | (merge-queue repos; CIFixCycles++) |
||
| Validate, Convergence | Budget exhausted | Validate, Paused | fabrik:paused, fabrik:awaiting-input |
fabrik:auto-merge-enabled |
|
| Validate, Convergence | User disables auto-merge in GitHub UI (non-queue repo) | Validate, Paused | fabrik:paused, fabrik:awaiting-input |
fabrik:auto-merge-enabled |
|
| Validate, Convergence | settle.Status=PRMergeQueued persisting past CIWaitTimeout (merge-group stall, ADR-058 D5) |
Validate, Paused | fabrik:paused, fabrik:awaiting-input |
fabrik:auto-merge-enabled |
(merge-queue repos; pauseForMergeGroupStall; operator must add on: merge_group CI + re-queue) |
5.6 Engine-Mechanized PR Creation (FABRIK_PR_CREATE Marker)
Motivation: If the Implement skill creates the draft PR directly via gh pr create, it can silently omit the Closes #N closing keyword. Without that keyword, GitHub’s closingIssuesReferences / closedByPullRequestsReferences fields return empty, breaking every downstream gate (review gate, CI gate, auto-merge). To eliminate this failure class, PR creation is mechanized: the skill emits a structured marker block, the engine creates the PR with Closes #N guaranteed as the first line.
Marker format:
FABRIK_PR_CREATE_BEGIN [owner/repo]
TITLE: <single-line PR title>
<PR body content — no closing keyword>
FABRIK_PR_CREATE_END
BEGINandENDmust each be on their own line.- Optional
owner/repoon theBEGINline (cross-repo target; v1: only same-repo supported). - First non-empty line inside the block must be the
TITLE: <title>header. The key is matched case-insensitively and tolerates surrounding whitespace (title:,Title:, ` TITLE :), via the shared header helpermatchHeader(engine/marker_header.go, ADR 2091) thatFABRIK_SPAWN_CHILD` also uses; the value keeps any later colons. Nothing else about the grammar changed. - Body is everything after the title line. Must be non-empty.
- The skill MUST NOT write any closing keyword (
Closes,Fixes,Resolves+#N); the engine prepends one.
Code path: processItem() → ParsePRCreateBlock() → processPRCreateMarker()
Processing (runs when stage.CreateDraftPR and a FABRIK_PR_CREATE_BEGIN/END block is found in output, before output is posted):
- Cross-repo guard: If the
BEGINline specifies a differentowner/repo, the engine pauses the issue withfabrik:pausedand a “cross-repo PR creation not supported in v1” comment. Does not create a PR. - Idempotency: Call
FetchLinkedPR()— if an open PR already exists onfabrik/issue-N, use it (record in Store, skip create). - Push branch: Call
PushBranch()(non-fatal warning if push fails — mirrorsensureDraftPR). - Compose body:
finalBody = "Closes #N\n\n" + block.Body. This is the mechanized guarantee. - Create PR: Call
CreateDraftPR()with the composed body and verbatim title. Up to 3 attempts with exponential backoff for transient errors. - On success: Cache write-through (
RecordPRLinkage),RegisterEcho("pull_request", "opened"), post acknowledgement comment"🏭 **Fabrik** — opened PR #M"on the issue. - On failure: Apply
fabrik:paused, post error comment, return error.handleStageCompleteis NOT called — the stage does not advance.
After the marker block is processed, the FABRIK_PR_CREATE_BEGIN/END content is stripped from the output string before posting the stage comment.
Malformed marker (missing TITLE, empty body, missing END, invalid owner/repo format): ParsePRCreateBlock returns an error. Each error quotes the line the parser actually found — the first non-empty line, truncated and sanitised for inline rendering by quoteFoundLine (newlines and control characters collapsed, backticks replaced, @ defused with a zero-width space, truncated via sanitizeToolsDeniedCommand) — or says the block was empty: block is empty, missing TITLE line … found <line>, TITLE line has no value, found <line>, PR body is empty after TITLE line <line>. Work is preserved first (ADR 2091): before posting the error, finalizeStageOutcome calls preserveWorkOnMarkerFailure — commitWIP (a no-op on the clean tree a completed Implement normally leaves) then pushBranchOnCancel, which never pauses, so the zero-ahead guard (§”Zero-ahead push guard”) cannot add a second pause. A commit or push failure (including a zero-ahead refusal) is logged only and never suppresses the comment or the pause. Then the engine pauses the issue with fabrik:paused and a comment naming the malformation. Does not create a PR. Un-pausing re-runs Implement on the already-pushed branch rather than from scratch (the stage was never marked complete, and StageAttempted is not recorded on this early-return path, as before).
Fallback to ensureDraftPR: If no FABRIK_PR_CREATE marker is found and stage.CreateDraftPR is true, the engine falls back to the legacy ensureDraftPR path (§5.1). The post-Implement linkage backstop (§5.7) runs in either case.
PRCreationFailed in-memory flag and retry path: When the marker path fails to create a PR, the same PRCreationFailed and R5 retry path as §5.5 applies.
| State | Trigger | Action | Outcome |
|---|---|---|---|
| Locked + In Progress | FABRIK_PR_CREATE block found + valid |
processPRCreateMarker → CreateDraftPR + acknowledgement comment |
PR created with Closes #N as first line |
| Locked + In Progress | FABRIK_PR_CREATE block malformed |
commitWIP + push, then fabrik:paused + comment quoting the found line |
Work on the remote branch; issue paused; no PR created |
| Locked + In Progress | FABRIK_PR_CREATE block → cross-repo target |
fabrik:paused + “cross-repo not supported” comment |
Issue paused; no PR created |
| Locked + In Progress | FABRIK_PR_CREATE block → CreateDraftPR fails |
fabrik:paused + error comment |
Issue paused; no PR created |
5.7 Post-Implement Linkage Backstop
Motivation: Belt-and-braces for the case where a PR exists but lacks the Closes #N keyword — either because a legacy skill called gh pr create directly, because a gh pr edit overwrote the body, or because the engine-mechanized path (§5.6) failed and a fallback PR was opened manually.
Trigger: After the Implement stage signals FABRIK_STAGE_COMPLETE (via either the marker path or the ensureDraftPR fallback), before handleStageComplete() is called. Only runs when stage.CreateDraftPR is true.
Code path: processItem() → verifyAndHealLinkage()
Flow:
- If
prNumber == 0(no PR exists): skip, return true (no linkage to verify). - Fast path (default-branch items only,
!itemHasBaseLabel(item)): callFetchItemDetails(&item)(GraphQLclosedByPullRequestsReferences) to get freshitem.LinkedPRNumber.- If
item.LinkedPRNumber != 0: linkage confirmed — return true. No further calls.
- If
- Otherwise —
item.LinkedPRNumber == 0, or the item carries abase:<branch>label and skipped the fast path entirely — delegate unconditionally toverifyAndHealLinkageByBody(see below).item.LinkedPRNumber == 0is never itself treated as “linkage missing” — it is read only as “the fast path didn’t confirm; fall through to the authoritative body check” (#1598, R1/R3).closedByPullRequestsReferencesis populated asynchronously, so a read taken within seconds of the write that should have set it (PR creation, or a heal that just edited the body) can legitimately still be empty; the old flow trusted that emptiness as definitive and paused a correctly-linked issue. verifyAndHealLinkageByBody: callFetchLinkedPR()to find the PR by branch name (fabrik/issue-N).- If no PR found: user has diverged from the
fabrik/issue-Nnaming convention; log warning and return true (skip heal — do not pause).
- If no PR found: user has diverged from the
- Call
FetchPRClosingIssues()(a direct REST fetch + regex parse of the PR body forCloses/Fixes/Resolves #N— synchronous, no propagation lag).- If the parsed set already contains the issue number: linkage confirmed — return true. No body edit.
- Otherwise call
attemptLinkageHeal:- R2 guard: fetch the current body (
GetIssueBody) and re-check it withgh.ParseClosingIssuesbefore writing anything. If the keyword is already present for this issue — the caller’s own presence check (step 5, or the fast path’sitem.LinkedPRNumber) raced a lagging read — log and return success without writing the body and without touching the idempotency guard. This is what stops the historical duplicate-Closes #Nwrite (#1598, R2). - Body-length guard (FR-015): if
len(currentBody) + len("Closes #N\n\n") > 65,300, pause withfabrik:paused+ “body too long for auto-heal” comment. Return false. - Idempotency guard: check
Snapshot.LinkageHealAttempted(stage.Name, prSHA). If true (a genuine heal — one that actually wrote the body — was already attempted once for this PR head SHA): pause withfabrik:paused+ copy-pastegh pr editrecovery command. Return false. - Record
LinkageHealAttempted{Repo, Number, StageName, PRSHA}mutation in Store (in-memory only), then heal:balanceFences(currentBody), prepend"Closes #N\n\n", callUpdateIssueBody().
- R2 guard: fetch the current body (
- Re-verify (only when a write happened in step 6): call
FetchPRClosingIssues()again (neverFetchItemDetails— re-reading the async derived field a second time is exactly the race this fix removes).- If the parsed set now contains the issue number: post
"🏭 **Fabrik** — PR body auto-corrected: Closes #N prepended"comment, return true. - If still absent: call
pauseForBrokenLinkage()—fabrik:paused+ copy-paste recovery command, with a reason string that is now unambiguous ("auto-heal completed but PR body still lacks the closing keyword") because the body was actually checked, not inferred from a timeout (R4). This condition can now only mean the write itself didn’t take (e.g. a further concurrent body edit) — the derived-field-lag case is structurally excluded by never consulting that field here.
- If the parsed set now contains the issue number: post
- If heal returns false:
handleStageCompleteis NOT called —stage:Implement:completeis NOT applied.
LinkageHealAttempted in-memory flag: StageState.LinkageHealAttempted map[string]string maps stage name to PR head SHA. In-memory only — does not survive engine restart. On restart, one more heal attempt is permitted (conservative / safe). Keyed by SHA so a force-push clears the guard naturally. Not recorded when step 6’s R2 guard finds the keyword already present and skips the write — no attempt occurred, so nothing should count against the guard.
pauseForBrokenLinkage comment: Names PR number, issue number, failure reason, and provides a copy-paste gh pr edit ... --body-file ... command the user can run to manually add the closing keyword. After running the command, the user removes fabrik:paused to resume.
| State | Trigger | Action | Outcome |
|---|---|---|---|
| Locked + FABRIK_STAGE_COMPLETE | Fast path (FetchItemDetails) confirms linkage |
Return true | handleStageComplete proceeds normally |
| Locked + FABRIK_STAGE_COMPLETE | Fast path empty/skipped, but PR body already carries the keyword (FetchPRClosingIssues) |
Return true — no write | handleStageComplete proceeds normally; no duplicate keyword |
| Locked + FABRIK_STAGE_COMPLETE | No linkage in body; PR found on branch; body short; no prior heal | Auto-heal: prepend Closes #N, re-verify via FetchPRClosingIssues |
If verified: comment + advance; if still missing: pause |
| Locked + FABRIK_STAGE_COMPLETE | No linkage in body; body too long (> 65,300 chars) | fabrik:paused + “body too long” comment |
Issue paused for manual recovery |
| Locked + FABRIK_STAGE_COMPLETE | No linkage in body; heal already attempted for this SHA | fabrik:paused + copy-paste recovery comment |
Issue paused for manual recovery |
| Locked + FABRIK_STAGE_COMPLETE | No PR found by branch name | Log warning; return true (user-diverged) | Stage advances normally |
Non-default-base branch (base:<branch>) exception — now the shared path, not a separate implementation: GitHub only populates closingIssuesReferences / closedByPullRequestsReferences for a PR targeting the repository’s default branch — the field is structurally empty for a PR targeting any other branch, regardless of body content (#1046). Before #1598, the default-branch flow (steps 2–9 above) and the base:<branch> flow were two parallel implementations of the same heal/re-verify shape, one trusting the derived field throughout and one trusting the PR body throughout. #1598 collapsed them: verifyAndHealLinkage now only ever owns the cheap item.LinkedPRNumber fast path (step 2, skipped entirely when itemHasBaseLabel(item) is true), and both a base:<branch> item and a default-branch item whose fast path didn’t confirm linkage fall through to the same verifyAndHealLinkageByBody — the body-based confirm/heal/re-verify logic described in steps 4–7. attemptLinkageHeal (step 6) is shared unchanged by both callers. The only remaining default-branch-specific behavior is the fast path itself; everything past it is now identical for both kinds of item. A default-branch item’s log lines read verifyAndHealLinkageByBody: ... whenever the fast path doesn’t confirm — a cosmetic consequence of the shared implementation, not a functional difference.
Review-gate data feed (closed — #1050): This section’s fix addresses linkage verification only. The broader review-gate data feed (LinkedPRReviewRequests, LinkedPRReviews — see §6.1) is nested inside the same closedByPullRequestsReferences GraphQL field and remains structurally empty for base:<branch> repos. That gap is closed by a separate, additive REST-fetch branch inside checkReviewGate (§6.1) — handleBrokenReviewLinkage above now also returns the PR number it resolves, which checkReviewGate reuses to fetch reviews/requested-reviewers directly via FetchPRReviews/FetchPRReviewRequests. LinkedPRReviewThreadComments (the inline per-line comments consumed by review reinvoke) is unaffected — that field remains structurally empty for base:<branch> repos and is out of scope here.
5.8 Merge-Queue Awareness (ADR-058 D3)
When the linked PR’s repository has GitHub’s merge queue enabled, Fabrik’s preemptive rebasing and branch mutations fight the queue: the queue already enforces “up-to-date at merge time,” and any push, rebase, or base-branch change to a PR currently in the queue ejects it. Two guards make every engine-initiated git/PR mutation queue-aware, across all paths (yolo, cruise, manual). Both guards source their signal exclusively from the GraphQL-populated ProjectItem fields — never from e.client.FetchLinkedPR (the REST pulls endpoint does not carry these flags, so it always reports false). The convergence-owner conflict-dispatch site no longer reads settle.PR.IsInMergeQueue inline: D4 (§5.5) consolidates queue membership into the PRMergeQueued settle status, which intercepts an in-queue PR before the conflict branch is reached.
Signal fields:
ProjectItem.LinkedPRIsInMergeQueue— the linked PR is currently in the merge queue (per-PR).ProjectItem.LinkedPRIsMergeQueueEnabled— the repository requires a merge queue (per-repo).
Guard 1 — per-PR in-queue skip (FR-1). When LinkedPRIsInMergeQueue is true, every branch/PR mutation is skipped. Fires on the in-queue signal alone, regardless of the merge_queue config kill-switch — a PR physically in the queue is queued no matter the config (it may have been queued before the operator flipped the switch). Helper: prInMergeQueue(item).
Guard 2 — per-repo stop-preemptive-rebase (FR-2). When LinkedPRIsMergeQueueEnabled is true and cfg.MergeQueue != "off", the preemptive rebase (behind-but-clean) in updateWorktreeFromMain is suppressed — the queue enforces up-to-date at merge time, so it is redundant. The merge_queue: off kill-switch restores legacy preemptive rebasing (mirrors the D2 enqueue kill-switch). Helper: (*Engine).suppressPreemptiveRebase(item).
FR-2 boundary — preemptive rebase vs. conflict resolution. Preemptive rebasing (behind-but-clean) lives exclusively in updateWorktreeFromMain, which git rebase --aborts on any conflict and never resolves it. Genuine conflict resolution lives exclusively in dispatchRebaseReinvoke, gated on settle.Status == PRMergeConflicting (⟺ mergeable_state == "dirty"). A behind-but-clean PR is "behind", never PRMergeConflicting, so suppressing the former (Guard 2) never suppresses the latter. Conflict resolution is therefore preserved and still runs when a PR is actually conflicting and not in the queue; it is skipped (via Guard 1) only when the PR is in the queue.
Guarded mutation sites (completeness — FR-1):
| Site | Mutation | Guard | FR |
|---|---|---|---|
engine/worktree.go updateWorktreeFromMain (via EnsureWorktree skipUpdate) |
per-stage git rebase origin/<base> |
item.go / comments.go OR prInMergeQueue \|\| suppressPreemptiveRebase into skipUpdate |
FR-1, FR-2 |
engine/pr.go ensureDraftPR |
git push --force-with-lease |
pushBranchForNewPR (in-queue skip only; no zero-ahead guard — no PR exists yet) |
FR-1 |
engine/pr.go markPRReady |
git push --force-with-lease |
pushBranchUnlessQueued (in-queue skip, then the zero-ahead guard, #2089) |
FR-1 |
engine/prcreate.go processPRCreateMarker |
git push --force-with-lease |
pushBranchForNewPR (in-queue skip only; runs after the existing-open-PR early return) |
FR-1 |
engine/item.go post-stage push; engine/comments.go spec push (#2034) |
git push --force-with-lease |
pushBranchUnlessQueued (in-queue skip, then the zero-ahead guard, #2089) |
FR-1 |
engine/item.go cancellation WIP push (R8, #1393) |
git push --force-with-lease |
pushBranchOnCancel (in-queue skip, then the zero-ahead refusal without the pause, #2089) |
FR-1 |
engine/pr.go syncPRBase |
UpdatePRBase (PR base change) |
early-return if prInMergeQueue(item) |
FR-1 |
engine/catch_up_handlers.go handleMergeAndCIGates |
dispatchRebaseReinvoke (synthetic rebase + force-push) |
for an in-queue PR settlePRMergeState returns PRMergeQueued, so checkMergeabilityGate clears mergeConflict and the rebase dispatch is never reached; the inline prInMergeQueue \|\| settle.PR.IsInMergeQueue guard at the dispatch site remains as a defensive backstop |
FR-1 |
engine/merge_gate.go checkAutoMergeConvergence |
dispatchRebaseReinvoke (synthetic rebase + force-push) |
D4 (landed): the in-queue PR is intercepted by the PRMergeQueued hand-off (step ②) before the conflict branch — a PRMergeConflicting settle is therefore guaranteed not-in-queue, so the former inline conflict-branch guard is removed (dead). After the PR leaves the queue, the conflict branch dispatches the rebase (the ejection→resolve composition); see §5.5 |
FR-1 |
engine/stages.go attemptMergeOnValidate MergePR fallback |
direct merge | no new guard — the D2 enqueue early-return (cfg.MergeQueue != "off" && pr.IsMergeQueueEnabled) returns before MergePR is reached on a queue repo; a queued PR is never directly merged |
FR-1 (audit) |
Non-mutating (no guard needed): engine/pr_terminal_advance.go rebase-needed handling is removeRebaseNeededLabel only (label cleanup, no git/PR mutation) and runs only after a PR is already merged/closed.
Skill-side complement (ADR-1364): the guards above cover only engine-initiated mutations. The validate skill’s own Pre-Completion Gate rebase is skill-initiated — it runs inside the worker subprocess via gh/git, with no access to the engine’s ProjectItem snapshot — so none of the guards in the table above reach it. ADR-1364 extends the same queue-awareness principle to that site with an in-session gh api graphql check (plus an up-to-date check and a CI-freshness check comparing the base branch’s commit time against the earliest start time among the PR’s required-check runs, requiring every one of them to be currently successful — start time, not completion time, since a long-running check can finish after the base has moved again, and earliest rather than most recent, since one fresh required check does not vouch for a sibling required check that hasn’t rerun), closing a gap where CI-fix re-invocation or fabrik:revalidate could otherwise push an already-enqueued PR out of its queue.
Scope boundary. These guards cover engine-initiated mutations. Claude-driven git push/rebase inside a stage (the default tool allowlist includes Bash(git:*)) is outside engine control. Stages rarely run while a PR is in the queue (queue entry happens at Validate completion), so the realistic exposure is the rebase-reinvoke synthetic comment — which is guarded at dispatch.
WIP-preservation trade-off. Skipping the post-stage push (item.go) when the PR is in the queue means in-flight WIP is not pushed for that cycle. This is acceptable: queue entry happens at Validate completion, so a stage rarely runs while the PR is queued.
Backward-compat (FR-3). Both signal fields are false-by-default, so on a non-queue repo (LinkedPRIsMergeQueueEnabled == false, LinkedPRIsInMergeQueue == false) every guard is a no-op and all rebase/mutation behavior is byte-for-byte unchanged (the ADR-058 D1 guarantee).
Zero-ahead push guard (#2089, ADR 2089)
An existing worktree’s local fabrik/issue-N branch is never synced from origin/fabrik/issue-N: updateWorktreeFromMain fetches and rebases onto the base branch only. If someone pushes to the issue branch out of band, the next engine push uses a bare --force-with-lease, which compares against the shared bare clone’s refs/remotes/origin/fabrik/issue-N — a ref many other paths refresh — so the lease no longer protects the remote. When the stale local branch carries no commits of its own, its head is the base tip, and the push makes the remote issue branch identical to base; GitHub then closes the PR as having no diff (report #2058).
pushBranchUnlessQueued (engine/merge_queue.go) therefore applies a second guard after the in-queue skip, zeroAheadPushRefused (engine/push_guard.go). It is a safety net, not a resync: it never fetches the issue branch, fast-forwards or resets the local branch, or sets an explicit lease SHA.
Trigger. The push is refused only when all of these hold, checked cheapest first:
- the base branch resolves (
baseBranchForItem, so abase:<branch>label is honoured); - the worktree’s HEAD is literally zero commits ahead of
origin/<base>(WorktreeManager.CommitsAheadOfRef,git rev-list --count, underwm.mu). This is notcommitsAheadOfBase: spec-only commits (#921) count as ahead here, because a refusal must be conservative; and - a live
FetchLinkedPRread (only when the count is zero) reports an open, unmerged PR.
Fail-open. An unresolvable base, a git error, or a PR read error all let the push proceed (logged as [#N push-guard] warn: …), as do no PR and a closed or merged PR. A branch that is only behind base but carries its own commits is not zero-ahead and pushes.
Refusal. The guard logs [#N push-guard] REFUSING to push …, leaves the remote ref untouched, pauses the item with fabrik:paused + fabrik:awaiting-input through pauseIssue, and posts one 🏭 **Fabrik — push refused: local branch has no commits** comment saying the local branch carries no commits ahead of origin/<base>, that the remote was left untouched, and naming the PR. It does not assert a cause (a rebase that dropped already-absorbed commits also yields zero ahead). Comment dedup follows #1408: if the item already carries that comment (hasPauseComment) the labels are re-applied (reapplyPauseLabels) without a second comment. pushBranchUnlessQueued returns ErrPushRefusedZeroAhead. A human comment after the pause resumes the item (ADR-1813); if the branch is unchanged the guard fires again.
Callers.
| Caller | Treatment |
|---|---|
item.go post-stage push (every stage that ran Claude) |
Guarded. On refusal processItem releases the lock and returns before the completion/blocked/retry chain, so handleStageComplete cannot strip fabrik:awaiting-input or advance/auto-merge a paused item; the stage stays incomplete. |
item.go cancellation WIP push |
Guarded through pushBranchOnCancel: the push is refused (remote untouched) and logged, but the item is not paused or commented from here — the daemon-stop / TUI-stop interruption that cancelled the worker already pauses it under pauseIssueMu, and a second pause would race it. The lock is released as before. |
comments.go publishCommentOutput spec push (#2034) |
Guarded; a refusal is logged. |
pr.go markPRReady |
Guarded; on refusal it returns before MarkPRReady (a stale PR is not flipped to ready-for-review) and returns true so its callers can act on it. All three callers (item.go post-stage completion, item.go R5 PR-creation retry, comments.go comment-path completion) skip handleStageComplete on true, so the paused item is neither advanced nor auto-merged; the comment path also restores the rework marker state via endStageRework(…, false). TestPushGuard_MarkPRReadyCallersActOnRefusal pins that every call site uses the result. |
pr.go ensureDraftPR, prcreate.go processPRCreateMarker |
Exempt (pushBranchForNewPR, in-queue skip only). They push because no open PR exists yet, so a zero-ahead push (for example a coordinator parent) is legitimate and there is no PR for the overwrite to close. |
Known gap. The guard catches only the zero-ahead shape. A stale local branch that carries commits of its own can still overwrite out-of-band work. Syncing the issue branch from origin before work and pushing with an explicit lease on the synced SHA (the logic PrepareCatchUp already has for the train path) is the follow-up, and is expected to subsume this guard.
Test seam. SetPushZeroAheadGuardDisabledForTest neutralises the guard so the refusal tests can show they bite (engine/push_guard_test.go, tests/sim/push_guard_test.go).
6. Review Gate and Review Reinvoke
6.1 Two-Phase Review Gate
The review gate has three paths that handle different timing scenarios:
Path 1: handleStageComplete() (inside worker goroutine)
- Runs immediately after a stage completes
- Reviewer data is STALE (reviewers are assigned only after
MarkPRReady, which just ran) - Optimistically applies
fabrik:awaiting-reviewlabel only whenwait_for_ci: false(stages withwait_for_ci: trueskip Path 1 seeding — Path 2 handles the gate after CI clears, avoiding spurious re-application offabrik:awaiting-reviewfrom stale data; #617) - Returns without advancing — defers to Path 2
Path 2: Catch-up loop in poll() (in poll goroutine)
- Runs on subsequent poll cycles for items with
stage:<X>:complete - Has FRESH reviewer data from
FetchItemDetails()(bothLinkedPRReviewRequestsandLinkedPRReviews) LinkedPRReviewRequestsis kept current via two independent paths: (1) an explicitpull_request.review_request_removedwebhook fires andapplyPullRequestDeltaappliesPRReviewRequestRemoved; (2) anypull_request_reviewevent (action=submitted,edited, ordismissed) is received —applyPullRequestReviewDeltaappliesPRReviewRequestRemovedas a side effect, even when thereview_request_removedside-effect webhook does not arrive (observed reliably for Copilotedited-action reviews). This means the cache can reflect the cleared reviewer list within seconds of a review arriving, without waiting for the next deep-fetch cycle.- Calls
checkReviewGate()for the real gate evaluation - Gate clears only when
len(outstanding) == 0AND at least one non-DISMISSED review exists. This means: no requested reviewers are outstanding AND at least one review withState != "DISMISSED"has been submitted. Waiting on submitted reviews (not just requests) is what catches bot reviewers like Copilot and Gemini that self-trigger via webhooks without ever appearing in the formal requested-reviewer list. DISMISSED reviews are excluded from this check because a dismissed review indicates the reviewer’s prior response was revoked — the gate must re-block until a new non-dismissed review arrives or the reviewer re-submits. This filter is necessary because thedismissedwebhook action is now processed (not silently dropped), and there is a race window between apull_request_review.dismissedevent and the subsequentpull_request.review_requestedevent that re-adds the reviewer to outstanding.reviewGateOutstanding(reviewRequests, reviews)is a pure function over explicit slices (notgh.ProjectItem) —checkReviewGateselects which slices to pass, per the base:branch below. This clearing condition itself is unchanged by `expected_reviewers` (below); only the *waiting* behavior that precedes it changes. - FR-2 fast path —
expected_reviewers: []. Immediately after computingoutstanding/hasReviews,checkReviewGate(and, identically,reviewGateBlocksLandingin Path 3) calls the pure functionreviewGateFastAdvance(outstanding, hasReviews, expected) bool, whereexpectedise.effectiveExpectedReviewers(item, stage)— the per-issueexpected-reviewers:<mode>label override (#1304) resolved on top ofstage.ExpectedReviewers; see the “Per-issue override” bullet below. It returns true — and the gate clears with(false, false, false)exactly as “cleared naturally” below — only whenoutstandingis empty,hasReviewsis false, and the stage explicitly declaredexpected_reviewers: [](see “Declared unrequested reviewers” below). A reviewer actually requested on the PR (outstandingnon-empty) or an already-submitted review always takes precedence over the declaration, so this narrows waiting for unrequested reviewers only — it is not equivalent towait_for_reviews: false, which ignores a requested reviewer outright. Placed afterhandleBrokenReviewLinkage/the base:REST-fallback resolution in both call sites, so it never fires on unconfirmed PR linkage. The advance is recorded via a distinct `e.logf(..., "awaiting-review", "expected_reviewers declared none expected and nothing was requested — advancing immediately\n")` line (the trail is `fabrik.log`/the TUI, not a new PR comment — every other "gate cleared naturally" path already posts nothing). - On a default-branch item, the slices passed to
reviewGateOutstanding/reviewGateAllBotsareitem.LinkedPRReviewRequests/item.LinkedPRReviews— the GraphQL-sourced fields, kept current via two independent paths: (1) an explicitpull_request.review_request_removedwebhook fires andapplyPullRequestDeltaappliesPRReviewRequestRemoved; (2) anypull_request_reviewevent (action=submitted,edited, ordismissed) is received —applyPullRequestReviewDeltaappliesPRReviewRequestRemovedas a side effect, even when thereview_request_removedside-effect webhook does not arrive (observed reliably for Copilotedited-action reviews). This means the cache can reflect the cleared reviewer list within seconds of a review arriving, without waiting for the next deep-fetch cycle. - On a
base:<branch>item (#1050):closedByPullRequestsReferences— and everything nested inside it, includingreviewRequests/latestReviews— is structurally empty there (GitHub only populates it for PRs targeting the repository default branch), soitem.LinkedPRReviewRequests/item.LinkedPRReviewsare always empty regardless of the PR’s actual review state. WhenitemHasBaseLabel(item)is true andhandleBrokenReviewLinkage(§5.7) has resolved a PR number > 0,checkReviewGateinstead callse.readClient.FetchPRReviews(owner, repo, prNumber)ande.readClient.FetchPRReviewRequests(owner, repo, prNumber)— REST endpoints keyed on PR number, unaffected by the PR’s base branch — and passes those slices toreviewGateOutstanding/reviewGateAllBotsinstead.FetchPRReviewRequests’sIsBotreproduces the GraphQL path’s dual signal (RESTuser.type == "Bot", with the same login-pattern fallback). If either REST call errors, the poll is treated conservatively as no-data (nilslices, gate stays blocked) rather than trusting a partial success — a warning is logged and the fetch retries on the next poll. Becauseitem.LinkedPRNumberis always 0 on abase:<branch>repo, this PR-number resolution is not a one-time linkage-repair path — it is the steady-state lookuphandleBrokenReviewLinkageperforms on every poll while the gate is open. Partially out of scope, updated by #1375:LinkedPRReviewThreadComments(the inline per-line comments consumed by review reinvoke, §6.2) is unaffected by this fix and remains structurally empty forbase:<branch>repos —buildReviewThreadCommentshas no REST fallback and this stays an accepted, pre-existing gap.buildReviewBodyComments(§6.2, Finding 4), however, now has its own REST fallback mirroring this one exactly: it resolves the PR number fromitem.LinkedPRNumberor, when zero, a side-effect-freeFetchLinkedPRcall (safe becausehandleReviewGateonly reaches it aftercheckReviewGate’s own linkage check already ran this same poll without terminating), then callsFetchPRReviewsdirectly — so aCHANGES_REQUESTEDreview’s body is picked up on abase:<branch>repo too, closing the gap a Pruefer review comment identified during #1375: without this, the reinvoke-before-block fix (Finding 1) would have been silently unreachable forbase:<branch>stages, sincebuildReviewFeedbackCommentswould always see an emptyitem.LinkedPRReviews. ReviewRequest.IsBotis populated fromrequestedReviewer.__typename == "Bot"in the GraphQL query, with a login-pattern fallback (*[bot],*-bot,copilot-*,dependabot,gemini-code-assist). This drives the bot-aware escalation ladder.- Broken-linkage guard (FR-013/FR-014): Before evaluating reviewers,
checkReviewGate()checks whetheritem.LinkedPRNumber == 0. If so, it callsFetchLinkedPR()to look for a PR on thefabrik/issue-Nbranch. If a PR is found open and unmerged,handleBrokenReviewLinkagefirst parses the PR body withFetchPRClosingIssueson every path (#1962). It no longer trustsclosedByPullRequestsReferences, which can lag a new PR, be a stale cached snapshot, or, as on 2026-09-30, not be populated at all. If the body references the issue, the linkage is fine and the gate proceeds. If the body read fails, nothing is paused and the check retries next poll. Only a body with no closing keyword counts as broken: on a default-branch item, the issue is paused withfabrik:pausedand a comment —"🏭 **Fabrik** — PR #M exists on branch fabrik/issue-N but is not linked to this issue via a closing keyword. Add Closes #N to the PR body and remove fabrik:paused to resume."— and(false, false, true)is returned.fabrik:awaiting-reviewis NOT applied (FR-014). Theterminated=truevalue (ADR-1223) signals that processing was already terminated via a directpauseIssuecall, distinct from “gate cleared naturally” —handleReviewGatechecks it first and claims the item so Phase 2 does not advance it in the same poll pass. On abase:<branch>item (itemHasBaseLabel(item)true —closedByPullRequestsReferencesis structurally empty there regardless of the PR body, soitem.LinkedPRNumber == 0alone does not indicate broken linkage),handleBrokenReviewLinkagefirst callsFetchPRClosingIssuesto parse the PR body directly: if it already containsCloses #N, linkage is confirmed and the function returns(false, prNumber)so the gate falls through to normal reviewer evaluation using the REST branch above (no pause); only if the body genuinely lacks the closing keyword does it pause, with wording that explains the base-branch limitation instead of implyingclosingIssuesReferencesalone is diagnostic. If no PR is found on the branch (either case): gate falls through to the normal reviewer evaluation path withprNumber == 0, so the REST branch above is skipped and the (empty) GraphQL-sourced slices are used — same behavior as before #1050 in that specific sub-case. - Default-base items with a missing link (#1962, round 4): the same REST fallback (
FetchPRReviews/FetchPRReviewRequests, keyed on the PR numberhandleBrokenReviewLinkageresolved) also applies on the default base whenreviewDataRidesMissingLink(item)holds, meaningLinkedPRNumber == 0and both GraphQL review slices are empty. The nested review data rides the sameclosedByPullRequestsReferenceslink, so when the link is missing that data is empty whatever the PR’s real review state. Once R1 stopped pausing on a body-confirmed link, the gate would otherwise have evaluated “no reviews” and waited out the timeout. The feedback gate’sresolveReviewsForFeedbackChecked(#1953) uses the same predicate. There, a failedFetchLinkedPR/FetchPRReviewssetsLive=falseand holds the landing decision (deferred=true); it never clears on unknown state. GraphQL review data that is present is still trusted. checkReviewGate()returns(blocked, timedOut, terminated bool)(ADR-1223). Four outcomes:(blocked=true, timedOut=false, terminated=false)— still waiting;fabrik:awaiting-reviewmaintained. Either outstanding requested reviewers remain, or no reviews submitted yet (bots may still be processing). Also returned after Phase 1 (bot re-prompt fired, waiting for Phase 2 window).(blocked=false, timedOut=false, terminated=false)— gate cleared naturally;fabrik:awaiting-reviewremoved;fabrik:bot-repromptedlabel cleaned if present; advance or reinvoke.(blocked=false, timedOut=true, terminated=false)— gate cleared by timeout;fabrik:awaiting-reviewremoved;pauseForReviewTimeout()pauses issue. Fires at1× ReviewWaitTimeoutfor mixed/pure-human outstanding (existing path) or at2× ReviewWaitTimeoutfor pure-bot outstanding (Phase 2 — after the re-prompt window expired).(blocked=false, timedOut=false, terminated=true)— processing already terminated via a directpauseIssuecall insidehandleBrokenReviewLinkage(see broken-linkage guard above). Before ADR-1223 this branch returned the same all-false tuple as “gate cleared naturally,” sohandleReviewGatedid not claim the item and could fall through tobuildReviewThreadComments/Phase 2, potentially dispatching a review reinvoke or advancing an item that had just been paused in the same poll pass.handleReviewGatenow checksterminatedfirst, ahead ofblocked/timedOut, and claims (return true) immediately when it is set.
Path 3: reviewGateBlocksLanding() inside attemptMergeOnValidate (landing decision)
- Runs immediately before any landing action for a Validate item: auto-merge enable, merge-queue enqueue, direct merge, or advance-to-
Queued. Positioned ahead of themerge_trainfork, so both modes are gated identically — turning the merge train on cannot weaken the gate. - Exists because Paths 1 and 2 leave a hole whenever
wait_for_cidefers completion: within the single poll pass that clears CI, Path 2 has already skipped itself on the frozenpctx.hasComplete, and Phase 2 reaches the landing decision in that same iteration. See §6.6.6 for the full mechanism (#1216). - Re-fetches review state live (
FetchPRReviews/FetchPRReviewRequests) rather than readingitem.LinkedPRReviewRequests/LinkedPRReviews, becauseattemptMergeOnValidate’s two callers have different freshness guarantees — and because a reviewer requested during the CI-await window must still block. PR number comes fromitem.LinkedPRNumber, falling back toFetchLinkedPRonly when it is 0. - Uses the same
reviewGateOutstandingclearing condition as Path 2, so the two sites cannot disagree. Blocks conservatively on any fetch error — the review fetches (both slices discarded) and theFetchLinkedPRfallback alike. Only a PR that is definitively absent lets the landing through. - Blocks and labels only — applies
fabrik:awaiting-reviewidempotently and returns(false, nil). It never runs the escalation ladder or the timeout: once it blocks,stage:X:completeis present, so Path 2 claims the item on the next poll and owns escalation from there.
Declared unrequested reviewers (expected_reviewers, #1283). Every real-world review bot — Pruefer, Gemini, CodeRabbit, Copilot — is unrequested and asynchronous: it polls or receives a webhook and posts a review without ever being formally requested, so it never appears in ReviewRequest/outstanding. Before #1283, reviewGateAllBots(reviewRequests, outstanding) computed allBots := len(outstanding) > 0, which is unconditionally false whenever nothing was formally requested — the escalation ladder below was therefore unreachable dead code for every self-submitting bot, the exact case it exists to handle; fabrik:bot-reprompted had been applied to zero issues, ever, in production. GitHub has no representation for “a reviewer will turn up here unrequested” (requesting these bots as reviewers is not possible; CODEOWNERS requires collaborator permissions a GitHub App does not hold; inferring from history fails open on a cold-started repo) — the information exists only in the operator’s head, so it must be declared.
- Declaration (
stages.Stage.ExpectedReviewers *[]string, YAMLexpected_reviewers). Mirrors the codebase’s*booltri-state idiom (WaitForReviews,WaitForCI):nil(key absent) — undeclared, unchanged default behavior (below);&[]string{}(expected_reviewers: []) — explicitly no unrequested reviewer is expected, enabling the FR-2 fast path above; non-empty — one or more declared identities, satisfied when any one responds (matches the existing any-authorhasReviewsclearing condition — requiring all would turn N declared bots into N independent points of failure, recreating the SPOF tracked in #1071). Only meaningful alongsidewait_for_reviews: true; inert otherwise, mirroringreview_authority’s precedent. - Load-time validation (
validateExpectedReviewers,stages/stages.go, FR-8). Each declared identity is trimmed and must be a bare, mention-resolvable handle: non-empty, no leading@, no trailing[bot]/[Bot]/[BOT]suffix (case-insensitive). A malformed identity failsLoadAlland stage load entirely — startup fails rather than silently never notifying anyone. This rejection is not academic: Fabrik’s own first-party bot Pruefer’s actual submitted-review author is<slug>[bot](confirmed forhandarbeit/fabrik’s own installation via both GraphQL, whereBot.loginomits the suffix, and REST, whereuser.logincarries it) — the declared form must be the suffix-free one, which is also GitHub’s live@-mention surface. - Identity matching (
reviewerIdentityMatches,stripBotSuffix,declaredReviewersOutstanding,engine/reviews.go). A declared identity is matched against a livePRReview.Authorby stripping a trailing[bot]from the author side, applying the existingbotMentionHandlecopilot-collapse to both sides, then comparing case-insensitively — tolerating both the suffixed (REST) and unsuffixed (GraphQL) forms the same login can arrive in.declaredReviewersOutstanding(declared, reviews) []stringreturns the declared names not yet matched to a non-DISMISSED review; this is the declared analogue ofoutstanding, computed independently of it (a declared identity is never a member ofReviewRequest, so it cannot be represented insideoutstandingitself). reviewGateAllBotsextended (FR-3). Signature is nowreviewGateAllBots(reviewRequests []gh.ReviewRequest, outstanding, declaredOutstanding []string) bool. Whenoutstandingis non-empty, behavior is byte-for-byte unchanged — declared reviewers never interfere with formally-requested-reviewer classification. Whenoutstandingis empty, it now returnslen(declaredOutstanding) > 0instead of unconditionalfalse— the one-line fix that makes the ladder below reachable for a declared self-submitting bot.- A declaration is not evidence the reviewer exists (FR-6). A declared-but-uninstalled bot is indistinguishable, at the gate, from a slow one: it still runs the same escalation ladder and still times out, naming the bot that never appeared.
fabrik:bot-repromptedidempotency guard is unrelaxed. The label remains the single fixed, bot-agnostic guard described below; adding the declared-reviewer re-prompt path does not introduce a second label or a per-declaration guard — this is what prevents the runaway-mention class tracked in #1083/#1088.- Per-issue override — the
expected-reviewers:<mode>label (#1304, extending #1283).expected_reviewersis a stage-YAML-only setting by default, applying repo-wide to every issue on that stage.expected-reviewers:none/expected-reviewers:declaredlets a single issue opt into (or out of) a declared value without editing stage config, mirroringreview-authority:<mode>’s per-issue override idiom (§6.1.1).extractExpectedReviewersOverride(issueNumber int, labels []string) *[]string(engine/reviews.go) scansitem.Labelsfor theexpected-reviewers:prefix:noneresolves to&[]string{};declaredresolves to&[]string{expectedReviewersSyntheticName}("e2e-synthetic-declared-reviewer"— a fixed testing/e2e identity that never posts a real review; applying it to a production issue runs out the full re-prompt ladder before pausing). Precedence followsextractReviewAuthorityOverride’s “pick deterministically, don’t arbitrate” convention: no label → stage config governs (nilstaysnil— FR-5 default preserved); exactly one recognized label → it overrides the stage config for this issue only; bothexpected-reviewers:noneandexpected-reviewers:declaredpresent → resolves todeclared(the more restrictive mode — it imposes waiting/re-prompt-ladder obligation, unlikenone’s immediate fast-advance), with a logged warning; a label whose suffix is neither exactlynonenordeclared(typo, casing, unknown value) → ignored with a logged warning, falls back to stage config — never a hard failure, never a silent escalation todeclared.effectiveExpectedReviewers(item gh.ProjectItem, stage *stages.Stage) *[]stringcomposes the extractor with the stage fallback and is the single resolution pointcheckReviewGate,reviewGateBlocksLanding, andpauseForReviewTimeout’s FR-4 status message all consult — none of the three readsstage.ExpectedReviewersdirectly anymore, so the advance gate, the landing gate, and the message describing declared-reviewer status can never disagree about which value applies to a given issue. The override changes nothing about whether the gate applies (wait_for_reviews: trueon the stage is still required) — only which reviewers are expected once the gate is active.
Bot-aware escalation ladder:
The ladder engages whenever reviewGateAllBots is true and item.LinkedPRNumber > 0 — either every outstanding requested reviewer is a bot (detected via ReviewRequest.IsBot, unchanged from before #1283), or nothing was requested and at least one declared expected_reviewers identity has not yet responded (declaredOutstanding, new in #1283). checkReviewGate applies a two-phase escalation instead of immediately pausing:
A declared bot never preempts an authoritative human escalation (Finding 2, #1375). At the checkReviewGate call site, the ladder’s allBots is computed as reviewGateAllBots(reviewRequests, outstanding, declaredOutstanding) && authorityReason == "" — not reviewGateAllBots(...) alone. authorityReason (§6.1.1) is only ever non-empty once a requested human has already responded but with a verdict authoritative mode does not accept (outstanding is empty by the time it’s set), so this cannot change behavior for the “human hasn’t responded yet” case — outstanding non-empty already forces allBots=false via the per-reviewer loop below. It only closes the gap where a human has responded with a blocking verdict and a stage-declared bot is separately still outstanding: before this fix, reviewGateAllBots’s declared-reviewer branch (len(outstanding)==0) saw only the bot and returned true, routing the item into the bot re-prompt ladder instead of the human-escalation timeout path — deferring the human’s CHANGES_REQUESTED verdict behind a full ReviewWaitTimeout bot-reprompt cycle that cannot possibly resolve it. reviewGateAllBots’s own signature and pure-function contract are unchanged; the gate is applied at the call site, preserving ADR-1283’s “shared function, minimal call-site divergence” discipline.
-
Phase 1 (fires at 1×
ReviewWaitTimeoutfromfabrik:awaiting-review): For each outstanding requested bot reviewer: deletes then re-adds the formal review request (DELETE+POST torequested_reviewers— the delete-then-add cycle is required to re-trigger the bot’s webhook; a plain POST is a silent no-op if the reviewer is already listed), posts an@<login> just checking incomment on the linked PR, usingbotMentionHandle(login)for the mention surface. For each declared-but-unrequested reviewer still outstanding (skipping any name already re-prompted above via the formal-request path, so a reviewer that is both requested and declared isn’t mentioned twice): posts the same@<name> just checking incomment directly, using the declared identity verbatim (already FR-8-validated to be mention-resolvable) — noDeleteReviewRequest/AddReviewRequestcall is made, since there is no GitHub review request to mutate for a reviewer that was never formally requested. This direct-mention path does not conflict withneutralizeBotMentions/#1141 (engine/mentions.go), which is scoped to Claude’s freeform stage output, not this engine-authored re-prompt — Phase 1 already posted a live mention for requested bots before #1283. After processing both groups, applies the single fixed labelfabrik:bot-reprompted(idempotency guard — ≤50 chars, GitHub REST API limit) exactly once. Returns(true, false)— still blocked. Phase 1 fires once per gate cycle (idempotency enforced by presence offabrik:bot-reprompted) regardless of which group(s) triggered it. -
Phase 2 (fires at 1×
ReviewWaitTimeoutfromfabrik:bot-reprompted): If no bot — requested or declared — has responded after a full additionalReviewWaitTimeoutwindow,fabrik:bot-repromptedis removed,fabrik:awaiting-reviewis removed, and(false, true)is returned. The caller firespauseForReviewTimeout(), which detects Phase 2 context from the pre-cleanupitem.Labelssnapshot and posts a named, contextual pause comment explaining that a re-prompt was already attempted. Bot logins in the Phase 2 comment are derived fromLinkedPRReviewRequests(requested bots that haven’t responded are still in that list) plusdeclaredOutstanding(declared-but-unrequested bots that never appear there at all) — folding both sources together is what lets the Phase 2 message flavor fire for a purely-declared reviewer, not just a formally requested one. Human-in-the-loop is preserved — the engine never auto-advances past await_for_reviews: truegate.
Mixed bot+human outstanding: Phase 1 does NOT fire. The gate falls through to the existing pauseForReviewTimeout() at 1× ReviewWaitTimeout from fabrik:awaiting-review, unchanged. Re-prompting humans is not appropriate (they have inbox notifications; webhooks don’t apply).
pauseForReviewTimeout() messaging (FR-4): In all timeout paths (Phase 2, mixed, pure-human), the pause comment names all requested outstanding reviewers and tags each as (bot) or (human) for easy triage, as before #1283. When the stage declares expected_reviewers (non-empty), the message additionally appends a per-declared-reviewer status line — e.g. Expected reviewers: `handarbeit-pruefer` reviewed; `gemini-code-assist` did not respond — built from live review data (resolvedReviewData, the same REST-fallback helper the authoritative-mode verdict messaging uses) via declaredReviewersOutstanding, so a partial response across multiple declared reviewers is diagnosable. This line is unconditional on which timeout branch fired (Phase 2 / mixed / generic), and replaces the old, pre-#1283 message’s generic “no reviewers were requested” framing — which was true but told the operator nothing about why nobody was requested. In Phase 2, the comment additionally notes that a re-prompt was already attempted and provides four recovery options.
ReviewWaitTimeout semantics (depends on outstanding-reviewer mix):
| Outstanding reviewers | Meaning of ReviewWaitTimeout |
|---|---|
| Pure bot(s) — requested, declared, or both | How long before Phase 1 fires (bot re-prompt); Phase 2 pause triggers at 2× (1× from fabrik:awaiting-review + 1× from fabrik:bot-reprompted) |
| Mixed bot(s) + human(s) | How long before the engine pauses for human input (existing behavior, Phase 1 does not fire) |
| Pure human(s) | How long before the engine pauses for human input (existing behavior, unchanged) |
| Nothing requested, nothing declared/expected (undeclared stage) | Same as pure bot(s) above — FR-5’s unchanged default |
Nothing requested, expected_reviewers: [] declared |
N/A — the FR-2 fast path advances immediately instead |
Label lifecycle for bot escalation:
fabrik:bot-reprompted— single fixed label (22 chars, well under GitHub’s 50-char REST API limit); applied once after Phase 1 finishes re-prompting all outstanding bots (requested and/or declared); removed when Phase 2 fires (as part of Phase 2 cleanup), when the gate clears naturally viaremoveAwaitingReviewLabel(all reviewers submitted), or when the issue is closed (defensive sweep bycleanupClosedIssueTransientLabels). Only exists while the bot escalation is in progress within a gate cycle.fabrik:awaiting-review— applied on first block; removed on natural gate clear (including the FR-2 fast-advance path) or when Phase 2 fires; persists while the issue is paused (mixed/human/Phase-2 timeout paths —pauseForReviewTimeoutdoes not remove it).
Startup notice for under-specified stages (stages.WarnUndeclaredReviewers, FR-7). On every startup, alongside the existing stage-drift check, Fabrik emits a one-line [startup] notice: for each wait_for_reviews: true stage with ExpectedReviewers == nil (undeclared). This is deliberately not part of drift detection (WarnStageDrift/FilterNoOpKeys), which stays silent by design here — an omitted declaration is a behaviorally-identical no-op under FR-5, so drift correctly has nothing to say about it. The notice answers a different question (“is this config under-specified?”, not “is this config outdated?”) and uses the same warnings.Record/warnings.Clear self-limiting idiom, keyed "undeclared_reviewers:" + stage.Name: informational only, never blocking, and it disappears the moment a declaration — including an explicit expected_reviewers: [] — is added for that stage.
6.1.1 review_authority: Verdict-Aware Clearing (Authoritative Mode)
Everything in §6.1 describes advisory mode — the default, and the only mode that existed before ADR-1250. Advisory clears the gate on existence: len(outstanding) == 0 && hasReviews, regardless of what any review actually says. A CHANGES_REQUESTED review does not, by itself, block advancement; a required reviewer only needs to submit, not approve. This is why advisory mode’s self-review escape hatch (§2.3) works at all: a COMMENTED review from the PR author is existence, not approval, and GitHub permits self-COMMENT while forbidding self-approval. Authoritative mode (below) layers an approval requirement on top of existence — on a repo with no reviewer and no branch-protection review requirement, a self-review still clears authoritative mode’s own-repo fallback verdict check (the “Fabrik’s own fallback” bullet below), since that check only fails on an active CHANGES_REQUESTED, not on the reviewer’s identity.
review_authority: authoritative is a per-stage YAML opt-in (stages.Stage.ReviewAuthority, {"", "advisory", "authoritative"}, default ""/advisory) that makes the same clearing branch additionally verdict-aware. It is orthogonal to Fabrik’s autonomy controls (yolo/cruise/manual) — autonomy answers “does Fabrik proceed without a human clicking go?”; authority answers “what must be true before proceeding is allowed?” See ADR-1250.
Where the check lives. reviewGateAuthorityVerdict(reviewDecision string, reviews []gh.PRReview) (satisfied bool, reason string) (engine/reviews.go) is a pure function called from inside the existing len(outstanding) == 0 && hasReviews branch of both checkReviewGate (Path 2, the advance gate) and reviewGateBlocksLanding (Path 3, the landing gate) — never as a parallel or replacement check. This is strictly additive: a PR with zero reviews stays blocked by the outer advisory condition regardless of mode, and advisory mode’s existing behavior is completely untouched (the new check is gated on effectiveReviewAuthority(item, stage) == "authoritative" and never runs otherwise — zero extra API calls, zero behavior change for advisory/default repos).
Per-issue override — the review-authority:<mode> label (#1261, extending ADR-1250). review_authority is a stage-YAML-only setting by default, applying repo-wide to every issue on that stage. review-authority:advisory/review-authority:authoritative lets a single issue opt into (or out of) authoritative mode without editing stage config, mirroring the established model:<name>/effort:<level>/base:<branch> per-issue override idiom. extractReviewAuthorityOverride(issueNumber int, labels []string) string (engine/reviews.go) scans item.Labels for the review-authority: prefix and validates the suffix against the same {"advisory", "authoritative"} value set stages.go’s validReviewAuthorities enforces at YAML-load time. Precedence follows extractEffortOverride’s “pick deterministically, don’t arbitrate” convention (rank array, prefer the higher-ranked/more-restrictive value), not extractModelOverride’s “first wins” convention: no label → stage config governs; exactly one recognized label → it overrides the stage config for this issue only; both review-authority:advisory and review-authority:authoritative present → resolves to authoritative, with a logged warning; a label whose suffix is neither exactly advisory nor authoritative (typo, casing, unknown value) → ignored with a logged warning, falls back to stage config — never a hard failure, never a silent escalation to authoritative. effectiveReviewAuthority(item gh.ProjectItem, stage *stages.Stage) string composes the extractor with the stage fallback (override, else stage.ReviewAuthority) and is the single resolution point checkReviewGate, reviewGateBlocksLanding, and pauseForReviewTimeout’s authoritative-mode pause message all consult — none of the three reads stage.ReviewAuthority directly anymore, so the advance gate, the landing gate, and the message describing why a pause happened can never disagree about which mode applies to a given issue. The override changes nothing about whether the gate applies (wait_for_reviews: true on the stage is still required for either gate to engage at all) — only the verdict-strictness once the gate is active. See ADR-1261.
Verdict source, preferred then fallback:
- GitHub’s
reviewDecision, when it reflects a real branch-protection review requirement — one ofAPPROVED,CHANGES_REQUESTED,REVIEW_REQUIRED(computed server-side, including CODEOWNERS-required approvals and required-approval-count rules — Fabrik does no CODEOWNERS parsing of its own). Satisfied only onAPPROVED.REVIEW_REQUIREDcovers both “zero reviews yet” and “some, but not enough, approvals” (e.g. a required-approval-count of 2 with only 1 approval so far) — GitHub reports the same value for both, andreviewGateAuthorityVerdictis only reached oncehasReviewsis already true, so in practice this branch represents the not-enough-approvals case. - Fabrik’s own fallback, when
reviewDecisionis""(no branch-protection review requirement configured on the repo) — satisfied unless any review in the caller’sreviewsslice hasState == "CHANGES_REQUESTED". This exists soauthoritativemode is never a silent no-op on a repo without branch protection; without it, opting in on such a repo would be indistinguishable from advisory. On thebase:<branch>REST fallback path,reviewscomes fromFetchPRReviews(github/prs.go), which collapses each author’s full review history down to one entry — critically, aCOMMENTEDfollow-up submission never overwrites a prior formal verdict (APPROVED/CHANGES_REQUESTED/DISMISSED) from the same author during that collapse, so a reviewer who requests changes and later leaves a comment-only follow-up still reads as an activeCHANGES_REQUESTEDhere, matching GitHub’s own review-decision semantics (COMMENTED is informational, not a state transition). - Any other non-empty
reviewDecision(an undocumented future GitHub enum value) blocks conservatively rather than falling through to the fallback in (2) — GitHub did report a real verdict, just not one Fabrik recognizes yet, so treating it as “no requirement configured” could wrongly satisfy the gate.
Fetching reviewDecision. FetchPRReviewDecision(owner, repo string, prNumber int) (string, error) (github/prs.go) is a GraphQL-by-PR-number query (repository(owner,name){ pullRequest(number){ reviewDecision } }), mirroring prNodeID’s shape rather than nesting inside closedByPullRequestsReferences — REST has no equivalent field, and this shape works identically for default-branch and base:<branch> items with no ProjectItem schema change, no webhook-delta freshness plumbing, and no caching (boardcache.CacheImpl.FetchPRReviewDecision always delegates, mirroring FetchPRReviews/FetchPRReviewRequests’ “review state is highly volatile” rationale). The call is made only when stage.ReviewAuthority == "authoritative" and the gate has already reached the “would otherwise clear” branch, so the extra GraphQL round-trip is paid only by stages that opt in, and only on the poll where it would matter.
Fetch failure blocks conservatively. A FetchPRReviewDecision error at either gate site does not clear the gate and does not silently fall back to the no-branch-protection computation — it is treated as unknown state, mirroring the existing FetchPRReviews/FetchPRReviewRequests failure handling in both functions. At checkReviewGate this falls through into the existing still-waiting/escalation logic with the reason "review verdict unreadable (fetch failed), blocking conservatively"; at reviewGateBlocksLanding it returns from holdLandingForReview with an equivalent message.
No PR number resolved. If checkReviewGate’s prNumber <= 0 at the point authoritative mode would consult it (linkage not yet resolved), the gate blocks with reason "review verdict unreadable — no PR number resolved" rather than skipping the check.
Non-clear → reinvoke first, pause only at MaxReviewCycles (R1, #1375, amending ADR-1250). review_authority governs merging, never working. A verdict that does not clear (most commonly an unresolved CHANGES_REQUESTED) does not make Fabrik sit idle waiting for a human — handleReviewGate (§6.2) computes buildReviewFeedbackComments() (unresolved inline thread comments plus unaddressed review bodies) before ever consuming checkReviewGate’s blocked/timedOut return, and dispatches a bounded reinvoke whenever that set is non-empty, regardless of blocked state. Per GitHub’s REST contract, a CHANGES_REQUESTED review always carries a non-empty body, so there is always something actionable to reinvoke on — the reinvoke, not a pause, is the primary response to a change request. The reinvoke loop is bounded by the pre-existing MaxReviewCycles/dispatchWithCycleLimit/pauseForReviewCycleLimit machinery (§6.2), plus a non-convergence terminal check independent of any single poll’s dedup outcome (R1/R2, ADR-1518): the cycle-limit pause fires whenever the gate is still blocked/timedOut and max(ReviewCycles, ReviewBlockedCycles) has reached MaxReviewCycles — including on a poll where the currently-outstanding feedback happens to have already been deduped as processed (snap.CommentProcessed), which is exactly the steady state immediately after the MaxReviewCycles-th reinvoke and was previously unreachable (#1518). pauseForReviewTimeout remains reserved for the genuinely different case: nothing actionable to reinvoke on at all — a plain “nobody has responded yet” block, which can only have a structurally-zero cycle count (a reinvoke can only ever fire in response to actual review content), so the two pause reasons cannot collide. See ADR-1375, which amends this ADR-1250 consequence, and ADR-1518 for the terminal-check/ReviewBlockedCycles mechanism.
Messaging distinguishes “no reviews yet” from “reviews exist but the verdict blocks.” checkAwaitingReviewTimeout takes an authorityReason parameter (non-empty only when the authoritative check blocked) and uses it verbatim as both the wait-log reason and the timeout-pause reason, instead of the generic "no reviews submitted yet" / "pending reviewers" text — which would otherwise misleadingly suggest nobody has reviewed at all. pauseForReviewTimeout’s standard (non-Phase-2) message additionally live-fetches the verdict and reuses reviewGateAuthorityVerdict itself when authoritative mode is on, appending its reason string verbatim (e.g. "no branch-protection review requirement; bob requested changes" — the reviewer’s login comes from reviewGateAuthorityVerdict’s own fallback-branch message, not a separate helper) — so this line can never disagree with why the gate is actually blocking, and covers REVIEW_REQUIRED/fetch-failure cases too, not just an active CHANGES_REQUESTED review. item.LinkedPRNumber/item.LinkedPRReviews are structurally empty for base:<branch> items (same gap as the existing pendingLine), so this resolves the PR and its reviews via the same FetchLinkedPR/FetchPRReviews REST fallback checkReviewGate/reviewGateBlocksLanding use, rather than omitting the line there.
yolo/cruise never bypass the gate — no new branching needed. Because the check lives inside the shared clearing predicate rather than at attemptMergeOnValidate/handleStageComplete’s yolo/cruise decision points, authoritative + yolo “waits” for free: attemptMergeOnValidate simply never reaches its merge/enqueue logic while reviewGateBlocksLanding returns true, exactly like an outstanding-reviewer block today. Once the verdict clears (a fix pushed → stale review dismissed on push → re-review → APPROVED, or a required approval lands), yolo merges immediately with no human click required — the timing is still automatic, only the gate has tightened. cruise composes identically at both layers: hasCruiseLabel is checked before reviewGateBlocksLanding is ever consulted (§1’s cruise-wins-over-yolo precedent — cruise is checked on the raw label, not cruiseActive — applies unchanged here), so a cruise item never even reaches the authoritative check — cruise’s “leave the PR for human merge” behavior is unaffected by review_authority.
Unit tests: TestReviewGateAuthorityVerdict (engine/reviews_test.go) is the pure-function table (APPROVED/CHANGES_REQUESTED/REVIEW_REQUIRED/empty-with-clean-reviews/empty-with-CHANGES_REQUESTED/empty-with-DISMISSED-CHANGES_REQUESTED). TestCheckReviewGate_Authoritative_* and TestCheckReviewGate_NonDefaultBase_Authoritative_* cover the advance gate across reviewDecision × branch-protection-present/absent × default-branch/base:<branch> × fetch-error. TestAttemptMergeOnValidate_Authoritative_* covers the identical matrix at the landing gate, plus TestAttemptMergeOnValidate_Authoritative_CruiseSkipsAutoMerge (cruise composition) — attemptMergeOnValidate’s own yolo-vs-advisory tests (TestAttemptMergeOnValidate_YoloEnablesAutoMerge etc.) already exercise the yolo path unconditionally, since the function has no yolo-specific branching to begin with. TestCheckReviewGate_Advisory_ChangesRequestedReview_StillClears and TestAttemptMergeOnValidate_Advisory_ChangesRequestedReview_StillMerges are explicit non-regression controls: advisory mode clears/merges regardless of a CHANGES_REQUESTED verdict, and FetchPRReviewDecision is asserted never called. For the per-issue label override (#1261): TestExtractReviewAuthorityOverride and TestEffectiveReviewAuthority (engine/reviews_test.go) cover the extractor and resolution-helper precedence matrix directly (no label, single label, both labels, malformed suffix). TestCheckReviewGate_LabelOverride_* and TestAttemptMergeOnValidate_LabelOverride_* (engine/reviews_test.go/engine/stages_test.go) re-run the label-authoritative-on-advisory-stage and label-advisory-on-authoritative-stage cases at both gate sites, plus TestAttemptMergeOnValidate_LabelOverride_Authoritative_CruiseSkipsAutoMerge/_YoloWaitsForGate confirming yolo/cruise compose with a label-resolved gate exactly as they do with a stage-YAML-resolved one.
6.2 Review Reinvoke Mechanics
The catch-up loop in poll() is split into two phases for every non-paused non-cleanup item that has either stage:<X>:complete OR fabrik:awaiting-ci (on a wait_for_ci: true stage):
Phase 1 — unconditional, ordered handler list (catchUpPhase1Handlers):
Phase 1 is implemented as an explicit ordered slice of named handler methods (catchUpPhase1Handlers in engine/catch_up_handlers.go). Each handler returns true to claim the item — Phase 2 is skipped for this item this cycle — or false to pass through to the next handler. Ordering is structurally enforced by slice position (ADR-056 D3); a future reorder requires a deliberate code change and triggers a test failure, rather than occurring as an accidental side effect of a continue insertion.
Handler 0: handleEngineUnpause (#1460 — numbered 0, not renumbering Handlers 1-4 below, to avoid an invasive cross-reference cascade) — runs first, unconditionally, ahead of every other Phase 1 handler. It is the actual resume trigger for the four cycle-limit pause sites (pauseForReviewCycleLimit, pauseForCIFixCycleLimit, pauseForRebaseCycleLimit, pauseForEnqueueCycleLimit; see §7.2 “Resumable Engine Pauses”): checks snap.PausedByEngine(stage.Name); if true — meaning the item reached this handler chain (so fabrik:paused is currently absent, per both the main catch-up loop’s and settleAwaitingCIScan’s admission filters) while still carrying a stale itemstate.EnginePaused record from before the label was removed — it calls clearFailedStage(), the same reset processItem’s own unpause gate already uses (zeroing ReviewCycles/CIFixCycles/RebaseCycles/EnqueueCycles, clearing Attempts, resetting LastAttemptAt, and resetting the comment circuit breaker). Always returns false — it only resets state, never claims — so the rest of the chain evaluates the just-reset counters in the same pass, letting a genuine resume converge in one poll rather than two. See ADR-1460 for why this handler exists at all (processItem’s pre-existing unpause gate is structurally unreachable for these four sites — they only ever fire post-stage-completion, where itemNeedsWork never re-dispatches processItem).
Handler 1: handleDependencies — thin wrapper around checkDependencies(). If the item has unresolved blocking dependencies, claims the item (returns true); otherwise passes through. checkDependencies’s own bool return is the Phase 1 claim signal directly (no translation layer, unlike the review/CI gates below) — so its cycle-detection branch (a bounded BFS finds the item is itself a transitive blocker of one of its own blockers) also returns true when it pauses the issue via pauseIssue (ADR-1223), rather than reusing the false value returned for “no open dependencies.” Before this fix the cycle-detection pause fell through unclaimed, the same overloaded-return bug present in the review/CI gates.
Handler 2: handleReviewGate — its gate half (checkReviewGate: the fabrik:awaiting-review write and the reviewer timeout) is only active when stage:<X>:complete is present (pctx.hasComplete == true; skipped during the CI-await window when fabrik:awaiting-ci is present but stage:X:complete is absent — prevents spurious fabrik:awaiting-review re-application from stale board data; #617). Since #1953 (R1) its feedback-detection and review-reinvoke dispatch half runs in every state: with !hasComplete it resolves reviews itself (resolveReviewsForFeedback), passes blocked = timedOut = false, writes no label, and returns false when nothing was dispatched so the CI gate still runs; and a completed stage without wait_for_reviews now also resolves reviews, so its review bodies are dispatched (before, checkReviewGate’s early return handed back no reviews and only threads were). Its review data comes from the item the poll admitted, i.e. the board cache, while the feedback gate reads live; a gate hold on feedback this handler cannot see is repaired by invalidating the cache (§6.6.6 “Dispatcher input agreement”, #2160). Because feedback now holds an advance or landing (§6.6.6), a reinvoke dispatched while feedback stands between the item and its advance (feedbackHoldsProgress: stage incomplete, or yolo/cruise/auto_advance would advance it) is also charged to the never-refunded ReviewBlockedCycles counter, so a reviewer producing fresh non-actionable bodies forever still ends in pauseForReviewCycleLimit (ADR-1953-a):
checkReviewGate()runs first (side effects — label application, the bot-reprompt ladder — happen regardless of what the caller does with the return value), yielding(blocked, timedOut, terminated).terminated(a direct pause already issued byhandleBrokenReviewLinkage, ADR-1223) is claimed immediately, exactly as before.- Reinvoke-before-block (R1, #1375, amending ADR-1250):
buildReviewFeedbackComments()— unresolved inline review-thread comments (buildReviewThreadComments(), unmodified) plus unaddressed review-body comments (buildReviewBodyComments(), new) — is computed unconditionally next, beforeblocked/timedOutare consumed. If that combined set is non-empty, the reinvoke below dispatches regardless of whether the gate is currentlyblocked— this is what makes the reinvoke reachable at all underreview_authority: authoritative, where an unresolvedCHANGES_REQUESTEDwould otherwise keepcheckReviewGatereturningblocked=trueindefinitely (Finding 1). Only when this set is empty does the handler fall through to the pre-existing tail: ifblocked, recordsCooldownRecorded{Reason: "review-blocked"}soitemMayNeedWork’s expiry path re-evaluates the item every 10 × PollSeconds, then claims; iftimedOut, callspauseForReviewTimeout()and claims.buildReviewBodyComments()turns each unaddressed non-DISMISSED/non-PENDINGreview’s top-level body into a syntheticgh.Comment(ID: "review-body:" + PRReview.DatabaseID,DatabaseID: 0) — as of #1045, widened fromCHANGES_REQUESTED-only.DISMISSED(no longer an active verdict) andPENDING(not yet submitted) are the only states skipped, plus empty bodies andDatabaseID == 0reviews (no stable ID to dedup on).CHANGES_REQUESTED,COMMENTED, andAPPROVEDbodies are all now treated as potentially actionable. Superseded rationale (#1375, kept for history): the original version skippedCOMMENTED/APPROVEDoutright, reasoning that automated reviewers like Copilot/Gemini routinely submit aCOMMENTEDreview whose body is a generic summary, not actionable feedback, and that admitting it would trigger a reinvoke on everywait_for_reviewsstage far beyond aCHANGES_REQUESTEDverdict’s scope. That reasoning held only while everyCOMMENTEDreviewer submitted generic summaries; it stopped holding once a reviewer that submits its entire finding set asCOMMENTED(Pruefer, severity-gated raise-onlyREQUEST_CHANGES, #1251) started running onhandarbeit/fabrik— the gate cleared normally on those reviews (a non-DISMISSEDreview from an author satisfieshasReviews), but the findings were never routed to the fixer, reproducing #1207’s “merge past unresolved feedback” failure shape through a filter instead of a race. #1045’s fix is to widen actionability rather than add an author-based discriminator (anexpected_reviewers-keyed approach was proposed and withdrawn — it would have relocated #1407’s “silently drops an undeclared identity’s findings” failure into the gate) and pay the reopened cost down via the no-op cycle exemption immediately below, rather than filtering on state.reviewGateAuthorityVerdict(§6.1.1, the separate merge/landing decision) is unaffected — it still treats onlyCHANGES_REQUESTEDas blocking; that split is unchanged by #1045. Dedup is viasnap.CommentProcessed(id)(and the durable marker below) only — GitHub’s REST reactions API has no endpoint for a top-level review body, so there is no REST ROCKET-reaction backstop the way a real thread comment gets (R7); this is what bounds the reinvoke to one dispatch per genuinely new review, not one per poll (the #1083 runaway shape). Reactions (#1953 R8, ADR-1953-b): the synthetic comment carries noDatabaseIDbut does carryReactionNodeID(the review’s GraphQL node ID, fromlatestReviews { id }/ RESTnode_id), soacknowledgeComments/finalizeCommentsreact 👀 at dispatch and 🚀 at completion through GraphQLaddReaction(AddReviewReaction) — the same two sites and conditions as ordinary comments. The reaction is only the human-visible signal (reactions can be added or removed by anyone); a failed reaction is a logged warning, and dedup never depends on it. Synthetics with neither ID (CI-fix, rebase) still skip. On abase:<branch>item,item.LinkedPRReviewsis structurally empty (§6.1’s base:note) — `buildReviewBodyComments()` REST-falls-back exactly like `checkReviewGate()` does (resolve PR number, then `FetchPRReviews`), so this reinvoke path is reachable there too. `buildReviewThreadComments()` has no equivalent fallback and stays an accepted gap for that repo shape. - Durable review-ids-addressed marker (R3, #1555).
snap.CommentProcessed(id)above is an in-memory record — wiped by every self-upgrade restart (syscall.Exec, §4.7’s root cause discussion). Because a review body has no reaction-endpoint backstop (R7, above), that restart-wipe is a single point of failure:handarbeit/fabrik#1254observed the same already-addressed review body redelivered seven consecutive times, each restart re-admitting it as “new.”buildReviewBodyCommentsFromReviews(engine/reviews.go) now consults a second, durable, GitHub-sourced fallback for any candidate not alreadysnap.CommentProcessed: every review-feedback PR commentformatReviewFeedbackCommentposts already lists which review bodies it addressed, so the fix embeds a machine-readable trailer —<!-- fabrik:review-ids-addressed: N,N -->— in that existing artifact.durablyAddressedReviewIDsresolves the linked PR and fetches its comments viaGitHubClient.FetchIssueComments(a new interface method, deliberately not added toboardcache.ReadClient— this is a one-shot, restart-triggered lookup, not a per-poll cache-worthy read), scanning for the marker. A hit backfillssnap.CommentProcessedso every subsequent call in the same process takes the fast, no-fetch path again. Lazy, not unconditional: the fetch only fires when at least one candidate isn’t already known-processed in memory — at steady state (nothing outstanding) this costs zero extra API calls; immediately after a restart it costs exactly one fetch per PR with an outstanding review body, not per poll. Accepted residual gap: the marker only exists once a review-feedback PR comment is actually posted — a cycle that lands no PR output (e.g. every candidate filtered upstream,output == "") still depends on the in-memory record alone for that narrower case; §4.7’sNoOpCommentCyclescounter is the safety net for it. See ADR-1555. - No-op cycle exemption (#1045).
handleReviewGatestill incrementsReviewCyclessynchronously before dispatch, exactly as before —dispatchWithCycleLimit(below) is unchanged and shared with rebase/CI-fix cycle dispatch.dispatchReviewReinvoke(engine/reviews.go) additionally snapshotsHEADviagitHeadSHA(workDir)in itsbuild()hook, beforeprocessComments()runs, and compares it again in a newafterhook onceprocessComments()returns. When the two match — and only when the “before” snapshot was itself readable (headBefore != "") — it appliesitemstate.ReviewCycleDecremented(floored at 0), netting that dispatch back to unchanged. This mirrorsdispatchCIFixReinvoke’s existinggitHeadSHAbefore/after pattern (below), with one deliberate divergence: CI-fix’s no-op record only short-circuits future dispatches for the same head SHA, it does not stop the counter incrementing for the no-op dispatch itself; the review-cycle exemption prevents the increment from sticking at all. The check is scoped to “no new commit” only, uniformly — it does not distinguish “Claude ran and did nothing” from “the batch was filtered to empty before Claude ran” (the #1221 chokepoint below). It does not reach the #1221 case:processComments()’s notice filter runs beforewm.EnsureWorktree, so when every candidate comment is filtered out, the worktree is never touched that cycle andgitHeadSHAfails identically inbuild()andafter()— there is noHEADto compare,headBefore == "", and no decrement fires.TestCatchUpLoop_ReviewReinvoke_AllNoticeThread_NoInvocation’sReviewCycles == 1assertion is therefore unchanged by #1045 — see the #1221 paragraph below. See ADR-1045. - Did-not-run cycle refund (#1812, ADR-1812). A reinvoke whose Claude invocation provably never ran is not an attempt to converge and must not spend the cycle budget.
dispatchReinvoke(engine/reinvoke.go, the shared scaffold for review, CI-fix and rebase — and the only place that also coverscheckAutoMergeConvergence’s inlineRebaseCycleIncremented/CIFixCycleIncrementedcharges) callsprocessCommentsClassified, which returns adidNotRunKindalongside the unchanged error:account-usage limited(the ADR-1120 usage-limit hold gate, which returnsnilbefore any side effect),usage-limit(claudeUsageLimitError, also anilreturn, excluded from the comment breakers),api-error(claudeAPIErrorExit, whose classifier already refuses a run that consumed turns and incurred cost — R4), andapi-key-helper(apiKeyHelperDetectedError; produced only by the stage-dispatch path today, so recognised here for forward-compatibility only). Any other error — turn limit, resume failure, tools denied, a mid-run crash, an output-publication failure — yields""and stays charged exactly as before (R3); ambiguity is always charged. On a non-empty kind the scaffold applies the dispatcher’s compensating mutations (ReviewCycleDecremented, plusReviewBlockedCycleDecrementedwhenhandleReviewGatecharged it, i.e.blocked || timedOut;CIFixCycleDecremented;RebaseCycleDecremented— each floored at zero and a no-op with noChangeon an already-zero counter), recordsDidNotRunReinvokeRecorded, logsinvocation did not run (<kind>) — refunding <counter> cycle counter for stage "<S>" (#1812), and skips the dispatcher’safterhook — so the #1045 HEAD-unchanged refund, CI-fix’s no-op-SHA debounce and rebase’s auto-merge re-enable, all meaningless when nothing ran, never fire. The two refund mechanisms are therefore disjoint and cannot double-refund.ReviewBlockedCycleskeeps ADR-1518’s never-refunded guarantee for a run that did execute and no-op’d; only a did-not-run exit compensates it. DidNotRunReinvokes— evidence for the pause message (R7, #1812). Because refunded cycles vanish from the counters,StageState.DidNotRunReinvokesis a tally of did-not-run reinvokes that is never refunded by a cycle refund, is cleared byEngineCyclesCleared, and is reset byDidNotRunReinvokesResetwhenever a reinvoke stays charged (dispatchReinvokeapplies it on every non-did-not-run exit, including ambiguous ones) — so it is the streak of never-ran reinvokes since the last genuine cycle, never an outage from days earlier that could flip the message for a later real non-converging loop.pauseForReviewCycleLimit,pauseForCIFixCycleLimitandpauseForRebaseCycleLimitconsult it viadidNotRunPauseNote: when that streak is at least the cycle count, the “reviewer is repeatedly requesting changes” (resp. “CI still failing” / “not mergeable”) sentence is replaced with a statement that the invocations never ran, naming the likely cause (a Claude API error such as a 429 session limit, or a usage limit), the elapsed-time discriminator (real cycles take minutes; a burst within seconds means nothing ran) and where to look (.fabrik/logs/<repo>/issue-N/); a smaller nonzero tally adds a one-line note; a zero tally leaves the message byte-identical. The stable dedup fragments (reviewCyclePauseFragmentetc.) are unchanged, so #1460’shasPauseCommentstill matches. Bound after the refund: a did-not-run storm no longer tripsMaxReviewCycles; it is bounded by the comment circuit breaker (§4.6, default 10 per 30 min) and the #1555 no-op breaker (default 10), both of which still count these exits — the false pause moves from cycle 5 to roughly cycle 10 with the breaker’s own message. Backoff on this path is out of scope; detecting the 429 upstream is what prevents the storm.ReviewBlockedCycles— the never-refunded non-convergence counter (R3, #1518, ADR-1518).#1045’s refund above is deliberately unconditional for its own scenario — five, or an unbounded number of, consecutive no-op reinvokes on non-blocking feedback (advisory-modeCOMMENTEDjunk overviews that never held up the gate at all) must keep being forgiven forever (TestHandleReviewGate_FiveNoOpReinvokes_DoNotExhaustBudget_SixthGenuineFindingAddressed). But that same unconditional refund can also mask a genuinely non-converging loop: if every reinvoke against an unresolved authoritative verdict happens to be a no-op onHEAD,ReviewCyclesnever accumulates past 1, and the cycle-limit check below never fires. The two scenarios are structurally distinguishable at dispatch time —checkReviewGate’s ownblocked/timedOutreturn (computed beforebuildReviewFeedbackComments()is even built) is true only when the gate itself is still failing to clear.handleReviewGatetherefore also appliesitemstate.ReviewBlockedCycleIncremented— alongside the existing pre-dispatchReviewCycleIncremented, never in place of it — exactly when a reinvoke is about to be dispatched andblocked || timedOutis true at that moment. It has no decrement counterpart: a no-op reinvoke dispatched while the gate was genuinely blocked is non-convergence evidence regardless of whether it happened to land a commit. A reinvoke dispatched while!blocked && !timedOut(#1045’s own shape) never increments this counter, so that refund stays unconditional exactly as #1045 requires.- Terminal check independent of this poll’s dedup outcome (R1/R2, #1518, ADR-1518). Both the cycle-limit check below (inside the
buildReviewFeedbackComments() > 0branch) and a new check in the pre-existingblocked/timedOutfallback tail now compare againstmax(snap.ReviewCycles(stage.Name), snap.ReviewBlockedCycles(stage.Name)), notReviewCyclesalone — closing the gapReviewBlockedCyclesexists to close. The fallback-tail check fires wheneverblocked || timedOutis true and that max has already reachedMaxReviewCycles, callingpauseForReviewCycleLimit()directly and claiming the item before the tail’s ownCooldownRecorded/pauseForReviewTimeoutbranches are reached — so a gate that is still failing to clear cannot fall through to the timeout pause merely because the currently-outstanding feedback happens to already be deduped as processed on this particular poll. Gated onblocked || timedOut, so the naturally-cleared case (buildReviewFeedbackComments()empty and the gate itself cleared) is untouched; a genuine “nobody has ever reviewed” block can only have a structurally-zero cycle count (no reinvoke has ever had content to fire on), so it can never satisfy the>=comparison and R4 (pauseForReviewTimeoutfor that case) holds without an explicit special case.
- Yolo-merge guard 2 (#1207): if the item carries
fabrik:auto-merge-enabledandcurrentHeadReviewFeedbackComments()(current-head thread comments, viacurrentHeadReviewThreadComments()’s existingisOutdatedfiltering, plus all review-body comments — a review body has no thread and therefore noisOutdatedsignal; every body comment is treated as always-current-head, an explicit, documented R8 simplification, not an oversight) is non-empty,disableAutoMergeForReviewThreads()runs here — before the worker/cycle-limit checks below, on the same poll. This has to live inside Handler 2 rather than inside Handler 3 (handleAutoMergeConvergence/checkAutoMergeConvergence): Handler 2 always claims the item and stops the Phase 1 chain wheneverbuildReviewFeedbackComments()is non-empty, so a check placed only in Handler 3 would never run in that same cycle. See §2.9 for the full guard-1/guard-2 design. - Worker guard (
snap.Worker() != nil): if a reinvoke goroutine from a previous poll cycle is still running, claims the item without dispatching or incrementing the cycle count. - Cycle limit check (
max(snap.ReviewCycles(stage.Name), snap.ReviewBlockedCycles(stage.Name))vsMaxReviewCycles, default 5 — themax, notReviewCyclesalone, is #1518/ADR-1518’s fix for the refund-masking gap above): if exceeded,pauseForReviewCycleLimit()addsfabrik:paused+fabrik:awaiting-inputand claims — this is the terminal fallback (R5) for a verdict that never converges, reached whether the gate cleared naturally or is still authoritatively blocked. If not exceeded: increment count viaReviewCycleIncremented(plusReviewBlockedCycleIncrementedwhenblocked || timedOutat that moment, per the bullet above), dispatchdispatchReviewReinvoke()(appliesWorkerEntered, acquires semaphore, callsprocessComments()asynchronously with the combined thread+body comment set), setadvancedItems[key] = true, claim. isReviewReinvoke()(engine/comments.go, consulted bypublishCommentOutputwhen the reinvoke’sprocessComments()call completes) recognizes a comment as review-reinvoke-eligible whenReviewThreadID != ""or itsIDcarries the"review-body:"prefix — so a reinvoke batch that is mixed (thread comments + a review body) or consists solely of review bodies still posts the PR feedback summary comment, not just a pure-thread-comment batch.
Handler 3: handleAutoMergeConvergence — if the item does not have fabrik:auto-merge-enabled, passes through. If it does: calls settlePRMergeState() to build a PRSettleResult, then forwards to checkAutoMergeConvergence() which consumes the settle result for PRMergeUnsettled/PRMergeConflicting branch decisions. Claims the item (returns true). The merge/CI gates remain bypassed — GitHub owns the merge decision for these items — but checkAutoMergeConvergence() no longer independently interprets mergeable_state or calls FetchPRMergeableFields/FetchCheckRuns.
Handler 4: handleMergeAndCIGates — calls settlePRMergeState() once (the settle call shared by both gates), then the merge-conflict gate, then the CI gate. Merge runs before CI per ADR-028: a PR made unmergeable by a base-branch advance must be rebased before the engine spins on CI-await polls. See §6.4 for the full settling primitive specification.
-
Settle call:
settlePRMergeState()fetchesmergeable,mergeable_state, and check runs in a single pass, returning a typedPRSettleResult. Both gates below consume this result and do not make additional REST calls — eliminating the split-brain where two separate REST calls within one poll cycle could observe different GitHub state. - Merge-conflict gate (
checkMergeabilityGate()interpretsPRSettleResult):PRMergeNoPRorPRMergeTerminal: no-op; fall through to the CI gatePRMergeReady: clear any stalefabrik:rebase-neededlabel; fall through to the CI gatePRMergeBlocked(CI checks failed): clear any stalefabrik:rebase-neededlabel; fall through to the CI gate socheckCIGatecan classify and dispatch the CI-fix reinvokePRMergeUnsettled(andPRMergeQueued): claims the item, blocking it for the rest of Phase 1 (no label churn — ADR-032 R10c), and records a shortmerge-unsettledcooldown (now + PollSeconds/2, #2096) so the item is re-evaluated on the next poll rather than after the periodic re-evaluation interval — see “Cooldown Cache-Key Strategy” in Appendix BPRMergeConflicting(mergeable == false): applyfabrik:rebase-neededidempotently, then worker guard (snap.Worker() != nil) + cycle limit check (snap.RebaseCycles(stage.Name)vsMaxRebaseCycles, default 3):- If exceeded:
pauseForRebaseCycleLimit()pauses issue and claims - If not exceeded: increment count via
RebaseCycleIncremented, dispatchdispatchRebaseReinvoke(), setadvancedItems[key] = true, claim. The CI gate is never reached while a conflict is outstanding — there is no point spinning on CI-await when the branch cannot merge.
- If exceeded:
- CI gate (
checkCIGate()interpretsPRSettleResult; only active whenstage.WaitForCI == true):PRMergeTerminal(merged): gate clears immediately (addCompleteLabelAndRemoveCI); handler returns false (advance falls through to Phase 2)PRMergeTerminal(closed, not merged):pauseForPRClosedNotMerged()addsfabrik:paused+fabrik:awaiting-input, removesfabrik:awaiting-ci,fabrik:awaiting-review, andfabrik:rebase-needed(ADR-1387 R6); handler returns falsePRMergeReady(mergeable_state == "clean", or all checks green under the classification chain below — ADR-1441): gate clears immediately viaaddCompleteLabelAndRemoveCI(). For thecleancase, per-check classification is skipped: GitHub’s branch protection is the source of truth once every check, required or not, has passed.mergeable_state == "unstable"no longer shortcuts here (ADR-1441, #1441) — it falls through to the check-run/required-context classification below, exactly likePRMergeBlocked/PRMergeUnsettled, so a confirmed failure on a non-required check now blocks instead of clearing.PRMergeBlockedorPRMergeUnsettledwithCheckRunspopulated: per-check classification viagithub.ClassifyCheckRuns()(#958) applies; any pending run (at any check name) → blocked, not failed — this holds even if a different check, or a stale lower-ID run of the same check, has already failed; only when nothing is pending does a failed run count as CI failed → flake re-run (#2072, §6.29: at most one re-run per head before the first dispatch and one after a recorded no-op; claims the item while it waits) + worker guard (snap.Worker() != nil) + no-op debounce guard (snap.LastCIFixNoOpSHA() == settle.PR.HeadSHA→ skip; §6.5.2) + cycle limit check (snap.CIFixCycles(stage.Name)vsMaxCiFixCycles): if exceeded,pauseForCIFixCycleLimit()claims; if not exceeded, increment count viaCIFixCycleIncremented, dispatchdispatchCIFixReinvoke(), setadvancedItems[key] = true, claim. Per ADR-1410 (R3), the failed-CI branch fires unconditionally — never gated on elapsed time. When still pending instead,checkCIGateapplies a liveness-stall dwell (ciProgressStalledSince,LinkedPRState.LastCIProgressAt) rather than an elapsed-time one — see §6.14.5.PRMergeUnsettledwith emptyCheckRunsandMergeableState == "blocked": R3 — checkFetchLabelAppliedAtdwell (elapsed-time, unchanged by ADR-1410 — no check-run signal exists here to observe progress on); if < CIWaitTimeout → dwell guard, not yet paused; if ≥ CIWaitTimeout →pauseForRequiredNeverRunningCheck()PRMergeUnsettledwith emptyCheckRunsandMergeableState ∉ {"", "unknown"}: non-empty branch-protection signal (e.g.behind,dirty,has_hooks) with no visible check_runs — checksFetchLabelAppliedAtinline (elapsed-time, unchanged by ADR-1410, same reasoning); if ≥ CIWaitTimeout, removesfabrik:awaiting-ciand returns(false, false, true)(caller callspauseForCITimeout); otherwise returns(true, false, false), re-evaluates next pollPRMergeUnsettledwith emptyCheckRunsand emptyMergeableState: hadChecks/dwell/post-push window — re-evaluates next poll; no label churn- Timed out (generic path):
pauseForCITimeout()pauses issue
After Phase 1 + Phase 2 — single-owner Validate advance (runValidatePRTerminalAdvance, R4 — ADR-056 D2, ADR-057): A single authoritative scan iterates deepFetchCandidates for all Validate-stage items not in the fabrik:auto-merge-enabled convergence flow, regardless of which gate label (fabrik:awaiting-ci, fabrik:awaiting-review, fabrik:rebase-needed, or any future label) is present. For each, e.client.FetchLinkedPR is called directly (not e.readClient — boardcache may have stale state). When pr.Merged == true: iterates all pipeline stages in ascending Order from the stage after the highest already-complete stage up to (but not including) the cleanup-terminal stage, adding stage:<N>:complete for every WaitForCI or WaitForReviews gate-checked stage whose label is absent (with cache write-through; fail-fast on error for idempotent retry). After all labels are added, removeAwaitingCILabel, removeAwaitingReviewLabel, and removeRebaseNeededLabel are called as applicable, fabrik:paused + fabrik:awaiting-input are removed, and advanceToNextStage() is called. Items already in advancedItems are skipped (prevents double-advance). This self-heals Validate-stage items merged externally regardless of which gate was active, without requiring a manual unpause. The function never dispatches workers or acquires e.sem — label-mutation-only path (see ADR-053 constraints; superseded in structure by ADR-057).
Phase 2 — gated (yolo/cruise/auto_advance only):
- Only runs when no Phase 1 handler claimed the item (i.e., all handlers returned false)
- Gated on:
e.cfg.YoloORfabrik:yololabel ORfabrik:cruiselabel OR stageauto_advance: true - Runs
attemptMergeOnValidate()(yolo only), skips if unprocessed comments exist, then callsadvanceToNextStage()
processComments widening: processComments() itself also merges any unresolved LinkedPRReviewThreadComments at entry, before Step 1. This closes the race where a user nudge arrives before the catch-up loop Phase 1 fires — the review thread comments are addressed in the same invocation as the nudge comment.
Bot service-notice chokepoint (#1221): immediately after that merge, and before Step 1, processComments() filters the fully-assembled comments slice through isBotServiceNotice (filterBotServiceNotices, §4.1 signal 4) — the single point every comment reaching processComments() passes through regardless of source (findNewComments(), the LinkedPRReviewThreadComments merge above, or a reinvoke dispatcher’s build() output). If the filtered result is empty, processComments() returns immediately, before any reaction, label, worktree setup, or Claude invocation — mirroring the claudeUsageLimitedUntilTime early-return earlier in the function. This closes a review-thread bypass: bot reviewers (Gemini, CodeRabbit) post non-actionable service notices via inline review comments, not only via issue comments, and buildReviewThreadComments() (used by both dispatchReviewReinvoke’s precheck and the merge above) filters only ROCKET reaction and CommentProcessed — it does not itself classify bot notices. Left unfixed by design: dispatchReviewReinvoke’s precheck and the catch-up loop’s ReviewCycles increment (§6.2 Handler 2) still use the unfiltered buildReviewThreadComments() count, so an un-watermarked, notice-only review thread still dispatches a worker each poll (immediately no-op’d by this chokepoint) and still consumes a review cycle — self-limiting via MaxReviewCycles/pauseForReviewCycleLimit rather than silently free. Watermarking bot notices within LinkedPRReviewThreadComments (extending settleBotServiceNoticesForItem, currently scoped to item.Comments only) is a deliberate follow-up, not required to close the processing risk this chokepoint addresses.
Why #1045’s no-op cycle exemption (§6.2 Handler 2) does not subsume this case. That exemption compares gitHeadSHA(workDir) before and after the reinvoke, and this chokepoint returns from processComments() at the point quoted above — before wm.EnsureWorktree ever runs. There is therefore no worktree to read a HEAD from at either snapshot point; gitHeadSHA fails identically before and after, and dispatchReviewReinvoke’s headBefore != "" guard means no decrement is ever applied. This is a structural consequence of the two mechanisms’ relative ordering, not a deliberate carve-out re-litigating this paragraph’s own trade-off — it happens to leave this paragraph’s documented cost exactly as it was.
Review thread resolution: Step 10 of processComments() resolves addressed review threads via ResolveReviewThread() after adding ROCKET reactions.
PR summary comment: Step 8b of processComments() posts a Fabrik-marked "<StageName> (review feedback addressed)" comment on the linked PR when the invocation is a review-reinvoke (all-ReviewThreadID batch) and output != "". The comment includes Claude’s cleaned output plus a machine-generated per-thread footer listing path:line — resolved for each unique ReviewThreadID in the input batch (deduped; line resolved from Comment.Line with OriginalLine fallback). This gives reviewers a visible record in the PR timeline that their inline feedback was addressed.
6.3 Review Reinvoke vs Regular Comment Processing
| Aspect | Regular Comments | Review Reinvoke |
|---|---|---|
| Source | item.Comments (issue comments) |
item.LinkedPRReviewThreadComments (PR inline comments) |
| Detection | findNewComments() |
buildReviewThreadComments() |
| Dispatch | Synchronous in processItem() |
Async goroutine via dispatchReviewReinvoke() |
| Cycle limits | None | MaxReviewCycles (default 5) |
| Timeout | None | Integrated with ReviewWaitTimeout |
| Thread resolution | Yes — processComments() merges unresolved LinkedPRReviewThreadComments at entry, so a user nudge resolves threads in the same invocation |
Yes — resolves review threads after processing |
| PR summary posting | None | Posts "<StageName> (review feedback addressed)" on the linked PR with per-thread footer (one path:line — resolved bullet per unique ReviewThreadID); skipped when output == "" or no linked PR |
| Worker guard | Uses dispatch loop’s snap.Worker() != nil |
Has its own snap.Worker() != nil check in catch-up loop |
6.4 PR Merge State Settling Primitive
settlePRMergeState() is called once per catch-up loop Phase 1 iteration to read mergeable, mergeable_state, and check runs in a single pass. It is called from two handlers:
handleAutoMergeConvergence(Handler 3): for items withfabrik:auto-merge-enabled;checkAutoMergeConvergence()consumes the result for unsettled/conflict branch decisions, replacing directmergeable_stateinterpretation andFetchCheckRunscalls. The merge/CI gates remain bypassed.handleMergeAndCIGates(Handler 4): for all other items; bothcheckMergeabilityGate()andcheckCIGate()receive the result and do not make their own REST calls for these fields.
In both cases, this eliminates the split-brain where two separate REST calls within one poll cycle could observe different GitHub state.
Return type: PRSettleResult carries:
Status PRMergeStatus— one of seven typed constants (see below)PR *gh.PRDetails— the fetched PR (nil when status isPRMergeNoPR)MergeableState string— rawmergeable_statefrom GitHub (empty when intentionally omitted — see invariant below)CheckRuns []gh.CheckRun— check runs for the PR head SHA (nil when not fetched)Reason string— human-readable explanation for the statusRequiredContextsStatus gh.RequiredContextStatus(ADR-933) —RequiredContextsSatisfied(zero value),RequiredContextsPending, orRequiredContextsFailed. The zero value is correct both whenrequired_status_contextsisn’t configured for the repo and when an earlier rule (terminal/queued/conflict/ADR-033-shortcut) returned before required-context classification was ever consulted — callers must not infer “checked and satisfied” from an unpopulated result;checkCIGateonly acts on this field when it equalsRequiredContextsFailed.RequiredMissing,RequiredPending,RequiredFailed []string(ADR-933) — the required context names in each bucket, populated only alongside a non-SatisfiedRequiredContextsStatus.
Seven status constants:
| Constant | Meaning |
|---|---|
PRMergeNoPR |
No linked PR on this item; both gates are no-ops |
PRMergeUnsettled |
GitHub has not yet computed a definitive merge state; gates block without label churn |
PRMergeReady |
PR is ready to merge (all checks green under rules 13–19 below, or mergeable_state == "clean" via the rule 9 fast path — ADR-1441 narrowed this from {clean, unstable}) |
PRMergeConflicting |
mergeable == false — confirmed base-branch conflict |
PRMergeBlocked |
CI checks have failed; checkCIGate dispatches dispatchCIFixReinvoke() |
PRMergeTerminal |
PR is merged or closed without merging; both gates handle terminal state |
PRMergeQueued |
PR is in GitHub’s native merge queue (ADR-058 D4) — a transient hand-off: the queue owns the PR and will merge or eject it. Both gates block like PRMergeUnsettled (no conflict, no label churn); the convergence monitor waits. The single canonical queue-membership signal (ADR-056 D1), evaluated strictly after the terminal check (the #913 merged-first invariant: a PR that merged via the queue is terminal, never “queued”). |
Settling rules (applied in order):
FetchLinkedPR()— if no linked PR: returnPRMergeNoPR.- PR merged (
pr.Merged == true): returnPRMergeTerminal. - PR closed, not merged (
pr.State == "closed"): returnPRMergeTerminal(re-confirmed via the authoritative single-PRFetchPRMergedendpoint, since the list endpoint reportsmerged == falsefor several seconds after a queue merge — the #913 trap). - PR in merge queue (
prInMergeQueue(item) || pr.IsInMergeQueue || pr.MergeQueueEntry.State ∈ {QUEUED, AWAITING_CHECKS}): returnPRMergeQueued. Evaluated after the terminal check (rules 2–3, the merged-first invariant) and before mergeable/CI classification — a queued PR is a transient hand-off, so the gates never interpret itsmergeable_state. Dual-sourced from the GraphQL-authoritativeProjectItemfield (reliable on every poll) andsettle.PR/MergeQueueEntry, so it holds on a boardcache miss (the RESTFetchLinkedPRfallback reports the flag as false). FetchPRMergeableFields()— fetchesmergeable(bool ptr) andmergeable_state(string) in one REST call (single-PR endpoint; the list endpoint used byFetchLinkedPRdoes not returnmergeable). API error → returnPRMergeUnsettled.mergeable == nil(GitHub still computing): returnPRMergeUnsettled(MergeableStateset).mergeable_state == "unknown"(transient): returnPRMergeUnsettled(MergeableStateset).mergeable == false: returnPRMergeConflicting(MergeableStateset).mergeable_state == "clean"(ADR-033, narrowed toclean-only by ADR-1441/#1441 — previouslymergeable_state ∈ {clean, unstable}per theMergeableStateAcceptedallowlist): returnPRMergeReady— GitHub reports no required check failing or pending. This is not proof every job ran (#1822): a job still queued for a runner has no check run, so GitHub reportscleanfor a green prefix.FetchCheckRunsis not called, but since #1822 the shortcut first consults the suite hold (see Suite-aware completeness below); a held suite returnsPRMergeUnsettledinstead. An emptyHeadSHAskips the shortcut and falls to rule 10.mergeable_state == "unstable"no longer matches this rule — it falls through to rules 10–19 below like any other non-cleanvalue (e.g."blocked"), so a confirmed failure on a non-required check is caught by rule 19 instead of being silently accepted here.- HeadSHA empty: return
PRMergeUnsettled(MergeableStateintentionally omitted — see invariant). FetchCheckRuns()— API error → returnPRMergeUnsettled.- Check runs non-empty: apply
PRChecksObservedstore event (setsHasHadChecks). - Check runs empty + a configured required context confirmed-failed (ADR-933):
classifyRequiredContexts()is consulted here, unconditionally ahead of rules 14–16 below, and returnsRequiredContextsFailed→ returnPRMergeBlocked(RequiredContextsStatus/RequiredFailedset). This ordering is deliberate: a confirmed required-context failure (e.g. a classic commit status with no check-run footprint at all — the local-CI-takeover case #933 was filed for) is a definitive, timing-independent signal, and rules 14–16 exist only to wait out transient GitHub propagation delays for check-run-based CI. Placing the failure check after them (as an early draft of this fix did) let the R3mergeable_statebranch (rule 15, which fires for almost any non-emptymergeable_state, e.g."blocked"— exactly the local-CI-takeover case) mask the failure as genericPRMergeUnsettledbefore it was ever reached. A repo with norequired_status_contextsconfigured always resolvesSatisfiedhere (no-op, zero behavior change). - Check runs empty +
hadChecks(store:LinkedPRState.HasHadChecks): returnPRMergeUnsettled(MergeableStateintentionally omitted — see invariant). - Check runs empty + post-push dwell active (
LastHeadSHAUpdatenon-zero andtime.Since < PostPushDwell, default 90 s): returnPRMergeUnsettled(MergeableStateintentionally omitted — see invariant). ZeroLastHeadSHAUpdate(cold start / post-restart) falls through. - Check runs empty +
mergeable_state ∉ {"", "unknown"}: returnPRMergeUnsettled(MergeableStateset — R3/branch-protection signal forcheckCIGate). - Check runs empty + a configured required context not yet confirmed satisfied (ADR-933):
classifyRequiredContexts()is consulted again; aRequiredContextsPendingresult (missing entirely, or reportedskipped/neutral/still-pending) → returnPRMergeUnsettled(RequiredContextsStatus/RequiredMissing/RequiredPendingset) instead of falling through to rule 18’s “no CI configured” — nothing has regressed (so no CI-fix reinvoke fires from this alone; seeclassifyCIFromRequiredContextsinengine/ci.go), it simply hasn’t reported yet. - Check runs empty +
mergeable_state ∈ {"", "unknown"}+ required contexts satisfied (or unconfigured): returnPRMergeReady(no CI configured) — but only if the suite hold (below) is clear: a suite reporting runs, or a run-less suite younger than the post-push dwell, contradicts “no CI” and returnsPRMergeUnsettledinstead (#1822). - Check runs non-empty: classify via
github.ClassifyCheckRuns(checkRuns)(#958) — reduces to the latest run per check name (highest ID wins), then: any pending (queued/in_progress) at any name → returnPRMergeUnsettled(CheckRunsset) regardless of any failed run, whether that failed run is a different check name or a stale (lower-ID) entry for the same name superseded by the pending rerun; else any failed (failure/timed_out/action_required) → returnPRMergeBlocked(CheckRunsset); else all green: before returningPRMergeReady(ADR-933),classifyRequiredContexts()is consulted a third time (fetching classic commit statuses viaFetchCombinedStatusonly when at least one required name isn’t already resolvable from the present check runs) — a non-Satisfiedresult returnsPRMergeBlocked/PRMergeUnsettledinstead (same Failed→Blocked, Pending→Unsettled mapping as rules 13/17), closing the exact hole #933 was filed for: an all-skipped/neutralcheck-run set on the head must never resolve toPRMergeReadywhen a required context hasn’t confirmed success. Only once required contexts are satisfied (or unconfigured) and the suite hold (below) is clear (#1822) does this rule returnPRMergeReady(CheckRunsset); a held suite returnsPRMergeUnsettledwithCheckRunsset andMergeableStateomitted. Pending-over-failed (withinClassifyCheckRunsitself) is still evaluated first, reversing the pre-#958 order where a failed run short-circuited toPRMergeBlockedbefore pending was even checked.
MergeableState omission invariant: checkCIGate uses settle.MergeableState to detect R3 (OPEN+BLOCKED+no-check-runs+fabrik:awaiting-ci elapsed → pauseForRequiredNeverRunningCheck). The primitive omits MergeableState from the PRSettleResult in the hadChecks, post-push dwell, and HeadSHA-empty cases. This prevents R3 from misfiring on cases where hadChecks == true (checks have been observed) or where GitHub simply hasn’t computed mergeability yet (post-push window). Only a non-empty MergeableState in the settle result is genuinely relevant for R3/branch-protection timeout checking.
Startup failure (#2052). settlePRMergeStateForCIGate (the form handleMergeAndCIGates calls; settlePRMergeState itself never retriggers) additionally treats a PR head with zero check runs and a startup_failure workflow run as CI infrastructure for a wait_for_ci stage — inside the clean shortcut (read the cached check runs first) and ahead of the hadChecks/dwell/R3 returns. Retrigger/wait return PRMergeUnsettled with MergeableState omitted; abandon sets PRSettleResult.StartupFailure. See §6.29.
Suite-aware completeness (#1822, ADR-1822). A check run exists only once its job has been scheduled onto a runner, so a needs:-dependent job (or one queued for a busy runner) is invisible to FetchCheckRuns, and an all-green prefix of a workflow reads as a complete pass — while GitHub itself reports mergeable_state == "clean" whenever no required check is outstanding. The check suite is created at queue time and stays non-completed until every job has run, so it is the signal missing from the run set. settlePRMergeState therefore consults ciSuiteHold (engine/ci_suites.go) at every point that would otherwise return PRMergeReady: rule 9, rule 18 and rule 19. It is deliberately consulted last and only on a would-be-green path, so a confirmed failure (PRMergeBlocked), a pending run, or an unsatisfied required context (ADR-933) is never masked, and a red or pending SHA does not pay for the extra call.
- Live read, never cached.
FetchCheckSuites(GET /repos/{o}/{r}/commits/{sha}/check-suites, paginated andtotal_count-verified likeFetchCheckRuns, #1539) is a pass-through onGitHubClient,boardcache.ReadClient,GitHubAdapterandCacheImpl: thecheck_suitewebhook is a no-op (§ webhook table), so there is no cached suite state to serve. It needs onlychecks: read, already held under App auth — no permission or manifest change. - Discriminator (
gh.OutstandingCheckSuites). A suite is outstanding whenstatus != "completed"and eitherlatest_check_runs_count > 0(real work in progress — the incident case) or it has zero runs but is younger than the post-push dwell, measured from the suite’s owncreated_at— or it belongs to thegithub-actionsApp (#1829), in which case age and run count are irrelevant: any non-completedgithub-actionssuite is outstanding, full stop. GitHub creates exactly one check suite per workflow run, so a run-lessgithub-actionssuite past the dwell always means a job hasn’t been scheduled onto a runner yet, never an idle installation — the incident’s ownE2E (Playwright)job waited 17 minutes for a runner, well past any plausible fixed dwell, which is the narrower window #1829 closed (the original #1822 incident happened to haveruns > 0on itsgithub-actionssuite by the time the gate was read, so it never hit this path). An old run-less non-completed suite from any other App is inert — an installed App (cursor,claude,coderabbitai,github-pages…) whose suite sitsqueuedforever — and is ignored. “Every suite must be completed” is explicitly rejected: it deadlocks on inert Apps. The rule is “any” suite, never “the” suite (GitHub Actions creates one suite per workflow run). Agithub-actionssuite that has genuinely settled tocompletedwith zero runs (e.g. an all-skipped-jobs workflow, or a startup/parse failure) still clears normally — thecompletedfilter is checked before the app-slug branch, so this carve-out cannot itself deadlock the gate. - Post-push mitigation. The window in which a real suite exists with zero runs before its first job registers is covered by the age bound above (a variant of option (b), reusing
PostPushDwell, default 90s, anchored on GitHub’screated_atso — unlike the in-memoryLastHeadSHAUpdateof rule 15 — it survives a restart), plus a scoped form of option (a): a suite reporting runs contradicts an empty run set, so rule 18 holds. A repo with genuinely no CI has no outstanding suite and still clears. Option (c) (the Actions workflow-run read) was rejected: at the time it neededactions: read, which the App was deliberately not granted (ADR-1756; since #2105actions: writeis required at startup), and it still does not generalize beyond GitHub Actions. A missing or unparseablecreated_aton a run-less suite counts as young (hold). - Fail-safe. A suite read error holds (
PRMergeUnsettled; the reason carries the error). The hold isPRMergeUnsettledwithMergeableStateomitted — it is a wait for workflow work, not an R3 case, so the invariant above is preserved.checkMergeabilityGateclaims it beforecheckCIGate, so it is bounded bysettleAwaitingCIScan’s unconditionalCIBackstopTimeout(ADR-1410, §6.14.5): a suite stuckin_progressforever escalates throughpauseForCITimeoutlike a stuck check run. That comment names the outstanding suite and app (suiteTimeoutNote, one best-effort live read at escalation time). - One primitive, every surface. The same
ciSuiteHoldgatesclassifyLandingCI(and through itsingletonFastPathEligible, §6.20) andpollTrainCI(both green returns), so the advance gate, the landing poll, the trial poll and the singleton fast path cannot disagree on one SHA. Each is aTrainCIPending(never green) on a hold or a read error. - Residual limits. No suite read can see a workflow that has not yet been triggered (
workflow_runchains, later events). The ordinary non-train yolo auto-merge path (attemptMergeOnValidate) has no CI classification of its own and is protected transitively —stage:Validate:complete, which precedes landing, is conjunctive on this gate;MergePR’sunstable-accepting self-gate is unchanged, and a Validate stage withwait_for_ci: falseis out of scope. A member that goes red after admission to the merge train is #1821’s concern.
FetchCheckRuns consolidation: Two callers consume settle.CheckRuns from the PRSettleResult instead of making independent FetchCheckRuns calls:
buildCIFixComment()(called insidedispatchCIFixReinvoke()) usessettle.CheckRunsfor the PR-head check runs — eliminating the separateFetchCheckRuns()call that previously lived in the CI-fix path.pauseForConvergenceFailed()usessettle.CheckRunsfor the CI summary in the convergence-budget-exhausted pause comment — no independentFetchCheckRunscall for the convergence path.
The base-branch FetchCheckRuns(baseSHA) inside buildCIFixComment() is a non-gate diagnostic read — it fetches check runs for the base branch’s HEAD SHA for regression-vs-pre-existing classification only, and is structurally independent (different SHA; result is not used by either gate).
Stale-cache self-healing on webhook-less deployments (#958 leg 3): settlePRMergeState()’s step 11 FetchCheckRuns() call routes through e.readClient, which on a live deployment is boardcache.CacheImpl.FetchCheckRuns(). Two additive fixes close the “cached failure never refreshes” gap:
- Store-side pruning:
internal/itemstate/store.go’sPRHeadSHAUpdatedhandler prunesLinkedPRState.CheckRunswhenever the head SHA genuinely changes (not on first linkage) — a run from a prior push can no longer shadow the current SHA’s classification.HasHadChecksis untouched (it tracks “has this PR ever had checks,” not per-SHA state). - Read-side re-validation:
CacheImpl.FetchCheckRuns()no longer trusts a cache hit unconditionally. It classifies the cached runs viagithub.ClassifyCheckRuns()first; a cached WAIT or READY classification is still served from cache, but a cached FAILED classification forces a live GitHub refetch before being trusted — the exact case where a stale cached failure (nocheck_runwebhook to refresh it) would otherwise keep the gate returning FAIL forever without manual label removal.
Together these mean a genuinely-stuck failure still reaches PRMergeBlocked (the live refetch confirms it), but a stale failure whose live status has since gone green or pending self-heals on the next poll without operator intervention.
6.5 CI Gate and CI-Fix Reinvoke
Channel events (#1968):
pauseForCITimeout(a fresh CI-gate timeout pause) emitsci-timeout; the CI-fix cycle counter reaching one short of its limit emitscycle-limit-near; see §7.14.
6.5.1 Two-Phase CI Gate
The CI gate has two paths that handle different timing scenarios:
Path 1: attemptMergeOnValidate() (Merge Guard)
- Embedded directly in the auto-merge path for Validate+yolo items. It performs no CI classification of its own (an earlier version fetched check runs with pending/timeout bookkeeping; that was removed and this section had drifted — corrected in #1822).
- For a non-train yolo item it calls
EnablePullRequestAutoMerge(GitHub enforces required checks natively), falling back toMergePRon failure, whose only CI protection is its ownmergeable_state ∈ {clean, unstable}self-gate (ADR-072). - CI protection is therefore transitive: landing follows
stage:Validate:complete, which for await_for_ci: truestage is itself conjunctive on Path 2 below — the suite-aware gate (§6.4, #1822). No separate landing-time CI gate exists here (it would need its own escalation owner to be bounded);MergePR’s self-gate andwait_for_ci: falsestages remain unprotected against a partial check set.
Path 2: settleAwaitingCIScan() (checkCIGate())
- Runs for open (non-closed), non-paused items with
fabrik:awaiting-cion stages withwait_for_ci: true. Sourced directly fromboard.Items, not from the main catch-up loop’sdeepFetchCandidates— as of #1270, this is the sole per-poll evaluator of the CI gate for these items, independent ofitemMayNeedWork,selectDeepFetchCandidates’s cooldown pre-filter, and the main loop’s own per-item admission gate (which now admitshasComplete-only items — see “Admission model” below). Field evidence (an item stranded 80+ minutes with zero"settle"-tagged log lines, despite sibling items processing normally the same poll) showed that shared, admission-gated pipeline could silently exclude anawaiting-ciitem without ever logging why;settleAwaitingCIScancloses that class of gap by never depending on it in the first place. - For each qualifying item, deep-fetches details (
FetchItemDetails), builds aphase1Ctx, and runs the identicalcatchUpPhase1Handlerschain the main catch-up loop uses (handleDependencies → handleReviewGate → handleAutoMergeConvergence → handleMergeAndCIGates) — it does not reimplement any gate logic, so it can never become a third, divergent owner ofstage:X:complete(see “Clearing-owner invariant” below). If the handler chain does not claim the item,runCatchUpPhase2runs immediately afterward — the same gated stage-advancement/landing step the main loop’s Phase 2 performs (see §6.6.6, which this scan now shares responsibility for on the CI-clears-this-poll path). - Admission model (post-#1270): the main catch-up loop’s per-item admission gate is
hasCompleteonly — it no longer has anfabrik:awaiting-ci/wait_for_cicarve-out. Sincefabrik:awaiting-ciandstage:X:completeare mutually exclusive in steady state, everyawaiting-ciitem is!hasCompleteand is therefore always routed tosettleAwaitingCIScan, never to the main loop, until the gate clears — so the two paths never both process the same item in the same poll (no double-dispatch risk forCIFixCyclesor worker dispatch). - Orphan-column handling: an
awaiting-ciitem whose board column resolves to no stage, a non-wait_for_cistage, or aHoldingStage/Unmanaged/CleanupWorktreestage cannot have its CI gate evaluated there (e.g.merge_train’sQueuedholding column has no CI-gate concept of its own — see §6.11/ADR-059 D3).settleAwaitingCIScanlogs the stray column and retries via the sharedrecordSettleRetry/escalateSettlesettle-scan helpers (the same pattern assettleNoWorkNeededScan,settleMergeTrainMemberCloses,settleNonDefaultBaseCloses,settleChildPlacements), keyed by the synthetic__awaiting_ci_orphan__retry-stage constant. AfterMaxRetriesfailed passes, the issue is paused (fabrik:pausedadded,fabrik:awaiting-ciremoved) with an explanatory comment naming the stray column — never left silent. A pass that resolves to a realwait_for_cistage clears the orphan-retry counter, so a transient bounce through a stray column doesn’t accumulate toward escalation. - Deep-fetch-failure handling: an item that does resolve to a real
wait_for_cistage but whoseFetchItemDetailscall fails (permissions, a deleted issue node, sustained API errors) is the other “gate genuinely cannot be evaluated” case — the scan can never reachcheckCIGatefor it either. This shares the same__awaiting_ci_orphan__counter andescalateAwaitingCIOrphanFailureescalation path as the orphan-column case (mirroringescalateNoWorkNeededFailure’s single generic-message counter for its own two failure causes), butescalateAwaitingCIOrphanFailurere-resolves the item’s current stage at escalation time so the posted comment describes whichever cause is actually current — a stray column, or a persistent fetch failure — rather than always claiming a stray column. - Has FRESH data from
FetchItemDetails()and receives aPRSettleResultfromsettlePRMergeState()(called once per Phase 1 iteration, before both gates) — no separate REST calls formergeable,mergeable_state, or PR-head check runs - Timeout tracking is dwell-specific (ADR-1410): the later of
FetchLabelAppliedAtonfabrik:awaiting-ci(durable across restarts) and the latestfabrik:pausedremoval (FetchLabelRemovedAt, §6.14.6) for the R3/mergeable-state-blocked elapsed dwells and thesettleAwaitingCIScanbackstop; in-memoryLinkedPRState.LastCIProgressAt(reset by a restart to its safe “never observed” default) for the check-run-pending liveness-stall dwell — see §6.14.5. checkCIGatereturns(blocked, ciFailure, timedOut, terminated bool)(ADR-1223). Four outcomes:(ciBlocked=true, ciFailure=false, ciTimedOut=false, ciTerminated=false)— checks still pending, and either no liveness dwell applies yet or it hasn’t elapsed; skip to next item (fabrik:awaiting-cialready present — no additional label needed)(ciBlocked=true, ciFailure=true, ciTimedOut=false, ciTerminated=false)— failure confirmed (viaclassifyCIFromCheckRunsor, since ADR-933,classifyCIFromRequiredContextsfor a confirmed required-context failure); fires unconditionally, regardless of elapsed time (ADR-1410, R3 — a verdict is never a timeout);fabrik:awaiting-ciapplied idempotently; dispatchdispatchCIFixReinvoke()or pause on cycle limit(ciBlocked=false, ciFailure=false, ciTimedOut=true, ciTerminated=false)— a genuine liveness dwell elapsed: either check runs are pending with no observed progress forCIWaitTimeout(LastCIProgressAt-anchored), or the R3/mergeable-state-blocked case’sfabrik:awaiting-cihas been present ≥CIWaitTimeout(FetchLabelAppliedAt-anchored) — never fires for a confirmed failure or for merely-elapsed time on healthy, progressing CI; pause viapauseForCITimeout()(ciBlocked=false, ciFailure=false, ciTimedOut=false, ciTerminated=true)— processing already terminated via a directpauseIssuecall: the R2 closed-without-merging branch (pauseForPRClosedNotMerged) or the R3 required-check-never-runs branch (pauseForRequiredNeverRunningCheck, insideclassifyCIFromMergeableState). Before ADR-1223, both branches fell through to the same all-false tuple used by “gate cleared,” sohandleMergeAndCIGatesdid not claim the item and Phase 2 could advance an issue that had just been paused in the same poll pass.handleMergeAndCIGatesnow checksciTerminatedfirst, ahead of the other three fields, and claims (return true) immediately when it is set.classifyCIFromRequiredContextsandclassifyCIFromCheckRunsnever call a pause helper, so every value they can produce always carriesciTerminated=false— their exemption from the ADR-1223 convention is by design (neither ever pauses an item), not an oversight. SeecheckCIGate’s Go doc comment inengine/ci.gofor the full per-branch outcome table.
- Gate cleared outcome: When all checks pass — and, since ADR-933, every configured required context has confirmed success on the exact head SHA (
settle.RequiredContextsStatus == RequiredContextsSatisfied, the default when unconfigured) — or no check runs exist and the same required-context condition holds (R5),checkCIGatecallsaddCompleteLabelAndRemoveCI: addsstage:X:completeand removesfabrik:awaiting-ci. This is whereitemstate.ValidateCompletedAtSHA(“validate-sha”) is recorded — ADR-933 exists precisely so this can never happen for a head whose required status hasn’t actually confirmed success. PRMergeQueuedhand-off (ADR-058 D4 FR-1): when the PR is in GitHub’s merge queue,checkCIGatereturns(true, false, false)— block with nofabrik:awaiting-cichurn, mirroring thePRMergeUnsettledfall-through. The queue owns the merge decision while the PR waits; the next poll re-evaluates.
Clearing-owner invariant for wait_for_ci: true stages: Exactly two code paths may add stage:X:complete for these stages; they are mutually exclusive by PR state at the time of evaluation:
- Normal path (
addCompleteLabelAndRemoveCI, called fromcheckCIGate): PR is still open; CI checks clear (or R5 — no CI configured). Runs insidehandleMergeAndCIGates, invoked from either the main catch-up loop’s Phase 1 (for items withstage:X:completealready present — a narrow re-evaluation window, not the common case) orsettleAwaitingCIScan(Path 2 above; the common case, since openawaiting-ciitems are routed there exclusively as of #1270). Both call sites reuse the samehandleMergeAndCIGates/checkCIGatecode —settleAwaitingCIScanis a second call site, not a second owner: it never mutatesstage:X:completeorfabrik:awaiting-cidirectly, only through this shared function. - PR-merged recovery path (
runValidatePRTerminalAdvance, ADR-056 D2): PR is already merged. Addsstage:X:completefor every gate-checked stage (WaitForCIorWaitForReviews) missing its completion label. Runs only for Validate-stage items withoutfabrik:auto-merge-enabled. See §3.2 (“Awaiting CI” table, R4 row).
Both paths call EnsureLabel, which is idempotent. The advancedItems map (keyed by issue number, populated by advanceToNextStage) prevents double-advancement within a single poll cycle even if both paths fire in the same poll. This supersedes the original ADR 032 single-owner claim; the two-path structure is captured by ADR-056 D2.
Two different timeout strategies:
- Path 1 (merge guard):
itemstate.Store→LinkedPRState.CIMergePendingSince. Acceptable because merge-guard state is transient — engine restarts simply re-evaluate CI on the next poll (store is in-memory only; not persisted across restarts). - Path 2 (catch-up loop): dwell-specific since ADR-1410 (see above) —
FetchLabelAppliedAtonfabrik:awaiting-ci(durable, label persists across restarts) for the R3/mergeable-state elapsed dwells and the backstop; in-memoryLastCIProgressAt(reset by a restart, safe cold-start default) for the check-run-pending liveness dwell. Both are accurate from the start of the relevant window: the label from the moment Claude emits FABRIK_STAGE_COMPLETE on await_for_ci: truestage,LastCIProgressAtfrom the first post-restart check-run observation.
6.5.2 CI Fix Reinvoke Mechanics
The catch-up loop Phase 1 calls checkCIGate() after the review gate check. When CI has failed:
- Worker guard (
snap.Worker() != nil): If a CI-fix goroutine from a previous poll is still running for this item, skip dispatch entirely (no further checks) - No-op debounce guard (
snap.LastCIFixNoOpSHA() == settle.PR.HeadSHA, #958 leg 2): If the most recent CI-fix reinvoke for this exact head SHA already observed no new commit pushed, skip dispatch entirely (no cycle-limit check, no increment) — dispatching again would just repeat the same no-op and burn cycle budget for nothing. The guard is keyed to the current head SHA, so it clears implicitly the moment a genuine fix advancesHeadSHA(before it applies,ciFlakeRerunre-runs the failed jobs once for that head — #2072 R2, §6.29);CIBackstopTimeoutremains the backstop if CI never resolves on the stuck SHA (ADR-1410). - Cycle limit check:
snap.CIFixCycles(stage.Name)is compared againstMaxCiFixCycles(default 5)- If exceeded:
pauseForCIFixCycleLimit()addsfabrik:paused+fabrik:awaiting-inputand posts comment - If not exceeded: increment count, dispatch reinvoke via
dispatchCIFixReinvoke():- Applies
WorkerEntered(prevents double-dispatch) - Acquires semaphore slot (respects
MaxConcurrent) - Snapshots
gitHeadSHA(workDir)before reinvoking (headBefore) — the samegit rev-parse HEADhelper used by the Implement/Review turn-extension progress check (§ detectProgress) - Calls
buildCIFixComment()to construct a syntheticgh.Comment(DatabaseID: 0) with a structured CI failure report — usessettle.CheckRunsfor the PR-head check runs (no separateFetchCheckRunscall); classifies each failed check as NEW REGRESSION (not failing on base branch) or pre-existing (also failing on base branch) by callingFetchCheckRuns(baseSHA)for the base-branch HEAD SHA (a non-gate diagnostic read — different SHA, independent of the settle result) - Failure context (#1991, ADR-1991): for each NEW REGRESSION check,
ciFailureDetail()appends the check’s failure detail beneath its line — up to 5 non-empty annotations (liveGET /check-runs/{id}/annotations?per_page=50, first page only, via the optionalcheckRunAnnotationFetcherinterface asserted one.client— never the board cache; Actions’ generic “Process completed with exit code N.” annotation is shown only when nothing more specific exists), then anyoutput.summary/output.textthe check set — each capped at ~500 chars, ~2000 chars per check (truncateMiddle). Annotations are the only channel that carries an Actions job’s::error::text to the API ($GITHUB_STEP_SUMMARYnever populatesoutput.summary, #916). Every failure here is soft: a fetch error is logged ([#N ci-fix-reinvoke] warn: …) and the prompt degrades to the check name and conclusion; a client without the capability (e.g. simgh) yields no annotations. Pre-existing failures get no detail (the prompt tells the agent not to fix them). - Prompt log line (#1991): the synthetic comment is never posted to GitHub, so the
buildhook logs it —[#N ci-fix-reinvoke] prompt (<n> bytes): "<%q-rendered body>"— bounded to 4 KB (truncateMiddle) and one line. The liveTestCIFixReinvokeasserts the line carries the failing check’s name and the run-ID nonce. A log line is not a charged cycle (ADR-1812 refunds a did-not-run reinvoke), so tests must not count lines as cycles. - Calls
processComments()with the synthetic comment and theci_fix_skill(falls back tocomment_skillif unset) - After
processComments()returns, snapshotsgitHeadSHA(workDir)again (headAfter). IfheadBefore == headAfter(and both are non-empty — no git-command failure), appliesCIFixNoOpRecorded{SHA: headAfter}, populatingLinkedPRState.LastCIFixNoOpSHAfor the guard above to consult on the next poll - On exit: releases semaphore, applies
WorkerExited
- Applies
- If exceeded:
DatabaseID: 0 guard: Synthetic CI-fix comments have DatabaseID: 0 and no ReactionNodeID, which skips the 👀 and 🚀 reaction steps in processComments() (reactions require a real GitHub comment ID or, for review bodies only, a review node ID — #1953 R8).
CI-fix cycle counter reset: StageState.CIFixCycles[stageName] is reset to 0 by clearFailedStage() (via EngineCyclesCleared mutation) when the user removes fabrik:paused from a paused-failed item, allowing fresh CI-fix attempts after human intervention. StageState.RebaseCycles[stageName] is reset in the same call for the same reason. LinkedPRState.LastCIFixNoOpSHA is not cleared by this reset — it is scoped to a specific SHA and self-obsoletes once the head SHA changes, so a stale no-op record from before the pause cannot block a fresh CI-fix attempt on a since-advanced head.
6.5.2a Startup Failure on the Gate (#2052)
For a wait_for_ci stage whose PR head has zero check runs, settlePRMergeStateForCIGate also reads the workflow runs; a startup_failure run is retriggered by close/reopen (at most twice) and then escalated through pauseForCIStartupFailure — a pause naming the startup-failed run — instead of waiting for the CI backstop, clearing the gate with CI never run (mergeable_state: clean), or ending in the R3 “required check never runs” pause. Full description in §6.29.
6.5.3 CI Fix Reinvoke vs Review Reinvoke
| Aspect | Review Reinvoke | CI Fix Reinvoke |
|---|---|---|
| Trigger | Unresolved PR review thread comments | CI check runs with failure/timed_out/action_required conclusion |
| Source data | item.LinkedPRReviewThreadComments |
settle.CheckRuns from settlePRMergeState() (no separate REST call for PR-head checks) |
| Label on waiting | fabrik:awaiting-review (always applied) |
fabrik:awaiting-ci (applied by handleStageComplete on FABRIK_STAGE_COMPLETE; present for both pending and failed checks — covers the full CI-await window) |
| Timeout tracking | In-memory ReviewWaitTimeout timer |
FetchLabelAppliedAt on fabrik:awaiting-ci (durable across restarts; label is present from FABRIK_STAGE_COMPLETE onwards) |
| Cycle counter | snap.ReviewCycles(stageName) / ReviewCycleIncremented |
snap.CIFixCycles(stageName) / CIFixCycleIncremented |
| Max cycles | MaxReviewCycles (default 5) |
MaxCiFixCycles (default 5) |
| Skill | comment_skill |
ci_fix_skill (falls back to comment_skill) |
| Synthetic comment | PR review thread text | Structured CI failure report with NEW REGRESSION classification |
| Worker guard | Yes — snap.Worker() != nil |
Yes — snap.Worker() != nil |
| Thread resolution | Yes — ResolveReviewThread() after processing |
No |
| PR summary comment | Yes — "<StageName> (review feedback addressed)" on linked PR |
No |
| Stage gate config | wait_for_reviews: true |
wait_for_ci: true |
6.6 Merge-Conflict Gate and Rebase Reinvoke
The merge-conflict gate is a third prong of the catch-up loop Phase 1, sitting between review reinvoke and the CI gate. It is the direct response to the failure mode in which a base-branch advance during the CI-await window leaves a PR unmergeable — the CI gate alone will happily keep polling check runs on the branch head while the real blocker is a conflict.
Note: attemptMergeOnValidate() (§5.4, the legacy auto-merge path for stages without wait_for_ci: true) now uses the same snap.RebaseCycles(stage.Name) + RebaseCycleIncremented, dispatchRebaseReinvoke(), and pauseForRebaseCycleLimit() pattern as the conjunctive gate described in this section. Both paths share the same per-item store field; MaxRebaseCycles applies to both.
Independence from MergePR’s CI precondition (ADR-933): the rebase-cycle machinery in this section is driven exclusively by settle.Status == PRMergeConflicting (a confirmed base-branch conflict, mergeable == false). MergePR’s own mergeable_state self-gate (§5, “MergePR’s own CI precondition”) is a separate, later check inside MergePR itself — a refusal there (gh.ErrNotMergeableCI, e.g. a required check still blocked) is not a conflict and is never observed by checkMergeabilityGate or checkAutoMergeConvergence. No call site reads MergePR’s return value to apply fabrik:rebase-needed or increment RebaseCycles, so a CI refusal cannot consume a rebase cycle or trigger pauseForRebaseCycleLimit.
6.6.1 Gate Mechanics
checkMergeabilityGate() runs only when stage.WaitForCI is true (the same opt-in that admits items to the catch-up window via fabrik:awaiting-ci). It returns (blocked, conflict):
(false, false)— clear:PRMergeNoPR,PRMergeTerminal,PRMergeReady, orPRMergeBlocked(CI failed — no base conflict; deferred to the CI gate). Any stalefabrik:rebase-neededlabel is removed. Caller falls through to the CI gate.(true, false)—PRMergeUnsettled(mergeable == null, still computing, or transient API error) orPRMergeQueued(the PR is in GitHub’s merge queue — ADR-058 D4). The gate blocks but no label is applied — unknown/queued states must not produce label churn (ADR-032 R10c). ForPRMergeQueuedthis is the FR-1 hand-off: a human-enqueued non-yolo PR at a gate-checked stage simply waits for the queue. Caller skips to the next item; the next poll re-evaluates.(true, true)—PRMergeConflicting(mergeable == false, confirmed conflict).fabrik:rebase-neededis applied idempotently. The caller inpoll()dispatches a rebase reinvoke or pauses on the cycle limit.
The gate’s inputs come from settlePRMergeState() (§6.4), called once per Phase 1 iteration before both gates. FetchPRMergeableFields() (single-PR endpoint, not the list endpoint used by FetchLinkedPR) provides both mergeable and mergeable_state in one request — the list endpoint does not return mergeable.
Clearing-owner invariant for fabrik:rebase-needed: Four code paths may remove this label; all call RemoveLabelFromIssue, which is idempotent:
- Primary clearing owner (
checkMergeabilityGate): whensettle.Status == PRMergeReadyorPRMergeBlocked(i.e.,mergeable == true— Claude’s rebase push landed and GitHub confirms no conflict). The gate falls through to the CI gate on the same poll. - PR-merged recovery path (
runValidatePRTerminalAdvance): when the PR is already merged; removes all gate labels includingfabrik:rebase-neededas part of terminal advance. Runs only for Validate-stage items withoutfabrik:auto-merge-enabled. - Convergence success path (
checkAutoMergeConvergence): when the PR merges under GitHub’s auto-merge after a conflict was resolved; removesfabrik:rebase-neededas part of convergence cleanup. - SHA-invalidation scan (
handleValidateSHAInvalidation, §2.16): when the linked PR’s HEAD SHA changes afterstage:Validate:completewas recorded. Distinct in kind from the first three, which each re-check live mergeability before clearing — this path clears without re-checking mergeability (apart from the landed/terminal and pure-catch-up exemptions of #2158, which leave the label alone), because a new push invalidates the entire prior Validate-completion episode the label was scoped to, regardless of whether the conflict determination would still hold (#1225).
Additionally, cleanupClosedIssueTransientLabels performs a defensive sweep when the issue is closed — except when the item resolves specifically to the Validate stage, where fabrik:rebase-needed is deliberately left for the settle-owner pair (runValidatePRTerminalAdvance/settleClosedValidateAdvance, §6.15) to clear atomically as part of its own transition, rather than being stripped independently by the generic sweep (R6, ADR-1387; see §6.15’s own discussion of why the sweep previously raced this label, and why the exclusion is scoped to stage.Name == "Validate" rather than stageIsGateChecked generally).
6.6.2 Ordering Against the CI Gate
The merge-conflict gate runs before the CI gate so that a confirmed conflict preempts CI-await polling. The rationale: a PR that cannot merge has no reason to wait for CI, and Claude on CI-fix reinvoke cannot productively act on a branch that must first be rebased. When the merge gate emits conflict, Phase 1 continues without reaching the CI gate.
When the merge gate clears (mergeable == true, CI failed, or no conflict), Phase 1 falls through to the CI gate on the same poll. When the merge gate is blocked (PRMergeUnsettled — mergeable == null or a transient API error), Phase 1 skips to the next item — the next poll re-evaluates once GitHub has a definite answer or the API recovers.
6.6.3 Rebase Reinvoke Mechanics
When checkMergeabilityGate returns conflict=true:
- Worker guard (
snap.Worker() != nil): if a rebase goroutine from a previous poll is still running for this item, skip dispatch entirely (no cycle-limit check). - Cycle limit check:
snap.RebaseCycles(stage.Name)is compared againstMaxRebaseCycles(default 3 — lower than review/CI because rebase either works in one shot or needs human judgment):- If exceeded:
pauseForRebaseCycleLimit()pauses the issue withfabrik:paused+fabrik:awaiting-input;fabrik:rebase-neededis intentionally left in place so the reason is visible. - If not exceeded: increment count, dispatch
dispatchRebaseReinvoke():- Applies
WorkerEntered(prevents double-dispatch) - Acquires semaphore slot (respects
MaxConcurrent) - Calls
buildRebaseComment()to construct a syntheticgh.Comment(DatabaseID: 0) instructing Claude togit fetch origin <base> && git rebase origin/<base>, resolve conflicts conservatively (never dropping code from base), watch for semantic collisions (duplicated ADR numbers, migration slots), run the project’s build + tests, and force-push with--force-with-lease. - Calls
processComments()with the synthetic comment and therebase_skill(falls back tocomment_skillif unset) - On exit: releases semaphore, applies
WorkerExited
- Applies
- If exceeded:
DatabaseID: 0 guard: like the CI-fix and review synthetic comments, the rebase synthetic comment uses DatabaseID: 0 (and no ReactionNodeID) so processComments() skips the 👀 and 🚀 reaction steps (no real GitHub comment exists to react to).
6.6.4 Why Claude Rebases (Not the Engine)
The engine could in principle run git fetch && git rebase directly from the worker, but does not. Automatic rebase is right most of the time and catastrophically wrong sometimes: two PRs independently pick adr-054.md, both PRs pick migration slot 0042, both PRs add a new line at the same point in a shared config file. A mechanical rebase drops one side silently; a Claude-driven rebase can rename, renumber, and keep both contributions. The synthetic comment explicitly flags this — “watch for semantic collisions” — so Claude’s judgment is applied where it matters.
The cost is a re-invocation rather than an inline exec.Cmd. This is why MaxRebaseCycles defaults to 3 rather than 5: if Claude cannot rebase cleanly in three attempts the conflict almost certainly needs a human.
6.6.5 Rebase Reinvoke vs CI Fix Reinvoke
| Aspect | CI Fix Reinvoke | Rebase Reinvoke |
|---|---|---|
| Trigger | CI check runs in failure state | mergeable == false on linked PR |
| Source data | settle.CheckRuns from settlePRMergeState() (no separate REST call for PR-head checks) |
settle.Status == PRMergeConflicting from settlePRMergeState() (FetchPRMergeableFields provides mergeable) |
| Label on waiting | fabrik:awaiting-ci (only on confirmed failure) |
fabrik:rebase-needed (only on confirmed mergeable == false) |
| Order in Phase 1 | After merge-conflict gate | Before CI gate |
| Cycle counter | snap.CIFixCycles(stageName) / CIFixCycleIncremented |
snap.RebaseCycles(stageName) / RebaseCycleIncremented |
| Max cycles | MaxCiFixCycles (default 5) |
MaxRebaseCycles (default 3) |
| Skill | ci_fix_skill (falls back to comment_skill) |
rebase_skill (falls back to comment_skill) |
| Synthetic comment | Structured CI failure report with NEW REGRESSION classification | Rebase instructions with explicit semantic-collision guidance |
| Thread resolution | No | No |
| PR summary comment | No | No |
| Stage gate config | wait_for_ci: true |
wait_for_ci: true (same opt-in — these are the stages that enter the catch-up window) |
| Label left on pause | fabrik:awaiting-ci removed before pause |
fabrik:rebase-needed retained on pause so the human sees the reason |
References: ADR-028: Merge-Conflict Gate and Rebase Reinvoke
6.6.6 CI ∧ Review Gate Joint-Clearing Sequence
Channel events (#1968): The moment both gates have cleared and the engine reaches
runCatchUpPhase2(or, for yolo, the landing decision inattemptMergeOnValidate) is wherevalidate-settledis emitted;pauseForReviewTimeoutemitsreview-timeout; see §7.14.
When a stage has both wait_for_ci: true and wait_for_reviews: true, the two gates are enforced by two distinct owners: handleReviewGate (which owns escalation and the timeout) and reviewGateBlocksLanding inside attemptMergeOnValidate (which owns the landing decision). The second owner exists because the Phase 1 handler chain alone cannot gate the poll pass in which CI clears.
Within a single poll pass, CI clearing and landing are not separated by a poll boundary. pctx.hasComplete is computed once, before the Phase 1 handler chain runs, and catchUpPhase1Handlers orders reviewGate ahead of mergeAndCIGates. As of #1270, a not-yet-complete fabrik:awaiting-ci item on a wait_for_ci stage reaches this chain via settleAwaitingCIScan (§6.5.1 Path 2), not the main catch-up loop — the main loop’s admission gate is hasComplete-only, so it never sees these items while CI is still pending. settleAwaitingCIScan runs runCatchUpPhase2 (the same gated stage-advancement/landing step the main loop’s Phase 2 performs) immediately after the handler chain if no handler claimed the item, so the same-poll handoff below is unchanged in outcome, only in which iteration performs it. So on the pass where CI turns green:
handleReviewGateruns first.stage:X:completeis still absent (the frozen snapshot was taken before the CI gate ran), so its gate half (checkReviewGate) is skipped — by design, per #617 — while its feedback detection still runs and dispatches any unprocessed feedback (#1953 R1).handleMergeAndCIGatesruns later in the same chain. CI clears;checkCIGatecallsaddCompleteLabelAndRemoveCI, addingstage:X:completeand removingfabrik:awaiting-ci. It returnsfalse(unclaimed) because the gate is now clear.- No handler claimed the item →
settleAwaitingCIScancallsrunCatchUpPhase2immediately, in the same scan pass. For ayoloActiveValidate item withoutfabrik:auto-merge-enabled, this callsattemptMergeOnValidatedirectly.
There is therefore no poll pass in which the item sits at Validate with hasComplete == true and handleReviewGate has not already skipped itself. Relying on the handler chain alone (with no immediate Phase-2-equivalent step) would leave the gate permanently unarmed whenever wait_for_ci defers completion — i.e. on every real PR, since CI is essentially never already green at the instant FABRIK_STAGE_COMPLETE is emitted (#1216).
Landing-decision gate (reviewGateBlocksLanding, engine/reviews.go). attemptMergeOnValidate is the single landing-decision owner for both merge_train modes (ADR-058/ADR-059 “invoke, don’t relocate”). The gate check sits after the cruise and fabrik:auto-merge-enabled early-returns and before the merge_train fork, so it covers auto-merge enable, merge-queue enqueue, direct merge, and advance-to-Queued identically:
- Opt-in first. Returns immediately unless
stage.wait_for_reviewsis true, so stages that don’t use the gate make zero extra API calls. - Live re-fetch, never the item snapshot.
FetchPRReviews+FetchPRReviewRequests, keyed onitem.LinkedPRNumber(falling back toFetchLinkedPRonly when it is 0, which is always the case on abase:<branch>repo).attemptMergeOnValidate’s two callers have different freshness guarantees —handleStageComplete’sitemis the pre-stage snapshot, stale by design because reviewer assignment happens insideMarkPRReady— and a reviewer requested during the CI-await window must still block. - Same clearing condition as
checkReviewGate, via the same shared pure functionreviewGateOutstanding: the gate clears only whenlen(outstanding) == 0 && hasReviews. The two gate sites can never disagree on “outstanding”; any change to that condition must be made in the shared function. - Conservative on error. A failure from either review fetch discards both slices and blocks — trusting whichever call succeeded could produce a false
len(outstanding) == 0read that clears the gate on unknown state. AFetchLinkedPRfailure blocks for the same reason: on abase:<branch>repo that fallback is the only PR-resolution route, so treating a transient error as “nothing to gate on” would land the item with the gate never evaluated at all. Every block path is bounded by thefabrik:awaiting-reviewtimeout (the label it applies is theFetchLabelAppliedAtanchorcheckReviewGatereads), so a sustained API outage pauses for a human rather than hanging. - No PR ⇒ no block; unreadable PR ⇒ block. A definitively absent PR (
pr == nil) means there are no reviewer requests, so the landing proceeds andhandleBrokenReviewLinkageowns the broken-linkage pause. A PR that merely could not be read is unknown state, not an absent PR — the two are deliberately distinguished, and only the former is a safe reason to let a landing through. - Blocks and labels only. It applies
fabrik:awaiting-review(idempotent) and returns; it deliberately does not duplicate the bot-escalation ladder or the wait timeout. Once it blocks,stage:X:completeis present, so on the next pollhandleReviewGateclaims the item withhasComplete == trueandcheckReviewGateowns all escalation from there. This also bounds the live fetch’s cost: it happens on the pass where CI clears, not on every poll.
#617 non-regression. The landing gate is provably unreachable while CI is genuinely still blocking: handleMergeAndCIGates claims the item in that case, so Phase 2 never runs and attemptMergeOnValidate is never called. No extra guard is needed, and catchUpPhase1Handlers ordering is unchanged.
wait_for_ci is not required for the gate to matter. handleStageComplete calls attemptMergeOnValidate whenever yoloActive && !waitForCI, which is before its own fabrik:awaiting-review seeding block runs. The landing gate covers that path too — which is why the fix lives in attemptMergeOnValidate rather than in the handler chain.
Feedback gate (feedbackGateBlocks, engine/feedback_gate.go, #1862 → #1953, ADR-1862, ADR-1953-a). The same choke point gates on unprocessed feedback — issue comments (#1862), and since #1953 also unresolved current-head review threads and unaddressed review bodies (a COMMENTED bot review used to clear every landing gate: reviewers outstanding and the authority verdict only). One predicate, unprocessedFeedback, answers “is anything unprocessed on this item right now”, read live and fail-closed (R2/R5), and is consulted at attemptMergeOnValidate (placed after reviewGateBlocksLanding, and again before the direct-merge fallback), at handleStageComplete’s advance and at runCatchUpPhase2’s non-Validate advance — never inside advanceToNextStage, which is also the terminal Done advance after a merge, and advanceConvergedPRToDone runs only after a confirmed merge so there is nothing to hold there. A hold is logged [#N feedback-gate] holding <where> — 1 unprocessed review body (5356334497) (R6). Review findings go through dropCatchUpFeedback first (#2044, §6.30): a bot review or thread comment made against the live head of a pure merge-train catch-up (self-authored marker comment, exact commit match) is non-actionable for the landing; everything else — humans, conflict-edited catch-ups, older heads, unverifiable markers — is unchanged. Feedback that arrives while a stage invocation is running is not injected into it: the run finishes, then the gate holds the completion from being final and the ordinary reinvoke path processes it (R3, complete-then-reinvoke; the reinvoke’s push restarts any CI wait, an accepted cost). Original #1862 text follows, and still describes the comment half. Without it, Phase 2 merged a Validate-complete item from a snapshot that already contained a fresh steering comment (typically posted just before a cruise → yolo swap), and dispatchCandidates — which runs after the catch-up loop in the same poll — then processed that comment against the merged branch, orphaning the rework.
- Placement. After the dependency guard’s live
FetchItemDetailsre-read (so it sees current comments, including at the stage-completion call site whose item is a pre-invocation snapshot) and immediately beforereviewGateBlocksLanding, i.e. ahead of themerge_trainfork: auto-merge enable, enqueue, direct merge and advance-to-Queuedare all gated identically, from bothrunCatchUpPhase2andhandleStageComplete. The direct-merge fallback re-checks against its own re-read just beforeMergePR. - Predicate (comment half).
findNewComments, unchanged — the same one the non-Validate advance guard inrunCatchUpPhase2uses, so the two can never disagree. It excludes🏭 **Fabrikcomments, 🚀’d comments,CommentProcessedwatermarks and bot service notices; Pruefer’s reviews and inline threads are not initem.Commentsat all. A third-party bot’s issue/PR comment is counted (dispatch processes and 🚀’s it, so it costs one cycle and cannot deadlock). - Return contract. A hold returns
(false, true, nil)— the “landing deferred” return guard 1 uses — sohandleStageCompletesetsshouldAdvance = falseand never advances an unmerged item to Done.stage:Validate:completeis still applied, so Phase 2 re-enters the decision. - Failed live read holds. If the re-read fails the comment state is unknown and the gate holds (conservative, as the review gate does); the dependency guard’s own fail-open is unchanged.
- Process, then re-evaluate. Dispatch processes the comment through the normal path. A rework commit changes the PR head SHA, which
settleSHAInvalidationScanturns into a Validate re-run so the CI gate verifies the new head before any merge; with no commit, the next poll finds the comment 🚀’d and lands. - Re-admission. Every hold records
CooldownRecorded{Reason: "feedback-pending"}(comment-pendingbefore #1953) (githubRecheckInterval()), mirroringhandleReviewGate’sreview-blocked, soitemMayNeedWork’s expiry path re-evaluates the held item without depending onupdatedAtmovement. It does not suppress comment dispatch (the plain new-comment path has no cooldown gate). - No deadlock. The gate counts exactly what dispatch processes. A comment dispatch cannot process ends in the existing pause (comment breakers, no-op breaker, tools-denied at its bound →
fabrik:paused, which the catch-up loop skips), never a silent block; a usage-limit hold or another instance’sfabrik:locked:<user>holds only as long as dispatch itself is skipped. - Dispatcher input agreement (#2160, ADR-2160). The gate and Phase 1’s
handleReviewGateshare one predicate (buildReviewBodyCommentsFromReviews,CommentProcessed, the durablereview-ids-addressedmarker) but read different facts: the gate reads live (e.client.FetchItemDetailson a copy), the dispatcher reads the board cache throughe.readClient. A review submitted before the cache’s staleness baseline — whichSelfWriteObserved(§7.9) moves to the local clock at every engine self-write — never makes the cache look stale, so the gate held on a review body the dispatcher could not see, and nothing dispatched until a restart zeroed the baseline (#2125). When the gate finds review bodies or threads it therefore also builds the dispatcher’s view (areadClient.FetchItemDetailson a copy of the caller’s original item, reduced to the review bodiesreviewBodyCandidatesoffers plus its thread comments,engine/feedback_dispatch_view.go; the processed and durable-marker filters are not re-run — the gate already applied the same ones — so the check adds no comment read); the held IDs that view lacks arependingFeedback.DispatcherGap. Abase:<branch>item, or an item with no PR link and no review data, resolves its reviews over REST on both sides and is never checked. A non-empty gap triggersrepairDispatcherBlindness: log[#N feedback-gate] dispatcher cannot see held feedback (…), applyDeepFetchInvalidated, and record afeedback-dispatcher-gapcooldown ofmergeGateRecheckInterval()so the next poll re-admits the item, performs a real fetch, and Phase 1 dispatches the ordinaryreview-reinvoke(in-flight bail, landed-PR bail,MaxReviewCycles/ReviewBlockedCyclesunchanged). The gate itself still never dispatches and stays fail-closed. The repair is bounded tofeedbackGapRepairBudget(2) per exact(item, gap IDs)key (Engine.feedbackGaps, in memory); past it the gate keeps holding and does not refetch. A steady-state hold the dispatcher also sees costs no extra GraphQL (a cache hit); a failed dispatcher-view read leaves the gap empty and never loosens the hold. Only review bodies and threads are compared — issue and PR comments come from the same live re-read the dispatcher uses. - Residual races (accepted, ADR-1862). The enable-auto-merge and enqueue paths have no second read, so a comment landing between the gate’s read and GitHub’s later merge is not caught; comments after
fabrik:auto-merge-enabledis set are not gated. The post-merge guard (§2.2 item 4) is the backstop where the merge already happened. Comments on a member already inQueuedare out of scope.
Reviewer review submitted during CI-await: a pull_request_review webhook applies PRReviewSubmitted to the board cache store. handleReviewGate’s gate half is still not evaluated during the CI-await window (!hasComplete guard, #617), but its feedback detection is (#1953 R1), so a review body or thread is dispatched as a review-reinvoke during the wait, and the landing gate re-reads review state live on the pass where CI clears, so the reviewer’s decision is honoured at the landing decision rather than one poll later.
Unit tests:
TestValidateLanding_ReviewGateBlocks_WhenCIClearsSamePollandTestValidateLanding_ReviewGateBlocks_MergeTrainOn(engine/poll_test.go) are the primary proof: a singlepoll()call where CI turns green and a reviewer is outstanding must addstage:Validate:completebut take no landing action, under bothmerge_trainmodes.TestValidateLanding_ReviewGateDoesNotArm_DuringGenuineCIAwaitis the #617 guard: nofabrik:awaiting-reviewwhile CI is genuinely pending.TestHandleStageComplete_ReviewGate_BlocksImmediateMergeTrainAdvancecovers thewait_for_ci-independent path.TestAttemptMergeOnValidate_ReviewGate_*(engine/stages_test.go) pin the gate’s own semantics: PR-number fallback, conservative fetch-error blocking, and zero cost whenwait_for_reviewsis unset.TestCIAndReviewGate_JointClearingHandoffremains the coverage for the ordinary across-polls path, wherehandleReviewGateowns escalation oncestage:X:completeis present.
review_authority: authoritative composes unchanged (ADR-1250). The verdict check described in §6.1.1 lives inside reviewGateBlocksLanding’s own len(outstanding) == 0 && hasReviews branch, so it participates in the joint-clearing sequence exactly as the advisory clearing condition does — no separate ordering, no new poll-gap risk. A CHANGES_REQUESTED verdict on the pass where CI turns green blocks the landing exactly as an outstanding reviewer would; the item still gets stage:Validate:complete (from handleMergeAndCIGates) with the landing withheld, and checkReviewGate (Path 2) owns escalation from the next poll onward, same as always.
References: ADR-1216: Review gate checked at the landing decision, ADR-1250: Review authority — advisory vs. authoritative, an axis orthogonal to autonomy
6.7 Pre-Implement Spawn Path
Channel events (#1968):
spawnChildren, once every child is created and wired andfabrik:children-spawnedis applied, emitschildren-spawned(scope grew); see §7.14.
Trigger: The Implement stage dispatcher calls preImplement() before the Claude invocation on every Implement dispatch. preImplement is a no-op unless the Plan stage comment contains FABRIK_SPAWN_CHILD_BEGIN/END blocks AND the parent issue does not yet have fabrik:children-spawned.
FABRIK_SPAWN_CHILD_* marker convention: Plan emits structured blocks in its output to declare sub-issues to spawn:
FABRIK_SPAWN_CHILD_BEGIN owner/repo
TITLE: <single-line title for the new issue>
DEPENDS_ON: <n> # optional — see below
<full scoped spec body — multiple paragraphs OK>
FABRIK_SPAWN_CHILD_END
These blocks persist in the Plan stage comment — they are data, not consumed-and-stripped signals. The preImplement engine step reads them from the most-recent Plan comment body at Implement dispatch time.
DEPENDS_ON: (ADR-1337, sibling ordering): an optional header line expressing a forward-only dependency on an earlier block in the same Plan output — the mechanism for the common case of sequentially dependent, same-repo decomposition slices, where without it Plan could only encode the ordering as prose (“Depends on: Slice N”) that checkDependencies cannot see. When present, DEPENDS_ON: must immediately follow TITLE: with no blank line between them; a DEPENDS_ON:-looking line separated from TITLE: by a blank line is parsed as ordinary body content, not a header. The value is a 1-based index into this Plan output’s own block list — DEPENDS_ON: 2 on block 3 means “block 3 depends on block 2” — never an issue number, since children don’t have issue numbers yet when Plan runs. References must be strictly forward (1 <= DEPENDS_ON < ownIndex); this makes sibling dependency cycles structurally impossible without any graph walk. An absent header leaves the block’s dependency behavior exactly as before this feature — the parallel-star graph. DEPENDS_ON is not restricted to same-repo blocks; the underlying AddBlockedByIssue primitive is already repo-agnostic.
On-thread spawn receipt (#1338): at the moment the Plan comment is posted — well before preImplement ever runs — the engine appends a deterministic note naming the count of well-formed blocks it parsed (via the same ParseSpawnBlocks preImplement itself uses) and stating that no sub-issues exist yet, that they will be created when the parent advances to Implement. This makes the declarative nature of the blocks visible on the thread itself, rather than only in docs/USER_GUIDE.md or the engine log; see docs/stage-lifecycle.md’s “Spawn Receipt Note” section for the posting-path details. A block that fails to parse (#2091) is reported instead: the note becomes a ⚠️ N spawn block(s) could not be parsed: block 2 — … found
Malformed blocks are held and paused (#2091, ADR 2091). ParseSpawnBlocksDetailed returns, besides the well-formed blocks, one SpawnBlockError{Index, Repo, Found, Reason} per BEGIN/END pair that failed to parse (ParseSpawnBlocks is a wrapper returning only the blocks). Header keys (TITLE, DEPENDS_ON) match case-insensitively and whitespace-tolerantly via matchHeader; DEPENDS_ON must still be on the line immediately after TITLE. preImplement and recoverMissingPlanComment call pauseForMalformedSpawn when any error exists: one [#N spawn] log line per block, a 🏭 **Fabrik — malformed FABRIK_SPAWN_CHILD marker** comment listing each block’s 1-based index (position among all pairs — what a human counts in the Plan), repo and quoted offending line, and fabrik:paused. The whole declaration is held — no child is created, including the well-formed siblings: spawning the siblings would shift the DEPENDS_ON indexes and the ADR-1583 fabrik:spawned-child:<index>:<n> markers and leave the parent’s blockedBy set half-populated. The stored Plan comment is immutable, so a bare un-pause re-pauses until the Plan is edited (or a comment gets it revised); the comment says so.
Own-line requirement (enforced by ParseSpawnBlocks): Both markers must stand on their own line, the same convention FABRIK_STAGE_COMPLETE follows. On the BEGIN line the target repo must be the only trailing token and a well-formed owner/repo; the END line must carry nothing else. Leading whitespace is tolerated, so a block nested under a list item still parses — trailing prose after the repo is not. A marker occurring inside prose (for example a backticked mention in the Plan’s own task checklist) is therefore ignored, and cannot consume, truncate, or suppress a genuine block later in the same comment. Until this was enforced, such a mention silently destroyed the real block and the spawn simply did not happen (#1263). When a BEGIN-looking line is present but no well-formed block parses, preImplement now logs that fact rather than returning the same silent “nothing to spawn” result as a Plan that declared no children.
Code path: processItem() → Implement stage dispatch → preImplement() (in engine/spawn.go)
Idempotency guard: If the parent issue has fabrik:children-spawned, preImplement returns immediately (no-op). Manual removal of this label (and closing of children created under the completed spawn) is required to trigger a fresh spawn. This guard runs before the recovery path below and must remain first — recovery must never bypass it, or repeated deferred-retry cycles risk double-spawning.
Missing-Plan-comment recovery (#982): findStageComment(item.Comments, "Plan") returning nil is ambiguous on its own — it means either “Plan genuinely declared nothing to spawn” or “the item snapshot’s Comments is stale” (Comments is a deep field whose cache freshness is updatedAt-keyed; see #957). preImplement disambiguates using the stage:Plan:complete label, which is populated through a different path and can be visible even when Comments lags:
- No
stage:Plan:completelabel — true no-op; Plan hasn’t run (or hasn’t been recorded as complete) yet. Returns(false, nil)with no further action. stage:Plan:completepresent but no Plan comment found — an inconsistent state.preImplementcallsrecoverMissingPlanComment, which:- Logs an actionable line naming the item, stage, and inconsistency — unconditionally, before any of the following steps, so it fires regardless of which outcome below is reached.
- Checks a per-item cooldown (
CooldownAt("spawn-recovery-deferred"), mirroring thedep-blockedcooldown pattern). If active, skips the live re-read and returns(false, errPreImplementDeferred)immediately — bounding repeated GraphQL load to once per cooldown window (PollSeconds*10seconds) during a sustained failure (e.g. #971-style rate-limit pressure). - Otherwise performs a live, uncached
e.client.FetchItemDetailsre-read (the same primitiveverifyAndHealLinkageuses for the closing-issue-linkage deep field) and re-checksfindStageCommentagainst the freshly fetched comments. - Live re-read fails: logs the error, records the cooldown, and returns
(false, errPreImplementDeferred)— the parent is not paused; it is retried on a subsequent poll cycle (the deferred outcome is returned beforeStageAttempted/LastAttemptAtis recorded, so no dispatch cooldown suppresses the retry). - Live re-read succeeds but still finds no Plan comment, or finds one with no spawn blocks: confirms there is genuinely nothing to spawn — returns
(false, nil). - Live re-read recovers a Plan comment with spawn blocks: proceeds to spawn via the same
spawnChildrenhelper the direct path uses (step-for-step identical to the Flow below), guaranteeingfabrik:children-spawnedis applied through a single code path regardless of which path (direct or recovered) triggered it.
A recovered live-read’s
Commentsare used locally for this call only — they are not written back into the cache/store; this keeps the fix scoped to thepreImplementdecision point rather than becoming a general-purpose cache-refresh utility.
spawnChildren is shared by three callers (ADR-1419). preImplement’s direct path, recoverMissingPlanComment’s recovery path, and — since #1419 — the Review/Validate mid-flight hook (§6.7.2) all funnel through the single spawnChildren function documented below. This is what makes a Review/Validate spawn “not a second-class path” relative to a Plan-spawned one: board-servability, assignment, blocked_by linking, and default labeling are identical regardless of which stage originated the spawn, because there is exactly one implementation to drift out of sync.
Flow (when spawn blocks are present and not yet spawned):
- Validate all
DEPENDS_ONheaders upfront, before any GitHub mutation:validateSpawnDependsOnchecks that every declared index is a strictly-forward reference to an earlier block (1 <= DEPENDS_ON < ownIndex) — purely structural, no created-issue data needed. Any invalid value (out-of-range, non-forward/self-referencing, or syntactically malformed — non-numeric, empty, zero, negative) is a hard failure: post an error comment (Created so far: none), addfabrik:paused, and stop — no children are created. A silent drop here would reproduce the exact bugDEPENDS_ONexists to fix. - Board-servability scope-check (ADR-1419, requirement 3). For each unique target repo in the spawn blocks,
spawnTargetServedByThisInstance(childOwner, childRepo)asks: does this instance’s own, already-known config cover this repo? A multi-repo instance (cfg.Repo == "") already legitimately serves any repo the org grants board access to — proceeds unchanged, exactly as before #1419 (this is why “Instance A processes multiple repos on board 5 without issue” in the originating report). Arepo:-scoped instance whose repo does not match the target has no basis to claim that repo is servable here — per the deliberately out-of-scope “no cross-instance board discovery” boundary, the fix cannot locate the correct board, only refuse to guess wrong. It posts an error comment naming the unservable target and this instance’s ownowner/reposcope, addsfabrik:paused, and stops — no children are created for the whole batch (a routing failure aborts the entire spawn, not just the offending block, mirroringDEPENDS_ONvalidation’s all-or-nothing semantics). This is the fix for failure mode 1 (a Plan-spawned cross-repo child silently registered onto the parent’s own board and left unassigned) — it does not relocate the child to the correct board (no data exists to do so), it makes the failure visible instead of silent. - Validate all target repos are clonable: call
ensureSpawnTargetReadyfor each unique target repo that passed the scope-check in step 2. If a repo cannot be bare-cloned, post an error comment, addfabrik:paused, and stop — no children are created. - For each
FABRIK_SPAWN_CHILD_BEGIN/ENDblock in document order (creation pass — unchanged byDEPENDS_ON, except that the block-index → child node-ID mapping is now retained for the sibling-wiring pass below):- Resume-safe retry (ADR-1583). Before the loop,
parseSpawnChildLabels(item.Labels)recovers which 1-based block indices already have afabrik:spawned-child:<blockIndex>:<childNumber>marker from a previous, interrupted attempt at this same batch; if any exist,item.BlockedByis refreshed live first (refreshForSpawnResume, mirroring the missing-Plan-comment recovery’s cooldown-gated live-read pattern below — a live-read failure or active cooldown returns the sameerrPreImplementDeferredoutcome, not a pause). For a block with an existing marker:CreateIssueis skipped — the child’s node ID is instead resolved viaFetchProjectItem(childOwner, childRepo, childNumber), pausing loud (naming the specific marker label to remove) if that recorded child can no longer be found. For a block with no marker, the sequence below runs exactly as it always has. CreateIssue(owner, repo, title, body+footer, assignees)via REST — returns(number, nodeID).assigneesis[cfg.User]in PAT mode (ADR-1419, requirement 4) — under GitHub App auth (#1893) it is the parent’s human assignees instead (spawnAssignees; possibly none), and an HTTP 422 on them is retried once with no assignees rather than aborting the batch. In PAT mode every spawned child, same-repo or cross-repo, is assigned to the user of the instance meant to process it; folded into the same POST as creation rather than a separate mutation, so a bad/misconfigureduser:value fails loud through this single already-fail-loud call rather than opening a second, silent failure point. Immediately after a successfulCreateIssue, the durablefabrik:spawned-child:<blockIndex>:<childNumber>marker is written on the parent (addLabelChecked, best-effort — a write failure is logged, not fatal, and only regresses that one child to pre-ADR-1583 behavior if a later step in the same attempt also fails).AddProjectV2ItemById(board.ProjectID, nodeID)— adds child to the same project board; returnschildItemID. When resuming a marked block, this is skipped in favor of aLookupIssueProjectItemcheck first — if the child is already on the board, its existing item ID (and current Status) is reused instead of callingAddProjectV2ItemByIdagain (unconfirmed against real GitHub whether a repeat call is safe — ADR-1583 checks rather than assumes). A fresh (unmarked) block always takes the direct-call path, unchanged.AddBlockedByIssue(parent.NodeID, nodeID)— links child as ablockedBydependency of the parent. When resuming, the (freshly refreshed)item.BlockedByis checked first; if the edge already exists, this call is skipped for the same unconfirmed-idempotency reason as the board-add above.AddLabelToIssueforfabrik:sub-issueon the child (informational; idempotent add, always attempted)- Inherited labels (#2090, ADR 2090) — applied before the Status placement below.
inheritChildLabelscopies, in order: the parent’sbase:<branch>label (sameowner/repoonly, compared case-insensitively; a cross-repo child gets none because the branch may not exist there, and keeps today’s missing-branch fallback-with-comment behavior), thenfabrik:yoloif the parent has it, thenfabrik:cruiseif the parent has it (any repo, unchanged). When the parent carries severalbase:labels the child gets only the one the parent itself resolves to —baseLabelFor, the same first-non-empty-wins rulebaseBranchForItemuses; a parent with none gives the child none. The label is the parent’s label as written, not anls-remote-verified branch: if the branch is missing from the remote, the child falls back to the default with the usual one-time comment at its own worktree creation. The ordering matters: the Status placement is what makes a child dispatchable, and its first dispatch’sEnsureWorktreereadsbase:, so a label added afterwards could lose that race and fork from the default branch. Thebase:write is fail-closed: if it fails,inheritChildLabelsreturns the error immediately andspawnChildrenpauses the parent (fabrik:paused, the same path as the other per-child failures) and returns an error before the Status placement, so the child is never dispatchable against the wrong base — a child placed withoutbase:would fork from and target the default branch, and under yolo land on it. The child already carries itsfabrik:spawned-child:<i>:<n>marker (resumable origins), so the ADR-1583 retry resumes it and the resume path’s missing-label check writesbase:then; no new retry machinery. A non-resumable origin (Review/Validate mid-flight) writes no marker, so its retry re-spawns as before — which means a failedbase:write there leaves the already-created child (on the board, linked asblockedByof the parent, labelledfabrik:sub-issue, but with no Status and therefore never dispatched) as an orphan that the retry does not reuse: the retry creates a duplicate, and the orphan must be closed by hand, or the parent stays blocked by it. This is the same pre-existing duplicate-on-retry exposure that a failed board-add orblockedBylink has on these origins (ADR-1583), not a new failure class. Thefabrik:yolo/fabrik:cruisewrites stay best-effort (failure logs a warning, spawn continues), since a missed autonomy label only reduces automation. When resuming, the child’s labels from theFetchProjectItemresult already in hand are checked and only the missing ones are written — a second retry makes no label writes — and this runs even for a child with a non-empty Status (its Status is not touched). A label cannot repair a child that already created its worktree from the default base. Because the same code path serves every origin, Plan, Review and Validate spawns inherit identically. Children of a non-default-base parent therefore take the ADR-1096/1097 explicit-close path and join the parent’s base merge-train partition (ADR-1648). UpdateProjectItemStatus(board.ProjectID, childItemID, sf.FieldID, specifyOptionID)— moves child to theSpecifycolumn (or first non-Backlog, non-terminal column as fallback). Non-fatal: if the call fails,e.statusFieldis nil (startup fetch failed), or no viable column exists, the child lands in Backlog, a warning is logged, spawn continues — andrecordChildPlacementFailurewrites the durablefabrik:awaiting-placementmarker on the child so the board-placement settle scan retries it (§6.9, ADR-062). When resuming a block whoseLookupIssueProjectItemcheck (above) found a non-empty Status, this call is skipped entirely rather than re-run — the child may have already progressed under its own pipeline while the parent sat paused, and blindly re-placing it intoSpecifywould clobber that progress (ADR-1583).- On any failure in the fatal steps (CreateIssue, AddProjectV2ItemById, AddBlockedByIssue, the inherited
base:label write, or resolving a marked-but-now-missing child): post error comment naming completed and failed children, addfabrik:pausedto parent, stop without addingfabrik:children-spawned. The comment no longer instructs closing orphaned children (ADR-1583) — it states that already-created children are tracked and will be resumed, not recreated, on the next retry.
- Resume-safe retry (ADR-1583). Before the loop,
- Sibling-wiring pass (second pass, after all children in step 4 exist): for each block that declared
DEPENDS_ON, callAddBlockedByIssue(childNodeIDs[ownIndex], childNodeIDs[dependsOnIndex])— the same primitive used for the parent edge in step 4, now linking two children. Immediately on success, applyBlockedByEdgeAddedto the Store too — keyed on the dependent child’s own(repo, number), not the parent’s, mirroring step 4’s parent-edge write (#1783 follow-up) so a dependent child is not exposed to the same stale-BlockedByrace this ADR closes for the parent. A block may depend on any earlier sibling regardless of where in step 4 its own creation happened to land. On failure: post an error comment (all children created so far are listed), addfabrik:pausedto parent, stop without addingfabrik:children-spawned— the same partial-creation-on-failure shape step 4 already uses, not a new failure mode. - After all children are created, linked to the parent, and all declared sibling edges are wired: best-effort remove every
fabrik:spawned-child:<blockIndex>:<childNumber>marker this call knows about (ADR-1583 — steady state carries none of them, they exist only transiently during an interrupted/in-progress spawn), then addfabrik:children-spawnedto the parent. The idempotency guard is applied for the full two-phase operation (steps 4+5) — after both, not immediately-after-step-4 — specifically so a sibling-wiring failure is retried on the next attempt rather than silently accepted as “done.”
After spawn: spawnChildren returns (spawned, true, nil), where spawned lists each child as owner/repo#N. From preImplement’s call site, processItem returns without invoking Claude — decomposition is a terminal outcome for that dispatch (R1, #1783).
The dispatch gate is a synchronous Store write, not a race against deep-fetch/cycleSet timing (#1783, ADR-1783). Before #1783, the only thing that could stop a later dispatch attempt from running Claude on the full parent was fabrik:blocked, and fabrik:blocked depended on checkDependencies seeing a freshly deep-fetched BlockedBy — which required a LabelsChanged change (from fabrik:children-spawned) to populate cycleSet, which required a subsequent poll’s selectDeepFetchCandidates to admit the item for a real FetchItemDetails call. Under GraphQL rate-limit pressure or a transient deep-fetch failure, that chain could take several poll cycles to complete — or, if the item was admitted to dispatch by some other route in the interim (e.g. a comment) while BlockedBy was still stale-empty, checkDependencies would report “not blocked” and Implement would build the whole parent as a monolith minutes after the children were filed (the #1659 report). Immediately after each child is linked via AddBlockedByIssue, spawnChildren now also applies a BlockedByEdgeAdded mutation (internal/itemstate) directly to the engine’s own Store, appending that one dependency edge with no other deep field touched. Because boardcache.CacheImpl.FetchProjectBoard reconstructs every item’s BlockedBy straight from the Store on every call (snapshotToProjectItem), the very next board fetch — this poll’s own continuation or the next poll, whichever comes first — already carries the correct edges, with zero dependency on cycleSet wake timing, deep-fetch admission, or a live re-fetch succeeding. checkDependencies then sees the new blockedBy edges — both parent edges and any sibling edges — on that board and adds fabrik:blocked, gating the parent until all children close. checkDependencies and PushUnblockObserver require no changes for sibling edges: both already operate generically over any item.BlockedBy entry regardless of whether it originated from the parent-wiring step or the sibling-wiring step. The BlockedByEdgeAdded write is unconditional per child — it fires whether the edge was freshly linked or found already-linked on an ADR-1583 resume — so a partial (paused) spawn’s already-linked children are reflected in the Store even though the batch as a whole did not complete (R5 non-regression).
Partial spawn failure: fabrik:children-spawned is NOT applied if any step fails — including a DEPENDS_ON validation failure (step 1, before any child exists), a board-servability refusal (step 2, before any child exists), or a sibling-wiring failure (step 5, after all children exist but some sibling edges may be unwired). On retry (after user removes fabrik:paused), the caller re-runs spawnChildren from the beginning — but resume-safe as of ADR-1583: any block already recorded via a fabrik:spawned-child:<blockIndex>:<childNumber> marker (step 4, above) is recognized and resumed rather than recreated. Users no longer need to manually close orphaned children before re-advancing for the three per-child steps this covers (CreateIssue, AddProjectV2ItemById, AddBlockedByIssue); the sibling DEPENDS_ON wiring pass (step 5) itself is unchanged — a raw re-call on every retry, deliberately out of this ADR’s scope (its own AddBlockedByIssue calls are expected, but not confirmed, to no-op on a repeat).
Recursive decomposition: A child issue created by preImplement runs the full Fabrik pipeline. If the child’s own Plan emits FABRIK_SPAWN_CHILD_* blocks, the child’s Implement dispatch triggers another preImplement — grandchildren are created identically. There is no depth limit.
References: ADR-048: Engine-Side Pre-Implement Spawn, ADR-1583: Spawn-Children Resume-Safe Retry via Durable Per-Child Markers, ADR-1783: Deterministic Post-Spawn Dependency Gate
6.7.1 Empty-Coordinator Completion
Trigger: A parent issue carries fabrik:children-spawned (§6.7) and its Implement stage completes (FABRIK_STAGE_COMPLETE, err == nil) with zero commits in the worktree ahead of origin/<baseBranch>. This is the case of a fully-delegated coordinator: all of the parent’s actual work was decomposed into spawned children, which have since merged to main, leaving the parent’s own worktree with nothing to commit.
Why this needs a dedicated check. Before this, a completed Implement invocation with create_draft_pr: true unconditionally attempted ensureDraftPR (§5.5). For a pure coordinator, GitHub’s CreateDraftPR call fails with a 422 (“No commits between main and fabrik/issue-N”) because there is genuinely nothing to diff. The failure repeats every retry (the worktree state doesn’t change), so the issue always reaches MaxRetries and lands in escalatePRCreationFailure — paused with fabrik:paused, requiring a manual close even though the work is substantively complete. ADR-048 anticipated a coordinator-only parent emitting FABRIK_NO_WORK_NEEDED itself, but the Implement skill prompt was never taught to detect this case, so in practice Claude just completes normally and falls into the PR-creation attempt.
Detection is commit-count-based, not label-based, and only runs after a completed Claude invocation. Checking fabrik:children-spawned alone, before Claude even runs, would be unsafe: specs/sub-issue-decomposition/spec.md explicitly supports a hybrid parent — one whose Plan both spawns children and describes its own implementation work, which runs normally in Implement after all children close. On the first post-unblock Implement dispatch, “zero commits so far” is true for both a pure coordinator and a hybrid parent that hasn’t started yet — the two are only distinguishable after Claude has had the chance to produce its own commits. Gating on completed (Claude ran to completion, no error) makes the check safe for both: a hybrid parent that does its own work ends up with commits ahead of base and is unaffected; a pure coordinator ends up with zero and is caught.
Implementation — commitsAheadOfBase(workDir, baseBranch string, issueNumber int) (int, error) (engine/item.go): lists git log --name-only origin/<baseBranch>..HEAD in the worktree and counts every commit except those whose only changes are the item’s own persisted spec (specs/<issueNumber>-*/spec.md, isSpecOnlyCommit — ADR 2034, §7.16), so the Specify stage’s engine-written spec commit never makes a fully-delegated coordinator look like it has work of its own. Empty and merge commits (no touched files) still count. Fails safe — any git or parse error returns a non-nil error, never an assumed zero, so callers must treat an error as “unknown” and fall through to the normal PR-creation path rather than short-circuiting to Done.
Two call sites, both folding the result into the existing noWorkNeeded boolean rather than introducing a separate completion path — this lets the fully-built, ADR-060-hardened handleNoWorkNeeded/settleNoWorkNeeded machinery (§6.8) handle everything else (durable marker, idempotent settle, Done move, issue close, retry/escalation) with no new code:
- Primary path —
finalizeStageOutcome(called after every Claude invocation completes): after computingnoWorkNeeded := err == nil && CheckNoWorkNeeded(output), if!noWorkNeeded && completed && stage.Name == "Implement" && hasLabel(item.Labels, "fabrik:children-spawned"), callscommitsAheadOfBase(workDir, baseBranch, item.Number). On success withahead == 0, setsnoWorkNeeded = true— routing the rest offinalizeStageOutcomethrough the samecompleted && noWorkNeededbranch a Claude-emittedFABRIK_NO_WORK_NEEDEDwould take (§6.8), instead of the plaincompletedbranch that callsensureDraftPR. This computation runs beforefinalizeStageOutcome’s own separate eagerensureDraftPRcall — the one gated oncompleted && stage.CreateDraftPR && stage.PostToPRthat exists purely to guarantee a PR exists before posting output to it — so that eager call is itself gated on!noWorkNeededand never fires for the empty-coordinator case either. Without this ordering, an empty coordinator would still trigger a realCreateDraftPR422 from that eager call even though the latercompleted && noWorkNeededbranch correctly skips escalation — the real Implement stage config has bothcreate_draft_pr: trueandpost_to_pr: true, so this eager call is always reachable in production. - R5 self-heal — the skip-Claude retry precheck (
processItem, §5.5): when an item already hasPRCreationFailed[stage.Name] == true(e.g. from a pre-fix retry cycle, or from before this check existed), the same guard runs unconditionally — reaching this branch already proves Claude completed the stage at least once and still produced zero commits, so there is no hybrid-parent ambiguity to protect against here. On a match, the lock is released,StageRetryCleared/EngineUnpausedare applied, andhandleNoWorkNeededis called directly — beforeensureDraftPRis attempted again. This lets an issue already stuck mid-retry-cycle when this fix ships (or a pre-fix issue withfabrik:pausedmanually removed) recover to Done on its next poll instead of continuing to hammerensureDraftPRtoward escalation.
Comment text is distinct from the marker-emitted case. noWorkNeededSkipComment (engine/no_work_needed_settle.go) composes the “skipped: no work needed” comment body: for an Implement stage with fabrik:children-spawned (this path), it posts _Skipped: no work needed — all work was delegated to spawned children and this issue has no commits of its own._ rather than the generic _Skipped: no work needed (FABRIK_NO_WORK_NEEDED emitted by Implement)._ text used everywhere else. Earlier revisions of this fix reused the generic text, but it asserts something false — Claude never emitted FABRIK_NO_WORK_NEEDED here, the engine inferred completion from zero commits — and a durable GitHub comment that misdescribes its own mechanism erodes trust in the record later. No separate completion path was introduced to fix this: hasSkippedComment’s idempotency check derives the same way, so retries remain safe.
A parent with own commits is unaffected. If the coordinator’s Implement invocation produces its own commits (hybrid parent, or a coordinator whose “own work” happens to be a no-op that still commits something), commitsAheadOfBase returns nonzero, noWorkNeeded stays false, and ensureDraftPR runs normally — the guard checks actual commit state, not just the label.
References: ADR-048: Engine-Side Pre-Implement Spawn, handarbeit/fabrik#921
6.7.2 Mid-flight Spawn Recognition (Review/Validate)
Trigger (ADR-1419). FABRIK_SPAWN_CHILD_BEGIN/END recognition is not exclusive to Plan. finalizeStageOutcome (engine/item.go) also scans for well-formed blocks when stage.Name is "Review" or "Validate", immediately after the FABRIK_PR_CREATE marker handling and before output is stripped/posted. This closes failure mode 2 from the originating report: a Review or Validate agent that discovers a blocker mid-flight previously had no sanctioned route to declare it and would call gh issue create directly — the engine never observed that spawn, so none of its wiring ran (no board registration, no assignee, no blocked_by edge), and the parent resumed and could report green on a tree that was, in fact, still missing its blocking dependency.
Why raw output, not a stored comment. Plan’s mechanism (§6.7) is “stage N’s output, read back later as a stored comment, drives stage N+1’s pre-dispatch step” — preImplement reads the most-recent comment literally named "Plan" via findStageComment. Review and Validate are post_to_pr: true stages, so nothing later re-reads their comment for this purpose; instead, ParseSpawnBlocks runs directly against the in-memory output string produced by this dispatch, before postOutputToPR/stripMarkers runs — structurally identical to the existing FABRIK_PR_CREATE_BEGIN/END handling immediately above it in the same function. This sidesteps the post_to_pr complication entirely: it does not matter whether the eventual comment lands on the issue or the PR, because parsing already happened on the pre-posting string.
No new idempotency label needed. output is this dispatch’s own fresh content, never replayed across dispatches — a block is processed exactly once by construction, the same guarantee FABRIK_STAGE_COMPLETE detection already relies on. This is unlike Plan’s fabrik:children-spawned guard, which exists because preImplement re-reads the same stored comment on every Implement dispatch.
Same shared spawnChildren, same wiring — except ADR-1583’s resume marker. The block list parses via the same ParseSpawnBlocks/spawnBeginRepo/isSpawnEndLine own-line-marker discipline documented in §6.7 (the #1263 hardening applies identically — a Review/Validate output that merely mentions the marker in prose cannot misfire) and is passed to e.spawnChildren(ctx, board, item, owner, repo, blocks, false) — the exact function §6.7’s Flow documents, including the board-servability scope-check, mandatory assignment, and blocked_by linking. Nothing about board registration, assignment, or dependency-edge creation is reimplemented for this call site. The final argument (resumable) is false here, unlike preImplement’s and recoverMissingPlanComment’s true: ADR-1583’s fabrik:spawned-child:<blockIndex>:<childNumber> marker assumes blocks reparses identically on every retry, which holds for Plan’s immutable, already-posted comment but not for this call site’s output — a fresh Claude generation on every dispatch, including a redispatch after a failed mid-flight spawn’s fabrik:paused is removed. With resumable=false, this origin never reads, writes, or leaves behind that marker; a retried mid-flight spawn always takes the fresh-CreateIssue path for every block, exactly as it did before ADR-1583 — carrying forward that ADR’s original duplicate-on-retry exposure for this origin specifically, rather than risking the marker resuming an unrelated, already-created child under a same-numbered but different block on a non-deterministic redispatch (caught in review on PR #1708).
On success: formatMidflightSpawnReceiptNote(spawned) — a present-tense sibling of Plan’s formatSpawnReceiptNote (§6.7’s on-thread spawn receipt) — is prepended to output, and stripSpawnBlocks(output) removes the raw FABRIK_SPAWN_CHILD_BEGIN/TITLE:/FABRIK_SPAWN_CHILD_END block(s) that were actually spawned before posting — mirroring the stripMarkers call already made for FABRIK_PR_CREATE/FABRIK_ISSUE_UPDATE immediately above in the same function. This differs from Plan’s own comment, which deliberately leaves its raw block visible (formatSpawnReceiptNote’s “declared above” wording depends on it staying there); the mid-flight note is self-contained and already names what was spawned, so leaving the internal marker syntax in a human-facing PR/issue comment would only duplicate that information verbatim (a pruefer review finding on the originating PR). stripSpawnBlocks re-uses ParseSpawnBlocks’s own line-scanning (spawnBeginRepo/isSpawnEndLine/parseTitleAndBody), so it removes exactly the blocks that were parsed and spawned. A malformed block never reaches it (#2091): finalizeStageOutcome parses with ParseSpawnBlocksDetailed first, and on any SpawnBlockError runs preserveWorkOnMarkerFailure (commit + push, as for FABRIK_PR_CREATE), pauseForMalformedSpawn (comment naming the block index and quoted line, fabrik:paused), releases the lock and returns — no spawnChildren, no stage completion; a retry re-runs the stage and regenerates the declaration, so there is no re-pause loop (unlike Plan). The leave-visible branch of stripSpawnBlocks is defensive only. It correctly handles a stage output declaring more than one blocker in the same dispatch. The stage’s own narrative output (and, when present, FABRIK_STAGE_COMPLETE) is otherwise untouched and posted normally — a mid-flight spawn does not, by itself, suppress or force the stage’s completion decision.
On failure: spawnChildren has already posted its own fail-loud pause comment and added fabrik:paused (the same path §6.7’s Flow documents for every fatal step). finalizeStageOutcome releases the dispatch lock and returns immediately — this dispatch’s own stage-complete comment/label is suppressed entirely, mirroring the FABRIK_PR_CREATE failure precedent immediately above it in the same function.
Validate dependency guard: live re-check before any landing decision. attemptMergeOnValidate (engine/stages.go) is reached from handleStageComplete synchronously, in the same dispatch that just ran finalizeStageOutcome’s mid-flight spawn hook above — before checkDependencies is ever called for this dispatch through the normal per-item admission path (checkDependencies only gates the shouldAdvance branch, which a successful auto-merge/direct-merge short-circuits to false, skipping it entirely). Left unguarded, a Validate output that both declares a genuine blocker and emits FABRIK_STAGE_COMPLETE in the same turn could have its PR merged before the freshly-created blocked_by edge was ever consulted — the exact same-turn race the mid-flight hook would otherwise introduce. attemptMergeOnValidate closes this itself: immediately after the fabrik:auto-merge-enabled idempotency check and before guard 1’s review-thread check, it re-fetches the item live (FetchItemDetails, the same pattern already used by the direct-merge fallback’s own live re-check further down this function) and calls checkDependencies against the fresh snapshot. A dependency created moments earlier in this exact dispatch is caught: checkDependencies applies fabrik:blocked and posts the usual blocked comment, and attemptMergeOnValidate returns (false, false, nil) without ever calling EnablePullRequestAutoMerge/MergePR/EnqueuePullRequest. Both callers of attemptMergeOnValidate (handleStageComplete and poll.go’s catch-up loop) already gate on yolo and return early once fabrik:auto-merge-enabled is present, so this extra live fetch is bounded to the same narrow per-Validate-completion window guard 1 already re-fetches live data in — it is not a new per-poll cost for items already converging toward merge. This is a genuine engine-side hard stop, not prompt-level mitigation alone; plugin/fabrik-workflows/skills/validate/SKILL.md’s “If You Discover a Blocking Issue” section still instructs the agent not to emit FABRIK_STAGE_COMPLETE alongside a genuine spawn (consistent with Validate’s own “Decision: Complete or Block” criteria), but the engine no longer depends on the agent following that instruction to avoid a premature merge. Review carries no equivalent hazard in the first place: checkDependencies runs (and correctly gates) at the next stage’s dispatch off a non-Validate completion, so completing Review while also spawning a blocker was always safe.
References: ADR-048: Engine-Side Pre-Implement Spawn, ADR-1419: Cross-repo spawn board-servability and mid-flight spawn recognition
6.8 No Work Needed Path
Trigger: Claude outputs both FABRIK_STAGE_COMPLETE and FABRIK_NO_WORK_NEEDED in the same invocation output. Expected most commonly from the Plan stage when Research findings show no code or documentation changes are needed.
Key invariant: FABRIK_NO_WORK_NEEDED alone has no effect. Both markers must co-occur. This keeps Claude honest — the emitting stage must declare itself complete before the bypass fires.
Marker priority: This path fires in the completed && noWorkNeeded branch, which precedes the plain completed branch in processItem()’s dispatch chain. This ensures PR creation (ensureDraftPR) and markPRReady — both in the plain completed branch — are not called.
Mutual exclusivity: FABRIK_NO_WORK_NEEDED is mutually exclusive with FABRIK_BLOCKED_ON_INPUT (structurally: blockedOnInput is only reached when completed is false).
Durable marker (fabrik:awaiting-done), written before anything else. The instant processItem decides completed && noWorkNeeded, handleNoWorkNeeded writes fabrik:awaiting-done as its very first mutation — before clearing any orphaned fabrik:awaiting-input, before the emitting stage’s stage:<X>:complete label, before anything else (idempotent — a no-op if already present, e.g. on a retried invocation). This closes a real failure window: handleNoWorkNeeded’s original form made upward of ten sequential GraphQL calls, any of which could be the one that first hits an exhausted rate limit (the observed real-world trigger — #981). Writing the marker unconditionally first is the only placement that survives every plausible failure point, including a full engine restart — see ADR-060 for why this must be a durable GitHub label rather than an itemstate.Store mutation (which does not persist across restarts).
Dispatch suppression, independent of board column. While fabrik:awaiting-done is present, itemMayNeedWork and itemNeedsWork (engine/item.go) return false for every non-cleanup stage — checked immediately after each function’s stage == nil guard, ahead of the HoldingStage check — regardless of item.Status. The item may still be sitting in whichever column the emitting stage ran in, because the board move to Done is exactly the step that may be outstanding. Cleanup stages (CleanupWorktree: true) are exempt so Done’s own worktree cleanup can still run once the marker is eventually cleared.
Code path: processItem() → handleNoWorkNeeded() (writes the marker, then delegates) → settleNoWorkNeeded() (idempotent; does the rest of the work) — and, on retry, the no-work-needed settle scan (settleNoWorkNeededScan, engine/poll_settle.go) calls settleNoWorkNeeded() directly.
handleNoWorkNeeded flow:
- If
fabrik:awaiting-doneis not already present, add it. - Delegate to
settleNoWorkNeeded.
settleNoWorkNeeded flow — idempotent; safe to call repeatedly, every sub-step checks current state before mutating:
- Only if
item.Status != "Done"(see below for why this guards the whole block): clear any orphanedfabrik:awaiting-inputlabel (same rationale ashandleStageComplete). - Add
stage:<emitting>:completeif not already present (hasLabelcheck) — sameitem.Status != "Done"guard. - Find
doneOrder: iteratee.cfg.Stagesfor the stage withCleanupWorktree == true; fall back tomath.MaxIntif none exists. - For each stage with
Order > emitting.Order && Order < doneOrder:- Add
stage:<name>:completeif not already present (checked independently per stage). - Post the one-line “skipped” comment (
_Skipped: no work needed (FABRIK_NO_WORK_NEEDED emitted by <emitting stage>)._, no rocket reaction) — but only once per decision, not once per stage. Every skip comment for a given decision carries identical text (it names the emitting stage, not the individual skipped stage — unchanged from the original behavior), sohasSkippedComment(item, emittingStage.Name)is checked once before the loop: if any skip comment already exists, the full comment set is assumed already posted.
- Add
- If any label/comment sub-step failed in this pass, stop here — do not attempt the status move or the close. Record a retry (below) and return; the next settle pass re-attempts only whatever is still missing.
- Move the issue to Done via
UpdateProjectItemStatus, unlessitem.Statusis already"Done"(a prior pass already succeeded). On failure, record a retry and return — no PR is created, andCloseIssueis not attempted, preserving the pre-existing invariant (TestHandleNoWorkNeeded_CloseIssueNotCalledOnStatusFailure) that the issue is never closed before the status move succeeds. - Close the issue via
CloseIssue, unlessitem.IsClosedis already true. On failure, record a retry and return. - Once both the status move and close have succeeded (or were already true), clear the durable marker: remove
fabrik:awaiting-doneand reset the retry counter.
Why steps 1–4 are skipped once item.Status == "Done": the Done move (step 6) is only ever attempted after every label/comment sub-step in steps 1–4 has already succeeded in that same pass (step 5 gates it), so item.Status == "Done" is itself proof that steps 1–4 already landed — there is nothing left for them to do. This isn’t just an optimization: skipping them also sidesteps a stage-resolution hazard that would otherwise corrupt state on a retry. The settle scan (below) re-derives stage from item.Status, and once the board has moved to "Done", that resolves to the cleanup stage itself, not the original emitting stage. Running steps 1–4 with that wrong stage would add a spurious stage:Done:complete label — before the real Done-stage worktree cleanup has ever run for the item — which itemNeedsWork’s CleanupWorktree branch (§1.1) reads as “cleanup already complete,” permanently skipping normal cleanup dispatch and leaving the worktree janitor (§8) as the only remaining path to reap it.
Retry-owner: the no-work-needed settle scan, settleNoWorkNeededScan (engine/poll_settle.go; called from poll() in engine/poll.go). Runs unconditionally once per poll, immediately after runValidatePRTerminalAdvance (ADR-057) — the closest existing structural analog, since it too must act independent of item.Status/current column and idempotently fill in whatever a prior partial pass left missing. Sourced directly from board.Items, not deepFetchCandidates (see #1220): itemMayNeedWork returns false for exactly the items this scan exists to retry — a non-cleanup-stage item carrying fabrik:awaiting-done — so a candidate list built through that filter would drop the item before the scan ever saw it. This is the same precedent already established by settleMergeTrainMemberCloses, settleNonDefaultBaseCloses, and settleChildPlacements (§6.9): retry eligibility and dispatch eligibility are different questions, and itemMayNeedWork/itemNeedsWork answer only the latter. For every item in board.Items carrying fabrik:awaiting-done and not fabrik:paused (an already-escalated or independently-paused item is left alone — see below): resolve stage := stages.FindStage(e.cfg.Stages, item.Status). If no stage matches the current column (e.g. the board column was renamed away), record a retry without calling settleNoWorkNeeded — there is no safe stage to derive the skip-list from. Otherwise, unless item.Status == "Done", perform a targeted inline FetchItemDetails call and write the result through via itemstate.ItemDeepFetched — this item was never deep-fetched by the normal per-poll pipeline (that’s precisely why it’s stranded), and settleNoWorkNeeded’s hasSkippedComment idempotency check (below) needs item.Comments populated to avoid re-posting the “skipped” comment set on every retry pass; a failed fetch records a retry (via itemstate.DeepFetchFailed) and skips this item for the pass without calling settleNoWorkNeeded. The fetch is skipped once item.Status == "Done", since hasSkippedComment is never consulted in that case (only the close-issue retry remains) and the extra GraphQL call would be wasted on the already-working “Done move succeeded, only CloseIssue failed” sub-case. Finally call settleNoWorkNeeded(board, item, stage) — note that once item.Status == "Done", the resolved stage here is the cleanup stage, not the emitting stage, but per the previous paragraph settleNoWorkNeeded no longer uses it for anything in that case.
Retry counting and escalation. Every failed settle pass calls recordNoWorkNeededRetry, which increments the existing itemstate.StageRetryIncremented/Attempts counter — but keyed by the dedicated constant "__no_work_needed__", not the emitting stage’s real name (the emitting stage’s own counter is cleared, not reused, immediately before handleNoWorkNeeded is called, so reusing it would be safe but conflates two different failure semantics; the constant is also unrepresentable as a real YAML stage name:, so it can never collide). Once Attempts("__no_work_needed__") >= e.cfg.MaxRetries, escalateNoWorkNeededFailure fires — mirroring escalatePRCreationFailure/escalateFailedStage (§5.5, §7.2): adds fabrik:paused, removes fabrik:awaiting-done (dispatch is already suppressed by the pre-existing paused-item gate once fabrik:paused is present, so the marker is no longer needed, and removing it prevents the settle scan from fighting an operator investigating the issue), posts an explanatory comment naming the failure and giving manual recovery steps (gh issue close <N> --repo <owner>/<repo> plus a note that the board column must be moved to Done by hand — there is no simple gh one-liner for Projects-v2 status fields, unlike the gh pr create workaround in §5.5), and applies itemstate.EnginePaused.
Marker clearing. fabrik:awaiting-done is removed in exactly two places: clearNoWorkNeededMarker (a fully successful settle pass — the normal, expected path) and escalateNoWorkNeededFailure (giving up after MaxRetries — the issue is paused for human intervention instead). It is never removed by cleanupClosedIssueTransientLabels’s closed-issue defensive sweep (§1.4), unlike most other gate labels: if a no-work-needed issue is closed out-of-band (e.g. manually) while retries are still below MaxRetries, stripping the marker prematurely would silently resurrect the exact bug this mechanism exists to prevent. The settle scan is itself closed-issue-aware (its item.IsClosed check in step 7 above skips the now-redundant CloseIssue call and still completes the status move), so it is the sole self-healing path for this label — not the sweep.
State transitions:
| Before | Trigger | After | Labels Added | Labels Removed |
|---|---|---|---|---|
Column <X>, Locked + In Progress |
FABRIK_STAGE_COMPLETE + FABRIK_NO_WORK_NEEDED |
Column <X>, Awaiting Done (marker present; board move outstanding) |
fabrik:awaiting-done, stage:<X>:complete, stage:<Y>:complete … (all subsequent non-cleanup stages) |
fabrik:locked:<user>, stage:<X>:in_progress |
Column <X>, Awaiting Done |
Settle pass succeeds (status move + close) | Done, Pending Cleanup | — | fabrik:awaiting-done |
Column <X>, Awaiting Done |
Settle pass fails Attempts >= MaxRetries times |
Column <X>, Paused |
fabrik:paused |
fabrik:awaiting-done |
References: ADR-045: No Work Needed Marker, ADR-060: Durable No-Work-Needed Marker
6.9 Child Board-Placement Retry
Trigger: spawnChildren (§6.7) creates a child issue, adds it to the project board, and links it as blockedBy of the parent — all of which must succeed, or the parent itself is paused and the whole spawn aborts (§6.7’s fatal branches). Only the final per-child step, moving the new board item’s Status to Specify (or the first non-Backlog, non-terminal column) via UpdateProjectItemStatus, can fail without aborting the spawn: the child, its board item, and its blockedBy link already exist by the time this call is attempted, so a failure here is recoverable in place rather than a reason to tear down what has already been created.
How this compares to §6.8’s settle scan. fabrik:awaiting-done (§6.8) is written on an item that is still sitting in the column of the real, matched stage that emitted the decision — stages.FindStage always resolves a non-nil stage for it. A child whose placement failed is different in kind: by construction it is sitting in whichever column GitHub defaulted it to when placement failed — typically Backlog, a column with no configured stage at all, so stages.FindStage returns nil for it. Both cases nonetheless fail itemMayNeedWork/itemNeedsWork and are absent from deepFetchCandidates — one at the stage == nil check (before either function even inspects labels, engine/item.go), the other at the explicit fabrik:awaiting-done label check a few lines later — so neither settle scan can be sourced from deepFetchCandidates. Both are, accordingly, sourced directly from board.Items (§6.8 was fixed to match this scan’s shape in #1220, having originally and incorrectly assumed it was already deepFetchCandidates-reachable). The one remaining difference: this scan needs no deep fetch at all — the shallow board query already includes everything an ordinary retry pass needs (ItemID, Status, Repo, Number, Labels) — whereas §6.8’s scan performs a conditional, per-item deep fetch to populate item.Comments for its comment-based idempotency check, which this scan has no equivalent of. itemMayNeedWork/itemNeedsWork are not modified by either fix — this marker’s retry work is independent of stage dispatch, so routing it through the dispatch pre-filter would solve an already-solved problem the wrong way.
Durable marker (fabrik:awaiting-placement), written on the child. recordChildPlacementFailure adds this label to the child issue (not the parent) at the UpdateProjectItemStatus call site in spawnChildren, covering all three failure branches: the call itself erroring, e.statusField being nil (status-field metadata never populated), and no suitable status option existing on the board (no exact "Specify" match and no non-Backlog, non-terminal fallback column). As with fabrik:awaiting-done (ADR-060), this must be a durable GitHub label rather than an in-memory itemstate.Store mutation: there is no idempotent artifact to safely redo on restart — a lost marker would silently resurrect the stall, exactly the failure mode this mechanism exists to prevent.
Code path: preImplement() → spawnChildren() (writes the marker on failure) — and, independently, the child board-placement settle scan (settleChildPlacements, engine/poll_settle.go) calls settleChildPlacement() every poll for any item carrying the marker.
Settle-scan flow (settleChildPlacements, engine/poll_settle.go; called from poll() in engine/poll.go, immediately after cleanupClosedIssueTransientLabels):
- Iterate
board.Itemsdirectly. Skip any item withoutfabrik:awaiting-placement, and skip any item also carryingfabrik:paused(either already escalated — marker already removed, so this is defense-in-depth — or an operator is independently investigating; the scan must not fight them). - Closed-child short-circuit: if
item.IsClosed, callclearChildPlacementMarkerdirectly and move on — no placement attempt, no escalation, no comment. A closed child needs no further board dispatch; the only purpose of correct placement was to let the pipeline process it, which no longer applies once the child is closed (manually, or resolved out-of-band). This is a departure from §6.8’s precedent:settleNoWorkNeededtreats issue-closed as a step to still complete (the Done move remains required work even on a closed issue), but this marker’s precondition (the child needs dispatching) is simply moot once closed. - Otherwise call
settleChildPlacement(board, item).
settleChildPlacement flow:
- Snapshot
e.statusFieldundere.mu(mirrorsspawnChildren’s own snapshot pattern) and callresolveSpecifyOptionID. - If no option resolves (nil
statusField, or no suitable column), record a retry and return — no API call attempted.e.statusFieldself-heals with no special-casing: it is re-populated whenevernilon any poll (existingpoll.gostartup/refresh behavior), so this failure branch requires no extra logic beyond “try again next pass.” - Otherwise call
UpdateProjectItemStatus(board.ProjectID, item.ItemID, sf.FieldID, optionID). On failure, record a retry and return. - On success, clear the durable marker and the retry counter.
No idempotency short-circuit (e.g., checking item.Status before calling UpdateProjectItemStatus) is performed — the call is idempotent, and once the marker is cleared the scan simply stops revisiting the item, so a pre-check would be an optimization without a correctness benefit.
Retry counting and escalation. Every failed settle pass calls recordChildPlacementRetry, which increments the existing itemstate.StageRetryIncremented/Attempts counter — keyed by the dedicated constant "__child_placement__" (mirroring "__no_work_needed__"’s double-underscore convention: unrepresentable as a real YAML stage name:, so it can never collide with a configured stage’s own counter, nor with noWorkNeededRetryStage). Once Attempts("__child_placement__") >= e.cfg.MaxRetries, escalateChildPlacementFailure fires:
- Adds
fabrik:pausedto the child. - Removes
fabrik:awaiting-placementfrom the child. - Posts an explanatory comment on the child naming the failure and manual recovery steps (move the board column to
Specify/first-processing by hand, then removefabrik:paused; notes that a board-configuration problem — no suitable column — may be the underlying cause). - Applies
itemstate.EnginePausedkeyed by"__child_placement__". - Calls
notifyParentOfStalledChild— a best-effort attempt to also comment on the parent issue (see below). This step runs last and its outcome never affects steps 1–4, which have already completed.
Parent notification and its link-recovery mechanism. Unlike fabrik:awaiting-done (which escalates on the same issue it’s already operating on), this marker’s escalation must also reach a different issue — the parent — since the parent has no other visibility into why it remains blocked (via blockedBy) on a child that will never close. There is no structured child→parent link: the only edge is the free-text back-reference childFooter writes into the child’s Body at spawn time ("...Spawned by Fabrik from parent issue owner/repo#N..."). notifyParentOfStalledChild therefore:
- Lazily deep-fetches the child’s
Bodyviae.readClient.FetchItemDetails— this is the only place in the child board-placement retry path that needs a deep fetch; every ordinary settle pass above uses only shallowboard.Itemsfields. - Regex-parses the parent’s
owner/repo#numberout of the footer text viaparseParentFromChildBody. - If the deep-fetch fails, or the footer cannot be found (e.g. a human edited the child’s body and removed it), logs a warning and returns — silently, without posting anything.
- Otherwise posts a best-effort comment on the parent naming the stalled child and pointing at the child’s own escalation comment for recovery steps. A failure to post this comment is logged and swallowed.
This is a deliberately fragile-by-design, best-effort mechanism: a parameterized label carrying the parent link verbatim (set alongside fabrik:sub-issue at spawn time) would be more robust against body edits and cheaper (no deep-fetch needed at escalation), but the spec frames parent notification as best-effort only, and adding a new label kind for a once-per-child, escalation-only lookup was judged disproportionate (ADR-062).
Marker clearing. fabrik:awaiting-placement is removed in exactly two places: clearChildPlacementMarker (a successful settle pass, or the closed-child short-circuit) and escalateChildPlacementFailure (giving up after MaxRetries — the child is paused for human intervention instead). It is never removed by cleanupClosedIssueTransientLabels’s closed-issue defensive sweep (§1.4) — for the same reason as fabrik:awaiting-done: if the sweep stripped the marker before the settle scan’s own closed-child short-circuit ran, a closed child mid-retry (below MaxRetries) would silently lose the marker without ever going through the deliberate, observable clear path. In practice this is a narrow window (the settle scan’s closed-child branch runs every poll, same as the sweep), but the exclusion keeps the marker’s removal paths exhaustively enumerable, matching §6.8’s own rationale.
No change to spawnChildren’s own idempotency guard from this section’s fix. fabrik:children-spawned continues to be added to the parent unconditionally once all children are created, added to the board, and linked — regardless of each individual child’s board-placement outcome. This section only makes the child’s own placement durable/recoverable; it does not make the parent’s guard conditional on it (per the issue’s explicit scope). (A separate mechanism, ADR-1583/§6.7, does change what a retried spawn does before this guard is ever reached — see “Resume-safe retry” in §6.7’s Flow step 4 — but that fix is orthogonal to board-placement recoverability, which this section covers.)
State transitions:
| Before | Trigger | After | Labels Added | Labels Removed |
|---|---|---|---|---|
(child created, board item added, blockedBy linked) |
UpdateProjectItemStatus fails (any of 3 branches) |
Column Backlog (or wherever GitHub defaulted it), Awaiting Placement |
fabrik:awaiting-placement |
— |
Column <X>, Awaiting Placement |
Settle pass succeeds | Column Specify (or first processing stage) |
— | fabrik:awaiting-placement |
Column <X>, Awaiting Placement |
Child observed closed | Column <X> (unchanged), closed |
— | fabrik:awaiting-placement |
Column <X>, Awaiting Placement |
Settle pass fails Attempts >= MaxRetries times |
Column <X>, Paused |
fabrik:paused |
fabrik:awaiting-placement |
References: ADR-060: Durable No-Work-Needed Marker, ADR-062: Child Board-Placement Retry Marker
6.10 Merge-Train Singleton Member-Issue Close Retry
Trigger: landSingleton (the merge-train one-at-a-time landing fallback, §1.3) calls e.client.CloseIssue for the member issue as its last action, after the PR merge, the Queued→Done board move, and the member-PR close have already succeeded. This call fails (e.g. rate limit, transient API error).
Key difference from §6.8’s fabrik:awaiting-done: that marker is written before a chain of up to ten sequential calls, because any of them could be the one that first hits a rate limit. Here there is exactly one at-risk call, positioned after every other side effect of landSingleton has already landed — so the marker is written only in the failure branch, not unconditionally beforehand (ADR-061).
Durable marker (fabrik:awaiting-member-close), written only on failure. markMergeTrainMemberCloseOutstanding adds the label (idempotently — a no-op if already present) only inside the if closeErr != nil branch of the member-issue close. landSingleton’s two calls immediately following the close — resetEjectionCount and resetTrialCounter — run unconditionally regardless of the close’s outcome, exactly as before this fix; the retry mechanism lives entirely outside landSingleton’s call frame and never re-enters it.
No dispatch-suppression wiring — deliberately. Unlike fabrik:awaiting-done, this marker is not checked by itemMayNeedWork/itemNeedsWork, and is not added to transientLifecycleLabels. By construction, the item has already reached its terminal singleton-landing outcome (Status == "Done" in the common case; Status == "<HoldingStage>"/Queued in the rarer case where the Done-move itself also failed) by the time this marker can exist — there is no per-stage redispatch this marker needs to prevent. HoldingStage items are never individually dispatched by itemMayNeedWork/itemNeedsWork regardless (“batch-scoped, not per-item”), and Done-column items are gated by the ordinary stage:Done:complete check in itemNeedsWork’s CleanupWorktree branch, untouched by this marker. This sidesteps the terminal-skip/transientLifecycleLabels interaction that a naive verbatim port of §6.8’s shape would have hit (see ADR-061 Context): once stage:Done:complete lands — which happens independently of this marker, on essentially the same poll — isTerminalPredicate (#689) would otherwise permanently skip the item from all further polling unless the marker were added to transientLifecycleLabels. Not depending on deepFetchCandidates/itemMayNeedWork at all avoids that dependency entirely.
Code path: landOneAtATime → landSingleton (writes the marker on close failure) — and, on retry, settleMergeTrainMemberCloses → settleMergeTrainMemberClose (both engine/merge_train_member_close_settle.go; settleMergeTrainMemberCloses is called from poll() in poll.go) directly.
Retry-owner: settleMergeTrainMemberCloses (engine/merge_train_member_close_settle.go; called from poll() in poll.go). Runs unconditionally once per poll, immediately after handleMergeTrainBatch — independent of merge_train: on/off, so a marker written while merge-train was enabled keeps draining even if the setting is later turned off. It iterates the raw board.Items (not deepFetchCandidates — this scan has no dependency on the deep-fetch/terminal-skip machinery at all). For every item carrying fabrik:awaiting-member-close and not fabrik:paused (mirroring the no-work-needed settle scan’s own paused-item guard — an operator investigating a paused item must not be fought), it calls settleMergeTrainMemberClose.
settleMergeTrainMemberClose flow — idempotent; safe to call repeatedly:
- If
item.IsClosedis already true (the member PR’s ownCloses #N, the landing PR’sCloses #N, or a prior settle pass already closed it — or GitHub’s own auto-close finally landed), skip the redundantCloseIssuecall and go straight to clearing the marker. - Otherwise call
CloseIssue. On success, clear the marker. On failure, record a retry and stop — the next settle pass re-attempts.
Retry counting and escalation. Every failed settle pass calls recordMergeTrainMemberCloseRetry, which increments the existing itemstate.StageRetryIncremented/Attempts counter — keyed by the dedicated constant "__merge_train_member_close__" (same double-underscore-wrapped, YAML-unrepresentable shape as "__no_work_needed__", so it can never collide with a configured stage’s own counter). MaxRetries <= 0 means unlimited retries, never escalate (same guard as recordNoWorkNeededRetry). Once Attempts("__merge_train_member_close__") >= e.cfg.MaxRetries, escalateMergeTrainMemberCloseFailure fires — mirroring escalateNoWorkNeededFailure: adds fabrik:paused, removes fabrik:awaiting-member-close, posts an explanatory comment with the manual recovery step (gh issue close <N> --repo <owner>/<repo>), and applies itemstate.EnginePaused.
Marker clearing. fabrik:awaiting-member-close is removed in exactly two places: clearMergeTrainMemberCloseMarker (a fully successful settle pass, or an already-closed issue found on a later pass — the normal, expected path) and escalateMergeTrainMemberCloseFailure (giving up after MaxRetries). It is not referenced by cleanupClosedIssueTransientLabels’s defensive sweep at all — unlike fabrik:awaiting-done, there is no “silently resurrect the bug” risk from an early strip, since the settle scan’s own item.IsClosed check already treats a closed issue as fully settled; the label is simply informational once the issue is closed, and clearMergeTrainMemberCloseMarker removes it on the very next poll regardless.
Scope: this mechanism covers only landSingleton’s member-issue close. landMergeTrainBatch’s structurally identical member-issue close (§1.3, FR-3 step d) is a deliberately deferred follow-up (ADR-061 §Sibling Audit) — the settle/escalate helpers are written generic over gh.ProjectItem so that follow-up can reuse mergeTrainAwaitingMemberCloseLabel/settleMergeTrainMemberCloses directly rather than duplicating them.
State transitions:
| Before | Trigger | After | Labels Added | Labels Removed |
|---|---|---|---|---|
| Done (or Queued, if the Done-move itself also failed) | landSingleton’s member-issue CloseIssue fails |
Same column, awaiting member close (marker present) | fabrik:awaiting-member-close |
— |
| Same column, awaiting member close | Settle pass succeeds (issue closed, or already closed) | Same column, settled | — | fabrik:awaiting-member-close |
| Same column, awaiting member close | Settle pass fails Attempts >= MaxRetries times |
Same column, Paused | fabrik:paused |
fabrik:awaiting-member-close |
References: ADR-060: Durable No-Work-Needed Marker, ADR-061: Merge-Train Singleton Member-Issue Close Retry
6.11 Closed-Item-At-Any-Stage Advance To Done
Trigger: An issue is closed (manually, or via GitHub’s own Closes #N auto-close on PR merge) while its board item sits at any column other than Done — Specify, Plan, Implement, Review, Backlog, a Holding stage (e.g. Queued), any other non-terminal configured column, or a column with no matching stage config at all (Backlog, or any custom/extra column). itemMayNeedWork/itemNeedsWork (engine/item.go:118/:195) admit a closed item only if the current stage is a cleanup stage or carries stage:<stage>:complete (§2.8, ADR-1387) — a closed item at Validate lacking stage:Validate:complete is, like any other incomplete non-cleanup stage, deliberately not admitted, but unlike every other such stage (which §6.11 below handles), Validate is the domain of §6.15’s settleClosedValidateAdvance instead — excluded from this scan (stage.Name == "Validate", step 3 of the Flow below — narrowed from the broader stageIsGateChecked(stage) in an ADR-1387 follow-up, see below) specifically so the two settle-owners never race. A closed item at any other stage matches neither admission condition and is dropped by the admission guard — it is never dispatched again, so its worktree is never reaped and it never gets archived. This includes items closed while sitting in Queued: the merge train’s own landing paths (landSingleton/landMergeTrainBatch) advance a batch member to Done as part of landing, but an item can reach Queued and then be closed some other way (a human gh pr merge outside the train, or — before issue #1072’s NextStage fix — the generic terminal-advance leak) with no train worker ever picking it up.
Why this can’t reuse §6.10’s (runValidatePRTerminalAdvance) shape verbatim. runValidatePRTerminalAdvance (ADR-057) already handles this exact transition, but is deliberately scoped to stage.Name == "Validate" only (fills gate-checked completion labels in ascending Order, clears gate labels, calls advanceToNextStage). A closed item at Specify/Plan/Implement/Review never reaches that function’s per-item loop at all — it iterates deepFetchCandidates, and a stranded closed item at a non-admitted stage never reaches deepFetchCandidates in the first place, for the identical reason §6.9’s child-placement item never does. Widening runValidatePRTerminalAdvance itself to non-Validate stages was rejected (ADR-064): its completion-label-filling logic exists specifically because Validate is gate-checked and downstream tooling depends on the gate-checked stages’ stage:X:complete labels being present; a closed item at an ordinary, non-gate-checked stage has no such labels to fill and no such downstream dependency — it only needs to reach Done so the existing cleanup machinery there takes over.
Why no durable marker is needed here — unlike every sibling in §6.7–6.10. Every other settle-owner in this section keys on a GitHub label because the failure it recovers from happens mid-sequence (a spawn step, a close call, a Done-move) and the marker records “this specific side effect is still outstanding” so a restart doesn’t lose the decision. This scan has almost no such sequence to protect: item.IsClosed && !(stage != nil && (stage.CleanupWorktree || stage.Name == "Validate")) — plus one additional in-memory (not durable) check for Holding stages, below — is otherwise a pure function of durable, re-derivable board state (the issue’s closed status and its current column), so nothing is lost on an engine restart; the predicate re-evaluates identically from board.Items on every poll. The predicate itself doubles as the idempotency check: once the item reaches the cleanup column, its resolved stage has CleanupWorktree true and the scan naturally stops touching it — no separate item.Status != "Done" comparison, and no marker to write or clear. Note the guard is intentionally stage != nil && (...), not stage == nil || (...): a column with no matching stage config (Backlog, or any custom/extra column) has nothing to protect it from advancing — it has no worktree and no stage bookkeeping — so it is treated the same as any other eligible column, not skipped.
Holding stages are a conditional exception, not a blanket skip (issue #1072). A closed item resolved to a HoldingStage (e.g. Queued) is skipped only while mergeTrainWorkerActive(repoKey) reports a merge-train worker currently in flight for that item’s owner/repo. mergeTrainWorkerActive reads Store.RepoWorkerActive(repoKey) (issue #1222) — the same repo-scoped in-memory liveness registry inside itemstate.Store that the auto-upgrade idle guard reads via Store.HasInFlightWorker() (§9.2), populated by dispatchMergeTrainWorker’s Store.EnterRepoWorker call and cleared by finishTrain’s Store.ExitRepoWorker call on worker exit (ADR-067) — in lockstep with the older Engine.mergeTrainInFlight sync.Map, which dispatchMergeTrainWorker/finishTrain still populate/clear for the atomic duplicate-launch claim, but which no consumer reads for a liveness answer anymore. This is the one real race a Holding-stage item poses that no other stage does: it can be closed-without-merging while still a live batch member mid-assembly or mid-bisection, and yanking it to Done out from under the worker would corrupt that in-flight batch. The check is deliberately in-memory rather than a durable label — the Store’s repo-worker registry only ever reflects “is a goroutine for this repo running right now,” which by construction cannot survive (or need to survive) a restart: a restarted engine has no in-flight worker for any repo, so every closed Holding-stage item is immediately eligible again, which is exactly correct. Once no worker owns the repo, item.IsClosed alone is sufficient — the same accepted predicate as any other stage (ADR-064) — with no PR-merge re-confirmation, since a closed issue at a non-terminal column is itself the durable, sufficient signal regardless of column. See ADR-1072 and ADR-1222.
Deliberately label-state-agnostic. Unlike every other settle-owner, this scan is not conditioned on fabrik:paused, fabrik:awaiting-input, fabrik:blocked, or any other in-flight label. A closed issue at a non-terminal column is itself a terminal state that supersedes any gate/lock label — no further pipeline work can occur on a closed issue sitting outside Done regardless of what else is set on it, so there is nothing for those labels to protect against here (ADR-064).
Code path: the settle scan, settleClosedItemsToDone (engine/closed_item_advance_settle.go), called unconditionally every poll from poll.go, immediately after the child board-placement settle scan (§6.9). Auto-archiving of Done items is covered separately by §6.12.
Flow:
- Resolve the cleanup (Done) stage via
cleanupStage(e.cfg)— the lowest-Orderstage withCleanupWorktree: true, mirroringholdingStage’s shape but by explicit min-Orderscan (the same conventionsettleNoWorkNeededuses) rather than first-encountered, sincee.cfg.Stagescould theoretically contain more than one cleanup-flagged stage. If none is configured, the scan is a no-op for the whole poll — no panic. - Iterate
board.Itemsdirectly — notdeepFetchCandidates— for the same sourcing reason as §6.9/§6.10: a stranded closed item at a non-admitted stage never reachesdeepFetchCandidates. All fields the predicate needs (IsClosed,Status,Labels) are present on the shallow board query already; no deep fetch is required. - For each item: resolve its current stage via
stages.FindStage(e.cfg.Stages, item.Status). Skip if!item.IsClosed, or if a resolved stage isstage.CleanupWorktreeorstage.Name == "Validate"(excludes Validate specifically — left torunValidatePRTerminalAdvance/settleClosedValidateAdvance, preventing any double-advance/race between the two settle-owners; notstageIsGateChecked(stage), a broader category that also matches any other stage configured withwait_for_reviews: true, such as the shipped default Review stage — see the ADR-1387 follow-up note below). If the resolved stage isstage.HoldingStage, skip only whenmergeTrainWorkerActive(repoKey)is true for the item’sowner/repo— otherwise fall through and advance it like any other eligible stage. Astage == nilcolumn (Backlog, or any custom/extra column) is not skipped — it is treated as eligible, since it has no worktree and no stage bookkeeping to protect. - Otherwise call
advanceClosedItemToDone, which mirrorsadvanceToQueued’s shape: look up the cleanup stage’s status option, callUpdateProjectItemStatus, write throughboardcache.CacheImpl.UpdateItemStatus, and register the webhook echo. No completion label is added — unlikeadvanceToQueued’sstage:Validate:complete, this scan has no per-stage bookkeeping to do; landing the item at the cleanup column is sufficient for the existingCleanupWorktreedispatch branch (engine/item.go) to take over on a later poll (worktree reaping when a worktree exists,stage:Done:completelabeling regardless — see the correction below). - On any failure (
e.statusField == nil, no matching status option, or the API call itself failing), log a warning and move on — no retry counter, no escalation. The next poll re-derives the same predicate fromboard.Itemsand retries automatically; there is nothing to lose between polls, so a bare retry-forever loop is sufficient (ADR-064; contrast with §6.7–6.10’sMaxRetries/escalation machinery, which exists to protect an at-risk sequence of calls or a durable marker this scan does not have).
Downstream: the cleanup-stage admission gates explicitly admit worktree-less closed items (#1224). Once an item lands at the cleanup column via this scan, it must still pass itemMayNeedWork/itemNeedsWork before handleCleanupStage (which adds stage:Done:complete) ever runs. Before #1224, both gates’ CleanupWorktree branches keyed solely on worktreeExistsForItem — correct for an item that reached Done via the normal pipeline (Implement always creates a worktree), but this scan’s whole reason for existing is to land items that may never have had one (a closed issue that never reached Implement, or whose worktree was already reaped). A worktree-less item therefore failed worktreeExistsForItem, was dropped by both gates, and was permanently stranded: un-labelled (no stage:Done:complete) and consequently invisible to §6.12’s archival scan, which requires that exact label. This was a real, load-bearing bug in this scan’s own “sufficient to take over” assumption below, not a hypothetical.
itemMayNeedWork and itemNeedsWork’s CleanupWorktree branches now read, in order: (1) if the completion label (stage:<Done>:complete) is already present, refuse — the item is done, and refusing here (rather than only in itemNeedsWork) is what keeps a labelled worktree-less item cheap to skip on every subsequent poll, matching a worktree-having item’s cost once its worktree is reaped; (2) if a worktree exists on disk, admit — the pre-#1224 behavior, unchanged; (3) otherwise admit only if item.IsClosed && !item.IsPR — the case this scan produces. PR board items are deliberately excluded from step (3) (handleCleanupStage already special-cases !item.IsPR for worktree removal, but widening admission to worktree-less PR items was left as a separate, deferred concern — see #1224’s Scope).
With that fix, this scan’s original design intent now holds: landing an item at the cleanup column — worktree or not — is sufficient for the existing CleanupWorktree dispatch branch to take over on a later poll and apply stage:Done:complete exactly as it does for every other route into Done (§6.10, the normal pipeline advance, settleNoWorkNeeded). This scan closes the reachability gap (worktree leak, permanent dispatch stall); the archival gap — removing the item from the board afterward — is closed by §6.12’s settle scan, which requires no additional wiring here: once this scan lands an item at the cleanup column and handleCleanupStage applies stage:Done:complete, §6.12 picks it up on a later poll like any other Done item, worktree-agnostically (confirmed, not merely assumed — see §6.12’s own note).
Out of scope for #1224 (left for a follow-up): the cold-start/restart path. engine/terminal.go’s isProbeOnlyTerminal/seedTerminalFromProbeItems is a second, independent mechanism that can seed TerminalFlagSet{Terminal: true} for a closed, CleanupWorktree-stage, worktree-absent item using only probe-only data — which does not include labels — bypassing itemMayNeedWork/itemNeedsWork entirely (selectDeepFetchCandidates skips any item already flagged terminal before either gate runs). If an engine restart observes a closed, worktree-less item this scan already landed at Done before handleCleanupStage had a chance to label it, that item can be seeded terminal without ever being labelled — stranding it exactly as before #1224’s fix, via a different trigger (restart racing an unlabelled item, rather than every poll unconditionally). This is a distinct, narrower failure mode from the one #1224 fixes and was deliberately left out of this issue’s scope (see #1224’s Plan-stage Key Decisions); this note exists so the gap is documented rather than silently implied to be closed.
State transitions:
| Before | Trigger | After | Labels Added | Labels Removed |
|---|---|---|---|---|
Column <X> (non-Done, non-cleanup, non-gate-checked), open |
Issue closed | Column <X>, closed |
— | — |
Column <X>, closed (including Holding, if no train worker is active for its repo) |
Settle pass succeeds | Column <Done>, closed, worktree present or absent |
— (stage:Done:complete added separately by the CleanupWorktree dispatch branch on a later poll, once itemMayNeedWork/itemNeedsWork admit the item — see the correction above; admission no longer depends on a worktree existing) |
— |
Column <Done> or Validate, closed |
Settle pass evaluated | No change (out of scope for this scan; Validate is settleClosedValidateAdvance/runValidatePRTerminalAdvance’s exclusive domain, §6.15) |
— | — |
| Holding column, closed, merge-train worker active for the item’s repo | Settle pass evaluated | No change (left to the in-flight worker; retried next poll) | — | — |
Column <X>, closed |
Settle pass fails (transient API error, missing status field/option) | Column <X> (unchanged) |
— | — |
ADR-1387 follow-up: exclusion narrowed from stageIsGateChecked(stage) to stage.Name == "Validate". Originally this scan excluded any stageIsGateChecked stage, on the theory that gate-checked stages were categorically deferred to the Validate-specific settle-owner pair. But that pair only ever processes stage.Name == "Validate" — and both shipped default stage templates (stages/examples/review.yaml, stages/examples/validate.yaml) set wait_for_reviews: true, making Review gate-checked too. Under the broader exclusion, a closed item stranded at Review with no stage:Review:complete had zero remaining owners: dispatch admission refused it (R1, §2.8), no settle-owner processed anything but Validate, and this scan skipped it as well — a permanent, silent strand, caught in review on PR #1388. The fix is safe because advanceClosedItemToDone (what this scan calls) never inspects the linked PR at all, unlike advanceValidateTerminalItem (§6.15) — it only moves the board column, exactly as it already does, unconditionally, for a closed item at Specify/Plan/Implement. Extending that same behavior to Review needs no new PR-state reasoning, since Fabrik’s own PR merge action (attemptMergeOnValidate) only ever runs as part of Validate completing.
References: ADR-057: Single-Owner Validate PR Terminal Advance, ADR-064: Closed-Item-At-Any-Stage Advance To Done, ADR-1072: Holding Stages Are Reachable Only Via Dedicated Code, Not Positional Advance, ADR-1387: Closed Items Are Never Dispatched (the exclusion-narrowing follow-up described above)
6.12 Done-Item Archival
Trigger: A board item’s current Status resolves to the Done (cleanup) stage and it carries that stage’s completion label (stage:<Done>:complete), and at least ArchiveAfter (default 168h/1 week; --archive-after/FABRIK_ARCHIVE_AFTER) has elapsed since that label was applied. Disabled entirely via --archive-done off / FABRIK_ARCHIVE_DONE=off.
Why this exists. Un-archived Done and closed items accumulate on the project board indefinitely; every one of them is fetched on each poll cycle, inflating per-poll GraphQL cost. A prior implementation, archiveDoneCompleteItems, archived Done items immediately on first observation — with no grace period, completed work appeared to vanish from the board the instant it finished, so its call site was deliberately commented out and the dead function later removed (#1025). This settle scan re-implements the capability with a grace period anchored to a restart-safe, GitHub-side timestamp instead (ADR-068).
Why FetchLabelAppliedAt, not a local timestamp. No GraphQL field exposes “when did Status become Done” directly; ProjectV2Item.updatedAt is a coalesced max of issue/item/linked-PR activity, bumped by unrelated edits, and cannot serve as a “time since Done” measurement. GitHubClient.FetchLabelAppliedAt (github/labels.go) — the same GitHub-side, restart-safe REST lookup already backing the CI-wait gate (§6.5), the review-wait timeout (§6.1’s ReviewWaitTimeout), and the post-Validate convergence budget (§5.5) — reads the actual stage:<Done>:complete label-applied timestamp. A separate, abandoned prior attempt (#687/#688) used a local itemstate field set once at label-observation time, falling back to item.UpdatedAt after a restart; that fallback reintroduces the exact premature-archival risk the grace period exists to prevent, since updatedAt has no contractual link to the Done transition. FetchLabelAppliedAt avoids the fallback problem entirely by having no local state to lose.
Bounding the REST cost. FetchLabelAppliedAt pages through the full issue-events history — not cheap to call every poll for every Done item sitting in its grace window. The computed eligible-at time (appliedAt + ArchiveAfter) is cached in itemstate.CooldownAt under the reason key "archive-eligible-at" after the first successful lookup; every subsequent poll is a pure time.Now() comparison against the cached value, with no REST call. The cache is a cost-bounding optimization only, not a correctness-load-bearing marker: losing it (e.g., on an engine restart, since itemstate.Store is in-memory) costs exactly one extra FetchLabelAppliedAt call, never an incorrect archival.
Bounding the restart burst. The CooldownAt cache above bounds the cost per item, but on the first poll after a restart the cache is empty for every still-Done item at once — without a further cap, a bloated board (this feature’s own target scenario) would fire that many synchronous FetchLabelAppliedAt calls within a single poll. settleArchiveDoneItems tracks a per-poll fetch budget, maxArchiveLabelFetchesPerPoll (default 20, a package-level var so tests can lower it): once exhausted, remaining cache-misses are left uncached and simply retried on the next poll — the same safe fallback as the “not yet known” path below, so this only spreads the burst across polls, never causes early, stuck, or duplicate archival.
Fail-open on unknown timestamp. FetchLabelAppliedAt returns the zero time with a nil error when the label-applied event isn’t found (a deliberate fail-open contract). This scan treats a zero result as “not yet known” — no eligible-at time is cached, and the item is retried on the next poll — so an unknown timestamp can never trigger early archival, only a delayed one.
Code path: the settle scan, settleArchiveDoneItems (engine/archive_done_settle.go), called unconditionally every poll from poll.go, immediately after §6.11’s settleClosedItemsToDone — so an item this same poll advances into Done becomes visible to this scan on the next poll.
Flow:
- If
ArchiveDone == "off", the entire scan is a no-op for the poll. - Resolve the cleanup (Done) stage via
cleanupStage(e.cfg). If none is configured, no-op. - Iterate
board.Itemsdirectly — notdeepFetchCandidates— for the same sourcing reason as §6.9–§6.11: a Done item with its worktree already reaped never passesitemMayNeedWork’s admission guard again, so it is never deep-fetched.StatusandLabelsare present on the shallow board query already. - For each item where
item.Status == cleanup.Name && hasLabel(item.Labels, "stage:<cleanup.Name>:complete"), callmaybeArchiveDoneItem, threading a per-pollfetchBudgetshared across all items this scan evaluates (see “Bounding the restart burst” above). maybeArchiveDoneItemfirst checksitemstate.CooldownAt("archive-eligible-at")for a cached eligible-at time. On a cache miss, it consultsfetchBudget: if exhausted, the item is left uncached and retried next poll; otherwise it callsFetchLabelAppliedAtonce and decrements the budget. A zero result skips (see above); otherwise it computesappliedAt + ArchiveAfter, caches it viaitemstate.CooldownRecorded, and uses it as the eligible-at time for this poll too.- If
time.Now()is before the eligible-at time, the item is left alone (still within its grace period). - Otherwise: call
ArchiveProjectItem(board.ProjectID, item.ItemID). On success, write through the cache viaboardcache.CacheImpl.RemoveItem(item.ItemID)(wrappingstore.RemoveByItemID, mirroring the webhook delta path’s"deleted"/"archived"case), andlogfthe archived issue for operator visibility. On failure, log a warning and retry on the next poll — no retry counter, no escalation. No webhook echo is registered —boardcache/delta.go’s"deleted"/"archived"case never callsmatchEchoFn(only"edited"does), so a registered echo here would always expire unmatched and could spuriously tripWebhookStreamUnhealthyon burst-archival; theRemoveItemwrite-through already gives immediate cache coherence, mirroring the identical no-echo rationale inengine/no_work_needed_settle.gofor issue-close.
Idempotency and restart-safety. Archival is terminal: an archived item stops appearing in default board queries, so there is nothing left to re-observe — no marker is needed to prevent double-archival. On an engine restart, the CooldownAt cache is empty, so the very next poll re-fetches FetchLabelAppliedAt once per still-Done item, up to the per-poll fetch budget; if the label was already applied long enough ago, the item archives on that same poll (one extra REST call, not one extra full grace-period wait). A restart during the grace-period wait therefore costs at most one extra FetchLabelAppliedAt call (spread across polls if the budget is exceeded) — it can never cause early archival, a stuck item, or a double-archive.
Accepted residual risk: shallow-label truncation. The board query’s labels(first: 30) may not surface the completion label for an item with 30+ labels, silently skipping its archival indefinitely. ADR-021 already accepted this trade-off for the original implementation; mitigating it would require a deep fetch, which would reintroduce the per-poll GraphQL cost this scan exists to eliminate.
Accepted residual risk: manual Done→away→Done round trip. stage:<Done>:complete is never removed by any engine code path (consistent with every other stage:<Name>:complete label, which are permanent provenance markers). If an operator manually drags an item’s Status away from Done and back — there is no engine-driven path that does this — FetchLabelAppliedAt still returns the original label-applied timestamp (no new labeled event exists to reflect it), so the item re-archives on the stale, already-elapsed eligibleAt instead of waiting a fresh grace period. Not mitigated: the trigger is a human manually undoing/redoing a board move on an already-terminal item, not a state the engine itself produces. See ADR-068 for why clearing the CooldownAt cache alone would not fix this either.
Accepted assumption: single cleanup stage. cleanupStage(e.cfg) resolves only the lowest-Order CleanupWorktree: true stage — a board configured with a second cleanup-marked column would have items there never evaluated for archival. Shared with §6.11’s settleClosedItemsToDone; Fabrik’s stage-config convention is a single terminal Done/cleanup stage.
Worktree-less items (#1224): no change needed here. This scan’s eligibility check (item.Status == cleanup.Name && hasLabel(item.Labels, completeLabel)) was already worktree-agnostic by construction — it never inspects the filesystem. §6.11’s admission-gate fix means a worktree-less closed item now reaches this exact same eligible state (Done column + completion label) as any worktree-having item; this scan picks it up identically, with no code change of its own. Confirmed by the end-to-end test covering the full closed-item → Done → labelled → archived chain (engine/worktreeless_done_e2e_test.go), not merely by inspection.
State transitions:
| Before | Trigger | After | Labels Added | Labels Removed |
|---|---|---|---|---|
Done, stage:Done:complete, no cached eligible-at |
Settle pass: cache miss, FetchLabelAppliedAt returns non-zero |
Done (unchanged); eligible-at cached in CooldownAt |
— | — |
Done, stage:Done:complete, no cached eligible-at |
Settle pass: FetchLabelAppliedAt returns zero (not found) |
Done (unchanged); nothing cached | — | — |
Done, stage:Done:complete, eligible-at cached, not yet elapsed |
Settle pass evaluated | Done (unchanged) | — | — |
Done, stage:Done:complete, eligible-at cached, elapsed |
Settle pass succeeds | Archived (removed from board) | — | — |
Done, stage:Done:complete, eligible-at elapsed |
ArchiveProjectItem call fails |
Done (unchanged); retried next poll | — | — |
Any non-Done column, or Done without stage:Done:complete |
Settle pass evaluated | No change (out of scope for this scan) | — | — |
References: ADR-021: Housekeeping Mutations on Shallow Data, ADR-068: Done-Item Archive Timing
6.13 Non-Default-Base Explicit Close Retry
Trigger: closeIssueIfNonDefaultBase (§2.8, ADR-1096) calls e.client.CloseIssue as the last step of the outbound explicit-close path, after the caller’s (runValidatePRTerminalAdvance’s cruise path, or advanceConvergedPRToDone’s non-train yolo path) board advance to Done has already succeeded. This call fails (e.g. rate limit, transient API error).
Key difference from §6.8’s fabrik:awaiting-done: that marker is written before a chain of up to ten sequential calls, because any of them could be the one that first hits a rate limit. Here there is exactly one at-risk call, positioned after every other side effect of the caller has already landed — so the marker is written only in the failure branch, not unconditionally beforehand (ADR-1097, mirroring ADR-061).
Durable marker (fabrik:awaiting-close), written only on failure. markNonDefaultBaseCloseOutstanding adds the label (idempotently — a no-op if already present) only inside the if err != nil branch of the explicit close, after the gh.ErrNotFound short-circuit (already-deleted issue is treated as success, never reaches this branch) and after the item.IsClosed short-circuit (already-closed issue never attempts the call at all).
No dispatch-suppression wiring — deliberately. Unlike fabrik:awaiting-done, this marker is not checked by itemMayNeedWork/itemNeedsWork, and is not added to transientLifecycleLabels. By construction, the item has already reached Done (both callers advance the board before calling closeIssueIfNonDefaultBase) by the time this marker can exist — there is no per-stage redispatch this marker needs to prevent. This sidesteps the terminal-skip/transientLifecycleLabels interaction a naive verbatim port of §6.8’s shape would have hit, identical to §6.10’s reasoning (ADR-061 Context).
Code path: runValidatePRTerminalAdvance / advanceConvergedPRToDone → closeIssueIfNonDefaultBase (writes the marker on close failure) — and, on retry, settleNonDefaultBaseCloses → settleNonDefaultBaseClose (both engine/close_nondefault_base_settle.go; settleNonDefaultBaseCloses is called from poll() in poll.go) directly.
Retry-owner: settleNonDefaultBaseCloses (engine/close_nondefault_base_settle.go; called from poll() in poll.go). Runs unconditionally once per poll, immediately after settleMergeTrainMemberCloses. It iterates the raw board.Items (not deepFetchCandidates — this scan has no dependency on the deep-fetch/terminal-skip machinery at all). For every item carrying fabrik:awaiting-close and not fabrik:paused (mirroring settleMergeTrainMemberCloses’s own paused-item guard — an operator investigating a paused item must not be fought), it calls settleNonDefaultBaseClose.
settleNonDefaultBaseClose flow — idempotent; safe to call repeatedly:
- If
item.IsClosedis already true (a prior settle pass already closed it, or a human closed it manually), skip the redundantCloseIssuecall and go straight to clearing the marker. - Otherwise call
CloseIssue. On success, clear the marker (and write through the boardcache viaApplyIssueClosed, matchingcloseIssueIfNonDefaultBase’s own success path). On failure, record a retry and stop — the next settle pass re-attempts.
Retry counting and escalation. Every failed settle pass calls recordNonDefaultBaseCloseRetry, which increments the existing itemstate.StageRetryIncremented/Attempts counter — keyed by the dedicated constant "__non_default_base_close__" (same double-underscore-wrapped, YAML-unrepresentable shape as "__merge_train_member_close__"/"__no_work_needed__", so it can never collide with a configured stage’s own counter). MaxRetries <= 0 means unlimited retries, never escalate (same guard as recordMergeTrainMemberCloseRetry). Once Attempts("__non_default_base_close__") >= e.cfg.MaxRetries, escalateNonDefaultBaseCloseFailure fires — mirroring escalateMergeTrainMemberCloseFailure: adds fabrik:paused, removes fabrik:awaiting-close, posts an explanatory comment naming the merged PR (item.LinkedPRNumber, when non-zero — already populated on the board item, so no new storage is needed to thread it from the original closeIssueIfNonDefaultBase(item, prNumber) call) with the manual recovery step (gh issue close <N> --repo <owner>/<repo>), and applies itemstate.EnginePaused.
Marker clearing. fabrik:awaiting-close is removed in exactly two places: clearNonDefaultBaseCloseMarker (a fully successful settle pass, or an already-closed issue found on a later pass — the normal, expected path) and escalateNonDefaultBaseCloseFailure (giving up after MaxRetries). It is not referenced by cleanupClosedIssueTransientLabels’s defensive sweep at all — unlike fabrik:awaiting-done, there is no “silently resurrect the bug” risk from an early strip, since the settle scan’s own item.IsClosed check already treats a closed issue as fully settled.
Scope: this mechanism covers only the two terminal merge-advance sites’ calls into closeIssueIfNonDefaultBase (the merge-train path and the FABRIK_NO_WORK_NEEDED short-circuit already close explicitly via their own paths and are unaffected — see §2.8). The generic recordSettleRetry/escalateSettle/clearSettleMarker helpers (engine/settle.go) this mechanism reuses are the same ones backing fabrik:awaiting-done (§6.8) and fabrik:awaiting-member-close (§6.10).
State transitions:
| Before | Trigger | After | Labels Added | Labels Removed |
|---|---|---|---|---|
| Done | closeIssueIfNonDefaultBase’s explicit CloseIssue fails |
Done, awaiting close (marker present) | fabrik:awaiting-close |
— |
| Done, awaiting close | Settle pass succeeds (issue closed, or already closed) | Done, settled | — | fabrik:awaiting-close |
| Done, awaiting close | Settle pass fails Attempts >= MaxRetries times |
Done, Paused | fabrik:paused |
fabrik:awaiting-close |
References: ADR-1096: Explicit Close on Non-Default-Base Merge, ADR-061: Merge-Train Singleton Member-Issue Close Retry, ADR-1097: Non-Default-Base Explicit Close Retry
6.14 Awaiting-CI Settle Scan
Trigger: Field evidence (issue #1270) showed an item can strand indefinitely with fabrik:awaiting-ci present and a confirmed CI failure: the shared, three-layer admission pipeline that fed the CI gate — itemMayNeedWork (engine/item.go), selectDeepFetchCandidates’s cooldown pre-filter, and the main catch-up loop’s own per-item admission gate — could silently exclude the item from ever reaching checkCIGate, with no log line marking the exclusion. dispatchCIFixReinvoke was never called, MaxCiFixCycles never tripped, and no timeout fired.
Same failure class as four prior fixes. fabrik:awaiting-ci was the one durable “awaiting-X” marker in the codebase still relying exclusively on the shared, admission-gated catch-up path rather than a dedicated board.Items-sourced settle scan — the same shape already fixed for fabrik:awaiting-done (§6.8, ADR-060), spawned-child board placement (§6.9, ADR-062), merge-train singleton member-issue close (§6.10, ADR-061), and non-default-base explicit close (§6.13, ADR-1097). settleAwaitingCIScan is the fifth instance of the pattern.
Code path: settleAwaitingCIScan (engine/ci_settle.go; called from poll() in poll.go, immediately after runValidatePRTerminalAdvance and before settleNoWorkNeededScan).
Not a durable-marker-write-on-failure mechanism, unlike §6.10/§6.13. fabrik:awaiting-ci is applied by handleStageComplete on FABRIK_STAGE_COMPLETE (§6.5.1 Path 1/2) — this scan does not write the marker, it is the marker’s sole evaluator for open items. See §6.5.1 Path 2 for the full evaluation mechanics (admission model, orphan-column handling, and the runCatchUpPhase2 same-poll landing handoff shared with §6.6.6).
Admission independence is the entire fix. The scan iterates the raw board.Items, filtering only on fabrik:awaiting-ci present, fabrik:paused absent, and item.IsClosed false (closed-issue recovery is runValidatePRTerminalAdvance’s job, ADR-056 D2 — duplicating it here would be a second owner for no benefit). It never calls itemMayNeedWork, is never filtered by selectDeepFetchCandidates’s cooldown logic, and does not depend on the item having appeared in deepFetchCandidates this poll at all — so whatever excluded #3915 from that pipeline in the field cannot exclude it from this scan.
No double-dispatch. The main catch-up loop’s per-item admission gate was narrowed to hasComplete-only as part of this fix (§6.5.1). Since fabrik:awaiting-ci and stage:X:complete are mutually exclusive in steady state, an item is always routed to exactly one of the two paths per poll — never both — so CIFixCycles/worker dispatch can never double-fire for the same item in the same poll.
Diagnosability. Every pass logs under the awaiting-ci-settle tag: a cache-hit/deep-fetch pre-check line per item (mirroring selectDeepFetchCandidates’s poll.go pattern); a stray-column skip names the column and the reason it can’t host the CI gate. A summary line reports two counts when non-zero — items examined (every fabrik:awaiting-ci item the scan looked at, including ones stuck in the orphan-column/deep-fetch-failure retry branches) and items that reached the CI gate — kept separate so a poll where every item is retrying without reaching the gate doesn’t read as “0 items, nothing happening.” This directly addresses the issue’s own root-cause complaint — an 80-minute silence with no log line explaining why the item wasn’t progressing. Corrected by #1303 (see §6.14.1): “reached the CI gate” now means exactly that — phase1Ctx.reachedCIGate is set only immediately before handleMergeAndCIGates calls checkCIGate, not merely “made it through catchUpPhase1Handlers” (an earlier handler in the chain, e.g. checkMergeabilityGate’s claim, can still consume an iteration without checkCIGate ever running).
State transitions:
| Before | Trigger | After | Labels Added | Labels Removed |
|---|---|---|---|---|
wait_for_ci stage, awaiting CI |
Settle pass: CI still pending | Unchanged | — | — |
wait_for_ci stage, awaiting CI |
Settle pass: CI fails | Unchanged (or paused at MaxCiFixCycles) |
fabrik:paused (cycle limit only) |
— |
wait_for_ci stage, awaiting CI |
Settle pass: CI clears | stage:X:complete present; landing decision reached via runCatchUpPhase2 |
stage:X:complete |
fabrik:awaiting-ci |
Non-wait_for_ci/HoldingStage/Unmanaged/CleanupWorktree column, awaiting CI |
Settle pass: orphan column detected | Unchanged | — | — |
wait_for_ci stage, awaiting CI |
Settle pass: FetchItemDetails fails with a transient/rate-limit error (isTransientAPIError) |
Unchanged — deferred, no counter change (issue #1313, see §6.14.2) | — | — |
wait_for_ci stage, awaiting CI |
Settle pass: FetchItemDetails fails with a non-transient (structural) error |
Unchanged — counts toward escalation | — | — |
Orphan column or repeated non-transient FetchItemDetails failure, awaiting CI |
Attempts("__awaiting_ci_orphan__") >= MaxRetries (the two causes share one counter — see below) |
Paused | fabrik:paused |
fabrik:awaiting-ci |
A repeated non-transient FetchItemDetails failure is the other “gate genuinely cannot be evaluated” case (in practice this is permissions, a deleted issue node, or another persistent, per-item error) — the scan can never reach checkCIGate for such an item either, even though it resolves to a perfectly valid wait_for_ci stage. It shares the same __awaiting_ci_orphan__ retry counter and escalateAwaitingCIOrphanFailure escalation path as the orphan-column case (mirroring escalateNoWorkNeededFailure’s precedent of one counter covering multiple failure causes with a generic message), so orphan-column retries and non-transient deep-fetch-failure retries accumulate toward the same MaxRetries threshold regardless of which cause fires on which pass. escalateAwaitingCIOrphanFailure re-resolves the item’s current stage at escalation time so the posted comment names whichever cause is actually current — a stray column, or a persistent non-transient fetch failure — rather than always claiming a stray column. A transient/rate-limit FetchItemDetails failure (issue #1313) never reaches this counter at all — see §6.14.2.
References: ADR-1270: Awaiting-CI Settle Scan, ADR-056 (convergence/settling consolidation), ADR-060 (fabrik:awaiting-done settle scan), ADR-061 (merge-train member-close retry), ADR-1097 (non-default-base close retry), ADR-1216 (§6.6.6, joint-clearing handoff), ADR-1223 (gate-pause-terminates-not-clears), #1135 (orphaned durable labels — same failure class)
6.14.1 Awaiting-CI Handler-Chain Stall (issue #1303)
Trigger: #1270 fixed the admission layer for fabrik:awaiting-ci (§6.14 above) — the scan reliably reaches catchUpPhase1Handlers every poll. Field evidence (fabrik-test-alpha#3930, PR #3932) showed a second, downstream stall: an item with a mergeable PR and a confirmed check-run failure could still sit indefinitely once inside the handler chain, with zero settle/ci-gate log output and a stale fabrik:rebase-needed label never cleared, for over 80 minutes with no CI-fix reinvoke, no pause, and no timeout.
Confirmed root cause: an unbounded cached CheckRunsPending classification. boardcache.CacheImpl.FetchCheckRuns() only forces a live GitHub refetch when the cached classification would read as CheckRunsFailed (the #958 leg 3 guard, §6.4 above) — a cached CheckRunsPending snapshot is served from cache indefinitely. On a webhook-less deployment nothing else ever supersedes it: no check_run event arrives to update the store, so a poll that happens to cache a PENDING snapshot moments before the real checks resolve latches the item permanently. This is a latent race, not a regression — the same code path is unchanged since the last known-good release; it only fires when a poll’s timing catches CI mid-flight, which explains why the field incident was intermittent (1 of 4 runs) rather than universal.
The resulting silent chain: settlePRMergeState()’s CheckRunsPending case returns PRMergeUnsettled → checkMergeabilityGate claims (blocked=true) → handleMergeAndCIGates’s if mergeBlocked { return true } claims the item → checkCIGate (and the 30-minute CIWaitTimeout escalation logic inside it) is never reached. Three of these four steps produced no log line at all before this fix, which is why the incident took over an hour to diagnose from logs alone.
Fix 1 — CacheImpl.RefreshCheckRunsLive(owner, repo, sha) (boardcache/boardcache.go). Superseded by #2180 (ADR 2180): FetchCheckRuns now confirms any cached PENDING or FAILED set live (a PENDING set confirmed within 15 s is served), which makes this per-poll prime redundant; RefreshCheckRunsLive and the settleAwaitingCIScan call are removed. The text below is the as-decided history. An unconditional live check-run fetch that bypasses FetchCheckRuns’s cache-trust check entirely and applies the result into the Store exactly as FetchCheckRuns’s own miss path does. settleAwaitingCIScan calls it once per fabrik:awaiting-ci item per poll — after FetchItemDetails, before the handler chain — guarded on item.LinkedPRHeadSHA/item.LinkedPRNumber being populated. Scoped narrowly to this one caller rather than inverting FetchCheckRuns’s general cache-trust contract: the general contract (still denylist — refetch only on a would-be-FAILED classification) is unchanged for its ~35 other call sites, and TestFetchCheckRuns_CachedPending_ServedFromCache continues to document that contract for everyone else. This closes the confirmed root cause: a stale cached Pending is now primed with genuinely fresh data before checkMergeabilityGate ever evaluates it. Correction (issue #1325, see §6.14.3): this guard’s item.LinkedPRHeadSHA does not always come from “that poll’s own fresh GraphQL deep-fetch” as originally stated here — on the far more common cache-hit path, FetchItemDetails never performs a GraphQL fetch at all, and until #1325 the guard’s SHA field was left unpopulated on that path, making this fix inert almost everywhere it needed to run. See §6.14.3 for the full story.
Fix 2 — an unconditional CIWaitTimeout backstop, ahead of the handler chain (engine/ci_settle.go). Fix 1 makes checkCIGate reachable for the confirmed cause, but does not bound the class: any other future silent claim earlier in the chain would reproduce the same unbounded stall, because the CI-wait timeout logic lives inside checkCIGate, which such a claim would make unreachable. settleAwaitingCIScan now checks fabrik:awaiting-ci’s FetchLabelAppliedAt age against ciWaitTimeout() before building phase1Ctx and running the handler chain at all; once exceeded, it calls pauseForCITimeout directly and escalates regardless of what any gate would otherwise classify or claim. Inert on the happy path — an item that reaches checkCIGate normally hits the identical timeout at the identical threshold; the backstop only fires when the item never gets that far. Correction (issue #1408, see §6.14.4): “idempotent… no-ops” understated what happened when hasCIGatePauseComment already matched — pauseForCITimeout silently did nothing at all (no labels, no return signal), and the backstop continued past the handler chain regardless, permanently stranding any item resumed after this fired. §6.14.4 describes the fix.
Fix 3 — every previously-silent Phase 1 claim path now logs. checkMergeabilityGate’s PRMergeUnsettled and PRMergeQueued cases (engine/merge_gate.go) and handleMergeAndCIGates’s if mergeBlocked { return true } branch (engine/catch_up_handlers.go) each log under their respective merge-gate/ci-gate tags, naming the branch and settle.Reason. handleDependencies and handleReviewGate were audited and already logged on every claim path (checkDependencies’s “waiting for…“/cycle-detection lines; checkReviewGate’s “waiting for reviewers…” and handleBrokenReviewLinkage’s broken-linkage lines) — no changes were needed there. These are logging-only additions; no claim/no-claim return value changed for any existing caller.
Fix 4 — gateReached now measures what its log line claims (engine/ci_settle.go). Before this fix, gateReached incremented unconditionally at the end of every loop iteration in settleAwaitingCIScan, regardless of which handler in catchUpPhase1Handlers claimed the item (or whether any did) — so the summary line (“examined N item(s), N reached the CI gate”) could report an item as having reached the CI gate when checkCIGate never actually ran. This misled triage of the very incident this section documents: an operator initially read “1 reached the CI gate” as proof the gate was being evaluated. phase1Ctx now carries a reachedCIGate bool field, set true by handleMergeAndCIGates immediately before its checkCIGate call — the one call site that matters — and settleAwaitingCIScan reads it after the handler loop instead of incrementing unconditionally.
Fix 5 — the in-flight worker marker can no longer outlive a pre-PID hang (engine/worker_liveness.go). Investigated as a candidate root cause and ruled out for this incident (the field log shows exactly one “already in-flight” skip, not a sustained one), but a real, narrower gap was found and closed: runWorkerDetectorScan previously skipped every WorkerHandle with PID <= 0 unconditionally — correct for the common case (a worker still on its way to onPIDReady), but a dispatch goroutine that hangs before onPIDReady fires (e.g. stuck in ensureRepoReady) never records a PID, so isWorkerStale’s signal-0 liveness check can never confirm it dead, and the marker (plus the fabrik:locked:<user>/stage:<name>:in_progress labels it gates) could outlive the hung goroutine indefinitely, permanently suppressing dispatch for the item. The detector now applies workerStaleTimeout() against w.StartedAt (not LastSignAt, which never starts ticking for a worker that never reached PID assignment) for the PID <= 0 case, and clears via the existing cleanupStaleWorker path on timeout — logged distinctly as “could not verify liveness” (a timeout-based clear, not a signal-0-confirmed-dead clear, since this path can never confirm death directly).
References: ADR-1270: Awaiting-CI Settle Scan (the admission-layer predecessor fix), #958 leg 3 (the pre-existing FAILED-refetch precedent this fix extends the same reasoning to for PENDING, scoped narrowly rather than generally), ADR-1223 (ciTerminated claim-ordering convention, unaffected by this change), issue #1303
6.14.2 Awaiting-CI Settle Scan: Transient vs. Structural FetchItemDetails Failures (issue #1313)
Trigger: Field evidence (fabrik-test-alpha#3989) showed settleAwaitingCIScan’s FetchItemDetails error path (§6.14) routed every error — transient or structural — through the same __awaiting_ci_orphan__ escalation counter. A GraphQL rate-limit exhaustion episode ("API rate limit already exceeded for user ID ...") affects every item on the board simultaneously, not just the one being processed, and resolves itself automatically at the next hourly reset. Three consecutive rate-limited settle passes exhausted MaxRetries and paused an otherwise-healthy issue — final labels included fabrik:cruise, fabrik:paused, stage:Specify:complete … stage:Review:complete — requiring manual fabrik:paused removal to recover. This is distinct from #1270’s original stall (an unevaluatable gate that never surfaced): here the gate was evaluatable, but a global, self-healing API condition was mistaken for a per-item, persistent one.
Fix — classify the error before calling recordAwaitingCIOrphanRetry. isTransientAPIError(err error) bool (engine/item.go, immediately after isTransientError) layers rate-limit/quota-exhaustion detection on top of the existing isTransientError predicate: it returns true for everything isTransientError already recognizes (network-layer errors, unexpected EOF, GitHub 5xx, connection reset, i/o timeout) plus GraphQL/REST rate-limit and secondary-rate-limit/abuse-detection error text, matched case-insensitively via rateLimitErrorPatterns ("api rate limit exceeded", "api rate limit already exceeded", "secondary rate limit", "abuse detection" — specific multi-word phrases, deliberately avoiding a bare "rate limit" or "api rate limit" substring per the precedent in botServiceNoticePatterns, engine/comments.go), plus a direct HTTP-429 status-code check ("github api returned 429") since 429 is used exclusively for rate-limiting by GitHub’s API and would otherwise be missed by a response body that doesn’t happen to match any known phrase. Both narrowings (splitting the bare "api rate limit" phrase, and adding the 429 status check) were added in PR review (pruefer) after the bare phrase was found to also match unrelated prose that merely mentions the concept without reporting an actual exhaustion. Anything not confidently matched falls through to false (structural) — including the plain, unwrapped "node not found" error FetchItemDetails returns for a deleted issue node, which has no underlying error to match against and correctly stays structural.
settleAwaitingCIScan’s FetchItemDetails error branch now checks isTransientAPIError(err) before deciding whether to call recordAwaitingCIOrphanRetry: a transient/global error logs distinctly and continues without calling recordAwaitingCIOrphanRetry at all — no counter touch, however many consecutive polls are affected, so a rate-limit episode of any length can never mass-pause the item. A non-transient error is handled exactly as before (§6.14): logged, recordAwaitingCIOrphanRetry called, escalating to fabrik:paused + comment at MaxRetries. The itemstate.DeepFetchFailed store mutation (bookkeeping only — records LastDeepFetchFailureAt) stays unconditional for both branches, since it has no bearing on the escalation counter.
Escalation now names the failure class. Because a transient/global error never reaches recordAwaitingCIOrphanRetry, escalateAwaitingCIOrphanFailure (§6.14) can only ever fire for the orphan-column case or a genuinely non-transient fetch failure — never a rate-limit episode. Its fetch-failure problem string was reworded to say the failure was non-transient (“rate-limiting and other transient conditions defer automatically and never reach this point”), so an operator reading the pause comment can immediately rule out a rate-limit episode as the cause.
Scope. isTransientAPIError is a sibling predicate, not an extension of isTransientError itself — isTransientError’s four other call sites (removeEditingLabel, dependencies.go, pr.go ×3, prcreate.go) are untouched and cannot be affected by this change. recordSettleRetry/escalateSettle (engine/settle.go, shared by four other settle scans per ADR-060/061/062/1097) are unchanged — this fix acts entirely at the ci_settle.go call site, deciding whether to call recordAwaitingCIOrphanRetry, never changing what it or its downstream helpers do. The orphan-column retry path (§6.14) is unaffected — it doesn’t call FetchItemDetails and still shares the same counter unconditionally, exactly as before this fix.
References: ADR-1270: Awaiting-CI Settle Scan (amended in place for this issue rather than superseded — see its “Why does a FetchItemDetails failure retry-then-escalate through the same path as an orphan column?” subsection), engine/backoff.go (shouldPauseForRESTRateLimit/isRateLimitNearZero, the pre-existing numeric rate-limit-aware poll-interval backoff — a structurally different mechanism from this issue’s per-error-string classification), engine/terminal.go (GraphQL-budget-aware polling pause via RateLimitStats()), issue #1313
6.14.3 Awaiting-CI Cache-Hit Guard Never Fired (issue #1325)
Trigger: #1303’s primary fix (§6.14.1, Fix 1) had never executed a single time in a multi-hour, multi-slice validation run — awk '/refreshing check runs live/' fabrik.log | wc -l returned 0 across the entire test-bed log, despite the cache being demonstrably active. A validation slice on main reproduced the original #1303 symptom exactly: fabrik-test-alpha#4157/PR #4158 reported CI checks pending for 89 minutes while the live PR head had already resolved to a definitive ci-fix-sentinel FAILURE within minutes. Only the CIWaitTimeout backstop (§6.14.1, Fix 2) stopped it — the primary fix contributed nothing, and the backstop’s sensible pause-with-comment behavior masked the wedge from the outside.
Confirmed root cause: copyDeepFieldsFromState never copied LinkedPRHeadSHA. The RefreshCheckRunsLive call site’s guard (engine/ci_settle.go) is item.LinkedPRHeadSHA != "" && item.LinkedPRNumber != 0, evaluated against the item value returned by settleAwaitingCIScan’s FetchItemDetails call. That call goes through boardcache.CacheImpl.FetchItemDetails, which on a cache hit calls copyDeepFieldsFromState to overlay deep fields from the store’s cached ItemState onto the passed *gh.ProjectItem — and, until this fix, that function copied LinkedPRNumber from s.LinkedPR but never assigned dst.LinkedPRHeadSHA, leaving it at its zero value ("") on every hit. On a cache miss, the live GraphQL path (applyLinkedPRs, github/project.go) populates both fields together from headRefOid, so the guard passed — but only in the one case (a fresh live fetch just happened) where RefreshCheckRunsLive’s rescue wasn’t needed. Net effect: the live refresh could only run when the cache was not serving a stale read, making it inert exactly when the #1303 wedge was in play.
Fix — copyDeepFieldsFromState (boardcache/boardcache.go) now also copies dst.LinkedPRHeadSHA = s.LinkedPR.HeadSHA, symmetric with the existing LinkedPRNumber copy from the same s.LinkedPR struct. This benefits every FetchItemDetails caller uniformly rather than adding a second, settleAwaitingCIScan-specific SHA-sourcing mechanism at the call site. ItemState.LinkedPR.HeadSHA is safe to trust on a cache hit: it’s written by three independent, pre-existing freshness mechanisms — every deep fetch (CacheImpl.FetchItemDetails’s own PRHeadSHAUpdated write on a miss), pull_request webhook deltas, and check_run webhook deltas (boardcache/delta.go) — the identical set of writers that already keep every other LinkedPR field (reviews, mergeability, queue state) current. No new, independently-stale cache is introduced. FetchCheckRuns’s general cache-trust contract (§6.14.1, Fix 1’s denylist-only-on-FAILED behavior, ~35 other call sites) is untouched — this fix only supplies the missing input to a guard whose own logic was already correct, and does not change the guard itself, the CIWaitTimeout backstop, or FetchCheckRuns’s scope.
Test-gap note. #1303’s own regression test (TestSettleAwaitingCIScan_StaleCachedPendingCheckRuns_EndToEnd) sets item.LinkedPRHeadSHA directly in its mock fetchItemDetailsFn, so its single FetchItemDetails call is always a cache miss — it never exercised copyDeepFieldsFromState and so never caught this defect. The new regression test (TestSettleAwaitingCIScan_CacheHitPath_RefreshCheckRunsLiveReached) primes the real boardcache.CacheImpl with a genuine prior deep fetch, then calls settleAwaitingCIScan with a fresh/zeroed board item and an unchanged UpdatedAt so FetchItemDetails takes the cache-hit branch — confirmed failing against the pre-fix copyDeepFieldsFromState and passing after.
References: §6.14.1 (issue #1303 — the fix this issue makes reachable), §6.14.2 (issue #1313), issue #1320/#1323 (unrelated e2e scenario defects that had independently prevented TestCIFixReinvokeCycleLimit from reaching cycle 2 and exposing this)
6.14.4 CI-Gate Pause Comment Reuse on Resume (issue #1408)
Trigger: Field evidence (verveguy/liminis-context-graph#342, 2026-08-04) showed an item that once hit the CIWaitTimeout backstop (§6.14.1, Fix 2) could never be resumed by the remedy its own pause comment instructs. The linked PR was CLEAN/APPROVED/MERGEABLE with every check green since 13:25:44Z; the issue sat stranded from 20:00:57Z onward carrying fabrik:awaiting-ci with no stage:Validate:complete and no fabrik:paused — the timeout comment had been posted once, at 13:21:56Z, and a human had since removed fabrik:paused to resume, exactly as instructed.
Confirmed root cause: hasCIGatePauseComment’s unscoped match, combined with a never-reset fabrik:awaiting-ci appliedAt. fabrik:awaiting-ci’s applied-at timestamp is set once, at handleStageComplete time (ADR-1314), and is never reset for the label’s lifetime — neither pauseForCITimeout nor pauseForCIFixCycleLimit remove or reapply fabrik:awaiting-ci itself, only checkCIGate’s classify helpers do, on a confirmed gate-clear or confirmed timeout. So once an item’s fabrik:awaiting-ci ages past CIWaitTimeout once, every subsequent appliedAt-based check still reads it as timed out, regardless of live CI state, until the label is genuinely removed or reapplied. hasCIGatePauseComment (engine/ci.go) was written to identify a single timeout/cycle-limit episode by scanning the issue’s entire comment history for the stable prose fragment — deliberately unscoped by time, to catch the same-poll two-call label-swap race described in §6.14 — but before this fix, pauseForCITimeout/pauseForCIFixCycleLimit treated any match as “nothing more to do” and returned without touching labels or reporting anything. The backstop’s caller (settleAwaitingCIScan) then continued past the handler chain unconditionally, on the assumption that calling pauseForCITimeout always means “this item was just escalated.” A human resuming the item by removing only fabrik:paused (per the pause comment’s own instructions) never touches the comment or fabrik:awaiting-ci, so the very next settle pass re-derived “timed out” from the same stale appliedAt, matched the old comment, and looped: no label churn, no comment, no re-evaluation — forever.
Why the no-op itself was not the bug. Suppressing a duplicate pause comment within the same episode (the two-call race) is correct and remains unchanged (R3). The defect was conflating that suppression with “escalated” at the call site: the backstop’s continue treated both outcomes identically, so a suppressed escalation was never followed up by anything that could re-evaluate the item.
Fix — split by whether a live CI read has already happened this call. The fix has two coordinated pieces; neither alone satisfies both the “green CI advances” and “still-failing CI re-escalates” requirements (R1/R4) — see the two rejected single-piece designs recorded in ADR-1408:
- The backstop (
engine/ci_settle.go) checkshasCIGatePauseCommentitself, before ever callingpauseForCITimeout. When no comment exists yet (a genuinely fresh timeout), it pauses andcontinues exactly as before. When a comment already exists for this episode, it does not callpauseForCITimeoutand does notcontinue— it logs that the item is deferred to the live-data-informed handler chain and falls through toRefreshCheckRunsLiveandcatchUpPhase1Handlers, unchanged otherwise. This is what lets a resumed item with green CI clear the gate cleanly (checkCIGate’sPRMergeReady/PRMergeTerminal/PRMergeNoPRcases short-circuit toaddCompleteLabelAndRemoveCIbefore any timeout check runs) with zero pause-label churn. pauseForCITimeout/pauseForCIFixCycleLimit(engine/ci.go) stop no-opping whenhasCIGatePauseCommentmatches. Every caller reaching this branch has, by construction, already done a live CI read this poll (either the backstop’s own fresh-timeout branch — irrelevant, since that branch never has a match — orhandleMergeAndCIGates, reached only aftercheckCIGate/classifyCIFrom*re-derivesciTimedOut/ciFailurefrom the same staleappliedAtfor a still-blocked item). So it is now safe to reapplyfabrik:paused+fabrik:awaiting-input(idempotent GitHub label writes, via a smallreapplyCIGatePauseLabelshelper) without posting a new comment — the existing episode’s comment is reused. Both functions now returnescalated bool:truefor a fresh post,falsefor a reused/relabeled-only pass (R2).
handleMergeAndCIGates (engine/catch_up_handlers.go) needed no changes — its existing unconditional calls into the two pause functions already do the right thing once those functions stopped no-opping, since every path into them already ran a live settle read this poll.
Why reuse the comment rather than repost (R4). hasCIGatePauseComment’s unscoped, full-history match is exactly what makes comment reuse possible without a new time-scoping mechanism (a per-poll dedup set does not survive the two-call race test’s two separate top-level settleAwaitingCIScan calls; comment-recency matching would require realistic CreatedAt stamping in test mocks that don’t currently provide it). The fix repurposes the existing check rather than adding one, and hasCIGatePauseComment’s matching logic itself is unchanged (R6 below covers only its doc comment and the two call sites’ log wording).
R4 — a still-blocked resumed item is genuinely re-escalatable, not silently unpausable. Because the reused-comment path still reapplies fabrik:paused + fabrik:awaiting-input every time it’s reached, a resumed item whose CI is genuinely still stalled (classifyCIFromCheckRuns’s own independent liveness check re-derives timedOut for a still-pending, no-progress case) is paused again on the very next settle pass, not left stranded a second time. Updated by ADR-1410: this description originally also covered “still failing” — at the time this fix landed, classifyCIFromCheckRuns re-derived timedOut identically for a confirmed failure past the deadline as for a still-pending one (the R3 defect ADR-1410 later fixed). Since ADR-1410, a confirmed failure instead re-derives ciFailure=true unconditionally and dispatches CI-fix reinvocation rather than re-pausing — see the table row below and TestSettleAwaitingCIScan_ResumedAfterTimeout_StillFailing_DispatchesCIFix (engine/ci_settle_test.go), which replaced this section’s original still-failing pause-based test for the identical fixture.
R5 — pauseForCIFixCycleLimit gets the identical fix, and the strand is reachable independent of the CIWaitTimeout backstop. pauseForCIFixCycleLimit has two call sites — handleMergeAndCIGates’s cycle-limit callback (engine/catch_up_handlers.go) and checkAutoMergeConvergence’s queue-repo ejection-recovery path on PRMergeBlocked (engine/merge_gate.go) — both always reached after a live CI classification this poll (ciFailure and settle.Status == PRMergeBlocked respectively; autoMergeConvergence and mergeAndCIGates are mutually-exclusive handlers in catchUpPhase1Handlers, so they can’t double-fire for the same item), so neither needs caller-side gating — only the same reapply-without-repost behavior internally. The strand is independently reachable via resume: clearFailedStage() only resets CIFixCycles when stage:<X>:failed is present or snap.PausedByEngine(stageName) is true, and a CI-fix-cycle-limit pause sets neither (pauseIssue never applies itemstate.EnginePaused) — so CIFixCycles survives a resume unreset, and a human resuming an item paused at the cycle limit before CIWaitTimeout has separately elapsed hits this exact path, entirely independent of the backstop.
R6 — corrected wording. hasCIGatePauseComment’s doc comment and both pause functions’ log lines previously described the match as “already posted this poll” — inaccurate, since the match is unscoped by time and identifies an entire episode, not a single poll. Both now describe the unscoped, full-episode match and the reapply-without-repost behavior it enables.
R7 — no behavior change for an item that has never posted a CI-gate pause comment. hasCIGatePauseComment returns false for such an item exactly as before; the backstop’s fresh-timeout branch (pause and continue) and both pause functions’ fresh-post branch (post via pauseIssue, return true) are byte-identical to pre-#1408 behavior.
Correction (issue #1937): the match is anchored to the pause-comment shape. The shared primitive behind hasCIGatePauseComment and the review/rebase/enqueue wrappers is hasPauseComment (engine/mutate.go). It still scans the whole comment history, so it still identifies an episode. But a comment now counts only if it has the engine’s own pause shape: a 🏭 **Fabrik — …** header line, a blank line, then the fragment at the start of the body text. Every pause message is built that way.
The previous unanchored substring match also hit a fragment merely quoted elsewhere. In the live case (release gate, TestCIFixReinvokeCycleLimit), a Research output recounted an earlier run’s “The stage Validate has been re-invoked to fix CI failures 2 time(s)…”. When the real limit was hit, pauseForCIFixCycleLimit took the reapply branch and paused with labels only: no explanatory comment was ever posted. A human quoting a pause message had the same effect.
State transitions (new rows; existing §6.14 rows unaffected):
| Before | Trigger | After | Labels Added | Labels Removed |
|---|---|---|---|---|
wait_for_ci stage, awaiting CI, existing pause comment, no fabrik:paused (resumed) |
Settle pass: live CI green | stage:X:complete present; landing decision reached via runCatchUpPhase2 |
stage:X:complete |
fabrik:awaiting-ci |
wait_for_ci stage, awaiting CI, existing pause comment, no fabrik:paused (resumed) |
Settle pass: live CI still genuinely stalled (no progress past the liveness dwell, or the R3/mergeable-state-blocked case) | Paused (re-escalated; no new comment) | fabrik:paused, fabrik:awaiting-input |
— |
wait_for_ci stage, awaiting CI, existing pause comment (from the old, pre-ADR-1410 timeout-labeled-as-failure defect), no fabrik:paused (resumed) |
Settle pass: live CI confirmed still failing | Not re-paused — ciFailure=true dispatches CI-fix reinvocation instead (ADR-1410, R3) |
fabrik:awaiting-ci (idempotent; already present) |
— |
wait_for_ci stage, awaiting CI, existing cycle-limit comment, no fabrik:paused (resumed), CIFixCycles at MaxCiFixCycles |
Settle pass: live CI still failing | Paused (re-escalated via pauseForCIFixCycleLimit’s reuse path; no new comment) |
fabrik:paused, fabrik:awaiting-input |
— |
References: §6.14 (Fix 2, corrected above), §6.14.1 (issue #1303, the backstop this fix modifies), ADR-1270: Awaiting-CI Settle Scan (the two-call label-swap race hasCIGatePauseComment was originally written for — its regression test, TestSettleAwaitingCIScan_RaceWithMainLoop_CycleLimitPause_NoDuplicateComment, is unaffected by this fix), ADR-1387: Closed Items Are Never Dispatched (structural precedent for a “zero remaining owners” defect), ADR-1408: CI-Gate Pause Comment Reuse on Resume, issue #1408
6.14.5 CI Gate: Liveness, Not Elapsed Time (issue #1410)
Trigger: CIWaitTimeout was a single wall clock applied uniformly to four structurally different CI states — check runs pending (alive and reporting), check runs failed (a verdict, not a wait), R3’s never-checked case, and a blocking mergeable_state with no check runs (both genuine liveness problems — no event is coming without a human). Only the last two are legitimately clock-worthy. Field evidence (verveguy/liminis-context-graph#342, 2026-08-04) showed the first misapplication in practice: fabrik:awaiting-ci anchored at 12:51:45Z, a rebase reinvocation consuming 15m46s of the 30-minute budget before pushing a fresh head at 13:07:31Z, an 18m04s CI suite starting at 13:07:40Z with only 14m05s left — Fabrik paused at 13:21:57Z, CI finished green at 13:25:44Z on a CLEAN/APPROVED/MERGEABLE PR. The failure row was its own defect: classifyCIFromCheckRuns’s CIWaitTimeout guard fired ahead of pending/failed classification, so a genuine failure occurring after the deadline was reported as a timeout — bypassing MaxCiFixCycles and the CI-fix reinvocation path, and posting a message claiming Fabrik “timed out waiting for checks to pass” when checks had in fact reported a verdict.
Fix — split the single clock into a verdict/liveness pair. See ADR-1410 for the full design and rationale; summary of the as-built mechanism:
- Verdicts never wait (R3).
classifyCIFromCheckRunsclassifies viagh.ClassifyCheckRunsfirst; onCheckRunsFailedit returnsciFailure=trueunconditionally, no clock.classifyCIFromRequiredContextsdrops its elapsed-time guard entirely and always returnsciFailure=trueonRequiredContextsFailed. Both route to the existingMaxCiFixCyclespath exactly as any fresh failure always has — see §6.4 rules 13/17/19 and §6.5.2, both unaffected by this change. - The pending branch gets a liveness dwell, not an elapsed one. A new
ciProgressStalledSince()helper (engine/ci.go) readsLinkedPRState.LastCIProgressAtfrom theitemstatestore;classifyCIFromCheckRunsescalates ((false, false, true)) only once that timestamp is both non-zero andCIWaitTimeoutor more in the past.LastCIProgressAtis stamped ininternal/itemstate/store.go:applyCheckRunCompleted, wheneverupsertCheckRunByIDreports actual content change (a new check-run ID, or an existing one’sStatus/Conclusiontransitioning — not on an identical duplicate observation); andPRHeadSHAUpdated’s existing SHA-change branch and pending-buffer drain (a fresh push resets CI entirely, which is progress in its own right — this is what removes #342’s dependence on when the wait started). All three check-run producers (RefreshCheckRunsLive,FetchCheckRuns’s cache-miss path,check_runwebhook deltas) already funnel throughapplyCheckRunCompleted, so one edit point covers all of them. classifyCIFromMergeableStateis unchanged. Its two cases (R3 never-checked, mergeable-state-blocked-with-no-checks) have no check-run signal to observe progress on at all, so they keep the originallabelAppliedAt-anchored elapsed dwell — the one classifier for which a plain clock remains the right instrument.- A new, separate
CIBackstopTimeout(240-minute default) boundssettleAwaitingCIScan’s unconditional backstop (R5) — see §6.14.1 Fix 2, now repointed at this setting instead ofCIWaitTimeout. It stays elapsed-time-based (FetchLabelAppliedAt, unchanged mechanism) since its purpose — bounding per-poll cost regardless of CI duration — is an elapsed-time question, deliberately decoupled from the liveness dwell’s own justification. - Merge-train’s blocking polls (
pollForMergeable,pollTrainCI,engine/merge_train.go) keep an elapsed bound, repointed atCIBackstopTimeout(R6). They are synchronous, single-goroutine blocking loops, not re-entrant poll-driven state — adopting liveness semantics would hold a worker goroutine open for a suite’s full duration. They already degrade gracefully on timeout (retry next merge-train cycle, no pause) rather than reproducing #342’s destructive pause; repointing at the much larger backstop removes the real remaining cost (a wasted trial-branch rebuild roughly every 30 minutes for a healthy-but-slow suite under the old, now-repurposedCIWaitTimeout). 5a. Suite-only holds are bounded the same way (#1822). A hold caused by an outstanding check suite (§6.4 Suite-aware completeness) isPRMergeUnsettled, claimed bycheckMergeabilityGatebeforecheckCIGate, so it is bounded byCIBackstopTimeout, not the liveness dwell (no check run is pending, soLastCIProgressAtnever advances for it).pauseForCITimeoutnames the outstanding suite and app. The train polls’ suite holds are likewise bounded by their existingCIBackstopTimeoutdeadline. pauseForMergeGroupStall/checkAutoMergeConvergence(§5.5 ②′,engine/merge_gate.go, ADR-058 D5) needed no code change — already liveness-shaped (fires only when merge-group CI has never reported), already anchored onlabelAppliedAtrather than the CI-await window’s own clock. It correctly keeps reading the repurposedCIWaitTimeoutdirectly.- No new pause function or comment text. The liveness-stall escalation reuses
pauseForCITimeout()verbatim (§6.14.4) — every caller of that function now represents a genuine stall, so its existing “timed out waiting for checks to pass” message stays truthful, and #1408’s episode-matching recoverability composes for free.
Architectural discovery — the pending-branch dwell is unreachable via the real async pipeline today. checkMergeabilityGate (§6.6, engine/merge_gate.go) unconditionally claims any PRMergeUnsettled classification, and settlePRMergeState (§6.4 rule 19) always returns PRMergeUnsettled for CheckRunsPending — there is no other path to that state. Since handleMergeAndCIGates returns immediately when the merge gate claims the item (§6.2 Handler 4), checkCIGate — and therefore classifyCIFromCheckRuns’s pending branch, item 2 above — is never reached for a merely-pending CI state via settleAwaitingCIScan/the catch-up loop. This was equally true of the old CIWaitTimeout guard before this issue (the pending-branch clock was already dead code from the pipeline’s perspective, reachable only via a direct checkCIGate call, as the TestCheckCIGate_Pending_* unit tests in engine/ci_test.go do). settleAwaitingCIScan’s backstop (§6.14.1 Fix 2) — not classifyCIFromCheckRuns’s inner guard — is and was the mechanism that actually escalated #342 in the field; TestSettleAwaitingCIScan_342Repro_SlowButHealthyCI_DoesNotPause (engine/ci_settle_test.go) reproduces the incident end-to-end through the backstop and confirms this, including a non-vacuous sub-test showing a 30-minute-equivalent CIBackstopTimeout does fire at #342’s own 34-elapsed-minute timeline. Item 2’s liveness dwell remains correct behavior for a direct checkCIGate caller and is defensive against a future change to the merge-gate claim priority, but item 4’s backstop resize is what actually resolves #342 for real traffic.
Consequence for §6.14.4 (R4). A resumed item (#1408) whose original pause was a genuine liveness stall, and which is still stalled on resume with no confirmed verdict, is not automatically re-escalated by the async gate for the same reason: the handler chain cannot re-derive “still stalled” for a pending-CI item the way it can for a confirmed-failing one. It falls back to being silently re-blocked by the merge gate, re-evaluated every poll, until CI produces a verdict or the backstop (independently anchored on labelAppliedAt, unaffected by the resume) fires again. This is a pre-existing property of the claim-priority architecture, not a regression introduced by this issue — the old elapsed-time guard had the identical unreachability — and closing it is out of this issue’s scope (Non-goals: no change to the conjunctive CI/review gate’s semantics, #895).
R8 — configuration compatibility. FABRIK_CI_WAIT_TIMEOUT/--ci-wait-timeout keep their name, flag, and 30-minute default (TestCiWaitTimeout_DefaultUnchanged pins the value; TestCheckCIGate_Failed_NeverTimesOut_RegardlessOfElapsedTime and its required-context sibling pin the behavior change). Only the meaning changes — cap on total CI wait → liveness-stall dwell — stated in this ADR, in docs/USER_GUIDE.md, and via a one-time, unconditional startup log line (logCIWaitTimeoutSemantics, engine/poll.go) so no operator discovers the change silently.
References: ADR-1410: CI Gate — Liveness, Not Elapsed Time, §6.14 (Fix 2 / backstop, repointed), §6.14.1 (issue #1303, the backstop’s origin), §6.14.4 (issue #1408, the recoverability model this reuses), §6.6.2 (merge-group stall precedent, ADR-058 D5), verveguy/liminis-context-graph#342 / PR #344 (field evidence)
6.14.6 CIBackstopTimeout Backstop: Anchor and Guards (issue #2059)
Trigger: On #2052 / PR #2053 (2026-10-05/06) a daemon clean stop paused an item mid-Validate while it carried fabrik:awaiting-ci. Eleven hours later a human’s steering comment resumed it (fabrik:paused + fabrik:awaiting-input removed). Three seconds later settleAwaitingCIScan’s backstop logged “exceeded CIBackstopTimeout … escalating” and re-paused it, in the same poll in which the resume had dispatched a comment worker; the worker then fixed the outstanding items under a pause that would not lift on its own. The #1408 “existing pause comment → defer” rule did not apply: the earlier pause was a clean-stop pause, so no CI-timeout pause comment existed for the episode.
Two defects. (1) The clock measured from fabrik:awaiting-ci’s applied time, which pausing and resuming never reset (ADR-1314), so it kept running through paused time and daemon downtime: an item paused overnight was already past the timeout on its first poll after the resume. (2) The scan had no in-flight check, so it could escalate an item whose worker had just been dispatched.
As-built. The backstop block (engine/ci_settle.go) now evaluates, cheapest first, and only escalates when every guard passes. The timeout value, the §6.14.4 deferral, the escalation message and the bound on per-poll cost are unchanged.
- Label anchor (unchanged).
time.Since(appliedAt) < ciBackstopTimeout()→ nothing to do. An unreadable or zeroappliedAtis also a no-op, as before. - Pause comment present (unchanged, §6.14.4). Defers to the live handler chain.
- R3 — worker in flight. If the store holds a worker for the item (
snap.Worker() != nil— set synchronously at dispatch, so it covers a worker started this poll by an earlier catch-up phase as well as one from a previous poll), the backstop is skipped for this poll and logged with the worker’s stage. Only the backstop is deferred; the handler chain still runs. The next poll re-evaluates, so the bound holds once the worker exits. - R2 — first evaluation since daemon start. The first time the scan evaluates an item in this process, the backstop does not escalate: the live-data chain (the live-confirmed check-run read, then
checkCIGateet al.) runs first, and the item is marked evaluated (Engine.ciBackstopLiveEvaluated) only once the chain has actually run — the earlycontinues (orphan column, deep-fetch failure, label changed after the fetch, escalation) do not count. If a gate silently claims the item on that pass, the result is “skip once, then enforce”. The marker is in memory by design: a restart clears it, so each awaiting-ci item gets one live evaluation per daemon start, and a genuinely stuck item is escalated one poll later than before. - R1 — the clock does not count paused time. The effective anchor is
max(appliedAt, latest fabrik:paused removal). The removal time comes fromFetchLabelRemovedAt(github/labels.go: the newestunlabeledevent for the label in GitHub’s issue event log, same paging andrestMaxPagesbound asFetchLabelAppliedAt), read live and never from the record-on-write label cache, so it survives restarts and sees removals made in the UI (the reasoning ofresumeAuthorised, ADR-1813). A failed read or an absent event falls back toappliedAtand still escalates — the opposite fail direction toresumeAuthorised; the backstop must never fail open into “never escalate”, or its per-poll cost bound is lost. A resume still inside the timeout skips the backstop (logged). - Resume-read memo. A recently resumed item would otherwise cost one events page-through per poll for up to the timeout.
Engine.ciBackstopResumeSeenmemoises the newest removal time read; a memoised resume still inside the timeout skips without a read. The memo is skip-only: it can only be older than the truth, so it can only produce a skip, and an escalation always follows a live read.
Sim fidelity. simgh.LabelEvent carries At (the injected clock at the mutation) and Sim.FetchLabelRemovedAt answers from that log (tests/sim/simgh/FIDELITY.md, “Label events”). tests/sim/ci_backstop_resume_test.go replays the #2052 sequence: awaiting-ci → clean-stop pause → restart → resume → comment worker → no re-pause → gate clears on green.
Sibling timeouts (#2064, ADR 2059). The other timeouts anchored on never-reset label timestamps use the same effectiveAnchor helper (engine/resume_anchor.go): the CIWaitTimeout classifiers classifyCIFromMergeableState (R3 never-checked and mergeable-state-blocked dwells, engine/ci.go, anchored on fabrik:awaiting-ci), the merge-queue stall dwell and ConvergenceBudget (engine/merge_gate.go, anchored on fabrik:auto-merge-enabled). Each measures from the later of its label anchor and the latest fabrik:paused removal, so paused time never counts. Each first tests the cheap label anchor and calls the helper (one FetchLabelRemovedAt read, memoised in the shared resume memo, tested against the caller’s own timeout) only once already over its timeout; an unreadable removal time falls back to the label anchor and still escalates. Operators, comparison operators (>=; > for ConvergenceBudget), disable-on-zero guards, timeout values and escalation messages are unchanged, and pauseForConvergenceFailed reports the elapsed time from the effective anchor. The classifyCIFromCheckRuns liveness stall uses in-memory LastCIProgressAt and is unaffected.
References: §6.14.1 (the backstop), §6.14.4 (the §6.14.4 deferral this keeps), §6.14.5 (CIBackstopTimeout), ADR-2059, ADR-1813, ADR-1314, #2052.
6.15 Closed-Item Validate Terminal Advance Settle Scan (ADR-1387)
Trigger: Field evidence (handarbeit/fabrik-test-alpha#4246) showed an issue closed out-of-band while parked at Validate — wait_for_ci: true, so handleStageComplete withholds stage:Validate:complete and applies fabrik:awaiting-ci instead (the conjunctive gate, §6.5) — was re-dispatched as a full Claude stage invocation every poll cycle, 87 times over ~14 hours. The loop composed from four individually-reasonable mechanisms: (1) the conjunctive gate defers stage:Validate:complete and applies fabrik:awaiting-ci as the interim suppressor; (2) cleanupClosedIssueTransientLabels (see R6 below) swept fabrik:awaiting-ci off the closed issue every poll, as ordinary transient-label hygiene; (3) the item was then left with neither the completion label (deferred) nor the suppressing label (swept) — still sitting at Validate; (4) the pre-ADR-1387 closed-issue admission guard (§2.8) admitted it anyway via a !stageIsGateChecked(stage) disjunct, so it was dispatched — returning to step 1. The loop only terminated by chance, when the model happened to emit FABRIK_NO_WORK_NEEDED instead of FABRIK_STAGE_COMPLETE.
Root cause: the settle-owner’s only feed was dispatch-admission-gated. Before ADR-1387, runValidatePRTerminalAdvance (§2.10, ADR-056 D2) — the settle-owner responsible for healing exactly this class of closed, paused, or awaiting-review Validate-stage item (the #874 class, commit 7311a14e) — was fed exclusively from deepFetchCandidates, which is itself filtered by itemMayNeedWork. The only way to make a closed item observable to the settle-owner was therefore to admit it to dispatch — conflating “let the settle-owner see this item” with “let this item receive a real Claude invocation.” 7311a14e widened admission to reach the settle-owner and, in doing so, also widened dispatch eligibility, since both consumers shared one gate.
Same failure class as §6.9/§6.10/§6.13/§6.14: a durable-marker/stranding class recovered by the shared, admission-gated catch-up path instead of a dedicated board.Items-sourced settle scan. settleClosedValidateAdvance gives the settle-owner a feed that is entirely independent of dispatch admission, mirroring settleAwaitingCIScan’s (§6.14, ADR-1270) “admission independence is the entire fix” shape — the same idiom applied to a different stranding class.
Code path: settleClosedValidateAdvance (engine/pr_terminal_advance.go; called from poll() in poll.go, immediately after runValidatePRTerminalAdvance).
Shared logic, split ownership. Both runValidatePRTerminalAdvance (§2.10, open items) and settleClosedValidateAdvance (closed items) delegate to the same extracted per-item function, advanceValidateTerminalItem(board, item, advancedItems) — the exact pre-ADR-1387 logic of runValidatePRTerminalAdvance, unchanged. This keeps ADR-057’s “single authoritative owner” contract intact in spirit: one union of healing logic, split only by feed (deepFetchCandidates vs. board.Items) and therefore by cost, not two independently-reasoning implementations that could drift apart. runValidatePRTerminalAdvance’s loop skips any item.IsClosed item (a redundant-but-explicit ownership boundary — admission no longer lets a closed item reach deepFetchCandidates at Validate in the first place, so this is unit-testable clarity, not load-bearing correctness) and settleClosedValidateAdvance’s loop skips any open item, so the two functions partition board.Items/deepFetchCandidates disjointly and can never both process the same item in the same poll.
No double-advance, by construction (R5). advancedItems[iKey] is checked before the live FetchLinkedPR call inside the shared per-item logic, and poll() is single-threaded/sequential, so even without the IsClosed partition above, whichever of the two callers ran second would find the item already marked and no-op. settleClosedItemsToDone (§6.11, ADR-064) excludes stage.Name == "Validate" — it defers closed Validate items to whichever function owns them, which only needed to actually be reachable independent of admission, which is exactly what this scan supplies. (An initial implementation of this fix left settleClosedItemsToDone’s exclusion unchanged at the broader stageIsGateChecked(stage), which turned out to need its own follow-up correction — see the note below.)
Cost: O(closed items currently sitting at Validate) — bounded by WIP at one late-pipeline stage, not board size, and structurally identical in kind to settleClosedItemsToDone’s and settleAwaitingCIScan’s existing unconditional per-poll board.Items scans. Calls only the lightweight FetchLinkedPR per candidate (never the expensive FetchItemDetails GraphQL deep-fetch), and only for items already known-closed from the shallow board snapshot — materially cheaper than what deepFetchCandidates was originally introduced to bound. No rate-limit concern identified.
R6 — cleanupClosedIssueTransientLabels no longer races the settle-owner. Before this fix, the per-poll transient-label sweep (engine/poll.go, #617) stripped fabrik:awaiting-ci (and fabrik:awaiting-review, fabrik:rebase-needed) off every closed issue unconditionally — including one sitting at Validate with its completion still deferred, which is exactly step 2 of the loop described above. The sweep now resolves each closed item’s stage and, when it is specifically stage.Name == "Validate" — not stageIsGateChecked(stage) generally — skips exactly fabrik:awaiting-ci, fabrik:awaiting-review, and fabrik:rebase-needed — the three labels the settle-owner clears itself, atomically, as part of a real merge/pause transition — while every other transient label (including fabrik:auto-merge-enabled, which per attemptMergeOnValidate’s design always coexists with stage:Validate:complete and so was never part of the stranding mechanism) continues to sweep unconditionally. Every closed item at any other column, gate-checked or not (e.g. a Review stage independently configured with wait_for_reviews: true, as in the shipped default stages/examples/review.yaml), is entirely unaffected — the sweep runs exactly as before there. This scoping is deliberate, not an oversight: the settle-owner pair only ever processes stage.Name == "Validate" (matching runValidatePRTerminalAdvance’s own pre-existing, unchanged hardcoding — see the settleClosedItemsToDone exclusion-narrowing note below); excluding a label at any other gate-checked stage from the sweep would strand it with no owner left to ever clear it. A first implementation of this fix used stageIsGateChecked(stage) here and was caught and corrected during Implement precisely because it would have created that stranding for a closed, incomplete Review-stage item.
fabrik:auto-merge-enabled-without-stage:Validate:complete is healed inline, not left to a delayed self-heal (caught in review on PR #1388, with the initial fix’s rationale itself corrected on a follow-up review pass). advanceValidateTerminalItem defers entirely to checkAutoMergeConvergence (Phase 1) whenever fabrik:auto-merge-enabled is present, on the assumption that attemptMergeOnValidate never applies the label before the gate has already cleared — true for both callers today. An initial implementation deferred unconditionally on the label alone and, when that assumption was violated (a labeling race, or a future caller of the label), only logged a warning before returning — claiming this “may permanently strand the item,” the same framing correctly used for the Review-stage strand documented above. That framing was itself wrong: fabrik:auto-merge-enabled is not among the labels cleanupClosedIssueTransientLabels withholds from its closed-item sweep (gateSettleOwnedTransientLabels, R6 above), so for a closed item the same poll’s later sweep step removes the label unconditionally, and the next poll’s call into this function falls through normally and heals it — a one-poll-cycle delay, not a permanent strand. (An open item has no such fallback, since the sweep only runs on closed issues — but this state is believed rare for either.) The fix itself is unaffected by that correction: it narrows the defer-to-Phase-1 skip to the invariant-respecting case only (fabrik:auto-merge-enabled and stage:Validate:complete both present) and, when the label is present without the completion label, logs the same warning but falls through into the ordinary merge-vs-pause flow below instead of returning — healing the item inline via the identical logic used for every other Validate terminal advance, rather than depending on sweep ordering and a wasted poll cycle (and closing the gap entirely for the open-item case, which has no sweep-based fallback at all).
advanceValidateTerminalItem intentionally hardcodes stage.Name == "Validate" rather than stageIsGateChecked(stage) — Fabrik’s own PR merge action (attemptMergeOnValidate) only ever runs as part of Validate completing, so the PR-fetch-and-decide logic (merged → fill labels/advance; closed-unmerged → pause) this function implements has no equivalent nuance to add for any other stage. This is unaffected by, and distinct from, settleClosedItemsToDone’s own exclusion below.
settleClosedItemsToDone (§6.11) excludes stage.Name == "Validate" specifically, not stageIsGateChecked(stage) generally (caught in review on PR #1388, after an initial implementation shipped the broader exclusion as a documented “known limitation”). Both shipped default stage templates, stages/examples/review.yaml and stages/examples/validate.yaml, set wait_for_reviews: true — so a deployment using the defaults genuinely has a second gate-checked stage (Review), not a hypothetical one. Under the broader stageIsGateChecked exclusion, a closed item stranded at Review with no stage:Review:complete was never dispatched (R1 held) but also never healed by any settle-owner — settleClosedItemsToDone skipped it (gate-checked), and the settle-owner pair skipped it too (not Validate). Zero remaining owners, a permanent silent strand — worse than the pre-ADR-1387 bug it replaced, which was at least loud and moving. Closing an abandoned or superseded issue mid-pipeline is ordinary operator behavior, not a rare edge case — it is exactly what produced the field evidence this section is built on. The fix needs no new reasoning about PR state: advanceClosedItemToDone (what this scan calls) never inspects the linked PR at all, it only moves the board column — exactly what it already does, unconditionally, for a closed item at Specify/Plan/Implement. Narrowing the exclusion to stage.Name == "Validate" simply stops incorrectly withholding that existing, already-correct behavior from Review. The per-poll “no settle-owner” warning this once required has been removed along with the condition that produced it.
Unresolved-PR polling cost, pre-existing and unchanged by this fix. For a closed item at Validate whose linked PR is neither merged nor closed (a human closed the issue without touching the PR — an unusual, out-of-band action, the same trigger class as the loop this fix targets), advanceValidateTerminalItem returns immediately with no state change, and settleClosedValidateAdvance re-evaluates that item every poll — one FetchLinkedPR call, indefinitely, until the PR resolves — with no timeout/escalation comparable to settleAwaitingCIScan’s CIWaitTimeout backstop (§6.14, ADR-1270). This exact no-escalation behavior already existed in the pre-ADR-1387 runValidatePRTerminalAdvance (verified against the pre-fix code); ADR-1387 does not introduce it, and only lowers its cost — pre-ADR-1387 the same item was also being fully re-dispatched to Claude every poll (the bug this fix targets), so the post-fix steady-state cost (one lightweight, read-only API call per poll) is strictly cheaper, not a new failure mode. A stuck-PR escalation path is unrelated to this issue’s R1–R7 and left out of scope.
State transitions:
| Before | Trigger | After | Labels Added | Labels Removed |
|---|---|---|---|---|
Validate, closed, no stage:Validate:complete, any gate label or none |
PR confirmed merged | Next stage (Done — stages.NextStage skips Holding stages like Queued per issue #1072, §1.3), closed |
stage:<gate-checked>:complete for each missing gate-checked stage up to and including Validate |
fabrik:awaiting-ci, fabrik:awaiting-review, fabrik:rebase-needed, fabrik:paused, fabrik:awaiting-input (whichever present) |
| Validate, closed, PR closed without merging, not already paused | Settle pass observes closed-unmerged PR | Unchanged column | fabrik:paused, fabrik:awaiting-input |
fabrik:awaiting-ci, fabrik:awaiting-review, fabrik:rebase-needed (whichever present — pauseForPRClosedNotMerged clears all three, not just fabrik:awaiting-ci; this is the only remaining chance to clear them once paused, since the sweep withholds them at Validate) |
| Validate, closed, PR closed without merging, already paused | Settle pass evaluated | No change (avoids duplicate comment) | — | — |
Validate, closed, fabrik:auto-merge-enabled present with stage:Validate:complete |
Settle pass evaluated | No change (owned exclusively by checkAutoMergeConvergence, Phase 1) |
— | — |
Validate, closed, fabrik:auto-merge-enabled present without stage:Validate:complete (invariant violation) |
PR confirmed merged or closed-unmerged | Healed via the normal merge-vs-pause flow above (same rows as the no-auto-merge-enabled cases) |
— | — |
Validate, closed, already in advancedItems this poll |
Settle pass evaluated | No change (dedup, R5) | — | — |
| Validate, open (any state) | Settle pass evaluated | No change (owned exclusively by runValidatePRTerminalAdvance) |
— | — |
R1 follow-up: comment-driven dispatch on a closed, stage-complete item (caught by Pruefer review, ADR-1387). The closed-issue admission guard’s retained exception — a closed item carrying stage:<X>:complete stays admitted so the catch-up loop can still act on it — interacted with a separate, older fast path: itemNeedsWork’s “new comments are always worth processing (even on completed stages)” check, and its mirror in processItem, had no item.IsClosed guard. A closed, stage-complete item that received a fresh comment was therefore still routed to processComments, a real Claude invocation — a narrower, pre-existing instance of the same class of bug this section fixes, reachable independently of the CI-gate/fabrik:awaiting-ci mechanism and predating commit 7311a14e. Both fast paths now skip when item.IsClosed: in itemNeedsWork, simply not short-circuiting is sufficient, since the “already completed this stage” check immediately below it already rejects a closed+stage:complete item once the fast path stops pre-empting it; in processItem the added guard is redundant-but-explicit, since itemNeedsWork’s gate already prevents processItem from being invoked at all for this case. A human comment on a closed, completed issue is no longer actionable by Fabrik — consistent with “a closed issue has no computable work left” applying to comment processing exactly as it applies to stage re-invocation.
References: ADR-056: Consolidate Convergence Gate Recovery (D2, the settle-owner’s original single-authoritative-owner contract), ADR-057: Single-Owner Validate PR Terminal Advance, ADR-064: Closed-Item-At-Any-Stage Advance To Done (§6.11, the sibling scan this one defers to/from), ADR-1270: Awaiting-CI Settle Scan (§6.14, the direct template for “board-sourced, admission-independent settle scan for a specific stranding class”), ADR-1387: Closed Items Are Never Dispatched, commit 7311a14e (the regression point whose intent — #874-class healing — this fix preserves, only changing its mechanism), issue #617 (the transient-label sweep whose interaction with the conjunctive gate is closed by R6 above)
6.16 Queued Review-Finding Ejection (Settle Scan)
Channel events (#1968):
ejectMemberandejectQueuedMemberForCommentsemitmerge-train-ejected; see §7.14.
Trigger: Issue #1208. While a merge-train member sits in the Queued holding stage awaiting a batch, Fabrik processes no PR feedback at all — the HoldingStage exclusion shared by itemMayNeedWork (§6.14’s admission-independence framing applies equally here: deepFetchCandidates never even deep-fetches a Queued item, so reviewThreads/LinkedPRReviewThreadComments are never populated) and the main catch-up loop’s own per-item admission gate blacks out review-reinvoke and comment processing for as long as the item remains Queued. This window is not brief: batch formation, trial-branch assembly, CI, and possibly inline conflict resolution and bisection can span tens of minutes, and a live merge-train worker can block synchronously inside pollTrainCI for up to CIWaitTimeout (default 30 min) — precisely the window in which a self-submitting review bot (Pruefer) is most likely to post a fresh finding, since it re-reviews on every new head SHA and train activity produces new SHAs. #1207 (§ADR-1207, the sibling Validate-side race) narrows this issue’s practical exposure to the mid-flight case — a finding that lands after the member is already in Queued — since a member with unresolved threads at the moment Validate would hand off to Queued is now guarded there, but the code path this section fixes has no such guard of its own.
Same failure class as §6.9/§6.10/§6.13/§6.14/§6.15: a durable condition invisible to the shared, admission-gated catch-up path, recovered by a dedicated board.Items-sourced settle scan. settleQueuedReviewFindings is the sixth instance of the ADR-1270 pattern. Unlike those five siblings, this scan does not merely make an already-evaluable gate reachable — it introduces a new ejection trigger on top of the pre-existing ejectMember mechanism (ADR-059), because a Queued member cannot address a review finding in place: pushing a fix would change the member’s head SHA mid-batch, invalidating a trial branch the train may have already assembled, CI’d, or bisected.
Code path: settleQueuedReviewFindings (engine/queued_review_settle.go; called from poll() in poll.go, immediately after handleMergeTrainBatch, gated on merge_train: on). Review bodies (#1953 R2): detection (queuedReviewFindings) is current-head thread comments plus unaddressed review bodies (a body-only COMMENTED review, the #616 shape), routed through the same direct/pending eject. A body-read error skips bodies for that poll only. Arrival before admission needs no admission-time gate (admitTrainMembers stays fail-open): a member flagged here is ejected directly when no worker holds it, or consumed by the worker at its first applyPendingReviewEjects checkpoint ahead of any trial or landing.
Catch-up recognition (#2044, §6.30). queuedReviewFindings passes the thread comments and reviews through dropCatchUpFeedback before building findings, so a bot review of a pure merge-train catch-up head neither flags a pending eject nor ejects the member directly — the same predicate unprocessedFeedback applies at the landing gate, so the two cannot disagree. A review of a conflict-edited catch-up is actionable as usual.
Interaction with the own-PR CI admission gate (§6.25, #1821). A pending eject signal for a member that the fresh-batch admission gate then defers for confirmed-red own-PR CI is consumed (takePendingReviewEject) and logged by the deferral, so the member is never double-routed or double-commented and the stale signal cannot later fire on a re-queued member. If the deferral’s reroute fails, the signal is left intact.
Detection. The scan groups the current Queued snapshot by owner/repo via groupQueuedByRepoAndBase — the identical holding-stage-column filter and closed/fabrik:paused exclusion routeQueuedGroup (§”Engine Selection” above) already applies, so a poisoner already excluded from batch formation is equally excluded here. For each remaining member it skips native-merge-queue items (MergeQueue != "off" && LinkedPRIsMergeQueueEnabled, or fabrik:auto-merge-enabled already present) — ejectMember/MaxMergeTrainEjections have no meaning for a member GitHub’s own queue is landing, mirroring routeQueuedGroup’s FR-3 precedence exactly — then calls FetchItemDetails itself (the deep-fetch deepFetchCandidates never performs for a Queued item) and checks currentHeadReviewThreadComments — the ADR-1207-canonical, current-head-scoped primitive (not raw buildReviewThreadComments), so a thread anchored to a commit the PR has since moved past never triggers a spurious ejection.
Direct-eject vs. pending-signal split (the concurrency design). The three pre-existing ejectMember call sites all run from inside the merge-train worker goroutine that owns the batch state, so there is no concurrent-mutation hazard: the same goroutine that decides to eject also owns the slice being mutated. This scan’s trigger is different in kind — it fires from the poll loop, which can run concurrently with a worker blocked for up to CIWaitTimeout inside pollTrainCI. Reaching into that goroutine’s own in-memory batch slice from outside it would race the worker’s own assemble/validate/land sequence. The scan therefore branches on batch membership, not repo-level activity: mergeTrainBatchMembers(repoKey) returns the in-flight worker’s batchNumbers — the issue-number set it was dispatched with, snapshotted once in dispatchMergeTrainWorker and immutable afterward (worker membership only ever shrinks via ejection/landing, never grows).
- Member’s issue number is in
batchNumbers(or no worker is registered for the repo at all): nothing else could be touching this specific member’s state, so the scan ejects it directly viaejectQueuedMemberForReviewFindings. - A worker is in flight and the member’s issue number IS in its
batchNumbers: the scan leaves a mutex-guarded pending-eject signal instead (markPendingReviewEject, keyedowner/repo→ issue number → finding count) — mirroring the pre-existing runaway guard’s own “poll writes a signal, worker consumes it at a checkpoint” shape (isRunawayTripped/mergeTrainTrialsMu, described under “Runaway Guard” above). The worker itself applies the signal from inside its own goroutine, at three checkpoints (applyPendingReviewEjects): immediately afterassembleAndValidatereturns inrunMergeTrainWorker’s re-form loop (right after the existing Hook 1 runaway-guard check); insidelandOneAtATime’s per-singleton loop, right before its green/red/pending outcome switch; and insidelandGreenBatch’s own main-moved rebase-and-revalidate loop, right after each rebase cycle’s re-validate returns — this third checkpoint exists because that loop can itself spend a full combined-Validate wait without ever returning control torunMergeTrainWorker’s re-form loop, so a finding arriving mid-rebase would otherwise reachlandMergeTrainBatchunchecked. At all three checkpoints, a flagged member’s current trial is discarded regardless of its own CI result — a green trial containing a flagged member must never reachlandMergeTrainBatch. The re-form loop andlandOneAtATimecontinue with the reduced membership (an empty remainder falls through to the re-form loop’s existing zero-survivors return; no special-casing needed);landGreenBatch’s rebase loop instead discards the whole trial and returns outright, leaving its non-flagged survivors untouched inQueuedto re-form fresh on a future poll’sdispatchMergeTrainWorkercall, rather than threading a resume-with-reduced-membership path back into that loop.
Why batch membership, not mergeTrainWorkerActive, decides the branch. A worker is always dispatched with its batch already capped to effectiveMaxBatchSize (default 5, capBatch/FR-4) — a Queued member beyond that cap is never part of any worker’s in-memory batch for as long as that worker runs. An earlier version of this scan branched on mergeTrainWorkerActive(repoKey) alone (any worker active for the repo → always the pending-signal path); found in review (issue #1208) to leave such an overflow member’s pending-eject signal permanently unconsumed, since none of the three checkpoints above ever iterate an issue number outside the dispatched batch — silently reproducing the very multi-poll-cycle blackout this settle scan exists to close, just for members outside the front of the queue. Checking batchNumbers membership directly closes this: an overflow member is ejected immediately, exactly as safely as the no-worker-at-all case, since a worker never looks at, fetches, or mutates an item outside the batch it was dispatched with.
This is checkpoint-based, not continuous preemption — a pending eject flagged mid-bisection is only applied at the next outer-loop checkpoint, the same granularity the runaway guard already has.
Reroute-then-eject ordering. ejectQueuedMemberForReviewFindings calls rerouteQueuedMemberOffHolding before posting the ejection comment or incrementing the MaxMergeTrainEjections counter. rerouteQueuedMemberOffHolding moves the board Status to stageBeforeHolding — the non-Unmanaged stage with the highest Order strictly less than the holding stage’s (derived structurally, not hardcoded to "Validate", so a custom stage config whose pre-Queued stage isn’t literally named Validate is still handled) — via a plain UpdateProjectItemStatus call, with the same cache write-through/SelfWriteObserved/webhook-echo bookkeeping advanceToQueued already uses for the reverse move. If the reroute fails (status-field metadata unavailable, or the mutation itself errors), ejectQueuedMemberForReviewFindings returns immediately — no comment is posted, no counter is incremented — so a transient board-mutation failure can never produce a duplicate ejection comment or double-count toward MaxMergeTrainEjections; the settle scan simply re-detects the same still-unresolved thread on a member still sitting in Queued and retries the whole operation next poll.
Distinct ejection wording (stayInQueue). ejectMember gained a stayInQueue bool parameter rather than a new sibling function, so the existing comment-posting/counter/pause-at-cap logic is reused unconditionally, exactly as the issue’s Requirements ask. All five pre-existing call sites (unfetchable linked PR, missing head SHA, unresolvable conflict, bisection-isolated poisoner, singleton-fails-in-isolation) pass stayInQueue=true and keep their unchanged closing sentence (“This issue remains in the Queued column and will be retried in a future train with a different composition.”). This new cause passes stayInQueue=false, producing the opposite, textually distinguishable sentence: “This issue has left the Queued column so the unresolved review-thread finding above can be addressed via the normal review pipeline. Once addressed and Validate completes again, it will re-queue and join a later batch.” — so an operator can tell from the ejection comment alone which of the four causes fired.
MaxReviewCycles composes across the eject/re-queue cycle “for free.” The reroute is a plain status move only — it never touches stage:Validate:complete (already present from the original Validate completion, and never removed by advanceToQueued/advanceToNextStage on the Queued visit) and never calls EngineCyclesCleared (the only mutation that zeroes ReviewCycles and, as of #1518/ADR-1518, ReviewBlockedCycles — otherwise reachable only via the manual-intervention paths in §7). Because both are left untouched, the very next poll admits the rerouted item to Phase 1 with hasComplete=true; handleReviewGate — completely unmodified by this issue, and, per #1518’s own Research pass, requiring no merge-train-specific change either — finds the same still-unresolved thread comment (dedup is keyed on the 🚀 reaction / ProcessedComments, not on ejection state) and dispatches through the existing dispatchWithCycleLimit/max(ReviewCycles, ReviewBlockedCycles) gate (§6.2 Handler 2), keyed by stage name exactly as ordinary CHANGES_REQUESTED reinvocation is. A member that keeps attracting findings therefore eventually escalates via pauseForReviewCycleLimit, the same terminal fallback any other Validate-stage item reaches — rather than oscillating Queued ↔ Validate unbounded. MaxMergeTrainEjections (a separate, pre-existing bound reused via ejectMember) fires independently of ReviewCycles/ReviewBlockedCycles from the same repeated-ejection counter every other ejectMember cause already increments — a member could in principle hit the train-ejection pause cap purely from repeated review findings before MaxReviewCycles ever fires; the two bounds are not substitutes for each other.
Scope: mid-flight ejection only, no admission-time filtering. A member with an unresolved thread is still admitted into a forming batch; it is ejected once this scan (or a worker checkpoint) observes the finding. groupQueuedByRepoAndBase/routeQueuedGroup’s batch-composition logic is untouched.
Comment cause: an unprocessed human comment ejects the member too (#1863, ADR-1863). Review-thread findings were the only feedback this scan noticed; an ordinary issue or PR comment posted while the member sat Queued was ignored until the member landed (where the #1862 post-merge guard declines it). The same scan now also detects an unprocessed human comment and ejects the member so the ordinary comment path can act on it.
- Detection:
filterHuman(findNewComments(item))on the item the scan already deep-fetched (FetchItemDetailsreturns the issue’s own comments plus every linked PR’s, with reactions — no extra API call).findNewCommentsis the one “unprocessed” predicate shared with the Validate landing gate (commentGateBlocksLanding, §2.2/§6.6.6) and the non-Validate advance guard; it already excludes store-watermarked,🏭 **Fabrik-prefixed, 🚀’d and bot-service-notice comments. It does not exclude every bot author, so the human restriction (filterHuman, the ADR-069 resume gate’s helper) is layered on top — never a second predicate. A bot or engine comment never ejects a Queued member. This is the one place the scan is deliberately narrower than the landing gate, and only in what triggers an eject, not in what counts as unprocessed. - Precedence. The comment branch (
settleQueuedCommentCause) runs only when the review-thread check found nothing. A member with both causes takes the review-finding path unchanged, since the ordinary path handles both in one invocation. - Same direct/pending routing as review findings. A member outside a live worker’s dispatched batch is ejected directly by
ejectQueuedMemberForComments. A member inside it gets a pending signal — a parallelqueuedCommentEjectsmap (markPendingCommentEject/takePendingCommentEject/clearPendingCommentEject, guarded by the samequeuedReviewEjectsMu, keyed by bareowner/repoper ADR-1648) — thatapplyPendingReviewEjectsconsumes at the same checkpoints. It takes both signals per member in one pass, so a member is consumed once; a review-finding signal wins and drops any comment signal. The return contract is unchanged (ejectedCount > 0→ the caller discards the trial and the rest of the batch continues). An in-batch member that is no longer flagged has its stale comment signal cleared by the scan, anddeferRedMember(§6.25) drops it on a CI deferral, so it cannot fire later on a re-queued member. - Not train churn.
ejectQueuedMemberForCommentsdoes not go throughejectMember. It callsrerouteQueuedMemberOffHoldingfirst (reroute-before-side-effects — on failure nothing is posted and the next poll retries), then posts one locally composed comment headed🏭 **Fabrik merge-train — ejected (unprocessed comment)**(the🏭 **Fabrikprefix keeps it from ever being re-flagged, including in PAT mode where Fabrik posts as the operator’s login). It never touchesmergeTrainEjectionCounts, never pauses, and does no label mutation,EngineCyclesClearedorReviewCycleschange (§5 of ADR-1208: a plain status move). UnlikeejectRedSingleton(ADR-1545) it need not pause: the unprocessed comment is itself the persistent signal the ordinary path re-detects. Prior genuine ejections keep counting — this path never clears the counter. - Pickup and re-queue. The reroute raises
StatusChanged, so the next poll admits the member at Validate;itemNeedsWorkadmits it viafindNewCommentsandprocessItemrunsprocessComments.stage:Validate:completesurvives the reroute (beginStageRework/endStageReworkclear and restore it around the run), and the member re-queues throughhandleStageCompleteorrunCatchUpPhase2oncecommentGateBlocksLandingclears. The scan runs after dispatch, so a member ejected in poll N is first dispatched in poll N+1. MaxReviewCyclesdoes not bound this cause.ReviewCycleIncrementedhas one production writer,handleReviewGate’s dispatch; the ordinary comment path never increments or resetsReviewCycles. The bound is structural instead: (1) the eject predicate is a subset of the landing gate’s, so the member cannot re-queue while any comment is unprocessed — a second eject needs a new human comment after re-queue, each cycle consuming a distinct comment, so there is no autonomous eject → Validate → re-queue → eject loop; (2) a comment the ordinary path cannot process stays unprocessed at Validate (not Queued, where this scan cannot see it) and the comment breakers pause it there (MaxNoOpCommentCycles,MaxCommentCyclesPerWindow, §4.6), lifted only by a later human comment (ADR-1813).- Accepted edge: integration-PR comments. A merge-train integration PR carries
Closes #Nfor each member; a human comment on it can appear in every such member’sitem.Commentsand eject the whole batch. The comment’sFromPRis not a reliable discriminator, and a second filter would fork the predicate, so this is accepted; #1862’s post-merge guard is the backstop for the late-arrival window (a comment landing after the batch is committed to landing is out of scope here). - Not covered: native-merge-queue members and
fabrik:auto-merge-enabledmembers (skipped, as above); closed andfabrik:pausedmembers (excluded, as above).
State transitions:
| Before | Trigger | After | Labels/State Added | Labels/State Removed |
|---|---|---|---|---|
| Queued, no unresolved review-thread findings | Settle pass evaluated | Unchanged | — | — |
Queued, unresolved finding(s), member’s issue number NOT in any live worker’s batchNumbers (no worker for the repo, or a worker is active but this member is beyond the batch cap) |
Settle pass detects finding(s) | Rerouted to stageBeforeHolding (e.g. Validate); ejection counter incremented |
Ejection comment (stayInQueue=false wording) |
— (stage:Validate:complete and ReviewCycles deliberately untouched) |
Queued, unresolved finding(s), member’s issue number IS in the live worker’s batchNumbers |
Settle pass detects finding(s) | Unchanged this poll — pending-eject signal recorded | — | — |
| Run holds a pending-eject signal for a member | Checkpoint reached (the run state machine’s verdict step for the main trial, the one-at-a-time per-singleton step, or landGreenBatch’s main-moved rebase loop) |
Trial discarded regardless of CI result; member rerouted + ejected exactly as the direct-eject row above; the re-form loop and landOneAtATime continue with the reduced membership, while landGreenBatch discards the whole trial and returns (non-flagged survivors re-form on a future poll) |
Ejection comment (stayInQueue=false wording) |
— |
Queued, no review findings, unprocessed human comment(s), member NOT in any live worker’s batchNumbers (#1863) |
Settle pass detects filterHuman(findNewComments) non-empty |
Rerouted to stageBeforeHolding; no ejection count, no pause |
Comment-eject comment (ejected (unprocessed comment)) |
— (stage:Validate:complete and ReviewCycles untouched) |
Queued, no review findings, unprocessed human comment(s), member IS in the live worker’s batchNumbers (#1863) |
Settle pass detects them | Unchanged this poll — comment pending-eject signal recorded | — | — |
| Worker holds a pending comment-eject signal for a member (#1863) | Same checkpoints as the review-finding row above | Trial discarded; member rerouted exactly as the direct row (no count, no pause); the rest of the batch continues | Comment-eject comment | — |
| Queued, only bot/engine/🚀’d/watermarked comments (#1863) | Settle pass evaluated | Unchanged — never ejected | — | — |
| Comment-ejected member, comment processed, Validate completes again (#1863) | handleStageComplete / runCatchUpPhase2 |
Back in Queued, eligible for a later batch | — | — |
Rerouted member reaches handleReviewGate on the next poll, max(ReviewCycles, ReviewBlockedCycles) < MaxReviewCycles |
Still-unresolved thread comment found | Review reinvoke dispatched (unmodified path) | — | — |
Rerouted member reaches handleReviewGate, max(ReviewCycles, ReviewBlockedCycles) >= MaxReviewCycles |
Still-unresolved thread comment found | Paused (pauseForReviewCycleLimit, unmodified path) |
fabrik:paused, fabrik:awaiting-input |
— |
Member ejected for this cause MaxMergeTrainEjections consecutive times (independent of ReviewCycles) |
Ejection counter reaches cap | Paused | fabrik:paused, fabrik:awaiting-input |
— |
| Member addresses the finding, Validate completes again | advanceToQueued |
Back in Queued, eligible for a later batch | stage:Validate:complete (already present, unchanged) |
— |
References: ADR-1208: Queued Review-Finding Ejection, ADR-059: Internal Merge Train (ejectMember’s original three causes, the runaway guard’s two-hook precedent this issue’s pending-eject signal mirrors), ADR-1420: Merge-Train Ejection Diagnostics (diag/otherMembers contract — this new cause passes nil, nil, consistent with the three pre-1420 call sites), ADR-1270: Awaiting-CI Settle Scan (the direct template for this settle scan’s admission-independence shape), ADR-1207: Yolo-Merge Review-Thread Guards (the sibling Validate-side race; establishes currentHeadReviewThreadComments as the canonical detection primitive this issue reuses), ADR-067 (finishTrain as the single clear point for mergeTrainInFlight/Store.ExitRepoWorker; the worker-checkpoint early-return paths added by this issue still route through it), issue #1208; ADR-1863: Queued Comment Ejection (the comment cause), issue #1863
6.17 Terminal-Advance Escalation and Settle Scan (ADR-1422)
Trigger: Issue #1422 (reported by @bdueck in #1082). A terminal advance (advanceToNextStage, called from advanceValidateTerminalItem’s merged-PR path and advanceConvergedPRToDone) that cannot resolve its target Status option — most commonly because the intermediate or Done column named by stages.NextStage has no matching option on the project board — previously only logged a warning and returned. The issue stayed open with every stage complete and a merged, green PR: nothing red, nothing paused, nothing requesting attention. Because native dependency edges clear on close, not on merge, every downstream issue that listed the stranded issue as a blocker kept holding fabrik:blocked, waiting on work that had already shipped. In the reported incident, three merged issues stranded five downstream slices, and the stall was found only by reading fabrik.log directly, after it had been in effect for some time — the most expensive shape of stall Fabrik can produce, because the natural operator response to “quietly working” is to wait longer.
Same failure class as §6.9/§6.10/§6.13/§6.14/§6.15/§6.16: a durable stranding condition invisible to the operator and only fragile-at-best self-healing via the shared, admission-gated catch-up path. settleAwaitingAdvanceScan is the seventh instance of the ADR-1270 pattern. Research (see the R6 audit below) found this call site was not, in fact, never retried — advanceValidateTerminalItem’s closed-item half is already board.Items-sourced via settleClosedValidateAdvance (§6.15, ADR-1387), and advanceConvergedPRToDone’s admission-gated path mostly kept retrying as long as stage:Validate:complete stayed present. The two missing pieces were visibility (R1 — nothing signaled the stall to the operator) and boundedness (R3 — an admission-gated retry has no escalation path when something else in the pipeline starves it on a given poll).
Durable marker (fabrik:awaiting-advance), written only on failure — mirrors fabrik:awaiting-close (§6.13), not fabrik:awaiting-done (§6.8). advanceToNextStage is the last mutation at both call sites — by the time it can fail, every other side effect (gate-label clearing, completion-label filling, fabrik:auto-merge-enabled removal) has already landed. There is nothing left to redo except the status move itself, so a generic retry of the bare advanceToNextStage call is both correct and sufficient — no re-derivation of “which caller failed” is needed, since stages.FindStage(e.cfg.Stages, item.Status) always resolves back to the same stage the original caller had (the move is exactly what didn’t happen).
One shared label/counter/scan for both call sites, not two. Both failures are the identical “final board-Status move failed” shape with identical recovery, so a single recordAdvanceOutcome wrapper — routed through by both advanceValidateTerminalItem and advanceConvergedPRToDone in place of calling advanceToNextStage directly — avoids duplicating the label/comment/escalation machinery for no behavioral difference.
First-failure comment, gated on the label’s own absence (R1, R5). Unlike fabrik:awaiting-close (which only comments at escalation), markAdvanceFailureOutstanding posts a one-time explanatory comment on the very first failure — naming the failing stage and embedding the underlying error verbatim (react=false, mirroring handleUsageLimitExit’s fabrik:claude-limit shape, §7.3). For the “no status option” case, advanceToNextStage’s own error text already names the missing option and the options that do exist (no status option %q found on project board (available: %v)), so no new typed-error plumbing is needed to satisfy R1. Every failing pass — first or repeat — still counts toward the retry budget via recordSettleRetry, but only the absent→present transition ever posts a comment, so repeated failures in the same episode never produce more than one (R5).
No dispatch-suppression wiring, no transientLifecycleLabels entry — mirrors fabrik:awaiting-close’s reasoning exactly (§6.13). By the time advanceToNextStage can fail at either call site, stage:<X>:complete (or the gate-checked completion labels advanceValidateTerminalItem fills first) is already present, so ordinary admission already treats the item as stage-complete-and-idle. Adding fabrik:awaiting-advance to itemMayNeedWork/itemNeedsWork would be inert (nothing left to suppress); adding it to transientLifecycleLabels would let the closed-issue sweep race the settle scan, reproducing the exact class of bug ADR-1387 R6 (§6.15) closed for fabrik:awaiting-ci/fabrik:awaiting-review/fabrik:rebase-needed at Validate.
Code path: advanceValidateTerminalItem / advanceConvergedPRToDone → recordAdvanceOutcome → markAdvanceFailureOutstanding (writes the marker + first-failure comment on failure) — and, on retry, settleAwaitingAdvanceScan → recordAdvanceOutcome (all in engine/advance_settle.go; settleAwaitingAdvanceScan is called from poll() in poll.go) directly.
Retry-owner: settleAwaitingAdvanceScan (engine/advance_settle.go; called from poll() in poll.go, immediately after settleClosedValidateAdvance). Runs unconditionally once per poll, iterating the raw board.Items — not deepFetchCandidates — for the same reason as every prior settle scan in this family: a stranded item’s stage is already complete, and admission does not reliably see it (this is exactly the fragility Research identified for advanceConvergedPRToDone’s pre-fix path). For every item carrying fabrik:awaiting-advance and not fabrik:paused, not already in this poll’s advancedItems dedup set, and resolving to a real configured stage via stages.FindStage(e.cfg.Stages, item.Status), it calls recordAdvanceOutcome again.
Shares the poll’s advancedItems dedup map with runValidatePRTerminalAdvance/settleClosedValidateAdvance (§6.15). For a closed item stuck at Validate, that pair already retries advanceToNextStage unconditionally every poll and, running earlier in the same poll() pass, will already have set advancedItems[iKey] = true before this scan reaches it — so this scan is a harmless, correctly-skipped no-op there. It is the exclusive retry-owner only for the two genuine gaps: an open item admission-gated out of runValidatePRTerminalAdvance’s deepFetchCandidates source, and advanceConvergedPRToDone’s path, reachable only via the Phase 1 catch-up handler chain and otherwise unowned on any poll where that chain doesn’t reach it.
Retry counting and escalation (R3). Every failing pass — inside markAdvanceFailureOutstanding, called from both the original call sites and the settle scan’s retries — calls recordSettleRetry, keyed by the dedicated constant "__awaiting_advance__" (same double-underscore-wrapped, YAML-unrepresentable shape as the rest of this family). MaxRetries <= 0 means unlimited retries, never escalate (same guard every settle scan in this family shares). Once Attempts("__awaiting_advance__") >= e.cfg.MaxRetries, escalateAwaitingAdvanceFailure fires: adds fabrik:paused and posts an explanatory comment naming the attempt count with the manual-fix instruction (check every stage name has a matching board Status option, then remove fabrik:paused) — but, unlike every other escalation in this family, does not remove fabrik:awaiting-advance (see below).
Marker persists through escalation — the one deliberate deviation from this family’s shared shape (Pruefer, PR #1469 review, second round). Every other settle scan in this family (fabrik:awaiting-done, fabrik:awaiting-member-close, fabrik:awaiting-close, etc.) removes its marker label on escalation via the shared escalateSettle helper, whose own doc comment justifies this with “dispatch/retry suppression is no longer needed once fabrik:paused takes over.” That holds for their direct call sites only because each one unconditionally re-creates the marker on its very next relevant poll regardless of pause state (e.g. closeIssueIfNonDefaultBase, §6.13, has no fabrik:paused guard at all). advanceConvergedPRToDone (merge_gate.go) does not have this property: removing fabrik:auto-merge-enabled up front structurally prevents checkAutoMergeConvergence from ever re-entering it, so it fires at most once per episode. For an item whose only driver is settleAwaitingAdvanceScan (i.e. this call site’s items specifically), removing the marker at escalation would permanently strand it — nothing would ever re-create it, even after an operator fixes the board and removes fabrik:paused alone, which the escalation comment’s own recovery instruction promises is sufficient. escalateAwaitingAdvanceFailure therefore does not call the shared escalateSettle helper; it inlines the same pause/comment/EnginePaused steps without the marker removal. This is safe: settleAwaitingAdvanceScan’s own admission guard already skips any item carrying fabrik:paused, so the marker sitting alongside fabrik:paused is inert until the pause is lifted.
Counter reset on unpause, in both retry owners. Because the marker (and hence Attempts("__awaiting_advance__"), already at or above MaxRetries) survives escalation, both awaitingAdvanceStuckOrReset (called by advanceValidateTerminalItem before it touches anything, so it can skip the whole branch while still paused) and awaitingAdvanceResetIfUnpaused (called by settleAwaitingAdvanceScan directly, whose own pre-filter already excludes paused items) reset the counter via StageRetryCleared the moment they observe the item is no longer paused — mirroring clearFailedStage’s reset, scoped to this synthetic retry key. Without this, a retry issued right after a manual unpause would immediately re-escalate on its very first subsequent failure instead of getting a fresh MaxRetries budget (Pruefer’s first-round second finding, which turned out to apply to both retry owners, not just the direct call sites).
Marker clearing. fabrik:awaiting-advance is removed in exactly one place: clearAwaitingAdvanceMarker, on a fully successful recordAdvanceOutcome retry pass. Adding the missing board option is sufficient to unstick the item on the very next poll — no engine restart, no manual re-dispatch, and (per the two points above) no manual re-labeling either, even for an item that was previously escalated and paused (R2, Acceptance #2).
R6 audit of the sibling warn-and-return call sites named in #1422’s Problem section:
engine/no_work_needed_settle.go:178— already fully covered, not a gap. This line is insidesettleNoWorkNeededitself (owned byfabrik:awaiting-done, §6.8, ADR-060), which already callsrecordNoWorkNeededRetry→ bounded retry →escalateNoWorkNeededFailure(pause + comment) on this exact failure. The issue body’s “warn, no escalation” characterization was stale by the time of this fix.engine/closed_item_advance_settle.go:76— already covered, deliberately unbounded by design (ADR-064).settleClosedItemsToDonealready retries every poll, sourced fromboard.Items, independent of dispatch admission — it already has an owner in the sense R2 cares about.MaxRetries-bounded escalation was deliberately not added there: ADR-064’s own rationale (“no marker to lose or leak… a bare retry-forever loop is sufficient”) is a different, and equally valid, terminal shape for that specific call site, since it never writes a durable marker in the first place — there is nothing for the settle scan to lose track of on an engine restart, unlike every marker-based settle scan in this family. Left unchanged; not retroactively extended by this fix.engine/pr_terminal_advance.go:191andengine/merge_gate.go:627— the two genuine gaps, both closed by this section’s mechanism.
Three other advanceToNextStage call sites exist in the codebase (poll.go, stages.go, merge_train.go) but are not named in #1422’s Scope section and are left untouched — a candidate follow-up, not fixed here.
Scope: only the bare status-move failure, nothing else. recordAdvanceOutcome does not duplicate closeIssueIfNonDefaultBase (§6.13) — that call already runs once, unconditionally, at both original call sites regardless of advanceToNextStage’s outcome, and has its own independent retry owner (fabrik:awaiting-close) if it also fails. “Closing the issue,” part of R2/Acceptance #2’s phrasing, needs no new code here: GitHub’s own Closes #N auto-close fires on merge for the default branch regardless of Fabrik’s board-Status bookkeeping, and closeIssueIfNonDefaultBase already owns the non-default-base explicit close independently. This section’s sole responsibility is the stuck board-Status move.
Corollary (Pruefer, PR #1469 review, third round): because closeIssueIfNonDefaultBase runs regardless of recordAdvanceOutcome’s outcome, a base:<branch> issue whose terminal advance just failed can still end up closed on the very same pass — fabrik:awaiting-advance and a stranded board Status coexisting with a closed issue. This is pre-existing ordering (unchanged by this section’s introduction), and harmless in practice: settleAwaitingAdvanceScan does not filter on IsClosed, so the retry (and, on exhaustion, the escalation pause/comment) still fire correctly — they just land on an issue that is already closed, which can read oddly to an operator glancing at the thread. No code change follows from this; it is recorded here so the ordering is discoverable without re-deriving it from the two call sites.
R4 — no silent fallback to Done. #1082 suggested falling back to Done when the intermediate option is absent. Rejected as the primary mechanism: it would paper over a misconfigured board and hide this exact bug. No fallback of any kind is implemented — a failed advance stays failed, visibly, until the board is fixed or MaxRetries escalates it.
State transitions:
| Before | Trigger | After | Labels Added | Labels Removed |
|---|---|---|---|---|
| Any active stage, stage-complete | recordAdvanceOutcome’s advanceToNextStage call fails (first failure) |
Unchanged column, stranded | fabrik:awaiting-advance |
— |
| Stranded (marker present) | Settle pass: advanceToNextStage still fails |
Unchanged (no second comment) | — | — |
| Stranded (marker present) | Settle pass: advanceToNextStage succeeds (board fixed) |
Next stage | — | fabrik:awaiting-advance |
| Stranded (marker present) | Attempts("__awaiting_advance__") >= MaxRetries |
Paused, still stranded | fabrik:paused |
— (marker deliberately retained — see above) |
| Paused, still stranded (both labels present) | Operator removes fabrik:paused only (board not yet fixed) |
Stranded, fresh retry budget | — | — (Attempts reset to 0 via awaitingAdvanceStuckOrReset/awaitingAdvanceResetIfUnpaused) |
| Stranded (marker present, post-reset) | advanceToNextStage succeeds (board now fixed) |
Next stage | — | fabrik:awaiting-advance |
References: ADR-1422: Terminal-Advance Escalation and Settle Scan, ADR-1097: Non-Default-Base Explicit Close Retry (§6.13, the closest structural precedent — marker-on-failure-only, single at-risk call after other side effects already landed), ADR-1387: Closed Items Are Never Dispatched (§6.15, defines the current closed-item admission rule that already retries advanceValidateTerminalItem’s closed half), ADR-1270: Awaiting-CI Settle Scan (the “admission-gated retry is fragile even when it mostly fires” rationale, and the direct template for this pattern), ADR-064: Closed-Item-At-Any-Stage Advance To Done (rationale for closed_item_advance_settle.go’s deliberately unbounded retry, audited above and left unchanged), ADR-060/061/062/1208 (the other instances of the dedicated-settle-scan pattern this section follows), issue #1422, issue #1082
6.18 Runaway Guard Alert Retry (ADR-1533)
Channel events (#1968):
fireRunawayGuardpausing a member emitsmerge-train-failed(causerunaway-guard); see §7.14.
Trigger: Issue #1533. fireRunawayGuard (see the “Runaway guard (ADR-059 D8)” section above for its full pause+alert contract) pauses every member it’s given, but its alert AddComment call and its fabrik:paused/fabrik:awaiting-input label calls were independent, unconditional GitHub API calls — a comment failure was logged and the loop continued straight into the label calls anyway. Because groupQueuedByRepoAndBase (Hook 2’s snapshot source) excludes any fabrik:paused member from every subsequent poll’s Queued snapshot, and Hook 1 (the worker goroutine) only ever knows the members it started with, a member whose comment failed had no path back to a retry — it stayed fabrik:paused with no explanation, permanently, unless an operator noticed and investigated by other means (e.g. a sibling member’s comment in the same batch). This is the same “stranded with no signal” shape ADR-060/#1422/#1408 already treat as a defect.
Same failure class as §6.8/§6.9/§6.10/§6.13/§6.14/§6.15/§6.16/§6.17 — the eighth instance of the ADR-1270 dedicated-settle-scan pattern. The durable marker (fabrik:awaiting-runaway-alert), retry-owner settle scan (settleRunawayGuardAlertScan), and MaxRetries-bounded escalation all reuse the shared recordSettleRetry/clearSettleMarker/escalateSettle helpers (engine/settle.go) this whole family shares.
The one structural difference from every sibling in this family: fabrik:paused is not a “later” state, it’s a “day one” state. For fabrik:awaiting-member-close/fabrik:awaiting-close/fabrik:awaiting-advance, the marker is written when some terminal step fails after the item has already reached its natural resting state — the item isn’t fabrik:paused yet, and each settle scan explicitly skips items that are (an operator investigating a pause must not be fought, and a paused item there means “already escalated, stop retrying”). Here, fabrik:paused is applied by fireRunawayGuard unconditionally, in the same loop iteration as the comment attempt, regardless of whether the comment succeeded — so a member carrying fabrik:awaiting-runaway-alert always also carries fabrik:paused, from the very first poll the marker exists. settleRunawayGuardAlertScan therefore does not gate on fabrik:paused’s absence (unlike settleMergeTrainMemberCloses/settleNonDefaultBaseCloses/settleAwaitingAdvanceScan) — doing so would make it a permanent no-op. The marker’s own presence/absence is the sole retry-eligibility signal.
Atomicity/idempotency is the primary fix; the settle scan is the residual-failure backstop. Most of #1533’s defect (a member appearing in two racing fireRunawayGuard calls’ item slices) is closed by mergeTrainRunawayMu/mergeTrainRunawayAlerted — see the “Atomicity and per-episode idempotency” paragraph in the Runaway guard section above. The settle scan exists for the residual case that atomicity alone cannot fix: a single call’s AddComment failing outright (network error, rate limit) for a member no future call will ever revisit, because that member is now fabrik:paused and therefore excluded from groupQueuedByRepoAndBase forever.
Code path: fireRunawayGuard → markRunawayAlertOutstanding (writes the marker on comment failure only; engine/runaway_alert_settle.go) — and, on retry, settleRunawayGuardAlertScan → settleRunawayGuardAlert (same file; called from poll() in poll.go, immediately after settleMergeTrainMemberCloses) directly.
Retry-owner: settleRunawayGuardAlertScan. Runs unconditionally once per poll — independent of merge_train: on/off, so a marker written while it was enabled keeps draining even if the setting is later turned off (mirroring settleMergeTrainMemberCloses) — iterating the raw board.Items, not groupQueuedByRepoAndBase or deepFetchCandidates: the member has already reached its terminal “paused, awaiting operator” state by the time the marker is written, and groupQueuedByRepoAndBase’s own fabrik:paused exclusion (load-bearing for the poison-well guard, unrelated to this scan) would otherwise hide exactly the items this scan exists to find. For every item carrying fabrik:awaiting-runaway-alert, it calls settleRunawayGuardAlert, which acquires mergeTrainRunawayMu for its entire body — the same critical section fireRunawayGuard uses — before re-reading the count/window live via isRunawayTripped/effectiveTrialWindow (no storage persists the original firing’s exact count — a live re-read is accurate enough for the retry comment’s wording) and retrying the postComment call. Holding the mutex across the retry, not just the map update afterward, matters: a member carrying this marker can still appear in a later Hook 1 call’s own current/survivors (the worker’s own in-flight member list, not a fresh board read that would exclude an already-fabrik:paused item — unlike Hook 2’s groupQueuedByRepoAndBase), so without the lock spanning the comment post itself, a racing fireRunawayGuard call and this settle retry could both observe “not yet alerted” and post the comment concurrently. On success, the member is recorded in mergeTrainRunawayAlerted (so a still-in-flight or later fireRunawayGuard call for the same member, within the same episode, skips it rather than double-posting) and the marker is cleared.
Retry counting and escalation. Every failed retry calls recordRunawayAlertRetry, keyed by the dedicated constant "__awaiting_runaway_alert__" (same double-underscore-wrapped, YAML-unrepresentable shape as the rest of this family). MaxRetries <= 0 means unlimited retries, never escalate. Once Attempts("__awaiting_runaway_alert__") >= e.cfg.MaxRetries, escalateRunawayAlertFailure fires.
Escalation removes the marker only if its own fallback comment succeeds — a deliberate departure from the rest of the family. fabrik:paused is already present (re-applying it here is a harmless no-op) and is not the thing being escalated into — the escalation’s job is purely to guarantee the member ends up with some explanation. escalateRunawayAlertFailure posts a fallback comment ("🏭 **Fabrik merge-train — runaway guard alert delivery failed**") embedding the same runawayGuardAlertMessage body the original alert would have carried. Unlike every other escalation in this family, it does not delegate to the shared escalateSettle helper: escalateSettle unconditionally removes the marker and discards its comment closure’s error (via postItemComment), which is fine when the escalation comment is purely informational — but here the fallback comment is the last remaining delivery of the explanation R1 requires. escalateRunawayAlertFailure therefore checks the fallback’s own result: on success, the marker is cleared and the member is recorded alerted (mergeTrainRunawayAlerted); on failure — a persistent AddComment outage outlasting MaxRetries, not just a transient one — the marker stays, so the next settleRunawayGuardAlertScan pass keeps retrying the primary alert, then the fallback, every poll, indefinitely, until a comment actually lands (#1533 review, finding 1: the original shape removed the marker regardless of the fallback’s outcome, which could erase the only diagnostic signal that the alert never landed).
Marker clearing. fabrik:awaiting-runaway-alert is removed in exactly two places: clearRunawayAlertMarker (a fully successful settleRunawayGuardAlert pass, or a racing fireRunawayGuard call for the same member succeeding first — both go through the identical helper) and escalateRunawayAlertFailure (giving up after MaxRetries, as described above).
State transitions:
| Before | Trigger | After | Labels Added | Labels Removed |
|---|---|---|---|---|
| Queued (or any column), not runaway-flagged | fireRunawayGuard: AddComment succeeds |
Same column, paused + alerted | fabrik:paused, fabrik:awaiting-input |
— |
| Queued (or any column), not runaway-flagged | fireRunawayGuard: AddComment fails |
Same column, paused + alert outstanding | fabrik:paused, fabrik:awaiting-input, fabrik:awaiting-runaway-alert |
— |
| Paused, alert outstanding (marker present) | Settle pass: AddComment succeeds |
Same column, paused + alerted | — | fabrik:awaiting-runaway-alert |
| Paused, alert outstanding (marker present) | Settle pass: AddComment still fails |
Unchanged | — | — |
| Paused, alert outstanding (marker present) | Attempts("__awaiting_runaway_alert__") >= MaxRetries, fallback comment succeeds |
Paused, fallback comment posted | — | fabrik:awaiting-runaway-alert |
| Paused, alert outstanding (marker present) | Attempts("__awaiting_runaway_alert__") >= MaxRetries, fallback comment also fails |
Unchanged — still retried every poll | — | — |
References: ADR-1533: Runaway Guard Atomic Pause and Alert, ADR-059: Internal Merge Train §D8 (original runaway guard design — see ADR-1533’s correction note), ADR-061: Merge-Train Singleton Member-Issue Close Retry (§6.10, the closest structural precedent among this family), issue #1533
6.19 Post-Done Landing Verification (ADR-1616)
Channel events (#1968): The
fabrik:awaiting-landing-verificationlabel (applied right after a merge-attributable Done transition) is what emitsmerged;failLandingVerificationapplyingfabrik:landing-verification-failedemitslanding-verification-failed; see §7.14.
Trigger: Issue #1614. A merge-train batch attribution bug closed two issues as COMPLETED whose work was never actually merged — the Done transition is driven entirely by inferred success (a merge call returned, or a PR was believed merged), never by observing the credited work actually landing. This is silent and self-certifying: the issue reads COMPLETED, the PR reads landed, and no dispatch path ever revisits a Done item, so there is no signal for anything to notice later. Issue #1615 fixes the specific root cause; this is the cause-agnostic backstop that would have caught it — and would catch the next one, whatever its cause.
Mechanism: merge-state of the crediting PR — not branch-tip ancestry. The issue’s Prior Discussion thread recorded a live false-positive audit: a literal git merge-base --is-ancestor-style check, hand-run against 25 real closed issues in this repo, flagged three (#1573, #1523, #1520) as “not landed” that had, in fact, landed — because their branches were rebased onto a later base after landing, so the same content exists on base under different commit SHAs while the branch tip is no longer an ancestor. A merge-train member is especially prone to this: the train merges the trial branch, and the member’s own branch then drifts independently. Separately, under --auto-merge-strategy SQUASH/REBASE, GitHub creates a new commit object on the base branch, so a merged PR’s original branch-tip commit is never a literal ancestor — branch-tip ancestry would false-positive on every correctly-shipped issue in a non-MERGE-strategy repo, not just drifted ones. Both findings point the same way: the property this issue needs cannot be expressed as a fact about the item’s own branch tip at all. The chosen mechanism instead asserts that the PR credited for the Done transition reached MERGED (FetchPRMerged) — inherently strategy-agnostic (GitHub’s REST merged boolean means the same thing under MERGE/SQUASH/REBASE) and immune by construction to both false-positive classes, since it never inspects the item’s own branch. R3 (a branch legitimately deleted after a real merge is not a failure) is satisfied the same way — the branch is simply never examined.
Discovery: a durable label recorded at the Done transition itself, not comment-parsing. Three landing paths write to Done, and none of them carries PR/merge identity forward through advanceToNextStage’s bare Status-field mutation. What differs is which PR is credited and how it’s later rediscoverable:
| Path | Credited PR | How it’s found later |
|---|---|---|
Ordinary auto-merge (advanceConvergedPRToDone, advanceValidateTerminalItem’s merged branch) |
The issue’s own linked/closing PR | FetchLinkedPR(owner, repo, issueNumber) — durably rediscoverable, no new storage needed |
Merge-train batch (landMergeTrainBatch) |
The integration PR — distinct from the member’s own PR, which stays closed-not-merged | fabrik:credited-pr:<N>, applied at the Done transition |
Merge-train singleton (landSingleton) |
A dedicated singleton landing PR — again distinct from the member’s own PR | fabrik:credited-pr:<N>, applied at the Done transition |
Merge-train singleton fast path (trySingletonFastPath, §6.20, ADR-1644) |
The member’s own PR — no separate PR is ever minted | FetchLinkedPR(owner, repo, issueNumber) — durably rediscoverable, same as the ordinary auto-merge path, no fabrik:credited-pr:<N> needed |
An earlier design considered parsing the existing "🏭 **Fabrik merge-train** — Landed via batch PR #%d." comment landMergeTrainBatch/landSingleton already post — rejected because addLandedCommentWithRetry is best-effort (three attempts, then a silent log warning) and the pattern is coupled to exact prose with no test tying the two together; a verifier that goes quiet exactly when the landing path is already misbehaving is not a backstop (the same failure shape as #1614 itself — a mechanism that appears to protect while silently doing nothing). Recording fabrik:credited-pr:<N> as a label at the transition — cheap (the number is already in hand), durable (survives a restart, unlike an itemstate field — itemstate has no cross-restart persistence and GitHub exposes no change-history equivalent to backfill it), and matching this repo’s “labels are state” convention — removes that dependency entirely. Not applied by the ordinary auto-merge path, whose credited PR needs no such capture.
Marker-driven scope, not backlog-driven. fabrik:awaiting-landing-verification is applied at the Done transition itself by all three call sites, immediately after their own advanceToNextStage/recordAdvanceOutcome call succeeds — not unconditionally the way closeIssueIfNonDefaultBase runs regardless of the advance’s outcome (§6.13), since this marker’s entire purpose is to verify a Done transition that must have actually happened. settleLandingVerification owns items carrying the marker exclusively, sourced from raw board.Items (ADR-1270 precedent), never deepFetchCandidates. Old Done items — anything that landed before this feature shipped — never had the marker applied, so they are naturally out of scope: no backfill pass, no risk of misclassifying a pre-feature merge-train landing (whose member PR is legitimately closed-not-merged) as a failure, and AC1’s “within one poll cycle” is trivially satisfied since the marker exists from the same poll the transition happens in.
R4 (base:<branch>) is satisfied by the mechanism itself, not by an active check. Verifying “did the credited PR merge” needs no base-branch resolution at all — the credited PR’s own base is whatever it was opened against, already base:<branch>-correct at PR-creation time (#1554). baseBranchForItem/WorktreeManager resolution (resolveLandingVerificationBranchInfo) is used only to name the base in the failure comment — mirroring closeIssueIfNonDefaultBase’s existing best-effort, degrade-gracefully pattern (§6.13) — never as an input to the merge/no-merge decision. A repo with no registered WorktreeManager degrades to an unnamed base in the comment, never a blocked scan.
Two failure regimes, never conflated:
- Confirmed failure — the credited PR resolves and
FetchPRMergeddefinitively reportsfalse.failLandingVerificationfires immediately, on the same pass that reaches this determination — never gated by the retry counter, so AC1’s “within one poll cycle” holds regardless ofMaxRetries. - Inconclusive — a
FetchPRMerged/FetchLinkedPRAPI error, or no crediting reference discoverable at all (nofabrik:credited-pr:<N>label and no linked PR). Absence of evidence is not evidence of a falseCOMPLETED, so this goes through the samerecordSettleRetry/escalateSettleidiom (engine/settle.go) every settle-scan sibling in this family shares, bounded byMaxRetries, escalating tofabrik:paused— never reopens the issue on ambiguity (R5, AC4).
Conflating these would either reopen a correctly-closed issue on a single transient hiccup, or silently retry a confirmed false-COMPLETED forever.
Code path: landMergeTrainBatch/landSingleton/finishSingletonFastPathLanding (engine/merge_train.go), advanceConvergedPRToDone (engine/merge_gate.go), advanceValidateTerminalItem’s merged-PR branch (engine/pr_terminal_advance.go) — each applies fabrik:awaiting-landing-verification (and, for landMergeTrainBatch/landSingleton only, fabrik:credited-pr:<N>) immediately after its own successful Done transition. finishSingletonFastPathLanding (§6.20, ADR-1644) follows advanceConvergedPRToDone’s labeling exactly — see the table above. Retry-owner: settleLandingVerification → settleLandingVerificationItem (engine/landing_verification_settle.go; settleLandingVerification called from poll() in engine/poll.go, immediately after settleNonDefaultBaseCloses).
settleLandingVerificationItem flow:
- Resolve the credited PR:
fabrik:credited-pr:<N>label if present →N; elseFetchLinkedPR→ its number. Neither found → inconclusive (recordLandingVerificationRetry). FetchPRMerged(owner, repo, creditedPR).- Error → inconclusive (
recordLandingVerificationRetry). merged == true→clearLandingVerificationMarkersremovesfabrik:awaiting-landing-verificationand anyfabrik:credited-pr:<N>, clears the retry counter. No further action (AC2).merged == false→failLandingVerification: reopens the issue (ReopenIssue, if closed), moves the item’s board Status back to the fixed targetValidate(moveItemToValidate— a directUpdateProjectItemStatuscall with a looked-upoptionID, sinceadvanceToNextStageonly ever moves forward to the next configured stage and has no “move backward to a caller-chosen target” mode), appliesfabrik:landing-verification-failed, removesfabrik:awaiting-landing-verification/fabrik:credited-pr:<N>, and posts a comment naming the branch (fabrik/issue-<N>), the resolved base (best-effort — blank if noWorktreeManageris registered), and the credited PR checked and found not merged. Every step is independently logged and best-effort — aReopenIssuefailure must not prevent the board move, label, or comment from still happening.
- Error → inconclusive (
The remediation is human-gated, and the comment says so. failLandingVerification moves the board Status back to Validate but deliberately leaves stage:Validate:complete in place, so the move alone does not re-dispatch the stage — matching R2’s “park it for inspection” posture rather than silently re-running a landing that just failed verification. Because a board move that doesn’t re-dispatch is exactly the hazard §6.16/#1545 R4 documents elsewhere, the failure comment states this explicitly and names fabrik:revalidate as the label that does re-run Validate, so an operator is never left waiting on a re-dispatch that will not come.
Restart safety at the merge-train landing sites. advanceToNextStage and the marker writes are separate, non-atomic API calls, so a run can advance a member to Done and die before markCreditedLanding executes. Both merge-train landing paths have a pre-existing “member already in Done column” fast path for exactly that restart shape — and both now call markCreditedLanding on that path too, rather than skipping straight past it. Without the backfill the item would be permanently unverified: nothing ever revisits a Done item, so the marker would never be applied and the backstop would be silently absent for precisely the merge-train restart scenario #1614 came from. Both label writes are idempotent, so re-marking an item whose verification already completed costs at most one redundant FetchPRMerged that confirms merged and clears the markers again.
“Move it off Done” target: Validate, uniform across all three paths. Validate already knows how to attempt/re-attempt landing (auto-merge, or re-entry into the merge-train’s holding stage on its next cycle) — a human-gated posture (parked for inspection via the reopen + distinguishing label + comment), not a silently self-healing one. "Validate" is a literal, hardcoded stage name here, the same convention used throughout this package (pr_terminal_advance.go, poll_settle.go).
Retry counting and escalation. Every inconclusive pass calls recordLandingVerificationRetry, keyed by the dedicated constant "__landing_verification__" (same double-underscore-wrapped, YAML-unrepresentable shape as the rest of this family, governing only the inconclusive regime — never the confirmed-failure regime). MaxRetries <= 0 means unlimited retries, never escalate. Once Attempts("__landing_verification__") >= e.cfg.MaxRetries, escalateLandingVerificationFailure fires: pauses the issue, removes the marker, and posts a comment deliberately worded differently from failLandingVerification’s confirmed-failure comment — explicitly not a confirmed failure, so an operator reading it does not conclude work was actually lost.
A known, narrow blind spot: a merge-train label write that fails outright. The two merge-train landing paths write their labels through markCreditedLanding, which applies fabrik:credited-pr:<N> first, with its error checked, and applies fabrik:awaiting-landing-verification only if that credited-PR record actually landed on GitHub. This ordering and check are load-bearing, not defensive habit: the two labels are separate GitHub calls, so the credited-PR write can fail on its own (rate limit, momentary 5xx). Were the marker applied regardless, the scan would find no fabrik:credited-pr:<N> label and fall back to FetchLinkedPR — which for a merge-train member resolves to the member’s own PR, closed and deliberately never merged. FetchPRMerged would report false and the scan would fire its immediate confirmed-failure path against a correctly-landed issue, precisely the false positive AC2 forbids.
So a lost credited-PR write leaves the item unmarked — not verified (the pre-#1616 status quo), logged loudly at the landing call site — rather than marked-and-unresolvable. Not verifying is a coverage gap; falsely reversing a good landing is a regression, and between the two the gap is strictly safer. A board snapshot taken between the two writes can therefore only ever show the credited-PR label without the marker, which the scan ignores — never the reverse.
Scope: covers only the three merge-attributable Done-transition call sites. No change to merge-train assembly, bisection, or batching — every write this feature makes is a label applied after advanceToNextStage has already succeeded, never a change to the landing logic itself. Issue #1615’s trial-identity root-cause fix is independent and out of scope here — this backstop detects and reverses a false COMPLETED within a poll cycle; it does not prevent one.
State transitions:
| Before | Trigger | After | Labels Added | Labels Removed |
|---|---|---|---|---|
| (any Done-transition call site) | Own advanceToNextStage/recordAdvanceOutcome call succeeds |
Done, awaiting landing verification | fabrik:awaiting-landing-verification (+ fabrik:credited-pr:<N> for merge-train paths) |
— |
| Done, awaiting landing verification | Settle pass: credited PR confirmed merged | Done, verified | — | fabrik:awaiting-landing-verification, fabrik:credited-pr:<N> |
| Done, awaiting landing verification | Settle pass: credited PR confirmed not merged | Validate, reopened, failure-labeled | fabrik:landing-verification-failed |
fabrik:awaiting-landing-verification, fabrik:credited-pr:<N> |
| Done, awaiting landing verification | Settle pass: inconclusive (API error, or no crediting reference found) | Unchanged | — | — |
| Done, awaiting landing verification | Attempts("__landing_verification__") >= MaxRetries (inconclusive regime exhausted) |
Done, Paused | fabrik:paused |
fabrik:awaiting-landing-verification |
References: ADR-1616: Post-Done Landing Verification, ADR-1270: Awaiting-CI Settle Scan (the settle-scan pattern this is the ninth instance of), ADR-1096: Explicit Close on Non-Default-Base Merge / §6.13 (closest structural precedent — merge-adjacent, best-effort, settle-scan-backed), issue #1614 (the motivating incident), issue #1615 (its independent root-cause fix)
6.20 Merge-Train Singleton Fast Path (ADR-1644)
Trigger: Issue #1644. A merge-train batch containing exactly one Queued member pays the full trial cycle — a disposable trial worktree, a pushed branch, a draft CI PR, a complete CI run, a landing integration PR — even when the member’s own PR has already been validated against the exact tree the trial would produce. Fabrik’s own Validate-stage convention (rebase onto base before signalling completion) means a member arriving fresh at Queued is typically already up to date with base, making the trial’s re-validation redundant by construction on the common path. This is not a bug — ADR-1153’s check-completeness gate and pollTrainCI’s backstop timeout both behave correctly throughout — it is avoidable, correctly-performed, duplicate work.
Mechanism: skip the trial only on positive evidence, never on absence of evidence. trySingletonFastPath (engine/merge_train.go), called from runMergeTrainWorker’s re-form loop on every iteration where len(current) == 1, immediately before a trial would otherwise be built. singletonFastPathEligible requires all four of the following to hold; any single one failing — including any API error — falls through to the unmodified trial path:
- No re-push since batch formation (TOCTOU guard). The member’s live PR head SHA (fetched fresh via
FetchPRDetails) still equals thetrainMember.headSHAsnapshot captured once byfetchTrainMembersat worker start. A member re-pushed between batch formation and this check must never be landed on stale evidence. - The pinned base SHA is an ancestor of the member’s head.
FetchCommitsBehind(owner, repo, p.baseSHA, m.headSHA) == 0— checked againsttrialParams.baseSHA, the base pinned once at batch start (ADR-059 D-b), never a liveorigin/<base>read (R3), so a base that moves mid-flight cannot make a stale decision look current.behind_by == 0means the pinned base is a fast-forward ancestor of the member’s head, so the trial’s tree (pinned_base + member) would be byte-identical to the member’s own PR head tree. - The PR is mergeable.
gh.MergeableStateAccepted(pr.MergeableState)(ADR-072) — the same{clean, unstable}acceptance testMergePR’s own self-gate uses; adirtystate means an actual conflict. - The member’s own CI is green and complete.
FetchCheckRunsmust return at least one check run — zero check runs is rejected outright, beforeclassifyLandingCIis even consulted, regardless ofmergeable_state(see below for why this is stricter thanclassifyLandingCI’s own zero-check-runs handling). Given at least one check run,classifyLandingCI(ADR-1153/ADR-1441, reused unmodified — the same check-run-based classifierpollForMergeableuses for the ordinary landing path) must returnTrainCIGreen. Since #1822classifyLandingCIadditionally returnsTrainCIPendingwhile any check suite has outstanding work or the suite read fails (§6.4 Suite-aware completeness), so a member whose own all-green runs are only a prefix of a still-running workflow is not eligible and falls through to the trial path.
Every branch that cannot positively confirm a condition returns (false, reason) — there is no code path that returns eligible except by every check having been positively satisfied. This is the deliberate inverse of #1614’s failure mode: there the train wrongly skipped a merge; here a wrong skip would mean skipping validation, so only positive evidence may authorize it (R2).
Why zero check runs is rejected outright, not delegated to classifyLandingCI’s own zero-check-runs fallback. classifyLandingCI’s zero-check-runs branch falls back to mergeable_state alone (ADR-933’s “Actions disabled” case) — correct for its usual caller, pollForMergeable, which judges a landing PR whose tree a just-polled trial CI run already validated, so zero check runs there is a benign GitHub quirk, not an absence of validation. This fast path never builds a trial, so that precondition doesn’t hold: zero check runs on the member’s own PR head means nothing has actually validated that tree, not “green by absence.” singletonFastPathEligible therefore rejects on zero check runs before classifyLandingCI is ever consulted — a fast-path-specific precondition layered in front of the reused classifier, not a fork of it.
Why this is a loop-level guard, not a change to assembleAndValidate. assembleAndValidate has four call sites: the main re-form loop (the only genuinely-fresh Queued-batch decision), bisect’s poisoner-isolation sub-trials (a member’s own-PR CI passing says nothing about whether it’s the poisoner in an already-red combination — fast-pathing here would be a correctness bug, not just scope creep), landOneAtATime’s one-at-a-time fallback (explicitly out of scope — landSingleton’s existing fallback role is unchanged), and landGreenBatch’s main-moved rebase loop (D5 recovery of an already-green trial after base moved — a different problem from “should a trial be built at all”). Placing the guard inside assembleAndValidate itself would silently reach all four; placing it in the loop, immediately before the call, reaches only the one call site this feature targets. Applied on every loop iteration, not just the first, so a batch that bisects down to a single clean survivor and re-forms is also eligible.
A member that is merely behind is caught up, not tried (#2044). When check 2 fails only because the pinned base is ahead of the member’s head, the worker does not go straight to a trial: trySingletonCatchUp (§6.30) first merges the pinned base into the member’s own branch, waits for its own CI and re-enters this same eligibility check, unmodified, on the new head. Every other ineligibility, and singleton_catch_up: off, still builds the trial exactly as before.
Pending review-finding ejects (#1208) are applied before the fast path is even considered. applyPendingReviewEjects (§6.16) existed at three checkpoints before this feature, all downstream of a call to assembleAndValidate — the re-form loop’s own Hook 2, landOneAtATime’s per-singleton loop, landGreenBatch’s main-moved rebase loop. The fast path never calls assembleAndValidate, so without a fourth checkpoint specific to it, a member flagged mid-trial for unresolved review-thread findings could be landed directly, never reaching any of the three existing checkpoints. The guard therefore applies applyPendingReviewEjects to the length-1 current slice before calling trySingletonFastPath: an ejected member leaves current empty, and the loop continues into the top-of-loop zero-survivors return; an unflagged member falls through to the fast path exactly as before. (Found in review — the original implementation had no such checkpoint.)
Landing shape: advanceConvergedPRToDone, not landSingleton. landSingleton (the bisection fallback’s per-member landing helper) is built entirely around minting a dedicated landing PR off a validated trial branch — a shape this feature has no trial branch to produce. Instead, finishSingletonFastPathLanding follows the ordinary auto-merge path’s own Done-transition template: MergePRAtHeadSHA(owner, repo, m.prNum, m.headSHA) merges the member’s own PR directly, pinned to the exact head SHA singletonFastPathEligible validated (see below), recordAdvanceOutcome advances Queued → Done (with the same durable failure-retry/escalation every terminal advance gets, §6.17), closeIssueIfNonDefaultBase runs unconditionally on the confirmed merge (see below), and — only once the advance itself succeeded — awaitingLandingVerificationLabel is applied with no fabrik:credited-pr:<N> label, since the merged PR is the item’s own linked PR and is durably rediscoverable via the ordinary FetchLinkedPR fallback (§6.19’s table, row 1) — unlike merge-train’s other two landing paths, whose credited PR is never the member’s own. resetEjectionCount/resetTrialCounter still run, exactly as every other successful land performs.
The landed comment, and why its shape differs from the other two paths. Every merge-train landing path posts a 🏭 **Fabrik merge-train** — Landed … comment on the member’s own PR — the sole cross-landing-path, member-scoped record of which PR actually landed the change (#1275). finishSingletonFastPathLanding posts Landed via singleton fast path PR #<n>., and here <n> is the member’s own PR number, because that PR is the merge vehicle — there is no distinct integration or singleton PR to name, by construction. Under landMergeTrainBatch (Landed via batch PR #<n>.) and landSingleton (Landed one-at-a-time via singleton PR #<n>.) the cited PR is always distinct from the member’s, and a comment citing the member’s own PR there would indicate a direct merge that bypassed the train. Anything reading these comments must therefore key off the path named in the text, not off distinctness alone (tests/e2e/mergetrain_helpers.go’s landedPRPattern/waitForLandingPRDetail do exactly this).
Unlike the other two paths, this one is guarded, rather than posted unconditionally. landMergeTrainBatch/landSingleton close the member’s PR immediately after commenting, so they can never be re-entered for the same member; this path leaves the PR to GitHub’s own merge-close and can legitimately be re-entered while the member is still Queued (the pr.Merged restart branch above, and a retry after a failed recordAdvanceOutcome). The guard is two-part, and both halves are load-bearing. The comment posts only when this call actually recorded a completed landing — the same condition that applies fabrik:awaiting-landing-verification, i.e. a successful recordAdvanceOutcome — and only when that label is absent from the item’s own snapshot, meaning no earlier call already did. The label’s absence alone is insufficient: it is applied only after the advance succeeds, so on a failed advance (a missing board Status option, say — the case fabrik:awaiting-advance exists to retry, §6.17) the member stays Queued with its PR already merged and re-enters every poll with the guard still open, posting one duplicate per poll until MaxRetries escalates to fabrik:paused. Tying the comment to the successful-label condition closes that window without suppressing it permanently: no advance, no comment, and the retry that eventually advances posts exactly one. The landed half alone is equally insufficient, since the already-Done restart branch re-applies the label idempotently on every re-entry. As with the other paths, the comment gates nothing: addLandedCommentWithRetry retries transient failures and then warns and continues — a landing is never blocked or delayed on it.
Why MergePRAtHeadSHA, not the plain MergePR every other merge-train landing path uses. MergePR’s own self-gate re-reads mergeable/mergeable_state live immediately before merging, but never confirms the head is still the exact commit some earlier, separate check validated — immaterial for landSingleton/landMergeTrainBatch/the ordinary auto-merge path, since all three merge a Fabrik-owned trial, integration, or landing branch nobody else pushes to. This fast path is the first merge-train landing path to merge the member’s own, externally-writable PR branch directly after a separate, earlier CI/ancestry classification — so a commit landing on that branch between the classification and the merge call would otherwise be merged having never been validated at all. MergePRAtHeadSHA closes this by passing the expected head SHA through to GitHub’s merge endpoint’s own sha field: a request whose live head no longer matches is refused (gh.ErrConflict, from GitHub’s 409) rather than silently merging whatever is actually there. A refusal is handled identically to any other merge failure at this call site (see the state-transition table below) — the member stays in Queued for a fresh, correctly re-evaluated attempt next poll.
closeIssueIfNonDefaultBase is required here for the first time in merge-train. Both existing merge-train landing paths (landMergeTrainBatch, landSingleton) always call CloseIssue explicitly regardless of base, so neither ever needed this guard. This fast path relies on the merged PR’s own Closes #N (R5’s stated rationale for landing the member’s own PR directly) — but GitHub only auto-honours that keyword into the repository’s default branch (#1096). A base:<branch>-labeled Queued member would otherwise land but never close; closeIssueIfNonDefaultBase is the same best-effort, fabrik:awaiting-close-backed guard (§6.13) the ordinary auto-merge path already uses.
Restart safety is close to free. MergePRAtHeadSHA shares MergePR’s idempotent-on-already-merged behavior (returns immediately once the PR reports merged). trySingletonFastPath checks pr.Merged before running eligibility at all: if a prior run merged the member’s own PR but died before the Done transition, the next poll’s fast-path check resumes straight to finishSingletonFastPathLanding without re-evaluating eligibility (which could plausibly now read differently against an already-merged PR). finishSingletonFastPathLanding itself mirrors landMergeTrainBatch’s “already Done” restart branch: when m.item.Status == "Done", it still re-applies closeIssueIfNonDefaultBase and the landing-verification label, both idempotent.
No interaction with the runaway guard or trial counter. e.recordTrial (the wrapper around assembleAndValidate that feeds isRunawayTripped) only fires for non-green results and is never invoked by the fast path, since the fast path never calls assembleAndValidate at all. A landed fast path is unambiguous progress, identical in kind to any other green land.
Contrast with batch admission (§6.25). The fast path is fail-closed: it skips validation, so only positive evidence of green authorises it. §6.25’s admission gate at fresh batch formation is the deliberate inverse, fail-open: it removes a member from a batch, so only positive evidence of red (classifyLandingCI → TrainCIRed, reused unmodified) acts, and every ambiguity admits.
Logging (R4). Every disposition — eligible-and-landed, or the specific unmet condition that forced a trial — is logged from trySingletonFastPath/singletonFastPathEligible, so “why didn’t it fast-path?” is answerable from logs without inference. On a completed landing finishSingletonFastPathLanding additionally emits the repo-scoped landing complete for <trainKey> (singleton fast path, PR #<n>, 1 member) line that landMergeTrainBatch emits for a batch — the train’s terminal progress signal on its TUI job row, and the only log-visible statement that a train actually reached a landing. It is repo-scoped (logfRepo), not per-issue: a per-issue line would not reach the train’s job row. Unlike the landed comment above it is emitted unconditionally rather than guarded, since a repeat on re-entry is accurate and is itself the signal that a re-entry happened.
Code path: trySingletonFastPath, singletonFastPathEligible, finishSingletonFastPathLanding (engine/merge_train.go), called from runMergeTrainWorker’s re-form loop. See §6.19’s table for how this path’s credited PR is discovered post-Done.
State transitions:
| Before | Trigger | After | Labels Added | Labels Removed |
|---|---|---|---|---|
| Queued, single member, eligibility unresolved | trySingletonFastPath: any condition not positively confirmed (TOCTOU mismatch, ancestry not confirmed, not mergeable, CI not green/complete, or an API error) |
Queued (unchanged) — falls through to the ordinary trial path | — | — |
| Queued, single member, pinned base ahead of the head (behind) | trySingletonFastPath declines; trySingletonCatchUp (§6.30) |
Catch-up merge pushed, then — once own CI is green — the fast path re-runs on the new head; otherwise deferred or the trial (§6.30) | — | — |
| Queued, single member, all conditions positively confirmed | MergePRAtHeadSHA succeeds; recordAdvanceOutcome succeeds |
Done, awaiting landing verification | fabrik:awaiting-landing-verification |
— |
| Queued, single member, all conditions positively confirmed | MergePRAtHeadSHA succeeds; recordAdvanceOutcome fails |
Queued (unchanged); advance failure durably retried (§6.17) | — | — |
| Queued, single member, all conditions positively confirmed | MergePRAtHeadSHA fails (transient, or the pinned head SHA no longer matches — gh.ErrConflict) |
Queued (unchanged); retried next poll | — | — |
| Queued, single member, PR already merged (restart) | trySingletonFastPath observes pr.Merged == true and the member’s live Status is still the holding stage (§6.28) |
Resumes straight to the Done transition, as above | (as above) | — |
| Snapshot-Queued single member, PR already merged, live Status already Done | pr.Merged == true, liveLandingState → moved (§6.28, #1871) |
Disposition decided; nothing re-landed, no trial built | — | — |
| Queued, single member, pending review-finding eject flagged (#1208) or pending unprocessed-comment eject flagged (#1863) | applyPendingReviewEjects, applied before the fast path is considered |
Rerouted off Queued (§6.16) — never reaches trySingletonFastPath |
— | — |
References: ADR-1644: Merge-Train Singleton Fast Path, ADR-1153: Train CI Completeness over Mergeable State, ADR-1441: Unstable Requires Check-Run Classification, ADR-072: MergePR Self-Gates on Mergeable State, ADR-1208: Queued Review-Finding Ejection, §6.13 (closeIssueIfNonDefaultBase), §6.16 (applyPendingReviewEjects), §6.19 (post-Done landing verification), issue #1644
6.21 Non-Default-Base Linkage Notice (#1649)
Trigger: GitHub’s Closes #N inertness on a non-default base branch (§2.8, ADR-1096) goes further than suppressing auto-close: per GitHub’s own docs, closing keywords “are interpreted only when the pull request targets the repository’s default branch… otherwise these keywords are ignored, no links are created” — so a base:<branch> item’s issue shows no PR at all in its Development panel. Fabrik itself is unaffected (FetchLinkedPR resolves by the fabrik/issue-N head-branch convention, never GitHub’s linkage graph), but the operator looking at the board has no visual cue and has to go hunting. Reported as the “adjacent, lower priority” half of #1646; ADR-1096/1097 already cover the closing half — this covers the purely human-visible half.
Durable marker (fabrik:nondefault-base-pr-noted), written unconditionally after the post attempt. Unlike fabrik:awaiting-close (§6.13), this label is not written only in a failure branch — it’s added right after postItemComment is called, whether or not that call actually succeeded. There is no durable outstanding action to distinguish a “written on failure, retry later” state from: a missed comment (transient AddComment error) is simply gone, not queued. This mirrors fabrik:api-key-helper-detected’s accepted trade-off (CLAUDE.md), not ADR-1097’s retry/escalation shape.
Why no retry/escalation machinery, deliberately. fabrik:awaiting-close’s retry loop exists because a stalled explicit close has a real downstream cost — dependent issues that unblock on close (not merge) stay blocked indefinitely. A missed one-time informational comment has no such cost: nothing gates on its delivery, no dependent issue waits on it, and the underlying fact it reports (this PR has no GitHub linkage) remains discoverable by other means (the issue’s base:<branch> label itself, or a cross-referenced timeline event GitHub still creates). Adding recordSettleRetry/escalateSettle here would be retry machinery in search of a problem it doesn’t have. R2’s “must survive restarts and repeated polls without duplicating” is satisfied entirely by the label’s own presence — it says nothing about guaranteeing eventual delivery.
Code path: noteNonDefaultBasePRLinkage (engine/nondefault_base_linkage_notice.go), reusing itemHasBaseLabel/baseBranchForItem (§6.13’s closeIssueIfNonDefaultBase precedent) for the base-vs-default comparison. Retry-owner (detection only, not delivery): settleNonDefaultBaseLinkageNotice, called from poll() in engine/poll.go, immediately after settleNonDefaultBaseCloses.
settleNonDefaultBaseLinkageNotice flow, run unconditionally every poll over raw board.Items (ADR-1270 family, ninth-or-later instance — ordinal not load-bearing), independent of itemMayNeedWork/itemNeedsWork dispatch and of the item’s board column (eligibility is “has a base: label and a discoverable PR,” not “reached a particular stage or Done”):
item.IsPR→ skip.!itemHasBaseLabel(item)→ skip (R3’s fast path: perbaseBranchForItem’s documented contract, an item with nobase:label always resolves to exactly the default, so the overwhelming common case never touches aWorktreeManageror does a git shell-out).item.LinkedPRNumber == 0→ skip (R1: no PR discovered yet —LinkedPRNumberis already populated on the board snapshot via the deep-fetch cache’sFetchLinkedPR, so this costs no new API call, R4).hasLabel(item.Labels, "fabrik:nondefault-base-pr-noted")→ skip (R2 idempotency).- Non-panicking
WorktreeManagerlookup (e.mu.Lock(); wm, ok := e.worktreeManagers[key]; e.mu.Unlock()), mirroringcloseIssueIfNonDefaultBase— skip-and-log if unregistered (self-heals next poll once a dispatch registers it; same restart-window caveat as ADR-1096). base, err := e.baseBranchForItem(item, wm)→ skip-and-log on error.def, err := wm.DefaultBaseBranch()→ skip-and-log on error.base == def→ skip (R3: covers both a genuinely default-base item and the case wherebaseBranchForItem’s own remote-fallback already reduced the effective base to default — e.g. the labeled branch doesn’t exist on the remote, in which case GitHub’s ordinary linkage applies and no notice is warranted).- Otherwise: post the one-time comment naming the PR (
item.LinkedPRNumber), the resolved base, and the repository default; then unconditionally addfabrik:nondefault-base-pr-noted.
A distinct comment from baseBranchForItem’s own fallback notice. Step 6 can itself post a different one-time comment (“base branch not found on remote, falling back to default”), gated by an in-memory-only sync.Map (not restart-durable, unlike this feature’s own label-gated marker). When that fallback fires, step 8’s base == def comparison correctly no-ops this feature’s notice — but the two comments are independent and must not be conflated when reading an issue’s thread or writing tests against it.
Closed issues are not excluded. Unlike closeIssueIfNonDefaultBase (which treats an already-closed issue as nothing-to-do, since closing an already-closed issue is a no-op action), this scan posts regardless of item.IsClosed — the information (“this PR was never GitHub-linked”) remains true and useful for an issue closed by any means, and posting an informational comment is not the same kind of action as an idempotent-by-necessity state mutation.
State transitions:
| Before | Trigger | After | Labels Added | Labels Removed |
|---|---|---|---|---|
base:<branch> item, PR discovered, base ≠ default |
Settle pass reaches step 9 | Notice posted | fabrik:nondefault-base-pr-noted |
— |
base:<branch> item, no PR yet, or base == default (incl. remote-fallback) |
Settle pass reaches step 3 or 8 | Unchanged | — | — |
Default-base item (no base: label) |
Settle pass reaches step 2 | Unchanged | — | — |
References: ADR-1649: Non-Default-Base Linkage Notice, ADR-1096: Explicit Close on Non-Default-Base Merge / ADR-1097: Non-Default-Base Close Retry / §6.13 (the closing half of this same gap, and the structural precedent this feature’s base-resolution logic reuses verbatim), issue #1646 (the original community report), issue #1649 (this feature).
6.22 Mark-PR-Ready Durable Retry (ADR-1582)
Trigger: markPRReady (§5.2, engine/pr.go) calls e.client.MarkPRReady as the final step of transitioning a stage’s draft PR to ready-for-review, after its own in-process 3-attempt retry (#599). This call fails non-transiently, or exhausts that retry budget. handleStageComplete proceeds regardless — stage:<name>:complete is granted whether or not the PR ever became ready — and, before this feature, nothing ever retried the call again: the PR could remain draft forever, which attemptMergeOnValidate’s direct-merge fallback treats as unconditionally not-CI-clean (mergeable_state == "draft", checked ahead of any check-run/dirty logic), permanently blocking the issue from reaching Done via that path.
Key difference from §6.13’s fabrik:awaiting-close: structurally the same single-at-risk-call shape, but this marker is not terminal-only. mark_pr_ready_on_complete: true can be configured on a non-terminal stage (e.g. Implement, with Review and Validate still ahead), so the marker can be written while the item is still mid-pipeline — and it is deliberately left free to keep advancing through subsequent stages while the marker is outstanding, since nothing about a later stage’s dispatch depends on the PR being ready.
Durable marker (fabrik:awaiting-pr-ready), written only on failure, inline inside markPRReady. markPRReadyOutstanding adds the label (idempotently — a no-op if already present) in exactly the two places markPRReady previously just logged and returned: the non-transient-error branch, and the retry-exhausted fallthrough at the end of its loop. No call-site changes were needed at any of markPRReady’s three callers for this marker (the later bool return, #2089, is unrelated to it — see §”Zero-ahead push guard”) (engine/item.go ×2, engine/comments.go) — every value the marker write needs (item, owner, repo, the resolved prNumber) was already in scope at both failure points. markPRReady’s success path also calls clearPRReadyMarker when the marker happens to already be present, so an issue previously escalated and un-paused self-heals the moment its own next markPRReady call succeeds directly, without waiting for the settle scan below to notice.
No dispatch-suppression wiring — deliberately, and unlike most of this family. This marker is not checked by itemMayNeedWork/itemNeedsWork, and is not added to transientLifecycleLabels. Unlike fabrik:awaiting-close (§6.13) or fabrik:awaiting-member-close (§6.10), which only ever exist after the item has already reached Done, this marker can be present on an item still actively progressing through later stages — gating stage completion on MarkPRReady succeeding was explicitly out of scope for #1582 (a bigger change to the conjunctive-gate model than the bug called for).
Code path: markPRReady (engine/pr.go, writes the marker on failure, clears it on a direct success) — and, on retry, settlePRReadyScan → settlePRReady (both engine/pr_ready_settle.go; settlePRReadyScan is called from poll() in poll.go) directly.
Retry-owner: settlePRReadyScan (engine/pr_ready_settle.go; called from poll() in poll.go). Runs unconditionally once per poll. It iterates the raw board.Items (not deepFetchCandidates — this scan has no dependency on the deep-fetch/terminal-skip machinery at all). For every item carrying fabrik:awaiting-pr-ready and not fabrik:paused (mirroring every other settle scan’s own paused-item guard), it calls settlePRReady.
settlePRReady flow — idempotent; safe to call repeatedly. Re-resolves the PR live every pass (via FetchLinkedPR) rather than trusting a stashed PR number — a GitHub label carries no payload, and this scan runs on a completely independent poll cadence from when the marker was written:
- No linked PR found at all → clear the marker (nothing left to mark ready — the PR was deleted, or the marker survived a scenario that no longer applies).
- PR closed or merged → clear the marker (the issue was abandoned, or it landed via a path that never called
markPRReady). - PR found and already not-draft → clear the marker without calling
MarkPRReadyagain (self-healed by a later stage’s ownmarkPRReadycall, or a human clicking “Ready for review” manually). - Otherwise (genuinely still draft and open): call
e.client.MarkPRReadyagain — safe regardless of how many times this fires, since GitHub’smarkPullRequestReadyForReviewmutation is a documented no-op success on an already-ready PR. On success, clear the marker. On failure, record a retry and stop — the next settle pass re-attempts. - A
FetchLinkedPRAPI error (distinct from “no PR found”) retries without clearing the marker — the scan cannot tell whether the PR is still draft, so it must not assume either outcome.
Retry counting and escalation. Every failed settle pass calls recordPRReadyRetry, which increments the existing itemstate.StageRetryIncremented/Attempts counter — keyed by the dedicated constant "__awaiting_pr_ready__" (same double-underscore-wrapped, YAML-unrepresentable shape as its siblings, so it can never collide with a configured stage’s own counter). MaxRetries <= 0 means unlimited retries, never escalate (same guard as every sibling counter). Once Attempts("__awaiting_pr_ready__") >= e.cfg.MaxRetries, escalatePRReadyFailure fires — mirroring escalateNonDefaultBaseCloseFailure: adds fabrik:paused, removes fabrik:awaiting-pr-ready, posts an explanatory comment naming the draft PR (item.LinkedPRNumber, when non-zero) with the manual recovery step (gh pr ready <N>), and applies itemstate.EnginePaused.
Marker clearing. fabrik:awaiting-pr-ready is removed in three places: clearPRReadyMarker via a fully successful settle pass (PR found not-draft, closed, merged, or not found at all — the normal, expected path), markPRReady’s own success path (a direct self-heal, without waiting for the settle scan), and escalatePRReadyFailure (giving up after MaxRetries).
Scope: this mechanism covers only markPRReady’s own client.MarkPRReady call (engine/pr.go, called from engine/item.go and engine/comments.go for any stage configured with mark_pr_ready_on_complete: true). The merge-train’s own inline MarkPRReady call (engine/merge_train.go, promoting a reused draft integration PR to ready before landing) is out of scope — a failure there already leaves the affected members in Queued and is retried naturally on the worker’s own next iteration. The generic recordSettleRetry/escalateSettle/clearSettleMarker helpers (engine/settle.go) this mechanism reuses are the same ones backing fabrik:awaiting-done (§6.8), fabrik:awaiting-member-close (§6.10), and fabrik:awaiting-close (§6.13).
Interaction with attemptMergeOnValidate: no change needed. That function already retries indefinitely on every poll while un-gated. Once settlePRReadyScan flips the PR out of draft, the very next poll’s existing merge attempt sees a non-draft mergeable_state and proceeds normally — fixing retryability at the source is sufficient to unblock the direct-merge fallback described in this section’s Trigger.
State transitions:
| Before | Trigger | After | Labels Added | Labels Removed |
|---|---|---|---|---|
Any stage, mark_pr_ready_on_complete: true |
markPRReady’s client.MarkPRReady call fails (non-transient, or retry-exhausted) |
Same stage, awaiting pr-ready (marker present); item continues advancing normally | fabrik:awaiting-pr-ready |
— |
| Awaiting pr-ready | Settle pass succeeds (PR marked ready, or found already not-draft/closed/merged/missing) | Settled | — | fabrik:awaiting-pr-ready |
| Awaiting pr-ready | A later markPRReady call succeeds directly (self-heal) |
Settled | — | fabrik:awaiting-pr-ready |
| Awaiting pr-ready | Settle pass fails Attempts >= MaxRetries times |
Paused | fabrik:paused |
fabrik:awaiting-pr-ready |
References: ADR-1582: Durable Mark-PR-Ready Retry, ADR-1097: Non-Default-Base Explicit Close Retry (§6.13, the closest structural precedent), ADR-061: Merge-Train Singleton Member-Issue Close Retry, issue #599 (the in-process retry this feature’s durable follow-up builds on), issue #1582 (this feature).
6.23 Autonomy-Label Live Re-Read at Both Advance Decisions (#1769)
Trigger: An operator adds or removes fabrik:yolo/fabrik:cruise on an issue while a stage is in flight, intending to change the advance/auto-merge decision the engine is about to make for that issue.
The problem this section fixes. Two independent advance paths read hasYoloLabel(item)/hasCruiseLabel(item) off item.Labels at the moment they decide whether to advance the stage (and, at Validate, whether to enable auto-merge):
- Path 1 —
handleStageComplete(engine/stages.go), which runs synchronously right after a stage’sFABRIK_STAGE_COMPLETE.itemhere is the pre-invocation dispatch-time snapshot, which can be stale by tens of minutes on a long-running stage. - Path 2 — the catch-up loop’s
runCatchUpPhase2(engine/poll.go), reached wheneverstage:<name>:completeis already present — which is everywait_for_ci: truestage, sincehandleStageCompletedefers to the catch-up loop there (§6.5) rather than reaching its own advance logic. On the default stage set this is Validate, where advancing under withdrawn authority means auto-merging the PR.
Before this fix, Path 1 attempted a mid-run label re-fetch but targeted e.cfg.Owner/e.cfg.Repo — empty in multi-repo mode (a 404, silently swallowed by an err == nil && guard) and, on a single-repo instance whose config repo differs from the item’s own, the wrong issue’s labels entirely. Path 2 performed no re-fetch at all: item.Labels is not part of the deep-fetch cache contract (copyDeepFieldsFromState excludes Labels), so even an item that went through a “fresh” deep fetch earlier in the same poll pass could still carry stale autonomy labels at the point runCatchUpPhase2 reads them. Both defects meant a label removal made mid-run was never observed by either path — the stage would complete and auto-advance (or auto-merge) under authority the operator had already withdrawn, only picking up the change on the next full board refresh.
The fix: one shared helper, consulted by both paths. refreshAutonomyLabels(item *gh.ProjectItem) (engine/stages.go) performs a live FetchLabels call scoped to the item’s own resolved owner, repo (itemOwnerRepo, never e.cfg.Owner/e.cfg.Repo) via e.client (the raw, uncached GitHubClient — never e.readClient, since Labels is not part of that cache’s deep-field contract regardless), and mutates item.Labels in place. Both handleStageComplete and runCatchUpPhase2 call it as the very first step of their autonomy-decision logic, before any hasYoloLabel/hasCruiseLabel read — mirroring effectiveReviewAuthority/effectiveExpectedReviewers’s “one shared helper, every gate” precedent (ADR-1250, ADR-1283, engine/reviews.go), except — unlike those two, which only resolve a label already present on the snapshot — this helper performs the fetch itself.
On a fetch error, item.Labels is left untouched and a warning naming the item and the error is logged — silent fallback is exactly what hid this defect, and does not survive the fix. On a successful fetch, item.Labels is replaced unconditionally, including when the fresh result is an empty label set: a real fetch returning zero labels is data, not a failure, and must not be discarded the way the old len(freshLabels) > 0 guard discarded it.
One fetch covers every downstream autonomy-label read. attemptMergeOnValidate (the single landing-decision function both paths call — ADR-1216) receives item by value and performs its own hasCruiseLabel(item) check. Because both callers invoke refreshAutonomyLabels upstream of every attemptMergeOnValidate call site, that check is fresh too, without a second REST call from this fix — bounding the autonomy-label fetch cost to exactly one live FetchLabels per handleStageComplete/runCatchUpPhase2 invocation (i.e. at most once per item per poll, since the two paths are mutually exclusive admission for the same item in the same pass). This is narrower than “one fetch total,” though: attemptMergeOnValidate already performs its own live re-read for an unrelated purpose — the pre-existing ADR-1419 FetchItemDetails dependency guard (for BlockedBy) — and that call’s GraphQL query resets item.Labels as a side effect, overwriting the REST-fresh snapshot refreshAutonomyLabels just produced. On the Validate/yolo merge path, item.Labels is genuinely fetched twice per invocation, not once. This is harmless today (the dependency guard runs after both the hasCruiseLabel check and the fabrik:auto-merge-enabled idempotency check, and nothing later in the function reads hasYoloLabel/hasCruiseLabel), but a future label check added after the dependency guard must not assume it is reading the REST-fresh snapshot.
Code path: refreshAutonomyLabels (engine/stages.go); called from handleStageComplete (engine/stages.go) and runCatchUpPhase2 (engine/poll.go).
Precedent: this is the same class of fix as checkDependencies’s live BlockedBy re-read (ADR-1419) and reviewGateBlocksLanding’s live review-state re-read (ADR-1216) — a decision with real consequences (advance, or at Validate, merge) must not be made on a cached snapshot that a concurrent operator action can invalidate.
References: ADR-1769: Live Re-Read of Autonomy Labels at Both Advance Decision Points, ADR-1419: Cross-Repo Spawn Servability and Mid-flight Recognition, ADR-1216: Review Gate at Landing Decision, ADR-1250: Review Authority Orthogonal to Autonomy, ADR-1283: Declared Unrequested Reviewers, issue #1768 (originating report), issue #1769 (this fix).
6.24 Merge-Train Base Sanity Check (#1773)
Trigger: In #1688, the merge train opened an integration PR against a protected main while
every member of the batch carried base:develop, and it self-merged nine minutes later with no
human in the loop — a production incident. #1772 fixes the specific cause (an unhydrated member
silently partitioning to the default base at grouping time). This is a deliberately independent
second line of defence: a check on the outcome the train is about to act on, immediately before
it acts, rather than another cause-specific fix — this is already the second time this class of
defect (ADR-1647/ADR-1648 fixed the first, #1637/#1646) has reached production, so a check that
catches the outcome regardless of cause is warranted.
Mechanism: fires only on the default partition, reads live, refuses rather than repairs.
refuseIfBaseContradictsMembers(owner, repo, baseBranch, trainKey string, members []trainMember)
bool (engine/merge_train.go) is called immediately before either of the two landing paths that
can mint a new integration/landing PR:
isDefaultPartitionKey(owner, repo, trainKey)—trainKey == owner+"/"+repo— decides whether the pinned base is the repository default. This is exactlymergeTrainKey’s own default-sentinel condition (§ above, ADR-1648), not an approximation of it, so it needs no additional git call and costs nothing on abase:<branch>-partitioned train (R5): the check returns immediately, false, without touchinge.clientat all.- Only when triggered, each member’s labels are read live via
e.client.FetchLabels— nevere.readClient(boardcache.CacheImpl.FetchLabelsserves the cached snapshot whenever the cache isn’t paused) and never the member’s own cachedtrainMember.item.Labels. This is the entire point of the check (R2): the #1688 incident shape is exactly a cached/stale label disagreeing with a live one, so a check reusing either cached path would have passed cleanly in that incident. nonDefaultBaseLabelValue(labels []string, pinnedBase string) stringscans for abase:<branch>label whose value is non-empty and differs frompinnedBase. It deliberately does not resolve remote branch existence or fall back to a default the waybaseBranchForItem(§6.13/§6.21) does — R3 forbids repairing a contradiction found here, only refusing.- Any contradiction, or any per-member
FetchLabelserror (treated identically — fail-closed, matching #1772’s own posture), causes the caller to refuse: no PR is opened or reused, the member(s) stay in Queued, ande.logfReponames the pinned base and every contradicting member with its declared base (R4). A healthy train — including the ordinary all-default-base case — produces no new log output at all.
Both CreatePR sites that can mint a new PR are covered; the singleton fast path is exempt by
construction. landMergeTrainBatch calls the check immediately after the closed-unmerged-trial
escalation and before its if integrationPR != nil { reuse } else { create } split — covering both
the PR-open and PR-reuse branches uniformly, since a reused PR (a promoted draft CI PR) is just as
capable of carrying a stale pinned base as a freshly-created one. The check is skipped, however,
when the found integrationPR is already merged (integrationPR != nil && integrationPR.Merged
— the FR-2 restart-safety state, e.g. an engine restart between merging the integration PR and
advancing its members to Done): nothing is being opened or reused in that state, so refusing there
would only strand an already-landed batch in Queued forever on a transient FetchLabels error or a
label edited after the merge, blocking FR-3’s advance-to-Done/close-PR logic for no protective
benefit — found during PR review and fixed in the same PR (TestLandMergeTrainBatch_AlreadyMergedPR_SkipsBaseCheck).
landSingleton (the FR-5 one-at-a-time fallback, not a rare corner case — any bisection failure
routes through it) calls the check immediately before its own CreatePR call; it never searches for
or reuses an existing PR at all (by design — see the ADR-059 D4 data-loss note in its own doc
comment), so it has no equivalent already-merged state to guard against. The singleton fast path
(trySingletonFastPath/finishSingletonFastPathLanding, §6.20) never calls CreatePR at all — it
merges the member’s own existing PR via MergePRAtHeadSHA, targeting whatever base that PR was
already opened against — so it structurally cannot reproduce this contradiction and needs no
equivalent check.
Self-healing, not sticky. The check carries no state of its own across polls — a refusal simply returns without opening or reusing a PR, leaving the batch’s members in Queued exactly as they were. The next poll re-forms the batch and re-runs the check from scratch; once the underlying label disagreement resolves (by any means — a human fixing a stale label, a cache entry expiring, or #1772 preventing the bad partition in the first place), the very next attempt proceeds normally.
No new label, no escalation machinery. Unlike most of this file’s other guard mechanisms, a
refusal here is not durably marked on the board — an operator distinguishes it from ordinary
merge-train activity by the merge-train-tagged log line alone. This is a deliberate, narrower
scope than the pattern most fabrik:awaiting-* siblings follow (§6.9–§6.22): the acceptance
criteria for this feature are fully satisfiable with logging and self-healing alone, and building a
durable repeat-counter + fabrik:paused escalation now would be speculative state for a scope not
yet shown to need it (see ADR-1773’s Rejected Alternatives).
Independent of #1772, deliberately. #1772 fixes the partition-time cause of a member ending up in the wrong batch; this check fires at landing time, downstream of and structurally unrelated to that fix, sharing no code. With #1772 also applied, a member it excludes at partition time never reaches this check at all (there is nothing left to contradict); this check still fires for any other cause of the same pinned-base-vs-declared-base disagreement, including causes not yet discovered — which is the entire reason this is a second, independent line of defence rather than a refactor of the first.
Code path: refuseIfBaseContradictsMembers, isDefaultPartitionKey, nonDefaultBaseLabelValue
(engine/merge_train.go), called from landMergeTrainBatch and landSingleton.
State transitions:
| Before | Trigger | After | Labels Added | Labels Removed |
|---|---|---|---|---|
| Queued batch, pinned base == repo default, all members agree (or genuinely default-base) | refuseIfBaseContradictsMembers returns false |
Landing proceeds normally (open or reuse the integration/landing PR) | — | — |
| Queued batch, pinned base == a non-default partition base | isDefaultPartitionKey is false — check never fires |
Landing proceeds normally, no FetchLabels calls made |
— | — |
| Queued batch, pinned base == repo default, ≥1 member’s live label declares a different base | refuseIfBaseContradictsMembers returns true |
No PR opened or reused; members remain in Queued; loud log line naming base + members | — | — |
Queued batch, pinned base == repo default, a member’s live FetchLabels call errors |
refuseIfBaseContradictsMembers returns true (fail-closed) |
No PR opened or reused; members remain in Queued; loud log line naming base + members | — | — |
landMergeTrainBatch found an existing integration PR that is already merged (FR-2 restart-safety state) |
integrationPR != nil && integrationPR.Merged — the check is skipped entirely, regardless of member labels |
FR-3 (advance to Done, close member PR/issue) proceeds normally; no FetchLabels calls made, no refusal possible |
— | — |
References: ADR-1773: Merge-Train Base Sanity Check at Landing Time, ADR-1647: Merge-Train Non-Default-Base Exclusion, ADR-1648: Merge-Train Batches Partitioned Per (Repo, Base), §6.13 (baseBranchForItem’s remote-resolution semantics, deliberately not reused here), §6.20 (the singleton fast path, confirmed exempt), issue #1688 (the community report that motivated this issue), issue #1773 (this feature).
6.25 Merge-Train Own-PR CI Admission Gate (ADR-1821)
Trigger: Issue #1821. fetchTrainMembers validated exactly two things per Queued member — a linked PR exists and has a head SHA — and never consulted the member’s CI. A member whose own PR checks were already red was therefore admitted, merged into a trial, and only discovered by that trial’s CI (or by bisection): on verveguy/concept-maps two such members cost roughly 2 hours each (an ~54-minute assembly, a 24-minute CI cycle, then re-bisection) to rediscover a fact that was readable off the members’ own check-runs in one API call. classifyLandingCI already answers the question and is already trusted for a landing decision (§6.20 step 4), but was only applied on the singleton fast path, never at admission.
Mechanism: a fail-open admission gate at fresh batch formation. admitTrainMembers (engine/merge_train_admission.go) is called from prepareTrainWorker immediately after fetchTrainMembers. For each member it reads e.client.FetchCheckRuns(owner, repo, m.headSHA) — the live REST client, at the head SHA fetchTrainMembers snapshotted, so the evidence corresponds to the exact commit that would enter the trial — and classifies it with classifyLandingCI, unmodified. The gate has no notion of red of its own: “confirmed red” is exactly TrainCIRed, which includes a required-context failure.
Polarity is fail-open — the inverse of §6.20. The singleton fast path skips validation, so it authorises only on positive evidence and every ambiguity blocks it. The gate removes a member from a batch, so it acts only on positive evidence of failure and every ambiguity admits:
classifyLandingCI / read outcome |
Gate action |
|---|---|
TrainCIRed (failed check run, or required-context failure) |
Defer (below) |
TrainCIPending (incl. a rerun in progress masking an older failure) |
Admit |
TrainCIGreen |
Admit |
| Zero check runs, no required-context failure | Admit (the fast path’s fail-closed zero-run rejection is deliberately not copied) |
FetchCheckRuns error |
Admit, logged as fetch-error |
| Context cancelled mid-gate | Admit the remainder unchecked |
Admitting is byte-identical to pre-#1821 behavior. Deferring on ambiguity would silently drain the queue and strand members that have done nothing wrong; the cost of the fail-open choice is that a red member with a pending run, a zero-run head, or an unreadable read is discovered by the trial exactly as it always was.
Scope: fresh formation only (R7). The gate lives in prepareTrainWorker, after reconstructTrainState has returned true for both restart routes (completeDeferredLanding of an already-merged marker PR, and resumeTrain of an existing in-flight trial), and it is not inside fetchTrainMembers itself. Those routes re-resolve an already-decided batch and must not re-litigate admission. Bisection sub-trials, landOneAtATime, and landGreenBatch never pass through it either. The gate needs state.projectID for the reroute, which fetchTrainMembers does not have.
Precondition: the reroute target must have wait_for_ci (R9). A deferred member is rerouted to stageBeforeHolding (normally Validate) with its stage:<X>:complete label intact. What re-picks it up is Phase 1’s handleMergeAndCIGates: settlePRMergeState classifies red checks as PRMergeBlocked, checkCIGate returns ciFailure, fabrik:awaiting-ci is applied and dispatchCIFixReinvoke runs (bounded by MaxCiFixCycles, with the no-op-SHA debounce). Phase 2 — the only place attemptMergeOnValidate → advanceToQueued runs — is never reached for a claimed item, so a red member is not advanced straight back to Queued, even under fabrik:yolo. But checkCIGate is a no-op unless the stage has wait_for_ci: true; without it nothing would claim the member (under yolo it would bounce back to Queued and be re-deferred every poll; under cruise it would strand). The gate therefore checks the target stage first and, if it lacks wait_for_ci, admits every member without reading anything, logging once per formation. The shipped and live validate.yaml both set wait_for_ci: true; a custom config that omits it simply gets no gate.
Deferral (deferRedMember). Modelled on ejectRedSingleton (#1545) minus the pause:
- The member is dropped from the batch unconditionally — it is confirmed red.
rerouteQueuedMemberOffHoldingruns first (the ADR-1208 reroute-before-side-effects primitive). On failure nothing is posted, consumed, or recorded; the next poll’s fresh formation retries the whole operation. The member stays out of the current batch regardless.- On success, any pending review-finding eject signal for the member (
takePendingReviewEject, §6.16) — and any pending unprocessed-comment eject signal (takePendingCommentEject, #1863) — is consumed and logged, so a member flagged both ways is never double-routed or double-commented and a stale signal cannot later fire on a re-queued member that already addressed the finding. - One comment is posted: header
🏭 **Fabrik merge-train — deferred (own CI failing)**, the failing check(s) viarenderFailedChecks(falling back to the classifier’sdetailfor a required-context red, which has no failing runs to render), a statement that the failure is the PR’s own and not a train interaction, and that the member has not been paused — the CI gate will dispatch a fix from the target stage and the issue re-queues once CI is green and that stage completes again. It is composed locally:ejectMember(stayInQueue=false)hard-codes review-finding wording, andrenderDiagnosticBlock/renderBatchContext/reentryInstructioncarry trial/integration-PR wording and a pause-assuming recovery instruction that are wrong for this cause.
Never paused, never counted (R5). A deferral does not touch mergeTrainEjectionCounts and never calls pauseMergeTrainMember, so it cannot accumulate toward MaxMergeTrainEjections. This is the opposite of ejectRedSingleton’s reasoning: there the failure was only ever observed on a synthetic trial branch, so nothing external would re-detect it and a pause was the only human gate. Here the failure is observable on the member’s own PR check-runs, so the ordinary CI gate has an external signal to act on.
Ping-pong backstop (R9). mergeTrainCIDeferred (owner/repo#N → head SHA, in memory, mergeTrainCIDeferredMu) records the SHA each member was last deferred at. A repeat deferral of the same SHA — reachable only if the live read shows red while Phase 1’s cache-backed read does not — still reroutes but posts no second comment. The entry is cleared when the member is next admitted non-red. It resets on restart, costing at most one duplicate comment.
Cost. One FetchCheckRuns per member per fresh batch formation, sequentially, bounded by MaxBatchSize. A pending or timed-out trial re-forms the batch on the next poll, so the read repeats per member per poll until a landing or eject. classifyLandingCI additionally calls FetchCombinedStatus (cache-backed) only when required_status_contexts is configured and a required name is absent from the check runs. A one-member batch reads check runs again in singletonFastPathEligible. The small per-poll cost is set against ~2 hours avoided per red member. Every disposition — defer, or admit with its reason (pending, green, fetch-error, no-wait-for-ci) — is logged with the classifier’s detail.
A new sustained label state. A deferred member carries both stage:<X>:complete and fabrik:awaiting-ci for the whole CI-fix cycle (classifyCIFromCheckRuns adds the label but never removes the completion label). Both the main Phase 1 loop and settleAwaitingCIScan (ADR-1270) run the same handler chain on it; duplicate dispatch is prevented by the snap.Worker() != nil guard and duplicate pause comments by hasCIGatePauseComment. MaxCiFixCycles and the CIBackstopTimeout can still pause a slow fix — an ordinary Validate pause, not a deferral pause.
Crash window. The reroute and the comment are two calls; a crash between them leaves a rerouted member with no comment, matching ADR-1208 and ADR-1545.
Code path: admitTrainMembers, deferRedMember, composeCIDeferralBody (engine/merge_train_admission.go), called from prepareTrainWorker (engine/merge_train.go).
State transitions:
| Before | Trigger | After | Labels Added | Labels Removed |
|---|---|---|---|---|
Queued member, fresh batch formation, reroute target has wait_for_ci |
classifyLandingCI → TrainCIRed on the member’s own head; reroute succeeds |
Rerouted to the stage before Queued; excluded from the batch; one comment; ordinary CI gate takes over | — | — |
| (as above) | TrainCIRed; reroute fails |
Queued (unchanged), excluded from this batch; no comment, no dedupe record; whole operation retried next poll | — | — |
| (as above) | TrainCIRed again at the same head SHA (e.g. stale-cache bounce) |
Rerouted again; no repeat comment | — | — |
| (as above) | Pending, green, zero check runs, or FetchCheckRuns error |
Admitted to the batch, unchanged from pre-#1821 | — | — |
Queued member, reroute target lacks wait_for_ci |
Gate skipped before any read | Admitted, unchanged | — | — |
Queued member, restart reconstruction (Route 1 / resumeTrain), bisection, landOneAtATime, landGreenBatch |
Never reaches the gate | Unchanged | — | — |
| Deferred member at the reroute target, red checks | Phase 1 handleMergeAndCIGates → ciFailure |
CI-fix reinvoke dispatched; not advanced (Phase 2 unreached) | fabrik:awaiting-ci |
— |
References: ADR-1821: Merge-Train Own-PR CI Admission Gate, ADR-1644: Merge-Train Singleton Fast Path (the inverse fail-closed polarity), ADR-1208: Queued Review-Finding Ejection (reroute primitive), ADR-1545: Red-Singleton Reroute Before Pause, ADR-1420: Merge-Train Ejection Diagnostics, §6.16 (applyPendingReviewEjects), §6.20 (the singleton fast path), issue #1821
6.26 Merge-Train Conflict Resolution Replay via git rerere (ADR-1834)
Trigger: Issue #1834. A merge-train conflict resolution lives only on its trial branch — cleanupTrialArtifacts deletes the trial worktree and branch after every trial, including bisection sub-trials, win or lose. When a trial is abandoned (bisected, re-formed after an eject, rebuilt after the base moves), the resolution is deleted with it, and the next assembly pays a fresh Claude invocation for the byte-identical conflict. On one operator’s repo, the same file’s conflict was resolved twice in 80 minutes across two trials for the same members.
Mechanism: git rerere enabled repo-wide, replay detected via the merge’s own output. enableRerere (engine/worktree.go) sets rerere.enabled/rerere.autoupdate on the bare clone, called from ensureBareClone alongside the existing setCommitterIdentity — every worktree derived from that bare clone (regular per-issue worktrees and merge-train trial worktrees alike) shares the enablement and the rr-cache it writes to. For a bare repo, bareDir is $GIT_DIR, so <bareDir>/rr-cache is already scoped correctly and survives CleanupTrainWorktree’s git worktree remove/git branch -D untouched — no new cache-location plumbing was needed.
Scoping is repo-wide, not worktree-scoped — a correction made during implementation. The originally planned narrower approach — enabling rerere only for merge-train trial worktrees via Git’s per-worktree config extension (extensions.worktreeConfig + git config --worktree) — was implemented, then reverted after direct reproduction showed it is unsafe: turning on extensions.worktreeConfig for a bare clone makes core.bare (true, a bare repo’s own top-level value) the extension’s shared/common config value, and any linked worktree that doesn’t explicitly override it inherits that value — breaking every ordinary git operation in that worktree with fatal: this operation must be run in a work tree. Since Fabrik’s every worktree for a given repo — every open issue’s worktree, not just merge-train trials — shares one bare clone (NewWorktreeManagerForRepo), this would have broken every currently-open worktree for the repo the instant the extension was first turned on, not merely the newly-created trial worktree. Repo-wide enablement carries no such risk: a rerere lookup with no recorded resolution is a pure no-op, so a regular per-issue worktree’s Validate-stage rebase conflict is, at worst, itself replayed for free if it happens to exactly match a merge-train-recorded resolution — a strict improvement, never a regression.
git merge always exits non-zero on a conflict, even when rerere fully replays it. Git’s merge machinery invokes rerere synchronously inside git merge itself: it hashes each conflict hunk and, given a recorded resolution, rewrites the file in place. With rerere.autoupdate=true, a file whose every hunk was replayed is also automatically staged — but git merge still reports a conflict and a non-zero exit regardless of how much rerere already resolved, so mergeErr != nil in assembleTrialBranch is uninformative on its own about whether anything is actually left to do.
Three-way dispatch in resolveTrainConflict, keyed on the post-merge unmerged-path set:
unmergedPaths after the failed merge |
Dispatch |
|---|---|
Empty, and no declared generated path was named in the merge’s own CONFLICT (...) output |
commitRerereReplayedMerge finalizes directly (git add -A + git commit) — no Claude invocation |
Empty, but a declared generated path was named in the merge’s CONFLICT (...) output |
regenerateAndCommit runs the declared regeneration command fresh — no Claude invocation, and the replayed content is never trusted (see the ADR-1235 interaction below) |
| Non-empty | Unchanged from before this issue: classifyConflictedPaths runs against whatever remains, dispatching Claude for the non-generated remainder (if any) and/or regeneration for generated paths, exactly as ADR-1235 already specified. Claude’s own git status naturally shows only what rerere left behind — buildTrainConflictComment was never handed a precomputed path list for the plain case, so no change was needed here at all |
The empty-unmergedPaths cases above are the ones this issue adds: previously, an empty result fell through to the same “no generated paths in the conflict” branch as an ordinary clean-Claude-dispatch case, invoking Claude even when there was nothing left to resolve.
The ADR-1235 interaction: rerere has no awareness that a path is a declared generated file. If a conflict confined to a declared generated path (e.g. docs/llms-full.txt) was already resolved via regenerateAndCommit once, and the identical conflict recurs, rerere.autoupdate would — left unchecked — silently replay and re-stage that old committed content before resolveTrainConflict’s unmergedPaths-based classification ever ran, so matched would come back empty and the regeneration command would never re-run, landing stale content instead of ADR-1235’s guaranteed fresh regeneration of the trial’s actual current sources. The fix is conflictedGeneratedSpecsFromMergeOutput (engine/generated_files.go): git always prints a CONFLICT (...) line naming every originally-conflicted path, independent of whether rerere then silently resolved it, so a substring check of each declared generated path against those lines recovers the conflict’s original membership regardless of what unmergedPaths shows now. resolveTrainConflict unions that recovered set with whatever classifyConflictedPaths finds in the current (post-rerere) unmerged set via unionGeneratedSpecsByPath, before any dispatch decision — so a declared generated path is always force-regenerated, whether it’s still genuinely unmerged or was already silently replayed.
Trust boundary is unchanged (Requirement 4). A rerere replay is committed and pushed exactly like a fresh Claude resolution, then validated by the trial’s own combined CI — the same gate, with no weakening. Nothing about being “already resolved before” skips or shortens downstream validation.
Red-trial resolution hygiene: forgetPoisonerResolutions (Requirement 5). Bisection sub-trial worktrees are destroyed (cleanupTrialArtifacts) immediately after each sub-trial’s own CI result is known — before the poisoner is identified, since bisection only learns the poisoner once every sub-trial has run. This means the porcelain git rerere forget <path> command (which re-derives the current conflict’s hash from a live conflicted working tree — it cannot be pointed at an arbitrary historical hash from outside a live conflict) can’t be run against the sub-trial that actually observed the poisoner’s conflict; that worktree is already gone. Once handleRedBatch isolates the poisoner, forgetPoisonerResolutions reconstructs a fresh live conflict instead — matching the episode’s original merge order, not merely poisoner-vs-base: assembleTrialBranch merges members strictly in order into one accumulating branch, so the conflict that actually produced a poisoner’s recorded resolution is usually poisoner vs. the trial-so-far (base plus every earlier-in-order red member already merged in), not poisoner vs. base alone — a solo poisoner-vs-base reconstruction can silently miss the resolution entirely (a review caught this in the initial implementation; see ADR-1834’s Decision 4). forgetPoisonerResolutions takes red (the full episode’s red batch, in its original order — bisect only ever slices it, never reorders it) and poisoner’s position idx within it: it replays red[:idx] into a disposable worktree forked off the same pinned p.baseSHA every trial in the episode used (each earlier member merged with --no-ff --no-edit, relying on rerere.autoupdate to silently replay that member’s own already-recorded resolution — never invoking Claude; an earlier member that doesn’t fully replay aborts the reconstruction and skips forgetting, fail-safe), then merges poisoner.headSHA last. If that final merge conflicts, git rerere forget runs for every path named in that merge’s own output (conflictedPathsFromMergeOutput, the same mergeOut-derived technique as the ADR-1235 guard above, not unmergedPaths: once rerere’s autoupdate has replayed and staged a path, git status no longer reports it as unmerged even though there is still a recorded resolution worth forgetting there). If it doesn’t conflict at all, there is nothing to forget for this episode’s primary trial order — silently, with no warning logged. This is a narrower guarantee than “every resolution poisoner was ever party to”: a bisection sub-trial that combined poisoner with a different partial subset of red can record a distinct resolution this reconstruction doesn’t replay and so can’t forget — accepted, not solved, since the unchanged CI trust boundary still gates any such stale replay before it lands. Entirely best-effort throughout: every git failure here is logged and none of them affect the eject that already happened. Skipped entirely under the trainValidateFn test seam, which never runs assembleTrialBranch’s real git assembly in the first place, so there is no real rr-cache entry from that episode to forget.
Pruning: periodic git rerere gc. Nothing in the codebase invoked git rerere gc before this issue, so gc.rerereResolved/gc.rerereUnresolved’s default expiries (60 days / 15 days) were moot and rr-cache would grow unbounded. runMergeTrainWorker now runs it once per worker invocation, immediately after prepareTrainWorker succeeds (mirroring that function’s own base-SHA-pinning gate: skipped under the trainValidateFn test seam), best-effort. The defaults themselves are accepted as-is — rr-cache entries are small text blobs and the merge-train’s own observed reuse window (minutes to weeks, per the motivating incident) comfortably fits inside 60 days.
Accepted, not mitigated: concurrent rr-cache writers. Two merge-train workers for the same repo but different (repo, base) partitions (ADR-1648) can run truly concurrently, each in its own trial worktree off the same shared bare clone, both writing into the same rr-cache with no mutex serializing the git merge/git commit sequence between them. Git’s rr-cache writes are per-hash-directory, so two different conflicts recorded concurrently don’t collide; the only real race is two workers resolving the identical conflict hash at the same moment — narrow, low-probability, and low-severity (worst case, one valid resolution’s postimage is overwritten by another, still gated by the unchanged CI trust boundary above). No code mitigates this; it is a conscious trade-off, not an oversight.
No configuration toggle. rerere is unconditionally enabled whenever a bare clone is created or repaired; there is no merge_train_rerere-style opt-out, matching the issue’s own requirements (which never mention one).
Interaction with #1841’s turn-limit/prompt changes. #1841 changed resolveConflictWithClaude’s invocation shape (dropped the build/test verification instruction, added turn-limited-exit handling and InvokeOptions.NoResume) but touches none of the three-way dispatch table above: rerere’s replay-or-not decision (resolveTrainConflict’s unmergedPaths/conflictedGeneratedSpecsFromMergeOutput classification) runs entirely before resolveConflictWithClaude is ever called for the plain and mixed cases, so it is unaffected by what Claude is told or how a turn-limited exit is subsequently classified. A conflict rerere fully replays still short-circuits to commitRerereReplayedMerge/regenerateAndCommit with no Claude invocation at all, exactly as before.
Code path: enableRerere (engine/worktree.go); conflictedGeneratedSpecsFromMergeOutput, unionGeneratedSpecsByPath (engine/generated_files.go); commitRerereReplayedMerge, resolveTrainConflict, conflictedPathsFromMergeOutput, forgetPoisonerResolutions (engine/merge_train.go, called from handleRedBatch), the git rerere gc call in runMergeTrainWorker.
References: ADR-1834: Merge-Train Conflict Resolution Replay via git rerere, ADR-1235: Generated-File Regeneration in Merge Train (the interaction this issue must preserve), ADR-059: Internal Merge Train (D3 conflict resolution, D4 bisection, D-e non-isolable interaction), ADR-1648: Merge-Train Per-Base Partitioning (the accepted concurrent-writer surface), ADR-1833: Deterministic Queued Batch Ordering (the prerequisite for rerere’s hit rate), issue #1834
6.27 Merge-Train Trial Prefix Reuse (ADR-1835)
Trigger: Issue #1835. A merge-train trial is a linear chain of merge commits — pinned base → +m1 → +m2 → … → +mn (assembleTrialBranch). Whenever a trial is abandoned — a bisection sub-trial, a re-form after ejecting a poisoner, or any other re-assembly — the whole chain was discarded and the next assembly started again from the pinned base, redoing merge work (including Claude conflict resolution) for members whose result was still valid: everything before the first changed input is completely unaffected by that change. The motivating incident (verveguy/concept-maps, 2026-09-20) shows bisection’s first probe landing on exactly the first four merges of the red trial’s own eight-member chain — a prefix by construction — yet the train rebuilt it from base anyway, re-paying 4.5 minutes of Claude conflict resolution for a commit that had existed minutes earlier.
Mechanism: a content-addressed hash chain, scoped to one worker invocation. trainPrefixCache (engine/merge_train_prefix.go) records h0 = chainHashSeed(trainKey, baseSHA) — computed per lookup from the baseSHA passed to lookup(p.baseSHA, members), never fixed at construction, because landOneAtATime’s per-singleton re-pin and the main-moved rebuild loop both change p.baseSHA mid-invocation while sharing the cache — and h_i = chainHashStep(h_{i-1}, member[i].Number, member[i].headSHA) for each member that merges successfully; only successful merges advance the chain — an ejected member (either assembleTrialBranch’s own mid-assembly unresolvable-conflict ejection, or bisection’s post-CI poisoner ejection) contributes no step, so the next successful member’s hash is computed against whatever hash preceded the ejected one. Each h_i maps to the resulting merge commit’s SHA. assembleTrialBranch looks up the new member list’s longest matching prefix by walking the identical recurrence forward from h0 and stopping at the first miss; on any match it forks the trial worktree (EnsureTrainWorktreeAt) from that prefix’s commit instead of the pinned base SHA, pre-seeds survivors with the matched members, and merges only what’s left. This single mechanism, with no separate invalidation logic, covers every re-assembly path named in the issue:
- Bisect’s first half (
red[:mid],bisectnever reordersred) is by construction an exact prefix of the red trial’s own chain — full match, zero merges, zero Claude invocations. - Re-form after ejecting the isolated poisoner at position k — the survivor list is
redminus the poisoner, order preserved — matches positions1..k-1and stops exactly there (the poisoner’s own hash step is absent from the survivor list’s walk), sobase..m(k-1)is reused and only members after the eject point are merged. - A changed pinned base SHA, or a member re-pushed to a new head SHA, simply produces a lookup miss at that point (a different
h0— including a mid-invocation re-pin ofp.baseSHAon the same live cache — or a differenth_ifrom that position onward) — the prefix ends there and the rest re-merges normally, with ADR-1834’s rerere replaying any conflict resolution that didn’t involve the changed input.
Scope is one merge-train RUN (#2051: the cache belongs to the run, so it survives the worker goroutine exiting between bisection steps) — never persisted across restarts. The cache is constructed right after prepareTrainWorker succeeds (same spot as the existing git rerere gc call, skipped identically under the trainValidateFn test seam, which never runs real git assembly at all) and threaded through trialParams.prefixCache to every trial the invocation assembles — the main-loop re-form and every bisection sub-trial share the same pointer. p.baseSHA and every member’s headSHA are pinned/fetched exactly once per worker dispatch and held constant for its whole lifetime, so a cache that only lives as long as those values are guaranteed stable needs no cross-invocation staleness tracking (“did a member land or leave Queued since this was recorded?”) at all. A restart, or even the very next poll’s fresh worker dispatch, starts from a nil-equivalent cache — by construction this can only ever reproduce full re-assembly, never a wrongly-trusted reuse, since there is no persisted state to misapply.
Reachability: non-pushed refs, gc-protection only. Each successful merge is recorded into the in-memory entries map (the only thing lookup ever consults for correctness) and, in the same step, as a git ref under refs/fabrik/merge-train-prefix/<sha256(trainKey)>/<h_i> in the shared bare clone — never pushed to origin — so git gc cannot prune a commit the cache might still reuse. The ref’s existence is never itself evidence of validity; it exists purely so the commit survives external garbage collection while the in-memory entry that governs reuse is still live. At the start of a worker invocation, sweepStaleRefs deletes everything under this trainKey’s own ref directory before any lookup or record — mopping up refs a prior invocation left behind if it crashed before reaching its own cleanup — and at the end of every invocation (any exit path, via defer), the same sweep removes everything the invocation created, unconditionally: nothing will ever look up this invocation’s chain again once it’s gone. The ref directory is keyed by a hash of trainKey alone, independent of the pinned base SHA — this makes the crash-recovery sweep provably safe against a concurrently-running sibling (repo, base) partition sharing the same bare clone (ADR-1648): the in-flight guard already guarantees no other goroutine is using this exact trainKey right now, and a different trainKey hashes to a different directory.
Trust boundary is unchanged (Requirement 7). A reused prefix commit is pushed as a new trial branch and validated by the trial’s own combined CI exactly like any freshly-assembled one — this saves merge work, not CI runs, and a wrong prefix match would at worst waste a CI run on a wrongly-composed trial, never silently land anything incorrect.
forgetPoisonerResolutions (ADR-1834) is deliberately not rewired to consume this cache — it’s a best-effort hygiene helper outside this issue’s stated scope, not one of the re-assembly paths above, and it runs after the poisoner has already been ejected and the batch’s trial artifacts are already gone, a different lifecycle than the one this cache covers.
Test-only observation seams. trainPrefixLookupHookFn (matched length, total length, and member numbers at the start of every prefix lookup) and trainMergeAttemptHookFn (fires once per member an assembly actually attempts to merge, whatever the outcome) are both nil in production. The second exists because ADR-1834’s rerere replay can resolve a repeated identical conflict without ever invoking Claude — so the Claude-invocation counter alone can’t distinguish “the merge never ran at all” (this issue’s reuse) from “the merge ran but its conflict resolved for free” (rerere, unrelated); the merge-attempt hook is the literal, rerere-immune “zero merges” signal. mergeTrainPrefixReuseDisabledForTest (mirroring mergeTrainQueueSortDisabledForTest, ADR-1833) lets a test disable reuse entirely and observe the previously-suppressed merges return, proving the reuse tests are not vacuously true of a scenario that never needed any merge work to begin with.
Code path: trainPrefixCache, newTrainPrefixCache, chainHashSeed, chainHashStep (engine/merge_train_prefix.go); assembleTrialBranch’s prefix lookup/fork/record, trialParams.prefixCache, the cache construction/sweep/defer-cleanup in runMergeTrainWorker (engine/merge_train.go).
References: ADR-1835: Merge-Train Trial Prefix Reuse, ADR-1834: Merge-Train Conflict Resolution Replay via git rerere (the direct dependency and complementary mechanism), ADR-1648: Merge-Train Per-Base Partitioning (the shared-bare-clone concurrency this issue’s ref namespacing accounts for), ADR-1833: Deterministic Queued Batch Ordering (the prerequisite for stable member-list comparison), ADR-059: Internal Merge Train (D-b base-SHA pinning, D4 bisection), issue #1835
6.28 Merge-Train Live-Status Resume Guard (ADR-1871)
Trigger: Issue #1871. Right after the train lands a member, the next batch formation could still see it in Queued and run its landing a second time — a second Done move, close, Landed via … comment and landing-verification write (3 of 25 landings on verveguy/concept-maps, on both landing paths, 5–6 s apart). Cause: poll() reads the board once (engine/poll.go) and reaches handleMergeTrainBatch seconds later with that same object; meanwhile the previous worker moved the member to Done and finishTrain released the in-flight marker, so LoadOrStore succeeds and a fresh worker starts from the pre-landing snapshot. Both “resume an interrupted landing” branches then read Queued off that snapshot as “interrupted”. Their own Status == "Done" / fabrik:awaiting-landing-verification checks read snapshot fields too, so could not catch it.
Mechanism: a live Status read at the two resume decision points. liveLandingState (engine/merge_train.go) reads each member’s Status via e.client.FetchProjectItemStatus(item.ItemID) (falling back to LookupIssueProjectItem when the snapshot has no ItemID) — never e.readClient (boardcache-backed), item.Status or item.Labels; the same live-read discipline as §6.24. It is called from completeDeferredLanding (Route 1 of restart reconstruction, a per-member filter applied before fetchTrainMembers), from the pr.Merged branch of trySingletonFastPath (§6.20), and — via dropLiveLandedMembers — at fresh batch formation in prepareTrainWorker, right after fetchTrainMembers and before admitTrainMembers (§6.25).
| Live read | Outcome | Action |
|---|---|---|
| Status equals the holding stage | liveHolding |
Resume as before — a genuine interrupted landing (crash between merge and Done) or the fabrik:awaiting-advance retry (§6.17), where the member stays Queued |
| Status non-empty, not the holding stage | liveMoved |
The landing already completed (or a human moved the item): skip — no Done move, close, comment or label write; logged |
| Status empty (no error) | liveUnknown |
Resume (no positive evidence of completion) |
| Read error | liveReadFailed |
Defer this poll, logged as deferring resume of #N: live status read failed |
Polarity. Skip only on positive evidence (a confirmed non-empty, non-holding Status); an ambiguous empty Status resumes. A read error fails closed: both resume branches re-run every poll (each poll re-dispatches, reconstructTrainState re-probes, the fast path re-fetches FetchPRDetails), so deferral delays but never strands a member, whereas failing open would keep the duplicate landing exactly when reads are flaky.
Integration-PR path. Members that are moved or read-failed drop out of this poll’s landing; the rest still land (a merged batch PR can have some members Done live and others still Queued). When none remain the existing “no still-Queued members” early return applies and reconstructTrainState still reports handled, so no fresh batch forms that poll. Singleton fast path. liveMoved/liveReadFailed return true (“disposition decided”) — no trial is built and no landing is re-run.
Fresh batch formation. A stale snapshot can also put just-landed members into a fresh batch: when no merged marker PR matches (any landing that carries no batch marker, e.g. a series of singleton fast-path landings), reconstructTrainState returns false and prepareTrainWorker falls through to formation. fetchTrainMembers only checks that a linked PR exists with a head SHA (a merged PR passes) and admitTrainMembers only classifies the member’s own CI (green), so the trial — whose landed members’ heads are already in base, making the merges no-ops — would pass and landMergeTrainBatch would re-land each one. dropLiveLandedMembers therefore drops liveMoved members and excludes liveReadFailed members from this batch only (the next poll retries them); liveHolding/liveUnknown members are kept, so a genuinely Queued member is never dropped. If nothing remains the worker logs “no survivors” and returns. The resume-branch guards stay: they cover the restart case formation never reaches. Cost: one live read per member per fresh formation (including polls where the batch is merely pending).
Consequence for the landing functions’ own Done branches. landMergeTrainBatch’s Status == "Done" branch and finishSingletonFastPathLanding’s Done-status branch (both commented as restart safety) are now reachable only if the Status changes between the guard’s read and their own snapshot check.
Known gap (out of scope, ADR-1871). landSingleton’s dedicated landing PR carries no batch marker, so Route 1 never matches it; a stale-Queued member from that path goes to fresh formation and was not traced. Bisection, landOneAtATime and landGreenBatch share the snapshot source but are not entered from either resume branch. Follow-up to be filed separately.
Test seams. Engine.SetMergeTrainLandingGuardDisabledForTest makes the guard report liveHolding (pre-#1871 behaviour) so the sim scenarios can show the duplicate with the guard off. simgh.Sim.LagBoardStatus makes the bulk board reads keep reporting a held Status/labels while the single-node reads stay truthful.
Code path: landingLiveState, liveLandingState, dropLiveLandedMembers, prepareTrainWorker, completeDeferredLanding, trySingletonFastPath (engine/merge_train.go).
State transitions:
| Before | Trigger | After | Labels Added | Labels Removed |
|---|---|---|---|---|
| Snapshot Queued, live Done (or any other non-holding column), merged PR found | Resume branch runs liveLandingState → liveMoved |
Unchanged; nothing written | — | — |
| Snapshot Queued, live Queued, merged PR found | liveHolding |
Landing completed once (as §6.20 / Route 1) | (as the landing path) | — |
| Snapshot Queued, live read errors | liveReadFailed |
Unchanged; retried next poll | — | — |
| Fresh batch lists a member that is live Done (or any non-holding column) | dropLiveLandedMembers → liveMoved |
Member dropped from the batch; nothing written | — | — |
| Fresh batch lists a member whose live read errors | dropLiveLandedMembers → liveReadFailed |
Member excluded from this batch; retried next poll | — | — |
References: ADR-1871: Merge-Train Live-Status Resume Guard, ADR-059: Internal Merge Train (FR-2 restart safety, unchanged), ADR-1644: Merge-Train Singleton Fast Path, ADR-1773: Merge-Train Base Sanity Check (live-read discipline), §6.20, §6.24, issue #1871
6.29 CI Infrastructure Failures: Startup Failure and Flake Re-run (#2052, ADR 2052)
Trigger: Two failures of the CI provider were read as the members’ fault or as “still pending”. A workflow run that fails before creating any job (startup_failure) leaves zero check runs, which every gate reads as “CI has not reported yet” — pollTrainCI polled it to CIBackstopTimeout (4h) and then threw the trial away, and with no required checks (mergeable_state: clean) the zero-check shortcut even returned green and landed a batch whose CI never ran. A single flaky test made a trial red and started a bisection of full CI cycles that could end by blaming an unrelated member.
Detection (engine/ci_startup.go). isStartupFailure(run): the workflow run is completed and either concluded startup_failure or has zero jobs and concluded failure. Any other zero-job completion is not one: cancelled (a concurrency-group supersession, #2033), action_required (a run held for a human’s approval), skipped, stale, success. Runs come from FetchWorkflowRuns (GET /actions/runs?head_sha=… plus one jobs?per_page=1 read per completed run for the job count), called only on the zero-check-run path. With no startup failure in view, a workflow run that is queued or in_progress holds the zero-check green shortcut (CI is on its way, so the absence of check runs is not completeness); a run in waiting or pending does not, so such a trial behaves as before #2052. startupWatch.observe considers only runs absent from the baseline snapshot taken when the last retrigger was issued (R3a: the dead run stays on the head SHA and would otherwise burn the budget in one loop) and returns none, retrigger, wait or abandon.
Retrigger. retriggerPR calls CloseIssue then ReopenIssue back to back (the reopen is retried reopenAttempts times; a PR that cannot be reopened is reported as an error and the caller abandons). A pull_request reopened event is what starts a fresh run; GitHub refuses to re-run a run that never started (403). The bound is maxCIRetriggers (2); after a retrigger, if no new run appears within ciRetriggerNewRunDwell (5m — e.g. a workflow whose types: omits reopened) the trial is abandoned. These are minutes, never the 4h backstop.
TrainCIInfra. The abandon outcome. It is never red and never pending, so no call site can fall into a charging or green-reading default::
| Site | Behaviour |
|---|---|
assembleAndValidate wrapper |
does not recordTrial (runaway guard stays uncharged) |
runMergeTrainWorker |
cleanup, cooldown, return — no bisection, ejection, pause, comment or board move |
bisect / handleRedBatch |
a sub-trial returning infra aborts the whole episode (5th/4th return value); nothing is ejected because ejection follows bisection |
landOneAtATime |
cleanup, cooldown, stop (the remaining singletons would hit the same failure) |
landGreenBatch rebase loop |
cleanup, cooldown, return — no dissolveBatch comment |
resumeTrain |
keeps today’s dissolve-and-re-form, plus the cooldown |
After an abandon dispatchMergeTrainWorker refuses the (repo, base) partition for trainInfraAbandonCooldown (5m): nothing is charged, so the runaway guard cannot bound a permanently broken workflow (a member whose PR breaks the workflow YAML); the cooldown limits it to one trial per cooldown. Known residual: such a batch re-forms indefinitely at that cadence; a cross-trial pause is a possible follow-up.
Ownership of the closed window. The close/reopen runs only inside the owning worker’s pollTrainCI. reconstructTrainState’s stale-PR sweep is reached only through prepareTrainWorker, after dispatchMergeTrainWorker’s LoadOrStore on mergeTrainInFlight; a second dispatch for an in-flight partition returns before it. escalateClosedUnmergedTrial is reachable only from landMergeTrainBatch, after green — never from pollTrainCI. If the daemon crashes inside the window, the trial PR is left closed with no worker: the next dispatch’s sweep treats a closed-unmerged trial PR as neither open nor merged, so it does not close, reopen, land or escalate it, orphaned trial branches are cleaned silently, and a fresh train forms. Pinned by TestMergeTrainInfra_*.
Flake re-run (R5). On the first CheckRunsFailed the engine takes the workflow-run ID from each failed check’s DetailsURL (…/actions/runs/<id>/job/<n>) and calls RerunFailedJobs once per distinct run, then keeps polling. A failed check with no Actions run behind it (a third-party status) or an erroring re-run request stays red at once, exactly as before. The second failure counts only if a failing latest-per-name check run has an ID absent from the first failure (the stale failure is still visible for a poll or two after the re-run is accepted); if only stale IDs are visible, polling continues until trainRerunSettleDwell (2m) elapses; the stale failure is then red unless one of the workflow runs that was re-run is still queued or in_progress (rerunInFlight, one FetchWorkflowRuns read, matched by run ID — an unrelated workflow, or a run held in waiting/pending, never postpones the verdict). The in-flight read is made before the check-run read of the same poll, never after: read the other way round, a re-run finishing between the two reads shows as a stale failure plus “nothing in flight” and the stale failure is judged the second one. A re-run waiting for a runner is waited for at most trainRerunMaxWait (30m) past the dwell — its own bound, far below CIBackstopTimeout. A read error or refused actions permission falls back to red. The state is local to one pollTrainCI call, so every bisection sub-trial gets its own single re-run. RequiredContextsFailed and dirty stay immediately red. ciSuiteHold keeps the gate unsettled while the re-run is in flight. This softens ADR-1153’s strict policy by one re-run: a genuinely red trial takes one extra CI cycle to detect.
Permissions and degradation (R7). Both endpoints need the actions permission (read for the list, write for the re-run). actions: write is in RequiredGitHubAppPermissions (#2105, ADR 2105), so the fail-hard startup grant check and fabrik init --github-app refuse an installation without it (with PermissionShortfallRemedy’s two manual steps), and a new App’s manifest requests it. The runtime degrade below covers a permission revoked after startup. A refused call (gh.ErrForbidden for a non-rate-limit 403, or ErrNotFound) logs once per process and the engine behaves exactly as before.
Stage wait_for_ci gate (R6). settlePRMergeStateForCIGate (used by handleMergeAndCIGates; the auto-merge convergence path keeps the plain settlePRMergeState) has the same blind spot in two forms: clean with zero check runs cleared the gate with CI never run (rule 9), and blocked waited to the backstop and could end in a misleading R3 “required check never runs” pause. For a stage with wait_for_ci: true and zero check runs (inside the clean shortcut, which now reads the cached check runs, and ahead of the hadChecks/dwell/R3 returns) it applies the same startupWatch per PR head SHA (Engine.startupWatches, in memory — a restart may grant one extra retrigger; a push resets it). A retrigger or wait returns PRMergeUnsettled with MergeableState omitted; an abandon (or a reopen that failed) sets PRSettleResult.StartupFailure, which handleMergeAndCIGates routes ahead of every other gate to pauseForCIStartupFailure — a pause naming the startup-failed run. The pause also drops the PR’s retrigger state (resetStartupState): the fix a human makes (a workflow repair on the base branch, or the end of an outage) does not move the PR’s head SHA, so a kept state — spent budget, old baseline — would make the resumed item abandon and re-pause at once without ever retriggering. A resume therefore gets a fresh budget of maxCIRetriggers. Unlike the train, an item’s own PR has no in-flight marker, so the closed window cannot be made exclusive; it is mitigated by closing and reopening back to back, and a PR reading closed within ciRetriggerGrace (2m) of an engine retrigger is PRMergeUnsettled, not PRMergeTerminal (which would pause the item as “closed without merging”).
Stage wait_for_ci gate: flake re-run (#2072, ADR 2072). The gate’s counterpart of the train’s R5 re-run. In handleMergeAndCIGates’ ciFailure branch, ahead of the ci-fix-reinvoke dispatch, ciFlakeRerun (engine/ci_flake_rerun.go) acts only when gh.ClassifyCheckRuns(settle.CheckRuns) is CheckRunsFailed and no worker is in flight (a required-context- or commit-status-only failure has no check run to re-run and dispatches as before). Per PR head (Engine.flakeReruns, keyed owner/repo#PR, reset when the head SHA moves, pruned past startupStateTTL, in memory only — a restart may grant one extra re-run) it grants at most one re-run before the first CI-fix dispatch (R1) and one after a recorded no-op fix (R2: LastCIFixNoOpSHA() == head); the budget is marked under flakeRerunMu before RerunFailedJobs is called, so the poll loop and settleAwaitingCIScan can never both re-run a head. A re-run calls the train’s rerunFailedWorkflowRuns (run IDs from each failed check’s DetailsURL) and claims the item instead of dispatching; no issue comment is posted — a [#N ci-gate] flake re-run (R1|R2) log line names the PR, head and workflow run IDs. CIFixCycles is never charged. Stale-failure guard (across polls): the failed check-run IDs at re-run time are recorded; while only those IDs are failing the gate waits for rerunSettleDwell (engine clock, e.now()), then for as long as a re-run workflow run is queued/in_progress (rerunInFlight) up to rerunMaxWait; after that the stale failure is the verdict. A failing check run with a new ID counts immediately. After the R1 re-run a failure therefore dispatches the CI-fix worker, and after both re-runs a failure follows the no-op skip and then CIBackstopTimeout. Degrade (R4): a check with no Actions run ID (external CI), ErrForbidden/ErrNotFound (no actions: write) or any other re-run error marks the head degraded — logged once, never retried for that head, never a pause or failure — and the gate proceeds to today’s dispatch. The auto-merge convergence dispatch (merge_gate.go) and the train are out of scope. SetCIFlakeRerunDisabledForTest is the test-only neutralisation seam the R1/R2 tests’ twins use.
Tests: engine/ci_flake_rerun_test.go, tests/sim/ci_flake_rerun_test.go, engine/ci_startup_test.go, engine/merge_train_ci_infra_test.go, engine/merge_train_infra_test.go, engine/pr_settle_infra_test.go, github/workflowruns_test.go, tests/sim/mergetrain_ci_infra_test.go. There is no live e2e twin: a startup_failure cannot be induced on demand against the bed (recorded in tests/sim/simgh/FIDELITY.md).
6.30 Merge-Train Singleton Catch-Up (#2044, ADR-2044)
Trigger: Issue #2044 (reported in #2041; proposal 1 of #1826). A singleton whose own branch is behind the pinned base never takes the §6.20 fast path (its ancestry check requires FetchCommitsBehind == 0), so it built a trial branch, opened a draft integration PR and ran a full CI run — then discarded the trial and left the member’s branch as stale as before. On a busy board that is most landings.
Mechanism. trySingletonCatchUp (engine/merge_train_catchup.go), called from runMergeTrainWorker’s re-form loop for a length-1 batch after trySingletonFastPath declines and before the trial is built (never inside assembleAndValidate, so bisection sub-trials and landOneAtATime are out of reach; batches of two or more keep trial branches, and landOneAtATime keeps trials by decision — ADR-2044). Gated on singleton_catch_up (merge default, off restores the trial path exactly), a known pinned base, FetchCommitsBehind(p.baseSHA, m.headSHA) > 0 (an unreadable ancestry builds the trial) and the per-member attempt cap (effectiveMaxTrainRebaseCycles(), in memory, reset on landing/eject; past it the trial takes over). Then:
- Ownership (R4). The member is deferred — left Queued, nothing charged, no trial this poll — when a stage session is in flight (
itemstateworker), a liveFetchLabelsread showsfabrik:editing(or cannot be read), the issue worktree is dirty or mid-merge, or the remote branch is no longer the batch-snapshot head (WorktreeManager.PrepareCatchUp:ErrCatchUpBusy). The checks run again just before the push. A localfabrik/issue-Nthat is ahead of or diverged from the remote (ErrCatchUpLocalDiverged) carries unpushed work: it is never touched and the member takes the trial — unless it is a pure rebase of the remote (#2150, ADR-2150): the same patches on a newer base, with no local-only patch (compared by patch-id,git rev-list --right-only --cherry-pick --no-merges <remote>...HEAD ^origin/<base>; plaingit cherrywould count the new base commits as local) and no local-only merge commit. That is the footprint of an engine pre-invocation rebase that was never pushed (§4.4); it holds no work the remote lacks, soPrepareCatchUplogscatch-up: local branch is a pure rebase of the remote — resetting to the remote head, runsgit reset --hard <expected head>and proceeds from the remote head — the head CI validated. The local rebase is never pushed. Any local-only patch or merge commit, a conflict-edited rebase (its patch-id changed), or any git error or failed base fetch keeps the refusal.SetCatchUpPureRebaseDisabledForTestis the neutralisation seam. - Merge (R1). In the member’s own issue worktree (so the local branch stays equal to the remote),
git merge --no-ff <p.baseSHA>— the pinned SHA, never a liveorigin/<base>; a merge commit, no rebase. A conflict runs the trial’s path (resolveTrainConflict: rerere → generated-file regeneration → Claude,NoResume,p.maxTurnsOverride; prompt re-worded viaInvokeOptions.CatchUpBaseSHA). The result must be a two-parent merge of the member head and the pinned base (anything else restores the worktree and takes the trial). A clean merge is pure; any conflict resolution makes it not pure. An unresolvable conflict restores the worktree and ejects the member throughejectMember(stay in queue), as a trial would; the account-wide Claude usage-limit sentinel defers without ejecting (ADR-1120). - Push. A
Fabrik-Train-Catch-Up: <base sha>trailer is stamped (audit only), thenPushCatchUppushes with--force-with-lease=<ref>:<expected head>— a compare-and-swap, never a real force — and advances the remote-tracking ref so the next stage’sPushBranchlease starts from the catch-up commit. A rejection (branch protection, signed commits, the member moved) resets the worktree to the member’s head and falls back to the trial, uncharged: norecordTrial, no ejection count, no pause. Policy memo (#2065, ADR-2065).PushCatchUpreturns a*CatchUpPushErrorcarrying git’s raw output;classifyCatchUpPushRejection(engine/catchup_policy.go) treats it as repo policy only on a positiveGH006(protected branch) orGH013(repository rule) marker, vetoed by any lease (stale info,fetch first,non-fast-forward) or transport/permission marker and by aGH013that mentions secrets/push protection (content-dependent). Free prose never matches. On a policy matchrunCatchUpGitreturns the fallback withpolicyReasonset, andtrySingletonCatchUprecords an in-memory memo keyed byowner/repo(not the(repo, base)partition) forcatchUpPolicyMemoTTL(24h, engine clock) and logs one line naming the repo, GitHub’s sanitised one-line reason (≤300 runes) andsingleton_catch_up: off. While the memo is unexpired,trySingletonCatchUpreturns to the trial path immediately after thebaseSHAcheck — beforeFetchCommitsBehind, so no GitHub call, worktree, merge or push — and logs nothing. A rejection arriving during an active memo neither extends it nor logs. Expiry and restart re-probe; nothing is persisted. Other fallbacks (refusedPrepareCatchUp, bad merge shape, CI never starting) are not memoised. - Marker. One comment on the member’s PR carrying
<!-- fabrik:train-catch-up head=<sha> base=<sha> pure=<bool> -->(see below). - CI wait (R2).
waitMemberCIblocks in the worker (only the worker keeps the pinned base meaningful; across polls a busy base would catch up again and again) on the member PR’s own CI at the new head:classifyLandingCI/ciSuiteHoldunmodified, bounded byCIBackstopTimeout. It applies pending review/comment ejects each iteration, treats zero check runs as pending until the no-CI dwell (default 10m → fall back to the trial; the fast path rejects zero runs anyway), treats a head that is not the pushed commit (after a short sync dwell), a closed/merged PR ordirtyas “moved” (member stays Queued), and never closes and reopens the member’s PR (a trial-PR technique). The first confirmed failure re-runs the failed jobs once (rerunFailedWorkflowRuns, §6.29); a second is red. - Land (R2/R5). Green →
trySingletonFastPathon the caught-up member: its four eligibility checks (§6.20 — live head == the caught-up commit is R5’s TOCTOU check) are the landing gate, unmodified;MergePRAtHeadSHApinned to the caught-up head. The landed comment becomes “Landed via singleton fast path PR #N after a catch-up…”. Red →ejectRedCatchUpSingleton(reroute off Queued, comment naming the catch-up commit and the diagnostic,fabrik:paused) — the red-singleton disposition (the “Red singleton short-circuit” in the merge-train overview, #1440/#1545). Backstop timeout, moved head, applied eject → member stays Queued/rerouted, nothing charged.
Review churn (R3). Pruefer reviews every head, and an unaddressed review body holds landing (§6.6.6) while the Queued settle scan ejects a member with review findings (§6.16) — which would eject the member the catch-up meant to land. A review or review-thread comment is non-actionable for the landing only if all hold (engine/catchup_feedback.go, catchUpSuppresses): the PR carries a marker comment authored by selfLogin() whose head equals the PR’s live head; pure=true; the finding’s commit (the review’s commit.oid — now selected by the board query, so PRReview.CommitID is populated on the GraphQL path — or the thread comment’s originalCommit.oid, Comment.CommitOID) equals that head exactly; and the author is a bot (gh.IsBotLogin or a declared expected_reviewers identity). Everything else stays actionable — a conflict-edited catch-up, a human, an older head, no attribution, no marker, a marker from another author, an unreadable marker. Human comments (findNewComments) are never filtered. It is applied in the two shared places — queuedReviewFindings (the settle scan and the worker’s checkpoints) and unprocessedFeedback (the landing/advance gate) — via dropCatchUpFeedback, which reads the marker (one FetchIssueComments) only when a bot finding was made against the live head. That lookup is cached per (PR, head) (catchUpMarkerCached): a found marker for good, “no marker” for catchUpNoMarkerTTL (2 min), a read error never — so a bot-reviewed head that was never caught up costs at most one extra read per TTL window, not one per poll. The worker drops the entry when it posts the head’s marker (forgetCatchUpMarker). The commit trailer is not trusted — the branch is writable by anyone with push access, so a forged trailer must not mute feedback. Member CI on the new head is the safety net for a genuine finding on a merge-only head.
Validate completion survives the catch-up (#2158, ADR-2158). The catch-up moves the member’s head past the SHA recorded at Validate completion. The SHA-invalidation scan (§2.16) must not read that as an external push: it skips an item that has landed (the catch-up lands the member’s own PR, so the Done item would otherwise lose stage:Validate:complete a poll later) and, for an open item, accepts a head named by this section’s self-authored pure=true marker — re-recording it as the completion SHA — while a pure=false catch-up, a foreign marker or a bare trailer still invalidates.
Restart safety. Nothing but the pushed commit and the marker is durable. A crash between the push and the marker leaves reviews actionable for one cycle (fail closed). The next worker pins a (possibly newer) base and re-evaluates; the in-memory attempt counter bounds the loop.
Code path: trySingletonCatchUp, runCatchUpGit, waitMemberCI, ejectRedCatchUpSingleton, postCatchUpMarker (engine/merge_train_catchup.go); PrepareCatchUp/PushCatchUp/localIsPureRebase (engine/worktree_catchup.go); RestoreUnpushedRebase (engine/worktree_restore.go); dropCatchUpFeedback/catchUpSuppresses (engine/catchup_feedback.go); trainMember.caughtUpFrom; tests engine/merge_train_catchup*_test.go, engine/catchup_feedback_test.go, engine/worktree_catchup_test.go, tests/sim/mergetrain_singleton_catchup_test.go, engine/worktree_restore_test.go, engine/comments_restore_test.go, tests/sim/mergetrain_singleton_catchup_rebase_test.go.
State transitions:
| Before | Trigger | After | Labels Added | Labels Removed |
|---|---|---|---|---|
| Queued singleton, behind the pinned base, owned | catch-up pushed; member CI green; fast path eligible; MergePRAtHeadSHA |
Done, awaiting landing verification (as §6.20) | fabrik:awaiting-landing-verification |
— |
| Queued singleton, behind | stage session / fabrik:editing / dirty tree / remote moved / usage-limit hold |
Queued (unchanged) — deferred, no trial, nothing charged | — | — |
| Queued singleton, behind, local branch is a pure rebase of the remote (#2150) | PrepareCatchUp resets the local branch to the remote head |
catch-up proceeds from the remote head (rows above) | — | — |
| Queued singleton, behind | push rejected (lease or ambiguous), local branch carries a local-only patch or merge commit, bad merge shape, CI never starts, attempt cap, singleton_catch_up: off |
Queued → ordinary trial path, uncharged | — | — |
| Queued singleton, behind | push rejected by repo policy (GH006/GH013) — first time |
Queued → trial path, uncharged; per-repo memo set (24h), one operator log line | — | — |
| Queued singleton, behind | per-repo policy memo active | Queued → trial path, no catch-up attempted (no merge, no push), uncharged | — | — |
| Queued singleton, behind | unresolvable conflict | stays in Queue via ejectMember (counted, as a trial ejection) |
— | — |
| Queued singleton, caught up | own CI red (after one failed-job re-run) | Rerouted off Queued, paused | fabrik:paused, fabrik:awaiting-input |
— |
| Queued singleton, caught up | CI wait timeout, head moved, PR closed/merged, dirty, applied review/comment eject |
Queued (or rerouted by the eject) — nothing charged | — | — |
References: ADR-2044: Merge-Train Singleton Catch-Up, ADR-1644, ADR-1953-a, ADR-1208/1863 (§6.16), ADR-2052 (§6.29), ADR-2150.
6.31 Merge-Train Red-Singleton Auto-Repair (#2045, ADR-2045)
Trigger: Issue #2045 (reported in #2039). ADR-1545 paused a red singleton for a human, who then applied fabrik:revalidate; on a busy board that step was the main reason finished work did not land. The engine now performs that step itself, bounded by an attempt cap that stands in for the external re-detection signal ADR-1545 found missing. Amends ADR-1545’s “pause for a human”; the pause remains the fallback.
Entry. ejectRedSingleton(projectID, owner, repo, m, p, diag) (both call sites: the true-singleton red trial and a member that fails alone in landOneAtATime; p is the trial’s trialParams, whose baseSHA is the pinned base — landOneAtATime re-pins it per member). Auto-repair is considered only when Config.MaxTrainAutoRepairAttempts > 0 (flag/env/yaml max_train_auto_repair_attempts, default 1 applied in cmd, 0 disables everything below), p.baseSHA is known and stageBeforeHolding is literally named Validate. Otherwise the unchanged ADR-1545 path (pauseRedSingleton) runs.
- R3 — already fixed (
memberFixedAgainstLiveBase). Re-fetch the PR, resolve the liveorigin/<base>(liveBaseSHA;trainLiveBaseFnis the test seam), rebuild atrainMemberfrom the live head and callsingletonFastPathEligibleunmodified (live head contains the live base, mergeable, own check runs present andclassifyLandingCIgreen incl.ciSuiteHold). Confirmed → return with no reroute, comment, label or pause; the member stays Queued. Fail-closed: any error is “not confirmed”. Allowed once per (member, pinned base SHA) (takeRequeue), tracked apart from the repair budget. - Cap check.
autoRepairAttemptsAt(member, p.baseSHA)counts recorded attempts for that base SHA only. At or past the cap →pauseRedSingleton, whose comment gains an “Auto-repair attempts made (N)” list (base SHA, member head SHA at failure, failing checks, time). - Repair (
startAutoRepair).- Reroute to Validate (
rerouteQueuedMemberOffHolding). A failure returns with nothing posted, counted or labelled (ADR-1208 ordering); the next poll retries. - Register the in-memory pending context (
autoRepairState.pending) before re-entry so the next dispatch cannot miss it. reenterValidate(extracted fromhandleRevalidateLabel, which still delegates to it): clearsstage:Validate:complete/:failed,fabrik:paused,fabrik:awaiting-input,fabrik:awaiting-ci,fabrik:auto-merge-enabledand resets the store’s Validate retry/pause/last-attempt/cycle state. If it defers (a label removal failed),fabrik:revalidateis applied sosettleRevalidateScanretries each poll. If that also fails the pending context is dropped and the member is paused as before (already rerouted, nothing counted).- Only once re-entry is secured: record the attempt, post
🏭 **Fabrik merge-train — auto-repair started**(failing checks, attempt N of cap) and emit the event.
- Reroute to Validate (
- Dispatch. Validate dispatches in the poll goroutine like any stage run (R5: the train worker starts no Claude and holds no slot).
writeContextFilescallswriteMergeTrainRepair, which for a non-commentValidateinvocation writes.fabrik-context/merge-train-repair.mdfrom the pending context without consuming it (a retry of the same Validate — incomplete run, turn-limit slice, tools-denied — is given the file again;handleStageCompletedrops the record when Validate completes); on every other invocation (or with nothing pending — e.g. after a restart) a leftover file is removed. The file holds the failing checks (renderDiagnosticBlock), the trial head, pinned base and member head SHAs and the instruction to rebase/merge onto the moved base and reconcile with what landed, never reverting or working around it. The same instruction is in the embeddedvalidateskill and one staticbuildPromptline. A pending context older than 2 hours (repairContextTTL) is discarded instead of written, and so is one recorded for a PR head other than the item’s currentLinkedPRHeadSHA— but that head check applies only before the context has first been written (repairContext.delivered): once delivered, the repair run itself rebases/merges and pushes, so a moved head is its own work and a retry of that run still gets the file, so a record that outlived its repair flow (item paused, blocked or usage limited) cannot leak into a later, unrelated Validate run. A missing context never blocks the dispatch. - Rejoining. The repaired Validate rejoins
Queuedthrough the path any Validate completion takes; the repair adds no enqueue path.
State. autoRepairState on Engine (mutex-guarded; the train worker writes, the poll goroutine reads, handleStageComplete drops on Validate completion): attempts per member (history capped at 20), spent R3 requeues, pending contexts. In memory only: a restart resets the counts and drops pending contexts. resetEjectionCount (every landing path) also clears a member’s auto-repair state.
Observers. emitTrainEvent emits merge-train-failed with cause red-singleton-auto-repair, auto_repair=true, attempt, cap and a reason saying no human action is needed; cause red-singleton is emitted only when the member is paused. The pause comment keeps the 🏭 **Fabrik prefix, so neither comment counts as steering.
Unchanged (R4). ejectMember (bisection-isolated poisoners, MaxMergeTrainEjections), the runaway guard, #1615 escalations and ejectRedCatchUpSingleton (§6.30) keep their human gate.
Code path: engine/merge_train_autorepair.go (startAutoRepair, memberFixedAgainstLiveBase, writeMergeTrainRepair, autoRepairState), ejectRedSingleton/pauseRedSingleton (engine/merge_train.go), reenterValidate (engine/item.go), writeContextFiles (engine/context.go).
State transitions:
| Before | Trigger | After | Labels Added | Labels Removed |
|---|---|---|---|---|
| Queued singleton, red trial | cap > 0, target Validate, attempts at this base < cap, not already fixed |
Validate (rerouted), Validate re-entered, repair context pending | — | stage:Validate:complete, stage:Validate:failed, fabrik:paused, fabrik:awaiting-input, fabrik:awaiting-ci, fabrik:auto-merge-enabled |
| Queued singleton, red trial | direct re-entry deferred | Validate (rerouted); settle scan retries | fabrik:revalidate |
— |
| Queued singleton, red trial | live head contains live base and own CI green (once per base) | Queued (unchanged) | — | — |
| Queued singleton, red trial | cap reached, cap 0, non-Validate target, or re-entry could not be secured |
Validate (rerouted), paused (ADR-1545) | fabrik:paused, fabrik:awaiting-input |
— |
| Queued singleton, red trial | reroute fails | Queued (unchanged); retried next poll, nothing posted or counted | — | — |
References: ADR-2045, ADR-1545, ADR-1208, ADR-1644, ADR-2044.
6.32 Merge-Train Overlap-Aware Batch Composition and Post-Landing Invalidation (#2047, ADR-2047)
Trigger: Issue #2047 (reported in #2040; proposal 2 of #1826). Batches were formed in entry order and conflicts were found only by spending trials; after a landing every Queued member that now conflicts with the new base cost a trial (and often a pause) to discover. Two mechanisms, with opposite failure polarities on purpose.
A. Fresh-batch composition (admitByOverlap, engine/merge_train_overlap.go). The third fresh-formation filter in prepareTrainWorker, after dropLiveLandedMembers (§6.28) and admitTrainMembers (§6.25). Fresh formation only — both restart routes return from reconstructTrainState first; bisection, landOneAtATime and landGreenBatch never call it. It runs on the already-capped batch (routeQueuedGroup caps first), so a deferral can leave the batch smaller than the cap.
- Order. Members that were deferred for overlap
overlapStarvationThreshold(3) consecutive formations of their partition are considered first (FR-003); otherwise today’s(StatusEnteredAt, Number)order (§6.16/ADR-1833). Survivors keep their original relative order. - Changed files.
FetchPRFileson the live client, read once per(repo, PR, head SHA)and cached in memory (a new head SHA overwrites). A read error, an empty/404 list or a list of 3000 or more (GitHub’s silent cap) is unreadable: the member is admitted unchecked, contributes nothing to the union, and an error is never cached. - Walk. Paths matching
merge_train_overlap_ignore(internal/pathglob:*never crosses/,**spans segments) are dropped from both sides. A member is kept if its remaining paths are disjoint from those already kept, else deferred. The first member considered is always admitted. - Deferral. The member simply stays Queued — no reroute, comment, ejection count or pause. One log line:
deferred #B: overlaps #A on <smallest shared path>. The skip count increments; it resets when the member is admitted or is absent from a formation of its partition (counts are pertrainKey).
B. Post-landing invalidation (invalidateConflictingQueued, engine/merge_train_invalidate.go). Every landing path (landSingleton, finishSingletonFastPathLanding, both branches of landMergeTrainBatch) records the landed member with noteTrainLanded. routeQueuedGroup consumes the record for a partition only while no worker is in flight for its trainKey (the ADR-1208 ownership rule: with no worker, the poll goroutine owns the Queued members), before the cap and before dispatch, and skips it under the trainValidateFn seam.
- Skipped (members untouched, logged) unless the reroute target stage has
wait_for_ci(ADR-1821 R9), aWorktreeManageris registered, the base resolves, andgit fetch originsucceeds. - For each candidate that is not a landed member and carries neither
fabrik:pausednorfabrik:editing: resolverefs/remotes/origin/fabrik/issue-Nand rungit merge-tree --write-tree --name-only --no-messages refs/remotes/origin/<base> <head>in the bare clone — no CI, no GitHub API reads. A conflict is accepted only when both refs resolve and stdout starts with a result-tree OID (git merge-treealso exits 1 for an unresolvable ref). - Conflict →
rerouteQueuedMemberOffHoldingfirst (reroute-before-side-effects, ADR-1208); on success any pending review/comment eject signal is consumed,fabrik:rebase-neededis applied best-effort (Validate’s mergeability gate re-derives it from live mergeability, so a wrong application self-heals, and applying it closes the yolo/cruise bounce window), oneinvalidated #N: conflicts with landed #M (merge-tree)line is logged (M = landed members whose cached file lists intersect the conflicted paths, else all landed this pass) and one🏭 **Fabrik merge-train — rerouted (conflicts with new base)**comment is posted, deduplicated per (member, head SHA). It never callsejectMember, never touchesmergeTrainEjectionCountsand never pauses (FR-009); the rebase cycle is charged normally bydispatchRebaseReinvoke. - Clean merge (even when the member overlaps the landed files) → left alone. Any error (git < 2.38, missing ref/object, failed fetch) → left Queued and logged (fail-safe). A failed reroute leaves the member Queued for a trial.
State. Engine.overlap (file cache, skip counts) and Engine.invalidate (landed record, comment dedupe), each behind its own mutex; in memory only, so a restart delays the starvation guard and drops a pending scan (cost: one trial, no worse than before). The signal is consumed once — no retry loop. Test seams: SetMergeTrainOverlapDisabledForTest, SetMergeTrainInvalidationDisabledForTest.
Out of scope. Bisection, restart reconstruction, one-at-a-time and green-batch landing paths; rerouting on file overlap alone; persisting counts or the cache.
Code path: engine/merge_train_overlap.go, engine/merge_train_invalidate.go, prepareTrainWorker and the landing sites (engine/merge_train.go), routeQueuedGroup (engine/poll.go), internal/pathglob, FetchPRFiles (github/prs.go, engine/interfaces.go).
State transitions:
| Before | Trigger | After | Labels Added | Labels Removed |
|---|---|---|---|---|
| Queued candidate | fresh formation, files overlap an already-admitted member | Queued (unchanged; skip count +1) | — | — |
| Queued candidate | fresh formation, files unreadable / not known complete | admitted as before | — | — |
| Queued candidate | skipped 3 consecutive formations | considered first in the next formation | — | — |
| Queued member | post-landing scan, merge-tree conflict |
Validate (rerouted), not paused, no ejection counted | fabrik:rebase-needed |
— |
| Queued member | post-landing scan, clean merge (overlap or not) | Queued (unchanged) | — | — |
| Queued member | post-landing scan, git error / unresolvable ref / fetch failed / no wait_for_ci target |
Queued (unchanged), logged | — | — |
References: ADR-2047, ADR-1821, ADR-1208, ADR-1833, ADR-1648, ADR-2044, ADR-2045.
7. Edge Case States
7.1 Cooldown Retry
When Claude runs but does not output any completion marker, the engine enters a cooldown retry loop. This applies both when Claude exits cleanly without a marker and when it exits with an error (e.g., timeout, crash). Only start failures (binary not found, exec.Error, os.PathError) skip the cooldown — the item is retried on the next poll instead.
- Cooldown duration:
retry_backoff(--retry-backoff/FABRIK_RETRY_BACKOFF, default 60s, minimum 1s), independent ofpoll(#1831). Its only job is to stop a stage that exits incomplete from hot-looping; runaway and slot-fairness control belong tomax_turns(per attempt — the worker slot is released when the invocation ends) andmax_retries(across attempts). Before #1831 this wasPollSeconds * 10, so raisingpollto protect a shared GitHub rate limit silently multiplied retry latency (180s poll → 30 min). AConfigbuilt without the CLI (tests, embedders) leavesRetryBackoffzero and falls back toPollSeconds * 10. - State: In-memory only —
LastAttemptAt(stage)initemstate.Store, written byStageAttemptedonly when Claude actually runs, and read through the one shared helperstageRetryBackoff()by the dispatch gate (itemNeedsWork),processItem’s gate, and the “will retry after” message, so all three always agree. No label is added for cooldown. This is distinct fromCooldownAt("periodic-re-eval"), which suppresses deep-fetches on the GitHub re-check cadence (githubRecheckInterval(), stillPollSeconds * 10because it is about API cost) — see ADR-1831. - Re-admission when the backoff ends (#2096, ADR-2096): the backoff records no cooldown of its own, and expired cooldowns are consumed on admission, so nothing would re-admit a backed-off item before the periodic re-evaluation (
githubRecheckInterval(), 10 ×poll). Where the worker’s exit churns a label the item is incycleSeton the next poll, which is usually still inside the backoff;itemNeedsWorkthen refuses it and the poll’s defer re-armsperiodic-re-eval. (An incomplete exit churns nothing — see the post-run record below — so this was only a backstop.) So whendispatchCandidatesrefuses an item only becauseLastAttemptAtis insidestageRetryBackoff()(itemNeedsWorkRetryAt; the backoff is the last gate, so every other check has passed), it recordsCooldownRecorded{Reason: "retry-backoff"}expiring after the remaining backoff. The expiry re-admits the item exactly once, through the consume path, on the first poll afterLastAttemptAt + stageRetryBackoff().LastAttemptAtis wall-clock (time.Now) while cooldown entries are one.now(), so the remaining duration is carried across the two domains, not an absolute instant. Backoff semantics andgithubRecheckInterval()are unchanged. The same path also recordssentinel-live(one poll,mergeGateRecheckInterval()) when dispatch is refused because a cleared worker’s process is still alive (R5, #1779), so that refusal is re-checked next poll instead of waiting out the periodic interval; the entry is consumed on each admission, so it recurs once per poll only while the process is alive. - Time slices skip the backoff (#2232, ADR-2232): the backoff exists to stop a stage that exits incomplete straight away (a crash at startup, a broken environment) from looping. A turn-cap stop (
error_max_turns, routed toSliceRetries) or amax_wall_timekill (claudeerr.KillWallTime) is the opposite: the worker just ran a full slice, andSliceRetries/max_retriesalready bound it.isTimeSlice(structural signals only, never run duration, so a fast error exit cannot qualify; an inactivity kill is a hang and keeps the backoff) marks the attempt (StageAttempted.Slice→StageState.BackoffExempt), both retry gates (itemNeedsWorkRetryAt,processItem) skip theLastAttemptAtbackoff for it, the decision point records an already-dueretry-backoffcooldown (Until = e.now()) and logswill retry on the next poll (time slice). TheWorkerExitedwake (inwakeChFlags) then runs a poll that admits the item through the consume path. The nextStageAttemptedrewrites the mark. Worker-slot fairness is §9.9.3. -
Every engine-scheduled retry owns an admission cooldown (#2232, ADR-2232): the dispatch-time record above only fires if something else admitted the item, and a worker that ends incomplete (a
max_wall_timeor inactivity kill, a turn-cap slice, a resume failure, a tools-denied or error exit) admits nothing: its commits, pushes and label writes are self-writes,InvocationChangedis in neither the wake set norcycleSet, andWorkerExitedis excluded fromcycleSetFlags(ADR-039). The item then waited forperiodic-re-eval(up to 10 ×poll). So the decision point itself records the cooldown: infinalizeStageOutcome’s non-escalating incomplete branch (recordPostRunRetryCooldown),handleAPIErrorExit,handleUsageLimitExit(until the later of the backoff and the account-wide hold) and the PR-creation-failed branch,CooldownRecorded{Reason: "retry-backoff", Until: e.now() + stageRetryBackoff()}— the same value that is logged aswill retry after <d>and passed toemitRetryScheduled(FR-004), so they cannot disagree. Only whenStageAttemptedwas stamped (claudeRan): a start failure has no backoff, logsdid not start — will retry on the next polland records nothing. Escalated and paused exits record nothing, and neither do exits that need a human or a config fix (apiKeyHelper detected, skill unresolved or failed to load, boundary violation).LastAttemptAtis wall-clock while the cooldown is one.now(); if the cooldown expires before the real-time gate opens, the dispatch-time record above re-records the remaining wait. - Lock behavior: The lock (
fabrik:locked:<user>andstage:<X>:in_progress) is NOT released during cooldown. This prevents other instances from picking up the item. - Resume behavior: On retry,
resume=trueis passed to Claude (resumes the session rather than starting fresh) - On restart: Cooldown state is lost. On clean shutdown,
fabrik:locked:<user>is removed by the deferredcleanupLockedIssues()path and is NOT present on restart; a daemon-wide clean stop (SIGINT/SIGTERM) additionally clearsstage:<X>:in_progressdirectly for every issue with a live worker, before the process exits (ADR-1393 R1/R2), rather than leaving that to chance. After a crash, a force-quit, or a shutdown-pause write that itself failed, a stalefabrik:locked:<user>label remains on GitHub and is detected byrunStartupCleanup()on next startup, which removes it (logging[#N startup] found stale lock label from prior crash — removing);stage:*:in_progresslabels are also removed by that same pass. Astage:<X>:in_progresslabel that survives with neither a lock label nor acomplete/failedsibling — the residual gap even the direct clear above cannot fully close — is healed independently byrunStartupBareInProgressScan()(§9.7, ADR-1393 R7). - Shutdown lock cleanup:
cleanupLockedIssues()is driven byLock.HeldByThis. It skips the REST removal only when the engine itself recorded removing the lock label (EngineWriteAt[label:<lock>]) and the cached label set confirms it is gone — the call would 404. A cached set that merely lacks the label is not proof (the board query truncates labels and a deep-fetch write-back can drop one), so without that record the removal is still attempted. Agh.ErrNotFoundfrom the removal is success, logged quietly. Either way it releases the lock in the store (#2159). A reinvoke (engine/reinvoke.go) marks the lock held without applying a label and nothing releases it; that leak is a separate follow-up. - Stage-complete exemption: Items where
stage:X:completeappears in the shallow label set are NOT subject to cooldown retry — they have no work to retry. When the cooldown-expired branch fires initemMayNeedWork(), the engine checks forstage:X:completein shallow labels before returningtrue; if present, it returnsfalse. This prevents perpetual deep-fetch loops for terminal items (cruise+Validate complete, paused+complete, closed-with-stage-complete) where every poll after cooldown expiry would otherwise trigger a no-op deep-fetch indefinitely.
7.2 Failed Stage / Pause on Retry Limit
Channel events (#1968): A pause or escalation is reported as
paused,awaiting-inputorescalated— the transition into the sameinternal/attentionstate the read API shows (§7.13) — and a counter one short of its limit ascycle-limit-near; see §7.14.
When a stage fails MaxRetries times (default: configurable, 0 disables):
escalateFailedStage()addsfabrik:paused+stage:<X>:failed- Posts an explanatory comment
- Sets
PausedByEngine(stageName)viaitemstate.EnginePausedmutation - Releases the lock
Degenerate output as a failure trigger: A stage whose stripped output trips the degenerate-output guard (§2.6 — a bare @file reference or absolute path, e.g. @/tmp/plan.md) counts as a non-completion for retry purposes, identical to “no marker present.” escalateFailedStage() takes an optional pre-formatted causeNote string, built by the caller; when the failure was caused by degenerate output, causeNote carries the offending reference and is appended to the pause comment as a **Cause:** paragraph naming it. On the first detection (before MaxRetries is reached), a one-time explanatory comment is also posted immediately so the operator isn’t left with silence until the final escalation.
No-artifact completion as a failure trigger (#1782): A stage whose invocation arrives with completed=false because artifactMissingOnComplete() fired upstream (§2.6) is a distinct cause from degenerate output — the CLI’s terminal result field (and every assistant turn) carried nothing beyond FABRIK_STAGE_COMPLETE itself — and counts against MaxRetries the same way. finalizeStageOutcome() recomputes artifactMissingOnComplete() against the pre-strip output specifically to build a distinct causeNote for escalateFailedStage() (never reusing the degenerate-output guard’s bare-file-reference wording, which would misdescribe this cause) and to post its own one-time first-detection comment (“no artifact harvested”) on first occurrence, mirroring the degenerate-output guard’s visibility. escalateFailedStage()’s causeNote parameter is exactly what makes both causes representable without either misdescribing the other — message construction lives at each call site, not inside escalateFailedStage() itself.
Recovery: User investigates, makes fixes, then removes fabrik:paused. On next poll, processItem() detects the failed label (or snap.PausedByEngine(stageName) from the store) and calls clearFailedStage(), which:
- Removes
stage:<X>:failed - Resets retry count (
StageRetryCleared), engine-paused flag (EngineUnpaused), cooldown (StageLastAttemptCleared), and review cycle count (EngineCyclesCleared)
7.2.1 Resumable Engine Pauses (ADR-1460)
The invariant: every pause comment Fabrik posts ends with “remove fabrik:paused to resume” — this must actually be true for every pause site, not just stage-failure escalation. Before #1460, four pause sites violated it: pauseForReviewCycleLimit, pauseForCIFixCycleLimit, pauseForRebaseCycleLimit, pauseForEnqueueCycleLimit never applied itemstate.EnginePaused, so removing fabrik:paused was a no-op — the triggering cycle counter (ReviewCycles/CIFixCycles/RebaseCycles/EnqueueCycles) stayed pinned at its limit and the item re-paused on its very next evaluation (reproduced live on #1208: five seconds from unpause to re-pause, byte-identical repost comment, no comment involved in the resume attempt at all).
Two resume gates, chosen by when the pause fires relative to stage completion:
- Pre-completion pauses (stage not yet
stage:<X>:complete) —escalateFailedStage,escalatePRCreationFailure,handleBoundaryViolation(§7.2 above),pauseForSliceLimit(§7.12) — resume viaprocessItem()’s ownwasPaused || hasFailedLabelgate, described above. Unchanged by #1460. - Post-completion pauses (stage already
stage:<X>:complete, or in thefabrik:awaiting-ciwindow) — the four cycle-limit sites — resume viahandleEngineUnpause, a Phase 1 catch-up-loop handler (§6.2, Handler 0) added by #1460.processItem()is never dispatched for a stage-complete item with no new comment (itemNeedsWorkreturns false), so gate 1 above is structurally unreachable for these four sites — this was the flaw in the issue’s own originally-suggested fix.handleEngineUnpauseruns first, unconditionally, in every Phase 1 pass: ifsnap.PausedByEngine(stage.Name)is true while the item is here at all (meaningfabrik:pausedis currently absent, per the catch-up loop’s own admission filter), it calls the identicalclearFailedStage()reset gate 1 uses, then lets the rest of the chain re-evaluate the just-reset counter in the same pass.
All four now-fixed sites apply itemstate.EnginePaused in both their fresh-pause branch and their reapply-existing-comment branch (see below) — the reapply branch must re-arm it too, or a second cycle-limit episode after a successful first resume would itself become a fresh one-way door.
No-repost on re-pause (R4): reusing the pattern from ADR-1408 (pauseForCITimeout/pauseForCIFixCycleLimit), each of the four sites checks hasPauseComment(item, <its own stable message fragment>) before posting — a mutate.go-shared helper generalized from ADR-1408’s CI-specific hasCIGatePauseComment. If a matching comment already exists for the episode, the labels are reapplied (reapplyPauseLabels) without a duplicate post.
Classification of every pause/counter site (full detail in ADR-1460):
| Category | Sites |
|---|---|
Already correct (applies EnginePaused, resumes via gate 1) |
escalateFailedStage, escalatePRCreationFailure, handleBoundaryViolation, escalateSettle (synthetic stage key, out of scope), pauseForSliceLimit (§7.12 — mid-stage, gate 1 reachable) |
Fixed by #1460 (applies EnginePaused, resumes via gate 2 / handleEngineUnpause) |
pauseForReviewCycleLimit, pauseForCIFixCycleLimit, pauseForRebaseCycleLimit, pauseForEnqueueCycleLimit |
| Independently resets (no fix needed) | pauseForReviewTimeout (strips its own fabrik:awaiting-review anchor before pausing), pauseForConvergenceFailed / pauseForMergeGroupStall / checkAutoMergeConvergence’s “auto-merge disabled” branch (all strip fabrik:auto-merge-enabled, their shared anchor), tripCommentBreaker (self-expiring window), checkDependencies (live-derived) |
| Condition-driven, no counter (no fix needed) | pauseForCITimeout (anchored to external fabrik:awaiting-ci state, not an attempt budget — re-pausing while CI is genuinely still unresolved is correct), pauseForPRClosedNotMerged, pauseForRequiredNeverRunningCheck, handleBrokenReviewLinkage, pauseForBrokenLinkage, ensureRepoReady/postSpawnCloneError/spawnChildren |
Accepted side effect: clearFailedStage()’s reset (reused unmodified by handleEngineUnpause) also zeroes Attempts(stage.Name) — the unrelated MaxRetries failure counter — via StageRetryCleared. A manual unpause of a cycle-limit-paused item therefore also gives that stage’s failure budget a clean slate. This is deliberate (“a human unpause is ‘try again’”), consistent with the same reset already applying to escalateFailedStage/escalatePRCreationFailure, and is a net-new intentional behavior for the four newly-fixed sites (they had no working “unchanged” baseline to preserve — they were broken before #1460).
Restart durability: handleEngineUnpause reads PausedByEngine, which is in-memory-only (does not survive an engine restart — internal/itemstate/snapshot.go). This is not a gap: ReviewCycles/CIFixCycles/RebaseCycles/EnqueueCycles live in the same in-memory StageState, constructed empty by NewStore(nil) on every process start with no persistence layer for any StageState field — so a restart clears the pause flag and the triggering counter together, never one without the other. A restart before a human unpauses an item is equivalent to handleEngineUnpause having already fired for free (both read back as zero); there is no interleaving that reproduces this section’s defect via a restart. See ADR-1460’s “Restart Durability” section.
7.3 Claude Usage-Limit Exemption
Channel events (#1968):
activateClaudeUsageLimit/clearClaudeUsageLimit, and the usage-limit hold’s deadline simply passing, emit the account-wideclaude-usage-limited/claude-usage-limit-lifted; see §7.14.
When a Claude invocation exits because the account’s usage limit was hit (e.g. You've hit your
session limit · resets 10:20pm (America/Edmonton)), it is a distinct condition from both §7.2
(genuine stage failure) and a process start failure (claudeRan == false, §1.4’s claudeRan
discussion) — the process started and ran, but produced no usable stage outcome, and retrying
immediately would only hammer the same limit. See ADR-1119 for why this reuses the same
StageAttempted-without-StageRetryIncremented split handleBoundaryViolation established for an
unrelated condition (cross-repo worktree boundary violations).
Detection: interpretClaudeResult (engine/claude.go) classifies a genuine usage-limit exit
structurally, from the CLI’s own parsed result object only — never from anything the assistant
wrote. classifyUsageLimitExit(resp claudeResponse, usage TokenUsage) checks
resp.TerminalReason == "blocking_limit" (the usageLimitTerminalReason constant), the same
terminal_reason field the sibling turn-cap check (resp.Subtype == "error_max_turns") already
consults. A second structural shape is also a usage-limit exit (#1811, ADR-1811): the CLI reports
some session-limit hits as terminal_reason == "api_error" with api_error_status == 429 (the HTTP
status behind the failure), which classifyUsageLimitExit claims ahead of §7.3a’s api_error
classifier. api_error_status decodes tolerantly (apiErrorStatus): only a JSON integer counts;
absent, null, zero, or wrong-typed values read as “not 429” and never drop the rest of the result
object. The result text is never consulted. Both shapes sit behind the same turns/cost exclusion
gate, and detection only runs when a result object actually parsed (ok == true). When Claude exits non-zero
without a FABRIK_STAGE_COMPLETE marker, no turn cap, and TerminalReason matches,
interpretClaudeResult returns a *claudeUsageLimitError{Message, ResetAt, ResetFallbackReason} sentinel
instead of the generic error wrapper; ResetAt/ResetFallbackReason come from the separate
structured rate_limit_event scan (see “Structured reset instant” below, ADR-1815) — never from the
result text, which for a 429 api_error names a reset time that is deliberately not parsed. An unparseable-JSON
invocation (ok == false — process killed mid-stream, truncated output) has no structured payload to
trust and is never classified as a usage-limit exit by any means; it falls through to ordinary
failure/timeout handling.
The turn-count/cost shape from #1184 is kept as a belt-and-suspenders exclusion, carried over from the
original prose-matching guard: classifyUsageLimitExit returns detected = false when
usage.TurnsUsed > 0 && usage.CostUSD > 0, because a real usage-limit exit terminates the invocation
immediately, before any work is billed, so an invocation that consumed turns and incurred cost cannot
have hit one regardless of what TerminalReason says. The exclusion is conjunctive (turns and
cost) so a partially-recorded usage struct cannot suppress a genuine detection.
This replaces the original usageLimitHitRE/usageLimitResetRE/detectUsageLimitExit prose match
over raw NDJSON stdout bytes, which matched Anthropic’s usage-limit exit message wherever it appeared
in output — including text the assistant itself had written. Without a structural check, any stage
whose output merely discusses usage limits self-triggers an account-wide usage-limit hold — on 2026-07-27 a
turn-capped run on #1178 that quoted #1084’s example message held dispatch for ~11 hours (#1183).
A diagnostic-only log line (claudeLog(..., "error exit with unmatched terminal_reason=%q ..."))
records any other non-empty TerminalReason seen on an error exit (e.g. the SDK’s documented
rapid_refill_breaker value, a plausibly short-lived burst throttle deliberately not treated as a
usage limit) so a real-world sighting is auditable without affecting classification. See ADR-1183.
Handling: finalizeStageOutcome() (engine/item.go) detects the sentinel via errors.As,
immediately after the existing engine-shutdown guard, and routes to handleUsageLimitExit(), which:
- Applies
itemstate.StageAttempted— the normal dispatch cooldown (retry_backoff, §7.1) applies, so the item does not retry on the very next poll and hammer the limit in a tight loop. Deliberately does not callStageRetryIncremented— the stage never ran, so this does not count againstMaxRetries. - If
fabrik:claude-limitis absent, posts an explanatory comment naming the condition and applies the label — gated on the label’s own absence, the same once-per-episode idiom asfabrik:awaiting-ci/fabrik:bot-reprompted. If already present (a repeated hit within the same episode), neither the comment nor a duplicate label-add fires. The comment names the reset time only when it was sourced from the structuredresetsAt(see “Structured reset instant” below); it is omitted on the fixed fallback. - Skips
commitWIP, push, andmarkCommentsSeenByStage— nothing was produced. - Releases the lock and returns — no
stage:<name>:failed, nofabrik:paused.
Auto-clear (per-issue): Immediately after the branch above, any invocation that reaches that point
(i.e. was not itself classified as a usage-limit exit — success, blocked-on-input, no-work-needed,
genuine failure/retry, or PR-creation failure alike) unconditionally removes fabrik:claude-limit if
present. This is broader than “only clears on success”: a subsequent genuine failure right after a
cleared usage-limit episode is still tracked fully independently via stage:<name>:failed/MaxRetries,
so clearing on any non-limit invocation correctly signals “not currently limited” without masking a real
failure.
This per-issue clear only fires when the issue is dispatched again. An issue that is labelled and then
paused, blocked, or simply not redispatched keeps fabrik:claude-limit indefinitely after the
account-wide usage-limit hold has already ended — see “Account-wide label sweep” below for the settle scan
that closes this gap.
Regression guard: a usage-limit hit followed by a genuine failure leaves the full MaxRetries
budget available for the genuine failure — the limit hit consumes zero attempts.
Account-wide usage-limit hold and backoff-to-reset (ADR-1120): the per-issue handling above is layered
under an account-wide gate, because the usage limit applies to the account, not the issue — without it,
every concurrently-dispatched item independently rediscovers the same limit and each waits out its own
per-issue cooldown. Engine carries a single usage-limit hold deadline (claudeUsageLimitedUntil time.Time,
guarded by a dedicated claudeUsageLimitMu, distinct from the general-purpose e.mu) that is checked
lazily — a cheap read — at each of the three places a Claude invocation can actually be attempted:
the stage-invocation loop (runInvocationWithExtension, engine/item.go), the comment-review loop
(processComments/runCommentExtensionLoop, engine/comments.go), and merge-train inline conflict
resolution (resolveConflictWithClaude, engine/merge_train.go).
- Structured reset instant (ADR-1815):
unifiedWindowsis not on the terminalresultline — it sits on a separate{"type":"rate_limit_event"}NDJSON line that arrives before it. OnceclassifyUsageLimitExithas fired,extractUsageLimitReset(engine/usage_limit_reset.go) makes a tolerant best-effort scan of the raw stream (independent ofparseClaudeJSON, so a malformed event can never break classification) and attaches the result to the error asResetAt, or aResetFallbackReason(absent/zero/malformed/no_exhausted_window) when none is usable. The lastrate_limit_eventcarryingunifiedWindowswins, and within it the reset is the latestresetsAtamong windows whoseutilizationis ≥ 1 — windows below 1 never contribute (a non-exhaustedseven_dayresets days after the blockingfive_hourand would over-hold the account). Window keys are read generically, so a future exhausted window is honoured.resultprose, assistant text andoverageStatus/overageResetsAtare never read (ADR-1183 applies to the deadline as it does to the trigger). The prose parser (parseUsageLimitResetTime) is deleted. This reverses ADR-1811’s dropped R6 and fulfils ADR-1183 §3’s deferred follow-up. - Deadline resolution:
resolveUsageLimitDeadline(limitErr, now)is the single place the R3 and R4 rules apply: a zeroResetAtfalls back toclaudeUsageLimitFallbackBackoff(1 hour, ADR-1120) with the decode-time reason;ResetAt <= nowfalls back with reasonpast;ResetAtmore thanusageLimitMaxResetHorizon(8 days — the 7-day window plus a day’s margin) ahead is clamped tonow + 8d; anything else is used as-is. Every fallback and clamp is logged with its named reason. Accepted edge: a fallback-derived deadline from one worker can extend an exact one that ends sooner than an hour away (extend-never-shorten is preserved unchanged); unreachable in practice, since every captured 429 exit carries the event. - Gating: each of the three call sites checks
claudeUsageLimitedUntilTime(time.Now())before attempting an invocation. If usage limited, the stage-invocation and merge-train paths short-circuit by returning/propagating a*claudeUsageLimitErrorwithout calling Claude at all (routed through the same per-issue handling described above);processCommentsreturnsnilimmediately, before any reaction, label, or worktree side effect, so the comment remains “new” and is retried on a later poll once dispatch resumes. Already-running invocations are unaffected — the gate only blocks the start of a new one. - Detecting at all three sites:
comments.goandmerge_train.godid not unwrapclaudeUsageLimitErrorbefore this issue landed, so a hit there was misattributed as ordinary failure — a comment-review usage-limit hit counted toward the comment circuit breaker, and a merge-train conflict-resolution usage-limit hit was indistinguishable from a genuinely unresolvable conflict and ejected a healthy member. Both paths now detect the sentinel viaerrors.Asafter theirInvokeForCommentscall:runCommentExtensionLoopactivates/clears the usage-limit hold andprocessCommentsskipscheckCommentBreakeron a usage-limit outcome;resolveConflictWithClaudereturns(bool, error)instead of a barebool, andassembleTrialBranch(viaresolveTrainConflict, issue #1235) treats a non-nil error as a fatal assembly failure (cleanup, retry next cycle) rather than runninggit merge --abort+ejectMember— an account-wide condition says nothing about whether that member’s conflict is resolvable.resolveConflictWithClaudeadditionally takes ageneratedPaths []stringparameter (#1235): non-empty only in the mixed generated/non-generated conflict case, where it changes the synthetic comment’s instructions and defers the unscopedgit diff --check+ commit toregenerateAndCommit, but does not change the usage-limit gate or detection above — the usage-limit hold check anderrors.Asunwrap run identically regardless ofgeneratedPaths. The all-generated case (no non-generated paths at all) never callsresolveConflictWithClaudeand so never touches this gate — regeneration has no Claude dispatch to hold. - Auto-resume: there is no ticker or wake mechanism tied to the deadline itself —
claudeUsageLimitedUntilTimesimply returns “not usage limited” oncenowreaches the deadline, so the very next dispatch attempt (which was going to happen anyway on the normal poll cadence) proceeds normally. Non-Claude engine work (board polling, settle scans, label reconciliation) is never gated by this mechanism. - Operator-triggered clear (restart-free, #1183): an operator can end an active usage-limit hold early —
e.g. after confirming a usage-limit hold was a false positive — by applying
fabrik:clear-claude-limitto any open board item (not necessarily one carryingfabrik:claude-limit, since the usage-limit hold is account-wide, not per-issue).settleClaudeLimitClearRequests(engine/claude_limit_settle.go), a per-poll settle scan run unconditionally over the raw board snapshot (mirroringfabrik:revalidate’s label-is-a-command idiom), reads the label each poll: if any open item carries it, the scan callsclearClaudeUsageLimitonce and removes the label from every carrying item. This is a one-shot command with no retry counter — a failed removal simply leaves the label for the next poll to consume again, which just re-runs the (idempotent) clear. This is the previously-missing restart-free remedy; SIGHUP/ctrl+r(full process re-exec, dropping all in-memory state, not just the usage-limit hold) remains available as a heavier fallback. - Account-wide label sweep (#1183):
settleClaudeLimitLabelSweep(engine/claude_limit_settle.go), the same per-poll settle scan file, closes the per-issue-only gap in “Auto-clear” above: onceclaudeUsageLimitedUntilTimereports the usage-limit hold has lifted (by either path — deadline reached or operator clear), it best-effort-removesfabrik:claude-limitfrom every open item still carrying it, regardless of whether that item is ever dispatched again. The label is purely cosmetic — dispatch is gated byclaudeUsageLimitedUntil, not the label — so a failedRemoveLabelFromIssuecall self-heals on the next poll and no retry/escalate machinery (recordSettleRetry/escalateSettle) is used, unlikesettleMergeTrainMemberCloses/settleNonDefaultBaseCloses, which guard an outstanding mutation with real consequences. Scoped to open items only —cleanupClosedIssueTransientLabelsalready sweepsfabrik:claude-limit(andfabrik:clear-claude-limit) from closed issues every poll viatransientLifecycleLabels. Both new scans are wired intopoll.goimmediately aftersettleNonDefaultBaseCloses, clear-requests first so an operator’s clear and the resulting sweep can land in the same poll cycle. - Early clear: any of the three call sites whose invocation succeeds (
err == nil) callsclearClaudeUsageLimit(reason), which clearsclaudeUsageLimitedUntil(logging + emittingtui.ClaudeUsageLimitAlertEvent{UsageLimited: false}) if a usage-limit hold was active — the parsed/fallback deadline is a hint, not a contract, so a successful invocation ahead of it lifts the usage-limit hold immediately rather than waiting it out. A generic, unrelated error is deliberately not treated as evidence the limit has cleared: it proves nothing about account-wide state, and clearing on it would race with a concurrently-running worker that just activated the usage-limit hold, undoing the detection moments after it happened. - TUI surfacing:
ClaudeUsageLimitAlertEvent(tui/events.go) drives a dedicatedClaudeUsageLimitBannerComponent(tui/usage_limit_banner.go), independent of the GitHub rate-limitAlertBannerComponentper the naming distinction above — both can be visible at once. The banner also self-clears on aTickEventwhose time has passed the reset, a cosmetic-only convenience that can run slightly ahead of the engine’s own lazy check (harmless, since the engine never trusts the banner’s state) —AlertBannerComponentmirrors this same self-clear-on-reset behavior independently per REST/GraphQL bucket (#1482), so both banner components now share the identical active/reset model. - Updated comment copy: the per-issue explanatory comment posted by
handleUsageLimitExitno longer says “Fabrik will keep retrying on the normal poll cooldown” — it now describes the account-wide usage-limit hold and automatic resume at the reset time (or as soon as any invocation succeeds), and carries the structured reset instant when one was sourced.
Out of scope (still deferred): a dedicated escalation/safety-net path for persistent or repeated
misdetection. Without MaxRetries counting against a usage-limit exit, a misdetected or unusually
long-lived condition still retries indefinitely (now spaced out by the account-wide usage-limit hold window
rather than the 5-minute cooldown) instead of ever escalating for human review — an accepted, explicit
limitation per ADR-1119 and ADR-1120. Cross-process coordination between multiple Fabrik instances
sharing one account is also out of scope: the usage-limit hold is per-process, so a fleet of instances will
each discover the limit once.
7.3a Transient api_error Exemption
Scope (#1811, ADR-1811): this section covers api_error exits whose api_error_status is not
- An
api_errorcarryingapi_error_status == 429is a session/usage limit — account-wide, not per-invocation — and is claimed byclassifyUsageLimitExitfirst, taking §7.3’s path (label, comment, account-wide usage-limit hold). Every liveapi_errorexit measured on 2026-09-19 (532 across three daemons) was such a 429; the eight-exit sample below almost certainly was too. Non-429, absent, or unparseable statuses keep the handling described here unchanged.
A Claude invocation that exits with resp.TerminalReason == "api_error" (the apiErrorTerminalReason
constant) is, like §7.3’s usage-limit exit, a condition where the stage never ran — but unlike a usage
limit, it is per-invocation and self-resolving, not account-wide. Observed live on 2026-08-08 (#1458):
eight such exits across two issues, each at 1 turn and $0.0000, with three consecutive hits on #1408
incorrectly consuming its entire MaxRetries budget and pausing a PR that was otherwise mergeable and
green.
Detection: classifyAPIErrorExit(resp claudeResponse, usage TokenUsage) (engine/claude.go)
mirrors classifyUsageLimitExit exactly — structural-only (resp.TerminalReason == "api_error", never
output prose), with the same conjunctive usage.TurnsUsed > 0 && usage.CostUSD > 0 exclusion carried
over unchanged (a real api_error sample’s CostUSD could not be empirically captured during Research,
so the guard is reused rather than dropped or re-derived — see ADR-1458). interpretClaudeResult
checks this immediately after the classifyUsageLimitExit branch and before the diagnostic-only
unmatched-terminal_reason log line, returning a *claudeAPIErrorExit{TerminalReason, NumTurns,
CostUSD} sentinel in place of the generic error wrapper.
Handling — deliberately a distinct type, not a reuse of claudeUsageLimitError: claudeUsageLimitError
is the trigger, by Go type via errors.As, for two behaviors beyond handleUsageLimitExit itself:
activateClaudeUsageLimit (account-wide, in both runInvocationWithExtension and
runCommentExtensionLoop) and the comment-processing circuit breaker bypass (processComments). Both
are correct only for a genuine account-wide usage limit — an api_error says nothing about the account
as a whole, so claudeAPIErrorExit is a separate sentinel type that simply never matches either
errors.As(&claudeUsageLimitError{}) check, avoiding both behaviors by construction rather than by an
added conditional. finalizeStageOutcome() detects it via its own errors.As branch (alongside the
usage-limit and apiKeyHelper branches) and routes to handleAPIErrorExit(), which:
- Applies
itemstate.StageAttempted— the normal dispatch cooldown (retry_backoff, §7.1) applies on the stage-dispatch path, exactly as for a usage-limit exit. Deliberately does not callStageRetryIncremented— the stage never ran, so this does not count againstMaxRetries. - Logs only. Unlike
handleUsageLimitExit/handleAPIKeyHelperDetected, posts no comment and applies no label — a fifth, more minimal “did not run” shape. The condition is per-invocation and self-resolving on the next attempt; a durable label here would be indistinguishable from the orphaned-durable-state leak ADR-1183’s sweep exists to clean up, and a comment for a self-healing event is noise (see #1414’s “comment when a human must act, log when the engine self-heals”). - Releases the lock and returns — no
stage:<name>:failed, nofabrik:paused, no account-wide usage-limit hold.
The comment-triggered dispatch path (engine/comments.go) is deliberately unmodified. That path has
no LastAttemptAt-based cooldown at all (§7.3’s cooldown only exists in itemNeedsWork’s stage-dispatch
fallthrough); its only bound is the comment-processing circuit breaker (checkCommentBreaker, #1089,
default 10 invocations/30 min). Because claudeAPIErrorExit is a distinct type from
claudeUsageLimitError, it automatically falls through to the same checkCommentBreaker(item, "") call
every other unclassified error already reaches — unlike a genuine usage-limit hit, which
processComments deliberately exempts from the breaker. Counting an api_error toward the breaker is
correct here: it says nothing account-wide that would justify exempting an issue’s comment thread from
“no forward progress” accounting, and the breaker is the only mechanism that bounds this path (the
5-invocations-in-31-seconds burst observed on #1208 came from comment-triggered redispatch, not the
5-minute-cadence stage-dispatch cooldown).
Reinvoke cycle counters (#1812, ADR-1812). The exemption extends to the reinvoke path’s cycle budgets: an api_error (or usage-limit, usage-limit-gated, or apiKeyHelper) exit on a review, CI-fix or rebase reinvoke is refunded from ReviewCycles/CIFixCycles/RebaseCycles (and ReviewBlockedCycles when it was charged) rather than being permanently charged — see §6.2’s “Did-not-run cycle refund”. The comment circuit breaker (above) still counts the exit; only the cycle counters are compensated.
Regression guard: an api_error exit followed by a genuine failure leaves the full MaxRetries
budget available for the genuine failure, exactly as §7.3 describes for a usage-limit exit. A genuine
stage failure with no matching TerminalReason is entirely unaffected by this section — it still counts
against MaxRetries and still escalates via escalateFailedStage() (§7.2) at the limit.
See ADR-1458 and #1458.
7.3b Tool-Permission Denial Exemption
On 2026-08-09, a Claude Code profile misconfiguration made every mutating tool (Edit, Write,
any write-capable Bash) return a permission denial to Fabrik’s stage workers, while read-only
operations kept working. The engine responded to the same condition two incompatible ways
depending entirely on whether the worker happened to also emit FABRIK_BLOCKED_ON_INPUT: two
issues paused cleanly via the marker path, and two others (#1456, #1462) retried three times,
burned their full MaxRetries budget, and marked themselves stage:<name>:failed — a durable
claim that the work itself was wrong, when the machine had simply been unable to write files. See
#1523.
Structurally unlike every sibling in this family (§7.3, §7.3a, fabrik:api-key-helper-detected):
a tool-permission denial does not abort the invocation — the CLI exits cleanly (is_error: false,
subtype: "success", terminal_reason: "completed"), and real, committable work may have happened
before the denial (e.g. several successful edits before one write is blocked). There is also no
dedicated terminal_reason enum value to key off, unlike "blocking_limit"/"api_error" — the
only structural signal is the CLI’s own permission_denials array on the terminal result line,
confirmed empirically against the installed CLI (2.1.227) using Fabrik’s own invocation flags
(--output-format stream-json --verbose --permission-mode dontAsk, no
--dangerously-skip-permissions): a PreToolUse-hook-denied tool call populates this array with
one entry per denial (tool_name, tool_use_id, tool_input), on an otherwise ordinary clean
exit.
Detection: classifyToolsDenied(resp claudeResponse) (engine/claude.go) returns the
deduplicated, first-seen-order list of denied tool names, alongside a per-denial (not deduplicated)
[]toolDenial carrying each denial’s tool name plus — for Bash only, best-effort decoded from the
CLI’s tool_input — the specific command that was denied (#1775; see §7.3d). interpretClaudeResult
consults it only in the clean-exit path (runErr == nil), gated on
!completed — a denial the model worked around and still completed the stage is ordinary success,
with no exemption and no label, exactly matching every incident report and this section’s own
empirical reproduction. When the gate matches, interpretClaudeResult returns a
*claudeToolsDeniedError{ToolNames} sentinel in place of the unconditional nil the clean-exit path
previously always returned. Detection is scoped to the clean-exit path only, per the evidence
above — a diagnostic-only log line records a permission_denials array seen on a non-clean exit
(evidence-gathering for a shape not yet observed, never a classification), mirroring §7.3’s
unmatched-terminal_reason diagnostic.
Handling — continue-processing, not short-circuit: because real work can precede the denial,
*claudeToolsDeniedError follows §7.3a’s sibling claudeTurnLimitError/claudeResumeFailureError
shape, not §7.3/§7.3a’s did-not-run early return. toolsDenied := errors.As(err, &toolsDeniedErr)
is computed in finalizeStageOutcome (engine/item.go) alongside the pre-existing turnLimited/
resumeFailed checks; commitWIP, the branch push, markCommentsSeenByStage, and
InvocationRecorded (with Errored excluding toolsDenied, alongside turnLimited) all still run
exactly as for any other incomplete run — a late-invocation denial never discards earlier valid
edits. In the final escalation block, toolsDenied is a fourth branch alongside turnLimited/
resumeFailed:
itemstate.StageAttemptedwas already recorded unconditionally above (the normal dispatch cooldown applies).itemstate.ToolsDeniedRetryIncrementedincrements a distinct counter — deliberately notStageRetryIncremented(R2) — so a tool-permission denial never counts againstMaxRetries.- If
fabrik:tools-deniedis absent, posts an explanatory comment naming the denied tool(s) (toolsDeniedErr.ToolNames, joined) — plus the first denied command when the CLI’stool_inputmade one decodable (firstToolsDeniedCommand/sanitizeToolsDeniedCommand, #1775) — stating that the denial is scoped to that one command, not the tool for the rest of the session, then applies the label — gated on the label’s own absence, the same once-per-episode idiom asfabrik:claude-limit/fabrik:awaiting-ci. A repeated detection within the same episode posts neither a duplicate comment nor a duplicate label-add. - Compares the running count against
MaxToolsDeniedRetries(default 3 — see “TheMaxToolsDeniedRetriesbound” below); at the bound,pauseForToolsDeniedLimitappliesfabrik:paused+fabrik:awaiting-input(via the sharedpauseIssue/EnginePausedprimitives, mirroringpauseForSliceLimit’s shape exactly) — neverstage:<name>:failed. The condition is never treated as a stage failure, even at the bound.
Label clearing (R3, per-worktree self-resolution): gated on !toolsDenied — unlike
fabrik:claude-limit/fabrik:api-key-helper-detected, this invocation itself may be the condition
being classified (the detection does not short-circuit), so an unconditional clear at the same site
those two labels use would immediately erase the label the branch above is about to apply. On the
next invocation that is genuinely not classified as tools-denied, the label clears — mirroring
fabrik:api-key-helper-detected’s per-worktree self-resolution (a human fixes the permission
configuration directly) rather than fabrik:claude-limit’s account-wide settle sweep, since the
cause here is always local to one worktree’s environment, never account-wide. fabrik:tools-denied
is included in transientLifecycleLabels for the closed-issue defensive sweep (§7.5).
Marker independence (R6): because classification is expressed as a non-nil error returned from
interpretClaudeResult rather than a separately-threaded boolean, blockedOnInput := err == nil &&
CheckBlockedOnInput(output) (engine/item.go) is structurally unreachable once a denial is
detected — no explicit marker-suppression code exists anywhere, and none is needed. The outcome
(label, comment, counter, escalation) is identical whether or not the worker’s output also contains
FABRIK_BLOCKED_ON_INPUT: that marker is a worker-side courtesy, never an engine guarantee, and the
engine’s own structural classification is what governs the outcome — the asymmetry that let #1453
and #1498 recover cleanly while #1456 and #1462 burned their retry budget on the identical condition
cannot recur.
The MaxToolsDeniedRetries bound (R5, ADR-1523): defaults to 3 (--max-tools-denied-retries /
FABRIK_MAX_TOOLS_DENIED_RETRIES), lower than MaxSliceRetries (10 — a turn-cap preemption is
routine and self-resolving by construction). A denial is command-scoped and often self-resolves once
the worker reshapes the offending command (#1775, §7.3d) — unlike a genuine permission
misconfiguration, which would recur identically — but a handful of cycles is still a reasonable place
to draw the line before asking a human to look, higher than
MaxResumeFailures (2) since the explanatory comment already reaches the operator on the very first
detection (R4); the extra cycles before escalating guard only against a single spurious/flaky
denial, never against expecting a retry to fix the underlying cause. An exempt condition that never
escalates would be its own failure mode — an issue silently retrying an unwatched environment
problem forever — which is exactly what this bound exists to prevent, the same rationale as §7.12’s
slice budget.
Regression guard: a tools-denied detection followed by a genuine failure leaves the full
MaxRetries budget available for the genuine failure, exactly as §7.3/§7.3a describe for their own
conditions — ToolsDeniedRetries is tracked entirely independently of Attempts/MaxRetries.
See ADR-1523 and #1523.
7.3c Tool-Permission Denial Coverage on Reinvoke Paths
§7.3b’s bound (ToolsDeniedRetries/MaxToolsDeniedRetries, pauseForToolsDeniedLimit) originally
existed only on the stage-dispatch path (finalizeStageOutcome, engine/item.go) — every reference
lived in that one file. But interpretClaudeResult (engine/claude.go) is the single seam both
invocation paths share: Invoke (stage dispatch) and InvokeForComments (comment processing) each
receive an identical *claudeToolsDeniedError under the identical detection rule. processComments
(engine/comments.go) is the funnel every comment-driven invocation runs through — ordinary comment
review and all three reinvoke dispatchers (dispatchCIFixReinvoke, dispatchReviewReinvoke,
dispatchRebaseReinvoke) via the shared dispatchReinvoke scaffold — so before #1704 a mode denial
detected on any of those four invocation shapes fell into processComments’s generic warn log and
the comment circuit breaker (§4.6), which is calibrated for a different failure shape (a runaway
comment loop triggered by frequency) than a slow, deterministic denial recurring once per retry
cooldown. At a poll: 120 cadence with a 20-minute stage-retry cooldown, a denial recurs roughly 1.5
times per 30-minute window — structurally unreachable against the breaker’s default threshold of 10
per window — so the loop never converged: no terminal state, ~$0.30–0.40 burned per attempt,
indefinitely. The ci-fix-reinvoke case was the sharpest instance of this: CI had failed, so
ci-fix-reinvoke was the designed repair path, and it was the one disabled by the fault it exists to
recover from (#1657, #1704).
Fix: a new errors.As(err, &toolsDeniedErr) branch in processComments’s error handling
(alongside the pre-existing usage-limit exclusion, in the if err != nil && !completed block) —
covering all four invocation shapes in the single place they already share, rather than duplicating
the check inside each of the three reinvokeOpts.after callbacks. It calls the same
recordToolsDeniedDetection helper §7.3b’s stage-dispatch branch calls (extracted from the previously
inline block so the two consumers cannot drift), and, once the shared counter reaches
MaxToolsDeniedRetries, the same unforked pauseForToolsDeniedLimit. The branch returns before either
checkNoOpCommentCycle or checkCommentBreaker runs, so a tools-denied cycle is bypassed from the
comment breaker entirely, exactly mirroring how the usage-limit exclusion immediately above it is
bypassed — the breaker’s own thresholds are untouched.
Consequences of sharing the counter: ToolsDeniedRetries is keyed by (repo, issue, stageName)
only, not by which code path incremented it — so a denial on the stage path followed by a denial on
ci-fix-reinvoke for the same stage accumulate against the same counter and the same
MaxToolsDeniedRetries bound, and the outcome (pause at the bound, exemption from max_retries) is
identical regardless of which invocation type detected it (R3). No change was needed in reinvoke.go
itself — dispatchReinvoke’s error handling only logs whatever processComments returns; the
classification, accounting, and escalation are already complete by the time control returns there.
Remedy naming (R4, corrected by #1775 — see §7.3d): pauseForToolsDeniedLimit’s escalation
comment states the per-command scope, then offers remediation in order: try re-running the step as
separate, simpler commands, or add a matching allowed_tools rule for the command that keeps getting
denied, first; fabrik:unrestricted is named only as a last resort — a headless worker has no
interactive prompt to grant a denied tool, so it is the fallback once the command-scoped fixes have
been tried — with its trade-off (it removes all tool restrictions, not just the denied one) stated
alongside it.
See ADR-1704 and #1704, #1657, #1523.
7.3d Command-Scoped Messaging and Measured Progress Wording (#1775, #1743)
Two corrections to the messaging §7.3b/§7.3c describe, both landing in #1775 since #1743 targets the same log line #1775 already touches:
Denials are command-scoped, not tool- or session-scoped. ADR-1523’s original reasoning — that a
tool missing from --allowedTools “self-reports it unavailable, with no permission_denials entry,”
so a real denial “most plausibly involved a hook” — was explicitly flagged in that ADR’s own text as
“by strong inference,” never observed. A community report (#1741) directly contradicts it: within one
session, an ordinary git status && … Bash call ran, and a later, differently-shaped
base_branch=$(gh pr view …) Bash call was denied — impossible if a hook or mode had disabled Bash
session-wide. The reporter’s own 745-denial census attributes denials to command shape (/tmp
redirects, unallowed binaries, env-prefix/variable-assignment forms, cd-chains, conditionals), not to
session or tool. ADR-1775 supersedes this specific piece of ADR-1523’s reasoning (and ADR-1704’s
repetition of the related decision_reason_type: "mode" framing) without rewriting either — both
remain as accurate historical records of the decisions as made at the time.
Consequently: classifyToolsDenied now also returns per-denial []toolDenial (tool name plus, for
Bash only, a best-effort tool_input.command decode — decodeToolCommand, silently degrading to
"" for absent/non-Bash/malformed input, never a panic). sanitizeToolsDeniedCommand renders a
denied command safely for inline display: embedded newlines collapse to spaces, backticks become
straight quotes, and the result is truncated via the existing truncateMiddle convention
(engine/merge_train.go) with short constants sized for a single command line rather than
truncateMiddle’s CI-log-sized defaults. Both recordToolsDeniedDetection’s initial comment and
pauseForToolsDeniedLimit’s escalation comment name the first available command this way, state the
per-command scope explicitly, and no longer contain “no retry can fix this on its own” — the phrase
this section’s earlier text described, now removed from both the comment templates and
claudeerr.ToolsDeniedError’s doc comment. ToolsDeniedError gained an additive Denials
[]ToolDenial field alongside the pre-existing ToolNames; every construction site that only ever set
ToolNames keeps compiling and behaving unchanged.
The tools-denied log line no longer asserts unmeasured progress (#1743). Independently of the
command-scoping fix, interpretClaudeResult’s tools-denied branch previously logged “stage did not
make progress” — a claim derived purely from !completed (the absence of FABRIK_STAGE_COMPLETE),
never from checking the worktree. This directly contradicted this document’s own §7.3b text (“real,
committable work may have happened before the denial”) and plugin/fabrik-workflows/LABELS.md’s
shipped fabrik:tools-denied entry (which already correctly says commits and pushes still happen). A
reported instance: an Implement comment-review invocation made three commits (the entire fix, pushed),
spent 80 turns, and was logged as having made no progress — the next run then treated the review
comment as already handled and did nothing. The fix: runClaude captures the worktree’s HEAD before
starting the Claude process and again immediately after cmd.Wait() returns (best-effort,
gitHeadSHA, mirroring the existing headBefore/headAfter pattern dispatchReviewReinvoke uses for
review-reinvoke, #1045), computes a commit count between them (gitCommitCountBetween, git rev-list
--count) when both SHAs are non-empty and differ, and passes that count into
interpretClaudeResult as a new parameter. The tools-denied branch reports “N commit(s) pushed, stage
did not signal completion” when a positive count was measured, or “stage did not signal completion”
— exactly what !completed actually establishes — when it wasn’t; it never asserts an unmeasured
claim, and never prints “0 commit(s)”. The string “did not make progress” no longer appears anywhere in
the engine.
See adrs/1775-command-scoped-tools-denied-messaging.md, #1775, #1741, #1743.
7.3e Toolchain Declaration Drift Detection
A long-lived daemon inherits PATH once, from whatever shell hosted it at process start. A worktree
can later declare a toolchain version (.nvmrc, package.json engines.node, go.mod’s
toolchain/go directive, or .tool-versions) newer than what that inherited PATH resolves, and
nothing notices — workers keep running the stale toolchain silently, indefinitely; restarting fabrik
itself does not help, since it re-inherits the same PATH from the same parent shell. Reported in
#1658: a daemon whose hosting shell started 2026-08-11 kept resolving Node 20.20.2 after the repo
pinned Node 24 via .nvmrc on 2026-08-23, producing 36 Unsupported engine warnings in a single
issue’s worker logs before the mismatch was noticed.
Detection is structurally unlike every other exemption in this section (§7.3–§7.3c): it never skips
the invocation. checkToolchainDrift (engine/toolchain.go) is a direct, best-effort side-effecting
call at the top of both invocation paths — runInvocationWithExtension (stage dispatch, immediately
after the apiKeyHelper check) and processComments (comment review, right after context files are
written) — never routed through finalizeStageOutcome, never returning an error, never touching
StageAttempted/StageRetryIncremented/max_retries. There is nothing for Fabrik to do about a
detected mismatch except tell someone: re-resolving PATH or driving a version manager (nvm use,
asdf install, …) is explicitly out of scope (#1786 R3).
detectToolchainDriftreads whichever declaration files exist in the worktree and, for each comparable declaration, resolves the corresponding binary (node --version/go version) viaexec.LookPathagainst the daemon’s own inheritedPATH. A binary entirely absent fromPATHis “cannot compare” — a distinct, pre-existing condition, never a mismatch. A successful resolution is memoized process-wide per tool for the daemon’s lifetime (resolveToolVersion): since the bug’s own premise is that thisPATHcannot change without a restart, a second check for the same tool, on any issue, is a cache hit rather than a second shell-out. A resolution failure (subprocess timeout, transient exec error) is deliberately never cached — a failure carries no such invariance guarantee, so caching it would permanently and silently disable detection for that tool after a single bad moment; every invocation with a matching declaration retries an uncached failure instead.- Comparison is deliberately narrow and fails closed on ambiguity:
.nvmrc,package.jsonengines.node, and.tool-versionscompare at major-version granularity only (an.nvmrcalias likelts/*, or anenginesrange using||/x/multiple clauses, is “not comparable” — silence, never a false-positive warning);go.mod’stoolchain/godirective compares at major.minor granularity using minimum-version (>=) semantics, since ago.moddirective states a floor, not a pin — andGOTOOLCHAIN=auto(the Go 1.21+ default) already self-corrects in the common case, so this format mainly catches theGOTOOLCHAIN=local/auto-download-disabled edge case. - On any comparable mismatch, gated on
fabrik:toolchain-stale’s own absence: an explanatory comment naming every mismatched declaration (source, declared version, resolved version) is posted and the label applied — once per episode (per issue), not once per invocation (#1786 R5). - If no comparable mismatch is found and the label is present, it is cleared — self-resolving once a
worktree’s declaration file is fixed, or the daemon is restarted with a corrected
PATH.
A daemon/worktree whose resolved toolchain already satisfies every declaration sees no new behavior and no new output at all (#1786 R4) — no comment, no label, no added latency worth noticing beyond a few small file reads and, at most once per tool per daemon lifetime, a version-check subprocess.
See §1.4’s fabrik:toolchain-stale row, docs/stage-lifecycle.md’s “Toolchain Declaration Drift
Detection (Worktree)”, and ADR-1786.
7.4 Multi-Instance Lock Protocol
Per ADR-007:
- Instance acquires its own lock label,
fabrik:locked:<identity>—e.lockLabel(), where the identity (e.lockIdentity()) iscfg.Userin PAT mode and the App-derived ID under GitHub App auth (see “Lock identity under App auth” below) - Waits
lockVerifyDelay(2 seconds) for competing instances to place their locks - Re-fetches labels via
FetchLabels() - If another
fabrik:locked:*label is present: lexicographic tie-break — lower username wins (proceeds), higher username loses (releases lock and skips) - Winner proceeds with stage invocation; loser returns nil
Edge cases:
- Identical identities: both proceed (unsupported configuration). Under App auth two instances never share an identity by construction, so this only arises in PAT mode with the same
user: - API error on re-fetch: winner proceeds (optimistic; logs warning)
- Lock is released on: completion, permanent failure (MaxRetries), blocked-on-input, or lock conflict loss. NOT released on cooldown retry.
Lock identity under App auth (#1893, ADR-1893). With no operator user:, the lock identity is <slug>-<hash>: the App slug (no [bot]) plus six hex characters of sha256(hostname NUL abs(fabrikDir)). Two local instances sharing one App installation therefore hold different labels, which keeps every check above meaningful — the label != lockLabel “held by someone else” test, the lexicographic tie-break, and the own-label-only cleanups. One shared fabrik:locked:<slug> label would make each instance read the other’s lock as its own, let a starting instance strip a live lock in runStartupCleanup, and reduce the tie-break to “both win”. fabrik:locked: (14 chars) leaves 36 of GitHub’s 50-character label limit, so the slug is truncated to 29 characters (deterministically; the hash keeps truncated slugs distinct). The label is stable across restarts on the same host and directory; moving the directory or renaming the host changes it and orphans a crashed instance’s old label until removed by hand. FABRIK_INSTANCE_ID, when set, replaces the hostname in the hash (container/pod deployments, where the hostname changes on every restart); startup logs the derived label. cfg.User, if still configured under App auth, is never used as the identity (R8); it is read only by the one-shot legacy sweep below. itemstate.LocalLockAcquired{User} and SeedLabels(…, lockedUser) carry the same identity.
Legacy lock sweep. A deployment that switches from PAT to App mode may have crashed holding fabrik:locked:<user>; under the new identity that label reads as “locked by another instance” and would strand the item. runStartupCleanup therefore additionally removes exactly fabrik:locked:<cfg.User> (only when App auth is active and user: is still set) from items with no live local Worker(). It does not sweep other fabrik:locked:* labels, which may belong to a live instance. Because nothing ties the <cfg.User> label to this instance (a PAT-mode instance still running as that user legitimately holds it), the sweep also skips a label applied less than legacyLockMinAge (2h) ago, or whose age cannot be read (FetchLabelAppliedAt, live), logging that the label was left for a human to remove.
Age-window limitation. The 2h window is a heuristic, not a liveness proof: no process can see whether another process is alive. It only matters when a PAT-mode instance and an App-mode instance share a username and a board. In that case a PAT-mode stage that runs longer than 2h (possible with no max_wall_time) could have its live fabrik:locked:<user> lock swept when the App-mode instance starts. Deployments without that mix are unaffected.
Host and directory instance locks (#2097, ADR-2097). The label protocol above arbitrates between engines that are already running; the flocks below keep a second engine from running at all. Run() takes them before anything else touches disk (acquireInstanceLocks, engine/instance_lock.go), in this order: (1) checkNotInWorktree(fabrikDir) refuses a directory with a .fabrik/worktrees path-component pair (symlinks resolved); (2) the directory lock, an exclusive non-blocking flock on <fabrikDir>/.fabrik/fabrik.lock whose content stays exactly "<pid>\n" (the live e2e gate parses it as a bare pid); (3) the board lock, a flock on <lockdir>/<slug>-<8 hex>.lock where <lockdir> is $FABRIK_LOCK_DIR or os.UserCacheDir()/fabrik/locks (mode 0700; no resolvable directory is a startup error, never an unlocked start) and the key is lower(host)/lower(owner)/projectNumber (empty ghes_host = github.com; OwnerType excluded). The board file holds one JSON line {"pid","dir","started"}, written in a single write only after the flock is owned. A refused engine reads that record through a separate open, retrying briefly, and names the holder’s pid and directory (or says the owner could not be identified); it releases the directory lock it had taken. All of this precedes log rotation, so a refused engine never rotates the winner’s log. Both fds are close-on-exec and never given to a child.
Per-poll verification (R3). Run()’s doPollCycle closure calls instanceLocks.Verify() before PollWithBackoff: for each lock, fstat of the held fd against stat of the path (dev+inode), then the recorded pid read from the held fd must equal os.Getpid(). It logs nothing on success. Failures are classified: a definitive loss — the path no longer exists (ENOENT), a dev/inode mismatch, a record naming another pid, or an empty/unparseable record — is fatal at once; any other fstat/stat/read error (a momentary EIO/ESTALE/EACCES/EINTR, e.g. an NFS home or a flaky cache mount) is inconclusive: it logs a warning and the lock is treated as still held, and only maxInconclusiveVerifies (3) consecutive inconclusive verifications of one lock count as a loss (a clean verification resets the count). acquireDirLock returns a failed pid write (e.g. disk full) at startup rather than leaving it to surface at the first poll as a lost lock. On a loss it prints lost the instance lock; another Fabrik may own this board (<which lock>: <reason>) to stderr and fabrik.log, sets Engine.lockLost, cancels the run context and returns errInstanceLockLost; drainAndExit then skips cleanupLockedIssues (in PAT mode another engine may legitimately hold fabrik:locked:<user>), never takes the SIGHUP branch, waits the bounded drain and returns the error, so the process exits non-zero. Verification is not in poll()/PollOnce (the sim seam deliberately skips the lock preamble, ADR-1449), and cycles triggered through the test-only poll seam bypass it.
Re-exec. performSighupRestart releases both locks immediately before syscall.Exec; a self-upgrade re-exec relies on close-on-exec. The pid is unchanged across exec, and Run() re-acquires both locks in the new image. If a competitor wins the gap it keeps the lock and the re-exec’d image exits. Older binaries and a second host are not covered.
7.5 Closed-Issue Catch-Up
Closed issues are normally skipped by itemMayNeedWork() and itemNeedsWork(). Exceptions:
- Cleanup stages: A closed issue in Done with a worktree still needs cleanup
- Complete-labeled items: A closed issue with
stage:<X>:completecan be advanced by the catch-up loop (e.g., PR merge closes the issue while it’s in Validate with the complete label — it needs to move to Done)
Stale lock cleanup: cleanupClosedIssueLocks() runs every poll cycle and removes fabrik:locked:<user> labels from any closed issues on the board. This handles stale locks left when an issue was closed while a stage was in-flight.
Transient label sweep: cleanupClosedIssueTransientLabels() runs every poll cycle (immediately after cleanupClosedIssueLocks) and removes the labels in transientLifecycleLabels (fabrik:awaiting-review, fabrik:awaiting-ci, fabrik:auto-merge-enabled, fabrik:awaiting-input, fabrik:rebase-needed, fabrik:bot-reprompted, fabrik:revalidate, fabrik:claude-limit, fabrik:clear-claude-limit, fabrik:api-key-helper-detected, fabrik:skill-unresolved, fabrik:tools-denied, fabrik:toolchain-stale) from any closed issues. These labels have no meaning once an issue is closed and leaving them behind can cause confusion. The sweep is idempotent (treats ErrNotFound as success) and non-fatal (API errors are logged as warnings and processing continues). Operates on cache data from e.readClient.FetchProjectBoard(...) — labels come from the most recent deep-fetch (FetchItemDetails, up to 20 labels). Items never deep-fetched carry no labels in the store (probe-based bootstrap does not populate labels). In practice, closed terminal items are seeded by seedTerminalFromProbeItems (called after BootstrapFromProbe) and skipped by runProbeAndDeepFetch, so they will not be deep-fetched; the sweep sees empty label sets for these items and skips them (no matching transient labels to remove). Active items receive labels on the first deep-fetch cycle. A one-shot startup scan (runStartupTransientLabelScan) runs after the first successful poll to handle stale labels on closed issues that may have been missed during a prior crash — it scans the Store directly, no extra GitHub call needed.
7.6 In-Memory vs Durable State
| State | In-Memory | Durable (Label/Reaction) | Behavior on Restart |
|---|---|---|---|
| Stage invocation timestamp | itemstate.Store → StageState.LastAttemptAt[stageName] |
None | Lost — item retried immediately |
| Periodic re-eval cooldown | itemstate.Store → ItemState.CooldownAt["periodic-re-eval"] |
None | Lost — item retried immediately |
| Retry count | itemstate.Store → StageState.Attempts[stageName] |
None | Lost — retries restart from 0 |
| Paused-due-to-retries | itemstate.Store → StageState.PausedByEngine[stageName] |
fabrik:paused + stage:<X>:failed |
Labels survive; in-memory flag lost but processItem() detects the failed label directly |
| Review cycle count | itemstate.Store → StageState.ReviewCycles[stageName] |
None | Lost — cycle count restarts from 0 |
| CI-fix cycle count | itemstate.Store → StageState.CIFixCycles[stageName] |
None | Lost — cycle count restarts from 0 |
| Rebase cycle count | itemstate.Store → StageState.RebaseCycles[stageName] |
None | Lost — cycle count restarts from 0 |
| CI merge pending since | itemstate.Store → LinkedPRState.CIMergePendingSince |
None | Lost — merge guard re-evaluates CI fresh on next poll |
| Comment processed | itemstate.Store → StageState.ProcessedComments[commentID] |
ROCKET (🚀) reaction (review bodies: the durable review-ids-addressed marker; the 🚀 is GraphQL and human-visible only, #1953 R8) |
Reaction survives restart; in-memory dedup is defense-in-depth |
| Lock tracking | itemstate.Store → ItemState.Lock |
fabrik:locked:<user> label |
Label may survive if process crashes; cleanupLockedIssues() runs on graceful shutdown. Lock is set only by stage dispatch; a reinvoke registers a Worker only (#2179, ADR 2179) |
| Change-feed set | Engine.mayNeedWork[iKey] (Engine.mayNeedWorkMu) |
None | Lost — all items re-evaluated on first poll |
| Deep-fetch failure | itemstate.Store → ItemState.LastDeepFetchFailureAt |
None | Lost — failed items retried immediately |
| Channel subscriptions, held events, dedup keys, settle-episode counters (#1968) | channelevents.Hub (mirrors of the files) |
.fabrik/state/channel/subscriptions.json, held-<hash>.json, dedup.json (atomic writes, corrupt files quarantined) |
Survive — loaded before the socket binds; held events are delivered in order on re-attach |
| Channel deriver’s per-item memory (last pause-family state, counter arming, last comment link) | engine.channelEvents (consumer goroutine only) |
None | Lost — re-seeded silently from the store, so a restart announces nothing about the existing board |
7.7 Invocation-Level Kill Mechanisms
Two proactive kill mechanisms cap how long a single Claude invocation can run. Both are implemented in runClaude() in engine/claude.go and operate independently of the engine-level context cancellation.
max_wall_time (per-stage YAML field)
- Configured as a Go duration string in stage YAML (e.g.,
max_wall_time: "45m"). Absent or zero means no cap. - Implemented via
context.WithTimeoutwrapping the invocation context; the clock starts when the process is spawned. - When the deadline fires,
cmd.Cancelexecutes the graceful kill sequence: SIGTERM to the process group, 10-second grace, SIGKILL. - Recommended for long-running stages (Implement, Review) to bound worst-case hang time.
- Scaled to match a per-invocation turn-budget override (
InvokeOptions.MaxTurnsOverride), viascaledWallTime(stage.MaxWallTime, effectiveBudget, baseBudget)inengine/claude.go.InvokeClaudeandInvokeClaudeForCommentseach compute this from the same values that already determine the per-call turn limit —effectiveBudget/stage.MaxTurnsfor stage invocations,limit/commentMaxTurns(stage)for comment processing — and pass the scaled duration torunClaudein place of the rawstage.MaxWallTime. The result is a no-op (basereturned unchanged) whenever there is no cap, no baseline to scale against (stage.MaxTurns == 0, i.e. unlimited turns), or no extension is in effect for this call (effectiveBudget <= baseBudget— true for every ordinary invocation and every progress-based extension iteration, both of which pass an unmultiplied budget). It scales proportionately only forfabrik:extend-turns’ label-gated first-invocation 2× pre-grant (§3.2, §4.3), whereeffectiveBudget = 2 × baseBudget— matching wall-clock headroom to the inflated turn budget instead of killing a legitimately-progressing extended run on a clock sized for the un-extended case. See ADR-1206. - Exception — merge-train conflict resolution always gets a nonzero deadline.
resolveConflictWithClaude(engine/merge_train.go) never dispatches with the holding stage’s rawMaxWallTime, even when it’s absent or zero.conflictResolutionStagesubstitutesmergeTrainConflictWallTimeFallback(30 minutes) via a local, non-shared copy of the holding stage whenever its ownmax_wall_timeis unset —holdingStage(e.cfg)returns the single process-wide*stages.Stageshared by every concurrently running (repo, base) merge-train worker (ADR-1648), so the substitution must never mutate that pointer in place. The fallback value flows into the ordinaryscaledWallTimecall above with no further wiring, sofabrik:extend-turnsscaling applies to it exactly as it would to an explicit YAML value. This closes the gap where a stuck-but-still-emitting-output conflict-resolution invocation could otherwise run unbounded, relying solely on the inactivity timeout below (which never fires while output keeps arriving). See ADR-1500.
Inactivity timeout (hardcoded 15 minutes)
- A watchdog goroutine resets a 15-minute timer on every byte of stdout received via
activityWriter. - When no output arrives for 15 consecutive minutes, the watchdog calls
killProcGroupGraceful()directly and setsinactivityFired. - Acts as backstop for stages with no
max_wall_time(or when the process produces occasional output but never completes).
Shared post-kill behavior:
extractTextFromAssistantTurns(rawOutput)scans the already-buffered output forFABRIK_STAGE_COMPLETEappearing in thetextcontent of any{"type":"assistant"}NDJSON line.- If found: returns
completed=true— the invocation is treated identically to a liveFABRIK_STAGE_COMPLETE. - If not found: returns
completed=false— routes to cooldown/retry (see §3.2 and §7.1), not a hard error.
wasTimedOut flag: inactivityFired.Load() || (stageCtx.Err() == context.DeadlineExceeded && ctx.Err() == nil) — distinguishes our kills from engine-shutdown context cancellation. When wasTimedOut=true, the no-marker path follows the same cooldown/retry flow as a clean exit without markers. When ctx.Err() != nil && !wasTimedOut (engine shutdown), runClaude returns immediately with zero output.
Kill sequence: killProcGroupGraceful(pid, issueNumber, label, reason, sigintGrace, sigtermGrace) runs the session-wide SIGINT → grace → SIGTERM → grace → SIGKILL escalation of internal/sessionreap over every process in the worker’s session and its sampled command sessions (#1989; see docs/stage-lifecycle.md “Kill escalation”), not a single SIGTERM to -pid. This terminates grandchild processes (e.g., background sleep spawned by Monitor tool) that would otherwise hold the stdout pipe open past cmd.WaitDelay. The post-exit killProcGroup group signal and the escalation’s listing-failure fallback are ownership-checked before they signal (#1957, “Stored-PID group kills” in the same doc).
No-op on Windows: killProcGroupGraceful is a no-op on Windows (process groups work differently). Both timeout mechanisms still fire and set their flags, but the kill is a best-effort cmd.Cancel.
7.8 Poll Loop Idle Backoff and Webhook Health State
The poll loop’s effective interval grows when there is nothing to do (idle backoff). The cap on that backoff depends on the webhook stream health state.
Idle backoff algorithm (computeEffectiveInterval in engine/backoff.go):
- Base interval:
PollSeconds(default 30s). - Activity multiplier: doubles each poll cycle where no work was dispatched, up to the idle cap.
- Activity reset: only a poll cycle where
result.Active == true(work was dispatched or a deep fetch ran) resetsidleStartand the multiplier back to 1×. AwakeChsignal triggers an immediate poll but does not unconditionally reset the backoff — the multiplier is only reset if that poll finds active work. Exception (#1716): this assumes the wake reaches a poll at all — while rate-limit backoff is active (see the wake-path gate below), it does not; see §7.9. - Rate-limit adjustment (
nextRateLimitLow+computeEffectiveIntervalinengine/backoff.go):- Activation: rate-limit backoff engages when remaining GraphQL quota drops below 20% (
rateLimitBackoffThreshold). The actual remaining fraction is passed tocomputeEffectiveIntervalasrateLimitRatio. - Clearance: backoff clears only when quota rises above 50% (
rateLimitHealthyThreshold). Between 20% and 50% the state is sticky — backoff remains active to prevent thrashing on boards where quota fluctuates near the activation point. - Stepwise escalation: the multiplier scales with depletion depth — 2× at >=10% remaining (includes the 20%–50% sticky zone), 4× at >=5% and <10%, 6× at >=1% and <5%, 10× (
rateLimitMaxBackoffMultiplier) below 1%. - No idle cap: the rate-limit component has no 5-minute ceiling; it is capped only at
rateLimitMaxBackoffMultiplier × configuredInterval. - Independence from idle backoff: activity detection (items dispatched) resets idle backoff but does NOT reset rate-limit backoff.
- Wake-path gate (#1716, ADR-1716): the escalated interval computed above only ever reached the poll loop via the ticker (
ticker.Reset(result.NextInterval)inRun()’sdoPollCycle) — the separatecase <-e.wakeCh:branch calleddoPollCycle()unconditionally, with no reference to backoff state at all. Under--webhooks, inbound GitHub events are not themselves rate-limited and kept arriving while every GraphQL call was being rejected, so a rate-limited daemon’s own webhook traffic (including its self-feedback loop, §7.9) fed the loop at whatever rate events arrived — observed in production sustaining tens of polls per second and draining a fresh hourly GraphQL budget within seconds of every reset.Engine.wakeBlockedByRateLimitBackoff()(engine/backoff.go) closes this:Run()’s wake branch now checks it before callingdoPollCycle(), returning true (drop the wake, no poll) wheneverbackoffRateLimitLoworbackoffRestPausedis set. See §7.9 for the drop-vs-coalesce design and the ordering guarantee that this does not delay the legitimate rate-limit-recovery self-wake below. - Poll-rate floor (R3, #1716): independent of the gate above,
PollWithBackoffitself refuses to reache.poll()more than once perminPollInterval(a fixed 500ms,engine/backoff.go) regardless of what triggered the call — ticker, wake, or any future caller. This is defense-in-depth, not the fix for the wake-path bug itself: it bounds any future bypass of the gate above to a slow leak rather than an unbounded-rate loop. A floor-blocked call returns immediately (PollBackoffResult{NextInterval: minPollInterval - elapsed}) without touchingRateLimitStats()ore.poll().
- Activation: rate-limit backoff engages when remaining GraphQL quota drops below 20% (
REST/core exhaustion is a hard pause, not a slowdown (shouldPauseForRESTRateLimit in engine/backoff.go): the interval adjustment above conserves the GraphQL budget (spent by the poll read), but the REST/core budget is spent by per-item mutations (reactions, labels, comments, merges) and janitor fetches, which interval-stretching does not throttle. So when REST remaining drops to near zero (≤1% of limit, isRateLimitNearZero), doPollCycle skips the entire work phase — fetch, dispatch, and mutations — until GitHub’s hourly reset (reset + rateLimitResetBuffer), emitting RateLimitAlertEvent{Bucket: tui.RateLimitBucketREST, Exhausted: true}. The independent worktree-janitor goroutine (runWorktreeJanitor) applies the same gate on entry, so it does not keep spending REST while the poll loop is paused. REST stats are refreshed from the headers of every REST response (including 403s), so the reset timestamp is authoritative even at zero budget. Introduced to stop a 403 retry storm after the REST budget was drained (#1118/#1121). RateLimitAlertEvent.Bucket (tui.RateLimitBucketREST |
tui.RateLimitBucketGraphQL) identifies which of the two independent budgets an emit/clear event concerns, so AlertBannerComponent (tui/alert.go) tracks each bucket’s active/reset state separately and never lets one bucket’s Exhausted: false clear a banner the other bucket raised — see #1482. The GraphQL-only banner string reads “polling backed off,” not “polling usage limited” (#1716) — GraphQL exhaustion only slows polls via the interval/gate machinery above, it does not stop them the way REST exhaustion’s hard gate does; the REST-only and combined banner strings are unchanged and remain accurate. |
Rate-limited probe logging is a per-attempt statement, not a usage-limit hold claim (R2, #1716). runProbeAndDeepFetch (engine/terminal.go) runs inside poll(), before PollWithBackoff’s GraphQL hysteresis block (above) computes for that cycle — so when its own ProbeProjectBoard call fails on a low/exhausted GraphQL budget, it has no visibility into whether the real backoff this section describes is active, and no usage-limit hold mechanism of its own. Its log line describes only what happened at this call site (one probe attempt was rejected for rate limit) rather than claiming a stand-down it does not control; the RateLimitAlertEvent{Bucket: GraphQL, Exhausted: true} emission is unchanged (it remains the sole source of the GraphQL-exhausted TUI signal). Log-rate protection (R4, #1716): this log line is routed through Engine.logfThrottled (engine/logthrottle.go) — a shared dedup helper that suppresses a repeated identical message under the same key until either the message changes or a fixed interval elapses, appropriate here since this line’s content (a minute-granular reset time, a mostly-stable error string) is genuinely static for the duration of one rate-limit episode. The three unconditional per-poll rate-limit stats/warning lines inside poll() (REST remaining, GraphQL remaining, GraphQL-low warning) instead use Engine.logfThrottledByInterval — a variant that throttles purely by elapsed time, ignoring message content entirely: their interpolated remaining-count changes on nearly every poll in ordinary (non-rate-limited) operation, so logfThrottled’s message-change-forces-log rule would have defeated the throttle for exactly the routine case it needed to cover (caught in review). Before this, a sustained rate-limited condition reproduced the same handful of lines every poll cycle; combined with the pre-#1716 wake-path bug this produced 39 log lines/second sustained in production (14 MB/hour). Both are thin wrappers over the same underlying dedup map — see engine/logthrottle.go’s doc comments for the exact semantics of each.
Idle cap selection (effectiveIdleCap in engine/backoff.go):
| Webhook stream state | Idle cap |
|---|---|
WebhookStreamHealthy |
60 minutes (webhookIdleCap) |
WebhookStreamStartingUp |
60 minutes (webhookIdleCap) |
WebhookStreamUnhealthy |
5 minutes (maxIdleBackoff) |
| Webhook mode disabled | 5 minutes (maxIdleBackoff) |
When the webhook stream is healthy or starting up, steady-state polling is suppressed to a 60-minute safety-net interval. Events that arrive on the webhook stream signal the wakeCh channel, which triggers an immediate poll regardless of the current interval. The backoff multiplier is preserved unless that poll finds active work (see “Activity reset” above).
Webhook stream health states (managed by webhookManager in engine/webhook.go):
| State | Meaning | Idle cap used | TUI indicator |
|---|---|---|---|
WebhookStreamStartingUp |
Subprocess launched; no reconcile tick has completed yet. State persists until the first reconcileTicker tick confirms no drift. |
60 min | Blue ○ |
WebhookStreamHealthy |
Most recent reconcileTicker tick found no drift between in-memory cache and GitHub |
60 min | Green ● |
WebhookStreamUnhealthy |
Cache is known to be out of sync (most recent reconcileTicker tick found drift, Pause/Reconcile/Resume in progress), OR subprocess failed to start / was permanently disabled by the 422 circuit-breaker |
5 min | Yellow ◌ |
State transitions (reconcile-driven — webhooks no longer drive health state):
StartingUp → Healthy: firstreconcileTickertick completes with no drift (or drift is reconciled).StartingUp/Healthy → Unhealthy:reconcileTickertick finds drift between cache and GitHub; triggers Pause → Reconcile(freshBoard) → Resume.Unhealthy → Healthy: immediately after Reconcile+Resume completes successfully within the same tick.* → StartingUp: subprocess restart (secret rotation, crash recovery) resets state; next tick re-evaluates.
Webhooks continue to apply deltas to the cache via ApplyDelta but no longer drive health state. Event silence does not produce health transitions — only LightReconcile drift detection does. LightReconcile discounts engine writes the listing cannot have seen (D.4, #2159), so an item whose only difference from the listing is a recent engine write neither marks the stream unhealthy nor triggers a reconcile.
The reconcile ticker runs independent of webhooks (poll-only correctness backstop). reconcileLoop (engine/reconcile.go) is launched unconditionally whenever the cache is active — it is not nested in the webhook-manager start path. This is required because webhooks are an optimization, not a correctness requirement: on a webhook-less deployment (or when the gh webhook forward subprocess fails to start), the reconcile ticker is the only mechanism that re-syncs the cache with GitHub, and it must still run. When the webhook manager is present it also receives the health-state transitions above; when it is absent (wm == nil) the drift detection and Pause → Reconcile → Resume repair still run, only the health-state signaling is skipped. LightReconcile drift detection compares status, updatedAt, and the fabrik-managed label set (fabrik:* / stage:* labels only — the board query truncates labels at 30, so the full set is not compared; the gate-label subset is small and never truncated). Comparing the gate-label subset closes the case where a store label set diverges from GitHub with a matching updatedAt (e.g. updatedAt advanced by a deep-fetch that syncs updatedAt but not labels), which would otherwise strand an item at a gate (e.g. fabrik:awaiting-ci missing from the store) indefinitely on a webhook-less deployment.
Webhook mode is always non-fatal. If the gh webhook forward subprocess fails to start, the stream state stays Unhealthy and the 5-minute idle cap applies. The poll loop (and the reconcile ticker) continues normally.
Idle backoff never sleeps past a due retry (#2232, ADR-2232). After computeEffectiveInterval, PollWithBackoff caps the interval at the earliest unexpired short engine-scheduled cooldown (retry-backoff, sentinel-live, merge-unsettled) found by one in-memory Store.Scan (earliestShortCooldown; cleanup-stage and closed items and expired entries are ignored), floored at 1 s (capIntervalToDue), and logs next poll capped to <d>. Long cooldowns (periodic-re-eval, dep-blocked, review-blocked, feedback-pending, spawn-*) deliberately do not cap, or the webhook-healthy 60-minute idle cap would shrink to 10 × poll. The cap only ever shortens the interval, so setNextPollDue/setPollPeriod stay correct, and it is skipped while GraphQL rate-limit backoff is active. The webhook wake path is unchanged.
References: ADR-032: Webhook-Driven Event Delivery
7.8a Test-Only Poll Hold/Trigger Seam (#1978)
Test-only — not a production feature. Enabled solely by the hidden environment variable
FABRIK_TEST_POLL_CONTROL(Config.PollControlFile), which the live e2e gate runner and harness set on the bed engine and nothing else does. It has no flag, no YAML key and no entry indocs/USER_GUIDE.md. Unset (the production state)Engine.pollSeamis nil, every hook below is a nil check, and the engine starts no goroutine, touches no file and logs nothing extra — the poll loop is behaviourally identical to one without the seam. Distinct from the sim’sEngine.PollOnceseam (ADR-1449), which bypassesRun(); this one gates the realRun()loop. See ADR-1978.
Some live scenarios need the engine to take one decision in one specific poll — their subject is a state window (TestMergeTrainQueuedDeeperThanBatchCap: “the engine sees exactly 7 Queued members at once”). With the seam enabled the harness can hold automatic polls, build the window, then trigger exactly one complete poll and read back when it finished.
What “held” covers. Every path that starts poll() or poll-equivalent work goes through one of two hooks, and a held cycle is dropped, never queued, so the window cannot move:
| Trigger | Hook |
|---|---|
Startup poll, ticker, wakeCh wake (including PollWithBackoff’s own GraphQL-recovery self-wake, which only ever sends on wakeCh) |
doPollCycle → pollSeam.ordinary; no select in Run() is restructured |
reconcileLoop tick (LightReconcile, cache reconcile) |
pollSeam.runUnlessHeld around the extracted reconcileTick |
No dispatch, catch-up phase, settle scan or light reconcile runs while held. Workers already dispatched by an earlier poll (item workers, a merge-train worker goroutine) keep running: hold gates new cycles only. A hold never blocks ctx.Done()/the drain — the held path returns immediately.
A trigger runs one complete poll: Run()’s own pollCycle closure (PollWithBackoff — minimum-poll-interval floor, REST hard gate, poll(), idle/GraphQL backoff bookkeeping — then the ticker reset), so backoff state is real, not reduced. PollBackoffResult.Ran is true only when poll() executed. A call that hit the 500 ms floor is retried a bounded number of times; anything else that ran no poll is reported blocked, never ran. A poll error is reported error.
Protocol (internal/pollctl; two single-writer JSON files, both replaced by atomic rename). The harness writes <bed>/.fabrik/poll-control.json (hold, hold_until, hold_gen, trigger_seq); a watcher goroutine — started only when the seam is enabled, polling every 200 ms — applies it and writes poll-control.ack.json (held, applied hold_gen, done_seq, outcome ∈ ran/error/blocked, detail). poll_seam.go’s pollMu serializes poll execution: an ordinary poll, a triggered poll and the application of a hold all take it, so two polls never overlap, a trigger during a running poll waits for it, and a hold is acknowledged only once no poll is running and none will start (reconMu does the same for reconcile ticks).
Lifecycle. The engine deletes any stale request at startup and writes a fresh released ack — its existence is the harness’s “seam is live on this bed” signal — so a hold never survives a restart (both bed launch sites, tests/gate/bed.go and tests/e2e/lifecycle.go, append pollctl.Env, so a restarted bed keeps the seam; enabled-but-released it is a normal free-running bed). A hold carries hold_until (the harness stamps now + pollctl.MaxHold, 10 min); past it the engine releases on its own, so a hard-killed harness cannot leave a bed held.
Who may use it. Holding polls starves the whole bed, so the harness helpers (HoldPolls/TriggerPoll/ReleasePolls, tests/e2e/poll_control.go over tests/e2e/pollhold) fail any test not marked exclusive in tests/e2e/registry/registry.json, and the registry check (plain go test ./...) fails any test that reaches them without being exclusive. HoldPolls registers the release with t.Cleanup. The only user is TestMergeTrainQueuedDeeperThanBatchCap; pipeline, convergence and gate tests stay free-running.
7.9 Webhook Wake Semantics: Burst Coalescence and Self-Feedback
Burst coalescence. wakeCh is a buffered channel with capacity 1. When multiple webhook events arrive in rapid succession, at most one wake is queued. The wakeChObserver uses a non-blocking send (select { case wakeCh <- struct{}{}: default: }), so additional fires while the channel is full are dropped. A burst of N simultaneous events produces at most 1 extra poll cycle. Test: TestHandleWebhookBurstCoalescence in engine/webhook_test.go.
Observer-based signaling (Phase 3-H). wakeCh is no longer signaled directly by the webhook handler. Instead, newWakeChObserver is registered on the shared store. The observer fires a non-blocking send whenever a Change includes any wakeChFlag (StatusChanged | LabelsChanged | CommentsChanged | LockChanged | LinkedPRChanged | AssigneesChanged | WorkerLifecycleChanged). The webhook handler calls deltaFn(eventType, body), which applies typed mutations to the cache store via boardcache delta functions; those Apply calls synchronously invoke all registered observers, including wakeChObserver. Changes that don’t affect dispatch eligibility (e.g., InvocationChanged, StageStateChanged, WorkerChanged from heartbeats/PID-sets) are filtered out and do not wake the poll loop. AssigneesChanged was added in issue #543; WorkerLifecycleChanged was added in Fix B (issue #544). WorkerLifecycleChanged is in wakeChFlags (for the wake channel) but excluded from cycleSetFlags (for mayNeedWorkObserver) to prevent early-return goroutine exits from bypassing the cooldown gate — see §9.2 and §9.9 (Fix B, issue #576).
Self-feedback loop (known gap). Fabrik runs as the human operator’s own GitHub account. Every API action Fabrik takes — label mutations, comment posts, status field updates, PR opens — generates webhook events from that account. These events arrive at the webhook handler and signal wakeCh (via the observer), triggering one extra poll cycle per burst of activity. The burst-coalescence guarantee bounds the damage: a stage advance producing N API actions generates at most 1 extra poll.
Sender-filter approach — considered and rejected. Suppressing wakeCh signals when sender.login matches cfg.User would eliminate these spurious wakes, but cfg.User is the human operator’s login. Filtering by it would also suppress every event the user generates (comments, label changes, PR reviews) — the most important input class. Sender filtering is only viable when Fabrik runs as a dedicated bot account separate from any human user. That is a future change; no sender filtering is currently implemented.
Backoff impact. Before the fix for issue #490, the case <-e.wakeCh: branch unconditionally reset e.idleStart and prevMultiplier = 1 before calling doPollCycle. Self-feedback events would therefore destroy the idle-backoff state on every stage advance, defeating the GraphQL budget savings webhooks provide. After the fix, those unconditional resets are removed. A webhook wake triggers an immediate poll, but backoff is preserved unless result.Active == true (see 7.7).
Wake-path rate-limit gate (#1716, ADR-1716). The idle-backoff interaction above assumes the wake reaches a poll cycle at all. It doesn’t, while rate-limit backoff is active: Run()’s case <-e.wakeCh: branch checks Engine.wakeBlockedByRateLimitBackoff() (§7.8) before calling doPollCycle(), and drops the wake entirely when it returns true — no doPollCycle() call, no ticker drain (an already-pending tick is left to fire on the next loop iteration rather than being silently discarded along with the dropped wake). This closes the production incident this section’s self-feedback loop made possible: under --webhooks, every self-feedback wake this section describes arrives while GraphQL or REST is rate-limited exactly when Fabrik’s own mutations (label changes, comments) are what triggered the exhaustion in the first place, and pre-#1716 each one walked straight through the backed-off ticker interval to trigger an immediate, immediately-rejected poll.
- Drop, not coalesce-with-re-arm. A wake dropped during backoff is not queued for redelivery once backoff clears — no
pendingWakeDuringBackoffflag exists. The already-backed-off ticker (reset toPollBackoffResult.NextIntervalby the priorPollWithBackoffcall) is what eventually re-polls; a board change that arrives via webhook mid-backoff and produces no further wake sits unprocessed for up to that interval, the same bound the ticker itself already had before this fix. The rejected alternative — an explicit re-arm flag that fires the instant backoff clears — was judged riskier: a dropped-with-no-re-arm bug in that design is silent, for a latency benefit already accepted as tolerable. - The legitimate recovery self-wake (§7.8’s rate-limit-adjustment bullet, the
e.wakeCh <- struct{}{}sent on the near-zero→recovered transition) is not blocked by this gate, and needs no special-casing to achieve that:PollWithBackoffruns synchronously to completion — including thebackoffRateLimitLowreassignment to its post-recovery value — beforedoPollCyclereturns, anddoPollCyclereturns beforeRun()’sselectcan consume the queued wake. So by the time the gate actually evaluates a wake sent this way,backoffRateLimitLowhas already settled tofalseand the gate passes it through. This is an ordering property of the current single-goroutine code (bothRun()’s select loop andPollWithBackoffexecute onRun()’s own goroutine), not an enforced invariant, and could be broken by a future change that makes either asynchronous — guarded byTestPollWithBackoff_RecoverySelfWake_NotBlockedByBackoffGate(engine/poll_test.go). - Acceptance/neutralization coverage.
TestRun_WakeDuringRateLimitBackoff_DropsWake(engine/engine_test.go) drives the realRun()goroutine with a sustained near-zero GraphQL budget and a burst ofwakeChsends, asserting the board-fetch count does not advance — confirmed (by hand, reverting the gate) to fail without it, exactly the “guards are deletable without failing a test” blind spot #1687 identified.TestPollWithBackoff_MinPollIntervalFloorcovers §7.8’s independent poll-rate floor.
7.10 Stall Detection and Corrective Re-Invocation
A stalled attempt — one where the worker backgrounded a long-running command (a dev server, build, or test run) and then idled waiting for a completion notification that never arrives in a headless environment — is distinguishable from a genuinely failed one by its turn-usage trend across retries within §7.1’s cooldown loop: a stage that did real work and failed burns turns roughly consistently across attempts; a stalled one shows a clean incomplete attempt followed by a retry that completes fewer turns, still without completing. A genuinely-progressing retry does not shrink like that. This holds whether the first attempt burned its full turn budget before giving up, or — the cleanest and most common real-world shape — the worker correctly recognized it was waiting on a backgrounded command and ended its turn early, deliberately, well under budget: the latter is textually the #1142 production incident (three clean, uncapped, declining Validate attempts — 45/100, 20/100, 7/100 — none of which armed the hint before the original turn-capped-predecessor precondition was widened). See issue #1146 and ADR-1146 for the original incident (#816) that motivated this, and #1767/ADR-1146’s amended Consequences for the #1142 gap and the widened signature below.
Detection (detectAndArmStallHint, engine/item.go). Runs inside §7.1’s incomplete-outcome branch, after StageRetryIncremented and the count >= MaxRetries escalation decision (§7.2), whenever claudeRan is true and this attempt is not the one triggering escalation (see “Escalating attempt” below) — unconditionally, regardless of whether this attempt stopped cleanly or errored out. A clean flag (err == nil, or err classifies as *claudeTurnLimitError per §7.2’s turnLimited) is passed in and gates only the second half (arming), not the first (recording) — see step 4:
- Reads the previous attempt’s
TurnsUsed/capped/clean status for this stage from the store (LastTurnsUsed,LastTurnsCapped,LastTurnsClean— zero-valued if this is the first attempt), and whether a hint has already armed this episode (StallEpisodeArmed). - Computes this attempt’s own capped status:
clean && usage.MaxTurns > 0 && usage.TurnsUsed >= usage.MaxTurns. Forcingcappedtofalsewhenever!cleanmatters independently of arming (step 4) — see the recording note below. Capped-ness is evaluated against this attempt’s ownMaxTurns, so it is unaffected byfabrik:extend-turnswidening the budget between attempts.LastTurnsCappedis still recorded and read (step 4), but only to phrase the stall comment correctly — it dropped out of the arming precondition itself in #1767. - Records this attempt’s
TurnsUsed/capped/clean status viaitemstate.StageTurnUsageRecorded, overwriting the previous values, on every call — clean or not. Recording a non-clean attempt withClean=false(regardless of its actual turn count) keeps it from going invisible to the trend: it immediately invalidates the “previous clean” precondition (step 4) for the next comparison, so a generic error sitting between two clean attempts breaks the chain at the point of the error rather than being skipped over. An earlier version of this fix gated the whole function call (recording included) behindclean, which leftLastTurnsUsed/LastTurnsCappeddescribing the last clean attempt across an intervening error — a later clean attempt would then compare against that non-consecutive predecessor and could still misfire. See ADR-1146’s Rationale for the incident this reproduced and the regression test that guards it. - If
cleanis true, the previous attempt was itself clean (LastTurnsClean— capped or not; #1767 dropped the “previous attempt was turn-capped” requirement, since it made the cleanest form of the stall — a worker that deliberately ends its turn early, well under budget, having recognized it’s waiting on a background job — invisible to the detector, exactly#1142’s shape), this attempt is not itself turn-capped, this attempt used more than zero turns but strictly fewer than the previous attempt, and no hint has already armed this episode (!StallEpisodeArmed), the pattern is a detected stall: appliesitemstate.StallHintArmed(which also setsStallEpisodeArmed) for the stage and posts a one-time informational comment citing both turn counts, phrased according to whether the predecessor was actually turn-capped (LastTurnsCapped, from step 2) or stopped short on its own — never asserting a turn-limit hit that didn’t happen. The “not itself capped” precondition guards against a false positive from the progress-based turn-extension loop (runInvocationWithExtension, see “Turn Limit Extension” above): that loop can widen the effective budget on one dispatch (e.g. to 2×/3×stage.MaxTurns) without the widening persisting to the next, separate dispatch, so two consecutive turn-capped attempts could otherwise show a smaller absoluteTurnsUsedon the second purely because its own budget reset lower, not because it is a declining, stalled retry. Thecleanprecondition guards against a different false positive:claudeRanis true for most invocation errors (network failure, git-push error, malformed CLI output), not only clean incomplete stops, so without it a generic error whose partialusage.TurnsUsedhappened to be smaller than a prior clean attempt’s would be misread as a declining, stalled retry and arm a specific, confident — but wrong — diagnosis.err == nilalone is not sufficient to computeclean, since a turn-cap exit is itself reported as a non-nil*claudeTurnLimitError(ADR-1178);turnLimitedis reused from the classification already computed earlier infinalizeStageOutcomefor theInvocationRecordedwrite, so both consult the same definition of “clean.” See ADR-1146 (as amended by #1767) for the fuller rationale. - One-shot per episode via an explicit guard, not an emergent property. Before #1767, the attempt that triggered arming was (by construction, per the then-precondition) not itself capped, so step 3’s overwrite cleared the “previous capped” precondition for arming again immediately — no separate one-shot guard was needed. Widening the precondition to “previous clean” removed that emergent property: the arming attempt is clean too (arming requires
cleanfor the current attempt), so it no longer clears its own precondition by being uncapped.StallEpisodeArmedis the explicit replacement: set byStallHintArmedthe first time a hint arms in a retry episode, checked before arming again, and cleared only byStageRetryCleared(see “Cross-stage isolation and cleanup” below) — so a continued multi-attempt decline (e.g.#1142’s real 45→20→7 sequence) still arms exactly once, not on every subsequent comparison.
A detected stall still applies StageRetryIncremented and counts against MaxRetries exactly like any other non-completion — unlike §7.3’s usage-limit exemption, the stage genuinely ran and spent real tokens; the fix targets spending the existing retry budget more effectively, not exempting stalls from it. Detection itself runs whenever claudeRan is true, independent of MaxRetries: max_retries: 0 (“unlimited retries”, §7.2) is a first-class configuration, not an edge case, and is exactly the setting where a stalled stage would otherwise grind identical retries forever with no mitigation — only the surrounding retry-count/escalation bookkeeping is gated on MaxRetries > 0.
Escalating attempt: arming is skipped. detectAndArmStallHint’s comment tells the operator “the next invocation will receive a corrective hint.” If the capped-then-declining pattern is detected on the same attempt whose incremented count also reaches MaxRetries, §7.2’s escalateFailedStage() pauses the issue immediately afterward, and a human’s later clearFailedStage() applies StageRetryCleared — which wipes StallHintPending before any further invocation happens. The promised hint would never be delivered, and the comment would be actively misleading about what already occurred. finalizeStageOutcome therefore computes the escalation decision (count >= MaxRetries) first and skips the detectAndArmStallHint call entirely when this attempt is the one being escalated; no stall comment is posted and nothing is armed in that case.
Injection (consumeStallHint, engine/item.go; buildPrompt, engine/claude.go). When building InvokeOptions for the next invocation of a stage (runInvocationWithExtension), the engine checks StallHintPending for that stage. If armed, it applies itemstate.StallHintConsumed (clearing the flag) and sets InvokeOptions.CorrectiveHint to a fixed, stage-agnostic, hedged callout describing the suspected cause and suggesting the worker run any long-running command in the foreground with an explicit timeout instead of backgrounding it — the same guidance already deployed worker-side per #1077. buildPrompt prepends this text as a callout when non-empty; it is otherwise absent from the prompt. Because retries --resume the same underlying Claude session (§1’s resume semantics; see also the “Retry after a turn-cap kill” section of docs/stage-lifecycle.md for the retry-trustworthiness fix, #1081, this must not regress), the hint reads as a new turn in a conversation that already has full context of its own prior attempt — it does not need to re-explain the task, only redirect the strategy.
runInvocationWithExtension checks §7.3’s account-wide usage-limit hold gate before building InvokeOptions (and therefore before calling consumeStallHint). This ordering matters: consumeStallHint destructively clears StallHintPending the moment it runs, so if the check happened after, a dispatch that lands while usage limited would consume an armed hint without ever reaching Claude — silently discarding the one-shot corrective re-invocation for that stall episode. Gating first means a usage limited dispatch leaves the hint pending for the next dispatch that actually reaches Claude.
Cross-stage isolation and cleanup. StageRetryCleared (applied by clearFailedStage() on unpause, and by a stage’s completion) deletes LastTurnsUsed, LastTurnsCapped, LastTurnsClean, StallHintPending, and StallEpisodeArmed for that stage alongside the existing Attempts reset — so a later, unrelated incomplete run of the same stage (e.g. after fabrik:revalidate) never inherits a stale armed hint (or a stale one-shot guard) from a long-past episode.
State: In-memory only (StageState.LastTurnsUsed/LastTurnsCapped/LastTurnsClean/StallHintPending/StallEpisodeArmed, all map[string]bool/map[string]int keyed by stage name), mirroring every other per-stage StageState field — same restart-gap tradeoff already accepted by ADR-030’s turn-extension baseline.
Scope note: this implements only the “cheaper to detect” trend signal from issue #1146 — a clean incomplete attempt (capped or not, #1767) followed by a declining, incomplete one. It does not add live NDJSON tool-use parsing to detect backgrounding-then-silence within a single invocation (the issue’s other suggested signal); that remains a possible follow-up if the trend signal proves insufficient in practice.
7.11 Stale Warning Sweep
.fabrik/warnings.json (warnings/warnings.go, ADR-052) backs the TUI’s Warnings panel. Several
engine-side detectors record an entry when a condition is true and clear it via warnings.Clear when
the same re-checked subject’s condition resolves — but a subject that stops being checked at all (a
repo leaves the board, a stage is renamed or its YAML deleted) never revisits the Clear branch, so its
entry is immortal. This is the same durable-state-leak shape as the orphaned stage:*:in_progress
labels (#1135) and the fabrik:claude-limit account-wide label before §7.3’s sweep (ADR-1183). Observed
in production on the verveguy project (#1348): four allow_auto_merge warnings for repos that had
left the board persisted indefinitely, pushing live warnings past the panel’s row cap.
warnings.ClearMissing(warningType string, present map[string]bool) ([]string, error) is the shared
bulk-predicate primitive both sweeps below use. warnings.Clear(key) always calls save() even when
key was never present — a caller-side loop over N possibly-stale keys would therefore write the file
on every poll unless every caller independently pre-filters correctly. ClearMissing removes that
footgun structurally: it loads the file once, removes every entry whose Type == warningType and whose
Key subject (the portion after "<warningType>:") is absent from present, and only calls save()
when at least one entry was actually removed. Entries of other Types are never touched regardless of
Key, and a present-subject entry is preserved whether or not it is Dismissed — the sweep is about
absent subjects, not about un-dismissing anything still around. Returns the full Key of each cleared
entry for the caller to log.
allow_auto_merge (repo-keyed, per-poll). sweepStaleAllowAutoMergeWarnings
(engine/allow_auto_merge_settle.go) is called from poll() immediately after seenRepos — the board
repo set poll() already computes for the label-seeding path (poll.go ~line 925) — is built, before
the label-seeding loop runs. It calls warnings.ClearMissing("allow_auto_merge", present) with present
= seenRepos unioned with e.defaultRepo() (single-repo mode only). The union matters:
checkAllowAutoMerge(e.cfg.Owner, e.cfg.Repo) fires unconditionally at Run() startup regardless of
whether the board currently has any open items for that repo, but seenRepos is built purely from
board.Items — without the union, a transient zero-open-items poll for the operator’s own configured
repo (e.g. everything currently in Done) would durably clear a legitimate warning for it, and because
checkedAutoMergeRepos never re-fires for that repo after the first startup call, the warning would not
reappear until a process restart. Multi-repo mode (e.cfg.Repo == "") has no equivalent always-present
repo, so defaultRepo() returning "" is a correct no-op there. Runs every poll, because seenRepos
itself is rebuilt every poll (a repo leaving/rejoining the board is visible immediately). No new API
calls: everything consumed is already computed this poll cycle. Each clear logs one line (tag "poll",
matching the cadence of the other per-poll log lines it sits next to — not "startup", which the
codebase reserves for once-per-process/once-per-repo events) naming the full key and the reason (“repo
no longer on the board”).
stage_drift / undeclared_reviewers (stage-name-keyed, startup-only). stages.SweepStaleWarnings
(stages/drift.go) is called once from Run(), immediately after the existing WarnStageDrift/
WarnUndeclaredReviewers calls (both the e.logFile != nil and the plain-stderr branch), with the same
unfiltered e.cfg.Stages those two functions themselves consume — using the Unmanaged-excluding
stageNames list built later in poll.go for SeedLabels would wrongly sweep a still-valid Unmanaged
stage’s warning. It builds the current stage-name set and calls warnings.ClearMissing twice, once per
Type. Startup-only (not per-poll) is correct here, unlike the repo sweep: e.cfg.Stages is fixed for
the life of the process and only changes across a restart (including the in-place SIGHUP restart, which
re-execs and re-runs Run() from scratch), so re-running the sweep every poll would be correct but
pointless. Each clear logs one line naming the full key and the reason (“stage no longer configured”),
written to the same io.Writer (os.Stderr, tee’d to fabrik.log when open) as the sibling drift/
undeclared-reviewers warnings.
version_skew: no sweep needed, by construction. checkVersionSkew’s (engine/upgrade.go) key
subject is the resolved on-disk executable path, re-derived fresh via filepath.EvalSymlinks on every
idle-upgrade check (poll.go), not looked up against a shrinking discovered set the way a board repo or
a configured stage name is. Its Clear branch (diskVersion == running) is therefore reachable on every
single evaluation, so the warning can never outlive the condition that produced it — see the doc comment
on checkVersionSkew for the same reasoning in code.
source_staleness: no sweep needed, by construction — same reasoning as version_skew.
checkSourceStaleness (engine/staleness.go, #1464) uses a single fixed key
("source_staleness") rather than a per-subject key, because there is exactly one source
checkout per Fabrik process — unlike allow_auto_merge/stage_drift (one entry per board
repo/configured stage, a set that shrinks as repos leave the board or stages are removed).
Every throttled evaluation (§9.2 below) re-derives the comparison from scratch via
selfupgrade.CompareDevBuild and either records or clears the same key, so its Clear branch
is reachable on every single evaluation the same way checkVersionSkew’s is — the warning
cannot outlive the condition that produced it, and there is no “subject went away” case to sweep
for.
What belongs in the panel (#2135, ADR-2135, FR-001). An operator-level condition: a configuration, environment or engine fault that blocks or degrades work and needs a person to act. Per-item states (paused, awaiting input, escalated), transient runtime states (poll errors and rate limits — the header’s backoff status shows them) and the output of interactive one-shot subcommands do not belong. Borderline lines are dispositioned individually, with a reason, in the audit table below.
The shared path (FR-005). warnings/report.go:
| Call | Use |
|---|---|
warnings.Report(w, Condition{Type, Subject, Title, Detail, FixAction?, FixParams?, Fingerprint?, LogLine?, Level?}) |
log line to w (nil = none) and record the entry, in one call. Safe every poll. |
warnings.Resolve(key) |
clear one entry when its re-checked condition no longer holds. |
warnings.Reconcile(w, type, []Condition) |
report a complete set, then ClearMissing(type, present). Only when the check ran to completion: an empty set clears the whole type, so a check that errored or returned early must not call it. |
Rules: a new operator-level warning is a Report/Reconcile call, never a bare log line followed (or not)
by Record; the writer is a parameter so the log-file-open and log-file-failed branches of Run() share
one startupW; FixAction == "" is valid (the panel shows title and detail; the TUI fix key is then a
no-op); keys are <type>:<subject> (the ClearMissing shape — Condition.KeyOverride exists only for the
legacy source_staleness key). warnings/scan_test.go fails on any [startup] warn…, [warn],
[upgrade] warning or WARNING: literal in engine/, cmd/, config/, stages/ that is not on its
audited exception list (each entry carries a reason). How a sibling adds a condition (skill resolution,
plugin loading, bare-name deprecation, fabrik upgrade migration — #2137/#2139/#2140): build a
warnings.Condition with a stable Type, a Subject, a Title, a Detail ending in Fix: …, and call
Report where the condition is detected and Resolve/Reconcile where the check passes (including every
early-return branch that means “checked, nothing wrong”); add one test that records, repeats (one entry) and
clears.
Dismissal and re-appearance. Entry.Fingerprint is the stable identity of the condition. A dismissed
entry stays dismissed while its fingerprint is unchanged and reappears when it changes; an empty new
fingerprint keeps the stored one (so the TUI’s [Fix failed] re-record never un-dismisses). Report
defaults it to Detail. Volatile recorders pin an explicit one: source_staleness constant (commit
counts), version_skew the version pair (PID), stage_drift the sorted missing-key set,
undeclared_reviewers constant. Record skips the write when nothing material changed and LastSeen is
under 10 minutes old (SC-003); Clear of an absent key does not write; the temp file is unique per write.
Audit (FR-004 / SC-002). Every operator-level warning in the daemon’s source, with its disposition:
| Source | Disposition | Type / clear rule |
|---|---|---|
source staleness, repo access, allow_auto_merge, version skew, stage drift, undeclared reviewers |
Panel (migrated onto the shared path) | unchanged keys; Resolve/ClearMissing as before |
unrecognised .fabrik/config.yaml key |
Panel (supersedes ADR-1544’s stderr-only scope) | config_key:<key>; Reconcile each start |
~60 [warn] … invalid; using default lines (cmd/root.go) |
Panel, one aggregated entry; per-setting log lines kept | config_value:settings; Reconcile each start |
fine-grained PAT (github_pat_) |
Panel | config_token:fine_grained_pat; cleared when a classic token is used |
.fabrik/config.yaml gitignored |
Panel | config_ignored:config.yaml; Reconcile each start |
| plugin state check failed / skill refresh failed / customised plugin skips auto-refresh | Panel (also still in the header) | plugin_state:{check_failed,refresh_failed,customized,migration_pending} (migration_pending = fabrik upgrade has a workflow migration to apply, §7.19); Reconcile each start, except refresh_failed is swept only on a start that attempted the refresh (plain-text mode with stale skills, or nothing stale) — a TUI-mode or customized start leaves an earlier failure in place |
| no git credential helper (HTTPS) | Panel | git_credentials:https; Resolve when a helper exists or SSH/App auth applies |
Claude settings file unreadable/unparseable (apiKeyHelper refusal may be bypassed) |
Panel | api_key_helper_check:settings; Reconcile each start |
FABRIK_WORKER_ENV_PASSTHROUGH entries ignored; Hookdeck env names not scrubbed |
Panel | worker_env:{ignored,hookdeck_dropped}; Reconcile each start |
CIBackstopTimeout / DrainDeadline ordering |
Panel | timeout_ordering:{ci_backstop,drain_deadline}; Resolve when valid |
| log rotation failed / log file unopenable | Panel | log_file:{rotation,open}; Reconcile each start |
| App installation does not cover N managed repos | Panel (aggregated) | hookdeck_coverage:repos; Reconcile each check |
| repos with no forwarding hook (poll-only) | Panel (aggregated) | webhook_coverage:repos; Reconcile each check |
board / status-field fetch failed at startup; GHES /meta fetch or parse failed |
Log only: transient, retried at the next start; the floor check is advisory | – |
board columns with no matching stage; holding stage without a column while merge_train is off |
Log only: neither blocks nor degrades work | – |
| board drift on a terminal item | Log only: per-item (FR-001) | – |
| could not determine repo access / list the installation’s repos | Log only: fail-open guess; the confirmed result is the repo_access entry |
– |
| could not resolve the bot user ID | Log only: cosmetic, fail-soft | – |
display-field (Fabrik, Last activity, Last run) write failed |
Log only: per-item, retried at the next transition | – |
| display field absent from the board / of the wrong type / lookup failed (#2218) | Panel (display_field:missing, display_field:mismatch:<label>, display_field_lookup:<label>) — see §7.17 |
fabrik repair-board --apply for missing (none on a user-owned board or for a mismatch) |
| local API fails to start (#2223) | Panel | local_api:unavailable; names the socket path and the error and says fabrik mcp/PM plugin tools will report the daemon as not running; no FixAction (the cause is the path’s directory, its permissions or a non-socket file, which is never replaced); fingerprint pinned to the path; Resolve on every successful start, so a restart, SIGHUP re-exec or self-upgrade clears it; the existing local API unavailable log line is kept |
| webhook runtime failures (subprocess crash, 422s, unsupported event, orphan cleanup) | Log only: runtime health is the header’s webhook ingestion status | – |
| hookdeck coverage read failed | Log only: transient; the prior entry is deliberately left | – |
webhook hook lookup errored for a repo (hook-coverage check failed) |
Log only: transient; the repo is unknown, not missing, so it stays out of webhook_coverage:repos and, when nothing else is confirmed missing, the prior entry is deliberately left |
– |
| the check for unrecognised keys itself failed | Log only: nothing is known, so reconciling would wrongly clear entries | – |
| re-exec path plugin messages | Log only: Execute re-evaluates and reports plugin_state right after |
– |
| the panel store itself failing (record/clear/sweep errors) | Log only | – |
fabrik init / upgrade / watch / repair-board output |
Out of scope: interactive one-shot subcommands | – |
per-item logf(item, "warn", …) lines; poll errors and rate limits |
Out of scope: per-item or transient runtime (FR-001) | – |
Non-goals. No TUI layout change (the panel stays flat — no severity), and this is not a general warning-expiry/TTL mechanism — only subjects that have provably gone away (absent from a known-good set already computed elsewhere) are ever cleared. See ADR-1348, ADR-2135.
7.12 Slice Budget / Turn-Cap Preemption Limit
max_retries (§7.2) is the failure counter: genuine errors, degenerate output, a completion
marker with no harvestable artifact (#1782), PR-creation failures, and a clean run that never emits
FABRIK_STAGE_COMPLETE. It was previously overloaded to
also bound turn-cap preemptions — a large job resuming across several slices looked identical to a
stage that kept genuinely failing, so a job needing more slices than max_retries was paused with
stage:<X>:failed while progressing normally (#816, #1114, #1183; reported independently as the
undocumented max_retries × max_turns ceiling in #1191). §3.1’s Turn Limit Extension subsection
describes what a turn cap means — a resumable preemption, never a failure, per ADR-1178. This
section describes how the retry accounting treats it (#1199).
The two counters:
| Counter | Field | Config | Default | Counts |
|---|---|---|---|---|
| Failure counter | StageState.Attempts |
MaxRetries / --max-retries / FABRIK_MAX_RETRIES |
3 (0 = unlimited) | Genuine errors, degenerate output, no harvestable artifact (#1782), PR-creation failures, clean run with no completion marker |
| Slice counter | StageState.SliceRetries |
MaxSliceRetries / --max-slice-retries / FABRIK_MAX_SLICE_RETRIES |
10 | Turn-cap preemptions only (turnLimited == true, CLI subtype: "error_max_turns") |
The slice counter’s default is intentionally higher than the failure counter’s: a job legitimately
needing several slices is routine, while a stage that keeps genuinely failing is not — the two want
different bounds, which was exactly #1191’s complaint about the single overloaded counter. MaxRetries
== 0 still means “unlimited failures”; there is no equivalent “unlimited slices” setting for
MaxSliceRetries — it is resolved to its default (10) whenever configured as 0 or unset, the same
tier MaxRebaseCycles/MaxEnqueueCycles use (flag + env only, no config.yaml field).
Detection: finalizeStageOutcome() (engine/item.go) already computes turnLimited :=
errors.As(err, &turnLimitErr) for the InvocationRecorded write (§3.1/ADR-1178) — the same boolean is
reused here, so no new detection work exists. This is a structural signal (the CLI’s own
subtype/terminal_reason, captured since #1178), not an inference: distinguishing a preemption from a
failure by a heuristic over output text would risk misclassifying either direction.
Handling: Inside the existing non-completed/non-blocked else branch — after StageAttempted,
commitWIP, the branch push, and markCommentsSeenByStage have already run unconditionally on
claudeRan exactly as before (ADR-1178 requires this machinery keep running for a turn-cap exit; unlike
§7.3’s usage-limit exemption, which is a stage-never-ran early return, a turn-capped invocation did
run and made progress) — the routing is a single branch on turnLimited:
turnLimited == true: appliesSliceRetryIncremented(neverStageRetryIncremented). If the resultingSliceRetries(stageName) >= MaxSliceRetries, callspauseForSliceLimit()instead ofescalateFailedStage()— modeled directly on the merge-gate’spauseForRebaseCycleLimit(§5, a live in-production precedent for a second, independently-bounded, non-failure counter): posts a comment that explicitly states this is not a failure, names--max-slice-retries/FABRIK_MAX_SLICE_RETRIESas the override, and suggestsfabrik:extend-turns(wider per-invocation turn budget → fewer slices needed) or splitting the issue. Appliesfabrik:pausedfabrik:awaiting-input— neverstage:<X>:failed. The degenerate-output first-detection comment (§7.2) does not apply on this branch; degenerate output is a marker-content condition orthogonal toturnLimited, and in practice a turn-capped exit’s stripped output is never checked for it.
turnLimited == false: unchanged from before this issue — appliesStageRetryIncremented, compares toMaxRetries, escalates viaescalateFailedStage()exactly as §7.2 describes. This covers both a genuine error and a clean run with no completion marker.
The no-FABRIK_STAGE_COMPLETE decision (deferred by the issue to Plan): a clean run
(err == nil) that never emits FABRIK_STAGE_COMPLETE counts against the failure counter, not the
slice counter. turnLimited is the only structurally verified signal available; a clean run with no
marker has no equivalent structural signal distinguishing it from “stopped for an unknown reason,” so it
is treated the same as a genuine error rather than inferred to be a slice. This preserves today’s
(already narrow) protection against a silently-non-terminating stage.
Recovery: StageRetryCleared — applied on normal stage completion and by clearFailedStage() on
manual unpause — zeroes SliceRetries(stageName) alongside Attempts(stageName); the two counters
share one reset point. pauseForSliceLimit() applies itemstate.EnginePaused (never
stage:<X>:failed), and — unlike the four cycle-limit sites #1460 fixes (§7.2.1) — resumes via
processItem()’s own gate directly, not via handleEngineUnpause: a slice-limit pause fires
mid-stage, before stage:<X>:complete is ever applied, so the item is not yet stage-complete when
paused and processItem()’s gate remains reachable. Without EnginePaused, processItem()’s unpause
guard (wasPaused || hasFailedLabel, both false for a pure slice-limit pause) would never fire
clearFailedStage(), SliceRetries would never reset, and removing fabrik:paused would be a no-op:
the very next dispatch takes exactly one more slice, re-checks a counter already at MaxSliceRetries,
and re-pauses immediately — the documented recovery path would not work. With EnginePaused applied,
snap.PausedByEngine(stageName) is true on the next pass, so removing fabrik:paused genuinely
triggers clearFailedStage() → StageRetryCleared, and the stage resumes with a fresh slice budget.
clearFailedStage()’s stage:<X>:failed label removal is a harmless no-op on this path, since that
label was never applied. (This section previously contrasted with pauseForRebaseCycleLimit, which at
the time did not apply EnginePaused either — #1460 fixed that site too, via the separate
handleEngineUnpause resume path described in §7.2.1, since a rebase-cycle-limit pause fires
post-completion, unlike this section’s mid-stage slice-limit pause.)
Regression guard: a job that has already taken more turn-cap slices than MaxRetries (the old,
accidental ceiling per #1191) is never paused as long as it stays within MaxSliceRetries — Attempts
is untouched by any number of turn-cap preemptions.
7.13 Overseer Attention Classification and the Local Read API (#1967)
ADR-1966-a. The running engine serves a read-only, zero-GitHub-cost view of its in-memory state on a per-daemon Unix socket; fabrik mcp proxies three tools to it (see USER_GUIDE §9, “Overseer Tools: fabrik mcp”). This section is the as-built definition of the attention classification those tools report, in internal/attention (pure — no engine, store or GitHub dependency — so the push-event work reuses it). It is a product definition, pinned by internal/attention/attention_test.go.
States. needs-human (the engine is paused and only a human reply resumes it), suspended (nothing is wrong and nothing will move until a human decides — a completed stage that does not auto-advance, or work blocked only by a parked dependency; #2120, below), escalated (the engine stopped on its own account), waiting (the engine is correctly waiting on something it owns), working (a worker holds a slot and progress is recent), stalled (nothing is correctly waited on, and no progress for longer than the threshold), idle.
Precedence — first match wins:
| # | Condition | State / code |
|---|---|---|
| 1 | Closed; or Terminal; or the cleanup stage; or an Unmanaged column; or a column with no configured stage |
idle (closed, done, unmanaged-column, no-stage) |
| 2 | fabrik:landing-verification-failed, or fabrik:paused with an escalation signal: StageState.PausedByEngine for the item’s stage, any cycle counter at its known positive limit (limit:<counter>), or fabrik:awaiting-runaway-alert |
escalated |
| 3 | fabrik:paused without an escalation signal |
needs-human (fabrik:awaiting-input when that label is present, else fabrik:paused) |
| 4 | Status == Validate, stage:Validate:complete, fabrik:cruise, an open unmerged linked PR, no worker |
needs-human, code awaiting-merge-decision (cruise never merges; ranked last in the attention view) |
| 4b | fabrik:api-key-helper-detected (the engine skips the item until a human removes apiKeyHelper from the worktree settings; no retry or deadline clears it) |
needs-human (fabrik:api-key-helper-detected, ranked with the paused items) |
| 4c | Suspended (#2120): a completed stage that does not auto-advance, or an item blocked only by work parked in an unmanaged column — see “Suspended” below | suspended (decision-pending, parked-blocker), ranked with needs-human; never stalled |
| 5 | A worker, or a merge-train conflict resolution, is queued or running (#2121) — above every wait label and cooldown (R4) | holding no slot yet: waiting, queued-for-slot, cause slot-starved, unranked; holding a slot: working, worker-in-flight, naming the kind, its trigger and (known) turns; stalled (worker-no-progress) only when progress age exceeds both the threshold and the stage’s wall-clock budget (max_wall_time, or the 15-minute Claude inactivity timeout when unset) plus a 5-minute margin — a healthy 30–45 minute run is working (its summary is flagged long-running past the threshold). While a worker is present no cooldown:* appears as a reason or a deadline |
| 6 | Any state-bearing waiting label (fabrik:awaiting-* except awaiting-input, fabrik:blocked, fabrik:rebase-needed, fabrik:bot-reprompted, fabrik:claude-limit, fabrik:tools-denied) |
waiting — unless a known deadline for it passed more than the stall threshold ago, then stalled (overdue); or, when it has no known deadline and no progress for longer than the stall threshold, stalled (no-progress) — except the open-ended waits fabrik:blocked, fabrik:claude-limit |
| 6a | No worker, no wait label, and a retry the engine scheduled after a killed or incomplete invocation (#2121) | waiting, retry-pending / retry-waiting-for-slot, cause wall-time-kill or max-turns when that is the kill; stalled (overdue) only when the retry was due more than the threshold ago and a slot is free |
| 6b | No worker, no wait label, and a gate holding the item’s advance (#2121) | waiting, gate-hold, cause gate-hold:<gate> |
| 7 | Waiting for dispatch (#2121): the item has work the engine will dispatch and no worker | waiting, waiting-for-dispatch, unranked; never idle — see below |
| 8 | An unexpired cooldown, or the holding stage | waiting (same deadline/stall handling as row 6; an unexpired cooldown and the holding stage are open-ended) |
| 9 | Otherwise | stalled (no-progress, or stage-complete-not-advanced when the stage’s complete label is present) past the threshold; else idle |
fabrik:toolchain-stale, fabrik:children-spawned and fabrik:nondefault-base-pr-noted are listed as reasons but never change the state.
Poll-stall attribution (post-pass, #2124, ADR 2124). Classify ends with attributePollStall, applied only when Input.PollStall.Active (the daemon’s poll loop is overdue, below). It never changes which row above matched; it changes what an engine-owned verdict says. A stalled result with code overdue, no-progress or stage-complete-not-advanced (rows 6, 6a and 9) becomes waiting, code waiting-for-poll, unranked, with summary and next_action waiting for the poll: blocked for N s, in step <step> (the step is omitted when unknown). A waiting result, or idle with code idle, keeps its state and code and has the same text prepended. Rows 1-4b (closed, escalated, needs-human, api-key-helper) and row 5 (a worker is running, including worker-no-progress, and a worker queued for a slot, queued-for-slot) are untouched: a poll does not unblock the first group, and a queued worker’s goroutine is already blocked on the slot semaphore, so blaming the poll for its slot wait would be wrong (#2121). With PollStall zero every classification is byte-identical to before. pauseFamilyKey maps only stalled to the stalled event, so an item whose only obstacle is the poll produces no item-level push: the cause is carried once, by the account-wide pipeline-stalled event (§7.14).
Cooldown exemption and item-wedged (#2170, ADR 2170). An active cooldown classifies as waiting, so a gate hold that re-stamps its cooldown on every poll (feedback-pending, review-blocked, merge-unsettled) never becomes stalled and produces no stalled push however long it lasts. Since #2121 a fresh recorded gate hold is additionally explained as gate-hold (row 6b) and is likewise never stalled; the cooldown handling itself is unchanged, and item-wedged (§7.14) covers that gap with its own detector, which does not consult cooldowns: a gate hold the engine keeps confirming, with no worker dispatched or queued since it began, is reported once it outlives its threshold.
Recovery grace. Input.PollRecoveredAt (the end of the last overdue episode) floors a known progress anchor, so the items the stall held back get a fresh stall window instead of flipping to stalled (and pushing) the moment the poll returns. An unknown anchor stays unknown.
Attention-view rank (lower is more urgent; items with no rank are omitted): needs-human and suspended 0 (a pause that needs an answer sorts ahead of a decision that waits), escalated 1, stalled 2, settled-at-Validate awaiting-merge-decision 3, an unaddressed red CI on the linked PR 4 (RankCIRed, #2122).
Red CI overlay (post-pass, #2122, ADR 2122). Classify ends with overlayCI, after attributePollStall. It is additional information about an item the table above has already placed, not a table row: it never changes State or Code, so every row’s precedence is unchanged. Input.CI (attention.CIFact, built by attentionCIFact from the Store alone) is non-nil only for a verdict whose HeadSHA equals the PR’s current head and whose PR is open and not merged. The overlay applies only when the verdict is red and not stale (attentionCIFact measures staleness against one poll interval times Engine.prFactsRotation, the number of polls settlePRFacts needs to rotate through all candidates at its per-poll check budget, so a red is not dropped between re-reads when more PRs can spend the budget than it covers (merged/closed PRs and greens read live within the recheck window do not count); the status tool’s own ci_stale still uses one interval), the item is in a managed stage and open, it does not carry fabrik:awaiting-ci (a CI-gated wait is untouched), and the table gave waiting, working, idle or stalled — needs-human, escalated and suspended keep their classification. It then (1) adds the reason ci:red, (2) prefixes the summary with CI is red on <failing checks>; ((a required status context failed) when no check run failed), (3) sets Result.CIFailing, and (4) either — with a worker or a merge-train conflict resolution — appends a CI-fix reinvoke is addressing it (kind ci-fix-reinvoke) or <kind> is running (every other kind, which is not claimed to address CI) and leaves the rank alone, or — with none, and the PR not a draft — sets the rank to RankCIRed (4) when the result had none or a lower urgency. A draft PR gets the text marked (draft PR) and no rank (Implement opens drafts early, so red there is routine). Green, pending, none, unknown and (beyond one rotation) stale verdicts add nothing. Reporting a red verdict does not make the engine dispatch anything.
Progress anchor. The latest of StatusEnteredAt, EligibilityChangedAt, every stage’s LastAttemptAt, every LabelAppliedAt entry, LinkedPR.LastCIProgressAt and Worker.StartedAt. EligibilityChangedAt (#2120) is stamped by the store whenever a before/after label diff of an item it already held touches fabrik:cruise, fabrik:yolo, fabrik:blocked, fabrik:paused or a stage:*:complete label (a population or baseline never stamps; unknown after a restart), so an autonomy change, a resolved decision or a cleared blocker restarts the stall clock: stalled measures time since the item could last have moved on its own, never time spent waiting for a human. It deliberately excludes Worker.LastSignAt (a ticker heartbeat from the engine’s own goroutine, fresh even while Claude is hung) and UpdatedAt (bumped by Fabrik’s own writes). An anchor of zero is unknown and never stalls an item. Stall threshold: --stall-threshold / FABRIK_STALL_THRESHOLD (minutes, default 30); fabrik_board accepts a per-call override; the effective value is echoed in every response. Zero disables stalling.
Restart caveat. StatusEnteredAt is reset to now on every store reset (ADR-1833), so right after a restart every item’s anchor is “just now”: nothing classifies stalled until one threshold has elapsed since the daemon started, and “time in column” means “since this daemon first saw it” (responses carry status_entered_basis and daemon uptime). This is a deliberate fail-safe for the stall decision. It no longer applies to a pending decision (#2120): that is anchored on a GitHub time, listed from the first served board, and never stalled (see “Suspended”).
Next action and deadlines. The result lists every deadline the engine will act on: unexpired cooldowns (cooldown:<reason>; only an entry still in the future is a deadline; an expired one is either history or an expiry poll admission has not yet acted on — the engine deletes it once it has, #2096 — and is neither a deadline nor a reason); the CI liveness stall (LinkedPR.LastCIProgressAt + CIWaitTimeout); the CI backstop (LabelAppliedAt[awaiting-ci] + CIBackstopTimeout); the review wait (LabelAppliedAt[awaiting-review] + ReviewWaitTimeout); the bot re-prompt escalation (LabelAppliedAt[bot-reprompted] + ReviewWaitTimeout); the end of the account-wide Claude usage-limit hold. next.at is the earliest known deadline. A deadline anchored on a value the cache does not hold — LabelAppliedAt is record-on-write and empty after a restart, and the API may not use the live FetchLabelAppliedAt fallback — is listed with an unknown time, and next.at is unknown when none is known; it is never “now”. Next-action honesty (#2120 R5): a state only a human or an external event moves — suspended, needs-human, escalated, awaiting-merge-decision — reports no deadline and no next.at, and a fabrik:blocked wait drops its cooldown deadlines, because a dispatch cooldown’s expiry does not change what happens to such an item (for #1342 the cooldown had been presented as its next action). The periodic-re-eval safety-net cooldown is never a deadline anywhere: it can only re-check, so it stays a reason and never reads “re-dispatched at …”.
Suspended (#2120, ADR 2120). A derived state — never a label — for the most common human decision point, which the pre-#2120 classifier called stalled after daemon uptime, or waiting. It is suspended when the item is a managed, non-holding stage and nothing the engine owns is going on: no worker, no fabrik:paused, no in-flight marker (fabrik:editing, fabrik:locked:*, stage:*:in_progress), and no state-bearing wait label. Cooldowns never mask it (they can only mean “re-check later”). Two cases:
- (a)
decision-pending:stage:<Status>:completeis present, the stage does not auto-advance for this item (stageAutoAdvances: nofabrik:cruise/fabrik:yolo, no global yolo, stageauto_advancenot true — the same predicaterunCatchUpPhase2uses), the stage is not Validate (its next stage is the cleanup stage;fabrik_advancerefuses it, and a cruise item settled there keeps itsawaiting-merge-decisionranking), and the cache holds no unprocessed feedback (the feedback gate’s pure cores).next_actionreads “advance to<next stage>(fabrik_advance) … or set cruise/yolo (fabrik_set_autonomy)”. - (b)
parked-blocker: every open blocker, directly or transitively, sits in an unmanaged parking column (stages.IsParkingColumn), or is itself blocked only by such items. Resolved cache-only (Peek/Scan, depth ≤ 8, cycle-safe); anything the cache cannot vouch for — a blocker it does not hold, an un-deep-fetchedBlockedBy, an active-column blocker with no open blockers of its own — fails closed towaiting.next_actionnames the blocker: “promote blocker #N out of Backlog (fabrik_promote); or remove the blocking dependency”.
Honest age. pending.since is the time the deciding fact became true and pending.since_basis says how far to trust it. For a completed stage it is the stage:<X>:complete label’s GitHub time (github-label); until the poll-side recorder has that, the cached Fabrik stage comment’s CreatedAt stands in (stage-comment, an estimate: the comment is posted before the label and a rework or step back can leave an older completion’s comment as the latest), else the daemon’s first sight (first-seen). A parked-blocker wait has two facts, the item’s fabrik:blocked label and the blocker being parked, and starts at the later of them: blocker-parked is reported only when the label’s GitHub time is known and the daemon watched every direct blocker move into its parking column (ItemState.StatusMovedAt, stamped only for a move between two known columns of an item the store already held; zero after a restart). A 30-day-old label says nothing about a blocker parked minutes ago, so otherwise — an unwatched move, or a transitive chain whose intermediate edges nobody observes — the basis is first-seen. A later EligibilityChangedAt wins over all of these (eligibility-change: the age starts at the change). github-label, eligibility-change and blocker-parked are exact; stage-comment and first-seen are not: then age_seconds is unknown, at_least_age_seconds carries the value and the summary reads “waiting at least since …” (first-seen) or “waiting since about …” (stage-comment); a daemon-observed age is never presented as the wait. The store stamps (EligibilityChangedAt, StatusMovedAt) come from the store clock (Store.SetClock, set to the engine clock by Engine.SetClock), so a sim clock drives them and the classifier together. The label time is read once per item per restart by the poll-side recorder (engine/decision_anchor.go, settleDecisionAnchors: at most 8 labelAppliedAt reads per poll, only for items the classifier would call suspended, a miss or error retried after 10 minutes) and written into the store’s LabelAppliedAt map; engine/localapi*.go and engine/channel_*.go still make no GitHub call.
Options. A decision-bearing result carries options — an ordered list of {action, tool}, tool empty when none performs it (merge PR #N is the human’s own merge). Decision pending: fabrik_advance, fabrik_set_autonomy cruise, fabrik_set_autonomy yolo. Parked blocker: one fabrik_promote per blocker, then “remove the blocking dependency”. Settled at Validate under cruise: merge PR #N, fabrik_set_autonomy yolo. Paused/needs-human/escalated: the resume method the engine’s own pause comment gives (R-options-2). resumeMethodOf walks back from the newest Fabrik comment to the newest state-bearing one and skips comments that say nothing about a pause, so an overseer audit comment (🏭 **Fabrik — overseer action:) or another informational Fabrik comment posted after the pause never shadows it: a comment that tells the operator to remove fabrik:paused (a daemon clean stop, a CI/spawn/merge-gate pause) gives “remove the fabrik:paused label”; one that asks for a reply or comment “to resume” gives “a human comment”; a later stage comment (an awaiting-input question) supersedes an older pause comment and, with no instruction of its own, leaves the method unknown — a plain awaiting-input question then reads “a human comment”. When the cache cannot say, the text points at the pause comment and claims neither.
Worker kinds, queued and in use (#2121, ADR 2121). The worker record (itemstate.WorkerHandle) carries a Kind — stage-run, comment-review, review-reinvoke, ci-fix-reinvoke, rebase-reinvoke or merge-train-conflict — a Trigger (the comment or review-body ids the worker is addressing, comma separated), a Queued flag with QueuedSince, and the live TurnsUsed/TurnsKnown. Kind is set where the worker is created (dispatchCandidates, dispatchReinvoke via reinvokeOpts.kind) or, for a comment review, by processCommentsClassified, which marks the stage placeholder comment-review for the pass and restores the prior kind afterwards; it never overrides a reinvoke kind. The trigger of a reinvoke is applied after its build step, once the worktree is ready and the ids are known. A kind the engine cannot determine is absent and the item is still working. A merge-train conflict resolution is not a Worker (the stale-worker detector, the shutdown pause set, the janitor and the overseer refusals all treat Worker != nil as a stage worker): it is a separate ItemState.ConflictResolution{Queued, Since} on the member, set around acquireTrainSlotEp and InvokeForComments in resolveConflictWithClaude and cleared in a defer; the classifier treats it like a worker.
Queued is not running. A handle stays non-nil while the worker waits for a slot — about a dozen Worker != nil guards (double-dispatch, shutdown pause, janitor, overseer refusals) mean “claimed” — with Queued set. dispatchCandidates takes its slot before WorkerEntered (ADR 2114); dispatchReinvoke used to apply it on the poll goroutine before its goroutine blocked on e.sem (F16) and now applies a queued WorkerEntered, then WorkerSlotAcquired immediately after the slot is taken: that clears Queued and sets StartedAt and LastSignAt to the acquisition time (WorkerChanged only, never WorkerLifecycleChanged, so it wakes nothing). TestWorkerEnteredSitesAreAudited pins the audited set of WorkerEntered sites, so a new site must be reviewed for this ordering. The stale-worker detector skips a queued handle (it holds no process or lock to be stale about; the goroutine’s deferred WorkerExited ends it). Status reports worker.queued, worker.queued_since and worker.in_flight: false until the slot is held; worker.started_at is the slot-acquired time. fabrik_health.workers.in_use is the number of item workers that hold a slot, plus any merge-train conflict resolution holding a general-pool slot (ConflictResolution.GeneralSlot, recorded by acquireTrainSlotIn), clamped to max_concurrent; queued is the number waiting. A resolution on the reserved merge-train slot is reported separately (train_in_use), not in in_use.
Live turns. InvokeOptions.OnTurn(turns, budget) is called by turnCountingWriter once with zero turns when an invocation starts and after every logical turn; liveTurnRecorder applies a WorkerTurnsRecorded for each call (a single in-memory field update that wakes nothing). The zero-turn call forgets the previous invocation’s count (several invocations can run under one worker handle); a final OnTurn(0, 0) when runClaude returns forgets the count and budget again, because the handle outlives the Claude process (result handling, pushes, later work) and must not report a finished invocation’s turns as live and records that invocation’s effective budget — the stage’s max_turns, a comment review’s own limit, or a fabrik:extend-turns/extension budget — as WorkerHandle.TurnsMax. worker.turns_used is unknown until the first turn of the current invocation is recorded, never zero, worker.max_turns is that invocation’s budget (unknown before it starts), and the working summary reads (turn n of max) when known.
Waiting for dispatch (#2121 R3). Input.Dispatchable (engine/attention_workers.go, cache-only, fails closed) is true when the item is an open, non-PR card in a managed stage, its stage:<Status>:complete is absent, nothing gates it (pause, fabrik:blocked with an open blocker, any fabrik:awaiting-*, fabrik:editing/reworking/lock/in-progress markers, api-key-helper-detected, skill-unresolved, claude-limit, an engine-paused stage, a held Claude usage limit, a cycle counter at its limit, a gate cooldown) and it has no worker, conflict resolution or gate hold. It is deliberately not conditioned on autonomy: a stage is dispatched whatever the autonomy label, which only governs advancing. The reason is exactly one of: all N of N worker slots are busy (cause slot-starved; the count is len/cap of e.sem), a cooldown that will really re-dispatch it (retry-backoff, sentinel-live — stated with its expiry), or the next poll. Gate cooldowns (feedback-pending, review-blocked, merge-unsettled), dep-blocked and periodic-re-eval are never offered as the reason. Only an unexplained gap (slots free, no real cooldown, past the stall threshold) falls through to stalled. A sim test pins the predicate against the real dispatcher.
Retry and gate-hold reasons (#2121). The facts the channel anomaly detectors held in memory only when a hub ran now also reach the Store, written from the same single hook points before the hub nil check: ItemState.PendingRetry{Stage, Cause, Due} from emitRetryScheduled (only on the paths where the retry will really happen, so an exhausted or escalated retry is never one; cleared on dispatch, completion, pause and close) and ItemState.GateHold{Gate, Reason, IDs, Since, ConfirmedAt} from noteGateHold/noteGateClear. Each gate ends its own hold the moment it stops holding — the feedback gate when nothing is unprocessed, the review gate when it neither blocks nor times out, the merge gate once mergeability settles, the CI gate once CI is no longer pending — so status stops naming a hold that has ended; a hold a gate site stopped confirming without clearing (a path that never reaches those points) is still ignored after wedgeLapseFor. A killed item is waiting with a summary naming the kill and the retry; the leftover stage:*:in_progress and fabrik:locked:* labels get the reason “left over from the killed run until the retry runs; not a live worker”. A held item names the gate and the pending feedback kind and id(s) (“+N more” past three). Neither is stalled.
Cause vocabulary (#2121). Result.Cause (attention.cause on the wire, omitted when empty) says why the item is in the reported state, from a fixed vocabulary, one cause per result, absent rather than guessed. Each is established by an explicit recorded fact:
| Cause | Establishing input |
|---|---|
wall-time-kill, max-turns |
PendingRetry.Cause (the retry rows). inactivity-kill, api-error and the other engine retry causes appear in the summary but yield no cause |
slot-starved |
a queued worker or conflict resolution, or waiting for dispatch with every slot busy |
gate-hold:<gate> |
a fresh GateHold; <gate> is feedback, review, ci or merge |
daemon-stop |
the item is paused and the newest state-bearing Fabrik comment carries the <!-- fabrik:pause-cause=daemon-stop --> marker that runShutdownPause adds, and the store has not watched an eligibility change (EligibilityChangedAt) more than two minutes after that comment — a resume followed by a bare re-pause that wrote no Fabrik comment would otherwise inherit the old marker. An unknown (zero) stamp proves nothing and keeps the cause. A bare stale in_progress label, a crash or a later pause comment yields no cause |
rebase-conflict-after-landing |
fabrik:rebase-needed is present and ItemState.RebaseCause records after-landing, written only by invalidateConflictingQueued (a merge-tree conflict found after a train landing) and cleared with the label. The ErrNotMergeable path writes none |
Label meanings (#2121 R5). Every state-bearing fabrik:* label has a one-line meaning in the reasons (attention.MeaningOf), including fabrik:editing, fabrik:reworking:<Stage>, fabrik:locked:<id>, fabrik:credited-pr:<N>, fabrik:spawned-child:<i>:<n>, the autonomy and operator-command labels. Families with a variable part resolve by prefix. internal/attention/labels_scan_test.go scans the engine, github, cmd and internal sources for fabrik:* label literals and fails for any with no meaning that is not on its explicit exemption list (each entry carries a reason); a non-vacuity test proves it fails for a fake label.
Zero GitHub cost. engine/localapi*.go reads the Store only through Peek (no FallbackFetcher) and Scan, and never the engine’s GitHub clients, Store.Get or labelAppliedAt — enforced by TestLocalAPISourceDoesNotReachGitHub and a nil-client test. The poll loop’s single-goroutine fields reach the API only through the mutex-guarded daemonHealth mirror.
PR facts and CI verdict (#2122, ADR 2122). The status tool’s pr block is rendered from two poll-side records on LinkedPRState, separate from the fields the gates read (State, Draft, Merged, Mergeable, MergeableState, CheckRuns), so writing them cannot change a gate. Facts (PRFacts: state, draft, merged, mergeable, mergeable state, the head they were read for, ObservedAt) is written by PRFactsObserved from two places: the per-poll board probe (runProbeAndDeepFetch applies the probe query’s linked-PR scalars state isDraft merged mergeable mergeStateStatus, which add no connection to the query, as FromProbe), and — for an item whose PR the probe cannot see (a base:<branch> PR, which closedByPullRequestsReferences omits) or whose facts the probe did not stamp this poll — a bounded REST FetchPRDetails in settlePRFacts (prFactsRESTPerPoll = 5 per poll). CIVerdict (CIVerdictRecord: verdict, the head it was computed for, failing check names, a fingerprint of the latest-per-name runs, ObservedAt, LiveReadAt) is written by CIVerdictObserved from settlePRFacts (engine/pr_facts.go, poll goroutine, after settleDecisionAnchors): for each managed open item with a linked open PR, least recently attempted first (in-memory attempt stamp; a PR known merged/closed is not re-read by REST inside the recheck window), it reads the head’s runs live through e.client.FetchCheckRuns — not the cache, whose READY set is trusted for ever and is never refreshed on a poll-only deployment; nothing is written to the Store from that read, so the gates see what they saw before — and classifies them with ciStatusVerdictFor, i.e. classifyLandingCI unmodified, so the overseer view and the gates cannot disagree about a head. A green verdict is restamped without a read only for prFactsGreenRecheck (5 minutes) after the last live read (CIVerdictRecord.LiveReadAt) — whether or not the Store’s cached runs are populated (they are empty on a poll-only deployment; when a webhook or gate filled them they must still be a READY set matching the verdict’s fingerprint) — then re-read, so a check rerun that fails on the same head surfaces; at most prFactsCheckBudgetPerPoll (8) evaluations that can cost a GitHub read run per poll, and an item past the budget is not refreshed and ages into stale. A failed read keeps the previous record unstamped. Both mutations return PRFactsChanged, which is not in wakeChFlags/cycleSetFlags, never call SelfWriteObserved, never move LastSeenPRUpdatedAt/LinkedPRUpdatedAt and never touch LastCIProgressAt; a verdict whose head is not the PR’s current head is dropped on write (and when the engine-recorded head and the head the facts were read for are both known and disagree there is no agreed head: AgreedHead() is empty, nothing is read or evaluated, any verdict is dropped and the API reports unknown until the two agree), PRHeadSHAUpdated clears it on a genuine head change (next to the CheckRuns prune), a merged or closed PR clears it, and the renderer shows a verdict only when CIHeadSHA equals head_sha. prInfo reports each group with facts_as_of/ci_as_of and facts_stale/ci_stale (older than one poll interval — the larger of the configured interval and the poll ticker’s effective period from the health mirror, so a back-off is not stale; unknown when the interval is not known). A fact never observed is unknown, not stale. The old ciVerdict(runs) summariser, which disagreed with the gates over reruns and cancelled/stale conclusions, was deleted. Seam: SetPRFactsRefreshDisabledForTest; sim twin tests/sim/pr_facts_test.go.
Poll liveness in fabrik_health (#2124, ADR 2124). poll() — the one function both PollWithBackoff (production) and PollOnce (the sim seam) run — records its lifecycle in daemonHealth: beginPoll at entry and, in a defer so an error or early return cannot leave it stuck, endPoll (in_progress, current_started_at, last_completed_at — completed means returned, with or without error; last_success_at keeps the success meaning), plus setPollStep at about ten phase boundaries (layer 2 status sweep, probe and deep fetch, fetching the project board, selecting deep-fetch candidates, settle scans, dispatch, merge train, post-dispatch settle scans, cleanup). A step is a phase name only; #2114 made worker-slot acquisition non-blocking, so there is no waiting for a worker slot step and nothing is invented. endPoll publishes next_due_at = end + max(configured interval, the period the poll ticker was last set to) as a fallback — an errored poll does not reset the ticker, so the next poll is due after the engine’s last chosen back-off period (and backoff_reason keeps that period’s reason), not after the configured interval, and one transient error in a deeply idle daemon is not reported as a missed poll; PollWithBackoff then overrides it on success with the engine’s own value — the effective interval after idle, GraphQL and webhook-cap back-off, or the REST reset on the hard-gate path — with backoff_reason idle, graphql or rest.
Overdue verdict (computePollVerdict, pure, engine/pipeline_liveness.go), with K = pipelineOverdueFactor = 2 (one constant, not an operator setting) and the configured interval: a poll in progress is overdue when now - started > K × interval (poll-in-progress); with no poll in progress it is overdue when now > next_due_at + K × interval (no-poll-started), where next_due_at falls back to the daemon start before the first poll. Measuring from the engine’s own next_due_at, not the configured interval, is what keeps an idle daemon (up to 60 min between polls with healthy webhooks) and a REST pause (up to ~1 h) from being reported as stalled. An unknown interval (poll_seconds unset, as in test engines) or baseline makes the verdict unknown, never healthy and never overdue. SetClock re-stamps the daemon start on the injected clock, and the infra-abandon cooldown reads e.now(), so tests control both.
Gauge ages. graphql_backoff.as_of is set only by setGraphQLBackoff, and rest_paused_as_of by setRestPaused, which the poll now calls on every attempt that passes the floor (a refresh, not only a transition). Neither moves with time alone; both are "unknown" until first refreshed.
Merge-train partitions. partitionStates joins the Store’s Queued rows (trainRows), the durable TrainRunPhases(), the worker markers and a non-mutating infraCooldownPeek (infraCooldownRemaining deletes an elapsed entry on read; the read path must not). A partition has an active run when it has an open run or an in-flight worker (the async driver lets the worker goroutine exit with a run still open, so worker_in_flight alone under-reports). With members and no active run computePartitionLiveness (pure; health and the deriver share it) gives idle_reason: poll-overdue, then infra-abandon-cooldown, else unknown — other holds (overlap deferral, the runaway guard, a base contradiction) are deliberately not duplicated in the read path. The partition stall condition is: merge_train: on, members, no active run, not in the infra cooldown, not inside a deliberate poll back-off, a known oldest_queued_since, and now - oldest_queued_since > K × interval. oldest_queued_since is the daemon-observed StatusEnteredAt, so it restarts with the daemon and is unknown without the board cache.
Read-surface polish (#2127, ADR 2127). Additive wire fields over state the daemon already holds; nothing reads GitHub and an input the cache lacks is unknown, never false or 0. fabrik_board: every result carries matched (items that passed the filter); the flow view’s excluded.closed counts matched closed items it leaves out, so sum(flow.count) + excluded.closed == matched; an attention view always serialises attention (BoardResult.MarshalJSON, [] when none), the flow view never does. fabrik_status: place.other_labels (labels that are neither stage: nor fabrik:), url derived from the repo and number (Engine.webURL, honouring ghes_host) when the cache holds none, no <column>:<kind> limit rows for an unmanaged or holding current column (attentionStageKind), last_invocation.figure_basis (duration and turns per-invocation, tokens and cost session-cumulative, as the CLI reports them; InvocationRecorded accounting is untouched) and pr.url / pr.reviewers. pr.reviewers and the validate-settled reviewers meta come from one pure builder, reviewerVerdicts (engine/review_verdict.go), evaluated at LinkedPRState.AgreedHead() (#2122): per reviewer’s latest non-dismissed review, at_head (review commit_id equals the head; unknown when either is missing - the pull_request_review webhook now carries commit_id), inline_at_head (unresolved thread comments by the reviewer with CommitOID == head; a thread comment has no review id, so author plus commit is the link), outstanding (unresolved non-outdated threads plus the latest review body when actionable and not in the durable review-ids-addressed marker, regardless of the processed/rocket mark), clean_pass (true only when at_head and outstanding == 0; false for an older head and for a CHANGES_REQUESTED review; unknown when at_head is unknown) and unaddressed (the feedback gate’s own cores, filterReviewThreadComments + dropOutdatedThreadComments + reviewBodyCandidates, per author). outstanding and unaddressed differ: a non-relaxed validate-settled is only announced when nothing is unaddressed, so outstanding is what distinguishes a clean pass from findings already worked through but never resolved. The builder lives outside the scanned globs and is itself pinned GitHub-free by TestReadPolishVerdictBuilderReachesNoGitHub. SetReadPolishDisabledForTest("R1".."R8") neutralises each requirement for the non-vacuity twins in engine/read_polish_test.go.
The situation view and fabrik_situation (#2171, ADR 2171). A fourth read method, situation, beside status / board / health. It is served through an optional interface (localapi.SituationBackend, set on Server.Situation, advertised by hello only when set; an older daemon answers unknown_method, which the shim reports as “this daemon predates fabrik_situation”), the same pattern as Streamer and Actor, so the three-method Backend is unchanged. Params {subscriber?, push?, force?}; result Envelope (as_of) plus view, plus pushed / push_note when a push was asked for. engine/localapi_situation.go holds the one builder, buildSituation(scope). It is a view over the existing classification, not a second classifier: each open, labelled item is run through attention.Classify (via attentionInput) and placed by its (State, Code) - Suspended → decisions; code awaiting-merge-decision → merge_decisions (review verdict reviewSummary, CI verdict cachedCIVerdict, the cores validate-settled uses); NeedsHuman / Escalated → human_input (with ResumeVia); Stalled → stalled; Waiting / Idle with a gate → gates. A gate is an unannounced anomalyState.holds entry (gate, reason, firstHeld), else the fabrik:awaiting-ci / fabrik:awaiting-review label wait with the label’s applied time when the daemon holds one; the hold start is "unknown", never “now”, when it began before this run. Because placement is a function of the one classification, an item is in at most one item section; a hold already announced as item-wedged is reported under anomalies (not again under gates). Items whose labels are not yet read are counted in unclassified_items, never placed, and stop nothing_needs_you. The account sections come from Health’s own sources (Health() is called for poll liveness, worker slots, the Claude hold and partitionStates); anomalies are the poll / partition stall verdicts, openAnomalyFacts (the open retry-starved / item-wedged episodes anomalyCatchUpEvents reports, now sharing that accessor) and cache-divergence - a point event, so anomalyState.recent keeps a bounded (50) memory of divergences inside cacheDivergenceWindow for it. Ordering is attention rank, then repo and number (gates: longest held first, unknown last); every section is cut at 10 before rendering with an exact omitted. The builder reads the Store (Scan, never nested with Peek), the health mirror and anom.mu only: localapi_situation.go and channel_situation.go match the localapi*.go / channel_*.go scan globs, so TestLocalAPISourceDoesNotReachGitHub and TestChannelEventsSourceDoesNotReachGitHub cover them, and TestSituationSourcesAreInsideTheGitHubFreeScans pins that. The tool is unscoped (account-wide); a scope narrows the item sections and item-scoped anomalies for a digest.
Local API lifecycle. Run() binds .fabrik/state/fabrik.sock (mode 0600) after the instance lock (§7.4) is held and after the webhook manager is assigned, before the first poll. A socket file already there is stale — replaced, but only if it is a socket. The socket is closed and unlinked when Run() returns and before the SIGHUP re-exec; the self-upgrade exec, a crash and an exec failure leave a file that the next start replaces. A bind failure only disables the API: it is logged (local API unavailable (continuing without it)) and recorded as the local_api:unavailable panel entry (§7.11), which the next successful start resolves.
7.14 Channel Events: Proactive Push to fabrik mcp Sessions (#1968)
ADR-1966-b. Where §7.13’s read API answers a question, channel events tell a session that a question is due. The daemon derives events from transitions it already observes, routes them to named subscribers through a persisted hub, and fabrik mcp writes them to the session as Claude Code Channels notifications/claude/channel messages (a research preview — see USER_GUIDE §9, “Proactive push”). Events observe; they never cause or alter an engine decision (R10), and they cost no GitHub call (R9).
Sources. (1) The itemstate observer: every non-no-op Store.Apply delivers a Change whose LabelDeltas list each label the mutation actually added or removed (a before/after diff, computed only for an item whose labels the store already knew (ItemState.LabelsKnown) — the first label-bearing population of an item, whichever fetch carries it, is a baseline, not a change; see “Baseline, arrivals, column moves and closes” below), plus its Origin (LocalLabel* write-through = the engine; webhook; anything else = a poll/reconcile) and, for webhooks, the sender.login and whether the engine’s echo registry matched. (2) A small number of nil-safe, void hooks at engine transition points that have no store signal of their own. (3) A 30 s ticker for the time-driven events. The observer only appends to a queue; one consumer goroutine owns all per-item memory and is the only publisher, which is what serialises ordering and dedup.
One event per actual change. The engine’s LocalLabelAdded write-through changes the store; the webhook echo that follows is an idempotent no-op that never reaches an observer. A reconcile that re-observes an unchanged label is a no-op diff. A reconcile that merely declines to revert a recent engine write (D.4, #2159) changes nothing in the store and so emits no event — previously it reverted the label and emitted a false unknown-actor event. Residual race: a webhook delivered between the REST call returning and the write-through is applied first, so its event is attributed by the webhook (actor = fabrik when the echo registry matched, else from the sender) — never duplicated.
Actor of a label event (engine/channel_events.go labelAttribute, #2125, ADR 2125), resolved in this order: overseer (the write was made by an overseer action — the requester’s sanitised subscriber name rides the write-through mutation and the echo-registry entry; the event also carries via=<subscriber>, and this takes precedence over the App-sender check), fabrik (the engine’s own autonomous write-through, a webhook that matched the engine’s echo registry with no requester, or — under GitHub App auth — a sender equal to the App’s bot login), bot (another [bot] sender), human (any other sender, including the operator’s own login under PAT auth, which cannot be told apart from Fabrik’s), unknown (no webhook sender: first observed by a poll, reconcile, deep fetch or probe, so nobody told us who did it). human and bot events carry the sender’s login as actor_login; overseer events carry via; fabrik and unknown carry neither. A requester is believed only from the engine’s own write or an echo-registry match (RegisterEchoVia/MatchEchoVia), never from an unmatched webhook, and a failed write registers nothing. Which unknown paths resolve: only the webhook sender (already on the delta). The events-API actor behind FetchLabelAppliedAt/FetchLabelRemovedAt is not used (it arrives after the event is built, and the labels those reads cover are almost all engine-applied), and the board/deep-fetch queries carry no timeline actors, so a human’s change first seen by polling stays unknown; no GitHub read is added for attribution.
Where each event comes from:
| Event | Produced by | Notes |
|---|---|---|
validate-settled |
runCatchUpPhase2 (the settle point — reached only when no Phase 1 handler claimed the item, i.e. dependencies, review gate, auto-merge convergence and the CI/merge gates are clear; §6.4, §6.6.6) via noteValidateSettled, called before the autonomy gate so cruise is covered; and attemptMergeOnValidate after its dependency, review-thread, review and feedback gates passed (via noteValidateLanding) — the settle point for yolo, whose landing happens inside that call and never reaches Phase 2. The yolo event is captured before the landing call but announced only if the merge, enqueue or move to Queued succeeded; noteValidateSettled at the top of Phase 2 announces only items that wait for a human, so a landing that fails (a conflict, an enqueue error) has announced nothing |
Never at FABRIK_STAGE_COMPLETE while fabrik:awaiting-ci is present (the complete label is deferred and the predicate requires it). A cache-only predicate confirms: Validate, open linked PR, stage:Validate:complete, none of awaiting-ci/awaiting-review/bot-reprompted/paused/blocked/rebase-needed/revalidate, no open blocker, no worker, and no unprocessed feedback (the feedback gate’s own pure cores, §2.2, read from the store; TestCachedFeedbackMatchesLiveGate pins the agreement). The event is captured on the calling goroutine, because a yolo item can be merged and moved to Done in the same pass. Episodes: one event per episode (meta.options — merge PR #N, fabrik_set_autonomy yolo — accompanies next=waiting-for-human); an episode ends when the complete label is removed (fabrik:revalidate), fabrik:rebase-needed is applied (the settle after the rebase is announced afresh), the item is closed or leaves Validate, or the PR head moves after Validate completed, or (non-yolo episodes) a worker is dispatched or unprocessed feedback arrives — each of those that the engine caused is announced with one settlement-withdrawn; the episode counter and dedup key persist, so a restart neither replays an announced episode nor loses an unannounced one. meta.next: waiting-for-human (cruise — the raw cruise label wins over yolo — or no autonomy), auto-merge (yolo, merge train off, or fabrik:auto-merge-enabled), merge-train (yolo, merge_train: on). meta.ci is read from cached check runs and is unknown when the cache holds none (the cache fills from check_run webhooks, so absence is not “no checks”); meta.ci_gate says whether the engine’s own CI gate was configured and cleared. meta.reviewers is a compact JSON array of per-reviewer verdicts at the PR head (§7.13 “Read-surface polish”); meta.ci_rerun (#2127 R7) names the failed checks the stage wait_for_ci gate re-ran (#2072 R1/R2) on this same head, and is present only when meta.ci is green - it reads Engine.flakeRerunChecks, the in-memory prFlakeState.rerunChecks (recorded only for a re-run actually made, never for a refused one), so it is absent after a daemon restart or startupStateTTL without a gate touch and for the merge train’s own (#2052) re-runs |
decision-pending |
engine/channel_decision.go: the consumer goroutine, at first sight, on a debounced re-evaluation and on the ticker, when the classifier (§7.13) reports an item suspended and no episode is open (#2120) |
one event per episode. Episode counters dp:<ref>/dpclosed:<ref> and dedup key dp:<ref>:<n> persist in the hub like validate-settled’s, so a restart neither replays an announced episode nor loses an unannounced one. Unlike the pause family it is announced at first sight (that is what makes a pending decision visible after a restart). meta: column, completed_stage, next_stage (decision) or blockers (parked), decision (advance/unblock), options (a JSON list of {action, tool}), since, since_basis, autonomy; the latest Fabrik comment link is always attached. An item whose GitHub time is not yet known waits decisionAnchorGrace (3 min) for the recorder before it is announced with since_basis=first-seen. The episode ends when the item stops being suspended |
settlement-withdrawn |
the same consumer, when a validate-settled or decision-pending episode ends because the engine took the item back (#2120 R-unsettle) |
meta.withdrew (the event type whose announcement no longer holds) and meta.reason, one of rebase-needed (fabrik:rebase-needed is present, or was applied since the last evaluation), ci-on-new-head (the PR head moved after Validate completed), worker-dispatched (a worker or fabrik:reworking:* marker) and new-feedback (unprocessed feedback, read from the cache); deduplicated per episode (swvs:<ref>:<n>, swdp:<ref>:<n>). A human acting — autonomy set, the item advanced or merged, closed, fabrik:revalidate, a blocker cleared, paused — is silent. Once the item resettles, a fresh episode announces itself afresh (R-unsettle-3). Immediate (never digested) |
label-applied, label-removed |
the observer’s LabelDeltas |
label, actor, stage, and unverified=true when the change is one a cache-divergence was raised for (#2170; the same marker is set on every other event built from such a change). Label patterns filter only these |
item-added |
the observer: a Change.Created mutation after the board was populated (Engine.boardPopulated, set when the first poll succeeds) registers a pending arrival; the consumer publishes it from evaluateItem once the snapshot has LabelsKnown, a board Status in a column Fabrik knows (a configured stage of any kind, or the legacy Backlog parking column per stages.IsParkingColumn), is open and is not a PR card (announceArrival, engine/channel_lifecycle.go) |
title, column, labels, autonomy (cruise wins over yolo; none), milestone (title, none, or absent when unknown), author, url. Event.Labels carries the arrival labels: a subscription’s labels include patterns match when any of them matches, and exclude_labels does not apply. Dedup key item-added:<ref>. Items present at startup, closed items, PR cards and items in a column with no configured stage never produce it; an unmanaged column (Backlog, Triage, …) does announce, and the later move into a managed column is a column-changed with no second item-added; an item first seen in an unconfigured column and later moved into a configured one gets column-changed only; a pending arrival whose labels or column never become known is dropped (with a log line) after the one-hour prune interval |
column-changed |
the observer, on a StatusChanged Change whose PrevStatus is known and differs from the new Status (a Created change or an unknown previous Status is a baseline and invents nothing) |
from, to, actor (+ via/actor_login, the §7.14 label vocabulary: fabrik for the engine’s LocalStatusUpdated write-through or an echo that matched, overseer with via=<subscriber> for a fabrik_promote move — the issue’s overseer:<subscriber> shorthand — human/bot when a projects_v2_item webhook named the sender, unknown for a move first seen by the Layer 2 sweep, probe, deep fetch or reconcile), status_at (the webhook’s updated_at; absent, never “now”, when unknown). One move is one event: the write-through changes the store first and the echo, sweep or probe re-observing the same Status is a no-op |
closed |
the observer, on a Change.Closed (open → closed on an item the store held) |
column (the board Status), actor (fabrik only when emitLanded announced the landing and the issue has not since been reopened or had its landing verification fail; a landing inferred from the snapshot alone keeps the webhook-derived actor), landing (true when emitLanded announced a landing for the item, or the snapshot shows fabrik:awaiting-landing-verification, a close-tracking label, or a merged linked PR). IssueClosed carries the webhook sender (no echo match: the engine registers no issues/closed echo); the engine’s ApplyIssueClosed write-through is OriginEngine |
paused, awaiting-input, escalated, stalled |
the same internal/attention classification the read API uses (§7.13), re-evaluated after a short debounce when an item’s labels, stage state, status or invocation change, and on the ticker; one event per transition into the state |
paused and awaiting-input are the classifier’s needs-human (fabrik:awaiting-input present → awaiting-input); escalated is its escalated (PausedByEngine — set by escalateSettle and every pauseFor* that arms it — a counter at its limit, or fabrik:awaiting-runaway-alert; the §6.x settle-scan exhaustion paths and §7.2 pause-on-retry-limit all land here); stalled is its stalled with the one --stall-threshold, so a push is never looser than fabrik_board and a subscription cannot override it. The debounce means a pause’s labels and EnginePaused are classified together, not as two events. A cruise item settled at Validate (awaiting-merge-decision) is validate-settled’s business, a suspended item is decision-pending’s, and a failed landing verification is its own event, so none is reported here. A pause-family event that waits on a human also carries meta.options and meta.resume (remove-label, human-comment or see-pause-comment): the resume method the engine’s own pause comment gives (§7.13 “Options”), never a generic one that contradicts it. A stalled event is never published for a pending decision. The event carries meta.cause (§7.13 “Cause vocabulary”) when the same classification gives one, e.g. daemon-stop on a paused event, and omits it otherwise (#2121) |
awaiting-input-stale |
the observer, when an invocation is recorded while fabrik:awaiting-input is present, that run did not emit FABRIK_BLOCKED_ON_INPUT and the engine did not pause the item itself |
once per awaiting-input episode; after a restart the invocation fields are empty, so no event (unknown, not guessed) |
cycle-limit-near |
the observer, on a stage-state change | a cycle/retry counter (not attempts) reaches limit - 1 (needs a known limit ≥ 2; zero config is unknown, never “limit 0”); re-arms when the counter drops below limit - 1, so a no-op refund (§2.9) followed by a new approach fires again |
merge-train-ejected |
ejectMember (§6.16: trial/conflict/bisection causes stay queued, review-findings leaves it) and ejectQueuedMemberForComments (§6.16, #1863) |
cause, reason, failing_checks (from the trainCIDiagnostic the engine already holds), ejections, max_ejections, stays_queued |
merge-train-failed |
ejectRedSingleton (own validation failing; “Merge-Train Red-Batch Bisection”; cause red-singleton-auto-repair when §6.31 started a repair instead of pausing), escalateStrandedTrainMember (landing failed, also the closed-unmerged-trial path) and fireRunawayGuard (§6.18) |
cause, reason, failing_checks when held |
ci-timeout |
pauseForCITimeout, only on a fresh pause (a reapplied pause on an episode that already announced itself is silent) |
timeout, ci |
review-timeout |
pauseForReviewTimeout |
reviews_before_wait names reviewers whose review predates the engine applying fabrik:awaiting-review — unknown after a restart, when that record-on-write timestamp is gone — and pending_reviewers |
merged |
emitLanded (engine/channel_lifecycle.go, #2123), called by the landing paths at the merge itself (attemptMergeOnValidate’s direct merge, the merge-gate rebase fallback, landSingleton, the singleton fast path, the batch loop) and as a backstop at the Done advance and in markCreditedLanding; fallback: the fabrik:awaiting-landing-verification label delta (§6.19) |
pr is the credited PR (fabrik:credited-pr:<N>, applied first, for merge-train members whose own PR is closed, not merged) else the linked PR; both producers share the dedup key merged:<ref>:<pr>, so exactly one is published. The early emit is what makes merged precede closed and the close-tracking labels (fabrik:awaiting-close, awaiting-member-close, credited-pr:*), which then carry landing=true in meta |
blocker-cleared |
the fabrik:blocked removal delta (the push-unblock path and the checkDependencies defense-in-depth path, see the fabrik:blocked label row) |
per dependent issue |
children-spawned |
spawnChildren, after every child is created, wired and fabrik:children-spawned applied (§6.7) |
children, count |
landing-verification-failed |
the fabrik:landing-verification-failed label delta (failLandingVerification, §6.19) |
|
claude-usage-limited, claude-usage-limit-lifted |
the edge of claudeUsageLimitedUntilTime (§7.3), checked on activateClaudeUsageLimit/clearClaudeUsageLimit and on the ticker, so a usage-limit hold that simply expires is also reported |
account-wide: no issue or repo; they ignore repo/issue/milestone scopes and reach every subscriber whose event filter admits them |
pipeline-stalled, pipeline-resumed |
engine/channel_pipeline.go (#2124, ADR 2124), checkPipeline from the deriver’s ticker, on the consumer goroutine — independent of the poll loop, so a blocked poll is still seen (FR-012). Poll scope (scope=poll): the §7.13 overdue verdict turns true (pipeline-stalled, cause poll-in-progress or no-poll-started, interval_seconds, blocked_seconds, overdue_after_seconds, step when a poll is running and the step is known) and ends (pipeline-resumed) when the verdict clears or the health mirror’s stall_epoch — bumped by beginPoll/endPoll whenever they close an overdue interval — moved, which catches an episode that ended and restarted between two ticks. Merge-train scope (scope=merge-train, partition, members, oldest_queued_seconds, idle_reason): a partition meets the §7.13 stall condition; resolved when a run starts (active run), the members leave Queued, or the partition disappears. A partition stall is published even while the poll stall is open; the poll event carries the poll cause, the partition event its own |
account-wide and immediate: no issue, they ignore repo/issue/milestone scopes. Dedup keys pipeline-stalled:<scope>:<anchor RFC3339Nano>:<seq> (and pipeline-resumed:… with the same suffix) are unique per episode — the anchor is the poll’s start (or missed due time) or the partition’s oldest queued time, seq a per-process episode counter — because the hub persists dedup keys and a static key would swallow every later episode. Episode memory is in memory only: a restarted daemon that is still stalled re-raises. Detection latency is at most K × interval plus the 30 s tick |
retry-starved, retry-resumed |
engine/channel_anomaly.go (#2170, ADR 2170), checkAnomalies from the deriver’s ticker and drain. finalizeStageOutcome’s retry branch (and the usage-limit short-circuit) calls the emitRetryScheduled hook with the cause and a due time stamped from e.now() (e.now() + stageRetryBackoff(), or the hold end), because LastAttemptAt is wall-clock and CooldownAt is consumed on admission (ADR 2096). dispatchCandidates calls emitDispatchDeferred where it logs deferred dispatch. An open fact becomes an episode when now - max(due, last usage-limit lift) >= threshold with no worker; reason is no worker slot (a deferral recorded since due, or the general pool full; slots_in_use/slots_max), else poll overdue (§7.13 verdict), else unknown. A fact that was never announced is dropped silently when the item is paused, blocked, awaiting input or done, left the stage or closed; an announced episode always closes with retry-resumed, whose how meta is dispatched when a worker enters and otherwise closed, advanced, held or gone. A deferral explains no worker slot only while it keeps recurring (recorded no earlier than the due time and within max(3 × poll, 2 min) of the check) |
item-scoped, immediate; dedup keys retry-starved:<ref>:<due RFC3339Nano>:<seq> and retry-resumed:… are unique per episode. In memory only. Reinvoke and comment-review dispatches and cooldown holds (dep-blocked, feedback-pending, review-blocked) are not stage retries and are not reported here |
item-wedged, item-unwedged |
engine/channel_anomaly.go (#2170). Gate sites call noteGateHold(gate, reason, ids) after their own hold (feedbackGateBlocks with the describePendingFeedback text it logs; the review, merge and CI holds in catch_up_handlers.go / ci.go) and noteGateClear when the feedback gate stops holding. A hold keeps firstHeld and lastConfirmed and lapses when not re-confirmed for max(3 × PollSeconds, 2m) (plus githubRecheckInterval(), 10 × PollSeconds, for the feedback, review and CI gates, which are re-evaluated that rarely once nothing bumps the item). It becomes an episode when held for at least the gate’s threshold (--item-wedged-threshold for feedback; max(threshold, backstop + 2 × PollSeconds) for review, CI and merge), the item is not paused, no worker was dispatched or deferred since firstHeld, and the poll is not overdue. Ends with item-unwedged on a worker entering, a column change, a clear or a lapse |
item-scoped, immediate; dedup keys item-wedged:<ref>:<gate>:<firstHeld>:<seq>. internal/attention treats an active cooldown as waiting, so a hold that re-stamps a cooldown every poll never reaches stalled; this event covers that gap without changing the classifier. meta.cause is gate-hold:<gate>, the status vocabulary’s name for the hold (#2121). In memory only |
cache-divergence |
engine/channel_events.go (#2170). onChange keeps a mutex-guarded ledger of recent engine writes (OriginEngine label deltas and EngineStatusWrite column changes, stamped with e.now()). A change whose Change.Path is reconcile (ShallowBoardItemUpdated, the batch board apply), deep-fetch (ItemDeepFetched) or probe (FromProbe) that contradicts an in-window entry (an opposite label delta, or a status change away from the written column) publishes one cache-divergence and consumes the entry; a later webhook or engine change to the same key clears it. Every event built from that change (label-applied/label-removed, the label-derived merged/blocker-cleared/landing-verification-failed, column-changed, closed) carries unverified=true (channelevents.MetaUnverified, defined once so #2169’s provenance reuses it) |
item-scoped, ordinary; no recovery event. A column revert may be a legitimate external move that only the reconcile observed |
daemon-unreachable, daemon-reachable |
the fabrik mcp shim, when the held connection stays down past a 10 s grace and when it re-attaches |
live only: a daemon that is down cannot hold them |
events-dropped, digest |
the hub (synthetic) | count; dropped_types on events-dropped (e.g. label-applied x4), plus expired when entries aged out |
stream-superseded |
the fabrik mcp shim (synthetic, immediate, not subscribable), when RunStream reports Superseded — a newer attach under the same name took the stream |
subscriber; live only |
| situation (#2171) | engine/channel_situation.go: renderSituation(buildSituation(scope)), published by Hub.PublishSituation. Triggers: the catch-up (forced), a request (fabrik_situation push) and the cadence (situation_seconds, from onTick → publishSituations) |
account-scoped, immediate; a per-subscriber fingerprint dedup (below) instead of a DedupKey |
catch-up (validate-settled, decision-pending, paused, awaiting-input, escalated, and an open pipeline-stalled, with catch_up=true) |
engine/channel_catchup.go: a snapshot of items currently settled at Validate and waiting on a human, currently suspended, or currently needs-human / escalated, plus (pipelineCatchUpEvents) any pipeline stall open now and (anomalyCatchUpEvents, #2170) any open retry-starved / item-wedged episode — current account state, unlike an item’s time-driven stalled, queued on a new subscription, on the first attach of a session (attach {catch_up: true}, never on a plain reconnect) and, since #2169, to every subscriber at daemon readiness after a start (see “Restart delivery” below); carries source=catch_up |
not a transition: no dedup key, no episode counter or per-item memory touched, and an event already held for the same (type, repo, issue) is skipped; Peek/Scan only |
Anomaly events (#2170, ADR 2170). retry-starved/retry-resumed, item-wedged/item-unwedged and cache-divergence report per-item conditions the engine already knew about and previously only logged. They are observation only (R5/R10): engine/channel_anomaly.go reads the Store through Peek, the poll verdict through the health mirror and the slot counts through len/cap of the semaphore; facts the Store lacks arrive through void, nil-safe, panic-recovered hooks in channel_hooks.go (emitRetryScheduled, emitDispatchDeferred, noteGateHold, noteGateClear) called from the engine files that know them. Episode state is in memory only (a restarted daemon that is still in the condition raises a fresh episode) and every dedup key carries the episode’s anchor time and a per-process sequence number.
- Thresholds (flags/env in the user guide, read once at startup, classified in
internal/workerenv):--retry-starved-threshold5 min,--item-wedged-threshold10 min,--cache-divergence-window5 min. The first two never fall below 2 × the poll interval. - Kill cause.
runClaudewraps the error of a run it killed on a timeout inclaudeerr.KilledError{Cause: wall-time-kill|inactivity-kill}(only when the run did not complete). The wrapper is transparent (Unwrap), so everyerrors.Asclassification and theAttempts/max_retriesaccounting are unchanged.retryCausemaps an incomplete outcome towall-time-kill,inactivity-kill,max-turns,resume-failure,tools-denied,errororincomplete; the usage-limit and api_error exits (handleUsageLimitExit,handleAPIErrorExit) registerclaude-usage-limitandapi-error. Escalated outcomes register nothing: there is no retry. - Divergence provenance.
itemstate.ChangecarriesPath(reconcileforShallowBoardItemUpdatedandBoardReconciled,deep-fetch,probe),StatusFrom/StatusToandEngineStatusWrite.Origincould not tell these paths apart. - Scope. Reinvoke and comment-review dispatches and cooldown holds (
dep-blocked,feedback-pending,review-blocked) are not stage retries and never produceretry-starved; a hold that never reaches a dispatch isitem-wedged’s business.
Baseline, arrivals, column moves and closes (#2123, ADR 2123). The line between “present at startup” and “added later” is drawn once. (1) Label baseline: gh.ProjectItem.LabelsAbsent marks a producer that did not observe labels (the probe bootstrap BootstrapFromProbe and the minimal item runProbeAndDeepFetch seeds); applyProjectItem/applyShallowItem then leave Labels alone and do not set ItemState.LabelsKnown. A mutation produces LabelDeltas only when before.LabelsKnown (explicit label writes — LocalLabel*, IssueLabeled/Unlabeled — always do), so a probe-seeded record followed by a deep fetch, a shallow reconcile or a board reconcile is a silent baseline for open, Done, Backlog and closed items alike; ItemDeepFetched always clears LabelsAbsent (a deep fetch carries labels). The flag is negative on purpose: a producer that forgets it stays noisy rather than silently swallowing real changes. (2) State-derived events: derive, seed and the ticker skip an item until LabelsKnown, so the first label-bearing evaluation takes the existing silent first-sight branch and a restart re-announces no pause, awaiting-input or escalation; subscribers wanting the current picture use catch-up. (3) Arrivals: Change.Created is set only for a record a running store learns about (never by Reset), and only after Engine.boardPopulated; items created between hub start and the end of the first successful poll are silent. The arrival rule (#2187, amending #2123 FR-004): item-added is published for an open, non-PR issue arriving after startup in any column that has a configured stage — managed or unmanaged, keyed on the flag and never the name — plus the legacy Backlog column with no stage (stages.IsParkingColumn); an item in a board column with no configured stage never produces it, whichever path first sees it. Parking-column items are seeded into the store by the probe loop (minimal IssueOpened, LabelsAbsent, no deep fetch) so a later hand move out of the column is a real column-changed; their item-added waits for the first label-bearing population (a webhook, or the next light reconcile). (4) Catch-up never replays item-added, column-changed or closed — they are transitions, not current state (fabrik_board is the pull alternative; TestCatchUpNeverReplaysLifecycleEvents). (5) Noise option: progress_via_column on a subscription drops label-applied/label-removed for stage:*:complete and stage:*:in_progress when the subscription takes column-changed (events empty or naming it; Validate rejects the option with an events list that omits it); it overrides labels and exclude_labels: [] for those two patterns, and the default exclusions are unchanged.
The situation digest and the catch-up (#2171, ADR 2171). situation is a subscribable event (Immediate, AccountWide: Matches ignores the repo / issue / milestone scopes, and the builder narrows by the subscriber’s own scopes instead - Subscription.ScopeMatches, factored out of Matches, over the union of the scopes of the subscriber’s subscriptions that take it). Subscription.TakesSituation() is true for an explicit events list naming it and for the semantic default (EventsDefault); a grandfathered subscription (empty events, no marker) and an explicit list without it do not. Subscription.SituationSeconds (situation_seconds, fabrik_subscribe’s situation_seconds; default 0 = off; 60 s - 24 h, channelevents.MinSituation / MaxSituation; refused unless the subscription takes situation) is the cadence, deliberately not digest_seconds, which batches ordinary events into the synthetic digest. Three triggers, one builder: fabrik_situation (push: true, force), the cadence and the catch-up all call Engine.pushSituation(name, trigger, force) → buildSituation + renderSituation, so their Content is identical for the same state (trigger meta and the catch-up tags catch_up=true, source=catch_up differ). Cadence: channelEvents.publishSituations, on the consumer goroutine from onTick (the 30 s tick, so granularity is the tick), once the daemon is ready, for each attached subscriber whose shortest cadence has elapsed on e.now() since its last evaluation (sitEval, in memory); a detached subscriber is caught up on attach instead. Dedup: Hub.PublishSituation(name, ev, fingerprint, force) sends unless !force and the fingerprint equals the one last sent to name; the memory is per subscriber and persisted in dedup.json (stateFile.Situation, forgotten when the last subscription goes), and the hub’s DedupKey mechanism is not used (it is global and permanent, and could not resend “A, then B, then A again”). situationFingerprint is an explicit allowlist projection of the view - identity, title, link, state, code, rank, options, gate name / reason / start, verdicts, resume method, pending kind and blockers, partition members / phase / idle reason, slots, train queued, overdue verdict, anomaly kind / subject / start, truncation counts - and excludes everything that moves with the clock or the poll cycle alone (the as_of envelope, every age or *_seconds, whether a poll is running, the step, the classifier’s prose reason). Held queue: a send first drops any situation entry already queued for that subscriber (queueFile.dropType), so a subscriber that is away holds at most one, always the newest. Catch-up: Engine.catchUp, catchUpAll and catchUpLate push a forced situation digest (trigger=catch_up) for the selected subscriptions that take it - which also records its fingerprint, so the first scheduled digest does not repeat it, and restarts the reminder clock of subscriptions with remind_after via Hub.RestartReminders - and give the per-item catchUpEvents snapshot, through Hub.CatchUpWhere(..., notTakingSituation), only to the subscriptions that do not take it. Timing, the readiness gate and who receives a catch-up (§ “Restart delivery”) are unchanged; an empty system still gets an explicit “nothing needs you” digest. Seams: SetSituationDedupDisabledForTest (every digest is sent - the sim’s non-vacuity check), SetSituationMinForTest.
Restart delivery and downtime deltas (#2169, ADR 2169). Supersedes ADR-1966-b’s “a reconnect never gets a catch-up” and “a restart is silent”. After a daemon start (also a self-upgrade or SIGHUP re-exec) a subscriber receives, in queue order: (1) daemon-reachable, written by the shim at the attach ack, before the hub drains; (2) events already held for it (real, previously published); (3) one catch-up snapshot; (4) the downtime deltas. Nothing replays unchanged state, and a reconnect that is not preceded by a daemon start gets no catch-up (unchanged).
- Ledger.
channelevents.Ledger(internal/channelevents/ledger.go) persists.fabrik/state/channel/items.json: per item its labels (minus the default churn exclusions, so startup cleanup offabrik:locked:*/stage:*:in_progressnever shows), board column, closed flag and the last announcedvalidate-settled(episode,head_sha,autonomy, revalidated). It records what the store observed, independent of any subscription filter, so it is correct for every subscriber. The observer only writes a map under the ledger’s own mutex; the file is written atomically (tmp+rename) within about a second of a change and flushed inCloseand the pre-exec hook (flushChannelEvents). After a crash the last second may be missing and a change already delivered can be reported once more (such events areunverified). An item whose downtime delta is still held while the daemon warms keeps its previous record until that delta has been published, so a crash or kill during warming leaves the delta to be diffed again by the next start rather than losing it. A missing, corrupt (quarantined) or version-mismatched file makes every item a baseline; the daemon logs it and subscribers hear nothing. Records of unobserved items are carried forward for 14 days. - Downtime diff.
downtimeCheck(engine/channel_downtime.go) takes the item’s prior record once per part: the column and closed state at the first observation with a column, the labels at the first withLabelsKnown(the store notifies observers whenLabelsKnownflips, even on an empty label set, because the item state itself changed). A difference is built with the live builders (label-applied/label-removed,column-changed,closed) and carriessource=downtime-delta,actor=unknown(a diff names no writer) andunverified=true. An item with no record is a baseline (so arrivals during downtime stay out, ADR 2123). When the downtime diff of a change reports a column move or a close,onChangedrops that same change’s livecolumn-changed/closed, so neither is reported twice. - Readiness gate. A catch-up built right after a restart would read a label-less store. The engine is “warming” from hub start until the board is populated and every open non-PR item in a managed stage has
LabelsKnown, or untilchannelReadyTimeout(3 min). While warming the deltas are buffered, and so is any later change queued for an item that has a delta held (holdWarm), so an item’s changes keep the order they happened in; subscribe-time / attach-time catch-up requests are no-ops. On the warming-to-ready transitioncheckReadyfirst queues one catch-up to every subscription that exists when the snapshot starts (attached or detached, whatevercatch_upthe shim sent, so older shims work;catchUpAllreturns the covered set), raises the ready flag, queues the snapshot to any subscription created in between (catchUpLate, which is not in the covered set and was refused its own catch-up), then publishes the buffered deltas. After readiness, subscribe-time and first-attach catch-up behave as before. Derived events (for example a first-sightdecision-pending) are not held back and can precede the catch-up; the hub’s held-dedup keeps them from doubling in it. A close that happened while the daemon was down is decided at the item’s first observation, whether or not it still has a column. - Default family.
Subscription.EventsDefault(events_default,omitempty) is set bychannelStreamer.Subscribewhen the normalisedeventslist is empty.Matchesthen admits every subscribable type exceptlabel-applied/label-removed; naming them restores them. A subscription persisted before #2169 has no marker and is grandfathered as “every type” (a running overseer never silently loses events); unsubscribe and subscribe again to move to the new default.canonical()ignores the marker, so a repeated subscribe with the same filters stays idempotent.labels,exclude_labelsandprogress_via_columnonly shape label events; theSubscriberesult carriesnotessaying so when label events are not in effect. - Provenance. Every event carries
source:live(stamped infinishEvent; a held event delivered late is still live),catch_up(the catch-up builders, beside the oldercatch_up=true) ordowntime-delta. There is no replay value. On the wire the key isorigin(wireMetaKeyininternal/mcpstdio/channel.go), because Claude Code sets the<channel>tag’s ownsourceattribute from the server name; the shim also defaultsorigin=livefor the events it generates. A label, column or closed event built from a change whoseOriginis a reconcile, deep fetch or probe, with no known write or webhook behind it, carriesunverified=true(widening #2170’s rule, which stays the narrowcache-divergencetrigger). Actors use the ADR 2125 vocabulary. - Re-settle cause. A
validate-settledthat is not the first for the item (ledgerSettleRecordpresent) carriescause:revalidated(the previous episode ended byfabrik:revalidate), elsenew-head(head_shadiffers), elseautonomy-changed(cruise/yolo/none differs), elseother. The first settle carries no cause (after a lost ledger the daemon cannot honestly say “initial”).autonomy-changedalso carries the autonomy writer’sactor(+via/actor_login):onChangeremembers the attribution of eachfabrik:cruise/fabrik:yolodelta (an added label wins over a removed one, as a swap is two deltas) and the consumer takes it when stamping; an unseen write isunknown. An autonomy change while the item stays settled ends the open episode (autonomyChangedSinceSettleincloseSettleEpisode) without asettlement-withdrawn, so the next settle point announces it again; under yolo the announcement still waits for landing success. Seams:SetDowntimeDeltasDisabledForTest,SetSettleCauseDisabledForTest,SetChannelReadyTimeoutForTest,StartChannelEventsWarmingForTest. The autonomy compared is the effective mode (labels plus the daemon-wideyolosetting), so restarting withyolotoggled re-announces each settled item once withcause=autonomy-changedand actorunknown: the flip changes what the engine does with them.
Title on every item-scoped event (#2127 R8). baseEvent, the one constructor of item-scoped events, sets meta.title from the item’s title (pushTitle: line breaks and backticks become spaces, whitespace collapses, cut at 200 runes). Account-wide events, and the two fallbacks that build an event for an item that has left the Store (retryResumedEvent, itemUnwedgedEvent), carry none; an event held before an upgrade lacks it and a held event keeps its creation-time title.
Hub, subscriptions and held delivery. Subscribers are keyed by a stable name (the operator’s topic or session name — never a PID; supplied by fabrik mcp --subscriber / FABRIK_SUBSCRIBER). A subscription ANDs its set scopes (repos, issues, milestone, label patterns for label events, event types) and ORs the entries within a list; an item whose milestone is unknown never matches a milestone scope. One event reaches a subscriber at most once however many of its subscriptions match. The hub appends to a per-subscriber persisted queue before any delivery; the attached session’s drain loop sends in order and removes an entry only after a successful write, so held events (no session attached) go out first, in order and individually, then live ones. The queue is bounded (--channel-held-max, default 200): overflow evicts the oldest non-immediate entry first and drops an immediate one (validate-settled, escalated, paused) only when nothing else is evictable, and the next delivery starts with one events-dropped notice naming the dropped event types (an entry that aged out is counted by type too, and the notice says expired rather than overflow); entries older than 7 days and subscribers not connected for 30 days are pruned (the clock is the last time a session was connected: it is refreshed while attached, on detach and at shutdown, so a long-lived session is never mistaken for an abandoned one). A second attach under the same name supersedes the first (it receives a superseded frame, the shim tells its session with one stream-superseded notice, and it stops reconnecting). Digest mode batches ordinary events per subscription (30 s–1 h); validate-settled, escalated, paused and daemon-unreachable bypass it, and a digest entry queued while the subscriber had no attached session is delivered individually, in order, on its next attach (not re-digested). A digest collected while a session was attached keeps batching across a brief reconnect (heartbeat reset, daemon re-exec) and is sent when due. Delivery means “written to the shim”: Claude Code never acknowledges a channel message and drops it silently when the session did not enable the channel, which the daemon cannot detect.
Subscribe under version skew (#2168, ADR 2168). The daemon self-upgrades while a fabrik mcp shim lives as long as its Claude session, so a shim routinely names event types its daemon no longer (or does not yet) know, and no daemon-side change reaches a schema baked into a running shim. channelStreamer.Subscribe therefore runs channelevents.NormalizeEvents on the request before the hub sees it: the two pre-#2129 names (claude-limit-suspended, claude-limit-lifted) are mapped to claude-usage-limited / claude-usage-limit-lifted, every other unknown (or synthetic) name is rejected, and the rest of the subscription is applied. The result reports both: mapped ([{from, to}]) and rejected ([]string), each omitted when empty, so a request of current names alone is unchanged. The kept list is de-duplicated, so ["claude-limit-suspended", "claude-usage-limited"] is one entry, and because mapping happens before the hub’s idempotency check, re-subscribing with an old name is the same subscription, not a second one. An empty event list means every event, so a request that named events and keeps none is refused (bad_request, naming each rejected type), stores nothing and leaves the subscriber’s existing subscriptions untouched. Hub and Subscription.Validate stay strict; the persisted-state migration in Open is unchanged. A new shim against an old daemon is not covered: the old daemon rejects the request as before.
Version skew is reported. hello carries daemon_version (omitted when the daemon has none), and fabrik_health carries the daemon’s version. The shim compares the daemon’s version with its own as opaque strings and, the first time they differ, appends one text block to a tool result naming both and advising a session restart; it is never repeated. fabrik_health also gets a “Versions — daemon X, MCP server Y” block. Every other tool checks with a hello until the notice has been shown, but a check that found no skew (equal builds, or a daemon without daemon_version) is trusted for five minutes (skewRecheckInterval), so a daemon upgrade mid-session is noticed within that time without a hello on every call, and the hello is made outside the lock so concurrent tool calls do not queue behind it; a failed hello is retried by the next call, a daemon without daemon_version or a shim without a version yields no notice and no error. Nothing on the daemon side depends on what the shim reports.
Unsubscribe is a hard stop (daemon side). Hub.Unsubscribe removes the subscription (all of the subscriber’s when no id is given), bumps the subscriber’s epoch, discards every queued entry that none of the remaining subscriptions matches (all entries and the drop counters when none remains; removing one of several subscriptions therefore also discards entries only it matched), and then waits for the drain loop’s current cycle: the drain loop holds sendMu across a peek→deliver→remove cycle, so an in-flight Deliver has finished before Unsubscribe returns. The drain loop for its part rechecks the epoch and its attachment immediately before every Deliver (a batch taken before the unsubscribe is abandoned unwritten) and, at the top of every cycle, re-matches the queue against a fresh snapshot of the subscriber’s subscriptions, which catches a Publish or CatchUp that resolved its targets before the unsubscribe and enqueued after it. The guarantee is: no Deliver for a removed subscription completes after Unsubscribe returns; the wait is bounded by the session’s 5 s write deadline. The attachment stays bound after the last removal (closing it would make the shim reconnect and re-attach), so a later subscribe under the same name resumes delivery on it. Lock order: h.mu is never held while taking s.mu or sendMu, and sendMu is never taken under h.mu. What the daemon cannot recall: events already accepted by the socket, the shim’s stdout pipe or Claude Code’s own queue when Unsubscribe returns may still surface in the session.
Durable vs in-memory. Subscriptions, held queues, dedup keys and settle-episode counters live under .fabrik/state/channel/ (subscriptions.json, held-<hash>.json, dedup.json), written atomically and quarantined (.corrupt-<unix>) when unreadable, like workers.json. Subscriptions and held queues are written on every change; dedup keys and counters are coalesced into one dedup.json write at most every 250 ms (Publish runs on the engine’s poll and tick paths and never does a per-event rewrite) and flushed synchronously by Hub.Close, which the SIGHUP path calls before the exec — so only a crash inside that window can lose the newest dedup keys and republish an event once. They are loaded before the socket binds, so a reconnect during startup finds them. The per-item memory the deriver keeps (last pause-family state, counter arming) is in-memory only and re-seeded silently from the store at startup, so a restart announces nothing about the existing board. Events that occur while the daemon is down, or during a self-upgrade exec, are not captured.
Reminders (#2120 R6a). A subscription may set remind_after (seconds, 600 – 2 592 000; default off, fabrik_subscribe’s remind_after). A still-open decision-pending episode is then re-sent to attached sessions of that subscription once per interval — meta.reminder=true, meta.reminder_count, a (reminder) content prefix, and the original event’s attributes and comment link (the reminder is built from the event as announced, so it cannot differ). The interval runs from the later of the hub’s persisted announcement time (its dedup entry) and the previous reminder (persisted per subscriber in dedup.json’s reminders), so a restart neither fires nor postpones one and no reminder is ever driven by daemon uptime; with no announcement on record a first look only plants the anchor. Reminders are never held for a detached subscriber (its catch-up on attach already carries the pending decision), have no dedup key, and end with the episode. A catch-up of an open decision-pending episode restarts that subscriber’s reminder clock (the catch-up event carries the episode key to Hub.CatchUp, which clears it before queueing), so a reminder already overdue at attach time does not fire right behind the catch-up. A stalled pending decision is never re-published as stalled.
Label events are not a readiness signal. A label such as stage:Validate:complete can be applied and removed again while the PR still needs a rebase, CI or a fix (it was re-applied and pushed while a PR was DIRTY). Only validate-settled (and decision-pending) say an item awaits a human, and settlement-withdrawn says that stopped being true before the human acted.
Connection lifetime (protocol v2). hello reports protocol_version 2 (and daemon_version) and lists subscribe/unsubscribe/attach only when the channel hub is running; a client that does not find attach degrades to read tools. After attach the server writes a heartbeat frame every 30 s (Server.HeartbeatInterval); every successful write moves the connection’s read deadline out by the idle timeout (Server.IdleTimeout, default 10 min), so a push-only subscriber is never dropped, and a write that fails (the 5 s write deadline) reaps a dead peer within one heartbeat. The client treats 3 missed heartbeats as a half-open connection and reconnects with jittered exponential backoff (250 ms → 30 s), re-attaching under the same name every time; a busy refusal at the 32-connection cap is retried. Each attached shim holds one of the 32 slots.
Exec paths. The SIGHUP restart closes the hub (closeChannelEvents) before the exec. The self-upgrade exec (checkAndUpgrade → selfupgrade’s PreExecHook → flushChannelEvents) only flushes the hub’s lazily written dedup and counter state immediately before syscall.Exec and tears nothing down, because a failed exec leaves the process running; so a settle announced within the 250 ms flush window of an upgrade is not announced again by the new process.
Known limits. Delivery is at-most-once to the session (no acknowledgement; a half-open connection can lose up to about one heartbeat window of events). fabrik_unsubscribe is a hard stop only for what the daemon has not yet written: pushes already in the kernel socket buffer, the shim’s stdout pipe or Claude Code’s queue (on the order of 200 notifications) cannot be recalled, so a few can still appear after it returns. Label events can be reordered in a narrow race between two concurrent applies of the same label. A restart or self-upgrade replays nothing: no re-published pause-family events and no label events for unchanged items; what it does deliver is the catch-up snapshot and the downtime deltas (#2169, “Restart delivery” above). Arrivals that happen while the daemon is down are not captured, and items created between hub start and the end of the first successful poll are not announced. A column move is attributed human/bot only when a projects_v2_item webhook named the sender; Status often reaches the daemon only through the Layer 2 sweep (§9), so unknown is common and status_at is then absent. merged precedes the close events of a landing only when the engine itself performs the merge: for a landing GitHub completes on its own (native auto-merge, merge queue) the issues closed webhook can reach the store before the engine observes the merge, and that closed is then published first with landing=false. A sweep that observes an engine move before the write-through lands attributes it unknown, and the write-through is then a no-op (still exactly one event). The label actor is unknown when only a poll saw the change (no cheap read names the actor; ADR 2125), via is a self-asserted advisory name, an echo that arrives before the engine registers it keeps today’s attribution, and under PAT auth a worker’s gh label write whose echo was missed reads as human. One incident can produce several related pushes (paused + ci-timeout + label-applied).
Zero GitHub cost and observation only. engine/channel_*.go and internal/channelevents read item state only through Peek/Scan and the guarded engine state, and never the engine’s GitHub clients, Store.Get, the live label-timestamp fallback or the live feedback/review gates — enforced by TestChannelEventsSourceDoesNotReachGitHub; TestChannelEventsAreObservationOnly (tests/sim) shows a run with a hub and a subscriber makes exactly the same GitHub mutations in the same order as a run without one.
7.15 Overseer Actions: the Mutating Local-API Method Set (#1969)
ADR-1966-c. §7.13’s read API and §7.14’s push stream never write to GitHub. The action method set — promote, advance, move_parked, set_autonomy, revalidate, clear_claude_limit, surfaced as the fabrik_promote / fabrik_advance / fabrik_move_parked / fabrik_set_autonomy / fabrik_revalidate / fabrik_clear_claude_limit MCP tools — is the first part of the local API that does. It adds no engine transition: each action performs one the engine already has (a board Status move out of a parking column or between two parking columns, the one-stage advance of a completed stage, the fabrik:cruise/fabrik:yolo labels, the one-shot fabrik:revalidate and fabrik:clear-claude-limit command labels) and the engine’s existing settle scans consume the command labels exactly as if an operator had applied them in the UI.
Separation. localapi.Actor is a second optional interface beside the three-method read Backend (the Streamer precedent). Server.handle routes an action method to Server.handleAction, the only caller of the Actor; a server with no Actor answers every action method unknown_method and hello does not list them. The engine implements Actor in engine/overseer_actions.go (advance in engine/overseer_advance.go, move_parked in engine/overseer_park.go) — deliberately not in localapi*.go or channel_*.go, whose static scans forbid the GitHub-client identifiers an action needs, so the read-path guarantee of §7.13 and §7.14 stays pinned. startLocalAPI wires it with one line. Refusals are CodeRefused with the item’s current state in Error.Data.
Per-action sequence (serialised with each other by one mutex; no per-item in-flight guard is taken, so an action races the poll loop exactly as a human applying a label or dragging a card does): (1) resolve the issue through the same resolver as fabrik_status, from Store.Peek only — a miss is not_found, a closed item is refused; (2) check the precondition on the cached snapshot, a missing fact counting as unknown and refusing; (3) for promote, advance and revalidate, one ID-keyed live FetchProjectItemStatus read confirms the column (the #1871 liveLandingState precedent; a read error refuses — fail closed — and the live column, not the cache, decides); (4) write through the engine’s own idioms — addLabelChecked / removeLabelChecked (error-returning, with cache write-through, SelfWriteObserved and echo registration) and moveItemToStatus (the status-move idiom of advanceToNextStage/moveItemToValidate, taking the project ID from the board cache, the item ID from the store and the field and option IDs from the loaded statusField, so no FetchStatusField, FetchProjectBoard or LookupIssueProjectItem is made); (5) post the audit comment through postComment (via the single postAudit helper). If set_autonomy fails part-way after at least one label write landed, the comment is posted anyway, worded as a partial application, and the error is still returned; if no write landed, no comment is posted.
| Action | Precondition (all on top of: one managed, open, non-PR item) | Writes |
|---|---|---|
promote |
to is a configured, non-unmanaged, non-holding, non-cleanup stage with a board option; the cached and live column is a parking column (stages.IsParkingColumn: an Unmanaged stage, or the legacy Backlog when no Backlog stage is configured); project ID, item ID and status field are known |
one UpdateProjectItemStatus |
advance |
(#2126, ADR 2126) cached and live column is a configured pipeline stage whose following stage (skipping parking columns) is not a holding or cleanup stage — so Validate, which leads to Queued/Done, is refused with “use the PR merge”; the next stage has a board option; no worker (ItemState.Worker, live fabrik:locked:* / fabrik:editing / stage:<X>:in_progress); live stage:<X>:complete present; no fabrik:paused / fabrik:awaiting-input / fabrik:blocked and none of fabrik:awaiting-ci, fabrik:awaiting-review, fabrik:rebase-needed, fabrik:auto-merge-enabled, fabrik:awaiting-done (labels read live, failing closed); then the read-only gates: no open dependency (openDependencies on a live FetchItemDetails copy), feedbackGateBlocks clear, and reviewGateHold clear on a wait_for_reviews stage |
one advanceToNextStage — the transition auto-advance uses, including clearStatusLine; no label write |
move_parked |
(#2188, ADR 2188) cached and live column is a parking column (stages.IsParkingColumn, as for promote); to is also a parking column — a configured Unmanaged stage, or the legacy Backlog when no Backlog stage is configured — that exists as a board option, and differs from the current (cached and live) column; project ID, item ID and status field are known. Pipeline, holding and cleanup columns are neither a source nor a destination. |
one UpdateProjectItemStatus via moveItemToStatus; no label write (a paused, blocked, cruise or yolo item moves like any other and keeps its labels) and no auto-advance note, since nothing is dispatched for an unmanaged column |
set_autonomy |
mode ∈ {cruise,yolo,none}; the labels are read live (FetchLabels; a failed read refuses) and if they already equal the mode, nothing is written |
add the wanted label first, then remove the other (none: yolo then cruise), so a partial failure leaves the conservative state |
revalidate |
cached and live column is Validate; no fabrik:revalidate; no fabrik:paused / fabrik:awaiting-input, checked on the cache and again on a live FetchLabels read that fails closed (handleRevalidateLabel clears both, which would be a label-based pause lift) |
add fabrik:revalidate (an in-flight Validate worker defers it to a later poll; the result notes this) |
clear_claude_limit |
claudeUsageLimitedUntilTime(now) reports an active usage-limit hold; no clear request already pending |
add fabrik:clear-claude-limit to the chosen item: open, non-PR, on the board; one carrying fabrik:claude-limit first, then the lowest number |
Promote refusal shape (#2113). Promote checks the source column (cached, then live) before it checks to. A refusal because the item is not in a parking column carries the current state but no valid_targets — the move is impossible, so a target list would contradict it. valid_targets is set only while the item is established as parked (a missing or invalid to); the stale-cache (confirmed live) refusal omits it too. The pipeline-column refusal also names fabrik_advance as the way to move a completed stage forward (#2126); the unknown-column and unparked refusals do not.
Move parked (#2188, ADR 2188). MoveParked is Promote with the destination restricted to parking columns, as its own method so that its name does not suggest work starts. The refusal order is: board metadata; the source column (cached; a pipeline, holding, cleanup or unconfigured column is refused, with no valid_targets); to (missing, not a parking destination, a parking stage with no board option, or no other parking column at all); to equal to the current column; the live column (a read error refuses; the live column is judged by the same predicate and the same-column check is re-run on it). valid_targets lists the parking destinations other than the current column and is set only while the item is established as parked. parkingDestinations decides membership with stages.IsParkingColumn plus the board-option check, so the legacy-Backlog rule is not redefined. A refusal writes nothing. tools/list is static, as for the other actions; a daemon without an Actor does not advertise move_parked in hello, and the shim reports an older daemon as not supporting overseer actions.
Advance (#2126, ADR 2126). Advance is the overseer’s version of the move a stage with auto-advance makes when it completes; it exists for stages that wait for a human (Specify, typically) and moves the item one stage without changing its autonomy. The refusal order is fixed so each case has one reason: board metadata; the column (parking column → fabrik_promote, unknown column, cleanup, holding, and “the move out of this stage is the engine’s landing” for Validate); the next stage’s board column; a worker (in memory, so it is checked before any live read); the live column and labels (the live column decides which stage is current, and a stale cache that already shows the next stage refuses as “not complete”, so two requests can never move the item two stages); completion; pause; block; engine-owned wait labels; dependencies; feedback; the review wait. Every gate is a read-only core shared with the engine — openDependencies (extracted from checkDependencies), feedbackGateBlocks, and reviewGateHold (extracted from reviewGateBlocksLanding, which now wraps it and applies fabrik:awaiting-review; it shares the pure verdict helpers with checkReviewGate but not its timeout, bot re-prompt ladder or pause — those stay with the engine’s catch-up gate, so a stuck review wait ends in the engine’s own pause and the advance only ever fails closed) — because stage:<X>:complete alone does not cover a review wait (for a wait_for_reviews stage it is added before the reviewers are recorded as outstanding). A refused advance writes nothing to GitHub; the only side effect of the refusal path is feedbackGateBlocks’ in-memory feedback-pending cooldown, the same one the engine’s own held advance records. advanceToNextStage reads e.statusField without the lock (as its other callers do); the project ID comes from the board cache, or the last polled board (Engine.boardProjectID) when none is wired. An item carrying fabrik:cruise or fabrik:yolo is not refused: the result notes it and the label is untouched. A “queued” item here means a worker handle (ItemState.Worker is set before the dispatch goroutine starts); retry cooldowns are not refusal conditions.
Attribution. The requester is the --subscriber name the shim was launched with; the shim refuses an action without it and the daemon re-validates it. It is reduced to [A-Za-z0-9._:/-] and truncated to 64 characters before it is rendered. The socket is unauthenticated (mode 0600), so the name is self-asserted: an audit aid, not authorisation. Since #2123 (ADR 2123) promote’s board move carries the requester too (moveItemToStatus → CacheImpl.UpdateItemStatusBy, and the projects_v2_item echo entry via RegisterEchoIfSubscribedVia), so the resulting column-changed event reads actor=overseer, via=<subscriber> (§7.14); move_parked’s board move does the same (one column-changed, actor overseer).
Audit comment and the comment-processing lifecycle (§2.2). The comment is 🏭 **Fabrik — overseer action: <action>**\n\n<what happened>; requested by subscriber
No pause lift, no free-form comment (R2). No method removes fabrik:paused or fabrik:awaiting-input and none posts a comment other than the audit comment; a test pins both the method set and the source of overseer_actions.go, overseer_advance.go and overseer_park.go (the last also writes no label). A pause is lifted only by resumeAuthorised (§4.4, ADR-1813).
Ordering and failure. Write first, then comment. A failed write returns the error and posts no comment (a multi-label set_autonomy names the writes that did land). A failed comment leaves the action in place and reports audit_comment: failed. The action’s own label write reaches the channel observer like any engine write-through and surfaces as a label-applied / label-removed event with actor overseer and via set to the requester’s subscriber name (#2125); the engine’s later removal of a one-shot command label (fabrik:revalidate, fabrik:clear-claude-limit) is its own write and reads fabrik.
7.16 Specify Spec Persistence (persist_spec, ADR 2034)
A stage with persist_spec: true (the default Specify config) has the engine project its canonical issue-body update to specs/<issue>-<slug>/spec.md in the issue’s worktree and commit that one file. The stage stays read_only: true — Claude writes nothing and gains no tool permission — so the stash/restore, the commitWIP skip and the worktree-boundary layers are unchanged. docs/stage-lifecycle.md “Read-Only Stage Stashing” covers the per-invocation view.
Where it runs. persistSpec (engine/spec_persist.go) is called immediately after a successful or failed UpdateIssueBody from the same extractUpdatedBody string, on both entry points:
| Path | Site | Push |
|---|---|---|
| Stage dispatch | finalizeStageOutcome, after the issue-body update (post stash-pop) |
the existing post-run pushBranchUnlessQueued (runs after any invocation where Claude ran) |
| Comment review | publishCommentOutput, after the issue-body update |
pushBranchUnlessQueued, only when persistSpec created a commit — processComments has no push of its own |
Because it keys on the body update rather than on completed, a round ending in FABRIK_BLOCKED_ON_INPUT persists the spec too. A round with no FABRIK_ISSUE_UPDATE block, or one that emits FABRIK_NO_WORK_NEEDED, writes nothing.
Rules.
- Slug lock. If
specs/<issue>-*/already exists in the worktree (lexically first usable match), it is reused. A directory is usable only if itsspec.mdis absent, exists only in the worktree (a round that wrote it and failed to commit), or was added by adocs(spec):commit (specOwnedByFabrik); a repo’s own hand-writtenspecs/<issue>-*/spec.mdsharing the issue number is skipped and never overwritten. Otherwise the slug is derived once fromitem.Title(lowercase ASCII alphanumerics, other runs collapsed to-, ≤ 50 characters cut on a-boundary,specwhen nothing remains). The filesystem is the only restart-safe lock (itemstateis in-memory); the directory is never renamed. If the derived path itself holds a spec Fabrik did not write, the round logs a warning and persists nothing. - Projection.
## Open Questions— the heading (with or without trailing hint text such as the template’s*(only if unresolved questions remain)*) through the next##heading or EOF — is stripped; the rest is written verbatim with one trailing newline. The file is never read back as input. - Commit.
git add -- <path>thengit commit -m "docs(spec): add|update <path> for #N" -- <path>; the pathspec scopes the commit so other dirty or staged worktree state is never captured. A projection that matches the file as committed in HEAD commits nothing (no empty commits); one that matches the file on disk but not HEAD — a previous round wrote it and then failed to commit — is committed on the next round, so a failed commit is retried rather than silently lost. - Failure. Every error is logged (
[#N warn]) and non-fatal; the issue body is already updated and remains canonical. - #921 interaction.
commitsAheadOfBaseignores commits touching only the item’s own spec (§6.7.1), so a delegated coordinator still self-heals to Done.
Upgrade behavior. persist_spec is opt-in per stage YAML. A specify.yaml predating it silently writes no file (no denial, no pause); startup drift detection reports the missing key (it is not a no-op default) and fabrik refresh-stages --apply adds it.
7.17 Display-Only Status-Line Field (#2048, ADR 2048)
The engine projects a one-line description of what an item is doing or waiting on into an optional ProjectV2 text field (project_fields.status_line, default name Fabrik; off disables). The field is display-only and is not state. No handler, gate, settle scan or cache reads it back — not from GitHub, not from the Store, not from the writer’s own record of what it wrote. Labels and the item store remain the only state; an operator editing the field by hand changes nothing the engine does.
Absent, mismatched and failed lookups (#2218, ADR 2218). fetchProjectFieldID (github/textfield.go) treats GitHub’s GraphQL error Could not resolve to a Unions::ProjectV2FieldConfiguration with the name … (on the field path, via the typed *github.GraphQLError) as absent, so a missing field reaches the existing one-line unavailable notice with no retry; any other error stays an error. A field of the name with another type returns *github.FieldTypeMismatchError. Each displayFieldState records its outcome (found / absent / mismatch / off), and resolveDisplayFields (called at startup and every poll) brings the warnings panel in line: one display_field:missing entry listing the enabled fields the board lacks, with FixAction fabrik_command (repair-board --apply) — omitted on a user-owned board, which repair-board refuses; one display_field:mismatch:<label> entry per wrong-type field, never with a FixAction; and display_field_lookup:<label> for a genuine lookup failure, reported once per outage and resolved by a later success or by off. warnings.Reconcile clears stale display_field entries only when every enabled field has an answer; an unresolved field never clears or is named as missing. An absent field is final for the process, so the missing entry clears at the next start after the fields are created, or when the missing ones are set to off.
Writer. engine/status_line.go: setStatusLine(item, line) / clearStatusLine(item) are the only entry points (setMembersStatusLine for a train’s members, withStatusLine for a bracketed activity such as comment review). The wording lives in engine/status_line_text.go.
- Write-on-change. An in-memory
issueKey → last line writtenmap (own mutex, per-item write lock) skips a write when the line equals the last one written; the record is updated only after a successful write, so a failed write is retried at the next transition. It is lost on restart: the first transition of an item afterwards may rewrite an identical line once, and a Done item with no record gets one blind clear. - Never per turn or per poll. Calls sit at the engine’s existing transitions only (below).
- Truncation. At most 60 characters (runes), ending in
…. - Missing / off. The field is resolved once (startup
checkStageColumnAlignment, or lazily from the poll loop) with a TEXT-only lookup. If it is absent, not a text field, or the feature is off, every call is a silent no-op and exactly one[startup] status-line field unavailable …line is logged. A field created later, or renamed, needs a restart. The engine never creates the field at runtime;fabrik init --create-boardandfabrik repair-board --applydo (#2110, ADR 2110), create-only and report-only on a type mismatch. - Failure. A failed write is logged (
[#N status-line]) and swallowed — it never alters, retries-in-a-blocking-way or pauses the transition that triggered it. - No self-write anchor. A successful write does not apply
itemstate.SelfWriteObserved(§7.9, #2180): the board and probe parsers already discount the project-item bump a display-field write causes, so nothing needs suppressing and anchoring the baseline would only hide a concurrent external change.
Transitions that write.
| Line | Hook |
|---|---|
queued · batch of N |
end of prepareTrainWorker (fresh formation) |
queued |
advanceToQueued; a trial abandoned (pending, infra, cancelled, assemble error); a member ejected but staying Queued; a dissolved batch; a catch-up decided for this poll |
deferred: overlaps #M |
admitByOverlap |
trial · resolving conflicts |
assembleTrialBranch, when a member’s merge conflicts (no trial PR exists yet, so no number) |
trial #<PR> · CI running |
assembleAndValidateInner, after the draft CI PR is created (PR number = the draft CI PR) |
bisecting · step i of n |
bisect, before each sub-trial (i = the validation about to run, n = the cost cap; sub-trials write no trial line of their own) |
catch-up · CI on PR #N |
trySingletonCatchUp, while waitMemberCI waits on the member’s own PR |
landing |
landMergeTrainBatch, landSingleton, the singleton fast path |
<Stage> · running |
acquireLockAndVerify, once the in-progress label is set (not for the cleanup stage) |
<Stage> · comment review |
processCommentsClassified, restored to the previous line afterwards unless something newer was written |
blocked by #N |
checkDependencies, on the first-time block |
paused: <reason> |
pauseIssue (reason = the pause comment’s header), blockOnInput, the merge-train pause paths, the settle escalations |
claude-limit until HH:MM |
handleUsageLimitExit, for the item whose invocation hit the limit only (daemon local time) |
awaiting CI on PR #N |
finalizeStageOutcome, when fabrik:awaiting-ci is first applied |
awaiting review |
the review gate (engine/reviews.go), when fabrik:awaiting-review is first applied |
Cleared by advanceToNextStage (merge-train landings, ordinary merge, and every stage-to-stage advance), by the two Done moves that bypass it (the no-work-needed settle and the closed-item advance), by rerouteQueuedMemberOffHolding, when fabrik:blocked is lifted, and by the stage worker’s release when its <Stage> · running line is still showing (clearStatusLineIfShowing: the stage ended into a state with no line of its own, so the card must not keep claiming to be running). Lines can still go stale where no transition fires (a crash mid-phase); they are overwritten by the item’s next transition or Done and are not repaired by a read-back.
No webhook echo is registered for a status-line write. RegisterEchoIfSubscribed("projects_v2_item", "edited", <ItemID>) has no field discriminator, so a registration would collide with Status-move registrations, and a delivery that never arrives would inflate the ADR-042 miss counter.
Cache and drift (R5). Three places must treat a display-field write as a no-op:
- Webhook.
applyProjectsV2ItemDelta(boardcache/delta.go) still runs the echo match first, then returns for any edit whosefield_typeis notsingle_select, before the Store is touched: no Store change (so no wake), nolocalDeltaAtstamp (so no invalidation), no drift.changes.field_value.tois decoded asjson.RawMessagebecause its type depends on the field type; a text edit’s value never causes an unmarshal error. updatedAt. A field write is assumed to bump the project item’supdatedAt(unverified for text fields). The board and probe queries therefore select the display field’s ownupdatedAt(statusLine: fieldValueByName(...) @include(if: $withStatusLine), set throughgithub.Client.SetStatusLineField), and the parsers drop the project-item contribution toUpdatedAt/EffectiveUpdatedAtwhen it is not after that value (2 s tolerance). The probe,LightReconcileand the store all read that one parsed value, so none count a display write as drift or a reason to deep-fetch. Status is compared directly, so a real Status move is still seen even if a display write shortly follows it. With the feature off the selection is not sent and parsing is unchanged.- Project-level gate.
project.updatedAtmay still advance, so the layer-2 status batch can run once per write burst; it wakes nothing.
Neutralisation seams: CacheImpl.SetTextEditSuppressionDisabledForTest (boardcache) and leaving SetStatusLineField unset (the discount); the tests that pin the behaviour are shown to fail without them.
7.17.1 Last activity and Last run fields (#2049, ADR 2049)
Two more display-only fields share the writer: Last activity (ProjectV2 date, project_fields.last_activity, default Last activity) and Last run (text, project_fields.last_run, default Last run); off disables each independently. They follow every rule above — display-only and never read back (not from GitHub, the Store, or the writer’s own record), write-on-change only, a missing / wrong-typed / off field is a silent no-op with one [startup] last-activity|last-run field unavailable … line, a failed write is logged and swallowed, a successful write applies no SelfWriteObserved (the parsers discount its updatedAt bump, #2180), and no webhook echo is registered. No new state is introduced: no ItemState field, no persistence, no counter; the only memory is the per-field skip-unchanged record (displayFieldState.last), forgotten on restart.
Writer. engine/status_line.go is generalised to a displayFieldState per field (the status line, Engine.lastActivity, Engine.lastRun), each with its own lookup, startup line, lookup-retry back-off and per-item write lock, and a displayFieldSpec carrying the field kind (text or date). engine/run_fields.go holds the two entry points and engine/run_fields_text.go the wording (lastActivityDate, lastRunLine, formatRunDuration).
Hooks — call sites, not a subscription. The values come from fields the TUI events also carry, but the writer is called at the sites that hold them rather than from the InvocationObserver / event channel: the observer would also see the Done-cleanup InvocationRecorded (not a run) and reads the board column at observation time rather than the stage that ran.
| Write | Hook |
|---|---|
Last activity ← UTC date of the start instant |
processItem, after acquireLockAndVerify succeeds (beside the JobStartedEvent); processCommentsClassified, beside its JobStartedEvent |
Last activity ← UTC date of the completion instant, Last run ← lastRunLine |
finalizeStageOutcome, beside InvocationRecorded; processCommentsClassified, beside its InvocationRecorded |
Never hooked: merge-train batch job events (repo-level, IssueNumber 0), the synthetic Skipped completions, TurnProgressEvent (per turn), and the Done-cleanup record. Nothing clears either field at Done.
Value. Last activity is the UTC YYYY-MM-DD of StartedAt / the completion time (both time.Now, the same source the events use). Last run is <Stage>[ · comment review] · <outcome>[ · <used>/<budget> turns][ · <duration>]; outcome precedence is turn-limited, blocked on input, completed, failed (errored, not completed), else incomplete; turns appear for completed and turn-limited runs, duration for completed runs only; capped at 60 characters like the status line.
Cache and drift. The same three layers as the status line, extended to all three fields: the webhook guard already ignores any non-single_select edit (a date edit has field_type: "date"; pinned by TestDeltaProjectsV2ItemDateEditIsNoOp); the board and probe queries additionally select lastActivity (ProjectV2ItemFieldDateValue { updatedAt }) and lastRun (ProjectV2ItemFieldTextValue { updatedAt }) by alias, and projectItemUpdatedAt discounts the project item’s updatedAt against the latest present display-field value updatedAt (2 s tolerance, nil values ignored). These fields are never cleared, so every bump of theirs has a value node to compare against (the status-line clear at Done remains the known uncovered bump). Whether GitHub bumps the item updatedAt for a date write is unverified; the sim models the conservative case.
7.17.2 Train phase hook, live row and episode outcomes (#2050, ADR 2050)
The merge-train TUI row and the board status line are two projections of one train, driven by one hook so they cannot disagree. Presentation only: no train decision, ordering or timing changes, and the board’s lines, dedupe and count are byte-identical to §7.17.
Hook. e.noteTrainPhase(ep, repoKey, members, phase) (engine/train_phase.go) is the single “train phase changed” call. Every merge-train phase site — prepareTrainWorker (admitted), the loop’s assembly, assembleTrialBranch (conflict), assembleAndValidateInner (trial CI), bisect, the three landing sites, landOneAtATime, trySingletonCatchUp, acquireTrainSlotEp (slot wait) and the worker-exit releases — calls it exactly once per transition, and it (1) writes the board line through the unchanged setMembersStatusLine builders (skipped for phases with no board text: assembling, waiting for slot, one-at-a-time), (2) records the phase and its start time on the episode, then (3) emits one tui.TrainRowEvent through emitStructural (never the droppable emit, never under the episode lock). TestTrainPhaseSites_GoThroughTheHook forbids the phase builders (statusLineTrial/Bisect/Landing/CatchUp/Queued) in engine/merge_train*.go outside comments, and TestMergeTrainTUI_BoardAndRowSeeSamePhaseSequence shows the TUI rows equal the hook records one to one and the board writes equal their board projection. Sub-trials under bisection stay quiet (quietTrialLines) on both projections.
| TUI phase | Board line |
|---|---|
assembling |
queued · batch of N (once, at batch formation); none on later re-forms |
waiting for slot |
none; the previous phase is restored once the slot is obtained |
resolving conflicts on #N |
trial · resolving conflicts |
trial CI #NNNN |
trial #NNNN · CI running |
bisecting (step n) |
bisecting · step n of cap |
landing |
landing |
one-at-a-time |
none (each singleton’s own trial writes its lines) |
catching up #N |
catch-up · CI on PR #N |
released |
queued |
Episode. trainEpisode (engine/train_episode.go) is created per merge-train run (one (repo, base) partition; since #2051 a run outlives its worker goroutine, and an adopted run rebuilds its episode — phase and phase start restored from the persisted record — so the row and the single History entry span the whole run), reachable as state.episode and trialParams.episode, nil-safe, with its own lock. It holds the original batch size, the held members, ejected and deferred members with short reasons, the current phase and its start, and the outcome facts. ejectMember is untouched (it also runs on the poll goroutine); ejections are recorded at worker-side call sites (noteTrainEjected / noteTrainDeferred), which refresh the row title and keep the phase clock. The live title is [base] N of M: #a #b (ejected #x) (deferred #y), lists capped at three plus +k.
Outcomes. Sites only record facts; the single tui.JobCompletedEvent (Skipped: false, Completed: true, Outcome, Detail, Success) is emitted when the run finishes (trainRun.finish, or directly when prepareTrainWorker fails — every exit) — exactly one History entry per episode, never zero, never two. resolve() applies a fixed precedence: abandoned (nothing landed and a fault/abandon cause was recorded; Success=false) > one-at-a-time > red → bisected > landed > ejected at assembly / ejected > dissolved (the loop ran out of members; nothing to land) > catch-all ended — nothing landed (Success=false). Success is true for everything except abandoned and the catch-all. Ejections and deferrals are always appended to the detail. “Landed” members are those the integration PR claims (and members already Done on replay), not survivors; the engine’s own main-moved dissolveBatch is reported as abandoned (cause: base moved) so dissolved keeps its “nothing to land” meaning. A restart mid-train produces no entry for the interrupted run.
Neutralisation. SetTrainOutcomeNeutralisedForTest restores the blanket Skipped: true completion; TestMergeTrainTUI_OutcomeTestFailsWhenNeutralised shows the per-outcome checks fail without the real emission.
Sibling (repo, base) partitions still share the repo-keyed row (ADR 1661 caveat, display-only); each partition’s episode emits its own History entry.
7.18 Stage Skill References (#2137, ADR 2137; runtime load failure #2138, ADR 2138)
Stage YAML names a skill as <plugin>:<skill> (split on the first colon, plugin.Split); the engine resolves each reference against the plugins actually loaded for the worker and the prompt carries the fully-qualified name.
Names. plugin.NamespaceSep (:) and plugin.BasePluginName (fabrik-stages since #2140, ADR 2140; it was fabrik) are each defined once (plugin/names.go), beside plugin.LegacyBasePluginName (fabrik), plugin.PMPluginID (fabrik@fabrik) and the single pre-rename table plugin.LegacyRef (fabrik:fabrik-X and bare fabrik-X → fabrik-stages:X, the twelve known names only). stages.loadOne rejects a malformed reference (empty plugin or skill half, whitespace) at load; whether the plugin exists is decided by the engine.
Catalogue. engine/skill_catalogue.go builds an immutable snapshot from the same directory list that becomes --plugin-dir (claudePluginDirs, §7 ADR 2136): per plugin, the manifest name (else the directory basename) and its skills, found as Claude Code finds them — skills/<dir>/SKILL.md, directories added by the manifest skills key (string or array, "." allowed, paths escaping the plugin root ignored), and a root SKILL.md as a single skill when there is no skills/ and no skills key. The skill identity is the directory name; commands/ entries are not skills. The snapshot lives on Engine.skillResolver (built in New() after resolvePluginDirs succeeds; nil on NewWithDeps engines) and is rebuilt once, under a mutex, on an unresolved lookup so a plugin edited or upgraded while the daemon runs is picked up.
Resolution (engine/skill_resolve.go, resolveStageSkills). A qualified reference resolves when its plugin is loaded and has the skill. A bare reference resolves to the loaded plugin that provides it — the plugin named plugin.BasePluginName if it does, otherwise the first in load order — and is flagged deprecated, naming any other providers. Anything else is unresolved. Legacy fallback (#2140, FR-004): the as-is lookup always runs first, so a customised, not yet migrated base plugin still named fabrik keeps its own fabrik-review; only on a miss is plugin.LegacyRef applied and the mapped name resolved, flagged deprecated (the resolver and fabrik upgrade share the one table, so they cannot disagree). An empty catalogue (no loaded plugin directory) passes every reference through unchanged. resolveStageSkills(stage, keys...) returns a copy of the stage carrying the resolved names (the configured stage is shared by concurrent workers and is never mutated), so buildPrompt and buildCommentReviewPrompt emit exactly the registered name.
Gates. Two choke points, each before any worker is spawned or reinvoke cycle spent:
| Path | Where | Key resolved |
|---|---|---|
| Stage dispatch | runInvocationWithExtension, after the suspension and apiKeyHelper gates |
skill |
| Comment review, CI-fix, rebase, review-reinvoke | processCommentsClassified, right after the suspension gate (stageVariant has already swapped ci_fix_skill/rebase_skill into comment_skill) |
comment_skill |
The merge-train conflict resolver’s stage has no skill keys and skips resolution.
Unresolved reference — a self-healing did-not-run. handleSkillUnresolved (engine/skill_unresolved.go): on the stage path StageAttempted is recorded (the dispatch cooldown applies) but StageRetryIncremented never is, and neither stage:<name>:failed nor fabrik:paused is applied; on the comment path the call returns didNotRunSkillUnresolved (so the reinvoke cycle charge is refunded, #1812) with a nil error (no comment breaker is fed). One comment naming the reference, the plugins searched and their paths is posted and fabrik:skill-unresolved applied, both gated on the label’s absence. The label is cleared by finalizeStageOutcome on the next invocation that reaches Claude, and clearSkillUnresolved removes the panel entry when a reference resolves again. The next cooldown-bounded dispatch re-resolves, so fixing the YAML or the plugin needs no human comment (the reason it is not a pause).
Runtime load failure — a pause, never a completion (#2138, ADR 2138). Pre-flight resolution cannot see a skill that is on disk but that Claude Code fails to register or load in the worker; a worker without its skill has no instructions, yet still prints FABRIK_STAGE_COMPLETE. runClaude therefore takes the invocation’s own configured skill (stage.Skill for stage dispatch, stage.CommentSkill for the comment/CI-fix/rebase paths) and, after interpretClaudeResult, calls detectSkillLoadFailure (engine/skill_load_detect.go) over the raw stream-json. The signal is structural only: an assistant tool_use named Skill whose input.skill matches the own skill (base names equal; when both sides are plugin-qualified the plugins must be equal; a bare side matches any plugin), and a tool_result for its tool_use_id with is_error: true. Content text is never used to decide, only quoted. It is a failure only when an own-skill call failed and none succeeded; no call, a call without a result, unparseable lines, sub-agent traffic (parent_tool_use_id set) and unrelated failing Skill calls or tools never fire, and a stage without a skill is inert. skillLoadOverride applies it to an unclassified outcome and to the marker-wins non-zero exit, so the marker no longer wins; usage-limit, api-error, tools-denied, turn-limit and resume-failure classifications keep precedence. The result is a claudeerr.SkillLoadError, handled by handleSkillLoadFailed (engine/skill_load_failed.go): one comment through pauseIssue (fabrik:paused + fabrik:awaiting-input) naming the stage, skill, the name the worker called, skillResolver.describe() (plugins loaded) and the sanitised error; a panel entry skill_load:<stage>.<key>; and EnginePaused. Unlike the pre-flight case this does not self-heal (resolution already passed), so it is a pause rather than a cooldown. It records no StageRetryIncremented and no stage:<name>:failed, and skips commitWIP and the push. On the stage path finalizeStageOutcome handles it beside skillUnresolvedError; on the comment path processCommentsClassified returns didNotRunSkillLoadFailed (cycle refunded via classifyDidNotRun, #1812; no comment breaker fed). The human lifts the pause by removing fabrik:paused and fabrik:awaiting-input; the panel entry clears on the next invocation of that stage and key that reaches Claude without the failure (clearSkillLoadFailed).
Operator surfaces (engine/skill_report.go, ADR 2135). reportStageSkills, called from Run() beside WarnUndeclaredReviewers, checks every skill-valued key of every stage and then Reconciles two panel types: skill_reference_deprecated:deprecated (LevelNotice, #2140; formerly subject bare; one entry and one log line listing every bare or pre-rename fabrik:fabrik-* reference as stage.key → qualified name, pointing at fabrik upgrade) and skill_reference:<stage>.<key> (LevelWarning; reference, reason, plugins searched with their paths). A dispatch-time failure reports the same entry.
Drift. stages/drift.go compares top-level key presence only, so a bare-vs-qualified difference is never reported as drift (pinned by TestWarnStageDrift_BareVsQualifiedSkillIsNotDrift). The embedded stage defaults are written qualified (fabrik-stages:<stage>, #2140).
7.19 Workflow Migration and local-workflows (#2139, ADR 2139)
fabrik upgrade — never the daemon — migrates a deployment to fully-qualified skill references and moves operator edits of the deployed plugin out of Fabrik’s own plugin name. It decides from what is on disk and a per-file table of shipped versions; there are no version-number checks, so a deployment that skipped releases reaches the same end state, and a second run changes nothing.
Per-file shipped table. plugin/shipped_files.json (embedded; plugin/shipped_files.go, IsShippedFile) maps each relative plugin file to every sha256 any release shipped; the current embed always counts as shipped at runtime. KnownEmbeddedVersions holds only aggregate fingerprints and cannot answer “does this one file differ from every shipped version”. The table is generated from the v* tags by tools/gen-shipped-files; scripts/cut-release.sh runs -add-current and -check beside the KnownEmbeddedVersions step and commits the file. Between releases the table need not cover the current embed (it always counts as shipped at runtime), so a PR that edits a plugin file does not touch the table.
Classification (plugin/classify.go). ClassifyDir hashes every file under a plugin tree (.installed-version excluded) and lists the unknown ones; a skill is edited when any file under skills/<name>/ is unknown (a skill only in the tree is edited). A tree is moved only when CheckPluginState reports customWorkflow and a file is unknown: a mixture of shipped versions is refreshed in place, and a dev-build tree (ADR 1297) is never moved.
Plan, then apply (cmd/upgrade_migrate.go). planMigration reads the disk and writes nothing; Pending() is true when it would change something. Skipped outright (reported, nothing moved) when FABRIK_PLUGIN_DIR is set or .fabrik/plugin is a symlink; blocked (nothing changed, the old refusal follows) when a move is needed but .fabrik/plugins/local-workflows exists — it is never overwritten, or when the manifest’s top-level name cannot be renamed safely (checked at plan time by checkManifestRenamable, before anything is moved). applyMigration runs in an order whose every intermediate state the next plan recognises: (1) rename .fabrik/plugin → .fabrik/plugins/local-workflows; (2) drop its .installed-version and change only the manifest name to plugin.LocalWorkflowsName (textual substitution of the top-level name token only — a nested author.name is never touched — verified by decoding both versions); (3) restore the base with RefreshPluginTo + WriteInstalledVersion; (4) rewrite the stage files. On a resume the edited skills are read from the existing local-workflows tree, and that directory is touched (manifest name, .installed-version) only when it is recognisably a moved base copy — its manifest still names plugin.BasePluginName, or is missing; a manifest naming any other plugin is the operator’s own, left byte-for-byte alone with a report note, and no stage is pointed at it. The base name comes only from plugin.BasePluginName.
Stage files (stages/skillrefs.go, RewriteSkillRefs). Only column-0 skill/comment_skill/ci_fix_skill/rebase_skill lines are edited, in place, so comments, quoting and CRLF survive and block-scalar text can never match; the original and result are both decoded and the only permitted difference is the rewritten values. A bare edited skill → local-workflows:<s>; a bare shipped skill → <BasePluginName>:<s>; a bare name a project plugin provides → that plugin (below); a bare name nothing provides stays bare and is reported (and does not keep the migration pending); a current qualified reference is never rewritten (if it names an edited base skill, the report names the local-workflows copy). Pre-rename forms (#2140, ADR 2140): fabrik:fabrik-X and bare fabrik-X go through plugin.LegacyRef — an unedited skill becomes fabrik-stages:X; an edited one becomes local-workflows:fabrik-X (the moved copy keeps its old skill names) when a usable local copy exists, otherwise it is left alone with a note, never silently pointed at the shipped copy. local-workflows:* and other plugins’ references are untouched. Project plugins (#2203, ADR 2140 amendment): a bare fabrik-X (and any bare non-shipped name) is first looked up in the project-plugin set (cmd/upgrade_projectplugins.go), the plugins the daemon would load besides the base: the operator’s own local-workflows (manifest naming something other than local-workflows/fabrik/fabrik-stages), then the plugin_dirs extras in order — FABRIK_PLUGIN_DIRS if set, else plugin_dirs in <dir>/.fabrik/config.yaml (.env is loaded first, as the daemon does, an already-set variable winning; there is no --plugin-dirs flag). Name and skill set come from engine.DescribePlugin (the resolver’s own discoverPlugin), the first provider in load order wins (as resolveAsIs’s providers[0] once the base is excluded) and a report note names it and any others. The base under either name, an extra named local-workflows, and the moved/about-to-be-moved copy of the base are never project plugins, so edited skills still go to local-workflows:… and unedited ones to fabrik-stages:X unless a project plugin provides them. An unreadable extra, a missing manifest or a duplicate name is skipped with a note, never an error. The edited-skill check runs first: a base skill the operator edited goes to local-workflows:… even when a project plugin also provides the name (the edit lives in the moved copy, which loads ahead of the extras; pinned by TestProjectPlugin_EditedBaseSkillOutranksExtra). The qualified pre-rename form fabrik:fabrik-X is not looked up. projectPluginLookupDisabledForTest restores the always-base rewrite. The “is this a moved base copy” manifest checks accept both LegacyBasePluginName and BasePluginName. Pruning: RefreshPluginTo (used by the base restore, the re-exec auto-refresh and fabrik init) also deletes files that are byte-identical to some shipped version (IsShippedFile) but absent from the current embed, then empty directories, so the rename leaves no duplicate skills/fabrik-*; edited files and .installed-version are never removed. Worker exclusion: buildClaudeArgs adds --settings {"enabledPlugins":{"fabrik@fabrik":false}} to every worker so the PM plugin never loads (docs/stage-lifecycle.md).
Engine. resolvePluginDirs loads <fabrikDir>/.fabrik/plugins/local-workflows right after the base and before plugin_dirs, through the same validation (readable manifest, unique name, path de-duplication); a copy whose manifest still says fabrik is refused loudly. The dev-build PostBuildHook runs fabrik upgrade --no-migrate so unattended upgrades never move files or edit tracked YAML; --no-migrate keeps the pre-#2139 refusal exactly.
Startup. After the plugin state check, cmd/root.go computes the plan (skipped under --plugin-dir) and reportPluginState records plugin_state:migration_pending through warnings.Reconcile; it clears on the first start after the migration ran. The daemon changes nothing.
8. Invalid / Unexpected States
The engine handles unexpected label combinations (from manual human manipulation) through its guard chain. The behavior is defined by the order of checks in itemMayNeedWork(), itemNeedsWork(), and processItem().
8.1 Guard Chain Order in processItem()
Guards are checked in this order. The first matching guard determines behavior:
| Priority | Guard | Check | Behavior |
|---|---|---|---|
| 1 | No matching stage | FindStage(stages, item.Status) == nil |
Skip (return nil) |
| 2 | Repo not ready | ensureRepoReady() fails |
Skip (ErrSkipItem) or return error |
| 3 | Locked by other user | fabrik:locked:<other> present |
Skip with log |
| 4 | Editing | fabrik:editing present |
Skip with log (defense-in-depth; primary gate is in itemNeedsWork) |
| 5 | Awaiting input + comment | isAwaitingInput() + new human comments created at/after the pause (resumeAuthorised()) |
Unblock → comment processing |
| 6 | Awaiting input, no comment | isAwaitingInput() |
Skip with log |
| 7 | Paused + comment | fabrik:paused + new human comments created at/after the pause (resumeAuthorised()) |
Unpause → fall through |
| 8 | Paused, no comment | fabrik:paused |
Skip with log |
| 9 | Dependencies blocked | checkDependencies() returns true |
Skip (label + comment handled by checkDependencies) |
| 10 | Cleanup stage | stage.CleanupWorktree |
Remove worktree, add complete label |
| 11 | Failed label + unpause detection | stage:<X>:failed present or snap.PausedByEngine(stage.Name) |
clearFailedStage() then continue |
| 12 | New comments | findNewComments() non-empty |
processComments() |
| 13 | PR item | item.IsPR |
Skip (PRs only support comments) |
| 14 | Stage complete | stage:<X>:complete present |
Skip |
| 15 | Cooldown active | snap.LastAttemptAt(stage.Name) within cooldown window |
Skip |
| 16 | (all guards pass) | — | Acquire lock → invoke Claude |
8.2 Notable Unexpected Scenarios
fabrik:editing without active comment processing:
If fabrik:editing is left orphaned by a prior crash (no active processComments goroutine), the engine skips the item — first at the itemNeedsWork pre-dispatch gate (preventing goroutine launch), then at guard 4 in processItem (defense-in-depth). On restart, runStartupCleanup() automatically removes stale fabrik:editing labels from items with no active Worker — the same startup self-healing mechanism that handles fabrik:locked:<user>. Both labels are cleaned up in parallel on restart, so a crashed Fabrik instance leaves no permanent stuck state. If a human manually applies fabrik:editing, the label must be manually removed for processing to resume (startup cleanup only runs once per restart and only for items with no active Worker).
fabrik:reworking:<Stage> without active comment processing (#1802):
If fabrik:reworking:<Stage> is left orphaned by a prior crash mid-rework, the item’s completion claim may be either still cleared or already durably resolved, depending on exactly where the crash landed — finalizeComments only removes this marker after handleStageComplete has made its own completion decision on a completing exit, not before (§4.2 step 8b/11). On restart, runStartupCleanup()’s third pass (mirroring the fabrik:editing pass immediately above it) reads the reworked stage’s name directly out of the marker label’s own suffix — <Stage> in fabrik:reworking:<Stage> — never from the item’s current board Status. This is a deliberate correction (a Pruefer review finding, #1802): a completing rework that also auto-advances the board (yolo/cruise/auto_advance: true) moves Status to the next stage synchronously inside handleStageComplete, before finalizeComments’s deferred removal of this marker ever runs. A crash in that window would leave Status pointing at a stage that never actually ran; resolving the reworked stage from Status there would falsely stamp stage:<NextStage>:complete onto it, which itemNeedsWork then reads as “already done,” silently and permanently skipping that stage’s real dispatch. Reading the stage name from the label itself avoids this entirely, regardless of how far Status has since moved. Using that resolved stage name, the pass first checks whether stage:<Stage>:complete or fabrik:awaiting-ci is already present: if either is found, the rework already settled correctly before the crash and only the stale marker is removed; otherwise it restores stage:<Stage>:complete directly (the crash landed before any completion decision was recorded — indistinguishable from, and treated the same as, a non-completing exit) and then removes fabrik:reworking:<Stage>. Either way this only runs for an item with no active Worker. If the marker is malformed (no stage name encoded in the suffix), it is left in place (logged loudly) rather than silently dropping the only remaining signal that a restore is owed — see ADR-1802.
stage:<X>:complete without board column advancement:
The item is skipped by guard 14 in processItem(). The catch-up loop will attempt to advance it if yolo/cruise/autoAdvance is active. Without auto-advance, it waits for a human to move the board column.
fabrik:paused on a complete item:
The catch-up loop checks fabrik:paused and skips paused items. The item will not be advanced until unpaused.
fabrik:awaiting-review without stage:<X>:complete:
The catch-up loop only processes items with the complete label, so fabrik:awaiting-review alone has no effect in the catch-up path. In processItem(), the label is not checked — it’s only relevant in the catch-up loop.
stage:<X>:failed without fabrik:paused:
processItem() guard 11 detects this as an “unpause” scenario — the user has already removed fabrik:paused, so clearFailedStage() resets retry state and processing continues.
Multiple stage:<X>:in_progress labels:
No special handling. Each is independent. The engine only checks the in_progress label for the current stage’s column.
9. Concurrency Model
9.1 Semaphore
Engine.sem is a buffered channel of size MaxConcurrent (default 5). With merge_train: on there is also Engine.trainSem, a separate one-slot channel used only by the merge train (#2115, Reserved train slot); stage work never touches it. The dispatch loop, dispatchReviewReinvoke(), and dispatchCIFixReinvoke() all acquire slots from this semaphore before invoking Claude. The dispatch loop’s acquire is non-blocking (#2114, §9.9.1) — a full semaphore defers the candidate and the poll carries on; the reinvoke dispatchers acquire inside their own goroutine. The merge train takes a slot only around each of its own Claude conflict-resolution invocations (acquireTrainSlot, #2046) — never for its lifecycle (see “Merge-Train Serialization + Main-Moved Recovery”, Worker-slot scope).
9.2 Worker In-Flight Guard (formerly inFlight Map)
Engine.inFlight (sync.Map) has been removed (Fix B, issue #544). The dispatch guard is now entirely Store-backed.
The dispatch loop uses snap.Worker() != nil to detect whether a goroutine is already running for an item. WorkerEntered{Repo, Number, StageName, StartedAt} is applied synchronously before wg.Add(1) and before the goroutine is launched, so the store guard is effective from the instant the goroutine is scheduled. The reinvoke dispatchers (dispatchReviewReinvoke, dispatchCIFixReinvoke, dispatchRebaseReinvoke) follow the same pattern. WorkerExited is deferred at the top of each goroutine in all four dispatchers — main (engine/poll.go) and the three reinvoke dispatchers — so it fires on every exit path, including processItem’s many early-return guards (paused, blocked, awaiting-input, locked-by-other, stage-complete, etc.).
- Set by: dispatch loop applies
WorkerEnteredbefore goroutine launch; each reinvoke dispatcher appliesWorkerEnteredbefore goroutine launch - Cleared by: goroutine-top
defer store.Apply(WorkerExited{...})— fires on any exit path including processItem early-returns, context cancel,ensureRepoReadyfailure, panic-after-recover, or normal completion - Read by:
snap.Worker() != nil— used by all dispatch guards (main loop, reinvoke catch-up guards) and the idle display (“workers active” log). The auto-upgrade guard (engine/poll.go, the idle-period check that refuses to fire while any worker is in-flight, to avoidsyscall.Execkilling live workers) instead callsStore.HasInFlightWorker()(issue #1222) — a single authoritative check that ORssnap.Worker() != nilacross all items withStore.RepoWorkerActive()over the repo-scoped registry described below, so a merge-train worker (which has no single(Repo, Number)home) also holds the guard closed. A separate startup check fires unconditionally before the firstdoPollCycle()call — no in-flight guard is needed there because no workers exist at that point. - Repo-scoped worker liveness (issue #1222):
Store.repoWorkers(keyed"owner/repo") is a second, independent liveness registry insideitemstate.Store, for workers — currently only merge-train workers — that operate on a batch spanning multiple issue numbers rather than a single(Repo, Number).dispatchMergeTrainWorkercallsStore.EnterRepoWorker(repoKey)synchronously, immediately after itsmergeTrainInFlight.LoadOrStoreduplicate-launch claim succeeds and before the worker goroutine is launched;finishTrain— already the single ADR-067-mandated clear point formergeTrainInFlight— also callsStore.ExitRepoWorker(repoKey), so every existing early-return path that already funnels throughfinishTrainclears both registries together.Store.RepoWorkerActive(repoKey)is read bymergeTrainWorkerActive(used bysettleClosedItemsToDoneto avoid advancing a closed batch member to Done out from under a live train) and byHasInFlightWorker().mergeTrainInFlightitself is unchanged and still owns the atomic duplicate-launch claim and the richerassembling/bisecting/CIResult/prNum/trialNamesub-state (§1.3.1’s Merge-Train Landing Lifecycle) — only the liveness answer was consolidated into the Store, not the claim mechanism or the sub-state. - Idempotent: the
WorkerExitedmutation handler (internal/itemstate/store.go) returns 0 changes whenitem.Worker == nil, so a redundant defer (e.g. an innerdefer WorkerExitedinsideprocessItemfiring in addition to the goroutine-top defer) does not produce spurious observer notifications.
Version-skew watchdog (#1074): Piggybacked on the same idle+no-in-flight gate described above, Engine.checkVersionSkew() (engine/upgrade.go) runs immediately before checkAndUpgrade() whenever AutoUpgrade is enabled and idleUpgradeThreshold is reached — no new counter or threshold. It resolves the on-disk executable path (versionSkewExecutableFn → os.Executable + filepath.EvalSymlinks — engine’s own private seam, distinct from internal/selfupgrade’s executableFn used by PerformReleaseUpgrade/CheckAndRebuildDev since ADR-1196 moved the self-upgrade trigger logic to its own package), spawns <exe> --version with a 5s timeout, and compares the trimmed output to e.cfg.Version. A mismatch is a general “the code on disk has moved on from what’s running” signal — it catches a SemverGreater bug, a syscall.Exec failure after a successful binary replace, or an externally-replaced binary (e.g. a fleet sharing ~/go/bin/fabrik) uniformly, without needing a dedicated warning per cause. The mismatch is recorded as a persistent warnings.Entry (Type: "version_skew", keyed by the resolved executable path, FixAction: "shell_command" suggesting kill -HUP <pid>) via the same warnings.Record/warnings.Clear pattern as checkAllowAutoMerge (ADR-052), so it survives process restarts and surfaces in the TUI Warnings panel. A matching version clears any existing entry for that key. All failures resolving the path or running the subprocess are logged and non-fatal — the check is simply skipped for that poll.
Source-checkout staleness reporting (#1464): deliberately not piggybacked on the idle
gate. Unlike the version-skew watchdog above, Engine.checkSourceStaleness()
(engine/staleness.go) does not run inside the idle+no-in-flight branch — it is called
unconditionally from poll(), before the dispatched == 0 check, throttled internally by
its own poll-count gate (maybeCheckSourceStaleness, every stalenessCheckPollInterval = 30
polls, ≈15 minutes at the default 30s --poll interval, firing on the very first poll too).
This is the fix for the gap the version-skew watchdog and checkAndUpgrade both share: both
require idleUpgradeThreshold consecutive fully-idle polls to ever run, so a board that
dispatches work (or has a merge-train worker in flight) on every single poll never reaches
either one — the daemon can run arbitrarily stale code indefinitely with no signal. Since
checkSourceStaleness only reports (via selfupgrade.CompareDevBuild, which never calls
git pull, go build, or execFn — see its doc comment), it has no in-flight-worker
precondition to honor: there’s nothing here that would kill a live Claude invocation, unlike
checkAndUpgrade’s syscall.Exec. CompareDevBuild is the same comparison
selfupgrade.CheckAndRebuildDev uses to decide whether to actually rebuild, so the staleness
warning and the real upgrade decision are structurally unable to disagree. Applicability
mirrors checkAndUpgrade’s own fork (e.cfg.Version must start with "dev", then
CompareDevBuild’s own IsSourceCheckout gate) — a release build never calls into
selfupgrade for this check at all, so it never runs a git fetch. Recorded via the same
warnings.Record/warnings.Clear pattern as version_skew, under the single fixed key
"source_staleness" (§7.11 above); the FixAction differs depending on e.cfg.AutoUpgrade,
since kill -HUP only self-heals staleness when the startup-time checkAndUpgrade() call that
a SIGHUP re-exec re-triggers is itself enabled.
Safe-point-gating race (investigated for #1074, none found): the concern was whether a webhook-triggered concurrent poll could dispatch a new worker while a synchronous checkReleaseUpgrade() download+exec is in flight on another poll cycle. It cannot: Engine.Run()’s single for { select {...} } loop is the only caller of doPollCycle(); webhook wakes are delivered through e.wakeCh and consumed serially by that same loop, so checkAndUpgrade() (and its syscall.Exec) always runs synchronously on the main goroutine between poll cycles — never concurrently with worker dispatch from a different cycle. The in-flight check at the top of this section (Store.HasInFlightWorker(), covering both snap.Worker() != nil over e.store.All() and the repo-scoped merge-train registry) already detects workers dispatched by any prior poll cycle regardless of which cycle started them, so no additional guard was needed. This also covers the same-cycle case: dispatchMergeTrainWorker calls Store.EnterRepoWorker synchronously before returning, so the very poll cycle that just launched a merge-train worker (which dispatchCandidates does not count in its dispatched return value) no longer misreports itself as idle.
Both mutations emit WorkerChanged | WorkerLifecycleChanged (§2.14). Only WorkerLifecycleChanged is in wakeChFlags, so worker entry and exit wake the poll loop via newWakeChObserver. WorkerHeartbeat and WorkerPIDSet emit only WorkerChanged and do not feed the wake pipeline.
WorkerLifecycleChanged remains in wakeChFlags (wake channel still fires on goroutine entry/exit) but is excluded from cycleSetFlags (used by newMayNeedWorkObserver). This prevents early-return goroutine exits — e.g. from a dep-blocked item — from bypassing the cooldown gate for items that did no useful work. Non-blocked items are still re-evaluated promptly: the wake channel fires, the poll loop runs, and any item that passes itemNeedsWork is dispatched. See §9.9 (pre-dispatch label gates and wake-loop avoidance) and ADR-039.
Wake channel scope: the wakeChObserver is registered only when a wake channel is configured (TUI mode or any other configuration that calls
SetWakeCh). In headless /--notuiruns,WorkerLifecycleChangeddoes not populatemayNeedWork(it is excluded fromcycleSetFlags— see Fix B, issue #576) and does not trigger an immediate wake (no wake channel). Re-evaluation in headless runs is driven byStatusChanged/LabelsChangedevents (which ARE incycleSetFlags) or the next ticker-driven poll (withinPollSeconds). The dispatch re-evaluation guarantee is therefore “by the next scheduled poll” (≤PollSeconds) for worker-exit events in headless mode, and “within milliseconds” in TUI mode (wake channel fires onWorkerLifecycleChangedviawakeChFlags).
9.3 Worktree Mutex
Git operations that write .git/config are not concurrent-safe. WorktreeManager.mu serializes worktree creation and updates within a single repo.
9.4 Catch-Up vs Dispatch Ordering
Within a single poll() call:
- Catch-up loop runs first — processes items with
stage:<X>:completelabels for yolo/cruise advancement, review gate evaluation, and review reinvoke dispatch - Dispatch loop runs second — processes items that need stage invocations or comment processing
The advancedItems map tracks items that the catch-up loop advanced during this poll cycle. Items in advancedItems are excluded from the deferred CooldownAt["periodic-re-eval"] refresh at the end of poll(), so they appear in the next poll cycle’s cycleSet naturally (via the observer that fires when their status changes) rather than being suppressed by cooldown; their expired cooldown (if any) is consumed at admission (#2096), so the status change — not a leftover expiry — is what admits them (#544). Items dispatched by dispatchReviewReinvoke() in the catch-up loop are guarded from double-dispatch by the dispatch loop via snap.Worker() != nil (see §9.2).
9.5 Engine.mu Mutex
Engine.mu (sync.Mutex) protects in-memory state that is not covered by its own synchronization primitive: totalTokens, lastReportedCost. Critical sections are kept small — typically a single map read/write. Engine.mayNeedWork is protected by its own Engine.mayNeedWorkMu (a separate mutex) so that observer callbacks writing to mayNeedWork from any goroutine don’t contend with Engine.mu-held code paths.
Per-item engine state previously stored in Engine.mu-protected maps has been migrated to itemstate.Store (Phase 3-E and 3-F; see ADR-036). Those fields — Lock, LastTokenUsage, LastInvocationCompleted, LastInvocationBlocked, LastDeepFetchFailureAt, LinkedPR.HasHadChecks, LinkedPR.CIMergePendingSince, plus the Phase 3-F fields StageState.LastAttemptAt, StageState.Attempts, StageState.PausedByEngine, StageState.ReviewCycles, StageState.CIFixCycles, StageState.RebaseCycles, StageState.ProcessedComments, and ItemState.CooldownAt — are now read via e.store.Get(repo, n) (returning an immutable Snapshot) and written via e.store.Apply(Mutation). The Store has its own internal mutex; no Engine.mu guard is needed for these fields.
9.6 Engine Internal State (itemstate.Store)
Per-item state lives in a single shared *itemstate.Store instance. The Engine creates it (sharedStore := itemstate.NewStore(nil) in New()), assigns it to eng.store, and passes it to boardcache.NewCacheImpl. All mutations — engine-side (locks, invocations, stage state) and webhook/reconcile-side (status, labels, comments, linked-PR fields) — flow through sharedStore.Apply. NewWithDeps (test factory) constructs its own independent store because it never creates a CacheImpl.
Phase 5 F2 (issue #562): The previously separate linkedPRs map[string]*gh.PRDetails and prNumToKey map[string]string CacheImpl maps were migrated into itemstate.Store: PR detail fields (Title, State, Merged, Draft) are now stored in LinkedPRState via PRDetailsUpdated mutations, and the PR→issue reverse-index is store.prToKey (populated by PRHeadSHAUpdated / PRDetailsUpdated; queried via store.GetByPRKey).
Phase 5 F4 (issue #563): The previously separate checkRuns map[string][]gh.CheckRun CacheImpl map (check-run lists keyed by commit SHA) was migrated into itemstate.Store as pendingCheckRuns map[string][]gh.CheckRun. This pre-linkage buffer holds check runs for SHAs not yet linked to any item. When PRHeadSHAUpdated fires (establishing the SHA→item mapping), the buffer is drained into LinkedPR.CheckRuns via upsert-by-ID. FetchCheckRuns now reads from store.CheckRunsBySHA, which returns the union of pendingCheckRuns[sha] and the linked item’s LinkedPR.CheckRuns; on total miss it falls back to GitHub and populates the Store via CheckRunCompleted mutations. See §9.8 for CheckRunChanged flag semantics.
Per-SHA pruning (#958 leg 3): gh.CheckRun carries no SHA field, so LinkedPR.CheckRuns is a flat, per-item (not per-SHA) slice — CheckRunsBySHA(sha) only used sha to resolve which item to read from shaToKey, then returned that item’s entire CheckRuns accumulation regardless of which SHA was actually requested. This let a run from a prior push shadow the current SHA’s classification indefinitely on a webhook-less deployment (no check_run event ever prunes it). PRHeadSHAUpdated now clears LinkedPR.CheckRuns whenever the head SHA genuinely changes (v.SHA != item.LinkedPR.HeadSHA && item.LinkedPR.HeadSHA != "" — excludes first linkage), restoring the informal “this only reflects the current SHA” invariant without a full map[SHA][]CheckRun storage rewrite (deferred to #957 if ever needed).
Field ownership is by mutation type, not by store identity: a reader calling store.Get("owner/repo", 1) receives a Snapshot with all field groups populated regardless of which code path applied each mutation. This is the single-Store design originally proposed in ADR-036 and completed in Phase 5 F3 (issue #537).
The following fields are stored in ItemState / LinkedPRState and accessed exclusively through the store:
ItemState field |
Mutation (write) | Snapshot accessor (read) | Former Engine map |
|---|---|---|---|
Lock (LockState) |
LocalLockAcquired, LocalLockReleased |
snap.Lock() |
e.lockedIssues[iKey] |
LastTokenUsage |
InvocationRecorded{Usage: ...} |
snap.State().LastTokenUsage |
e.lastUsage[iKey] |
LastInvocationCompleted |
InvocationRecorded{Completed: ...} |
snap.State().LastInvocationCompleted |
e.lastCompleted[iKey] |
LastInvocationBlocked |
InvocationRecorded{Blocked: ...} |
snap.State().LastInvocationBlocked |
e.lastBlocked[iKey] |
LastInvocationTurnLimited |
InvocationRecorded{TurnLimited: ...} |
snap.State().LastInvocationTurnLimited |
N/A (new in #1178) |
LastDeepFetchFailureAt |
DeepFetchFailed{At: ...} (set); ItemDeepFetched (clears) |
snap.State().LastDeepFetchFailureAt |
e.deepFetchFailureTime[iKey] |
LinkedPR.CheckRuns |
CheckRunCompleted (when SHA is already in shaToKey); PRHeadSHAUpdated drain (flushes pendingCheckRuns[sha]); PRHeadSHAUpdated prune (clears the slice when the head SHA genuinely changes, not on first linkage — #958 leg 3); fallback populate in FetchCheckRuns |
snap.LinkedPR().CheckRuns |
CacheImpl: c.checkRuns[sha] (Phase 5 F4: migrated to Store in #563) |
LinkedPR.LastCIFixNoOpSHA |
CIFixNoOpRecorded{SHA} (#958 leg 2) |
snap.LastCIFixNoOpSHA() |
N/A (new in #958) |
LinkedPR.HasHadChecks |
PRChecksObserved (REST path); CheckRunCompleted and PRHeadSHAUpdated drain (webhook path) |
snap.LinkedPR().HasHadChecks |
e.prHasHadChecks[iKey] |
LinkedPR.CIMergePendingSince |
CIMergePendingStarted{At: ...} (set); CIMergePendingCleared (clear) |
snap.LinkedPR().CIMergePendingSince |
e.ciMergePendingSince[iKey] |
LinkedPR.Number, LinkedPR.HeadSHA |
PRHeadSHAUpdated{LinkedPRNum, SHA} |
snap.LinkedPR().Number, snap.LinkedPR().HeadSHA |
CacheImpl: c.prNumToKey[pk] (routing), c.linkedPRs[pk].HeadSHA (Phase 5 F2: migrated in #562) |
LinkedPR.ValidateCompletedSHA |
ValidateCompletedAtSHA{Repo, Number, SHA} (set); ValidateCompletedAtSHACleared{Repo, Number} (clear) |
snap.ValidateCompletedSHA() |
N/A (engine-internal; not persisted across restarts) |
LinkedPR.Title, LinkedPR.State, LinkedPR.Merged, LinkedPR.Draft |
PRDetailsUpdated{Title, State, Merged, Draft} |
snap.LinkedPR().Title, etc. |
CacheImpl: c.linkedPRs[pk].* (Phase 5 F2: migrated in #562) |
LinkedPR.Facts, LinkedPR.CIVerdict (#2122) |
PRFactsObserved{PRNumber, Facts} (the probe apply and settlePRFacts’ REST fallback); CIVerdictObserved{Verdict} (settlePRFacts); CIVerdict is cleared by PRHeadSHAUpdated on a genuine head change and by PRFactsObserved for a merged or closed PR. Flag PRFactsChanged, not a wake flag |
snap.LinkedPR().Facts, .CIVerdict (local API and attention only; no gate reads them) |
N/A (new in #2122) |
StageState.LastAttemptAt |
StageAttempted{StageName, At} (set); StageLastAttemptCleared (clear) |
snap.LastAttemptAt(stageName) |
e.processedSet[stageKey] (invocation timestamp) |
ItemState.CooldownAt |
CooldownRecorded{Reason, Until} |
snap.CooldownAt(reason) |
e.processedSet[stageKey] (cooldown timestamp; same key, different semantic — now split) |
StageState.Attempts |
StageRetryIncremented (increment); StageRetryCleared (reset) |
snap.Attempts(stageName) |
e.retryCount[stageKey] |
StageState.PausedByEngine |
EnginePaused (set); EngineUnpaused (clear) |
snap.PausedByEngine(stageName) |
e.pausedDueToRetries[stageKey] |
StageState.ReviewCycles |
ReviewCycleIncremented (increment); EngineCyclesCleared (reset) |
snap.ReviewCycles(stageName) |
e.reviewCycleCount[stageKey] |
StageState.CIFixCycles |
CIFixCycleIncremented (increment); EngineCyclesCleared (reset) |
snap.CIFixCycles(stageName) |
e.ciFixCycleCount[stageKey] |
StageState.RebaseCycles |
RebaseCycleIncremented (increment); EngineCyclesCleared (reset) |
snap.RebaseCycles(stageName) |
e.rebaseCycleCount[stageKey] |
StageState.DidNotRunReinvokes |
DidNotRunReinvokeRecorded (increment); DidNotRunReinvokesReset (reset on a charged reinvoke); EngineCyclesCleared (reset) |
snap.DidNotRunReinvokes(stageName) |
— |
StageState.ProcessedComments |
CommentProcessed{CommentID, At} |
snap.CommentProcessed(commentID) |
e.processedSet["…comment-ID"] |
Worker (*WorkerHandle) |
WorkerEntered{StageName, StartedAt, Kind, Trigger, Queued, QueuedSince} (placeholder, before goroutine launch; Queued for a reinvoke that holds no slot yet — #2121); LocalLockAcquired{Worker: &WorkerHandle{...}} (full details, after lock acquired — stage dispatch); WorkerSlotAcquired{At} (a queued reinvoke now holds a slot: clears Queued, re-stamps StartedAt/LastSignAt, no lock claimed — #2179/#2121); WorkerKindSet/WorkerTriggerSet/WorkerTurnsRecorded (kind, trigger, live turns — #2121); WorkerPIDSet{PID} (update PID); WorkerHeartbeat{At} (update LastSignAt); WorkerExited (clear) |
snap.Worker() |
N/A (new in Phase 3-G; WorkerEntered added in Fix B, #544) |
mayNeedWork (Phase 3-H): The map e.mayNeedWork map[string]bool (protected by e.mayNeedWorkMu) is the Phase 3-H replacement for the removed e.seenUpdatedAt map. It is populated by the mayNeedWorkObserver registered once on the shared store whenever a Change includes any wakeChFlag. All mutation types — engine-side and webhook/reconcile-side — now fire through the same observer registration. Each poll cycle drains the map into a local cycleSet — only items in the set (or with bypass conditions) proceed to deep-fetch evaluation. See section 9.8 for the full observer pattern description.
See also: ADR-036 (adrs/036-reactive-cache-single-owner.md) for the full rationale and the Phase 5 F3 addendum documenting the unification. ADR-038 (adrs/038-dual-store-observer-wiring.md) documents the historical dual-store registration design (superseded by the unification).
9.7 Worker Liveness (Heartbeat and Stale-Lock Recovery)
Phase 3-G adds a heartbeat-based liveness system that allows the engine to detect and recover from stale fabrik:locked:<user> labels left by crashed worker processes.
WorkerHandle Struct
type WorkerHandle struct {
PID int // Claude subprocess PID (0 until cmd.Start() returns)
StageName string // name of the stage being invoked
StartedAt time.Time // time LocalLockAcquired was applied
LastSignAt time.Time // time of the most recent WorkerHeartbeat mutation
}
ItemState.Worker is non-nil while a worker goroutine is in flight; nil when no worker is active. snap.Worker() returns a deep copy (nil-safe).
Heartbeat Protocol
Every Claude-spawning dispatch path starts a heartbeat goroutine at dispatch time:
| Dispatch site | File |
|---|---|
processItem() — main stage invocation |
engine/item.go |
dispatchReviewReinvoke() — review comment processing |
engine/reviews.go |
dispatchCIFixReinvoke() — CI failure re-processing |
engine/ci.go |
dispatchRebaseReinvoke() — rebase conflict re-processing |
engine/merge_gate.go |
Lifecycle per dispatch path:
This is the engine-state lifecycle (goroutine boundary). A parallel TUI-presentation lifecycle (JobStartedEvent/JobCompletedEvent) fires at the work boundary — inside processItem/processComments past all early-return guards — not at goroutine launch. See Appendix E for the full comparison.
-
Before goroutine launch:
store.Apply(WorkerEntered{StageName, StartedAt})— placeholder Worker set sosnap.Worker() != nilis true from the instant the goroutine is scheduled.WorkerEnteredemitsWorkerLifecycleChanged, which wakes the poll loop. - Inside goroutine (after lock acquired):
store.Apply(LocalLockAcquired{Worker: &WorkerHandle{StageName, StartedAt, PID: 0}})— full Worker details replace the placeholder (PID=0 until subprocess starts).processItemalso emitsJobStartedEventhere (at the lock-acquired boundary) and defersJobCompletedEvent{Skipped:true}. - A
done := make(chan struct{})is created;defer close(done)is set on the dispatch goroutine. go startHeartbeat(ctx, repo, number, done)— heartbeat goroutine starts. It appliesWorkerHeartbeat{At: time.Now()}every 30 seconds untildoneis closed orctxis cancelled.WorkerHeartbeatemits onlyWorkerChanged(notWorkerLifecycleChanged) so it does not wake the poll loop.- Claude is invoked. After
cmd.Start(),opts.OnPIDReady(pid)appliesWorkerPIDSet{PID: pid}— the Claude subprocess PID is recorded in Worker. Emits onlyWorkerChanged. - When the dispatch goroutine exits (defer):
close(done)stops the heartbeat goroutine;store.Apply(WorkerExited{})setsWorker = nil.WorkerExitedemitsWorkerLifecycleChanged, deterministically waking the poll loop.
Reinvoke registration (#2179, ADR 2179). A review/CI-fix/rebase reinvoke (dispatchReinvoke) applies WorkerEntered before the goroutine, as a queued worker, then WorkerSlotAcquired the moment it holds a semaphore slot (clearing Queued and re-stamping StartedAt/LastSignAt so the PID<=0 detector below measures from then; the detector skips a queued handle, #2121), and WorkerExited in the goroutine’s defer on every exit path. It never applies LocalLockAcquired/LocalLockReleased and never adds a fabrik:locked:<id> label: Lock mirrors that label and is therefore set only by stage dispatch (the reinvoke’s cross-instance marker is fabrik:editing; in-process exclusion is Worker() != nil). cleanupLockedIssues consequently sees only real stage locks.
No-op semantics:
WorkerHeartbeatis a no-op whenWorker == nil(race-safe: heartbeat may fire just afterWorkerExited).WorkerPIDSetis a no-op whenWorker == nil.WorkerExitedis idempotent (safe to apply when already nil).
Stale-Worker Detector (Runtime Crash Case)
startWorkerDetector(ctx) launches a background goroutine in Run() that scans for stale workers every 60 seconds.
Detection requires BOTH conditions (implemented by isWorkerStale(w, threshold)):
time.Since(Worker.LastSignAt) > WorkerStaleTimeout(heartbeat is stale), AND- Signal-0 confirms the process is dead (
kill -0 PIDreturnsESRCH)
The “both conditions” requirement prevents spurious clearing of live workers whose heartbeat goroutine is delayed under system load. If the heartbeat is stale but the PID is alive, the engine logs a warning and re-checks on the next scan cycle.
WorkerStaleTimeout (default 5 minutes) is configurable via --worker-stale-timeout <N> (minutes) or FABRIK_WORKER_STALE_TIMEOUT=N. Must be longer than heartbeatInterval × 2 (currently > 60 s).
PID<=0 timeout-based clear (issue #1303): Workers with PID <= 0 (PID not yet set, or invalid) are in the narrow window between LocalLockAcquired and cmd.Start() — signal-0 has no PID to target, so the ordinary confirmed-dead check cannot run for them at all. An unconditional skip here would let a dispatch goroutine that hangs before onPIDReady fires (e.g. stuck in ensureRepoReady, before the child process is even started) outlive its own WorkerEntered marker indefinitely — and the fabrik:locked:<user> / stage:<name>:in_progress labels it gates — permanently suppressing dispatch for the item. Instead, the detector applies WorkerStaleTimeout against Worker.StartedAt (not LastSignAt, which may never have started ticking for a worker that hung before acquireLockAndVerify started the heartbeat goroutine): once a PID<=0 worker has been running longer than WorkerStaleTimeout, it is a candidate for a timeout-based clear — not a confirmed-dead clear, since nothing here has verified the process is actually gone.
Sentinel-probe verification tier (issue #1779): Before a PID<=0 worker past the threshold is actually cleared, the detector probes for a live process carrying that worker’s deterministic sessionNameSentinel (fabrik:<owner>/<repo>#<issue>:<stage> — see “Worker Session Naming” in docs/USER_GUIDE.md/docs/stage-lifecycle.md), already passed as --name on every worker invocation. This closes the gap #1303 left open: a real claude subprocess can be alive and progressing (the reported incident: cleared after 5 minutes, ran a further 1h36m, authored five files) even though the engine’s own Worker.PID field was never updated — the two facts are unrelated once the subprocess has actually started. handleUnverifiablePIDWorker (engine/worker_liveness.go) implements this as a three-way branch, gated on claudeNameFlagSupported (a one-time, process-lifetime capability probe — if the installed claude binary predates --name support, no worker in this process could ever carry a sentinel, so the probe is skipped entirely and behavior falls straight through to the plain #1303 clear below):
| Probe outcome | Action |
|---|---|
| Sentinel found live | Mirror the PID-recorded path’s posture: log [#N worker-liveness] sentinel "…" found live … — adopting PID, signal-0 liveness now governs (or “waiting for natural exit” if no PID was recovered) and do not clear. When the probe also yields the OS PID, WorkerPIDSet{PID} adopts it into the handle and a fresh WorkerHeartbeat{At: now} is applied, so the item’s Worker.PID > 0 on the very next scan cycle and ordinary signal-0 liveness governs from then on. |
| Sentinel affirmatively not found | Clear exactly as the plain #1303 timeout path: store.Apply(WorkerExited{}) + label removal + log [#N worker-liveness] worker never reached PID assignment (started T ago) — clearing, could not verify liveness (sentinel "…" not found). |
Probe itself failed (unsupported platform, ps error, non-zero exit, timeout) |
Neither clear nor defer forever: a per-(repo, issue) consecutive-failure counter (Engine.sentinelProbeFailures, in-memory, scan-goroutine-local — not itemstate.Store state) is incremented and compared against sentinelProbeUnverifiableCycleLimit (currently 3, ~3 scan cycles ≈ 3 minutes on top of WorkerStaleTimeout). Below the bound: log sentinel probe unverifiable (…) — deferring clear and leave the worker alone. At the bound: clear, logged as clearing UNVERIFIED after N consecutive sentinel-probe failures: <err>. The counter is cleared whenever the worker leaves this state — sentinel found live, sentinel confirmed not found, cleared at the bound, or a real PID reaches the store via the ordinary onPIDReady callback. |
claudeNameFlagSupported == false |
Probe skipped entirely (never invoked); clears exactly as the plain #1303 timeout path, logged without the “(sentinel … not found)” suffix. |
The probe (probeSentinelLive, unix-only; a Windows build-tag stub always reports itself unsupported, routing Windows through the bounded-unverifiable row above — the same posture isProcessAlive’s Windows stub takes for signal-0) shells out to ps -eo pid=,args= -w -w (no shell, short timeout) and matches the sentinel as a whole argv token — never a substring — so a sentinel for issue #2966 cannot match a decoy carrying #29660.
Dispatch-time backstop (R5, issue #1779): R1/R2 above are meant to make “two workers on one worktree” unreachable, but dispatchCandidates (engine/poll.go) asserts it independently rather than relying on that alone: immediately after the existing snap.Worker() != nil in-flight guard (which only catches an item this process still thinks is in-flight), it re-checks the same sentinel for the item’s resolved stage. A Live result refuses dispatch (logged refusing dispatch: sentinel "…" still live for a cleared worker) without ever applying WorkerEntered; “not found” or a probe error both fall through to ordinary dispatch — the guard fails open on a probe error, since treating an unverifiable probe as a hard block would let a broken ps wedge dispatch for every item on every poll, a strictly worse failure mode than the rare duplicate-writer it exists to prevent. Scoped to the main stage-dispatch path only; the structurally identical snap.Worker() != nil guards in the reinvoke dispatchers (reinvoke.go, reviews.go, ci.go, merge_gate.go) do not get this check.
Unlike the scan’s one-worker-at-a-time probeSentinelLive call, dispatchCandidates fetches the live process table at most once per call (listProcessArgvFn, lazily on first need) and matches every dispatch-eligible item’s sentinel against that single snapshot in-process (matchSentinelInArgvList) — not once per item. A poll with many simultaneously dispatch-eligible items (a startup burst, or a wide cooldown-expiry window) would otherwise serialize one ps subprocess spawn per item ahead of the first dispatch in that pass.
Signal-0 liveness check (syscall.Kill(pid, 0), for workers with PID > 0 — whether from the ordinary onPIDReady callback or adopted via the sentinel probe above):
| Outcome | Action |
|---|---|
| Signal-0 fails with ESRCH (PID dead) AND heartbeat stale | store.Apply(WorkerExited{}) + RemoveLabel(fabrik:locked:<user>) + RemoveLabel(stage:<StageName>:in_progress) + log [#N worker-liveness] stale worker handle (pid=P last_sign=T) — clearing |
| Signal-0 succeeds (PID alive) with stale heartbeat | Log [#N worker-detector] stale heartbeat for PID P — process alive, waiting for natural exit; do not remove labels or kill the process; re-check on next scan |
Heartbeat fresh (within WorkerStaleTimeout) |
Skip regardless of PID state |
StageName for label construction is taken from Worker.StageName, which was set at dispatch time.
Windows note: isProcessAlive always returns true on Windows (signal-0 is unsupported). The detector never clears stale workers on Windows via the signal-0 path; a PID<=0 worker on Windows always falls through the sentinel probe’s unsupported stub to the bounded-unverifiable row above. The startup cleanup pass handles the restart case on any platform.
Janitor Integration (Stale-Worker Awareness)
The worktree janitor’s in-flight-worker check (reaping criterion FR-4(d)) uses the same isWorkerStale() helper as the detector. A worker that the detector would clear is not permitted to block the janitor.
Effective guard: snap.Worker() != nil && !isWorkerStale(snap.Worker(), e.workerStaleTimeout())
- Worker with fresh heartbeat OR alive PID → treated as in-flight; janitor skips the worktree.
- Worker with stale heartbeat AND dead PID → treated as absent; janitor proceeds to reap if all other criteria hold.
This prevents the multi-hour stall scenario where a Claude process dies without the engine’s context-cancel path firing, leaving a permanent WorkerEntered block that the janitor would otherwise honour indefinitely.
Startup Cleanup Pass (Restart Case)
When a Fabrik process crashes, deferred cleanup never runs. On the next startup, stale fabrik:locked:<user> and stage:*:in_progress labels may remain on issues. The startup cleanup pass removes them.
Trigger: Immediately after the first doPollCycle() completes (not a wall-clock timer). This guarantees the store is populated before the scan in both webhook and non-webhook modes.
Grace period: Any lock acquired during the first poll cycle already has Worker != nil in the store (set by LocalLockAcquired). The scan skips these items — no artificial sleep is needed.
Detection: Scan store.All() for items where:
snap.Labels()containse.lockLabel()("fabrik:locked:" + e.lockIdentity()—cfg.Userin PAT mode, the App-derived ID under App auth, §7.4; raw label check —Lock.HeldByThisis nil on restart since noLocalLockAcquiredwas applied in this session), ANDsnap.Worker() == nil
Cleanup per item:
RemoveLabel(fabrik:locked:<user>)from GitHub (with write-through to cache).- For each label in
snap.Labels()matchingstrings.HasPrefix("stage:") && strings.HasSuffix(":in_progress"):RemoveLabel(label)from GitHub (with write-through).StageNameis unavailable; labels are identified by pattern. store.Apply(WorkerExited{})— no-op sinceWorkeris already nil; applied for idempotency.- Log the cleanup at
"startup"tag.
Startup Orphaned in_progress Scan (Broader Sweep)
The lock-gated pass above only reaches items that still hold a stale lock label. An in_progress label orphaned after its paired lock was already cleanly released — the normal outcome when release() (engine/item.go) succeeds at removing the lock but the paired removeInProgressLabel call fails transiently, since that call has no retry — is invisible to it and survives every restart. runStartupOrphanedInProgressScan() closes this gap with a second, broader sweep.
Trigger: Same call site as the lock-gated pass — runs immediately after runStartupCleanup(), inside the same if firstPollErr == nil block in poll.go.
Detection: Scan store.All() — every managed item in the store, not filtered by lock label. For each item, build the set of stage names carrying stage:<Name>:complete or stage:<Name>:failed. Then, for each stage:<Name>:in_progress label on the same item whose <Name> is in that set, the label is stale by definition — a stage cannot be both finished and in progress — and is removed.
No Worker() or lock-label filter. This is a deliberate divergence from the lock-gated pass, not an oversight: dispatch ordering already guarantees complete/failed and in_progress never legitimately co-occur for the same stage — removeFailedLabel runs before in_progress is re-added on retry, and complete blocks further dispatch of that stage entirely. Any occurrence of the co-occurrence is therefore stale regardless of whether a worker or lock is present, making the predicate self-justifying and safe to run unconditionally at startup.
Cleanup per item: for each stale in_progress label found, RemoveLabel(label) from GitHub (with write-through to cache on success), logged as [startup] removed stale label %q — the same format as the lock-gated pass. Errors are logged as warnings and not retried, matching the lock-gated pass’s behavior.
Additive, not a replacement: this pass runs alongside, not instead of, the lock-gated sweep — the two predicates (stale lock label vs. complete/failed co-occurrence) are independent and both run every startup.
Scope: startup-only. A periodic (mid-run) version of this sweep would additionally need to exclude items with a live in-process worker for that stage — the startup case is safe without that check only because no workers exist yet in a freshly-started process. Running this sweep periodically is not implemented (see issue #1135).
Startup Bare in_progress Scan (ADR-1393 R7)
The two passes above are each conditional: the lock-gated pass requires a stale fabrik:locked:<user> label; the orphaned-scan pass requires a stage:<Name>:complete/failed sibling. An item with stage:<Name>:in_progress, no lock label, and no terminal sibling matches neither — the shape produced when the lock is cleanly removed (e.g. by cleanupLockedIssues() on a daemon clean-stop) but in_progress itself is never cleared, because the shutdown-pause write that should have cleared it directly (ADR-1393 R2) either never ran (force-quit, crash) or itself failed. runStartupBareInProgressScan() closes this third, disjoint gap.
Trigger: Same call site as the other two passes — runs immediately after runStartupOrphanedInProgressScan(), inside the same if firstPollErr == nil block in poll.go.
Detection: Scan store.All(), skipping any item with an active Worker() (defensive — always nil this early in startup, mirroring the lock-gated pass’s own guard). For each remaining item, build the set of stage names carrying stage:<Name>:complete/failed, and check whether any fabrik:locked: label (any user) is present. Skip the item entirely if a lock label is present (owned by the lock-gated pass) or if the item carries no in_progress label outside the terminal-sibling set (owned by the orphaned-scan pass). What remains — stage:<Name>:in_progress with neither a lock label nor a terminal sibling for <Name> — is removed.
Deliberately a third, narrow pass, not a broadened existing one: the three passes’ predicates are disjoint by construction (lock-gated; terminal-sibling-gated; neither), so each stays individually self-justifying and none silently absorbs another’s responsibility.
Cleanup per item: for each bare in_progress label found, RemoveLabel(label) from GitHub (with write-through to cache on success), logged as [startup] removed bare in_progress label %q (no lock, no terminal sibling). Errors are logged as warnings and not retried, matching the other two passes.
Scope: startup-only, for the same reason as the orphaned-scan pass (§ above).
Updated fabrik:locked:<user> Label Lifecycle
The lock label is now removed by four paths (previously two):
| Path | Trigger | Function |
|---|---|---|
| Normal completion | Stage completes (any outcome) | releaseLock() |
| Graceful shutdown | Engine receives SIGINT/SIGTERM | cleanupLockedIssues() |
| Stale-worker detector | Worker PID dead + heartbeat stale > WorkerStaleTimeout (default 5 min) |
cleanupStaleWorker() |
| Startup cleanup | Engine restart; Worker nil + lock label present | runStartupCleanup() |
stage:<Name>:in_progress is additionally cleared directly by the daemon clean-stop pause
(runShutdownPause) and by handleStopRequest (ADR-1393 R2) — not just by releaseLock() — and, on
next startup, by runStartupBareInProgressScan() above for whatever survives despite that direct
clear.
9.8 Change-feed / Observer Pattern (Phase 3-H)
Phase 3-H wires itemstate.Store.Subscribe into the engine to replace polling-based “has this item changed?” detection with a reactive change-feed. The key concept: after every non-no-op Store.Apply call, registered observers are called synchronously (outside the store’s write-lock) with a Change value indicating which fields changed and for which item.
Single-Store Architecture (Phase 5 F3)
There is exactly one *itemstate.Store instance. Engine.New() creates it and passes it to boardcache.NewCacheImpl. Both eng.store and cacheImpl.store reference the same pointer. Field ownership is by mutation type:
| Mutation category | Mutation types | ChangeFlags produced |
|---|---|---|
| Engine-side | LocalLockAcquired, LocalLockReleased, InvocationRecorded, StageAttempted, CooldownRecorded, WorkerEntered, WorkerSlotAcquired, WorkerPIDSet, WorkerHeartbeat, WorkerExited, PRChecksObserved, … |
LockChanged, InvocationChanged, StageStateChanged, CooldownChanged, WorkerChanged, WorkerLifecycleChanged (only WorkerEntered/WorkerExited), LinkedPRChanged (partial) |
| Webhook/reconcile-side | IssueLabeled, IssueUnlabeled, IssueCommentCreated, PRReviewSubmitted, CheckRunCompleted, … |
LabelsChanged, CommentsChanged, LinkedPRChanged, AssigneesChanged, … |
| Both | LocalStatusUpdated (reconcile/webhook delta path and engine write-through via cacheImpl.UpdateItemStatus) |
StatusChanged |
Because both categories write to the same store, any store.Get(...) returns a Snapshot with all field groups populated. CacheImpl.Subscribe is a thin wrapper over c.store.Subscribe; since c.store is the shared store, engine code should call e.store.Subscribe directly rather than going through cacheImpl.Subscribe to avoid double-registration.
Boundary note: After Phase 5 F4 (issue #563), all per-item and pre-linkage state lives in itemstate.Store. CacheImpl retains only paused, recentMissCache, projectID/Title/OwnerType, and localDeltaAt — none of which are per-item state. checkRuns was migrated to Store.pendingCheckRuns (Phase 5 F4, #563); linkedPRs was migrated to LinkedPRState via PRDetailsUpdated (Phase 5 F2, #562). FetchLinkedPR and FetchCheckRuns now reconstruct their results from Store snapshots.
ChangeFlags and Their Trigger Mutations
All ChangeFlags are produced by the single shared store. Field ownership is by mutation type (see table above).
| ChangeFlag | Mutation category | Trigger mutations |
|---|---|---|
StatusChanged |
Both | LocalStatusUpdated (reconcile, board fetch, projects_v2_item webhook delta); also engine-side via cacheImpl.UpdateItemStatus called from advanceToNextStage/handleStageComplete (stages.go) |
LabelsChanged |
Webhook/reconcile | IssueLabeled, IssueUnlabeled, LocalLabelAdded, LocalLabelRemoved |
LockChanged |
Engine-side | LocalLockAcquired, LocalLockReleased |
LinkedPRChanged |
Both | PR review, check runs, head SHA updates (webhook); PRChecksObserved, CIMergePendingStarted/Cleared (engine) |
PRStateChanged |
Webhook/reconcile | PRDetailsUpdated only — a narrower sub-flag of LinkedPRChanged for genuine PR-level state transitions (merged, closed, draft↔ready). Emitted alongside LinkedPRChanged, never alone. Not set by PRReviewSubmitted/PRReviewCommentCreated/ReviewThreadCommentAdded — see CommentBreakerObserver (§4.6) for why that distinction is load-bearing. |
CommentsChanged |
Webhook/reconcile | IssueCommentCreated, LocalCommentAdded |
AssigneesChanged |
Webhook/reconcile | IssueAssigneesUpdated (via applyIssuesDelta on issues.assigned/issues.unassigned events) |
InvocationChanged |
Engine-side | InvocationRecorded (token usage, completed, blocked, turn-limited, IsComment) |
StageStateChanged |
Engine-side | StageAttempted, StageRetryIncremented, ReviewCycleIncremented, etc. |
WorkerChanged |
Engine-side | WorkerEntered, WorkerSlotAcquired, WorkerKindSet, WorkerTriggerSet, WorkerTurnsRecorded, LocalLockAcquired (with Worker), WorkerPIDSet, WorkerHeartbeat, WorkerExited — all worker-handle mutations |
WorkerLifecycleChanged |
Engine-side | WorkerEntered, WorkerExited only — lifecycle transitions that change dispatch eligibility; emitted alongside WorkerChanged; this is the wake-relevant sub-flag |
CooldownChanged |
Engine-side | CooldownRecorded, ExpiredCooldownsConsumed (only when it deleted an entry) |
DeepFetchChanged |
Engine-side | DeepFetchFailed, ItemDeepFetched |
ItemRemoved |
Reset | Store.Reset for items present in the old map but absent from the new items slice |
CheckRunChanged |
Webhook/reconcile | CheckRunCompleted when SHA is not yet in shaToKey — run buffered into Store.pendingCheckRuns; also emitted (combined with LinkedPRChanged) by PRHeadSHAUpdated when the buffer is drained. Not in wakeChFlags: the CI gate is catch-up-driven (the catch-up loop polls on every cycle), not wake-driven. The flag exists for observability (future TUI consumers) without forcing a poll wake. |
PRFactsChanged |
Poll-side | PRFactsObserved, CIVerdictObserved, and PRHeadSHAUpdated when it clears the CI verdict on a head change (#2122). Informational; not in wakeChFlags/cycleSetFlags, so a poll-side write never wakes the poll |
wakeChFlags: Which Changes Wake the Poll Loop
The wakeChObserver is registered once on the shared store. It fires a non-blocking send on wakeCh when Change.Fields & wakeChFlags != 0:
wakeChFlags = StatusChanged | LabelsChanged | CommentsChanged | LockChanged | LinkedPRChanged | AssigneesChanged | WorkerLifecycleChanged
AssigneesChanged was added in issue #543 so that assignment webhooks wake the dispatcher. WorkerLifecycleChanged was added in Fix B (issue #544) so that WorkerEntered and WorkerExited mutations deterministically wake the poll loop. The broader WorkerChanged flag (which also covers WorkerHeartbeat and WorkerPIDSet) is intentionally not in wakeChFlags — heartbeats fire every 30s per active worker and would otherwise cause repeated deep-fetch cycles for items that cannot be dispatched. See §2.14.
cycleSetFlags (issue #576): newMayNeedWorkObserver uses a narrower constant — cycleSetFlags = wakeChFlags &^ WorkerLifecycleChanged — rather than wakeChFlags. WorkerLifecycleChanged is excluded so that WorkerExited from an early-return goroutine (e.g. a dep-blocked item) does not bypass the cooldown gate by adding the item to mayNeedWork. The wake channel (newWakeChObserver) still uses the full wakeChFlags, so non-blocked items are re-evaluated promptly on worker exit. See §9.2 and §9.9.
Probe-origin writes never wake (#2080 R3). The probe loop (runProbeAndDeepFetch) runs at the start of the same poll() that dispatches, so everything it writes is already seen by that poll; waking Run() for those writes only schedules a redundant poll ~minPollInterval later, and a probe/cache disagreement that kept re-invalidating turned that into a forced poll every ~1.5 s. The probe loop therefore wraps its Store writes in itemstate.FromProbe (ProbeBoardItemUpdated, DeepFetchInvalidated, PRDetailsUpdated, PRHeadSHAUpdated, TerminalFlagSet, and — through CacheImpl.FetchItemDetailsFromProbe — the stale-item deep fetch’s ItemDeepFetched/PRHeadSHAUpdated), which tags the resulting Change.Origin as OriginProbe; newWakeChObserver ignores that origin. mayNeedWorkObserver is unaffected. Webhook, worker and REST writes are untagged and wake as before, as does the new-item path (IssueOpened). Any new probe-loop write must go through FromProbe or it will wake the poll again. The probe apply also honours the engine-write protection for Status and closed state (D.4, #2178).
Changes that do NOT fire wakeCh: InvocationChanged, StageStateChanged, CooldownChanged, WorkerChanged (from heartbeats/PID-sets/lock-with-worker), TitleBodyChanged, StateChanged, BlockedByChanged, DeepFetchChanged, BaseBranchChanged, ItemRemoved, CheckRunChanged (CI gate is catch-up-driven; see §9.8 ChangeFlag table).
The unconditional wakeCh <- struct{}{} send that previously lived in the webhook handler has been removed. The observer path is the sole mechanism for immediate wake — but wakeChObserver is only registered when wakeCh != nil. In headless runs (--notui or any configuration without a wake channel), mayNeedWorkObserver (always registered) alone populates the pre-filter set; wakeCh is never signalled and there is no immediate dispatcher wake. Because all mutation types flow through the single shared store, the single wakeChObserver registration (when present) is sufficient — no per-source registration is needed.
Registered Observers and Where They Live
| Observer | Registered on | Fires when | Emits |
|---|---|---|---|
wakeChObserver |
shared store (once) | Change.Fields & wakeChFlags != 0 |
wakeCh <- struct{}{} (non-blocking) |
mayNeedWorkObserver |
shared store (once) | Change.Fields & cycleSetFlags != 0 (excludes WorkerLifecycleChanged — see §9.9) |
adds repo#number to e.mayNeedWork |
InvocationObserver |
shared store | Change.Fields & InvocationChanged != 0 |
tui.JobCompletedEvent (Success/Errored are false/true respectively only for a genuine fault as of #1178 — a turn-cap exit (CLI subtype: "error_max_turns") sets TurnLimited: true instead and no longer counts as an error for rendering purposes, even though the underlying Claude process still exited non-zero and the existing retry/cooldown mechanics are unaffected; Completed remains a separate axis meaning “no FABRIK_STAGE_COMPLETE marker was seen”) |
StageChangeObserver |
shared store | Change.Fields & StatusChanged != 0 |
tui.StageChangedEvent |
| Pause observer (closure) | CacheImpl.SubscribePause |
Pause() / Resume() |
tui.WebhookStatusEvent |
All observers are registered in Engine.Run() after extracting cacheImpl. Their unsubscribe funcs are deferred so observers are cleaned up when Run() returns.
mayNeedWork Set: Poll Cycle Pre-Filter
Engine.mayNeedWork map[string]bool (protected by Engine.mayNeedWorkMu) is populated by mayNeedWorkObserver on every change that includes wakeChFlags. At the start of each poll cycle, poll() drains it to a local cycleSet:
cycleSet := func() map[string]bool {
e.mayNeedWorkMu.Lock()
defer e.mayNeedWorkMu.Unlock()
s := e.mayNeedWork
e.mayNeedWork = make(map[string]bool)
return s
}()
Each item in board.Items passes the pre-filter if any of these bypass conditions apply (authoritative source: poll.go:1291–1310, hasAwaitingLabel variable in selectDeepFetchCandidates):
| Bypass condition | Label / Condition | Rationale |
|---|---|---|
Item is in cycleSet |
(observer-driven) | Observer saw a relevant change since last poll |
Stage has CleanupWorktree: true |
(stage config) | Cleanup triggers on local filesystem state, not board changes |
Item has fabrik:awaiting-ci label |
fabrik:awaiting-ci |
CI check-run completions don’t bump the issue’s updatedAt; CI gate must be evaluated every poll |
Item has fabrik:rebase-needed label |
fabrik:rebase-needed |
Base-branch advances don’t bump updatedAt; rebase gate must be evaluated every poll |
Item has fabrik:awaiting-review label |
fabrik:awaiting-review |
Review-submission webhooks don’t bump updatedAt; gate clearance and Phase 1/Phase 2 reprompt timers require per-poll evaluation (issue #616) |
Item has fabrik:auto-merge-enabled label |
fabrik:auto-merge-enabled |
GitHub auto-merge state changes (e.g., PR merges) don’t bump the issue’s updatedAt; convergence monitor must be evaluated every poll |
Item has fabrik:revalidate label |
fabrik:revalidate |
Stuck-Validate items may have settled updatedAt long ago; the revalidate scan must still be reached on every poll |
snap.HasExpiredCooldown(now) is true |
(cooldown timer) | Periodic re-evaluation window has passed |
Items not meeting any bypass condition are skipped for that poll cycle (no deep-fetch, no dispatch). Items with an active CooldownAt (not yet expired) are also skipped — the CooldownAt[“periodic-re-eval”] entry gates time-based re-evaluation.
This replaces the removed engine.seenUpdatedAt map[string]time.Time, which performed an equivalent “has this item changed?” gate via timestamp comparison.
CacheImpl.SubscribePause
In addition to Store.Subscribe, CacheImpl exposes SubscribePause(fn func(bool)) func() for components that need to react to pause/resume transitions (e.g., the TUI WebhookStatusEvent). Pause observers are called outside c.mu — Pause() and Resume() snapshot the list before releasing c.mu, then call observers on the snapshot. This mirrors Store’s snapshot-then-call pattern. Observers registered via SubscribePause MUST NOT call Pause() or Resume() re-entrantly. Because c.mu is released before observers are called, calling other CacheImpl methods from an observer is deadlock-free; however, calling Pause/Resume from within a pause observer produces semantic re-entrancy (double-fire, inconsistent paused state).
Invariants
- I1 (Single fire): Each
Store.Applycall produces at most oneChangeper observer registration.Store.Resetproduces oneChangeper item (add, update, or removal). Observers are called in registration order. - I2 (Outside lock): Observers are called after the store’s write-lock is released. Observers may safely call
store.Getor other read methods without deadlock. - I3 (Single registration): Each observer is registered exactly once on the shared store. Since
engine.storeandcacheImpl.storeare the same pointer, calling bothe.store.Subscribe(obs)andcacheImpl.Subscribe(obs)registers the observer twice, causing every Apply to fire it twice. Register once viae.store.Subscribe. See ADR-038 for the historical dual-store design that this replaced. - I4 (Non-blocking wakeCh): The
wakeChObserveralways uses a non-blocking send. A fullwakeCh(capacity 1) means a poll is already pending; dropping the signal is correct (burst coalescence).
References: ADR-036: Reactive Cache Single-Owner, ADR-038: Observer Wiring (historical dual-store design).
9.9 Pre-Dispatch Label Gates and Wake-Loop Avoidance
Problem: When processItem returns early — without invoking Claude — WorkerExited fires with WorkerLifecycleChanged. Before Fix B (issue #576), WorkerLifecycleChanged was in wakeChFlags and was used by newMayNeedWorkObserver to populate mayNeedWork (the cycleSet). This caused the blocked item to bypass the cooldown gate on the next poll cycle, be deep-fetched, pass through itemNeedsWork, dispatch another goroutine, hit the same early-return — creating a tight wake-loop at 2-3 Hz that burned GraphQL budget.
Pattern: Labels that cause processItem to return early without invoking Claude must be checked in itemNeedsWork before goroutine launch. This is the pre-dispatch gate pattern. The gate prevents WorkerEntered/WorkerExited from firing at all — no goroutine means no WorkerLifecycleChanged, no wake signal, no cycleSet entry.
Currently gated labels in itemNeedsWork:
| Label | Gate Added | Notes |
|---|---|---|
fabrik:locked:<other> |
Always (original) | Items owned by another instance |
fabrik:editing |
Issue #550 | Items in active comment processing |
fabrik:blocked |
Issue #576 | Items waiting for blocking issues to close |
Defense-in-depth (Fix B, issue #576): cycleSetFlags excludes WorkerLifecycleChanged. Even if a pre-dispatch gate is missing for a given early-return label, WorkerExited no longer bypasses the cooldown gate via mayNeedWork. The item is still re-evaluated on cooldown expiry (CooldownAt["dep-blocked"]) or when a genuinely-new event fires LabelsChanged/StatusChanged. Fix A (pre-dispatch gate) eliminates goroutine launch; Fix B ensures the loop cannot self-perpetuate even if Fix A is absent for some future early-return path.
Other early-return paths: fabrik:awaiting-input and fabrik:paused are conditionally gated in itemNeedsWork — they return false when no new comment is present, preventing goroutine launch and the associated wake-loop. fabrik:awaiting-review is handled by the catch-up loop rather than by an early-return inside processItem, so it does not apply this pattern. Future additions to processItem that introduce early-return paths should add a corresponding pre-dispatch gate in itemNeedsWork; Fix B’s cycleSetFlags provides defense-in-depth while a gate is absent.
Cross-references: §1.4 (label semantics), §9.2 (cycleSetFlags split), ADR-039 (decision record for the wakeChFlags/cycleSetFlags split).
9.9.1 Non-Blocking Slot Acquire and Slot-Deferred Dispatch (#2114, ADR 2114)
dispatchCandidates acquires an Engine.sem slot with a non-blocking select (default branch). It never waits on a worker: everything poll() runs after dispatch (settleTrainRuns/handleMergeTrainBatch and the settle scans) and the next poll would otherwise stall for as long as the shortest-remaining worker, and a quiet worker extends that indefinitely. ctx.Err() is checked at the top of each loop iteration, since the default branch no longer observes cancellation.
When no slot is free the candidate is deferred: no WorkerEntered, issue context, StageAttempted, retry count, cooldown or label is applied, and the rest of the candidate loop still runs (once the semaphore is full every remaining admitted candidate is deferred; guard-skipped ones cost nothing extra). The snap.Worker() != nil guard, the R5 sentinel check (#1779), WorkerEntered-before-goroutine and the release-then-WorkerExited defer order are unchanged.
Logging. One poll-level line per poll with a deferral: [poll] deferred dispatch of N candidate(s) #A, #B: all worker slots busy (in_use/max in use), where in_use/max is len(e.sem)/cap(e.sem) — true slot occupancy including reinvoke and merge-train conflict-resolution holders (not the Worker-entry count the local API reports).
Re-admission. The WorkerExited wake alone does not re-admit a slot-deferred item: its cycleSet entry was consumed by the poll that deferred it, an end-of-poll periodic-re-eval stamp would leave it behind a cooldown of up to PollSeconds*10, and cycleSetFlags excludes WorkerLifecycleChanged (§9.2). So dispatchCandidates returns the deferred issueKeys; poll()’s end-of-poll stamp skips them, and requeueDeferredDispatch writes them into e.mayNeedWork (directly, under mayNeedWorkMu, so it causes no wake). The next poll — woken by the freed slot or by the ticker — admits them through cycleSet, bypassing the cooldown gate, without changing ADR-039’s flag split.
Re-admission survives a failed deep-fetch. The poll that re-admits a slot-deferred item does so through cycleSet alone, so no expired cooldown is consumed. If that poll then drops the item before it joins deepFetchCandidates — FetchItemDetails fails, or itemMayNeedWork refuses it inside the deep-fetch failure backoff — selectDeepFetchCandidates re-arms periodic-re-eval (rearmPeriodicReEval) for any cycleSet-admitted item, not only for one whose expiry was consumed. Otherwise the item would have no cooldown and no cycleSet entry and would sit stranded until an unrelated change. The re-arm interval equals the failure backoff (githubRecheckInterval), so a persistent failure does not become a per-poll API call.
Reinvokes can take a freed slot first. A review- or CI-fix reinvoke goroutine blocked on e.sem (engine/reinvoke.go) is already waiting when a slot frees, so it can take that slot before the poll’s non-blocking attempt. A deferred dispatch can therefore be delayed by reinvokes, but never stranded: it stays queued in mayNeedWork and is re-attempted on every following poll.
Known limits. A wake dropped during GraphQL/REST backoff (#1716) leaves the item to the backed-off ticker poll, which still re-admits it. A merge-train conflict-resolution slot release (acquireTrainSlot, ADR-2046) fires no WorkerExited, so it does not wake the poll; the next ticker poll covers it. Under sustained saturation each poll re-deep-fetches each deferred item (cache-served when the board cache is fresh). Sim coverage: tests/sim/slot_saturation_test.go.
9.9.3 Dispatch Order (#2232, ADR-2232)
dispatchCandidates hands free worker slots out by time without a worker, longest first (orderDispatchCandidates), not in board order: the oldest LastAttemptAt for the item’s current stage first, items that have never run before all others, ties by issue number so the order is deterministic. This matters because a time slice (§7.1) is re-dispatched with no backoff: the sliced item has just had a worker, so it sorts after every longer-waiting candidate and takes its own slot back only if one is still free after they have had their turn. A slot-deferred candidate (§9.9.1) is re-admitted by the next poll and sorted by the same rule, so a deferral never moves it to the back. Holding stages, settle scans and merge-train dispatch do not pass through it. LastAttemptAt is stamped when an invocation ends, so it approximates the instant the item last held a worker.
9.9.2 Post-Run Feedback Admission (#2232, ADR-2232)
Feedback that arrived while a worker ran is read into the cache during the run, so its arrival raises no change signal once the run ends: a poll woken by the comment drained its cycleSet entry and dropped the item because a worker was in flight (§9.9, snap.Worker() != nil), and the bare WorkerExited wake is excluded from cycleSetFlags. The item then fell through to periodic-re-eval. So the goroutines that run a worker (dispatchCandidates and the shared reinvoke scaffold dispatchReinvoke) defer postRunFeedbackAdmission, declared so it runs first on exit — before the slot release and the WorkerExited wake. It reads the item from the Store (Peek, no GitHub call) and applies cachedFeedbackArrivedSince — cachedPendingFeedbackOf (the cache-only twin of unprocessedFeedback, pinned by TestCachedFeedbackMatchesLiveGate) restricted to feedback created at or after the worker’s start (e.now() captured when the goroutine began; a comment, thread comment or review body with an unknown timestamp does not count). Feedback that was already pending when the run began is what the run was for: if it is still pending afterwards the pass failed, and admitting it again would bypass every cooldown for a pass that ended without clearing it and without pausing, so it takes the ordinary retry path instead. When anything that arrived during the run is pending (comments, current-head review threads, unaddressed review bodies) it sets e.mayNeedWork[key] (as requeueDeferredDispatch does), so the next poll admits the item through cycleSet, bypassing cooldowns; itemNeedsWork already lets a new comment past the retry backoff. The entry is drained once. It is conditional on known pending feedback — admitting every exit would undo ADR-039 — and is skipped on shutdown, for cleanup-stage, closed, PR and paused items (a pause lifts only on a human comment created after it, so cached feedback cannot resume the item). Admission is not a dispatch: the comment pass or review reinvoke it leads to stays bounded by the comment circuit breakers and the review cycle limit, and the stage’s own retry still waits out retry_backoff (LastAttemptAt). Feedback GitHub has but the cache lacks is still caught by feedbackGateBlocks at every advance and landing.
10. State Diagrams
10.1 Happy Path — Linear Stage Progression
stateDiagram-v2
direction TB
[*] --> Specify_Idle : Issue added to board
Specify_Idle --> Specify_Running : Poll tick
Specify_Running --> Specify_Complete : FABRIK_STAGE_COMPLETE
Specify_Complete --> Research_Idle : Auto-advance or human move
Research_Idle --> Research_Running : Poll tick
Research_Running --> Research_Complete : FABRIK_STAGE_COMPLETE
Research_Complete --> Plan_Idle : Auto-advance or human move
Plan_Idle --> Plan_Running : Poll tick
Plan_Running --> Plan_Complete : FABRIK_STAGE_COMPLETE
Plan_Complete --> Implement_Idle : Auto-advance or human move
Implement_Idle --> Implement_Running : Poll tick
Implement_Running --> Implement_Complete : FABRIK_STAGE_COMPLETE
note right of Implement_Running
Draft PR created
PR marked ready
end note
Implement_Complete --> Review_Idle : Auto-advance or human move
Review_Idle --> Review_Running : Poll tick
Review_Running --> Review_Complete : FABRIK_STAGE_COMPLETE
Review_Complete --> Validate_Idle : Auto-advance or human move
Validate_Idle --> Validate_Running : Poll tick
Validate_Running --> Validate_Complete : FABRIK_STAGE_COMPLETE
Validate_Complete --> Done_Pending : Yolo: merge + advance
Done_Pending --> Done_Complete : Worktree cleaned up
Done_Complete --> [*]
10.2 Off-Path Flows
stateDiagram-v2
state "Active Stage" as Active {
Idle --> Running : Poll tick
Running --> Complete : FABRIK_STAGE_COMPLETE
Running --> AwaitingInput : FABRIK_BLOCKED_ON_INPUT
Running --> Cooldown : No marker (incomplete)
Cooldown --> Running : Cooldown expired (retry)
Cooldown --> PausedFailed : MaxRetries exceeded
AwaitingInput --> CommentProcessing : Human comment
PausedFailed --> Idle : User removes fabrik:paused
Idle --> Paused : Human adds fabrik:paused
Paused --> CommentProcessing : Human comment (implicit unpause)
Paused --> Idle : Human removes fabrik:paused
Idle --> Blocked : Open dependencies
Blocked --> Idle : All dependencies closed
Complete --> AwaitingReview : wait_for_reviews gate
AwaitingReview --> ReviewReinvoke : Gate clears + unresolved threads
AwaitingReview --> AwaitingInput : Review timeout
AwaitingReview --> NextStage : Gate clears, no threads
ReviewReinvoke --> AwaitingReview : Reinvoke completes, new reviews arrive
ReviewReinvoke --> AwaitingInput : Cycle limit exceeded
Complete --> AwaitingCI : wait_for_ci gate (CI failure detected)
AwaitingCI --> CIFixReinvoke : CI failed, cycle ok
AwaitingCI --> NextStage : CI checks all pass
AwaitingCI --> AwaitingInput : CI timeout or cycle limit
CIFixReinvoke --> AwaitingCI : Reinvoke completes, CI re-evaluated next poll
CIFixReinvoke --> AwaitingInput : Cycle limit exceeded
Complete --> NextStage : Auto-advance or human move
Paused --> NextStage : Validate stage + PR merged externally (runValidatePRTerminalAdvance — any gate label, ADR-056 D2)
CommentProcessing --> Complete : FABRIK_STAGE_COMPLETE in comment output
CommentProcessing --> Idle : Comment processed (no completion)
}
state "Next Stage" as NextStage
NextStage --> Active : Board column updated
10.3 Review Reinvoke Cycle
stateDiagram-v2
direction TB
StageComplete --> CheckDependencies : Catch-up loop Phase 1 (unconditional — all items)
CheckDependencies --> Blocked : Has open blockers
CheckDependencies --> CheckReviewGate : No blockers
Blocked --> CheckDependencies : Next poll tick
CheckReviewGate --> WaitingForReviewers : Outstanding reviewers
CheckReviewGate --> TimedOut : Timeout elapsed
CheckReviewGate --> GateCleared : All reviewers submitted
WaitingForReviewers --> CheckReviewGate : Next poll tick
TimedOut --> PausedForTimeout : pauseForReviewTimeout()
note right of PausedForTimeout
fabrik:paused
fabrik:awaiting-input
end note
GateCleared --> CheckThreads : buildReviewThreadComments()
CheckThreads --> Phase2 : No unresolved threads → Phase 2
CheckThreads --> CheckInFlight : Unresolved threads exist
CheckInFlight --> SkipReinvoke : Already in-flight
CheckInFlight --> CheckCycleLimit : Not in-flight
CheckCycleLimit --> PausedForCycles : cycleCount >= MaxReviewCycles
note right of PausedForCycles
fabrik:paused
fabrik:awaiting-input
end note
CheckCycleLimit --> DispatchReinvoke : cycleCount < MaxReviewCycles
DispatchReinvoke --> ProcessComments : Async goroutine
ProcessComments --> CheckReviewGate : Next poll (if new reviews arrive)
ProcessComments --> Phase2 : Stage complete after addressing feedback
SkipReinvoke --> CheckReviewGate : Next poll tick
Phase2 --> Advance : yolo/cruise/auto_advance gate passes
Phase2 --> Idle : Gate not met — no advancement
note right of Idle
Item stays in stage:X:complete
until user advances manually
or adds yolo/cruise label
end note
10.4 Additional Flows
These flows are absent from 10.1–10.3 but represent reachable states documented in §2, §3.4, §5, and §6.8.
stateDiagram-v2
direction TB
%% Validate re-entry back-edges
ValidateComplete --> ValidateIdle : fabrik:revalidate applied (§2.15)\nRemoves stage:Validate:complete, gate labels, paused labels;\nresets retry counters; dispatches Validate on next poll
ValidateComplete --> ValidateIdle : SHA-invalidation scan (§2.16)\nPR HEAD SHA ≠ ValidateCompletedSHA\n(not landed, not a pure engine catch-up head);\nremoves stage:Validate:complete + gate labels
note right of ValidateIdle
ValidateIdle = Validate column,
Idle (no completion label)
end note
%% Convergence monitor deferred Done advance
ValidateAwaitingMerge --> Done : GitHub merges PR (§5.5)\ncheckAutoMergeConvergence detects pr.Merged == true;\nremoves fabrik:auto-merge-enabled; advanceToNextStage
note right of ValidateAwaitingMerge
ValidateAwaitingMerge =
stage:Validate:complete present
+ fabrik:auto-merge-enabled present
(yolo flow after handleStageComplete)
end note
%% FABRIK_PR_CREATE marker (Implement)
ImplementInProgress --> ImplementInProgress : FABRIK_PR_CREATE_BEGIN/END (§5.6)\nEngine creates draft PR; no stage advance\nPR URL posted as issue comment
%% FABRIK_NO_WORK_NEEDED durable-marker path (§6.8, ADR-060)
AnyStageInProgress --> AwaitingDone : FABRIK_STAGE_COMPLETE + FABRIK_NO_WORK_NEEDED\nfabrik:awaiting-done written FIRST (durable, survives restart);\nAll subsequent non-cleanup stages marked complete;\nno PR created; dispatch suppressed regardless of column
AwaitingDone --> AwaitingDone : Settle pass fails (status move or close)\nRetried every poll by the settle scan;\nfabrik:awaiting-done remains
AwaitingDone --> Done : Settle pass succeeds\nfabrik:awaiting-done removed
AwaitingDone --> Paused : Settle pass fails Attempts >= MaxRetries times\nescalateNoWorkNeededFailure: fabrik:paused added,\nfabrik:awaiting-done removed, comment posted
note right of AwaitingDone
AwaitingDone = fabrik:awaiting-done present
Board column may still be the
emitting stage's own column —
the Done move itself may be
what's failing
end note
note right of Paused
Paused (here) = fabrik:paused present,
fabrik:awaiting-done removed.
Escalation after MaxRetries settle
failures — human intervention required
end note
note right of Done
Done = Pending Cleanup
(worktree removal by janitor)
end note
%% Spawned child board-placement retry (§6.9, ADR-062)
ChildCreated --> AwaitingPlacement : UpdateProjectItemStatus fails\n(call error, nil statusField, or no suitable option)\nfabrik:awaiting-placement written on the CHILD;\nchild, board item, blockedBy link already exist
AwaitingPlacement --> AwaitingPlacement : Settle pass fails\nRetried every poll, sourced from board.Items directly\n(NOT deepFetchCandidates — child's column matches no stage);\nfabrik:awaiting-placement remains
AwaitingPlacement --> ChildPlaced : Settle pass succeeds\nfabrik:awaiting-placement removed
AwaitingPlacement --> ChildClosed : Child observed closed\nclearChildPlacementMarker: fabrik:awaiting-placement removed;\nno placement attempt, no pause, no comment
AwaitingPlacement --> ChildPaused : Settle pass fails Attempts >= MaxRetries times\nescalateChildPlacementFailure: fabrik:paused added to child,\nfabrik:awaiting-placement removed, comment posted on child,\nbest-effort comment posted on parent
note right of AwaitingPlacement
AwaitingPlacement = fabrik:awaiting-placement present
Board column is typically Backlog —
declared unmanaged (or, on installs
without backlog.yaml, still unrecognized),
so ordinary dispatch never revisits it
end note
note right of ChildPaused
ChildPaused = fabrik:paused present on the child,
fabrik:awaiting-placement removed.
Parent remains blocked (blockedBy) with
no other visibility — the best-effort
parent comment is the only signal
end note
note right of ChildClosed
ChildClosed = child issue closed;
no further board dispatch is needed,
so the marker is simply cleared
end note
Assignee transitions (§2.13): Assignee changes fire AssigneesChanged → wakeChObserver, re-evaluating the item on the next poll. No label is mutated. No separate diagram node is warranted.
11. Worktree Janitor
11.1 Purpose
The worktree janitor reaps orphaned git worktrees left behind when issues exit the
managed lifecycle without triggering the normal cleanup_worktree: true Done-stage path.
Three scenarios produce stranded worktrees:
- Off-board items — issues removed from the project board (archived manually by an operator, or deleted) are no longer iterated by the poll loop; their worktrees persist indefinitely.
- Narrow restart window — Fabrik dies after
attemptMergeOnValidateadvances the board to Done but before the next poll dispatches the Done stage. - Stage YAML drift — an operator who renames or removes the Done stage from their config orphans worktrees that will never again be visited by the dispatch loop.
11.2 Schedule
- Startup scan: runs once after the first successful poll cycle (so the Store is
hydrated) immediately after
runStartupTerminalScan. Gated onJanitorIntervalHours != 0. - Hourly recurring scan: a ticker goroutine started from
Engine.Run()fires atJanitorIntervalHours-hour intervals (default 1 hour). First tick is one interval after startup; the startup scan covers the “now” tick. - EC-3 (cycle overrun): the goroutine blocks inside
runWorktreeJanitorduring a long cycle; the ticker fires are dropped naturally. No special handling required.
11.3 Reaping Gate (FR-4)
A worktree is reaped only when all four conditions hold:
| Condition | Rationale |
|---|---|
| (a) Issue is closed | Open issues are never safe to reap — the issue may still be in progress, even if no worker is currently dispatched. |
| (b) Off-board OR cleanup-complete | If the issue is still on the board at an active (non-cleanup) stage, the dispatch loop may still visit it. A worktree at a cleanup_worktree: true stage with stage:<Name>:complete is safe to reap: CleanupWorktree runs before the label is applied, so the directory can only persist if CleanupWorktree errored. |
| (c) Clean worktree | git status --porcelain returns no non-engine-managed paths. Engine-managed files (.fabrik-context/, .fabrik/issue.md) are excluded by isEngineManagedPath. A dirty worktree may contain uncommitted work from an interrupted Claude session. |
| (d) No in-flight worker | snap.Worker() == nil from the Store. Concurrent reads are safe — Store uses RWMutex. |
All four conditions are evaluated in order (cheapest first). A worktree that fails any
condition is left in place and counted toward the skipped total.
Closed-status resolution (FR-5):
- Primary:
Store.Get(repo, number)→snap.IsClosed()(free; no API call). - Fallback (item not in Store):
GitHubClient.FetchIssue(owner, repo, number)— one REST call per stranded worktree per cycle; result cached in-memory for the cycle duration. - Error: skip with warning; treat as not-closed to avoid false positives.
Cleanup path (FR-6):
WorktreeManager.CleanupWorktree(number, false) — the same path used by the Done stage.
If the bare repo is absent (manually deleted), falls back to os.RemoveAll with a warning.
Both paths now run the cwd-rooted process reaper immediately before removing the directory
(including the os.RemoveAll fallback) — see
stage-lifecycle.md § Worktree Teardown Process Reaping
for detail.
11.4 Owner/Repo Resolution
The on-disk directory is named owner-repo (dash-separated), which is ambiguous for
hyphenated owner or repo names.
- Registered repos: looked up from the reverse map built from
e.worktreeManagers(copied undere.muat the start of each cycle; lock held only for the copy). - Unregistered repos (EC-2 — repo no longer on the board):
git remote get-url originis run from the first issue subdirectory found. If the git command fails (corrupt or missing.git), the entireowner-repodirectory is skipped with a warning.
11.5 Configuration
# .fabrik/config.yaml
janitor_interval_hours: 1 # default; set to 0 to disable the janitor entirely
Also settable via flag (--janitor-interval) or environment variable
(FABRIK_JANITOR_INTERVAL). EC-6: changing this value at runtime has no effect; Fabrik
must be restarted for a new cadence to take effect.
11.6 Summary Log (FR-7)
At the end of each cycle the janitor emits an INFO-level log line:
[janitor] cycle complete: scanned N worktrees, reaped K, skipped M (reasons: ...)
The reasons field tallies distinct skip reasons (e.g. issue open ×3, dirty worktree,
in-flight worker). Each reaped worktree is also logged individually at reap time.
11.7 Process-Sweep Janitor (#1798 R3)
A fourth periodic janitor, runProcessSweepJanitor, runs alongside the worktree/log/session
janitors above — same schedule (§11.2: startup scan, then hourly), same
JanitorIntervalHours != 0 gate, no separate config knob. Unlike the other three, it does
not scan .fabrik/worktrees/ at all: it sweeps the durable session-scoped descendant
registry (.fabrik/state/descendants.json) for orphans that escaped both the PGID-scoped
grandchild cleanup and the per-invocation session-scoped reap — see
stage-lifecycle.md § Session-Scoped Descendant Reaping
for the full mechanism, and ADR 1798 for
the design rationale.
Sweep condition: a registry entry is only ever acted on when its recorded worker PID is
confirmed dead — a live signal-0 probe and an identity re-check against the worker’s own
comm/lstart fingerprint (recorded alongside the entry at discovery time). A live PID whose
fingerprint no longer matches means the original worker exited and the OS has since reused its
PID for an unrelated process; treating that PID as “still alive” without the identity re-check
would leave the entry unswept forever, not merely delayed. An entry whose worker is still alive
and still the same worker is left untouched — that invocation is still in flight, and the
per-invocation reap will handle it at its own invocation end. This means the sweep only ever
reaps genuine orphans: descendants of a worker that crashed before reaching its own
invocation-end reap, descendants left behind by a previous engine run entirely (the case this
janitor exists for, since the registry is a JSON file on disk and survives a restart while the
engine’s own in-memory dispatch state does not), or descendants whose worker PID was reused.
Identity re-verification (R5): immediately before killing any entry, its recorded
comm+lstart identity fingerprint is re-checked against the live process at that PID. A
mismatch — the process already exited, or the PID was reused for something unrelated since it
was recorded — means the entry is dropped from the registry and never signalled.
Summary log:
[proc-janitor] cycle complete: scanned N registry entries, reaped K, skipped M
Each reaped PID is also logged individually, via the existing [#N kill] convention, at
sweep time.
Registry-independent session sweep (#1814): runProcessSweepJanitor additionally calls
sweepOrphanedWorkerSessions, because the registry sweep above can only act on a descendant that
was recorded — one spawned in its worker’s final seconds, or during a transient ps failure, has
no entry and is invisible to it. The worker record is written at spawn to
.fabrik/state/workers.json (a JSON array; one record per invocation, keyed by unique ID, never
by PID; an unparseable file is quarantined to workers.json.corrupt with a logged warning and
treated as empty, so a corrupt file cannot disable recording). The sweep takes one process-table scan (the shared cached one) and one Getsid pass, and
reaps every process whose SID equals the PID of a recorded worker classified as dead. The
invocation-end variant runs the same logic with a fresh scan (stage-lifecycle.md § Session-Scoped
Descendant Reaping).
| Worker PID state | Result |
|---|---|
| not alive | dead — sweep members |
| alive, fingerprint lookup errors | inconclusive — skip, keep record |
| alive, record fingerprint empty | skip (never signalled on liveness alone) |
alive, lstart equal to record |
live worker — skip |
alive, lstart differs |
recycled PID — sweep only members that started at or after the recorded worker and strictly before the current holder; equal/unparseable timestamps skip |
Ownership is by session lineage only (no comm/args allow-list), and only for a recorded worker:
a dead session leader with no record is never touched. Before each SIGKILL the candidate’s
fingerprint is re-read and Getsid re-checked; a lookup failure skips it. A record is pruned only
when the worker is dead, a successful scan found zero live members and nothing was skipped, on two
consecutive scans (EmptyScans >= 2, persisted across janitor passes); a scan error or any skip
resets the count and keeps the record. A scan error changes nothing.
Owner-marker reap (#2128, ADR-2128). A worker record also carries Owner: the path of the owner file named by the FABRIK_WORKER_OWNER the worker was started with (.fabrik/state/worker-owners/<nonce>.json, holding the worker’s PID and start token, written right after cmd.Start()). For a record classified dead or recycled — never live or inconclusive (R4) — both sweeps call sessionreap.ReapMarked before the session pass: every live process whose initial environment carries exactly that marker gets SIGTERM then SIGKILL, per exact PID, after re-checking its start token and marker and the CheckSID/CheckPGID refusals. A process with no marker is never signalled on shape (ppid 1, empty group) alone. The prune rule additionally requires the marked set to be empty (two consecutive scans), and pruning removes the owner file; the janitor also deletes owner files that no record references and that are older than 24h. The unreadable-environment cases (SIP-protected macOS binaries, other users, hidepid mounts) read as unmarked.
Summary log (a second line; the line above is unchanged):
[proc-janitor] session sweep complete: worker records N, session members reaped K, skipped M, records pruned P
Each reap is logged individually as [#N kill] sending SIGKILL to PID <pid> (<comm>) — orphaned
session member of dead worker PID <W> (reason=session_sweep_periodic). See
ADR 1814.
Appendix A: Two Paths to Stage Advancement
Stage advancement can occur through two code paths:
| Aspect | Path 1: handleStageComplete() |
Path 2: Catch-up loop in poll() |
|---|---|---|
| Runs in | Worker goroutine | Poll goroutine |
| Triggered by | Claude outputs FABRIK_STAGE_COMPLETE | Poll cycle finds stage:<X>:complete label |
| Review data | Stale (just ran MarkPRReady) | Fresh (from FetchItemDetails) |
| Review gate | Optimistic: applies fabrik:awaiting-review, returns |
Real: calls checkReviewGate(), evaluates timeout |
| Label freshness | Re-fetched via refreshAutonomyLabels() |
Re-fetched via refreshAutonomyLabels() |
| Merge at Validate | attemptMergeOnValidate() called directly |
attemptMergeOnValidate() called from catch-up (yolo only) |
| Advancement | advanceToNextStage() if should advance and no gate |
advanceToNextStage() after Phase 2 gate (yolo/cruise/auto_advance) |
Autonomy-label re-fetch, both paths (#1769, §6.23). handleStageComplete() and runCatchUpPhase2() each call the shared refreshAutonomyLabels() helper (engine/stages.go) as the first step of their autonomy-decision logic, before either reads hasYoloLabel/hasCruiseLabel. This row previously read “Already fresh from deep fetch” for Path 2 — that was false: Labels is not part of the deep-fetch cache contract (copyDeepFieldsFromState excludes it), so an item that went through a “fresh” deep fetch earlier in the same poll pass could still carry a stale fabrik:yolo/fabrik:cruise from before an operator removed it. Both paths now perform the same live, item-scoped FetchLabels() call (never e.cfg.Owner/e.cfg.Repo, which is empty in multi-repo mode) immediately before the autonomy decision, so a mid-run add or remove of either label is observed by whichever path makes the next advance decision — see §6.23 for the full rationale and the D1/D2 defects this replaced.
Path 2 is split into two phases:
| Phase | Gate | What it does |
|---|---|---|
| Phase 1 | Unconditional (all stage:<X>:complete non-paused non-cleanup items; also items with fabrik:awaiting-ci on wait_for_ci: true stages — they have no stage:<X>:complete until CI clears) |
checkDependencies() → checkReviewGate() → buildReviewThreadComments() / dispatchReviewReinvoke() → checkMergeabilityGate() → checkCIGate() / dispatchCIFixReinvoke() (review reinvoke and CI-fix reinvoke are mutually exclusive per poll cycle) |
| Phase 2 | fabrik:yolo (cfg or label) OR fabrik:cruise label OR stage auto_advance: true |
attemptMergeOnValidate() (yolo only) → findNewComments() deferral → advanceToNextStage() |
Phase 1 ensures inline PR review thread comments (from Copilot, Gemini, or human reviewers) are addressed and CI failures are fixed on all issues, not just yolo/cruise ones. Phase 2 keeps stage advancement gated as before. Items that dispatch a reinvoke in Phase 1 (review reinvoke or CI-fix reinvoke) skip Phase 2 on that poll cycle and are re-evaluated on the next poll.
Advancement invariant: Every code path that adds stage:X:complete must call advanceToNextStage() (or a well-defined deferral mechanism) in the same logical transaction. The board column is a derived view of label state; a column that diverges from a stage:X:complete label is a stuck issue. Authorized deferral mechanisms are:
fabrik:auto-merge-enabledpresent → advancement deferred tocheckAutoMergeConvergence(fires when GitHub auto-merge lands)fabrik:awaiting-cipresent → advancement deferred tocheckCIGate(fires when CI passes, R5)fabrik:pausedpresent → advancement deferred to the paused-item recovery loops (fire when the linked PR merges on the next poll)
Any new code path that adds stage:X:complete without one of these deferrals is a bug that will leave the board column stuck until a manual intervention.
Appendix B: Shallow Pre-Filter (Engine.poll() pre-filter + itemMayNeedWork())
Shallow pre-filtering is a two-pass process that avoids the expensive FetchItemDetails() GraphQL call for items that clearly don’t need work:
Pass 1 — Engine.poll() pre-filter (runs first, on every item in the board snapshot):
| Check | Passes If |
|---|---|
| cycleSet / cooldown pre-filter | Item is in cycleSet (a Store observer fired for a relevant change), OR has a bypass label (fabrik:awaiting-ci, fabrik:awaiting-review, fabrik:rebase-needed, fabrik:auto-merge-enabled, fabrik:revalidate — need per-poll evaluation because their unblocking events don’t bump updatedAt; see §9.8 for per-label rationale), OR cleanup stage (checks local filesystem), OR CooldownAt has an unconsumed expiry (periodic re-evaluation window — an expiry admits the item once, then is deleted; see “Cooldown Cache-Key Strategy”), OR item is not yet in the engine store. Items with an active CooldownAt and no other signal are suppressed. See “Cooldown Cache-Key Strategy” section in Appendix B below. |
Paused deep-fetch skip (runs after Pass 1 admits the item and after the cleanup-stage/FetchItemDetails split, inside selectDeepFetchCandidates — #1379, ADR-1379): if the item carries fabrik:paused and has no cycleSet entry (no observer-driven change this poll), the FetchItemDetails call is skipped — the item is still appended to deepFetchCandidates unfetched, so list-membership-only consumers (runValidatePRTerminalAdvance’s merged-while-paused self-heal, ADR-056 D2; settleRevalidateScan’s FR-5 guarantee) are unaffected. A paused item that also carries a bypass label (fabrik:awaiting-review, fabrik:rebase-needed, fabrik:auto-merge-enabled, fabrik:revalidate) is still skipped — pause wins over the bypass for the fetch decision, though the bypass label is still what got the item past Pass 1 in the first place. A paused item with a cycleSet entry (a real comment or label change since last poll) still gets fetched normally, so itemNeedsWork’s human-comment-resume check always sees fresh item.Comments. A paused item not yet recorded in the engine store gets a one-time baseline fetch regardless (mirrors Pass 1’s own not-in-store bypass). Unpausing re-admits within one poll, no restart required: board.Items[i].Labels is refreshed every poll by the shallow board-wide read (FetchProjectBoard) independent of this gate, so the moment fabrik:paused is removed on GitHub, this check reads false on the very next poll. Periodic re-evaluation reads through the cache (#1944, #2180, ADR 2180): the one exception to the skip is a paused item that is not in cycleSet and was admitted by an expired periodic-re-eval cooldown. It calls FetchItemDetails through the board cache. The cache’s freshness rule (§7.9 “Self-write staleness baseline”) sees a human reply that a self-write baseline would have masked, because GitHub’s issue comment total then differs from the cached comments, so the read costs nothing while the cache is genuinely fresh and refetches once when it is not. A long-parked item with a fresh cache is therefore still never fetched (#1379’s cost intent). Without a board cache every read is live and the skip is kept. This replaces the #1944 backstop (recentBaseline, a hand-applied DeepFetchInvalidated, ItemState.PausedBackstopBaseline, PausedBackstopServed, ADR 2096 item 5), which existed only because the baseline could hide a reply within about a second of the pause comment (0.0.83 release gate, TestBlockedOnInput). The same masking class affects review feedback on non-paused, stage-complete items (#2160): repairDispatcherBlindness (§6.6.6 “Dispatcher input agreement”) is kept as the gate-side guard; the PR-layer baseline above removes the updatedAt cause, but not the latestReviews cap or per-author collapse.
Pass 2 — itemMayNeedWork() (runs after the pre-filter admits the item, on shallow board data):
| Check | Passes If |
|---|---|
| Stage exists | FindStage(stages, item.Status) != nil |
| Closed issue | Not closed, OR cleanup stage, OR has stage:<X>:complete label |
| Repo write access | Repo not yet resolved by resolveRepoAccess (fail-open on “unknown”), OR resolved with CanPush: true. A repo cached with CanPush: false is never admitted, regardless of stage or status — see ADR-1347. Purely in-memory (no API call); the probe itself always runs earlier in the same poll cycle, in poll()’s seeding block. resolveRepoAccess’s probe branches on auth mode (#1750, ADR-1750): PAT mode is FetchRepoAccess’s permissions.push (unchanged, ADR-1347). App mode (e.ghAppAuth != nil) instead consults resolveAppRepoAccess against Engine.appAccessibleRepos — the pinned installation’s own GET /installation/repositories list, fetched once at startup (New()), since permissions.push is structurally meaningless for an installation token (GitHub always returns it false, regardless of actual grants). Ambiguous App-mode answers (the startup fetch never succeeded, or the list was truncated and the repo wasn’t found in it) fail open through the same if err != nil branch PAT mode’s probe errors already use — never a silent CanPush: false. A confirmed exclusion (App or PAT mode) additionally gets a persistent warnings.Record entry (Type: "repo_access"), not just the one-line startup log. A confirmed zero-repo installation is a hard startup refusal from New() instead of a per-repo determination. |
| Cleanup stage | Worktree exists on disk (local filesystem check only) |
| Deep-fetch failure cooldown | No recent FetchItemDetails failure, OR failure cooldown expired |
Note: itemMayNeedWork() intentionally does NOT check updatedAt, the cycleSet/cooldown gate, lock, editing, pause, or dependency labels — those are either handled by the poll.go pre-filter (Pass 1) or require the full label set from deep fetch and are checked in itemNeedsWork().
Cooldown Cache-Key Strategy
The engine uses two distinct in-memory stores (both in itemstate.Store) for cooldown tracking, split to prevent the #504 regression where invocation timestamps and periodic-re-eval cooldowns shared the same key:
StageState.LastAttemptAt[stageName]— written byStageAttemptedmutation when Claude actually runs for a stage. Read viasnap.LastAttemptAt(stageName). This is the “I already ran Claude on this stage” invocation gate.ItemState.CooldownAt[reason]— written byCooldownRecorded{Reason: reason}mutations for periodic re-evaluation gating. Read viasnap.HasActiveCooldown(now)/snap.HasExpiredCooldown(now)(iterate all reasons) orsnap.CooldownAt(reason)for a specific key. Active reason keys:"periodic-re-eval"(poll-rate throttle for all items),"dep-blocked"(dependency gate),"review-blocked"(review gate),"feedback-pending"(feedback gate, ADR-1953-a),"feedback-dispatcher-gap"(feedback gate found held review feedback the dispatcher’s cached view lacks;mergeGateRecheckInterval(), #2160),"spawn-resume-deferred"/"spawn-recovery-deferred"(spawn retry deferral),"merge-unsettled"(merge-gate claim, #2096),"retry-backoff"(stage retry backoff, #2096),"sentinel-live"(dispatch refused for a live cleared-worker process, #2096) and"archive-eligible-at"(an absolute “eligible since” cache for Done-stage archiving — not a gate, never consumed). This is the “don’t deep-fetch this item too frequently” cooldown; dispatch suppression for incomplete stages usesLastAttemptAt[stageName]instead.
What writes CooldownAt("periodic-re-eval"):
processItem()sets it when cleanup completes (cleanup-stage terminal path)- The deferred cache-write block in
Engine.poll()(invoked by thedoPollCycleclosure) sets it for non-advanced, non-cleanupdeepFetchCandidatesafter each full poll cycle — a belt-and-suspenders refresh that caps deep-fetch frequency to once per cooldown period. In-flight items (snap.Worker() != nil) are included: the cooldown prevents repeated deep-fetch evaluation (and the fallback GraphQL fetch when the cache is invalidated or disabled) for items the dispatch guard would block anyway. Prompt re-dispatch after the prior-cycle worker exits is guaranteed byWorkerExited → WorkerLifecycleChanged, which is inwakeChFlagsand adds the item tomayNeedWork, bypassing any active cooldown (#544). Thestage:X:completeterminal-only guard was removed in Phase 3-F:LastAttemptAt[stageName](notCooldownAt) now carries dispatch suppression for incomplete stages, so refreshingCooldownAt["periodic-re-eval"]for all non-cleanup items is safe regardless of completion state (#504 structural fix).
Expired cooldowns are consumed on admission (#2096, ADR-2096). An expired entry admits the item once: when the pre-filter admits a non-cleanup, in-store item that has an expired entry (on the strength of the expiry or of any other reason, including cycleSet), selectDeepFetchCandidates applies ExpiredCooldownsConsumed, which deletes every non-zero expired entry except archive-eligible-at. HasExpiredCooldown therefore reports only expiries the engine has not yet acted on (and ignores archive-eligible-at), which ends the old behaviour where one expired reason made the item admitted on every poll for ever (#2062). Every reader of CooldownAt outside admission tests only now.Before(t), so none depends on an expired entry persisting. Because the entry used to double as the periodic safety net, the net is re-armed explicitly: an item admitted and fetched is re-stamped by the poll() defer; an item admitted but dropped before joining deepFetchCandidates (itemMayNeedWork false, or a failed deep fetch) gets periodic-re-eval = now + githubRecheckInterval() at the drop site (rearmPeriodicReEval); an item advanced during the poll (advancedItems, #544) is skipped by the defer on purpose and is admitted next poll by its status change (StatusChanged puts it in cycleSet), not by a cooldown — the same path that always admitted an advanced item that had no cooldown entry. A failed non-Validate catch-up advance changes no status, so runCatchUpPhase2 leaves the item out of advancedItems; the defer then re-stamps periodic-re-eval and the advance is retried at the periodic interval.
What writes other CooldownAt reason keys:
CooldownAt("dep-blocked"):processItem()sets it viaCooldownRecorded{Reason: "dep-blocked"}each timecheckDependencies()returns true — blocked on dependenciesCooldownAt("review-blocked"):checkReviewGate()(catch-up loop) sets it viaCooldownRecorded{Reason: "review-blocked"}when the review gate blocks — ensures Phase 1/Phase 2 reprompt timers fire via the cooldown retry path even when noupdatedAtchange occursCooldownAt("merge-unsettled"):handleMergeAndCIGates(catch-up Phase 1) sets it viaCooldownRecorded{Reason: "merge-unsettled"}when the merge gate claims an item on transient state (PRMergeUnsettled/PRMergeQueued, themergeBlockedbranch only — never a conflict), expiring atnow + mergeGateRecheckInterval()(PollSeconds / 2, floored at 1s, deliberately notgithubRecheckInterval()). The claim applies nofabrik:awaiting-ci, and CI completion moves neither a label norupdatedAt, so without this the only re-admission was the periodic re-evaluation,10 × PollSecondslater (30 minutes atpoll: 180, #1943). The entry expires before the next poll, admits the item once, and is consumed — a persisting claim re-records it each poll.
What bypasses the cooldown gate (returns true regardless of cooldown):
fabrik:awaiting-cilabel: CI check-run completions don’t bumpupdatedAt, so forced re-evaluation is necessaryfabrik:awaiting-reviewlabel: review-submission webhooks don’t bumpupdatedAt; forced re-evaluation ensures gate clearance within one poll cycle of webhook receipt (issue #616)fabrik:rebase-neededlabel: base-branch advances don’t bumpupdatedAtstage:X:completelabel is ABSENT and cooldown has expired: retry for genuinely incomplete stages
What suppresses the cooldown gate (returns false despite expired cooldown):
stage:X:completelabel is PRESENT in shallow labels ANDfabrik:awaiting-reviewis absent: completed stages with no pending review need no retry (introduced in #488 to fix perpetual deep-fetch loop). Items with bothstage:X:completeandfabrik:awaiting-reviewbypass the cooldown gate entirely (viahasAwaitingLabelinpoll.go) and are retried every poll cycle so Phase 1/Phase 2 timers can fire — not merely every cooldown period (updated in #616).
Root-cause fix (#488): Terminal items (cruise+Validate complete, paused+complete, closed-with-stage-complete) triggered a perpetual deep-fetch loop: CooldownAt("periodic-re-eval") was only written by processItem() when work actually ran, so after cooldown expiry, itemMayNeedWork() returned true on every poll cycle indefinitely — each producing a no-op deep-fetch that did not update the cooldown. The fix has two parts: (1) primary — check stage:X:complete in shallow labels before returning true from the cooldown-expired branch, with an exemption for fabrik:awaiting-review items (which also carry stage:X:complete but need periodic re-evaluation for Phase 1/Phase 2 timers); (2) belt-and-suspenders — the deferred block in Engine.poll() sets CooldownAt("periodic-re-eval") for non-advanced deepFetchCandidates after each full cycle, capping deep-fetch frequency to once per cooldown period for all items (not just terminal ones). Phase 3-F structural fix (#504): The original Part 2 was scoped to terminal items only (stage:X:complete present in the full label set) because refreshing incomplete-stage entries reset the cooldown timer and prevented retries from firing. Phase 3-F removes this constraint: LastAttemptAt[stageName] (written only by StageAttempted, never by observation) now carries dispatch suppression independently of CooldownAt, so refreshing CooldownAt["periodic-re-eval"] for ALL non-cleanup items is safe — incomplete stages still retry after their LastAttemptAt cooldown expires, regardless of the CooldownAt["periodic-re-eval"] entry.
Appendix C: Guard Evaluation in itemNeedsWork() (Full Filter)
itemNeedsWork() runs after FetchItemDetails() has populated comments and the full label set.
| Priority | Check | Passes If |
|---|---|---|
| 1 | Stage exists | FindStage returns non-nil |
| 2 | Closed issue | Not closed, OR cleanup stage, OR has stage:<X>:complete |
| 3 | Cleanup stage | Not paused, not complete, worktree exists |
| 4 | Locked by other | No fabrik:locked:<other> label |
| 5 | Awaiting input | isAwaitingInput() true AND new human comments created at/after the pause exist (resumeAuthorised()) |
| 6 | Paused | Not paused, OR paused with new human comments created at/after the pause (resumeAuthorised()) |
| 7 | New comments | Any unprocessed comments → true |
| 8 | PR item | Not a PR (PRs only support comments, checked after comment check) |
| 9 | Stage complete | No stage:<X>:complete label |
| 10 | Cooldown | Not attempted, OR cooldown expired |
Appendix D: In-Memory Board Cache Lifecycle
The engine always maintains an in-memory cache of board state in boardcache.CacheImpl, wired as e.readClient for production runs (tests using engine.NewWithDeps bypass the cache and use boardcache.GitHubAdapter directly). The cache is the unified source of truth for downstream engine logic and reactive observers (PushUnblockObserver, StageChangeObserver, etc.) regardless of whether webhooks are configured. This appendix describes the cache lifecycle, delta semantics, reconciliation, and stream-health failover.
D.1 Bootstrap
The cache is bootstrapped via ProbeProjectBoard → BootstrapFromProbe. Both startup paths use the same probe-based cold-start:
-
Webhook startup path — when
--webhooksis enabled, beforewm.Start()is called (before the webhook listener binds), the engine callse.client.ProbeProjectBoard(...)and feeds the results tocacheImpl.BootstrapFromProbe(probeItems, projectID). AfterBootstrapFromProbereturns,e.seedTerminalFromProbeItems(probeItems)seedsTerminal=truefor qualifying items (see below). This pre-bootstrap closes the gap between webhook subprocess start and the first poll cycle, so deltas don’t land in an empty cache. If the probe returns zero items (transient indexer hiccup), the cache is left virgin and the first poll retries. -
First-poll path — when the cache is not yet bootstrapped (webhooks disabled, or webhook-startup probe failed), the first
poll()cycle callsProbeProjectBoardand passes the results toBootstrapFromProbe, then callsseedTerminalFromProbeItems. On subsequent polls (cache already bootstrapped), the probe-and-refresh path (runProbeAndDeepFetch) replaces the bootstrap; see D.4.
BootstrapFromProbe calls Store.Reset with synthetic items constructed from probe results. After Store.Reset, the engine calls seedTerminalFromProbeItems, which applies TerminalFlagSet{Terminal: true} for each item that is closed, in a cleanup stage (CleanupWorktree: true), and has no on-disk worktree. The worktree-presence check prevents stranding cleanup work: if the worktree still exists (the Done stage’s cleanup_worktree action hasn’t run yet), the item must not be treated as terminal — it must proceed through processItem so cleanup can execute. Seeding the terminal flag for worktree-absent items prevents runProbeAndDeepFetch from calling FetchItemDetails on already-cleaned Done items. Deep fields (comments, linked PR data) are not populated at bootstrap; they are fetched lazily on the first active-item probe cycle.
Store.Reset fires observer notifications for every item (one Change per item, with non-zero Fields). mayNeedWorkObserver is always registered in Engine.Run() before bootstrap and is the mechanism that makes bootstrapped items visible to the dispatch loop via the next ticker poll — no external webhook event is needed to unblock them. wakeChObserver is also registered before bootstrap, but only when wakeCh != nil; in headless runs there is no wake channel, so visibility of bootstrapped items relies solely on mayNeedWorkObserver and the next ticker-driven poll (within PollSeconds).
If the probe fails (transient network error), the cache stays unbootstrapped and the next poll cycle retries. No data is lost. Labels are not available from probe results; they are populated by FetchItemDetails on the first active-item deep-fetch cycle.
D.2 Delta Application
Every verified webhook payload is passed to cacheImpl.ApplyDelta(eventType, payload []byte) before the poll loop is woken. ApplyDelta is a no-op when IsPaused() returns true. Otherwise it dispatches to a typed handler.
D.2.1 Unknown-item fallback
When a delta handler looks up an item that is not yet in the cache (e.g., issues.labeled arrives before issues.opened), the handler calls ensureIssueInStore(owner, fullRepo, issueNumber). This helper:
- Fast path: if the item is already in the Store, returns immediately.
- Miss path: calls
fallback.FetchProjectItem(owner, repo, issueNumber)via REST GET/repos/{owner}/{repo}/issues/{number}, appliesIssueOpened{Item: *pi}to the Store, then continues with the original delta.
This resolves the “fabrik went deaf” bug class where webhooks for new issues were silently dropped because the issue had not yet been seen in a board reconcile.
Exception: issues.opened itself creates the item from the webhook payload directly (no REST call needed — the payload contains full issue data). issues.transferred and issues.deleted remove the item rather than ensuring it.
D.2.2 Webhook event coverage table
All handlers hold the write lock for their mutation only. No lock is held during network calls (the ensureIssueInStore fallback fetch and resolvePRLinkage auto-heal are performed without holding c.mu — ADR 037 lock-ordering invariant).
PR→issue auto-heal and the authoritative mapping path: When a pull_request, pull_request_review, pull_request_review_comment, or check_run webhook arrives for a PR whose linked issue is not yet indexed in store.prToKey, the handler calls resolvePRLinkage to determine which issue the PR closes. resolvePRLinkage first checks store.GetByPRKey for an authoritative engine-recorded mapping. Only when no authoritative mapping is present does it fall back to fetching the PR body via REST and scanning it with the reClosingKeyword regex. The authoritative mapping is written by CacheImpl.RecordPRLinkage, which the engine calls immediately after CreateDraftPR succeeds (and also when an existing PR is found on restart), so the index is populated before any webhooks arrive for the new PR. This eliminates mismaps from prose issue references in PR bodies (e.g., “before fixes #598 landed, Closes #602” would have matched #598 first without the authoritative path).
| Event type | Action | Handler decision | Cache mutation | Engine reaction |
|---|---|---|---|---|
issues |
opened |
Create item from payload | IssueOpened (builds item from full webhook payload; no API call) |
Item appears in next FetchProjectBoard |
issues |
closed |
Fallback if missing; update state | IssueClosed |
IsClosed=true; itemMayNeedWork skips unless cleanup stage |
issues |
reopened |
Fallback if missing; update state | IssueReopened |
IsClosed=false; re-enters dispatch |
issues |
transferred |
Remove item | store.Remove (cleans prToKey index automatically) |
Item gone from next board fetch |
issues |
deleted |
Remove item | store.Remove (cleans prToKey index automatically) |
Item gone from next board fetch |
issues |
edited |
Fallback if missing; update title+body | IssueEdited |
Title/Body updated; next FetchItemDetails serves updated body |
issues |
assigned |
Fallback if missing; update assignees | IssueAssigneesUpdated (full list replace) |
Assignees updated |
issues |
unassigned |
Fallback if missing; update assignees | IssueAssigneesUpdated (full list replace) |
Assignees updated |
issues |
labeled |
Fallback if missing; add label | IssueLabeled |
Labels updated; poll woken for next dispatch |
issues |
unlabeled |
Fallback if missing; remove label | IssueUnlabeled |
Labels updated; poll woken |
issues |
milestoned, demilestoned, locked, unlocked, pinned, unpinned |
No-op | — | No engine state depends on these fields |
issue_comment |
created |
Guard: item must be in Store; append comment | IssueCommentCreated |
Comment appears in next FetchItemDetails; poll woken for comment processing |
issue_comment |
edited, deleted |
No-op | — | Reaction-based state machine reads reactions not bodies; next deep-fetch heals |
pull_request |
opened, closed, reopened, synchronize |
Look up issue via store.GetByPRKey (authoritative mapping from RecordPRLinkage); auto-heal via resolvePRLinkage (REST + regex) if not found; update SHA and PR details |
PRHeadSHAUpdated + PRDetailsUpdated; DeepFetchInvalidated on auto-heal |
shaToKey + prToKey updated; CI gate can evaluate this SHA; LinkedPR.Title/State/Merged/Draft populated |
pull_request |
ready_for_review |
Look up linked issue via store.GetByPRKey; read current LinkedPR snapshot; apply Draft=false |
PRDetailsUpdated{Draft: false} (preserves Title/State/Merged from snapshot) |
LinkedPR.Draft=false; PR no longer draft; review bots can see it |
pull_request |
converted_to_draft |
Look up linked issue via store.GetByPRKey; read current LinkedPR snapshot; apply Draft=true |
PRDetailsUpdated{Draft: true} (preserves Title/State/Merged from snapshot) |
LinkedPR.Draft=true; PR back to draft |
pull_request |
review_requested |
Look up linked issue via store.GetByPRKey; replace reviewer list |
PRReviewRequested (full list replace) |
LinkedPRReviewRequests updated; review gate re-evaluated |
pull_request |
review_request_removed |
Look up linked issue via store.GetByPRKey; remove one reviewer |
PRReviewRequestRemoved (remove by login) |
LinkedPRReviewRequests updated |
pull_request |
labeled, unlabeled, assigned, unassigned, edited |
No-op | — | PR-level labels/assignees/metadata not tracked; Closes #N linkage healed by next Reconcile |
pull_request_review |
submitted, edited, dismissed |
Route via store.GetByPRKey; auto-heal if missing; all three actions carry the full review object and are processed identically (upsert by DatabaseID) |
PRReviewSubmitted + PRReviewRequestRemoved (side effect); PRHeadSHAUpdated + DeepFetchInvalidated on auto-heal |
LinkedPRReviews updated (state stored as-is, uppercased); LinkedPRReviewRequests also updated (reviewer removed) — covers cases where review_request_removed webhook is not fired (e.g. Copilot edited reviews); review gate re-evaluated; DISMISSED state excludes the review from hasReviews in checkReviewGate |
pull_request_review_comment |
created |
Route via store.GetByPRKey; auto-heal if missing |
ReviewThreadCommentAdded; DeepFetchInvalidated |
Thread comment appears; review reinvoke path can see it |
pull_request_review_comment |
edited, deleted |
No-op | — | Thread comment content not decision-relevant; next deep-fetch heals |
check_run |
completed |
Upsert in checkRuns[sha]; index via shaToKey; auto-heal if SHA unknown |
CheckRunCompleted; PRHeadSHAUpdated + DeepFetchInvalidated on auto-heal |
CI gate evaluates conclusion; LinkedPRCheckRuns updated |
check_run |
created, rerequested, requested_action |
No-op | — | Only terminal state (completed) matters for CI gate |
check_suite |
completed, requested, rerequested |
No-op | — | Check suites are coarse aggregates; individual runs tracked via check_run.completed. The store holds no suite state, so the suite-aware CI gate (#1822, §6.4) reads suites live via FetchCheckSuites instead of relying on this webhook |
projects_v2_item |
created |
Call FetchItemDetails by content_node_id; apply as new item |
IssueOpened |
Item appears in board; triggers dispatch on next poll |
projects_v2_item |
edited |
Update Status via itemIDToKey reverse lookup |
ProjectV2ItemEdited |
Status updated; stage selection uses new column |
projects_v2_item |
deleted, archived |
Remove item | store.RemoveByItemID (cleans prToKey index automatically) |
Item gone from next board fetch |
projects_v2_item |
restored |
No-op | — | Next Reconcile re-adds the item from the full board snapshot |
projects_v2_item |
reordered |
No-op | — | Column order not modeled in Fabrik’s cache or engine |
D.3 Cache Read Semantics
CacheImpl implements boardcache.ReadClient. When the poll loop calls a read method:
| Method | Cache behavior |
|---|---|
FetchProjectBoard |
Returns reconstructed board from items map; falls back to GitHub when cache is empty |
FetchItemDetails |
Serves deep fields from cache when deepFetched[key] is true; falls back to GitHub and populates on miss; logs [cache] miss for #N — fetching from GitHub |
FetchCheckRuns |
Serves from checkRuns[sha]; falls back and caches on miss |
FetchLinkedPR |
Cache hit when snap.LinkedPR().Title != "" (set by PRDetailsUpdated); reconstructs *gh.PRDetails from Store snapshot; falls back to GitHub REST on miss and applies PRDetailsUpdated write-back |
FetchPRMergeable |
Always delegates to fallback (mergeability changes without webhook events) |
FetchPRMergeableState |
Always delegates to fallback |
FetchLabels |
Serves from item.Labels when item is cached; falls back on miss |
FetchStatusField |
Always delegates to fallback (static field metadata) |
RateLimitStats |
Always delegates to fallback |
D.4 Reconciliation
Reconciliation operates at two levels:
Periodic status-only sweep (Layer 2)
At the top of every poll() cycle (~15 s), when the in-memory cache is bootstrapped (cacheImpl != nil and cacheImpl.ProjectID() != ""), the engine runs a two-step gate:
-
Gate step —
e.client.FetchProjectUpdatedAt(projectID): fetches the singleupdatedAtfield for the project node (node(id:$id){ ...on ProjectV2 { updatedAt } }). Cost: ~1 GraphQL point per poll cycle. If the returned timestamp equalse.lastProjectUpdatedAt(no project-level change since last cycle), the batch is skipped entirely. -
Batch step — fired only when the timestamp has advanced:
e.client.FetchProjectItemStatusBatch(projectID)returnsitemNodeID → statusNamefor every board item (paginated in pages of 100, no nested fields). The result is passed tocacheImpl.ApplyStatusBatch(updates).
ApplyStatusBatch holds the write lock for one pass:
- For each
(itemID, newStatus)pair, look up the cache key viaitemIDToKey. - If the status differs from the cached value, update
StatusandUpdatedAt. Unchanged items are skipped (no observer notifications). - Unknown item IDs (not yet bootstrapped) are silently skipped.
After a successful batch step, e.lastProjectUpdatedAt is updated to the new timestamp so the next poll cycle can compare correctly.
First poll cycle: e.lastProjectUpdatedAt starts at the zero value (time.Time{}). The first real updatedAt returned by GitHub is always later than zero, so the batch fires on the first cycle. This catches any Status drift that occurred before engine start.
GitHub API limitation: GitHub’s ProjectV2.items does not support a since filter or orderBy: UPDATED_AT. The full batch (FetchProjectItemStatusBatch) is the only way to identify which specific items changed once the gate fires. The Store’s existing diff in ApplyStatusBatch de-dupes unchanged-Status items so no spurious observer notifications are emitted.
Gate placement: The gate runs before e.readClient.FetchProjectBoard(...) in poll(), so the cache holds the latest Status values when the board is read and items are dispatched in the same cycle.
Gate inactive when: e.readClient is not a *boardcache.CacheImpl (non-cache mode), or cacheImpl.ProjectID() is empty (bootstrap not yet complete).
Per-poll probe (runProbeAndDeepFetch)
After the Layer 2 status gate above, every poll() cycle on a bootstrapped cache calls runProbeAndDeepFetch(cacheImpl). This replaces the previous unconditional full-shallow FetchProjectBoard + Reconcile call, reducing per-poll GraphQL cost ~5–10× on idle boards by eliminating the labels(first:30) and closedByPullRequestsReferences(first:5) nested connections.
The probe loop:
-
Calls
e.client.ProbeProjectBoard(...)— fetches only scalar fields per item plusclosedByPullRequestsReferences(first:1)for linked-PR drift detection. Nolabelsconnection at any nesting level. -
For each probe item, computes
effectiveUpdatedAt = max(issue.updatedAt, projectItem.updatedAt, linkedPR.updatedAt). - Stage-membership guard: checks
stages.FindStage(e.cfg.Stages, pi.Status).- New item (not yet in store): if the column has no matching Fabrik stage, the item is skipped entirely — no
store.Apply, noFetchItemDetails. It is still recorded in thenewKeystracking set before the guard fires, so it is not evicted by the post-loop tombstoning pass. - Existing item (already in store):
ProbeBoardItemUpdatedis applied soStatus,IsClosed, and related fields stay current (important when items drift into or out of unconfigured columns), butFetchItemDetailsis skipped.
- New item (not yet in store): if the column has no matching Fabrik stage, the item is skipped entirely — no
-
New item (not in store): seeds minimal state via
IssueOpened, then callsFetchItemDetailsunconditionally to populate labels and deep fields. -
Linkage drift (
probe.linkedPRNumber ≠ cached LinkedPR.Number) — on a warm cache (LastDeepFetchAtset), appliesDeepFetchInvalidatedto resetLastSeenSourceUpdatedAt, then falls through to the staleness check. The check is subject to three rules (#2080, ADR 2080):- Terminal items are exempt (R2). An item that is flagged
Terminalin the same cleanup stage, or is closed in a cleanup stage with no worktree on disk (the quietprobeOnlyTerminalpredicate — the same test asisProbeOnlyTerminal, without its log lines), skips the drift check entirely: nothing about its linkage changes Fabrik’s work. This runs ahead of the check, not after it. - A (cached, probe) pair invalidates once (R1). The engine-local
probeDriftLedger(engine/probe_drift.go) records the pair last invalidated per item. The probe’s0is ambiguous — it means “no PR” or “a PR thatclosedByPullRequestsReferencesomits” (a closed, unmerged PR, which neither the probe nor the deep fetch returns; a REST lookup finds it and writes it to the cache) — so a deep fetch can legitimately leave the cached number in place. A repeat of the same pair is therefore not invalidated again and is logged once (linkage disagreement persists after deep fetch); a different pair (a genuine PR A → PR B, or 0 → N) invalidates once; when cached and probe agree the pair is forgotten, so returning to an earlier pair invalidates again. The cachedLinkedPRis deliberately left untouched — a deep-fetch0is not authoritative (it would breakbase:items and PRs found through REST). The ledger is in memory, pruned when the item leaves the board or turns terminal; a restart costs one extra invalidation per affected item. - A loop is reported (R4). If one item causes more than
probeDriftLoopThreshold(10) invalidations withinprobeDriftLoopWindow(1 h, one.now()), oneprobe: linkage drift loopwarning names the item and both PR numbers. It re-arms once the count falls back to the threshold.
base:
<branch>suppression:closedByPullRequestsReferencesis only populated by GitHub for PRs targeting the repository’s default branch, so the shallow probe structurally always reportsLinkedPRNumber == 0for an item whose linked PR targets a non-defaultbase:<branch>, even while the warm deep cache correctly holds the real PR number (resolved via the base-independent linkage path). To avoid perpetual false-positive invalidation on every poll, the drift check is suppressed — left as a true no-op, noDeepFetchInvalidatedand noPRDetailsUpdated— when all three hold: the probe reportsLinkedPRNumber == 0, the cachedLinkedPR.Numberis non-zero, and the cached item (read from the Store’sItemState.Labels, never from the probe item — probe items carry noLabelsfield) satisfiesitemHasBaseLabel. A probe reporting a different non-zero PR number on a base-labeled item is still treated as genuine drift and invalidates as usual. Default-branch behavior (nobase:label) is unaffected — the suppression can never fire sinceitemHasBaseLabelis false. - Terminal items are exempt (R2). An item that is flagged
-
Applies
ProbeBoardItemUpdated(updatesIsClosed,State,IsPR,Status,UpdatedAt; explicitly skipsLabelsto preserve the cached label set populated by prior deep-fetches or webhook deltas). -
Staleness check: calls
cacheImpl.IsItemCacheFresh(repo, number, boardcache.EvidenceFromProbe(pi)). The cache is fresh only if the issue/project-itemupdatedAtis not afterLastSeenSourceUpdatedAt, the linked PR’supdatedAtis not afterLastSeenPRUpdatedAt, and GitHub’s issue comment total equals the cached issue comment count (the three-part rule under “Self-write staleness baseline” below, #2180). If fresh, continues without GitHub traffic. If stale, callsFetchItemDetailsto refresh. - After the item loop, removes from the store any items no longer present in the probe result (they left the board). A guard skips removal when the probe returns 0 items and the cache is non-empty (transient indexer hiccup).
FetchItemDetails writes ItemDeepFetched to the Store, updating LastSeenSourceUpdatedAt to the new effectiveUpdatedAt and LastSeenPRUpdatedAt to the linked PR’s own updatedAt. The candidates loop that runs later in the same poll cycle sees these items as fresh (via IsItemCacheFresh) and skips duplicate fetches.
Self-write staleness baseline (SelfWriteObserved)
The issue-layer part of the staleness check in step 7 compares the issue/project-item updatedAt against LastSeenSourceUpdatedAt, which is otherwise written only by a full deep-fetch (ItemDeepFetched). Without a second writer, every one of Fabrik’s own board mutations — which bump the real GitHub updatedAt just like an external edit — would make the very next probe cycle see the item as stale and force an immediate FetchItemDetails, even though nothing actionable changed (the #1083 incident: a self-posted reply forced its own re-fetch on the next poll, running the ping-pong loop as fast as polling allowed instead of backing off).
itemstate.SelfWriteObserved{Repo, Number} closes this gap: applied via e.store.Apply(...) directly (bypassing boardcache.CacheImpl, so it needs no cache-wired guard and is not gated on e.webhookMgr), it advances LastSeenSourceUpdatedAt to the local wall-clock time at the moment of the self-write — no deep-fetch, no other field touched. The advance is monotonic (store.go’s applyToItem only updates when time.Now() is after the current baseline), so it never regresses an earlier SelfWriteObserved, and a DeepFetchInvalidated reset (zero value) always “wins” against a stale SelfWriteObserved racing behind it. Only SelfWriteObserved is monotonic: ItemDeepFetched assigns the fetched value unconditionally, which is how a real refetch lowers a clock-anchored baseline.
It is wired into exactly five call-site groups, mirroring the existing webhook RegisterEcho/RegisterEchoIfSubscribed success-gating at each:
- Label add (
engine/mutate.gosyncLabelAdd) — unconditional; reaching the call site already impliesAddLabelToIssuesucceeded. - Label remove (
engine/mutate.gosyncLabelRemoval) — gated onecho, the same signal that suppresses the webhook echo on agh.ErrNotFoundno-op removal. - Comment post (
engine/mutate.gopostComment) — unconditional;AddCommentfailure returns before this point. - Issue body edit (
engine/comments.gopublishCommentOutput) — insideUpdateIssueBody’s success branch. - Project board status move (
engine/stages.goadvanceToQueuedandadvanceToNextStage,engine/no_work_needed_settle.go,engine/closed_item_advance_settle.go) — inside each site’sUpdateProjectItemStatussuccess branch.
A failed mutation at any of these sites never advances the baseline, so the next probe still correctly treats the item as stale. And because the mechanism only ever advances the baseline to the current wall clock, a genuine external change that lands with a later effectiveUpdatedAt (a human/bot comment, label, or status change arriving after our self-write) still compares as stale and triggers a real deep-fetch — the suppression only ever applies to the self-write’s own updatedAt bump, never to activity after it.
Display-field writes (status line, Last activity, Last run) are deliberately not call sites (#2180): the board and probe parsers already discount a project-item bump explained by a display field’s own updatedAt, so those writes move no updatedAt the cache compares and anchoring the baseline would only hide an external change.
Known scope boundary: this covers only the call sites above. PR-body self-writes (engine/pr.go’s updatePRVerification/ensurePRLinksIssue, engine/prcreate.go’s linkage-heal edit) and a second issue-body-edit site (engine/item.go’s stage-output publishing path) bump a component of effectiveUpdatedAt the same way but are not wired to SelfWriteObserved — a deliberate, documented gap (see ADR-044 Addendum 4), not an oversight, left as a candidate follow-up if it proves to matter in practice.
The freshness rule: self-write anchoring may absorb only what the engine can account for (#2180, ADR 2180)
One scalar compared with one scalar cannot tell “my write at T” from “an external change at T−ε plus my write at T”, so SelfWriteObserved alone masked external changes that predate it (the #2125 shape). boardcache.cacheIsStale(Evidence, ItemState) therefore serves a cached deep copy only when all three hold:
- Issue layer.
Evidence.IssueLayer(the later of the issue’s and the project item’supdatedAt, display-field bumps discounted, linked PR excluded) is not afterLastSeenSourceUpdatedAt. This is the baselineSelfWriteObservedadvances and the only layer the engine writes. - PR layer. The linked PR’s own
updatedAtis not afterItemState.LastSeenPRUpdatedAt, set only from GitHub values byItemDeepFetchedand zeroed byDeepFetchInvalidated.SelfWriteObservednever touches it, so an issue-side self-write cannot hide a review, thread comment or PR comment. - Comment evidence. GitHub’s
comments { totalCount }for the issue equals the number of issue comments in the cached copy (FromPR == 0, no path), unless the cached copy already holds a first page (100). The engine’s own comments are written through the cache (LocalCommentAdded, idempotent byDatabaseID), so a self-write keeps the two equal; a human comment hidden by the issue-layer baseline makes them differ.
A linked PR on the board that the cache has no record of (no PR baseline and no cached PR) means a PR appeared after the fetch and reads as stale. When a producer did not capture the layers (zero IssueLayer, or no PR baseline for a cached PR) the predicate compares the overall updatedAt with LastSeenSourceUpdatedAt as before; a zero overall updatedAt is fresh (no signal, no hot-loop). The evidence rides ProjectItem/BoardProbeItem (IssueLayerUpdatedAt, LinkedPRUpdatedAt, IssueCommentTotal/IssueCommentTotalKnown; the board and probe queries select comments { totalCount } on the Issue fragment) and is mirrored on ItemState. A webhook stamp (localDeltaAt) raises the issue layer too. Every refetch re-anchors all baselines and rebuilds the comment list, so a mismatch cannot persist across polls: engine self-writes on the issue layer cost no refetch (the #1090 guarantee), and an engine write on the PR side (pr.go, prcreate.go, a push) costs one deep fetch at the next probe.
CI is outside updatedAt. A check completion does not change any updatedAt the board reports, so no baseline rule can reveal it. boardcache.FetchCheckRuns serves a cached set on trust only when it classifies READY; PENDING and FAILED sets are confirmed live and written back, and a PENDING set confirmed within 15 s (checkRunsConfirmTTL) is served, so several readers in one poll cost one REST call. This subsumes the #1303 per-poll prime that settleAwaitingCIScan used to run (RefreshCheckRunsLive, removed).
Cost. The added comments { totalCount } selection measured 0 extra points on the dev board’s listing queries (probe 1 point per page of 100, board listing 2 points per page, with and without it).
Backstops. The #1944 paused-item backstop is removed (§7.9 paused deep-fetch skip); the ADR 2160 gate repair (§6.6.6) is kept because it also guards causes this rule cannot (the latestReviews cap, per-author collapse).
Terminal-item skip
An item is terminal once Fabrik has confirmed it has nothing left to do: its status is a cleanup stage (e.g. Done), its labels include stage:<StageName>:complete, and no transient lifecycle label (transientLifecycleLabels — see “Transient label sweep” above) or lock label (fabrik:locked:*) is present.
Terminal state is stored as a Terminal bool field in ItemState and set/cleared via the TerminalFlagSet mutation.
When the flag is set: after a successful FetchItemDetails call in either runProbeAndDeepFetch or the admit pass, if isTerminalPredicate(labels, status, stages) returns true. Logged once (first-time only) at [poll] terminal flag set.
When the flag is cleared:
applyProbeItemclearsTerminalwhenever the item’sStatuschanges (probe shows the item left the cleanup column).applyShallowItemclears it on any status change during reconciliation.applyProjectV2ItemEdited(Layer 2 status sweep) clears it when the board status changes.LocalStatusUpdatedclears it when the engine itself changes the item’s status.- The probe loop’s terminal gate and the admit pass terminal gate each clear the flag explicitly and log
[poll] terminal flag cleared (status drifted to ...)when the probe shows the item in a non-cleanup stage.
Probe-loop skip: the terminal gate in runProbeAndDeepFetch runs before ProbeBoardItemUpdated is applied (ordering constraint: the cached Terminal flag must be read before applyProbeItem would clear it on a status change). If Terminal == true and the probe’s status is still a cleanup stage, the item is skipped without any deep-fetch. If Terminal == true but the probe shows a non-cleanup status, the flag is cleared and the item falls through to normal probe processing. The linkage-drift check (step 5 above) now consults the same terminal condition first (R2, #2080), so a terminal item is never invalidated by drift on its way to being skipped.
Admit-pass skip: the same gate in the per-poll admit pass (before the cycleSet check) unconditionally skips items that are terminal and still in the cleanup stage. This guard is redundant for the normal probe path (where runProbeAndDeepFetch already handled the item) but covers non-cache and paused-cache modes.
Cold-start (startup scan): after runStartupTransientLabelScan removes stale transient labels, runStartupTerminalScan iterates all items in the Store and applies TerminalFlagSet{Terminal: true} to any that pass the terminal predicate. This uses bootstrap label data already present in the Store — no GitHub API call is needed. On a 362-item board with 340 terminal Done items, this eliminates ~340 deep-fetches on the first poll cycle.
Reconcile is drift-recovery only: cacheImpl.Reconcile(board) is called only during drift-recovery, independent of webhook mode (see below). It is no longer called on every poll cycle, and it is not called during cold-start bootstrap — that path uses BootstrapFromProbe instead.
Full reconcile — drift-recovery (runs independent of webhook mode)
LightReconcile returns the fresh board snapshot when drift is detected, which the reconcileTicker goroutine passes directly to cacheImpl.Reconcile(board) — avoiding a second API call. The older full-fetch-and-reconcile helper (reconcileCache) has been removed; reconcileLoop’s LightReconcile path is the sole reconciliation mechanism.
Reconcile performs a deep partial update:
- For each item in the fresh board snapshot: update shallow fields (
Status, shallowLabels,UpdatedAt, etc.) in-place. Deep fields (Comments,LinkedPR*, etc.) are preserved from the existing cache entry to avoid triggering a burst of FetchItemDetails calls after each reconciliation. - Items present in the cache but absent from the new snapshot are removed (they were archived or moved off the board).
itemIDToKeyandshaToKeyare rebuilt.- Logs
[reconciliation] N items differedat the end.
Engine-write protection (#2159, ADR 2159). A listing is authoritative about changes made outside the engine, not about the engine’s own writes made after the listing was taken — and GitHub’s board listing lags writes by seconds (the #1974 probe measures it). Without protection a reconcile that ran 1–2 s after an engine label write received the pre-write listing, counted the difference as drift and put the old value back (observed on 2026-10-11: fabrik:awaiting-ci back on #2091, fabrik:awaiting-landing-verification missing from #2097, stage:Review:complete missing from #2116), leaving every gate that reads the cache wrong until the next tick (~3 min) and emitting a false unknown-actor label event.
- Record.
ItemState.EngineWriteAt(in memory only; cleared on restart,Resetand item removal) mapslabel:<name>,statusandclosedto the time the engine last wrote that field through the cache write-through (LocalLabelAdded/LocalLabelRemoved/LocalStatusUpdated/LocalIssueClosed, i.e. after the REST call succeeded — a failed write is never recorded). The webhookIssueClosedand all other mutations record nothing. Entries older than the grace margin are pruned on every later record and every reconcile apply. - Listing fetch time.
LightReconcilestampsProjectBoard.FetchedAtwith the instant just beforeFetchProjectBoardstarts and snapshots the cache after the fetch, so a write made during the fetch is visible to the comparison. The set of keys cached before the fetch is recorded separately, and only those can count as “cached but not on the board” drift — an item that enters the cache while the fetch is in flight cannot be on the older listing. A zeroFetchedAt(any other caller ofReconcile) means “unknown”: no protection, the listing wins as before. - Rule (
itemstate.protectShallow, shared by the apply step and drift detection so they cannot disagree). A field with a record at W is protected whenFetchedAt < W + grace: the cache’s current value is kept — presence of that one label,Status, orState/IsClosed— and the listing’s differing value is ignored. The grace (itemstate.DefaultEngineWriteGrace, 30 s) covers listing lag beyond “the fetch was in flight during the write”. “Current value”, not “the value the engine wrote”, means a later webhook change (a human unlabelling after the engine labelled) keeps winning over a stale listing. Labels the engine never wrote, and other labels on the same item, reconcile normally; title, URL, milestone andUpdatedAtare applied as before. - When protection ends. A record is dropped when a listing fetched at or after the write agrees with the cache (confirmed), or when it expires (W + grace has passed). A listing fetched after W + grace that contradicts the write — an external removal, a missed webhook — wins, so real drift is corrected by the next tick as before; a lost webhook cannot leave the cache stale beyond the grace plus one tick.
- Events. The decision is made inside the Store before any label diff, so a kept write produces no
LabelDeltasand no channel event. A redundant engine write or a dropped record alone is not aChange(the bookkeeping map is excluded from no-op detection;EngineWriteRecordedChangedis not a wake flag). - Probe listing (#2178, ADR 2178). The probe loop’s apply (
ProbeBoardItemUpdated→applyProbeItem) honours the same rule forStatusandState/IsClosed.runProbeAndDeepFetchstampsProbeBoardItemUpdated.FetchedAtwithtime.Now()just beforeProbeProjectBoard(wall clock, note.now(): the Store stamps writes withtime.Now()); the decision is made inside the Store under its mutex throughprotectStatusClosed, the Status/closed half ofprotectShallow(which composes it withprotectLabels), and the sameDefaultEngineWriteGrace. A probe carries no labels, so it never callsprotectLabels: “absent from the listing” must not be read as confirming an engine label removal. A kept value produces noStatusChanged/StateChanged, no channel event and noStatusEnteredAtreset; a probe fetched at or after W that agrees with the cache drops thestatus/closedrecord; a probe fetched after W + grace overwrites as before, so a human move is applied one probe later. Status and closed state are decided independently.UpdatedAtis still assigned from the probe (a lagging listing can move it backwards; harmless). Around the apply,Store.EffectiveProbegives the probe loop the protected Status/closed state for its pre-apply reads (stage configuration, terminal-flag clear, probe-only terminal), and the stale-item deep fetch is seeded with the post-apply Store Status/closed state soItemDeepFetchedcannot write the stale probe value back. The seed protectsStatus(the deep fetch never sets it) and PR cards; for an issue cardFetchItemDetailsoverwritesIsClosedfrom the live issue node, so that closed-state write-back stays unprotected (the deep-fetch residual below). Bootstrap probes (BootstrapFromProbe→Reset) are not listings of an existing item and are unprotected by design. - Residuals. Writes that bypass the cache write-through (a direct client call with no
Apply*) are never recorded and stay unprotected. The deep-fetch write-back (ItemDeepFetched→applyProjectItem) also refreshesLabelsand, for an issue card, the closed state (read live from the issue node), and is not protected (ADR-957 scope). The probe-loop apply is protected forStatusand closed state (#2178, below) but not for anything else; it carries no labels, so it never touches label records. Because engine writes never bump the cachedUpdatedAt, a listing that confirms a write after its protection ended still looks newer and counts as drift once (pre-existing noise); while a record shields the item, theUpdatedAtcomparison is skipped.
D.5 Stream-Health Failover
Health transitions are driven by the reconcileTicker goroutine (engine/reconcile.go), which calls cacheImpl.LightReconcile(...) on each tick (default: every 3 minutes, configurable via --reconcile-interval / FABRIK_RECONCILE_INTERVAL):
LightReconcile result |
Action |
|---|---|
| No drift (a difference that is only a recent engine write is not drift — see D.4, #2159) | wm.transitionHealthState(Healthy, "") — no-op if already healthy; logs if recovering from unhealthy |
| Drift detected | wm.transitionHealthState(Unhealthy, reason) → cacheImpl.Pause() → cacheImpl.Reconcile(freshBoard) → cacheImpl.Resume() → wm.transitionHealthState(Healthy, "drift reconciled") |
| Network error | Log warning; no state change (treat as “unable to determine”, not drift) |
When IsPaused() is true:
ApplyDeltais a no-op (deltas are dropped rather than applied to a potentially stale cache).- All
CacheImplread methods fall through to thefallbackReadClient (live GitHub API), so correctness is preserved at the cost of increased GraphQL usage.
On drift recovery, LightReconcile returns the fresh board it already fetched; this is passed directly to cacheImpl.Reconcile(freshBoard) before Resume(), so the cache is coherent immediately when IsPaused() returns false.
API cost: Each LightReconcile tick costs ~1 GraphQL call (FetchProjectBoard). At the default 3-minute cadence: ≤ 20 calls/hour — negligible against the 5000-point/hour budget.
healthChangeFn removed (issue #641): The callback-based healthChangeFn pattern (ADR-034) has been replaced. The reconcileTicker goroutine owns the full Pause/Reconcile/Resume sequence directly; no indirection through a closure is required.
D.6 Cache Mode
The --board-cache flag has been removed. CacheImpl is now wired unconditionally for production runs. The cache is freshened by:
- Per-poll probe (
runProbeAndDeepFetch) (always active on bootstrapped cache) — see D.4 - Layer 2 status sweep (always active) — see D.4 and D.7
- Webhook deltas (when
--webhooksis enabled) — see D.2 - Light-reconcile drift recovery (always active, independent of
--webhooks) — see D.5
Tests that need to bypass the cache use engine.NewWithDeps, which wires boardcache.GitHubAdapter directly. This bypasses the Store-mutation pipeline and silently disables observers, so it is intended only for unit tests that don’t exercise Store-driven behavior.
D.7 Status field reconciliation in user mode
In user mode (PAT-based, repo-level webhooks via gh webhook forward), GitHub does not deliver projects_v2_item events. Board-column changes — including the Status field that drives stage selection — are invisible to the webhook stream. Fabrik uses a four-layer strategy to keep the cached Status current despite this gap:
| Layer | Mechanism | Latency | Cost |
|---|---|---|---|
| 0 Write-through | After any successful mutation call in the engine (status, labels, comments), the corresponding CacheImpl method is called immediately |
Zero | Zero |
| 1 Per-event refresh | After ApplyDelta for an issues or issue_comment event, fetches the current Status: fast path (FetchProjectItemStatus) when the item’s itemID is cached; fallback (LookupIssueProjectItem) when it isn’t yet (e.g., brand-new issues arriving via issues.opened before projects_v2_item.created) |
Seconds (best-effort) | ~1–5 pts/event |
| 2 Periodic status sweep | In-poll updatedAt gate: FetchProjectUpdatedAt (~1 pt) every poll cycle (~15 s); fires FetchProjectItemStatusBatch + ApplyStatusBatch only when project timestamp advances |
Up to 15 s | ~1 pt/cycle idle; O(⌈N/100⌉) pts when gate fires |
| Startup bootstrap | ProbeProjectBoard + BootstrapFromProbe on cold start (both webhook and polling-only modes) |
Startup only | ~250 nodes |
| Drift-recovery | Full FetchProjectBoard + Reconcile when reconcileTicker detects drift |
Minutes | Full board cost |
Residual latency: For external column moves (user moves issue on the board), the upper bound on detection latency is the main poll cadence (~15 s) — Layer 2 runs at the top of every poll() cycle. If the column move happens to coincide with any other issue activity (a comment, label change, etc.), Layer 1 may catch it sooner. Layer 0 applies only to Fabrik’s own mutations.
Layer 0 write-through convention: Every engine call site that mutates dispatch-relevant cache state must call the corresponding CacheImpl write-through method immediately after the API call succeeds. The safe type-assertion pattern used at every site:
if cacheImpl, ok := e.readClient.(*boardcache.CacheImpl); ok {
cacheImpl.<Method>(boardcache.ItemKey(item.Repo, item.Number), ...)
}
This is a no-op when e.readClient is a GitHubAdapter (cache-disabled mode), so the guard is always safe to include.
Covered mutation types and their CacheImpl methods:
| Mutation | CacheImpl method |
Dispatch relevance |
|---|---|---|
UpdateProjectItemStatus |
UpdateItemStatus |
Stage selection reads Status |
AddLabelToIssue |
ApplyLabelAdded |
Label guards (fabrik:locked, stage:*:complete, etc.) |
RemoveLabelFromIssue |
ApplyLabelRemoved |
Same |
AddComment (issue-targeted) |
ApplyCommentAdded |
Comment dispatch reads Comments for rocket-reaction checks |
Excluded mutation types (annotated with // no write-through: excluded — at each call site):
| Mutation | Reason excluded |
|---|---|
AddCommentReaction |
Reactions are not read from cache for dispatch decisions |
AddPRReviewCommentReaction |
Same |
AddComment targeting a PR number |
Posts to PR comment thread, not issue cache |
UpdateIssueBody |
Issue body is not read from cache for dispatch decisions |
CreateDraftPR |
PR existence is resolved live via FindPRForIssue; not cached |
MarkPRReady |
Draft state is not read from cache for dispatch decisions |
MergePR |
Merge state is not read from cache for dispatch decisions |
CacheImpl existence guard: All three new write-through methods (ApplyLabelAdded, ApplyLabelRemoved, ApplyCommentAdded) call c.store.Get(repo, number) before applying the mutation. If the key is not present in the Store (i.e., was never bootstrapped), the method returns without creating a phantom Store entry.
Layer 1 scope: Only issues and issue_comment event types trigger a per-event status fetch. pull_request, pull_request_review, check_run, and other event types are excluded from Layer 1 (finding the linked issue for a PR event requires an O(N) cache scan; check_run does not carry an item ID). Layer 2’s per-poll-cycle gate covers these cases within ~15 s. PR conversation comments are skipped (#2093): an issue_comment whose issue object carries a pull_request key returns before any cache or GraphQL access and logs nothing, because repository.issue(number:) cannot resolve a PR number.
For new issues whose itemID is not yet in the cache (e.g., added via issues.opened before a projects_v2_item.created event is received), Layer 1 falls back to LookupIssueProjectItem to populate both itemID and current Status in one GraphQL query (repository.issue.projectItems). If cache.ProjectID() is empty (Bootstrap not yet complete), the fallback is skipped. After a successful fallback, subsequent Layer 1 calls for the same issue use the cheaper FetchProjectItemStatus fast path.
Negative cache (#2093, engine/layer1_offboard.go): a fallback lookup that succeeds with no item (the issue is not on the board) is remembered in memory for 10 minutes per (project, repo, number), so further events for that issue make no call. Errors are never cached, so a transient failure is retried on the next event. The fast path (GetItemID) is consulted before the entry, so any path that gives the cache an itemID bypasses it, and every projects_v2_item created event clears all entries (that payload carries no repo/number to target one). A restart clears the cache.
Appendix E: Goroutine-Boundary vs. Work-Boundary Lifecycle
Two parallel lifecycle mechanisms track active work in Fabrik. They exist at different abstraction layers and must not be conflated.
E.1 Engine-State Lifecycle (Goroutine Boundary)
WorkerEntered / WorkerExited mark the goroutine boundary — the interval during which a dispatch goroutine is allocated and running. Introduced in Fix B (issue #544, PR #568).
| Event | Applied | Where |
|---|---|---|
WorkerEntered{StageName, StartedAt} |
Synchronously before wg.Add(1) and goroutine launch |
engine/poll.go (main dispatch), engine/reviews.go, engine/ci.go, engine/merge_gate.go |
WorkerExited{} |
Deferred at the top of each goroutine — fires on every exit path | Same four files |
Purpose: Prevent double-dispatch. snap.Worker() != nil is the guard. Because WorkerEntered is applied before the goroutine starts, the guard is effective from the instant the goroutine is scheduled. WorkerExited wakes the poll loop via WorkerLifecycleChanged so the next stage dispatches promptly after the goroutine exits.
Coverage: All exit paths including processItem early-returns, context cancel, ensureRepoReady failure, and normal completion.
E.2 TUI-Presentation Lifecycle (Work Boundary)
JobStartedEvent / JobCompletedEvent mark the work boundary — the interval during which Claude is actually running. Introduced in issue #578.
| Event | Emitted | Where |
|---|---|---|
JobStartedEvent{...} |
Past all early-return guards, at the “committed to real work” point | engine/item.go (after lock acquired + WorkerExited deferred); engine/comments.go (at function entry, after log line) |
JobCompletedEvent{Skipped:true} (cleanup guard, unconditional) |
Deferred immediately after JobStartedEvent; fires on every function return including success |
Same two sites — HistoryPaneComponent filters these out; only InvocationObserver’s Skipped:false event reaches history |
JobCompletedEvent{Skipped:false} (authoritative success path) |
By InvocationObserver when InvocationRecorded is applied to the store |
engine/observers.go |
Purpose: Drive the TUI active pane. “In Progress (N)” reflects items where Claude is running, not items whose goroutines have merely launched.
Why the distinction matters: A goroutine that launches and immediately hits a fabrik:blocked early-return allocates a worker slot for ~1ms but does not invoke Claude. The goroutine-boundary lifecycle must track it (to prevent double-dispatch). The TUI-presentation lifecycle must not track it (to avoid ghost “In Progress” entries in the active pane that never clear).
Constraint for contributors: Do NOT move JobStartedEvent emission back to the goroutine-launch site (dispatch goroutines in poll.go or reinvoke dispatchers). Ghost entries from early-return paths have no matching JobCompletedEvent and persist in the active pane indefinitely. Issue #578 was filed for exactly this bug. The constraint is documented in the ActivePaneComponent struct comment in tui/active.go.
E.3 Summary of Differences
| Dimension | Engine-State Lifecycle | TUI-Presentation Lifecycle |
|---|---|---|
| Events | WorkerEntered / WorkerExited |
JobStartedEvent / JobCompletedEvent |
| Boundary | Goroutine lifetime | Claude invocation interval |
| Applied at | Goroutine launch / goroutine exit | Past all early-return guards / invocation return |
| Coverage | Every goroutine including early-returns | Only goroutines that invoke Claude |
| Consumer | itemstate.Store observers (dispatch guard, wake channel) |
tui.ActivePaneComponent (active pane display) |
| Idempotent on double-fire? | Yes (WorkerExited is a no-op when Worker is nil) |
Yes (delete(a.active, key) is a no-op on missing key) |
| Introduced | PR #568 (Fix B, issue #544) | Issue #578 |