Grounding · 28 October 2026 · api-v0.3.116 live

The release gate, reworked from first principles

api-v0.3.114 rolled back twice on a gate that measured the CRM instead of the deploy. A day later the gate proves only what a release changes, every repository has shipped, and the next bottleneck is the cycle itself.

Read the changes Cycle, minute by minute

Staging APIapi-v0.3.116develop head · released 03:46Z

Connectorconnector-v0.5.6engine 0.28.0 · ops 0.6.2

Boxes2 of 2 linkedtenant-a, tenant-b · heartbeats under 30 s

Gate probepassed in 3 soc release probe --target staging --once

01 · Why

A release that changed nothing failed twice. The gate was the defect.

Before

The gate waited on the CRM

The old api.health gate waited for each connector box's heartbeat and relay to be fresh. The tenant-b box was stuck on a CRM event the platform kept refusing: an owner invitation for a unit the staging catalogue had never filed. The connector retried it behind every later event and heartbeat, the box went stale, and the gate rolled back api-v0.3.114, which carried no runtime change at all.

Heartbeat age measures one tenant's CRM state. No API release changes that. A gate that waits on it fails releases for reasons they cannot fix.

Principle

The gate proves only what the release changes.

Everything else is a monitor's job. Heartbeat and relay freshness stay on the monitoring route and never return to a release gate.

02 · What shipped

One pull request per repository, in dependency order

Unblock the box, change the gate, make the gate testable, make the deploy cheaper.

crm 194 · connector-v0.5.5

Park what the platform refuses

The publisher parks an event after 12 retryable refusals and frees its slot between retries. The heartbeat has its own lane.

One bad event can no longer freeze a box
The catalogue sweep is no longer starved
A parked event is heard afresh on the sale's next change

// publisher, tenant-b box event OwnerInvitationRequested Sale … reject unit_unknown retryable=false parked after 12 refusals · slot freed heartbeat lane 0 · age 7s

platform 2133 · api-v0.3.115

Gate = ready + linked

Identity answers unit_unknown as a terminal refusal. The connector Worker gains /internal/connector/linked: 200 once every configured box's Durable Object holds a socket.

api.health checks the two ready routes and linked
Heartbeat and relay stay on the monitoring route
First release through the linked gate: 5m59s

$ curl -s $CONNECTOR/internal/connector/linked 200 {"linked":true,"boxes":2,"sockets":2} $ oc verify api.health ✓ api.ready 142ms ✓ connector.ready 98ms ✓ connector.linked { linked: true }

engine PR 652 · engine-v0.28.0

Probe locally, hold what did not change

oc release probe --target staging --once runs the checkout's gate from a workstation with the readiness token. The Wrangler provider adds --containers-rollout=none when every released Container image already runs.

Gate changes are tried before a release, not during one
Unchanged container images keep their instances
The engine no longer builds the dashboard

$ oc release probe --target staging --once probe api.health · 3 nodes ✓ api.ready ✓ connector.ready ✓ connector.linked passed in 3.0s

platform 2135 · crm 195

fmtkit out of the pipeline

Engine 0.28.0 never downloads fmtkit on a runner. Rather than wire its turbo task into every consumer, both repositories dropped their Format workflow and the platform rulesets no longer require its context.

Workstations still run make format-all
Required contexts: three remain, Format removed
the engine repo keeps its own format workflow

$ git rm .github/workflows/format.yml $ gh api -X PUT repos/…/rulesets/1552087 required contexts ✓ Backend verdict ✓ Mobile verdict ✓ Static verdict − Code is formatted and within the complexity limits

03 · The cycle, minute by minute

From the merge of engine PR 652 to api-v0.3.116 on staging

46 minutes with three stops, none of them a release defect.

Wall clock46 min

02:56Z merge to 03:46Z release

Stops3

fmtkit twice, stale reference once

Clean cycle~35 min

first full cycle on 26 October: 75

Gate3.5 s

ready + linked, passed first time

02:56Z
engine PR 652 merged
The cycle starts from platform develop with pin PRs admin-merged once green.
03:05Z
engine-v0.28.0 released
35 commits on the engine main past 0.27.15, including the fmtkit resolver by another author.
03:08Z
Stop 1 · crm pin PR 195 Format
Engine 0.28.0 refuses to download fmtkit on a runner. Fixed by removing the Format workflow on the pin branch.
03:20Z
PR 195 merged · ops-v0.6.2 released
03:22Z
Stop 2 · platform pin PR 2136 verify-reference
The committed command reference lacked oc release probe and oc setup fmtkit. Regenerated with oc gen reference and oc gen schema on the pin branch. The platform Format workflow was removed in parallel by PR 2135.
03:34Z
connector-v0.5.6 released
The crm main had moved by one commit (PR 195), so the cycle cut a connector release too. Bundle 2m55s, shadow 2m21s, release 3m30s.
03:35Z
PR 2136 merged
03:46Z
api-v0.3.116 released gate passed in 3.5 s
Bundle 2m30s, shadow 2m43s, release 5m57s. Every pin at its newest release.
03:45–03:48Z
Both boxes dark for 2.5 minutes blind spot
The gate passed against the old Container instances; Cloudflare replaced them 22 seconds later. The new instances linked at 03:48:22Z.

Step	Duration	Mostly
Engine release	3m 35s	build and publish
CRM pin PR CI	~10m	waiting on the full Go gate
Ops release	1m 02s	publish
Connector bundle, shadow, release	8m 46s	three serial runs
Platform pin PR CI	~8m	every backend tier for a two-line lock change
API bundle, shadow, release	11m 10s	three serial runs, six minutes of them the release graph

04 · What we learned

Five things worth keeping

A gate measures only the release's own effect.

Heartbeat age measures the CRM, not the deploy. It never goes back into a gate.

Engine moves break consumers in ways no gate sees.

Engine 0.28.0 stopped the cycle three times: fmtkit no longer downloads on a runner, in the crm repo and then the platform repo, and the committed command reference went stale. Each was a consumer convention, not a release defect, and each cost a stop and a rerun.

Container replacement is asynchronous.

wrangler deploy returns before Cloudflare swaps the instances. The gate passed against the old sockets and the boxes then went dark while the new ones linked. API-only releases are now immune through the hold; releases that ship a new connector image are not yet.

The cycle is serial and mostly waiting.

Engine, crm pins, ops and connector, platform pins, API. Pin PR CI, trusted CI on the head, shadow runs and merges fill the wall clock.

A dry run proves the plan, not the consumers.

Each consumer's decoders, references and workflows only fail once the pin lands.

Rule

Never put heartbeat or relay freshness back into a release gate.

05 · What needs doing

In priority order

RG-1.1 engine repo · deploy provider

Wait for the Container rollout before health

When an image changes, poll the application's rollout until the new instances serve, then run the gate. Closes the blind spot.

RG-1.2 engine repo · oc release pin

Regenerate reference and schemas on an engine move

The pin step writes reference.json, oc.schema.json and plugin.schema.json itself. Removes a whole class of stops.

RG-0.1 next API-only release

Confirm the hold

The log should say the containers already run that image and keep their instances; heartbeat ages should stay small across the deploy.

RG-0.2 staging · tenant-b

Verify the catalogue files the new tower

Box 2 cursors advancing and project-prefixed rows in app.places. The 330 sweep failures still reported come from that chain.

06 · Plan to speed the release cycle

A clean cycle under 15 minutes

Many merges per release, never a cycle per merge.

01
Release only what moved
Engine 0.28.0 already carries rebuild (the pins whose moves cut a line) and gate (the one CI job a tag needs). Declare them in the cycle policy so an engine docs commit does not cut an engine release and a connector release is cut only when its code or an embedded pin moved. Today every upstream commit forces a release of its line.
biggest win
02
Narrow pin PR CI
A pin PR changes two lock files. Run pin, reference, workflows and static only, through the existing CI decider, not the full backend tiers.
−5 to 8 min per pin PR
03
Overlap the independent steps
Tag ops and shadow the connector while the platform pin PR runs. Dispatch the API bundle as soon as develop's head is known, since the pin merge does not change the bundle digest.
five waits become three
04
Shadow once per bundle
When the same bundle digest already shadowed green, skip the shadow for the connector and the API.
−2.5 min each
05
Hold what did not change
Shipped for Container images. Extend it to the migrate Worker when its image digest is unchanged.
shipped for containers
06
Batch merges
Run the cycle on demand or on a timer. PRs merge to develop all day through the queue; one cycle ships the batch with pin PRs admin-merged. The cycle already resumes from open pin PRs, so a failed batch retries without redoing passed steps.
one cycle per batch
07
Catch engine surprises upstream, not in the cycle
Before an engine tag, run each consumer's static oc verify tier against the engine candidate, so a 0.28.0-style break fails in the engine repo's CI first.
no more stops 1–2

07 · The changes

How we get there, in four phases

Every item has an identifier, RG-<phase>.<item>, lands as its own pull request, and is measured by the next cycle's log. Nothing waits on a redesign.

Today46 min

35 clean, 3 stops caused by the engine move

After phase 1~35 min

no stops; rollout waited for

After phase 2~24 min

pin PRs run the narrow tier

After phase 3< 15 min

steps overlap, shadows deduplicated

Phase 1 · stop the stops · engine repo

Make an engine move safe for its consumers

Every stop in today's cycle came from the engine moving under a consumer. These three changes make that impossible to miss before a tag exists.

RG-1.1 Wait for the Container rollout. When a released image differs from the running one, the deploy provider polls the application's rollout until every instance serves the new image, then runs health. Verified on the next connector release: heartbeat ages stay under 30 s across the deploy.
RG-1.2 Pin step regenerates the reference. oc release pin engine writes reference.json, oc.schema.json and plugin.schema.json in the pin commit. Verified when a pin PR's static tier passes with no manual regeneration.
RG-1.3 Consumer static tier in engine CI. Before an engine tag, a job checks out each consumer at its default branch and runs its static oc verify tier against the candidate binary. A 0.28.0-style break fails here first.

$ oc release staging --target staging apply connector · image changed rollout 0/2 instances on new image 12s rollout 1/2 instances on new image 41s rollout 2/2 instances on new image 58s verify api.health ✓ connector.linked { linked: true } passed in 3.2s · no instance replaced after the gate

Phase 2 · narrow pin PRs · platform and crm repos

A two-line lock change runs a two-minute tier

Pin PRs today run every backend tier: about 10 minutes in the crm repo and 8 in the platform repo. The CI decider already classifies a diff; it gains a pin-only class.

RG-2.1 Decider class pins. A diff that touches only the engine lock and the pins file selects pin, reference, workflows and static. The verdict jobs still run, so the rulesets' required contexts are unchanged. The merge commit on the default branch still runs everything before a tag is cut from it.
RG-2.2 Platform wiring. The backend workflow honours the class and skips the api-tests, db, bundle, go and e2e tiers for it.
RG-2.3 CRM wiring. The same for the crm repo's ci workflow, whose Go gate is the longest wait today.

$ oc ci decide-backend-ci --base develop changed cloud/oc.lock, cloud/pins.json class pins run pin · reference · workflows · static skip api-tests · db · bundle · go · e2e verdicts backend ✓ mobile ✓ static ✓ 2m 40s

Phase 3 · reshape the cycle · engine repo, oc release staging

Five waits become three, and nothing is rehearsed twice

The cycle is a fixed chain. It becomes a small graph that releases only what moved and overlaps whatever does not depend on a merge.

RG-3.1 Cycle policy reads rebuild and gate. Each line declares the pins whose move cuts it and the one CI job its tag needs. A docs-only engine commit cuts nothing; a connector release is cut only when its code or an embedded pin moved.
RG-3.2 Overlap. Tag ops and shadow the connector while the platform pin PR runs. Dispatch the API bundle as soon as the develop head is known, since the pin merge does not change the bundle digest.
RG-3.3 Shadow once per digest. A bundle digest that already shadowed green is not shadowed again. Saves 2.5 minutes on each of the connector and the API.
RG-3.4 Hold the migrate Worker when its image digest is unchanged, as the Container hold already does.

Cycle step engine rebuild: none moved skip Cycle step crm pins merged 2m 38s Cycle step ops ‖ connector shadow ‖ platform pins Cycle step connector digest 9f1c… already shadowed skip shadow Cycle step api bundle dispatched at develop head Cycle step api release passed in 3.4s Released api-v0.3.117 · cycle 13m 51s

Phase 4 · batch merges · engine repo + a timer

One cycle per batch, never one per merge

PRs merge to develop all day through the queue. The cycle runs on demand or on a timer, ships whatever landed, and is safe to rerun.

RG-4.1 Timer or on-demand trigger runs oc release staging --yes --merge --wait; a cycle that finds nothing moved exits in seconds. The cycle already resumes from open pin PRs, so a failed batch retries without redoing passed steps.
RG-4.2 One line per release in the log: what moved, what was skipped, gate time, rollout time. That log is how each phase above is checked.

Done when

Five consecutive cycles under 15 minutes with zero stops.

Measured from the cycle log, not estimated. Each phase lands as its own pull requests and is reported against the previous cycle's numbers.

Phase	Items	Repositories	Proved by
1 · Stop the stops	RG-1.1 · 1.2 · 1.3	engine	next connector release: heartbeats stay fresh; next engine move: no pin PR stop
2 · Narrow pin PRs	RG-2.1 · 2.2 · 2.3	engine, platform, crm	pin PR wall clock under 3 min in the cycle log
3 · Reshape the cycle	RG-3.1 · 3.2 · 3.3 · 3.4	engine	clean cycle under 15 min; skipped steps named in the log
4 · Batch merges	RG-4.1 · 4.2	engine, platform	one release per batch; rerun of a failed batch resumes

Delivery order: RG-1.2, RG-1.3, RG-1.1, RG-2.1, RG-2.2, RG-2.3, RG-3.1, RG-3.3, RG-3.2, RG-3.4, RG-4.1, RG-4.2. One pull request per item and repository, opened and left for review; each lands on its own and the next cycle measures it. Order matters only between phases 1 and 3: the cycle policy depends on the pin step regenerating references, and overlap depends on the rollout wait, otherwise an overlapped connector release could pass its gate against instances about to be replaced.