The script that shipped half a release
We release products with two stores and several kinds of evidence. Riff has an Android package and an iOS binary. QuipKey has a keyboard extension and more than one platform surface. WaveCraft has an editor, a share extension and store metadata that changes independently of its binary.
The first release automation treated each store operation as a step in a script:
- create a version;
- upload a build;
- upload metadata;
- submit for review;
- wait for the store.
The script was readable and it had no memory outside its process. A timeout after step three meant the next run had to guess whether the version existed. A callback arriving twice could advance the same release twice. A human changing a draft in App Store Connect could leave the local process believing it still owned the old contents. Play's single-draft rule and App Store's version-scoped resources made the guesses fail in different ways.
The change was to stop treating a release as a script. A release is a durable state machine, and the store is one of its observers.
The ledger owns the question “what are we doing?”
Mainline's release ledger records the intent before it performs an external mutation. It names the product, platform, candidate build, channel, requested metadata and the operation currently admitted. Every later callback or poll is reconciled against that record.
That ordering matters. If the process dies after creating an App Store version but before recording it, the next process cannot tell whether it should create another. If the intent is durable first, the reconciler can query the store and bind the observed resource to the existing operation. The same rule handles a callback that arrives before a network request returns: the event is evidence, not permission to create a second state transition.
The ledger separates three things that the script had collapsed:
- intent — what the release is trying to do;
- evidence — what the store, build system or review system has said;
- policy — whether that evidence is enough to advance.
A store adapter can report that a draft exists. It cannot decide that the draft belongs to the current candidate without the ledger's identity check. A build callback can report that an artifact uploaded. It cannot advance the release if the candidate identity does not match the one reserved in the ledger.
This is the same observation/policy boundary that matters in CI. The external system owns facts about its resources; Mainline owns the decision about its release.
Two stores, one domain
App Store Connect and Google Play do not share a vocabulary. One has version records, resource scopes and review submissions. The other has tracks, drafts, edit sessions and staged rollout. Trying to make the adapters expose a fake common API creates a second source of meaning: the adapter starts translating a store state into a lifecycle state that the domain never actually witnessed.
The domain model therefore owns the invariant and the adapters own translation.
For every candidate, Mainline admits one upload operation. The App Store adapter maps it to the version and build resource that the store requires. The Play adapter maps it to the edit and track state that Play accepts. Both return evidence in the same shape: resource identity, observed status, submitted payload digest and the timestamp of the observation. The domain reconciler decides whether the evidence closes the operation, schedules another attempt or moves the release to a terminal failure.
The adapters also preserve store-specific constraints rather than hiding them. Play rejects a track with two draft releases, so the Play operation must select or replace the existing draft before it can upload. App Store metadata belongs to a version or a product-level resource, so a marketing change must not create a version as a side effect. A common domain API that erases those constraints would make the caller look simple while moving the failure to a later, less attributable step.
Resource scope prevents accidental work
Metadata was the clearest case. A store has several resources that contain words: a product page, a version, a review submission and a release note. An old uploader treated “sync metadata” as a broad action and created whichever resource was missing. That made a harmless copy update mutate release state.
The new operation names its scope. A product-level marketing update can write only product-level resources. A version update must be bound to the candidate's version identity. A review submission reads a fixed envelope and checks its contents before it submits. If the requested resource is absent, the operation reports that it cannot proceed; it does not create a different resource because one happens to be available.
The rule is deliberately boring:
A write may create only the resource its intent names.
That rule makes a retry safe. A timeout after a version write can re-read the version and compare its digest. A timeout after a product-page write can do the same without accidentally finding or creating a version. The ledger contains the requested scope, so recovery does not need to infer it from a half-finished response.
Idempotence is the release feature
Release workers restart. Store callbacks repeat. Review status changes while the worker is offline. Network errors happen after the server has accepted a request but before the client receives the response. A release system that handles only the happy path is a system that asks a human to remember which side of a timeout won.
Every external operation now has an identity and a convergence rule. If the ledger says an upload is admitted, a repeated callback updates the observation rather than creating another upload. If a poll sees the store already in the desired state, reconciliation closes the operation. If the store has moved to a state the policy cannot accept, the ledger records a terminal reason instead of retrying forever.
The terminal state matters as much as the retry state. A metadata sync that has exhausted its attempts without one successful observation is not “still running” because a worker might come back. It is a terminal failure with evidence. A Play draft that cannot be replaced is not an anonymous 400 that the next loop will interpret differently; the release records the store's constraint and points at the operation that violated it.
Cleanup follows the same rule. Temporary uploads, stale drafts and detached artifacts are removed by a dedicated reconciler after the release has a terminal outcome. Cleanup failure is recorded as cleanup failure. It does not rewrite a successful build into a failed release, and it does not erase the evidence needed to retry cleanup later.
What a product team gets
The immediate effect is fewer mysterious store states. The more useful effect is that a product change can be described in product terms again.
When WaveCraft changes a paid capability, the metadata sync can update the right resource without creating a phantom version. When Riff ships an Android candidate, the Play track and draft rules are part of the release model rather than tribal knowledge in a script. When QuipKey has multiple platform artifacts, the candidate identity keeps a keyboard extension or an Android package from being attached to the wrong release.
The system is not pretending the stores are the same. It is making the difference explicit and keeping one owner for the decisions that must be consistent across them.
Takeaways
Persist release intent before making an external change. A durable ledger turns a timeout into a reconciliation question instead of a guess.
Keep store adapters factual. App Store Connect and Play translate their own resources and states; the release domain owns identity, admission and policy.
Name the resource scope of every write. A metadata operation must not create a version, and a retry must not change what the original intent meant.
Design recovery as the normal path. Duplicate callbacks, worker restarts and store-side edits are ordinary release events; idempotent reconciliation is how a release finishes without a human replaying a script from memory.