Two ways to say "done"
Mainline's analytics pipeline consumes from Kafka with franz-go. The library offers two ways for a consumer to report progress, and the difference between them is the difference between at-least-once delivery and silently-thrown-away rows.
The cheap one is committing the poll watermark: whatever offset the consumer last fetched. It costs a few lines, needs no bookkeeping, and is what you write the first time. It is also wrong in a way that produces no error, because the watermark advances on every poll and never rewinds.
Picture three records polled together — A, B, C. A handler for B fails with something transient. The consumer retries B, succeeds this time, and commits. It committed the watermark, which by now sits past C as well, so C is marked processed having never been handled at all. If C belonged to a partition nobody noticed, the row is simply gone: no exception, no metric, no lag. Our three workers — an enricher, a person merger, and the crash ingest — each carried their own version of this pattern, and each had been quietly wrong.
One owner for the cursor
The fix is not a better call at the call site. It is deleting the call sites. Foundation/Analytics/Go/mq.Resolver now owns one invariant, stated in the type's own documentation:
The consumer cursor never passes an unresolved record.
A consumer holds one Resolver, and Resolve takes the records of a poll and returns the subset that is safe to commit. Every consumer wires its handler into it, and the watermark APIs disappear from the codebase entirely.
The detail that decides whether the invariant holds is lifetime. The Resolver must outlive the poll that discovered the problem. An implementation that rebuilds its blocked-set each round looks correct and is not: round N refuses to commit past record X, round N+1 starts with an empty blocked set, processes everything including X and everything after it, and commits the watermark — past X. The loss does not happen inside a round, where the guard sees it. It happens between rounds, where nothing is watching.
Three outcomes, and shutdown is not one of them
Resolution has to distinguish more than success and failure, because collapsing "we gave up" into "this failed" loses the distinction that decides whether the pipeline stops or continues:
const (
settled outcome = iota // handled, or permanently failed and already parked elsewhere
unsettled // retry budget exhausted
canceled // context cancelled
)
canceled gets its own outcome for a reason worth stating plainly: shutdown is not failure. A record left unprocessed by a rolling deploy is not a record that failed, and blocking its partition until a human notices would turn every deploy into an incident. Cancellation truncates the batch and blocks nothing; on the next start the record is re-read from the committed offset, exactly as intended.
Truncation is per partition. A poison record is a per-partition fact — one unparseable row should stop the partition it landed in and nothing else. Truncating the whole batch would stall the healthy partitions interleaved behind it, and since their committed offsets would then stop moving, the stuck-partition monitor would report all of them as stuck. One bad record, an alarm that names four victims.
"Unlimited retries" turned out to be unimplementable
The retry policy looks like a plain struct until you ask what happens when a record can never succeed:
type RetryPolicy struct {
BaseBackoff time.Duration // first delay, doubling up to MaxBackoff
MaxBackoff time.Duration
MaxElapsed time.Duration // total backoff budget for one record; must be > 0
}
The zero value of MaxElapsed used to mean "retry forever." It now panics during normalization, and the comment explaining why is longer than the field:
Resolve is called synchronously in the consume goroutine. An infinite retry never returns, so the rest of the consume loop — the poll that would notice new data, the monitor that would notice this partition is not advancing — never executes. The blind spot closes over the exact thing that was supposed to watch it: process alive, health check green, no alert, one record that will never resolve.
The concrete damage took months to notice. Our install-attribution topic stopped making progress on 2026-06-19 and stayed stopped: an install old enough that the ad service had forgotten about it kept failing, and each retry cycle re-welded the partition shut. Nothing in the system could report that, because the reporting lived downstream of the loop that had stopped.
Two smaller decisions ride along in the same struct. The budget accrues against the unjittered backoff ladder, so how many attempts you get is a function of policy and not of how slow a given downstream happens to be — jitter exists to keep a fleet of replicas from stampeding a shared failure, and letting it perturb the retry count would make "when do we give up" a random number. And blocking is explicitly not losing: the partition stops until the process restarts, then re-reads the unresolved record from the committed offset.
Retry-exhausted is not one thing
The last piece is the one that turned a stuck pipeline into a healthy one. When the budget runs out, "there is no point retrying" means different things to different consumers:
// ExhaustedSettles decides what a spent budget means. true = the caller parked
// this record somewhere it cannot be lost from, so treat it as handled and let
// the cursor advance. false or nil = block the partition (the default).
ExhaustedSettles func(rec *kgo.Record, err error) bool
Blocking is the right default, and staying there is not laziness. Dropping a crash report loses evidence — the crash is the thing you built the pipeline to keep. But an ad-attribution lookup that returns 404 is not a failure at all: it is the correct, terminal answer to the question "did this install come from an ad?", namely no. Holding an entire pipeline hostage to a row whose content is "nothing happened" is a strictly worse outcome than losing the row.
The default stays block, and a caller may only opt out if it can say where the record went. A function that answers "just drop it" is not in the API, because "just drop it" is precisely the silent loss this package exists to prevent.
Then forbid the fix pattern
An invariant that every engineer has to remember is an invariant that decays. Once the resolver existed, the watermark APIs had no remaining legitimate caller — so a guard forbids them by name across the analytics trees:
id = "analytics-commit-granularity"
risk = "critical"
paths = ["Analytics/**/*.go", "Foundation/Analytics/**/*.go"]
forbid = [
"*uncommittedoffsets*",
"*commitmarkedoffsets*",
"*markcommitrecords*",
"*.commitoffsets(*",
"*.commitoffsetssync(*",
]
checks = ["guard:go"]
The globs are dotted rather than bare on purpose. A bare *commitoffsets* would also match the progress monitor's ObserveCommit… helpers and flag files that never touch a watermark at all; pinning the call shape keeps the blast radius to the five API entries that are actually the bug. (Because the matcher runs against whole added lines including comments, prose about these methods has to refer to them indirectly — a small price for a rule an agent cannot talk its way past.)
Takeaways
If your consumer library offers a "commit what I fetched" convenience, know what it costs before you take it: the convenience is exactly the property that makes loss invisible, because the offset moves whether or not the work happened. Give the cursor a single owner, and give that owner a lifetime longer than a single poll — the leak that matters is between rounds, not inside one.
Give "we gave up" more than one meaning, and make the caller name which one applies. Retry budgets should be measured against a policy, not a clock, or downstream latency silently becomes part of your correctness story. And when you find that an option was never implementable — "unlimited" in a synchronous retry loop is the standing example — delete it and make the compiler or the constructor say so, because a configuration value that can express an impossible state is an outage waiting for someone to type it.
Finally: an invariant nobody has to remember survives. Naming the forbidden API is what keeps a fix from being undone by the next person who finds the convenient shortcut.