What a monorepo does to a secret
We keep six Apple signing keys — development and distribution, three teams — under a directory in the repository. They went in on 2025-11-29, in a commit whose subject was about centralized signing and auto-import in CI. The motivation was entirely reasonable: before it, setting up a build machine meant manually transferring keychains by hand, and the fix made every machine identical.
They were still there when we found them, nearly eleven months later.
The reason a private key in a monorepo is categorically worse than a private key in a folder is distribution. A key in a secrets manager is reachable by the things you point at it. A key in the repository is reachable by:
- every clone, on every machine, forever, including the laptop that gets stolen;
- every parallel worktree, which is how we work — a developer running two branches has two full copies;
- every CI job that checks out the repository, across every runner, including runners hosted by people who are not us;
- every fork, every backup, every archive of the repository that somebody made in passing.
And it has the refresh latency of a pull. Nothing is pushed. Nothing is revoked. Every new machine that joins simply already has them.
None of this requires a breach. The realistic failure is not an attacker; it is that the set of people with access to the repository is much larger than the set of people who should be able to sign an application, and nobody ever made that distinction explicit because the key was in a folder.
The guard that was marked critical
The part that made this embarrassing, and the part worth writing about, is what we already had.
We have a guard system. It enforces architecture invariants across the codebase, it runs in CI, and one of its rules covers the release-signing directory and is marked critical — the highest severity, the one that has to pass before anything ships.
It would have reported green for all eleven months.
That rule is a mutation check. It asks whether the change breaks the shape the build requires — did you delete a profile this build needs, did you point a target at a certificate that is not there. It is a genuinely useful rule, and it is exactly what it says. But a mutation check and a credential scanner are different instruments, and putting them under one badge with one severity label meant that for eleven months nobody asked which one we had.
This is the failure mode I would defend against hardest, and it is not "we lacked a tool." We had a tool. It was green. The lesson is that a mechanism which has earned trust on several axes gets trusted on the axes it knows nothing about, and a severity label is the most efficient trust-delivery device ever designed. If your gate says critical, it is worth establishing what class of problem it can actually see before you relax anything else.
For what it is worth, we did have a secret-scanning script. It line-scans text files under one infrastructure directory, and it would not have found binary key files sitting in a directory it was not pointed at. The two systems coexisted for a year and neither was wrong; we just never asked which one was looking.
Moving them, and failing closed while doing it
The change itself is small. The keys move to a separate deployment repository, outside the source repository entirely, and CI stops reading them from a path inside the tree.
The interesting part is what the CI change does when it cannot find them:
certs_source: "{{ signing_dir | default('') }}"
- name: 校验 p12 源目录已指定
assert:
that: certs_source | length > 0
fail_msg: >-
未指定 p12 源目录。用 -e signing_dir=<绝对路径>/signing/ios 指定；
私钥已迁出本仓库，不再从仓库相对路径读取。
The variable defaults to the empty string and an assertion fails the run if it is still empty. That is a deliberate choice over the obvious alternative, which is to default it to the new expected path and let a missing directory fail later with an import error. An import error says this path is wrong. It does not say you must now supply a path from outside the repository, which is the message that makes the change stick and stops the next person from quietly adding the old relative path back.
Two smaller decisions in the same commit:
The keys are read from one machine and copied, not discovered. The task that locates the key files runs delegated to the operations host, so the private key is read from a local path there and then copied to the runner. A find running on the runner would work identically on day one — and would also mean every runner needs the repository's history and the old paths to be able to search them, which keeps the distribution channel open.
.gitignore names the extensions, not the paths. *.p12, *.jks, *.keystore, with a comment above them explaining where the keys now live. Path-based ignores are defeated the first time someone needs the file in a slightly different place; extension-based ignores are the last line, not the first, and they are honest about being the last line.
A keystore that was never a release identity
The same commit removed something else, and the reason it was there is worth more than the removal.
The Android build graph contained a release keystore. It had been there long enough that everyone assumed it was how we shipped. It was not: it had been added to sign debug builds on machines where a locally-generated key was inconvenient, and it had never signed anything that reached a store. We deleted it from the build graph on the same day we moved the Apple keys.
That is the pattern worth noticing. In both cases the finding was not "we leaked a secret" — it was "we had a secret-shaped thing that we believed was load-bearing, and nobody had ever asked what it was actually for." A signing key is the most convincing-looking object in a build graph; it looks important enough that nobody tests whether it is.
What we did not do
We did not rotate the keys in that change. Rotation is irreversible, touches every machine and every past build's reproducibility, and deserved its own change with its own plan rather than being smuggled in beside a path move.
That is a real gap and we want to be plain about it: moving a key out of a repository limits future distribution, but it does nothing about the eleven months of history. The key was in every clone. It is still in every clone. Until the keys are rotated and the old ones revoked at the issuer, the distribution channel we closed is closed for new clones only.
It is worth stating because the failure mode of the follow-up is obvious in hindsight: it is very easy to close a distribution channel, tick the box, and leave the keys valid — at which point the whole exercise accomplished nothing except tidiness.
Takeaways
Treat every secret in version control as a secret that has already left your perimeter, because mechanically it has. Every clone, worktree, runner, and backup has it, it is not expiring, and the pull is free. The distribution is the exposure; removing the file from the working tree does not remove it from the history.
Audit what your critical-severity checks can actually see. A check that the build's shape is intact is not a check that no credentials are present, and a severity label will keep both on your dashboard for a year. Ask which instrument you have, because the badge does not answer that.
Fail closed on the new configuration, with an error message that explains the change. An empty default plus an assertion costs one line and converts "this path is wrong" into "this is now your job to supply it" — which is the message that prevents the old path from being added back as a convenience.
And check that your signing material is actually load-bearing. The Android keystore that had never signed anything looked exactly as important as the ones that did, and nobody noticed for a year. A credential in a build graph is a claim, not a fact, and claims should expire.