Summary

Determining which of 470,000 repositories are still alive is a harder classification problem than it looks, and getting it wrong takes down production. How we model liveness as a graph property, why read activity is the signal that matters, and how we redesigned deletion to be survivable.

What problem are you addressing? What real engineering challenge, question, or experience is this session about?

We operate GitHub Enterprise Server for about 17,000 engineers: 470,000 repositories, 12 TB of Git data. Repository count at this scale is a load multiplier on nearly everything the platform does. Backup and replication windows scale with it. Every security scanner, dependency crawler, compliance job and inventory sync traverses the whole estate, so each dead repository consumes API budget, scan minutes and index space indefinitely. Fleet-wide jobs were degrading not because any individual job got worse, but because the denominator kept growing. Deleting repositories is a performance intervention, not a storage chore.

The difficulty is the determination. Liveness is not a property you can read off a repository. Last-commit recency marks finished, stable libraries as dead and bot-churned graveyards as alive. Declared owners have often left the company. We modelled the estate as a graph and classified against composed criteria, which produced roughly 46,000 org-level deletion candidates, narrowed to about 40,000 after a read-activity gate.

Then we deleted one that mattered. It backed a job running on a two-year cadence that had not failed in five years, and every engineer who built it had left — so there was nobody to say the flow depended on it. Its cadence simply exceeded our observation window. The outage took time to attribute because nothing linked the failure to a cleanup that had run weeks earlier. The restore we treated as a safety net was a Git bundle in S3, which returns code and nothing else.

Who is the intended audience?

Platform Engineering, SRE, DevOps, Infrastructure. Also relevant to engineering leaders responsible for developer platform cost and risk.

Level:

Intermediate

List one or two practical takeaways

1. Repository liveness is a graph property, not a repository attribute — and read activity, not write activity, is the signal that tells you whether anything still depends on it.
2. A Git bundle is not a repository backup. Capture configuration before deletion, and design the delete path as a staged ladder so that a wrong classification is recoverable rather than an outage.

What will you share?

Architecture and design decisions (graph model of the estate, composed classification labels over a single risk score), implementation detail (representative Cypher classification queries, the Splunk read-activity method across unicorn.log and gitauth.log, and the chunked windowing needed to run it without overloading the log platform), production experience (candidate counts before and after the activity gate, roughly 50,000 of 470,000 repositories classified in the 1st pass), an incident and its postmortem, failure modes and debugging (why attribution was slow, and a full inventory of what a bundle restore leaves behind), operational trade-offs, and the before-and-after shape of the deletion pipeline.

What is your experience with this problem?

Production system. Real incident or failure. Internal engineering project. Hard-earned engineering lesson.

What approaches failed, disappointed, or created unexpected problems?

Commit recency as a liveness signal — inverted for stable code. Declared ownership — stale wherever attrition has occurred, which is disproportionately the repositories you’re evaluating. Dependency edges alone — they catch code-level coupling but miss operational coupling, which is how a scheduled job holds a repository alive without anything importing from it.

The deeper failure was treating absence of signal as evidence of death. Five years without a break read as abandonment; it was actually a system working correctly on a cadence longer than anything we were watching.

And the restore. We assumed S3 Git bundles made deletion reversible. Code came back; webhooks did not, so checks stopped firing and the repository was present, correctly named, and still functionally dead. Branch protections, rulesets and pre-receive hook associations were also gone. We rebuilt by hand under the same name, and anything keyed to the repository ID rather than the path stayed broken while appearing fixed.

What will you do differently today?

Capture repository configuration as structured metadata before deletion — owners and collaborators, webhooks, rulesets, branch protection rules — not only Git data. Stage the deletion path rather than switching: 30-day owner opt-out, 15 days archived, 90 days soft-deleted, then S3-only. Treat multi-year workload cadence as a first-class blind spot rather than a long tail, and link cleanup actions into incident tooling so the programme is visible to whoever is debugging its consequences.

What trade-offs did you consider?

Graph over relational. Liveness depends on network position — who commits, whether those identities are current, what deploys from it, what depends on it. Expressing that as joins was unworkable; Cypher over a graph was not. The cost is that graph freshness becomes its own pipeline, and a stale graph classifies confidently and wrongly.

Composed labels over a single risk score. A score gives you a threshold argument with every owner you talk to. A label set lets you state precisely which condition was met. We optimised for defensibility in conversation over ranking convenience.

One-year read-activity retention. Extending the Splunk lookback further would have cost more server load than we chose to spend. We knowingly gave up visibility into anything with a cadence beyond a year — which is precisely the class of workload that broke us. This trade is still live.

Opt-out over opt-in sign-off. Requiring a live owner to approve deletion sounds safer, but most candidate repositories have no live owner, so opt-in becomes a permanent stall and the estate never shrinks. Opt-out shifts the default and still gives objectors a path. The honest cost: a repository whose owners have all left cannot object, which is exactly the failing case.

How can this help other practitioners?

1. A design approach for modelling liveness in any large asset estate, not just repositories. A specific mistake to avoid — assuming your backup is a restore, with a concrete inventory of what is missing and what breaks first.
2. A pattern for staged, reversible destructive operations. A way to evaluate signals for this kind of decision, including which ones invert at scale. And a way of thinking about observability retention as a hard bound on what you are permitted to conclude.

Current state

Production experience (pipeline live, classification ongoing)

#platformengineering #developerplatforms #failurestory #casestudy #devops #sre #scalability #observability #governance