Version Control for Data

A data scientist needs a snapshot of a dataset before a big transformation, so she copies the whole thing into a folder named with today’s date. A week later there are five such folders, nobody remembers which one fed the model that shipped, and the “version” is once again a string a human typed into a name. It is the footer-version problem from the pillar, at petabyte scale, and no amount of discipline fixes it, because the tool was never built to hold versions in the first place.

Every post in this series has leaned on one phrase: a versioned source of truth. It is time to be honest that the source of truth is not one thing. Git is perfect until the artifact is too big or too unstructured for git, and then you need a different substrate that keeps the same property. The principle survives; the medium changes with the shape of the data.

Text belongs in git

For text-shaped artifacts, the answer is the one this whole series assumed: git. Policies, standards, architecture decisions, prompts, configuration, the semantic layer of the business as prose and structured markup. It is small, it diffs cleanly, and it reviews in a pull request.

I made the fuller case in Markdown Is the Knowledge Layer: plain text with frontmatter, format not platform, one medium that scales from a personal note to an enterprise lattice. If the artifact is words and it fits in a repository, it goes in the repository, and you are done. The interesting problems start where git stops.

Unstructured at volume belongs in a lake

Git chokes on a terabyte of PDFs, images, audio, and transcripts. You can bolt large-file storage onto it, but you are fighting the tool. The substrate built for this is the data lake: object storage with a lineage and catalog layer on top, where versioning comes from snapshots and manifest files rather than commits.

The property you are preserving is the same one git gives you. A given state of the corpus is addressable, reproducible, and auditable. You can say what the collection looked like on a date, what changed, and where each object came from. The mechanism is different, the guarantee is not. And for the unstructured half of the Context pillar, the half an agent grounds on through retrieval, this is the store that holds it.

Structured data belongs in a lakehouse

For structured data at scale, the billion-row tables, something genuinely new arrived, and it deserves the headline of this post. Table formats like Delta Lake and Apache Iceberg give a table a transaction log, which means it has a history. You can query the table as of a past version, roll back a bad write, and branch or tag a dataset the way you branch code. That is not a metaphor for version control. It is version control, implemented for data, at a scale git was never going to reach.

This is the substrate under the lakehouse platforms, and I have written about the data layer it enables in The Zero-Copy Promise and The Data Was Always the Problem. I will not reopen the federate-versus-copy debate here. The point for this series is narrower: the structured heart of the enterprise, the entity model I called the floor, can now live under a source-of-truth discipline as real as the one code has enjoyed for decades.

The invariant that unifies them

Three substrates, and it would be easy to see three unrelated technologies. What makes them one thing is the property they share. Each is versioned. Each is diffable or time-travelable, so you can see and undo change. Each is governed, so access and lineage are controlled. And each is indexable, which is the word that matters most, because indexability is what makes the store legible to an agent.

That last property is why this post sits in the middle of the series rather than off to the side. The semantic layer everything-as-code builds is only useful to an AI if the AI can search it, and a source of truth the model cannot index is just expensive storage. Choosing the substrate by data shape is not a data-engineering footnote. It is how you make the whole business retrievable.

What the tooling can’t yet promise

The honest limit here is maturity, not concept. Git has had fifty years to make branching, merging, and blame feel obvious. Data versioning is a decade in and still rough: the three substrates have different mental models, lineage across them is stitched together rather than seamless, and reproducing an exact multi-substrate state, this text plus that lake snapshot plus that table version, is harder than it should be. Anyone who tells you it is solved is selling something.

But the direction is not in doubt, because the property is the same one software already proved out. Pick the substrate that fits the shape of the artifact, insist on versioned, diffable, governed, and indexable in all three, and the source of truth at the center of the AI-first enterprise stops being a slogan and becomes a thing you can actually point at. Git for the words, a lake for the mass, a lakehouse for the tables. One principle, wearing whatever the data requires.

---

If this resonated, read Markdown Is the Knowledge Layer, on the text substrate, and An ERD Is the Floor, Not the Ceiling, on the structured model this versions. Both sit in the Everything as Code series, under the pillar.

Find me on X @orestesgarcia or LinkedIn /in/setsero.