From Medallion to Meaning: Databricks as an Enterprise AI Context Layer

A data lake does not ground anything. You can pour a decade of clean, conformed records into it, hand an agent a pristine Gold table, and still watch it answer the wrong question. The storage was never the hard part. Meaning was.

That is the shift worth watching in Databricks right now. The company that taught the enterprise to think in lakehouses has spent the last two years building the layer above the data: the governed semantics that tell a model, and increasingly an agent, what a number actually means. The lakehouse was the foundation. The context layer is the product.

This post walks the stack from the bottom up, because the only way to understand why the top matters is to see what it stands on.

The lakehouse is the foundation, not the feature

Start with what is actually underneath, because the rest only makes sense on top of it.

A lakehouse is Databricks’ answer to a decades-old split. Data lakes were cheap and flexible but unreliable. Warehouses were reliable but rigid and expensive. The lakehouse keeps one governed copy of data in open formats on object storage and serves BI, machine learning, and streaming from it. No second copy, no nightly export into a warehouse that is already stale by the time anyone queries it.

The reliability comes from Delta Lake, the storage layer that adds ACID transactions, schema enforcement, and time travel to plain Parquet files. A file-based transaction log records every operation in order, and that one mechanism is what makes concurrent reads and writes safe at scale. Schema enforcement rejects writes that do not match the table, so a malformed upstream feed cannot quietly corrupt it. Time travel versions every operation, so you can query or restore any prior snapshot. These are warehouse guarantees on data-lake economics.

The open-format story matters more than it used to. Delta Lake, Apache Iceberg, and Hudi all sit on Parquet, and the real difference is the metadata. Databricks’ Universal Format (UniForm) writes one copy of the data and exposes it to any engine that speaks Delta or Iceberg, and in 2024 Databricks acquired Tabular, the company founded by Iceberg’s original creators. The message was deliberate: the table format should stop being a lock-in decision. Hold that thought, because the lock-in did not disappear. It moved up a floor.

The lakehouse is the layer I keep calling the real starting line in The Data Was Always the Problem. Databricks solves it well. It just does not finish the job, because clean storage and useful meaning are not the same thing.

Medallion: how raw data earns the right to ground AI

Clean data does not arrive clean. The medallion architecture is Databricks’ pattern for making it so, and it is the part most teams either live by or quietly skip. The idea is progressive refinement across three layers, each one improving the structure and quality of the data as it flows up.

Bronze is the raw landing zone. Data arrives from source systems as-is, with a few metadata columns bolted on: load time, source, a process ID. Nothing is cleaned here, on purpose. Bronze is the historical archive and the audit anchor, the thing you reprocess from when a downstream rule changes, without having to go back and re-read the source system.

Silver is where the data gets conformed. Records are matched, merged, deduplicated, and cleansed just enough to be trustworthy and joinable. Databricks describes the goal as an “enterprise view” of the key business entities and transactions, closer to a normalized model than a reporting one. The transformations are deliberately light. The point is a reliable spine, not a finished product.

Gold is the business layer. De-normalized, read-optimized, aggregated for a purpose: customer analytics, segmentation, the tables that feed dashboards and features. Databricks’ docs say the Gold layer “represents highly refined views of the data that drive downstream analytics, dashboards, ML, and applications.” It is the tier everything downstream is supposed to consume.

Here is the part the diagram cannot show, and the reason this post exists. A Gold table is clean. It is not self-explanatory. The column net_revenue_usd is a name, not a definition. The agent reading it still does not know whether that figure is gross or net of returns, which entities roll into it, or which of the four revenue columns the business actually certifies. Medallion gets you trustworthy data. It does not get you meaning. That is a different layer, and it is the one Databricks has been racing to build.

Unity Catalog and the semantic layer: where data becomes meaning

Meaning starts with governance, because you cannot describe what you cannot govern.

Unity Catalog is Databricks’ unified governance layer for data and AI: tables, files, machine learning models, and functions under one set of permissions, with lineage and discovery across all of it. Databricks open-sourced it in 2024. It is the metadata backbone, the map of what exists, who owns it, who is allowed to see it, and where it came from. That map is the precondition for everything above.

The semantic layer sits on top of the map. Databricks calls the current implementation metric views, or Unity Catalog business semantics. The move is to separate a metric’s definition from the dimensions you slice it by, so you define “net revenue per active customer” once and query it any way you like, grouped by region, by month, by segment, without redefining it each time. The definition compiles to deterministic SQL that the engine runs the same way on every call. The line that matters for AI is Databricks’ own: “every consumer, whether human or AI agent, gets the same result from the same definition.”

That is the context layer made concrete. Not retrieved text, meaning. Metric definitions, dimensions, certified values, display names, and synonyms, served at the moment of the query instead of left for the model to infer from column names. I argued the generic version of this in Banks Don’t Need Another Agent. They Need a Context Layer., and made the case that context engineering is infrastructure you own, not a feature you buy, in Context Engineering Is Infrastructure, Not a Skill. Metric views are what that layer looks like when a vendor builds it straight into the data platform.

The payoff shows up at the natural-language surface. AI/BI Genie, generally available since mid-2025, lets people ask questions in plain language and answers by generating SQL grounded in Unity Catalog metadata and metric definitions. The grounding is the whole trick. A value dictionary lets Genie map a request for “first level group” to the literal value “1st line support group” instead of guessing. In Databricks’ own benchmark of 28 real enterprise questions, Genie answered 84.5 percent correctly on the first attempt, against 52.4 percent for the strongest general-purpose coding agent they tested. The models underneath were comparable. The difference was the semantics.

Grounding the model: vector search, agents, and the serving tier

Structured metrics answer the questions you can phrase in SQL. A lot of enterprise knowledge does not live in tables, and that is where the rest of the stack comes in.

Vector Search is the managed retrieval layer for the unstructured half. It builds an index from a Delta table and keeps it synced as the source changes, so the embeddings do not silently drift out of date. You can let the platform handle chunking and embedding, or bring your own vectors. The detail that matters most is governance: the index is a Unity Catalog entity, bound by the same permissions as the data it indexes, so retrieval for a RAG app inherits those access controls instead of quietly routing around them. This is the retrieval side of the problem I wrote about in Grounding Is a Memory Problem, and it assumes less about document shape than the single-index habit I pushed on in Your RAG Assumes One Shape.

On top of retrieval sits Mosaic AI, the agent and model-serving layer. Agent Bricks, announced in 2025 and still early, lets you describe a task, connect governed data, and have the platform auto-generate evaluations and tune the agent for quality and cost. The evaluation piece is the part worth noticing. MLflow ships built-in groundedness judges that check whether a response is actually supported by the context it was handed, which is how you catch a confidently wrong answer before it reaches a user. And Lakebase, a managed Postgres tier added in 2025, gives agents a low-latency transactional store that still shares lineage and governance with the lakehouse, closing the gap between the analytical data and the operational state an app actually runs on.

Read the whole path end to end and the shape is clear. Governed data in Delta, refined through medallion, mapped by Unity Catalog, given meaning by metric views, retrieved through Vector Search, reasoned over by an agent, and checked by a groundedness judge. Every step shares one governance plane. That last property is the one that counts in a regulated shop, where the question an examiner asks about an agent is the same question they ask about a person: what did it access, under whose authority, and can you reconstruct the chain. A stack with one lineage graph can answer that. A stack stitched from six vendors usually cannot.

The honest limit: one vendor, one bill, one lock-in

Everything above is the capability case, and it is a strong one. The control case is just as real, and it is why this is an architecture decision rather than a purchase order.

Start with cost. Databricks bills on consumption, in compute units stacked on top of your cloud infrastructure bill, and consumption pricing is notoriously hard to forecast before the workload is real. The platform’s own efficiency claims, the “faster” and “cheaper” numbers scattered through its announcements, are vendor benchmarks measured on vendor terms. Useful as direction, useless as a budget.

Then complexity. A full lakehouse is cluster sizing, autoscaling, Spark tuning, and pipeline orchestration, and that operational weight lands on your team. For a single well-served BI workload it is overkill. The semantic layer in particular is young: metric views require a recent runtime, forbid some nested aggregations, and are authored artifacts that someone has to keep in sync as the tables beneath them change. There is also an irony worth naming out loud. The metric layer is deterministic, but Genie sitting on top of it is still probabilistic, so the promise that two queries return the same number does not fully survive the natural-language interface that made the feature attractive.

Then lock-in, which did not vanish when the table format opened up. Delta and Iceberg are open, and UniForm plus the Tabular acquisition make the raw storage genuinely portable. But the valuable thing was never the Parquet. It is the compiled meaning: the metric definitions, the lineage, the access policy, the accumulated corrections of a hundred arguments about what a number means. That asset lives in Unity Catalog, and leaving means rebuilding it. Microsoft is making the same play from the other side, which I covered in Foundry and Fabric, and the question for both is identical, the one I raised in The AI Context Portability Problem: who owns the meaning when it is time to leave.

The reconciliation is not to pick a side. It is to see clearly that the open format is the floor and the semantic layer is the ceiling, and the ceiling is where the lock-in now lives. A single governed stack is worth real money precisely because meaning served from one lineage graph is what makes enterprise AI defensible. You just want to walk in knowing which part you can take with you and which part you are renting.

What the lakehouse was always building toward

The lakehouse was never the point. It was the thing that had to exist first.

The reason Databricks matters in the AI conversation is not that it stores petabytes cheaply, though it does, across more than 20,000 organizations and most of the Fortune 500 by its late-2025 numbers. It is that the company turned storage into a place where meaning is governed once and served everywhere, to a dashboard, a SQL query, or an agent, with the same definition and the same lineage behind it. Raw data does not ground anything. A context layer does, and the companies that win the next round of enterprise AI will be the ones that built that layer on purpose instead of hoping the model would infer it.

The medallion gets your data clean. Unity Catalog and the semantic layer get it meaningful. The second half is the harder half, and it is the half that was missing.

---

The companion read is Banks Don’t Need Another Agent. They Need a Context Layer., which makes the platform-agnostic version of this argument. For where the compiled meaning should live when you can leave, see The AI Context Portability Problem, and for the same data-wall thesis seen through another stack, The Data Was Always the Problem.

I write about AI-assisted development, enterprise architecture, and building in regulated environments. If you’re weighing a lakehouse and its semantic layer against your own data estate, I’d like to compare notes. Find me on X @orestesgarcia or LinkedIn /in/setsero.