Your RAG Assumes One Shape of Knowledge

I argued that your knowledge belongs in plain text, in git, where you can read it and diff it. I still believe that. But storing knowledge well and retrieving it well are two different jobs, and the second one is where most systems fall apart. The flat wiki is the right store. A single vector index over it is the wrong retriever.

Here is the part the “just embed everything” crowd skips. Knowledge does not come in one shape. It comes in at least four, and a vector store is genuinely good at exactly one of them. Point it at the other three and it does not fail loudly. It returns a confident, plausible, wrong chunk, which in a regulated shop is the most expensive failure mode there is.

The four shapes, and why one retriever cannot serve them

Classic RAG had one loop: a user types a question, the system finds three semantically similar chunks, the model writes a paragraph. That worked when the answer lived in a couple of paragraphs. Agents do not ask a question and stop. They run a task, and a task needs operating context, not the three nearest neighbors.

Sort enterprise knowledge by its actual structure and you get four buckets:

• Fuzzy prose. Blog posts, documentation, emails, runbooks. This is what embeddings were built for, and they are genuinely good at it. If all your knowledge looked like this, one vector index would be the whole answer. It does not.
• Structured documents. Contracts, filings, policies, legal agreements. Chunk a contract and you sever the clause from the definitions section that gives it meaning, and from a schedule forty pages away that silently overrides a general term. The nearest-neighbor chunk reads fine and means the opposite of what the document says.
• Tables. ERP rows, CRM records, governed metrics, supplier data. Flattening a spreadsheet into text and asking a language model to reason over the prose is the wrong abstraction entirely. The number is right there in a governed column. You do not want a paraphrase of it.
• Graphs. Supplier-to-shipment chains, incident root causes, entity relationships. This is relational work, and neither chunks nor tables carry it. Following an edge is not a similarity operation, and no amount of cosine distance turns it into one.

The retrieval unit has to match the kind of work being done. A chunk works for an FAQ. A whole section works for a filing. A row and its schema work for financial analysis. A traversal works for a dependency question. Better embeddings do not fix this. All a better embedding does is find more relevant text, which is a 2025 answer to a 2026 problem. The choice was never “vector database X versus Y.” It is which shapes your agent actually touches.

When the vector-database company demotes vector search

You do not have to take the shape argument from a skeptic. Take it from Pinecone.

It is a strange thing when a vector-database company ships a product whose premise is that vector search is not enough. That is what Pinecone did with KnowQL. The pitch is a query language that makes retrieval carry more than similarity: intent, filters, access policy, provenance, response shape, confidence, and budget, all riding along with the query instead of being bolted on afterward. Pure semantic ranking gets demoted from “the job” to “one signal in the job.”

Read that honestly and it is an admission. Even the company whose entire brand is the vector index now says the vector index does not define the whole contract anymore. A chatbot needs related text. An agent needs operating context. Those are different problems, and the incumbent expanded the contract before someone else expanded it for them. When the market leader quietly reprices its own core primitive, that is the signal, not the noise.

Some documents should never be chunked at all

The structured-document problem got a sharper answer than “chunk smaller.” Page Index makes the case that a lot of documents should never be chunked in the first place.

Instead of splitting a filing into fragments and embedding each one, Page Index builds a hierarchical tree of the document, a table of contents with a summary on every node, and lets the model reason down through the tree to the right section. No embeddings. No vector similarity. The model navigates structure the way a human analyst flips to the right part of a 10-K, rather than groping for the three nearest fragments. On FinanceBench, the reported accuracy for the tree approach is 98.7 percent, and the mechanism is exactly the point: the retrieval unit is a section, because a section is the unit the work operates on. Treat your git-tracked markdown as a tree with real headings and you get the same benefit, which is one more reason the store and the retriever are separate design decisions. I made the store side of that case in Markdown Is the Knowledge Layer; the tree retriever is the missing half.

The table bet: SAP says embeddings are the wrong layer

The most direct rejection of vectors-for-everything is a spending decision, not a blog post. SAP put real money on the table, literally, betting that tables carry enterprise knowledge and embeddings do not.

SAP acquired two companies whose value has nothing to do with RAG in the classic sense: a lakehouse and semantic-layer vendor for governed, federated, permissioned, lineage-tracked access to business data, and a tabular-foundation-model group whose TabPFN work was published in Nature. Tabular foundation models exist for one reason: table-native reasoning beats text-derived reasoning on structured data. Churn risk, supplier risk, renewal forecasting, none of it survives being flattened into prose and handed to a language model as a paragraph.

The through-line is governance. For an enterprise agent, the source of truth on a revenue number is a governed table with a defined metric, not a sentence lifted from a PDF. Get that wrong and it is not a slightly-off summary, it is real money moving on a hallucinated figure. This is the entity-and-relationship layer I called the floor in The ERD Is the Floor: the graph of what relates to what, and the governed tables underneath it, are not a legacy inconvenience the agent reads around. They are the retrieval target. SAP is betting the knowledge layer is the data in the shape the business already stores it, and the LLM is a consumer of that layer, not a replacement for it.

The regulated wrinkle: grounding and a traceable citation

Everything above is an architecture argument until you put it in a regulated shop, and then it becomes a control requirement.

You cannot instruct a language model not to hallucinate any more than you can instruct autocomplete not to autocomplete. There is no separate truth-checking pass inside the weights for a prompt to hook into. Sullivan and Cromwell, one of the most prestigious firms in the world, filed a bankruptcy motion with dozens of fabricated or misquoted AI citations, and their own review missed it. The opposing side caught it. The model was not the problem. The environment around the model was.

The fix is grounding, and grounding is a shape problem. A regulated answer has to trace to a specific authoritative source: this clause in this executed contract, this row in this governed table, this node in this document tree. A vector store gives you “here is a chunk that scored 0.87,” which is a similarity claim, not a provenance claim. It cannot tell you whether the chunk is the authoritative version or a superseded draft, because ranking is not lineage. When an examiner asks why the agent said what it said, “it was semantically close” is not an answer. “It cited section 4.2 of the current executed agreement, which the tree retriever resolved and the semantic layer marked authoritative” is. The retriever that can produce that sentence is the one you can put into production, and it is almost never a bare vector index.

Write the bundle before you buy the database

The practical move is to stop shopping for a database and start writing down what the agent actually needs. For a refund agent that is the customer record, the plan, the region, the product version, the purchase history, the refund policy, the threshold, prior exceptions, the current ticket, and the approved response language, plus whether the agent can issue a refund or only draft one. That is a bundle, and writing it out makes three things obvious: most fields live in different systems, some need to be governed rather than merely retrieved, and the real work is assembling and reasoning over the bundle, not searching for documents.

Once the bundle is on paper the shopping is boring. Prose-heavy fields go to vector search and document trees. Governed business data goes to a semantic layer and tabular reasoning. Relational fields go to a graph. Most real agents need a mix, which is exactly why one index over everything is the wrong default. I laid out how the pieces compose in The Context Systems Blueprint; the bundle is the input that tells you which pieces you actually need.

Pinecone, Page Index, and SAP are not fighting over the same slot. They are solving for different shapes: prose, structured documents, and governed tables. Once you know your bundle, the argument about which one is “best” dissolves, because the honest answer is usually “three of them, for the three shapes you have.”

Why this is on the Karpathy thread

Andrej Karpathy’s point that the flat wiki does not scale is right, and it is right about retrieval, not storage. The store stays plain text in git, readable and diffable, and that is the part I will keep defending. What does not scale is pretending that one similarity index is an adequate way to get knowledge back out of that store when the knowledge is really four shapes wearing one file extension.

Own the store. Then own four retrievers, or as many as your shapes demand, and make each one produce a citation you could hand to an examiner. That is the version of the knowledge layer that survives contact with both the model and the regulator.

---

The store was the easy half. The retriever is where the architecture actually lives, and where the four shapes stop being a diagram and start being a build decision.

Find me on X @orestesgarcia or LinkedIn /in/setsero.