Don't Build the Hippocampus

Last week I argued that the enterprise second brain needs a body, and I called the recall organ the hippocampus: the part that remembers, promotes the signal worth keeping, and hands it back on demand. A reader pointed me at Hindsight, and the honest reaction was not surprise. It was relief.

Because the recall organ is already built. Not as a weekend repo that will rot by spring, but as a maintained open-source project with a company and a paid tier behind it, in use by a lot of people and still growing. That changes the question. It is no longer how to build the hippocampus. It is why you would.

This is the build-versus-buy edition of A Second Brain Needs a Body. Some organs you should adopt. Some you have to grow yourself. Telling them apart is the whole skill.

The recall organ is already a reference implementation

Give the thing its due, because it earned it. Hindsight, from Vectorize, is an MIT-licensed agent-memory system whose pitch is memory that learns rather than a transcript you replay. Its loop is Retain, Recall, Reflect: store a memory, retrieve it through several strategies at once, and periodically reflect to form new connections and update what it believes. It holds typed memory, world facts, experiences, observations, mental models, and it recalls by semantic similarity, keyword, graph, and time together rather than one blunt vector search.

More to the point for anyone shipping to real users, it already carries the properties I said in The Second Brain Doesn’t Ship arrive the moment strangers log in. Memory is isolated per tenant, in what it calls memory banks, so one customer’s memory never surfaces for another. It scans for secrets and PII on the way in. It runs on Postgres with pgvector, speaks to the usual agent frameworks, and exposes an MCP server.

The part that matters most is not any single feature. It is that a funded company maintains it and monetizes a hosted version. That is the difference between a dependency you can lean on and one that becomes your problem in six months when the sole author loses interest. A commodity organ that thousands of teams exercise and a business is paid to keep alive is exactly the kind of thing you should not be hand-rolling. The best code is the code you did not write, and the recall organ is now that code.

So adopt it. Treat it as the reference implementation of agent memory and move on. The relief is real, and it is earned.

But memory is not knowledge

Here is the line that keeps the relief honest. Hindsight remembers what the agent learned. It does not know what is true about the business.

Those are different organs. An agent’s memory is the accumulation of what it observed in conversations and tasks: this customer prefers reshipments, that workflow usually stalls at step three. Valuable, and exactly what Hindsight is for. But the authoritative definition of net revenue, the system of record for a customer’s positions, the governed table an examiner will ask about, none of those are the agent’s memories. They are the institution’s facts, and they live somewhere the agent was never the author.

Blur the two and you rebuild the collector’s fallacy inside the memory layer. A store that learned a definition from a conversation will happily recall a number that was never certified, with the same confidence it recalls a real one. Memory of what was said is not knowledge of what is true, and no amount of better recall closes that gap, because it is a sourcing problem, not a retrieval one. This is the grounding distinction I drew in Grounding Is a Memory Problem, seen from the other side: adopting a great memory organ does not hand you the knowledge organ. You still have to build that.

What still has to come from the lakehouse

The organs you cannot download are the ones that encode your business, because no company can sell you the meaning that makes you you. Three of them, and they are the hard three.

The metabolism, which is the governed business-data substrate. A memory store keeps what it is told. It has no pipeline from your systems of record, no notion of raw intake refined into trustworthy, business-ready data. That is the medallion architecture’s job, Bronze to Gold, which I walked through in From Medallion to Meaning. Hindsight remembers; the lakehouse metabolizes the actual data of the business into something an agent can stand on.

The shared meaning, which is the certified semantic layer. One deterministic definition of a metric, compiled once and resolved the same way for every consumer, human or agent. A memory bank can remember a definition. It cannot certify one, enforce it across the estate, or guarantee that two agents asking the same question get the same number. That guarantee lives in the semantic layer on the platform, not in any one app’s memory.

The governance plane, which is lineage, access control, and audit over everything. This is the subtle one. Hindsight governs its own memories well, per-bank isolation and PII scanning. But that is app-level governance of one store. The enterprise plane has to govern the whole estate: which human or which agent may touch which table, with lineage an auditor can follow from a number back to its source. Governing the memory layer is necessary and not remotely sufficient. The plane sits under all the organs, not inside one of them.

The architecture that falls out of this is not either-or. It is Hindsight wired into the lakehouse: the recall organ adopted whole, reaching into governed data and certified definitions for what it must not invent. Memory recalls. The substrate grounds. Neither is the other’s job.

The judgment is the whole post

Strip the anatomy away and this is a build-versus-buy decision, which means it is a judgment call about where your scarce engineering budget earns the most.

Reusing a maintained memory layer is not free, and the honest ledger has a second column. You take on a dependency and its release cadence. If you reach for the hosted tier you accept a data boundary and whatever egress it implies, the kind of question I raised in The AI Context Portability Problem. Even self-hosted, you inherit its memory model and a quieter lock-in to how it thinks about recall. Those are real, and in a regulated seat they are worth a careful read before you adopt.

But weigh that against the alternative, which is spending months rebuilding a commodity organ that a funded team already maintains and thousands already trust, and doing it instead of the work only you can do. The governed meaning of your business is the one asset no vendor can hand you, because it is the thing that differentiates you. That is where the build budget belongs. The memory organ is not.

So the rule is the unglamorous one that keeps being right. Adopt the parts that are good, maintained, and generic. Build the parts that are yours and only yours. Reuse the hippocampus. Spend yourself on the body.

---

This is a follow-up to A Second Brain Needs a Body, which makes the case that the enterprise second brain is a cortex in search of a body. For the lakehouse half of that body in detail, see From Medallion to Meaning, and for where the markdown version stops scaling, The Second Brain Doesn’t Ship.

I write about AI-assisted development, enterprise architecture, and building in regulated environments. If you’re drawing your own build-versus-buy line across the agent stack, I’d like to compare notes. Find me on X @orestesgarcia or LinkedIn /in/setsero.