You built the Event Hub. You stood up the cluster, set the replication factor, sized the partitions, and asked the vendor to produce to it. They agreed to “event streaming.” What they wired up was a bare HTTP POST that returns a 201 with an empty body.

The next morning the daily ETL batch is short. The counts do not tie out against the source system. Which events are missing? You cannot say. There is no offset to walk, no sequence number to diff, nothing to replay. The vendor got a 201 for every call it decided to make, and that 201 is the entire audit trail. This is the failure I want to take apart, because it is invisible until reconciliation, and by then the events are gone.

You own the broker; the vendor owes you a producer

Here is the part that gets lost in the sales call. Resilience in an event pipeline is a producer-side property. The broker can be flawless, replicated three ways, every partition healthy, and you will still lose events if the thing writing to it does not participate in the guarantees.

You built durability into the platform: replication, in-sync replica sets, durable offsets a consumer can rewind. None of it engages on the write path, because the vendor’s write path does not speak to any of it. A bare POST to an ingestion endpoint is acks=0 with extra steps and a friendlier status code. The producer fires, the socket closes, and whether the event was durably committed is a question nobody on the write side ever asked.

So the contract that matters is not “can you send us events.” It is “how do you produce,” and that is exactly the contract a POST leaves blank.

The producer ladder

Start with the vocabulary, because it is where the ambiguity hides. Kafka’s own delivery semantics name three guarantees: at-most-once (lost on failure, never redelivered), at-least-once (never lost, may duplicate), and exactly-once (once and only once, through failure). Every producer sits on one rung, and the rung is a function of the machinery, not the marketing.

Native Kafka producer, over TCP and TLS. The full surface. acks=all means the leader waits for every in-sync replica before it acknowledges, and paired with min.insync.replicas=2 a write is not durable until a majority has persisted it. enable.idempotence=true stamps each record with a producer ID and a per-partition sequence number, so the broker deduplicates retries and, critically, can detect a gap in the sequence. delivery.timeout.ms and retries turn a transient failure into a bounded, deduplicated retry instead of a lost event. Per-key ordering holds. This is the rung where “what got lost” is an answerable question.

Kafka REST bridge. Confluent’s REST Proxy keeps acks and lets you target a partition, but the idempotent producer and transactions are not in its capability set, a request addresses a single topic-partition, and every write now inherits HTTP timeout ambiguity: a dropped response leaves you unsure whether the write landed.

Event Hubs Kafka endpoint. A re-implementation of the Kafka protocol that runs none of Apache’s code. Idempotent producers are supported. Transactions and exactly-once are a Premium and Dedicated preview at best, and Microsoft’s own docs contradict each other on whether they are supported at all, so do not design a ledger on exactly-once here yet.

Bare HTTP POST. The rung the vendor called streaming. POST to a /messages endpoint, get a 201 and an empty body. No producer state machine, no acks contract surfaced to the caller, no idempotency, no sequence number. It collapses to at-most-once.

What “never made it” means, concretely

A POST loses events in ordinary, undramatic ways. The connection times out mid-write and the producer never learns whether the event committed. The broker fails a partition leader over in the same window and the un-acknowledged write evaporates. The producer process crashes with events still buffered in memory. The endpoint returns a 5xx and there is no retry policy behind the call, so the event is simply dropped.

Every one of those is a silent at-most-once loss. And the naive fix makes it worse: retry a POST after an ambiguous timeout with no idempotency key and you now duplicate, which in a ledger is its own reconciliation break in the other direction.

The deepest problem is the missing sequence number. An idempotent producer numbers its records per partition, so the broker can see that it received 5 and 7 but never 6. A POST carries no such counter. The broker cannot detect the gap, because from its side there is no gap, only fewer calls than the source system made.

Why next-day reconciliation can’t recover it

This is where the design decision comes due. Your T+1 ETL reconciliation diffs the events you consumed against the system of record and reports a delta: you are short by some number. It gives you a count, not a manifest. It cannot hand you the identities of the missing events, because the only artifact of a lost POST is its absence.

Contrast the rung you asked for. Per-partition sequence numbers make gaps detectable in-band, at write time, not the next morning. Durable consumer offsets make any range replayable, so a detected gap is also a recoverable one. On the POST path there is neither, so reconciliation degrades into a smoke alarm with no map to the fire. You know something burned. You cannot prove what, and you cannot re-request it, because there is no offset to rewind to and no idempotency key to ask about.

Why they get away with it

None of this is a secret to the vendor. It is a pricing decision. In core financial-services software there are few substitutes and the switching cost is measured in years and seven figures, so “no” is the vendor’s cheapest possible answer. Building a real producer costs engineering effort that a POST does not, and the guarantee surface it buys is invisible in a demo and unnamed in the purchase order. So the producer ships on the weakest rung, keeps the word “streaming,” and lets the delta surface on your side at reconciliation, where it is your operational problem, not their contractual one.

Make the producer contract auditable

The fix is to make the producer a written requirement, not a feature name. Specify the rung: a native Kafka or AMQP producer with acks=all and enable.idempotence=true, so retries are bounded and deduplicated and gaps are detectable. If HTTP is genuinely non-negotiable, then demand the parts that make it auditable: acknowledged writes, a client-supplied idempotency or sequence key on every message, and a replayable reconciliation feed, an offset or a queryable by-id endpoint, so a T+1 shortfall becomes a targeted re-request instead of a shrug.

The same discipline I argued for with data movement in The Zero-Copy Promise applies here: the contract that matters is the one that governs the data under failure, not the one that looks fine when everything is healthy. “Event streaming” has to mean a delivery guarantee you can audit the next morning, or it means nothing.

---

The switch is the moat, and a 201 with an empty body is what the moat lets them ship. Read the producer contract before the reconciliation reads it back to you.

If this resonated, the companion read is Foundry and Fabric, Not Foundry or Fabric, which works the same seam from the control-plane side: who owns the layer you depend on, and what it costs to leave.

I write about AI-assisted development, enterprise architecture, and the vendor relationships that quietly shape both. Find me on X @orestesgarcia or LinkedIn /in/setsero.