System Design · hard · Reported at Netflix
Design a data pipeline to ingest 1B events/day.
Asked in Data Engineer interviews, in the System Design round.
Short answer
Kafka ingestion -> stream/batch processing (Spark/Flink) -> warehouse/lake, partition & schema mgmt.
How to answer it
Requirements first
Say the numbers out loud before drawing anything. A billion a day is about 11,600 events per second on average, and peak is what you size for, usually three to five times that. At 1KB per event that is a terabyte a day, so retention and storage cost are part of the design rather than a footnote.
Then the questions that change the architecture:
- How fresh does this need to be? Seconds, minutes, or next morning?
- Can we lose an event? Usually no for billing, sometimes yes for telemetry.
- Exactly once, or at least once with idempotent consumers?
Architecture
The diagram above is the shape. Each box exists for a reason:
- Producers write to an append-only log such as Kafka or Kinesis. The log is the buffer that absorbs bursts, so a slow consumer creates lag rather than data loss.
- A stream job validates, enriches and routes. Anything unparseable goes to a dead-letter topic rather than the floor.
- Raw events land in object storage, immutable and partitioned by date. This is the replay source, and it is what turns a modeling bug into a rerun instead of an incident.
- A batch or streaming transform models the raw data into the warehouse for BI.
Delivery semantics
at most once fast, drops on failure. Rarely acceptable.
at least once the practical default. Duplicates WILL happen.
exactly once expensive end to end, and usually unnecessary.The affordable answer is at least once plus idempotent consumers keyed on an event id, which gives the same result as exactly once without the cost. Say that phrase; it is what the question is testing.
Scaling and cost
- Partition the log by a key with even distribution. Partitioning by customer id gives one enormous partition for your biggest customer.
- Parquet, not JSON, at rest. Columnar and compressed, so downstream reads cost a fraction.
- Tier storage: hot for 30 days, cold after. A terabyte a day is 365TB a year, and most of it is never read again.
- Scale consumers horizontally; partition count sets the ceiling on parallelism.
Failure modes
- Consumer lag growing: alert on it, because it is the leading indicator of every downstream problem.
- Schema change upstream: a schema registry with compatibility rules turns a silent corruption into a rejected write.
- Hot partition: skewed keys, so monitor per-partition throughput.
- Poison message: without a dead-letter topic, one bad event blocks a partition indefinitely.
Key points
- Do the arithmetic first: 11,600 a second, and size for peak.
- The log decouples producers from consumers. That is why it is there.
- Raw immutable storage makes bugs replayable rather than fatal.
- At least once plus idempotency is the pragmatic delivery guarantee.
- Monitor consumer lag, schema changes and partition skew.
FAQ
- Kafka or a managed queue? Kafka for high throughput and replay; a managed service to avoid running it. The design does not change.
- Why not write straight to the warehouse? It couples ingestion to the warehouse's availability and cost, and loses the replay buffer.
- Do I need a data lake at all? At this volume, yes. Storing raw in the warehouse is far more expensive for data you rarely read.
The 30-second answer
First the arithmetic: a billion a day is around eleven thousand six hundred events a second on average, and I would size for a peak several times that.
The shape I would draw is producers writing to an append-only log, a stream job that validates and enriches, raw events landing immutably in object storage partitioned by date, and a transform into the warehouse for BI. The log is there to decouple producers from consumers so a slow consumer creates lag rather than loss, and the immutable raw layer is there so a modeling bug becomes a replay rather than an incident.
On guarantees I would aim for at least once delivery with idempotent consumers keyed on an event id, because that gets exactly-once results without exactly-once cost. And anything unparseable goes to a dead-letter topic, so one bad event cannot block a partition.
In the interview
- What they are checking: a raw immutable layer, event-time partitioning, dedup, and replay.
- Common mistake: a diagram of boxes with named products and no answer to "what happens when a job fails at 3am".