System Design · hard · Asked at Netflix

Design a data pipeline to ingest 1B events/day.

Asked in Data Engineer interviews, in the System Design round.

Short answer

Kafka ingestion -> stream/batch processing (Spark/Flink) -> warehouse/lake, partition & schema mgmt.

How to answer it

A billion a day is about 12,000 events a second on average and perhaps 50,000 at peak. That is a real but ordinary load; the design is about durability and replayability more than raw throughput.

Walk it stage by stage:

The parts that show experience:

Back-of-envelope: at 1KB an event, a billion a day is a terabyte raw, roughly 100 to 200GB compressed columnar, which is small.

Related questions

Practice this for real