System Design · medium

Batch vs streaming: how do you decide for a given use case?

Asked in Data Engineer interviews, in the System Design round.

Short answer

Latency needs, cost, complexity, data freshness SLAs, and correctness guarantees.

How to answer it

Ask how quickly the consumer of the data needs to act on it, and what a wrong or late answer costs. Everything else follows.

Batch when:

Streaming when:

The middle, which is most cases: micro-batch every few minutes gives near-real-time freshness with batch semantics, and is much simpler to operate than true streaming.

The costs to state: streaming needs exactly-once or idempotent consumers, state management, handling of out-of-order and late events with watermarks, and 24/7 operation. A batch job that fails re-runs in the morning; a stream that falls behind needs someone now.

Worked line: "The dashboard refreshes hourly and the analysts look at it twice a day; that is batch, hourly. The fraud check has to answer before the payment completes; that is streaming, and it is a different system with a different team on call."

Related questions

Practice this for real