System Design · medium

A nightly pipeline fails halfway through. How do you design it so re-running is safe?

Asked in Data Engineer interviews, in the System Design round.

Short answer

Make each step idempotent and partition-scoped: a run owns a date partition and fully replaces it, so re-running converges instead of accumulating. Checkpoint at partition boundaries rather than per row — recovery granularity should match the unit you can safely recompute. Side effects that cannot be repeated (incrementing a counter, sending an alert, charging something) either move outside the retryable path or get keyed and deduplicated.

How to answer it

Design so that re-running from the start is always correct, and re-running from the failure point is merely faster. Both need the same property: every step is idempotent.

-- the shape of an idempotent step
BEGIN;
DELETE FROM daily_agg WHERE run_date = :d;
INSERT INTO daily_agg SELECT ... FROM raw WHERE event_date = :d;
COMMIT;

Then say what you do when you find a step that cannot be made idempotent: isolate it, make it the last step, and guard it with a "has this already been done for this date" check.

Related questions

Practice this for real