Make each step idempotent and partition-scoped: a run owns a date partition and fully replaces it, so re-running converges instead of accumulating. Checkpoint at partition boundaries rather than per row — recovery granularity should match the unit you can safely recompute. Side effects that cannot be repeated (incrementing a counter, sending an alert, charging something) either move outside the retryable path or get keyed and deduplicated.
How to answer it
Design so that re-running from the start is always correct, and re-running from the failure point is merely faster. Both need the same property: every step is idempotent.
Partition-scoped writes. A run owns a date partition and replaces it completely: delete-then-insert in a transaction, or write to a new location and swap. Append-only steps are the ones that double-count on retry.
Deterministic inputs. A step reads a fixed snapshot (a partition, a versioned file), not "whatever is in the source now", so a re-run at 9am produces the same output as the run at 2am would have.
Checkpoint at the boundary you can recompute. Per partition or per step, not per row. Row-level checkpoints are complex and rarely needed once steps are idempotent.
Side effects outside the retryable path. Sending the alert email, incrementing a counter, calling a payment API: either move them to a final step that runs once after everything succeeded, or key them so a repeat is deduplicated.
Orchestration that knows the dependency graph, so a failure in step 4 re-runs 4, 5, 6 and not 1 to 3, and marks the run as failed rather than partially done.
-- the shape of an idempotent step
BEGIN;
DELETE FROM daily_agg WHERE run_date = :d;
INSERT INTO daily_agg SELECT ... FROM raw WHERE event_date = :d;
COMMIT;
Then say what you do when you find a step that cannot be made idempotent: isolate it, make it the last step, and guard it with a "has this already been done for this date" check.
What they are checking: the word idempotent, partition-scoped replacement, and the treatment of side effects.
Common mistake: adding "retry 3 times" to the job and calling it resilient. Retrying a non-idempotent step three times makes three copies.