System Design · hard · Reported at Uber
How would you design a feature store for ML?
Asked in ML Engineer interviews, in the System Design round.
Short answer
Offline (batch) + online (low-latency) stores, feature registry, point-in-time correctness.
How to answer it
Requirements first
A feature store exists to solve two problems, and it is worth naming both before designing anything:
- Training-serving skew. The feature computed in a notebook and the one computed in production drift apart, so the model sees something different at inference than it learned on.
- Reuse. Ten teams each rebuild "orders in the last 30 days" slightly differently, and none of them match.
Architecture
Sources warehouse tables, event streams
Transformation ONE definition per feature, versioned in git,
materialised to both stores from the same code
Offline store columnar, historical, point-in-time correct
used for training and backfills
Online store low-latency key-value, current values only
used for serving, single-digit millisecond reads
Registry metadata: owner, definition, freshness, lineage, and which
models consume itThe single definition feeding both stores is the whole point. Two definitions means skew, and skew is what the thing exists to prevent.
Point-in-time correctness
The hardest requirement, and the one that separates a real answer from a diagram. Building a training set for an event at 3pm on 4 March must use feature values as they were at 3pm on 4 March, not today's values. A naive join on entity id pulls current values into historical rows, which is leakage, and it produces a model that scores beautifully offline and fails live.
That is an as-of join on the feature's own timestamp, exactly the same range join as a Type 2 dimension.
Tradeoffs
- Build or buy: Feast, Tecton or a warehouse-native approach against your own. At one or two models a store is overhead; the case appears with many models and shared features.
- Freshness against cost: streaming features are expensive, and many features are fine computed daily.
- Online store choice: Redis or DynamoDB for latency, at the price of another system to run.
Key points
- One definition, materialised to both stores, is what kills training-serving skew.
- Point-in-time correct joins are the core requirement, not a nice-to-have.
- The registry matters as much as the storage: ownership, freshness and lineage.
- Not every team needs one. With few models the complexity outweighs the benefit.
- Backfill is a first-class feature. A new feature is useless without history to train on.
FAQ
- Can the warehouse be the offline store? Yes, and that is often the pragmatic answer, with a small online cache for serving.
- What about feature versioning? Treat a changed definition as a new feature. Silently redefining one invalidates every model trained on it.
- How does this relate to leakage? Point-in-time correctness is the structural fix for temporal leakage.
The 30-second answer
A feature store exists to solve training-serving skew and reuse: one definition of a feature, versioned in code, materialised into both an offline store for training and an online store for low-latency serving.
The hard requirement, and where I would spend my time, is point-in-time correctness. When I build a training set for an event that happened at three in the afternoon on a given day, I need the feature values as they were at that moment, not as they are now. A naive join on entity id pulls today's values into historical rows, which is leakage, and produces a model that looks excellent offline and fails in production. That is an as-of join on the feature timestamp.
I would also say that not every team needs one. With two models it is overhead; the case appears when many models share features.
In the interview
- What they are checking: point-in-time correctness and the single-definition, dual-store idea.
- Common mistake: describing a feature store as a database of features, with no answer for training and serving skew.