System Design · medium
How do you validate a new dashboard before you ship it?
Asked in BI Developer interviews, in the System Design round.
Short answer
Reconcile totals against a trusted source (finance, the source system, a known-good query), spot-check a few rows end to end, and confirm filters and date logic behave at the edges (empty selection, a single day, a year boundary) at every drill level. A dashboard that is wrong once loses trust for good, so validation is the job, not an afterthought.
How to answer it
The principle
A dashboard is a product with users who will make decisions from it. Validate the numbers, the definitions and the reading, in that order, because a chart that is correct and misread is still a failure.
Validate the numbers
- Reconcile totals against a trusted source: the finance system, the production database, last month's signed-off report. A number that cannot be tied out should not ship.
- Recompute a sample by hand, or in a separately written query, and compare.
- Check the edges: earliest and latest dates, a segment with almost no data, a filter combination returning nothing.
- Check the grain. Sum the detail and confirm it equals the summary. This catches fan-out from a join.
- Compare against yesterday's figures if a version already exists, and explain any difference before shipping rather than after.
Validate the definitions
- Every metric has a written definition and an owner.
- Those definitions match the rest of the organization. If active user already exists, use that one rather than inventing a second.
- Filters are stated on the dashboard: excludes internal accounts, excludes test orders, UTC.
Validate the reading
Show it to someone in the audience without narrating it, and ask what they conclude. If they reach the wrong conclusion, the dashboard is wrong even though the numbers are right. Watch for truncated axes, dual axes, missing context on whether a number is good, and a default date range that flatters.
Before it ships
- Load it cold and time it. A dashboard nobody waits for is a dashboard nobody uses.
- Confirm the refresh schedule and add a data-as-of timestamp.
- Decide who is on the hook when it breaks, and add freshness and row-count alerts on the underlying models.
Key points
- Reconcile against an independent source of truth before anything else.
- Sum the detail and check it matches the summary, to catch join fan-out.
- Use existing metric definitions rather than inventing parallel ones.
- Test comprehension with a real user, silently.
- Ship a data-as-of timestamp and an owner, not just charts.
FAQ
- How much validation is enough? Proportional to the decisions it drives. A board metric deserves reconciliation; an exploratory chart does not.
- What if it disagrees with an existing dashboard? Resolve it before shipping. Two live numbers is the problem, not the discrepancy.
- Should stakeholders review it? Yes, but for interpretation. They cannot validate the pipeline for you.
The 30-second answer
I would validate three things in order: the numbers, the definitions and the reading.
For the numbers I would reconcile totals against an independent source of truth, recompute a sample with a separately written query, check edge cases like the earliest date and an empty segment, and sum the detail to confirm it matches the summary, because that catches a join fanning out.
For the definitions I would make sure every metric is written down with an owner and matches what the rest of the company already means by that word, rather than inventing a second definition of active user.
Then the reading. I would show it to someone in the audience without explaining it and ask what they conclude, because a chart that is correct and misread has still failed. And I would ship it with a data-as-of timestamp, an owner, and freshness alerts on the models behind it.
In the interview
- What they are checking: reconciliation against an external source, row-level tracing, and edge cases at every drill level.
- Common mistake: checking that the dashboard matches the query that built it. That checks the tool, not the number.