Python · medium

You need to process a 50GB CSV on a machine with 16GB of RAM. How?

Asked in Data Engineer interviews, in the Python round.

Short answer

Stream it instead of loading it: read_csv with chunksize, or hand it to an engine that spills to disk and reads only the columns you touch (DuckDB, Polars lazy, Spark). If it is a recurring job, convert once to Parquet — columnar plus compression usually cuts both IO and memory by an order of magnitude. The general move is to make the working set fit in memory, not the file.

How to answer it

Do not load it. Make the working set fit, not the file.

The three moves, from least to most change:

• Stream it. pd.read_csv(path, chunksize=1_000_000, usecols=[...], dtype={...}) and aggregate chunk by chunk. Reading only the columns you touch and giving pandas real dtypes (category for low-cardinality strings, int32 where it fits) often cuts memory by five to ten times on its own.
• Use an engine built for it. DuckDB or Polars in lazy mode scan the CSV with a query plan, push filters and projections down, and spill to disk when needed. SELECT region, SUM(amount) FROM read_csv_auto('big.csv') GROUP BY 1 runs on a laptop.
• Convert once. If this file is read more than once, write it to Parquet: columnar, compressed, typed. The 50GB becomes 5 to 10, and later reads touch only the columns asked for.

import duckdb
duckdb.sql('''
  SELECT region, SUM(amount) AS revenue
  FROM read_csv_auto('big.csv')
  GROUP BY region
''').df()

Spark is the answer when the data is on a cluster already; it is not the answer for one file on one machine.

• What they are checking: whether you know that column pruning and dtypes are usually enough, before reaching for a cluster.
• Common mistake: "use Spark", offered before asking what the job actually needs to compute.

Related questions

• How do you handle missing values in a pandas DataFrame?
• Merge two DataFrames on a key and keep unmatched rows from the left.
• Given a pandas DataFrame, compute the 30-day moving average of a column.
• Explain the difference between .apply(), .map(), and vectorized ops in pandas.
• Detect and remove outliers using the IQR method in pandas.
• Write a function to compute a rolling session count with a 30-min timeout.

Practice this for real

• Check your resume against a Data Engineer job description — free, no account needed.
• Every data interview question, by role and topic
• Scored AI mock interviews — two free, no card