Python · medium

Detect and remove outliers using the IQR method in pandas.

Asked in Data Scientist interviews, in the Python round.

Short answer

Compute Q1 and Q3 with quantile in pandas, take IQR as Q3 minus Q1, and keep rows between Q1 minus 1.5 IQR and Q3 plus 1.5 IQR. Report how many rows the fence removed.

Sample data

These tables are loaded into an in-browser engine on this page, so you can run your own answer against them. No account needed.

transactions

txn_idamount
122
235
328
431
526
6900
733
824
938
102
1129
1227

How to answer it

Definition

The interquartile range rule: anything more than 1.5 IQRs below the first quartile or above the third is flagged. It is the rule behind the whiskers on a box plot.

Formula

Q1    = 25th percentile
Q3    = 75th percentile
IQR   = Q3 - Q1
lower = Q1 - 1.5 * IQR
upper = Q3 + 1.5 * IQR

Example

# The practice editor preloads the sample as `transactions`.
df = transactions.copy()

def iqr_bounds(s, k=1.5):
    q1, q3 = s.quantile(0.25), s.quantile(0.75)
    iqr = q3 - q1
    return q1 - k * iqr, q3 + k * iqr

lower, upper = iqr_bounds(df["amount"])
outlier = df["amount"].notna() & ~df["amount"].between(lower, upper)

print(f"bounds: {lower:.2f} to {upper:.2f}")
print(f"flagged: {outlier.sum()} of {len(df)}")
print(f"flagged transaction IDs: {df.loc[outlier, 'txn_id'].tolist()}")

clean = df.loc[~outlier].copy()
clean

Output

bounds: 13.50 to 45.50
flagged: 2 of 12
flagged transaction IDs: [6, 10]

The returned clean table has ten rows. From the sample above, pandas' default linear interpolation gives Q1 = 25.50 and Q3 = 33.50. IQR is 8.00, so 1.5 IQR is 12.00. Amounts 900 and 2 fall outside the inclusive bounds 13.50 to 45.50. An outlier can be unusually low as well as unusually high.

Run it outside the practice editor

Create the same input first, then run the solution above:

import pandas as pd
transactions = pd.DataFrame({
    "txn_id": list(range(1, 13)),
    "amount": [22, 35, 28, 31, 26, 900, 33, 24, 38, 2, 29, 27],
})

Missing values and zero IQR

quantile ignores missing values. The explicit notna() check prevents a missing amount from being counted as an outlier; this solution retains it for a separate missing-data decision. If every amount is missing, no useful bounds can be estimated. If Q1 equals Q3, IQR is zero and any value different from that shared quartile is flagged. Inspect the distribution before applying that rule to sparse counts or mostly constant measurements.

Detect, remove, or cap?

Keep the boolean flag when unusual values are valuable evidence, such as fraud. Remove a row only when you can justify excluding it. To retain rows while limiting extremes, use df['amount'].clip(lower, upper) and preserve the original column for review. For a predictive model, estimate bounds on the training data and reuse them on validation/test data to avoid leakage.

Why IQR and not standard deviations

The mean and standard deviation are themselves dragged by the outliers you are trying to find, so a three-sigma rule on skewed data quietly widens its own threshold. Quartiles are order statistics and barely move, which is what makes the rule robust.

Key points

FAQ

The 30-second answer

Take the twenty-fifth and seventy-fifth percentiles with quantile, subtract them for the IQR, and flag anything below Q1 minus one and a half IQRs or above Q3 plus one and a half. In pandas that is a between mask, so I keep the rows I want rather than mutating values.
I would say why IQR rather than a standard-deviation rule: the mean and standard deviation are themselves pulled by the outliers you are hunting, so the threshold moves with the contamination, whereas quartiles barely shift.
The judgement part is what to do next. On skewed data like revenue the rule flags a long tail that is completely genuine, so I would look at the flagged rows before deleting anything, and often cap or flag rather than drop.

In the interview

Practice this for real