Describing data: centre, spread and shape

Before any modelling or testing, look at your data and describe it. Most bad analysis starts with skipping this step.

Centre

• The mean is the sum divided by the count. It uses every value, which also makes it sensitive to extreme values.
• The median is the middle value when sorted. It is robust to outliers.

For skewed data, such as incomes, order values or response times, the mean and median can be very different. If ten customers spend 20 and one spends 2,000, the mean is about 200 while the median is 20. Which one is "typical"? The median describes the typical customer, the mean is what you need for total revenue. Report the one that answers your question, and often both.

Spread

Averages alone hide a lot. Two delivery services can both average 30 minutes, while one is always between 28 and 32 and the other ranges from 5 to 90.

• Standard deviation measures typical distance from the mean, in the original units.
• Interquartile range (IQR) is the distance between the 25th and 75th percentiles, the middle half of the data. Like the median, it is robust to outliers.
• Percentiles describe the distribution directly. For response times, the 95th or 99th percentile often matters more than the mean.

orders["value"].describe(percentiles=[0.1, 0.5, 0.9, 0.99])

Shape

Always plot the distribution. A histogram or a box plot shows skew, multiple peaks and outliers that summary numbers hide:

orders["value"].plot.hist(bins=50)

Two peaks can mean two different groups mixed together, like weekday and weekend customers. Summarising them with one mean would describe neither group.

Outliers

An outlier is a value far from the rest. Before removing one, ask what it is:

• A data error, like a price of 99999 entered by mistake? Fix or remove it, and document that.
• A real but rare event, like a very large order? Keep it. Removing real extremes makes your data look calmer than reality.

Rules such as "more than 1.5 IQR beyond the quartiles" help you find candidates. They do not tell you to delete them.

Anscombe's lesson

Anscombe's quartet is four small datasets with nearly identical means, variances and correlations but completely different shapes when plotted. It is a classic reminder that summary statistics without a chart can mislead. Load it from seaborn's sample data and plot it yourself.