Exploratory
Data Analysis
The scattered points always know something before the model does. EDA is the discipline of looking - carefully, visually, skeptically - until the data tells you what it actually is.
What is EDA?
Exploratory Data Analysis (EDA) is the process of investigating a dataset with summary statistics and visualizations to understand its structure, quality, and relationships — before you fit a model or run a formal statistical test. The term comes from statistician John Tukey, who drew a line between exploratory analysis (letting data suggest hypotheses) and confirmatory analysis (testing a hypothesis you already had).
IMP — EDA is not a step you skip to save time — it is where most real modeling mistakes are prevented, cheaply, before they become expensive.
Where EDA sits in the data science pipeline
Why use EDA, and why does it matter?
Catches data-quality issues early
Missing values, duplicates, wrong types, impossible values — cheaper to fix now than after a model is trained on them.
Reveals the true shape of the data
Skew, multimodality, and outliers change which statistics and models are even valid to use.
Prevents misleading summaries
A mean and a correlation can look identical across very different datasets — only a plot exposes the difference.
Builds intuition before you model
You choose better features, better models, and better evaluation metrics when you've actually looked at the data.
In short: without EDA, "garbage in" quietly becomes "garbage out" — and you find out only after weeks of modeling work.
Theory & statistical foundations
The five-number summary
Minimum, first quartile (Q1), median, third quartile (Q3), and maximum. It's the numeric backbone of a box plot and the fastest way to describe a distribution's center and spread without assuming it's symmetric.
Shape: skewness & kurtosis
Skewness measures asymmetry (a long right tail = positive skew). Kurtosis measures tail weight — how likely extreme values are compared to a normal distribution.
Simpson's Paradox
A trend present in several groups can reverse when those groups are combined. Always check whether an aggregate pattern survives a group-by-group look.
Four datasets, each with nearly identical mean, variance, correlation, and regression line — yet each looks completely different when plotted: one is linear, one is curved, one has a single outlier driving the whole trend, and one is a vertical line with an outlier. It is the single most-cited proof that summary statistics can lie, and plots don't.
IMP — Correlation is not causation — EDA finds relationships, never proves what causes what.
Types of EDA
EDA splits along two independent axes: how many variables you look at together, and whether you use numbers or pictures.
Univariate — Non-Graphical
Summarizing a single variable with numbers: mean, median, mode, variance, standard deviation, range, and the five-number summary.
Univariate — Graphical
Visualizing one variable at a time: histograms, box plots, density (KDE) plots, bar charts, stem-and-leaf plots.
Multivariate — Non-Graphical
Cross-tabulations, correlation and covariance matrices, grouped summary statistics (groupby/pivot tables).
Multivariate — Graphical
Visualizing relationships between two or more variables: scatter plots, pair plots, heatmaps, violin plots, 3-D plots.
The EDA roadmap
A practical, repeatable order of operations for exploring any new dataset, from first glance to documented insight.
Understand the data
Read the data dictionary, check shape, dtypes, and the first/last rows. Ask: what does each column represent?
df.shape
df.dtypes
df.head()
df.tail()Handle missing & duplicate values
Quantify nulls, decide whether to impute, drop, or flag them. Remove exact duplicates.
df.isnull().sum()
df.duplicated().sum()
df.drop_duplicates(inplace=True)Univariate analysis
Look at each variable alone. Check distribution shape, spread, and central tendency.
df['age'].describe()
df['age'].hist(bins=30)Bivariate & multivariate analysis
Study relationships between variables — numeric-numeric, numeric-categorical, categorical-categorical.
sns.scatterplot(x='income', y='spend', data=df)
df.groupby('segment')['spend'].mean()Outlier & anomaly detection
Flag values far from the bulk of the data using IQR or z-score, then decide to cap, remove, or investigate them.
q1, q3 = df['x'].quantile([.25, .75])
iqr = q3 - q1
outliers = df[(df['x']<q1-1.5*iqr)|(df['x']>q3+1.5*iqr)]Correlation & feature relationships
Measure linear and rank relationships between numeric features to spot redundancy or strong predictors.
corr = df.corr(numeric_only=True)
sns.heatmap(corr, annot=True, cmap='viridis')Form hypotheses & document insights
Write down what surprised you, what needs a statistical test, and what should feed into feature engineering.
# insight: churn is 3x higher for month-to-month
# contracts — worth testing formally (chi-square)Code snippets
Copy-ready Python for every stage of EDA — pandas, seaborn, matplotlib, and scipy.
import pandas as pd
df = pd.read_csv("data.csv")
df.shape # (rows, cols)
df.info() # dtypes + non-null counts
df.describe(include="all") # summary stats
df.head()# Count and visualize missingness
missing = df.isnull().sum().sort_values(ascending=False)
missing_pct = (missing / len(df) * 100).round(2)
import missingno as msno
msno.matrix(df)
# Simple imputation
df["age"] = df["age"].fillna(df["age"].median())
df["city"] = df["city"].fillna("Unknown")import matplotlib.pyplot as plt
import seaborn as sns
fig, axes = plt.subplots(1, 2, figsize=(10, 4))
sns.histplot(df["price"], kde=True, ax=axes[0])
sns.boxplot(x=df["price"], ax=axes[1])
plt.tight_layout()
plt.show()df["category"].value_counts()
df["category"].value_counts(normalize=True) * 100
sns.countplot(y="category", data=df,
order=df["category"].value_counts().index)sns.scatterplot(x="sqft", y="price", hue="city", data=df)
sns.regplot(x="sqft", y="price", data=df, scatter_kws={"alpha": 0.4})
df[["sqft", "price"]].corr()sns.boxplot(x="category", y="price", data=df)
sns.violinplot(x="category", y="price", data=df)
df.groupby("category")["price"].agg(["mean", "median", "std"])sns.pairplot(df, hue="category", corner=True)
corr = df.corr(numeric_only=True)
sns.heatmap(corr, annot=True, fmt=".2f", cmap="coolwarm", center=0)# IQR method
q1, q3 = df["price"].quantile([0.25, 0.75])
iqr = q3 - q1
lower, upper = q1 - 1.5 * iqr, q3 + 1.5 * iqr
iqr_outliers = df[(df["price"] < lower) | (df["price"] > upper)]
# Z-score method
from scipy import stats
z = stats.zscore(df["price"])
z_outliers = df[abs(z) > 3]# pip install ydata-profiling
from ydata_profiling import ProfileReport
profile = ProfileReport(df, title="EDA Report", explorative=True)
profile.to_file("eda_report.html")
# Alternative: pip install sweetviz
import sweetviz as sv
sv.analyze(df).show_html("sweetviz_report.html")Diagrams & sketches
Distribution shapes
Box plot anatomy
Correlation heatmap, sketched
Cheat sheet
| Function | What it does | Library |
|---|---|---|
| df.shape | Rows & columns count | pandas |
| df.info() | Dtypes, non-null counts, memory | pandas |
| df.describe() | Count, mean, std, min, quartiles, max | pandas |
| df.isnull().sum() | Missing values per column | pandas |
| df.duplicated() | Flags duplicate rows | pandas |
| df.corr() | Pairwise correlation matrix | pandas |
| df.value_counts() | Frequency of unique values | pandas |
| df.groupby() | Aggregate by category | pandas |
| sns.histplot() | Distribution of one numeric variable | seaborn |
| sns.boxplot() | Spread, median, outliers | seaborn |
| sns.violinplot() | Distribution shape + density by group | seaborn |
| sns.scatterplot() | Relationship between 2 numeric vars | seaborn |
| sns.pairplot() | All pairwise relationships at once | seaborn |
| sns.heatmap() | Visualize a correlation matrix | seaborn |
| stats.zscore() | Standardize values to detect outliers | scipy |
| np.percentile() | Compute the Nth percentile | numpy |
| ProfileReport() | Full automated EDA report | ydata-profiling |
| msno.matrix() | Visualize missing-value pattern | missingno |
Use cases
Finance
Spotting fraud patterns, understanding risk distributions, stress-testing portfolios before modeling.
Healthcare
Studying patient vitals and outcomes, catching data-entry errors before they bias a clinical model.
E-commerce
Segmenting customers, understanding cart-abandonment patterns, sanity-checking A/B test data.
Manufacturing
Monitoring sensor readings for drift, finding the root cause of defect spikes on a line.
Sports Analytics
Comparing player distributions, finding undervalued performance metrics before building models.
Marketing
Understanding campaign response distributions and which channels actually correlate with conversions.
Features of good EDA
- Visual-first — a picture reveals what a summary statistic can hide.
- Iterative — every plot raises a new question worth checking.
- Assumption-light — you let the data speak before you model it.
- Hypothesis-generating, not hypothesis-confirming.
- Outlier- and distribution-aware, not just mean/variance-aware.
- Tool-agnostic — the mindset matters more than the library.
The future of EDA
AutoEDA tools
ydata-profiling, Sweetviz, D-Tale and AutoViz generate a full report in one line — great for a first pass.
LLM-assisted exploration
Natural-language queries over a dataframe ("show me the distribution of churn by contract type") are becoming standard in notebooks and spreadsheet copilots.
Real-time / streaming EDA
Dashboards that profile data as it arrives, catching schema drift and anomalies before they hit a model.
No-code EDA platforms
Drag-and-drop profiling for analysts who don't write code, lowering the barrier to a data-driven first look.
Tools will keep automating the mechanics of EDA — but the underlying skill, asking good questions of a dataset, stays a human one.
Why every data scientist starts here
Every dataset arrives with a story it hasn't told you yet. A column named "income" might be capped at a survey limit. A "date" field might silently mix two formats. A "churned" flag might be defined differently in two systems that were merged last quarter. None of this shows up in a model's loss curve until it's too late — it shows up in a histogram, a scatter plot, a value_counts() call, minutes into looking.
That is the quiet argument for EDA: it is cheap insurance against expensive mistakes. Tukey's insight was that data analysis has two very different moods. In the confirmatory mood, you already have a hypothesis and you're testing it rigorously. In the exploratory mood, you don't yet know what you're looking for — you're reading the data the way a detective reads a room, open to being surprised. Most real projects need both, in that order.
Anscombe's Quartet is the sharpest illustration of why the exploratory mood can't be skipped: four datasets that are numerically indistinguishable — same mean, same variance, same correlation, same regression line — turn out to be a straight line, a curve, a line with one wild outlier, and a vertical stack of points with a single stray value, respectively. If you had only read the statistics table, you would have believed they were the same dataset. Only the plot tells the truth.
In practice, good EDA is less about any one chart and more about a posture: assume nothing, check everything, and let a plot answer a question before you write a line of modeling code. It's the difference between a model that's technically correct on your validation set and one that actually understands the world it was trained on.
Download the complete EDA notes
EDA-Complete-Notes.md
Definition, theory, roadmap, code snippets, cheat sheet, use cases, and the future of EDA — all in one markdown file you can keep, print, or drop into your own notes.