Learn Data Science from the ground up

Data science is the discipline of turning raw, messy data into decisions someone can actually act on. It blends statistics (to reason correctly under uncertainty), programming (to handle data at scale), and domain expertise (to know which question is worth asking in the first place).

• The end product isn't a model or a number — it's a decision, a prediction, or a dashboard someone uses.
• Companies use it for things like fraud detection, demand forecasting, medical diagnosis support, and personalized recommendations.
• Knowing which question to ask of the data usually matters more than knowing every algorithm that exists.

Almost every data science project moves through the same five stages, even if it loops back on itself constantly in practice. Skipping straight to 'build a model' without the earlier steps is the most common way projects go wrong.

• Collect — pull data from databases, APIs, logs, spreadsheets, or sensors.
• Clean — handle missing values, duplicates, and inconsistent formats before trusting anything.
• Explore — look at distributions, outliers, and relationships (EDA) before modeling.
• Model — fit a statistical or machine learning model to answer the question.
• Communicate — turn the result into something a non-technical stakeholder can use.

# A typical project skeleton
raw_data = collect_from_source()
clean_data = handle_missing_and_duplicates(raw_data)
explore(clean_data)              # histograms, summary stats, correlations
model = fit_model(clean_data)
report = summarize_for_stakeholders(model)

Before you can analyze data correctly, you need to know what kind of data you're holding — the type determines which statistics and charts are even valid to use.

• Quantitative (numerical): measurable amounts, like age or revenue — further split into discrete (counts) and continuous (measurements).
• Qualitative (categorical): labels or categories, like country or product type — further split into nominal (no order) and ordinal (has order, like 'low/medium/high').
• Structured data fits neatly into rows and columns (a spreadsheet, a SQL table); unstructured data doesn't (raw text, images, audio).

Real-world data is never clean — sensors fail, forms get skipped, formats drift over time. Deciding how to handle missing or malformed values is one of the highest-leverage steps in the entire workflow, because every downstream number inherits that decision.

• Dropping rows with missing data is simple but can bias your dataset if the missingness isn't random.
• Filling in missing values (imputation) with the mean, median, or a model's prediction is often safer for small amounts of missing data.
• Always check *why* data is missing before choosing a strategy — 'missing' can itself be meaningful information.

import pandas as pd
 
df = pd.read_csv("sales.csv")
 
print(df.isna().sum())              # how much is missing, per column
 
df_dropped = df.dropna()             # simplest option
df_filled = df.fillna(df.mean(numeric_only=True))  # median/mean imputation

EDA is the step where you actually look at your data before doing anything clever with it — summary statistics, distributions, and simple charts catch problems that would otherwise sink a model weeks later.

• .describe() gives a fast first read on every numeric column: mean, spread, min/max.
• .value_counts() is the categorical equivalent — it shows you class imbalance immediately.
• A histogram or boxplot will show outliers and skew far faster than staring at raw numbers.

import pandas as pd
 
df = pd.read_csv("customers.csv")
 
print(df.describe())                 # count, mean, std, min, quartiles, max
print(df["segment"].value_counts())  # how many rows per category

       age       spend
count  500.0     500.0
mean    34.2      82.1
std      9.8      41.7
min     18.0       0.0
max     71.0     398.0
 
segment
consumer      312
corporate     140
home-office    48
Name: count, dtype: int64

Floating point numbers are stored in binary, so decimals like 0.1 can't be represented exactly. This bites people doing financial or scientific calculations who assume decimal arithmetic is exact.

• The error is tiny, but it compounds across large datasets or many operations.
• Never compare floats with == directly — round first, or compare against a small tolerance.
• This isn't a Python bug specifically; it's how IEEE 754 floating point works in almost every language.

>>> 0.1 + 0.2
0.30000000000000004
>>> round(0.1 + 0.2, 2) == 0.3
True

Swapping a Python for-loop for a vectorized NumPy operation can be 50-100x faster, because the vectorized version runs in compiled C instead of the slow Python interpreter, one element at a time.

• Vectorized operations apply to an entire array at once — no explicit loop needed.
• This matters more as datasets grow; a loop that's 'fine' at 1,000 rows can crawl at 10 million.
• Most of pandas is built on NumPy, so this habit pays off there too.

import numpy as np
a = np.arange(1_000_000)
 
# slow: python-level loop
total = 0
for x in a:
    total += x * x
 
# fast: vectorized
total = np.sum(a ** 2)

Sample means from almost any distribution start looking normal as sample size grows — which is why so many statistical tests assume normality even when the underlying data clearly isn't normal.

• This holds even if the original population is uniform, skewed, or bimodal.
• It's the theoretical foundation behind confidence intervals and t-tests.
• Larger sample sizes make the approximation to a normal distribution tighter.

import numpy as np
 
# uniform, NOT normal, population
population = np.random.uniform(0, 1, 100_000)
 
means = [np.mean(np.random.choice(population, 30))
         for _ in range(1000)]
# 'means' is approximately normal even though 'population' isn't

A single outlier can create or destroy a correlation, and a strong correlation coefficient can still describe wildly different-looking relationships. Anscombe's quartet is the classic demonstration: four datasets, nearly identical statistics, completely different shapes when plotted.

• Correlation measures a linear relationship's strength — it says nothing about the shape of that relationship.
• Always plot the data; a summary statistic alone can be dangerously misleading.
• Correlation never implies causation on its own — a third, unmeasured variable can drive both.

import numpy as np
 
x = np.array([10, 8, 13, 9, 11, 14, 6, 4, 12, 7, 5])
y = np.array([8.04, 6.95, 7.58, 8.81, 8.33, 9.96, 7.24, 4.26, 10.84, 4.82, 5.68])
 
print(np.corrcoef(x, y)[0, 1])  # ~0.82
# plot it - the relationship looks nothing like a clean line

NULL is never equal to NULL in SQL logic, but GROUP BY still groups every NULL row together into one bucket. Know this before you trust a 'missing category' count in a report.

• This is a common source of silently wrong dashboards — the NULL group is easy to miss.
• COALESCE(column, 'unknown') is a common way to make that bucket explicit instead of hidden.
• The same rule applies to DISTINCT — SQL treats NULLs as equal to each other there too.

SELECT country, COUNT(*) AS orders
FROM sales
GROUP BY country;
 
-- every row where country IS NULL lands in
-- a single "country = NULL" group

Feature engineering is the practice of creating new input columns that make patterns easier for a model to find — often the single highest-leverage step in a machine learning project, ahead of picking a fancier algorithm.

• Simple transforms — ratios, differences, date parts — often help more than switching models.
• One-hot encoding turns a categorical column into multiple 0/1 columns a model can use directly.
• Always create features using only information that would actually be available at prediction time.

import pandas as pd
 
df["order_date"] = pd.to_datetime(df["order_date"])
df["order_month"] = df["order_date"].dt.month
df["is_weekend"] = df["order_date"].dt.dayofweek >= 5
 
df = pd.get_dummies(df, columns=["segment"])  # one-hot encoding

A model that scores perfectly on training data has often just memorized it rather than learned a generalizable pattern. Always check performance on a held-out test set before celebrating any result.

• Splitting data into train/test sets simulates how the model will perform on data it hasn't seen.
• A big gap between training and test accuracy is the classic sign of overfitting.
• Cross-validation extends this idea by testing on several different splits instead of just one.

from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier
 
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
 
model = DecisionTreeClassifier()
model.fit(X_train, y_train)
 
print(model.score(X_train, y_train))  # often ~1.0
print(model.score(X_test, y_test))    # the number that actually matters

Accuracy alone can be dangerously misleading, especially when classes are imbalanced — a model that always predicts 'no fraud' can be 99% accurate and still useless. Precision and recall tell a fuller story.

• Precision: of everything the model flagged as positive, how much was actually positive?
• Recall: of everything that was actually positive, how much did the model catch?
• There's almost always a trade-off between the two — which one matters more depends on the cost of each type of mistake.

from sklearn.metrics import accuracy_score, precision_score, recall_score
 
y_pred = model.predict(X_test)
 
print("accuracy:", accuracy_score(y_test, y_pred))
print("precision:", precision_score(y_test, y_pred))
print("recall:", recall_score(y_test, y_pred))

A model trained on biased historical data will happily reproduce — and sometimes amplify — that bias, since it's only learning statistical patterns, not right from wrong. Being aware of where a dataset came from is part of the job, not an afterthought.

• If a group is underrepresented in training data, the model will typically perform worse for that group.
• Removing a sensitive attribute (like race or gender) from the data doesn't remove bias if other correlated columns still encode it.
• Fairness usually means picking a definition explicitly (equal accuracy across groups, equal false-positive rates, etc.) — there's no single universal standard.

The best analysis in the world is worthless if nobody understands or trusts it. Good visualization and clear writing are what turn a model into an actual decision — this is often the most underrated skill in the field.

• Pick the chart type for the question, not the other way around — trends want a line chart, comparisons want bars.
• Lead with the takeaway, not the methodology — most stakeholders want the 'so what' first.
• Uncertainty is part of the story too — a confidence interval is more honest than a single number.

The questions people getting started in data science ask most often.