Feature Engineering

Why raw data usually isn't ready for a model#

Most machine learning algorithms expect their input as numbers — specifically, numeric feature vectors of a fixed, known shape. Real-world data rarely arrives that way: it has categories ("New York", "red", "premium tier"), missing entries (a sensor that failed to log for an hour, a survey question a respondent skipped), and numeric columns on wildly different scales (age in single digits, income in the tens of thousands). Feature engineering is the practice of transforming raw data into a form a model can actually learn from effectively — and in most real projects, it has a bigger impact on final model quality than which specific algorithm gets used.

Encoding categorical variables#

A column like color with values "red", "blue", "green" has no inherent numeric meaning — a model can't multiply a weight by the string "red". Two common encodings handle this differently, and the choice matters:

One-hot encoding creates a separate binary column for each category, with exactly one column set to 1 per row:

import pandas as pd

df = pd.DataFrame({"color": ["red", "blue", "green", "red"]})
pd.get_dummies(df["color"])

   blue  green  red
0     0      0    1
1     1      0    0
2     0      1    0
3     0      0    1

This is the right default for nominal categories — ones with no natural order (color, city, product category) — because it doesn't imply any false ranking between categories.

Ordinal encoding maps each category to a single integer reflecting a genuine, meaningful order:

education_order = {"high_school": 0, "bachelors": 1, "masters": 2, "phd": 3}
df["education_encoded"] = df["education"].map(education_order)

Using ordinal encoding on a genuinely nominal variable (assigning red=0, blue=1, green=2) is a common, damaging mistake — it silently tells the model "green is greater than red," a relationship that doesn't exist and that a linear-ish model in particular can end up "learning" as if it were real signal.

Scaling and normalization#

Many algorithms (linear/logistic regression, k-means, anything based on distance between points or gradient descent) are sensitive to features being on very different numeric scales — a column ranging 0-1 and a column ranging 0-500,000 can cause the larger-scale feature to dominate the model's learned weights or distance calculations, regardless of its actual predictive importance.

Standardization (z-score scaling) rescales a feature to have mean 0 and standard deviation 1:

from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)  # fit only on training data
X_test_scaled = scaler.transform(X_test)        # apply the same transform to test data

Min-max normalization rescales a feature to a fixed range, typically 0 to 1:

from sklearn.preprocessing import MinMaxScaler

scaler = MinMaxScaler()
X_train_scaled = scaler.fit_transform(X_train)

Tree-based models (decision trees, random forests, gradient boosting) are a notable exception — they split on threshold comparisons per feature independently, so they're largely unaffected by feature scale and don't require this step at all.

Handling missing data#

Real datasets almost always have gaps, and the right handling depends on how much is missing and why:

• Drop the row — reasonable when missing values are rare and randomly distributed, and the dataset is large enough that losing a small number of rows doesn't meaningfully hurt.
• Impute with a simple statistic — fill missing numeric values with the column's mean or median (median is more robust to outliers), or fill missing categorical values with the most frequent category.
• Impute with a model-based estimate — predict the likely missing value from the other columns, using a simple model trained just for that purpose; more accurate than a flat statistic, at the cost of added complexity.
• Add a "was missing" indicator column — sometimes whether a value was missing is itself predictive (a skipped survey question can correlate with the outcome you're predicting), and encoding that fact as its own feature preserves a signal that a naive imputation would otherwise erase.

from sklearn.impute import SimpleImputer

imputer = SimpleImputer(strategy="median")
X_train_imputed = imputer.fit_transform(X_train)  # fit only on training data, same rule as scaling

Common mistakes#

• Fitting a scaler, encoder, or imputer on the entire dataset before splitting into train/test — this leaks statistics from the test set into preprocessing, the same data leakage problem covered in this track's ML workflow page; always fit on training data only, then apply that same fitted transform to validation/test data.
• Using ordinal encoding on a category with no real order — this silently injects a false numeric relationship between categories that a model can end up treating as real signal.
• Dropping every row with any missing value by default, without checking how much data that discards or whether "missingness" itself might be predictive — on a dataset where missing values are common, this can throw away a large fraction of otherwise-usable data.