Target Encoding: AI for Dummies #5

At the end of our One Hot encoding article, we left a small elephant in the room: what happens when your categorical column has 50 unique values instead of 3?

If you tried using one-hot encoding on something like postcodes, job titles, or car models, you probably noticed your dataset exploding into hundreds of binary columns. Most of them filled with zeros.

That’s known as feature explosion (or sparse data), and it makes models slow, memory-hungry, and often pretty confused.

In this post, we’ll look at Target Encoding, a simple trick that compresses any categorical column into a single numeric column without losing the underlying pattern.

So what is really our problem?

Let’s bring back our house price dataset. But this time, instead of 3 neighbourhoods (North, South, Downtown), we have 15 different locations across the city:

square meters	house_age	neighbourhood	price
150	10	Riverside	320,000
85	45	Old Town	195,000
220	3	Marina Bay	510,000
110	20	Suburbs North	230,000

If we use One-Hot Encoder here, we end up with 14 new binary columns (remembering `drop=’first’`). If we had 200 neighbourhoods, we’d get 199 columns. For a small dataset, that’s more columns than useful data points.

There has to be a better way than creating a massive grid of 1s and 0s.

Meet Target Encoding

The idea behind target encoding is simple: instead of turning a category into a dummy binary column, we replace the category name with the average target value (in our case, average house price) for that category.

So if houses in `Marina Bay` sell for an average of 500,000€, every house in `Marina Bay` gets the number `500000` in the `neighbourhood` column.

Think of it like a restaurant review score. If you’re picking a restaurant, you don’t need a separate yes/no column for every single restaurant name in town. You just look at its 1-to-5 star rating. Target encoding does the exact same thing for your machine learning model and it replaces raw labels with a number that already carries real meaning.

One column. That’s it 🙂

Let’s see this running

We’ll use `pandas` and `scikit-learn`. Scikit-Learn added built-in support for `TargetEncoder` in version 1.3, so we don’t even need extra libraries.

# requirements.txt

pandas==2.2.2
scikit-learn==1.5.0

Let’s create and active our virtual env and install these depedencies:

python -m venv venv

source venv/bin/activate

# venv\Scripts\activate on Windows

pip install -r requirements.txt

The Code

Here’s our execution scenario: we’ll generate a synthetic dataset with 15 neighbourhoods, split it properly into train and test sets, and compare One-Hot Encoding against Target Encoding using Linear Regression.

# demo.py
import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import TargetEncoder, OneHotEncoder
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.linear_model import Ridge
from sklearn.metrics import mean_absolute_error

# 1. Create a synthetic dataset with high-cardinality neighbourhood column
np.random.seed(42)
n_samples = 300

neighbourhoods = [f"Zone_{i}" for i in range(15)]
# Assign base prices to each neighbourhood (100k to £400 €)
neighbourhood_premiums = {name: 100000 + i * 20000 for i, name in enumerate(neighbourhoods)}

square_meters= np.random.randint(60, 300, size=n_samples)
house_age = np.random.randint(0, 50, size=n_samples)
neighbourhood_col = np.random.choice(neighbourhoods, size=n_samples)

price = (
    square_meters* 1200
    - house_age * 600
    + pd.Series(neighbourhood_col).map(neighbourhood_premiums).values
    + np.random.normal(0, 15000, n_samples)
)

df = pd.DataFrame({
    'square_meters': square_meters,
    'house_age': house_age,
    'neighbourhood': neighbourhood_col,
    'price': price
})

# 2. Train / Test Split
X = df.drop(columns='price')
y = df['price']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

categorical_features = ['neighbourhood']
numeric_features = ['square_meters', 'house_age']

# 3. Pipeline A: One-Hot Encoding
preprocessor_ohe = ColumnTransformer(transformers=[
    ('cat', OneHotEncoder(drop='first', handle_unknown='ignore'), categorical_features),
], remainder='passthrough')

pipeline_ohe = Pipeline(steps=[
    ('preprocessor', preprocessor_ohe),
    ('model', Ridge())
])

pipeline_ohe.fit(X_train, y_train)
y_pred_ohe = pipeline_ohe.predict(X_test)
mae_ohe = mean_absolute_error(y_test, y_pred_ohe)

# Inspect feature count after OHE
X_train_ohe = pipeline_ohe.named_steps['preprocessor'].transform(X_train)
print(f"One-Hot Encoding MAE: {mae_ohe:,.0f} € | Total features produced: {X_train_ohe.shape[1]}")

# 4. Pipeline B: Target Encoding
preprocessor_target = ColumnTransformer(transformers=[
    ('cat', TargetEncoder(smooth='auto', cv=5), categorical_features),
], remainder='passthrough')

pipeline_target = Pipeline(steps=[
    ('preprocessor', preprocessor_target),
    ('model', Ridge())
])

pipeline_target.fit(X_train, y_train)
y_pred_target = pipeline_target.predict(X_test)
mae_target = mean_absolute_error(y_test, y_pred_target)

# Inspect feature count after Target Encoding
X_train_target = pipeline_target.named_steps['preprocessor'].transform(X_train)
print(f"Target Encoding MAE:   {mae_target:,.0f}€ | Total features produced: {X_train_target.shape[1]}")

This will produce something like:

One-Hot Encoding MAE: 20,905 € | Total features produced: 16
Target Encoding MAE: 22,662€ | Total features produced: 3

Notice the feature count. Target encoding kept our dataset down to just 3 features (square_meters, house_age, encoded neighbourhood) while delivering equal or better accuracy.

Code Explanation

1. Synthetic data: We set up 15 distinct neighbourhoods (Zone_0 to Zone_14), each carrying a different baseline price premium.
2. One-Hot Pipeline: Transforms the 15 categories into 14 separate binary columns. Total features passed to the model: 16.
3. TargetEncoder: Replaces the neighbourhood string with its estimated target value. cv=5 performs internal 5-fold cross-validation during training to prevent data leakage (more on that below). Total features passed to the model: 3.
4. Ridge Regression: Fits a linear model on top of both feature sets.

Under the Hood: Two Traps to Watch Out For

(Took me a while to trust this technique, to be honest replacing names with average prices felt almost like cheating at first. But there are two specific pitfalls you need to understand.)

1. Data Leakage (The Golden Rule)

If you calculate the average house price for each neighbourhood using your entire dataset before doing train_test_split, your training set now secretly knows target values from the test set.

Your training error will look amazing. Then your model hits real-world data and collapses.

scikit-learn‘s TargetEncoder avoids this automatically when used inside a Pipeline. During fit(), it computes target statistics using out-of-fold cross-validation (cv=5), ensuring the encoder never sees the target values of the exact rows it’s encoding.

2. Rare Categories & Smoothing

What happens if Zone_12 only appears once in your training data, and by coincidence that single house sold for £900,000 because it had a golden roof?

If your encoder relies solely on that single row, it will assume all houses in Zone_12 are worth 900k €.

To fix this, target encoders use smoothing (blending). If a category has very few samples, the encoder blends its local average with the global average price across the entire dataset. As more houses in Zone_12 are observed, the encoder trusts the local average more and the global average less.

In scikit-learn, setting smooth='auto' handles this automatically.

A few points worth mentioning

• Target Encoding is for High Cardinality: If you only have 3 or 4 categories (like ['North', 'South', 'West']), stick with One-Hot Encoding. It’s simpler and doesn’t carry risk of target leakage.
• Regression vs Classification: Target encoding works for classification problems too. Instead of replacing categories with average house prices, it replaces them with the probability of the positive class (e.g. 0.85 for high risk).
• Tree-based models: XGBoost, LightGBM, and CatBoost have built-in categorical handling that often uses target encoding variants under the hood automatically.

And that should be it. Cheers!