Introduction
Categorical variables, which take on a limited set of discrete values rather than a continuous numerical range, are very common in real-world data sets. Examples include gender, country, profession, industry sector, and many other attributes.
While extremely useful for analysis, categorical variables pose a challenge for many machine learning algorithms, which expect numerical feature inputs. Scikit-learn models are no different – they require all features to be encoded as numbers before model training can begin.
In this comprehensive tutorial, we will explore 3 core strategies and techniques for encoding categorical features in scikit-learn:
- One-hot encoding
- Ordinal encoding
- Binary encoding
We will look at examples of implementing these encodings in scikit-learn using utilities like OneHotEncoder, OrdinalEncoder, and LabelEncoder. We will also discuss best practices like monitoring encoding quality, shuffling data, and avoiding target leakage.
By the end, you will have a solid grasp of recommended approaches for handling categorical variables in your scikit-learn modeling pipelines. Let’s get started!
One-Hot Encoding
One-hot encoding (also known as 1-of-K encoding) is a popular approach for encoding categorical variables with N possible values into N binary columns.
Each categorical value is mapped to a binary vector with a single “1” value representing the presence of that particular category, and “0” values otherwise.
For example, consider a “City” categorical feature with three possible values – “New York”, “Chicago”, and “Los Angeles”. Using one-hot encoding, this would get transformed into three separate binary columns:
City_NewYork: 1 0 0
City_Chicago: 0 1 0
City_LosAngeles: 0 0 1The main advantage of one-hot encoding is that it avoids imposing any ordinal relationship between categories that may not exist. Each category is independently represented in its own column.
In scikit-learn, one-hot encoding can be applied using the OneHotEncoder class:
from sklearn.preprocessing import OneHotEncoder
encoder = OneHotEncoder(sparse=False)
encoded = encoder.fit_transform(data[['City']])By default, the encoded output is a sparse matrix, but we set sparse=False here to get a dense NumPy array.
One potential downside of one-hot encoding is that the number of generated columns grows quickly as categorical variables take on more values. This can make the dataset very wide, causing sparsity and model overfitting issues.
Using a sparse matrix representation via scipy.sparse can help mitigate these problems for datasets with a large number of categories. Feature selection may also be required to reduce dimensionality.
Overall, one-hot encoding provides an effective and conceptually simple method for encoding categorical variables, especially those with relatively few categories.
Ordinal Encoding
Ordinal encoding assigns each unique categorical value to a different integer, based on the implicit ordinal relationship between categories.
For example, consider a categorical “Profession” variable with the following values:
- Lawyer
- Doctor
- Engineer
Using ordinal encoding, these could be mapped to:
Lawyer -> 1
Doctor -> 2
Engineer -> 3So “Lawyer” is encoded as 1, “Doctor” as 2, and so on. This integer encoding imposes an ordering between professions, even though the actual order is arbitrary.
In scikit-learn, ordinal encoding can be performed using the OrdinalEncoder class:
from sklearn.preprocessing import OrdinalEncoder
encoder = OrdinalEncoder()
encoded = encoder.fit_transform(data[['Profession']])A limitation of ordinal encoding is that it assumes an inherent ordering between categories that may not exist or make sense. The arbitrary integer mappings can incorrectly imply that certain categories are “greater than” or “less than” others in a meaningful way.
However, ordinal encoding does preserve some information about categorical variables that one-hot encoding does not. It can also help reduce dimensionality compared to one-hot encoding.
So ordinal encoding can be considered when an inherent ordering exists between categories and compact encodings are preferred.
Binary Encoding
For binary categorical variables where each observation belongs to one of two possible classes, simple binary encoding with 0s and 1s can be used.
For example, consider a column “IsStudent” with possible values “Yes” and “No”. This could be binarily encoded as:
Yes -> 1
No -> 0In scikit-learn, binary encoding is easily applied with the LabelBinarizer class:
from sklearn.preprocessing import LabelBinarizer
encoder = LabelBinarizer()
encoded = encoder.fit_transform(data[['IsStudent']])Compared to one-hot encoding, binary encoding is more compact and does not unnecessarily explode the number of columns for variables with only two classes.
Binary encoding can also be used as part of ordinal encoding – first ordinally encode multi-class categorical variables into integers, then binarily encode the integers.
Overall, binary encoding provides an efficient way to encode categorical variables with binary outcomes into a format digestible for scikit-learn models.
Conclusion
Encoding categorical variables is an essential step in preparing data for scikit-learn modeling. Strategies like one-hot, ordinal and binary encoding each have trade-offs and are suitable in different situations depending on the number of categories and inherent variable orderings.
Beyond encoding, best practices like monitoring encoding quality on hold-out data, shuffling data before encoding to avoid order bias, and avoiding target/information leakage from the future are key to generating clean, unbiased encoded datasets.
With the encoding techniques explored in this guide, you now have an expansive toolkit to handle categorical features in your scikit-learn pipelines.
The next step is to apply these encodings, critically evaluate their impact on model performance for your specific problem, and iterate to find the optimal encoding approach. This thoughtful encoding process will ensure you effectively leverage the value of categorical variables.
