data preparation is a critical step in the machine learning process, and can have a significant impact on the accuracy and effectiveness of the final model. it requires careful attention to detail and a thorough understanding of the data and the problem at hand.

let's discuss how data should be prepared in order to fit right with the model for better accuracy and outcome.

what is data preparation?

data preparation is the process of dealing with raw data i.e, cleaning, organizing and transforming it to align with the machine learning algorithms. data preparation is a continuous process, and has a huge impact on the performance of machine learning model. clean and structured data would result in better outcomes.

importance of data preparation

in machine learning, the model learns from the data that is fed. so, the algorithm can learn efficiently only if the data is organized and perfect. the quality of the data you use for your model can have a significant impact on the performance of the model.

few aspects that define the importance of data preparation in machine learning are −

improves model accuracy − machine learning algorithms reply completely on data. when you provide clean and structured data to models, the outcomes are accurate.
facilitates feature engineering − data preparation often includes the process of selecting or creating new features to train the model. hence, data preparation would make feature engineering easy.
data quality − collected data most often would contain inconsistencies, errors and irrelevant information. hence when tasks like data cleaning, transformation are applied, the data is formatted and neat. this can be used for gaining insights and patterns.
enables rate of prediction − prepared data makes it easier to analyze results and would yield accurate outcomes.

data preparation process steps

data preparation process involves a sequence of steps that is required to make data suitable for analysis and modeling. the goal of data preparation is to make sure that the data is accurate, complete, and relevant for the analysis.

the following are some of the key steps involved in data preparation −

data collection
data cleaning
data transformation
data reduction
data splitting

the process shown is not always sequential. you might, for example, split your data before you transform it. you might need to collect more data.

let's understand each of the above steps in detail −

data collection

data collection is the first step in the process of machine learning, where data from different sources is gathered to make decisions, answer research questions and statistical planning. different sources such as databases, text files, pictures, sound files, or web scraping may be used for data collection. once the data is selected, the data has to be preprocessed in order to gain insights. this process is carried out to put the data in an appropriate format that would be useful for problem solving. some time data collection follows the data integration step.

data integration involves combining data from multiple sources into a single dataset for analysis. this may involve matching or linking records across different datasets, or merging datasets based on common variables.

after selecting the raw data, the most important task is data preprocessing. in broad sense, data preprocessing will convert the selected data into a form we can work with or can feed to ml algorithms. we always need to preprocess our data so that it can be as per the expectation of machine learning algorithm. the data preprocessing includes data cleaning, transformation and reduction. let's discuss each of these three in detail.

data cleaning

data cleaning is the process of identifying and correcting errors, missing values, duplicate values and outliers, etc. in the data. this step is crucial in the process of machine learning as it ensures that the data is accurate, relevant and error free.

common techniques used for data cleaning include imputation, outlier detection and removal, etc. the following is a sequence of steps for data cleaning −

1. handling duplicate values

duplicates in the dataset means that there is repeated data, which might occur due to data entry errors or issues while collecting data. the technique used to remove duplicates is first they are identified and then deleted using drop_duplicates function in pandas.

2. fixing syntax errors

in this step, structural errors like inconsistencies in data format or naming conventions should be addressed. standardizing formats and fixing errors would ensure data consistence and accurate analysis.

3. dealing outliers

outliers are values that are unusual and differ greatly with the data. the techniques used to detect outliers include statistical methods like z-score or iqr method and machine learning methods like clustering and svm's.

4. handling missing values

missing values are the values or data that is not stored for some values in the dataset. there are several ways to handle missing data like:

imputation − in this process the missing values are substituted with different value, which can be a central tendency measure like mean, median or mode for numeric values and most frequency category for categorical data. some other methods in imputation include regression imputation and multiple imputation.
deletion − in this process the entire instances with missing values are removed. well, this is not a reliable method since there is loss of data.

5. validating the data

data validation is another stage that makes sure that the data aligns perfectly with the requirements so that the predicted outcome is accurate. some common data validation procedures it the correctness of data before storing them in databases are:

data type check
code check
format check
range check

data transformation

data transformation is the process of converting the data from its original format into a format that is suitable for analysis and modeling. this could include defining the structure, aligning the data, extracting data from source, and then storing it in an appropriate form.

there are many techniques available to transorm data into a sutable format. some commonly used data transformation techniques are as follows −

scaling
normalization − l1 & l2 normalizations
standardization
binarization
encoding
log transformation

lets discuss each of the above data transformation techniques in detail −

1. scaling

in most cases, the data we collected consists of attributes with varying scale, but we cannot provide such data to ml algorithm hence it requires rescaling. data scaling makes sure that attributes are at the same scale i.e, usually range of 0 to 1.

we can rescale the data with the help of minmaxscaler class of scikit-learn python library.

example

in this example we will rescale the data of pima indians diabetes dataset which we used earlier. first, the csv data will be loaded (as done in the previous chapters) and then with the help of minmaxscaler class, it will be rescaled in the range of 0 and 1.

the first few lines of the following script are same as we have written in previous chapters while loading csv data.

from pandas import read_csv
from numpy import set_printoptions
from sklearn import preprocessing
path = r'c:\pima-indians-diabetes.csv'
names = ['preg', 'plas', 'pres', 'skin', 'test', 'mass', 'pedi', 'age', 'class']
dataframe = read_csv(path, names=names)
array = dataframe.values

now, we can use minmaxscaler class to rescale the data in the range of 0 and 1.

data_scaler = preprocessing.minmaxscaler(feature_range=(0,1))
data_rescaled = data_scaler.fit_transform(array)

we can also summarize the data for output as per our choice. here, we are setting the precision to 1 and showing the first 10 rows in the output.

set_printoptions(precision=1)
print ("\nscaled data:\n", data_rescaled[0:10])

output

scaled data:
[
   [0.4 0.7 0.6 0.4 0.  0.5 0.2 0.5 1. ]
   [0.1 0.4 0.5 0.3 0.  0.4 0.1 0.2 0. ]
   [0.5 0.9 0.5 0.  0.  0.3 0.3 0.2 1. ]
   [0.1 0.4 0.5 0.2 0.1 0.4 0.  0.  0. ]
   [0.  0.7 0.3 0.4 0.2 0.6 0.9 0.2 1. ]
   [0.3 0.6 0.6 0.  0.  0.4 0.1 0.2 0. ]
   [0.2 0.4 0.4 0.3 0.1 0.5 0.1 0.1 1. ]
   [0.6 0.6 0.  0.  0.  0.5 0.  0.1 0. ]
   [0.1 1.  0.6 0.5 0.6 0.5 0.  0.5 1. ]
   [0.5 0.6 0.8 0.  0.  0.  0.1 0.6 1. ]
]

from the above output, all the data got rescaled into the range of 0 and 1.

2. normalization

normalization is used to rescale the data with a distribution value between 0 and 1. for every feature, the minimum value is set to 0 and the maximum value is set to 1.

this is used to rescale each row of data to have a length of 1. it is mainly useful in sparse dataset where we have lots of zeros. we can rescale the data with the help of normalizer class of scikit-learn python library.

in machine learning, there are two types of normalization preprocessing techniques as follows −

l1 normalization

it may be defined as the normalization technique that modifies the dataset values in a way that in each row the sum of the absolute values will always be up to 1. it is also called least absolute deviations.

example

in this example, we use l1 normalize technique to normalize the data of pima indians diabetes dataset which we used earlier. first, the csv data will be loaded and then with the help of normalizer class it will be normalized.

the first few lines of following script are same as we have written in previous chapters while loading csv data.

from pandas import read_csv
from numpy import set_printoptions
from sklearn.preprocessing import normalizer
path = r'c:\pima-indians-diabetes.csv'
names = ['preg', 'plas', 'pres', 'skin', 'test', 'mass', 'pedi', 'age', 'class']
dataframe = read_csv (path, names=names)
array = dataframe.values

now, we can use normalizer class with l1 to normalize the data.

data_normalizer = normalizer(norm='l1').fit(array)
data_normalized = data_normalizer.transform(array)

we can also summarize the data for output as per our choice. here, we are setting the precision to 2 and showing the first 3 rows in the output.

set_printoptions(precision=2)
print ("\nnormalized data:\n", data_normalized [0:3])

output

normalized data:
[
   [0.02 0.43 0.21 0.1  0. 0.1  0. 0.14 0. ]
   [0.   0.36 0.28 0.12 0. 0.11 0. 0.13 0. ]
   [0.03 0.59 0.21 0.   0. 0.07 0. 0.1  0. ]
]

l2 normalization

it may be defined as the normalization technique that modifies the dataset values in a way that in each row the sum of the squares will always be up to 1. it is also called least squares.

example

in this example, we use l2 normalization technique to normalize the data of pima indians diabetes dataset which we used earlier. first, the csv data will be loaded (as done in previous chapters) and then with the help of normalizer class it will be normalized.

the first few lines of following script are same as we have written in previous chapters while loading csv data.

from pandas import read_csv
from numpy import set_printoptions
from sklearn.preprocessing import normalizer
path = r'c:\pima-indians-diabetes.csv'
names = ['preg', 'plas', 'pres', 'skin', 'test', 'mass', 'pedi', 'age', 'class']
dataframe = read_csv (path, names=names)
array = dataframe.values

now, we can use normalizer class with l1 to normalize the data.

data_normalizer = normalizer(norm='l2').fit(array)
data_normalized = data_normalizer.transform(array)

we can also summarize the data for output as per our choice. here, we are setting the precision to 2 and showing the first 3 rows in the output.

set_printoptions(precision=2)
print ("\nnormalized data:\n", data_normalized [0:3])

output

normalized data:
[
   [0.03 0.83 0.4  0.2  0. 0.19 0. 0.28 0.01]
   [0.01 0.72 0.56 0.24 0. 0.22 0. 0.26 0.  ]
   [0.04 0.92 0.32 0.   0. 0.12 0. 0.16 0.01]
]

3. standardization

standardization is used to transform data attributes to a standard gaussian distribution with a mean of 0 and a standard deviation of 1. this technique is useful in ml algorithms like linear regression, logistic regression that assumes a gaussian distribution in input dataset and produce better results with rescaled data.

we can standardize the data (mean = 0 and sd =1) with the help of standardscaler class of scikit-learn python library.

example

in this example, we will rescale the data of pima indians diabetes dataset which we used earlier. first, the csv data will be loaded and then with the help of standardscaler class it will be converted into gaussian distribution with mean = 0 and sd = 1.

the first few lines of following script are same as we have written in previous chapters while loading csv data.

from sklearn.preprocessing import standardscaler
from pandas import read_csv
from numpy import set_printoptions
path = r'c:\pima-indians-diabetes.csv'
names = ['preg', 'plas', 'pres', 'skin', 'test', 'mass', 'pedi', 'age', 'class']
dataframe = read_csv(path, names=names)
array = dataframe.values

now, we can use standardscaler class to rescale the data.

data_scaler = standardscaler().fit(array)
data_rescaled = data_scaler.transform(array)

we can also summarize the data for output as per our choice. here, we are setting the precision to 2 and showing the first 5 rows in the output.

set_printoptions(precision=2)
print ("\nrescaled data:\n", data_rescaled [0:5])

output

rescaled data:
[
   [ 0.64  0.85  0.15  0.91 -0.69  0.2   0.47  1.43  1.37]
   [-0.84 -1.12 -0.16  0.53 -0.69 -0.68 -0.37 -0.19 -0.73]
   [ 1.23  1.94 -0.26 -1.29 -0.69 -1.1   0.6  -0.11  1.37]
   [-0.84 -1.   -0.16  0.15  0.12 -0.49 -0.92 -1.04 -0.73]
   [-1.14  0.5  -1.5   0.91  0.77  1.41  5.48 -0.02  1.37]
]

4. binarization

as the name suggests, this is the technique with the help of which we can make our data binary. we can use a binary threshold for making our data binary. the values above that threshold value will be converted to 1 and below that threshold will be converted to 0. for example, if we choose threshold value = 0.5, then the dataset value above it will become 1 and below this will become 0. that is why we can call it binarizing the data or thresholding the data. this technique is useful when we have probabilities in our dataset and want to convert them into crisp values.

we can binarize the data with the help of binarizer class of scikit-learn python library.

example

in this example, we will rescale the data of pima indians diabetes dataset which we used earlier. first, the csv data will be loaded and then with the help of binarizer class it will be converted into binary values i.e. 0 and 1 depending upon the threshold value. we are taking 0.5 as threshold value.

the first few lines of following script are same as we have written in previous chapters while loading csv data.

from pandas import read_csv
from sklearn.preprocessing import binarizer
path = r'c:\pima-indians-diabetes.csv'
names = ['preg', 'plas', 'pres', 'skin', 'test', 'mass', 'pedi', 'age', 'class']
dataframe = read_csv(path, names=names)
array = dataframe.values

now, we can use binarize class to convert the data into binary values.

binarizer = binarizer(threshold=0.5).fit(array)
data_binarized = binarizer.transform(array)

here, we are showing the first 5 rows in the output.

print ("\nbinary data:\n", data_binarized [0:5])

output

binary data:
[
   [1. 1. 1. 1. 0. 1. 1. 1. 1.]
   [1. 1. 1. 1. 0. 1. 0. 1. 0.]
   [1. 1. 1. 0. 0. 1. 1. 1. 1.]
   [1. 1. 1. 1. 1. 1. 0. 1. 0.]
   [0. 1. 1. 1. 1. 1. 1. 1. 1.]
]

5. encoding

this technique is used to convert categorical variables into numerical representations. some common encoding techniques include one-hot encoding, label encoding and target encoding.

label encoding

most of the sklearn functions expect that the data with number labels rather than word labels. hence, we need to convert such labels into number labels. this process is called label encoding. we can perform label encoding of data with the help of labelencoder() function of scikit-learn python library.

example

in the following example, python script will perform the label encoding.

first, import the required python libraries as follows −

import numpy as np
from sklearn import preprocessing

now, we need to provide the input labels as follows −

input_labels = ['red','black','red','green','black','yellow','white']

the next line of code will create the label encoder and train it.

encoder = preprocessing.labelencoder()
encoder.fit(input_labels)

the next lines of script will check the performance by encoding the random ordered list −

test_labels = ['green','red','black']
encoded_values = encoder.transform(test_labels)
print("\nlabels =", test_labels)
print("encoded values =", list(encoded_values))
encoded_values = [3,0,4,1]
decoded_list = encoder.inverse_transform(encoded_values)

we can get the list of encoded values with the help of following python script −

print("\nencoded values =", encoded_values)
print("\ndecoded labels =", list(decoded_list))

output

labels = ['green', 'red', 'black']
encoded values = [1, 2, 0]
encoded values = [3, 0, 4, 1]
decoded labels = ['white', 'black', 'yellow', 'green']

6. log transformation

this technique is usually used in handling skewed data. it involves apply natural logarithmic function for all values in the dataset to modify the scale of numeric values.

data reduction

data reduction is a technique to reduce the size of the dataset by selecting a subset of features or observations that are most relevant for the analysis. this can help to reduce noise and improve the accuracy of the model.

this is useful when the dataset is very large or when a dataset contains large amount of irrelevant data.

one of the most common technique used is dimensionality reduction, which reduces the size of the dataset without loosing the important information. other method is the discretization, where continuous values like time and temperature are converted to discrete categories which simplifies the data.

data splitting

data splitting is the last step in the preparation of data for machine learning, where the data is split into different sets -

training − subset which is used by the machine learning model for learning patterns.
validation − subset used to evaluate the performance of machine learning model while training.
testing − subset used to evaluate the performance and efficiency of the trained model.

python example

let's check an example of data preparation using the breast cancer dataset −

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import standardscaler

# load the dataset
data = load_breast_cancer()

# separate the features and target
x = data.data
y = data.target

# split the data into training and testing sets
x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.2, random_state=42)

# normalize the data using standardscaler
scaler = standardscaler()
x_train = scaler.fit_transform(x_train)
x_test = scaler.transform(x_test)

in this example, we first load the breast cancer dataset using load_breast_cancer function from scikit-learn. then we separate the features and target, and split the data into training and testing sets using train_test_split function.

finally, we normalize the data using standardscaler from scikit-learn, which subtracts the mean and scales the data to unit variance. this helps to bring all the features to a similar scale, which is particularly important for models like svm and neural networks.

data preparation and feature engineering

feature engineering involves creating new features from the existing data that may be more informative or useful for the analysis. it can involve combining or transforming existing features, or creating new features based on domain knowledge or insights. both data preparation and feature engineering go hand-in-hand in the overall data preprocessing pipeline.

Save or share this article