Data Pre-processing and Data Interpretation

Hello students, welcome to Kwickprep. If you cook with rotten vegetables, even the best recipe fails. AI works the same way with bad data. Today we learn how to clean data, normalise and encode it, split it for training and testing, and read the results responsibly.

First, the new term. Data collected from the real world is messy, with blanks, repeats and typing mistakes. Pre-processing means preparing that raw data so a model can learn from it. Data scientists say, garbage in, garbage out, because a model is only as good as its data. In real projects, this cleaning often takes more time than building the model.

Look at this small class register and spot the problems. Riya's row looks fine. Aman's marks are blank, which is a missing value. Neha's age is one hundred fifty, which must be a typing error. And Riya's row appears a second time, which is a duplicate.

Each problem has a standard fix. For a missing value, we either remove the row or fill the blank, for example with the class average. For a duplicate, we delete the extra copy, because counting Riya twice would bias the model. For an error, we correct it if we know the true value, or remove it if we do not.

Here is a small cleaning step in Python. In the list marks, None means the value is missing. The second line keeps only the marks that are not missing. Then we find their average, which is eighty one point two five. We could now fill Aman's blank with this average.

Next comes normalising. A feature is one column of data, like age or income. Features can have very different ranges. A family income may be in lakhs, while age is only in tens. Many models wrongly treat the bigger numbers as more important. Normalising rescales every feature to the same range, usually zero to one.

The most common method is min max normalisation. We subtract the smallest value, called lo, and divide by the range, which is hi minus lo. Here lo is forty and hi is one hundred. Forty becomes zero, seventy becomes point five, and one hundred becomes one. Pause and predict. What would fifty five become?

Fifty five becomes point two five, one quarter of the way up. Models understand numbers, not words, so text categories must be encoded, which means turned into numbers. Delhi can get label code zero, or a one hot pattern of one, zero, zero. Mumbai gets label one, or zero, one, zero. Chennai gets label two, or zero, zero, one. One hot encoding is safer for names, because Chennai is not really bigger than Delhi.

Now, how do we know if a model has really learned? We split the data into two parts. The training set is the part the model learns from. The testing set is kept hidden and used only to check the model on new data. A common split is eighty percent for training and twenty percent for testing. Always shuffle the rows first, and never test on data the model has already seen.

Here we have ten rows of data, numbered one to ten. Eighty percent of ten is eight, so cut is eight. The first eight rows go to train, and the last two go to test. In a real project, we shuffle the rows before cutting, so the test set is a fair sample.

Think of the testing set like a board exam paper. A teacher checks real learning by asking new questions. If the exam repeats the practice questions, students can pass by memory alone. When a model scores very high on training data but poorly on new data, we call it overfitting.

After modelling, we must interpret the results. Data interpretation means finding what the numbers really tell us. We read charts, averages and accuracy with care. We compare the model with a simple baseline, like always guessing the most common answer. And we check how many rows were used, because a result from ten students proves very little.

Responsible interpretation needs four habits. First, two things rising together does not mean one causes the other. Ice cream sales and heat stroke cases both rise in May, but ice cream does not cause heat stroke. Second, check for bias, for example data only from city schools. Third, report the limits of your result honestly. Fourth, protect the privacy of every person in the data.

Let us put the whole journey together. We start with raw data. First we clean it, fixing missing values, duplicates and errors. Next we normalise the numbers and encode the text. Then we split the data into training and testing sets. We train the model and evaluate it on the test set. Finally, we interpret the results responsibly.

Let us revise what we learned today. Cleaning fixes missing values, duplicates and errors. Normalising puts numbers on the same scale, and encoding turns text into numbers. We split data, often eighty percent for training and twenty for testing. We never test on data the model has already seen. And we interpret results honestly, checking bias, limits and privacy.