Classification and the k-Nearest Neighbours Algorithm

Hello students, welcome to Kwickprep. You see a new fruit in the market, and you judge it by the fruits it looks most similar to. Can a computer decide the same way? Yes, it can. Today we learn what classification means, how the k nearest neighbours algorithm votes, how to choose k, and one full worked example.

Let us start with a new word. Classification means putting a new item into one group from a fixed list of groups. These groups are called classes, or labels, like spam or not spam. The model first studies examples whose correct label is already known, so this is supervised learning. The answer it gives is always a category, never a number like a price.

Classification is all around us. An email filter reads a mail and labels it spam or not spam. A bank checks each card payment and labels it fraud or safe. A hospital tool studies an X-ray and labels it healthy or needing care. A farmer's app looks at a leaf photo and names the crop disease.

Board exams often ask you to tell classification apart from regression. Will a student pass or fail? The answer is a category, so it is classification. How many marks will the student score? That answer is a number, so it is regression. Which fruit is this? A category again, so classification.

Now meet a simple classification algorithm. Its name is k nearest neighbours, or k-NN for short. It is based on a simple idea: items that are alike sit close together, just as friends sit near each other in class. The letter k is the number of nearest neighbours we ask. It does not build rules in advance; it keeps every example and decides only when a new item arrives.

To find neighbours, we need a way to measure closeness. A feature is a quality we measure, like sweetness or weight. Each item becomes a point on a graph, using its features as coordinates. Distance tells how far apart two points are, using the distance formula you know from maths.

Here are the steps of k nearest neighbours as a flowchart. First, read the features of the new item. Next, find its distance to every labelled example. Then sort these distances and pick the k closest examples. Now count how many of these neighbours belong to each label. Finally, the label with the most votes is the answer, just like a class monitor election.

Choosing k is the most important decision in this algorithm. If k is very small, like one, a single unusual example can give a wrong answer. If k is very large, points that are far away also vote, and the answer blurs. With two classes, an odd k like three or five avoids a tie. The best way is to try several values of k on test data and keep the most accurate one.

Now a full worked example. Each fruit has two features, a sweetness score and a crunch score, out of ten. Fruit A is at five, six and is an orange. Fruit B is at five, eight and is an apple. Fruit C is at eight, five and is an apple. Fruit D is at one, two and is an orange. Fruit E is at nine, eight and is an orange.

A new fruit arrives with sweetness five and crunch five. We find its distance to each fruit, using the square root of the sum of squared differences. For A, the differences are zero and one, so the distance is one. For B, the differences are zero and three, so the distance is three. For C, it is three and zero, so again three. For D, the differences are four and three, so the distance is five. For E, it is also four and three, so five.

Now we vote, and see how k changes the answer. With k equal to one, only A votes, so the answer is orange. With k equal to three, A, B and C vote, and apple wins two votes to one. With k equal to five, everyone votes, and orange wins three to two. Pause and predict before we continue: which answer should we trust?

There is no magic answer, so we reason it out. With k equal to one, we trust a single fruit, which may be unusual. With k equal to five, far away fruits D and E also get a vote. Here k equal to three is a sensible balance, so we classify the new fruit as an apple. In a real project, we would confirm this choice by checking accuracy on test data.

Let us weigh k nearest neighbours fairly. It is easy to explain, but it must keep all the data in memory. It needs no training step, but predicting is slow, since it measures distance to every example. It works with many features, but features must use similar scales, or a big number like weight in grams will drown the rest.

Let us revise what we learned today. Classification puts a new item into one of a fixed set of labels. The k nearest neighbours algorithm finds the k closest labelled examples using distance. The majority label among those neighbours is the answer. For two classes, choose an odd k, and test several values. And remember, as our fruit example showed, a different k can change the answer. Try your own example with three new points on graph paper.