Text Processing: Normalisation, Tokenisation, Bag of Words and TF-IDF

Hello students, welcome to Kwickprep. A computer understands only numbers. So how can it read a sentence like, Riya plays cricket? Today we clean text step by step. Then we turn three sentences into numbers with Bag of Words, and calculate TF-IDF by hand, the way exam questions ask.

Let us learn three terms before we begin. A corpus is the whole collection of text we work on, like all the reviews of one app. A document is one piece of text in the corpus, like one single review. Text normalisation means cleaning the text and making it simpler, so the computer has less confusion.

Text normalisation happens in five steps, in this order. Sentence segmentation splits the whole text into separate sentences. Tokenisation splits each sentence into small pieces called tokens. Then we remove stop words, special characters and often numbers. Next, we convert everything to the same case, usually lower case. Finally, stemming or lemmatisation reduces each word to its root, and we get clean tokens.

Let us see the first two steps with a tiny example. Sentence segmentation looks for full stops and similar marks, so I won, Mom smiled, becomes two sentences. Tokenisation breaks a sentence into tokens, and a token is any word, number or symbol. So I won with an exclamation mark gives three tokens, I, won, and the exclamation mark.

Next, we remove what does not help. Stop words are very common words that add little meaning to the text. Examples are a, an, the, and, is, and to. We also remove special characters, and often numbers, unless they matter for the task. Then we convert to a common case, so capital LOVE and small love are counted as the same word.

Here are these steps in a few lines of Python. Lower makes every letter small, and replace removes the exclamation mark, giving t. Split breaks t into word tokens, stored in ws. We keep only the words that are not in our stop list. The output has four clean tokens, riya, aman, love and chess.

The last step reduces a word to its root form, and there are two ways. Stemming simply cuts off endings like i n g, so playing becomes play, which is fine. But stemming turns studies into s t u d i, which is not a real word. And caring becomes car, which changes the meaning completely. Lemmatisation is smarter and always gives a real dictionary word, like study and care, but it is slower.

Now we turn clean text into numbers with Bag of Words. It is called a bag because word order is ignored, and only counts matter. First, we normalise all documents. Second, we create a dictionary, which is the list of every unique word in the corpus. Third, for one document, we count each dictionary word, and this list of counts is its document vector. Fourth, we repeat this for every document, and we get a table of numbers.

Let us build it from three sentences. Document one is, Riya plays cricket. Document two is, Aman plays chess. Document three is, Riya and Aman love chess. After normalising, we drop the stop word and, and use lower case. So the dictionary has six unique words: riya, plays, cricket, aman, chess and love.

This program makes the document vectors. The list v is our dictionary of six words, in the same order as before. For each document, count tells how many times each dictionary word appears. Document one has riya, plays and cricket, so it starts with one, one, one, then three zeros. Pause and predict document two before reading it. It is zero, one, zero, one, one, zero.

Bag of Words treats every word as equally important, but that is not true. Term frequency, or TF, is how many times the word appears in this document. Document frequency, or DF, is the number of documents that contain the word. Inverse document frequency, or IDF, is N divided by DF, where N is the total number of documents. TF-IDF is TF multiplied by the log of N divided by DF, and log here means log to base ten.

Here N is three, because we have three documents. Riya appears in documents one and three, and plays appears in one and two, so both have DF two. Aman and chess also appear in two documents each. Cricket appears only in document one, so its DF is one. Love appears only in document three, so its DF is also one.

Now let us calculate TF-IDF for two words in document one, where each word appears once, so TF is one. For riya, log of three by two is about zero point one seven six. For cricket, log of three by one is about zero point four seven seven. So in document one, cricket gets a higher value than riya. Cricket is rare in the corpus, so it tells us more about document one.

So what does TF-IDF really tell us? If a word appears in every document, N by DF is one, and log of one is zero, so its value is zero. That is why stop words like the and is score very low. Rare words score high, and they often show what a document is really about. Search engines and keyword tools use this idea to find the most important words.

Let us revise what we learned today. Normalisation means sentence segmentation, tokenisation, removing stop words and symbols, and converting to a common case. Stemming just cuts endings, while lemmatisation always gives a real word. Bag of Words makes a dictionary, then a count vector for every document. And TF-IDF is TF times log of N by DF, so rare words get high values. Practise with three sentences of your own.