Python Tokens: Keywords, Identifiers, Literals, Operators and Punctuators

Hello students, welcome to Kwickprep. Is True a valid variable name in Python? And what about true with a small t? By the end of this lesson, you will know. Today we learn tokens, the small pieces every Python program is built from.

A token is the smallest meaningful unit in a program. Think of an English sentence, which is made of words and punctuation marks. In the same way, Python first splits each line into tokens, and only then runs it. There are five kinds of tokens: keywords, identifiers, literals, operators and punctuators.

Here is a small program for a bus fare. The base fare is one hundred twenty rupees, and we add thirty rupees for luggage. So it prints one hundred fifty. Now let us look closely at the second line, fare equals base plus thirty, and break it into tokens.

The line fare equals base plus thirty has five tokens. Fare is an identifier, because it is a name we created. The single equals sign is an operator that stores a value in a variable. Base is another identifier. The plus sign is an operator that adds. And thirty is a literal, because it is a fixed value written directly in the code.

The first kind of token is the keyword. A keyword is a word that Python has reserved for its own use. Each keyword has a fixed meaning, like if for decisions and for for loops. Keywords are written in small letters, except True, False and None, which begin with a capital letter. Current Python three versions have thirty five keywords.

You do not need to memorise all of them today. Keywords for decisions include if, elif and else. Loops use for, while, in, break and continue. Functions use def, return, lambda and pass. True, False and None are keyword values. And, or, not and is are used for logic. Import, from and as bring in extra code called modules. Match and case are soft keywords, which are special only in certain places, so they are not counted in the thirty five.

Python can tell you its own keywords. The keyword module holds a list called kw list. The function len counts the items, and it prints thirty five. Then iskeyword checks one word, and for is a keyword, so it prints True. Type print of keyword dot kw list to see the full list.

Why can a keyword not be used as your own name? Python always reads a keyword by its fixed meaning. If you write for equals five, Python expects a loop, not a variable. So it stops at once with a syntax error. The fix is simple: choose another name, like for underscore count.

The second kind of token is the identifier. An identifier is a name you create for a variable, a function or a class. It may use letters, digits and the underscore sign only, so no dash, dot or dollar sign. It cannot start with a digit, and it cannot contain a space. It cannot be a keyword. And it is case sensitive, so marks in small letters and Marks with a capital letter are two different names.

Let us test six names, a very common exam question. Total underscore marks is valid, because it uses letters and an underscore. Underscore count is valid, because a name may start with an underscore. Roll two is valid, because the digit is not first. Two marks is invalid, because it starts with a digit. Total space marks is invalid, because it has a space. And class is invalid, because it is a keyword.

Pause and predict. Remember our opening question. True with a capital letter is a keyword, so it cannot be a name. But true in small letters is not a keyword, so this program runs. Marks with a capital letter and marks in small letters are two different variables. So it prints ten, ninety and forty five. Even so, avoid such confusing names in real programs.

The third kind of token is the literal. A literal is a fixed value written directly in the code. Seventy two is an integer literal, a whole number. Four hundred fifty five point five zero is a float literal, which has a decimal point. Python prints it as four five five point five. The variable num holds a complex number, with a real part and an imaginary part marked by the letter j.

A string literal is text inside quotes, and single or double quotes both work. Surat and Train at five pm are string literals. A boolean literal is either True or False, nothing else. None is a special literal that means no value yet, like a match whose winner is not decided. So the output is Surat, Train at five pm, then True and None.

The fourth kind of token is the operator, a symbol or word that does work on values. Arithmetic operators like plus, minus, star for multiply and slash for divide do maths. Relational operators like double equals compare two values and give True or False. Assignment operators like a single equals sign store values. Logical operators, and, or and not, join conditions. We study every operator in detail in the operators lesson.

The fifth kind of token is the punctuator. Punctuators are symbols that give a program its shape, and the Python documents call them delimiters. Round brackets hold what we pass to a function, and they group parts of a sum. Square brackets make lists and pick items by position. Curly brackets make dictionaries, which store pairs of values. The comma separates items. The colon starts an indented block, in every if statement and loop. The semicolon lets two short statements share one line, though we rarely need it.

Look for the punctuators here. Square brackets make a list of three marks, and commas separate them. Curly brackets make a dictionary, where the colon links the key, name, to the value, Riya. Marks with one in square brackets picks the second item, seventy two, because counting starts at zero. The colon after the if starts a block. So it prints Riya and seventy two.

Some exams ask for answers in pseudocode, which is program steps written in simple fixed words. The same tokens appear here too. Declare, if, then, output and end if act like keywords. Fare is an identifier. One hundred twenty and Costly are literals. The left arrow is the assignment operator.

Let us revise what we learned today. A token is the smallest unit of a program. A keyword is a reserved word, so it can never be your own name. An identifier is a name you create, using letters, digits and underscore, without a digit first. A literal is a fixed value, like a number, a string, True, False or None. An operator does the work, and a punctuator shapes the code.