Descrição
token is the smallest individual unit in a program, often referring to a portion of a much larger data piece. For example, if a person’s name is John Thomas Wood, it can be broken into tokens; ‘John’, ‘Thomas’ and ‘Wood’. The programmer can then go on to use only the portion or token they wish to. Tokens are classified into keywords, identifiers, literals, operators, and punctuators.
