TokenInfo holds information about token types. Each has a name, a regex (regular expression describing how the token is represented) and an id.
def __init__(self, name, regex, create, continued):
Make one. Public variables:
the name of the token type
the regular expression that describes the tokens source form
a function that should expect a token source string
an optional regular expression. If it is present, it indicates that the token may be continued over multiple lines (if regex matches to the end of the current line) and it represents the kind of expression which will terminate the multi line token beginning with regex. All lines between the line that begins the token and the portion of a line which ends the token are considered to be part of the token.