TokenInfo

TokenInfo holds information about token types. Each has a name, a regex (regular expression describing how the token is represented) and an id.

Methods

__init__(self, name, regex, create, continued)

   def __init__(self, name, regex, create, continued):

Make one. Public variables:

name

the name of the token type

regex

the regular expression that describes the tokens source form

create

a function that should expect a token source string

continued

an optional regular expression. If it is present, it indicates that the token may be continued over multiple lines (if regex matches to the end of the current line) and it represents the kind of expression which will terminate the multi line token beginning with regex. All lines between the line that begins the token and the portion of a line which ends the token are considered to be part of the token.