Tokenizer

A Tokenizer is used to extract tokens from a source stream. Tokens are of the form normally accepted by C-like languages.

XXX This class really needs to become generic, with its C personality moved to CTokenizer

Methods

__init__(self, src)

   def __init__(self, src):

Constructor. Creates a new Tokenizer given a source file and (optionally) a set of flags.

__takeIt(self, tokType, toksrc)

   def __takeIt(self, tokType, toksrc):

Takes the next token off of the front of the buffer and returns it. If the token type continues onto the next line, continue it.

breakOff(self, toksrc)

   def breakOff(self, toksrc):

"breaks off" the regular expression match specified by toksrc from the buffer, and returns it.

fillBuffer(self)

   def fillBuffer(self):

Make sure that the buffer has data in it.

nextPreparsed(self)

   def nextPreparsed(self):

Return the next token in the preparsed list (the list of tokens that has already been parsed and were put back).

nextToken(self)

   def nextToken(self):

Returns the next token, either from the preparsed cache, or from the stream.

parseNextToken(self)

   def parseNextToken(self):

Parses the next token directly off of the stream. Clients should generally avoid using this. Use nextToken() instead, since that will use the preparsed queue if tokens have been put back.

putBack(self, tok)

   def putBack(self, tok):

Puts the given token back on the list.