Module spug.util.tok

Simple tokenizer framework.

At this time, all of the functionality of the CTokenizer is implemented in Tokenizer. Eventually, the CTokenizer personality should be broken out into the correct class.

Synopsis:


      from spug.util.tok import CTokenizer
      
      # create a new tokenizer to parse standard input
      toker = CTokenizer(sys.stdin)
      
      while 1:
      
         # get the next token
         tok = toker.nextToken()
         
         # check for end-of-stream
         if tok.isType(Token.end):
            break
         
         # print out the token type and the value
         print toker.tokTypes[tok.type - 1].name + ': ' + tok.val

Run "python tok.py" to see the action of the above code.

Classes

Functions