Simple tokenizer framework.
At this time, all of the functionality of the CTokenizer is implemented in Tokenizer. Eventually, the CTokenizer personality should be broken out into the correct class.
Synopsis:
from spug.util.tok import CTokenizer # create a new tokenizer to parse standard input toker = CTokenizer(sys.stdin) while 1: # get the next token tok = toker.nextToken() # check for end-of-stream if tok.isType(Token.end): break # print out the token type and the value print toker.tokTypes[tok.type - 1].name + ': ' + tok.val
Run "python tok.py" to see the action of the above code.