Simple tokenizer framework.
At this time, all of the functionality of the CTokenizer is implemented in Tokenizer. Eventually, the CTokenizer personality should be broken out into the correct class.
Synopsis:
from spug.util.tok import CTokenizer
# create a new tokenizer to parse standard input
toker = CTokenizer(sys.stdin)
while 1:
# get the next token
tok = toker.nextToken()
# check for end-of-stream
if tok.isType(Token.end):
break
# print out the token type and the value
print toker.tokTypes[tok.type - 1].name + ': ' + tok.val
Run "python tok.py" to see the action of the above code.