Why the heck don't I just use the Java StringTokenizer class from java.util? Well, I'm in the middle of reading Writing Compilers And Interpreters, and I thought it would be instructive to port the tokenizer described in Chapter 2 (aka, "token1.c") to Java. I also apparently have a masochistic streak. What is the tokenizer supposed to do? As it stands, it should read a designated text file line by line, separate "words" or "tokens" on each line, and decide whether the token is a number, a … [Read more...]