Now that it’s debugged, here’s some source for a lexer, and associated testapp.
- Matches literal tokens only at the moment, not regexps (which I leave as an exercise to the reader).
- Has Ansi and Unicode and Ordinal (numeric versions).
- Also has some nifty arbitrary size set handling code.
- Is not optimised: Some of the list and transition handling is (relatively) slow compared to compiled lexers. Also an exercise for the reader.