A Lexer.

Now that it’s debugged, here’s some source for a lexer, and associated testapp.

  • Matches literal tokens only at the moment, not regexps (which I leave as an exercise to the reader).
  • Has Ansi and Unicode and Ordinal (numeric versions).
  • Also has some nifty arbitrary size set handling code.
  • Is not optimised: Some of the list and transition handling is (relatively) slow compared to compiled lexers. Also an exercise for the reader.

Source code.


Leave a Reply

Your email address will not be published. Required fields are marked *