@hackage tokenify0.1.0.0

A regex based LR tokenizer

A LR tokenizier used to split text into tokens