tokenizer
nounEtymology From tokenize + -er.
1
Computing, Engineering, Mathematics, Natural sciences, Physical sciences, Sciences A system that parses an input stream into its component tokens.
Entry derived from the Wiktionary, under licence CC BY-SA 4.0 — list of authors.