tokenize
verbUS /ˈtoʊ.kən.aɪz/
Etymology From token + -ize.
1
Computing, Engineering, Mathematics, Natural sciences, Physical sciences, Sciences To reduce to a token or set of tokens by lexical analysis.
2
Computing, Engineering, Mathematics, Natural sciences, Physical sciences, Sciences To substitute sensitive data with meaningless placeholders.
3
To treat as a token minority.
- Derived terms
- detokenize, mistokenize, retokenize, tokenizable, tokenization, tokenizer, untokenized
- Related terms
- tokenism