tokenization
nounEtymology From token + -ization.
1
The act or process of tokenizing.
2
Something tokenized.
- This was an unlikely tokenization of the input string.
- Derived terms
- detokenization, retokenization
Entry derived from the Wiktionary, under licence CC BY-SA 4.0 — list of authors.