NLP & Language · beginner · concept 74 of 176

🗣️Tokenization

Breaking text into smaller units (tokens) for processing. Modern tokenizers use subword methods (BPE, SentencePiece), 'unhappiness' → ['un', 'happiness']. Token count determines cost and context window usage. Classical pipelines normalized words first: stemming crudely chops endings, lemmatization maps to dictionary forms; subword tokenizers made most of that unnecessary.