NLP & Language · beginner · concept 110 of 311 · Updated Nov 4, 2026

🗣️Tokenization

Breaking text into smaller units (tokens) for processing. Modern tokenizers use subword methods (BPE, SentencePiece). Splits vary: 'hamburger' alone is ham, bur, ger in GPT-2 but h, amburger in Qwen2.5. Token count determines cost and context window usage. Classical pipelines normalized words first: stemming crudely chops endings, lemmatization maps to dictionary forms; subword tokenizers made most of that unnecessary.