NLP & Language · beginner · concept 115 of 311 · Updated Nov 4, 2026

🧮Token Density and Information Density

Token density is how much meaning each token carries. Models read tokens, not words, and the same text costs more tokens in some languages: Petrov et al. found tokenization lengths differing by up to 15 times across languages, so some users pay more and fit less into a context window. Dense writing costs less and retrieves better. LLMLingua compresses prompts by dropping low-information tokens; tokenizer playgrounds show how text splits.