Architectures · intermediate · concept 86 of 311 · Updated Nov 4, 2026
BERT (Bidirectional Encoder)
An encoder-only Transformer that reads text bidirectionally (both left-to-right and right-to-left). Trained with masked language modeling. Still a workhorse for cheap, high-volume classification, extraction, and retrieval, where a small encoder beats paying for a generative model.
Top AI contributors on this
- Yoshua BengioLawZero / Université de Montréal
- Ilya SutskeverSafe Superintelligence Inc. (SSI)
- Noam ShazeerOpenAI
- Alec RadfordIndependent / Thinking Machines Lab (adviser)
- Jürgen SchmidhuberKAUST / IDSIA
- Oriol VinyalsDiscovery Loop
Key terms
Masked language modelingEncoder-onlyBidirectional contextFine-tuning
Learn these first
Where you meet it in the real world
Google Search, sentiment analysis, document classification, question answering
Videos
▶ Encoder-Only Transformers (like BERT) for RAG, Clearly Explained!!! ↗
StatQuest with Josh Starmer · YouTube
▶ Transformer models and BERT model: Overview ↗
Google Cloud Tech · YouTube
Guides and articles
This unlocks