Architectures · intermediate · concept 59 of 176
BERT (Bidirectional Encoder)
An encoder-only Transformer that reads text bidirectionally (both left-to-right and right-to-left). Trained with masked language modeling. Still a workhorse for cheap, high-volume classification, extraction, and retrieval, where a small encoder beats paying for a generative model.
Key terms
Masked language modelingEncoder-onlyBidirectional contextFine-tuning
Learn these first
Where you meet it in the real world
Google Search, sentiment analysis, document classification, question answering
Videos
▶ Encoder-Only Transformers (like BERT) for RAG, Clearly Explained!!! ↗
StatQuest with Josh Starmer · YouTube
▶ Transformer models and BERT model: Overview ↗
Google Cloud Tech · YouTube
Guides and articles
Courses, papers, and more
This unlocks