In October 2018, a Google research team published a paper with a title that felt almost deliberately modest: “BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding.” Within weeks, it had shattered eleven state-of-the-art benchmarks in natural language processing. Within a year, it was running inside Google Search, quietly reshaping how a billion people found information. The paper itself is now one of the most cited in the history of computer science.
What made BERT extraordinary wasn’t raw scale. The base model had 110 million parameters — laughably small by today’s standards. What made it extraordinary was a single, elegant idea: read the whole sentence at once.
Every serious language model before BERT processed text in one direction. You read left to right, or right to left, and you predicted the next token based on what came before. This felt natural — it mirrors the way we speak. But it’s a terrible way to understand language. The word “bank” means something completely different in “she sat by the river bank” versus “she deposited money at the bank,” and you often don’t know which until you’ve seen words that come after it. Sequential models were perpetually flying half-blind.
BERT’s solution was bidirectional attention through a training task called Masked Language Modeling. During pretraining, the model would randomly mask out roughly 15% of tokens in a sentence and then try to predict them — using context from both sides simultaneously. To do this well, the model had no choice but to build rich, contextual representations of every word in relation to every other word in the sentence. It couldn’t cheat by reading ahead sequentially, because ahead and behind were the same thing.
The second pretraining task, Next Sentence Prediction, pushed BERT to reason about relationships between pairs of sentences, not just individual token sequences. This gave it something earlier models badly lacked: a sense of discourse, of how ideas connect across sentence boundaries.
The downstream impact was immediate and almost disorienting. Fine-tuned on a modest labeled dataset, BERT reached human-level performance on the Stanford Question Answering Dataset. It crushed benchmarks on textual entailment, sentiment analysis, named entity recognition, and semantic similarity — all with the same pretrained weights, just a small task-specific layer added on top. The idea that one model, trained once on raw text, could then be lightly adapted to almost any language task was genuinely new. Transfer learning had worked in computer vision since AlexNet, but nobody had made it work this cleanly in NLP.
What BERT really proved was that language understanding wasn’t a collection of separate engineering problems requiring separate bespoke solutions. It was one problem: build a deep enough representation of meaning, and the specific tasks become easy. The field had spent decades building elaborate pipelines — part-of-speech taggers feeding into parsers feeding into semantic role labelers — and BERT more or less made all of it optional.
The Transformer architecture BERT built on had been introduced only a year earlier in the “Attention Is All You Need” paper. BERT showed that architecture wasn’t just useful for translation. It was a general-purpose meaning machine, if you trained it right.
Everything that followed — GPT-3, T5, RoBERTa, the entire lineage of large language models that defines the current moment — traces directly back to that October 2018 paper. The pretraining-then-finetuning paradigm BERT crystallized is still the dominant framework. Bidirectional attention remains a core design choice in models working at scales BERT’s authors never imagined.
Six years later, the models are millions of times more capable. But they’re still, in a fundamental sense, asking the same question BERT first learned to answer: what does this word mean, given everything around it? We’re still finding out how deep that question goes.