PAPERS-01-001 · Paper — The Lineage

The paper

This one is the track opener, so it has no single paper behind it — it is the map you read before the journey. Where every other entry in PAPERS-01 teaches one landmark paper in its lineage, this video walks the whole lineage as a single story: six ideas, published between 2013 and 2020, that compounded into the large language models everyone now uses. The refinements that came after — scaling laws, Chinchilla, RLHF, chain-of-thought, RAG, LoRA — are the polish on that foundation. Follow the track root to frontier and each paper hands off to the next; the lineage is the point.

Before this paper

Before any of it, meaning was hard for a machine to represent. A word was just a symbol — an index in a vocabulary — with no built-in sense of what it meant or what sat near it in meaning-space. Models could count words and match strings, but "king" and "queen" were as unrelated to a computer as "king" and "banana." Sequence tasks like translation were handled by brittle, hand-engineered pipelines that did not transfer from one problem to the next. The field needed a way to make meaning, and then sequence, and then attention, something a network could learn rather than something an engineer had to build.

The key idea

The big idea of the whole arc is that each advance answers the exact limitation of the one before it — the lineage is a chain of fixes. Word2Vec (2013) put meaning into geometry: similar words land near each other, and directions carry analogy. Seq2Seq (2014) mapped one whole sequence to another with a pair of networks. Bahdanau attention (2014) fixed Seq2Seq's memory bottleneck by letting the decoder look back. The Transformer (2017) made attention the entire model, dropping recurrence so training could parallelize. BERT and GPT (2018) pretrained on raw text; GPT-3 (2020) showed the pretrained model could learn a task from examples in the prompt alone.

How it works

Read the chain as a sequence of unlocks. Geometry gives you word vectors you can do arithmetic on. Seq2Seq stacks an encoder that reads a sentence into a vector and a decoder that writes the answer out — but a single vector loses detail on long inputs, so attention lets the decoder weight every input position it needs. The Transformer removes recurrence entirely and computes all positions in parallel, which is what made truly large models trainable. Pretraining then splits learning in two: learn language once on the whole internet, then adapt cheaply to any task — until, with enough scale, adaptation happens in the prompt itself.

See it — the road from word vectors to modern LLMs

timeline
    title The road to modern LLMs
    2013 : Word2Vec meaning as geometry
    2014 : Seq2Seq encoder to decoder
         : Bahdanau attention look back
    2017 : Transformer attention is the model
    2018 : BERT and GPT pretrain on text
    2020 : GPT-3 few-shot in the prompt

What they showed

Because this opener is a narrative rather than a single experiment, it reports no numbers of its own — each downstream video carries its paper's actual headline results, cited to the arXiv id. What the arc as a whole showed is qualitative and decisive: analogy arithmetic works on learned word vectors; attention beats a fixed-length bottleneck on long sequences; a recurrence-free model trains faster and scales further; and a single pretrained network, prompted with a handful of examples, can perform a task it was never fine-tuned for. Where a later paper in this very lineage revised an earlier claim — Chinchilla correcting the Kaplan scaling laws on how to spend compute — the track names the revision, because that correction is itself part of the story.

What it changed

Together these ideas turned natural-language processing from a discipline of bespoke pipelines into one of general, pretrained models you adapt — the shift that produced the assistants, copilots, and search tools in daily use. None of the ideas erased the last: each sits on the one before, and the modern LLM is all of them at once. The refinements — predictable scaling, compute-optimal training, alignment from human feedback, step-by-step reasoning, retrieval grounding, and low-rank adaptation — tune the machine rather than replace it. Next: Words as Vectors (Word2Vec).

The one-line takeaway

A modern LLM is not one idea — it is six ideas compounding, each answering the limit of the one before.

Read it / meet it

There is no arXiv id for the opener — it is the through-line, and each linked paper carries its own citation. You meet this lineage every time you type into a chat assistant: the word vectors, the attention, the pretraining, and the in-context learning are all still running under the hood. Start the track at the next video and follow it forward; by the end you will have read the field from its root paper to its frontier.

Named concepts: word embeddings, sequence-to-sequence, attention, Transformer, pretraining, in-context learning