PAPERS-01-005 · Paper — Attention Is All You Need

The paper

Attention Is All You Need, by Vaswani et al., 2017, published at NeurIPS (arXiv:1706.03762). It introduced the Transformer — an encoder-decoder built entirely from attention and feed-forward layers, with no recurrence and no convolution. The title is a deliberate provocation: attention had been a helpful add-on to recurrent translators, and the paper's claim was that it could be the whole model. That one architecture is the direct ancestor of every large language model in use today.

Before this paper

The state of the art in machine translation was the recurrent sequence-to-sequence model with attention: an LSTM or GRU read the source sentence one token at a time, carrying a hidden state forward, and a decoder used an attention mechanism to look back at the encoded words. It worked well, but its core was inherently sequential. Token ten could not be computed until token nine was finished, which capped how much of the training could run in parallel and made long-range dependencies costly to preserve across many steps. Convolutional approaches parallelized better but still needed many stacked layers to relate distant words.

The key idea

Remove the recurrence altogether and let self-attention do all the connecting. Instead of passing information hop-by-hop along a chain, every position attends directly to every other position in a single step — the path length between any two words becomes constant rather than growing with distance. If attention can relate any two tokens on its own, the sequential backbone is unnecessary. Hence the title: attention is all you need.

How it works

Self-attention scores each token against every other token, then blends in the ones that matter, so context is gathered in one parallel operation rather than a sequence of steps. The paper runs this in multi-head attention — several attention operations in parallel, each free to learn a different kind of relationship (syntax, agreement, reference) — and concatenates their results. Because pure attention is order-blind, the model adds positional encodings to the token representations so the network knows where each word sits. These pieces stack into an encoder (self-attention plus feed-forward blocks) that reads the input and a decoder that attends back to the encoder while generating the output.

See it — encoder reads, decoder attends back and writes

flowchart LR
    IN["Input tokens<br/>+ positional encoding"] --> ENC["Encoder<br/>self-attention"]
    ENC --> MEM["Encoded<br/>representation"]
    MEM --> DEC["Decoder<br/>attends to input"]
    DEC --> OUT["Output tokens"]

What they showed

On the WMT 2014 English-to-German translation benchmark the Transformer scored 28.4 BLEU, a new state of the art. On English-to-French it reached 41.8 BLEU, again the best reported result at the time. Crucially, it got there cheaply: the big model trained in about 3.5 days on 8 GPUs, a small fraction of the training cost of the previous best models. Better and cheaper was the whole point — the parallelism that self-attention allowed turned directly into faster, stronger training.

What it changed

The Transformer stopped being a translation trick and became a general-purpose substrate. Once you had an architecture that scaled with parallel hardware and related any two tokens directly, the open question was no longer how to build the model but what to train it on. This architecture is the substrate for everything after. Two teams now ask what to TRAIN it on. Next: BERT, then GPT.

The one-line takeaway

Drop the recurrence — self-attention alone, run in parallel, was enough to beat it and to launch the era of large language models.

Read it / meet it

Read it on arXiv: https://arxiv.org/abs/1706.03762. The Transformer now lives inside essentially every modern language model — BERT, the GPT family, and their descendants are all stacks of the attention and feed-forward blocks this paper defined. When you use an LLM today, you are running the architecture from this 2017 paper.

Named concepts: Transformer, self-attention, multi-head attention, positional encoding