PAPERS-01-004 · Paper — Learning to Align

The paper

Neural Machine Translation by Jointly Learning to Align and Translate, by Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Posted to arXiv in 2014 (id 1409.0473) and presented at ICLR 2015. It is the paper that introduced the attention mechanism — the single idea that, a few years later, would give the Transformer its name. Read today, it is a small, surgical change to an existing architecture that turned out to unlock the whole field.

Before this paper

The state of the art in neural translation was the sequence-to-sequence model: one recurrent network (the encoder) reads the entire source sentence and compresses it into a single fixed-length vector, and a second recurrent network (the decoder) reads that one vector and produces the translation word by word. It was elegant and end-to-end, and it worked — up to a point. The problem was the fixed-length vector itself: everything the model understood about a fifty-word sentence had to be packed into exactly the same size box as a five-word one. That box was a bottleneck, and the symptom was concrete: as sentences got longer, sequence-to-sequence quality fell off, because one vector simply could not hold the whole meaning.

The key idea

Bahdanau, Cho, and Bengio's insight was to stop forcing a single vector to carry everything. Instead of summarizing the source once, keep the encoder's representation of every input word around, and let the decoder look back over all of them at each output step, focusing on the parts it needs right now. Rather than a hard, pre-computed word-to-word alignment (as classical statistical translation used), the model learns a soft alignment: a distribution of attention weights over the whole input that it trains jointly with the translation itself. Hence the title — the network learns to align and to translate at the same time.

How it works

The encoder is run so that it produces one vector per source word (a bidirectional RNN, so each vector carries context from both sides). Then, at each step of generating the translation, the decoder computes a score for how well each source-word vector matches what it is about to produce. Those scores are pushed through a softmax, turning them into weights that are all positive and sum to one — the soft alignment. The context vector for that step is the weighted sum of all the encoder vectors, so it is dominated by the handful of source words that matter for the word being written. The decoder emits the next word from that focused context and repeats, building a fresh blend of the input for every single output word — no fixed bottleneck anywhere in the loop.

See it — attention re-blends the input for every output word

flowchart LR
  S["Source words"] --> E["Encoder<br/>one vector per word"]
  E --> A["Score + softmax<br/>weights per step"]
  A --> X["Context vector<br/>weighted sum"]
  X --> D["Decoder<br/>writes next word"]
  D -.->|"next step"| A

What they showed

The headline result was about exactly the weakness it set out to fix: the attention model did not degrade on long sentences. Where the fixed-vector baseline's quality dropped as inputs grew, the attention model held steady across sentence lengths — direct evidence that the bottleneck, not the recurrent network, had been the limit. The second result was interpretability: because the attention weights form a soft alignment over the source, you can plot them, and the plots show the model lining up each translated word with the source word it came from — a sensible, mostly monotonic alignment that no one hand-specified. The network discovered where to look, purely from the translation objective.

What it changed

Attention went from a one-off trick to the load-bearing idea of the field: soft, learned, differentiable focus over a set of representations. It made the encoder-decoder robust to length, gave models a built-in way to explain themselves, and generalized far beyond translation. But note where it lives in this paper — attention is bolted onto a recurrent network. The RNN still reads the sequence step by step; attention is an addition on top. That framing invites the obvious next question: if attention is what makes the model work, why keep the recurrence at all? What if attention were not an add-on but the whole model? That is precisely the leap the next paper takes. Next: Attention Is All You Need.

The one-line takeaway

Don't compress everything into one vector — let the model learn where to look.

Read it / meet it

Read the paper at arxiv.org/abs/1409.0473. The mechanism it introduced is now everywhere: every Transformer, and therefore every modern LLM you use, is built out of attention layers that are the direct descendants of this soft alignment — the same score-then-softmax-then-weighted-sum recipe, scaled up and stripped of the surrounding RNN.

Named concepts: attention mechanism, soft alignment, encoder-decoder