PAPERS-01-003 · Paper — Sequence to Sequence

The paper

Sequence to Sequence Learning with Neural Networks, by Ilya Sutskever, Oriol Vinyals, and Quoc V. Le, published at NeurIPS 2014 (arXiv:1409.3215). It is the paper that showed a single neural network — with no hand-built alignment, no phrase tables, no separate language model — could translate a whole sentence end to end. It reframed translation, and by extension almost any language task, as one clean problem: map an input sequence to an output sequence.

Before this paper

The best machine translation systems of the era were statistical phrase-based pipelines: they chopped sentences into phrases, looked up translation probabilities in enormous tables, and stitched the pieces back together with a separate language model and a reordering model. Neural networks had a role, but only as components bolted onto that pipeline — rescoring candidates, smoothing probabilities. Plain feedforward and recurrent nets were stuck on fixed-size inputs and outputs; nobody had a clean recipe for a net that ate a variable-length sentence and produced a variable-length translation of a different length.

The key idea

Use two recurrent networks. One network, the encoder, reads the source sentence one word at a time and compresses everything it has seen into a single fixed-length vector — a numerical summary of the whole sentence. A second network, the decoder, takes that vector as its starting state and generates the translation one word at a time, feeding each word it emits back in as the input for the next step. One net reads; the other writes. The bridge between them is that single context vector.

How it works

Both halves are multi-layer LSTMs — the gated recurrent cell that can carry information across many time steps without the gradient vanishing. The encoder runs to the end of the source and hands its final hidden state to the decoder as the seed. The decoder then produces words autoregressively until it emits an end-of-sentence token, and at inference a beam search keeps several candidate translations alive at once instead of greedily committing to one. The paper's most surprising trick is almost embarrassingly simple: reverse the word order of the source sentence before feeding it in. Reversing puts the first source words physically close to the first target words, shortening the dependencies the LSTM must remember and making optimization dramatically easier.

See it — encoder compresses to one vector, decoder writes from it

flowchart LR
  A["Source sentence<br/>(reversed order)"] --> B["Encoder LSTM<br/>reads word by word"]
  B --> C["Fixed-length<br/>context vector"]
  C --> D["Decoder LSTM<br/>writes word by word"]
  D --> E["Translation<br/>+ end-of-sentence"]

What they showed

On the WMT'14 English-to-French translation task, an ensemble of these LSTM models reached 34.8 BLEU, a strong result that was competitive with the mature phrase-based systems of the day — remarkable for an approach with no translation-specific engineering. The reversing trick was not a rounding-error improvement: it moved results by several BLEU points and turned a model that struggled on long sentences into one that handled them well. The learned sentence vectors also showed sensible structure — paraphrases landed near each other, and the representation was largely insensitive to word order in ways that tracked meaning.

What it changed

Sequence to sequence became the default template for a decade of language work: translation, summarization, dialogue, parsing, speech, and image captioning were all recast as "read one sequence, write another." The encoder-decoder shape it introduced is still the skeleton inside modern systems. But the paper carried its own weakness in plain sight: cramming an entire sentence into one fixed-length context vector is a bottleneck, and quality sags as sentences grow longer, because a single vector cannot hold everything. Next: Learning to Align — attention — which lets the decoder look back at any input word it needs instead of relying on one frozen summary.

The one-line takeaway

One network reads the sentence into a vector; another writes the translation out of it — end to end, no pipeline.

Read it / meet it

Read the original at arxiv.org/abs/1409.3215. The encoder-decoder idea it named lives today inside every translation model, every summarizer, and the decoder stack of every generative LLM — the Transformer simply swapped the LSTM for self-attention while keeping this read-then-write shape.

Named concepts: sequence-to-sequence learning, encoder-decoder, LSTM, fixed-length context vector