PAPERS-06-002 · Paper — Connectionist Temporal Classification (CTC)

The paper

The 2006 paper “Connectionist Temporal Classification: Labelling Unsegmented Sequence Data with Recurrent Neural Networks” by Alex Graves, Santiago Fernández, Faustino Gomez and Jürgen Schmidhuber, presented at ICML, is a landmark because it introduced the first practical way to train end‑to‑end speech and handwriting recognizers without needing frame‑by‑frame labels.

Before this paper

Before CTC, building speech or handwriting recognizers required data labeled at every time step. Researchers had to know exactly which slice of audio or pen trace corresponded to each character. This frame‑wise labeling was costly, fragile, and limited the scalability of sequence models. Consequently, truly end‑to‑end systems were out of reach.

The key idea

The core insight of CTC is to let the network emit a label or a special blank symbol at each time step and to train it by summing the probabilities of all possible alignments that collapse to the desired output sequence. In other words, the model does not need pre‑segmented labels; it learns the alignment itself. This turns a hard labeling problem into a tractable loss that can be optimized with standard back‑propagation.

How it works

During training, the network produces a probability distribution over the alphabet plus a blank token at every time step. A blank token means “no output here”. All possible ways of inserting blanks and repeating characters that reduce to the target string are enumerated implicitly using a dynamic‑programming forward‑backward algorithm. The algorithm efficiently computes the total probability of the correct sequence by adding up the probabilities of every valid alignment. The negative log of this total probability becomes the CTC loss, which is back‑propagated to update the network weights. At inference time, the most likely path is collapsed by removing blanks and merging repeated symbols to produce the final transcription.

See it — align-free labeling

flowchart LR
  A["Audio frames"] --> B["RNN per-frame<br/>predictions"]
  B --> C["Sum over all<br/>alignments (+ blank)"]
  C --> D["Output text"]

What they showed

The authors showed that CTC enabled the first practical end‑to‑end recognizers for unsegmented speech and handwriting, demonstrating that high‑quality models could be trained without any frame‑level annotation.

What it changed

CTC became a foundational training method for sequence recognition, making end‑to‑end speech recognition feasible for the first time. It removed the need for costly alignment data, opening the door to larger, more flexible models. The approach quickly spread to many other domains, becoming a standard component in modern speech and handwriting systems. Its impact is still felt in virtually every production recognizer today. Next: DeepSpeech.

The one-line takeaway

CTC taught neural networks to learn their own alignment, eliminating the need for hand‑crafted frame labels.

Read it / meet it

Reading the original paper gives a clear view of how the loss is derived and why it works, and it’s surprisingly accessible for motivated beginners. You’ll see the roots of today’s speech‑to‑text engines, which still rely on CTC or its modern variants.