PAPERS-09-003 · Paper — Word2Vec

The paper

Word2Vec, introduced by Tomas Mikolov and colleagues in 2013 at the NIPS conference, is a landmark because it showed how to learn useful word meanings directly from raw text, sparking a new era of dense language representations.

Before this paper

Before Word2Vec, words were encoded as one‑hot vectors that treated every term as unrelated to every other. This representation gave no notion of similarity, so "cat" and "dog" were as distant as "cat" and "car". The field struggled to capture semantic relationships without massive hand‑crafted resources, limiting many NLP tasks.

The key idea

The key insight is to let a word’s meaning emerge from the company it keeps: by training a model to predict words from their contexts (or vice‑versa), each word acquires a dense vector that naturally places similar words near each other in space.

How it works

First, a large corpus is scanned to collect sliding windows of words. In the CBOW variant, the model averages the vectors of surrounding words and tries to predict the central word; in skip‑gram, it takes a target word and tries to predict each surrounding word. The model’s parameters—one vector per word—are updated via stochastic gradient descent to reduce prediction errors. Because the training objective forces words that appear in similar contexts to have similar vectors, the resulting embeddings capture nuanced relationships. The process is computationally efficient, allowing training on billions of words.

See it — words become vectors

flowchart LR
  A["Word in context"] --> B["Skip-gram / CBOW"]
  B --> C["Dense vector"]
  C --> D["king - man + woman<br/>= queen"]

What they showed

The authors demonstrated that the learned embeddings captured both semantic and syntactic regularities so well that simple vector arithmetic—e.g., "king" − "man" + "woman"—landed near "queen", illustrating meaningful geometric structure.

What it changed

Word2Vec launched the era of dense word representations, replacing sparse one‑hot encodings and inspiring the explosion of embedding‑based methods across NLP. It showed that a single, trainable embedding layer could serve as the foundation for many downstream tasks, fundamentally reshaping how language models are built. Next: Dropout.

The one-line takeaway

Dense word vectors learned from context turned language into a geometry we can navigate.

Read it / meet it

Reading the original paper reveals the elegant simplicity behind the breakthrough and why it still matters. Today, its legacy lives on in every modern language model, from search engines to recommendation systems, where embeddings form the first layer of understanding.