PAPERS-01-002 · Paper — Words as Vectors

The paper

Efficient Estimation of Word Representations in Vector Space, by Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean, 2013, ICLR Workshop track, arXiv:1301.3781. The system it introduced is universally known as Word2Vec. It is short, almost blunt — its whole argument is that you can get better word vectors by using a simpler model, not a fancier one. That inversion is why it mattered.

Before this paper

The dominant way to hand a word to a model was one-hot encoding: a vocabulary-sized vector, all zeros except a single 1 in the word's slot. In that scheme every word is exactly the same distance from every other — "cat" is no closer to "dog" than to "Tuesday." Nothing about meaning survives. Researchers already knew that neural language models could learn richer, dense word vectors as a side effect of predicting text, but those models carried an expensive non-linear hidden layer that made training on large corpora painfully slow. Good vectors existed in principle; they were just too costly to learn at scale.

The key idea

Word2Vec's move was to throw the hidden layer away. Keep a deliberately shallow model whose only job is to predict which words appear near which other words, and train it on enormous amounts of raw text. The bet was that the representation — not the classifier on top — is what you actually want, and a simple predictive task is enough to shape it. Meaning, in this view, is a position in space: a word ends up near the words it keeps company with.

How it works

The paper offers two architectures. CBOW (Continuous Bag-of-Words) takes the surrounding context words and predicts the missing word in the middle. Skip-gram flips the direction: from a single word, it predicts the surrounding context. Both are trained by self-supervision — the text is its own label, so no human annotation is needed — and both produce a dense embedding, a few hundred real numbers per word, pulled from the model's weight matrix. Because there is no hidden non-linearity to compute, each training step is cheap, which is exactly what lets the model swallow far more data than its predecessors.

See it — CBOW predicts the word; Skip-gram predicts the context

flowchart LR
    C["context words"] --> M1["shallow model"]
    M1 --> W["target word (CBOW)"]
    X["one word"] --> M2["shallow model"]
    M2 --> CX["context (Skip-gram)"]

What they showed

Trained on about 1.6 billion words, the learned vectors did something that startled the field: vector arithmetic captured analogies. Take the vector for king, subtract man, add woman, and the nearest vector in the space is queen. Take Paris, subtract France, add Italy, and you land on Rome. The model had never been told about gender or capital cities — those relationships emerged as consistent directions in the space, purely from the task of predicting neighbors. And it did all this while being dramatically faster to train than the neural language models that came before it: higher quality and larger scale at the same time.

What it changed

Word2Vec turned embeddings into the default first layer of nearly every language system for the next several years — you looked up each word's dense vector, then built your model on top. It made "meaning as geometry" a working engineering fact rather than a theory. But the vectors were static: one word, one vector, forever. "Bank" the riverbank and "bank" the vault collapse onto a single point, no matter the sentence. Static one-vector-per-word is the limit this lineage spends the next decade removing. Next: Sequence to Sequence.

The one-line takeaway

Meaning can live in geometry — teach a model to predict a word's neighbors, and its coordinates learn what the word means.

Read it / meet it

Read the original on arXiv: https://arxiv.org/abs/1301.3781. The idea lives on in every embedding layer of every modern model, in vector search and retrieval systems, and in the sentence- and document-embedding APIs that power today's semantic search and RAG pipelines — all descendants of the notion that a word is a point in space.

Named concepts: word embeddings, Word2Vec, CBOW, Skip-gram, vector arithmetic