PAPERS-11-003 · Paper — Factorization Machines

The paper

The landmark paper “Factorization Machines” by Steffen Rendle, published in 2010 at the IEEE International Conference on Data Mining, introduced a versatile model that quickly became a cornerstone for recommendation and click‑through prediction tasks.

Before this paper

Before this work, matrix factorization could only handle user and item IDs, and adding side information such as time, context, or demographics required custom‑built models. Moreover, trying to model every possible pair of features directly was impossible on sparse data because the number of parameters exploded. Practitioners were stuck with either very limited models or highly engineered solutions that did not scale.

The key idea

The key insight is to give each feature its own low‑dimensional factor vector and to represent the interaction between any two features as the dot product of their vectors. This lets the model learn interactions even for feature pairs that appear rarely, because the parameters are shared across many pairs. In effect, the model captures rich pairwise relationships without needing a separate weight for every possible pair.

How it works

A factorization machine takes any real‑valued feature vector as input. First, it adds a linear term for each feature, just like ordinary linear regression. Then, for every pair of features it computes the dot product of their factor vectors and multiplies by the feature values, summing all these interaction terms. Because each interaction weight is expressed as a dot product, the total computation can be done in linear time with respect to the number of non‑zero features. Training proceeds by optimizing the factor vectors (and linear weights) using standard gradient‑based methods on the observed data, which works well even when the data are very sparse.

See it — factorize the interactions

flowchart LR
  A["Any feature vector<br/>(sparse)"] --> B["Linear terms"]
  A --> C["Pairwise interactions<br/>via factor vectors"]
  B --> D["Prediction"]
  C --> D

What they showed

The authors showed that factorization machines matched the performance of specialized factorization models while being able to work with arbitrary feature sets, effectively generalizing matrix factorization.

What it changed

This paper turned factorization machines into a strong, general baseline for both click‑through and rating prediction tasks, offering a single model that could replace many bespoke solutions. It demonstrated that a simple, unified approach could achieve state‑of‑the‑art results across diverse domains, encouraging researchers to adopt FMs as a default starting point. Consequently, the community began to view FMs as a go‑to tool for any problem involving sparse, high‑dimensional features. Next: Wide & Deep.

The one-line takeaway

Factorization machines unify linear models and matrix factorization by learning feature interactions through shared low‑dimensional vectors.

Read it / meet it

Reading the original paper reveals the elegant simplicity behind a model that still powers many modern recommendation and advertising systems. Its legacy lives on in deep learning extensions and continues to influence how we think about feature interactions today.