PAPERS-07-006 · Paper — Graph Attention Networks (GAT)

The paper

The 2018 paper “Graph Attention Networks” by Petar Veličković, et al., presented at ICLR, is a landmark because it introduced attention mechanisms to graph neural networks, enabling the model to learn which neighbors are most important.

Before this paper

Before GAT, popular graph neural networks like GCN and GraphSAGE treated all neighbors the same, using either uniform weighting or a fixed degree‑based normalization. This meant that subtle but crucial differences between neighbors were ignored. As a result, the models could not focus on the most informative connections, limiting their expressive power. Researchers recognized the need for a way to weight neighbors adaptively.

The key idea

The core insight of GAT is to let the network itself learn how much attention to give each neighbor instead of relying on preset weights. By computing a learned attention score from the features of a node and each of its neighbors, the model can highlight the most relevant connections. This simple idea brings the flexibility of attention, already successful in NLP, to graph data.

How it works

First, each node’s feature vector is linearly transformed. Then, for every edge, the transformed features of the two incident nodes are combined and passed through a shared attention mechanism to produce a raw attention coefficient. These coefficients are normalized across all neighbors of a node using a softmax, yielding attention weights that sum to one. Each neighbor’s feature is then multiplied by its weight, and the weighted features are summed to form the node’s updated representation. The whole process is fully parallelizable and respects the graph’s adjacency mask, so only existing edges contribute. Finally, multi‑head attention can be applied and the results concatenated or averaged for robustness.

See it — weighting the neighbors

flowchart LR
  A["Node + neighbors"] --> B["Attention scores<br/>per neighbor"]
  B --> C["Weighted sum"]
  C --> D["New representation"]

What they showed

The authors showed that GAT matched or beat prior graph networks on standard node‑classification benchmarks, while also providing interpretable neighbor weights that reveal which connections mattered most.

What it changed

GAT introduced the attention mechanism to graph neural networks, making attention a standard building block in modern graph models. This shift allowed subsequent works to design more expressive and adaptive GNNs that can handle heterogeneous and noisy neighborhoods. The paper’s success demonstrated that learned weighting of edges is both powerful and practical, influencing countless follow‑up studies and system implementations. Next: Graph Isomorphism Network (GIN).

The one-line takeaway

Attention lets graph networks focus on the most important neighbors, dramatically improving performance and interpretability.

Read it / meet it

Reading the original GAT paper gives a clear view of how attention can be integrated into graph learning, and the ideas are now visible in many state‑of‑the‑art graph systems and libraries. Exploring its citations will show how the attention paradigm has become a cornerstone of today’s graph AI.