PAPERS-02-004 · Paper — Deeper with Small Filters (VGG)

The paper

"Deeper with Small Filters (VGG)" by Simonyan and Zisserman (2014, ICLR 2015) is a landmark because it demonstrated that depth built from uniform 3×3 convolutions can dramatically improve visual recognition performance.

Before this paper

Before VGG, networks used a mix of large kernels and shallow depth, limiting the ability to capture hierarchical features and often requiring many parameters for a given receptive field. Researchers believed that increasing depth would cause optimization difficulties, so most architectures stayed relatively shallow. The community lacked a systematic study of how small, stacked filters affect depth and performance.

The key idea

The core idea is to replace large kernels with multiple stacked 3×3 convolutions, which achieve the same receptive field with fewer parameters and an extra non‑linearity. By stacking two 3×3 layers they emulate a 5×5 receptive field, and three layers emulate a 7×7 field. This design lets the network go much deeper while keeping the architecture simple and uniform.

How it works

VGG builds networks of 16 or 19 weight layers (VGG‑16 and VGG‑19) composed almost entirely of 3×3 convolutions followed by ReLU, with occasional max‑pooling to reduce spatial resolution. Each convolution uses a stride of 1 and padding to preserve size, so depth grows without shrinking the feature map too quickly. The repeated use of the same filter size simplifies implementation and encourages the learning of richer hierarchical representations. The final fully‑connected layers aggregate the deep features for classification, resulting in a model with about 138 M parameters.

See it — why stacked 3x3 convolutions win

The diagram below visualizes the VGG‑16 architecture, showing the cascade of 3×3 convolutions and pooling layers.

flowchart LR
  A["two 3x3 convs"] --> B["same receptive field<br/>as one 5x5"]
  B --> C["fewer parameters"]
  B --> D["more nonlinearity"]
  C --> E["deeper, stronger net"]
  D --> E

What they showed

On the ILSVRC‑2014 challenge VGG placed 1st in localization and 2nd in classification, achieving roughly a 7.3 % top‑5 error with an ensemble of models. The results confirmed that depth built from small filters can outperform shallower networks with larger kernels. The performance gap highlighted the importance of uniform depth over ad‑hoc architectural tweaks.

What it changed

VGG‑16’s pretrained weights became a de‑facto backbone for transfer learning across many vision tasks, thanks to its simple, uniform design. Its parameter‑heavy but effective architecture inspired later models to seek depth with more efficient building blocks (e.g., residual connections). The paper cemented the practice of using deep, homogeneous convolutional stacks as a starting point for new research.

Next: Efficient Depth (GoogLeNet / Inception).

The one-line takeaway

Stacking tiny filters is the secret sauce that turns depth into power.

Read it / meet it

Read the original VGG paper for the full architectural tables and training details, and explore its implementation in modern frameworks such as PyTorch’s torchvision.models.vgg16. You’ll encounter VGG‑derived backbones in countless transfer‑learning tutorials and benchmark suites today.