Interactive Lab · Training & Optimization
Every optimizer has the same goal: reach the bottom of the loss, the minimum. What differs is how it gets there. Line up SGD, Momentum, RMSProp and Adam at the same start, feed them the same slopes, and race. On a narrow valley plain SGD zig-zags and stalls, while the others reuse past gradients to power along the floor, all reaching the minimum, but at wildly different speeds. That speed gap is the difference between training that takes hours and days.
▶ The playground, start the race
Every model you train has one line that picks an optimizer, optimizer = Adam(model.parameters(), lr=3e-4). That single choice decides whether training is fast and stable or slow and stuck. Adam is the default that “just works” for most deep learning and LLMs; SGD + momentum still wins for many vision models. This race is that one line, made visible, so you know what you’re choosing and why.
The code that's actually running
On the narrow valley, crank the learning rate up and race again. SGD wants a big step to make progress along the valley floor, but a big step makes it bounce off the steep walls and diverge, so it's stuck choosing between "too slow" and "explodes." Adam sidesteps the dilemma by keeping a separate step size for each direction (small across the valley, large along it). That per-parameter adaptivity is the whole reason Adam is the default optimizer for training almost every modern model.
The takeaway