All modulesS20 · Deep learning III▾
Sooner or later, a network refuses to learn: the loss stays put, or the score is at guessing level. Panicking and changing things at random doesn't help. You have a checklist, in order, from simple to complex. Most of the time the problem is trivial and it's near the top of the list.
The test that solves half the cases
Before anything, check whether the model can overfit 10 examples. You take ten examples, turn off all regularization, and train until it should memorize them perfectly. If the loss gets close to zero, the learning machinery works and the problem is elsewhere (data, too aggressive regularization).
The checklist, in order
If the overfit-on-10-examples test fails, go through this list, top to bottom. These are the most common causes, roughly ordered by how often they show up.
- 01Learning rate: too big (chaotic loss or NaN) or too small (nothing moves). Try another value first.
- 02Data normalization: are the inputs scaled? A network struggles with data on huge or inconsistent ranges.
- 03Labels: are they aligned correctly with the inputs? Did you shuffle the order? Do you have the right loss function for the type of problem?
- 04zero_grad: are you calling optimizer.zero_grad() at each step? Without it, the gradients add up and training goes haywire.
- 05Initialization and gradients: check whether the gradients explode (become huge) or vanish (become zero). Batch norm and good initialization help.
How to read the loss curve
The loss curve is the main diagnostic tool. Look at it, not just the final score. Each shape tells you something different.
- Flat loss from the start: learning doesn't kick off. Most often the learning rate or the data.
- Loss that explodes or becomes NaN: learning rate too big or exploding gradients.
- Training loss drops, validation loss rises: overfitting, add regularization.
- Noisy but decreasing loss: probably normal, maybe a batch size that's too small.
- First test: can the model overfit 10 examples? If not, it's a bug.
- Debug in order: learning rate, normalization, labels, zero_grad, gradients.
- Don't change things at random; go down the list, one thing at a time.
- The loss curve tells you the cause: flat shape, explosion, or a train-validation gap.