All modulesS19 · Deep learning II▾
A good model trained badly doesn't learn. This module is about the training loop: how you choose the optimizer, how you tune the learning rate (the most important knob), and how you keep overfitting under control. The same layers, trained carefully or carelessly, give completely different results.
Epochs, batches, steps
Three words to lock in. A batch is a small group of examples you process at once. An epoch is one full pass through the whole training set. A step is one weight update, meaning one batch processed. You usually train for many epochs, each with many steps.
Why batches and not the whole set at once? Because it's faster and adds useful noise that helps the model not get stuck. Why not one example at a time? Because batches use the hardware better. A batch size between 32 and 256 is typical.
Optimizers
The optimizer decides how it uses the gradients to update the weights. SGD (stochastic gradient descent) with momentum is solid and reliable: momentum adds inertia, like a ball rolling, so it gets over small bumps.
Adam and AdamW adapt on their own: they take bigger steps where needed and smaller ones where not. They start more easily and need less tuning, which is why they're the default for many. AdamW handles weight decay more correctly. Start with Adam/AdamW if you're not sure.
Learning rate, the most important knob
The learning rate (η) is the step size at each update. It's the most important hyperparameter in all of deep learning. Too big: the loss jumps around chaotically or explodes, the model diverges. Too small: it learns painfully slowly or gets stuck before reaching anywhere good.
Don't guess it, search for it. Try values on a logarithmic scale (for example 0.1, 0.01, 0.001) and look at the loss curve. You want the largest learning rate where the loss still drops smoothly, not chaotically.
Schedulers change the learning rate during training. A recipe that often works: a short warmup at the start (you raise the rate gradually, so you don't destabilize it), followed by a smooth decay (cosine) toward the end, to settle finely into the minimum.
Regularization: keeping overfitting in check
Regularization is any technique that stops the model from memorizing the training set. Without it, a big network learns the examples by heart and falls apart on new data. You have several tools, often used together:
- Dropout: during training, it randomly switches off part of the neurons at each step. It forces the network not to rely on a single path, so it generalizes better.
- Weight decay: penalizes large weights, keeping them small and the model simple.
- Batch normalization: normalizes the activations in each batch, stabilizes and speeds up training.
- Early stopping: you stop when the validation score starts getting worse, even if the training one is still dropping.
Weight initialization matters more than it seems at first. Weights started badly can block training from the start. Luckily, PyTorch layers have good default initializations, so you rarely need to step in, but it's worth knowing it's a factor.
- Batch, epoch, step: one batch processed = one step; one pass through the whole set = one epoch.
- Adam/AdamW start easily and need little tuning; SGD with momentum is solid.
- The learning rate is the most important knob; search for it on a logarithmic scale.
- Warmup plus cosine decay is a good scheduler recipe.
- Dropout, weight decay, batch norm and early stopping keep overfitting in check.