I'm a second-year student at the University of Texas at Austin with an interest in engineering, math, and machine learning.
Overfitting means that the model fits the training data too well in the sense that while it has high accuracy when tested on examples it has already been trained on, it performs poorly on brand new examples it has not seen before and is thus unable to generalize. Another term for this is that the model has high variance. It is called high variance because if just one training example was different, the entire model would change drastically. On the other extreme, if the model does not fit the training data well, it is known as underfitting. Another term is that the model has high bias, meaning that the data has preconceptions that go against the existing data, which result in inaccurate predictions.
Underfitting is usually associated with too few features while overfitting is associated with too many features. Ideally, we want our models to be somewhere in the middle–neither underfitting nor overfitting the training data, and this means that we must be careful about what features to keep or discard when training the model.
NOTE: Overfitting applies to classification too. If our model is overfitting or underfitting, there is usually an issue in our decision boundary–it is either inaccurate or too perfect/aligned to the training data.
If your model experiences overfitting, what measures can you take to fix it?
Collect more training data. With a large training set, the model is more likely to tend towards a more generalizable algorithm as opposed to an overfit one.
Use fewer features. The more polynomial features you have, the more likely the model is to be overfit. By reducing the number of features, you can reduce the likelihood of the model becoming overfit. The process of choosing the right features to include is called feature selection. Note that you have to be extra cautious when selecting features, because by reducing features, you are also losing potentially critical information, which may impact the accuracy of the model. While you can use your intuition to determine which features are irrelevant, there may be non-obvious trends hidden in the features that you may inadvertently throw away by removing features.
Regularization. If you look at overfit models, you will notice that their parameters w and b tend to have relatively large values. By setting one or more of these parameters equal to 0, you are effectively removing them (which we talked about above). Regularization, on the other hand, is a less harsh way of subduing features–instead of outright removing them, regularization simply assigns a lower weight to them. This solves the problem of overfitting in cases where overfitting is caused by the features having an excessively strong effect on the output. Note that when we are talking about regularization, we are usually referring to reducing the weights (w values), not the b value.
