In a standard linear regression, the predictor coefficients that characterize the relationship between the predictors and an outcome variable are model parameters that are estimated from the data (via least squares). A machine learning model can also have parameters that are “learned” from the data, e.g. node weights in a neural net.

A machine learning algorithm will also have hyperparameters that are set by the user – how many layers to have in a neural net, for example, or how many levels to have in a decision tree. (Actually, this distinction is a bit artificial – often hyperparameter adjustments are embedded in an iterative cross-validation process to optimize them – but you could still imagine setting them by hand in a way that you could not for the core machine learning algorithm.)

In Bayesian statistics, the standard statistical modeling process is integrated with the estimation and specification of a “prior” distribution to incorporate more information about the problem than is provided by the sample of data at hand. Hyperparameters are the parameters of that prior distribution.

Learn more about hyperparameters in our Bayesian Statistics courses and our Predictive Analytics series.