The term “non-parametric” can be a bit misleading at first glance: it doesn’t mean that these models have no parameters at all. Instead, non-parametric models can become increasingly complex as the amount of data grows.
Parametric machine learning algorithm is a learning model that summarizes data with a set of parameters of fixed size (independent of the number of training examples) is called a parametric model. No matter how much data you throw at a parametric model, it won’t change its mind about how many parameters it needs. — Artificial Intelligence: A Modern Approach, page 737
Nonparametric methods are good when you have a lot of data and no prior knowledge, and when you don’t want to worry too much about choosing just the right features. — Artificial Intelligence: A Modern Approach, page 757
In parametric models, there is a finite number of parameters, whereas in non-parametric models, the number of parameters can be infinite. Essentially, the complexity of non-parametric models scales with the amount of training data, while parametric models have a fixed number of parameters.
Examples of parametric models include linear regression, logistic regression, and linear Support Vector Machines (SVMs), where the number of parameters (weight coefficients) is fixed. On the other hand, K-nearest neighbor, decision trees, and SVMs with an RBF kernel are non-parametric because their complexity grows with the size of the training dataset. Specifically, an RBF kernel SVM is non-parametric because the kernel matrix is constructed by calculating pair-wise distances between training points.
In statistics, parametric models are also associated with the assumption of a specific probability distribution for the data, characterized by a finite set of parameters (such as the mean and standard deviation in a normal distribution). Non-parametric models do not make such assumptions and are essentially distribution-free.
It’s important to note that the definitions of “parametric” and “non-parametric” are somewhat ambiguous. As stated in “The Handbook of Nonparametric Statistics 1 (1962),” there isn’t a precise and universally accepted definition of “non-parametric.” Generally, a statistical procedure is considered non-parametric if it works under assumptions that are reasonably general.
