
- Top 1: Gaussian Process
- a GP only works cleanly when distance is informative and the noise is Gaussian.
- The kernel assumption. A GP assumes that everything about your function (its smoothness, length scale, and usually stationarity) is captured by the covariance function you choose, and that all function values are jointly Gaussian.
- Why it matters most: if the kernel is wrong (for example, a smooth stationary RBF kernel on data with jumps or a varying length scale), both the predictions and the uncertainty estimates are wrong, and nothing in the output warns you.
- Also worth knowing: standard kernels degrade in high dimensions (more than about 10–20 inputs), and non-Gaussian likelihoods such as classification need approximate inference.
- Kernels like RBF and Matérn measure similarity by Euclidean distance, and in high dimensions distance stops being informative.
- GP regression has an exact formula only because a Gaussian prior times a Gaussian noise model gives a Gaussian posterior. If you change the noise model, the closed form disappears.