Summary of Probabilistic Machine Learning

## Introduction Generative models describe how data are generated by modeling a joint distribution $p(x,y)$ or $p(x, z)$, then using that model for tasks such as sampling, density estimation, or classification. This material focuses on parametric generative classifiers that assume class-conditional distributions for features and show how parameter estimates lead to practical prediction rules and decision boundaries. > **Definition:** A *generative classifier* models class priors $p(y)$ and class-conditional feature distributions $p(x\mid y)$, and predicts by choosing the class that maximizes the posterior $p(y\mid x)$. ## 1. From data to parameter estimates (intuitive steps) - We observe labeled data $\{(x_i,y_i)\}_{i=1}^n$ with $c$ classes. For each class $y$ let $n_y$ denote the number of examples with label $y$. - Estimate class prior probabilities by relative frequency: $\hat p(y)=\dfrac{n_y}{n}$. - For continuous features, adopt a parametric family for $p(x\mid y)$. A common choice is the Gaussian family (one-dimensional per feature or multivariate). Estimating the Gaussian parameters per class yields means and variances used in the classifier. > **Definition:** The *class prior* $p(y)$ is the marginal probability of class $y$. The *class-conditional distribution* $p(x\mid y)$ models how features are distributed within class $y$. ## 2. Gaussian class-conditionals (component-wise independence) Assume $d$ features and for each class $y$ the features are independent given $y$ and each feature $j$ follows a Gaussian with mean $\mu_{y,j}$ and variance $\sigma^2_{y,j}$. Then $$p(x\mid y)=\prod_{j=1}^d p(x_j\mid y)\,,\qquad p(x_j\mid y)=\dfrac{1}{\sqrt{2\pi\sigma^2_{y,j}}}\exp\left(-\dfrac{(x_j-\mu_{y,j})^2}{2\sigma^2_{y,j}}\right).$$ Parameter estimates used in practice (sample mean and sample variance within each class) are $$\hat\mu_{y,j}=\dfrac{1}{n_y}\sum_{i: y_i=y} x_{i,j}$$ $$\hat\sigma^2_{y,j}=\dfrac{1}{n_y}\sum_{i: y_i=y} \left(x_{i,j}-\hat\mu_{y,j}\right)^2.$$ Practical notes: - Use class-wise sample means as the center for each Gaussian component. - Use class-wise variances to capture spread; if $n_y$ is small, estimates can be noisy. - When dimensionality is large, a diagonal covariance (feature independence) reduces the number of parameters. ## 3. Prediction rule and discriminant function Given estimates, predict for a new point $x$ by maximizing the posterior $$\hat y=\operatorname{argmax}_y\ \hat p(y\mid x)=\operatorname{argmax}_y\ \hat p(y)\prod_{j=1}^d \hat p(x_j\mid y).$$ Taking logs to avoid numerical underflow yields the discriminant score $$\hat f_y(x)=\log\hat p(y)-\dfrac{1}{2}\sum_{j=1}^d\log\hat\sigma^2_{y,j}-\dfrac{1}{2}\sum_{j=1}^d\dfrac{(x_j-\hat\mu_{y,j})^2}{\hat\sigma^2_{y,j}}.$$ The predicted label is $\hat y=\operatorname{argmax}_y\hat f_y(x)$. For binary classification with labels $+1$ and $-1$, the classifier can be written using the discriminant difference $$\hat f(x)=\log\dfrac{\hat p(+1\mid x)}{\hat p(-1\mid x)}$$ and the predicted label is $\operatorname{sign}(\hat f(x))$. ## 4. Decision boundaries and geometry - Decision boundaries arise where two class scores are equal: $\hat f_{y_1}(x)=\hat f_{y_2}(x)$. Depending on the class covariances, boundaries can be linear or non-linear. - If each class has its own diagonal covariance, boundaries are generally curved (quadratic in $x$). > **Definition:** A *decision boundary* is the set of feature vectors $x$ for which two or more classes have equal posterior probability. ## 5. Shared-variance special case and Fisher-like linear discriminant If variances are shared across classes for each feature, i.e. $\sigma^2_{+1,j}=\sigma^2_{-1,j}=\sigma^2_j$, the quadratic terms in the discriminant cancel in the difference between classes and the decision boundary becomes linear in $x$. This yields a simpler linear classifier closely related to classical linear discriminant analysis. Consequence: when variances are equal across classes, the decision