Summary of Generative Models: LLMs and Transformers

Generative Models: LLMs & Transformers Explained for Students

Summary Knowledge test Flashcards Podcast Mindmap

Introduction

Machine learning paradigms organize how we formulate, train, evaluate, and choose models. This guide summarizes the main paradigms used in supervised and unsupervised learning, probabilistic interpretations, loss and regularization choices, common optimization methods, and model selection strategies. The goal is to give a compact, practical view you can use when designing or analyzing learning systems.

Definition: A learning paradigm is a high-level framework that prescribes the type of data, objective, model family, and learning algorithm used to solve a task.

1. High-level categories

Supervised learning

Trains models from labeled data $(x,y)$ to predict targets $y$ from inputs $x$.
Typical tasks: regression (continuous $y$) and classification (discrete $y$).

Definition: Supervised learning uses labeled examples to minimize a loss that measures prediction error on the labels.

Unsupervised learning

Learns structure from unlabeled data $x$ alone: clustering, density estimation, dimensionality reduction.

Definition: Unsupervised learning finds patterns, low-dimensional structure, or groups in data without explicit target labels.

Semi-supervised and generative-discriminative perspectives

Semi-supervised learning uses a mix of labeled and unlabeled data (e.g., semi-supervised Gaussian mixtures).
Discriminative models directly model $p(y,|,x)$ or decision boundaries; generative models model $p(x,y)$ or $p(x)$ and derive $p(y,|,x)$ via Bayes rule.

2. Loss functions and probabilistic interpretation

A loss function often corresponds to a negative log-likelihood under a probabilistic model.

Definition: Loss function is a mapping from predictions and true labels to a nonnegative penalty quantifying error; negative log-likelihood connects losses to probabilistic models.

Key examples (loss and probabilistic view):

Squared loss: corresponds to Gaussian likelihood. For regression, minimize expected squared error: $L(y,\hat y) = (y-\hat y)^2$.
Logistic loss: corresponds to Bernoulli likelihood for binary classification; connects to logistic regression.
Cross-entropy loss: corresponds to categorical likelihood for multi-class classification.
Hinge loss: used in SVMs for margin-based classification.
0/1 loss: direct task loss for classification (hard to optimize).
Reconstruction error: used in dimensionality reduction and autoencoders.

Practical note: choose a loss that matches the noise model and the optimization tractability you need.

3. Regularization and priors

Regularization penalizes model complexity to improve generalization and can be interpreted as priors in a Bayesian view.

Definition: Regularization is any penalty added to the loss to prefer simpler models and prevent overfitting.

Common penalties and their probabilistic equivalents:

$L_2$ penalty: $\lambda |w|_2^2$, corresponds to a Gaussian prior on parameters.
$L_1$ penalty: $\lambda |w|_1$, corresponds to a Laplace prior and encourages sparsity (e.g., Lasso, l1-SVM).
$L_0$ penalty: counts nonzero parameters; ideal for sparsity but combinatorial.
Early stopping: limits training iterations to act as implicit regularization.
Dropout: randomly remove features/units during training to reduce co-adaptation (used in neural nets; excluded from detailed coverage per instructions).

4. Common supervised models (overview)

Linear regression and ridge regression (least squares + $L_2$).
Lasso ($L_1$-regularized least squares) for sparse linear models.
Logistic regression for binary classification with logistic loss and optional $L_1$/$L_2$ regularization.
Support Vector Machines (SVMs): margin maximization with hinge loss; can use $L_1$ or $L_2$ regularization.
Kernelized methods: apply kernel functions to implicitly map features into high-dimensional spaces for kernelized SVM, kernel ridge regression.
k-Nearest Neighbor: nonparametric, instance-based classification/regressi

Zaregistruj se pro celé shrnutí

FlashcardsKnowledge testSummaryPodcastMindmap

Start for free

Already have an account? Sign in

Machine Learning Paradigms

Klíčové pojmy: Supervised learning uses labeled pairs $(x,y)$ to minimize a predictive loss, Unsupervised learning finds structure from unlabeled $x$ such as clusters or low-dimensional embeddings, Squared loss corresponds to Gaussian likelihood; logistic/cross-entropy correspond to Bernoulli/categorical likelihoods, $L_2$ regularization equals a Gaussian prior; $L_1$ equals a Laplace prior and promotes sparsity, Choose loss to match noise model and task-specific needs (e.g., F1 for imbalanced classes), Use GD/SGD for differentiable losses, EM for latent-variable models, and Lloyd’s algorithm for k-Means, Kernel methods allow nonlinear decision boundaries via implicit feature maps computed by kernels, Model selection via K-fold CV or Bayesian model comparison; evaluate with task-appropriate metrics

## Introduction Machine learning paradigms organize how we formulate, train, evaluate, and choose models. This guide summarizes the main paradigms used in supervised and unsupervised learning, probabilistic interpretations, loss and regularization choices, common optimization methods, and model selection strategies. The goal is to give a compact, practical view you can use when designing or analyzing learning systems. > Definition: A learning paradigm is a high-level framework that prescribes the type of data, objective, model family, and learning algorithm used to solve a task. ## 1. High-level categories ### Supervised learning - Trains models from labeled data $(x,y)$ to predict targets $y$ from inputs $x$. - Typical tasks: regression (continuous $y$) and classification (discrete $y$). > Definition: Supervised learning uses labeled examples to minimize a loss that measures prediction error on the labels. ### Unsupervised learning - Learns structure from unlabeled data $x$ alone: clustering, density estimation, dimensionality reduction. > Definition: Unsupervised learning finds patterns, low-dimensional structure, or groups in data without explicit target labels. ### Semi-supervised and generative-discriminative perspectives - Semi-supervised learning uses a mix of labeled and unlabeled data (e.g., semi-supervised Gaussian mixtures). - Discriminative models directly model $p(y\,|\,x)$ or decision boundaries; generative models model $p(x,y)$ or $p(x)$ and derive $p(y\,|\,x)$ via Bayes rule. ## 2. Loss functions and probabilistic interpretation - A loss function often corresponds to a negative log-likelihood under a probabilistic model. > Definition: Loss function is a mapping from predictions and true labels to a nonnegative penalty quantifying error; negative log-likelihood connects losses to probabilistic models. Key examples (loss and probabilistic view): - Squared loss: corresponds to Gaussian likelihood. For regression, minimize expected squared error: $L(y,\hat y) = (y-\hat y)^2$. - Logistic loss: corresponds to Bernoulli likelihood for binary classification; connects to logistic regression. - Cross-entropy loss: corresponds to categorical likelihood for multi-class classification. - Hinge loss: used in SVMs for margin-based classification. - 0/1 loss: direct task loss for classification (hard to optimize). - Reconstruction error: used in dimensionality reduction and autoencoders. Practical note: choose a loss that matches the noise model and the optimization tractability you need. ## 3. Regularization and priors - Regularization penalizes model complexity to improve generalization and can be interpreted as priors in a Bayesian view. > Definition: Regularization is any penalty added to the loss to prefer simpler models and prevent overfitting. Common penalties and their probabilistic equivalents: - $L_2$ penalty: $\lambda \|w\|_2^2$, corresponds to a Gaussian prior on parameters. - $L_1$ penalty: $\lambda \|w\|_1$, corresponds to a Laplace prior and encourages sparsity (e.g., Lasso, l1-SVM). - $L_0$ penalty: counts nonzero parameters; ideal for sparsity but combinatorial. - Early stopping: limits training iterations to act as implicit regularization. - Dropout: randomly remove features/units during training to reduce co-adaptation (used in neural nets; excluded from detailed coverage per instructions). ## 4. Common supervised models (overview) - Linear regression and ridge regression (least squares + $L_2$). - Lasso ($L_1$-regularized least squares) for sparse linear models. - Logistic regression for binary classification with logistic loss and optional $L_1$/$L_2$ regularization. - Support Vector Machines (SVMs): margin maximization with hinge loss; can use $L_1$ or $L_2$ regularization. - Kernelized methods: apply kernel functions to implicitly map features into high-dimensional spaces for kernelized SVM, kernel ridge regression. - k-Nearest Neighbor: nonparametric, instance-based classification/regressi