Summary of Introduction to Simple Regression Analysis

Introduction to Simple Regression Analysis Guide

Summary Knowledge test Flashcards Podcast Mindmap

Introduction

Linear regression inference and diagnostics explore how to interpret, evaluate and draw conclusions from a fitted linear model. This study note walks through the deviation form of the two-variable model, estimation of coefficients by ordinary least squares (OLS), measures of fit such as the coefficient of determination, formulas for sampling variances and standard errors, computation of the residual sum of squares and the estimator of the error variance, and key relationships used in inference and diagnostics.

Definition: The deviation form of a two-variable linear model subtracts sample means from each variable so the transformed model has zero means for the transformed variables. If $Y_i$ and $X_i$ are original variables and $\bar{Y}$, $\bar{X}$ their sample means, define $y_i = Y_i - \bar{Y}$ and $x_i = X_i - \bar{X}$. The deviation model is $y_i = \hat{\beta} x_i + \hat{u}_i$.

1. Deviation form and why it helps

Starting from the sample regression function with intercept:

$$Y_i = \hat{\alpha} + \hat{\beta} X_i + \hat{u}_i$$

Taking averages gives:

$$\bar{Y} = \hat{\alpha} + \hat{\beta} \bar{X}$$

Subtracting the mean relation from the original equation yields the deviation form (no intercept):

$$y_i = \hat{\beta} x_i + \hat{u}_i$$

Benefits of the deviation form:
- Simplifies algebra for slope estimation because $\sum x_i = 0$ and $\sum y_i = 0$.
- Makes some sums (like $\sum x_i y_i$) equivalent to centered sums.

Definition: The deviation sums are $\sum x_i y_i = \sum (X_i-\bar{X})(Y_i-\bar{Y})$, $\sum x_i^2 = \sum (X_i-\bar{X})^2$. These express covariance and variance up to scale.

2. OLS slope and intercept in deviation form

Slope estimator:

$$\hat{\beta} = \frac{\sum x_i y_i}{\sum x_i^2}$$

Intercept from sample means:

$$\hat{\alpha} = \bar{Y} - \hat{\beta} \bar{X}$$

Equivalent forms useful in practice (when working with raw $X,Y$):

$$\hat{\beta} = \frac{n\sum X_i Y_i - \sum X_i \sum Y_i}{n\sum X_i^2 - (\sum X_i)^2}$$$$\hat{\alpha} = \frac{\sum X_i^2\sum Y_i - \sum X_i \sum X_i Y_i}{n\sum X_i^2 - (\sum X_i)^2}$$

Practical example (price $X$, quantity $Y$ with $n=5$): given $\sum X=15$, $\sum Y=46$, $\sum X^2=55$, $\sum XY=113$, and using means $\bar{X}=3$, $\bar{Y}=9.2$ we get centered sums $\sum x_i^2=10$, $\sum x_i y_i=-25$, so

$$\hat{\beta} = \frac{-25}{10} = -2.5$$

$$\hat{\alpha} = 9.2 - (-2.5)\times 3 = 16.7$$

💡 Věděli jste?Fun fact: The deviation-form slope $\hat{\beta}$ is unchanged whether you compute it from centered sums or from raw sums using the full-form formula; centering only reorganizes the algebra.

3. Decomposing total variation (TSS = ESS + RSS)

Total sum of squares (TSS):

$$\text{TSS} = \sum (Y_i - \bar{Y})^2 = \sum y_i^2$$

Explained sum of squares (ESS):

$$\text{ESS} = \sum (\hat{Y}_i - \bar{Y})^2 = \sum \hat{y}_i^2$$

Residual sum of squares (RSS):

$$\text{RSS} = \sum (Y_i - \hat{Y}_i)^2 = \sum \hat{u}_i^2$$

Identity:

$$\text{TSS} = \text{ESS} + \text{RSS}$$

Coefficient of determination:

$$r^2 = \frac{\text{ESS}}{\text{TSS}} = 1 - \frac{\text{RSS}}{\text{TSS}}$$

Alternative expressions useful for centered variables:

$$r^2 = \frac{\hat{\beta}^2 \sum x_i^2}{\sum y_i^2}$$$$r^2 = \frac{(\sum x_i y_i)^2}{\sum x_i^2 \sum y_i^2}$$

Definition: The sample correlation coefficient between centered $x$ and $y$ is $r_{xy} = \frac{\sum x_i y_i}{\sqrt{\sum x_i^2}\sqrt{\sum y_i^2}}$. Then $r_{xy}^2 = r^2$ in the simple regression case. Did you know that in simple linear regression the slope can be written as $\hat{\beta}=r_{xy}\dfrac{S_y}{S_x}$ where $S_x$, $S_y$ are sample standard deviations? This links correlation, variances and slope directly.

4. Residual sums and estimator for error variance

Using deviation form with $\hat{y}_i = \hat{\beta} x_i$:

$$\sum y_i^2 = \hat{\beta}^2 \sum x_i^2 + \sum \hat{u}_i^2$$so

$$\sum \hat{u}_i^2 = \sum y_i^2 - \hat{\beta}^2 \sum x_i^2$$

Alternative algebraic form with raw sums:

$$\sum \hat{u}_i^2 = \

Zaregistruj se pro celé shrnutí

FlashcardsKnowledge testSummaryPodcastMindmap

Start for free

Already have an account? Sign in

Regression Inference Notes

Klíčové pojmy: Deviation form removes intercept so $\sum x_i=\sum y_i=0$, Slope estimator $\hat{\beta}=\frac{\sum x_i y_i}{\sum x_i^2}$, Intercept $\hat{\alpha}=\bar{Y}-\hat{\beta}\bar{X}$, TSS = ESS + RSS and $r^2=\frac{ESS}{TSS}$, Alternative $r^2=\dfrac{(\sum x_i y_i)^2}{\sum x_i^2\sum y_i^2}=r_{xy}^2$, RSS computed as $\sum \hat{u}_i^2=\sum y_i^2-\hat{\beta}^2\sum x_i^2$, $\hat{\sigma}^2=\dfrac{\sum \hat{u}_i^2}{n-2}$ gives unbiased variance estimate, $\operatorname{Var}(\hat{\beta})=\dfrac{\sigma^2}{\sum x_i^2}$ and $s.e.(\hat{\beta})=\sqrt{\dfrac{\sigma^2}{\sum x_i^2}}$, $\operatorname{Var}(\hat{\alpha})=\sigma^2\left(\dfrac{\bar{X}^2}{\sum x_i^2}+\dfrac{1}{n}\right)$, Covariance $\operatorname{Cov}(\hat{\alpha},\hat{\beta})=-\bar{X}\dfrac{\sigma^2}{\sum x_i^2}$, Use $\hat{\sigma}^2$ to compute estimated standard errors for t-tests, Formula invariance: computing with centered sums or raw sums yields identical estimators

## Introduction Linear regression inference and diagnostics explore how to interpret, evaluate and draw conclusions from a fitted linear model. This study note walks through the deviation form of the two-variable model, estimation of coefficients by ordinary least squares (OLS), measures of fit such as the coefficient of determination, formulas for sampling variances and standard errors, computation of the residual sum of squares and the estimator of the error variance, and key relationships used in inference and diagnostics. > Definition: The deviation form of a two-variable linear model subtracts sample means from each variable so the transformed model has zero means for the transformed variables. If $Y_i$ and $X_i$ are original variables and $\bar{Y}$, $\bar{X}$ their sample means, define $y_i = Y_i - \bar{Y}$ and $x_i = X_i - \bar{X}$. The deviation model is $y_i = \hat{\beta} x_i + \hat{u}_i$. ## 1. Deviation form and why it helps - Starting from the sample regression function with intercept: $$Y_i = \hat{\alpha} + \hat{\beta} X_i + \hat{u}_i$$ - Taking averages gives: $$\bar{Y} = \hat{\alpha} + \hat{\beta} \bar{X}$$ - Subtracting the mean relation from the original equation yields the deviation form (no intercept): $$y_i = \hat{\beta} x_i + \hat{u}_i$$ - Benefits of the deviation form: - Simplifies algebra for slope estimation because $\sum x_i = 0$ and $\sum y_i = 0$. - Makes some sums (like $\sum x_i y_i$) equivalent to centered sums. > Definition: The deviation sums are $\sum x_i y_i = \sum (X_i-\bar{X})(Y_i-\bar{Y})$, $\sum x_i^2 = \sum (X_i-\bar{X})^2$. These express covariance and variance up to scale. ## 2. OLS slope and intercept in deviation form - Slope estimator: $$\hat{\beta} = \frac{\sum x_i y_i}{\sum x_i^2}$$ - Intercept from sample means: $$\hat{\alpha} = \bar{Y} - \hat{\beta} \bar{X}$$ - Equivalent forms useful in practice (when working with raw $X,Y$): $$\hat{\beta} = \frac{n\sum X_i Y_i - \sum X_i \sum Y_i}{n\sum X_i^2 - (\sum X_i)^2}$$$$\hat{\alpha} = \frac{\sum X_i^2\sum Y_i - \sum X_i \sum X_i Y_i}{n\sum X_i^2 - (\sum X_i)^2}$$ Practical example (price $X$, quantity $Y$ with $n=5$): given $\sum X=15$, $\sum Y=46$, $\sum X^2=55$, $\sum XY=113$, and using means $\bar{X}=3$, $\bar{Y}=9.2$ we get centered sums $\sum x_i^2=10$, $\sum x_i y_i=-25$, so $$\hat{\beta} = \frac{-25}{10} = -2.5$$ $$\hat{\alpha} = 9.2 - (-2.5)\times 3 = 16.7$$ Fun fact: The deviation-form slope $\hat{\beta}$ is unchanged whether you compute it from centered sums or from raw sums using the full-form formula; centering only reorganizes the algebra. ## 3. Decomposing total variation (TSS = ESS + RSS) - Total sum of squares (TSS): $$\text{TSS} = \sum (Y_i - \bar{Y})^2 = \sum y_i^2$$ - Explained sum of squares (ESS): $$\text{ESS} = \sum (\hat{Y}_i - \bar{Y})^2 = \sum \hat{y}_i^2$$ - Residual sum of squares (RSS): $$\text{RSS} = \sum (Y_i - \hat{Y}_i)^2 = \sum \hat{u}_i^2$$ - Identity: $$\text{TSS} = \text{ESS} + \text{RSS}$$ - Coefficient of determination: $$r^2 = \frac{\text{ESS}}{\text{TSS}} = 1 - \frac{\text{RSS}}{\text{TSS}}$$ - Alternative expressions useful for centered variables: $$r^2 = \frac{\hat{\beta}^2 \sum x_i^2}{\sum y_i^2}$$$$r^2 = \frac{(\sum x_i y_i)^2}{\sum x_i^2 \sum y_i^2}$$ > Definition: The sample correlation coefficient between centered $x$ and $y$ is $r_{xy} = \frac{\sum x_i y_i}{\sqrt{\sum x_i^2}\sqrt{\sum y_i^2}}$. Then $r_{xy}^2 = r^2$ in the simple regression case. Did you know that in simple linear regression the slope can be written as $\hat{\beta}=r_{xy}\dfrac{S_y}{S_x}$ where $S_x$, $S_y$ are sample standard deviations? This links correlation, variances and slope directly. ## 4. Residual sums and estimator for error variance - Using deviation form with $\hat{y}_i = \hat{\beta} x_i$: $$\sum y_i^2 = \hat{\beta}^2 \sum x_i^2 + \sum \hat{u}_i^2$$so $$\sum \hat{u}_i^2 = \sum y_i^2 - \hat{\beta}^2 \sum x_i^2$$ - Alternative algebraic form with raw sums: $$\sum \hat{u}_i^2 = \