Simple Regression: OLS, Assumptions, and Properties

Simple Regression is a fundamental statistical tool used to understand the relationship between two variables. It helps us model how changes in one variable, the independent variable (X), might influence another, the dependent variable (Y). This article will break down the core concepts of Simple Regression, including the Ordinary Least Squares (OLS) method, its crucial assumptions, and the desirable properties of its estimators, making it easy to grasp for students.

Understanding Simple Regression: OLS, Assumptions, and Properties

What is Simple Regression?

The simple regression model involves one dependent variable (Y) and one independent variable (X). The underlying idea is that variations in Y are caused by changes in X. For instance, consider the quantity of airtime vouchers a consumer would purchase at different prices.

In a hypothetical illustration, as the price of airtime vouchers increases (from UGX 500 to UGX 5000), the quantity demanded by a rational consumer decreases. This demonstrates the dependence of one variable (quantity demanded) on another (price), which is the essence of regression analysis.

Population Regression Function (PRF) vs. Sample Regression Function (SRF)

When studying regression, it's important to distinguish between the Population Regression Function (PRF) and the Sample Regression Function (SRF). The population refers to the entire set of observations under study, while a sample is a representative subset.

The PRF describes the true, unobservable relationship in the entire population: $$ Y = E(Y/X) + U $$ $$ Y = \alpha + \beta X + U $$ (Equation 3.12) Here, $Y$ is the dependent variable, $X$ is the independent variable, $\alpha$ is the intercept, $\beta$ is the slope coefficient, and $U$ is the disturbance term.

The SRF, on the other hand, is estimated from sample data and is used to approximate the PRF: $$ Y = \hat{Y} + \hat{u} $$ $$ Y = \hat{\alpha} + \hat{\beta} X + \hat{u} $$ (Equation 3.13) In the SRF, $\hat{\alpha}$ and $\hat{\beta}$ are the estimated coefficients, and $\hat{u}$ is the residual or estimated error term. The disturbance term $U_i$ (or error, random, statistical, stochastic, white noise term) in the simple model captures all other factors not explicitly included.

Why Do We Include the Error Term?

The error term is a critical component of any regression model, absorbing the influences of factors not directly accounted for. Its inclusion is necessary for several reasons:

• Omission of relevant explanatory variables: Many real-world phenomena are influenced by myriad factors, some of which may be qualitative (e.g., tastes, expectations), unquantifiable, or unknown to the researcher.
• Incorrect model specification: Economic relationships are often complex and might be incorrectly simplified (e.g., linearized when non-linear).
• Aggregation of variables: When individual behaviors (with diverse parameters) are aggregated into macroscopic variables (e.g., aggregate consumption), individual peculiarities are lost.
• Randomness of human behavior: Human actions are inherently unpredictable and introduce variability that the error term reflects.
• Errors of measurement: Inaccuracies during data collection, compilation, and analysis (e.g., rounding off) lead to inexact estimations, necessitating the error term.
• Need for model parsimony: To keep models concise, the error term represents all excluded variables.

Deterministic vs. Stochastic Relationships

Understanding the nature of the relationship between variables is crucial:

• Deterministic Relationship: An exact mathematical relationship where both dependent and independent variables are non-random. If you have all the information, you can determine the outcome with 100% certainty. For example, the conversion of Celsius to Fahrenheit ($F = \frac{9}{5} C + 32$) is a deterministic relationship where all points lie perfectly on a line.
• Stochastic Relationship: An inexact relationship that combines both deterministic and random components. Most real-world occurrences, like weather patterns, exhibit both predictability and randomness. In a stochastic relationship, a trend exists, but points scatter around the regression line due to errors or inherent randomness.

Causation vs. Regression Analysis

It's important not to confuse regression with causation:

• Causation (Causality): Implies that one variable directly influences another, creating a cause-effect relationship. For example, rain causes a researcher to get wet. This is a unidirectional relationship.
• Regression: Focuses on the dependence among variables within a model. It helps estimate and predict the average value of a dependent variable based on known values of independent variables. However, regression does not always imply causation.

Types of Variables in Regression

In regression analysis, specific terminology is used for the variables:

The Ordinary Least Squares (OLS) Method Explained

Ordinary Least Squares (OLS) is a widely used econometric estimation method. Its primary goal is to find the values of $\hat{\alpha}$ (intercept) and $\hat{\beta}$ (slope coefficient) that best fit the data.

How OLS Works

The OLS method minimizes the sum of the squares of the residuals (RSS). A residual is the difference between the actual observed value of Y ($Y_i$) and the estimated predicted value of Y ($\hat{Y}_i$) for each observation. Mathematically, $\hat{u}_i = Y_i - \hat{Y}_i = Y_i - \hat{\alpha} - \hat{\beta} X_i$.

• Minimizing Residuals: We cannot simply minimize the sum of residuals ($\sum \hat{u}i$) because positive and negative residuals would cancel out, often resulting in a sum of zero, regardless of how well the line fits. Instead, OLS minimizes the Residual Sum of Squares (RSS): $RSS = \sum{i=1}^{n} \hat{u}i^2 = \sum{i=1}^{n} (Y_i - \hat{\alpha} - \hat{\beta} X_i)^2$.
• Normal Equations: To find $\hat{\alpha}$ and $\hat{\beta}$ that minimize RSS, we take partial derivatives of the RSS function with respect to $\hat{\alpha}$ and $\hat{\beta}$ and set them to zero. This yields the normal equations:
• $\sum Y_i = n \hat{\alpha} + \hat{\beta} \sum X_i$ (Equation 3.17)
• $\sum X_i Y_i = \hat{\alpha} \sum X_i + \hat{\beta} \sum X_i^2$ (Equation 3.18)

Solving for OLS Coefficients: Formulas and Methods

These normal equations can be solved using various methods, such as matrix inversion or Cramer's rule, to obtain the OLS estimators for $\hat{\alpha}$ and $\hat{\beta}$.

Formulas for $\hat{\alpha}$ and $\hat{\beta}$:

• $\hat{\alpha} = \frac{\sum X_i^2 \sum Y_i - \sum X_i \sum X_i Y_i}{n \sum X_i^2 - (\sum X_i)^2}$ (Equation 3.21)
• $\hat{\beta} = \frac{n \sum X_i Y_i - \sum X_i \sum Y_i}{n \sum X_i^2 - (\sum X_i)^2}$ (Equation 3.22)

Deviation Form Formulas: Often, calculations are simplified using mean deviations, where $x_i = X_i - \bar{X}$ and $y_i = Y_i - \bar{Y}$.

• $\hat{\beta} = \frac{\sum x_i y_i}{\sum x_i^2}$
• $\hat{\alpha} = \bar{Y} - \hat{\beta} \bar{X}$

Properties of OLS Residuals

OLS residuals have specific properties:

• The sum of the residuals is zero ($\sum \hat{u}_i = 0$). This means the mean of the error term is zero ($E[\hat{u}_i] = 0$).
• The estimated Y values ($\hat{y}_i$) and the residuals ($\hat{u}_i$) are uncorrelated ($\sum \hat{y}_i \hat{u}_i = 0$).
• The explanatory variables ($\hat{x}_i$) and the residuals ($\hat{u}_i$) are uncorrelated ($\sum x_i \hat{u}_i = 0$).

Assumptions of the Classical Linear Regression Model (CLRM)

For OLS estimators to have desirable properties, the Classical Linear Regression Model (CLRM) relies on several key assumptions. These assumptions allow us to draw reliable conclusions about the true population function from the sample regression function.

Assumptions Specifically on the Error Term ($U_i$)

1. Random Variable: The error term $u_i$ is a real random variable, meaning it occurs by chance and can take positive, negative, or zero values following a probability distribution.
2. Zero Mean: Given the values of $X$, the mean or expected value of the error term is zero: $E[u_i | X] = 0$.
3. No Autocorrelation: Error terms belonging to different observations are uncorrelated: $cov(u_i, u_j) = 0$ for all $i \neq j$. There is no serial correlation.
4. Homoscedasticity (Constant Variance): The variance of each error term, given $X$, is constant: $Var(u_i | X) = E(u^2) = \sigma^2$ for all $i$. If the variance is unequal ($ \sigma_i^2$), it's called heteroscedasticity.
5. Uncorrelated with Explanatory Variables (Exogeneity): The error term $u_i$ and the explanatory variables $X_i$ are uncorrelated: $E(X_i u_i) = 0$.
6. Normality: The error term $u_i$ is normally distributed with zero mean and constant variance: $u_i \sim N(0, \sigma^2)$. This assumption is particularly important for small sample sizes.
7. Accurately Measured Explanatory Variables: The explanatory variables are measured without error, and the disturbance term absorbs all measurement errors.

Other Assumptions of the CLRM

8. No Perfect Multicollinearity: There is no perfect linear relationship among the explanatory variables. This means the matrix of explanatory variables (X matrix) is full column rank, implying linear independence among columns.
9. Number of Observations (n) > Number of Parameters (k): The sample size must be greater than the number of parameters to be estimated (excluding the intercept).
10. Linearity in Parameters: The regression model must be linear in its parameters, though it may or may not be linear in the variables themselves. For example, $Y_i = \beta_0 + \beta_1 X_i + \varepsilon_i$ is linear in both parameters and variables, while $Y_i = \gamma_0 + \gamma_1 (1/X_i) + e_i$ is linear in parameters but not in variables.
11. Non-Stochastic Explanatory Variables (Fixed in Repeated Sampling): Explanatory variables are assumed to be fixed in repeated sampling. If they are stochastic, they are assumed to be independent of or at least uncorrelated with the error term.
12. Correct Model Specification: The regression model used is accurately specified, meaning there is no model specification bias or error.
13. Variance of Explanatory Variables is a Finite Positive Number: The explanatory variables in a given sample should not all be the same (i.e., they must have some variation).

Properties of OLS Estimators: The Gauss-Markov Theorem

An estimator is a formula used to compute a numerical value for a parameter (e.g., OLS formulas for $\hat{\alpha}$ and $\hat{\beta}$). An estimate is the actual numerical value obtained from a specific sample. The statistical properties of OLS estimators, based on the CLRM assumptions, are summarized by the Gauss-Markov Theorem.

The Gauss-Markov Theorem: BLUE Properties

The Gauss-Markov Theorem states that, given the assumptions of the CLRM, the OLS estimators are the Best Linear Unbiased Estimators (BLUE). This means they possess the following three desirable properties:

1. Linearity: The estimators ($\hat{\alpha}$ and $\hat{\beta}$) are linear functions of the dependent variable $Y_i$. We can prove $\hat{\beta} = \sum k_i Y_i$ and $\hat{\alpha} = \sum t_i Y_i$, where $k_i$ and $t_i$ are weights.
2. Unbiasedness: The expected value of the estimator is equal to the true population parameter. This means $E(\hat{\alpha}) = \alpha$ and $E(\hat{\beta}) = \beta$. An unbiased estimator has a bias of zero.
3. Minimum Variance (Efficiency): Among all linear unbiased estimators, OLS estimators have the smallest variance. This makes them the most efficient estimators. If an estimator has a smaller variance than another unbiased estimator, it is considered more efficient.

Other Desirable Properties of OLS Estimators

• Least Mean Square Error (MSE): The MSE measures the average of the squared differences between estimated and actual values. Researchers might choose an estimator with a slightly larger bias if it has a significantly smaller variance, leading to a lower MSE overall. $MSE = Var(\hat{\beta}) + (Bias(\hat{\beta}))^2$.
• Consistency: As the sample size ($n$) increases indefinitely, the estimator approaches the true population parameter. This involves two conditions:
• Asymptotic Unbiasedness: The estimator becomes more unbiased as $n \to \infty$.
• Convergence: The value of the estimator converges to the true parameter as $n \to \infty$. The variance of a consistent estimator tends to zero as the sample size grows infinitely large ($lim_{n \to \infty} var(\hat{\beta}) = 0$).

Variances and Standard Errors of OLS Estimators

After estimating the coefficients, it's crucial to assess their reliability and precision. This involves calculating their variances and standard errors.

Formulas for Variances and Standard Errors

• Variance of $\hat{\beta}$: $Var(\hat{\beta}) = \frac{\sigma^2}{\sum x_i^2}$
• Standard Error of $\hat{\beta}$: $s.e(\hat{\beta}) = \sqrt{Var(\hat{\beta})} = \sqrt{\frac{\sigma^2}{\sum x_i^2}}$
• Variance of $\hat{\alpha}$: $Var(\hat{\alpha}) = \sigma^2 \left( \frac{\bar{X}^2}{\sum x_i^2} + \frac{1}{n} \right) = \frac{\sigma^2 \sum X_i^2}{n \sum x_i^2}$
• Standard Error of $\hat{\alpha}$: $s.e(\hat{\alpha}) = \sqrt{Var(\hat{\alpha})} = \sqrt{\frac{\sigma^2 \sum X_i^2}{n \sum x_i^2}}$
• Covariance between $\hat{\alpha}$ and $\hat{\beta}$: $Cov(\hat{\alpha}, \hat{\beta}) = \frac{-\bar{X} \sigma^2}{\sum x_i^2}$

Note that $\sigma^2$ (the population variance of the error term) is usually unknown and is estimated by $\hat{\sigma}^2 = \frac{\sum \hat{u}^2}{n-2}$, where $(n-2)$ represents the degrees of freedom (number of observations minus number of estimated parameters).

Root Mean Square Error (RMSE)

Root Mean Squared Error (RMSE), also known as the standard error of the regression, is a measure of the typical size of the residuals. It's the positive square root of $\hat{\sigma}^2$: $$ RMSE = \hat{\sigma} = \sqrt{\frac{\sum \hat{u}^2}{n-2}} $$ RMSE indicates the goodness-of-fit of the predicted regression line; a smaller RMSE suggests a better fit.

Coefficient of Determination ($r^2$): Measure of Goodness-of-Fit

The coefficient of determination ($r^2$) is a statistic that tells us how well the sample regression line fits the data. It quantifies the proportion of the variance in the dependent variable (Y) that is predictable from the independent variable (X).

Understanding $r^2$

• Range: The value of $r^2$ ranges from 0 to 1 ($0 \leq r^2 \leq 1$).
• Interpretation: An $r^2$ of 1 indicates that the regression predictions perfectly fit the data, meaning all variation in Y is explained by X. An $r^2$ of 0 means X explains none of the variation in Y.
• Decomposition of Total Variation: The total variation in the dependent variable (Total Sum of Squares, TSS) can be broken down into two parts:
• Explained Sum of Squares (ESS): The variation in Y explained by the regression model (due to X).
• Residual Sum of Squares (RSS): The unexplained variation in Y (due to the error term).
• The relationship is: $TSS = ESS + RSS$
• Calculation: $r^2 = \frac{ESS}{TSS} = \frac{\sum \hat{y}_i^2}{\sum y_i^2}$. It can also be expressed as $r^2 = 1 - \frac{RSS}{TSS}$.

Relationship with the Correlation Coefficient

For a simple two-variable model, the square of the correlation coefficient ($r_{xy}$) is equal to the coefficient of determination ($r^2$). The correlation coefficient measures the magnitude and direction of the linear association between two variables, but unlike regression, it doesn't distinguish between dependent and independent variables, nor does it imply causation.

Frequently Asked Questions (FAQ)

What are the main differences between regression and correlation?

Regression analysis focuses on the dependence of one variable (dependent variable) on one or more others (independent variables) to estimate and predict mean values. It assumes fixed independent variables and stochastic dependent variables. Correlation, on the other hand, measures the symmetric linear association between two or more random variables, where no distinction is made between dependent and independent variables, and it does not imply causation.

Why is the error term normally distributed in CLRM?

The normality assumption of the error term ($u_i \sim N(0, \sigma^2)$) is crucial for statistical inference, especially when dealing with small sample sizes. It allows us to use t-tests and F-tests for hypothesis testing and construct confidence intervals for the regression coefficients. With a large sample, the Central Limit Theorem helps ensure that OLS estimators are approximately normally distributed even if the errors are not.

What does BLUE stand for in OLS properties?

BLUE stands for Best Linear Unbiased Estimator. According to the Gauss-Markov Theorem, OLS estimators possess these properties under the assumptions of the Classical Linear Regression Model (CLRM). "Best" means having the minimum variance, "Linear" means it's a linear function of the dependent variable, and "Unbiased" means its expected value equals the true population parameter.

How does multicollinearity affect OLS estimates?

Multicollinearity refers to a strong linear relationship among the independent variables. If there is perfect multicollinearity, OLS estimation becomes impossible because the normal equations cannot be solved uniquely. In cases of high (but not perfect) multicollinearity, OLS estimators remain BLUE, but their variances and standard errors become very large, leading to imprecise estimates and difficulty in determining the individual impact of each independent variable.