Regression · Part 2

Regression predicts a numerical response from one or more inputs. Begin with the observed pairs, specify a candidate mean function, choose its coefficients by least squares and evaluate its predictions. Then allow polynomial features to represent curvature.

Part 1 follows linear regression, training, model evaluation and polynomial regression. Part 2 continues with bias–variance, accuracy of estimates, multiple regression and assumptions. The notation and examples below make each step explicit.

A residual is an observed response minus the model’s prediction at the same input. RSS, the residual sum of squares, squares each training residual and adds the results. It is the total squared training error, so its value depends on the number of cases and the response units. Compare it on the same observations.

TSS, the total sum of squares, measures response variation around the mean. It is the squared error of a reference predictor that always returns that mean. It describes the data’s variation before asking how much the fitted model explains.

RSE, residual standard error, estimates the spread of the model’s unexplained random error, in response units. For a simple line it is the square root of the whole ratio RSS/(n − 2), allowing for fitting the intercept and slope. It is a training-based estimate of error spread, rather than a test-set prediction score.

R², the coefficient of determination, compares the model’s squared error with TSS. A value of 1 means no residual error, 0 means the same squared error as the mean-only baseline, and a negative value means worse than that baseline. The score requires nonzero TSS. It is not a percentage of predictions that are correct.

Test RMSE, root mean squared error, measures the size of prediction mistakes on observations not used to fit or choose the model. Square the test errors, average them and take the square root. This returns the score to response units and gives larger misses more influence. RSE describes estimated training noise; test RMSE assesses the fixed model’s prediction on new cases.

Ordinary least squares (OLS) estimates coefficients by minimizing RSS. Use validation to compare model choices and a final test set to assess the selected procedure. Underfitting misses a stable pattern, while overfitting follows training details that do not transfer. Repeating training on new samples lets us study model variability.

The RSE formula uses $q$ for the number of independently fitted coefficients, including the intercept. For a straight line $q=2$, so the denominator is $n-2$. For a polynomial of degree $p$ with independent feature columns, $q=p+1$. RSE requires positive residual degrees of freedom, $n>q$. The R² ratio requires $\text{TSS}>0$.

In the test-RMSE row, the observed responses and predictions belong to the held-out test set, and $n_{\text{test}}$ is its number of cases. The residuals in RSS and the mean in TSS describe the training observations in this recap.

Bias is the systematic difference between the average learned relationship and the true relationship. At a fixed input, compare the average fitted prediction across repeated training datasets with the true mean response. For example, a straight-line model cannot capture a genuinely quadratic mean over the input range, so its average fit can miss that pattern.

Variance describes how much the learned model changes across possible training datasets sampled from the same population, while keeping the fitting procedure and sample size fixed. At one input, this is the spread of predictions across those fitted models. It describes variation across repeated training, rather than the noise of one fresh observation. A model can have low variance and still be consistently off target.

The orange mark is the true mean, the teal mark is the average fitted prediction and the purple dots are individual fitted predictions. In real data we do not know the true mean; a simulation makes this comparison visible.

New observation: $y$ is an outcome not used for training. Its $\epsilon$ is the random difference from the true mean at $x$. For example, another crop trial with the same fertilizer dose can give a different yield. “Fresh” simply means new: this observation was not used for training. The noise $\epsilon$ is the observation’s deviation from the true mean, whereas a residual compares an observed value with a fitted model prediction.

Keep $x$ fixed and imagine training the same procedure on possible datasets $𝒟$ from the same population. The notation $f(x;𝒟)$ explicitly means the prediction at $x$ after training on $𝒟$. The semicolon separates the prediction input from the dataset used for learning.

${𝔼}_{𝒟}$ averages over those possible training datasets. The true mean response at this fixed input is ${𝔼}_{\epsilon }\left[y\right]$: average the new, unseen response $y$ over its random error $\epsilon$. Bias is the average learned prediction minus that true mean. It can be positive or negative; the error decomposition uses its square.

Prediction variance measures how fitted predictions spread around their own average across training datasets. Its full expectation form is:

A less flexible family can miss part of the true relationship while moving less across samples. A more flexible family can capture the relationship better while following sample-specific noise. These are tendencies, not a guarantee for every dataset. Bias here is unrelated to the intercept sometimes being called a bias term.

At a fixed input $x$, consider a new observation $y$ and the prediction $f(x;𝒟)$ learned from a random training dataset $𝒟$. The expectation ${𝔼}_{𝒟,\epsilon }$ averages over both repeated training and noise in a new observation. This is expected mean squared error (MSE), rather than the error of just one observed case.

New observation: $y$ is an outcome not used for training. Its $\epsilon$ is the random difference from the true mean at $x$. For example, another crop trial with the same fertilizer dose can give a different yield. This random error belongs to the new observation and is independent of the dataset used for training. It is not a training residual. Both sides of the displayed identity give the expected MSE: the right-hand side splits it into three contributions. MSE is the quantity inside the RMSE square root. Here the expectation is at one fixed input, rather than an empirical average over a finite test set.

The expected squared prediction error decomposes into the squared bias ${\left[\text{Bias}\left(f(x;𝒟)\right)\right]}^2$, prediction variance $\text{Var}\left(f(x;𝒟)\right)$ and noise variance $\text{Var}\left(\epsilon \right)$. The new observation’s random error $\epsilon$ is independent of the training dataset, has mean ${𝔼}_{\epsilon }\left[\epsilon \right]=0$ and finite variance. Bias compares the average prediction with ${𝔼}_{\epsilon }\left[y\right]$, the true mean response.

The simulation uses noise standard deviation 0.3, so $\text{Var}\left(\epsilon \right)=0.09$. Its known generating curve makes bias and variance measurable across repeated fits. They cannot generally be recovered exactly from one real dataset. MSE has squared response units; taking its square root gives RMSE.

Irreducible noise is relative to the information in the selected inputs. More relevant measured predictors can change that information. Use training data for fitting, validation for model choices, and the final test set for assessment.

The sum on the right is the expected Mean Squared Error (MSE).
MSE is RMSE without the square root: $\text{RMSE}=\sqrt{\text{MSE}}$.

The simulation knows the true mean curve ${𝔼}_{\epsilon }\left[y\right]=2+1.2x-0.9x^2$. It adds independent normal noise with standard deviation 0.3. It fits the chosen degree to 60 independently noisy training samples, each with the selected number of cases. The sample-set control selects a reproducible group of samples.

Each fitted curve represents $f(x;𝒟)$ for one training dataset. The chart draws 15 of these curves, their 60-model average and the true mean curve. At the selected input, all 60 predictions appear as purple dots. The average miss gives bias, and the spread gives prediction variance. The scores are squared bias, prediction variance, noise variance 0.09 and their sum: an approximation to expected test MSE at that input.

Try a constant model, the correctly shaped quadratic, and a flexible degree-9 model. Then change training sample size. The exercise concerns repeated training, not changing the test outcome or knowing the true function in a real application.

The blue dotted line is only a guide to the selected input. The purple dots are the predictions from the 60 fitted models. The simulation knows the true curve. With real data, compare validation error because the true bias and variance are usually unknown.

Try it: Compare degrees 0, 2 and 9. Purple dots show 60 fitted predictions. The blue dotted line marks the selected input x.

The diagram uses three disjoint subsets. The training set supplies the observations used to estimate coefficients. Both candidate degrees are fitted using those training cases. The validation set contains different observations: compare the candidates’ predictions with those targets. In this illustrative example, degree 2 has validation RMSE 0.32, lower than degree 9’s 0.58, so we select degree 2. These numbers illustrate the decision and are not measurements from the interactive simulation.

Only after fixing the procedure do we use the untouched test set. Its illustrative RMSE of 0.35 estimates the chosen model’s performance on new cases. Validation and test RMSE need not match because they use different observations. The test score does not choose the degree. Refitting the selected procedure using training and validation data before final testing is also possible, provided all choices are already fixed.

Fit each candidate on training observations and compare validation error or cross-validation estimates. Choose the complete procedure, including degree and learned transformations, before using the final test set. Keep preprocessing within the training portion of each split.

Validation supplies evidence for choices; testing supplies a final assessment of those choices. If test results guide another choice, they no longer provide an untouched final assessment. Model complexity, uncertainty of estimates and prediction error answer related but distinct questions.

The 90 observations have an estimated slope of about 0.1075. A nonzero estimate by itself does not demonstrate a nonzero population slope, because the responses include random error. We ask how unusual the observed result would be under a zero-slope model.

The following slides introduce the null and alternative hypotheses, distinguish response noise from uncertainty of the slope, and construct a t statistic and p-value. The interactive worked example comes after those quantities have been defined.

One sample gives one estimate

The fitted slope here is about 0.108. Another sample could give a different value.

The question: would a slope this far from zero be unusual* if the true population slope were zero?

A fitted number alone does not tell us how precise it is. We need its sampling uncertainty.

*Here, “unusual” means “unlikely to arise from sampling noise if the true population slope were zero.”

A fitted slope can be nonzero just because the sample is noisy. We state two hypotheses to check whether the data provide evidence of a linear association beyond this sampling variation. Before looking at a p-value, we specify the claim to test and the competing possibility. This makes the evidence question precise: how unusual would the observed estimate be under the zero-slope hypothesis? The test does not decide that either hypothesis is certainly true.

The null hypothesis H₀: ${\beta }_1$ = 0 says that the population linear mean has zero slope. The two-sided alternative H₁: ${\beta }_1$ ≠ 0 allows either a positive or negative slope. A fitted slope will rarely be exactly zero even if the null is true, because estimates vary across samples.

Compare the estimate with its sampling uncertainty instead of comparing it with zero alone. Rejecting this null supplies evidence of a linear association under the assumptions. It does not establish causality. Failure to reject does not prove no relationship, especially if precision is low or the true relationship is nonlinear.

The small plots show illustrative population mean functions, not fitted samples. Under the null, the mean is flat even though individual observations may still be noisy. Under the two-sided alternative, the slope may be positive or negative: the two colored lines show separate possibilities, not two simultaneous fitted lines.

Null hypothesis

$$H_0:{\beta }_1=0$$

The population slope is zero. A nonzero fitted slope can arise from sampling noise.

Two-sided alternative

$$H_1:{\beta }_1\ne 0$$

The population slope is positive or negative. The data may support a linear association.

Rejecting H₀ does not establish causation. Failing to reject does not establish no relationship.

The two panels answer different questions. The left panel uses one dataset and one fitted line. Each blue dot is an observed response. Each purple segment is a residual, the observed response minus the fitted prediction at the same input. RSE summarizes the estimated scatter around that line, in response units.

For the right panel, imagine repeating the study on new observations from the same population, while keeping the sample size, input values and fitting procedure fixed. Each dataset gives its own fitted intercept and slope. The upper plot shows three of these fitted lines. The matching colored dots on the lower numbered axis show their slope estimates. The other dots show the remaining estimates: one dot for each of 20 simulated datasets. Horizontal spread means the fitted slopes change across studies. The slope standard error describes the standard deviation of the slope estimate across such repetitions. In real data we estimate it from one sample using the formula introduced next. It has response units per input unit.

These teaching simulations use a mean response of 1 + 2x with independent normal errors. The left plot shows the first of the 20 simulated datasets. Each dataset contains 16 observations at the same inputs, with noise standard deviation 0.6. The lower axis uses all 20 slope estimates. The upper plot shows only three fitted lines to keep the comparison readable. A smaller slope SE means a more precise estimate. It does not by itself establish a large association or a causal effect.

RSE · Residual Standard Error

How scattered are the responses?

Use one dataset. RSE estimates the size of the random scatter around its fitted line.

Units: response units, such as kg.

Slope SE · Standard Error of the Slope

How much does the fitted slope change?

Repeat training on new datasets. Slope SE describes how much the slope estimates vary.

Units: response units per input unit.

The second block is a theoretical standardization, not an immediately computable sample statistic. Both the true population slope ${\beta }_1$ and the true sampling standard deviation $\sigma$ are unknown in ordinary observed data. The sample gives their estimates, not these population values themselves. The standardized expression can be evaluated only when the true values are supplied, for example by a simulation. It describes the sampling theory that motivates a practical test after a null slope is proposed and the standard error is estimated.

Imagine repeating the same simple-regression study while keeping its input values, sample size and fitting procedure fixed. The population slope ${\beta }_1$ is a fixed unknown number. Each new set of noisy responses gives a different fitted slope ${\hat{\beta }}_1$. Under the classical normal-error assumptions, the distribution of these estimates is $𝒩({\beta }_1,{\sigma }^2)$. Here $\sigma$ denotes the true sampling standard deviation of the slope estimate, also called its true standard error.

Subtracting ${\beta }_1$ centers the distribution at zero. Dividing by $\sigma$ measures distance in standard-error units. A value of +1 means one standard error above the true slope, and −1 means one standard error below it. The plots use an illustrative true slope 2 and true standard error 0.4. A fitted slope 2.4 therefore becomes (2.4 − 2) / 0.4 = 1. These illustrative values are separate from the 90-observation example.

Distinguish the slope standard error $\sigma$ from the response-error standard deviation ${\sigma }_{\epsilon }$. Their exact relationship in simple regression is:

$n$ is the number of observations, $x_i$ is input i, and $\overline{x}$ is the input mean. Equivalently, the slope variance is the response-error variance divided by the centered input sum of squares. Exact under a correct linear mean, varying fixed inputs, and independent normal errors with common variance. The response errors have conditional mean zero, their variance is finite and common across cases, the inputs are not all equal, and inference requires n > 2. In practice $\sigma$ is unknown. We estimate it on the next slide.

1 · Before: fitted slopes vary around the truth

$${\hat{\beta }}_1\sim 𝒩({\beta }_1,{\sigma }^2)$$

Center = ${\beta }_1$. Spread = $\sigma$.

2 · Theoretical standardization

$$\frac{{\hat{\beta }}_1-{\beta }_1}{\sigma }\sim 𝒩(0,1)$$

Center = 0, spread = 1.
Computing this needs the true ${\beta }_1$ and $\sigma$.

This theoretical quantity cannot be computed from the sample alone. A simulation can supply the true values. Next, estimate $\sigma$ from data.

The true standard deviation of the fitted slope across repeated samples is $\sigma$. We cannot observe it directly, so we calculate its estimate $\hat{\sigma }$ from one dataset. This is the same quantity often written $\text{SE}\left({\hat{\beta }}_1\right)$. It describes uncertainty in the estimated slope, not the scatter of a single response.

The first formula has a rise-over-run intuition. RSE is vertical response noise, measured in response units. The square root of the centered input sum of squares measures horizontal input spread. Dividing converts the estimated vertical noise into uncertainty about the line’s slope, in response units per input unit. This is an intuition for the slope standard-error formula, rather than the ordinary rise-over-run calculation of the fitted slope.

The hat in $\hat{\sigma }$ means that the standard error is estimated. It is not ${\sigma }^2$, and it is not the realized estimation error ${\hat{\beta }}_1-{\beta }_1$ of this particular fitted slope. It estimates the standard deviation of fitted slopes across hypothetical repetitions of the same model and sampling conditions. Smaller values mean less sampling variability in estimating the true population slope.

First estimate the response-error standard deviation ${\sigma }_{\epsilon }$ with RSE. Then divide by the square root of the centered input sum of squares. If the observed inputs cover a wider range, the data provide more information about the tilt of a line.

Next substitute RSE² = RSS / (n − 2). The denominator n − 2 reflects that the intercept and slope were estimated from the same observations. Finally, expand RSS as the sum of squared residuals. The three displayed formulas are equal, rather than three alternative methods.

The sum over i runs over all n cases. For case i, $x_i$ is the observed input, $y_i$ the observed response, and ${\hat{y}}_i$ its fitted prediction. $\overline{x}$ is the mean input. Larger response noise increases the estimated slope uncertainty, while larger input spread or more observations can reduce it, all else equal. The residual degrees of freedom must be positive, n > 2, and the inputs must vary.

More response noise increases slope uncertainty. More cases or more input spread reduce it, all else equal. Requires n > 2 and varying inputs.

This introduces a new, illustrative height–weight dataset before discussing its fitted slope. There are 16 simulated adults, with irregular heights from 155 to 195 cm as the input and weights in kilograms as the response. The first sample's weights range from about 61 to 91 kg. Each plotted dot is one paired observation. All 16 pairs appear in the plot, while the table lists four rounded examples. These are teaching data rather than a real survey.

The observations are exactly those in the less-scattered sample of the later side-by-side comparison. OLS on the full, unrounded observations gives slope 0.60 kg per cm and estimated slope standard error 0.06 kg per cm. This introductory plot shows observations only. The following slide introduces the fitted slope, and the comparison then shows its fitted line alongside a second, more variable sample using exactly the same height values.

Each dot is one person

What have we measured?

Input: height, measured in centimetres.
Response: weight, measured in kilograms.

Four of the 16 observations:

Height (cm)	Weight (kg)
155	61.3
168	74.6
180	76.0
195	91.2

Illustrative simulated data. The plot shows all 16 observations. Table values are rounded.

First look at the observations. Next, fit a line and ask how much evidence its upward slope gives.

The preceding slide introduced 16 simulated adult height–weight observations. Fitting a line to this sample gives slope 0.6 kg per cm. An extra centimetre of height is therefore associated with an extra 0.6 kg in the fitted mean weight, not an exact 0.6 kg difference between every pair of people. An upward sample line alone does not tell us whether the population slope is nonzero: sampling noise can produce a nonzero fitted slope even under a flat population mean.

The slope test temporarily assumes the zero-slope null $H_0:{\beta }_1=0$. We imagine repeating the study under that model and the same sampling setup, then ask how unusual a fitted result at least as far from zero as the observed one would be. If it would be very unusual under the zero-slope model, the data provide evidence against that model under the regression assumptions.

This is a conditional assessment of the observed result. It does not calculate the probability that the population slope is zero, and the hypothetical repetitions are a reference model rather than extra data we must collect. The next comparison holds the fitted slope, sample size and inputs fixed while changing residual noise, showing why uncertainty must enter the comparison. Evidence of a linear association does not establish causation.

We assess the observed result assuming a zero slope. We do not calculate the probability that the true slope is zero. A linear association does not establish causation.

This visual follows the same three steps as the preceding slide. First assume that mean weight does not vary with height: the population slope is zero. The horizontal teal line shows that assumed mean, not a claim that every person's weight is identical. The flat level is the mean response in the 16-observation example.

Second imagine new responses at the same 16 height values. For this illustration, generate 500 hypothetical samples by adding independent normal response noise to that flat mean. Its standard deviation is set to the RSE estimated from the less-scattered height–weight sample. Fit an intercept and slope separately to every generated sample. The middle panel shows five of those fitted lines, spanning negative, near-zero and positive estimates. The dashed teal line remains the same assumed population mean.

Third collect all 500 fitted slopes into a histogram. Each purple bar counts hypothetical samples in a slope interval, rather than people. The orange mark is the slope +0.60 kg/cm computed from our original, non-flat-looking sample. It is much farther from zero than the simulated reference cluster. The histogram includes both negative and positive fitted slopes because noise can tilt either way.

This is a visual, plug-in noise simulation to explain the testing logic. It does not calculate an exact p-value or the probability that the null hypothesis is true. A finite simulation with no result as extreme as the observed one does not mean the true tail probability is zero. The formal test introduced later uses a t statistic and Student's t reference, accounting for estimation of the noise scale. Exact inference requires the stated regression assumptions. The following comparison shows why the same observed slope can provide different evidence when its uncertainty changes.

1 · Assume a flat population line

The true slope is temporarily set to zero. Individual responses can still be noisy.

2 · Imagine repeated samples

Keep the same 16 heights. Fit a new line to each hypothetical sample. Five of 500 fits are shown.

3 · Compare with our result

Purple bars summarize all 500 fitted slopes. Our +0.60 kg/cm lies far from their cluster.

Both plots use the same 16 irregular adult height values, from 155 to 195 cm, and the same fitted slope of 0.60 kg per cm. They share the same fitted intercept, height coordinates, observation count and axis scales. The fitted line predicts 75 kg at 175 cm. Only the residual scatter differs. These are illustrative simulated observations, not a real survey.

The residuals are centered and orthogonal to centered height, so scaling them preserves the fitted slope and intercept. Their scales give slope standard errors 0.06 and 0.30 kg per cm. Recomputing OLS from the displayed coordinates gives t = 0.60 / 0.06 = 10 for the less-scattered sample and t = 0.60 / 0.30 = 2 for the more-scattered sample. The two panels illustrate separate samples at matching heights, not weight changes of the same people.

The estimated response scatter RSE is approximately 2.8206 kg in the first sample and 14.1032 kg in the second. Their weight ranges are about 61–91 kg and 55–108 kg. The larger residual scatter increases slope uncertainty. Dividing the same fitted slope by its larger estimated SE gives a smaller positive t. Slope and slope SE both have kg/cm units, which cancel in the t ratio.

Slope SE also depends on sample size and input spread, both deliberately held fixed here. The next slide gives the t-statistic formula. This comparison illustrates evidence against a zero slope under the regression assumptions, not causation.

Less vertical scatter

$${\hat{\beta }}_1=0.60\text{kg/cm},\hat{\sigma }=0.06\text{kg/cm},t=10$$

Less vertical noise gives a smaller slope SE. Dividing the same fitted slope by this smaller SE gives a larger positive t.

More vertical scatter

$${\hat{\beta }}_1=0.60\text{kg/cm},\hat{\sigma }=0.30\text{kg/cm},t=2$$

More vertical noise gives a larger slope SE. Dividing the same fitted slope by this larger SE gives a smaller positive t.

Slope SE also depends on sample size and horizontal input spread. Both are held fixed here, so this comparison isolates residual scatter. The t value is unitless.

Zero is subtracted because it is the population slope proposed by the null hypothesis. We temporarily assume the true slope is zero and assess how unusual the observed fitted slope would be under that assumption, relative to its estimated standard error. The numerator ${\hat{\beta }}_1-0$ is the signed distance between the slope fitted to our data and the zero slope being tested. It is positive above zero and negative below zero. Its absolute value gives distance regardless of direction.

The t value tells us how much the observed slope from our data deviates from a hypothetical zero slope, measured in estimated standard errors. The observed value $t_{\text{observed}}$ is fixed after fitting our dataset. Across hypothetical repetitions under the zero-slope null, the same calculation is random and is denoted $T$. This test does not assign a probability to the true population slope being zero.

The null hypothesis is $H_0$: ${\beta }_1=0$. Compare the fitted slope with that null value, then divide by its estimated standard error $\hat{\sigma }$. The observed ratio $t$ counts how many estimated standard errors the slope is from zero. Its sign indicates the direction, while its absolute value measures distance.

If the null and the classical assumptions hold, the random ratio from repeating the entire study follows Student’s t distribution with n − 2 degrees of freedom. Before data are observed this is a random quantity. Once a dataset is fitted, t is one realized value. We use the distribution to judge that realized value. Student’s t is symmetric and centered at zero, with heavier tails than the normal distribution. Estimating the unknown spread introduces the extra uncertainty that this reference distribution accounts for.

For the 90 displayed observations, the slope is 0.1075379991 and its estimated standard error is 0.0179471258. Their ratio is 5.991934335, on 88 degrees of freedom. The number 5.99 is a distance in standard-error units, not a probability. A large absolute t is evidence to investigate against the zero-slope null, not proof of a causal effect. Exact under a correct linear mean, varying fixed inputs, and independent normal errors with common variance.

Use the controls to change the fitted slope or its estimated standard error independently. Hold the slope fixed and increase the standard error: the absolute t statistic becomes smaller. Near zero and Far from zero provide simple examples. The 90-point button restores the values calculated from the supplied data. These illustration controls change the displayed estimate and uncertainty directly. The data-generation example later shows how new observations produce such changes.

The t value measures deviation from a hypothetical zero slope in estimated standard errors. It does not give the probability that the true slope is zero.

Assuming the true slope is zero, the p-value is the probability that a hypothetical repetition of the study gives a t value at least as far from zero as the t value we computed from our original data. These are hypothetical repetitions under the same model and sampling setup, not additional data we must collect. Compare absolute t values, rather than raw slopes.

The two quantities use the same t-statistic calculation but refer to different datasets. $T$ is a random statistic across hypothetical repeated samples under the zero-slope null. $t_{\text{observed}}$ is the fixed number already calculated from our actual dataset. The probability expression asks how often $\left|T\right|$, a random distance, is at least $\left|t_{\text{observed}}\right|$, our observed distance. The vertical bar means “assuming” or “given” $H_0$.

Absolute value means distance from zero regardless of sign. In the illustrative case $t_{\text{observed}}=2.2$, the outcomes counted are $T\le \mathrm{-2.2}$ or $T\ge 2.2$. An equally distant negative result is just as extreme as a positive one. “More extreme” does not mean a larger raw slope without regard to its standard error: it means a t statistic at least as far from zero.

A smaller p-value means that, if the true line were flat, a result like ours or more extreme would be rarer. Thus a small p-value is stronger evidence against a flat population line, under the regression assumptions. It does not give the probability that the population slope is zero, measure effect size, or establish causation. The p in p-value is unrelated to polynomial degree p.

If $t_{\text{observed}}=0$ and the estimated slope standard error is positive, the fitted slope is zero and the fitted line is flat. Every possible absolute random t value is at least zero, so the two-sided p-value is 1. This gives no evidence against the zero-slope null; it does not prove the population slope is zero.

Choose a significance threshold $\alpha$ before inspecting the result. The often-used value 0.05 is a convention whose appropriateness depends on the task and field. Reject the null when the p-value is below that threshold. Otherwise there is insufficient evidence to reject it, rather than proof of no relationship. A nonlinear relationship is not ruled out by failure to reject a zero linear slope.

The following interactive slide explores the same ideas. At observed t = 2.2 with 88 degrees of freedom, the two-sided p-value is approximately 0.0304. This is distinct from the supplied 90-observation fit, whose t is approximately 5.99193 and whose p-value is approximately 4.445 × 10⁻⁸. The null reference distribution and interpretation require the stated simple-regression assumptions: a correct linear mean, varying fixed inputs, and independent normal errors with common variance and mean zero.

Smaller p means stronger evidence against a flat population line. “More extreme” means farther from zero in t units, not a larger raw slope. It is not the probability that the null is true.

The curve is the Student t reference under the zero-slope population model. The control n is the number of observations in each hypothetical study. The calculation uses n − 2 degrees of freedom internally for a simple regression with an intercept and slope. Changing n updates the reference curve while holding the selected observed t fixed. It does not generate or refit a new dataset.

Start with t = 0: every absolute reference value is at least zero, so the two-sided p-value is 1. At observed t = 2.2 and n = 90 observations, the probability in both tails beyond −2.2 and +2.2 is approximately 0.0304. Reversing the sign of t leaves the two-sided p-value unchanged. Larger absolute t values leave less probability in the tails.

The decision compares the unrounded p-value with the selected significance threshold α. Show REJECT in red when p < α, meaning reject the zero-slope null. Show DON'T REJECT in yellow when p ≥ α, meaning there is insufficient evidence to reject that null. Equality belongs to DON'T REJECT. A failure to reject does not prove that the population slope is zero, and rejection does not establish causation. Changing α changes the decision threshold without changing the p-value.

The 90-point result button loads the fitted statistic from the supplied dataset and sets n to 90. Its observed t is approximately 5.99193 and its two-sided p-value approximately 4.445 × 10⁻⁸. This is a separate dataset from the 16-observation height–weight illustration. The default t = 2.2 is an independent illustration, not a fit to those 90 points.

The shaded probability includes tails beyond the plotted limits. Curve height is density, while shaded area represents probability. The reference and its interpretation require a correct linear mean, varying inputs, and independent normal errors with common variance and mean zero. These are hypothetical zero-slope repetitions, not additional data that must be collected.

The curve describes repeated studies under $H_0$. A larger distance from zero leaves less probability in the two tails.

Try it: Change the observed t, n or threshold α. Try t = 0, switch its sign, or load the 90-point result. Watch the decision change as p crosses α.

All quantities in this example are calculated from the same 90 plotted observations. First OLS fits an intercept and slope, choosing the coefficients that minimize RSS. The estimated slope is ${\hat{\beta }}_1\approx 0.107538$. This is the estimated change in mean response per unit of input.

Second, calculate the response scatter RSE ≈ 0.0496994 and use the centered input spread to obtain $\hat{\sigma }\approx 0.0179471$. RSE and the slope standard error have different roles and units. Third, test the zero-slope null by dividing the fitted slope by its estimated standard error: t ≈ 0.107538 / 0.0179471 ≈ 5.99193. The estimate is almost six standard errors above zero.

Fourth, use Student t with n − 2 = 90 − 2 = 88 degrees of freedom. The two-sided probability beyond ±5.99193 is about 4.445 × 10⁻⁸. At α = 0.05 this rejects the zero-slope null. The interpretation is evidence of a linear association under the assumptions, not proof of a practically large or causal effect.

Hiding the line only changes the picture, not the observations or calculated statistics. Zoom, pan and reset help inspect the scatter. The supplied dataset has the p-value reported above. The interpretation assumes a correct linear mean, varying inputs, and independent normal errors with common variance.

The data control begins with the supplied 90 observations and the same fitted statistics as the worked example. Choose Simulate new data or New simulated sample to explore a known linear population. The generator is $y_i=0.5+{\beta }_1{x}_i+{\epsilon }_i$, with independent normal errors of mean zero and the selected response-noise standard deviation. Inputs are evenly spaced around 0.5, over the selected narrow or wide range. A sample set selects a reproducible noise realization. The new-sample button changes that realization.

Vary one feature at a time: use fewer cases, increase noise, narrow the input spread, or set the true slope to zero. Each change refits OLS and recalculates the slope, its estimated standard error, t, the two-sided p-value, training RSE and RSS. Sampling variability means a nonzero population slope can fail to give a significant sample result, and a zero population slope can occasionally produce a false positive.

Compare OLS with the mean-only fit, which always predicts the observed response average and has zero fitted slope. Its RSS is TSS. Both fits use the same observations, and the displayed slope-test statistics continue to describe the OLS fit. The dashed generating mean is shown only for simulations, where that mean is known.

A tiny p-value supports a linear association under the model assumptions. The study alone does not establish causation.

The interpretation now spells out the reason for the decision in three linked steps. First divide the estimated slope minus the tested zero by its estimated slope standard error to calculate t. Its absolute value is the distance from zero in estimated slope SEs. Next compare the two-sided p-value with α = 0.05. Show REJECT if p < α and DON'T REJECT if p ≥ α. Finally explain whether a t value this far from zero would be rare under a flat population mean. A failure to reject is insufficient evidence against zero, not proof that the slope is zero. The displayed formulas are rounded for reading, while decisions use the unrounded p-value. The calculation and explanation update together when data settings change.

Try it: Start with the supplied data. Then simulate new samples: change their size, noise, true slope or input spread. Compare the OLS line with a mean-only fit.

Optional reading: this extends the slope test with confidence intervals. It is retained here as additional explanation and is not a separate slide in the main lecture.

A fitted slope is a point estimate. A confidence interval adds a sampling-uncertainty margin around that estimate. For a two-sided 95% interval, multiply the estimated slope standard error $\hat{\sigma }$ by the Student t critical value $t_{0.975,n-2}$, then subtract and add the resulting margin to ${\hat{\beta }}_1$.

The critical value is the 97.5th percentile: 97.5% of the reference probability is to its left and 2.5% to its right. Symmetry leaves another 2.5% below the negative critical value, so 95% lies between the two cutoffs. For n = 90 the degrees of freedom are 88 and the multiplier is about 1.987. Thus 0.107538 ± 1.987 × 0.0179471 gives approximately [0.0719, 0.1432].

The blue point is the fitted slope and the teal segment is the confidence interval. Zero lies outside it. Under the same assumptions, excluding zero agrees with rejecting the two-sided zero-slope test at α = 0.05. An interval that includes zero would not provide evidence to reject that null at this threshold.

The 95% refers to coverage across repeated studies: about 95% of intervals constructed by this procedure would contain the fixed true slope. It is not a 95% posterior probability that this already observed interval contains the slope, and it does not mean 95% of future responses fall in this interval. It concerns a population coefficient, not an individual future response. It also does not establish causality. Exact under a correct linear mean, varying fixed inputs, and independent normal errors with common variance.

The interactive example starts with the 95% interval calculated from the 90 supplied observations. Select 20 repeated studies, or press Repeat 20 studies, to generate 20 independent datasets with a known population slope of 0.10 and intercept 0.5. Each study uses the selected sample size and independent normal response errors with the selected standard deviation. Inputs are evenly spaced from 0 to 1. OLS and its standard error are recalculated separately for every dataset.

Each horizontal segment is one 95% confidence interval, with its fitted slope marked. The orange vertical line is the known true slope. Teal intervals cover that value and red intervals miss it. Use Inspect interval to see any row’s estimate, estimated standard error, t multiplier and endpoints. The selected fitted slope is marked in blue.

Increasing sample size or reducing response noise tends to narrow intervals. Repeat the group to see that the number covering the true slope varies. The 95% statement concerns the long-run coverage of the procedure, rather than an exact requirement that 19 out of each group of 20 intervals cover. All intervals retain the 95% level and use their own n − 2 degrees of freedom.

This is an interval for the population slope, not for an individual future response. It does not establish causation.

Yield can be associated with fertilizer, rainfall, temperature and other measured inputs. Multiple linear regression extends the same linear-coefficient model by fitting a coefficient for each included predictor.

There is still one response variable. With several predictors we can inspect two-dimensional projections, but a scatterplot against one predictor does not automatically show that predictor’s conditional coefficient.

Multiple linear regression combines several measured inputs in one prediction rule.

Multiple regression uses several predictors to model one numerical outcome. For each observation i, predictor j has value xᵢⱼ and coefficient βⱼ. Setting xᵢ₀ = 1 includes the intercept in the same summation. With m predictors there are m + 1 coefficients.

Stack the observation rows [1, xᵢ₁, …, xᵢₘ] into a design matrix X with n rows and m + 1 columns. The response vector y and error vector ε have n entries, while β has m + 1 entries. The matrix equation is the collection of the individual observation equations.

Fit the coefficients jointly. A coefficient describes a change in the modeled mean while the other included predictors are held fixed. A plot of response against one predictor is only a projection and does not remove the influence of the other predictors.

y and ε stack the n responses and errors. β stacks m + 1 coefficients. Each row of X is [1, xᵢ₁, …, xᵢₘ].

The coefficient βⱼ is the modeled change in mean response per unit of predictor xⱼ, holding the other included predictors fixed. The simple example uses predicted yield = 1 + 2F + 0.5R, where F is fertilizer and R is rainfall in illustrative units.

At F = 1 and R = 4 the model predicts 5. At F = 2 with the same R = 4 it predicts 7. The two-unit change for a one-unit increase in F is the fertilizer coefficient, 2. Holding the other inputs fixed defines the comparison; it does not by itself make it an experiment or a causal effect.

When included predictors are correlated, their individual coefficients can be unstable. Holding one input fixed while changing another may also describe combinations scarcely observed in the data, relying on extrapolation. Adding a variable changes the conditional question answered by a coefficient.

This is a model-based conditional association. A causal effect needs additional evidence about the study and mechanism.

Consider the familiar ice-cream and shark-attack example. Imagine recording sales as a predictor and attacks as an outcome across different times of year, then fitting a simple linear regression. The plotted values are illustrative normalized indices rather than a documented real-world survey or a causal experiment. The chart contains 80 training observations and 20 test observations, the same points used in the later interactive example.

The upward fitted line gives evidence of a positive association between the two indices. Recomputing OLS from the 80 training observations gives intercept 0.0703611652 and slope 0.7176136662. The training R² is 0.5300353268, training RSE is 0.1524972516, and RMSE on the separate 20 test points is 0.1487342609. R² compares the model with the mean-only baseline, RSE estimates residual scatter on training data, and test RMSE measures predictive error on the held-out cases. Error values use the normalized response scale.

The slope t statistic is 9.3792282024 on 78 degrees of freedom, with two-sided p-value approximately 1.987 × 10⁻¹⁴. The table reports p < 0.01 for readability. These results concern a nonzero population linear slope under the regression assumptions. The fitted model line matches OLS on the shown training points.

A tempting but unjustified leap is to conclude that eating or selling more ice cream causes more shark attacks. A positive slope, a moderate R² and a small association p-value cannot establish that intervention claim. The data show that sales and attacks vary together, without identifying the mechanism. A common cause could affect both. The next slide states the correlation-versus-causation principle with a humorous illustration, and the following slides investigate the shared-temperature explanation.

Predictor: sales → response: attacks

A tempting causal claim

“More ice cream must be causing the attacks!”

The line rises, R² ≈ 0.53, and the slope test gives p < 0.01. These support a linear association.

What can we actually conclude?

Sales and attacks vary together.
The regression alone does not explain why.

The joke is to treat the ice-cream vendor as the culprit because ice-cream sales and shark attacks rise together in the teaching example. The cone protests “I plead delicious”, and the shark replies “Don’t blame the sprinkles”. This deliberately absurd conclusion illustrates that an association, even one with a small regression p-value, does not by itself establish a cause.

Hot weather is a plausible common cause to investigate. Warmer conditions can increase demand for ice cream and encourage more swimming. More people in the water can increase exposure to sharks and contribute to the observed attack pattern. This is a proposed mechanism, not something established by the sales–attack scatterplot. The chart in the next slide uses illustrative indices and is not a causal experiment.

Correlation describes how two quantities vary together. Causation asks whether deliberately changing one would change the other. A sales–attack association alone therefore cannot tell us whether reducing ice-cream sales would reduce shark attacks. Common causes, study design and assumptions about the mechanism matter. Correlation may be compatible with causation, but it cannot establish it on its own. The next slides inspect the association and develop the common-temperature explanation, followed by a controlled adjustment example.

The absurd conclusion

A possible common cause: hot weather

Warm days can increase both ice-cream sales and swimming. More swimmers means more exposure to sharks.

The association alone does not show that ice cream causes shark attacks.

Correlation: two quantities vary together.
Causation: changing one produces a change in the other. A correlation alone cannot establish that.

More examples: Tyler Vigen’s Spurious Correlations ↗

The plotted training and test observations are illustrative normalized indices, not a causal experiment. Their association and fitted simple-regression slope can be computed from the shown points. Statistical evidence of association cannot determine whether sales directly affect attacks or whether both respond to a shared cause.

The plot is one two-dimensional projection. It contains no temperature measurement, so it cannot by itself estimate a temperature-adjusted coefficient. The following causal hypothesis and controlled simulation explain how that extra measurement changes the comparison.

Try it: These illustrative sales and attack indices rise together. Inspect the points: can this two-variable scatter tell us whether a third variable drives both?

A confounder is a common cause of a predictor and an outcome that can create or distort their association. The proposed diagram gives temperature arrows toward both ice-cream sales and shark attacks, with swimming exposure providing a plausible link to attacks.

A causal diagram records assumptions about a mechanism; it is not inferred from the two-variable scatterplot. Assessing the mechanism needs appropriate observations and study design. A statistically significant simple slope cannot distinguish a direct effect from this shared-cause explanation.

The model includes ice-cream sales and temperature as separate predictors of the shark-attack outcome. ${\beta }_1$ now describes the association with sales while holding temperature fixed; ${\beta }_2$ describes the association with temperature while holding sales fixed. This is a different comparison from a regression using sales alone.

A coefficient that becomes smaller or less statistically precise after adjustment is compatible with a shared-temperature explanation. This is a possible mechanism to investigate, not a claim established by the displayed scatterplot. Temperature observations are needed to estimate this adjusted model; the available sales–attack chart alone is insufficient. Appropriate study design remains necessary for causal conclusions.

A coefficient is the number multiplying a predictor in the model. “Sales coefficient” and “ice-cream coefficient” both mean the coefficient of ice-cream sales, not sales themselves or a correlation score. With temperature included, ${\beta }_1$ describes the modeled response change for one extra sales-index unit while temperature stays fixed, and ${\beta }_2$ describes the change for one extra temperature-index unit while sales stay fixed. The fitted model estimates these coefficients from the sample.

This controlled simulation generates 80 observations. Temperature varies, the sales index responds to temperature plus independent noise, and the attack-risk index responds to temperature plus independent noise. Sales has no direct term in the outcome generator. These are teaching indices rather than observed counts or evidence about real shark attacks.

In the sales-only view, the fitted slope combines the sales association with the shared temperature pattern. Include temperature to fit the intercept and both predictor coefficients jointly. When X shows sales, the adjusted line holds temperature at its sample mean. When X shows temperature, the line instead holds sales at their sample mean. Y always shows shark risk. Dot color shows the predictor not selected for X. Changing the model or axis leaves the same observations in place.

Sample set A gives a strong positive sales-only slope. After adjustment, the sales coefficient is close to zero and its p-value is much larger, while the temperature coefficient remains clear. The p-values are calculated from each model using the appropriate coefficient standard errors: 78 residual degrees of freedom for the simple line and 77 for the intercept plus two predictors.

Sample set B deliberately remains available even though random noise gives a small adjusted sales p-value in that sample. The generator still has no sales term. A significance decision can be a false positive; it is not a guarantee or a proof of mechanism. Sample set C gives another reproducible sample.

The ice-cream sales coefficient can also be called the sales coefficient or ice-cream coefficient: all three names refer to the same multiplier. It is the predicted change in shark-risk index per extra sales-index unit. In the model including temperature, temperature is held fixed for this comparison. The temperature coefficient gives the predicted change per extra temperature-index unit while holding sales fixed. The sales-only model has no temperature coefficient. The explanation below the plot translates each fitted value into a predicted increase or decrease.

Choose Sales only to plot sales on X. Choose Sales + temperature to allow either sales or temperature on X. Switching back to Sales only returns X to sales. Shark risk stays on Y, so every offered view shows a predictor against the model’s response.

The same 80 observations and fitted coefficients remain unchanged when X changes. The plotted model slice varies the selected predictor while holding the other included predictor at its sample mean. With X set to sales, the adjusted line holds temperature fixed. With X set to temperature, it holds sales fixed. Dot color represents the other predictor. These views all match the model outcome, shark risk.

Try it: Choose the model. In Sales + temperature, select sales or temperature on X. Shark risk stays on Y.

LINE is a memory aid for four assumptions of classical linear regression: L stands for Linearity, I for Independence, N for Normality, and E for Equal variance. These assumptions describe the modeled mean and the errors. They matter because classical standard errors and p-values rely on them to describe uncertainty in fitted coefficients. If the assumptions fail, the usual uncertainty calculations can be misleading. A small p-value or high R² cannot establish these assumptions.

Write the response as the modeled mean plus an unobserved error. ${\epsilon }_i$ is the true response minus the population mean specified by the model, for observation i. It differs from the residual, which subtracts the fitted sample prediction. $X$ denotes the observed feature matrix, with one row per observation. Conditional notation means that we fix those feature values when describing the errors.

L means that the conditional mean is a weighted sum of the selected features, with coefficients entering linearly. A polynomial uses powers of the input as features and can still satisfy this requirement. The zero conditional error mean is $\mathrm{E}\left[{\epsilon }_i\right|X]=0$. If the chosen features omit a real curve, averaging more observations does not remove that systematic mismatch.

I means independence of the underlying errors across observations, conditional on the predictors. Repeated measurements on the same person, time series or shared group effects can create dependence. Treating such related cases as independent can misstate uncertainty. Independence does not require the predictor values to be evenly spaced.

A separate identifiability requirement for unique coefficients is full column rank: no feature column can be written as a combination of the other feature columns. For example, including both height in centimeters and exactly the same height in meters makes those columns redundant. Least-squares solutions can still exist when this condition fails, but individual coefficient values need not be unique.

LINE is a memory aid for four assumptions of classical linear regression:
Linearity, Independence, Normality and Equal variance.
They help justify the standard errors and p-values used to judge uncertainty in fitted coefficients.

Linearity is in the coefficients. Using x² as a feature can still describe a curved mean (x = input).

N and E describe the distribution of the unobserved errors after accounting for the mean response. Normality is a conditional assumption at each fixed input. It does not require the predictor values or the raw responses across all inputs to form a bell curve. N describes distribution shape, while E describes whether its spread changes across inputs.

Normality means errors at each input have a normal distribution centered at zero. Under a correctly specified mean, independent normal errors, equal variance and a full-rank design, the usual finite-sample t tests and confidence intervals are exact. OLS itself can minimize the sum of squared residuals without normal errors.

Equal variance means $\mathrm{Var}\left({\epsilon }_i\right|X)={\sigma }_{\epsilon }^2$ for every case. ${\sigma }_{\epsilon }^2$ denotes the common error variance, and Var is the variance operator. This condition is called homoscedasticity. Unequal variance is called heteroscedasticity. A fan-shaped residual cloud is one possible clue. With unequal variance, the usual homoscedastic standard-error formula may misstate uncertainty even if the mean model is appropriate.

The four illustrations are schematics, not data from a particular study. True errors are unobserved, so we use fitted residuals as diagnostic clues. Residual plots can reveal curvature, changing spread or patterns over observation order. A normal Q–Q plot checks distribution shape. Fitted residuals are generally dependent because of fitting constraints, even if the underlying errors are independent. A featureless plot does not prove all assumptions.

N describes the shape of the errors. E describes their spread. Both refer to errors at fixed input values.

OLS can fit coefficients without normal errors. LINE supports classical inference about their uncertainty.

Use residual plots to choose a follow-up question. A curved mean pattern suggests checking the conditional mean and suitable transformations. A widening spread suggests investigating variance or scale. Ordered runs suggest temporal or sampling dependence. An isolated point warrants a data-quality and sensitivity check.

Normality needs a different diagnostic, such as a normal Q–Q plot, and visual patterns are evidence rather than automatic verdicts. Fitted residuals are not the independent errors in the model; the fitting constraints generally make residuals dependent. The goal is to investigate the underlying mean and error process.

A normal Q–Q plot compares residual quantiles with the quantiles of a normal distribution. Approximate alignment along a straight reference line supports a normal-shape approximation, while systematic deviations suggest a mismatch. Residual plots remain diagnostic clues rather than proofs.

A featureless plot does not prove every assumption. A normal Q–Q plot compares residual quantiles with normal-distribution quantiles to assess shape.

Start with a prediction or association question, an observation unit, relevant response units and the information available at prediction time. Choose a split representing the intended use; fit a candidate family with a reliable least-squares solver; compare validation performance and inspect residuals.

After fixing the procedure, assess it on held-out test cases. Report uncertainty together with its assumptions. Keep four claims distinct: fit to observed data, prediction on new cases, evidence about coefficients and evidence of causality. No one score establishes all four.

Prediction quality, coefficient significance and causal evidence answer different questions.

1. For n training cases, how many coefficients does a degree-5 polynomial have, and what is its full-rank RSE denominator?
2. Can a correctly calculated model score have negative R² on unseen test observations?
3. If the feature matrix X has redundant columns, can an ordinary least-squares solution still exist?
4. Does a slope p-value below 0.05 establish that ice-cream sales cause shark attacks?

Use definitions to justify each answer: count coefficients including the intercept; identify the baseline in R²; distinguish existence from uniqueness in least squares; and separate association from causality. The worked answers that follow explain each distinction.

For a degree-5 fit, there are six coefficients. A held-out R² may be negative if predictions are worse than its mean baseline. A singular XᵀX can still have least-squares solutions. A small p-value tests a statistical null under assumptions and does not supply a causal study design.

A degree-5 polynomial has six coefficients, ${\beta }_0$ through ${\beta }_5$, and full-rank RSE denominator n − 6. This requires n > 6. Held-out R² can be negative when its RSS exceeds the TSS of the held-out mean baseline.

A singular XᵀX means the inverse formula is unavailable and the coefficient minimizer may not be unique; least-squares solutions still exist. A pseudoinverse gives a minimum-norm one. A small association p-value does not establish that changing ice-cream sales would change attack outcomes; common causes and study design remain relevant.

A minimum-norm solution means the tied coefficient vector with the smallest total squared coefficient magnitude. The pseudoinverse selects that solution; the lack of a unique coefficient vector does not remove the least-squares fitted predictions.

Least squares chooses coefficients by minimizing squared residuals. Polynomial features preserve linearity in coefficients. Degree controls the candidate family; validation compares choices and the final test assesses the fixed procedure.

Repeated samples make fits vary. Bias–variance reasoning describes this variation, and standard errors, tests and intervals quantify coefficient uncertainty under assumptions. Multiple predictors change conditional comparisons. Residual diagnostics guide model checks, and association remains distinct from causality.

Next lecture: classification changes the target and the loss.

Further reading: An Introduction to Statistical Learning, Chapter 3; Penn State STAT 501 for regression assumptions and inference; and NumPy documentation for least-squares solvers and rank.

The local teaching examples illustrate fitting, evaluation and inference. Their results should be interpreted under the stated assumptions and should not be treated as domain-specific causal evidence.