Introduction to machine learning · Part 2

An observation in the fertilizer example is one field trial: one field receives a recorded fertilizer level, and its yield is measured later. A dataset row joins those values for the same trial. This identifies the unit of observation before introducing formulas.

The fertilizer example contains 25 trials and one predictor. Its physical dosage and yield units are not specified. Later, a clearly marked two-predictor toy example adds measured soil moisture to illustrate vectors and matrices; it is separate from the 25 fertilizer observations.

The symbol i numbers field trials. The value xᵢ is the fertilizer measurement for trial i, and yᵢ is that trial’s measured yield. The fertilizer measurements use an illustrative scale with unspecified physical units.

Uppercase symbols name variables shared across trials: X₁ is fertilizer level, X₂ is measured soil moisture, and Y is crop yield. This small two-predictor toy example is separate from the 25 fertilizer observations, which have only fertilizer as a predictor. It is reused later to explain vectors and matrices.

For trial 2, the fertilizer value is 0.6 and the soil-moisture value is 0.5. We write x₂₁ = 0.6 and x₂₂ = 0.5. The first index, i = 2, selects trial 2; the second index, j = 1 or j = 2, selects the predictor. Keeping i fixed and changing j moves across the input columns of the same row.

The outcome for that trial is y₂ = 1.1. Because this task has one outcome, its recorded value needs only the trial index. In general, xᵢⱼ records predictor j in trial i, while yᵢ records the target for that trial.

A variable is a feature recorded across experiments when possible. Examples include age, number of owned cars, hair colour and recorded sex categories. The table explains the variable types and gives examples of each.

The measurement or encoding determines the type. Age may be treated continuously or recorded in whole years. Binary describes two categories and can overlap with an ordered scheme; 0/1 category codes do not by themselves make a task quantitative. Ordinal codes do not imply equal spacing.

This formulation predicts one outcome Y from one or more predictors X₁, …, Xₘ; m is their number. A predictor is an input variable, also called a feature, independent variable or explanatory variable. The outcome is the output to predict, also called target, dependent/predicted variable or response.

These names describe task roles and may refer to a variable or its recorded measurement in context. “Independent” does not establish statistical independence, and “explanatory” does not establish causation. A proposed predictor may turn out to have no useful relationship with the outcome.

Inputs must be available at prediction time. Soil moisture measured before planting could be used to predict yield. Future realized rainfall would not be available for a prediction made beforehand; a forecast is a different input.

The supervised dataset $\mathrm{𝓓}={\left\{\right(x_i,y_i\left)\right\}}_{i=1}^n$ is a collection of n recorded input–target pairs. Here i identifies a trial, xᵢ records its input and yᵢ its target. With one predictor, xᵢ can be displayed as a single number; with several predictors it will be a vector.

The fertilizer example contains n = 25 observations and m = 1 predictor. The table displays its first three rows, rounded to four decimal places. The observations example at the start of Models and hypotheses retains all 25 observations.

The domain 𝓧ⱼ is the set of values allowed for predictor Xⱼ. The outcome domain is 𝓨. For quantitative regression, 𝓨 is typically a subset of the real numbers ℝ; for classification it is a finite set of class labels.

The full input space is the Cartesian product 𝓧 = 𝓧₁ × ··· × 𝓧ₘ. Here × means that we form tuples by choosing one value from each domain; it is not multiplication of measured values. With fertilizer and soil moisture as predictors, one input is a pair consisting of one fertilizer value and one moisture value.

The notation x ∈ 𝓧 means that x belongs to the input space. A space describes possible cases; a dataset contains only the particular cases we recorded. A suitable encoding is needed before categorical, image or text inputs can be used in a numerical design matrix.

In the diagram, three allowed fertilizer values and three moisture values give 3 × 3 = 9 possible input pairs. The three recorded toy observations occupy only three of those combinations.

Observation i consists of an input vector xᵢ and, in supervised training data, a target value yᵢ. The vector xᵢ = (xᵢ₁, …, xᵢₘ)ᵀ collects one value for each of the m predictors. The superscript T denotes transpose; we use it here to write the listed values as a column vector.

For an illustrative two-predictor example, let x₁ = (0.2, 0.4)ᵀ and y₁ = 0.8. The first input entry is fertilizer level and the second is soil moisture; the target is yield. These toy values illustrate vector and matrix notation. They are separate from the 25-observation fertilizer plot, which has only one predictor.

In xᵢⱼ, i identifies the observation and j identifies the predictor. The notation xᵢ ∈ 𝓧 and yᵢ ∈ 𝓨 states that recorded inputs and targets belong to their respective spaces.

In this notation, xᵢ is the observation or input data point, and xᵢⱼ ∈ 𝓧ⱼ records predictor j in analysis i. The target yᵢ ∈ 𝓨 is the outcome measurement for the same analysis. The terms observation or data point may also refer to the complete pair (xᵢ,yᵢ); here input vector and input–target pair make that distinction explicit. A dataset may contain survey answers, images or text as well as numerical measurements.

The design matrix X stores input observations as rows and predictors as columns. For n observations and m numerical predictors, X ∈ ℝ^n×m. Entry Xᵢⱼ is predictor j in observation i. Since xᵢ is a column vector, row i of the matrix is $x_i^{\mathrm{T}}$.

The target vector Y = (y₁, …, yₙ)ᵀ has one entry for each row. In the three-row toy example, the input matrix has rows (0.2, 0.4), (0.6, 0.5), (0.9, 0.3), and the target vector is (0.8, 1.1, 0.7)ᵀ. It has n = 3 and m = 2, whereas the fertilizer chart has one predictor.

Bold X names the whole matrix; Xⱼ names a predictor variable; xᵢ names an observed input vector. Bold Y names the vector of recorded targets, while Y names the outcome variable and yᵢ one recorded target. Matrix notation assumes suitable numerical representations: raw text, images or categories may need encoding.

The worked numerical matrix is an instance of this general formula. Stack one input row for every observation to form the matrix X, and place the corresponding target values in the same order in Y. Entry xᵢⱼ is predictor j in observation i. The displayed ellipses continue that pattern up to row n and predictor m.

Since xᵢ = (xᵢ₁, …, xᵢₘ)ᵀ is defined as a column vector, row i of X is $x_i^{\mathrm{T}}$. The transpose turns the input column vector into a matrix row. X ∈ ℝⁿˣᵐ has n rows and m columns; Y = (y₁, …, yₙ)ᵀ ∈ ℝⁿ has n matching targets.

Inspect the 25 fertilizer/yield observations. Hover over a point or choose its observation number to read its values. Drag to zoom, switch to Pan to move through the chart, and use Reset view to restore the full range.

Each point is one input–target pair. The horizontal position is the fertilizer input and the vertical position is the recorded yield. This fertilizer plot has n = 25 observations and m = 1 predictor; the earlier two-predictor matrix was a separate notation example.

Plotting the collected measurements reveals a possible curved relationship. Model training searches for a convincing prediction function that fits those recorded pairs. The assumed model restricts its possible form, such as a polynomial family. Model can also mean the fitted function after training.

A function f: 𝓧 → 𝓨 maps an input xᵢ to a prediction ŷᵢ = f(xᵢ). Its prediction should be approximately equal to the corresponding measured target yᵢ for recorded pairs (xᵢ,yᵢ) in the dataset 𝓓. The hat marks a prediction, ∈ means belongs to, and ≈ means approximately equal. The complete fitting equation is shown below.

The task is regression because the outcome is a quantitative magnitude. Approximate fit is expected: measurements need not lie exactly on one curve. The next step fixes a function form and identifies the parameters that training can change.

Following the course convention, a model is an assumed function form. Once that form is fixed, a hypothesis θ is one numerical parameter setting, and the hypothesis space Θ is the set of allowed settings. The corresponding prediction rule is written f_θ; the parameters need the assumed form to determine that function.

For the illustrative family f_θ(x) = ax² + bx + c, θ = (a,b,c). Here x is one numerical input, a multiplies the squared term, b multiplies the linear term, and c is the intercept: the prediction at x = 0. The three curves illustrate three different coefficient choices in the same family.

Training changes coefficients within the selected form. Choosing another maximum polynomial degree changes the model family. Because Θ = ℝ³ permits a = 0, the displayed family contains polynomials of degree at most two, including linear and constant special cases.

The course’s supplied example curve is explored separately. Its model family and fitting algorithm are not specified, so it is not asserted to be quadratic or an optimal fit.

The parameter setting θ selects one candidate function. The hypothesis space Θ is the set of allowed parameter settings, following the parameter-based convention used in the course. In the unrestricted quadratic example, Θ = ℝ³: every triple (a, b, c) of real coefficients is allowed.

For θ = (1, 0, 0), the selected function is f_θ(x) = x². For θ = (2, −1, 0.5), it is f_θ(x) = 2x² − x + 0.5. Both choices belong to the same space Θ and use the same functional form.

Another common convention calls the functions themselves hypotheses and calls their set the hypothesis space. Here θ names the coefficients and f_θ names the corresponding function, so the distinction stays explicit. Fitting chooses a θ; model selection can compare families or complexity settings.

The plot overlays the fertilizer observations and an illustrative hypothesis curve. Toggle traces with the legend or select an observation to compare its target yᵢ with the curve’s prediction ŷᵢ. Their vertical difference is the residual, yᵢ − ŷᵢ.

This supplied curve illustrates a candidate hypothesis; it is not asserted to be quadratic or an optimal fit. The following sections first discuss the gaps, then introduce mean squared error as one numerical way to assess disagreement when a visual comparison is insufficient.

The diagram zooms the same fertilizer measurements. Blue dots are measured yields and the purple line is the supplied curve. At fertilizer level 0.25, the measured yield is 2.225 while the curve predicts 2.475. The highlighted vertical segment compares these two values at the same input. Other points lie above or below the curve, so the curve captures an overall trend without passing through every observation.

The illustrative curve follows the overall trend but misses individual measurements. This discrepancy motivates two questions: could another hypothesis fit the recorded data better, and how can fit be judged when a visual plot is impractical, for example with many predictors?

Measurement error can contribute to these gaps. Unrecorded soil, weather or other conditions can also change yields, and the chosen model may miss part of the relationship. A visible disagreement alone does not identify its cause.

Fitting should capture a useful relationship despite fluctuations in the observations. Passing through every training point can reproduce accidental variation. The next slide defines residuals and MSE as one concrete way to measure disagreement before considering performance on new cases.

The prediction for observation i is ŷᵢ = f_θ(xᵢ). Its residual is rᵢ = yᵢ − ŷᵢ, the observed target minus the prediction. A positive residual means the model predicted too low; a negative one means it predicted too high.

Mean squared error (MSE) averages the squared residuals over n observations: $\mathrm{MSE}=\frac{1}{n}\sum _{i=1}^n{(y_i-{ŷ}_i)}^2$. The summation symbol ∑ means add the terms for i = 1 through n. Squaring prevents positive and negative residuals from cancelling and gives larger disagreements more weight.

MSE is one concrete score for regression, used here as a bridge to the regression lecture. Its units are squared target units. A lower training MSE means a closer fit to these recorded targets, not necessarily more accurate predictions on new cases.

A prediction function should work beyond the recorded pairs in 𝓓, on relevant unseen cases in the input and outcome domains. Generalization means predicting the outcome of relevant unseen inputs, rather than only explaining the measurements used to fit the function.

In the displayed new-case formula, x is an input and y is its associated outcome; ŷ = f_θ(x) is the fitted rule’s prediction. The pair belongs to 𝓧 × 𝓨 because these are its input and outcome domains. This membership specifies the possible types of pairs. It does not require fitting every arbitrary pairing of an input with an unrelated outcome.

With noisy outcomes, approximate agreement on useful new cases is the aim. A small training error is not sufficient evidence of this. Held-out observations estimate performance for the intended future use; the following practical sections separate fitting, model choice and final assessment.

A training set is used to fit model parameters and any learned preprocessing. A validation set is held out from that fitting step and used to compare development choices, such as polynomial degree. A test set is reserved for estimating performance after those choices are fixed.

Repeatedly choosing models from test results makes the test set part of development. It can then give an optimistic estimate of performance. The validation set serves that development role so that a separate test can remain a final assessment.

The split should match the intended use. To predict later dates, hold out later observations. To predict on new fields or people, keep the relevant groups separate. A random row split may be inappropriate when related records appear in both sets.

The illustration assigns 30 of 50 observations to training, 10 to validation and 10 to testing: a 60/20/20 split. The block widths and dot counts show the same proportions. Training commonly receives the largest share, but there is no mandatory ratio; the split should leave enough evidence for fitting, model selection and a meaningful final evaluation.

Choose a polynomial degree and noise level. The model is fitted using 16 training points; its validation score uses 40 separate points generated from the same synthetic process. The true (generating) curve is the known rule used by this simulator: y = 2 + 1.2x − 0.9x². Each dot is a value on that curve plus random noise. Choose Show true curve to reveal it. The fitted curve is instead learned from the training dots; the fitting method is not given the true rule. We can reveal that rule here because the data are artificial; with real data it is usually unknown.

Compare training and validation errors as degree increases. A more flexible family can lower training error while failing to improve, or worsening, validation error. This is the overfitting pattern: fitting details of the sample that do not transfer to new observations. A degree-9 polynomial has more flexibility than degree 2, but need not pass through all 16 training points.

Noise is random variation added to the generated targets. The seed makes a chosen dataset repeatable. Because you use validation results to compare degrees, those points are a development resource, not a final untouched test set. A later test would assess the selected procedure on further held-out observations.

The control σ (sigma) is the standard deviation of the added noise: a larger value produces more variation around the generating curve. Keeping the same seed repeats the sample so that model comparisons can use the same observations.

Here x is a synthetic numerical input and y is its outcome. Polynomial degree is the highest power allowed in the fitted function. The symbol σ (sigma) is the standard deviation controlling the simulated noise. MSE means mean squared error. A seed repeats the same sampled data.

Data leakage occurs when information unavailable in the intended prediction setting influences model development or evaluation. It can enter through input construction, preprocessing or a split that mixes related cases.

For example, computing feature means from all observations before splitting lets held-out inputs influence preprocessing, even if their target labels are never used. Estimate such preprocessing from the training data, then apply the fitted transformation to validation and test inputs.

Also inspect time and group relationships. If the goal is to predict future months for new fields, evaluation should hold out later dates and relevant fields. Merely hiding individual rows can leave closely related records available during training. Decide the deployment question before choosing the split.

Preprocessing transforms the inputs before fitting or prediction, such as subtracting a feature mean learned from data. Fit these learned quantities on training observations only and apply the same transformation to held-out observations.

These cases apply the notation and evaluation concepts introduced in Part 2. They concern the shape of input data, membership in a parameter space, the interpretation and units of residuals, and a split that matches deployment.

State the definition you use, then apply it to the case. The following worked answers explain the reasoning rather than only giving a number.

With 80 observations and four predictor columns, the design matrix has shape 80 × 4 and the target vector has 80 entries. The lookup ID is not an input merely because it is stored alongside the measurements.

Both (1,0,0) and (2,−1,0.5) belong to Θ = ℝ³. Fitting a negative coefficient changes the parameter setting without changing the allowed space or quadratic form.

In the toy yield case the residual is observed minus predicted: 4 − 6 = −2 kg. A negative residual means overprediction. Its square is 4 kg², illustrating why mean squared error has squared target units.

For future months and unseen fields, the held-out data should be later in time and contain different field identities. A row shuffle alone does not enforce either requirement.

A prediction problem begins with an observation unit, available predictors and an outcome. A dataset records input–target pairs; a matrix and vector organize their values. A model family and allowed parameter space then specify the candidate prediction rules.

Training fits a parameter setting. Validation compares development choices, and a held-out test assesses the selected procedure for the intended future use. The next regression lecture develops specific fitting methods within this framework.