Artificial Intelligence and Machine Learning · Lecture 08 of 16

Introduction to machine learning · Part 2

From data
to predictions

Variables and datasets. Models and hypotheses. Generalization.

Formulating a regression task

Introduction to machine learning · Part 2 · Start here

Now make the prediction task precise

Our starting point

Known fertilizer level

Predict crop yield

We have chosen a supervised regression task.

The next questions

How is one observation recorded?

Which prediction functions may we use?

How do we judge their predictions?

Predictor: an input used to make a prediction. Here: fertilizer level.Outcome (target): what we want to predict. Here: crop yield.

Handout explanation ↗Link to this slide

Introduction to machine learning · Part 2 · Start here

Formulating a regression task: three chapters

CHAPTER 1Variables and datasetsCollect variables and outcomes, then organize their measured values.CHAPTER 2Models and hypothesesInspect the observations, choose a function form and fit its parameters.CHAPTER 3GeneralizationInspect the fitted curve, quantify disagreement, then test predictions beyond the recorded data.

Variables and datasets → Models and hypotheses → Generalization. Evaluation is how we check generalization.

Handout explanation ↗Link to this slide

01 · Variables and datasets / 1.1

One observation is one field trial

A golden wheat field — One field trial: record fertilizer level, then measure the crop yield.

One row in the original example

Trial i	Fertilizer xᵢ	Yield yᵢ
1	0.0000	1.9200

Here fertilizer is the predictor (input); yield is the outcome (target).

i = field-trial number; xᵢ = its fertilizer value; yᵢ = its measured yield.

Handout explanation ↗Link to this slide

01 · Variables and datasets / 1.2

Variables name quantities; indexed values name records

Toy example: three field trials

Trial i	Fertilizer X₁	Soil moisture X₂	Yield Y
1	0.2	0.4	0.8
2	0.6	0.5	1.1
3	0.9	0.3	0.7

X₁ and X₂ name the inputs; Y names the outcome.

Read values from trial 2

x₂₁ = 0.6Trial 2, predictor 1: fertilizer.

x₂₂ = 0.5Trial 2, predictor 2: soil moisture.

y₂ = 1.1: the measured yield of trial 2.

First index i: choose the trial (row).Second index j: choose the predictor (input column).Uppercase X₁, X₂ and Y name variables.Lowercase xᵢⱼ and yᵢ record values for one trial.

Handout explanation ↗Link to this slide

01 · Variables and datasets / 1.3

Variable types: magnitudes and categories

A type tree: quantitative variables can be continuous or discrete; categorical variables can be nominal, ordinal or binary.

Continuous: values along a continuum.Discrete: separate quantitative values.Nominal: categories without a natural order.Ordinal: ordered categories, without equal spacing.Binary: two categories; these may also be ordered.

Handout explanation ↗Link to this slide

A quantitative variable expresses a magnitude for which numerical differences have meaning. Continuous variables are modeled as ranging along a continuum, such as temperature or yield. Discrete quantitative variables take separate values, such as the integer count of plants in a field. A categorical variable names a category. Nominal categories, such as species, have no natural order. Ordinal categories, such as low, medium and high, have an order but not necessarily equally spaced differences. Binary means there are two categories; a binary variable may also have an ordering. These meanings survive changes in storage. Coding low, medium and high as 0, 1 and 2 does not establish equal distances between the categories. Decide how to represent a variable from its meaning and the intended model.

01 · Variables and datasets / 1.4

Predictors are inputs; the outcome is the target

Predictors X₁, …, XₘAvailable when we predict

ModelUses the predictor values

Outcome YThe quantity to estimate

X₁: fertilizer → Y: yield

X₁, …, Xₘ = predictor variables; m = their number.Y = outcome variable, also called target or response.Predictor, input and feature describe the same role here.

Handout explanation ↗Link to this slide

Predictors X₁, …, Xₘ are the input variables; m is their number. The outcome Y is the variable whose value we want to predict. Other names for predictors include features, explanatory variables and independent variables. Other names for the outcome include target, response and dependent variable. These names describe roles in the prediction task. “Independent variable” does not establish statistical independence, and “explanatory” does not prove a causal effect. A variable can be an input in one task and the target in another. Every predictor must be available at prediction time. Soil moisture measured before planting is a possible input for predicting yield. Future realized rainfall is unavailable if the prediction is made before that rainfall occurs. A forecast of rainfall is different from its later realized value.

01 · Variables and datasets / 1.5

Collect the observed input–target pairs

𝓓 = {(x_{i}, y_{i})}_{i = 1}^{n}

Trial i	Fertilizer xᵢ	Yield yᵢ
1	0.0000	1.9200
2	0.0417	1.7144
3	0.0833	2.0449

n = 25observations

m = 1predictor per observation

𝓓 = dataset containing n input–target pairs.i = observation number, from 1 to n.xᵢ = recorded input; yᵢ = recorded target.Here n = 25 observations and m = 1 predictor.

Handout explanation ↗Link to this slide

01 · Variables and datasets / 1.6

Domains contain values; input spaces contain combinations

A separate two-predictor toy example

𝓧₁ = {0.2, 0.6, 0.9}
𝓧₂ = {0.3, 0.4, 0.5}

Example domains for fertilizer and measured soil moisture.

𝓧 = 𝓧₁ × 𝓧₂

The Cartesian product combines each of three fertilizer values with each of three soil-moisture values, giving nine possible input pairs.

X₁ = fertilizer; X₂ = measured soil moisture.𝓧ⱼ = allowed values for predictor Xⱼ; j identifies it.𝓧 = input space; 𝓨 = allowed outcome values.× forms every possible tuple; ∈ means “belongs to”.The 3 × 3 domain combinations give 9 possible inputs; they need not all be observed.

Handout explanation ↗Link to this slide

The domain 𝓧ⱼ is the set of values allowed for predictor Xⱼ. The outcome domain is 𝓨. For quantitative regression, 𝓨 is typically a subset of the real numbers ℝ; for classification it is a finite set of class labels. The full input space is the Cartesian product 𝓧 = 𝓧₁ × ··· × 𝓧ₘ. Here × means that we form tuples by choosing one value from each domain; it is not multiplication of measured values. With fertilizer and soil moisture as predictors, one input is a pair consisting of one fertilizer value and one moisture value. The notation x ∈ 𝓧 means that x belongs to the input space. A space describes possible cases; a dataset contains only the particular cases we recorded. A suitable encoding is needed before categorical, image or text inputs can be used in a numerical design matrix.

01 · Variables and datasets / 1.7

One input vector collects one value per predictor

A separate two-predictor toy example

Record	Fertilizer	Soil moisture	Yield
1	0.2	0.4	0.8
2	0.6	0.5	1.1
3	0.9	0.3	0.7

Read the first record as a column

x₁ = (0.2, 0.4)ᵀ
y₁ = 0.8

Fertilizer first, soil moisture second.

xᵢ = (xᵢ₁, …, xᵢₘ)ᵀ ∈ 𝓧
xᵢⱼ ∈ 𝓧ⱼ, yᵢ ∈ 𝓨

xᵢ = observation / input data point; yᵢ = its target.xᵢⱼ = measurement of predictor j; m = predictor count.𝓧ⱼ = predictor j’s domain; 𝓧 = full input space.𝓨 = outcome domain; ᵀ writes the entries as a column.

Handout explanation ↗Link to this slide

Observation i consists of an input vector xᵢ and, in supervised training data, a target value yᵢ. The vector xᵢ = (xᵢ₁, …, xᵢₘ)ᵀ collects one value for each of the m predictors. The superscript T denotes transpose; we use it here to write the listed values as a column vector. For an illustrative two-predictor example, let x₁ = (0.2, 0.4)ᵀ and y₁ = 0.8. The first input entry is fertilizer level and the second is soil moisture; the target is yield. These toy values illustrate vector and matrix notation. They are separate from the original 25-observation plot, which has only one predictor. In xᵢⱼ, i identifies the observation and j identifies the predictor. The notation xᵢ ∈ 𝓧 and yᵢ ∈ 𝓨 states that recorded inputs and targets belong to their respective spaces.

01 · Variables and datasets / 1.8

A design matrix stores input rows; a vector stores targets

Design matrix: bold X

X = 0.20.40.60.50.90.3

Toy inputs: fertilizer, then soil moisture; one row per trial.

Target vector: bold Y

Y = (0.8, 1.1, 0.7)ᵀ

Measured yield for each trial, in the same row order.

X ∈ ℝ^n×m, Y ∈ ℝⁿ
Row i of X is

x_{i}^{T}

; the target in row i is yᵢ.

n = number of observations; m = number of predictors.i = row number; j = predictor-column number.xᵢ = input column vector; ᵀ transposes it into a row.Bold X stores inputs; bold Y stores the target values.ℝ = real numbers; ℝ^n×m means n rows and m columns.Xⱼ names a predictor variable; its values fill column j.

Handout explanation ↗Link to this slide

The design matrix X stores input observations as rows and predictors as columns. For n observations and m numerical predictors, X ∈ ℝ n×m . Entry Xᵢⱼ is predictor j in observation i. Since xᵢ is a column vector, row i of the matrix is xᵢᵀ. The target vector Y = (y₁, …, yₙ)ᵀ has one entry for each row. In the three-row toy example, the input matrix has rows (0.2, 0.4), (0.6, 0.5), (0.9, 0.3), and the target vector is (0.8, 1.1, 0.7)ᵀ. It has n = 3 and m = 2, unlike the original one-predictor chart. Bold X names the whole matrix; Xⱼ names a predictor variable; xᵢ names an observed input vector. Bold Y names the vector of recorded targets, while Y names the outcome variable and yᵢ one recorded target. Matrix notation assumes suitable numerical representations: raw text, images or categories may need encoding.

01 · Variables and datasets / 1.9

General matrix notation: n rows and m predictors

The input matrix repeats the row pattern

X=

x_{1}^{T}

⋮

x_{n}^{T}

=x₁₁…x₁ₘ⋮⋱⋮xₙ₁…xₙₘ

X ∈ ℝ^n×m: n rows and m predictor columns.

The target vector follows the same order

Y=y₁⋮yₙ

Y ∈ ℝⁿ: one target for each input row.

n = observations; m = predictors; ℝ = real numbers.xᵢ = input column vector;

x_{i}^{T}

= row i of bold X.xᵢⱼ = predictor j’s measured value in observation i.yᵢ = recorded target for that same observation.The ellipsis … and vertical dots ⋮ mean “continue the same pattern”; ⋱ continues it diagonally.

Handout explanation ↗Link to this slide

02 · Models and hypotheses / 2.2

Training fits predictions to recorded targets

xᵢ ∈ 𝓧Recorded predictor values

f: 𝓧 → 𝓨The prediction rule

ŷᵢ ∈ 𝓨Predicted target

yᵢ ≈ ŷᵢ = f(xᵢ), (xᵢ, yᵢ) ∈ 𝓓

𝓓 = recorded input–target pairs; i = record index.xᵢ = input; yᵢ = observed target; ŷᵢ = its prediction.f: 𝓧 → 𝓨 maps inputs to predicted outcomes.≈ means approximately equal; ∈ means belongs to.Model training searches for a function f that fits the collected data. The chosen model restricts its possible shape.

Handout explanation ↗Link to this slide

02 · Models and hypotheses / 2.3

A model fixes a form; a hypothesis chooses its parameters

Polynomial family: degree at most 2

f_θ(x) = ax² + bx + c
θ = (a, b, c)

Keep this form fixed. A hypothesis θ chooses numerical values for a, b and c.

Three quadratic-family curves with coefficient triples (1,0,0), (2,-1,0.5), and (0.5,1,0).

x = one numerical input; f_θ(x) = predicted output.a = squared-term coefficient; b = linear coefficient.c = intercept: the prediction when x = 0.θ (theta) = the coefficient tuple (a, b, c).Course convention: θ is a hypothesis within a fixed function form; f_θ is the corresponding prediction rule.

Handout explanation ↗Link to this slide

02 · Models and hypotheses / 2.4

A hypothesis space contains the allowed choices

All allowed choices: Θ

Θ = ℝ³

Every real triple (a, b, c) is allowed in this quadratic family.

One selected choice: θ

θ = (1, 0, 0) ∈ Θf_θ(x) = x²

This tuple selects one candidate function.

Choose the family / allowed space

Fit one parameter setting

θ = one coefficient tuple (a, b, c).Θ (capital theta) = all allowed coefficient tuples.ℝ³ = triples of real numbers; ∈ means “belongs to”.x = numerical input; f_θ(x) = its prediction.For this family, f_θ(x) = ax² + bx + c; a, b and c are the fitted coefficients.

Handout explanation ↗Link to this slide

03 · Generalization / 3.2

Why does the curve miss some observed points?

Zoom of the original fertilizer observations and supplied curve. At fertilizer level 0.25, the measured yield is 2.225 and the curve prediction is 2.475; the vertical gap is highlighted.

A convincing curve raises two questions

Could another hypothesis fit these observations better?

How can we measure fit when a plot is impractical, for example with many predictors?

Read each gap vertically: measured yield versus the curve prediction at the same fertilizer level.Gaps can reflect measurement noise, unrecorded conditions or model mismatch.

Handout explanation ↗Link to this slide

03 · Generalization / 3.3

A residual is the gap between observation and prediction

At the same input, an observed target of 4 and a prediction of 3 give a vertical residual of 1.

From one gap to an average score

rᵢ = yᵢ − ŷᵢ

Observed minus predicted.

MSE = \frac{1}{n} \sum_{i = 1}^{n} r_{i}^{2}

Square the gaps, add them and divide by n.

yᵢ = observed target; ŷᵢ = predicted target.rᵢ = residual: observed minus predicted.i = record number; n = number scored; xᵢ = its input.MSE = mean squared error; ∑ adds the n squared residuals.MSE has squared target units. Low training MSE alone does not establish good new predictions.

Handout explanation ↗Link to this slide

03 · Generalization / 3.4

Generalization means predicting relevant new cases

Training observations are used to fit a rule; different new observations check its predictions.

Recorded data: yᵢ ≈ ŷᵢ = f_θ(xᵢ), (xᵢ, yᵢ) ∈ 𝓓New cases: y ≈ ŷ = f_θ(x), (x, y) ∈ 𝓧 × 𝓨

𝓓 = training input–target pairs; i = record index.𝓧 = input space; 𝓨 = outcome domain.x = an input; y = its associated outcome; ŷ = prediction.f_θ = prediction rule with fitted parameters θ.𝓧 × 𝓨 contains possible input–outcome pairs.The goal is useful predictions for relevant unseen cases. Noise can prevent exact agreement.

Handout explanation ↗Link to this slide

03 · Generalization / 3.5

Training, validation and test sets have different jobs

Illustrative 60/20/20 split: 30 training observations in the largest block, 10 validation observations and 10 test observations. Every dot represents one observation.

Training set: fit adjustable model settings and input transformations.Validation set: compare choices such as model complexity.Test set: assess the final selected procedure.Hold out later dates or different groups when those match the intended use.

Handout explanation ↗Link to this slide

03 · Generalization / 3.7

Data leakage lets evaluation information enter preparation

Computing preprocessing from all rows before splitting leaks evaluation inputs; split first, fit preprocessing on training only, then apply it to held-out inputs.

Preprocessing transforms inputs before prediction, for example by subtracting a learned mean. Fit that transformation on training data, then apply it unchanged to held-out inputs.

Handout explanation ↗Link to this slide

03 · Generalization / 3.8

Use the notation to reason about four cases

1 · Data

80 trials have four predictors, a lookup ID and a target. What is the shape of X? Does the ID belong in it?

2 · Model

Changing θ from (1,0,0) to (2,−1,0.5) changes signs. Does it require a different quadratic family?

3 · Error

In a toy case, observed yield is 4 kg and prediction is 6 kg. What does the residual mean? What units does its square have?

4 · Evaluation

Predict later months for fields never used in training. Which dates and field identities should be held out?

Bold X = input matrix; θ = coefficient tuple (a, b, c).Residual = observed target minus predicted target.Use the definitions to justify each answer; a lookup ID is not automatically an input.

Handout explanation ↗Link to this slide

03 · Generalization / 3.9

Reason from the definition, then apply it

1 · Four predictor columns

X is 80 × 4; Y has 80 targets. A lookup ID does not become a predictor just because it is stored.

2 · Same allowed space

Both tuples are in Θ = ℝ³. Negative coefficients are allowed; the functional form stays quadratic.

3 · Overprediction

r = 4 − 6 = −2 kg. The prediction is too high. Squaring gives 4 kg².

4 · Later, different fields

Train on earlier dates from one group of fields; evaluate later dates from other fields.

n = 80 observations; m = 4 predictor columns.θ = coefficient tuple; Θ = allowed tuples; ℝ³ = real triples.Here r = residual, measured minus predicted. The intended use determines the appropriate split.

Handout explanation ↗Link to this slide

03 · Generalization / 3.10

A complete prediction problem follows this chain

1 · Define the record

What is observed, and which outcome is predicted?

2 · Organize the data

Pairs in 𝓓; predictor rows in X; targets in Y.

3 · Choose candidates

A function family and its allowed parameter space Θ.

4 · Fit parameters

Select θ using training observations.

5 · Compare choices

Use validation evidence for model selection.

6 · Assess the result

Estimate new-case performance on a held-out test.

𝓓 = dataset; bold X = input matrix; bold Y = targets.θ = fitted parameters; Θ = allowed parameter choices.The regression lecture develops fitting methods within this same formulation.

Handout explanation ↗Link to this slide

Introduction to machine learning · Part 2 · Sources

Sources for the data, notation and evaluation

Course material

Introduction to Machine Learning · original course lesson

The variable, dataset and hypothesis definitions, and the two original fertilizer charts.

Evaluation guidance

scikit-learn · Common pitfalls and data leakage

Training-only preprocessing and appropriate held-out evaluation.

The original chart coordinates and interactive controls are retained locally for offline use.

Handout explanation ↗Link to this slide

From datato predictions

Now make the prediction task precise

Our starting point

The next questions

Formulating a regression task: three chapters

One observation is one field trial

One row in the original example

Variables name quantities; indexed values name records

Toy example: three field trials

Read values from trial 2

Variable types: magnitudes and categories

Predictors are inputs; the outcome is the target

Collect the observed input–target pairs

Domains contain values; input spaces contain combinations

A separate two-predictor toy example

One input vector collects one value per predictor

A separate two-predictor toy example

Read the first record as a column

A design matrix stores input rows; a vector stores targets

Design matrix: bold X

Target vector: bold Y

General matrix notation: n rows and m predictors

The input matrix repeats the row pattern

The target vector follows the same order

Inspect the original fertilizer observations

Training fits predictions to recorded targets

A model fixes a form; a hypothesis chooses its parameters

Polynomial family: degree at most 2

A hypothesis space contains the allowed choices

All allowed choices: Θ

One selected choice: θ

Compare the original observations and hypothesis

Why does the curve miss some observed points?

A convincing curve raises two questions

A residual is the gap between observation and prediction

From one gap to an average score

Generalization means predicting relevant new cases

Training, validation and test sets have different jobs

Fit the training points; compare unseen validation points

Data leakage lets evaluation information enter preparation

Use the notation to reason about four cases

1 · Data

2 · Model

3 · Error

4 · Evaluation

Reason from the definition, then apply it

1 · Four predictor columns

2 · Same allowed space

3 · Overprediction

4 · Later, different fields

A complete prediction problem follows this chain

1 · Define the record

2 · Organize the data

3 · Choose candidates

4 · Fit parameters

5 · Compare choices

6 · Assess the result

Sources for the data, notation and evaluation

Course material

Evaluation guidance

Continue the course

From data
to predictions