03-Regression-problems.pdf

(75 KB) Pobierz
Artificial Neural Networks
Prof. Dr. Sen Cheng
Oct 18, 2021
Problem Set 3: Regression
Tutors:
Sandhiya Vijayabaskaran (sandhiya.vijayabaskaran@rub.de), Behnam Ghazinouri (behnam.ghazinouri@rub.de)
Note:
All excercises should be done in python.
1.
Analytical solution of linear regression
Although in real-world problems you do not know the function that generated the dataset, here we assume that
we know the generating function, which is usually called the “ground-truth”. Let
y
=
4x
+
5
+
ε
be the ground-truth function, where
ε
is a uniform random variable.
(a) Generate a vector
x
containing 100 linearly spaced numbers from -3 to 3 (use
numpy.linspace)
and a
vector
ε
of 100 random numbers between
−0.5
and 0.5 (use
numpy.random.uniform).
Then compute
the vector
y
according to Eq. 1 and generate a scatter plot of
y
against
x.
(b) Use univariate linear regression (see lecture notes, section 3.1) to find the parameters
m
and
b
that min-
imize the
l
2
−
norm
loss function (or summed squared errors). Implement the equations yourself using
only elementary operations.
(c) Use the more general multiple linear regression (see lecture notes, section 3.3 ) on the data from part (a) to
ˆ
find
θ.
Implement the equations yourself using only elementary matrix operations. Compare your solution
with the solution in problem 1b.
(d) Compare the estimated parameters to the ground-truth. Visualize the goodness of fit by plotting the regres-
sion line together with the data points. Also, compute the explained variance, and generate the residual
plot.
(e) Use the scikit-learn python package to perform linear regression on the data from (a). From
sklearn
import
linear_model
and use the method
LinearRegression.
Fit the linear model to the data generated
in the first step. Look at the instructions provided by
help(linear_model.LinearRegression)
for
help. Calculate the explained variance
R
2
using
sklearn.metrics.r2_score.
2.
Polynomial regression
Given the stochastic polynomial function
y
=
−x
5
+
1.5x
3
−
2x
2
+
4x
+
3
+
ε,
where
−2
<
x
<
2 and
ε
is a random number uniformly distributed between -1 and 1.
(a) Generate
Y
, an array consisting of 100 data points.
ˆ
(b) Fit a 5th-order polynomial to
Y
, and determine its coefficients
θ.
Plot the data points and the fitted curve
on a single plot.
(2)
(1)
1
ˆ
(c) Compare the parameters,
θ,
you estimated to the ground-truth,
θ,
by using the
l
2
-norm as the error mea-
sure. Check how the error of the estimated parameters varies with the number of data points by plotting
the error in the estimates versus the number of data points N, for
N
∈
[1,
100].
3.
Incremental version of linear regression
Often in practice, datasets can be extremely large, which makes it infeasible to use the analytical solution. Use
stochastic gradient descent (SGD) to compute the regression parameters as follows:
(a) Write a Python function that performs stochastic gradient descent. Use the
l
2
-norm as the loss function
and assume your data can be modelled using multiple linear regression. Only use elementary operations.
(Hint: don’t forget to shuffle the data before iterating through it!)
(b) Use
y
=
4x
+
5
+
ε
as the input for your stochastic gradient descent function to test it.
(c) Compare the parameters you estimated to the ground-truth. Plot both the data and predicted values in
a single graph. Check how varying the number of data points (N
∈
[5,
500]) affects the estimates by
generating a plot as in Problem 2c. Use a constant learning rate of 0.01 and use 3 runs of SGD. Next, keep
the number of data points constant at 100 and use 3 epochs of SGD, and vary the learning rate in steps of
0.001 up to a range of 0.4. Generate a similar plot of error vs. learning rate.
(3)
2

Plik z chomika:

Inne pliki z tego folderu:

Inne foldery tego chomika:

Zgłoś jeśli naruszono regulamin