09-problems.pdf

(159 KB) Pobierz
Artificial Neural Networks
Prof. Dr. Sen Cheng
Nov 29, 2020
Problem Set 9: Multi-Layer Perceptron
Tutors:
Nicolas Diekmann (nicolas.diekmann@rub.de), Eloy Parra (eloy.parrabarrero@rub.de)
1. Consider a small multi-layer perceptron with one hidden layer. The input layer consists of two units, the hidden
layer, of two units with relu activation function, and the output layer, of one unit with hyperbolic tangent
activation function. The network weights are as follows (bias weights are already included):
W
hidden
=
−1
0.5 0.5
−0.5
0
1
W
out
=
−0.5
0.5 0.5
You are provided a training sample
x
with label
y:
x
=
1
1
y
=
1
Use paper and pencil, and an electronic calculator for numerical calculations, to solve the following problems
correctly to 3 decimal places.
(a) Compute the forward pass for the training sample
x.
(b) Using the summed squared error as loss function compute the backward pass (i.e. compute the
delta’s).
(c) Adjust the network weights using a learning rate
η
=
0.01.
Hint:
Derivatives of the activation functions are:
∂
tanh(x)
=
1
−
tanh
2
(x)
∂x
2.
Backpropagation algorithm
A skeleton of a class that models a multi-layer perceptron (MLP) with one hidden layer is provided in
prob
mlp.py.
The
main
function of the provided code already loads an XOR data set (Fig. 1) and splits it into
training and test sets (Note: While you are actively developing and testing your code, you may use a smaller
training set to speed things up). Class identities are coded with
+1
and
−1.
(a) Implement the method
train
that trains the MLP’s weights with backpropagation using stochastic gradient
descent. Use the following parameters for your network:
•
Hyperbolic tangent as activation function for hidden and output layers.
•
The squared error (SE) as loss function (Note: for a classification problem, it is better to use the
cross-entropy loss, but here we will use the squared error as it is easier to compute).
•
Hidden layer with 64 units. Also add a bias unit to the input and hidden layers and change the weights
initialization in
init
method accordingly.
(b) Implement a method
evaluate
in the MLP class. This method should compute the mean squared error
(MSE) on the test set.
1
∂
relu(x)
=
∂x
1, if
x
>
0
0, otherwise
XOR Data Set
1.0
0.8
0.6
X2
0.4
0.2
0.0
0.0
0.2
0.4
X1
0.6
0.8
1.0
Figure 1: XOR data set. Blue and orange points belong to the positive and negative classes, respectively.
(c) Train 30 networks for 15 epochs each with a learning rate of 0.005. After each epoch, evaluate the
performance of the networks. In a single figure, plot the MSE as a function of the epoch number for each
network. In addition, plot the evolution of the MSE averaged across networks.
3.
Xavier initialization
The MLP class provided initializes weights by drawing them from a normal distribution with zero mean and
standard deviation of 1. Extend the class so that weights can be optionally initialized using Xavier initialization.
Train the network as before, changing only the initialization scheme, and plot the results together with the
previous ones. Describe the differences in the training performance between the two initialization schemes.
2

Plik z chomika:

Inne pliki z tego folderu:

Inne foldery tego chomika:

Zgłoś jeśli naruszono regulamin