Question

A deep learning problem.

The following matrices describing a neural network were uncovered by scientists. The weights for the hidden layer are given in the matrix W[1] = [0 1] The bias for the hidden layer is given in the vector b[1] = [1] The weights for the output layer are given in the vector W[2] [8] 0 1 The biases for the output layer are 612] = -0.5 0.75 The input X is given in the vector X 1.25 -0.5 S The activation function for all units in the hidden and output layers is Relu: g(x) = max(0, 2) Determine dimensions of the output, draw the diagram of this network and cal- culate the output.

Answer 1

Network Architecture Dimension = R2 Dimension = R2 Hidden Layer b21 = -0.5 Dimension = R b11 = 1 z1 x1 = 1.25 x1 w11 = 0 a21 z1 w12 = 0 Input Layer a1 h1 Output Layer w12 = 1 x2 = -0.5 w 22 x2 w21 = 1 a22 z2 b22 = 0.75 X = (1.25] [-0.5] w[1] = [0] b[1] = [1] b[2] = (-0.5] Z = [21] [22] [1] [0.75]

For every neuron we can compute the pre-activation value and output by the following equation

pre-activation, a = W.X + b where W is the weight, X is input, b is the bias

Output can be compute using the activation function, here RELU, g(z) = max(0, z)

From the figure

Computing a1, which is the pre-activation for the first hidden layer

a1 = (1.25 x 0) + (-0.5 x 1) + 1 = -0.5 + 1 = 0.5

h1, is the output of the first neuron of the first hidden layer

Can compute using the activation function, RELU, g(z) = max(0,z)

Here, h1 = max(0, 0.5) = 0.5

h1 will be the input to the two neurons in the output layer

For the first neuron

a21 = (0.5 x 0) + (-0.5) = -0.5

z1 = max(0, -0.5) = 0

For the second neuron

a22 = (0.5 x 1) + 0.75 = 1.25

z2 = max(0, 1.25) = 1.25

So Final output

Z = [z1] = [0]

      [z2]      [1.25]

So the output is of two dimension R2