Question

Exercise Optimization in neural network Consider a very simple neural network with two input values, one output value, and a single neuron with sigmoid activation. Each input to the neuron has an associated weight, and the neuron has a bias. So the network represents functions of the form o(W1X1 + W222 + b). We train the neural network using least squares loss on a single piece of training data ((1, -1),0). Initially all weights and biases are set to 1. Carry out one iteration of gradient descent using a step size of 2.

Answer 1

ANSWER:

The least square loss is basically the sum of squares of the residuals. A residual is defined as the predicted value subtracted from the true value.

Predicted value at (x₁, x₂) = sigmoid(w₁x₁ + w₂x₂ + b)

Considering the single piece of training data and the given weights, the loss is equal to

L = (true value - observed value)²

= (0 - sigmoid(w₁x₁ + w₂x₂ + b))²

= (sigmoid( (1)(1)+ (1)(-1) + 1))²

= (sigmoid(1))²

We are asked to perform gradient descent, this means we need to change the values of the weights and biases in such a way that the loss is decreased as much as possible.

To find this optimal change at the given point, we consider the loss function without plugging in the values of weights and biases, i.e

L(w₁, w₂ , b) = (sigmoid(w₁(1)+ w₂ (-1)+ b))²

= (sigmoid(w₁ - w₂+ b))²

Now, we use the fact that partial derivatives indicate the change in value with respect to the given weight.

al дw д(а(wi — 02 + b))? дw1

θσ(w1 = 2σω- 2 + 6). Ow1 m2 + 6)

► 20(W1 - W2 + b)2(1 – 0(W1 - W2 +b))
    (Since
(0)0 - 1)(2) = d)
)

Similarly

OL диша — 20(W1 l2 +b)=(1 – (ші Ш2 + b))

მL მს 23(2u1 — x2 + 6) (1 — 7(u1 — + ))

Now for the given weights,

this means as w₁ is increasing loss is increasing therefore to decrease it, we have to take a step in the negative direction. Since the step size has already been mentioned. The new weight w₁ = 1 - 2 = -1

OL 001 = 2)- 0 e+1e+1

Similar to w₁ , b becomes 1-2 i.e b = -1

მL მს 2 e -)2, # +1 & +1 0

This is opposite to the above case and hence we increase the weight for w₂  i.e w₂ = 3

e OL Ow2 -2- 26 <0 e +1' 'e +1

NOTE:We can observe that the loss has not decreased, after taking the step. This is because the step size being too big, it overshooted.