Question

I'm not sure how to answer this question can anyone help me on this? Thanks

What is the extended delta update rule?

Answer 1

In machine learning, the Delta update rule is a gradient descent learning rule for updating the weights of the inputs to artificial neurons in a single-layer neural network. It is a special case of the more general backpropagation algorithm. For a neuron { j} with activation function {g(x)} ${\displaystyle g(x)}$ , the delta rule for {j}'s {i}th weight {w_{ji}} ${\displaystyle w_{ji}}$ is given by

${\displaystyle \Delta w_{ji}=\alpha (t_{j}-y_{j})g'(h_{j})x_{i}}$ ,

where

	$\alpha$ is a small constant called learning rate
	${\displaystyle g(x)}$ is the neuron's activation function
	${\displaystyle t_{j}}$ is the target output
	${\displaystyle h_{j}}$ is the weighted sum of the neuron's inputs
	${\displaystyle y_{j}}$ is the actual output
	$x_{i}$ is the {\displaystyle i}th input.

It holds that {{j}=\sum x_{i}w_{ji}} ${\displaystyle h_{j}=\sum x_{i}w_{ji}}$ and {y_{j}=g(h_{j})} ${\displaystyle y_{j}=g(h_{j})}$ .

The delta rule is commonly stated in simplified form for a neuron with a linear activation function as

{\Delta w_{ji}=\alpha (t_{j}-y_{j})x_{i}} ${\displaystyle \Delta w_{ji}=\alpha (t_{j}-y_{j})x_{i}}$

While the delta rule is similar to the perceptron's update rule, the derivation is different. The perceptron uses the Heaviside step function as the activation function {\displaystyle g(h)}, and that means that {\displaystyle g'(h)} does not exist at zero, and is equal to zero elsewhere, which makes the direct application of the delta rule impossible.