Question

A deep neural network has multiple layers with non-linear activation functions (e.g., ReLU) in between each layer, which allows it to learn a complex non-linear function. Suppose instead we had a deep neural network without any non- linear activation functions. Concisely describe what effect this would have on the network. (Hint: can it still be considered a deep network?)

Answer 1

Yes, the basic operation in neural networks are matrix multiplication (in most of the cases). Hence matrix multiplication over more than one matrix will induce a non linearity in the system. Hence a deep neural network with out activation function also can establish non linear relationship between data and output.