Question

This is more of a conceptual question.

I have learned about Neural Nets, and I have some clue as to how Support Vector Machines work. I read somewhere however that given the appropriate kernel (is that right?), the SVM is identical to the Neural Net. Could someone who understands this please enlighten me as to how that's possible?

Answer 1

For simplicity lets consider a simple single hidden layer feed forward neural net for binary prediction. At test time the neural network predicts

p(Y=1?X=x)=?(w??(Ax)),
where w is the vector of hidden to output connections, A is the matrix of input to hidden connections, ? is the logistic sigmoid function, ? is an element wise function (normally something like a sigmoid, tanh, rectified linear, etc), and I've ignored biases for simplicity.

Now skim through the section Nonlinear Classification in the SVM Wikipedia article. Notice how essentially we have the same thing above, but with everything happening in the neural network before the final dot product serving as the transform function ?, and that's it. The advantage of neural networks is they allow you to easily learn this transform, which can be very beneficial.