World's most popular travel blog for travel bloggers.

are the activations of hidden nodes in an ANN binary or real valued?

, , No Comments
Problem Detail: 

this may seem to be a pretty basic question, but it is something i have been puzzling over for some time.

when calculating the activations of nodes in a hidden layer in an ANN using sigmoid neurons for use with the backpropagation algorithm - should the output of the neuron be thresholded or not?

say the activation vector for layer $i$ is calculated by:

$\mathbf{a}_i = \sigma(\mathbf{w}_i\mathbf{a}_{i-1} + \mathbf{b}_i)$

where $\sigma$ is the sigmoid activation function

should it actually be:

$\mathbf{a}_i = (\sigma(\mathbf{w}_i\mathbf{a}_{i-1} + \mathbf{b}_i) > 0.5$)

to ensure that $\mathbf{a}_i$ is binary vector? or should the real value from $\sigma$ be passed on to the next layer?

my main reason for asking this is that biological neurons don't transmit real valued data, but i haven't really been able to find any definitive answer anywhere to this question or anyone who explicitly says to threshold the value; it seems like a pretty fundamental question to the functioning of ANNs, so either i am missing something in my reading, or it is considered so common-sense that it doesn't need to be mentioned much.

any help would be greatly appreciated

Asked By : guskenny83
Answered By : Tobias

When you do thresholding your activation function becomes a step function. The problem with that is that step functions are not differentiable, so you would not be able to do basic gradient descent to learn the parameters. You can see the sigmoid as a continuous approximation of a step function.

Best Answer from StackOverflow

Question Source :

3200 people like this

 Download Related Notes/Documents


Post a Comment

Let us know your responses and feedback