[Solved]: Z-score relations to perceptrons

Problem Detail: 

I am learning about preceptrons and my professor noted that z-scores are a commmon pre-processing step to normalizing input variables. Following this, I am having trouble understanding why z-scores are useful when training a preceptron?

My current understanding is that z-scores allows us to calculate the probability of a score landing inside the normal distribution and compare with scores from other normal distributions. However, how does this translate to perceptrons?

Asked By : Christian

Answered By : Nicholas Mancuso

There are two parts to your question that need to be addressed.

1. What transformation are you performing?
2. Why does that help learning?

The transformation of taking your input and turning them into "Z-scores" is really just centering and standardizing the variance for each variable. The reason I put "Z-score" in quotes is because there is no reason to assume that this transformed variable is distributed normally. All you have done is taken a random variable with some unknown distribution and transform it to a random variable with mean 0 and variance 1. That doesn't necessarily mean it is normally distributed. Formally, if we have $$X \sim f(\mu, \sigma^2)$$ where $X$ is a random variable $f$ is the distribution with mean $\mu$ and variance $\sigma^2$, then $$\frac{X - \mu}{\sigma} \sim f(0, 1)$$ using the basic properties of mean and variance. Nowhere does this require $X \sim N(\mu, \sigma^2)$, which is where Z-score comes from.

With that said, why does this help? By centering your data (removing the mean) from both your training data and labels, you don't need to estimate a bias term. Bias (or mean measured effect) is function of the mean of your data; therefore if your data have mean 0, you should have no measurable bias term. Setting all variables to have unit variance places them all on the same scale. Oftentimes in real data the different variables in your model may have widely different variances, which could lead to difficult to compare effect sizes. Setting everything to have similar variance simplifies interpretation.

Best Answer from StackOverflow

Question Source : http://cs.stackexchange.com/questions/64309

 Ask a Question

 Download Related Notes/Documents