# What does (a) the input of the activation function mean?

I am not very good at understanding the ANN theory, but please if anyone can explain me about what does (a) as input of the activation function mean? and how to get this value?

Thank you

• Maybe your supervisor doesn't understand it either. Commented Feb 3, 2018 at 10:59
• Many of us have not seen this before - could you provide a link with the basic equations/summary? From a statistics perspective - this looks like it may have some relationship to a Gaussian function, but without seeing it in context it is difficult to say. Commented Feb 3, 2018 at 11:03
• Having looked here, the activation function, in your case a sigmoid within hidden nodes, is simply a generic function that scales hidden node input to output in the range of 0 to 1. The a value would be the weighted average of the outputs of the input nodes connected to a specific hidden node. It is, in effect a response to information, as I see it. So by choosing a different activation function, you would change the nature of your model and it's response to information. Commented Feb 3, 2018 at 11:23
• Sorry correction - not weighted average but weighted sum. Commented Feb 3, 2018 at 11:30
• Well you have input values or parameters. You assign weighting factors to these depending upon their relative importance. Then the weighted values from the input nodes are added up. This becomes the a value for the activation function of the next layer hidden node that they (input nodes) are connected to. Because you are using a sigmoid, the output of the hidden node will be a value between 0 and 1. Commented Feb 3, 2018 at 11:59

Just a quick summary of my concluding thoughts on this step in your research. I understand that you have been advised to use the Sigmoidal Function, so this discussion will try to explain it's potential application.

First, I don't think that an Artificial Neural Network (ANN) would apply to your problem. Typically ANN would consider a range of factors on some outcome. When this process is considered, a lot of data is required along with specialized software. This is because training of the ANN is necessary to establish that the model works. Sometimes ANN is used to determine which factors are important in a non-linear, multivariate investigation. Let me provide an example that may illustrate its use.

## Application of ANN

Suppose we are experiencing a routine cracking problem of some concrete elements of a structure. We may collect data, on cracked elements and uncracked elements, to include:

1. Temperature of concrete.
2. Temperature of air.
3. Time of delivery.
4. Time of casting.
5. Wind speed.
7. Cement content.
8. Water content.

And potentially lots of other parameters (there could be dozens, even hundreds).

So in theory we could send perhaps thousands of replicated data sets (measurements) through a neural network to see which factors may or may not be predictive of cracking of concrete elements. That is how ANN is applied and it requires highly specialized software.

As a part of this process, we may also try different weighting factors, as $w$, on the input data to try to improve predictive ability. So, for example we could assemble a vector of weighting factors that can be used to adjust each of the input values. This is a trial and error process that may or may not result in a better outcome. We also may choose an activation function for the hidden nodes that receive the weighted input values. In your example the activation function is a sigmoid; there are many others. The activation function in simple terms will change the behaviour of your ANN in the sense that it changes the response to information. Also important is how the nodes are connected; a good software package will go a long way to doing this for you.

So I hope that this better explains what ANN is and how it may work. I do not think that it would apply to your project because I have experience of this technique being applied to large, international concrete durability research using thousands of data sets and the ANN method alone was enough for granting a PhD to a student from a major UK engineering school.

## Sigmoid Function

Now putting ANN aside, we look at your sigmoidal function from a different perspective. From Wikipedia:

"A wide variety of sigmoid functions have been used as the activation function of artificial neurons, including the logistic and hyperbolic tangent functions. Sigmoid curves are also common in statistics as cumulative distribution functions (which go from 0 to 1), such as the integrals of the logistic distribution, the normal distribution, and Student's t probability density functions."

At first glance I recognized that this function could be derived from the family of Gaussian distributions, and indeed it is. The Gaussian (Normal) distribution is the most commonly used statistical function in existence, and is often used as the basis of error computation and regression analysis. A Gaussian probability density function, or PDF, looks like the following:

You must think of a PDF as a special class of functions belonging to the class of Generalized (sometimes called Generic) functions. These are described as:

"In mathematics, generalized functions, or distributions, are objects extending the notion of functions. There is more than one recognized theory. Generalized functions are especially useful in making discontinuous functions more like smooth functions, and describing discrete physical phenomena such as point charges. They are applied extensively, especially in physics and engineering."

We can consider any probability density function to simply represent a state of knowledge. For example, the Uniform PDF illustrated below:

represents a maximum uncertainty in that, no matter what $z$ value we are considering within the bounds of $a$ and $b$, we will have the same probability of outcome. This is different from the Gaussian above because there is a much lower probability associated with the extreme values than the average value. With the Gaussian, as the standard deviation, or $\sigma$, gets smaller, we can consider that our state of knowledge has improved. Complete certainty, for example, would be where $\sigma \rightarrow 0$ (indeed this condition represents the Dirac Delta Function that is widely used in physics and engineering, especially Finite Element Analysis).

The beauty of using a PDF is that it transforms (or some may use the term normalizes) the data to a relative real range of 0 to 1 based upon our state of predictive knowledge. Thus we may convert and consider any problem in the relative, rather than absolute. This permits multiple data sets to be compared from different experiments and also allows prediction of behaviour, rather than magnitude.

The Sigmoid Function, just like a Gaussian PDF, is indeed one of these creatures. A Sigmoid function plot, with a replaced by $\sum{wx} + bias$, is shown below:

The outcome is a value between 0 and 1. The shape of the curve is very near to what is termed a Gaussian Cumulative Distribution Function or CDF. An excellent discussion of how this is derived may be found at TowardsDataScience.com.

## Possible Application

The application of the Sigmoid to data carries with it the assumption that your data would follow a Gaussian form. That is to say that extreme values are far less probable than others.

For your specific data, thickness or effective thickness, I would not expect that weighting factors $w$ would be necessary nor would there be a need for $bias$ but these decisions are up to you. You have a matrix of measurements that are to be transformed to the range of 0 to 1 using the Sigmoidal, or Gaussian CDF, function.

There are also Excel best practice guides describing how the Sigmoid Function may be used in regression analysis on experimental point data. This can be reviewed in a document that can be downloaded from ExcelCurveFitting.com.

How you apply it depends upon the specific needs of your work. For example, you could transform each value and run the full simulation, then transform collections of values within 50 by 50 cm grids to a single value and run it again, comparing the outcome of each. From this you may consider the need for weighting factors or bias values to improve the regression. It could be that this becomes a trial and error process that would eventually lead you to a deeper insight into the nature of your data and the problem you are attempting to solve.