# Statistical approach to defect detection and prioritization

My company has over 100k communication devices deployed. We collect failure information from the devices. We have two type of data failure count per hour for each device and we use that to calculate failure percentage. We are developing a pro active monitoring system which will help us better maintain the system. One of the challenge is to prioritize devices using failure count or failure percentage. Is there any statistical or mathematical theory which will help us create a priority list using both count and percentage?

Edit 1: All the devices are not the same. But for this process we will consider to be same. To have a good system performance we have to prioritize our troubleshooting.

• Are all of your devices the same? Do you have enough information to be able to be able put them in to categories? – hazzey Oct 16 '15 at 23:53

What you are considering, I think, is a system reliability problem. From your description my impression is that these devices may or may not be connected, that they form a network and that you are looking to improve the reliability of operation of the entire system.

# System Reliability

The solution you are seeking is complicated by the fact that the reliability of one part of your system may affect the reliability of another route within the system. So, in addition to identifying the reliability of individual components, you will need to identify how the network is connected; where events are mutually exclusive, where the communication flow is sequential, and where it may be in parallel. Of course if you have control of this network structure, ie you can redirect the routing of data within the system, then you may be able to optimize the network, which is one of the goals of your opening post, I believe.

The next issue of great importance is to establish whether the probability of failure of a device is independent of all other devices. You may decide that the reliability of a device is affected by the devices around, or connected to it. If so, this will affect the mathematics of your analysis considerably.

If all events are independent, then you can apply classical probability laws to your calculation. However, if they are not independent, then you will need to apply conditional probability, or Bayes' Theorem. Using a computer, it is feasible apply brute-force Monte Carlo methods by assigning random variables (probability mass or density functions) to each element to model the behaviour. A widely used domain specific language for this type of analysis is the R Language.

As an example, take the case illustrated below that is borrowed from Hahn and Shapiro, Statistical Models in Engineering:

We shall examine three stages in series ($S_1$, $S_2$, $S_3$) that determine the reliability of the whole system ($S$). If we constructed a Venn diagramme, we would be looking for the area of the diagramme where the three overlap - their intersection. We define these stages to behave as follows:

• Stage 1: Events are not mutually exclusive but are independent. Survival depends upon the survival of A and/or B.
• Stage 2: Survival depends upon the survival of C only.
• Stage 3: Events are not mutually exclusive but are independent. Survival depends upon the survival of D and/or E and/or F.

Where events are operating in parallel, and they are not mutually exclusive but are independent, then the collective survival is computed according to:

$$Pr = \bigcup Pr_i = 1 - \prod (1 - Pr_i)$$

From the diagramme and equations above, we now can determine the overall system reliability:

$$Pr_S = Pr_1 \times Pr_2 \times Pr_3$$ $$Pr_1 = 1 - ( 1 - 0.9 ) \times ( 1 - 0.8 ) = 0.98$$ $$Pr_2 = Pr_C = 0.95$$ $$Pr_3 = 1 - ( 1 - 0.9 ) \times ( 1 - 0.9 ) \times ( 1 - 0.5 ) = 0.995$$ $$Pr_S = 0.98 \times 0.95 \times 0.995 = 0.926$$

So the probability of survival for the above system is 0.926.

Where elements of the analysis are not independent, then we enter the realms of conditional probability and Bayes' Theorem must be applied. For example if the reliability of B depends upon the reliability of A, then the probability of event B given event A is:

$$Pr(B \mid A) = \frac{Pr(AB)}{Pr(A)}$$

This can be generalized to produce Bayes' Theorem:

$$Pr(A_i \mid B) = Pr(A_i) \times \frac{Pr(B \mid A_i)}{\sum Pr(B \mid A_i) \times Pr(A_i)}$$

where the $rhs$ term $Pr(A_i)$ is often identified as the prior probability and the remaining fractional term is the factor by which the prior probability is updated as more experimental data is obtained. For example, you could assign a prior probability to a device from a certain manufacturer based upon your belief or experience, until more data is acquired.

# Monte Carlo

The above analysis yields a single answer for the entire system. You may prefer instead to develop a range of possible outcomes as a probability density function. If this is desired, then instead of applying single reliability values for the analysis (as was done in the example), random variables expressed as density (continuous) or mass (discrete) functions would be assigned.

The random variables are perturbed by running a large number of computer simulations with pseudo random number generators. Each of the simulations is captured and the collection of results is used to develop an overall probability density function for the system reliability. With this you can better engineer your system in that you would then be able to look at variance as a key factor, something that is not possible to do under the previous scenario.

The selection of the correct representation for the random variable of interest is of great importance. The commonly used normal (Gaussian) distribution may not be appropriate in that it permits negative values as outcomes. Instead the variables should comply with the form:

$$\int_0^{\infty} f(x)dx$$

For time to failure analyses, an exponential distribution is often used. However, if a general frequency of failure is of interest, independent of time, then it may be possible to move between a normal distribution and log-normal distribution, as this permits compliance with the above form. Of course the appropriate selection depends upon the nature of the events or processes within the system to be simulated.

# Constraint Propagation

Another approach that may be beneficial is to model the system by considering each device (or subnet of devices) as a constraint. This is an inference net and the goal is to propagate values through interconnected logic constraints that obey the laws of probability. Instead of yielding a single value, as offered by the first scenario, or yielding a distribution of possible outcomes, as in the second scenario, all inputs and outputs are intervals over the range 0 to 1. This will result in an interval bounding the expected reliability, for example [0.92,1.0].

The technique is formally identified as constraint propagation by Winston and Horn and was originated by J. R. Quinlan, INFERNO: a cautious approach to uncertain inference, Comput. J. 26 (1983) 255-269. If you are considering applying this technique then I would strongly advise reading Quinlan's paper, who was a researcher at the RAND Institute at the time this was published.

An excellent review is found within Chapter 23 of Lisp 3rd Edtion, Winston and Horn and Chapter 3 of Artificial Intelligence, Winston. Within Winston and Horn you will find source code that will permit you to prototype your network rather quickly, according to the probability model you are using. If you are not familiar with Lisp you can download cross-platform versions of Clozure Common Lisp or CLISP at no cost. This technique is ideally suited for functional programming languages and I have noticed Haskell is used for this type of analysis too.

The benefit of using constraint propagation is that you can work with uncertainly as to your assessment of the risk of failure. This is more of a plausibility approach rather than strict probability. Each logic gate is assigned an upper and lower probability of reliability, within the range 0 to 1. The spread between these two bounds is a measure of your certainty as to the reliability of the element of interest. These values then propagate throughout the network in accordance with the laws of probability (not simple arithmetic) and this allows you to immediately consider the impact of a sudden change of a specific node on the reliability of the entire system.

The illustration below provides a schematic of an example set of constraint boxes, each with inputs, $L_i$ and $U_i$, and outputs, $O_i$. All of the values in the network must be contained within the interval [0,1], and the $Pr_i$ values must be within the interval [$L_i$,$U_i$].

The system initializes with all values as $L_i = 0.0$ and $U_i = 1.0$. As the nodal values are each updated, the system automatically propagates new values to all nodes in accordance with the laws of probability. The Or constraints consider survival if either of the inputs ($L_i$ or $U_i$) so indicate, the And constraints consider survival only when both values so indicate (more conservative).

The end result, in this case $O_C$, will yield an interval that will indicate your lower end reliability; for instance [0.92, 1.0]. If the upper end reliability were to drop below 1.0, it could portend serious network reliability problems.

This works forward and backwards, so it is possible to consider what-if scenarios, ie, what would be the effect on the entire system if a node were replaced, for example. It should also be a possible to produce colour-coded output of the network to quickly illustrate where problems are emerging, perhaps in real time, although I have not used it for this purpose.

# Closure

For a network with 100,000 nodes, consider breaking the network down into blocks of contiguous, like units. Model each of these in your system first, then develop beyond to look at the whole. No doubt you will discover potential benefits very quickly under this process.

• Data flow is independent of each equipment as they are the end point. But thanks for the info. Very helpful. – Jonab Kornell Oct 22 '15 at 18:32
• @JonabKornell : You are welcome - if you are just looking at a large collection of independent, unconnected elements, the classical approach to failure analysis is the use of the Hazard Function and it's derivatives which could be normal, exponential or Weibull distributions, for example. But these devices are not standing alone, are they, they are connected to something and this may make it difficult to consider them in isolation. – AsymLabs Oct 22 '15 at 19:50