A transfer function of a closed-loop feedback control system is written in the form:
$$ T(s) = \frac{H(s)}{G(s)} $$
where $G(s)$ is called the characteristic polynomial of the system. The poles and zeros of the system are defined:
- Zeros $\rightarrow$ Roots of $H(s)$
- Poles $\rightarrow$ Roots of $G(s)$
The stability of the closed-loop system can be determined by looking at the roots of the characteristic polynomial. Consider the general case at which the poles are complex numbers of the form $p = \sigma + j\omega$ (if $\omega = 0 \rightarrow$ poles are real numbers). Now, there will always be one of these three following cases:
- If at least one pole has positive real part (i.e. $\sigma > 0$) then the closed-loop system is unstable.
- If all the poles have negative real part (i.e. $\sigma <0$) then the closed-loop system is strictly stable.
- If all the poles have negative real parts and at least one has real part equal to $0$ (i.e. $\sigma = 0$) then the closed-loop system may be marginally stable or unstable. Generally, in this case you need to further investigate the stability of the system.
Finally, of course you can use software like MATLAB to determine the stability of an open-loop or closed-loop system. Let's consider the system:
$$ G(s) = \frac{10}{s^2+3s+2} $$
By using the following piece of code you can find the poles, zeros and check the stability of the system:
s = tf('s');
G = 10/(s^2+3*s+2);
zero(G)
pole(G)
isstable(G)
You can check the MATLAB official documentation for more information on these functions.