Suppose we have a third order polynomial in the form :
$$ s^3+a_2s^2+a_1s+a_0 = 0$$
There is nice caveat for third order systems which is derived from the Routh-Hurwitz stability criterion. In order for this polynomial to be stable the following three conditions have to be met (trying to derive the Routh-Hurwitz table will be a total mess for this particular system):
- $a_2 > 0$
- $a_0 > 0$
- $a_2a_1 > a_0$
The characteristic polynomial of the third order system is:
$$ (K+1)s^3+(12-2K)s^2+(10K-2)s+36K-24=0 $$
which by considering the fact that $K>0 \ (\Rightarrow K+1>1>0)$ can be rewritten:
$$ s^3+\frac{12-2K}{K+1}s^2+\frac{10K-2}{K+1}s+\frac{36K-24}{K+1}=0 $$
The above requirements for this particular polynomial are:
- $\frac{36K-24}{K+1} > 0 \ \Rightarrow \ K > 0.6667 $
- $\frac{12-2K}{K+1} > 0 \ \Rightarrow \ K < 6$
- $\frac{(12-2K)(10K-2)}{(K+1)^2} > \frac{36K-24}{K+1} \ \Rightarrow \ K\in[0 \ 2]$
Taking these into consideration we conclude that the gain $K$ should lie somewhere in between the interval:
$$ 0.6667 \ \le \ K \ \le \ 2 $$
If you indeed try the values $0.6666$ or $2.01$ for $K$ you will see that your system goes unstable. For your information, there is a same caveat for the second order polynomials of the form:
$$ s^2+a_1s+a_0 = 0$$
This polynomial is stable if only and only if $a_1,a_0 > 0$.
Now, regarding you root locus of your open loop function, it is somewhat challenging to derive it since there is some complexity going on. You can always use some software to obtain it. This is the root locus from MATLAB:
Below is the root locus of the closed loop system for a specific value of $K=2$. Notice that the marks for the closed loop poles are indeed located on the imaginary axis which means that the system is critically stable (not strictly stable).
The system becomes strictly stable for values of the gain $K$ which lie in the interval: $(0.6667 \ 2)$. For $K=1$ the root locus of the closed loop system becomes:
And as a last test the root locus for $K=0.6667$ of the closed loop system also includes one pole of the closed loop system on the imaginary axis which implies again that the system is critically stable and not strictly stable:
As you see the stability of the system is very well stated throught the Routh-Hurwitz criterion. There are some rules of thumb in order to obtain the root locus of a system, such as that the poles of the system "go" towards the zeros of the system. However, I encourage you to try and obtain some on your own and have some software package to check them. Check also these series on how to draw them by hand. They are really good.
https://www.youtube.com/playlist?list=PLUMWjy5jgHK3-ca6GP6PL0AgcNGHqn33f