When a plant is known to be unstable, does it mean it will have an oscillation response?
3 Answers
An example of control system with non oscillatory response.
$$ \frac{d x}{d t} = 0 + u\\ u = x $$
Closed loop equation of this unstable system is $$ \frac{d x}{d t} = x $$
and its response is $$ x(t) = x(0) \cdot e^t $$
which is unstable response and does not oscillate.
In general, for an unstable, linear system, the contribution of a pole on the positive real axis is a growing exponential (which doesn't have oscillations).
From a practical point of view (I am not an expert):
A stable system may also oscillate, however it will oscillate in a decreasing manner (i.e. the amplitude will always drop).
In an unstable system, the amplitude (may or may not oscillate) but the amplitude will increase with time (at least for some time until in some real life systems a limit is reached).
UPDATE
Figure : types of stable and unstable systems (Source roymech)
Theoretically, when amplitude of the response tends to infinity, it can be said that it is unstable. It doesn't matter it's oscillating or not, tending to infinity does matter.
About rotational systems, tending to infinity means one-way rotation around a center.