The short answer is, there is no amplitude used. Even more important though, is the fact that the displacements and stresses shown in the results of a modal analysis cannot be used to say anything about the physical behavior of the part in absolute terms.
The basic equation of motion is
$$[M][\ddot{U}]+[B][\dot{U}]+[K][U]=F(t)$$
$M$, $B$,and $K$ are matrices for mass, dampening, and stiffness, respectively, and $U$ is the displacement. The material properties are known, and we are solving for the displacement. These are specified for the individual nodes created when meshing an FEA model. For a modal analysis, we ignore the damping effects, and assume there are no loads present. This essentially poses the question "If we constrain the part in a certain manner but apply no load to it, what are the possible ways in which it will vibrate?"
Look at the equation of motion when we neglect the damping and apply no load (F=0)
$$[M][\ddot{U}]+[K][U]=0$$
We are trying to solve this equation for non-zero values of $U$; that is, points at which the inertial forces and spring forces of the material are equal. In this idealized case of no damping, the part could theoretically vibrate through these points forever after an initial displacement/force is applied.
In solving the idealized equation above, we also let
$$[\ddot{U}]=\lambda[U]$$
where $\lambda$ is an eigenvalue. This is a consequence of harmonic motion, which makes sense when you think about it a bit. If a part is oscillating between two positions and the points of the part take linear paths between their two positions, the acceleration vector will always be a linear multiple of the displacement vector.
In the final form of the equation, we see that we only have 4 distinct terms:
$$\lambda[M][U]+[K][U]=0$$
Since $M$ and $K$ are known, we're just looking for a displacement matrix for which a constant $\lambda$ exists. But given the nature of this equation, it's easy to see that for a matrix $[V]=A[U]$ where $A$ is an arbitrary constant, $\lambda_U=\lambda_V$.
Lastly, it's useful to note that the units of $\lambda$ are $s^{-2}$, meaning that $\lambda=\phi^2$ with $\phi$ being the resonant frequency of the vibration mode in question.
So as I said at the top, there is no amplitude used to the actual main calculations in a modal analysis. To display the results and give the "displacement" values in the results, there is some standard amplitude or normalization done, but those numbers are not what you're looking for in a modal analysis, and they shouldn't be used in absolute form; how you can use them is in determining the proportionality of displacements between nodes. If you see point A has a displacement of 2mm when point B has a displacement of 1mm, you know that 2:1 ratio will always exist for this vibration mode.
To determine how much a part will actually vibrate, you have to do a full vibrational analysis, defining the loads and their frequencies yourself, using the knowledge you've gained from the modal analysis.
Two good references: