How do I proceed with finding the transfer function?
The concept of (Laplace transform) transfer functions are applicable for linear systems. So, you need to linearise your equations of motion. Linearisation uses the concept of Taylor series.
In the example given in your question, that reduces to removing the term $\dot{\theta}^2$. This comes from the assumption used for linearization.
but I'm unable to figure out what this assumption is.but I'm unable to figure out what this assumption is.
The assumption is that values of $\dot{r},\ \ddot{r},\ \theta,\ \dot\theta,\ \ddot\theta$ are small such that
$\dot{r}^2,\ \ddot{r}^2,\ \theta^2,\ \dot\theta^2,\ \ddot\theta^2, \ \dot{\theta}\dot{r},\ \theta\dot\theta$ etc.
are negligibly small compared to $\dot{r},\ \ddot{r},\ \theta,\ \dot\theta,\ \ddot\theta$.
Alternatively, can I proceed with leaving the θdot term and changing it to the Laplace domain, and again find the transfer function from there?
Just to clarify, the papers (I think) are not cancelling $\dot{\theta}$ terms, they are cancelling terms which are the product of two or more "small-valued" variables; e.g. $\theta\dot{\theta}$ or $\dot{\theta}^2$ or $\theta^2$.
What about finding it with respect to θ
I assume that the input to your system is defined as $\theta$. If the input to the system was something else, like torque to the disc, then, you are right. The partial derivative with respect to θ does need to be taken.
edit
what if "θ˙^2" has a considerable value, such that the equation cannot be linearized. How would it be approached?
It is better to follow a text book for knowing the proper way to do this. I will attempt to illustrate the specific case below. I follow the Taylor series expansion as mentioned above.
Let the steady value of $\dot{\theta}$ be $a$.
Then,
$$
\dot{\theta}^2_{(a+\delta_\dot{\theta})}
=
a^2 + \frac{2\dot{\theta}|_{a}}{1!} \delta_{\dot{\theta}}
+ \frac{2}{2!} \delta^{2}_{\dot{\theta}}
=
\color{blue}{a^2} +
\color{red}{
\frac{2 a}{1!} \delta_{\dot{\theta}}
}
+ \frac{2}{2!} \delta^{2}_{\dot{\theta}}
$$
The term in blue is part of the steady state and is usually not accounted for deviations from steady state which we want to analyse. The term in red is the linear part which we will use for further analysis of deviations from steady state. The variable $\dot\theta$ is now replaced by its variation from steady state value $a$. The new variable which will appear in the linear analysis is denoted $\delta_{\dot{\theta}}$. However, a lot of textbooks will not introduce a new symbol for this purpose. They will reuse the original symbol $\dot{\theta}$ and the reader needs to understand from context, if they are referring to the original variable or the variation from its steady state value. Note that the term is red is linear in the new variable $\delta_{\dot{\theta}}$
The variation $\delta_{\dot{\theta}}$ is assumed to be a small value. If the variation from steady state value is also not small, then linear analysis is not suitable and one needs to do non linear analysis and transfer functions cannot be used (IMO).
