I have tried to compare two courses, and there seems to be some overlap, but not as much. Which method is better when controlling a nonlinear system? Also, what is the main difference between the two?
In nonlinear control theory, you will recognize most concepts such as controllability and observability where the linear case is often a special case of the nonlinear case. I would highly recommend digging into linear control theory first if you have not done so. Depending on the course you take, concepts such as Lyapunov stability are discussed here, which is a very important concept.
In contrast, optimal control is rather independent of the underlying system, be it linear or nonlinear and has therefore overlap with both, depending on what type of optimal control problem you look at. I would say that it has a broader scope since you will find optimization problems everywhere in different fields of research. Especially important is model predictive control which solves optimal control problems on a receding horizon and is one of the most-used control schemes in the industry (apart from PID control) because it can handle state and input constraints explicitly (which most other types of control fail to achieve).
All in all, I would recommend going this path:
Linear control theory => Nonlinear control theory => Optimal control theory
In control systems, the main focus is the design of a controller for machines or robots, here we mainly deal with linear systems application of linear control theory. While non-linear systems is an advanced topic, where we deal with advance and mathematically more complex systems.
If its ur first course in control systems or automatic control. Go for the linear control system, but if you have a basic background of controls then you can go for non-linear control.