I am trying to come up with a "stabilization" algorithm for an antenna dish. The dish can move in the azimuth (yaw) and elevation (pitch) directions. But the boat it is mounted to, moves in all three degrees (yaw pitch and roll). To start with, I am only given a heading and elevation the dish must point. How do I calculate for the heading and pitch corrections for once the boat starts to move in the yaw pitch and roll directions? I know the boat's movement in terms of Euler angles.
NASA has a serious widget for this after a couple embarrassing instances of not being able to aim stuff where they wanted and massive hours wasted creating custom control systems for every different satellite instrument. If this is something other than a one-off homework assignment, I'd suggest you study the SPICE conventions and use their methods where possible. This may feel a bit like using a howitzer to kill a bug at first, but once you have a general method that can be stacked, it becomes just a task for computers. In the case of gimbals or articulated arms with a telescope on the end, you may have to perform a dozen transformations to figure out what all the joint angles and step rates must be in order to hold a target.
So finding the endpoint is pretty straightforward if a bit tedious in practice. What isn't straightforward is the engineering of these systems to minimize drift or incorporate targeting feedback information. Some sort of system for fine tuning and error correction is usually required.
SPICE is maintained by NAIF, which is part of NASA. https://naif.jpl.nasa.gov/pub/naif/toolkit_docs/Tutorials/pdf/individual_docs/17_frames_and_coordinate_systems.pdf
If you start with the ship using a North-East-Down (NED) earth navigation and attitude matrix (NED to shipframe), you have two possibilities, but most likely is that you have an antenna target defined wrt the earth. So you need a target-to-earth matrix, and an earth-to-ship matrix (you are given this, apparently), a ship to gimbal matrix (defined by the antenna driver's geometry). You need to solve slew rates for the gimbals that zero the ship rates for target maintenance, and you need to be able to solve slew angles for changing the target. Slew rates can be derived from time stepping the position data, or they can be computed directly from the ship IMU rate data.
It does sound a bit like a specific euler angle homework problem. You can work it out in euler angles or unit vectors or both and observe the utility of euler angles.
Start with the "heading and elevation the dish must point" in world coordinates (as euler angle or unit vector).
Rotate by the inverse of the boat (you likely have the boat orientation in world coordinates). This gets you to the boat coordinate system. Simple example: If you only had heading/azimuth and your boat were 15degrees clockwise off north, then an item at north is 15 degrees counterclockwise from the boat's perspective.
Convert to azimuth and elevation (the goal of where to point, the usual end to the homework problem). Note that the range limits of your device should be considered here. If, for example you can do one full circle on both, there will often be two answers (you can face forward or bend over backwards) and you should aim for the closest to current position so you can get there quickly. You may want to limit the range to avoid wasting effort to point through the boat bottom if you won't do any good. If it is out of range, you might aim for the closest limit and wait for it to come back in range, issue a signal requesting to move the boat, or even use that time to move your system to a better position in the range (if for example you get 3 turns of azimuth in either direction and were close to a limit).
Finally answer: What should my devices do now? You can put together a control loop to drive your current dish orientation to your desired. Tune it to what you can sense and how you can actually react- PID is simple, low-pass filters are great for avoiding a waste of energy attempting to react to something that will be gone before you get a chance to move.
If you're up for some bonus, you might notice that you can apply boat orientation to your current azimuth and elevation to bring that into world coordinates and solve there too.