What is the difference between a state observer and a Kalman filter? Having implemented various types of Kalman filters, I'm still a bit confused, mainly because state observers require the selection of poles similar to a control system setup, whereas Kalman filters don't. Any clarification on the two and their differences would be great.
They are both different forms of the same thing. I will try to clarify this a bit without a lot of buzzwords, which many people seem to do.
In short: The Kalman filter is an optimal observer similar to how LQR is an optimal controller. Both together form an optimal state feedback controller.
A bit longer: You already noted that an observer is nothing more than a kind of controller controlling your estimate of a system towards the real system based on your sensor output, so just think of it as a controller. You can design a controller using frequency domain techniques giving you a transfer function with poles and zeros, using a state space representation and providing a control matrix yourself, or by solving a Riccati equation to find an optimal controller (LQR), requiring you to define cost matrices penalizing control effort and error. (And in many many other ways.)
The Kalman filter is just an optimal observer/estimator similar to how an LQR controller is an optimal controller. You can even design them using the same Riccati equation. (You just change your weighting matrices to the covariance matrices of process and observation noise and so on, but you are familiar with this.) The Kalman filter or Linear Quadratic Estimator (LQE) is a way of selecting the observer gains. You did this manually using pole placement to stabilize your error dynamics (see the controller analogy?). The Riccati equation gives you stabilizing gains by definition, but you lose control over where poles are placed exactly.
If you (or anyone else reading this) want to know more about this, the wikipedia page about Linear Quadratic Gaussian Control (LQG), which is nothing more than the combination of a Kalman Filter and a LQR controller might be a good starting point.
In observer-based state estimation, the focus is on providing a mathematical proof that the error of state estimation goes to zero (exponentially or in a finite time). For that, the proof is usually based on characterizing of dynamic of error, and therefore the formulation is quite close to stability proof of control systems (with pole placement or through Lyaponuv function). However, in most cases, the proof of the stability of estimation error dynamic is developed with "deterministic" assumption where measurement and process noise are not considered.
On the other hand, the Kalman filter approach focuses on the state estimation for stochastic processes (measurement and process noise exist and are addressed). In the Kalman filter, the problem is expressed as an optimization problem where is the existence of process and measurement noise, obtain the best estimation of state and characterize uncertainty for that and we are not looking for converging the estimation error to zero anymore since, due to stochastic nature, it is not feasible. Actually, it can be shown if the assumptions of the Kalman filter hold (Gaussian measurement and process noises and linear dynamic), its performance is better than any other state estimator. For nonlinear systems, variations of Kalman filters such as Extended Kalman Filter (EKF) and Unscented Kalman Filter(UKF) are available.