Power rating of a motor is the mechanical power out. Divide that by efficiency and you should roughly get electrical power in. Efficiency will be different if not at full-load. To get 6W mechanical @ 60% efficiency, you have to put 10W electrical in.
You have a problem. Driving motor correctly means generator is going in the incorrect direction, which may explain But, if I reversed the driving motor supply polarity, the readings stabilized. Small, cheap, dc motors can have brushes aligned to get more torque in one direction.
This simple view assumes I'm getting a nice flat DC waveform out, but I assume it's really like AC straight from a bridge rectifier (don't have a scope) so maybe RMS should come in somewhere?
Nope. You have a commutator, which the brushes ride on. You'd get DC with a ripple. You could add an inductor in series with output to filter ripple.
On your varying power consumption. What size wires and how far is power supply from motor? Your efficiency has to be applied to motor and generator. There is also a mechanical efficiency between the motor and generator. 10W means 6W (60%) from motor and 3.6W (60%) from generator.
Here you also have a problem. If your generator is at full-load, your motor will be overloaded. I suggest you graph out input and output (measure voltage and current at different loads).
Most motors (DC and three-phase ac) can be used as generators with some being better than others.
Efficiency has to deal with losses (copper loss, hysterisis, windage, bearings), so they apply to motor or generator. Although the motor will be at 60%, generator will be different, probably slightly higher because copper losses will be less (half-load means half current so I2R losses will be 25% of full-load copper losses). You have a small motor, but theory still applies.
Your erratic readings are due to motor going in the wrong direction. As proof of concept, you have a genset.
If you drive a BLDC, you can get a three-phase generator.