There are two types of generator sets of interest: regular generator sets and inverter generator sets.
Regular generator sets create 50 Hz or 60 Hz electricity directly all the time. Thus, the rotation rate of the generator always has to be 3000 RPM or 3600 RPM. However, this doesn't mean the engine is running at 3000 RPM or 3600 RPM. The engine may be connected to the generator by a belt, by a chain or by using gears. Nevertheless, the engine is all the time running at constant speed, whatever it may be. In directly connected generators (no gears, belts or chains), obviously the running speed of the engine has to be 3000 RPM or 3600 RPM.
Because the engine is running at constant speed, it is never "idling" if you don't have any load. The load on the engine decreases, but its speed doesn't. If it's a gasoline engine, the throttle plate will be nearly closed with no load, creating lots of pumping losses. Friction losses also don't decrease. Therefore, the fuel consumption will be reduced only very little if you disconnect all load. Diesel generators would be slightly better because they have no pumping losses, but they would still suffer from the constant friction in the engine. I don't have any figures for regular generator sets, but expect them to be poor below the rated load.
Inverter generator sets are different. They create high-voltage direct current, and then use pulse width modulation to synthesize sine wave, or for some early or really cheap (not common anymore nowadays) generator sets, modified square wave with three levels: positive, zero, negative.
Because the frequency of the output sine wave is synthesized, the engine may be running at any speed provided that the engine can supply enough power. Usually there's an "economy switch". Some cheap Chinese generator sets may say in the manual that "economy switch" must be off for high loads, better generator sets can even provide full load with "economy switch" on. What the "economy switch" does is it allow free RPMs. With "economy switch" on, RPM is typically variable, very low at no power output, very high at full power output. With "economy switch" off, RPM is always at the maximum power RPM. In better generators (not those cheap Chinese ones), you only need the "economy switch" off if you are planning to start a really large tool or appliance such as a large circular saw, that can require a high inrush current when starting, creating a momentary high load. Typically these inverter generator sets are designed to be lightweight, able to be carried with one hand, so the flywheel is lightweight, and given that the engine is single cylinder, it can stall if there's a huge inrush current. When you turn the "economy switch" off, you increase the RPM of the engine, allowing starting huge tools or appliances. Then you can immediately turn the switch on after the inrush current is gone and the engine stays at optimal RPM.
Even inverter generator sets that run on gasoline suffer from pumping losses. At very low load, the throttle plate is nearly closed, creating an intake vacuum that causes those pumping losses. Diesel generators would be better but heavier and more expensive. The main benefits of inverter generators are that (1) if RPM decreases, friction losses decrease, and (2) if RPM decreases, throttle plate can be opened more for same power output, which reduces but does not eliminate pumping losses. Also noise reduces if engine speed reduces.
Here's an example of fuel consumption of Honda EU10i (1000 watt inverter generator set): https://hsaoy.com/wp-content/uploads/PAKM_Honda_EU10i.pdf
In case the link goes stale, here are the figures:
0 W, 0.25 l/h
215 W, 0.29 l/h, 1.35 l/kWh
430 W, 0.38 l/h, 0.88 l/kWh
630 W, 0.47 l/h, 0.74 l/kWh
829 W, 0.59 l/h, 0.71 l/kWh
1029 W, 0.77 l/h, 0.75 l/kWh
So you can see that at full load, it's consuming about 3x the gasoline it would be consuming at no load. That's hardly optimal. A car cruising at 120 km/h is consuming maybe 7 liters / 100 km, or 8.4 liters per hour whereas a car idling would be consuming 0.5 liters per hour, 16.8x difference. And the "cruising at 120 km/h" wouldn't even be full load for the car. Whereas for these small inverter generators, full load consumes 3x the idle consumption.
Gasoline contains about 32 MJ/l or 8.89 kWh/l, so the best efficiency this generator set (Honda EU10i) can attain is below 16%. Typical efficiency at low load is maybe 10%. As you can see, most of the energy goes into keeping the engine running, and very little goes into the produced electrical power output, except at high loads where produced electrical power output exceeds the constant consumption.
A larger generator would be less efficient at same load. At 386W average load, my Champion 92001i (1900 watt unit) consumes 0.44 liters per hour of small engine gasoline (which would correspond to 0.42 liters per hour of ethanol-less regular gasoline -- the small engine gasoline has low energy density). The EU10i at same load would consume about 0.36 liters per hour (interpolated from the figures), not sure if the EU10i test was done with ethanol-less regular gasoline, with E10 gasoline or with small engine gasoline.