Go back to first principles. Engine horsepower depends upon:
i.) Cubic capacity.
iii.) Compression ratio.
There is also the inevitable ‘ proportionality constant ‘ , which tidies up all other constants relating to interchanging units.
i. & ii. together give you the amount of air passing through the engine in 1 minute, & hence the amount of fuel that can be burnt to generate power.
If you want a small & compact, lightweight, engine, keep the cubic capacity small, & hence the rpm will need to increase to achieve the same total ‘flow’.
The more you can compress the charge prior to ignition, the greater the relative expansion, & the more power that can be absorbed from the process. There is, of course, a practical limit. Due to the heating effect of compressing gases, petrol will tend to ignite prematurely, before the ideal point in the cylinder cycle - detonation, also known as ‘pinking’. This limits the available power, and damages the engine into the bargain. The upper limit is about 8.5 -9 :1. Above this, you have to put additives into the fuel, to deter pre-ignition. The most famous of which was tetra-ethyl lead, which is now banned.
Because diesel oil is basically a much heavier, less volatile hydrocarbon than petrol [ gasoline], and it is only injected directly into the cylinder at the point of almost maximum compression, much higher compression ratios are possible, a factor of maybe 2–3.
Of course, the diesel engine has to be much more robust, & hence heavier to survive these stresses. I’ve never heard of a practical diesel motorbike!!
They can also be slower revving, & longer stroke, which gives greater torque. I’ve driven trucks with 13–14 litre [ 780 -840 ] that produce 450 bhp, in a power band that runs from 1000–1500 rpm. The torque is truly awesome. My road car has a 1.9 litre twin turbo diesel, connected to a 6 speed gearbox, & redlined at 4500rpm, to yield 180bhp. It does not have the same long stroke, hence low[ish] torque, & cannot really pull away in 2nd gear.