Wheels surface area is very, very small, compared to say a bike, tricycle or other wheeled vehicle. That's fine on hard smooth surfaces like say a roller rink or a skateboard ramp. But terrible on pavement and gritty surfaces.
Rubber wheels slightly deform and compress to meet the surface contours they ride on. Hard wheels like skateboards and rollerblades ...
You can use a 'negative' rake to increase the stability - like this:
It even works without handlebars:
Or if you want suspension too:
... sorry about the volume of the music!