Based on your nicely constructed model, it appears that two axes of movement will be required. The vertical axis can be addressed by a pair of bearing assemblies securely mounted to the top and bottom of the center pole:

It may be necessary to find a different type of assembly which allows you to secure the inner race to the pole, but the image serves the purpose of this discussion.
I found this pillow block bearing which has a setscrew on the inner race:

These types of bearings are usually designed to tolerate axial loading but for light duty, slow speed operation will probably hold well enough. You may also find some other bearing types suited for thrust loads.
With a suitable bearing assembly on each end of the pole, one could create a trolley to attach to the bearing assembly. This trolley would ride in a track internal to the shelf structure.

This image above is for a rolling roof mechanism. It would be necessary to have four of these at each bearing assembly (at the corners?) and matching rails above and below both bearings.
None of this is to scale and a 40 kg load may require large scale components.
I've been building a CNC router which uses v-shaped wheels, riding in aluminum extrusions with matching v-grooves. This permits the carriage to travel from one end of the gantry to the other, but it's not carrying 40 kg, either.
3D printers use linear bearings which provide similar movement, but again, not 40 kg loads.
The distance to the wall will be based on the spacing of the travel wheels from the axle center as well as the size of the wheels. I can envision using in-line skate wheels with matching tracks, providing suitable load capacity. One could get away without using the wheel portion and having bearing contact directly on the travel rails, if there is sufficient clearance.