TL;DR: Its not the air that's enclosed inside that creates the force, but the lack of air - or more precisely pressure.
First of all some nomenclature (this is for the closed type, there is also the open type):
When you press a suction cup on a surface what happens is that the air inside it is pushed out. During the compression phase of the cup the air comes out of the lip to the atmospheric pressure. So if you have an original volume $V_0$ of the suction cup, and then the volume reduces to $V_2 = \frac{V_0}{2}$ (usually there is even less volume), then the mass of the air trapped inside the suction cup is about half the original (keep this in mind for later on).
Figure 2: Air exiting . (source: ijiset.com)
The dome is flexible, so while pushing it is easy to change its shape (thus force the the air out). Additionally the shape of the lip (at a tangent with the object makes it easy for the air to be driven out). However, after the external force is removed then because of the material elasticity, the suction cup is trying to return to the original shape.
Figure 3: Different stages. (source: Abetterchemtext)
Following on with the example, because the suction cup tries to recover (partially), as the volume of the suction cup tries to increase the air density reduces, and so does the pressure. So what you end up is a low pressure underneath the suction cup. If the suction cup were allowed to recover its original shape (it doesn't) then the pressure in the example would be half the pressure.
Figure 4: Low and high pressure regions. (source: wired)
Then the "suction" force is equal to :
$$F = A\cdot dP $$
where :
- A is the area of the suction cup (its projection)
- dP is the difference in pressure ($P_{High}- P_{low}$)
numerical example
Assuming a diameter of 50 mm, and a pressure difference at about half the pressure of the atmospheric pressure, then you can estimate that the pull out force would be approximately:
$$F = \frac{\pi d^2}{4} \Delta P $$ $$F \approx 2 \; [N]$$
Notice that if you double the diameter the force quadruples, so you can get significantly higher forces.
Why some surfaces are better
The reason why smooth non porous surfaces tend to behave better with suction cups is that there is not much leakage (from the outside in).
Air coming in the suction chamber would increase the pressure inside the chamber, and therefore reduce the pressure differential $\Delta P$.