Polymers are in fact used - wood is a very common construction material (although perhaps one should consider it a composite?).
I don't really have a straightforward answer, but polymers (= plastics) are also prone to degradation caused by the environment: UV light is bad for them, water might cause them to swell and go soft, they are prone to creep. Also, as you say, they are simply weak (lower Young's modulus). Their range of operating temperatures are very bad too: you can subject metals to fairly wide range of low and high temperatures without changing their properties very much, but this span is much narrower for polymers.
Like wood, they are easy to ignite (unless flame retardants are used which on the other hand they often are) but unlike wood, many plastics lose their capacity to carry loads at higher temperatures (just 100-200 °C). This also happens with metals, but they yield at much higher temperatures than polymers (plastics).