The primary reason is gravity. When humans first started building things structures that were perpendicular to the ground stayed up more. Stacking rocks for example, the straighter you could get it the more stable it would be, therefore the more 'square' the rocks you used, the easier it would be to get it straight. This has not changed, while it's possible to build walls at an angle, these need to be stronger and have more support. Actually if you build a long enough 'rectangular' structure, the top wouldn't be flat, it would bend with the curvature of the Earth. However, at scales we deal with on a day-to-day basis, the easiest shape to build with is rectangular.
Triangular structures can also be strong, but you get less usable space inside the structure for the same amount of material used.
Gravity comes into it again, though indirectly. Also geometry plays a big role in what shape our human devices are.
Due to gravity, there are lots of things to see on the surface of Earth, such as trees, rocks, wild animals, food, etc. but not all that much up in the air and downwards there is just the ground. As we evolved, it was more important to see things to the left and right than up and down, such as predators or potential food. So we evolved two eyes, arranged horizontally, giving us a better field of vision in the horizontal plane than the vertical one. Our arms and hands also evolved to allow us to manipulate things horizontally more than vertically. This means if you're looking at something or interacting with it (such as your monitor and keyboard) it's much easier to do so in a horizontal direction.
An arc shape (similar to the shape a car windscreen wiper makes) might actually be more ideal than a rectangle for interacting with, but it is more difficult to manufacture, so ease of manufacturing also comes into it. We do have curved interfaces, but generally when we need to pack lots of things into a small yet accessible space, such as a car dashboard, or an air-traffic-controller's desk, where function is more important than simplicity.
Rectangles are much less universal in smaller human devices. Circles can also be simple to make, for example mugs/vases/bowls on a potters wheel, table-legs on a lathe. This is where geometry comes in. If you want to enclose something, for example hold tea in a mug, using the same amount of material you can hold more with a circular shape than a square one. However if the purpose is to spread things out, to make them accessible (for example words on a page or items on a table) moving them away from each other at right angles will get them further apart for a shorter move-distance. So the function plays a much bigger role than physical viability for smaller items.
Design for aesthetics does of course also affect things, but they need to remain functional otherwise they would become useless, so this limits what can be done in this regard and guides the shape of things around us.