One size doesn't fit all.
Putting the US prefab houses mentioned by Muze and blown away by stronger wind, if you go with the european prefab, you can quickly end up with architecture like this:
If you ask anyone living there to list the disadvantages, you won't hear the end of the list. One could say the entire list of advantages consists of "it's affordable to build and maintain," "they can be built fast" and "one can live in there."
The biggest part of the prefab house problem, I think, is lack of individualism. If you make "short runs" of individualized prefabs you're losing the economy of scale; price goes up, list of other disadvantages remains. If you go large scale, it's hard not to end with monolithic, dull, depressing architecture.
Brick offers full flexibility. Fancy curved walls, levels that don't correspond to the integer 'floor#', small quirks, fancy decorative styles, and a house that is tailored to your specific needs. It also doesn't require the sort of heavy machinery heavy prefab construction does. It's extensible, modifiable, can be arbitrarily individualized; plays well with other technologies - wood, steel, glass, aluminum. In short, a brick house can be pretty much anything, from a multi-floor block of flats, to a little fancy castle:
Prefabs don't offer anything near this flexibility. Even room sizes come in small multiples of slab sizes: 1-wide, 2-long bedroom; 2-wide, 2-long living room, 1-wide 1-long bathroom and kitchen, plus half-slab wide, 2-long corridor... it gets depressing.