Cost is a factor, but gradation is probably more important. Round aggregate is very nice for making concrete: the same mix proportions with rounded aggregate will produce a higher slump (more workable) concrete than will crushed aggregate. Now it does turn out that crushed aggregate may produce a higher ultimate strength, because the crushed aggregate tends to interlock, whereas rounded aggregate tends to lose connection to the main mass.
ASTM C-33 specifies a gradation for sand which minimizes void space - the approved gradation doesn't need so much paste to bind everything together. Many unusable sands, like some beach sands, may be overrepresented in one size range, so to get the desirable overall gradation, most of the sand must be discarded (or used elsewhere). On the other hand, when the cement is the item to be tested, rounded sand of a narrow size range is used (Ottawa Sand, from Illinois, windblown). Paste cubes made from Ottawa Sand have a failure mechanism which is highly dependent on cement properties (strength, adhesion) rather than on aggregate (adhesion, interlocking).
But ultimately, cost does determine whether there is enough value in the sand, or does it need to be processed to make it usable.