A project makes economic sense when the return from the project exceeds the rate you can borrow money at. Usually we want a fairly large return before we concern ourselves with the project however. Lets say we want a 3 year break even or 33%APR. All aspects of that project should target that rate of return to be optimized.
To calculate this you first determine the return. In this case it is the amount of heat saved by upgrading to new windows (kW * 24hr * 365 days), multiplied by the cost of heat ($/kWhr). Then take this amout divided by the cost of the window project. If it is more than your target APR, consider increasing to the next quality level of windows. If it is less than your target APR, try decreasing the quality of the new windows or abandon that aspect of the project.
You can then run the same calculation for all the other items in your project: attic insulation, wall insulation, HVAC COP, etc.