In the physical sciences (which are physics, chemistry, astronomy, materials science, etc.), we learned that the uncertainty is +/- the smallest unit (which is 1) of the last significant figure if the uncertainty is not given in a recording of data. So, if we have a digital measuring device that measures to the nearest millimeter, has a manufacturer's stated uncertainty of +/- 1 mm, and gives a reading of 914 mm, then it will obviously be recorded as just "914 mm".

So, if a yard stick is specced to a length of 914 mm +/- 1 mm (I know, the tolerance is too loose for a measuring device, but this is just an example problem) does the true length actually lie somewhere between exactly 913 mm and exactly 915 mm, or may it stray outside even those numbers if higher precision is used? For example, if go down to the micrometer, is the uncertainty actually +/- 999 μm or +/- 1,499 μm according to the rules of significant figures? If we measure the same yardstick using a micrometer, is the reading guaranteed to be somewhere between 913,001 microns and 914,999 microns, or is it instead only guaranteed to be somewhere between 912,501 microns and 915,499 microns, respectively?

  • $\begingroup$ Precision varies as the task. $\endgroup$
    – Solar Mike
    Jul 3, 2022 at 12:48
  • $\begingroup$ Exact rules are those that govern existence that may only be approximated by physics and its derivatives. To ask for exact rules on approximation seems a bit... $\endgroup$
    – Abel
    Jul 3, 2022 at 13:10
  • $\begingroup$ The subjects you list are all factors in Quality Assurance. The ASQC ( American Society for Quality Assurance ) would be an excellent source if you want more information. $\endgroup$ Jul 4, 2022 at 1:20
  • 1
    $\begingroup$ Last sentence of first paragraph is problematic. If you actually have a quantitative estimate of the uncertainty, why would you rely on rounding/truncation to communicate that uncertainty, rather than writing down the quantitative estimate you've got? $\endgroup$ Jul 4, 2022 at 11:13

2 Answers 2


To begin with, in the title to your question are you confusing precision with accuracy?

Accuracy is how close or far off a given set of measurements (observations or readings) are to their true value, while precision is how close or dispersed the measurements are to each other.

In other words, precision is a description of random errors, a measure of statistical variability. Accuracy has two definitions:

  1. More commonly, it is a description of only systematic errors, a measure of statistical bias of a given measure of central tendency; low accuracy causes a difference between a result and a true value; ISO calls this trueness.

  2. Alternatively, ISO defines accuracy as describing a combination of
    both types of observational error (random and systematic), so high
    accuracy requires both high precision and high trueness.

If a device measures a length to the nearest millimeter, that is the accuracy of that device, not the accuracy of the true length of what is being measured. Another device might have an accuracy of 0.5 mm or 0.1 mm. How a person uses either device to take several measurements of the length a particular item is precision.

If a device that measures to the nearest millimeter measures a length of 914 mm, the true length of the item will be between 913 and 915 mm (914 ±1 mm) - if the measuring device is used correctly. If a second device, that measures to the nearest 0.1 mm measures the same length, the true length will be between 913.9 and 914.1 mm (914 ±0.1 mm).

The uncertainty has nothing to do with the item being measured but with the accuracy of the measuring device.

  • $\begingroup$ I'm assuming here that both devices are always accurate within +/- 1 in the smallest digit displayed, only that they have different precisions (resolutions). $\endgroup$ Jul 3, 2022 at 19:52

Exact rules are to specify tolerances where it matters, like the manufacturer of the measurement device specifies +/- 1mm. To "obviously be recorded as just "914 mm"" can obviously cause confusion. Rely not on significant figures unless the reader is the writer as they can easily be lost in translation. We have tolerances because significant figures are insufficient, and gd&t because tolerances are insufficient.

Supposing a measurement is 915 max and a +/-1 device measures 915.001, that is insufficient to conclude something is in or out of tolerance because it might be 914.001. If a +/-.0005 device measures 915.001, that would be conclusive since it must be at least 915.0005 which is greater than the max of 915.

GD&T tries to describe the fit rather than just specify numeric tolerances. That way the fit becomes a requirement, and if used correctly, one can conclude that if two parts do not fit, then one of the two must be out of tolerance.

  • $\begingroup$ I meant just "914 mm" as in recorded in a generic database, not a comprehensive one. Also, my main question is regarding that if a manufacturer specifies the length of something to be 914 +/- 1 mm, will being at 915.001 mm (which would round to exactly 915 mm when using the nearest smallest unit specified by the manufacturer) be a failure for example? Or will it be acceptable all the way until 915.499 mm (because it still rounds down to exactly 915 mm)? $\endgroup$ Jul 3, 2022 at 19:57
  • $\begingroup$ Out of tolerance is out of tolerance. You must however prove it is out of tolerance. If your +/-1 instrument claims it is 915.001, it might still be in tolerance. If the instrument is +/-.0005 then you can justifiably claim it is out of tolerance. $\endgroup$
    – Abel
    Jul 4, 2022 at 5:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.