# Quantifying Instrument Precision Using Multiple Measurements: Rounding

I'm working with a spectrometer that collects raw data as a 16-bit integer value, so the readout I get from a measurement always varies between 0 and 65,535.

I'm trying to avoid implementing data with false precision in subsequent analysis of the data. In order to quantify the level of instrument precision / repeatability, I took multiple measurements of the same target with consistent illumination.

The highest and lowest values measured for the consistent target varied by 1250 values.

How would one use these results to properly round the data collected by the spectrometer in order to avoid implementing false precision? Is there a set of 'best practices' for rounding measured values? Would I want to round my output to the nearest 1250 values? To the nearest 1000? The nearest 2000?

Let's say the output from the instrument for a measurement on a completely different target is 59279. Knowing the instrument is imprecise to 1250 values, what should the output be rounded to in order to avoid false precision?

• Take a look at engineering.stackexchange.com/questions/3315/… Also can you please share a example set data. Oct 11 '16 at 2:15
• @MahendraGunawardena I'm not sure I understand how the information at that link is applicable. Oct 11 '16 at 3:09

Roughly speaking, if your uncertainty is x then you shouldn't claim more precision than x . (Strictly speaking, there's nothing wrong with reporting a mean value of $3.1415926536 \pm 0.003$ but it's not particularly useful.