# Quantifying Instrument Precision Using Multiple Measurements: Rounding

I'm working with a spectrometer that collects raw data as a 16-bit integer value, so the readout I get from a measurement always varies between 0 and 65,535.

I'm trying to avoid implementing data with false precision in subsequent analysis of the data. In order to quantify the level of instrument precision / repeatability, I took multiple measurements of the same target with consistent illumination.

The highest and lowest values measured for the consistent target varied by 1250 values.

How would one use these results to properly round the data collected by the spectrometer in order to avoid implementing false precision? Is there a set of 'best practices' for rounding measured values? Would I want to round my output to the nearest 1250 values? To the nearest 1000? The nearest 2000?

Let's say the output from the instrument for a measurement on a completely different target is 59279. Knowing the instrument is imprecise to 1250 values, what should the output be rounded to in order to avoid false precision?

It's more than merely a question of absolute max and min. I recommend you read Bevington's "Data Reduction and Error Analysis" as a good, simple start.

Precision can only be estimated based on the statistics of your data, and that depends on your entire setup: temporal and spatial uniformity of the light source, for example. First, calculate the mean value and the standard deviation. Then wander over to formalisms such as the Student's T-test and the general area of "confidence intervals." These algorithms allow you to use the measured parameters (mean, standard deviation, etc) to estimate the uncertainty in your measurement.
Roughly speaking, if your uncertainty is x then you shouldn't claim more precision than x . (Strictly speaking, there's nothing wrong with reporting a mean value of $3.1415926536 \pm 0.003$ but it's not particularly useful.