I'm working with a spectrometer that collects raw data as a 16-bit integer value, so the readout I get from a measurement always varies between 0 and 65,535.
I'm trying to avoid implementing data with false precision in subsequent analysis of the data. In order to quantify the level of instrument precision / repeatability, I took multiple measurements of the same target with consistent illumination.
The highest and lowest values measured for the consistent target varied by 1250 values.
How would one use these results to properly round the data collected by the spectrometer in order to avoid implementing false precision? Is there a set of 'best practices' for rounding measured values? Would I want to round my output to the nearest 1250 values? To the nearest 1000? The nearest 2000?
Let's say the output from the instrument for a measurement on a completely different target is 59279. Knowing the instrument is imprecise to 1250 values, what should the output be rounded to in order to avoid false precision?