CHP Home GenChem     Analytical     Instrumentation     Index   

Measures of Precision


The following quantities are quantitative measures of precision. All of the equations below provide precision measures for a limited number of repetitive measurements, i.e., between 2 and 20.

These quantities provide a measure of the uncertainty in a mean value determined from a series of repetitive measurements. The mean or average, x-bar, is calculated from:


Where N is the number of measurements and xi is each individual measurement. x-bar is sometimes called the sample mean to differentiate it from the true or population mean, µ. The formula for µ is the same as above, but N must be at least 20 measurements.

Standard Deviation

The standard deviation, s, is a statistical measure of the precision for a series of repetitive measurements. The advantage of using s to quote uncertainty in a result is that it has the same units as the mean value. Under a normal distribution, ± one standard deviation encompasses 68% of the measurements and ± two standard deviations encompasses 96% of the measurements. It is calculated from:

standard deviation

Where N is the number of measurements, xi is each individual measurement, and x-bar is the mean of all measurements. The quantity (xi - x-bar) is called the "residual" or the "deviation from the mean" for each measurement. The quantity (N -1) is called the "degrees of freedom" for the measurement.

Relative Standard Deviation

The relative standard deviation (RSD) is useful for comparing the uncertainty between different measurements of varying absolute magnitude. The RSD is calculated from the standard deviation, s, and is commonly expressed as parts per thousand (ppt) or percentage (%):

RSD         coefficient of variance

The %-RSD is also called the "coefficient of variance" or CV.

Confidence Limits

The confidence limits are another statistical measure of the precision for a series of repetitive measurements. They are calculated from the standard deviation using:

confidence limits

You would say that with some confidence, for example 95%, the true value is between the confidence limits. The t term is taken from a table for the number of degrees of freedom and the degree of confidence desired.

t values for finding confidence limits

You might also encounter the term "confidence interval". The confidence interval is the span between the confidence limits:

confidence interval

Other Measures of Precision

The quantitative measures of precision described above are the most common for reporting analytical results. You might encounter other measures of precisions, and several other quantities are listed here for completeness.

standard error:

standard error



The advantage of working with variance is that variances from independent sources of variation may be summed to obtain a total variance for a measurement.

All of the equations above are for quantitating the precision of a relatively small numbers of replicate measurements. For 20 or more measurements, the true or population standard deviation, sigma, is:

population standard deviation

Related topics:

 Top of Page   
 Copyright © 2000 by Brian M. Tissue, all rights reserved.