CHP Home | GenChem Analytical Instrumentation Index |
---|

The following quantities are quantitative measures of precision. All of the equations below provide precision measures for a limited number of repetitive measurements, i.e., between 2 and 20.

These quantities provide a measure of the uncertainty in a mean value determined from a series of repetitive measurements. The mean or average, , is calculated from:

Where N is the number of measurements and x_{i} is each individual measurement.
is sometimes called the sample mean to differentiate it from the true or population mean, µ. The formula for µ is the same as above, but N must be at least 20 measurements.

The standard deviation, s, is a statistical measure of the precision for a series of repetitive measurements. The advantage of using s to quote uncertainty in a result is that it has the same units as the mean value. Under a normal distribution, ± one standard deviation encompasses 68% of the measurements and ± two standard deviations encompasses 96% of the measurements. It is calculated from:

Where N is the number of measurements, x_{i} is each individual measurement, and is the mean of all measurements. The quantity
(x_{i} - ) is called the "residual" or the "deviation from the mean" for each measurement. The quantity (N -1) is called the "degrees of freedom" for the measurement.

The relative standard deviation (RSD) is useful for comparing the uncertainty between different measurements of varying absolute magnitude. The RSD is calculated from the standard deviation, s, and is commonly expressed as parts per thousand (ppt) or percentage (%):

The %-RSD is also called the "coefficient of variance" or CV.

The confidence limits are another statistical measure of the precision for a series of repetitive measurements. They are calculated from the standard deviation using:

You would say that with some confidence, for example 95%, the true value is between the confidence limits. The t term is taken from a table for the number of degrees of freedom and the degree of confidence desired.

D.F. | 90% | 95% | 99% |
---|---|---|---|

1 | 6.31 | 12.71 | 63.66 |

2 | 2.92 | 4.30 | 9.93 |

3 | 2.35 | 3.18 | 5.84 |

4 | 2.13 | 2.78 | 4.60 |

5 | 2.01 | 2.57 | 4.03 |

6 | 1.94 | 2.45 | 3.71 |

7 | 1.90 | 2.37 | 3.50 |

15 | 1.75 | 2.13 | 2.95 |

1.65 | 1.96 | 2.58 |

You might also encounter the term "confidence interval". The confidence interval is the span between the confidence limits:

The quantitative measures of precision described above are the most common for reporting analytical results. You might encounter other measures of precisions, and several other quantities are listed here for completeness.

standard error:

variance:

The advantage of working with variance is that variances from independent sources of variation may be summed to obtain a total variance for a measurement.

All of the equations above are for quantitating the precision of a relatively small numbers of replicate measurements. For 20 or more measurements, the true or population standard deviation, , is:

Top of Page |