Quality control of the analytical examination process

Background

The purpose of a clinical laboratory test is to provide information on the pathophysiologic condition of an individual patient to assist with diagnosis, to guide or monitor therapy, or to assess risk for developing a disease or for progression of a disease. QC, also called internal QC, monitors a measurement procedure to verify that it meets performance specifications appropriate for patient care or that an error condition exists that must be corrected.

Content

Internal QC ensures that measurement procedures meet specifications at the time patient testing occurs. QC samples are measured at intervals along with patient samples. Recovery of the expected target values for the QC samples allows the laboratory to verify that a measurement procedure is working correctly and the results for patient samples can be reported. The QC plan specifies the number of controls, the frequency they are to be measured, and the rules to determine if the QC results are consistent with expected measurement procedure performance. External QC, also called external quality assessment (EQA) or proficiency testing (PT) , is an assessment process in which control samples are received from an independent external organization and the expected values are not known by the laboratory. The results for the EQA/PT samples are compared with target values assigned to the samples to verify that a laboratory’s measurement procedures conform to expected performance. In addition to internal and external QC, the results from patient sample testing (e.g., medians of patient results) can be used to assess and monitor the performance of measurement procedures.

Calibration traceability to a reference system

Chapter 7 describes that calibration of clinical laboratory measurement procedures should, whenever possible, be traceable to a higher order reference measurement procedure (RMP) or certified reference material. Such calibration ensures that results for patient samples are equivalent within medically acceptable limits irrespective of the measurement procedure or laboratory making the measurements. Calibration is provided by the in vitro diagnostic (IVD) manufacturer for commercially available measurement procedures. In the case of a laboratory developed test, the clinical laboratory produces the measurement procedure and is responsible for its calibration hierarchy including traceability to a reference system when available.

Internal QC is not intended to verify that a measurement procedure is calibrated to a higher order reference system. Rather, QC is intended to verify that the performance, for example, the bias and imprecision, of a measurement procedure remains within acceptable limits during use. A clinical laboratory may wish to verify that a measurement procedure’s calibration conforms to an IVD manufacturer’s claim for traceability to the reference system used for a given measurand. Some measurement procedure manufacturers provide materials specifically intended for this purpose. Such materials may be provided as measurement procedure-specific QC materials that typically have matrix characteristics and target values that are intended only for use with the specific measurement procedures claimed in the instructions for use and cannot be used with any other manufacturer’s measurement procedure.

A clinical laboratory has limited resources to verify the calibration traceability of a commercially available or laboratory developed measurement procedure. National and international certified reference materials are available for some measurands. As described in Chapter 7 , certified reference materials can be used to verify calibration when those certified reference materials are commutable with clinical samples for use with a specific measurement procedure. The certificate or published validation of a certified reference material should be reviewed for commutability documentation. A laboratory can split clinical samples with a laboratory that offers an RMP to verify calibration. In most cases, a clinical laboratory is dependent on the IVD manufacturer for metrologic traceability of calibration of measurement procedures.

Third-party QC materials intended for statistical process control (i.e., those provided by a manufacturer other than the routine measurement procedure’s manufacturer) are not suitable to verify calibration traceability. These materials are not validated for commutability with clinical samples for different routine measurement procedures, and they do not have target values that are traceable to higher-order RMPs. Such QC materials are designed to be used as QC samples, with target values and standard deviation (SD) values assigned as described later in this chapter. When third-party QC materials are used in an interlaboratory comparison program with measurement procedure-specific peer group mean values, these values can be used to confirm that a laboratory is using a specific measurement procedure in conformance with other users of the same measurement procedure when the results are not influenced by different reagent lots (see External Quality Assessment or Proficiency Testing section).

Analytical bias and imprecision

Fig. 6.2 illustrates the meaning of bias and imprecision that must be known to develop a QC plan for a measurement procedure. In Fig. 6.2 A, the horizontal axis represents the numeric value for an individual result, and the vertical axis represents the number of repeated measurements with the same value made on aliquots of a QC material. The red line shows the dispersion of results for repeated measurements of aliquots of the same QC material, which is the random imprecision of the measurement. Assuming that the dispersion of results follows a Gaussian (normal) distribution, it is described by the SD. The SD is a measure of expected imprecision in a measurement procedure when it is performing within specifications. Note that results near the mean (average value) occur more frequently than results farther away from the mean. An interval of ±1 SD includes 68% of the measured values, ±2 SDs includes 95% of the values. A result that is more than 2 SDs from the mean is expected to occur 5% of the time (100%–95%) with 2.5% of results in a positive and 2.5% of results in a negative direction from the mean value. Correct calibration of a measurement procedure eliminates systematic bias (within uncertainty limits), so the mean of repeated measurements of a QC sample becomes the expected or target value for that QC sample when the measurement procedure is performing within specifications.

Fig. 6.2 B, illustrates that if the calibration changes for any reason, a systematic bias is introduced into the results. The bias is the difference between the observed mean and the expected value for a QC material (for more discussion on bias, refer to Chapter 2 ). Note that the imprecision is the same as before the bias occurred because it is unlikely, although not impossible, that a change in imprecision would occur at the same time as a bias shift. The primary purpose of measuring QC samples is to statistically evaluate the measurement procedure performance to verify that it continues to perform within the specifications consistent with its acceptable expected stable condition or to identify that a change in performance occurred that needs to be corrected. Acceptance criteria for QC results, discussed in a later section, are based on the probability for an individual QC result to be different from the variability in results expected when the measurement procedure is performing within specifications.

The term accuracy is used for closeness of agreement of an individual result and a true value and is the combination of bias and imprecision that occurred for that specific measurement (refer to Chapter 2 for more discussion on accuracy). The bias for an individual patient sample includes any systematic bias in the measurement procedure and the influence of any interfering substances that could be present in that sample. An individual QC sample is only influenced by systematic bias and imprecision of the measurement procedure. Statistical QC does not evaluate possible interfering substances that may affect results for an individual patient sample. The influence of interfering substances needs to be examined during the evaluation that a measurement procedure is suitable for use (refer to Chapter 5 for additional discussion on interference). However, the imprecision observed for QC results provides a measure of the variability expected for an individual patient result caused by the inherent imprecision of a measurement procedure and is usually independent of interfering substances that typically affect the bias for an individual patient result.

The term trueness is inversely related to a bias that may be present in a measurement procedure. Trueness is an attribute of a measurement procedure that reflects how correctly its calibration is traceable to a reference system.

Fig. 6.3 shows a Levey-Jennings plot that was an adaptation for clinical laboratory measurements of the Shewhart plot developed for statistical process control in manufacturing. The Levey-Jennings plot is the most common presentation for evaluating QC results. This format shows each QC result sequentially over time and allows a quick visual assessment of performance. Assuming the measurement procedure is performing in a stable condition consistent with its specifications, the mean value represents the target (or expected) value for the QC result, and the SD lines represent the expected imprecision. Assuming a Gaussian (normal) distribution of imprecision, the results should be distributed uniformly around the mean with results observed more frequently closer to the mean than near the extremes of the distribution. Note that a few results in Fig. 6.3 are greater than 2 SDs, and two results slightly exceed 3 SDs, which is expected for a Gaussian distribution of imprecision. For a large number of repeated measurements, the number of results expected within the SD intervals is as follows:

• ±1 SD = 68.3% of observations

• ±2 SD = 95.4% of observations

• ±3 SD = 99.7% of observations

Interpretation of an individual QC result is based on its probability to be part of the expected distribution of results for the measurement procedure when the procedure is performing correctly. A later section provides details regarding interpretive rules for evaluation of QC results. Note that evaluation of individual QC results may be performed by computer algorithms without visual examination of a Levey-Jennings chart. However, the underlying logic of such algorithms is illustrated by the Levey-Jennings chart example.

Performance of a measurement procedure for its intended medical use

It is necessary to determine how the performance of a measurement procedure relates to the intended medical use for interpreting results in order to determine the frequency to measure QC samples and the criteria to use to evaluate the QC results. The sigma metric is commonly used to assess how well a measurement procedure performs relative to the analytical performance specifications that ideally should be based on the intended medical use of the results. Sigma is the Greek letter used to denote SD. The sigma scale expresses the variability of a measurement process in SDs in relation to the variability that is acceptable because it will not cause an error in diagnosis or treatment of a patient.

For laboratory measurements, the sigma metric is calculated as:

where TE _a is the total error allowed based on analytical performance specifications that ideally should be related to the intended medical use for interpreting results, and bias and SD refer to performance characteristics of the measurement procedure. The SD is estimated from the QC data as previously described. It is critically important that the estimate of SD be made using QC data that represent all or most components of variability that occur over an extended time period (see the section called Establishing the Quality Control Target Value and Standard Deviation That Represent a Stable Measurement Operating Condition). The bias is difficult for a laboratory to estimate because it is difficult to evaluate if a particular measurement procedure has a bias compared with a reliable estimate of a true value such as based on an RMP. For internal QC, a laboratory is usually interested to determine if a bias has occurred compared with the condition established by calibration of a measurement procedure. Such a bias represents a QC result that is sufficiently different from its target value that corrective action is needed. Consequently, the bias is usually assumed to be zero for calculating sigma.

However, a bias term may be needed in situations when there are two or more different measurement procedures used for the same measurand and those different measurement procedures have a bias between them that cannot be removed, or when changes in lots of reagents or calibrators introduce shifts in bias that cannot otherwise be corrected. Note that it is preferable to adjust the calibration of different measurement procedures or different lots of reagents or calibrators to provide equivalent results, but this solution may not be applicable for some technologies. In such cases, this relative bias can be estimated based on comparison of results for patient samples following a procedure such as described in Clinical and Laboratory Standards Institute (CLSI) document EP9 . That bias should be considered in determining the sigma metric and in establishing a QC plan for such measurement procedures.

TE _a represents the measurement procedure performance required to enable suitable medical decisions based on a test result. A test result may be used for different medical decisions in different disease conditions. In a main lab setting where samples from different medical practices are measured, the most stringent decision parameter should be used as the basis for the TE _a . In a setting where the samples are used for one specific clinical situation, for example, in a point-of-care (POC) setting, the medical requirements of the setting can be used as the basis for the TE _a . TE _a can be estimated using three models. The preferred model (model 1) to set a performance specification is to base it on an outcome study (i.e., investigating the impact of analytical performance of the measurement procedure on the clinical outcome). Outcome studies can be direct assessment of clinical outcome for a group of patients or “indirect” outcome when the consequences of analytical performance on, for example, clinical classifications or decisions and thereby on the probability of patient outcomes can be investigated. These probabilities can be discussed with clinical experts who then can recommend a performance specification.

Indirect outcome studies are often used to set TE _a in laboratory practice guidelines. For example, the National Cholesterol Education Program recommends that total cholesterol be measured with a TE _a of 9% or less, and the National Kidney Disease Education Program recommends that creatinine be measured with a TE _a of less than 7 to 10% in the concentration interval 1 to 1.5 mg/dL (76.3 to 114.4 mmol/L). The limitation of this model is that it is only useful when the links between the measurand, clinical decision-making, and clinical outcomes are strong, which is the case for a minority of measurands.

Another model (model 2) tries to minimize the ratio of the “analytical” noise to the “biologic signal” with an assumption that a small ratio will identify measurement procedure performance that relates to the medical requirements. The biological variation is composed of within and between subject variation. Performance specifications for imprecision, bias, and TE _a are based on a fraction of the within and between individual biologic variations of the measurand. ^, Tables of optimal, desirable, and minimal TE _a based on biologic variation are available and may provide useful information. ^, However, biologic variation–based estimates of TE _a should be used with caution because the estimates of biologic variation in many cases are based on limited data, and the experimental designs of the estimates and the process to select the estimates to list in the tables have been challenged. Estimates of biologic variation typically vary among different investigations. The newly established EFLM database on biological variation evaluates published reports on biological variation using a critical appraisal checklist and calculates point estimates with confidence intervals for each measurand after a meta-analysis of eligible reports. In addition, the way the TE _a is calculated is flawed because the calculation combines maximum allowable imprecision with maximum allowable bias (both based on a fraction of biologic variation) that has no theoretical basis and leads to overestimation of the TE _a. Another limitation is that the biologic variability has typically been derived from data for nondiseased individuals and may be different for pathologic conditions. Additional examples and discussion of biologic variation are provided in Chapter 8 .

Model 3 bases the performance specifications on the “state of the art.” The advantage of this model is that data are readily available from QC and EQA/PT information. The disadvantage is that there may be no relationship between what is technically achievable and what is needed to make a medical decision for diagnosis or treatment of a patient. It is generally agreed that preference should be given to model 1 whenever such information is available or to model 2 as a starting point to estimate TE _a . ^, A laboratory director should consult with clinical care providers to agree on an appropriate TE _a for the patient population served. An extended presentation of analytical performance specifications is given in Chapter 8 .

Because sigma assumes a Gaussian or normal distribution for repeated measurements, the probability of a defect (i.e., an erroneous laboratory result) can be predicted as shown in Table 6.1 . The sigma metric represents the probability that a given number of erroneous results that could cause risk of harm to a patient are expected to occur when the test measurement procedure is performing to its specifications. The phrase “six sigma” refers to a condition when the variability in the measurement process is sufficiently smaller than the medical requirement that erroneous results are very uncommon. A “four-sigma” measurement procedure would be less robust and have a higher probability that erroneous results could be produced but still at a fairly low frequency. A “two-sigma” measurement procedure would produce enough erroneous results even though it met its performance specifications that it would not be very reliable for patient care.

Fig. 6.4 shows how the sigma metric describes the performance of a laboratory test relative to the TE _a . Parts A and B show that a measurement procedure with the same analytical performance characteristics can have different sigma metrics depending on how the imprecision relates to the TE _a . Fig. 6.4 A shows a “six-sigma” measurement procedure that has the TE _a limits 6 SDs away from the center point of the distribution of variability in measurements when the measurement procedure is performing to its analytical specifications. In the “six-sigma” situation, a small amount of bias or increased imprecision will have little influence on the number of erroneous results produced, and less stringent QC will be appropriate because the risk of producing an erroneous result even with some loss of performance is very low.

Fig. 6.4 B shows a “three-sigma” measurement procedure that has the TE _a limits 3 SDs away from the center point of the expected distribution of variability in measurements when the measurement procedure is performing to its analytical specifications. In the “three-sigma” situation, a small amount of bias or increased imprecision will cause the number of erroneous results to increase substantially, and more stringent QC is needed to identify when such an error condition occurs so that corrective action can be initiated. Note that no amount of QC will improve the performance of a marginal measurement procedure. However, more frequent QC and more stringent acceptance criteria will allow the laboratory to more quickly identify when small changes in performance occur so they can be corrected to minimize the risk of harm to a patient from erroneous results being acted on to make medical care decisions. It is important to emphasize that the sigma calculations are dependent on what TE _a is chosen. As discussed earlier, an “objective” TE _a is often difficult to establish, and good data to set a TE _a are often lacking.

Selection of quality control materials

Generally, two different concentrations are necessary for adequate statistical QC. For quantitative measurement procedures, QC materials should be selected to provide measurand concentrations that monitor the analytical measuring interval of the measurement procedure. In practice, laboratories are frequently limited by concentrations available in commercial QC products. When possible, it is important to confirm that measurement procedure performance is stable near the limits of its analytical measuring interval because defects may affect these concentrations before others. Many quantitative measurement procedures have a linear response over the analytical measuring interval, and it is reasonable to assume that their performance over the interval is acceptable if the results near the interval limits are acceptable. In the case of nonlinear analytical response, it may be necessary to use additional controls at intermediate concentrations. Concentrations of control materials close to clinical decision values (e.g., glucose, therapeutic drugs, thyroid-stimulating hormone, prostate-specific antigen, hemoglobin A _1c [HbA _1c ], troponin) may also be appropriate for additional QC monitoring. In many cases, the imprecision near the limit of quantitation may be relatively large, in which case the concentration should be chosen to provide adequate SD for practical evaluation of QC results. For procedures with extraction or other pretreatment steps, controls must be used that are subject to the same pretreatment steps.

This chapter primarily focuses on QC procedures for quantitative measurement procedures. However, the principles can be adapted to most qualitative procedures with allowances for the lack of numeric results. For measurement procedures based on qualitative interpretation of quantitative measurements (e.g., drugs of abuse, human chorionic gonadotropin, hepatitis markers), the same principles of QC assessment can be applied to the numeric results even if they are only expressed as instrument signal values. For qualitative results, the negative and positive controls should be selected to have concentrations relatively near the clinical decision threshold to adequately control for discrimination between negative and positive. For qualitative procedures with graded responses (e.g., dipstick urinalysis), negative, positive, and graded response controls are required. For qualitative tests based on other properties (e.g., electrophoretic procedures, stain adequacy, immunofluorescence, organism identification), it is necessary to ensure that the QC procedure will appropriately evaluate that the measurement procedure correctly discriminates normal from pathologic conditions.

The QC samples selected must be manufactured to provide stable materials that can be used for an extended time period, preferably 1 or more years for stable measurands. Use of a single lot for an extended period allows reliable interpretive criteria to be established that will permit efficient identification of a measurement procedure problem, avoid false alerts caused by poorly defined expected ranges for the QC results, and minimize limitations in interpreting values after reagent and calibrator lot changes.

Limitations of quality control materials

Limitations are inherent in most commercially available QC materials. One limitation is that the QC material is frequently noncommutable with patient samples. Commutability is a property of a reference or control material that refers to how well that material mimics patient samples in measurement procedures. A commutable QC material is one that reacts in a measurement procedure to give a result that would closely agree with that expected for a patient sample containing the same amount of measurand. Fig. 6.5 A shows that results from patient samples and from commutable QC (or EQA/PT) samples have the same relationship between two measurement procedures or between two reagent lots used with the same measurement procedure. Fig. 6.5 B shows that noncommutable samples have a different relationship than observed for patient samples.

QC, as well as EQA/PT materials, are typically noncommutable with patient samples because the serum or other biologic fluid matrix is usually altered from that of a patient sample. The matrix alteration is due to processing of the biologic fluid during product manufacturing; use of partially purified human and nonhuman additives to achieve desired concentrations of the measurands; and various stabilization processes that alter proteins, cells, and other components. The impact of the matrix alteration on measurement of a measurand is not predictable and is frequently different for different lots of QC material, for different lots of reagent within a given measurement procedure, and for different measurement procedures. ^, Because of the noncommutability limitation, special procedures are required (discussed in later sections) when changing lots of reagent or comparing QC results among two or more measurement procedures.

A second limitation of QC materials is deterioration of the measurand during storage. Measurand stability during unopened storage is generally excellent, but slow deterioration eventually limits the shelf life of a product and can introduce a gradual drift into QC data that may require correction over the life of a lot of QC material. Measurand stability after reconstitution, thawing, or vial opening can be an important source of variability in QC results and can vary substantially among measurands in the same vial. Variables to be controlled are the time spent at room temperature and the time spent uncapped with the potential for evaporation. An expiration time after opening is provided by the QC material manufacturer but may need to be established by a laboratory for each QC material under the conditions of use in that laboratory and may be different for different measurands in the same QC product. For QC materials reconstituted by adding a diluent, vial-to-vial variability can be minimized by standardizing the pipetting procedure (e.g., using the same pipet or filling device, preferably an automated device, and having the same person prepare the controls) whenever practical.

Another limitation of QC materials is that measurand concentrations in multiconstituent control materials may not be at levels optimal for all measurement procedures. This limitation may be caused by solubility considerations or potential interactions between different constituents, particularly at higher concentrations. It may be necessary to use supplementary QC materials to adequately monitor the measuring interval and clinically important decision concentrations.

Frequency to measure quality control samples

Determining the frequency to measure QC samples should use a risk assessment approach. The frequency to measure QC samples is a function of several parameters:

• Analytical stability of the measurement procedure

• Risk of harm to a patient from clinical action being taken before a significant error is detected at the next scheduled QC event

• Number of patient results produced in a period of time when an error condition could exist but is not yet detected

• Scheduled events such as recalibration or maintenance that may alter the current performance condition of the measurement procedure

• Training and competency of the test operator, particularly for manual or semiautomated measurement procedures