The Role of Microsimulation Modeling in Evaluating the Outcomes and Effect of Screening

A Brief History of Microsimulation Modeling

Generally, Guy Orcutt, is acknowledged as providing the foundation for the field of microsimulation in 1957, by suggesting that microsimulation models consisting of “various sorts of interacting units which receive inputs and generate outputs” could be used to investigate “what would happen given specified external conditions.” However, it took some time before microsimulation modeling was widely used, mainly because of limitations in computing power and the lack of suitable data. With the rise of technology and data collection, these limitations were gradually overcome, and the use of microsimulation increased. Since then, microsimulation models have been used extensively for evaluating social and economic policies, such as income tax, social security, health benefits, or pension policies. Microsimulation of industrial processes to explore ways to improve them is also common. And microsimulation also underlies some newly popular methodologies for data analysis, specifically multiple imputation for dealing with missing data, and Metropolis-Hastings or Gibbs sampling for Bayesian data analysis.

The use of models to answer health policy questions began to evolve in the 1980s with application in a wide range of health care topics, such as hospital scheduling and organization, communicable disease, screening, costs of illness, and economic evaluation. For breast cancer screening, specifically, models were developed to extrapolate the findings of randomized trials. For example, when long-term outcomes from breast cancer screening trials were not yet available, models were used to predict the lifetime benefits, risks, and costs. Since the 1980s, the breadth of application, diversity, and complexity of model approaches have rapidly increased.

What Is Microsimulation Modeling?

According to Wikipedia, “ microsimulation (from microanalytic simulation) is a category of computerized analytical tools that perform highly detailed analysis of activities such as highway traffic flowing through an intersection, financial transactions, or pathogens spreading disease through a population. Microsimulation is often used to evaluate the effects of proposed interventions before they are implemented in the real world .”

An important/distinguishable feature of microsimulation models is that individuals (or individual components) are simulated. Typically, with microsimulation modeling, a life history is simulated for each individual using an algorithm that describes the events that make up that life history. Random draws decide if and when specific events occur. For health applications, these events typically include birth and death from other causes, as well as a model part that describes the target disease events, such as diagnosis and death from disease. Because individuals are simulated, microsimulation can produce individual-level artificial data that can be analyzed using standard statistical techniques, while population-level outcomes are obtained by aggregating individual life histories. The impact of changes in influential factors on key outcomes for subgroups as well as aggregates of the population can thus be assessed by testing “what if” scenarios. A disadvantage of microsimulation modeling is that it tends to be computationally intensive. In order to obtain stable estimates of stochastic outcomes, a large number of individuals must be simulated, particularly if small effects need to be estimated with precision.

Description of Some Available Models

To evaluate the effects of breast cancer screening several microsimulation models have been developed. We will describe some of the available microsimulation models below. A comprehensive review of all such models would be impractical, so we have selected those we feel have been most influential.

In 1980, Eddy articulated a mathematical theory of screening that closely resembles the approach used by most modern simulation models. He developed a Markov process-based simulation model, CAN*TROL, which he applied to the evaluation of cancer control policies for several cancer sites. He used his model to advise numerous countries about cancer screening programs. Although the underlying mathematical screening is similar to that used by other models reviewed in this chapter, the CAN*TROL model itself is not a microsimulation model.

One of the first microsimulation models, the “MIcrosimulation SCreening ANalysis” (MISCAN) model, was developed to examine the impact of cancer screening on morbidity and mortality. The MISCAN model was developed to evaluate mass screening for breast cancer, as well as for other types of cancers, such as cervical, prostate, and colorectal cancer. It is a microsimulation model with continuous time and discrete events that simulates a dynamic population. The simulated life histories include date of birth and age at death from other causes than the target disease, a preclinical, screen-detectable disease process, in which breast cancer progresses through states, and survival. The natural course of the disease may be influenced by screening when a preclinical lesion becomes screen-detected. Screen-detection can result in the detection of smaller tumors, which may entail a survival benefit.

Within the Cancer Intervention and Surveillance Modeling Network (CISNET), many models have been developed for different cancer sites. For breast cancer, seven models were developed and extensively discussed and compared. Six of the seven models were classified as microsimulation models and six of the seven models include a natural history component. The natural history of breast cancer is modeled with a time from birth to start of the preclinical screen detectable period (the duration of this period being known as the sojourn time), clinical diagnosis, and a period of survival that ends with either breast cancer death or death from other causes. The time of diagnosis can change if screening takes place during the preclinical screen detectable period, which, in turn, may alter the time and cause of death. In most models, therapy influences survival from time of diagnosis.

Despite the similarities, the models also differ in important ways, for example in their assumptions about sojourn time, the mechanism of screen detection, tumor characteristics at diagnosis, survival following clinical detection and survival following screen detection. Most CISNET models include ductal carcinoma in situ (DCIS), but one model includes only invasive breast cancer. Also, assumptions on nonprogression vary: several models specifically assume that some percentage of DCIS are nonprogressive; one model also assumes that some cases of small invasive cancer are nonprogressive. Some groups model breast cancer progression in discrete stages, whereas other models explicitly model continuous tumor growth. The models also differ as to whether treatment affects the hazard of death from breast cancer, results in a cure for some fraction of cases, or both.

Besides these models, many models have been developed over the past years, often to estimate the effects of breast cancer screening for specific countries, and often based on previously developed models. For example, models have been developed to determine the potential costs and effects of a breast cancer screening program for New Zealand as well as Slovenia. For both, it was explicitly stated that the model used was similar to MISCAN. In Spain, a model based on the Lee-Zelen approach (one of the CISNET models) was used to estimate the amount of overdiagnosis from breast cancer screening, as well as to estimate the cost-effectiveness of risk-based screening approaches and the cost implications of switching from film to digital mammography. A similar approach was used to estimate the cost-effectiveness of breast cancer screening in Korea. In addition, specific models have been developed to estimate overdiagnosis in France, Denmark, and the United Kingdom.

Model Development: Conceptualization and Parameter Estimation

Most models are developed to answer specific questions and the development of a model is ideally guided by an understanding and conceptualization of the problem that is being represented. The conceptualization of the problem involves addressing factors such as the disease or condition, patient population, analytic perspective, alternative interventions, health and other outcomes, and the time horizon.

With regard to the disease or condition, typically, a single disease (eg, breast cancer) or set of closely related diseases is of interest, but sometimes it is necessary to consider multiple diseases, for example if one is interested in estimating the impact on life expectancy of changes in a factor, such as smoking, that influences risk of multiple diseases. The modeled patient population should reflect the population of interest in terms of features relevant to the decision (eg, geography and patient characteristics, including comorbid conditions, disease prevalence, and stage). Analytic perspectives commonly considered are those of the patient, the health plan or insurer, and society. With regard to the diagnostic or therapeutic actions and interventions, it is crucial to model all practical interventions and their variations, because the choice of comparators has a major impact on estimated effectiveness and efficiency, and the results are only meaningful in relation to the set of interventions considered. In addition, the specific implementations of interventions might differ across countries and often across settings or subpopulations within countries. Thus, despite the same label (eg, “breast cancer screening”), effects may differ, depending on the practice patterns in the target population area. Sources of variation include the technology used (plain film vs digital), criteria for recall and biopsy recommendations, skill level, work load, and operating point of the radiologist, and differences in the underlying epidemiology of breast cancer itself. Moreover, the benefits of screening also vary with the range of available treatments and their accessibility to the population under study. It is therefore advisable to specify the components of the intervention in detail so that users can determine how well the analysis reflects their situations. Health outcomes of the model should be directly relevant to the question posed and might include number of cases of disease, number of deaths (or deaths averted by an intervention), life years gained, quality-adjusted life years (QALYs), and costs. The time horizon of the model should be long enough to capture relevant differences in outcomes across strategies; a lifetime time horizon is often required.

In order to make the appropriate choices on all these factors the modeling team should consult widely with clinical, epidemiologic, and health services experts and stakeholders to ensure that the model represents disease processes appropriately and adequately addresses the decision problem.

Once the problem has been clearly conceptualized, conceptualizing the model is the next step. To convert the problem conceptualization into an appropriate model structure, ensuring it reflects current disease knowledge and the process modeled, several methods might be used (expert consultations, influence diagrams, concept mapping). Several problem characteristics should be considered to decide which modeling method is most appropriate: Will the model represent individuals or groups, are there interactions among individuals, what time horizon is appropriate, should time be represented as continuous or discrete, do events occur more than once, and are resource constraints to be considered?

Another dimension is how complex a model should be. The appropriate level of model complexity depends on the disease process, the scientific questions to be addressed by the model, and data available to inform model parameters. There is a tension between the simplicity of a model and the complexity of the disease process. Simpler models include fewer parameters making it more likely that model parameters can be estimated using observed data. Simpler models are easier to describe, may entail fewer assumptions, and are less likely to contain hidden errors, making them more transparent. However, more complex models can be used to address a wider range of scientific questions. One approach is to begin with relatively simple models, extending these as needed to address specific research questions.

Once the model conceptualization has been finalized, parameter estimation can begin. All models have parameters that must be estimated. In doing so, when possible, analysts should conform to evidence-based medicine principles; for example, seek to consider all evidence, rather than selectively picking values that best accord with the modeler’s preferences; use best-practice methods to avoid potential biases, as when estimating treatment effectiveness from observational sources; and employ formal evidence synthesis techniques.

Some parameters might be estimated from observed data using various analytical techniques to produce a result that will be used as a direct model input. For example, data on treatment efficacy might be used by combining results from several trials on treatment effects, and using the obtained hazard rate reduction directly in the model.

Often, other parameters have to be estimated using the model itself. Typically, unknown model parameters are estimated by optimizing the fit between simulated and observed data, using, for example, least-squares, minimum chi-square, or maximum-likelihood methods. This process is usually referred to as calibration. Typically this process is used for model parameters that quantify unobservable aspects of the process being modeled. For example, a model might include the mammographic detection probability of a preclinical tumor. Such a parameter might be estimated by optimizing fit between simulation output and observed rates of incidence of screen-detected and interval (ie, between scheduled mammograms) cancers in a clinical trial. Fitting to data is well defined when a model is built for a particular intervention, and there are one or more randomized trials in which the effects of the intervention are reported in terms of end results and intermediate outcomes, which can be compared with model predictions. When observed data comes from multiple studies and disease registries, expert judgment is required for assessing data quality and prioritizing data for their importance in fitting.

Validation

Evaluation and optimization of cancer screening programs or scenarios using modeling is only useful if the estimated harms and benefits of screening are reasonably accurate. Testing the accuracy of models, that is, model validation, is, therefore, a crucial step in any modeling study.

Previously, five main types of validation have been distinguished: face validity, verification (or internal validity), cross validity, external validity, and predictive validity. Face validity is the extent to which a model and its assumptions and applications correspond to current science and evidence, as judged by people who have expertise in the problem. In other words, face validity exists when a plain explanation of the model structure and assumptions appears reasonable to a critical audience.

Verification (or internal validity) addresses whether the model’s parts behave as intended and the model has been implemented correctly. This can, for example, be tested by assessing whether the model responds as expected in simplistic boundary conditions such as zero incidence, no other-cause mortality, no benefit from screening or treatment, and other (typically contrived) sets of input parameters where expected model results can be determined beforehand.

Cross validation involves comparing a model with others and determining the extent to which they calculate similar results. If similar results are calculated by models using different methods, confidence in the results increases, especially if the models have been developed independently.

In external validation, a model is used to simulate a real scenario, such as a clinical trial, and the predicted outcomes are compared with the real-world ones. Ideally, external validation tests the model’s ability to reproduce observed results in a validation data set that was not used as a source of parameter estimates or calibration while building the model. Given, however, that breast cancer screening models typically include parameters that vary from setting to setting, or among subpopulation groups, some of them not directly observable, it is usually difficult to completely avoid some degree of reliance on the validation data. Completely independent external validation requires that the variable aspects of screening, natural history of disease, and treatment noted in the previous section are the same as those which prevailed for the data sources from which the model was built. For this reason, external model validation of breast cancer screening models is typically imperfect. In addition, even a fully independent external validation in a single data set, or a small number of them, should not, by itself, be taken as a definitive endorsement of the model: the number of “degrees of freedom” in any model complex enough to approach a realistic data generating mechanism will always be too large for one or a handful of fits to data to confirm.

Predictive validity involves using a model to forecast events and, after some time, comparing the forecasted outcomes to the actual ones. On the one hand, predictive validation is the most desirable type of validation, as it corresponds best to the purpose of most models, which is predicting what will happen. On the other hand, the observations are by definition in the future, making it often difficult to test a models predictive validity in a sufficiently timely manner to make practical use of its results. Predictive validity can sometimes be estimated by observing the model’s performance in simulating the results of somewhat similar events in existing data sets. For example, if the question of interest concerns the outcomes associated with different intensities of screening, it may be relevant to check how well the model can reproduce the changes in incidence and mortality that occurred during a time when a similar population introduced screening and its use disseminated.

Interpretation

The interpretation of observational studies, and sometimes that of clinical trials, is made difficult by the fact that in the real world, influential factors other than the object of study are often changing too. For example, during the same period that mammography screening became widely available and widely used in the United States, new forms of adjuvant therapy for breast cancer were also introduced, and the use of adjuvant chemotherapy became more common. When systematic screening programs are introduced, changes in incidence and mortality may also be influenced by concurrent changes in risk factor prevalence or treatment patterns. Differences in breast cancer outcomes between different populations can seldom be ascribed to a single cause: typically the prevalences of different risk factors, utilization and implementation of screening, and treatment patterns will all differ. Ordinary statistical analyses, such as regression equations, have at best limited ability to accommodate such confounding variables. Simulation models, however, because they can represent complicated and multifaceted data generating processes, can disentangle the concurrent effects and support attribution of portions of the observed effects to each cause by providing projections of the expected outcomes under both real and counterfactual combinations of these factors.