Evaluation of surgical evidence

Population or target condition

A study population is a group of participants selected from a general population based on specific characteristics. Having a clearly defined study population is key to ensuring research is clinically relevant and can answer a specific question. If a target condition is used as the basis for selecting a population, validated diagnostic criteria should be applied to identify such a population.

The basis for the inclusion of patients in a study should be systematic, to avoid bias. The best way of ensuring a sample is both representative and free from bias is to approach every eligible patient in a consecutive manner. This is often referred to as a consecutive sample.

Specific study populations may have special considerations that must be considered in the study design. An example of this may be an older age group, where visual or hearing impairment may present difficulties with particular data collection methods, such as telephone interviews.

Intervention or exposure

The intervention is the main variable changed within the treatment group. In observational research, patients and clinicians decide which treatment will be received. As no direct experimental intervention occurs, the variable is termed the ‘exposure’.

When considering an intervention for surgical research, particular care must be given to standardisation between patients and clinicians. Delivering interventions in a standardised manner is essential to ensure patients are comparable. Variation in how a treatment is delivered should be considered during the design process. For example, in a study of a new surgical technique or approach, training in the delivery of the new intervention requires both time and some means of ensuring all patients receive a similar treatment. Standardising the surgical intervention may include an assessment within the research study to determine whether an acceptable level of competence has been achieved. A good study protocol will help to address this (more about Study protocols can be found in the Bias section).

The acceptability of the intervention must also be given due thought. If an intervention is not acceptable to patients, it will be very difficult to convince patients (and research ethics committees) to participate. One means of ensuring interventions are acceptable is to involve patient representatives when designing research studies. These patient representatives are part of the study design team and provide feedback to investigators on the best means of ensuring studies are conducted in a feasible and acceptable manner.

Comparison

The comparison describes the control group the intervention or exposure of interest is being evaluated against. This group should be sufficiently comparable to the intervention group to ensure valid conclusions may be drawn about the true effects of the any exposure or intervention.

In controlled trials, the comparison group often uses a placebo to reduce bias and maintain blinding. In many studies, the comparison group receives the current gold standard of clinical care, to create a group where the new treatment can be compared directly with the current best or usual therapy.

If the control group differs from the intervention group, bias may be introduced leading to invalid conclusions. This is of particular concern in observational research, such as case-control studies, where the comparison group is selected by investigators.

Outcome

A study outcome is the variable by which the effect of the intervention or exposure of interest is measured. For an outcome to be useful, several properties of the chosen measure should be considered:

• Incidence: The outcome of interest should be sufficiently common. The more common the outcome of interest is, the smaller the sample size required to demonstrate a difference between treatment groups.

• Directness: The outcome of interest should directly measure what the intervention is ultimately intended to achieve. For example, intraoperative warming devices are designed to keep the patient warm, thereby reducing postoperative complications. Therefore, any study investigating the efficacy of intraoperative warming devices should use postoperative complications as a primary outcome rather than looking at differences in temperature (termed a ‘surrogate’ outcome).

• Definition: All outcomes should be measured using clearly defined criteria. Preferably these criteria would be used widely across the field, so all studies are measuring the same outcome in the same way. One example of such criteria would be the Response Evaluation Criteria In Solid Tumours (RECIST) criteria for the assessment of tumour progression.

• Relevance: Outcomes should be relevant to the research question and to patient care. What may be relevant to clinicians may not also be relevant to patients. To this end, patient representatives should be consulted when selecting study outcomes. Often quality of life and functional outcomes are of primary concern to patients.

• Timing: Outcomes must be measured at appropriate time intervals, which are relevant to the timeframe where the outcome of interest would be expected to occur. For example, measuring an outcome at 72 hours following a procedure may be relevant for postoperative bleeding, but not for surgical site infection.

• Reliability: The chosen outcome measure should be able to reliably detect an event and should be standardised for all patients. This is particularly important for multicentre research, or large-scale trials where multiple observers are judging outcomes. Here having a standardised means of assessing an outcome is important, as it must be ensured that all patients are being assessed in the same way, otherwise the results of the study may be inaccurate.

Patient-reported outcome measures (PROMs) are outcomes which are reported by patients themselves. This contrasts with outcomes where an investigator or clinician reports an outcome. Example of PROMS may include validated general quality of life questionnaires (such as the EQ-5D or HR-QoL). Alternatively, PROMs may focus on specific areas of interest such as sexual function and continence, which are of particular importance in pelvic surgery. ^,

Cost-effectiveness is sometimes used as an outcome measure. This is often used by clinical governance bodies to decide whether a treatment represents value for money and should be recommended for use.

Occasionally, different outcomes may be combined to form a single measure. This is known as a composite measure. The use of composite outcome measures can enable researchers to measure two or more outcomes simultaneously and gives better statistical efficiency for the number of patients in a study. An example may be unplanned critical care admission or death, where both represent a major complication, but they are pooled to increase the number of events, which reduces the sample size required for demonstrating the effectiveness of a treatment. While this is beneficial, it can result in outcome measures that prove challenging to interpret.

Study designs

There are many study designs available to answer clinical questions. Study design methodology is constantly evolving; however, all types of study can be broadly divided and sub-divided into the following categories:

• Primary research (research at the individual patient level)

  • Randomised trial

  • Prospective cohort study

  • Retrospective cohort study

  • Cross-sectional study

  • Case-control study

  • Case series

  • Case report

• Secondary research (research that considers multiple sources of primary research)

  • Systematic review

  • Systematic review with meta-analysis

Figure 1.1 below provides a useful means of identifying the type of study where a report may not be explicit as to its design:

Another way of classifying evidence is according to the Oxford Centre of Evidence-based Medicine (CEBM), which divides evidence according to the risk of bias. These are commonly referred to as the ‘Level of evidence’ ( Table 1.1 ).

One well-known way of classifying these studies is using the ‘pyramid of evidence’, which provides a simplified means to consider the advantages of one study design over another. This pyramid considers the inherent properties of each study design and classifies them according to how likely the study is to provide a reliable answer to the study question, as close to the ‘true value’ as possible.

The pyramid of evidence is accompanied by several caveats ( Fig. 1.2 ). First, not all questions can be answered by a randomised controlled trial either due to lack of equipoise (where it would be unethical to conduct a trial) or for logistical reasons (where a trial would be too expensive or simply unfeasible). Second, a poorly conducted study may give an inaccurate, or unreliable answer compared with a well-conducted study of a different design. For example, a small poorly conducted randomised trial may not provide a better answer to a question than a large, well-conducted prospective cohort study. Here is where the Oxford CEBM classification and Grading of Recommendations Assessment, Development and Evaluation (GRADE; discussed later) assessments are useful to help determine whether there can be certainty in the body of evidence for a given research question. ^,

Systematic reviews and meta-analysis

At the top of the pyramid are systematic reviews and meta-analyses, which take multiple sources of evidence for a given clinical question. These sources may take the form of randomised trials, cohort studies or even case-control studies ( Fig. 1.3 ). Systematic searching methods are used to attempt to find every possible study which may contain the answer to the clinical question of interest. These methods often include searching multiple databases, searching the references of key articles in the field and contacting experts to ask if they have suggestions for potential studies to include.

Once all studies that may be eligible are screened, each study is critically appraised individually (see Critical Appraisal). This approach of systematically identifying every possible study, followed by scrutiny of the study methods is then synthesised into the final review, which should present an overarching and balanced view of the evidence for a given clinical question.

Meta-analysis describes the use of statistical methods to combine the numerical results of similar studies in order to derive an estimate of how well the intervention of interest works (i.e. the treatment effect). This is often combined with a systematic review where the results of studies are combined after being screened for inclusion. Combining studies in this manner is referred to as ‘pooling’.

An important part of meta-analysis is an assessment of the similarity between different studies of the same clinical question. Variation between studies is termed heterogeneity and takes two forms, clinical and statistical heterogeneity. Clinical heterogeneity concerns how clinically similar a population in one study is to another. It is unlikely to make sense to combine the results of two trials examining the same treatment in two distinct populations, for instance, adults and children. Statistical heterogeneity refers to the differences in the actual results of included studies and whether these are likely to be due to chance (sampling error) or true differences in outcome. This is summarised by the I-squared statistic and is 0% when all included trials provide a similar result. If studies show completely contradicting results, the I-squared statistic is 100% meaning the combined result likely lacks any real meaning. A rule-of-thumb for the interpretation of the I-squared statistic is as follows:

• 0–30%: Low level of statistical heterogeneity

• 30–50%: Moderate level of statistical heterogeneity

• 50–100%: Substantial level of statistical heterogeneity

Randomised controlled trials

Randomised controlled trials are primary research studies that test the effect of an intervention using experimental methods. After participants are recruited, they are allocated to treatment groups at random, where the choice of treatment is not determined by the patient, clinician or any other person. Removing choice of treatment reduces the risk of selection bias, which is often the largest source of bias in medical research. Selection bias simply refers to how doctors and patients select treatments based upon certain patient characteristics.

The controlled part of the name refers to two integral components of high-quality trials. The first is that there is a comparable control group with which to compare the intervention of interest. The second is that the trial is conducted in strict accordance with a pre-defined study protocol.

Study protocols are important and are essentially a detailed instruction manual outlining how a piece of research will be conducted. The main advantages of a study protocol are that it ensures investigators stick to a pre-planned analysis of their data and in the case of multicentre research, promotes standardisation across centres. Study protocols are not unique to randomised controlled trials. It is highly recommended that all clinical research (including systematic reviews and cohort studies) should be conducted according to a pre-defined protocol. Many journals now stipulate this as a requirement for even considering a study for publication.

Randomised controlled trials can be grouped into two categories:

Randomisation

Randomisation allocates patients to treatment groups without the clinician or patient choosing the treatment.

If done correctly, randomisation aims to ensure treatment groups are equally balanced for confounding factors which may influence outcomes, both observed factors and unobserved factors. This enables us to draw fair comparisons between two treatments.

Randomisation ensures treatment groups are balanced for characteristics and confounding factors, both of which may be observed (data collected upon them) or unobserved (no data collected on or not yet discovered). Randomisation may be performed in two different ways:

Blinding

Blinding is the act of disguising which treatment a participant in a research study has received. Blinding in surgical studies often proves to be difficult, and requires some flair or creativity to identify a suitable strategy for implementing blinding successfully.

There are four main parties which can be blinded to an intervention or an outcome:

• Patients

• Those administering the treatment, i.e. the clinical team

• Those who are measuring the effect of the treatment, i.e. observers

• Those performing the data analysis, i.e. the statistician

In the past, where one of these groups was blinded the study was termed ‘single-blind’, where two groups were blind ‘double-blind’ etc. Nowadays it is best practice to specify explicitly who was blinded and what exactly was done rather than use terminology that may be unclear without qualification.

In studies where drug administration is involved, participants and clinicians can be blinded by using placebos (i.e. ‘dummy pills’). Where intravenous drugs are used, normal saline could be used in replacement for a drug. Coloured intravenous fluids may be masked by black tubing. If groups are receiving a different surgical procedure altogether, it can be difficult or impossible to blind patients and is impossible to blind the surgeon to the procedure. There are, however, means to address this.

Observers are often used in surgical studies where blinding patients or clinical teams is impossible. These observers measure outcomes but are independent from the clinical team and are blinded to which intervention the patient received.

In all these cases, blinding enables the person who is assessing the outcome to judge it fairly. If the person assessing the study outcome did know which treatment the patient received, they might have preconceptions and judge the outcome in a biased way to favour one treatment over the other.

Finally, it is important to ask patients at the end of the study which treatment group they believe they were allocated to. By asking every patient this question, it enables the adequacy of the blinding to be tested.

Cluster randomisation

Some studies may randomise at a group rather than patient level, e.g.by hospital. This can be useful when investigating healthcare processes, public health interventions or complex treatments. Cluster randomised trials ( Fig. 1.4 ) have specific considerations that must be taken on board whilst designing the study and are often more complex to conduct.

Some studies use methods which rely upon factors that appear to be random to allocate patients to treatment groups. These may include birthday, day of the week or even whether the patient’s hospital number is odd or even. Although these may appear to be random numbers, they are not. This is called ‘pseudo-randomisation’ or ‘quasi-randomisation’ and studies utilising this approach should not be classed as randomised trials.

Phases of clinical trials

To take a new drug or procedure from concept to clinical practice requires several steps. There are five phases of research trials ( Fig. 1.5 ):

Cohort studies

In a cohort study, a defined population with a specific condition or a population undergoing a specific type of surgery is identified and followed over a defined period. Cohort studies are typically observational research, which means instead of patients being allocated to treatment groups, patients and doctors are free to choose. Instead of an ‘intervention’, which implies allocation to treatment groups, the intervention is often referred to as an ‘exposure’ ( Fig. 1.3c ).

As cohort studies are simply observing what happens in clinical practice, they may be conducted looking forwards in time (prospective) or backwards in time (retrospective). There are advantages and disadvantages of each approach are listed in Box 1.1 .