Statistics and Evidence-Based Healthcare

Sampling And Inference To The Population At Large

The most fundamental issue of statistics is that one is trying to relate data taken from a relatively small ‘sample’ to a much larger group where it would be impractical to collect all the available data. In medical statistics, this large and rather nebulous group of subjects is known as ‘the population’ and it can often be hard to define. This may seem obvious but understanding the concept of sampling is crucial in the interpretation of results from studies. The application of statistics to research is an attempt to ensure that the results from your sample are in general agreement with the results that would have occurred if you had been able to conduct the experiment on all relevant members of the population in question. Fig. 14.1 demonstrates this simple relationship, and it is intuitive to see that, as the sample size increases, it moves closer to representing the population. In general, as the size of the sample increases, the bias in the study decreases; however, this is not always the case, and in some unusual statistical settings the bias will remain, but these are beyond the scope of this chapter (see Peacock and Peacock, 2011).

The most important ‘take-home’ message for you as an investigator is that, if you repeated your experiment, you would almost certainly get a different result. You may still draw the same conclusions from that result, but the numbers used in the statistical calculations would differ from sample to sample. On average, you would expect the results to be consistent, but when there is disagreement between sample and population, the following types of error (Type 1 and Type 2 errors) can occur and these should always be at the front of your mind when interpreting study results.

Type 1 Error

This occurs when ‘the sample’ used in your experiment generates a significant result for your hypothesis but there would not have been a significant result if you had performed the experiment on ‘the population’; in other words, it occurred by chance. When we set a ‘significant’ P value at .05, we are allowing a 5% chance of a type 1 error occurring for our study. To have zero chance of a type 1 error we would need to perform the experiment on ‘the population’ itself, and this is not possible as it implies recruiting an infinite number of subjects. The 5% level is entirely arbitrary and purely a convention that seems ‘reasonable’ in most research situations. Thus, P values should be interpreted cautiously, always bearing in mind the study design and the size of the difference observed between the comparative groups. P values are a continuum that runs from 0 to 1 and thresholds such as .05 are only used as a guide. On occasions, different levels are used, such as 1% ( P value of .01) and 10% ( P value of .10).

Another consideration when interpreting the results of a study is whether many comparisons have been made using the same sample. This is known as ‘multiple testing’ and it is a common flaw seen in a number of studies in the published literature (see Peacock and Peacock 2011 for further details on how to address this). By setting your P value for significance at .05, you are allowing a 5% chance of a type 1 error, that is, 1 in 20 statistical comparisons will produce a significant result by chance and thus, if a large number of variables in the dataset are tested for significance, there is a considerable risk of getting a type 1 error. Multiple testing often goes hand in hand with an unclear hypothesis and a poorly thought through study design. If you wish to test many outcomes and exposures using the same sample of patients, you need to account for this by setting yourself a more stringent P value for significance, for example, .01. In this case, you would need to make 100 comparisons in order for one of them to be significant by chance.

Type 2 Error

This occurs when ‘the sample’ used in your experiment fails to generate a significant result for your hypothesis but there would have been a significant result if you had performed the experiment on ‘the population’, that is, you have missed a real and possibly important effect. When we require a study to have 90% power, we are allowing a 10% chance that our sample will not detect a significant result that in truth exists in the population. This commonly occurs in small studies where there is insufficient power. Under-powered studies can be frustrating to interpret, particularly when there appears to be quite a large difference between the groups but the P value does not reach the magical threshold for acceptance as a significant result. Some statisticians argue that no under-powered studies should ever be undertaken as they cannot be interpreted and can raise ethical concerns in terms of unjustified patient research, and this would probably condense the world’s research output to a fraction of its current amount. Most research funders now demand power calculations, but sometimes the assumptions on which power calculations are based are grossly optimistic. However, in many cases a balance can still be achieved between attaining sufficient power and setting a pragmatic target for the sample size.

The Null And Active Hypotheses

It is always important to be able to define the null and active hypotheses for a study, and this means having clear definitions for both the outcome and the exposure or treatment. Under the null hypothesis, there is no difference between the groups that are being compared. This will tend to be the ‘default’ hypothesis unless the study sample accrues sufficient evidence to reject this null hypothesis and show that the active hypothesis is true.

Bias And Generalisability

When studies have been sampled in a non-random fashion, differences between the results from the sample and the true population can arise. Similarly, if treatments are allocated to patients non-randomly, estimates for differences between treatment groups can be biased. In other words, bias can arise if the sample is systematically unrepresentative of the population. However, bias and generalisability are not always the same thing. If one runs a well-powered randomised controlled trial (RCT), the results comparing randomised groups that are estimated from the sample are unlikely to be biased; however, they will only be generalisable to the sub-group of the population that met the inclusion criteria for the study. Thus, when interpreting the results from studies it is important to view them in relation to the study inclusion and exclusion criteria and the sampling methods that were employed when recruiting the subjects. There are many different types of bias and certain study designs are more prone to particular types of bias than others. This will be discussed in more depth in the section about types of study and experimental design ( pp 10 ).

Confidence Intervals, Accuracy And Precision

When interpreting a result from a sample, it is useful to express the result with a range of possible values that it might have taken if other samples of the same size had been selected. This range of values is called a confidence interval (CI), and we can set the level of confidence as a percentage; 95% confidence is typically used. For example, if we measure the birth weight of 50 babies and calculate a point estimate for the mean weight of 3360 g and a 95% CI of 3200 to 3520 g, this means that we can be 95% confident that, given this sample size, the true mean birth weight for all babies in the population relevant to this study lies somewhere between 3200 and 3520 g. In general, as the sample size increases, the CI becomes narrower. The term precision is used to describe how wide the CI is around the point estimate, whereas accuracy gives an indication of how close the point estimate from the sample is to the true unmeasurable population value and is therefore more related to bias or generalisability. The calculation of CIs will be discussed in more detail on p. 7 .

Independence And Matched Data

Many statistical tests make assumptions about the independence of the subjects analysed in the study. If data are not independent (e.g. the same mother can be included more than once in a study on childbirth) then account should be taken of this in the analysis. Many commonly used statistical tests assume that all observations are from separate individuals and, by including subjects more than once, you are making your sample less varied than it would be if subjects were only entered once. Similarly, if your study design selected cases and controls by matching them in terms of covariates such as age and ethnic group, then your analysis must account for this matching. In general, it is preferable not to match any subjects within a study as it is possible to adjust for potential differences between your groups at the analysis stage. Furthermore, in studies where subjects have been assessed before and after experiencing an exposure, they should be investigated in a ‘paired’ fashion by analysing the difference between the before and after measurements, as this accounts for the lack of independence between them. This also tends to improve the power of the study as within-patient differences tend to be less variable than absolute variation between patients.

The Normal Distribution

The normal distribution is one of the most important and widely used probability distributions in medical statistics. It can be described by a rather complex mathematical equation; however, if it is plotted in terms of probability, we can see that it generates the famous ‘bell-shaped’ curve shown in Fig. 14.2 . The x-axis is standardised such that the mean corresponds to zero (the most probable value) with units of standard deviation (SD) falling above and below this value. It can be seen that 95% of the area under the curve lies between the points that fall 1.96 SDs on either side of the mean value, and this number is particularly important as we can use it to give us an indicator of the range of values that would incorporate 95% of all possible values. In some cases, you may wish to know the range of values that incorporate 90% or even 99% of all values, and these ranges correspond to the 1.65 and 2.58 SDs on either side of the mean, respectively.

The beauty of the normal distribution is that this symmetrical property around the mean value holds whether we are plotting the actual data points from our sample or whether we are plotting the results of the study if we had repeated it over and over again. In this scenario, we would end up with the ‘mean of the mean of the samples’, and the range of mean values can be represented by the sampling distribution. The term standard error is essentially the SD of this sampling distribution and it is used throughout statistics to calculate CIs around point estimates. An example of how to calculate a CI for the mean is given on p. 7 .

Parametric And Non-Parametric Tests

Parametric statistical tests are ones where assumptions are made about which mathematical distribution best represents the sample and the population from which it was taken. Non-parametric statistical tests are ones where no assumption has been made about the distribution of the data. See Table 14.1 ‘A guide to unifactorial statistical methods’. In general, parametric tests tend to be more powerful and sensitive than non-parametric tests and therefore tend to be preferred, as fewer observations are required to provide evidence in favour of the hypothesis if it is true. A typical example of a parametric test is the use of a Student’s t -test to compare the mean values of a continuous variable between two groups. One of the test assumptions is that the continuous data measured in the sample can be assumed to follow the normal distribution. If this assumption is not valid, then the non-parametric Mann–Whitney U test can be used, which ranks the observations in order of size and compares the proportions that fall above and below the median value for each of the groups in question. Thus, the Mann–Whitney U test is less sensitive to large outlying values but also less informative as observations above or below the median are all treated in the same way.

Deciding Whether to Use Parametric or Non-Parametric Tests

For binary data, the assumption of a binomial distribution may be valid, as according to Peacock and Peacock, 2011, the normal distribution ‘can be used as an approximation to the Binomial distribution when n is large. In precise this works if np and n(1−p) are both greater than 5 (where np and n(1−p) are number of successes and number of failures)’. When comparing proportions across binary, categorical or ordinal data, the chi-squared distribution is often assumed; however, if the numbers in the categories in a 2 × 2 table become very small (<5 in any one category) then it is often more appropriate to use Fisher’s exact test, which is described in any standard statistical textbook.

Probably the most common example of deciding whether to use a parametric or non-parametric test is when you want to know whether the continuous data in your sample can be assumed to follow a normal distribution. In general, for small samples of less than about 15 observations, it is not safe to assume the data are normally distributed, and non-parametric methods should generally be employed. However, it should be remembered that these tests are less powerful and the sample size is small, which will make the statistical results hard to interpret. If you have a reasonably large sample size, the first thing to do is to plot your data points on a scatter graph or group the data into bins and plot them on a histogram. Inspection of the graphs or histograms is the simplest way of assessing whether your distribution assumptions are valid. Deviations from the normal distribution can lead to significant skewness or kurtosis. Fig. 14.3 demonstrates histograms for data that follow a normal distribution or have a positively or negatively skewed distribution, and Fig. 14.4 shows how data can deviate from the classic ‘bell-shaped’ curve seen in the normal distribution and exhibit kurtosis. Kurtosis is concerned with the shape of the distribution and can have a considerable impact on the statistical analysis that you choose to perform on your data. When kurtosis is extreme, non-parametric tests should be used. It is worth noting that for data that are perfectly normally distributed, the mean, median and mode values are all the same, whereas for positively skewed data, the mean tends to be larger than the median and vice versa for negatively skewed data. When summarising skewed data (and also reporting descriptive results from a non-parametric test), it is often better to quote the median and interquartile range rather than the mean and SD, which are generally used for summarising normally distributed data. The way to calculate these summary statistics is described in the next section. In some cases, it helps to convert skewed data into another variable that can be assumed to follow the normal distribution (this is called transformation). For example, data that are positively skewed can often be manipulated into a more normally distributed format by transforming them onto the log scale; the t -test can then be used on the log-transformed data. There are more formal ways of testing your assumptions about the normal distribution, such as normal plots and Shapiro–Francia or Shapiro–Wilk tests, but these should be used cautiously and, if there is doubt, you should revert to using non-parametric methods.

Mean

This is the sum of all the observations divided by the number of observations.

For example, mean systolic blood pressure at 20 weeks’ gestation = (105 + 107 + 107 + … + 190 + 199)/20 = 144.35 mmHg.

Similarly, the mean age = 32.05 years.