Statistics in Psychiatric Research

Concrete Examples of the Three Classes of Statistics in a Research Article

To provide a concrete example of these sometimes abstract concepts, consider a fictional study based on the simplest research design in psychiatric research: a randomized double-blind trial of a new drug versus a placebo pill for obsessive-compulsive disorder (OCD).

Figures 62-1 to 62-3 contain the annotated Method and Results sections for this fictional study, showing how the various psychometric statistics are presented in the Method section, while descriptive statistics are presented in the Method and Results sections, and inferential statistics are presented in the Results section (for definitions of terms used in these figures, refer to the section on statistical terms and their definitions ).

Experiment-wise Error Rate

Researchers should test only a few carefully selected hypotheses (specified before collecting their data!) if their obtained P-values are to have any meaning. The more statistical tests you perform, the greater the chance of finding at least one significant by chance alone (i.e., a false-positive result). Table 62-2 illustrates this phenomenon.

One should not be impressed by a researcher who conducts eight t-tests, finds one significant at P < 0.05, and proceeds to interpret the findings as confirming his theory. Table 62-2 shows us that with eight statistical tests at P < 0.05, the researcher had a 33% chance of finding at least one result significant by chance alone.

Selecting an Appropriate Statistical Method

The two key determinants in choosing a statistical method are (1) your research goal, and (2) the level of measurement of your outcome (or dependent) variable(s). Table 62-3 illustrates the key characteristics of the various levels of measurement and provides examples of each.

Once the level of measurement of your outcome variable has been determined, you will decide whether your research question will require you to compare two or more different groups of subjects, or to compare variables within a single group of subjects. Tables 62-4 and 62-5 will help you choose the appropriate statistical method once you have made these decisions. (Note that these tables consider only univariate statistical tests; multivariate tests are beyond the scope of this chapter.)

For example, if you want to conduct a study comparing a new drug to two control conditions, and your outcome measure is a continuous rating scale, Table 62-4 indicates that you would typically use the analysis of variance (ANOVA) to analyze your data. If you wanted to assess the association of two continuous measures of dissociation and anxiety in a single depressed sample, Table 62-5 indicates that you would usually select the Pearson correlation coefficient. (Note that the procedures listed for Ranked outcome measures are those typically referred to as “non-parametric tests.”)

The final consideration in selecting a statistical procedure is whether subjects are measured on more than one occasion, as in the typical longitudinal clinical trial. In cases such as these, special statistical methods for “repeated measures” are used. Traditionally, a repeated measures analysis of variance has been the most-used method of analyzing a study comparing two or more treatment conditions in a longitudinal design. However, when, as is commonly the case, there are missing data due to subject dropout, the preferred analysis method is a sophisticated approach referred to as the Mixed-effect Model Repeated Measure (MMRM) model.

The Importance of Assessing Statistical Power

A non-significant P-value is meaningless if the researcher studied too few subjects, resulting in low statistical power. Tables 62-6 to 62-8 will help you estimate the number of subjects required to have a reasonable chance (usually set at 80%, or power = 0.80) of detecting a true effect (or put another way, a 20% change of a false-negative finding).

For example, a researcher reports that she has compared two groups of 12 depressed patients, and found that a new drug was not significantly better than placebo at P < 0.05, by t-test. However, this negative result is not informative, because Table 62-6 indicates that with only 12 subjects per group, this researcher had statistical power of less than 0.80 to detect even a “large” effect; that is, even if the drug were truly effective, this study had less than a 50/50 chance of finding a significant difference.

Power analysis is now required as part of virtually all grant and institutional review board (IRB) applications submitted today.

Analysis of Covariance

Analysis of covariance (ANCOVA) is a form of ANOVA that tests the significance of differences between group means by adjusting for initial differences among the groups on one or more covariates. As an example, a psychologist interested in studying the effectiveness of a behavioral weight loss program versus self-dieting includes pre-treatment weights as a covariate.

Analysis of Variance

This is an optimal test of significance of difference among means from three or more independent groups. As an example, if a medical researcher wants to compare the effects of three or more different drugs on a single dependent measure, he or she would compute a one-way ANOVA. The more complex, factorial ANOVA also tests for interaction effects between multiple factors. For example, if the two factors being tested were “drug/placebo” and “male/female,” the ANOVA interaction test may find that the drug is more effective than the placebo in the female subjects only. The significance of the analysis of variance is tested by the F-statistic.

Analysis of Variance with Repeated Measure(s)

This is the optimal test of significance for comparing continuous variables that are obtained through repeated measurements of the same subjects (because each subject's scores are usually correlated, the regular ANOVA would give results that are “too significant”). An experimenter may select a repeated-measures design because these are generally more sensitive to treatment effects (i.e., they have high power), since score differences between subjects are ignored.

Bonferroni Correction

This is a conservative method of reducing the chances of false-positive findings by testing a set of statistical tests at a more conservative P-value. The standard Bonferroni correction divides the nominal P-value (say P < 0.05) by the total number of statistical tests being conducted. For example, with 10 t-tests conducted, each would be tested at P < 0.005 (i.e., 0.05/10) to determine significance.

Canonical Correlation

This is a generalization of multiple regression to the case of multiple independent variables and multiple dependent variables. It is rarely used today, except in neuroimaging studies with hundreds or thousands of correlated measurements. It is considered a multivariate statistical procedure because many inter-correlated variables are analyzed simultaneously.

Chi-Square ( χ ² ) Test

See Contingency Table Analysis by Chi-Square .

Cluster Analysis

This is a data-reduction technique used to group subjects together into subgroups (or “clusters”) based on their similarities or differences on a set of variables. This technique answers questions such as, “Do my subjects fall into subgroups?” and “What variables give a profile that distinguishes subgroups of my subjects?” A simple rule of thumb is “cluster analysis groups people, while factor analysis groups variables.” It is considered a multivariate statistical procedure because many inter-correlated variables are analyzed simultaneously.

Confidence Interval

A confidence interval does more than simply report that our observed mean difference between two groups is 2.5 points, significant at P < 0.05; it is far more informative to report that the 95% confidence interval around our observed mean is 0.6 to 4.4. Since 0 (the null hypothesis value of the mean difference) is not included in the 95% confidence interval, we know at a glance that the difference is significant at P < 0.05, but we also learn of the possible values of the actual mean difference between the groups, ranging from as low as 0.6 point to a high of 4.4 points (with 95% confidence). In the case of odds ratios, a confidence interval of 0.60 to 4.40 would not be significant at P < 0.05, because the null hypothesis would state that the odds ratio is 1.00, and this is included in the computed 95% confidence interval. Many journals require the reporting of confidence intervals, instead of using only P-values.

Contingency Table Analysis by Chi-Square ( χ ² )

This is a test to determine whether the frequencies in each cell of a contingency table are different from the proportions expected by chance. It is most commonly used on a 2 × 2 contingency table, represented as four cells forming a square.

A common use is to answer the following question: “Is there a difference between the occurrence of a given side effect in the drug group versus a placebo group?” In this case the table is arranged with drug versus placebo as the two rows, and side effect versus no side effect as the two columns. As the (squared) difference between the observed and expected frequencies in each cell increases, the chi-square statistic also increases, and the more significant the result becomes. If all cells contain exactly the frequencies that would be expected by chance, the chi-square statistic is zero. If the frequencies differ greatly from chance, the chi-square statistic gets larger and larger. The size of the chi-square statistic is based on the number of cells in the contingency table (since df = [# rows − 1] × [# columns − 1], a 2 × 2 table always has a single degree of freedom).

Correlation

See Pearson Correlation Coefficient .

Correlation Matrix

This is a “Table” or “Matrix” of correlation coefficients for all variables of interest.

Covariate

This is a variable that the investigator believes may influence the outcome or dependent variable and that is to be statistically adjusted for. For example, in a study of a new antidepressant, the baseline level of depression for subjects in each of the two groups may be used as a covariate.

Dependent t-Test

See t-Test for Dependent Means .

Dependent Variable

This is usually the outcome variable of interest in a study; it is also called an “end-point.” In a study of a new antidepressant drug, a depression rating scale may be used as the dependent variable.

Descriptive Statistics

These are statistics used to describe a single population. Descriptive statistics used commonly to summarize the central tendency of a group are the mean (or arithmetic average) for continuous measures and the median (or “middle score”) for ordinal or ranked measures. Descriptive statistics are used commonly to describe the variability within a group. These include the variance and its square root, the standard deviation for continuous measures, and the interquartile range for ranked measures. Researchers look at descriptive statistics first to get a “big picture” of their data (and also to look for data entry errors or obvious outliers).

Discriminant Function Analysis

This is the optimal procedure to distinguish statistically two or more groups based on a group of discriminating variables. It is an important and under-used procedure. It is considered a multivariate statistical procedure because many inter-correlated variables are analyzed simultaneously.