Meta-analysis and psychophysiology: A tutorial using depression and action-monitoring event-related potentials
Introduction
Most scientific questions are addressed by multiple studies conducted by independent research teams using a diverse range of methods rather than by a single study. Researchers understand and accept that the results of these studies will often vary and, in some cases, may directly contradict each other. Yet researchers also want to be able to use these varied and conflicting findings to come to a consensus regarding a body of work – for example, it is often desirable to determine whether the predictions of a theory have been supported or whether a finding has practical applications. For the greater part of the previous century, researchers from a number of fields including physics, psychology, ecology, zoology, archaeology, astronomy and medicine (Birge, 1929, Birge, 1932, Haidich, 2010, Petticrew, 2001) have relied on meta-analysis to quantitatively summarize a body of work and draw conclusions. “Meta-analysis” refers to a set of procedures that statistically analyze the results of primary studies (i.e. the original research) in order to synthesize the findings (Glass, 1976).
The purpose of this article is to provide a broad overview of what meta-analysis is as well as a practical tutorial aimed at psychophysiologists. This article is organized around a series of steps that nearly all meta-analyses will follow (adapted from Cooper, 2010, Cumming, 2012): formulating the problem, conducting the literature search, coding studies and extracting data, synthesizing effect sizes and assessing for heterogeneity, and assessing for threats to validity. Each of these sections will present tips, strategies and best-practices for conducting a meta-analysis. Each of the five sections will end with an illustrative example from a novel meta-analysis we performed on the relationship between depression and action-monitoring event-related potentials (ERPs), namely the error-related negativity (ERN) and the feedback negativity (FN). We conclude by identifying challenges to conducting robust meta-analyses, and offer some possible solutions for psychophysiologists to take up in planning, executing, and reporting on future studies.
Section snippets
Step 1: formulate the problem
Conducting a meta-analysis can take a great deal of time and effort. For example, one of the authors recently completed a meta-analysis which required approximately 16 months (Moran, in press). Given the work involved, one could legitimately wonder if summarizing the existing literature with a meta-analysis is a better use of one's time than trying to address an existing question with a new primary study or by summarizing the literature with a narrative review. The type of study that one
Step 2: conduct the literature search
Searching the literature for relevant studies is among the simplest steps in conducting a meta-analysis but it is also among the most important. The literature search determines which studies are eligible for inclusion in the meta-analysis and, therefore, the scope, validity and generality of the meta-analysis. Having a set of clear, well-formulated questions is critical for conducting a literature search as they will guide the inclusion/exclusion of primary studies.
In order to locate as many
Step 3: code studies and extract data
Once the meta-analyst has settled on a list of studies, coders must extract the relevant information including effect sizes and important study characteristics such as population, setting, task, recording parameters (psychophysiological recording system, filter settings etc.) and any other information that is relevant given the research question. This information can be used both to simply describe the studies under investigation as well as to test for potential moderating variables.
The
Step 4: synthesize effect sizes/assess heterogeneity
Once effect sizes have been computed for each study, the individual effects must be synthesized. There are many statistical models with which effects can be synthesized; the current article will focus on two of the most widely used: fixed effects and random effects (FE and RE, respectively) models. When combining effect sizes, the meta-analyst will note that, just like the results of individual participants, the results of individual studies are variable – often substantially so. The major
Step 5: assess for threats to validity
As with any statistical procedure, the results of a meta-analysis are only as good as the quality of the original data and the methodology employed by the researcher. Meta-analysis can be a powerful tool for summarizing a literature and for answering broader questions than any primary study; however, problems with the available data and the meta-analytic techniques can result in severely distorted findings. Thus, just as manipulation checks are necessary in many primary studies, assessing for
Discussion
Meta-analysis is a powerful statistical tool that can aid researchers in quantitatively summarizing a field, testing hypotheses, directing future research and assessing the field for potential threats to validity such as publication bias. Our goal in writing this tutorial was to point out the potential usefulness of meta-analysis to psychophysiologists unfamiliar with the method as well as to provide practical advice and best-practices to those psychophysiologists interested in conducting their
Conclusion
Meta-analysis is a powerful tool for quantitatively summarizing an existing literature. The present article aimed to introduce meta-analysis and provide a step-by-step tutorial aimed at psychophysiologists which included: formulating the problem, conducting the literature search, coding studies and extracting data, synthesizing effect sizes and assessing for heterogeneity, and assessing for threats to validity. Each of these steps was accompanied by a substantive example in which we
References (76)
- et al.
Neural response to reward and depressive symptoms in late childhood to early adolescence
Biol. Psychol.
(2012) - et al.
Depression and reduced sensitivity to non-rewards versus rewards: evidence from event-related potentials
Biol. Psychol.
(2009) - et al.
Reward dysfunction in major depression: multimodal neuroimaging evidence for refining the melancholic phenotype
NeuroImage
(2014) - et al.
Task-related dissociation in ERN amplitude as a function of obsessive-compulsive symptoms
Neuropsychologia
(2009) - et al.
Altered error-related brain activity in youth with major depression
Dev. Cog. Neurosci.
(2012) - et al.
The influence of anhedonia on feedback negativity in major depressive disorder
Neuropsychologia
(2014) - et al.
Global and regional burden of disease and risk factors, 2001: systemic analysis of population health data
Lancet
(2006) - et al.
Sex moderates the relationship between worry and performance monitoring brain activity in undergraduates
Int. J. Psychophysiol.
(2012) - et al.
The error-related negativity (ERN) and psychopathology: toward and endophenotypes
Clin. Psychol. Rev.
(2008) - et al.
Depression symptom severity and error-related brain activity
Psychiatry Res.
(2010)
Error processing in major depressive disorder: evidence from event-related potentials
J. Psychiatr. Res.
The error processing system in major depressive disorder: cortical phenotypal marker hypothesis
Biol. Psychol.
Action monitoring in major depressive disorder with psychomotor retardation
Cortex
Action monitoring and depressive symptom reduction in major depressive disorder
Int. J. Psychophysiol.
Hyperactivity within an extensive cortical distribution associated with excessive sensitivity in error processing in unmedicated depression: a combined event-related potential and sLORETA study
Int. J. Psychophysiol.
Increased error-related brain activity in generalized anxiety
Biol. Psychol.
Blunted neural response to errors as a trait marker or melancholic depression
Biol. Psychol.
Electrical brain imaging reveals the expression and timing of altered error monitoring in major depression
J. Abnorm. Psychol.
Event-related potentials in an emotional go/no-go task and remission if geriatric depression
Cog. Neurosci. Neuropsychol.
Probable values of the general physical constants
Physiol. Rev. Suppl.
The calculation of errors by the method of least squares
Phys. Rev.
Introduction to Meta-Analysis
Blunted neural response to rewards prospectively predicts depression in adolescent girls
Psychophysiology
Differentiating anxiety and depression in children and adolescents: evidence from event-related potentials
J. Clin. Child Adolesc. Psychol.
Anterior cingulate cortex and conflict detection: an update of theory and data
Cogn. Affect. Behav. Neurosci.
Individual differences in error monitoring in healthy adults: psychological symptoms and antisocial personality characteristics
Eur. J. Neurosci.
Neural evidence for enhanced error detection in major depressive disorder
Am. J. Psychiatr.
Statistical Power Analysis for the Behavioral Sciences
Error detection and posterior behavior in depressed undergraduates
Emotion
Research Synthesis and Meta-Analysis: A Step-by-Step Approach
The Handbook of Research Synthesis
Understanding the New Statistics: Effect Sizes, Confidence Intervals, and Meta-Analysis
Autonomic and electrophysiological correlates of emotional intensity in older and younger adults
Psychophysiology
Meta-Analysis by the Confidence Profile Method. The Statistical Synthesis of Evidence
Bias in meta-analysis detected by a simple, graphical test
Br. Med. J.
Performance monitoring in obsessive-compulsive disorder and social anxiety disorder
J. Abnorm. Psychol.
Event-related potential activity in the basal ganglia differentiates rewards from nonrewards: temporospatial principal components analysis and source localization of the feedback negativity
Hum. Brain Mapp.
The error-related negativity (ERN/Ne)
Cited by (43)
Intergenerational transmission of cognitive control capacity among children at risk for depression
2023, Biological PsychologyElectrophysiological evidence of mal-adaptation to error in remitted depression
2023, Biological PsychologyNo intolerance of errors: The effect of intolerance of uncertainty on performance monitoring revisited
2022, International Journal of PsychophysiologyCitation Excerpt :Our well-powered sample aims to disentangle discrepancies in the literature surrounding our constructs of interest. Specifically, mixed findings regarding the effect of IU subfactors on ERN amplitudes (Jackson et al., 2016; Sandre and Weinberg, 2019), mixed findings for direct relationships between depression error monitoring (Moran et al., 2017), and prior potential overestimation of the relationship between ERN and trait anxiety (Saunders and Inzlicht, 2020). We have attempted an experimental replication of Jackson et al. (2016) and Ruchensky et al. (2020), which seeks to verify previous findings using particular variables while adjusting the experimental procedure (Hudson, 2021).
Understanding the Error in Psychopathology: Notable Intraindividual Differences in Neural Variability of Performance Monitoring
2022, Biological Psychiatry: Cognitive Neuroscience and NeuroimagingNeural perspective on depression
2021, Encyclopedia of Behavioral Neuroscience: Second EditionEvent-related potential (ERP) measures of error processing as biomarkers of externalizing disorders: A narrative review
2021, International Journal of Psychophysiology