Meta-analysis and psychophysiology: A tutorial using depression and action-monitoring event-related potentials

https://doi.org/10.1016/j.ijpsycho.2016.07.001Get rights and content

Highlights

  • A meta-analysis tutorial is provided.

  • The concepts are demonstrated using the relationship between depression and the ERN/FN.

  • The depression/ERN relationship is contaminated by publication bias.

  • The depression/FN relationship is dependent on the use of gambling/guessing tasks.

Abstract

Meta-analyses are regularly used to quantitatively integrate the findings of a field, assess the consistency of an effect and make decisions based on extant research. The current article presents an overview and step-by-step tutorial of meta-analysis aimed at psychophysiological researchers. We also describe best-practices and steps that researchers can take to facilitate future meta-analysis in their sub-discipline. Lastly, we illustrate each of the steps by presenting a novel meta-analysis on the relationship between depression and action-monitoring event-related potentials – the error-related negativity (ERN) and the feedback negativity (FN). This meta-analysis found that the literature on depression and the ERN is contaminated by publication bias. With respect to the FN, the meta-analysis found that depression does predict the magnitude of the FN; however, this effect was dependent on the type of task used by the study.

Introduction

Most scientific questions are addressed by multiple studies conducted by independent research teams using a diverse range of methods rather than by a single study. Researchers understand and accept that the results of these studies will often vary and, in some cases, may directly contradict each other. Yet researchers also want to be able to use these varied and conflicting findings to come to a consensus regarding a body of work – for example, it is often desirable to determine whether the predictions of a theory have been supported or whether a finding has practical applications. For the greater part of the previous century, researchers from a number of fields including physics, psychology, ecology, zoology, archaeology, astronomy and medicine (Birge, 1929, Birge, 1932, Haidich, 2010, Petticrew, 2001) have relied on meta-analysis to quantitatively summarize a body of work and draw conclusions. “Meta-analysis” refers to a set of procedures that statistically analyze the results of primary studies (i.e. the original research) in order to synthesize the findings (Glass, 1976).

The purpose of this article is to provide a broad overview of what meta-analysis is as well as a practical tutorial aimed at psychophysiologists. This article is organized around a series of steps that nearly all meta-analyses will follow (adapted from Cooper, 2010, Cumming, 2012): formulating the problem, conducting the literature search, coding studies and extracting data, synthesizing effect sizes and assessing for heterogeneity, and assessing for threats to validity. Each of these sections will present tips, strategies and best-practices for conducting a meta-analysis. Each of the five sections will end with an illustrative example from a novel meta-analysis we performed on the relationship between depression and action-monitoring event-related potentials (ERPs), namely the error-related negativity (ERN) and the feedback negativity (FN). We conclude by identifying challenges to conducting robust meta-analyses, and offer some possible solutions for psychophysiologists to take up in planning, executing, and reporting on future studies.

Section snippets

Step 1: formulate the problem

Conducting a meta-analysis can take a great deal of time and effort. For example, one of the authors recently completed a meta-analysis which required approximately 16 months (Moran, in press). Given the work involved, one could legitimately wonder if summarizing the existing literature with a meta-analysis is a better use of one's time than trying to address an existing question with a new primary study or by summarizing the literature with a narrative review. The type of study that one

Step 2: conduct the literature search

Searching the literature for relevant studies is among the simplest steps in conducting a meta-analysis but it is also among the most important. The literature search determines which studies are eligible for inclusion in the meta-analysis and, therefore, the scope, validity and generality of the meta-analysis. Having a set of clear, well-formulated questions is critical for conducting a literature search as they will guide the inclusion/exclusion of primary studies.

In order to locate as many

Step 3: code studies and extract data

Once the meta-analyst has settled on a list of studies, coders must extract the relevant information including effect sizes and important study characteristics such as population, setting, task, recording parameters (psychophysiological recording system, filter settings etc.) and any other information that is relevant given the research question. This information can be used both to simply describe the studies under investigation as well as to test for potential moderating variables.

The

Step 4: synthesize effect sizes/assess heterogeneity

Once effect sizes have been computed for each study, the individual effects must be synthesized. There are many statistical models with which effects can be synthesized; the current article will focus on two of the most widely used: fixed effects and random effects (FE and RE, respectively) models. When combining effect sizes, the meta-analyst will note that, just like the results of individual participants, the results of individual studies are variable – often substantially so. The major

Step 5: assess for threats to validity

As with any statistical procedure, the results of a meta-analysis are only as good as the quality of the original data and the methodology employed by the researcher. Meta-analysis can be a powerful tool for summarizing a literature and for answering broader questions than any primary study; however, problems with the available data and the meta-analytic techniques can result in severely distorted findings. Thus, just as manipulation checks are necessary in many primary studies, assessing for

Discussion

Meta-analysis is a powerful statistical tool that can aid researchers in quantitatively summarizing a field, testing hypotheses, directing future research and assessing the field for potential threats to validity such as publication bias. Our goal in writing this tutorial was to point out the potential usefulness of meta-analysis to psychophysiologists unfamiliar with the method as well as to provide practical advice and best-practices to those psychophysiologists interested in conducting their

Conclusion

Meta-analysis is a powerful tool for quantitatively summarizing an existing literature. The present article aimed to introduce meta-analysis and provide a step-by-step tutorial aimed at psychophysiologists which included: formulating the problem, conducting the literature search, coding studies and extracting data, synthesizing effect sizes and assessing for heterogeneity, and assessing for threats to validity. Each of these steps was accompanied by a substantive example in which we

References (76)

  • M. Ruchsow et al.

    Error processing in major depressive disorder: evidence from event-related potentials

    J. Psychiatr. Res.

    (2006)
  • P.L.A. Schoenberg

    The error processing system in major depressive disorder: cortical phenotypal marker hypothesis

    Biol. Psychol.

    (2014)
  • D. Schrijvers et al.

    Action monitoring in major depressive disorder with psychomotor retardation

    Cortex

    (2008)
  • D. Schrijvers et al.

    Action monitoring and depressive symptom reduction in major depressive disorder

    Int. J. Psychophysiol.

    (2009)
  • Y. Tang et al.

    Hyperactivity within an extensive cortical distribution associated with excessive sensitivity in error processing in unmedicated depression: a combined event-related potential and sLORETA study

    Int. J. Psychophysiol.

    (2013)
  • A. Weinberg et al.

    Increased error-related brain activity in generalized anxiety

    Biol. Psychol.

    (2010)
  • A. Weinberg et al.

    Blunted neural response to errors as a trait marker or melancholic depression

    Biol. Psychol.

    (2016)
  • K. Aarts et al.

    Electrical brain imaging reveals the expression and timing of altered error monitoring in major depression

    J. Abnorm. Psychol.

    (2013)
  • G.S. Alexopoulos et al.

    Event-related potentials in an emotional go/no-go task and remission if geriatric depression

    Cog. Neurosci. Neuropsychol.

    (2007)
  • R.T. Birge

    Probable values of the general physical constants

    Physiol. Rev. Suppl.

    (1929)
  • R.T. Birge

    The calculation of errors by the method of least squares

    Phys. Rev.

    (1932)
  • M. Borenstein et al.

    Introduction to Meta-Analysis

    (2009)
  • J.N. Bress et al.

    Blunted neural response to rewards prospectively predicts depression in adolescent girls

    Psychophysiology

    (2013)
  • J.N. Bress et al.

    Differentiating anxiety and depression in children and adolescents: evidence from event-related potentials

    J. Clin. Child Adolesc. Psychol.

    (2015)
  • C.S. Carter et al.

    Anterior cingulate cortex and conflict detection: an update of theory and data

    Cogn. Affect. Behav. Neurosci.

    (2007)
  • W.-P. Chang et al.

    Individual differences in error monitoring in healthy adults: psychological symptoms and antisocial personality characteristics

    Eur. J. Neurosci.

    (2010)
  • P.H. Chiu et al.

    Neural evidence for enhanced error detection in major depressive disorder

    Am. J. Psychiatr.

    (2007)
  • J. Cohen

    Statistical Power Analysis for the Behavioral Sciences

    (1988)
  • R.J. Compton et al.

    Error detection and posterior behavior in depressed undergraduates

    Emotion

    (2008)
  • H.M. Cooper

    Research Synthesis and Meta-Analysis: A Step-by-Step Approach

    (2010)
  • H. Cooper et al.

    The Handbook of Research Synthesis

    (1994)
  • G. Cumming

    Understanding the New Statistics: Effect Sizes, Confidence Intervals, and Meta-Analysis

    (2012)
  • J. Dywan et al.

    Autonomic and electrophysiological correlates of emotional intensity in older and younger adults

    Psychophysiology

    (2008)
  • D.M. Eddy et al.

    Meta-Analysis by the Confidence Profile Method. The Statistical Synthesis of Evidence

    (1992)
  • M. Egger et al.

    Bias in meta-analysis detected by a simple, graphical test

    Br. Med. J.

    (1997)
  • T. Endrass et al.

    Performance monitoring in obsessive-compulsive disorder and social anxiety disorder

    J. Abnorm. Psychol.

    (2014)
  • D. Foti et al.

    Event-related potential activity in the basal ganglia differentiates rewards from nonrewards: temporospatial principal components analysis and source localization of the feedback negativity

    Hum. Brain Mapp.

    (2011)
  • W.J. Gehring et al.

    The error-related negativity (ERN/Ne)

  • Cited by (43)

    • No intolerance of errors: The effect of intolerance of uncertainty on performance monitoring revisited

      2022, International Journal of Psychophysiology
      Citation Excerpt :

      Our well-powered sample aims to disentangle discrepancies in the literature surrounding our constructs of interest. Specifically, mixed findings regarding the effect of IU subfactors on ERN amplitudes (Jackson et al., 2016; Sandre and Weinberg, 2019), mixed findings for direct relationships between depression error monitoring (Moran et al., 2017), and prior potential overestimation of the relationship between ERN and trait anxiety (Saunders and Inzlicht, 2020). We have attempted an experimental replication of Jackson et al. (2016) and Ruchensky et al. (2020), which seeks to verify previous findings using particular variables while adjusting the experimental procedure (Hudson, 2021).

    • Neural perspective on depression

      2021, Encyclopedia of Behavioral Neuroscience: Second Edition
    View all citing articles on Scopus
    View full text