Elsevier

Neuroscience & Biobehavioral Reviews

Volume 45, September 2014, Pages 100-118
Neuroscience & Biobehavioral Reviews

Review
Test-retest reliabilities of resting-state FMRI measurements in human brain functional connectomics: A systems neuroscience perspective

https://doi.org/10.1016/j.neubiorev.2014.05.009Get rights and content

Highlights

  • We review previous work on voxel-wise metrics and their reliabilities.

  • We conduct a meta-summary reliability analysis of these metrics.

  • Heteromodal associative functional networks are test-retest reliable.

  • ReHo, ICA and VMHC are most reliable across examined metrics.

  • Sub-optimal design and data processing options may hurt reliability.

Abstract

Resting-state functional magnetic resonance imaging (RFMRI) enables researchers to monitor fluctuations in the spontaneous brain activities of thousands of regions in the human brain simultaneously, representing a popular tool for macro-scale functional connectomics to characterize normal brain function, mind-brain associations, and the various disorders. However, the test-retest reliability of RFMRI remains largely unknown. We review previously published papers on the test-retest reliability of voxel-wise metrics and conduct a meta-summary reliability analysis of seven common brain networks. This analysis revealed that the heteromodal associative (default, control, and attention) networks were mostly reliable across the seven networks. Regarding examined metrics, independent component analysis with dual regression, local functional homogeneity and functional homotopic connectivity were the three mostly reliable RFMRI metrics. These observations can guide the use of reliable metrics and further improvement of test-retest reliability for other metics in functional connectomics. We discuss the main issues with low reliability related to sub-optimal design and the choice of data processing options. Future research should use large-sample test-retest data to rectify both the within-subject and between-subject variability of RFMRI measurements and accelerate the application of functional connectomics.

Introduction

Connectomics has rapidly become a revolution in basic brain research following the introduction of the key concept of the brain connectome, which conceptualizes the connection of the entire brain at different scales as a complex and dynamic system (Sporns et al., 2005, Sporns, 2013). The macro-scale brain structural connectome has been mapped out as a graph with 50–1000 nodes based on structural magnetic resonance imaging (MRI) (He et al., 2007) or diffusion tensor imaging (DTI) (Hagmann et al., 2008, Gong et al., 2009). Brain dynamics are shaped by the brain connectome structure and topology (Deco et al., 2011, Bullmore and Sporns, 2009, Bullmore and Sporns, 2012) and can be measured with resting-state functional MRI (RFMRI) with different spatial (millimeters) and temporal (seconds) resolutions (Biswal et al., 1995) as well as use of graph theory (Salvador et al., 2005, Achard et al., 2006). At this macro-scale, the functional connectivity of the entire brain connectome or brain graph (Bullmore and Bassett, 2011) has been termed the ‘functional connectome’ (Biswal et al., 2010, Zuo et al., 2012, Kelly et al., 2012), which, conceptually, is another term for the functional outcomes of the brain connectome.

Over the last two decades, macro-scale functional connectomics (i.e., the functional connectomes revealed with RFMRI) has rapidly become a powerful tool in the study of human brain function and associations with mind, behavior and disease (Craddock et al., 2013, Castellanos et al., 2013). For example, the default mode or default network (DMN) of the human brain represents a remarkable and successful indication of discovery science with functional connectomics (Buckner, 2012). This network originally referred to a set of brain regions including the posterior cingulate, hippocampus, medial prefrontal and inferior parietal cortex, which are functionally connected (Greicius et al., 2003) and reliably deactivates during most externally focused tasks (Shulman et al., 1997) but exhibits elevated metabolism during internal cognitive processes (Raichle et al., 2001). The function of the DMN has been related to episodic memory (Buckner et al., 2005), mind wandering (Mason et al., 2007) and various traits (e.g., Adelstein et al., 2011, personality). Furthermore, DMN alterations have been widely observed in neurodegenerative diseases such as Alzheimer's disease (AD) (Greicius et al., 2004) and psychiatric disorders (Greicius, 2008, Broyd et al., 2009, Fornito and Bullmore, 2010). Between-group differences or inter-individual variability in functional connectomics have greatly enriched our knowledge of specific cortical areas (e.g., (Leech and Sharp, 2014), the posterior cingulate cortex) and elucidated the meaningful and stable properties of the human brain functional connectivity (Buckner et al., 2013).

Although functional connectomes have demonstrated features that are temporally stable or exhibit statistically ignorable intra-individual variability, the temporal dynamics of these connectomes have recently received substantial attention (Hutchison et al., 2013). There are both neural and non-neural factors that likely contribute to the dynamic changes in resting-state functional connectivity. Although exploratory, emerging evidence suggests that dynamic RSFC patterns indicate the intrinsic functional architecture of the human brain (Deco and Corbetta, 2011) as it relates to normal cognition (Albert et al., 2009, Bassett et al., 2011, Mantzaris et al., 2013), behavior (Fox et al., 2007, Hesselmann et al., 2008, Sadaghiani et al., 2010) and clinical diseases (e.g., Jones et al., 2012, AD). Meanwhile, the intra-individual variability of resting-state functional connectivity can be partly attributed to various non-neural factors including scan conditions (Yan et al., 2009), head motion (Power et al., 2012, Van Dijk et al., 2012, Satterthwaite et al., 2012, Satterthwaite et al., 2013, Yan et al., 2013a), physiological noise (Birn et al., 2008, Chang et al., 2009, Chang and Glover, 2009a, Chang and Glover, 2009b), and data analysis/standardization strategies (Yan et al., 2013b). In summary, factors that can affect the intra-individual variability of RSFC patterns raise concerns regarding the test-retest reliabilities of functional metrics of the brain connectome with RFMRI.

High test-retest reliability requires both low intra-individual and high inter-individual variability (Barnhart et al., 2007). Low intra-individual variability indicates a high stability across different points in time, and high inter-individual variability implies highly differentiable measures across subjects. High test-retest reliability is particularly important for the development of biomarker-based clinical tests for early detection, timely interventions and diagnoses of brain disorders, especially psychiatric disorders, which currently lack a gold standard biological definition (Kapur et al., 2012). Surprisingly, however, the importance of test-retest reliability has been overlooked in functional connectomics until several recent studies on the test-retest reliability of RFMRI were published (Shehzad et al., 2009, Zuo et al., 2010a, Zuo et al., 2010b). As mentioned, RFMRI offers both high temporal and spatial resolutions to examine whole brain activity in vivo. A common spatial unit of RFMRI measurements is called the volumetric element (voxel) of several millimeters. While previous studies on functional connectomes conducted with large-scale parcellation provided great insight into brain network organization (Bullmore and Sporns, 2009, Bullmore and Sporns, 2012), regional variations in local functional homogeneity (Zang et al., 2004, Zuo et al., 2013) suggested that defining a node based on a large structural region and building an edge between a pair of nodes in a brain graph can be problematic. Simply averaging the voxel-wise time series in a large region ignores the fact that the strength of functional homogeneity within the region is typically low and highly variable across spatial locations (Jiang et al., 2014), leading to difficulties in interpreting the mean time series and derived functional metrics in relation to raw time series. This highlights the importance of voxel-wise assessments on the intrinsic functional architecture.

The present work will survey common voxel-wise metrics used in functional connectomics and their test-retest reliabilities. We also aim to provide recommendations on use of these functional metrics for characterizing the brain connectome at a systems neuroscience level by summarizing the voxel-wise reliability maps of seven functional networks of a large-scale functional brain parcellation. Fig. 1 demonstrates a flowchart of the overall analytic strategies. Specifically, we first introduce the RFMRI measurements of human functional connectomes at the voxel level (Fig. 1A–D) and then briefly review various voxel-wise functional metrics (Fig. 1D), the test-retest reliabilities of which have been examined previously (Fig. 1E). Finally, these reliability maps are summarized into a large-scale functional parcellation of the human brain as defined in (Yeo et al., 2011) (Fig. 1F) by comparing the distribution of voxel-wise test-retest reliability values (Fig. 1G), the proportion of reliable voxels (Fig. 1H) and the ratio of reliable voxels (Fig. 1I) across the seven functional networks.

Section snippets

Functional metrics of brain connectomes with RFMRI

The basal metabolism of the human brain measured during rest state (i.e., eyes closed, awake, and no specific cognitive task) represents 20% of the total body energy consumption, which highlights the importance of studying resting state brain function (Raichle, 2006, Raichle and Mintun, 2006). RFMRI was initially proposed to study low-frequency spontaneous neural activity in the motor system (Biswal et al., 1995). Since then, this technology has been widely used to study human brain function

Test-retest reliability in functional connectomics

We have briefly reviewed the high-resolution (i.e., voxel-wise) metrics of the human brain function. As mentioned, our aim with this review is to provide a systematic overview on the test-retest reliability of these voxel-wise functional metrics and further a reference on uses of these metrics to study the human functional connectome markers in brain disorders (Singh and Rose, 2009). In this section, the test-retest reliability of common functional metrics will be summarized at the level of

Further considerations and future directions

As noted, both within- and between-subject variability can affect the level of test-retest reliability. Any factor that changes within- or between-subject variability thus possibly influences test-retest reliability. Meanwhile, the findings revealed by the meta-summary analysis implied that there are remarkable differences in test-retest reliability across different functional metrics and functional networks. It becomes a crucial point in developing a highly reliable method to investigate how

Conclusions and recommendations

The overall spatial pattern of common RFMRI measurements can be reliably detected across different times and large-scale networks. The regional variations of test-retest reliability of various RFMRI metrics are reflected at the network level across the common seven large-scale neural networks, among which, the high-order associative networks have the highest reliability, whereas the limbic network exhibits the lowest reliability. The voxel-wise metrics of functional connectomes can exhibit

Funding

Dr. Xi-Nian Zuo acknowledges the funding support from the Hundred Talents Program, the Key Research Program (KSZD-EW-TZ-002) of the Chinese Academy of Sciences, the Major Joint Fund for International Cooperation and Exchange of the National Natural Science Foundation (81220108014). Dr. Xiu-Xia Xing's research is supported by the Fundamental Development Program on Mathematics and Statistics from Beijing University of Technology and the Natural Science Foundation of China (81201153). RFMRI data

Conflict of interest statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Acknowledgments

Dr. Xi-Nian Zuo is indebted to Yu-Feng Zang, Michael Peter Milham and F. Xavier Castellanos, who introduced him to open neuroscience with RFMRI and have provided endless supports. Without their encouragement and comments, this review would not have been possible. We thank Zarrar Shehzad and Chao-Gan Yan for sharing their reliability maps, Donna Dierker for discussions about the design and implementation of surface-based cortical analyses with the RFMRI data from the HCP Q3 release, Yong He and

References (232)

  • M.G. Bright et al.

    Reliable quantification of BOLD fMRI cerebrovascular reactivity despite poor breath-hold performance

    Neuroimage

    (2013)
  • M.G. Bright et al.

    Removing motion and physiological artifacts from intrinsic BOLD fluctuations using short echo data

    Neuroimage

    (2013)
  • S.J. Broyd et al.

    Default-mode brain dysfunction in mental disorders: a systematic review

    Neurosci. Biobehav. Rev.

    (2009)
  • R.L. Buckner

    The serendipitous discovery of the brain's default network

    Neuroimage

    (2012)
  • M. Cao et al.

    Topological organization of the human brain functional connectome across the lifespan

    Dev. Cogn. Neurosci.

    (2014)
  • F.X. Castellanos et al.

    Clinical applications of the functional connectome

    Neuroimage

    (2013)
  • C. Chang et al.

    Influence of heart rate on the BOLD signal: the cardiac response function

    Neuroimage

    (2009)
  • C. Chang et al.

    Effects of model-based physiological noise correction on default mode network anti-correlations and correlations

    Neuroimage

    (2009)
  • C. Chang et al.

    Relationship between respiration, end-tidal CO2, and BOLD signals in resting-state fMRI

    Neuroimage

    (2009)
  • G. Chen et al.

    Linear mixed-effects modeling approach to FMRI group analysis

    Neuroimage

    (2013)
  • D. Cordes et al.

    Hierarchical clustering to measure connectivity in fMRI resting-state data

    Magn. Reson. Imaging

    (2002)
  • R.W. Cox

    AFNI: what a long strange trip it's been

    Neuroimage

    (2012)
  • L. Deuker et al.

    Reproducibility of graph metrics of human brain functional networks

    Neuroimage

    (2009)
  • M. Ercsey-Ravasz et al.

    A predictive network model of cerebral cortical connectivity based on a distance rule

    Neuron

    (2013)
  • B. Fischl

    FreeSurfer

    Neuroimage

    (2012)
  • M.D. Fox et al.

    Intrinsic fluctuations within cortical systems account for intertrial variability in human behavior

    Neuron

    (2007)
  • K.J. Friston et al.

    Network discovery with DCM

    Neuroimage

    (2011)
  • D.D. Garrett et al.

    Moment-to-moment brain signal variability: a next frontier in human brain mapping?

    Neurosci. Biobehav. Rev.

    (2013)
  • M.F. Glasser et al.

    The minimal preprocessing pipelines for the Human Connectome Project

    Neuroimage

    (2013)
  • D.N. Greve et al.

    Accurate and robust brain image alignment using boundary-based registration

    Neuroimage

    (2009)
  • C.C. Guo et al.

    One-year test-retest reliability of intrinsic connectivity network fMRI in older adults

    Neuroimage

    (2012)
  • J. Hlinka et al.

    Functional connectivity in resting-state fMRI: is linear correlation sufficient?

    Neuroimage

    (2011)
  • R.M. Hutchison et al.

    Dynamic functional connectivity: promise, issues, and interpretations

    Neuroimage

    (2013)
  • M. Jenkinson et al.

    FSL

    Neuroimage

    (2012)
  • T. Jiang

    Brainnetome: a new -ome to understand the brain and its disorders

    Neuroimage

    (2013)
  • T. Jiang et al.

    Brainnetome-wide association studies in schizophrenia: the advances and future

    Neurosci. Biobehav. Rev.

    (2013)
  • S. Achard et al.

    A resilient, low-frequency, small-world human brain functional network with highly connected association cortical hubs

    J. Neurosci.

    (2006)
  • J.S. Adelstein et al.

    Personality is reflected in the brain's intrinsic functional architecture

    PLoS One

    (2011)
  • H. Akil et al.

    Challenges and opportunities in mining neuroscience data

    Science

    (2011)
  • A.P. Alivisatos et al.

    Nanotools for neuroscience and brain activity mapping

    ACS Nano

    (2013)
  • A.P. Alivisatos et al.

    Neuroscience. the brain activity map

    Science

    (2013)
  • E.A. Allen et al.

    Tracking whole-brain connectivity dynamics in the resting state

    Cereb. Cortex

    (2014)
  • J.S. Anderson et al.

    Decreased interhemispheric functional connectivity in autism

    Cereb. Cortex

    (2011)
  • J.S. Anderson et al.

    Reproducibility of single-subject functional connectivity measurements

    AJNR Am. J. Neuroradiol.

    (2011)
  • J.S. Anderson et al.

    Complexity of low-frequency blood oxygen level-dependent fluctuations covaries with local connectivity

    Hum. Brain Map.

    (2014)
  • H.X. Barnhart et al.

    An overview on assessing agreement with continuous measurements

    J. Biopharm. Stat.

    (2007)
  • D.S. Bassett et al.

    Dynamic reconfiguration of human brain networks during learning

    Proc. Natl. Acad. Sci. U. S. A.

    (2011)
  • C.F. Beckmann et al.

    Investigations into resting-state connectivity using independent component analysis

    Philos. Trans. R. Soc. Lond. B Biol. Sci.

    (2005)
  • J.L. Bernal-Rusiel et al.

    Alzheimer's Disease Neuroimaging I. Statistical analysis of longitudinal neuroimage data with linear mixed effects models

    Neuroimage

    (2012)
  • B. Biswal et al.

    Functional connectivity in the motor cortex of resting human brain using echo-planar MRI

    Magn. Reson. Med.

    (1995)
  • Cited by (512)

    • Test-Retest Reliability of Functional Connectivity in Adolescents With Depression

      2024, Biological Psychiatry: Cognitive Neuroscience and Neuroimaging
    View all citing articles on Scopus
    View full text