Elsevier

NeuroImage

Volume 59, Issue 2, 16 January 2012, Pages 1404-1412
NeuroImage

Test–retest reliability of resting-state connectivity network characteristics using fMRI and graph theoretical measures

https://doi.org/10.1016/j.neuroimage.2011.08.044Get rights and content

Abstract

Characterizing the brain connectome using neuroimaging data and measures derived from graph theory emerged as a new approach that has been applied to brain maturation, cognitive function and neuropsychiatric disorders. For a broad application of this method especially for clinical populations and longitudinal studies, the reliability of this approach and its robustness to confounding factors need to be explored. Here we investigated test–retest reliability of graph metrics of functional networks derived from functional magnetic resonance imaging (fMRI) recorded in 33 healthy subjects during rest. We constructed undirected networks based on the Anatomic-Automatic-Labeling (AAL) atlas template and calculated several commonly used measures from the field of graph theory, focusing on the influence of different strategies for confound correction. For each subject, method and session we computed the following graph metrics: clustering coefficient, characteristic path length, local and global efficiency, assortativity, modularity, hierarchy and the small-worldness scalar. Reliability of each graph metric was assessed using the intraclass correlation coefficient (ICC).

Overall ICCs ranged from low to high (0 to 0.763) depending on the method and metric. Methodologically, the use of a broader frequency band (0.008–0.15 Hz) yielded highest reliability indices (mean ICC = 0.484), followed by the use of global regression (mean ICC = 0.399). In general, the second order metrics (small-worldness, hierarchy, assortativity) studied here, tended to be more robust than first order metrics.

In conclusion, our study provides methodological recommendations which allow the computation of sufficiently robust markers of network organization using graph metrics derived from fMRI data at rest.

Highlights

► Test–retest reliability of graph metrics derived from resting-state fMRI. ► Different preprocessing and confound correction methods are examined. ► Moderate overall reliability, but differed between methods. ► Using a broad filter frequency range (0.008–0.15 Hz) yielded best results.

Introduction

The application of graph theory to networks extracted from functional imaging data has proven to be a promising tool to investigate the complex functional structure of the human brain that influences the dynamics underlying cognition and emotion (Bullmore and Sporns, 2009, Eryilmaz et al., 2011, Rubinov and Sporns, 2010). Several studies using different imaging modalities have consistently found the brain to be organized according to small-world properties, making it highly efficient in terms of information exchange while minimizing wiring costs (Achard and Bullmore, 2007, Micheloyannis et al., 2006, van den Heuvel et al., 2008). Further studies focusing on the finer architecture of the human brain network have shown the brain to exhibit an assortative, hierarchical modular structure (Ferrarini et al., 2009, Salvador et al., 2005) (see Materials and methods section for definitions) that can be altered in mental disease (Bassett et al., 2008, Rubinov et al., 2009, Supekar et al., 2008, Wang et al., 2009), is affected by age and sex (Gong et al., 2009, Micheloyannis et al., 2009, Tian et al., 2011) and can undergo rapid plastic changes (Achard and Bullmore, 2007, Polanía et al., 2011). The potential to characterize the entire brain connectome in a way that is both biologically meaningful and related to normal and disordered function across the age span makes these graph theoretical approaches attractive to clinical and basic research. The application of this methodology to resting-state fMRI data is especially interesting since it does not require a cognitive activation task and is therefore, in principle, applicable across the whole age range, at varying levels of awareness and cognitive ability, and even across species.

A variety of different parameters and methods can be used to construct a network graph from functional MRI data. Since a graph is formed by “nodes” (also called vertices or points) connected by “edges” (also called links or lines), the following two steps have to be performed to extract a network graph successfully:

  • 1)

    Definition of the nodes. In principle each voxel could be a node. In practice, a lower resolution is often preferred. When using a regional template with a resolution lower than the voxel size, additionally an estimator of regional activity is needed.

  • 2)

    Definition of the links. Two nodes are marked as linked if their respective time series show similarity measures (such as correlation) above a threshold. This requires

    • a.

      Correction for confounds such as physiological noise and/or global signal.

    • b.

      Decision about the frequency range.

    • c.

      Estimation of a measure of connectivity between nodes and definition of a significance threshold for selection of substantially contributing links.

Steps 1 and 2 are of fundamental importance, since the choice of nodes and links defines the graph and thus determines the network metrics that are then used for neurobiological interpretation (Rubinov and Sporns, 2010). Recently, several studies comparing different strategies for node and edge selection have been published. Fornito et al. (2010) investigated the influence of parcellation scale on different network parameters. They found whole-brain topological parameters to be quite robust over different scales while node specific parameters varied significantly depending on the parcellation scale. Consistent results were reported by Hayasaka and Laurienti (2010), who compared region of interest (ROI)- and voxel-based approaches, arguing for a slight superiority of the latter one. A study of Smith et al. (2011) focused on comparing different methods for the estimation of connectivity between nodes, representing network links. They found methods based on covariance to be quite sensitive to the underlying network, as well as several Bayesian net methods. Moreover, they noted that independently of the connectivity method, the definition of inadequate ROIs (nodes) could seriously compromise network characterization.

From the literature reviewed above, it follows that studies using different node and link selection methods cannot be simply compared quantitatively, especially when considering regional topological metrics based on single node measures.

In the present paper, we investigated two relevant methodological issues: First, we investigated the reliability of topological measures extracted from resting-state fMRI data. This issue is especially important for longitudinal studies or for repeated measurements in the context of a cognitive or drug challenge and lately has received considerable attention (Schwarz and McGonigle, 2011, Telesford et al., 2010).

Second, we studied the influence on these topological measures of different methods to correct for confounds, such as movement or global signal variation, and the choice of different frequency ranges, since these issues have been found relevant in conventional resting-state fMRI analyses (Fox et al., 2009, Murphy et al., 2009, Weissenbacher et al., 2009).

To this end, we systematically explored different estimators of local activity, several confound correction methods and different frequency ranges to construct a network, using the same fMRI resting-state reliability dataset. For each method, we investigated its performance in a two step validating procedure. First, we compared the obtained graph metrics to prominent findings in literature, e.g. whether the networks show small-world properties and are organized in a hierarchical modular fashion. And second, we compared test–retest reliability between methods, using the intra-class coefficient (ICC) as our metric.

Regarding the first step of network extraction outlined above, definition of nodes, we examined differences between estimators of local activity. Using a template with a lower resolution than the original data requires deciding on an estimator of average regional activity. Since most studies and especially those focusing on clinical aspects have used the Anatomic-Automatic-Labeling (AAL) template (Ferrarini et al., 2009, He et al., 2009, Liu et al., 2008, Tzourio-Mazoyer et al., 2002, Wang et al., 2009), we decided on using the AAL node approach as well, providing a certain degree of comparability. Most studies have used the mean time-series as an estimator, but this can be influenced disproportionally by potential outliers. Therefore, using the eigenvariate and median might offer some advantages, especially when using large and inhomogeneous regional templates. For the definition of links (step 2), we examined different methods for correction of confounds (step 2a). The blood-oxygen-level-dependent (BOLD) signal in fMRI is often confounded by fluctuations of non-neural origin (Cordes et al., 2001, Weissenbacher et al., 2009). This can be adjusted for by regressing the time-series of interest against time-series extracted from cerebrospinal fluid (CSF) and white matter (WM).

To address step 2b, we investigated the influence of different frequency ranges on reliability and network metrics. A bandpass filter is often used as an additional procedure to correct for noise. Neural oscillations in resting-state commonly occur at a frequency range between 0.0083 and 0.15 Hz. However, this quite broad frequency range could be obscured by aliased physiological influences like heart beat (0.6 to 1.2 Hz) or respiration (0.1 to 0.5 Hz) (Cordes et al., 2001). Thus, network studies often restrict themselves to smaller frequency bands (Achard et al., 2006, Fornito et al., 2010). We therefore compared the commonly used resting-state frequency range to a more stringent one, often used in network studies.

We did not restudy step 2c (estimation of a measure of connectivity between nodes and matrix thresholding), which is covered in the literature by the study of Smith et al. (Smith et al., 2011). However, as an additional analysis, we investigated the effects of different lengths of time-series on reproducibility of graph metrics. There is a great variety in how many data points are used to estimate the links between network nodes, ranging from 180 to 512 data points per subject. While averaging over long time-series might intuitively provide a better estimator of real connectivity, there are several potential drawbacks of long time-series. First, during resting-state, subjects are told not to focus their attention. However, this gets harder as measuring time increases and different networks become active, which might influence the connectivity measures. Second, while it might be feasible for healthy subjects to lie motionless in the scanner and refrain from external or internal stimuli for a period longer than 5 min, this might not apply to patients, for example suffering from Alzheimer's disease or schizophrenia. Therefore, if network metrics derived from short time-series provide equally reliable results as it was shown for conventional resting-state analysis (Van Dijk et al., 2010), it could also reduce the burden on patients.

Our investigation shows significant impacts of the methodology chosen for steps 1 and 2 on the reliability of topological measures derived from resting-state fMRI. We identified methods associated with reasonably high reliability that we recommend for further use of this approach, especially for longitudinal or interventional studies.

Section snippets

Subjects

A group of 33 healthy right-handed volunteers (mean age: 24.2 years; range: 19 to 29 years; 17 female) was recruited as part of an ongoing study on neurogenetic risk mechanisms for major mood disorders and schizophrenia (Erk et al., 2010, Esslinger et al., 2009) at two collaborating centers in Berlin and Mannheim. 14 participants were scanned in Berlin and 19 in Mannheim. All subjects were scanned twice two weeks apart at the same center. None of the participants reported a lifetime history of

Results

Independently of the estimator used, and across all density levels, all networks showed small-world characteristics indicated by gamma > 1 (gamma = Cnet/Crand) and lambda  1 (lambda = lnet/lrand) (see Supplementary Fig. 2) and exhibited an assortative, hierarchical and modular structure. The second order metrics studied were higher for lower densities, slowly decreasing as more links were added to the networks. The graphs for all metrics and every method can be found in Fig. 1.

Discussion

One primary goal of this study was to investigate the reliability of graph theoretical measures of human brain functional networks derived from fMRI resting-state data. Overall, we found that reliability was moderate (defined as ranging from 0.41 to 0.59) and highly dependent on the parameters used for the construction of networks and the networks' densities. This is in line with the results reported by Schwarz and McGonigle, who found an overall robust reliability (ICC ~ 0.5), but marked

Acknowledgments

This study was supported by NEWMEDS — Grant Agreement No. 115008. Funding for this study was provided by BMBF (NGFNplus MooDs) and DFG (SFB 636-B7).

Furthermore, we thank Dagmar Gass for her help with data collection and Anna Kopp for proof-reading the manuscript. All other authors reported no biomedical financial interests or potential conflicts of interest regarding this project.

References (62)

  • M. Rubinov et al.

    Complex network measures of brain connectivity: uses and interpretations

    NeuroImage

    (2010)
  • A.J. Schwarz et al.

    Negative edges and soft thresholding in complex network analysis of resting state functional connectivity data

    NeuroImage

    (2011)
  • S.M. Smith et al.

    Network modelling methods for FMRI

    NeuroImage

    (2011)
  • L. Tian et al.

    Hemisphere- and gender-related differences in small-world brain networks: a resting-state functional MRI study

    NeuroImage

    (2011)
  • N. Tzourio-Mazoyer et al.

    Automated anatomical labeling of activations in SPM using a macroscopic anatomical parcellation of the MNI MRI single-subject brain

    NeuroImage

    (2002)
  • P. Uhlhaas et al.

    Neural synchrony and the development of cortical networks

    Trends Cogn. Sci.

    (2010)
  • M.P. van den Heuvel et al.

    Small-world and scale-free organization of voxel-based resting-state functional connectivity in the human brain

    NeuroImage

    (2008)
  • L. Waldorp et al.

    Effective connectivity of fMRI data using ancestral graph theory: dealing with missing regions

    NeuroImage

    (2011)
  • A. Weissenbacher et al.

    Correlations and anticorrelations in resting-state functional connectivity MRI: a quantitative comparison of preprocessing strategies

    NeuroImage

    (2009)
  • S. Achard et al.

    Efficiency and cost of economical brain functional networks

    PLoS Comput. Biol.

    (2007)
  • S. Achard et al.

    A resilient, low-frequency, small-world human brain functional network with highly connected association cortical hubs

    J. Neurosci.

    (2006)
  • A.F. Alexander-Bloch et al.

    Disrupted modularity and local connectivity of brain functional networks in childhood-onset schizophrenia

    Front. Syst. Neurosci.

    (2010)
  • D.S. Bassett et al.

    Small-world brain networks

    Neuroscientist

    (2006)
  • D.S. Bassett et al.

    Hierarchical organization of human cortical networks in health and schizophrenia

    J. Neurosci.

    (2008)
  • E. Bullmore et al.

    Complex brain networks: graph theoretical analysis of structural and functional systems

    Nat. Rev. Neurosci.

    (2009)
  • G. Buzsáki et al.

    Neuronal oscillations in cortical networks

    Science

    (2004)
  • D.V. Cicchetti

    The precision of reliability and validity estimates re-visited: distinguishing between clinical and statistical significance of sample size requirements

    J. Clin. Exp. Neuropsychol.

    (2001)
  • D.V. Cicchetti et al.

    Developing criteria for establishing interrater reliability of specific items: applications to assessment of adaptive behavior

    Am. J. Ment. Defic.

    (1981)
  • D. Cordes et al.

    Frequencies contributing to functional connectivity in the cerebral cortex in “resting-state” data

    AJNR Am. J. Neuroradiol.

    (2001)
  • J. Damoiseaux et al.

    Consistent resting-state networks across healthy subjects

    Proc. Natl. Acad. Sci. U. S. A.

    (2006)
  • S. Erk et al.

    Brain function in carriers of a genome-wide supported bipolar disorder variant

    Arch. Gen. Psychiatr.

    (2010)
  • Cited by (0)

    View full text