Shared neuroimaging data is an important means of promoting an open and reproducible neuroimaging analysis culture. The ABIDE 1 dataset (RRID:SCR_003612, Di Martino et al. 2014) is a premier example of shared neuroimaging data that promotes exploration of the relationship between the autism diagnosis and the features accessible in structural and resting state functional MRI in over 1000 subjects. There are many factors that influence the reproducibility of neuroimaging findings, including selection of software tool (Tustison et al. 2014). In this report, we take advantage of the ABIDE Preprocessed Data Project (http://preprocessed-connectomes-project.org/abide/) which has performed a structural analysis of the ABIDE 1 data with both FreeSurfer, versions 5.1 and 5.3 (RRID:SCR_001847, Fischl 2012), and ANTS (RRID:SCR_004757, Das et al. 2009). In an ideal world, regional thickness data would be independent of the specific software tool used to generate the result, when applied to common data. We utilize this dataset to evaluate the extent to which the selection of software tool matters, and provide a common platform to support further exploration of these results.

We pooled analysis results available from the ABIDE Preprocessing project for the FreeSurfer v5.1 and ANTS results as well as an independant run of Freesurfer v5.3. We prepared a common results file (in csv format) that included the average thickness measure from each of the tools as defined over the regions of the Desikan-Killany-Tourville (DKT; Klein and Tourville 2012, Desikan et al. 2006) atlas. We identified the subset of (976 cases (from the 1112 ABIDE 1 original cases)) that had completed all three analysis. We also generated a demographics file (also in csv format) for these specific cases. Finally, we developed an R software package reader to ingest these files, and to perform a simple analysis of correlation between the various methods in order to test the similarity between methods in generating these regional thickness summaries.

The result of this effort is a GitHub repository, https://github.com/companat/compare-surf-tools, which contains the summary data tables, R reader, sample R-based analysis examples, and the results of an initial analysis of the correlation of each of the anatomic regions in the DKT atlas across each of the analysis methods. The DKT cortical atlas is comprised of 31 bi-lateral regions, resulting in 62 correlations between FreeSurfer 5.1 and 5.3; FreeSurfer 5.1 and ANTS, and FreeSurfer 5.3 and ANTS. Across this set of regions, the mean and range of correlations observed (mean, range) was: 0.875, [0.7647, 0.9387]; 0.4316, [0.1945 ,0.5912]; 0.4744, [0.1871, 0.6743), respectively.

The FreeSurfer analysis in this data presents excellent inter-version (v5.1 – v5.3) commonality. There are, however, substantial differences between the regional thickness results between the FreeSurfer and ANTS analysis. While this is not a study related to ground truth, the conclusion, however, is that the reporting of thickness measures should be qualified by software platform, as the results of these two different platforms reporting a nominally similar resulting metric, can be substantially different. In providing access to the data and analysis, we envision this as an ‘open’ project, a starting point to explore more details of this analysis in order to elucidate additional details regarding the specific factors that are important to the ultimate goal of more reproducible neuroimaging computation. We encourage interested parties in the community to join the research team for this project by joining the Slack team at: https://brainhack.slack.com/messages/comp_surf_tools/.

The authors would like to thank the organizers and attendees of Brainhack Vienna 2016.