Access the full text.
Sign up today, get DeepDyve free for 14 days.
Loading next page...
/lp/springer-journals/the-decision-decoding-toolbox-ddtbox-a-multivariate-pattern-analysis-poH7d75MPT
- Publisher
- Springer Journals
- Copyright
- Copyright © 2018 by The Author(s)
- Subject
- Biomedicine; Neurosciences; Bioinformatics; Computational Biology/Bioinformatics; Computer Appl. in Life Sciences; Neurology
- ISSN
- 1539-2791
- eISSN
- 1559-0089
- DOI
- 10.1007/s12021-018-9375-z
- Publisher site
- See Article on Publisher Site
Abstract
In recent years, neuroimaging research in cognitive neuroscience has increasingly used multivariate pattern analysis (MVPA) to investigate higher cognitive functions. Here we present DDTBOX, an open-source MVPA toolbox for electroencephalography (EEG) data. DDTBOX runs under MATLAB and is well integrated with the EEGLAB/ERPLAB and Fieldtrip toolboxes (Delorme and Makeig 2004; Lopez-Calderon and Luck 2014; Oostenveld et al. 2011). It trains support vector machines (SVMs) on patterns of event-related potential (ERP) amplitude data, following or preceding an event of interest, for classification or regression of experimental variables. These amplitude patterns can be extracted across space/electrodes (spatial decoding), time (temporal decoding), or both (spatiotemporal decoding). DDTBOX can also extract SVM feature weights, generate empirical chance distributions based on shuffled-labels decoding for group-level statistical testing, provide estimates of the prevalence of decodable information in the population, and perform a variety of corrections for multiple comparisons. It also includes plotting functions for single subject and group results. DDTBOX complements conventional analyses of ERP components, as subtle multivariate patterns can be detected that would be overlooked in standard analyses. It further allows for a more explorative search for information when no ERP component is known to be specifically linked to a cognitive process of interest. In summary, DDTBOX is an easy-to-use and open-source toolbox that allows for characterising the time-course of information related to various perceptual and cognitive processes. It can be applied to data from a large number of experimental paradigms and could therefore be a valuable tool for the neuroimaging community. . . . Keywords Multivariate pattern classification analysis (MVPA) Decoding Event-related potentials (ERPs) . . Electroencephalography (EEG) Toolbox Support vector machines General Introduction Beginning with Edelman and colleagues' (1998) and Haxby and colleagues'(2001)seminalstudies,applications of MVPA tofunc- In recent years, the use of multivariate pattern analysis (MVPA) tional magnetic resonance imaging (fMRI) data have become in- techniques for neuroimaging data has rapidly increased. creasingly popular, leading to a strong trend towards investigating representations and predicting Binformation^ (i.e. the content of cognition) in cognitive neuroscience (for reviews and comments Stefan Bode and Daniel Feuerriegel contributed equally to this see: Davis and Poldrack 2013;Haynes 2015; Haynes and Rees publication and are shared first authors. 2006; Heinzle et al. 2012; Hogendoorn 2015; Mur et al. 2009; Norman et al. 2006; Kriegeskorte et al. 2006; Tong and Pratte * Daniel Feuerriegel dfeuerriegel@unimelb.edu.au 2012; Woolgar et al. 2014, 2016) and to the publication of several toolboxes (Hanke et al. 2009a;Hebartetal. 2015; Oosterhof et al. 1 2016). Most variants of MVPA have in common that (usually Melbourne School of Psychological Sciences, The University of Melbourne, Melbourne, Australia supervised) classifiers are trained to predict the content of cogni- 2 tive processes directly from local spatial activation patterns, as School of Psychology, Social Work and Social Policy, University of measured by the blood-oxygen-level-dependent (BOLD) signal. South Australia, Adelaide, Australia Due to the poor temporal resolution of fMRI, however, it is often Princeton Neuroscience Institute, Princeton University, difficult to track any fast decision processes in time, and to deter- Princeton, New Jersey, USA mine the timecourse with which predictive information regarding Max Planck Institute for Psycholinguistics, these processes is represented in neural data. Nijmegen, The Netherlands 28 Neuroinform (2019) 17:27–42 One potential solution to this problem is to apply MVPA to properties of stimuli such as arousal (e.g., Bode et al. 2014). electroencephalography (EEG) data, as EEG has a far better Finally, MVPA on ERPs allows for a more explorative search temporal resolution in the range of milliseconds, as opposed for information when no ERP component is known to be to seconds in fMRI (other techniques such as specifically linked to the cognitive process of interest; it does magnetoencephalography, MEG, will not be discussed here; not require apriori knowledge of the location and timing of an for reviews see King and Dehaene 2014; Grootswagers et al. effect, which can vary substantially across experiments 2016). Multivariate analysis techniques were already applied to (discussed in Groppe et al. 2011). EEG data to investigate cognition several decades ago (Gevins We will now introduce the functionality of DDTBOX, et al. 1979), but are currently experiencing a strong revival (for which can be applied to data from a variety of experimental reviews and technical comments see: Bai et al. 2007; Blankertz paradigms (and is by no means restricted to decision-making et al. 2011;Continietal. in press; King and Dehaene 2014; research). DDTBOX requires minimal experience with Parra et al. 2005; Sajda et al. 2009;Stokes et al. 2015). By MATLAB coding, and integrates well with EEGLAB/ taking advantage of the multivariate nature of EEG signals, ERPLAB (Delorme and Makeig 2004; Lopez-Calderon and multivariate analysis techniques can, for example, predict the Luck 2014) and FieldTrip (Oostenveld et al. 2011), allowing outcomes of decisions, parameters of decision models, and de- users to prepare data using standard pre-processing pipelines cision errors directly from activity patterns (e.g., Blank et al. for ERP analyses with only minimal additions. DDTBOX can, 2013;Bodeetal. 2012, 2014;Bodeand Stahl 2014; Boldt and however, also use data preprocessed with many other com- Yeung 2015; Charles et al. 2014; Chung et al. 2015; Das et al. mercially available software packages. In the following, we 2010; Parra et al. 2002; Philiastides and Sajda 2006; will first briefly discuss basic principles of classification ap- Philiastides et al. 2006; Ratcliff et al. 2009; Tzovara et al. proaches for the study of EEG signals. Then, we will provide 2015; Van Vugt et al. 2012;for relatedapproachessee:ElEl an overview of the general architecture of the DDTBOX, Zein et al. 2015;Wyartetal. 2012, 2015). Others have used complementing our detailed online user manual (https:// similar techniques to investigate visual awareness (Hogendoorn github.com/DDTBOX/DDTBOX/wiki). This will be et al. 2015; Hogendoorn and Verstraten 2013;Fahrenfortetal. followed by a user-oriented introduction to using DDTBOX, 2017), multisensory integration (Chan et al. 2017), and auto- covering the general principles and features, the functional matic processing of semantic features of task-irrelevant stimuli structure of the toolbox, and a section on how to prepare (Bode et al. 2014). MVPA hasalsobeen extensivelyusedfor EEG data and how to configure analyses in DDTBOX. We constructing brain-computer interfaces (BCIs; e.g., Müller et al. will then provide examples of research which has already used 2004, 2008). beta-versions of DDTBOX, as well as limitations of analyses We present a novel open-source toolbox for MATLAB— offered by DDTBOX. Finally, we will conclude by giving an the Decision Decoding ToolBOX (DDTBOX)—that performs outlook into planned future developments and extensions, as MVPA on the high-temporal-resolution EEG data typically well as options for users to directly contribute to the toolbox. analysed using univariate analyses of event-related potentials Where relevant, we will make reference to the detailed, more (ERPs). However, instead of analysing signals at single elec- technical documentation available online, which is designed trodes (i.e. channels), or averaging across a group of selected as an additional hands-on guide to the DDTBOX. We have electrodes, for which ERP components have been described also provided an example dataset online that can be and linked to specific cognitive processes (e.g., Luck 2005), downloaded to learn how to use DDTBOX, available at MVPA is applied to data from all electrodes in a predefined https://osf.io/bndjg/. This dataset is under a GNU General analysis time window, which thereby serves as multivariate Public License (GPL) 3.0, which guarantees user rights to input to a classifier (Parra et al. 2005). Such patterns of am- share and modify the data. plitude data can be extracted across space (e.g., from data averaged over a time window for each channel: spatial decoding), time (e.g., from all timepoints for each channel Classification Based on ERP Data separately: temporal decoding), or both (from all timepoints for all channels: spatiotemporal decoding). This approach is Machine learning has recently gained strong popularity in arguably often more data- rather than hypothesis-driven com- systems neuroscience. In particular, a supervised-learning ap- pared to conventional ERP analyses, and has several benefits: proach using support vector machines (SVMs) has proved to First, even subtle multivariate EEG patterns that differ be- be a powerful tool for neuroimaging analysis (e.g., Haynes tween experimental conditions can be detected that would 2015; Grootswagers et al. 2016). The power of this approach otherwise be overlooked (Bode et al. 2012). Second, single- is derived from the fact that in most experiments, experi- trial patterns of activity can be directly linked to parameters in menters know the categories of interest apriori. These cate- decision models (e.g., Philiastides and Sajda 2006; Ratcliff gories of interest typically correspond to different experimen- et al. 2009; Tzovara et al. 2015), or used to predict subjective tal conditions or participant response types (e.g., participants Neuroinform (2019) 17:27–42 29 make decisions regarding object categories, make errors ver- DDTBOX can also perform a type of generalisation anal- sus correct responses, report different subjective experiences, ysis, testing whether patterns of information that discriminate etc.). The aim for analysis is then to find patterns of neural between categories are stable across experimental contexts. activity that distinguish these categories. For multivariate For this, the classifiers are trained on data from one context EEG signals, this would correspond to finding patterns in (e.g., correct and error responses in task A) and used to predict the signal across time and space (electrodes) that distin- the classes from data from another context (e.g., correct and guish the categories of interest. While this approach is still error responses in task B). Such an approach is known as a correlative, in the sense that it seeks to identify patterns of cross-condition classification analysis. This can reveal wheth- covariance between neural data and latent cognitive vari- er the neural patterns that discriminate between outcomes are ables, its great advantage is that the structure of the neural consistent across different task or stimulus presentation con- data need not map straightforwardly to known aspects of ditions, as the classifier should only be able to generalise from the cognitive variables. Instead, it is sufficient that the EEG one to the other if patterns are highly similar (for an example signal patterns predict the cognitive variables, thereby per- from fMRI, see Bode et al. 2013). mitting researchers to conclude that information regarding For cognitive variables of interest that are continuous rather cognitive variables is present in the neural data, either than categorical (such as response times), an alternative to decodable from specific electrodes, or from specific pro- SVM classification is support vector regression (SVR). SVR cessing time windows (Yarkoni et al. 2017). allows for trial-by-trial values of a continuous variable to be A detailed description of SVMs has been provided else- mapped to predicted values of that variable. DDTBOX offers where (Cortes and Vapnik 1995; Burges 1998; Hastieetal. both options, as we will outline below. 2001). Put simply, the general principle of SVM classification is to construct a hyperplane (i.e. a decision boundary) in mul- tidimensional feature space to optimally separate exemplars General Principles of DDTBOX into different categories (i.e. neural data mapping onto differ- ent experimental conditions). The nearest exemplars to this In order to perform a classification analysis, DDTBOX first hyperplane from each category are known as the support vec- requires the user to define discrimination groups, correspond- tors. The further away these exemplars are from the separating ing to categories of interest. These could be experimental con- hyperplane, the better the classification. To avoid circularity ditions (e.g., different object categories) but also participants’ (Kriegeskorte et al. 2009), estimation of the hyperplane must behaviour (e.g., correct and incorrect responses). The event- be performed on data independent from test data (left-out data locked ERP data, which is used for the analysis, has under- from the same experiment, or new data from an identical ex- gone all pre-processing steps such as artefact correction and is periment), which is subsequently used to evaluate the model epoched into time periods of interest as for conventional ERP by assigning category Blabels^ to each exemplar in the test analyses. Events can be either exogenous, such as stimulus data set. This usually involves k-fold cross-validation, in presentations, or endogenous, such as behavioural responses. which the data are divided into k subsets. The classifier is The epochs of ERP data are then sorted with respect to the trained on k-1 subsets and tested using the left-out subset. categories of interest, and each epoch is assigned a class label This procedure is then independently repeated with each sub- corresponding to its category. set serving as the test data set once while training on the others (for an example see Meyers and Kreiman 2011). The average SVM Back-End Software, Types of Analysis, accuracy across all cross-validation steps, often referred to as and Kernels Bclassification accuracy^ or Bdecoding accuracy^, can then be treated as an index for whether information about the catego- DDTBOX’s main function is to prepare exemplars of patterns ries of interest was represented in this specific pattern of brain of ERP data for each participant for SVM classification or activity. Statistically, this question can be assessed by submit- regression analyses. To perform such analyses, DDTBOX in- ting classification accuracy values to statistical testing, either teracts with existing machine learning software packages to against a theoretical chance performance level (e.g., with two perform classification/regression (similar to The Decoding balanced classes, the expected chance level = 50%), or against Toolbox for fMRI; Hebart et al. 2015). The user can choose an empirical test distribution, e.g., by comparing against re- between the LIBSVM package (Chang and Lin 2011), which sults of analyses using randomly-shuffled condition labels has been used by other toolboxes in the field (e.g., Hebart et al. (e.g., Stelzer et al. 2013). Although SVM classification is 2015), and LIBLINEAR (Fan et al. 2008), a less flexible but binary in nature, it can easily be extended to more complex faster implementation of commonly-used SVM classification multi-class classification problems by combining results from and regression algorithms (see the online documentation for all pair-wise class combinations or performing one-vs.-other details). Several SVM analysis options are available, includ- comparisons. ing different SVM fitting methods and kernels. We refer to the 30 Neuroinform (2019) 17:27–42 websites of these software packages for detailed explanations other channels are ignored, and the data points for the selected of these options (LIBSVM: https://www.csie.ntu.edu.tw/ channel that are included in the analysis time window (num- ~cjlin/libsvm; LIBLINEAR: https://www.csie.ntu.edu.tw/ ber of data points × 1) are treated as the activity pattern of ~cjlin/liblinear). For most research questions requiring interest. This analysis does not investigate spatially distributed classification, C-SVM (as implemented in LIBSVM) with a information, but instead focuses on information distributed in linear kernel and a default regularising parameter C = 1 ap- time for a given channel. This approach is complementary to pears to be adequate and standard in the field, and is therefore spatial classification, but it does not make use of all available the default option in DDTBOX. For multivariate regression, (spatial) information. Finally, one can consider both spatial DDTBOX uses SVR in LIBSVM with a linear kernel and information (over channels) and temporal information (over regularisation parameter C = 0.1 as the default option. timepoints) within a chosen analysis time window as the ac- tivity pattern (spatiotemporal analyses; number of data points Analysis Time Window Width x number of channels), as shown in Fig. 1D. DDTBOX performs analyses using a moving window ap- Averaging proach: the signals of interest during a prespecified analysis time window are extracted and analysed, and the analysis time DDTBOX further provides the user with the option to average window is then moved by a specified step-size through the across separate sets of exemplars first before training the clas- epoch (depicted in Fig. 1A). The user can specify the analysis sifier. The standard option is not to average, which means that window width and step-size. The optimal analysis time win- usually each experimental trial (or a part of such) is treated as dow width depends on the research question of interest, as one exemplar for one of the classes of interest. This usually information relating to some cognitive processes might be has the advantage of maintaining a large number of exemplars better captured by longer analysis time windows, while other for training and testing. However, if data from a large number short duration cognitive processes might be better captured of trials are available, one might consider averaging across using short analysis time windows. Our own previous work subsets of trials for the same reasons that averaging is per- has successfully utilised analysis time windows ranging from formed to obtain grand average ERPs: to optimise the 10 ms (e.g., Bode and Stahl 2014) toaslongas 80ms(e.g., signal-to-noise ratio. For example, if the experiment was split Bode et al. 2012). into 10 separate blocks, one could use block-averaged data for each class instead of single trials (e.g., see Bode et al. 2012). Analysis Time Window Step Size This is similar to first obtaining beta-estimates, or ‘regressors’, for separate functional ‘runs’ in fMRI, and then performing The analysis time window is moved through the trial at a user- MVPA on these estimates (representing the run-averaged defined step size, independently repeating decoding analyses model fit of a general linear model) instead of on single vol- each time with data from the new time window (depicted in umes from all trials. Averaging usually results in estimates of Fig. 1A). The step size could be the same as the analysis exemplars with a higher signal-to-noise ratio, and can improve window width to achieve non-overlapping analysis time win- classification performance in some cases (see Isik et al. 2014; dows (e.g., 10 ms windows moved in steps of 10 ms). Grootswagers et al. 2016). Alternatively, the step size could be finer than the window width (e.g., 20 ms windows moved in steps of 10 ms), leading Feature Weight Analyses to partial overlap of analysis time windows. This can be use- ful, for example, when one is interested in relatively fast cog- DDTBOX allows users to extract and analyse feature weights nitive processes, which might occur with a finer temporal from the fitted SVM classifiers. Much as regression coeffi- resolution than the window size and therefore be captured cients describe the contribution of each predictor to the depen- only partly by two consecutive larger analysis time windows. dent variable, feature weights in SVM describe the contribu- tion of each feature in determining the decision boundary, i.e. Spatial and/or Temporal Analyses separating classes. As such, feature weights are used in DDTBOX to estimate the relative importance of different fea- DDTBOX users can elect to run spatial analyses (Fig. 1B), tures (e.g., channels in spatial decoding analyses) for classifi- which involve averaging across all data points included in the cation or regression. Accordingly, feature weights are chosen analysis time window for each channel. This proce- analysed in DDTBOX to identify sources of information that dure results in one data point per channel (number of channels the classifier uses to distinguish between experimental catego- × 1 activity pattern). Alternatively, the user can choose to ries of interest. The ‘raw’ feature weights derived from SVMs disregard spatial patterns and perform temporal analyses are prone to erroneous interpretations regarding the sources of (Fig. 1C) using data from single channels. In this case, all information used for decoding, as they can be affected by Neuroinform (2019) 17:27–42 31 Fig. 1 Decoding approaches in DDTBOX. (a) Example of the windowed analysis approach. DDTBOX performs MVPA on time windows of EEG data (time windows outlined in blue). For each analysis the time window is moved through the trial by a predefined step size. (b)Example of spatial decoding. For each channel EEG data is averaged across timepoints within the analysis time window, resulting in one value per channel used for MVPA. (c) Example of temporal decoding. MVPA is performed using data from each timepoint within the analysis time window, for each channel separately. (d) Example of spatiotemporal decoding. All timepoints at all channels are used in combination for MVPA other statistically independent signals (such as noise generated and spatiotemporal analyses). Furthermore, as the sign of the by muscle activity, which as a feature may be strongly weight- feature weights indicate the importance of each feature for one ed but irrelevant). However, this can be corrected in or the other (arbitrary) category, and since the sign of each DDTBOX by employing the algorithm described by Haufe feature weight is therefore arbitrary, DDTBOX computes ab- et al. (2014). solute feature weights, which indicate the importance for the In spatiotemporal analyses (see above) the features are classification in general (i.e. for either category). However, the single timepoints within the analysis time window for each advanced user can also access the original signed feature channel. In DDTBOX feature weights are averaged across weights at individual timepoints within each analysis window. timepoints within each analysis window to output an averaged Lastly, feature weights from each analysis step are z- feature weight value for each channel (in consequence, group standardised to make them comparable between analyses. level feature weight analyses are only implemented for spatial Hence, the final output is one absolute, z-standardised feature 32 Neuroinform (2019) 17:27–42 weight value for each channel for each analysis time window. chance decoding (i.e. is a fixed-effect analysis; discussed in These are used for group-level statistical testing (see below). Allefeld et al. 2016). However, the method based on the min- imum statistic also provides lower bound estimates of the Statistical Testing prevalence of decodable information in the population. Analyses run in DDTBOX typically involve a large number The result of each single analysis for each participant is a of individual tests, requiring corrections for multiple compari- percentage value of correct classifications for all exemplars sons to control the family-wise error rate. The number of tests contained in the test-data set (for classification analyses), or performed depends on the number of analysis time windows, a Fisher-Z transformed correlation between the predicted la- which can be minimised by selecting a restricted search space bels and the true labels (for regression analyses). Then, after prior to running decoding analyses. DDTBOX offers a variety the k-fold cross-validation procedure, all k outcome values are of correction techniques for multiple comparisons, some of averaged to index the overall accuracy. As it is theoretically which exploit temporal autocorrelation of the classification ac- possible that accuracy estimates were inflated by chance due curacy results across time windows to preserve statistical pow- to the random assignments of exemplars to training and test er. Available corrections include the Holm-Bonferroni method sets, the default option in DDTBOX is to re-compute the sets (Holm 1979), maximum statistic and cluster-based permutation m times (i.e. a new, fully independent draw of k sets) and to tests (Blair and Karniski 1993; Maris and Oostenveld 2007), repeat all analyses for a user-specified number of iterations. generalised family-wise error rate control (Korn et al. 2004)and The default is to repeat all cross-validated analyses with inde- false discovery rate control (e.g. Benjamini and Hochberg pendently drawn sets ten times. For example, choosing k = 10 1995; Benjamini et al. 2006). In addition, the distributional for cross-validation, and m = 10 iterations will result in 10 × assumptions for paired-samples t-tests are often violated for 10 = 100 analyses, and the final accuracy will be the average samples of classification accuracy scores (Stelzer et al. 2013). of all 100 analyses. This procedure is designed to optimise DDTBOX can therefore also perform analyses using Yuen’s reliability of results rather than accuracy values. paired-samples t-test (Yuen 1974; Wilcox 2012), which is more Statistical testing at a group level is then performed on robust against violations of normality. average accuracy values obtained from the same analysis time DDTBOX further offers group-level statistical testing of window across participants. DDTBOX offers the option of feature weights using paired-samples t-tests, with corrections testing against theoretical chance level (e.g., 50% for a bal- for multiple comparisons over channels. Feature weights can anced two-class classification, 33% for balanced 3-class clas- be averaged over a number of analysis time windows before sification, etc.). However, this approach has been criticised statistical testing, if required. recently (Combrisson and Jerbi 2015). For example, increases in sample variance of accuracy values will also increase the Display Options chance of rejecting the null hypothesis when testing against theoretical chance (Allefeld et al. 2016). The default option in DDTBOX allows plotting of the decoding performance and DDTBOX is therefore to estimate the empirical chance distri- feature weight results at various stages. First, users can plot bution by running decoding analyses on data with permuted decoding accuracy scores (averaged over cross-validation condition labels. Specifically, DDTBOX repeats all original steps and independent analyses) for individual subjects, for analyses (e.g., m iterations of a k-fold cross-validation proce- all analysis time windows (spatial and spatiotemporal dure) with exactly the same data and the same category labels, analyses) or for all channels within a single time window but with assignment of labels to exemplars independently (temporal analyses).For spatial and spatiotemporal analyses randomised for each iteration. This means that any potential this is an ‘information time-course’, displaying the average biases in the original data (such as unbalanced numbers of ex- accuracies (y-axis) for each chosen analysis time window (x- emplars across categories) also affect the permuted-label analy- axis). Results of permuted condition labels analyses can also ses. The original and the permuted-label analyses are otherwise be plotted. This could be useful to quickly visually inspect the completely identical, and the results of the permuted-label anal- results for appropriateness of the chosen parameters (such as yses can then be statistically compared to the original results. the window widths or step size), and also to confirm that the Finally, group decoding accuracy at each analysis time shuffled-label control analysis produces chance results. By window can be tested for statistical significance using either contrast, temporal analysis results are plotted as a spatial paired-samples t-tests or a group-level analysis method de- map of accuracies for each channel, which are plotted as a scribed in Allefeld et al. (2016) based on the minimum statistic heat map with a topographic projection onto the scalp. (Friston et al. 1999). Importantly, both testing approaches do Similarly, at a group level information time-course plots not provide population inference as do t-tests on univariate can be generated for spatial and spatiotemporal analyses, measures, but instead test the null hypothesis that there are displaying the group-level accuracies (and optionally the per- at least some individuals within the sample that show above- muted labels analysis results in the same plot) with error bars Neuroinform (2019) 17:27–42 33 denoting standard errors of the mean. Users also have the the category/condition for classification (only one condi- option to include a vertical bar indicating the timing of the tion is used for support vector regression), timepoints are event of interest, as well as automatic marking of statistically thesingledata points, channels the included EEG chan- significant analysis time windows based on a user-specified nels, and trials the single trials of the experiment. This is alpha level. Axis labels are automatically generated (based on the general format for data storage, and each processing the included baseline period and sampling rate, as well as step will create a similar variable after the specified minimum and maximum accuracy values) but can be manu- manipulations. ally modified, if desired. The temporal analyses group results Reduction of data (Phase 3). Next, the data is reduced to are again heat maps displaying the colour-coded average the user-specified categories / conditions, which are used for group-level accuracy for each channel (note that standard er- the discrimination group of interest. This has the advantage rors are not included in this plot). that DDTBOX can operate within the memory constraints of For the display of group-level feature weight maps (spatial most desktop computers. and spatiotemporal analyses), two options are available. Balancing the number of included trials (Phase 4). A fre- Firstly, a matrix of z-standardised, absolute feature weights quent problem with classification analysis is that one might per channel (y-axis) can be displayed for user-selected analy- end up with an unequal number of trials per condition. This sis time windows (x-axis). Secondly, the z-standardised, ab- might be due to paradigms in which one condition is over- solute group-level feature weights can be displayed for single represented (e.g., oddball paradigms, flanker tasks, or any analysis time windows or averages of user-specified analysis other paradigm that requires more or less frequent events), time windows. Feature weights can also be plotted as maps responses of interest are not balanced (e.g., errors and correct thresholded by statistical significance. All figures are plotted responses, or most decision-making paradigms), or simply using MATLAB plotting routines, can be manually modified because by chance more trials are lost during EEG data pre- if desired, and exported to file formats including TIFF, JPG, processing for one than for another condition. While this is not PDF, EPS, and many others. necessarily a problem for classification analyses, DDTBOX takes a conservative approach and equalises the number of trials per category / condition before classification. Calculating block-average trials or pooling all trials Functional Structure of DDTBOX across blocks (Phase 5). The next step involves averaging across trials (i.e. exemplars) within each experimental block, The functional structure of DDTBOX is extensively described if this option was chosen. Alternatively, if there exist multiple in the wiki (https://github.com/DDTBOX/DDTBOX/wiki/ blocks, but the user chose to treat them all as one long exper- DDTBOX-Code-Structure) and will not be repeated here in iment, trials from each block are pooled at this stage. detail. The order of data processing steps in DDTBOX Sorting for classification (Phase 6). The data is now sorted for MVPA on single subject datasets is displayed in for the classification or regression process. For this, all trials Fig. 2A. The operations performed in DDTBOX for (or block-averaged trials) will be divided into the user- group-level statistical testing are illustrated in Fig. 2B. specified number of k sets (the default is k = 10), which also Advanced users, who might want to gain access to data specifies the number of cross-validation sets to be executed. after specific processing steps, or who are considering For each full cross-validation cycle (repeated m times; the expanding the toolbox at specific stages according to default is m = 10; see section 3.7 Statistical testing)trialsare their needs, can use this information to easily navigate through randomly assigned to one of the sets with the restriction that the code. no set can have more trials than the others (left-over trials are The following section will only provide a brief overview of excluded for this cycle). Of these sets, k – 1 are randomly the functional structure, which is divided into phases: assigned to the training data variable while the left-out set is assigned to the test data variable. All k combinations are Data Preparation (Phase 1). Includes preparation of the stored before the random assignment of trials to sets and their epoched data (see Section 5.1 below), as well as configuration sorting into training data and test data is performed again for of classification/regression analyses (as covered in the previ- all m iterations. For SVR, an additional matrix containing one ous section). value (the condition label) for each trial is used and substitutes Reading the data (Phase 2). Thisdataistransformed into a for the class labels. MATLAB cell array with the following format: Vector preparation (Phase 7): After sorting data into train- ing and test sets, DDTBOX extracts data from within the analysis time window and reshapes data from each trial into whereby run refers to the experimental block (if no a single vector. These vectors are then used for training and testing the SVM classification or regression model. separate blocks exist in the data, run will be 1), cond is 34 Neuroinform (2019) 17:27–42 Fig. 2 Functional structure of DDTBOX. (a) The single subject data accept single subject MVPA results and group analysis configuration decoding functions accept epoched data and analysis configuration parameters. Decoding performance and feature weights are aggregated parameters. Epoched data is extracted for selected analysis time over single subjects and are statistically tested at the group level. Multiple windows, and sorted for SVM classification or regression, for each comparisons corrections are applied as specified by the user. After anal- cross-validation step and each independent analysis (full set of cross- yses, DDTBOX can plot the group decoding accuracy and feature validation steps). SVM classification/regression is performed in weights results LIBSVM or LIBLINEAR. (b) Group-level statistical analysis functions Using DDTBOX can, in principle, also be organised within the same cell array structure for use with DDTBOX by advanced users (for more Preparing and running MVPA in DDTBOX involves four information see the online documentation). stages: preparing the data, configuring and running the decoding analyses, configuring and running group-level anal- yses, and plotting and interpreting the group results. Each of Configuring and Running the Decoding Analyses these are briefly described below. DDTBOX uses a decoding analysis configuration script for defining all relevant parameters and running decoding analy- Preparation of EEG Data ses. Within this script the user can define single subject data filepaths, EEG dataset information, experimental conditions FordecodinganalysesDDTBOX usesepoched data, asde- and discrimination groups, and a wide variety of multivariate scribed in section 4. Each participant dataset is saved as a classification/regression analysis parameters. Finally, the sub- separate data file. Epoched EEG data must be sorted by ex- jects and discrimination groups for analyses are defined, and perimental condition and run/block, and then stored in this the DDTBOX core decoding functions are called from this array. If applicable, SVR labels are stored within a separate script. Users can copy and adapt these scripts for their own cell array, with labels ordered in the same way as the corre- experiments; all parameters are clearly explained in the code sponding epochs in the EEG data array. A function for auto- comments of the script. matically converting EEG data epoched using EEGLAB or Once all the configuration parameters have been specified, ERPLAB is provided with the toolbox. This function can also the user can run decoding analyses by executing the extract epoched independent component activations in addi- MATLAB configuration script. SVM classification/ tion to EEG amplitudes. This function can further generate regression performance and feature weights information will SVR labels files for each condition. Other data types (such be stored in a separate file for each subject. as behavioural or steady-state visual evoked potential data) Neuroinform (2019) 17:27–42 35 Configuring and Running Group-Level Analyses LIBSVM to compare the first dataset to each of the other datasets with added signals. We used window and step sizes Group-level statistical analyses of classification/regression of 10 ms, 10-fold cross-validation and 10 independent repeti- performance and feature weights are configured and run tions of cross-validated analyses. We also calculated absolute using a group-level analysis configuration script. Within SVM feature weights corrected using the Haufe et al. (2014) this script the user must define the filepaths of decoding method. results files, EEG dataset information, group-level statisti- Fig. 4 displays the results of the validation analyses, show- cal analysis and plotting parameters, and must specify the ing chance-level classification performance during the first subjects and discrimination groups to use for analyses. 50 ms of the simulated epoch where there are no systematic Running this configuration script will perform all specified differences between the datasets. Classification accuracy then group-level statistical analyses on classification/regression increases above chance from 51 to 100 ms according to the performance and feature weights, which can also be plotted amount of signal (relative to the noise) in each dataset. Plots at this stage if desired. of feature weights for the time window 51-60 ms (when the signals were present) show that only the first 10 features (those Plotting and Interpreting the Group Results containing the signal) have large weights, and that feature weights are larger for datasets with higher signal-to-noise ratios. DDTBOX offers a variety of plotting options for classification/ regression performance and feature weights results at the group level. These may be performed when running group-level sta- Examples of Research Using DDTBOX tistical analyses, and can be replotted using a separate set of easy-to-configure plotting scripts. In this section, we briefly review some studies that have used For spatial and spatiotemporal decoding analyses group DDTBOX to investigate cognitive functions. We will use average classification/regression performance is plotted for these to illustrate some recent research questions for which each selected time window in the epoch, for results of both MVPA analysis has been profitably applied to ERP data; how- original and permuted labels decoding analyses (Fig. 3A). For ever, there are many other potential research questions for temporal decoding analyses group average performance for a which DDTBOX could be used. single analysis time window is plotted as a topographic heat DDTBOX owes its name to its first application in perceptual map (Fig. 3B). Feature weights are also plotted in this way, decision-making (Bode et al. 2012). In this EEG study, images of and can also be plotted as a map thresholded for statistical pianos and chairs were presented after a 100 ms forward mask significance (Fig. 3C). and longer backward mask (500 ms minus the duration of the target stimulus, which was either 16.7 ms, 33.3 ms, 50 ms, or 66.7 ms). A randomised response mapping screen was shown Toolbox Validation Using Simulated Data after the backward mask, circumventing early motor preparation. DDTBOX was used to predict the displayed object category, as To demonstrate that the toolbox functions correctly we ran well as participants’ category choices, at all four discriminability single subject decoding analyses using simulated EEG data. levels. First, a spatial classification approach was applied, using These analyses were designed to show that the basic functions 80 ms analysis time windows moved in steps of 20 ms. It was of the toolbox work properly, rather than to evaluate all capa- found that the spatial patterns of EEG data predicted the bilities of the toolbox. We created a subject dataset consisting displayed as well as the chosen category during the presentation of 100 timepoints, 64 channels and 1000 epochs. Samples for of the poststimulus mask, with decreasing accuracy and fewer each timepoint and channel in each epoch were sampled from predictive time windows with decreasing discriminability of the independent Gaussian random noise (mean = 0, standard de- objects (Bode et al. 2012). The study also presented phase- viation = 1). A second dataset was created in the same way, randomised visual noise images at the shortest presentation du- except that a signal value of 0.05 was added to the first 10 ration (16.7 ms), but participants believed themselves to be channels for timepoints 51–100. Three more datasets were guessing real object categories. Participants’ choices could be generated in this way, instead adding values of 0.1, 0.2 and predicted from activity patterns from the pre-stimulus time peri- 0.3. All datasets contained Gaussian noise only at timepoints od. This was interpreted as brain activity reflecting pre-existing 1–50, but differed (due to the addition of the signal values) at decision biases resulting from carry-over effects of decisions in timepoints 51–100. We note that temporally independent previous trials. To identify channels likely to contain this predic- noise is not typical of real EEG data, but it sufficient for the tive information, complementary temporal classification analy- purposes of our simulations. ses, using data from each channel separately, were performed for We then performed spatial decoding using DDTBOX func- selected time windows showing high group classification accu- tions and C-support vector classification as implemented in racy in the spatial decoding analysis. Temporal decoding 36 Neuroinform (2019) 17:27–42 Correct vs. Error Responses - Spatio-Temporal Decoding Actual Decoding Results Permuted-Labels Decoding Results Accuracy [%] p < .05 (after correction for multiple comparisons) -500 -450 -400 -350 -300 -250 -200 -150 -100 -50 0 50 100 150 200 250 300 350 Time From Response Onset [ms] B Temporal Decoding Results channels in red 53 Accuracy (corrected for multiple [%] comparisons) C Z-Standardised Absolute Feature Weights 0.8 0.6 0.4 0.2 channels in red 0 Z (uncorrected for multi - -0.2 ple comparisons) -0.4 -0.6 -0.8 Fig. 3 Examples of group-level results outputs produced by DDTBOX. scalp map plots group average classification accuracy for each channel. (a) Group average classification accuracy scores by time window from The map on the right highlights in red the channels showing decoding response onset. The black line represents the actual decoding results, blue accuracy scores that were statistically significantly above zero. (c)Feature line is the permuted-labels analysis results. Error bars represent standard weights results averaged over time windows spanning 100-300 ms from errors of the mean. Shaded time windows are statistically significant after response onset. The left scalp map displays group averages of z- correction for multiple comparisons. (b) Temporal decoding results. A standardised absolute feature weights. The map on the right highlights single time window was selected for temporal decoding analyses (100- in red the feature channels with feature weights with z-scores that were 300 ms from response onset). This time range approximates the timing of significantly above zero the error positivity ERP component in Bode and Stahl (2014). The left analyses showed that channels predominantly over the visual related information was found for both channels over visual cor- cortex encoded object information early after stimulus presenta- tex and prefrontal cortex during the pre-stimulus period. Taken tion, while prefrontal channels did so during later stages before together, these results demonstrate that the classification analyses response preparation. For the pure-noise condition, decision- as implemented in DDTBOX can indeed detect subtle decision- Neuroinform (2019) 17:27–42 37 Amount of Signal Present AB [Proportion of Gaussian Noise SD] 0.05 0.1 0.2 0.3 100 0.6 Signal Added 90 10 0.05 0.1 80 0.2 0.3 70 Average Accuracy Absolute Channel [%] Feature Weights 1-10 11-20 21-30 31-40 41-50 51-60 61-70 71-80 81-90 91-100 Time Steps [ms] Fig. 4 Results of toolbox validation analyses using simulated data. (a) scores were averaged across 10 cross-validation steps and 10 analysis Classification accuracies for four separate analyses, each classifying repetitions. (b) Absolute SVM feature weights during time window 51- between one noise dataset (consisting of independent samples of 60 ms, averaged over cross-validation steps and analysis repetitions. Gaussian noise, mean = 0, SD = 1) and one signal dataset consisting of Larger feature weights are visible for channels 1–10 in datasets with Gaussian noise plus a signal. This signal consisted of values 0.05, 0.1, 0.2 larger signals relative to the noise or 0.3 added to the first 10 channels during timepoints 51–100. Accuracy related information, which would have gone unnoticed in con- adults (Chan et al. 2017). Another application of ventional ERP analyses. In contrast to the MVPA results, no ERP DDTBOX has been the use of SVR to predict post- components selected for analyses showed differential activity experimental ratings of affective and abstract stimulus related to piano and chair decisions, or differences in prestimulus attributes of task-irrelevant images to inform theories baseline activity by category decision for the pure noise condi- of automatic processing of stimulus features during pas- tion. This is likely due to subject-specific patterns of EEG activity sive exposure (Bode et al. 2014; Turner et al. 2017). that differ in response to pianos and chairs, which may not be These latter examples demonstrate that DDTBOX is by consistent across subjects and ‘average-out’ in conventional uni- no means restricted to applications in decision-making. variate ERP analyses. On the contrary, it lends itself to many possible ques- In another EEG study, spatiotemporal classification was per- tions for which conventional ERP analyses might not be formed using DDTBOX to predict whether an upcoming re- suited, such as cases in which the specific encoding sponse for a parity decision in a speeded digit flanker task was patterns and the timing of these patterns are unknown correct or erroneous (Bode and Stahl 2014). Participants were prior to the experiment. asked to indicate, using one of two response buttons, whether a central digit on the screen was odd or even, in the presence of two flanker digits on each side that were also odd or even, thereby creating congruent or incongruent decision conditions. For Versioning and Release Management MVPA 10 ms analysis time windows were used, moved in steps of 10 ms through the trial, approaching the behavioural response To denote releases we use semantic versioning with the ver- onset. MVPA revealed that EEG activity patterns from 100 ms sion number format X.Y.Z (https://semver.org/spec/v2.0.0. before response execution already predicted whether the upcom- html). X denotes the major version number (e.g. v.1.0.0, v2. ing response would be erroneous, while conventional ERP anal- 0.0). Changes to X indicate backwards-incompatible changes yses found that the error-related negativity (ERN), which follows to the structure of the toolbox code. Changes to Y denote a a response by 80-100 ms, was the first ERP component to predict minor version release (e.g. v1.1.0, v1.2.0) indicating added decision errors (Bode and Stahl 2014). Follow-up analyses of features or capabilities. Changes to Z denote the patch version feature weights suggested that this early information originated releases (e.g. v.1.0.1, v1.0.2) which include bugfixes and code from channels over visual and motor cortices. In this study clas- documentation modifications. The current toolbox release is v. sification analyses performed using DDTBOX provided infor- 1.0.3, which incorporates minor bugfixes following the initial mation related to decision errors preceding the participants’ re- public release. sponses, and informed theories of how information about up- The numbered releases are tracked via Zenodo (https://doi. coming decision errors could accumulate over time to support org/10.5281/zenodo.593871), which archives a snapshot of online error monitoring processes (Bode and Stahl 2014). the code and assigns each release a DOI, allowing for users DDTBOX has also been used to investigate percep- to cite specific versions and guaranteeing the later availability tual categorisation of faces (Quek and Rossion 2017) of those versions. This serves to support reproducible and multi-sensory integration in elderly and younger analyses. 38 Neuroinform (2019) 17:27–42 Comparisons to other Packages first notable limitation is the support for MVPA using dif- for Time-Series Multivariate Data Analysis ferent types of input data. At this stage, DDTBOX can per- form MVPA on frequency domain and time-frequency data, Other toolboxes have also been developed for MVPA as well as component activations from principal compo- on EEG and neuroimaging data. Below we outline the nents analysis (PCA) or independent components analysis most influential toolboxes in the field and offer broad (ICA). However, DDTBOX does not yet offer result plot- comparisons to DDTBOX in terms of analysis options ting capabilities, or automatic conversion to DDTBOX- and compatibility with EEG datasets. Given that there is compatible data files, for these data types. Future support active development of DDTBOX and other toolboxes, for these data types will widen the applicability of any limitations of each toolbox may be overcome in DDTBOX for use with different experimental designs, for the near future. example studies examining multivariate patterns of steady- CoSMoMVPA (Oosterhof et al. 2016)isatoolbox state visual evoked potential (SSVEP) data (e.g., Jacques for MVPA of fMRI and M/EEG data running in et al. 2016). In particular, decoding with principal or inde- MATLAB and OCTAVE. CoSMoMVPA offers support pendent components may also help improve decoding ac- for a wide variety of MVPA methods, including some curacy compared to EEG amplitudes (Grootswagers et al. classification-based analyses not yet available in 2016). DDTBOX, such as temporal generalisation and repre- Another current limitation of DDTBOX is its restric- sentational similarity analysis (Kriegeskorte et al. tion to using the same analysis time window for train- 2008). However, CoSMoMVPA does not offer multivar- ing and testing. Others have suggested that one strength iate regression methods such as SVR or analyses of of the multivariate approach is that the temporal feature weights. DDTBOX also offers more extensive generalisability of patterns across time can be investigat- functionality for visualising results and performing ed (Meyers et al. 2008; Carlson et al. 2011;Kingand group-level analyses. CoSMoMVPA has extensive doc- Dehaene 2014; Fahrenfort et al. 2017). For this, a clas- umentation and tutorial material, but requires more ad- sifier could be trained on data from one time window vanced programming skills compared with DDTBOX. and thentestedatother time windowstoassess the MNE-Python (Gramfort et al. 2013)alsoprovidessup- duration for which the same training data successfully port for single-subject MVPA analyses, with direct sup- predicts the cognitive process (or content) of interest. port for temporal generalization and spatial decoding via By using all possible combinations of training and test spatial filters (Common Spatial, Effect-Matched-Spatial data, a full generalisation matrix can be compiled that is Filter). MNE-Python has extensive documentation and informative about the temporal dynamics of cognition tutorial material; however, it is nearly all focused on (c.f. Fig. 3 in King and Dehaene 2014;see also single-subject analyses with a strong MEG bias. While Hogendoorn 2015). Temporal generalisation analyses MNE-Python offers great flexibility it also requires ad- will be added to a future version of DDTBOX. vanced programming skills. A final noteworthy limitation is that the user is re- In addition, there are several MVPA toolboxes specialised for quired to extract epoched data from EEGLAB/ERPLAB, fMRI data with a more extensive range of analysis options than and to create a configuration script containing all nec- DDTBOX, including The Decoding Toolbox (Hebart et al. essary information about the study and planned analy- 2015), The Princeton MVPA Toolbox (http://code.google.com/ ses, before using DDTBOX. While we provide a user- p/princeton-mvpa-toolbox/), Pattern Recognition of friendly wiki, example configuration scripts, and func- Neuroimaging Toolbox (PRoNTo; Schrouff et al. 2013)and tions for automatically extracting data epoched using the RSA Toolbox (Nili et al. 2014) written in MATLAB, and EEGLAB/ERPLAB, the use of DDTBOX nevertheless PyMVPA (Hanke et al. 2009b) written in Python. These could, in requires some basic knowledge of MATLAB. Our aim principle, be applied to EEG data by those with advanced pro- is that the next release will also function as a plug-in gramming skills. However, these toolboxes currently offer limit- for EEGLAB, providing users with a graphical user in- ed options for visualisation of time series MVPA results and terface (GUI) within the EEGLAB environment to input preparation of EEG data for decoding analyses. all DDTBOX configuration parameters, and the option to use data directly from EEGLAB. However, we are confident that the current release will be of great benefit Limitations, Future Developments for the research community, and our toolbox can easily and Extensions be handled without a GUI. In addition to extensions planned by the core devel- Although it includes a range of analysis options, the current opers, user-contributed features are welcome and en- couraged. DDTBOX users have actively contributed to version of DDTBOX is still limited in several ways. The Neuroinform (2019) 17:27–42 39 the toolbox since the initial public release. Users have openly-available on GitHub, we invite all users to contribute to suggested new features, reported bugs and contributed DDTBOX by submitting their own extensions and improve- code to extend toolbox capabilities. Procedures and ments. Authors of accepted contributions will be acknowledged guidelines for submitting community-contributed code in future releases. With DDTBOX, we are hoping to provide a are available at https://github.com/DDTBOX/DDTBOX/ useful toolbox for multivariate EEG analysis that can grow with wiki/Contributing-to-DDTBOX. the needs of researchers and new directions in the field, driven To preserve the stability and usability of toolbox code we and developed further by an active community of users. will also add a suite of unit tests to a future DDTBOX release. This test suite will input simulated data into each function and ensure that the code runs without errors, and that the function Information Sharing Statement outputs match predetermined expected values. All user- and developer-made code modifications will need to pass these DDTBOX (RRID:SCR_015978) is freely-available at https:// unit tests before they can be incorporated into the toolbox. github.com/DDTBOX/DDTBOX with the respective software documentation at https://github.com/DDTBOX/DDTBOX/wiki. Data used for generating Fig. 3 are available for download at User Support https://osf.io/bndjg/. DDTBOX runs on MATLAB, available at http://www.mathworks.com/products/matlab. Technical support for DDTBOX users is available via our mail- ing list (https://www.freelists.org/list/ddtbox). Questions and Acknowledgements The DDTBOX was inspired by SB’s work with Prof discussion points can be posted to this list and will be John-Dylan Haynes on MVPA for fMRI, and some features of the code answered by the core developers as well as the broader were modelled from code developed in the Haynes lab. We acknowledge helpful input from Dr. Carsten Bogler and Dr. Chun Siong Soon during community of DDTBOX users. Bugs and requests for new this time. We are further thankful for important conceptual input and features can also be reported via the issue tracker on Github improvements resulting from collaborative work with Prof Jutta Stahl, (https://github.com/DDTBOX/DDTBOX/issues). Dr. Simon Lilburn, Prof Philip L. Smith, Dr. Elaine Corbett, Dr. Carsten Murawski and Dr. Owen Churches. Authors Contributions The DDTBOX has been developed and written Summary by SB, with significant contributions by DB, DF and PMA. All authors contributed to the online documentation and developed the learning ma- To conclude, DDTBOX is a freely available, open- terial. SB, DF wrote the paper. All authors contributed to and approved source toolbox for MATLAB that can be used for mul- the final version of the paper and agreed to be accountable for the content of this work. tivariate pattern classification and regression analyses on spatial, temporal and spatiotemporal patterns of EEG Funding SB was funded by an Australian Research Council Discovery data. It is useful for investigating cognitive processes Early Career Researcher Award (ARC DECRA DE140100350). related to decision-making, object categorisation, percep- tion, and potentially many other cognitive phenomena. Compliance with Ethical Standards This class of predictive methods can be used in a more explorative and data-driven fashion than conventional Conflict of Interest The authors declare no conflict of interest. No pay- ERP analyses. DDTBOX has been used in several pub- ments were received by neither the institutions nor funding agencies to create this toolbox, and institutions and funding agencies had no input lished studies and allows for detecting even subtle in- into the content of the work or the publication. formation that might be overlooked by standard ERP analyses. DDTBOX incorporates a variety of statistical Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http:// tests, and the option to perform permuted-labels analy- creativecommons.org/licenses/by/4.0/), which permits unrestricted use, ses to generate empirical chance distributions. It also distribution, and reproduction in any medium, provided you give appro- generates feature weight maps, which provide useful priate credit to the original author(s) and the source, provide a link to the estimates of the origins of the decodable information. Creative Commons license, and indicate if changes were made. DDTBOX is released under a GNU General Public License (GPL) v.2.0, meaning that users are free to References share, modify and extend the toolbox as desired. DDTBOX and the respective documentation is available Allefeld, C., Görgen, K., & Haynes, J. D. (2016). Valid population infer- at: https://github.com/DDTBOX/DDTBOX. ence for information-based imaging: From the second-level t-test to The developers are working on improving DDTBOX on a prevalence inference. NeuroImage, 141,378–392. regular basis. Users can subscribe to our mailing list and will be Bai, O., Lin, P., Vorbach, S., Li, J., Furlani, S., & Hallett, M. (2007). regularly updated about new releases and features. As the code is Exploration of computational methods for classification of 40 Neuroinform (2019) 17:27–42 movement intention during human voluntary movement from single Delorme, A., & Makeig, S. (2004). EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics including independent com- trial EEG. Clinical Neurophysiology, 118,2637–2655. Benjamini, Y., & Hochberg, Y. (1995). Controlling the false discovery ponent analysis. Journal of Neuroscience Methods, 134,9–21. rate: A practical and powerful approach to multiple testing. Journal Edelman, S., Grill-Spector, K., Kushnir, T., & Malach, R. (1998). Toward of the Royal Statistical Society Series B: Statistical Methodology, 57, direct visualization of the internal shape representation space by 289–300. fMRI. Psychobiology, 26,309–321. Benjamini, Y., Krieger, A. M., & Yekutieli, D. (2006). Adaptive linear El Zein, M., Wyart, V., & Grezes, J. (2015). Anxiety dissociates the step-up procedures that control the false discovery rate. Biometrika, adaptive functions of sensory and motor response enhancements to 491–507. social threats. eLife, 4, e10274. https://doi.org/10.7554/eLife.10274. Blair, R. C., & Karniski, W. (1993). An alternative method for signifi- Fahrenfort, J. J., van Leeuwen, J., Olivers, C. N., & Hogendoorn, H. cance testing of waveform difference potentials. Psychophysiology, (2017). Perceptual integration without conscious access. 30,518–524. Proceedings of the National Academy Sciences of the United Blank, H., Biele, G., Heekeren, H. R., & Philiastides, M. G. (2013). States of America, 114,3744–3749. Temporal characteristics of the influence of punishment on percep- Fan, R. E., Chang, K.-W., Hsieh, C. J., Wang, X. R., & Lin, C. J. (2008). tual decision making in the human brain. The Journal of LIBLINEAR: a library for large linear classification. Journal of Neuroscience, 33,3939–3952. Machine Learning Research, 9,1871–1874. Blankertz, B., Lemm, S., Treder, M., Haufe, S., & Müller, K. R. (2011). Friston, K., Holmes, A., Price, C., Buchel, C., & Worsley, K. (1999). Single-trial anal-ysis and classification of ERP components – A Multisubject fMRI studies and conjunction analysis. NeuroImage, tutorial. NeuroImage, 56,814–825. 10,385–396. Bode, S., & Stahl, J. (2014). Predicting errors from patterns of event- Gevins, A. S., Zeitlin, G. M., Yingling, C. D., Doyle, J. C., Dedon, M. F., related potentials preceding an overt response. Biological Schaffer, R. E., Roumasset, J. T., & Yeager, C. L. (1979). EEG patterns Psychology, 103,357–369. during ‘cognitive’ tasks. I. Methodology and analysis of complex be- Bode, S., Sewell, D. K., Lilburn, S., Forte, J., Smith, P. L., & Stahl, J. haviors. Electroencephalography and Clinical Neurophysiology, 47, (2012). Predicting perceptual decision biases from early brain activ- 793–703. ity. The Journal of Neuroscience, 32,12488–12498. Gramfort, A., Luessi, M., Larson, E., Engemann, D., Strohmeier, D., Bode, S., Bogler, C., & Haynes, J. D. (2013). Similar neural mechanisms Brodbeck, C., Goj, R., Jas, M., Brooks, T., Parkkonen, L., & for guesses and free decisions. NeuroImage, 65,456–465. Hämäläinen, M. (2013). MEG and EEG data analysis with MNE- Bode, S., Bennett, D., Stahl, J., & Murawski, C. (2014). Distributed Python. Frontiers in Neuroscience, 7. https://doi.org/10.3389/fnins. patterns of event-related potentials predict subsequent ratings of 2013.00267. abstract stimulus attributes. PLoS One, 9, e109070. https://doi.org/ Grootswagers, T., Wardle, S. G., & Carlson, T. A. (2016). Decoding dynamic 10.1371/journal.pone.0109070. brain patterns from evoked responses: A tutorial on multivariate pattern Boldt, A., & Yeung, N. (2015). Shared neural markers of decision confi- analysis applied to time series neuroimaging data. Journal of Cognitive dence and error detection. Journal of Neuroscience, 35,3467–3484. Neuroscience, 29, 677–697. Burges, C. J. (1998). A tutorial on support vector machines for pattern Groppe, D. M., Urbach, T. P., & Kutas, M. (2011). Mass univariate anal- recognition. Data Mining and Knowledge Discovery, 2,121–167. ysis of event-related brain potentials/fields I: A critical tutorial re- Carlson, T. A., Hogendoorn, H., Kanai, R., Mesik, J., & Turret, J. (2011). view. Psychophysiology, 48,1711–1725. High temporal resolution decoding of object position and category. Hanke, M., Halchenko, Y. O., Sederberg, P. B., Olivetti, E., Fründ, I., JournalofVision, 11. https://doi.org/10.1167/11.10.9. Rieger, J. W., Herrmann, C. S., Haxby, J. V., Hanson, S. J., & Chan, Y. M., Pianta, M. J., Bode, S., & McKendrick, A. M. (2017). Pollmann, S. (2009a). PyMVPA: A unifying approach to the analy- Neural correlates of audiovisual synchrony judgements in older sis of neuroscientific data. Frontiers in Neuroinformatics, 3,3. adults. Neurobiology of Aging, 55,38–48. https://doi.org/10.3389/neuro.11.003.2009. Chang, C. C., & Lin, C. J. (2011). LIBSVM: a library for support vector Hanke, M., Halchenko, Y. O., Sederberg, P. B., Hanson, S. J., Haxby, J. V., & machines. ACM Transactions on Intelligent Systems and Pollman, S. (2009b). PyMVPA: A python toolbox for multivariate pat- Technology, 2,27. tern analysis of fMRI data. Neuroinformatics, 7,37–53. Charles, L., King, J. R., & Dehaene, S. (2014). Decoding the dynamics of Hastie, T., Tibshirani, R., & Friedman, J. (2001). The elements of statis- action, intention, and error-detection for conscious and subliminal stim- tical learning. Heidelberg: Springer. uli. The Journal of Neuroscience, 34,1158–1170. Haufe, S., Meinecke, F., Görgen, K., Dähne, S., Haynes, J. D., Blankertz, Chung, D., Yun, K., & Jeong, J. (2015). Decoding covert motivations of B., & Bießmann, F. (2014). On the interpretation of weight vectors free riding and cooperation from multi-feature pattern analysis of of linear models in multivariate neuroimaging. NeuroImage, 87,96– EEG signals. Social Cognitive and Affective Neuroscience, 10, 1210–1218. Haxby, J. V., Gobbini, M. I., Furey, M. L., Ishai, A., Schouten, J. L., & Combrisson, E., & Jerbi, K. (2015). Exceeding chance level by chance: Pietrini, P. (2001). Distributed and overlapping representations of The caveat of theoretical chance levels in brain signal classification faces and objects in ventral temporal cortex. Science, 293, 2425– and statistical assessment of decoding accuracy. Journal of Neuroscience Methods, 250,126–136. Haynes, J. D. (2015). A Primer on Pattern-Based Approaches to fMRI: Contini, E. W., Wardle, S. G., & Carlson, T. A. (in press). Decoding the Principles, Pitfalls, and Perspectives. Neuron, 87,257–270. time-course of object recognition in the human brain: From visual Haynes, J. D., & Rees, G. (2006). Decoding mental states from brain features to categorical decisions. Neuropsychologia. activity in humans. Nature Review Neuroscience, 7,523–534. Cortes, C., & Vapnik, V. (1995). Support-vector networks. Machine Hebart, M. N., Görgen, K., & Haynes, J. D. (2015). The Decoding Learning, 20,273–297. https://doi.org/10.1007/BF00994018. Das, K., Giesbrecht, B., & Eckstein, M. P. (2010). Predicting variations of Toolbox (TDT): a versatile software package for multivariate anal- yses of functional imaging data. Frontiers in Neuroinformatics, 8, perceptual performance across individuals from neural activity using pattern classifiers. NeuroImage, 51,1425–1437. 88. https://doi.org/10.3389/fninf.2014.00088. Davis, T., & Poldrack, R. (2013). Measuring neural representations with Heinzle, J., Anders, S., Bode, S., Bogler, C., Chen, Y., Cichy, R. M., Hackmack, K., Kahnt, T., Kalberlah, C., Reverberi, C., Soon, S. fMRI: practices and pitfalls. Annals of the New York Academy of Sciences, 1296,108–134. C.,Tusche,A.,Weygandt,M.,&Haynes,J.D.(2012). Neuroinform (2019) 17:27–42 41 Multivariate decoding of fMRI data – Towards a content-based cog- Norman, K. A., Polyn, S. M., Detre, G. J., & Haxby, J. V. (2006). Beyond mind-reading: multi-voxel pattern analysis of fMRI data. Trends in nitive neuroscience. e-Neuroforum, 3,1–16. Cognitive Sciences, 10,424–430. Hogendoorn, H. (2015). From sensation to perception: Using multivariate classification of visual illusions to identify neural correlates of con- Oostenveld,R.,Fries,P.,Maris, E., and Schoffelen, J.-M. (2011). FieldTrip: Open source software for advanced analysis of MEG, scious awareness in space and time. Perception, 44,71–78. EEG, and invasive electrophysiological data. Computational Hogendoorn, H., & Verstraten, F. A. (2013). Decoding the motion after- Intelligence and Neuroscience, (2011). https://doi.org/10.1155/ effect in human visual cortex. NeuroImage, 82,426–432. 2011/156869. Hogendoorn, H., Verstraten, F. A., & Cavanagh, P. (2015). Strikingly Oosterhof, N. N., Connolly, A. C., & Haxby, J. V. (2016). CoSMoMVPA: rapid neural basis of motion-induced position shifts revealed by high Multi-modal multivariate pattern analysis of neuroimaging data in temporal-resolution EEG pattern classification. Vision Research, Matlab/GNU Octave. Frontiers in Neuroinformatics, 10,27. https:// 113,1–10. doi.org/10.3389/fninf.2016.00027. Holm, S. (1979). A simple sequentially rejective multiple test procedure. Parra, L. C., Alvino, C., Tang, A., Pearlmutter, B., Yeung, N., Osman, A., Scandanavian Journal of Statistics, 6,65–70. & Sajda, P. (2002). Linear spatial integration for single-trial detec- Isik, L., Meyers, E. M., Leibo, J. Z., & Poggio, T. (2014). The dynamics tion in encephalography. NeuroImage, 17,223–230. of invariant object recognition in the human visual system. Journal Parra, L. C., Spence, C. D., Gerson, A. D., & Sajda, P. (2005). Recipes for of Neurophysiology, 111,91–102. the linear analysis of EEG. NeuroImage, 28,326–341. Jacques, C., Retter, T. L., & Rossion, B. (2016). A single glance at natural Philiastides, M. G., & Sajda, P. (2006). Temporal characterization of the face images generate larger and qualitatively different category-selective neural correlates of perceptual decision making in the human brain. spatio-temporal signatures than other ecologically-relevant categories in Cerebral Cortex, 16,509–518. the human brain. NeuroImage, 137,21–33. Philiastides, M. G., Ratcliff, R., & Sajda, P. (2006). Neural representation of King, J. R., & Dehaene, S. (2014). Characterizing the dynamics of mental task difficulty and decision making during perceptual categorization: a representations: The temporal generalization method. Trends in timing diagram. The Journal of Neuroscience, 26, 8965–8975. Cognitive Sciences, 18,203–210. Quek, G.L., and Rossion, B. (2017). Category-selective human brain Korn, E. L., Troendle, J. F., Mcshane, L. M., & Simon, R. (2004). processes elicited in fast periodic visual stimulation streams are im- Controlling the number of false discoveries: application to high- mune to temporal predictability. Neuropsychologia, 104,182–200. dimensional genomic data. Journal of Statistical Planning and https://doi.org/10.1101/117135. Inferences, 124,379–398. Ratcliff, R., Philiastides, M. G., & Sajda, P. (2009). Quality of evidence for Kriegeskorte, N., Goebel, R., & Bandettini, P. (2006). Information-based perceptual decision making is indexed by trial-to-trial variability of the functional brain mapping. Proceedings of the National Academy of EEG. Proceedings of the National Academy of Sciences of the United Sciences of the United States of America, 103,3863–3868. States of America, 106,6539–6544. Kriegeskorte, N., Mur, M., & Bandettini, P. (2008). Representational Sajda, P., Philiastides, M. G., & Parra, L. C. (2009). Single-trial analysis similarity analysis - connecting the branches of systems neurosci- of neuroimaging data: inferring neural networks underlying percep- ence. Frontiers in Systems Neuroscience, 2,4. tual decision-making in the human brain. IEEE Reviews in Biomedical Engineering, 2,97–109. Kriegeskorte, N., Simmons, W. K., Bellgowan, P. S., & Baker, C. I. Schrouff, J., Rosa, M. J., Rondina, J. M., Marquand, A. F., Chu, C., (2009). Circular analysis in systems neuroscience: the danger of Ashburner, J., et al. (2013). PRoNTo: Pattern recognition for neuro- double dipping. Nature Neuroscience, 12,535–540. imaging toolbox. Neuroinformatics, 11,319–337. Lopez-Calderon, J., & Luck, S. (2014). ERPLAB: an open-source tool- Stelzer, J., Chen, Y., & Turner, R. (2013). Statistical inference and mul- box for the analysis of event-related potentials. Frontiers in Human tiple testing correction in classification-based multi-voxel pattern Neuroscience, 8,213. https://doi.org/10.3389/fnhum.2014.00213. analysis (MVPA): Random permutations and cluster size control. Luck, S. J. (2005). An introduction to the event-related potential NeuroImage, 65,69–82. technique.Cambridge:MIT. Stokes, M. G., Wolff, M. J., & Spaak, E. (2015). Decoding rich spatial Maris, E., & Oostenveld, R. (2007). Nonparametric statistical testing of information with high temporal resolution. Trends in Cognitive EEG- and MEG-data. Journal of Neuroscience Methods, 164,177– Sciences, 19,636–638. Tong, F., & Pratte, M. S. (2012). Decoding patterns of human brain Meyers, E. M., & Kreiman, G. (2011). Tutorial on pattern classification in activity. Annual Review of Psychology, 63,483–509. cell recordings. In N. Kriegeskorte & G. Kreiman (Eds.), Turner, W. F., Johnston, P., de Boer, K., Morawetz, C., & Bode, S. (2017). Understanding visual population codes. Boston: MIT Press. Multivariate pattern analysis of event-related potentials predicts the Meyers, E. M., Freedman, D. J., Kreiman, G., Miller, E. K., & Poggio, T. subjective relevance of everyday objects. Consciousness and (2008). Dynamic population coding of category information in in- Cognition. ferior temporal and prefrontal cortex. Journal of Neurophysiology, Tzovara, A., Chavarriaga, R., and De Lucia, M. (2015). Quantifying the 100,1407–1419. time for accurate EEG decoding of single value-based decisions. Müller, K. R., Krauledat, M., Dornhege, G., Curio, G., & Blankertz, B. Journal of Neuroscience Methods, 250,114–125. (2004). Machine learning techniques for brain-computer interfaces. Van Vugt, M. K., Simen, P., Nystrom, L. E., Holmes, P., & Cohen, J. D. Biomedical Technology, 49,11–24. (2012). EEG oscillations reveal neural correlates of evidence accu- Müller, K. R., Tangermann, M., Dornhege, D., Krauledat, M., Curio, G., mulation. Frontiers in Neuroscience, 6,106. https://doi.org/10. & Blankertz, B. (2008). Machine learning for real-time single-trial 3389/fnins.2012.00106. EEG-analysis: From brain-computer interfacing to mental state Wilcox, R. (2012). Introduction to Robust Estimation and Hypothesis monitoring. Journal of Neuroscience Methods, 167,82–90. Testing. Waltham MA: Academic Press. Mur, M., Bandettini, P. A., & Kriegeskorte, N. (2009). Revealing repre- Woolgar, A., Golland, P., & Bode, S. (2014). Coping with confounds in sentational content with pattern- information fMRI – an introductory multi-voxel analysis: what should we do about reaction time differ- guide. Social Cognitive and Affective Neuroscience, 4,101–109. ences? A comment on Todd, Nystrom & Cohen 2013. NeuroImage, Nili, H., Wingfield, C., Walther, A., Su, L., Marslen-Wilson, W., & 98,73–80. Kriegeskorte, N. (2014). A toolbox for representational similarity Woolgar, A., Jackson, J., & Duncan, J. (2016). Coding of visual, auditory, analysis. PLoS Computational Biology, 10, e1003553. rule, and response information in the brain: 10 years of multivoxel 42 Neuroinform (2019) 17:27–42 pattern analysis. Journal of Cognitive Neuroscience, 28, 1433– Yarkoni, T., and Westfall, J. (2017). Choosing prediction over explanation in psychology: Lessons from machine learning. Perspectives in Wyart, V., de Gardelle, V., Scholl, J., and Summerfiled, C. (2012). Psychological Science, 12(6), 1100–1122. https://doi.org/10.1177/ Rhythmic fluctuations in evidence accumulation during decision 1745691617693393 making in the human brain. Neuron, 76, 847–858. Yuen, K. K. (1974). The two-sample trimmed t for unequal population Wyart, V., Myers, N. E., & Summerfield, C. (2015). Neural mechanisms variances. Biometrika, 61,165–170. of human perceptual choice under focussed and divided attention. The Journal of Neuroscience, 35,3485–3498.
Journal
Neuroinformatics – Springer Journals
Published: May 2, 2018