TY - JOUR
AU - Soble, Jason, R
AB - Abstract Objective Performance validity research has emphasized the need for briefer measures and, more recently, abbreviated versions of established free-standing tests to minimize neuropsychological evaluation costs/time burden. This study examined the accuracy of multiple abbreviated versions of the Dot Counting Test (“quick” DCT) for detecting invalid performance in isolation and in combination with the Test of Memory Malingering Trial 1 (TOMMT1). Method Data from a mixed clinical sample of 107 veterans (80 valid/27 invalid per independent validity measures and structured criteria) were included in this cross-sectional study; 47% of valid participants were cognitively impaired. Sensitivities/specificities of various 6- and 4-card DCT combinations were calculated and compared to the full, 12-card DCT. Combined models with the most accurate 6- and 4-card combinations and TOMMT1 were then examined. Results Receiver operator characteristic curve analyses were significant for all 6- and 4-card DCT combinations with areas under the curve of .868–.897. The best 6-card combination (cards, 1-3-5-8-11-12) had 56% sensitivity/90% specificity (E-score cut-off, ≥14.5), and the best 4-card combination (cards, 3-4-8-11) had 63% sensitivity/94% specificity (cut-off, ≥16.75). The full DCT had 70% sensitivity/90% specificity (cut-off, ≥16.00). Logistic regression revealed 95% classification accuracy when 6-card or 4-card “quick” combinations were combined with TOMMT1, with the DCT combinations and TOMMT1 both emerging as significant predictors. Conclusions Abbreviated DCT versions utilizing 6- and 4-card combinations yielded comparable sensitivity/specificity as the full DCT. When these “quick” DCT combinations were further combined with an abbreviated memory-based performance validity test (i.e., TOMMT1), overall classification accuracy for identifying invalid performance was 95%. Assessment, Malingering/symptom validity testing, Mild cognitive impairment, Professional issues Introduction Assessment of performance validity is an essential component of neuropsychological evaluation that has become a practice standard, with professional organizations calling for routine use of both embedded and stand-alone performance validity tests (PVTs; Boone, 2009; Bush et al., 2005; Heilbronner et al., 2009). The necessity of PVTs is clear as there is potential for invalid performance in clinical settings because examinees may present with a variety of external incentives for a diagnosis of cognitive impairment (American Academy of Clinical Neuropsychology, 2007). Given that use of one PVT often does not allow for conclusive statements regarding credibility and evidence suggesting use of multiple PVTs allows for better classification of invalid performance, using ≥2 PVTs when there is not clear and abundant evidence of invalidity has become increasingly common in the field (Larrabee, 2014; Heilbronner et al., 2009). However, when trends of PVT use are examined, many clinicians report using measures that appear memory-based or rely on a forced-choice decision paradigm (Martin, Schroeder, & Odland, 2015; Young, Roper, & Arentsen, 2016). This is problematic given that relying on PVTs examining one domain may be ineffective for detecting invalid performance as examinees may opt to provide suboptimal performance in a different cognitive domain (Boone, 2009, 2012) or if PVTs within a domain vary in their diagnostic accuracy (e.g., Bailey, Soble, & O’Rourke, 2017). Indeed, recent research has also shown that memory-based and non-memory-based PVTs examine different underlying constructs (Webber, Critchfield, & Soble, 2018b). This provides a rationale for using multiple PVTs which appear to map onto different cognitive domains. Additionally, research supporting consideration of sample characteristics such as age and cognitive impairment status can guide adjustment of cut scores for commonly used PVTs (e.g., Bailey, Soble, Bain, & Fullen, 2018; Webber & Soble, 2018). Regardless of number of PVTs administered or where cut scores are set, a process approach to interpreting results can aid in ultimate validity status determination by providing supplementary data above and beyond the obtained raw score. Information can be gleaned from intratest data (e.g., lengthy response latency, verbally stating how difficult a test is while simultaneously giving a correct answer, being surprised if an examiner either purposefully or inadavertently gives incorrect feedback on a forced-choice measure where the examinee chose the incorrect response), as well as intertest data (e.g., comparing approach across multiple PVTs and cognitive tests). One primary process approach area that has been examined is reaction time (Gabel et al., 2019; Kanser, Rapport, Bashem, & Hanks, 2018; Stevens, Bahlo, Lich, Liske, & Vossler-Thies, 2016). Slowed reaction time has been identified as a potential covert means of differentiating valid from feigned performance (Kanser et al., 2018). Indeed, research has shown that delayed reaction time is often deliberately used by participants asked to simulate brain injury (Kanser et al., 2018; Tan et al., 2002). Reaction time may also be slowed due to the nature of invalid performance in that the person will come to an answer and then decide whether or not to give the correct answer (Bolan, Foster, Schmand, & Bolan, 2002). Several studies have found that examining reaction time in already-established PVTs resulted in greater incremental validity than relying on accuracy alone. Babikian, Boone, Lu, and Arnold (2006) showed similar discrimination potential for average time per digit (>1.5; 32% sensitivity/96% specificity) on Digit Span similar to that of Reliable Digit Span (RDS; ≤6; 45% sensitivity/93% specificity) in a mixed clinical sample. One study found that when examining a timed version of the Digit Span subtest from the from the Wechsler Adult Intelligence Scale-Fourth Edition (WAIS-IV; Wechsler, 2008), response latency was a better predictor of failure on the computerized version of the Test of Memory Malingering (TOMM; Tombaugh, 1996) than accuracy in a veteran population with history of mild traumatic brain injury (mTBI). Response time on non-PVT measures of attention also added to one’s ability to differentiate valid performance in a sample of compensation-seeking individuals with TBI/brain lesion, non-compensation-seeking individuals with TBI/brain lesion, and compensation-seeking individuals without TBI/brain lesion (Stevens et al., 2016). More recently, researchers have examined response patterns and test-taking behaviors in conjunction with PVT measures to allow for less time spent on administering the PVT (Webber et al., 2018a; Denning, 2014). When examining various versions of the embedded PVT in the Digit Span subset of the WAIS-IV, it was found that examining the age-corrected scaled score in combination with the Dot Counting Test (DCT; Boone, Lu, & Herzberg, 2002b) allowed for maximization of sensitivity and specificity in non-memory-based PVTs among cognitively unimpaired examinees (Webber et al., 2018b). It was also found that using behavioral response patterns on both the full TOMM and on the first 10 items (TOMMe10) resulted in slight increases in sensitivity for these measures (Denning, 2014). The utility of combining objective test scores with behavioral responses can allow for a more confident determination on performance validity using fewer, or in more extreme cases one, PVT if it is abundantly clear that poor engagement will result in invalid data and spoiling of protected cognitive tests. The DCT offers the clinical benefits of brief administration while examining performance validity in the context of reaction time/processing speed rather than memory. The DCT is a non-memory-based PVT that requires examinees to count grouped and ungrouped dots as quickly as possible, with the expectation that counting grouped dots will demand less time than ungrouped dots for valid performance populations. Although this measure was originally developed by Rey (1941), several administration (Frederick, 1997; Lezak, 1995; Rey, 1941) and scoring (Frederick, 1997; Frederick, Sarafaty, Johnston, & Powel, 1994; Martin, Hayes, & Gouvier, 1996; Paul, Franzen, Cohen, & Fremouw, 1992; Rose, Hall, & Szalda-Petree, 1998) procedures have been developed, which contributed to initially poor specificity and inconsistent findings in terms of classification accuracy (Boone, Lu, Back, et al., 2002a; Boone, Lu, & Herzberg, 2002b; Frederick, 2003; Rose et al., 1998). However, Boone and colleagues (2002a) demonstrated that the mean grouped dot counting time, the ratio of mean grouped to ungrouped dot counting time, and the number of errors (e.g., overall speed and accuracy) represented discrete constructs that, when combined, improved specificity and sensitivity. Specifically, using a cut-off score of ≥17 (known as the Effort Index or E-score) resulted in 75%–100% sensitivity and ≥90% specificity across forensic (i.e., civil litigation/disability, prison hospital), clinical (i.e., head injury, learning disability, stroke, schizophrenia, depression, mild dementia), and cognitively normal older adults populations. More recently, McCaul and colleagues (2018) provided preliminary evidence for non-rounded E-scores (≥13.8) or failure on any of the three obtained scores (i.e., ungrouped ≥4.2, grouped ≥8.85, or errors ≥4) yielding good sensitivity (64%–70%) while maintaining specificity of ≥90% for differentiating validity status in a nonoverlapping clinical/forensic sample. The accuracy of the DCT in classifying invalid performance does not appear to be affected by gender and appears only minimally affected by age, education, household income, and some psychiatric disorders (Back, Boone, Edwards, Burgoyne, & Silver, 1996; Boone, Lu, Back, et al., 2002a; Boone, Lu, & Herzberg, 2002b, Lee, 2000). However, research suggests that administration times may vary based on level of cognitive impairment (i.e., those with cognitive impairment were slower than those without across both grouped and ungrouped cards; Soble et al., 2018), though these differences are likely accounted for by the various cut-offs established for different clinical groups and do not appear to affect accuracy of classification (Boone et al., 2002b). Patients with suspected invalid performance may also take significantly longer on this PVT, as speed is one factor used to calculate cut-offs for classification accuracy of the DCT. While the DCT offers the benefit of a briefer administration time than other stand-alone PVTs, in the aforementioned populations, that time may be significantly longer, therefore negating a primary benefit of the DCT in clinical practice. The current study aimed to determine if a “quick” version of the DCT (i.e., utilizing various 6- and 4-card DCT combinations) could allow clinicians to efficiently test hypotheses about performance validity within a clinical battery. As such, the study sought to preliminarily validate various “quick” versions of the DCT in isolation, and combined with an abbreviated, memory-based (i.e., TOMM T1), to predict validity status determined by free-standing and embedded PVT measures. Additionally, preliminary validation of these versions in a mixed clinical sample will help determine which combination of ungrouped and grouped cards, combined with error frequency, is most sensitive and specific in determining validity status. Materials and Methods Participants Data from 112 veterans referred for neuropsychological assessment services for diagnostic clarification and treatment planning purposes due to patient and/or provider concerns for cognition from 2015 to 2017 who completed the Word Memory Test (WMT; Green, 2003); TOMMT1; Advanced Clinical Solutions Word Choice Test (WCT; Pearson, 2009); full DCT; and RDS (Greiffenstein, Baker, & Gola, 1994) from the WAIS-IV as part of a clinical evaluation conducted in English and consented to include their data as part of ongoing, IRB-approved database study were included. Given the breadth of evaluative settings served by the providers in this study (e.g., inpatient psychiatry/neurology, cognitive screening evaluations in an epilepsy clinic, etc.) a nonconsecutive approach was necessitated given the length of the battery. For the purposes of this study, the criteria for determining valid neuropsychological test performance were operationally defined as failing ≤1/3 criterion PVTs (i.e., WMT, WCT, RDS), allowing for one failure to remain classified as valid. Invalid performance was operationalized as failing ≥2/3 criterion PVTs and specific elements of the Slick criteria for detecting noncredible neurocognitive dysfunction (Slick, Sherman, & Iverson, 1999): notably: patterns of neuropsychological test scores that substantially deviate from accepted models of brain abnormality and between test data, observed behavior, patient or collateral reports, and/or documented clinical history in VA medical records, which often includes information pertaining to any external incentive (i.e., service-connection disability/compensation-seeking status) that may be present. Based on these criteria, 80 were valid, 27 were invalid, and there were 5 outliers. Further review of these five outliers revealed that four participants failed only one criterion PVT, but were clinically classified as invalid per the aforementioned criteria met Slick criteria when the entirety of their evaluative data was considered (i.e., presence of external incentive (3/4) and notable discrepancies between neuropsychological test data and accepted models of brain dysfunction, self-report, observed behavior, and documented history) and were therefore removed. One participant had documented borderline intellectual functioning with negative Slick criteria (i.e., consistent neuropsychological test data, collateral report, and documented history, as well as no identifiable external incentive), raising concern for the validity of the DCT (Dean, Victor, Boone, & Arnold, 2008). Notably, this participant’s DCT E-score was 5.16 SD above the mean for the valid group and was therefore excluded. The sample was predominately male (82%, n = 88), had a mean age of 54.66 years (SD = 14.91, Range = 24–84) with three modal ages 42/43 (n = 10), 52, (n = 6), and 63 (n = 6), and a mean education of 13.8 years (SD = 2.38, Range = 7–19). The sample was ethnically diverse with 46% (n = 49) Caucasian, 35% (n = 38) Hispanic, 15% (n = 16) African-American, and 4% (n = 4) other ethnic backgrounds. A total of 26% (n = 28) were English/Spanish bilingual. There were no significant differences between those classified as valid versus invalid by the WMT, WCT, and RDS due to age, F(1, 105) = 1.03, p = .31; education, F(1, 105) = .01, p = .91 race, X2 (3, N = 107) = 3.2, p = .36; or bilingualism, X2 (1, N = 107) = .05, p = .83. A total of 75% (n = 80) of the sample were classified as valid and 25% (n = 27) as invalid. Of the 80 valid participants, 47% (n = 38) were cognitively unimpaired, whereas 53% (N = 42) were diagnosed with a neurocognitive disorder following evaluation per the Diagnostic and Statistical Manual of Mental Disorders-Fifth Edition (DSM-5; APA, 2013) criteria. Primary diagnoses for the cognitively unimpaired participants were predominantly affective (depression n = 9; bipolar n = 1; anxiety n = 5), followed by PTSD (n = 3) and ADHD (n = 2). Individual participants were diagnosed with unspecified psychotic disorder, substance use disorder, and conversion disorders (n = 3), 4 had sleep disorders, and 11 did not meet criteria for any DSM condition. Of the cognitively impaired, 83% (N = 35) met criteria for a mild neurocognitive disorder and 17% (n = 7) for a major neurocognitive disorder. Vascular etiologies were predominant and accounted for 36% (n = 15) with the remainder of the sample being diagnosed with mild cognitive impairment-amnestic subtype/Alzheimer’s disease (n = 6; 14%), epilepsy (n = 4; 10%), severe traumatic brain injury (TBI; n = 3; 7%), other medical/neurological conditions (n = 8; 19%), or multiple etiologies (n = 6; 14%). Among the 27 invalid participants, psychiatric disorders predominated such that 18% (n = 5; 2 PTSD, 1 depression, 2 anxiety) had a primary psychiatric diagnosis, 67% (n = 18) had an active psychiatric diagnosis in the context of a remote TBI history (17 mild, 1 moderate), and 15% (n = 4) had an active psychiatric diagnosis along with a medical condition (i.e., stroke, tumor). Service connection data were available for 76% of the sample (n = 82). For the valid group, 78.8%/81.3% were service connected (M SC = 50.6/60.7%) at time of evaluation for cognitive unimpaired/impaired, respectively. A total of 82.4% of the invalid participants were service connected, with a mean SC percentage of 53.6%. A total of 37% of the invalid participants had an active SC claim pending at time of the evaluation compared to 5% of the valid participants. Measures Each participant was administered the full DCT. See Table 1 for passing/failing cut scores for the six PVTs included in this study protocol. The E-score for various 6- and 4-card DCT combinations was derived in the same manner as the full DCT (i.e., mean response time for ungrouped cards + mean response time for grouped cards + errors on any of the selected cards). TOMM T1 was selected as a second stand-alone PVT given empirical support for utility as a briefer version of a well-established PVT (Denning, 2012; Fazio, Denning, & Denney, 2017) and congruency with the rationale for the present study. Table 1 Primary PVTs and associated cut scores for test failure PVT Cut-off(s) for test failure Reference(s) WMT ≤82.5% on any primary effort subtest and no Genuine Memory Impairment Profile for those with clinical history of cognitive impairment Green (2003); Green et al. (2011); Alverson et al. (2019) TOMM T1 ≤40 Denning (2012) WCT ≤41 Bain and Soble (2017); Bain et al. (2019) DCT ≥14 Depression; PTSD; anxiety; mild TBI ≥15 Learning disorder ≥19 Mild neurocognitive disorder ≥20 Schizophrenia; TBI with positive imaging ≥21 Mild dementia Boone et al. (2002) RDS ≤6 Schroder et al. (2012) PVT Cut-off(s) for test failure Reference(s) WMT ≤82.5% on any primary effort subtest and no Genuine Memory Impairment Profile for those with clinical history of cognitive impairment Green (2003); Green et al. (2011); Alverson et al. (2019) TOMM T1 ≤40 Denning (2012) WCT ≤41 Bain and Soble (2017); Bain et al. (2019) DCT ≥14 Depression; PTSD; anxiety; mild TBI ≥15 Learning disorder ≥19 Mild neurocognitive disorder ≥20 Schizophrenia; TBI with positive imaging ≥21 Mild dementia Boone et al. (2002) RDS ≤6 Schroder et al. (2012) Note: PVT = Performance Validity Test; WMT = Word Memory Test; TOMM = Test of Memory Malingering; WCT = Advanced Clinical Solutions Word Choice Test; DCT = Dot Counting Testing; RDS = Wechsler Adult Intelligence Scale (WAIS-IV) Reliable Digit Span. Open in new tab Table 1 Primary PVTs and associated cut scores for test failure PVT Cut-off(s) for test failure Reference(s) WMT ≤82.5% on any primary effort subtest and no Genuine Memory Impairment Profile for those with clinical history of cognitive impairment Green (2003); Green et al. (2011); Alverson et al. (2019) TOMM T1 ≤40 Denning (2012) WCT ≤41 Bain and Soble (2017); Bain et al. (2019) DCT ≥14 Depression; PTSD; anxiety; mild TBI ≥15 Learning disorder ≥19 Mild neurocognitive disorder ≥20 Schizophrenia; TBI with positive imaging ≥21 Mild dementia Boone et al. (2002) RDS ≤6 Schroder et al. (2012) PVT Cut-off(s) for test failure Reference(s) WMT ≤82.5% on any primary effort subtest and no Genuine Memory Impairment Profile for those with clinical history of cognitive impairment Green (2003); Green et al. (2011); Alverson et al. (2019) TOMM T1 ≤40 Denning (2012) WCT ≤41 Bain and Soble (2017); Bain et al. (2019) DCT ≥14 Depression; PTSD; anxiety; mild TBI ≥15 Learning disorder ≥19 Mild neurocognitive disorder ≥20 Schizophrenia; TBI with positive imaging ≥21 Mild dementia Boone et al. (2002) RDS ≤6 Schroder et al. (2012) Note: PVT = Performance Validity Test; WMT = Word Memory Test; TOMM = Test of Memory Malingering; WCT = Advanced Clinical Solutions Word Choice Test; DCT = Dot Counting Testing; RDS = Wechsler Adult Intelligence Scale (WAIS-IV) Reliable Digit Span. Open in new tab Data Analyses Descriptive statistics were used to preliminarily determine the relative statistical differences between validity groups for completion times and error frequency for the DCT cards individually. Analyses of variance (ANOVAs) were utilized to compare difference in response times. Given multiple ANOVAs, a false discovery rate (FDR) procedure of .05 was used to control for the possibility of false positive results (Benjamini & Hochberg, 1995; Glickman, Rao, & Schultz, 2014). Chi-square analyses were then used to examine differences in errors between validity groups. Given that the DCT utilizes both completion time and errors to calculate a unitary E-score, 6- and 4-card combinations with largest effect size for completion time as well as largest significant chi-square analyses were further examined. Next, receiver operator characteristic (ROC) curve analyses were conducted to assess the diagnostic accuracy and sensitivity/specificity of the two “quick” DCT combinations for predicting validity status. Another ROC curve with the full DCT was also conducted to allow for a comparison between the two “quick” DCT versions and the traditional administration method. Given the recently published non-rounded E-score (McCaul et al., 2018) for the full DCT, a follow-up ROC curve analyses with five patients with amnestic mild neurocognitive disorder and seven with major neurocognitive disorder were excluded to more closely mirror the validation. A binary logistic regression with the two “quick” DCT card combinations as well as a second PVT (i.e., TOMM T1) was conducted to determine the combined model with these two abbreviated measures. Finally, descriptive statistics were used to identify failure rates for valid-unimpaired, valid-impaired, and invalid groups for the two “quick” DCT combinations and the full DCT as stand-alone measures and combined with TOMM T1. Results DCT completion time means were statistically significant for all ungrouped and grouped cards such that the invalid group had substantially longer completion times across all cards with medium to large effect sizes (see Table 2). Moreover, ungrouped cards 1, 3, and 5 and grouped cards 8, 11, and 12 had the most robust effect sizes. Error frequency analyses showed that ungrouped cards 3 and 4 and grouped card 11 evidenced the largest, most significant discrepancies (i.e., <.001) between validity groups, such that invalid examinees made substantially more errors on these three cards. Therefore, two 3-card ungrouped combinations (1, 3, and 5) and (3, 4, 5) and two 3-card grouped combinations (8, 9, and 11) and (8, 11, and 12) along with one 2-card ungrouped combination (3 and 4) and grouped combination (8 and 11) were selected for further analyses. Table 2 Dot Counting Test completion time and error frequency between validity groups Completion time Error analysis Valid n = 80 M (SD) Invalid n = 27 M (SD) F np2 Valid n = 80 N (%) Invalid n = 27 N (%) X2 Ungrouped  Card 1 3.58 (1.44) 6.37 (4.28) 25.89*** 0.20 11 (14) 8 (30) 3.49  Card 2 7.73 (3.0) 9.56 (3.76) 6.54* 0.06 21 (26) 13 (48) 4.47*  Card 3 5.07 (1.57) 9.59 (8.02) 23.16*** 0.18 5 (6) 11 (41) 18.88***  Card 4 10.4 (4.63) 16.07 (11.10) 13.94*** 0.12 24 (30) 20 (74) 16.20***  Card 5 12.19 (4.45) 19.22 (11.82) 20.19*** 0.16 28 (35) 16 (59) 4.91*  Card 6 2.41 (1.14) 3.41 (1.85) 10.93** 0.09 1 (1) 2 (7) 2.81 Grouped  Card 7 2.4 (1.52) 5.15 (3.31) 34.16*** 0.25 0 3 (11) 9.15**  Card 8 2.19 (2.25) 7 (5.34) 43.06*** 0.29 0 1 (4) 2.99  Card 9 3.41 (2.40) 7.33 (4.95) 29.80*** 0.22 3 (4) 6 (22) 8.94**  Card 10 4.91 (4.47) 8.63 (3.98) 14.73*** 0.12 5 (6) 2 (7) .04  Card 11 4.26 (2.53) 9.48 (5.32) 46.55*** 0.31 1 (1) 7 (26) 17.77***  Card 12 1.16 (.43) 2.33 (1.44) 42.18*** 0.29 0 (0) 1 (4) 2.99 Completion time Error analysis Valid n = 80 M (SD) Invalid n = 27 M (SD) F np2 Valid n = 80 N (%) Invalid n = 27 N (%) X2 Ungrouped  Card 1 3.58 (1.44) 6.37 (4.28) 25.89*** 0.20 11 (14) 8 (30) 3.49  Card 2 7.73 (3.0) 9.56 (3.76) 6.54* 0.06 21 (26) 13 (48) 4.47*  Card 3 5.07 (1.57) 9.59 (8.02) 23.16*** 0.18 5 (6) 11 (41) 18.88***  Card 4 10.4 (4.63) 16.07 (11.10) 13.94*** 0.12 24 (30) 20 (74) 16.20***  Card 5 12.19 (4.45) 19.22 (11.82) 20.19*** 0.16 28 (35) 16 (59) 4.91*  Card 6 2.41 (1.14) 3.41 (1.85) 10.93** 0.09 1 (1) 2 (7) 2.81 Grouped  Card 7 2.4 (1.52) 5.15 (3.31) 34.16*** 0.25 0 3 (11) 9.15**  Card 8 2.19 (2.25) 7 (5.34) 43.06*** 0.29 0 1 (4) 2.99  Card 9 3.41 (2.40) 7.33 (4.95) 29.80*** 0.22 3 (4) 6 (22) 8.94**  Card 10 4.91 (4.47) 8.63 (3.98) 14.73*** 0.12 5 (6) 2 (7) .04  Card 11 4.26 (2.53) 9.48 (5.32) 46.55*** 0.31 1 (1) 7 (26) 17.77***  Card 12 1.16 (.43) 2.33 (1.44) 42.18*** 0.29 0 (0) 1 (4) 2.99 Note: *p < .05, **p < .005, ***p < .001. All p-values indicate false discovery rate corrected p-values. Bold indicates cards with largest combined difference in means. Open in new tab Table 2 Dot Counting Test completion time and error frequency between validity groups Completion time Error analysis Valid n = 80 M (SD) Invalid n = 27 M (SD) F np2 Valid n = 80 N (%) Invalid n = 27 N (%) X2 Ungrouped  Card 1 3.58 (1.44) 6.37 (4.28) 25.89*** 0.20 11 (14) 8 (30) 3.49  Card 2 7.73 (3.0) 9.56 (3.76) 6.54* 0.06 21 (26) 13 (48) 4.47*  Card 3 5.07 (1.57) 9.59 (8.02) 23.16*** 0.18 5 (6) 11 (41) 18.88***  Card 4 10.4 (4.63) 16.07 (11.10) 13.94*** 0.12 24 (30) 20 (74) 16.20***  Card 5 12.19 (4.45) 19.22 (11.82) 20.19*** 0.16 28 (35) 16 (59) 4.91*  Card 6 2.41 (1.14) 3.41 (1.85) 10.93** 0.09 1 (1) 2 (7) 2.81 Grouped  Card 7 2.4 (1.52) 5.15 (3.31) 34.16*** 0.25 0 3 (11) 9.15**  Card 8 2.19 (2.25) 7 (5.34) 43.06*** 0.29 0 1 (4) 2.99  Card 9 3.41 (2.40) 7.33 (4.95) 29.80*** 0.22 3 (4) 6 (22) 8.94**  Card 10 4.91 (4.47) 8.63 (3.98) 14.73*** 0.12 5 (6) 2 (7) .04  Card 11 4.26 (2.53) 9.48 (5.32) 46.55*** 0.31 1 (1) 7 (26) 17.77***  Card 12 1.16 (.43) 2.33 (1.44) 42.18*** 0.29 0 (0) 1 (4) 2.99 Completion time Error analysis Valid n = 80 M (SD) Invalid n = 27 M (SD) F np2 Valid n = 80 N (%) Invalid n = 27 N (%) X2 Ungrouped  Card 1 3.58 (1.44) 6.37 (4.28) 25.89*** 0.20 11 (14) 8 (30) 3.49  Card 2 7.73 (3.0) 9.56 (3.76) 6.54* 0.06 21 (26) 13 (48) 4.47*  Card 3 5.07 (1.57) 9.59 (8.02) 23.16*** 0.18 5 (6) 11 (41) 18.88***  Card 4 10.4 (4.63) 16.07 (11.10) 13.94*** 0.12 24 (30) 20 (74) 16.20***  Card 5 12.19 (4.45) 19.22 (11.82) 20.19*** 0.16 28 (35) 16 (59) 4.91*  Card 6 2.41 (1.14) 3.41 (1.85) 10.93** 0.09 1 (1) 2 (7) 2.81 Grouped  Card 7 2.4 (1.52) 5.15 (3.31) 34.16*** 0.25 0 3 (11) 9.15**  Card 8 2.19 (2.25) 7 (5.34) 43.06*** 0.29 0 1 (4) 2.99  Card 9 3.41 (2.40) 7.33 (4.95) 29.80*** 0.22 3 (4) 6 (22) 8.94**  Card 10 4.91 (4.47) 8.63 (3.98) 14.73*** 0.12 5 (6) 2 (7) .04  Card 11 4.26 (2.53) 9.48 (5.32) 46.55*** 0.31 1 (1) 7 (26) 17.77***  Card 12 1.16 (.43) 2.33 (1.44) 42.18*** 0.29 0 (0) 1 (4) 2.99 Note: *p < .05, **p < .005, ***p < .001. All p-values indicate false discovery rate corrected p-values. Bold indicates cards with largest combined difference in means. Open in new tab All ROC curve analyses were significant (p < .001), with AUC’s ranging from .868 to .897. The ungrouped cards 1, 3 and 5 paired with grouped cards 8, 11, and 12 yielded the highest (.897) AUC, followed by cards 1, 3, and 5 paired with 8, 9, and 11 (.894). Ungrouped cards 3, 4 and 5 paired with 8, 9, and 11 yielded an AUC of .877, and 3, 4, and 5 paired with 8, 11, and 12 produced an AUC of .868. Interestingly, the 4-card combination of 3, 4 paired with 8 and 11 produced an AUC of .887, falling in between the four 6-card combinations. See Table 3 for cut-offs and accompanying sensitivity/specific values for the best performing DCT “quick” combinations (6-card = 1-3-5 and 8-11-12, 4-card = 3-4 and 8-11). An E-score of ≥14.5 for the 6-card “quick” combination yielded sensitivity of 55.6 and specificity of 90.0. An E-score of ≥16.75 for the 4-card combination yielded sensitivity of 63.0 and specificity of 93.7. The full DCT produced an AUC of .896 (p < .001), with a recently recommended E-score cut-off of ≥13.8 corresponding to a sensitivity of 82% and specificity of 76% (McCaul et al., 2018). A more conservative E-score cut-off of ≥16.00 yielded sensitivity of 70% and specificity of 90%. After removal of the 12 participants with amnestic mild neurocognitive disorder/major neurocognitive disorder the AUC was slightly increased to .899 (p < .001), with ≥13.8 corresponding to a sensitivity of 82% and specificity of 77%. Table 3 Sensitivity and specificity values associated with a “quick” 6- and 4-card DCT 6-card “quick” DCT 4-card “quick” DCT E-score Sensitivity Specificity E-score Sensitivity Specificity ≥12.17 88.9 77.5 ≥11.25 88.9 63.7 ≥12.33 88.9 77.5 ≥12 88.9 66.2 ≥12.50 88.9 81.2 ≥12.75 88.9 70.0 ≥12.67 81.5 83.8 ≥13.25 81.5 75.0 ≥12.83 77.8 83.8 ≥13.75 81.5 78.7 ≥13.17 74.1 86.3 ≥14.25 74.1 81.2 ≥13.50 70.4 86.3 ≥14.75 74.1 82.5 ≥13.67 70.4 87.5 ≥15.25 74.1 85.0 ≥13.83 59.3 88.7 ≥15.75 70.4 87.5 ≥14.17 55.6 88.7 ≥16.25 66.7 88.7 ≥14.50 55.6 90.0 ≥16.75 63.0 93.7 ≥14.83 51.9 91.2 ≥17.25 59.3 93.7 ≥15.17 51.9 92.5 ≥17.75 55.6 93.7 ≥15.33 51.9 92.5 ≥18.25 55.6 95.0 ≥16.00 51.9 95.0 ≥19.25 51.9 95.0 ≥16.83 51.9 96.3 ≥20.25 51.9 96.3 ≥17.33 48.1 96.3 ≥21 44.4 97.5 ≥17.83 44.4 96.3 ≥21.75 44.4 98.8 ≥18.17 40.7 97.5 ≥22.5 40.7 98.8 ≥19.17 37.0 98.8 ≥23.5 37.0 98.8 ≥20.33 37.0 100 ≥24.5 33.3 98.8 ≥21.17 33.3 100 ≥26 29.6 98.8 ≥23.17 29.6 100 ≥27.75 25.9 98.8 6-card “quick” DCT 4-card “quick” DCT E-score Sensitivity Specificity E-score Sensitivity Specificity ≥12.17 88.9 77.5 ≥11.25 88.9 63.7 ≥12.33 88.9 77.5 ≥12 88.9 66.2 ≥12.50 88.9 81.2 ≥12.75 88.9 70.0 ≥12.67 81.5 83.8 ≥13.25 81.5 75.0 ≥12.83 77.8 83.8 ≥13.75 81.5 78.7 ≥13.17 74.1 86.3 ≥14.25 74.1 81.2 ≥13.50 70.4 86.3 ≥14.75 74.1 82.5 ≥13.67 70.4 87.5 ≥15.25 74.1 85.0 ≥13.83 59.3 88.7 ≥15.75 70.4 87.5 ≥14.17 55.6 88.7 ≥16.25 66.7 88.7 ≥14.50 55.6 90.0 ≥16.75 63.0 93.7 ≥14.83 51.9 91.2 ≥17.25 59.3 93.7 ≥15.17 51.9 92.5 ≥17.75 55.6 93.7 ≥15.33 51.9 92.5 ≥18.25 55.6 95.0 ≥16.00 51.9 95.0 ≥19.25 51.9 95.0 ≥16.83 51.9 96.3 ≥20.25 51.9 96.3 ≥17.33 48.1 96.3 ≥21 44.4 97.5 ≥17.83 44.4 96.3 ≥21.75 44.4 98.8 ≥18.17 40.7 97.5 ≥22.5 40.7 98.8 ≥19.17 37.0 98.8 ≥23.5 37.0 98.8 ≥20.33 37.0 100 ≥24.5 33.3 98.8 ≥21.17 33.3 100 ≥26 29.6 98.8 ≥23.17 29.6 100 ≥27.75 25.9 98.8 Note: 6-card “quick” DCT = Cards 1-3-5 and 8-11-12; 4-card “quick” DCT = Cards 3–4 and 8–11 Open in new tab Table 3 Sensitivity and specificity values associated with a “quick” 6- and 4-card DCT 6-card “quick” DCT 4-card “quick” DCT E-score Sensitivity Specificity E-score Sensitivity Specificity ≥12.17 88.9 77.5 ≥11.25 88.9 63.7 ≥12.33 88.9 77.5 ≥12 88.9 66.2 ≥12.50 88.9 81.2 ≥12.75 88.9 70.0 ≥12.67 81.5 83.8 ≥13.25 81.5 75.0 ≥12.83 77.8 83.8 ≥13.75 81.5 78.7 ≥13.17 74.1 86.3 ≥14.25 74.1 81.2 ≥13.50 70.4 86.3 ≥14.75 74.1 82.5 ≥13.67 70.4 87.5 ≥15.25 74.1 85.0 ≥13.83 59.3 88.7 ≥15.75 70.4 87.5 ≥14.17 55.6 88.7 ≥16.25 66.7 88.7 ≥14.50 55.6 90.0 ≥16.75 63.0 93.7 ≥14.83 51.9 91.2 ≥17.25 59.3 93.7 ≥15.17 51.9 92.5 ≥17.75 55.6 93.7 ≥15.33 51.9 92.5 ≥18.25 55.6 95.0 ≥16.00 51.9 95.0 ≥19.25 51.9 95.0 ≥16.83 51.9 96.3 ≥20.25 51.9 96.3 ≥17.33 48.1 96.3 ≥21 44.4 97.5 ≥17.83 44.4 96.3 ≥21.75 44.4 98.8 ≥18.17 40.7 97.5 ≥22.5 40.7 98.8 ≥19.17 37.0 98.8 ≥23.5 37.0 98.8 ≥20.33 37.0 100 ≥24.5 33.3 98.8 ≥21.17 33.3 100 ≥26 29.6 98.8 ≥23.17 29.6 100 ≥27.75 25.9 98.8 6-card “quick” DCT 4-card “quick” DCT E-score Sensitivity Specificity E-score Sensitivity Specificity ≥12.17 88.9 77.5 ≥11.25 88.9 63.7 ≥12.33 88.9 77.5 ≥12 88.9 66.2 ≥12.50 88.9 81.2 ≥12.75 88.9 70.0 ≥12.67 81.5 83.8 ≥13.25 81.5 75.0 ≥12.83 77.8 83.8 ≥13.75 81.5 78.7 ≥13.17 74.1 86.3 ≥14.25 74.1 81.2 ≥13.50 70.4 86.3 ≥14.75 74.1 82.5 ≥13.67 70.4 87.5 ≥15.25 74.1 85.0 ≥13.83 59.3 88.7 ≥15.75 70.4 87.5 ≥14.17 55.6 88.7 ≥16.25 66.7 88.7 ≥14.50 55.6 90.0 ≥16.75 63.0 93.7 ≥14.83 51.9 91.2 ≥17.25 59.3 93.7 ≥15.17 51.9 92.5 ≥17.75 55.6 93.7 ≥15.33 51.9 92.5 ≥18.25 55.6 95.0 ≥16.00 51.9 95.0 ≥19.25 51.9 95.0 ≥16.83 51.9 96.3 ≥20.25 51.9 96.3 ≥17.33 48.1 96.3 ≥21 44.4 97.5 ≥17.83 44.4 96.3 ≥21.75 44.4 98.8 ≥18.17 40.7 97.5 ≥22.5 40.7 98.8 ≥19.17 37.0 98.8 ≥23.5 37.0 98.8 ≥20.33 37.0 100 ≥24.5 33.3 98.8 ≥21.17 33.3 100 ≥26 29.6 98.8 ≥23.17 29.6 100 ≥27.75 25.9 98.8 Note: 6-card “quick” DCT = Cards 1-3-5 and 8-11-12; 4-card “quick” DCT = Cards 3–4 and 8–11 Open in new tab To determine the classification accuracy of the 6- and 4-card “quick” DCT combinations along with a second free-standing PVT (i.e., TOMM T1), two separate logistic regressions were performed with a stepwise entry. The dependent variable was the validity status per criterion PVTs/slick criteria and the 6-card or 4-card “quick” DCT E-score was entered in step one, followed by TOMM T1 score. As indicated in Table 4, both models were significant with overall classification accuracy at roughly 95%. Additionally, both TOMM T1 and the 4-card and 6-card quick DCT variables emerged as significant predictors in each respective model. Table 4 Logistic regression analyses for “Quick” DCT combinations and TOMM T1 predicting validity classification Variable B Standard error Wald Significance Exp (B) 6-card “quick” DCT .29 .12 5.35 .02 1.33  TOMM T1 −.37 .09 16.35 <.001 .69  Constant 10.36 3.89 7.07 .01 31,571.91 −2 Log likelihood = 28.65 Nagelkerke R2 = .85 Classification accuracy (%) = 95.3 4-card “quick” DCT .20 .08 6.40 .01 1.22  TOMM T1 −.38 .09 17.55 <.001 .68  Constant 11.51 3.68 9.77 .002 99845.61 −2 Log likelihood = 29.96 Nagelkerke R2 = .85 Classification accuracy (%) = 95.3 Variable B Standard error Wald Significance Exp (B) 6-card “quick” DCT .29 .12 5.35 .02 1.33  TOMM T1 −.37 .09 16.35 <.001 .69  Constant 10.36 3.89 7.07 .01 31,571.91 −2 Log likelihood = 28.65 Nagelkerke R2 = .85 Classification accuracy (%) = 95.3 4-card “quick” DCT .20 .08 6.40 .01 1.22  TOMM T1 −.38 .09 17.55 <.001 .68  Constant 11.51 3.68 9.77 .002 99845.61 −2 Log likelihood = 29.96 Nagelkerke R2 = .85 Classification accuracy (%) = 95.3 Note: 6-card “quick” DCT = Cards 1-3-5 and 8-11-12; 4-card “quick” DCT = Cards 3–4 and 8–11 Open in new tab Table 4 Logistic regression analyses for “Quick” DCT combinations and TOMM T1 predicting validity classification Variable B Standard error Wald Significance Exp (B) 6-card “quick” DCT .29 .12 5.35 .02 1.33  TOMM T1 −.37 .09 16.35 <.001 .69  Constant 10.36 3.89 7.07 .01 31,571.91 −2 Log likelihood = 28.65 Nagelkerke R2 = .85 Classification accuracy (%) = 95.3 4-card “quick” DCT .20 .08 6.40 .01 1.22  TOMM T1 −.38 .09 17.55 <.001 .68  Constant 11.51 3.68 9.77 .002 99845.61 −2 Log likelihood = 29.96 Nagelkerke R2 = .85 Classification accuracy (%) = 95.3 Variable B Standard error Wald Significance Exp (B) 6-card “quick” DCT .29 .12 5.35 .02 1.33  TOMM T1 −.37 .09 16.35 <.001 .69  Constant 10.36 3.89 7.07 .01 31,571.91 −2 Log likelihood = 28.65 Nagelkerke R2 = .85 Classification accuracy (%) = 95.3 4-card “quick” DCT .20 .08 6.40 .01 1.22  TOMM T1 −.38 .09 17.55 <.001 .68  Constant 11.51 3.68 9.77 .002 99845.61 −2 Log likelihood = 29.96 Nagelkerke R2 = .85 Classification accuracy (%) = 95.3 Note: 6-card “quick” DCT = Cards 1-3-5 and 8-11-12; 4-card “quick” DCT = Cards 3–4 and 8–11 Open in new tab Returning to the sample, where cognitive impairment within the valid participants were well represented, failure rates on the two “quick” DCT versions are provided for cognitively unimpaired and impaired valid participants (see Table 5). Additionally, the TOMM T1 was included as a second stand-alone PVT. Equivalent data are provided for the full DCT with and without the TOMM T1. The highest failure rate was observed in the 6-card “quick” DCT in the valid-impaired group (19%), followed by the 4-card version at 12%. For the full DCT, the highest failure rate was observed in the valid-unimpaired group (11%), with lower rates (5%) in the valid-impaired group. Inclusion of the TOMM T1 substantially reduced potential misclassification to 2% for both 6- and 4-card combinations in participants with cognitive impairment, nearly mirroring the perfect classification of the full DCT and TOMM T1. Table 5 Failure rates with newly established DCT “Quick” cut scores and previously established DCT cut scores for single and two PVT combinations (N = 107) Valid (n = 80) Valid-unimpaired (n = 38) Valid-impaired (n = 42) Invalid (n = 27) Fail n (%) n (%) n (%) n (%) 6-card “quick” DCT 8 (10) 0 (0) 8 (19) 15 (56) 4-card “quick” DCT 5 (6) 0 (0) 5 (12) 17 (63) 6-card “quick DCT and TOMM T1 1 (1) 0 (0) 1 (2) 12 (44) 4-card “quick” DCT and TOMM T1 1 (1) 0 (0) 1 (2) 14 (52) Full DCT 6 (8) 4 (11) 2 (5) 20 (74) Full DCT and TOMM T1 0 (0) 0 (0) 0 (0) 15 (56) Valid (n = 80) Valid-unimpaired (n = 38) Valid-impaired (n = 42) Invalid (n = 27) Fail n (%) n (%) n (%) n (%) 6-card “quick” DCT 8 (10) 0 (0) 8 (19) 15 (56) 4-card “quick” DCT 5 (6) 0 (0) 5 (12) 17 (63) 6-card “quick DCT and TOMM T1 1 (1) 0 (0) 1 (2) 12 (44) 4-card “quick” DCT and TOMM T1 1 (1) 0 (0) 1 (2) 14 (52) Full DCT 6 (8) 4 (11) 2 (5) 20 (74) Full DCT and TOMM T1 0 (0) 0 (0) 0 (0) 15 (56) Note: PVT = Performance Validity Test; DCT = Dot Counting Test; 6-card “quick” DCT = Cards 1-3-5 and 8-11-12 (E-score cut-off ≥14.50); 4-card “quick” DCT = Cards 3–4 and 8–11 (E-score cut-off ≥16.75); Full DCT cut score per manual norms, TOMM T1 ≤40. Open in new tab Table 5 Failure rates with newly established DCT “Quick” cut scores and previously established DCT cut scores for single and two PVT combinations (N = 107) Valid (n = 80) Valid-unimpaired (n = 38) Valid-impaired (n = 42) Invalid (n = 27) Fail n (%) n (%) n (%) n (%) 6-card “quick” DCT 8 (10) 0 (0) 8 (19) 15 (56) 4-card “quick” DCT 5 (6) 0 (0) 5 (12) 17 (63) 6-card “quick DCT and TOMM T1 1 (1) 0 (0) 1 (2) 12 (44) 4-card “quick” DCT and TOMM T1 1 (1) 0 (0) 1 (2) 14 (52) Full DCT 6 (8) 4 (11) 2 (5) 20 (74) Full DCT and TOMM T1 0 (0) 0 (0) 0 (0) 15 (56) Valid (n = 80) Valid-unimpaired (n = 38) Valid-impaired (n = 42) Invalid (n = 27) Fail n (%) n (%) n (%) n (%) 6-card “quick” DCT 8 (10) 0 (0) 8 (19) 15 (56) 4-card “quick” DCT 5 (6) 0 (0) 5 (12) 17 (63) 6-card “quick DCT and TOMM T1 1 (1) 0 (0) 1 (2) 12 (44) 4-card “quick” DCT and TOMM T1 1 (1) 0 (0) 1 (2) 14 (52) Full DCT 6 (8) 4 (11) 2 (5) 20 (74) Full DCT and TOMM T1 0 (0) 0 (0) 0 (0) 15 (56) Note: PVT = Performance Validity Test; DCT = Dot Counting Test; 6-card “quick” DCT = Cards 1-3-5 and 8-11-12 (E-score cut-off ≥14.50); 4-card “quick” DCT = Cards 3–4 and 8–11 (E-score cut-off ≥16.75); Full DCT cut score per manual norms, TOMM T1 ≤40. Open in new tab Discussion There is increased recognition that time and cognitive resources are at a premium in the context of a neuropsychological evaluation. Several studies now support at least two strategies for increasing efficiency of evaluations vis-à-vis PVT interpretation. One proposed method for increasing efficiency and diagnostic accuracy involves using a process-oriented approach to PVT interpretation, such as evaluating reaction time and completion time of PVTs (e.g., Kanser et al., 2017; Tan et al., 2007). Abbreviating free-standing PVTs has gained interest and empirical support as another potential method, though existing empirical efforts have largely focused on abbreviating memory-based PVTs (e.g., Denning, 2012). Given concern that relying on PVTs that appear to measure one neurocognitive domain may occasionally be inefficient in detecting invalid performance (Boone, 2009, 2013), we made the first attempt to validate an abbreviated version of the most commonly used non-memory-based PVT—the DCT. We identified the three cards from each of the grouped and ungrouped formations on the DCT that had the largest effect sizes for differentiating valid and noncredible examinees. These six cards were used to create a 6-card “quick” DCT, while the two cards from the grouped and ungrouped formations with the largest effect sizes constituted the 4-card “quick” DCT. Consistent with practice standards in neuropsychological performance validity testing that recommend a set specificity of ≥ .90 for PVTs (Boone, 2012), ROC curve models exhibited moderate (and significant) AUC and suggested that E-score cut-offs of ≥14.5 and ≥16.75 best identified noncredible performance for the 6- and 4-card DCT versions, respectively. ROC curve-identified sensitivity was .56 and specificity was .90 for the 6-card version, while sensitivity was .63 and specificity was .94 for the 4-card version. These values are largely consistent with sensitivity (.70) and specificity (.90) of the 12-card manual administration of the DCT, with minimal attenuation in sensitivity and utilized the recently published and encouraged non-rounding procedure (McCaul et al., 2018). Notably, AUC was slightly higher in models that excluded patients with amnestic cognitive impairment to mirror the original DCT validation. Similar to the values from ROC curve models, observed sensitivity and specificity for the 6-card DCT cut-off of ≥14.5 in the total sample was .56 and .90, respectively, while the 4-card DCT cut-off of ≥16.75 exhibited observed sensitivity of .63 and specificity of .94. Both the 4- and 6-card versions had perfect specificity (1.0) when examinees with genuine cognitive impairment were excluded from analyses. The 4-card version was relatively more robust to cognitive impairment than the 6-card version, with respective specificities of .88 and .81. Logistic regression models that included another abbreviated free-standing PVT (i.e., TOMM T1) provided evidence for incremental utility of either the 4- or 6-card versions in predicting validity status. Similar to results for either the 4- or 6-card versions alone, combining the 4-card version with the TOMM T1 had slightly better observed sensitivity and was relatively more robust to cognitive impairment than combining the 6-card version with TOMM T1. Notably, specificity of either the 4- or 6-card combination with TOMM T1 was substantially improved in the subsample of participants with genuine cognitive impairment (.98) and only marginally lower than the full DCT combined with TOMM T1 (1.0). Notably, the current data are also consistent with burgeoning support for abbreviated administrations of free-standing PVTs (Denning, 2012; Webber et al., 2018a) and the notion that the distinct underlying constructs measured by non-memory-based and memory-based PVTs may meaningfully improve classification accuracy (Boone, 2009, 2013; Webber et al., 2018b). Compared to administration of the full DCT and TOMM (which can take considerable time in select evaluations), results suggest that administering these two abbreviated free-standing PVTs has the potential to substantially reduce administration time for a neuropsychological evaluation while minimizing the likelihood that a cognitively impaired examinee’s performance is falsely identified as noncredible. From a practical standpoint, the time saved with the abbreviated DCT approach could allow for an additional 1–2 embedded PVTs to be administered (e.g., forced choice on a verbal memory task). Given that the DCT is already one of the briefer PVTs in a neuropsychologists’ repertoire, the potential time savings from an abbreviated 4- or 6-card version may be questioned. On an inpatient setting or during a cognitive screening examination where the clinician is limited to 30 min, the potential to save 1–2 min could be of paramount. Examination of the total completion time in the invalid group, 26% of patients took longer than 2 min to complete the full DCT compared to only 1% from their valid counterparts. This finding is consistent with research indicating that reduced reaction time is common in patients feigning cognitive impairment (Boone, 2009). Future studies could examine the diagnostic accuracy of the abbreviated DCT in conjunction with errors on the first 10 items on the TOMM to further provide clinicians with administration flexibility when time is of the essence. An important consideration for future studies attempting to replicate these findings or validate the utility of other abbreviated PVTs is the possibility of order effects on response times or errors for each card, particularly considering evidence of order effects for PVTs that result in a sensitivity/specificity trade-off (Zuccato, Tyson, & Erdodi, 2018). Given that each participant was administered the full DCT, response times or errors on subsequent cards may be (at least in part) dictated by exposure to stimuli on previous cards. Relatedly, there was evidence for improved diagnostic accuracy (based on AUC) for the 4-card combination (.887) from some of the 6-card combinations (.868–.877), though the best AUCs were observed for other 6-card combinations (.894–897). It is possible that this finding reflects the impact of order effects on the accuracy of the cards within this 4-card combination given that these cards are presented following other cards that can be used to develop a counting strategy. In particular, cards 8 and 11 (within the 4-card combination) represent the “test” of translating one or more previously shown grouping patterns into a more elaborate pattern. It is possible that the valid group learned the concept of grouping five dots on card 7 and translated that to a more complex pattern of five dots on card 8. The same possibility applies for applying a counting strategy on card 10 to card 11. Alternatively, the invalid group may have provided a delayed response or feigned confusion on the more complex cards (rather than using the previously learned strategy), resulting in a larger discrepancy between completion time/errors. While this may have resulted in increased accuracy for these card combinations, this possibility should be interpreted cautiously given the exploratory nature of this study. These results should be interpreted within the context of several notable limitations. First, external validity is limited by use of a relatively more highly educated and predominantly male veteran sample. This limitation is offset by evidence that the DCT distinguishes valid from invalid examinees in several nonveteran samples and is largely uninfluenced by sex and education (Back et al., 1996; Boone et al., 2002a; Boone et al., 2002b). Our linguistically, ethnically, and (in general) clinically diverse sample also suggests that these tests may function well in a variety of clinical populations, thereby enhancing external validity. Relatedly, sample size restrictions limited our ability to test the utility of these PVTs across clinical, linguistic, and racial/ethnic groups, though evidence suggests the DCT is generally robust to variability in primary language/language preference and psychiatric diagnoses (Boone et al., 2002b; Robles, López, Salazar, Boone, & Glaser, 2015). There is notable value in future research testing the impact of variability in severity and etiology of cognitive impairment on these performance validity indicators, particularly considering that 83% and 36% of cognitively impaired examinees in the current sample were diagnosed with mild neurocognitive impairment and neurocognitive impairment due to cerebrovascular etiology, respectively. Deficits in processing speed and working memory—neurocognitive abilities utilized when completing the DCT (Boone et al., 2002a)—typically accompany neurocognitive impairment secondary to cerebrovascular disease (Román, Erkinjuntti, Wallin, Pantoni, & Chui, 2002), suggesting that groups of patients with different neurocognitive disorder etiologies may perform differentially on the 4- and 6-card DCT. Third, we used a clinically referred sample that did not include a traditional “healthy control” comparison group. While simulator studies provide the benefit of comparing broadly healthy participants that have no cognitive complaints with participants feigning cognitive impairment, our mixed clinical sample allowed us to evaluate these PVTs in an ecologically valid context. Similarly, we used an approach consistent with existing evidence (Larrabee, 2008, 2014) and practice standards (Slick et al., 1999) and created a noncredible group based on failing ≥2 PVTs to ensure well-defined groups. While there is evidence that using ≥2 PVT failures has strong positive predictive value for noncredible performance while also retaining the ability to minimize false positives (Critchfield et al., 2019; Larrabee, 2008)—suggesting that the ≥2 PVT failure threshold most efficiently and accurately identifies examinees with noncredible performance—many validation studies continue to rely on a single PVT to define validity groups. Recent evidence also suggests that number of criterion PVTs used to identify validity group membership in PVT research meaningfully contributes to diagnostic error variance (Schroeder et al., 2018), emphasizing the need for caution when comparing these findings with studies that utilize <2 criterion PVTs to define validity groups. A new but growing body of literature provides evidence that abbreviated versions of free-standing PVTs be exploited to the minimize time, cost, and cognitive burden of neuropsychological evaluations. We present the first data to support the utility of an abbreviated DCT administration that uses 4- and 6-card combinations (rather than the 12-card manual administration) for classification of noncredible performance. The abbreviated DCT tests (particularly for the 4-card version) did not result in an unacceptable decrease in classification accuracy and were almost identically robust to cognitive impairment as the full DCT administration, particularly when combined with another abbreviated free-standing PVT. While identification of performance invalidity should only very rarely be solely based on these abbreviated administrations, independent administration of the 4-card DCT may best facilitate use of multiple PVTs in neuropsychological evaluations while simultaneously prioritizing diagnostic accuracy. Acknowledgements The views expressed herein are those of the authors and do not necessarily reflect the views or the official policy of the Department of Veterans Affairs or U.S. Government. Funding This work has no relevant funding to report. Conflict of Interest None declared. References Alverson , W. A. , O’Rourke , J. J. F. , & Soble , J. R. ( 2019 ). The Word Memory Test genuine memory impairment profile discriminates genuine memory impairment from invalid performance in a mixed clinical sample with cognitive impairment . The Clinical Neuropsychologist . WorldCat Babikian , T. , Boone , K. B. , Lu , P. , & Arnold , G. ( 2006 ). Sensitivity and specificity of various digit span scores in the detection of suspect effort . The Clinical Neuropsychologist , 20 ( 1 ), 145 – 159 . doi:10.1080/13854040590947362 . Google Scholar Crossref Search ADS PubMed WorldCat Back , C. , Boone , K. B. , Edwards , C. , Burgoyne , K. , & Silver , B. ( 1996 ). The performance of schizophrenics on three cognitive tests of malingering, Rey 15-Item Memory test, Rey Dot Counting, and Hiscock Forced-Choice Method . Assessment , 3 , 449 – 458 . Google Scholar Crossref Search ADS WorldCat Bailey , K. C. , Soble , J. R. , & O’Rourke , J. J. F. ( 2017 ). Clinical utility of the Rey 15-Item Test, recognition trial, and error scores for detecting noncredible neuropsychological performance in a mixed clinical sample of veterans . The Clinical Neuropsychologist , Jan;32(1):119-131. doi:10.1080/13854046.2017.1333151 . WorldCat Bailey , K. C. , Soble , J. R. , Bain , K. M. , & Fullen , C. ( 2018 ). Embedded performance validity tests in the Hopkins Verbal Learning Test—Revised and the Brief Visuospatial Memory Test—Revised: A replication study . Archives of Clinical Neuropsychology , 33 , 895 – 900 . Google Scholar Crossref Search ADS PubMed WorldCat Bain , K. M. , & Soble , J. R. ( 2017 ). Validation of the Advanced Clinical Solutions Word Choice Test (WCT) in a mixed clinical sample: Establishing classification accuracy, sensitivity/specificity, and cutoff scores . Assessment . 2019 Oct;26(7):1320–1328. doi: 10.1177/1073191117725172. WorldCat Bain , K. M. , Soble , J. R. , Webber , T. A. , Messerly , J. M. , Bailey , K. C. , Kirton , J. W. , et al. ( 2019 ). Cross-validation of three Advanced Clinical Solutions performance validity tests: Examining combinations of measures to maximize classification of invalid performance . Applied Neuropsychology. Adult . 2019 Apr 15:1–11. doi: 10.1080/23279095.2019.1585352. WorldCat Benjamini , Y. , & Hochberg , Y. ( 1995 ). Controlling the false discovery rate: A practical and powerful approach to multiple testing . Journal of the Royal Statistical Society. Series B (Methodological) , 57 , 289 – 300 . Google Scholar Crossref Search ADS WorldCat Board of Directors. ( 2007 ). American Academy of Clinical Neuropsychology (AACN) Practice Guidelines for Neuropsychological Assessment and Consultation . The Clinical Neuropsychologist , 21 ( 2 ), 209 – 231 . doi:10.1080/13825580601025932. Crossref Search ADS PubMed WorldCat Bolan , B. , Foster , J. K. , Schmand , B. , & Bolan , S. ( 2002 ). A comparison of three tests to detect feigned amnesia: The effect of feedback and the measurement of response latency . Journal of Clinical and Experimental Neuropsychology , 24 ( 2 ), 154 – 167 . Google Scholar Crossref Search ADS PubMed WorldCat Boone , K. B. ( 2009 ). The need for continuous and comprehensive sampling of effort/responsebias during neuropsychological examinations . The Clinical Neuropsychologist , 23 , 729 – 741 . doi:10.1080/13854040802427803 . Google Scholar Crossref Search ADS PubMed WorldCat Boone , K. B. ( 2012 ). Clinical practice of forensic neuropsychology: An evidence-based approach . New York City, NY : Guilford Press . Google Preview WorldCat COPAC Boone , K. B. , Lu , P. , Back , C. , King , C. , Lee , A. , Philpott , L. , et al. ( 2002a ). Sensitivity and specificity of the Rey dot counting test in patients with suspect effort and various clinical samples . Archives of Clinical Neuropsychology , 17 , 625 – 642 . doi:10.1093/arclin/17.7.625 . Google Scholar Crossref Search ADS WorldCat Boone , K. B. , Lu , P. , & Herzberg , D. ( 2002b ). The Dot Counting Test manual . Los Angeles, CA : Western Psychological Services . Google Preview WorldCat COPAC Bush , S. S. , Ruff , R. M. , Troster , A. I. , Barth , J. T. , Koffler , S. P. , Pliskin , N. H. , et al. ( 2005 ). NAN position paper—Symptom validity assessment: Practice issues and medical necessity NAN Policy & Planning Committee . Archives of Clinical Neuropsychology , 20 , 419 – 426 . doi:10.1016/j.acn.2005.02.002 . Google Scholar Crossref Search ADS PubMed WorldCat Critchfield , E. A. , Soble , J. R. , Marceaux , J. C. , Bain , K. M. , Bailey , K. C. , Webber , T. A. , et al. ( 2019 ). Cognitive impairment does not cause performance validity failure: Analyzing performance patterns among unimpaired, impaired, and noncredible participants across six tests . The Clinical Neuropsychologist , 6 , 1083 – 1101 . Google Scholar Crossref Search ADS WorldCat Dean , A. C. , Victor , T. L. , Boone , K. B. , & Arnold , G. ( 2008 ). The relationship of IQ to effort test performance . The Clinical Neuropsychologist , 22 ( 4 ), 705 – 722 . doi:10.1080/13854040701440493 . Google Scholar Crossref Search ADS PubMed WorldCat Denning , J. H. ( 2012 ). The efficiency and accuracy of the Test of Memory Malingering Trial 1, errors on the first 10 items of the test of memory malingering, and five embedded measures in predicting invalid test performance . Archives of Clinical Neuropsychology , 27 ( 4 ), 417 – 432 . doi:10.1093/arclin/acs044 . Google Scholar Crossref Search ADS PubMed WorldCat Denning , J. H. ( 2014 ). Combining the test of memory malingering trial 1 with behavioral responses improves the detection of effort test failure . Applied Neuropsychology. Adult , 21 ( 4 ), 269 – 277 . doi:10.1080/23279095.2013.811076 . Google Scholar Crossref Search ADS PubMed WorldCat Fazio , R. L. , Denning , J. H. , & Denney , R. L. ( 2017 ). TOMM Trial 1 as a performance validity indicator in a criminal forensic sample . The Clinical Neuropsychologist , 31 ( 1 ), 251 – 267 . doi:10.1080/13854046.2016.1213316 . Google Scholar Crossref Search ADS PubMed WorldCat Frederick , R. I. ( 1997 ). Validity indicator profile manual . Minnetonka, MN : NCS Assessments . Google Preview WorldCat COPAC Frederick , R. I. ( 2003 ). A review of Rey’s strategies for detecting malingeredneuropsychological impairment . Journal of Forensic Neuropsychology , 2 ( 3–4 ), 1 – 25 . doi:10.1300/J151v02n03_01 . Google Scholar Crossref Search ADS WorldCat Frederick , R. I. , Sarafaty , S. D. , Johnston , J. D. , & Powel , J. ( 1994 ). Validation of a detector response bias on a forced choice test of non-verbal ability . Neuropsychology , 8 , 118 – 125 . doi:10.1037/0894-4105.8.1.118 . Google Scholar Crossref Search ADS WorldCat Gabel , N. M. , Waldron-Perrine , B. , Spencer , R. J. , Pangilinan , P. H. , Hale , A. C. , & Bieliauskas , L. A. ( 2019 ). Suspiciously slow: Timed digit span as an embedded performance validity measure in a sample of veterans with mTBI . Brain Injury , 33 ( 3 ), 377 – 382 . doi:10.1080/02699052.2018.1553311 . Google Scholar Crossref Search ADS PubMed WorldCat Glickman , M. E. , Rao , S. R. , & Schultz , M. R. ( 2014 ). False discovery rate control is a recommended alternative to Bonferroni-type adjustments in health studies . Journal of Clinical Epidemiology , 67 , 850 – 857 . doi:10.1016/j.jclinepi.2014.03.012 . Google Scholar Crossref Search ADS PubMed WorldCat Green , P. ( 2003 ). Manual for the word memory test for windows . Edmonton: Green’s Publishing. WorldCat Green , P. , Montijo , J. , & Brockhaus , R. ( 2011 ). High specificity of the Word Memory Test and Medical Symptom Validity Test in groups with severe verbal memory impairment . Applied Neuropsychology , 18 ( 2 ), 86 – 94 . Google Scholar Crossref Search ADS PubMed WorldCat Greiffenstein , M. F. , Baker , W. J. , & Gola , T. ( 1994 ). Validation of malingered amnesia measures with a large clinical sample . Psychological Assessment , 6 ( 3 ), 218 – 224 . Google Scholar Crossref Search ADS WorldCat Heilbronner , R. L. , Sweet , J. J. , Morgan , J. E. , Larrabee , G. J. , Millis , S. R. , & Participants , C. ( 2009 ). American Academy of Clinical Neuropsychology consensus conference statement on the neuropsychological assessment of effort, response bias, and malingering . The Clinical Neuropsychologist , 23 , 1093 – 1129 . doi:10.1080/13854040903155063 . Google Scholar Crossref Search ADS PubMed WorldCat Kanser , R. J. , Rapport , L. J. , Bashem , J. R. , Billings , N. M. , Hanks , R. A. , Axelrod , B. N. , et al. ( 2017 ). Strategies of successful and unsuccessful simulators coached to feign traumatic brain injury . The Clinical Neuropsychologist , 31 ( 3 ), 644 – 653 . doi:10.1080/13854046.2016.1278040 . Google Scholar Crossref Search ADS PubMed WorldCat Kanser , R. J. , Rapport , L. J. , Bashem , J. R. , & Hanks , R. A. ( 2018 ). Detecting malingering in traumatic brain injury: Combining response time with performance validity test accuracy . The Clinical Neuropsychologist , 2019 Jan;33(1):90–107. doi:10.1080/13854046.2018.1440006 . WorldCat Larrabee , G. J. ( 2008 ). Aggregation across multiple indicators improves the detection of malingering: Relationship to likelihood ratios . The Clinical Neuropsychologist , 22 , 666 – 679 . Google Scholar Crossref Search ADS PubMed WorldCat Larrabee , G. J. ( 2014 ). False-positive rates associated with the use of multiple performance and symptom validity tests . Archives of Clinical Neuropsychology , 29 ( 4 ), 364 – 373 . doi:10.1093/arclin/acu01 . Google Scholar Crossref Search ADS PubMed WorldCat Lee , A. , Boone , K. B. , Lesser , I. , Wohl , M. , Wilkins , S. , & Parks , C. ( 2000 ). Performance of older depressed patients on two cognitive malingering tests: False positive rates for the Rey 15-Item Memorization and Dot Counting Tests . The Clinical Neuropsychologist , 14 , 303 – 308 . doi:10.1076/1385-4046(200008)14:3;1-P;FT303 . Google Scholar Crossref Search ADS PubMed WorldCat Lezak , M. D. ( 1995 ). Neuropsychological assessment (3rd ed.). New York : Oxford University Press . Google Preview WorldCat COPAC Martin , P. K. , Schroeder , R. W. , & Odland , A. P. ( 2015 ). Neuropsychologists’ validity testing beliefs and practices: A survey of north American professionals . The Clinical Neuropsychologist , 29 ( 6 ), 741 – 776 . doi:10.1080/13854046.2015.1087597 . Google Scholar Crossref Search ADS PubMed WorldCat Martin , R. C. , Hayes , J. S. , & Gouvier , W. D. ( 1996 ). Differential vulnerability between postconcussion self-report and objective malingering tests in identifying simulated mild head injury . Journal of Clinical and Experimental Neuropsychology , 18 , 265 – 275 . doi:10.1080/01688639608408281 . Google Scholar Crossref Search ADS PubMed WorldCat McCaul , C. , Boone , K. B. , Ermshar , A. , Cottingham , M. , Victor , T. L. , Ziegler , E. , et al. ( 2018 ). Cross-validation of the Dot Counting Test in a large sample of credible and non-credible patients referred for neuropsychological testing . The Clinical Neuropsychologist , 32 ( 6 ), 1054 – 1067 . doi:10.1080/13854046.2018.1425481 . Google Scholar Crossref Search ADS PubMed WorldCat Pearson. ( 2009 ). Advanced Clinical Solutions for WAIS-IV and WMS-IV: Clinical and interpretive manual . San Antonio, TX: Pearson. WorldCat Paul , D. , Franzen , M. D. , Cohen , S. H. , & Fremouw , W. ( 1992 ). An investigation into the reliability and validity of two tests used in the detection of dissimulation . International Journal of Clinical Neuropsychology , 14 , 1 – 9 . WorldCat Rey , A. ( 1941 ). L’examen psychologique dans les cas d’encéphalopathie traumatique [The psychological examination in cases of traumatic encephalopathy] . Archives de Psychologie , 28 , 286 – 340 . WorldCat Robles , L. , López , E. , Salazar , X. , Boone , K. B. , & Glaser , D. F. ( 2015 ). Specificity data for the b Test, Dot Counting Test, Rey-15 Item Plus Recognition, and Rey Word Recognition Test in monolingual Spanish-speakers . Journal of Clinical and Experimental Neuropsychology , 37 ( 6 ), 614 – 621 . Google Scholar Crossref Search ADS PubMed WorldCat Román , G. C. , Erkinjuntti , T. , Wallin , A. , Pantoni , L. , & Chui , H. C. ( 2002 ). Subcortical ischaemic vascular dementia . The Lancet Neurology , 1 ( 7 ), 426 – 436 . Google Scholar Crossref Search ADS PubMed WorldCat Rose , F. E. , Hall , S. , & Szalda-Petree , A. D. ( 1998 ). A comparison of four tests of malingering and the effects of coaching . Archives of Clinical Neuropsychology , 13 , 349 – 363 . doi:10.1093/arclin/13.4.349 . Google Scholar Crossref Search ADS PubMed WorldCat Schroeder , R. W. , Martin , P. K. , Heinrichs , R. J. , & Baade , L. E. ( 2018 ). Research methods in performance validity testing studies: Criterion grouping approach impacts study outcomes . The Clinical Neuropsychologist, 1-12. Advance online publication. doi:10.1080/13854046.2018.1484517 . WorldCat Slick , D. J. , Sherman , E. M. S. , & Iverson , G. L. ( 1999 ). Diagnostic Criteria for Malingered Neurocognitive Dysfunction: Proposed Standards for Clinical Practice and Research . The Clinical Neuropsychologist (Neuropsychology, Development and Cognition: Section D) , 13 ( 4 ), 545 – 561 . doi:10.1076/1385-4046(199911)13:04;1-Y;FT545 . WorldCat Soble , J. R. , Santos , O. A. , Bain , K. M. , Kirton , J. W. , Bailey , K. C. , Critchfield , E. A. , et al. ( 2018 ). The dot counting test adds up: Validation and response pattern analysis in a mixed clinical veteran sample . Journal of Clinical and Experimental Neuropsychology , 40 ( 4 ), 317 – 325 . doi:10.1080/13803395.2017.1342773 . Google Scholar Crossref Search ADS PubMed WorldCat Stevens , A. , Bahlo , S. , Licha , C. , Liske , B. , & Vossler-Thies , E. ( 2016 ). Reaction time as an indicator of insufficient effort: Development and validation of an embedded performance validity parameter . Psychiatry Research , 245 , 74 – 82 . doi:10.1016/j.psychres.2016.08.022 . Google Scholar Crossref Search ADS PubMed WorldCat Tan , J. E. , Slick , D. J. , Strauss , E. , & Hultsch , D. F. ( 2002 ). How’d they do it? Malingering strategies on symptom validity tests . The Clinical Neuropsychologist , 16 ( 4 ), 495 – 505 . doi:10.1076/clin.16.4.495.13909 . Google Scholar Crossref Search ADS PubMed WorldCat Tombaugh , T. N. ( 1996 ). Test of memory malingering: TOMM . New York/Toronto: MHS. WorldCat Webber , T. A. , Bailey , K. C. , Alverson , W. A. , Critchfield , E. A. , Bain , K. M. , Messerly , J. M. , et al. ( 2018a ). Further validation of the Test of Memory Malingering (TOMM) Trial 1: Examination of false positives and convergence with other validity measures . Psychological Injury and Law , 11 , 325 – 335 . Google Scholar Crossref Search ADS WorldCat Webber , T. A. , Critchfield , E. A. , & Soble , J. R. ( 2018b ). Convergent, discriminant, and concurrent validity of nonmemory-based performance validity tests . Assessment , 2018 Oct 6:1073191118804874. doi:10.1177/1073191118804874 . [Epub ahead of print]. WorldCat Webber , T. A. , & Soble , J. R. ( 2018 ). Utility of various WAIS-IV digit span indices for identifying noncredible performance validity among cognitively impaired and unimpaired examinees . The Clinical Neuropsychologist , 32 ( 4 ), 657 – 670 . doi:10.1080/13854046.2017.1415374 . Google Scholar Crossref Search ADS PubMed WorldCat Wechsler , D. ( 2008 ). WAIS-IV: Administration and scoring manual . San Antonio, TX : Pearson . Google Preview WorldCat COPAC Young , J. C. , Roper , B. L. , & Arentsen , T. J. ( 2016 ). Validity testing and neuropsychology practice in the VA healthcare system: Results from recent practitioner survey . The Clinical Neuropsychologist , 30 ( 4 ), 497 – 514 . doi:10.1080/13854046.2016.1159730 . Google Scholar Crossref Search ADS PubMed WorldCat Zuccato , B. G. , Tyson , B. T. , & Erdodi , L. A. ( 2018 ). Early bird fails the PVT? The effects of timing artifacts on performance validity tests . Psychological Assessment , 30 ( 11 ), 1491 . Google Scholar Crossref Search ADS PubMed WorldCat © The Author(s) 2019. Published by Oxford University Press. All rights reserved. For permissions, please e-mail: journals.permission@oup.com. This work is written by US Government employees and is in the public domain in the US.
TI - When Time is of the Essence: Preliminary Findings for a Quick Administration of the Dot Counting Test
JF - Archives of Clinical Neuropsychology
DO - 10.1093/arclin/acz058
DA - 2020-05-14
UR - https://www.deepdyve.com/lp/oxford-university-press/when-time-is-of-the-essence-preliminary-findings-for-a-quick-i2fFJb2XE7
SP - 1
VL - Advance Article
IS - 
DP - DeepDyve
ER -