Embedded Performance Validity Tests in the Hopkins Verbal Learning Test—Revised and the Brief Visuospatial Memory Test—Revised: A Replication Study

Embedded Performance Validity Tests in the Hopkins Verbal Learning Test—Revised and the Brief... Abstract Objective Embedded performance validity tests (PVTs) within the Hopkins Verbal Learning Test—Revised (HVLT-R) and Brief Visuospatial Memory Test—Revised (BVMT-R) were recently identified. This study aimed to further validate/replicate these embedded PVTs. Method Eighty clinically referred veterans who underwent neuropsychological evaluation were included. Validity groups were established by passing/failing 2–3 well-validated PVTs, with 75% (n = 60) classified as valid and 25% (n = 20) noncredible. Fifty-two percent of valid participants were cognitively impaired. Results HVLT-R Recognition Discrimination (RD) of ≤5 yielded 67% sensitivity/80% specificity for identifying noncredible performance. Removal of seven valid participants with an amnestic profile who produced a false positive, improved specificity to 92%, which replicated the original findings. Replication efforts failed for BVMT-R Percent Retained; however, significant findings for RD were elucidated. Conclusion Replication efforts were positive for the HVLT-R embedded PVT, corroborating its ability to identify invalid performance in this heterogeneous clinical veteran sample with and without cognitive impairment. Performance validity assessment, HVLT-R, BVMT-R, Veterans, Psychometrics INTRODUCTION Performance validity tests (PVTs) are essential in neuropsychological assessment and research. Accordingly, standards of practice necessitate inclusion of PVTs in all evaluations to objectively verify the credibility of test performance, ensure patients’ true level of cognitive functioning is accurately measured, and establish appropriate diagnoses/treatment recommendations (Heilbronner et al., 2009; Lezak, Howieson, Bigler, & Tranel, 2012). However, performance validity is not a static construct that can be effectively captured by one test at a single time point; rather, accurate assessment requires multiple indices administered throughout evaluations (Institute of Medicine of the National Academies, 2015). Fortunately, neuropsychologists can integrate multiple sources of data including objective PVTs, behavioral observations, and analysis of test performance (i.e., patterns unbefitting patient clinical history or atypical profiles for presenting problems) to assess for invalid performance (Slick, Sherman, & Iverson, 1999). Along with well-validated, freestanding PVTs, embedded measures are commonly examined. Embedded PVTs make use of scores within traditional neuropsychological tests to identify noncredible performance. They allow for continuous validity assessment throughout the battery, which is critical as engagement can fluctuate. Furthermore, they reduce the need for additional freestanding PVT administration, testing time burden, patient fatigue, and healthcare costs. Embedded measures also facilitate a more comprehensive approach to assessing validity by providing several scores which can be examined for convergence with freestanding PVTs (Lezak et al., 2012). The Hopkins Verbal Learning Test—Revised (HVLT-R; Brandt & Benedict, 2001) and the Brief Visuospatial Memory Test—Revised (BVMT-R; Benedict, 1997) are widely used tests of verbal/visual memory in both clinical and research settings. Recently, Sawyer, Testa, and Dux (2017) examined various HVLT-R/BVMT-R scores as embedded PVTs among a clinically referred veteran sample and found an HVLT-R Recognition Discrimination (RD) of ≤5 produced adequate sensitivity (53%) and specificity (93%) for detecting invalid performance. BVMT-R Percent Retained (PR) of ≤58% had similar specificity (92%), but less sensitivity (31%). While initially promising, further replication of these embedded PVTs is essential prior to use in routine, evidence-based clinical practice. Thus, as recommended by Sawyer et al. (2017), and consistent with the need for replicability of research findings from single studies to reduce false positives (Wacholder, Chanock, Garcia-Closas, El Ghormli, & Rothman, 2004), this study aimed to cross-validate these HVLT-R/BVMT-R embedded PVTs by assessing their psychometric properties and clinical utility for identifying noncredible performance among a more demographically diverse, mixed clinical veteran population in a different geographic region of the country, and using a more stringent criterion for establishing validity groups. MATERIALS AND METHODS Participants This Institutional Review Board (IRB)-approved, cross-sectional study utilized data collected from 2015 to 2017 as part of a larger, ongoing study of clinically referred veterans who received neuropsychological assessment at a VA medical center and gave informed consent for data inclusion. Eighty participants who completed both criterion PVTs (i.e., Test of Memory Malingering [TOMM; Tombaugh, 1996] and Word Memory Test [WMT; Green, 2003]) as well as one or both of the variables of interest (i.e., HVLT-R/BVMT-R) were identified. Using the criteria described below, and supported by Larrabee’s (2008) recommendation to use failures on multiple well-validated PVTs to establish invalidity, 56 participants were initially classified as valid (i.e., passed both criterion PVTs) and 15 as invalid (i.e., failed both criterion PVTs). A third freestanding PVT (i.e., Dot Counting Test [DCT]; Boone, Lu, & Herzberg, 2002) was examined for the nine participants with inconsistent WMT/TOMM scores. Four passed the DCT and five failed, yielding a final sample of 60 valid (75%) and 20 noncredible (25%). The sample was largely male (86%; n = 69), and had good diversity in age (M = 56.2 years; SD = 15.3; range = 24–84 years), education (M = 13.83 years; SD = 2.21; range = 8–19 years), race/ethnicity (50% Caucasian [n = 40], 31% Hispanic [n = 25], 15% African-American [n = 12], 4% Other [n = 3]), and linguistic preference (75% monolingual English [n = 60]; 25% English/Spanish bilingual [n = 20]). Validity groups did not differ by age and education, Wilks’s Λ = .94, F(2, 77) = 2.26, p = .11, np2 = .06; race, X2 (3, N = 80) = 1.21, p = .75; or language, X2 (1, N = 80) = .00, p = 1.0. Among valid veterans, 52% (n = 31) met Diagnostic and Statistical Manual of Mental Disorders—Fifth Edition (DSM-5; APA, 2013) criteria for a mild (n = 25) or major (n = 6) neurocognitive disorder due to vascular etiology (n = 14; 45%), amnestic mild cognitive impairment (MCI)/Alzheimer disease (n = 6; 19%), severe traumatic brain injury ([TBI]; n = 2; 7%), and other/multiple (n = 9; 29%) etiologies. Diagnoses for the 29 unimpaired were no diagnosis (n = 9; 31%), PTSD (n = 4; 14%), depression (n = 5; 17%), anxiety (n = 4; 14%), sleep (n = 2; 7%), or other psychiatric (i.e., bipolar; psychotic; substance use; personality; somatic symptom; n = 5; 17%) disorder. Noncredible participants had a primary psychiatric disorder (n = 7; 35%), mild TBI with psychiatric comorbidity (n = 8; 40%), or were suspected malingering (n = 5; 25%) due to below chance performance on criterion PVTs, discrepancies between test data and observed behavior, atypical profiles, and evidence of external incentive (Slick et al., 1999). PTSD was the most common psychiatric diagnosis in isolation (n = 6), followed by depression (n = 1), anxiety (n = 1), and somatic symptom disorders (n = 1). Six noncredible patients had PTSD and another psychiatric disorder (i.e., depression, substance use, personality disorder). Measures Hopkins Verbal Learning Test—Revised (HVLT-R; Brandt & Benedict, 2001) and Brief Visuospatial Memory Test—Revised (BVMT-R; Benedict, 1997). The HVLT-R is a 12-word, verbal memory task comprised of three learning trials, a delayed recall trial, and a recognition trial containing the 12 list words intermixed with 12 foils. The BVMT-R is a visual memory task consisting of three learning trials, a delayed recall trial, a recognition trial, and an optional copy trial. The HVLT-R/BVMT-R also contain a PR score calculated by dividing the raw delayed free recall score by the higher raw score on either learning trial 2 or 3. Both measures include a RD score which is calculated by subtracting the number of false positives from the number of hits on respective recognition trials. Across both measures, age is the largest contributing factor, accounting for 19% of variance in the total recall of the HVLT-R (Brandt & Benedict, 2001), and 11% of BVMT-R variance across trials except for recognition (Benedict, Schretlen, Groninger, Dobraski, & Shpritz, 1996). All participants were administered HVLT-R/BVMT-R Form 1. Freestanding Criterion PVTs. The Test of Memory Malingering (TOMM; Tombaugh, 1996) and the Word Memory Test (WMT; Green, 2003) served as the two well-validated criterion measures for establishing validity groups. The reader is referred to the aforementioned publications for a more thorough description. In brief, TOMM Trial 1 has robust sensitivity/specificity and classification accuracy among outpatient veterans (Denning, 2012; Hilsabeck, Gordon, Hietpas-Wilson, & Zartman, 2011). A score of ≥41 was used as the passing criterion. The WMT yields three primary effort indices (i.e., Immediate Recognition, Delayed Recognition, and Consistency), two “easy” memory indices (i.e., Multiple Choice and Paired Associates), and one “difficult” memory index (i.e., Free Recall). A raw percentage correct of ≤82.5% on any of the three primary effort indices indicates task failure. In cases of significant memory problems, the genuine memory impairment profile (GMIP) algorithm (i.e., ≥30-point discrepancy between the mean percentage correct for the primary effort indices and memory indices) was developed to reduce the risk of false positives from genuine memory deficits (Green, Montijo, & Brockhaus, 2011). Among the 60 valid participants, 41 produced a valid WMT (>82.5% on all three primary effort indices), and 18 were identified using the GMIP algorithm in the context of a clinical/medical history consistent with cognitive impairment. Given the sample size, indeterminate participants with inconsistent TOMM/WMT findings (i.e., one pass; one failure) were classified using a third freestanding PVT, the DCT (Boone et al., 2002), previously validated in a similarly diverse veteran sample with a total Effort-score cutoff of ≥15 yielding 70% sensitivity and 88% specificity (Soble et al., 2017). Data Analyses Correlational analyses were conducted for HVLT-R/BVMT-R scores, demographic variables, and criterion PVTs. Independent samples t-tests were performed to examine differences in HVLT-R/BVMT-R variables between validity groups. The false discovery rate (FDR) procedure was used to minimize false positives associated with multiple t-tests (Benjamini & Hochberg, 1995). For variables found to significantly differ between groups, receiver operating characteristic (ROC) curves were generated to examine sensitivity/specificity values for identifying noncredible performance. RESULTS Correlation coefficients for valid participants on HVLT-R/BVMT-R variables, demographic variables, and criterion PVTs are presented in Supplementary Table S1 available online. In brief, age was negatively correlated with all memory indices, whereas education and bilingualism were not. Scores on HVLT-R and BVMT-R PR and RD were significantly correlated with all WMT indices. Interestingly, only HVLT-R RD was significantly correlated with TOMM T1. Per Table 1, noncredible participants scored significantly lower than their valid counterparts, with medium to large effect sizes, across all HVLT-R/BVMT-R variables except for the BVMT-R PR. Thus, the BVMT-R PR finding described by Sawyer et al. (2017) did not replicate in this sample. Table 1. Performance Comparison of HVLT-R and BVMT-R Between Validity Groups Variable  Validity Group  n  Mean (SD)  t  d  HVLT-R Total Learning  Valid  56  20.00 (5.05)  2.91*  .85  Invalid  15  15.73 (5.01)  HVLT-R Delayed Recall  Valid  56  4.70 (3.79)  2.25*  .71  Invalid  15  2.33 (2.79)  HVLT-R Recognition Disc.  Valid  56  7.54 (3.30)  4.86***  1.52  Invalid  15  3.07 (2.55)  HVLT-R Retention %  Valid  56  53.27 (37.84)  2.14*  .65  Invalid  15  30.33 (32.88)  BVMT-R Total Learning  Valid  60  18.40 (7.96)  2.69*  .73  Invalid  20  13.15 (6.23)  BVMT-R Delayed Recall  Valid  60  7.22 (3.63)  2.60*  .67  Invalid  20  4.80 (3.48)  BVMT-R Recognition Disc.  Valid  60  5.03 (1.24)  2.97*  .88  Invalid  20  3.20 (2.67)  BVMT-R Retention %  Valid  60  83.38 (27.69)  .925  .26  Invalid  20  74.00 (42.47)  Variable  Validity Group  n  Mean (SD)  t  d  HVLT-R Total Learning  Valid  56  20.00 (5.05)  2.91*  .85  Invalid  15  15.73 (5.01)  HVLT-R Delayed Recall  Valid  56  4.70 (3.79)  2.25*  .71  Invalid  15  2.33 (2.79)  HVLT-R Recognition Disc.  Valid  56  7.54 (3.30)  4.86***  1.52  Invalid  15  3.07 (2.55)  HVLT-R Retention %  Valid  56  53.27 (37.84)  2.14*  .65  Invalid  15  30.33 (32.88)  BVMT-R Total Learning  Valid  60  18.40 (7.96)  2.69*  .73  Invalid  20  13.15 (6.23)  BVMT-R Delayed Recall  Valid  60  7.22 (3.63)  2.60*  .67  Invalid  20  4.80 (3.48)  BVMT-R Recognition Disc.  Valid  60  5.03 (1.24)  2.97*  .88  Invalid  20  3.20 (2.67)  BVMT-R Retention %  Valid  60  83.38 (27.69)  .925  .26  Invalid  20  74.00 (42.47)  Note: HVLT-R = Hopkins Verbal Learning Test—Revised; BVMT-R = Brief Visuospatial Memory Test—Revised; Total Learning = Sum of learning trials 1, 2, & 3; Recognition Disc. = Recognition discrimination (or true positives minus false positives); Retention % = Percentage retained (or delayed recall divided by better score on trial 2 or 3). *p < .05, ***p < .001. All p-values reflect false discovery rate (FDR)-corrected p-values. Table 1. Performance Comparison of HVLT-R and BVMT-R Between Validity Groups Variable  Validity Group  n  Mean (SD)  t  d  HVLT-R Total Learning  Valid  56  20.00 (5.05)  2.91*  .85  Invalid  15  15.73 (5.01)  HVLT-R Delayed Recall  Valid  56  4.70 (3.79)  2.25*  .71  Invalid  15  2.33 (2.79)  HVLT-R Recognition Disc.  Valid  56  7.54 (3.30)  4.86***  1.52  Invalid  15  3.07 (2.55)  HVLT-R Retention %  Valid  56  53.27 (37.84)  2.14*  .65  Invalid  15  30.33 (32.88)  BVMT-R Total Learning  Valid  60  18.40 (7.96)  2.69*  .73  Invalid  20  13.15 (6.23)  BVMT-R Delayed Recall  Valid  60  7.22 (3.63)  2.60*  .67  Invalid  20  4.80 (3.48)  BVMT-R Recognition Disc.  Valid  60  5.03 (1.24)  2.97*  .88  Invalid  20  3.20 (2.67)  BVMT-R Retention %  Valid  60  83.38 (27.69)  .925  .26  Invalid  20  74.00 (42.47)  Variable  Validity Group  n  Mean (SD)  t  d  HVLT-R Total Learning  Valid  56  20.00 (5.05)  2.91*  .85  Invalid  15  15.73 (5.01)  HVLT-R Delayed Recall  Valid  56  4.70 (3.79)  2.25*  .71  Invalid  15  2.33 (2.79)  HVLT-R Recognition Disc.  Valid  56  7.54 (3.30)  4.86***  1.52  Invalid  15  3.07 (2.55)  HVLT-R Retention %  Valid  56  53.27 (37.84)  2.14*  .65  Invalid  15  30.33 (32.88)  BVMT-R Total Learning  Valid  60  18.40 (7.96)  2.69*  .73  Invalid  20  13.15 (6.23)  BVMT-R Delayed Recall  Valid  60  7.22 (3.63)  2.60*  .67  Invalid  20  4.80 (3.48)  BVMT-R Recognition Disc.  Valid  60  5.03 (1.24)  2.97*  .88  Invalid  20  3.20 (2.67)  BVMT-R Retention %  Valid  60  83.38 (27.69)  .925  .26  Invalid  20  74.00 (42.47)  Note: HVLT-R = Hopkins Verbal Learning Test—Revised; BVMT-R = Brief Visuospatial Memory Test—Revised; Total Learning = Sum of learning trials 1, 2, & 3; Recognition Disc. = Recognition discrimination (or true positives minus false positives); Retention % = Percentage retained (or delayed recall divided by better score on trial 2 or 3). *p < .05, ***p < .001. All p-values reflect false discovery rate (FDR)-corrected p-values. ROC curve analyses yielded significant areas under the curve (AUCs) of .670–.850 for HVLT-R variables and .678–.720 for BVMT-R variables. Similar to the t-tests, the HVLT-R RD (AUC = .850; CI = .760–.940) and BVMT-R RD (AUC = .720; CI = .576–.864) scores had the highest AUCs, so these were examined further. Table 2 presents sensitivity/specificity and positive/negative predictive values for various base rates of invalid performance for the HVLT-R and BVMT-R RD scores. For the HVLT-R, a cutoff of ≤3 maximized sensitivity (47%) and specificity (89%), whereas the previously described cutoff (≤5) had adequate sensitivity (67%), but unacceptable specificity (80%). Further examination revealed seven valid participants with an amnestic disorder (3 MCI-amnestic; 3 Alzheimer disease; 1 alcohol-induced dementia) who performed very poorly (i.e., ≤5) on the HVLT-R RD, yet passed the freestanding criterion PVTs. Reanalysis after removal of these seven false positives, yielded an improved AUC (.918) and an optimal cutoff of ≤5 (sensitivity: 67%; specificity: 92%), which replicates Sawyer et al. (2017) findings with improved sensitivity. For the BVMT-R RD, ROC analysis for the total sample yielded an optimal cutoff of ≤3 (sensitivity: 40%; specificity: 95%). With the seven previously noted amnestic patients removed, AUC increased to (.756) with an optimal cutoff of ≤4 (sensitivity: 50%; specificity: 93%). Table 2. Operational Characteristics for the HVLT-R and BVMT-R Recognition Discrimination Cutoff  SN  SP  Base rate (0.40)  Base rate (0.20)  PPV  NPV  PPV  NPV  HVLT-R Recognition Discrimination Operational Characteristics   Entire Sample (N = 71)    ≤1  0.27  0.98  0.9  0.67  0.77  0.84    ≤2  0.33  0.95  0.82  0.68  0.63  0.85    ≤3  0.47  0.89  0.74  0.71  0.51  0.87    ≤4  0.53  0.86  0.72  0.73  0.49  0.88    ≤5  0.67  0.8  0.69  0.78  0.45  0.91    ≤6  0.73  0.75  0.66  0.81  0.42  0.92    ≤7  0.93  0.64  0.64  0.94  0.4  0.97    ≤8  1.00  0.59  0.62  1.00  0.38  1.00   Seven False Positives Removed (N = 64)    ≤1  0.27  1.00  1.00  0.67  1.00  0.85    ≤2  0.33  1.00  1.00  0.69  1.00  0.86    ≤3  0.47  0.96  0.89  0.73  0.74  0.88    ≤4  0.53  0.96  0.9  0.76  0.77  0.89    ≤5  0.67  0.92  0.85  0.81  0.68  0.92    ≤6  0.73  0.86  0.78  0.83  0.57  0.93    ≤7  0.93  0.73  0.7  0.94  0.46  0.98    ≤8  1.00  0.67  0.67  1.00  0.43  1.00  BVMT-R Recognition Discrimination Operational Characteristics   Entire Sample (N = 80)    ≤1  .10  1.00  1.00  0.63  1.00  0.82    ≤2  .20  1.00  1.00  0.65  1.00  0.83    ≤3  .40  0.95  0.84  0.70  0.67  0.86    ≤4  .50  0.83  0.66  0.71  0.42  0.87    ≤5  .60  0.75  0.62  0.74  0.38  0.88    ≤6  .75  0.5  0.50  0.75  0.27  0.89   Seven False Positives Removed (N = 73)    ≤1  .10  1.00  1.00  0.63  1.00  0.82    ≤2  .20  1.00  1.00  0.65  1.00  0.83    ≤3  .40  1.00  1.00  0.71  1.00  0.87    ≤4  .50  0.93  0.83  0.74  0.64  0.88    ≤5  .60  0.83  0.70  0.76  0.47  0.89    ≤6  .75  0.55  0.53  0.77  0.29  0.90  Cutoff  SN  SP  Base rate (0.40)  Base rate (0.20)  PPV  NPV  PPV  NPV  HVLT-R Recognition Discrimination Operational Characteristics   Entire Sample (N = 71)    ≤1  0.27  0.98  0.9  0.67  0.77  0.84    ≤2  0.33  0.95  0.82  0.68  0.63  0.85    ≤3  0.47  0.89  0.74  0.71  0.51  0.87    ≤4  0.53  0.86  0.72  0.73  0.49  0.88    ≤5  0.67  0.8  0.69  0.78  0.45  0.91    ≤6  0.73  0.75  0.66  0.81  0.42  0.92    ≤7  0.93  0.64  0.64  0.94  0.4  0.97    ≤8  1.00  0.59  0.62  1.00  0.38  1.00   Seven False Positives Removed (N = 64)    ≤1  0.27  1.00  1.00  0.67  1.00  0.85    ≤2  0.33  1.00  1.00  0.69  1.00  0.86    ≤3  0.47  0.96  0.89  0.73  0.74  0.88    ≤4  0.53  0.96  0.9  0.76  0.77  0.89    ≤5  0.67  0.92  0.85  0.81  0.68  0.92    ≤6  0.73  0.86  0.78  0.83  0.57  0.93    ≤7  0.93  0.73  0.7  0.94  0.46  0.98    ≤8  1.00  0.67  0.67  1.00  0.43  1.00  BVMT-R Recognition Discrimination Operational Characteristics   Entire Sample (N = 80)    ≤1  .10  1.00  1.00  0.63  1.00  0.82    ≤2  .20  1.00  1.00  0.65  1.00  0.83    ≤3  .40  0.95  0.84  0.70  0.67  0.86    ≤4  .50  0.83  0.66  0.71  0.42  0.87    ≤5  .60  0.75  0.62  0.74  0.38  0.88    ≤6  .75  0.5  0.50  0.75  0.27  0.89   Seven False Positives Removed (N = 73)    ≤1  .10  1.00  1.00  0.63  1.00  0.82    ≤2  .20  1.00  1.00  0.65  1.00  0.83    ≤3  .40  1.00  1.00  0.71  1.00  0.87    ≤4  .50  0.93  0.83  0.74  0.64  0.88    ≤5  .60  0.83  0.70  0.76  0.47  0.89    ≤6  .75  0.55  0.53  0.77  0.29  0.90  Note: Bolded numbers indicate optimal cutoff; HVLT-R = Hopkins Verbal Learning Test—Revised; BVMT-R = Brief Visuospatial Memory Test—Revised; SN = Sensitivity; SP = Specificity; PPV = Positive Predictive Value; NPV = Negative Predictive Value. Table 2. Operational Characteristics for the HVLT-R and BVMT-R Recognition Discrimination Cutoff  SN  SP  Base rate (0.40)  Base rate (0.20)  PPV  NPV  PPV  NPV  HVLT-R Recognition Discrimination Operational Characteristics   Entire Sample (N = 71)    ≤1  0.27  0.98  0.9  0.67  0.77  0.84    ≤2  0.33  0.95  0.82  0.68  0.63  0.85    ≤3  0.47  0.89  0.74  0.71  0.51  0.87    ≤4  0.53  0.86  0.72  0.73  0.49  0.88    ≤5  0.67  0.8  0.69  0.78  0.45  0.91    ≤6  0.73  0.75  0.66  0.81  0.42  0.92    ≤7  0.93  0.64  0.64  0.94  0.4  0.97    ≤8  1.00  0.59  0.62  1.00  0.38  1.00   Seven False Positives Removed (N = 64)    ≤1  0.27  1.00  1.00  0.67  1.00  0.85    ≤2  0.33  1.00  1.00  0.69  1.00  0.86    ≤3  0.47  0.96  0.89  0.73  0.74  0.88    ≤4  0.53  0.96  0.9  0.76  0.77  0.89    ≤5  0.67  0.92  0.85  0.81  0.68  0.92    ≤6  0.73  0.86  0.78  0.83  0.57  0.93    ≤7  0.93  0.73  0.7  0.94  0.46  0.98    ≤8  1.00  0.67  0.67  1.00  0.43  1.00  BVMT-R Recognition Discrimination Operational Characteristics   Entire Sample (N = 80)    ≤1  .10  1.00  1.00  0.63  1.00  0.82    ≤2  .20  1.00  1.00  0.65  1.00  0.83    ≤3  .40  0.95  0.84  0.70  0.67  0.86    ≤4  .50  0.83  0.66  0.71  0.42  0.87    ≤5  .60  0.75  0.62  0.74  0.38  0.88    ≤6  .75  0.5  0.50  0.75  0.27  0.89   Seven False Positives Removed (N = 73)    ≤1  .10  1.00  1.00  0.63  1.00  0.82    ≤2  .20  1.00  1.00  0.65  1.00  0.83    ≤3  .40  1.00  1.00  0.71  1.00  0.87    ≤4  .50  0.93  0.83  0.74  0.64  0.88    ≤5  .60  0.83  0.70  0.76  0.47  0.89    ≤6  .75  0.55  0.53  0.77  0.29  0.90  Cutoff  SN  SP  Base rate (0.40)  Base rate (0.20)  PPV  NPV  PPV  NPV  HVLT-R Recognition Discrimination Operational Characteristics   Entire Sample (N = 71)    ≤1  0.27  0.98  0.9  0.67  0.77  0.84    ≤2  0.33  0.95  0.82  0.68  0.63  0.85    ≤3  0.47  0.89  0.74  0.71  0.51  0.87    ≤4  0.53  0.86  0.72  0.73  0.49  0.88    ≤5  0.67  0.8  0.69  0.78  0.45  0.91    ≤6  0.73  0.75  0.66  0.81  0.42  0.92    ≤7  0.93  0.64  0.64  0.94  0.4  0.97    ≤8  1.00  0.59  0.62  1.00  0.38  1.00   Seven False Positives Removed (N = 64)    ≤1  0.27  1.00  1.00  0.67  1.00  0.85    ≤2  0.33  1.00  1.00  0.69  1.00  0.86    ≤3  0.47  0.96  0.89  0.73  0.74  0.88    ≤4  0.53  0.96  0.9  0.76  0.77  0.89    ≤5  0.67  0.92  0.85  0.81  0.68  0.92    ≤6  0.73  0.86  0.78  0.83  0.57  0.93    ≤7  0.93  0.73  0.7  0.94  0.46  0.98    ≤8  1.00  0.67  0.67  1.00  0.43  1.00  BVMT-R Recognition Discrimination Operational Characteristics   Entire Sample (N = 80)    ≤1  .10  1.00  1.00  0.63  1.00  0.82    ≤2  .20  1.00  1.00  0.65  1.00  0.83    ≤3  .40  0.95  0.84  0.70  0.67  0.86    ≤4  .50  0.83  0.66  0.71  0.42  0.87    ≤5  .60  0.75  0.62  0.74  0.38  0.88    ≤6  .75  0.5  0.50  0.75  0.27  0.89   Seven False Positives Removed (N = 73)    ≤1  .10  1.00  1.00  0.63  1.00  0.82    ≤2  .20  1.00  1.00  0.65  1.00  0.83    ≤3  .40  1.00  1.00  0.71  1.00  0.87    ≤4  .50  0.93  0.83  0.74  0.64  0.88    ≤5  .60  0.83  0.70  0.76  0.47  0.89    ≤6  .75  0.55  0.53  0.77  0.29  0.90  Note: Bolded numbers indicate optimal cutoff; HVLT-R = Hopkins Verbal Learning Test—Revised; BVMT-R = Brief Visuospatial Memory Test—Revised; SN = Sensitivity; SP = Specificity; PPV = Positive Predictive Value; NPV = Negative Predictive Value. DISCUSSION This study aimed to replicate Sawyer et al. (2017) findings, which was the first published study to examine embedded PVTs within the commonly used HVLT-R and BVMT-R. In a clinical veteran sample with a 21% noncredible performance base rate, the HVLT-R RD score performed well using a cutoff of ≤5 (53% sensitivity/93% specificity), with less robust findings from the BVMT-R PR cutoff of ≤58% (31% sensitivity/92% specificity). In this study, replication findings were mixed. The HVLT-R RD significantly differed between validity groups with a large effect size (d = 1.52) in heterogeneous clinical groups with and without cognitive impairment. A cut-score of ≤5 yielded 67% sensitivity, but unacceptable specificity (80%), whereas ≤3 had 47% sensitivity and 89% specificity. Removal of seven amnestic participants replicated the previously reported cutoff of ≤5 with sensitivity remaining at 67% and specificity substantially increasing to 92%. This converging evidence supports the use of the HVLT-R RD for detecting noncredible performance, while also cautioning the specific cutoff selected (i.e., ≤3 versus ≤5) may need to be informed by whether an amnestic disorder versus other etiology of impairment is suspected. By contrast, BVMT-R PR results did not replicate, as these scores did not differ significantly between validity groups. One possible explanation for this nonsignificant finding is that BVMT-R PR can be heavily influenced by processing speed during the encoding trials and frank memory impairment during the recall phases of the BVMT-R (Tam & Schmitter-Edgecombe, 2013). Given that roughly half of the valid participants were cognitively impaired, it is reasonable to hypothesize that a combination of attention, processing speed, visuoperception, and memory difficulties combined to account for a larger portion of the variance than measurable “effort.” A novel finding in the current study was performance on BVMT-R RD distinguished validity groups well (cutoff ≤3; 40% sensitivity/95% specificity), with more robust classification after false positives were removed from the analysis (cutoff ≤4; 50% sensitivity/93% specificity). The current replication study bolstered Sawyer et al. (2017) initial findings through the use of a heterogeneous clinical sample with good diversity in terms of age, level of education, race/ethnicity, and language preference, and with performance validity categorized by a more stringent criterion (i.e., two well-validated, freestanding PVT failures). Further, we addressed a limitation of the original validation by including a sample with a higher percentage of patients with cognitive impairment. The finding of similar accuracy for HVLT-R RD in our sample preliminarily suggests this index may identify noncredible performance in the context of cognitive impairment, though special consideration may need to be taken in cases of patients with a suspected amnestic disorder. Notably, they may perform poorly on the HVLT-R on the whole and therefore be more likely to be incorrectly classified as invalid if a clinician were to use the ≤5 cutoff, especially in the absence of other converging PVT evidence of noncredible performance. An additional strength of this sample is that the validity groups were similarly educated, which was a potential limitation in the initial study. However, the fact the HVLT-R RD performed so similarly across samples supports Sawyer et al. (2017) position that the statistically significant difference in education between groups was not clinically meaningful. This study had some limitations, including retrospective design and use of a mixed clinical sample. Further validation should be conducted in specific, homogenous clinical samples (e.g., TBI, Alzheimer’s disease) to determine whether the HVLT-R/BVMT-R RD scores continue to demonstrate adequate classification accuracy across clinical groups and to establish optimal cutoff scores for various clinical groups, if indicated. It is noted that the rate of invalid performance in the current sample was 25%, which was similar to the 21% reported in the Sawyer et al. (2017) sample. Similar to the original study, data pertaining to potential external motivating factors for invalid performance (e.g., disability-seeking status) in the current sample generally were not collected, and it is therefore impossible to determine the source of the discrepancies between samples. We included PPV/NPV values for a range of base rates of invalid performance to address this weakness. Another limitation shared by Sawyer et al. (2017) was a small sample size, which could have contributed to inconsistent findings across BVMT-R PR and RD. Larger validation samples are required to increase clinician confidence with including an embedded PVT derived from the BVMT-R. Despite limitations, the current study provides replicative support for the HVLT-R RD score as an embedded PVT, and offers initial support for the BVMT-R RD. CONCLUSIONS Overall, this study extended previous findings which demonstrated the embedded PVT within the HVLT-R is robust within a clinically referred veteran sample. This study is significant to this literature due to the more ethnically and linguistically diverse sample, with higher rates of cognitive impairment and invalidity. Converging evidence generally supports clinical use of an HVLT-R RD cutoff of ≤5 (or ≤3 in cases of suspected amnestic disorders), in conjunction with other well-validated PVTs, to facilitate ongoing performance validity assessment throughout neuropsychological evaluations. Future research can use larger sample sizes and potentially combine other commonly used clinical measures to provide clinicians with a formulaic approach for determining performance validity with enhanced psychometric properties. SUPPLEMENTARY DATA Supplementary material are available at Archives of Clinical Neuropsychology online. FUNDING The authors have no financial interest with the subject matter discussed in the manuscript. CONFLICT OF INTEREST None declared. ACKNOWLEDGMENTS The views expressed herein are those of the authors and do not necessarily reflect the views or the official policy of the Department of Veterans Affairs or U.S. Government. REFERENCES American Psychiatric Association. ( 2013). Diagnostic and statistical manual of mental disorders—Fifth edition (DSM-5) . Washington, DC: American Psychiatric Publishing. Benedict, R. H. B. ( 1997). Brief Visuospatial Memory Test—Revised . Odessa, FL: Psychological Assessment Resources. Benedict, R. H. B., Schretlen, D., Groninger, L., Dobraski, M., & Shpritz, B. ( 1996). Revision of the Brief Visuospatial Memory Test: Studies of normal performance, reliability, and validity. Psychological Assessment , 8, 145– 153. Google Scholar CrossRef Search ADS   Benjamini, Y., & Hochberg, Y. ( 1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society Series B (Methodological) , 57, 289– 300. Boone, K. B., Lu, P., & Herzberg, D. ( 2002). The Dot Counting Test manual . Los Angeles, CA: Western Psychological Services. Brandt, J., & Benedict, R. H. B. ( 2001). Hopkins Verbal Learning Test—Revised . Odessa, FL: Psychological Assessment Resources. Denning, J. H. ( 2012). The efficiency and accuracy of the Test of Memory Malingering trial 1, errors on the first 10 items of the Test of Memory Malingering, and five embedded measures in predicting invalid test performance. Archives of Clinical Neuropsychology , 27, 417– 432. Google Scholar CrossRef Search ADS PubMed  Green, P. ( 2003). Green’s Word Memory Test for Windows: User’s manual . Edmonton, Canada: Green’s Publishing. Green, P., Montijo, J., & Brockhaus, R. ( 2011). High specificity of the Word Memory Test and Medical Symptom Validity Test in groups with severe verbal memory impairment. Applied Neuropsychology , 18, 86– 94. Google Scholar CrossRef Search ADS PubMed  Heilbronner, R. L., Sweet, J. J., Morgan, J. E., Larrabee, G. J., & Millis, S. R. ( 2009). American Academy of Clinical Neuropsychology consensus conference statement on the neuropsychological assessment of effort, response bias, and malingering. The Clinical Neuropsychologist , 23, 1093– 1129. Google Scholar CrossRef Search ADS PubMed  Hilsabeck, R. C., Gordon, S. N., Hietpas-Wilson, T., & Zartman, A. ( 2011). Use of trial 1 of the Test of Memory Malingering (TOMM) as a screening measure of effort: Suggested discontinuation rules. The Clinical Neuropsychologist , 25, 1228– 1238. Google Scholar CrossRef Search ADS PubMed  Institute of Medicine of the National Academies. ( 2015). Psychological testing in the service of disability determination . Washington, DC: National Academies Press (US). (Chapter 5). Larrabee, G. J. ( 2008). Aggregation across multiple indicators improves the detection of malingering: Relationship to likelihood ratios. The Clinical Neuropsychologist , 22, 666– 679. Google Scholar CrossRef Search ADS PubMed  Lezak, M. D., Howieson, D. B., Bigler, E. D., & Tranel, D. ( 2012). Neuropsychological assessment  ( 5th ed.). Oxford: Oxford University Press. Sawyer, R. J., Testa, S. M., & Dux, M. ( 2017). Embedded performance validity tests within the Hopkins Verbal Learning Test—Revised and the Brief Visuospatial Memory Test—Revised. The Clinical Neuropsychologist , 31, 207– 218. Google Scholar CrossRef Search ADS PubMed  Slick, D. J., Sherman, E. M., & Iverson, G. L. ( 1999). Diagnostic criteria for malingered neurocognitive dysfunction: Proposed standards for clinical practice and research. The Clinical Neuropsychologist , 13, 545– 561. Google Scholar CrossRef Search ADS PubMed  Soble, J. R., Santos, O. A., Bain, K. M., Kirton, J. W., Bailey, K. C., Critchfield, E. A., et al.  . ( 2017). The Dot Counting Test adds up: Validation and response pattern analysis in a mixed clinical veteran sample. Journal of Clinical and Experimental Neuropsychology , 1– 9. doi: 10.1080/13803395.2017.1342773. Tam, J. W., & Schmitter-Edgecombe, M. ( 2013). The role of processing speed in the Brief Visuospatial Memory Test—Revised. The Clinical Neuropsychologist , 27, 962– 972. Google Scholar CrossRef Search ADS PubMed  Tombaugh, T. N. ( 1996). Test of Memory Malingering (TOMM) . North Tonawanda, NY: Multi Health Systems. Wacholder, S., Chanock, S., Garcia-Closas, M., El Ghormli, L., & Rothman, N. ( 2004). Assessing the probability that a positive report is false: An approach for molecular epidemiology studies. Journal of the National Cancer Institute , 96, 434– 442. Google Scholar CrossRef Search ADS PubMed  Published by Oxford University Press 2017. This work is written by (a) US Government employee(s) and is in the public domain in the US. http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Archives of Clinical Neuropsychology Oxford University Press

Embedded Performance Validity Tests in the Hopkins Verbal Learning Test—Revised and the Brief Visuospatial Memory Test—Revised: A Replication Study

Loading next page...
 
/lp/ou_press/embedded-performance-validity-tests-in-the-hopkins-verbal-learning-NH4AkYEhxy
Publisher
Oxford University Press
Copyright
Published by Oxford University Press 2017. This work is written by (a) US Government employee(s) and is in the public domain in the US.
ISSN
0887-6177
eISSN
1873-5843
D.O.I.
10.1093/arclin/acx111
Publisher site
See Article on Publisher Site

Abstract

Abstract Objective Embedded performance validity tests (PVTs) within the Hopkins Verbal Learning Test—Revised (HVLT-R) and Brief Visuospatial Memory Test—Revised (BVMT-R) were recently identified. This study aimed to further validate/replicate these embedded PVTs. Method Eighty clinically referred veterans who underwent neuropsychological evaluation were included. Validity groups were established by passing/failing 2–3 well-validated PVTs, with 75% (n = 60) classified as valid and 25% (n = 20) noncredible. Fifty-two percent of valid participants were cognitively impaired. Results HVLT-R Recognition Discrimination (RD) of ≤5 yielded 67% sensitivity/80% specificity for identifying noncredible performance. Removal of seven valid participants with an amnestic profile who produced a false positive, improved specificity to 92%, which replicated the original findings. Replication efforts failed for BVMT-R Percent Retained; however, significant findings for RD were elucidated. Conclusion Replication efforts were positive for the HVLT-R embedded PVT, corroborating its ability to identify invalid performance in this heterogeneous clinical veteran sample with and without cognitive impairment. Performance validity assessment, HVLT-R, BVMT-R, Veterans, Psychometrics INTRODUCTION Performance validity tests (PVTs) are essential in neuropsychological assessment and research. Accordingly, standards of practice necessitate inclusion of PVTs in all evaluations to objectively verify the credibility of test performance, ensure patients’ true level of cognitive functioning is accurately measured, and establish appropriate diagnoses/treatment recommendations (Heilbronner et al., 2009; Lezak, Howieson, Bigler, & Tranel, 2012). However, performance validity is not a static construct that can be effectively captured by one test at a single time point; rather, accurate assessment requires multiple indices administered throughout evaluations (Institute of Medicine of the National Academies, 2015). Fortunately, neuropsychologists can integrate multiple sources of data including objective PVTs, behavioral observations, and analysis of test performance (i.e., patterns unbefitting patient clinical history or atypical profiles for presenting problems) to assess for invalid performance (Slick, Sherman, & Iverson, 1999). Along with well-validated, freestanding PVTs, embedded measures are commonly examined. Embedded PVTs make use of scores within traditional neuropsychological tests to identify noncredible performance. They allow for continuous validity assessment throughout the battery, which is critical as engagement can fluctuate. Furthermore, they reduce the need for additional freestanding PVT administration, testing time burden, patient fatigue, and healthcare costs. Embedded measures also facilitate a more comprehensive approach to assessing validity by providing several scores which can be examined for convergence with freestanding PVTs (Lezak et al., 2012). The Hopkins Verbal Learning Test—Revised (HVLT-R; Brandt & Benedict, 2001) and the Brief Visuospatial Memory Test—Revised (BVMT-R; Benedict, 1997) are widely used tests of verbal/visual memory in both clinical and research settings. Recently, Sawyer, Testa, and Dux (2017) examined various HVLT-R/BVMT-R scores as embedded PVTs among a clinically referred veteran sample and found an HVLT-R Recognition Discrimination (RD) of ≤5 produced adequate sensitivity (53%) and specificity (93%) for detecting invalid performance. BVMT-R Percent Retained (PR) of ≤58% had similar specificity (92%), but less sensitivity (31%). While initially promising, further replication of these embedded PVTs is essential prior to use in routine, evidence-based clinical practice. Thus, as recommended by Sawyer et al. (2017), and consistent with the need for replicability of research findings from single studies to reduce false positives (Wacholder, Chanock, Garcia-Closas, El Ghormli, & Rothman, 2004), this study aimed to cross-validate these HVLT-R/BVMT-R embedded PVTs by assessing their psychometric properties and clinical utility for identifying noncredible performance among a more demographically diverse, mixed clinical veteran population in a different geographic region of the country, and using a more stringent criterion for establishing validity groups. MATERIALS AND METHODS Participants This Institutional Review Board (IRB)-approved, cross-sectional study utilized data collected from 2015 to 2017 as part of a larger, ongoing study of clinically referred veterans who received neuropsychological assessment at a VA medical center and gave informed consent for data inclusion. Eighty participants who completed both criterion PVTs (i.e., Test of Memory Malingering [TOMM; Tombaugh, 1996] and Word Memory Test [WMT; Green, 2003]) as well as one or both of the variables of interest (i.e., HVLT-R/BVMT-R) were identified. Using the criteria described below, and supported by Larrabee’s (2008) recommendation to use failures on multiple well-validated PVTs to establish invalidity, 56 participants were initially classified as valid (i.e., passed both criterion PVTs) and 15 as invalid (i.e., failed both criterion PVTs). A third freestanding PVT (i.e., Dot Counting Test [DCT]; Boone, Lu, & Herzberg, 2002) was examined for the nine participants with inconsistent WMT/TOMM scores. Four passed the DCT and five failed, yielding a final sample of 60 valid (75%) and 20 noncredible (25%). The sample was largely male (86%; n = 69), and had good diversity in age (M = 56.2 years; SD = 15.3; range = 24–84 years), education (M = 13.83 years; SD = 2.21; range = 8–19 years), race/ethnicity (50% Caucasian [n = 40], 31% Hispanic [n = 25], 15% African-American [n = 12], 4% Other [n = 3]), and linguistic preference (75% monolingual English [n = 60]; 25% English/Spanish bilingual [n = 20]). Validity groups did not differ by age and education, Wilks’s Λ = .94, F(2, 77) = 2.26, p = .11, np2 = .06; race, X2 (3, N = 80) = 1.21, p = .75; or language, X2 (1, N = 80) = .00, p = 1.0. Among valid veterans, 52% (n = 31) met Diagnostic and Statistical Manual of Mental Disorders—Fifth Edition (DSM-5; APA, 2013) criteria for a mild (n = 25) or major (n = 6) neurocognitive disorder due to vascular etiology (n = 14; 45%), amnestic mild cognitive impairment (MCI)/Alzheimer disease (n = 6; 19%), severe traumatic brain injury ([TBI]; n = 2; 7%), and other/multiple (n = 9; 29%) etiologies. Diagnoses for the 29 unimpaired were no diagnosis (n = 9; 31%), PTSD (n = 4; 14%), depression (n = 5; 17%), anxiety (n = 4; 14%), sleep (n = 2; 7%), or other psychiatric (i.e., bipolar; psychotic; substance use; personality; somatic symptom; n = 5; 17%) disorder. Noncredible participants had a primary psychiatric disorder (n = 7; 35%), mild TBI with psychiatric comorbidity (n = 8; 40%), or were suspected malingering (n = 5; 25%) due to below chance performance on criterion PVTs, discrepancies between test data and observed behavior, atypical profiles, and evidence of external incentive (Slick et al., 1999). PTSD was the most common psychiatric diagnosis in isolation (n = 6), followed by depression (n = 1), anxiety (n = 1), and somatic symptom disorders (n = 1). Six noncredible patients had PTSD and another psychiatric disorder (i.e., depression, substance use, personality disorder). Measures Hopkins Verbal Learning Test—Revised (HVLT-R; Brandt & Benedict, 2001) and Brief Visuospatial Memory Test—Revised (BVMT-R; Benedict, 1997). The HVLT-R is a 12-word, verbal memory task comprised of three learning trials, a delayed recall trial, and a recognition trial containing the 12 list words intermixed with 12 foils. The BVMT-R is a visual memory task consisting of three learning trials, a delayed recall trial, a recognition trial, and an optional copy trial. The HVLT-R/BVMT-R also contain a PR score calculated by dividing the raw delayed free recall score by the higher raw score on either learning trial 2 or 3. Both measures include a RD score which is calculated by subtracting the number of false positives from the number of hits on respective recognition trials. Across both measures, age is the largest contributing factor, accounting for 19% of variance in the total recall of the HVLT-R (Brandt & Benedict, 2001), and 11% of BVMT-R variance across trials except for recognition (Benedict, Schretlen, Groninger, Dobraski, & Shpritz, 1996). All participants were administered HVLT-R/BVMT-R Form 1. Freestanding Criterion PVTs. The Test of Memory Malingering (TOMM; Tombaugh, 1996) and the Word Memory Test (WMT; Green, 2003) served as the two well-validated criterion measures for establishing validity groups. The reader is referred to the aforementioned publications for a more thorough description. In brief, TOMM Trial 1 has robust sensitivity/specificity and classification accuracy among outpatient veterans (Denning, 2012; Hilsabeck, Gordon, Hietpas-Wilson, & Zartman, 2011). A score of ≥41 was used as the passing criterion. The WMT yields three primary effort indices (i.e., Immediate Recognition, Delayed Recognition, and Consistency), two “easy” memory indices (i.e., Multiple Choice and Paired Associates), and one “difficult” memory index (i.e., Free Recall). A raw percentage correct of ≤82.5% on any of the three primary effort indices indicates task failure. In cases of significant memory problems, the genuine memory impairment profile (GMIP) algorithm (i.e., ≥30-point discrepancy between the mean percentage correct for the primary effort indices and memory indices) was developed to reduce the risk of false positives from genuine memory deficits (Green, Montijo, & Brockhaus, 2011). Among the 60 valid participants, 41 produced a valid WMT (>82.5% on all three primary effort indices), and 18 were identified using the GMIP algorithm in the context of a clinical/medical history consistent with cognitive impairment. Given the sample size, indeterminate participants with inconsistent TOMM/WMT findings (i.e., one pass; one failure) were classified using a third freestanding PVT, the DCT (Boone et al., 2002), previously validated in a similarly diverse veteran sample with a total Effort-score cutoff of ≥15 yielding 70% sensitivity and 88% specificity (Soble et al., 2017). Data Analyses Correlational analyses were conducted for HVLT-R/BVMT-R scores, demographic variables, and criterion PVTs. Independent samples t-tests were performed to examine differences in HVLT-R/BVMT-R variables between validity groups. The false discovery rate (FDR) procedure was used to minimize false positives associated with multiple t-tests (Benjamini & Hochberg, 1995). For variables found to significantly differ between groups, receiver operating characteristic (ROC) curves were generated to examine sensitivity/specificity values for identifying noncredible performance. RESULTS Correlation coefficients for valid participants on HVLT-R/BVMT-R variables, demographic variables, and criterion PVTs are presented in Supplementary Table S1 available online. In brief, age was negatively correlated with all memory indices, whereas education and bilingualism were not. Scores on HVLT-R and BVMT-R PR and RD were significantly correlated with all WMT indices. Interestingly, only HVLT-R RD was significantly correlated with TOMM T1. Per Table 1, noncredible participants scored significantly lower than their valid counterparts, with medium to large effect sizes, across all HVLT-R/BVMT-R variables except for the BVMT-R PR. Thus, the BVMT-R PR finding described by Sawyer et al. (2017) did not replicate in this sample. Table 1. Performance Comparison of HVLT-R and BVMT-R Between Validity Groups Variable  Validity Group  n  Mean (SD)  t  d  HVLT-R Total Learning  Valid  56  20.00 (5.05)  2.91*  .85  Invalid  15  15.73 (5.01)  HVLT-R Delayed Recall  Valid  56  4.70 (3.79)  2.25*  .71  Invalid  15  2.33 (2.79)  HVLT-R Recognition Disc.  Valid  56  7.54 (3.30)  4.86***  1.52  Invalid  15  3.07 (2.55)  HVLT-R Retention %  Valid  56  53.27 (37.84)  2.14*  .65  Invalid  15  30.33 (32.88)  BVMT-R Total Learning  Valid  60  18.40 (7.96)  2.69*  .73  Invalid  20  13.15 (6.23)  BVMT-R Delayed Recall  Valid  60  7.22 (3.63)  2.60*  .67  Invalid  20  4.80 (3.48)  BVMT-R Recognition Disc.  Valid  60  5.03 (1.24)  2.97*  .88  Invalid  20  3.20 (2.67)  BVMT-R Retention %  Valid  60  83.38 (27.69)  .925  .26  Invalid  20  74.00 (42.47)  Variable  Validity Group  n  Mean (SD)  t  d  HVLT-R Total Learning  Valid  56  20.00 (5.05)  2.91*  .85  Invalid  15  15.73 (5.01)  HVLT-R Delayed Recall  Valid  56  4.70 (3.79)  2.25*  .71  Invalid  15  2.33 (2.79)  HVLT-R Recognition Disc.  Valid  56  7.54 (3.30)  4.86***  1.52  Invalid  15  3.07 (2.55)  HVLT-R Retention %  Valid  56  53.27 (37.84)  2.14*  .65  Invalid  15  30.33 (32.88)  BVMT-R Total Learning  Valid  60  18.40 (7.96)  2.69*  .73  Invalid  20  13.15 (6.23)  BVMT-R Delayed Recall  Valid  60  7.22 (3.63)  2.60*  .67  Invalid  20  4.80 (3.48)  BVMT-R Recognition Disc.  Valid  60  5.03 (1.24)  2.97*  .88  Invalid  20  3.20 (2.67)  BVMT-R Retention %  Valid  60  83.38 (27.69)  .925  .26  Invalid  20  74.00 (42.47)  Note: HVLT-R = Hopkins Verbal Learning Test—Revised; BVMT-R = Brief Visuospatial Memory Test—Revised; Total Learning = Sum of learning trials 1, 2, & 3; Recognition Disc. = Recognition discrimination (or true positives minus false positives); Retention % = Percentage retained (or delayed recall divided by better score on trial 2 or 3). *p < .05, ***p < .001. All p-values reflect false discovery rate (FDR)-corrected p-values. Table 1. Performance Comparison of HVLT-R and BVMT-R Between Validity Groups Variable  Validity Group  n  Mean (SD)  t  d  HVLT-R Total Learning  Valid  56  20.00 (5.05)  2.91*  .85  Invalid  15  15.73 (5.01)  HVLT-R Delayed Recall  Valid  56  4.70 (3.79)  2.25*  .71  Invalid  15  2.33 (2.79)  HVLT-R Recognition Disc.  Valid  56  7.54 (3.30)  4.86***  1.52  Invalid  15  3.07 (2.55)  HVLT-R Retention %  Valid  56  53.27 (37.84)  2.14*  .65  Invalid  15  30.33 (32.88)  BVMT-R Total Learning  Valid  60  18.40 (7.96)  2.69*  .73  Invalid  20  13.15 (6.23)  BVMT-R Delayed Recall  Valid  60  7.22 (3.63)  2.60*  .67  Invalid  20  4.80 (3.48)  BVMT-R Recognition Disc.  Valid  60  5.03 (1.24)  2.97*  .88  Invalid  20  3.20 (2.67)  BVMT-R Retention %  Valid  60  83.38 (27.69)  .925  .26  Invalid  20  74.00 (42.47)  Variable  Validity Group  n  Mean (SD)  t  d  HVLT-R Total Learning  Valid  56  20.00 (5.05)  2.91*  .85  Invalid  15  15.73 (5.01)  HVLT-R Delayed Recall  Valid  56  4.70 (3.79)  2.25*  .71  Invalid  15  2.33 (2.79)  HVLT-R Recognition Disc.  Valid  56  7.54 (3.30)  4.86***  1.52  Invalid  15  3.07 (2.55)  HVLT-R Retention %  Valid  56  53.27 (37.84)  2.14*  .65  Invalid  15  30.33 (32.88)  BVMT-R Total Learning  Valid  60  18.40 (7.96)  2.69*  .73  Invalid  20  13.15 (6.23)  BVMT-R Delayed Recall  Valid  60  7.22 (3.63)  2.60*  .67  Invalid  20  4.80 (3.48)  BVMT-R Recognition Disc.  Valid  60  5.03 (1.24)  2.97*  .88  Invalid  20  3.20 (2.67)  BVMT-R Retention %  Valid  60  83.38 (27.69)  .925  .26  Invalid  20  74.00 (42.47)  Note: HVLT-R = Hopkins Verbal Learning Test—Revised; BVMT-R = Brief Visuospatial Memory Test—Revised; Total Learning = Sum of learning trials 1, 2, & 3; Recognition Disc. = Recognition discrimination (or true positives minus false positives); Retention % = Percentage retained (or delayed recall divided by better score on trial 2 or 3). *p < .05, ***p < .001. All p-values reflect false discovery rate (FDR)-corrected p-values. ROC curve analyses yielded significant areas under the curve (AUCs) of .670–.850 for HVLT-R variables and .678–.720 for BVMT-R variables. Similar to the t-tests, the HVLT-R RD (AUC = .850; CI = .760–.940) and BVMT-R RD (AUC = .720; CI = .576–.864) scores had the highest AUCs, so these were examined further. Table 2 presents sensitivity/specificity and positive/negative predictive values for various base rates of invalid performance for the HVLT-R and BVMT-R RD scores. For the HVLT-R, a cutoff of ≤3 maximized sensitivity (47%) and specificity (89%), whereas the previously described cutoff (≤5) had adequate sensitivity (67%), but unacceptable specificity (80%). Further examination revealed seven valid participants with an amnestic disorder (3 MCI-amnestic; 3 Alzheimer disease; 1 alcohol-induced dementia) who performed very poorly (i.e., ≤5) on the HVLT-R RD, yet passed the freestanding criterion PVTs. Reanalysis after removal of these seven false positives, yielded an improved AUC (.918) and an optimal cutoff of ≤5 (sensitivity: 67%; specificity: 92%), which replicates Sawyer et al. (2017) findings with improved sensitivity. For the BVMT-R RD, ROC analysis for the total sample yielded an optimal cutoff of ≤3 (sensitivity: 40%; specificity: 95%). With the seven previously noted amnestic patients removed, AUC increased to (.756) with an optimal cutoff of ≤4 (sensitivity: 50%; specificity: 93%). Table 2. Operational Characteristics for the HVLT-R and BVMT-R Recognition Discrimination Cutoff  SN  SP  Base rate (0.40)  Base rate (0.20)  PPV  NPV  PPV  NPV  HVLT-R Recognition Discrimination Operational Characteristics   Entire Sample (N = 71)    ≤1  0.27  0.98  0.9  0.67  0.77  0.84    ≤2  0.33  0.95  0.82  0.68  0.63  0.85    ≤3  0.47  0.89  0.74  0.71  0.51  0.87    ≤4  0.53  0.86  0.72  0.73  0.49  0.88    ≤5  0.67  0.8  0.69  0.78  0.45  0.91    ≤6  0.73  0.75  0.66  0.81  0.42  0.92    ≤7  0.93  0.64  0.64  0.94  0.4  0.97    ≤8  1.00  0.59  0.62  1.00  0.38  1.00   Seven False Positives Removed (N = 64)    ≤1  0.27  1.00  1.00  0.67  1.00  0.85    ≤2  0.33  1.00  1.00  0.69  1.00  0.86    ≤3  0.47  0.96  0.89  0.73  0.74  0.88    ≤4  0.53  0.96  0.9  0.76  0.77  0.89    ≤5  0.67  0.92  0.85  0.81  0.68  0.92    ≤6  0.73  0.86  0.78  0.83  0.57  0.93    ≤7  0.93  0.73  0.7  0.94  0.46  0.98    ≤8  1.00  0.67  0.67  1.00  0.43  1.00  BVMT-R Recognition Discrimination Operational Characteristics   Entire Sample (N = 80)    ≤1  .10  1.00  1.00  0.63  1.00  0.82    ≤2  .20  1.00  1.00  0.65  1.00  0.83    ≤3  .40  0.95  0.84  0.70  0.67  0.86    ≤4  .50  0.83  0.66  0.71  0.42  0.87    ≤5  .60  0.75  0.62  0.74  0.38  0.88    ≤6  .75  0.5  0.50  0.75  0.27  0.89   Seven False Positives Removed (N = 73)    ≤1  .10  1.00  1.00  0.63  1.00  0.82    ≤2  .20  1.00  1.00  0.65  1.00  0.83    ≤3  .40  1.00  1.00  0.71  1.00  0.87    ≤4  .50  0.93  0.83  0.74  0.64  0.88    ≤5  .60  0.83  0.70  0.76  0.47  0.89    ≤6  .75  0.55  0.53  0.77  0.29  0.90  Cutoff  SN  SP  Base rate (0.40)  Base rate (0.20)  PPV  NPV  PPV  NPV  HVLT-R Recognition Discrimination Operational Characteristics   Entire Sample (N = 71)    ≤1  0.27  0.98  0.9  0.67  0.77  0.84    ≤2  0.33  0.95  0.82  0.68  0.63  0.85    ≤3  0.47  0.89  0.74  0.71  0.51  0.87    ≤4  0.53  0.86  0.72  0.73  0.49  0.88    ≤5  0.67  0.8  0.69  0.78  0.45  0.91    ≤6  0.73  0.75  0.66  0.81  0.42  0.92    ≤7  0.93  0.64  0.64  0.94  0.4  0.97    ≤8  1.00  0.59  0.62  1.00  0.38  1.00   Seven False Positives Removed (N = 64)    ≤1  0.27  1.00  1.00  0.67  1.00  0.85    ≤2  0.33  1.00  1.00  0.69  1.00  0.86    ≤3  0.47  0.96  0.89  0.73  0.74  0.88    ≤4  0.53  0.96  0.9  0.76  0.77  0.89    ≤5  0.67  0.92  0.85  0.81  0.68  0.92    ≤6  0.73  0.86  0.78  0.83  0.57  0.93    ≤7  0.93  0.73  0.7  0.94  0.46  0.98    ≤8  1.00  0.67  0.67  1.00  0.43  1.00  BVMT-R Recognition Discrimination Operational Characteristics   Entire Sample (N = 80)    ≤1  .10  1.00  1.00  0.63  1.00  0.82    ≤2  .20  1.00  1.00  0.65  1.00  0.83    ≤3  .40  0.95  0.84  0.70  0.67  0.86    ≤4  .50  0.83  0.66  0.71  0.42  0.87    ≤5  .60  0.75  0.62  0.74  0.38  0.88    ≤6  .75  0.5  0.50  0.75  0.27  0.89   Seven False Positives Removed (N = 73)    ≤1  .10  1.00  1.00  0.63  1.00  0.82    ≤2  .20  1.00  1.00  0.65  1.00  0.83    ≤3  .40  1.00  1.00  0.71  1.00  0.87    ≤4  .50  0.93  0.83  0.74  0.64  0.88    ≤5  .60  0.83  0.70  0.76  0.47  0.89    ≤6  .75  0.55  0.53  0.77  0.29  0.90  Note: Bolded numbers indicate optimal cutoff; HVLT-R = Hopkins Verbal Learning Test—Revised; BVMT-R = Brief Visuospatial Memory Test—Revised; SN = Sensitivity; SP = Specificity; PPV = Positive Predictive Value; NPV = Negative Predictive Value. Table 2. Operational Characteristics for the HVLT-R and BVMT-R Recognition Discrimination Cutoff  SN  SP  Base rate (0.40)  Base rate (0.20)  PPV  NPV  PPV  NPV  HVLT-R Recognition Discrimination Operational Characteristics   Entire Sample (N = 71)    ≤1  0.27  0.98  0.9  0.67  0.77  0.84    ≤2  0.33  0.95  0.82  0.68  0.63  0.85    ≤3  0.47  0.89  0.74  0.71  0.51  0.87    ≤4  0.53  0.86  0.72  0.73  0.49  0.88    ≤5  0.67  0.8  0.69  0.78  0.45  0.91    ≤6  0.73  0.75  0.66  0.81  0.42  0.92    ≤7  0.93  0.64  0.64  0.94  0.4  0.97    ≤8  1.00  0.59  0.62  1.00  0.38  1.00   Seven False Positives Removed (N = 64)    ≤1  0.27  1.00  1.00  0.67  1.00  0.85    ≤2  0.33  1.00  1.00  0.69  1.00  0.86    ≤3  0.47  0.96  0.89  0.73  0.74  0.88    ≤4  0.53  0.96  0.9  0.76  0.77  0.89    ≤5  0.67  0.92  0.85  0.81  0.68  0.92    ≤6  0.73  0.86  0.78  0.83  0.57  0.93    ≤7  0.93  0.73  0.7  0.94  0.46  0.98    ≤8  1.00  0.67  0.67  1.00  0.43  1.00  BVMT-R Recognition Discrimination Operational Characteristics   Entire Sample (N = 80)    ≤1  .10  1.00  1.00  0.63  1.00  0.82    ≤2  .20  1.00  1.00  0.65  1.00  0.83    ≤3  .40  0.95  0.84  0.70  0.67  0.86    ≤4  .50  0.83  0.66  0.71  0.42  0.87    ≤5  .60  0.75  0.62  0.74  0.38  0.88    ≤6  .75  0.5  0.50  0.75  0.27  0.89   Seven False Positives Removed (N = 73)    ≤1  .10  1.00  1.00  0.63  1.00  0.82    ≤2  .20  1.00  1.00  0.65  1.00  0.83    ≤3  .40  1.00  1.00  0.71  1.00  0.87    ≤4  .50  0.93  0.83  0.74  0.64  0.88    ≤5  .60  0.83  0.70  0.76  0.47  0.89    ≤6  .75  0.55  0.53  0.77  0.29  0.90  Cutoff  SN  SP  Base rate (0.40)  Base rate (0.20)  PPV  NPV  PPV  NPV  HVLT-R Recognition Discrimination Operational Characteristics   Entire Sample (N = 71)    ≤1  0.27  0.98  0.9  0.67  0.77  0.84    ≤2  0.33  0.95  0.82  0.68  0.63  0.85    ≤3  0.47  0.89  0.74  0.71  0.51  0.87    ≤4  0.53  0.86  0.72  0.73  0.49  0.88    ≤5  0.67  0.8  0.69  0.78  0.45  0.91    ≤6  0.73  0.75  0.66  0.81  0.42  0.92    ≤7  0.93  0.64  0.64  0.94  0.4  0.97    ≤8  1.00  0.59  0.62  1.00  0.38  1.00   Seven False Positives Removed (N = 64)    ≤1  0.27  1.00  1.00  0.67  1.00  0.85    ≤2  0.33  1.00  1.00  0.69  1.00  0.86    ≤3  0.47  0.96  0.89  0.73  0.74  0.88    ≤4  0.53  0.96  0.9  0.76  0.77  0.89    ≤5  0.67  0.92  0.85  0.81  0.68  0.92    ≤6  0.73  0.86  0.78  0.83  0.57  0.93    ≤7  0.93  0.73  0.7  0.94  0.46  0.98    ≤8  1.00  0.67  0.67  1.00  0.43  1.00  BVMT-R Recognition Discrimination Operational Characteristics   Entire Sample (N = 80)    ≤1  .10  1.00  1.00  0.63  1.00  0.82    ≤2  .20  1.00  1.00  0.65  1.00  0.83    ≤3  .40  0.95  0.84  0.70  0.67  0.86    ≤4  .50  0.83  0.66  0.71  0.42  0.87    ≤5  .60  0.75  0.62  0.74  0.38  0.88    ≤6  .75  0.5  0.50  0.75  0.27  0.89   Seven False Positives Removed (N = 73)    ≤1  .10  1.00  1.00  0.63  1.00  0.82    ≤2  .20  1.00  1.00  0.65  1.00  0.83    ≤3  .40  1.00  1.00  0.71  1.00  0.87    ≤4  .50  0.93  0.83  0.74  0.64  0.88    ≤5  .60  0.83  0.70  0.76  0.47  0.89    ≤6  .75  0.55  0.53  0.77  0.29  0.90  Note: Bolded numbers indicate optimal cutoff; HVLT-R = Hopkins Verbal Learning Test—Revised; BVMT-R = Brief Visuospatial Memory Test—Revised; SN = Sensitivity; SP = Specificity; PPV = Positive Predictive Value; NPV = Negative Predictive Value. DISCUSSION This study aimed to replicate Sawyer et al. (2017) findings, which was the first published study to examine embedded PVTs within the commonly used HVLT-R and BVMT-R. In a clinical veteran sample with a 21% noncredible performance base rate, the HVLT-R RD score performed well using a cutoff of ≤5 (53% sensitivity/93% specificity), with less robust findings from the BVMT-R PR cutoff of ≤58% (31% sensitivity/92% specificity). In this study, replication findings were mixed. The HVLT-R RD significantly differed between validity groups with a large effect size (d = 1.52) in heterogeneous clinical groups with and without cognitive impairment. A cut-score of ≤5 yielded 67% sensitivity, but unacceptable specificity (80%), whereas ≤3 had 47% sensitivity and 89% specificity. Removal of seven amnestic participants replicated the previously reported cutoff of ≤5 with sensitivity remaining at 67% and specificity substantially increasing to 92%. This converging evidence supports the use of the HVLT-R RD for detecting noncredible performance, while also cautioning the specific cutoff selected (i.e., ≤3 versus ≤5) may need to be informed by whether an amnestic disorder versus other etiology of impairment is suspected. By contrast, BVMT-R PR results did not replicate, as these scores did not differ significantly between validity groups. One possible explanation for this nonsignificant finding is that BVMT-R PR can be heavily influenced by processing speed during the encoding trials and frank memory impairment during the recall phases of the BVMT-R (Tam & Schmitter-Edgecombe, 2013). Given that roughly half of the valid participants were cognitively impaired, it is reasonable to hypothesize that a combination of attention, processing speed, visuoperception, and memory difficulties combined to account for a larger portion of the variance than measurable “effort.” A novel finding in the current study was performance on BVMT-R RD distinguished validity groups well (cutoff ≤3; 40% sensitivity/95% specificity), with more robust classification after false positives were removed from the analysis (cutoff ≤4; 50% sensitivity/93% specificity). The current replication study bolstered Sawyer et al. (2017) initial findings through the use of a heterogeneous clinical sample with good diversity in terms of age, level of education, race/ethnicity, and language preference, and with performance validity categorized by a more stringent criterion (i.e., two well-validated, freestanding PVT failures). Further, we addressed a limitation of the original validation by including a sample with a higher percentage of patients with cognitive impairment. The finding of similar accuracy for HVLT-R RD in our sample preliminarily suggests this index may identify noncredible performance in the context of cognitive impairment, though special consideration may need to be taken in cases of patients with a suspected amnestic disorder. Notably, they may perform poorly on the HVLT-R on the whole and therefore be more likely to be incorrectly classified as invalid if a clinician were to use the ≤5 cutoff, especially in the absence of other converging PVT evidence of noncredible performance. An additional strength of this sample is that the validity groups were similarly educated, which was a potential limitation in the initial study. However, the fact the HVLT-R RD performed so similarly across samples supports Sawyer et al. (2017) position that the statistically significant difference in education between groups was not clinically meaningful. This study had some limitations, including retrospective design and use of a mixed clinical sample. Further validation should be conducted in specific, homogenous clinical samples (e.g., TBI, Alzheimer’s disease) to determine whether the HVLT-R/BVMT-R RD scores continue to demonstrate adequate classification accuracy across clinical groups and to establish optimal cutoff scores for various clinical groups, if indicated. It is noted that the rate of invalid performance in the current sample was 25%, which was similar to the 21% reported in the Sawyer et al. (2017) sample. Similar to the original study, data pertaining to potential external motivating factors for invalid performance (e.g., disability-seeking status) in the current sample generally were not collected, and it is therefore impossible to determine the source of the discrepancies between samples. We included PPV/NPV values for a range of base rates of invalid performance to address this weakness. Another limitation shared by Sawyer et al. (2017) was a small sample size, which could have contributed to inconsistent findings across BVMT-R PR and RD. Larger validation samples are required to increase clinician confidence with including an embedded PVT derived from the BVMT-R. Despite limitations, the current study provides replicative support for the HVLT-R RD score as an embedded PVT, and offers initial support for the BVMT-R RD. CONCLUSIONS Overall, this study extended previous findings which demonstrated the embedded PVT within the HVLT-R is robust within a clinically referred veteran sample. This study is significant to this literature due to the more ethnically and linguistically diverse sample, with higher rates of cognitive impairment and invalidity. Converging evidence generally supports clinical use of an HVLT-R RD cutoff of ≤5 (or ≤3 in cases of suspected amnestic disorders), in conjunction with other well-validated PVTs, to facilitate ongoing performance validity assessment throughout neuropsychological evaluations. Future research can use larger sample sizes and potentially combine other commonly used clinical measures to provide clinicians with a formulaic approach for determining performance validity with enhanced psychometric properties. SUPPLEMENTARY DATA Supplementary material are available at Archives of Clinical Neuropsychology online. FUNDING The authors have no financial interest with the subject matter discussed in the manuscript. CONFLICT OF INTEREST None declared. ACKNOWLEDGMENTS The views expressed herein are those of the authors and do not necessarily reflect the views or the official policy of the Department of Veterans Affairs or U.S. Government. REFERENCES American Psychiatric Association. ( 2013). Diagnostic and statistical manual of mental disorders—Fifth edition (DSM-5) . Washington, DC: American Psychiatric Publishing. Benedict, R. H. B. ( 1997). Brief Visuospatial Memory Test—Revised . Odessa, FL: Psychological Assessment Resources. Benedict, R. H. B., Schretlen, D., Groninger, L., Dobraski, M., & Shpritz, B. ( 1996). Revision of the Brief Visuospatial Memory Test: Studies of normal performance, reliability, and validity. Psychological Assessment , 8, 145– 153. Google Scholar CrossRef Search ADS   Benjamini, Y., & Hochberg, Y. ( 1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society Series B (Methodological) , 57, 289– 300. Boone, K. B., Lu, P., & Herzberg, D. ( 2002). The Dot Counting Test manual . Los Angeles, CA: Western Psychological Services. Brandt, J., & Benedict, R. H. B. ( 2001). Hopkins Verbal Learning Test—Revised . Odessa, FL: Psychological Assessment Resources. Denning, J. H. ( 2012). The efficiency and accuracy of the Test of Memory Malingering trial 1, errors on the first 10 items of the Test of Memory Malingering, and five embedded measures in predicting invalid test performance. Archives of Clinical Neuropsychology , 27, 417– 432. Google Scholar CrossRef Search ADS PubMed  Green, P. ( 2003). Green’s Word Memory Test for Windows: User’s manual . Edmonton, Canada: Green’s Publishing. Green, P., Montijo, J., & Brockhaus, R. ( 2011). High specificity of the Word Memory Test and Medical Symptom Validity Test in groups with severe verbal memory impairment. Applied Neuropsychology , 18, 86– 94. Google Scholar CrossRef Search ADS PubMed  Heilbronner, R. L., Sweet, J. J., Morgan, J. E., Larrabee, G. J., & Millis, S. R. ( 2009). American Academy of Clinical Neuropsychology consensus conference statement on the neuropsychological assessment of effort, response bias, and malingering. The Clinical Neuropsychologist , 23, 1093– 1129. Google Scholar CrossRef Search ADS PubMed  Hilsabeck, R. C., Gordon, S. N., Hietpas-Wilson, T., & Zartman, A. ( 2011). Use of trial 1 of the Test of Memory Malingering (TOMM) as a screening measure of effort: Suggested discontinuation rules. The Clinical Neuropsychologist , 25, 1228– 1238. Google Scholar CrossRef Search ADS PubMed  Institute of Medicine of the National Academies. ( 2015). Psychological testing in the service of disability determination . Washington, DC: National Academies Press (US). (Chapter 5). Larrabee, G. J. ( 2008). Aggregation across multiple indicators improves the detection of malingering: Relationship to likelihood ratios. The Clinical Neuropsychologist , 22, 666– 679. Google Scholar CrossRef Search ADS PubMed  Lezak, M. D., Howieson, D. B., Bigler, E. D., & Tranel, D. ( 2012). Neuropsychological assessment  ( 5th ed.). Oxford: Oxford University Press. Sawyer, R. J., Testa, S. M., & Dux, M. ( 2017). Embedded performance validity tests within the Hopkins Verbal Learning Test—Revised and the Brief Visuospatial Memory Test—Revised. The Clinical Neuropsychologist , 31, 207– 218. Google Scholar CrossRef Search ADS PubMed  Slick, D. J., Sherman, E. M., & Iverson, G. L. ( 1999). Diagnostic criteria for malingered neurocognitive dysfunction: Proposed standards for clinical practice and research. The Clinical Neuropsychologist , 13, 545– 561. Google Scholar CrossRef Search ADS PubMed  Soble, J. R., Santos, O. A., Bain, K. M., Kirton, J. W., Bailey, K. C., Critchfield, E. A., et al.  . ( 2017). The Dot Counting Test adds up: Validation and response pattern analysis in a mixed clinical veteran sample. Journal of Clinical and Experimental Neuropsychology , 1– 9. doi: 10.1080/13803395.2017.1342773. Tam, J. W., & Schmitter-Edgecombe, M. ( 2013). The role of processing speed in the Brief Visuospatial Memory Test—Revised. The Clinical Neuropsychologist , 27, 962– 972. Google Scholar CrossRef Search ADS PubMed  Tombaugh, T. N. ( 1996). Test of Memory Malingering (TOMM) . North Tonawanda, NY: Multi Health Systems. Wacholder, S., Chanock, S., Garcia-Closas, M., El Ghormli, L., & Rothman, N. ( 2004). Assessing the probability that a positive report is false: An approach for molecular epidemiology studies. Journal of the National Cancer Institute , 96, 434– 442. Google Scholar CrossRef Search ADS PubMed  Published by Oxford University Press 2017. This work is written by (a) US Government employee(s) and is in the public domain in the US.

Journal

Archives of Clinical NeuropsychologyOxford University Press

Published: Nov 17, 2017

There are no references for this article.

You’re reading a free preview. Subscribe to read the entire article.


DeepDyve is your
personal research library

It’s your single place to instantly
discover and read the research
that matters to you.

Enjoy affordable access to
over 18 million articles from more than
15,000 peer-reviewed journals.

All for just $49/month

Explore the DeepDyve Library

Search

Query the DeepDyve database, plus search all of PubMed and Google Scholar seamlessly

Organize

Save any article or search result from DeepDyve, PubMed, and Google Scholar... all in one place.

Access

Get unlimited, online access to over 18 million full-text articles from more than 15,000 scientific journals.

Your journals are on DeepDyve

Read from thousands of the leading scholarly journals from SpringerNature, Elsevier, Wiley-Blackwell, Oxford University Press and more.

All the latest content is available, no embargo periods.

See the journals in your area

DeepDyve

Freelancer

DeepDyve

Pro

Price

FREE

$49/month
$360/year

Save searches from
Google Scholar,
PubMed

Create lists to
organize your research

Export lists, citations

Read DeepDyve articles

Abstract access only

Unlimited access to over
18 million full-text articles

Print

20 pages / month

PDF Discount

20% off