A Reliable and Valid Assessment of Sustained Attention for Patients With Schizophrenia: The Computerized Digit Vigilance Test

A Reliable and Valid Assessment of Sustained Attention for Patients With Schizophrenia: The... Abstract Objective The purposes of this study were to examine the test–retest reliability, concurrent validity, and ecological validity of the Computerized Digit Vigilance Test (C-DVT) in patients with schizophrenia. Method Each participant was assessed four times, with 1-week intervals. In each assessment, the participants completed both the C-DVT and the original DVT. The participants were also assessed using the Lawton Instrumental Activities of Daily Living Scale (LIADL) and the Personal and Social Performance Scale (PSP). Results Forty-nine participants were recruited in this study. The results showed that the test–retest agreement of the C-DVT was good-to-excellent (intraclass correlation coefficient = 0.71–0.89). The random measurement errors of the C-DVT were acceptable (percentages of minimal detectable change = 12.9%–24.1%). The practice effect of the C-DVT reached a plateau after three assessments (effect size <0.20). The concurrent validity of the C-DVT was good (r = .75–.79 with DVT) when we controlled for the randomized administration order of the two tests. The ecological validity of the C-DVT was good (r = −.44 with the LIADL; r = −.45 with the PSP). Conclusions The C-DVT had acceptable test–retest reliability, sound concurrent validity, and sound ecological validity in patients with schizophrenia. These findings indicate that the C-DVT has the potential to be a reliable and valid test of sustained attention in patients with schizophrenia. Sustained attention, Psychometric property, Schizophrenia, Cognition Introduction Sustained attention can be defined as the ability to maintain consistent behavioral performance of a task for a reasonable duration of time (at least 3 min) (Betts, Mckay, Maruff, & Anderson, 2006; Loetscher & Lincoln, 2013; Sohlberg & Mateer, 1987). Sustained attention is a fundamental cognitive function involved in many daily activities, such as reading, writing, or working for a long time. Sustained attention deficit appears to be a common deficit in patients with schizophrenia (Liu et al., 2002). The deficit of sustained attention in patients with schizophrenia is associated with difficulties in social functioning (e.g., interpersonal communication and social activities) and instrumental activities of daily living (IADL, e.g., shopping, financial management, and taking medicine) (Evans et al., 2003; Green, 1996; Lipskaya, Jarus, & Kotler, 2011; Woonings, Appelo, Kluiter, Slooff, & van den Bosch, 2003). Because deficits of sustained attention are closely associated with the functional outcomes of patients with schizophrenia, the assessment and management of sustained attention deficits are crucial for clinicians in helping patients live independently in the community. The Digit Vigilance Test (DVT) is an often-used sustained attention test in patient with schizophrenia (Chan, Yip, & Lee, 2004; Depp et al., 2007; Lahiri & Singh, 2016; Raghavan, Shanmugiah, Bharathi, & Jeyaprakash, 2016). However, a previous study has indicated that the practice effect and random measurement error of the DVT are substantial in patients with schizophrenia (Lee, Li, Liu, & Hsieh, 2011). These drawbacks can cause inconsistencies in the results of the DVT across repeated assessments (Lee et al., 2011). The inconsistencies in the results may lead to misinterpretation of the performance changes in sustained attention even when subjects’ function of sustained attention may have been stable. To assess sustained attention with a limited practice effect and random measurement error, the Computerized Digit Vigilance Test (C-DVT) was developed based on the DVT (Yang, Lin, Chen, Hsueh, & Hsieh, 2015). The proposed C-DVT in the current study incorporated a design for reducing the practice effect: increasing the number of practice trials (Yang et al., 2015). This design helped subjects become fully familiar with the operation rules and methods of the C-DVT (Yang et al., 2015) before they performed the formal testing. To reduce random measurement error caused by patients’ poor visual acuity and visual scanning, the C-DVT adopted a large font size for the stimuli and fewer stimuli (one column of five stimuli) on each page. Because of the aforementioned features of the C-DVT, the C-DVT has the potential to assess sustained attention with a smaller practice effect and smaller random measurement error for patients with schizophrenia. The sound concurrent validity of the C-DVT has been validated in patients with stroke, which supports the C-DVT as a measure of sustained attention (Yang et al., 2015). Additionally, the test–retest reliability and ecological validity of the C-DVT are also sufficient in patients (Yang et al., 2015). However, such positive findings cannot be assumed for patients with schizophrenia because psychometric properties are population dependent (Hambleton, 2000). Thus, the psychometric properties of the C-DVT in patients with schizophrenia remain unknown. The purposes of this study were to examine the test–retest reliability, concurrent validity, and ecological validity of the C-DVT in patients with schizophrenia. Methods Participants We recruited patients by convenience sampling from two community-based psychiatric rehabilitation day centers affiliated with a medical center in northern Taiwan. Patients were included in this study if they met the following criteria: (1) diagnosis of schizophrenia according to the criteria of the Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition (APA, 2013), (2) age ≥ 20 years, and (3) score of the Mini-Mental Status Examination (MMSE) ≥21 points. The exclusion criteria of this study were patients with (1) diagnosis of other neurological or psychiatric diseases affecting cognition (e.g., stroke or depression), (2) another severe medical condition or psychiatric disorder that required treatment during study, or (3) unstable severity of illness [specifically, the scores of the Clinical Global Impressions Scale-Severity (CGI-S) were different between the first and last assessments of the study]. This study was approved by the Institutional Review Board of the medical center. All participants signed consent forms before participating in this study. The sample size needed for the test–retest reliability analysis was estimated using power analysis with alpha = 0.05 and power = 0.80 (Doros & Lew, 2010). In addition, we set the width of the 95% confidence interval of the ICC at ≤0.30 and the ICC of the C-DVT at ≥0.70 (Yang et al., 2015). The power analysis using the aforementioned parameters showed that the appropriate sample size was 50 participants. Procedure This study contained four assessment sessions (referred to as Assessments 1–4) with 1-week intervals between two adjacent sessions. The participants completed both the C-DVT and DVT in all four assessment sessions. The testing order of the C-DVT and DVT was randomized for all participants and every assessment session. Between the administrations of the C-DVT and DVT, the participants rested for 3 min. We also administered the Lawton Instrumental Activities of Daily Living Scale (LIADL) in Assessment 3, the Personal and Social Performance Scale (PSP) in Assessment 4, and the CGI-S twice, in Assessments 1 and 4. One occupational therapist assessed all participants across four assessment sessions. In addition, we collected the patients’ demographic characteristics from chart review. Measures C-DVT (Yang et al., 2015): The C-DVT, which is based on the DVT, can be used to assess sustained attention (Yang et al., 2015). The C-DVT contains 28 trials in a practice session and 120 trials in a formal test. The operation interface of the C-DVT includes a computer screen and an external keyboard with only two buttons [a circle (“O”) and an X (“X”)]. When performing the C-DVT, subjects have to judge whether the screen displays the digit “6” in a column of 5 digits. If the screen displays the digit “6”, subjects should press the button with a circle; otherwise, subjects should press the button with an X. The C-DVT automatically records the total time (in s) needed for completing the test and the number of errors. The total time for completing the C-DVT is used as a primary index for evaluating participants’ ability of sustained attention. A shorter time for completing the C-DVT indicates better sustained attention. DVT (Lewis, 1995): The DVT is presented as a two-page test sheet containing 59 rows of 35 digits (i.e., 0–9) in a 12-point (pt) font size. To perform the DVT, subjects have to cross out all occurrences of the digit “6” on the test sheets. The rater of the DVT records the total time (in s) needed to complete the test sheet and the number of errors. The total time for completing the DVT is treated as a primary index of sustained attention. In this study, we used the DVT as an external criterion for examining the concurrent validity of the C-DVT. LIADL (Lawton & Brody, 1969): The LIADL contains eight items, which assess patients’ performance on eight IADL tasks (i.e., using a telephone, shopping, food preparation, housekeeping, laundry, transportation, responsibility for own medications, and ability to handle finances). The LIADL is commonly administered by face-to-face interview, and the items of the LIADL are scored according to whether a participant can independently perform the IADL tasks. The score range of the LIADL is 0–23. A higher score represents better IADL function. We used the LIADL as an external criterion for examining the ecological validity of the C-DVT. PSP (Morosini, Magliano, Brambilla, Ugolini, & Pioli, 2000): The PSP measures personal and social function in four sub-domains: socially useful activities (e.g., work and study), personal and social relationships, self-care, and disturbing and aggressive behaviors (Morosini et al., 2000). The PSP is rated through observing a participant’s performance on four sub-domains. The score of a sub-domain is scored on a 6-point scale (from 1: absent disability to 6: very severe disability). A total score (0–100) is transformed from the scores of the sub-domains. A higher total score represents better personal and social function. We used the PSP as an external criterion for examining the ecological validity of the C-DVT. CGI-S (Haro et al., 2003): The CGI-S assesses severity of psychiatric illness on a 7-point scale (1–7). The rating of the CGI-S is based on observed and self-reported symptoms, behavior, and function within 1 week. One point of the CGI-S represents that a patient is not ill, and 7 points represents that a patient is most severely ill. We used the CGI-S for examining whether the illness severity of the participants was stable during the study. Data Analysis Test–retest reliability Test–retest reliability has been defined as the extent to which repeated test scores of patients, who have not changed in the characteristic to be measured, are consistent (Mokkink et al., 2010). To examine test–retest reliability, we analyzed the test–retest agreement, random measurement error, and practice effect of the C-DVT and DVT in patients with schizophrenia. To examine test–retest agreement of the C-DVT and DVT, intraclass correlation coefficients (ICCs) (Bartko, 1966) were used for two contexts: the ICC across four repeated assessments and the ICC of two adjacent assessments (i.e., Assessment 1 vs. Assessment 2, Assessment 2 vs. Assessment 3, and Assessment 3 vs. Assessment 4). The purpose of calculating the ICC across four repeated assessments was to indicate the overall test–retest agreement of the C-DVT and DVT. The purpose of calculating the ICC of two adjacent assessments was to indicate the change in test–retest agreement as the number of assessment sessions increased. The ICC was defined as the ratio of between-subject variance to total variance (between-subject variance + within-subject variance), and was calculated based on a two-way random model of analysis of variance (Bartko, 1966). An ICC value > 0.80 was considered to indicate excellent test–retest agreement; 0.60–0.79, good agreement; 0.40–0.59, moderate; and <0.40, poor (Bushnell, Johnston, & Goldstein, 2001). To visually assess each participant’s agreements of the C-DVT and DVT scores, we drew Bland–Altman plots using each participant’s mean score of the first two assessments against the difference of scores between the two assessments (Bland & Altman, 1986). The limits of agreement (LOAs) were calculated to illustrate the width of distribution of the score differences using the following formula: the mean difference between assessments ± 1.96 × standard deviation of the difference between assessments. We calculated the minimal detectable change (MDC) and MDC percentage (MDC%) to examine the random measurement error of the C-DVT and DVT. The MDC was calculated using the following formula (Haley & Fragala-Pinkham, 2006):   MDC=1.96×SDfirst assessment of each 2 adjacent assessments×2×(1-ICC2 adjacent assessments) The MDC estimated the amount of a score change that could be considered a real change (beyond the score change caused by random measurement error) at the 95% confidence level. The MDC% was calculated by dividing the MDC by the mean scores of two adjacent assessments and then multiplying by 100%. An MDC% below 20% was considered acceptable random measurement error (Huang et al., 2011). To examine the practice effects of the C-DVT and DVT, we analyzed the mean differences between each two adjacent assessments using both paired t-test and effect size. The significance level, alpha, of the paired t-test was set at 0.017 (0.05/3 times of comparisons) for reducing Type I errors of multiple comparisons between each two-adjacent assessment of the four assessment sessions. Furthermore, the effect size was calculated by dividing the mean score differences of each two adjacent assessments by the SD of the score differences. An effect size >0.80 was considered to indicate a large practice effect; 0.50–0.79, medium; 0.20–0.49, small; and <0.20, negligible (Cohen, 1988). In addition, we examined whether the practice effect had reached a plateau phase by the following two criteria (Chiu et al., 2014): (1) the effect size of two adjacent assessments was smaller than 0.20 (e.g., whether the effect size between Assessments 2–3 was smaller than 0.20) and (2) the later effect size of two adjacent effect sizes was smaller than the previous practice effect (e.g., whether the effect size of Assessments 2–3 was smaller than that of Assessments 1–2). Concurrent validity Concurrent validity can be defined as the extent of correlation between a new or revised test (e.g., the C-DVT) and its original test (e.g., the DVT) (La Porta et al., 2011). We examined the concurrent validity of the C-DVT by analyzing the levels of correlation between the C-DVT and DVT scores across the four assessment sessions using the Pearson’s correlation coefficient (r). However, the results of the concurrent validity examination might have been attenuated by systematic bias due to the study design (Ibrahim et al., 2015). We designed the random administration order of the C-DVT and DVT for each participant at each session. Therefore, about half of the participants completed the C-DVT first, and the other half of the participants completed the DVT first. The random administration order of the two tests might have caused systematic bias. For example, because both tests instruct participants to detect the digit “6”, participants might have become more familiar with detecting the digit “6” after performing the first test. Thus, the results of the second test might have been improved by the experience of performing the first test (Ibrahim et al., 2015). To control for the possible systematic bias so as to examine the concurrent validity, we adopted the methods proposed by Ibrahim and colleagues (2015). First, we adjusted the scores of the second tests by conducting linear regression models to estimate the amounts of change score caused by systematic bias. Then we used the estimated change score to adjust the scores of the second tests (i.e., the change score was added to the scores of the second test for each participant). Second, we used the raw scores and adjusted scores to analyze the levels of correlation between the C-DVT and DVT across four assessment sessions. An absolute value of r > .75 was considered to indicate high concurrent validity; .40–.74, acceptable; and <.40, poor (Salter et al., 2005). Ecological validity Ecological validity indicates the extent of correlation between a patient’s performance on a test and a patient’s daily function (Chaytor & Schmitter-Edgecombe, 2003). In addition, we examined the ecological validity of the C-DVT and DVT by analyzing the levels of correlation between the C-DVT/DVT and both the LIADL and PSP using Pearson’s r. The C-DVT/DVT score and the LIADL score (or the PSP score) obtained at the same session were analyzed. That is, the C-DVT/DVT score of Assessment 3 and the LIADL score were used. The C-DVT/DVT scores of Assessment 4 and the PSP score were analyzed. An absolute value of an r > .40 was considered to indicate good ecological validity; <.40, poor (Rempfer, Hamera, Brown, & Cromwell, 2003; Woonings et al., 2003). The criteria of the ecological validity examination were looser than those of the concurrent validity examination. The reason was that the extent of the correlation between two tests assessing related constructs (examination of the ecological validity), in theory, should be smaller than that between two tests assessing the same constructs (examination of the concurrent validity) (Mokkink et al., 2010). Results During our study, we recruited 74 patients and excluded 24 of them due to inconsistencies in their CGI-S scores (n = 23, range of the CGI-S scores difference = −2, −1, and 1). Among the 51 patients included in this study, two patients were lost to follow-up. Finally, 49 patients completed this study, and their data were analyzed. All the participants were Asians (Taiwanese), and most were females (69.4%) with mild severity (mean score of the CGI-S = 1.4). The number of years after onset was about 19 on average (mean ± SD = 18.9 ± 8.6). Regarding the C-DVT assessments, the completion time of the participants was about 4.8 min on average, and the number of errors was very low (1st–3rd quartile = 0–3) for most of the participants. The average completion time of the DVT was about 8.5 min, and the number of errors was low (1st–3rd quartile = 0–10) for most of the participants. Table 1 shows further characteristics of the participants. Table 1. Demographic characteristics of the participants (n = 49) Characteristic  Mean ± SD  Number (%)  Gender   Male    15 (30.6)   Female    34 (69.4)  Age, years  41.3 ± 10.8    Education level   Elementary    3 (4.2)   Middle school    11 (15.3)   High school or vocational school    22 (30.6)  University    13 (18.1)  Year after onset  18.9 ± 8.6    Number of admissions  2.3 ± 2.2    Type of antipsychoticsa   First generation    7 (14.3)   Second generation    40 (81.6)  Age of onset, years  20.6 ± 6.9    Work before onseta   Service industry    7 (14.3)   Industry    6 (12.2)   Clerk    3 (6.1)   Not employed    31 (63.3)  Living situationa   Living with family    46 (93.9)   Living alone    1 (2.0)   Living in an institution    1 (2.0)  Marital status   Married    1 (1.4)   Unmarried    42 (58.3)   Divorced    5 (6.9)   Other    1 (1.4)  Severity   Mild    21 (29.2)   Moderate    26 (36.1)   Severe    2 (2.8)  CGI-S  1.4 ± 0.6    MMSE  28.4 ± 2.1    LIADL  6.8 ± 1.2    PSP  64.9 ± 7.3    C-DVT  Number of errors, median (1st–3rd quartile)   1st assessment session  1.0 (0.0–3.0)     2nd assessment session  1.0 (0.0–1.5)     3rd assessment session  1.0 (0.0–2.0)     4th assessment session  0.5 (0.0–2.0)    DVT  Number of errors, median (1st–3rd quartile)   1st assessment session  3.0 (1.0–10.0)     2nd assessment session  3.0 (1.0–7.5)     3rd assessment session  3.0 (1.0–6.0)     4th assessment session  3.0 (0.0–5.0)    Characteristic  Mean ± SD  Number (%)  Gender   Male    15 (30.6)   Female    34 (69.4)  Age, years  41.3 ± 10.8    Education level   Elementary    3 (4.2)   Middle school    11 (15.3)   High school or vocational school    22 (30.6)  University    13 (18.1)  Year after onset  18.9 ± 8.6    Number of admissions  2.3 ± 2.2    Type of antipsychoticsa   First generation    7 (14.3)   Second generation    40 (81.6)  Age of onset, years  20.6 ± 6.9    Work before onseta   Service industry    7 (14.3)   Industry    6 (12.2)   Clerk    3 (6.1)   Not employed    31 (63.3)  Living situationa   Living with family    46 (93.9)   Living alone    1 (2.0)   Living in an institution    1 (2.0)  Marital status   Married    1 (1.4)   Unmarried    42 (58.3)   Divorced    5 (6.9)   Other    1 (1.4)  Severity   Mild    21 (29.2)   Moderate    26 (36.1)   Severe    2 (2.8)  CGI-S  1.4 ± 0.6    MMSE  28.4 ± 2.1    LIADL  6.8 ± 1.2    PSP  64.9 ± 7.3    C-DVT  Number of errors, median (1st–3rd quartile)   1st assessment session  1.0 (0.0–3.0)     2nd assessment session  1.0 (0.0–1.5)     3rd assessment session  1.0 (0.0–2.0)     4th assessment session  0.5 (0.0–2.0)    DVT  Number of errors, median (1st–3rd quartile)   1st assessment session  3.0 (1.0–10.0)     2nd assessment session  3.0 (1.0–7.5)     3rd assessment session  3.0 (1.0–6.0)     4th assessment session  3.0 (0.0–5.0)    Note: CGI-S = Clinical Global Impression-Schizophrenia Scale; MMSE = Mini-Mental State Examination; ADL = activities of daily living; PSP = Personal and Social Performance Scale; C-DVT = Computerized Digit Vigilance Test; DVT = Digit Vigilance Test. aThe total percentage was not 100% because some participants’ information could not be obtained. Test–Retest Reliability Tables 2 and 3 show the results of the test–retest reliability analyses. The ICCs of the C-DVT were larger than 0.70 (ranging from 0.71 to 0.89), and those of the DVT were larger than 0.90 (ranging from 0.95 to 0.96). The Bland–Altman plots of the C-DVT and DVT are presented in Figs. 1 and 2, respectively. The plots show that the points were generally scattered randomly. The LOAs of the C-DVT were −55.7 and 69.3, and those of the DVT were −139.1 and 139.3. Table 2. Mean, SD, and ICC of the C-DVT and DVT (n = 49) Tests  Time 1  Time 2  Time 3  Time 4  ICC (95% CI)  Mean ± SD  Mean ± SD  Mean ± SD  Mean ± SD  C-DVT  287.2 ± 45.9  280.4 ± 38.5  272.6 ± 38.6  277.1 ± 44.2  0.77 (0.67–0.85)  DVT  525.7 ± 212.7  525.6 ± 217.6  502.9 ± 177.7  488.6 ± 175.2  0.93 (0.89–0.96)  Tests  Time 1  Time 2  Time 3  Time 4  ICC (95% CI)  Mean ± SD  Mean ± SD  Mean ± SD  Mean ± SD  C-DVT  287.2 ± 45.9  280.4 ± 38.5  272.6 ± 38.6  277.1 ± 44.2  0.77 (0.67–0.85)  DVT  525.7 ± 212.7  525.6 ± 217.6  502.9 ± 177.7  488.6 ± 175.2  0.93 (0.89–0.96)  Note: C-DVT = Computerized Digit Vigilance Test; DVT = Digit Vigilance Test; SD = standard deviation; CI = confidence interval; ICC = intraclass correlation coefficient (estimation of four assessments). Table 3. Parameters of test–retest agreement, random measurement error, and practice effect of the C-DVT and DVT (n = 49) Tests  Adjacent assessments  Test–retest agreement  Random measurement error  Practice effect  ICC (95% CI)  MDC  MDC%  Differencea (Mean ± SD)  Effect sizeb  pc  C-DVT  Time 1–2  0.71 (0.54–0.83)  68.5  24.1%  6.8 ± 31.9  0.21  .14  Time 2–3  0.83 (0.70–0.90)  44.0  15.9%  7.8 ± 21.7  0.36  .02  Time 3–4  0.89 (0.81–0.93)  35.5  12.9%  −4.6 ± 19.6  −0.23  .11  DVT  Time 1–2  0.95 (0.91–0.97)  156.0  29.7%  0.1 ± 71.0  0.00  .99  Time 2–3  0.95 (0.90–0.97)  134.9  26.2%  22.7 ± 61.6  0.37  .01  Time 3–4  0.96 (0.93–0.98)  110.1  22.2%  14.3 ± 47.3  0.30  .04  Tests  Adjacent assessments  Test–retest agreement  Random measurement error  Practice effect  ICC (95% CI)  MDC  MDC%  Differencea (Mean ± SD)  Effect sizeb  pc  C-DVT  Time 1–2  0.71 (0.54–0.83)  68.5  24.1%  6.8 ± 31.9  0.21  .14  Time 2–3  0.83 (0.70–0.90)  44.0  15.9%  7.8 ± 21.7  0.36  .02  Time 3–4  0.89 (0.81–0.93)  35.5  12.9%  −4.6 ± 19.6  −0.23  .11  DVT  Time 1–2  0.95 (0.91–0.97)  156.0  29.7%  0.1 ± 71.0  0.00  .99  Time 2–3  0.95 (0.90–0.97)  134.9  26.2%  22.7 ± 61.6  0.37  .01  Time 3–4  0.96 (0.93–0.98)  110.1  22.2%  14.3 ± 47.3  0.30  .04  Note: C-DVT = Computerized Digit Vigilance Test; DVT = Digit Vigilance Test; ICC = intraclass correlation coefficient (estimation of two adjacent assessments); CI = confidence interval; MDC = minimal detectable change. aDifference = Scorepre – Scorepost. bEffect size = MeanDifferences/SDDifferences. cSignificance level = 0.02 (i.e., 0.05/3). Fig. 1. View largeDownload slide The Bland–Altman plot of the C-DVT (Times 1 and 2). Fig. 1. View largeDownload slide The Bland–Altman plot of the C-DVT (Times 1 and 2). Fig. 2. View largeDownload slide The Bland–Altman plot of the DVT (Times 1 and 2). Fig. 2. View largeDownload slide The Bland–Altman plot of the DVT (Times 1 and 2). The MDC%s of the C-DVT were smaller than 20% (12.9%–24.1%), except the MDC% between Assessments 1 and 2 (24.1%). However, the MDC%s of the DVT were all larger than 20% (22.2%–29.7%) across Assessments 1 to 4. In addition, the MDCs of the C-DVT ranged from about 0.5 to 1 min, and those of the DVT ranged from about 2 to 2.5 min. The analyses of the practice effect revealed that the effect sizes of score change in the C-DVT were larger than 0.20 among Assessments 1–3 (effect size = 0.21–0.36), but smaller than 0.20 between Assessments 3 and 4 (effect size = −0.23). Conversely, the effect sizes of score change in the DVT were larger than 0.20 among Assessments 2–4 (effect size = 0.30 to 0.37), but smaller than 0.20 between Assessments 1 and 2 (effect size = 0.00). Concurrent Validity The raw scores of the C-DVT were moderately correlated with those of the DVT (r = .53–.64, p < .01). However, high correlations were found between the adjusted scores of the C-DVT and DVT (r = .75–.79, p < .01). Ecological Validity The C-DVT scores were moderately correlated with both the LIADL (r = −.44, p < .01) and PSP scores (r = −.45, p < .01). Furthermore, the DVT scores were weakly correlated with the LIADL (r = −.15, p = .31) and PSP scores (r = −.30, p = .03). Discussion The analyses of the test–retest agreement in patients with schizophrenia revealed that the ICCs of the C-DVT were higher than 0.75 (0.76–0.87) and that the ICCs of the DVT were higher than 0.90 (0.93–0.95). In addition, the Bland–Altman plots of the C-DVT and DVT showed no obvious systematic trend of bias (i.e., difference of scores) across the mean scores (e.g., the difference increases as the mean scores increases or decrease) (Bland & Altman, 1986).These results indicate that the test–retest agreement of the C-DVT is good-to-excellent, and that of the DVT is excellent, in patients with schizophrenia. The test–retest agreement of the DVT appeared better than that of the C-DVT, possibly due to the DVT’s higher ratio of between-subject variance (which can be represented by the SD2 of the scores in each assessment) to within-subject variance (which can be represented by the SD2 of the difference scores between two adjacent assessments) (Lexell & Downham, 2005). The higher ratio of the DVT (the ratio of the DVT = 9.0–14.1 vs. that of the C-DVT = 2.1–3.9) tends to increase the estimated ICCs and might have caused overestimation of the test–retest agreement of the DVT in this study (Lexell & Downham, 2005). Because the values of the ICC tend to be influenced by the variance of the test scores, we should interpret the test–retest reliability from another reliability examination, including random measurement error and practice effect. From the analyses of random measurement error, we found that most of the MDC%s of the C-DVT were smaller than 20% in patients with schizophrenia (12.9%–15.9%), except the MDC% between Assessments 1 and 2 (24.1%). However, the MDC%s of the DVT were larger than 20% (22.2%–29.7%) across adjacent assessments. These findings indicate that the C-DVT has acceptable random measurement error in patients with schizophrenia, whereas the random measurement error of the DVT was somewhat larger. Such a difference may be ascribed to the design of the C-DVT, which is intended to help the participants see the stimuli clearly (i.e., the font size of the stimuli in the C-DVT [36 pt] is larger than that in the DVT [12 pt]) (Yang et al., 2015) and requires less visual searching (i.e., the C-DVT presents only one column of 5 digits on each page, whereas the DVT presents 59 rows of 35 digits on each page). Therefore, these designs of the C-DVT may remove potential confounding influences from variability in visual acuity and visual searching skill across the participants. These findings support that those designs of the C-DVT can be helpful in reducing random measurement error. The MDC is a useful indicator for clinicians and researchers in judging whether a participant’s change score of repeated measurements is caused by random measurement error (i.e., the change score ≤ MDC) or by a real change in the participant’s ability (i.e., the change score > MDC). The MDCs of the C-DVT in patients with schizophrenia were 35.5–68.5 s, and those of the DVT were 110.1–156.0 s. These results indicate that a patient’s change score of the C-DVT will only need to be larger than 68.5 s (MDC% = 24.1%), which can be regarded as a real change in sustained attention. In contrast, the change score of the DVT should be larger than 156.0 s (MDC% = 29.7%) for users to interpret that the patient has a real change in sustained attention. Because the C-DVT had smaller MDC% than the DVT, the C-DVT appears to be a more useful test to detect real change in sustained attention in patients with schizophrenia. Therefore, our results show the potential of the C-DVT for use as an outcome measure for examining treatment effects on sustained attention in patients with schizophrenia. The analyses of the C-DVT’s practice effect showed that the effect size of Assessments 3–4 (−0.23) was smaller than 0.20 and smaller than that of Assessments 2–3 (0.36). These findings indicate that the practice effect of the C-DVT scores reached a plateau after three assessments. In contrast, the effect size of the DVT (0.30) was larger than 0.20 even between Assessments 3 and 4, which indicates that the practice effect of the DVT scores did not reach a plateau even after three assessments. The C-DVT scores reaching a plateau earlier than the DVT scores may have been due to two specific design characteristics of the C-DVT: the random presentation of the targets and the large number of practice trials before the formal testing (Yang et al., 2015). The practice effect of the DVT can cause users to overestimate improvement or underestimate deterioration of sustained attention in patients with schizophrenia. Therefore, the C-DVT appears to be a more suitable measure for repeatedly assessing sustained attention in patients with schizophrenia. In the examination of the concurrent validity, we found that the extent of the correlation between the adjusted scores (r = .75–.79) of the C-DVT and DVT was higher than that between the raw scores (r = .53–.64). This phenomenon indicates that the raw scores of the two tests might have been affected by the systematic bias of the administration order (Ibrahim et al., 2015). Therefore, the extent of the correlation between the raw scores of both tests was attenuated by the systematic bias (Ibrahim et al., 2015). In other words, the concurrent validity may have been underestimated by examination of the correlation between raw scores. On the other hand, the adjusted scores were calculated to control for the systematic bias of the administration order. Thus, the extent of correlation from the adjusted scores may more plausibly represent the concurrent validity of the C-DVT. The high correlations from the adjusted scores support that the concurrent validity of the C-DVT is good in patients with schizophrenia. The analyses of ecological validity showed that the C-DVT score was moderately correlated with the scores of both the PSP and the LIADL (r = −.43 and −.42, respectively). These results were in agreement with the results of previous studies, which reported the small-to-moderate correlations between sustained attention and both social function (r = .42–.59) (Meyer & Kurtz, 2009; Nienow, Docherty, Cohen, & Dinzeo, 2006; Ohno et al., 2000) and IADL (r = .36–.54) (Evans et al., 2003; Lipskaya et al., 2011) in patients with schizophrenia. In contrast, the DVT score was weakly correlated with both the PSP and the LIADL scores (r = −.30 and −.15, respectively). These findings indicate that the C-DVT has good ecological validity in patients with schizophrenia, but the ecological validity of the DVT was not supported. The difference in the ecological validities of the C-DVT and DVT might have been due to the larger random measurement error of the DVT (Muchinsky, 1996). The substantial amount of random measurement error of the DVT may have reduced the magnitude of the correlations between the DVT score and either the PSP score or the LIADL score. In contrast, the good ecological validity of the C-DVT may make it more appropriate than the DVT for inferring a patient’s social function and IADL function: If a patient with schizophrenia has a better score on the C-DVT, he/she is likely to have better social function and IADL function. Comparing the completion times of the C-DVT and DVT, we found that the C-DVT required only about half the completion time of the DVT in formal testing (C-DVT: 4.5–4.8 min vs. DVT: 8.1–8.8 min). The results indicate that the efficiency of the C-DVT assessment is better than that of the DVT assessment. Additionally, although the completion time of the C-DVT is shorter than that of the DVT, the completion time of the C-DVT for the participants was longer than 3 min, which is regarded as the minimum time for testing sustained attention. Therefore, the C-DVT can help users reduce the administration time for assessing sustained attention in patients with schizophrenia (Betts et al., 2006; Loetscher & Lincoln, 2013). In addition to the completion time, the total error counts of both the C-DVT and the DVT were also recorded. However, we did not analyze the error counts as a measure of the psychometric properties. The reason was that the error counts of both tests were too low (1st–3rd quartiles of the C-DVT and DVT were 0–3 and 0–10, respectively) to be analyzed as continuous scores (e.g., completion times). The low error counts of both tests may suggest that this score cannot differentiate the sustained attention abilities of patients with schizophrenia. For a user of both tests, the error counts could provide information to judge whether the C-DVT or DVT is suitable for a patient (Lewis, 1995). If a patient has more errors on the tests (e.g., more than 10 errors on the C-DVT), the patient may not have followed the testing rules. In this situation, the assessor could provide further instruction or demonstration of the testing rules for the patient. Two limitations of this study should be noted. First, the severity of the participants’ symptoms might have been unstable during the periods between Assessments 2 and 3, for we assessed the severity of illness (CGI-S) only at Assessments 1 and 4. The unstable symptom severities might have caused underestimation of the test–retest reliability of the C-DVT and the DVT. Second, the demographic homogeneity of our participants may have impeded the generalization of our findings to the general population of schizophrenia. All the participants were Asians, and most had mild severity of illness and a long period after onset. Further studies should recruit patients with different demographic characteristics to cross validate the psychometric properties of the C-DVT. In conclusion, our results showed that the C-DVT had acceptable test–retest reliability, good concurrent validity, and good ecological validity in patients with schizophrenia. These findings indicate that the C-DVT has the potential to become a reliable and valid test of sustained attention in patients with schizophrenia. Funding This study was supported by a research grant from the Taipei City Hospital (TPCH-104-059). The funding source had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication. Conflict of interest No authors report any potential conflicts of interest. Acknowledgments We thank all of the participants and researchers for their generosity of time and effort to participate in this study. References APA. ( 2013). Diagnostic and statistical manual of mental disorders (DSM-5)  ( 5th ed.). Washington, DC: American Psychiatric Publishing. Bartko, J. J. ( 1966). The intraclass correlation coefficient as a measure of reliability. Psychological Reports , 19, 3– 11. Google Scholar CrossRef Search ADS PubMed  Betts, J., Mckay, J., Maruff, P., & Anderson, V. ( 2006). The development of sustained attention in children: The effect of age and task load. Child Neuropsychology , 12, 205– 221. Google Scholar CrossRef Search ADS PubMed  Bland, J. M., & Altman, D. ( 1986). Statistical methods for assessing agreement between two methods of clinical measurement. The Lancet , 327, 307– 310. Google Scholar CrossRef Search ADS   Bushnell, C. D., Johnston, D. C., & Goldstein, L. B. ( 2001). Retrospective assessment of initial stroke severity comparison of the NIH Stroke Scale and the Canadian Neurological Scale. Stroke; A Journal of Cerebral Circulation , 32, 656– 660. Google Scholar CrossRef Search ADS   Chan, M. W. C., Yip, J. T. H., & Lee, T. M. C. ( 2004). Differential impairment on measures of attention in patients with paranoid and nonparanoid schizophrenia. Journal of Psychiatric Research , 38, 145– 152. Google Scholar CrossRef Search ADS PubMed  Chaytor, N., & Schmitter-Edgecombe, M. ( 2003). The ecological validity of neuropsychological tests: A review of the literature on everyday cognitive skills. Neuropsychology Review , 13, 181– 197. Google Scholar CrossRef Search ADS PubMed  Chiu, E. C., Koh, C. L., Tsai, C. Y., Lu, W. S., Sheu, C. F., Hsueh, I. P., et al.  . ( 2014). Practice effects and test–re-test reliability of the Five Digit Test in patients with stroke over four serial assessments. Brain Injury , 28, 1726– 1733. Google Scholar CrossRef Search ADS PubMed  Cohen, J. ( 1988). Statistical power analysis for the behavioral sciences  ( 2nd ed.). Hillsdale, NJ: Lawrence Erlbaum Associates. Depp, C. A., Moore, D. J., Sitzer, D., Palmer, B. W., Eyler, L. T., Roesch, S., et al.  . ( 2007). Neurocognitive impairment in middle-aged and older adults with bipolar disorder: Comparison to schizophrenia and normal comparison subjects. Journal of Affective Disorders , 101, 201– 209. Google Scholar CrossRef Search ADS PubMed  Doros, G., & Lew, R. ( 2010). Design based on intra-class correlation coefficients. American Journal of Biostatistics , 1 ( 1), 1– 8. Evans, J. D., Heaton, R. K., Paulsen, J. S., Palmer, B. W., Patterson, T., & Jeste, D. V. ( 2003). The relationship of neuropsychological abilities to specific domains of functional capacity in older schizophrenia patients. Biological Psychiatry , 53, 422– 430. Google Scholar CrossRef Search ADS PubMed  Green, M. F. ( 1996). What are the functional consequences of neurocognitive deficits in schizophrenia? American Journal of Psychiatry , 153, 321– 330. Google Scholar CrossRef Search ADS PubMed  Haley, S. M., & Fragala-Pinkham, M. A. ( 2006). Interpreting change scores of tests and measures used in physical therapy. Physical Therapy , 86, 735– 743. Google Scholar PubMed  Hambleton, R. K. ( 2000). Response to Hays et al. and McHorney and Cohen: Emergence of item response modeling in instrument development and data analysis. Medical Care , 38, II-60– II-65. Google Scholar CrossRef Search ADS   Haro, J., Kamath, S., Ochoa, S., Novick, D., Rele, K., Fargas, A., et al.  . ( 2003). The Clinical Global Impression–Schizophrenia scale: A simple instrument to measure the diversity of symptoms present in schizophrenia. Acta Psychiatrica Scandinavica , 107, 16– 23. Google Scholar CrossRef Search ADS   Huang, S. L., Hsieh, C. L., Wu, R. M., Tai, C. H., Lin, C. H., & Lu, W. S. ( 2011). Minimal detectable change of the Timed “Up & Go” Test and the Dynamic Gait Index in people with Parkinson disease. Physical Therapy , 91, 114– 121. Google Scholar CrossRef Search ADS PubMed  Ibrahim, I., Tobar, S., Elassy, M., Mansour, H., Chen, K., Wood, J., et al.  . ( 2015). Practice effects distort translational validity estimates for a Neurocognitive Battery. Journal of Clinical and Experimental Neuropsychology , 37, 530– 537. Google Scholar CrossRef Search ADS PubMed  La Porta, F., Franceschini, M., Caselli, S., Susassi, S., Cavallini, P., & Tennant, A. ( 2011). Unified Balance Scale: Classic psychometric and clinical properties. Journal of Rehabilitation Medicine , 43, 445– 453. Google Scholar CrossRef Search ADS PubMed  Lahiri, D., & Singh, A. R. S. ( 2016). Attentional impairment and minor physical anomalies in early onset schizophrenia. Eastern Journal of Psychiatry , 17, 2– 11. Lawton, M. P., & Brody, E. M. ( 1969). Assessment of older people: Self-maintaining and instrumental activities of daily living. The Gerontologist , 9, 179– 186. Google Scholar CrossRef Search ADS PubMed  Lee, P., Li, P. C., Liu, C. H., & Hsieh, C. L. ( 2011). Test–retest reliability of two attention tests in schizophrenia. Archives of Clinical Neuropsychology , 26, 405– 411. Google Scholar CrossRef Search ADS PubMed  Lewis, R. F. ( 1995). Digit vigilance test . Odessa, FL: Psychological Assessment Resources. Lexell, J. E., & Downham, D. Y. ( 2005). How to assess the reliability of measurements in rehabilitation. American Journal of Physical Medicine and Rehabilitation , 84, 719– 723. Google Scholar CrossRef Search ADS PubMed  Lipskaya, L., Jarus, T., & Kotler, M. ( 2011). Influence of cognition and symptoms of schizophrenia on IADL performance. Scandinavian Journal of Occupational Therapy , 18, 180– 187. Google Scholar CrossRef Search ADS PubMed  Liu, S. K., Chiu, C. H., Chang, C. J., Hwang, T. J., Hwu, H. G., & Chen, W. J. ( 2002). Deficits in sustained attention in schizophrenia and affective disorders: Stable versus state-dependent markers. American Journal of Psychiatry , 159, 975– 982. Google Scholar CrossRef Search ADS PubMed  Loetscher, T., & Lincoln, N. B. ( 2013). Cognitive rehabilitation for attention deficits following stroke. Cochrane Database of Systematic Reviews , 4, CD008391. Meyer, M. B., & Kurtz, M. M. ( 2009). Elementary neurocognitive function, facial affect recognition and social-skills in schizophrenia. Schizophrenia Research , 110, 173– 179. Google Scholar CrossRef Search ADS PubMed  Mokkink, L. B., Terwee, C. B., Patrick, D. L., Alonso, J., Stratford, P. W., Knol, D. L., et al.  . ( 2010). The COSMIN study reached international consensus on taxonomy, terminology, and definitions of measurement properties for health-related patient-reported outcomes. Journal of Clinical Epidemiology , 63, 737– 745. Google Scholar CrossRef Search ADS PubMed  Morosini, P., Magliano, L., Brambilla, L., Ugolini, S., & Pioli, R. ( 2000). Development, reliability and acceptability of a new version of the DSM‐IV Social and Occupational Functioning Assessment Scale (SOFAS) to assess routine social funtioning. Acta Psychiatrica Scandinavica , 101, 323– 329. Google Scholar PubMed  Muchinsky, P. M. ( 1996). The correction for attenuation. Educational and Psychological Measurement , 56, 63– 75. Google Scholar CrossRef Search ADS   Nienow, T. M., Docherty, N. M., Cohen, A. S., & Dinzeo, T. J. ( 2006). Attentional dysfunction, social perception, and social competence: What is the nature of the relationship? Journal of Abnormal Psychology , 115, 408– 417. Google Scholar CrossRef Search ADS PubMed  Ohno, T., Ikebuchi, E., Henomatsu, K., Kasai, K., Nakagome, K., Iwanami, A., et al.  . ( 2000). Psychophysiological correlates of social skills deficits in persons with schizophrenia. Psychiatry Research , 100, 155– 167. Google Scholar CrossRef Search ADS PubMed  Raghavan, D. V., Shanmugiah, A., Bharathi, P., & Jeyaprakash, R. ( 2016). P300 and neuropsychological measurements in patients with schizophrenia and their healthy biological siblings. Indian Journal of Psychiatry , 58, 454. Google Scholar CrossRef Search ADS PubMed  Rempfer, M. V., Hamera, E. K., Brown, C. E., & Cromwell, R. L. ( 2003). The relations between cognition and the independent living skill of shopping in people with schizophrenia. Psychiatry Research , 117, 103– 112. Google Scholar CrossRef Search ADS PubMed  Salter, K., Jutai, J. W., Teasell, R., Foley, N. C., Bitensky, J., & Bayley, M. ( 2005). Issues for selection of outcome measures in stroke rehabilitation: ICF activity. Disability and Rehabilitation , 27, 315– 340. Google Scholar CrossRef Search ADS PubMed  Sohlberg, M. M., & Mateer, C. A. ( 1987). Effectiveness of an attention-training program. Journal of Clinical and Experimental Neuropsychology , 9, 117– 130. Google Scholar CrossRef Search ADS PubMed  Woonings, F. M. J., Appelo, M. T., Kluiter, H., Slooff, C. J., & van den Bosch, R. J. ( 2003). Learning (potential) and social functioning in schizophrenia. Schizophrenia Research , 59, 287– 296. Google Scholar CrossRef Search ADS PubMed  Yang, C. M., Lin, G. H., Chen, M. H., Hsueh, I. P., & Hsieh, C. L. ( 2015). Development of a Computerized Digit Vigilance Test and validation in patients with stroke. Journal of Rehabilitation Medicine , 47, 311– 317. Google Scholar CrossRef Search ADS PubMed  © The Author 2017. Published by Oxford University Press. All rights reserved. For permissions, please e-mail: journals.permissions@oup.com. http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Archives of Clinical Neuropsychology Oxford University Press

A Reliable and Valid Assessment of Sustained Attention for Patients With Schizophrenia: The Computerized Digit Vigilance Test

Loading next page...
 
/lp/ou_press/a-reliable-and-valid-assessment-of-sustained-attention-for-patients-RIg8TH0Ol2
Publisher
Oxford University Press
Copyright
© The Author 2017. Published by Oxford University Press. All rights reserved. For permissions, please e-mail: journals.permissions@oup.com.
ISSN
0887-6177
eISSN
1873-5843
D.O.I.
10.1093/arclin/acx064
Publisher site
See Article on Publisher Site

Abstract

Abstract Objective The purposes of this study were to examine the test–retest reliability, concurrent validity, and ecological validity of the Computerized Digit Vigilance Test (C-DVT) in patients with schizophrenia. Method Each participant was assessed four times, with 1-week intervals. In each assessment, the participants completed both the C-DVT and the original DVT. The participants were also assessed using the Lawton Instrumental Activities of Daily Living Scale (LIADL) and the Personal and Social Performance Scale (PSP). Results Forty-nine participants were recruited in this study. The results showed that the test–retest agreement of the C-DVT was good-to-excellent (intraclass correlation coefficient = 0.71–0.89). The random measurement errors of the C-DVT were acceptable (percentages of minimal detectable change = 12.9%–24.1%). The practice effect of the C-DVT reached a plateau after three assessments (effect size <0.20). The concurrent validity of the C-DVT was good (r = .75–.79 with DVT) when we controlled for the randomized administration order of the two tests. The ecological validity of the C-DVT was good (r = −.44 with the LIADL; r = −.45 with the PSP). Conclusions The C-DVT had acceptable test–retest reliability, sound concurrent validity, and sound ecological validity in patients with schizophrenia. These findings indicate that the C-DVT has the potential to be a reliable and valid test of sustained attention in patients with schizophrenia. Sustained attention, Psychometric property, Schizophrenia, Cognition Introduction Sustained attention can be defined as the ability to maintain consistent behavioral performance of a task for a reasonable duration of time (at least 3 min) (Betts, Mckay, Maruff, & Anderson, 2006; Loetscher & Lincoln, 2013; Sohlberg & Mateer, 1987). Sustained attention is a fundamental cognitive function involved in many daily activities, such as reading, writing, or working for a long time. Sustained attention deficit appears to be a common deficit in patients with schizophrenia (Liu et al., 2002). The deficit of sustained attention in patients with schizophrenia is associated with difficulties in social functioning (e.g., interpersonal communication and social activities) and instrumental activities of daily living (IADL, e.g., shopping, financial management, and taking medicine) (Evans et al., 2003; Green, 1996; Lipskaya, Jarus, & Kotler, 2011; Woonings, Appelo, Kluiter, Slooff, & van den Bosch, 2003). Because deficits of sustained attention are closely associated with the functional outcomes of patients with schizophrenia, the assessment and management of sustained attention deficits are crucial for clinicians in helping patients live independently in the community. The Digit Vigilance Test (DVT) is an often-used sustained attention test in patient with schizophrenia (Chan, Yip, & Lee, 2004; Depp et al., 2007; Lahiri & Singh, 2016; Raghavan, Shanmugiah, Bharathi, & Jeyaprakash, 2016). However, a previous study has indicated that the practice effect and random measurement error of the DVT are substantial in patients with schizophrenia (Lee, Li, Liu, & Hsieh, 2011). These drawbacks can cause inconsistencies in the results of the DVT across repeated assessments (Lee et al., 2011). The inconsistencies in the results may lead to misinterpretation of the performance changes in sustained attention even when subjects’ function of sustained attention may have been stable. To assess sustained attention with a limited practice effect and random measurement error, the Computerized Digit Vigilance Test (C-DVT) was developed based on the DVT (Yang, Lin, Chen, Hsueh, & Hsieh, 2015). The proposed C-DVT in the current study incorporated a design for reducing the practice effect: increasing the number of practice trials (Yang et al., 2015). This design helped subjects become fully familiar with the operation rules and methods of the C-DVT (Yang et al., 2015) before they performed the formal testing. To reduce random measurement error caused by patients’ poor visual acuity and visual scanning, the C-DVT adopted a large font size for the stimuli and fewer stimuli (one column of five stimuli) on each page. Because of the aforementioned features of the C-DVT, the C-DVT has the potential to assess sustained attention with a smaller practice effect and smaller random measurement error for patients with schizophrenia. The sound concurrent validity of the C-DVT has been validated in patients with stroke, which supports the C-DVT as a measure of sustained attention (Yang et al., 2015). Additionally, the test–retest reliability and ecological validity of the C-DVT are also sufficient in patients (Yang et al., 2015). However, such positive findings cannot be assumed for patients with schizophrenia because psychometric properties are population dependent (Hambleton, 2000). Thus, the psychometric properties of the C-DVT in patients with schizophrenia remain unknown. The purposes of this study were to examine the test–retest reliability, concurrent validity, and ecological validity of the C-DVT in patients with schizophrenia. Methods Participants We recruited patients by convenience sampling from two community-based psychiatric rehabilitation day centers affiliated with a medical center in northern Taiwan. Patients were included in this study if they met the following criteria: (1) diagnosis of schizophrenia according to the criteria of the Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition (APA, 2013), (2) age ≥ 20 years, and (3) score of the Mini-Mental Status Examination (MMSE) ≥21 points. The exclusion criteria of this study were patients with (1) diagnosis of other neurological or psychiatric diseases affecting cognition (e.g., stroke or depression), (2) another severe medical condition or psychiatric disorder that required treatment during study, or (3) unstable severity of illness [specifically, the scores of the Clinical Global Impressions Scale-Severity (CGI-S) were different between the first and last assessments of the study]. This study was approved by the Institutional Review Board of the medical center. All participants signed consent forms before participating in this study. The sample size needed for the test–retest reliability analysis was estimated using power analysis with alpha = 0.05 and power = 0.80 (Doros & Lew, 2010). In addition, we set the width of the 95% confidence interval of the ICC at ≤0.30 and the ICC of the C-DVT at ≥0.70 (Yang et al., 2015). The power analysis using the aforementioned parameters showed that the appropriate sample size was 50 participants. Procedure This study contained four assessment sessions (referred to as Assessments 1–4) with 1-week intervals between two adjacent sessions. The participants completed both the C-DVT and DVT in all four assessment sessions. The testing order of the C-DVT and DVT was randomized for all participants and every assessment session. Between the administrations of the C-DVT and DVT, the participants rested for 3 min. We also administered the Lawton Instrumental Activities of Daily Living Scale (LIADL) in Assessment 3, the Personal and Social Performance Scale (PSP) in Assessment 4, and the CGI-S twice, in Assessments 1 and 4. One occupational therapist assessed all participants across four assessment sessions. In addition, we collected the patients’ demographic characteristics from chart review. Measures C-DVT (Yang et al., 2015): The C-DVT, which is based on the DVT, can be used to assess sustained attention (Yang et al., 2015). The C-DVT contains 28 trials in a practice session and 120 trials in a formal test. The operation interface of the C-DVT includes a computer screen and an external keyboard with only two buttons [a circle (“O”) and an X (“X”)]. When performing the C-DVT, subjects have to judge whether the screen displays the digit “6” in a column of 5 digits. If the screen displays the digit “6”, subjects should press the button with a circle; otherwise, subjects should press the button with an X. The C-DVT automatically records the total time (in s) needed for completing the test and the number of errors. The total time for completing the C-DVT is used as a primary index for evaluating participants’ ability of sustained attention. A shorter time for completing the C-DVT indicates better sustained attention. DVT (Lewis, 1995): The DVT is presented as a two-page test sheet containing 59 rows of 35 digits (i.e., 0–9) in a 12-point (pt) font size. To perform the DVT, subjects have to cross out all occurrences of the digit “6” on the test sheets. The rater of the DVT records the total time (in s) needed to complete the test sheet and the number of errors. The total time for completing the DVT is treated as a primary index of sustained attention. In this study, we used the DVT as an external criterion for examining the concurrent validity of the C-DVT. LIADL (Lawton & Brody, 1969): The LIADL contains eight items, which assess patients’ performance on eight IADL tasks (i.e., using a telephone, shopping, food preparation, housekeeping, laundry, transportation, responsibility for own medications, and ability to handle finances). The LIADL is commonly administered by face-to-face interview, and the items of the LIADL are scored according to whether a participant can independently perform the IADL tasks. The score range of the LIADL is 0–23. A higher score represents better IADL function. We used the LIADL as an external criterion for examining the ecological validity of the C-DVT. PSP (Morosini, Magliano, Brambilla, Ugolini, & Pioli, 2000): The PSP measures personal and social function in four sub-domains: socially useful activities (e.g., work and study), personal and social relationships, self-care, and disturbing and aggressive behaviors (Morosini et al., 2000). The PSP is rated through observing a participant’s performance on four sub-domains. The score of a sub-domain is scored on a 6-point scale (from 1: absent disability to 6: very severe disability). A total score (0–100) is transformed from the scores of the sub-domains. A higher total score represents better personal and social function. We used the PSP as an external criterion for examining the ecological validity of the C-DVT. CGI-S (Haro et al., 2003): The CGI-S assesses severity of psychiatric illness on a 7-point scale (1–7). The rating of the CGI-S is based on observed and self-reported symptoms, behavior, and function within 1 week. One point of the CGI-S represents that a patient is not ill, and 7 points represents that a patient is most severely ill. We used the CGI-S for examining whether the illness severity of the participants was stable during the study. Data Analysis Test–retest reliability Test–retest reliability has been defined as the extent to which repeated test scores of patients, who have not changed in the characteristic to be measured, are consistent (Mokkink et al., 2010). To examine test–retest reliability, we analyzed the test–retest agreement, random measurement error, and practice effect of the C-DVT and DVT in patients with schizophrenia. To examine test–retest agreement of the C-DVT and DVT, intraclass correlation coefficients (ICCs) (Bartko, 1966) were used for two contexts: the ICC across four repeated assessments and the ICC of two adjacent assessments (i.e., Assessment 1 vs. Assessment 2, Assessment 2 vs. Assessment 3, and Assessment 3 vs. Assessment 4). The purpose of calculating the ICC across four repeated assessments was to indicate the overall test–retest agreement of the C-DVT and DVT. The purpose of calculating the ICC of two adjacent assessments was to indicate the change in test–retest agreement as the number of assessment sessions increased. The ICC was defined as the ratio of between-subject variance to total variance (between-subject variance + within-subject variance), and was calculated based on a two-way random model of analysis of variance (Bartko, 1966). An ICC value > 0.80 was considered to indicate excellent test–retest agreement; 0.60–0.79, good agreement; 0.40–0.59, moderate; and <0.40, poor (Bushnell, Johnston, & Goldstein, 2001). To visually assess each participant’s agreements of the C-DVT and DVT scores, we drew Bland–Altman plots using each participant’s mean score of the first two assessments against the difference of scores between the two assessments (Bland & Altman, 1986). The limits of agreement (LOAs) were calculated to illustrate the width of distribution of the score differences using the following formula: the mean difference between assessments ± 1.96 × standard deviation of the difference between assessments. We calculated the minimal detectable change (MDC) and MDC percentage (MDC%) to examine the random measurement error of the C-DVT and DVT. The MDC was calculated using the following formula (Haley & Fragala-Pinkham, 2006):   MDC=1.96×SDfirst assessment of each 2 adjacent assessments×2×(1-ICC2 adjacent assessments) The MDC estimated the amount of a score change that could be considered a real change (beyond the score change caused by random measurement error) at the 95% confidence level. The MDC% was calculated by dividing the MDC by the mean scores of two adjacent assessments and then multiplying by 100%. An MDC% below 20% was considered acceptable random measurement error (Huang et al., 2011). To examine the practice effects of the C-DVT and DVT, we analyzed the mean differences between each two adjacent assessments using both paired t-test and effect size. The significance level, alpha, of the paired t-test was set at 0.017 (0.05/3 times of comparisons) for reducing Type I errors of multiple comparisons between each two-adjacent assessment of the four assessment sessions. Furthermore, the effect size was calculated by dividing the mean score differences of each two adjacent assessments by the SD of the score differences. An effect size >0.80 was considered to indicate a large practice effect; 0.50–0.79, medium; 0.20–0.49, small; and <0.20, negligible (Cohen, 1988). In addition, we examined whether the practice effect had reached a plateau phase by the following two criteria (Chiu et al., 2014): (1) the effect size of two adjacent assessments was smaller than 0.20 (e.g., whether the effect size between Assessments 2–3 was smaller than 0.20) and (2) the later effect size of two adjacent effect sizes was smaller than the previous practice effect (e.g., whether the effect size of Assessments 2–3 was smaller than that of Assessments 1–2). Concurrent validity Concurrent validity can be defined as the extent of correlation between a new or revised test (e.g., the C-DVT) and its original test (e.g., the DVT) (La Porta et al., 2011). We examined the concurrent validity of the C-DVT by analyzing the levels of correlation between the C-DVT and DVT scores across the four assessment sessions using the Pearson’s correlation coefficient (r). However, the results of the concurrent validity examination might have been attenuated by systematic bias due to the study design (Ibrahim et al., 2015). We designed the random administration order of the C-DVT and DVT for each participant at each session. Therefore, about half of the participants completed the C-DVT first, and the other half of the participants completed the DVT first. The random administration order of the two tests might have caused systematic bias. For example, because both tests instruct participants to detect the digit “6”, participants might have become more familiar with detecting the digit “6” after performing the first test. Thus, the results of the second test might have been improved by the experience of performing the first test (Ibrahim et al., 2015). To control for the possible systematic bias so as to examine the concurrent validity, we adopted the methods proposed by Ibrahim and colleagues (2015). First, we adjusted the scores of the second tests by conducting linear regression models to estimate the amounts of change score caused by systematic bias. Then we used the estimated change score to adjust the scores of the second tests (i.e., the change score was added to the scores of the second test for each participant). Second, we used the raw scores and adjusted scores to analyze the levels of correlation between the C-DVT and DVT across four assessment sessions. An absolute value of r > .75 was considered to indicate high concurrent validity; .40–.74, acceptable; and <.40, poor (Salter et al., 2005). Ecological validity Ecological validity indicates the extent of correlation between a patient’s performance on a test and a patient’s daily function (Chaytor & Schmitter-Edgecombe, 2003). In addition, we examined the ecological validity of the C-DVT and DVT by analyzing the levels of correlation between the C-DVT/DVT and both the LIADL and PSP using Pearson’s r. The C-DVT/DVT score and the LIADL score (or the PSP score) obtained at the same session were analyzed. That is, the C-DVT/DVT score of Assessment 3 and the LIADL score were used. The C-DVT/DVT scores of Assessment 4 and the PSP score were analyzed. An absolute value of an r > .40 was considered to indicate good ecological validity; <.40, poor (Rempfer, Hamera, Brown, & Cromwell, 2003; Woonings et al., 2003). The criteria of the ecological validity examination were looser than those of the concurrent validity examination. The reason was that the extent of the correlation between two tests assessing related constructs (examination of the ecological validity), in theory, should be smaller than that between two tests assessing the same constructs (examination of the concurrent validity) (Mokkink et al., 2010). Results During our study, we recruited 74 patients and excluded 24 of them due to inconsistencies in their CGI-S scores (n = 23, range of the CGI-S scores difference = −2, −1, and 1). Among the 51 patients included in this study, two patients were lost to follow-up. Finally, 49 patients completed this study, and their data were analyzed. All the participants were Asians (Taiwanese), and most were females (69.4%) with mild severity (mean score of the CGI-S = 1.4). The number of years after onset was about 19 on average (mean ± SD = 18.9 ± 8.6). Regarding the C-DVT assessments, the completion time of the participants was about 4.8 min on average, and the number of errors was very low (1st–3rd quartile = 0–3) for most of the participants. The average completion time of the DVT was about 8.5 min, and the number of errors was low (1st–3rd quartile = 0–10) for most of the participants. Table 1 shows further characteristics of the participants. Table 1. Demographic characteristics of the participants (n = 49) Characteristic  Mean ± SD  Number (%)  Gender   Male    15 (30.6)   Female    34 (69.4)  Age, years  41.3 ± 10.8    Education level   Elementary    3 (4.2)   Middle school    11 (15.3)   High school or vocational school    22 (30.6)  University    13 (18.1)  Year after onset  18.9 ± 8.6    Number of admissions  2.3 ± 2.2    Type of antipsychoticsa   First generation    7 (14.3)   Second generation    40 (81.6)  Age of onset, years  20.6 ± 6.9    Work before onseta   Service industry    7 (14.3)   Industry    6 (12.2)   Clerk    3 (6.1)   Not employed    31 (63.3)  Living situationa   Living with family    46 (93.9)   Living alone    1 (2.0)   Living in an institution    1 (2.0)  Marital status   Married    1 (1.4)   Unmarried    42 (58.3)   Divorced    5 (6.9)   Other    1 (1.4)  Severity   Mild    21 (29.2)   Moderate    26 (36.1)   Severe    2 (2.8)  CGI-S  1.4 ± 0.6    MMSE  28.4 ± 2.1    LIADL  6.8 ± 1.2    PSP  64.9 ± 7.3    C-DVT  Number of errors, median (1st–3rd quartile)   1st assessment session  1.0 (0.0–3.0)     2nd assessment session  1.0 (0.0–1.5)     3rd assessment session  1.0 (0.0–2.0)     4th assessment session  0.5 (0.0–2.0)    DVT  Number of errors, median (1st–3rd quartile)   1st assessment session  3.0 (1.0–10.0)     2nd assessment session  3.0 (1.0–7.5)     3rd assessment session  3.0 (1.0–6.0)     4th assessment session  3.0 (0.0–5.0)    Characteristic  Mean ± SD  Number (%)  Gender   Male    15 (30.6)   Female    34 (69.4)  Age, years  41.3 ± 10.8    Education level   Elementary    3 (4.2)   Middle school    11 (15.3)   High school or vocational school    22 (30.6)  University    13 (18.1)  Year after onset  18.9 ± 8.6    Number of admissions  2.3 ± 2.2    Type of antipsychoticsa   First generation    7 (14.3)   Second generation    40 (81.6)  Age of onset, years  20.6 ± 6.9    Work before onseta   Service industry    7 (14.3)   Industry    6 (12.2)   Clerk    3 (6.1)   Not employed    31 (63.3)  Living situationa   Living with family    46 (93.9)   Living alone    1 (2.0)   Living in an institution    1 (2.0)  Marital status   Married    1 (1.4)   Unmarried    42 (58.3)   Divorced    5 (6.9)   Other    1 (1.4)  Severity   Mild    21 (29.2)   Moderate    26 (36.1)   Severe    2 (2.8)  CGI-S  1.4 ± 0.6    MMSE  28.4 ± 2.1    LIADL  6.8 ± 1.2    PSP  64.9 ± 7.3    C-DVT  Number of errors, median (1st–3rd quartile)   1st assessment session  1.0 (0.0–3.0)     2nd assessment session  1.0 (0.0–1.5)     3rd assessment session  1.0 (0.0–2.0)     4th assessment session  0.5 (0.0–2.0)    DVT  Number of errors, median (1st–3rd quartile)   1st assessment session  3.0 (1.0–10.0)     2nd assessment session  3.0 (1.0–7.5)     3rd assessment session  3.0 (1.0–6.0)     4th assessment session  3.0 (0.0–5.0)    Note: CGI-S = Clinical Global Impression-Schizophrenia Scale; MMSE = Mini-Mental State Examination; ADL = activities of daily living; PSP = Personal and Social Performance Scale; C-DVT = Computerized Digit Vigilance Test; DVT = Digit Vigilance Test. aThe total percentage was not 100% because some participants’ information could not be obtained. Test–Retest Reliability Tables 2 and 3 show the results of the test–retest reliability analyses. The ICCs of the C-DVT were larger than 0.70 (ranging from 0.71 to 0.89), and those of the DVT were larger than 0.90 (ranging from 0.95 to 0.96). The Bland–Altman plots of the C-DVT and DVT are presented in Figs. 1 and 2, respectively. The plots show that the points were generally scattered randomly. The LOAs of the C-DVT were −55.7 and 69.3, and those of the DVT were −139.1 and 139.3. Table 2. Mean, SD, and ICC of the C-DVT and DVT (n = 49) Tests  Time 1  Time 2  Time 3  Time 4  ICC (95% CI)  Mean ± SD  Mean ± SD  Mean ± SD  Mean ± SD  C-DVT  287.2 ± 45.9  280.4 ± 38.5  272.6 ± 38.6  277.1 ± 44.2  0.77 (0.67–0.85)  DVT  525.7 ± 212.7  525.6 ± 217.6  502.9 ± 177.7  488.6 ± 175.2  0.93 (0.89–0.96)  Tests  Time 1  Time 2  Time 3  Time 4  ICC (95% CI)  Mean ± SD  Mean ± SD  Mean ± SD  Mean ± SD  C-DVT  287.2 ± 45.9  280.4 ± 38.5  272.6 ± 38.6  277.1 ± 44.2  0.77 (0.67–0.85)  DVT  525.7 ± 212.7  525.6 ± 217.6  502.9 ± 177.7  488.6 ± 175.2  0.93 (0.89–0.96)  Note: C-DVT = Computerized Digit Vigilance Test; DVT = Digit Vigilance Test; SD = standard deviation; CI = confidence interval; ICC = intraclass correlation coefficient (estimation of four assessments). Table 3. Parameters of test–retest agreement, random measurement error, and practice effect of the C-DVT and DVT (n = 49) Tests  Adjacent assessments  Test–retest agreement  Random measurement error  Practice effect  ICC (95% CI)  MDC  MDC%  Differencea (Mean ± SD)  Effect sizeb  pc  C-DVT  Time 1–2  0.71 (0.54–0.83)  68.5  24.1%  6.8 ± 31.9  0.21  .14  Time 2–3  0.83 (0.70–0.90)  44.0  15.9%  7.8 ± 21.7  0.36  .02  Time 3–4  0.89 (0.81–0.93)  35.5  12.9%  −4.6 ± 19.6  −0.23  .11  DVT  Time 1–2  0.95 (0.91–0.97)  156.0  29.7%  0.1 ± 71.0  0.00  .99  Time 2–3  0.95 (0.90–0.97)  134.9  26.2%  22.7 ± 61.6  0.37  .01  Time 3–4  0.96 (0.93–0.98)  110.1  22.2%  14.3 ± 47.3  0.30  .04  Tests  Adjacent assessments  Test–retest agreement  Random measurement error  Practice effect  ICC (95% CI)  MDC  MDC%  Differencea (Mean ± SD)  Effect sizeb  pc  C-DVT  Time 1–2  0.71 (0.54–0.83)  68.5  24.1%  6.8 ± 31.9  0.21  .14  Time 2–3  0.83 (0.70–0.90)  44.0  15.9%  7.8 ± 21.7  0.36  .02  Time 3–4  0.89 (0.81–0.93)  35.5  12.9%  −4.6 ± 19.6  −0.23  .11  DVT  Time 1–2  0.95 (0.91–0.97)  156.0  29.7%  0.1 ± 71.0  0.00  .99  Time 2–3  0.95 (0.90–0.97)  134.9  26.2%  22.7 ± 61.6  0.37  .01  Time 3–4  0.96 (0.93–0.98)  110.1  22.2%  14.3 ± 47.3  0.30  .04  Note: C-DVT = Computerized Digit Vigilance Test; DVT = Digit Vigilance Test; ICC = intraclass correlation coefficient (estimation of two adjacent assessments); CI = confidence interval; MDC = minimal detectable change. aDifference = Scorepre – Scorepost. bEffect size = MeanDifferences/SDDifferences. cSignificance level = 0.02 (i.e., 0.05/3). Fig. 1. View largeDownload slide The Bland–Altman plot of the C-DVT (Times 1 and 2). Fig. 1. View largeDownload slide The Bland–Altman plot of the C-DVT (Times 1 and 2). Fig. 2. View largeDownload slide The Bland–Altman plot of the DVT (Times 1 and 2). Fig. 2. View largeDownload slide The Bland–Altman plot of the DVT (Times 1 and 2). The MDC%s of the C-DVT were smaller than 20% (12.9%–24.1%), except the MDC% between Assessments 1 and 2 (24.1%). However, the MDC%s of the DVT were all larger than 20% (22.2%–29.7%) across Assessments 1 to 4. In addition, the MDCs of the C-DVT ranged from about 0.5 to 1 min, and those of the DVT ranged from about 2 to 2.5 min. The analyses of the practice effect revealed that the effect sizes of score change in the C-DVT were larger than 0.20 among Assessments 1–3 (effect size = 0.21–0.36), but smaller than 0.20 between Assessments 3 and 4 (effect size = −0.23). Conversely, the effect sizes of score change in the DVT were larger than 0.20 among Assessments 2–4 (effect size = 0.30 to 0.37), but smaller than 0.20 between Assessments 1 and 2 (effect size = 0.00). Concurrent Validity The raw scores of the C-DVT were moderately correlated with those of the DVT (r = .53–.64, p < .01). However, high correlations were found between the adjusted scores of the C-DVT and DVT (r = .75–.79, p < .01). Ecological Validity The C-DVT scores were moderately correlated with both the LIADL (r = −.44, p < .01) and PSP scores (r = −.45, p < .01). Furthermore, the DVT scores were weakly correlated with the LIADL (r = −.15, p = .31) and PSP scores (r = −.30, p = .03). Discussion The analyses of the test–retest agreement in patients with schizophrenia revealed that the ICCs of the C-DVT were higher than 0.75 (0.76–0.87) and that the ICCs of the DVT were higher than 0.90 (0.93–0.95). In addition, the Bland–Altman plots of the C-DVT and DVT showed no obvious systematic trend of bias (i.e., difference of scores) across the mean scores (e.g., the difference increases as the mean scores increases or decrease) (Bland & Altman, 1986).These results indicate that the test–retest agreement of the C-DVT is good-to-excellent, and that of the DVT is excellent, in patients with schizophrenia. The test–retest agreement of the DVT appeared better than that of the C-DVT, possibly due to the DVT’s higher ratio of between-subject variance (which can be represented by the SD2 of the scores in each assessment) to within-subject variance (which can be represented by the SD2 of the difference scores between two adjacent assessments) (Lexell & Downham, 2005). The higher ratio of the DVT (the ratio of the DVT = 9.0–14.1 vs. that of the C-DVT = 2.1–3.9) tends to increase the estimated ICCs and might have caused overestimation of the test–retest agreement of the DVT in this study (Lexell & Downham, 2005). Because the values of the ICC tend to be influenced by the variance of the test scores, we should interpret the test–retest reliability from another reliability examination, including random measurement error and practice effect. From the analyses of random measurement error, we found that most of the MDC%s of the C-DVT were smaller than 20% in patients with schizophrenia (12.9%–15.9%), except the MDC% between Assessments 1 and 2 (24.1%). However, the MDC%s of the DVT were larger than 20% (22.2%–29.7%) across adjacent assessments. These findings indicate that the C-DVT has acceptable random measurement error in patients with schizophrenia, whereas the random measurement error of the DVT was somewhat larger. Such a difference may be ascribed to the design of the C-DVT, which is intended to help the participants see the stimuli clearly (i.e., the font size of the stimuli in the C-DVT [36 pt] is larger than that in the DVT [12 pt]) (Yang et al., 2015) and requires less visual searching (i.e., the C-DVT presents only one column of 5 digits on each page, whereas the DVT presents 59 rows of 35 digits on each page). Therefore, these designs of the C-DVT may remove potential confounding influences from variability in visual acuity and visual searching skill across the participants. These findings support that those designs of the C-DVT can be helpful in reducing random measurement error. The MDC is a useful indicator for clinicians and researchers in judging whether a participant’s change score of repeated measurements is caused by random measurement error (i.e., the change score ≤ MDC) or by a real change in the participant’s ability (i.e., the change score > MDC). The MDCs of the C-DVT in patients with schizophrenia were 35.5–68.5 s, and those of the DVT were 110.1–156.0 s. These results indicate that a patient’s change score of the C-DVT will only need to be larger than 68.5 s (MDC% = 24.1%), which can be regarded as a real change in sustained attention. In contrast, the change score of the DVT should be larger than 156.0 s (MDC% = 29.7%) for users to interpret that the patient has a real change in sustained attention. Because the C-DVT had smaller MDC% than the DVT, the C-DVT appears to be a more useful test to detect real change in sustained attention in patients with schizophrenia. Therefore, our results show the potential of the C-DVT for use as an outcome measure for examining treatment effects on sustained attention in patients with schizophrenia. The analyses of the C-DVT’s practice effect showed that the effect size of Assessments 3–4 (−0.23) was smaller than 0.20 and smaller than that of Assessments 2–3 (0.36). These findings indicate that the practice effect of the C-DVT scores reached a plateau after three assessments. In contrast, the effect size of the DVT (0.30) was larger than 0.20 even between Assessments 3 and 4, which indicates that the practice effect of the DVT scores did not reach a plateau even after three assessments. The C-DVT scores reaching a plateau earlier than the DVT scores may have been due to two specific design characteristics of the C-DVT: the random presentation of the targets and the large number of practice trials before the formal testing (Yang et al., 2015). The practice effect of the DVT can cause users to overestimate improvement or underestimate deterioration of sustained attention in patients with schizophrenia. Therefore, the C-DVT appears to be a more suitable measure for repeatedly assessing sustained attention in patients with schizophrenia. In the examination of the concurrent validity, we found that the extent of the correlation between the adjusted scores (r = .75–.79) of the C-DVT and DVT was higher than that between the raw scores (r = .53–.64). This phenomenon indicates that the raw scores of the two tests might have been affected by the systematic bias of the administration order (Ibrahim et al., 2015). Therefore, the extent of the correlation between the raw scores of both tests was attenuated by the systematic bias (Ibrahim et al., 2015). In other words, the concurrent validity may have been underestimated by examination of the correlation between raw scores. On the other hand, the adjusted scores were calculated to control for the systematic bias of the administration order. Thus, the extent of correlation from the adjusted scores may more plausibly represent the concurrent validity of the C-DVT. The high correlations from the adjusted scores support that the concurrent validity of the C-DVT is good in patients with schizophrenia. The analyses of ecological validity showed that the C-DVT score was moderately correlated with the scores of both the PSP and the LIADL (r = −.43 and −.42, respectively). These results were in agreement with the results of previous studies, which reported the small-to-moderate correlations between sustained attention and both social function (r = .42–.59) (Meyer & Kurtz, 2009; Nienow, Docherty, Cohen, & Dinzeo, 2006; Ohno et al., 2000) and IADL (r = .36–.54) (Evans et al., 2003; Lipskaya et al., 2011) in patients with schizophrenia. In contrast, the DVT score was weakly correlated with both the PSP and the LIADL scores (r = −.30 and −.15, respectively). These findings indicate that the C-DVT has good ecological validity in patients with schizophrenia, but the ecological validity of the DVT was not supported. The difference in the ecological validities of the C-DVT and DVT might have been due to the larger random measurement error of the DVT (Muchinsky, 1996). The substantial amount of random measurement error of the DVT may have reduced the magnitude of the correlations between the DVT score and either the PSP score or the LIADL score. In contrast, the good ecological validity of the C-DVT may make it more appropriate than the DVT for inferring a patient’s social function and IADL function: If a patient with schizophrenia has a better score on the C-DVT, he/she is likely to have better social function and IADL function. Comparing the completion times of the C-DVT and DVT, we found that the C-DVT required only about half the completion time of the DVT in formal testing (C-DVT: 4.5–4.8 min vs. DVT: 8.1–8.8 min). The results indicate that the efficiency of the C-DVT assessment is better than that of the DVT assessment. Additionally, although the completion time of the C-DVT is shorter than that of the DVT, the completion time of the C-DVT for the participants was longer than 3 min, which is regarded as the minimum time for testing sustained attention. Therefore, the C-DVT can help users reduce the administration time for assessing sustained attention in patients with schizophrenia (Betts et al., 2006; Loetscher & Lincoln, 2013). In addition to the completion time, the total error counts of both the C-DVT and the DVT were also recorded. However, we did not analyze the error counts as a measure of the psychometric properties. The reason was that the error counts of both tests were too low (1st–3rd quartiles of the C-DVT and DVT were 0–3 and 0–10, respectively) to be analyzed as continuous scores (e.g., completion times). The low error counts of both tests may suggest that this score cannot differentiate the sustained attention abilities of patients with schizophrenia. For a user of both tests, the error counts could provide information to judge whether the C-DVT or DVT is suitable for a patient (Lewis, 1995). If a patient has more errors on the tests (e.g., more than 10 errors on the C-DVT), the patient may not have followed the testing rules. In this situation, the assessor could provide further instruction or demonstration of the testing rules for the patient. Two limitations of this study should be noted. First, the severity of the participants’ symptoms might have been unstable during the periods between Assessments 2 and 3, for we assessed the severity of illness (CGI-S) only at Assessments 1 and 4. The unstable symptom severities might have caused underestimation of the test–retest reliability of the C-DVT and the DVT. Second, the demographic homogeneity of our participants may have impeded the generalization of our findings to the general population of schizophrenia. All the participants were Asians, and most had mild severity of illness and a long period after onset. Further studies should recruit patients with different demographic characteristics to cross validate the psychometric properties of the C-DVT. In conclusion, our results showed that the C-DVT had acceptable test–retest reliability, good concurrent validity, and good ecological validity in patients with schizophrenia. These findings indicate that the C-DVT has the potential to become a reliable and valid test of sustained attention in patients with schizophrenia. Funding This study was supported by a research grant from the Taipei City Hospital (TPCH-104-059). The funding source had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication. Conflict of interest No authors report any potential conflicts of interest. Acknowledgments We thank all of the participants and researchers for their generosity of time and effort to participate in this study. References APA. ( 2013). Diagnostic and statistical manual of mental disorders (DSM-5)  ( 5th ed.). Washington, DC: American Psychiatric Publishing. Bartko, J. J. ( 1966). The intraclass correlation coefficient as a measure of reliability. Psychological Reports , 19, 3– 11. Google Scholar CrossRef Search ADS PubMed  Betts, J., Mckay, J., Maruff, P., & Anderson, V. ( 2006). The development of sustained attention in children: The effect of age and task load. Child Neuropsychology , 12, 205– 221. Google Scholar CrossRef Search ADS PubMed  Bland, J. M., & Altman, D. ( 1986). Statistical methods for assessing agreement between two methods of clinical measurement. The Lancet , 327, 307– 310. Google Scholar CrossRef Search ADS   Bushnell, C. D., Johnston, D. C., & Goldstein, L. B. ( 2001). Retrospective assessment of initial stroke severity comparison of the NIH Stroke Scale and the Canadian Neurological Scale. Stroke; A Journal of Cerebral Circulation , 32, 656– 660. Google Scholar CrossRef Search ADS   Chan, M. W. C., Yip, J. T. H., & Lee, T. M. C. ( 2004). Differential impairment on measures of attention in patients with paranoid and nonparanoid schizophrenia. Journal of Psychiatric Research , 38, 145– 152. Google Scholar CrossRef Search ADS PubMed  Chaytor, N., & Schmitter-Edgecombe, M. ( 2003). The ecological validity of neuropsychological tests: A review of the literature on everyday cognitive skills. Neuropsychology Review , 13, 181– 197. Google Scholar CrossRef Search ADS PubMed  Chiu, E. C., Koh, C. L., Tsai, C. Y., Lu, W. S., Sheu, C. F., Hsueh, I. P., et al.  . ( 2014). Practice effects and test–re-test reliability of the Five Digit Test in patients with stroke over four serial assessments. Brain Injury , 28, 1726– 1733. Google Scholar CrossRef Search ADS PubMed  Cohen, J. ( 1988). Statistical power analysis for the behavioral sciences  ( 2nd ed.). Hillsdale, NJ: Lawrence Erlbaum Associates. Depp, C. A., Moore, D. J., Sitzer, D., Palmer, B. W., Eyler, L. T., Roesch, S., et al.  . ( 2007). Neurocognitive impairment in middle-aged and older adults with bipolar disorder: Comparison to schizophrenia and normal comparison subjects. Journal of Affective Disorders , 101, 201– 209. Google Scholar CrossRef Search ADS PubMed  Doros, G., & Lew, R. ( 2010). Design based on intra-class correlation coefficients. American Journal of Biostatistics , 1 ( 1), 1– 8. Evans, J. D., Heaton, R. K., Paulsen, J. S., Palmer, B. W., Patterson, T., & Jeste, D. V. ( 2003). The relationship of neuropsychological abilities to specific domains of functional capacity in older schizophrenia patients. Biological Psychiatry , 53, 422– 430. Google Scholar CrossRef Search ADS PubMed  Green, M. F. ( 1996). What are the functional consequences of neurocognitive deficits in schizophrenia? American Journal of Psychiatry , 153, 321– 330. Google Scholar CrossRef Search ADS PubMed  Haley, S. M., & Fragala-Pinkham, M. A. ( 2006). Interpreting change scores of tests and measures used in physical therapy. Physical Therapy , 86, 735– 743. Google Scholar PubMed  Hambleton, R. K. ( 2000). Response to Hays et al. and McHorney and Cohen: Emergence of item response modeling in instrument development and data analysis. Medical Care , 38, II-60– II-65. Google Scholar CrossRef Search ADS   Haro, J., Kamath, S., Ochoa, S., Novick, D., Rele, K., Fargas, A., et al.  . ( 2003). The Clinical Global Impression–Schizophrenia scale: A simple instrument to measure the diversity of symptoms present in schizophrenia. Acta Psychiatrica Scandinavica , 107, 16– 23. Google Scholar CrossRef Search ADS   Huang, S. L., Hsieh, C. L., Wu, R. M., Tai, C. H., Lin, C. H., & Lu, W. S. ( 2011). Minimal detectable change of the Timed “Up & Go” Test and the Dynamic Gait Index in people with Parkinson disease. Physical Therapy , 91, 114– 121. Google Scholar CrossRef Search ADS PubMed  Ibrahim, I., Tobar, S., Elassy, M., Mansour, H., Chen, K., Wood, J., et al.  . ( 2015). Practice effects distort translational validity estimates for a Neurocognitive Battery. Journal of Clinical and Experimental Neuropsychology , 37, 530– 537. Google Scholar CrossRef Search ADS PubMed  La Porta, F., Franceschini, M., Caselli, S., Susassi, S., Cavallini, P., & Tennant, A. ( 2011). Unified Balance Scale: Classic psychometric and clinical properties. Journal of Rehabilitation Medicine , 43, 445– 453. Google Scholar CrossRef Search ADS PubMed  Lahiri, D., & Singh, A. R. S. ( 2016). Attentional impairment and minor physical anomalies in early onset schizophrenia. Eastern Journal of Psychiatry , 17, 2– 11. Lawton, M. P., & Brody, E. M. ( 1969). Assessment of older people: Self-maintaining and instrumental activities of daily living. The Gerontologist , 9, 179– 186. Google Scholar CrossRef Search ADS PubMed  Lee, P., Li, P. C., Liu, C. H., & Hsieh, C. L. ( 2011). Test–retest reliability of two attention tests in schizophrenia. Archives of Clinical Neuropsychology , 26, 405– 411. Google Scholar CrossRef Search ADS PubMed  Lewis, R. F. ( 1995). Digit vigilance test . Odessa, FL: Psychological Assessment Resources. Lexell, J. E., & Downham, D. Y. ( 2005). How to assess the reliability of measurements in rehabilitation. American Journal of Physical Medicine and Rehabilitation , 84, 719– 723. Google Scholar CrossRef Search ADS PubMed  Lipskaya, L., Jarus, T., & Kotler, M. ( 2011). Influence of cognition and symptoms of schizophrenia on IADL performance. Scandinavian Journal of Occupational Therapy , 18, 180– 187. Google Scholar CrossRef Search ADS PubMed  Liu, S. K., Chiu, C. H., Chang, C. J., Hwang, T. J., Hwu, H. G., & Chen, W. J. ( 2002). Deficits in sustained attention in schizophrenia and affective disorders: Stable versus state-dependent markers. American Journal of Psychiatry , 159, 975– 982. Google Scholar CrossRef Search ADS PubMed  Loetscher, T., & Lincoln, N. B. ( 2013). Cognitive rehabilitation for attention deficits following stroke. Cochrane Database of Systematic Reviews , 4, CD008391. Meyer, M. B., & Kurtz, M. M. ( 2009). Elementary neurocognitive function, facial affect recognition and social-skills in schizophrenia. Schizophrenia Research , 110, 173– 179. Google Scholar CrossRef Search ADS PubMed  Mokkink, L. B., Terwee, C. B., Patrick, D. L., Alonso, J., Stratford, P. W., Knol, D. L., et al.  . ( 2010). The COSMIN study reached international consensus on taxonomy, terminology, and definitions of measurement properties for health-related patient-reported outcomes. Journal of Clinical Epidemiology , 63, 737– 745. Google Scholar CrossRef Search ADS PubMed  Morosini, P., Magliano, L., Brambilla, L., Ugolini, S., & Pioli, R. ( 2000). Development, reliability and acceptability of a new version of the DSM‐IV Social and Occupational Functioning Assessment Scale (SOFAS) to assess routine social funtioning. Acta Psychiatrica Scandinavica , 101, 323– 329. Google Scholar PubMed  Muchinsky, P. M. ( 1996). The correction for attenuation. Educational and Psychological Measurement , 56, 63– 75. Google Scholar CrossRef Search ADS   Nienow, T. M., Docherty, N. M., Cohen, A. S., & Dinzeo, T. J. ( 2006). Attentional dysfunction, social perception, and social competence: What is the nature of the relationship? Journal of Abnormal Psychology , 115, 408– 417. Google Scholar CrossRef Search ADS PubMed  Ohno, T., Ikebuchi, E., Henomatsu, K., Kasai, K., Nakagome, K., Iwanami, A., et al.  . ( 2000). Psychophysiological correlates of social skills deficits in persons with schizophrenia. Psychiatry Research , 100, 155– 167. Google Scholar CrossRef Search ADS PubMed  Raghavan, D. V., Shanmugiah, A., Bharathi, P., & Jeyaprakash, R. ( 2016). P300 and neuropsychological measurements in patients with schizophrenia and their healthy biological siblings. Indian Journal of Psychiatry , 58, 454. Google Scholar CrossRef Search ADS PubMed  Rempfer, M. V., Hamera, E. K., Brown, C. E., & Cromwell, R. L. ( 2003). The relations between cognition and the independent living skill of shopping in people with schizophrenia. Psychiatry Research , 117, 103– 112. Google Scholar CrossRef Search ADS PubMed  Salter, K., Jutai, J. W., Teasell, R., Foley, N. C., Bitensky, J., & Bayley, M. ( 2005). Issues for selection of outcome measures in stroke rehabilitation: ICF activity. Disability and Rehabilitation , 27, 315– 340. Google Scholar CrossRef Search ADS PubMed  Sohlberg, M. M., & Mateer, C. A. ( 1987). Effectiveness of an attention-training program. Journal of Clinical and Experimental Neuropsychology , 9, 117– 130. Google Scholar CrossRef Search ADS PubMed  Woonings, F. M. J., Appelo, M. T., Kluiter, H., Slooff, C. J., & van den Bosch, R. J. ( 2003). Learning (potential) and social functioning in schizophrenia. Schizophrenia Research , 59, 287– 296. Google Scholar CrossRef Search ADS PubMed  Yang, C. M., Lin, G. H., Chen, M. H., Hsueh, I. P., & Hsieh, C. L. ( 2015). Development of a Computerized Digit Vigilance Test and validation in patients with stroke. Journal of Rehabilitation Medicine , 47, 311– 317. Google Scholar CrossRef Search ADS PubMed  © The Author 2017. Published by Oxford University Press. All rights reserved. For permissions, please e-mail: journals.permissions@oup.com.

Journal

Archives of Clinical NeuropsychologyOxford University Press

Published: Mar 1, 2018

There are no references for this article.

You’re reading a free preview. Subscribe to read the entire article.


DeepDyve is your
personal research library

It’s your single place to instantly
discover and read the research
that matters to you.

Enjoy affordable access to
over 12 million articles from more than
10,000 peer-reviewed journals.

All for just $49/month

Explore the DeepDyve Library

Unlimited reading

Read as many articles as you need. Full articles with original layout, charts and figures. Read online, from anywhere.

Stay up to date

Keep up with your field with Personalized Recommendations and Follow Journals to get automatic updates.

Organize your research

It’s easy to organize your research with our built-in tools.

Your journals are on DeepDyve

Read from thousands of the leading scholarly journals from SpringerNature, Elsevier, Wiley-Blackwell, Oxford University Press and more.

All the latest content is available, no embargo periods.

See the journals in your area

DeepDyve Freelancer

DeepDyve Pro

Price
FREE
$49/month

$360/year
Save searches from Google Scholar, PubMed
Create lists to organize your research
Export lists, citations
Access to DeepDyve database
Abstract access only
Unlimited access to over
18 million full-text articles
Print
20 pages/month
PDF Discount
20% off