JPP Student Journal Club Commentary: Smartphone-Delivered Interventions for Pediatric Populations: Improving Methodologies to Address Concerns of Feasibility and Efficacy

JPP Student Journal Club Commentary: Smartphone-Delivered Interventions for Pediatric... Smartphones demonstrate growing popularity as a delivery mechanism for pediatric interventions for multiple reasons. First, smartphone ownership is growing exponentially in the United States (Pew Research Center, 2018) and internationally (Poushter, 2016). Many now rely on smartphones as their sole source of Internet access (Smith, 2015), particularly users who are younger, minorities, and of lower socioeconomic status (Pew Research Center, 2018). Smartphones therefore promote access to large numbers of diverse groups, and do not require users to have costly broadband and/or computer access nor to take the time to physically sit down in front of a computer to complete intervention tasks. Second, smartphones tend to be carried throughout the day, promoting in-the-moment and ongoing opportunities for assessment and intervention (Proudfoot, 2013). Smartphones therefore also represent a delivery mechanism that is already highly used in daily life, promoting the likelihood of higher intervention use, and potentially overcome barriers to traditional delivery mechanisms, such as forgetfulness (Mohr et al., 2010). However, while smartphones are growing in popularity as daily tools and as a delivery mechanism for evidence-based pediatric interventions (Fedele, Cushing, Fritz, Amaro, & Ortega, 2017), concerns are present in the mobile health (mHealth) literature around their efficacy for specific pediatric populations (Grist, Porter, & Stallard, 2017). The present work by Schwebel et al. (2017) used a nonrandomized trial methodology to evaluate a smartphone-based virtual reality (VR) intervention to increase pedestrian safety in 56 schoolchildren in China (Schwebel et al, 2017). This work contributes to the literature on smartphones as a delivery mechanism and provides an opportunity to consider current practices in mHealth evaluations, including (1) strengths, (2) limitations, and (3) usability testing as a means to increase generalizability. This study has multiple strengths. First, the methodology is described clearly, with enough detail provided around the design to allow for replicability. Unfortunately, likely to comply with journal constraints in manuscript length, studies often sacrifice clear descriptions of the design. This gap in describing a critical methodological aspect of mHealth interventions typically limits generalizability and insights into the potential mechanisms impacting outcomes. Second, the authors used multimethod outcomes, such that pre/post comparisons could be made with both coded behavioral observations and child self-report. This is unfortunately only applied to roughly half of the sample (25 and 24 children’s actual pedestrian behavior was captured preintervention and postintervention, respectively; it is unclear if these children differ in any way from those not captured) but bolsters the significant findings for changes in self-reported data. Finally, the authors highlight that their intervention has minimal cultural specificity. While slight design modifications would be necessary for deployment in other countries and cultures (e.g., language translation), the majority of the described design appears to allow for evaluation and implementation for schoolchildren outside of China. This is an important strength, both for this study and mHealth interventions, from the perspective of delivering global health interventions to pediatric populations, particularly in resource challenged areas. While it is important to note that this study is a pilot intervention, there are also several limitations of the current work that highlight targets for improvement. The authors note: (1) the cost of equipment (i.e., $160 US per unit); (2) the possible introduction of bias (i.e., participants wore identifiable buttons and bright colors when crossing the street); (3) protocol gaps (i.e., managing participant excitement and inattention when introducing the intervention); (4) lack of a control group; and (5) no long-term follow-up data (Schwebel et al., 2017). However, other issues central to mHealth interventions also warrant discussion. These include the clinical significance of findings (e.g., at postintervention, ∼80% of children still selected “Maybe” or “No” to a prompt assessing if they feel safe crossing a busy street) and the usability of the system. Usability indicates how intuitive and easy to use a technology may be; usability testing is a systematic, formative, or summative evaluation to improve design (Tullis & Albert, 2008). Given that the current study was a pilot, the authors may have been constrained in their ability to measure usability or to report usability given their focus on the intervention. Previous usability testing of the VR system occurred with 68 adults, who rated the realism and their level of motion sickness (Schwebel, Severson, & He, 2017). However, usability testing guidelines specific to children have been developed and implemented (Hanna, Risden, & Alexander, 1997), and typically target five specific attributes: learnability, efficiency, memorability, errors, and satisfaction (Nielsen, 1993). Given the authors’ aim to implement learned skills for successful pedestrian behaviors, opportunities to evaluate usability metrics would improve generalizability. Indeed, usability testing of attributes such as learnability (i.e., to evaluate how well users may learn an intervention skill through use of an app) have begun to be increasingly implemented before trials or public deployment with adults (Sarkar, et al., 2016, Stiles-Shields et al., 2017, Vilardaga et al., 2018). Satisfaction is another critical item to evaluate. If users are not satisfied with a technology, they are unlikely to use it. The authors identified that the children found the system to be “fun” and “educational.” However, determining satisfaction from the perspective of the children, teachers, and parents would better speak to the generalizability of this intervention to schools or families. Future trials limited in their ability to conduct formal usability testing may also consider evaluating metrics of usability via a variety of validated self-report questionnaires (e.g., System Usability Scale; Brooke, 1996). The assessment of usability is a critical element of interpreting how well a smartphone-delivered intervention performs during interactions with intended users. Therefore, usability testing improves the claims of feasibility, efficacy, and generalizability of mHealth findings. The present work stands as a positive representation of evaluating a smartphone-delivered intervention to improve a crucial public health target for global pediatric populations. However, it also highlights opportunities for improving generalizability. For example, just as a review of the training, competency, and expertise of therapists delivering a face-to-face intervention are commonplace in interpreting the merit of a traditionally delivered intervention, usability findings should be provided in all reports in future mHealth literature. Funding This research was supported in part by grants from the National Institute of Nursing Research and the Office of Behavioral and Social Sciences Research (grant number R01 NR016235), National Institute of Child Health and Human Development (grant number R01 HD048629), and the March of Dimes Birth Defects Foundation (grant number 12-FY13-271). Conflicts of interest: None declared. References Brooke J. ( 1996). SUS: A “quick and dirty” usability scale. In Jordan P. W., Thomas B., Weerdmeester B. A. (Eds.), Usability evaluation in industry  (pp. 189– 194). London, UK: Taylor & Francis. Fedele D. A., Cushing C. C., Fritz A., Amaro C. M., Ortega A. ( 2017). Mobile health interventions for improving health outcomes in youth: A meta-analysis. JAMA Pediatrics , 171, 461– 469. Google Scholar CrossRef Search ADS PubMed  Grist R., Porter J., Stallard P. ( 2017). Mental health mobile apps for preadolescents and adolescents: A systematic review. Journal of Medical Internet Research , 19, e176. Google Scholar CrossRef Search ADS PubMed  Hanna L., Risden K., Alexander K. ( 1997). Guidelines for usability testing with children. Interactions , 4, 9– 14. Google Scholar CrossRef Search ADS   Mohr D. C., Ho J., Duffecy J., Baron K. G., Lehman K. A., Jin L., Reifler D. ( 2010). Perceived barriers to psychological treatments and their relationship to depression. Journal of Clinical Psychology , 66, 394– 409. Google Scholar PubMed  Nielsen J. ( 1993). Usability engineering . New York: Academic. Pew Research Center. ( 2018, February 5). Mobile fact sheet. Retrieved from http://www.pewinternet.org/fact-sheet/mobile/ Proudfoot J. ( 2013). The future is in our hands: The role of mobile phones in the prevention and management of mental disorders. Australian and New Zealand Journal of Psychiatry , 47, 111– 113. Google Scholar CrossRef Search ADS PubMed  Poushter J. ( 2016, February 22). Smartphone ownership and internet usage continues to climb in emerging economies . Pew Research Center. Retrieved from http://www.pewglobal.org/2016/02/22/smartphone-ownership-and-internet-usage-continues-to-climb-in-emerging-economies/ Sarkar U., Gourley G. I., Lyles C. R., Tieu L., Clarity C., Newmark L., Singh K., Bates D. W. ( 2016). Usability of commercially available mobile applications for diverse patients. Journal of General Internal Medicine , 31, 1417– 1426. Google Scholar CrossRef Search ADS PubMed  Schwebel D. C., Severson J., He Y. ( 2017). Using smartphone technology to deliver a virtual pedestrian environment: Usability and validation. Virtual Reality , 21, 145– 152. Google Scholar CrossRef Search ADS PubMed  Schwebel D. C., Wu Y., Li P., Severson J., He Y., Xiang H., Hu G. ( 2018). Evaluating smartphone-based virtual reality to improve Chinese schoolchildren’s pedestrian safety: A nonrandomized trial. Journal of Pediatric Psychology , 43, 473– 484. doi: https://doi.org/10.1093/jpepsy/jsx147 Stiles-Shields C., Montague E., Lattie E. G., Schueller S. M., Kwasny M. J., Mohr D. C. ( 2017). Exploring user learnability and learning performance in an app for depression: Usability study. JMIR Human Factors , 4, e18. Google Scholar CrossRef Search ADS PubMed  Smith A. ( 2015, April 1). U.S. smartphone use in 2015. Pew Research Center. Retrieved from http://www.pewinternet.org/2015/04/01/us-smartphone-use-in-2015/ Tullis T., Albert B. ( 2008). Measuring the user experience: Collecting, analyzing, and presenting usability metrics . Burlington, MA: Morgan Kaufmann Publishers. Vilardaga R., Rizo J., Zeng E., Kientz J. A., Ries R., Otis C., Hernandez K. ( 2018). User-centered design of Learn to Quit, a smoking cessation smartphone app for people with serious mental illness. JMIR Serious Games , 6, e2. Google Scholar CrossRef Search ADS PubMed  © The Author(s) 2018. Published by Oxford University Press on behalf of the Society of Pediatric Psychology. All rights reserved. For permissions, please e-mail: journals.permissions@oup.com This article is published and distributed under the terms of the Oxford University Press, Standard Journals Publication Model (https://academic.oup.com/journals/pages/about_us/legal/notices) http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Journal of Pediatric Psychology Oxford University Press

JPP Student Journal Club Commentary: Smartphone-Delivered Interventions for Pediatric Populations: Improving Methodologies to Address Concerns of Feasibility and Efficacy

Loading next page...
 
/lp/ou_press/jpp-student-journal-club-commentary-smartphone-delivered-interventions-Bo0Vxja00e
Publisher
Oxford University Press
Copyright
© The Author(s) 2018. Published by Oxford University Press on behalf of the Society of Pediatric Psychology. All rights reserved. For permissions, please e-mail: journals.permissions@oup.com
ISSN
0146-8693
eISSN
1465-735X
D.O.I.
10.1093/jpepsy/jsy018
Publisher site
See Article on Publisher Site

Abstract

Smartphones demonstrate growing popularity as a delivery mechanism for pediatric interventions for multiple reasons. First, smartphone ownership is growing exponentially in the United States (Pew Research Center, 2018) and internationally (Poushter, 2016). Many now rely on smartphones as their sole source of Internet access (Smith, 2015), particularly users who are younger, minorities, and of lower socioeconomic status (Pew Research Center, 2018). Smartphones therefore promote access to large numbers of diverse groups, and do not require users to have costly broadband and/or computer access nor to take the time to physically sit down in front of a computer to complete intervention tasks. Second, smartphones tend to be carried throughout the day, promoting in-the-moment and ongoing opportunities for assessment and intervention (Proudfoot, 2013). Smartphones therefore also represent a delivery mechanism that is already highly used in daily life, promoting the likelihood of higher intervention use, and potentially overcome barriers to traditional delivery mechanisms, such as forgetfulness (Mohr et al., 2010). However, while smartphones are growing in popularity as daily tools and as a delivery mechanism for evidence-based pediatric interventions (Fedele, Cushing, Fritz, Amaro, & Ortega, 2017), concerns are present in the mobile health (mHealth) literature around their efficacy for specific pediatric populations (Grist, Porter, & Stallard, 2017). The present work by Schwebel et al. (2017) used a nonrandomized trial methodology to evaluate a smartphone-based virtual reality (VR) intervention to increase pedestrian safety in 56 schoolchildren in China (Schwebel et al, 2017). This work contributes to the literature on smartphones as a delivery mechanism and provides an opportunity to consider current practices in mHealth evaluations, including (1) strengths, (2) limitations, and (3) usability testing as a means to increase generalizability. This study has multiple strengths. First, the methodology is described clearly, with enough detail provided around the design to allow for replicability. Unfortunately, likely to comply with journal constraints in manuscript length, studies often sacrifice clear descriptions of the design. This gap in describing a critical methodological aspect of mHealth interventions typically limits generalizability and insights into the potential mechanisms impacting outcomes. Second, the authors used multimethod outcomes, such that pre/post comparisons could be made with both coded behavioral observations and child self-report. This is unfortunately only applied to roughly half of the sample (25 and 24 children’s actual pedestrian behavior was captured preintervention and postintervention, respectively; it is unclear if these children differ in any way from those not captured) but bolsters the significant findings for changes in self-reported data. Finally, the authors highlight that their intervention has minimal cultural specificity. While slight design modifications would be necessary for deployment in other countries and cultures (e.g., language translation), the majority of the described design appears to allow for evaluation and implementation for schoolchildren outside of China. This is an important strength, both for this study and mHealth interventions, from the perspective of delivering global health interventions to pediatric populations, particularly in resource challenged areas. While it is important to note that this study is a pilot intervention, there are also several limitations of the current work that highlight targets for improvement. The authors note: (1) the cost of equipment (i.e., $160 US per unit); (2) the possible introduction of bias (i.e., participants wore identifiable buttons and bright colors when crossing the street); (3) protocol gaps (i.e., managing participant excitement and inattention when introducing the intervention); (4) lack of a control group; and (5) no long-term follow-up data (Schwebel et al., 2017). However, other issues central to mHealth interventions also warrant discussion. These include the clinical significance of findings (e.g., at postintervention, ∼80% of children still selected “Maybe” or “No” to a prompt assessing if they feel safe crossing a busy street) and the usability of the system. Usability indicates how intuitive and easy to use a technology may be; usability testing is a systematic, formative, or summative evaluation to improve design (Tullis & Albert, 2008). Given that the current study was a pilot, the authors may have been constrained in their ability to measure usability or to report usability given their focus on the intervention. Previous usability testing of the VR system occurred with 68 adults, who rated the realism and their level of motion sickness (Schwebel, Severson, & He, 2017). However, usability testing guidelines specific to children have been developed and implemented (Hanna, Risden, & Alexander, 1997), and typically target five specific attributes: learnability, efficiency, memorability, errors, and satisfaction (Nielsen, 1993). Given the authors’ aim to implement learned skills for successful pedestrian behaviors, opportunities to evaluate usability metrics would improve generalizability. Indeed, usability testing of attributes such as learnability (i.e., to evaluate how well users may learn an intervention skill through use of an app) have begun to be increasingly implemented before trials or public deployment with adults (Sarkar, et al., 2016, Stiles-Shields et al., 2017, Vilardaga et al., 2018). Satisfaction is another critical item to evaluate. If users are not satisfied with a technology, they are unlikely to use it. The authors identified that the children found the system to be “fun” and “educational.” However, determining satisfaction from the perspective of the children, teachers, and parents would better speak to the generalizability of this intervention to schools or families. Future trials limited in their ability to conduct formal usability testing may also consider evaluating metrics of usability via a variety of validated self-report questionnaires (e.g., System Usability Scale; Brooke, 1996). The assessment of usability is a critical element of interpreting how well a smartphone-delivered intervention performs during interactions with intended users. Therefore, usability testing improves the claims of feasibility, efficacy, and generalizability of mHealth findings. The present work stands as a positive representation of evaluating a smartphone-delivered intervention to improve a crucial public health target for global pediatric populations. However, it also highlights opportunities for improving generalizability. For example, just as a review of the training, competency, and expertise of therapists delivering a face-to-face intervention are commonplace in interpreting the merit of a traditionally delivered intervention, usability findings should be provided in all reports in future mHealth literature. Funding This research was supported in part by grants from the National Institute of Nursing Research and the Office of Behavioral and Social Sciences Research (grant number R01 NR016235), National Institute of Child Health and Human Development (grant number R01 HD048629), and the March of Dimes Birth Defects Foundation (grant number 12-FY13-271). Conflicts of interest: None declared. References Brooke J. ( 1996). SUS: A “quick and dirty” usability scale. In Jordan P. W., Thomas B., Weerdmeester B. A. (Eds.), Usability evaluation in industry  (pp. 189– 194). London, UK: Taylor & Francis. Fedele D. A., Cushing C. C., Fritz A., Amaro C. M., Ortega A. ( 2017). Mobile health interventions for improving health outcomes in youth: A meta-analysis. JAMA Pediatrics , 171, 461– 469. Google Scholar CrossRef Search ADS PubMed  Grist R., Porter J., Stallard P. ( 2017). Mental health mobile apps for preadolescents and adolescents: A systematic review. Journal of Medical Internet Research , 19, e176. Google Scholar CrossRef Search ADS PubMed  Hanna L., Risden K., Alexander K. ( 1997). Guidelines for usability testing with children. Interactions , 4, 9– 14. Google Scholar CrossRef Search ADS   Mohr D. C., Ho J., Duffecy J., Baron K. G., Lehman K. A., Jin L., Reifler D. ( 2010). Perceived barriers to psychological treatments and their relationship to depression. Journal of Clinical Psychology , 66, 394– 409. Google Scholar PubMed  Nielsen J. ( 1993). Usability engineering . New York: Academic. Pew Research Center. ( 2018, February 5). Mobile fact sheet. Retrieved from http://www.pewinternet.org/fact-sheet/mobile/ Proudfoot J. ( 2013). The future is in our hands: The role of mobile phones in the prevention and management of mental disorders. Australian and New Zealand Journal of Psychiatry , 47, 111– 113. Google Scholar CrossRef Search ADS PubMed  Poushter J. ( 2016, February 22). Smartphone ownership and internet usage continues to climb in emerging economies . Pew Research Center. Retrieved from http://www.pewglobal.org/2016/02/22/smartphone-ownership-and-internet-usage-continues-to-climb-in-emerging-economies/ Sarkar U., Gourley G. I., Lyles C. R., Tieu L., Clarity C., Newmark L., Singh K., Bates D. W. ( 2016). Usability of commercially available mobile applications for diverse patients. Journal of General Internal Medicine , 31, 1417– 1426. Google Scholar CrossRef Search ADS PubMed  Schwebel D. C., Severson J., He Y. ( 2017). Using smartphone technology to deliver a virtual pedestrian environment: Usability and validation. Virtual Reality , 21, 145– 152. Google Scholar CrossRef Search ADS PubMed  Schwebel D. C., Wu Y., Li P., Severson J., He Y., Xiang H., Hu G. ( 2018). Evaluating smartphone-based virtual reality to improve Chinese schoolchildren’s pedestrian safety: A nonrandomized trial. Journal of Pediatric Psychology , 43, 473– 484. doi: https://doi.org/10.1093/jpepsy/jsx147 Stiles-Shields C., Montague E., Lattie E. G., Schueller S. M., Kwasny M. J., Mohr D. C. ( 2017). Exploring user learnability and learning performance in an app for depression: Usability study. JMIR Human Factors , 4, e18. Google Scholar CrossRef Search ADS PubMed  Smith A. ( 2015, April 1). U.S. smartphone use in 2015. Pew Research Center. Retrieved from http://www.pewinternet.org/2015/04/01/us-smartphone-use-in-2015/ Tullis T., Albert B. ( 2008). Measuring the user experience: Collecting, analyzing, and presenting usability metrics . Burlington, MA: Morgan Kaufmann Publishers. Vilardaga R., Rizo J., Zeng E., Kientz J. A., Ries R., Otis C., Hernandez K. ( 2018). User-centered design of Learn to Quit, a smoking cessation smartphone app for people with serious mental illness. JMIR Serious Games , 6, e2. Google Scholar CrossRef Search ADS PubMed  © The Author(s) 2018. Published by Oxford University Press on behalf of the Society of Pediatric Psychology. All rights reserved. For permissions, please e-mail: journals.permissions@oup.com This article is published and distributed under the terms of the Oxford University Press, Standard Journals Publication Model (https://academic.oup.com/journals/pages/about_us/legal/notices)

Journal

Journal of Pediatric PsychologyOxford University Press

Published: Mar 13, 2018

There are no references for this article.

You’re reading a free preview. Subscribe to read the entire article.


DeepDyve is your
personal research library

It’s your single place to instantly
discover and read the research
that matters to you.

Enjoy affordable access to
over 18 million articles from more than
15,000 peer-reviewed journals.

All for just $49/month

Explore the DeepDyve Library

Search

Query the DeepDyve database, plus search all of PubMed and Google Scholar seamlessly

Organize

Save any article or search result from DeepDyve, PubMed, and Google Scholar... all in one place.

Access

Get unlimited, online access to over 18 million full-text articles from more than 15,000 scientific journals.

Your journals are on DeepDyve

Read from thousands of the leading scholarly journals from SpringerNature, Elsevier, Wiley-Blackwell, Oxford University Press and more.

All the latest content is available, no embargo periods.

See the journals in your area

DeepDyve

Freelancer

DeepDyve

Pro

Price

FREE

$49/month
$360/year

Save searches from
Google Scholar,
PubMed

Create lists to
organize your research

Export lists, citations

Read DeepDyve articles

Abstract access only

Unlimited access to over
18 million full-text articles

Print

20 pages / month

PDF Discount

20% off