Adult experts’ perceptions of telemental health for youth: A Delphi study

Adult experts’ perceptions of telemental health for youth: A Delphi study Downloaded from https://academic.oup.com/jamiaopen/article-abstract/1/1/67/4980801 by Ed 'DeepDyve' Gillespie user on 07 November 2018 JAMIA Open, 1(1), 2018, 67–74 doi: 10.1093/jamiaopen/ooy002 Advance Access Publication Date: 20 April 2018 Research and Applications Research and Applications Adult experts’ perceptions of telemental health for youth: A Delphi study 1 1,2 1,3 1 Abigail Howard, Mindy Flanagan, Michelle Drouin, Maria Carpenter, 4 5 1 Elizabeth M. Chen, Catherine Duchovic, and Tammy Toscos 1 2 Parkview Research Center, Parkview Health, Fort Wayne, Indiana, USA, Indiana University Center for Health Services Research, School of Medicine, Indianapolis, Indiana, USA, Indiana University–Purdue University Fort Wayne, Fort Wayne, 4 5 Indiana, USA, Marian University College of Osteopathic Medicine, Indianapolis, Indiana, USA and Park Center, Inc. Fort Wayne, Indiana, USA Corresponding Author: Tammy Toscos, Ph.D., Director of Informatics, 10622 Parkview Plaza Dr, Fort Wayne, Indiana 46845, USA (Tammy.Toscos@parkview.com) Received 2 January 2018; Revised 2 February 2018; Editorial Decision 0 Month 0000; Accepted 22 February 2018 ABSTRACT Objectives: Our objectives were to measure experts’ opinions and develop consensus via the Delphi process on the barriers, applications, and concerns associated with telemental health (TMH) for youth. Materials and methods: We delivered 3 online surveys over 2 months in Summer, 2016–2025 adult experts, in- cluding adults who experienced youth depression or suicidality, parents of youth with lived experience, and professionals (ie youth mental health researchers, clinicians/staff, or educators). We used the Delphi method to construct Likert and open-ended questions, developing expert consensus over 3 iterative surveys on the bar- riers and benefits of TMH for youth. Results: Adult experts identified stigma and knowledge barriers to youth mental health care. Although TMH is perceived as beneficial for screening, education, follow-up, and emotional support, no single delivery method (eg websites or instant messaging) was deemed universally beneficial. Discussion: Adults are the developers, administrators, and gatekeepers of youth mental health care. Although adult experts see potential for TMH to supplement traditional therapy via familiar technologies, there is no con- sensus on the technologies by which TMH should be delivered. However, there is consensus that family mem- bers and friends provide potential pathways to care; thus, an online TMH toolkit for youth would be beneficial for both caretakers and practitioners. Conclusion: Telemental health may not overcome barriers for crisis management but adult experts agreed that TMH had potential benefits for youth. Health care organizations should conduct research and provide training and education to youth caretakers and practitioners on potential barriers and benefits of TMH technologies for youth. Key words: telemedicine, mental health, Delphi technique, delivery of health care, youth INTRODUCTION stigma and lack of knowledge about accessing mental health serv- ices, as well as structural barriers, such as a lack of transportation 3–9 Background and significance and provider shortages (particularly for those in rural settings). Many youth struggling with mental health problems do not receive Telemental health (TMH) technologies—which can be broadly de- 1,2 mental health care. For this group, barriers to gaining access to fined as technologies used to make clinical assessments or to deliver mental health care include both psychological barriers, such as mental health care, education, or information—may offer potential V The Author(s) 2018. Published by Oxford University Press on behalf of the American Medical Informatics Association. This is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/4.0/), which permits non-commercial re-use, distribution, and reproduction in any medium, provided the original work is properly cited. For commercial re-use, please contact journals.permissions@oup.com 67 Downloaded from https://academic.oup.com/jamiaopen/article-abstract/1/1/67/4980801 by Ed 'DeepDyve' Gillespie user on 07 November 2018 68 JAMIA Open, 2018, Vol. 1, No. 1 solutions, providing a means for mental health care that is widely in considering diverse opinions and options and identifying consen- 2–10 21,22 available, less stigmatized, and easier to access. The pervasive- sus knowledge and priorities. As Delphi-based mental health re- ness of daily technology use among adolescents and young adults search is now using both professionals and those with lived 23,24 raises the possibility that TMH technologies may provide a pathway experience as experts, we included both youth mental health for mental health services for these populations. professionals and those with lived experience of youth mental health Over the past decade, in line with the surge of mobile phone use issues (ie adults who experienced depression or suicidal ideation and ownership, researchers have been developing and investigat- during youth and parents of youth with depression or suicidal idea- ing the effectiveness of TMH across a broad range of demographic tion) in our study. groups. Telemental health can include, but is not limited to, the provision of mental health services through telecommunications Objectives technology (including training rural health care workers online), Our goal was to engage a panel of adult experts in the Delphi pro- videoconferencing with consultations on behavioral health issues, cess to yield consensus on the barriers, applications, and concerns “virtual” case management, and smartphone applications targeted associated with TMH for youth. This Delphi study was the first part at teaching mental health skills. A systematic review of this research of broader research project on youth mental health issues and serv- has shown that TMH technologies offer convenience, confidential- ices, with the primary goals of identifying barriers to mental health ity, familiarity, and integration into everyday life. Additionally, care for youth ages 14 to 24 and examining the potential role of patients who utilize TMH display a higher adherence for return TMH technologies in overcoming these barriers and addressing the appointments than with traditional therapy and report high levels of mental health needs of youth. The results of this Delphi study in- patient satisfaction. In short, TMH is effective, and it has been formed later focus group and survey questions delivered to youth, shown to be effective among diverse groups, including youth their families, and caretakers. As adult experts may steer the devel- 13–16 populations. opment, usage, and acceptability of TMH technologies, we saw this With regard to the effectiveness of TMH for youth specifically, Delphi process as an essential step in the further refinement of TMH recent reviews show that telepsychiatry (ie clinicians using videocon- resources for youth. ferencing for psychiatric care) is an effective mental health care deliv- ery method for youth with various psychiatric issues (eg anxiety, attention deficit hyperactivity disorder, oppositional defiant disorder, MATERIALS AND METHODS etc.) in a variety of settings (eg juvenile detention centers, schools, 14–16 Procedure and data collection and inpatient units). For the most part, youth patients who re- As per Parkview Health’s Institutional Review Board approval, pro- ceived telepsychiatric care had comparable outcomes to patients re- spective panel members were recruited through listservs of 5 local ceiving face-to-face services, and both parents and youth expressed 14–16 mental health organizations in Northeast Indiana in Summer, 2016. satisfaction with TMH. However, some participants expressed These mental health organizations served both urban and rural com- concerns about the modality, such as privacy issues and video limita- munities in the largest county in this region. All listserv recipients re- tions. Moreover, telepsychiatry represents only one type of TMH ceived a secure link to a prequestionnaire, where they were directed technology; there are numerous applications and websites that are to select categorizations that applied to them from a list of lived now available that also offer mental health support to consumers experiences (eg parent of youth with depression or suicidal ideation) without therapist engagement. These programs, which are often and professional backgrounds (eg youth mental health clinician, re- based on cognitive behavioral therapy (CBT) principles and delivered searcher, or educator). As per our study protocol, 40 individuals via interactive websites and multimedia applications, have proven were selected randomly after the 260 replies were screened for meet- effective for reducing anxiety and depression among youth. ing inclusion criteria. Of the 40 adult experts identified for possible Youth-based TMH usability research is far less advanced than participation, 25 consented to participate in the Delphi panel pro- effectiveness research. In the studies that have measured benefits cess. Panel members were compensated with $30 Amazon gift cards and barriers of specific technologies, youth have responded with 18,19 20 after completing the first and third surveys. positive and negative feedback (eg SPARX and CopeSmart ). For example, adolescents stated that while CopeSmart (a TMH smartphone application focused on emotional self-monitoring and Participants positive coping strategies) was easy to use and provided helpful in- Overall, 25 experts (21 women, 4 men) consented to participate via formation, it was not very engaging or effective. However, these an online informed consent process, and 24 participated in all 3 findings are difficult to generalize, as the users’ opinions were fo- rounds of surveys. Most of the expert panel members identified as cused on one specific technology as compared to TMH, generally. non-Hispanic, White (92%), and 8% identified as another ethnicity. Therefore, more generalized assessments of barriers and benefits for In terms of age groups represented, 28% of the sample was 24–34, youth-based TMH are necessary. 32% of the sample was 35–49, and 40% of the sample was 50–64. As TMH use among youth is not yet widespread, opinions from With regard to their expert role, 4 people (16%) identified them- adults (who have experience with mental health issues, mental selves as having lived experience (ie adult with experience of youth health care, or both) may provide critical information for developing depression or suicidality or parent of youth with lived experience), and refining youth-based TMH technologies. Securing feedback 12 (48%) identified themselves as professionals (ie mental health re- from these stakeholders aligns with Brofenbrenner’s ecological sys- searcher, mental health clinician/staff at mental health organization, tems theory (1979), which posits that youth are affected by various educator that works with youth), and 9 (36%) as having both lived systems, including the family and wider culture. Within cultural con- experience and experience working in a professional capacity with texts, expert opinions are paramount for decision-making; therefore, youth. Finally, in terms of their existing knowledge of TMH, 14 ex- in this study, we employed the Delphi method using adult experts in pert panel members (56%) were familiar with TMH, and 8 expert youth mental health. The Delphi method has proven highly effective panel members (32%) had previously used TMH. Downloaded from https://academic.oup.com/jamiaopen/article-abstract/1/1/67/4980801 by Ed 'DeepDyve' Gillespie user on 07 November 2018 JAMIA Open, 2018, Vol. 1, No. 1 69 Measures comfortable using TMH to supplement therapy. All items used 5- point Likert response sets (ie 1 ¼ strongly disagree,5 ¼ strongly Expert panel members completed surveys online and agree;1 ¼ no, absolutely not,5 ¼ yes, definitely). anonymously—a distinguishing feature of the Delphi method—to help avoid pitfalls of face-to-face meetings (eg group think, irratio- nal adherence to an opinion, etc.) . Surveys included quantitative Analytic strategy and/or qualitative questions and were created via iterative processes, Quantitative data analysis included calculating descriptive statistics based on previous survey replies. (frequency, mean) for demographic characteristics and survey items. The percentage of responses by valence (1 ¼ strongly disagree, dis- Survey 1: broadly identifying barriers and telemental health agree;2 ¼ neutral;3 ¼ agree, strongly agree) was calculated for each usefulness for youth item. Additionally, a consensus score was calculated for each item Round 1 survey questions were created by the research team based which quantifies proximities in ordinal scales and consistent on consensus, literature review, and clinical experiences. Panel mem- responses between experts. Consensus was defined as: bers indicated the extent to which they viewed items as barriers for youth receiving mental health services (ie stigma, cost, transporta- jX  l j Consensus ¼ 1 þ p log 1 tion). For this same list of barriers, panel members indicated the ex- d i¼1 tent to which these barriers to youth receiving services could be In this equation, p is the probability for each Likert score; l is i X overcome with TMH. Panel members indicated the extent to which the mean of X; d is the possible range of X, d ¼ X  X (eg X X max min they viewed items as benefits of traditional in-person therapy com- 5–1). Here, a higher consensus score is reached between proximal pared to TMH (ie easier to talk in person, more comfortable in per- responses (ie strongly agree and agree rating); dissension is evident son). Panel members also rated potential benefits of incorporating with wider dispersion between responses (ie strongly agree and TMH for youth (ie scheduling appointments, positive text messages, strongly disagree). Consensus was operationalized as consensus mood assessment). For these closed-ended questions, panel members scores at 0.75 or above. At this score, no more than one expert can indicated agreement using a 5-point Likert scale (1 ¼ not at all; respond with an oppositely valenced rating than the others (with the 5 ¼ very much). The remaining 10 questions were open-ended, fo- exception for scores clustered around 3, but this case was not ob- cused on drawbacks and benefits of TMH delivery methods (ie video served). Below this score, experts’ ratings were distributed across a conferencing, websites, social media), concerns and benefits of using wider range of agreement/disagreement scores or had a few people TMH for youth, personal experience witnessing benefits and nega- who strongly disagreed with statements (while the rest agreed or tive outcomes for youth TMH use, and potential for TMH to im- were neutral). Data analyzes for this article were conducted using prove access to mental health services for youth. Microsoft Excel and SAS 9.4 software. Survey 2: refining barriers and benefits of telemental health for youth RESULTS Thematic analysis was conducted on open-ended responses from Round 1 to identify consistencies in responses about concerns and Perceptions of telemental health for youth benefits of TMH for youth. These responses were used to create Round 1 responses revealed that the majority of experts perceived closed-ended questions for the Round 2 survey, using 5-point Likert barriers for youth mental health access. With regard to specific bar- scales (1 ¼ not at all,5 ¼ very much;1 ¼ strongly disagree, riers, experts were in consensus that stigma (92%) and knowledge 5 ¼ strongly agree). Items assessed were benefits of seeing a therapist about where or how to obtain services (80%) were primary barriers in person (vs online), benefits of using technology to address mental to youth seeking mental health care (see Table 1). However, there health needs in youth, concerns of using technology to address youth was no consensus on the barriers that TMH tools could overcome. mental health needs, and benefits of using technology to supplement Among open-ended responses, 7 respondents indicated parents in-person therapy. Additionally, benefits of different TMH delivery may be a barrier for youth seeking mental health services. In one methods for youth were assessed on various parameters (see case, a mental health professional stated: Table 3). “In many cases parents need to give permission, especially for ac- cess to insurance, and they often do not see the importance or Survey 3: reaching consensus for the benefits of telemental health need for their teen to access services. Often youth do not share for youth with their parents that they may feel the need to access these serv- The goal of the Round 3 survey was consensus among panel mem- ices and it is often difficult for them to access services on their bers about prominent benefits and concerns of TMH use for youth. own.” For granularity, we separated some responses into individual items (eg “parents, friends, family” was divided into 3 individual items) Meanwhile, with regard to benefits of TMH, experts were in and converted others from qualitative to quantitative items (eg par- consensus that positive text messages, automatic messaging before ticular uses of TMH). Panel members indicated to what extent they appointments, and self-management for anxiety were beneficial thought TMH could support the following aspects of health care for applications of TMH tools for youth. Experts also agreed that youth: crisis support, emotional support, education, self-tracking, parents or other family, physicians or therapists, and faith leaders screenings, monitoring progress, coping tools, connecting people have responsibility to connect youth to mental health services. Col- who are struggling, and virtual support groups. Another item lege student counselors were viewed as responsible for connecting assessed which individuals were effective in connecting youth to college-aged youth to mental health services. Notably, experts did mental health resources. Additionally, panel members indicated not show consensus on youth benefits of seeing a therapist in person the extent to which they were open to using, planning to use, and (vs online). Downloaded from https://academic.oup.com/jamiaopen/article-abstract/1/1/67/4980801 by Ed 'DeepDyve' Gillespie user on 07 November 2018 70 JAMIA Open, 2018, Vol. 1, No. 1 Table 1. Round 1 expert consensus: barriers, benefits, and responsibility for telemental health for youth ages 14–24 years Question Strongly disagree or disagree % Neutral % Agree or strongly agree % Consensus Barriers to access mental health services Stigma 0 8 92 0.77 Knowledge about where/how to obtain services 4 16 80 0.80 Beneficial uses of TMH Positive text messages 0 4 96 0.79 Automatic messaging before an appointment 0 8 92 0.82 Self-manage anxiety 4 24 72 0.76 Who youth trust to help them get mental health care Internet/search engine 0 32 68 0.78 Responsible party to ensure high school students receive mental health services Parents/guardians/siblings/other relatives 0 4 96 0.85 Physician/therapist 0 12 88 0.75 Faith leader 4 24 72 0.77 Responsible party to ensure college students receive mental health services College student counseling center 0 12 88 0.77 Note: Percentage of participants by response category and consensus score (n ¼ 25). Among round 2 responses, experts agreed on the following appli- following components: education (100%), emotional support cations of technology to supplement therapy: follow-up on therapy- (100%), self-tracking (100%), screening (96%), and monitoring related assignments, enhance medication adherence, emotional sup- progress (92%). However, experts did not reach consensus on using port, questions for therapist between sessions, and appointment TMH tools for crisis support (either in favor or opposed; con- reminders (See Table 2). Meanwhile, in open-ended responses, panel sensus¼ 0.52 for crisis support with youth not receiving care, con- members identified TMH as convenient for continuous access at all sensus¼ 0.61 for crisis support with youth receiving care). Most times of day and night. A mental health professional indicated in an experts were open to using technology to supplement traditional, open-ended response: face-to-face therapy (88%) and planned to use TMH tools in the fu- ture with youth (75%). “24-hour responses to crisis situations, this age group may be comfortable send texts/talking later at night when negative thoughts occur before sleep.” DISCUSSION The benefits of TMH delivery methods are displayed in Table 3. Notably, experts did not consistently indicate that it is easier for Youth engagement with TMH resources will depend, at least par- youth to share emotions (“open up”) via technology for any of the tially, on buy-in from adult stakeholders who develop, recommend, TMH delivery methods assessed. Additionally, across TMH delivery and administer these mental health technologies. Through an itera- methods, the inability to observe nonverbal communication was tive Delphi process, youth mental health experts, including profes- viewed as a drawback (for online chat, websites, social media, and sionals and those with lived experience of youth mental health text messaging). Experts viewed technological problems as a draw- issues, identified various benefits and barriers to TMH for youth. back to video conferencing and computer literacy/computer access Most experts agreed that stigma is a primary barrier to youth problems as a drawback to websites. Experts did not reach consen- accessing mental health services. Unfortunately, previous research sus on which TMH delivery method would be most beneficial and has shown that parents’ perceptions of stigma are associated with with the least drawbacks for youth. less willingness to seek services for their children ; therefore, this In open-ended responses, many panel members expressed that overarching stigma may affect uptake of mental health services, gen- the usefulness of TMH services would depend on the youth and the erally, and TMH technologies, specifically. More importantly, al- situation. For example: though our experts acknowledged stigma as a barrier to mental health treatment, they did not reach consensus that TMH overcomes “I am not sure if video conferencing will help my client open up stigma. Hence, at least from the viewpoint of adult experts, TMH more, but I suppose it just depends on the personality of the does not abolish this significant barrier to youth mental health client.” treatment. Poor knowledge about obtaining services was also viewed as a “(TMH services could be) helpful for people uncomfortable with face to face communication.” barrier to youth mental health treatment. These findings are consis- tent with other research indicating there are pragmatic barriers that “Impacts youth who live in poverty with no computer or internet 28–31 prevent youth from receiving services. However, the majority service.” of our panel members were familiar with TMH, and 32% had used See Appendix A for additional responses related to each technology. TMH, suggesting that TMH is gaining popularity among adult By Round 3, experts agreed that TMH could help youth who are stakeholders. Future education and marketing efforts concerning not engaged with traditional face-to-face mental health in the TMH for youth should be targeted towards family members, peers, following ways: education (100%), emotional support (92%), and therapists, faith leaders, and college counseling centers, parties screening (88%, see Table 4). Additionally, experts indicated deemed by our experts as responsible for connecting youth to mental that mental health care could be supported by technology for the health resources. Health care organizations may be the ideal vehicle Downloaded from https://academic.oup.com/jamiaopen/article-abstract/1/1/67/4980801 by Ed 'DeepDyve' Gillespie user on 07 November 2018 JAMIA Open, 2018, Vol. 1, No. 1 71 Table 2. Round 2 expert consensus: benefits of telemental health for youth ages 14–24 years Strongly disagree Agree or strongly or disagree % Neutral % agree % Consensus Benefit of technology supplementing in-person therapy Follow-up on therapy-related assignments 0 4 96 0.80 Enhance medication adherence through messages, reminders, and education 0 8 92 0.77 Emotional support/encouragement 0 8 92 0.79 Ask therapist questions between sessions 4 0 96 0.77 Appointment reminders 0 0 100 0.97 Note: Percentage of participants by response category and consensus score (n ¼ 24). Table 3. Round 2 expert consensus (consensus> 0.75): benefits of TMH delivery methods for youth ages 14–24 years Assessed for benefit but no consensus that TMH Benefit Beneficial TMH delivery methods method provides benefit Improved access to mental health care Video conference, Online chat/Instant Message, Websites, Social media, Wearable technologies Mobile apps, Text messaging Easier to open up via technology (vs face- No consensus Video conference, Online chat/Instant Message, to-face) Mobile apps, Websites, Text messaging, Wearable technologies Emotional/social support No consensus Video conference, social media, text messaging Youth familiarity for technology-based Online chat/Instant Message, Mobile apps, Websites communication Social media, Text messaging, Wearable technologies Quick response Online chat/Instant Message, Text messaging Mobile apps, Websites, Social media, Wearable technologies Anonymity No consensus Online chat/Instant Message, Mobile apps Convenience Online chat/Instant Message, Mobile apps, Websites Wearable technologies Educational/raise awareness Websites, Social media Mobile apps, Text messaging Early intervention tool No consensus Websites Motivational No consensus Text messaging, Wearable technologies Addresses stigma Video conference Note: Not all TMH technologies were assessed for each benefit (n ¼ 24). 14–16 for such promotion, as they have connections in the community and are quite recent, it may take some time for the positive reviews monetary resources for large-scale health education initiatives. of TMH and accompanying guidelines to percolate through the Health care organizations are also in a prime position to provide health care system to clinicians and consumers. However, there also support to rural health practitioners for implementation and use of needs to be more focus on whether TMH is suitable for acute care, TMH in their practice, which could be particularly helpful in over- as experts’ opinions on it are clearly divided. coming access barriers for rural youth. Within the health care sys- Telemental health is delivered through a variety of methods, tem, TMH education could entail workshops or conferences, but it from websites, to apps, to videoconferencing; additionally, gaming might also be provided through secure messaging via patient portals and virtual reality are being explored as potential delivery modali- in primary care/pediatric offices. Future research should explore the ties. From an informatics standpoint, we expected that experts’ effectiveness of these alternate avenues for TMH education within feedback on delivery methods might prove most useful for TMH de- different populations. velopment. However, unfortunately, there was no one modality that With regard to benefits, experts were in agreement that TMH was endorsed universally as having beneficial features. Instead, ben- technologies could be helpful for certain aspects of mental health efits depended on the medium. For example, experts were in agree- treatment, such as appointment and medication reminders, connect- ment that online chat/instant messaging provided benefits in terms ing with therapists and completing follow-up work between ses- of improved access, convenience, improved access, and youth famil- sions, and emotional support. Moreover, experts regard TMH as iarity with the modality. However, wearable technologies were cited useful for education and screening, agreeing that TMH could be as beneficial only in terms of familiarity and convenience. These used as a supplement to traditional, face-to-face therapy. However, findings have at least 2 general, practical implications. First, just as notably, there was no consensus on crisis management, which there is no one-size-fits-all approach to face-to-face therapy, like- emphasizes that although adult experts see potential uses for TMH wise, there does not seem to be a single solution for TMH for youth. technology, they have not yet fully embraced TMH as a critical Consequently, developers should think creatively, remaining open treatment mechanism. As comprehensive reviews on youth TMH to the benefits and barriers of different delivery methods, and Downloaded from https://academic.oup.com/jamiaopen/article-abstract/1/1/67/4980801 by Ed 'DeepDyve' Gillespie user on 07 November 2018 72 JAMIA Open, 2018, Vol. 1, No. 1 Table 4. Round 3 expert consensus: telemental health uses, comfort, and openness to using telemental health, pathways to telemental health for youth ages 14–24 years Strongly disagree Agree or strongly Question or disagree % Neutral % agree % Consensus Technology uses for youth not seeking mental health care Education; Information 0 0 100 0.83 Emotional support; encouragement 0 8 92 0.80 Screening (for depression, anxiety, substance abuse) 0 13 88 0.79 Mental health care components potentially supported by technology Education; information 0 0 100 0.86 Emotional support; encouragement 0 0 100 0.81 Self-tracking (behaviors, mood, symptoms, feelings) 0 0 100 0.81 Screening (for depression, anxiety, substance abuse) 0 4 96 0.79 Monitoring progress; giving feedback, support 0 8 92 0.78 Effective way to connect youth to mental health resources Parents 0 0 100 0.81 Friends 0 12 88 0.75 Grandparents 4 4 92 0.78 Siblings 0 37 63 0.76 Aunts/uncles 0 17 83 0.79 Note: Percentage of participants by response category and consensus score (n ¼24). developing alternative mechanisms for addressing different types of of these adult experts regarded TMH as beneficial for mental health mental health problems within different populations. Ideally, a cen- screening, education, and support, but there was no single TMH de- tralized platform of available TMH technologies (ie a TMH toolkit livery method that was universally endorsed. Further, although adult for providers) would be created that highlights the different fea- experts expressed openness to using TMH with youth, they were tures of each TMH tool, which providers, school officials, parents, not in agreement about the use of TMH in crisis management, and peers could draw from as needed. Second, as technology which exemplifies an oft-overlooked stigma that plagues mental changes rapidly, developers must be mindful that today’s familiar health care—that online therapies are not as effective as face-to-face and convenient technologies may soon be pass e. Thus, adaptability treatment. As TMH research with youth populations continues to in this market is critical, and health care delivery systems must find emerge, parents, therapists, and youth must remain open to the a mechanism to help embedded mental health care providers be ever-shifting landscape of mental health care and the possibilities attuned to shifts in new technology offerings that are appealing to that TMH may provide. Moreover, health care information technol- youth. ogists must be at the forefront of these shifts, providing the research The present study has a number of limitations, primarily the and tools that these stakeholders need to make informed decisions limited sample size and the lack of youths’ perspectives to corrobo- about which TMH technologies are best suited for youth with vari- rate results. Additionally, the sample was mostly Caucasian and fe- ous mental health care needs. male, recruited from one state in the United States, which may limit the generalizability of the findings to other demographic and geographic areas. That said, the Delphi method was successful in CONTRIBUTORS soliciting a consensus view from adult experts, an exploratory ap- proach that can lay the groundwork for surveys, focus groups, and Tammy Toscos, Maria Carpenter, Elizabeth Chen, and Catherine additional Delphi panels with larger, more demographically di- Duchovic helped to design and conduct this study. Abigail How- verse samples, including youth. Specifically, future studies should ard, Mindy Flanagan, and Michelle Drouin analyzed and inter- examine the extent to which adults and youth share similar views preted the data and wrote the manuscript with input from all of TMH benefits and drawbacks in order to understand how bar- authors. Tammy Toscos was in charge of overall direction and riers may be created or overcome with technology. Further, as planning of the project. Additionally, all authors drafted or revised adults are often the gatekeepers to youth mental health care, it will this manuscript critically for important intellectual content, ap- be important to develop educational programs and/or advertise- proved the version to be published, and agree to be accountable for ments to bridge any disparate perceptions of TMH among adults all aspects of the work in ensuring that questions related to the ac- and youth. curacy or integrity of any part of the work are appropriately inves- tigated and resolved. CONCLUSION FUNDING Our Delphi approach showed that adult experts in youth mental health (ie those with lived experience with youth mental health This work was supported by the Robert Wood Johnson Foundation grant issues and mental health professionals) perceive significant barriers number 73055. to youth accessing mental health services. Unfortunately, they did not view TMH as a panacea to those barriers. Instead, the majority Conflict of interest statement. None declared. Downloaded from https://academic.oup.com/jamiaopen/article-abstract/1/1/67/4980801 by Ed 'DeepDyve' Gillespie user on 07 November 2018 JAMIA Open, 2018, Vol. 1, No. 1 73 18. Lucassen MFG, Hatcher S, Stasiak K, et al. The views of lesbian, gay and REFERENCES bisexual youth regarding computerised self-help for depression: an explor- 1. Merikangas KR, He JP, Burstein M, et al. Service utilization for lifetime atory study. Adv Mental Health 2013;12(1):22–33. mental disorders in U.S. adolescents: results of the National Comorbidity 19. Bobier C, Stasiak K, Mountford H, et al. When ‘e’ therapy enters the hos- Survey-Adolescent Supplement (NCS-A). J Am Acad Child Adolesc Psy- pital: Examination of the feasibility and acceptability of SPARX (a cCBT chiatry 2011;50(1):32–45. programme) in an adolescent inpatient unit. Advances in Mental Health 2. Goldstein F, Myers K. Telemental health: a new collaboration for pediatri- 2013;11(3):286–92. cians and child psychiatrists. Pediatr Ann 2014;43(2):79–84. 20. Kenny R, Dooley B, Fitzgerald A. Feasibility of “CopeSmart”: a telemen- 3. American Psychology Association. 2008: APA survey of psychology tal health app for adolescents. JMIR Ment Health 2015;2(3):e22. health service providers. http://www.apa.org/workforce/publications/15- 21. Akins RB, Tolson H, Cole BR. Stability of response characteristics of a health-service-providers/index.aspx. Accessed December 28, 2017. Delphi panel: application of bootstrap data expansion. BMC Med Res 4. Boydell KM, Pong R, Volpe T, et al. Family perspectives on pathways to Methodol 2005;5(1):37–49. mental health care for children and youth in rural communities. J Rural 22. Hsu C, Sandford BA. The Delphi technique: making sense of consensus. Health 2006;22(2):182–8. Pract Assess Res Eval 2007;12(10):1–8. 5. Myers KM, Vander Stoep A, McCarty CA, et al. Child and adolescent tele- 23. Jorm AF. Using the Delphi expert consensus method in mental health re- psychiatry: variations in utilization, referral patterns and practice trends. J search. Aust N Z J Psychiatry 2015;49(10):887–97. Telemed Telecare 2010;16(3):128–33. 24. Ross AM, Kelly CM, Jorm AF. Re-development of mental health first aid 6. Stamm B. Rural Behavioral Health Care: An Interdisciplinary Guide. guidelines for suicidal ideation and behaviour: a Delphi study. BMC Psy- American Psychological Association; 2003. chiatry 2014;14(1):241. 7. Lambert DZE, Lenardson JD. Rural Children Don’t Receive the Mental 25. Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psy- Health Care They Need (Research & Policy Brief). Portland, ME: Univer- chol 2006;3(2):77–101. sity of Southern Maine, Muskie School of Public Service, Maine Rural 25. Tastle WJ, Wierman MJ. Consensus and dissention: a measure of ordinal Health Research Center; 2009. dispersion. Int J Approx Reasoning 2007;45(3):531–45. 8. Nelson EL, Bui T. Rural telepsychology services for children and adoles- 26. Polaha J, Williams SL, Heflinger CA, et al. The perceived stigma of mental cents. J Clin Psychol 2010;66(5):490–501. health services among rural parents of children with psychosocial con- 9. Goldstein F, Glueck D. Developing rapport and therapeutic alliance dur- cerns. J Pediatr Psychol 2015;40(10):1095–104. doi: 10.1093/jpepsy/ ing telemental health sessions with children and adolescents. J Child Ado- jsv054. lesc Psychopharmacol 2016;26(3):204–11. 27. Faulkner LR, Juul D, Andrade NN, et al. Recent trends in American Board 10. Pruitt LD, Luxton DD, Shore P. Additional clinical benefits of home- of Psychiatry and neurology psychiatric subspecialties. Acad Psychiatry based telemental health treatments. Prof Psychol: Res Pr 2011;35(1):35–9. 2014;45(5):340–6. 28. Comer JS, Furr JM, Cooper-Vince CE, et al. Internet-delivered, family- 11. Birnbaum ML, Rizvi AF, Correll CU, et al. Role of social media and the based treatment for early-onset OCD: a preliminary case series. J Clin Internet in pathways to care for adolescents and young adults with psy- Child Adolesc Psychol 2014;43(1):74–87. chotic disorders and non-psychotic mood disorders. Early Interv Psychia- 29. Thomas CR, Holzer CE. The continuing shortage of child and try 2017;11(4):290–5. adolescent psychiatrists. J Am Acad Child Adolesc Psychiatry 2006;45(9): 12. Pew Research Center. Mobile phone ownership. http://www.pewinternet.org/ 1023–31. chart/mobile-phone-ownership. Accessed December 28, 2017. 30. Thomas L, Capistrant G. American Telemedicine Association. 50 state 13. Hilty DM, Ferrer DC, Parish MB, et al. The effectiveness of telemental telemedicine gaps analysis coverage & reimbursement American Telemed- health: a 2013 review. Telemed J E Health 2013;19(6):444–54. icine Association. Secondary 50 state telemedicine gaps analysis coverage 14. Gloff NE, LeNoue SR, Novins DK, et al. Telemental health for children & reimbursement. http://www.americantelemed.org/docs/default-source/ and adolescents. Int Rev Psychiatry 2015;27(6):513–24. policy/50-state-telemedicine-gaps-analysis—coverage-and-reimbursement. 15. Myers K, Nelson E-L, Rabinowitz T, et al. American Telemedicine Associ- pdf? sfvrsn¼8. Accessed December 28, 2017. ation practice guidelines for telemental health with children and adoles- 31. Reynolds J, Griffiths KM, Cunningham JA, et al. Clinical practice models cents. Telemed J E Health 2017;23(10):779–804. for the use of e-mental health resources in primary health care by health 16. American Academy of Child and Adolescent Psychiatry (AACAP) Com- professionals and peer workers: a conceptual framework. JMIR Mental mittee on Telepsychiatry and AACAP Committee on Quality Issues. Clini- Health 2015;2(1):e6. cal update: telepsychiatry with children and adolescents. J Am Acad Child 32. Pignatiello A, Teshima J, Boydell KM, Minden D, Volpe T, Braunberger Adolescent Psychiatry 2017;56(10):875–93. PG. Child and Youth Telepsychiatry in Rural and Remote Primary Care. 17. Stasiak K, Fleming T, Lucassen MF, et al. Computer-based and online Child Adolesc Psychiatr Clin N Am 2011;20(1):13–28. therapy for depression and anxiety in children and adolescents. J Child Adolesc Psychopharmacol 2016;26(3):235–45. Downloaded from https://academic.oup.com/jamiaopen/article-abstract/1/1/67/4980801 by Ed 'DeepDyve' Gillespie user on 07 November 2018 74 JAMIA Open, 2018, Vol. 1, No. 1 APPENDIX Appendix A. Quotes from experts about the benefits of online chat, mobile apps, and video conferencing in TMH for youth Delivery method Benefits Online Chat “Current method of communicating with this population” “This can be beneficial if the client is needing to speak to someone immediately and unable to get to the office to see the therapist or is in a situation where this is their only option” “Instant feedback any time you want it” “no need for transportation, easier access, user friendly for age group” “more comfortable for less verbal clients.. .clients who may process more slowly and be able to think before they write out their responses” Mobile Apps “since people are using apps for other areas of their lives, this would seem to be a natural next step for mental health help” “apps are on phone, and most youth always have their phones on them, so services could always be available” “ability to take interventions with them wherever they go, privacy to look at them without people knowing they are looking at coping information” Video Conferencing “ability to talk to someone face to face regardless of transportation” “In areas where there aren’t adequate mental health services this option would be beneficial for people who would other- wise not have the ability to see a therapist in person” “Convenient, no transportation needed, ideally lower cost, ideally can have more schedule options” “No need for transportation, (video conferencing) can reach those with social anxiety in comfortable setting” Appendix B. Table of demographic characteristics of expert panel members Adults with lived Parents of teens with Mental health Mental health clinicians and Educators Total experience lived experience researchers those who work at mental who work with teens/ (n ¼ 25) (n ¼ 5) (n ¼ 4) (n ¼ 5) health organizations (n ¼ 6) young adults (n ¼ 5) Gender Male 20.0% (1) 25.0% (1) 0.0% (0) 16.7% (1) 20.0% (1) 16.0% (4) Female 80.0% (4) 75.0% (3) 100.0% (5) 83.3% (5) 80.0% (4) 84.0% (21) Age 24–34 20.0% (1) 75.0% (3) 0.0% (0) 33.3% (2) 20.0% (1) 28.0% (7) 35–49 20.0% (1) 0.0% (0) 60.0% (3) 33.3% (2) 40.0% (2) 32.0% (8) 50–64 60.0% (3) 25.0% (1) 40.0% (2) 33.3% (2) 40.0% (2) 40.0% (10) Race White 80.0% (4) 100.0% (4) 100.0% (5) 100.0% (6) 80.0% (4) 92.0% (23) Latino 20.0% (1) 0.0% (0) 0.0% (0) 0.0% (0) 0.0% (0) 4.0% (1) Multi-Racial 0.0% (0) 0.0% (0) 0.0% (0) 0.0% (0) 20.0% (1) 4.0% (1) Familiarity with telemental health Yes 80.0% (4) 25.0% (1) 80.0% (4) 33.3% (2) 60.0% (3) 56.0% (14) No 20.0% (1) 75.0% (3) 20.0% (1) 66.7% (4) 40.0% (2) 44.0% (11) Use of telemental health Yes 40.0% (2) 25.0% (1) 0.0% (0) 50.0% (3) 40.0% (2) 32.0% (8) No 60.0% (3) 75.0% (3) 100.0% (5) 50.0% (3) 60.0% (3) 68.0% (17) http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png JAMIA Open Oxford University Press

Adult experts’ perceptions of telemental health for youth: A Delphi study

Free
8 pages

Loading next page...
 
/lp/ou_press/adult-experts-perceptions-of-telemental-health-for-youth-a-delphi-JnduM0cXb3
Copyright
© The Author(s) 2018. Published by Oxford University Press on behalf of the American Medical Informatics Association.
eISSN
2574-2531
D.O.I.
10.1093/jamiaopen/ooy002
Publisher site
See Article on Publisher Site

Abstract

Downloaded from https://academic.oup.com/jamiaopen/article-abstract/1/1/67/4980801 by Ed 'DeepDyve' Gillespie user on 07 November 2018 JAMIA Open, 1(1), 2018, 67–74 doi: 10.1093/jamiaopen/ooy002 Advance Access Publication Date: 20 April 2018 Research and Applications Research and Applications Adult experts’ perceptions of telemental health for youth: A Delphi study 1 1,2 1,3 1 Abigail Howard, Mindy Flanagan, Michelle Drouin, Maria Carpenter, 4 5 1 Elizabeth M. Chen, Catherine Duchovic, and Tammy Toscos 1 2 Parkview Research Center, Parkview Health, Fort Wayne, Indiana, USA, Indiana University Center for Health Services Research, School of Medicine, Indianapolis, Indiana, USA, Indiana University–Purdue University Fort Wayne, Fort Wayne, 4 5 Indiana, USA, Marian University College of Osteopathic Medicine, Indianapolis, Indiana, USA and Park Center, Inc. Fort Wayne, Indiana, USA Corresponding Author: Tammy Toscos, Ph.D., Director of Informatics, 10622 Parkview Plaza Dr, Fort Wayne, Indiana 46845, USA (Tammy.Toscos@parkview.com) Received 2 January 2018; Revised 2 February 2018; Editorial Decision 0 Month 0000; Accepted 22 February 2018 ABSTRACT Objectives: Our objectives were to measure experts’ opinions and develop consensus via the Delphi process on the barriers, applications, and concerns associated with telemental health (TMH) for youth. Materials and methods: We delivered 3 online surveys over 2 months in Summer, 2016–2025 adult experts, in- cluding adults who experienced youth depression or suicidality, parents of youth with lived experience, and professionals (ie youth mental health researchers, clinicians/staff, or educators). We used the Delphi method to construct Likert and open-ended questions, developing expert consensus over 3 iterative surveys on the bar- riers and benefits of TMH for youth. Results: Adult experts identified stigma and knowledge barriers to youth mental health care. Although TMH is perceived as beneficial for screening, education, follow-up, and emotional support, no single delivery method (eg websites or instant messaging) was deemed universally beneficial. Discussion: Adults are the developers, administrators, and gatekeepers of youth mental health care. Although adult experts see potential for TMH to supplement traditional therapy via familiar technologies, there is no con- sensus on the technologies by which TMH should be delivered. However, there is consensus that family mem- bers and friends provide potential pathways to care; thus, an online TMH toolkit for youth would be beneficial for both caretakers and practitioners. Conclusion: Telemental health may not overcome barriers for crisis management but adult experts agreed that TMH had potential benefits for youth. Health care organizations should conduct research and provide training and education to youth caretakers and practitioners on potential barriers and benefits of TMH technologies for youth. Key words: telemedicine, mental health, Delphi technique, delivery of health care, youth INTRODUCTION stigma and lack of knowledge about accessing mental health serv- ices, as well as structural barriers, such as a lack of transportation 3–9 Background and significance and provider shortages (particularly for those in rural settings). Many youth struggling with mental health problems do not receive Telemental health (TMH) technologies—which can be broadly de- 1,2 mental health care. For this group, barriers to gaining access to fined as technologies used to make clinical assessments or to deliver mental health care include both psychological barriers, such as mental health care, education, or information—may offer potential V The Author(s) 2018. Published by Oxford University Press on behalf of the American Medical Informatics Association. This is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/4.0/), which permits non-commercial re-use, distribution, and reproduction in any medium, provided the original work is properly cited. For commercial re-use, please contact journals.permissions@oup.com 67 Downloaded from https://academic.oup.com/jamiaopen/article-abstract/1/1/67/4980801 by Ed 'DeepDyve' Gillespie user on 07 November 2018 68 JAMIA Open, 2018, Vol. 1, No. 1 solutions, providing a means for mental health care that is widely in considering diverse opinions and options and identifying consen- 2–10 21,22 available, less stigmatized, and easier to access. The pervasive- sus knowledge and priorities. As Delphi-based mental health re- ness of daily technology use among adolescents and young adults search is now using both professionals and those with lived 23,24 raises the possibility that TMH technologies may provide a pathway experience as experts, we included both youth mental health for mental health services for these populations. professionals and those with lived experience of youth mental health Over the past decade, in line with the surge of mobile phone use issues (ie adults who experienced depression or suicidal ideation and ownership, researchers have been developing and investigat- during youth and parents of youth with depression or suicidal idea- ing the effectiveness of TMH across a broad range of demographic tion) in our study. groups. Telemental health can include, but is not limited to, the provision of mental health services through telecommunications Objectives technology (including training rural health care workers online), Our goal was to engage a panel of adult experts in the Delphi pro- videoconferencing with consultations on behavioral health issues, cess to yield consensus on the barriers, applications, and concerns “virtual” case management, and smartphone applications targeted associated with TMH for youth. This Delphi study was the first part at teaching mental health skills. A systematic review of this research of broader research project on youth mental health issues and serv- has shown that TMH technologies offer convenience, confidential- ices, with the primary goals of identifying barriers to mental health ity, familiarity, and integration into everyday life. Additionally, care for youth ages 14 to 24 and examining the potential role of patients who utilize TMH display a higher adherence for return TMH technologies in overcoming these barriers and addressing the appointments than with traditional therapy and report high levels of mental health needs of youth. The results of this Delphi study in- patient satisfaction. In short, TMH is effective, and it has been formed later focus group and survey questions delivered to youth, shown to be effective among diverse groups, including youth their families, and caretakers. As adult experts may steer the devel- 13–16 populations. opment, usage, and acceptability of TMH technologies, we saw this With regard to the effectiveness of TMH for youth specifically, Delphi process as an essential step in the further refinement of TMH recent reviews show that telepsychiatry (ie clinicians using videocon- resources for youth. ferencing for psychiatric care) is an effective mental health care deliv- ery method for youth with various psychiatric issues (eg anxiety, attention deficit hyperactivity disorder, oppositional defiant disorder, MATERIALS AND METHODS etc.) in a variety of settings (eg juvenile detention centers, schools, 14–16 Procedure and data collection and inpatient units). For the most part, youth patients who re- As per Parkview Health’s Institutional Review Board approval, pro- ceived telepsychiatric care had comparable outcomes to patients re- spective panel members were recruited through listservs of 5 local ceiving face-to-face services, and both parents and youth expressed 14–16 mental health organizations in Northeast Indiana in Summer, 2016. satisfaction with TMH. However, some participants expressed These mental health organizations served both urban and rural com- concerns about the modality, such as privacy issues and video limita- munities in the largest county in this region. All listserv recipients re- tions. Moreover, telepsychiatry represents only one type of TMH ceived a secure link to a prequestionnaire, where they were directed technology; there are numerous applications and websites that are to select categorizations that applied to them from a list of lived now available that also offer mental health support to consumers experiences (eg parent of youth with depression or suicidal ideation) without therapist engagement. These programs, which are often and professional backgrounds (eg youth mental health clinician, re- based on cognitive behavioral therapy (CBT) principles and delivered searcher, or educator). As per our study protocol, 40 individuals via interactive websites and multimedia applications, have proven were selected randomly after the 260 replies were screened for meet- effective for reducing anxiety and depression among youth. ing inclusion criteria. Of the 40 adult experts identified for possible Youth-based TMH usability research is far less advanced than participation, 25 consented to participate in the Delphi panel pro- effectiveness research. In the studies that have measured benefits cess. Panel members were compensated with $30 Amazon gift cards and barriers of specific technologies, youth have responded with 18,19 20 after completing the first and third surveys. positive and negative feedback (eg SPARX and CopeSmart ). For example, adolescents stated that while CopeSmart (a TMH smartphone application focused on emotional self-monitoring and Participants positive coping strategies) was easy to use and provided helpful in- Overall, 25 experts (21 women, 4 men) consented to participate via formation, it was not very engaging or effective. However, these an online informed consent process, and 24 participated in all 3 findings are difficult to generalize, as the users’ opinions were fo- rounds of surveys. Most of the expert panel members identified as cused on one specific technology as compared to TMH, generally. non-Hispanic, White (92%), and 8% identified as another ethnicity. Therefore, more generalized assessments of barriers and benefits for In terms of age groups represented, 28% of the sample was 24–34, youth-based TMH are necessary. 32% of the sample was 35–49, and 40% of the sample was 50–64. As TMH use among youth is not yet widespread, opinions from With regard to their expert role, 4 people (16%) identified them- adults (who have experience with mental health issues, mental selves as having lived experience (ie adult with experience of youth health care, or both) may provide critical information for developing depression or suicidality or parent of youth with lived experience), and refining youth-based TMH technologies. Securing feedback 12 (48%) identified themselves as professionals (ie mental health re- from these stakeholders aligns with Brofenbrenner’s ecological sys- searcher, mental health clinician/staff at mental health organization, tems theory (1979), which posits that youth are affected by various educator that works with youth), and 9 (36%) as having both lived systems, including the family and wider culture. Within cultural con- experience and experience working in a professional capacity with texts, expert opinions are paramount for decision-making; therefore, youth. Finally, in terms of their existing knowledge of TMH, 14 ex- in this study, we employed the Delphi method using adult experts in pert panel members (56%) were familiar with TMH, and 8 expert youth mental health. The Delphi method has proven highly effective panel members (32%) had previously used TMH. Downloaded from https://academic.oup.com/jamiaopen/article-abstract/1/1/67/4980801 by Ed 'DeepDyve' Gillespie user on 07 November 2018 JAMIA Open, 2018, Vol. 1, No. 1 69 Measures comfortable using TMH to supplement therapy. All items used 5- point Likert response sets (ie 1 ¼ strongly disagree,5 ¼ strongly Expert panel members completed surveys online and agree;1 ¼ no, absolutely not,5 ¼ yes, definitely). anonymously—a distinguishing feature of the Delphi method—to help avoid pitfalls of face-to-face meetings (eg group think, irratio- nal adherence to an opinion, etc.) . Surveys included quantitative Analytic strategy and/or qualitative questions and were created via iterative processes, Quantitative data analysis included calculating descriptive statistics based on previous survey replies. (frequency, mean) for demographic characteristics and survey items. The percentage of responses by valence (1 ¼ strongly disagree, dis- Survey 1: broadly identifying barriers and telemental health agree;2 ¼ neutral;3 ¼ agree, strongly agree) was calculated for each usefulness for youth item. Additionally, a consensus score was calculated for each item Round 1 survey questions were created by the research team based which quantifies proximities in ordinal scales and consistent on consensus, literature review, and clinical experiences. Panel mem- responses between experts. Consensus was defined as: bers indicated the extent to which they viewed items as barriers for youth receiving mental health services (ie stigma, cost, transporta- jX  l j Consensus ¼ 1 þ p log 1 tion). For this same list of barriers, panel members indicated the ex- d i¼1 tent to which these barriers to youth receiving services could be In this equation, p is the probability for each Likert score; l is i X overcome with TMH. Panel members indicated the extent to which the mean of X; d is the possible range of X, d ¼ X  X (eg X X max min they viewed items as benefits of traditional in-person therapy com- 5–1). Here, a higher consensus score is reached between proximal pared to TMH (ie easier to talk in person, more comfortable in per- responses (ie strongly agree and agree rating); dissension is evident son). Panel members also rated potential benefits of incorporating with wider dispersion between responses (ie strongly agree and TMH for youth (ie scheduling appointments, positive text messages, strongly disagree). Consensus was operationalized as consensus mood assessment). For these closed-ended questions, panel members scores at 0.75 or above. At this score, no more than one expert can indicated agreement using a 5-point Likert scale (1 ¼ not at all; respond with an oppositely valenced rating than the others (with the 5 ¼ very much). The remaining 10 questions were open-ended, fo- exception for scores clustered around 3, but this case was not ob- cused on drawbacks and benefits of TMH delivery methods (ie video served). Below this score, experts’ ratings were distributed across a conferencing, websites, social media), concerns and benefits of using wider range of agreement/disagreement scores or had a few people TMH for youth, personal experience witnessing benefits and nega- who strongly disagreed with statements (while the rest agreed or tive outcomes for youth TMH use, and potential for TMH to im- were neutral). Data analyzes for this article were conducted using prove access to mental health services for youth. Microsoft Excel and SAS 9.4 software. Survey 2: refining barriers and benefits of telemental health for youth RESULTS Thematic analysis was conducted on open-ended responses from Round 1 to identify consistencies in responses about concerns and Perceptions of telemental health for youth benefits of TMH for youth. These responses were used to create Round 1 responses revealed that the majority of experts perceived closed-ended questions for the Round 2 survey, using 5-point Likert barriers for youth mental health access. With regard to specific bar- scales (1 ¼ not at all,5 ¼ very much;1 ¼ strongly disagree, riers, experts were in consensus that stigma (92%) and knowledge 5 ¼ strongly agree). Items assessed were benefits of seeing a therapist about where or how to obtain services (80%) were primary barriers in person (vs online), benefits of using technology to address mental to youth seeking mental health care (see Table 1). However, there health needs in youth, concerns of using technology to address youth was no consensus on the barriers that TMH tools could overcome. mental health needs, and benefits of using technology to supplement Among open-ended responses, 7 respondents indicated parents in-person therapy. Additionally, benefits of different TMH delivery may be a barrier for youth seeking mental health services. In one methods for youth were assessed on various parameters (see case, a mental health professional stated: Table 3). “In many cases parents need to give permission, especially for ac- cess to insurance, and they often do not see the importance or Survey 3: reaching consensus for the benefits of telemental health need for their teen to access services. Often youth do not share for youth with their parents that they may feel the need to access these serv- The goal of the Round 3 survey was consensus among panel mem- ices and it is often difficult for them to access services on their bers about prominent benefits and concerns of TMH use for youth. own.” For granularity, we separated some responses into individual items (eg “parents, friends, family” was divided into 3 individual items) Meanwhile, with regard to benefits of TMH, experts were in and converted others from qualitative to quantitative items (eg par- consensus that positive text messages, automatic messaging before ticular uses of TMH). Panel members indicated to what extent they appointments, and self-management for anxiety were beneficial thought TMH could support the following aspects of health care for applications of TMH tools for youth. Experts also agreed that youth: crisis support, emotional support, education, self-tracking, parents or other family, physicians or therapists, and faith leaders screenings, monitoring progress, coping tools, connecting people have responsibility to connect youth to mental health services. Col- who are struggling, and virtual support groups. Another item lege student counselors were viewed as responsible for connecting assessed which individuals were effective in connecting youth to college-aged youth to mental health services. Notably, experts did mental health resources. Additionally, panel members indicated not show consensus on youth benefits of seeing a therapist in person the extent to which they were open to using, planning to use, and (vs online). Downloaded from https://academic.oup.com/jamiaopen/article-abstract/1/1/67/4980801 by Ed 'DeepDyve' Gillespie user on 07 November 2018 70 JAMIA Open, 2018, Vol. 1, No. 1 Table 1. Round 1 expert consensus: barriers, benefits, and responsibility for telemental health for youth ages 14–24 years Question Strongly disagree or disagree % Neutral % Agree or strongly agree % Consensus Barriers to access mental health services Stigma 0 8 92 0.77 Knowledge about where/how to obtain services 4 16 80 0.80 Beneficial uses of TMH Positive text messages 0 4 96 0.79 Automatic messaging before an appointment 0 8 92 0.82 Self-manage anxiety 4 24 72 0.76 Who youth trust to help them get mental health care Internet/search engine 0 32 68 0.78 Responsible party to ensure high school students receive mental health services Parents/guardians/siblings/other relatives 0 4 96 0.85 Physician/therapist 0 12 88 0.75 Faith leader 4 24 72 0.77 Responsible party to ensure college students receive mental health services College student counseling center 0 12 88 0.77 Note: Percentage of participants by response category and consensus score (n ¼ 25). Among round 2 responses, experts agreed on the following appli- following components: education (100%), emotional support cations of technology to supplement therapy: follow-up on therapy- (100%), self-tracking (100%), screening (96%), and monitoring related assignments, enhance medication adherence, emotional sup- progress (92%). However, experts did not reach consensus on using port, questions for therapist between sessions, and appointment TMH tools for crisis support (either in favor or opposed; con- reminders (See Table 2). Meanwhile, in open-ended responses, panel sensus¼ 0.52 for crisis support with youth not receiving care, con- members identified TMH as convenient for continuous access at all sensus¼ 0.61 for crisis support with youth receiving care). Most times of day and night. A mental health professional indicated in an experts were open to using technology to supplement traditional, open-ended response: face-to-face therapy (88%) and planned to use TMH tools in the fu- ture with youth (75%). “24-hour responses to crisis situations, this age group may be comfortable send texts/talking later at night when negative thoughts occur before sleep.” DISCUSSION The benefits of TMH delivery methods are displayed in Table 3. Notably, experts did not consistently indicate that it is easier for Youth engagement with TMH resources will depend, at least par- youth to share emotions (“open up”) via technology for any of the tially, on buy-in from adult stakeholders who develop, recommend, TMH delivery methods assessed. Additionally, across TMH delivery and administer these mental health technologies. Through an itera- methods, the inability to observe nonverbal communication was tive Delphi process, youth mental health experts, including profes- viewed as a drawback (for online chat, websites, social media, and sionals and those with lived experience of youth mental health text messaging). Experts viewed technological problems as a draw- issues, identified various benefits and barriers to TMH for youth. back to video conferencing and computer literacy/computer access Most experts agreed that stigma is a primary barrier to youth problems as a drawback to websites. Experts did not reach consen- accessing mental health services. Unfortunately, previous research sus on which TMH delivery method would be most beneficial and has shown that parents’ perceptions of stigma are associated with with the least drawbacks for youth. less willingness to seek services for their children ; therefore, this In open-ended responses, many panel members expressed that overarching stigma may affect uptake of mental health services, gen- the usefulness of TMH services would depend on the youth and the erally, and TMH technologies, specifically. More importantly, al- situation. For example: though our experts acknowledged stigma as a barrier to mental health treatment, they did not reach consensus that TMH overcomes “I am not sure if video conferencing will help my client open up stigma. Hence, at least from the viewpoint of adult experts, TMH more, but I suppose it just depends on the personality of the does not abolish this significant barrier to youth mental health client.” treatment. Poor knowledge about obtaining services was also viewed as a “(TMH services could be) helpful for people uncomfortable with face to face communication.” barrier to youth mental health treatment. These findings are consis- tent with other research indicating there are pragmatic barriers that “Impacts youth who live in poverty with no computer or internet 28–31 prevent youth from receiving services. However, the majority service.” of our panel members were familiar with TMH, and 32% had used See Appendix A for additional responses related to each technology. TMH, suggesting that TMH is gaining popularity among adult By Round 3, experts agreed that TMH could help youth who are stakeholders. Future education and marketing efforts concerning not engaged with traditional face-to-face mental health in the TMH for youth should be targeted towards family members, peers, following ways: education (100%), emotional support (92%), and therapists, faith leaders, and college counseling centers, parties screening (88%, see Table 4). Additionally, experts indicated deemed by our experts as responsible for connecting youth to mental that mental health care could be supported by technology for the health resources. Health care organizations may be the ideal vehicle Downloaded from https://academic.oup.com/jamiaopen/article-abstract/1/1/67/4980801 by Ed 'DeepDyve' Gillespie user on 07 November 2018 JAMIA Open, 2018, Vol. 1, No. 1 71 Table 2. Round 2 expert consensus: benefits of telemental health for youth ages 14–24 years Strongly disagree Agree or strongly or disagree % Neutral % agree % Consensus Benefit of technology supplementing in-person therapy Follow-up on therapy-related assignments 0 4 96 0.80 Enhance medication adherence through messages, reminders, and education 0 8 92 0.77 Emotional support/encouragement 0 8 92 0.79 Ask therapist questions between sessions 4 0 96 0.77 Appointment reminders 0 0 100 0.97 Note: Percentage of participants by response category and consensus score (n ¼ 24). Table 3. Round 2 expert consensus (consensus> 0.75): benefits of TMH delivery methods for youth ages 14–24 years Assessed for benefit but no consensus that TMH Benefit Beneficial TMH delivery methods method provides benefit Improved access to mental health care Video conference, Online chat/Instant Message, Websites, Social media, Wearable technologies Mobile apps, Text messaging Easier to open up via technology (vs face- No consensus Video conference, Online chat/Instant Message, to-face) Mobile apps, Websites, Text messaging, Wearable technologies Emotional/social support No consensus Video conference, social media, text messaging Youth familiarity for technology-based Online chat/Instant Message, Mobile apps, Websites communication Social media, Text messaging, Wearable technologies Quick response Online chat/Instant Message, Text messaging Mobile apps, Websites, Social media, Wearable technologies Anonymity No consensus Online chat/Instant Message, Mobile apps Convenience Online chat/Instant Message, Mobile apps, Websites Wearable technologies Educational/raise awareness Websites, Social media Mobile apps, Text messaging Early intervention tool No consensus Websites Motivational No consensus Text messaging, Wearable technologies Addresses stigma Video conference Note: Not all TMH technologies were assessed for each benefit (n ¼ 24). 14–16 for such promotion, as they have connections in the community and are quite recent, it may take some time for the positive reviews monetary resources for large-scale health education initiatives. of TMH and accompanying guidelines to percolate through the Health care organizations are also in a prime position to provide health care system to clinicians and consumers. However, there also support to rural health practitioners for implementation and use of needs to be more focus on whether TMH is suitable for acute care, TMH in their practice, which could be particularly helpful in over- as experts’ opinions on it are clearly divided. coming access barriers for rural youth. Within the health care sys- Telemental health is delivered through a variety of methods, tem, TMH education could entail workshops or conferences, but it from websites, to apps, to videoconferencing; additionally, gaming might also be provided through secure messaging via patient portals and virtual reality are being explored as potential delivery modali- in primary care/pediatric offices. Future research should explore the ties. From an informatics standpoint, we expected that experts’ effectiveness of these alternate avenues for TMH education within feedback on delivery methods might prove most useful for TMH de- different populations. velopment. However, unfortunately, there was no one modality that With regard to benefits, experts were in agreement that TMH was endorsed universally as having beneficial features. Instead, ben- technologies could be helpful for certain aspects of mental health efits depended on the medium. For example, experts were in agree- treatment, such as appointment and medication reminders, connect- ment that online chat/instant messaging provided benefits in terms ing with therapists and completing follow-up work between ses- of improved access, convenience, improved access, and youth famil- sions, and emotional support. Moreover, experts regard TMH as iarity with the modality. However, wearable technologies were cited useful for education and screening, agreeing that TMH could be as beneficial only in terms of familiarity and convenience. These used as a supplement to traditional, face-to-face therapy. However, findings have at least 2 general, practical implications. First, just as notably, there was no consensus on crisis management, which there is no one-size-fits-all approach to face-to-face therapy, like- emphasizes that although adult experts see potential uses for TMH wise, there does not seem to be a single solution for TMH for youth. technology, they have not yet fully embraced TMH as a critical Consequently, developers should think creatively, remaining open treatment mechanism. As comprehensive reviews on youth TMH to the benefits and barriers of different delivery methods, and Downloaded from https://academic.oup.com/jamiaopen/article-abstract/1/1/67/4980801 by Ed 'DeepDyve' Gillespie user on 07 November 2018 72 JAMIA Open, 2018, Vol. 1, No. 1 Table 4. Round 3 expert consensus: telemental health uses, comfort, and openness to using telemental health, pathways to telemental health for youth ages 14–24 years Strongly disagree Agree or strongly Question or disagree % Neutral % agree % Consensus Technology uses for youth not seeking mental health care Education; Information 0 0 100 0.83 Emotional support; encouragement 0 8 92 0.80 Screening (for depression, anxiety, substance abuse) 0 13 88 0.79 Mental health care components potentially supported by technology Education; information 0 0 100 0.86 Emotional support; encouragement 0 0 100 0.81 Self-tracking (behaviors, mood, symptoms, feelings) 0 0 100 0.81 Screening (for depression, anxiety, substance abuse) 0 4 96 0.79 Monitoring progress; giving feedback, support 0 8 92 0.78 Effective way to connect youth to mental health resources Parents 0 0 100 0.81 Friends 0 12 88 0.75 Grandparents 4 4 92 0.78 Siblings 0 37 63 0.76 Aunts/uncles 0 17 83 0.79 Note: Percentage of participants by response category and consensus score (n ¼24). developing alternative mechanisms for addressing different types of of these adult experts regarded TMH as beneficial for mental health mental health problems within different populations. Ideally, a cen- screening, education, and support, but there was no single TMH de- tralized platform of available TMH technologies (ie a TMH toolkit livery method that was universally endorsed. Further, although adult for providers) would be created that highlights the different fea- experts expressed openness to using TMH with youth, they were tures of each TMH tool, which providers, school officials, parents, not in agreement about the use of TMH in crisis management, and peers could draw from as needed. Second, as technology which exemplifies an oft-overlooked stigma that plagues mental changes rapidly, developers must be mindful that today’s familiar health care—that online therapies are not as effective as face-to-face and convenient technologies may soon be pass e. Thus, adaptability treatment. As TMH research with youth populations continues to in this market is critical, and health care delivery systems must find emerge, parents, therapists, and youth must remain open to the a mechanism to help embedded mental health care providers be ever-shifting landscape of mental health care and the possibilities attuned to shifts in new technology offerings that are appealing to that TMH may provide. Moreover, health care information technol- youth. ogists must be at the forefront of these shifts, providing the research The present study has a number of limitations, primarily the and tools that these stakeholders need to make informed decisions limited sample size and the lack of youths’ perspectives to corrobo- about which TMH technologies are best suited for youth with vari- rate results. Additionally, the sample was mostly Caucasian and fe- ous mental health care needs. male, recruited from one state in the United States, which may limit the generalizability of the findings to other demographic and geographic areas. That said, the Delphi method was successful in CONTRIBUTORS soliciting a consensus view from adult experts, an exploratory ap- proach that can lay the groundwork for surveys, focus groups, and Tammy Toscos, Maria Carpenter, Elizabeth Chen, and Catherine additional Delphi panels with larger, more demographically di- Duchovic helped to design and conduct this study. Abigail How- verse samples, including youth. Specifically, future studies should ard, Mindy Flanagan, and Michelle Drouin analyzed and inter- examine the extent to which adults and youth share similar views preted the data and wrote the manuscript with input from all of TMH benefits and drawbacks in order to understand how bar- authors. Tammy Toscos was in charge of overall direction and riers may be created or overcome with technology. Further, as planning of the project. Additionally, all authors drafted or revised adults are often the gatekeepers to youth mental health care, it will this manuscript critically for important intellectual content, ap- be important to develop educational programs and/or advertise- proved the version to be published, and agree to be accountable for ments to bridge any disparate perceptions of TMH among adults all aspects of the work in ensuring that questions related to the ac- and youth. curacy or integrity of any part of the work are appropriately inves- tigated and resolved. CONCLUSION FUNDING Our Delphi approach showed that adult experts in youth mental health (ie those with lived experience with youth mental health This work was supported by the Robert Wood Johnson Foundation grant issues and mental health professionals) perceive significant barriers number 73055. to youth accessing mental health services. Unfortunately, they did not view TMH as a panacea to those barriers. Instead, the majority Conflict of interest statement. None declared. Downloaded from https://academic.oup.com/jamiaopen/article-abstract/1/1/67/4980801 by Ed 'DeepDyve' Gillespie user on 07 November 2018 JAMIA Open, 2018, Vol. 1, No. 1 73 18. Lucassen MFG, Hatcher S, Stasiak K, et al. The views of lesbian, gay and REFERENCES bisexual youth regarding computerised self-help for depression: an explor- 1. Merikangas KR, He JP, Burstein M, et al. Service utilization for lifetime atory study. Adv Mental Health 2013;12(1):22–33. mental disorders in U.S. adolescents: results of the National Comorbidity 19. Bobier C, Stasiak K, Mountford H, et al. When ‘e’ therapy enters the hos- Survey-Adolescent Supplement (NCS-A). J Am Acad Child Adolesc Psy- pital: Examination of the feasibility and acceptability of SPARX (a cCBT chiatry 2011;50(1):32–45. programme) in an adolescent inpatient unit. Advances in Mental Health 2. Goldstein F, Myers K. Telemental health: a new collaboration for pediatri- 2013;11(3):286–92. cians and child psychiatrists. Pediatr Ann 2014;43(2):79–84. 20. Kenny R, Dooley B, Fitzgerald A. Feasibility of “CopeSmart”: a telemen- 3. American Psychology Association. 2008: APA survey of psychology tal health app for adolescents. JMIR Ment Health 2015;2(3):e22. health service providers. http://www.apa.org/workforce/publications/15- 21. Akins RB, Tolson H, Cole BR. Stability of response characteristics of a health-service-providers/index.aspx. Accessed December 28, 2017. Delphi panel: application of bootstrap data expansion. BMC Med Res 4. Boydell KM, Pong R, Volpe T, et al. Family perspectives on pathways to Methodol 2005;5(1):37–49. mental health care for children and youth in rural communities. J Rural 22. Hsu C, Sandford BA. The Delphi technique: making sense of consensus. Health 2006;22(2):182–8. Pract Assess Res Eval 2007;12(10):1–8. 5. Myers KM, Vander Stoep A, McCarty CA, et al. Child and adolescent tele- 23. Jorm AF. Using the Delphi expert consensus method in mental health re- psychiatry: variations in utilization, referral patterns and practice trends. J search. Aust N Z J Psychiatry 2015;49(10):887–97. Telemed Telecare 2010;16(3):128–33. 24. Ross AM, Kelly CM, Jorm AF. Re-development of mental health first aid 6. Stamm B. Rural Behavioral Health Care: An Interdisciplinary Guide. guidelines for suicidal ideation and behaviour: a Delphi study. BMC Psy- American Psychological Association; 2003. chiatry 2014;14(1):241. 7. Lambert DZE, Lenardson JD. Rural Children Don’t Receive the Mental 25. Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psy- Health Care They Need (Research & Policy Brief). Portland, ME: Univer- chol 2006;3(2):77–101. sity of Southern Maine, Muskie School of Public Service, Maine Rural 25. Tastle WJ, Wierman MJ. Consensus and dissention: a measure of ordinal Health Research Center; 2009. dispersion. Int J Approx Reasoning 2007;45(3):531–45. 8. Nelson EL, Bui T. Rural telepsychology services for children and adoles- 26. Polaha J, Williams SL, Heflinger CA, et al. The perceived stigma of mental cents. J Clin Psychol 2010;66(5):490–501. health services among rural parents of children with psychosocial con- 9. Goldstein F, Glueck D. Developing rapport and therapeutic alliance dur- cerns. J Pediatr Psychol 2015;40(10):1095–104. doi: 10.1093/jpepsy/ ing telemental health sessions with children and adolescents. J Child Ado- jsv054. lesc Psychopharmacol 2016;26(3):204–11. 27. Faulkner LR, Juul D, Andrade NN, et al. Recent trends in American Board 10. Pruitt LD, Luxton DD, Shore P. Additional clinical benefits of home- of Psychiatry and neurology psychiatric subspecialties. Acad Psychiatry based telemental health treatments. Prof Psychol: Res Pr 2011;35(1):35–9. 2014;45(5):340–6. 28. Comer JS, Furr JM, Cooper-Vince CE, et al. Internet-delivered, family- 11. Birnbaum ML, Rizvi AF, Correll CU, et al. Role of social media and the based treatment for early-onset OCD: a preliminary case series. J Clin Internet in pathways to care for adolescents and young adults with psy- Child Adolesc Psychol 2014;43(1):74–87. chotic disorders and non-psychotic mood disorders. Early Interv Psychia- 29. Thomas CR, Holzer CE. The continuing shortage of child and try 2017;11(4):290–5. adolescent psychiatrists. J Am Acad Child Adolesc Psychiatry 2006;45(9): 12. Pew Research Center. Mobile phone ownership. http://www.pewinternet.org/ 1023–31. chart/mobile-phone-ownership. Accessed December 28, 2017. 30. Thomas L, Capistrant G. American Telemedicine Association. 50 state 13. Hilty DM, Ferrer DC, Parish MB, et al. The effectiveness of telemental telemedicine gaps analysis coverage & reimbursement American Telemed- health: a 2013 review. Telemed J E Health 2013;19(6):444–54. icine Association. Secondary 50 state telemedicine gaps analysis coverage 14. Gloff NE, LeNoue SR, Novins DK, et al. Telemental health for children & reimbursement. http://www.americantelemed.org/docs/default-source/ and adolescents. Int Rev Psychiatry 2015;27(6):513–24. policy/50-state-telemedicine-gaps-analysis—coverage-and-reimbursement. 15. Myers K, Nelson E-L, Rabinowitz T, et al. American Telemedicine Associ- pdf? sfvrsn¼8. Accessed December 28, 2017. ation practice guidelines for telemental health with children and adoles- 31. Reynolds J, Griffiths KM, Cunningham JA, et al. Clinical practice models cents. Telemed J E Health 2017;23(10):779–804. for the use of e-mental health resources in primary health care by health 16. American Academy of Child and Adolescent Psychiatry (AACAP) Com- professionals and peer workers: a conceptual framework. JMIR Mental mittee on Telepsychiatry and AACAP Committee on Quality Issues. Clini- Health 2015;2(1):e6. cal update: telepsychiatry with children and adolescents. J Am Acad Child 32. Pignatiello A, Teshima J, Boydell KM, Minden D, Volpe T, Braunberger Adolescent Psychiatry 2017;56(10):875–93. PG. Child and Youth Telepsychiatry in Rural and Remote Primary Care. 17. Stasiak K, Fleming T, Lucassen MF, et al. Computer-based and online Child Adolesc Psychiatr Clin N Am 2011;20(1):13–28. therapy for depression and anxiety in children and adolescents. J Child Adolesc Psychopharmacol 2016;26(3):235–45. Downloaded from https://academic.oup.com/jamiaopen/article-abstract/1/1/67/4980801 by Ed 'DeepDyve' Gillespie user on 07 November 2018 74 JAMIA Open, 2018, Vol. 1, No. 1 APPENDIX Appendix A. Quotes from experts about the benefits of online chat, mobile apps, and video conferencing in TMH for youth Delivery method Benefits Online Chat “Current method of communicating with this population” “This can be beneficial if the client is needing to speak to someone immediately and unable to get to the office to see the therapist or is in a situation where this is their only option” “Instant feedback any time you want it” “no need for transportation, easier access, user friendly for age group” “more comfortable for less verbal clients.. .clients who may process more slowly and be able to think before they write out their responses” Mobile Apps “since people are using apps for other areas of their lives, this would seem to be a natural next step for mental health help” “apps are on phone, and most youth always have their phones on them, so services could always be available” “ability to take interventions with them wherever they go, privacy to look at them without people knowing they are looking at coping information” Video Conferencing “ability to talk to someone face to face regardless of transportation” “In areas where there aren’t adequate mental health services this option would be beneficial for people who would other- wise not have the ability to see a therapist in person” “Convenient, no transportation needed, ideally lower cost, ideally can have more schedule options” “No need for transportation, (video conferencing) can reach those with social anxiety in comfortable setting” Appendix B. Table of demographic characteristics of expert panel members Adults with lived Parents of teens with Mental health Mental health clinicians and Educators Total experience lived experience researchers those who work at mental who work with teens/ (n ¼ 25) (n ¼ 5) (n ¼ 4) (n ¼ 5) health organizations (n ¼ 6) young adults (n ¼ 5) Gender Male 20.0% (1) 25.0% (1) 0.0% (0) 16.7% (1) 20.0% (1) 16.0% (4) Female 80.0% (4) 75.0% (3) 100.0% (5) 83.3% (5) 80.0% (4) 84.0% (21) Age 24–34 20.0% (1) 75.0% (3) 0.0% (0) 33.3% (2) 20.0% (1) 28.0% (7) 35–49 20.0% (1) 0.0% (0) 60.0% (3) 33.3% (2) 40.0% (2) 32.0% (8) 50–64 60.0% (3) 25.0% (1) 40.0% (2) 33.3% (2) 40.0% (2) 40.0% (10) Race White 80.0% (4) 100.0% (4) 100.0% (5) 100.0% (6) 80.0% (4) 92.0% (23) Latino 20.0% (1) 0.0% (0) 0.0% (0) 0.0% (0) 0.0% (0) 4.0% (1) Multi-Racial 0.0% (0) 0.0% (0) 0.0% (0) 0.0% (0) 20.0% (1) 4.0% (1) Familiarity with telemental health Yes 80.0% (4) 25.0% (1) 80.0% (4) 33.3% (2) 60.0% (3) 56.0% (14) No 20.0% (1) 75.0% (3) 20.0% (1) 66.7% (4) 40.0% (2) 44.0% (11) Use of telemental health Yes 40.0% (2) 25.0% (1) 0.0% (0) 50.0% (3) 40.0% (2) 32.0% (8) No 60.0% (3) 75.0% (3) 100.0% (5) 50.0% (3) 60.0% (3) 68.0% (17)

Journal

JAMIA OpenOxford University Press

Published: Jul 1, 2018

References

You’re reading a free preview. Subscribe to read the entire article.


DeepDyve is your
personal research library

It’s your single place to instantly
discover and read the research
that matters to you.

Enjoy affordable access to
over 18 million articles from more than
15,000 peer-reviewed journals.

All for just $49/month

Explore the DeepDyve Library

Search

Query the DeepDyve database, plus search all of PubMed and Google Scholar seamlessly

Organize

Save any article or search result from DeepDyve, PubMed, and Google Scholar... all in one place.

Access

Get unlimited, online access to over 18 million full-text articles from more than 15,000 scientific journals.

Your journals are on DeepDyve

Read from thousands of the leading scholarly journals from SpringerNature, Elsevier, Wiley-Blackwell, Oxford University Press and more.

All the latest content is available, no embargo periods.

See the journals in your area

DeepDyve

Freelancer

DeepDyve

Pro

Price

FREE

$49/month
$360/year

Save searches from
Google Scholar,
PubMed

Create lists to
organize your research

Export lists, citations

Read DeepDyve articles

Abstract access only

Unlimited access to over
18 million full-text articles

Print

20 pages / month

PDF Discount

20% off