Abstract In response to data collection challenges during mass immunization events, Denver Public Health developed a mobile application to support efficient public health immunization and prophylaxis activities. The Handheld Automated Notification for Drugs and Immunizations (HANDI) system has been used since 2012 to capture influenza vaccination data during Denver Health’s annual employee influenza campaign. HANDI has supported timely and efficient administration and reporting of influenza vaccinations through standardized data capture and database entry. HANDI’s mobility allows employee work locations and schedules to be accommodated without the need for a paper-based data collection system and subsequent manual data entry after vaccination. HANDI offers a readily extensible model for mobile data collection to streamline vaccination documentation and reporting, while improving data quality and completeness. mobile health, mHealth, mass vaccination, data collection methods, public health INTRODUCTION Despite having a core mission of assuring access to and delivery of key interventions for populations, many local health departments find monitoring their efforts challenging. Local public health agencies (LPHAs) are funded to plan for mass immunization and prophylaxis in response to emergencies.1,2 The Centers for Disease Control and Prevention’s Strategic National Stockpile provides essential treatment and prophylaxis countermeasure resources for local use.3 It is focused on ready deployment and dispensing, and less emphasis has been placed on documenting to whom medical countermeasures have been administered. Most LPHAs have “point of dispensing”4 plans consistent with national preparedness goals, with community partnerships that expand closed (or institutional-based) point of dispensing capacity.5 With greater external responsibility for distributing medical countermeasures, most LPHAs still lack efficient electronic methods to document and monitor these efforts.6–9 In 2009–2010, limited H1N1 influenza vaccine required targeting delivery to certain risk groups.10 Tracking high-risk target groups and measuring immunization coverage while serving as a supply chain control for health care providers and conducting mass immunization clinics was challenging.10 After 124 million doses were distributed, few state health departments could accurately report who had been vaccinated. Based on self-reported national survey data, only an estimated 81 million people reported receiving the H1N1 vaccine.11 Of Denver’s 200 000 doses received, Denver Public Health administered approximately 20 000 vaccines but was unable to report on risk group coverage, because a method to rapidly capture large volumes of data was lacking (Davidson AJ. 2016 personal communication). The absence of timely and accurate vaccine coverage measures emphasizes a need for real-time data collection to facilitate efficient vaccine distribution, administration, and monitoring. During emergencies, mass vaccination clinic workflows typically include time-consuming paper-based patient registration, eligibility and contraindication assessment, and vaccine delivery. Subsequent resource-intensive manual data entry of demographics and vaccine documentation (eg, vaccine manufacturer, lot number, injection site) is variably completed. For H1N1, few health departments were able to keep up with this demand and fewer transferred the information to a registry. Delayed and incomplete information jeopardized the monitoring of vaccine coverage for those at highest risk. Seeking to help manage and transform large volumes of data into information, this paper describes our experience with an informatics solution developed as a data collection and monitoring tool for use during a mass vaccination campaign. METHODS Study setting Denver Health (DH), an integrated health care delivery system, employs >6000 people, including outpatient, inpatient, and public health services. DH conducts annual employee influenza vaccination campaigns12 and instituted a mandatory influenza vaccination policy in 2011. Study population As part of a DH interdisciplinary approach, representatives from the Center for Occupational Safety and Health (COSH), Infection Control, Denver Public Health (DPH), and eHealth Services (collectively referred to as the DH Employee Flu Task Force) collaboratively supported the annual employee influenza vaccination campaigns by building processes and tools to efficiently document vaccine administration and monitor coverage. Informatics approach Application development Business requirements gathered information from: (1) existing business processes, (2) H1N1 mass immunization clinic experiences, (3) existing forms and databases, and (4) interviews with DPH immunization, epidemiology, and preparedness staff. The employee influenza vaccination campaign use case was chosen because it is of large scale, is conducted annually, and requires vaccination reporting. DPH’s Handheld Automated Notification for Drugs and Immunizations (HANDI) application was developed as an Apple iOS mobile app supported by a Microsoft Windows server and Microsoft SQL server database. A peripheral scanner encased an iPod Touch to scan magnetic stripes and 1- or 2-dimensional barcodes. The HANDI administration application was hosted on a server where administrators defined interventions (ie, clinic location, vaccine type, vaccine lot number, manufacturer, and expiration date). All data were encrypted13 and mobile device management software was installed on devices.14 Pilot test The employee influenza campaign was piloted in 2011 in 5 DH clinics, and the workflow was divided into 3 stations: (1) consent, (2) registration/eligibility, and (3) vaccination. Employees signed vaccination consent forms at station 1 and proceeded to stations 2 and 3, where HANDI collected demographic information by scanning their driver’s license, capturing answers to influenza-related screening questions (eg, “Are you sick today?”), and recording vaccination data (eg, lot number, injection site, vaccinator). HANDI users included volunteer DPH staff from the epidemiology, preparedness, administration, and immunization departments, who completed a 5-question HANDI usability survey. The pilot captured 242 vaccinations, for which average time from consent to injection was 4 min (std 2). HANDI users reported rapid and easy data entry, with straightforward, intuitive usability. Nineteen percent of driver’s licenses failed to scan because they were damaged (eg, bent, scratched, worn). For these, the device scanned the magnetic stripe or barcode, or allowed keypad data entry. Based on 2011 pilot test results, HANDI was recommended for use during subsequent employee influenza campaigns. Employee influenza vaccine campaign For annual employee influenza vaccination campaigns, a web-based application, FluTracker, supported preregistration, vaccine consent, and reporting functionality. HANDI was modified to recognize the magnetic stripe on employee badges. Employees preregistered, answered influenza-related screening questions, and consented to receiving vaccine using FluTracker. Once registered, employees presented for vaccination at mass clinics, COSH, or DH clinics, where HANDI scanned employee badges. HANDI retrieved preregistration information and influenza vaccination–related responses for review and captured vaccination information (Figure 1). FluTracker automatically sent documentation e-mails to employees following vaccination and provided ongoing access to vaccination records. Figure 1. View largeDownload slide Denver Health employee influenza campaign information systems and data flow. Figure 1. View largeDownload slide Denver Health employee influenza campaign information systems and data flow. HANDI was used at mass vaccination clinics and in other DH facilities. Roving HANDI-equipped vaccination teams vaccinated employees in different areas of the main campus and circulated among 24 community clinics and off-campus facilities. The FluTracker dashboard displayed number of employees preregistered, number vaccinated, and number of vaccines administered by vaccinator. DH managers used FluTracker for real-time vaccination coverage progress and monitoring of their employees’ vaccination status. In 2012, designated HANDI users were trained to staff the mass clinics. A 3-min training session occurred at the beginning of the user’s shift. HANDI user guides including step-by-step instructions, and screenshots were available for reference. After 2012, HANDI was operated by members of the DH Employee Flu Task Force, with less need for additional HANDI users. Off-campus location teams were trained by an Employee Flu Task Force member, or self-trained using the HANDI user guide. Analysis Performance measures included: (1) number of people served, (2) number of clinic staff, (3) time required to use HANDI and administer influenza vaccine, and (4) percentage of vaccinations occurring at mass clinics versus alternate DH vaccination locations. HANDI users completed an evaluation about using the application. Data from 2011 were included in the results and designated as “pre-HANDI” to compare vaccination measures before and after HANDI and FluTracker were deployed. RESULTS Between 2012 and 2015, 22 390 influenza vaccinations were administered to DH employees using FluTracker and HANDI applications. Comparing 2011 (pre-HANDI) and 2012 mass vaccination clinics (Table 1): in 2011, 3110 employees were vaccinated by 12 clinic staff during 98 clinic hours, and in 2012, 3512 employees were vaccinated by 6 clinic staff during 73 clinic hours. Mass clinic time and staffing were further reduced in 2013, 2014, and 2015, and, using Poisson regression, a significant increasing trend in employee vaccinations per hour (P < .0001) was found. From 2012 to 2015, vaccinations administered off campus increased from 27% to 63% (Table 2). HANDI vaccination times averaged ≤1 min. Similar to the pilot test user surveys, HANDI users reported easy and fast data entry and said they would use HANDI again. Table 1. Comparison of vaccination productivity measures for annual employee influenza mass vaccination clinics, Denver Health, 2011–2015 Measure 2011 (pre-HANDI) 2012 2013 2014 2015 Vaccinations 3110 3512 2588 2186 2375 Clinic staff 12 6 6 4 2 Hours 98 73 60 42 29 Vaccinations per staff 259 585 431 547 1188 Vaccinations per hour 32 48 43 52 82 Measure 2011 (pre-HANDI) 2012 2013 2014 2015 Vaccinations 3110 3512 2588 2186 2375 Clinic staff 12 6 6 4 2 Hours 98 73 60 42 29 Vaccinations per staff 259 585 431 547 1188 Vaccinations per hour 32 48 43 52 82 Table 2. Distribution by site of vaccination for annual employee influenza vaccination campaigns, Denver Health, 2011–2015 Site 2011 (pre-HANDI), n (%) 2012, n (%) 2013, n (%) 2014, n (%) 2015, n (%) Mass clinic 3110 (58) 3512 (73) 2588 (47) 2186 (38) 2375 (37) Non-mass clinic 2230 (42) 1269 (27) 2906 (53) 3518 (62) 4036 (63) Total 5340 4781 5494 5704 6411 Site 2011 (pre-HANDI), n (%) 2012, n (%) 2013, n (%) 2014, n (%) 2015, n (%) Mass clinic 3110 (58) 3512 (73) 2588 (47) 2186 (38) 2375 (37) Non-mass clinic 2230 (42) 1269 (27) 2906 (53) 3518 (62) 4036 (63) Total 5340 4781 5494 5704 6411 DISCUSSION This study demonstrates increased efficiency using a web- and mobile-based application to support mass vaccinations. The FluTracker and HANDI solution eased the burden on all groups involved with mass vaccinations through improved patient flow and throughput, reduced staffing requirements, and streamlined data collection. Impact on employees was reduced due to efficient preregistration and numerous mobile vaccination venues. Using HANDI and FluTracker, supervisors monitored employee vaccination status and encouraged staff vaccinations. HANDI’s mobility accommodated employee work locations and schedules. From 2012 to 2015, employee vaccination rates at non–mass clinic venues increased from 27% to 63%. While employees conveniently preregistered at any work station, HANDI devices permitted off-campus and roving vaccination teams, which reduced employee travel time to mass vaccination clinic sites. Faster employee vaccination delivery reduced mass clinic staffing requirements. By 2015, mass clinic schedules were reduced to 29 h (with 2 staff vaccinators), yet the clinics still vaccinated 2375 employees, demonstrating markedly improved efficiency over previous campaigns. More employees were vaccinated in less time using FluTracker and HANDI compared with prior campaigns. FluTracker and HANDI eliminated paper-based vaccination consent and documentation. Preregistering using FluTracker captured employee consent, and HANDI provided real-time electronic vaccination documentation. Proof of vaccination was readily accessible, as data were stored without burdensome manual data entry. No cumbersome stacks of consent forms to prove vaccination status were maintained, as HANDI leveraged standardized employee badges, scanning technology, and mobile device technologies. A significant limitation of this study was the inability to compare pre/post accuracy and timeliness measures. Prior paper-based campaigns required employees to sign paper consent forms and vaccinators to complete vaccination information on forms, and data were manually entered. Information was not available on the time required to manage this data entry process or its accuracy. Prior to 2011, employee influenza vaccination was not mandatory and there was limited need to track and report employee vaccinations, unlike the current state.15 Various tools support improved methods of data capture during mass and routine vaccination events.16 Several health departments used online registration applications to reduce the time required for registration and paper form completion; patients registered online and printed a voucher or ticket to present at the clinic.17,18 These applications improved demographic data accuracy, but typically relied on paper forms to capture vaccination information. Magnetic stripe and barcode scanners, although similar to HANDI in their ability to scan demographic and vaccination data, cannot wirelessly transfer real-time data to immunization information systems or other vaccination registries. HANDI’s use of common iOS devices provides flexible access to e-mail, phone, and Internet. Web-based data collection systems register patients, schedule appointments, collect vaccination information, and forward data to immunization registries.19,20 These applications offer functionality similar to HANDI, but typically require a synchronous Internet connection, are not as mobile, and rely less on scanning technology. HANDI vaccination times averaging ≤1 min were similar to other electronic systems, which reported an average of 104 seconds,21 and faster than systems using hybrid methods, including electronic patient registration, which reported mean times ranging from 2 to 8 min.17,21 While we deployed a preregistration and tracking system at DH, for a large-scale event, a public-facing preregistration tool would be implemented. The employee influenza vaccination campaigns relied on the DH network for communication between mobile devices and server; mass events would require a more scalable solution. Technology advances, such as secure cloud-based platforms,22–24 would simplify connectivity, facilitate wider deployment to other jurisdictions, and offer greater flexibility to address distinct field requirements. Additionally, organizations seeking to utilize mobile data collection should consider a sustainability plan that includes retiring obsolete hardware and acquiring new equipment to keep technology uniform and current. This DH HANDI version supported 1 immunization per employee. DPH has since expanded the data model to accommodate a wider range of public health use cases beyond immunization (eg, multiple intervention delivery, international traveler and health care worker monitoring, community-based service registration). Increased flexibility and versatility should raise the value of mobile technology for health departments; mobile data collection is applicable to multiple public health business needs.25,26 This study demonstrates the utility of mobile technology in large-scale vaccination campaigns. A previous study used HANDI to capture medical countermeasure delivery and validated linkage of HANDI collected data to a health care system and then to a national surveillance system.27 HANDI was also used during a pertussis vaccination campaign for childcare workers. Nurse vaccinators traveled to Denver childcare sites and used HANDI to scan driver’s licenses and capture Tdap vaccination information on 405 childcare providers (McClung MW. 2017 personal communication). A mobile data collection tool such as HANDI, adaptable to a variety of clinic workflows and environments, has the potential to reduce the time necessary to serve the population and significantly enhance the capacity to monitor efforts to protect the public’s health. FUNDING This project was supported by the Centers for Disease and Control preparedness funding, award no. 1U90TP000510. CONTRIBUTORS MM and AD drafted the manuscript. MM is the project manager for the HANDI project. AD provided project oversight. SG developed the FluTracker application and provided technical support for the DH employee influenza campaigns. MB and AM planned and conducted the DH employee influenza campaign that utilized the HANDI and FluTracker applications. BK used HANDI during the mass clinics and provided the statistics for the manuscript. COMPETING INTERESTS The authors have no competing interests. References 1 Office of Public Health Preparedness and Response. State and Local Readiness . Atlanta: Centers for Disease Control and Prevention; June 17, 2016. www.cdc.gov/phpr/readiness/mcm.html. Accessed May 31, 2017. 2 Office of Public Health Preparedness and Response. Public Health Preparedness and Response National Snapshot . 2017. Atlanta: Centers for Disease Control and Prevention; March 2017. www.cdc.gov/phpr/whyitmatters/00_docs/2017_PublicHealthPreparednessSnapshot_508.pdf. Accessed May 31, 2017. 3 Office of Public Health Preparedness and Response. Strategic National Stockpile . Atlanta: Centers for Disease Control and Prevention; March 2, 2017. www.cdc.gov/phpr/stockpile/index.htm. Accessed May 31, 2017. 4 Office of Public Health Preparedness and Response. Public Health Preparedness Capabilities: National Standards for State and Local Planning . Atlanta: Centers for Disease Control and Prevention; April 26, 2017. www.cdc.gov/phpr/readiness/capabilities.htm. Accessed May 31, 2017. 5 Rebmann T, Loux TM, Swick Zet al. , A national study examining closed points of dispensing (PODs): existence, preparedness, exercise participation, and training provided. Biosecur Bioterror. 2014; 12 4: 208– 16. Google Scholar CrossRef Search ADS PubMed 6 Billittier AJ, Lupiani P, Masterson Get al. , Electronic patient registration and tracking at mass vaccination clinics: a clinical study. J Public Health Manag Pract. 2003; 9 5: 401– 10. Google Scholar CrossRef Search ADS PubMed 7 Gursky EA, Bice G. Assessing a decade of public health preparedness: progress on the precipice? Biosecur Bioterror. 2012; 10 1: 55– 65. Google Scholar CrossRef Search ADS PubMed 8 Russell PK, Gronvall GK. U.S. medical countermeasure development since 2001: a long way yet to go. Biosecur Bioterror. 2012; 10 1: 66– 76. Google Scholar CrossRef Search ADS PubMed 9 Kilianski A, O'Rourke AT, Carlson CLet al. , The planning, execution, and evaluation of a mass prophylaxis full-scale exercise in Cook County, IL. Biosecur Bioterror. 2014; 12 2: 106– 16. Google Scholar CrossRef Search ADS PubMed 10 Hunter JC, Rodriguez DC, Aragon TJ. Public health management of antiviral drugs during the 2009 H1N1 influenza pandemic: a survey of local health departments in California. BMC Public Health. 2012; 12: 82. Google Scholar CrossRef Search ADS PubMed 11 Centers for Disease Control and Prevention, National Center for Immunization and Respiratory Diseases. Final Estimates for 2009–10 Seasonal Influenza and Influenza A (H1N1) 2009 Monovalent Vaccination Coverage – United States, August 2009 through May, 2010 . Atlanta: Centers for Disease Control and Prevention; May 13, 2011. www.cdc.gov/flu/fluvaxview/coverage_0910estimates.htm. Accessed May 31, 2017. 12 Centers for Disease Control and Prevention Advisory Committee on Immunization Practices. Immunization of Health-Care Personnel: Recommendations of the Advisory Committee on Immunization Practices (ACIP). MMWR Morb Mortal Wkly Rep. 2011;60(RR07):1–45. 13 National Institute of Standards and Technology. Advanced Encryption Standard (AES) (FIPS PUB 197) . November 26, 2001. http://nvlpubs.nist.gov/nistpubs/FIPS/NIST.FIPS.197.pdf. Accessed May 31, 2017. 14 BlackBerry (formerly Good Technology). http://us.blackberry.com/. Accessed May 31, 2017. 15 The Joint Commission. R3 Report, Influenza Vaccination for Licensed Independent Practitioners and Staff . May 30, 2012. www.jointcommission.org/assets/1/18/R3_Report_Issue_3_5_18_12_final.pdf. Accessed May 31, 2017. 16 Oak Ridge Associate Universities. Catalog of Electronic Technologies Used for Data Collection at Vaccination Clinics . December 31, 2015. http://toolbox.naccho.org/pages/tool-view.html?id=4839. Accessed May 31, 2017. 17 Larimer County. Harvard School of Government honors Health Department for “Bright Idea” innovation . September 26, 2012. www.larimer.org/news/newsDetail.cfm?id=1774. Accessed May 31, 2017. 18 Johnson County Department of Health and Environment. Dispense Assist . www.dispenseassist.net/. Accessed May 31, 2017. 19 Centers for Disease Control and Prevention. CDC Countermeasure Tracking Systems . www.cdc.gov/cts/cra/documents/cra-fact-sheet.pdf. Accessed May 31, 2017. 20 New York State Department of Health. New York State Clinical Data Management System . September 18, 2014. www.youtube.com/watch?v=IYDCiGZQl2I. Accessed May 31, 2017. 21 Quach S, Hamid JS, Pereira JAet al. , Time and motion study to compare electronic and hybrid data collection systems during the pandemic (H1N1) 2009 influenza vaccination campaign. Vaccine. 2011; 29 10: 1997– 2003. Google Scholar CrossRef Search ADS PubMed 22 Amazon Web Services (AWS). https://aws.amazon.com/. Accessed May 31, 2017. 23 Box. www.box.com. Accessed May 31, 2017. 24 Microsoft Azure. https://azure.microsoft.com/en-us/. Accessed May 31, 2017. 25 Griebel L, Prokosch HU, Kopcke Fet al. , A scoping review of cloud computing in healthcare. BMC Med Inform Decis Mak. 2015; 15: 17. Google Scholar CrossRef Search ADS PubMed 26 Wilson K, Atkinson KM, Westeinde J. Apps for immunization: Leveraging mobile devices to place the individual at the center of care. Hum Vaccin Immunother. 2015; 11 10: 2395– 99. Google Scholar CrossRef Search ADS PubMed 27 Daley MF, Goddard K, McClung Met al. , Using a handheld device for patient data collection: a pilot for medical countermeasures surveillance. Public Health Rep 2016; 131 1: 30– 34. Google Scholar CrossRef Search ADS PubMed © The Author 2017. Published by Oxford University Press on behalf of the American Medical Informatics Association. All rights reserved. For Permissions, please email: firstname.lastname@example.org This article is published and distributed under the terms of the Oxford University Press, Standard Journals Publication Model (https://academic.oup.com/journals/pages/about_us/legal/notices)
Journal of the American Medical Informatics Association – Oxford University Press
Published: Apr 1, 2018
It’s your single place to instantly
discover and read the research
that matters to you.
Enjoy affordable access to
over 12 million articles from more than
10,000 peer-reviewed journals.
All for just $49/month
Read as many articles as you need. Full articles with original layout, charts and figures. Read online, from anywhere.
Keep up with your field with Personalized Recommendations and Follow Journals to get automatic updates.
It’s easy to organize your research with our built-in tools.
Read from thousands of the leading scholarly journals from SpringerNature, Elsevier, Wiley-Blackwell, Oxford University Press and more.
All the latest content is available, no embargo periods.
“Hi guys, I cannot tell you how much I love this resource. Incredible. I really believe you've hit the nail on the head with this site in regards to solving the research-purchase issue.”Daniel C.
“Whoa! It’s like Spotify but for academic articles.”@Phil_Robichaud
“I must say, @deepdyve is a fabulous solution to the independent researcher's problem of #access to #information.”@deepthiw
“My last article couldn't be possible without the platform @deepdyve that makes journal papers cheaper.”@JoseServera