TY - JOUR
AU - Garey, Kevin, W.
AB - Purpose The purpose of this study was to quantify the impact of robotic technology on efficiency, accuracy, and cost in a satellite oncology pharmacy. Methods A 33-week quasi-experimental study was conducted at an academic, quaternary care institution with 1,119 licensed beds from June 2016 to February 2017 to evaluate the turnaround time (TAT) for preparations compounded by automated robotic compounding technology (ARCT) versus historical procedures. Secondary endpoints included mean preparation time and percentage of doses with a TAT of <30 minutes before and after the implementation of ARCT and were evaluated using time-segmented regression analysis. The cost savings in the satellite oncology pharmacy was determined by comparing usage of closed-system transfer devices (CSTDs) and labor costs between study phases. Accuracy of the intervention was expressed through a descriptive analysis of mean ARCT dose preparation deviations and preparation failures. Results Data for 1,453 preparations were included for analysis. The mean ± S.D. preimplementation TAT was 64.1 ± 27.9 minutes, which decreased to 53.2 ± 32.2 minutes after ARCT implementation (p < 0.01). Financial benefit was demonstrated through supply cost savings. Breakeven was estimated at 8.6 years after capital expenditure, with an annualized projected savings of $129,477. The mean ± S.D. deviation of the doses compounded using ARCT was −0.58% ± 0.01% from the ordered dosage. Conclusion Adoption of ARCT for compounding of admixtures containing 4 oncology agents reduced TAT and preparation time and led to lower expenditures for CSTDs. automation, chemotherapy, cost, efficiency, oncological, robotics Using sterile compounding practices is one of the most important duties of inpatient pharmacy operations due to the high risk that improper preparation can pose to patient and employee safety.1 For example, a multistate meningitis outbreak that resulted in 64 deaths was caused by improper aseptic technique at a commercial sterile compounding center.2 Another example was documented in a report, which described a compounding error involving high-alert medications that resulted in a patient’s death.3 Pharmacy staff who compound hazardous drugs may be exposed to toxic drug concentrations unless appropriate safeguards are in place.4,5 The addition of supplemental engineering controls, such as closed-system transfer devices (CSTDs), to the compounding process has significantly reduced but not eliminated surface residue.6 A 2014 survey of U.S. hospitals found that CSTD use differed significantly by hospital size, with the smallest hospitals being the least likely to use the device during medication preparation.7 To maintain strict control of this practice, the United States Pharmacopeia (USP) chapter 800 addresses the regulatory requirements for hazardous sterile compounding practices across all healthcare settings.8 Hospitals and health systems must comply with regulatory standards and strive to adopt best practices to ensure the quality of sterile compounding practices to support the safety of patients and their employees. View largeDownload slide Sunny B. Bhakta, Pharm.D., is currently a postgraduate year 2 health-system pharmacy resident at Houston Methodist Hospital. He received his doctor of pharmacy degree, summa cum laude, from the University of Houston College of Pharmacy in 2016. Dr. Bhakta is currently completing an ASHP-accredited residency in health-system pharmacy administration and a concurrent master of science degree in pharmacy leadership and administration at Houston Methodist Hospital and the University of Houston, respectively. Dr. Bhakta is a newly appointed member of the ASHP Section of Pharmacy Practice Managers’ Advisory Group on Management of the Pharmacy Enterprise and is also a member of the Vizient Medication Use Informatics and Technology Committee. Dr. Bhakta has led the implementation and evaluation of a sterile compounding robot at the Houston Methodist Cancer Center pharmacy and was integral in the implementation of a departmental communication and engagement strategy through the leveraging of a pharmacy SharePoint website’s capabilities. Dr. Bhakta was recently awarded an ASHP Research and Education Foundation research grant to improve patient safety and reduce alert fatigue by systematically optimizing the quantity and quality of medication alerts in the health system. Dr. Bhakta’s current research interests include health-system pharmacy leadership and administration, information technology and automation, and oncology pharmaceutical care and management. View largeDownload slide Sunny B. Bhakta, Pharm.D., is currently a postgraduate year 2 health-system pharmacy resident at Houston Methodist Hospital. He received his doctor of pharmacy degree, summa cum laude, from the University of Houston College of Pharmacy in 2016. Dr. Bhakta is currently completing an ASHP-accredited residency in health-system pharmacy administration and a concurrent master of science degree in pharmacy leadership and administration at Houston Methodist Hospital and the University of Houston, respectively. Dr. Bhakta is a newly appointed member of the ASHP Section of Pharmacy Practice Managers’ Advisory Group on Management of the Pharmacy Enterprise and is also a member of the Vizient Medication Use Informatics and Technology Committee. Dr. Bhakta has led the implementation and evaluation of a sterile compounding robot at the Houston Methodist Cancer Center pharmacy and was integral in the implementation of a departmental communication and engagement strategy through the leveraging of a pharmacy SharePoint website’s capabilities. Dr. Bhakta was recently awarded an ASHP Research and Education Foundation research grant to improve patient safety and reduce alert fatigue by systematically optimizing the quantity and quality of medication alerts in the health system. Dr. Bhakta’s current research interests include health-system pharmacy leadership and administration, information technology and automation, and oncology pharmaceutical care and management. The complexity and number of human steps involved in sterile compounding create opportunities for errors.1 Vulnerabilities secondary to variation throughout assembly, compounding, waste management, and administration are inherent without process controls. Advanced techniques such as photo validation, gravimetric dose validation, and barcode scanning are available to improve safety and accuracy during sterile compounding; however, most of these techniques are not widely used, partially due to the lack of research studies demonstrating their cost-effectiveness.7,9,10 Furthermore, recommendations from the Institute for Safe Medication Practices call for augmentation of the traditional compounding process with barcode scanning and gravimetrics for high-alert medications whenever possible.11 Automated robotic compounding technology (ARCT) has a number of advantages, such as consistency of preparation, utilization of ultraviolet-light sterilization, and ability to handle products that present hazards to personnel during preparation.9 Robotic automation devices have demonstrated safety benefits, with mixed results on operational efficiency and pharmacy costs.9,12,13 The purpose of this study was to quantify the impact of new-generation robotic technology on efficiency, accuracy, and costs in a satellite oncology pharmacy. Design and setting. This study was conducted at an academic, qua ternary care institution with 1,119 licensed beds in the Texas Medical Center from June 2016 to February 2017. The hospital’s satellite oncology pharmacy serves a 50-bed inpatient cancer center, a 15-bed bone marrow transplantation center for cell and gene therapy, and an outpatient infusion center with 40 infusion chairs. The satellite oncology pharmacy provided approximately 40,000 compounded sterile preparations in 2015. Using a 33-week quasi-experimental study design, variables during the use of ARCT (i.v.STATION ONCO, Omni-cell, Bakersfield, CA) were compared with those from the preimplementation period. The study was reviewed by the Houston Methodist institutional review board and designated as not constituting human subjects research. This study incorporated interrupted time-series design with time-segmented regression analysis to statistically evaluate the results. The primary endpoint was medication turnaround time (TAT) in the preimplementation and postimplementation phases. TAT was defined as the time from when a technician started preparing a dose to the time when a pharmacist completed final dose verification. Preparation time was defined as the time that elapsed between the start and finish of technician dose preparation. Robotic preparation time was defined as the time that elapsed between the beginning of dose preparation to the point of removal of the completed preparation from the robot. Preparation start times were electronically recorded in both the robotic and i.v. workflow management technology and were correlated to the actual start time of product preparation by either the robotic device or compounding technician. The time of the final pharmacist check was also electronically recorded in the systems, and this documentation was required before the involved dose could be dispensed. Secondary endpoints included mean preparation time and percentage of doses with a TAT of <30 minutes before and after the implementation of ARCT and were evaluated using time-segmented regression analysis. The cost savings in the satellite oncology pharmacy was determined by comparing usage of CSTDs. Accuracy of the intervention was expressed through a descriptive analysis of mean ARCT dose preparation deviations and preparation failures (i.e., ARCT doses that either were not eligible to be evaluated by gravimetrics due to mechanical or technical errors in the ARCT system or doses that exceeded the 5% threshold). Preimplementation workflow and implementation of ARCT. Before implementation of ARCT, manual hazardous compounding utilized i.v. workflow management technology interfaced with the electronic medical record system (Epic Systems, Verona, WI) using computerized prescriber order entry (CPOE) of oncology orders through prebuilt treatment plans. The i.v. workflow management system was implemented at the study site in February 2016, approximately 4 months before study initiation and 7 months before ARCT implementation. Once orders were verified by the pharmacist, pharmacy technicians compounded the preparation using barcode technology and photographs to verify each step of the compounding process and allow for an inprocess check before the addition of the hazardous agent into the diluent container. A CSTD (PhaSeal, BD, Franklin Lakes, NJ) was used to reduce the exposure of personnel involved in compounding to oncology agents. Once the dose was prepared, the label was scanned by the pharmacist to verify accuracy of compounding, and a system-generated verification label was printed to replace the unverified status of the current label. Time points from beginning the preparation to final pharmacist check were available through reporting methods in the software. In September 2016, the department adopted ARCT for chemotherapy preparation in the satellite oncology pharmacy, which incorporated barcoding, digital photographic verification, gravimetric controls, and physical isolation features for hazardous drug compounding. The ARCT was intended to serve as an additional production stream within the pharmacy, which operated during 2 shifts daily on weekdays. The ARCT was initially configured with 4 drugs: cyclophosphamide, cisplatin, carboplatin, and oxaliplatin. Using an interface via Health Level Seven standards,14 CPOE orders were automatically queued to the automated robotic compounding system to generate labels and begin preparation. The robot operator was allowed to select drug vials with lot and expiration date information via the interface to initiate the loading procedure. The loading process required barcode verification for drug vials and fluids before loading them into the device. A preestablished drug vial and syringe photograph library was used to establish photo verification protocols for vial and syringe loading. Accuracy features included a high-precision balance to weigh fluid bags and source drug vials before and after volume transfer of the requested dose and specific gravity measurements of the compounded product to validate dosage accuracy. The acceptable deviation for the final preparation was set at ±5%, in accordance with USP chapter 797 and previous reports that have involved robotics.9,12,15 A final visual product inspection was performed with both technology systems. Analysis plan. Efficiency. Weekly mean drug TAT and preparation time for the 4 oncological agents were assessed before and after the intervention using segmented regression analysis.16,17 Coefficient values of the parameter estimates, standard errors, and p values from segmented regression models predicting the mean TAT and preparation time were calculated. Autocorrelation was assessed using the Durbin-Watson statistic; positive autocorrelation was evaluated through autoregressive modeling. Modified winsorization was used to exclude outliers at the 95th percentile of distribution. This methodology was decided a priori, as prepared admixtures were being held purposefully for patient care issues. The percentage of doses with a TAT of <30 minutes before and after ARCT implementation was also analyzed using the aforementioned methodology. In addition, mean TAT and preparation time were assessed before and after ARCT implementation in aggregate using a 2-sample t test. All statistical analyses were performed using SAS, version 9.4 (SAS Institute, Cary, NC). The a priori level of significance was 0.05 for all endpoints evaluated. Accuracy. Descriptive analyses for accuracy of the robotic compounding device included the collection of variation of ARCT prepared doses and a description of robotic preparation failures. Cost. The supply cost of a CSTD was defined as the mean cost per dose using average wholesale price (AWP), which incorporated costs of the vial protector and syringe adapter.18 A 5% annual inflation rate was applied to supply costs. The mean cost of supplies per dose compounded with ARCT was based on dose volume and vial size to calculate the quantities of protectors and adaptors per dose. Calculations were also adjusted for robot downtime, which was estimated to be 10% of annual operational days. Financial breakeven was determined by subtracting the projected supply cost savings from the capital expense and annual maintenance fees. Results Operational efficiency in TAT. A total of 1,556 preparations of the 4 oncological agents were compounded during the study period. Among the data excluded were those associated with preparations made during the week of ARCT implementation (n = 53) and those with data that were considered outliers via winsorization (n = 50), leaving data for 1,453 preparations available for analysis. A total of 509 preparations were compounded during the preimplementation, compared with 944 during the postimplementation phase, with 514 doses prepared by ARCT during postimplementation. The mean ± S.D. preimplementation TAT was 64.1 ± 27.9 minutes, which decreased to 53.2 ± 32.2 minutes after ARCT implementation (p < 0.01). The segmented regression analysis showed a nonsignificant change in TAT during preimplementation and postimplementation (−0.5 and 0.5 minutes, respectively; p > 0.05) (Figure 1). The interrupted time-series regression analysis found a significant reduction (9.2 ± 3.7 minutes, p = 0.01) in mean ± S.D. TAT after ARCT implementation. After adjusting for first-degree autocorrelation (Durbin-Watson statistic = 1.75), the mean ± S.D. preparation time was reduced by 6.3 ± 2.5 minutes (p = 0.01), after ARCT implementation in the segmented regression analysis. The mean ± S.D. aggregate preparation times were 44.2 ± 20.0 minutes preimplementation and 34.7 ± 26.8 minutes postimplementation (p < 0.01). There were nonsignificant changes in preparation times within the preimplementation and postimplementation phases (−0.4 and 0.4 minutes, respectively; p > 0.05). The number of preparations completed in under 30 minutes also increased significantly in the postimplementation period using ARCT compared with the preimplementation period (20% versus 7.6% of preparations, p = 0.02) (Figure 2). Figure 1 View largeDownload slide Drug preparation time (dotted line) and turnaround time (solid line) for 4 selected oncology agents during the study. Implementation of automated robotic compounding technology (ARCT) occurred at week 14. Figure 1 View largeDownload slide Drug preparation time (dotted line) and turnaround time (solid line) for 4 selected oncology agents during the study. Implementation of automated robotic compounding technology (ARCT) occurred at week 14. Figure 2 View largeDownload slide Percentage of preparations of 4 selected oncology agents with a turnaround time of <30 minutes during the study. Implementation of automated robotic compounding technology (ARCT) occurred at week 14. Figure 2 View largeDownload slide Percentage of preparations of 4 selected oncology agents with a turnaround time of <30 minutes during the study. Implementation of automated robotic compounding technology (ARCT) occurred at week 14. Accuracy. With ARCT, the mean ± S.D. from the prescribed dose for cyclophosphamide, cisplatin, oxaliplatin, and carboplatin were −1.28% ± 2.37%, −1.22% ± 1.66%, −0.59% ± 0.53%, and −2.29% ± 3.91%, respectively; the mean ± S.D. deviation for all products combined was −0.58% ± 0.01%. Of the 525 orders that were routed to ARCT, 11 preparations (2.1%) failed to meet evaluation standards. Of the 11 preparations, 7 failed due to deviations outside of the acceptable deviation of ±5%; all 7 had a deficiency of >5% compared with the prescribed dose. Four of these 7 deviations were due to operator error in the placement of the bag in the ARCT, which resulted in pre mature termination of the preparation and an evaluation error outside of the target acceptable variation. One failure was attributed to the loading of a partial vial, which resulted in inadequate drug delivery to the final container. The remaining 2 deviations involved cyclophosphamide and carboplatin and were attributed to device dosage deviations. They occurred early during ARCT implementation, requiring recalibration of the robot. The remaining 4 failed preparations were attributed to device and engineering control issues that occurred early during the adoption of the technology. These events involved the needle alignment device and drug-doser apparatus and required calibration to ensure adequate drug withdrawal and transfer to the syringe and final container. Cost-effectiveness. The mean savings per dose ranged from $14.83 to $85.43 based on AWP costs of the components required to make the doses. An average savings of $27.56 per unit produced during ARCT was utilized to perform cost-savings pro jections. Reduction of CSTD utilization due to implementation of ARCT led to a projected annualized operational cost savings of $129,477 and an expected target preparation volume of 20 doses per operational day. The operational period of the robot accounted for production across 2 shifts daily exclusively on weekdays with 10% downtime. Robotic preparation volume was approximately 41% of the target in the postimplementation period. The projected annual savings from reduced use of CSTDs led to a capital expenditure breakeven point at 8.6 years postimplementation, exclusive of labor efficiencies and inclusive of maintenance and service expenses. Discussion This quasi-experimental study at a single institution found the implementation of ARCT reduced TAT and preparation time for 4 oncological agents. Robotic preparation resulted in a failure rate of approximately 2% during the study period for these 4 agents. Cost analysis revealed a break- even point of 8.6 years postimplementation based solely on projected supply cost savings. Our findings contrast with those of studies examining the implementation of other robotic solutions for chemotherapy compounding. Our institution used i.v. workflow management technology before implementation of the ARCT, which allowed us to compare data on preparation and verification times. Comparison of these 2 technology workflows has not previously been described. Other studies showed a longer mean preparation time for robot-prepared doses; however, we found a similar preparation time and overall reduction in both preparation time and TAT after ARCT implementation.7,10,11 This finding may be due to the fact our institution utilized i.v. workflow management technology before the implementation of ARCT. During the study period, no other performance-improvement initiatives were attempted in an effort to isolate the effect of the ARCT implementation. Alternative explanations of the impact on efficiency could include improved productivity of the pharmacy staff postimplementation. During the postimplementation phase of the study, 5 additional oncological agents were configured in the device, which may have influenced TAT for the original 4 agents given that there were greater utilization and demand of the ARCT for doses other than those being studied. Also, our institution adopted a new electronic medical record 3 weeks before initiation of the study, which may have influenced TAT in the preimplementation period; however, the study design and analysis considered changes over time, and the analysis did not reveal statistically significant trends before or after implementation. In addition, our study did not examine time of delivery or administration for prepared doses, which can be a limitation in the context of infusion center wait times and chair time. The failure rate in our study period was similar to the rates reported from observations in 2 different ARCT trials by Seger et al.12 and Yaniv et al.13 (0.9% and 1.2%, respectively). Notably, the study by Yaniv et al. utilized a more narrow accuracy range of ±4% as opposed to ±5% used by others.9,12 A study by Nurgat et al.,9 which evaluated the same ARCT utilized by Seger et al.,12 had a trend toward less accuracy as the number of ARCT doses increased over time. At our institution, an improvement to the ARCT was implemented due to an error where a used vial was loaded into the device. This improvement included setting a threshold deviation for all drug vials and bags to allow for a mechanism to reject partial products from being used in the device. Preparation failures and spills within the device were not completely resolved during the study period, and both of these issues suggest that more maturity is needed from engineering and support service perspectives before widespread adoption. Our study supported the accuracy benefits that have been shown with the adoption of similar robotic solutions to improve safety.11,12 Supply cost savings from the reduction of CSTDs provided the opportunity to maintain a financial return during the anticipated life of ARCT; approximately $120,000 annually would be saved based on 20 doses per operational day. Our current robot configuration includes 9 drugs as opposed to the 4 that were studied, and actual trends are projected to achieve 20 preparations per day. Labor cost avoidance was excluded from this study and only used for internal purposes given the lack of external validity in other settings with different staffing models and technology. Therefore, CSTD savings were used for purposes of financial return calculations in this report. Waste reduction was not factored into our cost analysis, given that a majority of preparations were eligible for waste billing. Limitations to the generalizability of our findings include different workflows and technologies in other hospitals, differences in calculations of economic impact from other studies, and other factors that may vary across institutions, including salvage value, operating term, annual service fees, labor cost-avoidance calculations, and the use of CSTDs. The preparation time and TAT measures included only the specific steps mentioned in the methods, which did not include loading of the robot or gathering supplies before compounding. Our CSTD cost savings were based on the AWP of the individual components used for dose preparation; institutions may vary in which a CSTD is used and the respective purchase agreement cost. Our study did not directly compare the dose variation between manual and automated preparations, since a gravimetric process was not used before ARCT implementation. Increased numbers of preparations and growth in oncology pharmaceutical care played a significant role in our study, but the actual effect on other satellite oncology pharmacies may vary depending on workload, productivity, and growth. Conclusion Adoption of ARCT for compounding of admixtures containing 4 oncology agents reduced TAT and preparation time and led to lower expenditures for CSTDs. Disclosures The authors have declared no potential conflicts of interest. Additional information Presented at the ASHP Midyear Clinical Meeting, Orlando, FL, December 6, 2017. References 1 Rich DS Fricker MP Jr Cohen MR Levine SR . Guidelines for the safe preparation of sterile compounds: results of the ISMP sterile preparation compounding safety summit of October 2011 . Hosp Pharm . 2013 ; 48 : 282 – 94 . Google Scholar Crossref Search ADS PubMed 2 Centers for Disease Control and Prevention . Multistate outbreak of fungal meningitis and other infection ( October 30 , 2015 ). www.cdc.gov/hai/outbreaks/meningitis.html (accessed 2017 Jan 14). 3 Institute for Safe Medication Practices . Tragic error with neuromuscular blocker should prompt risk assessment by all hospitals ( December 18 , 2014 ). www.ismp.org/newsletters/acutecare/showarticle.aspx?id=97 (accessed 2017 Feb 3). 4 Anderson RW Puckett WH Jr Dana WJ et al. Risk of handling injectable antineoplastic agents . Am J Hosp Pharm . 1982 ; 39 : 1881 – 7 . Google Scholar PubMed 5 Connor TH Theiss JC Anderson RW et al. Re-evaluation of urine mutagenicity of pharmacy personnel exposed to antineoplastic agents . Am J Hosp Pharm . 1986 ; 43 : 1236 – 9 . Google Scholar PubMed 6 Sessink PJ Connor TH Jorgenson JA Tyler TG . Reduction in surface contamination with antineoplastic drugs in 22 hospital pharmacies in the US following implementation of a closed-system drug transfer device . J Oncol Pharm Pract . 2011 ; 17 : 39 – 48 . Google Scholar Crossref Search ADS PubMed 7 Pedersen CA Schneider PJ Scheckelhoff DJ . ASHP national survey of pharmacy practice in hospital settings: dispensing and administration—2014 . Am J Health-Syst Pharm . 2015 ; 72 : 1119 – 37 . Google Scholar Crossref Search ADS PubMed 8 United States Pharmacopeia . Briefing: <800> hazardous drugs–handling in healthcare settings . www.usp.org/compounding/general-chapter-hazardous-drugs-handling-healthcare (accessed 2018 Mar 5). 9 Nurgat Z Faris D Mominah M et al. A three-year study of a first-generation chemotherapy-compounding robot . Am J Health-Syst Pharm . 2015 ; 72 : 1036 – 45 . Google Scholar Crossref Search ADS PubMed 10 Reece KM Lozano MA Roux R Spivey SM . Implementation and evaluation of a gravimetric i.v. work-flow software system in an oncology ambulatory care pharmacy . Am J Health-Syst Pharm . 2016 ; 73 : 165 – 73 . Google Scholar Crossref Search ADS PubMed 11 Institute for Safe Medication Practices . ISMP guidelines for safe preparation of compounded sterile preparations (revised 2016 ). www.ismp.org/tools/guidelines/ivsummit/ivcguide-lines.pdf (accessed 2018 Mar 5). 12 Seger AC Churchill WW Keohane CA et al. Impact of robotic antineoplastic preparation on safety, workflow, and costs . J Oncol Pract . 2012 ; 8 : 344 – 9 . Google Scholar Crossref Search ADS PubMed 13 Yaniv AW Knoer SJ . Implementation of an i.v.-compounding robot in a hospital-based cancer center pharmacy . Am J Health-Syst Pharm . 2013 ; 70 : 2030 – 7 . Google Scholar Crossref Search ADS PubMed 14 Health Level Seven International . Introduction to HL7 standards . www.hl7.org/implement/standards/ (accessed 2018 Mar 5). 15 Pharmaceutical compounding—sterile preparations (general chapter 797) . In: United States pharmacopeia, 37th rev., and The national formulary , 32nd ed. Rockville, MD : United States Pharmacopeial Convention ; 2014 . 16 Garey KW Lai D Dao-Tran TK et al. Interrupted time series analysis of vancomycin compared to cefuroxime for surgical prophylaxis in patients undergoing cardiac surgery . Antimicrob Agents Chemother . 2008 ; 52 : 446 – 51 . Google Scholar Crossref Search ADS PubMed 17 Wagner AK Soumerai SB Zhang F Ross-Degnan D . Segmented regression analysis of interrupted time series studies in medication use research . J Clin Pharm Ther . 2002 ; 27 : 299 – 309 . Google Scholar Crossref Search ADS PubMed 18 Gold Medical Supplies. Search results . www.goldmedicalsup-plies.com/search.php?search_query=phaseal&Search=l (accessed 2017 Jan 18). Copyright © 2018 by the American Society of Health-System Pharmacists, Inc. All rights reserved.
TI - Implementation and evaluation of a sterile compounding robot in a satellite oncology pharmacy
JF - American Journal of Health-System Pharmacy
DO - 10.2146/ajhp170461
DA - 2018-06-01
UR - https://www.deepdyve.com/lp/oxford-university-press/implementation-and-evaluation-of-a-sterile-compounding-robot-in-a-yuVoV1gPR0
SP - S51
VL - 75
IS - 11_Supplement_2
DP - DeepDyve
ER -