TY - JOUR AU - Kim,, Hyunwook AB - Abstract This study aims to evaluate the accuracy, precision and conservatism of three models (ECETOC TRA 3.1, Stoffenmanager 7.0 and ART 1.5) by comparing model predictions and repeated exposure measurements in Korea. We collected the exposure measurements (n = 390) and detailed contextual information extracted from 10 survey reports published by Ministry of Employment and Labour in the mid-2000s. Using these three models, seven occupational health professionals predicted inhalation exposure to 10 solvents used for cleaning tasks in 51 situations at 33 companies in 15 industries. We applied four comparison approaches previously used by several European studies and calculated the lack of agreement (bias, relative bias, precision), Pearson correlation coefficients (r), level of conservatism, and residuals between the log-transformed predicted estimates and measured exposures on both individual and situation levels. The overall bias and precision were −0.53 ± 2.11 with ART, −1.32 ± 5.88 with Stoffenmanager, and −1.03 ± 8.88 with ECETOC TRA. Pearson correlation coefficients were significantly high in ART (r = 0.95) and Stoffenmanager (r = 0.82), but moderate in ECETOC TRA (r = 0.58). We found ART to be the most accurate model, and Stoffenmanager was the most balanced model in terms of good accuracy, high correlation, and medium conservatism in the model predictions. However, ECETOC TRA showed less accurate outcomes and lower level of conservatism but still had moderate correlations. We observed a systematic tendency to overestimate low exposures and underestimate higher exposures in all models, similar to previous studies. Therefore, our findings suggest that these European models can be used to predict occupational exposure to solvents in Korea. Advanced REACH Tool (ART), ECETOC TRA, exposure modelling, occupational exposure assessment, STOFFENMANAGER® Introduction The Registration, Evaluation, Authorisation and Restriction of CHemicals (REACH) Regulation (EC 1907/2006) aims to ensure a high level of protection of human health and the environment and requires that manufacturers, importers, suppliers or relevant users demonstrate ‘safe use’ of substances and adequate control of the risks arising from these substances. For general registration purposes, information on exposure assessment, including the generation of exposure scenarios and exposure estimations that quantitatively or qualitatively estimate the dose/concentration of the substance to which humans and the environment may be exposed, should be included in a chemical safety report (EC, 1999). To support exposure assessments, exposure estimations using the exposure modelling tools recommended by the European Chemicals Agency (ECHA) are performed to predict real-life situations and exposure levels and describe exposure scenarios from different processes or tasks in various circumstances and industries when the measured exposure data are not available. Currently, Tier 1 (ECETOC Targeted Risk Assessment, EMKG-Expo-Tool, etc.), Tier 1.5 (Stoffenmanager®, hereafter referred to as Stoffenmanager) and Tier 2 (Advanced REACH Tool, hereafter referred to as ART) models have been developed and are widely used to predict occupational exposure estimates for solids, liquids and dusts to conservatively and reliably cover the exposure situations in many European countries (ECHA, 2016). Under REACH, Tier 1 screening models are used to identify the exposure situations and provide rough estimates of occupational inhalation exposures in a given exposure situation, but some Tier 1 models do not always produce sufficiently conservative assessment outcomes (ECHA, 2016; van Tongeren et al., 2017). Higher Tier models (ART and Stoffenmanager), on the other hands, are more refined, robust and sophisticated models than ECETOC Targeted Risk Assessment (TRA) (Riedmann et al., 2015), thus these models can produce highly accurate, conservative and more realistic estimates of occupational exposure by entering more principal modifying factors or input parameters (Hofstetter et al., 2013; Spinazzè et al., 2017; Savic et al., 2017b). However, higher Tier models sometimes show partly inaccurate and less conservative exposure estimates for certain types of substances in some exposure situations and industries (Mc Donnell et al., 2011; Landberg et al., 2017; Savic et al., 2017a). Recently, several validation studies were conducted to evaluate the accuracy, conservatism, reliability and robustness of the Tier 1 and 2 models by calculating the bias, relative bias, precision, and correlation coefficients between the modelled outputs and the measured exposures from different tasks, situations and industries within the European Union (EU) (Kupczewska-Dobecka et al., 2011; Koppisch et al., 2012; Marquart et al., 2017; Savic et al., 2017a; Spinazzè et al., 2017; van Tongeren et al., 2017). Most validation studies showed good agreements between model estimates and measurement data on higher Tier models (>80%) but moderate for several Tier 1 models (30–60%), and the study results also suggested that these models were generally conservative in the exposure modelling. Between-user variability was also high, thus resulting in inconsistencies in the modelling outcomes of multiple users, which differed by several orders of magnitude when the users or assessors performed the exposure assessment modelling for the same exposure situations using either Tier 1 or 2 tools (Schinkel et al., 2014; Landberg et al., 2015; Lamb et al., 2017). These studies recommended that additional technical support, extensive training, and quality control before the use of those models should be implemented to minimize unpredictable between-user variability thus increase consistency in the modelled outcomes among various model users. Most importantly, these studies found that a Tier 1 model (ECETOC TRA) usually overestimated occupational exposure to organic solvents (80%) (Spinazzè et al., 2017), volatile liquids (94% for manufacturing, 78% for painting work) (Ishii et al., 2017), and toluene (a ratio of predicted to measured value was a factor of 3.61) (Hofstetter et al., 2013) in given exposure situations. Higher Tier models (ART and Stoffenmanager) showed better agreement and consistency (75%) with measured exposures of volatile liquids (Spinazzè et al., 2017) but moderate correlations due to underestimation of inhalation exposures to liquids and powders in general (Landberg et al., 2017) compared with the Tier 1 models (van Tongeren et al., 2017). However, the higher Tier models (ART and Stoffenmanager) overestimated low exposures and underestimated higher exposures, that is, they had a systematic tendency to underestimate or overestimate exposures, especially for liquids and powders (Landberg et al., 2017). As for ART, this over- or underestimation resulted in a significant decrease in the level of agreement with measured exposures in different situations and industries (Savic et al., 2017a). In this regard, the authors suggested that further studies were necessary to investigate whether or not the systematic trend resulted from drawbacks of the models themselves or an artefact of the exposure measurement database in a certain country (Savic et al., 2017a) and were required to further validate the models by comparing the tool estimates with a large number of new exposure measurements, including contextual information on operational conditions and risk management measures, collected from a wide range of various exposure scenarios and situations in different companies, industries and countries (Fransman, 2017; van Tongeren et al., 2017). In Korea, there was only one study conducted by the Korea Occupational Safety and Health Agency (KOSHA) that evaluated a Tier 1 model (ECETOC TRA) by comparing the predicted estimates with 8 h-TWA exposure measurements (n = 15) for six different chemicals (e.g. toluene, isopropyl alcohol and lead) in 15 industrial processes (i.e. exposure situations) in 11 different industries (Lee et al., 2012). The authors found that ECETOC TRA mostly overestimated the measured exposures (n = 11), and the ratio of measurements to predictions was less than 1 (on average 0.23). In this regard, the authors suggested that further studies were required to calibrate the model by modifying PROcess Category (PROC) applicable to Korean industries. To date, however, no study has evaluated the accuracy, precision and conservatism of either Tier 1 or 2 models by comparing the model predictions with large-scale exposure measurements collected in Korea. Furthermore, there has been no specific information regarding which model would be best to use for the prediction of occupational exposure to various substances in exposure situations of chemical-specific tasks performed in Korean workplaces and industries in the applicability domain of the models. Therefore, the present study aims to evaluate the accuracy, precision and conservatism of three well-known exposure models (ECETOC TRA 3.1, Stoffenmanager 7.0 and ART 1.5) originally developed and widely used in Europe and across the globe. We compared the model predictions with measured exposures on task-based exposure situations of solvent cleaning tasks and ultimately examined the feasibility of generic applicability of these European models to Korean situations and workplaces. Materials and methods Study design Seven occupational health (OH) professionals, including authors in the academic fields of occupational and environmental medicine, exposure science and industrial hygiene, each with at least 10 years of work experience in occupational exposure assessment from various exposure situations, companies and industries in Korea, participated in this study. All authors had hands-on technical practice and research experience in exposure assessment and modelling with professional training on the three exposure models for various exposure situations and industries as well as the calculation of model accuracy and conservatism in academic and private sectors. First, we intensively reviewed the governmental survey reports on study projects conducted by Ministry of Employment and Labour (MoEL) from 2005 to 2006, primarily aiming to revise the outdated Korean occupational exposure limits (KOELs) of various chemicals under the Korea Occupational Safety and Health Act (Jeong et al., 2010). A total of 126 MoEL survey reports originally contained a large volume of exposure monitoring datasets, including over 6000 repeated exposure measurements collected for various types of substances (solid, liquid, dust, etc.) in 450 task-based situations at several hundred different companies in Korea. The contextual information on the exposure situations, including physicochemical and toxicity data, detailed task descriptions, frequency and duration of tasks, working conditions, engineering controls, respiratory protective equipment and ventilation systems with several photographs taken during the tasks performed were also included in the MoEL reports. In this study, we used the collected exposure measurements for 10 organic solvents (n = 390) and contextual information on the selected exposure situations extracted from the only 10 of 126 MoEL survey reports for comparison of model predictions of volatile liquids with exposure measurement data using three models. Seven OH professionals, including the authors, performed exposure assessment modelling to predict inhalation exposure to 10 different solvents associated with 23 cleaning tasks in 51 exposure situations (34 for 8 h-TWA and 17 for short-term periods) at 33 companies in 15 industries in Korea (Table 1). Then, we compared the model predictions with the repeated exposure measurements extracted from the MoEL reports. Before performing exposure assessment modelling, four OH professionals, grouped into two teams, made site visits to 12 companies that mainly produced small electrical appliances, flat display panels, personal mobile devices and other products from January to June of 2016. The purpose of these site visits was to observe, identify and determine the principal factors, input parameters and related characteristics to enter them into each model for each exposure situation. Table 1. General characteristics of agent, industry, task, situation, and number of exposure measurement included in this study. Agent Industry Task Task description Company (n) Situation (n) Measurement (n) Reference Trichloroethylene Plastic manufacturing Electronics manufacturing Car parts manufacturing Zinc plating Parts cleaning Parts cleaning Parts cleaning Cleaning Dipping in cleaning bath Spraying solvent Spraying in cleaner Dipping in cleaning bath 9 14  8 h-TWA: 9  Short-term: 5 70  8 h-TWA: 42  Short-term: 28 Lee et al. (2006) Perchloroethylene Optical lens manufacturing Electronics manufacturing Gas manufacturing Fibre manufacturing Lens cleaning Parts cleaning Tank cleaning Product cleaning Spraying in cleaner Spraying solvent Spraying solvent Dipping in cleaning bath 5 6  8 h-TWA: 5  Short-term: 1 45  8 h-TWA: 41  Short-term: 4 Yoon et al. (2005b) Allyl alcohol Optical lens manufacturing Surfactant manufacturing Parts cleaning Tool cleaning Spraying solvent Spraying in cleaner 3 6  8 h-TWA: 3  Short-term: 3 68  8 h-TWA: 41  Short-term: 27 Kim et al. (2005) Acetone Electronics manufacturing Parts cleaning Tool cleaning Spraying solvent Spraying in cleaner 5 10  8 h-TWA: 5  Short-term: 5 104  8 h-TWA: 64  Short-term: 40 Lee et al. (2005b) Glutaraldehyde Detergent manufacturing Tool cleaning Spraying in cleaner 1 2  8 h-TWA: 1  Short-term: 1 7  8 h-TWA: 5  Short-term: 2 Paek et al. (2005b) Acetonitrile Drug manufacturing Parts cleaning Spraying solvent 1 2  8 h-TWA: 1  Short-term: 1 8  8 h-TWA: 4  Short-term: 4 Paek et al. (2005a) Toluene Display panel manufacturing Panel cleaning Spraying in cleaner 1 2  8 h-TWA: 1  Short-term: 1 6  8 h-TWA: 4  Short-term: 2 Lee et al. (2005a) 1-bromopropane Electronics manufacturing Military manufacturing Car parts manufacturing Parts cleaning Parts cleaning Parts cleaning Dipping in cleaning bath Dipping in cleaning bath Dipping in cleaning bath 3 3  8 h-TWA: 3 51  8 h-TWA: 51 Byun et al. (2005a) Cyclohexanone Screen printing Electronics manufacturing Printer cleaning Lens cleaning Pouring in cleaning bath Spraying in cleaner 2 2  8 h-TWA: 2 11  8 h-TWA: 11 Yoon et al. (2005a) HCFC-141b Mould manufacturing Metal manufacturing Display panel manufacturing Mould cleaning Cleaning Parts cleaning Spraying in cleaner Spraying in cleaner Spraying in cleaner 3 4  8 h-TWA: 4 20  8 h-TWA: 20 Byun et al. (2005b) Total 15 23 — 33 51  8 h-TWA: 34  Short-term: 17 390  8 h-TWA: 283  Short-term: 107 — Agent Industry Task Task description Company (n) Situation (n) Measurement (n) Reference Trichloroethylene Plastic manufacturing Electronics manufacturing Car parts manufacturing Zinc plating Parts cleaning Parts cleaning Parts cleaning Cleaning Dipping in cleaning bath Spraying solvent Spraying in cleaner Dipping in cleaning bath 9 14  8 h-TWA: 9  Short-term: 5 70  8 h-TWA: 42  Short-term: 28 Lee et al. (2006) Perchloroethylene Optical lens manufacturing Electronics manufacturing Gas manufacturing Fibre manufacturing Lens cleaning Parts cleaning Tank cleaning Product cleaning Spraying in cleaner Spraying solvent Spraying solvent Dipping in cleaning bath 5 6  8 h-TWA: 5  Short-term: 1 45  8 h-TWA: 41  Short-term: 4 Yoon et al. (2005b) Allyl alcohol Optical lens manufacturing Surfactant manufacturing Parts cleaning Tool cleaning Spraying solvent Spraying in cleaner 3 6  8 h-TWA: 3  Short-term: 3 68  8 h-TWA: 41  Short-term: 27 Kim et al. (2005) Acetone Electronics manufacturing Parts cleaning Tool cleaning Spraying solvent Spraying in cleaner 5 10  8 h-TWA: 5  Short-term: 5 104  8 h-TWA: 64  Short-term: 40 Lee et al. (2005b) Glutaraldehyde Detergent manufacturing Tool cleaning Spraying in cleaner 1 2  8 h-TWA: 1  Short-term: 1 7  8 h-TWA: 5  Short-term: 2 Paek et al. (2005b) Acetonitrile Drug manufacturing Parts cleaning Spraying solvent 1 2  8 h-TWA: 1  Short-term: 1 8  8 h-TWA: 4  Short-term: 4 Paek et al. (2005a) Toluene Display panel manufacturing Panel cleaning Spraying in cleaner 1 2  8 h-TWA: 1  Short-term: 1 6  8 h-TWA: 4  Short-term: 2 Lee et al. (2005a) 1-bromopropane Electronics manufacturing Military manufacturing Car parts manufacturing Parts cleaning Parts cleaning Parts cleaning Dipping in cleaning bath Dipping in cleaning bath Dipping in cleaning bath 3 3  8 h-TWA: 3 51  8 h-TWA: 51 Byun et al. (2005a) Cyclohexanone Screen printing Electronics manufacturing Printer cleaning Lens cleaning Pouring in cleaning bath Spraying in cleaner 2 2  8 h-TWA: 2 11  8 h-TWA: 11 Yoon et al. (2005a) HCFC-141b Mould manufacturing Metal manufacturing Display panel manufacturing Mould cleaning Cleaning Parts cleaning Spraying in cleaner Spraying in cleaner Spraying in cleaner 3 4  8 h-TWA: 4 20  8 h-TWA: 20 Byun et al. (2005b) Total 15 23 — 33 51  8 h-TWA: 34  Short-term: 17 390  8 h-TWA: 283  Short-term: 107 — HCFC-141b: 1,1-dichloro-1-fluoroethane. View Large Table 1. General characteristics of agent, industry, task, situation, and number of exposure measurement included in this study. Agent Industry Task Task description Company (n) Situation (n) Measurement (n) Reference Trichloroethylene Plastic manufacturing Electronics manufacturing Car parts manufacturing Zinc plating Parts cleaning Parts cleaning Parts cleaning Cleaning Dipping in cleaning bath Spraying solvent Spraying in cleaner Dipping in cleaning bath 9 14  8 h-TWA: 9  Short-term: 5 70  8 h-TWA: 42  Short-term: 28 Lee et al. (2006) Perchloroethylene Optical lens manufacturing Electronics manufacturing Gas manufacturing Fibre manufacturing Lens cleaning Parts cleaning Tank cleaning Product cleaning Spraying in cleaner Spraying solvent Spraying solvent Dipping in cleaning bath 5 6  8 h-TWA: 5  Short-term: 1 45  8 h-TWA: 41  Short-term: 4 Yoon et al. (2005b) Allyl alcohol Optical lens manufacturing Surfactant manufacturing Parts cleaning Tool cleaning Spraying solvent Spraying in cleaner 3 6  8 h-TWA: 3  Short-term: 3 68  8 h-TWA: 41  Short-term: 27 Kim et al. (2005) Acetone Electronics manufacturing Parts cleaning Tool cleaning Spraying solvent Spraying in cleaner 5 10  8 h-TWA: 5  Short-term: 5 104  8 h-TWA: 64  Short-term: 40 Lee et al. (2005b) Glutaraldehyde Detergent manufacturing Tool cleaning Spraying in cleaner 1 2  8 h-TWA: 1  Short-term: 1 7  8 h-TWA: 5  Short-term: 2 Paek et al. (2005b) Acetonitrile Drug manufacturing Parts cleaning Spraying solvent 1 2  8 h-TWA: 1  Short-term: 1 8  8 h-TWA: 4  Short-term: 4 Paek et al. (2005a) Toluene Display panel manufacturing Panel cleaning Spraying in cleaner 1 2  8 h-TWA: 1  Short-term: 1 6  8 h-TWA: 4  Short-term: 2 Lee et al. (2005a) 1-bromopropane Electronics manufacturing Military manufacturing Car parts manufacturing Parts cleaning Parts cleaning Parts cleaning Dipping in cleaning bath Dipping in cleaning bath Dipping in cleaning bath 3 3  8 h-TWA: 3 51  8 h-TWA: 51 Byun et al. (2005a) Cyclohexanone Screen printing Electronics manufacturing Printer cleaning Lens cleaning Pouring in cleaning bath Spraying in cleaner 2 2  8 h-TWA: 2 11  8 h-TWA: 11 Yoon et al. (2005a) HCFC-141b Mould manufacturing Metal manufacturing Display panel manufacturing Mould cleaning Cleaning Parts cleaning Spraying in cleaner Spraying in cleaner Spraying in cleaner 3 4  8 h-TWA: 4 20  8 h-TWA: 20 Byun et al. (2005b) Total 15 23 — 33 51  8 h-TWA: 34  Short-term: 17 390  8 h-TWA: 283  Short-term: 107 — Agent Industry Task Task description Company (n) Situation (n) Measurement (n) Reference Trichloroethylene Plastic manufacturing Electronics manufacturing Car parts manufacturing Zinc plating Parts cleaning Parts cleaning Parts cleaning Cleaning Dipping in cleaning bath Spraying solvent Spraying in cleaner Dipping in cleaning bath 9 14  8 h-TWA: 9  Short-term: 5 70  8 h-TWA: 42  Short-term: 28 Lee et al. (2006) Perchloroethylene Optical lens manufacturing Electronics manufacturing Gas manufacturing Fibre manufacturing Lens cleaning Parts cleaning Tank cleaning Product cleaning Spraying in cleaner Spraying solvent Spraying solvent Dipping in cleaning bath 5 6  8 h-TWA: 5  Short-term: 1 45  8 h-TWA: 41  Short-term: 4 Yoon et al. (2005b) Allyl alcohol Optical lens manufacturing Surfactant manufacturing Parts cleaning Tool cleaning Spraying solvent Spraying in cleaner 3 6  8 h-TWA: 3  Short-term: 3 68  8 h-TWA: 41  Short-term: 27 Kim et al. (2005) Acetone Electronics manufacturing Parts cleaning Tool cleaning Spraying solvent Spraying in cleaner 5 10  8 h-TWA: 5  Short-term: 5 104  8 h-TWA: 64  Short-term: 40 Lee et al. (2005b) Glutaraldehyde Detergent manufacturing Tool cleaning Spraying in cleaner 1 2  8 h-TWA: 1  Short-term: 1 7  8 h-TWA: 5  Short-term: 2 Paek et al. (2005b) Acetonitrile Drug manufacturing Parts cleaning Spraying solvent 1 2  8 h-TWA: 1  Short-term: 1 8  8 h-TWA: 4  Short-term: 4 Paek et al. (2005a) Toluene Display panel manufacturing Panel cleaning Spraying in cleaner 1 2  8 h-TWA: 1  Short-term: 1 6  8 h-TWA: 4  Short-term: 2 Lee et al. (2005a) 1-bromopropane Electronics manufacturing Military manufacturing Car parts manufacturing Parts cleaning Parts cleaning Parts cleaning Dipping in cleaning bath Dipping in cleaning bath Dipping in cleaning bath 3 3  8 h-TWA: 3 51  8 h-TWA: 51 Byun et al. (2005a) Cyclohexanone Screen printing Electronics manufacturing Printer cleaning Lens cleaning Pouring in cleaning bath Spraying in cleaner 2 2  8 h-TWA: 2 11  8 h-TWA: 11 Yoon et al. (2005a) HCFC-141b Mould manufacturing Metal manufacturing Display panel manufacturing Mould cleaning Cleaning Parts cleaning Spraying in cleaner Spraying in cleaner Spraying in cleaner 3 4  8 h-TWA: 4 20  8 h-TWA: 20 Byun et al. (2005b) Total 15 23 — 33 51  8 h-TWA: 34  Short-term: 17 390  8 h-TWA: 283  Short-term: 107 — HCFC-141b: 1,1-dichloro-1-fluoroethane. View Large Fortunately, the cleaning tasks and working conditions, previously assessed by a number of authors a decade ago, remained in the workplaces of the companies that we visited, and a number of cleaning workers performed similar tasks using the same solvents, although the products were manufactured in upgraded process tools with more efficient ventilation systems and engineering technologies. While visiting Korean companies, we observed chemical-specific task activities with the naked eye and collected the necessary information on the exposure situations, determinants and characteristics from the beginning to the end of the cleaning tasks performed. In general, we observed that the manufacturing processes, task activities and working conditions as well as the type and quantity of chemicals used remained considerably similar to those used when the exposure monitoring events were conducted by MoEL in the mid-2000s. Data collection We collected a total of 390 exposure measurements for 10 organic solvents, including 1-bromopropane (Byun et al., 2005a), acetone (Lee et al., 2005b), acetonitrile (Paek et al. 2005a), allyl alcohol (Kim et al., 2005), cyclohexanone (Yoon et al., 2005a), glutaraldehyde (Paek et al., 2005b), 1,1-dichloro-1-fluoroethane (HCFC-141b) (Byun et al., 2005b), perchloroethylene (Yoon et al., 2005b), toluene (Lee et al., 2005a), and trichloroethylene (Lee et al., 2006), from 10 MoEL survey reports. The exposure datasets included the repeated and task-related personal exposure measurements collected only for the cleaning tasks performed in the same workplaces at the same time (repeated) but on different workers. The repeated exposure measurements consisted of 107 short-term (from 15 to 30 min) and 283 time-weighted average (8 h-TWA) concentrations in parts per million (ppm) quantified using certified analytical methods (mostly NIOSH methods). We also collated detailed contextual information on exposure situations per chemical, including type of industry, task descriptions, process temperature (°C), sampling duration (min), number of workers, ventilation rates, type of local exhaust ventilation (LEV), efficiency of LEV, respiratory protection equipment (RPE), task duration and frequency, distance from the source, room volume (m3), quantity of chemical used (kg day−1), current KOELs (8 h-TWA, STEL or ceiling values) and chemical properties (≥99%), from the MoEL reports. Basic information related to chemical and physical properties, including molecular weight (g mol−1), vapour pressure (pa), boiling point (°C), water solubility (mg ml−1), partition coefficient (Kow) and mole fraction (%), were also collected from the official PubChem website, an open chemistry and biological activity database managed by the United States National Institutes of Health (NIH) (https://pubchem.ncbi.nlm.nih.gov/). The collected exposure measurements were mostly right-skewed, and some exposure levels (n = 22) for five chemical agents, such as trichloroethylene, allyl alcohol, toluene, 1-bromopropane and HCFC-141b, were left-censored and below the limit of detection (25%), medium (11–25%), or high (≤10%) level of conservatism of the modelled outcomes. Third, scatterplots were drawn to determine the correlations between the log-transformed predicted estimates and measured exposures by model, with a fitted line with the corresponding 95% CI. Pearson correlation coefficients (r) between the log-transformed 75th (ECETOC TRA) and 90th (ART and Stoffenmanager) percentile estimates and measured exposures in each situation were calculated to determine the model fitness in the scatterplots on situation level (n = 51) (Fig. 1). Figure 1. View largeDownload slide Scatterplots of the log-transformed predicted estimates and measured exposures for all organic solvents collected for 8 h-TWA and short-term situations on situation level by model: (a, d) ART (90th); (b, e) Stoffenmanager (90th); (c, f) ECETOC TRA (75th). Three figures (a, b, c) demonstrate the correlation for 8 h-TWA situations (n = 34), and the other three (d, e, f) show for short-term peak situations (n = 17). The dashed lines indicate the corresponding 95% confidence intervals (CI). Figure 1. View largeDownload slide Scatterplots of the log-transformed predicted estimates and measured exposures for all organic solvents collected for 8 h-TWA and short-term situations on situation level by model: (a, d) ART (90th); (b, e) Stoffenmanager (90th); (c, f) ECETOC TRA (75th). Three figures (a, b, c) demonstrate the correlation for 8 h-TWA situations (n = 34), and the other three (d, e, f) show for short-term peak situations (n = 17). The dashed lines indicate the corresponding 95% confidence intervals (CI). Finally, residuals were calculated as the log differences between the predicted estimates ( ŷ) and measured exposures (y) using equation (4): Residual =log(y^)−log(y) (4) The 75th (ECETOC TRA) and 90th percentiles (ART and Stoffenmanager) of the predicted estimate distributions were used for the calculation of log differences with individual measurements (n = 390). Similar to the bias, a positive residual indicated overestimation, whereas a negative residual meant underestimation of the modelled outcomes. Residual plots were also drawn to observe the log differences in the predicted estimates and individual exposure measurements by model on individual level (n = 390) (Savic et al., 2017a). All statistical analyses were performed using the SAS 9.4 software package (SAS, Inc., Cary, NC, USA). All exposure measurements and model predictions were log-transformed before calculating descriptive statistics, the lack of agreement, Pearson correlation coefficients and residuals in data analysis. The Anderson–Darling normality test was also performed to observe if the log-transformed data were log-normally distributed at a significance level of 0.05. Results The descriptive statistics of the Korean exposure measurements (n = 390) for the 10 organic solvents are shown in Tables 3 and 4. Table 3 presents AM, GM, and GSD of the 8 h-TWA measurements (n = 283) in 34 situations, and Table 4 shows for short-term measurements (n = 107) in 17 situations for each agent. Overall, the levels of exposure measurements for the 8 h-TWA and short-term situations were below the legal exposure limits (MoEL’s KOELs), and the mean levels of exposure in the short-term measurements were significantly higher than those in 8 h-TWAs (P < 0.05) (data not shown). Tables 5 and 6 present comparisons of the predicted estimates (50th and 90th percentiles for ART and Stoffenmanager, 75th percentile for ECETOC TRA) and the measured exposures (GM, 75th and 90th percentiles) for 8 h-TWA in Table 5 and the short-term period in Table 6 by agent. In general, the predicted outcomes from ART and Stoffenmanager models were more accurate than those from ECETOC TRA model for 8 h-TWA and short-term exposures on situation level (n = 51). Table 3. Descriptive statistics of Korean exposure measurement data (8 h-TWA) for organic solvents by situation. Agent Industry Task Exposure situation Duration (min) n predicted estimates Bias Relative bias (%) Precision Bias Relative bias (%) Precision Bias Relative bias (%) Precision ARTa (90th) STOFFENa (90th) ECETOCb (75th) Overall 390 −0.53 −41.17 2.11 −1.32 −73.38 5.88 −1.03 −64.35 8.88 16.67 14.36 52.31  8-h TWA 283 −0.29 −25.20 1.66 −1.94 −85.57 4.18 −5.29 −99.50 5.94 15.90 15.55 54.06   Trichloroethylene 42 0.22 24.93 0.75 −2.67 −93.09 3.20 −2.88 −94.41 4.82 14.29 16.67 61.90   Perchloroethylene 41 −0.13 −12.11 0.27 −1.11 −67.08 1.46 −0.98 −62.42 1.55 7.32 26.83 63.41   Allyl alcohol 41 0.18 19.79 0.37 0.86 136.43 1.03 −0.09 −8.22 0.59 29.27 0 21.95   Acetone 64 0.20 22.41 0.55 −0.59 −44.67 0.48 −0.91 −59.57 0.75 1.56 31.25 76.56   Glutaraldehyde 5 −0.15 −13.91 0.30 0.39 47.97 0.78 0.54 70.88 1.07 40.00 0 0   Acetonitrile 4 0.03 2.73 0.05 0.89 144.34 1.55 0.73 107.59 1.27 0 0 0   Toluene 4 −0.47 −37.77 0.82 0.31 36.78 0.54 0.53 69.90 0.92 50.00 0 0   1-bromopropane 51 0.07 7.03 0.05 −0.37 −30.81 0.54 −0.26 −22.57 0.37 27.45 7.84 43.14   Cyclohexanone 11 −0.10 −9.33 0.09 0.26 30.30 0.31 −0.22 −20.12 0.62 0 0 72.73   HCFC-141b 20 −0.14 −13.03 0.59 0.08 8.67 0.29 −1.75 −82.71 2.15 25.00 10.00 65.00  Short-term 107 −0.24 −21.35 1.31 0.61 84.54 1.86 4.26 6995.88 3.93 18.69 11.21 47.66   Trichloroethylene 28 −0.24 −21.62 0.36 −0.58 −44.17 0.64 −0.71 −51.08 1.09 17.86 0 71.43   Perchloroethylene 4 0.10 10.58 0.17 −0.22 −19.73 0.38 0.18 20.29 0.32 0 25.00 0   Allyl alcohol 27 −0.11 −10.54 0.38 0.13 13.77 0.35 0.42 51.82 1.15 44.44 18.52 18.52   Acetone 40 0.04 4.20 0.31 −1.07 −65.78 0.42 −0.89 −58.92 0.42 5.00 15.00 65.00   Glutaraldehyde 2 0.85 134.10 0.85 0.88 140.82 0.88 1.85 533.79 1.85 0 0 0   Acetonitrile 4 −0.19 −17.42 0.33 0.59 80.57 1.02 1.20 230.89 2.07 0 0 0   Toluene 2 −0.69 −49.65 0.69 0.89 143.23 0.89 2.22 821.95 2.22 50.00 0 0 Agent n ART (50th) STOFFEN (50th) ECETOC (75th) % measured exposures > predicted estimates Bias Relative bias (%) Precision Bias Relative bias (%) Precision Bias Relative bias (%) Precision ARTa (90th) STOFFENa (90th) ECETOCb (75th) Overall 390 −0.53 −41.17 2.11 −1.32 −73.38 5.88 −1.03 −64.35 8.88 16.67 14.36 52.31  8-h TWA 283 −0.29 −25.20 1.66 −1.94 −85.57 4.18 −5.29 −99.50 5.94 15.90 15.55 54.06   Trichloroethylene 42 0.22 24.93 0.75 −2.67 −93.09 3.20 −2.88 −94.41 4.82 14.29 16.67 61.90   Perchloroethylene 41 −0.13 −12.11 0.27 −1.11 −67.08 1.46 −0.98 −62.42 1.55 7.32 26.83 63.41   Allyl alcohol 41 0.18 19.79 0.37 0.86 136.43 1.03 −0.09 −8.22 0.59 29.27 0 21.95   Acetone 64 0.20 22.41 0.55 −0.59 −44.67 0.48 −0.91 −59.57 0.75 1.56 31.25 76.56   Glutaraldehyde 5 −0.15 −13.91 0.30 0.39 47.97 0.78 0.54 70.88 1.07 40.00 0 0   Acetonitrile 4 0.03 2.73 0.05 0.89 144.34 1.55 0.73 107.59 1.27 0 0 0   Toluene 4 −0.47 −37.77 0.82 0.31 36.78 0.54 0.53 69.90 0.92 50.00 0 0   1-bromopropane 51 0.07 7.03 0.05 −0.37 −30.81 0.54 −0.26 −22.57 0.37 27.45 7.84 43.14   Cyclohexanone 11 −0.10 −9.33 0.09 0.26 30.30 0.31 −0.22 −20.12 0.62 0 0 72.73   HCFC-141b 20 −0.14 −13.03 0.59 0.08 8.67 0.29 −1.75 −82.71 2.15 25.00 10.00 65.00  Short-term 107 −0.24 −21.35 1.31 0.61 84.54 1.86 4.26 6995.88 3.93 18.69 11.21 47.66   Trichloroethylene 28 −0.24 −21.62 0.36 −0.58 −44.17 0.64 −0.71 −51.08 1.09 17.86 0 71.43   Perchloroethylene 4 0.10 10.58 0.17 −0.22 −19.73 0.38 0.18 20.29 0.32 0 25.00 0   Allyl alcohol 27 −0.11 −10.54 0.38 0.13 13.77 0.35 0.42 51.82 1.15 44.44 18.52 18.52   Acetone 40 0.04 4.20 0.31 −1.07 −65.78 0.42 −0.89 −58.92 0.42 5.00 15.00 65.00   Glutaraldehyde 2 0.85 134.10 0.85 0.88 140.82 0.88 1.85 533.79 1.85 0 0 0   Acetonitrile 4 −0.19 −17.42 0.33 0.59 80.57 1.02 1.20 230.89 2.07 0 0 0   Toluene 2 −0.69 −49.65 0.69 0.89 143.23 0.89 2.22 821.95 2.22 50.00 0 0 ART, Advanced Reach Tool; STOFFEN, Stoffenmanager; ECETOC, ECETOC TRA. a90th percentiles of the predicted exposure distributions were used to evaluate the conservatism of ART and Stoffenmanager (<10%). b75th percentiles of the predicted exposure distributions were used to evaluate the conservatism of ECETOC TRA (<25%). View Large Table 7. Calculation of the lack of agreement for the predicted estimates with measured exposures by model. Agent n ART (50th) STOFFEN (50th) ECETOC (75th) % measured exposures > predicted estimates Bias Relative bias (%) Precision Bias Relative bias (%) Precision Bias Relative bias (%) Precision ARTa (90th) STOFFENa (90th) ECETOCb (75th) Overall 390 −0.53 −41.17 2.11 −1.32 −73.38 5.88 −1.03 −64.35 8.88 16.67 14.36 52.31  8-h TWA 283 −0.29 −25.20 1.66 −1.94 −85.57 4.18 −5.29 −99.50 5.94 15.90 15.55 54.06   Trichloroethylene 42 0.22 24.93 0.75 −2.67 −93.09 3.20 −2.88 −94.41 4.82 14.29 16.67 61.90   Perchloroethylene 41 −0.13 −12.11 0.27 −1.11 −67.08 1.46 −0.98 −62.42 1.55 7.32 26.83 63.41   Allyl alcohol 41 0.18 19.79 0.37 0.86 136.43 1.03 −0.09 −8.22 0.59 29.27 0 21.95   Acetone 64 0.20 22.41 0.55 −0.59 −44.67 0.48 −0.91 −59.57 0.75 1.56 31.25 76.56   Glutaraldehyde 5 −0.15 −13.91 0.30 0.39 47.97 0.78 0.54 70.88 1.07 40.00 0 0   Acetonitrile 4 0.03 2.73 0.05 0.89 144.34 1.55 0.73 107.59 1.27 0 0 0   Toluene 4 −0.47 −37.77 0.82 0.31 36.78 0.54 0.53 69.90 0.92 50.00 0 0   1-bromopropane 51 0.07 7.03 0.05 −0.37 −30.81 0.54 −0.26 −22.57 0.37 27.45 7.84 43.14   Cyclohexanone 11 −0.10 −9.33 0.09 0.26 30.30 0.31 −0.22 −20.12 0.62 0 0 72.73   HCFC-141b 20 −0.14 −13.03 0.59 0.08 8.67 0.29 −1.75 −82.71 2.15 25.00 10.00 65.00  Short-term 107 −0.24 −21.35 1.31 0.61 84.54 1.86 4.26 6995.88 3.93 18.69 11.21 47.66   Trichloroethylene 28 −0.24 −21.62 0.36 −0.58 −44.17 0.64 −0.71 −51.08 1.09 17.86 0 71.43   Perchloroethylene 4 0.10 10.58 0.17 −0.22 −19.73 0.38 0.18 20.29 0.32 0 25.00 0   Allyl alcohol 27 −0.11 −10.54 0.38 0.13 13.77 0.35 0.42 51.82 1.15 44.44 18.52 18.52   Acetone 40 0.04 4.20 0.31 −1.07 −65.78 0.42 −0.89 −58.92 0.42 5.00 15.00 65.00   Glutaraldehyde 2 0.85 134.10 0.85 0.88 140.82 0.88 1.85 533.79 1.85 0 0 0   Acetonitrile 4 −0.19 −17.42 0.33 0.59 80.57 1.02 1.20 230.89 2.07 0 0 0   Toluene 2 −0.69 −49.65 0.69 0.89 143.23 0.89 2.22 821.95 2.22 50.00 0 0 Agent n ART (50th) STOFFEN (50th) ECETOC (75th) % measured exposures > predicted estimates Bias Relative bias (%) Precision Bias Relative bias (%) Precision Bias Relative bias (%) Precision ARTa (90th) STOFFENa (90th) ECETOCb (75th) Overall 390 −0.53 −41.17 2.11 −1.32 −73.38 5.88 −1.03 −64.35 8.88 16.67 14.36 52.31  8-h TWA 283 −0.29 −25.20 1.66 −1.94 −85.57 4.18 −5.29 −99.50 5.94 15.90 15.55 54.06   Trichloroethylene 42 0.22 24.93 0.75 −2.67 −93.09 3.20 −2.88 −94.41 4.82 14.29 16.67 61.90   Perchloroethylene 41 −0.13 −12.11 0.27 −1.11 −67.08 1.46 −0.98 −62.42 1.55 7.32 26.83 63.41   Allyl alcohol 41 0.18 19.79 0.37 0.86 136.43 1.03 −0.09 −8.22 0.59 29.27 0 21.95   Acetone 64 0.20 22.41 0.55 −0.59 −44.67 0.48 −0.91 −59.57 0.75 1.56 31.25 76.56   Glutaraldehyde 5 −0.15 −13.91 0.30 0.39 47.97 0.78 0.54 70.88 1.07 40.00 0 0   Acetonitrile 4 0.03 2.73 0.05 0.89 144.34 1.55 0.73 107.59 1.27 0 0 0   Toluene 4 −0.47 −37.77 0.82 0.31 36.78 0.54 0.53 69.90 0.92 50.00 0 0   1-bromopropane 51 0.07 7.03 0.05 −0.37 −30.81 0.54 −0.26 −22.57 0.37 27.45 7.84 43.14   Cyclohexanone 11 −0.10 −9.33 0.09 0.26 30.30 0.31 −0.22 −20.12 0.62 0 0 72.73   HCFC-141b 20 −0.14 −13.03 0.59 0.08 8.67 0.29 −1.75 −82.71 2.15 25.00 10.00 65.00  Short-term 107 −0.24 −21.35 1.31 0.61 84.54 1.86 4.26 6995.88 3.93 18.69 11.21 47.66   Trichloroethylene 28 −0.24 −21.62 0.36 −0.58 −44.17 0.64 −0.71 −51.08 1.09 17.86 0 71.43   Perchloroethylene 4 0.10 10.58 0.17 −0.22 −19.73 0.38 0.18 20.29 0.32 0 25.00 0   Allyl alcohol 27 −0.11 −10.54 0.38 0.13 13.77 0.35 0.42 51.82 1.15 44.44 18.52 18.52   Acetone 40 0.04 4.20 0.31 −1.07 −65.78 0.42 −0.89 −58.92 0.42 5.00 15.00 65.00   Glutaraldehyde 2 0.85 134.10 0.85 0.88 140.82 0.88 1.85 533.79 1.85 0 0 0   Acetonitrile 4 −0.19 −17.42 0.33 0.59 80.57 1.02 1.20 230.89 2.07 0 0 0   Toluene 2 −0.69 −49.65 0.69 0.89 143.23 0.89 2.22 821.95 2.22 50.00 0 0 ART, Advanced Reach Tool; STOFFEN, Stoffenmanager; ECETOC, ECETOC TRA. a90th percentiles of the predicted exposure distributions were used to evaluate the conservatism of ART and Stoffenmanager (<10%). b75th percentiles of the predicted exposure distributions were used to evaluate the conservatism of ECETOC TRA (<25%). View Large Table 7 also presents the proportions of individual measurements exceeding the predicted percentile estimates (90th for ART and Stoffenmanager, 75th for ECETOC TRA) on individual level (n = 390). The overall percentages were 16.67% for ART, 14.36% for Stoffenmanager and 52.31% for ECETOC TRA. The model predictions showed medium conservative for ART and Stoffenmanager, but a low level of conservatism for ECETOC TRA, indicating that all model predictions were conservative in general. However, Stoffenmanager was relatively more conservative (15.55% in 8 h-TWA and 11.21% in short-term situations) than the other two models (ART and ECETOC TRA). Figure 1 demonstrates scatterplots of the log-transformed predicted estimates (90th percentiles for ART and Stoffenmanager, 75th percentile for ECETOC TRA) and measured exposures with Pearson correlation coefficients (r) and the corresponding 95% CI by model on situation level (n = 51). Overall, the correlations (8 h-TWA and short-term exposures combined) were significantly higher with ART (r = 0.95) and relatively high with Stoffenmanager (r = 0.82), but moderate with ECETOC TRA (r = 0.58) (P < 0.01) (data not shown). The correlations for each 8 h-TWA and short-term exposure situation were also high with ART (r = 0.96 for 8 h-TWA in Fig. 1a; r = 0.93 for short-term in Fig. 1d) and Stoffenmanager (r = 0.74 for 8 h-TWA in Fig. 1b; r = 0.90 for short-term in Fig. 1e), but moderate with ECETOC TRA (r = 0.55 for 8 h-TWA in Fig. 1c; r = 0.61 for short-term in Fig. 1f) (P < 0.01). Figure 2 shows residual plots of the differences between the log-transformed predicted estimates (90th percentiles for ART and Stoffenmanager, 75th percentile for ECETOC TRA) and measured exposures on individual level (n = 390) by model. For the 8 h-TWA exposures, the log mean differences were 1.31 ± 1.78 for ART (Fig. 2a), 1.91 ± 2.22 for Stoffenmanager (Fig. 2b), and 0.02 ± 2.58 for ECETOC TRA (Fig. 2c). The short-term peak exposures were 0.85 ± 1.68 (Fig. 2d), 1.76 ± 1.92 (Fig. 2e), and 0.88 ± 2.70 (Fig. 2f) for ART, Stoffenmanager and ECETOC TRA, respectively. Overall, a systematic tendency to overestimate low-level exposures and underestimate higher exposures was observed in all models. Figure 2. View largeDownload slide Residual plots of the differences in log-transformed predicted estimates and measured exposures for all organic solvents collected for 8 h-TWA and short-term situations on individual level by model: (a, d) ART (90th); (b, e) Stoffenmanager (90th); (c, f) ECETOC TRA (75th). Three figures (a, b, c) demonstrate the residuals for 8 h-TWA measurements (n = 283), and the other three (d, e, f) show for short-term peak measurements (n = 107). The log mean differences of ART [(a) 8 h-TWA: 1.31 ± 1.78; (d) short-term: 0.85 ± 1.68], Stoffenmanager [(b) 8 h-TWA: 1.91 ± 2.22; (e) short-term: 1.76 ± 1.92], and ECETOC TRA [(c) 8 h-TWA: 0.02 ± 2.58; (f) short-term: 0.88 ± 2.70] are marked with dashed lines. Figure 2. View largeDownload slide Residual plots of the differences in log-transformed predicted estimates and measured exposures for all organic solvents collected for 8 h-TWA and short-term situations on individual level by model: (a, d) ART (90th); (b, e) Stoffenmanager (90th); (c, f) ECETOC TRA (75th). Three figures (a, b, c) demonstrate the residuals for 8 h-TWA measurements (n = 283), and the other three (d, e, f) show for short-term peak measurements (n = 107). The log mean differences of ART [(a) 8 h-TWA: 1.31 ± 1.78; (d) short-term: 0.85 ± 1.68], Stoffenmanager [(b) 8 h-TWA: 1.91 ± 2.22; (e) short-term: 1.76 ± 1.92], and ECETOC TRA [(c) 8 h-TWA: 0.02 ± 2.58; (f) short-term: 0.88 ± 2.70] are marked with dashed lines. Discussion In this study, we evaluated the accuracy, precision and conservatism of the ART, Stoffenmanager and ECETOC TRA models by comparing the predicted estimates with Korean exposure measurements. We found that ART was the most accurate and precise model among the three models, and the predicted 90th percentile estimates were highly correlated with the measured 90th percentile exposures. Stoffenmanager was less accurate than ART but still showed high agreement and medium conservatism (i.e. the most balanced model), which was comparable or even higher than results from previous studies (Tielemans et al., 2008; Koppisch et al., 2012; van Tongeren et al., 2017). Unlike higher Tier models, ECETOC TRA showed less accurate outcomes for the 8 h-TWA (underestimation) and short-term (overestimation) exposure situations in terms of the lack of agreement, but it had moderate correlations with measured exposures. More importantly, we observed a systematic tendency to overestimate low exposures and underestimate higher exposures in all three models, similarly shown in previous studies (Landberg et al., 2017; Marquart et al., 2017; Savic et al., 2017a). Despite these results, we observed systematic error with ranges of the lack of agreement (bias ± precision) including zero, but the model outcomes were generally conservative with the overall percentages of the individual measurements exceeding the predicted estimates in all models. In this regard, two higher Tier models (ART and Stoffenmanager) showed medium levels of conservatism, but the model predictions estimated by a Tier 1 model (ECETOC TRA) showed a low level of conservatism. However, the overall levels of model conservatism in this study were still comparable to those from previous studies (Koppisch et al., 2012; Spinazzè et al., 2017; van Tongeren et al., 2017). Recently, Lamb et al. suggested that conservatism of Tier 1 models could be evaluated and classified into three categories, ‘high’ (≤10%), ‘medium’ (11–25%), or ‘low’ (>25%), depending on the percentages of individual measurements exceeding the predicted estimates (Lamb et al., 2015). When applying the same criteria of conservatism to our study (although it is not statistically relevant or valid for high Tier models), ART and Stoffenmanager had medium levels of conservatism (11–25%), but ECETOC TRA still had low conservatism (>25%). Our study results demonstrated that higher Tier models were more accurate, precise and conservative than Tier 1 models, especially for the prediction of exposure to volatile liquids in Korean exposure situations. Our study results showed good consistency and similarity with previous studies conducted in the United States and several European countries. In this study, ART produced the most accurate and precise outcomes, with higher agreement between the predicted estimates and Korean exposure measurements, but the model requires more detailed contextual information on the task-based situations evaluated. Mc Donnell et al. similarly found that ART was accurate but underestimated GM exposures to various substances for the scenarios in the pharmaceutical industry (with an uncertainty factor of 4.4) (Mc Donnell et al., 2011). Hofstetter et al. also found that ART performed better in terms of the experimental measurement results but still overestimated toluene exposures by a factor of 2.92 (Hofstetter et al., 2013). In Switzerland, Savic et al. conducted a validation study for a large exposure dataset (n = 584), and they found that the model predictions were still within the same order of magnitude as sufficient conservatism (Savic et al., 2017a). Landberg et al. also examined the validity of ART and Stoffenmanager in Sweden, and the authors found that ART had higher agreement with the exposure measurements than did Stoffenmanager, but ART underestimated the exposure to liquids (−0.55 ± 0.88) and powders (−1.4 ± 1.6) (Landberg et al., 2017). In Italy, Spinazzè et al. also evaluated the model predictions of ART, Stoffenmanager, and ECETOC TRA for solvents (n = 32) and pesticides (n = 14) in 14 exposure scenarios. The authors concluded that ART had the best accuracy but tended to underestimate the exposure to pesticides (Spinazzè et al., 2017). Furthermore, we observed that Stoffenmanager (between Tier 1 and 2 models) was the most balanced model, with good accuracy and high agreement with Korean exposure measurements and medium conservatism for the model predictions, which was similar to the results of previous studies. In the Netherlands, Schinkel et al. evaluated the model by comparing Stoffenmanager scores with inhalation exposure measurements for solid (n = 142) and volatile liquid (n = 112) scenarios. The authors found that Stoffenmanager underestimated exposure to solid and dust but was sufficiently conservative; thus, the model could be used as a Tier 1 screening model for regulatory risk assessment in this cross-validation study (Schinkel et al., 2010). Similarly, Koppisch et al. found that the overall correlation between Stoffenmanager scores and individual measurements (German exposure database named MEGA) was moderate but tended to underestimate most scenarios. Stoffenmanager also showed sufficient conservatism (11% for powders and granules, 7% for machining), indicating that the model could be regarded as a Tier 1 model (Koppisch et al., 2012). In Switzerland, Riedmann et al. conducted a sensitivity study to identify dominant factors in the three models, and the authors found that Stoffenmanager was the most balanced and robust model and was associated with less uncertainties in modifying factors than the other models (Riedmann et al., 2015). In the ETEAM Project, Stoffenmanager showed a medium level of conservatism (11% for volatile liquids), demonstrating that the model showed the most balanced performance in situations with liquids and powders (Lamb et al., 2015). Spinazzè et al. also concluded that Stoffenmanager was the best choice as an alternative model since the model showed neither underestimation nor overestimation in the scenarios, and it demonstrated acceptable results within an order of magnitude (Spinazzè et al., 2017). Landberg et al. noted that Stoffenmanager showed low agreement, with mean differences of 0.22 ± 1.00 (liquids) and −0.024 ± 0.66 (powders), and the Swedish authors also observed a systematic tendency to overestimate low exposures and underestimate higher ones (Landberg et al., 2017). Therefore, many studies recommended that Stoffenmanager was the most balanced, robust and conservative model among the three models. Unlike ART and Stoffenmanager, ECETOC TRA showed less accurate outcomes, lower level of conservatism and weaker (but moderate) correlations than the two high Tier models in our study. In Poland, Kupczewska-Dobecka et al. conducted a study to evaluate the ECETOC TRA model in terms of inhalation exposure to organic solvents in several scenarios. The authors found that ECETOC TRA had several advantages, such as easy use, clear structure (algorithm) and few data requirements. However, the model underestimated the high level of solvent exposure, and the predicted outcomes were still within an order of magnitude (Kupczewska-Dobecka et al., 2011). However, Hofstetter et al. found that ECETOC TRA overestimated the 8 h-TWA toluene measurements by a factor of 3.61 in the United States (Hofstetter et al., 2013). In Italy, Spinazzè et al. found that ECETOC TRA produced inaccurate and unrealistic exposure estimates (overestimation) and thus concluded that a Tier 1 model should be used as the first screening tool for inhalation exposure assessment, especially for low-volatile chemicals (Spinazzè et al., 2017). In the ETEAM Project, all Tier 1 models generally showed good conservatism, but some models were not always sufficiently conservative when considering the reasonable worst-case predictions in all exposure situations (Tischer et al., 2017). van Tongeren et al. also demonstrated similar results, in that the correlations between individual measurements and model predictions for volatile liquids were relatively low in the ECETOC TRA model (r = 0.34) (van Tongeren et al., 2017). Therefore, most studies similarly recommended that ECETOC TRA be used as a Tier 1 screening model due to lower accuracy, correlations and conservatism than higher Tier models, and the model overestimated but sometimes underestimated the exposures to various substances in general, as was similarly observed in our study. Despite the consistency with the results of previous studies, there are several factors that could produce differences in exposure situations in Korea and European countries. First, the time periods of exposure measurements collected in our study (between 2005 and 2006) are earlier than those in previous studies (between 2007 and 2015) (Koppisch et al., 2012; Landberg et al., 2017; Savic et al., 2017a; Spinazzè et al., 2017). Secondly, occupational health and safety (OSH) management systems, engineering technologies, administrative policies and OSH laws and regulations (especially for chemical exposures) in Korea are different from those in Europe (Jeong, 2004; Jeong et al., 2010; Cho, 2015). These differences might have increased the inconsistencies with different patterns, trends or directions in the model predictions, thus suggesting a lack of applicability for the models in Korean situations. Furthermore, the model algorithms were originally developed, rigorously calibrated and refined for prediction of occupational exposures in European countries, not in Korea. Although we did not adjust the differences or modify any parameters or factors in the models, we observed significantly high accuracy and precision with medium conservatism in the ART and Stoffenmanager models in Korean situations. Our results are highly consistent and comparable to those of several European studies. Therefore, our study findings confirm that these European models under REACH can be used as generic models to predict occupational inhalation exposures to organic solvents in several task-based situations in Korean workplaces. Our study has several strengths. To our knowledge, this study is the first study that not only evaluates model predictions of three European models on task-based situations of solvent cleaning but also examines the generic applicability of the models to Korean workplaces. We collated a large-scale exposure dataset and detailed contextual information on each situation using solvent cleaning from reliable MoEL reports published in Korea. Using the collected information on the situations and working conditions, we were able to adequately enter correct model parameters and modifying factors in each model through direct translation and coding processes with expert judgement, if necessary. Second, by applying the same approaches introduced by several European studies, we predicted the long-term (8 h-TWA) and short-term (15–30 min) peak exposures in the selected cleaning task situations and evaluated model accuracy, precision and conservatism by comparing the predicted and measured exposures on individual and situation levels. Most importantly, our study not only showed sufficient evidence of the feasibility of generic applicability of the models across countries (especially in an Asian country), but it also presented good information on the best fitted and balanced model for Korean users (OH professionals). The present study also provides useful information on model validation for a wide range of situations to model developers, downstream users and various stakeholders, including exposure scientists, consultants, governmental administrators, manufacturers and importers in Europe. Despite these strengths, there are some limitations in our study. First, there were uncertainties in the decisions regarding input parameters and modifying factors entered into the models. For example, we were unable to select PROC 28 (manual maintenance for cleaning) related to cleaning tasks or processes in ECETOC TRA due to its absence. Instead, we entered similar factors, such as PROC 10 and 13, into the model. In the Stoffenmanager model, we were unable to select specific types of LEV (e.g. capturing or enclosing hood, fixed or movable system) and ventilation rates since no option was available. Second, we could not visit all Korean workplaces and companies where the exposure measurements were actually collected in the mid-2000s because two-thirds of the companies had already closed. We failed to confirm whether or not the contextual information accurately or fully described the task-based exposure situations in the MoEL reports. This issue might have resulted in potential mistranslation of the information on the situations, although we carefully entered and coded parameters into all models through a high-level translation process. Furthermore, we could not observe any actual exposure monitoring event while we visited the Korean workplaces, and we could not check where the exposure samples were collected (either inside or outside RPE). In general, the exposure measurements are collected outside RPE but in the breathing zone under the Korea Occupational Safety and Health Act. These uncertainties probably came from a discrepancy between the information described (written) in MoEL reports and information observed in real-world workplaces, where the tasks and processes were actually performed. Finally, we could not identify which factor significantly affected model under- or overestimation in our study. To identify the most dominant factors influencing model predictions for various types of substances, a sensitivity analysis, such as that conducted in the Riedmann study (Riedmann et al., 2015), might help demonstrate how model predictions vary from entering the revised or adjusted parameters into the models, where uncertainties were produced (e.g. information discrepancy or between-user variability), as well as the results (under- or overestimation) expected in the model predictions compared with collected exposure measurements. Conclusion In summary, our study shows that ART was the most accurate model, and Stoffenmanager was the most balanced model, with good accuracy, high correlation and medium conservatism in the model predictions for the selected situations of solvent cleaning in Korea. Our study results are highly consistent with several European studies, suggesting these European models may be used as generic models to predict occupational inhalation exposure in Korea. Further studies are certainly needed to calibrate and refine the models and ultimately select the best fitted one for the applicability domain. Uncertainties should also be reduced by adjusting the differences between Korea and European countries as well as discrepancies between the information in the written documents and real-world workplaces in the future. Funding There is no financial support for this study. Acknowledgements The authors are grateful to several professionals, an occupational and environmental medicine physician (Dr Koh) and three Korean occupational health professionals (Dr Choi, Dr Kim and Mr Shin), who provided valuable time, helpful advices and technical supports for data collection and statistical analysis, site visits and exposure modelling in this study. Conflict of Interest The authors declare no conflict of interest relating to the material presented in this article. Its contents, including any opinions and/or conclusions expressed, are solely those of the authors. References Byun S , Kim Y , Choi J et al. ( 2005a ) A study on revision of Korean occupational exposure limits (1-Bromopropane) . Sejong, Republic of Korea : Ministry of Employment and Labour (MoEL) . Byun S , Kim Y , Choi J et al. ( 2005b ) A study on revision of Korean occupational exposure limits (1,1-Dichloro-1-fluoroethane) . Sejong, Republic of Korea : Ministry of Employment and Labour (MoEL) . Cho K-S . ( 2015 ) A study on the updates of Korean Occupational Health & Safety Laws and Regulations . Labor Review . EC . ( 1999 ) No 1907/2006 of the European Parliament and of the Council of 18 December 2006 concerning the registration. Evaluation, Authorisation and Restriction of Chemicals (REACH), establishing a European Chemicals Agency, amending Directive ; 45 : 1 – 849 . ECHA . ( 2016 ) Chapter R.14: occupational exposure assessment . In Guidance on information requirements and chemical safety assessment . Helsinki, Finland : European Chemicals Agency . Finkelstein MM , Verma DK . ( 2001 ) Exposure estimation in the presence of nondetectable values: another look . AIHAJ ; 62 : 195 – 8 . Google Scholar PubMed Fransman W . ( 2017 ) How accurate and reliable are exposure models ? Ann Work Expo Health ; 61 : 907 – 10 . Google Scholar Crossref Search ADS PubMed Helsel D . ( 2010 ) Much ado about next to nothing: incorporating nondetects in science . Ann Occup Hyg ; 54 : 257 – 62 . Google Scholar PubMed Hewett P , Ganser GH . ( 2007 ) A comparison of several methods for analyzing censored data . Ann Occup Hyg ; 51 : 611 – 32 . Google Scholar PubMed Hofstetter E , Spencer JW , Hiteshew K et al. ( 2013 ) Evaluation of recommended REACH exposure modeling tools and near-field, far-field model in assessing occupational exposure to toluene from spray paint . Ann Occup Hyg ; 57 : 210 – 20 . Google Scholar PubMed Ishii S et al. ( 2017 ) Evaluation of the ECETOC TRA model for workplace inhalation exposure to ethylbenzene in Japan . J Chem Health Safety ; 24 : 8 – 20 . Google Scholar Crossref Search ADS Jeong H . ( 2004 ) Global trends of hazardous chemical management in Europe and the U.S . Korean Soc Environ Health Toxicol ; 5 : 9 – 25 . Jeong JY , Choi S , Kho YL et al. ( 2010 ) Extensive changes to occupational exposure limits in Korea . Regul Toxicol Pharmacol ; 58 : 345 – 8 . Google Scholar Crossref Search ADS PubMed Kim C , Roh J , Won J et al. ( 2005 ) A study on revision of Korean occupational exposure limits (Allyl alcohol) . Sejong, Republic of Korea : Ministry of Employment and Labour (MoEL) . Koppisch D , Schinkel J , Gabriel S et al. ( 2012 ) Use of the MEGA exposure database for the validation of the Stoffenmanager model . Ann Occup Hyg ; 56 : 426 – 39 . Google Scholar PubMed Kupczewska-Dobecka M , Czerczak S , Jakubowski M . ( 2011 ) Evaluation of the TRA ECETOC model for inhalation workplace exposure to different organic solvents for selected process categories . Int J Occup Med Environ Health ; 24 : 208 – 17 . Google Scholar Crossref Search ADS PubMed Lamb J , Galea KS , Miller BG et al. ( 2017 ) Between-user reliability of tier 1 exposure assessment tools used under REACH . Ann Work Expo Health ; 61 : 939 – 53 . Google Scholar Crossref Search ADS PubMed Lamb J et al. ( 2015 ) Evaluation of Tier 1 exposure assessment models under REACH (eteam) Project . Bundesanstalt für Arbeitsschutz und Arbeitsmedizin (BAuA) . Landberg HE , Axmon A , Westberg H et al. ( 2017 ) A study of the validity of two exposure assessment tools: Stoffenmanager and the Advanced REACH Tool . Ann Work Expo Health ; 61 : 575 – 88 . Google Scholar Crossref Search ADS PubMed Landberg HE , Berg P , Andersson L et al. ( 2015 ) Comparison and evaluation of multiple users’ usage of the exposure and risk tool: Stoffenmanager 5.1 . Ann Occup Hyg ; 59 : 821 – 35 . Google Scholar Crossref Search ADS PubMed Lee S , Kim H , Kuh J et al. ( 2005a ) A study on revision of Korean occupational exposure limits (Toluene) . Sejong, Republic of Korea : Ministry of Employment and Labour (MoEL) . Lee S , Kuh J , Jeong H et al. ( 2005b ) A study on revision of Korean occupational exposure limits (Acetone) . Sejong, Republic of Korea : Ministry of Employment and Labour (MoEL) . Lee S , Kim H , Lim H et al. ( 2006 ) A study on revision of Korean occupational exposure limits (Trichloroethylene) . Ulsan, Republic of Korea : Korea Occupational Safety and Health Agency (KOSHA) . Lee J et al. ( 2012 ) Evaluation of the application of a European chemical risk assessment tool in Korea . J Korean Soc Occup Environ Hyg ; 22 : 191 – 9 . Marquart H , Franken R , Goede H et al. ( 2017 ) Validation of the dermal exposure model in ECETOC TRA . Ann Work Expo Health ; 61 : 854 – 71 . Google Scholar Crossref Search ADS PubMed Mc Donnell PE , Schinkel JM , Coggins MA et al. ( 2011 ) Validation of the inhalable dust algorithm of the Advanced REACH Tool using a dataset from the pharmaceutical industry . J Environ Monit ; 13 : 1597 – 606 . Google Scholar Crossref Search ADS PubMed Paek J , Lee Y , Ryu B et al. ( 2005a ) A study on revision of Korean occupational exposure limits (Acetonitrile) . Sejong, Republic of Korea : Ministry of Employment and Labour (MoEL) . Paek N , Jeong H , Choi K et al. ( 2005b ) A study on revision of Korean occupational exposure limits (Glutaraldehyde) . Sejong, Republic of Korea : Ministry of Employment and Labour (MoEL) . Riedmann RA , Gasic B , Vernez D . ( 2015 ) Sensitivity analysis, dominant factors, and robustness of the ECETOC TRA v3, Stoffenmanager 4.5, and ART 1.5 occupational exposure models . Risk Anal ; 35 : 211 – 25 . Google Scholar Crossref Search ADS PubMed Savic N , Gasic B , Schinkel J et al. ( 2017a ) Comparing the advanced REACH tool’s (ART) estimates with Switzerland’s occupational exposure data . Ann Work Expo Health ; 61 : 954 – 64 . Google Scholar Crossref Search ADS Savic N , Gasic B , Vernez D . ( 2017b ) ART, Stoffenmanager, and TRA: a systematic comparison of exposure estimates using the TREXMO translation system . Ann Work Expo Health ; 62 : 72 – 87 . Google Scholar Crossref Search ADS Schinkel J , Fransman W , Heussen H et al. ( 2010 ) Cross-validation and refinement of the Stoffenmanager as a first tier exposure assessment tool for REACH . Occup Environ Med ; 67 : 125 – 32 . Google Scholar Crossref Search ADS PubMed Schinkel J , Fransman W , McDonnell PE et al. ( 2014 ) Reliability of the Advanced REACH Tool (ART) . Ann Occup Hyg ; 58 : 450 – 68 . Google Scholar PubMed Spinazzè A , Lunghini F , Campagnolo D et al. ( 2017 ) Accuracy evaluation of three modelling tools for occupational exposure assessment . Ann Work Expo Health ; 61 : 284 – 98 . Google Scholar Crossref Search ADS PubMed Tielemans E , Noy D , Schinkel J et al. ( 2008 ) Stoffenmanager exposure model: development of a quantitative algorithm . Ann Occup Hyg ; 52 : 443 – 54 . Google Scholar PubMed Tischer M , Lamb J , Hesse S et al. ( 2017 ) Evaluation of Tier One Exposure Assessment Models (ETEAM): project overview and methods . Ann Work Expo Health ; 61 : 911 – 20 . Google Scholar Crossref Search ADS PubMed van Tongeren M , Lamb J , Cherrie JW et al. ( 2017 ) Validation of lower tier exposure tools used for REACH: comparison of tools estimates with available exposure measurements . Ann Work Expo Health ; 61 : 921 – 38 . Google Scholar Crossref Search ADS PubMed Yoon C , Kim Y , Park D et al. ( 2005a ) A study on revision of Korean occupational exposure limits (Cyclohexanone) . Sejong, Republic of Korea : Ministry of Employment and Labour (MoEL) . Yoon C , Kim Y , Park D et al. ( 2005b ) A study on revision of Korean occupational exposure limits (Perchloroethylene) . Sejong, Republic of Korea : Ministry of Employment and Labour (MoEL) . © The Author(s) 2018. Published by Oxford University Press on behalf of the British Occupational Hygiene Society. This article is published and distributed under the terms of the Oxford University Press, Standard Journals Publication Model (https://academic.oup.com/journals/pages/open_access/funder_policies/chorus/standard_publication_model) TI - Comparison of Quantitative Exposure Models for Occupational Exposure to Organic Solvents in Korea JF - Annals of Work Exposures and Health (formerly Annals Of Occupational Hygiene) DO - 10.1093/annweh/wxy087 DA - 2019-02-16 UR - https://www.deepdyve.com/lp/oxford-university-press/comparison-of-quantitative-exposure-models-for-occupational-exposure-9oVHc0sOKC SP - 197 VL - 63 IS - 2 DP - DeepDyve ER -