Intra-observer reliability and agreement of manual and digital orthodontic model analysis

Intra-observer reliability and agreement of manual and digital orthodontic model analysis Summary Background and aim Digital orthodontic model analysis is gaining acceptance in orthodontics, but its reliability is dependent on the digitalisation hardware and software used. We thus investigated intra-observer reliability and agreement / conformity of a particular digital model analysis work-flow in relation to traditional manual plaster model analysis. Materials and methods Forty-eight plaster casts of the upper/lower dentition were collected. Virtual models were obtained with orthoX®scan (Dentaurum) and analysed with ivoris®analyze3D (Computer konkret). Manual model analyses were done with a dial caliper (0.1 mm). Common parameters were measured on each plaster cast and its virtual counterpart five times each by an experienced observer. We assessed intra-observer reliability within method (ICC), agreement/conformity between methods (Bland–Altman analyses and Lin’s concordance correlation), and changing bias (regression analyses). Results Intra-observer reliability was substantial within each method (ICC ≥ 0.7), except for five manual outcomes (12.8 per cent). Bias between methods was statistically significant, but less than 0.5 mm for 87.2 per cent of the outcomes. In general, larger tooth sizes were measured digitally. Total difference maxilla and mandible had wide limits of agreement (−3.25/6.15 and −2.31/4.57 mm), but bias between methods was mostly smaller than intra-observer variation within each method with substantial conformity of manual and digital measurements in general. No changing bias was detected. Conclusions Although both work-flows were reliable, the investigated digital work-flow proved to be more reliable and yielded on average larger tooth sizes. Averaged differences between methods were within 0.5 mm for directly measured outcomes but wide ranges are expected for some computed space parameters due to cumulative error. Introduction Three-dimensional model analyses, based on plaster casts of the upper and lower dentition of a patient, are an integral part of orthodontic diagnostics. Due to limitations associated with conventional plaster dental casts, such as increased need of storage space and possible damage due to handling, digital scanners for obtaining virtual models and a number of software programs for digital model analyses have been developed and propagated in orthodontics. However, as a relatively new diagnostic procedure, its validity compared to the traditional model analysis with calipers remains to be assessed, especially, since a variety of hardware and software solutions are available to the user. Systematic reviews comparing various manual and digital work-flows generally agree that the digital work-flow itself can be trusted for clinical purposes (1–3). Due to increasing clinical acceptance of virtual models and a digital work-flow for model analysis, many new hardware and software solutions are commercially available. The orthoX®scan (DENTAURUM, Ispringen, Germany) is an optical 3D scanner, which allows the merging of pictures acquired with a camera recording a stripe-light projection into a 3D virtual model. The manufacturer states a scanning accuracy of less than 20 µm (http://www.dentaurum.de/eng/orthox-scan-24165.aspx, accessed on 21 December 2016). Exported data can be saved in open STL data format, which provides flexibility, since the scanner can be incorporated in existing software solutions. The software ivoris®analyze 3D (Computer konkret, Falkenstein, Germany) can import STL data and a variety of digital model analyses are available to the user. Until now, no study has been conducted to evaluate the performance of orthoX®scan and ivoris®analyze 3D. Therefore, we aimed to comparatively assess the traditional manual model analysis on plaster casts and digital model analysis with ivoris®analyze 3D, for which virtual models were acquired with orthoX®scan. Materials and methods This paper was reported according to Guidelines for Reporting Reliability and Agreement Studies (4). Dental plaster casts of orthodontic patients meeting the criteria listed in Table 1 were consecutively collected from the archives of the Department of Orthodontics, University Hospital Regensburg, Germany until 48 plaster casts were available for analysis. Characteristics of the sample are provided in Supplementary Table 1. Plaster casts were scanned once with orthoX®scan according to the manufacturer’s instructions. The three-dimensional virtual models were visually controlled for their precision and in case of inadequacies, scanning was repeated until high quality virtual models were acquired. After scanning, virtual models were imported into ivoris®analyze 3D for digital model analyses. The same plaster casts were then manually analysed. Table 1. Criteria for collecting dental casts. Permanent dentition (full erupted posterior segment apart from second molars) or early mixed dentition (full erupted incisors and first molars)  Good condition of dental casts (not fractured or degraded)  No missing teeth up to first molars (permanent dentitions) or first molars and incisors (early mixed dentitions)  No more than one dental cast for each patient  Irrespective of crowding, malocclusion, and previous orthodontic treatment  Permanent dentition (full erupted posterior segment apart from second molars) or early mixed dentition (full erupted incisors and first molars)  Good condition of dental casts (not fractured or degraded)  No missing teeth up to first molars (permanent dentitions) or first molars and incisors (early mixed dentitions)  No more than one dental cast for each patient  Irrespective of crowding, malocclusion, and previous orthodontic treatment  View Large We performed five manual and five digital repeated measurements for each outcome and case investigated. Manual model analyses were done with a conventional (identical for all the measurements) dial caliper at a precision of 0.1 mm and digital model analyses with ivoris®analyze 3D by one experienced investigator, familiarised with the software, in a random order regarding the patients’ casts and method of measurement (manual/digital) per round of repeated measurements. After a round of repeated measurements was completed, the next one was initiated until each outcome was measured five times on each plaster cast and its virtual counterpart. The number of repeated measurements was chosen in accordance with previous studies to allow a reliable assessment of central tendency. The observer was not blinded to patients’ personal information, such as name and date of birth, but was blinded to patients’ initial diagnosis, treatment plan, and previous repeated measurements with either method. As it usually happens in practices or clinics, the measuring process was often discontinued. Since we had not set a desired number of model analyses to be completed per day, measurements were performed with neither urge nor delay. Thus, measuring conditions approximated those in orthodontic clinical routine. All the measurements were completed within 2 months. Manual data were directly recorded into a pre-validated Excel spreadsheet, whereas digital data were first exported into a separate spreadsheet and then copied into the pre-validated spreadsheet. For both manual and digital methods, computations were automatically performed with Excel functions in the same way. Measurement points and the corresponding computations represent commonly used parameters in clinical practice and are listed in Table 2. Duration of the scanning process and model analysis for two repeated measurements per method was also recorded. Table 2. Definitions of measurements and corresponding computations performed for model analyses. Measurements*   Tooth sizes (15–25, 35–45)  Mesiodistal width of the respective tooth (only the anterior teeth in case of the early mixed dentition)   Available space at the posterior segment (15–13, 23–25, 35–33, 43–45)  Respective distance between the distal surfaces of the lateral incisors and the mesial surfaces of the first molars in the four quadrants of the jaw   Available space at the anterior segment (12–22, 32–42)  Sum of the distances between the distal surface of the lateral incisor and the mesial surface of the central incisor in the right and left section of the respective dental arch.   Interpremolar and intermolar widths maxilla  Distance between the centre of the fissures of the first maxillary premolars (or first deciduous molars) and distance between the deepest points of the main fissures of the maxillary first molars   Interpremolar and intermolar widths mandible  Distance between the distal contact points of the first mandibular premolars (or first deciduous molars) and distance between the distobuccal cusps of the mandibular first molars   Duration  Time needed for the aforementioned measurements to be completed (time needed for computations not included)  Computations**   Sum 12–22 (SI)  Sum of mesiodistal widths of maxillary incisors   Sum 32–42 (si)  Sum of mesiodistal widths of mandibular incisors   Total difference maxilla  Difference between the total available space in the upper dental arch (sum of available space at the anterior and posterior segments) and the sum of the mesiodistal widths of teeth 15–25 (Moyers analysis at 75% probability) in case of the early mixed dentitions (20)   Total difference mandible  Difference between the total available space in the lower dental arch (sum of available space at the anterior and posterior segments) and the sum of the mesiodistal widths of teeth 35–45 (Moyers analysis at 75% probability) in case of the early mixed dentitions (20)   Bolton 3-3  Ratio of the sum of mesiodistal widths of teeth 33–43 and 13–33 (not applicable to early mixed dentitions)   Bolton 5-5  Ratio of the sum of mesiodistal widths of teeth 35–45 and 15–25 (not applicable to early mixed dentitions)   Tonn  SI = 4/3 si + k (k = 0.4 if si < 22.2; k = 0.5 if 22.3 < si < 28.1; k = 0.6 if si > 28.1)   Transversal premolar relationship  Difference between maxillary and mandibular interpremolar widths   Transversal molar relationship  Difference between maxillary and mandibular intermolar widths  Measurements*   Tooth sizes (15–25, 35–45)  Mesiodistal width of the respective tooth (only the anterior teeth in case of the early mixed dentition)   Available space at the posterior segment (15–13, 23–25, 35–33, 43–45)  Respective distance between the distal surfaces of the lateral incisors and the mesial surfaces of the first molars in the four quadrants of the jaw   Available space at the anterior segment (12–22, 32–42)  Sum of the distances between the distal surface of the lateral incisor and the mesial surface of the central incisor in the right and left section of the respective dental arch.   Interpremolar and intermolar widths maxilla  Distance between the centre of the fissures of the first maxillary premolars (or first deciduous molars) and distance between the deepest points of the main fissures of the maxillary first molars   Interpremolar and intermolar widths mandible  Distance between the distal contact points of the first mandibular premolars (or first deciduous molars) and distance between the distobuccal cusps of the mandibular first molars   Duration  Time needed for the aforementioned measurements to be completed (time needed for computations not included)  Computations**   Sum 12–22 (SI)  Sum of mesiodistal widths of maxillary incisors   Sum 32–42 (si)  Sum of mesiodistal widths of mandibular incisors   Total difference maxilla  Difference between the total available space in the upper dental arch (sum of available space at the anterior and posterior segments) and the sum of the mesiodistal widths of teeth 15–25 (Moyers analysis at 75% probability) in case of the early mixed dentitions (20)   Total difference mandible  Difference between the total available space in the lower dental arch (sum of available space at the anterior and posterior segments) and the sum of the mesiodistal widths of teeth 35–45 (Moyers analysis at 75% probability) in case of the early mixed dentitions (20)   Bolton 3-3  Ratio of the sum of mesiodistal widths of teeth 33–43 and 13–33 (not applicable to early mixed dentitions)   Bolton 5-5  Ratio of the sum of mesiodistal widths of teeth 35–45 and 15–25 (not applicable to early mixed dentitions)   Tonn  SI = 4/3 si + k (k = 0.4 if si < 22.2; k = 0.5 if 22.3 < si < 28.1; k = 0.6 if si > 28.1)   Transversal premolar relationship  Difference between maxillary and mandibular interpremolar widths   Transversal molar relationship  Difference between maxillary and mandibular intermolar widths  *Although it was originally planned, overjet and overbite could not be digitally measured with ivoris®analyze 3D due to limitations of the analysis software. **Computed from the direct measurements on casts. View Large Medians of the five repeated measurements were calculated for each variable of interest per method (manual/digital) and considered for statistical analyses, unless otherwise stated. To represent the central tendency of our data, median was chosen over mean because it is less likely to be influenced from the extremes of a dataset (5), thus better representing the middle measured value(s) that are more likely to correspond to the true values. The intra-observer reliability was estimated with intraclass correlation coefficients (ICC) on all five repeated measurements per variable and method. We additionally used the Dahlberg’s formula to evaluate method error within methods. To assess agreement between methods, Bland–Altman analyses were performed with corresponding limits of agreement and 95 per cent confidence intervals (6). Conformity was assessed with Lin’s concordance correlation coefficient (CCC), which is robust for more than 10 pairs of data (7). For the ICC and CCC correlation coefficients, agreement was considered substantial, if values exceeded 0.7. Differences in means/systematic bias (bias, Δ) were considered clinically acceptable, if values were smaller than 0.5 mm (8, 9). Changing bias relative to the true values was investigated by plotting the data and running regression analyses (10). Normality of the data was confirmed with Shapiro–Wilk tests and a visual inspection of histograms. Power calculation was performed post hoc in G*Power 3.1 (11) by taking into account a bivariate normal model and an effect size of 0.6, which is well below effect sizes reported in literature and the cut-off value adopted in this study. Statistical analyses were performed with Microsoft Excel® 2010 (Microsoft Corp., Redmond, USA), IBM SPSS® Statistics software version 23.0 (IBM Corp., Armonk, NY), and the online NIWA calculator (https://www.niwa.co.nz/node/104318/concordance, accessed on 21 December 2016). An overview of the study design as well as statistical procedures is provided in Supplementary Figure 1. Results According to the post hoc power analyses with α (two-tailed) = 0.05, effect size = 0.6, and n = 48 or n = 24, achieved power (1 − β) of the study was 0.99 or 0.90, respectively. Results for intra-observer reliability and agreement/conformity are provided in Table 3. Descriptive statistics are additionally provided in Supplementary Table 2, and Bland–Altman plots in Supplementary Figure 2. Table 3. Results on intra-observer reliability and agreement.     Intra-examiner reliability (within method)  Intra-examiner agreement (between methods)  Outcomes  Manual ICC (95% CI)*  Digital ICC (95% CI)*  Manual Dahlberg’s error** mm  Digital Dahlberg’s error** mm  Mean of Δ (SD) mm  Δ different from 0 P value  Lower LOA (95% CI) mm  Upper LOA (95% CI) mm  Range of lower and upper LOA mm  Changing bias regression r / P value  Conformity CCC (95% CI)  2-landmark measurements on casts  Tooth size 15 (n = 24)  0.73 (0.56 / 0.86)  0.78 (0.65 / 0.88)  0.33  0.26  −0.28 (0.27)  <0.001  −0.8 (−0.98 / −0.61)  0.25 (0.06 / 0.43)  1.05  0.03 / 0.892  0.64 (0.4 / 0.8)  Tooth size 14 (n = 24)  0.72 (0.56 / 0.84)  0.82 (0.7 / 0.9)  0.3  0.22  −0.35 (0.28)  <0.001  −0.89 (−1.08 / −0.7)  0.19 (0 / 0.38)  1.08  0.07 / 0.739  0.61 (0.38 / 0.77)  Tooth size 13 (n = 24)  0.82 (0.7 / 0.9)  0.77 (0.64 / 0.88)  0.2  0.25  0.03 (0.25)  0.577  −0.47 (−0.64 / −0.29)  0.52 (0.35 / 0.7)  0.99  0.22 / 0.305  0.79 (0.58 / 0.9)  Tooth size 12  0.86 (0.8 / 0.91)  0.88 (0.81 / 0.92)  0.21  0.22  −0.13 (0.4)  0.026  −0.91 (−1.1 / −0.72)  0.65 (0.45 / 0.84)  1.56  0.13 / 0.374  0.66 (0.47 / 0.79)  Tooth size 11  0.9 (0.85 / 0.94)  0.92 (0.88 / 0.95)  0.21  0.19  −0.03 (0.31)  0.549  −0.64 (−0.79 / −0.49)  0.58 (0.43 / 0.73)  1.22  0.21 / 0.16  0.82 (0.7 / 0.9)  Tooth size 21  0.92 (0.87 / 0.95)  0.88 (0.8 / 0.92)  0.17  0.25  −0.2 (0.34)  <0.001  −0.87 (−1.04 / −0.71)  0.47 (0.3 / 0.63)  1.34  0.04 / 0.81  0.75 (0.6 / 0.84)  Tooth size 22  0.88 (0.82 / 0.92)  0.84 (0.76 / 0.9)  0.17  0.25  −0.17 (0.4)  0.006  −0.96 (−1.16 / −0.77)  0.62 (0.43 / 0.82)  1.58  0.11 / 0.446  0.61 (0.43 / 0.75)  Tooth size 23 (n = 24)  0.87 (0.78 / 0.93)  0.7 (0.55 / 0.84)  0.17  0.3  −0.06 (0.31)  0.371  −0.67 (−0.89 / −0.46)  0.56 (0.34 / 0.77)  1.23  0.46 / 0.023  0.72 (0.46 / 0.87)  Tooth size 24 (n = 24)  0.81 (0.69 / 0.9)  0.81 (0.69 / 0.9)  0.21  0.3  −0.29 (0.29)  <0.001  −0.86 (−1.06 / −0.66)  0.29 (0.09 / 0.49)  1.15  0.02 / 0.915  0.64 (0.4 / 0.8)  Tooth size 25 (n = 24)  0.76 (0.63 / 0.87)  0.72 (0.57 / 0.85)  0.27  0.24  −0.34 (0.19)  <0.001  −0.72 (−0.85 / −0.58)  0.04 (−0.09 / 0.17)  0.76  0.27 / 0.209  0.65 (0.45 / 0.79)  Tooth size 35 (n = 24)  0.67 (0.51 / 0.82)  0.76 (0.59 / 0.88)  0.33  0.2  −0.22 (0.27)  0.001  −0.74 (−0.93 / −0.56)  0.31 (0.13 / 0.49)  1.05  0.16 / 0.47  0.72 (0.49 / 0.85)  Tooth size 34 (n = 24)  0.77 (0.63 / 0.88)  0.83 (0.72 / 0.91)  0.27  0.23  −0.52 (0.26)  <0.001  −1.04 (−1.22 / −0.85)  0 (−0.18 / 0.18)  1.04  0.01 / 0.975  0.47 (0.26 / 0.63)  Tooth size 33 (n = 24)  0.76 (0.63 / 0.87)  0.88 (0.79 / 0.94)  0.24  0.19  0.06 (0.24)  0.238  −0.4 (−0.57 / −0.24)  0.52 (0.36 / 0.68)  0.92  0.05 / 0.807  0.81 (0.63 / 0.91)  Tooth size 32  0.79 (0.7 / 0.86)  0.84 (0.78 / 0.9)  0.17  0.17  −0.06 (0.24)  0.104  −0.54 (−0.65 / −0.42)  0.42 (0.3 / 0.54)  0.96  0.23 / 0.118  0.7 (0.53 / 0.82)  Tooth size 31  0.79 (0.69 / 0.86)  0.86 (0.8 / 0.91)  0.15  0.14  −0.07 (0.19)  0.015  −0.44 (−0.53 / −0.35)  0.3 (0.21 / 0.39)  0.74  0.04 / 0.807  0.8 (0.67 / 0.88)  Tooth size 41  0.68 (0.56 / 0.78)  0.79 (0.71 / 0.86)  0.18  0.19  −0.01 (0.19)  0.82  −0.38 (−0.47 / −0.29)  0.37 (0.27 / 0.46)  0.75  0.09 / 0.546  0.77 (0.62 / 0.86)  Tooth size 42  0.75 (0.65 / 0.83)  0.86 (0.8 / 0.91)  0.18  0.19  0 (0.23)  0.951  −0.46 (−0.57 / −0.34)  0.45 (0.34 / 0.56)  0.91  0.12 / 0.409  0.79 (0.66 / 0.88)  Tooth size 43 (n = 24)  0.87 (0.79 / 0.94)  0.82 (0.71 / 0.91)  0.13  0.19  0.06 (0.16)  0.09  −0.26 (−0.37 / −0.15)  0.37 (0.26 / 0.48)  0.63  0.1 / 0.654  0.91 (0.81 / 0.96)  Tooth size 44 (n = 24)  0.77 (0.61 / 0.88)  0.8 (0.68 / 0.89)  0.22  0.23  −0.32 (0.21)  <0.001  −0.74 (−0.88 / −0.59)  0.1 (−0.05 / 0.24)  0.84  0.03 / 0.882  0.69 (0.5 / 0.82)  Tooth size 45 (n = 24)  0.85 (0.73 / 0.92)  0.81 (0.7 / 0.9)  0.21  0.24  −0.3 (0.29)  <0.001  −0.86 (−1.05 / −0.66)  0.27 (0.07 / 0.46)  1.13  0.13 / 0.546  0.67 (0.44 / 0.82)  Available space 15–13  0.95 (0.91 / 0.97)  0.88 (0.83 / 0.92)  0.21  0.32  0.11 (0.43)  0.089  −0.74 (−0.95 / −0.53)  0.95 (0.75 / 1.16)  1.69  0.1 / 0.487  0.95 (0.92 / 0.97)  Available space 12–22  0.89 (0.84 / 0.93)  0.96 (0.95 / 0.98)  0.51  0.33  0.19 (0.43)  0.004  −0.66 (−0.87 / −0.45)  1.03 (0.83 / 1.24)  1.69  0.15 / 0.311  0.96 (0.92 / 0.97)  Available space 23–25  0.95 (0.93 / 0.97)  0.96 (0.94 / 0.98)  0.22  0.26  −0.02 (0.38)  0.762  −0.76 (−0.94 / −0.58)  0.73 (0.54 / 0.91)  1.49  0.04 / 0.805  0.96 (0.93 / 0.98)  Available space 35–33  0.98 (0.97 / 0.99)  0.98 (0.97 / 0.99)  0.17  0.2  0.06 (0.24)  0.085  −0.41 (−0.52 / −0.29)  0.53 (0.41 / 0.64)  0.94  0.42 / 0.003  0.99 (0.98 / 0.99)  Available space 32–42  0.91 (0.87 / 0.94)  0.97 (0.95 / 0.98)  0.39  0.22  0.30 (0.41)  <0.001  −0.5 (−0.7 / −0.3)  1.1 (0.9 / 1.29)  1.6  0.29 / 0.048  0.9 (0.84 / 0.94)  Available space 43–45  0.98 (0.96 / 0.99)  0.98 (0.97 / 0.99)  0.19  0.26  0.01 (0.28)  0.836  −0.54 (−0.67 / −0.4)  0.55 (0.42 / 0.69)  1.09  0.25 / 0.082  0.98 (0.97 / 0.99)  Interpremolar width maxilla  0.99 (0.98 / 0.99)  0.99 (0.99 / 1)  0.24  0.22  0.27 (0.27)  <0.001  −0.26 (−0.39 / −0.12)  0.8 (0.67 / 0.93)  1.06  0.06 / 0.699  0.99 (0.98 / 0.99)  Interpremolar width mandible  0.96 (0.94 / 0.98)  0.99 (0.99 / 1)  0.52  0.18  0.08 (0.26)  0.044  −0.44 (−0.57 / −0.31)  0.6 (0.47 / 0.73)  1.04  0.07 / 0.635  0.99 (0.98 / 0.99)  Intermolar width maxilla  0.96 (0.93 / 0.97)  0.98 (0.97 / 0.99)  0.56  0.47  0.71 (0.39)  <0.001  −0.05 (−0.24 / 0.14)  1.46 (1.28 / 1.65)  1.51  0.1 / 0.497  0.95 (0.93 / 0.97)  Intermolar width mandible  0.95 (0.92 / 0.97)  0.97 (0.95 / 0.98)  0.34  0.63  0.63 (0.38)  <0.001  −0.12 (−0.3 / 0.06)  1.37 (1.19 / 1.55)  1.49  0.01 / 0.944  0.95 (0.91 / 0.97)  Computed measurements  Sum 12–22  0.92 (0.89 / 0.95)  0.92 (0.85 / 0.96)  0.52  0.6  −0.48 (1.12)  0.005  −2.67 (−3.21 / −2.13)  1.71 (1.17 / 2.25)  4.38  0.11 / 0.451  0.74 (0.59 / 0.84)  Sum 32–42  0.82 (0.74 / 0.88)  0.93 (0.89 / 0.96)  0.46  0.4  −0.19 (0.61)  0.036  −1.39 (−1.68 / −1.09)  1.01 (0.71 / 1.3)  2.4  0.21 / 0.143  0.84 (0.73 / 0.9)  Total difference maxilla  0.82 (0.73 / 0.89)  0.86 (0.74 / 0.92)  1.15  1.05  1.45 (2.4)  <0.001  −3.25 (−4.41 / −2.09)  6.15 (4.99 / 7.31)  9.4  0.18 / 0.221  0.56 (0.37 / 0.71)  Total difference mandible  0.83 (0.73 / 0.89)  0.91 (0.84 / 0.95)  1.08  0.94  1.13 (1.76)  <0.001  −2.31 (−3.16 / −1.46)  4.57 (3.72 / 5.42)  6.88  0.1 / 0.501  0.72 (0.57 / 0.83)  Bolton 3-3 (n = 24)  0.66 (0.49 / 0.81)  0.81 (0.7 / 0.9)  0.01  0.01  0.01 (0.02)  0.004  −0.02 (−0.04 / −0.01)  0.05 (0.04 / 0.06)  0.07  0.01 / 0.981  0.56 (0.27 / 0.76)  Bolton 5-5 (n = 24)  0.59 (0.41 / 0.76)  0.81 (0.69 / 0.9)  0.01  0.01  0.01 (0.02)  0.177  −0.03 (−0.04 / −0.02)  0.04 (0.03 / 0.05)  0.07  0.02 / 0.913  0.55 (0.23 / 0.76)  Tonn  0.62 (0.5 / 0.74)  0.83 (0.75 / 0.89)  0.61  0.55  −0.27 (0.91)  0.041  −2.05 (−2.49 / −1.61)  1.5 (1.06 / 1.94)  3.55  0.05 / 0.762  0.62 (0.43 / 0.76)  Transversal premolar relationship  0.93 (0.89 / 0.95)  0.98 (0.96 / 0.98)  0.55  0.26  0.19 (0.37)  0.001  −0.52 (−0.7 / −0.35)  0.91 (0.73 / 1.09)  1.43  0.34 / 0.017  0.96 (0.94 / 0.98)  Transversal molar relationship  0.87 (0.78 / 0.93)  0.92 (0.88 / 0.95)  0.66  0.67  0.02 (0.57)  0.821  −1.1 (−1.37 / −0.82)  1.14 (0.86 / 1.41)  2.24  0.02 / 0.891  0.94 (0.9 / 0.97)      Intra-examiner reliability (within method)  Intra-examiner agreement (between methods)  Outcomes  Manual ICC (95% CI)*  Digital ICC (95% CI)*  Manual Dahlberg’s error** mm  Digital Dahlberg’s error** mm  Mean of Δ (SD) mm  Δ different from 0 P value  Lower LOA (95% CI) mm  Upper LOA (95% CI) mm  Range of lower and upper LOA mm  Changing bias regression r / P value  Conformity CCC (95% CI)  2-landmark measurements on casts  Tooth size 15 (n = 24)  0.73 (0.56 / 0.86)  0.78 (0.65 / 0.88)  0.33  0.26  −0.28 (0.27)  <0.001  −0.8 (−0.98 / −0.61)  0.25 (0.06 / 0.43)  1.05  0.03 / 0.892  0.64 (0.4 / 0.8)  Tooth size 14 (n = 24)  0.72 (0.56 / 0.84)  0.82 (0.7 / 0.9)  0.3  0.22  −0.35 (0.28)  <0.001  −0.89 (−1.08 / −0.7)  0.19 (0 / 0.38)  1.08  0.07 / 0.739  0.61 (0.38 / 0.77)  Tooth size 13 (n = 24)  0.82 (0.7 / 0.9)  0.77 (0.64 / 0.88)  0.2  0.25  0.03 (0.25)  0.577  −0.47 (−0.64 / −0.29)  0.52 (0.35 / 0.7)  0.99  0.22 / 0.305  0.79 (0.58 / 0.9)  Tooth size 12  0.86 (0.8 / 0.91)  0.88 (0.81 / 0.92)  0.21  0.22  −0.13 (0.4)  0.026  −0.91 (−1.1 / −0.72)  0.65 (0.45 / 0.84)  1.56  0.13 / 0.374  0.66 (0.47 / 0.79)  Tooth size 11  0.9 (0.85 / 0.94)  0.92 (0.88 / 0.95)  0.21  0.19  −0.03 (0.31)  0.549  −0.64 (−0.79 / −0.49)  0.58 (0.43 / 0.73)  1.22  0.21 / 0.16  0.82 (0.7 / 0.9)  Tooth size 21  0.92 (0.87 / 0.95)  0.88 (0.8 / 0.92)  0.17  0.25  −0.2 (0.34)  <0.001  −0.87 (−1.04 / −0.71)  0.47 (0.3 / 0.63)  1.34  0.04 / 0.81  0.75 (0.6 / 0.84)  Tooth size 22  0.88 (0.82 / 0.92)  0.84 (0.76 / 0.9)  0.17  0.25  −0.17 (0.4)  0.006  −0.96 (−1.16 / −0.77)  0.62 (0.43 / 0.82)  1.58  0.11 / 0.446  0.61 (0.43 / 0.75)  Tooth size 23 (n = 24)  0.87 (0.78 / 0.93)  0.7 (0.55 / 0.84)  0.17  0.3  −0.06 (0.31)  0.371  −0.67 (−0.89 / −0.46)  0.56 (0.34 / 0.77)  1.23  0.46 / 0.023  0.72 (0.46 / 0.87)  Tooth size 24 (n = 24)  0.81 (0.69 / 0.9)  0.81 (0.69 / 0.9)  0.21  0.3  −0.29 (0.29)  <0.001  −0.86 (−1.06 / −0.66)  0.29 (0.09 / 0.49)  1.15  0.02 / 0.915  0.64 (0.4 / 0.8)  Tooth size 25 (n = 24)  0.76 (0.63 / 0.87)  0.72 (0.57 / 0.85)  0.27  0.24  −0.34 (0.19)  <0.001  −0.72 (−0.85 / −0.58)  0.04 (−0.09 / 0.17)  0.76  0.27 / 0.209  0.65 (0.45 / 0.79)  Tooth size 35 (n = 24)  0.67 (0.51 / 0.82)  0.76 (0.59 / 0.88)  0.33  0.2  −0.22 (0.27)  0.001  −0.74 (−0.93 / −0.56)  0.31 (0.13 / 0.49)  1.05  0.16 / 0.47  0.72 (0.49 / 0.85)  Tooth size 34 (n = 24)  0.77 (0.63 / 0.88)  0.83 (0.72 / 0.91)  0.27  0.23  −0.52 (0.26)  <0.001  −1.04 (−1.22 / −0.85)  0 (−0.18 / 0.18)  1.04  0.01 / 0.975  0.47 (0.26 / 0.63)  Tooth size 33 (n = 24)  0.76 (0.63 / 0.87)  0.88 (0.79 / 0.94)  0.24  0.19  0.06 (0.24)  0.238  −0.4 (−0.57 / −0.24)  0.52 (0.36 / 0.68)  0.92  0.05 / 0.807  0.81 (0.63 / 0.91)  Tooth size 32  0.79 (0.7 / 0.86)  0.84 (0.78 / 0.9)  0.17  0.17  −0.06 (0.24)  0.104  −0.54 (−0.65 / −0.42)  0.42 (0.3 / 0.54)  0.96  0.23 / 0.118  0.7 (0.53 / 0.82)  Tooth size 31  0.79 (0.69 / 0.86)  0.86 (0.8 / 0.91)  0.15  0.14  −0.07 (0.19)  0.015  −0.44 (−0.53 / −0.35)  0.3 (0.21 / 0.39)  0.74  0.04 / 0.807  0.8 (0.67 / 0.88)  Tooth size 41  0.68 (0.56 / 0.78)  0.79 (0.71 / 0.86)  0.18  0.19  −0.01 (0.19)  0.82  −0.38 (−0.47 / −0.29)  0.37 (0.27 / 0.46)  0.75  0.09 / 0.546  0.77 (0.62 / 0.86)  Tooth size 42  0.75 (0.65 / 0.83)  0.86 (0.8 / 0.91)  0.18  0.19  0 (0.23)  0.951  −0.46 (−0.57 / −0.34)  0.45 (0.34 / 0.56)  0.91  0.12 / 0.409  0.79 (0.66 / 0.88)  Tooth size 43 (n = 24)  0.87 (0.79 / 0.94)  0.82 (0.71 / 0.91)  0.13  0.19  0.06 (0.16)  0.09  −0.26 (−0.37 / −0.15)  0.37 (0.26 / 0.48)  0.63  0.1 / 0.654  0.91 (0.81 / 0.96)  Tooth size 44 (n = 24)  0.77 (0.61 / 0.88)  0.8 (0.68 / 0.89)  0.22  0.23  −0.32 (0.21)  <0.001  −0.74 (−0.88 / −0.59)  0.1 (−0.05 / 0.24)  0.84  0.03 / 0.882  0.69 (0.5 / 0.82)  Tooth size 45 (n = 24)  0.85 (0.73 / 0.92)  0.81 (0.7 / 0.9)  0.21  0.24  −0.3 (0.29)  <0.001  −0.86 (−1.05 / −0.66)  0.27 (0.07 / 0.46)  1.13  0.13 / 0.546  0.67 (0.44 / 0.82)  Available space 15–13  0.95 (0.91 / 0.97)  0.88 (0.83 / 0.92)  0.21  0.32  0.11 (0.43)  0.089  −0.74 (−0.95 / −0.53)  0.95 (0.75 / 1.16)  1.69  0.1 / 0.487  0.95 (0.92 / 0.97)  Available space 12–22  0.89 (0.84 / 0.93)  0.96 (0.95 / 0.98)  0.51  0.33  0.19 (0.43)  0.004  −0.66 (−0.87 / −0.45)  1.03 (0.83 / 1.24)  1.69  0.15 / 0.311  0.96 (0.92 / 0.97)  Available space 23–25  0.95 (0.93 / 0.97)  0.96 (0.94 / 0.98)  0.22  0.26  −0.02 (0.38)  0.762  −0.76 (−0.94 / −0.58)  0.73 (0.54 / 0.91)  1.49  0.04 / 0.805  0.96 (0.93 / 0.98)  Available space 35–33  0.98 (0.97 / 0.99)  0.98 (0.97 / 0.99)  0.17  0.2  0.06 (0.24)  0.085  −0.41 (−0.52 / −0.29)  0.53 (0.41 / 0.64)  0.94  0.42 / 0.003  0.99 (0.98 / 0.99)  Available space 32–42  0.91 (0.87 / 0.94)  0.97 (0.95 / 0.98)  0.39  0.22  0.30 (0.41)  <0.001  −0.5 (−0.7 / −0.3)  1.1 (0.9 / 1.29)  1.6  0.29 / 0.048  0.9 (0.84 / 0.94)  Available space 43–45  0.98 (0.96 / 0.99)  0.98 (0.97 / 0.99)  0.19  0.26  0.01 (0.28)  0.836  −0.54 (−0.67 / −0.4)  0.55 (0.42 / 0.69)  1.09  0.25 / 0.082  0.98 (0.97 / 0.99)  Interpremolar width maxilla  0.99 (0.98 / 0.99)  0.99 (0.99 / 1)  0.24  0.22  0.27 (0.27)  <0.001  −0.26 (−0.39 / −0.12)  0.8 (0.67 / 0.93)  1.06  0.06 / 0.699  0.99 (0.98 / 0.99)  Interpremolar width mandible  0.96 (0.94 / 0.98)  0.99 (0.99 / 1)  0.52  0.18  0.08 (0.26)  0.044  −0.44 (−0.57 / −0.31)  0.6 (0.47 / 0.73)  1.04  0.07 / 0.635  0.99 (0.98 / 0.99)  Intermolar width maxilla  0.96 (0.93 / 0.97)  0.98 (0.97 / 0.99)  0.56  0.47  0.71 (0.39)  <0.001  −0.05 (−0.24 / 0.14)  1.46 (1.28 / 1.65)  1.51  0.1 / 0.497  0.95 (0.93 / 0.97)  Intermolar width mandible  0.95 (0.92 / 0.97)  0.97 (0.95 / 0.98)  0.34  0.63  0.63 (0.38)  <0.001  −0.12 (−0.3 / 0.06)  1.37 (1.19 / 1.55)  1.49  0.01 / 0.944  0.95 (0.91 / 0.97)  Computed measurements  Sum 12–22  0.92 (0.89 / 0.95)  0.92 (0.85 / 0.96)  0.52  0.6  −0.48 (1.12)  0.005  −2.67 (−3.21 / −2.13)  1.71 (1.17 / 2.25)  4.38  0.11 / 0.451  0.74 (0.59 / 0.84)  Sum 32–42  0.82 (0.74 / 0.88)  0.93 (0.89 / 0.96)  0.46  0.4  −0.19 (0.61)  0.036  −1.39 (−1.68 / −1.09)  1.01 (0.71 / 1.3)  2.4  0.21 / 0.143  0.84 (0.73 / 0.9)  Total difference maxilla  0.82 (0.73 / 0.89)  0.86 (0.74 / 0.92)  1.15  1.05  1.45 (2.4)  <0.001  −3.25 (−4.41 / −2.09)  6.15 (4.99 / 7.31)  9.4  0.18 / 0.221  0.56 (0.37 / 0.71)  Total difference mandible  0.83 (0.73 / 0.89)  0.91 (0.84 / 0.95)  1.08  0.94  1.13 (1.76)  <0.001  −2.31 (−3.16 / −1.46)  4.57 (3.72 / 5.42)  6.88  0.1 / 0.501  0.72 (0.57 / 0.83)  Bolton 3-3 (n = 24)  0.66 (0.49 / 0.81)  0.81 (0.7 / 0.9)  0.01  0.01  0.01 (0.02)  0.004  −0.02 (−0.04 / −0.01)  0.05 (0.04 / 0.06)  0.07  0.01 / 0.981  0.56 (0.27 / 0.76)  Bolton 5-5 (n = 24)  0.59 (0.41 / 0.76)  0.81 (0.69 / 0.9)  0.01  0.01  0.01 (0.02)  0.177  −0.03 (−0.04 / −0.02)  0.04 (0.03 / 0.05)  0.07  0.02 / 0.913  0.55 (0.23 / 0.76)  Tonn  0.62 (0.5 / 0.74)  0.83 (0.75 / 0.89)  0.61  0.55  −0.27 (0.91)  0.041  −2.05 (−2.49 / −1.61)  1.5 (1.06 / 1.94)  3.55  0.05 / 0.762  0.62 (0.43 / 0.76)  Transversal premolar relationship  0.93 (0.89 / 0.95)  0.98 (0.96 / 0.98)  0.55  0.26  0.19 (0.37)  0.001  −0.52 (−0.7 / −0.35)  0.91 (0.73 / 1.09)  1.43  0.34 / 0.017  0.96 (0.94 / 0.98)  Transversal molar relationship  0.87 (0.78 / 0.93)  0.92 (0.88 / 0.95)  0.66  0.67  0.02 (0.57)  0.821  −1.1 (−1.37 / −0.82)  1.14 (0.86 / 1.41)  2.24  0.02 / 0.891  0.94 (0.9 / 0.97)  Apart from columns 8 and 12, bold values refer to correlation coefficients smaller than 0.7 or differences greater than 0.5 mm. ICC, intraclass correlation coefficient; CI, confidence intervals; Δ, manual (median of 5 repeated measurements) − digital (median of 5 repeated measurements); SD, standard deviation; LOA, limits of agreement; CCC, Lin’s concordance correlation coefficient. *values for single measurements with a mixed model and absolute agreement. **calculated only on the first two repeated measurements. View Large Intra-observer reliability proved to be substantial in both manual and digital methods, except for five outcomes (12.8 per cent) in the manual method (0.59 ≤ ICC ≤ 0.68). Three of these outcomes were tooth size discrepancy indices (0.59 ≤ ICC ≤ 0.66). Reliability was higher in the digital (ICC ≥ 0.7 for all the outcomes) than the manual method. Dahlberg’s error was comparable between the methods and approximately 1 mm for the outcomes total difference in maxilla and mandible. In general, systematic bias was close to zero (−0.5 mm ≤ Δ ≤ 0.5 mm) for most measurements (87.2 per cent), therefore manual and digital methods agree to the nearest 0.5 mm on average, although P values for one sample t-tests on the difference of Δ from zero were significant for most outcomes. Outcomes that showed distinct bias were total space difference of maxilla and mandible (Δ = 1.45 mm and Δ = 1.13 mm, respectively) and to a lesser degree intermolar width of maxilla and mandible (Δ = 0.71 mm and Δ = 0.63 mm, respectively). Furthermore, tooth sizes were larger when measured digitally, since the sign of Δ was negative for 80 per cent of the teeth measured, whereas 90 per cent of the directly measured distances had a positive sign. Limits of agreement (LOA) on Δ were relatively wide for the computed outcomes sum 12–22 (−2.67 mm ≤ LOA ≤ 1.71 mm), sum 32–42 (−1.39 mm ≤ LOA ≤ 1.01 mm), total difference maxilla (−3.25 mm ≤ LOA ≤ 6.15 mm), total difference mandible (−2.31 mm ≤ LOA ≤ 4.57 mm), and Tonn (−2.05 mm ≤ LOA ≤ 1.5 mm). However, the range between the lower and upper LOA (Table 3) was in general lower than the range of repeated intra-observer measurement variation for both manual and digital methods (Supplementary Table 2). Thus, the differences detected between both methods did not exceed intra-method error/variation. Manual and digital methods seem to conform well with each other in general (0.47 ≤ CCC ≤ 0.99), since the majority of outcomes (66.7 per cent) had a concordance correlation coefficient of more than 0.7 (substantial conformity). Premolars and tooth size discrepancy indices (computed outcomes) seem to produce results, which do not agree between methods (0.47 ≤ CCC ≤ 0.69 and 0.55 ≤ CCC ≤ 0.62, respectively). Finally, no signs of changing bias were present, as the P values from the regression slopes were sporadically significant. Therefore, bias between methods changes at random and not according to the true values as measured with the manual method. The corresponding plots are provided in Supplementary Figure 3. Discussion In this study, we investigated the intra-observer reliability and agreement between traditional manual model analysis, based on plaster casts and a dial caliper, and digital model analysis, based on virtual models acquired with orthoX®scan and analysed with ivoris®analyze 3D. We found that intra-observer reliability within each method was generally substantial, whereas agreement/conformity between the two methods was relatively limited, especially for computed outcomes, which are of particular diagnostic interest to orthodontists. However, digital model analysis proved to be more reliable with larger tooth sizes, but smaller distances measured on average. Among the factors influencing measurement error is observer’s experience (12). In our study, the observer was experienced and familiar with the software used and completed all the measurements within 2 months. Experienced observers generally produce more consistent readings in repeated measurements (12). However, this is not always the case and repeated measurements can vary even though they were performed by the same observer, especially when the interval between repeated measurements increases (13). In any case, intra-observer reliability is generally higher than inter-observer reliability and that is attributed to random error (14). Due to this fact, in measurement error studies, the number of observers should be ideally considered as the sample size of the study, in order to draw inferences for the population of observers in general (10). Point identification and positioning is perhaps the greatest source of measurement error (14, 15). Thus, reliability of point identification most likely directly relates to the reliability of the model analysis itself. Whereas point identification correlates with point definition, shape of the anatomical structure measured, and observer’s experience/judgment (16), point positioning mostly depends on properties of the measuring instrument and measured item. In the present study, we adopted common definitions of anatomical points. However, some imprecision in point identification is always present, which is inherent in model analysis itself irrespective of the method used, i.e. a contact point may actually be a contact area, causing variation in point identification. The shape of the anatomical structure being measured also plays a role on point identification, since it was found that points located at the edges of anatomical structures are more precisely identifiable than points located at curved anatomical structures (16). This could be observed in our results as well, since mesiodistal sizes of premolars presented in general lower conformity between the manual and digital methods than mesiodistal sizes of incisors. Another aspect possibly contributing to these results is the limited sample size for outcomes only measurable on casts of permanent dentitions, such as premolar width (n = 24). Finally, with respect to the point positioning, calipers cannot completely access the maximum mesiodistal diameter of teeth, especially if crowding is present, and impression materials do not exactly imprint the space between crowded teeth in plaster casts (17). On the contrary, software solutions for digital model analysis provide a variety of functions, such as zooming or view rotation, which facilitate point positioning particularly at proximal contacts, even if teeth are crowded. Our results indicated that mesiodistal tooth sizes were measured larger digitally than manually, which is in agreement with other findings (18, 19). Furthermore, variation in point identification and positioning is greater for outcomes consisting of more than one point (16). This leads to cumulative error in outcomes that are computed from more than one tooth, such as the sum of upper or lower incisors widths. This is also the case with measurements derived by mathematical operations, as in the case of tooth size discrepancy indices (Bolton/Tonn). Moreover, in our study, the total width of the posterior teeth was calculated according to Moyers (20) at 75 per cent probability level and not directly measured in the early mixed dentitions. These factors could explain our results regarding imprecision in tooth size discrepancy indices within the manual method as well as the wide LOA ranges of computed outcomes between methods. Despite their higher imprecision, computed outcomes are of particular diagnostic interest to orthodontists. Lower crowding, lower lip to E-plane, upper crowding, and overjet are important diagnostic variables in Class I extraction treatments (21), whereas lower anterior crowding, molar relationship, and growth pattern are the most influential factors in Class II extraction treatments (22). Therefore, a precise and accurate space analysis is needed for meaningful treatment decisions (23). Although bias between methods was within 1–1.5 mm for space measurements, LOA indicated that 95 per cent of the differences between methods could vary from zero across a wide range, which could eventually influence clinical decisions. With respect to the averaged time for manual and digital model analyses per model, our data suggested that differences were negligible (manual mean 7 minutes and 59 seconds and SD 1 minutes and 36 seconds; digital mean 8 minutes and 36 seconds and SD 2 minutes and 10 seconds). However, one should also consider the scanning process, which could be long for dental casts with unstable occlusions, such as open bites. The average duration of the scanning process was 6 minutes and 43 seconds (SD 13 seconds), but for open bites approximately 2 more minutes were needed, since fixating open bites onto the model holder was more time consuming than stable occluded dentitions. Finally, some models had to be scanned twice resulting in an averaged duration of 11 minutes and 53 seconds (SD 28 seconds). Conclusions The following conclusions can be drawn from the present study regarding orthodontic model analysis with plaster casts and a dial caliper and virtual models obtained with orthoX®scan and analysed with ivoris®analyze 3D: - Both methods were reliable but the digital model analysis was more reliable than the manual one with less variability encountered in repeated measurements. - The digital method yielded on average larger tooth sizes but smaller other distances than the manual method. - Methods agree substantially up to 0.5 mm with each other for most direct measurements in model analysis. Caution is warranted for some computed space parameters, since bias is expected to have a wide range between methods due to cumulative error. Supplementary material Supplementary material is available at European Journal of Orthodontics online. Conflicts of interest None to declare. References 1. Fleming P.S. Marinho V. and Johal A. ( 2011) Orthodontic measurements on digital study models compared with plaster models: a systematic review. Orthodontics & Craniofacial Research , 14, 1– 16. Google Scholar CrossRef Search ADS PubMed  2. Luu N.S. Nikolcheva L.G. Retrouvey J.-M. Flores-Mir C. El-Bialy T. Carey J.P. and Major P.W. ( 2012) Linear measurements using virtual study models. Angle Orthodontist , 82, 1098– 1106. Google Scholar CrossRef Search ADS PubMed  3. Rossini G. Parrini S. Castroflorio T. Deregibus A. and Debernardi C.L. ( 2016) Diagnostic accuracy and measurement sensitivity of digital models for orthodontic purposes: A systematic review. American Journal of Orthodontics and Dentofacial Orthopedics , 149, 161– 170. Google Scholar CrossRef Search ADS PubMed  4. Kottner J. Audigé L. Brorson S. Donner A. Gajewski B.J. Hróbjartsson A. Roberts C. Shoukri M. and Streiner D.L. ( 2011) Guidelines for Reporting Reliability and Agreement Studies (GRRAS) were proposed. Journal of Clinical Epidemiology , 64, 96– 106. Google Scholar CrossRef Search ADS PubMed  5. Lomax R.G. and Hahs-Vaughn D.L. ( 2012) An Introduction to Statistical Concepts . Routledge, New York. 6. Bland J.M. and Altman D. ( 1986) Statistical methods for assessing agreement between two methods of clinical measurement. Lancet , 327, 307– 310. Google Scholar CrossRef Search ADS   7. Lin L.I. ( 1989) A concordance correlation coefficient to evaluate reproducibility. Biometrics , 45, 255– 268. Google Scholar CrossRef Search ADS PubMed  8. Asquith J. Gillgrass T. and Mossey P. ( 2007) Three-dimensional imaging of orthodontic models: a pilot study. European Journal of Orthodontics , 29, 517– 522. Google Scholar CrossRef Search ADS PubMed  9. Luu N.S. Mandich M.-A. Flores-Mir C. El-Bialy T. Heo G. Carey J.P. and Major P.W. ( 2014) The validity, reliability, and time requirement of study model analysis using cone-beam computed tomography-generated virtual study models. Orthodontics & Craniofacial Research , 17, 14– 26. Google Scholar CrossRef Search ADS PubMed  10. Bartlett J.W. and Frost C. ( 2008) Reliability, repeatability and reproducibility: analysis of measurement errors in continuous variables. Ultrasound in Obstetrics & Gynecology , 31, 466– 475. Google Scholar CrossRef Search ADS   11. Faul F. Erdfelder E. Buchner A. and Lang A.G. ( 2009) Statistical power analyses using G*Power 3.1: tests for correlation and regression analyses. Behavior Research Methods , 41, 1149– 1160. Google Scholar CrossRef Search ADS PubMed  12. Gesch D. Schröder W. Kocher T. and Kirbschus A. ( 2006) Examiner differences in the assessment of different malocclusions. Journal of Orofacial Orthopedics , 67, 404– 413. Google Scholar CrossRef Search ADS PubMed  13. Kieser J.A. Groeneveld H.T. McKee J. and Cameron N. ( 1990) Measurement error in human dental mensuration. Annals of Human Biology , 17, 523– 528. Google Scholar CrossRef Search ADS PubMed  14. Kamoen A. Dermaut L. and Verbeeck R. ( 2001) The clinical significance of error measurement in the interpretation of treatment results. European Journal of Orthodontics , 23, 569– 578. Google Scholar CrossRef Search ADS PubMed  15. Houston W.J. Maher R.E. McElroy D. and Sherriff M. ( 1986) Sources of error in measurements from cephalometric radiographs. European Journal of Orthodontics , 8, 149– 151. Google Scholar CrossRef Search ADS PubMed  16. Baumrind S. and Frantz R.C. ( 1971) The reliability of head film measurements. American Journal of Orthodontics , 60, 111– 127. Google Scholar CrossRef Search ADS PubMed  17. Harris E.F. and Smith R.N. ( 2009) Accounting for measurement error: a critical but often overlooked process. Archives of Oral Biology , 54 ( suppl 1), S107– S117. Google Scholar CrossRef Search ADS PubMed  18. Abizadeh N. Moles D.R. O’Neill J. and Noar J.H. ( 2012) Digital versus plaster study models: how accurate and reproducible are they? Journal of Orthodontics , 39, 151– 159. Google Scholar CrossRef Search ADS PubMed  19. Czarnota J. Hey J. and Fuhrmann R. ( 2016) Measurements using orthodontic analysis software on digital models obtained by 3D scans of plaster casts. Journal of Orofacial Orthopedics , 77, 22– 30. Google Scholar CrossRef Search ADS PubMed  20. Moyers R. ( 1988) Handbook of Orthodontics . Year Book Medical Publishers, Chicago, IL, 4th edn. 21. Konstantonis D. Anthopoulou C. and Makou M. ( 2013) Extraction decision and identification of treatment predictors in Class I malocclusions. Progress in Orthodontics , 14, 47. Google Scholar CrossRef Search ADS PubMed  22. Guo Y. Han X. Xu H. Ai D. Zeng H. and Bai D. ( 2014) Morphological characteristics influencing the orthodontic extraction strategies for Angle’s class II division 1 malocclusions. Progress in Orthodontics , 15, 44. Google Scholar CrossRef Search ADS PubMed  23. Naish H. Dunbar C. Crouch-Baker J. Shah K. Wallis C. Atack N.E. Sherriff M. Sandy J.R. and Ireland A.J. ( 2016) Does a true knowledge of dental crowding affect orthodontic treatment decisions? European Journal of Orthodontics , 38, 66– 70. Google Scholar CrossRef Search ADS PubMed  © The Author(s) 2017. Published by Oxford University Press on behalf of the European Orthodontic Society. All rights reserved. For permissions, please email: journals.permissions@oup.com http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png The European Journal of Orthodontics Oxford University Press

Intra-observer reliability and agreement of manual and digital orthodontic model analysis

Loading next page...
 
/lp/ou_press/intra-observer-reliability-and-agreement-of-manual-and-digital-0Rypg0u2ws
Publisher
Oxford University Press
Copyright
© The Author(s) 2017. Published by Oxford University Press on behalf of the European Orthodontic Society. All rights reserved. For permissions, please email: journals.permissions@oup.com
ISSN
0141-5387
eISSN
1460-2210
D.O.I.
10.1093/ejo/cjx040
Publisher site
See Article on Publisher Site

Abstract

Summary Background and aim Digital orthodontic model analysis is gaining acceptance in orthodontics, but its reliability is dependent on the digitalisation hardware and software used. We thus investigated intra-observer reliability and agreement / conformity of a particular digital model analysis work-flow in relation to traditional manual plaster model analysis. Materials and methods Forty-eight plaster casts of the upper/lower dentition were collected. Virtual models were obtained with orthoX®scan (Dentaurum) and analysed with ivoris®analyze3D (Computer konkret). Manual model analyses were done with a dial caliper (0.1 mm). Common parameters were measured on each plaster cast and its virtual counterpart five times each by an experienced observer. We assessed intra-observer reliability within method (ICC), agreement/conformity between methods (Bland–Altman analyses and Lin’s concordance correlation), and changing bias (regression analyses). Results Intra-observer reliability was substantial within each method (ICC ≥ 0.7), except for five manual outcomes (12.8 per cent). Bias between methods was statistically significant, but less than 0.5 mm for 87.2 per cent of the outcomes. In general, larger tooth sizes were measured digitally. Total difference maxilla and mandible had wide limits of agreement (−3.25/6.15 and −2.31/4.57 mm), but bias between methods was mostly smaller than intra-observer variation within each method with substantial conformity of manual and digital measurements in general. No changing bias was detected. Conclusions Although both work-flows were reliable, the investigated digital work-flow proved to be more reliable and yielded on average larger tooth sizes. Averaged differences between methods were within 0.5 mm for directly measured outcomes but wide ranges are expected for some computed space parameters due to cumulative error. Introduction Three-dimensional model analyses, based on plaster casts of the upper and lower dentition of a patient, are an integral part of orthodontic diagnostics. Due to limitations associated with conventional plaster dental casts, such as increased need of storage space and possible damage due to handling, digital scanners for obtaining virtual models and a number of software programs for digital model analyses have been developed and propagated in orthodontics. However, as a relatively new diagnostic procedure, its validity compared to the traditional model analysis with calipers remains to be assessed, especially, since a variety of hardware and software solutions are available to the user. Systematic reviews comparing various manual and digital work-flows generally agree that the digital work-flow itself can be trusted for clinical purposes (1–3). Due to increasing clinical acceptance of virtual models and a digital work-flow for model analysis, many new hardware and software solutions are commercially available. The orthoX®scan (DENTAURUM, Ispringen, Germany) is an optical 3D scanner, which allows the merging of pictures acquired with a camera recording a stripe-light projection into a 3D virtual model. The manufacturer states a scanning accuracy of less than 20 µm (http://www.dentaurum.de/eng/orthox-scan-24165.aspx, accessed on 21 December 2016). Exported data can be saved in open STL data format, which provides flexibility, since the scanner can be incorporated in existing software solutions. The software ivoris®analyze 3D (Computer konkret, Falkenstein, Germany) can import STL data and a variety of digital model analyses are available to the user. Until now, no study has been conducted to evaluate the performance of orthoX®scan and ivoris®analyze 3D. Therefore, we aimed to comparatively assess the traditional manual model analysis on plaster casts and digital model analysis with ivoris®analyze 3D, for which virtual models were acquired with orthoX®scan. Materials and methods This paper was reported according to Guidelines for Reporting Reliability and Agreement Studies (4). Dental plaster casts of orthodontic patients meeting the criteria listed in Table 1 were consecutively collected from the archives of the Department of Orthodontics, University Hospital Regensburg, Germany until 48 plaster casts were available for analysis. Characteristics of the sample are provided in Supplementary Table 1. Plaster casts were scanned once with orthoX®scan according to the manufacturer’s instructions. The three-dimensional virtual models were visually controlled for their precision and in case of inadequacies, scanning was repeated until high quality virtual models were acquired. After scanning, virtual models were imported into ivoris®analyze 3D for digital model analyses. The same plaster casts were then manually analysed. Table 1. Criteria for collecting dental casts. Permanent dentition (full erupted posterior segment apart from second molars) or early mixed dentition (full erupted incisors and first molars)  Good condition of dental casts (not fractured or degraded)  No missing teeth up to first molars (permanent dentitions) or first molars and incisors (early mixed dentitions)  No more than one dental cast for each patient  Irrespective of crowding, malocclusion, and previous orthodontic treatment  Permanent dentition (full erupted posterior segment apart from second molars) or early mixed dentition (full erupted incisors and first molars)  Good condition of dental casts (not fractured or degraded)  No missing teeth up to first molars (permanent dentitions) or first molars and incisors (early mixed dentitions)  No more than one dental cast for each patient  Irrespective of crowding, malocclusion, and previous orthodontic treatment  View Large We performed five manual and five digital repeated measurements for each outcome and case investigated. Manual model analyses were done with a conventional (identical for all the measurements) dial caliper at a precision of 0.1 mm and digital model analyses with ivoris®analyze 3D by one experienced investigator, familiarised with the software, in a random order regarding the patients’ casts and method of measurement (manual/digital) per round of repeated measurements. After a round of repeated measurements was completed, the next one was initiated until each outcome was measured five times on each plaster cast and its virtual counterpart. The number of repeated measurements was chosen in accordance with previous studies to allow a reliable assessment of central tendency. The observer was not blinded to patients’ personal information, such as name and date of birth, but was blinded to patients’ initial diagnosis, treatment plan, and previous repeated measurements with either method. As it usually happens in practices or clinics, the measuring process was often discontinued. Since we had not set a desired number of model analyses to be completed per day, measurements were performed with neither urge nor delay. Thus, measuring conditions approximated those in orthodontic clinical routine. All the measurements were completed within 2 months. Manual data were directly recorded into a pre-validated Excel spreadsheet, whereas digital data were first exported into a separate spreadsheet and then copied into the pre-validated spreadsheet. For both manual and digital methods, computations were automatically performed with Excel functions in the same way. Measurement points and the corresponding computations represent commonly used parameters in clinical practice and are listed in Table 2. Duration of the scanning process and model analysis for two repeated measurements per method was also recorded. Table 2. Definitions of measurements and corresponding computations performed for model analyses. Measurements*   Tooth sizes (15–25, 35–45)  Mesiodistal width of the respective tooth (only the anterior teeth in case of the early mixed dentition)   Available space at the posterior segment (15–13, 23–25, 35–33, 43–45)  Respective distance between the distal surfaces of the lateral incisors and the mesial surfaces of the first molars in the four quadrants of the jaw   Available space at the anterior segment (12–22, 32–42)  Sum of the distances between the distal surface of the lateral incisor and the mesial surface of the central incisor in the right and left section of the respective dental arch.   Interpremolar and intermolar widths maxilla  Distance between the centre of the fissures of the first maxillary premolars (or first deciduous molars) and distance between the deepest points of the main fissures of the maxillary first molars   Interpremolar and intermolar widths mandible  Distance between the distal contact points of the first mandibular premolars (or first deciduous molars) and distance between the distobuccal cusps of the mandibular first molars   Duration  Time needed for the aforementioned measurements to be completed (time needed for computations not included)  Computations**   Sum 12–22 (SI)  Sum of mesiodistal widths of maxillary incisors   Sum 32–42 (si)  Sum of mesiodistal widths of mandibular incisors   Total difference maxilla  Difference between the total available space in the upper dental arch (sum of available space at the anterior and posterior segments) and the sum of the mesiodistal widths of teeth 15–25 (Moyers analysis at 75% probability) in case of the early mixed dentitions (20)   Total difference mandible  Difference between the total available space in the lower dental arch (sum of available space at the anterior and posterior segments) and the sum of the mesiodistal widths of teeth 35–45 (Moyers analysis at 75% probability) in case of the early mixed dentitions (20)   Bolton 3-3  Ratio of the sum of mesiodistal widths of teeth 33–43 and 13–33 (not applicable to early mixed dentitions)   Bolton 5-5  Ratio of the sum of mesiodistal widths of teeth 35–45 and 15–25 (not applicable to early mixed dentitions)   Tonn  SI = 4/3 si + k (k = 0.4 if si < 22.2; k = 0.5 if 22.3 < si < 28.1; k = 0.6 if si > 28.1)   Transversal premolar relationship  Difference between maxillary and mandibular interpremolar widths   Transversal molar relationship  Difference between maxillary and mandibular intermolar widths  Measurements*   Tooth sizes (15–25, 35–45)  Mesiodistal width of the respective tooth (only the anterior teeth in case of the early mixed dentition)   Available space at the posterior segment (15–13, 23–25, 35–33, 43–45)  Respective distance between the distal surfaces of the lateral incisors and the mesial surfaces of the first molars in the four quadrants of the jaw   Available space at the anterior segment (12–22, 32–42)  Sum of the distances between the distal surface of the lateral incisor and the mesial surface of the central incisor in the right and left section of the respective dental arch.   Interpremolar and intermolar widths maxilla  Distance between the centre of the fissures of the first maxillary premolars (or first deciduous molars) and distance between the deepest points of the main fissures of the maxillary first molars   Interpremolar and intermolar widths mandible  Distance between the distal contact points of the first mandibular premolars (or first deciduous molars) and distance between the distobuccal cusps of the mandibular first molars   Duration  Time needed for the aforementioned measurements to be completed (time needed for computations not included)  Computations**   Sum 12–22 (SI)  Sum of mesiodistal widths of maxillary incisors   Sum 32–42 (si)  Sum of mesiodistal widths of mandibular incisors   Total difference maxilla  Difference between the total available space in the upper dental arch (sum of available space at the anterior and posterior segments) and the sum of the mesiodistal widths of teeth 15–25 (Moyers analysis at 75% probability) in case of the early mixed dentitions (20)   Total difference mandible  Difference between the total available space in the lower dental arch (sum of available space at the anterior and posterior segments) and the sum of the mesiodistal widths of teeth 35–45 (Moyers analysis at 75% probability) in case of the early mixed dentitions (20)   Bolton 3-3  Ratio of the sum of mesiodistal widths of teeth 33–43 and 13–33 (not applicable to early mixed dentitions)   Bolton 5-5  Ratio of the sum of mesiodistal widths of teeth 35–45 and 15–25 (not applicable to early mixed dentitions)   Tonn  SI = 4/3 si + k (k = 0.4 if si < 22.2; k = 0.5 if 22.3 < si < 28.1; k = 0.6 if si > 28.1)   Transversal premolar relationship  Difference between maxillary and mandibular interpremolar widths   Transversal molar relationship  Difference between maxillary and mandibular intermolar widths  *Although it was originally planned, overjet and overbite could not be digitally measured with ivoris®analyze 3D due to limitations of the analysis software. **Computed from the direct measurements on casts. View Large Medians of the five repeated measurements were calculated for each variable of interest per method (manual/digital) and considered for statistical analyses, unless otherwise stated. To represent the central tendency of our data, median was chosen over mean because it is less likely to be influenced from the extremes of a dataset (5), thus better representing the middle measured value(s) that are more likely to correspond to the true values. The intra-observer reliability was estimated with intraclass correlation coefficients (ICC) on all five repeated measurements per variable and method. We additionally used the Dahlberg’s formula to evaluate method error within methods. To assess agreement between methods, Bland–Altman analyses were performed with corresponding limits of agreement and 95 per cent confidence intervals (6). Conformity was assessed with Lin’s concordance correlation coefficient (CCC), which is robust for more than 10 pairs of data (7). For the ICC and CCC correlation coefficients, agreement was considered substantial, if values exceeded 0.7. Differences in means/systematic bias (bias, Δ) were considered clinically acceptable, if values were smaller than 0.5 mm (8, 9). Changing bias relative to the true values was investigated by plotting the data and running regression analyses (10). Normality of the data was confirmed with Shapiro–Wilk tests and a visual inspection of histograms. Power calculation was performed post hoc in G*Power 3.1 (11) by taking into account a bivariate normal model and an effect size of 0.6, which is well below effect sizes reported in literature and the cut-off value adopted in this study. Statistical analyses were performed with Microsoft Excel® 2010 (Microsoft Corp., Redmond, USA), IBM SPSS® Statistics software version 23.0 (IBM Corp., Armonk, NY), and the online NIWA calculator (https://www.niwa.co.nz/node/104318/concordance, accessed on 21 December 2016). An overview of the study design as well as statistical procedures is provided in Supplementary Figure 1. Results According to the post hoc power analyses with α (two-tailed) = 0.05, effect size = 0.6, and n = 48 or n = 24, achieved power (1 − β) of the study was 0.99 or 0.90, respectively. Results for intra-observer reliability and agreement/conformity are provided in Table 3. Descriptive statistics are additionally provided in Supplementary Table 2, and Bland–Altman plots in Supplementary Figure 2. Table 3. Results on intra-observer reliability and agreement.     Intra-examiner reliability (within method)  Intra-examiner agreement (between methods)  Outcomes  Manual ICC (95% CI)*  Digital ICC (95% CI)*  Manual Dahlberg’s error** mm  Digital Dahlberg’s error** mm  Mean of Δ (SD) mm  Δ different from 0 P value  Lower LOA (95% CI) mm  Upper LOA (95% CI) mm  Range of lower and upper LOA mm  Changing bias regression r / P value  Conformity CCC (95% CI)  2-landmark measurements on casts  Tooth size 15 (n = 24)  0.73 (0.56 / 0.86)  0.78 (0.65 / 0.88)  0.33  0.26  −0.28 (0.27)  <0.001  −0.8 (−0.98 / −0.61)  0.25 (0.06 / 0.43)  1.05  0.03 / 0.892  0.64 (0.4 / 0.8)  Tooth size 14 (n = 24)  0.72 (0.56 / 0.84)  0.82 (0.7 / 0.9)  0.3  0.22  −0.35 (0.28)  <0.001  −0.89 (−1.08 / −0.7)  0.19 (0 / 0.38)  1.08  0.07 / 0.739  0.61 (0.38 / 0.77)  Tooth size 13 (n = 24)  0.82 (0.7 / 0.9)  0.77 (0.64 / 0.88)  0.2  0.25  0.03 (0.25)  0.577  −0.47 (−0.64 / −0.29)  0.52 (0.35 / 0.7)  0.99  0.22 / 0.305  0.79 (0.58 / 0.9)  Tooth size 12  0.86 (0.8 / 0.91)  0.88 (0.81 / 0.92)  0.21  0.22  −0.13 (0.4)  0.026  −0.91 (−1.1 / −0.72)  0.65 (0.45 / 0.84)  1.56  0.13 / 0.374  0.66 (0.47 / 0.79)  Tooth size 11  0.9 (0.85 / 0.94)  0.92 (0.88 / 0.95)  0.21  0.19  −0.03 (0.31)  0.549  −0.64 (−0.79 / −0.49)  0.58 (0.43 / 0.73)  1.22  0.21 / 0.16  0.82 (0.7 / 0.9)  Tooth size 21  0.92 (0.87 / 0.95)  0.88 (0.8 / 0.92)  0.17  0.25  −0.2 (0.34)  <0.001  −0.87 (−1.04 / −0.71)  0.47 (0.3 / 0.63)  1.34  0.04 / 0.81  0.75 (0.6 / 0.84)  Tooth size 22  0.88 (0.82 / 0.92)  0.84 (0.76 / 0.9)  0.17  0.25  −0.17 (0.4)  0.006  −0.96 (−1.16 / −0.77)  0.62 (0.43 / 0.82)  1.58  0.11 / 0.446  0.61 (0.43 / 0.75)  Tooth size 23 (n = 24)  0.87 (0.78 / 0.93)  0.7 (0.55 / 0.84)  0.17  0.3  −0.06 (0.31)  0.371  −0.67 (−0.89 / −0.46)  0.56 (0.34 / 0.77)  1.23  0.46 / 0.023  0.72 (0.46 / 0.87)  Tooth size 24 (n = 24)  0.81 (0.69 / 0.9)  0.81 (0.69 / 0.9)  0.21  0.3  −0.29 (0.29)  <0.001  −0.86 (−1.06 / −0.66)  0.29 (0.09 / 0.49)  1.15  0.02 / 0.915  0.64 (0.4 / 0.8)  Tooth size 25 (n = 24)  0.76 (0.63 / 0.87)  0.72 (0.57 / 0.85)  0.27  0.24  −0.34 (0.19)  <0.001  −0.72 (−0.85 / −0.58)  0.04 (−0.09 / 0.17)  0.76  0.27 / 0.209  0.65 (0.45 / 0.79)  Tooth size 35 (n = 24)  0.67 (0.51 / 0.82)  0.76 (0.59 / 0.88)  0.33  0.2  −0.22 (0.27)  0.001  −0.74 (−0.93 / −0.56)  0.31 (0.13 / 0.49)  1.05  0.16 / 0.47  0.72 (0.49 / 0.85)  Tooth size 34 (n = 24)  0.77 (0.63 / 0.88)  0.83 (0.72 / 0.91)  0.27  0.23  −0.52 (0.26)  <0.001  −1.04 (−1.22 / −0.85)  0 (−0.18 / 0.18)  1.04  0.01 / 0.975  0.47 (0.26 / 0.63)  Tooth size 33 (n = 24)  0.76 (0.63 / 0.87)  0.88 (0.79 / 0.94)  0.24  0.19  0.06 (0.24)  0.238  −0.4 (−0.57 / −0.24)  0.52 (0.36 / 0.68)  0.92  0.05 / 0.807  0.81 (0.63 / 0.91)  Tooth size 32  0.79 (0.7 / 0.86)  0.84 (0.78 / 0.9)  0.17  0.17  −0.06 (0.24)  0.104  −0.54 (−0.65 / −0.42)  0.42 (0.3 / 0.54)  0.96  0.23 / 0.118  0.7 (0.53 / 0.82)  Tooth size 31  0.79 (0.69 / 0.86)  0.86 (0.8 / 0.91)  0.15  0.14  −0.07 (0.19)  0.015  −0.44 (−0.53 / −0.35)  0.3 (0.21 / 0.39)  0.74  0.04 / 0.807  0.8 (0.67 / 0.88)  Tooth size 41  0.68 (0.56 / 0.78)  0.79 (0.71 / 0.86)  0.18  0.19  −0.01 (0.19)  0.82  −0.38 (−0.47 / −0.29)  0.37 (0.27 / 0.46)  0.75  0.09 / 0.546  0.77 (0.62 / 0.86)  Tooth size 42  0.75 (0.65 / 0.83)  0.86 (0.8 / 0.91)  0.18  0.19  0 (0.23)  0.951  −0.46 (−0.57 / −0.34)  0.45 (0.34 / 0.56)  0.91  0.12 / 0.409  0.79 (0.66 / 0.88)  Tooth size 43 (n = 24)  0.87 (0.79 / 0.94)  0.82 (0.71 / 0.91)  0.13  0.19  0.06 (0.16)  0.09  −0.26 (−0.37 / −0.15)  0.37 (0.26 / 0.48)  0.63  0.1 / 0.654  0.91 (0.81 / 0.96)  Tooth size 44 (n = 24)  0.77 (0.61 / 0.88)  0.8 (0.68 / 0.89)  0.22  0.23  −0.32 (0.21)  <0.001  −0.74 (−0.88 / −0.59)  0.1 (−0.05 / 0.24)  0.84  0.03 / 0.882  0.69 (0.5 / 0.82)  Tooth size 45 (n = 24)  0.85 (0.73 / 0.92)  0.81 (0.7 / 0.9)  0.21  0.24  −0.3 (0.29)  <0.001  −0.86 (−1.05 / −0.66)  0.27 (0.07 / 0.46)  1.13  0.13 / 0.546  0.67 (0.44 / 0.82)  Available space 15–13  0.95 (0.91 / 0.97)  0.88 (0.83 / 0.92)  0.21  0.32  0.11 (0.43)  0.089  −0.74 (−0.95 / −0.53)  0.95 (0.75 / 1.16)  1.69  0.1 / 0.487  0.95 (0.92 / 0.97)  Available space 12–22  0.89 (0.84 / 0.93)  0.96 (0.95 / 0.98)  0.51  0.33  0.19 (0.43)  0.004  −0.66 (−0.87 / −0.45)  1.03 (0.83 / 1.24)  1.69  0.15 / 0.311  0.96 (0.92 / 0.97)  Available space 23–25  0.95 (0.93 / 0.97)  0.96 (0.94 / 0.98)  0.22  0.26  −0.02 (0.38)  0.762  −0.76 (−0.94 / −0.58)  0.73 (0.54 / 0.91)  1.49  0.04 / 0.805  0.96 (0.93 / 0.98)  Available space 35–33  0.98 (0.97 / 0.99)  0.98 (0.97 / 0.99)  0.17  0.2  0.06 (0.24)  0.085  −0.41 (−0.52 / −0.29)  0.53 (0.41 / 0.64)  0.94  0.42 / 0.003  0.99 (0.98 / 0.99)  Available space 32–42  0.91 (0.87 / 0.94)  0.97 (0.95 / 0.98)  0.39  0.22  0.30 (0.41)  <0.001  −0.5 (−0.7 / −0.3)  1.1 (0.9 / 1.29)  1.6  0.29 / 0.048  0.9 (0.84 / 0.94)  Available space 43–45  0.98 (0.96 / 0.99)  0.98 (0.97 / 0.99)  0.19  0.26  0.01 (0.28)  0.836  −0.54 (−0.67 / −0.4)  0.55 (0.42 / 0.69)  1.09  0.25 / 0.082  0.98 (0.97 / 0.99)  Interpremolar width maxilla  0.99 (0.98 / 0.99)  0.99 (0.99 / 1)  0.24  0.22  0.27 (0.27)  <0.001  −0.26 (−0.39 / −0.12)  0.8 (0.67 / 0.93)  1.06  0.06 / 0.699  0.99 (0.98 / 0.99)  Interpremolar width mandible  0.96 (0.94 / 0.98)  0.99 (0.99 / 1)  0.52  0.18  0.08 (0.26)  0.044  −0.44 (−0.57 / −0.31)  0.6 (0.47 / 0.73)  1.04  0.07 / 0.635  0.99 (0.98 / 0.99)  Intermolar width maxilla  0.96 (0.93 / 0.97)  0.98 (0.97 / 0.99)  0.56  0.47  0.71 (0.39)  <0.001  −0.05 (−0.24 / 0.14)  1.46 (1.28 / 1.65)  1.51  0.1 / 0.497  0.95 (0.93 / 0.97)  Intermolar width mandible  0.95 (0.92 / 0.97)  0.97 (0.95 / 0.98)  0.34  0.63  0.63 (0.38)  <0.001  −0.12 (−0.3 / 0.06)  1.37 (1.19 / 1.55)  1.49  0.01 / 0.944  0.95 (0.91 / 0.97)  Computed measurements  Sum 12–22  0.92 (0.89 / 0.95)  0.92 (0.85 / 0.96)  0.52  0.6  −0.48 (1.12)  0.005  −2.67 (−3.21 / −2.13)  1.71 (1.17 / 2.25)  4.38  0.11 / 0.451  0.74 (0.59 / 0.84)  Sum 32–42  0.82 (0.74 / 0.88)  0.93 (0.89 / 0.96)  0.46  0.4  −0.19 (0.61)  0.036  −1.39 (−1.68 / −1.09)  1.01 (0.71 / 1.3)  2.4  0.21 / 0.143  0.84 (0.73 / 0.9)  Total difference maxilla  0.82 (0.73 / 0.89)  0.86 (0.74 / 0.92)  1.15  1.05  1.45 (2.4)  <0.001  −3.25 (−4.41 / −2.09)  6.15 (4.99 / 7.31)  9.4  0.18 / 0.221  0.56 (0.37 / 0.71)  Total difference mandible  0.83 (0.73 / 0.89)  0.91 (0.84 / 0.95)  1.08  0.94  1.13 (1.76)  <0.001  −2.31 (−3.16 / −1.46)  4.57 (3.72 / 5.42)  6.88  0.1 / 0.501  0.72 (0.57 / 0.83)  Bolton 3-3 (n = 24)  0.66 (0.49 / 0.81)  0.81 (0.7 / 0.9)  0.01  0.01  0.01 (0.02)  0.004  −0.02 (−0.04 / −0.01)  0.05 (0.04 / 0.06)  0.07  0.01 / 0.981  0.56 (0.27 / 0.76)  Bolton 5-5 (n = 24)  0.59 (0.41 / 0.76)  0.81 (0.69 / 0.9)  0.01  0.01  0.01 (0.02)  0.177  −0.03 (−0.04 / −0.02)  0.04 (0.03 / 0.05)  0.07  0.02 / 0.913  0.55 (0.23 / 0.76)  Tonn  0.62 (0.5 / 0.74)  0.83 (0.75 / 0.89)  0.61  0.55  −0.27 (0.91)  0.041  −2.05 (−2.49 / −1.61)  1.5 (1.06 / 1.94)  3.55  0.05 / 0.762  0.62 (0.43 / 0.76)  Transversal premolar relationship  0.93 (0.89 / 0.95)  0.98 (0.96 / 0.98)  0.55  0.26  0.19 (0.37)  0.001  −0.52 (−0.7 / −0.35)  0.91 (0.73 / 1.09)  1.43  0.34 / 0.017  0.96 (0.94 / 0.98)  Transversal molar relationship  0.87 (0.78 / 0.93)  0.92 (0.88 / 0.95)  0.66  0.67  0.02 (0.57)  0.821  −1.1 (−1.37 / −0.82)  1.14 (0.86 / 1.41)  2.24  0.02 / 0.891  0.94 (0.9 / 0.97)      Intra-examiner reliability (within method)  Intra-examiner agreement (between methods)  Outcomes  Manual ICC (95% CI)*  Digital ICC (95% CI)*  Manual Dahlberg’s error** mm  Digital Dahlberg’s error** mm  Mean of Δ (SD) mm  Δ different from 0 P value  Lower LOA (95% CI) mm  Upper LOA (95% CI) mm  Range of lower and upper LOA mm  Changing bias regression r / P value  Conformity CCC (95% CI)  2-landmark measurements on casts  Tooth size 15 (n = 24)  0.73 (0.56 / 0.86)  0.78 (0.65 / 0.88)  0.33  0.26  −0.28 (0.27)  <0.001  −0.8 (−0.98 / −0.61)  0.25 (0.06 / 0.43)  1.05  0.03 / 0.892  0.64 (0.4 / 0.8)  Tooth size 14 (n = 24)  0.72 (0.56 / 0.84)  0.82 (0.7 / 0.9)  0.3  0.22  −0.35 (0.28)  <0.001  −0.89 (−1.08 / −0.7)  0.19 (0 / 0.38)  1.08  0.07 / 0.739  0.61 (0.38 / 0.77)  Tooth size 13 (n = 24)  0.82 (0.7 / 0.9)  0.77 (0.64 / 0.88)  0.2  0.25  0.03 (0.25)  0.577  −0.47 (−0.64 / −0.29)  0.52 (0.35 / 0.7)  0.99  0.22 / 0.305  0.79 (0.58 / 0.9)  Tooth size 12  0.86 (0.8 / 0.91)  0.88 (0.81 / 0.92)  0.21  0.22  −0.13 (0.4)  0.026  −0.91 (−1.1 / −0.72)  0.65 (0.45 / 0.84)  1.56  0.13 / 0.374  0.66 (0.47 / 0.79)  Tooth size 11  0.9 (0.85 / 0.94)  0.92 (0.88 / 0.95)  0.21  0.19  −0.03 (0.31)  0.549  −0.64 (−0.79 / −0.49)  0.58 (0.43 / 0.73)  1.22  0.21 / 0.16  0.82 (0.7 / 0.9)  Tooth size 21  0.92 (0.87 / 0.95)  0.88 (0.8 / 0.92)  0.17  0.25  −0.2 (0.34)  <0.001  −0.87 (−1.04 / −0.71)  0.47 (0.3 / 0.63)  1.34  0.04 / 0.81  0.75 (0.6 / 0.84)  Tooth size 22  0.88 (0.82 / 0.92)  0.84 (0.76 / 0.9)  0.17  0.25  −0.17 (0.4)  0.006  −0.96 (−1.16 / −0.77)  0.62 (0.43 / 0.82)  1.58  0.11 / 0.446  0.61 (0.43 / 0.75)  Tooth size 23 (n = 24)  0.87 (0.78 / 0.93)  0.7 (0.55 / 0.84)  0.17  0.3  −0.06 (0.31)  0.371  −0.67 (−0.89 / −0.46)  0.56 (0.34 / 0.77)  1.23  0.46 / 0.023  0.72 (0.46 / 0.87)  Tooth size 24 (n = 24)  0.81 (0.69 / 0.9)  0.81 (0.69 / 0.9)  0.21  0.3  −0.29 (0.29)  <0.001  −0.86 (−1.06 / −0.66)  0.29 (0.09 / 0.49)  1.15  0.02 / 0.915  0.64 (0.4 / 0.8)  Tooth size 25 (n = 24)  0.76 (0.63 / 0.87)  0.72 (0.57 / 0.85)  0.27  0.24  −0.34 (0.19)  <0.001  −0.72 (−0.85 / −0.58)  0.04 (−0.09 / 0.17)  0.76  0.27 / 0.209  0.65 (0.45 / 0.79)  Tooth size 35 (n = 24)  0.67 (0.51 / 0.82)  0.76 (0.59 / 0.88)  0.33  0.2  −0.22 (0.27)  0.001  −0.74 (−0.93 / −0.56)  0.31 (0.13 / 0.49)  1.05  0.16 / 0.47  0.72 (0.49 / 0.85)  Tooth size 34 (n = 24)  0.77 (0.63 / 0.88)  0.83 (0.72 / 0.91)  0.27  0.23  −0.52 (0.26)  <0.001  −1.04 (−1.22 / −0.85)  0 (−0.18 / 0.18)  1.04  0.01 / 0.975  0.47 (0.26 / 0.63)  Tooth size 33 (n = 24)  0.76 (0.63 / 0.87)  0.88 (0.79 / 0.94)  0.24  0.19  0.06 (0.24)  0.238  −0.4 (−0.57 / −0.24)  0.52 (0.36 / 0.68)  0.92  0.05 / 0.807  0.81 (0.63 / 0.91)  Tooth size 32  0.79 (0.7 / 0.86)  0.84 (0.78 / 0.9)  0.17  0.17  −0.06 (0.24)  0.104  −0.54 (−0.65 / −0.42)  0.42 (0.3 / 0.54)  0.96  0.23 / 0.118  0.7 (0.53 / 0.82)  Tooth size 31  0.79 (0.69 / 0.86)  0.86 (0.8 / 0.91)  0.15  0.14  −0.07 (0.19)  0.015  −0.44 (−0.53 / −0.35)  0.3 (0.21 / 0.39)  0.74  0.04 / 0.807  0.8 (0.67 / 0.88)  Tooth size 41  0.68 (0.56 / 0.78)  0.79 (0.71 / 0.86)  0.18  0.19  −0.01 (0.19)  0.82  −0.38 (−0.47 / −0.29)  0.37 (0.27 / 0.46)  0.75  0.09 / 0.546  0.77 (0.62 / 0.86)  Tooth size 42  0.75 (0.65 / 0.83)  0.86 (0.8 / 0.91)  0.18  0.19  0 (0.23)  0.951  −0.46 (−0.57 / −0.34)  0.45 (0.34 / 0.56)  0.91  0.12 / 0.409  0.79 (0.66 / 0.88)  Tooth size 43 (n = 24)  0.87 (0.79 / 0.94)  0.82 (0.71 / 0.91)  0.13  0.19  0.06 (0.16)  0.09  −0.26 (−0.37 / −0.15)  0.37 (0.26 / 0.48)  0.63  0.1 / 0.654  0.91 (0.81 / 0.96)  Tooth size 44 (n = 24)  0.77 (0.61 / 0.88)  0.8 (0.68 / 0.89)  0.22  0.23  −0.32 (0.21)  <0.001  −0.74 (−0.88 / −0.59)  0.1 (−0.05 / 0.24)  0.84  0.03 / 0.882  0.69 (0.5 / 0.82)  Tooth size 45 (n = 24)  0.85 (0.73 / 0.92)  0.81 (0.7 / 0.9)  0.21  0.24  −0.3 (0.29)  <0.001  −0.86 (−1.05 / −0.66)  0.27 (0.07 / 0.46)  1.13  0.13 / 0.546  0.67 (0.44 / 0.82)  Available space 15–13  0.95 (0.91 / 0.97)  0.88 (0.83 / 0.92)  0.21  0.32  0.11 (0.43)  0.089  −0.74 (−0.95 / −0.53)  0.95 (0.75 / 1.16)  1.69  0.1 / 0.487  0.95 (0.92 / 0.97)  Available space 12–22  0.89 (0.84 / 0.93)  0.96 (0.95 / 0.98)  0.51  0.33  0.19 (0.43)  0.004  −0.66 (−0.87 / −0.45)  1.03 (0.83 / 1.24)  1.69  0.15 / 0.311  0.96 (0.92 / 0.97)  Available space 23–25  0.95 (0.93 / 0.97)  0.96 (0.94 / 0.98)  0.22  0.26  −0.02 (0.38)  0.762  −0.76 (−0.94 / −0.58)  0.73 (0.54 / 0.91)  1.49  0.04 / 0.805  0.96 (0.93 / 0.98)  Available space 35–33  0.98 (0.97 / 0.99)  0.98 (0.97 / 0.99)  0.17  0.2  0.06 (0.24)  0.085  −0.41 (−0.52 / −0.29)  0.53 (0.41 / 0.64)  0.94  0.42 / 0.003  0.99 (0.98 / 0.99)  Available space 32–42  0.91 (0.87 / 0.94)  0.97 (0.95 / 0.98)  0.39  0.22  0.30 (0.41)  <0.001  −0.5 (−0.7 / −0.3)  1.1 (0.9 / 1.29)  1.6  0.29 / 0.048  0.9 (0.84 / 0.94)  Available space 43–45  0.98 (0.96 / 0.99)  0.98 (0.97 / 0.99)  0.19  0.26  0.01 (0.28)  0.836  −0.54 (−0.67 / −0.4)  0.55 (0.42 / 0.69)  1.09  0.25 / 0.082  0.98 (0.97 / 0.99)  Interpremolar width maxilla  0.99 (0.98 / 0.99)  0.99 (0.99 / 1)  0.24  0.22  0.27 (0.27)  <0.001  −0.26 (−0.39 / −0.12)  0.8 (0.67 / 0.93)  1.06  0.06 / 0.699  0.99 (0.98 / 0.99)  Interpremolar width mandible  0.96 (0.94 / 0.98)  0.99 (0.99 / 1)  0.52  0.18  0.08 (0.26)  0.044  −0.44 (−0.57 / −0.31)  0.6 (0.47 / 0.73)  1.04  0.07 / 0.635  0.99 (0.98 / 0.99)  Intermolar width maxilla  0.96 (0.93 / 0.97)  0.98 (0.97 / 0.99)  0.56  0.47  0.71 (0.39)  <0.001  −0.05 (−0.24 / 0.14)  1.46 (1.28 / 1.65)  1.51  0.1 / 0.497  0.95 (0.93 / 0.97)  Intermolar width mandible  0.95 (0.92 / 0.97)  0.97 (0.95 / 0.98)  0.34  0.63  0.63 (0.38)  <0.001  −0.12 (−0.3 / 0.06)  1.37 (1.19 / 1.55)  1.49  0.01 / 0.944  0.95 (0.91 / 0.97)  Computed measurements  Sum 12–22  0.92 (0.89 / 0.95)  0.92 (0.85 / 0.96)  0.52  0.6  −0.48 (1.12)  0.005  −2.67 (−3.21 / −2.13)  1.71 (1.17 / 2.25)  4.38  0.11 / 0.451  0.74 (0.59 / 0.84)  Sum 32–42  0.82 (0.74 / 0.88)  0.93 (0.89 / 0.96)  0.46  0.4  −0.19 (0.61)  0.036  −1.39 (−1.68 / −1.09)  1.01 (0.71 / 1.3)  2.4  0.21 / 0.143  0.84 (0.73 / 0.9)  Total difference maxilla  0.82 (0.73 / 0.89)  0.86 (0.74 / 0.92)  1.15  1.05  1.45 (2.4)  <0.001  −3.25 (−4.41 / −2.09)  6.15 (4.99 / 7.31)  9.4  0.18 / 0.221  0.56 (0.37 / 0.71)  Total difference mandible  0.83 (0.73 / 0.89)  0.91 (0.84 / 0.95)  1.08  0.94  1.13 (1.76)  <0.001  −2.31 (−3.16 / −1.46)  4.57 (3.72 / 5.42)  6.88  0.1 / 0.501  0.72 (0.57 / 0.83)  Bolton 3-3 (n = 24)  0.66 (0.49 / 0.81)  0.81 (0.7 / 0.9)  0.01  0.01  0.01 (0.02)  0.004  −0.02 (−0.04 / −0.01)  0.05 (0.04 / 0.06)  0.07  0.01 / 0.981  0.56 (0.27 / 0.76)  Bolton 5-5 (n = 24)  0.59 (0.41 / 0.76)  0.81 (0.69 / 0.9)  0.01  0.01  0.01 (0.02)  0.177  −0.03 (−0.04 / −0.02)  0.04 (0.03 / 0.05)  0.07  0.02 / 0.913  0.55 (0.23 / 0.76)  Tonn  0.62 (0.5 / 0.74)  0.83 (0.75 / 0.89)  0.61  0.55  −0.27 (0.91)  0.041  −2.05 (−2.49 / −1.61)  1.5 (1.06 / 1.94)  3.55  0.05 / 0.762  0.62 (0.43 / 0.76)  Transversal premolar relationship  0.93 (0.89 / 0.95)  0.98 (0.96 / 0.98)  0.55  0.26  0.19 (0.37)  0.001  −0.52 (−0.7 / −0.35)  0.91 (0.73 / 1.09)  1.43  0.34 / 0.017  0.96 (0.94 / 0.98)  Transversal molar relationship  0.87 (0.78 / 0.93)  0.92 (0.88 / 0.95)  0.66  0.67  0.02 (0.57)  0.821  −1.1 (−1.37 / −0.82)  1.14 (0.86 / 1.41)  2.24  0.02 / 0.891  0.94 (0.9 / 0.97)  Apart from columns 8 and 12, bold values refer to correlation coefficients smaller than 0.7 or differences greater than 0.5 mm. ICC, intraclass correlation coefficient; CI, confidence intervals; Δ, manual (median of 5 repeated measurements) − digital (median of 5 repeated measurements); SD, standard deviation; LOA, limits of agreement; CCC, Lin’s concordance correlation coefficient. *values for single measurements with a mixed model and absolute agreement. **calculated only on the first two repeated measurements. View Large Intra-observer reliability proved to be substantial in both manual and digital methods, except for five outcomes (12.8 per cent) in the manual method (0.59 ≤ ICC ≤ 0.68). Three of these outcomes were tooth size discrepancy indices (0.59 ≤ ICC ≤ 0.66). Reliability was higher in the digital (ICC ≥ 0.7 for all the outcomes) than the manual method. Dahlberg’s error was comparable between the methods and approximately 1 mm for the outcomes total difference in maxilla and mandible. In general, systematic bias was close to zero (−0.5 mm ≤ Δ ≤ 0.5 mm) for most measurements (87.2 per cent), therefore manual and digital methods agree to the nearest 0.5 mm on average, although P values for one sample t-tests on the difference of Δ from zero were significant for most outcomes. Outcomes that showed distinct bias were total space difference of maxilla and mandible (Δ = 1.45 mm and Δ = 1.13 mm, respectively) and to a lesser degree intermolar width of maxilla and mandible (Δ = 0.71 mm and Δ = 0.63 mm, respectively). Furthermore, tooth sizes were larger when measured digitally, since the sign of Δ was negative for 80 per cent of the teeth measured, whereas 90 per cent of the directly measured distances had a positive sign. Limits of agreement (LOA) on Δ were relatively wide for the computed outcomes sum 12–22 (−2.67 mm ≤ LOA ≤ 1.71 mm), sum 32–42 (−1.39 mm ≤ LOA ≤ 1.01 mm), total difference maxilla (−3.25 mm ≤ LOA ≤ 6.15 mm), total difference mandible (−2.31 mm ≤ LOA ≤ 4.57 mm), and Tonn (−2.05 mm ≤ LOA ≤ 1.5 mm). However, the range between the lower and upper LOA (Table 3) was in general lower than the range of repeated intra-observer measurement variation for both manual and digital methods (Supplementary Table 2). Thus, the differences detected between both methods did not exceed intra-method error/variation. Manual and digital methods seem to conform well with each other in general (0.47 ≤ CCC ≤ 0.99), since the majority of outcomes (66.7 per cent) had a concordance correlation coefficient of more than 0.7 (substantial conformity). Premolars and tooth size discrepancy indices (computed outcomes) seem to produce results, which do not agree between methods (0.47 ≤ CCC ≤ 0.69 and 0.55 ≤ CCC ≤ 0.62, respectively). Finally, no signs of changing bias were present, as the P values from the regression slopes were sporadically significant. Therefore, bias between methods changes at random and not according to the true values as measured with the manual method. The corresponding plots are provided in Supplementary Figure 3. Discussion In this study, we investigated the intra-observer reliability and agreement between traditional manual model analysis, based on plaster casts and a dial caliper, and digital model analysis, based on virtual models acquired with orthoX®scan and analysed with ivoris®analyze 3D. We found that intra-observer reliability within each method was generally substantial, whereas agreement/conformity between the two methods was relatively limited, especially for computed outcomes, which are of particular diagnostic interest to orthodontists. However, digital model analysis proved to be more reliable with larger tooth sizes, but smaller distances measured on average. Among the factors influencing measurement error is observer’s experience (12). In our study, the observer was experienced and familiar with the software used and completed all the measurements within 2 months. Experienced observers generally produce more consistent readings in repeated measurements (12). However, this is not always the case and repeated measurements can vary even though they were performed by the same observer, especially when the interval between repeated measurements increases (13). In any case, intra-observer reliability is generally higher than inter-observer reliability and that is attributed to random error (14). Due to this fact, in measurement error studies, the number of observers should be ideally considered as the sample size of the study, in order to draw inferences for the population of observers in general (10). Point identification and positioning is perhaps the greatest source of measurement error (14, 15). Thus, reliability of point identification most likely directly relates to the reliability of the model analysis itself. Whereas point identification correlates with point definition, shape of the anatomical structure measured, and observer’s experience/judgment (16), point positioning mostly depends on properties of the measuring instrument and measured item. In the present study, we adopted common definitions of anatomical points. However, some imprecision in point identification is always present, which is inherent in model analysis itself irrespective of the method used, i.e. a contact point may actually be a contact area, causing variation in point identification. The shape of the anatomical structure being measured also plays a role on point identification, since it was found that points located at the edges of anatomical structures are more precisely identifiable than points located at curved anatomical structures (16). This could be observed in our results as well, since mesiodistal sizes of premolars presented in general lower conformity between the manual and digital methods than mesiodistal sizes of incisors. Another aspect possibly contributing to these results is the limited sample size for outcomes only measurable on casts of permanent dentitions, such as premolar width (n = 24). Finally, with respect to the point positioning, calipers cannot completely access the maximum mesiodistal diameter of teeth, especially if crowding is present, and impression materials do not exactly imprint the space between crowded teeth in plaster casts (17). On the contrary, software solutions for digital model analysis provide a variety of functions, such as zooming or view rotation, which facilitate point positioning particularly at proximal contacts, even if teeth are crowded. Our results indicated that mesiodistal tooth sizes were measured larger digitally than manually, which is in agreement with other findings (18, 19). Furthermore, variation in point identification and positioning is greater for outcomes consisting of more than one point (16). This leads to cumulative error in outcomes that are computed from more than one tooth, such as the sum of upper or lower incisors widths. This is also the case with measurements derived by mathematical operations, as in the case of tooth size discrepancy indices (Bolton/Tonn). Moreover, in our study, the total width of the posterior teeth was calculated according to Moyers (20) at 75 per cent probability level and not directly measured in the early mixed dentitions. These factors could explain our results regarding imprecision in tooth size discrepancy indices within the manual method as well as the wide LOA ranges of computed outcomes between methods. Despite their higher imprecision, computed outcomes are of particular diagnostic interest to orthodontists. Lower crowding, lower lip to E-plane, upper crowding, and overjet are important diagnostic variables in Class I extraction treatments (21), whereas lower anterior crowding, molar relationship, and growth pattern are the most influential factors in Class II extraction treatments (22). Therefore, a precise and accurate space analysis is needed for meaningful treatment decisions (23). Although bias between methods was within 1–1.5 mm for space measurements, LOA indicated that 95 per cent of the differences between methods could vary from zero across a wide range, which could eventually influence clinical decisions. With respect to the averaged time for manual and digital model analyses per model, our data suggested that differences were negligible (manual mean 7 minutes and 59 seconds and SD 1 minutes and 36 seconds; digital mean 8 minutes and 36 seconds and SD 2 minutes and 10 seconds). However, one should also consider the scanning process, which could be long for dental casts with unstable occlusions, such as open bites. The average duration of the scanning process was 6 minutes and 43 seconds (SD 13 seconds), but for open bites approximately 2 more minutes were needed, since fixating open bites onto the model holder was more time consuming than stable occluded dentitions. Finally, some models had to be scanned twice resulting in an averaged duration of 11 minutes and 53 seconds (SD 28 seconds). Conclusions The following conclusions can be drawn from the present study regarding orthodontic model analysis with plaster casts and a dial caliper and virtual models obtained with orthoX®scan and analysed with ivoris®analyze 3D: - Both methods were reliable but the digital model analysis was more reliable than the manual one with less variability encountered in repeated measurements. - The digital method yielded on average larger tooth sizes but smaller other distances than the manual method. - Methods agree substantially up to 0.5 mm with each other for most direct measurements in model analysis. Caution is warranted for some computed space parameters, since bias is expected to have a wide range between methods due to cumulative error. Supplementary material Supplementary material is available at European Journal of Orthodontics online. Conflicts of interest None to declare. References 1. Fleming P.S. Marinho V. and Johal A. ( 2011) Orthodontic measurements on digital study models compared with plaster models: a systematic review. Orthodontics & Craniofacial Research , 14, 1– 16. Google Scholar CrossRef Search ADS PubMed  2. Luu N.S. Nikolcheva L.G. Retrouvey J.-M. Flores-Mir C. El-Bialy T. Carey J.P. and Major P.W. ( 2012) Linear measurements using virtual study models. Angle Orthodontist , 82, 1098– 1106. Google Scholar CrossRef Search ADS PubMed  3. Rossini G. Parrini S. Castroflorio T. Deregibus A. and Debernardi C.L. ( 2016) Diagnostic accuracy and measurement sensitivity of digital models for orthodontic purposes: A systematic review. American Journal of Orthodontics and Dentofacial Orthopedics , 149, 161– 170. Google Scholar CrossRef Search ADS PubMed  4. Kottner J. Audigé L. Brorson S. Donner A. Gajewski B.J. Hróbjartsson A. Roberts C. Shoukri M. and Streiner D.L. ( 2011) Guidelines for Reporting Reliability and Agreement Studies (GRRAS) were proposed. Journal of Clinical Epidemiology , 64, 96– 106. Google Scholar CrossRef Search ADS PubMed  5. Lomax R.G. and Hahs-Vaughn D.L. ( 2012) An Introduction to Statistical Concepts . Routledge, New York. 6. Bland J.M. and Altman D. ( 1986) Statistical methods for assessing agreement between two methods of clinical measurement. Lancet , 327, 307– 310. Google Scholar CrossRef Search ADS   7. Lin L.I. ( 1989) A concordance correlation coefficient to evaluate reproducibility. Biometrics , 45, 255– 268. Google Scholar CrossRef Search ADS PubMed  8. Asquith J. Gillgrass T. and Mossey P. ( 2007) Three-dimensional imaging of orthodontic models: a pilot study. European Journal of Orthodontics , 29, 517– 522. Google Scholar CrossRef Search ADS PubMed  9. Luu N.S. Mandich M.-A. Flores-Mir C. El-Bialy T. Heo G. Carey J.P. and Major P.W. ( 2014) The validity, reliability, and time requirement of study model analysis using cone-beam computed tomography-generated virtual study models. Orthodontics & Craniofacial Research , 17, 14– 26. Google Scholar CrossRef Search ADS PubMed  10. Bartlett J.W. and Frost C. ( 2008) Reliability, repeatability and reproducibility: analysis of measurement errors in continuous variables. Ultrasound in Obstetrics & Gynecology , 31, 466– 475. Google Scholar CrossRef Search ADS   11. Faul F. Erdfelder E. Buchner A. and Lang A.G. ( 2009) Statistical power analyses using G*Power 3.1: tests for correlation and regression analyses. Behavior Research Methods , 41, 1149– 1160. Google Scholar CrossRef Search ADS PubMed  12. Gesch D. Schröder W. Kocher T. and Kirbschus A. ( 2006) Examiner differences in the assessment of different malocclusions. Journal of Orofacial Orthopedics , 67, 404– 413. Google Scholar CrossRef Search ADS PubMed  13. Kieser J.A. Groeneveld H.T. McKee J. and Cameron N. ( 1990) Measurement error in human dental mensuration. Annals of Human Biology , 17, 523– 528. Google Scholar CrossRef Search ADS PubMed  14. Kamoen A. Dermaut L. and Verbeeck R. ( 2001) The clinical significance of error measurement in the interpretation of treatment results. European Journal of Orthodontics , 23, 569– 578. Google Scholar CrossRef Search ADS PubMed  15. Houston W.J. Maher R.E. McElroy D. and Sherriff M. ( 1986) Sources of error in measurements from cephalometric radiographs. European Journal of Orthodontics , 8, 149– 151. Google Scholar CrossRef Search ADS PubMed  16. Baumrind S. and Frantz R.C. ( 1971) The reliability of head film measurements. American Journal of Orthodontics , 60, 111– 127. Google Scholar CrossRef Search ADS PubMed  17. Harris E.F. and Smith R.N. ( 2009) Accounting for measurement error: a critical but often overlooked process. Archives of Oral Biology , 54 ( suppl 1), S107– S117. Google Scholar CrossRef Search ADS PubMed  18. Abizadeh N. Moles D.R. O’Neill J. and Noar J.H. ( 2012) Digital versus plaster study models: how accurate and reproducible are they? Journal of Orthodontics , 39, 151– 159. Google Scholar CrossRef Search ADS PubMed  19. Czarnota J. Hey J. and Fuhrmann R. ( 2016) Measurements using orthodontic analysis software on digital models obtained by 3D scans of plaster casts. Journal of Orofacial Orthopedics , 77, 22– 30. Google Scholar CrossRef Search ADS PubMed  20. Moyers R. ( 1988) Handbook of Orthodontics . Year Book Medical Publishers, Chicago, IL, 4th edn. 21. Konstantonis D. Anthopoulou C. and Makou M. ( 2013) Extraction decision and identification of treatment predictors in Class I malocclusions. Progress in Orthodontics , 14, 47. Google Scholar CrossRef Search ADS PubMed  22. Guo Y. Han X. Xu H. Ai D. Zeng H. and Bai D. ( 2014) Morphological characteristics influencing the orthodontic extraction strategies for Angle’s class II division 1 malocclusions. Progress in Orthodontics , 15, 44. Google Scholar CrossRef Search ADS PubMed  23. Naish H. Dunbar C. Crouch-Baker J. Shah K. Wallis C. Atack N.E. Sherriff M. Sandy J.R. and Ireland A.J. ( 2016) Does a true knowledge of dental crowding affect orthodontic treatment decisions? European Journal of Orthodontics , 38, 66– 70. Google Scholar CrossRef Search ADS PubMed  © The Author(s) 2017. Published by Oxford University Press on behalf of the European Orthodontic Society. All rights reserved. For permissions, please email: journals.permissions@oup.com

Journal

The European Journal of OrthodonticsOxford University Press

Published: Feb 1, 2018

There are no references for this article.

You’re reading a free preview. Subscribe to read the entire article.


DeepDyve is your
personal research library

It’s your single place to instantly
discover and read the research
that matters to you.

Enjoy affordable access to
over 12 million articles from more than
10,000 peer-reviewed journals.

All for just $49/month

Explore the DeepDyve Library

Unlimited reading

Read as many articles as you need. Full articles with original layout, charts and figures. Read online, from anywhere.

Stay up to date

Keep up with your field with Personalized Recommendations and Follow Journals to get automatic updates.

Organize your research

It’s easy to organize your research with our built-in tools.

Your journals are on DeepDyve

Read from thousands of the leading scholarly journals from SpringerNature, Elsevier, Wiley-Blackwell, Oxford University Press and more.

All the latest content is available, no embargo periods.

See the journals in your area

Monthly Plan

  • Read unlimited articles
  • Personalized recommendations
  • No expiration
  • Print 20 pages per month
  • 20% off on PDF purchases
  • Organize your research
  • Get updates on your journals and topic searches

$49/month

Start Free Trial

14-day Free Trial

Best Deal — 39% off

Annual Plan

  • All the features of the Professional Plan, but for 39% off!
  • Billed annually
  • No expiration
  • For the normal price of 10 articles elsewhere, you get one full year of unlimited access to articles.

$588

$360/year

billed annually
Start Free Trial

14-day Free Trial