Quality measurement in physician-staffed emergency medical services: a systematic literature review

Quality measurement in physician-staffed emergency medical services: a systematic literature review Abstract Purpose Quality measurement of physician-staffed emergency medical services (P-EMS) is necessary to improve service quality. Knowledge and consensus on this topic are scarce, making quality measurement of P-EMS a high-priority research area. The aim of this review was to identify, describe and evaluate studies of quality measurement in P-EMS. Data sources The databases of MEDLINE and Embase were searched initially, followed by a search for included article citations in Scopus. Study selection The study eligibility criteria were: (1) articles describing the use of one quality indicator (QI) or more in P-EMS, (2) original manuscripts, (3) articles published from 1 January 1968 until 5 October 2016. The literature search identified 4699 records. 4543 were excluded after reviewing title and abstract. An additional 129 were excluded based on a full-text review. The remaining 27 papers were included in the analysis. Methodological quality was assessed using an adapted critical appraisal tool. Data extraction The description of used QIs and methods of quality measurement was extracted. Variables describing the involved P-EMSs were extracted as well. Results of data synthesis In the included papers, a common understanding of which QIs to use in P-EMS did not exist. Fifteen papers used only a single QI. The most widely used QIs were ‘Adherence to medical protocols’, ‘Provision of advanced interventions’, ‘Response time’ and ‘Adverse events’. Conclusion The review demonstrated a lack of shared understanding of which QIs to use in P-EMS. Moreover, papers using only one QI dominated the literature, thus increasing the risk of a narrow perspective in quality measurement. Future quality measurement in P-EMS should rely on a set of consensus-based QIs, ensuring a comprehensive approach to quality measurement. quality measurement < quality management, quality improvement < quality management, emergency care < setting of care Background Emergency medical services (EMS) provide pre-hospital treatment and transportation to definitive care for patients in need of urgent medical care. EMSs are well integrated in health services in most countries and normally consist of ground ambulances staffed by paramedics, emergency medical technicians or nurses [1–6]. As a supplement to regular EMS, physician-staffed rapid response vehicles or helicopters exist in some areas [7]. Although the operational concept of this physician-staffed EMS (P-EMS) may differ, a common feature is the involvement of a specially trained physician in pre-hospital care of critically ill or injured patients. Depending on the country, these physicians are often anaesthesiologists, surgeons, internists or emergency physicians [1, 3]. The focus on quality measurement in healthcare is increasing [8–12]. As an example, quality dimensions such as efficiency, patient-centeredness and safety have been assessed in several emergency departments [13–15]. In P-EMS, a valid model for quality assessment is needed to achieve appropriate governance, quality assurance and quality improvement [16]. Snooks et al. [17] define the development of meaningful quality indicators (QIs) for EMS as the most important issue for future research in emergency pre-hospital care. For P-EMS, measuring quality of care is considered a priority area of research [18]. Quality measurement can be defined as measuring the extent to which set targets are achieved [19]. A QI is used to measure performance against a recognized standard of care. Donabedian defines three categories of QIs: structure, process and outcome of healthcare [20, 21]. Structure indicators describe the infrastructure of a healthcare system, such as competence of the staff, equipment and deployment and response times. Process indicators evaluate the care provided to the patient, and outcome indicators address the change in patient health status. None of these categories of indicators provide a complete description of the quality of care but address single components. Thus, different types of QIs should be combined to assess the quality of a service [19]. QIs inform clinicians and organizations how the health system performs and aid in the improvement in care. Ideally, all QIs are based upon evidence of their relevance and importance. The process of developing QIs generally includes stakeholders who evaluate the evidence and define the QI parameters [22]. These QIs for P-EMS should be evaluated against patient-oriented outcomes, e.g. pain intensity, morbidity or mortality. However, P-EMS quality can also relate to system factors such as training of traditional EMS, major incident management and the concept of providing equity of access to healthcare. Different stakeholders have different perspectives on what represents quality in healthcare [23, 24], and various QIs for P-EMS are possible. A widely cited definition of quality that also might be applicable for P-EMS systems is ‘the degree to which health services for individuals and populations increase the likelihood of desired health outcomes and are consistent with current professional knowledge’ [9]. Often, a few high impact clinical conditions are identified (in terms of morbidity, mortality, costs and incidence). These conditions are termed ‘tracer conditions’. Examples of tracer conditions for emergency medicine are cardiac arrest or trauma patients with severely reduced consciousness (Glasgow Coma Score < 8). Measuring the outcomes of tracer conditions can predict a system’s response to other clinical states and the overall quality of a service [25]. This systematic review aims to identify, describe and appraise the methodological quality of the literature pertaining to the quality assessment of P-EMS. Methods For the purpose of this review, physicians who staff P-EMS should be trained in critical care, exceeding the competency of a general practitioner on call [26]. Moreover, we define the term ‘pre-hospital’ as relating to procedures administered or care provided prior to patient arrival at the hospital [8]. The studies identified in the review do not address the potential benefit from P-EMS compared to other EMS. Literature search strategy A systematic literature search of MEDLINE and Embase to identify relevant literature was conducted (see Additional file 1 for search strategy). Four different sets of entry terms were applied and combined. These entry terms describe pre-hospital setting, emergency care, physician staffing and finally the concept of quality measurement. All records were collated in an Endnote bibliographic database (©2007 Thompson Reuters). The study followed the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-analyses) guidelines, including the PICOS (Population, Intervention, Comparator, Outcome, Study design) methodology [27]. Here, the participants were all the identified articles describing quality estimation in P-EMS. Our evaluation of intervention, comparison and outcomes was carried out using the data extraction and quality appraisal variables in Tables 1 and 3. The study was registered at PROSPERO (http://www.crd.york.ac.uk//prospero/, registration number CRD42015024421). Table 1 Reasons for excluding 129 out of 156 full-text studies in the eligibility-check of the systematic review Reasons for exclusion  No.  Wrong study design  77  Not about quality measurement in P-EMS  29  Not enough information for quality appraisal and data extraction  9  Only abstracts  6  Not about P-EMS  3  Comparative studies pertaining to new procedures  2  Commentary, letter to editor or editorial  1  Not original article  1  Duplicate  1  Reasons for exclusion  No.  Wrong study design  77  Not about quality measurement in P-EMS  29  Not enough information for quality appraisal and data extraction  9  Only abstracts  6  Not about P-EMS  3  Comparative studies pertaining to new procedures  2  Commentary, letter to editor or editorial  1  Not original article  1  Duplicate  1  Table 1 Reasons for excluding 129 out of 156 full-text studies in the eligibility-check of the systematic review Reasons for exclusion  No.  Wrong study design  77  Not about quality measurement in P-EMS  29  Not enough information for quality appraisal and data extraction  9  Only abstracts  6  Not about P-EMS  3  Comparative studies pertaining to new procedures  2  Commentary, letter to editor or editorial  1  Not original article  1  Duplicate  1  Reasons for exclusion  No.  Wrong study design  77  Not about quality measurement in P-EMS  29  Not enough information for quality appraisal and data extraction  9  Only abstracts  6  Not about P-EMS  3  Comparative studies pertaining to new procedures  2  Commentary, letter to editor or editorial  1  Not original article  1  Duplicate  1  Inclusion/exclusion criteria Articles were included in the systematic literature review if they fulfilled all the following criteria: (1) literature describing methods of quality estimation in P-EMS, i.e. the use of one or more QIs based on quantitative methods, qualitative methods or both; (2) original manuscripts and (3) literature published after 1 January 1968 and until the date of the literature search, 5 October 2016. The rationale for including literature from 1 January 1968, is the establishment of the world’s first civilian physician-staffed helicopter EMS in Munich this year [28]. Articles without abstract, book chapters, editorials, comments and letters to the editor were excluded. Articles in English, French, German and Scandinavian languages were identified. The translation competency of these languages was present in the author group. Literature identification The records from the literature search were exported to www.covidence.org. Here, all titles and corresponding abstracts were screened independently by two of the authors for eligibility (O.U. and H.H.) (Figure 1). In the case of uncertainty, a third reviewer (M.R.) reviewed the title/abstract. Articles clearly not meeting the inclusion criteria were excluded. Articles accepted for full-text screening were assessed in pairs of authors (O.U. and M.R., A.J.K. and H.H.) using the inclusion/exclusion criteria listed above. Excluded articles were listed with the reason for exclusion. If there was any uncertainty about whether an article should be included, there was a discussion until consensus was reached among the authors. One author (H.H.) performed data extraction and quality appraisal and consulted another author (M.R.) in uncertain cases. Further, one author hand-searched references in included articles to identify additional relevant articles (H.H.). Finally, a search in Scopus was conducted to identify articles citing the included articles. Figure 1 View largeDownload slide Information flow through the different phases of the systematic review. Figure 1 View largeDownload slide Information flow through the different phases of the systematic review. Prior to the literature search, the authors made templates for data extraction and quality appraisal. The data extraction and quality appraisal variables were based on the authors’ assumptions on what is important to report in quality measurement studies in P-EMS. However, these variables do not represent a reference standard, since such a standard does not exist, to our knowledge. As a part of data extraction, fixed-system variables were included. Fixed-system variables relate to system characteristics concerning the organization, staffing and operational capacities of the service and are necessary for interpreting the results [29]. Results A total of 4699 articles were identified by the search strategy, 156 of which were accepted for full-text screening. Of these, 129 articles were excluded. The main reason for excluding articles were ‘Wrong study design’, pertaining to articles exclusively comparing different treatment modalities and without any quality measurement objectives (Table 1). A total of 26 articles from the main database search were included for data extraction and quality appraisal [30–55]. One additional article was included from the Scopus search for citing articles [56]. A review of the literature lists of included articles did not result in additional findings. None of the papers gave a complete report of fixed-system variables, thus complicating the comparison of involved P-EMS concepts (Table 2). Twenty-four of the 27 papers use QIs that can be identified as process indicators. Structure indicators and outcome indicators are used less frequently, in two and seven papers, respectively. Twenty-five different QIs were identified, all of which were considered suitable for international use and transferable to other P-EMSs (Table 3). The most widely used QI was ‘Adherence to medical protocols’. This QI measures if medical guidelines are followed, as done by Viergutz et al. [53] who investigated whether guidelines for preclinical care of patients with traumatic head injury were followed. The second most used QI was ‘Provision of advanced interventions’, investigating if the P-EMS unit provided treatment that exceeds the competences of the attending EMS, as done by Mikkelsen et al. [44]. The two following QIs are ‘Response time’ and ‘Adverse events’, the latter exemplified by Nakstad et al. [45], who studied the incidence of desaturation during pre-hospital rapid sequence intubation. Fifteen papers used one single QI, and twelve papers applied a set of QIs. Moreover, three papers used tracer conditions as their approach to quality assessment (Table 2). Pertaining to the internal validity of the papers, ten of the 27 papers did not clearly explain the methodology for developing the QIs (Table 4). Table 2 Data extraction of included articles   Are the following fixed-system variables reported?  Quality indicators used  Multiple, mixed quality indicators  Tracer conditions used    Transportation mode  SAR-capability  Rural and/or urban  Response types  Speciality of physician  Physician experience  Funding  Description of dispatch system  Other fixed-system variables  Structure  Process  Outcome  Patient satisfaction      Akin Paker [30]  ✓  ✗  ✗  ✓  ✗  ✗  ✗  ✗  ✗  ✗  ✓  ✗  ✗  ✓  ✗  Anadelic [31]  ✓  ✗  ✓  Partly  ✗  ✗  ✗  ✓  ✓  ✓  ✓  ✗  ✗  ✓  ✗  Arntz [32]  ✓  ✗  ✓  ✓  ✓  Partly  ✓  ✓  ✓  ✗  ✓  ✗  ✗  ✓  ✗  Breckwoldt [33]  ✓  ✗  ✓  ✗  ✓  ✓  ✗  ✗  ✓  ✗  ✓  ✓  ✗  ✗  ✗  Duchateau [34]  ✓  ✗  ✓  ✗  ✗  ✗  ✗  ✗  ✓  ✗  ✓  ✗  ✗  ✗  ✗  Fjaeldstad [35]  ✓  ✗  ✓  ✓  ✓  ✗  ✓  ✓  ✓  ✗  ✓  ✗  ✗  ✗  ✗  Flabouris [36]  ✓  ✓  ✗  ✓  ✓  ✓  ✗  ✓  ✓  ✗  ✓  ✗  ✗  ✓  ✗  Helm [37]  ✓  ✗  ✗  ✗  ✗  ✗  ✗  ✗  ✗  ✗  ✓  ✗  ✗  ✗  ✗  Helm [38]  ✓  ✗  ✗  ✗  ✓  ✗  ✗  ✗  ✗  ✗  ✓  ✗  ✗  ✗  ✗  Hennes [39]  ✓  ✗  ✓  ✓  ✗  ✗  ✗  ✗  ✗  ✗  ✗  ✓  ✗  ✗  ✗  Klemenc-Ketis [40]  ✓  ✓  ✓  ✓  ✗  ✗  ✓  ✓  ✓  ✗  ✓  ✗  ✗  ✓  ✗  Leicht [41]  ✓  ✗  ✗  ✓  ✓  ✓  ✗  ✗  ✓  ✗  ✓  ✗  ✗  ✗  ✗  Lossius [42]  ✓  ✗  ✓  ✓  ✓  ✓  ✗  Partly  ✗  ✗  ✗  ✓  ✗  ✗  ✗  Messelken [43]  ✓  ✗  ✓  ✓  ✓  ✗  ✗  ✗  ✗  ✗  ✓  ✓  ✗  ✓  ✓  Mikkelsen [44]  ✓  ✗  ✓  ✓  ✓  ✓  ✗  ✓  ✓  ✗  ✓  ✓  ✗  ✗  ✗  Nakstad [45]  ✓  ✗  ✓  ✓  ✓  ✓  ✗  ✗  ✓  ✗  ✓  ✗  ✗  ✗  ✗  Neukamm [46]  ✓  ✗  ✓  ✓  ✗  ✗  ✗  Partly  ✓  ✓  ✓  ✓  ✗  ✓  ✗  Pedersen [47]  ✓  ✗  ✓  ✓  ✗  ✗  ✗  ✗  ✗  ✗  ✗  ✗  ✓  ✗  ✓  Regel [48]  ✓  ✗  ✓  ✓  ✗  ✗  ✗  ✗  ✓  ✗  ✓  ✗  ✗  ✓  ✗  Rognås [49]  ✓  ✗  ✓  ✓  ✓  ✓  ✗  ✓  ✓  ✗  ✓  ✗  ✗  ✗  ✗  Rognås [50]  ✓  ✗  ✓  ✓  ✓  ✗  ✗  Partly  ✓  ✗  ✓  ✗  ✗  ✗  ✗  Sollid [51]  ✓  ✗  ✓  ✓  ✓  ✗  ✗  ✗  ✓  ✗  ✓  ✗  ✗  ✗  ✗  Van der Velden [52]  ✓  ✗  ✓  ✓  ✗  ✗  ✗  ✓  ✓  ✗  ✓  ✗  ✗  ✗  ✗  Viergutz [53]  ✓  ✗  ✓  ✓  ✗  ✗  ✗  ✗  ✓  ✗  ✓  ✗  ✗  ✓  ✗  Von Knobelsdorff [54]  ✓  ✗  ✓  ✓  ✗  ✗  ✗  ✗  ✗  ✗  ✓  ✓  ✗  ✓  ✗  Weltermann [55]  ✓  ✗  ✓  ✓  ✗  ✗  ✗  ✗  ✗  ✗  ✓  ✗  ✗  ✓  ✗  Schlechtriemen [56]  ✓  ✗  ✗  ✗  ✗  ✗  ✗  ✗  ✗  ✗  ✓  ✗  ✗  ✓  ✓    Are the following fixed-system variables reported?  Quality indicators used  Multiple, mixed quality indicators  Tracer conditions used    Transportation mode  SAR-capability  Rural and/or urban  Response types  Speciality of physician  Physician experience  Funding  Description of dispatch system  Other fixed-system variables  Structure  Process  Outcome  Patient satisfaction      Akin Paker [30]  ✓  ✗  ✗  ✓  ✗  ✗  ✗  ✗  ✗  ✗  ✓  ✗  ✗  ✓  ✗  Anadelic [31]  ✓  ✗  ✓  Partly  ✗  ✗  ✗  ✓  ✓  ✓  ✓  ✗  ✗  ✓  ✗  Arntz [32]  ✓  ✗  ✓  ✓  ✓  Partly  ✓  ✓  ✓  ✗  ✓  ✗  ✗  ✓  ✗  Breckwoldt [33]  ✓  ✗  ✓  ✗  ✓  ✓  ✗  ✗  ✓  ✗  ✓  ✓  ✗  ✗  ✗  Duchateau [34]  ✓  ✗  ✓  ✗  ✗  ✗  ✗  ✗  ✓  ✗  ✓  ✗  ✗  ✗  ✗  Fjaeldstad [35]  ✓  ✗  ✓  ✓  ✓  ✗  ✓  ✓  ✓  ✗  ✓  ✗  ✗  ✗  ✗  Flabouris [36]  ✓  ✓  ✗  ✓  ✓  ✓  ✗  ✓  ✓  ✗  ✓  ✗  ✗  ✓  ✗  Helm [37]  ✓  ✗  ✗  ✗  ✗  ✗  ✗  ✗  ✗  ✗  ✓  ✗  ✗  ✗  ✗  Helm [38]  ✓  ✗  ✗  ✗  ✓  ✗  ✗  ✗  ✗  ✗  ✓  ✗  ✗  ✗  ✗  Hennes [39]  ✓  ✗  ✓  ✓  ✗  ✗  ✗  ✗  ✗  ✗  ✗  ✓  ✗  ✗  ✗  Klemenc-Ketis [40]  ✓  ✓  ✓  ✓  ✗  ✗  ✓  ✓  ✓  ✗  ✓  ✗  ✗  ✓  ✗  Leicht [41]  ✓  ✗  ✗  ✓  ✓  ✓  ✗  ✗  ✓  ✗  ✓  ✗  ✗  ✗  ✗  Lossius [42]  ✓  ✗  ✓  ✓  ✓  ✓  ✗  Partly  ✗  ✗  ✗  ✓  ✗  ✗  ✗  Messelken [43]  ✓  ✗  ✓  ✓  ✓  ✗  ✗  ✗  ✗  ✗  ✓  ✓  ✗  ✓  ✓  Mikkelsen [44]  ✓  ✗  ✓  ✓  ✓  ✓  ✗  ✓  ✓  ✗  ✓  ✓  ✗  ✗  ✗  Nakstad [45]  ✓  ✗  ✓  ✓  ✓  ✓  ✗  ✗  ✓  ✗  ✓  ✗  ✗  ✗  ✗  Neukamm [46]  ✓  ✗  ✓  ✓  ✗  ✗  ✗  Partly  ✓  ✓  ✓  ✓  ✗  ✓  ✗  Pedersen [47]  ✓  ✗  ✓  ✓  ✗  ✗  ✗  ✗  ✗  ✗  ✗  ✗  ✓  ✗  ✓  Regel [48]  ✓  ✗  ✓  ✓  ✗  ✗  ✗  ✗  ✓  ✗  ✓  ✗  ✗  ✓  ✗  Rognås [49]  ✓  ✗  ✓  ✓  ✓  ✓  ✗  ✓  ✓  ✗  ✓  ✗  ✗  ✗  ✗  Rognås [50]  ✓  ✗  ✓  ✓  ✓  ✗  ✗  Partly  ✓  ✗  ✓  ✗  ✗  ✗  ✗  Sollid [51]  ✓  ✗  ✓  ✓  ✓  ✗  ✗  ✗  ✓  ✗  ✓  ✗  ✗  ✗  ✗  Van der Velden [52]  ✓  ✗  ✓  ✓  ✗  ✗  ✗  ✓  ✓  ✗  ✓  ✗  ✗  ✗  ✗  Viergutz [53]  ✓  ✗  ✓  ✓  ✗  ✗  ✗  ✗  ✓  ✗  ✓  ✗  ✗  ✓  ✗  Von Knobelsdorff [54]  ✓  ✗  ✓  ✓  ✗  ✗  ✗  ✗  ✗  ✗  ✓  ✓  ✗  ✓  ✗  Weltermann [55]  ✓  ✗  ✓  ✓  ✗  ✗  ✗  ✗  ✗  ✗  ✓  ✗  ✗  ✓  ✗  Schlechtriemen [56]  ✓  ✗  ✗  ✗  ✗  ✗  ✗  ✗  ✗  ✗  ✓  ✗  ✗  ✓  ✓  ✓ = yes; ✗ = no. Table 2 Data extraction of included articles   Are the following fixed-system variables reported?  Quality indicators used  Multiple, mixed quality indicators  Tracer conditions used    Transportation mode  SAR-capability  Rural and/or urban  Response types  Speciality of physician  Physician experience  Funding  Description of dispatch system  Other fixed-system variables  Structure  Process  Outcome  Patient satisfaction      Akin Paker [30]  ✓  ✗  ✗  ✓  ✗  ✗  ✗  ✗  ✗  ✗  ✓  ✗  ✗  ✓  ✗  Anadelic [31]  ✓  ✗  ✓  Partly  ✗  ✗  ✗  ✓  ✓  ✓  ✓  ✗  ✗  ✓  ✗  Arntz [32]  ✓  ✗  ✓  ✓  ✓  Partly  ✓  ✓  ✓  ✗  ✓  ✗  ✗  ✓  ✗  Breckwoldt [33]  ✓  ✗  ✓  ✗  ✓  ✓  ✗  ✗  ✓  ✗  ✓  ✓  ✗  ✗  ✗  Duchateau [34]  ✓  ✗  ✓  ✗  ✗  ✗  ✗  ✗  ✓  ✗  ✓  ✗  ✗  ✗  ✗  Fjaeldstad [35]  ✓  ✗  ✓  ✓  ✓  ✗  ✓  ✓  ✓  ✗  ✓  ✗  ✗  ✗  ✗  Flabouris [36]  ✓  ✓  ✗  ✓  ✓  ✓  ✗  ✓  ✓  ✗  ✓  ✗  ✗  ✓  ✗  Helm [37]  ✓  ✗  ✗  ✗  ✗  ✗  ✗  ✗  ✗  ✗  ✓  ✗  ✗  ✗  ✗  Helm [38]  ✓  ✗  ✗  ✗  ✓  ✗  ✗  ✗  ✗  ✗  ✓  ✗  ✗  ✗  ✗  Hennes [39]  ✓  ✗  ✓  ✓  ✗  ✗  ✗  ✗  ✗  ✗  ✗  ✓  ✗  ✗  ✗  Klemenc-Ketis [40]  ✓  ✓  ✓  ✓  ✗  ✗  ✓  ✓  ✓  ✗  ✓  ✗  ✗  ✓  ✗  Leicht [41]  ✓  ✗  ✗  ✓  ✓  ✓  ✗  ✗  ✓  ✗  ✓  ✗  ✗  ✗  ✗  Lossius [42]  ✓  ✗  ✓  ✓  ✓  ✓  ✗  Partly  ✗  ✗  ✗  ✓  ✗  ✗  ✗  Messelken [43]  ✓  ✗  ✓  ✓  ✓  ✗  ✗  ✗  ✗  ✗  ✓  ✓  ✗  ✓  ✓  Mikkelsen [44]  ✓  ✗  ✓  ✓  ✓  ✓  ✗  ✓  ✓  ✗  ✓  ✓  ✗  ✗  ✗  Nakstad [45]  ✓  ✗  ✓  ✓  ✓  ✓  ✗  ✗  ✓  ✗  ✓  ✗  ✗  ✗  ✗  Neukamm [46]  ✓  ✗  ✓  ✓  ✗  ✗  ✗  Partly  ✓  ✓  ✓  ✓  ✗  ✓  ✗  Pedersen [47]  ✓  ✗  ✓  ✓  ✗  ✗  ✗  ✗  ✗  ✗  ✗  ✗  ✓  ✗  ✓  Regel [48]  ✓  ✗  ✓  ✓  ✗  ✗  ✗  ✗  ✓  ✗  ✓  ✗  ✗  ✓  ✗  Rognås [49]  ✓  ✗  ✓  ✓  ✓  ✓  ✗  ✓  ✓  ✗  ✓  ✗  ✗  ✗  ✗  Rognås [50]  ✓  ✗  ✓  ✓  ✓  ✗  ✗  Partly  ✓  ✗  ✓  ✗  ✗  ✗  ✗  Sollid [51]  ✓  ✗  ✓  ✓  ✓  ✗  ✗  ✗  ✓  ✗  ✓  ✗  ✗  ✗  ✗  Van der Velden [52]  ✓  ✗  ✓  ✓  ✗  ✗  ✗  ✓  ✓  ✗  ✓  ✗  ✗  ✗  ✗  Viergutz [53]  ✓  ✗  ✓  ✓  ✗  ✗  ✗  ✗  ✓  ✗  ✓  ✗  ✗  ✓  ✗  Von Knobelsdorff [54]  ✓  ✗  ✓  ✓  ✗  ✗  ✗  ✗  ✗  ✗  ✓  ✓  ✗  ✓  ✗  Weltermann [55]  ✓  ✗  ✓  ✓  ✗  ✗  ✗  ✗  ✗  ✗  ✓  ✗  ✗  ✓  ✗  Schlechtriemen [56]  ✓  ✗  ✗  ✗  ✗  ✗  ✗  ✗  ✗  ✗  ✓  ✗  ✗  ✓  ✓    Are the following fixed-system variables reported?  Quality indicators used  Multiple, mixed quality indicators  Tracer conditions used    Transportation mode  SAR-capability  Rural and/or urban  Response types  Speciality of physician  Physician experience  Funding  Description of dispatch system  Other fixed-system variables  Structure  Process  Outcome  Patient satisfaction      Akin Paker [30]  ✓  ✗  ✗  ✓  ✗  ✗  ✗  ✗  ✗  ✗  ✓  ✗  ✗  ✓  ✗  Anadelic [31]  ✓  ✗  ✓  Partly  ✗  ✗  ✗  ✓  ✓  ✓  ✓  ✗  ✗  ✓  ✗  Arntz [32]  ✓  ✗  ✓  ✓  ✓  Partly  ✓  ✓  ✓  ✗  ✓  ✗  ✗  ✓  ✗  Breckwoldt [33]  ✓  ✗  ✓  ✗  ✓  ✓  ✗  ✗  ✓  ✗  ✓  ✓  ✗  ✗  ✗  Duchateau [34]  ✓  ✗  ✓  ✗  ✗  ✗  ✗  ✗  ✓  ✗  ✓  ✗  ✗  ✗  ✗  Fjaeldstad [35]  ✓  ✗  ✓  ✓  ✓  ✗  ✓  ✓  ✓  ✗  ✓  ✗  ✗  ✗  ✗  Flabouris [36]  ✓  ✓  ✗  ✓  ✓  ✓  ✗  ✓  ✓  ✗  ✓  ✗  ✗  ✓  ✗  Helm [37]  ✓  ✗  ✗  ✗  ✗  ✗  ✗  ✗  ✗  ✗  ✓  ✗  ✗  ✗  ✗  Helm [38]  ✓  ✗  ✗  ✗  ✓  ✗  ✗  ✗  ✗  ✗  ✓  ✗  ✗  ✗  ✗  Hennes [39]  ✓  ✗  ✓  ✓  ✗  ✗  ✗  ✗  ✗  ✗  ✗  ✓  ✗  ✗  ✗  Klemenc-Ketis [40]  ✓  ✓  ✓  ✓  ✗  ✗  ✓  ✓  ✓  ✗  ✓  ✗  ✗  ✓  ✗  Leicht [41]  ✓  ✗  ✗  ✓  ✓  ✓  ✗  ✗  ✓  ✗  ✓  ✗  ✗  ✗  ✗  Lossius [42]  ✓  ✗  ✓  ✓  ✓  ✓  ✗  Partly  ✗  ✗  ✗  ✓  ✗  ✗  ✗  Messelken [43]  ✓  ✗  ✓  ✓  ✓  ✗  ✗  ✗  ✗  ✗  ✓  ✓  ✗  ✓  ✓  Mikkelsen [44]  ✓  ✗  ✓  ✓  ✓  ✓  ✗  ✓  ✓  ✗  ✓  ✓  ✗  ✗  ✗  Nakstad [45]  ✓  ✗  ✓  ✓  ✓  ✓  ✗  ✗  ✓  ✗  ✓  ✗  ✗  ✗  ✗  Neukamm [46]  ✓  ✗  ✓  ✓  ✗  ✗  ✗  Partly  ✓  ✓  ✓  ✓  ✗  ✓  ✗  Pedersen [47]  ✓  ✗  ✓  ✓  ✗  ✗  ✗  ✗  ✗  ✗  ✗  ✗  ✓  ✗  ✓  Regel [48]  ✓  ✗  ✓  ✓  ✗  ✗  ✗  ✗  ✓  ✗  ✓  ✗  ✗  ✓  ✗  Rognås [49]  ✓  ✗  ✓  ✓  ✓  ✓  ✗  ✓  ✓  ✗  ✓  ✗  ✗  ✗  ✗  Rognås [50]  ✓  ✗  ✓  ✓  ✓  ✗  ✗  Partly  ✓  ✗  ✓  ✗  ✗  ✗  ✗  Sollid [51]  ✓  ✗  ✓  ✓  ✓  ✗  ✗  ✗  ✓  ✗  ✓  ✗  ✗  ✗  ✗  Van der Velden [52]  ✓  ✗  ✓  ✓  ✗  ✗  ✗  ✓  ✓  ✗  ✓  ✗  ✗  ✗  ✗  Viergutz [53]  ✓  ✗  ✓  ✓  ✗  ✗  ✗  ✗  ✓  ✗  ✓  ✗  ✗  ✓  ✗  Von Knobelsdorff [54]  ✓  ✗  ✓  ✓  ✗  ✗  ✗  ✗  ✗  ✗  ✓  ✓  ✗  ✓  ✗  Weltermann [55]  ✓  ✗  ✓  ✓  ✗  ✗  ✗  ✗  ✗  ✗  ✓  ✗  ✗  ✓  ✗  Schlechtriemen [56]  ✓  ✗  ✗  ✗  ✗  ✗  ✗  ✗  ✗  ✗  ✓  ✗  ✗  ✓  ✓  ✓ = yes; ✗ = no. Table 3 Quality indicators used in the included literature Quality indicator  Category  No. of papers it is used in  Adherence to medical protocols  Process  11  Provision of advanced interventions  Process  8  Response time  Process  7  Adverse events  Process  7  Medication administration  Process  5  Transport to appropriate facility  Process  4  Time on scene  Process  3  Improved care due to clinical decision making  Process  3  Reliability of the primary diagnosis made by the P-EMS physician  Process  3  Survival  Outcome  3  ROSC in cardiac arrest  Outcome  3  Time from alarm to patient handover  Process  2  Time from arrival at patient until hospital admission  Process  2  Altered physiology  Outcome  2  Pain management  Outcome  2  Proportion of intubated patients adequately oxygenated and ventilated  Process  2  Time gain by air transportation  Process  1  The number of patients with a NACA-score ≥ 4 with an intravenous line  Process  1  The proportion of patients successfully intubated  Process  1  Life years gained  Outcome  1  Morbidity/disability  Outcome  1  Amount of yearly CPR training  Structure  1  Precision of dispatch  Process  1  Rate of CPR started within 8 min of the call to the dispatch center  Process  1  Patient satisfaction  Outcome  1  Quality indicator  Category  No. of papers it is used in  Adherence to medical protocols  Process  11  Provision of advanced interventions  Process  8  Response time  Process  7  Adverse events  Process  7  Medication administration  Process  5  Transport to appropriate facility  Process  4  Time on scene  Process  3  Improved care due to clinical decision making  Process  3  Reliability of the primary diagnosis made by the P-EMS physician  Process  3  Survival  Outcome  3  ROSC in cardiac arrest  Outcome  3  Time from alarm to patient handover  Process  2  Time from arrival at patient until hospital admission  Process  2  Altered physiology  Outcome  2  Pain management  Outcome  2  Proportion of intubated patients adequately oxygenated and ventilated  Process  2  Time gain by air transportation  Process  1  The number of patients with a NACA-score ≥ 4 with an intravenous line  Process  1  The proportion of patients successfully intubated  Process  1  Life years gained  Outcome  1  Morbidity/disability  Outcome  1  Amount of yearly CPR training  Structure  1  Precision of dispatch  Process  1  Rate of CPR started within 8 min of the call to the dispatch center  Process  1  Patient satisfaction  Outcome  1  ROSC, return of spontaneous circulation; NACA, National Advisory Committee for Aeronautics (see Additional file 2); CPR, cardiopulmonary resuscitation. Table 3 Quality indicators used in the included literature Quality indicator  Category  No. of papers it is used in  Adherence to medical protocols  Process  11  Provision of advanced interventions  Process  8  Response time  Process  7  Adverse events  Process  7  Medication administration  Process  5  Transport to appropriate facility  Process  4  Time on scene  Process  3  Improved care due to clinical decision making  Process  3  Reliability of the primary diagnosis made by the P-EMS physician  Process  3  Survival  Outcome  3  ROSC in cardiac arrest  Outcome  3  Time from alarm to patient handover  Process  2  Time from arrival at patient until hospital admission  Process  2  Altered physiology  Outcome  2  Pain management  Outcome  2  Proportion of intubated patients adequately oxygenated and ventilated  Process  2  Time gain by air transportation  Process  1  The number of patients with a NACA-score ≥ 4 with an intravenous line  Process  1  The proportion of patients successfully intubated  Process  1  Life years gained  Outcome  1  Morbidity/disability  Outcome  1  Amount of yearly CPR training  Structure  1  Precision of dispatch  Process  1  Rate of CPR started within 8 min of the call to the dispatch center  Process  1  Patient satisfaction  Outcome  1  Quality indicator  Category  No. of papers it is used in  Adherence to medical protocols  Process  11  Provision of advanced interventions  Process  8  Response time  Process  7  Adverse events  Process  7  Medication administration  Process  5  Transport to appropriate facility  Process  4  Time on scene  Process  3  Improved care due to clinical decision making  Process  3  Reliability of the primary diagnosis made by the P-EMS physician  Process  3  Survival  Outcome  3  ROSC in cardiac arrest  Outcome  3  Time from alarm to patient handover  Process  2  Time from arrival at patient until hospital admission  Process  2  Altered physiology  Outcome  2  Pain management  Outcome  2  Proportion of intubated patients adequately oxygenated and ventilated  Process  2  Time gain by air transportation  Process  1  The number of patients with a NACA-score ≥ 4 with an intravenous line  Process  1  The proportion of patients successfully intubated  Process  1  Life years gained  Outcome  1  Morbidity/disability  Outcome  1  Amount of yearly CPR training  Structure  1  Precision of dispatch  Process  1  Rate of CPR started within 8 min of the call to the dispatch center  Process  1  Patient satisfaction  Outcome  1  ROSC, return of spontaneous circulation; NACA, National Advisory Committee for Aeronautics (see Additional file 2); CPR, cardiopulmonary resuscitation. Table 4 Quality appraisal of included articles   Internal validity  External validity  Was the methodology for developing the QIs clearly explained?  Are the QIs clearly defined?  Is the rationale for the QIs sufficiently described?  Is handling of missing data sufficiently described?  Do the authors address conflicts of interest?  Has an ethics committee approved the study?  Is professional background and funding of those involved in QI development stated?  Were the QIs developed by the systematic work of a group of experts?  Is it reported in which country/organization the QIs were developed?  Is (are) the P-EMS (s) involved in quality measurement sufficiently described?  Are the QIs transferable to other countries or P-EMSs?  Are limitations of the study discussed?  Are possible sources of bias discussed?  Have the QIs been used in other publications?  Was the feasibility of the QIs evaluated?  Akin Paker [30]  ✗  ✗  ✗  ✓  ✗  ✓  ✗  ?  ✓  ✗  ✓  ✓  ✓  ?  ✓  Anadelic [31]  ✓  ✓  ✓  ✗  ✗  ?  ✓  ?  ✓  ✗  ✓  ✗  ✗  ?  ✓  Arntz [32]  ✓  ✗  ✓  ✓  ✗  ?  ✗  ?  ✓  ✓  ✓  ✗  ✗  ?  ✗  Breckwoldt [33]  ✓  ✗  ✓  ✗  ✓  ?  ✗  ?  ✓  ✗  ✓  ✗  ✗  ?  ✗  Duchateau [34]  ✓  ✓  ✗  ✗  ✓  ?  ✗  ?  ✓  ✗  ✓  ✗  ✗  ✓  ✗  Fjaeldstad [35]  ✗  ✓  ✗  ✓  ✓  ?  ✗  ?  ✓  ✗  ✓  ✓  ✗  ?  ✗  Flabouris [36]  ✓  ✓  ✓  ✓  ✗  ?  ✗  ?  ✓  ✓  ✓  ✓  ✓  ?  ✗  Helm [37]  ✓  ✓  ✓  ✗  ✓  ✓  ✗  ✗  ✓  ✗  ✓  ✓  ✗  ✓  ✗  Helm [38]  ✗  ✓  ✓  ✗  ✗  ?  ✗  ✗  ✓  ✗  ✓  ✓  ✗  ?  ✗  Hennes [39]  ✓  ✓  ✓  ✗  ✗  ?  ✗  ?  ✓  ✗  ✓  ✓  ✓  ✓  ✓  Klemenc-Ketis [40]  ✗  ✓  ✓  ✗  ✗  ✓  ✗  ✗  ✓  ✓  ✓  ✓  ✓  ?  ✗  Leicht [41]  ✗  ✓  ✓  ✗  ✗  ?  ✗  ✗  ✓  ✗  ✓  ✓  ✓  ?  ✗  Lossius [42]  ✓  ✓  ✓  ✗  ✗  ✓  ✓  ?  ✓  ✓  ✓  ✓  ✓  ✓  ✗  Messelken [43]  ✓  ✓  ✓  ✓  ✓  ?  ✗  ?  ✓  ✗  ✓  ✓  ✗  ✓  ✗  Mikkelsen [44]  ✓  ✓  ✓  ✓  ✓  ?  ✓  ✗  ✓  ✓  ✓  ✓  ✓  ?  ✗  Nakstad [45]  ✗  ✓  ✓  ✓  ✗  ✓  ✗  ?  ✓  ✓  ✓  ✗  ✓  ✓  ✗  Neukamm [46]  ✗  ✓  ✗  ✗  ✓  ✓  ✗  ?  ✓  ✓  ✓  ✓  ✓  ?  ✗  Pedersen [47]  ✗  ✓  ✓  ✗  ✓  ✓  ✗  ✗  ✓  ✗  ✓  ✓  ✓  ✗  ✗  Regel [48]  ✓  ✓  ✓  ✗  ✗  ?  ✗  ?  ✓  ✗  ✓  ✓  ✓  ✗  ✗  Rognås [49]  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✗  ✗  Rognås [50]  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✗  Sollid [51]  ✗  ✗  ✓  ✗  ✓  ✓  ✓  ✗  ✓  ✓  ✓  ✓  ✓  ?  ✗  Van der Velden [52]  ✓  ✓  ✓  ✗  ✓  ?  ✓  ✗  ✓  ✗  ✓  ✓  ✓  ?  ✗  Viergutz [53]  ✓  ✓  ✓  ✓  ✓  ✓  ✗  ✗  ✓  ✗  ✓  ✓  ✓  ?  ✗  Von Knobelsdorff [54]  ✗  ✓  ✓  ✓  ✗  ?  ✗  ?  ✓  ✗  ✓  ✗  ✓  ?  ✗  Weltermann [55]  ✓  ✓  ✓  ✗  ✗  ?  ✗  ?  ✓  ✗  ✓  ✓  ✗  ✓  ✗  Schlechtriemen [56]  ✓  ✓  ✓  –  ✗  –  ✗  ?  ✓  –  ✓  ✓  ✓  ?  ✗    Internal validity  External validity  Was the methodology for developing the QIs clearly explained?  Are the QIs clearly defined?  Is the rationale for the QIs sufficiently described?  Is handling of missing data sufficiently described?  Do the authors address conflicts of interest?  Has an ethics committee approved the study?  Is professional background and funding of those involved in QI development stated?  Were the QIs developed by the systematic work of a group of experts?  Is it reported in which country/organization the QIs were developed?  Is (are) the P-EMS (s) involved in quality measurement sufficiently described?  Are the QIs transferable to other countries or P-EMSs?  Are limitations of the study discussed?  Are possible sources of bias discussed?  Have the QIs been used in other publications?  Was the feasibility of the QIs evaluated?  Akin Paker [30]  ✗  ✗  ✗  ✓  ✗  ✓  ✗  ?  ✓  ✗  ✓  ✓  ✓  ?  ✓  Anadelic [31]  ✓  ✓  ✓  ✗  ✗  ?  ✓  ?  ✓  ✗  ✓  ✗  ✗  ?  ✓  Arntz [32]  ✓  ✗  ✓  ✓  ✗  ?  ✗  ?  ✓  ✓  ✓  ✗  ✗  ?  ✗  Breckwoldt [33]  ✓  ✗  ✓  ✗  ✓  ?  ✗  ?  ✓  ✗  ✓  ✗  ✗  ?  ✗  Duchateau [34]  ✓  ✓  ✗  ✗  ✓  ?  ✗  ?  ✓  ✗  ✓  ✗  ✗  ✓  ✗  Fjaeldstad [35]  ✗  ✓  ✗  ✓  ✓  ?  ✗  ?  ✓  ✗  ✓  ✓  ✗  ?  ✗  Flabouris [36]  ✓  ✓  ✓  ✓  ✗  ?  ✗  ?  ✓  ✓  ✓  ✓  ✓  ?  ✗  Helm [37]  ✓  ✓  ✓  ✗  ✓  ✓  ✗  ✗  ✓  ✗  ✓  ✓  ✗  ✓  ✗  Helm [38]  ✗  ✓  ✓  ✗  ✗  ?  ✗  ✗  ✓  ✗  ✓  ✓  ✗  ?  ✗  Hennes [39]  ✓  ✓  ✓  ✗  ✗  ?  ✗  ?  ✓  ✗  ✓  ✓  ✓  ✓  ✓  Klemenc-Ketis [40]  ✗  ✓  ✓  ✗  ✗  ✓  ✗  ✗  ✓  ✓  ✓  ✓  ✓  ?  ✗  Leicht [41]  ✗  ✓  ✓  ✗  ✗  ?  ✗  ✗  ✓  ✗  ✓  ✓  ✓  ?  ✗  Lossius [42]  ✓  ✓  ✓  ✗  ✗  ✓  ✓  ?  ✓  ✓  ✓  ✓  ✓  ✓  ✗  Messelken [43]  ✓  ✓  ✓  ✓  ✓  ?  ✗  ?  ✓  ✗  ✓  ✓  ✗  ✓  ✗  Mikkelsen [44]  ✓  ✓  ✓  ✓  ✓  ?  ✓  ✗  ✓  ✓  ✓  ✓  ✓  ?  ✗  Nakstad [45]  ✗  ✓  ✓  ✓  ✗  ✓  ✗  ?  ✓  ✓  ✓  ✗  ✓  ✓  ✗  Neukamm [46]  ✗  ✓  ✗  ✗  ✓  ✓  ✗  ?  ✓  ✓  ✓  ✓  ✓  ?  ✗  Pedersen [47]  ✗  ✓  ✓  ✗  ✓  ✓  ✗  ✗  ✓  ✗  ✓  ✓  ✓  ✗  ✗  Regel [48]  ✓  ✓  ✓  ✗  ✗  ?  ✗  ?  ✓  ✗  ✓  ✓  ✓  ✗  ✗  Rognås [49]  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✗  ✗  Rognås [50]  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✗  Sollid [51]  ✗  ✗  ✓  ✗  ✓  ✓  ✓  ✗  ✓  ✓  ✓  ✓  ✓  ?  ✗  Van der Velden [52]  ✓  ✓  ✓  ✗  ✓  ?  ✓  ✗  ✓  ✗  ✓  ✓  ✓  ?  ✗  Viergutz [53]  ✓  ✓  ✓  ✓  ✓  ✓  ✗  ✗  ✓  ✗  ✓  ✓  ✓  ?  ✗  Von Knobelsdorff [54]  ✗  ✓  ✓  ✓  ✗  ?  ✗  ?  ✓  ✗  ✓  ✗  ✓  ?  ✗  Weltermann [55]  ✓  ✓  ✓  ✗  ✗  ?  ✗  ?  ✓  ✗  ✓  ✓  ✗  ✓  ✗  Schlechtriemen [56]  ✓  ✓  ✓  –  ✗  –  ✗  ?  ✓  –  ✓  ✓  ✓  ?  ✗  ✓ = yes; ✗ = no; ? = uncertain; – = not relevant. Table 4 Quality appraisal of included articles   Internal validity  External validity  Was the methodology for developing the QIs clearly explained?  Are the QIs clearly defined?  Is the rationale for the QIs sufficiently described?  Is handling of missing data sufficiently described?  Do the authors address conflicts of interest?  Has an ethics committee approved the study?  Is professional background and funding of those involved in QI development stated?  Were the QIs developed by the systematic work of a group of experts?  Is it reported in which country/organization the QIs were developed?  Is (are) the P-EMS (s) involved in quality measurement sufficiently described?  Are the QIs transferable to other countries or P-EMSs?  Are limitations of the study discussed?  Are possible sources of bias discussed?  Have the QIs been used in other publications?  Was the feasibility of the QIs evaluated?  Akin Paker [30]  ✗  ✗  ✗  ✓  ✗  ✓  ✗  ?  ✓  ✗  ✓  ✓  ✓  ?  ✓  Anadelic [31]  ✓  ✓  ✓  ✗  ✗  ?  ✓  ?  ✓  ✗  ✓  ✗  ✗  ?  ✓  Arntz [32]  ✓  ✗  ✓  ✓  ✗  ?  ✗  ?  ✓  ✓  ✓  ✗  ✗  ?  ✗  Breckwoldt [33]  ✓  ✗  ✓  ✗  ✓  ?  ✗  ?  ✓  ✗  ✓  ✗  ✗  ?  ✗  Duchateau [34]  ✓  ✓  ✗  ✗  ✓  ?  ✗  ?  ✓  ✗  ✓  ✗  ✗  ✓  ✗  Fjaeldstad [35]  ✗  ✓  ✗  ✓  ✓  ?  ✗  ?  ✓  ✗  ✓  ✓  ✗  ?  ✗  Flabouris [36]  ✓  ✓  ✓  ✓  ✗  ?  ✗  ?  ✓  ✓  ✓  ✓  ✓  ?  ✗  Helm [37]  ✓  ✓  ✓  ✗  ✓  ✓  ✗  ✗  ✓  ✗  ✓  ✓  ✗  ✓  ✗  Helm [38]  ✗  ✓  ✓  ✗  ✗  ?  ✗  ✗  ✓  ✗  ✓  ✓  ✗  ?  ✗  Hennes [39]  ✓  ✓  ✓  ✗  ✗  ?  ✗  ?  ✓  ✗  ✓  ✓  ✓  ✓  ✓  Klemenc-Ketis [40]  ✗  ✓  ✓  ✗  ✗  ✓  ✗  ✗  ✓  ✓  ✓  ✓  ✓  ?  ✗  Leicht [41]  ✗  ✓  ✓  ✗  ✗  ?  ✗  ✗  ✓  ✗  ✓  ✓  ✓  ?  ✗  Lossius [42]  ✓  ✓  ✓  ✗  ✗  ✓  ✓  ?  ✓  ✓  ✓  ✓  ✓  ✓  ✗  Messelken [43]  ✓  ✓  ✓  ✓  ✓  ?  ✗  ?  ✓  ✗  ✓  ✓  ✗  ✓  ✗  Mikkelsen [44]  ✓  ✓  ✓  ✓  ✓  ?  ✓  ✗  ✓  ✓  ✓  ✓  ✓  ?  ✗  Nakstad [45]  ✗  ✓  ✓  ✓  ✗  ✓  ✗  ?  ✓  ✓  ✓  ✗  ✓  ✓  ✗  Neukamm [46]  ✗  ✓  ✗  ✗  ✓  ✓  ✗  ?  ✓  ✓  ✓  ✓  ✓  ?  ✗  Pedersen [47]  ✗  ✓  ✓  ✗  ✓  ✓  ✗  ✗  ✓  ✗  ✓  ✓  ✓  ✗  ✗  Regel [48]  ✓  ✓  ✓  ✗  ✗  ?  ✗  ?  ✓  ✗  ✓  ✓  ✓  ✗  ✗  Rognås [49]  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✗  ✗  Rognås [50]  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✗  Sollid [51]  ✗  ✗  ✓  ✗  ✓  ✓  ✓  ✗  ✓  ✓  ✓  ✓  ✓  ?  ✗  Van der Velden [52]  ✓  ✓  ✓  ✗  ✓  ?  ✓  ✗  ✓  ✗  ✓  ✓  ✓  ?  ✗  Viergutz [53]  ✓  ✓  ✓  ✓  ✓  ✓  ✗  ✗  ✓  ✗  ✓  ✓  ✓  ?  ✗  Von Knobelsdorff [54]  ✗  ✓  ✓  ✓  ✗  ?  ✗  ?  ✓  ✗  ✓  ✗  ✓  ?  ✗  Weltermann [55]  ✓  ✓  ✓  ✗  ✗  ?  ✗  ?  ✓  ✗  ✓  ✓  ✗  ✓  ✗  Schlechtriemen [56]  ✓  ✓  ✓  –  ✗  –  ✗  ?  ✓  –  ✓  ✓  ✓  ?  ✗    Internal validity  External validity  Was the methodology for developing the QIs clearly explained?  Are the QIs clearly defined?  Is the rationale for the QIs sufficiently described?  Is handling of missing data sufficiently described?  Do the authors address conflicts of interest?  Has an ethics committee approved the study?  Is professional background and funding of those involved in QI development stated?  Were the QIs developed by the systematic work of a group of experts?  Is it reported in which country/organization the QIs were developed?  Is (are) the P-EMS (s) involved in quality measurement sufficiently described?  Are the QIs transferable to other countries or P-EMSs?  Are limitations of the study discussed?  Are possible sources of bias discussed?  Have the QIs been used in other publications?  Was the feasibility of the QIs evaluated?  Akin Paker [30]  ✗  ✗  ✗  ✓  ✗  ✓  ✗  ?  ✓  ✗  ✓  ✓  ✓  ?  ✓  Anadelic [31]  ✓  ✓  ✓  ✗  ✗  ?  ✓  ?  ✓  ✗  ✓  ✗  ✗  ?  ✓  Arntz [32]  ✓  ✗  ✓  ✓  ✗  ?  ✗  ?  ✓  ✓  ✓  ✗  ✗  ?  ✗  Breckwoldt [33]  ✓  ✗  ✓  ✗  ✓  ?  ✗  ?  ✓  ✗  ✓  ✗  ✗  ?  ✗  Duchateau [34]  ✓  ✓  ✗  ✗  ✓  ?  ✗  ?  ✓  ✗  ✓  ✗  ✗  ✓  ✗  Fjaeldstad [35]  ✗  ✓  ✗  ✓  ✓  ?  ✗  ?  ✓  ✗  ✓  ✓  ✗  ?  ✗  Flabouris [36]  ✓  ✓  ✓  ✓  ✗  ?  ✗  ?  ✓  ✓  ✓  ✓  ✓  ?  ✗  Helm [37]  ✓  ✓  ✓  ✗  ✓  ✓  ✗  ✗  ✓  ✗  ✓  ✓  ✗  ✓  ✗  Helm [38]  ✗  ✓  ✓  ✗  ✗  ?  ✗  ✗  ✓  ✗  ✓  ✓  ✗  ?  ✗  Hennes [39]  ✓  ✓  ✓  ✗  ✗  ?  ✗  ?  ✓  ✗  ✓  ✓  ✓  ✓  ✓  Klemenc-Ketis [40]  ✗  ✓  ✓  ✗  ✗  ✓  ✗  ✗  ✓  ✓  ✓  ✓  ✓  ?  ✗  Leicht [41]  ✗  ✓  ✓  ✗  ✗  ?  ✗  ✗  ✓  ✗  ✓  ✓  ✓  ?  ✗  Lossius [42]  ✓  ✓  ✓  ✗  ✗  ✓  ✓  ?  ✓  ✓  ✓  ✓  ✓  ✓  ✗  Messelken [43]  ✓  ✓  ✓  ✓  ✓  ?  ✗  ?  ✓  ✗  ✓  ✓  ✗  ✓  ✗  Mikkelsen [44]  ✓  ✓  ✓  ✓  ✓  ?  ✓  ✗  ✓  ✓  ✓  ✓  ✓  ?  ✗  Nakstad [45]  ✗  ✓  ✓  ✓  ✗  ✓  ✗  ?  ✓  ✓  ✓  ✗  ✓  ✓  ✗  Neukamm [46]  ✗  ✓  ✗  ✗  ✓  ✓  ✗  ?  ✓  ✓  ✓  ✓  ✓  ?  ✗  Pedersen [47]  ✗  ✓  ✓  ✗  ✓  ✓  ✗  ✗  ✓  ✗  ✓  ✓  ✓  ✗  ✗  Regel [48]  ✓  ✓  ✓  ✗  ✗  ?  ✗  ?  ✓  ✗  ✓  ✓  ✓  ✗  ✗  Rognås [49]  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✗  ✗  Rognås [50]  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✗  Sollid [51]  ✗  ✗  ✓  ✗  ✓  ✓  ✓  ✗  ✓  ✓  ✓  ✓  ✓  ?  ✗  Van der Velden [52]  ✓  ✓  ✓  ✗  ✓  ?  ✓  ✗  ✓  ✗  ✓  ✓  ✓  ?  ✗  Viergutz [53]  ✓  ✓  ✓  ✓  ✓  ✓  ✗  ✗  ✓  ✗  ✓  ✓  ✓  ?  ✗  Von Knobelsdorff [54]  ✗  ✓  ✓  ✓  ✗  ?  ✗  ?  ✓  ✗  ✓  ✗  ✓  ?  ✗  Weltermann [55]  ✓  ✓  ✓  ✗  ✗  ?  ✗  ?  ✓  ✗  ✓  ✓  ✗  ✓  ✗  Schlechtriemen [56]  ✓  ✓  ✓  –  ✗  –  ✗  ?  ✓  –  ✓  ✓  ✓  ?  ✗  ✓ = yes; ✗ = no; ? = uncertain; – = not relevant. Discussion This systematic review identified 27 papers that reported the use of QIs in P-EMS. Fifteen papers used one single QI, and twelve papers applied a set of QIs. Twenty-four of the 27 papers used QIs pertaining to process. Generally, the systematic review demonstrated a lack of a shared understanding of which QI to use in P-EMS. The lack of agreement on QIs has also been described for regular EMS [57, 58]. This lack of consensus on QI, the heterogeneity of diagnosis, and the challenge of isolating the effect of pre-hospital care from the effect of in-hospital care complicates quality measurement in pre-hospital emergency medicine [59, 60]. The characteristics of the P-EMSs in the identified literature vary and are described sufficiently in only ten papers. This complicates the comparison of studies because the concept of involved P-EMSs remains unclear. Five services use helicopters, eleven use rapid response cars, and another eleven use both transportation modes. Further, nine services are urban, one is rural and eleven are both urban and rural. This mixed representation of transportation modes and urban versus rural profile seems to reflect the heterogeneity of P-EMS. Another relevant aspect of P-EMS is the specialty training and competency of the staffing physicians. Fourteen papers do not report the physicians’ medical specialty. Emergency medicine and anesthesiology are reported as the physicians’ specialties in one and seven papers, respectively. Five studies report a mix of medical specialties in the actual services. All mixes are different, with anesthesiology as the only medical specialty represented in all five papers. Regarding country, eleven of the 27 papers are German. Germany has a long history of P-EMS as an integrated and natural part of the emergency medical system. However, the considerable contribution of German papers is not only because of the service’s long existence. Quality measurement in German hospitals has developed substantially during the last two decades [61], and gradually, German P-EMS has adopted this quality measurement initiative. In addition to a formalized and common understanding of the need for quality measurement in P-EMS [62], the establishment of common documentation systems seems to have been the first necessary step towards quality measurement in German P-EMS and might set an example for QI work in P-EMS in other countries [63, 64]. In 23 papers, the QIs are well defined. However, for many of the QIs identified in this systematic review, the development process does not seem optimal. Ten papers do not clearly describe the methodology for developing the QIs, and only in two papers were the QIs systematically developed by a group of experts. Finally, only seven papers report the professional background and funding of those involved in the development of QIs. A key point in the development of QIs is a systematical and objective approach; this allows for the assessment of the evidence base for the QIs and secures legitimacy. Inadequate QI development may influence the validity, reliability and feasibility of the QIs. Structure indicators and outcome indicators are used by only two and seven papers, respectively. This may be because process indicators are easier to collect. Routine use of outcome indicators will require automated data exchange between pre- and in-hospital databases, which might not be feasible. This calls for a more integrated electronic patient chart system covering patient data capture through all phases of care. Nevertheless, the major role of process indicators in quality measurement literature of P-EMS is as expected. Process indicators are considered useful for short time frames and when it is difficult to adjust for patient factors [65], and process indicators are therefore particularly relevant for P-EMS. Moreover, process indicators provide a direct assessment of quality of care, as opposed to structure and outcome indicators, which measure care quality by an indirect approach [66]. Process indicators are easy to interpret and well suited for the evaluation of adherence to medical protocols and other quality improvement programmes. As seen in Table 3, many of the process indicators are time variables. When setting targets and measuring EMS quality, time variables have been widely applied [17, 66]. EMS evolved as a response to the need for rapid access to healthcare in time critical conditions (war injuries, cardiac arrest, major trauma) [66, 67]. However, the current patient population for P-EMS is increasingly heterogeneous, including conditions that are not highly time critical. For a high proportion of patients, shorter pre-hospital time intervals do not improve outcome, and in some cases, longer on-scene time for initial treatment and stabilization is desirable [68, 69]. Accordingly, if P-EMS quality measurement focuses too much on time variables, the results will have poor relevance for a high proportion of the services’ patient population. Moreover, it can lead to an undesired attention shift, resulting in decreased quality for quality dimensions not measured. However, this undesired attention shift is not specific for time variables but rather a universal challenge in quality measurement when the used QIs are few. Finally, our findings indicate that literature pertaining to pre-hospital time variables often lacks information about the competency of the responding unit and the quality of provided care, resulting in even less information from the time variables measured. Adherence to medical protocols is the most frequently used QI in the included papers. Guidelines and protocols are developed to improve quality of care and to reduce unwanted variation in care. Quality measurement can be defined as measuring the extent to which set targets are achieved [19], and to explore the gap between guidelines and clinical practice is an adequate approach in quality measurement. Ebben et al. [70] demonstrated a wide variation in different EMS professionals’ adherence to guidelines and protocols, indicating that a substantial number of patients do not receive appropriate pre-hospital care. However, this conclusion presumes that there is an evident relationship between adherence to guidelines and patient outcome. Few studies have explored this relationship, and in the review by Ebben et al., only three studies showed that adherence to guidelines improved patient outcomes [71–73]. Finally, it is recommended that guidelines should define QIs to aid monitoring and assessment of guideline adherence [74, 75]. Thus, QIs should ideally be a part of the guideline development process. For the fifteen papers relying on only one single QI, there is a risk for narrowing the perspective on quality, and important aspects of quality in the actual healthcare service may be ignored. Using a set of mixed QIs that cover different aspects of the service is preferable [76]. Three papers evaluated the care quality of selected ‘tracer conditions’, i.e. high-priority clinical conditions [77]. The evaluation of a service’s response to tracer conditions with condition-specific QIs is used to predict the overall performance of P-EMS. The chosen tracer conditions are not identical, illustrating the following challenge: when different clinical conditions are used for quality measurement, it may complicate the evaluation and comparison of the quality of different P-EMS. To overcome this, the Institute for Healthcare Improvement (IHI) has recommended the use of ‘whole system measures’, defined as a set of QIs aligned with the Institute of Medicine’s (IOM’s) six dimensions of quality, which are not disease- or condition-specific [78, 79]. The six quality dimensions that define high-quality care are timeliness, safety, efficiency, equity, effectiveness and patient-centeredness. Each of these is distinct, and all are equally important. To obtain an adequate and comprehensive quality measurement of P-EMS, future quality measurement should therefore cover these six dimensions as far as possible. This will require the use of multiple QIs, developed specifically for P-EMS. A set of QIs for this purpose has been developed recently, with IOM’s six quality dimensions as the most important part of the conceptual framework [76]. Strengths and limitations We recognize some strengths and limitations of this study. A strength is that in the literature search, several languages were eligible, allowing inclusion of non-English literature. This is important as eleven of the 27 identified papers were non-English. First, a limitation is that as always in a systematic review, screening, eligibility-check and qualitative synthesis of literature is a product of the review authors’ judgements, allowing subjective interpretations of the content of the studies. Although screening and eligibility assessment were conducted in pairs, data extraction and quality appraisal were primarily conducted by one author. Moreover, no established tools for data extraction or quality appraisal were available. Second, some of the included studies are not explicitly presented as quality measurement studies. However, when screening the literature, we recognized that some studies are quality measurement studies despite not using the quality measurement terminology. A possible reason for this might be the fact that quality measurement terminology is still quite new in P-EMS and not widely used. Accordingly, we have focused on the actual content of the screened literature when deciding if a study should be included—not on the presence or absence of correct quality measurement terminology. Doing this, we have appraised if a paper ‘concerns an initiative to improve healthcare; broadly defined to include the quality, safety, effectiveness, patient-centeredness, timeliness, cost, efficiency and equity of healthcare’, as stated by the SQUIRE guidelines [80]. This approach was chosen to avoid overlooking potentially relevant aspects of quality measurement studies in P-EMS. Conclusion This systematic literature review served the purpose of identifying, describing and evaluating studies of quality measurement in P-EMS. The review demonstrated a lack of a shared understanding of which QIs to use in P-EMS. Process indicators were dominant in the included papers, and the most emphasized QIs were ‘Adherence to medical protocols’, ‘Provision of advanced interventions’, ‘Response time’ and ‘Adverse events’. Moreover, fifteen of the 27 papers used only a single QI to measure quality, thus increasing the risk of a narrow perspective on quality. The remaining papers used multiple QIs in their quality measurement, which is considered preferable. Future quality measurement in P-EMS should rely on a set of consensus-based QIs, securing a comprehensive approach to quality measurement and offering the possibility of comparing results from different P-EMS systems. Supplementary material Supplementary material is available at International Journal for Quality in Health Care online. Acknowledgements The authors would like to thank Mrs Ingrid Ingeborg Riphagen, Unit for Applied Clinical Research, Norwegian University of Science and Technology, for her competent and invaluable assistance in designing and conducting the literature search. We would also like to thank members and donors of the Norwegian Air Ambulance Foundation for their financial contribution to this study. Additional file Additional file 1: ‘Search strategy’. Additional file 2: ‘NACA-score; elaboration document’. Authors’ contribution H.H. and M.R. conceived the idea. All authors were part of the study design. H.H. and O.U. screened the identified literature. H.H., O.U., A.J.K. and M.R. considered the eligibility of uncertain literature. H.H. performed data extraction and quality appraisal of the included literature. All authors have approved the final version of the manuscript. Ethics approval and consent to participate Not applicable. References 1 Roessler M, Zuzan O. EMS systems in Germany. Resuscitation  2006; 68: 45– 9. doi:10.1016/j.resuscitation.2005.08.004. Google Scholar CrossRef Search ADS PubMed  2 Langhelle A, Lossius HM, Silfvast T et al.  . International EMS Systems: the Nordic countries. Resuscitation  2004; 61: 9– 21. doi:10.1016/j.resuscitation.2003.12.008. Google Scholar CrossRef Search ADS PubMed  3 Black JJ, Davies GD. International EMS systems: United Kingdom. Resuscitation  2005; 64: 21– 9. doi:10.1016/j.resuscitation.2004.10.004. Google Scholar CrossRef Search ADS PubMed  4 Adnet F, Lapostolle F. International EMS systems: France. Resuscitation  2004; 63: 7– 9. doi:10.1016/j.resuscitation.2004.04.001. Google Scholar CrossRef Search ADS PubMed  5 Trevithick S, Flabouris A, Tall G et al.  . International EMS systems: New South Wales, Australia. Resuscitation  2003; 59: 165– 70. Google Scholar CrossRef Search ADS PubMed  6 Pozner CN, Zane R, Nelson SJ et al.  . International EMS systems: the United States: past, present, and future. Resuscitation  2004; 60: 239– 44. doi:10.1016/j.resuscitation.2003.11.004. Google Scholar CrossRef Search ADS PubMed  7 Garner AA. The role of physician staffing of helicopter emergency medical services in prehospital trauma response. Emerg Med Australas  2004; 16: 318– 23. doi:10.1111/j.1742-6723.2004.00636.x. Google Scholar CrossRef Search ADS PubMed  8 Porter ME. What is value in health care? N Engl J Med  2010; 363: 2477– 81. Google Scholar CrossRef Search ADS PubMed  9 Institute of Medicine. Emergency Medical Services at a Crossroads . Washington DC: The National Academies Press, 2006. PubMed PubMed  10 World Health Organization. Quality of care: a process for making strategic choices in health systems 2006. 11 National Committee for Quality Assurance. The essential guide to health care quality, https://www.ncqa.org/Portals/0/Publications/Resource%20Library/NCQA_Primer_web.pdf. (1 March 2016, date last accessed). 12 Scott KW, Jha AK. Putting quality on the global health agenda. N Engl J Med  2014; 371: 3– 5. doi:10.1056/NEJMp1402157. Google Scholar CrossRef Search ADS PubMed  13 Hsia RY, Niedzwiecki M. Avoidable emergency department visits: a starting point. Int J Qual Health Care  2017; 29: 642– 5. doi:10.1093/intqhc/mzx081. Google Scholar CrossRef Search ADS PubMed  14 Male L, Noble A, Atkinson J et al.  . Measuring patient experience: a systematic review to evaluate psychometric properties of patient reported experience measures (PREMs) for emergency care service provision. Int J Qual Health Care  2017; 29: 314– 26. doi:10.1093/intqhc/mzx027. Google Scholar CrossRef Search ADS PubMed  15 Stenlund M, Sjodahl R, Pia Yngman-Uhlin RN. Incidence and potential risk factors for hospital-acquired pneumonia in an emergency department of surgery. Int J Qual Health Care  2017; 29: 290– 4. doi:10.1093/intqhc/mzx018. Google Scholar CrossRef Search ADS PubMed  16 Rehn M, Kruger AJ. Quality improvement in pre-hospital critical care: increased value through research and publication. Scand J Trauma Resusc Emerg Med  2014; 22: 34. Google Scholar CrossRef Search ADS PubMed  17 Snooks H, Evans A, Wells B et al.  . What are the highest priorities for research in emergency prehospital care? Emerg Med J  2009; 26: 549– 50. doi:10.1136/emj.2008.065862. Google Scholar CrossRef Search ADS PubMed  18 Fevang E, Lockey D, Thompson J et al.  . The top five research priorities in physician-provided pre-hospital critical care: a consensus report from a European research collaboration. Scand J Trauma Resusc Emerg Med  2011; 19: 57. doi:10.1186/1757-7241-19-57. Google Scholar CrossRef Search ADS PubMed  19 Mainz J. Defining and classifying clinical indicators for quality improvement. Int J Qual Health Care  2003; 15: 523– 30. Google Scholar CrossRef Search ADS PubMed  20 Donabedian A. The quality of care. How can it be assessed? JAMA  1988; 260: 1743– 8. Google Scholar CrossRef Search ADS PubMed  21 Donabedian A. Evaluating the quality of medical care. Milbank Mem Fund Q  1966; 44: 166– 206. Google Scholar CrossRef Search ADS   22 FAMILIESUSA. Measuring Health Care Quality: An Overview of Quality Measures. www.familiesusa.org. (15 April 2015, date last accessed). 23 Grol R. Improving the quality of medical care: building bridges among professional pride, payer profit, and patient satisfaction. JAMA  2001; 286: 2578– 85. Google Scholar CrossRef Search ADS PubMed  24 Blumenthal D. Part 1: Quality of care—what is it? N Engl J Med  1996; 335: 891– 4. doi:10.1056/NEJM199609193351213. Google Scholar CrossRef Search ADS PubMed  25 El Sayed MJ. Measuring quality in emergency medical services: a review of clinical performance indicators. Emerg Med Int  2012; 2012: 161630. doi:10.1155/2012/161630. Google Scholar CrossRef Search ADS PubMed  26 Kruger AJ, Skogvoll E, Castren M et al.  . ScanDoc Phase 1a Study Group. Scandinavian pre-hospital physician-manned Emergency Medical Services—same concept across borders? Resuscitation  2010; 81: 427– 33. doi:10.1016/j.resuscitation.2009.12.019. Google Scholar CrossRef Search ADS PubMed  27 Moher D, Liberati A, Tetzlaff J et al.  . Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med  2009; 6: e1000097. doi:10.1371/journal.pmed.1000097. Google Scholar CrossRef Search ADS PubMed  28 G K. ADACOPTER-Aufzeichnungen einer Entwicklung München. Werner Wolfsfellner Verlag; 2002. 29 Kruger AJ, Lockey D, Kurola J et al.  . A consensus-based template for documenting and reporting in physician-staffed pre-hospital services. Scand J Trauma Resusc Emerg Med  2011; 19: 71. doi:10.1186/1757-7241-19-71. Google Scholar CrossRef Search ADS PubMed  30 Akin Paker S, Dagar S, Gunay E et al.  . Assessment of prehospital medical care for the patients transported to emergency department by ambulance. Turk J Emerg Med  2015; 15: 122– 5. doi:10.1016/j.tjem.2015.11.005. Google Scholar CrossRef Search ADS PubMed  31 Anadelic S, Ivancevic N, Bogunovic S. Reaction times as indicators of the quality of expert work of belgrade municipal institutions for emergency medical services. Cent Eur J Med  2013; 8: 90– 5. doi:10.2478/s11536-012-0102-0. 32 Arntz HR, Klatt S, Stern R et al.  . Are emergency physicians’ prehospital diagnoses reliable?. [German]. Anaesthesist  1996; 45: 163– 70. doi:10.1007/s001010050251. Google Scholar CrossRef Search ADS PubMed  33 Breckwoldt J, Muller D, Overbeck M et al.  . Prehospital care of acute coronary syndrome by anaesthetists. Prospective comparison with the care standards of cardiologists. [German]. Anaesthesist  2008; 57: 131– 8. doi:10.1007/s00101-007-1290-x. Google Scholar CrossRef Search ADS PubMed  34 Duchateau FX, Garnier-Connois D, Ricard-Hibon A et al.  . Response time evaluation for emergency medical service as a part of its performance. Emerg Med J  2013; 30: 763– 5. doi:10.1136/emermed-2012-201714. Google Scholar CrossRef Search ADS PubMed  35 Fjaeldstad A, Kirk MH, Knudsen L et al.  . Physician-staffed emergency helicopter reduces transportation time from alarm call to highly specialized centre. Dan Med J  2013; 60: A4666. Google Scholar PubMed  36 Flabouris A. A description of events associated with scene response by helicopter based medical retrieval teams. Injury  2003; 34: 847– 52. doi:10.1016/S0020-1383%2803%2900033-0. Google Scholar CrossRef Search ADS PubMed  37 Helm M, Faul M, Unger T et al.  . Reliability of emergency medical field triage. Exemplified by traffic accident victims. [German]. Anaesthesist  2013; 62: 973– 80. doi:10.1007/s00101-013-2255-x. Google Scholar CrossRef Search ADS PubMed  38 Helm M, Hauke J, Sauermuller G et al.  . About the quality of prehospital emergency ventilation—A prospective study in trauma patients. [German]. Unfallchirurg  1999; 102: 347– 53. doi:10.1007/s001130050416. Google Scholar CrossRef Search ADS PubMed  39 Hennes HJ, Reinhardt T, Otto S et al.  . Prehospital care: the efficacy of prehospital intervention by emergency medicine physicians. [German]. Anaesthesist  1993; 42: 455– 61. Google Scholar PubMed  40 Klemenc-Ketis Z, Tomazin I, Kersnik J. HEMS in Slovenia: one country, four models, different quality outcomes. Air Med J  2012; 31: 298– 304. doi:10.1016/j.amj.2012.09.002. Google Scholar CrossRef Search ADS PubMed  41 Leicht MJ, Melanson SW, Albright D. Air medical tracheal intubation: establishing a threshold for this QA indicator. J Air Med Transp  1991; 10: 15– 20. Google Scholar CrossRef Search ADS PubMed  42 Lossius HM, Soreide E, Hotvedt R et al.  . Prehospital advanced life support provided by specially trained physicians: is there a benefit in terms of life years gained? Acta Anaesthesiol Scand  2002; 46: 771– 8. Google Scholar CrossRef Search ADS PubMed  43 Messelken M, Kehrberger E, Dirks B et al.  . The quality of emergency medical care in Baden-Wurttemberg (Germany): four years in focus. [German, English]. Dtsch Arztebl  2010; 107: 523– 30. doi:10.3238/arztebl.2010.0523. 44 Mikkelsen S, Kruger AJ, Zwisler ST et al.  . Outcome following physician supervised prehospital resuscitation: a retrospective study. BMJ Open  2015; 5: e006167. doi:10.1136/bmjopen-2014-006167. Google Scholar CrossRef Search ADS PubMed  45 Nakstad AR, Heimdal HJ, Strand T et al.  . Incidence of desaturation during prehospital rapid sequence intubation in a physician-based helicopter emergency service. Am J Emerg Med  2011; 29: 639– 44. doi:10.1016/j.ajem.2010.01.013. Google Scholar CrossRef Search ADS PubMed  46 Neukamm J, Grasner JT, Schewe JC et al.  . The impact of response time reliability on CPR incidence and resuscitation success: a benchmark study from the German Resuscitation Registry. Crit Care  2011; 15: R282. doi:10.1186/cc10566. Google Scholar CrossRef Search ADS PubMed  47 Pedersen UG, Jorgensen S, Nielsen SL. A survey of patients’ satisfaction with the Mobile Emergency Care Unit in Copenhagen. [Danish]. Ugeskr Laeger  2006; 168: 3322– 5. Google Scholar PubMed  48 Regel G, Seekamp A, Pohlemann T et al.  . Does the accident patient need to be protected from the emergency doctor?. [German]. Unfallchirurg  1998; 101: 160– 75. doi:10.1007/s001130050250. Google Scholar CrossRef Search ADS PubMed  49 Rognas L, Hansen TM, Kirkegaard H et al.  . Pre-hospital advanced airway management by experienced anaesthesiologists: a prospective descriptive study. Scand J Trauma Resusc Emerg Med  2013; 21: 58. doi:10.1186/1757-7241-21-58. Google Scholar CrossRef Search ADS PubMed  50 Rognas L, Hansen TM, Kirkegaard H et al.  . Anaesthesiologist-provided prehospital airway management in patients with traumatic brain injury: an observational study. Eur J Emerg Med  2014; 21: 418– 23. doi:10.1097/MEJ.0000000000000103. Google Scholar CrossRef Search ADS PubMed  51 Sollid SJ, Lossius HM, Soreide E. Pre-hospital intubation by anaesthesiologists in patients with severe trauma: an audit of a Norwegian helicopter emergency medical service. Scand J Trauma Resusc Emerg Med  2010; 18: 30. doi:10.1186/1757-7241-18-30. Google Scholar CrossRef Search ADS PubMed  52 van der Velden MW, Ringburg AN, Bergs EA et al.  . Prehospital interventions: time wasted or time saved? An observational cohort study of management in initial trauma care. Emerg Med J  2008; 25: 444– 9. doi:10.1136/emj.2007.052662. Google Scholar CrossRef Search ADS PubMed  53 Viergutz T, Schmittner MD, Bess S et al.  . Quality management of patients with severe traumatic brain injury in emergency medicine. [German]. Notfall und Rettungsmedizin  2015; 18: 381– 90. doi:10.1007/s10049-015-0033-9. Google Scholar CrossRef Search ADS   54 Von Knobelsdorff G, Reiferscheid F, Straakholder TM et al.  . Do paramedics and emergency physicians adhere to the algorithm of the European Resuscitation Council (ERC) for advanced cardiovascular life support (ACLS)?. [German]. Intensivmedizin und Notfallmedizin  2004; 41: 22– 8. doi:10.1007/s00390-004-0447-y. Google Scholar CrossRef Search ADS   55 Weltermann B, Vom Eyser D, Kleine-Zander R et al.  . Deployment of emergency physicians for patients with stroke in the Munster area: a cross-sectional study of the quality of medical care in one region. [German]. Dtsch Med Wochenschr  1999; 124: 1192– 6. Google Scholar CrossRef Search ADS PubMed  56 Schlechtriemen T, Lackner CK, Moecke H et al.  . Medizinisches Qualitätsmanagement mit Hilfe ausgewählter Zieldiagnosen. [German]. Notfall Rettungsmedizin  2003; 6: 175– 88. doi:10.1007/s10049-003-0561-6. Google Scholar CrossRef Search ADS   57 Sobo EJ, Andriese S, Stroup C et al.  . Developing indicators for emergency medical services (EMS) system evaluation and quality improvement: a statewide demonstration and planning project. Jt Comm J Qual Improv  2001; 27: 138– 54. Google Scholar PubMed  58 Spaite DW, Criss EA, Valenzuela TD et al.  . Emergency medical service systems research: problems of the past, challenges of the future. Ann Emerg Med  1995; 26: 146– 52. Google Scholar CrossRef Search ADS PubMed  59 Spaite DW, Maio R, Garrison HG et al.  . Emergency Medical Services Outcomes Project (EMSOP) II: developing the foundation and conceptual models for out-of-hospital outcomes research. Ann Emerg Med  2001; 37: 657– 63. Google Scholar CrossRef Search ADS PubMed  60 Maio RF, Garrison HG, Spaite DW et al.  . Emergency medical services outcomes project I (EMSOP I): prioritizing conditions for outcomes research. Ann Emerg Med  1999; 33: 423– 32. Google Scholar CrossRef Search ADS PubMed  61 Busse R, Nimptsch U. Mansky T. Measuring, monitoring, and managing quality in Germany’s hospitals. Health Aff  2009; 28: 294– 304. doi:10.1377/hlthaff.28.2.w294. Google Scholar CrossRef Search ADS   62 DIVI. Gemeinsame Empfehlung der Deutschen Interdisziplinaren Vereinigung fur Intensiv- und Notfallsmedizin (DIVI) und des Vorstandes des Bundesarztekammer zum Qualitatsmanagement in der Notfallsmedizin. [German]. Intensivmed  1995; 32: 387– 8. 63 Moecke H, Schafer J, Herden H et al.  . Das Bundeseinheitliche Rettungsdienstprotokoll-Empfehlung der DIVI. [German]. Intensivmed  1994; 31: 96– 9. 64 Herden H, Moecke H. Bundeseinheitliches Notarzteinsatzprotokoll. [German]. Anasth Intensivmed  1991; 33: 166– 9. 65 Palmer RH. Using health outcomes data to compare plans, networks and providers. Int J Qual Health Care  1998; 10: 477– 83. Google Scholar CrossRef Search ADS PubMed  66 El Sayed MJ. Measuring quality in emergency medical services: a review of clinical performance indicators. Emerg Med Int  2012; 2012: 161630. Google Scholar CrossRef Search ADS PubMed  67 Nilsen JE, Tjelmeland IBM, Halvorsen KS et al.  . Kvalitetsindikatorer i den akuttmedisinske kjeden. [Norwegian]. www.nakos.no: Nasjonal kompetansetjeneste for prehospital akuttmedisin 2015. 68 Department of Health. Emergency care ten years on: reforming emergency care . London: Department of Health, 2007. 69 Cooke M, Fischer J, Mc Leod E et al.   Reducing attendances and waits in emergency departments: a systematic review of present innovations NHS Service and Delivery Organisation 2005. 70 Ebben RH, Vloet LC, Verhofstad MH et al.  . Mintjes-de Groot JA, van Achterberg T. Adherence to guidelines and protocols in the prehospital and emergency care setting: a systematic review. Scand J Trauma Resusc Emerg Med  2013; 21: 9. doi:10.1186/1757-7241-21-9. Google Scholar CrossRef Search ADS PubMed  71 Charpentier S, Sagnes-Raffy C, Cournot M et al.  . Determinants and prognostic impact of compliance with guidelines in reperfusion therapy for ST-segment elevation myocardial infarction: results from the ESTIM Midi-Pyrenees Area. Arch Cardiovasc Dis  2009; 102: 387– 96. doi:10.1016/j.acvd.2009.02.011. Google Scholar CrossRef Search ADS PubMed  72 Aylwin CJ, Konig TC, Brennan NW et al.  . Reduction in critical mortality in urban mass casualty incidents: analysis of triage, surge, and resource use after the London bombings on July 7, 2005. Lancet  2006; 368: 2219– 25. doi:10.1016/S0140-6736(06)69896-6. Google Scholar CrossRef Search ADS PubMed  73 Kirves H, Skrifvars MB, Vahakuopus M et al.  . Adherence to resuscitation guidelines during prehospital care of cardiac arrest patients. Eur J Emerg Med  2007; 14: 75– 81. doi:10.1097/MEJ.0b013e328013f88c. Google Scholar CrossRef Search ADS PubMed  74 Wollersheim H, Burgers J, Grol R. Clinical guidelines to improve patient care. Neth J Med  2005; 63: 188– 92. Google Scholar PubMed  75 Forsner T, Wistedt AA, Brommels M et al.  . An approach to measure compliance to clinical guidelines in psychiatric care. BMC Psychiatry  2008; 8: 64. doi:10.1186/1471-244×-8-64. Google Scholar CrossRef Search ADS PubMed  76 Haugland H, Rehn M, Klepstad P et al.  , EQUIPE collaboration group. Developing quality indicators for physician-staffed emergency medical services: a consensus process. Scand J Trauma Resusc Emerg Med  2017; 25: 14. doi:10.1186/s13049-017-0362-4. Google Scholar CrossRef Search ADS PubMed  77 Kessner DM, Kalk CE, Singer J. Assessing health quality—the case for tracers. N Engl J Med  1973; 288: 189– 94. doi:10.1056/NEJM197301252880406. Google Scholar CrossRef Search ADS PubMed  78 Martin LA, Nelson EC, Lloyd RC, Nolan TW. Whole system measures . Cambridge, Massachusetts: Institute for Healthcare Improvement, 2007. 79 Institute of Medicine. Crossing the quality chasm: a new health system for the twenty-first century . Washington: National Academies Press, 2001. 80 Ogrinc G, Davies L, Goodman D, Batalden P, Davidoff F, Stevens D. SQUIRE 2.0 (Standards for QUality Improvement Reporting Excellence): revised publication guidelines from a detailed consensus process. BMJ Qual Saf  2016; 25: 986– 92. doi:10.1136/bmjqs-2015-004411. Google Scholar CrossRef Search ADS PubMed  © The Author(s) 2018. Published by Oxford University Press in association with the International Society for Quality in Health Care. This is an Open Access article distributed under the terms of the Creative Commons Attribution-NonCommercial-NoDerivs licence (http://creativecommons.org/licenses/by-nc-nd/4.0/), which permits non-commercial reproduction and distribution of the work, in any medium, provided the original work is not altered or transformed in any way, and that the work is properly cited. For commercial re-use, please contact journals.permissions@oup.com http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png International Journal for Quality in Health Care Oxford University Press

Quality measurement in physician-staffed emergency medical services: a systematic literature review

Loading next page...
 
/lp/ou_press/quality-measurement-in-physician-staffed-emergency-medical-services-a-jaK4pnRruQ
Publisher
Oxford University Press
Copyright
© The Author(s) 2018. Published by Oxford University Press in association with the International Society for Quality in Health Care.
ISSN
1353-4505
eISSN
1464-3677
D.O.I.
10.1093/intqhc/mzy106
Publisher site
See Article on Publisher Site

Abstract

Abstract Purpose Quality measurement of physician-staffed emergency medical services (P-EMS) is necessary to improve service quality. Knowledge and consensus on this topic are scarce, making quality measurement of P-EMS a high-priority research area. The aim of this review was to identify, describe and evaluate studies of quality measurement in P-EMS. Data sources The databases of MEDLINE and Embase were searched initially, followed by a search for included article citations in Scopus. Study selection The study eligibility criteria were: (1) articles describing the use of one quality indicator (QI) or more in P-EMS, (2) original manuscripts, (3) articles published from 1 January 1968 until 5 October 2016. The literature search identified 4699 records. 4543 were excluded after reviewing title and abstract. An additional 129 were excluded based on a full-text review. The remaining 27 papers were included in the analysis. Methodological quality was assessed using an adapted critical appraisal tool. Data extraction The description of used QIs and methods of quality measurement was extracted. Variables describing the involved P-EMSs were extracted as well. Results of data synthesis In the included papers, a common understanding of which QIs to use in P-EMS did not exist. Fifteen papers used only a single QI. The most widely used QIs were ‘Adherence to medical protocols’, ‘Provision of advanced interventions’, ‘Response time’ and ‘Adverse events’. Conclusion The review demonstrated a lack of shared understanding of which QIs to use in P-EMS. Moreover, papers using only one QI dominated the literature, thus increasing the risk of a narrow perspective in quality measurement. Future quality measurement in P-EMS should rely on a set of consensus-based QIs, ensuring a comprehensive approach to quality measurement. quality measurement < quality management, quality improvement < quality management, emergency care < setting of care Background Emergency medical services (EMS) provide pre-hospital treatment and transportation to definitive care for patients in need of urgent medical care. EMSs are well integrated in health services in most countries and normally consist of ground ambulances staffed by paramedics, emergency medical technicians or nurses [1–6]. As a supplement to regular EMS, physician-staffed rapid response vehicles or helicopters exist in some areas [7]. Although the operational concept of this physician-staffed EMS (P-EMS) may differ, a common feature is the involvement of a specially trained physician in pre-hospital care of critically ill or injured patients. Depending on the country, these physicians are often anaesthesiologists, surgeons, internists or emergency physicians [1, 3]. The focus on quality measurement in healthcare is increasing [8–12]. As an example, quality dimensions such as efficiency, patient-centeredness and safety have been assessed in several emergency departments [13–15]. In P-EMS, a valid model for quality assessment is needed to achieve appropriate governance, quality assurance and quality improvement [16]. Snooks et al. [17] define the development of meaningful quality indicators (QIs) for EMS as the most important issue for future research in emergency pre-hospital care. For P-EMS, measuring quality of care is considered a priority area of research [18]. Quality measurement can be defined as measuring the extent to which set targets are achieved [19]. A QI is used to measure performance against a recognized standard of care. Donabedian defines three categories of QIs: structure, process and outcome of healthcare [20, 21]. Structure indicators describe the infrastructure of a healthcare system, such as competence of the staff, equipment and deployment and response times. Process indicators evaluate the care provided to the patient, and outcome indicators address the change in patient health status. None of these categories of indicators provide a complete description of the quality of care but address single components. Thus, different types of QIs should be combined to assess the quality of a service [19]. QIs inform clinicians and organizations how the health system performs and aid in the improvement in care. Ideally, all QIs are based upon evidence of their relevance and importance. The process of developing QIs generally includes stakeholders who evaluate the evidence and define the QI parameters [22]. These QIs for P-EMS should be evaluated against patient-oriented outcomes, e.g. pain intensity, morbidity or mortality. However, P-EMS quality can also relate to system factors such as training of traditional EMS, major incident management and the concept of providing equity of access to healthcare. Different stakeholders have different perspectives on what represents quality in healthcare [23, 24], and various QIs for P-EMS are possible. A widely cited definition of quality that also might be applicable for P-EMS systems is ‘the degree to which health services for individuals and populations increase the likelihood of desired health outcomes and are consistent with current professional knowledge’ [9]. Often, a few high impact clinical conditions are identified (in terms of morbidity, mortality, costs and incidence). These conditions are termed ‘tracer conditions’. Examples of tracer conditions for emergency medicine are cardiac arrest or trauma patients with severely reduced consciousness (Glasgow Coma Score < 8). Measuring the outcomes of tracer conditions can predict a system’s response to other clinical states and the overall quality of a service [25]. This systematic review aims to identify, describe and appraise the methodological quality of the literature pertaining to the quality assessment of P-EMS. Methods For the purpose of this review, physicians who staff P-EMS should be trained in critical care, exceeding the competency of a general practitioner on call [26]. Moreover, we define the term ‘pre-hospital’ as relating to procedures administered or care provided prior to patient arrival at the hospital [8]. The studies identified in the review do not address the potential benefit from P-EMS compared to other EMS. Literature search strategy A systematic literature search of MEDLINE and Embase to identify relevant literature was conducted (see Additional file 1 for search strategy). Four different sets of entry terms were applied and combined. These entry terms describe pre-hospital setting, emergency care, physician staffing and finally the concept of quality measurement. All records were collated in an Endnote bibliographic database (©2007 Thompson Reuters). The study followed the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-analyses) guidelines, including the PICOS (Population, Intervention, Comparator, Outcome, Study design) methodology [27]. Here, the participants were all the identified articles describing quality estimation in P-EMS. Our evaluation of intervention, comparison and outcomes was carried out using the data extraction and quality appraisal variables in Tables 1 and 3. The study was registered at PROSPERO (http://www.crd.york.ac.uk//prospero/, registration number CRD42015024421). Table 1 Reasons for excluding 129 out of 156 full-text studies in the eligibility-check of the systematic review Reasons for exclusion  No.  Wrong study design  77  Not about quality measurement in P-EMS  29  Not enough information for quality appraisal and data extraction  9  Only abstracts  6  Not about P-EMS  3  Comparative studies pertaining to new procedures  2  Commentary, letter to editor or editorial  1  Not original article  1  Duplicate  1  Reasons for exclusion  No.  Wrong study design  77  Not about quality measurement in P-EMS  29  Not enough information for quality appraisal and data extraction  9  Only abstracts  6  Not about P-EMS  3  Comparative studies pertaining to new procedures  2  Commentary, letter to editor or editorial  1  Not original article  1  Duplicate  1  Table 1 Reasons for excluding 129 out of 156 full-text studies in the eligibility-check of the systematic review Reasons for exclusion  No.  Wrong study design  77  Not about quality measurement in P-EMS  29  Not enough information for quality appraisal and data extraction  9  Only abstracts  6  Not about P-EMS  3  Comparative studies pertaining to new procedures  2  Commentary, letter to editor or editorial  1  Not original article  1  Duplicate  1  Reasons for exclusion  No.  Wrong study design  77  Not about quality measurement in P-EMS  29  Not enough information for quality appraisal and data extraction  9  Only abstracts  6  Not about P-EMS  3  Comparative studies pertaining to new procedures  2  Commentary, letter to editor or editorial  1  Not original article  1  Duplicate  1  Inclusion/exclusion criteria Articles were included in the systematic literature review if they fulfilled all the following criteria: (1) literature describing methods of quality estimation in P-EMS, i.e. the use of one or more QIs based on quantitative methods, qualitative methods or both; (2) original manuscripts and (3) literature published after 1 January 1968 and until the date of the literature search, 5 October 2016. The rationale for including literature from 1 January 1968, is the establishment of the world’s first civilian physician-staffed helicopter EMS in Munich this year [28]. Articles without abstract, book chapters, editorials, comments and letters to the editor were excluded. Articles in English, French, German and Scandinavian languages were identified. The translation competency of these languages was present in the author group. Literature identification The records from the literature search were exported to www.covidence.org. Here, all titles and corresponding abstracts were screened independently by two of the authors for eligibility (O.U. and H.H.) (Figure 1). In the case of uncertainty, a third reviewer (M.R.) reviewed the title/abstract. Articles clearly not meeting the inclusion criteria were excluded. Articles accepted for full-text screening were assessed in pairs of authors (O.U. and M.R., A.J.K. and H.H.) using the inclusion/exclusion criteria listed above. Excluded articles were listed with the reason for exclusion. If there was any uncertainty about whether an article should be included, there was a discussion until consensus was reached among the authors. One author (H.H.) performed data extraction and quality appraisal and consulted another author (M.R.) in uncertain cases. Further, one author hand-searched references in included articles to identify additional relevant articles (H.H.). Finally, a search in Scopus was conducted to identify articles citing the included articles. Figure 1 View largeDownload slide Information flow through the different phases of the systematic review. Figure 1 View largeDownload slide Information flow through the different phases of the systematic review. Prior to the literature search, the authors made templates for data extraction and quality appraisal. The data extraction and quality appraisal variables were based on the authors’ assumptions on what is important to report in quality measurement studies in P-EMS. However, these variables do not represent a reference standard, since such a standard does not exist, to our knowledge. As a part of data extraction, fixed-system variables were included. Fixed-system variables relate to system characteristics concerning the organization, staffing and operational capacities of the service and are necessary for interpreting the results [29]. Results A total of 4699 articles were identified by the search strategy, 156 of which were accepted for full-text screening. Of these, 129 articles were excluded. The main reason for excluding articles were ‘Wrong study design’, pertaining to articles exclusively comparing different treatment modalities and without any quality measurement objectives (Table 1). A total of 26 articles from the main database search were included for data extraction and quality appraisal [30–55]. One additional article was included from the Scopus search for citing articles [56]. A review of the literature lists of included articles did not result in additional findings. None of the papers gave a complete report of fixed-system variables, thus complicating the comparison of involved P-EMS concepts (Table 2). Twenty-four of the 27 papers use QIs that can be identified as process indicators. Structure indicators and outcome indicators are used less frequently, in two and seven papers, respectively. Twenty-five different QIs were identified, all of which were considered suitable for international use and transferable to other P-EMSs (Table 3). The most widely used QI was ‘Adherence to medical protocols’. This QI measures if medical guidelines are followed, as done by Viergutz et al. [53] who investigated whether guidelines for preclinical care of patients with traumatic head injury were followed. The second most used QI was ‘Provision of advanced interventions’, investigating if the P-EMS unit provided treatment that exceeds the competences of the attending EMS, as done by Mikkelsen et al. [44]. The two following QIs are ‘Response time’ and ‘Adverse events’, the latter exemplified by Nakstad et al. [45], who studied the incidence of desaturation during pre-hospital rapid sequence intubation. Fifteen papers used one single QI, and twelve papers applied a set of QIs. Moreover, three papers used tracer conditions as their approach to quality assessment (Table 2). Pertaining to the internal validity of the papers, ten of the 27 papers did not clearly explain the methodology for developing the QIs (Table 4). Table 2 Data extraction of included articles   Are the following fixed-system variables reported?  Quality indicators used  Multiple, mixed quality indicators  Tracer conditions used    Transportation mode  SAR-capability  Rural and/or urban  Response types  Speciality of physician  Physician experience  Funding  Description of dispatch system  Other fixed-system variables  Structure  Process  Outcome  Patient satisfaction      Akin Paker [30]  ✓  ✗  ✗  ✓  ✗  ✗  ✗  ✗  ✗  ✗  ✓  ✗  ✗  ✓  ✗  Anadelic [31]  ✓  ✗  ✓  Partly  ✗  ✗  ✗  ✓  ✓  ✓  ✓  ✗  ✗  ✓  ✗  Arntz [32]  ✓  ✗  ✓  ✓  ✓  Partly  ✓  ✓  ✓  ✗  ✓  ✗  ✗  ✓  ✗  Breckwoldt [33]  ✓  ✗  ✓  ✗  ✓  ✓  ✗  ✗  ✓  ✗  ✓  ✓  ✗  ✗  ✗  Duchateau [34]  ✓  ✗  ✓  ✗  ✗  ✗  ✗  ✗  ✓  ✗  ✓  ✗  ✗  ✗  ✗  Fjaeldstad [35]  ✓  ✗  ✓  ✓  ✓  ✗  ✓  ✓  ✓  ✗  ✓  ✗  ✗  ✗  ✗  Flabouris [36]  ✓  ✓  ✗  ✓  ✓  ✓  ✗  ✓  ✓  ✗  ✓  ✗  ✗  ✓  ✗  Helm [37]  ✓  ✗  ✗  ✗  ✗  ✗  ✗  ✗  ✗  ✗  ✓  ✗  ✗  ✗  ✗  Helm [38]  ✓  ✗  ✗  ✗  ✓  ✗  ✗  ✗  ✗  ✗  ✓  ✗  ✗  ✗  ✗  Hennes [39]  ✓  ✗  ✓  ✓  ✗  ✗  ✗  ✗  ✗  ✗  ✗  ✓  ✗  ✗  ✗  Klemenc-Ketis [40]  ✓  ✓  ✓  ✓  ✗  ✗  ✓  ✓  ✓  ✗  ✓  ✗  ✗  ✓  ✗  Leicht [41]  ✓  ✗  ✗  ✓  ✓  ✓  ✗  ✗  ✓  ✗  ✓  ✗  ✗  ✗  ✗  Lossius [42]  ✓  ✗  ✓  ✓  ✓  ✓  ✗  Partly  ✗  ✗  ✗  ✓  ✗  ✗  ✗  Messelken [43]  ✓  ✗  ✓  ✓  ✓  ✗  ✗  ✗  ✗  ✗  ✓  ✓  ✗  ✓  ✓  Mikkelsen [44]  ✓  ✗  ✓  ✓  ✓  ✓  ✗  ✓  ✓  ✗  ✓  ✓  ✗  ✗  ✗  Nakstad [45]  ✓  ✗  ✓  ✓  ✓  ✓  ✗  ✗  ✓  ✗  ✓  ✗  ✗  ✗  ✗  Neukamm [46]  ✓  ✗  ✓  ✓  ✗  ✗  ✗  Partly  ✓  ✓  ✓  ✓  ✗  ✓  ✗  Pedersen [47]  ✓  ✗  ✓  ✓  ✗  ✗  ✗  ✗  ✗  ✗  ✗  ✗  ✓  ✗  ✓  Regel [48]  ✓  ✗  ✓  ✓  ✗  ✗  ✗  ✗  ✓  ✗  ✓  ✗  ✗  ✓  ✗  Rognås [49]  ✓  ✗  ✓  ✓  ✓  ✓  ✗  ✓  ✓  ✗  ✓  ✗  ✗  ✗  ✗  Rognås [50]  ✓  ✗  ✓  ✓  ✓  ✗  ✗  Partly  ✓  ✗  ✓  ✗  ✗  ✗  ✗  Sollid [51]  ✓  ✗  ✓  ✓  ✓  ✗  ✗  ✗  ✓  ✗  ✓  ✗  ✗  ✗  ✗  Van der Velden [52]  ✓  ✗  ✓  ✓  ✗  ✗  ✗  ✓  ✓  ✗  ✓  ✗  ✗  ✗  ✗  Viergutz [53]  ✓  ✗  ✓  ✓  ✗  ✗  ✗  ✗  ✓  ✗  ✓  ✗  ✗  ✓  ✗  Von Knobelsdorff [54]  ✓  ✗  ✓  ✓  ✗  ✗  ✗  ✗  ✗  ✗  ✓  ✓  ✗  ✓  ✗  Weltermann [55]  ✓  ✗  ✓  ✓  ✗  ✗  ✗  ✗  ✗  ✗  ✓  ✗  ✗  ✓  ✗  Schlechtriemen [56]  ✓  ✗  ✗  ✗  ✗  ✗  ✗  ✗  ✗  ✗  ✓  ✗  ✗  ✓  ✓    Are the following fixed-system variables reported?  Quality indicators used  Multiple, mixed quality indicators  Tracer conditions used    Transportation mode  SAR-capability  Rural and/or urban  Response types  Speciality of physician  Physician experience  Funding  Description of dispatch system  Other fixed-system variables  Structure  Process  Outcome  Patient satisfaction      Akin Paker [30]  ✓  ✗  ✗  ✓  ✗  ✗  ✗  ✗  ✗  ✗  ✓  ✗  ✗  ✓  ✗  Anadelic [31]  ✓  ✗  ✓  Partly  ✗  ✗  ✗  ✓  ✓  ✓  ✓  ✗  ✗  ✓  ✗  Arntz [32]  ✓  ✗  ✓  ✓  ✓  Partly  ✓  ✓  ✓  ✗  ✓  ✗  ✗  ✓  ✗  Breckwoldt [33]  ✓  ✗  ✓  ✗  ✓  ✓  ✗  ✗  ✓  ✗  ✓  ✓  ✗  ✗  ✗  Duchateau [34]  ✓  ✗  ✓  ✗  ✗  ✗  ✗  ✗  ✓  ✗  ✓  ✗  ✗  ✗  ✗  Fjaeldstad [35]  ✓  ✗  ✓  ✓  ✓  ✗  ✓  ✓  ✓  ✗  ✓  ✗  ✗  ✗  ✗  Flabouris [36]  ✓  ✓  ✗  ✓  ✓  ✓  ✗  ✓  ✓  ✗  ✓  ✗  ✗  ✓  ✗  Helm [37]  ✓  ✗  ✗  ✗  ✗  ✗  ✗  ✗  ✗  ✗  ✓  ✗  ✗  ✗  ✗  Helm [38]  ✓  ✗  ✗  ✗  ✓  ✗  ✗  ✗  ✗  ✗  ✓  ✗  ✗  ✗  ✗  Hennes [39]  ✓  ✗  ✓  ✓  ✗  ✗  ✗  ✗  ✗  ✗  ✗  ✓  ✗  ✗  ✗  Klemenc-Ketis [40]  ✓  ✓  ✓  ✓  ✗  ✗  ✓  ✓  ✓  ✗  ✓  ✗  ✗  ✓  ✗  Leicht [41]  ✓  ✗  ✗  ✓  ✓  ✓  ✗  ✗  ✓  ✗  ✓  ✗  ✗  ✗  ✗  Lossius [42]  ✓  ✗  ✓  ✓  ✓  ✓  ✗  Partly  ✗  ✗  ✗  ✓  ✗  ✗  ✗  Messelken [43]  ✓  ✗  ✓  ✓  ✓  ✗  ✗  ✗  ✗  ✗  ✓  ✓  ✗  ✓  ✓  Mikkelsen [44]  ✓  ✗  ✓  ✓  ✓  ✓  ✗  ✓  ✓  ✗  ✓  ✓  ✗  ✗  ✗  Nakstad [45]  ✓  ✗  ✓  ✓  ✓  ✓  ✗  ✗  ✓  ✗  ✓  ✗  ✗  ✗  ✗  Neukamm [46]  ✓  ✗  ✓  ✓  ✗  ✗  ✗  Partly  ✓  ✓  ✓  ✓  ✗  ✓  ✗  Pedersen [47]  ✓  ✗  ✓  ✓  ✗  ✗  ✗  ✗  ✗  ✗  ✗  ✗  ✓  ✗  ✓  Regel [48]  ✓  ✗  ✓  ✓  ✗  ✗  ✗  ✗  ✓  ✗  ✓  ✗  ✗  ✓  ✗  Rognås [49]  ✓  ✗  ✓  ✓  ✓  ✓  ✗  ✓  ✓  ✗  ✓  ✗  ✗  ✗  ✗  Rognås [50]  ✓  ✗  ✓  ✓  ✓  ✗  ✗  Partly  ✓  ✗  ✓  ✗  ✗  ✗  ✗  Sollid [51]  ✓  ✗  ✓  ✓  ✓  ✗  ✗  ✗  ✓  ✗  ✓  ✗  ✗  ✗  ✗  Van der Velden [52]  ✓  ✗  ✓  ✓  ✗  ✗  ✗  ✓  ✓  ✗  ✓  ✗  ✗  ✗  ✗  Viergutz [53]  ✓  ✗  ✓  ✓  ✗  ✗  ✗  ✗  ✓  ✗  ✓  ✗  ✗  ✓  ✗  Von Knobelsdorff [54]  ✓  ✗  ✓  ✓  ✗  ✗  ✗  ✗  ✗  ✗  ✓  ✓  ✗  ✓  ✗  Weltermann [55]  ✓  ✗  ✓  ✓  ✗  ✗  ✗  ✗  ✗  ✗  ✓  ✗  ✗  ✓  ✗  Schlechtriemen [56]  ✓  ✗  ✗  ✗  ✗  ✗  ✗  ✗  ✗  ✗  ✓  ✗  ✗  ✓  ✓  ✓ = yes; ✗ = no. Table 2 Data extraction of included articles   Are the following fixed-system variables reported?  Quality indicators used  Multiple, mixed quality indicators  Tracer conditions used    Transportation mode  SAR-capability  Rural and/or urban  Response types  Speciality of physician  Physician experience  Funding  Description of dispatch system  Other fixed-system variables  Structure  Process  Outcome  Patient satisfaction      Akin Paker [30]  ✓  ✗  ✗  ✓  ✗  ✗  ✗  ✗  ✗  ✗  ✓  ✗  ✗  ✓  ✗  Anadelic [31]  ✓  ✗  ✓  Partly  ✗  ✗  ✗  ✓  ✓  ✓  ✓  ✗  ✗  ✓  ✗  Arntz [32]  ✓  ✗  ✓  ✓  ✓  Partly  ✓  ✓  ✓  ✗  ✓  ✗  ✗  ✓  ✗  Breckwoldt [33]  ✓  ✗  ✓  ✗  ✓  ✓  ✗  ✗  ✓  ✗  ✓  ✓  ✗  ✗  ✗  Duchateau [34]  ✓  ✗  ✓  ✗  ✗  ✗  ✗  ✗  ✓  ✗  ✓  ✗  ✗  ✗  ✗  Fjaeldstad [35]  ✓  ✗  ✓  ✓  ✓  ✗  ✓  ✓  ✓  ✗  ✓  ✗  ✗  ✗  ✗  Flabouris [36]  ✓  ✓  ✗  ✓  ✓  ✓  ✗  ✓  ✓  ✗  ✓  ✗  ✗  ✓  ✗  Helm [37]  ✓  ✗  ✗  ✗  ✗  ✗  ✗  ✗  ✗  ✗  ✓  ✗  ✗  ✗  ✗  Helm [38]  ✓  ✗  ✗  ✗  ✓  ✗  ✗  ✗  ✗  ✗  ✓  ✗  ✗  ✗  ✗  Hennes [39]  ✓  ✗  ✓  ✓  ✗  ✗  ✗  ✗  ✗  ✗  ✗  ✓  ✗  ✗  ✗  Klemenc-Ketis [40]  ✓  ✓  ✓  ✓  ✗  ✗  ✓  ✓  ✓  ✗  ✓  ✗  ✗  ✓  ✗  Leicht [41]  ✓  ✗  ✗  ✓  ✓  ✓  ✗  ✗  ✓  ✗  ✓  ✗  ✗  ✗  ✗  Lossius [42]  ✓  ✗  ✓  ✓  ✓  ✓  ✗  Partly  ✗  ✗  ✗  ✓  ✗  ✗  ✗  Messelken [43]  ✓  ✗  ✓  ✓  ✓  ✗  ✗  ✗  ✗  ✗  ✓  ✓  ✗  ✓  ✓  Mikkelsen [44]  ✓  ✗  ✓  ✓  ✓  ✓  ✗  ✓  ✓  ✗  ✓  ✓  ✗  ✗  ✗  Nakstad [45]  ✓  ✗  ✓  ✓  ✓  ✓  ✗  ✗  ✓  ✗  ✓  ✗  ✗  ✗  ✗  Neukamm [46]  ✓  ✗  ✓  ✓  ✗  ✗  ✗  Partly  ✓  ✓  ✓  ✓  ✗  ✓  ✗  Pedersen [47]  ✓  ✗  ✓  ✓  ✗  ✗  ✗  ✗  ✗  ✗  ✗  ✗  ✓  ✗  ✓  Regel [48]  ✓  ✗  ✓  ✓  ✗  ✗  ✗  ✗  ✓  ✗  ✓  ✗  ✗  ✓  ✗  Rognås [49]  ✓  ✗  ✓  ✓  ✓  ✓  ✗  ✓  ✓  ✗  ✓  ✗  ✗  ✗  ✗  Rognås [50]  ✓  ✗  ✓  ✓  ✓  ✗  ✗  Partly  ✓  ✗  ✓  ✗  ✗  ✗  ✗  Sollid [51]  ✓  ✗  ✓  ✓  ✓  ✗  ✗  ✗  ✓  ✗  ✓  ✗  ✗  ✗  ✗  Van der Velden [52]  ✓  ✗  ✓  ✓  ✗  ✗  ✗  ✓  ✓  ✗  ✓  ✗  ✗  ✗  ✗  Viergutz [53]  ✓  ✗  ✓  ✓  ✗  ✗  ✗  ✗  ✓  ✗  ✓  ✗  ✗  ✓  ✗  Von Knobelsdorff [54]  ✓  ✗  ✓  ✓  ✗  ✗  ✗  ✗  ✗  ✗  ✓  ✓  ✗  ✓  ✗  Weltermann [55]  ✓  ✗  ✓  ✓  ✗  ✗  ✗  ✗  ✗  ✗  ✓  ✗  ✗  ✓  ✗  Schlechtriemen [56]  ✓  ✗  ✗  ✗  ✗  ✗  ✗  ✗  ✗  ✗  ✓  ✗  ✗  ✓  ✓    Are the following fixed-system variables reported?  Quality indicators used  Multiple, mixed quality indicators  Tracer conditions used    Transportation mode  SAR-capability  Rural and/or urban  Response types  Speciality of physician  Physician experience  Funding  Description of dispatch system  Other fixed-system variables  Structure  Process  Outcome  Patient satisfaction      Akin Paker [30]  ✓  ✗  ✗  ✓  ✗  ✗  ✗  ✗  ✗  ✗  ✓  ✗  ✗  ✓  ✗  Anadelic [31]  ✓  ✗  ✓  Partly  ✗  ✗  ✗  ✓  ✓  ✓  ✓  ✗  ✗  ✓  ✗  Arntz [32]  ✓  ✗  ✓  ✓  ✓  Partly  ✓  ✓  ✓  ✗  ✓  ✗  ✗  ✓  ✗  Breckwoldt [33]  ✓  ✗  ✓  ✗  ✓  ✓  ✗  ✗  ✓  ✗  ✓  ✓  ✗  ✗  ✗  Duchateau [34]  ✓  ✗  ✓  ✗  ✗  ✗  ✗  ✗  ✓  ✗  ✓  ✗  ✗  ✗  ✗  Fjaeldstad [35]  ✓  ✗  ✓  ✓  ✓  ✗  ✓  ✓  ✓  ✗  ✓  ✗  ✗  ✗  ✗  Flabouris [36]  ✓  ✓  ✗  ✓  ✓  ✓  ✗  ✓  ✓  ✗  ✓  ✗  ✗  ✓  ✗  Helm [37]  ✓  ✗  ✗  ✗  ✗  ✗  ✗  ✗  ✗  ✗  ✓  ✗  ✗  ✗  ✗  Helm [38]  ✓  ✗  ✗  ✗  ✓  ✗  ✗  ✗  ✗  ✗  ✓  ✗  ✗  ✗  ✗  Hennes [39]  ✓  ✗  ✓  ✓  ✗  ✗  ✗  ✗  ✗  ✗  ✗  ✓  ✗  ✗  ✗  Klemenc-Ketis [40]  ✓  ✓  ✓  ✓  ✗  ✗  ✓  ✓  ✓  ✗  ✓  ✗  ✗  ✓  ✗  Leicht [41]  ✓  ✗  ✗  ✓  ✓  ✓  ✗  ✗  ✓  ✗  ✓  ✗  ✗  ✗  ✗  Lossius [42]  ✓  ✗  ✓  ✓  ✓  ✓  ✗  Partly  ✗  ✗  ✗  ✓  ✗  ✗  ✗  Messelken [43]  ✓  ✗  ✓  ✓  ✓  ✗  ✗  ✗  ✗  ✗  ✓  ✓  ✗  ✓  ✓  Mikkelsen [44]  ✓  ✗  ✓  ✓  ✓  ✓  ✗  ✓  ✓  ✗  ✓  ✓  ✗  ✗  ✗  Nakstad [45]  ✓  ✗  ✓  ✓  ✓  ✓  ✗  ✗  ✓  ✗  ✓  ✗  ✗  ✗  ✗  Neukamm [46]  ✓  ✗  ✓  ✓  ✗  ✗  ✗  Partly  ✓  ✓  ✓  ✓  ✗  ✓  ✗  Pedersen [47]  ✓  ✗  ✓  ✓  ✗  ✗  ✗  ✗  ✗  ✗  ✗  ✗  ✓  ✗  ✓  Regel [48]  ✓  ✗  ✓  ✓  ✗  ✗  ✗  ✗  ✓  ✗  ✓  ✗  ✗  ✓  ✗  Rognås [49]  ✓  ✗  ✓  ✓  ✓  ✓  ✗  ✓  ✓  ✗  ✓  ✗  ✗  ✗  ✗  Rognås [50]  ✓  ✗  ✓  ✓  ✓  ✗  ✗  Partly  ✓  ✗  ✓  ✗  ✗  ✗  ✗  Sollid [51]  ✓  ✗  ✓  ✓  ✓  ✗  ✗  ✗  ✓  ✗  ✓  ✗  ✗  ✗  ✗  Van der Velden [52]  ✓  ✗  ✓  ✓  ✗  ✗  ✗  ✓  ✓  ✗  ✓  ✗  ✗  ✗  ✗  Viergutz [53]  ✓  ✗  ✓  ✓  ✗  ✗  ✗  ✗  ✓  ✗  ✓  ✗  ✗  ✓  ✗  Von Knobelsdorff [54]  ✓  ✗  ✓  ✓  ✗  ✗  ✗  ✗  ✗  ✗  ✓  ✓  ✗  ✓  ✗  Weltermann [55]  ✓  ✗  ✓  ✓  ✗  ✗  ✗  ✗  ✗  ✗  ✓  ✗  ✗  ✓  ✗  Schlechtriemen [56]  ✓  ✗  ✗  ✗  ✗  ✗  ✗  ✗  ✗  ✗  ✓  ✗  ✗  ✓  ✓  ✓ = yes; ✗ = no. Table 3 Quality indicators used in the included literature Quality indicator  Category  No. of papers it is used in  Adherence to medical protocols  Process  11  Provision of advanced interventions  Process  8  Response time  Process  7  Adverse events  Process  7  Medication administration  Process  5  Transport to appropriate facility  Process  4  Time on scene  Process  3  Improved care due to clinical decision making  Process  3  Reliability of the primary diagnosis made by the P-EMS physician  Process  3  Survival  Outcome  3  ROSC in cardiac arrest  Outcome  3  Time from alarm to patient handover  Process  2  Time from arrival at patient until hospital admission  Process  2  Altered physiology  Outcome  2  Pain management  Outcome  2  Proportion of intubated patients adequately oxygenated and ventilated  Process  2  Time gain by air transportation  Process  1  The number of patients with a NACA-score ≥ 4 with an intravenous line  Process  1  The proportion of patients successfully intubated  Process  1  Life years gained  Outcome  1  Morbidity/disability  Outcome  1  Amount of yearly CPR training  Structure  1  Precision of dispatch  Process  1  Rate of CPR started within 8 min of the call to the dispatch center  Process  1  Patient satisfaction  Outcome  1  Quality indicator  Category  No. of papers it is used in  Adherence to medical protocols  Process  11  Provision of advanced interventions  Process  8  Response time  Process  7  Adverse events  Process  7  Medication administration  Process  5  Transport to appropriate facility  Process  4  Time on scene  Process  3  Improved care due to clinical decision making  Process  3  Reliability of the primary diagnosis made by the P-EMS physician  Process  3  Survival  Outcome  3  ROSC in cardiac arrest  Outcome  3  Time from alarm to patient handover  Process  2  Time from arrival at patient until hospital admission  Process  2  Altered physiology  Outcome  2  Pain management  Outcome  2  Proportion of intubated patients adequately oxygenated and ventilated  Process  2  Time gain by air transportation  Process  1  The number of patients with a NACA-score ≥ 4 with an intravenous line  Process  1  The proportion of patients successfully intubated  Process  1  Life years gained  Outcome  1  Morbidity/disability  Outcome  1  Amount of yearly CPR training  Structure  1  Precision of dispatch  Process  1  Rate of CPR started within 8 min of the call to the dispatch center  Process  1  Patient satisfaction  Outcome  1  ROSC, return of spontaneous circulation; NACA, National Advisory Committee for Aeronautics (see Additional file 2); CPR, cardiopulmonary resuscitation. Table 3 Quality indicators used in the included literature Quality indicator  Category  No. of papers it is used in  Adherence to medical protocols  Process  11  Provision of advanced interventions  Process  8  Response time  Process  7  Adverse events  Process  7  Medication administration  Process  5  Transport to appropriate facility  Process  4  Time on scene  Process  3  Improved care due to clinical decision making  Process  3  Reliability of the primary diagnosis made by the P-EMS physician  Process  3  Survival  Outcome  3  ROSC in cardiac arrest  Outcome  3  Time from alarm to patient handover  Process  2  Time from arrival at patient until hospital admission  Process  2  Altered physiology  Outcome  2  Pain management  Outcome  2  Proportion of intubated patients adequately oxygenated and ventilated  Process  2  Time gain by air transportation  Process  1  The number of patients with a NACA-score ≥ 4 with an intravenous line  Process  1  The proportion of patients successfully intubated  Process  1  Life years gained  Outcome  1  Morbidity/disability  Outcome  1  Amount of yearly CPR training  Structure  1  Precision of dispatch  Process  1  Rate of CPR started within 8 min of the call to the dispatch center  Process  1  Patient satisfaction  Outcome  1  Quality indicator  Category  No. of papers it is used in  Adherence to medical protocols  Process  11  Provision of advanced interventions  Process  8  Response time  Process  7  Adverse events  Process  7  Medication administration  Process  5  Transport to appropriate facility  Process  4  Time on scene  Process  3  Improved care due to clinical decision making  Process  3  Reliability of the primary diagnosis made by the P-EMS physician  Process  3  Survival  Outcome  3  ROSC in cardiac arrest  Outcome  3  Time from alarm to patient handover  Process  2  Time from arrival at patient until hospital admission  Process  2  Altered physiology  Outcome  2  Pain management  Outcome  2  Proportion of intubated patients adequately oxygenated and ventilated  Process  2  Time gain by air transportation  Process  1  The number of patients with a NACA-score ≥ 4 with an intravenous line  Process  1  The proportion of patients successfully intubated  Process  1  Life years gained  Outcome  1  Morbidity/disability  Outcome  1  Amount of yearly CPR training  Structure  1  Precision of dispatch  Process  1  Rate of CPR started within 8 min of the call to the dispatch center  Process  1  Patient satisfaction  Outcome  1  ROSC, return of spontaneous circulation; NACA, National Advisory Committee for Aeronautics (see Additional file 2); CPR, cardiopulmonary resuscitation. Table 4 Quality appraisal of included articles   Internal validity  External validity  Was the methodology for developing the QIs clearly explained?  Are the QIs clearly defined?  Is the rationale for the QIs sufficiently described?  Is handling of missing data sufficiently described?  Do the authors address conflicts of interest?  Has an ethics committee approved the study?  Is professional background and funding of those involved in QI development stated?  Were the QIs developed by the systematic work of a group of experts?  Is it reported in which country/organization the QIs were developed?  Is (are) the P-EMS (s) involved in quality measurement sufficiently described?  Are the QIs transferable to other countries or P-EMSs?  Are limitations of the study discussed?  Are possible sources of bias discussed?  Have the QIs been used in other publications?  Was the feasibility of the QIs evaluated?  Akin Paker [30]  ✗  ✗  ✗  ✓  ✗  ✓  ✗  ?  ✓  ✗  ✓  ✓  ✓  ?  ✓  Anadelic [31]  ✓  ✓  ✓  ✗  ✗  ?  ✓  ?  ✓  ✗  ✓  ✗  ✗  ?  ✓  Arntz [32]  ✓  ✗  ✓  ✓  ✗  ?  ✗  ?  ✓  ✓  ✓  ✗  ✗  ?  ✗  Breckwoldt [33]  ✓  ✗  ✓  ✗  ✓  ?  ✗  ?  ✓  ✗  ✓  ✗  ✗  ?  ✗  Duchateau [34]  ✓  ✓  ✗  ✗  ✓  ?  ✗  ?  ✓  ✗  ✓  ✗  ✗  ✓  ✗  Fjaeldstad [35]  ✗  ✓  ✗  ✓  ✓  ?  ✗  ?  ✓  ✗  ✓  ✓  ✗  ?  ✗  Flabouris [36]  ✓  ✓  ✓  ✓  ✗  ?  ✗  ?  ✓  ✓  ✓  ✓  ✓  ?  ✗  Helm [37]  ✓  ✓  ✓  ✗  ✓  ✓  ✗  ✗  ✓  ✗  ✓  ✓  ✗  ✓  ✗  Helm [38]  ✗  ✓  ✓  ✗  ✗  ?  ✗  ✗  ✓  ✗  ✓  ✓  ✗  ?  ✗  Hennes [39]  ✓  ✓  ✓  ✗  ✗  ?  ✗  ?  ✓  ✗  ✓  ✓  ✓  ✓  ✓  Klemenc-Ketis [40]  ✗  ✓  ✓  ✗  ✗  ✓  ✗  ✗  ✓  ✓  ✓  ✓  ✓  ?  ✗  Leicht [41]  ✗  ✓  ✓  ✗  ✗  ?  ✗  ✗  ✓  ✗  ✓  ✓  ✓  ?  ✗  Lossius [42]  ✓  ✓  ✓  ✗  ✗  ✓  ✓  ?  ✓  ✓  ✓  ✓  ✓  ✓  ✗  Messelken [43]  ✓  ✓  ✓  ✓  ✓  ?  ✗  ?  ✓  ✗  ✓  ✓  ✗  ✓  ✗  Mikkelsen [44]  ✓  ✓  ✓  ✓  ✓  ?  ✓  ✗  ✓  ✓  ✓  ✓  ✓  ?  ✗  Nakstad [45]  ✗  ✓  ✓  ✓  ✗  ✓  ✗  ?  ✓  ✓  ✓  ✗  ✓  ✓  ✗  Neukamm [46]  ✗  ✓  ✗  ✗  ✓  ✓  ✗  ?  ✓  ✓  ✓  ✓  ✓  ?  ✗  Pedersen [47]  ✗  ✓  ✓  ✗  ✓  ✓  ✗  ✗  ✓  ✗  ✓  ✓  ✓  ✗  ✗  Regel [48]  ✓  ✓  ✓  ✗  ✗  ?  ✗  ?  ✓  ✗  ✓  ✓  ✓  ✗  ✗  Rognås [49]  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✗  ✗  Rognås [50]  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✗  Sollid [51]  ✗  ✗  ✓  ✗  ✓  ✓  ✓  ✗  ✓  ✓  ✓  ✓  ✓  ?  ✗  Van der Velden [52]  ✓  ✓  ✓  ✗  ✓  ?  ✓  ✗  ✓  ✗  ✓  ✓  ✓  ?  ✗  Viergutz [53]  ✓  ✓  ✓  ✓  ✓  ✓  ✗  ✗  ✓  ✗  ✓  ✓  ✓  ?  ✗  Von Knobelsdorff [54]  ✗  ✓  ✓  ✓  ✗  ?  ✗  ?  ✓  ✗  ✓  ✗  ✓  ?  ✗  Weltermann [55]  ✓  ✓  ✓  ✗  ✗  ?  ✗  ?  ✓  ✗  ✓  ✓  ✗  ✓  ✗  Schlechtriemen [56]  ✓  ✓  ✓  –  ✗  –  ✗  ?  ✓  –  ✓  ✓  ✓  ?  ✗    Internal validity  External validity  Was the methodology for developing the QIs clearly explained?  Are the QIs clearly defined?  Is the rationale for the QIs sufficiently described?  Is handling of missing data sufficiently described?  Do the authors address conflicts of interest?  Has an ethics committee approved the study?  Is professional background and funding of those involved in QI development stated?  Were the QIs developed by the systematic work of a group of experts?  Is it reported in which country/organization the QIs were developed?  Is (are) the P-EMS (s) involved in quality measurement sufficiently described?  Are the QIs transferable to other countries or P-EMSs?  Are limitations of the study discussed?  Are possible sources of bias discussed?  Have the QIs been used in other publications?  Was the feasibility of the QIs evaluated?  Akin Paker [30]  ✗  ✗  ✗  ✓  ✗  ✓  ✗  ?  ✓  ✗  ✓  ✓  ✓  ?  ✓  Anadelic [31]  ✓  ✓  ✓  ✗  ✗  ?  ✓  ?  ✓  ✗  ✓  ✗  ✗  ?  ✓  Arntz [32]  ✓  ✗  ✓  ✓  ✗  ?  ✗  ?  ✓  ✓  ✓  ✗  ✗  ?  ✗  Breckwoldt [33]  ✓  ✗  ✓  ✗  ✓  ?  ✗  ?  ✓  ✗  ✓  ✗  ✗  ?  ✗  Duchateau [34]  ✓  ✓  ✗  ✗  ✓  ?  ✗  ?  ✓  ✗  ✓  ✗  ✗  ✓  ✗  Fjaeldstad [35]  ✗  ✓  ✗  ✓  ✓  ?  ✗  ?  ✓  ✗  ✓  ✓  ✗  ?  ✗  Flabouris [36]  ✓  ✓  ✓  ✓  ✗  ?  ✗  ?  ✓  ✓  ✓  ✓  ✓  ?  ✗  Helm [37]  ✓  ✓  ✓  ✗  ✓  ✓  ✗  ✗  ✓  ✗  ✓  ✓  ✗  ✓  ✗  Helm [38]  ✗  ✓  ✓  ✗  ✗  ?  ✗  ✗  ✓  ✗  ✓  ✓  ✗  ?  ✗  Hennes [39]  ✓  ✓  ✓  ✗  ✗  ?  ✗  ?  ✓  ✗  ✓  ✓  ✓  ✓  ✓  Klemenc-Ketis [40]  ✗  ✓  ✓  ✗  ✗  ✓  ✗  ✗  ✓  ✓  ✓  ✓  ✓  ?  ✗  Leicht [41]  ✗  ✓  ✓  ✗  ✗  ?  ✗  ✗  ✓  ✗  ✓  ✓  ✓  ?  ✗  Lossius [42]  ✓  ✓  ✓  ✗  ✗  ✓  ✓  ?  ✓  ✓  ✓  ✓  ✓  ✓  ✗  Messelken [43]  ✓  ✓  ✓  ✓  ✓  ?  ✗  ?  ✓  ✗  ✓  ✓  ✗  ✓  ✗  Mikkelsen [44]  ✓  ✓  ✓  ✓  ✓  ?  ✓  ✗  ✓  ✓  ✓  ✓  ✓  ?  ✗  Nakstad [45]  ✗  ✓  ✓  ✓  ✗  ✓  ✗  ?  ✓  ✓  ✓  ✗  ✓  ✓  ✗  Neukamm [46]  ✗  ✓  ✗  ✗  ✓  ✓  ✗  ?  ✓  ✓  ✓  ✓  ✓  ?  ✗  Pedersen [47]  ✗  ✓  ✓  ✗  ✓  ✓  ✗  ✗  ✓  ✗  ✓  ✓  ✓  ✗  ✗  Regel [48]  ✓  ✓  ✓  ✗  ✗  ?  ✗  ?  ✓  ✗  ✓  ✓  ✓  ✗  ✗  Rognås [49]  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✗  ✗  Rognås [50]  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✗  Sollid [51]  ✗  ✗  ✓  ✗  ✓  ✓  ✓  ✗  ✓  ✓  ✓  ✓  ✓  ?  ✗  Van der Velden [52]  ✓  ✓  ✓  ✗  ✓  ?  ✓  ✗  ✓  ✗  ✓  ✓  ✓  ?  ✗  Viergutz [53]  ✓  ✓  ✓  ✓  ✓  ✓  ✗  ✗  ✓  ✗  ✓  ✓  ✓  ?  ✗  Von Knobelsdorff [54]  ✗  ✓  ✓  ✓  ✗  ?  ✗  ?  ✓  ✗  ✓  ✗  ✓  ?  ✗  Weltermann [55]  ✓  ✓  ✓  ✗  ✗  ?  ✗  ?  ✓  ✗  ✓  ✓  ✗  ✓  ✗  Schlechtriemen [56]  ✓  ✓  ✓  –  ✗  –  ✗  ?  ✓  –  ✓  ✓  ✓  ?  ✗  ✓ = yes; ✗ = no; ? = uncertain; – = not relevant. Table 4 Quality appraisal of included articles   Internal validity  External validity  Was the methodology for developing the QIs clearly explained?  Are the QIs clearly defined?  Is the rationale for the QIs sufficiently described?  Is handling of missing data sufficiently described?  Do the authors address conflicts of interest?  Has an ethics committee approved the study?  Is professional background and funding of those involved in QI development stated?  Were the QIs developed by the systematic work of a group of experts?  Is it reported in which country/organization the QIs were developed?  Is (are) the P-EMS (s) involved in quality measurement sufficiently described?  Are the QIs transferable to other countries or P-EMSs?  Are limitations of the study discussed?  Are possible sources of bias discussed?  Have the QIs been used in other publications?  Was the feasibility of the QIs evaluated?  Akin Paker [30]  ✗  ✗  ✗  ✓  ✗  ✓  ✗  ?  ✓  ✗  ✓  ✓  ✓  ?  ✓  Anadelic [31]  ✓  ✓  ✓  ✗  ✗  ?  ✓  ?  ✓  ✗  ✓  ✗  ✗  ?  ✓  Arntz [32]  ✓  ✗  ✓  ✓  ✗  ?  ✗  ?  ✓  ✓  ✓  ✗  ✗  ?  ✗  Breckwoldt [33]  ✓  ✗  ✓  ✗  ✓  ?  ✗  ?  ✓  ✗  ✓  ✗  ✗  ?  ✗  Duchateau [34]  ✓  ✓  ✗  ✗  ✓  ?  ✗  ?  ✓  ✗  ✓  ✗  ✗  ✓  ✗  Fjaeldstad [35]  ✗  ✓  ✗  ✓  ✓  ?  ✗  ?  ✓  ✗  ✓  ✓  ✗  ?  ✗  Flabouris [36]  ✓  ✓  ✓  ✓  ✗  ?  ✗  ?  ✓  ✓  ✓  ✓  ✓  ?  ✗  Helm [37]  ✓  ✓  ✓  ✗  ✓  ✓  ✗  ✗  ✓  ✗  ✓  ✓  ✗  ✓  ✗  Helm [38]  ✗  ✓  ✓  ✗  ✗  ?  ✗  ✗  ✓  ✗  ✓  ✓  ✗  ?  ✗  Hennes [39]  ✓  ✓  ✓  ✗  ✗  ?  ✗  ?  ✓  ✗  ✓  ✓  ✓  ✓  ✓  Klemenc-Ketis [40]  ✗  ✓  ✓  ✗  ✗  ✓  ✗  ✗  ✓  ✓  ✓  ✓  ✓  ?  ✗  Leicht [41]  ✗  ✓  ✓  ✗  ✗  ?  ✗  ✗  ✓  ✗  ✓  ✓  ✓  ?  ✗  Lossius [42]  ✓  ✓  ✓  ✗  ✗  ✓  ✓  ?  ✓  ✓  ✓  ✓  ✓  ✓  ✗  Messelken [43]  ✓  ✓  ✓  ✓  ✓  ?  ✗  ?  ✓  ✗  ✓  ✓  ✗  ✓  ✗  Mikkelsen [44]  ✓  ✓  ✓  ✓  ✓  ?  ✓  ✗  ✓  ✓  ✓  ✓  ✓  ?  ✗  Nakstad [45]  ✗  ✓  ✓  ✓  ✗  ✓  ✗  ?  ✓  ✓  ✓  ✗  ✓  ✓  ✗  Neukamm [46]  ✗  ✓  ✗  ✗  ✓  ✓  ✗  ?  ✓  ✓  ✓  ✓  ✓  ?  ✗  Pedersen [47]  ✗  ✓  ✓  ✗  ✓  ✓  ✗  ✗  ✓  ✗  ✓  ✓  ✓  ✗  ✗  Regel [48]  ✓  ✓  ✓  ✗  ✗  ?  ✗  ?  ✓  ✗  ✓  ✓  ✓  ✗  ✗  Rognås [49]  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✗  ✗  Rognås [50]  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✗  Sollid [51]  ✗  ✗  ✓  ✗  ✓  ✓  ✓  ✗  ✓  ✓  ✓  ✓  ✓  ?  ✗  Van der Velden [52]  ✓  ✓  ✓  ✗  ✓  ?  ✓  ✗  ✓  ✗  ✓  ✓  ✓  ?  ✗  Viergutz [53]  ✓  ✓  ✓  ✓  ✓  ✓  ✗  ✗  ✓  ✗  ✓  ✓  ✓  ?  ✗  Von Knobelsdorff [54]  ✗  ✓  ✓  ✓  ✗  ?  ✗  ?  ✓  ✗  ✓  ✗  ✓  ?  ✗  Weltermann [55]  ✓  ✓  ✓  ✗  ✗  ?  ✗  ?  ✓  ✗  ✓  ✓  ✗  ✓  ✗  Schlechtriemen [56]  ✓  ✓  ✓  –  ✗  –  ✗  ?  ✓  –  ✓  ✓  ✓  ?  ✗    Internal validity  External validity  Was the methodology for developing the QIs clearly explained?  Are the QIs clearly defined?  Is the rationale for the QIs sufficiently described?  Is handling of missing data sufficiently described?  Do the authors address conflicts of interest?  Has an ethics committee approved the study?  Is professional background and funding of those involved in QI development stated?  Were the QIs developed by the systematic work of a group of experts?  Is it reported in which country/organization the QIs were developed?  Is (are) the P-EMS (s) involved in quality measurement sufficiently described?  Are the QIs transferable to other countries or P-EMSs?  Are limitations of the study discussed?  Are possible sources of bias discussed?  Have the QIs been used in other publications?  Was the feasibility of the QIs evaluated?  Akin Paker [30]  ✗  ✗  ✗  ✓  ✗  ✓  ✗  ?  ✓  ✗  ✓  ✓  ✓  ?  ✓  Anadelic [31]  ✓  ✓  ✓  ✗  ✗  ?  ✓  ?  ✓  ✗  ✓  ✗  ✗  ?  ✓  Arntz [32]  ✓  ✗  ✓  ✓  ✗  ?  ✗  ?  ✓  ✓  ✓  ✗  ✗  ?  ✗  Breckwoldt [33]  ✓  ✗  ✓  ✗  ✓  ?  ✗  ?  ✓  ✗  ✓  ✗  ✗  ?  ✗  Duchateau [34]  ✓  ✓  ✗  ✗  ✓  ?  ✗  ?  ✓  ✗  ✓  ✗  ✗  ✓  ✗  Fjaeldstad [35]  ✗  ✓  ✗  ✓  ✓  ?  ✗  ?  ✓  ✗  ✓  ✓  ✗  ?  ✗  Flabouris [36]  ✓  ✓  ✓  ✓  ✗  ?  ✗  ?  ✓  ✓  ✓  ✓  ✓  ?  ✗  Helm [37]  ✓  ✓  ✓  ✗  ✓  ✓  ✗  ✗  ✓  ✗  ✓  ✓  ✗  ✓  ✗  Helm [38]  ✗  ✓  ✓  ✗  ✗  ?  ✗  ✗  ✓  ✗  ✓  ✓  ✗  ?  ✗  Hennes [39]  ✓  ✓  ✓  ✗  ✗  ?  ✗  ?  ✓  ✗  ✓  ✓  ✓  ✓  ✓  Klemenc-Ketis [40]  ✗  ✓  ✓  ✗  ✗  ✓  ✗  ✗  ✓  ✓  ✓  ✓  ✓  ?  ✗  Leicht [41]  ✗  ✓  ✓  ✗  ✗  ?  ✗  ✗  ✓  ✗  ✓  ✓  ✓  ?  ✗  Lossius [42]  ✓  ✓  ✓  ✗  ✗  ✓  ✓  ?  ✓  ✓  ✓  ✓  ✓  ✓  ✗  Messelken [43]  ✓  ✓  ✓  ✓  ✓  ?  ✗  ?  ✓  ✗  ✓  ✓  ✗  ✓  ✗  Mikkelsen [44]  ✓  ✓  ✓  ✓  ✓  ?  ✓  ✗  ✓  ✓  ✓  ✓  ✓  ?  ✗  Nakstad [45]  ✗  ✓  ✓  ✓  ✗  ✓  ✗  ?  ✓  ✓  ✓  ✗  ✓  ✓  ✗  Neukamm [46]  ✗  ✓  ✗  ✗  ✓  ✓  ✗  ?  ✓  ✓  ✓  ✓  ✓  ?  ✗  Pedersen [47]  ✗  ✓  ✓  ✗  ✓  ✓  ✗  ✗  ✓  ✗  ✓  ✓  ✓  ✗  ✗  Regel [48]  ✓  ✓  ✓  ✗  ✗  ?  ✗  ?  ✓  ✗  ✓  ✓  ✓  ✗  ✗  Rognås [49]  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✗  ✗  Rognås [50]  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✓  ✗  Sollid [51]  ✗  ✗  ✓  ✗  ✓  ✓  ✓  ✗  ✓  ✓  ✓  ✓  ✓  ?  ✗  Van der Velden [52]  ✓  ✓  ✓  ✗  ✓  ?  ✓  ✗  ✓  ✗  ✓  ✓  ✓  ?  ✗  Viergutz [53]  ✓  ✓  ✓  ✓  ✓  ✓  ✗  ✗  ✓  ✗  ✓  ✓  ✓  ?  ✗  Von Knobelsdorff [54]  ✗  ✓  ✓  ✓  ✗  ?  ✗  ?  ✓  ✗  ✓  ✗  ✓  ?  ✗  Weltermann [55]  ✓  ✓  ✓  ✗  ✗  ?  ✗  ?  ✓  ✗  ✓  ✓  ✗  ✓  ✗  Schlechtriemen [56]  ✓  ✓  ✓  –  ✗  –  ✗  ?  ✓  –  ✓  ✓  ✓  ?  ✗  ✓ = yes; ✗ = no; ? = uncertain; – = not relevant. Discussion This systematic review identified 27 papers that reported the use of QIs in P-EMS. Fifteen papers used one single QI, and twelve papers applied a set of QIs. Twenty-four of the 27 papers used QIs pertaining to process. Generally, the systematic review demonstrated a lack of a shared understanding of which QI to use in P-EMS. The lack of agreement on QIs has also been described for regular EMS [57, 58]. This lack of consensus on QI, the heterogeneity of diagnosis, and the challenge of isolating the effect of pre-hospital care from the effect of in-hospital care complicates quality measurement in pre-hospital emergency medicine [59, 60]. The characteristics of the P-EMSs in the identified literature vary and are described sufficiently in only ten papers. This complicates the comparison of studies because the concept of involved P-EMSs remains unclear. Five services use helicopters, eleven use rapid response cars, and another eleven use both transportation modes. Further, nine services are urban, one is rural and eleven are both urban and rural. This mixed representation of transportation modes and urban versus rural profile seems to reflect the heterogeneity of P-EMS. Another relevant aspect of P-EMS is the specialty training and competency of the staffing physicians. Fourteen papers do not report the physicians’ medical specialty. Emergency medicine and anesthesiology are reported as the physicians’ specialties in one and seven papers, respectively. Five studies report a mix of medical specialties in the actual services. All mixes are different, with anesthesiology as the only medical specialty represented in all five papers. Regarding country, eleven of the 27 papers are German. Germany has a long history of P-EMS as an integrated and natural part of the emergency medical system. However, the considerable contribution of German papers is not only because of the service’s long existence. Quality measurement in German hospitals has developed substantially during the last two decades [61], and gradually, German P-EMS has adopted this quality measurement initiative. In addition to a formalized and common understanding of the need for quality measurement in P-EMS [62], the establishment of common documentation systems seems to have been the first necessary step towards quality measurement in German P-EMS and might set an example for QI work in P-EMS in other countries [63, 64]. In 23 papers, the QIs are well defined. However, for many of the QIs identified in this systematic review, the development process does not seem optimal. Ten papers do not clearly describe the methodology for developing the QIs, and only in two papers were the QIs systematically developed by a group of experts. Finally, only seven papers report the professional background and funding of those involved in the development of QIs. A key point in the development of QIs is a systematical and objective approach; this allows for the assessment of the evidence base for the QIs and secures legitimacy. Inadequate QI development may influence the validity, reliability and feasibility of the QIs. Structure indicators and outcome indicators are used by only two and seven papers, respectively. This may be because process indicators are easier to collect. Routine use of outcome indicators will require automated data exchange between pre- and in-hospital databases, which might not be feasible. This calls for a more integrated electronic patient chart system covering patient data capture through all phases of care. Nevertheless, the major role of process indicators in quality measurement literature of P-EMS is as expected. Process indicators are considered useful for short time frames and when it is difficult to adjust for patient factors [65], and process indicators are therefore particularly relevant for P-EMS. Moreover, process indicators provide a direct assessment of quality of care, as opposed to structure and outcome indicators, which measure care quality by an indirect approach [66]. Process indicators are easy to interpret and well suited for the evaluation of adherence to medical protocols and other quality improvement programmes. As seen in Table 3, many of the process indicators are time variables. When setting targets and measuring EMS quality, time variables have been widely applied [17, 66]. EMS evolved as a response to the need for rapid access to healthcare in time critical conditions (war injuries, cardiac arrest, major trauma) [66, 67]. However, the current patient population for P-EMS is increasingly heterogeneous, including conditions that are not highly time critical. For a high proportion of patients, shorter pre-hospital time intervals do not improve outcome, and in some cases, longer on-scene time for initial treatment and stabilization is desirable [68, 69]. Accordingly, if P-EMS quality measurement focuses too much on time variables, the results will have poor relevance for a high proportion of the services’ patient population. Moreover, it can lead to an undesired attention shift, resulting in decreased quality for quality dimensions not measured. However, this undesired attention shift is not specific for time variables but rather a universal challenge in quality measurement when the used QIs are few. Finally, our findings indicate that literature pertaining to pre-hospital time variables often lacks information about the competency of the responding unit and the quality of provided care, resulting in even less information from the time variables measured. Adherence to medical protocols is the most frequently used QI in the included papers. Guidelines and protocols are developed to improve quality of care and to reduce unwanted variation in care. Quality measurement can be defined as measuring the extent to which set targets are achieved [19], and to explore the gap between guidelines and clinical practice is an adequate approach in quality measurement. Ebben et al. [70] demonstrated a wide variation in different EMS professionals’ adherence to guidelines and protocols, indicating that a substantial number of patients do not receive appropriate pre-hospital care. However, this conclusion presumes that there is an evident relationship between adherence to guidelines and patient outcome. Few studies have explored this relationship, and in the review by Ebben et al., only three studies showed that adherence to guidelines improved patient outcomes [71–73]. Finally, it is recommended that guidelines should define QIs to aid monitoring and assessment of guideline adherence [74, 75]. Thus, QIs should ideally be a part of the guideline development process. For the fifteen papers relying on only one single QI, there is a risk for narrowing the perspective on quality, and important aspects of quality in the actual healthcare service may be ignored. Using a set of mixed QIs that cover different aspects of the service is preferable [76]. Three papers evaluated the care quality of selected ‘tracer conditions’, i.e. high-priority clinical conditions [77]. The evaluation of a service’s response to tracer conditions with condition-specific QIs is used to predict the overall performance of P-EMS. The chosen tracer conditions are not identical, illustrating the following challenge: when different clinical conditions are used for quality measurement, it may complicate the evaluation and comparison of the quality of different P-EMS. To overcome this, the Institute for Healthcare Improvement (IHI) has recommended the use of ‘whole system measures’, defined as a set of QIs aligned with the Institute of Medicine’s (IOM’s) six dimensions of quality, which are not disease- or condition-specific [78, 79]. The six quality dimensions that define high-quality care are timeliness, safety, efficiency, equity, effectiveness and patient-centeredness. Each of these is distinct, and all are equally important. To obtain an adequate and comprehensive quality measurement of P-EMS, future quality measurement should therefore cover these six dimensions as far as possible. This will require the use of multiple QIs, developed specifically for P-EMS. A set of QIs for this purpose has been developed recently, with IOM’s six quality dimensions as the most important part of the conceptual framework [76]. Strengths and limitations We recognize some strengths and limitations of this study. A strength is that in the literature search, several languages were eligible, allowing inclusion of non-English literature. This is important as eleven of the 27 identified papers were non-English. First, a limitation is that as always in a systematic review, screening, eligibility-check and qualitative synthesis of literature is a product of the review authors’ judgements, allowing subjective interpretations of the content of the studies. Although screening and eligibility assessment were conducted in pairs, data extraction and quality appraisal were primarily conducted by one author. Moreover, no established tools for data extraction or quality appraisal were available. Second, some of the included studies are not explicitly presented as quality measurement studies. However, when screening the literature, we recognized that some studies are quality measurement studies despite not using the quality measurement terminology. A possible reason for this might be the fact that quality measurement terminology is still quite new in P-EMS and not widely used. Accordingly, we have focused on the actual content of the screened literature when deciding if a study should be included—not on the presence or absence of correct quality measurement terminology. Doing this, we have appraised if a paper ‘concerns an initiative to improve healthcare; broadly defined to include the quality, safety, effectiveness, patient-centeredness, timeliness, cost, efficiency and equity of healthcare’, as stated by the SQUIRE guidelines [80]. This approach was chosen to avoid overlooking potentially relevant aspects of quality measurement studies in P-EMS. Conclusion This systematic literature review served the purpose of identifying, describing and evaluating studies of quality measurement in P-EMS. The review demonstrated a lack of a shared understanding of which QIs to use in P-EMS. Process indicators were dominant in the included papers, and the most emphasized QIs were ‘Adherence to medical protocols’, ‘Provision of advanced interventions’, ‘Response time’ and ‘Adverse events’. Moreover, fifteen of the 27 papers used only a single QI to measure quality, thus increasing the risk of a narrow perspective on quality. The remaining papers used multiple QIs in their quality measurement, which is considered preferable. Future quality measurement in P-EMS should rely on a set of consensus-based QIs, securing a comprehensive approach to quality measurement and offering the possibility of comparing results from different P-EMS systems. Supplementary material Supplementary material is available at International Journal for Quality in Health Care online. Acknowledgements The authors would like to thank Mrs Ingrid Ingeborg Riphagen, Unit for Applied Clinical Research, Norwegian University of Science and Technology, for her competent and invaluable assistance in designing and conducting the literature search. We would also like to thank members and donors of the Norwegian Air Ambulance Foundation for their financial contribution to this study. Additional file Additional file 1: ‘Search strategy’. Additional file 2: ‘NACA-score; elaboration document’. Authors’ contribution H.H. and M.R. conceived the idea. All authors were part of the study design. H.H. and O.U. screened the identified literature. H.H., O.U., A.J.K. and M.R. considered the eligibility of uncertain literature. H.H. performed data extraction and quality appraisal of the included literature. All authors have approved the final version of the manuscript. Ethics approval and consent to participate Not applicable. References 1 Roessler M, Zuzan O. EMS systems in Germany. Resuscitation  2006; 68: 45– 9. doi:10.1016/j.resuscitation.2005.08.004. Google Scholar CrossRef Search ADS PubMed  2 Langhelle A, Lossius HM, Silfvast T et al.  . International EMS Systems: the Nordic countries. Resuscitation  2004; 61: 9– 21. doi:10.1016/j.resuscitation.2003.12.008. Google Scholar CrossRef Search ADS PubMed  3 Black JJ, Davies GD. International EMS systems: United Kingdom. Resuscitation  2005; 64: 21– 9. doi:10.1016/j.resuscitation.2004.10.004. Google Scholar CrossRef Search ADS PubMed  4 Adnet F, Lapostolle F. International EMS systems: France. Resuscitation  2004; 63: 7– 9. doi:10.1016/j.resuscitation.2004.04.001. Google Scholar CrossRef Search ADS PubMed  5 Trevithick S, Flabouris A, Tall G et al.  . International EMS systems: New South Wales, Australia. Resuscitation  2003; 59: 165– 70. Google Scholar CrossRef Search ADS PubMed  6 Pozner CN, Zane R, Nelson SJ et al.  . International EMS systems: the United States: past, present, and future. Resuscitation  2004; 60: 239– 44. doi:10.1016/j.resuscitation.2003.11.004. Google Scholar CrossRef Search ADS PubMed  7 Garner AA. The role of physician staffing of helicopter emergency medical services in prehospital trauma response. Emerg Med Australas  2004; 16: 318– 23. doi:10.1111/j.1742-6723.2004.00636.x. Google Scholar CrossRef Search ADS PubMed  8 Porter ME. What is value in health care? N Engl J Med  2010; 363: 2477– 81. Google Scholar CrossRef Search ADS PubMed  9 Institute of Medicine. Emergency Medical Services at a Crossroads . Washington DC: The National Academies Press, 2006. PubMed PubMed  10 World Health Organization. Quality of care: a process for making strategic choices in health systems 2006. 11 National Committee for Quality Assurance. The essential guide to health care quality, https://www.ncqa.org/Portals/0/Publications/Resource%20Library/NCQA_Primer_web.pdf. (1 March 2016, date last accessed). 12 Scott KW, Jha AK. Putting quality on the global health agenda. N Engl J Med  2014; 371: 3– 5. doi:10.1056/NEJMp1402157. Google Scholar CrossRef Search ADS PubMed  13 Hsia RY, Niedzwiecki M. Avoidable emergency department visits: a starting point. Int J Qual Health Care  2017; 29: 642– 5. doi:10.1093/intqhc/mzx081. Google Scholar CrossRef Search ADS PubMed  14 Male L, Noble A, Atkinson J et al.  . Measuring patient experience: a systematic review to evaluate psychometric properties of patient reported experience measures (PREMs) for emergency care service provision. Int J Qual Health Care  2017; 29: 314– 26. doi:10.1093/intqhc/mzx027. Google Scholar CrossRef Search ADS PubMed  15 Stenlund M, Sjodahl R, Pia Yngman-Uhlin RN. Incidence and potential risk factors for hospital-acquired pneumonia in an emergency department of surgery. Int J Qual Health Care  2017; 29: 290– 4. doi:10.1093/intqhc/mzx018. Google Scholar CrossRef Search ADS PubMed  16 Rehn M, Kruger AJ. Quality improvement in pre-hospital critical care: increased value through research and publication. Scand J Trauma Resusc Emerg Med  2014; 22: 34. Google Scholar CrossRef Search ADS PubMed  17 Snooks H, Evans A, Wells B et al.  . What are the highest priorities for research in emergency prehospital care? Emerg Med J  2009; 26: 549– 50. doi:10.1136/emj.2008.065862. Google Scholar CrossRef Search ADS PubMed  18 Fevang E, Lockey D, Thompson J et al.  . The top five research priorities in physician-provided pre-hospital critical care: a consensus report from a European research collaboration. Scand J Trauma Resusc Emerg Med  2011; 19: 57. doi:10.1186/1757-7241-19-57. Google Scholar CrossRef Search ADS PubMed  19 Mainz J. Defining and classifying clinical indicators for quality improvement. Int J Qual Health Care  2003; 15: 523– 30. Google Scholar CrossRef Search ADS PubMed  20 Donabedian A. The quality of care. How can it be assessed? JAMA  1988; 260: 1743– 8. Google Scholar CrossRef Search ADS PubMed  21 Donabedian A. Evaluating the quality of medical care. Milbank Mem Fund Q  1966; 44: 166– 206. Google Scholar CrossRef Search ADS   22 FAMILIESUSA. Measuring Health Care Quality: An Overview of Quality Measures. www.familiesusa.org. (15 April 2015, date last accessed). 23 Grol R. Improving the quality of medical care: building bridges among professional pride, payer profit, and patient satisfaction. JAMA  2001; 286: 2578– 85. Google Scholar CrossRef Search ADS PubMed  24 Blumenthal D. Part 1: Quality of care—what is it? N Engl J Med  1996; 335: 891– 4. doi:10.1056/NEJM199609193351213. Google Scholar CrossRef Search ADS PubMed  25 El Sayed MJ. Measuring quality in emergency medical services: a review of clinical performance indicators. Emerg Med Int  2012; 2012: 161630. doi:10.1155/2012/161630. Google Scholar CrossRef Search ADS PubMed  26 Kruger AJ, Skogvoll E, Castren M et al.  . ScanDoc Phase 1a Study Group. Scandinavian pre-hospital physician-manned Emergency Medical Services—same concept across borders? Resuscitation  2010; 81: 427– 33. doi:10.1016/j.resuscitation.2009.12.019. Google Scholar CrossRef Search ADS PubMed  27 Moher D, Liberati A, Tetzlaff J et al.  . Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med  2009; 6: e1000097. doi:10.1371/journal.pmed.1000097. Google Scholar CrossRef Search ADS PubMed  28 G K. ADACOPTER-Aufzeichnungen einer Entwicklung München. Werner Wolfsfellner Verlag; 2002. 29 Kruger AJ, Lockey D, Kurola J et al.  . A consensus-based template for documenting and reporting in physician-staffed pre-hospital services. Scand J Trauma Resusc Emerg Med  2011; 19: 71. doi:10.1186/1757-7241-19-71. Google Scholar CrossRef Search ADS PubMed  30 Akin Paker S, Dagar S, Gunay E et al.  . Assessment of prehospital medical care for the patients transported to emergency department by ambulance. Turk J Emerg Med  2015; 15: 122– 5. doi:10.1016/j.tjem.2015.11.005. Google Scholar CrossRef Search ADS PubMed  31 Anadelic S, Ivancevic N, Bogunovic S. Reaction times as indicators of the quality of expert work of belgrade municipal institutions for emergency medical services. Cent Eur J Med  2013; 8: 90– 5. doi:10.2478/s11536-012-0102-0. 32 Arntz HR, Klatt S, Stern R et al.  . Are emergency physicians’ prehospital diagnoses reliable?. [German]. Anaesthesist  1996; 45: 163– 70. doi:10.1007/s001010050251. Google Scholar CrossRef Search ADS PubMed  33 Breckwoldt J, Muller D, Overbeck M et al.  . Prehospital care of acute coronary syndrome by anaesthetists. Prospective comparison with the care standards of cardiologists. [German]. Anaesthesist  2008; 57: 131– 8. doi:10.1007/s00101-007-1290-x. Google Scholar CrossRef Search ADS PubMed  34 Duchateau FX, Garnier-Connois D, Ricard-Hibon A et al.  . Response time evaluation for emergency medical service as a part of its performance. Emerg Med J  2013; 30: 763– 5. doi:10.1136/emermed-2012-201714. Google Scholar CrossRef Search ADS PubMed  35 Fjaeldstad A, Kirk MH, Knudsen L et al.  . Physician-staffed emergency helicopter reduces transportation time from alarm call to highly specialized centre. Dan Med J  2013; 60: A4666. Google Scholar PubMed  36 Flabouris A. A description of events associated with scene response by helicopter based medical retrieval teams. Injury  2003; 34: 847– 52. doi:10.1016/S0020-1383%2803%2900033-0. Google Scholar CrossRef Search ADS PubMed  37 Helm M, Faul M, Unger T et al.  . Reliability of emergency medical field triage. Exemplified by traffic accident victims. [German]. Anaesthesist  2013; 62: 973– 80. doi:10.1007/s00101-013-2255-x. Google Scholar CrossRef Search ADS PubMed  38 Helm M, Hauke J, Sauermuller G et al.  . About the quality of prehospital emergency ventilation—A prospective study in trauma patients. [German]. Unfallchirurg  1999; 102: 347– 53. doi:10.1007/s001130050416. Google Scholar CrossRef Search ADS PubMed  39 Hennes HJ, Reinhardt T, Otto S et al.  . Prehospital care: the efficacy of prehospital intervention by emergency medicine physicians. [German]. Anaesthesist  1993; 42: 455– 61. Google Scholar PubMed  40 Klemenc-Ketis Z, Tomazin I, Kersnik J. HEMS in Slovenia: one country, four models, different quality outcomes. Air Med J  2012; 31: 298– 304. doi:10.1016/j.amj.2012.09.002. Google Scholar CrossRef Search ADS PubMed  41 Leicht MJ, Melanson SW, Albright D. Air medical tracheal intubation: establishing a threshold for this QA indicator. J Air Med Transp  1991; 10: 15– 20. Google Scholar CrossRef Search ADS PubMed  42 Lossius HM, Soreide E, Hotvedt R et al.  . Prehospital advanced life support provided by specially trained physicians: is there a benefit in terms of life years gained? Acta Anaesthesiol Scand  2002; 46: 771– 8. Google Scholar CrossRef Search ADS PubMed  43 Messelken M, Kehrberger E, Dirks B et al.  . The quality of emergency medical care in Baden-Wurttemberg (Germany): four years in focus. [German, English]. Dtsch Arztebl  2010; 107: 523– 30. doi:10.3238/arztebl.2010.0523. 44 Mikkelsen S, Kruger AJ, Zwisler ST et al.  . Outcome following physician supervised prehospital resuscitation: a retrospective study. BMJ Open  2015; 5: e006167. doi:10.1136/bmjopen-2014-006167. Google Scholar CrossRef Search ADS PubMed  45 Nakstad AR, Heimdal HJ, Strand T et al.  . Incidence of desaturation during prehospital rapid sequence intubation in a physician-based helicopter emergency service. Am J Emerg Med  2011; 29: 639– 44. doi:10.1016/j.ajem.2010.01.013. Google Scholar CrossRef Search ADS PubMed  46 Neukamm J, Grasner JT, Schewe JC et al.  . The impact of response time reliability on CPR incidence and resuscitation success: a benchmark study from the German Resuscitation Registry. Crit Care  2011; 15: R282. doi:10.1186/cc10566. Google Scholar CrossRef Search ADS PubMed  47 Pedersen UG, Jorgensen S, Nielsen SL. A survey of patients’ satisfaction with the Mobile Emergency Care Unit in Copenhagen. [Danish]. Ugeskr Laeger  2006; 168: 3322– 5. Google Scholar PubMed  48 Regel G, Seekamp A, Pohlemann T et al.  . Does the accident patient need to be protected from the emergency doctor?. [German]. Unfallchirurg  1998; 101: 160– 75. doi:10.1007/s001130050250. Google Scholar CrossRef Search ADS PubMed  49 Rognas L, Hansen TM, Kirkegaard H et al.  . Pre-hospital advanced airway management by experienced anaesthesiologists: a prospective descriptive study. Scand J Trauma Resusc Emerg Med  2013; 21: 58. doi:10.1186/1757-7241-21-58. Google Scholar CrossRef Search ADS PubMed  50 Rognas L, Hansen TM, Kirkegaard H et al.  . Anaesthesiologist-provided prehospital airway management in patients with traumatic brain injury: an observational study. Eur J Emerg Med  2014; 21: 418– 23. doi:10.1097/MEJ.0000000000000103. Google Scholar CrossRef Search ADS PubMed  51 Sollid SJ, Lossius HM, Soreide E. Pre-hospital intubation by anaesthesiologists in patients with severe trauma: an audit of a Norwegian helicopter emergency medical service. Scand J Trauma Resusc Emerg Med  2010; 18: 30. doi:10.1186/1757-7241-18-30. Google Scholar CrossRef Search ADS PubMed  52 van der Velden MW, Ringburg AN, Bergs EA et al.  . Prehospital interventions: time wasted or time saved? An observational cohort study of management in initial trauma care. Emerg Med J  2008; 25: 444– 9. doi:10.1136/emj.2007.052662. Google Scholar CrossRef Search ADS PubMed  53 Viergutz T, Schmittner MD, Bess S et al.  . Quality management of patients with severe traumatic brain injury in emergency medicine. [German]. Notfall und Rettungsmedizin  2015; 18: 381– 90. doi:10.1007/s10049-015-0033-9. Google Scholar CrossRef Search ADS   54 Von Knobelsdorff G, Reiferscheid F, Straakholder TM et al.  . Do paramedics and emergency physicians adhere to the algorithm of the European Resuscitation Council (ERC) for advanced cardiovascular life support (ACLS)?. [German]. Intensivmedizin und Notfallmedizin  2004; 41: 22– 8. doi:10.1007/s00390-004-0447-y. Google Scholar CrossRef Search ADS   55 Weltermann B, Vom Eyser D, Kleine-Zander R et al.  . Deployment of emergency physicians for patients with stroke in the Munster area: a cross-sectional study of the quality of medical care in one region. [German]. Dtsch Med Wochenschr  1999; 124: 1192– 6. Google Scholar CrossRef Search ADS PubMed  56 Schlechtriemen T, Lackner CK, Moecke H et al.  . Medizinisches Qualitätsmanagement mit Hilfe ausgewählter Zieldiagnosen. [German]. Notfall Rettungsmedizin  2003; 6: 175– 88. doi:10.1007/s10049-003-0561-6. Google Scholar CrossRef Search ADS   57 Sobo EJ, Andriese S, Stroup C et al.  . Developing indicators for emergency medical services (EMS) system evaluation and quality improvement: a statewide demonstration and planning project. Jt Comm J Qual Improv  2001; 27: 138– 54. Google Scholar PubMed  58 Spaite DW, Criss EA, Valenzuela TD et al.  . Emergency medical service systems research: problems of the past, challenges of the future. Ann Emerg Med  1995; 26: 146– 52. Google Scholar CrossRef Search ADS PubMed  59 Spaite DW, Maio R, Garrison HG et al.  . Emergency Medical Services Outcomes Project (EMSOP) II: developing the foundation and conceptual models for out-of-hospital outcomes research. Ann Emerg Med  2001; 37: 657– 63. Google Scholar CrossRef Search ADS PubMed  60 Maio RF, Garrison HG, Spaite DW et al.  . Emergency medical services outcomes project I (EMSOP I): prioritizing conditions for outcomes research. Ann Emerg Med  1999; 33: 423– 32. Google Scholar CrossRef Search ADS PubMed  61 Busse R, Nimptsch U. Mansky T. Measuring, monitoring, and managing quality in Germany’s hospitals. Health Aff  2009; 28: 294– 304. doi:10.1377/hlthaff.28.2.w294. Google Scholar CrossRef Search ADS   62 DIVI. Gemeinsame Empfehlung der Deutschen Interdisziplinaren Vereinigung fur Intensiv- und Notfallsmedizin (DIVI) und des Vorstandes des Bundesarztekammer zum Qualitatsmanagement in der Notfallsmedizin. [German]. Intensivmed  1995; 32: 387– 8. 63 Moecke H, Schafer J, Herden H et al.  . Das Bundeseinheitliche Rettungsdienstprotokoll-Empfehlung der DIVI. [German]. Intensivmed  1994; 31: 96– 9. 64 Herden H, Moecke H. Bundeseinheitliches Notarzteinsatzprotokoll. [German]. Anasth Intensivmed  1991; 33: 166– 9. 65 Palmer RH. Using health outcomes data to compare plans, networks and providers. Int J Qual Health Care  1998; 10: 477– 83. Google Scholar CrossRef Search ADS PubMed  66 El Sayed MJ. Measuring quality in emergency medical services: a review of clinical performance indicators. Emerg Med Int  2012; 2012: 161630. Google Scholar CrossRef Search ADS PubMed  67 Nilsen JE, Tjelmeland IBM, Halvorsen KS et al.  . Kvalitetsindikatorer i den akuttmedisinske kjeden. [Norwegian]. www.nakos.no: Nasjonal kompetansetjeneste for prehospital akuttmedisin 2015. 68 Department of Health. Emergency care ten years on: reforming emergency care . London: Department of Health, 2007. 69 Cooke M, Fischer J, Mc Leod E et al.   Reducing attendances and waits in emergency departments: a systematic review of present innovations NHS Service and Delivery Organisation 2005. 70 Ebben RH, Vloet LC, Verhofstad MH et al.  . Mintjes-de Groot JA, van Achterberg T. Adherence to guidelines and protocols in the prehospital and emergency care setting: a systematic review. Scand J Trauma Resusc Emerg Med  2013; 21: 9. doi:10.1186/1757-7241-21-9. Google Scholar CrossRef Search ADS PubMed  71 Charpentier S, Sagnes-Raffy C, Cournot M et al.  . Determinants and prognostic impact of compliance with guidelines in reperfusion therapy for ST-segment elevation myocardial infarction: results from the ESTIM Midi-Pyrenees Area. Arch Cardiovasc Dis  2009; 102: 387– 96. doi:10.1016/j.acvd.2009.02.011. Google Scholar CrossRef Search ADS PubMed  72 Aylwin CJ, Konig TC, Brennan NW et al.  . Reduction in critical mortality in urban mass casualty incidents: analysis of triage, surge, and resource use after the London bombings on July 7, 2005. Lancet  2006; 368: 2219– 25. doi:10.1016/S0140-6736(06)69896-6. Google Scholar CrossRef Search ADS PubMed  73 Kirves H, Skrifvars MB, Vahakuopus M et al.  . Adherence to resuscitation guidelines during prehospital care of cardiac arrest patients. Eur J Emerg Med  2007; 14: 75– 81. doi:10.1097/MEJ.0b013e328013f88c. Google Scholar CrossRef Search ADS PubMed  74 Wollersheim H, Burgers J, Grol R. Clinical guidelines to improve patient care. Neth J Med  2005; 63: 188– 92. Google Scholar PubMed  75 Forsner T, Wistedt AA, Brommels M et al.  . An approach to measure compliance to clinical guidelines in psychiatric care. BMC Psychiatry  2008; 8: 64. doi:10.1186/1471-244×-8-64. Google Scholar CrossRef Search ADS PubMed  76 Haugland H, Rehn M, Klepstad P et al.  , EQUIPE collaboration group. Developing quality indicators for physician-staffed emergency medical services: a consensus process. Scand J Trauma Resusc Emerg Med  2017; 25: 14. doi:10.1186/s13049-017-0362-4. Google Scholar CrossRef Search ADS PubMed  77 Kessner DM, Kalk CE, Singer J. Assessing health quality—the case for tracers. N Engl J Med  1973; 288: 189– 94. doi:10.1056/NEJM197301252880406. Google Scholar CrossRef Search ADS PubMed  78 Martin LA, Nelson EC, Lloyd RC, Nolan TW. Whole system measures . Cambridge, Massachusetts: Institute for Healthcare Improvement, 2007. 79 Institute of Medicine. Crossing the quality chasm: a new health system for the twenty-first century . Washington: National Academies Press, 2001. 80 Ogrinc G, Davies L, Goodman D, Batalden P, Davidoff F, Stevens D. SQUIRE 2.0 (Standards for QUality Improvement Reporting Excellence): revised publication guidelines from a detailed consensus process. BMJ Qual Saf  2016; 25: 986– 92. doi:10.1136/bmjqs-2015-004411. Google Scholar CrossRef Search ADS PubMed  © The Author(s) 2018. Published by Oxford University Press in association with the International Society for Quality in Health Care. This is an Open Access article distributed under the terms of the Creative Commons Attribution-NonCommercial-NoDerivs licence (http://creativecommons.org/licenses/by-nc-nd/4.0/), which permits non-commercial reproduction and distribution of the work, in any medium, provided the original work is not altered or transformed in any way, and that the work is properly cited. For commercial re-use, please contact journals.permissions@oup.com

Journal

International Journal for Quality in Health CareOxford University Press

Published: May 15, 2018

There are no references for this article.

You’re reading a free preview. Subscribe to read the entire article.


DeepDyve is your
personal research library

It’s your single place to instantly
discover and read the research
that matters to you.

Enjoy affordable access to
over 18 million articles from more than
15,000 peer-reviewed journals.

All for just $49/month

Explore the DeepDyve Library

Search

Query the DeepDyve database, plus search all of PubMed and Google Scholar seamlessly

Organize

Save any article or search result from DeepDyve, PubMed, and Google Scholar... all in one place.

Access

Get unlimited, online access to over 18 million full-text articles from more than 15,000 scientific journals.

Your journals are on DeepDyve

Read from thousands of the leading scholarly journals from SpringerNature, Elsevier, Wiley-Blackwell, Oxford University Press and more.

All the latest content is available, no embargo periods.

See the journals in your area

DeepDyve

Freelancer

DeepDyve

Pro

Price

FREE

$49/month
$360/year

Save searches from
Google Scholar,
PubMed

Create lists to
organize your research

Export lists, citations

Read DeepDyve articles

Abstract access only

Unlimited access to over
18 million full-text articles

Print

20 pages / month

PDF Discount

20% off