A process for developing standards to promote quality in general practice

A process for developing standards to promote quality in general practice Abstract Background Since 1991, the Royal Australian College of General Practitioners’ (RACGP) Standards for General Practices (the Standards) have provided a framework for quality care, risk management and best practice in the operation of Australian general practices. The Standards are also linked to incentives for general practice remuneration. These Standards were revised in 2017. Objective The objective of this study is to describe the process undertaken to develop the fifth edition Standards published in 2017 to inform future standards development both nationally and internationally. Method A modified Delphi process was deployed to develop the fifth edition Standards. Development was directed by the RACGP and led by an expert panel of GPs and representatives of stakeholder groups who were assisted and facilitated by a team from RACGP. Each draft was released for stakeholder feedback and tested twice before the final version was submitted for approval by the RACGP board. Results Four rounds of consultation and two rounds of piloting were carried out over 32 months. The Standards were redrafted after each round. One hundred and fifty-two individuals and 225 stakeholder groups participated in the development of the Standards. Twenty-three new indicators were recommended and grouped into three sections in a new modular structure that was different from the previous edition. Conclusion The Standards represent the consensus view of national stakeholders on the indicators of quality and safety in Australian general practice and primary care. Delphi technique, general practice, health care, primary care, quality assurance, quality improvement, quality of health care Introduction Quality and safety standards are used in health services across the world as a means of promoting excellence in patient care (1). Accreditation against such standards has been shown to promote leadership, enhance corporate culture and improve clinical performance (2). Since 1991, the Royal Australian College of General Practitioners’ (RACGP) Standards for General Practices (the Standards) have provided a framework for quality care, risk management and best practice in the operation of Australian general practices. The Standards comprise a series of indicators grouped into sections that can be assessed. To press, ~80% of Australian general practices are voluntarily accredited against the Standards (3). Funding of general practice services in Australia is based on a fee for service model administered by the government through Medicare. Accreditation is a prerequisite to access a portion of the payments administered separately as the Practice Incentives Program (PIP). PIP payments incentivize government priorities for general practice (4). Independent agencies accredit general practices against the RACGP Standards. Two surveyors, a GP and a non-GP surveyor with recent and relevant general practice experience and working for one of the agencies, undertake an onsite peer review and assessment of the practice against these standards. All four previous editions of the Standards were developed by an expert committee in conjunction with the Standards Unit (SU), a business unit within the RACGP, and input via stakeholder engagement, consultation and piloting in general practice. In 2013, the expert committee reviewed the fourth edition Standards and identified areas where significant change was required to (i) maintain currency and reflect contemporary general practice, (ii) improve usability by providing flexibility for all general practices (regardless of location, size or patient cohort) and (iii) extend the framework for quality care and risk management to a variety of settings where primary care services are delivered. The fifth edition Standards were developed over 32 months using a modified Delphi process (5,6). The impact of accreditation against standards as a mechanism for promoting quality has been reported in the literature (7–10). However, there is a gap in the literature describing the process of developing such standards. This article details how the fifth edition Standards were developed in Australia. Method A modified Delphi process (5,6) was deployed to draft the fifth edition Standards (Fig. 1). It comprised four rounds of workshops to develop the final version. The process was led by the Standards committee (the expert panel) and the SU, a team employed by the RACGP. Figure 1. View largeDownload slide Modified Delphi process for the development of the fifth edition Standards for General Practices Figure 1. View largeDownload slide Modified Delphi process for the development of the fifth edition Standards for General Practices The expert panel Members were invited to nominate to the panel through a general call to the RACGP membership or were nominated by relevant peak bodies. A representative expert panel included 10 GPs (with clinical, research and academic experience), a practice nurse, a practice manager and a consumer. Five existing members had also been involved in the development of previous editions of the Standards. Panel members were from most Australian states and practice locations including urban, regional and rural sites. The clinical expertise of the panel included general practice, Indigenous health and emergency medicine. Five members of the panel were also accreditation surveyors with experience in the assessment of the Standards in practice. The Standards Unit The SU comprised a manager and project support staff. It supported the expert committee by drafting the Standards, sourcing relevant resources, and engaging with stakeholders including the accreditation agencies. Modified Delphi process for the development of the 5th edition Standards Preparation phase (September 2014–February 2015) The preparation phase comprised five workshops involving the expert panel and the SU. Two of these workshops were facilitated. The workshops canvassed expert opinion on the future models of the Standards and accreditation system; developed the process that would be used to structure the fifth edition Standards; current issues affecting general practice; a review of the available evidence; and an initial analysis and mapping of the fourth edition indicators to the proposed fifth edition structure. The proposed fifth edition structure was available for feedback at the conclusion of this phase and included 124 indicators. Stakeholder consultation and feedback (February–May 2015) Consultation and feedback was sought on the existing fourth edition Standards—current issues impacting on the delivery of general practice and the proposed new structure for the fifth edition. This included not only informal commentary from stakeholders but also more formal discussion in the workshops held across Australia. The workshops were facilitated by the expert panel members supported by the SU and local RACGP faculties. The average number of participants per workshop was eight (range of 5–17). Round 1 (June 2015–April 2016) The stakeholder feedback received during the first consultation was reviewed by the SU using a traffic light system and presented to the expert panel. Suggestions for change were categorized as green if they were appropriate for inclusion, amber if items required further consideration, or red if the suggestion was deemed inappropriate for inclusion or out of scope. A total of eight workshops were held during Round 1. Where unanimous consensus could not be reached, an issue was put to the expert panel for a vote. This process was used throughout the rounds of stakeholder consultation and workshops. During Round 1, an evidence review and benchmarking analysis of comparable national and international standards was undertaken by and presented to the expert panel. The SU reviewed accessible standards relevant to the Australian primary health care system from the UK, New Zealand, Canada, Denmark and Australia against the fourth edition Standards to identify gaps. From this process, the expert panel were asked to consider inclusions in the fifth edition Standards. At the conclusion of the workshops, the first draft of the fifth edition Standards was released to stakeholders for feedback. This preliminary pilot of the draft Standards was conducted concurrently at the end of Round 1. Twelve practices and five surveyors were remunerated to participate in this phase. Although this sample had limited membership, the first pilot aimed to test the following measures for each indicator: feasibility, acceptability, achievability, applicability and ease of assessment. Thresholds were set for each measure as assessed by the surveyors and the practice representatives involved in the mock survey. Indicators needed to be assessed as ‘met’ by a minimum of 75% of the surveyors and practice representatives on feasible, acceptable and achievable measures using a rating of met, not met and partially met; a minimum 50% for ease of assessment using a rating of very difficult, difficult, easy, very easy and not sure. Three of the surveyors were members of the expert panel. The SU held two information sessions (n = 16 attendees) with the surveyors and practices to provide guidance on how the indicators could be assessed. Round 2 (April–October 2016) Stakeholder feedback from the first draft and pilot was reviewed by the SU and presented to the expert panel for consideration using the traffic light system as described in Round 1. The expert panel further considered indicators assessed during the first pilot that did not meet the threshold measures. Qualitative feedback from surveyors and practices was also reviewed. Over two workshops, indicators were added, removed and modified to develop the second draft of the Standards. Stakeholders were then invited to provide formal and informal feedback on the second draft. A larger pilot was then undertaken to field test the second draft using the same measures as Round 1. A workshop and webinar information session were held with the surveyors prior to the commencement of pilot visits. This second field test evaluated the Standards in a variety of general practice settings stratified by rural and urban location; solo, small and large practices based on full time equivalent of GPs; corporate and private business models; and Aboriginal Medical Services. For efficiency, the second pilot involved two concurrent processes: (i) dual-process pilot for practices also undergoing fourth edition accreditation. This process comprised accreditation against the fourth edition Standards and a review of new fifth edition indicators; and (ii) pure fifth edition pilot for practices who have recently completed fourth edition accreditation. This process comprised a mock survey visit and assessment based on the second draft fifth edition Standards. For both processes, practices were required to complete a self-assessment against the draft Standards using an RACGP-developed tool. Round 3 (November 2016–January 2017) Stakeholder feedback from the second draft and pilot was reviewed by the SU and presented to the expert panel for consideration using the traffic light system as per Round 1. Indicators assessed during the pilot that did not meet the threshold measures were further considered by the expert panel. One workshop was held during this stage to edit the third draft. A final targeted review of the last 16 revised indicators was conducted. All surveyors and practices who participated in the pilots were invited to undertake the targeted review. For efficiency, a telephone survey was conducted between practices and surveyors. As well as including free text comments, practices and surveyors used the rating scale noted above when reviewing the revised indicators. Round 4 (January–March 2017) The feedback from the targeted review was reviewed by the SU and presented to the expert panel for consideration. Final edits were made to the third draft. One workshop between the SU and the expert panel was held during this stage. Consensus (March–April 2017) The final draft of the fifth edition Standards and supplementary resources approved by the expert panel was presented to the RACGP board for endorsement. Results The project was completed over 32 months. Preparation phase The revised structure of the fifth edition Standards was developed in the preparation phase (Fig. 1). A modular structure for the fifth edition Standards was adopted by expert panel consensus. Round 1 A total of eight workshops were conducted (Fig. 1; Round 1). During these workshops, the expert panel reviewed all feedback presented by the SU. Consultation yielded feedback from 212 stakeholders comprising 106 questionnaire responses, 44 submissions, 58 workshop attendees and 4 verbal feedback responses. The expert panel developed 19 new indicators and amended 42 existing indicators with the intention to remove duplication and facilitate the move to outcomes-focused indicators (Table 1). The number of indicators that did not meet set thresholds in the first pilot is shown in Table 2 and was reviewed by the expert panel in the second round. Table 1. Summary of the development process of indicators for the fifth edition Standards for General Practices Rounds Changes to the standards Fourth edition standards: 5-part structure with 140 indicators. New modular structure proposed in the preparation phase. Inclusions Exclusions Modified 1 n = 8 workshops Generation of first draft adding new indicators in the fifth edition Standards (n = 19) Removing duplication (n = 8) Merging of fourth edition indicators that shared a similar theme (n = 42) 2 n = 2 workshops Generation of second draft 0 indicators added 0 indicators removed 4 indicators amended after consultation phase and first pilot 3 n = 1 workshops Generation of third draft 3 indicators added 0 indicators removed 1 indicator amended after consultation phase and second pilot 4 n = 1 workshops Generation of final draft 0 indicators added 0 indicators removed The wording of 16 indicators refined after targeted review Consensus Fifth edition Standards final version comprised a new structure with 123 indicators with revised criterion, explanatory headings and the development of supplementary resource guides. In total, there were 23 new indicators, 8 not included from the fourth edition and 47 indicators from the fourth edition merged to create 20 of the fifth edition indicators. Rounds Changes to the standards Fourth edition standards: 5-part structure with 140 indicators. New modular structure proposed in the preparation phase. Inclusions Exclusions Modified 1 n = 8 workshops Generation of first draft adding new indicators in the fifth edition Standards (n = 19) Removing duplication (n = 8) Merging of fourth edition indicators that shared a similar theme (n = 42) 2 n = 2 workshops Generation of second draft 0 indicators added 0 indicators removed 4 indicators amended after consultation phase and first pilot 3 n = 1 workshops Generation of third draft 3 indicators added 0 indicators removed 1 indicator amended after consultation phase and second pilot 4 n = 1 workshops Generation of final draft 0 indicators added 0 indicators removed The wording of 16 indicators refined after targeted review Consensus Fifth edition Standards final version comprised a new structure with 123 indicators with revised criterion, explanatory headings and the development of supplementary resource guides. In total, there were 23 new indicators, 8 not included from the fourth edition and 47 indicators from the fourth edition merged to create 20 of the fifth edition indicators. View Large Table 1. Summary of the development process of indicators for the fifth edition Standards for General Practices Rounds Changes to the standards Fourth edition standards: 5-part structure with 140 indicators. New modular structure proposed in the preparation phase. Inclusions Exclusions Modified 1 n = 8 workshops Generation of first draft adding new indicators in the fifth edition Standards (n = 19) Removing duplication (n = 8) Merging of fourth edition indicators that shared a similar theme (n = 42) 2 n = 2 workshops Generation of second draft 0 indicators added 0 indicators removed 4 indicators amended after consultation phase and first pilot 3 n = 1 workshops Generation of third draft 3 indicators added 0 indicators removed 1 indicator amended after consultation phase and second pilot 4 n = 1 workshops Generation of final draft 0 indicators added 0 indicators removed The wording of 16 indicators refined after targeted review Consensus Fifth edition Standards final version comprised a new structure with 123 indicators with revised criterion, explanatory headings and the development of supplementary resource guides. In total, there were 23 new indicators, 8 not included from the fourth edition and 47 indicators from the fourth edition merged to create 20 of the fifth edition indicators. Rounds Changes to the standards Fourth edition standards: 5-part structure with 140 indicators. New modular structure proposed in the preparation phase. Inclusions Exclusions Modified 1 n = 8 workshops Generation of first draft adding new indicators in the fifth edition Standards (n = 19) Removing duplication (n = 8) Merging of fourth edition indicators that shared a similar theme (n = 42) 2 n = 2 workshops Generation of second draft 0 indicators added 0 indicators removed 4 indicators amended after consultation phase and first pilot 3 n = 1 workshops Generation of third draft 3 indicators added 0 indicators removed 1 indicator amended after consultation phase and second pilot 4 n = 1 workshops Generation of final draft 0 indicators added 0 indicators removed The wording of 16 indicators refined after targeted review Consensus Fifth edition Standards final version comprised a new structure with 123 indicators with revised criterion, explanatory headings and the development of supplementary resource guides. In total, there were 23 new indicators, 8 not included from the fourth edition and 47 indicators from the fourth edition merged to create 20 of the fifth edition indicators. View Large Table 2. Indicators that did not meet the thresholds in the first pilot Threshold Number of indicators Percentage range Indicators that score ≤75% for any of feasible, acceptable, achievable or applicable 7 58–75 Indicators assessed as ≤75% met 3 58–75 Indicators that are ≥50% difficult to assess 4 50–58 Threshold Number of indicators Percentage range Indicators that score ≤75% for any of feasible, acceptable, achievable or applicable 7 58–75 Indicators assessed as ≤75% met 3 58–75 Indicators that are ≥50% difficult to assess 4 50–58 View Large Table 2. Indicators that did not meet the thresholds in the first pilot Threshold Number of indicators Percentage range Indicators that score ≤75% for any of feasible, acceptable, achievable or applicable 7 58–75 Indicators assessed as ≤75% met 3 58–75 Indicators that are ≥50% difficult to assess 4 50–58 Threshold Number of indicators Percentage range Indicators that score ≤75% for any of feasible, acceptable, achievable or applicable 7 58–75 Indicators assessed as ≤75% met 3 58–75 Indicators that are ≥50% difficult to assess 4 50–58 View Large Round 2 The results of the first pilot and consultation on the draft Standards were presented to the expert panel and considered over two workshops (Fig. 1; Round 2). Over 80 stakeholders had provided feedback on the first draft. In total, 330 separate items of feedback were received. The feedback was comprised of 25 submissions, 25 email responses, 22 workshop attendees and 5 other responses (including verbal communications). The feedback could be broadly categorized into expansion, deletion and amendments to the Standards and potential challenges to implementation. The SU analysed the feedback using the traffic light system and presented to the expert panel for review. The expert panel amended four existing indicators in response to piloting and stakeholder feedback (Table 1). The subsequent second draft was piloted in 66 practices, with 34 GP surveyors and 29 co-surveyors completing the second pilot visits. Indicators that did not meet set thresholds for the second pilot (Table 3) were reviewed in Round 3. Table 3. Feedback from GP surveyors and co-surveyors on indicators that did not meet thresholds in the second pilot Threshold Co-surveyors GP surveyors Number of indicators Percentage range Number of indicators Percentage range Indicators assessed as ≤75% met 9 52.4–73.8 15 34–73.6 Indicators that are ≥50% difficult to assess 9 46.6–50 15 17–50 Threshold Co-surveyors GP surveyors Number of indicators Percentage range Number of indicators Percentage range Indicators assessed as ≤75% met 9 52.4–73.8 15 34–73.6 Indicators that are ≥50% difficult to assess 9 46.6–50 15 17–50 View Large Table 3. Feedback from GP surveyors and co-surveyors on indicators that did not meet thresholds in the second pilot Threshold Co-surveyors GP surveyors Number of indicators Percentage range Number of indicators Percentage range Indicators assessed as ≤75% met 9 52.4–73.8 15 34–73.6 Indicators that are ≥50% difficult to assess 9 46.6–50 15 17–50 Threshold Co-surveyors GP surveyors Number of indicators Percentage range Number of indicators Percentage range Indicators assessed as ≤75% met 9 52.4–73.8 15 34–73.6 Indicators that are ≥50% difficult to assess 9 46.6–50 15 17–50 View Large Round 3 As per Round 2, the SU presented feedback from the second pilot and consultation on the second draft to the expert panel. This comprised 650 individual items of feedback from over 56 stakeholders, including 25 organizations and 29 individuals and medico-legal commentators. Public consultation ceased at the end of Round 3. Evaluation of feedback by the expert panel resulted in the generation of the third draft of the Standards (Table 1). Indicators identified as not meeting threshold in the second pilot were evaluated further in a targeted review (Fig. 1; Round 3). Round 4 A total of 34 responses were received from the targeted review. They comprised 5 telephone interviews, 9 email responses from surveyors and 20 email responses from practices. Of the 16 revised indicators, 6 required further consideration by the expert panel. All of these indicators were edited in the third draft. Consensus Edits of the third draft were conducted by the SU and expert panel to produce the final draft of the fifth edition Standards. Panel members were required to make a recommendation in the affirmative or negative and where consensus was not clear the decision was put to a vote. There were three occasions where issues were put to a vote following robust discussion and differences of opinion. Discussion The fifth edition Standards were endorsed by the RACGP board and at the time of writing have been adopted as the RACGP Standards for general practice and primary care in Australia. Whilst elements of the Standards development process have been captured in some of the literature, no comprehensive description of standards development in health care has been published to our knowledge (11–17). The fifth edition Standards were developed using the Delphi technique as deployed in other settings (18–20) with common elements including consensus, expert opinion, identifying opportunities and problem areas to create new standards, developing iterative drafts, piloting and testing (21–25). As described earlier, the approach is an iterative process, seeking and incorporating feedback and enhancing the scope for widespread consultation through as many channels as possible. Limitations Many challenges are known to be associated with the technique, including estimating the number of rounds of feedback, creating or maintaining channels for feedback and adhering to a timeframe for completion. However, Delphi techniques have been widely deployed to achieve consensus and can be modified to meet the needs of the project (26–29). It was challenging to facilitate stakeholder engagement to the point of saturation as well as managing multiple stakeholder expectations. Some stakeholders inappropriately expected that their suggestions would immediately be incorporated without further consideration. Maintaining the momentum of all participants for the duration of the project was a vital component of delivering a set of agreed standards on time and within budget. The project timeframe to develop the fifth edition Standards was significantly longer than that allocated to the fourth edition (3 years as opposed to 12 months). This meant developing the latest edition was more costly and labour intensive. Often the same stakeholders provided feedback in each round. By the end of the process, a saturation of themes was evident and it was apparent that further rounds would not yield new information. At each stage, a summary of feedback was shared with stakeholders. While the heterogeneous composition of the expert panel enabled detailed discussions from varied perspectives, on only three occasions was there a call for a formal vote. Strengths Despite the challenges, the fifth edition Standards incorporated 17 fewer indicators than the previous edition. This was considered an improvement by reducing the burden on those who will ultimately be assessed against the Standards. A number of elements within the process were integral to its success. A consistent membership of the expert panel throughout the process ensured expertise and knowledge was retained. Momentum was maintained through effective leadership and maintaining the commitment of the expert panel and SU to achieve objectives. These elements were especially evident when there were differences of opinion on the indicators. Ongoing consistent support of the board for the duration was vital given the scope of the project as well as effective project management. Conclusion The production of the fifth edition Standards was led by an expert panel through multiple rounds of consultation and piloting. Through engagement nationwide with GPs and other key stakeholders, the fifth edition Standards were deemed feasible, acceptable to the profession and assessable by accreditation surveyors. Declaration Ethical approval: The RACGP National Research and Evaluation Ethics Committee (16-002) granted full ethics approval for this project. Conflicts of interest: None declared. Acknowledgements The authors would like to acknowledge the RACGP Expert Committee—Standards for General Practices and the University of Notre Dame—School of Medicine and Melbourne Clinical School for supporting this study. References 1. Greenfield D , Braithwaite J . Health sector accreditation research: a systematic review . Int J Qual Health Care 2008 ; 20 : 172 – 83 . Google Scholar CrossRef Search ADS PubMed 2. Braithwaite J , Greenfield D , Westbrook J , et al. Health service accreditation as a predictor of clinical and organisational performance: a blinded, random, stratified study . Qual Saf Health Care 2010 ; 19 : 14 – 21 . Google Scholar CrossRef Search ADS PubMed 3. Services AGDoH . 2015–2016 Annual Report . Canberra, Australia , 2016 . 4. Services AGDoH . Practice Incentives Program , 2017 . https://www.humanservices.gov.au/organisations/health-professionals/services/medicare/practice-incentives-program (accessed on 29 August 2017 ). 5. Keeney S , McKenna H , Hasson F. The Delphi Technique in Nursing and Health Research . John Wiley & Sons , 2010 . 6. Hasson F , Keeney S , McKenna H . Research guidelines for the Delphi survey technique . J Adv Nurs 2000 ; 32 : 1008 – 15 . Google Scholar PubMed 7. Andersen MK , Pedersen LB , Siersma V , et al. Accreditation in general practice in Denmark: study protocol for a cluster-randomized controlled trial . Trials 2017 ; 18 : 69 . Google Scholar CrossRef Search ADS PubMed 8. Alkhenizan A , Shaw C . Impact of accreditation on the quality of healthcare services: a systematic review of the literature . Ann Saudi Med 2011 ; 31 : 407 – 16 . Google Scholar CrossRef Search ADS PubMed 9. Pomey MP , Lemieux-Charles L , Champagne F , et al. Does accreditation stimulate change? A study of the impact of the accreditation process on Canadian healthcare organizations . Implement Sci 2010 ; 5 : 31 . Google Scholar CrossRef Search ADS PubMed 10. Greenfield D , Pawsey M , Hinchcliff R , Moldovan M , Braithwaite J . The standard of healthcare accreditation standards: a review of empirical research underpinning their development and impact . BMC Health Serv Res 2012 ; 12 : 329 . Google Scholar CrossRef Search ADS PubMed 11. Standardization IOf . International Organization for Standardization , 2018 . https://www.iso.org/developing-standards.html (accessed on 18 April 2018 ). 12. Institute ANS . Overview of the U.S Standardization System , 2018 . https://www.ansi.org/about_ansi/introduction/introduction?menuid=1 (accessed on 18 April, 2018 ). 13. Australia S . Standards Development , 2018 . https://www.standards.org.au/standards-development (accessed on 18 April 2018 ). 14. Canada SCo . Developing Standards , 2017 . https://www.scc.ca/en/standards/developing-standards (accessed on 18 April 2018 ). 15. Institution TBS . Standards Development , 2018 . https://standardsdevelopment.bsigroup.com/Home/About (accessed on 18 April, 2018 ). 16. Care TISfQiH . Guidelines and Principles for the Development of Health and Social Care Standards . 4th edn . ISQua ; 2014 . 17. Braithwaite J , Westbrook J , Pawsey M , et al. A prospective, multi-method, multi-disciplinary, multi-level, collaborative, social-organisational design for researching health sector accreditation [LP0560737] . BMC Health Serv Res 2006 ; 6 : 113 . Google Scholar CrossRef Search ADS PubMed 18. ni Riain A , Vahey C , Kennedy C , Campbell S , Collins C . Roadmap for developing a national quality indicator set for general practice . Int J Health Care Qual Assur 2015 ; 28 : 382 – 93 . Google Scholar CrossRef Search ADS PubMed 19. Tabrizi JS , Farahbakhsh M , Shahgoli J , et al. Designing excellence and quality model for training centers of primary health care: a Delphi Method study . Iran J Public Health 2015 ; 44 : 1367 – 75 . Google Scholar PubMed 20. Authority HIaQ . National Standards for Safer Better Healthcare . Ireland : Authority HIaQ , 2012 . 21. Bukonda N , Tavrow P , Abdallah H , Hoffner K , Tembo J . Implementing a national hospital accreditation program: the Zambian experience . Int J Qual Health Care 2003 ; 14 : 7 – 16 . 22. El-Jardali F , Hemadeh R , Jaafar M , et al. The impact of accreditation of primary healthcare centers: successes, challenges and policy implications as perceived by healthcare providers and directors in Lebanon . BMC Health Serv Res 2014 ; 14 : 86 . Google Scholar CrossRef Search ADS PubMed 23. Shaw CD . External quality mechanisms for health care: summary of the ExPeRT project on visitatie, accreditation, EFQM and ISO assessment in European Union countries. External Peer Review Techniques. European Foundation for Quality Management. International Organization for Standardization . Int J Qual Health Care 2000 ; 12 : 169 – 75 . Google Scholar CrossRef Search ADS PubMed 24. Hearnshaw H , Harker R , Cheater F , Baker R , Grimshaw G . Expert consensus on the desirable characteristics of review criteria for improvement of health care quality . BMJ Qual Saf 2001 ; 10 : 173 – 8 . Google Scholar CrossRef Search ADS 25. Campbell SM , Braspenning J , Hutchinson A , Marshall M . Research methods used in developing and applying quality indicators in primary care . Qual Saf Health Care 2002 ; 11 : 358 – 64 . Google Scholar CrossRef Search ADS PubMed 26. Mullen PM . Delphi: myths and reality . J Health Organ Manag 2003 ; 17 : 37 – 52 . Google Scholar CrossRef Search ADS PubMed 27. Keeney S , Hasson F , McKenna H . Consulting the oracle: ten lessons from using the Delphi technique in nursing research . J Adv Nurs 2006 ; 53 : 205 – 12 . Google Scholar CrossRef Search ADS PubMed 28. Powell C . The Delphi technique: myths and realities . J Adv Nurs 2003 ; 41 : 376 – 82 . Google Scholar CrossRef Search ADS PubMed 29. Hsu CC , Sandford BA . The Delphi technique: making sense of consensus . Pract Assess Res Eval 2007 ; 12 : 1 – 8 . © The Author(s) 2018. Published by Oxford University Press. This is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/4.0/), which permits non-commercial re-use, distribution, and reproduction in any medium, provided the original work is properly cited. For commercial re-use, please contact journals.permissions@oup.com http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Family Practice Oxford University Press

A process for developing standards to promote quality in general practice

Loading next page...
 
/lp/ou_press/a-process-for-developing-standards-to-promote-quality-in-general-j0QsvfQevh
Publisher
Oxford University Press
Copyright
© The Author(s) 2018. Published by Oxford University Press.
ISSN
0263-2136
eISSN
1460-2229
D.O.I.
10.1093/fampra/cmy049
Publisher site
See Article on Publisher Site

Abstract

Abstract Background Since 1991, the Royal Australian College of General Practitioners’ (RACGP) Standards for General Practices (the Standards) have provided a framework for quality care, risk management and best practice in the operation of Australian general practices. The Standards are also linked to incentives for general practice remuneration. These Standards were revised in 2017. Objective The objective of this study is to describe the process undertaken to develop the fifth edition Standards published in 2017 to inform future standards development both nationally and internationally. Method A modified Delphi process was deployed to develop the fifth edition Standards. Development was directed by the RACGP and led by an expert panel of GPs and representatives of stakeholder groups who were assisted and facilitated by a team from RACGP. Each draft was released for stakeholder feedback and tested twice before the final version was submitted for approval by the RACGP board. Results Four rounds of consultation and two rounds of piloting were carried out over 32 months. The Standards were redrafted after each round. One hundred and fifty-two individuals and 225 stakeholder groups participated in the development of the Standards. Twenty-three new indicators were recommended and grouped into three sections in a new modular structure that was different from the previous edition. Conclusion The Standards represent the consensus view of national stakeholders on the indicators of quality and safety in Australian general practice and primary care. Delphi technique, general practice, health care, primary care, quality assurance, quality improvement, quality of health care Introduction Quality and safety standards are used in health services across the world as a means of promoting excellence in patient care (1). Accreditation against such standards has been shown to promote leadership, enhance corporate culture and improve clinical performance (2). Since 1991, the Royal Australian College of General Practitioners’ (RACGP) Standards for General Practices (the Standards) have provided a framework for quality care, risk management and best practice in the operation of Australian general practices. The Standards comprise a series of indicators grouped into sections that can be assessed. To press, ~80% of Australian general practices are voluntarily accredited against the Standards (3). Funding of general practice services in Australia is based on a fee for service model administered by the government through Medicare. Accreditation is a prerequisite to access a portion of the payments administered separately as the Practice Incentives Program (PIP). PIP payments incentivize government priorities for general practice (4). Independent agencies accredit general practices against the RACGP Standards. Two surveyors, a GP and a non-GP surveyor with recent and relevant general practice experience and working for one of the agencies, undertake an onsite peer review and assessment of the practice against these standards. All four previous editions of the Standards were developed by an expert committee in conjunction with the Standards Unit (SU), a business unit within the RACGP, and input via stakeholder engagement, consultation and piloting in general practice. In 2013, the expert committee reviewed the fourth edition Standards and identified areas where significant change was required to (i) maintain currency and reflect contemporary general practice, (ii) improve usability by providing flexibility for all general practices (regardless of location, size or patient cohort) and (iii) extend the framework for quality care and risk management to a variety of settings where primary care services are delivered. The fifth edition Standards were developed over 32 months using a modified Delphi process (5,6). The impact of accreditation against standards as a mechanism for promoting quality has been reported in the literature (7–10). However, there is a gap in the literature describing the process of developing such standards. This article details how the fifth edition Standards were developed in Australia. Method A modified Delphi process (5,6) was deployed to draft the fifth edition Standards (Fig. 1). It comprised four rounds of workshops to develop the final version. The process was led by the Standards committee (the expert panel) and the SU, a team employed by the RACGP. Figure 1. View largeDownload slide Modified Delphi process for the development of the fifth edition Standards for General Practices Figure 1. View largeDownload slide Modified Delphi process for the development of the fifth edition Standards for General Practices The expert panel Members were invited to nominate to the panel through a general call to the RACGP membership or were nominated by relevant peak bodies. A representative expert panel included 10 GPs (with clinical, research and academic experience), a practice nurse, a practice manager and a consumer. Five existing members had also been involved in the development of previous editions of the Standards. Panel members were from most Australian states and practice locations including urban, regional and rural sites. The clinical expertise of the panel included general practice, Indigenous health and emergency medicine. Five members of the panel were also accreditation surveyors with experience in the assessment of the Standards in practice. The Standards Unit The SU comprised a manager and project support staff. It supported the expert committee by drafting the Standards, sourcing relevant resources, and engaging with stakeholders including the accreditation agencies. Modified Delphi process for the development of the 5th edition Standards Preparation phase (September 2014–February 2015) The preparation phase comprised five workshops involving the expert panel and the SU. Two of these workshops were facilitated. The workshops canvassed expert opinion on the future models of the Standards and accreditation system; developed the process that would be used to structure the fifth edition Standards; current issues affecting general practice; a review of the available evidence; and an initial analysis and mapping of the fourth edition indicators to the proposed fifth edition structure. The proposed fifth edition structure was available for feedback at the conclusion of this phase and included 124 indicators. Stakeholder consultation and feedback (February–May 2015) Consultation and feedback was sought on the existing fourth edition Standards—current issues impacting on the delivery of general practice and the proposed new structure for the fifth edition. This included not only informal commentary from stakeholders but also more formal discussion in the workshops held across Australia. The workshops were facilitated by the expert panel members supported by the SU and local RACGP faculties. The average number of participants per workshop was eight (range of 5–17). Round 1 (June 2015–April 2016) The stakeholder feedback received during the first consultation was reviewed by the SU using a traffic light system and presented to the expert panel. Suggestions for change were categorized as green if they were appropriate for inclusion, amber if items required further consideration, or red if the suggestion was deemed inappropriate for inclusion or out of scope. A total of eight workshops were held during Round 1. Where unanimous consensus could not be reached, an issue was put to the expert panel for a vote. This process was used throughout the rounds of stakeholder consultation and workshops. During Round 1, an evidence review and benchmarking analysis of comparable national and international standards was undertaken by and presented to the expert panel. The SU reviewed accessible standards relevant to the Australian primary health care system from the UK, New Zealand, Canada, Denmark and Australia against the fourth edition Standards to identify gaps. From this process, the expert panel were asked to consider inclusions in the fifth edition Standards. At the conclusion of the workshops, the first draft of the fifth edition Standards was released to stakeholders for feedback. This preliminary pilot of the draft Standards was conducted concurrently at the end of Round 1. Twelve practices and five surveyors were remunerated to participate in this phase. Although this sample had limited membership, the first pilot aimed to test the following measures for each indicator: feasibility, acceptability, achievability, applicability and ease of assessment. Thresholds were set for each measure as assessed by the surveyors and the practice representatives involved in the mock survey. Indicators needed to be assessed as ‘met’ by a minimum of 75% of the surveyors and practice representatives on feasible, acceptable and achievable measures using a rating of met, not met and partially met; a minimum 50% for ease of assessment using a rating of very difficult, difficult, easy, very easy and not sure. Three of the surveyors were members of the expert panel. The SU held two information sessions (n = 16 attendees) with the surveyors and practices to provide guidance on how the indicators could be assessed. Round 2 (April–October 2016) Stakeholder feedback from the first draft and pilot was reviewed by the SU and presented to the expert panel for consideration using the traffic light system as described in Round 1. The expert panel further considered indicators assessed during the first pilot that did not meet the threshold measures. Qualitative feedback from surveyors and practices was also reviewed. Over two workshops, indicators were added, removed and modified to develop the second draft of the Standards. Stakeholders were then invited to provide formal and informal feedback on the second draft. A larger pilot was then undertaken to field test the second draft using the same measures as Round 1. A workshop and webinar information session were held with the surveyors prior to the commencement of pilot visits. This second field test evaluated the Standards in a variety of general practice settings stratified by rural and urban location; solo, small and large practices based on full time equivalent of GPs; corporate and private business models; and Aboriginal Medical Services. For efficiency, the second pilot involved two concurrent processes: (i) dual-process pilot for practices also undergoing fourth edition accreditation. This process comprised accreditation against the fourth edition Standards and a review of new fifth edition indicators; and (ii) pure fifth edition pilot for practices who have recently completed fourth edition accreditation. This process comprised a mock survey visit and assessment based on the second draft fifth edition Standards. For both processes, practices were required to complete a self-assessment against the draft Standards using an RACGP-developed tool. Round 3 (November 2016–January 2017) Stakeholder feedback from the second draft and pilot was reviewed by the SU and presented to the expert panel for consideration using the traffic light system as per Round 1. Indicators assessed during the pilot that did not meet the threshold measures were further considered by the expert panel. One workshop was held during this stage to edit the third draft. A final targeted review of the last 16 revised indicators was conducted. All surveyors and practices who participated in the pilots were invited to undertake the targeted review. For efficiency, a telephone survey was conducted between practices and surveyors. As well as including free text comments, practices and surveyors used the rating scale noted above when reviewing the revised indicators. Round 4 (January–March 2017) The feedback from the targeted review was reviewed by the SU and presented to the expert panel for consideration. Final edits were made to the third draft. One workshop between the SU and the expert panel was held during this stage. Consensus (March–April 2017) The final draft of the fifth edition Standards and supplementary resources approved by the expert panel was presented to the RACGP board for endorsement. Results The project was completed over 32 months. Preparation phase The revised structure of the fifth edition Standards was developed in the preparation phase (Fig. 1). A modular structure for the fifth edition Standards was adopted by expert panel consensus. Round 1 A total of eight workshops were conducted (Fig. 1; Round 1). During these workshops, the expert panel reviewed all feedback presented by the SU. Consultation yielded feedback from 212 stakeholders comprising 106 questionnaire responses, 44 submissions, 58 workshop attendees and 4 verbal feedback responses. The expert panel developed 19 new indicators and amended 42 existing indicators with the intention to remove duplication and facilitate the move to outcomes-focused indicators (Table 1). The number of indicators that did not meet set thresholds in the first pilot is shown in Table 2 and was reviewed by the expert panel in the second round. Table 1. Summary of the development process of indicators for the fifth edition Standards for General Practices Rounds Changes to the standards Fourth edition standards: 5-part structure with 140 indicators. New modular structure proposed in the preparation phase. Inclusions Exclusions Modified 1 n = 8 workshops Generation of first draft adding new indicators in the fifth edition Standards (n = 19) Removing duplication (n = 8) Merging of fourth edition indicators that shared a similar theme (n = 42) 2 n = 2 workshops Generation of second draft 0 indicators added 0 indicators removed 4 indicators amended after consultation phase and first pilot 3 n = 1 workshops Generation of third draft 3 indicators added 0 indicators removed 1 indicator amended after consultation phase and second pilot 4 n = 1 workshops Generation of final draft 0 indicators added 0 indicators removed The wording of 16 indicators refined after targeted review Consensus Fifth edition Standards final version comprised a new structure with 123 indicators with revised criterion, explanatory headings and the development of supplementary resource guides. In total, there were 23 new indicators, 8 not included from the fourth edition and 47 indicators from the fourth edition merged to create 20 of the fifth edition indicators. Rounds Changes to the standards Fourth edition standards: 5-part structure with 140 indicators. New modular structure proposed in the preparation phase. Inclusions Exclusions Modified 1 n = 8 workshops Generation of first draft adding new indicators in the fifth edition Standards (n = 19) Removing duplication (n = 8) Merging of fourth edition indicators that shared a similar theme (n = 42) 2 n = 2 workshops Generation of second draft 0 indicators added 0 indicators removed 4 indicators amended after consultation phase and first pilot 3 n = 1 workshops Generation of third draft 3 indicators added 0 indicators removed 1 indicator amended after consultation phase and second pilot 4 n = 1 workshops Generation of final draft 0 indicators added 0 indicators removed The wording of 16 indicators refined after targeted review Consensus Fifth edition Standards final version comprised a new structure with 123 indicators with revised criterion, explanatory headings and the development of supplementary resource guides. In total, there were 23 new indicators, 8 not included from the fourth edition and 47 indicators from the fourth edition merged to create 20 of the fifth edition indicators. View Large Table 1. Summary of the development process of indicators for the fifth edition Standards for General Practices Rounds Changes to the standards Fourth edition standards: 5-part structure with 140 indicators. New modular structure proposed in the preparation phase. Inclusions Exclusions Modified 1 n = 8 workshops Generation of first draft adding new indicators in the fifth edition Standards (n = 19) Removing duplication (n = 8) Merging of fourth edition indicators that shared a similar theme (n = 42) 2 n = 2 workshops Generation of second draft 0 indicators added 0 indicators removed 4 indicators amended after consultation phase and first pilot 3 n = 1 workshops Generation of third draft 3 indicators added 0 indicators removed 1 indicator amended after consultation phase and second pilot 4 n = 1 workshops Generation of final draft 0 indicators added 0 indicators removed The wording of 16 indicators refined after targeted review Consensus Fifth edition Standards final version comprised a new structure with 123 indicators with revised criterion, explanatory headings and the development of supplementary resource guides. In total, there were 23 new indicators, 8 not included from the fourth edition and 47 indicators from the fourth edition merged to create 20 of the fifth edition indicators. Rounds Changes to the standards Fourth edition standards: 5-part structure with 140 indicators. New modular structure proposed in the preparation phase. Inclusions Exclusions Modified 1 n = 8 workshops Generation of first draft adding new indicators in the fifth edition Standards (n = 19) Removing duplication (n = 8) Merging of fourth edition indicators that shared a similar theme (n = 42) 2 n = 2 workshops Generation of second draft 0 indicators added 0 indicators removed 4 indicators amended after consultation phase and first pilot 3 n = 1 workshops Generation of third draft 3 indicators added 0 indicators removed 1 indicator amended after consultation phase and second pilot 4 n = 1 workshops Generation of final draft 0 indicators added 0 indicators removed The wording of 16 indicators refined after targeted review Consensus Fifth edition Standards final version comprised a new structure with 123 indicators with revised criterion, explanatory headings and the development of supplementary resource guides. In total, there were 23 new indicators, 8 not included from the fourth edition and 47 indicators from the fourth edition merged to create 20 of the fifth edition indicators. View Large Table 2. Indicators that did not meet the thresholds in the first pilot Threshold Number of indicators Percentage range Indicators that score ≤75% for any of feasible, acceptable, achievable or applicable 7 58–75 Indicators assessed as ≤75% met 3 58–75 Indicators that are ≥50% difficult to assess 4 50–58 Threshold Number of indicators Percentage range Indicators that score ≤75% for any of feasible, acceptable, achievable or applicable 7 58–75 Indicators assessed as ≤75% met 3 58–75 Indicators that are ≥50% difficult to assess 4 50–58 View Large Table 2. Indicators that did not meet the thresholds in the first pilot Threshold Number of indicators Percentage range Indicators that score ≤75% for any of feasible, acceptable, achievable or applicable 7 58–75 Indicators assessed as ≤75% met 3 58–75 Indicators that are ≥50% difficult to assess 4 50–58 Threshold Number of indicators Percentage range Indicators that score ≤75% for any of feasible, acceptable, achievable or applicable 7 58–75 Indicators assessed as ≤75% met 3 58–75 Indicators that are ≥50% difficult to assess 4 50–58 View Large Round 2 The results of the first pilot and consultation on the draft Standards were presented to the expert panel and considered over two workshops (Fig. 1; Round 2). Over 80 stakeholders had provided feedback on the first draft. In total, 330 separate items of feedback were received. The feedback was comprised of 25 submissions, 25 email responses, 22 workshop attendees and 5 other responses (including verbal communications). The feedback could be broadly categorized into expansion, deletion and amendments to the Standards and potential challenges to implementation. The SU analysed the feedback using the traffic light system and presented to the expert panel for review. The expert panel amended four existing indicators in response to piloting and stakeholder feedback (Table 1). The subsequent second draft was piloted in 66 practices, with 34 GP surveyors and 29 co-surveyors completing the second pilot visits. Indicators that did not meet set thresholds for the second pilot (Table 3) were reviewed in Round 3. Table 3. Feedback from GP surveyors and co-surveyors on indicators that did not meet thresholds in the second pilot Threshold Co-surveyors GP surveyors Number of indicators Percentage range Number of indicators Percentage range Indicators assessed as ≤75% met 9 52.4–73.8 15 34–73.6 Indicators that are ≥50% difficult to assess 9 46.6–50 15 17–50 Threshold Co-surveyors GP surveyors Number of indicators Percentage range Number of indicators Percentage range Indicators assessed as ≤75% met 9 52.4–73.8 15 34–73.6 Indicators that are ≥50% difficult to assess 9 46.6–50 15 17–50 View Large Table 3. Feedback from GP surveyors and co-surveyors on indicators that did not meet thresholds in the second pilot Threshold Co-surveyors GP surveyors Number of indicators Percentage range Number of indicators Percentage range Indicators assessed as ≤75% met 9 52.4–73.8 15 34–73.6 Indicators that are ≥50% difficult to assess 9 46.6–50 15 17–50 Threshold Co-surveyors GP surveyors Number of indicators Percentage range Number of indicators Percentage range Indicators assessed as ≤75% met 9 52.4–73.8 15 34–73.6 Indicators that are ≥50% difficult to assess 9 46.6–50 15 17–50 View Large Round 3 As per Round 2, the SU presented feedback from the second pilot and consultation on the second draft to the expert panel. This comprised 650 individual items of feedback from over 56 stakeholders, including 25 organizations and 29 individuals and medico-legal commentators. Public consultation ceased at the end of Round 3. Evaluation of feedback by the expert panel resulted in the generation of the third draft of the Standards (Table 1). Indicators identified as not meeting threshold in the second pilot were evaluated further in a targeted review (Fig. 1; Round 3). Round 4 A total of 34 responses were received from the targeted review. They comprised 5 telephone interviews, 9 email responses from surveyors and 20 email responses from practices. Of the 16 revised indicators, 6 required further consideration by the expert panel. All of these indicators were edited in the third draft. Consensus Edits of the third draft were conducted by the SU and expert panel to produce the final draft of the fifth edition Standards. Panel members were required to make a recommendation in the affirmative or negative and where consensus was not clear the decision was put to a vote. There were three occasions where issues were put to a vote following robust discussion and differences of opinion. Discussion The fifth edition Standards were endorsed by the RACGP board and at the time of writing have been adopted as the RACGP Standards for general practice and primary care in Australia. Whilst elements of the Standards development process have been captured in some of the literature, no comprehensive description of standards development in health care has been published to our knowledge (11–17). The fifth edition Standards were developed using the Delphi technique as deployed in other settings (18–20) with common elements including consensus, expert opinion, identifying opportunities and problem areas to create new standards, developing iterative drafts, piloting and testing (21–25). As described earlier, the approach is an iterative process, seeking and incorporating feedback and enhancing the scope for widespread consultation through as many channels as possible. Limitations Many challenges are known to be associated with the technique, including estimating the number of rounds of feedback, creating or maintaining channels for feedback and adhering to a timeframe for completion. However, Delphi techniques have been widely deployed to achieve consensus and can be modified to meet the needs of the project (26–29). It was challenging to facilitate stakeholder engagement to the point of saturation as well as managing multiple stakeholder expectations. Some stakeholders inappropriately expected that their suggestions would immediately be incorporated without further consideration. Maintaining the momentum of all participants for the duration of the project was a vital component of delivering a set of agreed standards on time and within budget. The project timeframe to develop the fifth edition Standards was significantly longer than that allocated to the fourth edition (3 years as opposed to 12 months). This meant developing the latest edition was more costly and labour intensive. Often the same stakeholders provided feedback in each round. By the end of the process, a saturation of themes was evident and it was apparent that further rounds would not yield new information. At each stage, a summary of feedback was shared with stakeholders. While the heterogeneous composition of the expert panel enabled detailed discussions from varied perspectives, on only three occasions was there a call for a formal vote. Strengths Despite the challenges, the fifth edition Standards incorporated 17 fewer indicators than the previous edition. This was considered an improvement by reducing the burden on those who will ultimately be assessed against the Standards. A number of elements within the process were integral to its success. A consistent membership of the expert panel throughout the process ensured expertise and knowledge was retained. Momentum was maintained through effective leadership and maintaining the commitment of the expert panel and SU to achieve objectives. These elements were especially evident when there were differences of opinion on the indicators. Ongoing consistent support of the board for the duration was vital given the scope of the project as well as effective project management. Conclusion The production of the fifth edition Standards was led by an expert panel through multiple rounds of consultation and piloting. Through engagement nationwide with GPs and other key stakeholders, the fifth edition Standards were deemed feasible, acceptable to the profession and assessable by accreditation surveyors. Declaration Ethical approval: The RACGP National Research and Evaluation Ethics Committee (16-002) granted full ethics approval for this project. Conflicts of interest: None declared. Acknowledgements The authors would like to acknowledge the RACGP Expert Committee—Standards for General Practices and the University of Notre Dame—School of Medicine and Melbourne Clinical School for supporting this study. References 1. Greenfield D , Braithwaite J . Health sector accreditation research: a systematic review . Int J Qual Health Care 2008 ; 20 : 172 – 83 . Google Scholar CrossRef Search ADS PubMed 2. Braithwaite J , Greenfield D , Westbrook J , et al. Health service accreditation as a predictor of clinical and organisational performance: a blinded, random, stratified study . Qual Saf Health Care 2010 ; 19 : 14 – 21 . Google Scholar CrossRef Search ADS PubMed 3. Services AGDoH . 2015–2016 Annual Report . Canberra, Australia , 2016 . 4. Services AGDoH . Practice Incentives Program , 2017 . https://www.humanservices.gov.au/organisations/health-professionals/services/medicare/practice-incentives-program (accessed on 29 August 2017 ). 5. Keeney S , McKenna H , Hasson F. The Delphi Technique in Nursing and Health Research . John Wiley & Sons , 2010 . 6. Hasson F , Keeney S , McKenna H . Research guidelines for the Delphi survey technique . J Adv Nurs 2000 ; 32 : 1008 – 15 . Google Scholar PubMed 7. Andersen MK , Pedersen LB , Siersma V , et al. Accreditation in general practice in Denmark: study protocol for a cluster-randomized controlled trial . Trials 2017 ; 18 : 69 . Google Scholar CrossRef Search ADS PubMed 8. Alkhenizan A , Shaw C . Impact of accreditation on the quality of healthcare services: a systematic review of the literature . Ann Saudi Med 2011 ; 31 : 407 – 16 . Google Scholar CrossRef Search ADS PubMed 9. Pomey MP , Lemieux-Charles L , Champagne F , et al. Does accreditation stimulate change? A study of the impact of the accreditation process on Canadian healthcare organizations . Implement Sci 2010 ; 5 : 31 . Google Scholar CrossRef Search ADS PubMed 10. Greenfield D , Pawsey M , Hinchcliff R , Moldovan M , Braithwaite J . The standard of healthcare accreditation standards: a review of empirical research underpinning their development and impact . BMC Health Serv Res 2012 ; 12 : 329 . Google Scholar CrossRef Search ADS PubMed 11. Standardization IOf . International Organization for Standardization , 2018 . https://www.iso.org/developing-standards.html (accessed on 18 April 2018 ). 12. Institute ANS . Overview of the U.S Standardization System , 2018 . https://www.ansi.org/about_ansi/introduction/introduction?menuid=1 (accessed on 18 April, 2018 ). 13. Australia S . Standards Development , 2018 . https://www.standards.org.au/standards-development (accessed on 18 April 2018 ). 14. Canada SCo . Developing Standards , 2017 . https://www.scc.ca/en/standards/developing-standards (accessed on 18 April 2018 ). 15. Institution TBS . Standards Development , 2018 . https://standardsdevelopment.bsigroup.com/Home/About (accessed on 18 April, 2018 ). 16. Care TISfQiH . Guidelines and Principles for the Development of Health and Social Care Standards . 4th edn . ISQua ; 2014 . 17. Braithwaite J , Westbrook J , Pawsey M , et al. A prospective, multi-method, multi-disciplinary, multi-level, collaborative, social-organisational design for researching health sector accreditation [LP0560737] . BMC Health Serv Res 2006 ; 6 : 113 . Google Scholar CrossRef Search ADS PubMed 18. ni Riain A , Vahey C , Kennedy C , Campbell S , Collins C . Roadmap for developing a national quality indicator set for general practice . Int J Health Care Qual Assur 2015 ; 28 : 382 – 93 . Google Scholar CrossRef Search ADS PubMed 19. Tabrizi JS , Farahbakhsh M , Shahgoli J , et al. Designing excellence and quality model for training centers of primary health care: a Delphi Method study . Iran J Public Health 2015 ; 44 : 1367 – 75 . Google Scholar PubMed 20. Authority HIaQ . National Standards for Safer Better Healthcare . Ireland : Authority HIaQ , 2012 . 21. Bukonda N , Tavrow P , Abdallah H , Hoffner K , Tembo J . Implementing a national hospital accreditation program: the Zambian experience . Int J Qual Health Care 2003 ; 14 : 7 – 16 . 22. El-Jardali F , Hemadeh R , Jaafar M , et al. The impact of accreditation of primary healthcare centers: successes, challenges and policy implications as perceived by healthcare providers and directors in Lebanon . BMC Health Serv Res 2014 ; 14 : 86 . Google Scholar CrossRef Search ADS PubMed 23. Shaw CD . External quality mechanisms for health care: summary of the ExPeRT project on visitatie, accreditation, EFQM and ISO assessment in European Union countries. External Peer Review Techniques. European Foundation for Quality Management. International Organization for Standardization . Int J Qual Health Care 2000 ; 12 : 169 – 75 . Google Scholar CrossRef Search ADS PubMed 24. Hearnshaw H , Harker R , Cheater F , Baker R , Grimshaw G . Expert consensus on the desirable characteristics of review criteria for improvement of health care quality . BMJ Qual Saf 2001 ; 10 : 173 – 8 . Google Scholar CrossRef Search ADS 25. Campbell SM , Braspenning J , Hutchinson A , Marshall M . Research methods used in developing and applying quality indicators in primary care . Qual Saf Health Care 2002 ; 11 : 358 – 64 . Google Scholar CrossRef Search ADS PubMed 26. Mullen PM . Delphi: myths and reality . J Health Organ Manag 2003 ; 17 : 37 – 52 . Google Scholar CrossRef Search ADS PubMed 27. Keeney S , Hasson F , McKenna H . Consulting the oracle: ten lessons from using the Delphi technique in nursing research . J Adv Nurs 2006 ; 53 : 205 – 12 . Google Scholar CrossRef Search ADS PubMed 28. Powell C . The Delphi technique: myths and realities . J Adv Nurs 2003 ; 41 : 376 – 82 . Google Scholar CrossRef Search ADS PubMed 29. Hsu CC , Sandford BA . The Delphi technique: making sense of consensus . Pract Assess Res Eval 2007 ; 12 : 1 – 8 . © The Author(s) 2018. Published by Oxford University Press. This is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/4.0/), which permits non-commercial re-use, distribution, and reproduction in any medium, provided the original work is properly cited. For commercial re-use, please contact journals.permissions@oup.com

Journal

Family PracticeOxford University Press

Published: Jun 2, 2018

There are no references for this article.

You’re reading a free preview. Subscribe to read the entire article.


DeepDyve is your
personal research library

It’s your single place to instantly
discover and read the research
that matters to you.

Enjoy affordable access to
over 18 million articles from more than
15,000 peer-reviewed journals.

All for just $49/month

Explore the DeepDyve Library

Search

Query the DeepDyve database, plus search all of PubMed and Google Scholar seamlessly

Organize

Save any article or search result from DeepDyve, PubMed, and Google Scholar... all in one place.

Access

Get unlimited, online access to over 18 million full-text articles from more than 15,000 scientific journals.

Your journals are on DeepDyve

Read from thousands of the leading scholarly journals from SpringerNature, Elsevier, Wiley-Blackwell, Oxford University Press and more.

All the latest content is available, no embargo periods.

See the journals in your area

DeepDyve

Freelancer

DeepDyve

Pro

Price

FREE

$49/month
$360/year

Save searches from
Google Scholar,
PubMed

Create lists to
organize your research

Export lists, citations

Read DeepDyve articles

Abstract access only

Unlimited access to over
18 million full-text articles

Print

20 pages / month

PDF Discount

20% off