Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

Evaluating health policy capacity: Learning from international and Australian experience

Evaluating health policy capacity: Learning from international and Australian experience Background: The health sector in Australia faces major challenges that include an ageing population, spiralling health care costs, continuing poor Aboriginal health, and emerging threats to public health. At the same time, the environment for policy-making is becoming increasingly complex. In this context, strong policy capacity – broadly understood as the capacity of government to make "intelligent choices" between policy options – is essential if governments and societies are to address the continuing and emerging problems effectively. Results: This paper explores the question: "What are the factors that contribute to policy capacity in the health sector?" In the absence of health sector-specific research on this topic, a review of Australian and international public sector policy capacity research was undertaken. Studies from the United Kingdom, Canada, New Zealand and Australia were analysed to identify common themes in the research findings. This paper discusses these policy capacity studies in relation to context, models and methods for policy capacity research, elements of policy capacity and recommendations for building capacity. Conclusion: Based on this analysis, the paper discusses the organisational and individual factors that are likely to contribute to health policy capacity, highlights the need for further research in the health sector and points to some of the conceptual and methodological issues that need to be taken into consideration in such research. Background emerging threats to public health, such as new communi- The context for health policy-making cable diseases [8]. Social and health problems are also Health policy-makers in Australia face a broad range of becoming more complex and are increasingly recognised entrenched (and in some cases, worsening) public health as being inter-dependent [9,10]. problems, as well as emerging issues that create challenges for the health system into the future. These include, inter For these challenges to be met, health bureaucracies at alia: continuing poor Aboriginal health [1,2]; widening both Commonwealth and state levels need the capacity to inequalities in health [3]; an ageing population [4]; plan effectively and put policies in place to ensure the increasing health care costs [5]; environmental destabili- health system (including health care and public health) is sation [6]; rising incidence of chronic disease [7]; and organised, funded, delivered and coordinated in the most Page 1 of 15 (page number not for citation purposes) Australia and New Zealand Health Policy 2009, 6:3 http://www.anzhealthpolicy.com/content/6/1/3 effective and sustainable way. Doing so requires overcom- These changes in the policy environment have resulted in ing significant barriers arising from the complexity of greater uncertainty and complexity in policy-making. It is health system financing and program delivery. For exam- increasingly difficult to predict the impact of policy ple, rising chronic disease rates mean that the health sys- changes into the long term. With increasing scale and tem must be re-engineered to focus more on prevention. complexity, balancing the rational/technical and the However, the capacity to invest in prevention initiatives is political aspects of policy development appears increas- hampered by a health system largely focused on episodes ingly difficult. Emerging ideas in the policy literature such of acute care, fee-for-service funding for general practice, as evidence-based policy-making [29,30] and calls for poor coordination between acute and community care greater policy coherence or 'horizontal government' sectors, and workforce shortages in most health profes- [31,32] also bring with them implications for the skills sions [11]. Other obstacles to health reform include the required of policy-makers and for the institutional sup- division of responsibility for health between the Com- ports required to change ways of working. Policy-makers monwealth and the states (which gives rise to coordina- increasingly need skills in coordinated and cooperative tion problems, overlap and duplication of services, gaps policy work, networking, negotiating, collaborating, and in service delivery and cost-shifting) [12-14]; the complex flexible policy implementation [22,33]. mix of public and private insurers and providers [15,16] and the presence of entrenched powerful interests that Policy capacity limit the ability of the state to bring about change [17,18]. From the mid-1990s onwards, there was increasing con- Many policy problems (such as social exclusion and cern about declining policy capacity in many jurisdic- health inequalities) also cross jurisdictional and portfolio tions, and governments in many countries turned their boundaries and require policy-makers and public sector attention to building (or re-building) policy capacity dur- organisations to work in new ways in order to be effective. ing the period from 1995 to 2005. Concerns about policy capacity have been attributed to a number of shifts in pub- Health policy-makers also face challenges in terms of bal- lic sector environments, including public management ancing different health system goals such as equity, effi- reforms such as privatisation, down-sizing and contract- ciency and quality. In an environment of limited ing out as well as shifts in the external environment such resources, there need to be some trade-offs between these as globalisation and changes to the state-society interface goals [19,20]. Ensuring equity through a universal health [34,35]. Concerns continue to be expressed about the pol- care system is an important value on which the Australian icy capacity of public sector agencies, even in wealthy health system is based [21]; however, balancing universal- countries [36]. There is a growing appreciation interna- ity (i.e. equity with respect to both access and levels of tionally that capacity building in policy-making can con- service) with cost containment is a significant issue for tribute significantly to improved public policy outcomes policy-makers. [23,24,26-28,37] although at this stage there is little empirical evidence to show that this is so. Increasingly complex policy environments The policy-making environment is said to have become The term 'policy capacity' is generally used to refer to the more complex over the last few decades due to major capacity of public sector agencies to develop and imple- shifts in the relationship between government and soci- ment "good" policy (although players outside govern- ety, brought about by large-scale forces such as globalisa- ment, including a range of non-government tion (understood as being the increasingly international organisations, universities, research agencies and service nature of the political and economic forces which shape organisations also contribute to the policy capacity of policy) and the increasing use of technology [22-25]. nations and states, the term is most often used to describe Increased reliance on market mechanisms such as man- the policy-making capacity of governments and the public aged competition as policy tools has also resulted in service). However, policy choices are always normative changes to the role of the public service in policy-making. with differential implications for different stakeholders. Greater involvement of third parties (including citizens What constitutes 'good' policy in any particular context and stakeholders) in service delivery results in more com- may be highly contested [38]. plex policy environments with more players outside gov- ernment, and creates challenges in terms of involving The scholarly literature offers a number of different defi- those who are responsible for service provision in policy nitions of policy capacity that highlight different dimen- development [24,26-28]. These developments shape and sions. Among the more useful of these is the definition reflect an increasingly complex health policy environ- offered by Painter and Pierre [1] of policy capacity as "the ment. ability to marshal the necessary resources to make intelli- gent collective choices about and set strategic directions for the allocation of scarce resources to public ends". At Page 2 of 15 (page number not for citation purposes) Australia and New Zealand Health Policy 2009, 6:3 http://www.anzhealthpolicy.com/content/6/1/3 the core of most conceptualisations of policy capacity is United Kingdom, New Zealand, Canada and Australia. A the capacity to make decisions or "intelligent choices" comprehensive literature search was undertaken includ- [1,35,38,39]. A focus on intelligent choice highlights the ing the published health policy, public policy and public quality of public sector policy workers as a central capacity administration literature (searched using library cata- issue, and suggests that recruiting, retaining and develop- logues, journal and thesis databases) as well as govern- ing the "best and brightest" policy workers should be a ment reports and grey literature (sourced from databases, primary focus of capacity building [39]. search engines and government and direct searching of government and organisational websites). The search was Other dimensions of policy capacity suggested by com- restricted to Canada, New Zealand, the United Kingdom mentators in the political science and public administra- and Australia due to their shared political traditions, tion literature include the capacity to utilise resources including the Westminster tradition, and broadly similar [1,40]; the capacity to implement policy decisions [35,38] approaches to health care delivery. The conclusions of this and coordination of the policy-making process across review ought likewise to be regarded as applying to these government [41]. Peters provides the additional insight kinds of jurisdictions. Evaluations of policy capacity or that policy capacity includes the ability to make not only the quality of policy advice at both federal/national and incremental choices, but also more strategic choices state/provincial levels were included in the data set. No which involve larger deviations from the status quo [38]. health sector-specific studies of policy capacity were found, although some studies included health depart- If the capacity of the health sector to develop and deliver ments among the units surveyed. effective health policy (however defined) is to be strength- ened, a better understanding of the elements of policy Table 1 shows the documents included in the comparative capacity is needed. However, empirical research on the review. The studies were analysed according to the follow- topic of health policy capacity is lacking. This paper exam- ing variables: context and themes; models and methods; ines international and Australian experience in conceptu- findings and recommendations for building policy capac- alising and evaluating public sector policy capacity more ity. In each area, common themes and key differences generally and draws out some lessons for building health were drawn out. The analysis was informed by the peer- policy capacity and for further research in this area. reviewed literature on policy capacity and public policy as well as the health policy literature. Based on this analysis, some hypotheses about elements of policy capacity in the Methods This paper reviews publicly available public sector policy health sector were developed. capacity research undertaken between 1995–2005 in the Table 1: Documents included in the review Jurisdiction Document Canada Canadian Government (1996) Strengthening Our Policy Capacity. Report of the Task Force on Strengthening the Policy Capacity of the Federal Government. Manitoba Office of the Provincial Auditor (2001) A Review of the Policy Development Capacity Within Government Departments. United Kingdom UK Cabinet Office (1999) Professional Policy Making for the Twenty-First Century. Strategic Policy Making Team. Bullock, H, Mountford J et al (1999) Better Policy Making. London, Centre for Management and Policy Studies, Cabinet Office New Zealand State Services Commission (1999) Essential Ingredients: Improving the Quality of Policy Advice. State Services Commission (1999) Looping the Loop: Evaluating Outcomes and Other Risky Feats. State Services Commission (1999) High Fliers: Developing High Performing Policy Units. State Services Commission (2000) Pieces of the Puzzle: Machinery of Government and the Quality of Policy Advice. State Services Commission (2000) Gaining Through Training: Developing High Performing Policy Advisors. Wright (1999) Strategic Policy Advice: Improving the Information Base. New Zealand State Services Commission. Wolf (2000) Building Advice: The Craft of the Policy Professional. New Zealand State Services Commission. Australia Australian National Audit Office (2002) Developing Policy Advice. Audit Report No. 21, 2001–2002. Victorian Auditor-General's Office (2004) Report on Public Sector Agencies: Results of Special Reviews and Other Studies, August 2004. Australian Public Service Commission (2004) Connecting Government: Whole of Government Responses to Australia's Priority Challenges. Management Advisory Committee, APSC, Commonwealth of Australia. Page 3 of 15 (page number not for citation purposes) Australia and New Zealand Health Policy 2009, 6:3 http://www.anzhealthpolicy.com/content/6/1/3 to meet the needs of decision-makers; so the emphasis in Results Introduction to the studies: Context, themes, models and this report is largely on improving the outputs of the pol- methods icy process [26]. The methodology involved developing Significant differences existed between the policy capacity criteria for effective policy development capacity in three studies undertaken in the four jurisdictions, in terms of domains (policy process, organisational context and the the context in which they were initiated, the drivers for the "policy product") based on a literature review and consul- review of policy capacity, the actors undertaking the stud- tations with people involved in policy development. ies, the focus and major themes driving the reviews, the These criteria were used as the basis for interviews with models of policy process underpinning the studies, and senior managers in departments and political policy-mak- the methods used for evaluating policy capacity. These ers including Cabinet Ministers and the Policy Secretariat. differences are discussed in the section below. Interviewees were asked to rank the criteria according to their relative importance. Canada The Canadian Government began to focus on rebuilding The United Kingdom A major program of public sector reform began in the policy capacity in the mid-1990s, after two decades of "new public management" and public sector downsizing United Kingdom in 1999 with the Modernising Government were perceived to have resulted in a decline of policy White Paper [27] in which policy-making was identified capacity [42,43]. A task force instigated by the Clerk of the as one of five key areas for reform. Key themes in the Privy Council Office in 1995–96 investigated the policy White Paper included "forward looking" and consultative capacity of the federal government and explored ways in policy-making and "joined-up" government [27]. which it could be strengthened [44]. Task Force data-gath- Improving the use of evidence in policy-making was also ering processes included interviews with officials in most a strong focus of Modernising Government and the subse- departments and roundtables with junior policy officers quent policy capacity studies undertaken in the UK [[47] and external policy research experts [44]. Major themes of p.13]. Academic literature from the UK has also focused the ensuing report included strengthening policy capacity extensively on both joined-up government [48,49] and across government for dealing with horizontal issues (as improving the utilisation of research in policy-making well as within departments), workforce development, and [30,47]. building more effective links with the policy research community [44]. The study draws extensively on an anal- The UK Cabinet Office followed Modernising Government ysis of seven "policy functions": theoretical research; sta- with a study of "professional policy-making" [37], which tistics, applied research and modelling; environmental involved an audit of policy-making using a model of scanning, trends analysis and forecasting; policy analysis "modernised policy-making". This study involved collect- and advice; consultation and managing relations; com- ing case studies of "good practice" policy-making, as well munications; and program design, implementation, mon- as interviews with officials and advisers, focus groups with itoring and evaluation [44]. This approach aligns closely policy staff and "recent leavers", and a training needs anal- with rational, "stagist" models of the policy process, such ysis involving ministers and policy staff [37]. A further as the Bridgman and Davis [45] policy cycle. study by the Cabinet Office entitled Better Policy-Making [50] examined approaches to modernising the policy The themes identified in the pivotal Canadian Govern- process, and looked more closely at enabling factors and ment Task Force report have been echoed in the scholarly barriers to change. A survey of civil servants in all ministe- literature from Canada, such as strengthening coordina- rial departments was undertaken for this study. tion and coherence across government [31,32], and policy analytic capacity (in particular the challenges of recruit- The UK policy capacity initiative was unusual in several ment and retention) [36,39,43]. This report also led to a respects. First, it was initiated by politicians rather than by focus on strengthening policy research capacity, and the public servants – this gave it a breadth and scope beyond Policy Research Initiative was subsequently established to the studies done in other countries. It also ensured that improve the generation and use of research on cross-cut- the judgements of "good" policy reflected the political pri- ting policy issues [46]. orities and orientation of the government of the day. Sec- ond, it was based on a complex conceptual model of Also in Canada, the Office of the Provincial Auditor in policy-making that included a set of characteristics of Manitoba Province undertook a smaller scale evaluation "modernised" policy-making with three overarching of the policy process, organisational structures and proc- themes and nine core competencies. The UK studies have esses and policy outputs in 2001. This initiative seems to a strong focus on competencies, many of which (such as have been driven by concerns about the inability of brief- "outward looking" and "joined up") seem to be used not ings and policy documents produced by the bureaucracy only to describe the capabilities of policy staff but to Page 4 of 15 (page number not for citation purposes) Australia and New Zealand Health Policy 2009, 6:3 http://www.anzhealthpolicy.com/content/6/1/3 describe organisational capabilities, and, even more advising [57]. The use of PMRs seems to have petered out broadly, to characterise the approach to "modernised pol- after the trials, largely due to a recognition of the limits of icy-making" [[37] p.12]. The conclusions of this survey, applying program evaluation principles to the inherently however, appear to be largely embedded in the methods political policy process [57,59]. and the assumptions that went into the methods, particu- larly with respect to the desired policy competencies. At a later stage, performance audits of policy advice sys- tems and processes were undertaken by the Australian New Zealand National Audit Office [60] and the State of Victoria's Audi- In New Zealand, the State Services Commission under- tor-General's Office [61]. These audits focused largely on took a suite of studies in 1998–99 as part of a project improving the process through which policy is made and called "Improving the Quality of Policy Advice". As the the quality of policy advice outputs such as briefings and title indicates, the project was driven by concerns about submissions. The scope of these studies is relatively lim- the quality of policy advice developed through the ited in comparison with some of the other studies consid- bureaucracy [42]. The focus of this project was largely on ered in this paper. The ANAO report was based on the quality of inputs (particularly information and discussions with departmental employees and their staff, research, but also evaluation, consultation and coordina- case studies of quality management arrangements and tion) to the policy process [24,51-56]. Key themes document analysis [[60] p.13]. Quality management sys- included: lack of effective use of information, evidence tems for the provision of policy advice were assessed and evaluation; emphasis on short-term outputs and "fast against criteria based on a literature review and other solutions" in policy development at the expense of a sources and a set of "better practice principles" for longer term, more strategic focus; shortages of appropri- enhancing management and quality assurance of the pro- ately skilled staff and lack of policy knowledge and expe- vision of policy advice were developed. The Victorian rience [24,51-56]. This project was strongly based in the audit assessed policy briefs and the policy development public policy literature with a number of comprehensive process against a set of "good practice" principles devel- literature reviews [51,52]. Data collection involved inter- oped using a literature review and expert interviews [61]. views with central agency officials and chief executives Like the New Zealand studies, these performance audits and policy managers [24]. One of the working papers also were focused on the development of policy advice and examined six high performing policy units, using case their recommendations were directed towards the partic- studies and interviews [54]. ular departments audited. Although the New Zealand studies were quite broad in The Australian Public Service Commission undertook a terms of the issues covered, the emphasis on improving major initiative to improve inter-sectoral approaches to policy advice meant that the recommendations of the policy-making, culminating in the Connecting Government studies were generally directed towards senior managers report in 2004 [23]. The report was based on a compre- in departments, and were therefore narrower in scope hensive literature review and a series of case studies of than the some of the capacity building strategies devel- "whole of government" policy-making. Although the oped in Canada and the UK. focus of this report is on whole-of-government responses to major cross-portfolio issues, it identifies a number of Australia areas where capacity could be improved and some of the Australian experience with evaluating policy capacity findings relating to institutional culture and the skills set began in the mid-1990s with a series of trials in evaluating for policy-makers are useful for thinking about depart- the policy advice function of Commonwealth Govern- mental capacity. Unfortunately "whole of government" ment central agencies using Policy Management Reviews was limited to "'whole of Commonwealth Government", (PMRs) based on performance assessment techniques so the challenges of intergovernmental policy coordina- [57,58]. PMRs were an attempt to subject the policy devel- tion were not addressed. opment and advice functions of public sector agencies to Key findings and recommendations for building policy the same scrutiny as that given to administration at the time, and were driven by concerns with outcomes and capacity accountability [57,58]. These trials raised methodological The findings and recommendations of these various pol- debates about the criteria for evaluating policy advice (dis- icy capacity studies point towards two broad areas as the cussed later in this paper). Di Francesco argues that an focus of capacity building: organisational capacity and original intention to evaluate policy advice according to individual competencies. To analyse the elements of pol- its outputs (such as briefs), the policy itself, and the out- icy capacity in each of these areas, the authors created a comes (the impact of policy in the real world) was aban- table with the main analytical themes, and categorised the doned in favour of a focus on the processes of policy Page 5 of 15 (page number not for citation purposes) Australia and New Zealand Health Policy 2009, 6:3 http://www.anzhealthpolicy.com/content/6/1/3 issues and recommendations in the documents accord- tion and evaluation) and it is the overall mix of skills ingly, using an iterative process. which is important for policy capacity. The individual competencies include: Knowledge Studies identified a number of different types of knowl- � Knowledge and experience edge important for policy-making. These included knowl- edge of context, both the context of the problem and of � The practical skills of policy making the policy, including the organisational, political and wider social context [[26], p.37,52]. Various disciplines � Personal attributes such as creativity, intuition and were identified as contributing to policy-making, includ- judgement. ing law, economics, accountancy, statistics, the social sci- ences, project management and information technology The elements of organisational capacity which emerge [37]. Knowledge of systems and developments in other from these studies include: countries and the ability to "learn lessons" from these was also a key competency in developing more "outward look- � Access to and use of information and evidence ing" policy-making identified by the UK Cabinet Office [37]. � Personnel management and workforce development Practical skills of policy-making � Consultation and communication Policy-making skills described in the reports included analytic skills such as the ability to frame problems, � Inter-departmental coordination and networking appraise research evidence, predict the likely conse- quences of policy choices and evaluate associated risks � Implementation [26,37,52]. Skills in the daily work of policy development (such as drafting, researching, consulting, evaluation and � Monitoring, evaluation and review project management) were also mentioned [26,37,52]. High level interpersonal and communication skills were � Strategic management and leadership highlighted in several studies [23,37,52]. The ability to utilise information technology, to manage risks and to � Institutional culture. learn new skills were also highlighted by the UK Cabinet Office [37]. The next section of the paper explores findings of the pol- icy capacity evaluations in each jurisdiction in each of Creativity, intuition and judgement these domains. These attributes were emphasised in one of the UK reports, which described the need for policy-makers to be Individual competencies "flexible and innovative", willing to question the status The knowledge, skills and capabilities of policy staff were quo and prepared to try out new ideas and work in new frequently referred to in all the studies. In the Manitoba ways [37]. The Canadian Task Force also discussed the report, the knowledge and skills of policy staff were iden- need for intellectual curiosity, intuition and the ability to tified as being the most important factor contributing to be "comfortable with the uncertainties of policy-making" excellence in policy development and were also the most [[44], p. 24]. The Manitoba study also highlighted the frequently cited area needing improvement [[26], pp. 34– importance of creativity and good judgement [26]. The 36]. Despite this emphasis, however, there was little in- emphasis on these attributes in the studies highlights the depth exploration of the knowledge and skills that policy significance of personal and second-hand experience with practitioners need to do their work and how the presence cases (stories of episodes of policy-making that illustrate or absence of these capabilities impacted on the outputs principles and insights that can then be used to inform or outcomes of policies. practice). Cases become an important source of knowl- edge upon which judgement and intuition are built. In this paper the capabilities of policy-makers described in the studies are considered under the headings of: knowl- Organisational capacity Information and evidence edge; practical skills of policy-making; and creativity, intu- ition and judgement. As the Canadian Government Task The importance of adequate and timely information and Force [44] notes, the skill sets and relevant knowledge evidence to inform policy-making was a strong theme requirements differ for different types of policy personnel across all the studies, and was particularly prominent in (with different roles in policy development, implementa- New Zealand and the United Kingdom. This reflects a Page 6 of 15 (page number not for citation purposes) Australia and New Zealand Health Policy 2009, 6:3 http://www.anzhealthpolicy.com/content/6/1/3 growing emphasis in the published literature on evidence- management infrastructure to facilitate sharing of knowl- based (or evidence-informed) policy-making [see, for edge across the bureaucracy. Strategies proposed included example, [29,30]]. The studies reviewed were concerned policy "knowledge pools" and "knowledge networks". not just with evidence derived from empirical research, The UK report Professional Policy Making for the Twenty First however, but also with other types of evidence, including Century [37] recommended the establishment of a policy information about the circumstances in which policy is knowledge pool including information about policies being made (a distinction that is often not clearly made in (such as policy objectives, impact assessments and evalu- the evidence-based policy literature). ation results, consultation, and evidence drawn upon) in a standard format. Concerns about the use of evidence in policy-making were common. The UK Cabinet Office studies found that evi- Knowledge management was also highlighted as an dence was not always utilised effectively, even when important factor in facilitating joined-up responses in the research had been commissioned by the departments Australian whole-of-government study [23] and identi- themselves [37]. The Canadian Government study also fied as a "better practice principle" in the ANAO audit. The found that relevant statistics for policy-making were not ANAO [[60], p. 81] also suggested including directories of always available and capacity for applied research and subject expertise and information about the policy agenda quantitative modelling varied between departments [[44], as well as information about specific projects in the p. 5]. Models available for forecasting, analysing contin- knowledge pools, and recommended cross-agency net- gencies and predicting scenarios were not utilised as fully works of policy advisers "to identify, research and coordi- as they could have been [[44], p. 5–6]. The Manitoba nate policy based on themes". Office of the Provincial Auditor [26] also found that there was limited use of both quantitative and qualitative anal- Personnel management and workforce development ysis for both issue analysis and for determining policy A common theme in the studies was the importance of an options. adequate supply of highly skilled policy personnel and an appropriate mix of skills within units or departments. The New Zealand State Services Commission identified a Staff shortages presented barriers to longer term and more number of factors affecting the use of information and strategic policy-making in many studies. For example, the evidence, including: adequate time to consult with Canadian Government study [44] noted shortages of key researchers and to commission research where evidence skills, particularly for "policy generalists", and highlighted does not exist; skills in using information; ability to coor- the importance of personnel management. Areas in which dinate information resources between departments; the personnel management needed improvement included production of longer-term and strategic research; and pro- rotation of staff, recruitment processes, performance ductive relationships with external research organisations review, and mobility and variation in experience. The rec- [24]. Ideas for improving the use of evidence include com- ommendations for improvements in these areas were missioning more long-term and strategic research, mostly cast in terms of self-evident principles; few sugges- increasing central-agency expectations and improving tions were made about how to give effect to these princi- coordination across the public service [24]. ples. Many of these ideas in the Canadian and New Zealand Training needs assessment and the provision of training studies are echoed in the UK Cabinet Office study Profes- for new policy staff, policy staff new to the field and non- sional Policy Making for the Twenty First Century, which policy officers engaged in policy work were recommended identified the need to improve the ability of policy-mak- by the Canadian Government [[44], p. 24–30]. Access to ers to use evidence and also to ensure that relevant evi- training in a range of areas (including analytical thinking, dence was available [37]. Strategies to achieve this particular disciplines important for policy work and the included development of systematic research strategies practical skills of policy development, implementation within the departments, coordination of research effort and evaluation) was also identified by senior managers in across government and a range of strategies to improve the Manitoba study as important for improving policy the accessibility of research evidence including establish- competencies [[26], p. 30]. Lack of both time and money ing a "Centre for Evidence-Based Policy" to develop inter- were found to present barriers to training [[26], p. 31]. national policy networks, databases and information resources. The UK Cabinet Office also advocated the The UK initiative recommended that training should establishment of a "policy researcher" role with a particu- involve both political and bureaucratic policy-makers (i.e. lar focus on gathering evidence and presenting it in a for- ministers as well as public servants) [37]; a strategy that mat accessible to policy-makers [37]. Some studies encourages relationship building as well as improved recommended the deliberate development of knowledge knowledge sharing. This was echoed by the ANAO report, Page 7 of 15 (page number not for citation purposes) Australia and New Zealand Health Policy 2009, 6:3 http://www.anzhealthpolicy.com/content/6/1/3 which recommended the establishment of a "senior gov- Inter-departmental coordination and networking ernment network in which ministers, senior government Common findings in the studies included inadequate officials and other senior policy-makers can meet from coordination between departments due to a "silo" men- time to time for focused seminars on top-level manage- tality and few opportunities for collaborative reflection on ment issues" [[60], p. 108]. best practice. Joined-up approaches across departments (for cross-cutting issues), within departments and Consultation and communication between departments, service deliverers and those respon- Timely and comprehensive consultation with a range of sible for implementing policies were a major focus in the stakeholders was highlighted as a key factor contributing UK reports [37]. For these approaches to work, they to good policy outcomes in many studies. Consultation needed to be supported by compatible information sys- with both internal stakeholders (within the department, tems, and by organisational cultures and processes [37]. other departments, and central government) and external Improving communication between people developing stakeholders (including clients, the public, policy-makers and implementing policies in different portfolio areas was in other jurisdictions, professional associations, academ- also seen as critical [37]. Sharing best practice was the ics and researchers) was considered important. most frequently identified "enabler" for modernising pol- icy making in the UK report Better Policy Making [[50], p. The Manitoba study stressed the importance of involving 10]. The Canadian Task Force also recommended the stakeholders – including clients and the public – in policy establishment of interdepartmental policy communities development from the beginning, including during the among senior policy executives to share ideas about best initial data gathering stage [[26], p. 26]. Consultation was practice and the struggles of policy work [[44], p. 14]. one of the core competencies explored by the UK Cabinet Office, which found that although "good practice" com- Implementation munication was reasonably widespread, the resources Only a few references to implementation were found in were not always available to undertake comprehensive the studies reviewed. The lack of attention to implementa- consultations [37]. Issues related to consultation were tion in the policy capacity studies is notable, given the also raised in the Victorian Auditor-General's Office report extensive literature which points to the importance of full [61], which identified weaknesses in consultation plan- consideration of implementation issues during policy ning and recommended the use of project management development [45,62-65]. The Canadian study found poor disciplines to promote more systematic planning for con- links between policy development and implementation sultation. [[44], p. 8]. "Greater consideration to policy implementa- tion" was one of the enablers of change identified in the The ability to keep abreast of international developments UK Cabinet Office report Better Policy Making [[50], p. 10]. and also communicate across jurisdictional boundaries Barriers to the integration of policy development and was considered important by the UK Cabinet Office [37], implementation identified in this report included institu- which recommended raising awareness of the "political tional separation and incompatibility of information and wider context", planning for communication, careful technology [[50], p. 41]. There are few clues in the studies, targeting, coordination, and fostering relationships with however, as to how links between policy development other jurisdictions. Cooperation with "external groups and implementation could be strengthened. Some strate- such as community organisations, businesses and other gies that could be explored in further policy capacity jurisdictions" and "strong external links at the political research include: improving the capacity for piloting and level – ministers, members of parliament, ministerial demonstration; strengthening processes for monitoring staff" were also seen by the Australian Public Service Com- and adjustment of incremental policy development and mission [[23], p. 8] as being important for cross-portfolio implementation; and involving implementation manag- policy work. Collaboration with policy researchers was ers in policy development. highlighted in the Canadian research, which recom- mended opportunities for more exchanges [44]. The New Monitoring, evaluation and review Zealand State Services Commission [24] found that build- Monitoring and evaluation were areas where weaknesses ing effective consultation into the policy advice process were commonly identified, as they tended to be neglected, required investment of sufficient time and resources as of insufficient quality or did not feed into policy develop- well as developing particular competencies including ment. For example, the Canadian Task Force found that negotiation and communication skills. Again, the recom- these functions were frequently separated institutionally mendations for improvements seem to be cast in very gen- from (and therefore poorly integrated with) policy devel- eral terms and there is little evidence on which to base opment [[44], p. 11]. specific strategies. Page 8 of 15 (page number not for citation purposes) Australia and New Zealand Health Policy 2009, 6:3 http://www.anzhealthpolicy.com/content/6/1/3 Outcome evaluation was identified as a significant input Fliers: Developing High Performing Policy Units [54] high- to policy advice by the New Zealand SSC [24], which lighted the importance of leadership and strategic man- found that evaluation was most often used for improving agement for improving the performance of "policy units". delivery and implementation and less frequently contrib- This included "policy leadership" (in terms of developing uted to better policy-making. The neglect of evaluation as a coherent overall direction and policy frameworks) and an input to policy-making was attributed to low demand "management leadership" to provide infrastructure and from ministers, political "short-termism", methodologi- support to policy work [[54], p. 10]. Recommendations cal problems with evaluation, lack of evaluation skills and for developing leadership were generally framed around manipulation of evaluation results for political ends [53]. improvements to personnel management (as described To improve effective outcome evaluation, the SSC recom- above) and institutional culture (see below). mended increasing demand from ministers, central agen- cies and parliament for evaluation information, better Supportive institutional culture specification of outcomes, increasing attention to out- Institutional culture was not a strong (or at least an comes (rather than just outputs), improving incentives for explicit) focus of many of the studies reviewed for this reprioritisation and evaluation and improving the skills of paper, with the exception of those undertaken by the UK policy staff to both carry out evaluations and to manage Cabinet Office, the APSC and the New Zealand SSC. external evaluations [24]. The UK work in particular highlights the importance of The Manitoba audit identified deficiencies in perform- institutional culture in recognising the goals of "modern- ance monitoring of policies and found there was an "ad ised" policy-making, such as a long-term and more strate- hoc", rather than systematic, approach to policy evalua- gic focus [37]. A culture of innovation and preparedness tion. However senior policy managers did not rank evalu- to take risks was considered critical to fostering innova- ation among the most important criteria for policy tion [37]. A "risk averse" culture was identified as a barrier capacity and tended to favour a "selective" approach to to improving policy-making [50]. Some recommenda- policy evaluation (as this was perceived to be a more effi- tions for changing institutional culture included bringing cient use of resources) [[26], pp. 27–28]. in staff from outside the public service, providing second- ments for policy officers and improving networking both The UK Cabinet Office found there were few opportuni- within government and with external agencies and other ties for policy-makers to learn from their own and others' jurisdictions [37]. experience [37]. Encouraging a culture of evaluation and improving the quality of evaluations undertaken were The APSC's Connecting Government report [23] identified a both areas that needed improvement [37]. A major rec- number of features of organisational culture important ommendation arising from the UK initiative was the use for the success of whole-of-government approaches. Like of peer review processes to allow sharing of "good prac- the UK reports, it emphasised the importance of innova- tice" and organisational learning, and also to encourage tion and the ability to manage risk. It also highlighted an cultural change [37]. Peer review was also recommended environment of teamwork and trust and "encouragement by the ANAO [[60], p. 115] and the Victorian Auditor- of the expression of diverse views" and the ability to "bal- General's Office. Other mechanisms suggested by the UK ance the tension between short-term and long-term goals" Cabinet Office [37] for improving policy evaluation [23]. Exposure of public servants to different organisa- included strategic management of the evaluation process, tional cultures (through secondments and networking) better resource allocation, developing a "centre of excel- was recommended as one way of fostering a different lence" devoted to policy evaluation and establishing proc- organisational culture [23]. esses for people implementing policies to feed back information about the effectiveness and acceptability of Clarity of policy direction and policy frameworks was policies. highlighted as important by the Manitoba Office of the Provincial Auditor [[26], pp. 25–6], which described a Strategic management and leadership lack of clear principles or conceptual frameworks for gen- Strategic management and leadership of the policy devel- erating or evaluating policy options – although this was opment process were common themes. The Canadian identified as a very important aspect of policy develop- Government Task Force [[44], p. 11] found that system- ment by senior policy managers. It was also found by the atic management of the policy process was patchy, and the New Zealand SSC to be important for improving the per- need for it was underestimated. Leadership direction and formance of policy units [54]. support was ranked as very important or important by 100% of interviewees in the Manitoba audit of policy capacity [[26], p. 22]. The New Zealand SSC report High Page 9 of 15 (page number not for citation purposes) Australia and New Zealand Health Policy 2009, 6:3 http://www.anzhealthpolicy.com/content/6/1/3 which they interact and the processes by which change at Discussion Implications for policy capacity building and workforce the individual level can affect the organisation – and vice development in the health sector versa. The policy capacity evaluations and audits reviewed in this paper suggest some of the factors likely to contribute Models and methods for policy capacity research to policy capacity in the health sector. Based on the find- There are continuing uncertainties about the conceptuali- ings of this review, some propositions about the organisa- sation of policy capacity and the methods for researching tional structures and processes, organisational cultures it. All the models for conceptualising policy work and the and individual competencies that are likely to contribute methods for evaluating policy capacity used in the studies to health policy capacity were developed by the authors. reviewed in this paper have their own limitations. Some These are listed in Figures 1, 2 and 3. studies used relatively linear models of the policy process and fairly narrowly conceived methodologies (particu- There is clearly a need for further research to explore how larly the state/provincial level audits), which provided lit- these general principles might be applied in the health tle scope for examining the political and contextual sector and to examine more closely the institutional aspects of policy-making. Policy work was often con- arrangements and competencies which create the condi- ceived as the provision of policy advice to political deci- tions for good policy outcomes in the health sector. This sion-makers; an approach which may not capture the full is the focus of research in which the authors are currently range of possible policy engagements and the more active engaged. Questions which might usefully be addressed in role that policy workers often play in shaping policy. such research include: Most of the studies reviewed for this paper used multiple � To what extent do the generic elements of policy capac- data collection methods. In most cases, interviews with ity apply to health policy in particular? senior officials were used to collect data. These were often supplemented by interviews with other players such as � Are some elements of policy capacity more important junior policy officers and external policy research experts than others in the context of health policy development in Canada, recent leavers in the UK and cabinet ministers and implementation? in Manitoba. In some cases interviews were combined with other data collection methods including case studies Grounding health policy capacity research within the of policy episodes or policy units, focus group interviews study of the health policy development process would or round tables, surveys, and training needs analysis. A provide the potential to examine more closely the ena- number of appropriate methods are available and the blers and barriers to good health policy process and ena- choice of particular methods depends on the context and ble more specific recommendations for capacity building focus of the study. Triangulation of different methods to be developed. appears to offer advantages. Jurisdictional differences are clearly important in shaping Analysis of case studies appears to provide a useful way of both policy capacity itself and the sorts of capacity build- exploring the conditions for good policy-making, particu- ing strategies that might be needed. Some factors that larly institutional culture, within context. However, in all might be important include differences in national sys- the studies reviewed that included analysis of case studies, tems (such as unitary governments or federal systems), the cases had been chosen by policy-makers themselves differences in health system contexts, and different insti- and put forward as examples of good practice. Methods tutional arrangements and policy coordination mecha- for policy capacity research could be strengthened by nisms (e.g. the Council of Australian Governments). more careful selection of case studies according to their Further research involving different jurisdictions is potential to illuminate the elements of policy capacity, needed to explore the extent to which comparative differ- both positive and negative. ences matter in the context of health policy capacity. None of the studies included in this review attempted to A further issue which should be addressed in future evaluate policy capacity against the outcomes of the pol- research is the relationship between individual and organ- icy process, although some studies evaluated policy capac- isational capacity. Clearly they are closely interrelated. ity in terms of outputs, such as the quality of policy briefs The capacity of individuals is shaped and constrained by or the satisfaction of ministers. the organisational context; the organisational context can only be changed through the efforts of people. Previous There has been extensive debate in the literature about the policy capacity research does not tell us very much about difficulties inherent in evaluating policy outputs and out- the nexus between the two, the mechanisms through comes. First, there are problems attributing policy out- Page 10 of 15 (page number not for citation purposes) Australia and New Zealand Health Policy 2009, 6:3 http://www.anzhealthpolicy.com/content/6/1/3 Or ganisation al str uctur es and p r ocesses which supp or t health policy capacity includ e: evidence and infor mation – provision for generating evidence and collecting information for policy development knowled ge management – systematic provision for recognising and accessing the individuals and units across the organisation (and beyond) which have specialised knowledge which might inform policy development; incorporating history and experience in knowledge management frameworks wor kfor ce development – programs in place to produce and reproduce a broad mix of human resources to support policy making and to develop individual policy practitioners con sultation and communication channels an d p r ocesses – relationships with other levels of government, other sectors, non-government organisations, civil society, stakeholders and politicians; processes for timely and meaningful consultation; relationships between the minister’s office and departmental policy makers inter -dep ar tmen tal coor d in ation and n etwor king – processes and support systems for working collaboratively across departmental boundaries imp lementation lin kage – capacity for piloting and demonstration where appropriate; capacity for incremental implementation with evaluation and adjustment as needed; and involvement of implementation managers in policy analysis and development mon itor in g, evaluation and r eview – continuity between policy development and evaluation phases; access to evaluation expertise; processes for monitoring and incremental policy adjustment; peer review of the policy development process str ategic man agemen t, coor dination and accountab ility – policy leadership and accountability for the quality of such leadership Organi Figure 1 sational structures and processes that support health policy capacity Organisational structures and processes that support health policy capacity. Page 11 of 15 (page number not for citation purposes) Australia and New Zealand Health Policy 2009, 6:3 http://www.anzhealthpolicy.com/content/6/1/3 Or ganisation al cultur es which suppor t h ealth policy cap acity ar e: open to innovation and different ways of seeing things; flexible with respect to process and different ways of working; able to provide a plurality of frameworks to guide and facilitate policy work; able to build strategic alliances, collaboration and trust; able to capitalise on windows of opportunity, tolerate mistakes and manage risk; open to naming problems (before solutions are evident); tolerant of constructive conflict and the engagement of diverse views; able to work across agency boundaries; able to balance long and short term goals; supportive of effective teamwork; conscious of their own history; and committed to valuing corporate memory. F Figure 2 eatures of organisational culture that support health policy capacity Features of organisational culture that support health policy capacity. comes to policy capacity or even to particular policies. It is ured. Where policy outputs are evaluated, it is best to difficult to link policy advice with decisions and their out- employ qualitative methods using professional judge- comes [66,67]. Even the notion of isolating a discrete area ment and peer review [66,67]. Nicholson argued that sen- of policy in order to assess outcomes and trace the policy ior policy advisers who are highly regarded by their peers development process is questionable when policies are are likely to be the best judges of the quality of policy interwoven and cross-portfolio in nature [66]. Many other advice [67]. variables impact on policy decision-making and its out- comes [57,68], and often the advice of public servants is Retrospectively judging the outcomes of policy from the only one source of advice used by politicians [67]. Sec- position of an outsider carries the additional danger of ond, there is the issue of timeframes. Policy outcomes are ignoring the contingencies of all the scenarios that did not often not evident for many years. The policy development eventuate, whereas at the point of action, the policy oper- itself can often take a long time, and there is often a long ative would have been confronting a wider range of possi- lag time between policy development and implementa- ble scenarios, all of which needed to be weighed and tion [67]. Incremental changes to policy during imple- considered. Any assessment of policy capacity must take mentation may also mean that the long-term outcomes into account the range of possible scenarios and contin- are no longer traceable to a particular period of policy gencies confronting the policy-makers at the time the pol- development [66]. Third, judgements about the value or icy episode took place. This suggests that policy capacity "goodness" of policy vary widely as there is political and research needs to draw on the accounts of the policy prac- ideological disagreement between different interest titioners themselves of their experiences of the policy groups and actors over goals and outcomes [38,69]. development process and the environments in which the Finally, there are no simple or universally applicable policy episode took place. models available for evaluating policy work; it is generally agreed that there would be little value in trying to develop Despite these concerns, some attempts have been made to a set of generic criteria to use for evaluating policy capacity delineate criteria for evaluating policy capacity. Thissen [58,69]. and Twaalfhoven [69] described three detailed sets of cri- teria for evaluating policy analytic activities, based on dif- For all these reasons, there are no objective standards ferent views of policy analysis: as information provision; against which policy outcomes or outputs can be meas- as a participative policy-oriented process; and as a set of Page 12 of 15 (page number not for citation purposes) x Australia and New Zealand Health Policy 2009, 6:3 http://www.anzhealthpolicy.com/content/6/1/3 Competencies that sup por t h ealth policy cap acity in clud e: Knowledge and exper ience contextual knowledge – of institutional and administrative arrangements, and contemporary policy environments sector al knowled ge – of the health and/or human services field generally histor ical and comp ar ative knowled ge – including the genealogy and track record of various dynamics, models and methods in health and welfare systems theor y fr om acr oss the academic disciplin es – and the ability to use such theory in description, analysis, interpretation, explanation and strategy Skills of policy pr actice basic wor king skills – research and evaluation; writing (and re-drafting); communication skills (both exposition and listening) skills in functional (r ational) an alysis – ability to name the problem, articulate causes, identify and evaluate options, and use evidence to evaluate causes and options skills in p olitical analysis – ability to map the political field, conceive interventions, imagine, project and evaluate scenarios inter -sector al skills – capacity to communicate and work across different disciplines and sectors Per sonal attr ibu tes cr eativity, intuition and judgemen t Individual com Figure 3 petencies that support health policy capacity Individual competencies that support health policy capacity. methods and tools. The framework for organising these ticular circumstances of the case. Painter and Pierre [1] criteria distinguishes between inputs, content, process, suggested a set of evaluative criteria which include a set of results, use, and effects. The criteria proposed by Thissen values (coherence, "public-regardingness", credibility, and Twaalfhoven are designed for evaluating policy ana- decisiveness and resoluteness) and a set of support sys- lytic activities related to specific policy issues, rather than tems (collective decision processes, planning and evalua- the policy analytic capacity of organisations. Thissen and tion, information and analysis, and coordination Twaalfhoven [69] also noted that a single set of univer- procedures). They point out, however, that the evaluative sally applicable criteria is not feasible and the choice of criteria will "necessarily be contested" [1]. While these cri- criteria depends on the perspective adopted and the par- teria may provide useful checklists for future policy capac- Page 13 of 15 (page number not for citation purposes) x Australia and New Zealand Health Policy 2009, 6:3 http://www.anzhealthpolicy.com/content/6/1/3 ity research, they do not yield "objective" measures of Health policy is difficult; policy-makers are working with policy process because of the methodological problems complexity, conflict and uncertainty. However, much is at noted. stake. Further research into the conditions for effective policy-making in the health sector would appear to offer The conceptualisation of policy capacity underpinning significant returns. much of the policy capacity research has been criticised more broadly for its reliance on a rational model of pol- Competing interests icy-making that privileges some aspects of policy-making The authors declare that they have no competing interests. over others. The implicit assumption is that improving and systematising the technical aspects of policy-making Authors' contributions will result in better policy. Brans and Vancoppenolle [42] DHG carried out the data collection and comparative pointed out that this approach conflicts with other per- analysis and drafted the manuscript. DGL and DON par- spectives on improving policy-making, such as "interac- ticipated in the design of the study and the data analysis, tive governance", which emphasise citizen involvement and critically reviewed drafts of the manuscript. All and participation and the need for government to be authors read and approved the final manuscript. responsive and flexible. This suggests there is a dimension to policy capacity that is not amenable to technical solu- Acknowledgements The authors would like to thank Alison Hughes for her thoughtful com- tions and that requires more attention to the interface ments on a draft of this paper. We would also like to thank the two anon- between the bureaucracy and other players outside gov- ymous reviewers whose very helpful feedback assisted us in improving the ernment, including industry stakeholders and the wider quality of the paper. society [[27], p. 12, 70]. These relational aspects of policy capacity should be a stronger focus in future policy capac- References ity research. 1. Painter M, Pierre J: Unpacking policy capacity: Issues and themes. In Challenges to State Policy Capacity Edited by: Painter M, Pierre J. Hampshire and New York: Palgrave MacMillan; 2005:1-8. Conclusion 2. Australian Institute of Health and Welfare: Aboriginal and Torres Strait In the absence of research into health policy capacity, Islander Health Performance Framework, 2006 Report: Detailed Analyses. some lessons can be drawn from experience in evaluating AIHW cat. no. IHW 20. Canberra: AIHW 2007. 3. Hayes LJ, Quine S, Taylor R, Berry G: Socio-economic mortality public sector policy capacity in Australia and other coun- differentials in Sydney over a quarter of a century, 1970–94. tries. Aust N Z J Public Health 2002, 26:311-317. 4. Andrews K: National Strategy for an Ageing Australia. An Older Australia: Challenges and Opportunities for All Canberra: Commonwealth of Aus- The review of public policy capacity research undertaken tralia; 2002. for this paper suggests a number of elements or domains 5. Australian Institute of Health and Welfare: Health Expenditure Australia 2002–03 Canberra: Australian Institute of Health and Welfare; 2004. of policy capacity that could be used to provide a concep- 6. Butler CD, Douglas B, McMichael AJ: Globalisation and environ- tual framework for future research. These include: infor- mental change: implications for health and health inequali- ties. In The Social Origins of Health and Well-Being Edited by: Eckersley mation and evidence; personnel management and R, Dixon J, Douglas B. Melbourne: Oxford University Press; 2001. workforce development; consultation and communica- 7. Australian Institute of Health and Welfare: Chronic Diseases and Asso- tion; implementation; monitoring, evaluation and ciated Risk Factors in Australia, 2001 Canberra: Australian Institute of Health and Welfare; 2002. review; strategic management and leadership; and sup- 8. Australian Government Department of Health and Ageing: Protecting portive organisational culture. Individual competencies Australia from Communicable Diseases: Everybody's Business. A special for policy practitioners identified in the studies included: report from the Commonwealth Chief Medical Officer. Canberra 2004. 9. Eckersley R, Dixon J, Douglas B, Eds: The Social Origins of Health knowledge (of context, different disciplines, and systems and Well-being. Melbourne: Cambridge University Press; 2001. and developments in other countries); practical skills of 10. Marmot MG, Wilkinson RG, Eds: Social Determinants of Health. Oxford, New York: Oxford University Press; 1999. policy-making, such as analytic, technical and communi- 11. Armstrong BK, Gillespie J, Leeder SR, Rubin GL, Russell LM: Chal- cation skills; and personal attributes such as creativity, lenges in health and health care for Australia. MJA 2007, intuition and judgment. 187:485-489. 12. Duckett SJ: Commonwealth/state relations in health. In Health Policy in the Market State Edited by: Hancock L. St Leonards, NSW: The review of models and methods suggests that future Allen & Unwin; 1999:71-85. 13. Duckett SJ: The Australian Health Care System 2nd edition. South Mel- policy capacity research should be grounded in careful bourne: Oxford University Press; 2004. description and analysis of the policy process that is not 14. Commonwealth of Australia: The Blame Game: Report of the Inquiry into limited to a single narrow perspective. Such research Health Funding House of Representatives Standing Committee on Health and Ageing; 2006. should also use a range of qualitative methods that 15. Hancock L: Australian federalism, politics and health. In Health employ the judgement of policy practitioners themselves Policy in Australia Edited by: Gardner H, Barraclough S. South Mel- in evaluating policy capacity, rather than attempting to bourne: Oxford University Press; 2002:49-78. 16. Swerissen H, Duckett SJ: Health policy and financing. In Health measure policy capacity according to "objective" meas- Policy in Australia Edited by: Gardner H, Barraclough S. Melbourne: ures of policy outcomes or explicit evaluative criteria. Oxford University Press; 2002:13-48. Page 14 of 15 (page number not for citation purposes) Australia and New Zealand Health Policy 2009, 6:3 http://www.anzhealthpolicy.com/content/6/1/3 17. Hancock L: Policy, power and interests. In Health Policy in the Mar- 43. Bakvis H: Rebuilding policy capacity in the era of the fiscal div- ket State Edited by: Hancock L. St Leonards, NSW: Allen & Unwin; idend: a report from Canada. Governance 2000, 13:71-103. 1999:19-47. 44. Canadian Government: Strengthening Our Policy Capacity. Report of the 18. Lewis JM: Health Policy and Politics: Networks, Ideas and Power East Haw- Task Force on Strengthening the Policy Capacity of the Federal Government thorn: IP Communications; 2005. 1996. 19. Ham C: The background. In Health Care Reform: Learning from Inter- 45. Bridgman P, Davis G: The Australian Policy Handbook Crows Nest, national Experience Edited by: Ham C. Bristol, PA: Open University NSW: Allen & Unwin; 1998. Press; 1997:1-20. 46. Policy Research Initiative [http://www.policyresearch.gc.ca/] 20. Ross B, Snasdell-Taylor J, Cass Y, Azmi S: Health Financing in Australia: 47. Nutley S, Webb J: Evidence and the policy process. In What The Objectives and the Players Canberra: Commonwealth Department Works? Evidence-based Policy and Practice in Public Services Edited by: of Health and Aged Care; 1999. Davies HTO, Nutley SM, Smith PC. Bristol: The Policy Press; 21. Hall J: Incremental change in the Australian Health Care Sys- 2000:13-41. tem. Health Aff 1999, 18:95-109. 48. Bogdanor V, Ed: Joined up Government. London: Oxford Univer- 22. Australian Public Service Commission: The Australian Experience of sity Press for the British Academy; 2005. Public Sector Reform. Occasional Paper Two Commonwealth of Aus- 49. Mulgan G: Joined up government in the United Kingdom: tralia; 2003. Past, present and future. Canberra Bulletin Pub Adm 2002, 23. Australian Public Service Commission: Connecting Government: Whole 105:25-29. of Government Responses to Australia's Priority Challenges Management 50. Bullock H, Mountford J, Stanley R: Better Policy-Making London: Cen- Advisory Committee, APSC, Commonwealth of Australia; 2004. tre for Management and Policy Studies, Cabinet Office, United King- 24. State Services Commission: Essential Ingredients: Improving the Quality dom; 2001. of Policy Advice. Occasional Paper No. 9. Wellington 1999. 51. Wright JC: Strategic Social Policy Advice: Improving the Information Base. 25. OECD: Synthesis of Reform Experiences in Nine OECD Countries: Govern- Working Paper No. 8 New Zealand State Services Commission; 1999. ment Roles and Functions, and Public Management: Government of the 52. Wolf A: Building Advice: The Craft of the Policy Professional. Working Future: Getting from Here to There. Paris: OECD 1999. Paper No. 7 Wellington: New Zealand State Services Commission; 26. Manitoba Office of the Provincial Auditor: A Review of the Policy Devel- 2000. opment Capacity within Government Departments 2001. 53. State Services Commission: Looping the Loop: Evaluating Outcomes and 27. UK Government: Modernising Government 1999. Other Risky Feats. Occasional Paper No. 7. Wellington 1999. 28. Office of the Auditor General of Canada: Public Service Management 54. State Services Commission: High Fliers: Developing High Performing Pol- Reform: Progress, Setbacks and Challenges. Ottawa 2001. icy Units. Occasional Paper No. 22. Wellington 1999. 29. Lin V, Gibson B, Eds: Evidence-based Health Policy: Problems 55. State Services Commission: Pieces of the Puzzle: Machinery of Govern- & Possibilities. South Melbourne: Oxford University Press; 2003. ment and the Quality of Policy Advice. Working Paper No. 4. Wellington 30. Davies HTO, Nutley SM, Smith PC, Eds: What Works? Evidence- 2000. based Policy and Practice in Public Services. Bristol: The Policy 56. State Services Commission: Gaining Through Training: Developing High Press; 2000. Performing Policy Advisors. Working Paper No. 2. Wellington 2000. 31. Peters BG: Managing Horizontal Government: The Politics of Coordination. 57. Di Francesco M: An evaluation crucible: Evaluating policy Research Paper No. 21 Canadian Centre for Management Develop- advice in Australian central agencies. Australian Journal of Public ment; 1998. Administration 2000, 59:36-48. 32. Lindquist E: Strategy, capacity and horizontal governance: 58. Uhr J, Mackay K, Eds: Evaluating Policy Advice: Learning from Perspectives from Australia and Canada. Optimum Online 2004, Commonwealth Experience. Canberra: Federalism Research 34:. Centre, Australian National University and Commonwealth Depart- 33. Hess M, Adams D: Knowing and skilling in contemporary pub- ment of Finance; 1996. lic administration. Aust J Public Adm 2002, 61:68-79. 59. Di Francesco M: Process not outcomes in new public manage- 34. Painter M, Pierre J: Conclusions: Challenges to policy capacity. ment? 'Policy coherence' in Australian Government. The In Challenges to State Policy Capacity: Global Trends and Comparative Per- Drawing Board: Aust Rev Public Aff 2001, 1:103-116. spectives Edited by: Painter M, Pierre J. Hampshire and New York: Pal- 60. Australian National Audit Office: Developing Policy Advice. Audit Report grave Macmillan; 2005:255-261. No. 21, 2001–2002 2002. 35. Davis G: Conclusion: policy capacity and the future of govern- 61. Victorian Auditor-General's Office: Report on Public Sector Agencies: ance. In The Future of Governance Edited by: Davis G, Keating M. St Results of Special Reviews and Other Studies, August 2004 2004. Leonards NSW: Allen & Unwin; 2000. 62. Pressman JL, Wildavsky A: Implementation: How Great Expectations in 36. Howlett M: Enhanced policy analytical capacity as a prerequi- Washington are Dashed in Oakland; or, Why it's Amazing that Federal Pro- site for effective evidence-based policy-making: Theory, con- grams Work at all Berkeley: University of California Press; 1973. cepts and lessons from the Canadian case. In Workshop on 63. Ham C, Hill M: The Policy Process in the Modern Capitalist State 2nd edi- Evidence-Based Policy; International Research Symposium on Public Man- tion. New York, London: Harvester Wheatsheaf; 1993. agement XII Queensland University of Technology, Brisbane; 2008. 64. Younis T, Ed: Implementation in Public Policy. Aldershot: Dart- 37. UK Cabinet Office: Professional Policy Making for the Twenty-First Cen- mouth Pub. Co; 1990. tury Strategic Policy Making Team, Cabinet Office; 1999. 65. Hogwood BW, Gunn LA: Policy Analysis for the Real World Oxford: 38. Peters BG: The Policy Capacity of Government. Research Paper No. 18 Oxford University Press; 1984. Canada: Canadian Centre for Management Development; 1996. 66. Waller M: Evaluating policy advice. Aust Journal Public Adm 1992, 39. Aucoin P, Bakvis H: Public service reform and policy capacity: 51:440-449. Recruiting and retaining the best and the brightest? In Chal- 67. Nicholson J: Measures for monitoring policy advice. In Evaluat- lenges to State Policy Capacity: Global Trends and Comparative Perspectives ing Policy Advice: Learning from Commonwealth Experience Edited by: Uhr Edited by: Painter M, Pierre J. Hampshire and New York: Palgrave J, Mackay K. Canberra: Federalism Research Centre, The Australian Macmillan; 2005:164-184. National University and Commonwealth Department of Finance; 40. Janicke M: The political system's capacity for environmental 1996:26-40. policy. In National Environmental Policies: A Comparative Study of 68. Keating M: Defining the policy advising function. In Evaluating Capacity-Building Edited by: Janicke M, Weidner H. Berlin: Springer- Policy Advice: Learning from Commonwealth Experience Edited by: Uhr J, Verlag; 1997:1-24. Mackay K. Canberra: Federalism Research Centre, The Australian 41. Parsons W: Not just steering but weaving: Relevant knowl- National University and Commonwealth Department of Finance; edge and the craft of building policy capacity and coherence. 1996:61-68. Aust J Public Adm 2004, 63:43-57. 69. Thissen WAH, Twaalfhoven PGH: Towards a conceptual struc- 42. Brans M, Vancoppenolle D: Policy-making reforms and civil ture for evaluating policy analytic activities. Euro Journal Oper- service systems: An exploration of agendas and conse- ational Res 2001, 219:627-649. quences. In Challenges to State Policy Capacity Edited by: Painter M, 70. Jayasuriya K: Capacity beyond the boundary: New regulatory Pierre J. Hampshire and New York: Palgrave Macmillan; state, fragmentation and relational capacity. In Challenges to 2005:164-184. State Policy Capacity Edited by: Painter M, Pierre J. Hampshire and New York: Palgrave Macmillan; 2005. Page 15 of 15 (page number not for citation purposes) http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Australia and New Zealand Health Policy Springer Journals

Evaluating health policy capacity: Learning from international and Australian experience

Loading next page...
 
/lp/springer-journals/evaluating-health-policy-capacity-learning-from-international-and-YZ5DL60TH0
Publisher
Springer Journals
Copyright
Copyright © 2009 by Gleeson et al; licensee BioMed Central Ltd.
Subject
Medicine & Public Health; Public Health; Social Policy
eISSN
1743-8462
DOI
10.1186/1743-8462-6-3
pmid
19245704
Publisher site
See Article on Publisher Site

Abstract

Background: The health sector in Australia faces major challenges that include an ageing population, spiralling health care costs, continuing poor Aboriginal health, and emerging threats to public health. At the same time, the environment for policy-making is becoming increasingly complex. In this context, strong policy capacity – broadly understood as the capacity of government to make "intelligent choices" between policy options – is essential if governments and societies are to address the continuing and emerging problems effectively. Results: This paper explores the question: "What are the factors that contribute to policy capacity in the health sector?" In the absence of health sector-specific research on this topic, a review of Australian and international public sector policy capacity research was undertaken. Studies from the United Kingdom, Canada, New Zealand and Australia were analysed to identify common themes in the research findings. This paper discusses these policy capacity studies in relation to context, models and methods for policy capacity research, elements of policy capacity and recommendations for building capacity. Conclusion: Based on this analysis, the paper discusses the organisational and individual factors that are likely to contribute to health policy capacity, highlights the need for further research in the health sector and points to some of the conceptual and methodological issues that need to be taken into consideration in such research. Background emerging threats to public health, such as new communi- The context for health policy-making cable diseases [8]. Social and health problems are also Health policy-makers in Australia face a broad range of becoming more complex and are increasingly recognised entrenched (and in some cases, worsening) public health as being inter-dependent [9,10]. problems, as well as emerging issues that create challenges for the health system into the future. These include, inter For these challenges to be met, health bureaucracies at alia: continuing poor Aboriginal health [1,2]; widening both Commonwealth and state levels need the capacity to inequalities in health [3]; an ageing population [4]; plan effectively and put policies in place to ensure the increasing health care costs [5]; environmental destabili- health system (including health care and public health) is sation [6]; rising incidence of chronic disease [7]; and organised, funded, delivered and coordinated in the most Page 1 of 15 (page number not for citation purposes) Australia and New Zealand Health Policy 2009, 6:3 http://www.anzhealthpolicy.com/content/6/1/3 effective and sustainable way. Doing so requires overcom- These changes in the policy environment have resulted in ing significant barriers arising from the complexity of greater uncertainty and complexity in policy-making. It is health system financing and program delivery. For exam- increasingly difficult to predict the impact of policy ple, rising chronic disease rates mean that the health sys- changes into the long term. With increasing scale and tem must be re-engineered to focus more on prevention. complexity, balancing the rational/technical and the However, the capacity to invest in prevention initiatives is political aspects of policy development appears increas- hampered by a health system largely focused on episodes ingly difficult. Emerging ideas in the policy literature such of acute care, fee-for-service funding for general practice, as evidence-based policy-making [29,30] and calls for poor coordination between acute and community care greater policy coherence or 'horizontal government' sectors, and workforce shortages in most health profes- [31,32] also bring with them implications for the skills sions [11]. Other obstacles to health reform include the required of policy-makers and for the institutional sup- division of responsibility for health between the Com- ports required to change ways of working. Policy-makers monwealth and the states (which gives rise to coordina- increasingly need skills in coordinated and cooperative tion problems, overlap and duplication of services, gaps policy work, networking, negotiating, collaborating, and in service delivery and cost-shifting) [12-14]; the complex flexible policy implementation [22,33]. mix of public and private insurers and providers [15,16] and the presence of entrenched powerful interests that Policy capacity limit the ability of the state to bring about change [17,18]. From the mid-1990s onwards, there was increasing con- Many policy problems (such as social exclusion and cern about declining policy capacity in many jurisdic- health inequalities) also cross jurisdictional and portfolio tions, and governments in many countries turned their boundaries and require policy-makers and public sector attention to building (or re-building) policy capacity dur- organisations to work in new ways in order to be effective. ing the period from 1995 to 2005. Concerns about policy capacity have been attributed to a number of shifts in pub- Health policy-makers also face challenges in terms of bal- lic sector environments, including public management ancing different health system goals such as equity, effi- reforms such as privatisation, down-sizing and contract- ciency and quality. In an environment of limited ing out as well as shifts in the external environment such resources, there need to be some trade-offs between these as globalisation and changes to the state-society interface goals [19,20]. Ensuring equity through a universal health [34,35]. Concerns continue to be expressed about the pol- care system is an important value on which the Australian icy capacity of public sector agencies, even in wealthy health system is based [21]; however, balancing universal- countries [36]. There is a growing appreciation interna- ity (i.e. equity with respect to both access and levels of tionally that capacity building in policy-making can con- service) with cost containment is a significant issue for tribute significantly to improved public policy outcomes policy-makers. [23,24,26-28,37] although at this stage there is little empirical evidence to show that this is so. Increasingly complex policy environments The policy-making environment is said to have become The term 'policy capacity' is generally used to refer to the more complex over the last few decades due to major capacity of public sector agencies to develop and imple- shifts in the relationship between government and soci- ment "good" policy (although players outside govern- ety, brought about by large-scale forces such as globalisa- ment, including a range of non-government tion (understood as being the increasingly international organisations, universities, research agencies and service nature of the political and economic forces which shape organisations also contribute to the policy capacity of policy) and the increasing use of technology [22-25]. nations and states, the term is most often used to describe Increased reliance on market mechanisms such as man- the policy-making capacity of governments and the public aged competition as policy tools has also resulted in service). However, policy choices are always normative changes to the role of the public service in policy-making. with differential implications for different stakeholders. Greater involvement of third parties (including citizens What constitutes 'good' policy in any particular context and stakeholders) in service delivery results in more com- may be highly contested [38]. plex policy environments with more players outside gov- ernment, and creates challenges in terms of involving The scholarly literature offers a number of different defi- those who are responsible for service provision in policy nitions of policy capacity that highlight different dimen- development [24,26-28]. These developments shape and sions. Among the more useful of these is the definition reflect an increasingly complex health policy environ- offered by Painter and Pierre [1] of policy capacity as "the ment. ability to marshal the necessary resources to make intelli- gent collective choices about and set strategic directions for the allocation of scarce resources to public ends". At Page 2 of 15 (page number not for citation purposes) Australia and New Zealand Health Policy 2009, 6:3 http://www.anzhealthpolicy.com/content/6/1/3 the core of most conceptualisations of policy capacity is United Kingdom, New Zealand, Canada and Australia. A the capacity to make decisions or "intelligent choices" comprehensive literature search was undertaken includ- [1,35,38,39]. A focus on intelligent choice highlights the ing the published health policy, public policy and public quality of public sector policy workers as a central capacity administration literature (searched using library cata- issue, and suggests that recruiting, retaining and develop- logues, journal and thesis databases) as well as govern- ing the "best and brightest" policy workers should be a ment reports and grey literature (sourced from databases, primary focus of capacity building [39]. search engines and government and direct searching of government and organisational websites). The search was Other dimensions of policy capacity suggested by com- restricted to Canada, New Zealand, the United Kingdom mentators in the political science and public administra- and Australia due to their shared political traditions, tion literature include the capacity to utilise resources including the Westminster tradition, and broadly similar [1,40]; the capacity to implement policy decisions [35,38] approaches to health care delivery. The conclusions of this and coordination of the policy-making process across review ought likewise to be regarded as applying to these government [41]. Peters provides the additional insight kinds of jurisdictions. Evaluations of policy capacity or that policy capacity includes the ability to make not only the quality of policy advice at both federal/national and incremental choices, but also more strategic choices state/provincial levels were included in the data set. No which involve larger deviations from the status quo [38]. health sector-specific studies of policy capacity were found, although some studies included health depart- If the capacity of the health sector to develop and deliver ments among the units surveyed. effective health policy (however defined) is to be strength- ened, a better understanding of the elements of policy Table 1 shows the documents included in the comparative capacity is needed. However, empirical research on the review. The studies were analysed according to the follow- topic of health policy capacity is lacking. This paper exam- ing variables: context and themes; models and methods; ines international and Australian experience in conceptu- findings and recommendations for building policy capac- alising and evaluating public sector policy capacity more ity. In each area, common themes and key differences generally and draws out some lessons for building health were drawn out. The analysis was informed by the peer- policy capacity and for further research in this area. reviewed literature on policy capacity and public policy as well as the health policy literature. Based on this analysis, some hypotheses about elements of policy capacity in the Methods This paper reviews publicly available public sector policy health sector were developed. capacity research undertaken between 1995–2005 in the Table 1: Documents included in the review Jurisdiction Document Canada Canadian Government (1996) Strengthening Our Policy Capacity. Report of the Task Force on Strengthening the Policy Capacity of the Federal Government. Manitoba Office of the Provincial Auditor (2001) A Review of the Policy Development Capacity Within Government Departments. United Kingdom UK Cabinet Office (1999) Professional Policy Making for the Twenty-First Century. Strategic Policy Making Team. Bullock, H, Mountford J et al (1999) Better Policy Making. London, Centre for Management and Policy Studies, Cabinet Office New Zealand State Services Commission (1999) Essential Ingredients: Improving the Quality of Policy Advice. State Services Commission (1999) Looping the Loop: Evaluating Outcomes and Other Risky Feats. State Services Commission (1999) High Fliers: Developing High Performing Policy Units. State Services Commission (2000) Pieces of the Puzzle: Machinery of Government and the Quality of Policy Advice. State Services Commission (2000) Gaining Through Training: Developing High Performing Policy Advisors. Wright (1999) Strategic Policy Advice: Improving the Information Base. New Zealand State Services Commission. Wolf (2000) Building Advice: The Craft of the Policy Professional. New Zealand State Services Commission. Australia Australian National Audit Office (2002) Developing Policy Advice. Audit Report No. 21, 2001–2002. Victorian Auditor-General's Office (2004) Report on Public Sector Agencies: Results of Special Reviews and Other Studies, August 2004. Australian Public Service Commission (2004) Connecting Government: Whole of Government Responses to Australia's Priority Challenges. Management Advisory Committee, APSC, Commonwealth of Australia. Page 3 of 15 (page number not for citation purposes) Australia and New Zealand Health Policy 2009, 6:3 http://www.anzhealthpolicy.com/content/6/1/3 to meet the needs of decision-makers; so the emphasis in Results Introduction to the studies: Context, themes, models and this report is largely on improving the outputs of the pol- methods icy process [26]. The methodology involved developing Significant differences existed between the policy capacity criteria for effective policy development capacity in three studies undertaken in the four jurisdictions, in terms of domains (policy process, organisational context and the the context in which they were initiated, the drivers for the "policy product") based on a literature review and consul- review of policy capacity, the actors undertaking the stud- tations with people involved in policy development. ies, the focus and major themes driving the reviews, the These criteria were used as the basis for interviews with models of policy process underpinning the studies, and senior managers in departments and political policy-mak- the methods used for evaluating policy capacity. These ers including Cabinet Ministers and the Policy Secretariat. differences are discussed in the section below. Interviewees were asked to rank the criteria according to their relative importance. Canada The Canadian Government began to focus on rebuilding The United Kingdom A major program of public sector reform began in the policy capacity in the mid-1990s, after two decades of "new public management" and public sector downsizing United Kingdom in 1999 with the Modernising Government were perceived to have resulted in a decline of policy White Paper [27] in which policy-making was identified capacity [42,43]. A task force instigated by the Clerk of the as one of five key areas for reform. Key themes in the Privy Council Office in 1995–96 investigated the policy White Paper included "forward looking" and consultative capacity of the federal government and explored ways in policy-making and "joined-up" government [27]. which it could be strengthened [44]. Task Force data-gath- Improving the use of evidence in policy-making was also ering processes included interviews with officials in most a strong focus of Modernising Government and the subse- departments and roundtables with junior policy officers quent policy capacity studies undertaken in the UK [[47] and external policy research experts [44]. Major themes of p.13]. Academic literature from the UK has also focused the ensuing report included strengthening policy capacity extensively on both joined-up government [48,49] and across government for dealing with horizontal issues (as improving the utilisation of research in policy-making well as within departments), workforce development, and [30,47]. building more effective links with the policy research community [44]. The study draws extensively on an anal- The UK Cabinet Office followed Modernising Government ysis of seven "policy functions": theoretical research; sta- with a study of "professional policy-making" [37], which tistics, applied research and modelling; environmental involved an audit of policy-making using a model of scanning, trends analysis and forecasting; policy analysis "modernised policy-making". This study involved collect- and advice; consultation and managing relations; com- ing case studies of "good practice" policy-making, as well munications; and program design, implementation, mon- as interviews with officials and advisers, focus groups with itoring and evaluation [44]. This approach aligns closely policy staff and "recent leavers", and a training needs anal- with rational, "stagist" models of the policy process, such ysis involving ministers and policy staff [37]. A further as the Bridgman and Davis [45] policy cycle. study by the Cabinet Office entitled Better Policy-Making [50] examined approaches to modernising the policy The themes identified in the pivotal Canadian Govern- process, and looked more closely at enabling factors and ment Task Force report have been echoed in the scholarly barriers to change. A survey of civil servants in all ministe- literature from Canada, such as strengthening coordina- rial departments was undertaken for this study. tion and coherence across government [31,32], and policy analytic capacity (in particular the challenges of recruit- The UK policy capacity initiative was unusual in several ment and retention) [36,39,43]. This report also led to a respects. First, it was initiated by politicians rather than by focus on strengthening policy research capacity, and the public servants – this gave it a breadth and scope beyond Policy Research Initiative was subsequently established to the studies done in other countries. It also ensured that improve the generation and use of research on cross-cut- the judgements of "good" policy reflected the political pri- ting policy issues [46]. orities and orientation of the government of the day. Sec- ond, it was based on a complex conceptual model of Also in Canada, the Office of the Provincial Auditor in policy-making that included a set of characteristics of Manitoba Province undertook a smaller scale evaluation "modernised" policy-making with three overarching of the policy process, organisational structures and proc- themes and nine core competencies. The UK studies have esses and policy outputs in 2001. This initiative seems to a strong focus on competencies, many of which (such as have been driven by concerns about the inability of brief- "outward looking" and "joined up") seem to be used not ings and policy documents produced by the bureaucracy only to describe the capabilities of policy staff but to Page 4 of 15 (page number not for citation purposes) Australia and New Zealand Health Policy 2009, 6:3 http://www.anzhealthpolicy.com/content/6/1/3 describe organisational capabilities, and, even more advising [57]. The use of PMRs seems to have petered out broadly, to characterise the approach to "modernised pol- after the trials, largely due to a recognition of the limits of icy-making" [[37] p.12]. The conclusions of this survey, applying program evaluation principles to the inherently however, appear to be largely embedded in the methods political policy process [57,59]. and the assumptions that went into the methods, particu- larly with respect to the desired policy competencies. At a later stage, performance audits of policy advice sys- tems and processes were undertaken by the Australian New Zealand National Audit Office [60] and the State of Victoria's Audi- In New Zealand, the State Services Commission under- tor-General's Office [61]. These audits focused largely on took a suite of studies in 1998–99 as part of a project improving the process through which policy is made and called "Improving the Quality of Policy Advice". As the the quality of policy advice outputs such as briefings and title indicates, the project was driven by concerns about submissions. The scope of these studies is relatively lim- the quality of policy advice developed through the ited in comparison with some of the other studies consid- bureaucracy [42]. The focus of this project was largely on ered in this paper. The ANAO report was based on the quality of inputs (particularly information and discussions with departmental employees and their staff, research, but also evaluation, consultation and coordina- case studies of quality management arrangements and tion) to the policy process [24,51-56]. Key themes document analysis [[60] p.13]. Quality management sys- included: lack of effective use of information, evidence tems for the provision of policy advice were assessed and evaluation; emphasis on short-term outputs and "fast against criteria based on a literature review and other solutions" in policy development at the expense of a sources and a set of "better practice principles" for longer term, more strategic focus; shortages of appropri- enhancing management and quality assurance of the pro- ately skilled staff and lack of policy knowledge and expe- vision of policy advice were developed. The Victorian rience [24,51-56]. This project was strongly based in the audit assessed policy briefs and the policy development public policy literature with a number of comprehensive process against a set of "good practice" principles devel- literature reviews [51,52]. Data collection involved inter- oped using a literature review and expert interviews [61]. views with central agency officials and chief executives Like the New Zealand studies, these performance audits and policy managers [24]. One of the working papers also were focused on the development of policy advice and examined six high performing policy units, using case their recommendations were directed towards the partic- studies and interviews [54]. ular departments audited. Although the New Zealand studies were quite broad in The Australian Public Service Commission undertook a terms of the issues covered, the emphasis on improving major initiative to improve inter-sectoral approaches to policy advice meant that the recommendations of the policy-making, culminating in the Connecting Government studies were generally directed towards senior managers report in 2004 [23]. The report was based on a compre- in departments, and were therefore narrower in scope hensive literature review and a series of case studies of than the some of the capacity building strategies devel- "whole of government" policy-making. Although the oped in Canada and the UK. focus of this report is on whole-of-government responses to major cross-portfolio issues, it identifies a number of Australia areas where capacity could be improved and some of the Australian experience with evaluating policy capacity findings relating to institutional culture and the skills set began in the mid-1990s with a series of trials in evaluating for policy-makers are useful for thinking about depart- the policy advice function of Commonwealth Govern- mental capacity. Unfortunately "whole of government" ment central agencies using Policy Management Reviews was limited to "'whole of Commonwealth Government", (PMRs) based on performance assessment techniques so the challenges of intergovernmental policy coordina- [57,58]. PMRs were an attempt to subject the policy devel- tion were not addressed. opment and advice functions of public sector agencies to Key findings and recommendations for building policy the same scrutiny as that given to administration at the time, and were driven by concerns with outcomes and capacity accountability [57,58]. These trials raised methodological The findings and recommendations of these various pol- debates about the criteria for evaluating policy advice (dis- icy capacity studies point towards two broad areas as the cussed later in this paper). Di Francesco argues that an focus of capacity building: organisational capacity and original intention to evaluate policy advice according to individual competencies. To analyse the elements of pol- its outputs (such as briefs), the policy itself, and the out- icy capacity in each of these areas, the authors created a comes (the impact of policy in the real world) was aban- table with the main analytical themes, and categorised the doned in favour of a focus on the processes of policy Page 5 of 15 (page number not for citation purposes) Australia and New Zealand Health Policy 2009, 6:3 http://www.anzhealthpolicy.com/content/6/1/3 issues and recommendations in the documents accord- tion and evaluation) and it is the overall mix of skills ingly, using an iterative process. which is important for policy capacity. The individual competencies include: Knowledge Studies identified a number of different types of knowl- � Knowledge and experience edge important for policy-making. These included knowl- edge of context, both the context of the problem and of � The practical skills of policy making the policy, including the organisational, political and wider social context [[26], p.37,52]. Various disciplines � Personal attributes such as creativity, intuition and were identified as contributing to policy-making, includ- judgement. ing law, economics, accountancy, statistics, the social sci- ences, project management and information technology The elements of organisational capacity which emerge [37]. Knowledge of systems and developments in other from these studies include: countries and the ability to "learn lessons" from these was also a key competency in developing more "outward look- � Access to and use of information and evidence ing" policy-making identified by the UK Cabinet Office [37]. � Personnel management and workforce development Practical skills of policy-making � Consultation and communication Policy-making skills described in the reports included analytic skills such as the ability to frame problems, � Inter-departmental coordination and networking appraise research evidence, predict the likely conse- quences of policy choices and evaluate associated risks � Implementation [26,37,52]. Skills in the daily work of policy development (such as drafting, researching, consulting, evaluation and � Monitoring, evaluation and review project management) were also mentioned [26,37,52]. High level interpersonal and communication skills were � Strategic management and leadership highlighted in several studies [23,37,52]. The ability to utilise information technology, to manage risks and to � Institutional culture. learn new skills were also highlighted by the UK Cabinet Office [37]. The next section of the paper explores findings of the pol- icy capacity evaluations in each jurisdiction in each of Creativity, intuition and judgement these domains. These attributes were emphasised in one of the UK reports, which described the need for policy-makers to be Individual competencies "flexible and innovative", willing to question the status The knowledge, skills and capabilities of policy staff were quo and prepared to try out new ideas and work in new frequently referred to in all the studies. In the Manitoba ways [37]. The Canadian Task Force also discussed the report, the knowledge and skills of policy staff were iden- need for intellectual curiosity, intuition and the ability to tified as being the most important factor contributing to be "comfortable with the uncertainties of policy-making" excellence in policy development and were also the most [[44], p. 24]. The Manitoba study also highlighted the frequently cited area needing improvement [[26], pp. 34– importance of creativity and good judgement [26]. The 36]. Despite this emphasis, however, there was little in- emphasis on these attributes in the studies highlights the depth exploration of the knowledge and skills that policy significance of personal and second-hand experience with practitioners need to do their work and how the presence cases (stories of episodes of policy-making that illustrate or absence of these capabilities impacted on the outputs principles and insights that can then be used to inform or outcomes of policies. practice). Cases become an important source of knowl- edge upon which judgement and intuition are built. In this paper the capabilities of policy-makers described in the studies are considered under the headings of: knowl- Organisational capacity Information and evidence edge; practical skills of policy-making; and creativity, intu- ition and judgement. As the Canadian Government Task The importance of adequate and timely information and Force [44] notes, the skill sets and relevant knowledge evidence to inform policy-making was a strong theme requirements differ for different types of policy personnel across all the studies, and was particularly prominent in (with different roles in policy development, implementa- New Zealand and the United Kingdom. This reflects a Page 6 of 15 (page number not for citation purposes) Australia and New Zealand Health Policy 2009, 6:3 http://www.anzhealthpolicy.com/content/6/1/3 growing emphasis in the published literature on evidence- management infrastructure to facilitate sharing of knowl- based (or evidence-informed) policy-making [see, for edge across the bureaucracy. Strategies proposed included example, [29,30]]. The studies reviewed were concerned policy "knowledge pools" and "knowledge networks". not just with evidence derived from empirical research, The UK report Professional Policy Making for the Twenty First however, but also with other types of evidence, including Century [37] recommended the establishment of a policy information about the circumstances in which policy is knowledge pool including information about policies being made (a distinction that is often not clearly made in (such as policy objectives, impact assessments and evalu- the evidence-based policy literature). ation results, consultation, and evidence drawn upon) in a standard format. Concerns about the use of evidence in policy-making were common. The UK Cabinet Office studies found that evi- Knowledge management was also highlighted as an dence was not always utilised effectively, even when important factor in facilitating joined-up responses in the research had been commissioned by the departments Australian whole-of-government study [23] and identi- themselves [37]. The Canadian Government study also fied as a "better practice principle" in the ANAO audit. The found that relevant statistics for policy-making were not ANAO [[60], p. 81] also suggested including directories of always available and capacity for applied research and subject expertise and information about the policy agenda quantitative modelling varied between departments [[44], as well as information about specific projects in the p. 5]. Models available for forecasting, analysing contin- knowledge pools, and recommended cross-agency net- gencies and predicting scenarios were not utilised as fully works of policy advisers "to identify, research and coordi- as they could have been [[44], p. 5–6]. The Manitoba nate policy based on themes". Office of the Provincial Auditor [26] also found that there was limited use of both quantitative and qualitative anal- Personnel management and workforce development ysis for both issue analysis and for determining policy A common theme in the studies was the importance of an options. adequate supply of highly skilled policy personnel and an appropriate mix of skills within units or departments. The New Zealand State Services Commission identified a Staff shortages presented barriers to longer term and more number of factors affecting the use of information and strategic policy-making in many studies. For example, the evidence, including: adequate time to consult with Canadian Government study [44] noted shortages of key researchers and to commission research where evidence skills, particularly for "policy generalists", and highlighted does not exist; skills in using information; ability to coor- the importance of personnel management. Areas in which dinate information resources between departments; the personnel management needed improvement included production of longer-term and strategic research; and pro- rotation of staff, recruitment processes, performance ductive relationships with external research organisations review, and mobility and variation in experience. The rec- [24]. Ideas for improving the use of evidence include com- ommendations for improvements in these areas were missioning more long-term and strategic research, mostly cast in terms of self-evident principles; few sugges- increasing central-agency expectations and improving tions were made about how to give effect to these princi- coordination across the public service [24]. ples. Many of these ideas in the Canadian and New Zealand Training needs assessment and the provision of training studies are echoed in the UK Cabinet Office study Profes- for new policy staff, policy staff new to the field and non- sional Policy Making for the Twenty First Century, which policy officers engaged in policy work were recommended identified the need to improve the ability of policy-mak- by the Canadian Government [[44], p. 24–30]. Access to ers to use evidence and also to ensure that relevant evi- training in a range of areas (including analytical thinking, dence was available [37]. Strategies to achieve this particular disciplines important for policy work and the included development of systematic research strategies practical skills of policy development, implementation within the departments, coordination of research effort and evaluation) was also identified by senior managers in across government and a range of strategies to improve the Manitoba study as important for improving policy the accessibility of research evidence including establish- competencies [[26], p. 30]. Lack of both time and money ing a "Centre for Evidence-Based Policy" to develop inter- were found to present barriers to training [[26], p. 31]. national policy networks, databases and information resources. The UK Cabinet Office also advocated the The UK initiative recommended that training should establishment of a "policy researcher" role with a particu- involve both political and bureaucratic policy-makers (i.e. lar focus on gathering evidence and presenting it in a for- ministers as well as public servants) [37]; a strategy that mat accessible to policy-makers [37]. Some studies encourages relationship building as well as improved recommended the deliberate development of knowledge knowledge sharing. This was echoed by the ANAO report, Page 7 of 15 (page number not for citation purposes) Australia and New Zealand Health Policy 2009, 6:3 http://www.anzhealthpolicy.com/content/6/1/3 which recommended the establishment of a "senior gov- Inter-departmental coordination and networking ernment network in which ministers, senior government Common findings in the studies included inadequate officials and other senior policy-makers can meet from coordination between departments due to a "silo" men- time to time for focused seminars on top-level manage- tality and few opportunities for collaborative reflection on ment issues" [[60], p. 108]. best practice. Joined-up approaches across departments (for cross-cutting issues), within departments and Consultation and communication between departments, service deliverers and those respon- Timely and comprehensive consultation with a range of sible for implementing policies were a major focus in the stakeholders was highlighted as a key factor contributing UK reports [37]. For these approaches to work, they to good policy outcomes in many studies. Consultation needed to be supported by compatible information sys- with both internal stakeholders (within the department, tems, and by organisational cultures and processes [37]. other departments, and central government) and external Improving communication between people developing stakeholders (including clients, the public, policy-makers and implementing policies in different portfolio areas was in other jurisdictions, professional associations, academ- also seen as critical [37]. Sharing best practice was the ics and researchers) was considered important. most frequently identified "enabler" for modernising pol- icy making in the UK report Better Policy Making [[50], p. The Manitoba study stressed the importance of involving 10]. The Canadian Task Force also recommended the stakeholders – including clients and the public – in policy establishment of interdepartmental policy communities development from the beginning, including during the among senior policy executives to share ideas about best initial data gathering stage [[26], p. 26]. Consultation was practice and the struggles of policy work [[44], p. 14]. one of the core competencies explored by the UK Cabinet Office, which found that although "good practice" com- Implementation munication was reasonably widespread, the resources Only a few references to implementation were found in were not always available to undertake comprehensive the studies reviewed. The lack of attention to implementa- consultations [37]. Issues related to consultation were tion in the policy capacity studies is notable, given the also raised in the Victorian Auditor-General's Office report extensive literature which points to the importance of full [61], which identified weaknesses in consultation plan- consideration of implementation issues during policy ning and recommended the use of project management development [45,62-65]. The Canadian study found poor disciplines to promote more systematic planning for con- links between policy development and implementation sultation. [[44], p. 8]. "Greater consideration to policy implementa- tion" was one of the enablers of change identified in the The ability to keep abreast of international developments UK Cabinet Office report Better Policy Making [[50], p. 10]. and also communicate across jurisdictional boundaries Barriers to the integration of policy development and was considered important by the UK Cabinet Office [37], implementation identified in this report included institu- which recommended raising awareness of the "political tional separation and incompatibility of information and wider context", planning for communication, careful technology [[50], p. 41]. There are few clues in the studies, targeting, coordination, and fostering relationships with however, as to how links between policy development other jurisdictions. Cooperation with "external groups and implementation could be strengthened. Some strate- such as community organisations, businesses and other gies that could be explored in further policy capacity jurisdictions" and "strong external links at the political research include: improving the capacity for piloting and level – ministers, members of parliament, ministerial demonstration; strengthening processes for monitoring staff" were also seen by the Australian Public Service Com- and adjustment of incremental policy development and mission [[23], p. 8] as being important for cross-portfolio implementation; and involving implementation manag- policy work. Collaboration with policy researchers was ers in policy development. highlighted in the Canadian research, which recom- mended opportunities for more exchanges [44]. The New Monitoring, evaluation and review Zealand State Services Commission [24] found that build- Monitoring and evaluation were areas where weaknesses ing effective consultation into the policy advice process were commonly identified, as they tended to be neglected, required investment of sufficient time and resources as of insufficient quality or did not feed into policy develop- well as developing particular competencies including ment. For example, the Canadian Task Force found that negotiation and communication skills. Again, the recom- these functions were frequently separated institutionally mendations for improvements seem to be cast in very gen- from (and therefore poorly integrated with) policy devel- eral terms and there is little evidence on which to base opment [[44], p. 11]. specific strategies. Page 8 of 15 (page number not for citation purposes) Australia and New Zealand Health Policy 2009, 6:3 http://www.anzhealthpolicy.com/content/6/1/3 Outcome evaluation was identified as a significant input Fliers: Developing High Performing Policy Units [54] high- to policy advice by the New Zealand SSC [24], which lighted the importance of leadership and strategic man- found that evaluation was most often used for improving agement for improving the performance of "policy units". delivery and implementation and less frequently contrib- This included "policy leadership" (in terms of developing uted to better policy-making. The neglect of evaluation as a coherent overall direction and policy frameworks) and an input to policy-making was attributed to low demand "management leadership" to provide infrastructure and from ministers, political "short-termism", methodologi- support to policy work [[54], p. 10]. Recommendations cal problems with evaluation, lack of evaluation skills and for developing leadership were generally framed around manipulation of evaluation results for political ends [53]. improvements to personnel management (as described To improve effective outcome evaluation, the SSC recom- above) and institutional culture (see below). mended increasing demand from ministers, central agen- cies and parliament for evaluation information, better Supportive institutional culture specification of outcomes, increasing attention to out- Institutional culture was not a strong (or at least an comes (rather than just outputs), improving incentives for explicit) focus of many of the studies reviewed for this reprioritisation and evaluation and improving the skills of paper, with the exception of those undertaken by the UK policy staff to both carry out evaluations and to manage Cabinet Office, the APSC and the New Zealand SSC. external evaluations [24]. The UK work in particular highlights the importance of The Manitoba audit identified deficiencies in perform- institutional culture in recognising the goals of "modern- ance monitoring of policies and found there was an "ad ised" policy-making, such as a long-term and more strate- hoc", rather than systematic, approach to policy evalua- gic focus [37]. A culture of innovation and preparedness tion. However senior policy managers did not rank evalu- to take risks was considered critical to fostering innova- ation among the most important criteria for policy tion [37]. A "risk averse" culture was identified as a barrier capacity and tended to favour a "selective" approach to to improving policy-making [50]. Some recommenda- policy evaluation (as this was perceived to be a more effi- tions for changing institutional culture included bringing cient use of resources) [[26], pp. 27–28]. in staff from outside the public service, providing second- ments for policy officers and improving networking both The UK Cabinet Office found there were few opportuni- within government and with external agencies and other ties for policy-makers to learn from their own and others' jurisdictions [37]. experience [37]. Encouraging a culture of evaluation and improving the quality of evaluations undertaken were The APSC's Connecting Government report [23] identified a both areas that needed improvement [37]. A major rec- number of features of organisational culture important ommendation arising from the UK initiative was the use for the success of whole-of-government approaches. Like of peer review processes to allow sharing of "good prac- the UK reports, it emphasised the importance of innova- tice" and organisational learning, and also to encourage tion and the ability to manage risk. It also highlighted an cultural change [37]. Peer review was also recommended environment of teamwork and trust and "encouragement by the ANAO [[60], p. 115] and the Victorian Auditor- of the expression of diverse views" and the ability to "bal- General's Office. Other mechanisms suggested by the UK ance the tension between short-term and long-term goals" Cabinet Office [37] for improving policy evaluation [23]. Exposure of public servants to different organisa- included strategic management of the evaluation process, tional cultures (through secondments and networking) better resource allocation, developing a "centre of excel- was recommended as one way of fostering a different lence" devoted to policy evaluation and establishing proc- organisational culture [23]. esses for people implementing policies to feed back information about the effectiveness and acceptability of Clarity of policy direction and policy frameworks was policies. highlighted as important by the Manitoba Office of the Provincial Auditor [[26], pp. 25–6], which described a Strategic management and leadership lack of clear principles or conceptual frameworks for gen- Strategic management and leadership of the policy devel- erating or evaluating policy options – although this was opment process were common themes. The Canadian identified as a very important aspect of policy develop- Government Task Force [[44], p. 11] found that system- ment by senior policy managers. It was also found by the atic management of the policy process was patchy, and the New Zealand SSC to be important for improving the per- need for it was underestimated. Leadership direction and formance of policy units [54]. support was ranked as very important or important by 100% of interviewees in the Manitoba audit of policy capacity [[26], p. 22]. The New Zealand SSC report High Page 9 of 15 (page number not for citation purposes) Australia and New Zealand Health Policy 2009, 6:3 http://www.anzhealthpolicy.com/content/6/1/3 which they interact and the processes by which change at Discussion Implications for policy capacity building and workforce the individual level can affect the organisation – and vice development in the health sector versa. The policy capacity evaluations and audits reviewed in this paper suggest some of the factors likely to contribute Models and methods for policy capacity research to policy capacity in the health sector. Based on the find- There are continuing uncertainties about the conceptuali- ings of this review, some propositions about the organisa- sation of policy capacity and the methods for researching tional structures and processes, organisational cultures it. All the models for conceptualising policy work and the and individual competencies that are likely to contribute methods for evaluating policy capacity used in the studies to health policy capacity were developed by the authors. reviewed in this paper have their own limitations. Some These are listed in Figures 1, 2 and 3. studies used relatively linear models of the policy process and fairly narrowly conceived methodologies (particu- There is clearly a need for further research to explore how larly the state/provincial level audits), which provided lit- these general principles might be applied in the health tle scope for examining the political and contextual sector and to examine more closely the institutional aspects of policy-making. Policy work was often con- arrangements and competencies which create the condi- ceived as the provision of policy advice to political deci- tions for good policy outcomes in the health sector. This sion-makers; an approach which may not capture the full is the focus of research in which the authors are currently range of possible policy engagements and the more active engaged. Questions which might usefully be addressed in role that policy workers often play in shaping policy. such research include: Most of the studies reviewed for this paper used multiple � To what extent do the generic elements of policy capac- data collection methods. In most cases, interviews with ity apply to health policy in particular? senior officials were used to collect data. These were often supplemented by interviews with other players such as � Are some elements of policy capacity more important junior policy officers and external policy research experts than others in the context of health policy development in Canada, recent leavers in the UK and cabinet ministers and implementation? in Manitoba. In some cases interviews were combined with other data collection methods including case studies Grounding health policy capacity research within the of policy episodes or policy units, focus group interviews study of the health policy development process would or round tables, surveys, and training needs analysis. A provide the potential to examine more closely the ena- number of appropriate methods are available and the blers and barriers to good health policy process and ena- choice of particular methods depends on the context and ble more specific recommendations for capacity building focus of the study. Triangulation of different methods to be developed. appears to offer advantages. Jurisdictional differences are clearly important in shaping Analysis of case studies appears to provide a useful way of both policy capacity itself and the sorts of capacity build- exploring the conditions for good policy-making, particu- ing strategies that might be needed. Some factors that larly institutional culture, within context. However, in all might be important include differences in national sys- the studies reviewed that included analysis of case studies, tems (such as unitary governments or federal systems), the cases had been chosen by policy-makers themselves differences in health system contexts, and different insti- and put forward as examples of good practice. Methods tutional arrangements and policy coordination mecha- for policy capacity research could be strengthened by nisms (e.g. the Council of Australian Governments). more careful selection of case studies according to their Further research involving different jurisdictions is potential to illuminate the elements of policy capacity, needed to explore the extent to which comparative differ- both positive and negative. ences matter in the context of health policy capacity. None of the studies included in this review attempted to A further issue which should be addressed in future evaluate policy capacity against the outcomes of the pol- research is the relationship between individual and organ- icy process, although some studies evaluated policy capac- isational capacity. Clearly they are closely interrelated. ity in terms of outputs, such as the quality of policy briefs The capacity of individuals is shaped and constrained by or the satisfaction of ministers. the organisational context; the organisational context can only be changed through the efforts of people. Previous There has been extensive debate in the literature about the policy capacity research does not tell us very much about difficulties inherent in evaluating policy outputs and out- the nexus between the two, the mechanisms through comes. First, there are problems attributing policy out- Page 10 of 15 (page number not for citation purposes) Australia and New Zealand Health Policy 2009, 6:3 http://www.anzhealthpolicy.com/content/6/1/3 Or ganisation al str uctur es and p r ocesses which supp or t health policy capacity includ e: evidence and infor mation – provision for generating evidence and collecting information for policy development knowled ge management – systematic provision for recognising and accessing the individuals and units across the organisation (and beyond) which have specialised knowledge which might inform policy development; incorporating history and experience in knowledge management frameworks wor kfor ce development – programs in place to produce and reproduce a broad mix of human resources to support policy making and to develop individual policy practitioners con sultation and communication channels an d p r ocesses – relationships with other levels of government, other sectors, non-government organisations, civil society, stakeholders and politicians; processes for timely and meaningful consultation; relationships between the minister’s office and departmental policy makers inter -dep ar tmen tal coor d in ation and n etwor king – processes and support systems for working collaboratively across departmental boundaries imp lementation lin kage – capacity for piloting and demonstration where appropriate; capacity for incremental implementation with evaluation and adjustment as needed; and involvement of implementation managers in policy analysis and development mon itor in g, evaluation and r eview – continuity between policy development and evaluation phases; access to evaluation expertise; processes for monitoring and incremental policy adjustment; peer review of the policy development process str ategic man agemen t, coor dination and accountab ility – policy leadership and accountability for the quality of such leadership Organi Figure 1 sational structures and processes that support health policy capacity Organisational structures and processes that support health policy capacity. Page 11 of 15 (page number not for citation purposes) Australia and New Zealand Health Policy 2009, 6:3 http://www.anzhealthpolicy.com/content/6/1/3 Or ganisation al cultur es which suppor t h ealth policy cap acity ar e: open to innovation and different ways of seeing things; flexible with respect to process and different ways of working; able to provide a plurality of frameworks to guide and facilitate policy work; able to build strategic alliances, collaboration and trust; able to capitalise on windows of opportunity, tolerate mistakes and manage risk; open to naming problems (before solutions are evident); tolerant of constructive conflict and the engagement of diverse views; able to work across agency boundaries; able to balance long and short term goals; supportive of effective teamwork; conscious of their own history; and committed to valuing corporate memory. F Figure 2 eatures of organisational culture that support health policy capacity Features of organisational culture that support health policy capacity. comes to policy capacity or even to particular policies. It is ured. Where policy outputs are evaluated, it is best to difficult to link policy advice with decisions and their out- employ qualitative methods using professional judge- comes [66,67]. Even the notion of isolating a discrete area ment and peer review [66,67]. Nicholson argued that sen- of policy in order to assess outcomes and trace the policy ior policy advisers who are highly regarded by their peers development process is questionable when policies are are likely to be the best judges of the quality of policy interwoven and cross-portfolio in nature [66]. Many other advice [67]. variables impact on policy decision-making and its out- comes [57,68], and often the advice of public servants is Retrospectively judging the outcomes of policy from the only one source of advice used by politicians [67]. Sec- position of an outsider carries the additional danger of ond, there is the issue of timeframes. Policy outcomes are ignoring the contingencies of all the scenarios that did not often not evident for many years. The policy development eventuate, whereas at the point of action, the policy oper- itself can often take a long time, and there is often a long ative would have been confronting a wider range of possi- lag time between policy development and implementa- ble scenarios, all of which needed to be weighed and tion [67]. Incremental changes to policy during imple- considered. Any assessment of policy capacity must take mentation may also mean that the long-term outcomes into account the range of possible scenarios and contin- are no longer traceable to a particular period of policy gencies confronting the policy-makers at the time the pol- development [66]. Third, judgements about the value or icy episode took place. This suggests that policy capacity "goodness" of policy vary widely as there is political and research needs to draw on the accounts of the policy prac- ideological disagreement between different interest titioners themselves of their experiences of the policy groups and actors over goals and outcomes [38,69]. development process and the environments in which the Finally, there are no simple or universally applicable policy episode took place. models available for evaluating policy work; it is generally agreed that there would be little value in trying to develop Despite these concerns, some attempts have been made to a set of generic criteria to use for evaluating policy capacity delineate criteria for evaluating policy capacity. Thissen [58,69]. and Twaalfhoven [69] described three detailed sets of cri- teria for evaluating policy analytic activities, based on dif- For all these reasons, there are no objective standards ferent views of policy analysis: as information provision; against which policy outcomes or outputs can be meas- as a participative policy-oriented process; and as a set of Page 12 of 15 (page number not for citation purposes) x Australia and New Zealand Health Policy 2009, 6:3 http://www.anzhealthpolicy.com/content/6/1/3 Competencies that sup por t h ealth policy cap acity in clud e: Knowledge and exper ience contextual knowledge – of institutional and administrative arrangements, and contemporary policy environments sector al knowled ge – of the health and/or human services field generally histor ical and comp ar ative knowled ge – including the genealogy and track record of various dynamics, models and methods in health and welfare systems theor y fr om acr oss the academic disciplin es – and the ability to use such theory in description, analysis, interpretation, explanation and strategy Skills of policy pr actice basic wor king skills – research and evaluation; writing (and re-drafting); communication skills (both exposition and listening) skills in functional (r ational) an alysis – ability to name the problem, articulate causes, identify and evaluate options, and use evidence to evaluate causes and options skills in p olitical analysis – ability to map the political field, conceive interventions, imagine, project and evaluate scenarios inter -sector al skills – capacity to communicate and work across different disciplines and sectors Per sonal attr ibu tes cr eativity, intuition and judgemen t Individual com Figure 3 petencies that support health policy capacity Individual competencies that support health policy capacity. methods and tools. The framework for organising these ticular circumstances of the case. Painter and Pierre [1] criteria distinguishes between inputs, content, process, suggested a set of evaluative criteria which include a set of results, use, and effects. The criteria proposed by Thissen values (coherence, "public-regardingness", credibility, and Twaalfhoven are designed for evaluating policy ana- decisiveness and resoluteness) and a set of support sys- lytic activities related to specific policy issues, rather than tems (collective decision processes, planning and evalua- the policy analytic capacity of organisations. Thissen and tion, information and analysis, and coordination Twaalfhoven [69] also noted that a single set of univer- procedures). They point out, however, that the evaluative sally applicable criteria is not feasible and the choice of criteria will "necessarily be contested" [1]. While these cri- criteria depends on the perspective adopted and the par- teria may provide useful checklists for future policy capac- Page 13 of 15 (page number not for citation purposes) x Australia and New Zealand Health Policy 2009, 6:3 http://www.anzhealthpolicy.com/content/6/1/3 ity research, they do not yield "objective" measures of Health policy is difficult; policy-makers are working with policy process because of the methodological problems complexity, conflict and uncertainty. However, much is at noted. stake. Further research into the conditions for effective policy-making in the health sector would appear to offer The conceptualisation of policy capacity underpinning significant returns. much of the policy capacity research has been criticised more broadly for its reliance on a rational model of pol- Competing interests icy-making that privileges some aspects of policy-making The authors declare that they have no competing interests. over others. The implicit assumption is that improving and systematising the technical aspects of policy-making Authors' contributions will result in better policy. Brans and Vancoppenolle [42] DHG carried out the data collection and comparative pointed out that this approach conflicts with other per- analysis and drafted the manuscript. DGL and DON par- spectives on improving policy-making, such as "interac- ticipated in the design of the study and the data analysis, tive governance", which emphasise citizen involvement and critically reviewed drafts of the manuscript. All and participation and the need for government to be authors read and approved the final manuscript. responsive and flexible. This suggests there is a dimension to policy capacity that is not amenable to technical solu- Acknowledgements The authors would like to thank Alison Hughes for her thoughtful com- tions and that requires more attention to the interface ments on a draft of this paper. We would also like to thank the two anon- between the bureaucracy and other players outside gov- ymous reviewers whose very helpful feedback assisted us in improving the ernment, including industry stakeholders and the wider quality of the paper. society [[27], p. 12, 70]. These relational aspects of policy capacity should be a stronger focus in future policy capac- References ity research. 1. Painter M, Pierre J: Unpacking policy capacity: Issues and themes. In Challenges to State Policy Capacity Edited by: Painter M, Pierre J. Hampshire and New York: Palgrave MacMillan; 2005:1-8. Conclusion 2. Australian Institute of Health and Welfare: Aboriginal and Torres Strait In the absence of research into health policy capacity, Islander Health Performance Framework, 2006 Report: Detailed Analyses. some lessons can be drawn from experience in evaluating AIHW cat. no. IHW 20. Canberra: AIHW 2007. 3. Hayes LJ, Quine S, Taylor R, Berry G: Socio-economic mortality public sector policy capacity in Australia and other coun- differentials in Sydney over a quarter of a century, 1970–94. tries. Aust N Z J Public Health 2002, 26:311-317. 4. Andrews K: National Strategy for an Ageing Australia. An Older Australia: Challenges and Opportunities for All Canberra: Commonwealth of Aus- The review of public policy capacity research undertaken tralia; 2002. for this paper suggests a number of elements or domains 5. Australian Institute of Health and Welfare: Health Expenditure Australia 2002–03 Canberra: Australian Institute of Health and Welfare; 2004. of policy capacity that could be used to provide a concep- 6. Butler CD, Douglas B, McMichael AJ: Globalisation and environ- tual framework for future research. These include: infor- mental change: implications for health and health inequali- ties. In The Social Origins of Health and Well-Being Edited by: Eckersley mation and evidence; personnel management and R, Dixon J, Douglas B. Melbourne: Oxford University Press; 2001. workforce development; consultation and communica- 7. Australian Institute of Health and Welfare: Chronic Diseases and Asso- tion; implementation; monitoring, evaluation and ciated Risk Factors in Australia, 2001 Canberra: Australian Institute of Health and Welfare; 2002. review; strategic management and leadership; and sup- 8. Australian Government Department of Health and Ageing: Protecting portive organisational culture. Individual competencies Australia from Communicable Diseases: Everybody's Business. A special for policy practitioners identified in the studies included: report from the Commonwealth Chief Medical Officer. Canberra 2004. 9. Eckersley R, Dixon J, Douglas B, Eds: The Social Origins of Health knowledge (of context, different disciplines, and systems and Well-being. Melbourne: Cambridge University Press; 2001. and developments in other countries); practical skills of 10. Marmot MG, Wilkinson RG, Eds: Social Determinants of Health. Oxford, New York: Oxford University Press; 1999. policy-making, such as analytic, technical and communi- 11. Armstrong BK, Gillespie J, Leeder SR, Rubin GL, Russell LM: Chal- cation skills; and personal attributes such as creativity, lenges in health and health care for Australia. MJA 2007, intuition and judgment. 187:485-489. 12. Duckett SJ: Commonwealth/state relations in health. In Health Policy in the Market State Edited by: Hancock L. St Leonards, NSW: The review of models and methods suggests that future Allen & Unwin; 1999:71-85. 13. Duckett SJ: The Australian Health Care System 2nd edition. South Mel- policy capacity research should be grounded in careful bourne: Oxford University Press; 2004. description and analysis of the policy process that is not 14. Commonwealth of Australia: The Blame Game: Report of the Inquiry into limited to a single narrow perspective. Such research Health Funding House of Representatives Standing Committee on Health and Ageing; 2006. should also use a range of qualitative methods that 15. Hancock L: Australian federalism, politics and health. In Health employ the judgement of policy practitioners themselves Policy in Australia Edited by: Gardner H, Barraclough S. South Mel- in evaluating policy capacity, rather than attempting to bourne: Oxford University Press; 2002:49-78. 16. Swerissen H, Duckett SJ: Health policy and financing. In Health measure policy capacity according to "objective" meas- Policy in Australia Edited by: Gardner H, Barraclough S. Melbourne: ures of policy outcomes or explicit evaluative criteria. Oxford University Press; 2002:13-48. Page 14 of 15 (page number not for citation purposes) Australia and New Zealand Health Policy 2009, 6:3 http://www.anzhealthpolicy.com/content/6/1/3 17. Hancock L: Policy, power and interests. In Health Policy in the Mar- 43. Bakvis H: Rebuilding policy capacity in the era of the fiscal div- ket State Edited by: Hancock L. St Leonards, NSW: Allen & Unwin; idend: a report from Canada. Governance 2000, 13:71-103. 1999:19-47. 44. Canadian Government: Strengthening Our Policy Capacity. Report of the 18. Lewis JM: Health Policy and Politics: Networks, Ideas and Power East Haw- Task Force on Strengthening the Policy Capacity of the Federal Government thorn: IP Communications; 2005. 1996. 19. Ham C: The background. In Health Care Reform: Learning from Inter- 45. Bridgman P, Davis G: The Australian Policy Handbook Crows Nest, national Experience Edited by: Ham C. Bristol, PA: Open University NSW: Allen & Unwin; 1998. Press; 1997:1-20. 46. Policy Research Initiative [http://www.policyresearch.gc.ca/] 20. Ross B, Snasdell-Taylor J, Cass Y, Azmi S: Health Financing in Australia: 47. Nutley S, Webb J: Evidence and the policy process. In What The Objectives and the Players Canberra: Commonwealth Department Works? Evidence-based Policy and Practice in Public Services Edited by: of Health and Aged Care; 1999. Davies HTO, Nutley SM, Smith PC. Bristol: The Policy Press; 21. Hall J: Incremental change in the Australian Health Care Sys- 2000:13-41. tem. Health Aff 1999, 18:95-109. 48. Bogdanor V, Ed: Joined up Government. London: Oxford Univer- 22. Australian Public Service Commission: The Australian Experience of sity Press for the British Academy; 2005. Public Sector Reform. Occasional Paper Two Commonwealth of Aus- 49. Mulgan G: Joined up government in the United Kingdom: tralia; 2003. Past, present and future. Canberra Bulletin Pub Adm 2002, 23. Australian Public Service Commission: Connecting Government: Whole 105:25-29. of Government Responses to Australia's Priority Challenges Management 50. Bullock H, Mountford J, Stanley R: Better Policy-Making London: Cen- Advisory Committee, APSC, Commonwealth of Australia; 2004. tre for Management and Policy Studies, Cabinet Office, United King- 24. State Services Commission: Essential Ingredients: Improving the Quality dom; 2001. of Policy Advice. Occasional Paper No. 9. Wellington 1999. 51. Wright JC: Strategic Social Policy Advice: Improving the Information Base. 25. OECD: Synthesis of Reform Experiences in Nine OECD Countries: Govern- Working Paper No. 8 New Zealand State Services Commission; 1999. ment Roles and Functions, and Public Management: Government of the 52. Wolf A: Building Advice: The Craft of the Policy Professional. Working Future: Getting from Here to There. Paris: OECD 1999. Paper No. 7 Wellington: New Zealand State Services Commission; 26. Manitoba Office of the Provincial Auditor: A Review of the Policy Devel- 2000. opment Capacity within Government Departments 2001. 53. State Services Commission: Looping the Loop: Evaluating Outcomes and 27. UK Government: Modernising Government 1999. Other Risky Feats. Occasional Paper No. 7. Wellington 1999. 28. Office of the Auditor General of Canada: Public Service Management 54. State Services Commission: High Fliers: Developing High Performing Pol- Reform: Progress, Setbacks and Challenges. Ottawa 2001. icy Units. Occasional Paper No. 22. Wellington 1999. 29. Lin V, Gibson B, Eds: Evidence-based Health Policy: Problems 55. State Services Commission: Pieces of the Puzzle: Machinery of Govern- & Possibilities. South Melbourne: Oxford University Press; 2003. ment and the Quality of Policy Advice. Working Paper No. 4. Wellington 30. Davies HTO, Nutley SM, Smith PC, Eds: What Works? Evidence- 2000. based Policy and Practice in Public Services. Bristol: The Policy 56. State Services Commission: Gaining Through Training: Developing High Press; 2000. Performing Policy Advisors. Working Paper No. 2. Wellington 2000. 31. Peters BG: Managing Horizontal Government: The Politics of Coordination. 57. Di Francesco M: An evaluation crucible: Evaluating policy Research Paper No. 21 Canadian Centre for Management Develop- advice in Australian central agencies. Australian Journal of Public ment; 1998. Administration 2000, 59:36-48. 32. Lindquist E: Strategy, capacity and horizontal governance: 58. Uhr J, Mackay K, Eds: Evaluating Policy Advice: Learning from Perspectives from Australia and Canada. Optimum Online 2004, Commonwealth Experience. Canberra: Federalism Research 34:. Centre, Australian National University and Commonwealth Depart- 33. Hess M, Adams D: Knowing and skilling in contemporary pub- ment of Finance; 1996. lic administration. Aust J Public Adm 2002, 61:68-79. 59. Di Francesco M: Process not outcomes in new public manage- 34. Painter M, Pierre J: Conclusions: Challenges to policy capacity. ment? 'Policy coherence' in Australian Government. The In Challenges to State Policy Capacity: Global Trends and Comparative Per- Drawing Board: Aust Rev Public Aff 2001, 1:103-116. spectives Edited by: Painter M, Pierre J. Hampshire and New York: Pal- 60. Australian National Audit Office: Developing Policy Advice. Audit Report grave Macmillan; 2005:255-261. No. 21, 2001–2002 2002. 35. Davis G: Conclusion: policy capacity and the future of govern- 61. Victorian Auditor-General's Office: Report on Public Sector Agencies: ance. In The Future of Governance Edited by: Davis G, Keating M. St Results of Special Reviews and Other Studies, August 2004 2004. Leonards NSW: Allen & Unwin; 2000. 62. Pressman JL, Wildavsky A: Implementation: How Great Expectations in 36. Howlett M: Enhanced policy analytical capacity as a prerequi- Washington are Dashed in Oakland; or, Why it's Amazing that Federal Pro- site for effective evidence-based policy-making: Theory, con- grams Work at all Berkeley: University of California Press; 1973. cepts and lessons from the Canadian case. In Workshop on 63. Ham C, Hill M: The Policy Process in the Modern Capitalist State 2nd edi- Evidence-Based Policy; International Research Symposium on Public Man- tion. New York, London: Harvester Wheatsheaf; 1993. agement XII Queensland University of Technology, Brisbane; 2008. 64. Younis T, Ed: Implementation in Public Policy. Aldershot: Dart- 37. UK Cabinet Office: Professional Policy Making for the Twenty-First Cen- mouth Pub. Co; 1990. tury Strategic Policy Making Team, Cabinet Office; 1999. 65. Hogwood BW, Gunn LA: Policy Analysis for the Real World Oxford: 38. Peters BG: The Policy Capacity of Government. Research Paper No. 18 Oxford University Press; 1984. Canada: Canadian Centre for Management Development; 1996. 66. Waller M: Evaluating policy advice. Aust Journal Public Adm 1992, 39. Aucoin P, Bakvis H: Public service reform and policy capacity: 51:440-449. Recruiting and retaining the best and the brightest? In Chal- 67. Nicholson J: Measures for monitoring policy advice. In Evaluat- lenges to State Policy Capacity: Global Trends and Comparative Perspectives ing Policy Advice: Learning from Commonwealth Experience Edited by: Uhr Edited by: Painter M, Pierre J. Hampshire and New York: Palgrave J, Mackay K. Canberra: Federalism Research Centre, The Australian Macmillan; 2005:164-184. National University and Commonwealth Department of Finance; 40. Janicke M: The political system's capacity for environmental 1996:26-40. policy. In National Environmental Policies: A Comparative Study of 68. Keating M: Defining the policy advising function. In Evaluating Capacity-Building Edited by: Janicke M, Weidner H. Berlin: Springer- Policy Advice: Learning from Commonwealth Experience Edited by: Uhr J, Verlag; 1997:1-24. Mackay K. Canberra: Federalism Research Centre, The Australian 41. Parsons W: Not just steering but weaving: Relevant knowl- National University and Commonwealth Department of Finance; edge and the craft of building policy capacity and coherence. 1996:61-68. Aust J Public Adm 2004, 63:43-57. 69. Thissen WAH, Twaalfhoven PGH: Towards a conceptual struc- 42. Brans M, Vancoppenolle D: Policy-making reforms and civil ture for evaluating policy analytic activities. Euro Journal Oper- service systems: An exploration of agendas and conse- ational Res 2001, 219:627-649. quences. In Challenges to State Policy Capacity Edited by: Painter M, 70. Jayasuriya K: Capacity beyond the boundary: New regulatory Pierre J. Hampshire and New York: Palgrave Macmillan; state, fragmentation and relational capacity. In Challenges to 2005:164-184. State Policy Capacity Edited by: Painter M, Pierre J. Hampshire and New York: Palgrave Macmillan; 2005. Page 15 of 15 (page number not for citation purposes)

Journal

Australia and New Zealand Health PolicySpringer Journals

Published: Feb 26, 2009

References