Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

Opening the black box: Presenting a model for evaluating organizational-level interventions

Opening the black box: Presenting a model for evaluating organizational-level interventions European Journal of Work and Organizational Psychology, 2013 Vol. 22, No. 5, 601–617, http://dx.doi.org/10.1080/1359432X.2012.690556 Opening the black box: Presenting a model for evaluating organizational-level interventions 1 2 Karina Nielsen and Raymond Randall National Research Centre for the Working Environment, Copenhagen, Denmark Department of Psychology, University of Leicester, Leicester, UK Organizational-level occupational health interventions are often recommended when improvements in working conditions, employee health, and well-being are sought within organizations. Research has revealed that these interventions result in inconsistent effects despite being based on theoretical frameworks. This inconsistency indicates that intervention studies need to be designed to examine directly how and why such interventions bring about change and why they sometimes fail. We argue that intervention studies should include a process evaluation that includes a close examination of the psychological and organizational mechanisms that hinder and facilitate desired intervention outcomes. By drawing on existing intervention literature we present an evidence-based model containing three levels of elements that appear to be crucial in process evaluation. We describe how this model may be applied and developed in future research to identify better the mechanisms that link intervention processes to intervention outcomes. Keywords: Methods; Organizational-level occupational health interventions; Process evaluation. Organizational-level occupational health interven- the small body of evaluation research (Egan, Bambra, tions can be defined as: ‘‘planned, behavioral, Petticrew, & Whitehead, 2009; Murta, Sanderson, & science-based actions to remove or modify the causes Oldenburg, 2007; Semmer, 2011). In their review, of job stress’’ (Mikkelsen, 2005, p. 152). In current LaMontagne, Keegel, Louie, Ostry, and Landsbergis European legislation there is a clear emphasis on the (2007) concluded that the published literature focused use of such interventions (i.e., changes in the design, on effect evaluation and contained relatively little, organization, and management of work) as the potentially important, process evaluation data about preferred way of improving working conditions and how interventions were planned and implemented. In tackling problems such as work stress (EU-OSHA, studies where process evaluation is attempted, it is 2010). This strategy has, however, been criticized often based on anecdotal data that have not been because of the lack of a large and consistent body of subjected to structured analysis (Bambra, Egan, evidence that shows these interventions to have a Thomas, Petticrew, & Whitehead, 2007; Murta et al., positive impact on working conditions and employee 2007; Roen, Arai, Roberts, & Popay, 2006). In this health and well-being (e.g., Briner & Reynolds, 1999; article we present a three-level evidence-based process Richardson & Rothstein, 2008). Others have argued evaluation model. This is intended to provide a that there is evidence of the positive impact of structure that researchers can use to guide the rigorous organizational-level occupational health interven- collection of detailed process evaluation data. We tions but that too few studies have examined why argue that the model can be used to strengthen the and how such interventions have succeeded or failed evaluation of organizational-level occupational health thus placing limits on the external validity of much of interventions in a number of ways. Correspondence should be addressed to Karina Nielsen, National Research Centre for the Working Environment, Lersoe Park Alle 105, DK-2100 Copenhagen, Denmark. Email: [email protected]. Raymond Randall’s current affiliation is the School of Business and Economics, Loughborough University, Loughborough, UK. This research was funded by the National Work Environment Research Fund (Grant 16-2004-09). The authors declared no potential conflicts of interests with respects to the authorship and/or the publication of this article. © 2013 The Author(s). Published by Taylor & Francis. This is an Open Access article. Non-commercial re-use, distribution, and reproduction in any medium, provided the original work is properly attributed, cited, and is not altered, transformed, or built upon in any way, is permitted. The moral rights of the named author(s) have been asserted. 602 NIELSEN AND RANDALL Organizational-level occupational health interven- actions in implementing any intervention and their tions are proactive in that they are focused on influence on the overall result of the intervention’’ reducing or eliminating the sources of job stress (Nytrø, Saksvik, Mikkelsen, Bohle, & Quinlan, 2000, (Hurrell & Murphy, 1996; LaMontagne et al., 2007). p. 214). This means that process evaluation (PE) may Researchers have found that some show no effects, be used to (1) provide feedback for improving others have been linked to improvements in working interventions, (2) replicate interventions in other conditions and employee health and well-being and a settings minimizing the number of pitfalls associated small number have prefaced deteriorations in these with a given intervention, (3) interpret the outcomes variables (Bambra et al., 2007; Egan et al., 2007; of interventions (Goldenhar, LaMontagne, Katz, Semmer, 2003, 2006). This mixed evidence has led to Heaney, & Landsbergis, 2001), and (4) help us considerable confusion and debate about how re- conclude on the generalizability, applicability, and search findings should be used to guide practice. The transferability of interventions studies (Armstrong problem stems from a prevailing focus on effect only et al., 2008). In summary, PE is needed to evaluate evaluation (Ruotsalainen et al., 2006). It is difficult to the generalizability of an intervention (to answer conclude why and how an intervention worked from questions such as ‘‘Under which circumstances will effect evaluation data only (Lipsey, 1996; Rychetnik, an intervention work?’’ and ‘‘Which were the Frommer, & Shiell, 2002) because effect-only evalua- processes that facilitated the change?’’) so that it tion data masks intervention effects that are sensitive can be implemented successfully in a variety of to variations in intervention processes (Lipsey, 1996). settings (Cooper, Dewe, & O’Driscoll, 2001). The nature of organizational-level occupational One of the reasons why PE is lacking in current health interventions indicates that their working research on organizational-level occupational health mechanisms are unlikely to be separate from the interventions may be that researchers are uncertain systems within which they operate. These interven- about what should be included in such evaluation. A tions require changes to complex social systems and number of papers have listed concepts that may be may be met with much resistance and have unintended examined in PE (Egan et al., 2009; Lipsey & Cordray, side-effects (Semmer, 2003). The internal validity of 2000; Murta et al., 2007; Nytrø et al., 2000). These evaluation studies of these interventions may be include organizational contexts, intervention reach, threatened by concurrent changes that worked against dose delivered and dose received, intervention fide- the intervention plan: This means that the same lity, support and available resources, recruitment, intervention could yield powerful effects if the context and attitudes towards the intervention. However, was less disruptive to the intervention plans and there is no integrated, evidence-based framework that processes (Mikkelsen, 2005). Moreover, changing describes the elements that need to be included in complex systems requires wide-ranging multifaceted process evaluations of organizational-level occupa- activities and diluted or disrupted intervention activ- tional health interventions. ities may not be intense enough to have an impact We propose that the factors that may have an impact (Bambra et al., 2007). Considering the complexity of on the outcomes of an organizational-level occupa- organizational-level occupational health interven- tional health intervention can be grouped into three tions, evaluation models and methods are needed that themes. The themes are: the intervention design and can be used to identify how the potential effects of implementation, the intervention contexts, and parti- interventions on health and well-being are moderated cipants’ mental models (of the intervention and their and mediated by intervention processes. work situation). The first theme determines the max- Such a shift in evaluation strategy represents a imum levels of intervention exposure that can be move away from ‘‘black box’’ and a move towards an achieved; the latter two represent the factors that may approach that can elaborate on the mechanisms moderate or mediate the link between any intervention through which changes in the outcomes operate: exposure and its outcomes. Within each theme we Looking inside the black box reveals various sources identify a number of specific questions about the of variation (Lipsey & Cordray, 2000). A fundamental intervention processes that need to be answered objective of this shift is to differentiate between theory/ through the collection of process evaluation data. programme failure (that the theory behind the Together, the collection of data across these themes is intervention did not address the problem) and likely to provide the evaluator with some useful insights implementation failure (that the way the intervention into the factors that influence the outcomes of an was implemented was incomplete or designed in such a organizational-level occupational health intervention. way that the intervention would have failed even if the In this article we have chosen to focus on the theory behind the intervention was correct) (Harachi, literature on health and well-being interventions at Abbott, Catalan, Haggerty, & Fleming, 1999). the organizational level and the factors and elements Intervention process has been defined as ‘‘indivi- identified in this literature. This is because PE dual, collective and management perceptions and models are needed that fit with the measurement PROCESS EVALUATION TOOL 603 opportunities and constraints operating in this with employees in two organizations, one private- domain; however, models of PE have been used with and one public-sector, to identify further themes and considerable success in other disciplines such as confirm existing themes (references withheld to public health, organizational development, and orga- ensure anonymity). nizational change (Armenakis & Bedeian, 1999; Burke & Litwin, 1992; Cummings & Worley, 2009; Rossi, Lipsey, & Freeman, 2004; Steckler & Linnan, A MODEL OF PROCESS EVALUATION 2002), and many of the factors and elements Intervention design and implementation identified in these disciplines may also be relevant in a model of the evaluation of the processes of In the following section we discuss the elements that organizational-level occupational health interven- should be included to document the intervention design tions. In developing a model specific to organiza- and implementation. We focus on three overarching tional-level occupational health interventions, we elements: initiation, intervention activities, and imple- drew on four sources of information. First, we mentation strategy. These themes are not orthogonal: identified two rigorous review articles focusing on Because we are describing interlinked and complex process factors (Egan et al., 2009; Murta et al., 2007) organizational processes, issues within a theme may and from here we drew the relevant factors identified also interact with other issues in other themes. The in our model. Second, we identified a number of model is presented in Figure 1. To allow readers to see papers that focused on the topic of intervention more clearly how the model may be translated into PE implementation (Cooper et al., 2001; Guastello, 1993; tools and methods, the key issues are presented as Lipsey, 1996; Nytrø et al., 2000; Pettigrew, 1990; questions that should be addressed during PE. Semmer, 2003, 2006, 2011; Shannon, Robson, & Guastello, 1999; Vedung, 2006). Third, we reviewed Initiation: Who initiated the intervention and for existing intervention studies to identify any analysis, what purpose? The motivation driving an however anecdotal, of process factors such as mental intervention may be related to problems internal to models, context, and/or intervention design and the organization (to deal with a crisis, to improve implementation. These are the papers that are quality and productivity or to become a healthy referred to throughout this article. Finally, we workplace) or external challenges (e.g., legislative developed a preliminary model which we tested in requirements) or a combination of both (Kompier, semistructured interviews (N ¼ 54) with key stake- Geurts, Grundemann, Vink, & Smulders, 1998; holders including human resources practitioners, Shannon & Cole, 2004). Any intervention can internal consultants, managers, and 44 focus groups stabilize or displace current power structures and Figure 1. Model of process evaluation. 604 NIELSEN AND RANDALL therefore reasons for the intervention are likely to who is responsible and who were the targets of influence the buy-in of key stakeholders (Fredslund & intervention (Nielsen, Randall, Holten, & Rial Strandgaard, 2005). It is therefore important to Gonza´ lez, 2010). The contents of action plans also explore who defined the problem, who decided what often highlight the potential ‘‘active ingredients’’ of should be done, and who should implement change. the intervention that could be linked to intervention This means that it is also important to identify the outcomes (Nielsen, Randall, & Christensen, 2010). key stakeholders in the initiation process: These may be managers, employees, union representatives, Implementing intervention activities: Did the occupational health practitioners, and clients. This intervention reach the target group? Careful identification may help understand the reactions and documentation of the actual implementation of actions of other key stakeholders (see later). intervention activities is a vital element of process Some of the effects of this decision-making process evaluation (Semmer, 2003, 2006, 2011) because this have been identified by Egan et al. (2007). They will highlight any discrepancies between the planned concluded that interventions initiated for perfor- intervention and its implementation (Roen et al., mance reasons were found to have an adverse impact 2006). Important questions include: ‘‘Which aspects on employee health and well-being, whereas inter- of the intervention activities brought about ventions whose rationale was to improve employee noticeable changes?’’ ‘‘How many changes were health and well-being was found to have a positive delivered to whom?’’ ‘‘Who noticed/reported these effect on these same outcome measures. changes?’’ The potential active ingredients of the intervention also need to be reexamined and Developing intervention activities: Did the intervention compared to the active ingredients identified in the activities target the problems of the workplace? It has intervention plan. This identification helps to rule out been argued by many researchers that the correct rival hypotheses for intervention results, i.e., that tailoring of an intervention to the needs of other factors than the intervention account for stakeholders requires a thorough risk assessment observed changes or lack of change (Lipsey, 1996). (LaMontagne et al., 2007; Nielsen, Randall, Holten, The importance of documenting differences be- &RialGonza´ lez, 2010; Noblet & LaMontagne, tween planned and actual exposure has been high- 2009). A thorough risk assessment is a crucial lighted in a number of studies. Nielsen, Fredslund, diagnostic process (Kompier et al., 1998) and Christensen, and Albertsen (2006) found that in a provides information that can be used to check designated intervention group no changes in well- whether intervention activities addressed the being were detected because intervention activities problems perceived by organizational members. had not been implemented, whereas changes were Context-independent organizational-level strategies observed in a designated control group because the have been described as unlikely to succeed as each manager had initiated activities intended only for the organization is unique and therefore require unique intervention groups. Similarly, Landsbergis and solutions (Hurrell & Murphy, 1996). Thus, it is Vivona-Vaughan (1995) found that in an intervention important during the PE to examine whether department where no effects were found, a planned intervention activities were tailored to the problem ‘‘policy and procedures’’ manual had not been as it is manifested itself in the specific organizational completed and implemented. context. Tailoring of organizational-level interventions Implementation strategy does not usually include adapting interventions to meet the requirements of specific individual employ- In this section we focus on the roles and behaviours ees. This has been cited as a potential problem with of key stakeholders. Later in the section on mental organizational interventions: An optimal strategy models we focus on the appraisals and perceptions of may be to use a combination of different interven- key stakeholders and how these may drive key tions (LaMontagne et al., 2007). Individual-level stakeholders’ behaviours, thereby indirectly influen- activities implemented during the development of cing intervention outcomes. organizational-level interventions may prime partici- pants to support and engage in organizational-level Drivers of change and the roles of key stakeholders: changes when they are implemented (Nielsen, Ran- Who were/are the drivers of change? In complex dall, Brenner, & Albertsen, 2009; Nielsen, Randall, & interventions, there are often many stakeholders in Christensen, 2010). Developing structured action the intervention process and therefore many potential plans may also facilitate effective intervention: Such drivers of change. In PE these stakeholders must be plans describe intervention activities in terms of the identified and their role in the change process resources needed, the activities undertaken, and how explored. It is important to identify who has the the intervention is implemented, including identifying power to make changes in order to identify how PROCESS EVALUATION TOOL 605 much they were involved in intervention activities intervention projects (i.e., the resources needed to (Nytrø et al., 2000). Next we discuss in further detail plan, implemen, and evaluate the project), they may act the roles of key change agents. as role models through their attitudes to the intervention, and they may be actively involved in Participatory approaches—involving employees: Did intervention activities (Giga, Noblet, Faragher, & employees participate significantly in decision making Cooper, 2003; Lindstro¨ m, 1995; Randall, Cox, & and how many were involved? The participatory Griffiths., 2007). However, because of their seniority approach has been advocated as a desirable and own work demands they are rarely able to follow intervention strategy and plays a major role in well- closely intervention development activity and known organizational occupational health implemen-tation (Nytrø et al., 2000). Although the intervention approaches (Nielsen, Randall, Holten, & importance of senior management support is often Rial Gonza´ lez, 2010). Participation in the development discussed, it is seldom formally evaluated (Nielsen, of health promoting activities is also included in the Randall, Holten, & Rial Gonza´ lez, 2010; Semmer, guidelines of the World Health Organization and the 2011). What research there is suggests that suchsupport European Network for Workplace Health Promotion can be important. In a study of stress coping training, (European Network for Workplace Health Lindquist and Cooper (1999) found that when senior Promotion, 2007). The essence of participation is a management released staff from their duties to conscious and intended effort made by individuals at a participate in workshops, attendance was 100%. In higher level in an organization to provide visible contrast, at follow-up when staff had to participate extrarole or role-expanding opportunities and during their leisure time, participation dropped to 66%. enhanced control for individuals or groups at a lower Saksvik, Nytrø, Dahl-Jørgensen, and Mikkelsen (2002) level in the organization (e.g. to have a greater voice). reported decreased opportunities for staff to take part Participation can take different forms: it may be in participatory workshops were due in part to informal or delivered through formal changes in roles constraints imposed by senior management only and responsibilities, directly experienced or indirectly allowing employees time to participate in short through union representatives, and the breadth and workshops. In this example, the lack of support from depth of participation and the extent of influence senior management also had a ‘‘trickle down’’ effect on linked to the participatory activities can also vary middle managers, who reported they did not support (Lines, 2004). the intervention project as they were allocated no Several pieces of research discuss how both resources to implement initiatives. qualitative (type of) and quantitative (amount of) participation might have influenced intervention Middle managers: What was the role of middle outcomes. Nielsen et al. (2006) found that employees managers? While senior managers often make the with little formal education benefited most from a decision to implement the intervention it is usually directive type of participation where they were told middle managers that are subsequently responsible what to do. Aust, Rugulies, Finken, and Jensen for communicating and implementing change (Guth (2010) found that employees reacted negatively to & Macmillan, 1986). Therefore, middle managers only having influence over parts of the intervention play a crucial role in many organizational-level programme, i.e., limited influence over the scope of occupational health interventions (Nielsen & the problem. Concerning the amount of participa- Randall, 2009; Randall et al., 2007). Kompier, tion, Lines (2004) found that this was negatively Cooper, and Geurts (2000) found that in all 11 case related to resistance of change, and positively related studies they collected from across Europe middle to achievement of goals and organizational commit- managers were primarily responsible for stress ment. Similarly, Nielsen, Randall, and Albertsen prevention interventions. This puts middle (2007) found that high levels of reported participa- managers in a position to hinder or facilitate the tion in change were associated with low levels of change. For example, Dahl-Jørgensen and Saksvik behavioural stress symptoms and higher job satisfac- (2005) reported that middle managers resisted change tion after intervention. Eklo¨ f, Ingelga˚ rd, and Hag- by restricting the time spent on interventions by berg (2004) found that the degree of participation in employees. Similarly, Nielsen and Randall (2009) the resolution of occupational health concerns was found that where middle managers were perceived as consistently associated with decreased work de- supportive and took an active part in implementing mands, increased social support, and lowered stress change, employees reported better working levels. conditions and higher levels of psychological well- being after the intervention. Middle managers can Senior management support: What was the role of also bring about changes in intended intervention senior managers? Senior managers are often exposure patterns. For example, Nielsen et al. (2006) involved in the allocation of resources to found that a new manager who resented being in the 606 NIELSEN AND RANDALL control group initiated and implemented activities change and changes in work roles linked to the similar to those planned in the inter-vention groups intervention (Øyum, Kvernberg Andersen, Pettersen with positive outcomes. In a process evaluation of Buvik, Knutstad, & Skarholt, 2006). It has also been seven intervention projects, Saksvik et al. (2002) found that open communication helps employees to found that middle managers had often exerted understand the intentions behind organizational-level passive resistance that had damaged and diluted occupational health interventions, thus improving some intervention activities. Together these findings employee commitment to and participation in the indicate that middle managers’ motivation for intervention (Nytrø et al., 2000). Communication is implementing change should be documented along likely to influence employees’ sense making (e.g., their with the actions they take to facilitate or obstruct perception of the motives and objectives of the change. This documentation is especially important intervention) and this appears to be closely linked because such data may not be captured in their to their commitment to intervention activities (Weick, performance appraisals and in times of pressure they Sufcliffe, & Obstfeld, 2005). Therefore, it is important may therefore choose to prioritize other aspects of to examine what kind of information has been their job over and above intervention activities distributed, to whom, and how it has been received (Saksvik et al., 2002). and perceived. This means that three important questions need to Consultants: What was the role of be answered. (1) Were participants informed about consultants? Large intervention projects often use the project? Nielsen et al. (2007) found that receiving external consultants to design, implement and adequate information about an intervention project facilitate aspects of the intervention process predicted the extent to which employees participated (Nielsen, Randall, Holten, & Rial Gonza´ lez, 2010). in intervention activities. One important caveat to Lindstro¨ m (1995) reported that process consultants this finding was that where information was not facilitated organizational changes by giving feedback followed up by actual activities, employees were on the progress of change and on group dynamics. In disappointed and reported negative results. (2) Were another study the consultants played a role in the risk assessment results fed back? This feedback has proliferation of the intervention activities as they sold been found to lead to more intervention activities similar services to those implemented to other groups (Eklo¨ f, Hagberg, Toomingas, & Tornqvist, 2004). In in the organization (Nielsen et al., 2006). However, a later study, Eklo¨ f and Hagberg (2006) found the few intervention studies have included an evaluation most significant changes in social support were of the role or the competencies of the external observed in parts of the organization where super- consultant(s) (Semmer, 2006). In order to isolate visors, and to a lesser extent work groups, had intervention effectiveness it is important to evaluate received detailed information about the problems whether the consultants had the necessary skills and identified in the risk assessment. (3) To what extent abilities to enhance the intervention process by are all participants updated about progress? Land- motivating and guiding participants through the sbergis and Vivona-Vaughan (1995) found that those intervention process (Landsbergis & Vivona- employees not directly involved in intervention Vaughan, 1995). It may also be that when external planning and implementation tended to be less aware consultants have total responsibility for change they of the progress of the intervention: These employees leave no infrastructure within the organization for also reported that the intervention had little effect. sustaining and continuing improvements they initiated, thus reducing long-term intervention Context effects (Dahl-Jørgensen & Saksvik, 2005). Nielsen, Cox, and Griffiths (2002) argued that, for Field studies are conducted to enhance the ecological intervention effects to be maintained in the long validity of intervention research. However, this term, a shift must take place in which organizational validity can only be achieved if the influence of the members gradually take more responsibility for the social and organizational context on intervention intervention from the consultant. outcomes is measured and analysed (Heaney et al., 1993; Rousseau & Fried, 2001). Context can be Information and communication about the defined as ‘‘situational opportunities and constraints intervention: What kind of information was provided that affect the occurrence and meaning of organiza- to participants during the study? It has been shown tional behaviour as well as functional relationships that the level of information and communication between variables’’ (Johns, 2006, p. 386). The context plays an important role in the effects of interventions may either facilitate or hinder successful implementa- (Jimmieson, Terry, & Callan, 2004). Providing tion. Intervention context can provide a link between information about a change keeps employees up to intervention plans and intervention exposure (i.e., date about anticipated events, the consequences of work as a mediator) or can dilute or strengthen the PROCESS EVALUATION TOOL 607 effects of intervention activities (i.e., work as a and national level (e.g., economic recession; moderator). The overall question is to ask: ‘‘Which Landsbergis & Vivona-Vaughan, 1995; Mikkelsen & hindering and facilitating factors in the context Saksvik, 1999; Nielsen, Randall, & Christensen, influenced intervention outcomes?’’ Because the 2010) should be considered. context is diverse and multifaceted the concepts of omnibus and discrete context (Johns, 2006) provide a Mental models useful framework for PE. Recently, research has begun to examine how Omnibus context. This refers to the story told and individuals’ perceptions and appraisals of an prompts several process evaluation questions organizational-level occupational health interven- including: ‘‘Who are the participants in the inter- tion are linked to outcomes through how they vention and who drives the intervention?’’ ‘‘Where drive the behaviours of key stakeholders. Employ- does it take place?’’ ‘‘When did the intervention take ees, managers and other key stakeholders may have place?’’ The underlying theme of these questions is: diverse and potentially conflicting agendas that may ‘‘How did the intervention fit in with the culture and influence how they behave and react to the conditions of the intervention group?’’ Dahl- intervention. These underlying psychological pro- Jørgensen and Saksvik (2005) found that a context cesses may help to explain change outcomes (Nytrø in which there were high job demands often hindered et al., 2000) but have rarely been measured directly participation in interventions. Organizational culture in intervention research. The main question to ask may also play a role: Saksvik et al. (2002) found that a here is: ‘‘What is the role of participants’ mental bureaucratic organizational structure or being part of models in determining their response to the inter- a larger, international organization hindered the vention?’’ Mental models are used to make sense of development of inter -vention activities. the world and explicit efforts at sense making take Furthermore, it is important to ask: ‘‘What capacity place when the world is perceived to be different does the organization have to conduct interventions?’’ from the expected state of the world, e.g., when The preintervention healthiness of an organization changes at work are occurring (Weick et al., 2005). and its past use of and experience with such Translating this into an intervention context, interventions have been found to affect mental models determine how participants react to organizational occupational health intervention the intervention and its activities and help explain outcomes (Semmer, 2006). Workplaces with low the behaviours of key stakeholders throughout the demands, high levels of support, and low stress intervention project. levels may have more time and resources to involve For example, it has been found that different workers and managers in participation and stakeholders have conflicting mental models about integration of interventions. On the other hand, what constitutes success (Cole et al., 2003; Shannon healthy organizations with low levels of stress and a & Cole, 2004). Detecting these different perspectives good working environment may not need in- may help to explain how different motivations drive terventions (Taris et al., 2003). Indeed, ceiling differences in key stakeholders’ behaviours during the effects may prevent further improvement in intervention process. Saksvik et al. (2002) found that intervention outcomes, even if the theory behind managers preferred individual-level interventions, intervention is correct (Nielsen et al., 2006). This i.e., putting responsibility for change at the indivi- represents an intervention paradox whereby the dual. In contrast, employees held negative attitudes omnibus context can inhibit intervention where it is to these interventions because they felt that this needed most. strategy was a way for managers to escape respon- sibility thus failing to address the problems in the Discrete context. This aspect of context focuses workplace. on specific events that may have influenced the effects For interventions to be effective, it has been of the intervention. The question asked here is: argued that employees should perceive that they have ‘‘Which events took place during the intervention problems that need to be addressed, believe that the phase?’’ Some factors that have been identified here intervention will be effective in addressing those are new project management demands (Nielsen et al., problems, and be motivated to actively support the 2006), conflicting priorities, concurrent use of intervention by participating in intervention activities multiple change programmes (Guastello, 1993), and (Nytrø et al., 2000). This implies that an important lack of integration of the intervention with important PE question is: ‘‘To what extent are participants corporate strategic decision-making activities ready for change?’’ Readiness for change has been (Schurman & Israel, 1995). Factors at both the widely researched, but rarely within the organiza- intraorganizational (e.g., introducing conflicting tional-level occupational health intervention litera- initiatives; Nielsen et al., 2006; Randall et al., 2007) ture. There is an abundance of literature that has 608 NIELSEN AND RANDALL linked the degree to which employees welcome and they felt there was no need for risk assessment since actively support the implementation of change to their perception was that stress was not a problem organizational development and change outcomes (Biron, Gatrell, & Cooper, 2010). In a study of (Weiner, Amick, & Lee, 2008). A number of studies organizational improvement programme and an have discussed the importance of mental models of individual-level stress management programme, readiness for change in organizational-level occupa- Bunce and West (1996) found that the perceived tional health intervention research. Randall, Grif- smoothness of implementation of the training pro- fiths, and Cox (2005) found that a change in gramme and the depth of programme content were responsibility had not been communicated to staff related to lower levels of stress and higher levels of because managers felt the intervention would have a job satisfaction after the programme. Nielsen et al. detrimental outcome on their own working condi- (2007) found that individuals’ appraisal of the quality tions: They were not ready for the change because and sustainability of intervention activities were their appraisal was that the intervention would positively linked to postintervention well-being. In damage their own working conditions. Another participatory action research projects where employ- mental model concerning readiness of change is ee representatives are involved in making change, it is initiative fatigue: If organizational-level occupational particularly important to examine the mental models health interventions have previously been conducted of those not directly involved in decision making but but little learning has taken place, this failure may who were targeted by the intervention (Landsbergis & have a detrimental impact on participants’ percep- Vivona-Vaughan, 1995). In some instances, percep- tions of later initiatives and their willingness to tions of key stakeholders’ expertise may also be participate in intervention activities (Nytrø et al., influential. Nielsen et al. (2006) reported that employ- 2000; Saksvik et al., 2002). ees with little education and little experience with Over time, people who work closely together may dealing with occupational health issues appraised an develop similar models to interpret and react to the external occupational health practitioner more posi- world (Mathieu, Heffner, Goodwin, Salas, & Can- tively because of her directive approach and a focus non-Bowers, 2000). In an intervention context on individual issues. participants with shared mental models may perceive the intervention and its activities in a similar manner Changes in mental models of the job. In order for and as a result react in similar ways to the introduced real changes to happen as a result of organizational- intervention. Where shared mental models have not level occupational health interventions, it has been been developed individuals may have conflicting pointed out that participants and key stakeholders agendas that may hinder effective implementation must unlearn old mental maps of their working and lead to diversity in intervention experiences and conditions and learn new ones (Schurman & Israel, outcomes. Recent research has shown that a lack of 1995). An important driver of this change is the support for an intervention programme was the result degree to which intervention activities prompt a shift of differences in stakeholder views about the most from espoused theories about the intervention to effective intervention option: Consultants felt the theories-in-use (Argyris, 1976, 1995). Theories-in-use focus of the intervention programme should be are the mental models that guide our behaviour, leadership development, whereas employees thought whereas espoused theories are the attitudes and it should have focused on employee involvement beliefs that we tell others guide our behaviour. (Aust et al., 2010).Therefore, process evaluation According to Argyris (1976, 1995) real change only should be used to examine the mental models and happens when individuals change their theories-in- the degree to which mental models are shared by use. Therefore, an important part of process participants using the question: ‘‘To what degree do evaluation should be the measurement of change in participants have shared mental models?’’ employees’ knowledge of the intervention, their An equally important mental model to explore is: expectations that the intervention can bring about ‘‘How did participants perceive the intervention and changes, and that these changes can have an impact its activities?’’ Employees’ perceptions of the drivers and be sustained as part of continuous improvements of change and the intervention objectives are likely to at work (Schurman & Israel, 1995). The important influence their willingness to participate in interven- question here is: ‘‘Did the intervention bring about a tion activities. If they believe activities do not address change in participants’ mental models?’’ In a the problems raised or are of a poor quality this may randomized, controlled study, Nielsen, Randall, and reduce their engagement. Christensen (2010) found that team manager training A study on the implementation of a Stress Risk that produced changes in mental models was need to Assessment tool revealed that middle managers (who bring about led to changes in managers’ behaviours were responsible for using the tool) failed to use the and, as a consequence, their subordinates’ tool after they attended a training course because involvement and job satisfaction. PROCESS EVALUATION TOOL 609 stakeholders had a hidden agenda such as employees PE methods wanting to get rid of an unpopular manager or In the previous sections, we have described a set of unpopular colleagues, managers wanting to be seen topics which we argue should be considered when to be ‘‘doing something’’, or internal consultants determining the validity and generalizability of creating work for themselves, justifying their organizational-level occupational health interven- existence). tions. This leads to an important question: ‘‘How do we get this information?’’ We argue that process Intervention activities. The risk assessment evaluation calls for a mixed methods approach. method itself and feedback reports provide Detecting the different active ingredients of organiza- information on the ‘‘objective’’ part of risk tional-level occupational health interventions requires assessment. This element of PE should include the the use of a range of different methods. The measure- collection of stakeholders’ experiences of the ment of some active ingredients requires the collection feedback of risk assessment results. Through of observer or objective data, e.g., what can be seen to questionnaires at follow-up, employees can be asked be happening. Other PE constructs are appraisals (e.g., whether they are aware of the results of risk how participants appraised the intervention influences assessment, have participated in meetings feeding intervention outcomes) that cannot be directly ob- back results, or whether the results have been served. In addition, the wide range of PE constructs discussed with managers and colleagues. It may also and the importance of context indicate that a flexible be examined whether participants felt the risk approach to data collection and analysis offered by assessment tool examined the appropriate topics. qualitative methods is likely to have utility. Combining Appraisals of the risk assessment may be collected both qualitative and quantitative results through through obtaining minutes from feedback meetings, mixed methods may offer four important benefits: direct observation of feedback activities, and Identification of the mechanisms behind any changes inspection of feedback documents. Such data can brought about by the intervention; added meaning to help determine how the risk assessment method and the results of outcome evaluation; cross-validated and content was determined upon, and which strategies triangulated results; and identification of the impact of were employed to feed results back to employees. the intervention context on the processes and out- Interviews provide an important source of data to comes of change (Greene, Benjamin, & Goodyear, gain information about the impact of risk 2007; Hugentobler, Israel, & Schurman, 1992; John- assessments: Have the results been used and how? son, Onwuegbuzie, & Turner, 2007; Nastasi et al., Independent reviews of documented actions plans 2007). Unfortunately, research on interventions rarely provide a method of obtaining information about the includes interview schedules, observation strategies, or content and objectives of action plans. It is possible to detailed information about how these data were detect the level of detail, e.g., do action plans include collected and used. It is therefore difficult to refer to vague objectives such as: ‘‘We want to be better at specific examples that provide guidance on how this communicating’’ or do they include detailed concrete may be done. Therefore, in the following sections we descriptions of the objective and associated activities, describe the data that may be collected and provide deadlines, required resources needed, and allocate preliminary estimates of the suitability of different responsibilities for key tasks? Comparisons can also be data collection methods (see Table 1). made between the results of risk assessment and how these fed into action plans. Cognitive mapping techniques may be useful in making the connection between problems present in the organization and Documentation and appraisal of the planned intervention activities (Harris, Daniels, & intervention design and implementation Briner, 2002): This will help to determine if there is a Initiation. In the first step it is important to clear link between the results of the risk assessment document the precursors of intervention activities. and the activities outlined in actions plans. This provides some documentary evidence of the Through observations of action planning meetings process. Such information can be obtained through and interviews with those involved, it is possible to sources such as meeting minutes and other explore the translation of risk assessment results into organizational material that records intervention- action plans. These data collection methods should related events. These may reveal who took the address questions about how and why activities were initiative to initiate a project, how the intervention prioritized and by whom. It is also possible to objectives were formulated, and who participated in document the degree to which actions plans were making the decision to design the project. Interviews implemented: This may be obtained through ques- may be better suited to the exploration of mental tionnaire data collected from various stakeholders. models (e.g., it can be revealed whether any Through listing the activities planned in question- 610 NIELSEN AND RANDALL TABLE 1 Process evaluation checklist Questions Necessary data ‘‘objective’’ Additional data ‘‘appraisal’’ The intervention Who initiated the intervention and for what purpose? Organizational data: Interviews: Who took the initiative? Detection of hidden agendas What was the official objective? Did the intervention activities target the problems of the workplace? Comparison of risk assessment to action Interviews: . What means of risk assessment was used? plans Appropriateness of intervention tools including . What was the implementation strategy? Documentation of activities developed workshop tools and questionnaires . How many changes were planned? Review of workshop material Questionnaires, Appropriateness of intervention . Was the focus on one large change or rather many small steps to make activities a change? . Were actions plans sufficiently detailed? . To which extent were activities tailored to the organization? . To which extent did the activities target multiple levels? . How was risk assessment translated into activities? Did the intervention reach the target group? Questionnaires, meeting minutes Interviews: . Why were intervention activities not implemented? Actual implementation Degree of intervention activity implementation . Which aspects of the activities brought about changes? . How much were delivered to whom? . Who noticed changes? . Were positive and negative ‘‘side effects’’ monitored? . Was a ‘‘plan B’’ developed to address any lack of progress? Who were the drivers of change? Questionnaires: Interviews: . Did the roles and commitment of key stakeholders change over time? The extent of involvement Identification of drivers of change throughout all . Did employees participate in real decision making and how many were phases of the project involved? Key stakeholders role of involvement . Did employees assume responsibility for the project and the completion of activities? . What was the role of formal representatives? . What was the role of senior managers? Were the necessary resources allocated? Did they support the project throughout – and how was support manifested? . What was the role of middle managers? Did they support the project throughout – and how was support manifested? Did they function as the link between senior management and employees? Did they encourage active participation by employees? . What was the role of consultants? Did they create a supportive, trusting atmosphere? Did they use tools to facilitate the process, e.g. visualizing progress? And how did they work? Did the organizational consultant enable organizational ownership? (continued overleaf ) PROCESS EVALUATION TOOL 611 TABLE 1 (Continued ) Questions Necessary data ‘‘objective’’ Additional data ‘‘appraisal’’ What kind of information was provided to participants during the study? Organizational data: Interviews: Were participants informed about the project? Review of meeting minutes, memos, Appropriateness and quality of information . Were risk assessment results fed back? correspondence, posters, leaflets . To which extent are all participants updated about progress? . Were small successes celebrated? The context Which hindering and facilitating factors in the context influenced intervention Organizational data, news media Interviews: outcomes? What happened in the discrete context? Identification of hindering and facilitating factors in . Why were intervention activities not implemented? the omnibus and discrete context . How did the intervention fit with the culture and conditions of the Degree of intervention activity implementation intervention group?(omnibus context) . What capacity did the organization have to conduct interventions? (omnibus context) . Which events took place during the intervention phase?(discrete context) . Did a change in management take place? . Did organizational restructuring take place during the intervention phase? . Did changes outside the organization take place during the intervention phase that may have influenced intervention outcomes, e.g., changes in legislation, recession etc.? Participants’ mental models What is the role of participants’ mental models? Questionnaires: Interviews and observations: . To which extent are participants ready for change? Degree of intervention openness to change Changes in employees’ perceptions of themselves, . To which degree do participants have shared mental models? Quality, quantity, and appropriateness of their colleagues and their workplace In case of resistance, what were the threats appraised by key intervention activities post-intervention stakeholders? Degree of intervention activity implementation In case of divergence, how did mental models differ? . How did participants perceive the intervention and its activities? . Did the intervention bring about a change in participants’ mental models? . Why were intervention activities not implemented? *The question ‘‘Why were intervention activities not implemented?’’ can be asked at all three levels, but the answers are different for each level. At the intervention level, the answer will reside in the actions of key stakeholders, whereas at the context and mental models level, we may obtain information about the reasons behind these behaviours. 612 NIELSEN AND RANDALL naires, and asking respondents to indicate which they memos, and correspondence provide ‘‘objective data’’ have participated in, or which procedures have been on how the intervention was communicated. How- changed, it is possible to document the reach of ever, this organizational data should be supplemen- activities (see Nielsen et al., 2006). Through inter- ted by interviews and questionnaires to ensure that views we may gather data on why activities may not this information actually reached participants and have been implemented: Questions can be asked how it was perceived by recipients (either by interview about whether the necessary resources were available or questionnaire methods). As has already been and if not why participants perceived this to be the discussed the intentions driving the intervention case (e.g., was it because senior managers were may not match the way these are perceived by its reluctant to allocate resources or did no-one assume recipients. A question for intervention recipients responsibility for implementing actions?). should therefore be: Was information sufficient and The drivers of change may be identified through was it easy to understand and relate to daily work? If interviews, meeting minutes, and questionnaires. Data middle managers were given leaflets did they dis- may be collected by asking employees about the extent tribute them? Did participants read emails about the to which they were involved in developing and project, updates in personnel magazines, or informa- implementing change (see Randall, Nielsen, & Tvedt, tion sent to them? It is also important to document 2009). A recent criticism of action research approaches the reach of information about different parts of the is that often only few employees are involved through intervention process. Are employees informed about steering groups or health circles (Nielsen, Randall, the initiation of the project but then hear nothing Holten, & Rial Gonza´ lez, 2010). It has been argued about the results of risk assessment, the developed that only through involving all employees the advan- action plans, and the implemented changes? If this is tages of participation can be achieved (Hurrell, 2005): the case they are likely to develop a perception that Therefore, information about participation should be the project was unsuccessful. Even if improvements collected from as many of the intended beneficiaries of are being implemented they fail to see the link the intervention as possible. between the intervention project and improvements if While interviews provide rich information about these are not clearly communicated. how employees were involved and the perceptions of such involvement these do not provide information Documenting the importance of context about how many had the opportunity to influence the intervention process. A questionnaire has now been Qualitative methods may be best suited to unpredict- developed which can be used to examine the degree to able, complex, and difficult settings as a means of which all employees were involved in the intervention mapping critical events that may influence outcomes process (Randall et al., 2009). Information about (Yin, 1994). Qualitative studies may be suitable for senior and middle management support can also be examining contextual levels that may affect the obtained through interviews and questionnaires in a intervention outcomes (the employment of a man- similar manner (e.g., Nielsen & Randall, 2009). ager that supports change) and explore the full range Through observing activities that are designed to of behaviours and attitudes that the context might enhance participation it is possible to collect data on, affect thereby working backwards to make inferences for example, how many employees were invited to about the situation. Data may be obtained from attend the activity, how many did actually participate, meeting notes, interviews, and organizational mate- how many participants were active during the meet- rial. Data may also be collected on any concurrent ings, and what role middle managers, employees, events such as changes in management and/or senior managers, and, if applicable, consultants took ongoing restructuring. during the activities. The following questions could be Through news media, information may be ob- addressed using these data: Did all organization tained about the national context, e.g., changes in members make suggestions for planning and imple- legislation and media attention (e.g., in some mentation the intervention and its activities? How was countries the teaching profession receives negative the general climate between different key stake- media attention on a regular basis). This material, holders—did employees have an actual say or were however, should always be analysed in terms of how decisions made by the management levels? Did any it relates to the intervention itself. Further informa- stakeholders show resistance? How were the meetings tion about the impact of such factors may be run, e.g., were tools used to facilitate discussions? obtained through interviews, meeting minutes, and other organizational material. Also at this level, the question ‘‘Why were Communication and information activities not implemented?’’ could be asked but with Organizational material and archive data such as additional probe questions about answers that point posters, meeting notes, mail communication, leaflets, to the impact of context: It may be that employees PROCESS EVALUATION TOOL 613 with substantive contact with clients found it hard to leaders became aware of exerting transformational prioritize participating in activities that took away leadership and employees started taking over time from clients or that concurrent restructuring responsibilities such as rota planning and inde- took up all the time and energy of managers and pendent problem solving. These data can be collected employees. Such information may be best obtained by asking employees whether there have been any through interviews. changes in behaviours and work procedures during the intervention project and whether these are seen to be ascribed to the intervention. Evaluating the impact of mental models Some aspects of mental models such as readiness for Timings of data collection change can be explored through questionnaires (Weiner et al., 2008). However, changes in multi- It has been argued that process evaluation questions faceted and complex mental models and the diversity should be included at follow-up (Randall et al., 2009), of mental models may be more difficult to detect but it is also important to gather data throughout the using quantitative methods. In order to understand process. Ongoing data collection is important for two the impact of an intervention we must also gain an reasons. First, collecting data on the process after the understanding of how the intervention was perceived intervention project may result on retrospective by recipients. Through interviews and reviews of sensemaking (Weick et al., 2005). Participants may meeting minutes we can obtain information about the seek explanations of why or why not the intervention extent to which participants felt intervention activ- programme had a desired effect, e.g., that middle ities addressed the issues raised in the risk assessment. managers did not fulfil their role as change agents or The question ‘‘Why were intervention activities concurrent changes created conflicts. Second, ele- not implemented?’’ may also be answered at this ments of the process may change over time. For level. The answer to this question resides within how example, a project may be initiated for one purpose, participants perceived the intervention activities. This e.g., the legal requirement to conduct risk assess- involves asking questions such as: Were key stake- ments but may be sustained because participants see holders not supportive—and if so what were their the added value of the project. The role of key mental models of why the intervention presented a stakeholders may also change over time. For exam- threat? Were activities not perceived to be appro- ple, senior managers may be supportive of the project priate? How was the occupational health consultant at the beginning, but focus their attention on new received—did he or she show an understanding of the projects over time or lose interest if the project does culture and problems of the organization? All these not bring about changes at the speed expected. questions may help explain the actions of key Participation may also change over time. One of the stakeholders, e.g., a middle manager who initially core elements of participatory processes is empower- supported the project stopped doing so, once s/he ment: Over time employees develop the skills and realized that it may threaten her/his status within the start feeling responsible for creating a good working organization. Interviews may be particularly impor- environment. It is important to document any such tant to reveal the mental models of participants in changes, e.g., do they emerge during risk assessment, terms of how they evaluate the appropriateness and during action planning, or during implementation of quality of interventions. However, to get a broader activities? but less detailed picture from the wider population, Concurrent data collection throughout the project questionnaires may be used (cf. Nielsen et al., 2007, is time consuming for researchers and participants 2009). Focus groups may be a particular powerful alike. This means that measures need to be developed method for revealing the extent to which shared that are both easy to use and analyse. Although mental models have developed (Mohammed, Ferzan- observation of meetings and of work and interviews di, & Hamilton, 2010). are time consuming, reviewing meeting minutes and developing short questionnaires may be less so. The Changes in mental models of the job. To explore scales by Randall et al. (2009) are fairly lengthy; double-loop learning questionnaires can be used to however, it is also possible to construct short explore the degree to which different procedures have questionnaires and interviews schedules (e.g., using been introduced to ensure that decisions and telephone interviews) that can be used to collect data behaviours are not taken for granted but openly from key stakeholders (e.g., the union representative explored (Randall et al., 2009). In addition obser- and middle managers) on a monthly basis. Such data vations and interventions may help reveal changes in collection need not take much more than a few procedures, e.g., Nielsen, Randall, and Christensen minutes. Another possibility is to develop a short list (2010) found that leaders and employees changed of questions that managers and employees discuss at behaviours after a teamwork implementation in that regular meetings in the organization. Such a strategy 614 NIELSEN AND RANDALL also ensures that the project stays on the agenda in changes in subordinates’ working conditions and well- the organization and that organization members being. A limitation of all of these studies is that they all reflect on progress. Collecting short measures may focus on isolated specific elements of the process and not only be an advantage to the researchers, it may do not contain comprehensive process evaluation also be used as formative evaluation for the strategies like the one described in this article. organization as it provides the opportunity to reveal Some examples of more comprehensive ap- gaps in the current implementation process and take proaches to analysing process data have been corrective action (Patton, 2002). Using the Experi- reported. Murta et al. (2007) listed in their review a ence Sampling Method offers the opportunity to number of factors that may serve as an analytic measure daily experiences (Hektner, Schmidt, & framework. Analyses could be conducted where Csikszentmihalyi, 2007) and to examine how experi- recruitment, context, reach (attendance rates), dose ences vary on a daily basis, e.g., levels of participation delivered (intervention content), dose received (the and middle managers’ actions. In this way PE may extent to which participants apply activities), parti- provide managers with dashboard data that can be cipants’ attitudes, fidelity (whether activities were used to inform decisions about changes to the implemented as planned), and the link between intervention process. process and outcome should be analysed in a step- wise manner to detect at which level effects can be identified. Randall et al. (2007) provided a template Analysing process evaluation data for analysing process data. They suggested analysing Research including systematic process evaluation is processes and outcomes of intervention at three still in its infancy; however, there are a few studies levels: (1) microprocesses that concern the interven- which can be drawn upon when seeking inspiration on tion implementation: magnitude, valance, effect on how to analyse process evaluation data. It may be working conditions of participants and others; (2) considered optimal to integrate process evaluation macroprocesses describing the design, delivery, and data into effect evaluation to fully understand how maintenance of interventions; and (3) intervention processes have influenced intervention outcomes. To context analysing the importance of the context at test the effects of actual exposure, Randall et al. (2005) different levels from the context of the intervention, divided intervention participants into two groups: the department, the organization, the demands of the those who had been exposed to the intervention and sector, and the national level. those who had not. They found clear differences in exhaustion levels, concluding that those whom the CONCLUDING REMARKS intervention had reached reported better health postintervention. More sophisticated analyses may In this article we presented a framework for how the be conducted of the patterns of interaction between processes of interventions may be evaluated. It does process and outcomes. Using Structural Equation not present a complete model to cover all interven- Modelling, Nielsen et al. (2007) combined ‘‘objective’’ tions; rather, it is intended to provide a guiding process data (information received about the inter- framework from which elements may be selected for vention programme, and participation in intervention inclusion in evaluations of organizational-level occu- activities) and ‘‘appraisal’’ process data (quality of pational health interventions. There is a conflicting intervention activities) and linked these to intervention demand on evaluation research: It should be cost outcomes. Another study by Nielsen and Randall effective, ecologically valid, and practically impor- (2009) linked preexisting working conditions (as a tant, while at the same time be of scientific proxy for organizational maturity) to the role of the importance and generalizable (Bussing & Glaser, middle manager during the intervention and found 1999). We believe a model that prompts careful this role predicted intervention outcomes. There may consideration of the information that is necessary to be instances where process data are not easily evaluate the effectiveness of the intervention and integrated into effect analysis, e.g., when describing under which circumstances the results may be the impact of context. Aust et al. (2010), Biron et al. transferable to other contexts will help to develop (2010), and Nielsen et al. (2006) all present examples of our understanding of organizational-level occupa- how quantitative effect data and qualitative process tional health interventions. data may be combined in such circumstances. In a It may be worthwhile changing our opinion of study of middle manager training, Nielsen, Randall, what represents success in organizational-level occu- and Christensen (2010) outlined four levels at which pational health interventions. In line with Lipsey quantitative and qualitative data were integrated in (1996), we propose three indicators for success: (1) order to detect effects at four levels (satisfaction with changes in working conditions, employee health, and the intervention, changes in values and attitudes, well-being; (2) intermediate indicators that focus on changes in middle managers’ own behaviours, and the intervention activities implemented, whether PROCESS EVALUATION TOOL 615 Aust, B., Rugulies, R., Finken, A., & Jensen, C. (2010). When participants were aware of the intervention and its workplace interventions lead to negative effects: Learning from activities, whether participants held positive attitudes failures. Scandinavian Journal of Public Health, 38, 106–119. towards the intervention, whether participants devel- Bambra, C., Egan, M., Thomas, S., Petticrew, M., & Whitehead, oped new skills, and whether the intervention M. (2007). The psychosocial and health effects of workplace changed participants’ behaviour; and, finally, (3) reorganisation. 2. A systemic review of task restructuring interventions. Journal of Epidemiology and Community Health, indirect indicators that focus on whether the organi- 61, 1028–1037. zation changes their strategies and tools to deal with Biron, C., Gatrell, K., & Cooper, C. (2010) Autopsy of a failure: occupational health issues. Evaluating process and contextual issues in an organizational- PE should be carried out in both intervention and level work stress intervention. International Journal of Stress control groups to get an understanding of changes in Management, 17, 135–158. Bond, F. W., Flaxman, P. E., & Bunce, D. (2008). The influence of both. It is often the case that control groups and psychological flexibility on work redesign: Mediated modera- intervention groups interact with each other and this tion of a work reorganization intervention. Journal of Applied should be considered in PE (Nielsen et al., 2006). Psychology, 93, 645–654. Problems during the process may be lack of perfect Briner, R., & Reynolds, S. (1999). The costs, benefits, and blinding, and crossover between groups, and it is limitations of organizational level stress interventions. Journal of Organizational Behavior, 20, 647–664. important to understand any interaction between the Bunce, D., & West, M. (1996). Stress management and innovation two (Hurrell & Murphy, 1996). interventions. Human Relations, 49, 209–232. The focus is on process evaluation in this article, Burke, W. W., & Litwin, G. H. (1992). A causal model of but we do not mean to say that effect evaluation is organizational performance and change. Journal of Manage- not important; rather, these should supplement each ment, 18, 523–545. Bussing, A., & Glaser, J. (1999). Work stressors in nursing in the other (Nielsen, Taris, & Cox, 2010). Others have course of redesign: Implications for burnout and interactional discussed what should be assessed in effect evalua- stress. European Journal of Work and Organizational Psychol- tion, appropriate follow-up times and the amount of ogy, 8, 401–426. follow-ups necessary (Kompier et al., 1998, 2000; Cole, D. C., Wells, R. P., Frazer, M. B., Kerr, M. S., Neumann, W. Mikkelsen, 2005; Pettigrew, 1990; Taris & Kompier, P., Laing, A. C., & The Ergonomic Intervention Evaluation Research Group. (2003). Methodological issues in evaluating 2003): It is beyond the scope of this article to discuss workplace interventions to reduce work-related musculoskeletal these. disorders through mechanical exposure reduction. Scandinavian Although we discussed which methods may be Journal of Work and Environmental Health, 29, 396–405. used to collect data at the three levels, there is yet a Cooper, C., Dewe, P., & O’Driscoll, M. (2001). Organizational long way to determine which methods may best be interventions. In C. Cooper, P. Dewe, & M. O’Driscoll (Eds.), Organizational stress: A review and critique of theory, research, used to collect information at certain levels. One of the and applications. (pp. 187–232). Thousand Oaks, CA: Sage. problems mentioned in the introduction is the lack of Cummings, T. G., & Worley, C. G. (2009). Organization integration between process evaluation and effect development and change. Mason, OH: Cengage Learning. evaluation. Few studies have integrated quantitative Dahl-Jørgensen, C., & Saksvik, P. Ø. (2005). The impact of two process evaluation and effect evaluation (e.g., Bond, organizational interventions on the health of service sector workers. International Journal of Health Services, 35, 529–549. Flaxman, & Bunce, 2008; Nielsen et al., 2007, 2009) Egan, M., Bambra, C., Petticrew, M., & Whitehead, M. (2009). but this may be a viable way forward. Finally, we Reviewing evidence on complex social interventions: Apprais- would like to emphasize that we are not against quasi- ing implementation in systemic reviews of the health effects of experimental designs; rather, than these cannot stand organisational-level workplace interventions. Journal of Epide- alone in a simple effect evaluation. Quasi-experimental miology and Community Health, 63, 4–11. Egan, M., Bambra, C., Thomas, S., Petticrew, M., Whitehead, M., designs may control for biases that process evaluation & Thomson, H. (2007). The psychosocial and health effects of cannot do on their own; instead, we should try to workplace reorganisation. 2. A systematic review of task of integrate process evaluation measures in the strongest organisational-level interventions that aim to increase employee designs possible. control. Journal of Epidemiology and Community Health, 61, 945–954. Eklo¨ f, M., & Hagberg, M. (2006). Are simple feedback interven- tions involving workplace data associated with better working REFERENCES environment and health? A cluster randomized controlled study Argyris, C. (1976). Theories of action that inhibit individual among Swedish VDU workers. Applied Ergonomics, 32, 201–210. learning. The American Psychologist, 31, 638–654. Eklo¨ f, M., Hagberg, M., Toomingas, A., & Tornqvist, E. W. Argyris, C. (1995). Action science and organizational learning. (2004). Feedback of workplace data to individual workers, Journal of Managerial Psychology, 10, 20–26. work groups and supervisors as a way to stimulate working Armenakis, A., & Bedeian, A. (1999). Organizational change: A environment activity: A cluster randomized controlled study. review of theory and research in the 1990s. Journal of International Archives of Occupational and Environmental Management, 25, 293–315. Health, 77, 505–514. Armstrong, R., Waters, E., Moore, L., Riggs, E., Cuervo, L. G., Eklo¨ f, M., Ingelga˚ rd, A., & Hagberg, M. (2004). Is participative Lumbiganon, P., et al. (2008). Improving the reporting of ergonomics associated with better working environment and public health intervention research: Advancing TREND and health? A study among Swedish white-collar VDU users. CONSORT. Journal of Public Health, 19, 1–7. International Journal of Industrial Ergonomics, 34, 355–366. 616 NIELSEN AND RANDALL EU-OSHA (European Agency for Safety and Health at Work). LaMontagne, A. D., Keegel, T., Louie, A. M., Ostry, A., & (2010). European Survey of Enterprises on New and Emerging Landsbergis, P. A. (2007). A systematic review of the job- Risks, 2010. Retrieved from www.esener.eu stress intervention evaluation literature, 1990–2005. International European Network for Workplace Health Promotion. (2007). Journal of Occupational and Environmental Medicine, 13,268– Luxembourg Declaration on Workplace Health Promotion in the 280. European Union. Retrieved from http://www.enwhp.org/file Landsbergis, P., & Vivona-Vaughan, E. (1995). Evaluation of an admin/rs-dokumente/dateien/Luxembourg_Declaration.pdf occupational stress intervention in a public agency. Journal of Fredslund, H., & Strandgaard, J. (2005). Methods for process Organizational Behavior, 16, 29–48. evaluation of work environment interventions. In J. Houdmont Lindquist, T. L., & Cooper, C. L. (1999). Using life style and & S. McIntyre (Eds.), Occupational health psychology: Key coping to reduce job stress and improve health in ‘‘at risk’’ papers of the European Academy of Occupational Health office workers. Stress Medicine, 143–152. Psychology (pp. 109–117). Oporto, Portugal: ISMAI Publishers. Lindstro¨ m, K. (1995). Finnish research in organizational develop- Giga, S., Noblet, A. L., Faragher, B., & Cooper, C. L. (2003). The ment and job redesign. In L. Murphy, J. Hurrell, S. Sauter, & UK perspective: A review of research on organisational stress G. Keita (Eds.), Job stress interventions (pp. 283–293). management interventions. The Australian Psychologist, 38, Washington, DC: American Psychological Association. 156–164. Lines, R. (2004). Influence of participation in strategy change: Goldenhar, L., LaMontagne, A., Katz, T., Heaney, C., & resistance, organizational commitment and change goal Landsbergis, P. (2001). The intervention research process in achievement. Journal of Change Management, 4, 193–215. occupational safety and health: An overview from the National Lipsey, M. (1996). Key issues in intervention research: A program Occupational Research Agenda Intervention Effectiveness evaluation perspective. American Journal of Industrial Medicine, Research Team. Journal of Occupational and Environmental 29, 298–302. Medicine, 43, 616–622. Lipsey, M., & Cordray, D. (2000). Evaluation methods for social Greene, J. C., Benjamin, L., & Goodyear, L. (2007). The merits of intervention. Annual Review of Psychology, 51, 345–375. mixed methods in evaluation. Evaluation, 7, 25–44. Mathieu, J. E., Heffner, T. S., Goodwin, G. F., Salas, E., & Guastello, S. (1993). Do we really know how well our occupational Cannon-Bowers, J. A. (2000). The influence of shared mental accident prevention programs work? Safety Science, 16, 445–463. models on team process and performance. Journal of Applied Guth, W. D., & Macmillan, I. C. (1986). Strategy implementation Psychology, 85, 273–283. versus middle manager self-interest. Strategic Management Mikkelsen, A. (2005). Methodological challenges in the study of Journal, 7, 313–327. organizational interventions in flexible organizations. In A. M. Harachi, T., Abbott, R., Catalan, R., Haggerty, K., & Fleming, C. Fuglseth & I. A. Kleppe (Eds.), Anthology for Kjell Grønhaug in (1999). Opening the black box: Using process evaluation celebration of his 70th birthday (pp. 150–178). Bergen, Norway: measures to assess implementation and theory building. Fagbokforlaget. American Journal of Community Psychology, 27, 711–731. Mikkelsen, A., & Saksvik, P. Ø. (1999). Impact of a participatory Harris, C., Daniels, K., & Briner, R.B. (2002).Using cognitive organizational intervention on job characteristics and job mapping for psychosocial risk assessment. Risk Management: stress. International Journal of Health Services, 29, 871–893. An International Journal, 4, 7–21. Mohammed, S., Ferzandi, L., & Hamilton, K. (2010). Metaphor Heaney, C., Israel, B., Schurman, S., Baker, E., House, J., & no more: A 15-year review of the team mental model construct. Hugentobler, M. (1993). Industrial relations, worksite stress Journal of Management, 36, 876–910. reduction, and employee well-being: A participatory action Murta, S. G., Sanderson, K., & Oldenburg, B. (2007). Process research investigation. Journal of Organizational Behavior, 14, evaluation in occupational stress management programs: A 495–510. systematic review. American Journal of Health Promotion, 21, Hektner, J., Schmidt, J. A., & Csikszentmihalyi, M. (2007). 248–254. Experience sampling method: Measuring the quality of everyday Nastasi, B. K., Hitchcock, J., Sarkar, S., Burkholder, G., Varjas, life. Thousand Oaks, CA: Sage. K., & Jayasena, A. (2007). Mixed methods in intervention Hugentobler, M., Israel, B., & Schurman, S. (1992). An action research: Theory to adaptation. Journal of Mixed Methods research approach to workplace health: Integrating methods. Research, 1, 164–182. Health Education Quarterly, 19, 55–76. Nielsen, K., Cox, T., & Griffiths, A. (2002). Developing interven- Hurrell, J. (2005). Organizational stress intervention. In J. Barling, tions for risk management: The Action Innovation Process. In E. K. Kelloway, & M. R. Frone (Eds.), Handbook of work stress C. Weikert, E. Torkelson, & J. Pryce (Eds.), Occupational health (pp. 623–645). Thousand Oaks, CA: Sage. psychology: Empowerment, participation and health at work (pp. Hurrell, J., & Murphy, L. (1996). Occupational stress interventions. 135–140). Vienna, Austria: Allgemeine Unvallsversicherung- American Journal of Industrial Medicine, 29, 338–341. sanstalt (AUVA). Jimmieson, N. L., Terry, D. J., & Callan, V. J. (2004). A Nielsen, K., Fredslund, H., Christensen, K. B., & Albertsen, K. longitudinal study of employee adaptation to organizational (2006). Success or failure? Interpreting and understanding the change: The role of change-related information and change- impact of interventions in four similar worksites. Work and related self-efficacy. Journal of Occupational Health Psychology, Stress, 20, 272–287. 9, 11–27. Nielsen, K., & Randall, R. (2009). Managers’ active support when Johns, G. (2006). The essential impact of context on organizational implementing teams: The impact on employee well-being. behavior. Academy of Management Review, 31, 386–408. Applied Psychology: Health and Well-being, 1, 374–390. Johnson, R. B., Onwuegbuzie, A. J., & Turner, L. A. (2007). Nielsen, K., Randall, R., & Albertsen, K. (2007). Participants’ Toward a definition of mixed methods research. Journal of appraisals of process issues and the effects of stress manage- Mixed Methods Research, 1, 112–133. ment interventions. Journal of Organizational Behavior, 28, Kompier, M., Cooper, C., & Geurts, S. (2000). A multiple case 793–810. study approach to work stress prevention in Europe. European Nielsen, K., Randall, R., Brenner, S.-O., & Albertsen, K. (2009). Journal of Work and Organizational Psychology, 9, 371–400. Developing a framework for the ‘‘why’’ in change outcomes: Kompier, M., Geurts, S., Grundemann, R., Vink, P., & Smulders, P. The importance of employees’ appraisal of changes. In P. Ø. (1998). Cases in stress prevention: The success of a participative Saksvik (Ed.), Prerequisites for healthy organizational change and stepwise approach. Stress Medicine, 14, 155–168. (pp. 76–86). Bussum, The Netherlands: Bentham Publishers. PROCESS EVALUATION TOOL 617 Nielsen, K., Randall, R., & Christensen, K. B. (2010). Does Rychetnik, L., Frommer, M., & Shiell, A. (2002). Criteria for training managers enhance the effects of implementing team- evaluating evidence on public health. Journal of Epidemiology working? A longitudinal, mixed methods field study. Human and Community Health, 56, 119–127. Relations, 11, 1719–1741. Saksvik, P. Ø., Nytrø, K., Dahl-Jørgensen, C., & Mikkelsen, A. Nielsen, K., Randall, R., Holten, A. L., & Rial Gonza´ lez, E. (2002). A process evaluation of individual and organizational (2010). Conducting organizational-level occupational health occupational stress and health interventions. Work and Stress, interventions: What works? Work and Stress, 24, 234–259. 16, 37–57. Nielsen, K., Taris, T. W., & Cox, T. (2010). The future of Schurman, S., & Israel, B. (1995). Redesigning work systems to organizational interventions: Addressing the challenges of reduce stress: A participatory action research approach to today’s organizations. Work and Stress, 24, 219–233. creating change. In L. Murphy, J. Hurrell, S. Sauter, & G. Noblet, A. J., & LaMontagne, A. D. (2009). The challenges of Keita (Eds.), Job stress interventions (pp. 235–263). Washing- developing, implementing, and evaluating interventions. In S. ton, DC: American Psychological Association. Cartwright & C. L. Cooper (Eds.), The Oxford handbook of Semmer, N. K. (2003). Job stress interventions and organization of organizational well-being (pp. 466–496). Oxford, UK: Oxford work. In L. Tetrick & J. C. Quick (Eds.), Handbook of University Press. occupational health psychology (pp. 325–353). Washington, DC: Nytrø, K., Saksvik, P. Ø., Mikkelsen, A., Bohle, P., & Quinlan, M. American Psychological Association. (2000). An appraisal of key factors in the implementation of Semmer, N. K. (2006). Job stress interventions and the organiza- occupational stress interventions. Work and Stress, 14, 213–225. tion of work. Scandinavian Journal of Work and Environmental Øyum, L., Kvernberg Andersen, T., Pettersen Buvik, M., Health, 32, 515–527. Knutstad, G. A., & Skarholt, K. (2006). God ledelsespraksis i Semmer, N. K. (2011). Job stress interventions and organization of endringsprocesser: Eksempler pa hvordan ledere har gjort til en work. In J. C. Quick & L. E. Tetrick (Eds.), Handbook of positiv erfaring for de ansatte [Good management practice during occupational health psychology (2nd ed., pp. 299–318). Wa- change processes: Examples of how managers have made it a shington, DC: American Psychological Association. positive experience for employees] (Rep. No. 567). København, Shannon, H., Robson, L., & Guastello, S. (1999). Methodological Denmark: Nordisk Ministerra˚ d. criteria for evaluating occupational safety intervention re- Patton, M. Q. (2002). Qualitative research and evaluation methods. search. Safety Science, 31, 161–179. Thousand Oaks, CA: Sage. Shannon, H. S., & Cole, D. C. (2004). Commentary III making Pettigrew, A. M. (1990). Longitudinal field research on change: workplaces healthier: generating better evidence on work Theory and practice. Organization Science, 1, 267–292. organization intervention research. Sozial Praventivmedizin, Randall, R., Cox, T., & Griffiths, A. (2007). Participants’ accounts 49, 92–94. of a stress management intervention. Human Relations, 60, Steckler, A. B., & Linnan, L. (2002). Process evaluation for public 1181–1209. health interventions and research. San Francisco, CA: Jossey- Randall, R., Griffiths, A., & Cox, T. (2005). Evaluating organiza- Bass. tional stress-management interventions using adapted study Taris, T., Kompier, M., Geurts, S., Schreurs, P., Shaufeli, P., de designs. European Journal of Work and Organizational Psychol- Boer, E. et al. (2003). Stress management interventions in the ogy, 14, 23–41. Dutch domiciliary care sector: Findings from 81 organisations. Randall, R., Nielsen, K., & Tvedt, S. D. (2009). The development International Journal of Stress Management, 10, 297–325. of 5 scales to measure participants’ appraisal of organizational- Taris, T. W., & Kompier, M. (2003). Challenges in longitudinal level stress management interventions. Work and Stress, 23,1–23. designs in occupational health psychology. Scandinavian Richardson, K. M., & Rothstein, H. R. (2008). Effects of Journal of Work, Environment and Health, 29, 1–4. occupational stress management intervention programs: A meta- Vedung, E. (2006). Process evaluation and implementation theory. analysis. Journal of Occupational Health Psychology, 13, 69–93. In E. Vedung (Ed.), Public policy and program evaluation (pp. Roen, K., Arai, L., Roberts, H., & Popay, J. (2006). Extending 209–245). Piscataway, NJ: Transaction Publishers. systematic reviews to include evidence on implementation: Weick, K. E., Sufcliffe, K. M., & Obstfeld, D. (2005). Organizing Methodological work on a review of community-based and the process of sensemaking. Organization Science, 16, 409– initiative to prevent injuries. Social Science and Medicine, 63, 421. 1060–1071. Weiner, B. J., Amick, H., & Lee, S.-Y. D. (2008). Conceptualiza- Rossi, P. H., Lipsey, R. W., & Freeman, H. E. (2004). Evaluation: tion and measurement of organizational readiness for change: A systematic approach. London, UK: Sage. A review of the literature in health services research and other Rousseau, D. M., & Fried, Y. (2001). Location, location, location: fields. Medical Care Research and Review, 65, 379–436. Contextualizing organizational research. Journal of Organiza- Yin, R. K. (1994). Case study research: Design and methods (2nd tional Behavior, 22, 1–13. ed.). Thousand Oaks, CA: Sage. Ruotsalainen, J. H., Verbeek, J. H., Salmi, J. A., Jauhiainen, M., Persternack, I., & Husman, K. (2006). Evidence on the Original manuscript received February 2011 effectiveness of occupational health interventions. American Revised manuscript received February 2012 Journal of Industrial Medicine, 49, 865–872. First published online July 2012 http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png European Journal of Work and Organizational Psychology Taylor & Francis

Opening the black box: Presenting a model for evaluating organizational-level interventions

Opening the black box: Presenting a model for evaluating organizational-level interventions

Abstract

Organizational-level occupational health interventions are often recommended when improvements in working conditions, employee health, and well-being are sought within organizations. Research has revealed that these interventions result in inconsistent effects despite being based on theoretical frameworks. This inconsistency indicates that intervention studies need to be designed to examine directly how and why such interventions bring about change and why they sometimes fail. We argue that intervention studies should include a process evaluation that includes a close examination of the psychological and organizational mechanisms that hinder and facilitate desired intervention outcomes. By drawing on existing intervention literature we present an evidence-based model containing three levels of elements that appear to be crucial in process evaluation. We describe how this model may be applied and developed in future research to identify better the mechanisms that link intervention processes to intervention outcomes.

Loading next page...
 
/lp/taylor-francis/opening-the-black-box-presenting-a-model-for-evaluating-organizational-hc607zN4cA

References (95)

Publisher
Taylor & Francis
Copyright
© 2013 The Author(s). Published by Taylor & Francis
ISSN
1464-0643
eISSN
1359-432X
DOI
10.1080/1359432X.2012.690556
Publisher site
See Article on Publisher Site

Abstract

European Journal of Work and Organizational Psychology, 2013 Vol. 22, No. 5, 601–617, http://dx.doi.org/10.1080/1359432X.2012.690556 Opening the black box: Presenting a model for evaluating organizational-level interventions 1 2 Karina Nielsen and Raymond Randall National Research Centre for the Working Environment, Copenhagen, Denmark Department of Psychology, University of Leicester, Leicester, UK Organizational-level occupational health interventions are often recommended when improvements in working conditions, employee health, and well-being are sought within organizations. Research has revealed that these interventions result in inconsistent effects despite being based on theoretical frameworks. This inconsistency indicates that intervention studies need to be designed to examine directly how and why such interventions bring about change and why they sometimes fail. We argue that intervention studies should include a process evaluation that includes a close examination of the psychological and organizational mechanisms that hinder and facilitate desired intervention outcomes. By drawing on existing intervention literature we present an evidence-based model containing three levels of elements that appear to be crucial in process evaluation. We describe how this model may be applied and developed in future research to identify better the mechanisms that link intervention processes to intervention outcomes. Keywords: Methods; Organizational-level occupational health interventions; Process evaluation. Organizational-level occupational health interven- the small body of evaluation research (Egan, Bambra, tions can be defined as: ‘‘planned, behavioral, Petticrew, & Whitehead, 2009; Murta, Sanderson, & science-based actions to remove or modify the causes Oldenburg, 2007; Semmer, 2011). In their review, of job stress’’ (Mikkelsen, 2005, p. 152). In current LaMontagne, Keegel, Louie, Ostry, and Landsbergis European legislation there is a clear emphasis on the (2007) concluded that the published literature focused use of such interventions (i.e., changes in the design, on effect evaluation and contained relatively little, organization, and management of work) as the potentially important, process evaluation data about preferred way of improving working conditions and how interventions were planned and implemented. In tackling problems such as work stress (EU-OSHA, studies where process evaluation is attempted, it is 2010). This strategy has, however, been criticized often based on anecdotal data that have not been because of the lack of a large and consistent body of subjected to structured analysis (Bambra, Egan, evidence that shows these interventions to have a Thomas, Petticrew, & Whitehead, 2007; Murta et al., positive impact on working conditions and employee 2007; Roen, Arai, Roberts, & Popay, 2006). In this health and well-being (e.g., Briner & Reynolds, 1999; article we present a three-level evidence-based process Richardson & Rothstein, 2008). Others have argued evaluation model. This is intended to provide a that there is evidence of the positive impact of structure that researchers can use to guide the rigorous organizational-level occupational health interven- collection of detailed process evaluation data. We tions but that too few studies have examined why argue that the model can be used to strengthen the and how such interventions have succeeded or failed evaluation of organizational-level occupational health thus placing limits on the external validity of much of interventions in a number of ways. Correspondence should be addressed to Karina Nielsen, National Research Centre for the Working Environment, Lersoe Park Alle 105, DK-2100 Copenhagen, Denmark. Email: [email protected]. Raymond Randall’s current affiliation is the School of Business and Economics, Loughborough University, Loughborough, UK. This research was funded by the National Work Environment Research Fund (Grant 16-2004-09). The authors declared no potential conflicts of interests with respects to the authorship and/or the publication of this article. © 2013 The Author(s). Published by Taylor & Francis. This is an Open Access article. Non-commercial re-use, distribution, and reproduction in any medium, provided the original work is properly attributed, cited, and is not altered, transformed, or built upon in any way, is permitted. The moral rights of the named author(s) have been asserted. 602 NIELSEN AND RANDALL Organizational-level occupational health interven- actions in implementing any intervention and their tions are proactive in that they are focused on influence on the overall result of the intervention’’ reducing or eliminating the sources of job stress (Nytrø, Saksvik, Mikkelsen, Bohle, & Quinlan, 2000, (Hurrell & Murphy, 1996; LaMontagne et al., 2007). p. 214). This means that process evaluation (PE) may Researchers have found that some show no effects, be used to (1) provide feedback for improving others have been linked to improvements in working interventions, (2) replicate interventions in other conditions and employee health and well-being and a settings minimizing the number of pitfalls associated small number have prefaced deteriorations in these with a given intervention, (3) interpret the outcomes variables (Bambra et al., 2007; Egan et al., 2007; of interventions (Goldenhar, LaMontagne, Katz, Semmer, 2003, 2006). This mixed evidence has led to Heaney, & Landsbergis, 2001), and (4) help us considerable confusion and debate about how re- conclude on the generalizability, applicability, and search findings should be used to guide practice. The transferability of interventions studies (Armstrong problem stems from a prevailing focus on effect only et al., 2008). In summary, PE is needed to evaluate evaluation (Ruotsalainen et al., 2006). It is difficult to the generalizability of an intervention (to answer conclude why and how an intervention worked from questions such as ‘‘Under which circumstances will effect evaluation data only (Lipsey, 1996; Rychetnik, an intervention work?’’ and ‘‘Which were the Frommer, & Shiell, 2002) because effect-only evalua- processes that facilitated the change?’’) so that it tion data masks intervention effects that are sensitive can be implemented successfully in a variety of to variations in intervention processes (Lipsey, 1996). settings (Cooper, Dewe, & O’Driscoll, 2001). The nature of organizational-level occupational One of the reasons why PE is lacking in current health interventions indicates that their working research on organizational-level occupational health mechanisms are unlikely to be separate from the interventions may be that researchers are uncertain systems within which they operate. These interven- about what should be included in such evaluation. A tions require changes to complex social systems and number of papers have listed concepts that may be may be met with much resistance and have unintended examined in PE (Egan et al., 2009; Lipsey & Cordray, side-effects (Semmer, 2003). The internal validity of 2000; Murta et al., 2007; Nytrø et al., 2000). These evaluation studies of these interventions may be include organizational contexts, intervention reach, threatened by concurrent changes that worked against dose delivered and dose received, intervention fide- the intervention plan: This means that the same lity, support and available resources, recruitment, intervention could yield powerful effects if the context and attitudes towards the intervention. However, was less disruptive to the intervention plans and there is no integrated, evidence-based framework that processes (Mikkelsen, 2005). Moreover, changing describes the elements that need to be included in complex systems requires wide-ranging multifaceted process evaluations of organizational-level occupa- activities and diluted or disrupted intervention activ- tional health interventions. ities may not be intense enough to have an impact We propose that the factors that may have an impact (Bambra et al., 2007). Considering the complexity of on the outcomes of an organizational-level occupa- organizational-level occupational health interven- tional health intervention can be grouped into three tions, evaluation models and methods are needed that themes. The themes are: the intervention design and can be used to identify how the potential effects of implementation, the intervention contexts, and parti- interventions on health and well-being are moderated cipants’ mental models (of the intervention and their and mediated by intervention processes. work situation). The first theme determines the max- Such a shift in evaluation strategy represents a imum levels of intervention exposure that can be move away from ‘‘black box’’ and a move towards an achieved; the latter two represent the factors that may approach that can elaborate on the mechanisms moderate or mediate the link between any intervention through which changes in the outcomes operate: exposure and its outcomes. Within each theme we Looking inside the black box reveals various sources identify a number of specific questions about the of variation (Lipsey & Cordray, 2000). A fundamental intervention processes that need to be answered objective of this shift is to differentiate between theory/ through the collection of process evaluation data. programme failure (that the theory behind the Together, the collection of data across these themes is intervention did not address the problem) and likely to provide the evaluator with some useful insights implementation failure (that the way the intervention into the factors that influence the outcomes of an was implemented was incomplete or designed in such a organizational-level occupational health intervention. way that the intervention would have failed even if the In this article we have chosen to focus on the theory behind the intervention was correct) (Harachi, literature on health and well-being interventions at Abbott, Catalan, Haggerty, & Fleming, 1999). the organizational level and the factors and elements Intervention process has been defined as ‘‘indivi- identified in this literature. This is because PE dual, collective and management perceptions and models are needed that fit with the measurement PROCESS EVALUATION TOOL 603 opportunities and constraints operating in this with employees in two organizations, one private- domain; however, models of PE have been used with and one public-sector, to identify further themes and considerable success in other disciplines such as confirm existing themes (references withheld to public health, organizational development, and orga- ensure anonymity). nizational change (Armenakis & Bedeian, 1999; Burke & Litwin, 1992; Cummings & Worley, 2009; Rossi, Lipsey, & Freeman, 2004; Steckler & Linnan, A MODEL OF PROCESS EVALUATION 2002), and many of the factors and elements Intervention design and implementation identified in these disciplines may also be relevant in a model of the evaluation of the processes of In the following section we discuss the elements that organizational-level occupational health interven- should be included to document the intervention design tions. In developing a model specific to organiza- and implementation. We focus on three overarching tional-level occupational health interventions, we elements: initiation, intervention activities, and imple- drew on four sources of information. First, we mentation strategy. These themes are not orthogonal: identified two rigorous review articles focusing on Because we are describing interlinked and complex process factors (Egan et al., 2009; Murta et al., 2007) organizational processes, issues within a theme may and from here we drew the relevant factors identified also interact with other issues in other themes. The in our model. Second, we identified a number of model is presented in Figure 1. To allow readers to see papers that focused on the topic of intervention more clearly how the model may be translated into PE implementation (Cooper et al., 2001; Guastello, 1993; tools and methods, the key issues are presented as Lipsey, 1996; Nytrø et al., 2000; Pettigrew, 1990; questions that should be addressed during PE. Semmer, 2003, 2006, 2011; Shannon, Robson, & Guastello, 1999; Vedung, 2006). Third, we reviewed Initiation: Who initiated the intervention and for existing intervention studies to identify any analysis, what purpose? The motivation driving an however anecdotal, of process factors such as mental intervention may be related to problems internal to models, context, and/or intervention design and the organization (to deal with a crisis, to improve implementation. These are the papers that are quality and productivity or to become a healthy referred to throughout this article. Finally, we workplace) or external challenges (e.g., legislative developed a preliminary model which we tested in requirements) or a combination of both (Kompier, semistructured interviews (N ¼ 54) with key stake- Geurts, Grundemann, Vink, & Smulders, 1998; holders including human resources practitioners, Shannon & Cole, 2004). Any intervention can internal consultants, managers, and 44 focus groups stabilize or displace current power structures and Figure 1. Model of process evaluation. 604 NIELSEN AND RANDALL therefore reasons for the intervention are likely to who is responsible and who were the targets of influence the buy-in of key stakeholders (Fredslund & intervention (Nielsen, Randall, Holten, & Rial Strandgaard, 2005). It is therefore important to Gonza´ lez, 2010). The contents of action plans also explore who defined the problem, who decided what often highlight the potential ‘‘active ingredients’’ of should be done, and who should implement change. the intervention that could be linked to intervention This means that it is also important to identify the outcomes (Nielsen, Randall, & Christensen, 2010). key stakeholders in the initiation process: These may be managers, employees, union representatives, Implementing intervention activities: Did the occupational health practitioners, and clients. This intervention reach the target group? Careful identification may help understand the reactions and documentation of the actual implementation of actions of other key stakeholders (see later). intervention activities is a vital element of process Some of the effects of this decision-making process evaluation (Semmer, 2003, 2006, 2011) because this have been identified by Egan et al. (2007). They will highlight any discrepancies between the planned concluded that interventions initiated for perfor- intervention and its implementation (Roen et al., mance reasons were found to have an adverse impact 2006). Important questions include: ‘‘Which aspects on employee health and well-being, whereas inter- of the intervention activities brought about ventions whose rationale was to improve employee noticeable changes?’’ ‘‘How many changes were health and well-being was found to have a positive delivered to whom?’’ ‘‘Who noticed/reported these effect on these same outcome measures. changes?’’ The potential active ingredients of the intervention also need to be reexamined and Developing intervention activities: Did the intervention compared to the active ingredients identified in the activities target the problems of the workplace? It has intervention plan. This identification helps to rule out been argued by many researchers that the correct rival hypotheses for intervention results, i.e., that tailoring of an intervention to the needs of other factors than the intervention account for stakeholders requires a thorough risk assessment observed changes or lack of change (Lipsey, 1996). (LaMontagne et al., 2007; Nielsen, Randall, Holten, The importance of documenting differences be- &RialGonza´ lez, 2010; Noblet & LaMontagne, tween planned and actual exposure has been high- 2009). A thorough risk assessment is a crucial lighted in a number of studies. Nielsen, Fredslund, diagnostic process (Kompier et al., 1998) and Christensen, and Albertsen (2006) found that in a provides information that can be used to check designated intervention group no changes in well- whether intervention activities addressed the being were detected because intervention activities problems perceived by organizational members. had not been implemented, whereas changes were Context-independent organizational-level strategies observed in a designated control group because the have been described as unlikely to succeed as each manager had initiated activities intended only for the organization is unique and therefore require unique intervention groups. Similarly, Landsbergis and solutions (Hurrell & Murphy, 1996). Thus, it is Vivona-Vaughan (1995) found that in an intervention important during the PE to examine whether department where no effects were found, a planned intervention activities were tailored to the problem ‘‘policy and procedures’’ manual had not been as it is manifested itself in the specific organizational completed and implemented. context. Tailoring of organizational-level interventions Implementation strategy does not usually include adapting interventions to meet the requirements of specific individual employ- In this section we focus on the roles and behaviours ees. This has been cited as a potential problem with of key stakeholders. Later in the section on mental organizational interventions: An optimal strategy models we focus on the appraisals and perceptions of may be to use a combination of different interven- key stakeholders and how these may drive key tions (LaMontagne et al., 2007). Individual-level stakeholders’ behaviours, thereby indirectly influen- activities implemented during the development of cing intervention outcomes. organizational-level interventions may prime partici- pants to support and engage in organizational-level Drivers of change and the roles of key stakeholders: changes when they are implemented (Nielsen, Ran- Who were/are the drivers of change? In complex dall, Brenner, & Albertsen, 2009; Nielsen, Randall, & interventions, there are often many stakeholders in Christensen, 2010). Developing structured action the intervention process and therefore many potential plans may also facilitate effective intervention: Such drivers of change. In PE these stakeholders must be plans describe intervention activities in terms of the identified and their role in the change process resources needed, the activities undertaken, and how explored. It is important to identify who has the the intervention is implemented, including identifying power to make changes in order to identify how PROCESS EVALUATION TOOL 605 much they were involved in intervention activities intervention projects (i.e., the resources needed to (Nytrø et al., 2000). Next we discuss in further detail plan, implemen, and evaluate the project), they may act the roles of key change agents. as role models through their attitudes to the intervention, and they may be actively involved in Participatory approaches—involving employees: Did intervention activities (Giga, Noblet, Faragher, & employees participate significantly in decision making Cooper, 2003; Lindstro¨ m, 1995; Randall, Cox, & and how many were involved? The participatory Griffiths., 2007). However, because of their seniority approach has been advocated as a desirable and own work demands they are rarely able to follow intervention strategy and plays a major role in well- closely intervention development activity and known organizational occupational health implemen-tation (Nytrø et al., 2000). Although the intervention approaches (Nielsen, Randall, Holten, & importance of senior management support is often Rial Gonza´ lez, 2010). Participation in the development discussed, it is seldom formally evaluated (Nielsen, of health promoting activities is also included in the Randall, Holten, & Rial Gonza´ lez, 2010; Semmer, guidelines of the World Health Organization and the 2011). What research there is suggests that suchsupport European Network for Workplace Health Promotion can be important. In a study of stress coping training, (European Network for Workplace Health Lindquist and Cooper (1999) found that when senior Promotion, 2007). The essence of participation is a management released staff from their duties to conscious and intended effort made by individuals at a participate in workshops, attendance was 100%. In higher level in an organization to provide visible contrast, at follow-up when staff had to participate extrarole or role-expanding opportunities and during their leisure time, participation dropped to 66%. enhanced control for individuals or groups at a lower Saksvik, Nytrø, Dahl-Jørgensen, and Mikkelsen (2002) level in the organization (e.g. to have a greater voice). reported decreased opportunities for staff to take part Participation can take different forms: it may be in participatory workshops were due in part to informal or delivered through formal changes in roles constraints imposed by senior management only and responsibilities, directly experienced or indirectly allowing employees time to participate in short through union representatives, and the breadth and workshops. In this example, the lack of support from depth of participation and the extent of influence senior management also had a ‘‘trickle down’’ effect on linked to the participatory activities can also vary middle managers, who reported they did not support (Lines, 2004). the intervention project as they were allocated no Several pieces of research discuss how both resources to implement initiatives. qualitative (type of) and quantitative (amount of) participation might have influenced intervention Middle managers: What was the role of middle outcomes. Nielsen et al. (2006) found that employees managers? While senior managers often make the with little formal education benefited most from a decision to implement the intervention it is usually directive type of participation where they were told middle managers that are subsequently responsible what to do. Aust, Rugulies, Finken, and Jensen for communicating and implementing change (Guth (2010) found that employees reacted negatively to & Macmillan, 1986). Therefore, middle managers only having influence over parts of the intervention play a crucial role in many organizational-level programme, i.e., limited influence over the scope of occupational health interventions (Nielsen & the problem. Concerning the amount of participa- Randall, 2009; Randall et al., 2007). Kompier, tion, Lines (2004) found that this was negatively Cooper, and Geurts (2000) found that in all 11 case related to resistance of change, and positively related studies they collected from across Europe middle to achievement of goals and organizational commit- managers were primarily responsible for stress ment. Similarly, Nielsen, Randall, and Albertsen prevention interventions. This puts middle (2007) found that high levels of reported participa- managers in a position to hinder or facilitate the tion in change were associated with low levels of change. For example, Dahl-Jørgensen and Saksvik behavioural stress symptoms and higher job satisfac- (2005) reported that middle managers resisted change tion after intervention. Eklo¨ f, Ingelga˚ rd, and Hag- by restricting the time spent on interventions by berg (2004) found that the degree of participation in employees. Similarly, Nielsen and Randall (2009) the resolution of occupational health concerns was found that where middle managers were perceived as consistently associated with decreased work de- supportive and took an active part in implementing mands, increased social support, and lowered stress change, employees reported better working levels. conditions and higher levels of psychological well- being after the intervention. Middle managers can Senior management support: What was the role of also bring about changes in intended intervention senior managers? Senior managers are often exposure patterns. For example, Nielsen et al. (2006) involved in the allocation of resources to found that a new manager who resented being in the 606 NIELSEN AND RANDALL control group initiated and implemented activities change and changes in work roles linked to the similar to those planned in the inter-vention groups intervention (Øyum, Kvernberg Andersen, Pettersen with positive outcomes. In a process evaluation of Buvik, Knutstad, & Skarholt, 2006). It has also been seven intervention projects, Saksvik et al. (2002) found that open communication helps employees to found that middle managers had often exerted understand the intentions behind organizational-level passive resistance that had damaged and diluted occupational health interventions, thus improving some intervention activities. Together these findings employee commitment to and participation in the indicate that middle managers’ motivation for intervention (Nytrø et al., 2000). Communication is implementing change should be documented along likely to influence employees’ sense making (e.g., their with the actions they take to facilitate or obstruct perception of the motives and objectives of the change. This documentation is especially important intervention) and this appears to be closely linked because such data may not be captured in their to their commitment to intervention activities (Weick, performance appraisals and in times of pressure they Sufcliffe, & Obstfeld, 2005). Therefore, it is important may therefore choose to prioritize other aspects of to examine what kind of information has been their job over and above intervention activities distributed, to whom, and how it has been received (Saksvik et al., 2002). and perceived. This means that three important questions need to Consultants: What was the role of be answered. (1) Were participants informed about consultants? Large intervention projects often use the project? Nielsen et al. (2007) found that receiving external consultants to design, implement and adequate information about an intervention project facilitate aspects of the intervention process predicted the extent to which employees participated (Nielsen, Randall, Holten, & Rial Gonza´ lez, 2010). in intervention activities. One important caveat to Lindstro¨ m (1995) reported that process consultants this finding was that where information was not facilitated organizational changes by giving feedback followed up by actual activities, employees were on the progress of change and on group dynamics. In disappointed and reported negative results. (2) Were another study the consultants played a role in the risk assessment results fed back? This feedback has proliferation of the intervention activities as they sold been found to lead to more intervention activities similar services to those implemented to other groups (Eklo¨ f, Hagberg, Toomingas, & Tornqvist, 2004). In in the organization (Nielsen et al., 2006). However, a later study, Eklo¨ f and Hagberg (2006) found the few intervention studies have included an evaluation most significant changes in social support were of the role or the competencies of the external observed in parts of the organization where super- consultant(s) (Semmer, 2006). In order to isolate visors, and to a lesser extent work groups, had intervention effectiveness it is important to evaluate received detailed information about the problems whether the consultants had the necessary skills and identified in the risk assessment. (3) To what extent abilities to enhance the intervention process by are all participants updated about progress? Land- motivating and guiding participants through the sbergis and Vivona-Vaughan (1995) found that those intervention process (Landsbergis & Vivona- employees not directly involved in intervention Vaughan, 1995). It may also be that when external planning and implementation tended to be less aware consultants have total responsibility for change they of the progress of the intervention: These employees leave no infrastructure within the organization for also reported that the intervention had little effect. sustaining and continuing improvements they initiated, thus reducing long-term intervention Context effects (Dahl-Jørgensen & Saksvik, 2005). Nielsen, Cox, and Griffiths (2002) argued that, for Field studies are conducted to enhance the ecological intervention effects to be maintained in the long validity of intervention research. However, this term, a shift must take place in which organizational validity can only be achieved if the influence of the members gradually take more responsibility for the social and organizational context on intervention intervention from the consultant. outcomes is measured and analysed (Heaney et al., 1993; Rousseau & Fried, 2001). Context can be Information and communication about the defined as ‘‘situational opportunities and constraints intervention: What kind of information was provided that affect the occurrence and meaning of organiza- to participants during the study? It has been shown tional behaviour as well as functional relationships that the level of information and communication between variables’’ (Johns, 2006, p. 386). The context plays an important role in the effects of interventions may either facilitate or hinder successful implementa- (Jimmieson, Terry, & Callan, 2004). Providing tion. Intervention context can provide a link between information about a change keeps employees up to intervention plans and intervention exposure (i.e., date about anticipated events, the consequences of work as a mediator) or can dilute or strengthen the PROCESS EVALUATION TOOL 607 effects of intervention activities (i.e., work as a and national level (e.g., economic recession; moderator). The overall question is to ask: ‘‘Which Landsbergis & Vivona-Vaughan, 1995; Mikkelsen & hindering and facilitating factors in the context Saksvik, 1999; Nielsen, Randall, & Christensen, influenced intervention outcomes?’’ Because the 2010) should be considered. context is diverse and multifaceted the concepts of omnibus and discrete context (Johns, 2006) provide a Mental models useful framework for PE. Recently, research has begun to examine how Omnibus context. This refers to the story told and individuals’ perceptions and appraisals of an prompts several process evaluation questions organizational-level occupational health interven- including: ‘‘Who are the participants in the inter- tion are linked to outcomes through how they vention and who drives the intervention?’’ ‘‘Where drive the behaviours of key stakeholders. Employ- does it take place?’’ ‘‘When did the intervention take ees, managers and other key stakeholders may have place?’’ The underlying theme of these questions is: diverse and potentially conflicting agendas that may ‘‘How did the intervention fit in with the culture and influence how they behave and react to the conditions of the intervention group?’’ Dahl- intervention. These underlying psychological pro- Jørgensen and Saksvik (2005) found that a context cesses may help to explain change outcomes (Nytrø in which there were high job demands often hindered et al., 2000) but have rarely been measured directly participation in interventions. Organizational culture in intervention research. The main question to ask may also play a role: Saksvik et al. (2002) found that a here is: ‘‘What is the role of participants’ mental bureaucratic organizational structure or being part of models in determining their response to the inter- a larger, international organization hindered the vention?’’ Mental models are used to make sense of development of inter -vention activities. the world and explicit efforts at sense making take Furthermore, it is important to ask: ‘‘What capacity place when the world is perceived to be different does the organization have to conduct interventions?’’ from the expected state of the world, e.g., when The preintervention healthiness of an organization changes at work are occurring (Weick et al., 2005). and its past use of and experience with such Translating this into an intervention context, interventions have been found to affect mental models determine how participants react to organizational occupational health intervention the intervention and its activities and help explain outcomes (Semmer, 2006). Workplaces with low the behaviours of key stakeholders throughout the demands, high levels of support, and low stress intervention project. levels may have more time and resources to involve For example, it has been found that different workers and managers in participation and stakeholders have conflicting mental models about integration of interventions. On the other hand, what constitutes success (Cole et al., 2003; Shannon healthy organizations with low levels of stress and a & Cole, 2004). Detecting these different perspectives good working environment may not need in- may help to explain how different motivations drive terventions (Taris et al., 2003). Indeed, ceiling differences in key stakeholders’ behaviours during the effects may prevent further improvement in intervention process. Saksvik et al. (2002) found that intervention outcomes, even if the theory behind managers preferred individual-level interventions, intervention is correct (Nielsen et al., 2006). This i.e., putting responsibility for change at the indivi- represents an intervention paradox whereby the dual. In contrast, employees held negative attitudes omnibus context can inhibit intervention where it is to these interventions because they felt that this needed most. strategy was a way for managers to escape respon- sibility thus failing to address the problems in the Discrete context. This aspect of context focuses workplace. on specific events that may have influenced the effects For interventions to be effective, it has been of the intervention. The question asked here is: argued that employees should perceive that they have ‘‘Which events took place during the intervention problems that need to be addressed, believe that the phase?’’ Some factors that have been identified here intervention will be effective in addressing those are new project management demands (Nielsen et al., problems, and be motivated to actively support the 2006), conflicting priorities, concurrent use of intervention by participating in intervention activities multiple change programmes (Guastello, 1993), and (Nytrø et al., 2000). This implies that an important lack of integration of the intervention with important PE question is: ‘‘To what extent are participants corporate strategic decision-making activities ready for change?’’ Readiness for change has been (Schurman & Israel, 1995). Factors at both the widely researched, but rarely within the organiza- intraorganizational (e.g., introducing conflicting tional-level occupational health intervention litera- initiatives; Nielsen et al., 2006; Randall et al., 2007) ture. There is an abundance of literature that has 608 NIELSEN AND RANDALL linked the degree to which employees welcome and they felt there was no need for risk assessment since actively support the implementation of change to their perception was that stress was not a problem organizational development and change outcomes (Biron, Gatrell, & Cooper, 2010). In a study of (Weiner, Amick, & Lee, 2008). A number of studies organizational improvement programme and an have discussed the importance of mental models of individual-level stress management programme, readiness for change in organizational-level occupa- Bunce and West (1996) found that the perceived tional health intervention research. Randall, Grif- smoothness of implementation of the training pro- fiths, and Cox (2005) found that a change in gramme and the depth of programme content were responsibility had not been communicated to staff related to lower levels of stress and higher levels of because managers felt the intervention would have a job satisfaction after the programme. Nielsen et al. detrimental outcome on their own working condi- (2007) found that individuals’ appraisal of the quality tions: They were not ready for the change because and sustainability of intervention activities were their appraisal was that the intervention would positively linked to postintervention well-being. In damage their own working conditions. Another participatory action research projects where employ- mental model concerning readiness of change is ee representatives are involved in making change, it is initiative fatigue: If organizational-level occupational particularly important to examine the mental models health interventions have previously been conducted of those not directly involved in decision making but but little learning has taken place, this failure may who were targeted by the intervention (Landsbergis & have a detrimental impact on participants’ percep- Vivona-Vaughan, 1995). In some instances, percep- tions of later initiatives and their willingness to tions of key stakeholders’ expertise may also be participate in intervention activities (Nytrø et al., influential. Nielsen et al. (2006) reported that employ- 2000; Saksvik et al., 2002). ees with little education and little experience with Over time, people who work closely together may dealing with occupational health issues appraised an develop similar models to interpret and react to the external occupational health practitioner more posi- world (Mathieu, Heffner, Goodwin, Salas, & Can- tively because of her directive approach and a focus non-Bowers, 2000). In an intervention context on individual issues. participants with shared mental models may perceive the intervention and its activities in a similar manner Changes in mental models of the job. In order for and as a result react in similar ways to the introduced real changes to happen as a result of organizational- intervention. Where shared mental models have not level occupational health interventions, it has been been developed individuals may have conflicting pointed out that participants and key stakeholders agendas that may hinder effective implementation must unlearn old mental maps of their working and lead to diversity in intervention experiences and conditions and learn new ones (Schurman & Israel, outcomes. Recent research has shown that a lack of 1995). An important driver of this change is the support for an intervention programme was the result degree to which intervention activities prompt a shift of differences in stakeholder views about the most from espoused theories about the intervention to effective intervention option: Consultants felt the theories-in-use (Argyris, 1976, 1995). Theories-in-use focus of the intervention programme should be are the mental models that guide our behaviour, leadership development, whereas employees thought whereas espoused theories are the attitudes and it should have focused on employee involvement beliefs that we tell others guide our behaviour. (Aust et al., 2010).Therefore, process evaluation According to Argyris (1976, 1995) real change only should be used to examine the mental models and happens when individuals change their theories-in- the degree to which mental models are shared by use. Therefore, an important part of process participants using the question: ‘‘To what degree do evaluation should be the measurement of change in participants have shared mental models?’’ employees’ knowledge of the intervention, their An equally important mental model to explore is: expectations that the intervention can bring about ‘‘How did participants perceive the intervention and changes, and that these changes can have an impact its activities?’’ Employees’ perceptions of the drivers and be sustained as part of continuous improvements of change and the intervention objectives are likely to at work (Schurman & Israel, 1995). The important influence their willingness to participate in interven- question here is: ‘‘Did the intervention bring about a tion activities. If they believe activities do not address change in participants’ mental models?’’ In a the problems raised or are of a poor quality this may randomized, controlled study, Nielsen, Randall, and reduce their engagement. Christensen (2010) found that team manager training A study on the implementation of a Stress Risk that produced changes in mental models was need to Assessment tool revealed that middle managers (who bring about led to changes in managers’ behaviours were responsible for using the tool) failed to use the and, as a consequence, their subordinates’ tool after they attended a training course because involvement and job satisfaction. PROCESS EVALUATION TOOL 609 stakeholders had a hidden agenda such as employees PE methods wanting to get rid of an unpopular manager or In the previous sections, we have described a set of unpopular colleagues, managers wanting to be seen topics which we argue should be considered when to be ‘‘doing something’’, or internal consultants determining the validity and generalizability of creating work for themselves, justifying their organizational-level occupational health interven- existence). tions. This leads to an important question: ‘‘How do we get this information?’’ We argue that process Intervention activities. The risk assessment evaluation calls for a mixed methods approach. method itself and feedback reports provide Detecting the different active ingredients of organiza- information on the ‘‘objective’’ part of risk tional-level occupational health interventions requires assessment. This element of PE should include the the use of a range of different methods. The measure- collection of stakeholders’ experiences of the ment of some active ingredients requires the collection feedback of risk assessment results. Through of observer or objective data, e.g., what can be seen to questionnaires at follow-up, employees can be asked be happening. Other PE constructs are appraisals (e.g., whether they are aware of the results of risk how participants appraised the intervention influences assessment, have participated in meetings feeding intervention outcomes) that cannot be directly ob- back results, or whether the results have been served. In addition, the wide range of PE constructs discussed with managers and colleagues. It may also and the importance of context indicate that a flexible be examined whether participants felt the risk approach to data collection and analysis offered by assessment tool examined the appropriate topics. qualitative methods is likely to have utility. Combining Appraisals of the risk assessment may be collected both qualitative and quantitative results through through obtaining minutes from feedback meetings, mixed methods may offer four important benefits: direct observation of feedback activities, and Identification of the mechanisms behind any changes inspection of feedback documents. Such data can brought about by the intervention; added meaning to help determine how the risk assessment method and the results of outcome evaluation; cross-validated and content was determined upon, and which strategies triangulated results; and identification of the impact of were employed to feed results back to employees. the intervention context on the processes and out- Interviews provide an important source of data to comes of change (Greene, Benjamin, & Goodyear, gain information about the impact of risk 2007; Hugentobler, Israel, & Schurman, 1992; John- assessments: Have the results been used and how? son, Onwuegbuzie, & Turner, 2007; Nastasi et al., Independent reviews of documented actions plans 2007). Unfortunately, research on interventions rarely provide a method of obtaining information about the includes interview schedules, observation strategies, or content and objectives of action plans. It is possible to detailed information about how these data were detect the level of detail, e.g., do action plans include collected and used. It is therefore difficult to refer to vague objectives such as: ‘‘We want to be better at specific examples that provide guidance on how this communicating’’ or do they include detailed concrete may be done. Therefore, in the following sections we descriptions of the objective and associated activities, describe the data that may be collected and provide deadlines, required resources needed, and allocate preliminary estimates of the suitability of different responsibilities for key tasks? Comparisons can also be data collection methods (see Table 1). made between the results of risk assessment and how these fed into action plans. Cognitive mapping techniques may be useful in making the connection between problems present in the organization and Documentation and appraisal of the planned intervention activities (Harris, Daniels, & intervention design and implementation Briner, 2002): This will help to determine if there is a Initiation. In the first step it is important to clear link between the results of the risk assessment document the precursors of intervention activities. and the activities outlined in actions plans. This provides some documentary evidence of the Through observations of action planning meetings process. Such information can be obtained through and interviews with those involved, it is possible to sources such as meeting minutes and other explore the translation of risk assessment results into organizational material that records intervention- action plans. These data collection methods should related events. These may reveal who took the address questions about how and why activities were initiative to initiate a project, how the intervention prioritized and by whom. It is also possible to objectives were formulated, and who participated in document the degree to which actions plans were making the decision to design the project. Interviews implemented: This may be obtained through ques- may be better suited to the exploration of mental tionnaire data collected from various stakeholders. models (e.g., it can be revealed whether any Through listing the activities planned in question- 610 NIELSEN AND RANDALL TABLE 1 Process evaluation checklist Questions Necessary data ‘‘objective’’ Additional data ‘‘appraisal’’ The intervention Who initiated the intervention and for what purpose? Organizational data: Interviews: Who took the initiative? Detection of hidden agendas What was the official objective? Did the intervention activities target the problems of the workplace? Comparison of risk assessment to action Interviews: . What means of risk assessment was used? plans Appropriateness of intervention tools including . What was the implementation strategy? Documentation of activities developed workshop tools and questionnaires . How many changes were planned? Review of workshop material Questionnaires, Appropriateness of intervention . Was the focus on one large change or rather many small steps to make activities a change? . Were actions plans sufficiently detailed? . To which extent were activities tailored to the organization? . To which extent did the activities target multiple levels? . How was risk assessment translated into activities? Did the intervention reach the target group? Questionnaires, meeting minutes Interviews: . Why were intervention activities not implemented? Actual implementation Degree of intervention activity implementation . Which aspects of the activities brought about changes? . How much were delivered to whom? . Who noticed changes? . Were positive and negative ‘‘side effects’’ monitored? . Was a ‘‘plan B’’ developed to address any lack of progress? Who were the drivers of change? Questionnaires: Interviews: . Did the roles and commitment of key stakeholders change over time? The extent of involvement Identification of drivers of change throughout all . Did employees participate in real decision making and how many were phases of the project involved? Key stakeholders role of involvement . Did employees assume responsibility for the project and the completion of activities? . What was the role of formal representatives? . What was the role of senior managers? Were the necessary resources allocated? Did they support the project throughout – and how was support manifested? . What was the role of middle managers? Did they support the project throughout – and how was support manifested? Did they function as the link between senior management and employees? Did they encourage active participation by employees? . What was the role of consultants? Did they create a supportive, trusting atmosphere? Did they use tools to facilitate the process, e.g. visualizing progress? And how did they work? Did the organizational consultant enable organizational ownership? (continued overleaf ) PROCESS EVALUATION TOOL 611 TABLE 1 (Continued ) Questions Necessary data ‘‘objective’’ Additional data ‘‘appraisal’’ What kind of information was provided to participants during the study? Organizational data: Interviews: Were participants informed about the project? Review of meeting minutes, memos, Appropriateness and quality of information . Were risk assessment results fed back? correspondence, posters, leaflets . To which extent are all participants updated about progress? . Were small successes celebrated? The context Which hindering and facilitating factors in the context influenced intervention Organizational data, news media Interviews: outcomes? What happened in the discrete context? Identification of hindering and facilitating factors in . Why were intervention activities not implemented? the omnibus and discrete context . How did the intervention fit with the culture and conditions of the Degree of intervention activity implementation intervention group?(omnibus context) . What capacity did the organization have to conduct interventions? (omnibus context) . Which events took place during the intervention phase?(discrete context) . Did a change in management take place? . Did organizational restructuring take place during the intervention phase? . Did changes outside the organization take place during the intervention phase that may have influenced intervention outcomes, e.g., changes in legislation, recession etc.? Participants’ mental models What is the role of participants’ mental models? Questionnaires: Interviews and observations: . To which extent are participants ready for change? Degree of intervention openness to change Changes in employees’ perceptions of themselves, . To which degree do participants have shared mental models? Quality, quantity, and appropriateness of their colleagues and their workplace In case of resistance, what were the threats appraised by key intervention activities post-intervention stakeholders? Degree of intervention activity implementation In case of divergence, how did mental models differ? . How did participants perceive the intervention and its activities? . Did the intervention bring about a change in participants’ mental models? . Why were intervention activities not implemented? *The question ‘‘Why were intervention activities not implemented?’’ can be asked at all three levels, but the answers are different for each level. At the intervention level, the answer will reside in the actions of key stakeholders, whereas at the context and mental models level, we may obtain information about the reasons behind these behaviours. 612 NIELSEN AND RANDALL naires, and asking respondents to indicate which they memos, and correspondence provide ‘‘objective data’’ have participated in, or which procedures have been on how the intervention was communicated. How- changed, it is possible to document the reach of ever, this organizational data should be supplemen- activities (see Nielsen et al., 2006). Through inter- ted by interviews and questionnaires to ensure that views we may gather data on why activities may not this information actually reached participants and have been implemented: Questions can be asked how it was perceived by recipients (either by interview about whether the necessary resources were available or questionnaire methods). As has already been and if not why participants perceived this to be the discussed the intentions driving the intervention case (e.g., was it because senior managers were may not match the way these are perceived by its reluctant to allocate resources or did no-one assume recipients. A question for intervention recipients responsibility for implementing actions?). should therefore be: Was information sufficient and The drivers of change may be identified through was it easy to understand and relate to daily work? If interviews, meeting minutes, and questionnaires. Data middle managers were given leaflets did they dis- may be collected by asking employees about the extent tribute them? Did participants read emails about the to which they were involved in developing and project, updates in personnel magazines, or informa- implementing change (see Randall, Nielsen, & Tvedt, tion sent to them? It is also important to document 2009). A recent criticism of action research approaches the reach of information about different parts of the is that often only few employees are involved through intervention process. Are employees informed about steering groups or health circles (Nielsen, Randall, the initiation of the project but then hear nothing Holten, & Rial Gonza´ lez, 2010). It has been argued about the results of risk assessment, the developed that only through involving all employees the advan- action plans, and the implemented changes? If this is tages of participation can be achieved (Hurrell, 2005): the case they are likely to develop a perception that Therefore, information about participation should be the project was unsuccessful. Even if improvements collected from as many of the intended beneficiaries of are being implemented they fail to see the link the intervention as possible. between the intervention project and improvements if While interviews provide rich information about these are not clearly communicated. how employees were involved and the perceptions of such involvement these do not provide information Documenting the importance of context about how many had the opportunity to influence the intervention process. A questionnaire has now been Qualitative methods may be best suited to unpredict- developed which can be used to examine the degree to able, complex, and difficult settings as a means of which all employees were involved in the intervention mapping critical events that may influence outcomes process (Randall et al., 2009). Information about (Yin, 1994). Qualitative studies may be suitable for senior and middle management support can also be examining contextual levels that may affect the obtained through interviews and questionnaires in a intervention outcomes (the employment of a man- similar manner (e.g., Nielsen & Randall, 2009). ager that supports change) and explore the full range Through observing activities that are designed to of behaviours and attitudes that the context might enhance participation it is possible to collect data on, affect thereby working backwards to make inferences for example, how many employees were invited to about the situation. Data may be obtained from attend the activity, how many did actually participate, meeting notes, interviews, and organizational mate- how many participants were active during the meet- rial. Data may also be collected on any concurrent ings, and what role middle managers, employees, events such as changes in management and/or senior managers, and, if applicable, consultants took ongoing restructuring. during the activities. The following questions could be Through news media, information may be ob- addressed using these data: Did all organization tained about the national context, e.g., changes in members make suggestions for planning and imple- legislation and media attention (e.g., in some mentation the intervention and its activities? How was countries the teaching profession receives negative the general climate between different key stake- media attention on a regular basis). This material, holders—did employees have an actual say or were however, should always be analysed in terms of how decisions made by the management levels? Did any it relates to the intervention itself. Further informa- stakeholders show resistance? How were the meetings tion about the impact of such factors may be run, e.g., were tools used to facilitate discussions? obtained through interviews, meeting minutes, and other organizational material. Also at this level, the question ‘‘Why were Communication and information activities not implemented?’’ could be asked but with Organizational material and archive data such as additional probe questions about answers that point posters, meeting notes, mail communication, leaflets, to the impact of context: It may be that employees PROCESS EVALUATION TOOL 613 with substantive contact with clients found it hard to leaders became aware of exerting transformational prioritize participating in activities that took away leadership and employees started taking over time from clients or that concurrent restructuring responsibilities such as rota planning and inde- took up all the time and energy of managers and pendent problem solving. These data can be collected employees. Such information may be best obtained by asking employees whether there have been any through interviews. changes in behaviours and work procedures during the intervention project and whether these are seen to be ascribed to the intervention. Evaluating the impact of mental models Some aspects of mental models such as readiness for Timings of data collection change can be explored through questionnaires (Weiner et al., 2008). However, changes in multi- It has been argued that process evaluation questions faceted and complex mental models and the diversity should be included at follow-up (Randall et al., 2009), of mental models may be more difficult to detect but it is also important to gather data throughout the using quantitative methods. In order to understand process. Ongoing data collection is important for two the impact of an intervention we must also gain an reasons. First, collecting data on the process after the understanding of how the intervention was perceived intervention project may result on retrospective by recipients. Through interviews and reviews of sensemaking (Weick et al., 2005). Participants may meeting minutes we can obtain information about the seek explanations of why or why not the intervention extent to which participants felt intervention activ- programme had a desired effect, e.g., that middle ities addressed the issues raised in the risk assessment. managers did not fulfil their role as change agents or The question ‘‘Why were intervention activities concurrent changes created conflicts. Second, ele- not implemented?’’ may also be answered at this ments of the process may change over time. For level. The answer to this question resides within how example, a project may be initiated for one purpose, participants perceived the intervention activities. This e.g., the legal requirement to conduct risk assess- involves asking questions such as: Were key stake- ments but may be sustained because participants see holders not supportive—and if so what were their the added value of the project. The role of key mental models of why the intervention presented a stakeholders may also change over time. For exam- threat? Were activities not perceived to be appro- ple, senior managers may be supportive of the project priate? How was the occupational health consultant at the beginning, but focus their attention on new received—did he or she show an understanding of the projects over time or lose interest if the project does culture and problems of the organization? All these not bring about changes at the speed expected. questions may help explain the actions of key Participation may also change over time. One of the stakeholders, e.g., a middle manager who initially core elements of participatory processes is empower- supported the project stopped doing so, once s/he ment: Over time employees develop the skills and realized that it may threaten her/his status within the start feeling responsible for creating a good working organization. Interviews may be particularly impor- environment. It is important to document any such tant to reveal the mental models of participants in changes, e.g., do they emerge during risk assessment, terms of how they evaluate the appropriateness and during action planning, or during implementation of quality of interventions. However, to get a broader activities? but less detailed picture from the wider population, Concurrent data collection throughout the project questionnaires may be used (cf. Nielsen et al., 2007, is time consuming for researchers and participants 2009). Focus groups may be a particular powerful alike. This means that measures need to be developed method for revealing the extent to which shared that are both easy to use and analyse. Although mental models have developed (Mohammed, Ferzan- observation of meetings and of work and interviews di, & Hamilton, 2010). are time consuming, reviewing meeting minutes and developing short questionnaires may be less so. The Changes in mental models of the job. To explore scales by Randall et al. (2009) are fairly lengthy; double-loop learning questionnaires can be used to however, it is also possible to construct short explore the degree to which different procedures have questionnaires and interviews schedules (e.g., using been introduced to ensure that decisions and telephone interviews) that can be used to collect data behaviours are not taken for granted but openly from key stakeholders (e.g., the union representative explored (Randall et al., 2009). In addition obser- and middle managers) on a monthly basis. Such data vations and interventions may help reveal changes in collection need not take much more than a few procedures, e.g., Nielsen, Randall, and Christensen minutes. Another possibility is to develop a short list (2010) found that leaders and employees changed of questions that managers and employees discuss at behaviours after a teamwork implementation in that regular meetings in the organization. Such a strategy 614 NIELSEN AND RANDALL also ensures that the project stays on the agenda in changes in subordinates’ working conditions and well- the organization and that organization members being. A limitation of all of these studies is that they all reflect on progress. Collecting short measures may focus on isolated specific elements of the process and not only be an advantage to the researchers, it may do not contain comprehensive process evaluation also be used as formative evaluation for the strategies like the one described in this article. organization as it provides the opportunity to reveal Some examples of more comprehensive ap- gaps in the current implementation process and take proaches to analysing process data have been corrective action (Patton, 2002). Using the Experi- reported. Murta et al. (2007) listed in their review a ence Sampling Method offers the opportunity to number of factors that may serve as an analytic measure daily experiences (Hektner, Schmidt, & framework. Analyses could be conducted where Csikszentmihalyi, 2007) and to examine how experi- recruitment, context, reach (attendance rates), dose ences vary on a daily basis, e.g., levels of participation delivered (intervention content), dose received (the and middle managers’ actions. In this way PE may extent to which participants apply activities), parti- provide managers with dashboard data that can be cipants’ attitudes, fidelity (whether activities were used to inform decisions about changes to the implemented as planned), and the link between intervention process. process and outcome should be analysed in a step- wise manner to detect at which level effects can be identified. Randall et al. (2007) provided a template Analysing process evaluation data for analysing process data. They suggested analysing Research including systematic process evaluation is processes and outcomes of intervention at three still in its infancy; however, there are a few studies levels: (1) microprocesses that concern the interven- which can be drawn upon when seeking inspiration on tion implementation: magnitude, valance, effect on how to analyse process evaluation data. It may be working conditions of participants and others; (2) considered optimal to integrate process evaluation macroprocesses describing the design, delivery, and data into effect evaluation to fully understand how maintenance of interventions; and (3) intervention processes have influenced intervention outcomes. To context analysing the importance of the context at test the effects of actual exposure, Randall et al. (2005) different levels from the context of the intervention, divided intervention participants into two groups: the department, the organization, the demands of the those who had been exposed to the intervention and sector, and the national level. those who had not. They found clear differences in exhaustion levels, concluding that those whom the CONCLUDING REMARKS intervention had reached reported better health postintervention. More sophisticated analyses may In this article we presented a framework for how the be conducted of the patterns of interaction between processes of interventions may be evaluated. It does process and outcomes. Using Structural Equation not present a complete model to cover all interven- Modelling, Nielsen et al. (2007) combined ‘‘objective’’ tions; rather, it is intended to provide a guiding process data (information received about the inter- framework from which elements may be selected for vention programme, and participation in intervention inclusion in evaluations of organizational-level occu- activities) and ‘‘appraisal’’ process data (quality of pational health interventions. There is a conflicting intervention activities) and linked these to intervention demand on evaluation research: It should be cost outcomes. Another study by Nielsen and Randall effective, ecologically valid, and practically impor- (2009) linked preexisting working conditions (as a tant, while at the same time be of scientific proxy for organizational maturity) to the role of the importance and generalizable (Bussing & Glaser, middle manager during the intervention and found 1999). We believe a model that prompts careful this role predicted intervention outcomes. There may consideration of the information that is necessary to be instances where process data are not easily evaluate the effectiveness of the intervention and integrated into effect analysis, e.g., when describing under which circumstances the results may be the impact of context. Aust et al. (2010), Biron et al. transferable to other contexts will help to develop (2010), and Nielsen et al. (2006) all present examples of our understanding of organizational-level occupa- how quantitative effect data and qualitative process tional health interventions. data may be combined in such circumstances. In a It may be worthwhile changing our opinion of study of middle manager training, Nielsen, Randall, what represents success in organizational-level occu- and Christensen (2010) outlined four levels at which pational health interventions. In line with Lipsey quantitative and qualitative data were integrated in (1996), we propose three indicators for success: (1) order to detect effects at four levels (satisfaction with changes in working conditions, employee health, and the intervention, changes in values and attitudes, well-being; (2) intermediate indicators that focus on changes in middle managers’ own behaviours, and the intervention activities implemented, whether PROCESS EVALUATION TOOL 615 Aust, B., Rugulies, R., Finken, A., & Jensen, C. (2010). When participants were aware of the intervention and its workplace interventions lead to negative effects: Learning from activities, whether participants held positive attitudes failures. Scandinavian Journal of Public Health, 38, 106–119. towards the intervention, whether participants devel- Bambra, C., Egan, M., Thomas, S., Petticrew, M., & Whitehead, oped new skills, and whether the intervention M. (2007). The psychosocial and health effects of workplace changed participants’ behaviour; and, finally, (3) reorganisation. 2. A systemic review of task restructuring interventions. Journal of Epidemiology and Community Health, indirect indicators that focus on whether the organi- 61, 1028–1037. zation changes their strategies and tools to deal with Biron, C., Gatrell, K., & Cooper, C. (2010) Autopsy of a failure: occupational health issues. Evaluating process and contextual issues in an organizational- PE should be carried out in both intervention and level work stress intervention. International Journal of Stress control groups to get an understanding of changes in Management, 17, 135–158. Bond, F. W., Flaxman, P. E., & Bunce, D. (2008). The influence of both. It is often the case that control groups and psychological flexibility on work redesign: Mediated modera- intervention groups interact with each other and this tion of a work reorganization intervention. Journal of Applied should be considered in PE (Nielsen et al., 2006). Psychology, 93, 645–654. Problems during the process may be lack of perfect Briner, R., & Reynolds, S. (1999). The costs, benefits, and blinding, and crossover between groups, and it is limitations of organizational level stress interventions. Journal of Organizational Behavior, 20, 647–664. important to understand any interaction between the Bunce, D., & West, M. (1996). Stress management and innovation two (Hurrell & Murphy, 1996). interventions. Human Relations, 49, 209–232. The focus is on process evaluation in this article, Burke, W. W., & Litwin, G. H. (1992). A causal model of but we do not mean to say that effect evaluation is organizational performance and change. Journal of Manage- not important; rather, these should supplement each ment, 18, 523–545. Bussing, A., & Glaser, J. (1999). Work stressors in nursing in the other (Nielsen, Taris, & Cox, 2010). Others have course of redesign: Implications for burnout and interactional discussed what should be assessed in effect evalua- stress. European Journal of Work and Organizational Psychol- tion, appropriate follow-up times and the amount of ogy, 8, 401–426. follow-ups necessary (Kompier et al., 1998, 2000; Cole, D. C., Wells, R. P., Frazer, M. B., Kerr, M. S., Neumann, W. Mikkelsen, 2005; Pettigrew, 1990; Taris & Kompier, P., Laing, A. C., & The Ergonomic Intervention Evaluation Research Group. (2003). Methodological issues in evaluating 2003): It is beyond the scope of this article to discuss workplace interventions to reduce work-related musculoskeletal these. disorders through mechanical exposure reduction. Scandinavian Although we discussed which methods may be Journal of Work and Environmental Health, 29, 396–405. used to collect data at the three levels, there is yet a Cooper, C., Dewe, P., & O’Driscoll, M. (2001). Organizational long way to determine which methods may best be interventions. In C. Cooper, P. Dewe, & M. O’Driscoll (Eds.), Organizational stress: A review and critique of theory, research, used to collect information at certain levels. One of the and applications. (pp. 187–232). Thousand Oaks, CA: Sage. problems mentioned in the introduction is the lack of Cummings, T. G., & Worley, C. G. (2009). Organization integration between process evaluation and effect development and change. Mason, OH: Cengage Learning. evaluation. Few studies have integrated quantitative Dahl-Jørgensen, C., & Saksvik, P. Ø. (2005). The impact of two process evaluation and effect evaluation (e.g., Bond, organizational interventions on the health of service sector workers. International Journal of Health Services, 35, 529–549. Flaxman, & Bunce, 2008; Nielsen et al., 2007, 2009) Egan, M., Bambra, C., Petticrew, M., & Whitehead, M. (2009). but this may be a viable way forward. Finally, we Reviewing evidence on complex social interventions: Apprais- would like to emphasize that we are not against quasi- ing implementation in systemic reviews of the health effects of experimental designs; rather, than these cannot stand organisational-level workplace interventions. Journal of Epide- alone in a simple effect evaluation. Quasi-experimental miology and Community Health, 63, 4–11. Egan, M., Bambra, C., Thomas, S., Petticrew, M., Whitehead, M., designs may control for biases that process evaluation & Thomson, H. (2007). The psychosocial and health effects of cannot do on their own; instead, we should try to workplace reorganisation. 2. A systematic review of task of integrate process evaluation measures in the strongest organisational-level interventions that aim to increase employee designs possible. control. Journal of Epidemiology and Community Health, 61, 945–954. Eklo¨ f, M., & Hagberg, M. (2006). Are simple feedback interven- tions involving workplace data associated with better working REFERENCES environment and health? A cluster randomized controlled study Argyris, C. (1976). Theories of action that inhibit individual among Swedish VDU workers. Applied Ergonomics, 32, 201–210. learning. The American Psychologist, 31, 638–654. Eklo¨ f, M., Hagberg, M., Toomingas, A., & Tornqvist, E. W. Argyris, C. (1995). Action science and organizational learning. (2004). Feedback of workplace data to individual workers, Journal of Managerial Psychology, 10, 20–26. work groups and supervisors as a way to stimulate working Armenakis, A., & Bedeian, A. (1999). Organizational change: A environment activity: A cluster randomized controlled study. review of theory and research in the 1990s. Journal of International Archives of Occupational and Environmental Management, 25, 293–315. Health, 77, 505–514. Armstrong, R., Waters, E., Moore, L., Riggs, E., Cuervo, L. G., Eklo¨ f, M., Ingelga˚ rd, A., & Hagberg, M. (2004). Is participative Lumbiganon, P., et al. (2008). Improving the reporting of ergonomics associated with better working environment and public health intervention research: Advancing TREND and health? A study among Swedish white-collar VDU users. CONSORT. Journal of Public Health, 19, 1–7. International Journal of Industrial Ergonomics, 34, 355–366. 616 NIELSEN AND RANDALL EU-OSHA (European Agency for Safety and Health at Work). LaMontagne, A. D., Keegel, T., Louie, A. M., Ostry, A., & (2010). European Survey of Enterprises on New and Emerging Landsbergis, P. A. (2007). A systematic review of the job- Risks, 2010. Retrieved from www.esener.eu stress intervention evaluation literature, 1990–2005. International European Network for Workplace Health Promotion. (2007). Journal of Occupational and Environmental Medicine, 13,268– Luxembourg Declaration on Workplace Health Promotion in the 280. European Union. Retrieved from http://www.enwhp.org/file Landsbergis, P., & Vivona-Vaughan, E. (1995). Evaluation of an admin/rs-dokumente/dateien/Luxembourg_Declaration.pdf occupational stress intervention in a public agency. Journal of Fredslund, H., & Strandgaard, J. (2005). Methods for process Organizational Behavior, 16, 29–48. evaluation of work environment interventions. In J. Houdmont Lindquist, T. L., & Cooper, C. L. (1999). Using life style and & S. McIntyre (Eds.), Occupational health psychology: Key coping to reduce job stress and improve health in ‘‘at risk’’ papers of the European Academy of Occupational Health office workers. Stress Medicine, 143–152. Psychology (pp. 109–117). Oporto, Portugal: ISMAI Publishers. Lindstro¨ m, K. (1995). Finnish research in organizational develop- Giga, S., Noblet, A. L., Faragher, B., & Cooper, C. L. (2003). The ment and job redesign. In L. Murphy, J. Hurrell, S. Sauter, & UK perspective: A review of research on organisational stress G. Keita (Eds.), Job stress interventions (pp. 283–293). management interventions. The Australian Psychologist, 38, Washington, DC: American Psychological Association. 156–164. Lines, R. (2004). Influence of participation in strategy change: Goldenhar, L., LaMontagne, A., Katz, T., Heaney, C., & resistance, organizational commitment and change goal Landsbergis, P. (2001). The intervention research process in achievement. Journal of Change Management, 4, 193–215. occupational safety and health: An overview from the National Lipsey, M. (1996). Key issues in intervention research: A program Occupational Research Agenda Intervention Effectiveness evaluation perspective. American Journal of Industrial Medicine, Research Team. Journal of Occupational and Environmental 29, 298–302. Medicine, 43, 616–622. Lipsey, M., & Cordray, D. (2000). Evaluation methods for social Greene, J. C., Benjamin, L., & Goodyear, L. (2007). The merits of intervention. Annual Review of Psychology, 51, 345–375. mixed methods in evaluation. Evaluation, 7, 25–44. Mathieu, J. E., Heffner, T. S., Goodwin, G. F., Salas, E., & Guastello, S. (1993). Do we really know how well our occupational Cannon-Bowers, J. A. (2000). The influence of shared mental accident prevention programs work? Safety Science, 16, 445–463. models on team process and performance. Journal of Applied Guth, W. D., & Macmillan, I. C. (1986). Strategy implementation Psychology, 85, 273–283. versus middle manager self-interest. Strategic Management Mikkelsen, A. (2005). Methodological challenges in the study of Journal, 7, 313–327. organizational interventions in flexible organizations. In A. M. Harachi, T., Abbott, R., Catalan, R., Haggerty, K., & Fleming, C. Fuglseth & I. A. Kleppe (Eds.), Anthology for Kjell Grønhaug in (1999). Opening the black box: Using process evaluation celebration of his 70th birthday (pp. 150–178). Bergen, Norway: measures to assess implementation and theory building. Fagbokforlaget. American Journal of Community Psychology, 27, 711–731. Mikkelsen, A., & Saksvik, P. Ø. (1999). Impact of a participatory Harris, C., Daniels, K., & Briner, R.B. (2002).Using cognitive organizational intervention on job characteristics and job mapping for psychosocial risk assessment. Risk Management: stress. International Journal of Health Services, 29, 871–893. An International Journal, 4, 7–21. Mohammed, S., Ferzandi, L., & Hamilton, K. (2010). Metaphor Heaney, C., Israel, B., Schurman, S., Baker, E., House, J., & no more: A 15-year review of the team mental model construct. Hugentobler, M. (1993). Industrial relations, worksite stress Journal of Management, 36, 876–910. reduction, and employee well-being: A participatory action Murta, S. G., Sanderson, K., & Oldenburg, B. (2007). Process research investigation. Journal of Organizational Behavior, 14, evaluation in occupational stress management programs: A 495–510. systematic review. American Journal of Health Promotion, 21, Hektner, J., Schmidt, J. A., & Csikszentmihalyi, M. (2007). 248–254. Experience sampling method: Measuring the quality of everyday Nastasi, B. K., Hitchcock, J., Sarkar, S., Burkholder, G., Varjas, life. Thousand Oaks, CA: Sage. K., & Jayasena, A. (2007). Mixed methods in intervention Hugentobler, M., Israel, B., & Schurman, S. (1992). An action research: Theory to adaptation. Journal of Mixed Methods research approach to workplace health: Integrating methods. Research, 1, 164–182. Health Education Quarterly, 19, 55–76. Nielsen, K., Cox, T., & Griffiths, A. (2002). Developing interven- Hurrell, J. (2005). Organizational stress intervention. In J. Barling, tions for risk management: The Action Innovation Process. In E. K. Kelloway, & M. R. Frone (Eds.), Handbook of work stress C. Weikert, E. Torkelson, & J. Pryce (Eds.), Occupational health (pp. 623–645). Thousand Oaks, CA: Sage. psychology: Empowerment, participation and health at work (pp. Hurrell, J., & Murphy, L. (1996). Occupational stress interventions. 135–140). Vienna, Austria: Allgemeine Unvallsversicherung- American Journal of Industrial Medicine, 29, 338–341. sanstalt (AUVA). Jimmieson, N. L., Terry, D. J., & Callan, V. J. (2004). A Nielsen, K., Fredslund, H., Christensen, K. B., & Albertsen, K. longitudinal study of employee adaptation to organizational (2006). Success or failure? Interpreting and understanding the change: The role of change-related information and change- impact of interventions in four similar worksites. Work and related self-efficacy. Journal of Occupational Health Psychology, Stress, 20, 272–287. 9, 11–27. Nielsen, K., & Randall, R. (2009). Managers’ active support when Johns, G. (2006). The essential impact of context on organizational implementing teams: The impact on employee well-being. behavior. Academy of Management Review, 31, 386–408. Applied Psychology: Health and Well-being, 1, 374–390. Johnson, R. B., Onwuegbuzie, A. J., & Turner, L. A. (2007). Nielsen, K., Randall, R., & Albertsen, K. (2007). Participants’ Toward a definition of mixed methods research. Journal of appraisals of process issues and the effects of stress manage- Mixed Methods Research, 1, 112–133. ment interventions. Journal of Organizational Behavior, 28, Kompier, M., Cooper, C., & Geurts, S. (2000). A multiple case 793–810. study approach to work stress prevention in Europe. European Nielsen, K., Randall, R., Brenner, S.-O., & Albertsen, K. (2009). Journal of Work and Organizational Psychology, 9, 371–400. Developing a framework for the ‘‘why’’ in change outcomes: Kompier, M., Geurts, S., Grundemann, R., Vink, P., & Smulders, P. The importance of employees’ appraisal of changes. In P. Ø. (1998). Cases in stress prevention: The success of a participative Saksvik (Ed.), Prerequisites for healthy organizational change and stepwise approach. Stress Medicine, 14, 155–168. (pp. 76–86). Bussum, The Netherlands: Bentham Publishers. PROCESS EVALUATION TOOL 617 Nielsen, K., Randall, R., & Christensen, K. B. (2010). Does Rychetnik, L., Frommer, M., & Shiell, A. (2002). Criteria for training managers enhance the effects of implementing team- evaluating evidence on public health. Journal of Epidemiology working? A longitudinal, mixed methods field study. Human and Community Health, 56, 119–127. Relations, 11, 1719–1741. Saksvik, P. Ø., Nytrø, K., Dahl-Jørgensen, C., & Mikkelsen, A. Nielsen, K., Randall, R., Holten, A. L., & Rial Gonza´ lez, E. (2002). A process evaluation of individual and organizational (2010). Conducting organizational-level occupational health occupational stress and health interventions. Work and Stress, interventions: What works? Work and Stress, 24, 234–259. 16, 37–57. Nielsen, K., Taris, T. W., & Cox, T. (2010). The future of Schurman, S., & Israel, B. (1995). Redesigning work systems to organizational interventions: Addressing the challenges of reduce stress: A participatory action research approach to today’s organizations. Work and Stress, 24, 219–233. creating change. In L. Murphy, J. Hurrell, S. Sauter, & G. Noblet, A. J., & LaMontagne, A. D. (2009). The challenges of Keita (Eds.), Job stress interventions (pp. 235–263). Washing- developing, implementing, and evaluating interventions. In S. ton, DC: American Psychological Association. Cartwright & C. L. Cooper (Eds.), The Oxford handbook of Semmer, N. K. (2003). Job stress interventions and organization of organizational well-being (pp. 466–496). Oxford, UK: Oxford work. In L. Tetrick & J. C. Quick (Eds.), Handbook of University Press. occupational health psychology (pp. 325–353). Washington, DC: Nytrø, K., Saksvik, P. Ø., Mikkelsen, A., Bohle, P., & Quinlan, M. American Psychological Association. (2000). An appraisal of key factors in the implementation of Semmer, N. K. (2006). Job stress interventions and the organiza- occupational stress interventions. Work and Stress, 14, 213–225. tion of work. Scandinavian Journal of Work and Environmental Øyum, L., Kvernberg Andersen, T., Pettersen Buvik, M., Health, 32, 515–527. Knutstad, G. A., & Skarholt, K. (2006). God ledelsespraksis i Semmer, N. K. (2011). Job stress interventions and organization of endringsprocesser: Eksempler pa hvordan ledere har gjort til en work. In J. C. Quick & L. E. Tetrick (Eds.), Handbook of positiv erfaring for de ansatte [Good management practice during occupational health psychology (2nd ed., pp. 299–318). Wa- change processes: Examples of how managers have made it a shington, DC: American Psychological Association. positive experience for employees] (Rep. No. 567). København, Shannon, H., Robson, L., & Guastello, S. (1999). Methodological Denmark: Nordisk Ministerra˚ d. criteria for evaluating occupational safety intervention re- Patton, M. Q. (2002). Qualitative research and evaluation methods. search. Safety Science, 31, 161–179. Thousand Oaks, CA: Sage. Shannon, H. S., & Cole, D. C. (2004). Commentary III making Pettigrew, A. M. (1990). Longitudinal field research on change: workplaces healthier: generating better evidence on work Theory and practice. Organization Science, 1, 267–292. organization intervention research. Sozial Praventivmedizin, Randall, R., Cox, T., & Griffiths, A. (2007). Participants’ accounts 49, 92–94. of a stress management intervention. Human Relations, 60, Steckler, A. B., & Linnan, L. (2002). Process evaluation for public 1181–1209. health interventions and research. San Francisco, CA: Jossey- Randall, R., Griffiths, A., & Cox, T. (2005). Evaluating organiza- Bass. tional stress-management interventions using adapted study Taris, T., Kompier, M., Geurts, S., Schreurs, P., Shaufeli, P., de designs. European Journal of Work and Organizational Psychol- Boer, E. et al. (2003). Stress management interventions in the ogy, 14, 23–41. Dutch domiciliary care sector: Findings from 81 organisations. Randall, R., Nielsen, K., & Tvedt, S. D. (2009). The development International Journal of Stress Management, 10, 297–325. of 5 scales to measure participants’ appraisal of organizational- Taris, T. W., & Kompier, M. (2003). Challenges in longitudinal level stress management interventions. Work and Stress, 23,1–23. designs in occupational health psychology. Scandinavian Richardson, K. M., & Rothstein, H. R. (2008). Effects of Journal of Work, Environment and Health, 29, 1–4. occupational stress management intervention programs: A meta- Vedung, E. (2006). Process evaluation and implementation theory. analysis. Journal of Occupational Health Psychology, 13, 69–93. In E. Vedung (Ed.), Public policy and program evaluation (pp. Roen, K., Arai, L., Roberts, H., & Popay, J. (2006). Extending 209–245). Piscataway, NJ: Transaction Publishers. systematic reviews to include evidence on implementation: Weick, K. E., Sufcliffe, K. M., & Obstfeld, D. (2005). Organizing Methodological work on a review of community-based and the process of sensemaking. Organization Science, 16, 409– initiative to prevent injuries. Social Science and Medicine, 63, 421. 1060–1071. Weiner, B. J., Amick, H., & Lee, S.-Y. D. (2008). Conceptualiza- Rossi, P. H., Lipsey, R. W., & Freeman, H. E. (2004). Evaluation: tion and measurement of organizational readiness for change: A systematic approach. London, UK: Sage. A review of the literature in health services research and other Rousseau, D. M., & Fried, Y. (2001). Location, location, location: fields. Medical Care Research and Review, 65, 379–436. Contextualizing organizational research. Journal of Organiza- Yin, R. K. (1994). Case study research: Design and methods (2nd tional Behavior, 22, 1–13. ed.). Thousand Oaks, CA: Sage. Ruotsalainen, J. H., Verbeek, J. H., Salmi, J. A., Jauhiainen, M., Persternack, I., & Husman, K. (2006). Evidence on the Original manuscript received February 2011 effectiveness of occupational health interventions. American Revised manuscript received February 2012 Journal of Industrial Medicine, 49, 865–872. First published online July 2012

Journal

European Journal of Work and Organizational PsychologyTaylor & Francis

Published: Oct 1, 2013

Keywords: Methods; Organizational-level occupational health interventions; Process evaluation

There are no references for this article.