TY - JOUR
AU - Broerse, Jacqueline E, W
AB - Abstract Although academic interest for patient involvement in health research decision-making is growing, in practice it proves challenging to involve patients meaningfully and sustainably. This article aims to unravel systemic factors influencing the embedding of meaningful patients’ involvement in proposal appraisal at a private cancer research funding organization. In the case study, the emergent and participatory research approach Reflexive Monitoring in Action was applied. At first, a positive stance of stakeholders towards patient involvement was accompanied by a lack of tools and structures. This lack of tools led to the execution of various interventions. Analysis reveals that structural interventions succeeded in the bureaucratic context. The required cultural shift was also addressed, but took longer to materialize. This may have reduced the meaningful involvement of patients in proposal appraisal in practice. To overcome the underlying systemic barriers, an approach is needed that encourages stakeholders’ reflexivity in the structural, cultural, and practical domain. 1. Patient involvement in research proposal appraisal The responsiveness of Western health research systems has been increasingly scrutinized in recent years: real-life problems in health practices and the patients’ needs are insufficiently addressed by newly developed innovations or knowledge, and the system is criticized for being supply-driven rather than demand-based (Chalmers 1995; Tallon et al. 2000; Elberse et al. 2017). Research tends to be ‘directed towards function enhancement rather than function recovery or prevention, it is disease-oriented rather than focused on integral approaches, and it is intervention-oriented rather than supporting individual responsibility or addressing implementation problems’ (Broerse et al. 2010). The limited set of stakeholders currently dominating decision-making in each step of the research process, comprising researchers, policy makers, and medical specialists can be associated with the supply-driven nature of health research systems (Broerse et al. 2010). In this article, we focus on the stakeholders involved in decision-making during proposal appraisal. Authors have argued that including (ex)patients in the process of reviewing grant proposals can increase the societal relevance of the research performed by using the complementary nature of experiential knowledge (Popay and Williams 1996; Entwistle et al. 1998; Caron-Flinterman et al. 2005). Furthermore, the call for more patient involvement coincides with an increased societal demand for public accountability of research and better translation of research outcomes to improvements in care practice (Entwistle et al. 1998; Gibbons 1999; Nowotny et al. 2001; Whitstock 2003; Abma and Broerse 2010). Some health funds have initiated patient involvement in their proposal appraisal procedures (Andejeski et al. 2002; Teunissen et al. 2013; Elberse et al. 2017). Notable pioneers in this field in the UK are the Alzheimer’s Society which has set up a Research Network with patient reviewers, and the National Institute of Health Research which includes lay reviewers on many of its funding panels (Alzheimers_Society 2018; NIHR 2018). However, with the exception of relatively recent academic interest into this topic little is known about the impact and embedding of such reviewing involvement in health research decision-making (O’Donnell and Entwistle 2004; Staley and Doherty 2016; Elberse et al. 2017). Scholars have investigated the range of issues which may be considered by patient reviewers in proposal appraisal (Saunders et al. 2007; Teunissen et al. 2013). These studies have demonstrated that patient reviewers can examine the information provided to participants on aspects such as comprehensibility and probity. Also, patients’ experiential knowledge allows them to identify prospective clinical subjects by ‘walking in their shoes’, hence verifying the feasibility of including patients with certain health conditions and the burden of the proposed research (Maguire and Britten 2017). This demonstrates the value of including patients in the process of knowledge co-production, in which knowledge is developed between science and society (Regeer and Bunders 2009). Here, patients’ experiential knowledge is articulated and integrated with researchers and healthcare professionals’ expertise (Pittens 2013). 2. Meaningful and sustainable patient involvement Despite its argued and demonstrated benefits in academic literature, in practice, patient involvement in reviewing grant proposals is currently often tokenistic and largely confined to isolated initiatives. The observation that patients can potentially contribute meaningfully to knowledge co-production does not imply that this involvement is by definition meaningful in every initiative (Rowe and Frewer 2000). For example, while the Innovative Medicine Initiative claimed that the majority of their first EU-funded projects involved patients in research, the patients were actually used as subjects to be studied rather than having an influence on the research itself (Supple et al. 2015). de Wit et al. (2015) demonstrated that the explicit acknowledgement of patients as a valuable source of knowledge is essential to avoid tokenism, and, similarly, Li et al. (2015) describe a tension between the genuine desire to involve the public in health policy decision-making and more strategic and political reasons to support patient involvement. Following the Li et al. study that describes the influence of public involvement in health policy decision-making, we define meaningful patient involvement in the reviewing process as a contribution which is based on the patients’ experiential knowledge and which is taken into serious consideration during deliberations and may affect the funding decision. In addition to the danger of tokenism, the organizational embedding of the patient perspective proves challenging. Although successful initiatives are being replicated on the individual level, Elberse (2012) argues that patient involvement is not yet ‘anchored in the system’, and Boaz et al. (2016) provide evidence that health researchers maintain ‘Mode 1 academic attitudes and values, despite the changing currents on the surface’. This implies that the health research system is currently not configured to meaningfully incorporate the patients’ perspective in proposal appraisal processes, and that it encounters systemic barriers to change this. A system innovation is therefore needed to overcome these barriers (Grin et al. 2010; Schuitmaker 2012; Broerse and Grin 2017). To include patients in decision-making procedures, involved scientists, policy-staff of a health foundation, and patients themselves need to change their perceptions and values regarding proposal appraisal (culture), the organizational rules, regulations, and procedures facilitating the proposal appraisal process (structure), and the actors’ means for performing the proposal appraisal (practice) (van Raak 2010). Moreover, the actors functioning within a dominant regime tend to reproduce the existing system, thereby counteracting innovations to the system (Schuitmaker 2012). These Lamarckian notions aid in explaining a regime’s resistance to change (Grin et al. 2010) and why such changes are typically gradual and continuous processes (Rotmans et al. 2001). Consider the cultural change accompanying patients’ involvement in reviewing grant proposals. The system’s criteria for good research, i.e. worthy of funding, need to be reconsidered (Andejeski et al. 2002; Fleurence et al. 2014). In addition to scientists focusing on the scientific relevance, innovativeness, and methodological quality of research proposals, the patients’ judgments of the proposal’s clinical relevance and feasibility need to be incorporated in the final assessment of the grant proposal. This means that the assessment whether the proposed research addresses a problem deemed relevant by patients, and whether the expected benefits of the solution for patients outweigh the burden of the research for participants, gains importance. Secondly, the existing structure of the grant proposal reviewing process comprises a set of routines with a clear balance of power, which structure daily activities, communication, and decision-making amongst all those involved (Frenk 1992). Incorporating the patients’ perspectives in this system tends to destabilize the system, as it requires adjustments to established procedures and routines to accommodate the patients’ reviewing process needs. Thirdly, involving patients in the decision-making process demands the establishment of long-term relationships between different stakeholders, extending people’s competences and requiring the availability of time, money, and resources (Gray et al. 2000; Howe et al. 2006; Lindenmeyer et al. 2007). For example, when patient reviewers are invited to join the decision-advising committee, scientific experts have to adjust their discourse to allow non-scientific experts to join their conversation (Elberse et al. 2011). The discrepancy between the promised benefits of patient involvement in proposal appraisal and the current lack of its structural embedding in the health research system calls for an analysis of this situation from a system’s perspective. An analysis of the systemic factors facilitating or impeding meaningful patient involvement in health research proposal appraisal may offer recommendations on how to enhance the meaningful and sustainable involvement of patients in proposal appraisal. 3. Case study We conducted a case study on the involvement of patients reviewing grant proposals at the Dutch Cancer Society (DCS), the largest privately funded health foundation of the Netherlands. With an annual budget of approximately €120 m funding approximately half of all project-based oncology research conducted in the Netherlands, DCS is a key player in the Dutch oncological health research system (Hanselaar and Boerrigter 2011; KWF Kankerbestrijding, 2015). The organization’s mission goals are as follows: more cures, less prevalence, and a better life quality for people with cancer. DCS aims to achieve these goals by increasing the impact of its funded research on the patients’ prospects and quality of life, and by increasing the role of patients in its operating procedures (Kankerbestrijding 2015). In 2011, DCS installed a Patient Advisory Committee (PAC). This committee originally consisted of ∼25 (ex)cancer patients who reviewed clinical study proposals submitted to DCS, using criteria they deemed relevant. Between 2015 and 2017, DCS reorganized its overall process of proposal appraisal in the Program Research and Implementation (PR&I). As part of this reform, patient involvement in proposal appraisal was anchored by formally installing patient reviewers in the Scientific Council (SC) of the organization in 2016. This study commenced in 2014 when DCS invited an external research organization to evaluate the PAC’s activities and to analyze the factors impeding and facilitating the embedding of the PAC at DCS. The study covered a critical phase of the changes taking place at DCS, in which patient involvement in proposal appraisal gained momentum but was not yet consolidated. Taking the goals of DCS regarding patient involvement into account, we formulated the following research question; How can the embedding of meaningful patient involvement in proposal appraisal at DCS be enhanced, considering the facilitating and impeding systemic factors influencing such a development? 4. Methodology The study adopted the emergent, action-oriented, and participatory research approach Reflexive Monitoring in Action (RMA) (van Mierlo et al. 2010). In a system innovation, stakeholders jointly need to change the structure, culture, and practice through recurrent reflection on these domains. RMA stimulates this reflection; it aids actors in the process of shaping and adapting to the new reality. An independent observer monitors and evaluates the innovation, initiating and supporting interventions when needed. The interventions in RMA can be characterized by the action-learning spiral, emphasizing reflexivity by: (1) identifying a challenge, (2) developing an action plan, (3) performing the intervention, (4) observing and evaluating the intervention, (5) adjusting the action plan with regard to the new insights, and (6) performing the adjusted intervention (Hanselaar and Boerrigter 2011). This method aids in identifying facilitating factors and areas for improvement. The insights can be readily translated into recommendations, allowing stakeholders to apply them directly. RMA thus integrates reflexivity in the process of change (van Mierlo et al. 2010). 5. Three-phased reflection and action study Data were collected between July 2014 and December 2016 in three largely consecutive phases; see Fig. 1 for a chronological overview. Figure 1. View largeDownload slide Chronological overview of the three-phase reflection and action study, in which phase 1 is medium gray (purple), phase 2 is dark gray (dark blue), and phase 3 is light gray (aquamarine). Key decisions made by DCS are above the timeline (in green). Colour version of this figure is available online at SCIPOL Journal. Figure 1. View largeDownload slide Chronological overview of the three-phase reflection and action study, in which phase 1 is medium gray (purple), phase 2 is dark gray (dark blue), and phase 3 is light gray (aquamarine). Key decisions made by DCS are above the timeline (in green). Colour version of this figure is available online at SCIPOL Journal. A project team consisted of three policy staff members of DCS, all specifically dedicated to enhancing patient involvement at DCS, and three independent academic researchers (one was replaced halfway through the study). One of the academic researchers performed most of the monitoring activities; she is henceforth called ‘the monitor’ (author A.S.). She was working at the DCS office on average two days a week, giving the study an ethnographic character. The other two academic researchers assisted in the design, data collection, and analysis of and reflection on the data collected. The project team regularly jointly evaluated the progress and findings of the study. 5.1 Phase 1: insights in early patient involvement experiences (2011–2015) To initiate the action-learning spiral, the initial culture, structure, and practice prevalent at the organization were analyzed as follows:(a) Fifteen semi-structured interviews were conducted by the monitor in July 2014 with patient reviewers (n = 7), policy staff at DCS (n =  5) and researchers in the SC (n  = 3). Interviewees discussed experiences in or with the PAC and their expectations regarding the PAC. Two members of the executive board of DCS were interviewed to provide insight into DCS’ management position on patient involvement.(b) In 2014, patient reviewers assessed grant proposals in six research-funding programs. In 2015, PAC’s involvement continued in three programs. The monitor attended seven assessment meetings through participant observation and informally evaluated with patient reviewers (n = 4), DCS staff (n = 3) and SC chairs (n = 1) in interviews lasting about 30 minutes. 5.2 Phase 2: interventions in patient involvement (2015) Based on the insights gained in the first phase of this study, we aimed at supporting sustainable change in the culture and structure of and practice at DCS. Therefore, we devised an intervention plan containing four elements: (a) The process of patient involvement in proposal appraisal was standardized according to a process scheme. In four workshops facilitated by the monitor and an additional researcher, one or two issues in the process scheme were discussed. Three to five patient reviewers attended each workshop, and nine participants attended between one and all four workshops. (b) The recruitment process for new PAC members was co-designed with current patient reviewers (in collaboration with all staff members of the project team and the monitor). A recruitment profile was published, candidates attended an introductory meeting, and new patient reviewers followed a training session. A staff member of the project team and the monitor jointly evaluated the recruitment process with all new members (n = 16) by phone. (c) In a workshop (organized by a staff member of the project team and facilitated by the monitor and an additional researcher), DCS research coordinators discussed their own role, required skills, and competences to involve patient reviewers in proposal appraisal as part of the PR&I reform. (d) All PAC-involved grant applications tendered in 2015 were analyzed by the monitor. Assessments by PAC and the SC were classified as positive, mediocre, negative, or no assessment/revision. The funding advice to DCS was either positive or negative. The initial scoring given by the PAC and the SC and the final advice to DCS were compared to each other. 5.3 Phase 3: analysis of patient involvement process (2016) As part of the action-learning spiral, the interventions were continuously evaluated in numerous informal meetings and discussions. Additionally, two formal evaluations were performed: (a) Nine semi-structured interviews with patient reviewers (n = 4) and DCS-staff members (n = 5) were conducted by the monitor in November–December 2016. Five patient reviewers and one DCS-staff member had been interviewed at the start of the study. Interviewees reflected on the past 2.5 years regarding patient involvement in proposal appraisal at DCS. (b) The monitor attended an evaluative meeting in June and in December 2016 with patient reviewers who had represented the PAC at SC meetings. During the meetings, patient reviewers evaluated their experiences with the SC so far and discussed the future plans for the PAC. 5.4 Data analysis The monitor recorded all formal interviews after verbal consent, transcribed the interviews verbatim, and summarized them. Minutes were made of all formal meetings and workshops that were facilitated by the monitor and/or an additional researcher. The summaries of the interviews and minutes of the meetings were sent to the participants for member-check within two weeks after the event. At all other meetings where the monitor was present and during all informal (phone) interviews conducted by the monitor and/or other academic researchers, observations were recorded in a logbook. Also, the monitor took field notes at the office of DCS. The action-oriented and emergent nature of the research design called for an iterative data analysis. Preliminary findings were regularly discussed with the entire project team to guide further interventions. In-depth analysis was based on thematic coding (Green and Thorogood 2004), performed by the first and second author on the logbook, transcripts, and summaries to identify systemic barriers and facilitators on culture, structure, and practice regarding meaningful and sustainable patient involvement. The themes were identified following the definition of culture (set of values, perceptions, and interpretative frames shared by most of the involved actors), structure (physical, legal, organizational, power, economic, and financial structures facilitating and/or constraining the behaviour of the involved actors) and practice (actual actions (operations) undertaken by the involved actors) as described by van Raak (2010). Next, open coding was performed by the first and second author to uncover patterns within the themes. The academic researchers regularly discussed the coding process. During these discussions, three overarching issues were identified in which the planned and performed interventions could be classified: Structural adjustments to the PAC-assessment criteria Formalization of PAC’s position Exploitation of patient reviewers’ experiential knowledge The monitor performed the document analysis of the grant applications which was added to this qualitative dataset. The analysis was subsequently reviewed with the staff members of the project team, to discuss the interpretation of the data. Differences in interpretation were extensively discussed until agreement was achieved. 5.5 Ethical considerations The ethical aspects of this research were considered an ongoing process of deliberation and reflection. All participants voluntarily, anonymously, and confidentially took part in the research activities; due to the emergent nature of the research design, we regularly re-evaluated their verbal informed consent. Also, by means of member checks we continuously scrutinized our own interpretation of the data, reviewing our potential preconceptions. We adhered to the nationally established code of ethics for research in the social and behavioural sciences involving human participants, as interpreted and explicated by the involved research institution (Scientific and Ethical Review Board 2016). The involvement of patient reviewers did not concern medical treatment or invasion of their integrity, hence approval of an accredited Dutch medical research ethics committee was not needed. 6. Results In this section, the initial culture, structure, and practice at DCS are described based on the data collected in phase 1 of the study. Subsequently, the performed interventions and stakeholders’ perspectives on the impact of the interventions are described based on the data collected in the second and third phase of this study. 6.1 Contextual and retrospective insights in early patient involvement experiences (2011–2015) Phase 1 of this study elicited the initial culture, structure, and practice at DCS through analyzing the positions and viewpoints of the main stakeholders involved and the organizational context of DCS. DCS staff members characterize their organization with a propensity to formulate regulations, procedures, and protocols. The relatively large amount of resources available enhances the organization’s societal visibility and increases the need for transparency and public accountability of its resource allocation. This fuels the need to legitimize the organization’s expenditure through standardized proposal appraisal procedures. The policies needed to involve patients are designed and implemented by the Team Patient Involvement (team PI) at DCS. The monitor was considered an external member of this team for the duration of the study. Supervision of team PI repeatedly shifted from an executive board member of DCS to lower ranks of team leaders. This is indicative not only of the broad scope of the team but also of its ‘outsider’ position in the organization. Emblematic was the team’s self-imposed characterization as gadfly of DCS—team PI members repeatedly reminded their colleagues of the value to involve patients meaningfully in proposal appraisal. This indicates that team PI members felt the need for a cultural shift within the organization, as the rest of the DCS-staff was less aware of this issue. Patient reviewers on their own behalf take a seat in the PAC and all share the experience of having been diagnosed with cancer. The group is relatively demographically diverse in age, gender, and the type, severity, and treatment of their tumours; however, higher-educated patients with Dutch ethnicity are overrepresented. DCS supported research includes all cancer types with divergent survival rates and treatment options and includes fundamental science to psycho-social support. This multi-levelled diversity complicates the comparisons and tradeoffs made in assessing grant proposals. Hence, interviewed stakeholders (including patient reviewers) raise questions about the legitimacy of patient reviewers: Are patients eligible to assess research proposals on cancer types they have not been confronted with? Or contrarily, do they favour proposals investigating their type of cancer? To what extent can they assess the value of preventive research, as they are already confronted with cancer? Some patient reviewers are especially knowledgeable about the type of cancer they have had themselves; they alert researchers to new collaborations within the field, or they question the scientific innovativeness of a research proposal. While some patient reviewers consider it their duty to have considerable state-of-the-art knowledge on oncological research, for example, by staying up-to-date in scientific literature, others disagree. We need more people [patient reviewers] with technological expertise. For example, the diagnostic tools. The technological developments are going so fast and we know too little about it. As a patient reviewer you need to keep abreast of such developments But shouldn’t I just know what it feels like to be lying in such a scanner? Instead of knowing how it works? (patient reviewers in evaluative meeting of PAC representatives) In addition to the experiential knowledge, patients also possess knowledge from (former) occupational experience. Patient reviewers do not always consciously separate this when assessing research proposals; they sometimes evaluate the feasibility or quality of a research proposal based on their professional knowledge. For example, when members of the SC expressed their doubts on scientific grounds, a patient reviewer defended a proposal on immunotherapy. In the evaluation, the patient reviewer acknowledged that her professional background as immunologist contributed to her defence. Leading scientists in the Dutch oncological research field are gathered in the SC to provide DCS advice regarding the funding of grant proposals. Researchers take seat for a period of 3 years, which can be extended once. The SC is divided in sub-committees that peer-review grant proposals on topics close to the members’ research fields. Each committee is chaired by a prominent researcher and supported by a research coordinator of DCS. The interviewed SC-chairs cherish the contribution of the patient reviewers, but some struggle how to take the PAC-assessment into consideration in the funding advice, as they do not want their scientific assessment to be influenced by the assessment of the patient reviewers. 6.2 Interventions in patient involvement in proposal appraisal (2015–2016) To enhance the sustainable embedding of meaningful patient involvement in proposal appraisal, several interventions were designed and performed based on the first phase of this study that elicited the initial culture, structure, and practice at DCS. The stakeholders indicated the need for adjustments during interviews or assessment meetings, or the monitor initiated activities. Improvement of patient involvement performance (practice) was often addressed by attending to the structural domain. While interventions aimed at addressing underlying cultural differences between the stakeholders, the effects of these efforts were less promptly visible. Three issues illustrate this claim. First, the issue is described, after which the intervention is presented and analyzed. 6.2.1 Issue 1: structural adjustments to improve interpretation and weighting the PAC-assessment criteria In the first phase of this study, patient reviewers and DCS-staff frequently mentioned the lack of and need for consensus regarding the interpretation and weighting of PAC-assessment criteria. The PAC was asked to assess: (1) the quality and readability of the Dutch summary, (2) societal relevance of the research, (3) burden and risks to participants compared to the expected health benefits, (4) involvement of patients, and (5) the quality and readability of the patient information for prospective clinical participants. The scoring sheet did not weigh the importance of each criterion or explicate its interpretation. The consequential variation and discussion amongst patient reviewers were observed in the assessment meetings. For example, ‘relevance’ was interpreted differently. People debated whether research that promises small progress is ‘messing in the margins’ or whether it can be a stepping stone for substantial health gains in the long run. Neither was there consensus whether the size of a patient group should be considered in assessing relevance. It’s about only 300 patients. Should that be an argument? I feel we should impede this. It’s about the individual patient. I disagree. Looking at the matter from a business perspective, I’ll go for a treatment with which you can save thousands of patients. (patient reviewers in PAC-assessment meeting) Regarding the weighting of the criteria, some assessors prioritized the proposal’s relevance. Another patient reviewer took serious account of the burden and hardly considered other aspects. If the research uses existing biomaterial, the burden for the patient is zero. Then I say, it’s a perfect study. And I hardly consider the rest. (patient reviewer in PAC-assessment meeting) During the PAC-assessment meetings, patient reviewers always attempted to reach consensus regarding their assessment of the proposal or—in exceptional cases of disagreement—to clearly denote the different lines of argumentation within the PAC during the SC meeting. To discuss the issue of interpretation of assessment criteria in more depth and generalized beyond specific proposals, the project team organized a workshop at which four patient reviewers were present. The assessment form for the PAC was adjusted based on this workshop and in dialogue with DCS-staff; guiding questions were added to help explicate the criteria. The form separated the readability and quality of the Dutch summary from the main assessment categories. The feasibility, quality, relevance, and the involvement of patients in the proposal were taken into account in coming to a final assessment. Additionally, all proposals were discussed at a PAC-assessment meeting facilitated by team PI (see issue 2). During those meetings, sometimes observations beyond the scope of the assessment form were discussed; for example, the dissemination of the prospective results or the interests of the pharmaceutical industry compared to DCS’ mission goals in the proposals. In the evaluation phase, most patient reviewers and DCS-staff agreed that the adjusted assessment forms and the facilitated PAC-assessment meeting resulted in more consistent, better structured, and more thoroughly substantiated arguments. Also, interviewees claimed there are fewer opportunities for patient reviewers to incorporate their favourite topics in the assessments. 6.2.2 Issue 2: formalization of PAC’s position and its influence on funding advice Based on the analysis in phase 1, we detected a need to consolidate the position of the PAC in the organization. This was achieved by mainly interventions on the structural domain. At the start of this study, the position of the PAC within DCS was not formalized and its activities were conducted in an exploratory manner. Patient reviewers and DCS-staff viewed the PAC as fertile testing-ground for patient involvement and praised its zeal and pioneering attitude. However, the embedding of the PAC in the assessment structure was limited, restricting its efficacy and influence. The feeling of urgency of patient reviewers to expand their input and influence sometimes collided with the, according to them, tedious developments at DCS. Patient reviewers considered it their task to convince the SC of the value of their involvement. We are the ones to decide on our involvement, not them [the SC]. It’s important to explain the essence of patient participation. We have to educate them [the SC] on this. (patient reviewers in interview) To consolidate the position of the PAC, the pilot programs in which patient reviewers participated in 2014 served to formalize and standardize patient involvement in proposal appraisal. Variables of change in these programs were: pre-applications assessed by PAC, facilitation of PAC-assessment meeting by DCS, formal weighting of assessments of PAC and SC, and PAC-representation at the SC meeting. Based on these insights, a process scheme (Fig. 2) established the procedures and extended them to all research programs except for fundamental science. Figure 2. View largeDownload slide Process scheme of PAC involvement in proposal appraisal, established for all research programs except for fundamental science. Figure 2. View largeDownload slide Process scheme of PAC involvement in proposal appraisal, established for all research programs except for fundamental science. Following this process scheme, from 2015 onwards SC-members, patient reviewers, and DCS-staff were present at the SC-assessment meetings to discuss the funding advice. The position of the PAC was formalized in the grant allocation structure in 2016. Patient reviewers were pleased with this development, calling their representation in the SC ‘beyond dreams.’ Despite this achievement, however, many patient reviewers wished to measure the direct impact of their presence on the funding advice. We were being listened to, there was a dialogue. The chair decides, that was all fine. But on closer inspection, the question arises how much influence we’ve actually had. Should we be equal discussion partners? Well, quite a lot needs to be done to achieve that, and I’m worried. What is our impact? We may feel it, but I want to actually see it. (patient reviewers in evaluative meeting of PAC-representatives) The project team therefore analyzed the congruency of the assessments by the PAC and SC to the funding advice. A total of 63 research proposals (in five different research programs), all tendered in 2015, were analyzed. This analysis corroborates with the doubts of the PAC regarding the quantifiable impact of their assessments on the funding advice. The PAC assessed 37 proposals positively, 18 mediocre, and 5 negatively. The PAC was inclined to give applying researchers the benefit of the doubt when the proposals were not entirely clear to them; this could, in part, explain their overall positive stance towards most proposals. In comparison: the SC assessed 24 proposals positively, 8 mediocre, and 28 negatively. Regarding the proposals over which the PAC and the SC did not agree; for three proposals, the SC assessment was positive but the PAC assessment was mediocre. These proposals were all funded, and the comments of the PAC were forwarded to the applying researchers as additional feedback. There were no proposals in which the SC filed a positive assessment and the PAC a negative one. In 13 cases, the PAC assessment was positive but the SC assessment was negative, all resulting in a negative funding advice. As DCS applies the principle to solely fund research projects of sufficient scientific quality, a negative SC assessment on scientific quality seemed to outweigh the patient reviewers’ assessments on relevance. All applying researchers received the assessments of the PAC as feedback, providing them with suggestions to improve their proposals and making them aware of the patients’ perspectives. As such, the PAC assessments were not only part of the reviewing process of the SC, but also instrumental in raising awareness in the research field regarding the value of patient involvement in writing research proposals. Hence, the impact of the PAC-assessments was not limited to a quantifiable measure. Patient reviewers often expressed their hopes that, in the future, ‘we [patient reviewers] are not needed anymore,’ as applying researchers would take the patients’ interests into account as a matter of course. The influence of PAC-representatives in the SC-assessment meetings was also assessed by participant observation and discussed in the evaluative interviews. All stakeholders were unanimously enthusiastic about their presence, as ‘the patient gets a face’. Their presence [of PAC-representatives] helps SC-members realize there is a difference between knowing about the existence of a problem and experiencing the problem. (DCS-staff member in evaluative interview) Yet, the project team observed that DCS-staff and SC-members need to get accustomed to the presence of the PAC at the assessment meetings. This was mentioned in the workshop with research coordinators of DCS and frequently confirmed in the evaluative interviews. Although the DCS-staff aims at a dialogue between patient reviewers and SC-members, this is not automatically put in practice. Based on the observations during the assessment meetings, several factors could be discerned which influence the discussion at the SC meeting. First, the assessment-meeting chair can affect the input of the patient reviewers and reinforce the PAC’s position by creating the right conditions, such as providing sufficient time to engage in a meaningful dialogue and give the floor to the patient reviewers explicitly. Also, starting an assessment meeting with all participants at the same time prevents PAC-representatives from entering the meeting with a disadvantage. Second, in the SC-assessment meetings, we observed that the composition of the SC influences to what extent the contribution of the patient reviewers is appreciated. Plurality in scientific disciplines encourages an atmosphere in which the patient perspective is acknowledged. Patient reviewers contributed less to the discussions in SC-committees with a more homogeneous composition. Third, in the workshop the research coordinators indicated they needed to develop the skills and competences for a neutral, yet active role to combine the PAC-assessment and the SC-assessment into a final funding advice. From these insights, we can derive that the formal position of the PAC has been firmly established in the organization; on the structural domain, adjustments can be identified such as the establishment and implementation of the process scheme. The effect of these measures was promptly visible. This issue also demonstrates that a cultural shift is required to exploit the patients’ perspectives on proposals to its full potential. Examples of efforts focusing on this cultural domain are providing circumstances to encourage the dialogue between patient reviewers and SC-members and feedback to applying researchers from the patient reviewers’ perspectives. However, this issue demonstrates that effects to become evident on this cultural domain take time, as such a shift is a long-haul process. 6.2.3 Issue 3: exploitation of patient reviewers’ experiential knowledge With respect to culture, many stakeholders, including DCS staff, SC members, and patient reviewers, share the opinion that, to converse at an equal level with each other, patient reviewers need to have some know-how of the scientific process, basic knowledge of the mechanisms of cancer, and be aware of current trends in oncological research. According to a DCS-staff member, this basic knowledge prevents patient reviewers from saying ‘foolish things’, thereby losing their credibility and ‘the goodwill’ of the SC. A number of patient reviewers attended an external training on epistemological issues specifically tailored for this audience to feel sufficiently equipped for their proposal appraisal tasks. Other stakeholders emphasized that patient reviewers also need a ‘backpack’, consisting of patient experiences, to inform them when assessing grant proposals. Similarly, in both interviews and during the assessment meetings SC members warned patient reviewers not to use scientific arguments to assess grant proposals. The project team therefore aimed to encourage patient reviewers to tap into their experiential knowledge rather than professionalize them regarding oncological research. For this purpose, the workshop ‘Connect with a broader patient perspective’ was designed to make patient reviewers aware of the need to regularly calibrate their patient perspective by expanding their ‘backpack’ of fellow patient experiences. However, due to time and organizational constraints, this workshop was postponed: it will become part of the support program for patient reviewers in 2018. Additionally, in the workshop with patient reviewers the need for a limited term was recognized: similar to the SC from 2018 onwards, new members will be invited to the PAC for a maximum of six years. Analyzing this issue, the value of experiential knowledge was emphasized. The necessity to support patient reviewers in articulating this knowledge was acknowledged by team PI. Especially in an environment inclined to highly value scientific knowledge and with patient reviewers gaining confidence through scientific training to converse with the SC on an appropriate level, experiential knowledge runs the risk of undervaluation. The difficulty to establish such a cultural change at DCS was illustrated as stakeholders outside of team PI seemed less aware of this issue. 7. Discussion 7.1 Contextualization of research findings This research aimed at unraveling systemic factors influencing the embedding of meaningful patient involvement in proposal appraisal. By providing a detailed account of the developments at DCS, two overarching observations can be made. First, the changes in the structural domain at DCS are substantial. With the aid of adjusted forms, rules, and regulations, the PAC’s position within the organization is firmly established and the patients’ perspectives are officially represented in the SC. Li et al. (2015) have described similar emphasis on the structural aspects of patient involvement in health policy decision-making: ‘[…] our findings reveal that there is a prevailing belief among decision-makers that such activities must meet robust design criteria to be considered a credible information source […]’. Second, it has become apparent that an accompanying cultural change among all stakeholders at DCS is required, but that the effects of interventions aimed at this domain are less evident. The organizational context, with a tendency to structure daily work into standardized protocols and regulations, can be described as bureaucratic following Westrum’s typology (Westrum 2004). Possibly, this also complicated reflexive discussions regarding the nature of stakeholders’ contributions. This may have hampered changes in daily routines and practices (practical domain) of patient involvement in proposal appraisal. The three issues described in the results section of this article support this analysis. For example, the first issue illustrates how the positivist orientation of oncological research is traditionally prevalent at DCS, and how this seems to be at odds with the subjective nature of experiential knowledge The PAC-members were asked, on the basis of their experiential knowledge, to help assess the appropriateness or relevance of research questions; the acceptability of research procedures and research instruments to patients; and the acceptability of treatments and interventions to patients. This perspective of the PAC-members provided a complementary view to the proposal assessment process. As patient reviewers assessed the proposals on these issues in their personal capacity, explicitly not representing a larger patient community, the individuality of their experiential knowledge was highlighted. On the other hand, the interventions contributed to better formulated assessments and more consistency in the weighing of the assessment criteria, but limited the flexibility of patient reviewers to diverge from these criteria. In this respect, one may question to what extent the involved stakeholders, including the patient reviewers themselves, were trying to fit the subjective experiential knowledge into the (bio)medical research paradigm. In other words, we observed how the (bio)medical research paradigm pervades within a research funding agency preventing integration of the subjective experiential knowledge brought in by patients. Similar tensions between the current research paradigm and the flexibility needed to acknowledge experiential knowledge in oncological research have been described (Gray et al. 2000; Saunders et al. 2007), as patient representatives themselves have been concerned about the representativeness of their involvement (O’Shea et al. 2017). Solutions have often been sought by increasing standardization, training of involved patients to reduce the subjectivity of experiential knowledge, and improving representativeness of participants (Andejeski et al. 2002; Saunders et al. 2007; Li et al. 2015). However, acknowledging the subjectivity of experiential knowledge could aid the knowledge co-creation process, especially in contexts where robustness of the assessment process is a criterion for the credibility of patient involvement (Pittens 2013; Li et al. 2015). Implementation of this notion in the assessment structure requires reflexivity of the stakeholders to acknowledge the different knowledge types. Using the workshop method on knowledge spaces described by (Gibson et al. 2017) could be helpful to enhance this collective reflexivity among all stakeholders, by explicating the four different dimensions of knowledge exchange. We hypothesize that in such an endeavour, the desire of PAC members to move towards more expressive action and the goal of DCS to become an organization open to change, would visualize clearly. Amongst researchers, for example those supported by DCS and who involve patients in their research agenda setting and/or research conduct, the Patient Involvement Impact Assessment Framework (PiiAF) might be a very helpful tool to enhance this reflexivity. PiiAF aids in formulating stakeholders’ values regarding patient involvement, and manage diverging values (Popay and Collins 2014). The second issue highlights that, in DCS’ new requirements to fund proposals, scientific arguments are complemented with the patients’ perspectives. However, in terms of ‘influence,’ the PAC has had little measurable effect on the funding decisions (Li et al. 2015). This finding corresponds to research by Andejeski et al. (2002), who similarly claim that, upon quantitative analysis, patient involvement in proposal appraisal does not alter which research gets funded. Although the effects of the interventions on the cultural level are less measurable, they are not less valuable. Explicit attention to and reflexivity regarding the attitudes and values of participants aids in realizing a truly equal dialogue, especially when the PAC and SC contribute incongruent assessments (Boaz et al. 2016). In this light, we agree with the conclusion by Staley and Doherty (2016) that ‘the information from patients does not [..] need to function as ‘evidence’ in the same way as the clinical and economic data [..]. It is better understood as an interpretive tool to aid [scientific reviewers] in their evaluation and deliberation of the existing clinical and economic data –it enables them to consider this evidence ‘in a different light’’ (p. 9) (Staley and Doherty 2016). Team PI has taken on the challenge of operationalizing these recommendations into concrete interventions taking place beyond the temporal scope of this article. The third issue demonstrates the pressure on patient reviewers to adjust to the scientific dialogue. This phenomenon has been described as proto-professionalism in research and healthcare service development (Caron-Flinterman et al. 2005; El Enany et al. 2013). Although team PI was aware of the necessity of a cultural shift on this issue, other DCS-staff and SC-members seemed less inclined to acknowledge this. Possibly, the needed reflexive practices regarding the patient perspective were undermined by the structure-laden nature of the organization, as flexibility is required to implement the results of this discourse. Similarly, Boaz et al. (2016) argue that ‘the focus on the hardware of engagement (the how to, methods, approaches, guidelines, etc.) rather than the software of values, norms and codes that shape scientific practice helps to explain how the policy to promote [Patient and Public Involvement] has made little impact on the attitudes of academic researchers’. Our findings illustrate that leadership is required to achieve this cultural change: the attitude of the assessment-meeting chair is particularly important in facilitating an equal dialogue between the stakeholders. Also, the need for neutral and independent facilitation of heterogeneous meetings corroborates research by (Rowe and Frewer 2000) on the process requirements of effective patient involvement. 7.2 Generalizability of research findings beyond the organizational context and proposal appraisal This research is based on a single case study, analyzed over a prolonged period of time (2.5 years). However, we argue that our findings are applicable beyond the organization studied, and partly also beyond the limits of proposal appraisal in the research cycle. By distinguishing between systemic barriers in culture, structure, and practice versus contingencies, three overarching issues could be discerned analytically. First, the incompatibility of the objectivity-focused positivist research paradigm with the subjective nature of patient expertise was repeatedly highlighted in this case study. We can state that this paradigm is prevalent throughout health research conduct and decision-making structures and the paradigm has been criticized for its neglect of patient perspectives (Darlenski et al. 2010; Rosner 2012). This study both unravels the real-world implications of such a prevalent paradigm in the decision-making structures of a research funding agency, and explains the difficulty to change such structures. We argue that these insights are valuable for other research funding organizations and may also be applicable to other patient involvement activities (not involving proposal appraisal). Second, the influence of patient reviewers in the proposal assessments was examined in this case study. We acknowledge that there is a difference between quantifiable impact of patient involvement as described by Li et al. (2015) and Andejeski et al. (2002), and a focus on insight rather than impact as highlighted by Staley and Doherty (2016); Li et al. 2015; Andejeski et al. 2002). In our study, the quantifiable impact of the patient reviewers can be described as limited, whereas the insights they provided in the SC-assessment meetings left important traces in the organization. We consider this finding relevant mainly for other health research funding agencies as it may affect their evaluation measures of patient involvement in proposal appraisal. Last, the pressure on patient reviewers to adjust to the scientific social norms has been observed in other instances of patient involvement in health research (Schölvinck et al. in press). In attempts to enhance stakeholders’ reflexivity regarding this issue, it would be interesting to further explore the role of leadership in achieving such a cultural change. 7.3 Methodological considerations Patient involvement in proposal appraisal at DCS was studied by analyzing the role and performance of the PAC. The RMA methodology provided the project team with the opportunity to closely examine the dynamics between the various actors and the system in which they operate. The monitor was frequently present at numerous (in)formal meetings and discussions for 2.5 years, enhancing trust and confidence with all relevant stakeholders. The action-oriented, yet reflexive research approach provided insights on recurring patterns in the system innovation; interventions were designed, executed, and evaluated in collaboration with the project team. As such, systemic factors influencing the system innovation could be distinguished from contingencies. However, the closeness of the monitor to the organization can also be characterized as a methodological limitation to the study; it required effort to zoom out of the day-to-day business to identify the systemic factors influencing the innovation. Regular and explicit moments of process reflection during the course of the study were needed to provide sufficient detachment from daily practical concerns. Similarly, a fruitful collaboration between the monitor and the team PI existed in which constructive feedback was appreciated, whilst all stakeholders acknowledged the constraining and enabling contextual factors. However, a much stronger stand—in which the monitor would have taken a little bit more a confrontational stand—on the need for activities enhancing reflexivity throughout the organization could have supported the system innovation more effectively. Although encouraging all stakeholders (including DCS-staff other than team PI) to reflect on their role and position in a system is theoretically an explicit component of the employed methodology, it proved difficult to organize this in a bureaucratic organization. Recognition of this propensity and explicit attention to its consequences may aid in incorporating the patient perspective more meaningfully in proposal appraisal. 8. Conclusion DCS has successfully implemented several structural adjustments reinforcing the patients’ perspectives in proposal appraisal. However, an excessive focus on the structural domain systematically reproduces the danger of tokenistic patient involvement. Therefore, the cultural domain was addressed too, as both domains can fuel a practical change in working routines and habits. However, the effects of the efforts on the cultural domain take longer to materialize, possibly due to the bureaucratic context. We argue that more reflexivity is needed to incorporate patient involvement into the ‘DNA’ of all stakeholders. Emphasis on rules and regulations at an organization should be considered a starting point and backbone of the system’s innovation, rather than a context-bound factor. References Abma T. A. , Broerse J. E. W. ( 2010 ) ‘ Patient Participation as Dialogue: Setting Research Agendas ’, Health Expectations , 13 / 2 : 160 – 73 . Google Scholar Crossref Search ADS PubMed Alzheimers Society. The Research Network [Internet]. [cited 1 Apr 2018 ]. Available at: https://www.alzheimers.org.uk/info/20063/patient_and_public_involvement/43/the_research_network/3 Andejeski Y. , Bisceglio I. T. , Dickersin K. , Johnson J. E. et al. ( 2002 ) ‘ Quantitative Impact of Including Consumers in the Scientific Review of Breast Cancer Research Proposals ’, Journal of Women’s Health & Gender Based Medicine , 11 / 4 : 379 – 89 . Google Scholar Crossref Search ADS PubMed Andejeski Y. , Breslau E. S. , Hart E. , Lythcott N. et al. ( 2002 ) ‘ Benefits and Drawbacks of Including Consumer Reviewers in the Scientific Merit Review of Breast Cancer Research ’, Journal of Women’s Health & Gender Based Medicine , 11 / 2 : 119 – 36 . Google Scholar Crossref Search ADS PubMed Boaz A. , Biri D. , Mckevitt C. ( 2016 ) ‘ Rethinking the Relationship Between Science and Society: Has There Been a Shift in Attitudes to Patient and Public Involvement and Public Engagement in Science in the United Kingdom? ’, Health Expectations , 19 / 3 : 592 – 601 . Google Scholar Crossref Search ADS PubMed Broerse J. E. W. , Grin J. ( 2017 ) Toward Sustainable Transitions in Healthcare Systems . New York : Routledge . Broerse J. E. W. , Elberse J. E. , Caron-Flinterman J. F. , Zweekhorst M. B. M. ( 2010 ) ‘Enhancing a Transition Towards a Needs-Oriented Health Research System Through Patient Participation’. In: Broerse J.E.W. , Bunders J.F.G. (eds). Transitions in Health Systems: Dealing with Persistent Problems, pp . 181 – 208 . Amsterdam : VU University Press . Caron-Flinterman J. F. , Broerse J. E. W. , Bunders J. F. G. ( 2005 ) ‘ The Experiential Knowledge of Patients: a New Resource for Biomedical Research? ’, Social Science and Medicine , 60 / 11 : 2575 – 84 . Google Scholar Crossref Search ADS PubMed Chalmers I. ( 1995 ) ‘ What Do I Want from Health Research and Researchers When I am a Patient? ’, British Medical Journal , 310 : 1315 – 8 . Google Scholar Crossref Search ADS PubMed Darlenski R. B. , Neykov N. V. , Vlahov V. D. , Tsankov N. K. ( 2010 ) ‘ Evidence-Based Medicine: Facts and Controversies ’, Clinics in Dermatology , 28 / 5 : 553 – 7 . Google Scholar Crossref Search ADS PubMed de Wit M. P. T. , Elberse J. E. , Broerse J. E. W. , Abma T. A. ( 2015 ) ‘ Do Not Forget the Professional - the Value of the FIRST Model for Guiding the Structural Involvement of Patients in Rheumatology Research ’, Health Expectations , 18 / 4 : 489 – 503 . Google Scholar Crossref Search ADS PubMed El Enany N. , Currie G. , Lockett A. ( 2013 ) ‘ A Paradox in Healthcare Service Development: Professionalization of Service Users ’, Social Science & Medicine , 80 : 24 – 30 . Google Scholar Crossref Search ADS Elberse J. E. ( 2012 ) Changing the Health Research System, Patient Participation in Health Research . Hertogenbosch : BOXPress . pp. 163 – 79 p. Elberse J. E. , Caron-Flinterman J. F. , Broerse J. E. W. ( 2011 ) ‘ Patient-Expert Partnerships in Research: How to Stimulate Inclusion of Patient Perspectives ’, Health Expectations , 14 / 3 : 225 – 39 . Google Scholar Crossref Search ADS PubMed Elberse J. E. , de Boer W. I. , Broerse J. E. W. ( 2017 ) Towards a Need-Oriented Health Research System: Involving Patients in Health Research. In: Broerse J.E.W. , Grin J. (eds). Toward Sustainable Transitions in Healthcare Systems . New York : Routledge . Entwistle V. A. , Renfrew M. J. , Yearley S. , Forrester J. et al. ( 1998 ) ‘ Lay Perspectives: Advantages for Health Research ’, British Medical Journal , 316 / 7129 : 463 – 6 . Google Scholar Crossref Search ADS PubMed Fleurence R. L. , Forsythe L. P. , Lauer M. , Rotter J. et al. ( 2014 ) ‘ Engaging Patients and Stakeholders in Research Proposal Review: The Patient-Centered Outcomes Research Institute ’, Annals of Internal Medicine , 161 / 2 : 122 – 30 . Google Scholar Crossref Search ADS PubMed Frenk J. ( 1992 ) ‘ Balancing Relevance and Excellence: Organizational Responses to Link Research with Decision Making ’, Social Science & Medicine , 35 / 11 : 1397 – 404 . Google Scholar Crossref Search ADS Gibbons M. T. ( 1999 ) ‘ Science’s New Social Contract with Society ’, Nature , 402 /December: 81 – 4 . Google Scholar Crossref Search ADS Gibson A. , Welsman J. , Britten N. ( 2017 ) ‘ Evaluating Patient and Public Involvement in Health Research: From Theoretical Model to Practical Workshop ’, Health Expectations , 20 / 5 : 826 – 35 . Google Scholar Crossref Search ADS PubMed Gray R. E. , Fitch M. , Davis C. , Phillips C. ( 2000 ) ‘ Challenges of Participatory Research: Reflections on a Study with Breast Cancer Self-Help Groups ’, Health Expectations , 3 / 4 : 243 – 52 . Google Scholar Crossref Search ADS PubMed Green J. , Thorogood N. ( 2004 ) ‘Analysing Qualitative Data’. Qualitative Methods for Health Research , pp. 173 – 200 . London . Grin J. , Rotmans J. , Schot J. ( 2010 ) Transitions to Sustainable Development: New Directions in the Study of Long Term Transformative Change , p. 397 . New York/Oxon : Routledge . Hanselaar A. G. J. M. , Boerrigter G. H. ( 2011 ) De Strijd Tegen Kanker in Nederland. In: van de Velde C.J.H. et al. (eds). Oncologie , 8 th edn, pp. 683 – 8 . Houten : Bohn Stafleu van Loghum . Hartstichting. ( 2016 ) Commissie Maatschappelijke Kwaliteit (CMK) [Internet]’, [cited 11 Aug]. Available at: http://www.webcitation.org/query? url=https%3A%2F%2Fwww.hartstichting.nl%2Fwetenschappers%2Fcmk%23&date=2017-05-02 Howe A. , Macdonald H. , Barrett B. , Little B. ( 2006 ) ‘ Ensuring Public and Patient Participation in Research: a Case Study in Infrastructure Development in One UK Research and Development Consortium’ , Primary Health Care Research & Development , 7 : 60 – 7 . Google Scholar Crossref Search ADS Kankerbestrijding K. W. F. ( 2015 ) Beleidsvisie . Amsterdam : Beleidsvisie . KWF Kankerbestrijding. ( 2015 ) Jaarverslag . Amsterdam . Li K. K. , Abelson J. , Giacomini M. , Contandriopoulos D. ( 2015 ) ‘ Conceptualizing the Use of Public Involvement in Health Policy Decision-Making ’, Social Science & Medicine , 138 : 14 – 21 . Google Scholar Crossref Search ADS Lindenmeyer A. , Hearnshaw H. , Sturt J. , Ormerod R. et al. ( 2007 ) ‘ Assessment of the Benefits of User Involvement in Health Research from the Warwick Diabetes Care Research User Group: A Qualitative Case Study ’, Health Expectations , 10 / 3 : 268 – 77 . Google Scholar Crossref Search ADS PubMed Maguire K. , Britten N. ( 2017 ) ‘ How Can Anybody be Representative for those Kind of People? Forms of Patient Representation in Health Research, and Why it is Always Contestable ’, Social Science & Medicine , 183 : 62 – 9 . Google Scholar Crossref Search ADS NIHR. ‘Review a Research Application [Internet]’. [cited 1 Apr 2018]. Available at: https://www.nihr.ac.uk/patients-and-public/how-to-join-in/review-a-research-application.htm Nowotny H. , Scott P. B. , Gibbons M. T. ( 2001 ) Re-Thinking Science: Knowledge and the Public in an Age of Uncertainty . Cambridge, UK : Polity Press . O’Donnell M. , Entwistle V. A. ( 2004 ) ‘ Consumer Involvement in Decisions About What Health-Related Research is Funded ’, Health Policy , 70 / 3 : 281 – 90 . Google Scholar Crossref Search ADS PubMed O’Shea A. , Chambers M. , Boaz A. ( 2017 ) ‘ Whose Voices? Patient and Public Involvement in Clinical Commissioning ’, Health Expectations , 20 / 3 : 484 – 94 . Google Scholar Crossref Search ADS PubMed Pittens C. A. C. M. ( 2013 ) Knowledge Co-production in Health Research, Policy and Care Practice . Hertogenbosch : BOXPress . Popay J. , Collins M. ( 2014 ) PiiAF Executive Summary . Lancaster , University of Lancaster . Popay J. , Williams G. ( 1996 ) ‘ Public Health Research and Lay Knowledge ’, Social Science & Medicine , 42 / 5 : 759 – 68 . Google Scholar Crossref Search ADS Regeer B. J. , Bunders J. F. G. ( 2009 ) Knowledge Co-creation: Interaction Between Science and Society. The Hague , The Netherlands : RMNO , pp. 1 – 131 . Rosner A. L. ( 2012 ) ‘ Evidence-Based Medicine: Revisiting the Pyramid of Priorities ’, Journal of Bodywork and Movement Therapies , 16 / 1 : 42 – 9 . Google Scholar Crossref Search ADS PubMed Rotmans J. , Kamp R. , van Asselt M. ( 2001 ) ‘ More Evolution than Revolution: Transition Management in Public Policy ’, Foresight , 3 / 1 : 15 – 31 . Google Scholar Crossref Search ADS Rowe G. , Frewer L. J. ( 2000 ) ‘ Public Participation Methods: A Framework for Evaluation ’, Science, Technology, & Human Values , 25 / 1 : 3 – 29 . Google Scholar Crossref Search ADS Saunders C. , Girgis A. , Butow P. , Crossing S. et al. ( 2007 ) ‘ Beyond Scientific Rigour: Funding Cancer Research of Public Value ’, Health Policy , 84 / 2–3 : 234 – 42 . Google Scholar Crossref Search ADS PubMed Schölvinck A.-F. M. , Schuitmaker T. J. , Broerse J. E. W. (in press) ‘ Meaningful and Sustainable Collaborations with Patients in Research Decision-Making; an Intervention Study at a Dutch Academic Hospital’ , Research Involvement and Engagement . Schuitmaker T. J. ( 2012 ) ‘ Identifying and Unravelling Persistent Problems ’, Technological Forecasting and Social Change , 79 / 6 : 1021 – 31 . Google Scholar Crossref Search ADS Scientific and Ethical Review Board. ( 2016 ) Code of Ethics for Research in the Social and Behavioural Sciences Involving Human Participants . Amsterdam . Staley K. , Doherty C. ( 2016 ) ‘ It’s Not Evidence, it’s Insight: Bringing Patients’ Perspectives into Health Technology Appraisal at NICE ’, Research Involvement and Engagement , 2 / 4 : 1 – 12 . Google Scholar PubMed Supple D. , Roberts A. , Hudson V. , Masefield S. et al. ( 2015 ) ‘ From Tokenism to Meaningful Engagement: Best Practices in Patient Involvement in an EU Project ’, Research Involvement and Engagement , 1 / 5 : 1 – 9 . Google Scholar PubMed Tallon D. , Chard J. , Dieppe P. ( 2000 ) ‘ Relation Between Agendas of the Research Community and the Research Consumer ’, Lancet , 355 : 2037 – 40 . Google Scholar Crossref Search ADS PubMed Teunissen T. , Visse M. A. , de Boer W. I. , Abma T. A. ( 2013 ) ‘ Patient Issues in Health Research and Quality of Care: An Inventory and Data Synthesis ’, Health Expectations , 16 / 4 : 308 – 22 . Google Scholar Crossref Search ADS PubMed Teunissen T. , Visse M. A. , Laan D. , de Boer W. I. et al. ( 2013 ) ‘ Patient Involvement in Lung Foundation Research: A Seven Year Longitudinal Case Study ’, Health , 5 / 2 : 320 – 30 . Google Scholar Crossref Search ADS van Mierlo B. , Regeer B. J. , van Amstel M. , Arkesteijn M. et al. ( 2010 ) Reflexive Monitoring in Action. A Guide for Monitoring System Innovation Projects . Wageningen & Amsterdam : Communication and Innovation Studies, WUR & Athena Institute, VU . van Raak R. ( 2010 ) ‘The Transition (Management) Perspective on Long-Term Change in Health Care’. In: Broerse J.E. , Bunders J.F.G. , (eds). Transitions in Health Systems: Dealing with Persistent Problems , pp. 49 – 86 . Amsterdam : VU University Press . Westrum R. ( 2004 ) ‘ A Typology of Organisational Cultures ’, Quality & Safety in Health Care , 13 : ii22 – 7 . Google Scholar Crossref Search ADS PubMed Whitstock M. T. ( 2003 ) ‘ Seeking Evidence from Medical Research Consumers as Part of the Medical Research Process Could Improve the Uptake of Research Evidence ’, Journal of Evaluation in Clinical Practice , 9 / 2 : 213 – 24 . Google Scholar Crossref Search ADS PubMed Zon M. W. ( 2017 ) ‘Beleidsspeerpunt Participatie [Internet]’. [cited 19 Apr 2017]. Available at: http://www.webcitation.org/query? url=https%3A%2F%2Fwww.zonmw.nl%2Fnl%2Fover-zonmw%2Fparticipatie%2F&date=2017-05-02 © The Author(s) 2018. Published by Oxford University Press. All rights reserved. For permissions, please email: journals.permissions@oup.com This article is published and distributed under the terms of the Oxford University Press, Standard Journals Publication Model (https://academic.oup.com/journals/pages/open_access/funder_policies/chorus/standard_publication_model)
TI - Embedding meaningful patient involvement in the process of proposal appraisal at the Dutch Cancer Society
JO - Science and Public Policy
DO - 10.1093/scipol/scy055
DA - 2019-04-01
UR - https://www.deepdyve.com/lp/oxford-university-press/embedding-meaningful-patient-involvement-in-the-process-of-proposal-dUQw6nQoAb
SP - 254
VL - 46
IS - 2
DP - DeepDyve
ER -