Abstract Research has a role to play in society’s endeavour for sustainable development. This is particularly true for agricultural research, since agriculture is at the nexus between numerous sustainable development goals. Yet, generally accepted methods for linking research outcomes to sustainability impacts are missing. We conducted a review of scientific literature to analyse how impacts of agricultural research were assessed and what types of impacts were covered. A total of 171 papers published between 2008 and 2016 were reviewed. Our analytical framework covered three categories: (1) the assessment level of research (policy, programme, organization, project, technology, or other); (2) the type of assessment method (conceptual, qualitative, or quantitative); and (3) the impact areas (economic, social, environmental, or sustainability). The analysis revealed that most papers (56%) addressed economic impacts, such as cost-effectiveness of research funding or macroeconomic effects. In total, 42% analysed social impacts, like food security or aspects of equity. Very few papers (2%) examined environmental impacts, such as climate effects or ecosystem change. Only one paper considered all three sustainability dimensions. We found a majority of papers assessing research impacts at the level of technologies, particularly for economic impacts. There was a tendency of preferring quantitative methods for economic impacts, and qualitative methods for social impacts. The most striking finding was the ‘blind eye’ towards environmental and sustainability implications in research impact assessments. Efforts have to be made to close this gap and to develop integrated research assessment approaches, such as those available for policy impact assessments. 1. Introduction Research has multiple impacts on society. In the light of the international discourse on grand societal challenges and sustainable development, the debate is reinforced about the role of research on economic growth, societal well-being, and environmental integrity (1). Research impact assessment (RIA) is a key instrument to exploring this role (2). A number of countries have begun using RIA to base decisions for allocation of funding on it, and to justify the value of investments in research to taxpayers (3). The so-called scientometric assessments with a focus on bibliometric and exploitable results such as patents are the main basis for current RIA practices (4–6). However, neither academic values of science, based on the assumption of ‘knowledge as progress’, nor market values frameworks (‘profit as progress’) seem adequate for achieving and assessing broader public values (7). Those approaches do not explicitly acknowledge the contribution of research to solving societal challenges, although they are sufficient to measure scientific excellence (8) or academic impact. RIA may however represent a vital element for designing socially responsible research processes with orientation towards responsibility for a sustainable development (9, 10). In the past, RIAs occurred to focus on output indicators and on links between science and productivity while hardly exploring the wider societal impacts of science (11). RIA should entail the consideration of intended and non-intended, positive and negative, and long- and short-term impacts of research (12). Indeed, there has been a broadening of impact assessments to include, for example, cultural and social returns to society (13). RIA is conceptually and methodologically not yet sufficiently equipped to capture wider societal implications, though (14). This is due to the specific challenges associated with RIA, including inter alia unknown time lags between research processes and their impacts (15–17). Independent from their orientation, RIAs are likely to influence research policies for years to come (18). Research on RIA and its potential to cover wider societal impacts has examined assessment methods and approaches in specific fields of research, and in specific research organizations. The European Science Foundation (19) and Guthrie et al. (20) provided overviews of a range of methods usable in assessment exercises. They discuss generic methods (e.g. economic analyses, surveys, and case studies) with view to their selection for RIAs. Methods need to fit the objectives of the assessment and the characteristics of the disciplines examined. Econometric methods consider the rate of return over investment (21), indicators for ‘productive interactions’ between the stakeholders try to capture the social impact of research (22), and case study-based approaches map the ‘public values’ of research programmes (8, 23). No approach is generally favourable over another, while challenges exist in understanding which impact areas are relevant in what contexts. Penfield et al. (6) looked at the different methods and frameworks employed in assessment approaches worldwide, with a focus on the UK Research Excellence Framework. They argue that there is a need for RIA approaches based on types of impact rather than research discipline. They point to the need for tools and systems to assist in RIAs and highlight different types of information needed along the output-outcome-impact-chain to provide for a comprehensive assessment. In the field of public health research, a minority of RIAs exhibit a wider scope on impacts, and these studies highlight the relevance of case studies (24). However, case studies often rely on principal investigator interviews and/or peer review, not taking into account the views of end users. Evaluation practices in environment-related research organizations tend to focus on research uptake and management processes, but partially show a broader scope and longer-term outcomes. Establishing attribution of environmental research to different types of impacts was identified to be a key challenge (25). Other authors tested impact frameworks or impact patterns in disciplinary public research organizations. For example, Gaunand et al. (26) analysed an internal database of the French Agricultural research organization INRA with 1,048 entries to identify seven impact areas, with five going beyond traditional types of impacts (e.g. conservation of natural resources or scientific advice). Besides, for the case of agricultural research, no systematic review of RIA methods exists in the academic literature that would allow for an overview of available approaches covering different impact areas of research. Against this background, the objective of this study was to review in how far RIAs of agricultural research capture wider societal implications. We understand agricultural research as being a prime example for the consideration of wider research impacts. This is because agriculture is a sector which has direct and severe implications for a range of the UN Sustainable Development Goals. It has a strong practice orientation and is just beginning to develop a common understanding of innovation processes (27). 2. Analytical framework The analysis of the identified literature on agricultural RIA (for details, see next section ‘Literature search’) built on a framework from a preliminary study presented at the ImpAR Conference 2015 (28). It was based on three categories to explore the impact areas that were addressed and the design of RIA. In particular, the analytical framework consisted of: (1) the assessment level of research; (2) the type of assessment method; and (3) the impact areas covered. On the side, we additionally explored the time dimension of RIA, i.e. whether the assessment was done ex ante or ex post (see Fig. 1). Figure 1. View largeDownload slide Analytical framework for the review of non-scientometric impact assessment literature of agricultural research. Figure 1. View largeDownload slide Analytical framework for the review of non-scientometric impact assessment literature of agricultural research. Agricultural research and the ramifications following from that refer to different levels of assessment (or levels of evaluation, (29)). We defined six assessment levels that can be the subject of a RIA: policy, programme, organization, project, technology, and other. The assessment level of the RIA is a relevant category, since it shapes the approach to the RIA (e.g. the impact chain of a research project differs to that at policy level). The assessment level was clearly stated in all of the analysed papers and in no case more than one assessment level was addressed. Articles were assigned to the policy level, if a certain public technology policy (30) or science policy, implemented by governments to directly or indirectly affect the conduct of science, was considered. Exemplary topics are research funding, transfer of research results to application, or contribution to economic development. Research programmes were understood as instruments that are adopted by government departments, or other organizational entities to implement research policies and fund research activities in a specific research field (e.g. programmes to promote research on a certain crop or cultivation technique). Articles dealing with the organizational level assess the impact of research activities of a specific research organization. The term research organization comprises public or private research institutes, associations, networks, or partnerships (e.g. the Consultative Group on International Agricultural Research (CGIAR) and its research centres). A research project is the level at which research is actually carried out, e.g. as part of a research programme. The assessment of a research project would consider the impacts of the whole project, from planning through implementation to evaluation instead of focusing on a specific project output, like a certain agricultural innovation. The technology level was considered to be complementary to the other assessment levels of research and comprises studies with a strong focus on specific agricultural machinery or other agricultural innovation such as new crops or crop rotations, fertilizer applications, pest control, or tillage practices, irrespective of the agricultural system (e.g. smallholder or high-technology farming, or organic, integrated, or conventional farming). The category ‘other’ included one article addressing RIA at the level of individual researchers (see (31)). We categorized the impact areas along the three dimensions of sustainable development by drawing upon the European Commission’s impact assessment guidelines (cf. (32)). The guidelines entail a list of 7 environmental impacts, such as natural resource use, climate change, or aspects of nature conservation; 12 social impacts, such as employment and working conditions, security, education, or aspects of equity; and 10 economic impacts, including business competitiveness, increased trade, and several macroeconomic aspects. The European Commission’s impact assessment guidelines were used as a classification framework because it is one of the most advanced impact assessment frameworks established until to date (33). In addition, we opened a separate category for those articles exploring joint impacts on the three sustainability dimensions. Few articles addressed impacts in two sustainability dimensions which we assigned to the dominating impact area. To categorize the type of RIA method, we distinguished between conceptual, qualitative, and quantitative. Conceptual analyses include the development of frameworks or concepts for measuring impacts of agricultural research (e.g. tracking of innovation pathways or the identification of barriers and supporting factors for impact generation). Qualitative and quantitative methods were identified by the use of qualitative data or quantitative data, respectively (cf. (34–36)). Qualitative data can be scaled nominally or ordinally. It is generated by interviews, questionnaires, surveys or choice experiments to gauge stakeholder attitudes to new technologies, their willingness to pay, and their preference for adoption measures. The generation of quantitative data involves a numeric measurement in a standardized way. Such data are on a metric scale and are often used for modelling. The used categorization is rather simple. We assigned approaches which employed mixed-method approaches according to their dominant method. We preferred this over more sophisticated typologies to achieve a high level of abstraction and because the focus of our analysis was on impact areas rather than methods. However, to show consistencies with existing typologies of impact assessment methods (19, 37), we provide an overview of the categorization chosen and give examples of the most relevant types of methods. To additionally explore the approach of the assessment (38), the dimensions ex ante and ex post were identified. The two approaches are complementary: whereas ex ante impact assessments are usually conducted for strategic and planning purposes to set priorities, ex post impact assessments serve as accountability validation and control against a baseline. The studies in our sample that employed an ex ante approach to RIA usually made this explicit, while in the majority of ex post impact assessments, this was indicated rather implicitly. 3. Literature search This study was performed as a literature review based on Thomson Reuters Web of ScienceTM Core Collection, indexed in the Science Citation Index Expanded (SCI-Exp) and the Social Sciences Citation Index (SSCI). The motivation for restricting the analysis to articles from ISI-listed journals was to stay within the boundaries of internationally accepted scientific quality management and worldwide access. The advantages of a search based on Elsevier’s Scopus® (more journals and alternative publications, and more articles from social and health science covered) would not apply for this literature review, with regard to the drawbacks of an index system based on abstracts instead of citation indexes, which is not as transparent as the Core Collection regarding the database definable by the user. We selected the years of 2008 to mid-2016 for the analysis (numbers last updated on2June 2016). First, because most performance-based funding systems have been introduced since 2000, allowing sufficient time for the RIA approaches to evolve and literature to be published. Secondly, in 2008 two key publications on RIA of agricultural research triggered the topic: Kelley, et al. (38) published the lessons learned from the Standing Panel on Impact Assessment of CGIAR; Watts, et al. (39) summarized several central pitfalls of impact assessment concerning agricultural research. We took these publications as a starting point for the literature search. We searched in TOPIC and therefore, the terms had to appear in the title, abstract, author keywords, or keywords plus®. The search query1 filtered for agricultural research in relation to research impact. To cover similar expressions, we used science, ‘R&D’, and innovation interchangeably with research, and we searched for assessment, evaluation, criteria, benefit, adoption, or adaptation of research. We combined the TOPIC search with a less strict search query2 in TITLE using the same groups of terms, as these searches contained approximately two-thirds non-overlapping papers. Together they consisted of 315 papers. Of these, we reviewed 282 after excluding all document types other than articles and reviews (19 papers were not peer-reviewed journal articles) and all papers not written in English language (14 papers). After going through them, 171 proved to be topic-relevant and were included in the analysis. 4. Results The application of the analytical framework allowed for the classification of the 171 reviewed articles regarding the assessment level of the research, the impact areas considered, and the type of method applied. These results are structured in a quantitative matrix (Table 1). Table 1. Analysis matrix showing the number of reviewed articles, each categorized to an assessment level and an impact area (social, economic, environmental, or all three (sustainability)). Additionally, the type of analytical method (conceptual, quantitative, and qualitative) is itemized Assessment level Policy Programme Organization Project Technology Others Sum Impact area Social issues 18 9 10 16 19 0 72 Conceptual 7 3 1 4 6 0 21 Qualitative 6 4 4 10 11 0 35 Quantitative 5 2 5 2 2 0 16 Economy 13 12 11 12 46 1 95 Conceptual 3 4 6 5 13 1 32 Qualitative 2 2 1 3 19 0 27 Quantitative 8 6 4 4 14 0 36 Environment 0 0 0 1 2 0 3 Conceptual 0 0 0 0 1 0 1 Qualitative 0 0 0 1 0 0 1 Quantitative 0 0 0 0 1 0 1 Sustainability 0 0 0 0 1 0 1 Conceptual 0 0 0 0 1 0 1 Qualitative 0 0 0 0 0 0 0 Quantitative 0 0 0 0 0 0 0 Total 31 21 21 29 68 1 171 Assessment level Policy Programme Organization Project Technology Others Sum Impact area Social issues 18 9 10 16 19 0 72 Conceptual 7 3 1 4 6 0 21 Qualitative 6 4 4 10 11 0 35 Quantitative 5 2 5 2 2 0 16 Economy 13 12 11 12 46 1 95 Conceptual 3 4 6 5 13 1 32 Qualitative 2 2 1 3 19 0 27 Quantitative 8 6 4 4 14 0 36 Environment 0 0 0 1 2 0 3 Conceptual 0 0 0 0 1 0 1 Qualitative 0 0 0 1 0 0 1 Quantitative 0 0 0 0 1 0 1 Sustainability 0 0 0 0 1 0 1 Conceptual 0 0 0 0 1 0 1 Qualitative 0 0 0 0 0 0 0 Quantitative 0 0 0 0 0 0 0 Total 31 21 21 29 68 1 171 In the agricultural RIA, the core assessment level of the reviewed articles was technology (39%), while the other levels were almost equally represented (with the exception of ‘other’). Generally, most papers (56%) addressed economic research impacts, closely followed by social research impacts (42%); however, only three papers (2%) addressed environmental research impacts and only 1 of 171 papers addressed all three dimensions of sustainable development. Assessments at the level of research policy slightly emphasized social impacts over economic impacts (18 papers, or 58%), whereas assessments at the level of technology clearly focused primarily on economic impacts (46 papers, or 68%). The methods used for agricultural RIA showed no preference for one method type (see Table 1). Approximately 31% of the papers assessed research impacts quantitatively, whereas 37% used qualitative methods. Conceptual considerations on research impact were applied by 32% of the studies. A noticeable high number of qualitative studies were conducted to assess social impacts. At the evaluation level of research policy and research programmes, we found a focus on quantitative methods, if economic impacts were assessed. A non-exhaustive list of methods applied in the analysed studies is provided in Table 2. The list is meant to illustrate what is behind the three categories of conceptual, qualitative, and quantitative RIAs used in our review (method Type I) referring to typologies (method Type II) from the European Science Foundation (19) and RAND Europe (37). Table 2. Overview on type of methods used for agricultural RIA Method Type I Method Type II Example Conceptual Review Document analysis, literature review, argumentation, anecdotes Framework development Conceptual innovation Qualitative Survey Questionnaire, interview, expert surveys, etc. Quantitative Stochastic method Regression analysis, Bayesian probabilistic method Economic valuation Econometric analysis, cost–benefit analysis, cost-effectiveness Mixed Participatory evaluationa Individual rating, group voting, actor mapping, evaluation of assessment tools Case studiesb Detailed analysis of individual research projects, programmes, etc. Method Type I Method Type II Example Conceptual Review Document analysis, literature review, argumentation, anecdotes Framework development Conceptual innovation Qualitative Survey Questionnaire, interview, expert surveys, etc. Quantitative Stochastic method Regression analysis, Bayesian probabilistic method Economic valuation Econometric analysis, cost–benefit analysis, cost-effectiveness Mixed Participatory evaluationa Individual rating, group voting, actor mapping, evaluation of assessment tools Case studiesb Detailed analysis of individual research projects, programmes, etc. aMix of conceptual and qualitative methods. bMix of conceptual, qualitative, and quantitative methods. Additionally, 37 ex ante studies, compared to 134 ex post studies, revealed that the latter clearly dominated, but no robust relation to any other investigated characteristic was found. Of the three environmental impact studies, none assessed ex ante, while the one study exploring sustainability impacts did. The share of ex ante assessments regarding social impacts was very similar to those regarding economic impacts. Within the assessment levels of research (excluding ‘others’ with only one paper), no notable difference between the shares of ex ante assessments occurred as they ranged between 13 and 28%. 5. Discussion The most relevant outcome of the review analysis was that only 3 of the 171 papers focus on the environmental impacts of agricultural research. This seems surprising because agriculture is dependent on an intact environment. However, this finding is supported by two recent reviews: one from Bennett, et al. (40) and one from Maredia and Raitzer (41). Both note that not only international agricultural research in general but also research on natural resource management shows a lack regarding large-scale assessments of environmental impacts. The CGIAR also recognized the necessity to deepen the understanding of the environmental impacts of its work because RIAs had largely ignored environmental benefits (42). A few papers explicitly include environmental impacts of research in addition to their main focus. Raitzer and Maredia (43) address water depletion, greenhouse gas emissions, and landscape effects; however, their overall focus is on poverty reduction. Ajayi et al. (44) report the improvement of soil physical properties and soil biodiversity from introducing fertilizer trees but predominantly measure economic and social effects. Cavallo, et al. (45) investigate users’ attitudes towards the environmental impact of agricultural tractors (considered as technological innovation) but do not measure the environmental impact. Briones, et al. (46) configure an environmental ‘modification’ of economic surplus analysis, but they do not prioritize environmental impacts. Of course, the environmental impacts of agricultural practices were the topic of many studies in recent decades, such as Kyllmar, et al. (47), Skinner, et al. (48), Van der Werf and Petit (49), among many others. However, we found very little evidence for the impact of agricultural research on the environment. A study on environmental management systems that examined technology adoption rates though not the environmental impacts is exemplarily for this (50). One possible explanation is based on the observation made by Morris, et al. (51) and Watts, et al. (39). They see impact assessments tending to accentuate the success stories because studies are often commissioned strategically as to demonstrate a certain outcome. This would mean to avoid carving out negative environmental impacts that conflict with, when indicated, the positive economic or societal impacts of the assessed research activity. In analogy to policy impact assessments, this points to the need of incentives to equally explore intended and unintended, expected and non-expected impacts from scratch (52). From those tasked with an RIA, this again requires an open attitude in ‘doing RIA’ and towards the findings of their RIA. Another possible explanation was given by Bennett, et al. (40): a lack of skills in ecology or environmental economics to cope with the technically complex and data-intensive integration of environmental impacts. Although such a lack of skills or data could also apply to social and economic impacts, continuous monitoring of environmental data related to agricultural practices is particularly scarce. A third possible explanation is a conceptual oversight, as environmental impacts may be thought to be covered by the plenty of environmental impact assessments of agricultural activities itself. The impression of a ‘blind eye’ on the environment in agricultural RIA may change when publications beyond Web of ScienceTM Core Collection are considered (53) or sources other than peer-reviewed journal articles are analysed (e.g. reports; conference proceedings). See, for example, Kelley, et al. (38), Maredia and Pingali (54), or FAO (55). Additionally, scientific publications of the highest quality standard (indicated by reviews and articles being listed in the Web of ScienceTM Core Collection) seem to not yet reflect experiences and advancements from assessment applications on research and innovation policy that usually include the environmental impact (56). Since their beginnings, RIAs have begun to move away from narrow exercises concerned with economic impacts (11) and expanded their scope to social impacts. However, we only found one sustainability approach in our review that would cover all three impact areas of agricultural research (see (57)). In contrast, progressive approaches to policy impact assessment largely attempt to cover the full range of environmental, social, and economic impacts of policy (33, 58). RIAs may learn from them. Additionally, the focus of agricultural research on technological innovation seems evident. Although the word innovation is sometimes still used for new technology (as in ‘diffusion of innovations’), it is increasingly used for the process of technical and institutional change at the farm level and higher levels of impact. Technology production increasingly is embedded in innovation systems (59). The review revealed a diversity of methods (see Table 2) applied in impact assessments of agricultural research. In the early phases of RIA, the methods drawn from agricultural economics were considered as good standard for an impact assessment of international agricultural research (39). However, quantitative methods most often address economic impacts. In addition, the reliability of assessments based on econometric models is often disputed because of strong relationships between modelling assumptions and respective results. Regarding environmental (or sustainability) impacts of agricultural research, the portfolio of assessment methods could be extended by learning from RIAs in other impact areas. In our literature sample, only review, framework development (e.g. key barrier typologies, environmental costing, or payments for ecosystem services), life-cycle assessment, and semi-structured interviews were used for environmental impacts of agricultural research. In total, 42 of the 171 analysed papers assessed the impact of participatory research. A co-management of public research acknowledges the influence of the surrounding ecological, social, and political system and allows different types of stakeholder knowledge to shape innovation (60). Schut, et al. (36) conceptualize an agricultural innovation support system, which considers multi-stakeholder dynamics next to multilevel interactions within the agricultural system and multiple dimensions of the agricultural problem. Another type of participation in RIAs is the involvement of stakeholders to the evaluation process. A comparatively low number of six papers considered participatory evaluation of research impact, of them three in combination with impact assessment of participatory research. Approximately 22% of the articles in our sample on agricultural research reported that they conducted their assessments ex ante, but most studies were ex post assessments. Watts, et al. (39) considered ex ante impact assessment to be more instructive than ex post assessment because it can directly guide the design of research towards maximizing beneficial impacts. This is particularly true when an ex ante assessment is conducted as a comparative assessment comprising a set of alternative options (61). Many authors of the studies analysed were not explicit about the time frames considered in their ex post studies. The potential latency of impacts from research points to the need for ex post (and ex ante) studies to account for and analyse longer time periods, either considering ‘decades’ (62, 63) or a lag distribution covering up to 50 years, with a peak approximately in the middle of the impact period (64). This finding is in line with the perspective of impact assessments as an ongoing process throughout a project’s life cycle and not as a one-off process at the end (51). Nevertheless, ex post assessments are an important component of a comprehensive evaluation package, which includes ex ante impact assessment, impact pathway analysis, programme peer reviews, performance monitoring and evaluation, and process evaluations, among others (38). RIA is conceptually and methodologically not yet sufficiently equipped to capture wider societal implications, though (14). This is due to the specific challenges associated with RIA, including inter alia unknown time lags between research processes and their impacts (15–17). Independent from their orientation, RIAs are likely to influence research policies for years to come (18). However, in the cases in which a RIA is carried out, an increase in the positive impacts (or avoidance of negative impacts) of agricultural research does not follow automatically. Lilja and Dixon (65) state the following methodological reasons for the missing impact of impact studies: no accountability with internal learning, no developed scaling out, the overlap of monitoring and evaluation and impact assessment, the intrinsic nature of functional and empowering farmer participation, the persistent lack of widespread attention to gender, and the operational and political complexity of multi-stakeholder impact assessment. In contrast, a desired impact of research could be reached or boosted by specific measures without making an impact assessment at all. Kristjanson, et al. (66), for example, proposed seven framework conditions for agricultural research to bridge the gap between scientific knowledge and action towards sustainable development. RIA should develop into process-oriented evaluations, in contrast to outcome-oriented evaluation (67), for addressing the intended kind of impacts, the scope of assessment, and for choosing the appropriate assessment method (19). 6. Conclusion This review aimed at providing an overview of impact assessment activities reported in academic agricultural literature with regard to their coverage of impact areas and type of assessment method used. We found a remarkable body of non-scientometric RIA at all evaluation levels of agricultural research but a major interest in economic impacts of new agricultural technologies. These are closely followed by an interest in social impacts at multiple assessments levels that usually focus on food security and poverty reduction and rely slightly more on qualitative assessment methods. In contrast, the assessment of the environmental impacts of agricultural research or comprehensive sustainability assessments was exceptionally limited. They may have been systematically overlooked in the past, for the reason of expected negative results, thought to be covered by other impact studies or methodological challenges. RIA could learn from user-oriented policy impact assessments that usually include environmental impacts. Frameworks for RIA should avoid narrowing the assessment focus and instead considering intended and unintended impacts in several impact areas equally. It seems fruitful to invest in assessment teams’ environmental analytic skills and to expand several of the already developed methods for economic or social impact to the environmental impacts. Only then, the complex and comprehensive contribution of agricultural research to sustainable development can be revealed. Acknowledgement The authors would like to thank Jana Rumler and Claus Dalchow for their support in the Web of Science analysis and Melanie Gutschker for her support in the quantitative literature analysis. Funding This work was supported by the project LIAISE (Linking Impact Assessment to Sustainability Expertise, www.liaisenoe.eu), which was funded by Framework Programme 7 of the European Commission and co-funded by the Leibniz-Centre for Agricultural Landscape Research. The research was further inspired and supported by funding from the ‘Guidelines for Sustainability Management’ project for non-university research institutes in Germany (‘Leitfaden Nachhaltigkeitsmanagement’, BMBF grant 311 number 13NKE003A). References 1 Seidl R. et al. ( 2013) ‘ Science with Society in the Anthropocene’, Ambio , 42/ 1: 5– 12. Google Scholar CrossRef Search ADS PubMed 2 OECD. ( 2010) ‘Performance-Based Funding for Public Research in Tertiary Education Institutions’, Workshop Proceedings ' 2010. Paris: Organisation for Economic Co-operation and Development. 3 Hicks D. ( 2012) ‘ Performance-based University Research Funding Systems’, Research Policy , 41/ 2: 251– 61. Google Scholar CrossRef Search ADS 4 Martin B. R. ( 1996) ‘ The Use of Multiple Indicators in the Assessment of Basic Research’, Scientometrics , 36/ 3: 343– 62. Google Scholar CrossRef Search ADS 5 Moed H. F., Halevi G. ( 2015) ‘ Multidimensional Assessment of Scholarly Research Impact’, Journal of the Association for Information Science and Technology , 66: 1988– 2002. Google Scholar CrossRef Search ADS 6 Penfield T. et al. ( 2014) ‘ Assessment, Evaluations, and Definitions of Research Impact: A Review’, Research Evaluation , 23/ 1: 21– 32. Google Scholar CrossRef Search ADS 7 Meyer R. ( 2011) ‘ The Public Values Failures of Climate Science in the US’, Minerva , 49/ 1: 47– 70. Google Scholar CrossRef Search ADS 8 Bozeman B., Sarewitz D. ( 2011) ‘ Public Value Mapping and Science Policy Evaluation’, Minerva , 49/ 1: 1– 23. Google Scholar CrossRef Search ADS 9 Helming K. et al. ( 2016) ‘ Forschen für nachhaltige Entwicklung. Kriterien für gesellschaftlich verantwortliche Forschungsprozesse (Research for Sustainable Development. Criteria for Socially Responsible Research Processes)’, GAIA , 25/ 3: 161– 5. Google Scholar CrossRef Search ADS 10 Cagnin C., Amanatidou E., Keenan M. ( 2012) ‘ Orienting European Innovation Systems Towards Grand Challenges and the Roles that FTA Can Play’, Science and Public Policy , 39/ 2: 140– 52. Google Scholar CrossRef Search ADS 11 Godin B., Doré C. ( 2004) Measuring the Impacts of Science: Beyond the Economic Dimension. Montréal (Québec): Centre Urbanisation Culture Société (INRS). 12 Ferretti J. et al. ( 2016) Reflexionsrahmen für Forschen in gesellschaftlicher Verantwortung. (Framework for Reflecting Research in Societal Responsibility). Berlin: Federal Ministry of Education and Research (BMBF). 13 Jacobsson S., Vico E. P., Hellsmark H. ( 2014) ‘ The Many Ways of Academic Researchers: How is Science Made Useful?’, Science and Public Policy , 41: 641– 57. Google Scholar CrossRef Search ADS 14 Levitt R. et al. ( 2010) Assessing the Impact of Arts and Humanities Research at the University of Cambridge. Cambridge: University of Cambridge. 15 Donovan C. ( 2011) ‘ State of the Art in Assessing Research Impact: Introduction to a Special Issue’, Research Evaluation , 20/ 3: 175– 9. Google Scholar CrossRef Search ADS 16 Ekboir J. ( 2003) ‘ Why Impact Analysis Should not be Used for Research Evaluation and what the Alternatives Are’, Agricultural Systems , 78/ 2: 166– 84. Google Scholar CrossRef Search ADS 17 Morton S. ( 2015) ‘ Progressing Research Impact Assessment: A ‘Contributions’ Approach’, Research Evaluation , 24: 405– 19. Google Scholar CrossRef Search ADS 18 Reinhardt A. ( 2013) ‘Different Pathways to Impact? “Impact” and Research Fund Allocation in Selected European Countries’, in Dean A., Wykes M., Stevens H. (eds) 7 Essays on Impact. DESCRIBE Project Report for Jisc , pp. 88– 101. Exeter: University of Exeter. 19 European Science Foundation. ( 2012) The Challenges of Impact Assessment. Working Group 2: Impact Assessment. Strasbourg: European Science Foundation. 20 Guthrie S. et al. ( 2013) Measuring Research. A Guide to Research Evaluation Frameworks and Tools. Cambridge: RAND Corporation. 21 Alston J. M. et al. ( 2011) ‘ The Economic Returns to US Public Agricultural Research’, American Journal of Agricultural Economics , 93/ 5: 1257– 77. Google Scholar CrossRef Search ADS 22 Spaapen J., Drooge L. ( 2011) ‘ Introducing' Productive Interactions' in Social Impact Assessment’, Research Evaluation , 20/ 3: 211– 18. Google Scholar CrossRef Search ADS 23 Bozeman B. ( 2003) Public Value Mapping of Science Outcomes: Theory and Method. Washington: Center for Science, Policy and Outcomes. 24 Milat A. J., Bauman A. E., Redman S. ( 2015) ‘ A Narrative Review of Research Impact Assessment Models and Methods’, Health Research Policy and Systems , 13/ 1: 18. Google Scholar CrossRef Search ADS PubMed 25 Bell S., Shaw B., Boaz A. ( 2011) ‘ Real-world Approaches to Assessing the Impact of Environmental Research on Policy’, Research Evaluation , 20/ 3: 227– 37. Google Scholar CrossRef Search ADS 26 Gaunand A. et al. ( 2015) ‘ How Does Public Agricultural Research Impact Society? A Characterization of Various Patterns’, Research Policy , 44/ 4: 849– 61. Google Scholar CrossRef Search ADS 27 Bokelmann W. et al. ( 2012) Sector Study on the Analysis of the Innovation of German Agriculture (Sektorstudie zur Untersuchung des Innovationssystems der deutschen Landwirtschaft). Berlin: Federal Office for Agriculture and Food (BLE). 28 Weißhuhn P., Helming K. ( 2015) ‘Methods for Assessing the Non-Scientometric Impacts of Agricultural Research: A Review’. In ImpAR Conference 2015: Impacts of Agricultural Research-Towards an Approach of Societal Values. Paris: INRA. 29 European Science Foundation. ( 2009) Evaluation in National Research Funding Agencies: Approaches, Experiences and Case Studies. Strasbourg: European Science Foundation. 30 Bozeman B. ( 2000) ‘ Technology Transfer and Public Policy: A Review of Research and Theory’, Research Policy , 29/ 4: 627– 55. Google Scholar CrossRef Search ADS 31 Hummer K. E., Hancock J. F. ( 2015) ‘ Vavilovian Centers of Plant Diversity: Implications and Impacts’, Hortscience , 50/ 6: 780– 3. 32 EC. ( 2015) Better Regulation “Toolbox”. Brussels: European Commission. 33 Helming K. et al. ( 2013) ‘ Mainstreaming Ecosystem Services in European Policy Impact Assessment’, Ecosystem Services in EIA and SEA , 40: 82– 7. 34 Thapa D. B. et al. ( 2009) ‘ Identifying Superior Wheat Cultivars in Participatory Research on Resource Poor Farms’, Field Crops Research , 112/ 2–3: 124– 30. Google Scholar CrossRef Search ADS 35 Holdsworth M. et al. ( 2015) ‘ African Stakeholders' Views of Research Options to Improve Nutritional Status in Sub-Saharan Africa’, Health Policy and Planning , 30/ 7: 863– 74. Google Scholar CrossRef Search ADS PubMed 36 Schut M. et al. ( 2015) ‘ RAAIS: Rapid Appraisal of Agricultural Innovation Systems (Part I). A Diagnostic Tool for Integrated Analysis of Complex Problems and Innovation Capacity’, Agricultural Systems , 132: 1– 11. Google Scholar CrossRef Search ADS 37 Jones M. M., Grant J. ( 2013) ’Making the Grade: Methodologies for assessing and evidencing research impact’. In Dean A., Wykes M., Stevens H. (eds) 7 Essays on Impact. DESCRIBE Project Report for Jisc , pp. 25– 43. Exeter: University of Exeter. 38 Kelley T., Ryan J., Gregersen H. ( 2008) ‘ Enhancing Ex Post Impact Assessment of Agricultural Research: The CGIAR Experience’, Research Evaluation , 17/ 3: 201– 12. Google Scholar CrossRef Search ADS 39 Watts J. et al. ( 2008) ‘ Transforming Impact Assessment: Beginning the Quiet Revolution of Institutional Learning and Change’, Experimental Agriculture , 44/ 1: 21– 35. Google Scholar CrossRef Search ADS 40 Bennett J. W., Kelley T. G., Maredia M. K. ( 2012) ‘ Integration of Environmental Impacts Into Ex-post Assessments of International Agricultural Research: Conceptual Issues, Applications, and the Way Forward’, Research Evaluation , 21/ 3: 216– 28. Google Scholar CrossRef Search ADS 41 Maredia M. K., Raitzer D. A. ( 2012) ‘ Review and Analysis of Documented Patterns of Agricultural Research Impacts in Southeast Asia’, Agricultural Systems , 106/ 1: 46– 58. Google Scholar CrossRef Search ADS 42 Renkow M., Byerlee D. ( 2010) ‘ The Impacts of CGIAR Research: A Review of Recent Evidence’, Food Policy , 35/ 5: 391– 402. Google Scholar CrossRef Search ADS 43 Raitzer D. A., Maredia M. K. ( 2012) ‘ Analysis of Agricultural Research Investment Priorities for Sustainable Poverty Reduction in Southeast Asia’, Food Policy , 37/ 4: 412– 26. Google Scholar CrossRef Search ADS 44 Ajayi O. C. et al. ( 2011) ‘ Agricultural Success from Africa: The Case of Fertilizer Tree Systems in Southern Africa (Malawi, Tanzania, Mozambique, Zambia and Zimbabwe)’, International Journal of Agricultural Sustainability , 9/ 1: 129– 36. Google Scholar CrossRef Search ADS 45 Cavallo E. et al. ( 2014) ‘ Strategic Management Implications for the Adoption of Technological Innovations in Agricultural Tractor: The Role of Scale Factors and Environmental Attitude’, Technology Analysis and Strategic Management , 26/ 7: 765– 79. Google Scholar CrossRef Search ADS 46 Briones R. M. et al. ( 2008) ‘ Priority Setting for Research on Aquatic Resources: An Application of Modified Economic Surplus Analysis to Natural Resource Systems’, Agricultural Economics , 39/ 2: 231– 43. Google Scholar CrossRef Search ADS 47 Kyllmar K. et al. ( 2014) ‘ Small Agricultural Monitoring Catchments in Sweden Representing Environmental Impact’, Agriculture, Ecosystems and Environment , 198: 25– 35. Google Scholar CrossRef Search ADS 48 Skinner J. et al. ( 1997) ‘ An Overview of the Environmental Impact of Agriculture in the UK’, Journal of Environmental Management , 50/ 2: 111– 28. Google Scholar CrossRef Search ADS 49 Van der Werf H. M., Petit J. ( 2002) ‘ Evaluation of the Environmental Impact of Agriculture at the Farm Level: A Comparison and Analysis of 12 Indicator-based Methods’, Agriculture, Ecosystems and Environment , 93/ 1: 131– 45. Google Scholar CrossRef Search ADS 50 Carruthers G., Vanclay F. ( 2012) ‘ The Intrinsic Features of Environmental Management Systems that Facilitate Adoption and Encourage Innovation in Primary Industries’, Journal of Environmental Management , 110: 125– 34. Google Scholar CrossRef Search ADS PubMed 51 Morris M. et al. ( 2003) ‘ Assessing the Impact of Agricultural Research: An Overview’, Quarterly Journal of International Agriculture , 42/ 2: 127– 48. 52 Podhora A. et al. ( 2013) ‘ The Policy-Relevancy of Impact Assessment Tools: Evaluating Nine Years of European Research Funding’, Environmental Science and Policy , 31: 85– 95. Google Scholar CrossRef Search ADS 53 Rodrigues G. S., de Almeida Buschinelli C. C., Dias Avila A. F. ( 2010) ‘ An Environmental Impact Assessment System for Agricultural Research and Development II: Institutional Learning Experience at Embrapa’, Journal of Technology Management and Innovation , 5/ 4: 38– 56. Google Scholar CrossRef Search ADS 54 Maredia M., Pingali P. ( 2001) Environmental Impacts of Productivity-Enhancing Crop Research: A Critical Review. Durban: CGIAR. 55 FAO. ( 2011) ‘Environmental Impact Assessment', Guideline for FAO field projects. Rome: Food and Agriculture Organization of the United Nations. 56 Miedzinski M. et al. ( 2013) Assessing Environmental Impacts of Research and Innovation Policy. 57 Ervin D. E., Glenna L. L., Jussaume R. A. ( 2011) ‘ The Theory and Practice of Genetically Engineered Crops and Agricultural Sustainability’, Sustainability , 3/ 6: 847– 74. Google Scholar CrossRef Search ADS 58 Jacob K. et al. ( 2012) ‘Sustainability in Impact Assessments - A Review of Impact Assessment Systems in selected OECD countries and the European Commission’. Paris: Organisation for Economic Co-operation and Development. 59 Röling N. ( 2009) ‘ Pathways for Impact: Scientists' Different Perspectives on Agricultural Innovation’, International Journal of Agricultural Sustainability , 7/ 2: 83– 94. Google Scholar CrossRef Search ADS 60 Dentoni D., Klerkx L. ( 2015) ‘ Co-managing Public Research in Australian Fisheries Through Convergence-Divergence Processes’, Marine Policy , 60: 259– 71. Google Scholar CrossRef Search ADS 61 Helming K. et al. ( 2011) ‘ Ex Ante Impact Assessment of Policies Affecting Land Use, Part A: Analytical Framework’, Ecology and Society , 16/ 1: 27. Google Scholar CrossRef Search ADS 62 Stads G. J., Beintema N. ( 2015) ‘ Agricultural R&D Expenditure in Africa: An Analysis of Growth and Volatility’, European Journal of Development Research , 27/ 3: 391– 406. Google Scholar CrossRef Search ADS 63 Raitzer D. A., Kelley T. G. ( 2008) ‘ Benefit-cost Meta-analysis of Investment in the International Agricultural Research Centers of the CGIAR’, Agricultural Systems , 96/ 1-3: 108– 23. Google Scholar CrossRef Search ADS 64 Andersen M. A. ( 2015) ‘ Public Investment in US Agricultural R&D and the Economic Benefits’, Food Policy , 51: 38– 43. Google Scholar CrossRef Search ADS 65 Lilja N., Dixon J. ( 2008) ‘ Responding to the Challenges of Impact Assessment of Participatory Research and Gender Analysis’, Experimental Agriculture , 44/ 1: 3– 19. 66 Kristjanson P. et al. ( 2009) ‘ Linking International Agricultural Research Knowledge with Action for Sustainable Development’, Proceedings of the National Academy of Sciences United States of America , 106/ 13: 5047– 52. Google Scholar CrossRef Search ADS 67 Upton S., Vallance P., Goddard J. ( 2014) ‘ From Outcomes to Process: Evidence for a New Approach to Research Impact Assessment’, Research Evaluation , 23: 352– 65. Google Scholar CrossRef Search ADS Footnotes 1 The exact TOPIC query was: agricult* NEAR/1 (research* OR *scien* OR "R&D" OR innovati*) AND (research* OR *scien* OR "R&D" OR innovati*) NEAR/2 (impact* OR assess* OR evaluat* OR criteria* OR benefit* OR adoption* OR adaptation*) 2 The exact TITLE query was: agricult* AND (research* OR *scien* OR "R&D" OR innovati*) AND (impact* OR assess* OR evaluat* OR criteria* OR benefit* OR adoption* OR adaptation*) © The Author 2017. Published by Oxford University Press. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted reuse, distribution, and reproduction in any medium, provided the original work is properly cited.
Research Evaluation – Oxford University Press
Published: Jan 1, 2018
It’s your single place to instantly
discover and read the research
that matters to you.
Enjoy affordable access to
over 12 million articles from more than
10,000 peer-reviewed journals.
All for just $49/month
Read as many articles as you need. Full articles with original layout, charts and figures. Read online, from anywhere.
Keep up with your field with Personalized Recommendations and Follow Journals to get automatic updates.
It’s easy to organize your research with our built-in tools.
Read from thousands of the leading scholarly journals from SpringerNature, Elsevier, Wiley-Blackwell, Oxford University Press and more.
All the latest content is available, no embargo periods.
“Hi guys, I cannot tell you how much I love this resource. Incredible. I really believe you've hit the nail on the head with this site in regards to solving the research-purchase issue.”Daniel C.
“Whoa! It’s like Spotify but for academic articles.”@Phil_Robichaud
“I must say, @deepdyve is a fabulous solution to the independent researcher's problem of #access to #information.”@deepthiw
“My last article couldn't be possible without the platform @deepdyve that makes journal papers cheaper.”@JoseServera