TY - JOUR
AU1 - Cañibano,, Carolina
AU2 - Vilardell,, Immaculada
AU3 - Corona,, Carmen
AU4 - Benito-Amat,, Carlos
AB - Abstract This article addresses how the progressive development of a strong research evaluation system in Spain has affected research practices in two highly prestigious university departments in History. Our research finds that researchers perceive that evaluation mechanisms relying on bibliometric indicators pressure them to publish more and to publish in international indexed journals. Researchers feel the time available to develop their research before publishing their results has been reduced, which does not allow them to address research questions with the former rigour and to fully exploit information sources. The pressure is also perceived as an obstacle to conceiving long-term intellectual projects. The strong evaluation culture and its accompanying mechanisms are thus perceived as directly undermining the fundamental pillars on which the academic prestige of these departments was built, namely originality, scientific rigour and methodological innovations. 1. Introduction Universities played a key role in the reconstruction of the Spanish society after the end of Franco’s regime in 1975. Their major contribution was the rebuilding of the critical spirit and the regeneration of intellectual stimuli that spread throughout classrooms and departments. According to Baldó-Lacomba (2012), the intellectual and cultural opening up that accompanied and made the political transition possible was primarily led by reflexive academics who were open to new ideas and connected to the modernity of their time. These academics embarked on new paths and raised new questions, looking at Europe and at the rest of the world through a different lens (Baldó-Lacomba 2012: 212). These academics often led the departments that were either created or reconstructed in the 1970s. They trained a generation of students who were highly politically active and intellectually engaged. In particular, the Humanities schools are well known for having hosted much of the ‘student movement’ of opposition to the dictatorship during the 1960s and 1970s (Puyol 2002). This article focuses on two departments in History embedded in human sciences schools that became very prestigious for the quality and innovative character of their work during the democratic transition. Over the years, they have been able to maintain their academic reputations in Spain and to become internationally visible. The current institutional landscape is completely different to the one prevailing in the 1970s. The Spanish university has faced the subsequent challenges of mass higher education and rapid growth in student numbers particularly from the 1970s. A significant number of legislative changes and reforms, the progressive institutionalization of a quality evaluation system, and the deep budgetary cuts extending from the recent economic crisis have impacted universities in profound ways. This article focuses on how the implementation of what may be considered as a ‘strong research evaluation system’, according to Whitley’s (2007) definition, has affected the dynamics of knowledge production in two History university departments in Spain. According to Whitley (2007: 9) a weak research evaluation system (RES) is characterized by low levels of formalization and standardization of evaluation rules and procedures. In weak RES, evaluation assessments are used as informative instruments to encourage improvements in organizations but not to rank research groups or to directly determine research funding decisions. In contrast, strong RES are characterized by highly formalized and standardized evaluation rules and procedures and by the use of evaluation assessments to rank research groups and to make decisions regarding funding. In Spain, the development of a strong RES has gone hand-in-hand with the generalization of an understanding of academic quality as a form of work performance that is susceptible to being measured and assessed by means of standardized metrics, and compared using quantitative indicators that are most often based on bibliometric techniques (Paradeise and Thoenig 2015). Since the late 1980s, Spain has progressively developed academic quality evaluation schemes and institutionalized behavioral norms that have profoundly affected academic life and careers. In this article, we investigate how these changes have affected two prestigious History departments in Spain. In particular, we question how the advancing tide of research quality and excellence evaluation has affected research practices and knowledge production in these two departments. Whitley argues that the implementation of a strong RES is likely to have a number of effects on knowledge production, which are accentuated in systems where few research funding agencies and few competing evaluation systems exist (Whitley 2007: 10–19). This is the case in Spain. He further argues that effects will also vary across scientific fields and will be accentuated in fields characterized by a large variety of research goals and approaches, and a large diversity of research audiences (Whitley 2007: 19–24), which characterizes the social sciences and humanities (Hammarfelt and De Rijcke 2015: 3–4). Among the specific effects of strong RES on knowledge production Whitley identifies the decline ‘in the diversity of intellectual goals and approaches (…), especially where they challenge current orthodoxies’ (Whitley 2007: 10). In their review of the literature on effects of evaluation practices and indicator use, De Rijcke et al. (2016) also identify the following potential effects of the use of metrics and assessment systems in academic evaluation: (1) avoidance of risk in the selection of research topics, (2) displacement of research quality considerations in favor of quantitative goals when funding is linked to the number of publications, (3) negative effects on interdisciplinary work, (4) abandonment of certain tasks and particularly devalued types of work in favor of publication oriented activities, and (5) changes in publication behavior. De Rijcke et al. also acknowledge that the above effects have been ‘scarcely documented and analyzed in empirical research’ and that there is a lack of empirical studies addressing how evaluation systems affect researchers, and how researchers respond to them (Hammarfelt and De Rijcke 2015; De Rijcke et al. 2016: 161). In this article, we contribute to partly filling this knowledge gap, by focusing on the effects of the implementation of a strong RES on researchers in two History departments in Spanish universities. In particular, we address the effects of the evaluation system on individual practices by collecting researchers’ perceptions regarding changes in research practices and goals and publication strategies. Publications data for the departments studied allow us to compare, to a limited extent, whether perceptions regarding these changes match the actual evolution of published output.1 In Section 2, we briefly describe the progressive institutionalization of the RES in Spain over the last 30 years. Section 3 describes the data and methods used for our empirical study, while Section 4 presents the History of the departments under focus. Section 5 presents the results of the empirical study, which focus on the original sources of academic prestige and the perceived effects of the evaluation system. Individual perceptions are compared and contrasted with the information provided by bibliometric information regarding the evolution of research output in these departments. Section 6 concludes and discusses the results. 2. From prestige to excellence: implementation of the Spanish RES The transition from the so-called ‘old university’ to the ‘new university’ can initially be situated in the decade of the 1960s, due to the emergence of new social and economic conditions and new regulation frameworks (Hernández-Díaz 1997). The 1960s are known as the ‘development years’ induced by the progressive international opening up of the economy, which had remained totally isolated between the end of the civil war in 1939 and 1959. In the 1960s and 1970s, Spain opened its doors to imports that allowed the modernization of industry, to foreign investment, tourism, and inflows of emigrants remittances. New universities were created at the end of the 1960s to respond to new social needs and the General Education Law was approved in 1970 as an attempt to bury the old ‘Francoist’ university (Hernández Díaz 1997). However, it was not until after the transition to democracy, that the higher education system was reformed, with the recognition of Universities’ autonomy in the Spanish Constitution of 1978 and the University Reform Law of 1983. This law explicitly considered academic quality as a necessary requisite and as part of universities’ accountability to society. The Reform Law acknowledged the teaching and research missions of universities for the first time (BOE 1983). The culture of evaluation and competition in the access to public resources for the funding of research was initiated with the approval of the Science Law of 1986 (BOE 1986). In the same year, the National Agency for Evaluation and Foresight (ANEP) was created, to respond to the need for independent scientific evaluation mechanisms to support decisions regarding the funding of research projects and other R&D initiatives. The National Commission for the Evaluation of Research Activity (CNEAI) was created in 1989. This was the first major step toward the institutionalization of processes for evaluating research performed by academic staff in universities and by research personnel employed at the Spanish National Research Council (CSIC). The mission of the CNEAI is to validate the research performance of individual tenured researchers working in academia and public research organizations on a 6-yearly basis (‘sexenios’ in Spanish). The CNEAI evaluation process was originally conceived to identify academics that were ‘research active’ and to recognize this activity with a modest salary increase. Over the years, sexenios have been increasingly used as a criterion required for supervising and evaluating doctoral researchers and for participating in decision-making committees and recruitment panels. Following the approval of a governmental decree in 2012, academics without an active sexenio (i.e. having failed to obtain a positive research evaluation for the most recent prior 6 years) have a substantially higher teaching load (BOE 2012).2 Thus, sexenios are currently a powerful instrument that determines salaries, individual prestige, access to research resources, and the teaching loads of academics.3 The activity of the CNEAI is one of the most important elements of the implementation of the Spanish RES, which has a strong focus on the retrospective evaluation of individual researchers (Cruz-Castro and Sanz-Menéndez 2007; Osuna et al. 2011). According to Jiménez-Contreras et al. (2003) and to Ruiz-Pérez et al. (2010), the CNEAI evaluation mechanism has encouraged research activity, international publications, and the standardization of evaluation procedures. However, while Osuna et al.’s study (2011) supports the idea that the CNEAI has most likely played a role in the standardization of publication patterns and in the progressive internationalization of publications, they do not find evidence of a significant increase in publication productivity following the creation of the CNEAI. In 2001, a new Law, the ‘Ley Orgánica de Universidades’ (LOU), was approved. One of the most important objectives of the LOU was ‘to improve the quality of the university system in all its dimensions’ (LOU, Chapter I, BOE 2001). The LOU transformed the existing structures for the measurement and evaluation of quality by creating a new agency, The National Agency for Quality Evaluation and Accreditation (ANECA). To promote academic excellence, the LOU also established a new accreditation system to guarantee that recruitment processes would prioritize the recognition of merit and achievement. ANECA is the responsible agency for individual accreditation processes that gives access to academic contracts and civil servant positions, and also for the evaluation of teaching programs. Many Spanish Autonomous Communities (CAs) have also created regional evaluation agencies that complement (and often duplicate) the functions of ANECA. The two regions where the departments studied in this article are located have a regional evaluation agency. The rise of evaluation structures and procedures has caused an increase in evaluation standards and a complication of criteria, norms, and administrative processes (Cruz-Castro and Sanz-Menéndez 20074). Of all the evaluation agencies created, it is the CNEAI that establishes the minimum criteria in all fields and in all regions that is necessary for obtaining a positive research evaluation for a given period. The indicator of quality preferred is the number of articles published in journals that are indexed in ISI Journal Citation Reports. For some fields, other forms and formats of diffusion of research results are also recognized. However, in general terms, the evaluation criteria in force have been strongly opposed and criticized by researchers from the Humanities on the grounds that they do not take into account the particularities of Humanities fields, notably their local and culturally embedded character and the choice of Spanish as the preferred language for diffusion (FECYT 2006: 16). Of course, this is not a particularly Spanish problem; rather it derives from the specificities of research in the Human and Social Sciences for which bibliometric indicators are of limited use (FECYT 2006; Hicks et al. 2015). In the case of History, the CNEAI limits ‘quality production’ to: (1) articles published in journals indexed in the Web of Science or well-positioned in Scopus; (2) books, and chapters in books, published by ‘prestigious publishers’ that accept manuscripts following the procedures described in the Scholarly Publisher Indicators;5 (3) other productions including mainly reviews and conference proceedings. A major remaining point of controversy is the lack of recognition of contributions to local journals that are favored for research where the topic concerned is locally or regionally focused. A researcher who wishes to apply for an evaluation of a 6-year period of research activity must select the top five contributions of the period.6 The minimum requisite to obtain a positive evaluation in History is that: (1) at least one of the contributions is a book that is internationally referenced and disseminated; or (2) two of the contributions are articles published in journals indexed in international databases; or (3) one contribution is an article published in an indexed international journal and another one is a chapter in a book fulfilling the requisites to be considered a ‘quality publication’ (BOE 2015). The supremacy of high impact publications persists in the criteria of evaluations performed by regional evaluation agencies and by the ANECA to grant access to specific job categories. Currently, the criteria applied to grant access to career entry positions (assistant professor, contracted doctor) weights articles published in international journals or in prestigious national journals more heavily than books and book chapters, which reflects the ‘Journal Citation Report-centrism’ of the system (Ruiz-Pérez et al. 2010). Against this background, our research question addresses how the evaluation system and culture that has been consolidated since the approval of the LOU in 2001 is affecting the practices and strategies of researchers in History. The researches studied belong to two highly prestigious Spanish university departments in the field. The following section describes the data collection and analysis methods employed. 3. Data and methods The data for this article derive from a series of in-depth narrative interviews conducted in two university History departments located in two different regions of Spain. The study is part of a research project funded by the Spanish National Plan for R&D + I, which adapts to the Spanish case the research questions and methodology of the seminal French project ‘Prest/Ence: from Prestige to Excellence, The fabrication of academic quality’.7 In Spain, excellent departments were selected on the basis of their position in international and national university rankings. The universities to which the selected departments belong are classified in the range 151–200 for the History subject, according to the QS international ranking by subject. In several national rankings, both universities occupy one of the top five positions for the History subject. In addition, both departments were selected on the basis of opinions and recommendations from members of the corresponding scientific community regarding the most prestigious departments in their academic discipline. These opinions are confirmed in part by the fact that both departments have been recipients of prestigious research awards in the field (i.e. National History Award, Paul Adams Award). Interviews attempted to cover a broad range of academic categories and jobs, including faculty, doctoral students, and supporting administrative staff. A total of twenty-five interviews were conducted between 2014 and 2015: eleven interviews in Department A and fourteen interviews in Department B. Table 1 shows the distribution of interviewees by professional category: Table 1. Distribution of interviewees by professional category. Category Interviewees Permanent academic staff Full Professor 11 Associate Professor 5 Non-permanent academic staff Postdoctoral fellow 2 Lecturer 3 PhD student (teaching/research assistant) 2 Scholarship researcher 1 Category Interviewees Permanent academic staff Full Professor 11 Associate Professor 5 Non-permanent academic staff Postdoctoral fellow 2 Lecturer 3 PhD student (teaching/research assistant) 2 Scholarship researcher 1 Table 1. Distribution of interviewees by professional category. Category Interviewees Permanent academic staff Full Professor 11 Associate Professor 5 Non-permanent academic staff Postdoctoral fellow 2 Lecturer 3 PhD student (teaching/research assistant) 2 Scholarship researcher 1 Category Interviewees Permanent academic staff Full Professor 11 Associate Professor 5 Non-permanent academic staff Postdoctoral fellow 2 Lecturer 3 PhD student (teaching/research assistant) 2 Scholarship researcher 1 The semi-structured interviews focused on each individual’s trajectory and experience within the department and the university, and on collective and individual strategies to meet the requirements of quality evaluation systems. Research topics, methodologies, and collaboration and publication practices were discussed, along with the department’s organizational strategies to improve the quality of research and teaching—with a special focus on recruitment and promotion strategies. Interviews with professors with a lengthy tenure in the department allowed us to address in detail the evolution of practices over time and the changes in organizational and individual behavior encouraged by the modification of institutional and regulation frameworks. Interviews lasted between 60 and 90 min and were conducted in Spanish and Catalan. They were recorded and fully transcribed. They were analyzed using qualitative content analysis (Gläser and Laudel 2013) and then coded and managed using N-VIVO software for qualitative data analysis. Content analysis is built inductively throughout the process of analyzing each interview (Becker 1970). The themes extracted allowed for a high level of comparability between the two departments studied and between the different job categories included. In keeping with the standard practice of qualitative studies, this article provides quotes to illustrate the results. However, individual quotes are confidential in nature and masking has been used in some places to ensure it is not possible to identify the interviewer or their department. Similarly, only individual job categories are provided to identify the seniority and tenure of participants. To complement our interview data, we use bibliometric information to address the evolution of the characteristics of the publication output in the two departments. This is done while also taking into account the specific characteristics of publication practices in the Humanities, which means it is not appropriate to use the large international bibliographic databases. Instead, we use Dialnet, which is the most comprehensive bibliographic source of Spanish scientific production. Dialnet was launched in 2001 as a cooperative effort of most Spanish university libraries. It now contains >5.2 million documents including scholarly journal articles and magazines, books, conference papers, dissertations, and other scholarly documents.8 To analyze the evolution of the publication output of the departments under study, we first identified current department members from the relevant department websites. Setting no starting date limit, we then used the author names retrieved to search Dialnet and downloaded all resulting records up to 2014, except those corresponding to book reviews, dissertations, and interviews published in the media. The resulting output set contains 2,615 items for Department A and 1,062 for Department B. Table 2 shows the distribution and percentages of the different types of publications retrieved for the two departments combined. Table 2. Publications retrieved from Dialnet for Departments A and B, by type of output, for the period 1972–2014. Books % Books Book chapters % Book chapters Journal articles % Journal articles Total 596 16% 1151 31% 1930 52% 3677 Books % Books Book chapters % Book chapters Journal articles % Journal articles Total 596 16% 1151 31% 1930 52% 3677 Table 2. Publications retrieved from Dialnet for Departments A and B, by type of output, for the period 1972–2014. Books % Books Book chapters % Book chapters Journal articles % Journal articles Total 596 16% 1151 31% 1930 52% 3677 Books % Books Book chapters % Book chapters Journal articles % Journal articles Total 596 16% 1151 31% 1930 52% 3677 4. Historical evolution of the university departments under study The two departments studied belong to universities that follow the classical and comprehensive European model, conducting teaching and research in a large number of academic fields. The two departments have a similar number of faculty members. Given that they belong to the Humanities, they impart teaching in multiple undergraduate degrees and collaborate with other departments in the provision of postgraduate training. Department B belongs to a five-century-old university, while Department A belongs to a university born in 1968. Nevertheless, it is possible to identify a common starting point in recent History for both departments, in the decade of 1970, characterized by the rapid increase in the number of students enrolling in higher education. The socio-political context in which we situate the birth of the two departments is characterized by the economic and intellectual openness that emerged in the transition to the democratic regime in the 1970s. Department A was formally established in 1968, at the same time as the university, and started to incorporate faculty, most of whom have spent their entire academic career within it. In the early 1970s, Department B appointed a number of new full Professors who used their positions as a way to transition to other universities. This practice left the department void of leading figures in the second half of the 1970s. The remaining young faculty in the department had to reconstruct it almost ‘from scratch’. We may thus argue that the current identities of both departments were de facto created in the same decade. The two departments also share the presence, either temporary or permanent, of leading academic figures that benefited from the intellectual openness of those years. These academics renewed the way to conceive of their research field. They planted the seeds of the good reputation earned by these departments over the last 50 years. They started research lines and groups and applied new work methods that were subsequently consolidated and diversified. The high level of scientific production of both departments is due to their sustained capacity to gather funding to support their research projects and lines, normally under the leadership of a well-regarded scholar. An additional common feature of both departments is the fact they were created on the basis of the recruitment of internal talent. In the 1970s, research assistants were selected from among the best undergraduate students. They were encouraged to enroll in PhD programs and were automatically incorporated to the Department after the completion of the PhD. Currently, 87 per cent of academic staff in Department B and 75 per cent in Department A obtained their PhD in the same university. Both departments include one particularly outstanding intellectual figure who has supervised the PhD of more than a quarter of the current faculty. The faculty has, therefore, been strongly developed in-house, but the departments were not closed in the sense that a big emphasis was placed on gathering funding to support international training. Encouraging international research visits of the departments’ staff was a collective priority, supported by reasonable teaching loads. Researchers took turns to go abroad for a period and to cover the teaching of colleagues on visits. International openness enhanced their exposure to different interpretative frames and represented an important source of intellectual enrichment. The mobile researchers of that period are today’s full professors. The construction of the two departments has been marked by the changing conditions affecting the recruitment of personnel and access to tenured positions, which in Spain take mainly the form of ‘civil-servant’ academic positions. The Spanish constitution of 1978 defines the general institutional framework for universities and grants them autonomy in the selection of personnel and the administration of resources. However, the management of human resources in public universities is subject to a general legislative framework, currently defined by the LOMLOU 2007,9 containing a number of conditions which define the conditions of access, accreditation, and compensation of research and teaching personnel. The LOMLOU (title IX) establishes two personnel groups: civil servants belonging to the Universities teaching staff; and personnel contracted on either a temporary or permanent basis. The Law and the corresponding regional regulations define the characteristics and salary limits of each professional category, while the universities’ statutes regulate the conditions and procedures for offering positions in the different categories. The LOMLOU establishes that access to contract positions must be open and subject to public contest, with the exception of positions for visiting professors. Access to civil servant positions is mediated by accreditation agencies and also subject to open public contests. Even though the Constitution grants autonomy to universities to provide positions and determine the criteria for recruitment and promotion, the current legislative framework limits the capacity of universities to design their own personnel policies. These limitations have been accentuated in recent years by the budgetary restrictions imposed following the financial and economic crisis, which have severely limited the capacity to recruit new faculty and even to maintain the contracts of temporary academic staff.10 Between the 1970s and the 1990s, the Spanish higher education system went through a phase of growth and expansion that multiplied by three the number of students and academic staff. This period was followed by one of relative stability until the beginning of the financial crisis in 2008, which caused a substantial reduction in universities’ budgets. The most critical years of the economic recession have encompassed a substantial growth in the number of graduates from public university Arts and Humanities departments, in parallel with a substantial decrease in academic staff in public universities. This has been particularly the case with regard to tenured civil servant positions, as shown in Figs. 1–2. Only the number of permanent ‘contratado doctor’ (contracted doctorate holder) positions has grown in the period following the 2001 reform, with these requiring an accreditation granted by the ANECA. Figure 1. View largeDownload slide Total graduates in Arts and Humanities by public universities, 2001–15. Source: Ministry for Education, Culture, and Sport. Figure 1. View largeDownload slide Total graduates in Arts and Humanities by public universities, 2001–15. Source: Ministry for Education, Culture, and Sport. Figure 2. View largeDownload slide Academic staff in public universities, 2004–15. Source: Ministry for Education, Culture, and Sport. Figure 2. View largeDownload slide Academic staff in public universities, 2004–15. Source: Ministry for Education, Culture, and Sport. The departments we are focusing on have applied different policies in the face of these budgetary restrictions, which help to explain the differences in the changing structure of their staff profiles by job category, as shown in Fig 3a and b. Figure 3. View largeDownload slide (A) Academic staff by professional category in Departments A, 2001–15 and (B) Academic staff by professional category in Department B, 2001–15. Figure 3. View largeDownload slide (A) Academic staff by professional category in Departments A, 2001–15 and (B) Academic staff by professional category in Department B, 2001–15. Over the recent years, the university where Department A is located has developed a policy according to which permanent academic staff have not been promoted and the positions of retired personnel have not been replaced. The number of contracted teaching assistants has also been reduced. As a consequence, the number of academic staff in academic year 2015 was lower than what it was in the mid-2000s. Between 2005 and 2016 only three full Professors have been appointed. The policy and strategy followed by Department B’s University have been different. Until 2009, promotion was practically guaranteed to those researchers fulfilling the necessary requisites. Between 2005 and 2012, eight Associate Professors were promoted to full Professors. Overall, the department has twelve full Professors and has managed to maintain the total number of staff in comparison with the mid-2000s. Despite these differences, there is a sense in both departments that the budgetary restrictions endanger the necessary generational transition in research staff and put at risk the prestige accumulated over the years. Members of both departments express concerns regarding the future. Budgetary cuts have also limited substantially the funding available for the support of research. Support for sabbaticals and international mobility has been eliminated and the number of funded PhD students has been reduced, as has the budget allocated to research projects. It is much less attractive today to pursue a doctorate in the field considering the limited options to develop a successful academic career. Despite the difficulties, the academic field and the two departments seem to maintain their attractiveness to prospective students. Economic difficulties and the rise in unemployment encourage people to enroll in graduate programs, but not so much in postgraduate studies in this field. The growth in the number of students along with the reduction in academic personnel makes it particularly difficult to adapt to the requisites of the European Higher Education System. Teaching and administrative loads have increased for researchers, reducing their time available for research—at the same moment evaluation systems are ever more focused on quantitative research output metrics. 5. Changes in the evaluation system and the dynamics of research and academic production 5.1 Factors supporting prestige and good reputation According to the information collected in the interviews, the reputations of the departments under study build on three fundamental pillars: work values and methods, public visibility, and thematic originality. The two departments are characterized by the level and rigour of individual work which adds up to high standards of collective achievement. Faculty members are conscious of the level attained by the previous generations and the need to maintain it ‘… the very high level of the reference figures – who are the older ones in the department - has been a source of emulation and inspiration for all. Because you know that if you don’t reach a certain level, you lag behind and you can neither talk informally nor academically with people that have a much higher level’ (B).11 ‘It’s a continuous process of self-development, of not lagging behind the others and progressively achieving the required levels. I think good quality work builds on scientific rigour and on the respect of a scientific methodology within the discipline, which means selecting and interpreting the sources appropriately’ (B). The dominant work values of the departments’ founders were rigor, integrity, and the search for knowledge, which according to current staff added to their capacity ‘to attract, fascinate and captivate students and to train disciples’ (A). Intellectual leadership was the cornerstone around which research groups were created. The high value and capacity of the people that created the departments and of research groups ‘attracted more valuable young people to them’ (B). A cumulative beneficial effect of these intellectual foundations has thus shaped their evolution over time. The prestige of both departments within their disciplinary community builds on the renewal of the way to conceive their field of research. A researcher from Department B stated ‘what characterizes this department is a renovation of the way to conceive History … the attempt to understand History not only in the classical terms of the description of facts and problems but also to raise questions regarding the social and political fabric underlying those problems’ (B). Regarding Department A, as a faculty member stated, ‘our department became prestigious because a number of researchers and professors were able to initiate work lines that made very important contributions to the field. Not only were they capable of opening up a field of interest but above all to make methodological innovations in their contributions. I think this is what made their work attractive, their methodological proposal and their contributions’ (A). Indeed, one of the differentiating characteristics on which the departments’ prestige was constructed was the originality of work methods. In Department B, a research line was started based on the ‘case study method’. ‘The case study, as in the rest of sciences, was the study of concrete situations in a variety of contexts, which could be addressed from many different sources’ (B). The case study addressed general problems, transcended the Spanish problematic alone, to focus on general questions relevant to the broader European context. One of the virtues ingrained in the subsequent generations of researchers in both departments has been the willingness and capacity to move forward and renew topics and research questions. They have continued opening up new research areas, breaking with consolidated schemes and becoming national and international reference points, including by being the first to address a topic. The thematic originality was partly the result of international mobility, which allowed researchers to become part of international work groups performing other types of work and focusing on other topics. The training received abroad, research visits, attendance at international conferences, and other types of international exchanges have been crucial for the establishment of contacts and the formation of collaboration networks. The national and international presence and visibility of both departments is quite notable and is consolidated through inter-university postgraduate programs, collaborative research projects, and multiple invitations to collaborate in different ways with universities from all regions of Spain and around the world. 5.2 Observed changes in the dynamics of work and production RESs in the Humanities have been conceived following the design of those established for the natural and experimental sciences, in Spain as much as in other European countries (Buela-Casal 2007; Hammarfelt and De Rijcke 2015). Evaluation systems tied to performance-based funding condition the distribution of resources and the development of researchers’ careers. Their inadequacy in relation to the Humanities disciplines has been widely criticized (Whitley 2007; Hicks et al. 2015). The controversy appears justified, when the local character of some of the investigations is considered the preference for publishing some of these results in Spanish and other languages of the Spanish state. The main diffusion channels of research results are books and monographs, which are not included in international reference indexes. Researchers from the departments studied are, therefore, critical of the criteria applied to evaluate their work. However, they have adapted some of their practices to meet these criteria and in some cases also value some of their implications, as we show below. 5.2.1 Ways of researching One of the changes in work practices that researchers pointed out in the interviews, and that has been observed for the whole of the Humanities, is the transition toward more collaborative forms of work and organization. The Humanities are characterized by a tradition of individual work (Must 2012), which has historically resulted in single author publications. In the last 20 years, science policy in Spain has favored the constitution of research groups organized around projects, as the target unit for the allocation of resources. Researchers compete for research funding by designing projects that are developed and applied for by research groups. The system is thus designed in such a way that it is research groups that compete for funding. However, the outcome of research is evaluated ex-post at higher levels of aggregation (institutes, departments, schools, and universities), which conditions the overall distribution of resources in the public system. Evaluation also takes place at the individual level by means of the accreditation processes conditioning recruitment and promotion and by the allocation of salary complements or sexenios, as described earlier. The formation of groups becomes thus important to compete for the funding that supports the production of research outputs that can facilitate good evaluation outcomes at both the individual and collective level. However, one consequence of the tension between the level at which research funding is distributed (the project group) and the fact that it is individuals who are subject to ex-post evaluations which condition their prestige and career development, seems to be the formation of somewhat artificial groups. Some groups are designed to access funding under a generic theme, inside which work is still performed individually. A researcher pointed out that ‘the work dynamic is to have a project under a general theme within which each person works on her specific topic. However one feels the moral obligation to work on something that is connected with the general topic of the project’ (B). Nevertheless, other interviewees acknowledge a change toward non-artificial group dynamics: ‘it’s no longer about getting together to get some money, but about getting together to do better work, to project ourselves, to improve our research because we are working on similar topics or with similar methodologies’ (A). The benefits of collaborative work are further acknowledged when researchers point out that groups ‘allow exchanges and allow doing lots of things that go beyond getting the funding to do them (…) they create spaces for debate, spaces to present your research, seminars, etc. I believe this has been very important because I would have not achieved a tenth of what I have done if it was not for the group’ (B). Individual work remains important, but the departments studied are characterized by the presence of highly consolidated and recognized research groups that are capable to attract sustained funding and that have produced many results over the years, including a significant number of doctoral theses. There is no evidence of an immediate causal effect between the implementation of the RES in Spain and the transition toward more collective and collaborative forms of work in History, but researchers acknowledge the progressive development of a ‘project culture’, of research habits and practices that revolve around funded projects, which we may assume to be somewhat connected with the way in which the science system is organized and the research funding is allocated. 5.2.2 Publication strategies and their implications The production and diffusion of research results in the Humanities have three major peculiarities compared to the experimental sciences: the maturing timeframe, the presentation format and the audience to which they are directed (Nederhof 2006). The work of researchers in History is slow. It requires a considerable investment of time, but its outputs do not become obsolete at the same rate as the results from some other fields (Archambault and Vignola-Gagné 2004; FECYT 2006). Historians in particular, and researchers in the Humanities in general, may use very ancient sources to test their hypotheses and build new theories. This work dynamics differs substantially to work in the experimental sciences, characterized by the extremely short life of results’ utility in many cases. Publication timeframes are also completely different as described by one interviewee: ‘A paper may take an average of two years to be published in a Humanities journal compared to the six months in which a paper can be published in experimental fields. Considering this is different, the way to measure the output of researchers from different fields should also be different’ (A). The use of language is also different. The presentation of research results in History often requires lengthy argumentation that allows authors to show the development of their methodologies. When asked about the role of language and their choices of different publication formats interviewees pointed out that ‘a complex explanation requires space for its presentation and its demonstration. Demonstrations in History are hard to make’ (A). ‘Therefore, a very small detail may require a note of four or five lines to explain that you have the proof for your argument, which is not a mathematical formula. Explanations tend to be extended. The paper format is too short. The products that are really recognized and prestigious are books, which result from trajectories of two or three years working on the same topic. Books are produced after having published a few articles but they are not the sum of the articles. Articles permit us to develop partial ideas that later on will lead to the full picture’ (A). ‘Articles allow us to advertise ourselves and an idea, or a part of an idea, but when the number of words is limited the article often ends up being shorter than it should be’ (A). Monographs are the core output in the Humanities (Nederhof 2006; Cronin 2012; Thelwall and Sud 2014), but interviewees perceive that this differs from the prime currency in evaluations: ‘It is striking that articles in indexed journals are the most valued (…) the evaluation criteria have been established on the bases of the research practices of other sciences. Our research is proved in monographs. You may find it in articles but articles have to be brief which often does not allow you to develop a long argument, which is fundamental to our discipline, to our historic methodology’ (B). The perceived pressure to publish articles rather than books seems to be generalized among interviewees who feel that having visibility means publishing in prestigious international journals: ‘The only possibility to have a future in the field is to publish articles in international high impact journals. It is the only way to compensate for the fact of not being a Doctor from Yale and to have some visibility, as much here as outside’ (B). Researchers from older generations feel the pressure and declare they have modified their publication strategies. A full Professor stated: ‘We write now many more articles than we used to. We target certain journals, not only for the interest in externalizing our research but also because this will count when applying for research projects or sexenios’ (A). Researchers also say that they have not stopped publishing books or book chapters, ‘but we publish many more articles because the system forces us to go that way’ (A). The evaluation criteria are perceived as excessively quantitative and reductionist. Interviewees feel that there is no flexibility to adapt to the characteristics of different scientific inquiry methods and that ‘evaluation scales should be flexible’ (B). In addition, the pressure to publish articles in internationally indexed journals entails a number of difficulties which were stressed by interviewees. The change in language is the most immediate difficulty (and is addressed further in the next subsection), but it also implies that researchers have to learn a different type of academic language, which according to an interviewee ‘is not easy because we have been trained in a certain way because our journals have a certain focus. Academic standards are different’ (A). An added difficulty identified with regard to publishing in international journals is national bias. A researcher stated I have tried to publish in the United States. I have invested a lot of money to translate the article. I have tried to make it suitable for over three years. I keep trying and it keeps being rejected. I talk about this personal example because it upsets me. Because we write an article with a Hispano-American perspective, we send it to a journal in the United States, one of the top 10. We discuss it over and over among the authors. My co-author asked why don’t we go for something less ambitious? No, because it is the first time and we need to know where the threshold is (…) The experience has been devastating, our pride is hurt. Some of the comments received were to do with parochialism and partisanship, like ‘you are not one of ours’. This is terrible because there is a lot of effort behind it. There are too many obstacles. If it is only about quality I am ready to stand up as many times as I am rejected to reach the required level but there are other things. I feel like to attain the required standards of quality and excellence I have to overcome obstacles that have nothing to do with quality and excellence (B). The debate between research quality and output quantity has become a priority question in the discipline. The evaluation system is perceived as conditioning the way work is done and published in the field and encourages a generational divide between senior faculty and younger researchers. Senior researchers did not start their careers under the pressing agenda marked by the need to publish. The dominant values they were trained with were the production and sharing of knowledge whereas young researchers ‘are trying to tick the boxes’ (B). Nobody wants to publish in local journals, not even research assistants. The general aspiration is to publish in high impact journals to be awarded more ‘points’ by evaluation agencies. Young doctors and PhD students have to respond to many demands. A doctoral student stated ‘on top of teaching and writing my thesis I have to go to conferences and I should try to be part of some administrative commission to fulfill all the requisites for the accreditation process’ (B). Thus, career strategy may be in conflict or tension with intellectual development. Those with the ‘most competitive’ curricula may be those that have better organized their careers, not necessarily those who are most intellectually capable. Referring to younger generations, a senior researcher reported ‘the ability to sell their research product has become more important than its intrinsic quality’ (B). Another researcher said ‘what matters is to augment the production and its publication. Evaluation criteria affect the orientation of what researchers want to publish and where they want to publish’ (B). The previous quote addresses the effects of evaluation criteria on the actual content of work. The combined pressure to publish in a certain format (articles) and at a certain rate of productivity is perceived as affecting the quality and content of research. According to a senior researcher ‘there is no time to elaborate solidly founded and well documented work’ (B), which makes theoretical work more complicated. The same researcher states that in many articles ‘theoretical frameworks are reutilized; the stating of the questions is repeated’ (B). A teaching associate also pointed out that many articles ‘do not contribute much, they do not allow the discipline to progress, do not contain substantial knowledge … they reproduce theoretical approaches or address very limited empirical problems that are not really significant …. The journal [article] system does not work for History’ (B).12 In sum, the researchers interviewed perceive pressure coming from the evaluation criteria to which they are subjected, particularly to publish journal articles (preferably international) rather than books—which are traditionally the most prestigious type of publication for research in History. In addition, they feel a pressure to be productive (meaning publishing a certain unspecified number of articles). They have the perception that these pressures are affecting the quality of their work because the imposed publication formats and production timeframes are not suitable to the specificities of their discipline. 5.2.3 Research content and language The choice of research topics and questions is subject to a number of conditioning factors. History could be characterized as a fragmented and rural research field according to Whitley’s (2007) definitions. It is fragmented in the sense that topics and methodologies are varied and so are the audiences to which results are addressed. The level of coordination in the field is weak (Hammarfelt 2016). It is rural in the sense that the concentration of researchers working on the same topic is generally low. According to Whitley (2007), the implementation of a strong RES is likely to reinforce concentration of focus in certain problems, therefore, reducing fragmentation, and to diminish attention on peripheral questions. This plays against topics with a markedly local or territorial character, which in Spain are often addressed in local languages. As mentioned, many of the research results in the Humanities are published in Spanish or in other regional languages, which makes international visibility more difficult (FECYT 2006: 16; Soriano 2008). The empirical evidence collected in our two departments points to an increased preference for transnational topics which are more likely to: (1) make international research visits more profitable, (2) increase the visibility and impact of the research, and (3) increase the likelihood of publishing in international indexed journals. Interviewees perceive that internationalization is necessary to maintain their academic prestige. As mentioned earlier, intellectual openness is what characterized the construction of both departments from the start, which contributed to developing ‘a large variety of historiographic perspectives’ (B). Openness was achieved through internationalization and collective efforts to promote and support international mobility. Among the transnational topics mentioned by interviewees when discussing their international experiences and networks are the history of women, the relationships between the Spanish and Italian fascist regimes, agricultural history and American history. A researcher stated that ‘the internationalization projection should be an important priority; a lot has been done over the last ten years but a lot remains to be done; it is fundamental to maintain the prestige’ (B). The same researcher said that ‘the pressure is on, not only in terms of the publications but also in terms of conducting research elsewhere. We now have more doctoral students writing theses on the history of other countries. They are not only capable of writing a thesis about Catalonia or Spain. This would have been unthinkable 15 years ago’ (B). An interviewee acknowledged the benefits of internationalization for individual performance evaluations by recognizing that one of the effects of international visits is that they increase ‘the possibilities to obtain sexenios’(B). Publishing internationally is also perceived as important since ‘not publishing in English has consequences for the profession. It means that even if Spanish research is of very high quality it remains unknown to researchers who do not read Spanish or the rest of our local languages. This implies that the interpretation of a particular problem will be biased as it will leave aside the contribution made here. Spanish researchers who wish to have an international impact have to publish in English’ (A). Researchers are aware that addressing transnational topics increases their chances to publish and to be positively evaluated. But it is not possible to derive from the evidence collected a strong connection between the implementation of the RES and the move toward more transnational topics in the field. Rather than being seen simply as an instrumental ‘rite of passage’ (Ackers 2013) necessary to overcome evaluations and accreditations, internationalization also seems to be perceived as a way to open up the mind and its perspectives, to establish contacts, to maintain academic prestige and to make work visible and accessible to the international community. In addition, the evidence collected does not allow us to identify a connection between evaluation practices and a decline in interdisciplinary work in History. As described earlier, De Rijcke et al.’s review of studies of RES impacts identifies bias against interdisciplinarity among the reported effects. Rather, one interviewee saw interdisciplinarity as a possible way to increase publication productivity in the future: ‘I work on the same topics as some colleagues from other disciplines, especially economics, we share materials, we have discussions, is this interdisciplinarity? No. Should we become more interdisciplinary? Yes, because we would be able to publish in my field since I am a historian and in my colleague’s field, because he is an economist and we would manage to improve our career and gain many points. We have to think about it. It did not seem serious to us to work like this but it might be the way to work now’ (B). Some researchers also seem to perceive that interdisciplinary topics or teams are more likely to obtain project funding under the current system: ‘it seems like if you don’t submit an interdisciplinary project you don’t get funded’ (B) ‘If I think about the big research projects … I believe they are more and more interdisciplinary because research topics are becoming more open’ (B). These statements suggest that rather than discouraging interdisciplinarity, the Spanish RES seems to make researchers think it is an interesting option that can advantage their careers. Even Historians, who have not traditionally been inclined to do interdisciplinary work, pointed this out in several interviews. 5.2.4 Publication output In this section, we complement our qualitative data, regarding perceptions of changes in research practices and of pressures from evaluation systems, with data on publication outputs from each department over the period 1972–2014. All figures shown portray the results of data extracted from the Dialnet database on the type of publication and the scope and quality of journals. The figures aggregate the production of both departments studied. Fig. 4 shows a steady increase in the volume of output over time. It is indeed the production of journal articles that has grown the most, although book chapters tracked this growth until quite recently. Books and book chapters show a very similar pattern of evolution until the early 1990s, after which the production of books stabilized. We observe a decline in the overall publication of books in recent years, from thirty-six books between the two departments in 2008, to fourteen in 2014. The two departments published seventy-seven journal articles in 2001, eighty-one in 2008, and eighty-six in 2014. In line with our participants’ descriptions, the publication of monographs seems to be in decline. Journal articles have been the predominant type of publication throughout almost the entire period. Figure 4. View largeDownload slide Total number of publications, by type of publication, 1972–2014. Figure 4. View largeDownload slide Total number of publications, by type of publication, 1972–2014. Fig. 5 shows the evolution of per-author output by type of publication and year. Much of the increase registered in total output is due to the enlargement of both departments over time. The series starts with four authors in 1972, rises to forty authors in 1990, and continues increasing to finish with 100 authors between the two departments in 2014. Per capita output registers large variations between years. Overall, we observe an increase in the per capita production of articles and book chapters after 1994. Before that year the number of yearly articles oscillated around 0.6. Since the end of the 1990s, the production moves to around one article per author and year. We do not observe, however, an increasing trend in per capita output over the 2000s. The number of registered articles per author per year is the same in 1999 as in 2014: 0.9. The publication of books per author has oscillated around 0.3 per year throughout the entire period. We observe a decline in recent years that situates this level at 0.2 books per author per year. The most important rise in productivity seems to date from the mid-1990s. The evidence of increased production following the approval of the LOU in 2001 and the creation of evaluation agencies and accreditation systems is somewhat mixed. Comments such as ‘we write now many more articles than we used to’ or ‘we publish many more articles because the system forces us to go that way’ therefore, need to be interpreted with caution. They might well refer to a more elongated comparison of practices between the early 1980s and the present. Figure 5. View largeDownload slide Publications per author and year by type of publication, 1972–2014. Figure 5. View largeDownload slide Publications per author and year by type of publication, 1972–2014. Fig. 6 shows the scope of the journals in which the articles of the two departments were published over the period 1972–2014, distinguishing between journals published by local or regional publishers, including the university and the department itself, national journals, and international journals. International publication was practically non-existent until the early 1990s, when the output was equally divided between national and more local journals. Since the 1990s, publication is mainly in domestic national level journals. During the 2000s, publication in international journals has remained at around the 5 per cent of total published articles. Despite the incentives of evaluation schemes regarding international publications, and the fact that researchers talk of international publishing as a necessary target to achieve, international publication of articles remains a rather limited practice in these departments. Figure 6. View largeDownload slide Publication of journal articles by scope of the journal, 1972–2014. Figure 6. View largeDownload slide Publication of journal articles by scope of the journal, 1972–2014. We also classified articles using the Spanish classification system used for research evaluation: Integrated Classification of Scientific Journals (CIRC)13 and by publication year. The CIRC classification is used by Spanish evaluation agencies and classifies a total of 20,831 domestic and foreign journals. Journals indexed in international multidisciplinary databases like the Social Sciences Citation Index and Scopus are ranked higher and grouped as Categories A and B of the CIRC classification. Journals meeting the criteria to be included in other databases such as ERIH or Latindex are classified in Category C by CIRC. Journals and periodicals not included in such indexes are grouped in Category D. We classify publications according to these categories and distinguishing between the sum of Categories A and B, Category C and Category D. These results are shown in Fig. 7a and b. The period prior 1986 is characterized by the predominance of publications in Category D journals. Between the mid-1980s and the early 2000s publications in journals classified as A and B dominated, spiking in 1998. Over the 2000s, the weighting of publications in the three categories of journals has been even, with small variations between years. Figure 7. View largeDownload slide A. Percentage distribution of articles by journal CIRC category, 1972–2014 and B. Total journal articles published by journal CIRC category, 1972–2014. Figure 7. View largeDownload slide A. Percentage distribution of articles by journal CIRC category, 1972–2014 and B. Total journal articles published by journal CIRC category, 1972–2014. These data suggest there may be an association between the introduction of the evaluation system in 1989 and the rise in the number of publications indexed in international multidisciplinary databases. However, the nature of this association remains somewhat unclear. According to Osuna et al. (2011: 588) the CNEAI has contributed to the maturation of the Spanish Research System by playing a role in the standardization of publication patterns and the encouragement of the switch to English in scientific communication. For the departments under focus, we observe that the mid-1980s mark the beginning of very modest levels of international publishing. However, there is little evidence of change in the journal impact and geographical scope of publication output following the more recent institutional changes, in particular the creation of the ANECA and the introduction of the accreditation system to access academic jobs in 2001. 6. Discussion and conclusions This article has addressed the effects of the institutionalization of a strong RES. Whitley characterizes a strong RES as having highly formalized and standardized evaluation rules and procedures and using evaluation assessments to rank research groups and to make decisions regarding funding. In many countries, the implementation of a RES has gone hand in hand with the progressive development of tools and indicators ‘measuring outcomes of investments in human resources’ (such as number of degrees delivered or bibliometric indicators) (Paradeise et al. 2009: 215). Spain joined this trend in 1989 with the creation of the National Commission for the Evaluation of Research Quality. Our study has assessed the effects of this specific RES upon the research practices of researchers working in two prestigious university History departments in Spain. The university department studied have a very good reputation in their field among students and national peers. They are embedded in international collaboration networks and are well placed in national rankings and field-specific international rankings. Technically, they were both established in the years of the transition to the democratic regime—1970s—characterized by bourgeoning intellectual and political activity and by progressive internationalization. Both departments were created and consolidated under the influence and leadership of prominent intellectual figures, committed to the renewal of their research field and to scientific rigor and integrity. These leaders were able to attract and train enthusiastic students the most brilliant of whom were incorporated into the departments, which were growing very rapidly due to the expansion of higher education in those years. According to the testimonies of the members of these departments, the prestige acquired and maintained over the years is mainly due to their thematic and methodological originality, their commitment to scientific rigor, and their national and international visibility. Researchers from these departments do not give the impression of resenting the evaluation system. The quality of their work allows them to easily satisfy evaluation and accreditation processes. The absence of strongly negative sentiments toward the evaluation system makes these departments good settings to consider the effects of the evaluation system. The results obtained from the study do not derive from the perceptions of researchers unable to obtain positive evaluations of their work, but rather the contrary. Our study has been guided by the work of Whitley (2007) and De Rijcke et al. (2016) regarding the effects of strong RES on individual research practices and knowledge production. Derived from their work, we expected that a strong RES is likely to: (1) cause a decline in the diversity of research goals and approaches and encourage risk aversion in the selection of research topics, (2) cause changes in publication behavior, (3) displace research quality considerations in favor of quantitative goals when funding is linked to the number of publications, (4) have negative effects on interdisciplinary work, and (5) cause the abandonment of certain tasks in favor of publication oriented activities. The results from the qualitative study provide evidence supporting (1), (2), and (3) to certain extents. We do not find evidence supporting the perception of effects undermining interdisciplinary work or causing an abandonment of particular tasks. Our research found that the focus of evaluations on the production of articles published in internationally indexed academic journals is perceived as a factor limiting originality and restricting innovation in topics and research questions, which was a crucial factor for the initial development of prestige in these departments. Each academic journal in the field is identified with certain theoretical frames and ways of understanding. The incentive (or obligation for young researchers) to publish in certain journals is perceived by interviewees as implying the necessity to locate topics and questions into well-established frames and to stay away from theoretical inquiries and innovations. According to our interviewees, the evaluation system encourages theoretical stagnation and repetitiveness. It is also perceived as imposing work time frames that limit the capacity to work toward long-term intellectual projects and objectives, which undermines future intellectual aspirations within the field. Interviewees also suggested that the RES had caused changes in publication behavior. They perceive that the system privileges articles in international indexed journals and that they are now forced to publish many more articles than years ago even when books and monographs are the most prestigious and preferred mechanism for the diffusion of research results in the field. In particular, young researchers perceive they do not have a choice; they must publish articles in high impact journals and preferably on the international stage. The bibliometric analysis confirms a decline over time in the publication of monographs in favor of shorter contributions such as articles and book chapters. However, journal articles do appear as the most frequent form of publication throughout the period analyzed (1972–2014). Our interviewees find that the pressure to publish papers works against the natural unfolding of rigorous research trajectories in the field, which normally exploits a topic over 2 or 3 years. The ‘journal article system’ is perceived as imposing temporal work time frames that are considered to be too short and publication formats considered too limited for a rigorous and thorough exploitation of information sources and for the correct development of arguments and presentation of methodologies. Evidence thus seems to confirm that the RES is perceived by these Historians as causing a displacement in research quality (rigor, thorough exploitation of sources) in favor of more quantitative goals (publishing a certain number of articles). According to our complementary information regarding per capita publication output and the quality and geographical scope of recipient journals, researchers from the two departments started to publish more after 1989, joining a generalized trend in the production dynamics of the Spanish science system that Osuna et al. (2011) link to its maturation process. The historians under study began to publish modest numbers of papers internationally after that year, and the number of publications in non-indexed journals declined substantially. However, we do not observe significant changes in per capita output, or in the scientific level of recipient journals over the 15 years since the creation of the National Agency for Accreditation and Foresight in 2001. International publications do not seem to be increasing as might be expected according to participants’ voiced expectations. This could be interpreted as some researchers managing to maintain or access positions without meeting the very demanding requirements that researchers consider exist. Another interpretation might be that these departments were already operating quite close to ‘full capacity’ in terms of the volume of their scholarly output. Researchers in these prestigious departments have always felt pressured to achieve high research quality standards. In the years following their creation in the 1970s, pressure came from the culture created and encouraged by their founding and leading intellectual figures: a culture of research rigor and intellectual ambition which showed up in the renovation of research topics and methodologies and in an international vocation. Researchers felt that they had to do good work simply to socialize and ‘be able to talk’ with their colleagues. This included traveling internationally and opening up to other interpretative frameworks. They were subject to ‘weak’ forms of evaluation (Whitley 2007), informal, and unstandardized but apparently highly effective according to the researchers’ discourses and the available bibliometric evidence. The perception of the pressure exerted by the evaluation system instituted in 2001 might not be translating into more outputs, but rather a heavier burden in terms of perceived formal demands, while maintaining what was already a high level of research productivity induced by earlier more informal, internal, and therefore, better accepted assessment mechanisms. These results raise concerns regarding the effects of the Spanish strong RES for fields with similar traditional research practices and work and publication patterns as those described in History, and particularly for highly performing, innovative, or prestigious research groups. For the History departments studied, not only do evaluation mechanisms not seem to be encouraging greater research output or impact (measured in terms of the geographical scope and level of journals), but they are perceived as undermining the fundamental pillars on which the academic prestige of the departments was built: the privileging of innovative topics and of new methodologies, and the rigorous exploitation of information sources. The bibliometric analysis performed has not gone as far as addressing the prevalence of topic or methodological switching (Gläser and Laudel 2016) of the output published by these departments. The extent to which the reported perceptions reflect an observable limitation in academic innovation thus remains an open and important question. Acknowledgements The authors would like to acknowledge the support and guidance received from Prof Catherine Paradeise, coordinator of the seminal project funded by the Agence Nationale de la Recherche in France ‘Prest/Ence: From prestige to excellence. The fabrication of academic quality.’ They also wish to express their gratitude toward all the researchers who have made this study possible by generously contributing with their time and narratives. Funding The Spanish Ministry of Economy and Competitiveness provided the funding to perform this research in the framework of the Spanish State Plan for R&D and as part of the project ‘Prest-Ence Spain: the Spanish University from Prestige to Excellence. An international comparison’ (Grant Number: CSO2013-45377-P). Conflict of interest statement: None declared. References Ackers L. ( 2013 ) ‘Internet Mobility, Co-Presence and Purpose: Contextualizing Internationalization in Research Careers’ , Sociology and Technoscience , 3 : 117 – 41 . Archambault E. , Vignola-Gagné E. ( 2004 ) The Use of Bibliometrics in the Social Sciences and Humanities. Science-Metrix Final Report prepared for the Social Sciences and Humanities Research Council of Canada. <http://www.science-metrix.com/pdf/SM_2004_008_SSHRC_Bibliometrics_Social_Science.pdf> accessed 15 Feb 2018. Baldó-Lacomba M. ( 2012 ) ‘Universidad española y oposición al franquismo: la regeneración universitaria (1956-1975)’. In Torremocha M. (ed.) El estudio general de Palencia. Historia de ocho siglos de la Universidad española . Universidad de Valladolid. Serie Historia y Sociedad . Chapter 9: 211 – 30 . Becker H. S. ( 1970 ) Sociological Work: Method and Substance . Chicago, IL : Aldine . BOE ( 1983 ) Ley 11/1983. de 25 de agosto, de Reforma Universitaria . España : Boletín Oficial del Estado 209 de 1 de septiembre de 1983 BOE ( 1986 ) Ley 13/1986. de 14 de abril, de Fomento y Coordinación General de la Investigación Científica y Técnica . España : Boletín Oficial del Estado 93 de 18 de abril de 1983 . BOE ( 2001 ) Ley Orgánica 6/2001, de 21 de diciembre, de Universidades . España : Boletín Oficial del Estado 307 de 24 de diciembre de 2001 . BOE ( 2012 ) Real Decreto 14/2012, de 20 de abril, de medidas urgentes de racionalización del gasto público en el ámbito educativo . España : Boletín Oficial del Estado 96 de 21 de abril de 2012 . BOE ( 2015 ) Resolución de 26 de noviembre de 2015, de la Comisión Nacional Evaluadora de La Actividad Investigadora, por la que se publican los criterios específicos aprobados para cada uno de los campos evaluación . España : Boletín Oficial del Estado 286, de 30 de noviembre de 2015 . Buela-Casal G. ( 2007 ) ‘ Reflexiones Sobre El Sistema de Acreditación Del Profesorado Funcionario de Universidad En España ’, Psicothema , 19 / 3 : 473 – 482 . Google Scholar PubMed Cronin B. ( 2012 ) ‘ Collaboration in Art and Science: approaches to Attribution, Authorship and Acknowledgment ’, Information & Culture , 47 / 1 : 18 – 37 . Google Scholar Crossref Search ADS Cruz-Castro L. , Sanz-Menéndez L. ( 2007 ) ‘Research Evaluation in Transition: Individual Versus Organisational Assessment in Spain’. In Whitley R. , Gläser J. (eds) The Changing Governance of the Sciences. The Advent of Research Evaluation Systems. Sociology of the Sciences Yearbook , vol. 26 . Springer , Chapter 1: 3 – 27 . De Rijcke S. , Wouters P. F. , Rushforth A. D. et al. ( 2016 ) ‘ Evaluation Practices and Effects of Indicator Use-a Literature Review ’, Research Evaluation , 25 / 2 : 161 – 69 . Google Scholar Crossref Search ADS Fundación Española para la Ciencia y la Tecnología (FECYT) ( 2006 ), Libro Blanco de la Investigación en Humanidades 2006. Madrid. Edición electrónica: <https://www.fecyt.es/es/publicacion/libro-blanco-en-investigacion-en-humanidades> accessed 15 Feb 2018. Gläser J. , Laudel G. ( 2013 ) ‘ Life With and Without Coding: Two Methods for Early-Stage Data Analysis in Qualitative Research Aiming at Causal Explanations [96 paragraphs] ’, Forum: Qualitative Social Research , 14 / 2 : Art. 5 . http://www.qualitative-research.net/index.php/fqs/article/view/1886 accessed 15 Feb 2018. Gläser J. , Laudel G. ( 2016 ) ‘ Governing Science ’, European Journal of Sociology , 57 / 01 : 117 – 68 . Google Scholar Crossref Search ADS Hammarfelt B. , De Rijcke S. ( 2015 ) ‘ Accountability in Context: Effects of Research Evaluation Systems on Publication Practices, Disciplinary Norms, and Individual Working Routines in the Faculty of Arts at Uppsala University ’, Research Evaluation , 24 / 1 : 63 – 77 . Google Scholar Crossref Search ADS Hammarfelt B. ( 2016 ) ‘Beyond Coverage: Toward a Bibliometrics for the Humanities’. In: Ochsner M. et al. (eds) Research Assessment in the Humanities. Towards Criteria and Procedures , pp. 115 – 31 . Berlin: Springer . Hernández-Díaz J. M. ( 1997 ) La Universidad en España, del Antiguo Régimen a la LRU (1983). Hitos y cuestiones destacadas . Aula, Revista de Pedagogía de la Universidad de Salamanca 9 : 19 – 44 . Hicks D. , Wouters P. , Waltman L. et al. ( 2015 ) ‘ The Leiden Manifesto for Research Metrics ’, Nature , 520 / 7548 : 9 – 11 . Google Scholar Crossref Search ADS Jiménez-Contreras E. , de Moya-Anegón F. , Delgado López-Cózar E. ( 2003 ) ‘ The Evolution of Research Activity in Spain. The Impact of the National Commission for the Evaluation of Research Activity (CNEAI) ’, Research Policy , 32 : 123 – 42 . Google Scholar Crossref Search ADS Molas-Gallart J. ( 2012 ) ‘ Research Governance and the Role of Evaluation ’, A Comparative Study. American Journal of Evaluation , 33 / 4 : 583 – 98 . Google Scholar Crossref Search ADS Must U. ( 2012 ) ‘ Alone or Together: Examples from History Research ’, Scientometrics , 91 / 2 : 527 – 37 . Google Scholar Crossref Search ADS Nederhof A. J. ( 2006 ) ‘ Bibliometric Monitoring of Research Performance in the Social Sciences and the Humanities: A Review ’, Scientometrics , 66 / 1 : 81 – 100 . Google Scholar Crossref Search ADS Osuna C. , Cruz-Castro L. , Sanz-Menéndez L. ( 2011 ) ‘ Overturning Some Assumptions About the Effects of Evaluation Systems on Publication Performance ’, Scientometrics , 86 : 575 – 92 . Google Scholar Crossref Search ADS Paradeise C. , Reale E. , Goastellec G. ( 2009 ) ‘A Comparative Approach to Higher Education Reforms in Western European Countries’. In: Paradeise C. , et al. (eds) University Governance , Netherlands: Springer . Chapter 9 : 197 – 225 . Paradeise C. , Thoenig J. C. ( 2015 ) In Search of Academic Quality . New York: Palgrave Macmillan . Puyol R. ( 2002 ) El papel de la Universidad en la transición. Paper presented at the workshop “La integración europea y la transición política en España”, Madrid, 9 December 2017. <http://www.fundacionfaes.org/file_upload/publication/pdf/20130425184824el-papel-de-la-universidad-en-la-transicion.pdf> accessed 15 Feb 2018. Rafols I. , Molas-Gallart J. , Chávarro D. et al. ( 2016 ) ‘On the Dominance of Quantitative Evaluation in “peripheral” Countries: Auditing Research with Technologies of Distance’, Paper presented at the workshop ‘Excellence policies in science’, Leiden, June 2–3. <https://papers.ssrn.com/sol3/papers.cfm? abstract_id=2818335> accessed 15 Feb 2018. Ruiz-Pérez R. , Delgado E. , López E. ( 2010 ) ‘ Principios Y Criterios Utilizados En España Por La Comisión Nacional Evaluadora de La Actividad Investigadora (CNEAI) Para La Valoración de Las Publicaciones Científicas : 1989-2009 ’, Psicothema , 22 : 898 – 908 . <http://www.psicothema.com/pdf/3818.pdf> accessed 14 Jul 2015. Google Scholar PubMed Soriano J. ( 2008 ) El efecto ANECA. Conference paper available at Research Gate. <https://www.researchgate.net/publication/261780208_El_efecto_ANECA> accessed 15 Feb 2018. Thelwall M. , Sud P. ( 2014 ) ‘ No Citation Advantage for Monograph-Based Collaborations? ’, Journal of Infometrics , 8 / 1 : 276 – 83 . Google Scholar Crossref Search ADS Whitley R. ( 2007 ) ‘Changing Governance of the Public Sciences: The Consequences of Establishing Research Evaluation Systems for Knowledge Production in Different Countries and Scientific Fields’. In: Whitley R. , Gläser J. (eds) The Changing Governance of the Sciences. The Advent of Research Evaluation Systems. Sociology of the Sciences Yearbook , vol. 26 . Dordrecht: Springer , Chapter 1: 3 – 27 . Footnotes 1 Hammarfelt and De Rijcke (2015) also apply a mixed method methodology by combining bibliometric and survey data to address how the performance-based Swedish evaluation system affects the publication strategies and behaviors of researchers at the faculty of Arts of a Swedish university. 2 Academics with no recognized sexenio teach 32 credits per year compared to 24 credits for those witn one active sexenio, 16 credits for those with three or more sexenios. (Real Decreto-ley 14/2012, 20 April) 3 Even if originally sexenios could have been seen as an incentive mechanism conditioning the distribution of resources, according to Molas-Gallart (2012) and Rafols et al. (2016), they are rather a mechanism of control of the academic community. 4 Cruz-Castro and Sanz-Menéndez (2007) provide a thorough chronological review of the development of the Spanish RES. 5 The evaluation criteria may be found at: http://www.mecd.gob.es/servicios-al-ciudadano-mecd/catalogo/general/educacion/050920/ficha.html#dc 6 Exceptionally, the number of contributions may be smaller if they have an extraordinary quality or have had a very high scientific impact (BOE 2015). 7 Information regarding the original project, including its output, may be found at: http://prestence.ifris.org/ 8 To provide an idea of Dialnet’s coverage, we find through this database that one of the most prolific Spanish researchers in Humanities has published 39 books, 113 book chapters and 136 papers in research journals and magazines. The Web of Science covers just six of these papers while Scopus contains just one. 9 Organic Law 4/2007, approved on 12 April, which modifies the Universities Organic Law 6/2001, from 21 December. 10 The Royal Law Decret 20/2011 of 30 December, approved urgent budgetary and financial measures for the correction of the public deficit. It forbid the recruitment of new personnel in the public sector with only a few exceptions, and limited the substitution rate of retired personnel to 10 per cent. The measures from this Decret have still been enforced in subsequent budget laws over the last years. In 2015 the replacement rate for retired personnel rate was increased to 50 per cent. 11 For each interview quote, we indicate the letter of the corresponding department: A or B. All quotes included were translated from the original Spanish or Catalan into English by the authors. 12 In line with these statements, Hammarfelt and De Rijcke (2015: 69) report the following affirmation from an established Swedish historian: ‘… we are moving from publishing monographs to publishing articles, and this is not always beneficial for the humanities’. This is addressed in the context of the discussion of effects of the implementation of a performance-based evaluation system in Sweden. 13 Our classification derives from the Integrated Classification of Scientific Journals (http://clasificacioncirc.es/clasificacion-circ) that assists the evaluation of research outcomes in the social sciences and humanities. © The Author(s) 2018. Published by Oxford University Press. All rights reserved. For Permissions, please email: journals.permissions@oup.com This article is published and distributed under the terms of the Oxford University Press, Standard Journals Publication Model (https://academic.oup.com/journals/pages/open_access/funder_policies/chorus/standard_publication_model)
TI - The evaluation of research excellence and the dynamics of knowledge production in the humanities: The case of history in Spain
JF - Science and Public Policy
DO - 10.1093/scipol/scy025
DA - 2018-12-01
UR - https://www.deepdyve.com/lp/oxford-university-press/the-evaluation-of-research-excellence-and-the-dynamics-of-knowledge-WcmDSpYkAC
SP - 775
VL - 45
IS - 6
DP - DeepDyve
ER -