The peer-review process: The most valued dimensions according to the researcher’s scientific career

The peer-review process: The most valued dimensions according to the researcher’s scientific... Abstract Scientific activities are being assessed permanently. The best well-known and well-established evaluation process is peer review. Peer-review-based systems may have different goals; therefore several guidelines are normally set to be followed by individual experts. Normally, the components to be evaluated are known to the whole interested community, but peers make use of their own criteria to evaluate the performance on these components, introducing subjectivity in the whole process. This article reports on an attempt to better understand the decisions of peer-review panels and the role that bibliometric analysis might play in supporting the evaluation of scientific merit in peer-review processes. A particular evaluation process for the national selection of junior and senior researchers is considered. The results show that the dimensions more highly valued by the peers differ depending on the applicant’s phase in the scientific career. For applicants with shorter careers, international collaboration appears to be the dimension more highly valued. In the case of applicants at an intermediate phase of the scientific career, the impact dimension showed to be the most relevant. 1. Introduction The assessment of scientific results of individual researchers, groups, or institutions is part of the everyday life of a research community. It is used to assess if the goals originally set for a given body (e.g. funding agency, research institution, or individual researcher) are being achieved. It is part of the follow-up process to decide what is needed to improve, to adjust, or to strengthen in a research institution or in a national scientific system. The main process used when the goal is to access scientific performance is the peer-review-based systems. It is used in very different contexts: (1) to assess the quality of papers submitted to publication in a given journal or other channel of communication, (2) to take decisions on promotion in the academic context (Vieira, Cabral and Gomes 2014a, 2014b), and (3) in the competitive allocation of funds for research activities at several levels (Abramo and D'Angelo 2015; Bornmann, Wallon and Ledin 2008; Cabezas-Clavijo et al. 2013; Neufeld, Huber and Wegner 2013; Ramos and Sarrico 2016; Rinia et al. 1998; Taylor 2011; van Raan 2006; Vieira and Gomes 2015). When applied at the researcher level, the stage of development of the scientific career of the applicants is normally taken into account (ERC 2015; FCT 2012a, 2014), and therefore, it is understandable that peers might consider different criteria as more relevant. Normally, the parameters to be evaluated are defined by the organization responsible for the assessment exercise together with the guidelines that peers should follow; peers follow these guidelines but make the evaluation taking into account their own understanding of quality. This, inevitably, introduces some subjectivity in the assessment, as the peers may value differently the same individual component. Despite this important limitation of the peer-review-based systems, there is not consensus on a more objective alternative. In the past decades, research has been done on quantitative and objective indicators as instruments to support the peer-review process. Funding agencies and national policy bodies became interested in the use of indicators to avoid/limit the known problems of peer review and to make the process fast and reliable at a reasonable cost. This interest did not translate into a consensus on which indicators should be used. There is several research on the choice of simple indicators and in the construction of composite indicators that perform better in reproducing or predicting peer-review decisions (Bornmann and Leydesdorff 2013; Cabezas-Clavijo et al. 2013; Neufeld et al. 2013; Vieira et al. 2014a, 2014b). The use of these indicators in supporting peer-review panels stills an open question. From the large set of performance indicators that have been developed over time, we give special attention to those based on the number of publications, citations, and on the collaboration practices. These parameters have a set of advantages that make them attractive as an auxiliary tool in assessment exercises: (1) they allow the assessment of a very large number of documents; (2) they provide very simple and objective information about scientific performance; (3) they are relatively inexpensive to collect and simple to implement when compared with peer-review evaluation; (4) they allow the measure of the multidimensional nature of the research activities; and (5) they are well understood by the community, even if they are not above the critical view of many. In this study, using the results of the programme Investigador FCT (a highly competitive scheme designed by the Fundação para a Ciência e Tecnologia in Portugal to provide 5-year support for the most talented and creative researchers), we try to understand if bibliometric indicators are able to predict the final decisions of the peers and the dimensions that peers value the most when considering applicants at different phase of their scientific career. The final goal of the exercise is that of selecting the candidates that are more promising in terms of their expectable future contribution to science. To achieve this, the panels are required to assess the past performance of each candidate and his or her future work plan. We are looking for the best indicators to predict the results of this complex process of peer assessment and decision. This is a very important process, as the development, renewal, and sustainability of a national research system require the definition of a set of strategies that guarantee that highly qualified and internationally competitive researchers are being progressively integrated into the system, and carefully chosen indicators may anticipate their future performance. The use of performance indicators allows an objective assessment, but the choice of these indicators may be said to be subjective or discretionary. This article attempts to propose some justification for the choice of indicators to be used to make the whole process hopefully more objective. 2. Literature review Several studies tried to address the role of bibliometric indicators within a peer-review process. Rinia et al. 1998 studied the correlation between bibliometric indicators and the outcomes of peer judgments made by expert committees of physics in The Netherlands. Using Spearman's rank-correlation coefficients the authors found several significant correlations. The highest values (between 0.5 and 0.7) were found for indicators based on citations (with and without normalization). Norris and Oppenheim 2003, using the results of the peer-review-based process of the 2001 RAE, found a high correlation between the total number of citations and the final decisions of the peers panel in the area of Archaeology (Spearman’s coefficient around 0.8). Similar results were obtained in previous analysis using the results of the 1992 RAE (Oppenheim 1997). Aksnes and Taxt 2004 using the peer ratings of the assessment exercise of 34 research groups at the University of Bergen and a set of five bibliometric indicators (number of papers per scientific personnel; number of citations per person; relative citation rate; relative subfield citedness; relative publication strategy) looked at the correlation between the results. They found weak but significant correlations. The highest Pearson’s correlation was observed between peer ratings and relative publication strategy (an indicator that compares the average citation rate of the journals in which the group’s articles were published with the average citation rates of the subfields covered by each journal), 0.48.van Raan 2006, for 147 university chemistry research groups in The Netherlands, studied the relation between the h index, the CPP/FCSm, and peers’ judgement. The author found that both indicators are able to discriminate very well between the sets of documents that received rating 3 and the sets of documents that received ratings 4 and 5. Jensen, Rouquier, and Croissant 2009 for a set of 400 researchers at CNR explored the correlation between bibliometric indicators and the results of a peer-review process concerning the promotion of researchers. The authors found that the h index, the h index divided by the ‘scientific age’, the number of citations, the number of publications, and the average number of citations per publication allow for a distinction between promoted and not promoted researchers. Using peer decisions at the Valutazione Triennale della Ricerca (VTR), in the hard sciences, and the impact factor of the journals where the documents were published, Abramo, D'Angelo, and Caprasecca 2009 found correlations between the two results that are in the interval 0.336 and 0.876. Franceschet and Costantini 2011 found positive correlations between peers’ decisions at VTR and the mean number of citations per document and the impact factor of the journal. For the number of citations per document the authors found correlation coefficients above 0.7 for Physics and Earth Sciences and between 0.3 and 0.5 for Mathematics and Computer Sciences, Civil Engineering and Architecture, and Economics and Statistics. Using the impact factor the authors found high correlations with peers’ decisions for Chemistry and Biology, while for fields such as Economics and Statistics, Civil Engineering and Architecture, and Industrial and Information Engineering, the correlation coefficient varies between 0.3 and 0.5. The strength of the association seems to be dependent on the scientific field. Butler and McAllister 2011 tried to replicate the peer-review-based outcome of the 2001 RAE in Chemistry and Political Science. They used the mean citation rate, the department size, the research culture, and the presence on the RAE panel of staff from the department under evaluation as predictors of the peeŕs decisions. The impact measured by the mean citation rate is a good predictor of the final decisions of the peer review. Bornmann and Leydesdorff 2013 using a set of 125 papers from F1000 and seven bibliometric indicators from InCites studied the correlation between the scores attributed to each paper by the reviewers (FFa) and the results given by each bibliometric indicator. The authors concluded that the Percentile in Subject Area and Category Actual/Expected Citations, both, should be preferred over other metrics, in research evaluation studies, as they were identified as those most correlated with the ratings. Cabezas-Clavijo et al. 2013 analysed the relationship between peers’ ratings and bibliometric indicators for 2,333 Spanish researchers in the 2007 National R&D Plan. The authors found that the number of published articles and the papers published in journals that belong to the first quartile ranking of the Journal Citations Report are the main indicators in predicting the peers’ rating. Neufeld et al. 2013 analysed the dependence of funding decisions on past publication performance among applicants of the Starting Grants Programme, offered by the European Research Council. Applicants from the Life Sciences and Physical Sciences and Engineering were considered in the study. The authors found that the number of publications is not able to distinguish between funded and not funded applicants, the mean Journal Impact Factor is able to distinguish between the two groups for some of the sub-areas of Life Sciences and Physical Sciences and Engineering, and the field normalized citation rate and the number of highly cited papers (top 10%) are able to distinguish between funded and not funded applicants in the Physical Sciences and Engineering. Vieira et al. 2014a using academic contests, which took place in several Portuguese universities, found that two composite indicators (combination of two/three bibliometric indicators) are able to predict the peers’ final decision. For 75% of the orderings of the contests, the composite indicators were able to predict the final decision of the peers. Aimed at determining the role of quantitative indicators (indicators based on citation and altmetrics) in research assessment and management, Wilsdon et al. 2015 carried out a large study using the data submitted to the REF2014. Fifteen indicators were determined and the results compared with the score obtained by each author-publication in the evaluation made by the peers. Consistently, high correlations were found for several metrics in Clinical Medicine, Economics, and Econometrics. The authors also found that the quantitative indicators with significant impact in predicting peers’ evaluation are different among scientific field. Bedsides quantitative analysis the authors tried to understand the opinion of several experts who participated in REF in relation to the use of quantitative indicators; the experts suggested that peer review should remain central to the process, but that there is some support for the additional use of quantitative data. The studies described above lead us to summarize the main findings in the following two points: Correlations between the results given by quantitative indicators and the results from peer evaluation were observed at several levels (evaluation of research programmes, university’s departments, individual researchers, and individual papers). However, the strength of the correlations varies depending on the scientific field. Several quantitative indicators are able to reproduce in part the peers’ decisions, h index, and normalized indices (CPP/FCSm, and/or highly cited papers). Bedsides the large numbers of studies that have been done none of the studies, as far as we know, address the fact that the bibliometric indicators able to describe the final decisions of the peers might be different depending on the phase of the scientific career of the researchers being evaluated. Taking into account the findings of the studies described above, our study tries to answer to the following research question: Q1: Do the dimensions of scientific performance, represented by bibliometric indicators, more valued by the peers differ between starting and development grants? There are several concerns about the use of bibliometric indicators to describe the scientific performance of young researchers; thus it is important to explore the subject. On the other hand, if bibliometric indicators are able to describe peers’ decisions, it is important to study if the set of bibliometric indicators differs according to the type of grant proposal. 3. Methods and data 3.1 Programme investigador FCT Investigador FCT is a programme carried out since 2012 by the FCT, in Portugal, with the aim of giving financial support for research activities at the individual level. Through this programme, FCT intends to select a set of highly motivated researchers that are internationally competitive and hope to establish themselves as independent researchers, and already independent researchers who wish to consolidate research skills and establish leadership in their research field, in Portugal. The call comprises three levels of grants, and taking into account the number of years after the award of the PhD degree and the number of years working as an independent researcher, a given researcher can apply to one of the following grants FCT 2012b: Starting grants: For researchers with a maximum of 5 years after the award of the PhD and being an independent researcher is not required. Development grants: For researchers with the PhD degree awarded for more than 6 years and less than 12 years and being independent researchers for less than 6 years. Advanced grants: For researchers with more than 6 years as independent. In 2012, the opening took place in two phases. At the first phase the applicants were asked to submit CV synopsis and the research project and career development synopsis for evaluation by the peers. In addition the applicants were also asked to identify their scientific area from the Organisation for Economic Co-operation and Development (OECD)´s adopted Field of Science and Technology (FOS) classification, the main and secondary scientific area (FCT 2012a). At this phase only 25% of the total applications were selected to forward to the second phase. Each proposal was evaluated by two reviewers belonging to the four panels responsible for the preliminary reviewing of all applications (first phase); CV (60%) research project (20%), and career development plan (20%) (FCT 2012b). In terms of CV, peers were asked to use the following evaluation criteria (for all types of grants) for the scientific merit (FCT 2012b): scientific productivity; abilities and skills to execute the proposed project; degree of internationalization; and degree of success in previous calls for grant application and doctoral and post-doctoral training. For more information related to the evaluation criteria for the remaining components see (FCT 2012b). The applicants selected at the first level were asked to submit a full application (i.e. the extended versions of the CV, research project, and career development plan) for evaluation by two external mail referees. The external mail referees produced evaluation reports submitted to the evaluation panel members responsible for the final decision. The evaluation panel comprised 10 members (one chair and three members for each scientific domain; Life Sciences, Physical Sciences and Engineering, and Social Sciences and the Humanities) who had access to all applications (FCT 2012b). Total 54% of the total applicants who passed to the second phase were selected for funding, and they represent about 13% of the total applicants (first and second phases). Table 1 showed the distribution of the applicants of the programme launched in 2012, by type of grant and field for each phase of the peer-review process. Table 1. Distribution of applicants by type of grant and scientific domain Type of grant  Area  Number of applicants Phase 1  Number of applicants Phase 2  Applicants selected for funding  Starting  Agricultural Sciences  37  3  1  Engineering and Technology  177  20  10  Exact Sciences  122  12  6  Humanities  73  15  8  Medical and Health Sciences  83  20  12  Natural Sciences  147  30  21  Social Sciences  87  16  9  Development  Agricultural Sciences  26  7  2  Engineering and Technology  76  28  12  Exact Sciences  92  48  28  Humanities  20  8  5  Medical and Health Sciences  33  21  14  Natural Sciences  76  23  15  Social Sciences  29  11  2  Advanced  Agricultural Sciences  1  –  –  Engineering and Technology  17  4  2  Exact Sciences  31  6  5  Humanities  11  4  –  Medical and Health Sciences  7  4  2  Natural Sciences  15  7  2  Social Sciences  15  5  2  Total    1,175  292  158  Type of grant  Area  Number of applicants Phase 1  Number of applicants Phase 2  Applicants selected for funding  Starting  Agricultural Sciences  37  3  1  Engineering and Technology  177  20  10  Exact Sciences  122  12  6  Humanities  73  15  8  Medical and Health Sciences  83  20  12  Natural Sciences  147  30  21  Social Sciences  87  16  9  Development  Agricultural Sciences  26  7  2  Engineering and Technology  76  28  12  Exact Sciences  92  48  28  Humanities  20  8  5  Medical and Health Sciences  33  21  14  Natural Sciences  76  23  15  Social Sciences  29  11  2  Advanced  Agricultural Sciences  1  –  –  Engineering and Technology  17  4  2  Exact Sciences  31  6  5  Humanities  11  4  –  Medical and Health Sciences  7  4  2  Natural Sciences  15  7  2  Social Sciences  15  5  2  Total    1,175  292  158  Table 1. Distribution of applicants by type of grant and scientific domain Type of grant  Area  Number of applicants Phase 1  Number of applicants Phase 2  Applicants selected for funding  Starting  Agricultural Sciences  37  3  1  Engineering and Technology  177  20  10  Exact Sciences  122  12  6  Humanities  73  15  8  Medical and Health Sciences  83  20  12  Natural Sciences  147  30  21  Social Sciences  87  16  9  Development  Agricultural Sciences  26  7  2  Engineering and Technology  76  28  12  Exact Sciences  92  48  28  Humanities  20  8  5  Medical and Health Sciences  33  21  14  Natural Sciences  76  23  15  Social Sciences  29  11  2  Advanced  Agricultural Sciences  1  –  –  Engineering and Technology  17  4  2  Exact Sciences  31  6  5  Humanities  11  4  –  Medical and Health Sciences  7  4  2  Natural Sciences  15  7  2  Social Sciences  15  5  2  Total    1,175  292  158  Type of grant  Area  Number of applicants Phase 1  Number of applicants Phase 2  Applicants selected for funding  Starting  Agricultural Sciences  37  3  1  Engineering and Technology  177  20  10  Exact Sciences  122  12  6  Humanities  73  15  8  Medical and Health Sciences  83  20  12  Natural Sciences  147  30  21  Social Sciences  87  16  9  Development  Agricultural Sciences  26  7  2  Engineering and Technology  76  28  12  Exact Sciences  92  48  28  Humanities  20  8  5  Medical and Health Sciences  33  21  14  Natural Sciences  76  23  15  Social Sciences  29  11  2  Advanced  Agricultural Sciences  1  –  –  Engineering and Technology  17  4  2  Exact Sciences  31  6  5  Humanities  11  4  –  Medical and Health Sciences  7  4  2  Natural Sciences  15  7  2  Social Sciences  15  5  2  Total    1,175  292  158  3.2 Data set A total of 7,559 documents were retrieved from the Web of Science Core Collection (Science Citation Index Expanded; Social Sciences Citation Index; Arts and Humanities Citation Index; Conference Proceedings Citation Index—Science; Conference Proceedings Citation Index—Social Science and Humanities). These documents are those mentioned in the grant proposal by each applicant and were made available by the FCT. We retrieved the information about the publications for each applicant from the Web of Science Core Collection using expressions that combine the different homonymous used by each applicant and the research institutions where they have been working at. The documents have been published by the 167 applicants that were selected to proceed to the second phase of the peer-review process: 101 of 167 are applicants to the development grants (Engineering and Technology (26 applicants), Natural Sciences (21), Exact Sciences (40), and Medial and Health Sciences (14)). In the case of starting grants the sample comprises 17 applicants from Engineering and Technology, 25 applicants from Natural Sciences, 9 applicants from Exact Sciences, and 15 applicants from Medial and Health Sciences. Not all the applicants selected to go to the second phase of the evaluation process were considered in our study due to the aspects presented below. The call was also available to those researchers from the Social Sciences and Humanities, but we did not consider these applicants as we are using a database that has several limitations in the coverage of publications in these fields. The proposals within Agricultural Sciences (all grants) and the proposals to advanced grants were also excluded, as they were present in a small number (see Table 1). We should use a considerable number of publications in calculating the bibliometric indicators. We use a sample composed of applicants with at least 10 documents published and indexed in the Web of Science at the moment of the call. The threshold used is lower than that suggested in the literature (Lehmann, Jackson and Lautrup 2008), but we want to avoid the exclusion of a large number of applicants, mainly in the case of the starting grants where, due to the very initial phase of the scientific career, applicants are expected to have a low number of publications. 3.3 Bibliometric indicators The evaluation guidelines to be used by the peers in the process were made available by the FCT, but these criteria are very general and most of them do not allow associating a bibliometric indicator easily. As an example we have (FCT 2012): ‘Scientific productivity of the applicant evaluated according to criteria accepted internationally by the different scientific communities’. ‘Indicators for scientific merit of the applicant include the main academic and professional degrees, publications in top specialty peer-reviewed journals and/or in major multidisciplinary international peer-reviewed journals’. Taking this into account, we selected a set of bibliometric indicators that considers the multidimensional nature of scientific achievement. Many indicators have been developed over the years, and we were forced to limit our choice for this study. We selected 15 indicators, which explore different dimensions: production, impact, collaboration, and scientific independence. Most of the indicators or their variants were already used in a previous study and showed to be important in representing the final decision of the peer-review process. Other indicators may be understood as a proxy for more objective guidelines of the peer-review process. The definition of each of the indicators is presented. The h index, hnf, HCD, NI, SNIPm, SJRm, PQ1, and DIC were determined using the documents classified by the database as articles, reviews, and proceedings paper. TD: The total number of documents indexed in the Web of Science Core Collection for each applicant. NDF: The total number of documents published by the applicants after fractionation. Each document was divided by the total number of authors (N) of the list and associated with (1/N). We considered all the types of documents indexed (Vieira et al. 2014a). PA: The percentage of the total number of documents of the applicant that are classified as articles in the Web of Science Core Collection. PR: The percentage of the total number of documents of the applicant that are classified as reviews in the Web of Science Core Collection. PP: The percentage of the total number of documents of the applicant that are classified as proceedings paper in the Web of Science Core Collection. PAP: The percentage of the total number of documents of the applicant that are classified simultaneously as proceedings paper and articles in the Web of Science Core Collection. PDAC: The percentage of documents where the applicant is the corresponding author. h index: As defined by J. E. Hirsch ‘A scientist has index h if h of his or her Np papers have at least h citations each and the other (Np - h) papers have ≤h citations each’ (Hirsch 2005). hnf index: The indicator is calculated in a way similar to the h index, but considers the different citation cultures among fields and the number of authors per publication (Vieira and Gomes 2011). HCD: Highly cited documents. This indicator gives the percentage of documents of a given researcher that are in the top 10% most cited EU_27 documents. The category, type of document, and publication year are taken into account and a 5-year citation window used in collecting the number of citations. NI: Normalized impact. The indicator compares the average number of citations per publication of the applicant with the EU_27 average. A value of 1 indicates that the documents of the applicant behave as the average and a value of 1.20 means that the average value of citations per document of the applicant is 20% above of the average of the EU_27. The category, type of document, and publication year are taken into account in the normalization process. We used a 5-year citation window in collecting the number of citations. SNIPm: For each document, indexed in the Web of Science Core Collection, we collected the SNIP (Moed 2010) of the journal. SNIPmrepresents the median value of the distribution of the journals where the documents have been published in. The SNIP takes into account the scientific domain where the journals operate, smoothing differences between field-specific properties such as the number of citations per paper and the speed of the publication process. SJRm: For each document, indexed in the Web of Science Core Collection, we collected the SJR (Gonzalez-Pereira, Guerrero-Bote and Moya-Anegon 2010) of the journal. SJRmrepresents the median value of the distribution of the journals where the documents have been published in. The SJR takes into account the prestige of the citing journal; the citations received by the journals are weighted according to the SJR of the citing journal. The SNIP and SJR were retrieved from Scopus for the year of publication of each document. PQ1: Percentage of documents that were published in journals that belong to the first quartile in the corresponding Web of Science category and year of publication, according to the impact factor for 2 years, in the Journal Citation Reports. When the journal belongs to more than one category, and it is only in the first quartile in one category, it is considered in the set of documents in the first quartile. DIC: The percentage of the total documents with at least one international collaboration. The indicators PA, PP, PR, and PAP were selected due to the fact that we are considering applicants from different scientific domains and therefore with very different culture of publications. As example we have the case of Engineering and Technology and Computer Science, where the acceptance levels in conferences are rather low (Meyer et al, 2009) showing the importance of the proceedings paper. The PDAC is a proxy for ‘scientific independence’. We selected the PDAC, as in the evaluation guide one of the parameters that peers are asked to evaluate is scientific independence. FCT suggests as criteria for assessing scientific independence the number of publications where the applicant is the corresponding author (FCT 2012a). Other suggested criteria are as follows: ‘– being a PI or group leader of a research team, – having obtained funding as principal investigator in competitive calls launched by national and/or international funding agencies’. We were not able to use these criteria in our study, as the information submitted by the applicants was not made available. In the calculation of the hnf, HCD, and NI, in the normalization of citations we used the Web of Science categories defined at the journals level. We adopted a cited-side normalization procedure and a full counting method. Indicators that use a different normalization process, for example the citing-side, could be used. There is no indicator which is entirely without drawbacks, as every standardized indicator has its advantages and limitations. In the literature there are studies showing that the method of normalization has only a slight influence on the validity of the indicators (Bornmann and Marx 2015). The unavailability of the data necessary to calculate the indicators with citing-side normalization did not allow using such type of indicators in our study. However, they should be explored in future studies. The SNIP and SJR are based on the information from Scopus database, and both normalized indicators are used frequently for comparing journal impact. While being aware of the fact that we are using two indicators based on the information from another database, we consider that this does not invalidate our results. The Web of Science journal impact factor (JIF) has been shown to correlate well with SNIP and SJR indicators for several scientific domains (Gonzalez-Pereira et al. 2010; Torres-Salinas and Jimenez-Contreras 2010); the lowest correlations were observed for the Social Sciences and Humanities and these domains are not considered in the study. The DIC is a proxy for the ‘Degree of internationalization’ (FCT 2012b). 3.4 Statistical analysis First, we used bootstrapping techniques (with replacement) and the t-test to identify those dimensions for which the scientific performance of the set of selected and not selected applicants is statistical significant different. The results provide preliminary evidence on the indicators that might be important in predicting peers’ decisions. To understand the substantive significance of the findings, we made use of the Cohen’s d statistic. The use of bootstrapping techniques allows us to deal with some features, less suitable for statistical analysis, of the original data, as small sample size and absence of normality of the data. To investigate the distribution of the indicators and find preliminary evidence on their correlations, we also did a descriptive analysis (see Appendix). Secondly, using logistic regression, we identify those bibliometric indicators that contain information about the dimensions more valued in the evaluation exercise for different phases of the researcher’s scientific career. We determine the bibliometric indicators with significant impact implicit in the peer judgments and the size of their effect (Hosmer, Lemeshow and Sturdivant 2013). Logistic regression is used to estimate the probability of observing a given outcome (normally a binary, variable) based on one or more independent variables. Such probability is given by the following expression:   Pi=exp⁡(β0+β×Xi)1+exp⁡(β0+β×Xi), (1) where β0+β×Xi is the utility function and represents a linear function of the explanatory indicators (Xi) of applicant i; the β is the vector of coefficients and, β0 the constant. As we are dealing with small sample size, we used cross-validation for estimating the performance of the predictive model. A set of fit measures (sensitivity, specificity, false positive, false negative, prediction, and area under the ROC (AUC)) is also determined to study the robustness of the main findings. Thirdly, we discuss in a detailed way the results obtained and compare them with the findings of previous studies. All the statistical analyses were carried out using MATLAB and Stata software. 4. Results 4.1 Differences between selected and not selected applicants To have some preliminary understanding of the selection process, we identified the bibliometric indicators able to distinguish, in a significant way, between the two sets of applicants, those selected and not selected. In Tables 2 and 3, we present the average value for each indicator and highlight those cases where the results from the t-test were statistically significant (P-value < 0.05). Table 2. Average values for the two groups of applicants (not selected and selected) for the starting grants Indicator  Engineering and Technology   Natural Sciences   Exact Sciences   Medical and Health Sciences   Selected  Not selected  Selected  Not selected  Selected  Not selected  Selected  Not selected  TD  53.22  25.38  20.24  29.88  22.75  29.80  27.33  19.17  NDF  9.95  5.84  4.68  6.66  3.26  6.75  3.69  2.69  PA  71.60  64.14  86.36  85.90  59.91  79.16  65.56  74.48  PR  3.46  2.95  3.79  5.48  1.71  6.64  12.51  8.51  PP  5.23  17.62  2.08  0.00  17.66  3.19  0.46  1.57  PAP  16.03  13.12  3.55  5.66  8.21  11.00  3.10  2.04  PDAC  18.52  21.46  31.53  26.88  20.45  19.00  6.45  16.30  h  14.56  7.75  8.88  11.00  7.00  11.40  10.00  8.17  hnf  4.94  2.76  2.68  4.10  1.79  3.99  2.36  2.03  HCD  14.24  6.84  18.78  23.07  3.64  13.54  29.80  10.90  NI  1.16  1.04  1.56  1.15  0.85  1.34  1.02  0.81  SJRm  1.02  0.85  1.34  1.20  0.87  1.18  1.91  1.17  SNIPm  1.30  1.11  1.19  1.24  1.04  1.10  1.30  1.09  PQ1  45.30  47.06  50.68  46.38  47.72  53.34  65.24  64.01  DIC  37.86  23.27  45.17  25.39  53.29  38.25  36.69  32.73  Indicator  Engineering and Technology   Natural Sciences   Exact Sciences   Medical and Health Sciences   Selected  Not selected  Selected  Not selected  Selected  Not selected  Selected  Not selected  TD  53.22  25.38  20.24  29.88  22.75  29.80  27.33  19.17  NDF  9.95  5.84  4.68  6.66  3.26  6.75  3.69  2.69  PA  71.60  64.14  86.36  85.90  59.91  79.16  65.56  74.48  PR  3.46  2.95  3.79  5.48  1.71  6.64  12.51  8.51  PP  5.23  17.62  2.08  0.00  17.66  3.19  0.46  1.57  PAP  16.03  13.12  3.55  5.66  8.21  11.00  3.10  2.04  PDAC  18.52  21.46  31.53  26.88  20.45  19.00  6.45  16.30  h  14.56  7.75  8.88  11.00  7.00  11.40  10.00  8.17  hnf  4.94  2.76  2.68  4.10  1.79  3.99  2.36  2.03  HCD  14.24  6.84  18.78  23.07  3.64  13.54  29.80  10.90  NI  1.16  1.04  1.56  1.15  0.85  1.34  1.02  0.81  SJRm  1.02  0.85  1.34  1.20  0.87  1.18  1.91  1.17  SNIPm  1.30  1.11  1.19  1.24  1.04  1.10  1.30  1.09  PQ1  45.30  47.06  50.68  46.38  47.72  53.34  65.24  64.01  DIC  37.86  23.27  45.17  25.39  53.29  38.25  36.69  32.73  Note: The values in bold represent the indicators where we observed statistical significant differences; P-value < 0.05. Table 2. Average values for the two groups of applicants (not selected and selected) for the starting grants Indicator  Engineering and Technology   Natural Sciences   Exact Sciences   Medical and Health Sciences   Selected  Not selected  Selected  Not selected  Selected  Not selected  Selected  Not selected  TD  53.22  25.38  20.24  29.88  22.75  29.80  27.33  19.17  NDF  9.95  5.84  4.68  6.66  3.26  6.75  3.69  2.69  PA  71.60  64.14  86.36  85.90  59.91  79.16  65.56  74.48  PR  3.46  2.95  3.79  5.48  1.71  6.64  12.51  8.51  PP  5.23  17.62  2.08  0.00  17.66  3.19  0.46  1.57  PAP  16.03  13.12  3.55  5.66  8.21  11.00  3.10  2.04  PDAC  18.52  21.46  31.53  26.88  20.45  19.00  6.45  16.30  h  14.56  7.75  8.88  11.00  7.00  11.40  10.00  8.17  hnf  4.94  2.76  2.68  4.10  1.79  3.99  2.36  2.03  HCD  14.24  6.84  18.78  23.07  3.64  13.54  29.80  10.90  NI  1.16  1.04  1.56  1.15  0.85  1.34  1.02  0.81  SJRm  1.02  0.85  1.34  1.20  0.87  1.18  1.91  1.17  SNIPm  1.30  1.11  1.19  1.24  1.04  1.10  1.30  1.09  PQ1  45.30  47.06  50.68  46.38  47.72  53.34  65.24  64.01  DIC  37.86  23.27  45.17  25.39  53.29  38.25  36.69  32.73  Indicator  Engineering and Technology   Natural Sciences   Exact Sciences   Medical and Health Sciences   Selected  Not selected  Selected  Not selected  Selected  Not selected  Selected  Not selected  TD  53.22  25.38  20.24  29.88  22.75  29.80  27.33  19.17  NDF  9.95  5.84  4.68  6.66  3.26  6.75  3.69  2.69  PA  71.60  64.14  86.36  85.90  59.91  79.16  65.56  74.48  PR  3.46  2.95  3.79  5.48  1.71  6.64  12.51  8.51  PP  5.23  17.62  2.08  0.00  17.66  3.19  0.46  1.57  PAP  16.03  13.12  3.55  5.66  8.21  11.00  3.10  2.04  PDAC  18.52  21.46  31.53  26.88  20.45  19.00  6.45  16.30  h  14.56  7.75  8.88  11.00  7.00  11.40  10.00  8.17  hnf  4.94  2.76  2.68  4.10  1.79  3.99  2.36  2.03  HCD  14.24  6.84  18.78  23.07  3.64  13.54  29.80  10.90  NI  1.16  1.04  1.56  1.15  0.85  1.34  1.02  0.81  SJRm  1.02  0.85  1.34  1.20  0.87  1.18  1.91  1.17  SNIPm  1.30  1.11  1.19  1.24  1.04  1.10  1.30  1.09  PQ1  45.30  47.06  50.68  46.38  47.72  53.34  65.24  64.01  DIC  37.86  23.27  45.17  25.39  53.29  38.25  36.69  32.73  Note: The values in bold represent the indicators where we observed statistical significant differences; P-value < 0.05. Table 3. Average values for the two groups of applicants (not selected and selected) for the development grants Indicator  Engineering and Technology   Natural Sciences   Exact Sciences   Medical and Health Sciences   Selected  Not selected  Selected  Not selected  Selected  Not selected  Selected  Not selected  TD  48.18  34.40  35.93  42.57  49.17  47.06  24.67  37.60  NDF  11.02  12.02  8.25  10.51  12.79  9.31  4.61  8.15  PA  71.94  67.22  81.13  81.65  81.88  76.40  77.19  55.66  PR  2.41  2.77  5.41  2.36  2.71  2.70  4.15  7.50  PP  8.57  17.37  4.92  2.38  4.40  6.34  0.91  12.02  PAP  13.12  10.69  2.87  7.21  9.42  13.56  9.41  4.67  PDAC  20.11  33.88  22.47  41.86  29.11  34.70  17.91  19.75  H  15.73  10.13  13.43  13.57  15.13  12.76  11.89  12.60  hnf  5.31  4.19  4.14  4.85  5.59  4.08  2.85  3.61  HCD  25.39  10.04  17.93  9.69  15.08  8.61  19.63  20.74  NI  1.42  0.86  1.59  0.95  1.25  1.49  1.17  1.46  SJRm  1.24  0.93  2.08  1.33  1.32  1.32  2.09  0.87  SNIPm  1.25  1.09  1.26  1.20  1.13  1.16  1.21  1.01  PQ1  49.05  42.08  66.61  61.10  56.31  57.70  68.70  49.21  DIC  48.77  24.72  46.78  52.02  50.94  55.19  44.95  32.55  Indicator  Engineering and Technology   Natural Sciences   Exact Sciences   Medical and Health Sciences   Selected  Not selected  Selected  Not selected  Selected  Not selected  Selected  Not selected  TD  48.18  34.40  35.93  42.57  49.17  47.06  24.67  37.60  NDF  11.02  12.02  8.25  10.51  12.79  9.31  4.61  8.15  PA  71.94  67.22  81.13  81.65  81.88  76.40  77.19  55.66  PR  2.41  2.77  5.41  2.36  2.71  2.70  4.15  7.50  PP  8.57  17.37  4.92  2.38  4.40  6.34  0.91  12.02  PAP  13.12  10.69  2.87  7.21  9.42  13.56  9.41  4.67  PDAC  20.11  33.88  22.47  41.86  29.11  34.70  17.91  19.75  H  15.73  10.13  13.43  13.57  15.13  12.76  11.89  12.60  hnf  5.31  4.19  4.14  4.85  5.59  4.08  2.85  3.61  HCD  25.39  10.04  17.93  9.69  15.08  8.61  19.63  20.74  NI  1.42  0.86  1.59  0.95  1.25  1.49  1.17  1.46  SJRm  1.24  0.93  2.08  1.33  1.32  1.32  2.09  0.87  SNIPm  1.25  1.09  1.26  1.20  1.13  1.16  1.21  1.01  PQ1  49.05  42.08  66.61  61.10  56.31  57.70  68.70  49.21  DIC  48.77  24.72  46.78  52.02  50.94  55.19  44.95  32.55  Note: The values in bold represent the indicators where we observed statistical significant differences; P-value < 0.05. Table 3. Average values for the two groups of applicants (not selected and selected) for the development grants Indicator  Engineering and Technology   Natural Sciences   Exact Sciences   Medical and Health Sciences   Selected  Not selected  Selected  Not selected  Selected  Not selected  Selected  Not selected  TD  48.18  34.40  35.93  42.57  49.17  47.06  24.67  37.60  NDF  11.02  12.02  8.25  10.51  12.79  9.31  4.61  8.15  PA  71.94  67.22  81.13  81.65  81.88  76.40  77.19  55.66  PR  2.41  2.77  5.41  2.36  2.71  2.70  4.15  7.50  PP  8.57  17.37  4.92  2.38  4.40  6.34  0.91  12.02  PAP  13.12  10.69  2.87  7.21  9.42  13.56  9.41  4.67  PDAC  20.11  33.88  22.47  41.86  29.11  34.70  17.91  19.75  H  15.73  10.13  13.43  13.57  15.13  12.76  11.89  12.60  hnf  5.31  4.19  4.14  4.85  5.59  4.08  2.85  3.61  HCD  25.39  10.04  17.93  9.69  15.08  8.61  19.63  20.74  NI  1.42  0.86  1.59  0.95  1.25  1.49  1.17  1.46  SJRm  1.24  0.93  2.08  1.33  1.32  1.32  2.09  0.87  SNIPm  1.25  1.09  1.26  1.20  1.13  1.16  1.21  1.01  PQ1  49.05  42.08  66.61  61.10  56.31  57.70  68.70  49.21  DIC  48.77  24.72  46.78  52.02  50.94  55.19  44.95  32.55  Indicator  Engineering and Technology   Natural Sciences   Exact Sciences   Medical and Health Sciences   Selected  Not selected  Selected  Not selected  Selected  Not selected  Selected  Not selected  TD  48.18  34.40  35.93  42.57  49.17  47.06  24.67  37.60  NDF  11.02  12.02  8.25  10.51  12.79  9.31  4.61  8.15  PA  71.94  67.22  81.13  81.65  81.88  76.40  77.19  55.66  PR  2.41  2.77  5.41  2.36  2.71  2.70  4.15  7.50  PP  8.57  17.37  4.92  2.38  4.40  6.34  0.91  12.02  PAP  13.12  10.69  2.87  7.21  9.42  13.56  9.41  4.67  PDAC  20.11  33.88  22.47  41.86  29.11  34.70  17.91  19.75  H  15.73  10.13  13.43  13.57  15.13  12.76  11.89  12.60  hnf  5.31  4.19  4.14  4.85  5.59  4.08  2.85  3.61  HCD  25.39  10.04  17.93  9.69  15.08  8.61  19.63  20.74  NI  1.42  0.86  1.59  0.95  1.25  1.49  1.17  1.46  SJRm  1.24  0.93  2.08  1.33  1.32  1.32  2.09  0.87  SNIPm  1.25  1.09  1.26  1.20  1.13  1.16  1.21  1.01  PQ1  49.05  42.08  66.61  61.10  56.31  57.70  68.70  49.21  DIC  48.77  24.72  46.78  52.02  50.94  55.19  44.95  32.55  Note: The values in bold represent the indicators where we observed statistical significant differences; P-value < 0.05. The indicators for which we found differences are not the same among scientific fields. In Engineering and Technology indicators related to quantity (TD and NDF) and collaboration (NDF) are those able to distinguish between the applicants not selected and selected by the peers (statistical significance is observed). In addition to these indicators we also have indicators related to impact (the h and hnf index). However, between the two indicators we would say that the hnf is the most appropriate for comparing the applicants, as it is a normalized indicator. In Natural Sciences there is statistical significant evidence to reject the null hypotheses in the case of the variables DIC and hnf; the two groups of applicants are different in what concerns to international collaboration; in average, the not selected applicants have lower practices of international collaboration than selected applicants. The hnf is able to distinguish between selected and not selected applicants (there is statistical evidence to say that the means between the two groups are not equal). However, the mean value is greater in the case of not selected applicants, which is against our expectations. The result is unclear for us (we do not identified outliers within the set of not selected applicants). In the case of Exact Sciences, statistical significant differences are not observed between the two groups of applicants. Nevertheless, the Cohen’s d indicates that there are indicators for which the differences between the means are large; the PA, h, hnf, and HCD (see Table A5). Table 5. Results for the logistic regression Grant type  Coefficients, standardized   SJRm (%)  HCD (%)  DIC (%)  Intercept (%)  Development  1.130* (18.1)  0.952* (9.2)  −  −1.891* (12.6)  Starting  –  –  0.692* (18.2)  −1.328 (12.1)  Grant type  Coefficients, standardized   SJRm (%)  HCD (%)  DIC (%)  Intercept (%)  Development  1.130* (18.1)  0.952* (9.2)  −  −1.891* (12.6)  Starting  –  –  0.692* (18.2)  −1.328 (12.1)  Note: * P-value < 0.05. The coefficient value is the mean value; cross-validation was used in adjusting the model. Between brackets is the coefficient of variation for each coefficient. Table 5. Results for the logistic regression Grant type  Coefficients, standardized   SJRm (%)  HCD (%)  DIC (%)  Intercept (%)  Development  1.130* (18.1)  0.952* (9.2)  −  −1.891* (12.6)  Starting  –  –  0.692* (18.2)  −1.328 (12.1)  Grant type  Coefficients, standardized   SJRm (%)  HCD (%)  DIC (%)  Intercept (%)  Development  1.130* (18.1)  0.952* (9.2)  −  −1.891* (12.6)  Starting  –  –  0.692* (18.2)  −1.328 (12.1)  Note: * P-value < 0.05. The coefficient value is the mean value; cross-validation was used in adjusting the model. Between brackets is the coefficient of variation for each coefficient. In Medical and Health Sciences we found statistical significant differences, between selected and not selected applicants, for indicators related to the impact and prestige of the journals (SNIPm and SJRm). It might appear that peers are using these metrics as a proxy of the quality of the research in an article. However, this result might be explained if we consider that in the evaluation guidelines it is mentioned that the scientific merit of the applicants should be evaluated considering the publications in top specialty peer-reviewed journals. In this situation we would say that the impact and the prestige of the journal where the applicants have been published in are important parameters. On the other hand, it might happen that the papers published in these journals did not have enough time to collect a high number of citations (we are considering young researchers), but peers using their expertise are able to identify the impact of the research of such papers. For all the indicators where we observed statistical significant differences between groups, the average value is higher in the group of selected applicants with exception of Exact Sciences for the variable hnf. The confidence intervals (CIs) presented in Table A5 corroborate the obtained results in the cases where we found statistical significant differences. The Cohen’s d indicates that the differences between selected and not selected applicants in the cases where there is statistical evidence to reject the null hypothesis are large with exception of Natural Sciences for the variable related to international collaboration (see Table A5). In the case of Engineering and Technology we have similarities, if we compare with starting grants, in what concerns to the dimensions where we observe statistical significant differences. However, the indicators describing such differences are not the same. The performance in terms of impact of the publications measured by the h index, by the percentage of papers highly cited (HCD), and by the average number of citations per paper is statistically different between selected and not selected applicants. Collaboration measured by the percentage of papers with international collaboration (DIC) is another dimension where we observed statistical significant differences. The TD, PDAC, and SJRm are slightly non-significant (P-value < 0.1). For all these variables, with exception for the PDAC, the selected applicants perform better than not selected applicants, in average. Due to the importance of conferences in this field, we expected to see the percentage of proceedings papers (PP) and/or the percentage of publications classified as article and proceedings paper (PAP) to be implicit in the peers’ judgments (statistical significant). However, the results of the t-test showed that both indicators are not statistically significant and, in the case of the PAP, we can state that it is highly non-significant (P-value = 0.586). In Natural Sciences statistical significant differences in performance are observed for the dimension impact when the average number of citations per document (NI) is used and the dimension related to the scientific independence as measured by the percentage of documents where the applicants is the correspondent author (PDAC). However, for the PDAC the mean value is the highest in the case of not selected applicants. This result is unclear for us, as we expected to find the highest value in the set of selected applicants (FCT suggests as criteria for assessing scientific independence the publications where the applicant is the corresponding author). In Exact Sciences, we only found statistical significant differences for the indicator hnf which is related to the impact of the individual publications. In average, the set of selected applicants has better performance than not selected applicants, and the difference is large (see Table A4). Table 4. Mean values for the two groups of applicants in the case of starting and development grants complemented by the CIs for the mean differences and the Cohen’s d statistic (effect size) Indicator  Development grants   Starting grants   Average   CI (95%)  Effect size  Average   CI (95%)  Effect size  Selected  Not selected  Selected  Not selected  TD  41.86  40.95  [−10.36; 10.07]  −0.03  29.74  26.15  [−11.26; 4.54]  −0.20  NDF  10.04  10.29  [−2.76; 3.20]  0.03  5.52  5.55  [−1.77; 1.95]  0.03  PA  79.04  71.75  [−13.97; −0.70]  −0.41  75.44  75.67  [−7.76; 9.00]  0.04  PR  3.54  3.22  [−1.94; 1.51]  −0.08  5.51  5.62  [−3.29; 3.83]  0.04  PP  4.78  10.12  [−0.76; 10.16]  0.45  4.03  6.16  [−3.14; 6.80]  0.20  PAP  8.52  10.56  [−2.22; 6.46]  0.18  6.81  8.05  [−3.77; 6.74]  0.10  PDAC  23.97  33.86  [2.26; 17.03]  0.53  21.61  21.47  [−10.89; 7.99]  −0.04  h  14.32  11.98  [−4.01; −0.51]  −0.51  10.26  9.48  [−2.65; 1.57]  −0.14  hnf  4.74  4.19  [−1.25; 0.14]  −0.29  3.04  3.22  [−0.66; 1.08]  0.10  HCD  18.49  10.65  [13.14; −2.12]  −0.54  18.72  13.79  [−14.33; 4.20]  −0.30  NI  1.35  1.19  [0.47; 0.19]  −0.20  1.27  1.08  [−0.75; 0.25]  −0.20  SJRm  1.61  1.14  [−0.87; −0.16]  −0.48  1.35  1.09  [−0.65; −0.034]  −0.50  SNIPm  1.2  1.13  [−0.17; 0.03]  −0.27  1.22  1.14  [−0.23; 0.05]  −0.23  PQ1  59.4  51.95  [−15.50; 0.76]  −0.36  52.49  51.79  [−13.54; 10.39]  −0.07  DIC  48.55  41.72  [−15.79; 1.82]  −0.33  42.36  28.77  [−22.05; −4.00]  −0.60  Indicator  Development grants   Starting grants   Average   CI (95%)  Effect size  Average   CI (95%)  Effect size  Selected  Not selected  Selected  Not selected  TD  41.86  40.95  [−10.36; 10.07]  −0.03  29.74  26.15  [−11.26; 4.54]  −0.20  NDF  10.04  10.29  [−2.76; 3.20]  0.03  5.52  5.55  [−1.77; 1.95]  0.03  PA  79.04  71.75  [−13.97; −0.70]  −0.41  75.44  75.67  [−7.76; 9.00]  0.04  PR  3.54  3.22  [−1.94; 1.51]  −0.08  5.51  5.62  [−3.29; 3.83]  0.04  PP  4.78  10.12  [−0.76; 10.16]  0.45  4.03  6.16  [−3.14; 6.80]  0.20  PAP  8.52  10.56  [−2.22; 6.46]  0.18  6.81  8.05  [−3.77; 6.74]  0.10  PDAC  23.97  33.86  [2.26; 17.03]  0.53  21.61  21.47  [−10.89; 7.99]  −0.04  h  14.32  11.98  [−4.01; −0.51]  −0.51  10.26  9.48  [−2.65; 1.57]  −0.14  hnf  4.74  4.19  [−1.25; 0.14]  −0.29  3.04  3.22  [−0.66; 1.08]  0.10  HCD  18.49  10.65  [13.14; −2.12]  −0.54  18.72  13.79  [−14.33; 4.20]  −0.30  NI  1.35  1.19  [0.47; 0.19]  −0.20  1.27  1.08  [−0.75; 0.25]  −0.20  SJRm  1.61  1.14  [−0.87; −0.16]  −0.48  1.35  1.09  [−0.65; −0.034]  −0.50  SNIPm  1.2  1.13  [−0.17; 0.03]  −0.27  1.22  1.14  [−0.23; 0.05]  −0.23  PQ1  59.4  51.95  [−15.50; 0.76]  −0.36  52.49  51.79  [−13.54; 10.39]  −0.07  DIC  48.55  41.72  [−15.79; 1.82]  −0.33  42.36  28.77  [−22.05; −4.00]  −0.60  Note: Values in bold represent variables with P-values < 0.05 obtained using the t-test. Table 4. Mean values for the two groups of applicants in the case of starting and development grants complemented by the CIs for the mean differences and the Cohen’s d statistic (effect size) Indicator  Development grants   Starting grants   Average   CI (95%)  Effect size  Average   CI (95%)  Effect size  Selected  Not selected  Selected  Not selected  TD  41.86  40.95  [−10.36; 10.07]  −0.03  29.74  26.15  [−11.26; 4.54]  −0.20  NDF  10.04  10.29  [−2.76; 3.20]  0.03  5.52  5.55  [−1.77; 1.95]  0.03  PA  79.04  71.75  [−13.97; −0.70]  −0.41  75.44  75.67  [−7.76; 9.00]  0.04  PR  3.54  3.22  [−1.94; 1.51]  −0.08  5.51  5.62  [−3.29; 3.83]  0.04  PP  4.78  10.12  [−0.76; 10.16]  0.45  4.03  6.16  [−3.14; 6.80]  0.20  PAP  8.52  10.56  [−2.22; 6.46]  0.18  6.81  8.05  [−3.77; 6.74]  0.10  PDAC  23.97  33.86  [2.26; 17.03]  0.53  21.61  21.47  [−10.89; 7.99]  −0.04  h  14.32  11.98  [−4.01; −0.51]  −0.51  10.26  9.48  [−2.65; 1.57]  −0.14  hnf  4.74  4.19  [−1.25; 0.14]  −0.29  3.04  3.22  [−0.66; 1.08]  0.10  HCD  18.49  10.65  [13.14; −2.12]  −0.54  18.72  13.79  [−14.33; 4.20]  −0.30  NI  1.35  1.19  [0.47; 0.19]  −0.20  1.27  1.08  [−0.75; 0.25]  −0.20  SJRm  1.61  1.14  [−0.87; −0.16]  −0.48  1.35  1.09  [−0.65; −0.034]  −0.50  SNIPm  1.2  1.13  [−0.17; 0.03]  −0.27  1.22  1.14  [−0.23; 0.05]  −0.23  PQ1  59.4  51.95  [−15.50; 0.76]  −0.36  52.49  51.79  [−13.54; 10.39]  −0.07  DIC  48.55  41.72  [−15.79; 1.82]  −0.33  42.36  28.77  [−22.05; −4.00]  −0.60  Indicator  Development grants   Starting grants   Average   CI (95%)  Effect size  Average   CI (95%)  Effect size  Selected  Not selected  Selected  Not selected  TD  41.86  40.95  [−10.36; 10.07]  −0.03  29.74  26.15  [−11.26; 4.54]  −0.20  NDF  10.04  10.29  [−2.76; 3.20]  0.03  5.52  5.55  [−1.77; 1.95]  0.03  PA  79.04  71.75  [−13.97; −0.70]  −0.41  75.44  75.67  [−7.76; 9.00]  0.04  PR  3.54  3.22  [−1.94; 1.51]  −0.08  5.51  5.62  [−3.29; 3.83]  0.04  PP  4.78  10.12  [−0.76; 10.16]  0.45  4.03  6.16  [−3.14; 6.80]  0.20  PAP  8.52  10.56  [−2.22; 6.46]  0.18  6.81  8.05  [−3.77; 6.74]  0.10  PDAC  23.97  33.86  [2.26; 17.03]  0.53  21.61  21.47  [−10.89; 7.99]  −0.04  h  14.32  11.98  [−4.01; −0.51]  −0.51  10.26  9.48  [−2.65; 1.57]  −0.14  hnf  4.74  4.19  [−1.25; 0.14]  −0.29  3.04  3.22  [−0.66; 1.08]  0.10  HCD  18.49  10.65  [13.14; −2.12]  −0.54  18.72  13.79  [−14.33; 4.20]  −0.30  NI  1.35  1.19  [0.47; 0.19]  −0.20  1.27  1.08  [−0.75; 0.25]  −0.20  SJRm  1.61  1.14  [−0.87; −0.16]  −0.48  1.35  1.09  [−0.65; −0.034]  −0.50  SNIPm  1.2  1.13  [−0.17; 0.03]  −0.27  1.22  1.14  [−0.23; 0.05]  −0.23  PQ1  59.4  51.95  [−15.50; 0.76]  −0.36  52.49  51.79  [−13.54; 10.39]  −0.07  DIC  48.55  41.72  [−15.79; 1.82]  −0.33  42.36  28.77  [−22.05; −4.00]  −0.60  Note: Values in bold represent variables with P-values < 0.05 obtained using the t-test. Within the dimension production and collaboration in Medical and Health Sciences, we have statistical evidence of differences between the two groups in case of variables TD and NDF. In average, the values for these indicators are high for the set of not selected applicants. If we think in terms of performance, we would say that the better the researcher is, the higher the number of publications. However, this can be explained looking to the remaining indicators. For the selected applicants we have, in average, a higher percentage of publications classified as articles than for the not selected applicants. We looked at the percentage of meeting abstracts, as this type of publications is widely used within this scientific field and found that for not selected applicants, the percentage of meetings abstracts is in general higher than for the selected applicants. Meeting abstracts are used frequently in the dissemination of current data, although, while timely and succinct, this type of documents is frequently considered less prestigious due to a simplified peer review and incomplete bibliographic description. These characteristics of the meeting abstracts might explain why applicants with a high number of publications, in average, and a high percentage of meeting abstracts were not selected. The impact and prestige of the journal where the applicants have been published in, represented by the SJRm, is, in average, high in the case of selected applicants, and the difference is statistically significant. In terms of substantive significance the Cohen’s d indicates large effect for all the variables where there is statistical evidence to state that the means between the two groups are not equal. The CIs corroborate our results (see Table A4). If we consider, in Tables 2 and 3, those indicators where we observed statistical significant differences we can conclude that there are differences within the same dimension. In Engineering and Technology we found statistical significant differences for indicators related to the dimension quantity, impact, and collaboration for both type of grants. However, the indicators are different; for example in the starting grants collaboration is represented by the NDF, while in the development grants it is represented by the DIC. Over all the results obtained, at this point, are very interesting, as they show that: among scientific fields different dimensions are implicit in the peers’ judgments; the dimensions implicit in the peers’ judgments are different if we considered applicants at different phase of their scientific career; the use of several bibliometric indicators allows us to understand better the scientific production of a given researcher. The indicator PA was not found statistically significant in the case of Medical and Health Sciences for the development grants, but it is very important in helping to understand the average values in the case of the TD and NDF. The requirements to apply to each type of grant are different, but the guidelines/criteria proposed by the FCT for evaluation are, practically, the same among type of grants and scientific fields. The results in Tables 2 and 3 show that statistical significant differences in performance for the two set of applicants are evidenced by different dimensions and indicators. In fact, applicants to starting grants are supposed to have a maximum of 5 years after the award of the PhD degree. These are researchers at the beginning of the career, most of them with a few papers published and are building their reputation in the field. Here, some metrics can be useful, but special attention should be given to those based on citations due to the low number of papers and time needed to the maturation of the citations (Glanzel and Schoepflin 1995). In such situations, the use of variables other than those associated with citations might be more appropriated. This might explain why in the case of starting grants just a few indicators based on the individual impact of the publications were able to show statistical significant differences between the selected and not selected applicants. The main findings suggest that the aspects considered more relevant in terms of scientific performance depend on the scientific field. This result was not unexpected in our study, despite the same criteria/guidelines to be used by the peers. As a well-known example, we have the field of Mathematics where collaboration activities are not common; in average, the number of authors per publication is lower than in Chemistry and Physics (Franceschet and Costantini 2011; Vieira and Gomes 2010). In this field the use of indicators of collaboration might not be relevant to assess scientific performance. An opposite case is the case of physics where international collaboration is common. Nevertheless, we were not able to make the analysis at this level in our study; we do not have enough information to justify the differences observed among scientific fields. Only a close discussion with the peers would provide information about the differences observed both at the different phase of the scientific career and at the level of scientific field. We were able to identify those indicators where there are statistical significant differences between the two groups of applicants, but we cannot discuss their relevance in explaining peers’ judgments. This can be explored using a regression analysis, more specifically a logistic regression. However, the low number of applicants in each field makes this methodology not suitable. For this reason, the remainder of the section deals with the analysis of the whole set, taking together all different scientific areas. In reality, we know that there are marked differences of culture among areas both in terms of publication and evaluation, but the general criteria defined by the FCT are the same for all areas. Information about the dispersion in the data set is presented in the box plots, which follows for each type of grant (more descriptive statistics can be found in Table A1). In Figs 1 and 2 we can see that for most of the bibliometric indicators the distributions are skewed (e.g. SJRm and h index); there are some outstanding researchers, and high dispersion is a feature of the data set. This is also confirmed by the results presented in Table A1. Figure 1. View largeDownload slide Distribution of the data for selected (Code 1) and not selected (Code 0) applicants according to each indicator for the starting grants. Figure 1. View largeDownload slide Distribution of the data for selected (Code 1) and not selected (Code 0) applicants according to each indicator for the starting grants. Figure 2. View largeDownload slide Distribution of the data for selected (Code 1) and not selected (Code 0) applicants according to each indicator in the development grants. Figure 2. View largeDownload slide Distribution of the data for selected (Code 1) and not selected (Code 0) applicants according to each indicator in the development grants. There are applicants with a large number of documents published, but a detailed inspection of the data revealed that for most of these applicants the mean impact of the publications as measured by the NI is below the average value of the reference set. It means applicants with high productivity, but low impact. The distribution for the DIC indicator shows researchers seeking to work in an environment that is characterized by international collaborations. This might be a consequence of the scientific policies that encourage and promote research in such environment. Some applicants present very high value for this indicator. There are researchers with a very high number of highly cited publications (HCD). Research of high standards are being carried out by these researchers; this indicator is considered in the scientific community as an indicator of excellence (Bornmann, Anegon and Leydesdorff 2012; Tijssen, Visser and van Leeuwen 2002; Waltman et al. 2012). For the indicator NI, we can observe some outliers. Most of these outliers are due to the presence of one or two documents highly cited. The NI being an average value is sensible to these documents. This clearly shows the advantage of using several bibliometric indicators even when describing performance within the same dimension. Outliers are essentially observed for the indicators related to the type of documents. The outliers observed for the PP and PAP are, mainly, applicants from the Engineering and Technology field. These results are in agreement with the literature, i.e. proceedings paper is an important vehicle of dissemination of the results of the research activities in the Engineering and Technology (Butler 2008; Montesi and Owen 2008). In the case of the h index we also observe outliers. These researchers have a large number of documents published; previous studies show a positive association of the h index with the absolute number of publications (Costas and Bordons 2007; van Raan 2006). The correlation tables(Tables A2 and A3) show moderate and significant correlations between the total number of published documents and the h index. As the indicator does not consider any correction for the different culture of publication among fields, researchers with a high number of publications are expected to have high h index. In Fig. 2 we show information for the development grants. The overlap between the distributions of the selected and not selected applicants is lower than in the starting grants. Outliers are present, in general terms, for the same indicators as in the starting grants, and the factors explaining such behaviour are the same. Obviously, a large number of not selected applicants show higher indicator value than some selected applicants. This does not mean a questionable peer-review process. This might mean that if applicants pass a certain threshold, other evaluation criteria (e.g. grants awarded, abilities and skills to adequately execute the proposed project, and relevance and originality of the project proposed (see FCT 2012b)) dominate the final decision. We present, in Table 4, the observed average value for each indicator within the group of selected and not selected applicants. In the case of starting grants the results of the t-test shows that the SJRm and DIC are the unique variables for which statistical significant differences are observed between the selected and not selected applicants. The CIs corroborate the results and the differences between the two groups vary between moderate and large. For the remaining variables there is no statistical evidence to consider that the performance of the groups is different, and the differences are considered of small effect. In the case of development grants for 6 of 15 indicators we identified statistical significant differences between the two groups being predominant moderate differences. The indicators represent several dimensions: quantity, impact, and scientific independence. 4.2 The impact of different dimensions The results presented above showed that there are statistically significant differences between the sets of selected and not selected applicants, although we do not have information about how the indicators can be combined in a way that contains information about the effect size and the capability of representing the peers’ final decision. The logistic regression was applied to the applicants of each grant type, using as independent variables, the bibliometric indicators that showed statistically significant differences between the two groups, and as dependent variable, a binary variable that represents the final decision (1—applicant selected; 0—applicant not selected). The coefficient value is the mean value; cross-validation was used in adjusting the model. Between brackets is the coefficient of variation for each coefficient. Several models were obtained using the variables for which the t-test showed statistically significant differences. In Table 5, we only present the model that showed to have the highest performance in explaining the final decisions of the peers. The results obtained allow us to answer to our research question; the dimensions implicit in peers’ judgments and most valued are different between type of grants. In the case of development grants the impact and the prestige of the source used to disseminate the research findings (as measured by the SJRm) are implicit in the peer judgments. The same is likely to be true for the research that is considered highly cited or excellent (Bornmann et al. 2012; Tijssen et al. 2002; Waltman et al. 2012) as measured by the HCD. Both indicators have a positive impact, and the effect size is similar with a slight advantage for applicants with high SJRm. For the starting grants the indicator that gives information about the traffic of international collaboration is the only one with significant and positive impact, and, in fact, the degree of internationalization of the applicant is suggested in the guide for peer review. The coefficients of the models were obtained using cross-validation due to the small amount of data. The values for the coefficients represent the mean value, and within brackets is the coefficient of variation; the values of the coefficient of variation suggest that our models are robust in relation to the used data set. Considering 50% as probability of cut-off in studying the several measures of fit of the models, presented in Table A6, we conclude that the models fit well the data. The percentage of cases predicted correctly is similar for both models, about 70%. It is important to highlight that there is a set of criteria that could not be considered in our study as the research project and the career development plan, which have a weight of 40% on the total final decision, as well as additional information about scientific merit (grants and/or projects awarded in very competitive calls, doctoral and post-doctoral training). Considering the limited scope of analysis of our indicators in relation to the whole set of features the peers are asked to assess, the 70% overlap found here is very good indeed. Table 6. Overlapping between the results given by the models and the peer decisions among scientific fields Scientific field  Correct prediction (%)   Starting grant  Development grant  Engineering and Technology  71  76  Natural Sciences  76  73  Exact Sciences  67  67  Medical and Health Sciences  47  69  All areas  69.7  68.8  Scientific field  Correct prediction (%)   Starting grant  Development grant  Engineering and Technology  71  76  Natural Sciences  76  73  Exact Sciences  67  67  Medical and Health Sciences  47  69  All areas  69.7  68.8  Table 6. Overlapping between the results given by the models and the peer decisions among scientific fields Scientific field  Correct prediction (%)   Starting grant  Development grant  Engineering and Technology  71  76  Natural Sciences  76  73  Exact Sciences  67  67  Medical and Health Sciences  47  69  All areas  69.7  68.8  Scientific field  Correct prediction (%)   Starting grant  Development grant  Engineering and Technology  71  76  Natural Sciences  76  73  Exact Sciences  67  67  Medical and Health Sciences  47  69  All areas  69.7  68.8  For the development grants, actually selected applicants for funding for whom the model predicts a probability of being selected by the peers higher than 50% is about 71% and for the starting grants is about 80% showing the high sensitivity of the models. In relation to specificity, the models show a lower performance, only for 65% and 55.6% (development and starting grant respectively) of the total applicants not selected for funding the models predict a probability of being selected by the peers lower than the value established as cut-off, which implies a considerable amount of the false positives, especially for the starting grant (44.4%). Concerning the discrimination power of the models (AUC, Table A6), the literature suggests a poor discrimination (0.5 < AUC < 0.7) for the model obtained for the starting grants; for the development grants, the power discrimination is acceptable (0.7 < AUC < 0.8) (Hosmer, Lemeshow and Sturdivant 2013). Visual observation of the ROC curve is presented in Fig. A1. This calls for some caution in the interpretation of the results in the case of starting grants. Taking into account the predicted probabilities for each observation, we analysed the percentage of observations predicted correctly in each scientific domain (Table 6). Not all the indicators presented in Table 5 were found to be important in identifying statistical significant differences in the performance between selected and not selected applicants for some fields, in Tables 2 and 3. For these fields we expect a lower percentage of cases predicted correctly. In the case of development grants for the field Engineering and Technology using the obtained model, the percentage of cases predicted correctly is the highest; the SJRm is slightly non-significant (0.5 < P-value < 0.1), and the HCD is significant in Table 3. All in all when interpreting the results given by our models, we have to be aware about the limitations introduced by the design of the study. In finding information about the effect size and capability of the indicators in representing the peers’ final decisions, we are not controlling for the effect of scientific field, and this might introduce some unquantifiable biases. We were able to predict a reasonable overlap between the results from bibliometric analysis and peers’ decisions, but our results would improve if we had considered the effect of the scientific field. The results obtained are in concordance with previous studies. For studies at the level of individuals, the HCD was found with positive and significant influence in predicting the peers’ judgments (Vieira et al. 2014a, 2014b). The impact of the journal where the applicants have published in was also found relevant in (Cabezas-Clavijo et al. 2013), although the authors used a different indicator, the JIF. We deem the indicator SJRm more appropriate, as it is a normalized indicator for the different culture of citations among fields. 5. Discussion When using bibliometric indicators for understanding the peer-reviewed process at the programme Investigador FCT, we are looking for variables that to some extent are related to those dimensions that peers appear to consider more relevant at each phase of the scientific career. The general criteria of evaluation were defined by FCT for all peers, but they use these criteria in accordance with their own understanding of scientific merit and quality. The bibliometric indicators used in this study were not made available to the peers in a formal way, and it is unlikely that most peers will have calculated them, especially those that require a normalization procedure. All we can test is that certain parameters correlate well with the decision not implying necessarily a direct causal relation. We will argue that some of the indicators found to be relevant may be understood as some measure of what the peers may be looking at. In the case of the starting grants, the results from Table 5 show that international collaboration is able to reproduce peer decisions to some extent. We must address the following questions: Why is international collaboration so important? What does international collaboration tell us about scientific merit? On one hand, it may be argued that peers are considering as important criteria in the evaluation process aspects that are correlated with international collaboration. For example, peers might be considering as relevant the grants/projects awarded in competitive calls as this was mentioned in the guidelines for evaluation. The literature shows that grants/projects awarded facilitate in establishing international collaboration (Bloch, Graversen and Pedersen 2014). On the other hand, peers are valuing the expected future performance, and the literature shows that international collaboration is determinant in enhancing high scientific performance (Bordons, Aparicio and Costas 2013; Glanzel 2001). In this situation we can state that peers are considering past and actual international collaborations as a predictor of a career of high performance. In the case of the development grants the dimension impact, represented by indicators SJRm and HCD, is the most relevant in predicting the application success. There are several situations where indicators related to the impact and prestige of the journals are being used in the individual assessment of publications as a way of replacing citation statistics. This practice raised several debates in the scientific community, mainly within bibliometricians, who condemn this behaviour. Taking into account the debate in the scientific community we would like to highlight that the results obtained in our study cannot be interpreted without looking to the evaluation criteria. In fact, in the document related to the guidelines for peer review it is said: ‘Indicators for scientific merit of the applicant include the main academic and professional degrees, publications in top specialty peer-reviewed journals and/or in major multidisciplinary international peer-reviewed journals’ (FCT 2012b: 4). Therefore, the results obtained are in agreement with some of the criteria that are supposed to be used in the evaluation process. The HCD variable is normally considered to point to excellent research (Tijssen et al. 2002; Waltman et al. 2012); in fact, in the majority of the cases, research with a very large number of citations in the field means high impact in the community; impact is a necessary condition for researchers and institutions to achieve broad visibility and success in terms of scientific recognition. At this level we can conclude that peers are looking for scientists that produce research of high standards and that are already established as research leaders in their own right. 6. Conclusions This work seeks a better understanding of the selection process carried out by peers at the individual level, in this specific case, at the programme Investigador FCT, in Portugal. The programme aims at selecting outstanding researchers into Portugal’s R&D institutions. Applicants should have an exceptional track record and a clear potential to develop innovative research. Using a set of bibliometric indicators, chosen taking into account previous studies and the evaluation guidelines provided by the FCT to the peers, we tried to understand the peer-review process looking for the indicators that are implicit in the peers’ judgments, identifying the dimensions of the scientific performance more valued by the peers. Our empirical results show that bibliometric indicators allow an understanding of the selection process identifying differences on scientific performance among applicants as seen by the peers: we highlight here the dimensions more valued by the community of peers. These results may be used to attenuate the subjectivity or the bias always associated with peer-review decisions by small expert panels. For applicants with a very short career, we found that the percentage of documents with international collaboration is able to predict the final decision of the peers for 70% of the applicants. International collaboration depends on a network of co-authors that somehow acknowledge the applicant’s research. This suggests that better researchers tend to have more intense international collaboration. Furthermore, we cannot rule out that international collaboration may be related to other criteria used by the peers that we were not able to use in this study. More refined conclusions in relation to the meaning of international collaboration can only be obtained if these results were discussed in detail with the peer panels. In the case of development grants the indicators that showed better performance in representing the final decisions were the SJRm and the HCD; for 70% of the applicants the prediction given by the bibliometric indicators overlaps with the peer decisions. If both indicators were made available to the peers, they would have a positive and significant impact on the final decision. The impact and prestige of the journals as measured by the SJRm would be the variable with the highest impact. When scientific domains are considered separately, different bibliometric indicators showed differences among applicants reflecting the role of the very different cultures of publication and citation of the scientific domains. However, a more in-depth analysis of this behaviour is necessary aimed at determining the relevance of each indicator in predicting the final peers’ decisions. This was not possible as the number of applicants in each field is not enough to draw statistically robust results. The overlap between the results given by the bibliometric indicators and the peers’ decisions is very encouraging, as there are several other factors that might influence peers’ decisions. Most of the indicators used are based on citation data, and this is perceived as a measure of impact of the scientific output (its actual influence on surrounding research activities at a given time). However, peers when assessing scientific production might also consider their personal assessment of the relevance (the influence of research on the advance of scientific knowledge) and technical quality of the research, whether it is free from obvious flaws, how aesthetically pleasing the mathematical formulations are, how original the conclusions are, and so on. On the other hand, peers were also asked, in the evaluation guidelines, to take into account the number of grants awarded, doctoral and post-doctoral training, the career development plan, and the research project, and this information was not made available for our study. The results presented in the article suggest that bibliometric indicators can have a role in peer assessment exercises. They were not available to the peer review in a formal way as far as we know; therefore in our study we can only state that the indicators found important in predicting the final decisions are implicit in the judgements made by the peers. Bibliometric indicators are known by their objectivity, and we might interrogate about the design of national evaluation exercises where the assessment is purely based on the results given by the bibliometric indicators. An evaluation exercise based only in bibliometric indicators requires a set of procedures that go from building an appropriate infrastructure to store all the data needed to the calculation of the bibliometric indicators until a clear and objective definition of the several aspects to be evaluated. The last point inevitably implies a massive interaction of bibliometric experts and experts belonging to the several fields under evaluation aimed at defining clearly what the main cultures of publication and citation are. On the other side, despite the attractive features of bibliometric indicators, there are several situations where they are not able to replace the role of the peers; e.g, it will be very difficult to assess the impact of a recent publication if we measure impact counting citations. A citation window between 3 and 5 years is required to collect the citations of publications. In a case where a system purely based on bibliometric indicators is to be used, it will be very difficult to have a clear picture about the impact of recent publications. Another example where bibliometric indicators fail is in book-oriented fields as counting citations from articles, which dominate traditional citation indexes, seems insufficient to assess the impact of books. The two examples show that bibliometric indicators can work better as a complementary tool of peer-review processes than as tool to replace them. The results presented here should be seen as preliminary results. We have identified the most important bibliometric indicators in predicting the final decisions from our set of 15 indicators. However, we should not discard the possibility of using other indicators that contribute to a better understanding of the scientific performance, even if they do not have a decisive role in building the final decisions. The very general evaluation criteria presented by the FCT do not allow us to extend our discussion into the design of an evaluation system, with the same goals as that of the FCT, where bibliometric indicators are to be used as complementary tool. Such design implies a clear and objective definition of the criteria to be evaluated and a strong interaction between bibliometricians and peers aimed at defining the most relevant aspects of the scientific performance in each domain and the most appropriate bibliometric indicators and the way they should be used. Acknowledgements The authors wish to acknowledge the data on the Investigador FCT applicants that were kindly made available by Fundação para a Ciência e a Tecnologia. Funding This work was supported by the Fundação para a Ciência e a Tecnologia (FCT) from Portugal, [grant number SFRH/BPD/99246/2013]. References Abramo G., D'Angelo C. A. ( 2015) ‘ The VQR, Italy's Second National Research Assessment: Methodological Failures and Ranking Distortions’, Journal of the Association for Information Science and Technology , 66/ 11: 2202– 14. Google Scholar CrossRef Search ADS   Abramo G., D'Angelo C. A., Caprasecca A. ( 2009) ‘ Allocative Efficiency in Public Research Funding: Can Bibliometrics Help?’, Research Policy , 38/ 1: 206– 15. Google Scholar CrossRef Search ADS   Aksnes D. W., Taxt R. E. ( 2004) ‘ Peer Reviews and Bibliometric Indicators: A Comparative Study at a Norwegian University’, Research Evaluation , 13/ 1: 33– 41. Google Scholar CrossRef Search ADS   Bloch C., Graversen E. K., Pedersen H. S. ( 2014) ‘ Competitive Research Grants and Their Impact on Career Performance’, Minerva , 52/ 1: 77– 96. Google Scholar CrossRef Search ADS   Bordons M., Aparicio J., Costas R. ( 2013) ‘ Heterogeneity of Collaboration and Its Relationship with Research Impact in a Biomedical Field’, Scientometrics , 96/ 2: 443– 66. Google Scholar CrossRef Search ADS   Bornmann L., Anegon F. D., Leydesdorff L. ( 2012) ‘ The New Excellence Indicator in the World Report of the SCImago Institutions Rankings 2011’, Journal of Informetrics , 6/ 2: 333– 5. Google Scholar CrossRef Search ADS   Bornmann L., Leydesdorff L. ( 2013) ‘ The Validation of (Advanced) Bibliometric Indicators Through Peer Assessments: A Comparative Study Using Data from InCites and F1000’, Journal of Informetrics , 7/ 2: 286– 91. Google Scholar CrossRef Search ADS   Bornmann L., Marx W. ( 2015) ‘ Methods for the Generation of Normalized Citation Impact Scores in Bibliometrics: Which Method Best Reflects the Judgements of Experts?’, Journal of Informetrics , 9/ 2: 408– 18. Google Scholar CrossRef Search ADS   Bornmann L., Wallon G., Ledin A. ( 2008) ‘ Does the Committee Peer Review Select the Best Applicants for Funding? An Investigation of the Selection Process for Two European Molecular Biology Organization Programmes’, PLoS One , 3/ 10: e3480. Google Scholar CrossRef Search ADS PubMed  Butler L. ( 2008) ‘ ICT Assessment: Moving beyond Journal Outputs’, Scientometrics , 74/ 1: 39– 55. Google Scholar CrossRef Search ADS   Butler L., McAllister I. ( 2011) ‘ Evaluating University Research Performance Using Metrics’, European Political Science , 10/ 1: 44– 58. Google Scholar CrossRef Search ADS   Cabezas-Clavijo A. et al.   ( 2013) ‘ Reviewers' Ratings and Bibliometric Indicators: Hand in Hand When Assessing over Research Proposals?’, PLoS One , 8/ 6: e68258. Google Scholar CrossRef Search ADS PubMed  Costas R., Bordons M. ( 2007) ‘ The h-Index: Advantages, Limitations and Its Relation with Other Bibliometric Indicators at the Micro Level’, Journal of Informetrics , 1/ 3: 193– 203. Google Scholar CrossRef Search ADS   ERC ( 2015). ERC Frontier Research Grants—Guide for Peer Reviewers’, <https://erc.europa.eu/sites/default/files/document/file/Guide-for-Peer_reviewers_StG_CoG_AdG_2015.pdf> accessed March 2016. FCT ( 2012a). ‘Guide for Writing and Submitting Applications for the FCT Investigator Grants’, <http://www.fct.pt/apoios/contratacaodoutorados/investigador-fct/2012/docs/ApplicationGuide.pdf> accessed December 2015. FCT ( 2012b). ‘Investigator FCT Grants—Guide for Peer Reviewers’, <http://www.fct.pt/apoios/contratacaodoutorados/investigador-fct/2012/docs/GuidelinesPeerReview.pdf> accessed December 2015. FCT ( 2014). ‘Evaluation guide for the 2014FCT Investigator Grant’, <http://www.fct.pt/apoios/contratacaodoutorados/investigador-fct/2014/docs/EvaluationGuide_IF2014.pdf> accessed December 2015. Franceschet M., Costantini A. ( 2011) ‘ The First Italian Research Assessment Exercise: A Bibliometric Perspective’, Journal of Informetrics , 5/ 2: 275– 91. Google Scholar CrossRef Search ADS   Glanzel W. ( 2001) ‘ National Characteristics in International Scientific Co-Authorship Relations’, Scientometrics , 51/ 1: 69– 115. Google Scholar CrossRef Search ADS   Glanzel W., Schoepflin U. ( 1995) ‘ A Bibliometric Study on Aging and Reception Processes of Scientific Literature’, Journal of Information Science , 21/ 1: 37– 53. Google Scholar CrossRef Search ADS   Gonzalez-Pereira B., Guerrero-Bote V. P., Moya-Anegon F. ( 2010) ‘ A New Approach to the Metric of Journals' Scientific Prestige: The SJR Indicator’, Journal of Informetrics , 4/ 3: 379– 91. Google Scholar CrossRef Search ADS   Hirsch J. E. ( 2005) ‘ An Index to Quantify an Individual's Scientific Research Output’, Proceedings of the National Academy of Sciences of the United States of America , 102/ 46: 16569– 72. Google Scholar CrossRef Search ADS PubMed  Hosmer D. W.Jr., Lemeshow S., Sturdivant R. X. ( 2013). Applied Logistic Regression , 3rd edn. Wiley (April 1, 2013). Google Scholar CrossRef Search ADS   Jensen P., Rouquier J. B., Croissant Y. ( 2009) ‘ Testing Bibliometric Indicators by Their Prediction of Scientists Promotions’, Scientometrics , 78/ 3: 467– 79. Google Scholar CrossRef Search ADS   Lehmann S., Jackson A. D., Lautrup B. E. ( 2008) ‘ A Quantitative Analysis of Indicators of Scientific Performance’, Scientometrics , 76/ 2: 369– 90. Google Scholar CrossRef Search ADS   Meyer B. et al.   ( 2009) ‘ Research Evaluation for Computer Science’, Communications of the ACM , 52/ 4: 31– 4. Google Scholar CrossRef Search ADS   Moed H. F. ( 2010) ‘ Measuring Contextual Citation Impact of Scientific Journals’, Journal of Informetrics , 4/ 3: 265– 77. Google Scholar CrossRef Search ADS   Montesi M., Owen J. M. ( 2008) ‘ From Conference to Journal Publication: How Conference Papers in Software Engineering are Extended for Publication in Journals’, Journal of the American Society for Information Science and Technology , 59/ 5: 816– 29. Google Scholar CrossRef Search ADS   Neufeld J., Huber N., Wegner A. ( 2013) ‘ Peer Review-Based Selection Decisions in Individual Research Funding, Applicants' Publication Strategies and Performance: The Case of the ERC Starting Grants’, Research Evaluation , 22/ 4: 237– 47. Google Scholar CrossRef Search ADS   Norris M., Oppenheim C. ( 2003) ‘ Citation Counts and the Research Assessment Exercise V—Archaeology and the 2001 RAE’, Journal of Documentation , 59/ 6: 709– 30. Google Scholar CrossRef Search ADS   Oppenheim C. ( 1997) ‘ The Correlation between Citation Counts and the 1992 Research Assessment Exercise Ratings for British Research in Genetics, Anatomy and Archaeology’, Journal of Documentation , 53/ 5: 477– 87. Google Scholar CrossRef Search ADS   Ramos A., Sarrico C. S. ( 2016) ‘ Past Performance does not Guarantee Future Results: Lessons from the Evaluation of Research Units in Portugal’, Research Evaluation , 25: 94– 106. Google Scholar CrossRef Search ADS   Rinia E. J. et al.   ( 1998) ‘ Comparative Analysis of a Set of Bibliometric Indicators and Central Peer Review Criteria—Evaluation of Condensed Matter Physics in The Netherlands’, Research Policy , 27/ 1: 95– 107. Google Scholar CrossRef Search ADS   Taylor J. ( 2011) ‘ The Assessment of Research Quality in UK Universities: Peer Review or Metrics?’, British Journal of Management , 22/ 2: 202– 17. Google Scholar CrossRef Search ADS   Tijssen R. J. W., Visser M. S., van Leeuwen T. N. ( 2002) ‘ Benchmarking International Scientific Excellence: Are Highly Cited Research Papers an Appropriate Frame of Reference?’, Scientometrics , 54/ 3: 381– 97. Google Scholar CrossRef Search ADS   Torres-Salinas D., Jimenez-Contreras E. ( 2010) ‘ Introduction and Comparative Study of the New Scientific Journals Citation Indicators in Journal Citation Reports and Scopus’, Profesional De La Informacion , 19/ 2: 201– 7. Google Scholar CrossRef Search ADS   van Raan A. F. J. ( 2006) ‘ Comparison of the Hirsch-Index with Standard Bibliometric Indicators and with Peer Judgment for 147 Chemistry Research Groups’, Scientometrics , 67/ 3: 491– 502. Google Scholar CrossRef Search ADS   Vieira E. S., Cabral J. A. S., Gomes J. A. N. F. ( 2014a) ‘ Definition of a Model Based on Bibliometric Indicators for Assessing Applicants to Academic Positions’, Journal of the Association for Information Science and Technology , 65/ 3: 560– 77. Google Scholar CrossRef Search ADS   Vieira E. S., Cabral J. A. S., Gomes J. A. N. F. ( 2014b) ‘ How Good is a Model Based on Bibliometric Indicators in Predicting the Final Decisions Made by Peers?’, Journal of Informetrics , 8/ 2: 390– 405. Google Scholar CrossRef Search ADS   Vieira E. S., Gomes J. A. N. F. ( 2010) ‘ Citations to Scientific Articles: Its Distribution and Dependence on the Article Features’, Journal of Informetrics , 4/ 1: 1– 13. Google Scholar CrossRef Search ADS   Vieira E. S., Gomes J. A. N. F. ( 2011) ‘ An Impact Indicator for Researchers’, Scientometrics , 89/ 2: 607– 29. Google Scholar CrossRef Search ADS   Vieira E. S., Gomes J. A. N. F. ( 2015) ‘ The Bibliometric Indicators as Predictors of the Final Decision of the Peer Review’, Research Evaluation , 25: 170– 83. Google Scholar CrossRef Search ADS   Waltman L. et al.   ( 2012) ‘ The Leiden Ranking 2011/2012: Data Collection, Indicators, and Interpretation’, Journal of the American Society for Information Science and Technology , 63/ 12: 2419– 32. Google Scholar CrossRef Search ADS   Wilsdon J. et al.   ( 2015). ‘The Metric Tide: Report of the Independent Review of the Role of Metrics in Research Assessment and Management’. Appendix Descriptive results Significance test and substantive significance of effects In Tables that follows we present the CIs obtained using bootstrapping techniques and the Cohen’s d statistics used to complement the results of the t-test. Logistic regression We determined a set of measures that allow assessing the performance of the models and present them in Table A6. Table A1. Descriptive statistics for starting and development grants Variable  Development grant   Starting grant   Mean  p50  SD  Maximum  Minimum  Mean  p50  SD  Maximum  Minimum  TD  41.85  36.00  27.39  208.00  14.00  28.27  24.00  16.93  74.00  10.00  NDF  10.32  7.90  7.76  45.75  2.78  5.53  4.26  3.78  19.01  0.85  PA  76.44  81.37  17.83  98.28  32.50  75.53  76.83  18.13  100.00  14.81  PR  3.31  2.33  4.27  20.00  0.00  5.56  2.82  7.43  30.77  0.00  PP  6.75  2.33  11.84  51.35  0.00  4.90  0.00  10.32  40.00  0.00  PAP  9.32  5.33  11.62  55.56  0.00  7.32  2.74  10.95  60.61  0.00  PDAC  27.95  24.72  19.47  84.21  0.00  21.55  16.87  21.31  82.05  0.00  h  13.39  13.00  4.75  30.00  5.00  9.94  10.00  4.33  25.00  4.00  hnf  4.55  4.24  1.94  12.92  1.85  3.11  2.82  1.76  8.04  0.22  HCD  14.48  9.31  14.85  66.67  0.00  16.70  12.50  18.94  100.00  0.00  NI  1.29  1.09  0.87  6.21  0.29  1.19  1.00  0.95  7.09  0.16  SNIPm  1.16  1.18  0.24  2.19  0.40  1.19  1.19  0.26  1.99  0.32  SJRm  1.40  1.17  1.02  7.85  0.20  1.24  1.08  0.66  3.59  0.00  PQ1  55.68  56.24  20.75  100.00  4.55  52.21  53.14  22.23  89.36  5.26  DIC  60.96  63.00  28.05  100.00  0.00  36.80  35.31  22.18  82.35  0.00  Variable  Development grant   Starting grant   Mean  p50  SD  Maximum  Minimum  Mean  p50  SD  Maximum  Minimum  TD  41.85  36.00  27.39  208.00  14.00  28.27  24.00  16.93  74.00  10.00  NDF  10.32  7.90  7.76  45.75  2.78  5.53  4.26  3.78  19.01  0.85  PA  76.44  81.37  17.83  98.28  32.50  75.53  76.83  18.13  100.00  14.81  PR  3.31  2.33  4.27  20.00  0.00  5.56  2.82  7.43  30.77  0.00  PP  6.75  2.33  11.84  51.35  0.00  4.90  0.00  10.32  40.00  0.00  PAP  9.32  5.33  11.62  55.56  0.00  7.32  2.74  10.95  60.61  0.00  PDAC  27.95  24.72  19.47  84.21  0.00  21.55  16.87  21.31  82.05  0.00  h  13.39  13.00  4.75  30.00  5.00  9.94  10.00  4.33  25.00  4.00  hnf  4.55  4.24  1.94  12.92  1.85  3.11  2.82  1.76  8.04  0.22  HCD  14.48  9.31  14.85  66.67  0.00  16.70  12.50  18.94  100.00  0.00  NI  1.29  1.09  0.87  6.21  0.29  1.19  1.00  0.95  7.09  0.16  SNIPm  1.16  1.18  0.24  2.19  0.40  1.19  1.19  0.26  1.99  0.32  SJRm  1.40  1.17  1.02  7.85  0.20  1.24  1.08  0.66  3.59  0.00  PQ1  55.68  56.24  20.75  100.00  4.55  52.21  53.14  22.23  89.36  5.26  DIC  60.96  63.00  28.05  100.00  0.00  36.80  35.31  22.18  82.35  0.00  Table A1. Descriptive statistics for starting and development grants Variable  Development grant   Starting grant   Mean  p50  SD  Maximum  Minimum  Mean  p50  SD  Maximum  Minimum  TD  41.85  36.00  27.39  208.00  14.00  28.27  24.00  16.93  74.00  10.00  NDF  10.32  7.90  7.76  45.75  2.78  5.53  4.26  3.78  19.01  0.85  PA  76.44  81.37  17.83  98.28  32.50  75.53  76.83  18.13  100.00  14.81  PR  3.31  2.33  4.27  20.00  0.00  5.56  2.82  7.43  30.77  0.00  PP  6.75  2.33  11.84  51.35  0.00  4.90  0.00  10.32  40.00  0.00  PAP  9.32  5.33  11.62  55.56  0.00  7.32  2.74  10.95  60.61  0.00  PDAC  27.95  24.72  19.47  84.21  0.00  21.55  16.87  21.31  82.05  0.00  h  13.39  13.00  4.75  30.00  5.00  9.94  10.00  4.33  25.00  4.00  hnf  4.55  4.24  1.94  12.92  1.85  3.11  2.82  1.76  8.04  0.22  HCD  14.48  9.31  14.85  66.67  0.00  16.70  12.50  18.94  100.00  0.00  NI  1.29  1.09  0.87  6.21  0.29  1.19  1.00  0.95  7.09  0.16  SNIPm  1.16  1.18  0.24  2.19  0.40  1.19  1.19  0.26  1.99  0.32  SJRm  1.40  1.17  1.02  7.85  0.20  1.24  1.08  0.66  3.59  0.00  PQ1  55.68  56.24  20.75  100.00  4.55  52.21  53.14  22.23  89.36  5.26  DIC  60.96  63.00  28.05  100.00  0.00  36.80  35.31  22.18  82.35  0.00  Variable  Development grant   Starting grant   Mean  p50  SD  Maximum  Minimum  Mean  p50  SD  Maximum  Minimum  TD  41.85  36.00  27.39  208.00  14.00  28.27  24.00  16.93  74.00  10.00  NDF  10.32  7.90  7.76  45.75  2.78  5.53  4.26  3.78  19.01  0.85  PA  76.44  81.37  17.83  98.28  32.50  75.53  76.83  18.13  100.00  14.81  PR  3.31  2.33  4.27  20.00  0.00  5.56  2.82  7.43  30.77  0.00  PP  6.75  2.33  11.84  51.35  0.00  4.90  0.00  10.32  40.00  0.00  PAP  9.32  5.33  11.62  55.56  0.00  7.32  2.74  10.95  60.61  0.00  PDAC  27.95  24.72  19.47  84.21  0.00  21.55  16.87  21.31  82.05  0.00  h  13.39  13.00  4.75  30.00  5.00  9.94  10.00  4.33  25.00  4.00  hnf  4.55  4.24  1.94  12.92  1.85  3.11  2.82  1.76  8.04  0.22  HCD  14.48  9.31  14.85  66.67  0.00  16.70  12.50  18.94  100.00  0.00  NI  1.29  1.09  0.87  6.21  0.29  1.19  1.00  0.95  7.09  0.16  SNIPm  1.16  1.18  0.24  2.19  0.40  1.19  1.19  0.26  1.99  0.32  SJRm  1.40  1.17  1.02  7.85  0.20  1.24  1.08  0.66  3.59  0.00  PQ1  55.68  56.24  20.75  100.00  4.55  52.21  53.14  22.23  89.36  5.26  DIC  60.96  63.00  28.05  100.00  0.00  36.80  35.31  22.18  82.35  0.00  Table A2. Correlations among variables for the development grant Indicators  SJRm  SNIPm  h  hnf  NDF  NI  HCD  DIC  PA  PR  PAP  PP  TD  PQ1  PDAC  SJRm  1.00                              SNIPm  0.69  1.00                            h  0.16  0.36  1.00                          hnf  −0.06  0.10  0.57  1.00                        NDF  −0.13  −0.08  0.36  0.84  1.00                      NI  0.27  0.32  0.47  −0.08  −0.14  1.00                    HCD  0.19  0.34  0.46  0.04  −0.10  0.55  1.00                  DIC  0.29  0.26  0.07  −0.05  −0.03  0.30  0.15  1.00                PA  0.39  0.45  0.22  0.25  0.06  0.06  0.08  0.11  1.00              PR  0.19  0.17  0.22  −0.07  −0.19  0.22  0.35  −0.02  0.05  1.00            PAP  −0.31  −0.26  −0.11  −0.11  −0.07  −0.11  −0.21  0.05  −0.61  −0.27  1.00          PP  −0.34  −0.51  −0.33  −0.14  0.08  −0.18  −0.22  −0.15  −0.66  −0.29  0.18  1.00        TD  −0.17  0.00  0.65  0.51  0.60  0.31  0.04  0.07  −0.09  −0.16  0.17  0.02  1.00      PQ1  0.63  0.73  0.32  0.04  −0.10  0.32  0.27  0.33  0.40  0.23  −0.22  −0.53  −0.03  1.00    PDAC  −0.16  −0.19  −0.30  0.20  0.30  −0.25  −0.22  −0.05  −0.05  −0.18  −0.04  0.28  −0.10  −0.24  1.00  Indicators  SJRm  SNIPm  h  hnf  NDF  NI  HCD  DIC  PA  PR  PAP  PP  TD  PQ1  PDAC  SJRm  1.00                              SNIPm  0.69  1.00                            h  0.16  0.36  1.00                          hnf  −0.06  0.10  0.57  1.00                        NDF  −0.13  −0.08  0.36  0.84  1.00                      NI  0.27  0.32  0.47  −0.08  −0.14  1.00                    HCD  0.19  0.34  0.46  0.04  −0.10  0.55  1.00                  DIC  0.29  0.26  0.07  −0.05  −0.03  0.30  0.15  1.00                PA  0.39  0.45  0.22  0.25  0.06  0.06  0.08  0.11  1.00              PR  0.19  0.17  0.22  −0.07  −0.19  0.22  0.35  −0.02  0.05  1.00            PAP  −0.31  −0.26  −0.11  −0.11  −0.07  −0.11  −0.21  0.05  −0.61  −0.27  1.00          PP  −0.34  −0.51  −0.33  −0.14  0.08  −0.18  −0.22  −0.15  −0.66  −0.29  0.18  1.00        TD  −0.17  0.00  0.65  0.51  0.60  0.31  0.04  0.07  −0.09  −0.16  0.17  0.02  1.00      PQ1  0.63  0.73  0.32  0.04  −0.10  0.32  0.27  0.33  0.40  0.23  −0.22  −0.53  −0.03  1.00    PDAC  −0.16  −0.19  −0.30  0.20  0.30  −0.25  −0.22  −0.05  −0.05  −0.18  −0.04  0.28  −0.10  −0.24  1.00  Note: Values in the bold show significant correlations. Table A2. Correlations among variables for the development grant Indicators  SJRm  SNIPm  h  hnf  NDF  NI  HCD  DIC  PA  PR  PAP  PP  TD  PQ1  PDAC  SJRm  1.00                              SNIPm  0.69  1.00                            h  0.16  0.36  1.00                          hnf  −0.06  0.10  0.57  1.00                        NDF  −0.13  −0.08  0.36  0.84  1.00                      NI  0.27  0.32  0.47  −0.08  −0.14  1.00                    HCD  0.19  0.34  0.46  0.04  −0.10  0.55  1.00                  DIC  0.29  0.26  0.07  −0.05  −0.03  0.30  0.15  1.00                PA  0.39  0.45  0.22  0.25  0.06  0.06  0.08  0.11  1.00              PR  0.19  0.17  0.22  −0.07  −0.19  0.22  0.35  −0.02  0.05  1.00            PAP  −0.31  −0.26  −0.11  −0.11  −0.07  −0.11  −0.21  0.05  −0.61  −0.27  1.00          PP  −0.34  −0.51  −0.33  −0.14  0.08  −0.18  −0.22  −0.15  −0.66  −0.29  0.18  1.00        TD  −0.17  0.00  0.65  0.51  0.60  0.31  0.04  0.07  −0.09  −0.16  0.17  0.02  1.00      PQ1  0.63  0.73  0.32  0.04  −0.10  0.32  0.27  0.33  0.40  0.23  −0.22  −0.53  −0.03  1.00    PDAC  −0.16  −0.19  −0.30  0.20  0.30  −0.25  −0.22  −0.05  −0.05  −0.18  −0.04  0.28  −0.10  −0.24  1.00  Indicators  SJRm  SNIPm  h  hnf  NDF  NI  HCD  DIC  PA  PR  PAP  PP  TD  PQ1  PDAC  SJRm  1.00                              SNIPm  0.69  1.00                            h  0.16  0.36  1.00                          hnf  −0.06  0.10  0.57  1.00                        NDF  −0.13  −0.08  0.36  0.84  1.00                      NI  0.27  0.32  0.47  −0.08  −0.14  1.00                    HCD  0.19  0.34  0.46  0.04  −0.10  0.55  1.00                  DIC  0.29  0.26  0.07  −0.05  −0.03  0.30  0.15  1.00                PA  0.39  0.45  0.22  0.25  0.06  0.06  0.08  0.11  1.00              PR  0.19  0.17  0.22  −0.07  −0.19  0.22  0.35  −0.02  0.05  1.00            PAP  −0.31  −0.26  −0.11  −0.11  −0.07  −0.11  −0.21  0.05  −0.61  −0.27  1.00          PP  −0.34  −0.51  −0.33  −0.14  0.08  −0.18  −0.22  −0.15  −0.66  −0.29  0.18  1.00        TD  −0.17  0.00  0.65  0.51  0.60  0.31  0.04  0.07  −0.09  −0.16  0.17  0.02  1.00      PQ1  0.63  0.73  0.32  0.04  −0.10  0.32  0.27  0.33  0.40  0.23  −0.22  −0.53  −0.03  1.00    PDAC  −0.16  −0.19  −0.30  0.20  0.30  −0.25  −0.22  −0.05  −0.05  −0.18  −0.04  0.28  −0.10  −0.24  1.00  Note: Values in the bold show significant correlations. Table A3. Correlations among variables for the starting grant Indicators  SJRm  SNIPm  h  hnf  NDF  NI  HCD  DIC  PA  PR  PAP  PP  TD  PQ1  PDAC  SJRm  1.00                              SNIPm  0.57  1.00                            h  0.02  0.20  1.00                          hnf  −0.22  0.10  0.79  1.00                        NDF  −0.30  0.09  0.61  0.91  1.00                      NI  0.24  0.11  0.13  −0.01  −0.08  1.00                    HCD  0.19  0.21  0.27  0.11  0.02  0.44  1.00                  DIC  0.09  −0.02  0.10  −0.10  −0.07  −0.10  −0.02  1.00                PA  0.01  −0.03  0.21  0.23  0.15  −0.02  0.08  0.29  1.00              PR  0.44  0.14  0.01  −0.11  −0.23  0.20  0.15  −0.31  −0.28  1.00            PAP  −0.25  0.05  0.21  0.28  0.32  −0.13  −0.14  −0.05  −0.42  −0.19  1.00          PP  −0.27  −0.29  −0.35  −0.27  −0.10  0.00  −0.26  −0.06  −0.43  −0.18  0.03  1.00        TD  −0.22  0.16  0.69  0.73  0.80  −0.09  0.03  −0.02  −0.14  −0.20  0.42  −0.14  1.00      PQ1  0.60  0.63  0.00  −0.13  −0.15  0.17  0.32  −0.08  −0.11  0.36  −0.17  −0.24  −0.10  1.00    PDAC  −0.28  −0.18  −0.23  0.10  0.14  −0.15  −0.19  −0.04  0.04  −0.18  0.06  0.18  −0.14  −0.10  1.00  Indicators  SJRm  SNIPm  h  hnf  NDF  NI  HCD  DIC  PA  PR  PAP  PP  TD  PQ1  PDAC  SJRm  1.00                              SNIPm  0.57  1.00                            h  0.02  0.20  1.00                          hnf  −0.22  0.10  0.79  1.00                        NDF  −0.30  0.09  0.61  0.91  1.00                      NI  0.24  0.11  0.13  −0.01  −0.08  1.00                    HCD  0.19  0.21  0.27  0.11  0.02  0.44  1.00                  DIC  0.09  −0.02  0.10  −0.10  −0.07  −0.10  −0.02  1.00                PA  0.01  −0.03  0.21  0.23  0.15  −0.02  0.08  0.29  1.00              PR  0.44  0.14  0.01  −0.11  −0.23  0.20  0.15  −0.31  −0.28  1.00            PAP  −0.25  0.05  0.21  0.28  0.32  −0.13  −0.14  −0.05  −0.42  −0.19  1.00          PP  −0.27  −0.29  −0.35  −0.27  −0.10  0.00  −0.26  −0.06  −0.43  −0.18  0.03  1.00        TD  −0.22  0.16  0.69  0.73  0.80  −0.09  0.03  −0.02  −0.14  −0.20  0.42  −0.14  1.00      PQ1  0.60  0.63  0.00  −0.13  −0.15  0.17  0.32  −0.08  −0.11  0.36  −0.17  −0.24  −0.10  1.00    PDAC  −0.28  −0.18  −0.23  0.10  0.14  −0.15  −0.19  −0.04  0.04  −0.18  0.06  0.18  −0.14  −0.10  1.00  Note: Values in the bold show significant correlations. Table A3. Correlations among variables for the starting grant Indicators  SJRm  SNIPm  h  hnf  NDF  NI  HCD  DIC  PA  PR  PAP  PP  TD  PQ1  PDAC  SJRm  1.00                              SNIPm  0.57  1.00                            h  0.02  0.20  1.00                          hnf  −0.22  0.10  0.79  1.00                        NDF  −0.30  0.09  0.61  0.91  1.00                      NI  0.24  0.11  0.13  −0.01  −0.08  1.00                    HCD  0.19  0.21  0.27  0.11  0.02  0.44  1.00                  DIC  0.09  −0.02  0.10  −0.10  −0.07  −0.10  −0.02  1.00                PA  0.01  −0.03  0.21  0.23  0.15  −0.02  0.08  0.29  1.00              PR  0.44  0.14  0.01  −0.11  −0.23  0.20  0.15  −0.31  −0.28  1.00            PAP  −0.25  0.05  0.21  0.28  0.32  −0.13  −0.14  −0.05  −0.42  −0.19  1.00          PP  −0.27  −0.29  −0.35  −0.27  −0.10  0.00  −0.26  −0.06  −0.43  −0.18  0.03  1.00        TD  −0.22  0.16  0.69  0.73  0.80  −0.09  0.03  −0.02  −0.14  −0.20  0.42  −0.14  1.00      PQ1  0.60  0.63  0.00  −0.13  −0.15  0.17  0.32  −0.08  −0.11  0.36  −0.17  −0.24  −0.10  1.00    PDAC  −0.28  −0.18  −0.23  0.10  0.14  −0.15  −0.19  −0.04  0.04  −0.18  0.06  0.18  −0.14  −0.10  1.00  Indicators  SJRm  SNIPm  h  hnf  NDF  NI  HCD  DIC  PA  PR  PAP  PP  TD  PQ1  PDAC  SJRm  1.00                              SNIPm  0.57  1.00                            h  0.02  0.20  1.00                          hnf  −0.22  0.10  0.79  1.00                        NDF  −0.30  0.09  0.61  0.91  1.00                      NI  0.24  0.11  0.13  −0.01  −0.08  1.00                    HCD  0.19  0.21  0.27  0.11  0.02  0.44  1.00                  DIC  0.09  −0.02  0.10  −0.10  −0.07  −0.10  −0.02  1.00                PA  0.01  −0.03  0.21  0.23  0.15  −0.02  0.08  0.29  1.00              PR  0.44  0.14  0.01  −0.11  −0.23  0.20  0.15  −0.31  −0.28  1.00            PAP  −0.25  0.05  0.21  0.28  0.32  −0.13  −0.14  −0.05  −0.42  −0.19  1.00          PP  −0.27  −0.29  −0.35  −0.27  −0.10  0.00  −0.26  −0.06  −0.43  −0.18  0.03  1.00        TD  −0.22  0.16  0.69  0.73  0.80  −0.09  0.03  −0.02  −0.14  −0.20  0.42  −0.14  1.00      PQ1  0.60  0.63  0.00  −0.13  −0.15  0.17  0.32  −0.08  −0.11  0.36  −0.17  −0.24  −0.10  1.00    PDAC  −0.28  −0.18  −0.23  0.10  0.14  −0.15  −0.19  −0.04  0.04  −0.18  0.06  0.18  −0.14  −0.10  1.00  Note: Values in the bold show significant correlations. Table A4. CIs for the mean differences (bias-corrected CIs), obtained using bootstrapping, and the effect size (Cohen’s d) in the case of development grants Indicator  Development grant   Engineering and Technology   Natural Sciences   Excat Sciences   Medical and Health Sciences   CI (95%)  Effect size  CI (95%)  Effect size  CI (95%)  Effect size  CI (95%)  Effect size  TD  [−29.45; 2.03]  −0.71  [−4.96; 18.55]  0.43  [−21.56; 20.63]  −0.06  [3.02; 21.39]  1.69  NDF  [−4.24; 6.39]  0.11  [−1.59; 5.62]  0.48  [−7.98, 0.56]  −0.41  [1.75; 5.26]  2.41  PA  [−17.77; 7.68]  −0.26  [−9.24; 10.85]  0.03  [−15.95; 4.76]  −0.35  [−41.50; 0.98]  −1.05  PR  [−2.31; 3.03]  0.10  [−5.92; 0.17]  −0.83  [−2.80; 2.52]  0.00  [−1.27; 35.89]  0.51  PP  [−4.26; 22.39]  0.54  [−9.89; 3.41]  −0.26  [−1.97; 6.17]  0.27  [−3.99; 11.99]  0.86  PAP  [−10.87; 6.43]  −0.23  [−0.71; 9.33]  0.74  [−2.79; 11.58]  0.32  [−20.63; 5.80]  −0.34  PDAC  [−1.33; 30.44]  0.69  [3.07; 35.66]  1.15  [−6.81; 17.35]  0.29  [−16.55; 22.05]  0.14  h  [−9.47; −1.68]  −1.22  [−2.61; 2.95]  0.04  [−5.30; 0.84]  −0.47  [−3.26; 4.91]  0.20  hnf  [−2.45; 0.3]  −0.56  [−0.41; 1.71]  0.51  [−2.67; −0.50]  −0.75  [−0.42; 2.03]  0.72  HCD  [−27.52; −0.41]  −1.09  [−23.61; 8.52]  −0.47  [−14.59; 1.16]  −0.53  [−16.43; 18.15]  0.07  NI  [−0.93; −0.19]  −1.26  [−1.16; −0.13]  −0.96  [−0.40; 0.96]  0.22  [−0.54; 1.18]  0.38  SJRm  [−0.59; −0.03]  −0.78  [−2.11; 0.40]  −0.44  [−0.35; 0.33]  −0.01  [−2.14; −0.41]  −1.07  SNIPm  [−0.35; 0.02]  −0.58  [−0.28; 0.17]  −0.19  [−0.09; 0.15]  0.18  [−0.53; 0.10]  −0.65  PQ1  [−22.00; 8.07]  −0.37  [−19.52; 9.02]  −0.28  [−11.50; 14.89]  0.07  [−41.287; 1.414]  −0.94  DIC  [−34.16; −14.11]  −1.70  [−18.21; 25.36]  0.24  [−8.14; 17.33]  0.20  [−28.69; 4.22]  −0.87  Indicator  Development grant   Engineering and Technology   Natural Sciences   Excat Sciences   Medical and Health Sciences   CI (95%)  Effect size  CI (95%)  Effect size  CI (95%)  Effect size  CI (95%)  Effect size  TD  [−29.45; 2.03]  −0.71  [−4.96; 18.55]  0.43  [−21.56; 20.63]  −0.06  [3.02; 21.39]  1.69  NDF  [−4.24; 6.39]  0.11  [−1.59; 5.62]  0.48  [−7.98, 0.56]  −0.41  [1.75; 5.26]  2.41  PA  [−17.77; 7.68]  −0.26  [−9.24; 10.85]  0.03  [−15.95; 4.76]  −0.35  [−41.50; 0.98]  −1.05  PR  [−2.31; 3.03]  0.10  [−5.92; 0.17]  −0.83  [−2.80; 2.52]  0.00  [−1.27; 35.89]  0.51  PP  [−4.26; 22.39]  0.54  [−9.89; 3.41]  −0.26  [−1.97; 6.17]  0.27  [−3.99; 11.99]  0.86  PAP  [−10.87; 6.43]  −0.23  [−0.71; 9.33]  0.74  [−2.79; 11.58]  0.32  [−20.63; 5.80]  −0.34  PDAC  [−1.33; 30.44]  0.69  [3.07; 35.66]  1.15  [−6.81; 17.35]  0.29  [−16.55; 22.05]  0.14  h  [−9.47; −1.68]  −1.22  [−2.61; 2.95]  0.04  [−5.30; 0.84]  −0.47  [−3.26; 4.91]  0.20  hnf  [−2.45; 0.3]  −0.56  [−0.41; 1.71]  0.51  [−2.67; −0.50]  −0.75  [−0.42; 2.03]  0.72  HCD  [−27.52; −0.41]  −1.09  [−23.61; 8.52]  −0.47  [−14.59; 1.16]  −0.53  [−16.43; 18.15]  0.07  NI  [−0.93; −0.19]  −1.26  [−1.16; −0.13]  −0.96  [−0.40; 0.96]  0.22  [−0.54; 1.18]  0.38  SJRm  [−0.59; −0.03]  −0.78  [−2.11; 0.40]  −0.44  [−0.35; 0.33]  −0.01  [−2.14; −0.41]  −1.07  SNIPm  [−0.35; 0.02]  −0.58  [−0.28; 0.17]  −0.19  [−0.09; 0.15]  0.18  [−0.53; 0.10]  −0.65  PQ1  [−22.00; 8.07]  −0.37  [−19.52; 9.02]  −0.28  [−11.50; 14.89]  0.07  [−41.287; 1.414]  −0.94  DIC  [−34.16; −14.11]  −1.70  [−18.21; 25.36]  0.24  [−8.14; 17.33]  0.20  [−28.69; 4.22]  −0.87  Note: Significance of bold values: p-value<0.05. Table A4. CIs for the mean differences (bias-corrected CIs), obtained using bootstrapping, and the effect size (Cohen’s d) in the case of development grants Indicator  Development grant   Engineering and Technology   Natural Sciences   Excat Sciences   Medical and Health Sciences   CI (95%)  Effect size  CI (95%)  Effect size  CI (95%)  Effect size  CI (95%)  Effect size  TD  [−29.45; 2.03]  −0.71  [−4.96; 18.55]  0.43  [−21.56; 20.63]  −0.06  [3.02; 21.39]  1.69  NDF  [−4.24; 6.39]  0.11  [−1.59; 5.62]  0.48  [−7.98, 0.56]  −0.41  [1.75; 5.26]  2.41  PA  [−17.77; 7.68]  −0.26  [−9.24; 10.85]  0.03  [−15.95; 4.76]  −0.35  [−41.50; 0.98]  −1.05  PR  [−2.31; 3.03]  0.10  [−5.92; 0.17]  −0.83  [−2.80; 2.52]  0.00  [−1.27; 35.89]  0.51  PP  [−4.26; 22.39]  0.54  [−9.89; 3.41]  −0.26  [−1.97; 6.17]  0.27  [−3.99; 11.99]  0.86  PAP  [−10.87; 6.43]  −0.23  [−0.71; 9.33]  0.74  [−2.79; 11.58]  0.32  [−20.63; 5.80]  −0.34  PDAC  [−1.33; 30.44]  0.69  [3.07; 35.66]  1.15  [−6.81; 17.35]  0.29  [−16.55; 22.05]  0.14  h  [−9.47; −1.68]  −1.22  [−2.61; 2.95]  0.04  [−5.30; 0.84]  −0.47  [−3.26; 4.91]  0.20  hnf  [−2.45; 0.3]  −0.56  [−0.41; 1.71]  0.51  [−2.67; −0.50]  −0.75  [−0.42; 2.03]  0.72  HCD  [−27.52; −0.41]  −1.09  [−23.61; 8.52]  −0.47  [−14.59; 1.16]  −0.53  [−16.43; 18.15]  0.07  NI  [−0.93; −0.19]  −1.26  [−1.16; −0.13]  −0.96  [−0.40; 0.96]  0.22  [−0.54; 1.18]  0.38  SJRm  [−0.59; −0.03]  −0.78  [−2.11; 0.40]  −0.44  [−0.35; 0.33]  −0.01  [−2.14; −0.41]  −1.07  SNIPm  [−0.35; 0.02]  −0.58  [−0.28; 0.17]  −0.19  [−0.09; 0.15]  0.18  [−0.53; 0.10]  −0.65  PQ1  [−22.00; 8.07]  −0.37  [−19.52; 9.02]  −0.28  [−11.50; 14.89]  0.07  [−41.287; 1.414]  −0.94  DIC  [−34.16; −14.11]  −1.70  [−18.21; 25.36]  0.24  [−8.14; 17.33]  0.20  [−28.69; 4.22]  −0.87  Indicator  Development grant   Engineering and Technology   Natural Sciences   Excat Sciences   Medical and Health Sciences   CI (95%)  Effect size  CI (95%)  Effect size  CI (95%)  Effect size  CI (95%)  Effect size  TD  [−29.45; 2.03]  −0.71  [−4.96; 18.55]  0.43  [−21.56; 20.63]  −0.06  [3.02; 21.39]  1.69  NDF  [−4.24; 6.39]  0.11  [−1.59; 5.62]  0.48  [−7.98, 0.56]  −0.41  [1.75; 5.26]  2.41  PA  [−17.77; 7.68]  −0.26  [−9.24; 10.85]  0.03  [−15.95; 4.76]  −0.35  [−41.50; 0.98]  −1.05  PR  [−2.31; 3.03]  0.10  [−5.92; 0.17]  −0.83  [−2.80; 2.52]  0.00  [−1.27; 35.89]  0.51  PP  [−4.26; 22.39]  0.54  [−9.89; 3.41]  −0.26  [−1.97; 6.17]  0.27  [−3.99; 11.99]  0.86  PAP  [−10.87; 6.43]  −0.23  [−0.71; 9.33]  0.74  [−2.79; 11.58]  0.32  [−20.63; 5.80]  −0.34  PDAC  [−1.33; 30.44]  0.69  [3.07; 35.66]  1.15  [−6.81; 17.35]  0.29  [−16.55; 22.05]  0.14  h  [−9.47; −1.68]  −1.22  [−2.61; 2.95]  0.04  [−5.30; 0.84]  −0.47  [−3.26; 4.91]  0.20  hnf  [−2.45; 0.3]  −0.56  [−0.41; 1.71]  0.51  [−2.67; −0.50]  −0.75  [−0.42; 2.03]  0.72  HCD  [−27.52; −0.41]  −1.09  [−23.61; 8.52]  −0.47  [−14.59; 1.16]  −0.53  [−16.43; 18.15]  0.07  NI  [−0.93; −0.19]  −1.26  [−1.16; −0.13]  −0.96  [−0.40; 0.96]  0.22  [−0.54; 1.18]  0.38  SJRm  [−0.59; −0.03]  −0.78  [−2.11; 0.40]  −0.44  [−0.35; 0.33]  −0.01  [−2.14; −0.41]  −1.07  SNIPm  [−0.35; 0.02]  −0.58  [−0.28; 0.17]  −0.19  [−0.09; 0.15]  0.18  [−0.53; 0.10]  −0.65  PQ1  [−22.00; 8.07]  −0.37  [−19.52; 9.02]  −0.28  [−11.50; 14.89]  0.07  [−41.287; 1.414]  −0.94  DIC  [−34.16; −14.11]  −1.70  [−18.21; 25.36]  0.24  [−8.14; 17.33]  0.20  [−28.69; 4.22]  −0.87  Note: Significance of bold values: p-value<0.05. Table A5. CIs for the mean differences (bias-corrected CIs), obtained using bootstrapping, and the effect size (Cohen’s d) in the case of starting grants Indicator  Starting grant   Engineering and Technology   Natural Sciences   Excat Sciences   Medical and Health Sciences   CI (95%)  Effect size  CI (95%)  Effect size  CI (95%)  Effect size  CI (95%)  Effect size  TD  [−43.16; −10.61]  −1.60  [−2.41; 23.94]  0.70  [−10.44; 24.03]  0.50  [−19.41; 1.30]  −0.70  NDF  [−6.92; −1.04]  −1.40  [−1.54; 5.47]  0.50  [−0.26; 7.24]  0.14  [−2.21; 0.12]  −0.77  PA  [−23.03; 7.36]  −0.43  [−12.03; 9.20]  −0.04  [5.28; 35.03]  1.30  [−8.61; 27.84]  0.43  PR  [−4.53; 3.85]  −0.10  [−3.63; 8.32]  0.24  [−2.16; 14.15]  0.70  [−13.37; 7.04]  −0.41  PP  [−2.67; 26.51]  0.93  [−4.89; −0.37]  −0.50  [−33.19; 4.16]  −1.00  [−0.84; 3.53]  0.54  PAP  [−15.83; 12.71]  −0.19  [−3.26; 9.24]  0.29  [−15.74; 19.85]  0.20  [−5.91; 2.96]  −0.23  PDAC  [−13.56; 20.02]  0.17  [−26.41; 16.89]  −0.18  [−19.24; 17.13]  −0.10  [−4.33; 24.33]  0.56  h  [−12.03; −1.62]  −1.30  [−0.31; 4.62]  0.70  [0.84; 8.47]  1.20  [−5.61; 1.36]  −0.50  hnf  [−3.66; −0.71]  −1.40  [−0.05; 2.80]  0.90  [0.23; 4.29]  1.20  [−1.37; 0.73]  −0.32  HCD  [−22.87; 7.07]  −0.50  [−8.95; 18.17]  0.24  [−3.73; 22.88]  1.00  [−42.79; 1.37]  −0.79  NI  [−0.65; 0.34]  −0.25  [−1.42; 0.40]  −0.40  [−0.52; 1.56]  0.70  [−0.65; 0.18]  −0.49  SJRm  [−0.39; 0.03]  −0.76  [−0.60; 0.35]  −0.15  [−0.67; 1.20]  0.51  [−1.34; −0.17]  −1.19  SNIPm  [−0.52; 0.11]  −0.59  [−0.14; 0.23]  0.00  [−0.24; 0.38]  0.20  [−0.37; −0.06]  −1.13  PQ1  [−15.87; 19.89]  0.10  [−22.05; 13.93]  −0.19  [−22.00; 32.76]  0.30  [−24.94; 20.80]  −0.05  DIC  [−35.78; 10.95]  −0.62  [−33.60; −4.36]  −0.19  [−36.00; 6.38]  −0.70  [−25.11; 17.27]  −0.19  Indicator  Starting grant   Engineering and Technology   Natural Sciences   Excat Sciences   Medical and Health Sciences   CI (95%)  Effect size  CI (95%)  Effect size  CI (95%)  Effect size  CI (95%)  Effect size  TD  [−43.16; −10.61]  −1.60  [−2.41; 23.94]  0.70  [−10.44; 24.03]  0.50  [−19.41; 1.30]  −0.70  NDF  [−6.92; −1.04]  −1.40  [−1.54; 5.47]  0.50  [−0.26; 7.24]  0.14  [−2.21; 0.12]  −0.77  PA  [−23.03; 7.36]  −0.43  [−12.03; 9.20]  −0.04  [5.28; 35.03]  1.30  [−8.61; 27.84]  0.43  PR  [−4.53; 3.85]  −0.10  [−3.63; 8.32]  0.24  [−2.16; 14.15]  0.70  [−13.37; 7.04]  −0.41  PP  [−2.67; 26.51]  0.93  [−4.89; −0.37]  −0.50  [−33.19; 4.16]  −1.00  [−0.84; 3.53]  0.54  PAP  [−15.83; 12.71]  −0.19  [−3.26; 9.24]  0.29  [−15.74; 19.85]  0.20  [−5.91; 2.96]  −0.23  PDAC  [−13.56; 20.02]  0.17  [−26.41; 16.89]  −0.18  [−19.24; 17.13]  −0.10  [−4.33; 24.33]  0.56  h  [−12.03; −1.62]  −1.30  [−0.31; 4.62]  0.70  [0.84; 8.47]  1.20  [−5.61; 1.36]  −0.50  hnf  [−3.66; −0.71]  −1.40  [−0.05; 2.80]  0.90  [0.23; 4.29]  1.20  [−1.37; 0.73]  −0.32  HCD  [−22.87; 7.07]  −0.50  [−8.95; 18.17]  0.24  [−3.73; 22.88]  1.00  [−42.79; 1.37]  −0.79  NI  [−0.65; 0.34]  −0.25  [−1.42; 0.40]  −0.40  [−0.52; 1.56]  0.70  [−0.65; 0.18]  −0.49  SJRm  [−0.39; 0.03]  −0.76  [−0.60; 0.35]  −0.15  [−0.67; 1.20]  0.51  [−1.34; −0.17]  −1.19  SNIPm  [−0.52; 0.11]  −0.59  [−0.14; 0.23]  0.00  [−0.24; 0.38]  0.20  [−0.37; −0.06]  −1.13  PQ1  [−15.87; 19.89]  0.10  [−22.05; 13.93]  −0.19  [−22.00; 32.76]  0.30  [−24.94; 20.80]  −0.05  DIC  [−35.78; 10.95]  −0.62  [−33.60; −4.36]  −0.19  [−36.00; 6.38]  −0.70  [−25.11; 17.27]  −0.19  Note: Significance of bold values: p-value<0.05. Table A5. CIs for the mean differences (bias-corrected CIs), obtained using bootstrapping, and the effect size (Cohen’s d) in the case of starting grants Indicator  Starting grant   Engineering and Technology   Natural Sciences   Excat Sciences   Medical and Health Sciences   CI (95%)  Effect size  CI (95%)  Effect size  CI (95%)  Effect size  CI (95%)  Effect size  TD  [−43.16; −10.61]  −1.60  [−2.41; 23.94]  0.70  [−10.44; 24.03]  0.50  [−19.41; 1.30]  −0.70  NDF  [−6.92; −1.04]  −1.40  [−1.54; 5.47]  0.50  [−0.26; 7.24]  0.14  [−2.21; 0.12]  −0.77  PA  [−23.03; 7.36]  −0.43  [−12.03; 9.20]  −0.04  [5.28; 35.03]  1.30  [−8.61; 27.84]  0.43  PR  [−4.53; 3.85]  −0.10  [−3.63; 8.32]  0.24  [−2.16; 14.15]  0.70  [−13.37; 7.04]  −0.41  PP  [−2.67; 26.51]  0.93  [−4.89; −0.37]  −0.50  [−33.19; 4.16]  −1.00  [−0.84; 3.53]  0.54  PAP  [−15.83; 12.71]  −0.19  [−3.26; 9.24]  0.29  [−15.74; 19.85]  0.20  [−5.91; 2.96]  −0.23  PDAC  [−13.56; 20.02]  0.17  [−26.41; 16.89]  −0.18  [−19.24; 17.13]  −0.10  [−4.33; 24.33]  0.56  h  [−12.03; −1.62]  −1.30  [−0.31; 4.62]  0.70  [0.84; 8.47]  1.20  [−5.61; 1.36]  −0.50  hnf  [−3.66; −0.71]  −1.40  [−0.05; 2.80]  0.90  [0.23; 4.29]  1.20  [−1.37; 0.73]  −0.32  HCD  [−22.87; 7.07]  −0.50  [−8.95; 18.17]  0.24  [−3.73; 22.88]  1.00  [−42.79; 1.37]  −0.79  NI  [−0.65; 0.34]  −0.25  [−1.42; 0.40]  −0.40  [−0.52; 1.56]  0.70  [−0.65; 0.18]  −0.49  SJRm  [−0.39; 0.03]  −0.76  [−0.60; 0.35]  −0.15  [−0.67; 1.20]  0.51  [−1.34; −0.17]  −1.19  SNIPm  [−0.52; 0.11]  −0.59  [−0.14; 0.23]  0.00  [−0.24; 0.38]  0.20  [−0.37; −0.06]  −1.13  PQ1  [−15.87; 19.89]  0.10  [−22.05; 13.93]  −0.19  [−22.00; 32.76]  0.30  [−24.94; 20.80]  −0.05  DIC  [−35.78; 10.95]  −0.62  [−33.60; −4.36]  −0.19  [−36.00; 6.38]  −0.70  [−25.11; 17.27]  −0.19  Indicator  Starting grant   Engineering and Technology   Natural Sciences   Excat Sciences   Medical and Health Sciences   CI (95%)  Effect size  CI (95%)  Effect size  CI (95%)  Effect size  CI (95%)  Effect size  TD  [−43.16; −10.61]  −1.60  [−2.41; 23.94]  0.70  [−10.44; 24.03]  0.50  [−19.41; 1.30]  −0.70  NDF  [−6.92; −1.04]  −1.40  [−1.54; 5.47]  0.50  [−0.26; 7.24]  0.14  [−2.21; 0.12]  −0.77  PA  [−23.03; 7.36]  −0.43  [−12.03; 9.20]  −0.04  [5.28; 35.03]  1.30  [−8.61; 27.84]  0.43  PR  [−4.53; 3.85]  −0.10  [−3.63; 8.32]  0.24  [−2.16; 14.15]  0.70  [−13.37; 7.04]  −0.41  PP  [−2.67; 26.51]  0.93  [−4.89; −0.37]  −0.50  [−33.19; 4.16]  −1.00  [−0.84; 3.53]  0.54  PAP  [−15.83; 12.71]  −0.19  [−3.26; 9.24]  0.29  [−15.74; 19.85]  0.20  [−5.91; 2.96]  −0.23  PDAC  [−13.56; 20.02]  0.17  [−26.41; 16.89]  −0.18  [−19.24; 17.13]  −0.10  [−4.33; 24.33]  0.56  h  [−12.03; −1.62]  −1.30  [−0.31; 4.62]  0.70  [0.84; 8.47]  1.20  [−5.61; 1.36]  −0.50  hnf  [−3.66; −0.71]  −1.40  [−0.05; 2.80]  0.90  [0.23; 4.29]  1.20  [−1.37; 0.73]  −0.32  HCD  [−22.87; 7.07]  −0.50  [−8.95; 18.17]  0.24  [−3.73; 22.88]  1.00  [−42.79; 1.37]  −0.79  NI  [−0.65; 0.34]  −0.25  [−1.42; 0.40]  −0.40  [−0.52; 1.56]  0.70  [−0.65; 0.18]  −0.49  SJRm  [−0.39; 0.03]  −0.76  [−0.60; 0.35]  −0.15  [−0.67; 1.20]  0.51  [−1.34; −0.17]  −1.19  SNIPm  [−0.52; 0.11]  −0.59  [−0.14; 0.23]  0.00  [−0.24; 0.38]  0.20  [−0.37; −0.06]  −1.13  PQ1  [−15.87; 19.89]  0.10  [−22.05; 13.93]  −0.19  [−22.00; 32.76]  0.30  [−24.94; 20.80]  −0.05  DIC  [−35.78; 10.95]  −0.62  [−33.60; −4.36]  −0.19  [−36.00; 6.38]  −0.70  [−25.11; 17.27]  −0.19  Note: Significance of bold values: p-value<0.05. Table A6. Measures of fit for the obtained models Grant type  Sensitivity (%)  Specificity (%)  False positive  False negative  Prediction  AUC  Development  71.4  65.0  35.0  28.6  68.8  73.6 [61.58; 82.36]  Starting  79.5  55.6  44.4  20.5  69.7  66.2 [50.8; 79.4]  Grant type  Sensitivity (%)  Specificity (%)  False positive  False negative  Prediction  AUC  Development  71.4  65.0  35.0  28.6  68.8  73.6 [61.58; 82.36]  Starting  79.5  55.6  44.4  20.5  69.7  66.2 [50.8; 79.4]  Note: In brackets the CIs (95%) for the AUC. Table A6. Measures of fit for the obtained models Grant type  Sensitivity (%)  Specificity (%)  False positive  False negative  Prediction  AUC  Development  71.4  65.0  35.0  28.6  68.8  73.6 [61.58; 82.36]  Starting  79.5  55.6  44.4  20.5  69.7  66.2 [50.8; 79.4]  Grant type  Sensitivity (%)  Specificity (%)  False positive  False negative  Prediction  AUC  Development  71.4  65.0  35.0  28.6  68.8  73.6 [61.58; 82.36]  Starting  79.5  55.6  44.4  20.5  69.7  66.2 [50.8; 79.4]  Note: In brackets the CIs (95%) for the AUC. Figure A1. View largeDownload slide Curve ROC for the development grant (a) and starting grant (b). Figure A1. View largeDownload slide Curve ROC for the development grant (a) and starting grant (b). © The Author(s) 2018. Published by Oxford University Press. All rights reserved. For permissions, please email: journals.permissions@oup.com This article is published and distributed under the terms of the Oxford University Press, Standard Journals Publication Model (https://academic.oup.com/journals/pages/about_us/legal/notices) http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Research Evaluation Oxford University Press

The peer-review process: The most valued dimensions according to the researcher’s scientific career

Loading next page...
 
/lp/ou_press/the-peer-review-process-the-most-valued-dimensions-according-to-the-ja4r7tGcLs
Publisher
Oxford University Press
Copyright
© The Author(s) 2018. Published by Oxford University Press. All rights reserved. For permissions, please email: journals.permissions@oup.com
ISSN
0958-2029
eISSN
1471-5449
D.O.I.
10.1093/reseval/rvy009
Publisher site
See Article on Publisher Site

Abstract

Abstract Scientific activities are being assessed permanently. The best well-known and well-established evaluation process is peer review. Peer-review-based systems may have different goals; therefore several guidelines are normally set to be followed by individual experts. Normally, the components to be evaluated are known to the whole interested community, but peers make use of their own criteria to evaluate the performance on these components, introducing subjectivity in the whole process. This article reports on an attempt to better understand the decisions of peer-review panels and the role that bibliometric analysis might play in supporting the evaluation of scientific merit in peer-review processes. A particular evaluation process for the national selection of junior and senior researchers is considered. The results show that the dimensions more highly valued by the peers differ depending on the applicant’s phase in the scientific career. For applicants with shorter careers, international collaboration appears to be the dimension more highly valued. In the case of applicants at an intermediate phase of the scientific career, the impact dimension showed to be the most relevant. 1. Introduction The assessment of scientific results of individual researchers, groups, or institutions is part of the everyday life of a research community. It is used to assess if the goals originally set for a given body (e.g. funding agency, research institution, or individual researcher) are being achieved. It is part of the follow-up process to decide what is needed to improve, to adjust, or to strengthen in a research institution or in a national scientific system. The main process used when the goal is to access scientific performance is the peer-review-based systems. It is used in very different contexts: (1) to assess the quality of papers submitted to publication in a given journal or other channel of communication, (2) to take decisions on promotion in the academic context (Vieira, Cabral and Gomes 2014a, 2014b), and (3) in the competitive allocation of funds for research activities at several levels (Abramo and D'Angelo 2015; Bornmann, Wallon and Ledin 2008; Cabezas-Clavijo et al. 2013; Neufeld, Huber and Wegner 2013; Ramos and Sarrico 2016; Rinia et al. 1998; Taylor 2011; van Raan 2006; Vieira and Gomes 2015). When applied at the researcher level, the stage of development of the scientific career of the applicants is normally taken into account (ERC 2015; FCT 2012a, 2014), and therefore, it is understandable that peers might consider different criteria as more relevant. Normally, the parameters to be evaluated are defined by the organization responsible for the assessment exercise together with the guidelines that peers should follow; peers follow these guidelines but make the evaluation taking into account their own understanding of quality. This, inevitably, introduces some subjectivity in the assessment, as the peers may value differently the same individual component. Despite this important limitation of the peer-review-based systems, there is not consensus on a more objective alternative. In the past decades, research has been done on quantitative and objective indicators as instruments to support the peer-review process. Funding agencies and national policy bodies became interested in the use of indicators to avoid/limit the known problems of peer review and to make the process fast and reliable at a reasonable cost. This interest did not translate into a consensus on which indicators should be used. There is several research on the choice of simple indicators and in the construction of composite indicators that perform better in reproducing or predicting peer-review decisions (Bornmann and Leydesdorff 2013; Cabezas-Clavijo et al. 2013; Neufeld et al. 2013; Vieira et al. 2014a, 2014b). The use of these indicators in supporting peer-review panels stills an open question. From the large set of performance indicators that have been developed over time, we give special attention to those based on the number of publications, citations, and on the collaboration practices. These parameters have a set of advantages that make them attractive as an auxiliary tool in assessment exercises: (1) they allow the assessment of a very large number of documents; (2) they provide very simple and objective information about scientific performance; (3) they are relatively inexpensive to collect and simple to implement when compared with peer-review evaluation; (4) they allow the measure of the multidimensional nature of the research activities; and (5) they are well understood by the community, even if they are not above the critical view of many. In this study, using the results of the programme Investigador FCT (a highly competitive scheme designed by the Fundação para a Ciência e Tecnologia in Portugal to provide 5-year support for the most talented and creative researchers), we try to understand if bibliometric indicators are able to predict the final decisions of the peers and the dimensions that peers value the most when considering applicants at different phase of their scientific career. The final goal of the exercise is that of selecting the candidates that are more promising in terms of their expectable future contribution to science. To achieve this, the panels are required to assess the past performance of each candidate and his or her future work plan. We are looking for the best indicators to predict the results of this complex process of peer assessment and decision. This is a very important process, as the development, renewal, and sustainability of a national research system require the definition of a set of strategies that guarantee that highly qualified and internationally competitive researchers are being progressively integrated into the system, and carefully chosen indicators may anticipate their future performance. The use of performance indicators allows an objective assessment, but the choice of these indicators may be said to be subjective or discretionary. This article attempts to propose some justification for the choice of indicators to be used to make the whole process hopefully more objective. 2. Literature review Several studies tried to address the role of bibliometric indicators within a peer-review process. Rinia et al. 1998 studied the correlation between bibliometric indicators and the outcomes of peer judgments made by expert committees of physics in The Netherlands. Using Spearman's rank-correlation coefficients the authors found several significant correlations. The highest values (between 0.5 and 0.7) were found for indicators based on citations (with and without normalization). Norris and Oppenheim 2003, using the results of the peer-review-based process of the 2001 RAE, found a high correlation between the total number of citations and the final decisions of the peers panel in the area of Archaeology (Spearman’s coefficient around 0.8). Similar results were obtained in previous analysis using the results of the 1992 RAE (Oppenheim 1997). Aksnes and Taxt 2004 using the peer ratings of the assessment exercise of 34 research groups at the University of Bergen and a set of five bibliometric indicators (number of papers per scientific personnel; number of citations per person; relative citation rate; relative subfield citedness; relative publication strategy) looked at the correlation between the results. They found weak but significant correlations. The highest Pearson’s correlation was observed between peer ratings and relative publication strategy (an indicator that compares the average citation rate of the journals in which the group’s articles were published with the average citation rates of the subfields covered by each journal), 0.48.van Raan 2006, for 147 university chemistry research groups in The Netherlands, studied the relation between the h index, the CPP/FCSm, and peers’ judgement. The author found that both indicators are able to discriminate very well between the sets of documents that received rating 3 and the sets of documents that received ratings 4 and 5. Jensen, Rouquier, and Croissant 2009 for a set of 400 researchers at CNR explored the correlation between bibliometric indicators and the results of a peer-review process concerning the promotion of researchers. The authors found that the h index, the h index divided by the ‘scientific age’, the number of citations, the number of publications, and the average number of citations per publication allow for a distinction between promoted and not promoted researchers. Using peer decisions at the Valutazione Triennale della Ricerca (VTR), in the hard sciences, and the impact factor of the journals where the documents were published, Abramo, D'Angelo, and Caprasecca 2009 found correlations between the two results that are in the interval 0.336 and 0.876. Franceschet and Costantini 2011 found positive correlations between peers’ decisions at VTR and the mean number of citations per document and the impact factor of the journal. For the number of citations per document the authors found correlation coefficients above 0.7 for Physics and Earth Sciences and between 0.3 and 0.5 for Mathematics and Computer Sciences, Civil Engineering and Architecture, and Economics and Statistics. Using the impact factor the authors found high correlations with peers’ decisions for Chemistry and Biology, while for fields such as Economics and Statistics, Civil Engineering and Architecture, and Industrial and Information Engineering, the correlation coefficient varies between 0.3 and 0.5. The strength of the association seems to be dependent on the scientific field. Butler and McAllister 2011 tried to replicate the peer-review-based outcome of the 2001 RAE in Chemistry and Political Science. They used the mean citation rate, the department size, the research culture, and the presence on the RAE panel of staff from the department under evaluation as predictors of the peeŕs decisions. The impact measured by the mean citation rate is a good predictor of the final decisions of the peer review. Bornmann and Leydesdorff 2013 using a set of 125 papers from F1000 and seven bibliometric indicators from InCites studied the correlation between the scores attributed to each paper by the reviewers (FFa) and the results given by each bibliometric indicator. The authors concluded that the Percentile in Subject Area and Category Actual/Expected Citations, both, should be preferred over other metrics, in research evaluation studies, as they were identified as those most correlated with the ratings. Cabezas-Clavijo et al. 2013 analysed the relationship between peers’ ratings and bibliometric indicators for 2,333 Spanish researchers in the 2007 National R&D Plan. The authors found that the number of published articles and the papers published in journals that belong to the first quartile ranking of the Journal Citations Report are the main indicators in predicting the peers’ rating. Neufeld et al. 2013 analysed the dependence of funding decisions on past publication performance among applicants of the Starting Grants Programme, offered by the European Research Council. Applicants from the Life Sciences and Physical Sciences and Engineering were considered in the study. The authors found that the number of publications is not able to distinguish between funded and not funded applicants, the mean Journal Impact Factor is able to distinguish between the two groups for some of the sub-areas of Life Sciences and Physical Sciences and Engineering, and the field normalized citation rate and the number of highly cited papers (top 10%) are able to distinguish between funded and not funded applicants in the Physical Sciences and Engineering. Vieira et al. 2014a using academic contests, which took place in several Portuguese universities, found that two composite indicators (combination of two/three bibliometric indicators) are able to predict the peers’ final decision. For 75% of the orderings of the contests, the composite indicators were able to predict the final decision of the peers. Aimed at determining the role of quantitative indicators (indicators based on citation and altmetrics) in research assessment and management, Wilsdon et al. 2015 carried out a large study using the data submitted to the REF2014. Fifteen indicators were determined and the results compared with the score obtained by each author-publication in the evaluation made by the peers. Consistently, high correlations were found for several metrics in Clinical Medicine, Economics, and Econometrics. The authors also found that the quantitative indicators with significant impact in predicting peers’ evaluation are different among scientific field. Bedsides quantitative analysis the authors tried to understand the opinion of several experts who participated in REF in relation to the use of quantitative indicators; the experts suggested that peer review should remain central to the process, but that there is some support for the additional use of quantitative data. The studies described above lead us to summarize the main findings in the following two points: Correlations between the results given by quantitative indicators and the results from peer evaluation were observed at several levels (evaluation of research programmes, university’s departments, individual researchers, and individual papers). However, the strength of the correlations varies depending on the scientific field. Several quantitative indicators are able to reproduce in part the peers’ decisions, h index, and normalized indices (CPP/FCSm, and/or highly cited papers). Bedsides the large numbers of studies that have been done none of the studies, as far as we know, address the fact that the bibliometric indicators able to describe the final decisions of the peers might be different depending on the phase of the scientific career of the researchers being evaluated. Taking into account the findings of the studies described above, our study tries to answer to the following research question: Q1: Do the dimensions of scientific performance, represented by bibliometric indicators, more valued by the peers differ between starting and development grants? There are several concerns about the use of bibliometric indicators to describe the scientific performance of young researchers; thus it is important to explore the subject. On the other hand, if bibliometric indicators are able to describe peers’ decisions, it is important to study if the set of bibliometric indicators differs according to the type of grant proposal. 3. Methods and data 3.1 Programme investigador FCT Investigador FCT is a programme carried out since 2012 by the FCT, in Portugal, with the aim of giving financial support for research activities at the individual level. Through this programme, FCT intends to select a set of highly motivated researchers that are internationally competitive and hope to establish themselves as independent researchers, and already independent researchers who wish to consolidate research skills and establish leadership in their research field, in Portugal. The call comprises three levels of grants, and taking into account the number of years after the award of the PhD degree and the number of years working as an independent researcher, a given researcher can apply to one of the following grants FCT 2012b: Starting grants: For researchers with a maximum of 5 years after the award of the PhD and being an independent researcher is not required. Development grants: For researchers with the PhD degree awarded for more than 6 years and less than 12 years and being independent researchers for less than 6 years. Advanced grants: For researchers with more than 6 years as independent. In 2012, the opening took place in two phases. At the first phase the applicants were asked to submit CV synopsis and the research project and career development synopsis for evaluation by the peers. In addition the applicants were also asked to identify their scientific area from the Organisation for Economic Co-operation and Development (OECD)´s adopted Field of Science and Technology (FOS) classification, the main and secondary scientific area (FCT 2012a). At this phase only 25% of the total applications were selected to forward to the second phase. Each proposal was evaluated by two reviewers belonging to the four panels responsible for the preliminary reviewing of all applications (first phase); CV (60%) research project (20%), and career development plan (20%) (FCT 2012b). In terms of CV, peers were asked to use the following evaluation criteria (for all types of grants) for the scientific merit (FCT 2012b): scientific productivity; abilities and skills to execute the proposed project; degree of internationalization; and degree of success in previous calls for grant application and doctoral and post-doctoral training. For more information related to the evaluation criteria for the remaining components see (FCT 2012b). The applicants selected at the first level were asked to submit a full application (i.e. the extended versions of the CV, research project, and career development plan) for evaluation by two external mail referees. The external mail referees produced evaluation reports submitted to the evaluation panel members responsible for the final decision. The evaluation panel comprised 10 members (one chair and three members for each scientific domain; Life Sciences, Physical Sciences and Engineering, and Social Sciences and the Humanities) who had access to all applications (FCT 2012b). Total 54% of the total applicants who passed to the second phase were selected for funding, and they represent about 13% of the total applicants (first and second phases). Table 1 showed the distribution of the applicants of the programme launched in 2012, by type of grant and field for each phase of the peer-review process. Table 1. Distribution of applicants by type of grant and scientific domain Type of grant  Area  Number of applicants Phase 1  Number of applicants Phase 2  Applicants selected for funding  Starting  Agricultural Sciences  37  3  1  Engineering and Technology  177  20  10  Exact Sciences  122  12  6  Humanities  73  15  8  Medical and Health Sciences  83  20  12  Natural Sciences  147  30  21  Social Sciences  87  16  9  Development  Agricultural Sciences  26  7  2  Engineering and Technology  76  28  12  Exact Sciences  92  48  28  Humanities  20  8  5  Medical and Health Sciences  33  21  14  Natural Sciences  76  23  15  Social Sciences  29  11  2  Advanced  Agricultural Sciences  1  –  –  Engineering and Technology  17  4  2  Exact Sciences  31  6  5  Humanities  11  4  –  Medical and Health Sciences  7  4  2  Natural Sciences  15  7  2  Social Sciences  15  5  2  Total    1,175  292  158  Type of grant  Area  Number of applicants Phase 1  Number of applicants Phase 2  Applicants selected for funding  Starting  Agricultural Sciences  37  3  1  Engineering and Technology  177  20  10  Exact Sciences  122  12  6  Humanities  73  15  8  Medical and Health Sciences  83  20  12  Natural Sciences  147  30  21  Social Sciences  87  16  9  Development  Agricultural Sciences  26  7  2  Engineering and Technology  76  28  12  Exact Sciences  92  48  28  Humanities  20  8  5  Medical and Health Sciences  33  21  14  Natural Sciences  76  23  15  Social Sciences  29  11  2  Advanced  Agricultural Sciences  1  –  –  Engineering and Technology  17  4  2  Exact Sciences  31  6  5  Humanities  11  4  –  Medical and Health Sciences  7  4  2  Natural Sciences  15  7  2  Social Sciences  15  5  2  Total    1,175  292  158  Table 1. Distribution of applicants by type of grant and scientific domain Type of grant  Area  Number of applicants Phase 1  Number of applicants Phase 2  Applicants selected for funding  Starting  Agricultural Sciences  37  3  1  Engineering and Technology  177  20  10  Exact Sciences  122  12  6  Humanities  73  15  8  Medical and Health Sciences  83  20  12  Natural Sciences  147  30  21  Social Sciences  87  16  9  Development  Agricultural Sciences  26  7  2  Engineering and Technology  76  28  12  Exact Sciences  92  48  28  Humanities  20  8  5  Medical and Health Sciences  33  21  14  Natural Sciences  76  23  15  Social Sciences  29  11  2  Advanced  Agricultural Sciences  1  –  –  Engineering and Technology  17  4  2  Exact Sciences  31  6  5  Humanities  11  4  –  Medical and Health Sciences  7  4  2  Natural Sciences  15  7  2  Social Sciences  15  5  2  Total    1,175  292  158  Type of grant  Area  Number of applicants Phase 1  Number of applicants Phase 2  Applicants selected for funding  Starting  Agricultural Sciences  37  3  1  Engineering and Technology  177  20  10  Exact Sciences  122  12  6  Humanities  73  15  8  Medical and Health Sciences  83  20  12  Natural Sciences  147  30  21  Social Sciences  87  16  9  Development  Agricultural Sciences  26  7  2  Engineering and Technology  76  28  12  Exact Sciences  92  48  28  Humanities  20  8  5  Medical and Health Sciences  33  21  14  Natural Sciences  76  23  15  Social Sciences  29  11  2  Advanced  Agricultural Sciences  1  –  –  Engineering and Technology  17  4  2  Exact Sciences  31  6  5  Humanities  11  4  –  Medical and Health Sciences  7  4  2  Natural Sciences  15  7  2  Social Sciences  15  5  2  Total    1,175  292  158  3.2 Data set A total of 7,559 documents were retrieved from the Web of Science Core Collection (Science Citation Index Expanded; Social Sciences Citation Index; Arts and Humanities Citation Index; Conference Proceedings Citation Index—Science; Conference Proceedings Citation Index—Social Science and Humanities). These documents are those mentioned in the grant proposal by each applicant and were made available by the FCT. We retrieved the information about the publications for each applicant from the Web of Science Core Collection using expressions that combine the different homonymous used by each applicant and the research institutions where they have been working at. The documents have been published by the 167 applicants that were selected to proceed to the second phase of the peer-review process: 101 of 167 are applicants to the development grants (Engineering and Technology (26 applicants), Natural Sciences (21), Exact Sciences (40), and Medial and Health Sciences (14)). In the case of starting grants the sample comprises 17 applicants from Engineering and Technology, 25 applicants from Natural Sciences, 9 applicants from Exact Sciences, and 15 applicants from Medial and Health Sciences. Not all the applicants selected to go to the second phase of the evaluation process were considered in our study due to the aspects presented below. The call was also available to those researchers from the Social Sciences and Humanities, but we did not consider these applicants as we are using a database that has several limitations in the coverage of publications in these fields. The proposals within Agricultural Sciences (all grants) and the proposals to advanced grants were also excluded, as they were present in a small number (see Table 1). We should use a considerable number of publications in calculating the bibliometric indicators. We use a sample composed of applicants with at least 10 documents published and indexed in the Web of Science at the moment of the call. The threshold used is lower than that suggested in the literature (Lehmann, Jackson and Lautrup 2008), but we want to avoid the exclusion of a large number of applicants, mainly in the case of the starting grants where, due to the very initial phase of the scientific career, applicants are expected to have a low number of publications. 3.3 Bibliometric indicators The evaluation guidelines to be used by the peers in the process were made available by the FCT, but these criteria are very general and most of them do not allow associating a bibliometric indicator easily. As an example we have (FCT 2012): ‘Scientific productivity of the applicant evaluated according to criteria accepted internationally by the different scientific communities’. ‘Indicators for scientific merit of the applicant include the main academic and professional degrees, publications in top specialty peer-reviewed journals and/or in major multidisciplinary international peer-reviewed journals’. Taking this into account, we selected a set of bibliometric indicators that considers the multidimensional nature of scientific achievement. Many indicators have been developed over the years, and we were forced to limit our choice for this study. We selected 15 indicators, which explore different dimensions: production, impact, collaboration, and scientific independence. Most of the indicators or their variants were already used in a previous study and showed to be important in representing the final decision of the peer-review process. Other indicators may be understood as a proxy for more objective guidelines of the peer-review process. The definition of each of the indicators is presented. The h index, hnf, HCD, NI, SNIPm, SJRm, PQ1, and DIC were determined using the documents classified by the database as articles, reviews, and proceedings paper. TD: The total number of documents indexed in the Web of Science Core Collection for each applicant. NDF: The total number of documents published by the applicants after fractionation. Each document was divided by the total number of authors (N) of the list and associated with (1/N). We considered all the types of documents indexed (Vieira et al. 2014a). PA: The percentage of the total number of documents of the applicant that are classified as articles in the Web of Science Core Collection. PR: The percentage of the total number of documents of the applicant that are classified as reviews in the Web of Science Core Collection. PP: The percentage of the total number of documents of the applicant that are classified as proceedings paper in the Web of Science Core Collection. PAP: The percentage of the total number of documents of the applicant that are classified simultaneously as proceedings paper and articles in the Web of Science Core Collection. PDAC: The percentage of documents where the applicant is the corresponding author. h index: As defined by J. E. Hirsch ‘A scientist has index h if h of his or her Np papers have at least h citations each and the other (Np - h) papers have ≤h citations each’ (Hirsch 2005). hnf index: The indicator is calculated in a way similar to the h index, but considers the different citation cultures among fields and the number of authors per publication (Vieira and Gomes 2011). HCD: Highly cited documents. This indicator gives the percentage of documents of a given researcher that are in the top 10% most cited EU_27 documents. The category, type of document, and publication year are taken into account and a 5-year citation window used in collecting the number of citations. NI: Normalized impact. The indicator compares the average number of citations per publication of the applicant with the EU_27 average. A value of 1 indicates that the documents of the applicant behave as the average and a value of 1.20 means that the average value of citations per document of the applicant is 20% above of the average of the EU_27. The category, type of document, and publication year are taken into account in the normalization process. We used a 5-year citation window in collecting the number of citations. SNIPm: For each document, indexed in the Web of Science Core Collection, we collected the SNIP (Moed 2010) of the journal. SNIPmrepresents the median value of the distribution of the journals where the documents have been published in. The SNIP takes into account the scientific domain where the journals operate, smoothing differences between field-specific properties such as the number of citations per paper and the speed of the publication process. SJRm: For each document, indexed in the Web of Science Core Collection, we collected the SJR (Gonzalez-Pereira, Guerrero-Bote and Moya-Anegon 2010) of the journal. SJRmrepresents the median value of the distribution of the journals where the documents have been published in. The SJR takes into account the prestige of the citing journal; the citations received by the journals are weighted according to the SJR of the citing journal. The SNIP and SJR were retrieved from Scopus for the year of publication of each document. PQ1: Percentage of documents that were published in journals that belong to the first quartile in the corresponding Web of Science category and year of publication, according to the impact factor for 2 years, in the Journal Citation Reports. When the journal belongs to more than one category, and it is only in the first quartile in one category, it is considered in the set of documents in the first quartile. DIC: The percentage of the total documents with at least one international collaboration. The indicators PA, PP, PR, and PAP were selected due to the fact that we are considering applicants from different scientific domains and therefore with very different culture of publications. As example we have the case of Engineering and Technology and Computer Science, where the acceptance levels in conferences are rather low (Meyer et al, 2009) showing the importance of the proceedings paper. The PDAC is a proxy for ‘scientific independence’. We selected the PDAC, as in the evaluation guide one of the parameters that peers are asked to evaluate is scientific independence. FCT suggests as criteria for assessing scientific independence the number of publications where the applicant is the corresponding author (FCT 2012a). Other suggested criteria are as follows: ‘– being a PI or group leader of a research team, – having obtained funding as principal investigator in competitive calls launched by national and/or international funding agencies’. We were not able to use these criteria in our study, as the information submitted by the applicants was not made available. In the calculation of the hnf, HCD, and NI, in the normalization of citations we used the Web of Science categories defined at the journals level. We adopted a cited-side normalization procedure and a full counting method. Indicators that use a different normalization process, for example the citing-side, could be used. There is no indicator which is entirely without drawbacks, as every standardized indicator has its advantages and limitations. In the literature there are studies showing that the method of normalization has only a slight influence on the validity of the indicators (Bornmann and Marx 2015). The unavailability of the data necessary to calculate the indicators with citing-side normalization did not allow using such type of indicators in our study. However, they should be explored in future studies. The SNIP and SJR are based on the information from Scopus database, and both normalized indicators are used frequently for comparing journal impact. While being aware of the fact that we are using two indicators based on the information from another database, we consider that this does not invalidate our results. The Web of Science journal impact factor (JIF) has been shown to correlate well with SNIP and SJR indicators for several scientific domains (Gonzalez-Pereira et al. 2010; Torres-Salinas and Jimenez-Contreras 2010); the lowest correlations were observed for the Social Sciences and Humanities and these domains are not considered in the study. The DIC is a proxy for the ‘Degree of internationalization’ (FCT 2012b). 3.4 Statistical analysis First, we used bootstrapping techniques (with replacement) and the t-test to identify those dimensions for which the scientific performance of the set of selected and not selected applicants is statistical significant different. The results provide preliminary evidence on the indicators that might be important in predicting peers’ decisions. To understand the substantive significance of the findings, we made use of the Cohen’s d statistic. The use of bootstrapping techniques allows us to deal with some features, less suitable for statistical analysis, of the original data, as small sample size and absence of normality of the data. To investigate the distribution of the indicators and find preliminary evidence on their correlations, we also did a descriptive analysis (see Appendix). Secondly, using logistic regression, we identify those bibliometric indicators that contain information about the dimensions more valued in the evaluation exercise for different phases of the researcher’s scientific career. We determine the bibliometric indicators with significant impact implicit in the peer judgments and the size of their effect (Hosmer, Lemeshow and Sturdivant 2013). Logistic regression is used to estimate the probability of observing a given outcome (normally a binary, variable) based on one or more independent variables. Such probability is given by the following expression:   Pi=exp⁡(β0+β×Xi)1+exp⁡(β0+β×Xi), (1) where β0+β×Xi is the utility function and represents a linear function of the explanatory indicators (Xi) of applicant i; the β is the vector of coefficients and, β0 the constant. As we are dealing with small sample size, we used cross-validation for estimating the performance of the predictive model. A set of fit measures (sensitivity, specificity, false positive, false negative, prediction, and area under the ROC (AUC)) is also determined to study the robustness of the main findings. Thirdly, we discuss in a detailed way the results obtained and compare them with the findings of previous studies. All the statistical analyses were carried out using MATLAB and Stata software. 4. Results 4.1 Differences between selected and not selected applicants To have some preliminary understanding of the selection process, we identified the bibliometric indicators able to distinguish, in a significant way, between the two sets of applicants, those selected and not selected. In Tables 2 and 3, we present the average value for each indicator and highlight those cases where the results from the t-test were statistically significant (P-value < 0.05). Table 2. Average values for the two groups of applicants (not selected and selected) for the starting grants Indicator  Engineering and Technology   Natural Sciences   Exact Sciences   Medical and Health Sciences   Selected  Not selected  Selected  Not selected  Selected  Not selected  Selected  Not selected  TD  53.22  25.38  20.24  29.88  22.75  29.80  27.33  19.17  NDF  9.95  5.84  4.68  6.66  3.26  6.75  3.69  2.69  PA  71.60  64.14  86.36  85.90  59.91  79.16  65.56  74.48  PR  3.46  2.95  3.79  5.48  1.71  6.64  12.51  8.51  PP  5.23  17.62  2.08  0.00  17.66  3.19  0.46  1.57  PAP  16.03  13.12  3.55  5.66  8.21  11.00  3.10  2.04  PDAC  18.52  21.46  31.53  26.88  20.45  19.00  6.45  16.30  h  14.56  7.75  8.88  11.00  7.00  11.40  10.00  8.17  hnf  4.94  2.76  2.68  4.10  1.79  3.99  2.36  2.03  HCD  14.24  6.84  18.78  23.07  3.64  13.54  29.80  10.90  NI  1.16  1.04  1.56  1.15  0.85  1.34  1.02  0.81  SJRm  1.02  0.85  1.34  1.20  0.87  1.18  1.91  1.17  SNIPm  1.30  1.11  1.19  1.24  1.04  1.10  1.30  1.09  PQ1  45.30  47.06  50.68  46.38  47.72  53.34  65.24  64.01  DIC  37.86  23.27  45.17  25.39  53.29  38.25  36.69  32.73  Indicator  Engineering and Technology   Natural Sciences   Exact Sciences   Medical and Health Sciences   Selected  Not selected  Selected  Not selected  Selected  Not selected  Selected  Not selected  TD  53.22  25.38  20.24  29.88  22.75  29.80  27.33  19.17  NDF  9.95  5.84  4.68  6.66  3.26  6.75  3.69  2.69  PA  71.60  64.14  86.36  85.90  59.91  79.16  65.56  74.48  PR  3.46  2.95  3.79  5.48  1.71  6.64  12.51  8.51  PP  5.23  17.62  2.08  0.00  17.66  3.19  0.46  1.57  PAP  16.03  13.12  3.55  5.66  8.21  11.00  3.10  2.04  PDAC  18.52  21.46  31.53  26.88  20.45  19.00  6.45  16.30  h  14.56  7.75  8.88  11.00  7.00  11.40  10.00  8.17  hnf  4.94  2.76  2.68  4.10  1.79  3.99  2.36  2.03  HCD  14.24  6.84  18.78  23.07  3.64  13.54  29.80  10.90  NI  1.16  1.04  1.56  1.15  0.85  1.34  1.02  0.81  SJRm  1.02  0.85  1.34  1.20  0.87  1.18  1.91  1.17  SNIPm  1.30  1.11  1.19  1.24  1.04  1.10  1.30  1.09  PQ1  45.30  47.06  50.68  46.38  47.72  53.34  65.24  64.01  DIC  37.86  23.27  45.17  25.39  53.29  38.25  36.69  32.73  Note: The values in bold represent the indicators where we observed statistical significant differences; P-value < 0.05. Table 2. Average values for the two groups of applicants (not selected and selected) for the starting grants Indicator  Engineering and Technology   Natural Sciences   Exact Sciences   Medical and Health Sciences   Selected  Not selected  Selected  Not selected  Selected  Not selected  Selected  Not selected  TD  53.22  25.38  20.24  29.88  22.75  29.80  27.33  19.17  NDF  9.95  5.84  4.68  6.66  3.26  6.75  3.69  2.69  PA  71.60  64.14  86.36  85.90  59.91  79.16  65.56  74.48  PR  3.46  2.95  3.79  5.48  1.71  6.64  12.51  8.51  PP  5.23  17.62  2.08  0.00  17.66  3.19  0.46  1.57  PAP  16.03  13.12  3.55  5.66  8.21  11.00  3.10  2.04  PDAC  18.52  21.46  31.53  26.88  20.45  19.00  6.45  16.30  h  14.56  7.75  8.88  11.00  7.00  11.40  10.00  8.17  hnf  4.94  2.76  2.68  4.10  1.79  3.99  2.36  2.03  HCD  14.24  6.84  18.78  23.07  3.64  13.54  29.80  10.90  NI  1.16  1.04  1.56  1.15  0.85  1.34  1.02  0.81  SJRm  1.02  0.85  1.34  1.20  0.87  1.18  1.91  1.17  SNIPm  1.30  1.11  1.19  1.24  1.04  1.10  1.30  1.09  PQ1  45.30  47.06  50.68  46.38  47.72  53.34  65.24  64.01  DIC  37.86  23.27  45.17  25.39  53.29  38.25  36.69  32.73  Indicator  Engineering and Technology   Natural Sciences   Exact Sciences   Medical and Health Sciences   Selected  Not selected  Selected  Not selected  Selected  Not selected  Selected  Not selected  TD  53.22  25.38  20.24  29.88  22.75  29.80  27.33  19.17  NDF  9.95  5.84  4.68  6.66  3.26  6.75  3.69  2.69  PA  71.60  64.14  86.36  85.90  59.91  79.16  65.56  74.48  PR  3.46  2.95  3.79  5.48  1.71  6.64  12.51  8.51  PP  5.23  17.62  2.08  0.00  17.66  3.19  0.46  1.57  PAP  16.03  13.12  3.55  5.66  8.21  11.00  3.10  2.04  PDAC  18.52  21.46  31.53  26.88  20.45  19.00  6.45  16.30  h  14.56  7.75  8.88  11.00  7.00  11.40  10.00  8.17  hnf  4.94  2.76  2.68  4.10  1.79  3.99  2.36  2.03  HCD  14.24  6.84  18.78  23.07  3.64  13.54  29.80  10.90  NI  1.16  1.04  1.56  1.15  0.85  1.34  1.02  0.81  SJRm  1.02  0.85  1.34  1.20  0.87  1.18  1.91  1.17  SNIPm  1.30  1.11  1.19  1.24  1.04  1.10  1.30  1.09  PQ1  45.30  47.06  50.68  46.38  47.72  53.34  65.24  64.01  DIC  37.86  23.27  45.17  25.39  53.29  38.25  36.69  32.73  Note: The values in bold represent the indicators where we observed statistical significant differences; P-value < 0.05. Table 3. Average values for the two groups of applicants (not selected and selected) for the development grants Indicator  Engineering and Technology   Natural Sciences   Exact Sciences   Medical and Health Sciences   Selected  Not selected  Selected  Not selected  Selected  Not selected  Selected  Not selected  TD  48.18  34.40  35.93  42.57  49.17  47.06  24.67  37.60  NDF  11.02  12.02  8.25  10.51  12.79  9.31  4.61  8.15  PA  71.94  67.22  81.13  81.65  81.88  76.40  77.19  55.66  PR  2.41  2.77  5.41  2.36  2.71  2.70  4.15  7.50  PP  8.57  17.37  4.92  2.38  4.40  6.34  0.91  12.02  PAP  13.12  10.69  2.87  7.21  9.42  13.56  9.41  4.67  PDAC  20.11  33.88  22.47  41.86  29.11  34.70  17.91  19.75  H  15.73  10.13  13.43  13.57  15.13  12.76  11.89  12.60  hnf  5.31  4.19  4.14  4.85  5.59  4.08  2.85  3.61  HCD  25.39  10.04  17.93  9.69  15.08  8.61  19.63  20.74  NI  1.42  0.86  1.59  0.95  1.25  1.49  1.17  1.46  SJRm  1.24  0.93  2.08  1.33  1.32  1.32  2.09  0.87  SNIPm  1.25  1.09  1.26  1.20  1.13  1.16  1.21  1.01  PQ1  49.05  42.08  66.61  61.10  56.31  57.70  68.70  49.21  DIC  48.77  24.72  46.78  52.02  50.94  55.19  44.95  32.55  Indicator  Engineering and Technology   Natural Sciences   Exact Sciences   Medical and Health Sciences   Selected  Not selected  Selected  Not selected  Selected  Not selected  Selected  Not selected  TD  48.18  34.40  35.93  42.57  49.17  47.06  24.67  37.60  NDF  11.02  12.02  8.25  10.51  12.79  9.31  4.61  8.15  PA  71.94  67.22  81.13  81.65  81.88  76.40  77.19  55.66  PR  2.41  2.77  5.41  2.36  2.71  2.70  4.15  7.50  PP  8.57  17.37  4.92  2.38  4.40  6.34  0.91  12.02  PAP  13.12  10.69  2.87  7.21  9.42  13.56  9.41  4.67  PDAC  20.11  33.88  22.47  41.86  29.11  34.70  17.91  19.75  H  15.73  10.13  13.43  13.57  15.13  12.76  11.89  12.60  hnf  5.31  4.19  4.14  4.85  5.59  4.08  2.85  3.61  HCD  25.39  10.04  17.93  9.69  15.08  8.61  19.63  20.74  NI  1.42  0.86  1.59  0.95  1.25  1.49  1.17  1.46  SJRm  1.24  0.93  2.08  1.33  1.32  1.32  2.09  0.87  SNIPm  1.25  1.09  1.26  1.20  1.13  1.16  1.21  1.01  PQ1  49.05  42.08  66.61  61.10  56.31  57.70  68.70  49.21  DIC  48.77  24.72  46.78  52.02  50.94  55.19  44.95  32.55  Note: The values in bold represent the indicators where we observed statistical significant differences; P-value < 0.05. Table 3. Average values for the two groups of applicants (not selected and selected) for the development grants Indicator  Engineering and Technology   Natural Sciences   Exact Sciences   Medical and Health Sciences   Selected  Not selected  Selected  Not selected  Selected  Not selected  Selected  Not selected  TD  48.18  34.40  35.93  42.57  49.17  47.06  24.67  37.60  NDF  11.02  12.02  8.25  10.51  12.79  9.31  4.61  8.15  PA  71.94  67.22  81.13  81.65  81.88  76.40  77.19  55.66  PR  2.41  2.77  5.41  2.36  2.71  2.70  4.15  7.50  PP  8.57  17.37  4.92  2.38  4.40  6.34  0.91  12.02  PAP  13.12  10.69  2.87  7.21  9.42  13.56  9.41  4.67  PDAC  20.11  33.88  22.47  41.86  29.11  34.70  17.91  19.75  H  15.73  10.13  13.43  13.57  15.13  12.76  11.89  12.60  hnf  5.31  4.19  4.14  4.85  5.59  4.08  2.85  3.61  HCD  25.39  10.04  17.93  9.69  15.08  8.61  19.63  20.74  NI  1.42  0.86  1.59  0.95  1.25  1.49  1.17  1.46  SJRm  1.24  0.93  2.08  1.33  1.32  1.32  2.09  0.87  SNIPm  1.25  1.09  1.26  1.20  1.13  1.16  1.21  1.01  PQ1  49.05  42.08  66.61  61.10  56.31  57.70  68.70  49.21  DIC  48.77  24.72  46.78  52.02  50.94  55.19  44.95  32.55  Indicator  Engineering and Technology   Natural Sciences   Exact Sciences   Medical and Health Sciences   Selected  Not selected  Selected  Not selected  Selected  Not selected  Selected  Not selected  TD  48.18  34.40  35.93  42.57  49.17  47.06  24.67  37.60  NDF  11.02  12.02  8.25  10.51  12.79  9.31  4.61  8.15  PA  71.94  67.22  81.13  81.65  81.88  76.40  77.19  55.66  PR  2.41  2.77  5.41  2.36  2.71  2.70  4.15  7.50  PP  8.57  17.37  4.92  2.38  4.40  6.34  0.91  12.02  PAP  13.12  10.69  2.87  7.21  9.42  13.56  9.41  4.67  PDAC  20.11  33.88  22.47  41.86  29.11  34.70  17.91  19.75  H  15.73  10.13  13.43  13.57  15.13  12.76  11.89  12.60  hnf  5.31  4.19  4.14  4.85  5.59  4.08  2.85  3.61  HCD  25.39  10.04  17.93  9.69  15.08  8.61  19.63  20.74  NI  1.42  0.86  1.59  0.95  1.25  1.49  1.17  1.46  SJRm  1.24  0.93  2.08  1.33  1.32  1.32  2.09  0.87  SNIPm  1.25  1.09  1.26  1.20  1.13  1.16  1.21  1.01  PQ1  49.05  42.08  66.61  61.10  56.31  57.70  68.70  49.21  DIC  48.77  24.72  46.78  52.02  50.94  55.19  44.95  32.55  Note: The values in bold represent the indicators where we observed statistical significant differences; P-value < 0.05. The indicators for which we found differences are not the same among scientific fields. In Engineering and Technology indicators related to quantity (TD and NDF) and collaboration (NDF) are those able to distinguish between the applicants not selected and selected by the peers (statistical significance is observed). In addition to these indicators we also have indicators related to impact (the h and hnf index). However, between the two indicators we would say that the hnf is the most appropriate for comparing the applicants, as it is a normalized indicator. In Natural Sciences there is statistical significant evidence to reject the null hypotheses in the case of the variables DIC and hnf; the two groups of applicants are different in what concerns to international collaboration; in average, the not selected applicants have lower practices of international collaboration than selected applicants. The hnf is able to distinguish between selected and not selected applicants (there is statistical evidence to say that the means between the two groups are not equal). However, the mean value is greater in the case of not selected applicants, which is against our expectations. The result is unclear for us (we do not identified outliers within the set of not selected applicants). In the case of Exact Sciences, statistical significant differences are not observed between the two groups of applicants. Nevertheless, the Cohen’s d indicates that there are indicators for which the differences between the means are large; the PA, h, hnf, and HCD (see Table A5). Table 5. Results for the logistic regression Grant type  Coefficients, standardized   SJRm (%)  HCD (%)  DIC (%)  Intercept (%)  Development  1.130* (18.1)  0.952* (9.2)  −  −1.891* (12.6)  Starting  –  –  0.692* (18.2)  −1.328 (12.1)  Grant type  Coefficients, standardized   SJRm (%)  HCD (%)  DIC (%)  Intercept (%)  Development  1.130* (18.1)  0.952* (9.2)  −  −1.891* (12.6)  Starting  –  –  0.692* (18.2)  −1.328 (12.1)  Note: * P-value < 0.05. The coefficient value is the mean value; cross-validation was used in adjusting the model. Between brackets is the coefficient of variation for each coefficient. Table 5. Results for the logistic regression Grant type  Coefficients, standardized   SJRm (%)  HCD (%)  DIC (%)  Intercept (%)  Development  1.130* (18.1)  0.952* (9.2)  −  −1.891* (12.6)  Starting  –  –  0.692* (18.2)  −1.328 (12.1)  Grant type  Coefficients, standardized   SJRm (%)  HCD (%)  DIC (%)  Intercept (%)  Development  1.130* (18.1)  0.952* (9.2)  −  −1.891* (12.6)  Starting  –  –  0.692* (18.2)  −1.328 (12.1)  Note: * P-value < 0.05. The coefficient value is the mean value; cross-validation was used in adjusting the model. Between brackets is the coefficient of variation for each coefficient. In Medical and Health Sciences we found statistical significant differences, between selected and not selected applicants, for indicators related to the impact and prestige of the journals (SNIPm and SJRm). It might appear that peers are using these metrics as a proxy of the quality of the research in an article. However, this result might be explained if we consider that in the evaluation guidelines it is mentioned that the scientific merit of the applicants should be evaluated considering the publications in top specialty peer-reviewed journals. In this situation we would say that the impact and the prestige of the journal where the applicants have been published in are important parameters. On the other hand, it might happen that the papers published in these journals did not have enough time to collect a high number of citations (we are considering young researchers), but peers using their expertise are able to identify the impact of the research of such papers. For all the indicators where we observed statistical significant differences between groups, the average value is higher in the group of selected applicants with exception of Exact Sciences for the variable hnf. The confidence intervals (CIs) presented in Table A5 corroborate the obtained results in the cases where we found statistical significant differences. The Cohen’s d indicates that the differences between selected and not selected applicants in the cases where there is statistical evidence to reject the null hypothesis are large with exception of Natural Sciences for the variable related to international collaboration (see Table A5). In the case of Engineering and Technology we have similarities, if we compare with starting grants, in what concerns to the dimensions where we observe statistical significant differences. However, the indicators describing such differences are not the same. The performance in terms of impact of the publications measured by the h index, by the percentage of papers highly cited (HCD), and by the average number of citations per paper is statistically different between selected and not selected applicants. Collaboration measured by the percentage of papers with international collaboration (DIC) is another dimension where we observed statistical significant differences. The TD, PDAC, and SJRm are slightly non-significant (P-value < 0.1). For all these variables, with exception for the PDAC, the selected applicants perform better than not selected applicants, in average. Due to the importance of conferences in this field, we expected to see the percentage of proceedings papers (PP) and/or the percentage of publications classified as article and proceedings paper (PAP) to be implicit in the peers’ judgments (statistical significant). However, the results of the t-test showed that both indicators are not statistically significant and, in the case of the PAP, we can state that it is highly non-significant (P-value = 0.586). In Natural Sciences statistical significant differences in performance are observed for the dimension impact when the average number of citations per document (NI) is used and the dimension related to the scientific independence as measured by the percentage of documents where the applicants is the correspondent author (PDAC). However, for the PDAC the mean value is the highest in the case of not selected applicants. This result is unclear for us, as we expected to find the highest value in the set of selected applicants (FCT suggests as criteria for assessing scientific independence the publications where the applicant is the corresponding author). In Exact Sciences, we only found statistical significant differences for the indicator hnf which is related to the impact of the individual publications. In average, the set of selected applicants has better performance than not selected applicants, and the difference is large (see Table A4). Table 4. Mean values for the two groups of applicants in the case of starting and development grants complemented by the CIs for the mean differences and the Cohen’s d statistic (effect size) Indicator  Development grants   Starting grants   Average   CI (95%)  Effect size  Average   CI (95%)  Effect size  Selected  Not selected  Selected  Not selected  TD  41.86  40.95  [−10.36; 10.07]  −0.03  29.74  26.15  [−11.26; 4.54]  −0.20  NDF  10.04  10.29  [−2.76; 3.20]  0.03  5.52  5.55  [−1.77; 1.95]  0.03  PA  79.04  71.75  [−13.97; −0.70]  −0.41  75.44  75.67  [−7.76; 9.00]  0.04  PR  3.54  3.22  [−1.94; 1.51]  −0.08  5.51  5.62  [−3.29; 3.83]  0.04  PP  4.78  10.12  [−0.76; 10.16]  0.45  4.03  6.16  [−3.14; 6.80]  0.20  PAP  8.52  10.56  [−2.22; 6.46]  0.18  6.81  8.05  [−3.77; 6.74]  0.10  PDAC  23.97  33.86  [2.26; 17.03]  0.53  21.61  21.47  [−10.89; 7.99]  −0.04  h  14.32  11.98  [−4.01; −0.51]  −0.51  10.26  9.48  [−2.65; 1.57]  −0.14  hnf  4.74  4.19  [−1.25; 0.14]  −0.29  3.04  3.22  [−0.66; 1.08]  0.10  HCD  18.49  10.65  [13.14; −2.12]  −0.54  18.72  13.79  [−14.33; 4.20]  −0.30  NI  1.35  1.19  [0.47; 0.19]  −0.20  1.27  1.08  [−0.75; 0.25]  −0.20  SJRm  1.61  1.14  [−0.87; −0.16]  −0.48  1.35  1.09  [−0.65; −0.034]  −0.50  SNIPm  1.2  1.13  [−0.17; 0.03]  −0.27  1.22  1.14  [−0.23; 0.05]  −0.23  PQ1  59.4  51.95  [−15.50; 0.76]  −0.36  52.49  51.79  [−13.54; 10.39]  −0.07  DIC  48.55  41.72  [−15.79; 1.82]  −0.33  42.36  28.77  [−22.05; −4.00]  −0.60  Indicator  Development grants   Starting grants   Average   CI (95%)  Effect size  Average   CI (95%)  Effect size  Selected  Not selected  Selected  Not selected  TD  41.86  40.95  [−10.36; 10.07]  −0.03  29.74  26.15  [−11.26; 4.54]  −0.20  NDF  10.04  10.29  [−2.76; 3.20]  0.03  5.52  5.55  [−1.77; 1.95]  0.03  PA  79.04  71.75  [−13.97; −0.70]  −0.41  75.44  75.67  [−7.76; 9.00]  0.04  PR  3.54  3.22  [−1.94; 1.51]  −0.08  5.51  5.62  [−3.29; 3.83]  0.04  PP  4.78  10.12  [−0.76; 10.16]  0.45  4.03  6.16  [−3.14; 6.80]  0.20  PAP  8.52  10.56  [−2.22; 6.46]  0.18  6.81  8.05  [−3.77; 6.74]  0.10  PDAC  23.97  33.86  [2.26; 17.03]  0.53  21.61  21.47  [−10.89; 7.99]  −0.04  h  14.32  11.98  [−4.01; −0.51]  −0.51  10.26  9.48  [−2.65; 1.57]  −0.14  hnf  4.74  4.19  [−1.25; 0.14]  −0.29  3.04  3.22  [−0.66; 1.08]  0.10  HCD  18.49  10.65  [13.14; −2.12]  −0.54  18.72  13.79  [−14.33; 4.20]  −0.30  NI  1.35  1.19  [0.47; 0.19]  −0.20  1.27  1.08  [−0.75; 0.25]  −0.20  SJRm  1.61  1.14  [−0.87; −0.16]  −0.48  1.35  1.09  [−0.65; −0.034]  −0.50  SNIPm  1.2  1.13  [−0.17; 0.03]  −0.27  1.22  1.14  [−0.23; 0.05]  −0.23  PQ1  59.4  51.95  [−15.50; 0.76]  −0.36  52.49  51.79  [−13.54; 10.39]  −0.07  DIC  48.55  41.72  [−15.79; 1.82]  −0.33  42.36  28.77  [−22.05; −4.00]  −0.60  Note: Values in bold represent variables with P-values < 0.05 obtained using the t-test. Table 4. Mean values for the two groups of applicants in the case of starting and development grants complemented by the CIs for the mean differences and the Cohen’s d statistic (effect size) Indicator  Development grants   Starting grants   Average   CI (95%)  Effect size  Average   CI (95%)  Effect size  Selected  Not selected  Selected  Not selected  TD  41.86  40.95  [−10.36; 10.07]  −0.03  29.74  26.15  [−11.26; 4.54]  −0.20  NDF  10.04  10.29  [−2.76; 3.20]  0.03  5.52  5.55  [−1.77; 1.95]  0.03  PA  79.04  71.75  [−13.97; −0.70]  −0.41  75.44  75.67  [−7.76; 9.00]  0.04  PR  3.54  3.22  [−1.94; 1.51]  −0.08  5.51  5.62  [−3.29; 3.83]  0.04  PP  4.78  10.12  [−0.76; 10.16]  0.45  4.03  6.16  [−3.14; 6.80]  0.20  PAP  8.52  10.56  [−2.22; 6.46]  0.18  6.81  8.05  [−3.77; 6.74]  0.10  PDAC  23.97  33.86  [2.26; 17.03]  0.53  21.61  21.47  [−10.89; 7.99]  −0.04  h  14.32  11.98  [−4.01; −0.51]  −0.51  10.26  9.48  [−2.65; 1.57]  −0.14  hnf  4.74  4.19  [−1.25; 0.14]  −0.29  3.04  3.22  [−0.66; 1.08]  0.10  HCD  18.49  10.65  [13.14; −2.12]  −0.54  18.72  13.79  [−14.33; 4.20]  −0.30  NI  1.35  1.19  [0.47; 0.19]  −0.20  1.27  1.08  [−0.75; 0.25]  −0.20  SJRm  1.61  1.14  [−0.87; −0.16]  −0.48  1.35  1.09  [−0.65; −0.034]  −0.50  SNIPm  1.2  1.13  [−0.17; 0.03]  −0.27  1.22  1.14  [−0.23; 0.05]  −0.23  PQ1  59.4  51.95  [−15.50; 0.76]  −0.36  52.49  51.79  [−13.54; 10.39]  −0.07  DIC  48.55  41.72  [−15.79; 1.82]  −0.33  42.36  28.77  [−22.05; −4.00]  −0.60  Indicator  Development grants   Starting grants   Average   CI (95%)  Effect size  Average   CI (95%)  Effect size  Selected  Not selected  Selected  Not selected  TD  41.86  40.95  [−10.36; 10.07]  −0.03  29.74  26.15  [−11.26; 4.54]  −0.20  NDF  10.04  10.29  [−2.76; 3.20]  0.03  5.52  5.55  [−1.77; 1.95]  0.03  PA  79.04  71.75  [−13.97; −0.70]  −0.41  75.44  75.67  [−7.76; 9.00]  0.04  PR  3.54  3.22  [−1.94; 1.51]  −0.08  5.51  5.62  [−3.29; 3.83]  0.04  PP  4.78  10.12  [−0.76; 10.16]  0.45  4.03  6.16  [−3.14; 6.80]  0.20  PAP  8.52  10.56  [−2.22; 6.46]  0.18  6.81  8.05  [−3.77; 6.74]  0.10  PDAC  23.97  33.86  [2.26; 17.03]  0.53  21.61  21.47  [−10.89; 7.99]  −0.04  h  14.32  11.98  [−4.01; −0.51]  −0.51  10.26  9.48  [−2.65; 1.57]  −0.14  hnf  4.74  4.19  [−1.25; 0.14]  −0.29  3.04  3.22  [−0.66; 1.08]  0.10  HCD  18.49  10.65  [13.14; −2.12]  −0.54  18.72  13.79  [−14.33; 4.20]  −0.30  NI  1.35  1.19  [0.47; 0.19]  −0.20  1.27  1.08  [−0.75; 0.25]  −0.20  SJRm  1.61  1.14  [−0.87; −0.16]  −0.48  1.35  1.09  [−0.65; −0.034]  −0.50  SNIPm  1.2  1.13  [−0.17; 0.03]  −0.27  1.22  1.14  [−0.23; 0.05]  −0.23  PQ1  59.4  51.95  [−15.50; 0.76]  −0.36  52.49  51.79  [−13.54; 10.39]  −0.07  DIC  48.55  41.72  [−15.79; 1.82]  −0.33  42.36  28.77  [−22.05; −4.00]  −0.60  Note: Values in bold represent variables with P-values < 0.05 obtained using the t-test. Within the dimension production and collaboration in Medical and Health Sciences, we have statistical evidence of differences between the two groups in case of variables TD and NDF. In average, the values for these indicators are high for the set of not selected applicants. If we think in terms of performance, we would say that the better the researcher is, the higher the number of publications. However, this can be explained looking to the remaining indicators. For the selected applicants we have, in average, a higher percentage of publications classified as articles than for the not selected applicants. We looked at the percentage of meeting abstracts, as this type of publications is widely used within this scientific field and found that for not selected applicants, the percentage of meetings abstracts is in general higher than for the selected applicants. Meeting abstracts are used frequently in the dissemination of current data, although, while timely and succinct, this type of documents is frequently considered less prestigious due to a simplified peer review and incomplete bibliographic description. These characteristics of the meeting abstracts might explain why applicants with a high number of publications, in average, and a high percentage of meeting abstracts were not selected. The impact and prestige of the journal where the applicants have been published in, represented by the SJRm, is, in average, high in the case of selected applicants, and the difference is statistically significant. In terms of substantive significance the Cohen’s d indicates large effect for all the variables where there is statistical evidence to state that the means between the two groups are not equal. The CIs corroborate our results (see Table A4). If we consider, in Tables 2 and 3, those indicators where we observed statistical significant differences we can conclude that there are differences within the same dimension. In Engineering and Technology we found statistical significant differences for indicators related to the dimension quantity, impact, and collaboration for both type of grants. However, the indicators are different; for example in the starting grants collaboration is represented by the NDF, while in the development grants it is represented by the DIC. Over all the results obtained, at this point, are very interesting, as they show that: among scientific fields different dimensions are implicit in the peers’ judgments; the dimensions implicit in the peers’ judgments are different if we considered applicants at different phase of their scientific career; the use of several bibliometric indicators allows us to understand better the scientific production of a given researcher. The indicator PA was not found statistically significant in the case of Medical and Health Sciences for the development grants, but it is very important in helping to understand the average values in the case of the TD and NDF. The requirements to apply to each type of grant are different, but the guidelines/criteria proposed by the FCT for evaluation are, practically, the same among type of grants and scientific fields. The results in Tables 2 and 3 show that statistical significant differences in performance for the two set of applicants are evidenced by different dimensions and indicators. In fact, applicants to starting grants are supposed to have a maximum of 5 years after the award of the PhD degree. These are researchers at the beginning of the career, most of them with a few papers published and are building their reputation in the field. Here, some metrics can be useful, but special attention should be given to those based on citations due to the low number of papers and time needed to the maturation of the citations (Glanzel and Schoepflin 1995). In such situations, the use of variables other than those associated with citations might be more appropriated. This might explain why in the case of starting grants just a few indicators based on the individual impact of the publications were able to show statistical significant differences between the selected and not selected applicants. The main findings suggest that the aspects considered more relevant in terms of scientific performance depend on the scientific field. This result was not unexpected in our study, despite the same criteria/guidelines to be used by the peers. As a well-known example, we have the field of Mathematics where collaboration activities are not common; in average, the number of authors per publication is lower than in Chemistry and Physics (Franceschet and Costantini 2011; Vieira and Gomes 2010). In this field the use of indicators of collaboration might not be relevant to assess scientific performance. An opposite case is the case of physics where international collaboration is common. Nevertheless, we were not able to make the analysis at this level in our study; we do not have enough information to justify the differences observed among scientific fields. Only a close discussion with the peers would provide information about the differences observed both at the different phase of the scientific career and at the level of scientific field. We were able to identify those indicators where there are statistical significant differences between the two groups of applicants, but we cannot discuss their relevance in explaining peers’ judgments. This can be explored using a regression analysis, more specifically a logistic regression. However, the low number of applicants in each field makes this methodology not suitable. For this reason, the remainder of the section deals with the analysis of the whole set, taking together all different scientific areas. In reality, we know that there are marked differences of culture among areas both in terms of publication and evaluation, but the general criteria defined by the FCT are the same for all areas. Information about the dispersion in the data set is presented in the box plots, which follows for each type of grant (more descriptive statistics can be found in Table A1). In Figs 1 and 2 we can see that for most of the bibliometric indicators the distributions are skewed (e.g. SJRm and h index); there are some outstanding researchers, and high dispersion is a feature of the data set. This is also confirmed by the results presented in Table A1. Figure 1. View largeDownload slide Distribution of the data for selected (Code 1) and not selected (Code 0) applicants according to each indicator for the starting grants. Figure 1. View largeDownload slide Distribution of the data for selected (Code 1) and not selected (Code 0) applicants according to each indicator for the starting grants. Figure 2. View largeDownload slide Distribution of the data for selected (Code 1) and not selected (Code 0) applicants according to each indicator in the development grants. Figure 2. View largeDownload slide Distribution of the data for selected (Code 1) and not selected (Code 0) applicants according to each indicator in the development grants. There are applicants with a large number of documents published, but a detailed inspection of the data revealed that for most of these applicants the mean impact of the publications as measured by the NI is below the average value of the reference set. It means applicants with high productivity, but low impact. The distribution for the DIC indicator shows researchers seeking to work in an environment that is characterized by international collaborations. This might be a consequence of the scientific policies that encourage and promote research in such environment. Some applicants present very high value for this indicator. There are researchers with a very high number of highly cited publications (HCD). Research of high standards are being carried out by these researchers; this indicator is considered in the scientific community as an indicator of excellence (Bornmann, Anegon and Leydesdorff 2012; Tijssen, Visser and van Leeuwen 2002; Waltman et al. 2012). For the indicator NI, we can observe some outliers. Most of these outliers are due to the presence of one or two documents highly cited. The NI being an average value is sensible to these documents. This clearly shows the advantage of using several bibliometric indicators even when describing performance within the same dimension. Outliers are essentially observed for the indicators related to the type of documents. The outliers observed for the PP and PAP are, mainly, applicants from the Engineering and Technology field. These results are in agreement with the literature, i.e. proceedings paper is an important vehicle of dissemination of the results of the research activities in the Engineering and Technology (Butler 2008; Montesi and Owen 2008). In the case of the h index we also observe outliers. These researchers have a large number of documents published; previous studies show a positive association of the h index with the absolute number of publications (Costas and Bordons 2007; van Raan 2006). The correlation tables(Tables A2 and A3) show moderate and significant correlations between the total number of published documents and the h index. As the indicator does not consider any correction for the different culture of publication among fields, researchers with a high number of publications are expected to have high h index. In Fig. 2 we show information for the development grants. The overlap between the distributions of the selected and not selected applicants is lower than in the starting grants. Outliers are present, in general terms, for the same indicators as in the starting grants, and the factors explaining such behaviour are the same. Obviously, a large number of not selected applicants show higher indicator value than some selected applicants. This does not mean a questionable peer-review process. This might mean that if applicants pass a certain threshold, other evaluation criteria (e.g. grants awarded, abilities and skills to adequately execute the proposed project, and relevance and originality of the project proposed (see FCT 2012b)) dominate the final decision. We present, in Table 4, the observed average value for each indicator within the group of selected and not selected applicants. In the case of starting grants the results of the t-test shows that the SJRm and DIC are the unique variables for which statistical significant differences are observed between the selected and not selected applicants. The CIs corroborate the results and the differences between the two groups vary between moderate and large. For the remaining variables there is no statistical evidence to consider that the performance of the groups is different, and the differences are considered of small effect. In the case of development grants for 6 of 15 indicators we identified statistical significant differences between the two groups being predominant moderate differences. The indicators represent several dimensions: quantity, impact, and scientific independence. 4.2 The impact of different dimensions The results presented above showed that there are statistically significant differences between the sets of selected and not selected applicants, although we do not have information about how the indicators can be combined in a way that contains information about the effect size and the capability of representing the peers’ final decision. The logistic regression was applied to the applicants of each grant type, using as independent variables, the bibliometric indicators that showed statistically significant differences between the two groups, and as dependent variable, a binary variable that represents the final decision (1—applicant selected; 0—applicant not selected). The coefficient value is the mean value; cross-validation was used in adjusting the model. Between brackets is the coefficient of variation for each coefficient. Several models were obtained using the variables for which the t-test showed statistically significant differences. In Table 5, we only present the model that showed to have the highest performance in explaining the final decisions of the peers. The results obtained allow us to answer to our research question; the dimensions implicit in peers’ judgments and most valued are different between type of grants. In the case of development grants the impact and the prestige of the source used to disseminate the research findings (as measured by the SJRm) are implicit in the peer judgments. The same is likely to be true for the research that is considered highly cited or excellent (Bornmann et al. 2012; Tijssen et al. 2002; Waltman et al. 2012) as measured by the HCD. Both indicators have a positive impact, and the effect size is similar with a slight advantage for applicants with high SJRm. For the starting grants the indicator that gives information about the traffic of international collaboration is the only one with significant and positive impact, and, in fact, the degree of internationalization of the applicant is suggested in the guide for peer review. The coefficients of the models were obtained using cross-validation due to the small amount of data. The values for the coefficients represent the mean value, and within brackets is the coefficient of variation; the values of the coefficient of variation suggest that our models are robust in relation to the used data set. Considering 50% as probability of cut-off in studying the several measures of fit of the models, presented in Table A6, we conclude that the models fit well the data. The percentage of cases predicted correctly is similar for both models, about 70%. It is important to highlight that there is a set of criteria that could not be considered in our study as the research project and the career development plan, which have a weight of 40% on the total final decision, as well as additional information about scientific merit (grants and/or projects awarded in very competitive calls, doctoral and post-doctoral training). Considering the limited scope of analysis of our indicators in relation to the whole set of features the peers are asked to assess, the 70% overlap found here is very good indeed. Table 6. Overlapping between the results given by the models and the peer decisions among scientific fields Scientific field  Correct prediction (%)   Starting grant  Development grant  Engineering and Technology  71  76  Natural Sciences  76  73  Exact Sciences  67  67  Medical and Health Sciences  47  69  All areas  69.7  68.8  Scientific field  Correct prediction (%)   Starting grant  Development grant  Engineering and Technology  71  76  Natural Sciences  76  73  Exact Sciences  67  67  Medical and Health Sciences  47  69  All areas  69.7  68.8  Table 6. Overlapping between the results given by the models and the peer decisions among scientific fields Scientific field  Correct prediction (%)   Starting grant  Development grant  Engineering and Technology  71  76  Natural Sciences  76  73  Exact Sciences  67  67  Medical and Health Sciences  47  69  All areas  69.7  68.8  Scientific field  Correct prediction (%)   Starting grant  Development grant  Engineering and Technology  71  76  Natural Sciences  76  73  Exact Sciences  67  67  Medical and Health Sciences  47  69  All areas  69.7  68.8  For the development grants, actually selected applicants for funding for whom the model predicts a probability of being selected by the peers higher than 50% is about 71% and for the starting grants is about 80% showing the high sensitivity of the models. In relation to specificity, the models show a lower performance, only for 65% and 55.6% (development and starting grant respectively) of the total applicants not selected for funding the models predict a probability of being selected by the peers lower than the value established as cut-off, which implies a considerable amount of the false positives, especially for the starting grant (44.4%). Concerning the discrimination power of the models (AUC, Table A6), the literature suggests a poor discrimination (0.5 < AUC < 0.7) for the model obtained for the starting grants; for the development grants, the power discrimination is acceptable (0.7 < AUC < 0.8) (Hosmer, Lemeshow and Sturdivant 2013). Visual observation of the ROC curve is presented in Fig. A1. This calls for some caution in the interpretation of the results in the case of starting grants. Taking into account the predicted probabilities for each observation, we analysed the percentage of observations predicted correctly in each scientific domain (Table 6). Not all the indicators presented in Table 5 were found to be important in identifying statistical significant differences in the performance between selected and not selected applicants for some fields, in Tables 2 and 3. For these fields we expect a lower percentage of cases predicted correctly. In the case of development grants for the field Engineering and Technology using the obtained model, the percentage of cases predicted correctly is the highest; the SJRm is slightly non-significant (0.5 < P-value < 0.1), and the HCD is significant in Table 3. All in all when interpreting the results given by our models, we have to be aware about the limitations introduced by the design of the study. In finding information about the effect size and capability of the indicators in representing the peers’ final decisions, we are not controlling for the effect of scientific field, and this might introduce some unquantifiable biases. We were able to predict a reasonable overlap between the results from bibliometric analysis and peers’ decisions, but our results would improve if we had considered the effect of the scientific field. The results obtained are in concordance with previous studies. For studies at the level of individuals, the HCD was found with positive and significant influence in predicting the peers’ judgments (Vieira et al. 2014a, 2014b). The impact of the journal where the applicants have published in was also found relevant in (Cabezas-Clavijo et al. 2013), although the authors used a different indicator, the JIF. We deem the indicator SJRm more appropriate, as it is a normalized indicator for the different culture of citations among fields. 5. Discussion When using bibliometric indicators for understanding the peer-reviewed process at the programme Investigador FCT, we are looking for variables that to some extent are related to those dimensions that peers appear to consider more relevant at each phase of the scientific career. The general criteria of evaluation were defined by FCT for all peers, but they use these criteria in accordance with their own understanding of scientific merit and quality. The bibliometric indicators used in this study were not made available to the peers in a formal way, and it is unlikely that most peers will have calculated them, especially those that require a normalization procedure. All we can test is that certain parameters correlate well with the decision not implying necessarily a direct causal relation. We will argue that some of the indicators found to be relevant may be understood as some measure of what the peers may be looking at. In the case of the starting grants, the results from Table 5 show that international collaboration is able to reproduce peer decisions to some extent. We must address the following questions: Why is international collaboration so important? What does international collaboration tell us about scientific merit? On one hand, it may be argued that peers are considering as important criteria in the evaluation process aspects that are correlated with international collaboration. For example, peers might be considering as relevant the grants/projects awarded in competitive calls as this was mentioned in the guidelines for evaluation. The literature shows that grants/projects awarded facilitate in establishing international collaboration (Bloch, Graversen and Pedersen 2014). On the other hand, peers are valuing the expected future performance, and the literature shows that international collaboration is determinant in enhancing high scientific performance (Bordons, Aparicio and Costas 2013; Glanzel 2001). In this situation we can state that peers are considering past and actual international collaborations as a predictor of a career of high performance. In the case of the development grants the dimension impact, represented by indicators SJRm and HCD, is the most relevant in predicting the application success. There are several situations where indicators related to the impact and prestige of the journals are being used in the individual assessment of publications as a way of replacing citation statistics. This practice raised several debates in the scientific community, mainly within bibliometricians, who condemn this behaviour. Taking into account the debate in the scientific community we would like to highlight that the results obtained in our study cannot be interpreted without looking to the evaluation criteria. In fact, in the document related to the guidelines for peer review it is said: ‘Indicators for scientific merit of the applicant include the main academic and professional degrees, publications in top specialty peer-reviewed journals and/or in major multidisciplinary international peer-reviewed journals’ (FCT 2012b: 4). Therefore, the results obtained are in agreement with some of the criteria that are supposed to be used in the evaluation process. The HCD variable is normally considered to point to excellent research (Tijssen et al. 2002; Waltman et al. 2012); in fact, in the majority of the cases, research with a very large number of citations in the field means high impact in the community; impact is a necessary condition for researchers and institutions to achieve broad visibility and success in terms of scientific recognition. At this level we can conclude that peers are looking for scientists that produce research of high standards and that are already established as research leaders in their own right. 6. Conclusions This work seeks a better understanding of the selection process carried out by peers at the individual level, in this specific case, at the programme Investigador FCT, in Portugal. The programme aims at selecting outstanding researchers into Portugal’s R&D institutions. Applicants should have an exceptional track record and a clear potential to develop innovative research. Using a set of bibliometric indicators, chosen taking into account previous studies and the evaluation guidelines provided by the FCT to the peers, we tried to understand the peer-review process looking for the indicators that are implicit in the peers’ judgments, identifying the dimensions of the scientific performance more valued by the peers. Our empirical results show that bibliometric indicators allow an understanding of the selection process identifying differences on scientific performance among applicants as seen by the peers: we highlight here the dimensions more valued by the community of peers. These results may be used to attenuate the subjectivity or the bias always associated with peer-review decisions by small expert panels. For applicants with a very short career, we found that the percentage of documents with international collaboration is able to predict the final decision of the peers for 70% of the applicants. International collaboration depends on a network of co-authors that somehow acknowledge the applicant’s research. This suggests that better researchers tend to have more intense international collaboration. Furthermore, we cannot rule out that international collaboration may be related to other criteria used by the peers that we were not able to use in this study. More refined conclusions in relation to the meaning of international collaboration can only be obtained if these results were discussed in detail with the peer panels. In the case of development grants the indicators that showed better performance in representing the final decisions were the SJRm and the HCD; for 70% of the applicants the prediction given by the bibliometric indicators overlaps with the peer decisions. If both indicators were made available to the peers, they would have a positive and significant impact on the final decision. The impact and prestige of the journals as measured by the SJRm would be the variable with the highest impact. When scientific domains are considered separately, different bibliometric indicators showed differences among applicants reflecting the role of the very different cultures of publication and citation of the scientific domains. However, a more in-depth analysis of this behaviour is necessary aimed at determining the relevance of each indicator in predicting the final peers’ decisions. This was not possible as the number of applicants in each field is not enough to draw statistically robust results. The overlap between the results given by the bibliometric indicators and the peers’ decisions is very encouraging, as there are several other factors that might influence peers’ decisions. Most of the indicators used are based on citation data, and this is perceived as a measure of impact of the scientific output (its actual influence on surrounding research activities at a given time). However, peers when assessing scientific production might also consider their personal assessment of the relevance (the influence of research on the advance of scientific knowledge) and technical quality of the research, whether it is free from obvious flaws, how aesthetically pleasing the mathematical formulations are, how original the conclusions are, and so on. On the other hand, peers were also asked, in the evaluation guidelines, to take into account the number of grants awarded, doctoral and post-doctoral training, the career development plan, and the research project, and this information was not made available for our study. The results presented in the article suggest that bibliometric indicators can have a role in peer assessment exercises. They were not available to the peer review in a formal way as far as we know; therefore in our study we can only state that the indicators found important in predicting the final decisions are implicit in the judgements made by the peers. Bibliometric indicators are known by their objectivity, and we might interrogate about the design of national evaluation exercises where the assessment is purely based on the results given by the bibliometric indicators. An evaluation exercise based only in bibliometric indicators requires a set of procedures that go from building an appropriate infrastructure to store all the data needed to the calculation of the bibliometric indicators until a clear and objective definition of the several aspects to be evaluated. The last point inevitably implies a massive interaction of bibliometric experts and experts belonging to the several fields under evaluation aimed at defining clearly what the main cultures of publication and citation are. On the other side, despite the attractive features of bibliometric indicators, there are several situations where they are not able to replace the role of the peers; e.g, it will be very difficult to assess the impact of a recent publication if we measure impact counting citations. A citation window between 3 and 5 years is required to collect the citations of publications. In a case where a system purely based on bibliometric indicators is to be used, it will be very difficult to have a clear picture about the impact of recent publications. Another example where bibliometric indicators fail is in book-oriented fields as counting citations from articles, which dominate traditional citation indexes, seems insufficient to assess the impact of books. The two examples show that bibliometric indicators can work better as a complementary tool of peer-review processes than as tool to replace them. The results presented here should be seen as preliminary results. We have identified the most important bibliometric indicators in predicting the final decisions from our set of 15 indicators. However, we should not discard the possibility of using other indicators that contribute to a better understanding of the scientific performance, even if they do not have a decisive role in building the final decisions. The very general evaluation criteria presented by the FCT do not allow us to extend our discussion into the design of an evaluation system, with the same goals as that of the FCT, where bibliometric indicators are to be used as complementary tool. Such design implies a clear and objective definition of the criteria to be evaluated and a strong interaction between bibliometricians and peers aimed at defining the most relevant aspects of the scientific performance in each domain and the most appropriate bibliometric indicators and the way they should be used. Acknowledgements The authors wish to acknowledge the data on the Investigador FCT applicants that were kindly made available by Fundação para a Ciência e a Tecnologia. Funding This work was supported by the Fundação para a Ciência e a Tecnologia (FCT) from Portugal, [grant number SFRH/BPD/99246/2013]. References Abramo G., D'Angelo C. A. ( 2015) ‘ The VQR, Italy's Second National Research Assessment: Methodological Failures and Ranking Distortions’, Journal of the Association for Information Science and Technology , 66/ 11: 2202– 14. Google Scholar CrossRef Search ADS   Abramo G., D'Angelo C. A., Caprasecca A. ( 2009) ‘ Allocative Efficiency in Public Research Funding: Can Bibliometrics Help?’, Research Policy , 38/ 1: 206– 15. Google Scholar CrossRef Search ADS   Aksnes D. W., Taxt R. E. ( 2004) ‘ Peer Reviews and Bibliometric Indicators: A Comparative Study at a Norwegian University’, Research Evaluation , 13/ 1: 33– 41. Google Scholar CrossRef Search ADS   Bloch C., Graversen E. K., Pedersen H. S. ( 2014) ‘ Competitive Research Grants and Their Impact on Career Performance’, Minerva , 52/ 1: 77– 96. Google Scholar CrossRef Search ADS   Bordons M., Aparicio J., Costas R. ( 2013) ‘ Heterogeneity of Collaboration and Its Relationship with Research Impact in a Biomedical Field’, Scientometrics , 96/ 2: 443– 66. Google Scholar CrossRef Search ADS   Bornmann L., Anegon F. D., Leydesdorff L. ( 2012) ‘ The New Excellence Indicator in the World Report of the SCImago Institutions Rankings 2011’, Journal of Informetrics , 6/ 2: 333– 5. Google Scholar CrossRef Search ADS   Bornmann L., Leydesdorff L. ( 2013) ‘ The Validation of (Advanced) Bibliometric Indicators Through Peer Assessments: A Comparative Study Using Data from InCites and F1000’, Journal of Informetrics , 7/ 2: 286– 91. Google Scholar CrossRef Search ADS   Bornmann L., Marx W. ( 2015) ‘ Methods for the Generation of Normalized Citation Impact Scores in Bibliometrics: Which Method Best Reflects the Judgements of Experts?’, Journal of Informetrics , 9/ 2: 408– 18. Google Scholar CrossRef Search ADS   Bornmann L., Wallon G., Ledin A. ( 2008) ‘ Does the Committee Peer Review Select the Best Applicants for Funding? An Investigation of the Selection Process for Two European Molecular Biology Organization Programmes’, PLoS One , 3/ 10: e3480. Google Scholar CrossRef Search ADS PubMed  Butler L. ( 2008) ‘ ICT Assessment: Moving beyond Journal Outputs’, Scientometrics , 74/ 1: 39– 55. Google Scholar CrossRef Search ADS   Butler L., McAllister I. ( 2011) ‘ Evaluating University Research Performance Using Metrics’, European Political Science , 10/ 1: 44– 58. Google Scholar CrossRef Search ADS   Cabezas-Clavijo A. et al.   ( 2013) ‘ Reviewers' Ratings and Bibliometric Indicators: Hand in Hand When Assessing over Research Proposals?’, PLoS One , 8/ 6: e68258. Google Scholar CrossRef Search ADS PubMed  Costas R., Bordons M. ( 2007) ‘ The h-Index: Advantages, Limitations and Its Relation with Other Bibliometric Indicators at the Micro Level’, Journal of Informetrics , 1/ 3: 193– 203. Google Scholar CrossRef Search ADS   ERC ( 2015). ERC Frontier Research Grants—Guide for Peer Reviewers’, <https://erc.europa.eu/sites/default/files/document/file/Guide-for-Peer_reviewers_StG_CoG_AdG_2015.pdf> accessed March 2016. FCT ( 2012a). ‘Guide for Writing and Submitting Applications for the FCT Investigator Grants’, <http://www.fct.pt/apoios/contratacaodoutorados/investigador-fct/2012/docs/ApplicationGuide.pdf> accessed December 2015. FCT ( 2012b). ‘Investigator FCT Grants—Guide for Peer Reviewers’, <http://www.fct.pt/apoios/contratacaodoutorados/investigador-fct/2012/docs/GuidelinesPeerReview.pdf> accessed December 2015. FCT ( 2014). ‘Evaluation guide for the 2014FCT Investigator Grant’, <http://www.fct.pt/apoios/contratacaodoutorados/investigador-fct/2014/docs/EvaluationGuide_IF2014.pdf> accessed December 2015. Franceschet M., Costantini A. ( 2011) ‘ The First Italian Research Assessment Exercise: A Bibliometric Perspective’, Journal of Informetrics , 5/ 2: 275– 91. Google Scholar CrossRef Search ADS   Glanzel W. ( 2001) ‘ National Characteristics in International Scientific Co-Authorship Relations’, Scientometrics , 51/ 1: 69– 115. Google Scholar CrossRef Search ADS   Glanzel W., Schoepflin U. ( 1995) ‘ A Bibliometric Study on Aging and Reception Processes of Scientific Literature’, Journal of Information Science , 21/ 1: 37– 53. Google Scholar CrossRef Search ADS   Gonzalez-Pereira B., Guerrero-Bote V. P., Moya-Anegon F. ( 2010) ‘ A New Approach to the Metric of Journals' Scientific Prestige: The SJR Indicator’, Journal of Informetrics , 4/ 3: 379– 91. Google Scholar CrossRef Search ADS   Hirsch J. E. ( 2005) ‘ An Index to Quantify an Individual's Scientific Research Output’, Proceedings of the National Academy of Sciences of the United States of America , 102/ 46: 16569– 72. Google Scholar CrossRef Search ADS PubMed  Hosmer D. W.Jr., Lemeshow S., Sturdivant R. X. ( 2013). Applied Logistic Regression , 3rd edn. Wiley (April 1, 2013). Google Scholar CrossRef Search ADS   Jensen P., Rouquier J. B., Croissant Y. ( 2009) ‘ Testing Bibliometric Indicators by Their Prediction of Scientists Promotions’, Scientometrics , 78/ 3: 467– 79. Google Scholar CrossRef Search ADS   Lehmann S., Jackson A. D., Lautrup B. E. ( 2008) ‘ A Quantitative Analysis of Indicators of Scientific Performance’, Scientometrics , 76/ 2: 369– 90. Google Scholar CrossRef Search ADS   Meyer B. et al.   ( 2009) ‘ Research Evaluation for Computer Science’, Communications of the ACM , 52/ 4: 31– 4. Google Scholar CrossRef Search ADS   Moed H. F. ( 2010) ‘ Measuring Contextual Citation Impact of Scientific Journals’, Journal of Informetrics , 4/ 3: 265– 77. Google Scholar CrossRef Search ADS   Montesi M., Owen J. M. ( 2008) ‘ From Conference to Journal Publication: How Conference Papers in Software Engineering are Extended for Publication in Journals’, Journal of the American Society for Information Science and Technology , 59/ 5: 816– 29. Google Scholar CrossRef Search ADS   Neufeld J., Huber N., Wegner A. ( 2013) ‘ Peer Review-Based Selection Decisions in Individual Research Funding, Applicants' Publication Strategies and Performance: The Case of the ERC Starting Grants’, Research Evaluation , 22/ 4: 237– 47. Google Scholar CrossRef Search ADS   Norris M., Oppenheim C. ( 2003) ‘ Citation Counts and the Research Assessment Exercise V—Archaeology and the 2001 RAE’, Journal of Documentation , 59/ 6: 709– 30. Google Scholar CrossRef Search ADS   Oppenheim C. ( 1997) ‘ The Correlation between Citation Counts and the 1992 Research Assessment Exercise Ratings for British Research in Genetics, Anatomy and Archaeology’, Journal of Documentation , 53/ 5: 477– 87. Google Scholar CrossRef Search ADS   Ramos A., Sarrico C. S. ( 2016) ‘ Past Performance does not Guarantee Future Results: Lessons from the Evaluation of Research Units in Portugal’, Research Evaluation , 25: 94– 106. Google Scholar CrossRef Search ADS   Rinia E. J. et al.   ( 1998) ‘ Comparative Analysis of a Set of Bibliometric Indicators and Central Peer Review Criteria—Evaluation of Condensed Matter Physics in The Netherlands’, Research Policy , 27/ 1: 95– 107. Google Scholar CrossRef Search ADS   Taylor J. ( 2011) ‘ The Assessment of Research Quality in UK Universities: Peer Review or Metrics?’, British Journal of Management , 22/ 2: 202– 17. Google Scholar CrossRef Search ADS   Tijssen R. J. W., Visser M. S., van Leeuwen T. N. ( 2002) ‘ Benchmarking International Scientific Excellence: Are Highly Cited Research Papers an Appropriate Frame of Reference?’, Scientometrics , 54/ 3: 381– 97. Google Scholar CrossRef Search ADS   Torres-Salinas D., Jimenez-Contreras E. ( 2010) ‘ Introduction and Comparative Study of the New Scientific Journals Citation Indicators in Journal Citation Reports and Scopus’, Profesional De La Informacion , 19/ 2: 201– 7. Google Scholar CrossRef Search ADS   van Raan A. F. J. ( 2006) ‘ Comparison of the Hirsch-Index with Standard Bibliometric Indicators and with Peer Judgment for 147 Chemistry Research Groups’, Scientometrics , 67/ 3: 491– 502. Google Scholar CrossRef Search ADS   Vieira E. S., Cabral J. A. S., Gomes J. A. N. F. ( 2014a) ‘ Definition of a Model Based on Bibliometric Indicators for Assessing Applicants to Academic Positions’, Journal of the Association for Information Science and Technology , 65/ 3: 560– 77. Google Scholar CrossRef Search ADS   Vieira E. S., Cabral J. A. S., Gomes J. A. N. F. ( 2014b) ‘ How Good is a Model Based on Bibliometric Indicators in Predicting the Final Decisions Made by Peers?’, Journal of Informetrics , 8/ 2: 390– 405. Google Scholar CrossRef Search ADS   Vieira E. S., Gomes J. A. N. F. ( 2010) ‘ Citations to Scientific Articles: Its Distribution and Dependence on the Article Features’, Journal of Informetrics , 4/ 1: 1– 13. Google Scholar CrossRef Search ADS   Vieira E. S., Gomes J. A. N. F. ( 2011) ‘ An Impact Indicator for Researchers’, Scientometrics , 89/ 2: 607– 29. Google Scholar CrossRef Search ADS   Vieira E. S., Gomes J. A. N. F. ( 2015) ‘ The Bibliometric Indicators as Predictors of the Final Decision of the Peer Review’, Research Evaluation , 25: 170– 83. Google Scholar CrossRef Search ADS   Waltman L. et al.   ( 2012) ‘ The Leiden Ranking 2011/2012: Data Collection, Indicators, and Interpretation’, Journal of the American Society for Information Science and Technology , 63/ 12: 2419– 32. Google Scholar CrossRef Search ADS   Wilsdon J. et al.   ( 2015). ‘The Metric Tide: Report of the Independent Review of the Role of Metrics in Research Assessment and Management’. Appendix Descriptive results Significance test and substantive significance of effects In Tables that follows we present the CIs obtained using bootstrapping techniques and the Cohen’s d statistics used to complement the results of the t-test. Logistic regression We determined a set of measures that allow assessing the performance of the models and present them in Table A6. Table A1. Descriptive statistics for starting and development grants Variable  Development grant   Starting grant   Mean  p50  SD  Maximum  Minimum  Mean  p50  SD  Maximum  Minimum  TD  41.85  36.00  27.39  208.00  14.00  28.27  24.00  16.93  74.00  10.00  NDF  10.32  7.90  7.76  45.75  2.78  5.53  4.26  3.78  19.01  0.85  PA  76.44  81.37  17.83  98.28  32.50  75.53  76.83  18.13  100.00  14.81  PR  3.31  2.33  4.27  20.00  0.00  5.56  2.82  7.43  30.77  0.00  PP  6.75  2.33  11.84  51.35  0.00  4.90  0.00  10.32  40.00  0.00  PAP  9.32  5.33  11.62  55.56  0.00  7.32  2.74  10.95  60.61  0.00  PDAC  27.95  24.72  19.47  84.21  0.00  21.55  16.87  21.31  82.05  0.00  h  13.39  13.00  4.75  30.00  5.00  9.94  10.00  4.33  25.00  4.00  hnf  4.55  4.24  1.94  12.92  1.85  3.11  2.82  1.76  8.04  0.22  HCD  14.48  9.31  14.85  66.67  0.00  16.70  12.50  18.94  100.00  0.00  NI  1.29  1.09  0.87  6.21  0.29  1.19  1.00  0.95  7.09  0.16  SNIPm  1.16  1.18  0.24  2.19  0.40  1.19  1.19  0.26  1.99  0.32  SJRm  1.40  1.17  1.02  7.85  0.20  1.24  1.08  0.66  3.59  0.00  PQ1  55.68  56.24  20.75  100.00  4.55  52.21  53.14  22.23  89.36  5.26  DIC  60.96  63.00  28.05  100.00  0.00  36.80  35.31  22.18  82.35  0.00  Variable  Development grant   Starting grant   Mean  p50  SD  Maximum  Minimum  Mean  p50  SD  Maximum  Minimum  TD  41.85  36.00  27.39  208.00  14.00  28.27  24.00  16.93  74.00  10.00  NDF  10.32  7.90  7.76  45.75  2.78  5.53  4.26  3.78  19.01  0.85  PA  76.44  81.37  17.83  98.28  32.50  75.53  76.83  18.13  100.00  14.81  PR  3.31  2.33  4.27  20.00  0.00  5.56  2.82  7.43  30.77  0.00  PP  6.75  2.33  11.84  51.35  0.00  4.90  0.00  10.32  40.00  0.00  PAP  9.32  5.33  11.62  55.56  0.00  7.32  2.74  10.95  60.61  0.00  PDAC  27.95  24.72  19.47  84.21  0.00  21.55  16.87  21.31  82.05  0.00  h  13.39  13.00  4.75  30.00  5.00  9.94  10.00  4.33  25.00  4.00  hnf  4.55  4.24  1.94  12.92  1.85  3.11  2.82  1.76  8.04  0.22  HCD  14.48  9.31  14.85  66.67  0.00  16.70  12.50  18.94  100.00  0.00  NI  1.29  1.09  0.87  6.21  0.29  1.19  1.00  0.95  7.09  0.16  SNIPm  1.16  1.18  0.24  2.19  0.40  1.19  1.19  0.26  1.99  0.32  SJRm  1.40  1.17  1.02  7.85  0.20  1.24  1.08  0.66  3.59  0.00  PQ1  55.68  56.24  20.75  100.00  4.55  52.21  53.14  22.23  89.36  5.26  DIC  60.96  63.00  28.05  100.00  0.00  36.80  35.31  22.18  82.35  0.00  Table A1. Descriptive statistics for starting and development grants Variable  Development grant   Starting grant   Mean  p50  SD  Maximum  Minimum  Mean  p50  SD  Maximum  Minimum  TD  41.85  36.00  27.39  208.00  14.00  28.27  24.00  16.93  74.00  10.00  NDF  10.32  7.90  7.76  45.75  2.78  5.53  4.26  3.78  19.01  0.85  PA  76.44  81.37  17.83  98.28  32.50  75.53  76.83  18.13  100.00  14.81  PR  3.31  2.33  4.27  20.00  0.00  5.56  2.82  7.43  30.77  0.00  PP  6.75  2.33  11.84  51.35  0.00  4.90  0.00  10.32  40.00  0.00  PAP  9.32  5.33  11.62  55.56  0.00  7.32  2.74  10.95  60.61  0.00  PDAC  27.95  24.72  19.47  84.21  0.00  21.55  16.87  21.31  82.05  0.00  h  13.39  13.00  4.75  30.00  5.00  9.94  10.00  4.33  25.00  4.00  hnf  4.55  4.24  1.94  12.92  1.85  3.11  2.82  1.76  8.04  0.22  HCD  14.48  9.31  14.85  66.67  0.00  16.70  12.50  18.94  100.00  0.00  NI  1.29  1.09  0.87  6.21  0.29  1.19  1.00  0.95  7.09  0.16  SNIPm  1.16  1.18  0.24  2.19  0.40  1.19  1.19  0.26  1.99  0.32  SJRm  1.40  1.17  1.02  7.85  0.20  1.24  1.08  0.66  3.59  0.00  PQ1  55.68  56.24  20.75  100.00  4.55  52.21  53.14  22.23  89.36  5.26  DIC  60.96  63.00  28.05  100.00  0.00  36.80  35.31  22.18  82.35  0.00  Variable  Development grant   Starting grant   Mean  p50  SD  Maximum  Minimum  Mean  p50  SD  Maximum  Minimum  TD  41.85  36.00  27.39  208.00  14.00  28.27  24.00  16.93  74.00  10.00  NDF  10.32  7.90  7.76  45.75  2.78  5.53  4.26  3.78  19.01  0.85  PA  76.44  81.37  17.83  98.28  32.50  75.53  76.83  18.13  100.00  14.81  PR  3.31  2.33  4.27  20.00  0.00  5.56  2.82  7.43  30.77  0.00  PP  6.75  2.33  11.84  51.35  0.00  4.90  0.00  10.32  40.00  0.00  PAP  9.32  5.33  11.62  55.56  0.00  7.32  2.74  10.95  60.61  0.00  PDAC  27.95  24.72  19.47  84.21  0.00  21.55  16.87  21.31  82.05  0.00  h  13.39  13.00  4.75  30.00  5.00  9.94  10.00  4.33  25.00  4.00  hnf  4.55  4.24  1.94  12.92  1.85  3.11  2.82  1.76  8.04  0.22  HCD  14.48  9.31  14.85  66.67  0.00  16.70  12.50  18.94  100.00  0.00  NI  1.29  1.09  0.87  6.21  0.29  1.19  1.00  0.95  7.09  0.16  SNIPm  1.16  1.18  0.24  2.19  0.40  1.19  1.19  0.26  1.99  0.32  SJRm  1.40  1.17  1.02  7.85  0.20  1.24  1.08  0.66  3.59  0.00  PQ1  55.68  56.24  20.75  100.00  4.55  52.21  53.14  22.23  89.36  5.26  DIC  60.96  63.00  28.05  100.00  0.00  36.80  35.31  22.18  82.35  0.00  Table A2. Correlations among variables for the development grant Indicators  SJRm  SNIPm  h  hnf  NDF  NI  HCD  DIC  PA  PR  PAP  PP  TD  PQ1  PDAC  SJRm  1.00                              SNIPm  0.69  1.00                            h  0.16  0.36  1.00                          hnf  −0.06  0.10  0.57  1.00                        NDF  −0.13  −0.08  0.36  0.84  1.00                      NI  0.27  0.32  0.47  −0.08  −0.14  1.00                    HCD  0.19  0.34  0.46  0.04  −0.10  0.55  1.00                  DIC  0.29  0.26  0.07  −0.05  −0.03  0.30  0.15  1.00                PA  0.39  0.45  0.22  0.25  0.06  0.06  0.08  0.11  1.00              PR  0.19  0.17  0.22  −0.07  −0.19  0.22  0.35  −0.02  0.05  1.00            PAP  −0.31  −0.26  −0.11  −0.11  −0.07  −0.11  −0.21  0.05  −0.61  −0.27  1.00          PP  −0.34  −0.51  −0.33  −0.14  0.08  −0.18  −0.22  −0.15  −0.66  −0.29  0.18  1.00        TD  −0.17  0.00  0.65  0.51  0.60  0.31  0.04  0.07  −0.09  −0.16  0.17  0.02  1.00      PQ1  0.63  0.73  0.32  0.04  −0.10  0.32  0.27  0.33  0.40  0.23  −0.22  −0.53  −0.03  1.00    PDAC  −0.16  −0.19  −0.30  0.20  0.30  −0.25  −0.22  −0.05  −0.05  −0.18  −0.04  0.28  −0.10  −0.24  1.00  Indicators  SJRm  SNIPm  h  hnf  NDF  NI  HCD  DIC  PA  PR  PAP  PP  TD  PQ1  PDAC  SJRm  1.00                              SNIPm  0.69  1.00                            h  0.16  0.36  1.00                          hnf  −0.06  0.10  0.57  1.00                        NDF  −0.13  −0.08  0.36  0.84  1.00                      NI  0.27  0.32  0.47  −0.08  −0.14  1.00                    HCD  0.19  0.34  0.46  0.04  −0.10  0.55  1.00                  DIC  0.29  0.26  0.07  −0.05  −0.03  0.30  0.15  1.00                PA  0.39  0.45  0.22  0.25  0.06  0.06  0.08  0.11  1.00              PR  0.19  0.17  0.22  −0.07  −0.19  0.22  0.35  −0.02  0.05  1.00            PAP  −0.31  −0.26  −0.11  −0.11  −0.07  −0.11  −0.21  0.05  −0.61  −0.27  1.00          PP  −0.34  −0.51  −0.33  −0.14  0.08  −0.18  −0.22  −0.15  −0.66  −0.29  0.18  1.00        TD  −0.17  0.00  0.65  0.51  0.60  0.31  0.04  0.07  −0.09  −0.16  0.17  0.02  1.00      PQ1  0.63  0.73  0.32  0.04  −0.10  0.32  0.27  0.33  0.40  0.23  −0.22  −0.53  −0.03  1.00    PDAC  −0.16  −0.19  −0.30  0.20  0.30  −0.25  −0.22  −0.05  −0.05  −0.18  −0.04  0.28  −0.10  −0.24  1.00  Note: Values in the bold show significant correlations. Table A2. Correlations among variables for the development grant Indicators  SJRm  SNIPm  h  hnf  NDF  NI  HCD  DIC  PA  PR  PAP  PP  TD  PQ1  PDAC  SJRm  1.00                              SNIPm  0.69  1.00                            h  0.16  0.36  1.00                          hnf  −0.06  0.10  0.57  1.00                        NDF  −0.13  −0.08  0.36  0.84  1.00                      NI  0.27  0.32  0.47  −0.08  −0.14  1.00                    HCD  0.19  0.34  0.46  0.04  −0.10  0.55  1.00                  DIC  0.29  0.26  0.07  −0.05  −0.03  0.30  0.15  1.00                PA  0.39  0.45  0.22  0.25  0.06  0.06  0.08  0.11  1.00              PR  0.19  0.17  0.22  −0.07  −0.19  0.22  0.35  −0.02  0.05  1.00            PAP  −0.31  −0.26  −0.11  −0.11  −0.07  −0.11  −0.21  0.05  −0.61  −0.27  1.00          PP  −0.34  −0.51  −0.33  −0.14  0.08  −0.18  −0.22  −0.15  −0.66  −0.29  0.18  1.00        TD  −0.17  0.00  0.65  0.51  0.60  0.31  0.04  0.07  −0.09  −0.16  0.17  0.02  1.00      PQ1  0.63  0.73  0.32  0.04  −0.10  0.32  0.27  0.33  0.40  0.23  −0.22  −0.53  −0.03  1.00    PDAC  −0.16  −0.19  −0.30  0.20  0.30  −0.25  −0.22  −0.05  −0.05  −0.18  −0.04  0.28  −0.10  −0.24  1.00  Indicators  SJRm  SNIPm  h  hnf  NDF  NI  HCD  DIC  PA  PR  PAP  PP  TD  PQ1  PDAC  SJRm  1.00                              SNIPm  0.69  1.00                            h  0.16  0.36  1.00                          hnf  −0.06  0.10  0.57  1.00                        NDF  −0.13  −0.08  0.36  0.84  1.00                      NI  0.27  0.32  0.47  −0.08  −0.14  1.00                    HCD  0.19  0.34  0.46  0.04  −0.10  0.55  1.00                  DIC  0.29  0.26  0.07  −0.05  −0.03  0.30  0.15  1.00                PA  0.39  0.45  0.22  0.25  0.06  0.06  0.08  0.11  1.00              PR  0.19  0.17  0.22  −0.07  −0.19  0.22  0.35  −0.02  0.05  1.00            PAP  −0.31  −0.26  −0.11  −0.11  −0.07  −0.11  −0.21  0.05  −0.61  −0.27  1.00          PP  −0.34  −0.51  −0.33  −0.14  0.08  −0.18  −0.22  −0.15  −0.66  −0.29  0.18  1.00        TD  −0.17  0.00  0.65  0.51  0.60  0.31  0.04  0.07  −0.09  −0.16  0.17  0.02  1.00      PQ1  0.63  0.73  0.32  0.04  −0.10  0.32  0.27  0.33  0.40  0.23  −0.22  −0.53  −0.03  1.00    PDAC  −0.16  −0.19  −0.30  0.20  0.30  −0.25  −0.22  −0.05  −0.05  −0.18  −0.04  0.28  −0.10  −0.24  1.00  Note: Values in the bold show significant correlations. Table A3. Correlations among variables for the starting grant Indicators  SJRm  SNIPm  h  hnf  NDF  NI  HCD  DIC  PA  PR  PAP  PP  TD  PQ1  PDAC  SJRm  1.00                              SNIPm  0.57  1.00                            h  0.02  0.20  1.00                          hnf  −0.22  0.10  0.79  1.00                        NDF  −0.30  0.09  0.61  0.91  1.00                      NI  0.24  0.11  0.13  −0.01  −0.08  1.00                    HCD  0.19  0.21  0.27  0.11  0.02  0.44  1.00                  DIC  0.09  −0.02  0.10  −0.10  −0.07  −0.10  −0.02  1.00                PA  0.01  −0.03  0.21  0.23  0.15  −0.02  0.08  0.29  1.00              PR  0.44  0.14  0.01  −0.11  −0.23  0.20  0.15  −0.31  −0.28  1.00            PAP  −0.25  0.05  0.21  0.28  0.32  −0.13  −0.14  −0.05  −0.42  −0.19  1.00          PP  −0.27  −0.29  −0.35  −0.27  −0.10  0.00  −0.26  −0.06  −0.43  −0.18  0.03  1.00        TD  −0.22  0.16  0.69  0.73  0.80  −0.09  0.03  −0.02  −0.14  −0.20  0.42  −0.14  1.00      PQ1  0.60  0.63  0.00  −0.13  −0.15  0.17  0.32  −0.08  −0.11  0.36  −0.17  −0.24  −0.10  1.00    PDAC  −0.28  −0.18  −0.23  0.10  0.14  −0.15  −0.19  −0.04  0.04  −0.18  0.06  0.18  −0.14  −0.10  1.00  Indicators  SJRm  SNIPm  h  hnf  NDF  NI  HCD  DIC  PA  PR  PAP  PP  TD  PQ1  PDAC  SJRm  1.00                              SNIPm  0.57  1.00                            h  0.02  0.20  1.00                          hnf  −0.22  0.10  0.79  1.00                        NDF  −0.30  0.09  0.61  0.91  1.00                      NI  0.24  0.11  0.13  −0.01  −0.08  1.00                    HCD  0.19  0.21  0.27  0.11  0.02  0.44  1.00                  DIC  0.09  −0.02  0.10  −0.10  −0.07  −0.10  −0.02  1.00                PA  0.01  −0.03  0.21  0.23  0.15  −0.02  0.08  0.29  1.00              PR  0.44  0.14  0.01  −0.11  −0.23  0.20  0.15  −0.31  −0.28  1.00            PAP  −0.25  0.05  0.21  0.28  0.32  −0.13  −0.14  −0.05  −0.42  −0.19  1.00          PP  −0.27  −0.29  −0.35  −0.27  −0.10  0.00  −0.26  −0.06  −0.43  −0.18  0.03  1.00        TD  −0.22  0.16  0.69  0.73  0.80  −0.09  0.03  −0.02  −0.14  −0.20  0.42  −0.14  1.00      PQ1  0.60  0.63  0.00  −0.13  −0.15  0.17  0.32  −0.08  −0.11  0.36  −0.17  −0.24  −0.10  1.00    PDAC  −0.28  −0.18  −0.23  0.10  0.14  −0.15  −0.19  −0.04  0.04  −0.18  0.06  0.18  −0.14  −0.10  1.00  Note: Values in the bold show significant correlations. Table A3. Correlations among variables for the starting grant Indicators  SJRm  SNIPm  h  hnf  NDF  NI  HCD  DIC  PA  PR  PAP  PP  TD  PQ1  PDAC  SJRm  1.00                              SNIPm  0.57  1.00                            h  0.02  0.20  1.00                          hnf  −0.22  0.10  0.79  1.00                        NDF  −0.30  0.09  0.61  0.91  1.00                      NI  0.24  0.11  0.13  −0.01  −0.08  1.00                    HCD  0.19  0.21  0.27  0.11  0.02  0.44  1.00                  DIC  0.09  −0.02  0.10  −0.10  −0.07  −0.10  −0.02  1.00                PA  0.01  −0.03  0.21  0.23  0.15  −0.02  0.08  0.29  1.00              PR  0.44  0.14  0.01  −0.11  −0.23  0.20  0.15  −0.31  −0.28  1.00            PAP  −0.25  0.05  0.21  0.28  0.32  −0.13  −0.14  −0.05  −0.42  −0.19  1.00          PP  −0.27  −0.29  −0.35  −0.27  −0.10  0.00  −0.26  −0.06  −0.43  −0.18  0.03  1.00        TD  −0.22  0.16  0.69  0.73  0.80  −0.09  0.03  −0.02  −0.14  −0.20  0.42  −0.14  1.00      PQ1  0.60  0.63  0.00  −0.13  −0.15  0.17  0.32  −0.08  −0.11  0.36  −0.17  −0.24  −0.10  1.00    PDAC  −0.28  −0.18  −0.23  0.10  0.14  −0.15  −0.19  −0.04  0.04  −0.18  0.06  0.18  −0.14  −0.10  1.00  Indicators  SJRm  SNIPm  h  hnf  NDF  NI  HCD  DIC  PA  PR  PAP  PP  TD  PQ1  PDAC  SJRm  1.00                              SNIPm  0.57  1.00                            h  0.02  0.20  1.00                          hnf  −0.22  0.10  0.79  1.00                        NDF  −0.30  0.09  0.61  0.91  1.00                      NI  0.24  0.11  0.13  −0.01  −0.08  1.00                    HCD  0.19  0.21  0.27  0.11  0.02  0.44  1.00                  DIC  0.09  −0.02  0.10  −0.10  −0.07  −0.10  −0.02  1.00                PA  0.01  −0.03  0.21  0.23  0.15  −0.02  0.08  0.29  1.00              PR  0.44  0.14  0.01  −0.11  −0.23  0.20  0.15  −0.31  −0.28  1.00            PAP  −0.25  0.05  0.21  0.28  0.32  −0.13  −0.14  −0.05  −0.42  −0.19  1.00          PP  −0.27  −0.29  −0.35  −0.27  −0.10  0.00  −0.26  −0.06  −0.43  −0.18  0.03  1.00        TD  −0.22  0.16  0.69  0.73  0.80  −0.09  0.03  −0.02  −0.14  −0.20  0.42  −0.14  1.00      PQ1  0.60  0.63  0.00  −0.13  −0.15  0.17  0.32  −0.08  −0.11  0.36  −0.17  −0.24  −0.10  1.00    PDAC  −0.28  −0.18  −0.23  0.10  0.14  −0.15  −0.19  −0.04  0.04  −0.18  0.06  0.18  −0.14  −0.10  1.00  Note: Values in the bold show significant correlations. Table A4. CIs for the mean differences (bias-corrected CIs), obtained using bootstrapping, and the effect size (Cohen’s d) in the case of development grants Indicator  Development grant   Engineering and Technology   Natural Sciences   Excat Sciences   Medical and Health Sciences   CI (95%)  Effect size  CI (95%)  Effect size  CI (95%)  Effect size  CI (95%)  Effect size  TD  [−29.45; 2.03]  −0.71  [−4.96; 18.55]  0.43  [−21.56; 20.63]  −0.06  [3.02; 21.39]  1.69  NDF  [−4.24; 6.39]  0.11  [−1.59; 5.62]  0.48  [−7.98, 0.56]  −0.41  [1.75; 5.26]  2.41  PA  [−17.77; 7.68]  −0.26  [−9.24; 10.85]  0.03  [−15.95; 4.76]  −0.35  [−41.50; 0.98]  −1.05  PR  [−2.31; 3.03]  0.10  [−5.92; 0.17]  −0.83  [−2.80; 2.52]  0.00  [−1.27; 35.89]  0.51  PP  [−4.26; 22.39]  0.54  [−9.89; 3.41]  −0.26  [−1.97; 6.17]  0.27  [−3.99; 11.99]  0.86  PAP  [−10.87; 6.43]  −0.23  [−0.71; 9.33]  0.74  [−2.79; 11.58]  0.32  [−20.63; 5.80]  −0.34  PDAC  [−1.33; 30.44]  0.69  [3.07; 35.66]  1.15  [−6.81; 17.35]  0.29  [−16.55; 22.05]  0.14  h  [−9.47; −1.68]  −1.22  [−2.61; 2.95]  0.04  [−5.30; 0.84]  −0.47  [−3.26; 4.91]  0.20  hnf  [−2.45; 0.3]  −0.56  [−0.41; 1.71]  0.51  [−2.67; −0.50]  −0.75  [−0.42; 2.03]  0.72  HCD  [−27.52; −0.41]  −1.09  [−23.61; 8.52]  −0.47  [−14.59; 1.16]  −0.53  [−16.43; 18.15]  0.07  NI  [−0.93; −0.19]  −1.26  [−1.16; −0.13]  −0.96  [−0.40; 0.96]  0.22  [−0.54; 1.18]  0.38  SJRm  [−0.59; −0.03]  −0.78  [−2.11; 0.40]  −0.44  [−0.35; 0.33]  −0.01  [−2.14; −0.41]  −1.07  SNIPm  [−0.35; 0.02]  −0.58  [−0.28; 0.17]  −0.19  [−0.09; 0.15]  0.18  [−0.53; 0.10]  −0.65  PQ1  [−22.00; 8.07]  −0.37  [−19.52; 9.02]  −0.28  [−11.50; 14.89]  0.07  [−41.287; 1.414]  −0.94  DIC  [−34.16; −14.11]  −1.70  [−18.21; 25.36]  0.24  [−8.14; 17.33]  0.20  [−28.69; 4.22]  −0.87  Indicator  Development grant   Engineering and Technology   Natural Sciences   Excat Sciences   Medical and Health Sciences   CI (95%)  Effect size  CI (95%)  Effect size  CI (95%)  Effect size  CI (95%)  Effect size  TD  [−29.45; 2.03]  −0.71  [−4.96; 18.55]  0.43  [−21.56; 20.63]  −0.06  [3.02; 21.39]  1.69  NDF  [−4.24; 6.39]  0.11  [−1.59; 5.62]  0.48  [−7.98, 0.56]  −0.41  [1.75; 5.26]  2.41  PA  [−17.77; 7.68]  −0.26  [−9.24; 10.85]  0.03  [−15.95; 4.76]  −0.35  [−41.50; 0.98]  −1.05  PR  [−2.31; 3.03]  0.10  [−5.92; 0.17]  −0.83  [−2.80; 2.52]  0.00  [−1.27; 35.89]  0.51  PP  [−4.26; 22.39]  0.54  [−9.89; 3.41]  −0.26  [−1.97; 6.17]  0.27  [−3.99; 11.99]  0.86  PAP  [−10.87; 6.43]  −0.23  [−0.71; 9.33]  0.74  [−2.79; 11.58]  0.32  [−20.63; 5.80]  −0.34  PDAC  [−1.33; 30.44]  0.69  [3.07; 35.66]  1.15  [−6.81; 17.35]  0.29  [−16.55; 22.05]  0.14  h  [−9.47; −1.68]  −1.22  [−2.61; 2.95]  0.04  [−5.30; 0.84]  −0.47  [−3.26; 4.91]  0.20  hnf  [−2.45; 0.3]  −0.56  [−0.41; 1.71]  0.51  [−2.67; −0.50]  −0.75  [−0.42; 2.03]  0.72  HCD  [−27.52; −0.41]  −1.09  [−23.61; 8.52]  −0.47  [−14.59; 1.16]  −0.53  [−16.43; 18.15]  0.07  NI  [−0.93; −0.19]  −1.26  [−1.16; −0.13]  −0.96  [−0.40; 0.96]  0.22  [−0.54; 1.18]  0.38  SJRm  [−0.59; −0.03]  −0.78  [−2.11; 0.40]  −0.44  [−0.35; 0.33]  −0.01  [−2.14; −0.41]  −1.07  SNIPm  [−0.35; 0.02]  −0.58  [−0.28; 0.17]  −0.19  [−0.09; 0.15]  0.18  [−0.53; 0.10]  −0.65  PQ1  [−22.00; 8.07]  −0.37  [−19.52; 9.02]  −0.28  [−11.50; 14.89]  0.07  [−41.287; 1.414]  −0.94  DIC  [−34.16; −14.11]  −1.70  [−18.21; 25.36]  0.24  [−8.14; 17.33]  0.20  [−28.69; 4.22]  −0.87  Note: Significance of bold values: p-value<0.05. Table A4. CIs for the mean differences (bias-corrected CIs), obtained using bootstrapping, and the effect size (Cohen’s d) in the case of development grants Indicator  Development grant   Engineering and Technology   Natural Sciences   Excat Sciences   Medical and Health Sciences   CI (95%)  Effect size  CI (95%)  Effect size  CI (95%)  Effect size  CI (95%)  Effect size  TD  [−29.45; 2.03]  −0.71  [−4.96; 18.55]  0.43  [−21.56; 20.63]  −0.06  [3.02; 21.39]  1.69  NDF  [−4.24; 6.39]  0.11  [−1.59; 5.62]  0.48  [−7.98, 0.56]  −0.41  [1.75; 5.26]  2.41  PA  [−17.77; 7.68]  −0.26  [−9.24; 10.85]  0.03  [−15.95; 4.76]  −0.35  [−41.50; 0.98]  −1.05  PR  [−2.31; 3.03]  0.10  [−5.92; 0.17]  −0.83  [−2.80; 2.52]  0.00  [−1.27; 35.89]  0.51  PP  [−4.26; 22.39]  0.54  [−9.89; 3.41]  −0.26  [−1.97; 6.17]  0.27  [−3.99; 11.99]  0.86  PAP  [−10.87; 6.43]  −0.23  [−0.71; 9.33]  0.74  [−2.79; 11.58]  0.32  [−20.63; 5.80]  −0.34  PDAC  [−1.33; 30.44]  0.69  [3.07; 35.66]  1.15  [−6.81; 17.35]  0.29  [−16.55; 22.05]  0.14  h  [−9.47; −1.68]  −1.22  [−2.61; 2.95]  0.04  [−5.30; 0.84]  −0.47  [−3.26; 4.91]  0.20  hnf  [−2.45; 0.3]  −0.56  [−0.41; 1.71]  0.51  [−2.67; −0.50]  −0.75  [−0.42; 2.03]  0.72  HCD  [−27.52; −0.41]  −1.09  [−23.61; 8.52]  −0.47  [−14.59; 1.16]  −0.53  [−16.43; 18.15]  0.07  NI  [−0.93; −0.19]  −1.26  [−1.16; −0.13]  −0.96  [−0.40; 0.96]  0.22  [−0.54; 1.18]  0.38  SJRm  [−0.59; −0.03]  −0.78  [−2.11; 0.40]  −0.44  [−0.35; 0.33]  −0.01  [−2.14; −0.41]  −1.07  SNIPm  [−0.35; 0.02]  −0.58  [−0.28; 0.17]  −0.19  [−0.09; 0.15]  0.18  [−0.53; 0.10]  −0.65  PQ1  [−22.00; 8.07]  −0.37  [−19.52; 9.02]  −0.28  [−11.50; 14.89]  0.07  [−41.287; 1.414]  −0.94  DIC  [−34.16; −14.11]  −1.70  [−18.21; 25.36]  0.24  [−8.14; 17.33]  0.20  [−28.69; 4.22]  −0.87  Indicator  Development grant   Engineering and Technology   Natural Sciences   Excat Sciences   Medical and Health Sciences   CI (95%)  Effect size  CI (95%)  Effect size  CI (95%)  Effect size  CI (95%)  Effect size  TD  [−29.45; 2.03]  −0.71  [−4.96; 18.55]  0.43  [−21.56; 20.63]  −0.06  [3.02; 21.39]  1.69  NDF  [−4.24; 6.39]  0.11  [−1.59; 5.62]  0.48  [−7.98, 0.56]  −0.41  [1.75; 5.26]  2.41  PA  [−17.77; 7.68]  −0.26  [−9.24; 10.85]  0.03  [−15.95; 4.76]  −0.35  [−41.50; 0.98]  −1.05  PR  [−2.31; 3.03]  0.10  [−5.92; 0.17]  −0.83  [−2.80; 2.52]  0.00  [−1.27; 35.89]  0.51  PP  [−4.26; 22.39]  0.54  [−9.89; 3.41]  −0.26  [−1.97; 6.17]  0.27  [−3.99; 11.99]  0.86  PAP  [−10.87; 6.43]  −0.23  [−0.71; 9.33]  0.74  [−2.79; 11.58]  0.32  [−20.63; 5.80]  −0.34  PDAC  [−1.33; 30.44]  0.69  [3.07; 35.66]  1.15  [−6.81; 17.35]  0.29  [−16.55; 22.05]  0.14  h  [−9.47; −1.68]  −1.22  [−2.61; 2.95]  0.04  [−5.30; 0.84]  −0.47  [−3.26; 4.91]  0.20  hnf  [−2.45; 0.3]  −0.56  [−0.41; 1.71]  0.51  [−2.67; −0.50]  −0.75  [−0.42; 2.03]  0.72  HCD  [−27.52; −0.41]  −1.09  [−23.61; 8.52]  −0.47  [−14.59; 1.16]  −0.53  [−16.43; 18.15]  0.07  NI  [−0.93; −0.19]  −1.26  [−1.16; −0.13]  −0.96  [−0.40; 0.96]  0.22  [−0.54; 1.18]  0.38  SJRm  [−0.59; −0.03]  −0.78  [−2.11; 0.40]  −0.44  [−0.35; 0.33]  −0.01  [−2.14; −0.41]  −1.07  SNIPm  [−0.35; 0.02]  −0.58  [−0.28; 0.17]  −0.19  [−0.09; 0.15]  0.18  [−0.53; 0.10]  −0.65  PQ1  [−22.00; 8.07]  −0.37  [−19.52; 9.02]  −0.28  [−11.50; 14.89]  0.07  [−41.287; 1.414]  −0.94  DIC  [−34.16; −14.11]  −1.70  [−18.21; 25.36]  0.24  [−8.14; 17.33]  0.20  [−28.69; 4.22]  −0.87  Note: Significance of bold values: p-value<0.05. Table A5. CIs for the mean differences (bias-corrected CIs), obtained using bootstrapping, and the effect size (Cohen’s d) in the case of starting grants Indicator  Starting grant   Engineering and Technology   Natural Sciences   Excat Sciences   Medical and Health Sciences   CI (95%)  Effect size  CI (95%)  Effect size  CI (95%)  Effect size  CI (95%)  Effect size  TD  [−43.16; −10.61]  −1.60  [−2.41; 23.94]  0.70  [−10.44; 24.03]  0.50  [−19.41; 1.30]  −0.70  NDF  [−6.92; −1.04]  −1.40  [−1.54; 5.47]  0.50  [−0.26; 7.24]  0.14  [−2.21; 0.12]  −0.77  PA  [−23.03; 7.36]  −0.43  [−12.03; 9.20]  −0.04  [5.28; 35.03]  1.30  [−8.61; 27.84]  0.43  PR  [−4.53; 3.85]  −0.10  [−3.63; 8.32]  0.24  [−2.16; 14.15]  0.70  [−13.37; 7.04]  −0.41  PP  [−2.67; 26.51]  0.93  [−4.89; −0.37]  −0.50  [−33.19; 4.16]  −1.00  [−0.84; 3.53]  0.54  PAP  [−15.83; 12.71]  −0.19  [−3.26; 9.24]  0.29  [−15.74; 19.85]  0.20  [−5.91; 2.96]  −0.23  PDAC  [−13.56; 20.02]  0.17  [−26.41; 16.89]  −0.18  [−19.24; 17.13]  −0.10  [−4.33; 24.33]  0.56  h  [−12.03; −1.62]  −1.30  [−0.31; 4.62]  0.70  [0.84; 8.47]  1.20  [−5.61; 1.36]  −0.50  hnf  [−3.66; −0.71]  −1.40  [−0.05; 2.80]  0.90  [0.23; 4.29]  1.20  [−1.37; 0.73]  −0.32  HCD  [−22.87; 7.07]  −0.50  [−8.95; 18.17]  0.24  [−3.73; 22.88]  1.00  [−42.79; 1.37]  −0.79  NI  [−0.65; 0.34]  −0.25  [−1.42; 0.40]  −0.40  [−0.52; 1.56]  0.70  [−0.65; 0.18]  −0.49  SJRm  [−0.39; 0.03]  −0.76  [−0.60; 0.35]  −0.15  [−0.67; 1.20]  0.51  [−1.34; −0.17]  −1.19  SNIPm  [−0.52; 0.11]  −0.59  [−0.14; 0.23]  0.00  [−0.24; 0.38]  0.20  [−0.37; −0.06]  −1.13  PQ1  [−15.87; 19.89]  0.10  [−22.05; 13.93]  −0.19  [−22.00; 32.76]  0.30  [−24.94; 20.80]  −0.05  DIC  [−35.78; 10.95]  −0.62  [−33.60; −4.36]  −0.19  [−36.00; 6.38]  −0.70  [−25.11; 17.27]  −0.19  Indicator  Starting grant   Engineering and Technology   Natural Sciences   Excat Sciences   Medical and Health Sciences   CI (95%)  Effect size  CI (95%)  Effect size  CI (95%)  Effect size  CI (95%)  Effect size  TD  [−43.16; −10.61]  −1.60  [−2.41; 23.94]  0.70  [−10.44; 24.03]  0.50  [−19.41; 1.30]  −0.70  NDF  [−6.92; −1.04]  −1.40  [−1.54; 5.47]  0.50  [−0.26; 7.24]  0.14  [−2.21; 0.12]  −0.77  PA  [−23.03; 7.36]  −0.43  [−12.03; 9.20]  −0.04  [5.28; 35.03]  1.30  [−8.61; 27.84]  0.43  PR  [−4.53; 3.85]  −0.10  [−3.63; 8.32]  0.24  [−2.16; 14.15]  0.70  [−13.37; 7.04]  −0.41  PP  [−2.67; 26.51]  0.93  [−4.89; −0.37]  −0.50  [−33.19; 4.16]  −1.00  [−0.84; 3.53]  0.54  PAP  [−15.83; 12.71]  −0.19  [−3.26; 9.24]  0.29  [−15.74; 19.85]  0.20  [−5.91; 2.96]  −0.23  PDAC  [−13.56; 20.02]  0.17  [−26.41; 16.89]  −0.18  [−19.24; 17.13]  −0.10  [−4.33; 24.33]  0.56  h  [−12.03; −1.62]  −1.30  [−0.31; 4.62]  0.70  [0.84; 8.47]  1.20  [−5.61; 1.36]  −0.50  hnf  [−3.66; −0.71]  −1.40  [−0.05; 2.80]  0.90  [0.23; 4.29]  1.20  [−1.37; 0.73]  −0.32  HCD  [−22.87; 7.07]  −0.50  [−8.95; 18.17]  0.24  [−3.73; 22.88]  1.00  [−42.79; 1.37]  −0.79  NI  [−0.65; 0.34]  −0.25  [−1.42; 0.40]  −0.40  [−0.52; 1.56]  0.70  [−0.65; 0.18]  −0.49  SJRm  [−0.39; 0.03]  −0.76  [−0.60; 0.35]  −0.15  [−0.67; 1.20]  0.51  [−1.34; −0.17]  −1.19  SNIPm  [−0.52; 0.11]  −0.59  [−0.14; 0.23]  0.00  [−0.24; 0.38]  0.20  [−0.37; −0.06]  −1.13  PQ1  [−15.87; 19.89]  0.10  [−22.05; 13.93]  −0.19  [−22.00; 32.76]  0.30  [−24.94; 20.80]  −0.05  DIC  [−35.78; 10.95]  −0.62  [−33.60; −4.36]  −0.19  [−36.00; 6.38]  −0.70  [−25.11; 17.27]  −0.19  Note: Significance of bold values: p-value<0.05. Table A5. CIs for the mean differences (bias-corrected CIs), obtained using bootstrapping, and the effect size (Cohen’s d) in the case of starting grants Indicator  Starting grant   Engineering and Technology   Natural Sciences   Excat Sciences   Medical and Health Sciences   CI (95%)  Effect size  CI (95%)  Effect size  CI (95%)  Effect size  CI (95%)  Effect size  TD  [−43.16; −10.61]  −1.60  [−2.41; 23.94]  0.70  [−10.44; 24.03]  0.50  [−19.41; 1.30]  −0.70  NDF  [−6.92; −1.04]  −1.40  [−1.54; 5.47]  0.50  [−0.26; 7.24]  0.14  [−2.21; 0.12]  −0.77  PA  [−23.03; 7.36]  −0.43  [−12.03; 9.20]  −0.04  [5.28; 35.03]  1.30  [−8.61; 27.84]  0.43  PR  [−4.53; 3.85]  −0.10  [−3.63; 8.32]  0.24  [−2.16; 14.15]  0.70  [−13.37; 7.04]  −0.41  PP  [−2.67; 26.51]  0.93  [−4.89; −0.37]  −0.50  [−33.19; 4.16]  −1.00  [−0.84; 3.53]  0.54  PAP  [−15.83; 12.71]  −0.19  [−3.26; 9.24]  0.29  [−15.74; 19.85]  0.20  [−5.91; 2.96]  −0.23  PDAC  [−13.56; 20.02]  0.17  [−26.41; 16.89]  −0.18  [−19.24; 17.13]  −0.10  [−4.33; 24.33]  0.56  h  [−12.03; −1.62]  −1.30  [−0.31; 4.62]  0.70  [0.84; 8.47]  1.20  [−5.61; 1.36]  −0.50  hnf  [−3.66; −0.71]  −1.40  [−0.05; 2.80]  0.90  [0.23; 4.29]  1.20  [−1.37; 0.73]  −0.32  HCD  [−22.87; 7.07]  −0.50  [−8.95; 18.17]  0.24  [−3.73; 22.88]  1.00  [−42.79; 1.37]  −0.79  NI  [−0.65; 0.34]  −0.25  [−1.42; 0.40]  −0.40  [−0.52; 1.56]  0.70  [−0.65; 0.18]  −0.49  SJRm  [−0.39; 0.03]  −0.76  [−0.60; 0.35]  −0.15  [−0.67; 1.20]  0.51  [−1.34; −0.17]  −1.19  SNIPm  [−0.52; 0.11]  −0.59  [−0.14; 0.23]  0.00  [−0.24; 0.38]  0.20  [−0.37; −0.06]  −1.13  PQ1  [−15.87; 19.89]  0.10  [−22.05; 13.93]  −0.19  [−22.00; 32.76]  0.30  [−24.94; 20.80]  −0.05  DIC  [−35.78; 10.95]  −0.62  [−33.60; −4.36]  −0.19  [−36.00; 6.38]  −0.70  [−25.11; 17.27]  −0.19  Indicator  Starting grant   Engineering and Technology   Natural Sciences   Excat Sciences   Medical and Health Sciences   CI (95%)  Effect size  CI (95%)  Effect size  CI (95%)  Effect size  CI (95%)  Effect size  TD  [−43.16; −10.61]  −1.60  [−2.41; 23.94]  0.70  [−10.44; 24.03]  0.50  [−19.41; 1.30]  −0.70  NDF  [−6.92; −1.04]  −1.40  [−1.54; 5.47]  0.50  [−0.26; 7.24]  0.14  [−2.21; 0.12]  −0.77  PA  [−23.03; 7.36]  −0.43  [−12.03; 9.20]  −0.04  [5.28; 35.03]  1.30  [−8.61; 27.84]  0.43  PR  [−4.53; 3.85]  −0.10  [−3.63; 8.32]  0.24  [−2.16; 14.15]  0.70  [−13.37; 7.04]  −0.41  PP  [−2.67; 26.51]  0.93  [−4.89; −0.37]  −0.50  [−33.19; 4.16]  −1.00  [−0.84; 3.53]  0.54  PAP  [−15.83; 12.71]  −0.19  [−3.26; 9.24]  0.29  [−15.74; 19.85]  0.20  [−5.91; 2.96]  −0.23  PDAC  [−13.56; 20.02]  0.17  [−26.41; 16.89]  −0.18  [−19.24; 17.13]  −0.10  [−4.33; 24.33]  0.56  h  [−12.03; −1.62]  −1.30  [−0.31; 4.62]  0.70  [0.84; 8.47]  1.20  [−5.61; 1.36]  −0.50  hnf  [−3.66; −0.71]  −1.40  [−0.05; 2.80]  0.90  [0.23; 4.29]  1.20  [−1.37; 0.73]  −0.32  HCD  [−22.87; 7.07]  −0.50  [−8.95; 18.17]  0.24  [−3.73; 22.88]  1.00  [−42.79; 1.37]  −0.79  NI  [−0.65; 0.34]  −0.25  [−1.42; 0.40]  −0.40  [−0.52; 1.56]  0.70  [−0.65; 0.18]  −0.49  SJRm  [−0.39; 0.03]  −0.76  [−0.60; 0.35]  −0.15  [−0.67; 1.20]  0.51  [−1.34; −0.17]  −1.19  SNIPm  [−0.52; 0.11]  −0.59  [−0.14; 0.23]  0.00  [−0.24; 0.38]  0.20  [−0.37; −0.06]  −1.13  PQ1  [−15.87; 19.89]  0.10  [−22.05; 13.93]  −0.19  [−22.00; 32.76]  0.30  [−24.94; 20.80]  −0.05  DIC  [−35.78; 10.95]  −0.62  [−33.60; −4.36]  −0.19  [−36.00; 6.38]  −0.70  [−25.11; 17.27]  −0.19  Note: Significance of bold values: p-value<0.05. Table A6. Measures of fit for the obtained models Grant type  Sensitivity (%)  Specificity (%)  False positive  False negative  Prediction  AUC  Development  71.4  65.0  35.0  28.6  68.8  73.6 [61.58; 82.36]  Starting  79.5  55.6  44.4  20.5  69.7  66.2 [50.8; 79.4]  Grant type  Sensitivity (%)  Specificity (%)  False positive  False negative  Prediction  AUC  Development  71.4  65.0  35.0  28.6  68.8  73.6 [61.58; 82.36]  Starting  79.5  55.6  44.4  20.5  69.7  66.2 [50.8; 79.4]  Note: In brackets the CIs (95%) for the AUC. Table A6. Measures of fit for the obtained models Grant type  Sensitivity (%)  Specificity (%)  False positive  False negative  Prediction  AUC  Development  71.4  65.0  35.0  28.6  68.8  73.6 [61.58; 82.36]  Starting  79.5  55.6  44.4  20.5  69.7  66.2 [50.8; 79.4]  Grant type  Sensitivity (%)  Specificity (%)  False positive  False negative  Prediction  AUC  Development  71.4  65.0  35.0  28.6  68.8  73.6 [61.58; 82.36]  Starting  79.5  55.6  44.4  20.5  69.7  66.2 [50.8; 79.4]  Note: In brackets the CIs (95%) for the AUC. Figure A1. View largeDownload slide Curve ROC for the development grant (a) and starting grant (b). Figure A1. View largeDownload slide Curve ROC for the development grant (a) and starting grant (b). © The Author(s) 2018. Published by Oxford University Press. All rights reserved. For permissions, please email: journals.permissions@oup.com This article is published and distributed under the terms of the Oxford University Press, Standard Journals Publication Model (https://academic.oup.com/journals/pages/about_us/legal/notices)

Journal

Research EvaluationOxford University Press

Published: Apr 3, 2018

There are no references for this article.

You’re reading a free preview. Subscribe to read the entire article.


DeepDyve is your
personal research library

It’s your single place to instantly
discover and read the research
that matters to you.

Enjoy affordable access to
over 18 million articles from more than
15,000 peer-reviewed journals.

All for just $49/month

Explore the DeepDyve Library

Search

Query the DeepDyve database, plus search all of PubMed and Google Scholar seamlessly

Organize

Save any article or search result from DeepDyve, PubMed, and Google Scholar... all in one place.

Access

Get unlimited, online access to over 18 million full-text articles from more than 15,000 scientific journals.</