On the meaning of innovation performance: Is the synthetic indicator of the Innovation Union Scoreboard flawed?

On the meaning of innovation performance: Is the synthetic indicator of the Innovation Union... Abstract The European Union (EU) annually publishes an Innovation Union Scoreboard (IUS) as a tool to measure the innovation performance of EU Member States by means of a composite index, called the Summary Innovation Index (SII). The SII is constituted by an average of 25 indicators. The SII is claimed to rank Member States according to their innovation performance. This means that the higher the average value of the 25 indicators, the better the innovation performance is said to be. The first purpose of this article is to assess whether the SII constitutes a meaningful measure of innovation performance. Our conclusion is that it does not. Our second purpose is to develop alternative, productivity or efficiency-based, measures of innovation system performance based on a simple index number, and complement it with advanced and robust nonparametric Data Envelopment Analysis techniques. By doing so, the article offers a critical review of the SII, and proposes to put more emphasis on the identification of and relation between input and output innovation indicators. The data provided by the 2014 and 2015 editions of the IUS are here used to analyze the innovation performance of all 28 EU national innovation systems. A theoretical background and reasons for selecting the indicators used are presented, and our new ranking of the innovation performance using bias-corrected efficiency scores of all EU countries is calculated. We find that the results differ substantially between the SII and the ranking based on our method, with significant consequences for the design of innovation policies. 1. Introduction The European Union (EU) has set ambitious objectives in five areas to be reached by 2020. In addition to climate and energy, education, employment, and social inclusion, innovation is one of the five pillars to form a so-called ‘Innovation Union’ (European Commission 2013). To support the establishment of the Innovation Union, the European Commission is using the Innovation Union Scoreboard (IUS). The objective of the IUS is to provide a ‘comparative assessment’ of the ‘innovation performance’ of EU Member States (European Commission 2011: 3). To assess the ‘innovation performance’ of the Member States, a Summary Innovation Index (SII) is calculated by the IUS. The SII synthetizes the 25 indicators included in the IUS, through their arithmetic average, into a single measure. As a result, the SII ranks all EU Member States based on what is explicitly called ‘EU Member States’ Innovation Performance’ (European Union 2014: 5). The first aim of this article is to assess whether the SII constitutes a meaningful measure of innovation performance, as argued by the IUS. Following Zabala-Iturriagagoitia et al. (2007a) or Nasierowski and Arcelus (2012), among others, the second aim is to develop alternative, productivity or efficiency-based, measures of innovation performance. By doing so, we provide a critical review of the SII, propose to place more emphasis on the identification of and relation between innovation input and output indicators, and on this basis we provide an alternative assessment and ranking of innovation performance. When developing an alternative approach to the one used by IUS, we use exclusively and exactly the same data as provided by the IUS 2014 and 2015 to assess and compare the performance of all EU28 national innovation systems. We reexamine these data using our different approach, which relies on a productivity—or relative efficiency—rationale. We do not discuss the quality and accuracy of the IUS data. We simply use the existing data provided by the IUS but seen through a different (productivity) lens, with the purpose of increasing the breadth of sight of innovation policy makers and politicians.1 A similar efficiency approach has been used by the Global Innovation Index (GII). The ‘Innovation Efficiency Ratio’ provided by the GII is ‘the ratio of the Output Sub-Index score over the Input Sub-Index score. It shows how much innovation output a given country is getting for its inputs’ (Dutta et al. 2017). The GII aggregates 81 different indicators for 143 countries, using different subindices to aggregate them. Following the theoretical bases used in productivity and efficiency measurement methods (Farrell 1957; Coelli et al. 2005), we single out a number of input (N = 4) and output (M = 8) innovation indicators from the 25 included in the IUS editions of 2014 and 2015. They are used to compare the innovation outputs with the innovation inputs of each of the EU28 countries, so as to get a measure of the performance of the innovation systems (i.e. the relationship between the innovation inputs and outputs). We conclude that the SII does not constitute a meaningful measure of innovation performance and that this indicator is not useful from the point of view of innovation policy design. Our alternative approach results in a radically different ranking of EU Member States’ innovation performance. Our performance and ranking results, however, lead to counterintuitive conclusions, which we discuss and evaluate. This means that the analysis has to be refined and developed in future research. The rest of the article is organized as follows. Section 2 discusses the concept of innovation performance and presents the rationale for the approach we have chosen. In Section 3 we present the index number and DEA methods and discuss the selection of output and input indicators. The empirical analysis is presented in Section 4. There, we use the normalized IUS scores for each of the selected indicators, provide the rankings for both innovation inputs and outputs, and calculate the efficiency of all 28 innovation systems by relating the innovation outputs and inputs to each other through the index number and robust DEA scores. Section 5 provides a discussion of the main findings and its relevance for the practice of innovation policy making, and also makes some proposals for future research. 2. Theoretical background During the past decade an increasing literature has dealt with the development and use of indicators to improve the measurement and characterization of innovation systems (e.g. Archibugi, Denni, and Filippetti 2009). The European Commission has been one of the most active agents in this sense, with the development of the European Innovation Scoreboard (since 2011, IUS) and the implementation of the Community Innovation Surveys. As indicated in Section 1, the IUS claims that the SII calculated is measuring innovation performance of EU Member States. But what is then meant by ‘innovation performance’? Let us discuss this concept by a metaphor. Let us imagine that two countries are trying to send a rocket to the moon, and both succeed. However, the first required $1 billion to achieve that goal, while the second required $1. If only outputs are considered, both countries would have achieved the same level of ‘performance’. However, it can be strongly argued that the second country might have achieved a remarkably higher performance, since the amount of resources used in the two cases is so unalike. This approach is also shared by Carayannis and Grigoroudis (2014) and Liu, Lu, and Ho (2014: 318) who consider that the measurement of the performance of an innovation system should be accomplished through ‘a linear process-oriented approach, whereby a nation’s innovation system is treated as a process in which certain output factors… are seen as products produced from a set of input resources’. This means that if we are to compare performance in a meaningful way the achievement (e.g. landing on the moon) must be related to the resources used (e.g. a budget constraint). In other words, outputs must be related to inputs in some way. In economics, productivity is defined as the value of output produced per unit of input, e.g. labor productivity is total production divided by employment. This is similar to measuring performance of innovation systems in terms of productivity or efficiency. From the policy evaluation literature we have learned that (innovation) policies must be context specific. These policies will define the goals of the innovation policy, identify the problems in the innovation system, and specify or develop and implement the instruments to mitigate the problems and thereby achieve the objectives (Edquist 2018). In other words, the policy must define where to go and how to get there. However, if we do not know where we are, how can we get anywhere? Indeed, it is in fact very difficult to improve what cannot be measured. It is not possible to say whether certain innovation intensities are high or low in a specific system if there is no comparison with those in other systems (Edquist 2011). This has to do with the fact that we cannot identify optimal or ideal innovation intensities (just as we cannot specify an optimal innovation system). There appears to be general support ‘to the premise that all performance evaluations involve comparisons’ (Mersha 1989: 163). Hence the measurement of innovation performance must be comparative between existing systems (Zabala-Iturriagagoitia et al. 2007b). Only then can they be helpful to policy design by identifying problems to be solved or mitigated by the implementation of policy instruments. In spite of the flags often raised in the literature as to the misuse of synthetic indicators in policy making spheres (e.g. Grupp and Schubert 2010), composite indicators are repeatedly used for the design of innovation policies (Saltelli 2007). One of the risks of using synthetic indicators is that, once they are used and accepted, their abuse makes them become policy targets, and hence, loose their meaning as a tool to explain certain realities. Saltelli refers to the work by Nardo et al. (2005) when discussing the steps required before creating a composite indicator.2 The SII falls short in meeting several of those steps. We claim that to assess the innovation performance of any system, the performance measure should be based on a productivity relation, defined as a ratio of aggregate innovation outputs (numerator) to aggregate innovation inputs (denominator). The most simple way of doing so is to classify indicators as either inputs or outputs, combine them using an aggregator function (e.g. the arithmetic average), and then relate one to another by using a simple fraction. A more advanced method relies on robust nonparametric data envelopment analysis (DEA) techniques to calculate a productivity index for each country using individual—most favorable—aggregating functions, which also allow accounting for the nature of returns to scale (increasing, constant, or decreasing). In this article we use both methods. An important difference between the simple aggregate productivity index and its DEA associate is that the former, being a basic index number, can be easily calculated with low effort and high immediacy (e.g. using a spreadsheet), while the latter relays on linear programing techniques that require solver optimization with bootstrapping. But, in return, the latter yields relevant information about the innovation processes that are evaluated (benchmark peers for inefficient observations, nature of returns to scale, etc.) as well as the statistical significance of the productivity estimates. 3. Methodology and relevant indicators The IUS identifies ιl, l = 1,…, 25, indicators, which are divided into three categories and eight dimensions (see Fig. 1). The three categories considered are Enablers, Firm activities, and Outputs. According to the IUS, the enablers ‘capture the main drivers of innovation performance external to the firm’ (European Union 2014: 4) and cover three innovation dimensions: human resources, open, excellent and attractive research systems, and finance and support. Firm activities ‘capture the innovation efforts at the level of the firm’ (ibid) and are also grouped in three innovation dimensions: firm investments, linkages and entrepreneurship, and intellectual assets. Finally, outputs cover ‘the effects of firms’ innovation activities’ (ibid) in two innovation dimensions: innovators and economic effects. Figure 1. View largeDownload slide Measurement framework of the IUS. Source:European Union (2014: 8). Figure 1. View largeDownload slide Measurement framework of the IUS. Source:European Union (2014: 8). From a numerical perspective, to adjust the values of each indicator observed in country i in period t to a notionally common scale, the IUS normalizes each series so their values belong to the following range: ι^lit∈[0,1].3 Subsequently, the IUS calculates the SII using the arithmetic mean as aggregating function, in which all indicators are given the same weight υl: SIIit=∑l=125υ¯lι^lit, υ¯l=1/25, ι^lit∈[0,1]. (1) Note that the SIIit index does not necessarily yield a distribution whose maximum value is equal to 1, thereby identifying the most productive innovation system with such value. To ensure this desirable property that is helpful to highlight best performance, and rank innovation systems against a single scalar—and that is shared with the forthcoming productivity indices, we normalize SIIit by dividing by the maximum observed value across the set of j = 1,…, J countries: SIÎit=∑l=125υ¯lι^litmax j(SIIjt), υ¯l=1/25, j=1,…J, SIÎit∈[0,1]. (2) The IUS draws the conclusion that the country with the highest SIIit―or SIÎit―is also the best innovation performer regardless of whether the indicators used measure the input or output side of innovation or something else. In addition, the IUS provides no explicit definition of innovation performance, which is quite surprising, since this is the most central concept in the scoreboard reports. However, implicitly and in practice, the SII score can be said to be the IUS definition of innovation performance. On this basis, EU Member States are ranked according to the SII in the following groups: innovation leaders (more than 20% above EU average), innovation followers (less than 20% above, or more than 90% of the EU average), moderate innovators (relative performance rates between 50 and 90% of the EU average), and modest innovators (less than 50% of the EU average) (European Union 2014: 11). 3.1 Identification of input and output indicators We have argued that innovation performance must be measured as a ratio between a numerator and a denominator. Our way of doing so is to classify indicators as either inputs or outputs and then relate them to one another. We will now discern which of the 25 IUS indicators in Fig. 1 can be qualified as inputs, as outputs, or as something else (e.g. indicators measuring intermediate phenomena, or consequences of innovations). Following the guidelines provided by the Oslo Manual (OECD/Eurostat 2005), we define input and output indicators as follows: Innovation input indicatorsrefer to the resources (human, material, and financial; private as well as governmental) used not only to create innovations but also to bring them to the market. Innovation output indicatorsrefer to new products and processes, new designs, and community trademarks, as well as marketing and organizational innovations, which are connected to the market, and which can either be new to the world, new to the industry, and/or new to the firm. We classify eight IUS indicators as measuring innovation output and four as measuring innovation input. The remaining 13 indicators from the IUS are certainly not irrelevant. In fact, many of these indicators can be regarded as determinants of innovation processes. However, further research is needed on the factors influencing (fostering or hindering) the development and the diffusion of innovations (see Edquist 2005). When more evidence is available about these determinants, the view on what indicators can be regarded as inputs or outputs will certainly have to be adapted. Below we provide our reasons to include some indicators and not others in these categories. Below we list the eight output indicators considered in this article.4 2.2.1: Small and medium-sized enterprises (SMEs) innovating in-house (% of SMEs) 2.3.3: Community trademarks per billion gross domestic product (GDP) (in Purchasing Parity Power (PPP)€) 2.3.4: Community designs per billion GDP (in PPP€) 3.1.1: SMEs introducing product or process innovations (% of SMEs) 3.1.2: SMEs introducing marketing or organizational innovations (% of SMEs) 3.2.2: Contribution of medium and high-tech products exports to the trade balance 3.2.3: Knowledge-intensive services exports (as % of total service exports) 3.2.4: Sales of new-to-market and new-to-firm innovations (as % of turnover) The notion of innovation output, according to our definition, is partly different than the IUS category of ‘outputs’, which is defined as ‘the effects of firms’ innovation activities’ (European Union 2014: 4).5 We contend that indicators 2.2.1, 2.3.3, and 2.3.4 should be categorized as output indicators, despite they are classified in the IUS as ‘firm activities’ rather than ‘outputs’. For example, the indicator 2.2.1, ‘SMEs innovating in-house’, refers to the SMEs that have succeeded in the introduction of new or significantly improved products and/or processes, and which may have been innovated inside the company. Therefore, this indicator is in fact capturing an output of an innovation system. Similar arguments hold for indicators 2.3.3 ‘Community trademarks per billion GDP’ and 2.3.4 ‘Community designs per billion GDP’. We consider that community trademarks and designs are significant aspects of actual product innovations, as they signal intellectual property rights related to a specific new product that is connected to the market (see also Mendonça, Santos Pereira, and Mira Godinho 2004). Indicators 3.1.1 through 3.2.5 are considered to be ‘outputs’ by the IUS. However, a conceptual difference exists between the label ‘outputs’ as used in the IUS and the concept of innovation output used in this article. It is for this reason that we do not classify as innovation outputs the following three indicators referred to in the IUS as ‘outputs’: 3.1.3, ‘Employment in fast-growing firms of innovative sectors’; 3.2.1, ‘Employment in knowledge-intensive activities’; and 3.2.5, ’License and patent revenues from abroad’. Indicators 3.1.3 and 3.2.1 measure employment. Employment may be a consequence of innovations or the result of other economic forces rather than an innovation as such—just as with economic growth. As to the indicator 3.2.5, licenses and patent revenues, it refers to sales of intellectual property rights, and cannot be considered an indicator of innovation output according to the previous definition.6 Classifications of individual indicators in a certain category (outputs here) can certainly be questioned and discussed. We therefore carried out a sensitivity analysis, which is reported in Section 4.1, in which we classified 12 indicators instead of 8 as innovation outputs.7 Looking further at the measurement framework of the IUS (Fig. 1), it becomes clear that while one of the main categories is considered to be a measure of innovation Outputs, there is no category explicitly referring to innovation inputs, or providing a clear specification of what such inputs could be. We consider four indicators to fulfill the requirements for being classified as innovation inputs—see below. Two of these indicators are categorized in the IUS as ‘enablers’ and two as ‘firm activities’. Two measure R&D expenditures from the public and private sector, which are both important determinants of innovation.8 Venture capital, which is important ‘for the relative dynamism of new business creation’ (European Union 2014: 87), is especially needed for risk and cost-intensive innovation and is also required to enhance innovation through commercialization of R&D results. In addition to R&D-intensive investments, companies need to invest in non-R&D innovation expenditures as well. Below we list the four input indicators considered. 1.3.1: R&D expenditure in the public sector (% of GDP) 1.3.2: Venture capital (% of GDP) 2.1.1: R&D expenditure in the business sector (% of GDP) 2.1.2: Non-R&D innovation expenditures (% of turnover) We carried out a sensitivity analysis also for the inputs (see Section 4.2), in which we classified seven indicators instead of four as input indicators.9 The four input indicators proposed above are linked to innovation activities and are undertaken to enhance innovation, at least in part, as the Oslo Manual highlights. There are, of course, other determinants of innovation processes. Ideally we should include all such determinants as input indicators. However, we would then need a fully articulated systemic and holistic approach in which all determinants of innovation processes, the feedback loops among them, and in which the relative importance of the different determinants of innovation processes are accounted for (Edquist 2014a, 2014c, 2018). That we do not have (yet). Admittedly, this is unsatisfactory, but a fact. For example, in the IUS (and in this article) no account is taken of determinants of innovation processes operating from the demand side (see Edquist et al. 2015). If all innovation input and all innovation output indicators were included, we would be able to calculate something corresponding to total factor productivity (or multifactor productivity) in the field of innovation. As indicated, we will be satisfied here with a limited number of indicators on both sides (i.e. we will only be able to provide a partial measure of innovation performance). 3.2 Productivity measures based on simple index numbers and robust DEA Once the indicators that we deem most relevant for the purposes of this article have been selected, we gather all the data from the IUS 2014 and 2015, to calculate both the conventional and normalized productivity measures for all EU28 countries. As a subset of the previous ι^lit normalized indicators in (1), and following the preceding discussion, we denote innovation input indicators of country i in period t by x^it= (x^1it,…,x^Nit) ∈ R+N, while the innovation outputs indicators are represented by the vector y^it= (y^1it,…,y^Mit) ∈ R+M—in the selected model, N = 4 and M = 8. Relying on the simplest index number definition, the Productivity Innovation Index (PII) is calculated as the relation (i.e. ratio) between the normalized input and output indicators― y^mit∈[0,1], x^nit∈[0,1], using once again the arithmetic mean as aggregating function; i.e. PIIit=∑m=1Mμ¯my^mit∑n=1Nν¯nx^nit, μ¯m=1/M, ν¯n=1/N,  (3) where ν¯n and μ¯m are the input and output weights. As its SIIit counterpart, (3) must be normalized by the maximum observed value across all J countries, so its value is bounded above by 1. PIÎit=PIIitmax j(PIIjt), j=1,…,J,  PIÎit∈[0,1]. (4) When compared to (2), a high score for the input indicators in the denominator means that a great deal of effort and resources has been devoted to stimulating innovation. Similarly, a high score for the output indicators in the numerator shows a high production of innovations. However, if the input side is, relatively speaking, much larger than the output side, the performance of the system as a whole is low (i.e. note the moon project). This implies that the efforts to produce or stimulate innovation have not led to a corresponding actual production of innovations. The different interpretations of SIÎit in (2) as an average measure of indiscriminate innovation indicators and the productivity measure PIÎit above, should now be clear. The properties of PIÎit can be easily determined by resorting to index number theory. Balk (2008) discusses the desirable properties that a productivity index should fulfill from the so-called axiomatic or test approach (monotonicity, homogeneity, identity, proportionality, etc.). Nevertheless, we are aware of the potential limits that using a deterministic formulation like (4) entails for the evaluation of innovation performance. Potential problems are the existence of sampling variation and measurement errors that may polarize results and bias the productivity values and associated rankings. For this reason we propose the use of an alternative approach that allows addressing the bias emanating from the above issues. This approach extends with bootstrapping statistical techniques the nonparametric DEA methods for assessing efficiency already used by Zabala-Iturriagagoitia et al. (2007a). Bootstrapped DEA constitutes a sophisticated method to estimate innovation productivity, differing from the simple index number aggregate in a subtle and fundamental way. Through mathematical programming, in each run the input and output weights νn and μm are now the result of the optimization process that maximizes a country’s productivity relative to its counterparts. We can rely on the seminal contribution by Charnes, Cooper, and Rhodes (1978) to illustrate the productivity interpretation of DEA. Indeed, based on the normalized indicators, their original ratio-form formulation, known as CCR, computes the productivity of country’s i innovation system in the following way:10 max vmt,μmt ∑m=1Mμmty^mit∑n=1Nνntx^nit=EIÎit∈(0,1], (5) s.t.  ∑m=1Mμmty^mjt∑n=1Nνntx^njt≤1, j=1,…,J,μmt≥0, νnt≥0, where  νnt* and μmt* denote the optimal input and output weights when evaluating the relative productivity with respect to all j = 1,…, J countries, including itself. Contrary to the equal weights formulations (2) and (3), program (5) identifies the most favorable inputs and outputs weights that result in the maximum feasible productivity level of ( xit, yit) relative to that of the remaining innovation systems. Note that since the weights  νnt and μmt constitute aggregator functions, both the objective function and set of j = 1,…, J restrictions represent proper definitions of productivity—a ratio of aggregate output to aggregate input, normalizing maximum productivity to 1. This normalization also allows to deem the solution to (5) as a relative (in)efficiency measure, justifying the efficiency innovation index notation: EIÎit. When country i under evaluation maximizes relative productivity or efficiency, EIÎit = 1. On the contrary, if EIÎit < 1, the country does not maximize the relative productivity of its innovation system, and the lower the efficiency value, the worse the innovation performance, as in the previous indicators. The productivity values calculated from (5) can be considered as estimates, since their values are subject to uncertainty due to sampling variation, measurement errors, etc. Simar and Wilson (1998) introduce bootstrap methods that, based on resampling, provide estimation bias, confidence intervals and allow hypotheses testing. We rely on the algorithms presented by these authors, following Daraio and Simar (2007) and Bogetoft and Otto (2011), to test the statistical significance of the attained results. The bootstrap algorithm can be summarized in the following steps: Selection of B independent bootstrap samples—drawn from the original data set with replacements (i.e. 2000); Calculate an initial estimate for the productivity or efficiency score of each country EIÎitwith respect to each bootstrapped sample and smooth their distribution by perturbating them with a random noise generated from a kernel density function with scale given with bandwith h; Correct the original estimates for the mean and variance of the smoothed values; Obtain a second set of bootstrapped samples generating inefficient countries inside the DEA attainable set and conditional on the original input—or output—mix; Repeat the process and estimate the efficiency scores for each original country with respect to that second set, so as to obtain a set of B bootstrap estimates; and, finally, Based on this distribution calculate the threshold values that truncate it according to the predetermined significance value α (i.e. 0.05), so as to determine the confidence intervals for the efficiency score of each country. In addition, the bootstrapped scores can be used to obtain an estimate of the bias of the true efficiency value, and thereby a bias-corrected estimator, which we denote by bEIÎit. In the empirical section the bias-corrected productivity values are presented along with their corresponding confidence intervals. 4. Empirical analysis This section assesses the innovation performance of the Swedish national innovation system in relation to the rest of the EU28 Member States. However, what follows in this section is just an example of how our approach can be developed with regard to one country. 4.1 Output orientation Table 1 summarizes the normalized scores for the eight output indicators considered, and the relative position that Sweden holds in relation to the EU28 countries for the latest year for which data are available. It also gives an average ranking and normalized score for Sweden for all output indicators. Table 1. The innovation output indicators of the Swedish national innovation system Indicator Score in 2014 [2015] Ranking (out of 28) in 2014 Ranking (out of 28) in 2015 EU 28 average in 2014 [2015] Leading countries (top 3) in 2014 Leading countries (top 3) in 2015 2.2.1 SMEs innovating in-house as % of SMEs 0.729 [0.779] 8 4 0.570 [0.513] Germany (0.933) Cyprus (0.833) Denmark (0.813) The Netherlands (0.797) Ireland (0.792) Germany (0.787) 2.3.3 Community trademarks per billion GDP (in PPP-€) 0.573 [0.661] 7 8 0.444 [0.580] Luxembourg (1.0) Cyprus (1.0) Malta (1.0) Cyprus (1.000) Luxembourg (1.000) Malta (1.000) 2.3.4 Community designs per billion GDP (in PPP-€) 0.574 [0.999] 8 3 0.566 [0.569] Luxembourg (1.0) Austria (1.0) Denmark (0.971) Denmark (1.000) Luxembourg (1.000) Sweden (0.999) 3.1.1 SMEs introducing product or process innovations as % of SMEs 0.781 [0.656] 4 6 0.577 [0.432] Germany (1.0) Belgium (0.848) Luxembourg (0.792) Luxembourg (0.732) Germany (0.717) Belgium (0.713) 3.1.2 SMEs introducing marketing or organizational innovations as % of SMEs 0.605 [0.540] 10 12 0.566 [0.495] Germany (1.0) Luxembourg (0.960) Greece (0.801) Luxembourg (0.851) Ireland (0.797) Germany (0.720) 3.2.2 Contribution of medium and high-tech product exports to trade balance 0.579 [0.648] 15 9 0.553 [0.658] Germany (0.930) Slovenia (0.802) Hungary (0.756) Hungary (0.899) Germany (0.892) Slovakia (0.850) 3.2.3 Knowledge-intensive services exports as % total service exports 0.510 [0.524] 10 11 0.606 [0.665] Ireland (1.0) Luxembourg (1.0) Denmark (0.959) Ireland (1.000) Luxembourg (1.000) Denmark (1.000) 3.2.4 Sales of new-to-market and new-to-firm innovations as % of turnover 0.248 [0.156] 21 24 0.664 [0.488] Greece (1.0) Slovakia (1.0) Spain (0.982) Denmark (1.000) Slovakia (0.869) Spain (0.590) Average output result12 0.575 [0.620] 10 4 0.568 [0.550] Germany (0.859) Luxembourg (0.754) Denmark (0.701) Luxembourg (0.772) Denmark (0.728) Germany (0.723) Indicator Score in 2014 [2015] Ranking (out of 28) in 2014 Ranking (out of 28) in 2015 EU 28 average in 2014 [2015] Leading countries (top 3) in 2014 Leading countries (top 3) in 2015 2.2.1 SMEs innovating in-house as % of SMEs 0.729 [0.779] 8 4 0.570 [0.513] Germany (0.933) Cyprus (0.833) Denmark (0.813) The Netherlands (0.797) Ireland (0.792) Germany (0.787) 2.3.3 Community trademarks per billion GDP (in PPP-€) 0.573 [0.661] 7 8 0.444 [0.580] Luxembourg (1.0) Cyprus (1.0) Malta (1.0) Cyprus (1.000) Luxembourg (1.000) Malta (1.000) 2.3.4 Community designs per billion GDP (in PPP-€) 0.574 [0.999] 8 3 0.566 [0.569] Luxembourg (1.0) Austria (1.0) Denmark (0.971) Denmark (1.000) Luxembourg (1.000) Sweden (0.999) 3.1.1 SMEs introducing product or process innovations as % of SMEs 0.781 [0.656] 4 6 0.577 [0.432] Germany (1.0) Belgium (0.848) Luxembourg (0.792) Luxembourg (0.732) Germany (0.717) Belgium (0.713) 3.1.2 SMEs introducing marketing or organizational innovations as % of SMEs 0.605 [0.540] 10 12 0.566 [0.495] Germany (1.0) Luxembourg (0.960) Greece (0.801) Luxembourg (0.851) Ireland (0.797) Germany (0.720) 3.2.2 Contribution of medium and high-tech product exports to trade balance 0.579 [0.648] 15 9 0.553 [0.658] Germany (0.930) Slovenia (0.802) Hungary (0.756) Hungary (0.899) Germany (0.892) Slovakia (0.850) 3.2.3 Knowledge-intensive services exports as % total service exports 0.510 [0.524] 10 11 0.606 [0.665] Ireland (1.0) Luxembourg (1.0) Denmark (0.959) Ireland (1.000) Luxembourg (1.000) Denmark (1.000) 3.2.4 Sales of new-to-market and new-to-firm innovations as % of turnover 0.248 [0.156] 21 24 0.664 [0.488] Greece (1.0) Slovakia (1.0) Spain (0.982) Denmark (1.000) Slovakia (0.869) Spain (0.590) Average output result12 0.575 [0.620] 10 4 0.568 [0.550] Germany (0.859) Luxembourg (0.754) Denmark (0.701) Luxembourg (0.772) Denmark (0.728) Germany (0.723) Source: Own elaboration based on European Union (2014, 2015). Table 1. The innovation output indicators of the Swedish national innovation system Indicator Score in 2014 [2015] Ranking (out of 28) in 2014 Ranking (out of 28) in 2015 EU 28 average in 2014 [2015] Leading countries (top 3) in 2014 Leading countries (top 3) in 2015 2.2.1 SMEs innovating in-house as % of SMEs 0.729 [0.779] 8 4 0.570 [0.513] Germany (0.933) Cyprus (0.833) Denmark (0.813) The Netherlands (0.797) Ireland (0.792) Germany (0.787) 2.3.3 Community trademarks per billion GDP (in PPP-€) 0.573 [0.661] 7 8 0.444 [0.580] Luxembourg (1.0) Cyprus (1.0) Malta (1.0) Cyprus (1.000) Luxembourg (1.000) Malta (1.000) 2.3.4 Community designs per billion GDP (in PPP-€) 0.574 [0.999] 8 3 0.566 [0.569] Luxembourg (1.0) Austria (1.0) Denmark (0.971) Denmark (1.000) Luxembourg (1.000) Sweden (0.999) 3.1.1 SMEs introducing product or process innovations as % of SMEs 0.781 [0.656] 4 6 0.577 [0.432] Germany (1.0) Belgium (0.848) Luxembourg (0.792) Luxembourg (0.732) Germany (0.717) Belgium (0.713) 3.1.2 SMEs introducing marketing or organizational innovations as % of SMEs 0.605 [0.540] 10 12 0.566 [0.495] Germany (1.0) Luxembourg (0.960) Greece (0.801) Luxembourg (0.851) Ireland (0.797) Germany (0.720) 3.2.2 Contribution of medium and high-tech product exports to trade balance 0.579 [0.648] 15 9 0.553 [0.658] Germany (0.930) Slovenia (0.802) Hungary (0.756) Hungary (0.899) Germany (0.892) Slovakia (0.850) 3.2.3 Knowledge-intensive services exports as % total service exports 0.510 [0.524] 10 11 0.606 [0.665] Ireland (1.0) Luxembourg (1.0) Denmark (0.959) Ireland (1.000) Luxembourg (1.000) Denmark (1.000) 3.2.4 Sales of new-to-market and new-to-firm innovations as % of turnover 0.248 [0.156] 21 24 0.664 [0.488] Greece (1.0) Slovakia (1.0) Spain (0.982) Denmark (1.000) Slovakia (0.869) Spain (0.590) Average output result12 0.575 [0.620] 10 4 0.568 [0.550] Germany (0.859) Luxembourg (0.754) Denmark (0.701) Luxembourg (0.772) Denmark (0.728) Germany (0.723) Indicator Score in 2014 [2015] Ranking (out of 28) in 2014 Ranking (out of 28) in 2015 EU 28 average in 2014 [2015] Leading countries (top 3) in 2014 Leading countries (top 3) in 2015 2.2.1 SMEs innovating in-house as % of SMEs 0.729 [0.779] 8 4 0.570 [0.513] Germany (0.933) Cyprus (0.833) Denmark (0.813) The Netherlands (0.797) Ireland (0.792) Germany (0.787) 2.3.3 Community trademarks per billion GDP (in PPP-€) 0.573 [0.661] 7 8 0.444 [0.580] Luxembourg (1.0) Cyprus (1.0) Malta (1.0) Cyprus (1.000) Luxembourg (1.000) Malta (1.000) 2.3.4 Community designs per billion GDP (in PPP-€) 0.574 [0.999] 8 3 0.566 [0.569] Luxembourg (1.0) Austria (1.0) Denmark (0.971) Denmark (1.000) Luxembourg (1.000) Sweden (0.999) 3.1.1 SMEs introducing product or process innovations as % of SMEs 0.781 [0.656] 4 6 0.577 [0.432] Germany (1.0) Belgium (0.848) Luxembourg (0.792) Luxembourg (0.732) Germany (0.717) Belgium (0.713) 3.1.2 SMEs introducing marketing or organizational innovations as % of SMEs 0.605 [0.540] 10 12 0.566 [0.495] Germany (1.0) Luxembourg (0.960) Greece (0.801) Luxembourg (0.851) Ireland (0.797) Germany (0.720) 3.2.2 Contribution of medium and high-tech product exports to trade balance 0.579 [0.648] 15 9 0.553 [0.658] Germany (0.930) Slovenia (0.802) Hungary (0.756) Hungary (0.899) Germany (0.892) Slovakia (0.850) 3.2.3 Knowledge-intensive services exports as % total service exports 0.510 [0.524] 10 11 0.606 [0.665] Ireland (1.0) Luxembourg (1.0) Denmark (0.959) Ireland (1.000) Luxembourg (1.000) Denmark (1.000) 3.2.4 Sales of new-to-market and new-to-firm innovations as % of turnover 0.248 [0.156] 21 24 0.664 [0.488] Greece (1.0) Slovakia (1.0) Spain (0.982) Denmark (1.000) Slovakia (0.869) Spain (0.590) Average output result12 0.575 [0.620] 10 4 0.568 [0.550] Germany (0.859) Luxembourg (0.754) Denmark (0.701) Luxembourg (0.772) Denmark (0.728) Germany (0.723) Source: Own elaboration based on European Union (2014, 2015). According to the IUS 2014 and 2015, the SII for Denmark, Finland, Germany, and Sweden are well above those of the EU average. These countries are labeled the ‘innovation leaders’, and Sweden is ranked number one. However, Table 1 gives a sharply different picture for Sweden. Taking into account the normalized values observed for the eight output indicators discussed above, and according to the IUS 2014 data, Sweden has an average normalized score of 0.575 for the innovation output indicators (0.620 for the IUS 2015), which is very close to the EU28 average of 0.568 (0.550 for the IUS 2015). Sweden thereby holds the 10th position among the EU28 (4th in the IUS 2015). Our results mean that nearly a third of all EU countries have higher innovation outputs than Sweden based on the IUS 2014 data. The highest ranked countries, based on the IUS 2014 data, and with regard to innovation output, are Germany (0.859), Luxembourg (0.754), and Denmark (0.701). In turn, the highest ranked countries, based on the IUS 2015, are Luxembourg (0.772), Denmark (0.728), and Germany (0.723). Figure 2 includes a sensitivity analysis of the results discussed above, showing the 2015 data with bars, and the 2014 data with markers. Countries have been sorted by the respective output scores for year 2015, according to the values achieved when 8 and 12 outputs are considered, respectively. Using this procedure with the IUS 2014 data, the ranking is led by Germany with a normalized score of 0.809, followed by Luxembourg (0.746) and Denmark (0.720). Sweden ranks 4th with a normalized value of 0.686. When comparing the average values with both approaches (M = 8 or M = 12) we get a correlation of R2 = 0.88. This implies that the average output results and the subsequent rankings with 8 and 12 outputs show very similar values for the EU28 countries.11 Figure 2. View largeDownload slide Output orientation of European innovation systems. Figure 2. View largeDownload slide Output orientation of European innovation systems. 4.2 Input orientation The four IUS indicators which we consider as being important for the input side of innovation processes are listed in Table 2, where we also summarize Sweden’s normalized scores and rankings for illustrative purposes. Table 2. The innovation input indicators of the Swedish national innovation system Indicator Score in 2014 [2015] Ranking (out of 28) in 2014 Ranking (out of 28) in 2015 EU28 average in 2014 [2015] Leading countries (top 3) in IUS 2014 Leading countries (top 3) in IUS 2015 1.3.1 Public R&D expenditures as % of GDP 0.979 [0.957] 2 3 0.639 [0.641] Finland (0.990) Sweden (0.979) Denmark (0.918) Denmark (0.989) Finland (0.957) Sweden (0.957) 1.3.2 Venture capital investments 0.503 [0.536] 8 7 0.478 [0.472] Luxembourg (1.0) UK (0.762) Finland (0.544) Luxembourg (0.858) UK (0.672) Denmark (0.604) 2.1.1 Business R&D expenditures as % of GDP 0.991 [0.956] 2 2 0.558 [0.559] Finland (1.0) Sweden (0.991) Slovenia (0.926) Finland (1.000) Sweden (0.956) Denmark (0.868) 2.1.2 Non-R&D innovation expenditures as % of turnover 0.319 [0.412] 10 10 0.275 [0.349] Cyprus (0.936) Lithuania (0.701) Estonia (0.557) Estonia (0.871) Latvia (0.764) Germany (0.746) Mean aggregate input result13 0.698 [0.715] 1 2 0.488 [0.505] Sweden (0.698) Finland (0.694) Germany (0.631) Germany (0.718) Sweden (0.715) Estonia (0.688) Indicator Score in 2014 [2015] Ranking (out of 28) in 2014 Ranking (out of 28) in 2015 EU28 average in 2014 [2015] Leading countries (top 3) in IUS 2014 Leading countries (top 3) in IUS 2015 1.3.1 Public R&D expenditures as % of GDP 0.979 [0.957] 2 3 0.639 [0.641] Finland (0.990) Sweden (0.979) Denmark (0.918) Denmark (0.989) Finland (0.957) Sweden (0.957) 1.3.2 Venture capital investments 0.503 [0.536] 8 7 0.478 [0.472] Luxembourg (1.0) UK (0.762) Finland (0.544) Luxembourg (0.858) UK (0.672) Denmark (0.604) 2.1.1 Business R&D expenditures as % of GDP 0.991 [0.956] 2 2 0.558 [0.559] Finland (1.0) Sweden (0.991) Slovenia (0.926) Finland (1.000) Sweden (0.956) Denmark (0.868) 2.1.2 Non-R&D innovation expenditures as % of turnover 0.319 [0.412] 10 10 0.275 [0.349] Cyprus (0.936) Lithuania (0.701) Estonia (0.557) Estonia (0.871) Latvia (0.764) Germany (0.746) Mean aggregate input result13 0.698 [0.715] 1 2 0.488 [0.505] Sweden (0.698) Finland (0.694) Germany (0.631) Germany (0.718) Sweden (0.715) Estonia (0.688) Source: Own elaboration based on European Union (2014, 2015). Table 2. The innovation input indicators of the Swedish national innovation system Indicator Score in 2014 [2015] Ranking (out of 28) in 2014 Ranking (out of 28) in 2015 EU28 average in 2014 [2015] Leading countries (top 3) in IUS 2014 Leading countries (top 3) in IUS 2015 1.3.1 Public R&D expenditures as % of GDP 0.979 [0.957] 2 3 0.639 [0.641] Finland (0.990) Sweden (0.979) Denmark (0.918) Denmark (0.989) Finland (0.957) Sweden (0.957) 1.3.2 Venture capital investments 0.503 [0.536] 8 7 0.478 [0.472] Luxembourg (1.0) UK (0.762) Finland (0.544) Luxembourg (0.858) UK (0.672) Denmark (0.604) 2.1.1 Business R&D expenditures as % of GDP 0.991 [0.956] 2 2 0.558 [0.559] Finland (1.0) Sweden (0.991) Slovenia (0.926) Finland (1.000) Sweden (0.956) Denmark (0.868) 2.1.2 Non-R&D innovation expenditures as % of turnover 0.319 [0.412] 10 10 0.275 [0.349] Cyprus (0.936) Lithuania (0.701) Estonia (0.557) Estonia (0.871) Latvia (0.764) Germany (0.746) Mean aggregate input result13 0.698 [0.715] 1 2 0.488 [0.505] Sweden (0.698) Finland (0.694) Germany (0.631) Germany (0.718) Sweden (0.715) Estonia (0.688) Indicator Score in 2014 [2015] Ranking (out of 28) in 2014 Ranking (out of 28) in 2015 EU28 average in 2014 [2015] Leading countries (top 3) in IUS 2014 Leading countries (top 3) in IUS 2015 1.3.1 Public R&D expenditures as % of GDP 0.979 [0.957] 2 3 0.639 [0.641] Finland (0.990) Sweden (0.979) Denmark (0.918) Denmark (0.989) Finland (0.957) Sweden (0.957) 1.3.2 Venture capital investments 0.503 [0.536] 8 7 0.478 [0.472] Luxembourg (1.0) UK (0.762) Finland (0.544) Luxembourg (0.858) UK (0.672) Denmark (0.604) 2.1.1 Business R&D expenditures as % of GDP 0.991 [0.956] 2 2 0.558 [0.559] Finland (1.0) Sweden (0.991) Slovenia (0.926) Finland (1.000) Sweden (0.956) Denmark (0.868) 2.1.2 Non-R&D innovation expenditures as % of turnover 0.319 [0.412] 10 10 0.275 [0.349] Cyprus (0.936) Lithuania (0.701) Estonia (0.557) Estonia (0.871) Latvia (0.764) Germany (0.746) Mean aggregate input result13 0.698 [0.715] 1 2 0.488 [0.505] Sweden (0.698) Finland (0.694) Germany (0.631) Germany (0.718) Sweden (0.715) Estonia (0.688) Source: Own elaboration based on European Union (2014, 2015). According to our results, Sweden is at the very top with regard to its average input ranking (ranking number one in 2014 and second in 2015, with 0.698 and 0.715, respectively) among the other EU28 Member States. According to the IUS 2014 data, Finland has ranking number 2 (0.694) and Germany has ranking number 3 (0.631). Using the IUS 2015 data, Germany has ranking number 1 (0.718), and Estonia has ranking number 3 (0.688). Here too we have conducted a sensitivity analysis to check to what extent the average input result for Sweden changes when additional input indicators are considered (N = 4 or N = 7). The results of this sensitivity analysis are plotted in Fig. 3. The ranking is still led by Sweden with a score of 0.771 in 2014 and 2015. When comparing the average values and rankings with both approaches (N= 4 inputs or N= 7), we get a correlation of R2 = 0.88 in 2014 and R2 = 0.83 in 2015. Figure 3. View largeDownload slide Input orientation of European innovation systems. Figure 3. View largeDownload slide Input orientation of European innovation systems. 4.3 Innovation performance In this subsection we focus on the relation between the input and the output sides, i.e. on a measure of the innovation performance of national innovation systems (see Table 3). We define the innovation performance of a system as the ratio between innovation outputs and innovation inputs, using both the index number ( PIÎit) and the standard ( EIÎit) and bootstrapped ( bEIÎit) DEA approaches. Such ratios, show, in alternative ways to SIÎit, how efficiently the countries or systems use their inputs, as SIÎit simply calculates the mean of all the 25 indicators, and does not relate them to each other in any other way. Table 3. The innovation performance of the Swedish national innovation system14 Performance index Year Scores Ranking (out of 28) Leading countries in IUS (top 3) Summary innovation index (IUS) (2): SIÎit 2014 1.000 (=0.750/0.750) 1 Sweden (1.000), Denmark (0.971), Germany (0.945) 2015 1.000 (=0.740/0.740) 1 Sweden (1.000), Denmark (0.995), Finland (0.914) Productivity Innovation Index (4): PIÎit 2014 0.327 = (0.824 = 0.575/0.698)/2.540 24 Greece (1.000), Bulgaria (0.869), Italy (0.786) 2015 0.285 = (0.870 = 0.620/0.715)/3.040 22 Cyprus (1.000), Luxembourg (0.662), Romania (0.650) Efficiency Innovation Index (5): EIÎit 2014 0.604 27 17 countries are DEA efficient: EIÎit= 1.000a 2015 0.706 23 15 countries are DEA efficient: EIÎit= 1.000b Bootstrapped Efficiency Innovation Index, bEIÎit 2014 0.562* 27 Slovakia (0.917*), Croatia (0.913*), Portugal (0.904*) 2015 0.647* 24 Slovenia (0.902*), Poland (0.897*), United Kingdom (0.881*) Performance index Year Scores Ranking (out of 28) Leading countries in IUS (top 3) Summary innovation index (IUS) (2): SIÎit 2014 1.000 (=0.750/0.750) 1 Sweden (1.000), Denmark (0.971), Germany (0.945) 2015 1.000 (=0.740/0.740) 1 Sweden (1.000), Denmark (0.995), Finland (0.914) Productivity Innovation Index (4): PIÎit 2014 0.327 = (0.824 = 0.575/0.698)/2.540 24 Greece (1.000), Bulgaria (0.869), Italy (0.786) 2015 0.285 = (0.870 = 0.620/0.715)/3.040 22 Cyprus (1.000), Luxembourg (0.662), Romania (0.650) Efficiency Innovation Index (5): EIÎit 2014 0.604 27 17 countries are DEA efficient: EIÎit= 1.000a 2015 0.706 23 15 countries are DEA efficient: EIÎit= 1.000b Bootstrapped Efficiency Innovation Index, bEIÎit 2014 0.562* 27 Slovakia (0.917*), Croatia (0.913*), Portugal (0.904*) 2015 0.647* 24 Slovenia (0.902*), Poland (0.897*), United Kingdom (0.881*) a Austria, Bulgaria, Croatia, Cyprus, Czech Republic, France, Greece, Hungary, Ireland, Italy, Latvia, Luxembourg, Malta, Portugal, Romania, Slovakia, Spain. b Austria, Bulgaria, Cyprus, Czech Republic, Denmark, Greece, Ireland, Italy, Luxembourg, Malta, Romania, The Netherlands, Slovakia, Spain, United Kingdom. * denotes that the bootstrapped bias-corrected indices are statistically significant at the 5% significance level. Source: Own elaboration based on data from the European Union (2014, 2015). Table 3. The innovation performance of the Swedish national innovation system14 Performance index Year Scores Ranking (out of 28) Leading countries in IUS (top 3) Summary innovation index (IUS) (2): SIÎit 2014 1.000 (=0.750/0.750) 1 Sweden (1.000), Denmark (0.971), Germany (0.945) 2015 1.000 (=0.740/0.740) 1 Sweden (1.000), Denmark (0.995), Finland (0.914) Productivity Innovation Index (4): PIÎit 2014 0.327 = (0.824 = 0.575/0.698)/2.540 24 Greece (1.000), Bulgaria (0.869), Italy (0.786) 2015 0.285 = (0.870 = 0.620/0.715)/3.040 22 Cyprus (1.000), Luxembourg (0.662), Romania (0.650) Efficiency Innovation Index (5): EIÎit 2014 0.604 27 17 countries are DEA efficient: EIÎit= 1.000a 2015 0.706 23 15 countries are DEA efficient: EIÎit= 1.000b Bootstrapped Efficiency Innovation Index, bEIÎit 2014 0.562* 27 Slovakia (0.917*), Croatia (0.913*), Portugal (0.904*) 2015 0.647* 24 Slovenia (0.902*), Poland (0.897*), United Kingdom (0.881*) Performance index Year Scores Ranking (out of 28) Leading countries in IUS (top 3) Summary innovation index (IUS) (2): SIÎit 2014 1.000 (=0.750/0.750) 1 Sweden (1.000), Denmark (0.971), Germany (0.945) 2015 1.000 (=0.740/0.740) 1 Sweden (1.000), Denmark (0.995), Finland (0.914) Productivity Innovation Index (4): PIÎit 2014 0.327 = (0.824 = 0.575/0.698)/2.540 24 Greece (1.000), Bulgaria (0.869), Italy (0.786) 2015 0.285 = (0.870 = 0.620/0.715)/3.040 22 Cyprus (1.000), Luxembourg (0.662), Romania (0.650) Efficiency Innovation Index (5): EIÎit 2014 0.604 27 17 countries are DEA efficient: EIÎit= 1.000a 2015 0.706 23 15 countries are DEA efficient: EIÎit= 1.000b Bootstrapped Efficiency Innovation Index, bEIÎit 2014 0.562* 27 Slovakia (0.917*), Croatia (0.913*), Portugal (0.904*) 2015 0.647* 24 Slovenia (0.902*), Poland (0.897*), United Kingdom (0.881*) a Austria, Bulgaria, Croatia, Cyprus, Czech Republic, France, Greece, Hungary, Ireland, Italy, Latvia, Luxembourg, Malta, Portugal, Romania, Slovakia, Spain. b Austria, Bulgaria, Cyprus, Czech Republic, Denmark, Greece, Ireland, Italy, Luxembourg, Malta, Romania, The Netherlands, Slovakia, Spain, United Kingdom. * denotes that the bootstrapped bias-corrected indices are statistically significant at the 5% significance level. Source: Own elaboration based on data from the European Union (2014, 2015). Based on the data provided by the IUS 2014, Sweden is ranked extremely high with regard to the mean input indicator (0.698, 1st place in Table 2) and significantly lower with regard to output (0.575, 10th, Table 1). However, the IUS 2015 data show some improvement, particularly on the output side, as its rank improves to the 4th place―0.620. As shown in Table 3, this brings Sweden to the 1st place according to the overall SII ranking in both years. Indeed, Sweden presents the highest absolute and normalized values, e.g. in 2015, SIISE2015 = 0.750 and SIÎSE2015 = 1.000―as 0.750 is the maximum SIIj2015 score. However, when assessing the performance of the Swedish innovation system with regard to its productivity, this leads to very low rankings regardless the definition employed, either the index number, the standard, or the robust DEA estimate. Considering the index number definition, the absolute value corresponding to the ratio of the aggregate output to the aggregate input is PIISE2015 = 0.824 = 0.575/0.698. Once this value is normalized by the maximum productivity observed across all countries, corresponding to Cyprus: max j(PIIj2015)= 3.040, Sweden ranks in the 22nd place with a normalized score of PIÎSE2015= 0.327 = 0.824/3.040. Its 2015 rank is two positions better than in 2014 (24th place), as a result of the improvements in the aggregate output, but nevertheless places Sweden consistently at the lower tail of the productivity distribution. Obviously, the national innovation system in Sweden cannot be said to perform well at all from this efficiency point of view, and Sweden can certainly not be seen as an innovation leader in the EU, judging from the output and performance data. This is further confirmed by the standard and bootstrapped DEA results. Even if DEA is benevolent when assessing the relative productivity by searching for the most advantageous input and output weights:  νnt* and μmt*, it is to no avail. The standard and bootstrapped DEA scores: EIÎSE2014 = 0.604 and bEIÎSE2014 = 0.562 place Sweden in the second to last position in the rankings. This relative position marginally improves for 2015. It is worth highlighting that these results are statistically significant from the perspective of the bootstrapping methods, as all efficiency scores bEIÎit fall within the bootstrap confidence intervals—as presented in the last columns of the Appendix. We note in passing that the bootstrapped efficiency scores represent a valid solution to the low discriminatory power of standard DEA techniques. In both 2014 and 2015 a significant number of countries (16 and 15) are efficient with a unitary score (see Table 3). This precludes an individualized ranking, as these countries are tied in the first place. Therefore, bootstrapping techniques break the tie while informing about the statistical significance of their individual values, thereby providing comprehensive rankings. Regarding our case study on Sweden, we conclude that whether one uses the fixed input and outputs weights  ν¯nt and μ¯mt, or the optimal—most favorable—weights  νnt* and μmt*, the innovation performance of Sweden can be consistently categorized as modest using the IUS denomination for countries below 50% of the EU average. Therefore, and quite unexpectedly, the fact that the DEA methods represent an optimistic approach that give observations the opportunity to perform the best possible way with respect to its peers does not result in significant improvements. This really tells us about how strongly dominated is Sweden by a large set of reference peers. On the contrary, compared to PIÎSEt in 2014 and 2015, both the standard and booststrapped DEA results rank Sweden in lowest positions. For the same reason, as shown in Table 3 and the Appendix, unsuspected countries present innovation systems that, from a productivity perspective, perform fairly well, attaining high relative levels of outputs given the availability of innovation inputs. This is the case of Slovakia and Slovenia, leading the bootstrapped efficiency rankings in 2014 and 2015, respectively. We now proceed to compare the three alternative performance measures ( PIÎit, EIÎit, bEIÎit). Figure 4 shows the distribution of the scores obtained using SIÎit, and our simplest productivity index number— PIÎit—(using four input and eight output indicators) for both the IUS 2014 and 2015. If the two rankings coincided, one would expect the majority of countries to be distributed along a 45° line; i.e. laying on the southwest and northeast quadrants. However, this is certainly not the case. As can be observed, Sweden is not the only country where the two rankings are radically different, laying in the ‘wrong quadrants’. In fact, this is the case for most countries, including innovation leaders, innovation followers, moderate innovators, and modest innovators. Figure 4. View largeDownload slide Comparison of the normalized IUS SIÎit and the normalized PIÎit. (a) IUS 2014. (b) IUS 2015. Source: Own elaboration. Figure 4. View largeDownload slide Comparison of the normalized IUS SIÎit and the normalized PIÎit. (a) IUS 2014. (b) IUS 2015. Source: Own elaboration. As discussed by Zabala-Iturriagagoitia et al. (2007a), it is not evident that the innovation systems that invest larger amount of resources (i.e. inputs) are also the most efficient ones. This result is also confirmed by Chen and Guan (2012) and by Mahroum and Al-Saleh (2013) among others, pointing out that territories making modest investments in innovation still can enjoy significant innovation outputs, and therefore high performance in terms of efficiency. Similarly, Fig. 5 provides a comparison of the EIÎit and the bEIÎit for years 2014 and 2015. Countries have been sorted according to the bEIÎit, which is represented together with the upper and lower bounds of the confidence intervals. This figure complements the previous Fig. 4, offering a concise but complete visual overview of the results. Figure 5. View largeDownload slide Comparison of the EIÎit and the bootstrapped efficiency innovation index ( bEIÎit). (a) IUS 2014. (b)- IUS 2015. Figure 5. View largeDownload slide Comparison of the EIÎit and the bootstrapped efficiency innovation index ( bEIÎit). (a) IUS 2014. (b)- IUS 2015. We also confirm the compatibility of the fixed weight PIÎit results with those obtained for its deterministic and bootstrapped DEA counterparts allowing for flexible weights. To explore the similarity between the productivity distributions, Table 4 shows the results of a series of tests checking alternative hypotheses. The null hypotheses of the Wilcoxon rank sum test and the two-sample t-test are that both samples come from distributions with equal medians and equal means, respectively, against the alternative that they are not. For both years the null hypothesis is rejected at the 99% confidence level, implying that the median and the mean between the two bilateral pairs of efficiency measures are different ( PIÎit vs. EIÎit, and PIÎit vs. bEIÎit). However from the perspective of the rankings that these distributions yield, the Kendall and Spearman rank correlations show relatively high and significant values, particularly between the deterministic indices PIÎit and EIÎit. Unsurprisingly, as reported in the last two columns, the bootstrapped results bEIÎit show that their 2014 and 2015 distributions are also different from their deterministic counterparts EIÎit in both years. Nevertheless, once again, the rankings correlate to a large extent. Table 4. Tests of hypotheses between alternative productivity measures Test PIÎit vs. EIÎit PIÎit vs. bEIÎit EIÎit vs. bEIÎit 2014 2015 2014 2015 2014 2015 Wilcoxon rank sum testa −5.649 −5.803 −5.170 −5.416 2.504 2.474 P-value 0.000 0.000 0.000 0.000 0.015 0.017 Two-sample t-testb −8.855 −10.214 −7.361 −8.685 0.992 0.990 P-value 0.000 0.000 0.000 0.000 0.000 0.000 Kendall’s τ rank correlationc 0.549 0.694 0.365 0.265 0.800 0.520 P-value 0.000 0.000 0.006 0.050 0.000 0.000 Spearman’s ρ correlationc 0.706 0.843 0.531 0.402 0.881 0.630 P-value 0.000 0.000 0.004 0.035 0.000 0.000 Test PIÎit vs. EIÎit PIÎit vs. bEIÎit EIÎit vs. bEIÎit 2014 2015 2014 2015 2014 2015 Wilcoxon rank sum testa −5.649 −5.803 −5.170 −5.416 2.504 2.474 P-value 0.000 0.000 0.000 0.000 0.015 0.017 Two-sample t-testb −8.855 −10.214 −7.361 −8.685 0.992 0.990 P-value 0.000 0.000 0.000 0.000 0.000 0.000 Kendall’s τ rank correlationc 0.549 0.694 0.365 0.265 0.800 0.520 P-value 0.000 0.000 0.006 0.050 0.000 0.000 Spearman’s ρ correlationc 0.706 0.843 0.531 0.402 0.881 0.630 P-value 0.000 0.000 0.004 0.035 0.000 0.000 a H0 is that both distributions are the same. b H0 is the equality of means. c H0 is that both variables are independent. Table 4. Tests of hypotheses between alternative productivity measures Test PIÎit vs. EIÎit PIÎit vs. bEIÎit EIÎit vs. bEIÎit 2014 2015 2014 2015 2014 2015 Wilcoxon rank sum testa −5.649 −5.803 −5.170 −5.416 2.504 2.474 P-value 0.000 0.000 0.000 0.000 0.015 0.017 Two-sample t-testb −8.855 −10.214 −7.361 −8.685 0.992 0.990 P-value 0.000 0.000 0.000 0.000 0.000 0.000 Kendall’s τ rank correlationc 0.549 0.694 0.365 0.265 0.800 0.520 P-value 0.000 0.000 0.006 0.050 0.000 0.000 Spearman’s ρ correlationc 0.706 0.843 0.531 0.402 0.881 0.630 P-value 0.000 0.000 0.004 0.035 0.000 0.000 Test PIÎit vs. EIÎit PIÎit vs. bEIÎit EIÎit vs. bEIÎit 2014 2015 2014 2015 2014 2015 Wilcoxon rank sum testa −5.649 −5.803 −5.170 −5.416 2.504 2.474 P-value 0.000 0.000 0.000 0.000 0.015 0.017 Two-sample t-testb −8.855 −10.214 −7.361 −8.685 0.992 0.990 P-value 0.000 0.000 0.000 0.000 0.000 0.000 Kendall’s τ rank correlationc 0.549 0.694 0.365 0.265 0.800 0.520 P-value 0.000 0.000 0.006 0.050 0.000 0.000 Spearman’s ρ correlationc 0.706 0.843 0.531 0.402 0.881 0.630 P-value 0.000 0.000 0.004 0.035 0.000 0.000 a H0 is that both distributions are the same. b H0 is the equality of means. c H0 is that both variables are independent. 5. Discussion, conclusions, and research agenda According to the IUS, innovation performance implicitly means an average of 25 indicators measuring the determinants of innovations, inputs, innovations, as well as their consequences. In this article, we have discussed why innovation performance should be understood as the relationships between innovation inputs and outputs, as done in productivity and efficiency research. The first purpose of this article was to assess whether the SII of the European Innovation Scoreboard (IUS) is a meaningful measure of innovation performance. Our conclusion is that it is not. The main reason is that it mixes input and output innovation indicators and calculates an average of them. In addition, it lacks an explicit definition of what is meant by innovation performance, beyond the indiscriminate arithmetic mean of all 25 IUS indicators. The second purpose was to develop an alternative, productivity, or efficiency-based, measure of innovation system performance. By doing so, the article offers a critical review of the SII, and proposes to place more emphasis on the identification of and relation between input and output indicators. For this purpose we rely on the simplest productivity index number formulation, but we also supplement it with state-of-the-art productivity measurement methods in the form of robust (bootstrapped) DEA techniques that address the weaknesses of deterministic indices. Our article can be understood as a contribution to the need for a ‘substantial research effort on conceptual foundations as well as empirics of innovation measurement’ (Janger et al. 2017: 39). We use the existing data provided by the IUS but seen through a different (productivity) lens, with the purpose of increasing the breadth of sight of innovation policy makers and politicians. By using both fixed and flexible weights we have assessed the performance of EU28 Member States by comparing alternative multi-input/multi-output relationships in a benchmark model, including four inputs and eight outputs. We justify this choice on theoretical grounds but encourage other researchers to improve on this ‘standard model’. Our intention is to mimic the simplicity of the methods followed by the SII, though from a different (productivity) point of view. The simple productivity index and its sophisticated DEA counterparts result in very similar rankings. The more sophisticated method is actually not needed to reach the basic result. If one is interested in this result, rather than in specific and individual numerical values (because distributions have been shown to be different), the simplest productivity definition performs well. However, the relevance of statistically robust methods such as bootstrapped DEA is clear, as they allow addressing some of the weaknesses of the deterministic approached; most particularity, the presence of innovation performance outliers emerging from the (IUS) data. As can be observed in the Appendix, the top 10 EU national innovations systems ranked in terms of the bootstrapped efficiency score for 2015 are Slovenia, Poland, the UK, Malta, Austria, France, Denmark, Italy, Portugal, and Spain. Sweden is ranked number 23. No matter how unbelievable it may appear, this efficiency ranking is appropriate to the extent that the IUS data are correct and our approach to dealing with them is sensible and robust. The ranking goes against stylized facts and seems counterintuitive, since many of the top-ranked countries are less developed economies with less comprehensive innovation systems. In turn, the countries that traditionally lead innovation rankings (such as the SII) get lower standings with regard to innovation performance. To fully explain the reasons for these results in a detailed manner, we would need sophisticated methods that allow controlling for key aspects such as sectoral structure, diversification, degree of internationalization, firm size, and institutional conditions, which remains for further research. One partial explanation to these seemingly contradictory results is that many of these small EU countries devote very limited resources to inputs. According to the approach followed here, the lower the inputs, the higher the efficiency. Besides, they manage to obtain reasonable outputs in relation to the invested inputs. For example, Sweden invested 7.35 times more than Bulgaria in innovation inputs, according to IUS 2014 (see Appendix). At the same time Sweden achieved an output figure that is only 2.77 times higher than Bulgaria’s. The figures are similar for the other EU countries that rank high on traditional innovation performance calculations (such as the SII). The above figures imply that some less developed countries manage to use their (more limited) resources in a more productive/efficient way. It also indicates that there might be Decreasing Returns to Scale (DRS) obtained by investments in innovation input activities, so countries investing a large amount of innovations resources cannot achieve large returns. The DEA methodology provides ancillary results that shed light on the role that returns to scale (i.e. input size) play for inefficient performance. The last column in the Appendix shows whether IRS, CRS, or DRS prevail locally for each country—see the footnote to program (5) on how to determine the nature of returns to scale. The suspected widespread presence of DRS is confirmed, with as many as 10 countries exhibiting a suboptimal scale size as a result of an excessive relative usage of inputs. Among this, we find several SII leading countries, including Sweden, Germany, Finland, etc. On the contrary, incurring in innovation inefficiency due to IRS (i.e. relatively low use of inputs) does not seem to be an issue, as only three countries, Croatia, Latvia, and Lithuania, exhibit this type of returns. As Zabala-Iturriagagoitia et al. discuss (2007a: 667), another partial explanation may be that countries with comprehensive innovation systems are usually, oriented toward radical innovation and the development of new industries, often in knowledge-intensive sectors and high-tech industries. Such innovation efforts are characterized by uncertainty, high-risk, failures, and large time lags. In contrast, countries with smaller innovation input resources tend to absorb and adopt embodied knowledge and innovations from other countries. Such absorption involves lower innovation input costs, but may, at the same time, be more efficient, as they may avoid the inherent risk involved in the development of these innovations, so that the ‘new’ knowledge is more rapidly and cheaply adapted and adopted than in the country where it was developed. According to Freeman and Soete (1988), the efficient diffusion of innovations is often much more important for development and growth than being the lead innovator. Countries regarded as innovation leaders by the IUS may thus be more prone to the creation of new-to-the-world innovations, while follower countries are more prone to the absorption and later diffusion of these innovations, as long as the required levels of absorptive capacity and learning are in place. This is highly relevant from an innovation policy point of view, a partial objective of which may be to catch up with the leading countries by absorbing innovations from abroad. We want to stress the irrelevance of the notion of optimality in an innovation context, given the impossibility to specify an ideal or optimal innovation system. The only way to identify problems that should be subject to innovation policy is therefore to compare existing innovation systems with each other. Unfortunately, the ranking of the efficiency of the EU28 innovation systems in the Appendix provides limited guidelines from the point of view of innovation policy development. There are no reasons whatsoever to benchmark Sweden’s national innovation system with those of Slovenia, Poland, Greece, Cyprus, etc., in attempting to develop an analysis to form a basis for innovation policy changes. Such comparisons should, instead, be made between innovation systems that are more similar in a structural sense and at a similar level of development, that have the same size, that score similarly on innovation output or innovation input, etc. A restricted DEA analysis for countries with similar characteristics could be helpful here. One implication of this analysis is that considerable efforts should be made to identify the sources of the inefficiencies and problems in the national systems of innovation. Existing innovation intensities are influenced by a number of forces that affect innovation processes, forces which we call determinants of innovations. Many of these determinants can be influenced by the state. When the state (through its public agencies) influences these determinants to increase innovation intensities of certain kinds or change the direction of innovation processes, it is actually pursuing innovation policy. Public bodies should be able to know and monitor the evolution of these determinants and their impact on innovation. The SII does however not help to do so. Since it includes inputs, determinants, innovations, and consequences, this synthetic indicator is not useful from the point of view of innovation policy design. The SII score will increase if a country puts more (input) resources into its innovation system, regardless of how the resources are used or what the (innovation) output might be. A worrisome property of the SII index is that its value increases even if the innovation output resulting from more inputs is 0 (i.e. the marginal productivity of inputs is 0). These results should call for a serious reconsideration of who the real European ‘innovation leaders’ may be, and in what sense they are leaders. These findings also call into question the way that the European Commission analyzes the innovation data presented in the IUS. The IUS is intended to have a real impact on the design of innovation policies of the EU Member States. According to our results, the SII is flawed with regard to measuring innovation performance and may therefore mislead researchers, policy makers, politicians, as well as the general public. Policy makers in the field of innovation should be able to identify the policy problems in their innovation systems and relate them to their causes to be able to select policy instruments to mitigate the problems. Our approach can be useful for the design of policy. For example, if we know that Sweden is much weaker on the output side than on the input side, our approach can tell policy makers and politicians to concentrate more on making a more efficient use of existing inputs than on increasing the volume of the inputs, for example through the articulation of an instrument-mix that considers demand-side innovation policy instruments. To further develop such a kind of disaggregated analysis to identify policy problems is much more useful as a basis for innovation policy design, than to aggregate data into single ranking numbers of countries. In this way, the individual indicators that constitute composite indicators should be analyzed in much more depth, and this should be done in a comparative manner. We believe that the simplicity of our definition of productivity as a ratio of (aggregate) outputs to (aggregate) inputs, along with the proved confirmation by using DEA methods, reinforces the credibility of our results. Although the analyses can be further refined, we do claim that this trajectory is also a fruitful one for future research as remarked in the different extensions that we propose: use of bootstrapped Malmquist productivity indices where the relative sources of productivity change, such as technological progress vs. catching-up processes, could be measured, restricted frontiers by country categories, etc. It can be argued that the rationale used in this article follows a linear logic, as opposed to a systemic and holistic one (e.g. Edquist 2014a, 2014b, 2014c, 2018). As mentioned at the end of Section 3.1, an ideal basis for the design of innovation policy should aim to include a fully articulated holistic innovation systems approach. Such an approach would include knowledge about all determinants of the development and diffusion of innovations—knowledge which we do not possess. This would mean overcoming the linear model of innovation that still dominates innovation policy in all countries, despite the fact that it has been completely rejected in innovation research (Edquist 2014a, 2018). Developing such a holistic view would make it possible to account for all determinants of innovations as inputs when measuring innovation performance and efficiency of innovation systems. This would mean that the linear approach (a focus on input–output relationships) used out of necessity in this article, could be overcome and transcended. The linear and holistic approaches would become compatible and reconciled. However, we are not yet there. Footnotes 1 For illustrative purposes, we use Sweden as an example when we discuss the data, methodology, and results. However, in the Appendix, we present the data for all 28 EU Member States. This means that the reader can substitute Sweden by any other of the 27 EU countries. 2 These steps include development of a theoretical framework, data selection, development of an exploratory multivariate analysis, imputation of missing data, normalization, weighting and aggregation, and robustness and sensibility checks among others (see Nardo et al. 2005). 3 The IUS performs a max–min normalization by subtracting the minimum value and diving by the observed range: ι^lit=(ιlit−min (ιljt))/(max (ιljt)−min (ιljt)), j = 1,…, 34 countries. Therefore, ‘The maximum re-scaled score is thus equal to 1 and the minimum re-scaled score is equal to 0’ (European Union 2015: 79). 4 For the detailed interpretation of each of the output and input indicators considered, please see European Union (2014: 86–90). All 25 indicators are also listed in Fig. 1. 5 Following Hagedoorn and Cloodt (2003: 1367), for whom the concept of innovation performance excludes ‘the possible economic success of innovations as such’, we do neither consider the consequences (i.e. outcomes or impacts) of innovations, such as economic growth or employment, to be innovation output indicators. 6 The exclusion of indicator 2.3.1 (Patent Cooperation Treaty (PCT) patent applications) is made for a different reason. As long as a product or a process has not been commercialized and connected to the market, it cannot be considered to be an innovation. An application for a patent is far from being a product introduced to the market. Not even granted patents are product innovations, although they may be a basis for future innovations. Patents are rather an indicator of research results or inventions, as they reflect that something is technologically new. This implies that patents are a throughput, rather than an output of innovation (see also Lee 2015; Janger et al. 2017). 7 The additional indicators considered were PCT patent applications (2.3.1), Employment in fast-growing enterprises in innovative sectors (3.1.3), Employment in knowledge-intensive activities (3.2.1), and License and patent revenues from abroad (3.2.5). 8 Of these two indicators, R&D expenditures in the business sector are certainly to a very large extent directly undertaken to enhance innovation (see Lee 2015). R&D expenditures in the public sector are to a lesser extent undertaken directly for this purpose, since a substantial proportion is pursued to develop new scientific knowledge, part of which may result (or not) in innovations. In spite of this, we include both of these indicators in the category of input indicators, although a part of the public sector R&D expenditures may result in innovations only after a substantial time lag, or not at all for some kinds of public R&D financing. 9 The additional indicators considered were New doctorate graduates (1.1.1), Percentage of population aged 30–34 years having completed tertiary education (1.1.2), and Percentage youth aged 20–24 years having attained at least upper secondary education (1.1.3). 10 Program (5) corresponds to a fractional (ratio form) formulation, which after some transformations can be expressed as a linear program, and whose resulting objective function is linearly homogeneous of degree one, thereby defining a productivity frontier characterized by constant returns to scale (CRS), Cooper, Seiford, and Tone (2007: chapter 2). Its dual−primal−formulation corresponds to the input-oriented envelopment form: EII^it=min θ,λ{θ:∑j=1Jλjtx^jnt≤θx^nt, n=1,…,N; ∑j=1Jλjty^jmt≥y^mt, m=1,…,M;  ∑j=1Jλjt=1, λj≥0, j=1,…,J}, where λj are intensity variable identifying the benchmark countries conforming the reference facet. Resorting to the envelopment form, it is possible to determine the nature of returns to scale at this benchmark frontier: increasing, constant, and decreasing, by examining the value of the sum of the optimal lambdas—Cooper, Seiford, and Tone (2007: chapter 5): Decreasing Returns to Scale (DRS) prevails for (xit,yit)⇔∑jλj*>1 for all optimal solutions. Increasing Returns to Scale (IRS) prevails for (xit,yit)⇔∑jλj*<1 for all optimal solutions. CRS prevails for (xit,yit)⇔∑jλj*=1 for some optimal solutions. Therefore, by solving the dual to program (5) we can determine what type of returns to scale locally predominate for (xit,yit)—which, in turn, implies that a suboptimal scale is at play if the innovation system turns out to be inefficient. In the discussion section we recall this result to substantiate one of the possible reasons why innovation performance is deficient depending on the innovation size of the countries. 11 When using the IUS 2015 data, the ranking is led by Luxembourg (0.765), followed by Germany (0.725) and Denmark (0.724). Sweden ranks 5th with a normalized value of 0.690. When comparing the average values with both approaches (M = 8 or M = 12) a correlation of R2 = 0.93 in 2015 is obtained. 12 Calculation is based on the arithmetic mean of the normalized scores of each output indicator. 13 Calculation is based on the arithmetic mean of the normalized scores for each input indicator. 14 The data and rankings for all EU28 Member Countries are presented in the Appendix. Acknowledgements A preliminary version of this article (Edquist and Zabala-Iturriagagoitia, 2015) received the 2015 VII UAM-Accenture Chair Scientific Award in Economics and Innovation Management in March 2016. The authors are indebted to the editor and two anonymous referees for their comments on an earlier draft. The authors appreciate Hannes Toivanen, Jakob Edler, Nick Vonortas, Olof Ejermo, Martin Andersson, Sylvia Schwaag Serger, Espen Solberg, Lennart Stenberg, Torben Schubert, Lorenzo Isella, Athina Karvounaraki, and Daniel Vertesy for their comments. The authors are also grateful to Michael Nauruschat for his assistance with the data collection and to Margaret Newman-Nowicka for the language editing of the text. J.M.Z-I. wishes to acknowledge the financial support from the Basque Government Department of Education, Language Policy and Culture (IT 885-16), and C.E. would like to acknowledge support from the Rausing Foundation. The views expressed are purely those of the authors and may not in any circumstances be regarded as stating an official position of the European Commission. References Archibugi D. , Denni M. , Filippetti A. ( 2009 ) ‘The Technological Capabilities of Nations: The State of the Art of Synthetic Indicators’ , Technological Forecasting and Social Change , 76 : 917 – 31 . Google Scholar CrossRef Search ADS Balk B. M. ( 2008 ) Price and Quantity Index Numbers: Models for Measuring Aggregate Change and Difference . New York, NY : Cambridge University Press . Google Scholar CrossRef Search ADS Bogetoft P. , Otto L. ( 2011 ). Benchmarking with DEA, SFA, and R . New York, NY : Springer . Google Scholar CrossRef Search ADS Carayannis E. , Grigoroudis E. ( 2014 ) ‘ Linking Innovation, Productivity, and Competitiveness: Implications for Policy and Practice ’, Journal of Technology Transfer , 39 : 199 – 218 . Google Scholar CrossRef Search ADS Charnes A. , Cooper W. W. , Rhodes E. ( 1978 ) ‘ Measuring the Efficiency of Decision Making Units ’, European Journal of Operational Research , 2 / 6 : 429 – 44 . Google Scholar CrossRef Search ADS Chen K. , Guan J. ( 2012 ) ‘ Measuring the Efficiency of China’s Regional Innovation Systens: Application of Network Data Envelopment Analysis (DEA) ’, Regional Studies , 46 / 3 : 355 – 77 . Google Scholar CrossRef Search ADS Coelli T. J. et al. ( 2005 ) An Introduction to Efficiency and Productivity Analysis . New York, NY : Springer . Cooper W. W. , Seiford L. M. , Tone K. ( 2007 ) Data Envelopment Analysis: A Comprehensive Text with Models, Applications, References and DEA-Solver Software . New York, NY : Springer . Daraio C. , Simar L. ( 2007 ) Advanced Robust and Nonparametric Methods in Efficiency Analysis , New York, NY : Springer Science + Business Media, LLC . Dutta S. , Lanvin B. , Wunsch-Vincent S. ( 2017 ) ‘The Global Innovation Index 2017, Innovation Feeding the World’ . Cornell University, INSEAD, WIPO. Ithaca, Fontainebleau, and Geneva. Edquist C. ( 2005 ) ‘Systems of Innovation: Perspectives and Challenges’, in Fagerberg J. , Mowery D. , Nelson R. (eds) Oxford Handbook of Innovation , pp. 181 – 208 . Oxford : Oxford University Press . Edquist C. ( 2011 ) ‘ Design of Innovation Policy through Diagnostic Analysis: Identification of Systemic Problems (or Failures) ’, Industrial and Corporate Change , 20 / 6 : 1725 – 53 . Google Scholar CrossRef Search ADS Edquist C. ( 2014a ) ‘ Striving Towards a Holistic Innovation Policy in European Countries—But Linearity Still Prevails! ’, STI Policy Review , 5 / 2 : 1 – 19 . Edquist C. ( 2014b ) ‘En Helhetlig Innovationspolitik—Varför, Vad Och Hur? (a Holistic Innovation Policy—Why, What and How?)’, in Wanger M. Ö. (ed.) Position Sverige—Om Innovation, Hållbarhet Och Arbetsmarknad—En Debattantologi (Position Sweden—On Innovation, Sustainability and Labour Market—A Debate Anthology—In Swedish) , pp. 59 – 80 . Stockholm : Ekerlids Förlag . Edquist C. ( 2014c ) ‘Efficiency of Research and Innovation Systems for Economic Growth and Employment’, CIRCLE Working Paper 2014/08. Sweden: Lund University. Edquist C. ( 2018 ) ‘Towards a Holistic Innovation Policy: Can the Swedish National Innovation Council Serve as a Role Model?’, Papers in Innovation Studies, Paper no. 2018/02. Sweden: CIRCLE, Lund University. Edquist C. et al. , eds ( 2015 ) Public Procurement for Innovation . Cheltenham : Edward Elgar . Google Scholar CrossRef Search ADS European Commission ( 2011 ) Innovation Union Scoreboard 2010. The Innovation Union's performance scoreboard for Research and Innovation. Brussels: European Commission. European Commission ( 2013 ) State of the Innovation Union 2012: Accelerating change. European Commission, Directorate General for Research and Innovation, Brussels. 21.3.2013. COM(2013) 149 final. Brussels: European Commission. European Union ( 2014 ) Innovation Union Scoreboard 2014 . Brussels : European Commission . European Union ( 2015 ) Innovation Union Scoreboard 2015 . Brussels : European Commission . Farrell M. ( 1957 ) ‘ The Measurement of Productive Efficiency ’, Journal of the Royal Statistical Society: Series A General , 120 / 3 : 253 – 81 . Google Scholar CrossRef Search ADS Freeman C. , Soete L. ( 1988 ) ‘ Innovation Diffusion and Employment Policies ’, Ricerche Economiche , 40 : 836 – 54 . Grupp H. , Schubert T. ( 2010 ) ‘ Review and New Evidence on Composite Innovation Indicators for Evaluating National Performance ’, Research Policy , 39 : 67 – 78 . Google Scholar CrossRef Search ADS Hagedoorn J. , Cloodt M. ( 2003 ) ‘ Measuring Innovative Performance: Is There an Advantage in Using Multiple Indicators ’, Research Policy , 32 / 8 : 1365 – 79 . Google Scholar CrossRef Search ADS Janger J. et al. ( 2017 ) ‘ The EU2020 Innovation Indicator: A Step Forward in Measuring Innovation Outputs and Outcomes ’, Research Polucy , 46 : 30 – 42 . Google Scholar CrossRef Search ADS Lee Y.-N. ( 2015 ) ‘ Evaluating and Extending Innovation Indicators for Innovation Policy ’, Research Evaluation , 24 : 471 – 88 . Google Scholar CrossRef Search ADS Liu J. S. , Lu W. M. , Ho M. H. C. ( 2014 ) ‘ National Characteristics: Innovation Systems from the Process Efficiency Perspective ’, R&D Management , 45 / 4 : 317 – 38 . Google Scholar CrossRef Search ADS Mahroum S. , Al-Saleh Y. ( 2013 ) ‘ Towards a Functional Framework for Measuring National Innovation Efficacy ’, Technovation , 33 : 320 – 32 . Google Scholar CrossRef Search ADS Mersha T. ( 1989 ) ‘ Output and Performance Measurement in Outpatient Care ’, Omega , 17 / 2 : 159 – 67 . Google Scholar CrossRef Search ADS Mendonça S. , Santos Pereira T. , Mira Godinho M. ( 2004 ) ‘ Trademarks as an Indicator of Innovation and Industrial Change ’, Research Policy , 33 : 1385 – 404 . Google Scholar CrossRef Search ADS Nardo M. et al. ( 2005 ) ‘ Handbook on Constructing Composite Indicators: Methodology and User Guide ’, OECD Statistics Working Papers 2005/03, OECD Publishing, Paris. Nasierowski W. , Arcelus F. J. ( 2012 ) ‘ About Efficiency of Innovations: What Can Be Learned from the Innovation Union Scoreboard Index ’, Procedia—Social and Behavioral Sciences , 58 : 792 – 801 . Google Scholar CrossRef Search ADS OECD/Eurostat ( 2005 ) Oslo Manual. Guidelines for Collecting and Interpreting Innovation Data , 3rd edn . Paris: OECD Publishing . Saltelli A. ( 2007 ) ‘ Composite Indicators between Analysis and Advocacy ’, Social Indicators Research , 81 : 65 – 77 . Google Scholar CrossRef Search ADS Simar L. , Wilson P. W. ( 1998 ) ‘ Sensitivity Analysis of Efficiency Scores: How to Bootstrap in Nonparametric Frontier Models ’, Management Science , 44 / 1 : 49 – 61 . Google Scholar CrossRef Search ADS Zabala-Iturriagagoitia J. M. et al. ( 2007a ) ‘ Regional Innovation Systems: How to Assess Performance ’, Regional Studies , 41 / 5 : 661 – 72 . Google Scholar CrossRef Search ADS Zabala-Iturriagagoitia J. M. et al. ( 2007b ) ‘ What Indicators Do (or Do Not) Tell Us about Regional Innovation Systems ’, Scientometrics , 70 / 1 : 85 – 106 . Google Scholar CrossRef Search ADS Appendix The Efficiency of the EU28 Innovation Systems Table A1. Year 2014 Country Output (M = 8) Input (N = 4) Summary Innovation Index SIIit Normalized Summary Innovation Index SIÎit Ranking SIÎit Productivity Innovation Index PIIit Normalized Productivity Innovation Index PIÎit Ranking PIÎit Efficiency Innovation Index EIÎit Ranking EIÎit Bootstrapped Efficiency Innovation Index bEIÎit Ranking bEIÎit Lower bound Upper bound Returns to scale (IRS, CRS, DRS) Austria 0.637 0.488 0.599 0.799 10 1.305 0.519 12 1.000 1 0.878 17 0.679 0.995 CRS Belgium 0.603 0.507 0.627 0.836 7 1.189 0.473 15 0.827 22 0.768 22 0.698 0.822 DRS Bulgaria 0.207 0.095 0.188 0.251 28 2.179 0.866 2 1.000 1 0.886 9 0.687 0.997 CRS Croatia 0.281 0.243 0.306 0.408 23 1.156 0.459 16 1.000 1 0.914 2 0.836 0.994 CRS Cyprus 0.66 0.39 0.501 0.668 14 1.692 0.672 6 1.000 1 0.882 15 0.678 0.995 CRS Czech Rep. 0.497 0.395 0.422 0.563 16 1.258 0.500 14 1.000 1 0.899 4 0.773 0.996 CRS Denmark 0.701 0.63 0.728 0.971 2 1.113 0.442 17 0.920 19 0.849 19 0.771 0.915 DRS Estonia 0.544 0.628 0.502 0.669 13 0.866 0.344 22 0.653 26 0.610 26 0.564 0.650 DRS Finland 0.579 0.694 0.684 0.912 4 0.834 0.331 23 0.704 25 0.652 25 0.590 0.700 DRS France 0.52 0.479 0.571 0.761 11 1.086 0.431 18 1.000 1 0.883 12 0.683 0.996 CRS Germany 0.859 0.631 0.709 0.945 3 1.361 0.541 11 0.839 21 0.780 21 0.718 0.835 DRS Greece 0.516 0.205 0.384 0.512 19 2.517 1.000 1 1.000 1 0.885 10 0.690 0.996 CRS Hungary 0.273 0.304 0.351 0.468 20 0.898 0.357 21 1.000 1 0.888 6 0.751 0.995 CRS Ireland 0.578 0.339 0.606 0.808 9 1.705 0.677 5 1.000 1 0.882 16 0.684 0.996 CRS Italy 0.591 0.299 0.443 0.591 15 1.977 0.785 3 1.000 1 0.887 8 0.740 0.995 CRS Latvia 0.19 0.2 0.221 0.295 27 0.950 0.377 20 1.000 1 0.887 7 0.731 0.995 CRS Lithuania 0.193 0.447 0.289 0.385 24 0.432 0.172 28 0.503 28 0.467 28 0.418 0.501 IRS Luxembourg 0.761 0.461 0.646 0.861 5 1.651 0.656 7 1.000 1 0.882 14 0.689 0.996 CRS Malta 0.391 0.309 0.319 0.425 21 1.265 0.503 13 1.000 1 0.891 5 0.751 0.995 CRS Netherlands 0.538 0.543 0.629 0.839 6 0.991 0.394 19 0.756 24 0.703 24 0.647 0.753 DRS Poland 0.253 0.38 0.279 0.372 25 0.666 0.265 27 0.921 18 0.860 18 0.772 0.918 DRS Portugal 0.566 0.366 0.41 0.547 18 1.546 0.614 8 1.000 1 0.905 3 0.802 0.995 CRS Romania 0.283 0.157 0.237 0.316 26 1.803 0.716 4 1.000 1 0.884 11 0.685 0.995 CRS Slovakia 0.391 0.275 0.318 0.424 22 1.422 0.565 10 1.000 1 0.918 1 0.818 0.995 CRS Slovenia 0.44 0.571 0.513 0.684 12 0.771 0.306 25 0.861 20 0.806 20 0.742 0.856 DRS Spain 0.464 0.315 0.414 0.552 17 1.473 0.585 9 1.000 1 0.882 13 0.700 0.995 CRS Sweden 0.575 0.698 0.75 1.000 1 0.824 0.327 24 0.604 27 0.563 27 0.513 0.601 DRS UK 0.439 0.577 0.613 0.817 8 0.761 0.302 26 0.815 23 0.758 23 0.691 0.811 DRS Country Output (M = 8) Input (N = 4) Summary Innovation Index SIIit Normalized Summary Innovation Index SIÎit Ranking SIÎit Productivity Innovation Index PIIit Normalized Productivity Innovation Index PIÎit Ranking PIÎit Efficiency Innovation Index EIÎit Ranking EIÎit Bootstrapped Efficiency Innovation Index bEIÎit Ranking bEIÎit Lower bound Upper bound Returns to scale (IRS, CRS, DRS) Austria 0.637 0.488 0.599 0.799 10 1.305 0.519 12 1.000 1 0.878 17 0.679 0.995 CRS Belgium 0.603 0.507 0.627 0.836 7 1.189 0.473 15 0.827 22 0.768 22 0.698 0.822 DRS Bulgaria 0.207 0.095 0.188 0.251 28 2.179 0.866 2 1.000 1 0.886 9 0.687 0.997 CRS Croatia 0.281 0.243 0.306 0.408 23 1.156 0.459 16 1.000 1 0.914 2 0.836 0.994 CRS Cyprus 0.66 0.39 0.501 0.668 14 1.692 0.672 6 1.000 1 0.882 15 0.678 0.995 CRS Czech Rep. 0.497 0.395 0.422 0.563 16 1.258 0.500 14 1.000 1 0.899 4 0.773 0.996 CRS Denmark 0.701 0.63 0.728 0.971 2 1.113 0.442 17 0.920 19 0.849 19 0.771 0.915 DRS Estonia 0.544 0.628 0.502 0.669 13 0.866 0.344 22 0.653 26 0.610 26 0.564 0.650 DRS Finland 0.579 0.694 0.684 0.912 4 0.834 0.331 23 0.704 25 0.652 25 0.590 0.700 DRS France 0.52 0.479 0.571 0.761 11 1.086 0.431 18 1.000 1 0.883 12 0.683 0.996 CRS Germany 0.859 0.631 0.709 0.945 3 1.361 0.541 11 0.839 21 0.780 21 0.718 0.835 DRS Greece 0.516 0.205 0.384 0.512 19 2.517 1.000 1 1.000 1 0.885 10 0.690 0.996 CRS Hungary 0.273 0.304 0.351 0.468 20 0.898 0.357 21 1.000 1 0.888 6 0.751 0.995 CRS Ireland 0.578 0.339 0.606 0.808 9 1.705 0.677 5 1.000 1 0.882 16 0.684 0.996 CRS Italy 0.591 0.299 0.443 0.591 15 1.977 0.785 3 1.000 1 0.887 8 0.740 0.995 CRS Latvia 0.19 0.2 0.221 0.295 27 0.950 0.377 20 1.000 1 0.887 7 0.731 0.995 CRS Lithuania 0.193 0.447 0.289 0.385 24 0.432 0.172 28 0.503 28 0.467 28 0.418 0.501 IRS Luxembourg 0.761 0.461 0.646 0.861 5 1.651 0.656 7 1.000 1 0.882 14 0.689 0.996 CRS Malta 0.391 0.309 0.319 0.425 21 1.265 0.503 13 1.000 1 0.891 5 0.751 0.995 CRS Netherlands 0.538 0.543 0.629 0.839 6 0.991 0.394 19 0.756 24 0.703 24 0.647 0.753 DRS Poland 0.253 0.38 0.279 0.372 25 0.666 0.265 27 0.921 18 0.860 18 0.772 0.918 DRS Portugal 0.566 0.366 0.41 0.547 18 1.546 0.614 8 1.000 1 0.905 3 0.802 0.995 CRS Romania 0.283 0.157 0.237 0.316 26 1.803 0.716 4 1.000 1 0.884 11 0.685 0.995 CRS Slovakia 0.391 0.275 0.318 0.424 22 1.422 0.565 10 1.000 1 0.918 1 0.818 0.995 CRS Slovenia 0.44 0.571 0.513 0.684 12 0.771 0.306 25 0.861 20 0.806 20 0.742 0.856 DRS Spain 0.464 0.315 0.414 0.552 17 1.473 0.585 9 1.000 1 0.882 13 0.700 0.995 CRS Sweden 0.575 0.698 0.75 1.000 1 0.824 0.327 24 0.604 27 0.563 27 0.513 0.601 DRS UK 0.439 0.577 0.613 0.817 8 0.761 0.302 26 0.815 23 0.758 23 0.691 0.811 DRS Source: Own elaboration from European Union (2014). Table A1. Year 2014 Country Output (M = 8) Input (N = 4) Summary Innovation Index SIIit Normalized Summary Innovation Index SIÎit Ranking SIÎit Productivity Innovation Index PIIit Normalized Productivity Innovation Index PIÎit Ranking PIÎit Efficiency Innovation Index EIÎit Ranking EIÎit Bootstrapped Efficiency Innovation Index bEIÎit Ranking bEIÎit Lower bound Upper bound Returns to scale (IRS, CRS, DRS) Austria 0.637 0.488 0.599 0.799 10 1.305 0.519 12 1.000 1 0.878 17 0.679 0.995 CRS Belgium 0.603 0.507 0.627 0.836 7 1.189 0.473 15 0.827 22 0.768 22 0.698 0.822 DRS Bulgaria 0.207 0.095 0.188 0.251 28 2.179 0.866 2 1.000 1 0.886 9 0.687 0.997 CRS Croatia 0.281 0.243 0.306 0.408 23 1.156 0.459 16 1.000 1 0.914 2 0.836 0.994 CRS Cyprus 0.66 0.39 0.501 0.668 14 1.692 0.672 6 1.000 1 0.882 15 0.678 0.995 CRS Czech Rep. 0.497 0.395 0.422 0.563 16 1.258 0.500 14 1.000 1 0.899 4 0.773 0.996 CRS Denmark 0.701 0.63 0.728 0.971 2 1.113 0.442 17 0.920 19 0.849 19 0.771 0.915 DRS Estonia 0.544 0.628 0.502 0.669 13 0.866 0.344 22 0.653 26 0.610 26 0.564 0.650 DRS Finland 0.579 0.694 0.684 0.912 4 0.834 0.331 23 0.704 25 0.652 25 0.590 0.700 DRS France 0.52 0.479 0.571 0.761 11 1.086 0.431 18 1.000 1 0.883 12 0.683 0.996 CRS Germany 0.859 0.631 0.709 0.945 3 1.361 0.541 11 0.839 21 0.780 21 0.718 0.835 DRS Greece 0.516 0.205 0.384 0.512 19 2.517 1.000 1 1.000 1 0.885 10 0.690 0.996 CRS Hungary 0.273 0.304 0.351 0.468 20 0.898 0.357 21 1.000 1 0.888 6 0.751 0.995 CRS Ireland 0.578 0.339 0.606 0.808 9 1.705 0.677 5 1.000 1 0.882 16 0.684 0.996 CRS Italy 0.591 0.299 0.443 0.591 15 1.977 0.785 3 1.000 1 0.887 8 0.740 0.995 CRS Latvia 0.19 0.2 0.221 0.295 27 0.950 0.377 20 1.000 1 0.887 7 0.731 0.995 CRS Lithuania 0.193 0.447 0.289 0.385 24 0.432 0.172 28 0.503 28 0.467 28 0.418 0.501 IRS Luxembourg 0.761 0.461 0.646 0.861 5 1.651 0.656 7 1.000 1 0.882 14 0.689 0.996 CRS Malta 0.391 0.309 0.319 0.425 21 1.265 0.503 13 1.000 1 0.891 5 0.751 0.995 CRS Netherlands 0.538 0.543 0.629 0.839 6 0.991 0.394 19 0.756 24 0.703 24 0.647 0.753 DRS Poland 0.253 0.38 0.279 0.372 25 0.666 0.265 27 0.921 18 0.860 18 0.772 0.918 DRS Portugal 0.566 0.366 0.41 0.547 18 1.546 0.614 8 1.000 1 0.905 3 0.802 0.995 CRS Romania 0.283 0.157 0.237 0.316 26 1.803 0.716 4 1.000 1 0.884 11 0.685 0.995 CRS Slovakia 0.391 0.275 0.318 0.424 22 1.422 0.565 10 1.000 1 0.918 1 0.818 0.995 CRS Slovenia 0.44 0.571 0.513 0.684 12 0.771 0.306 25 0.861 20 0.806 20 0.742 0.856 DRS Spain 0.464 0.315 0.414 0.552 17 1.473 0.585 9 1.000 1 0.882 13 0.700 0.995 CRS Sweden 0.575 0.698 0.75 1.000 1 0.824 0.327 24 0.604 27 0.563 27 0.513 0.601 DRS UK 0.439 0.577 0.613 0.817 8 0.761 0.302 26 0.815 23 0.758 23 0.691 0.811 DRS Country Output (M = 8) Input (N = 4) Summary Innovation Index SIIit Normalized Summary Innovation Index SIÎit Ranking SIÎit Productivity Innovation Index PIIit Normalized Productivity Innovation Index PIÎit Ranking PIÎit Efficiency Innovation Index EIÎit Ranking EIÎit Bootstrapped Efficiency Innovation Index bEIÎit Ranking bEIÎit Lower bound Upper bound Returns to scale (IRS, CRS, DRS) Austria 0.637 0.488 0.599 0.799 10 1.305 0.519 12 1.000 1 0.878 17 0.679 0.995 CRS Belgium 0.603 0.507 0.627 0.836 7 1.189 0.473 15 0.827 22 0.768 22 0.698 0.822 DRS Bulgaria 0.207 0.095 0.188 0.251 28 2.179 0.866 2 1.000 1 0.886 9 0.687 0.997 CRS Croatia 0.281 0.243 0.306 0.408 23 1.156 0.459 16 1.000 1 0.914 2 0.836 0.994 CRS Cyprus 0.66 0.39 0.501 0.668 14 1.692 0.672 6 1.000 1 0.882 15 0.678 0.995 CRS Czech Rep. 0.497 0.395 0.422 0.563 16 1.258 0.500 14 1.000 1 0.899 4 0.773 0.996 CRS Denmark 0.701 0.63 0.728 0.971 2 1.113 0.442 17 0.920 19 0.849 19 0.771 0.915 DRS Estonia 0.544 0.628 0.502 0.669 13 0.866 0.344 22 0.653 26 0.610 26 0.564 0.650 DRS Finland 0.579 0.694 0.684 0.912 4 0.834 0.331 23 0.704 25 0.652 25 0.590 0.700 DRS France 0.52 0.479 0.571 0.761 11 1.086 0.431 18 1.000 1 0.883 12 0.683 0.996 CRS Germany 0.859 0.631 0.709 0.945 3 1.361 0.541 11 0.839 21 0.780 21 0.718 0.835 DRS Greece 0.516 0.205 0.384 0.512 19 2.517 1.000 1 1.000 1 0.885 10 0.690 0.996 CRS Hungary 0.273 0.304 0.351 0.468 20 0.898 0.357 21 1.000 1 0.888 6 0.751 0.995 CRS Ireland 0.578 0.339 0.606 0.808 9 1.705 0.677 5 1.000 1 0.882 16 0.684 0.996 CRS Italy 0.591 0.299 0.443 0.591 15 1.977 0.785 3 1.000 1 0.887 8 0.740 0.995 CRS Latvia 0.19 0.2 0.221 0.295 27 0.950 0.377 20 1.000 1 0.887 7 0.731 0.995 CRS Lithuania 0.193 0.447 0.289 0.385 24 0.432 0.172 28 0.503 28 0.467 28 0.418 0.501 IRS Luxembourg 0.761 0.461 0.646 0.861 5 1.651 0.656 7 1.000 1 0.882 14 0.689 0.996 CRS Malta 0.391 0.309 0.319 0.425 21 1.265 0.503 13 1.000 1 0.891 5 0.751 0.995 CRS Netherlands 0.538 0.543 0.629 0.839 6 0.991 0.394 19 0.756 24 0.703 24 0.647 0.753 DRS Poland 0.253 0.38 0.279 0.372 25 0.666 0.265 27 0.921 18 0.860 18 0.772 0.918 DRS Portugal 0.566 0.366 0.41 0.547 18 1.546 0.614 8 1.000 1 0.905 3 0.802 0.995 CRS Romania 0.283 0.157 0.237 0.316 26 1.803 0.716 4 1.000 1 0.884 11 0.685 0.995 CRS Slovakia 0.391 0.275 0.318 0.424 22 1.422 0.565 10 1.000 1 0.918 1 0.818 0.995 CRS Slovenia 0.44 0.571 0.513 0.684 12 0.771 0.306 25 0.861 20 0.806 20 0.742 0.856 DRS Spain 0.464 0.315 0.414 0.552 17 1.473 0.585 9 1.000 1 0.882 13 0.700 0.995 CRS Sweden 0.575 0.698 0.75 1.000 1 0.824 0.327 24 0.604 27 0.563 27 0.513 0.601 DRS UK 0.439 0.577 0.613 0.817 8 0.761 0.302 26 0.815 23 0.758 23 0.691 0.811 DRS Source: Own elaboration from European Union (2014). Table A2. Year 2015 Country Output (M = 8) Input (N = 4) Summary Innovation Index SIIit Normalized Summary Innovation Index SIÎit Ranking SIÎit Productivity Innovation Index PIIit Normalized Productivity Innovation Index PIÎit Ranking PIÎit Efficiency Innovation Index EIÎit Ranking EIÎit Bootstrapped Efficiency Innovation Index bEIÎit Ranking bEIÎit Lower bound Upper bound Returns to scale (IRS, CRS, DRS) Austria 0.599 0.519 0.585 0.791 11 1.154 0.380 15 1.000 1 0.867 5 0.714 0.993 CRS Belgium 0.566 0.543 0.619 0.836 9 1.044 0.343 18 0.767 22 0.708 22 0.639 0.764 DRS Bulgaria 0.227 0.144 0.229 0.309 27 1.582 0.520 7 1.000 1 0.857 18 0.636 0.993 CRS Croatia 0.244 0.328 0.313 0.423 23 0.744 0.245 26 0.654 25 0.599 25 0.537 0.649 IRS Cyprus 0.540 0.178 0.445 0.601 15 3.040 1.000 1 1.000 1 0.858 16 0.639 0.994 CRS Czech Republic 0.510 0.415 0.447 0.604 14 1.228 0.404 14 1.000 1 0.860 15 0.639 0.994 CRS Denmark 0.728 0.655 0.736 0.995 2 1.112 0.366 16 1.000 1 0.866 7 0.723 0.995 CRS Estonia 0.526 0.688 0.489 0.661 13 0.765 0.252 25 0.622 26 0.562 26 0.504 0.617 DRS Finland 0.603 0.669 0.676 0.914 3 0.902 0.297 21 0.889 21 0.815 21 0.742 0.884 DRS France 0.544 0.502 0.591 0.799 10 1.082 0.356 17 0.944 19 0.867 6 0.797 0.940 DRS Germany 0.723 0.718 0.676 0.914 4 1.007 0.331 19 0.701 24 0.636 24 0.555 0.697 DRS Greece 0.408 0.244 0.365 0.493 21 1.677 0.552 6 1.000 1 0.862 13 0.638 0.995 CRS Hungary 0.289 0.370 0.369 0.499 20 0.782 0.257 24 0.910 20 0.828 20 0.720 0.903 DRS Ireland 0.606 0.352 0.628 0.849 8 1.721 0.566 5 1.000 1 0.862 12 0.707 0.994 CRS Italy 0.572 0.305 0.439 0.593 16 1.873 0.616 4 1.000 1 0.866 8 0.717 0.994 CRS Latvia 0.234 0.385 0.272 0.368 26 0.607 0.200 27 0.524 27 0.477 27 0.410 0.520 IRS Lithuania 0.194 0.441 0.283 0.382 25 0.438 0.144 28 0.395 28 0.359 28 0.311 0.392 IRS Luxembourg 0.772 0.383 0.642 0.868 6 2.014 0.662 2 1.000 1 0.858 17 0.638 0.993 CRS Malta 0.589 0.379 0.397 0.536 18 1.554 0.511 8 1.000 1 0.872 4 0.743 0.994 CRS The Netherlands 0.570 0.452 0.647 0.874 5 1.261 0.415 12 1.000 1 0.861 14 0.678 0.995 CRS Poland 0.298 0.362 0.313 0.423 24 0.823 0.271 23 0.986 17 0.897 2 0.797 0.980 DRS Portugal 0.510 0.365 0.403 0.545 17 1.399 0.460 9 0.944 18 0.865 9 0.780 0.938 DRS Romania 0.223 0.113 0.204 0.276 28 1.976 0.650 3 1.000 1 0.851 19 0.635 0.993 CRS Slovakia 0.398 0.304 0.360 0.486 22 1.311 0.431 11 1.000 1 0.863 11 0.693 0.993 CRS Slovenia 0.523 0.536 0.534 0.722 12 0.974 0.320 20 0.987 16 0.903 1 0.816 0.980 DRS Spain 0.370 0.299 0.385 0.520 19 1.239 0.408 13 1.000 1 0.864 10 0.704 0.994 CRS Sweden 0.620 0.715 0.740 1.000 1 0.867 0.285 22 0.706 23 0.647 23 0.588 0.702 DRS United Kingdom 0.586 0.426 0.636 0.859 7 1.376 0.453 10 1.000 1 0.881 3 0.769 0.994 CRS Country Output (M = 8) Input (N = 4) Summary Innovation Index SIIit Normalized Summary Innovation Index SIÎit Ranking SIÎit Productivity Innovation Index PIIit Normalized Productivity Innovation Index PIÎit Ranking PIÎit Efficiency Innovation Index EIÎit Ranking EIÎit Bootstrapped Efficiency Innovation Index bEIÎit Ranking bEIÎit Lower bound Upper bound Returns to scale (IRS, CRS, DRS) Austria 0.599 0.519 0.585 0.791 11 1.154 0.380 15 1.000 1 0.867 5 0.714 0.993 CRS Belgium 0.566 0.543 0.619 0.836 9 1.044 0.343 18 0.767 22 0.708 22 0.639 0.764 DRS Bulgaria 0.227 0.144 0.229 0.309 27 1.582 0.520 7 1.000 1 0.857 18 0.636 0.993 CRS Croatia 0.244 0.328 0.313 0.423 23 0.744 0.245 26 0.654 25 0.599 25 0.537 0.649 IRS Cyprus 0.540 0.178 0.445 0.601 15 3.040 1.000 1 1.000 1 0.858 16 0.639 0.994 CRS Czech Republic 0.510 0.415 0.447 0.604 14 1.228 0.404 14 1.000 1 0.860 15 0.639 0.994 CRS Denmark 0.728 0.655 0.736 0.995 2 1.112 0.366 16 1.000 1 0.866 7 0.723 0.995 CRS Estonia 0.526 0.688 0.489 0.661 13 0.765 0.252 25 0.622 26 0.562 26 0.504 0.617 DRS Finland 0.603 0.669 0.676 0.914 3 0.902 0.297 21 0.889 21 0.815 21 0.742 0.884 DRS France 0.544 0.502 0.591 0.799 10 1.082 0.356 17 0.944 19 0.867 6 0.797 0.940 DRS Germany 0.723 0.718 0.676 0.914 4 1.007 0.331 19 0.701 24 0.636 24 0.555 0.697 DRS Greece 0.408 0.244 0.365 0.493 21 1.677 0.552 6 1.000 1 0.862 13 0.638 0.995 CRS Hungary 0.289 0.370 0.369 0.499 20 0.782 0.257 24 0.910 20 0.828 20 0.720 0.903 DRS Ireland 0.606 0.352 0.628 0.849 8 1.721 0.566 5 1.000 1 0.862 12 0.707 0.994 CRS Italy 0.572 0.305 0.439 0.593 16 1.873 0.616 4 1.000 1 0.866 8 0.717 0.994 CRS Latvia 0.234 0.385 0.272 0.368 26 0.607 0.200 27 0.524 27 0.477 27 0.410 0.520 IRS Lithuania 0.194 0.441 0.283 0.382 25 0.438 0.144 28 0.395 28 0.359 28 0.311 0.392 IRS Luxembourg 0.772 0.383 0.642 0.868 6 2.014 0.662 2 1.000 1 0.858 17 0.638 0.993 CRS Malta 0.589 0.379 0.397 0.536 18 1.554 0.511 8 1.000 1 0.872 4 0.743 0.994 CRS The Netherlands 0.570 0.452 0.647 0.874 5 1.261 0.415 12 1.000 1 0.861 14 0.678 0.995 CRS Poland 0.298 0.362 0.313 0.423 24 0.823 0.271 23 0.986 17 0.897 2 0.797 0.980 DRS Portugal 0.510 0.365 0.403 0.545 17 1.399 0.460 9 0.944 18 0.865 9 0.780 0.938 DRS Romania 0.223 0.113 0.204 0.276 28 1.976 0.650 3 1.000 1 0.851 19 0.635 0.993 CRS Slovakia 0.398 0.304 0.360 0.486 22 1.311 0.431 11 1.000 1 0.863 11 0.693 0.993 CRS Slovenia 0.523 0.536 0.534 0.722 12 0.974 0.320 20 0.987 16 0.903 1 0.816 0.980 DRS Spain 0.370 0.299 0.385 0.520 19 1.239 0.408 13 1.000 1 0.864 10 0.704 0.994 CRS Sweden 0.620 0.715 0.740 1.000 1 0.867 0.285 22 0.706 23 0.647 23 0.588 0.702 DRS United Kingdom 0.586 0.426 0.636 0.859 7 1.376 0.453 10 1.000 1 0.881 3 0.769 0.994 CRS Source: Own elaboration from European Union (2015). Table A2. Year 2015 Country Output (M = 8) Input (N = 4) Summary Innovation Index SIIit Normalized Summary Innovation Index SIÎit Ranking SIÎit Productivity Innovation Index PIIit Normalized Productivity Innovation Index PIÎit Ranking PIÎit Efficiency Innovation Index EIÎit Ranking EIÎit Bootstrapped Efficiency Innovation Index bEIÎit Ranking bEIÎit Lower bound Upper bound Returns to scale (IRS, CRS, DRS) Austria 0.599 0.519 0.585 0.791 11 1.154 0.380 15 1.000 1 0.867 5 0.714 0.993 CRS Belgium 0.566 0.543 0.619 0.836 9 1.044 0.343 18 0.767 22 0.708 22 0.639 0.764 DRS Bulgaria 0.227 0.144 0.229 0.309 27 1.582 0.520 7 1.000 1 0.857 18 0.636 0.993 CRS Croatia 0.244 0.328 0.313 0.423 23 0.744 0.245 26 0.654 25 0.599 25 0.537 0.649 IRS Cyprus 0.540 0.178 0.445 0.601 15 3.040 1.000 1 1.000 1 0.858 16 0.639 0.994 CRS Czech Republic 0.510 0.415 0.447 0.604 14 1.228 0.404 14 1.000 1 0.860 15 0.639 0.994 CRS Denmark 0.728 0.655 0.736 0.995 2 1.112 0.366 16 1.000 1 0.866 7 0.723 0.995 CRS Estonia 0.526 0.688 0.489 0.661 13 0.765 0.252 25 0.622 26 0.562 26 0.504 0.617 DRS Finland 0.603 0.669 0.676 0.914 3 0.902 0.297 21 0.889 21 0.815 21 0.742 0.884 DRS France 0.544 0.502 0.591 0.799 10 1.082 0.356 17 0.944 19 0.867 6 0.797 0.940 DRS Germany 0.723 0.718 0.676 0.914 4 1.007 0.331 19 0.701 24 0.636 24 0.555 0.697 DRS Greece 0.408 0.244 0.365 0.493 21 1.677 0.552 6 1.000 1 0.862 13 0.638 0.995 CRS Hungary 0.289 0.370 0.369 0.499 20 0.782 0.257 24 0.910 20 0.828 20 0.720 0.903 DRS Ireland 0.606 0.352 0.628 0.849 8 1.721 0.566 5 1.000 1 0.862 12 0.707 0.994 CRS Italy 0.572 0.305 0.439 0.593 16 1.873 0.616 4 1.000 1 0.866 8 0.717 0.994 CRS Latvia 0.234 0.385 0.272 0.368 26 0.607 0.200 27 0.524 27 0.477 27 0.410 0.520 IRS Lithuania 0.194 0.441 0.283 0.382 25 0.438 0.144 28 0.395 28 0.359 28 0.311 0.392 IRS Luxembourg 0.772 0.383 0.642 0.868 6 2.014 0.662 2 1.000 1 0.858 17 0.638 0.993 CRS Malta 0.589 0.379 0.397 0.536 18 1.554 0.511 8 1.000 1 0.872 4 0.743 0.994 CRS The Netherlands 0.570 0.452 0.647 0.874 5 1.261 0.415 12 1.000 1 0.861 14 0.678 0.995 CRS Poland 0.298 0.362 0.313 0.423 24 0.823 0.271 23 0.986 17 0.897 2 0.797 0.980 DRS Portugal 0.510 0.365 0.403 0.545 17 1.399 0.460 9 0.944 18 0.865 9 0.780 0.938 DRS Romania 0.223 0.113 0.204 0.276 28 1.976 0.650 3 1.000 1 0.851 19 0.635 0.993 CRS Slovakia 0.398 0.304 0.360 0.486 22 1.311 0.431 11 1.000 1 0.863 11 0.693 0.993 CRS Slovenia 0.523 0.536 0.534 0.722 12 0.974 0.320 20 0.987 16 0.903 1 0.816 0.980 DRS Spain 0.370 0.299 0.385 0.520 19 1.239 0.408 13 1.000 1 0.864 10 0.704 0.994 CRS Sweden 0.620 0.715 0.740 1.000 1 0.867 0.285 22 0.706 23 0.647 23 0.588 0.702 DRS United Kingdom 0.586 0.426 0.636 0.859 7 1.376 0.453 10 1.000 1 0.881 3 0.769 0.994 CRS Country Output (M = 8) Input (N = 4) Summary Innovation Index SIIit Normalized Summary Innovation Index SIÎit Ranking SIÎit Productivity Innovation Index PIIit Normalized Productivity Innovation Index PIÎit Ranking PIÎit Efficiency Innovation Index EIÎit Ranking EIÎit Bootstrapped Efficiency Innovation Index bEIÎit Ranking bEIÎit Lower bound Upper bound Returns to scale (IRS, CRS, DRS) Austria 0.599 0.519 0.585 0.791 11 1.154 0.380 15 1.000 1 0.867 5 0.714 0.993 CRS Belgium 0.566 0.543 0.619 0.836 9 1.044 0.343 18 0.767 22 0.708 22 0.639 0.764 DRS Bulgaria 0.227 0.144 0.229 0.309 27 1.582 0.520 7 1.000 1 0.857 18 0.636 0.993 CRS Croatia 0.244 0.328 0.313 0.423 23 0.744 0.245 26 0.654 25 0.599 25 0.537 0.649 IRS Cyprus 0.540 0.178 0.445 0.601 15 3.040 1.000 1 1.000 1 0.858 16 0.639 0.994 CRS Czech Republic 0.510 0.415 0.447 0.604 14 1.228 0.404 14 1.000 1 0.860 15 0.639 0.994 CRS Denmark 0.728 0.655 0.736 0.995 2 1.112 0.366 16 1.000 1 0.866 7 0.723 0.995 CRS Estonia 0.526 0.688 0.489 0.661 13 0.765 0.252 25 0.622 26 0.562 26 0.504 0.617 DRS Finland 0.603 0.669 0.676 0.914 3 0.902 0.297 21 0.889 21 0.815 21 0.742 0.884 DRS France 0.544 0.502 0.591 0.799 10 1.082 0.356 17 0.944 19 0.867 6 0.797 0.940 DRS Germany 0.723 0.718 0.676 0.914 4 1.007 0.331 19 0.701 24 0.636 24 0.555 0.697 DRS Greece 0.408 0.244 0.365 0.493 21 1.677 0.552 6 1.000 1 0.862 13 0.638 0.995 CRS Hungary 0.289 0.370 0.369 0.499 20 0.782 0.257 24 0.910 20 0.828 20 0.720 0.903 DRS Ireland 0.606 0.352 0.628 0.849 8 1.721 0.566 5 1.000 1 0.862 12 0.707 0.994 CRS Italy 0.572 0.305 0.439 0.593 16 1.873 0.616 4 1.000 1 0.866 8 0.717 0.994 CRS Latvia 0.234 0.385 0.272 0.368 26 0.607 0.200 27 0.524 27 0.477 27 0.410 0.520 IRS Lithuania 0.194 0.441 0.283 0.382 25 0.438 0.144 28 0.395 28 0.359 28 0.311 0.392 IRS Luxembourg 0.772 0.383 0.642 0.868 6 2.014 0.662 2 1.000 1 0.858 17 0.638 0.993 CRS Malta 0.589 0.379 0.397 0.536 18 1.554 0.511 8 1.000 1 0.872 4 0.743 0.994 CRS The Netherlands 0.570 0.452 0.647 0.874 5 1.261 0.415 12 1.000 1 0.861 14 0.678 0.995 CRS Poland 0.298 0.362 0.313 0.423 24 0.823 0.271 23 0.986 17 0.897 2 0.797 0.980 DRS Portugal 0.510 0.365 0.403 0.545 17 1.399 0.460 9 0.944 18 0.865 9 0.780 0.938 DRS Romania 0.223 0.113 0.204 0.276 28 1.976 0.650 3 1.000 1 0.851 19 0.635 0.993 CRS Slovakia 0.398 0.304 0.360 0.486 22 1.311 0.431 11 1.000 1 0.863 11 0.693 0.993 CRS Slovenia 0.523 0.536 0.534 0.722 12 0.974 0.320 20 0.987 16 0.903 1 0.816 0.980 DRS Spain 0.370 0.299 0.385 0.520 19 1.239 0.408 13 1.000 1 0.864 10 0.704 0.994 CRS Sweden 0.620 0.715 0.740 1.000 1 0.867 0.285 22 0.706 23 0.647 23 0.588 0.702 DRS United Kingdom 0.586 0.426 0.636 0.859 7 1.376 0.453 10 1.000 1 0.881 3 0.769 0.994 CRS Source: Own elaboration from European Union (2015). © The Author(s) 2018. Published by Oxford University Press. All rights reserved. For permissions, please email: journals.permissions@oup.com This article is published and distributed under the terms of the Oxford University Press, Standard Journals Publication Model (https://academic.oup.com/journals/pages/about_us/legal/notices) http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Research Evaluation Oxford University Press

On the meaning of innovation performance: Is the synthetic indicator of the Innovation Union Scoreboard flawed?

Research Evaluation , Volume Advance Article (3) – May 3, 2018

Loading next page...
 
/lp/ou_press/on-the-meaning-of-innovation-performance-is-the-synthetic-indicator-of-vVHJe0cglF
Publisher
Oxford University Press
Copyright
© The Author(s) 2018. Published by Oxford University Press. All rights reserved. For permissions, please email: journals.permissions@oup.com
ISSN
0958-2029
eISSN
1471-5449
D.O.I.
10.1093/reseval/rvy011
Publisher site
See Article on Publisher Site

Abstract

Abstract The European Union (EU) annually publishes an Innovation Union Scoreboard (IUS) as a tool to measure the innovation performance of EU Member States by means of a composite index, called the Summary Innovation Index (SII). The SII is constituted by an average of 25 indicators. The SII is claimed to rank Member States according to their innovation performance. This means that the higher the average value of the 25 indicators, the better the innovation performance is said to be. The first purpose of this article is to assess whether the SII constitutes a meaningful measure of innovation performance. Our conclusion is that it does not. Our second purpose is to develop alternative, productivity or efficiency-based, measures of innovation system performance based on a simple index number, and complement it with advanced and robust nonparametric Data Envelopment Analysis techniques. By doing so, the article offers a critical review of the SII, and proposes to put more emphasis on the identification of and relation between input and output innovation indicators. The data provided by the 2014 and 2015 editions of the IUS are here used to analyze the innovation performance of all 28 EU national innovation systems. A theoretical background and reasons for selecting the indicators used are presented, and our new ranking of the innovation performance using bias-corrected efficiency scores of all EU countries is calculated. We find that the results differ substantially between the SII and the ranking based on our method, with significant consequences for the design of innovation policies. 1. Introduction The European Union (EU) has set ambitious objectives in five areas to be reached by 2020. In addition to climate and energy, education, employment, and social inclusion, innovation is one of the five pillars to form a so-called ‘Innovation Union’ (European Commission 2013). To support the establishment of the Innovation Union, the European Commission is using the Innovation Union Scoreboard (IUS). The objective of the IUS is to provide a ‘comparative assessment’ of the ‘innovation performance’ of EU Member States (European Commission 2011: 3). To assess the ‘innovation performance’ of the Member States, a Summary Innovation Index (SII) is calculated by the IUS. The SII synthetizes the 25 indicators included in the IUS, through their arithmetic average, into a single measure. As a result, the SII ranks all EU Member States based on what is explicitly called ‘EU Member States’ Innovation Performance’ (European Union 2014: 5). The first aim of this article is to assess whether the SII constitutes a meaningful measure of innovation performance, as argued by the IUS. Following Zabala-Iturriagagoitia et al. (2007a) or Nasierowski and Arcelus (2012), among others, the second aim is to develop alternative, productivity or efficiency-based, measures of innovation performance. By doing so, we provide a critical review of the SII, propose to place more emphasis on the identification of and relation between innovation input and output indicators, and on this basis we provide an alternative assessment and ranking of innovation performance. When developing an alternative approach to the one used by IUS, we use exclusively and exactly the same data as provided by the IUS 2014 and 2015 to assess and compare the performance of all EU28 national innovation systems. We reexamine these data using our different approach, which relies on a productivity—or relative efficiency—rationale. We do not discuss the quality and accuracy of the IUS data. We simply use the existing data provided by the IUS but seen through a different (productivity) lens, with the purpose of increasing the breadth of sight of innovation policy makers and politicians.1 A similar efficiency approach has been used by the Global Innovation Index (GII). The ‘Innovation Efficiency Ratio’ provided by the GII is ‘the ratio of the Output Sub-Index score over the Input Sub-Index score. It shows how much innovation output a given country is getting for its inputs’ (Dutta et al. 2017). The GII aggregates 81 different indicators for 143 countries, using different subindices to aggregate them. Following the theoretical bases used in productivity and efficiency measurement methods (Farrell 1957; Coelli et al. 2005), we single out a number of input (N = 4) and output (M = 8) innovation indicators from the 25 included in the IUS editions of 2014 and 2015. They are used to compare the innovation outputs with the innovation inputs of each of the EU28 countries, so as to get a measure of the performance of the innovation systems (i.e. the relationship between the innovation inputs and outputs). We conclude that the SII does not constitute a meaningful measure of innovation performance and that this indicator is not useful from the point of view of innovation policy design. Our alternative approach results in a radically different ranking of EU Member States’ innovation performance. Our performance and ranking results, however, lead to counterintuitive conclusions, which we discuss and evaluate. This means that the analysis has to be refined and developed in future research. The rest of the article is organized as follows. Section 2 discusses the concept of innovation performance and presents the rationale for the approach we have chosen. In Section 3 we present the index number and DEA methods and discuss the selection of output and input indicators. The empirical analysis is presented in Section 4. There, we use the normalized IUS scores for each of the selected indicators, provide the rankings for both innovation inputs and outputs, and calculate the efficiency of all 28 innovation systems by relating the innovation outputs and inputs to each other through the index number and robust DEA scores. Section 5 provides a discussion of the main findings and its relevance for the practice of innovation policy making, and also makes some proposals for future research. 2. Theoretical background During the past decade an increasing literature has dealt with the development and use of indicators to improve the measurement and characterization of innovation systems (e.g. Archibugi, Denni, and Filippetti 2009). The European Commission has been one of the most active agents in this sense, with the development of the European Innovation Scoreboard (since 2011, IUS) and the implementation of the Community Innovation Surveys. As indicated in Section 1, the IUS claims that the SII calculated is measuring innovation performance of EU Member States. But what is then meant by ‘innovation performance’? Let us discuss this concept by a metaphor. Let us imagine that two countries are trying to send a rocket to the moon, and both succeed. However, the first required $1 billion to achieve that goal, while the second required $1. If only outputs are considered, both countries would have achieved the same level of ‘performance’. However, it can be strongly argued that the second country might have achieved a remarkably higher performance, since the amount of resources used in the two cases is so unalike. This approach is also shared by Carayannis and Grigoroudis (2014) and Liu, Lu, and Ho (2014: 318) who consider that the measurement of the performance of an innovation system should be accomplished through ‘a linear process-oriented approach, whereby a nation’s innovation system is treated as a process in which certain output factors… are seen as products produced from a set of input resources’. This means that if we are to compare performance in a meaningful way the achievement (e.g. landing on the moon) must be related to the resources used (e.g. a budget constraint). In other words, outputs must be related to inputs in some way. In economics, productivity is defined as the value of output produced per unit of input, e.g. labor productivity is total production divided by employment. This is similar to measuring performance of innovation systems in terms of productivity or efficiency. From the policy evaluation literature we have learned that (innovation) policies must be context specific. These policies will define the goals of the innovation policy, identify the problems in the innovation system, and specify or develop and implement the instruments to mitigate the problems and thereby achieve the objectives (Edquist 2018). In other words, the policy must define where to go and how to get there. However, if we do not know where we are, how can we get anywhere? Indeed, it is in fact very difficult to improve what cannot be measured. It is not possible to say whether certain innovation intensities are high or low in a specific system if there is no comparison with those in other systems (Edquist 2011). This has to do with the fact that we cannot identify optimal or ideal innovation intensities (just as we cannot specify an optimal innovation system). There appears to be general support ‘to the premise that all performance evaluations involve comparisons’ (Mersha 1989: 163). Hence the measurement of innovation performance must be comparative between existing systems (Zabala-Iturriagagoitia et al. 2007b). Only then can they be helpful to policy design by identifying problems to be solved or mitigated by the implementation of policy instruments. In spite of the flags often raised in the literature as to the misuse of synthetic indicators in policy making spheres (e.g. Grupp and Schubert 2010), composite indicators are repeatedly used for the design of innovation policies (Saltelli 2007). One of the risks of using synthetic indicators is that, once they are used and accepted, their abuse makes them become policy targets, and hence, loose their meaning as a tool to explain certain realities. Saltelli refers to the work by Nardo et al. (2005) when discussing the steps required before creating a composite indicator.2 The SII falls short in meeting several of those steps. We claim that to assess the innovation performance of any system, the performance measure should be based on a productivity relation, defined as a ratio of aggregate innovation outputs (numerator) to aggregate innovation inputs (denominator). The most simple way of doing so is to classify indicators as either inputs or outputs, combine them using an aggregator function (e.g. the arithmetic average), and then relate one to another by using a simple fraction. A more advanced method relies on robust nonparametric data envelopment analysis (DEA) techniques to calculate a productivity index for each country using individual—most favorable—aggregating functions, which also allow accounting for the nature of returns to scale (increasing, constant, or decreasing). In this article we use both methods. An important difference between the simple aggregate productivity index and its DEA associate is that the former, being a basic index number, can be easily calculated with low effort and high immediacy (e.g. using a spreadsheet), while the latter relays on linear programing techniques that require solver optimization with bootstrapping. But, in return, the latter yields relevant information about the innovation processes that are evaluated (benchmark peers for inefficient observations, nature of returns to scale, etc.) as well as the statistical significance of the productivity estimates. 3. Methodology and relevant indicators The IUS identifies ιl, l = 1,…, 25, indicators, which are divided into three categories and eight dimensions (see Fig. 1). The three categories considered are Enablers, Firm activities, and Outputs. According to the IUS, the enablers ‘capture the main drivers of innovation performance external to the firm’ (European Union 2014: 4) and cover three innovation dimensions: human resources, open, excellent and attractive research systems, and finance and support. Firm activities ‘capture the innovation efforts at the level of the firm’ (ibid) and are also grouped in three innovation dimensions: firm investments, linkages and entrepreneurship, and intellectual assets. Finally, outputs cover ‘the effects of firms’ innovation activities’ (ibid) in two innovation dimensions: innovators and economic effects. Figure 1. View largeDownload slide Measurement framework of the IUS. Source:European Union (2014: 8). Figure 1. View largeDownload slide Measurement framework of the IUS. Source:European Union (2014: 8). From a numerical perspective, to adjust the values of each indicator observed in country i in period t to a notionally common scale, the IUS normalizes each series so their values belong to the following range: ι^lit∈[0,1].3 Subsequently, the IUS calculates the SII using the arithmetic mean as aggregating function, in which all indicators are given the same weight υl: SIIit=∑l=125υ¯lι^lit, υ¯l=1/25, ι^lit∈[0,1]. (1) Note that the SIIit index does not necessarily yield a distribution whose maximum value is equal to 1, thereby identifying the most productive innovation system with such value. To ensure this desirable property that is helpful to highlight best performance, and rank innovation systems against a single scalar—and that is shared with the forthcoming productivity indices, we normalize SIIit by dividing by the maximum observed value across the set of j = 1,…, J countries: SIÎit=∑l=125υ¯lι^litmax j(SIIjt), υ¯l=1/25, j=1,…J, SIÎit∈[0,1]. (2) The IUS draws the conclusion that the country with the highest SIIit―or SIÎit―is also the best innovation performer regardless of whether the indicators used measure the input or output side of innovation or something else. In addition, the IUS provides no explicit definition of innovation performance, which is quite surprising, since this is the most central concept in the scoreboard reports. However, implicitly and in practice, the SII score can be said to be the IUS definition of innovation performance. On this basis, EU Member States are ranked according to the SII in the following groups: innovation leaders (more than 20% above EU average), innovation followers (less than 20% above, or more than 90% of the EU average), moderate innovators (relative performance rates between 50 and 90% of the EU average), and modest innovators (less than 50% of the EU average) (European Union 2014: 11). 3.1 Identification of input and output indicators We have argued that innovation performance must be measured as a ratio between a numerator and a denominator. Our way of doing so is to classify indicators as either inputs or outputs and then relate them to one another. We will now discern which of the 25 IUS indicators in Fig. 1 can be qualified as inputs, as outputs, or as something else (e.g. indicators measuring intermediate phenomena, or consequences of innovations). Following the guidelines provided by the Oslo Manual (OECD/Eurostat 2005), we define input and output indicators as follows: Innovation input indicatorsrefer to the resources (human, material, and financial; private as well as governmental) used not only to create innovations but also to bring them to the market. Innovation output indicatorsrefer to new products and processes, new designs, and community trademarks, as well as marketing and organizational innovations, which are connected to the market, and which can either be new to the world, new to the industry, and/or new to the firm. We classify eight IUS indicators as measuring innovation output and four as measuring innovation input. The remaining 13 indicators from the IUS are certainly not irrelevant. In fact, many of these indicators can be regarded as determinants of innovation processes. However, further research is needed on the factors influencing (fostering or hindering) the development and the diffusion of innovations (see Edquist 2005). When more evidence is available about these determinants, the view on what indicators can be regarded as inputs or outputs will certainly have to be adapted. Below we provide our reasons to include some indicators and not others in these categories. Below we list the eight output indicators considered in this article.4 2.2.1: Small and medium-sized enterprises (SMEs) innovating in-house (% of SMEs) 2.3.3: Community trademarks per billion gross domestic product (GDP) (in Purchasing Parity Power (PPP)€) 2.3.4: Community designs per billion GDP (in PPP€) 3.1.1: SMEs introducing product or process innovations (% of SMEs) 3.1.2: SMEs introducing marketing or organizational innovations (% of SMEs) 3.2.2: Contribution of medium and high-tech products exports to the trade balance 3.2.3: Knowledge-intensive services exports (as % of total service exports) 3.2.4: Sales of new-to-market and new-to-firm innovations (as % of turnover) The notion of innovation output, according to our definition, is partly different than the IUS category of ‘outputs’, which is defined as ‘the effects of firms’ innovation activities’ (European Union 2014: 4).5 We contend that indicators 2.2.1, 2.3.3, and 2.3.4 should be categorized as output indicators, despite they are classified in the IUS as ‘firm activities’ rather than ‘outputs’. For example, the indicator 2.2.1, ‘SMEs innovating in-house’, refers to the SMEs that have succeeded in the introduction of new or significantly improved products and/or processes, and which may have been innovated inside the company. Therefore, this indicator is in fact capturing an output of an innovation system. Similar arguments hold for indicators 2.3.3 ‘Community trademarks per billion GDP’ and 2.3.4 ‘Community designs per billion GDP’. We consider that community trademarks and designs are significant aspects of actual product innovations, as they signal intellectual property rights related to a specific new product that is connected to the market (see also Mendonça, Santos Pereira, and Mira Godinho 2004). Indicators 3.1.1 through 3.2.5 are considered to be ‘outputs’ by the IUS. However, a conceptual difference exists between the label ‘outputs’ as used in the IUS and the concept of innovation output used in this article. It is for this reason that we do not classify as innovation outputs the following three indicators referred to in the IUS as ‘outputs’: 3.1.3, ‘Employment in fast-growing firms of innovative sectors’; 3.2.1, ‘Employment in knowledge-intensive activities’; and 3.2.5, ’License and patent revenues from abroad’. Indicators 3.1.3 and 3.2.1 measure employment. Employment may be a consequence of innovations or the result of other economic forces rather than an innovation as such—just as with economic growth. As to the indicator 3.2.5, licenses and patent revenues, it refers to sales of intellectual property rights, and cannot be considered an indicator of innovation output according to the previous definition.6 Classifications of individual indicators in a certain category (outputs here) can certainly be questioned and discussed. We therefore carried out a sensitivity analysis, which is reported in Section 4.1, in which we classified 12 indicators instead of 8 as innovation outputs.7 Looking further at the measurement framework of the IUS (Fig. 1), it becomes clear that while one of the main categories is considered to be a measure of innovation Outputs, there is no category explicitly referring to innovation inputs, or providing a clear specification of what such inputs could be. We consider four indicators to fulfill the requirements for being classified as innovation inputs—see below. Two of these indicators are categorized in the IUS as ‘enablers’ and two as ‘firm activities’. Two measure R&D expenditures from the public and private sector, which are both important determinants of innovation.8 Venture capital, which is important ‘for the relative dynamism of new business creation’ (European Union 2014: 87), is especially needed for risk and cost-intensive innovation and is also required to enhance innovation through commercialization of R&D results. In addition to R&D-intensive investments, companies need to invest in non-R&D innovation expenditures as well. Below we list the four input indicators considered. 1.3.1: R&D expenditure in the public sector (% of GDP) 1.3.2: Venture capital (% of GDP) 2.1.1: R&D expenditure in the business sector (% of GDP) 2.1.2: Non-R&D innovation expenditures (% of turnover) We carried out a sensitivity analysis also for the inputs (see Section 4.2), in which we classified seven indicators instead of four as input indicators.9 The four input indicators proposed above are linked to innovation activities and are undertaken to enhance innovation, at least in part, as the Oslo Manual highlights. There are, of course, other determinants of innovation processes. Ideally we should include all such determinants as input indicators. However, we would then need a fully articulated systemic and holistic approach in which all determinants of innovation processes, the feedback loops among them, and in which the relative importance of the different determinants of innovation processes are accounted for (Edquist 2014a, 2014c, 2018). That we do not have (yet). Admittedly, this is unsatisfactory, but a fact. For example, in the IUS (and in this article) no account is taken of determinants of innovation processes operating from the demand side (see Edquist et al. 2015). If all innovation input and all innovation output indicators were included, we would be able to calculate something corresponding to total factor productivity (or multifactor productivity) in the field of innovation. As indicated, we will be satisfied here with a limited number of indicators on both sides (i.e. we will only be able to provide a partial measure of innovation performance). 3.2 Productivity measures based on simple index numbers and robust DEA Once the indicators that we deem most relevant for the purposes of this article have been selected, we gather all the data from the IUS 2014 and 2015, to calculate both the conventional and normalized productivity measures for all EU28 countries. As a subset of the previous ι^lit normalized indicators in (1), and following the preceding discussion, we denote innovation input indicators of country i in period t by x^it= (x^1it,…,x^Nit) ∈ R+N, while the innovation outputs indicators are represented by the vector y^it= (y^1it,…,y^Mit) ∈ R+M—in the selected model, N = 4 and M = 8. Relying on the simplest index number definition, the Productivity Innovation Index (PII) is calculated as the relation (i.e. ratio) between the normalized input and output indicators― y^mit∈[0,1], x^nit∈[0,1], using once again the arithmetic mean as aggregating function; i.e. PIIit=∑m=1Mμ¯my^mit∑n=1Nν¯nx^nit, μ¯m=1/M, ν¯n=1/N,  (3) where ν¯n and μ¯m are the input and output weights. As its SIIit counterpart, (3) must be normalized by the maximum observed value across all J countries, so its value is bounded above by 1. PIÎit=PIIitmax j(PIIjt), j=1,…,J,  PIÎit∈[0,1]. (4) When compared to (2), a high score for the input indicators in the denominator means that a great deal of effort and resources has been devoted to stimulating innovation. Similarly, a high score for the output indicators in the numerator shows a high production of innovations. However, if the input side is, relatively speaking, much larger than the output side, the performance of the system as a whole is low (i.e. note the moon project). This implies that the efforts to produce or stimulate innovation have not led to a corresponding actual production of innovations. The different interpretations of SIÎit in (2) as an average measure of indiscriminate innovation indicators and the productivity measure PIÎit above, should now be clear. The properties of PIÎit can be easily determined by resorting to index number theory. Balk (2008) discusses the desirable properties that a productivity index should fulfill from the so-called axiomatic or test approach (monotonicity, homogeneity, identity, proportionality, etc.). Nevertheless, we are aware of the potential limits that using a deterministic formulation like (4) entails for the evaluation of innovation performance. Potential problems are the existence of sampling variation and measurement errors that may polarize results and bias the productivity values and associated rankings. For this reason we propose the use of an alternative approach that allows addressing the bias emanating from the above issues. This approach extends with bootstrapping statistical techniques the nonparametric DEA methods for assessing efficiency already used by Zabala-Iturriagagoitia et al. (2007a). Bootstrapped DEA constitutes a sophisticated method to estimate innovation productivity, differing from the simple index number aggregate in a subtle and fundamental way. Through mathematical programming, in each run the input and output weights νn and μm are now the result of the optimization process that maximizes a country’s productivity relative to its counterparts. We can rely on the seminal contribution by Charnes, Cooper, and Rhodes (1978) to illustrate the productivity interpretation of DEA. Indeed, based on the normalized indicators, their original ratio-form formulation, known as CCR, computes the productivity of country’s i innovation system in the following way:10 max vmt,μmt ∑m=1Mμmty^mit∑n=1Nνntx^nit=EIÎit∈(0,1], (5) s.t.  ∑m=1Mμmty^mjt∑n=1Nνntx^njt≤1, j=1,…,J,μmt≥0, νnt≥0, where  νnt* and μmt* denote the optimal input and output weights when evaluating the relative productivity with respect to all j = 1,…, J countries, including itself. Contrary to the equal weights formulations (2) and (3), program (5) identifies the most favorable inputs and outputs weights that result in the maximum feasible productivity level of ( xit, yit) relative to that of the remaining innovation systems. Note that since the weights  νnt and μmt constitute aggregator functions, both the objective function and set of j = 1,…, J restrictions represent proper definitions of productivity—a ratio of aggregate output to aggregate input, normalizing maximum productivity to 1. This normalization also allows to deem the solution to (5) as a relative (in)efficiency measure, justifying the efficiency innovation index notation: EIÎit. When country i under evaluation maximizes relative productivity or efficiency, EIÎit = 1. On the contrary, if EIÎit < 1, the country does not maximize the relative productivity of its innovation system, and the lower the efficiency value, the worse the innovation performance, as in the previous indicators. The productivity values calculated from (5) can be considered as estimates, since their values are subject to uncertainty due to sampling variation, measurement errors, etc. Simar and Wilson (1998) introduce bootstrap methods that, based on resampling, provide estimation bias, confidence intervals and allow hypotheses testing. We rely on the algorithms presented by these authors, following Daraio and Simar (2007) and Bogetoft and Otto (2011), to test the statistical significance of the attained results. The bootstrap algorithm can be summarized in the following steps: Selection of B independent bootstrap samples—drawn from the original data set with replacements (i.e. 2000); Calculate an initial estimate for the productivity or efficiency score of each country EIÎitwith respect to each bootstrapped sample and smooth their distribution by perturbating them with a random noise generated from a kernel density function with scale given with bandwith h; Correct the original estimates for the mean and variance of the smoothed values; Obtain a second set of bootstrapped samples generating inefficient countries inside the DEA attainable set and conditional on the original input—or output—mix; Repeat the process and estimate the efficiency scores for each original country with respect to that second set, so as to obtain a set of B bootstrap estimates; and, finally, Based on this distribution calculate the threshold values that truncate it according to the predetermined significance value α (i.e. 0.05), so as to determine the confidence intervals for the efficiency score of each country. In addition, the bootstrapped scores can be used to obtain an estimate of the bias of the true efficiency value, and thereby a bias-corrected estimator, which we denote by bEIÎit. In the empirical section the bias-corrected productivity values are presented along with their corresponding confidence intervals. 4. Empirical analysis This section assesses the innovation performance of the Swedish national innovation system in relation to the rest of the EU28 Member States. However, what follows in this section is just an example of how our approach can be developed with regard to one country. 4.1 Output orientation Table 1 summarizes the normalized scores for the eight output indicators considered, and the relative position that Sweden holds in relation to the EU28 countries for the latest year for which data are available. It also gives an average ranking and normalized score for Sweden for all output indicators. Table 1. The innovation output indicators of the Swedish national innovation system Indicator Score in 2014 [2015] Ranking (out of 28) in 2014 Ranking (out of 28) in 2015 EU 28 average in 2014 [2015] Leading countries (top 3) in 2014 Leading countries (top 3) in 2015 2.2.1 SMEs innovating in-house as % of SMEs 0.729 [0.779] 8 4 0.570 [0.513] Germany (0.933) Cyprus (0.833) Denmark (0.813) The Netherlands (0.797) Ireland (0.792) Germany (0.787) 2.3.3 Community trademarks per billion GDP (in PPP-€) 0.573 [0.661] 7 8 0.444 [0.580] Luxembourg (1.0) Cyprus (1.0) Malta (1.0) Cyprus (1.000) Luxembourg (1.000) Malta (1.000) 2.3.4 Community designs per billion GDP (in PPP-€) 0.574 [0.999] 8 3 0.566 [0.569] Luxembourg (1.0) Austria (1.0) Denmark (0.971) Denmark (1.000) Luxembourg (1.000) Sweden (0.999) 3.1.1 SMEs introducing product or process innovations as % of SMEs 0.781 [0.656] 4 6 0.577 [0.432] Germany (1.0) Belgium (0.848) Luxembourg (0.792) Luxembourg (0.732) Germany (0.717) Belgium (0.713) 3.1.2 SMEs introducing marketing or organizational innovations as % of SMEs 0.605 [0.540] 10 12 0.566 [0.495] Germany (1.0) Luxembourg (0.960) Greece (0.801) Luxembourg (0.851) Ireland (0.797) Germany (0.720) 3.2.2 Contribution of medium and high-tech product exports to trade balance 0.579 [0.648] 15 9 0.553 [0.658] Germany (0.930) Slovenia (0.802) Hungary (0.756) Hungary (0.899) Germany (0.892) Slovakia (0.850) 3.2.3 Knowledge-intensive services exports as % total service exports 0.510 [0.524] 10 11 0.606 [0.665] Ireland (1.0) Luxembourg (1.0) Denmark (0.959) Ireland (1.000) Luxembourg (1.000) Denmark (1.000) 3.2.4 Sales of new-to-market and new-to-firm innovations as % of turnover 0.248 [0.156] 21 24 0.664 [0.488] Greece (1.0) Slovakia (1.0) Spain (0.982) Denmark (1.000) Slovakia (0.869) Spain (0.590) Average output result12 0.575 [0.620] 10 4 0.568 [0.550] Germany (0.859) Luxembourg (0.754) Denmark (0.701) Luxembourg (0.772) Denmark (0.728) Germany (0.723) Indicator Score in 2014 [2015] Ranking (out of 28) in 2014 Ranking (out of 28) in 2015 EU 28 average in 2014 [2015] Leading countries (top 3) in 2014 Leading countries (top 3) in 2015 2.2.1 SMEs innovating in-house as % of SMEs 0.729 [0.779] 8 4 0.570 [0.513] Germany (0.933) Cyprus (0.833) Denmark (0.813) The Netherlands (0.797) Ireland (0.792) Germany (0.787) 2.3.3 Community trademarks per billion GDP (in PPP-€) 0.573 [0.661] 7 8 0.444 [0.580] Luxembourg (1.0) Cyprus (1.0) Malta (1.0) Cyprus (1.000) Luxembourg (1.000) Malta (1.000) 2.3.4 Community designs per billion GDP (in PPP-€) 0.574 [0.999] 8 3 0.566 [0.569] Luxembourg (1.0) Austria (1.0) Denmark (0.971) Denmark (1.000) Luxembourg (1.000) Sweden (0.999) 3.1.1 SMEs introducing product or process innovations as % of SMEs 0.781 [0.656] 4 6 0.577 [0.432] Germany (1.0) Belgium (0.848) Luxembourg (0.792) Luxembourg (0.732) Germany (0.717) Belgium (0.713) 3.1.2 SMEs introducing marketing or organizational innovations as % of SMEs 0.605 [0.540] 10 12 0.566 [0.495] Germany (1.0) Luxembourg (0.960) Greece (0.801) Luxembourg (0.851) Ireland (0.797) Germany (0.720) 3.2.2 Contribution of medium and high-tech product exports to trade balance 0.579 [0.648] 15 9 0.553 [0.658] Germany (0.930) Slovenia (0.802) Hungary (0.756) Hungary (0.899) Germany (0.892) Slovakia (0.850) 3.2.3 Knowledge-intensive services exports as % total service exports 0.510 [0.524] 10 11 0.606 [0.665] Ireland (1.0) Luxembourg (1.0) Denmark (0.959) Ireland (1.000) Luxembourg (1.000) Denmark (1.000) 3.2.4 Sales of new-to-market and new-to-firm innovations as % of turnover 0.248 [0.156] 21 24 0.664 [0.488] Greece (1.0) Slovakia (1.0) Spain (0.982) Denmark (1.000) Slovakia (0.869) Spain (0.590) Average output result12 0.575 [0.620] 10 4 0.568 [0.550] Germany (0.859) Luxembourg (0.754) Denmark (0.701) Luxembourg (0.772) Denmark (0.728) Germany (0.723) Source: Own elaboration based on European Union (2014, 2015). Table 1. The innovation output indicators of the Swedish national innovation system Indicator Score in 2014 [2015] Ranking (out of 28) in 2014 Ranking (out of 28) in 2015 EU 28 average in 2014 [2015] Leading countries (top 3) in 2014 Leading countries (top 3) in 2015 2.2.1 SMEs innovating in-house as % of SMEs 0.729 [0.779] 8 4 0.570 [0.513] Germany (0.933) Cyprus (0.833) Denmark (0.813) The Netherlands (0.797) Ireland (0.792) Germany (0.787) 2.3.3 Community trademarks per billion GDP (in PPP-€) 0.573 [0.661] 7 8 0.444 [0.580] Luxembourg (1.0) Cyprus (1.0) Malta (1.0) Cyprus (1.000) Luxembourg (1.000) Malta (1.000) 2.3.4 Community designs per billion GDP (in PPP-€) 0.574 [0.999] 8 3 0.566 [0.569] Luxembourg (1.0) Austria (1.0) Denmark (0.971) Denmark (1.000) Luxembourg (1.000) Sweden (0.999) 3.1.1 SMEs introducing product or process innovations as % of SMEs 0.781 [0.656] 4 6 0.577 [0.432] Germany (1.0) Belgium (0.848) Luxembourg (0.792) Luxembourg (0.732) Germany (0.717) Belgium (0.713) 3.1.2 SMEs introducing marketing or organizational innovations as % of SMEs 0.605 [0.540] 10 12 0.566 [0.495] Germany (1.0) Luxembourg (0.960) Greece (0.801) Luxembourg (0.851) Ireland (0.797) Germany (0.720) 3.2.2 Contribution of medium and high-tech product exports to trade balance 0.579 [0.648] 15 9 0.553 [0.658] Germany (0.930) Slovenia (0.802) Hungary (0.756) Hungary (0.899) Germany (0.892) Slovakia (0.850) 3.2.3 Knowledge-intensive services exports as % total service exports 0.510 [0.524] 10 11 0.606 [0.665] Ireland (1.0) Luxembourg (1.0) Denmark (0.959) Ireland (1.000) Luxembourg (1.000) Denmark (1.000) 3.2.4 Sales of new-to-market and new-to-firm innovations as % of turnover 0.248 [0.156] 21 24 0.664 [0.488] Greece (1.0) Slovakia (1.0) Spain (0.982) Denmark (1.000) Slovakia (0.869) Spain (0.590) Average output result12 0.575 [0.620] 10 4 0.568 [0.550] Germany (0.859) Luxembourg (0.754) Denmark (0.701) Luxembourg (0.772) Denmark (0.728) Germany (0.723) Indicator Score in 2014 [2015] Ranking (out of 28) in 2014 Ranking (out of 28) in 2015 EU 28 average in 2014 [2015] Leading countries (top 3) in 2014 Leading countries (top 3) in 2015 2.2.1 SMEs innovating in-house as % of SMEs 0.729 [0.779] 8 4 0.570 [0.513] Germany (0.933) Cyprus (0.833) Denmark (0.813) The Netherlands (0.797) Ireland (0.792) Germany (0.787) 2.3.3 Community trademarks per billion GDP (in PPP-€) 0.573 [0.661] 7 8 0.444 [0.580] Luxembourg (1.0) Cyprus (1.0) Malta (1.0) Cyprus (1.000) Luxembourg (1.000) Malta (1.000) 2.3.4 Community designs per billion GDP (in PPP-€) 0.574 [0.999] 8 3 0.566 [0.569] Luxembourg (1.0) Austria (1.0) Denmark (0.971) Denmark (1.000) Luxembourg (1.000) Sweden (0.999) 3.1.1 SMEs introducing product or process innovations as % of SMEs 0.781 [0.656] 4 6 0.577 [0.432] Germany (1.0) Belgium (0.848) Luxembourg (0.792) Luxembourg (0.732) Germany (0.717) Belgium (0.713) 3.1.2 SMEs introducing marketing or organizational innovations as % of SMEs 0.605 [0.540] 10 12 0.566 [0.495] Germany (1.0) Luxembourg (0.960) Greece (0.801) Luxembourg (0.851) Ireland (0.797) Germany (0.720) 3.2.2 Contribution of medium and high-tech product exports to trade balance 0.579 [0.648] 15 9 0.553 [0.658] Germany (0.930) Slovenia (0.802) Hungary (0.756) Hungary (0.899) Germany (0.892) Slovakia (0.850) 3.2.3 Knowledge-intensive services exports as % total service exports 0.510 [0.524] 10 11 0.606 [0.665] Ireland (1.0) Luxembourg (1.0) Denmark (0.959) Ireland (1.000) Luxembourg (1.000) Denmark (1.000) 3.2.4 Sales of new-to-market and new-to-firm innovations as % of turnover 0.248 [0.156] 21 24 0.664 [0.488] Greece (1.0) Slovakia (1.0) Spain (0.982) Denmark (1.000) Slovakia (0.869) Spain (0.590) Average output result12 0.575 [0.620] 10 4 0.568 [0.550] Germany (0.859) Luxembourg (0.754) Denmark (0.701) Luxembourg (0.772) Denmark (0.728) Germany (0.723) Source: Own elaboration based on European Union (2014, 2015). According to the IUS 2014 and 2015, the SII for Denmark, Finland, Germany, and Sweden are well above those of the EU average. These countries are labeled the ‘innovation leaders’, and Sweden is ranked number one. However, Table 1 gives a sharply different picture for Sweden. Taking into account the normalized values observed for the eight output indicators discussed above, and according to the IUS 2014 data, Sweden has an average normalized score of 0.575 for the innovation output indicators (0.620 for the IUS 2015), which is very close to the EU28 average of 0.568 (0.550 for the IUS 2015). Sweden thereby holds the 10th position among the EU28 (4th in the IUS 2015). Our results mean that nearly a third of all EU countries have higher innovation outputs than Sweden based on the IUS 2014 data. The highest ranked countries, based on the IUS 2014 data, and with regard to innovation output, are Germany (0.859), Luxembourg (0.754), and Denmark (0.701). In turn, the highest ranked countries, based on the IUS 2015, are Luxembourg (0.772), Denmark (0.728), and Germany (0.723). Figure 2 includes a sensitivity analysis of the results discussed above, showing the 2015 data with bars, and the 2014 data with markers. Countries have been sorted by the respective output scores for year 2015, according to the values achieved when 8 and 12 outputs are considered, respectively. Using this procedure with the IUS 2014 data, the ranking is led by Germany with a normalized score of 0.809, followed by Luxembourg (0.746) and Denmark (0.720). Sweden ranks 4th with a normalized value of 0.686. When comparing the average values with both approaches (M = 8 or M = 12) we get a correlation of R2 = 0.88. This implies that the average output results and the subsequent rankings with 8 and 12 outputs show very similar values for the EU28 countries.11 Figure 2. View largeDownload slide Output orientation of European innovation systems. Figure 2. View largeDownload slide Output orientation of European innovation systems. 4.2 Input orientation The four IUS indicators which we consider as being important for the input side of innovation processes are listed in Table 2, where we also summarize Sweden’s normalized scores and rankings for illustrative purposes. Table 2. The innovation input indicators of the Swedish national innovation system Indicator Score in 2014 [2015] Ranking (out of 28) in 2014 Ranking (out of 28) in 2015 EU28 average in 2014 [2015] Leading countries (top 3) in IUS 2014 Leading countries (top 3) in IUS 2015 1.3.1 Public R&D expenditures as % of GDP 0.979 [0.957] 2 3 0.639 [0.641] Finland (0.990) Sweden (0.979) Denmark (0.918) Denmark (0.989) Finland (0.957) Sweden (0.957) 1.3.2 Venture capital investments 0.503 [0.536] 8 7 0.478 [0.472] Luxembourg (1.0) UK (0.762) Finland (0.544) Luxembourg (0.858) UK (0.672) Denmark (0.604) 2.1.1 Business R&D expenditures as % of GDP 0.991 [0.956] 2 2 0.558 [0.559] Finland (1.0) Sweden (0.991) Slovenia (0.926) Finland (1.000) Sweden (0.956) Denmark (0.868) 2.1.2 Non-R&D innovation expenditures as % of turnover 0.319 [0.412] 10 10 0.275 [0.349] Cyprus (0.936) Lithuania (0.701) Estonia (0.557) Estonia (0.871) Latvia (0.764) Germany (0.746) Mean aggregate input result13 0.698 [0.715] 1 2 0.488 [0.505] Sweden (0.698) Finland (0.694) Germany (0.631) Germany (0.718) Sweden (0.715) Estonia (0.688) Indicator Score in 2014 [2015] Ranking (out of 28) in 2014 Ranking (out of 28) in 2015 EU28 average in 2014 [2015] Leading countries (top 3) in IUS 2014 Leading countries (top 3) in IUS 2015 1.3.1 Public R&D expenditures as % of GDP 0.979 [0.957] 2 3 0.639 [0.641] Finland (0.990) Sweden (0.979) Denmark (0.918) Denmark (0.989) Finland (0.957) Sweden (0.957) 1.3.2 Venture capital investments 0.503 [0.536] 8 7 0.478 [0.472] Luxembourg (1.0) UK (0.762) Finland (0.544) Luxembourg (0.858) UK (0.672) Denmark (0.604) 2.1.1 Business R&D expenditures as % of GDP 0.991 [0.956] 2 2 0.558 [0.559] Finland (1.0) Sweden (0.991) Slovenia (0.926) Finland (1.000) Sweden (0.956) Denmark (0.868) 2.1.2 Non-R&D innovation expenditures as % of turnover 0.319 [0.412] 10 10 0.275 [0.349] Cyprus (0.936) Lithuania (0.701) Estonia (0.557) Estonia (0.871) Latvia (0.764) Germany (0.746) Mean aggregate input result13 0.698 [0.715] 1 2 0.488 [0.505] Sweden (0.698) Finland (0.694) Germany (0.631) Germany (0.718) Sweden (0.715) Estonia (0.688) Source: Own elaboration based on European Union (2014, 2015). Table 2. The innovation input indicators of the Swedish national innovation system Indicator Score in 2014 [2015] Ranking (out of 28) in 2014 Ranking (out of 28) in 2015 EU28 average in 2014 [2015] Leading countries (top 3) in IUS 2014 Leading countries (top 3) in IUS 2015 1.3.1 Public R&D expenditures as % of GDP 0.979 [0.957] 2 3 0.639 [0.641] Finland (0.990) Sweden (0.979) Denmark (0.918) Denmark (0.989) Finland (0.957) Sweden (0.957) 1.3.2 Venture capital investments 0.503 [0.536] 8 7 0.478 [0.472] Luxembourg (1.0) UK (0.762) Finland (0.544) Luxembourg (0.858) UK (0.672) Denmark (0.604) 2.1.1 Business R&D expenditures as % of GDP 0.991 [0.956] 2 2 0.558 [0.559] Finland (1.0) Sweden (0.991) Slovenia (0.926) Finland (1.000) Sweden (0.956) Denmark (0.868) 2.1.2 Non-R&D innovation expenditures as % of turnover 0.319 [0.412] 10 10 0.275 [0.349] Cyprus (0.936) Lithuania (0.701) Estonia (0.557) Estonia (0.871) Latvia (0.764) Germany (0.746) Mean aggregate input result13 0.698 [0.715] 1 2 0.488 [0.505] Sweden (0.698) Finland (0.694) Germany (0.631) Germany (0.718) Sweden (0.715) Estonia (0.688) Indicator Score in 2014 [2015] Ranking (out of 28) in 2014 Ranking (out of 28) in 2015 EU28 average in 2014 [2015] Leading countries (top 3) in IUS 2014 Leading countries (top 3) in IUS 2015 1.3.1 Public R&D expenditures as % of GDP 0.979 [0.957] 2 3 0.639 [0.641] Finland (0.990) Sweden (0.979) Denmark (0.918) Denmark (0.989) Finland (0.957) Sweden (0.957) 1.3.2 Venture capital investments 0.503 [0.536] 8 7 0.478 [0.472] Luxembourg (1.0) UK (0.762) Finland (0.544) Luxembourg (0.858) UK (0.672) Denmark (0.604) 2.1.1 Business R&D expenditures as % of GDP 0.991 [0.956] 2 2 0.558 [0.559] Finland (1.0) Sweden (0.991) Slovenia (0.926) Finland (1.000) Sweden (0.956) Denmark (0.868) 2.1.2 Non-R&D innovation expenditures as % of turnover 0.319 [0.412] 10 10 0.275 [0.349] Cyprus (0.936) Lithuania (0.701) Estonia (0.557) Estonia (0.871) Latvia (0.764) Germany (0.746) Mean aggregate input result13 0.698 [0.715] 1 2 0.488 [0.505] Sweden (0.698) Finland (0.694) Germany (0.631) Germany (0.718) Sweden (0.715) Estonia (0.688) Source: Own elaboration based on European Union (2014, 2015). According to our results, Sweden is at the very top with regard to its average input ranking (ranking number one in 2014 and second in 2015, with 0.698 and 0.715, respectively) among the other EU28 Member States. According to the IUS 2014 data, Finland has ranking number 2 (0.694) and Germany has ranking number 3 (0.631). Using the IUS 2015 data, Germany has ranking number 1 (0.718), and Estonia has ranking number 3 (0.688). Here too we have conducted a sensitivity analysis to check to what extent the average input result for Sweden changes when additional input indicators are considered (N = 4 or N = 7). The results of this sensitivity analysis are plotted in Fig. 3. The ranking is still led by Sweden with a score of 0.771 in 2014 and 2015. When comparing the average values and rankings with both approaches (N= 4 inputs or N= 7), we get a correlation of R2 = 0.88 in 2014 and R2 = 0.83 in 2015. Figure 3. View largeDownload slide Input orientation of European innovation systems. Figure 3. View largeDownload slide Input orientation of European innovation systems. 4.3 Innovation performance In this subsection we focus on the relation between the input and the output sides, i.e. on a measure of the innovation performance of national innovation systems (see Table 3). We define the innovation performance of a system as the ratio between innovation outputs and innovation inputs, using both the index number ( PIÎit) and the standard ( EIÎit) and bootstrapped ( bEIÎit) DEA approaches. Such ratios, show, in alternative ways to SIÎit, how efficiently the countries or systems use their inputs, as SIÎit simply calculates the mean of all the 25 indicators, and does not relate them to each other in any other way. Table 3. The innovation performance of the Swedish national innovation system14 Performance index Year Scores Ranking (out of 28) Leading countries in IUS (top 3) Summary innovation index (IUS) (2): SIÎit 2014 1.000 (=0.750/0.750) 1 Sweden (1.000), Denmark (0.971), Germany (0.945) 2015 1.000 (=0.740/0.740) 1 Sweden (1.000), Denmark (0.995), Finland (0.914) Productivity Innovation Index (4): PIÎit 2014 0.327 = (0.824 = 0.575/0.698)/2.540 24 Greece (1.000), Bulgaria (0.869), Italy (0.786) 2015 0.285 = (0.870 = 0.620/0.715)/3.040 22 Cyprus (1.000), Luxembourg (0.662), Romania (0.650) Efficiency Innovation Index (5): EIÎit 2014 0.604 27 17 countries are DEA efficient: EIÎit= 1.000a 2015 0.706 23 15 countries are DEA efficient: EIÎit= 1.000b Bootstrapped Efficiency Innovation Index, bEIÎit 2014 0.562* 27 Slovakia (0.917*), Croatia (0.913*), Portugal (0.904*) 2015 0.647* 24 Slovenia (0.902*), Poland (0.897*), United Kingdom (0.881*) Performance index Year Scores Ranking (out of 28) Leading countries in IUS (top 3) Summary innovation index (IUS) (2): SIÎit 2014 1.000 (=0.750/0.750) 1 Sweden (1.000), Denmark (0.971), Germany (0.945) 2015 1.000 (=0.740/0.740) 1 Sweden (1.000), Denmark (0.995), Finland (0.914) Productivity Innovation Index (4): PIÎit 2014 0.327 = (0.824 = 0.575/0.698)/2.540 24 Greece (1.000), Bulgaria (0.869), Italy (0.786) 2015 0.285 = (0.870 = 0.620/0.715)/3.040 22 Cyprus (1.000), Luxembourg (0.662), Romania (0.650) Efficiency Innovation Index (5): EIÎit 2014 0.604 27 17 countries are DEA efficient: EIÎit= 1.000a 2015 0.706 23 15 countries are DEA efficient: EIÎit= 1.000b Bootstrapped Efficiency Innovation Index, bEIÎit 2014 0.562* 27 Slovakia (0.917*), Croatia (0.913*), Portugal (0.904*) 2015 0.647* 24 Slovenia (0.902*), Poland (0.897*), United Kingdom (0.881*) a Austria, Bulgaria, Croatia, Cyprus, Czech Republic, France, Greece, Hungary, Ireland, Italy, Latvia, Luxembourg, Malta, Portugal, Romania, Slovakia, Spain. b Austria, Bulgaria, Cyprus, Czech Republic, Denmark, Greece, Ireland, Italy, Luxembourg, Malta, Romania, The Netherlands, Slovakia, Spain, United Kingdom. * denotes that the bootstrapped bias-corrected indices are statistically significant at the 5% significance level. Source: Own elaboration based on data from the European Union (2014, 2015). Table 3. The innovation performance of the Swedish national innovation system14 Performance index Year Scores Ranking (out of 28) Leading countries in IUS (top 3) Summary innovation index (IUS) (2): SIÎit 2014 1.000 (=0.750/0.750) 1 Sweden (1.000), Denmark (0.971), Germany (0.945) 2015 1.000 (=0.740/0.740) 1 Sweden (1.000), Denmark (0.995), Finland (0.914) Productivity Innovation Index (4): PIÎit 2014 0.327 = (0.824 = 0.575/0.698)/2.540 24 Greece (1.000), Bulgaria (0.869), Italy (0.786) 2015 0.285 = (0.870 = 0.620/0.715)/3.040 22 Cyprus (1.000), Luxembourg (0.662), Romania (0.650) Efficiency Innovation Index (5): EIÎit 2014 0.604 27 17 countries are DEA efficient: EIÎit= 1.000a 2015 0.706 23 15 countries are DEA efficient: EIÎit= 1.000b Bootstrapped Efficiency Innovation Index, bEIÎit 2014 0.562* 27 Slovakia (0.917*), Croatia (0.913*), Portugal (0.904*) 2015 0.647* 24 Slovenia (0.902*), Poland (0.897*), United Kingdom (0.881*) Performance index Year Scores Ranking (out of 28) Leading countries in IUS (top 3) Summary innovation index (IUS) (2): SIÎit 2014 1.000 (=0.750/0.750) 1 Sweden (1.000), Denmark (0.971), Germany (0.945) 2015 1.000 (=0.740/0.740) 1 Sweden (1.000), Denmark (0.995), Finland (0.914) Productivity Innovation Index (4): PIÎit 2014 0.327 = (0.824 = 0.575/0.698)/2.540 24 Greece (1.000), Bulgaria (0.869), Italy (0.786) 2015 0.285 = (0.870 = 0.620/0.715)/3.040 22 Cyprus (1.000), Luxembourg (0.662), Romania (0.650) Efficiency Innovation Index (5): EIÎit 2014 0.604 27 17 countries are DEA efficient: EIÎit= 1.000a 2015 0.706 23 15 countries are DEA efficient: EIÎit= 1.000b Bootstrapped Efficiency Innovation Index, bEIÎit 2014 0.562* 27 Slovakia (0.917*), Croatia (0.913*), Portugal (0.904*) 2015 0.647* 24 Slovenia (0.902*), Poland (0.897*), United Kingdom (0.881*) a Austria, Bulgaria, Croatia, Cyprus, Czech Republic, France, Greece, Hungary, Ireland, Italy, Latvia, Luxembourg, Malta, Portugal, Romania, Slovakia, Spain. b Austria, Bulgaria, Cyprus, Czech Republic, Denmark, Greece, Ireland, Italy, Luxembourg, Malta, Romania, The Netherlands, Slovakia, Spain, United Kingdom. * denotes that the bootstrapped bias-corrected indices are statistically significant at the 5% significance level. Source: Own elaboration based on data from the European Union (2014, 2015). Based on the data provided by the IUS 2014, Sweden is ranked extremely high with regard to the mean input indicator (0.698, 1st place in Table 2) and significantly lower with regard to output (0.575, 10th, Table 1). However, the IUS 2015 data show some improvement, particularly on the output side, as its rank improves to the 4th place―0.620. As shown in Table 3, this brings Sweden to the 1st place according to the overall SII ranking in both years. Indeed, Sweden presents the highest absolute and normalized values, e.g. in 2015, SIISE2015 = 0.750 and SIÎSE2015 = 1.000―as 0.750 is the maximum SIIj2015 score. However, when assessing the performance of the Swedish innovation system with regard to its productivity, this leads to very low rankings regardless the definition employed, either the index number, the standard, or the robust DEA estimate. Considering the index number definition, the absolute value corresponding to the ratio of the aggregate output to the aggregate input is PIISE2015 = 0.824 = 0.575/0.698. Once this value is normalized by the maximum productivity observed across all countries, corresponding to Cyprus: max j(PIIj2015)= 3.040, Sweden ranks in the 22nd place with a normalized score of PIÎSE2015= 0.327 = 0.824/3.040. Its 2015 rank is two positions better than in 2014 (24th place), as a result of the improvements in the aggregate output, but nevertheless places Sweden consistently at the lower tail of the productivity distribution. Obviously, the national innovation system in Sweden cannot be said to perform well at all from this efficiency point of view, and Sweden can certainly not be seen as an innovation leader in the EU, judging from the output and performance data. This is further confirmed by the standard and bootstrapped DEA results. Even if DEA is benevolent when assessing the relative productivity by searching for the most advantageous input and output weights:  νnt* and μmt*, it is to no avail. The standard and bootstrapped DEA scores: EIÎSE2014 = 0.604 and bEIÎSE2014 = 0.562 place Sweden in the second to last position in the rankings. This relative position marginally improves for 2015. It is worth highlighting that these results are statistically significant from the perspective of the bootstrapping methods, as all efficiency scores bEIÎit fall within the bootstrap confidence intervals—as presented in the last columns of the Appendix. We note in passing that the bootstrapped efficiency scores represent a valid solution to the low discriminatory power of standard DEA techniques. In both 2014 and 2015 a significant number of countries (16 and 15) are efficient with a unitary score (see Table 3). This precludes an individualized ranking, as these countries are tied in the first place. Therefore, bootstrapping techniques break the tie while informing about the statistical significance of their individual values, thereby providing comprehensive rankings. Regarding our case study on Sweden, we conclude that whether one uses the fixed input and outputs weights  ν¯nt and μ¯mt, or the optimal—most favorable—weights  νnt* and μmt*, the innovation performance of Sweden can be consistently categorized as modest using the IUS denomination for countries below 50% of the EU average. Therefore, and quite unexpectedly, the fact that the DEA methods represent an optimistic approach that give observations the opportunity to perform the best possible way with respect to its peers does not result in significant improvements. This really tells us about how strongly dominated is Sweden by a large set of reference peers. On the contrary, compared to PIÎSEt in 2014 and 2015, both the standard and booststrapped DEA results rank Sweden in lowest positions. For the same reason, as shown in Table 3 and the Appendix, unsuspected countries present innovation systems that, from a productivity perspective, perform fairly well, attaining high relative levels of outputs given the availability of innovation inputs. This is the case of Slovakia and Slovenia, leading the bootstrapped efficiency rankings in 2014 and 2015, respectively. We now proceed to compare the three alternative performance measures ( PIÎit, EIÎit, bEIÎit). Figure 4 shows the distribution of the scores obtained using SIÎit, and our simplest productivity index number— PIÎit—(using four input and eight output indicators) for both the IUS 2014 and 2015. If the two rankings coincided, one would expect the majority of countries to be distributed along a 45° line; i.e. laying on the southwest and northeast quadrants. However, this is certainly not the case. As can be observed, Sweden is not the only country where the two rankings are radically different, laying in the ‘wrong quadrants’. In fact, this is the case for most countries, including innovation leaders, innovation followers, moderate innovators, and modest innovators. Figure 4. View largeDownload slide Comparison of the normalized IUS SIÎit and the normalized PIÎit. (a) IUS 2014. (b) IUS 2015. Source: Own elaboration. Figure 4. View largeDownload slide Comparison of the normalized IUS SIÎit and the normalized PIÎit. (a) IUS 2014. (b) IUS 2015. Source: Own elaboration. As discussed by Zabala-Iturriagagoitia et al. (2007a), it is not evident that the innovation systems that invest larger amount of resources (i.e. inputs) are also the most efficient ones. This result is also confirmed by Chen and Guan (2012) and by Mahroum and Al-Saleh (2013) among others, pointing out that territories making modest investments in innovation still can enjoy significant innovation outputs, and therefore high performance in terms of efficiency. Similarly, Fig. 5 provides a comparison of the EIÎit and the bEIÎit for years 2014 and 2015. Countries have been sorted according to the bEIÎit, which is represented together with the upper and lower bounds of the confidence intervals. This figure complements the previous Fig. 4, offering a concise but complete visual overview of the results. Figure 5. View largeDownload slide Comparison of the EIÎit and the bootstrapped efficiency innovation index ( bEIÎit). (a) IUS 2014. (b)- IUS 2015. Figure 5. View largeDownload slide Comparison of the EIÎit and the bootstrapped efficiency innovation index ( bEIÎit). (a) IUS 2014. (b)- IUS 2015. We also confirm the compatibility of the fixed weight PIÎit results with those obtained for its deterministic and bootstrapped DEA counterparts allowing for flexible weights. To explore the similarity between the productivity distributions, Table 4 shows the results of a series of tests checking alternative hypotheses. The null hypotheses of the Wilcoxon rank sum test and the two-sample t-test are that both samples come from distributions with equal medians and equal means, respectively, against the alternative that they are not. For both years the null hypothesis is rejected at the 99% confidence level, implying that the median and the mean between the two bilateral pairs of efficiency measures are different ( PIÎit vs. EIÎit, and PIÎit vs. bEIÎit). However from the perspective of the rankings that these distributions yield, the Kendall and Spearman rank correlations show relatively high and significant values, particularly between the deterministic indices PIÎit and EIÎit. Unsurprisingly, as reported in the last two columns, the bootstrapped results bEIÎit show that their 2014 and 2015 distributions are also different from their deterministic counterparts EIÎit in both years. Nevertheless, once again, the rankings correlate to a large extent. Table 4. Tests of hypotheses between alternative productivity measures Test PIÎit vs. EIÎit PIÎit vs. bEIÎit EIÎit vs. bEIÎit 2014 2015 2014 2015 2014 2015 Wilcoxon rank sum testa −5.649 −5.803 −5.170 −5.416 2.504 2.474 P-value 0.000 0.000 0.000 0.000 0.015 0.017 Two-sample t-testb −8.855 −10.214 −7.361 −8.685 0.992 0.990 P-value 0.000 0.000 0.000 0.000 0.000 0.000 Kendall’s τ rank correlationc 0.549 0.694 0.365 0.265 0.800 0.520 P-value 0.000 0.000 0.006 0.050 0.000 0.000 Spearman’s ρ correlationc 0.706 0.843 0.531 0.402 0.881 0.630 P-value 0.000 0.000 0.004 0.035 0.000 0.000 Test PIÎit vs. EIÎit PIÎit vs. bEIÎit EIÎit vs. bEIÎit 2014 2015 2014 2015 2014 2015 Wilcoxon rank sum testa −5.649 −5.803 −5.170 −5.416 2.504 2.474 P-value 0.000 0.000 0.000 0.000 0.015 0.017 Two-sample t-testb −8.855 −10.214 −7.361 −8.685 0.992 0.990 P-value 0.000 0.000 0.000 0.000 0.000 0.000 Kendall’s τ rank correlationc 0.549 0.694 0.365 0.265 0.800 0.520 P-value 0.000 0.000 0.006 0.050 0.000 0.000 Spearman’s ρ correlationc 0.706 0.843 0.531 0.402 0.881 0.630 P-value 0.000 0.000 0.004 0.035 0.000 0.000 a H0 is that both distributions are the same. b H0 is the equality of means. c H0 is that both variables are independent. Table 4. Tests of hypotheses between alternative productivity measures Test PIÎit vs. EIÎit PIÎit vs. bEIÎit EIÎit vs. bEIÎit 2014 2015 2014 2015 2014 2015 Wilcoxon rank sum testa −5.649 −5.803 −5.170 −5.416 2.504 2.474 P-value 0.000 0.000 0.000 0.000 0.015 0.017 Two-sample t-testb −8.855 −10.214 −7.361 −8.685 0.992 0.990 P-value 0.000 0.000 0.000 0.000 0.000 0.000 Kendall’s τ rank correlationc 0.549 0.694 0.365 0.265 0.800 0.520 P-value 0.000 0.000 0.006 0.050 0.000 0.000 Spearman’s ρ correlationc 0.706 0.843 0.531 0.402 0.881 0.630 P-value 0.000 0.000 0.004 0.035 0.000 0.000 Test PIÎit vs. EIÎit PIÎit vs. bEIÎit EIÎit vs. bEIÎit 2014 2015 2014 2015 2014 2015 Wilcoxon rank sum testa −5.649 −5.803 −5.170 −5.416 2.504 2.474 P-value 0.000 0.000 0.000 0.000 0.015 0.017 Two-sample t-testb −8.855 −10.214 −7.361 −8.685 0.992 0.990 P-value 0.000 0.000 0.000 0.000 0.000 0.000 Kendall’s τ rank correlationc 0.549 0.694 0.365 0.265 0.800 0.520 P-value 0.000 0.000 0.006 0.050 0.000 0.000 Spearman’s ρ correlationc 0.706 0.843 0.531 0.402 0.881 0.630 P-value 0.000 0.000 0.004 0.035 0.000 0.000 a H0 is that both distributions are the same. b H0 is the equality of means. c H0 is that both variables are independent. 5. Discussion, conclusions, and research agenda According to the IUS, innovation performance implicitly means an average of 25 indicators measuring the determinants of innovations, inputs, innovations, as well as their consequences. In this article, we have discussed why innovation performance should be understood as the relationships between innovation inputs and outputs, as done in productivity and efficiency research. The first purpose of this article was to assess whether the SII of the European Innovation Scoreboard (IUS) is a meaningful measure of innovation performance. Our conclusion is that it is not. The main reason is that it mixes input and output innovation indicators and calculates an average of them. In addition, it lacks an explicit definition of what is meant by innovation performance, beyond the indiscriminate arithmetic mean of all 25 IUS indicators. The second purpose was to develop an alternative, productivity, or efficiency-based, measure of innovation system performance. By doing so, the article offers a critical review of the SII, and proposes to place more emphasis on the identification of and relation between input and output indicators. For this purpose we rely on the simplest productivity index number formulation, but we also supplement it with state-of-the-art productivity measurement methods in the form of robust (bootstrapped) DEA techniques that address the weaknesses of deterministic indices. Our article can be understood as a contribution to the need for a ‘substantial research effort on conceptual foundations as well as empirics of innovation measurement’ (Janger et al. 2017: 39). We use the existing data provided by the IUS but seen through a different (productivity) lens, with the purpose of increasing the breadth of sight of innovation policy makers and politicians. By using both fixed and flexible weights we have assessed the performance of EU28 Member States by comparing alternative multi-input/multi-output relationships in a benchmark model, including four inputs and eight outputs. We justify this choice on theoretical grounds but encourage other researchers to improve on this ‘standard model’. Our intention is to mimic the simplicity of the methods followed by the SII, though from a different (productivity) point of view. The simple productivity index and its sophisticated DEA counterparts result in very similar rankings. The more sophisticated method is actually not needed to reach the basic result. If one is interested in this result, rather than in specific and individual numerical values (because distributions have been shown to be different), the simplest productivity definition performs well. However, the relevance of statistically robust methods such as bootstrapped DEA is clear, as they allow addressing some of the weaknesses of the deterministic approached; most particularity, the presence of innovation performance outliers emerging from the (IUS) data. As can be observed in the Appendix, the top 10 EU national innovations systems ranked in terms of the bootstrapped efficiency score for 2015 are Slovenia, Poland, the UK, Malta, Austria, France, Denmark, Italy, Portugal, and Spain. Sweden is ranked number 23. No matter how unbelievable it may appear, this efficiency ranking is appropriate to the extent that the IUS data are correct and our approach to dealing with them is sensible and robust. The ranking goes against stylized facts and seems counterintuitive, since many of the top-ranked countries are less developed economies with less comprehensive innovation systems. In turn, the countries that traditionally lead innovation rankings (such as the SII) get lower standings with regard to innovation performance. To fully explain the reasons for these results in a detailed manner, we would need sophisticated methods that allow controlling for key aspects such as sectoral structure, diversification, degree of internationalization, firm size, and institutional conditions, which remains for further research. One partial explanation to these seemingly contradictory results is that many of these small EU countries devote very limited resources to inputs. According to the approach followed here, the lower the inputs, the higher the efficiency. Besides, they manage to obtain reasonable outputs in relation to the invested inputs. For example, Sweden invested 7.35 times more than Bulgaria in innovation inputs, according to IUS 2014 (see Appendix). At the same time Sweden achieved an output figure that is only 2.77 times higher than Bulgaria’s. The figures are similar for the other EU countries that rank high on traditional innovation performance calculations (such as the SII). The above figures imply that some less developed countries manage to use their (more limited) resources in a more productive/efficient way. It also indicates that there might be Decreasing Returns to Scale (DRS) obtained by investments in innovation input activities, so countries investing a large amount of innovations resources cannot achieve large returns. The DEA methodology provides ancillary results that shed light on the role that returns to scale (i.e. input size) play for inefficient performance. The last column in the Appendix shows whether IRS, CRS, or DRS prevail locally for each country—see the footnote to program (5) on how to determine the nature of returns to scale. The suspected widespread presence of DRS is confirmed, with as many as 10 countries exhibiting a suboptimal scale size as a result of an excessive relative usage of inputs. Among this, we find several SII leading countries, including Sweden, Germany, Finland, etc. On the contrary, incurring in innovation inefficiency due to IRS (i.e. relatively low use of inputs) does not seem to be an issue, as only three countries, Croatia, Latvia, and Lithuania, exhibit this type of returns. As Zabala-Iturriagagoitia et al. discuss (2007a: 667), another partial explanation may be that countries with comprehensive innovation systems are usually, oriented toward radical innovation and the development of new industries, often in knowledge-intensive sectors and high-tech industries. Such innovation efforts are characterized by uncertainty, high-risk, failures, and large time lags. In contrast, countries with smaller innovation input resources tend to absorb and adopt embodied knowledge and innovations from other countries. Such absorption involves lower innovation input costs, but may, at the same time, be more efficient, as they may avoid the inherent risk involved in the development of these innovations, so that the ‘new’ knowledge is more rapidly and cheaply adapted and adopted than in the country where it was developed. According to Freeman and Soete (1988), the efficient diffusion of innovations is often much more important for development and growth than being the lead innovator. Countries regarded as innovation leaders by the IUS may thus be more prone to the creation of new-to-the-world innovations, while follower countries are more prone to the absorption and later diffusion of these innovations, as long as the required levels of absorptive capacity and learning are in place. This is highly relevant from an innovation policy point of view, a partial objective of which may be to catch up with the leading countries by absorbing innovations from abroad. We want to stress the irrelevance of the notion of optimality in an innovation context, given the impossibility to specify an ideal or optimal innovation system. The only way to identify problems that should be subject to innovation policy is therefore to compare existing innovation systems with each other. Unfortunately, the ranking of the efficiency of the EU28 innovation systems in the Appendix provides limited guidelines from the point of view of innovation policy development. There are no reasons whatsoever to benchmark Sweden’s national innovation system with those of Slovenia, Poland, Greece, Cyprus, etc., in attempting to develop an analysis to form a basis for innovation policy changes. Such comparisons should, instead, be made between innovation systems that are more similar in a structural sense and at a similar level of development, that have the same size, that score similarly on innovation output or innovation input, etc. A restricted DEA analysis for countries with similar characteristics could be helpful here. One implication of this analysis is that considerable efforts should be made to identify the sources of the inefficiencies and problems in the national systems of innovation. Existing innovation intensities are influenced by a number of forces that affect innovation processes, forces which we call determinants of innovations. Many of these determinants can be influenced by the state. When the state (through its public agencies) influences these determinants to increase innovation intensities of certain kinds or change the direction of innovation processes, it is actually pursuing innovation policy. Public bodies should be able to know and monitor the evolution of these determinants and their impact on innovation. The SII does however not help to do so. Since it includes inputs, determinants, innovations, and consequences, this synthetic indicator is not useful from the point of view of innovation policy design. The SII score will increase if a country puts more (input) resources into its innovation system, regardless of how the resources are used or what the (innovation) output might be. A worrisome property of the SII index is that its value increases even if the innovation output resulting from more inputs is 0 (i.e. the marginal productivity of inputs is 0). These results should call for a serious reconsideration of who the real European ‘innovation leaders’ may be, and in what sense they are leaders. These findings also call into question the way that the European Commission analyzes the innovation data presented in the IUS. The IUS is intended to have a real impact on the design of innovation policies of the EU Member States. According to our results, the SII is flawed with regard to measuring innovation performance and may therefore mislead researchers, policy makers, politicians, as well as the general public. Policy makers in the field of innovation should be able to identify the policy problems in their innovation systems and relate them to their causes to be able to select policy instruments to mitigate the problems. Our approach can be useful for the design of policy. For example, if we know that Sweden is much weaker on the output side than on the input side, our approach can tell policy makers and politicians to concentrate more on making a more efficient use of existing inputs than on increasing the volume of the inputs, for example through the articulation of an instrument-mix that considers demand-side innovation policy instruments. To further develop such a kind of disaggregated analysis to identify policy problems is much more useful as a basis for innovation policy design, than to aggregate data into single ranking numbers of countries. In this way, the individual indicators that constitute composite indicators should be analyzed in much more depth, and this should be done in a comparative manner. We believe that the simplicity of our definition of productivity as a ratio of (aggregate) outputs to (aggregate) inputs, along with the proved confirmation by using DEA methods, reinforces the credibility of our results. Although the analyses can be further refined, we do claim that this trajectory is also a fruitful one for future research as remarked in the different extensions that we propose: use of bootstrapped Malmquist productivity indices where the relative sources of productivity change, such as technological progress vs. catching-up processes, could be measured, restricted frontiers by country categories, etc. It can be argued that the rationale used in this article follows a linear logic, as opposed to a systemic and holistic one (e.g. Edquist 2014a, 2014b, 2014c, 2018). As mentioned at the end of Section 3.1, an ideal basis for the design of innovation policy should aim to include a fully articulated holistic innovation systems approach. Such an approach would include knowledge about all determinants of the development and diffusion of innovations—knowledge which we do not possess. This would mean overcoming the linear model of innovation that still dominates innovation policy in all countries, despite the fact that it has been completely rejected in innovation research (Edquist 2014a, 2018). Developing such a holistic view would make it possible to account for all determinants of innovations as inputs when measuring innovation performance and efficiency of innovation systems. This would mean that the linear approach (a focus on input–output relationships) used out of necessity in this article, could be overcome and transcended. The linear and holistic approaches would become compatible and reconciled. However, we are not yet there. Footnotes 1 For illustrative purposes, we use Sweden as an example when we discuss the data, methodology, and results. However, in the Appendix, we present the data for all 28 EU Member States. This means that the reader can substitute Sweden by any other of the 27 EU countries. 2 These steps include development of a theoretical framework, data selection, development of an exploratory multivariate analysis, imputation of missing data, normalization, weighting and aggregation, and robustness and sensibility checks among others (see Nardo et al. 2005). 3 The IUS performs a max–min normalization by subtracting the minimum value and diving by the observed range: ι^lit=(ιlit−min (ιljt))/(max (ιljt)−min (ιljt)), j = 1,…, 34 countries. Therefore, ‘The maximum re-scaled score is thus equal to 1 and the minimum re-scaled score is equal to 0’ (European Union 2015: 79). 4 For the detailed interpretation of each of the output and input indicators considered, please see European Union (2014: 86–90). All 25 indicators are also listed in Fig. 1. 5 Following Hagedoorn and Cloodt (2003: 1367), for whom the concept of innovation performance excludes ‘the possible economic success of innovations as such’, we do neither consider the consequences (i.e. outcomes or impacts) of innovations, such as economic growth or employment, to be innovation output indicators. 6 The exclusion of indicator 2.3.1 (Patent Cooperation Treaty (PCT) patent applications) is made for a different reason. As long as a product or a process has not been commercialized and connected to the market, it cannot be considered to be an innovation. An application for a patent is far from being a product introduced to the market. Not even granted patents are product innovations, although they may be a basis for future innovations. Patents are rather an indicator of research results or inventions, as they reflect that something is technologically new. This implies that patents are a throughput, rather than an output of innovation (see also Lee 2015; Janger et al. 2017). 7 The additional indicators considered were PCT patent applications (2.3.1), Employment in fast-growing enterprises in innovative sectors (3.1.3), Employment in knowledge-intensive activities (3.2.1), and License and patent revenues from abroad (3.2.5). 8 Of these two indicators, R&D expenditures in the business sector are certainly to a very large extent directly undertaken to enhance innovation (see Lee 2015). R&D expenditures in the public sector are to a lesser extent undertaken directly for this purpose, since a substantial proportion is pursued to develop new scientific knowledge, part of which may result (or not) in innovations. In spite of this, we include both of these indicators in the category of input indicators, although a part of the public sector R&D expenditures may result in innovations only after a substantial time lag, or not at all for some kinds of public R&D financing. 9 The additional indicators considered were New doctorate graduates (1.1.1), Percentage of population aged 30–34 years having completed tertiary education (1.1.2), and Percentage youth aged 20–24 years having attained at least upper secondary education (1.1.3). 10 Program (5) corresponds to a fractional (ratio form) formulation, which after some transformations can be expressed as a linear program, and whose resulting objective function is linearly homogeneous of degree one, thereby defining a productivity frontier characterized by constant returns to scale (CRS), Cooper, Seiford, and Tone (2007: chapter 2). Its dual−primal−formulation corresponds to the input-oriented envelopment form: EII^it=min θ,λ{θ:∑j=1Jλjtx^jnt≤θx^nt, n=1,…,N; ∑j=1Jλjty^jmt≥y^mt, m=1,…,M;  ∑j=1Jλjt=1, λj≥0, j=1,…,J}, where λj are intensity variable identifying the benchmark countries conforming the reference facet. Resorting to the envelopment form, it is possible to determine the nature of returns to scale at this benchmark frontier: increasing, constant, and decreasing, by examining the value of the sum of the optimal lambdas—Cooper, Seiford, and Tone (2007: chapter 5): Decreasing Returns to Scale (DRS) prevails for (xit,yit)⇔∑jλj*>1 for all optimal solutions. Increasing Returns to Scale (IRS) prevails for (xit,yit)⇔∑jλj*<1 for all optimal solutions. CRS prevails for (xit,yit)⇔∑jλj*=1 for some optimal solutions. Therefore, by solving the dual to program (5) we can determine what type of returns to scale locally predominate for (xit,yit)—which, in turn, implies that a suboptimal scale is at play if the innovation system turns out to be inefficient. In the discussion section we recall this result to substantiate one of the possible reasons why innovation performance is deficient depending on the innovation size of the countries. 11 When using the IUS 2015 data, the ranking is led by Luxembourg (0.765), followed by Germany (0.725) and Denmark (0.724). Sweden ranks 5th with a normalized value of 0.690. When comparing the average values with both approaches (M = 8 or M = 12) a correlation of R2 = 0.93 in 2015 is obtained. 12 Calculation is based on the arithmetic mean of the normalized scores of each output indicator. 13 Calculation is based on the arithmetic mean of the normalized scores for each input indicator. 14 The data and rankings for all EU28 Member Countries are presented in the Appendix. Acknowledgements A preliminary version of this article (Edquist and Zabala-Iturriagagoitia, 2015) received the 2015 VII UAM-Accenture Chair Scientific Award in Economics and Innovation Management in March 2016. The authors are indebted to the editor and two anonymous referees for their comments on an earlier draft. The authors appreciate Hannes Toivanen, Jakob Edler, Nick Vonortas, Olof Ejermo, Martin Andersson, Sylvia Schwaag Serger, Espen Solberg, Lennart Stenberg, Torben Schubert, Lorenzo Isella, Athina Karvounaraki, and Daniel Vertesy for their comments. The authors are also grateful to Michael Nauruschat for his assistance with the data collection and to Margaret Newman-Nowicka for the language editing of the text. J.M.Z-I. wishes to acknowledge the financial support from the Basque Government Department of Education, Language Policy and Culture (IT 885-16), and C.E. would like to acknowledge support from the Rausing Foundation. The views expressed are purely those of the authors and may not in any circumstances be regarded as stating an official position of the European Commission. References Archibugi D. , Denni M. , Filippetti A. ( 2009 ) ‘The Technological Capabilities of Nations: The State of the Art of Synthetic Indicators’ , Technological Forecasting and Social Change , 76 : 917 – 31 . Google Scholar CrossRef Search ADS Balk B. M. ( 2008 ) Price and Quantity Index Numbers: Models for Measuring Aggregate Change and Difference . New York, NY : Cambridge University Press . Google Scholar CrossRef Search ADS Bogetoft P. , Otto L. ( 2011 ). Benchmarking with DEA, SFA, and R . New York, NY : Springer . Google Scholar CrossRef Search ADS Carayannis E. , Grigoroudis E. ( 2014 ) ‘ Linking Innovation, Productivity, and Competitiveness: Implications for Policy and Practice ’, Journal of Technology Transfer , 39 : 199 – 218 . Google Scholar CrossRef Search ADS Charnes A. , Cooper W. W. , Rhodes E. ( 1978 ) ‘ Measuring the Efficiency of Decision Making Units ’, European Journal of Operational Research , 2 / 6 : 429 – 44 . Google Scholar CrossRef Search ADS Chen K. , Guan J. ( 2012 ) ‘ Measuring the Efficiency of China’s Regional Innovation Systens: Application of Network Data Envelopment Analysis (DEA) ’, Regional Studies , 46 / 3 : 355 – 77 . Google Scholar CrossRef Search ADS Coelli T. J. et al. ( 2005 ) An Introduction to Efficiency and Productivity Analysis . New York, NY : Springer . Cooper W. W. , Seiford L. M. , Tone K. ( 2007 ) Data Envelopment Analysis: A Comprehensive Text with Models, Applications, References and DEA-Solver Software . New York, NY : Springer . Daraio C. , Simar L. ( 2007 ) Advanced Robust and Nonparametric Methods in Efficiency Analysis , New York, NY : Springer Science + Business Media, LLC . Dutta S. , Lanvin B. , Wunsch-Vincent S. ( 2017 ) ‘The Global Innovation Index 2017, Innovation Feeding the World’ . Cornell University, INSEAD, WIPO. Ithaca, Fontainebleau, and Geneva. Edquist C. ( 2005 ) ‘Systems of Innovation: Perspectives and Challenges’, in Fagerberg J. , Mowery D. , Nelson R. (eds) Oxford Handbook of Innovation , pp. 181 – 208 . Oxford : Oxford University Press . Edquist C. ( 2011 ) ‘ Design of Innovation Policy through Diagnostic Analysis: Identification of Systemic Problems (or Failures) ’, Industrial and Corporate Change , 20 / 6 : 1725 – 53 . Google Scholar CrossRef Search ADS Edquist C. ( 2014a ) ‘ Striving Towards a Holistic Innovation Policy in European Countries—But Linearity Still Prevails! ’, STI Policy Review , 5 / 2 : 1 – 19 . Edquist C. ( 2014b ) ‘En Helhetlig Innovationspolitik—Varför, Vad Och Hur? (a Holistic Innovation Policy—Why, What and How?)’, in Wanger M. Ö. (ed.) Position Sverige—Om Innovation, Hållbarhet Och Arbetsmarknad—En Debattantologi (Position Sweden—On Innovation, Sustainability and Labour Market—A Debate Anthology—In Swedish) , pp. 59 – 80 . Stockholm : Ekerlids Förlag . Edquist C. ( 2014c ) ‘Efficiency of Research and Innovation Systems for Economic Growth and Employment’, CIRCLE Working Paper 2014/08. Sweden: Lund University. Edquist C. ( 2018 ) ‘Towards a Holistic Innovation Policy: Can the Swedish National Innovation Council Serve as a Role Model?’, Papers in Innovation Studies, Paper no. 2018/02. Sweden: CIRCLE, Lund University. Edquist C. et al. , eds ( 2015 ) Public Procurement for Innovation . Cheltenham : Edward Elgar . Google Scholar CrossRef Search ADS European Commission ( 2011 ) Innovation Union Scoreboard 2010. The Innovation Union's performance scoreboard for Research and Innovation. Brussels: European Commission. European Commission ( 2013 ) State of the Innovation Union 2012: Accelerating change. European Commission, Directorate General for Research and Innovation, Brussels. 21.3.2013. COM(2013) 149 final. Brussels: European Commission. European Union ( 2014 ) Innovation Union Scoreboard 2014 . Brussels : European Commission . European Union ( 2015 ) Innovation Union Scoreboard 2015 . Brussels : European Commission . Farrell M. ( 1957 ) ‘ The Measurement of Productive Efficiency ’, Journal of the Royal Statistical Society: Series A General , 120 / 3 : 253 – 81 . Google Scholar CrossRef Search ADS Freeman C. , Soete L. ( 1988 ) ‘ Innovation Diffusion and Employment Policies ’, Ricerche Economiche , 40 : 836 – 54 . Grupp H. , Schubert T. ( 2010 ) ‘ Review and New Evidence on Composite Innovation Indicators for Evaluating National Performance ’, Research Policy , 39 : 67 – 78 . Google Scholar CrossRef Search ADS Hagedoorn J. , Cloodt M. ( 2003 ) ‘ Measuring Innovative Performance: Is There an Advantage in Using Multiple Indicators ’, Research Policy , 32 / 8 : 1365 – 79 . Google Scholar CrossRef Search ADS Janger J. et al. ( 2017 ) ‘ The EU2020 Innovation Indicator: A Step Forward in Measuring Innovation Outputs and Outcomes ’, Research Polucy , 46 : 30 – 42 . Google Scholar CrossRef Search ADS Lee Y.-N. ( 2015 ) ‘ Evaluating and Extending Innovation Indicators for Innovation Policy ’, Research Evaluation , 24 : 471 – 88 . Google Scholar CrossRef Search ADS Liu J. S. , Lu W. M. , Ho M. H. C. ( 2014 ) ‘ National Characteristics: Innovation Systems from the Process Efficiency Perspective ’, R&D Management , 45 / 4 : 317 – 38 . Google Scholar CrossRef Search ADS Mahroum S. , Al-Saleh Y. ( 2013 ) ‘ Towards a Functional Framework for Measuring National Innovation Efficacy ’, Technovation , 33 : 320 – 32 . Google Scholar CrossRef Search ADS Mersha T. ( 1989 ) ‘ Output and Performance Measurement in Outpatient Care ’, Omega , 17 / 2 : 159 – 67 . Google Scholar CrossRef Search ADS Mendonça S. , Santos Pereira T. , Mira Godinho M. ( 2004 ) ‘ Trademarks as an Indicator of Innovation and Industrial Change ’, Research Policy , 33 : 1385 – 404 . Google Scholar CrossRef Search ADS Nardo M. et al. ( 2005 ) ‘ Handbook on Constructing Composite Indicators: Methodology and User Guide ’, OECD Statistics Working Papers 2005/03, OECD Publishing, Paris. Nasierowski W. , Arcelus F. J. ( 2012 ) ‘ About Efficiency of Innovations: What Can Be Learned from the Innovation Union Scoreboard Index ’, Procedia—Social and Behavioral Sciences , 58 : 792 – 801 . Google Scholar CrossRef Search ADS OECD/Eurostat ( 2005 ) Oslo Manual. Guidelines for Collecting and Interpreting Innovation Data , 3rd edn . Paris: OECD Publishing . Saltelli A. ( 2007 ) ‘ Composite Indicators between Analysis and Advocacy ’, Social Indicators Research , 81 : 65 – 77 . Google Scholar CrossRef Search ADS Simar L. , Wilson P. W. ( 1998 ) ‘ Sensitivity Analysis of Efficiency Scores: How to Bootstrap in Nonparametric Frontier Models ’, Management Science , 44 / 1 : 49 – 61 . Google Scholar CrossRef Search ADS Zabala-Iturriagagoitia J. M. et al. ( 2007a ) ‘ Regional Innovation Systems: How to Assess Performance ’, Regional Studies , 41 / 5 : 661 – 72 . Google Scholar CrossRef Search ADS Zabala-Iturriagagoitia J. M. et al. ( 2007b ) ‘ What Indicators Do (or Do Not) Tell Us about Regional Innovation Systems ’, Scientometrics , 70 / 1 : 85 – 106 . Google Scholar CrossRef Search ADS Appendix The Efficiency of the EU28 Innovation Systems Table A1. Year 2014 Country Output (M = 8) Input (N = 4) Summary Innovation Index SIIit Normalized Summary Innovation Index SIÎit Ranking SIÎit Productivity Innovation Index PIIit Normalized Productivity Innovation Index PIÎit Ranking PIÎit Efficiency Innovation Index EIÎit Ranking EIÎit Bootstrapped Efficiency Innovation Index bEIÎit Ranking bEIÎit Lower bound Upper bound Returns to scale (IRS, CRS, DRS) Austria 0.637 0.488 0.599 0.799 10 1.305 0.519 12 1.000 1 0.878 17 0.679 0.995 CRS Belgium 0.603 0.507 0.627 0.836 7 1.189 0.473 15 0.827 22 0.768 22 0.698 0.822 DRS Bulgaria 0.207 0.095 0.188 0.251 28 2.179 0.866 2 1.000 1 0.886 9 0.687 0.997 CRS Croatia 0.281 0.243 0.306 0.408 23 1.156 0.459 16 1.000 1 0.914 2 0.836 0.994 CRS Cyprus 0.66 0.39 0.501 0.668 14 1.692 0.672 6 1.000 1 0.882 15 0.678 0.995 CRS Czech Rep. 0.497 0.395 0.422 0.563 16 1.258 0.500 14 1.000 1 0.899 4 0.773 0.996 CRS Denmark 0.701 0.63 0.728 0.971 2 1.113 0.442 17 0.920 19 0.849 19 0.771 0.915 DRS Estonia 0.544 0.628 0.502 0.669 13 0.866 0.344 22 0.653 26 0.610 26 0.564 0.650 DRS Finland 0.579 0.694 0.684 0.912 4 0.834 0.331 23 0.704 25 0.652 25 0.590 0.700 DRS France 0.52 0.479 0.571 0.761 11 1.086 0.431 18 1.000 1 0.883 12 0.683 0.996 CRS Germany 0.859 0.631 0.709 0.945 3 1.361 0.541 11 0.839 21 0.780 21 0.718 0.835 DRS Greece 0.516 0.205 0.384 0.512 19 2.517 1.000 1 1.000 1 0.885 10 0.690 0.996 CRS Hungary 0.273 0.304 0.351 0.468 20 0.898 0.357 21 1.000 1 0.888 6 0.751 0.995 CRS Ireland 0.578 0.339 0.606 0.808 9 1.705 0.677 5 1.000 1 0.882 16 0.684 0.996 CRS Italy 0.591 0.299 0.443 0.591 15 1.977 0.785 3 1.000 1 0.887 8 0.740 0.995 CRS Latvia 0.19 0.2 0.221 0.295 27 0.950 0.377 20 1.000 1 0.887 7 0.731 0.995 CRS Lithuania 0.193 0.447 0.289 0.385 24 0.432 0.172 28 0.503 28 0.467 28 0.418 0.501 IRS Luxembourg 0.761 0.461 0.646 0.861 5 1.651 0.656 7 1.000 1 0.882 14 0.689 0.996 CRS Malta 0.391 0.309 0.319 0.425 21 1.265 0.503 13 1.000 1 0.891 5 0.751 0.995 CRS Netherlands 0.538 0.543 0.629 0.839 6 0.991 0.394 19 0.756 24 0.703 24 0.647 0.753 DRS Poland 0.253 0.38 0.279 0.372 25 0.666 0.265 27 0.921 18 0.860 18 0.772 0.918 DRS Portugal 0.566 0.366 0.41 0.547 18 1.546 0.614 8 1.000 1 0.905 3 0.802 0.995 CRS Romania 0.283 0.157 0.237 0.316 26 1.803 0.716 4 1.000 1 0.884 11 0.685 0.995 CRS Slovakia 0.391 0.275 0.318 0.424 22 1.422 0.565 10 1.000 1 0.918 1 0.818 0.995 CRS Slovenia 0.44 0.571 0.513 0.684 12 0.771 0.306 25 0.861 20 0.806 20 0.742 0.856 DRS Spain 0.464 0.315 0.414 0.552 17 1.473 0.585 9 1.000 1 0.882 13 0.700 0.995 CRS Sweden 0.575 0.698 0.75 1.000 1 0.824 0.327 24 0.604 27 0.563 27 0.513 0.601 DRS UK 0.439 0.577 0.613 0.817 8 0.761 0.302 26 0.815 23 0.758 23 0.691 0.811 DRS Country Output (M = 8) Input (N = 4) Summary Innovation Index SIIit Normalized Summary Innovation Index SIÎit Ranking SIÎit Productivity Innovation Index PIIit Normalized Productivity Innovation Index PIÎit Ranking PIÎit Efficiency Innovation Index EIÎit Ranking EIÎit Bootstrapped Efficiency Innovation Index bEIÎit Ranking bEIÎit Lower bound Upper bound Returns to scale (IRS, CRS, DRS) Austria 0.637 0.488 0.599 0.799 10 1.305 0.519 12 1.000 1 0.878 17 0.679 0.995 CRS Belgium 0.603 0.507 0.627 0.836 7 1.189 0.473 15 0.827 22 0.768 22 0.698 0.822 DRS Bulgaria 0.207 0.095 0.188 0.251 28 2.179 0.866 2 1.000 1 0.886 9 0.687 0.997 CRS Croatia 0.281 0.243 0.306 0.408 23 1.156 0.459 16 1.000 1 0.914 2 0.836 0.994 CRS Cyprus 0.66 0.39 0.501 0.668 14 1.692 0.672 6 1.000 1 0.882 15 0.678 0.995 CRS Czech Rep. 0.497 0.395 0.422 0.563 16 1.258 0.500 14 1.000 1 0.899 4 0.773 0.996 CRS Denmark 0.701 0.63 0.728 0.971 2 1.113 0.442 17 0.920 19 0.849 19 0.771 0.915 DRS Estonia 0.544 0.628 0.502 0.669 13 0.866 0.344 22 0.653 26 0.610 26 0.564 0.650 DRS Finland 0.579 0.694 0.684 0.912 4 0.834 0.331 23 0.704 25 0.652 25 0.590 0.700 DRS France 0.52 0.479 0.571 0.761 11 1.086 0.431 18 1.000 1 0.883 12 0.683 0.996 CRS Germany 0.859 0.631 0.709 0.945 3 1.361 0.541 11 0.839 21 0.780 21 0.718 0.835 DRS Greece 0.516 0.205 0.384 0.512 19 2.517 1.000 1 1.000 1 0.885 10 0.690 0.996 CRS Hungary 0.273 0.304 0.351 0.468 20 0.898 0.357 21 1.000 1 0.888 6 0.751 0.995 CRS Ireland 0.578 0.339 0.606 0.808 9 1.705 0.677 5 1.000 1 0.882 16 0.684 0.996 CRS Italy 0.591 0.299 0.443 0.591 15 1.977 0.785 3 1.000 1 0.887 8 0.740 0.995 CRS Latvia 0.19 0.2 0.221 0.295 27 0.950 0.377 20 1.000 1 0.887 7 0.731 0.995 CRS Lithuania 0.193 0.447 0.289 0.385 24 0.432 0.172 28 0.503 28 0.467 28 0.418 0.501 IRS Luxembourg 0.761 0.461 0.646 0.861 5 1.651 0.656 7 1.000 1 0.882 14 0.689 0.996 CRS Malta 0.391 0.309 0.319 0.425 21 1.265 0.503 13 1.000 1 0.891 5 0.751 0.995 CRS Netherlands 0.538 0.543 0.629 0.839 6 0.991 0.394 19 0.756 24 0.703 24 0.647 0.753 DRS Poland 0.253 0.38 0.279 0.372 25 0.666 0.265 27 0.921 18 0.860 18 0.772 0.918 DRS Portugal 0.566 0.366 0.41 0.547 18 1.546 0.614 8 1.000 1 0.905 3 0.802 0.995 CRS Romania 0.283 0.157 0.237 0.316 26 1.803 0.716 4 1.000 1 0.884 11 0.685 0.995 CRS Slovakia 0.391 0.275 0.318 0.424 22 1.422 0.565 10 1.000 1 0.918 1 0.818 0.995 CRS Slovenia 0.44 0.571 0.513 0.684 12 0.771 0.306 25 0.861 20 0.806 20 0.742 0.856 DRS Spain 0.464 0.315 0.414 0.552 17 1.473 0.585 9 1.000 1 0.882 13 0.700 0.995 CRS Sweden 0.575 0.698 0.75 1.000 1 0.824 0.327 24 0.604 27 0.563 27 0.513 0.601 DRS UK 0.439 0.577 0.613 0.817 8 0.761 0.302 26 0.815 23 0.758 23 0.691 0.811 DRS Source: Own elaboration from European Union (2014). Table A1. Year 2014 Country Output (M = 8) Input (N = 4) Summary Innovation Index SIIit Normalized Summary Innovation Index SIÎit Ranking SIÎit Productivity Innovation Index PIIit Normalized Productivity Innovation Index PIÎit Ranking PIÎit Efficiency Innovation Index EIÎit Ranking EIÎit Bootstrapped Efficiency Innovation Index bEIÎit Ranking bEIÎit Lower bound Upper bound Returns to scale (IRS, CRS, DRS) Austria 0.637 0.488 0.599 0.799 10 1.305 0.519 12 1.000 1 0.878 17 0.679 0.995 CRS Belgium 0.603 0.507 0.627 0.836 7 1.189 0.473 15 0.827 22 0.768 22 0.698 0.822 DRS Bulgaria 0.207 0.095 0.188 0.251 28 2.179 0.866 2 1.000 1 0.886 9 0.687 0.997 CRS Croatia 0.281 0.243 0.306 0.408 23 1.156 0.459 16 1.000 1 0.914 2 0.836 0.994 CRS Cyprus 0.66 0.39 0.501 0.668 14 1.692 0.672 6 1.000 1 0.882 15 0.678 0.995 CRS Czech Rep. 0.497 0.395 0.422 0.563 16 1.258 0.500 14 1.000 1 0.899 4 0.773 0.996 CRS Denmark 0.701 0.63 0.728 0.971 2 1.113 0.442 17 0.920 19 0.849 19 0.771 0.915 DRS Estonia 0.544 0.628 0.502 0.669 13 0.866 0.344 22 0.653 26 0.610 26 0.564 0.650 DRS Finland 0.579 0.694 0.684 0.912 4 0.834 0.331 23 0.704 25 0.652 25 0.590 0.700 DRS France 0.52 0.479 0.571 0.761 11 1.086 0.431 18 1.000 1 0.883 12 0.683 0.996 CRS Germany 0.859 0.631 0.709 0.945 3 1.361 0.541 11 0.839 21 0.780 21 0.718 0.835 DRS Greece 0.516 0.205 0.384 0.512 19 2.517 1.000 1 1.000 1 0.885 10 0.690 0.996 CRS Hungary 0.273 0.304 0.351 0.468 20 0.898 0.357 21 1.000 1 0.888 6 0.751 0.995 CRS Ireland 0.578 0.339 0.606 0.808 9 1.705 0.677 5 1.000 1 0.882 16 0.684 0.996 CRS Italy 0.591 0.299 0.443 0.591 15 1.977 0.785 3 1.000 1 0.887 8 0.740 0.995 CRS Latvia 0.19 0.2 0.221 0.295 27 0.950 0.377 20 1.000 1 0.887 7 0.731 0.995 CRS Lithuania 0.193 0.447 0.289 0.385 24 0.432 0.172 28 0.503 28 0.467 28 0.418 0.501 IRS Luxembourg 0.761 0.461 0.646 0.861 5 1.651 0.656 7 1.000 1 0.882 14 0.689 0.996 CRS Malta 0.391 0.309 0.319 0.425 21 1.265 0.503 13 1.000 1 0.891 5 0.751 0.995 CRS Netherlands 0.538 0.543 0.629 0.839 6 0.991 0.394 19 0.756 24 0.703 24 0.647 0.753 DRS Poland 0.253 0.38 0.279 0.372 25 0.666 0.265 27 0.921 18 0.860 18 0.772 0.918 DRS Portugal 0.566 0.366 0.41 0.547 18 1.546 0.614 8 1.000 1 0.905 3 0.802 0.995 CRS Romania 0.283 0.157 0.237 0.316 26 1.803 0.716 4 1.000 1 0.884 11 0.685 0.995 CRS Slovakia 0.391 0.275 0.318 0.424 22 1.422 0.565 10 1.000 1 0.918 1 0.818 0.995 CRS Slovenia 0.44 0.571 0.513 0.684 12 0.771 0.306 25 0.861 20 0.806 20 0.742 0.856 DRS Spain 0.464 0.315 0.414 0.552 17 1.473 0.585 9 1.000 1 0.882 13 0.700 0.995 CRS Sweden 0.575 0.698 0.75 1.000 1 0.824 0.327 24 0.604 27 0.563 27 0.513 0.601 DRS UK 0.439 0.577 0.613 0.817 8 0.761 0.302 26 0.815 23 0.758 23 0.691 0.811 DRS Country Output (M = 8) Input (N = 4) Summary Innovation Index SIIit Normalized Summary Innovation Index SIÎit Ranking SIÎit Productivity Innovation Index PIIit Normalized Productivity Innovation Index PIÎit Ranking PIÎit Efficiency Innovation Index EIÎit Ranking EIÎit Bootstrapped Efficiency Innovation Index bEIÎit Ranking bEIÎit Lower bound Upper bound Returns to scale (IRS, CRS, DRS) Austria 0.637 0.488 0.599 0.799 10 1.305 0.519 12 1.000 1 0.878 17 0.679 0.995 CRS Belgium 0.603 0.507 0.627 0.836 7 1.189 0.473 15 0.827 22 0.768 22 0.698 0.822 DRS Bulgaria 0.207 0.095 0.188 0.251 28 2.179 0.866 2 1.000 1 0.886 9 0.687 0.997 CRS Croatia 0.281 0.243 0.306 0.408 23 1.156 0.459 16 1.000 1 0.914 2 0.836 0.994 CRS Cyprus 0.66 0.39 0.501 0.668 14 1.692 0.672 6 1.000 1 0.882 15 0.678 0.995 CRS Czech Rep. 0.497 0.395 0.422 0.563 16 1.258 0.500 14 1.000 1 0.899 4 0.773 0.996 CRS Denmark 0.701 0.63 0.728 0.971 2 1.113 0.442 17 0.920 19 0.849 19 0.771 0.915 DRS Estonia 0.544 0.628 0.502 0.669 13 0.866 0.344 22 0.653 26 0.610 26 0.564 0.650 DRS Finland 0.579 0.694 0.684 0.912 4 0.834 0.331 23 0.704 25 0.652 25 0.590 0.700 DRS France 0.52 0.479 0.571 0.761 11 1.086 0.431 18 1.000 1 0.883 12 0.683 0.996 CRS Germany 0.859 0.631 0.709 0.945 3 1.361 0.541 11 0.839 21 0.780 21 0.718 0.835 DRS Greece 0.516 0.205 0.384 0.512 19 2.517 1.000 1 1.000 1 0.885 10 0.690 0.996 CRS Hungary 0.273 0.304 0.351 0.468 20 0.898 0.357 21 1.000 1 0.888 6 0.751 0.995 CRS Ireland 0.578 0.339 0.606 0.808 9 1.705 0.677 5 1.000 1 0.882 16 0.684 0.996 CRS Italy 0.591 0.299 0.443 0.591 15 1.977 0.785 3 1.000 1 0.887 8 0.740 0.995 CRS Latvia 0.19 0.2 0.221 0.295 27 0.950 0.377 20 1.000 1 0.887 7 0.731 0.995 CRS Lithuania 0.193 0.447 0.289 0.385 24 0.432 0.172 28 0.503 28 0.467 28 0.418 0.501 IRS Luxembourg 0.761 0.461 0.646 0.861 5 1.651 0.656 7 1.000 1 0.882 14 0.689 0.996 CRS Malta 0.391 0.309 0.319 0.425 21 1.265 0.503 13 1.000 1 0.891 5 0.751 0.995 CRS Netherlands 0.538 0.543 0.629 0.839 6 0.991 0.394 19 0.756 24 0.703 24 0.647 0.753 DRS Poland 0.253 0.38 0.279 0.372 25 0.666 0.265 27 0.921 18 0.860 18 0.772 0.918 DRS Portugal 0.566 0.366 0.41 0.547 18 1.546 0.614 8 1.000 1 0.905 3 0.802 0.995 CRS Romania 0.283 0.157 0.237 0.316 26 1.803 0.716 4 1.000 1 0.884 11 0.685 0.995 CRS Slovakia 0.391 0.275 0.318 0.424 22 1.422 0.565 10 1.000 1 0.918 1 0.818 0.995 CRS Slovenia 0.44 0.571 0.513 0.684 12 0.771 0.306 25 0.861 20 0.806 20 0.742 0.856 DRS Spain 0.464 0.315 0.414 0.552 17 1.473 0.585 9 1.000 1 0.882 13 0.700 0.995 CRS Sweden 0.575 0.698 0.75 1.000 1 0.824 0.327 24 0.604 27 0.563 27 0.513 0.601 DRS UK 0.439 0.577 0.613 0.817 8 0.761 0.302 26 0.815 23 0.758 23 0.691 0.811 DRS Source: Own elaboration from European Union (2014). Table A2. Year 2015 Country Output (M = 8) Input (N = 4) Summary Innovation Index SIIit Normalized Summary Innovation Index SIÎit Ranking SIÎit Productivity Innovation Index PIIit Normalized Productivity Innovation Index PIÎit Ranking PIÎit Efficiency Innovation Index EIÎit Ranking EIÎit Bootstrapped Efficiency Innovation Index bEIÎit Ranking bEIÎit Lower bound Upper bound Returns to scale (IRS, CRS, DRS) Austria 0.599 0.519 0.585 0.791 11 1.154 0.380 15 1.000 1 0.867 5 0.714 0.993 CRS Belgium 0.566 0.543 0.619 0.836 9 1.044 0.343 18 0.767 22 0.708 22 0.639 0.764 DRS Bulgaria 0.227 0.144 0.229 0.309 27 1.582 0.520 7 1.000 1 0.857 18 0.636 0.993 CRS Croatia 0.244 0.328 0.313 0.423 23 0.744 0.245 26 0.654 25 0.599 25 0.537 0.649 IRS Cyprus 0.540 0.178 0.445 0.601 15 3.040 1.000 1 1.000 1 0.858 16 0.639 0.994 CRS Czech Republic 0.510 0.415 0.447 0.604 14 1.228 0.404 14 1.000 1 0.860 15 0.639 0.994 CRS Denmark 0.728 0.655 0.736 0.995 2 1.112 0.366 16 1.000 1 0.866 7 0.723 0.995 CRS Estonia 0.526 0.688 0.489 0.661 13 0.765 0.252 25 0.622 26 0.562 26 0.504 0.617 DRS Finland 0.603 0.669 0.676 0.914 3 0.902 0.297 21 0.889 21 0.815 21 0.742 0.884 DRS France 0.544 0.502 0.591 0.799 10 1.082 0.356 17 0.944 19 0.867 6 0.797 0.940 DRS Germany 0.723 0.718 0.676 0.914 4 1.007 0.331 19 0.701 24 0.636 24 0.555 0.697 DRS Greece 0.408 0.244 0.365 0.493 21 1.677 0.552 6 1.000 1 0.862 13 0.638 0.995 CRS Hungary 0.289 0.370 0.369 0.499 20 0.782 0.257 24 0.910 20 0.828 20 0.720 0.903 DRS Ireland 0.606 0.352 0.628 0.849 8 1.721 0.566 5 1.000 1 0.862 12 0.707 0.994 CRS Italy 0.572 0.305 0.439 0.593 16 1.873 0.616 4 1.000 1 0.866 8 0.717 0.994 CRS Latvia 0.234 0.385 0.272 0.368 26 0.607 0.200 27 0.524 27 0.477 27 0.410 0.520 IRS Lithuania 0.194 0.441 0.283 0.382 25 0.438 0.144 28 0.395 28 0.359 28 0.311 0.392 IRS Luxembourg 0.772 0.383 0.642 0.868 6 2.014 0.662 2 1.000 1 0.858 17 0.638 0.993 CRS Malta 0.589 0.379 0.397 0.536 18 1.554 0.511 8 1.000 1 0.872 4 0.743 0.994 CRS The Netherlands 0.570 0.452 0.647 0.874 5 1.261 0.415 12 1.000 1 0.861 14 0.678 0.995 CRS Poland 0.298 0.362 0.313 0.423 24 0.823 0.271 23 0.986 17 0.897 2 0.797 0.980 DRS Portugal 0.510 0.365 0.403 0.545 17 1.399 0.460 9 0.944 18 0.865 9 0.780 0.938 DRS Romania 0.223 0.113 0.204 0.276 28 1.976 0.650 3 1.000 1 0.851 19 0.635 0.993 CRS Slovakia 0.398 0.304 0.360 0.486 22 1.311 0.431 11 1.000 1 0.863 11 0.693 0.993 CRS Slovenia 0.523 0.536 0.534 0.722 12 0.974 0.320 20 0.987 16 0.903 1 0.816 0.980 DRS Spain 0.370 0.299 0.385 0.520 19 1.239 0.408 13 1.000 1 0.864 10 0.704 0.994 CRS Sweden 0.620 0.715 0.740 1.000 1 0.867 0.285 22 0.706 23 0.647 23 0.588 0.702 DRS United Kingdom 0.586 0.426 0.636 0.859 7 1.376 0.453 10 1.000 1 0.881 3 0.769 0.994 CRS Country Output (M = 8) Input (N = 4) Summary Innovation Index SIIit Normalized Summary Innovation Index SIÎit Ranking SIÎit Productivity Innovation Index PIIit Normalized Productivity Innovation Index PIÎit Ranking PIÎit Efficiency Innovation Index EIÎit Ranking EIÎit Bootstrapped Efficiency Innovation Index bEIÎit Ranking bEIÎit Lower bound Upper bound Returns to scale (IRS, CRS, DRS) Austria 0.599 0.519 0.585 0.791 11 1.154 0.380 15 1.000 1 0.867 5 0.714 0.993 CRS Belgium 0.566 0.543 0.619 0.836 9 1.044 0.343 18 0.767 22 0.708 22 0.639 0.764 DRS Bulgaria 0.227 0.144 0.229 0.309 27 1.582 0.520 7 1.000 1 0.857 18 0.636 0.993 CRS Croatia 0.244 0.328 0.313 0.423 23 0.744 0.245 26 0.654 25 0.599 25 0.537 0.649 IRS Cyprus 0.540 0.178 0.445 0.601 15 3.040 1.000 1 1.000 1 0.858 16 0.639 0.994 CRS Czech Republic 0.510 0.415 0.447 0.604 14 1.228 0.404 14 1.000 1 0.860 15 0.639 0.994 CRS Denmark 0.728 0.655 0.736 0.995 2 1.112 0.366 16 1.000 1 0.866 7 0.723 0.995 CRS Estonia 0.526 0.688 0.489 0.661 13 0.765 0.252 25 0.622 26 0.562 26 0.504 0.617 DRS Finland 0.603 0.669 0.676 0.914 3 0.902 0.297 21 0.889 21 0.815 21 0.742 0.884 DRS France 0.544 0.502 0.591 0.799 10 1.082 0.356 17 0.944 19 0.867 6 0.797 0.940 DRS Germany 0.723 0.718 0.676 0.914 4 1.007 0.331 19 0.701 24 0.636 24 0.555 0.697 DRS Greece 0.408 0.244 0.365 0.493 21 1.677 0.552 6 1.000 1 0.862 13 0.638 0.995 CRS Hungary 0.289 0.370 0.369 0.499 20 0.782 0.257 24 0.910 20 0.828 20 0.720 0.903 DRS Ireland 0.606 0.352 0.628 0.849 8 1.721 0.566 5 1.000 1 0.862 12 0.707 0.994 CRS Italy 0.572 0.305 0.439 0.593 16 1.873 0.616 4 1.000 1 0.866 8 0.717 0.994 CRS Latvia 0.234 0.385 0.272 0.368 26 0.607 0.200 27 0.524 27 0.477 27 0.410 0.520 IRS Lithuania 0.194 0.441 0.283 0.382 25 0.438 0.144 28 0.395 28 0.359 28 0.311 0.392 IRS Luxembourg 0.772 0.383 0.642 0.868 6 2.014 0.662 2 1.000 1 0.858 17 0.638 0.993 CRS Malta 0.589 0.379 0.397 0.536 18 1.554 0.511 8 1.000 1 0.872 4 0.743 0.994 CRS The Netherlands 0.570 0.452 0.647 0.874 5 1.261 0.415 12 1.000 1 0.861 14 0.678 0.995 CRS Poland 0.298 0.362 0.313 0.423 24 0.823 0.271 23 0.986 17 0.897 2 0.797 0.980 DRS Portugal 0.510 0.365 0.403 0.545 17 1.399 0.460 9 0.944 18 0.865 9 0.780 0.938 DRS Romania 0.223 0.113 0.204 0.276 28 1.976 0.650 3 1.000 1 0.851 19 0.635 0.993 CRS Slovakia 0.398 0.304 0.360 0.486 22 1.311 0.431 11 1.000 1 0.863 11 0.693 0.993 CRS Slovenia 0.523 0.536 0.534 0.722 12 0.974 0.320 20 0.987 16 0.903 1 0.816 0.980 DRS Spain 0.370 0.299 0.385 0.520 19 1.239 0.408 13 1.000 1 0.864 10 0.704 0.994 CRS Sweden 0.620 0.715 0.740 1.000 1 0.867 0.285 22 0.706 23 0.647 23 0.588 0.702 DRS United Kingdom 0.586 0.426 0.636 0.859 7 1.376 0.453 10 1.000 1 0.881 3 0.769 0.994 CRS Source: Own elaboration from European Union (2015). Table A2. Year 2015 Country Output (M = 8) Input (N = 4) Summary Innovation Index SIIit Normalized Summary Innovation Index SIÎit Ranking SIÎit Productivity Innovation Index PIIit Normalized Productivity Innovation Index PIÎit Ranking PIÎit Efficiency Innovation Index EIÎit Ranking EIÎit Bootstrapped Efficiency Innovation Index bEIÎit Ranking bEIÎit Lower bound Upper bound Returns to scale (IRS, CRS, DRS) Austria 0.599 0.519 0.585 0.791 11 1.154 0.380 15 1.000 1 0.867 5 0.714 0.993 CRS Belgium 0.566 0.543 0.619 0.836 9 1.044 0.343 18 0.767 22 0.708 22 0.639 0.764 DRS Bulgaria 0.227 0.144 0.229 0.309 27 1.582 0.520 7 1.000 1 0.857 18 0.636 0.993 CRS Croatia 0.244 0.328 0.313 0.423 23 0.744 0.245 26 0.654 25 0.599 25 0.537 0.649 IRS Cyprus 0.540 0.178 0.445 0.601 15 3.040 1.000 1 1.000 1 0.858 16 0.639 0.994 CRS Czech Republic 0.510 0.415 0.447 0.604 14 1.228 0.404 14 1.000 1 0.860 15 0.639 0.994 CRS Denmark 0.728 0.655 0.736 0.995 2 1.112 0.366 16 1.000 1 0.866 7 0.723 0.995 CRS Estonia 0.526 0.688 0.489 0.661 13 0.765 0.252 25 0.622 26 0.562 26 0.504 0.617 DRS Finland 0.603 0.669 0.676 0.914 3 0.902 0.297 21 0.889 21 0.815 21 0.742 0.884 DRS France 0.544 0.502 0.591 0.799 10 1.082 0.356 17 0.944 19 0.867 6 0.797 0.940 DRS Germany 0.723 0.718 0.676 0.914 4 1.007 0.331 19 0.701 24 0.636 24 0.555 0.697 DRS Greece 0.408 0.244 0.365 0.493 21 1.677 0.552 6 1.000 1 0.862 13 0.638 0.995 CRS Hungary 0.289 0.370 0.369 0.499 20 0.782 0.257 24 0.910 20 0.828 20 0.720 0.903 DRS Ireland 0.606 0.352 0.628 0.849 8 1.721 0.566 5 1.000 1 0.862 12 0.707 0.994 CRS Italy 0.572 0.305 0.439 0.593 16 1.873 0.616 4 1.000 1 0.866 8 0.717 0.994 CRS Latvia 0.234 0.385 0.272 0.368 26 0.607 0.200 27 0.524 27 0.477 27 0.410 0.520 IRS Lithuania 0.194 0.441 0.283 0.382 25 0.438 0.144 28 0.395 28 0.359 28 0.311 0.392 IRS Luxembourg 0.772 0.383 0.642 0.868 6 2.014 0.662 2 1.000 1 0.858 17 0.638 0.993 CRS Malta 0.589 0.379 0.397 0.536 18 1.554 0.511 8 1.000 1 0.872 4 0.743 0.994 CRS The Netherlands 0.570 0.452 0.647 0.874 5 1.261 0.415 12 1.000 1 0.861 14 0.678 0.995 CRS Poland 0.298 0.362 0.313 0.423 24 0.823 0.271 23 0.986 17 0.897 2 0.797 0.980 DRS Portugal 0.510 0.365 0.403 0.545 17 1.399 0.460 9 0.944 18 0.865 9 0.780 0.938 DRS Romania 0.223 0.113 0.204 0.276 28 1.976 0.650 3 1.000 1 0.851 19 0.635 0.993 CRS Slovakia 0.398 0.304 0.360 0.486 22 1.311 0.431 11 1.000 1 0.863 11 0.693 0.993 CRS Slovenia 0.523 0.536 0.534 0.722 12 0.974 0.320 20 0.987 16 0.903 1 0.816 0.980 DRS Spain 0.370 0.299 0.385 0.520 19 1.239 0.408 13 1.000 1 0.864 10 0.704 0.994 CRS Sweden 0.620 0.715 0.740 1.000 1 0.867 0.285 22 0.706 23 0.647 23 0.588 0.702 DRS United Kingdom 0.586 0.426 0.636 0.859 7 1.376 0.453 10 1.000 1 0.881 3 0.769 0.994 CRS Country Output (M = 8) Input (N = 4) Summary Innovation Index SIIit Normalized Summary Innovation Index SIÎit Ranking SIÎit Productivity Innovation Index PIIit Normalized Productivity Innovation Index PIÎit Ranking PIÎit Efficiency Innovation Index EIÎit Ranking EIÎit Bootstrapped Efficiency Innovation Index bEIÎit Ranking bEIÎit Lower bound Upper bound Returns to scale (IRS, CRS, DRS) Austria 0.599 0.519 0.585 0.791 11 1.154 0.380 15 1.000 1 0.867 5 0.714 0.993 CRS Belgium 0.566 0.543 0.619 0.836 9 1.044 0.343 18 0.767 22 0.708 22 0.639 0.764 DRS Bulgaria 0.227 0.144 0.229 0.309 27 1.582 0.520 7 1.000 1 0.857 18 0.636 0.993 CRS Croatia 0.244 0.328 0.313 0.423 23 0.744 0.245 26 0.654 25 0.599 25 0.537 0.649 IRS Cyprus 0.540 0.178 0.445 0.601 15 3.040 1.000 1 1.000 1 0.858 16 0.639 0.994 CRS Czech Republic 0.510 0.415 0.447 0.604 14 1.228 0.404 14 1.000 1 0.860 15 0.639 0.994 CRS Denmark 0.728 0.655 0.736 0.995 2 1.112 0.366 16 1.000 1 0.866 7 0.723 0.995 CRS Estonia 0.526 0.688 0.489 0.661 13 0.765 0.252 25 0.622 26 0.562 26 0.504 0.617 DRS Finland 0.603 0.669 0.676 0.914 3 0.902 0.297 21 0.889 21 0.815 21 0.742 0.884 DRS France 0.544 0.502 0.591 0.799 10 1.082 0.356 17 0.944 19 0.867 6 0.797 0.940 DRS Germany 0.723 0.718 0.676 0.914 4 1.007 0.331 19 0.701 24 0.636 24 0.555 0.697 DRS Greece 0.408 0.244 0.365 0.493 21 1.677 0.552 6 1.000 1 0.862 13 0.638 0.995 CRS Hungary 0.289 0.370 0.369 0.499 20 0.782 0.257 24 0.910 20 0.828 20 0.720 0.903 DRS Ireland 0.606 0.352 0.628 0.849 8 1.721 0.566 5 1.000 1 0.862 12 0.707 0.994 CRS Italy 0.572 0.305 0.439 0.593 16 1.873 0.616 4 1.000 1 0.866 8 0.717 0.994 CRS Latvia 0.234 0.385 0.272 0.368 26 0.607 0.200 27 0.524 27 0.477 27 0.410 0.520 IRS Lithuania 0.194 0.441 0.283 0.382 25 0.438 0.144 28 0.395 28 0.359 28 0.311 0.392 IRS Luxembourg 0.772 0.383 0.642 0.868 6 2.014 0.662 2 1.000 1 0.858 17 0.638 0.993 CRS Malta 0.589 0.379 0.397 0.536 18 1.554 0.511 8 1.000 1 0.872 4 0.743 0.994 CRS The Netherlands 0.570 0.452 0.647 0.874 5 1.261 0.415 12 1.000 1 0.861 14 0.678 0.995 CRS Poland 0.298 0.362 0.313 0.423 24 0.823 0.271 23 0.986 17 0.897 2 0.797 0.980 DRS Portugal 0.510 0.365 0.403 0.545 17 1.399 0.460 9 0.944 18 0.865 9 0.780 0.938 DRS Romania 0.223 0.113 0.204 0.276 28 1.976 0.650 3 1.000 1 0.851 19 0.635 0.993 CRS Slovakia 0.398 0.304 0.360 0.486 22 1.311 0.431 11 1.000 1 0.863 11 0.693 0.993 CRS Slovenia 0.523 0.536 0.534 0.722 12 0.974 0.320 20 0.987 16 0.903 1 0.816 0.980 DRS Spain 0.370 0.299 0.385 0.520 19 1.239 0.408 13 1.000 1 0.864 10 0.704 0.994 CRS Sweden 0.620 0.715 0.740 1.000 1 0.867 0.285 22 0.706 23 0.647 23 0.588 0.702 DRS United Kingdom 0.586 0.426 0.636 0.859 7 1.376 0.453 10 1.000 1 0.881 3 0.769 0.994 CRS Source: Own elaboration from European Union (2015). © The Author(s) 2018. Published by Oxford University Press. All rights reserved. For permissions, please email: journals.permissions@oup.com This article is published and distributed under the terms of the Oxford University Press, Standard Journals Publication Model (https://academic.oup.com/journals/pages/about_us/legal/notices)

Journal

Research EvaluationOxford University Press

Published: May 3, 2018

There are no references for this article.

You’re reading a free preview. Subscribe to read the entire article.


DeepDyve is your
personal research library

It’s your single place to instantly
discover and read the research
that matters to you.

Enjoy affordable access to
over 18 million articles from more than
15,000 peer-reviewed journals.

All for just $49/month

Explore the DeepDyve Library

Search

Query the DeepDyve database, plus search all of PubMed and Google Scholar seamlessly

Organize

Save any article or search result from DeepDyve, PubMed, and Google Scholar... all in one place.

Access

Get unlimited, online access to over 18 million full-text articles from more than 15,000 scientific journals.

Your journals are on DeepDyve

Read from thousands of the leading scholarly journals from SpringerNature, Elsevier, Wiley-Blackwell, Oxford University Press and more.

All the latest content is available, no embargo periods.

See the journals in your area

DeepDyve

Freelancer

DeepDyve

Pro

Price

FREE

$49/month
$360/year

Save searches from
Google Scholar,
PubMed

Create lists to
organize your research

Export lists, citations

Read DeepDyve articles

Abstract access only

Unlimited access to over
18 million full-text articles

Print

20 pages / month

PDF Discount

20% off