Oppgrader til nyeste versjon av Internet eksplorer for best mulig visning av siden. Klikk her for for å skjule denne meldingen
Ikke pålogget
{{session.user.firstName}} {{session.user.lastName}}
Du har tilgang til Idunn gjennom , & {{sessionPartyGroup.name}}

2 Is there coherence between policy formulation and implementation of public funding of innovation projects?

A study of project selection mechanisms in the Norwegian Research Council
Erik Nesset er dr.polit i samfunnsøkonomi fra Universitetet i Bergen og professor i næringsøkonomi ved NTNU, Avdeling for Internasjonal Business (AIB) i Ålesund. Han har publisert en rekke artikler innen næringsøkonomi og markedsanalyser i internasjonale tidsskrift.
Bjørn G. Bergem er utdannet siviløkonom fra Norges Handelshøyskole og arbeider som seniorrådgiver ved Møreforsking Molde. Bergem har lang erfaring fra evaluering og analyser av virkemidler for innovasjon, og har spesielt arbeidet med resultat- og effektmålinger av brukerstyrte forsknings- og innovasjonsprosjekter i Forskningsrådet. Et annet hovedområde er klynge- og ringvirkningsanalyser, spesielt knyttet til de maritime og petroleumsrelaterte næringene i Møre og Romsdal.
Helge Bremnes har hovedfag i samfunnsøkonomi fra Universitetet i Bergen og er forskningsleder ved Møreforsking Molde. Hans hovedarbeidsfelt er næringsanalyser, og han har publisert en rekke artikler i internasjonale tidsskrifter.
SammendragEngelsk sammendrag


I denne studien utnyttes forskningsrådsdata som viser evalueringer av ulike aspekter ved norske bedrifters søknader om støtte til brukerstyrte innovasjonsprosjekter. Disse dataene er koblet sammen med et datasett som rapporterer ex post kortsiktige suksessindikatorer for de bedriftene som har fått slik støtte. Den empiriske analysen utføres i to steg i tråd med Heckmans korreksjonsprosedyre. I det første steget estimeres sannsynligheten for å få støtte med utgangspunkt i skår på evalueringskriteriene, samt andre kontrollvariabler. I det siste steget analyseres sammenhenger mellom ex ante evalueringsskår og ex post suksessindikatorer knyttet til samfunnsmessige eksterne effekter i form av kunnskapsmessig spredning og teknologispredning. De avhengige variablene er tellinger av ulike eksterne effekter som f.eks. antall publiseringer og antall ganger ny teknologi (innovasjon) utviklet i prosjektet er implementert av andre bedrifter. Resultatene viser at det er en viss form for samsvar mellom ex ante-evalueringer og ex post-resultat knyttet til kunnskapsspredning. Det er imidlertid lite samsvar mellom ex ante-evalueringer og ex post-resultater knyttet til teknologispredning. I tillegg viser det seg at store prosjekter gir større effekt både i form av kunnskaps- og teknologispredning, men de har samtidig mindre sannsynlighet for å få støtte enn mindre prosjekter.


In this study a unique dataset covering different aspects of official evaluators’ ex ante evaluation of Norwegian firms’ applications for public funding of user oriented innovation projects has been made accessible for empirical scrutiny. In addition, objective ex post short term success indicators can be linked to these evaluations. The empirical study is conducted in two stages following a Heckman correction procedure. In the second stage the relationship between the ex ante evaluation scores and a set of different objective measures of ex post spillover effects of the supported projects are estimated by a set of negative binomial regression models. The dependent variables at this stage are measured as counts of different spillover scores. There is to some extent coherence between ex ante evaluations of spillover effects and ex post short term knowledge diffusion effects. However, there seems to be a general lack of coherence between ex ante evaluations and ex post spillover effects connected to technology diffusion. In addition, larger projects produce more ex post spillover effects both in terms of knowledge and technology diffusion, but there is a significant negative relationship between the monetary amount applied for and the likelihood of being selected for support.

Keywords: Innovation projects,Public funding,Project selection,Heckman correction,Knowledge diffusion,Technology diffusion

2.1 Introduction

In contrast to many other areas of private investments, decision making regarding investments in research and development (R&D) and innovation are among the more difficult tasks for evaluators to perform. The reason for this is obvious: investment in R&D involves multiple sources of uncertainty that work together in complicated ways. The outcome of an R&D project is dependent on interactions between firm strategies, competitor strategies, and the stochastic macroeconomic environment, which may have an impact on investment costs (Hall et al., 2009). In addition to the fact that investment costs are uncertain, there is also uncertainty about the time it takes to complete the R&D project. In essence, there is learning while investing. Moreover, there is also the possibility of technological barriers which can put an end to the project.

For public funded projects there is an extra edge to the complexity of R&D investments. Hsu et al. (2003) offer two reasons for this:

…public funding of R&D projects generally involves strategic and long-term investment, implying that conventional financial justification approaches probably are inadequate …the allocation of R&D resources in the public sector may be influenced by political factors and a variety of interest groups (Hsu et al., 2003, p. 541).

Bozeman and Rogers (2001) point to the same two reasons, but also add differences regarding time horizons and restrictions connected to annual budgets. Being able to select the «right» R&D project eligible for public support is, nevertheless, paramount in order to be successful in the public R&D policy area. Acquiring knowledge of the R&D project selection process is thus important.

While there is a vast empirical literature focusing on economic effects of public R&D support (see e.g. Hall et al. 2009, for an extensive guide through the empirical literature), surprisingly little has been done regarding empirical analyses of public R&D project selection mechanisms. At least two factors may explain this. First, there is a general lack of adequate data suitable for such analyses. Second, the existing literature lacks well-founded theoretical models that incorporate a throughout description of the process of project selection. However, in recent years there has been a growing interest for this subject. Roper et al. (2004) suggest a framework for the ex ante evaluation of regional benefits based on ex post evaluations. In Desmet et al. (2004) they analyze the difference between ex ante evaluation criteria and ex post implementation. Another more recent example is Santamaria et al. (2010), where they develop an analytical model of the selection process for R&D cooperative projects to study the factors that motivate public project selection and corresponding funding. There is, however, still a lack of comprehensive empirical analyses focusing on the links between theoretical arguments for support and more objectively measured (ex post) spillover effects.

In the present study, a unique dataset covering different aspects of official evaluators’ ex ante evaluation of Norwegian private firms’ research support applications, in addition to objective ex post success indicators, has been made accessible for empirical scrutiny. Focusing on the so-called User Oriented Innovation Projects (UOIP), the objective of this study is to analyze the project evaluation process employed by the government agency Research Council of Norway (RCN). The main research questions addressed are:
  1. What is the relationship between the ex ante project evaluations and the ex post short term objective goal achievements in terms of different spillover effects?
  2. Are project selections in coherence with the main policy formulation?
The analysis should be of interest for several reasons. One obvious reason is that R&D is expensive and one would like to secure a positive return from public spending. Further, an examination of the project selection criteria will determine the characteristics of the projects that are actually implemented, and thus uncover the real objectives of the policy makers. In addition, they can affect not only responses to future calls, but also the definition and content of project proposals.

The rest of the paper is organized as follows. Section 2 sets out the theoretical platform and places the contribution of this study in the international literature on public R&D programs. Section 3 provides the essence of the Norwegian government strategy for supporting private companies. Section 4 describes the dataset and the empirical methodology, while section 5 presents the empirical analysis. Section 6 concludes the paper.

2.2 Theoretical platform and related literature

Despite its substantial complexity, it is widely recognized that R&D plays a key role as one of the main contributors to sustainable growth in highly industrialized economies. It is also generally accepted that the social return to R&D investments exceeds the private rate, leading to suboptimal rate of investment in R&D. Two popular reasons for underinvestment in private R&D are often mentioned in the literature. First, firms often have better information about success factors than their potential lenders. The combination of such informational asymmetries and risky investments – which often is the case for R&D – may typically lead to credit rationing in the financial markets. This will reduce the level of R&D investments (see e.g. Bernanke and Gertler, 1990; Blinder and Stiglitz, 1983). Second, it is argued that firms that undertake R&D investments are faced with the risk of not being able to appropriate the return from R&D. This may result in underinvestment from the social planner’s perspective (Schumpeter, 1976; Arrow, 1962; Nelson, 1959). Both arguments mentioned above provide a strong justification for government support for R&D activities.

Another argument for R&D support stems from the new endogenous growth theory (e.g. Romer, 1986; 1990), which sees knowledge as a non-rival partial excludable good. Because of weak or incomplete patent protection, reverse engineering and imitation, some of the knowledge and benefits from R&D are not kept within the firm. As a consequence there may be limited incentives for knowledge production by individual firms, but strong inter-firm knowledge spillover once knowledge production is coordinated on a broad basis.

Closely related to the discussion of theoretical arguments for public funding of R&D is the concept of additionality of the public support. Research projects with low risk profile will have easier access to private funding than riskier projects. User oriented R&D projects focusing too much on economic yield may face the danger of attracting projects with low risk, low innovation and acceptable yield, but that would have been realized without public support. From a pure market failure policy point of view it is the combination of externalities (spillover effects) and additionality that matters. The social economic benefit of R&D support is dependent on the degree of both of these aspects, and four different cases can be distinguished, as shown in figure 2.1.

Figure 2.1 Degrees of additionality and external effects: a classification of projects.

High degree of both additionality and spillover (external) effects is obviously the «first best case» for public support. With high degree of spillover effects but low degree of additionality (the «Coase-case»), where spillover effects are internalized without any public intervention, no support is needed. Low degree of spillover effects and high degree of additionality is the «second best case» for public support, while support in the case of low degree on both the additionality and the spillover effect dimensions apparently is a waste of public money. First best R&D support policy thus implies that a public R&D agency like RCN should select projects with both high degree of additionality and high degree of spillover effects.

In addition to these classical market failure arguments, other market failure arguments have been brought up within the fast growing literature on innovations and economic growth. Early innovation theory (Nelson, 1993; Rothwell, 1992; Lundvall, 1992) also emphasizes knowledge spillover, but in a much more faceted way than the former approaches, and with focus on the various channels for knowledge spillover. Such channels may, for example, be links between basic or applied science and innovative output, collaborative research between producers (networks), intermediate market linkages where R&D is embodied in the traded products, and so on. In the more recent geographically oriented innovation literature the focus is much stronger on the effects of sectoral proximity, «related variety», and localized socio-institutional networks on innovation and economic growth (e.g. Frenken et al., 2007; Boschma et al., 2009; Rodriguez-Pose and Crescenzi, 2008; Eriksson and Lindgren, 2009). Based on recent empirical analyses of firm innovation in Norway, Fitjar and Rodriguez-Pose (2011) and Frøystad and Nesset (2015) claim that interaction with («pipelines» to) a diversity of international partners is more important for innovation than interaction with local partners. This means that the degree of international cooperation should be considered when evaluating R&D projects applying for public support. There is also an important distinction made in the literature between formal and informal diffusion of knowledge. Storper and Venables (2004), introduced the term «buzz» to describe more informal face-to-face and mouth-to-ear informational flow, and showed how this can flourish in localized regions and create innovations. Externalities may, however, also be spillover due to pure market size effects that create demand and/or cost linkages between firms. Such pecuniary spillover effects are focused in the economic geography models of e.g. Krugman (1991) and Krugman and Venables (1996).

From a capability and system failure perspective, effects of public R&D support is often regarded as restricted by lack of R&D absorptive capacity within firms or by failures in R&D-institutions and -networks. Proponents of the National Innovation System (NIS) approach frequently advocate such a view. If a supported project with a matching grant is large relative to the supported firm’s R&D capability, i.e., a high support level intensity, R&D resources that otherwise would have been allocated to other forms of private R&D efforts (for example, other projects) may be tied up entirely on the supported project (see, for example, Nelson, 1993). This problem is probably of particular significance to the SME’s. R&D support mainly aimed at such firms should therefore not only be guided by the market failure correction perspective, but also by the possibility of R&D capability failures.

The foregoing discussion reflects some of the multifaceted difficulties that need to be addressed when R&D support programs are designed and implemented. Despite its complexity, it is essential to recognize the consequences of how the design of a R&D support program influences the way the program is implemented. R&D support programs are not invariant to the design and implementation (Desmet et al., 2004). In the majority of the literature where impact of public support to R&D is analyzed, it is often difficult to interpret results when design and implementation differ, or if information on implementation is lacking (Klette et al., 2000; David et al., 2000; Trajtenberg, 2002; Lach, 2002). The findings from this literature are often rooted on the actual selected projects. Lach (2002) suggests that the effect of R&D subsidies may be upward biased, because government bureaucrats tend to be under pressure to select projects with good prospects of success. However, there is no way of backing up this suspicion. Various authors go some way in uncovering implementation biases. For example, Lerner (1999) shows that there is evidence of government support of firms in backward regions in the case of the Small Business Research Innovation program.

2.3 RCN and the user oriented innovation program

RCN is responsible for the development and implementation of the national research strategy, and for enhancing Norway's knowledge base, as well as promoting basic and applied research and innovation in order to meet research needs within the society. According to their strategy documents, RCN activity is embedded in several different main goals (RCN, 2010). Among others, important goals are:
  • Enhanced quality and capacity: The Research Council will work to enhance the capacity and quality of, and promote the diversity in, Norwegian research.
  • Thematic priorities: The Research Council will work to strengthen research in areas of particular importance for trade and industry development.
  • Structure: The Research Council will work to promote constructive cooperation, distribution of responsibility and structures in the research system.
  • Learning: The Research Council will assist the translation of theoretical research into practical applications.
It is obvious that handling such a multidimensional objective is a challenging task. RCN administers various programs to implement its objective. One such program, which has increased its share of the total financial budget, is the so-called User Oriented Innovation Projects (UOIP). UIOP represents one part of RCN’s industrial R&D programs, directed towards private companies. The idea of letting companies contribute in the governing of research council programs, provided that they share their part of the financial burden, is an old one. In Norway such programs are dated as far back as to the middle of the 1960’s.

UOIP is one of RCN’s important funding instruments for promoting industrial research and innovation. This instrument constitutes a part of an overall public policy system to increase research-based value creation in Norway. UOIP is part of a broader innovation system to provide incentives for companies to cooperate with research institutions on R&D projects that reflect the strategies and knowledge needs of the companies. The decision to support R&D projects with a fixed percentage of matching grants involves an evaluation and a ranking of projects eligible for public support as well as the actual decision of which projects to support and at what support levels. RCN has specific guidelines for this evaluation and ranking process. These guidelines are described in five different evaluation steps, which are elaborated in the next section. The final decision of support is based on the evaluations and recommendations from the evaluators and, of course, the budgetary situation.

As a means for collecting data on the evaluation process RCN has implemented an evaluation tool, called PROVIS, which has been utilized since 2000 on all projects that have applied for public support. The data available in PROVIS gives information both on how the criteria effectively were used, and of their relative importance in the selection process.

2.4 Data and methodology

2.4.1 Data

To examine the relationship between the evaluators’ ex ante evaluations, by which selection of projects is based, and the short term ex post realizations from the projects, data from three different sources are utilized.

The ex ante data is collected from RCN’s two databases in the administration of user driven programs: FORISS and PROVIS. The PROVIS database contains data on how the executive officers in RCN and an expert panel assess project applications at the time the application is reviewed. For projects classified as UOIP, this assessment results in a grading of nine different aspects concerning quality of the project. The grading is measured on a seven-point Likert scale, with 1 indicating low degree, and 7 high degree of quality achievement. The project assessment is executed in several steps. First, a judgment of the General project quality (A1) is performed. Then, five different aspects that are considered important according to RCN’s R&D strategy are evaluated (Level of innovation (A2), Research content (A3), International cooperation (A4), Commercial benefits (A5) and Spillover effects (A6)). Evaluation of these five aspects is required to be independent of the different research programs, and the evaluator should not take into account whether the project is eligible for support or not. The next step is to evaluate the effect of the support, or the Additionality (A9), and the Relevance to the calls for proposals (A10). The last step in the evaluation process is to give a Total score (A11) of the project. For further description, see appendix A. The FORISS database supplements PROVIS with more objective data related to the project, which include, among others, company location, amount applied for, previous history record, year of application and duration of the project. These indicators are used as control variables in the different analyses. (See appendix C for a description of these control variables.)

Observations of ex post effects are based on a different dataset. For every project supported by RCN there is an obligation to report all achieved results according to a predefined list of success indicators. This dataset is based on compulsory reporting of quantitative characteristics from the project-owner to RCN. The list contains several count variables, such as number of publications, number of conferences, mass media articles, new products and new processes (see appendix B). The data bases are, however, mainly project specific. This means that they do not include many firm specific variables, as e.g. firm size. Table 2.1. provides descriptive statistics and correlations of the independent variables used in analyses 1 and 2.

Mean Sd. dev. (1) (2) (3) (4) (5) (6) (7) (8) (9) (10) (11) (12) (13) (14)
General project quality (A1) 4.77 1.15 1.00
Level of innovation (A2 4.51 1.21 0.56 1.00
Research content (A3) 4.33 1.35 0.56 0.64 1.00
International cooperation (A4) 3.37 1.65 0.29 0.31 0.35 1.00
Commercial benefits (A5) 4.39 1.04 0.53 0.55 0.45 0.25 1.00
Spillover effects (A6) 4.40 1.13 0.48 0.52 0.48 0.28 0.51 1.00
Additionality (A9) 4.48 1.15 0.38 0.41 0.45 0.29 0.33 0.40 1.00
Relevance to call for proposal (A10) 4.93 1.19 0.47 0.41 0.42 0.31 0.37 0.40 0.54 1.00
Total score (A11) 4.53 1.19 0.74 0.65 0.66 0.40 0.56 0.57 0.57 0.67 1.00
Amount applied for 0.38 0.51 0.02 0.18 0.17 0.15 0.15 0.14 0.13 0.08 0.09 1.00
District 0.47 0.50 0.01 0.02 –0.01 –0.09 0.01 –0.05 0.01 –0.01 0.01 –0.02 1.00
Research 0.55 0.50 0.19 0.24 0.39 0.10 0.16 0.21 0.26 0.22 0.28 0.10 0.03 1.00
Previous R&D experience 0.62 0.49 0.19 0.17 0.23 0.15 0.14 0.21 0.10 0.21 0.23 0.08 –0.03 0.31 1.00
Prioritized research area 0.40 0.49 –0.02 –0.03 –0.04 –0.02 –0.03 0.07 –0.01 0.01 –0.02 –0.05 0.01 –0.06 0.02 1.00

There are strong correlations between most of the ex ante evaluation aspects, but in particular between the three aspects Total score (A11), General project quality (A1) and Relevance to the calls for proposals (A10). These three variables are also strongly correlated to the other aspects, which in turn indicate that these variables to a large extent are measuring the same characteristics. In addition, RCN states that these three variables are prominent, and need high scores if a project is going to be selected. Thus, these variables reflect either more general assessments (A11), or are less specifically connected to the different theoretical arguments for R&D support (A1 and A10) than the other aspects.

2.4.2 Methodology

The empirical study is conducted in two stages, following a Heckman correction procedure (Heckman, 1979). In the first stage, data from all projects that applied for UIOP-support in the period 2000–2008 are used to examine how ex ante evaluation criteria scores influence the probability of being selected for support. The dependent variable is a dichotomous support status variable, and the econometric model is estimated by a probit regression. The independent variables are the different evaluation aspects (A1–A11). In addition, project size (amount applied for), a research dummy differentiating between pure research projects (value 1) and innovation projects (value 0), and dummy variables for different years of the proposed call, are included as control variables. The year dummies are meant to capture annual fluctuations in budgets. In order to control for these other factors beyond the evaluated aspects from the ex ante evaluation, three different regression models are specified. They differ with respect to the set of control variables included. The results from the first stage probit analysis are used to calculate the Inverse Mills Ratio (IMR) for each of the projects, which will be used in the second stage to overcome possible selection bias problems. This ratio is estimated by inserting the probit parameter estimates in the probit regression equation to calculate individual probability scores (IPS). The IPS-values are next used to compute the Heckman control factor LAMBDA, which is equivalent to the IMR.1

In the second stage the relationship between the ex ante evaluation scores and a set of different objective measures of ex post spillover effects of the supported projects are estimated. The dependent variables in this stage are measured as counts of different spillover scores, ranging from zero to many. The means of these data are significantly different from their variances, and a negative binomial regression model2 was thus chosen as the analytical estimating technique (Bound et al., 1982; Hausman et al., 1984; Gurmu and Trivedi, 1992). Further, as the dependent variables only contain information about projects that have been selected, the possibility of selection bias is apparent. To deal with this problem, the inverse Mills Ratio is estimated from model (3) in the first analysis, and included as a covariate to account for the selection bias in the second analysis. Robust standard errors are estimated in order to adjust for heterogeneity and misspecification problems in the model.

The explanatory variables are divided in three different sets. The first set of variables consists of the different evaluation aspects minus aspect A10: relevance to call for proposals. This variable serves as an important exclusion restriction. Relevance to call for proposals is obviously an important variable for selecting projects – i.e. high value of this aspect increases the probability of being selected. On the other hand, this aspect should have no particular effect on the outcome variables used in the second stage. The second set of independent variables consists of the same control variables as in the first stage estimation (project size, research and year dummies). The last set of variables consists of the Inverse Mills Ratio, the number of collaborating partners, and the two dummy variables R&D experience and University region. R&D experience is a dichotomous variable indicating whether the company has experience from earlier R&D projects (value 1) or not (value 0). This variable will probably, to some extent, pick up R&D absorptive capacity within the firm. The variable University region will probably capture a mixture of the different proximity effects that are addressed in the geographical oriented innovation theory approach.

2.5 Empirical results

2.5.1 Coherence between project selection criteria and theoretical arguments?

Table 2.2. presents empirical evidence of the influential factors in the selection process of user oriented innovations projects eligible for public funding in the years 2000–2008. Standard errors are computed using the robust estimator. The dependent variable is a dichotomous award status variable, which is equal to one if the firm received funding and zero otherwise. The sample consists of 2433 different projects of which 1007 received support.

Model (1) provides the baseline model with the evaluation indicators as independent variables. Models (2) and (3) add controls for a set of company attributes, together with dummies for different years of the proposal call to capture annual budgetary fluctuations.

For all three specifications it appears that project selection is in accordance with the evaluation scores given by the government agency. The regression analysis in model (1) shows that high scores given by an evaluator for the three more general evaluation aspects A1 (general project quality), A10 (relevance to call for proposals) and A11 (total score) are associated with higher probability of being selected, and this is true at a 1 percent level. In addition, the higher scores on evaluation aspect A9 (additionality) and A3 (research content) also increase the probability of being selected, which is in accordance with traditional theoretical arguments for public R&D support.

The above result is relatively robust after controlling for other factors that may influence a firm’s chance of winning government funding (Model 2 and 3). However, A3 (research content) does not have a significant effect in these model versions. There is also a negative relationship between the monetary amount of support which is applied for and the likelihood of being selected. This may indicate a preference for small non capital-demanding projects. However, this negative relationship may also originate from the fact that the research programs within UOIP sooner or later face budget constraints and in particular at the end of a program period. Thus, in these periods the government agency is tied to budget constraints when selecting projects. The regression result indicates that projects focusing on innovation rather than basic research have higher probability of being selected.

Dependent variable: Project selection (1) Coeff. Std.err.# (2) Coeff. Std.err.# (3) Coeff. Std.err.#
General project quality (A1) 0.384 *** (0.056) 0.388 *** (0.070) 0.373 *** (0.083)
Level of innovation (A2) –0.037 (0.053) –0.043 (0.067) –0.014 (0.078)
Research content (A3) 0.136 *** (0.045) 0.201 *** (0.058) 0.082 (0.071)
International cooperation (A4) –0.093 *** (0.030) –0.054 (0.035) –0.081 ** (0.038)
Commercial benefits (A5) –0.139 ** (0.056) 0.008 (0.071) –0.045 (0.079)
Spillover effects (A6) 0.016 (0.047) 0.026 (0.063) 0.047 (0.069)
Additionality (A9) 0.149 *** (0.050) 0.180 *** (0.063) 0.116 * (0.069)
Relevance to call for proposal (A10) 0.596 *** (0.055) 0.693 *** (0.068) 0.613 *** (0.077)
Total score (A11) 1.055 *** (0.075) 1.119 *** (0.104) 1.407 *** (0.126)
Amount applied for –0.542 *** (0.056) –0.443 *** (0.064)
Research/innovation –0.162 (0.112) –0.309 ** (0.130)
Year 2000 1.117 *** (0.301)
Year 2001 1.994 *** (0.271)
Year 2002 2.408 *** (0.296)
Year 2003 2.435 *** (0.274)
Year 2004 2.557 *** (0.255)
Year 2005 2.197 *** (0.263)
Year 2006 1.593 *** (0.238)
Year 2007 1.684 *** (0.257)
Constant –9.545 *** (0.357) –12.252 *** (0.545) –26.388 *** (1.369)
Number of observations 2433 2433 2433
Pseudo R2 0.402 0.436 0.479
–2Log likelihood 3236.3 2243.4 2014.9
# Robust standard errors based on bootstrapping: *: p <0.10, **: p <0.05, ***: p <0.01

2.5.2 Ex ante evaluations from RCN officials and objective spillover effects of the projects

The effectiveness of a research project is both ambiguous and multiple faceted and thus not possible to capture empirically with only one single indicator or variable. Because of this elusiveness, five different measures of goal achievement were used as dependent variables in the empirical analysis. Applying different success variables makes it possible to analyze various aspects of goal achievement. More specifically, the variables reflect spillovers connected both to knowledge diffusion and technology diffusion. Additionally, this multiple set of variables enables a check of the robustness of the findings.

Three of the dependent variables measure different aspects of knowledge diffusion: i) refereed publications, ii) participation in international conferences and iii) other ways of communicating the research result. Two of the variables focus on introduction of new technology, either to collaborating partners or to companies outside the project. All these five measures are, to some extent, related to spillover effects from the project.

The empirical results for the negative binomial regression equations are presented in tables 3 and 4 below. The negative binomial regression model predicting the different success factors is statistically significant in all regressions, as indicated by a Wald ?2 statistics. Further, the outcome of the likelihood-ratio tests for over-dispersion suggests that the probability of the data being generated from a Poisson process is very low, confirming that the choice of regression method were appropriate. In order to account for selection bias the inverse Mills ratio computed from the probit regression is included as a covariate in the negative binomial regressions. This covariate was significant in only one of five regressions. For this regression the use of this variable enables unbiased estimates.

Examining the impact of the control variables indicate that some paramount inferential conclusions may be drawn. The most obvious finding is, unsurprisingly, that money matters. The coefficient of the variable amount applied for is positive and statistically significant in all five regressions, suggesting that the more wide-ranging a project is, the greater are the numbers on the dependent success variables. Another important result from all the five regressions is that none of the more general or prominent aspects (i.e. project quality (A1), and total score (A11)) had any significant effect on spillovers, regardless of type of spillover. Neither did additionality (A9) have any significant effect.

Spillover: knowledge (information) diffusion

The empirical results for the negative binomial regression equations on factors that may have an influence on the quantity of informational diffusion of the results from the projects are presented in Table 2.3.

TABLE 2.3. Effects of evaluation scores and control factors on knowledge (information) diffusion. Estimated with negative binomial regression.
Publication Conference Communication
Coeff.. St.err.# Coeff. St.err.# Coeff. St.err.#
General project quality (A1) 0.095 (0.129) 0.098 (0.070) –00.85 (0.151)
Level of innovation (A2) –0.013 (0.120) –0.193 ** (0.067) 0.225 (0.128)
Research content (A3) 0.576 *** (0.089) 0.314 *** (0.058) –0.102 (0.111)
International cooperation (A4) –0.033 (0.063) 0.161 *** (0.035) 0.118 ** (0.050)
Commercial benefits (A5) –0.172 * (0.101) –0.075 (0.071) –0.373 *** (0.112)
Spillover effects (A6) 0.058 (0.104) –0.045 (0.063) 0.385 *** (0.105)
Additionality (A9) –0.013 (0.114) –0.005 (0.063) –0.190 * (0.101)
Total score (A11) –0.233 (0.257) 0.096 (0.068) –0.313 (0.233)
Amount applied for 0.613 *** (0.075) 0.513 *** (0.104) 0.827 *** (0.109)
Research or innovation 0.211 (0.211) 0.055 (0.112) –0.032 (0.191)
Year 2000 1.862 *** (0.676) 1.237 *** (0.314) 1.176 *** (0.391)
Year 2001 1.219 ** (0.593) 1.138 *** (0.336) 0.948 ** (0.428)
Year 2002 1.993 *** (0.754) 1.228 *** (0.409) 0.882 (0.542)
Year 2003 1.599 ** (0.605) 1.343 *** (0.296) 1.116 ** (0.504)
Year 2004 1.823 *** (0.601) 1.560 *** (0.301) 1.462 ** (0.539)
Year 2005 1.662 *** (0.594) 1.227 *** (0.293) 1.028 ** (0.422)
Year 2006 1.556 *** (0.552) 1.254 *** (0.255) 0.813 * (0.372)
Year 2007 1.806 *** (0.598) 1.130 *** (0.289) 0.938 ** (0.399)
Constant 10.965 ** (4.441) 8.281 *** (2.156) 11.015 *** (3.337)
Inverse Mills Ratio –0.102 (0.198) 0.120 (0.109) –0.451 ** (0.207)
Previous R&D experience 0.468 ** (0.206) 0.157 (0.275) –0.872 *** (0.193)
University region 0.233 (0.177) 0.152 (0.118) –0.072 (0.172)
Prioritized research area –0.255 (0.183) –0.055 (0.129) –0.032 (0.191)
Number of collaborating partners 0.032 (0.021) 0.041 *** (0.014) 0.005 (0.003)
Number of observations 1007 1007 1007
Log likelihood –1095.1 –1782.5 –1808.7
Pearson ?2/df 2.511 2.122 4.134
# Robust standard errors based on bootstrapping: *: p <0.10, **: p <0.05, ***: p <0.01

The results give some support for the claim that projects with high ratings on research content (A3) according to ex ante evaluation by RCN officials are also those projects that are more frequently made accessible to the public – either through articles in academic journals or presented on international conferences. The indicator that measures spillover effects (A6) has a highly significant and positive effect on objectively measured spillovers only in the regression with communication as the dependent variable. Further, the PROVIS indicator international collaboration is highly significant in the regressions that used international conferences and communication as dependent variables, but turns out to be insignificant for the regression where academic publication was dependent variable. Interestingly, the PROVIS indicator commercial benefit (A5) is estimated with a negative effect in all three regressions, but is only significant in two of the regressions. Although this is not a clear cut conclusion, it may moderately indicate that projects that are assessed by RCN officials to generate high private returns to a larger extent are less willing to publish the results from the research project. Further, applicants with previous R&D experience are also more likely to produce a greater amount of publishing. One may thus conclude that there is only partly coherence between the ex ante evaluation and ex post spillovers in terms of knowledge diffusion.


In this section the relationship between the score indicators from PROVIS and two measures of technology diffusion, i.e. diffusion of technology to collaborating partners and diffusion outside the project, is examined. The results from this analysis are reported in Table 2.4.

When technology diffusion is defined pursuant to collaborating partners, only one of the aspects shows significant effect on technology diffusion. Projects with high ex ante scores on commercial benefits (A5) had less technology diffusion ex post. In contrast, when estimating effects on technology diffusion outside the project, none of the evaluated aspects are significant. However, large projects and projects in firms located in university regions reports significantly higher technology diffusion both to collaborating partners and to actors outside the project, than other projects. Another interesting finding is the negative effect of the research dummy on technology diffusion to collaborating partners. This finding implies that innovation projects produce more technology diffusion among partners than projects with more focus on pure research. When looking at technology diffusion outside the project, projects focusing on research produce more technology diffusion. This lack of consistence emphasizes a potential problem of mapping evaluation criteria on one hand, with success indicators on the other. Put differently, terms like spillover effects can be defined and measured in contradictory dimensions. It is therefore difficult to base an evaluation on one single-dimensional criterion alone.

TABLE 2.4. Effects of evaluation scores and control factors on short term technology diffusion. Estimated with negative binomial regression.
Technology diffusion collaborating partners Technology diffusion outside the project
Coeff. St.err.# Coeff. St.err.#
General project quality (A1) 0.055 (0.113) 0.024 (0.186)
Level of innovation (A2) –0.119 (0.109) 0.174 (0.258)
Research content (A3) 0.039 (0.088) 0.016 (0.224)
International cooperation (A4) –0.005 (0.047) 0.018 (0.083)
Commercial benefits (A5) 0.291 *** (0.109) –0.067 (0.244)
Spillover effects (A6) –0.133 (0.081) 0.002 (0.143)
Additionality (A9) 0.073 (0.090) 0.236 (0.174)
Total score (A11) –0.014 (0.215) –0.235 (0.481)
Amount applied for 0.434 *** (0.088) 0.633 *** (0.184)
Research –0.317 * (0.181) 0.537 * (0.398)
Year 2000 3.422 *** (0.719) 4.221 *** (1.034)
Year 2001 2.911 *** (0.711) 3.246 *** (1.027)
Year 2002 2.964 *** (0.750) 3.580 *** (1.107)
Year 2003 3.031 *** (0.715) 2.559 ** (1.117)
Year 2004 3.313 *** (0.724) 3.930 *** (1.060)
Year 2005 2.630 *** (0.731) 2.466 ** (1.046)
Year 2006 2.875 *** (0.714) 3.493 *** (1.021)
Year 2007 1.400 * (0.809) 0.820 (1.332)
Constant 17.806 ** (5.093) 19.186 *** (7.489)
Inverse Mills Ratio –0.003 (0.159) 0.117 (0.322)
Previous R&D experience –0.417 ** (0.179) –0.940 * (0.378)
University region 0.334 ** (0.177) 0.646 * (0.297)
Prioritized research area –0.106 (0.185) –0.119 (0.312)
Number of collaborating partners 0.067 *** (0.016) 0.036 (0.025)
Number of observations 1007 1007
Log likelihood –789.9 –568.1
Pearson ?2/df 1.390 2.775
# Robust standard errors based on bootstrapping: *: p <0.10, **: p <0.05, ***: p <0.01

On the basis of the results concerning effects on technology diffusion, one must conclude that there is no coherence between the ex ante project evaluations and ex post short term spillover effects.

2.6 Conclusions

This paper provides a contribution to the discussion on the selection process when choosing projects that are eligible for public support within the RCN’s User Oriented Innovation Projects. The objective was to analyze the process of selection initiative from two different angles: (i) the coherence between theoretical motivated ex ante evaluation and the project selection, and (ii) the coherence between the ex ante evaluation and different objective measures of ex post spillover results obtained during the project period.

The main results from the first analysis (in section 2.5.1) indicate that the ex ante evaluation performed by RCN officials has a significant impact on the probability of being selected. In this sense, the ex ante project evaluation and project selection performed by RCN seem to be in coherence with classical theoretical arguments for support. The second analysis (in section 2.5.2), however, shows only partly empirical coherence between ex ante evaluation and ex post spillover effects. While there is some coherence between evaluation and spillover achievements related to knowledge diffusion (publications and other communications), there is a complete lack of coherence related to technology diffusion.

It is also interesting to note that, on one hand, there is a significant negative relationship between the monetary amount applied for and the likelihood of being selected, indicating that small projects are preferred. On the other hand, it is also evident that larger projects produce more ex post spillover effects both in terms of knowledge and technology diffusion. Another inconsistency is the ex ante priority of projects applied for by firms in more rural locations, and the fact that the ex post spillover results favors projects in firms located in more urban areas closer to universities.

One important lesson learnt from this empirical analysis is the need of taking the measuring complexity regarding assessment indicators seriously. The need for congruity between policy goals, assessment indicators and accompanying documentation of relevant success indicators is essential. Further analyses are, however, required in order to be able to give adequate advice to government agencies supporting private R&D. The data set needs to be extended in several ways. Future research should include more firm specific control variables and also more objective information on long term spillover effects.


Appendix A

Description of aspects
Description Evaluation by
A1 General project quality General project quality is an expression of how well the project complies with the requirements that should be posed to every project, independent of project content and type. Project quality includes project content and the players, and the following factors will be evaluated: whether the idea and objective are clearly defined, the overall project concept, the verifiability of the project’s objectives, the project plan (with milestones and a description of the results), and foundation in strategy, the players’ implementation capacity, and the capability to further exploit the results. Expert panel
A2 Level of innovation The term «innovation» is to be understood in the context of value creation. Evaluation will be focused on the level of innovation compared with the «state of the art» in a field, e.g. at the corporate level, in a particular industry or in a global context. Expert panel
A3 Research content This criterion will be used to rank the project on a scale ranging from simple development work to advanced scientific research. Evaluation will be focused on the extent to which the project produces new knowledge of significance for professional development in the field covered by the research, and the status of the project with regard to the international research front. Expert panel
A4 International cooperation Evaluation will be focused on the extent to which the project will contribute to the internationalization of Norwegian research and/or industry in the relevant field, and the plans for accomplishing this. Further, consideration will be given to whether the selection of international partners will help to enhance the project’s quality and feasibility. Administration
A5 Commercial benefits Evaluation will be focused on the project’s potential benefits for the participating enterprises. The potential refers to anticipated financial gains as a result of industrialization and commercialization, and will be compared against the aggregate expenses for the entire period (i.e. beyond the R&D project’s duration and expenses per se). Expert panel
A6 Spillover effects This is an indication of the impact of a project on society outside of the commercial benefits to the participating enterprises. It is the effects external to the companies involved that are to be assessed. Expert panel
A9 Additionality Evaluation will be focused on the extent to which support from the Research Council will trigger inputs, actions, results and effects assumed not to be feasible without the support. Administration
A10 Relevance to the call for proposals The project will be evaluated in relation to the guidelines set out in the call for proposals for the relevant activity/program. Administration
A11 Total score   Administration

Appendix B

Description of objective ex post success indicators
Variable name Description Computed as
Publication Number of times scientific articles related to project has been accepted in journals with a referee process. Count variable
Conference Number of times results from the project has been communicated in international conferences Count variable
Communication Number of times results from project has received publicity in mass media Count variable
Introduced new technology to collaborating partners Number of times new technology, as a result from the project, has been imported by collaborating partners Count variable
Introduced new technology outside the project Number of times new technology, as a result of the project, has been imported by non-involving firms Count variable

Appendix C

Description of control variables
Variable name Description Computed as
Amount applied for Total amount applied for. Log amount applied for
University region Indicates whether localization of the firm is close to universities. 1 = localized in one of the three counties where the largest universities are placed, 0 = otherwise. Dichotomous variable
Research Indicates whether the project primarily emphases on edge leading research. 1 = primary emphasis on innovation, 0 = primary emphasis on edge leading research. Dichotomous variable
Previous R&D experience Indicates whether the firm has a R&D record. 0 = no earlier experience, 1= has a track record of R&D experience. Dichotomous variable
Year xxxx  The year when RCN received the application. 1 = year equal to xxx, 0 = otherwise Dichotomous variable


Arrow, K.J. (1962). Economic welfare and the allocation of resources for invention. I R.R. Nelson (red.), The Rate and Direction of Inventive Activity. Princeton NJ: Princeton University Press.

Bernanke, B. and Gertler, M. (1990). Financial Fragility and Economic Performance. The Quarterly Journal of Economics, 105(1), 87?114.

Blinder, A.S. and Stiglitz, J.E. (1983). Money, Credit Constraints, and Economic Activity. American Economic Review, Papers and Proceedings, 73(2), 297?302.

Boschma, R.A., Eriksson, R. and Lundgren, U. (2009). How does labour mobility affect the performance of plants? The importance of relatedness and geographical proximity. Journal of Economic Geography, 9(2), 169?190.

Bound, J., Cummins, C., Griliches, Z., Hall, B.H. and Jaffe, A. (1982). Who Does R&D and Who Patents? National Bureau of Economic Research, Working Paper Series No. 908.

Bozeman, B. and Rogers, J. (2001). Strategic management of government-sponsored R&D portfolios. Environment and Planning C: Government and Policy, 19, 413–442.

David, P.A., Hall, B.H. and Toole, A.A. (2000). Is Public R&D a Complement or Substitute for Private R&D? A Review of the Econometric Evidence. Research Policy, 29, 1165?1183.

Desmet, K., Kujal, P. and Lobo, F. (2004). Implementing R&D policies: an analysis of Spain’s pharmaceutical research program. Research Policy, 33, 1493–1507.

Eriksson, R. and Lindgren, U. (2009). Localized mobility clusters: Impacts of labour market externalities on firm performance. Journal of Economic Geography, 9(1), 33–53.

Fitjar, R.D. and Rodriguez-Pose, A. (2011). When local interaction does not suffice: Sources of firm innovation in urban Norway. IMDEA Institutes, Working papers series in Economics and Social Sciences 2011/05.

Frenken, K., van Oort, F. and Verburg, T. (2007). Related variety, unrelated variety and regional economic growth. Regional Studies, 41(5), 685–697.

Frøystad, M.K. and Nesset, E. (2015). Geographical sources of company innovation for upstream companies in a regional maritime cluster. In Ø. Strand, E. Nesset and H. Yndestad (ed.), Fragmentering eller mobilisering? Fjordantologien 2014, 131–159. Oslo: Forlag1.

Gurmu, S. and Trivedi, P. (1992). Overdispersion tests for truncated Poisson regression models. Journal of Econometrics, 54, 347–370.

Hall, B.H., Mairesse, J. and Mohnen, P. (2009). Measuring the Returns to R&D, National Bureau of Economic Research. Working Paper Series No. 15622.

Hausman, J., Hall, B.H. and Griliches, Z. (1984). Econometric models for count data with an application to the patents R&D relationship. Econometrica, 52(4), 909–938.

Heckman, J. (1979). Sample selection bias as a specification error. Econometrica, 47, 153–161.

Hsu, Y.G., Tzeng, G.H. and Shyu, J.Z. (2003). Fuzzy multiple criteria selection of government-sponsored frontier technology R&D projects. R&D Management, 33(5), 539–551.

Klette, T.J., Møen, J. and Griliches, Z. (2000). Do subsidies to commercial R&D reduce market failures? Microeconometric evaluation studies. Research Policy, 29, 471–495.

Krugman, P.R. (1991). Increasing returns and economic geography. Journal of Political Economy, 99, 483–499.

Krugman, P.R. and Venables, A.J. (1996). Integration, specialization and adjustment. European Economic Review, 40, 959–968.

Lach, S. (2002). Do R&D subsidies stimulate or displace private R&D? Evidence from Israel. Journal of Industrial Economics, 50, 369–390.

Lerner, J. (1999). The Government as Venture Capitalist: The Long-Run Impact of the SBIR Program. Journal of Business, 72(3), 285–318.

Lundvall, B.Å. (ed.) (1992). National System of Innovation: Towards a Theory of Innovation and Interactive Learning. Pinter Publishers, London.

Nelson, R.R. (1959). The Simple Economics of Basic Scientific Research. Journal of Political Economy, 67, 297–315.

Nelson, R. (ed.) (1993). National Innovation System: a Comparative Analysis. Oxford: Oxford University Press.

RCN (2010). Norsk veikart for forskningsinfrastruktur. Forskningsrådet.

Rodriguez-Pose, A. and Crescenzi, R. (2008). Research and development, spillovers, Innovation systems, and the genesis of regional growth in Europe. Regional Studies, 42(1), 51–67.

Romer, P.M. (1986). Increasing Returns and long-run Growth. Journal of Political Economy, 94, 1002–1037.

Romer, P.M. (1990). Endogenous Technological Change. Journal of Political Economy, 98, 71–102.

Roper, S., Hewitt-Dundas, N. and Love, J.H. (2004). An ex ante framework for the regional benefits of publicly supported R&D projects. Research Policy, 33, 487–509.

Rothwell, R. (1992). Successful Industrial Innovation: Critical factors for the 1990’s. R&D Management, 14, 247–259.

Santamaria, L., Barge-Gil, A. and Modrego, A. (2010). Public selection and ?nancing of R&D cooperative projects: Credit versus subsidy funding. Research Policy, 39, 549–563.

Schumpeter, J.A. (1976). Capitalism, Socialism and Democracy. Routledge.

Storper, M. and Venables, A.J. (2004). Buzz. Face-to-face contact and the urban economy. Journal of Economic Geography, 4(4), 351–370.

Trajtenberg, M. (2002). Government Support for Commercial R&D: Lessons from the Israeli Experience. NBER: MIT Press.

1The following formula is used to estimate LAMBDA (and IMR): LAMBDA= ((1/sqrt(2?)) · (exp(–(IPS2/2))) / cdfnorm(IPS), where the nominator is the standard normal probability density function and the denominator (cdfnorm) is the standard normal cumulative distribution function, evaluated at the predicted probability value.
2Due to data variation larger than the mean (overdispersion), a negative binomial regression is used instead of a Poisson regression. The negative binomial regression has one more parameter compared to the Poisson regression, which is used to adjust the special case of overdispersion.

Idunn bruker informasjonskapsler (cookies). Ved å fortsette å bruke nettsiden godtar du dette. Klikk her for mer informasjon