Some Reflections upon the Paper “ERA Indicators and ERA Monitoring

Heike Belitz

Discussant session 1.8

 

I. Reduced set of ERA Headline Indicators versus a Composite Indicator

 

The group’s mission was to define indicators which allow a monitoring of the progress toward the ERA (“ERA making”) and an assessment the efficiency of the ERA in promoting a European knowledge society (“Lisbon objectives”).

Indicators are a useful tool in monitoring scientific, economic, and social development like the progress toward the ERA (“ERA making”). In their framework the experts differentiate between three subsets of indicators: the comprehensive set of about 60 indicators, 16 ERA Headline indicators, and a subset of 6 Lisbon-oriented indicators.

The ERA process is a complex, multi-dimensional process. Therefore, only the broader set of a battery of many separate indicators can capture all facets of the ERA policies, the ERA making and the effects. However, it is not easy to interpret the large amount of information. Furthermore it does not facilitate the communication with the general public which is an important point in the ERA monitoring. According to the invitation of the Council to identify a limited and consistent set of indicators which will serve as an operational tool the expert group therefore decided to reduce the full set of indicators into the selected ERA Headline indicators (and a smaller subset of Lisbon-oriented indicators).

Another solution of the problem of complexity of information would be to construct a composite indicator, i.e. a synthetic indicator based on the aggregation of as many indicators as there are elements to be considered.

This possibility is mentioned in the paper, but finally rejected. Arguments of the expert group against the use of a composite indicator methodology are

• the difficult choice of the elements to be accounted for and
• their weighting for computing the aggregate synthetic indicator.

Each one of these rests in fine on political decisions and if not, the indicator could legitimately be considered opaque (‘black box’).

These arguments need to be further discussed. With respect to the first argument it has to be emphasized that it is in any case necessary to make a choice of the “elements”, irrespective whether these are chosen for a composite indicator or for Headline indicators to measure the ERA process in an appropriate way. In both cases the choice should be based on a theory or a model and this underlying representation has to be made transparent. Therefore, the choice problem cannot be solved by making use of Headline indicator instead of a composite indicator.

The second problem i.e. the weighting of single indicators in the aggregation procedure can be solved in different ways as the example of a composite indicator of national innovative capacity for Germany and 16 other leading industrial countries shows.[1] Of course weighting procedures are subject of debates, a fact that can be viewed as an advantage as well, when using (composite) indicators.  In many cases in ad-hoc assessment processes based on studies, expert advice and policy makers working groups the important weighting problem of single indicators is only implicit and no subject to essential scientific und political dispute.

 

The above mentioned composite ‘Innovation Indicator for Germany’ is developed incrementally from one stage to the next, from the basics upwards via several intermediate levels, with the top level representing the overall score of national innovative capacity.

The summarized indicators are calculated on every level as a weighted average of the components. The weighting is determined empirically on the lower levels of indicator formation (i.e. based on the data) using the statistical procedure of principal component analysis (PCA). Using the first principal component, this analytical procedure calculates exactly that weighted average of the individual indicators that exhibits the largest variation between the countries being compared. It does so by allocating a relatively high impact to those indicators which exhibit a high variance from one country to the next, while harmonizing well with the cross-country variance of the other individual indicators with which it is being aggregated. This is based on the following concept: differences in the innovative capacity of the countries being compared are to be looked for wherever the indicators between the countries vary the most.

Another method of weighting is used on the second-to-last level, where seven sub-indicators are summarized on the side depicting the innovation system, the weighting is supported by empirical findings obtained from a survey of executives employed by innovative German companies.

To asses the robustness of composite indicators a sensitivity analysis using different weights or including and excluding single indicators has to be undertaken.

To sum up:  the composite ‘Innovation Indicator for Germany’ is one example of a growing number of composite indicators created to statistically capture rather complex socio-economic phenomena like the ERA process. These phenomena have in common that they may not be adequately represented by a small number of individual indicators.

 

II. Questions and Remarks on some proposed individual ERA Headline Indicators 

 

II.1        In my view the indicators to measure ERA progress should be selected such that – based on theoretical and empirical research results – it can presumed that a higher value is “better” than a lower one (i.e., that “ERA progress” in a country in total rises in tandem with the variable). Do you generally agree? Do you agree the assumption ‘the higher value the better’ i.e. in the following cases?

• Share of National Public Funds to Trans-nationally Coordinated Research.

• Share of publicly-performed research which is financed by business.
• Responses in surveys expressing interest and confidence of the citizens in S&T.

 

II.2        In many countries R&D tax incentives or tax credits play an important role in the system of financial support for R&D. Are these regarding their volume important tax credits part of the indicator Public funding of R&D and higher education as a share of GDP?

 

II.3        Smaller countries have to cooperate more intense internationally because the chance to find partners within the own country is lower. The same holds true for mobility. Smaller countries can’t offer graduate programmes in a broad range of scientific disciplines. Do you agree that one has to account for the size of the country in the following indicators of ‘ERA Making’ regarding international cooperation and networking?

• Share of co-publications (as regard to publications and to co-publications) which are with EU partners, among which with the the10 less scientifically developed MS  (why only the10 less scientifically developed MS ?)
• Percentage of Doctoral degree Holders who obtained their doctorate in another EU country and/or have worked in another EU country (why only in another EU country?)

 

II.4        The international attractiveness of Europe for business innovation and investment is proposed to measure by the share of business R&D expenditures by foreign affiliates. Do you have any evidence for the assumption that this share provides an indicator for the international attractiveness of an R&D location. According to OECD statistics the shares of foreign affiliates in BERD are around 14 percent in the USA, 26 percent in Germany and 38 percent in the United Kingdom. What do these figures tell us about differences in the attractiveness of these countries for R&D activities?

 

II.5        The Human Resource Base of the ERA is proposed to be measured by the Importance of tertiary education graduates in Europe. The definition of this indicator is not fully clear and should be specified.

Today more than 50 percent of university graduates in most MS are women. Yet the percentage of women active in the academic world falls dramatically over the period from the completion of doctoral work to the achievement of full-professor status. This percentage is particularly low in engineering, mathematics and the natural sciences (the MINT fields). This in turn negatively impacts the number of highly qualified women who are employed in the private sector. A significantly higher number of men are active in science and research. By contrast, a very large percentage of highly qualified women are employed in less innovative areas of the public sector (e.g., health care, education, social affairs). In many European countries there is a tremendous latent potential for the mobilization of women in the R&D and innovation process. Therefore a central ERA Headline Indicator should capture the participation of women in public and private R&D.