Report of the Expert Group “ERA Indicators and ERA Monitoring”

Rémi Barré

Main speaker session 1.8

 

This document is the draft executive summary of a report which will be further adapted taking into consideration ongoing policy discussions. It will be finalised the 15 October 2009 in order to take into account feedback of CREST from presentations made at a specific workshop on 14 September, at the full CREST meeting on 29 September 2009 and from ERA groups focusing on each of the five ERA initiatives. The final report of this expert group is an input to the further elaboration of a proposal by the Commission on indicators for monitoring progress towards a European Research Area (ERA), responding to the formal demand made by the Council at its meetings 1-2 December 2008 and 18 May 2009.

 

The Mandate

 

The overall objective of the group is “to promote and contribute to the development of an evidence-based monitoring system on progress towards the ERA and a knowledge-based economy”[1]. It is an integral part of the “Ljubljana Process” that aims to define and build the ERA – which defines the European way to excellence in research and is a major driver of EU competitiveness in a globalised world.

 

Concretely, our group’s mission is to define three subsets of indicators: a) a comprehensive set of indicators to fully understand progress towards the ERA and the European knowledge economy; b) a subset of key ERA indicators to monitor progress toward the ERA in a synthetic way linked to key ERA objectives derived from the ERA Vision 2020 (‘ERA-Headline’ indicators); c) an even smaller subset of  indicators serving as references for targets of the contribution of the ERA in promoting a European knowledge society (‘Lisbon-related indicators’).  In addition to these three subsets, more focused indicators on the five ERA initiatives will be developed by the ERA groups.

  

1. Methodology: The ERA indicators Framework

 

To ensure that this report is tightly related to the definition of ERA set up by Governments, we based our work of identifying relevant indicators on a detailed analysis of the ERA Vision 2020 document which expresses all the relevant facets of ERA – and on the other key document which is the Commission report on the five policy initiatives. We define our model of the ERA along two structural dimensions: the “components” of the ERA and the “types of concern” which its monitoring supposes.

 

The Five Components of the ERA

 

Component 1- Knowledge Activities: Volume and Quality

“The ERA defines the European way to excellence in research and is a major driver of

EU competitiveness in a globalised world”

Component 2 – Knowledge Triangle: Flows and dynamics

“Strong interactions within the “knowledge triangle” (education, research and

innovation)  are promoted at all levels”

Component 3 – Fifth freedom: intra and extra-EU openness and circulation

“The ERA provides a seamless area of freedom and opportunities for dialogue,

exchange and interaction, open to the world”

Component 4 – The Societal Dimension

“The ERA is firmly rooted in society and responsive to its needs and ambitions”

Component 5 – Sustainable development and Grand challenges

“The ERA is firmly rooted in society in pursuit of sustainable development”

 

The Four types of concern for the ERA monitoring

 

Type A1 – Member states (MS) level policy actions                    

Type A2 – EU level policy actions

Type B   – ERA progress; state of the ERA

Type C   – ERA effects; Lisbon objectives

 

Table 1: ERA monitoring indicators: the overall framework to build the indicators

 

Components of the system     Types of concern		Component 1. K activities in EU [volume & quality]	Component 2. Knowledge ∆ [local, national, EU-wide]	Component 3. Fifth Freedom [conditions for EU-wide mobility and circulation  single market for K]	Component 4.  Societal Dimensions of ERA [Science in society]	Component 5. Sustainable Development and Grand Challenges
Type A Policy actions	Type A1 Member States level	▪ Public RD investment ▪Attractiveness policies ▪ Incentives for private RD investment	▪  MS Knowledge ∆  policies ▪  Coordination of  ∆ policies within MS	▪ Preparation of inter-operability of HE and R systems ▪ Opening public procurement, nat. programmes… ▪ Autonomous R. institutions ▪ Attractive working conditions for researchers	▪ Societal platforms ▪ involvement of stakeholders ▪ TA (Technology Assessment),, foresight	▪ SD policies and actions
	Type A2 EU-level and coordination across MS	▪ FP volume & structure ▪ ERC ▪ Joint programming. & instruments ▪ Speaking with one voice in international fora ▪ ESFRI & instruments	▪ Coordination of ∆ policies within EU ▪ EIT (European Institute of Technology) ▪ EU innovation policy and public-private interactions	▪ Common market for knowledge and its production factors across EU ▪ High performance EU-wide info systems	▪ Societal platforms ▪ involvement of stakeholders ▪ TA (Technology Assessment), foresight ▪ Ethical principles ▪ Cohesion and equity concerns	▪ Strategic partnerships between community & MS SD policies and actions
Type B ERA progress state of the ERA as EU R-I system		▪ Integration – coordination among MS of public R funds	▪ Intra-MS and intra EU flows between HE-R-I ▪ Public-private interactions & flows	▪ Intra-EU collaboration ▪ Knowledge flows ▪ K production factors  circulation  intra ERA ▪  Level of competition in EU for K production factors ▪ Access to complementary K & capacities across EU ▪Accessible world class R infrast.	▪ Science society activities ▪ Common foresights ▪ Social, regional, geographic cohesion	▪  Joint SD activities
Type C ERA Effects – Lisbon objectives towards a K society		▪ K activities (Volume, quality) ▪ World class  research ▪ Structural change:       – K intensity       – Specialisation (sectoral, geographic)       – Dynamics of firms ▪ Revealed attractiveness  of ERA ▪ Linkages – networks between ERA and the world; openness of ERA to the world			▪ Mutual trust & dialogue between society – S&T ▪ Public attitude to S&T ▪ equity: geographic, social, gender	▪ EU leadership in addressing global challenges and reaching SD goals

K: knowledge ; K∆ : knowledge triangle (higher education – research – innovation) ; ∆ policies: triangle policies

MS: member state (and, when relevant, associated countries)

HE: Higher education; R: research; I : innovation; SD: sustainable development

2. The proposed indicators

All the indicators presented:

1. Should be computed at the level of each MS plus associated states and at EU level as well

    as ERA level (including associated states)[2]

2. For at least two dates for analysing trends

3. In relevant cases with comparisons with at least the US, Japan, China

4. With ratio to account for size, which can be GDP, but also population

5. For the financial indicators, growth rate in real terms is to be systematically considered

6. In relevant cases, the indicators should be computed at the level of sub-groups of  

   countries, which have similar characteristics regarding their research base

 

For the Lisbon–oriented indicators list and the ERA-Headline indicators list, we present for each indicator, first the notion expressing what is needed for the monitoring of the ERA (“Intention”) and then a proposed quantitative characterisation of the notion to be addressed (“Indicator”), indicating its source and availability[3]. While the list of “Intentions” is meant to have lasting significance, the relevant indicator for a given notion can (and should) change over time, when new data become available or new ideas of indicators emerge; in a sense, the indicators presented here can be seen as examples of what can be done since there are often several possible indicators for characterising an intention[4].

 

 

 

Indicators Sets
Lisbon-oriented indicators Target indicators	ERA Headline indicators Key ERA indicators	Comprehensive set
Public investments in knowledge	Idem	Idem
European integration of research    Systems	Idem	Idem
Strength of the business research base of Europe	Idem	Idem
Transition towards a knowledge-based economy – structural change	Idem	Idem
Productivity of the economy	Idem	Idem
Contribution of research to address grand societal challenges	Idem	Idem
	ERA research actors cooperation and cohesion	Idem
	International cooperation in S&T and opening up to the world	Idem
	Mobility of researchers and research careers	Idem
	Knowledge transfer between public and private sector	Idem
	Pan-European research Infrastructures	Idem
	Activity level in knowledge-producing activities	Idem
	Excellence of the S&T base	Idem
	Human resource base of the ERA	Idem
	Knowledge-based innovation in Europe	Idem
	Firm dynamics – structural change	Idem
	International attractiveness of Europe for Business innovation and investment	Idem
	Confidence of society in science and the S&T community	Idem
		42 additional indicators on ERA

 

The Lisbon-oriented indicators

 

● PUBLIC INVESTMENTS IN KNOWLEDGE

Intention: Even though not sufficient, adequate funding levels are necessary for knowledge generation. In a knowledge society, public investment in R&D (both public and private), higher education and innovation is crucial.

 

Indicator: Public funding of R&D and higher education as a share of GDP

 

● EUROPEAN INTEGRATION OF RESEARCH SYSTEMS (policies)

Intention: The issue addressed is the “de-fragmentation” of the EU research systems and how it can be overcome by integrating (parts of) the national funding systems. This refers to National funds for trans-nationally coordinated Research.  Joint Programming (ERA Initiative) is one part of this.

 

Indicator: Share of National Public Funds to Trans-nationally Coordinated Research.

 

● STRENGTH OF THE BUSINESS RESEARCH BASE OF EUROPE

Intention: This strength is measured by the business expenditures in R&D and represents an important aspect of the innovation potential.

 

Indicator: Business RD expenditure (BERD) / GDP (or population) and growth in real terms

 

● TRANSITION TOWARDS A KNOWLEDGE-BASED ECONOMY – STRUCTURAL CHANGE

Intention: The knowledge economy develops largely through the structural evolution of economic activities towards more knowledge-intensive ones; this can be monitored by observing the evolution of the relative weight of the most knowledge intensive activities.

 

Indicator: Evolution of the share of total value added contributed by sectors with higher proportions of tertiary educated employees work force  

 

● PRODUCTIVITY OF THE ECONOMY

Intention: Productivity growth is the key factor behind competitive economies and sustainable long-term economic growth and living standards. The intention is to get a synthetic measure of the overall capacity of the economy to provide economic and social benefits to the people; of course distribution aspects would need to be considered to address the issue in a more complete way. This proposed indicator incorporates indirectly the impact of the knowledge economy on competitiveness through innovation.

 

Indicator: Growth rate of labour productivity per hour both for the whole economy and for the knowledge intensive part of it (as defined for indicator 4, above)

 

● CONTRIBUTION OF RESEARCH TO ADDRESS GRAND SOCIETAL CHALLENGES

Intention: Mobilising R&D to address Grand Societal Challenges and fostering the contribution of S&T to sustainable development and competitiveness are the overarching goals assigned to research policy in the ERA 2020 Vision. Optimally, a consistent methodology should be applied for all areas where EU-level agreements will be made for Grand Societal Challenges. Leadership and responsiveness of RD in the Grand Societal Challenges fields are aimed at.

 

Indicators:

(a) Leadership: World shares of scientific publications and European patent office (EPO) applications in the fields of the Grand Societal Challenges

(b) Responsiveness: World shares of scientific publications and EPO applications in the fields of the Grand Societal Challenges / World shares of scientific publications and EPO applications in all fields (‘specialisation’ in the fields of Grand Societal Challenges).

 

First area available: Climate change; data on environmentally related energy technology (SET-Plan themes)

 

 

The ERA Headline indicators

 

 

● Public investment in knowledge

Indicator: Public funding of R&D and higher education as a share of GDP

 

 

 

● European integration of research systems (policies)

Indicator: Share of National Public Funds for Trans-nationally Coordinated Research.

 

 

 

● ERA research actors cooperation and cohesion

Indicator: Share of co-publications (as regard to publications and to co-publications) which are with EU partners, among which with the 10 Member States with the lowest R&D intensity

 

● International cooperation in S & T and opening to the world (ERA Initiative)

Indicator: Share of co-publications (as regard to publications and to co-publications) which are with non- EU partners

 

● Mobility of researchers and research careers (ERA Initiative)

Indicator: Percentage of Doctoral degree Holders who obtained their doctorate in another EU country and/or have worked in another EU country

 

● Knowledge transfer between public and private sector (ERA Initiative)

Indicator: Share of publicly-performed research which is financed by business

 

● Pan-European research infrastructures

Indicator: Amount of funding committed to new pan-European research infrastcurtures in the framework of ESFRI, ERIC or other transnational agreements

 

 

 

● Activity level in knowledge-producing activities

Indicator:  share of R&D expenditures in the Gross domestic product

 

● Strength of the Business research base of Europe

Indicator: Business expenditure in R&D (BERD) / GDP or population; growth in real terms

 

● Excellence of the S&T Base

Indicators:

a) World share in top 10% most cited publications divided by world share of publications

b) World share in top 250 most academic research intensive universities

 

● The Human Resource Base of the ERA

Indicator: Importance of tertiary education graduates in Europe

 

● Transition towards a knowledge based economy – Structural change (1)

Indicator: Evolution of the share of total value added contributed by sectors with higher proportions of tertiary educated employees

 

● Knowledge based innovation

Indicator:  % of innovators as a percentage of all firms (Innovation of firms based on own research as well as adaptation of knowledge developed by others

● Firm Dynamics - Structural Change (2) 

Indicator: Percentage of high-growth firms.

 

● International attractiveness of Europe for Business innovation and investment

Indicator: Share of R&D expenditures by non-EU foreign affiliates in total business R&D expenditures and Share of R&D expenditures by non-EU foreign affiliates /their share of VA

 

● Productivity of the economy

Indicator: Growth rate of labour productivity per hour both for the whole economy and for the knowledge intensive part of it

 

● Mobilising R&D to address Grand Challenges – Contribution of S&T to sustainable development and competitiveness

Indicators:

(a) Leadership: World shares of scientific publications and EPO applications in the fields of the Grand Challenges

(b) Responsiveness: World shares of scientific publications and EPO applications in the fields of the Grand Challenges / World shares of scientific publications and EPO applications in all fields (‘specialisation’ in the fields of Grand challenges).

 

● Confidence of society in science and the S&T community

Indicator: responses in survey expressing interest and confidence of the citizens in S&T

 

 

The Comprehensive Set of Indicators

 

The purpose of this section is to propose a comprehensive set of indicators that covers in a systematic way the entries of the overall framework proposed above. This would facilitate, an understanding of the development of the various issues related to STI policies in the European context, but would also allow an analysis in terms of policy actions, ERA building and Lisbon objectives. This comprehensive set of indicators aims at contributing to the future versions of the STC report. In the main text of this report we only propose about 60 indicators that are readily available or quite easily obtainable.

 

 

3. The use of indicators for monitoring the ERA

 

Following our terms of reference we address now the issue of the monitoring system. In due time the appropriate mechanisms have to be chosen to execute that monitoring dependent on the results of the discussions regarding governance in CREST and the preferences of the Council in this respect.  

In this context, the aim of this section is focussed on highlighting the elements which are important for indicators to play a meaningful role in the monitoring process. 

 

The situation and challenges of the monitoring of the ERA

 

With the advent of ERA, the issue of monitoring is substantially changed for two reasons:

– the ERA is about the contribution of member states to realising it, with the Commission (and the FP) largely in a role of a catalyst for national systems and programmes. integration – coordination: the issue is to monitor national reforms and the integration of national programmes (policies) and systems

– the ERA is about integrating research into a “knowledge society” : “knowledge triangle” (higher education, research, innovation) and free circulation of knowledge (“5^th freedom”) are at the core of ERA and are related to policies beyond research policy.

 

The challenges presented by this situation have been widely recognised: the ERA monitoring and governance issues are prominent in the Ljubljana process. So, there is an on-going move towards a new scheme for monitoring.

 

But how could indicators fit into such a scheme and make a specific contribution? Such is the purpose of this report, which raises the question of how, in principle, such quantitative measurements can contribute to public policies.

 

The significance of indicators for the monitoring of public policies

 

If we seriously consider that indicators have a substantial role to play in such an eminently political process as the development of the ERA, then we need to clarify how indicators can be articulated to political processes.

 

We suggest the following understanding of the nature of indicators: that indicators are intrinsically dependent on a representation (or model or theory) of the topic at stake and are therefore debatable. The whole difficulty – and interest of the indicators for policy decisions – is to make explicit the underlying representation (or model or theory).

 

In this view, indicators are neither truth nor fallacy, but a common language with a high potential for collective deepening of issues with their underlying values, as long as certain methodological and procedural rules are respected. In this condition, they can be a powerful media for complex and high stakes policy monitoring – such as the ERA.

 

This understanding of the indicators makes them highly relevant for this task, provided the following conditions are fulfilled:

– the indicators are produced in a way which specifies the source data, treatments, approximations, the definition and rationale for the classifications used, the reasons for the proposed interpretation….

– possibilities are provided for the criticism of the indicators, for revealing the underlying assumptions and proxies, for questioning the classifications

- opportunities are given for alternative approaches, classifications, hierarchy of parameters and models of functioning of the system, leading to other indicators, or alternative interpretations or at least argued questioning of the interpretation of the indicators presented.

 

Towards using indicators for the monitoring of the ERA

 

Indicators are valuable in a monitoring process to the extent they enable the actors to reveal, express and discuss their representation of the issue at stake through their interpretation, criticism and eventually reconstruction of indicators.

 

Using indicators for the monitoring of the ERA would thus mean they are one of the vectors of the interaction among the actors, in tow possible contexts:

– a multi-actors assessment of the ERA-Headline indicators producing a thorough understanding of the building of ERA,

– key-issues ERA assessment (on ERA-Headline indicators) done by government representatives and the Commission, feeding into ministerial-level meetings (focussing on Lisbon oriented indicators).

 

Such ERA indicators assessment undoubtedly require significant preparation which takes time, resources and expertise in terms of the decision-making processes, the production of indicators and the collective learning methodologies. This point is crucial. Insufficient attention to it will lead to superficial work.

 

The large number of States involved, the objective complexity of the ERA making, its pluri-sectoral dimension (Knowledge triangle)…make the governance of the process and its monitoring a difficult task indeed. Indicators have a potential for addressing real issues in a universal language, but the condition for this potential to be realised is to have top-level and professional preparation. This requires a dedicated structure (body) with a significant operational capacity, as well as a high degree of legitimacy, both professional and political.

 

 

Conclusion: Towards a responsible and efficient use of indicators for the monitoring of ERA

 

Analyzing indicators with a systemic perspective

The suggestion is to interpret indicators not one by one but jointly, by subgroups of related indicators. The framework proposed for analyzing the ERA should be useful in clustering the indicators for such joint interpretation efforts.

 

A note of caution about using indicators to set targets

At least some of the Lisbon orientated indicators could be completed by the definition of a quantitative target of political significance. The risk is that since, by definition, an indicator measures a part (which is measurable) as a substitute or proxy to a larger picture (which is not measurable), setting a target based on the indicator leads to take care of the part (on which the indicator – target is set) and not of the larger picture (which is the real concern).

 

Setting targets and benchmarks for groups of countries

For the ERA monitoring process to be politically meaningful to all member states, it is suggested that interpretations, as well as targets and benchmarks be set up per groups of countries having similarities as regard to the ERA issues.

 

Linkage between the ERA and the national monitoring processes

Since the ERA is about the synergies between national policies articulated with EU level policies, it would be logical for monitoring processes at national level to be concerned with ERA monitoring, and reciprocally. This interaction would be greatly simplified if the National ‘Lisbon documents’ relied on similar and coherent methods, indicators and processes.

 

The indicators lists and reports as “living documents”

There are technical (data availability) and political (for example new Grand challenges) reasons for the lists of indicators to evolve. This is why there is a need of a formalised decision process for adjusting (for example every year) the lists of indicators with their precise technical definition.

 

Broad issues not fit for direct qualitative measurements and the question of composite indicators

A major point is the need to address broad issues, which are central for the monitoring of ERA. At least three such issues can be identified:

– the framework conditions in each country, also influenced by EU-level decisions,

– the policy decisions and roadmaps for reforms,

– the efficiency of the research systems and related public expenditures.

 

Such meta-issues combine a large number of complex elements, the relevance of which is itself a matter of debate and even political vision.

 

An approach sometimes proposed to monitor these broad issues is to build composite indicators, i.e. synthetic indicators based on the aggregation of as many indicators as there are elements to be considered. This is indeed a possibility and some composite indicators are widely used and well known (the Human development index of the United Nations for example). The difficulty in designing composite indicators lies both in the choice of the elements to be accounted for and in then their weighting for computing the aggregate synthetic indicator. In many cases, it seems easier to acknowledge the complex and qualitative nature of the issue, and to develop ad-hoc processes based on assessment studies or evaluations, expert advice and policy makers working groups.