WP7. Monitoring, Evaluation and Quality Assurance

The overall objective of monitoring, evaluation and quality assurance of the STELLARnet activities is to critically reflect on the development of the Network of Excellence in terms of quality and effectiveness in order to a) learn from the obtained evaluation results; b) improve practices and outputs and c) on a case by case basis, to substantiate a mediation between partners.

The monitoring and evaluation process will be closely linked with the strategic management of the Network. At the beginning of the exercise the focus will be more on management issues and the organisation of basic integrative processes, whereas attention will shift towards the assessment of outputs and outcomes (valorisation, products and impact) as the network evolves.

Networks of Excellence (NoE) are one of the most complex instruments within the European Research Area (ERA) with high structural characteristics as stipulated in several programmatic publications of the EC (not at least in the Proposal of the European Commission on FP7). NoE explicitly target the fabric of the ERA in order to overcome the fragmentation of European research. Thus, the programmatic features of a NoE demand strong commitments of the participating partners along several dimensions.

Amongst others, the dimensions of a full-fledged NoE include

Especially the integrative aspects posed difficulties to many Networks of Excellence funded under the previous European Framework Programme for RTD and are, thus, the major concern in our evaluation approach.

In contrast to standard mid-term or ex-post evaluations of mainstream collaborative research projects with their strong thematically centred deliverable structures, the evaluation approach for assessing the relevance, efficiency, quality, and especially the impact and outcome of a Network of Excellence requires a far more sophisticated monitoring and evaluation design.

The main categories to assess the quality of integration within our Network include the following:

In order to provide the network members, the EC and other stakeholders, with reliable, pre- dominantly quantitative evaluation data, a mix of several monitoring, evaluation and quality assurance methods is proposed: 1) Longitudinal Graph-Analytical and Social Network Analysis 2) The Assessment of Progress Along Joint Intellectual Capital Indicators 3) An analysis of the Joint Infrastructure 4) Analysis of Feedback-Questionnaires of Participants at Training Events, Workshops and Conferences 5) Peer Review Based Quality Assurance of Deliverables.

The Monitoring Task is methodologically implemented by a longitudinal graph analytical Social Network Analysis. Our graph-analytical study design is based upon the analysis of the density (inclusion or exclusion) as well as the intensity (quality) of participation of the network partners in exchange relations with each other. In addition, the development of trust relations will be considered as a dimension of a more emotional quality. Data will be gathered at several points in time throughout the project implementation: at the beginning of the project (t1), and from then on a yearly basis. Thus, longitudinal results will be available to show to which extent the Network development in terms of progress and durability of integration met programme expectations. At the beginning of the social network analysis, realistic, but ambitious benchmarks will be pre-defined to assess whether the integration effects will have been substantial or not throughout the Network’s development. In order to avoid a carelessly arbitrary definition of benchmarks, relevant literature findings will be considered and the experiences of the previous, at least partially thematically comparable Network projects Prolearn and Kaleidoscope will be critically assessed. The quantitatively available results of the social network analysis will be assessed, visualised and communicated to the concerned parties.

Evaluation of STELLARnet means that results and impacts are regularly assessed via a yearly analysis based upon Joint Intellectual Capital Progress Indicators. In order to upgrade the evidence-base in quantitative terms for a critical reviewer, certain aspects of the relevant Network categories will be shown through an indicator system.

The proposed indicators are defined along standards of intellectual capital balance sheets, which has the advantage that certain data can be directly extracted from the already existing intellectual capital balance sheets of the Network partners. If intellectual capital balance sheets are not available, data for the proposed indicators can be easily gathered through a questionnaire. It is important to note, that the indicators will be constructed in such a way to grasp the specific requirements of a Network, meaning that next to standard performance and resource mobilisation aspects we will also address the collaborative and integrative aspects of STELLARnet. The indicators are presented in Table 2, which are adapted to the categories of a Network of Excellence, will be collected on a yearly basis.

Human Capital Development Structural Capital Education and Knowledge Dissemination Relational Capital Core Results
Total number of participants involved in the STELLARnet activities per year (expressed in full-time equivalents) Investments in research infrastructure in the thematic field of STELLARnet spent by each partner in € per calendar year Number of established joint study programmes (joint or double degrees) Number of new research proposals involving at least three of the partners of the network Number of new competitively acquired nationally (or locally) funded research projects in the thematic field of STELLARnet
Total number of researchers involved in the STELLARnet activities per year (expressed in full-time equivalents) Average success rate of proposal of NoE partners in national competitions in the thematic field of STELLARnet (e.g. basic national research funds, other funds) Number of jointly developed new higher education (or equivalent) training courses in the thematic field of STELLARnet Number of researchers from international cooperation partner countries (ICPCs or 3rd countries) involved in the STELLARnet activities Partner’s budget in € of new competitively acquired nationally (or locally) funded research projects in the thematic field of STELLARnet
Total number of young researchers (< 35 years) involved in the STELLARnet activities per year (expressed in full-time equivalents) Average success rate of proposal of NoE partners in international competitions in the thematic field of STELLARnet (e.g. FP7) Number of updated university (or equivalent) training courses in the thematic field of STELLARnet Number of researches from other disciplinary fields (units) involved in the STELLARnet activities Number of new competitively acquired internationally (e.g. European) funded research projects in the thematic field of STELLARnet
Total number of female researchers involved in the STELLARnet activities per year (expressed in full-time equivalents) Assessment of the physical research infrastructure relevant for STELLARnet by each team leader on a yearly basis (ranging from ‘international competitive infrastructure’ to ‘obsolete infrastructure’) Number of updated (including new) teaching textbooks and manuals in the thematic field of STELLARnet Participation of the STELLARnet staff in scientific, industrial and political fora (number of fora or gremial) Partner’s budget in € of new competitively acquired internationally (e.g. European) funded research projects in the thematic field of STELLARnet
Average number of days of an involved NoE researcher spent for continued education in the thematic field of STELLARnet (e.g. participation in relevant workshops, training seminars) Number of non STELLARnet members using STELLARnet services Number of young researchers (< 35) involved in research projects of the network partners in the thematic field of STELLARnet Number of consulting projects performed by the STELLARnet staff per year Number of publications of STELLARnet participants in reviewed scientific journals
Number of days spent by NoE researchers at the premises of another STELLARnet partner organisation (staff exchange) Sources of funding for joint projects (e.g. other programmes apart from FP7, bilateral programmes, etc.) Number of students (pre master level) involved in research projects of the network partners in the thematic field of STELLARnet Number of concluded technology transfers resulting from STELLARnet activities Number of joint publications between STELLARnet participants in reviewed scientific journals
Number of partner organisations participating in staff exchange   Number of PhD-candidates supervised by more than one STELLARnet partner in the thematic field under scrutiny Number of clients to whom STELLARnet software has been distributed (commercial or open source) Number of published monographs (books)
Number of NoE researchers recruited within the network (staff transfer) etc.   Number of diploma-candidates supervised by more than one STELLARnet partner in the thematic field under scrutiny Number of spin-offs from the STELLARnet activities Number of jointly published monographs (books)
    Number of events in which STELLARnet activities or results are presented Number of special cooperation agreements signed between some STELLARnet partners for exploitation of results Number of jointly published other scientific papers between STELLARnet participants (e.g. in conference proceedings, discussion papers etc.)
    Number of participants at STELLARnet events Number of specific projects with industry involvement Number of new requested common patent applications (resulting directly from the STELLARnet activities)
    Geographical coverage of these events   Number of new granted common patent applications (resulting directly from the STELLARnet activities)
        Number of concluded dissertations in the thematic field of STELLARnet under supervision of STELLARnet participants
        Number of joint presentations (or poster sessions) at scientific conferences by STELLARnet participants

STELLARnet Sustainability planning. The continuous monitoring and the evaluation results will feed important insights into the planning of the network sustainability. A core group of leading partners will be engaged in this task from the beginning of the project. Based on the experiences that partners bring in from their previous involvement in networks of excellence a roadmap for STELLARnet sustainability will be drafted. This roadmap will be an important strategic instrument for the network. Sustainability will be built on the main intellectual capital assets of STELLARnet, such as: structural capital (e.g. joint infrastructure); relational capital (e.g. cooperation agreements amongst partners); human capital (e.g. staff exchange programmes); education and knowledge dissemination (e.g. joint training courses); results (e.g. joint publication medium)

STELLARnet Centralised Quality Assurance. Besides the internal quality assurance (QA) procedures which each partner assigns to his or her tasks, two centralised QA functions will be delivered. They include Analysis of Feedback-Questionnaires (e.g. from STELLARnet events) and a managed Peer- Review process for all Deliverables.