Create a book
Quantitative evaluation
From Stellar Deliverable 6.4
This section builds on the monitoring plan from D6.2 “Monitoring the success of the open archive”. The evaluation of the Stellar OA is addressed along three modes:
- use as input repository (deposit of resources)
- use as source of information (download of resources)
- use as scientific networking place (discussion threads)
The key indicators applicable to the SOA focus on the affiliated institutions and registered users, the user‐generated content (resources submitted, interactive contributions, feeds provided), the content popularity, activity at the interface and enrolment.
The raw data we get about the visits of SOA shows that nothing happened until the time of the Stellar General Assembly in Nice, in September 2009. This is indeed an effect of the summer break. Then the traffic increased at the time of the Stellar GA, and again at the beginning of 2010 what reflects the activity related to the bootstrapping effort. At the present time, these data are not significant of a real use of the OA by Stellar members.
Contents |
1 Accounts
- number of affiliated institutions : 16 Stellar partners
- number of registered users : 23
- active users : 4 (4 online submissions)
All Stellar contractors are affiliated to the OA. This affiliation, due to the fact that the OA function on the model of institutional repositories, is compulsory. The number of registered users is currently rather low, but at this stage and given the strategy eventually adopted this is not surprising. The main origin of the resources are feeds treated by the SOA workforce, so the individual contributions are not the majority; the small number of active registered users has the same origin.
2 User‐generated content
- Number of resources submitted : 705
- bibliographical reference alone (notice): 10
- notice plus link to the document : 683
- notice plus associated document (pdf) : 11
- number of videos : 1
- Number of interactive contributions to threads attached to resources:
- Blog : 2 comments, 1 user
- forum : 6 posts, 2 users
- science corner : 1 post, 1 user
- Number of institutions providing a feed : in progress (see section 4)
The number of resources uploaded, a majority of bibliographical descriptions (BD) including a link to the actual document, has increased fast given the effort of the SOA workforce. These BD are obtained through a semi-automatic and supervised process which is being implemented to fit the users’ requirements and expectations. This will be explained below; it may be relevant to already mention that this is at the price of the quality of these BD when compared to the requirements of TeLearn. This choice is pragmatic and based on a Science 2.0 wisdom.
3 Content popularity
Most popular resources (since Jul 31, 2009 – to Jan 31, 2010)
These figures show by evidence two kind of elements.
- The absolute values may seem rather low, but this is related to the relative youth of the archive, the low general traffic (lack of notoriety), and the relative dearth of publications.
- The relative values indicate that users tend to favour resources for which material is downloadable. But, as the only downloadable material in the archive is at now the one linked to Stellar events (ECTEL workshops, Alpine RDV), none can tell if such resources appear at the head of the list because of its downloadable status or because of the Stellar event.
4 Click‐throughs
(clicks on the link to read the PDF‐file of the paper)
Most popular downloads (since Jul 31, 2009 – to Jan 31, 2010)
As stated previously, these figures may not be yet statistically significant enough to make assumptions. Nonetheless it seems that, building on this figure compared with the previous one, when a paper looks like one with a scientific content and with an attached document, in 30% to 50% of the occasions, the paper is downloaded.
5 Reflection
resources related to the resource the user has just viewed;
While this kind of recommendation system exists since the launching of the scientific portal, it is hardly that the performance of this system can be measured. Most of the users being anonymous ones, we still lack an analysis mechanism to assess if, when and how far, a particular visitor takes in account the recommendations he/she is provided with.
6 Enrolment
monitoring of feeds consumption; (powered by Google FeedBurner)
Monitoring of Stellar OA feeds ( 1) since Oct 5, 2009, 2) since Nov 20, 2009, 3) since Nov 30, 2009)
The monitoring of feed consumption was established after specification of indicators by deliverable D6.2 and thus implemented fall 2009. The average number of subscribers is low, in relation with the general low traffic at the SOA.
Monitoring of Latest Stellar OA publications feed
Considering the ‘Latest SOA publications’ feed the green line shows that the average number of subscribers started to improve significantly in January 2010. At the same time, the blue line shows ‘Reach’ evolution (i.e. the total number of people who have taken action — viewed or clicked — on the content in the feed).
7 Enrolment of a set of OA facilitators
number of facilitators : 16 (1 per stellar partner)
| Partners | UKOU | UJF | LMU | UB | UNOTT | LUH | KMRC | ZSI | EPFL | KUL | CNR-ITD | OUNL | KC | ALU-FR | ATOS | SCI |
| geographical spread | UK | FR | DE | UK | UK | DE | DE | AT | CH | BE | IT | NL | AT | DE | ES | IT |
| number of users brought | 5 | - | ||||||||||||||
| Number of documents brought | 1 | - | 1 | 1 | 1 |
The group of facilitators has been created easily, with a very good commitment of the persons and the institutions. Some of them tested the uploading of resources manually but quickly reported that they would prefer to support the submission by feeds to be treated automatically. This is reflected in the responses to the questionnaire we analyse below. This means that the indicators we chosen initially have to be modified to account for this type of use.