Create a book
Qualitative evaluation - Questionnaire & Interviews
From Stellar Deliverable 6.4
To complement the quantitative analysis, a set of online interviews ( 30’ Flashmeetings) have been conducted with some STELLAR workpackage leaders and a questionnaire has been sent to every SOA facilitator. This should ensure that needs and expectations are targeted at best.
Both interviews and questionnaires address the same issues:
- Description of the use (actual or intended) of the Stellar OA, in and out
- Which aspects is satisfactory and which aspects prevent the use
- View on the current status of the tools and services provided by Stellar
- Expectations for the coming two years in terms of resources or services
- Any other issues or topic ...
1 Summary of Interviews
2 Intended and actual use of the Stellar OA
WP1 (Grand Challenges) has not yet used SOA as such or explored its possibilities with respect to the workpackage agenda. It is recognized that there is a potential, especially to stay informed of the newest publications in TEL (possibility to use a feed) and the current trends. Indeed, the question arises of the place of such a resource when on the other hand a tool like Google Scholar is available and seems to provide a lot of information. WP3 (Researcher capacity) sees the benefit from uploading resources on SOA (e.g. LMU group can evidence that publications present on TeLearn are more cited than before). Theme Teams and Incubators can get a support from SOA to promote their product and select resources. SOA can also provide a support to the Alpine Rendez-vous (ARv) thanks to the access to the documents, the attached discussions (i.e. things working as a platform for the ARv), including a specific space for ARv archiving. WP4 (Next generation capacity) as well has not yet make use of SOA, but sees there a good support to the PhD community of practice, in collaboration with Graaasp (platform developed at EPFL); the main issue is less the technology than ensuring that students will adopt it.
The needs of WP2 (Strategic capacity) is of a different type. SOA will be use to invite contributors to get there a picture of the current trends and Stellar contributions, but there will not be contribution to the content of the repository since this workpackage does not plan this type of outcome. Things are different for the podcast which could be disseminated via the SOA scientific portal, some technical and strategic decisions will be taken for this purpose.
The case of WP5 (Community capacity) is rather specific. This workpackage main action is the design and implementation of an interactive stakeholder platform (TEL-Europe.eu) offering space and resources to a large variety of users including researchers and PhD students. The relation with SOA has indeed to be clarified. On the one hand WP5 view is that SOA will provide TEL-Europe with resources and TEL-Europe can provide as a feedback (rating of relevance, usability, etc.). On the other hand the presence of these two platforms invites to think about the specificity of the communication towards researchers versus general stakeholders. The perspective of WP7 is specific as well, it will essentially use SOA as a place where to get information on the evolution of Stellar networking and integration process (SNA analysis), for this purpose some data and metadata must be made available.
To sum up, the potential of SOA is well identified but at this stage all concrete actions are more projects than implementations due to the relative youth of the network (the first Theme Team and incubator are not yet created, the PhD community is just emerging, the stakeholder community and its platform have just been created, etc.). So ensuring a good awareness of the existence and potential of SOA is a challenge and the facilitation of its use must be part of the strategy (the role of the facilitators is critical in that respect). Two concerns dominate: first to avoid time consuming efforts (e.g. manual feeding) and clarifying the copyright issues.
3 Comments on the functionalities and state of SOA
The functionalities proposed by SOA are appreciated and discussed, depending on the usage habit and perspective of the interviewees. The emphasis is on getting information about the newest publications and trends, quotation index, etc. including statistics for SNA. But may be the most important expectation is to see the SOA being more significantly populated –the threshold of 100 000 bibliographical descriptions is mentioned by WP2 leader. Then it can be seen as a resource either for researchers, PhD students or stakeholders; the presence of Google Scholar which is easy to use and already largely populated question the type of service a specialized OA can offer. The strategy has to be collectively adopted and supported, as yet there is not enough coordination: where to start integration given the large number of possibility and offers.
4 Comments on the current status of Stellar tools and services
Stellar offers a large set of tools and services (Stellar project website, TELeurope.eu, SOA, Stellar WP6 Groupme.org, Stellar Flickr gallery, Stellar nextspace, etc.) of which some are initiated by Stellar instruments, working groups or individuals. This diversity of offer is much in the Web2.0 spirit, but is somewhat destabilizing for some users who get lost. This is increased by the possibilities of the Web2.0 (e.g. Mendeley, CiteUlike, etc.). There is a need for a clearer picture and structuration of all these. The situation is quite new, unexpected, linked to the boost of Web2.0 offers on Internet; eventually a challenge to Stellar integration.
The existence of the community as mentioned by WP1 could be an advantage which should be better exploited. This can allow designing more specific services (eg clustering publications, better adapted visualization tools, etc.), better adapted to the real needs (e.g. an easy to use and with a proactive support to Arv). On this respect a synergy must be looked for between SOA and TELeurope.eu.
5 WPs’ expectations for the coming two years in terms of resources or services
The priority should be on streamlining and consolidating, while keeping room for experimentation. What SOA can offer for the PhD students deserves a specific focus. Actually, while tools and discourses are of a 2.0 flavour, the practices are still quite 1.0. A strategic issue is to understand the internal motivations (of researchers, PhD students, stakeholders) and to adapt accordingly.
6 Additionnal comments
Communication among members of Stellar and the workpackages must be enhanced in order to have better integration and synchronization between the instruments (cross platform widgets may help). WP5 will provide SOA with recommendations coming from requirements of users of TELeurope.eu. There is an emerging opinion that SOA is first a place where is aggregated information about what is available (artifacts of all sorts). But, definitely the hot issue is to balance a stronger coordination for the network with the emergence of a Web2.0 culture.
7 Survey of SOA facilitators’ feedback
We present in this section the synthesis of the responses of the SOA facilitators to the questionnaire; full answers are available in Annex 7.1.
1. Description of the *use you have made or intend to make* of the Stellar OA, including providing feeds of data if not uploading material. Don't miss that "use" means also exploiting the resource (not only feeding it)
SOA is mostly acknowledged as a valuable source of information and publications focused on TEL. There is a wide range of uses, varying from ‘null’ to ‘full’, through ‘occasional’ and ‘in progress’. The lowest ones are explained by deeply-held practices for searching scientific information which favour the use of mainstreams (such as Google, Google scholar and other sources). These are practices that are going to be slowly changed. For the mid-ones, the archive is better seen as a complementary platform (e.g. to journal publications) or a platform dedicated mainly to Stellar material and authors. Regular users get current publications via RSS feed, upload their publications, download publications relevant to their work. And some consider promoting the usage of the OA by their partners in other projects. Advanced users are impatient to exploit rich metadata coming from the archive, and intend to integrate data and services from the SOA into science2.0 applications and widgets.
Most of the partners are willing to provide a publication feed from their institution, to feed the archive
2. *Which aspects make you (as a group) happy* (even if you have not yet used it) and *which aspects prevent* you from using the SOA service (including any disappointing experience, indeed).
Of course there are pros and cons and no feature makes unanimity. Even when a feature is largely applauded, still an opposite feeling can be found. Nevertheless the global picture shows as:
- Positive
The very existence of a TEL archive is largely appreciated. As a single point of reference, it makes retrieval of documents easier. The provision of downloadable materials, especially papers and grey literature as well as the quality control (meta-data quality is high) are important.
The design and functionalities look attractive, with an intuitive interface. The browsing facilities, the FAQ section are fine. The search functionality is comprehensive.
Useful features are the Events section, the pointers to related work for a publication. Also the variety of resources available from the OA home page: publication, news, videos, podcasts and blogs is seen as attractive and relevant
Though to be improved, the existing interfaces to the OA (RSS feeds for latest additions and search results, OAI interface for meta-search engine) are well-considered.
- Deterrent
At the principle level, feeding and using the SOA is sometimes considered too costly in regard of intended benefits. Given the scale and notoriety of the archive, it procures a low visibility. Is it worth the effort?
At the content level, the database is not enough exhaustive and not enough original vs. other tools (Google, Scholar…). There are even no such new documents vs. Telearn. The critical mass of content is not reached (papers from major TEL conferences should be included asap)
At the features level, the website seems rather confusing at first sight. No possibility to sign in with Stellar usernames. The search section should be made more salient. Some features are pepeating to the ones in TEL-Europe (Science corner).
At the interface level, manual filling and updating of the archive is really deterrent. The archive is missing an easy way to automatically extract publications. And in general no easy input (by feed) nor output (for analysing purpose) for the archive data can be found.
3. Your view of the *current status of the tools and services provided by Stellar* either from the perspective of the senior researchers or the PhD students
This question was understood in two ways regarding the tools and services mentioned: those of Stellar in general or those of the SOA in particular.
At the SOA level, the portal is first highly appreciated when in conjunction with Stellar specific events (like workshops …). Then, a lot of very interesting and useful instruments (for browsing for interesting news) are in general appreciated, though it cannot compete with large operators (Googles…) when looking for a specific paper. And the daily habits and routines, regarding search for information, are seen as difficult to change. Integration of OA efforts with other endeavours (such as Mendeley or Research Gate) should be investigated, in order to broaden the scope and limit the amount of tools to be used. Some marketing approach might be considered (in order to help the archive grow in popularity and also data input). Last, if the main concern is to increase coherence, room should be maintained for experimentation.
At the Stellar level, the lack of coherent strategy concerning the different portals (open archive, website, TEL Europe), and tools and services, was mentioned. A directory is missing, hence tools are hard to find. A better integration of STELLAR site, TEL-Europe and SOA is necessary. And above all, the main concern could be, not the tool and services, but the community: effort is required to strengthen the feeling of a community (especially for PhD students).
4. A brief expression of *the expectations of your group* for the coming two years in terms of resources or services
A first general concern is the clarification of the respective roles of TEL-Europe vs. Open Archive. Integration of TEL-Europe and the OA is seen as a mean of attracting stakeholders to use the OA as their main source of TEL publications. Attention should focus on services fulfilling a certain goal and not lead to tools overlapping each other in function.
A second one is the improvement of the amount of data available, and thus the availability of a feeds importation mechanism.
The expectations address then three domains likely to promote and enhance the TEL community use of the SOA:
- Personalisation
- - A way of matching the user’s profile to resources; i.e. new resources matching the research interests, alerting service and feeds of relevant publications
- - Regular feedback messages on how often people have searched for, accessed, downloaded, etc. your own publications.
- Social networking services and resources grouping facilities
- - article review, comments and rating, tag clouds, other automated summaries
- - authors CV
- - reference management, citation index
- - visualizations and rankings of co-author / similar topic / co-citation analysis
- - implementing a "one paper per day" recommendation service, to which PhD students could subscribe
- - publishing newsletters, providing a list of paper recently added, recommended readings, description of the SOA features.
- - including help to prepare further workshops, meetings (‘how to manuals’)
- Exploitation
- An API for service-level access to the data would be welcome. This API would ideally include interfaces to the publication meta-data, the documents itself, and to usage statistics. It would not only allow reading but to some extent also creating and updating of entries.
- Data dissemination should be enhanced by considering integration with the “linked web” initiative. Data should as far as possible be made available also as linked data.
Last, the sustainability of all the resources and services is already a major concern.