Home

From Stellar Deliverable 6.2

1 D6.2 Monitoring plan including indicators.

In this deliverable, we identify quantitative measures which we can obtain through the usage of the STELLAR portal, mash-ups and repository. The indicators will enable us to analyse the uptake of tools and services within the TEL community and to react appropriately in an iterative development approach.

The set of indicators proposed here allow us to study the impact of our work with respect to the uptake of tools and services. It goes without saying that this picture is complex and that there is no single indicator which provides us with an absolute benchmark for our success. Moreover, the tools and services differ in how their success can be analysed. That is why this deliverable focuses on the methodology needed to investigate the success of our work.

Several science 2.0 support tools have already been set up in STELLAR. In particular, these include wikis to support the development of the deliverables, flashmeeting as a virtual conferencing and brainstorming tool, a podcasting infrastructure and repository, a STELLAR blog, and a (not yet released) mobility tracker. Most of these tools are accessible directly under the roof of the stellarnet.eu portal.

In this deliverable, we present indicators for analysing the success of these tools and the overall portal. The approach consists of:

monitoring usage (cf. Section 1.1)
monitoring online buzz (cf. Section 1.2)
monitoring (technical) availability (cf. Section 1.3)
complemented by more qualitative data from surveys (cf. Section 1.4).

Section 1.5 elaborates on the applicability of the metrics to the STELLAR portal, repository, mashups, blog, podcast infrastructure and mobility tracker. Section 1.6 presents a monitoring plan that will be used as a basis to guide and report the WP6 monitoring and evaluation activities.

1.1 Monitoring usage

1.1.1 Why monitor usage?

Logging applications provides us with information about what users are doing with the application, who these users are and where they come from. Knowing what they do will raise more questions, like:

why do these users do these things with the application?
how did they find the application?
how did they use the application?
could they accomplish the tasks they wanted to do with the application?
did they encounter errors?
etc.

Answers to these questions can help to understand the success and problems of the application. They can provide hints on how to enlarge the user base and how to improve the usability of the application.

1.1.2 Which metrics can measure usage?

Croll and Power (2009) have presented an extensive overview of metrics for monitoring web usage. In this section, we describe some of the metrics and their relevance for measuring usage within STELLAR.

conversion and abandonment:

The conversion rate shows the application effectiveness in getting users to accomplish tasks or goals, for example register or find a relevant research paper. By adjusting the application so that more visitors achieve desired goals and fewer give up, conversion is improved. The inverse of conversion is abandonment. The abandonment rate thus reveals the failure in getting users to accomplish the goals.

To measure conversion and abandonment, a set of actions and/or goals has to be defined for the application. Activity is then analysed along these actions/goals to obtain an indication of the achievement of desired goals and to calculate the conversion and abandonment rates (cf. Figure 1b).

Figure 1: monitoring usage, based on [Teltzrow and Berendt, 2003]

click-throughs

For some applications, click-throughs can be important. For STELLAR, the application of this metric depends on the functionalities of the application. In the case of a paper recommender mash-up, it could for example be clicking on the link to read the PDF-file of the paper.

To get an idea of the amount of click-throughs, good metrics are:

the ratio of links served to links clicked,
clicks by visitors,
demographic data, and
the correlation to click-through ratio.

user-generated content

In many applications, user contribution to the content is key, e.g. wiki edits, ratings and comments. User-generated content contribution can be seen as the ratio of media consumed to media created. There are three types of user-generated content:

new content,
edits on content, and
responses.

If the application relies on user-generated content, it is useful to know:

how many people are adding to the site (cf. Figure 1d),
how many edits they make and how large these edits are,
whether they are creating content that other users of the site or application find useful,
the read-to-post ratio, and
difference between average users and super users.

In a wiki, there are other interesting metrics which make it possible to measure the expansion of the wiki, e.g. the number of incipient links (links within a wiki that point to another unwritten wiki entry).

Risks and issues: spamming or frequent editing due to disagreements can provide false indications of user-generated content contribution.

accounts

Another measure of continuing activity on the site or application is the number of new accounts created in a time period. When the application uses accounts, it is also possible to track active users and idle users and thus measure the activity of the community. Alternatively, cookies can be used as a basis for measuring web activity.

To use this metric, it has to be defined what “idle” and “active” mean; e.g. active users have to log in once a month or they have to accomplish some task.

traffic sources

It is interesting to know where the visitors of the site or application come from. This might help in understanding who these users are. It can also enable increasing the number of visitors by, for example:

understanding the search terms that people use and improve the findability of the site or application on major search engines;
encouraging sites to send more traffic by contacting them or sponsoring.
advertising on referring sites.
finding out where they are discussing the site or application on the internet and possibly joining the discussion.

referring websites

When a user visits the site or application, the browser sends a request for a page. If the user clicked a link to get to the site or application, the browser includes a referring URL of that page with the link to the request. The URL of a referring website can be used to track the visits back to the site that sent them and can be used to figure out what drives the traffic to the site or application. It is also interesting to relate this back to the conversion rate to get an idea who is sending converted users.

An interesting metric is the traffic volume per referring URL.

Risks and issues:

With some technologies though it is impossible to track where the users are coming from. For instance, Javascript within webpages and Flash plugins might not pass the necessary information.
URL-shortening services like tinyurl.com may not show the original referring URL.

inbound links from social networks

Today, an increasing number of visitors come from social network sites. These sources are very diverse, from social news aggregators, such as reddit.com or Digg.com, to blogs and Twitter.com.

Interesting metrics to watch are:

sudden changes in unique visitors,
enrollments from social networks, and
search results from social aggregators and microblogs

Risks and issues: Tracking social network referrals is difficult because all social network sites are different. It may be difficult to track down the original source of the message. The social network site might require a login to see the content at the referring URL or the link could be obfuscated in complex comment threads.

findability and search effectiveness

Visitors often use search functionality of a site or application to quickly find what they are looking for, instead of wading through hierarchies. If the site or application offers search functionality, it is interesting to know whether users find relevant content or data. If not, the findability can be optimised by labeling the site content better.

Interesting metrics to find out about these problems are:

the amount of searches that resulted in another search,
the amount of searches that resulted in a return to the home page or in leaving the site or application; and
the most popular terms causing abandonment and leading to a second search term.

content popularity

It can be helpful to know which content is popular.

Interesting metrics can be the content popularity by the number of visitors, the bounce rate, the appearance on social network sites, e.g. how many diggs did it get on Digg.com?

Risks and issues: Measuring content popularity is complex; popular stories might be one-hit wonders, the popularity can decay over time. It is important to know who is attracted by the content and what they do after consumption of the content. This should be related to the goals of the site or application.

loyalty

Loyalty is an indication of users that keep coming back. This can be tracked with browser cookies.

Interesting metrics are:

the ratio of new to returning visitors,
the average time between visits gives, and
the time since the last login.

enrollment

Beyond visiting the site or using the application, you can reach people in other ways, for example through RSS feeds and mailings. This can be measured by looking at the number of signups for a mailing, the RSS subscription rates and email churn (addresses that are no longer valid).

reflection

It is also possible to support or guide the user with some kind of reflection. This reflection might be used to provide the user with links related to the topics the user has just viewed. It could also show other users, online or not, who are interested in the same topics. During the utilisation of such a reflection tool it is easy to measure the activities of the user. The measurements which could be done might be at first, to give an idea what the intention of the user might be, what the user is looking for and perhaps why the user is visiting a page. Secondly, it is easy to find out what the user is doing, which references the user is interested in and where the user is going to.

1.1.3 Which tools can be used?

Google Analytics

Google provides a free service that tracks visitors of a website or application and presents the statistics in different views. Metrics can be provided for the different pages of the website or application, ranging from visitor quantity to technical info on the user computer systems. Google Analytics can track information about pages as well as events, for example the event when a user is added to the database. The latter is especially useful for rich internet applications (RIA). It can be used with Javascript as well as Flash applications.

Monitoring with Google Analytics provides a wide variety of general information about usage of the website or application. With the event system, Google Analytics can be used to track some of the functionalities of the application by passing the data related to the event to Google, such as who edited something from a wiki page. If Google is provided with this kind of data, it also provides statistics for that data, e.g. who edits the wiki the most.

Examples of what Google Analytics can track include:

visitors: number of visitors, unique visitors, new visits, average time on site
pages: number of pages per visit, page views, bounce rates, number of times each tracked page is visited and when, the percentage of direct traffic, referring traffic and traffic coming from search engines
navigation analysis:
- for each page, the percentage of entries on that page, exits from that page, next pages and previous pages navigated to from that page;
- entrance paths allow to analyse the path followed by the users
- entrance sources show from which sites users come
- entrance keywords show which keywords are used on search engines to find the site or application
user profiles and technical info: languages, geographical location, browsers, operating systems, network location, screen resolution, screen colors, Java and Flash support, connection speed
events: total number of events, visits with events, events per visit, top events, top event categories, top category actions, top category labels, hostnames where the events come from

Quantcast

Quantcast (http://www.quantcast.com/) provides usage statistics of websites and rates websites by ranks. The gathering of usage data is done by using a javascript plugin on the website, like with Google Analytics. With this technique, Quantcast collects thorough information on the users, e.g. the gender, the annual income range, the age group and education level of the visitor. This information comes from using inference through comparing and correlating between data from different web sites tracked by Quantcast. Quantcast can provide STELLAR with more detailed information on the visitor demographics of the website and mashups.

A very similar service is Alexa (http://www.alexa.com). Alexa tracks demographics through a browser toolbar installed by users.

Feedburner

Monitoring feed consumption becomes possible by redirecting feeds through feed management tools such as Feedburner (www.feedburner.com). Feedburner is a freely available web feed management tool that provides custom RSS feeds and management tools to bloggers, podcasters and web content publishers. Following quantitative statistics are provided:

number of subscribers
- in total
  - per country
  - per feed reader/aggregator
  - per period of time
number of views
- per item
number of clicks
- per item

Furthermore, Feedburner reports uncommon uses, i.e. republishing and mash-ups. This could be the starting point of a qualitative analysis of feed usage.

Attention Metadata Framework

Data describing the behaviour of how users use content and tools is called attention metadata. It can be tracked, merged and analysed to statistiscs on the usage behaviour and therefore enables conclusions on how people work with technologies and tools in a specific context.

The Contextualized Attention Metadata framework [Wolpers et al., 2007] tracks user attention in different tools they may use. Attention metadata can be captured on the server-side by relying on suitable server-side logging facilities. Attention metadata from these sources will be stored in an attention metadata store through web service interfaces.

With attention metadata, we can track the use of the application and all its functionalities and how the user interacts with them. What can be tracked highly depends on what kind of functionalities the application offers. The attention metadata format is general enough to cover about any kind of user action in an application and has to be integrated in the STELLAR applications. It would therefore be a good candidate to track the data that cannot be tracked with the other tools.

The table below summarises which tools can be used for monitoring the usage indicators described in Section 1.1.2.

metrics	Google Analytics	Attention Metadata tracking	Feedburner
conversion and abandonment	+	+/-	+/-
click-throughs	+	+	+
user-generated content	+/-	+	-
accounts	+/-	+	-
traffic sources	+	-	+
referring websites	+	-	+
inbound links from social networks	+	-	+
findability and search effectiveness	+/-	+	-
content popularity	+/-	+	+/-
community rankings and rewards	-	+	-
loyalty	+	+	+
enrollment	+/-	+	+

1.2 Monitoring online buzz

1.2.1 why monitor online buzz?

With today's social web and more and more people creating their own online content, tracking of users moves beyond visiting a website or application. An important part of knowing the users is knowing what other people are saying about the site or application online, for example in chat rooms, blogs, social network sites, news aggregators and user groups.

1.2.2 which metrics can measure online buzz?

In this section, metrics that can be used to measure online buzz are presented.

what are people saying about me:

Google Alerts (http://www.google.be/alerts) and Rollyo (http://rollyo.com/) can be used to search for new content on the internet. Community listening platforms monitor online buzz on community sites and social networks, e.g. Radian6 (http://www.radian6.com/), Techrigy (http://www.techrigy.com/), Scoutlabs (http://www.scoutlabs.com/), Sysomos (http://www.sysomos.com/), Keenkong (http://www.keenkong.com/), etc.

site reputation

High ranking on a search engine is important. Google PageRank is a measure of how relevant and significant Google thinks your site is. Technorati (http://technorati.com/) uses similar techniques and monitors what is going on in the blogosphere. StumbleUpon (http://www.stumbleupon.com/) rating can also be used as an indicator.

trends

Trends in searching can be monitored with the help of Google Trends (http://www.google.be/trends) and Yahoo! Buzz (http://buzz.yahoo.com/), which show the popularity of search terms. Google Insights (http://www.google.com/insights/search/) tracks popularity over time. These tools are good to understand the relative popularity. Mentions of the STELLAR project and the mash-ups over time can be useful to track.

social network activity

Many social networks have search functionality for their internal content that can be used to see whether a site or project is mentioned. Search results for the site or project, URL, persons in the project, relevant keywords across social sites like Digg, Summize and Twitter can be used as metrics. In the case of Twitter, the travel path of the messages (how many re-tweets have been done) can also be measured. It can also be interesting to see how many people bookmarked the site or application on online bookmarking sites like del.icio.us. Del.icio.us nowadays also presents the number of people who tweeted about the bookmarked URL.

site popularity and ranking

Site popularity can be measured with a visitor count, considered as the site's ability to reach people. Relevance-based search engine rankings are another good indication, e.g. PageRank of Google. Several services are available that estimate online site popularity, like compete.com and Alexa. A high number of visitors need to be listed on these sites. There are also other simple ways to count online popularity. It can be counted how many times people type the URL of the site or application into the search engine (navigational search). Since all these people know the URL, it can be a good indicator of popularity. This can be measured with tools like Google Trends or Google Insights.

who are the referring sites

Just looking at referring sites can be taken a step further by looking at the popularity and reputation of the referring sites to get an idea of their importance.

1.2.3 which tools can be used?

The tools highly depend on the metrics and on which platform we want to apply the metrics. To measure online buzz, we will need to make use of the tools and platforms where the buzz is happening, e.g social network sites (Digg and del.icio.us), and of tools that search the internet and online communities, e.g. Google Trends and Technorati. It is important to do these metrics over time and for all the different applications to allow comparison. Otherwise, it will be hard to know whether for example 50 tweets is a large or low number.

1.3 Monitoring availability

1.3.1 why monitor availability?

Next to knowing what people are doing with the site or application, it is also important to know whether they could do it and how quickly they could do it. If the tool is down, it cannot be used and this will reflect in the usage statistics.

1.3.2 which metrics can achieve this?

availability and performance

The availability of the web site is one of the most basic metrics. Availability is the percentage of tests that successfully retrieved a page. It can be calculated for cached and uncached pages. The performance is how long the user had to wait to interact with the site. Availability and performance can also be measured on parts of the site, e.g. important webpages. It is important to watch both metrics over time. Other typical metrics for availability and performance are: the uptime of the server, network availability, response time and the number of network errors.

impact performance on outcomes

Poor performance has a direct impact on different usage metrics like conversion rates and user productivity. Slow sites and applications encourage distraction. The relationship between conversion and performance can be measured by tracking the performance and by segmenting the conversion rates for visitors who experienced different levels of page latency. Another way to relate conversion and performance is to compare aggregate page latency and aggregate conversion rates. This can be done on a daily basis, showing the daily conversion rates compared to summaries of performance and availability.

1.3.3 which tools can be used?

A wide choice of free tools are available to monitor availability. On most of them, it is just a matter of signing up and filling out a form to specify which URL needs to be monitored. Some have more advanced features, like checking the text on the page, to detect a possible error message displayed on the page while the server can still be up and running. Examples are Mon.itor.us (http://mon.itor.us) and BinaryCanary (http://binarycanary.com/). Wikipedia has a comparison of web monitoring tools: [1].

To monitor the impact of performance on outcomes, the results of the tools used to track availability and performance have to be combined with the tools used for usage tracking.

1.4 Collecting qualitative data

1.4.1 why collect qualitative data?

It is hard to figure out what is going on in the head of a visitor just by looking at quantitative indicators. Therefore, qualitative data often yield interesting results. They can help to discover user motivations, goals and needs, improve predictions about future usage, and to identify usability issues. Other uses are to gain insight on how they discovered the site or application and what other sites and applications they are using.

1.4.2 which methods can achieve this?

surveys and polls

The most popular methods to collect qualitative data include surveys and polls. Polls consist of one question with several response options. Surveys, on the other hand, consist of more than one question, usually alternating between open and closed questions. Surveys can be conducted on-site (e.g. with a non-intrusive popup asking whether the visitor would be willing to fill out a survey), or may be disseminated via social networks, e-mail (e.g. the STELLAR mailing list), or offline. While the former technique has the advantage that one reaches the users in the context of the site/application, the latter allows one to not only reach out to visitors but also to non-users.

usability testing

Usability testing typically involves several users conducting common tasks with the target site/application whilst having their actions and utterances recorded. There exist several methods that can be used within STELLAR to evaluate the usability of applications. One approach consists of offline evaluations in a controlled lab setting. This method is very good in finding usability problems which cannot be derived directly from usage data. For example, a certain feature of a mash-up might not be used because users do not understand its iconography.

1.4.3 which tools can be used?

Numerous tools are available for conducting online questionnaires. Google Docs (http://docs.google.com) provides the possibility to create a form and share it with others. The following types of questions are available:

Open (Text and Paragraph Text)
Multiple choice
- one answer (List and Radio Buttons)
- one or more answers (Checkboxes)
Likert (Scale)

polldaddy (http://www.polldaddy.com) allows creating unlimited polls and on-site surveys which can be integrated as JavaScript into your own site/application. With the polls option one can create a poll with single/multiple choice answers and an optional open answers field. The surveys option allows for creating a sophisticated questionnaire corresponding to and in some aspects exceeding the offering of Google Docs. On the downside, polldaddy only allows for recording a hundred responses per month in the free version.

1.5 Applying indicators for monitoring the success of the STELLAR tools and the overall portal

This section elaborates on the applicability of the metrics to the STELLAR portal, repository, mashups, blog, podcast infrastructure and mobility tracker. Note that this is a first indication only of the potential relevance of metrics to the STELLAR tools and website. Detailed sets of indicators for the individual tools will be defined and implemented upon delivery of the tools (cf. Section 1.6).

metrics	Open Archive	Website	Mashups	Wikis	FlashMeeting	Blog
conversion and abandonment	+	+	(+)	+	+	+
click-throughs	+	+	(+)	+	-	+
user-generated content	+	-	(+)	+	-	+
accounts	+	-	(+)	+	+	+
traffic sources	+	+	(+)	+	+	+
referring websites	+	+	(+)	+	+	+
inbound links from social networks	+	+	(+)	+	+	+
findability and search effectiveness	+	+	(+)	+	-	+
loyalty	+	+	(+)	+	+	+
enrollment	+	+	(+)	+	+	+
what are people saying about me	+	+	(+)	+	+	+
site reputation	+	+	(+)	+	-	+
trends	+	+	(+)	-	-	-
social network activity	+	+	(+)	+	-	+
site popularity and ranking	+	+	(+)	+	-	+
who are the referring sites	+	+	(+)	+	+	+
availability and performance	+	+	(+)	+	+	+
impact performance on outcomes	+	+	(+)	+	+	+
surveys and polls	+	+	(+)	+	+	+
usability testing	+	+	(+)	+	+	+

1.5.1 Monitoring the success of the STELLAR web site

Most of the metrics proposed to measure usage, online buzz and availability are applicable to monitor the success of the STELLAR web site. Of particular interest are indidators that analyse:

conversion and abandonment;
traffic sources, including referring websites and inbound links from social networks;
findability and search effectiveness;
enrollment;
online buzz (what people are saying about me, site reputation, popularity and ranking, etc.);
as well as surveys and polls to collect qualitative data.

1.5.2 Monitoring the success of the open archive

The evaluation of the Stellar OA impact must be two fold, on the one hand it must address the use of the OA to upload bibliographical references and upload of original material, on the other hand it must address the use of the OA as scientific information resource for researchers, including visits, participation in online discussion of resources and download of material.

Most of the metrics proposed to measure usage, online buzz and availability are applicable to monitor the success of the SOA. Key indicators of the SOA include:

accounts : number of affiliated institutions and of registered users, active ones etc.
user-generated content :
- number of resources submitted: bibliographical reference alone , or plus link to the document, or plus associated document. And number of videos, tools, data sets.
- number of interactive contributions to blog and forum (cf. 1.5.5) specifically attached to resources
- number of institutions providing a feed
content popularity: popular resources, downloads…
click-throughs : clicks on the link to read the PDF-file of the paper.
reflection: resources related to the resource the user has just viewed
enrollment : monitoring of feeds consumption
enrollment of a set of OA facilitators : number of facilitators, their geographical spread, the number of users, documents, ... they bring to the OA

These indicators will be analysed against the use of the OA by the Stellar institutions and researchers, either home based or provided by other institutions. Qualitative data collection will be achieved by the interview of a selected set of Stellar researchers (senior and PhD), assessing their use of open access scientific information.

Indicators and interviews will be the material for a qualitative assessment of the Stellar OA impact on the consortium research integration.

1.5.3 Monitoring the success of the podcasting infrastructure

1.5.4 Monitoring the success of the STELLAR mashups

In this case, the appropriate metrics largely depend on the kind of functionality offered by the mash-ups. For each mashup, a set of indicators relevant for measuring the impact will be defined and implemented (cf. Section 1.6).

1.5.5 Monitoring the success of the STELLAR blog and wikis

Key indicators of the STELLAR blog and wikis include:

user-generated content contribution
- how many people are adding,
- how many edits they make and how large these edits are,
- whether they are creating content that other users of the site or application find useful,
- the read-to-post ratio, and
- the difference between average users and super users.
traffic sources, including referring websites and inbound links from social networks;
findability and search effectiveness;
enrollment; and
online buzz (what people are saying about me, site reputation, popularity and ranking, etc.);

1.5.6 Monitoring the success of the STELLAR mobility tracker

1.6 Monitoring Plan

In this deliverable, we have presented indicators and tools that can be used as a basis for measuring the potential impact of the STELLAR applications and the overall portal.

Detailed monitoring strategies that combine relevant indicators for monitoring the success of the individual applications will be specified upon delivery of initial versions of these applications. These monitoring strategies will define appropriate benchmarks and, where applicable, set sensible thresholds.

The table below outlines the timeline for delivering applications and their monitoring plan (cf. (1) and (2)). The indicators will be shared every month on the STELLAR WP6 mailing list (3). An analysis will be carried out and reported upon every six months (4). These reports will include an analysis of the indicators and their trends, as well as lessons learned and actions proposed and will also be shared on the WP6 mailing list.

A scientific conference paper on the results of the first experiences and a detailed scientific journal paper on the methodology for building and evaluating the STELLAR applications will be delivered in M20 and M36, respectively.

	Deliverable	Timeline
1	First set of STELLAR mash-ups, including individual monitoring strategies (D6.3)	M9, continuously updated
2	STELLAR Open Archive, including set of indicators and first analysis (D6.4)	M12
3	Monthly data report on indicators (shared via WP6 mailing list)	starting M10, monthly
4	Detailed analysis of the indicators and their trends, as well as lessons learned and actions proposed (shared via WP6 mailing list)	initial report M16, updated M22, M28, M34, M40
5	Scientific conference paper on first experiences with a mash-up of technology enhanced services (D6.5)	M20
6	Scientific journal paper on the methodology for building and evaluating a mash-up of technology enhanced learning services, including lessons learned (D6.6)	M36

1.7 References

Croll, A. and Power, S. (2009). Complete Web Monitoring. Oreilly & Associates Inc, June 2009, 662 pages.

Teltzrow, M. and Berendt, B. (2003). Web-Usage-Based Success Metrics for Multi-Channel Businesses. In Proceedings of the 5th ACM WebKDD 2003 Web Mining for E-Commerce Workshop "Webmining as a Premise to Effective and Intelligent Web Applications", New York: ACM Press, 2003, pp. 17-27.

Wolpers, M., Najjar, J., Verbert, K., Duval, E. (2007) Tracking Actual Usage: the Attention Metadata Approach. Educational Technology & Society 10 (3), 2007, pp. 106-121.