Create a book
Review 2
From Benchmarking-Tel
Contents |
1 Peer Review Report 1 for STELLAR Deliverable D7.1
1.1 Basic Information
Deliverable Reference: D 7.1
Deliverable Title: Report on the state of the art in TEL
Main Author/Editor: Wild, Scott
Due date of the Deliverable: M3
Peer Reviewer (Institution, Person):
Date of Receipt of Deliverable: 19/06/2009
Date of Sending out the completed Template to the Main Author and WP7: 23/06/2009
1.2 Length of the deliverable
- Is the length of the deliverable justified?
The deliverable is a little long, but the length may be justified by the fact that this is the first STELLAR report on the state of the art.
- Please indicate parts that are superfluous, irrelevant, redundant, unspecific, and/or are written in flowery language?
1) In the section “Introduction” I would avoid the excursus concerning the Apple iPod, as well as the one concerning the UK Minister’s speech (pag. 10); these are not so relevant to the treatment of the section and distract the reader’s attention.
Editors comment: done. Piled these passages in a separate wiki section.
On the contrary, I would stress more the concept of fragmentation in the field of TEL, which is overall too weakly addressed in the del.
Editors comment: Especially the introduction and the conclusion have been extended to clarify this. Furthermore, all sections now have own conclusion subsections summarising the key findings on fragmentation.
2) Under section 4. I would avoid the part concerning “Personal Recommendations” (4.2) which is not particularly relevant to this del.
Editors comment: Done.
1.3 Content
- Does the deliverable meet the objectives of the deliverable described in the respective Work Package Work Description? If not, please indicate the parts where improvement is necessary.
Overall the deliverable reflects the description provided in the DOW.
Nonetheless, in the DOW we stated that the state of the art would include the “analysis of the fragmentation” in the field of TEL. As a matter of fact, the concept of fragmentation is shortly addressed in the section “Introduction”, but then fragmentation is not mentioned anymore; even the results of the analyses conducted (on EDMEDIA, on DBLP, on TELELEARN, on the Funding and the Networks) as they are, do not really provide evidence of the fragmentation or at least such fragmentation is not particularly stressed. I suggest that the concept is taken up in section “8. Conclusions”.
Editors comment: Has been addressed (see above): new intro, added conclusion, and individual conclusions per section.
Moreover, the DOW mentions also the benchmarks as possible baseline for the progress monitoring of STELLAR. Maybe the section “8. Conclusions” may also explain how the benchmarks will serve to this purpose.
Editors comment: Has been addressed.
- Is the content of the deliverable focused and presented in a precise and to-the-point manner? If not, please indicate the parts where improvement is necessary.
see the following answer
- Does the deliverable require substantial revision or rewriting? If yes, please give concrete suggestions how to improve the deliverable.
No, I wouldn’t say so. Nonetheless, I would recommend the followings:
1) the structure of the Executive Summary, which – as I guess – is intended to reflect the one of the del. itself, is not so clear at a first sight. I would suggest to say explicitly something like: “ This del. will be structured around the following sections….”.
Editors comment: Done.
2) at a structural level it is evident that there is no balance among the sections. In particular, while sections 3, 4 and 5 are exhaustive and sound complete, section 6 (Funding) is far more limited and would benefit from an in-depth analysis. To this end, an idea could be not to limit the analysis to the present, but to provide an historical view on the funded projects (for example in the period 2000-2008, as done for the conferences);
Editors comment: Has been balanced: 5 has been condensed a little, 6 has been extended. However, a historical perspective has not been added so far. Though we will start to do that in the wiki (basing on some slides I have used in a research funding training in the western balkan countries last year).
3) in the section “Introduction” there is a part concerning the horizon scan (pag. 10-12), which is quite interesting. Nonetheless, there is no evidence of a direct link between the results of these scan with STELLAR. I suggest that in the end (pag. 12) you stress more what we – as STELLARs - are going to do with the challenges mentioned in the previous pages. Do they serve as an input to STELLAR? How?;
Editors comment: Has been merged with the Prolearn roadmap section into an own chapter. A section conclusion hopefully clarifies its value for STELLAR now.
4) At the beginning of section “5. Additional data from DBLP and Telearn” I would suggest to explain better the differences between the two studies (this one and the previous one concerning ED-MEDIA), by making it clearer why it was necessary to carry out both the studies. This would help also avoiding the impression of a simple juxtaposition of the two studies;
Editors comment: The complete Section 5 has been re-worked to address this.
5) since some of the Figures are not so clear, I would suggest to extend the descriptive text aimed at illustrating the data represented in the Figures themselves. For example Fig. 4 (pag. 45) is not clear, so perhaps a short description in words of the data would help the reader;
Editors comment: especially in this section, more narrative explaining the graphics has been added.
6) at pag. 50 you mention that, in order to compensate the difference in number between DBLP and Telearn, you have “included the abstracts of the papers”. What does this mean? Do you mean that you have made the Worlde Clouds by inputting both the tiles and the abstracts together? If it is so, I am not sure this really compensate the existing difference… Anyway, the concept needs to be clarified;
Editors comment: This sentence is misleading and has been deleted. A comment has been added further down in the section -- as some analysis artefacts can be related to including the abstracts in the Telearn analysis.
7) section “6. Benchmarks: Funding” is limited and does not contain any final reflections. I would certainly suggest to introduce some final considerations (in the light of fragmentation!) and – if possible – I would also enlarge this part by providing an historical perspective (i.e. add an analysis of the funded projects of the period 2000-2008 and see whether topics have changed). Moreover, at pag. 55 there is a sentence which is not very clear: “From an engineering educational point of view, advances are especially expected…..”. Is this an excerpt from the Work Programme?
Editors comment: see above: conclusion has been added, historical perspective leaves room for future work.
8) section “7. Benchmarks: Networks” contains an interesting analysis of the previous networks. Nonetheless the section “7.1 The Prolearn Roadmap” is not explained enough, nor it is clear why it is mentioned here and what will be the use of such a Roadmap within STELLAR.
Editors comment: Roadmap has been merged with the second horizon scanning report into an own chapter and a conclusion has been added to clarify its value for STELLAR.
- Please indicate the grammatical errors and/or typographical errors and/or incomprehensive sentences.
pag, 16: “This paper present a bibliometric and scientometric analysis…” I suggest to avoid the world “paper”
Editors comment: done.
pag. 40 (second item of the list): “There is a large and dynamic ….. at its core and….”
Editors comment: done.
1.4 Major strengths
The deliverable is overall well written and the general impact on the reader is good.
In particular the analysis of ED-MEDIA is very interesting and exhaustive. Even the analysis of DBLP and the tentative comparison with Telearn are useful and rich.
1.5 Major weaknesses
The major weakness is the lack of an overall discussion of the results obtained, which I guess will be tackled in section “8. Conclusions”. Here it will be necessary to explain also the relationships among the analyses conducted, so to provide a unifying interpretation of the presented results (that is missing at the moment!).
Editors comment: has been added.
In view of upcoming final reflections, please consider also that not only the two main analyses (ED-MEDIA and DBLP+Telearn) at the moment are not linked to each other (you should explain the differences between the two and the reason why you chose to include them both in this del.), but none of the two really tackles the issue of “fragmentation”.
On the contrary, sometimes it seems as if these analyses tend to demonstrate the existence of an internal coherence (see for example pag. 37 “….This means that the majority of ED-MEDIA authors are related and that the authors can be considered a coherent community….”). This is not a demonstration of fragmentation… So I think the final section of Conclusions should take up the concept of fragmentation and explain where and how the presented results demonstrate that the TEL field is fragmented.
Editors comment: This certainly has been correct, but it says nothing about the strength of the connection. Moreover, as could be seen from the analysis results presented in that section, the network actually is not very strongly connected. This section has been reworked to provide clarification on this. Similarily all other sections have been revised to provide clear insights into fragmentation.
Besides, another weakness of the deliverable is that sections 6. and 7. are far poorer than the previous sections (especially section 6. lacks some conclusive reflections; the reader does not understand the meaning of reading a list of project descriptions).
Editors comment: section 6 has been fitted with a conclusion reflecting on the implications. Similarly section 7 has been overhauled to now only focus on the networks, not the forecasting outcomes. Forecasting has been merged with the findings of the Horizon Report into an own section (and a conclusion has been added there).
6. Review Summary
The overall rating of the deliverable is 4 - good
Comments:
The current version of the deliverable is 2: applicable, but requires minor revisions
Comments:
see above for minor revisions
Is it necessary for the revised deliverables to be reviewed again before submitting it to the EC (1:Yes, 2: No)? [ No ]
Other remarks: