Reviewer has chosen to be Anonymous

Summary of paper in a few sentences:

This paper describes an Expert-Driven Automatic Methodology (EDAM) to assist researchers in performing systematic reviews. It consists of 7 components where ontology learning acts as the key step. EDAM is evaluated on Software Architecture against six senior researchers. The result shows that the performance of EDAM is comparable to the senior researchers.

Reasons to accept:

This paper provides a good option to speed up systematic reviews.
EDAM which incorporates ontology learning is a bit novel

Reasons to reject:

I have several concerns regarding the evaluation:
(1) for the evaluation of the primary study classification, the dataset (25 papers) is to small to draw a conclusion. As the human agreement is usually less than 50% in table 2, the authors should consider a more effective way for evaluation, e.g., providing the ontology to the researchers and asking them to pick up to 5 categories.

(2) the authors should compare their method with other possible approaches, e.g., LDA based approaches, which can cluster studies to multiple groups based on their topical distribution, then each cluster can also be automatically assigned with some topical terms.

Nanopublication comments:

Further comments:

By Anita Dewaard https://orcid.org/0000-0002-9034-4119

Review Details

---

Reviewer has chosen not to be Anonymous

Summary of paper in a few sentences:

The paper says it proposes a method for developing automated Systematic Reviews (SRs): summaries of work in a specific domain, addressing a particular question or research areas. The problem of the difficulty to scale SRs and a summary of background and related work in this area are all very well-written and clearly motivated. Unfortunately, after this excellent beginning, the paper takes an odd turn: first of all, instead of proposing how to construct a SR, it focuses only on identifying which papers in the literature concern a specific topic, given as a keyword. Though source selection is of course an important step in developing a SR, there is no reason to assume (nor is evidence given) that this is the most difficult or even most important step in writing a SR: surely, most of the work (and skill) that goes into writing a good and useful SR goes into evaluating these papers, comparing and contrasting them, and summarizing the knowledge gleaned from this collection of content? None of these aspects are addressed in the paper and to me, it is disingenuous to present the tool as an automated SR creation tool when really, all it does is find papers related to a specific term. It would be much more accurate to treat this as an information retrieval or indexing task, than as a writing-support task.
Secondly, the use case taken in the paper, Software Engineering, is very unconvincing. The keywords used, such as Service-based Architectures and Software Design are all very vague and seem to be overlapping. Though I am not a software engineering, I think it is highly unlikely that the outputs presented here would support the author of an SR to detect trends in this domain. There is no evidence provided that suggests that the trends shown in the graphs of various terms are not simply due to chance or fads, and I do not see how any insight can be gleaned from these seemingly random correlations.
Lastly, Table 2 is intended to show that the tool, EDAM, delivers better than human inter-annotator agreement with a set of 6 annotators. To me, the main point of Table 2 is that in fact the human inter-annotator agreement in appalling: Possibly, this is again due to the fact that the terms assigned are so vague, and probably have a great amount of overlap in meaning. The majority of terms have very little agreement between most of the annotators. So: given that the current mode of identifying papers is largely the authors choice, what does this tell us? Are these the right terms to classify papers under? Why is the agreement so low? Would a different taxonomy, or a different domain, have delivered a different outcome?
In summary, my experience in reading the paper is that I am excited by the premise and careful writing in the introduction, but disappointed by what EDAM and the tool actually deliver to what I was quite excited about learning about, an automated SR authoring tool.
I suggest that the authors decide either expand the introduction and write an overview of current work related to this topic, or write up the methods and results of this paper as a small conference paper. If the latter, I would like to see EDAM applied to a different domain and taxonomy, to see whether the bad inter-annotator agreement is structural or incidental. If the prior, I would love to see EDAM applied for writing an automated systematic review of automated systematic reviews.

Reasons to accept:

- Interesting topic
- Good introduction & Background of problem, related work

Reasons to reject:

- Methods & results not living up to promise of introduction, needs rescoping/clarification
- Results unclear: why this taxonomy? How does term overlap affect outcomes? Why not try a different topic?
- Not a homogenous quality/level: propose to split into one of two papers, either small paper on Methods/Results, with a check for other topic, or an overview of related work in this field: if latter, exciting to use tool described to write that review.

Nanopublication comments:

Further comments:

By Philipp Cimiano https://orcid.org/0000-0002-4771-441X

Review Details

---

Reviewer has chosen not to be Anonymous

Summary of paper in a few sentences:

This paper presents a new methodology that the authors call EDAM (Expert-Driven Automatic Methodology) that supports the process of generating systematic reviews. The method is applied to the domain of software engineering in the paper, but is clearly applicable beyond this domain.
The method attempts to automatize some steps of the systematic reviews process for two reasons: to reduce the burden of manual work involved in selecting relevant primary papers, but ii) to make the selection process more objective. The paper proposes an ontology-based approach that supports the steps of i) selection of relevant papers, ii) keywording, and iii) creation of a classification schema. For this purpose, the authors build on an existing ontology learning method that has been specifically designed for learning topic hierarchies in the scientific domain. The method proposed in the paper is briefly evaluated on a case study in software engineering. Further, the accuracy of the classification approach to classify the primary literature is evaluated by comparing it to a number of annotators.

Reasons to accept:

The paper tackles an important problem, that is the problem of supporting the process of generating systematic reviews. It proposes to automatize some parts of the task, in particular the one of selecting relevant papers, filtering them down and classifying the papers into topics.
The method is based on an ontology learning approach that generates a hierarchy of relevant concepts starting from a seed term. Experts can then interact with the ontology that is displayed in terms of a tree diagram in Excel. The fact that experts can directly refine the ontology is a very positive aspect of the method.
The method is evaluated on a use case in software engineering and limitations are discussed. The methodology for evaluation is sound, empirically evaluating the classification step of the approach by comparing automatically generated annotations to those of a number of annotates that are partially experts on the domain. The paper describes some typical case studies that the methodology can support as well as limitations (Section 5.2).
Overall, the proposed methodology is novel and sound.

Reasons to reject:

No major reason to reject the paper, see below some comments to improve the paper.

Nanopublication comments:

Further comments:

There is no evaluation of the usability of the tool for interacting with the ontology. The paper mentions that the experts were able to modify the ontology, but does not say anything about how usable the Excel-based editing process was from the point of view of experts.

The paper does not explain the ontology learning approach used. It references an existing method, Klink-2, that was developed by the same authors. To make the paper self-contained it would be good to have a concise description of how this ontology learning algorithm works.

It is not clear what language can be used to specify filter criteria. It is clear from the paper that the filtering is based on matching concepts in the ontology over the set of papers. However, it is not clear if one can use operators such as NOT, AND, OR etc. It is not clear if there is a formal language in which a set of matching criteria can be defined.

The authors mention that any method can be used to map papers into concepts / topics, even a learned classifier. However, it seems that in this paper they rely on a straightforward approach that annotates a paper with a concept from the ontology if the concept or any of its subconcepts appears in the ontology. This should be made clear in the evaluation section where the annotations are compared to the set of annotations by experts.

In this same section, the authors compute agreement in terms of overlap, but ignore that agreement can also be by chance. Many measures for quantifying inter-annotator agreement such as Kappa take this into account. I wonder why the authors have not considered to use such methods to quantify agreement.

Minor corrections:

Page 3: aimed to at improving -> aimed at

Page 4: by both [21] and [14] -> bad style, do not use references as words

Page 8: a Semantic Web ontology of 58 topics*s*

Page 18: limitations based on the categorization given in [45] -> do not use references as words

Page 19: requires human expertize -> expertise

Page 20: performance of entity extraction and linking tooks -> tools

Two comments on the title of the paper: as the methodology is not specific for software engineering, I wonder why the authors do not choose simply the title: "Reducing the effort for systematic reviews" without the domain qualification. Instead, the authors could think about adding some qualifiers that specify which parts of the SR process their method supports and add them to the title. When I first read the title, I did not have a clear idea what aspects of an SR could be supported. This is clarified only later in the introduction, but could be made more specific in the title of the paper.

A second point: I missed the qualifier "supporting systematic reviews" in the name of the methodology. Expert-Driven Automatic Methodology does not say anything about the fact that the methodology is supposed to support SR. This is odd.