Reviewer has chosen not to be AnonymousOverall Impression:
UndecidedTechnical Quality of the paper:
Limited noveltyData availability:
All used and produced data are FAIR and openly available in established data repositoriesLength of the manuscript:
The length of this manuscript is about right
Summary of paper in a few sentences:
This paper introduces a methodology called "compositional and iterative semantic enhancement" (CISE) and argues that it is feasible and worthwhile to automatically enhance scholarly papers with semantic markup. This should be done in several stages, starting from low-level syntactic mappings to the highest levels of scientific arguments and the position of a paper in the whole discipline.
Reasons to accept:
- Interesting and controversial claims
- Built upon an impressive collection of previous work in this area
- Addresses highly relevant problem
Reasons to reject:
- The theoretical links are not convincing
- Lack of discussion on limitations, possible downsides, and assumptions made
I think this paper addresses a highly relevant problem, and the author is able to build upon an impressive collection of previous work in this area. The paper furthermore makes interesting and controversial claims, which is great for a position paper. However, I found the theoretical connection to the principle of compositionality and the Curry-Howard isomorphism not convincing. I think Section 4 is the most interesting part, but unfortunately only the lowest layers are described in detail (which, in my view, are the least interesting for the main point the paper is making). I suggest to put less focus on the theoretical links and more on the practical experiences and preliminary findings at the higher levels.
Specifically, I didn't understand why the Curry-Howard isomorphism is relevant to the points of this paper. Unlike the principle of compositionality, it doesn't seem to be adding anything to the argument (to the point where I could follow it).
I don't think, however, that the principle of compositionality can really carry all the argumentative weight that is put onto it here. After all, there is no final proof that this principle really holds for natural languages in their entirety. There are in fact many known cases where compositionality doesn't hold, such as for idiomatic expressions or sarcasm.
Furthermore, even if we assert the principle of compositionality as a fact, it would only tell us that we could in principle semantically parse papers in an automated fashion, but it would not allow us to conclude that this is feasible in any realistic setting. In fact, over and over again, all kinds of ambitious natural language processing has been proven to be very difficult and often infeasible with current technology.
This leads me to what I think is the second main shortcoming of the paper: There is basically no discussion on the limitations of the approach and on many of the implicit assumptions made. Specifically:
- At what accuracy do you think we can perform such a full parse of scientific papers? Automated approaches are never perfect (often with accuracy levels below 70% for non-trivial NLP tasks), and this seems to heavily affect the arguments made in the paper.
- When do you think we will be able to perform complete semantic analyses of scientific papers? In 5, 10, 50 years from now? What should we be doing until then?
- Sometimes authors write in ambiguous sentences (also for human readers), and deliberately or accidentally leave out important information. With your approach, we are stuck with incomplete information in these cases, whereas involving authors in the process could solve this. This shortcoming of the approach is not discussed.
So, in summary, I think the paper has clear merits as a position paper but needs to improve on the aspects explained above.
Below I list some more minor comments:
- The "iterative" part of the "compositional and iterative semantic enhancement" is not really explained. Is "iterative" referring to applying one layer after the other? To me, this wouldn't be an intuitive use of the term "iterative". I think "iterative" would imply to go through all the layers (or the individual layers) several times.
- I think "the" in the title should be omitted: "Automating Semantic Publishing" instead of "Automating the Semantic Publishing" (and same for the first sentence of the abstract)
- In general, I suggest to have a native speaker check the document with respect to grammar and style. At several places, I think that some of the used grammar constructs are awkward if not incorrect, but not being a native speaker either, I don't feel confident in my own judgment in what might be borderline cases or simply a matter of taste.
- The first paragraph of Section 1 contains many links but no citations (except for the last sentence) that would provide evidence for claims like "... have resulted in ... acceleration of the publishing workflow".
- "... which is very close to the recent proposal of the FAIR principles for scholarly data": Very close in what sense? What are the differences?
- "generally only a very low number of semantic statements (if none at all) is specified by the authors": Can you be more specific? What are the average/median/maximum values?
- "incentives such us prizes" > "such as"
- With respect to the paragraphs connecting to Genuine Semantic Publishing, I am not sure whether an average reader is given enough background to understand this discussion. Maybe the issue of "should we or shouldn't we require authors to make a significant extra effort?" could be stated more clearly and more explicitly.
- "The idea is that the aforementioned approaches can work correctly only if used with documents stored in a particular format ...": Do these *approaches* really only work with a particular format, or is it just the current *implementations* of these approaches? I think this is an important difference.
- Contrary to "... if the text to process is written in a particular language such as English, as happens for FRED ", I read on the linked website that "FRED is [...] able to parse natural language text in 48 different languages". This should be clarified.
- "It is worth mentioning that this approach is not merely theoretical, but rather it has been implemented ...": An important qualification here is that is has been *partially* implemented. None of these grammar correctly represent an entire natural language.
- I didn't understand why "hierarchical markup" is needed as an assumption in Section 3. If you assume that natural language sentences can be automatically parsed at great accuracy (as you seem to be assuming), then certainly you can automatically detect the hierarchical structure of documents as well.
- "there is no need of having a prior knowledge about the particular natural language used for writing the scholarly article": I don't understand what you mean by "prior knowledge" here. Somebody or something would need some knowledge (in fact deep knowledge) about the language to semantically parse the text at all the layers.
- Figure 1: I think I understand the meaning of the colors in this figure, but I failed to understand the meaning of the x and y axes. This should be explained better.
- Section 4: I would have liked to learn a bit more about the ontologies, tools, and existing studies on the layers 4 to 8.
- Section 4: I would expect some of the most difficult but also most interesting kind of knowledge to extract from a paper to be domain knowledge, i.e. what the authors have found out about the world (e.g. about living organisms in the case of biology). I don't see this aspect anywhere in the 8 presented layers. This seems to be another limitation that is not discussed.