By Stephen Pettifer https://orcid.org/0000-0002-1809-5621

Review Details

---

Reviewer has chosen not to be Anonymous

Summary of paper in a few sentences:

This article proposes a framework or approach to automatic annotation of scholarly articles that begins with an author writing regular prose, and results through an incremental process in a fully semantic marked-up document.

Reasons to accept:

There's a really important story here trying to get out; if the story can be freed from what I think are unnecessary formalisms then this could be an important piece of work.

Reasons to reject:

I should make it clear before I begin that I’ve worked with the author before, and have co-authored a paper with him. I have no connections beyond that.

I should also confess at the start that I found this paper difficult to read and understand. Much of that is because it requires more knowledge of formalisms than I have; part of it I suspect is because of issues with the article itself. It’s difficult for me to judge how much is the former and how much is the latter, so the rest of this review should be read with that in mind. My review doesn't follow the standard structure requested for Data Science reviews because I'd end up writing 'N/A' or 'don't know' against most of the headings. Instead I'm returning a narrative verdict in the hope that this is helpful to the author.

The general gist of the article is that it proposes a framework or approach to automatic annotation of scholarly articles that begins with an author writing regular prose, and results through an incremental process in a fully semantic marked-up document.

This is a laudable aim — and I agree wholeheartedly that providing authors with tools to simplify the process of adding machine-readable semantics to articles is crucial if we are to move in that direction.

This is also an exceptionally ambitious aim, and one which seems a long way off being reached by existing technology. My difficulty with this article — probably a misunderstanding — is that it seems to suggest that this is already achievable (although the suggestion in surrounded by, in my opinion, unnecessarily complex and slightly flakey discussion of Curry-Howard isomorphism) and that the rest of the aims can be achieved by layering on further semantic transformations.

I think if this article was saying “here is a collection of ontologies and frameworks that form a kind of hierarchy that could in an ideal world be used to describe the perfect semantic publication; and maybe you can even reason about the relationships between the layers” then I’d be a lot more comfortable with it. As it stands, either I’m not following the formal glue that’s used to hold the different steps together, or perhaps the formal glue isn’t quite as cohesive as it might be. Or some combination of these two things.

Overall I think there’s an important message in here, but it’s currently squirming to get out a bit.

This may sound somewhat negative, but I really do think the author has something important to say here. I’d suggest simplifying the formalisms (I really don’t think they add clarity), and including concrete examples. I think then it could make an excellent position paper that unifies several already significant pieces of work.

A few specific comments:

"Automating the Semantic Publishing" -> should this be “Automating Semantic Publishing” ? I’m not sure what the definite article is doing here.
I think the sub title is too dense for a general CS audience (or even for a ‘data science’ audience). It may make sense to someone that’s into formal methods. Maybe.

Gutemburg -> Gutenburgh

Throughout the article needs a fair amount of attention to the written English; too many minor infelicities to list here. Because the article is technically dense, these risk tripping the reader rather too often.

Nanopublication comments:

Further comments:

By Karin Verspoor https://orcid.org/0000-0002-8661-1544

Review Details

---

Reviewer has chosen not to be Anonymous

Summary of paper in a few sentences:

This opinion paper introduces the reader to a proposal for formally modelling the structure of scholarly papers, and supporting combinations of lower-level structures to imply higher-level structures. The author argues that this approach will allow automatic enrichment of documents with semantic information, based on a sort of projection from syntactic patterns to the semantics implied implied by those patterns.

Reasons to accept:

The notion of, as I have understood it, reasoning over combinations of annotations to infer higher-order semantic types is interesting. While the discussion here focuses mostly on formal characterisation of the relationships/patterns, I can imagine that this could also be relevant to empirically derived models, e.g. statistical models derived from data mining/machine learning over semantic features. The author's comment that a majority of authors follow consistent patterns does not surprise me and I agree could be leveraged (whether rule-based or statistical) for compliance checks or feedback purposes. It makes sense to think of the structure of documents as compositional and the approach outlined here enables that.

There is pragmatic value in the approach; to the extent that semantic characteristics can be inferred from a document itself rather than demanding additional manual effort, the more likely those characteristics will be captured and represented.

Reasons to reject:

There is relevant literature that is not considered here, including the role of writing assistants for semantic enrichment [1] as well as the literature on web annotation and provenance from various W3C communities [2]-[3] and indeed the linguistic annotation communities (which often use stand-off annotation) [9]. Some work on compositionality in such models also exists [4]. The authors further mention "several tools that have been developed for addressing the automatic annotation..." without providing any details or references. I note that there is substantial work on issues such as document zoning [5] and rhetorical structure [6] and even citation analysis [7-8]. Finally the more recent and less well-developed writing of Robert Allen may be relevant [11]. While perhaps these don't (all) address the author's desire for the analysis to be independent of the language of the document content, and may not be amenable to algorithmic processing, they are certainly relevant to the broader context and argumentation -- at least for understanding how the proposals here differ.

Note that the citation provided for categorial grammar is probably not the canonical/most relevant citation, see [10] inter alia.

In addition, I found some of the aspects of the theory and algorithms described in section 4.1 not entirely clear. The interpretation of some of the core structural patterns as named isn't intuitive, and the notion of "coherence" that is relevant to pattern assignment is not entirely clear, especially given that every possible combination of t,s,T exists.

Regarding the point related to syntactical visualisation -- is this about generalisation of the patterns from a few examples? I would see this less as "compliance" (implying guidelines exist that are expected to be followed) and more as conventions that exist.

I also wonder about the use of the term "validity" in the Conclusions. In what way could this approach (as a whole) be validated as opposed to shown to apply in particular contexts?

[1] DOI: 10.1186/1471-2105-11-103
[2] https://www.w3.org/TR/annotation-model/
[3] https://www.w3.org/TR/prov-dm/
[4] DOI: 10.1186/2041-1480-4-38
[5] lots of citations -- a cursory search reveals:
Hollingsworth, Bill, Ian Lewin, and Dan Tidhar. "Retrieving hierarchical text structure from typeset scientific articles–a prerequisite for e-science text mining." Proc. of the 4th UK E-Science All Hands Meeting. 2005.
Taghva, Kazem, Allen Condit, and Julie Borsack. "Autotag: A tool for creating structured document collections from printed materials." Electronic Publishing, Artistic Imaging, and Digital Typography. Springer Berlin Heidelberg, 1998. 420-431.
Kim, Jongwoo, Daniel X. Le, and George R. Thoma. "Automated labeling algorithms for biomedical document images." Proc. 7th World Multiconference on Systemics, Cybernetics and Informatics. Vol. 5. 2003.
[6] https://doi.org/10.1093/bioinformatics/bts071
[7] http://clair.si.umich.edu/~radev/papers/192.pdf
[8] Nakov, P. I., Schwartz, A. S., and Hearst, M. A. 2004. Citances: Citation sentences for semantic analysis of bioscience text. In In Proceedings of the SIGIR’04 workshop on Search and Discovery in Bioinformatics.
[9] again, lots of citations. representative:
Bird, Steven, and Mark Liberman. "A formal framework for linguistic annotation." Speech communication 33.1 (2001): 23-60.
https://doi.org/10.1093/bioinformatics/bts071
Chiarcos, Christian, Sebastian Nordhoff, and Sebastian Hellmann, eds. Linked data in linguistics: Representing and connecting language data and language metadata. Springer Science & Business Media, 2012.
BioC: https://doi.org/10.1093/database/bat064
[10] Steedman, Mark. The syntactic process. Vol. 24. Cambridge: MIT press, 2000.
[11] e.g. DOI: 10.1007/978-3-319-49304-6_26

Nanopublication comments:

Further comments:

Note there are a few typos and language usage issues. I would suggest "the Semantic Publishing" should be simply "Semantic Publishing" in the title and elsewhere.
Others (not exhaustive)
Montegue -> Montague
dependent -> dependant
"for automatically generate" -> "to automatically generate"
What is "a rhetoric/rhetorics" (i.e. as a noun)?
What is an "easily complex" operation?
Reconsider use of "actually" in Abstract (can be safely removed)

By Tobias Kuhn https://orcid.org/0000-0002-1267-0234

Review Details

---

Reviewer has chosen not to be Anonymous

Summary of paper in a few sentences:

This paper introduces a methodology called "compositional and iterative semantic enhancement" (CISE) and argues that it is feasible and worthwhile to automatically enhance scholarly papers with semantic markup. This should be done in several stages, starting from low-level syntactic mappings to the highest levels of scientific arguments and the position of a paper in the whole discipline.

Reasons to accept:

- Interesting and controversial claims
- Built upon an impressive collection of previous work in this area
- Addresses highly relevant problem

Reasons to reject:

- The theoretical links are not convincing
- Lack of discussion on limitations, possible downsides, and assumptions made

Nanopublication comments:

Further comments:

I think this paper addresses a highly relevant problem, and the author is able to build upon an impressive collection of previous work in this area. The paper furthermore makes interesting and controversial claims, which is great for a position paper. However, I found the theoretical connection to the principle of compositionality and the Curry-Howard isomorphism not convincing. I think Section 4 is the most interesting part, but unfortunately only the lowest layers are described in detail (which, in my view, are the least interesting for the main point the paper is making). I suggest to put less focus on the theoretical links and more on the practical experiences and preliminary findings at the higher levels.

Specifically, I didn't understand why the Curry-Howard isomorphism is relevant to the points of this paper. Unlike the principle of compositionality, it doesn't seem to be adding anything to the argument (to the point where I could follow it).

I don't think, however, that the principle of compositionality can really carry all the argumentative weight that is put onto it here. After all, there is no final proof that this principle really holds for natural languages in their entirety. There are in fact many known cases where compositionality doesn't hold, such as for idiomatic expressions or sarcasm.

Furthermore, even if we assert the principle of compositionality as a fact, it would only tell us that we could in principle semantically parse papers in an automated fashion, but it would not allow us to conclude that this is feasible in any realistic setting. In fact, over and over again, all kinds of ambitious natural language processing has been proven to be very difficult and often infeasible with current technology.

This leads me to what I think is the second main shortcoming of the paper: There is basically no discussion on the limitations of the approach and on many of the implicit assumptions made. Specifically:

- At what accuracy do you think we can perform such a full parse of scientific papers? Automated approaches are never perfect (often with accuracy levels below 70% for non-trivial NLP tasks), and this seems to heavily affect the arguments made in the paper.

- When do you think we will be able to perform complete semantic analyses of scientific papers? In 5, 10, 50 years from now? What should we be doing until then?

- Sometimes authors write in ambiguous sentences (also for human readers), and deliberately or accidentally leave out important information. With your approach, we are stuck with incomplete information in these cases, whereas involving authors in the process could solve this. This shortcoming of the approach is not discussed.

So, in summary, I think the paper has clear merits as a position paper but needs to improve on the aspects explained above.

Below I list some more minor comments:

- The "iterative" part of the "compositional and iterative semantic enhancement" is not really explained. Is "iterative" referring to applying one layer after the other? To me, this wouldn't be an intuitive use of the term "iterative". I think "iterative" would imply to go through all the layers (or the individual layers) several times.

- I think "the" in the title should be omitted: "Automating Semantic Publishing" instead of "Automating the Semantic Publishing" (and same for the first sentence of the abstract)

- In general, I suggest to have a native speaker check the document with respect to grammar and style. At several places, I think that some of the used grammar constructs are awkward if not incorrect, but not being a native speaker either, I don't feel confident in my own judgment in what might be borderline cases or simply a matter of taste.

- The first paragraph of Section 1 contains many links but no citations (except for the last sentence) that would provide evidence for claims like "... have resulted in ... acceleration of the publishing workflow".

- "... which is very close to the recent proposal of the FAIR principles for scholarly data": Very close in what sense? What are the differences?

- "generally only a very low number of semantic statements (if none at all) is specified by the authors": Can you be more specific? What are the average/median/maximum values?

- "incentives such us prizes" > "such as"

- With respect to the paragraphs connecting to Genuine Semantic Publishing, I am not sure whether an average reader is given enough background to understand this discussion. Maybe the issue of "should we or shouldn't we require authors to make a significant extra effort?" could be stated more clearly and more explicitly.

- "The idea is that the aforementioned approaches can work correctly only if used with documents stored in a particular format ...": Do these *approaches* really only work with a particular format, or is it just the current *implementations* of these approaches? I think this is an important difference.

- Contrary to "... if the text to process is written in a particular language such as English, as happens for FRED [9]", I read on the linked website that "FRED is [...] able to parse natural language text in 48 different languages". This should be clarified.

- "It is worth mentioning that this approach is not merely theoretical, but rather it has been implemented ...": An important qualification here is that is has been *partially* implemented. None of these grammar correctly represent an entire natural language.

- I didn't understand why "hierarchical markup" is needed as an assumption in Section 3. If you assume that natural language sentences can be automatically parsed at great accuracy (as you seem to be assuming), then certainly you can automatically detect the hierarchical structure of documents as well.

- "there is no need of having a prior knowledge about the particular natural language used for writing the scholarly article": I don't understand what you mean by "prior knowledge" here. Somebody or something would need some knowledge (in fact deep knowledge) about the language to semantically parse the text at all the layers.

- Figure 1: I think I understand the meaning of the colors in this figure, but I failed to understand the meaning of the x and y axes. This should be explained better.

- Section 4: I would have liked to learn a bit more about the ontologies, tools, and existing studies on the layers 4 to 8.

- Section 4: I would expect some of the most difficult but also most interesting kind of knowledge to extract from a paper to be domain knowledge, i.e. what the authors have found out about the world (e.g. about living organisms in the case of biology). I don't see this aspect anywhere in the 8 presented layers. This seems to be another limitation that is not discussed.