By Ruben Taelman https://orcid.org/0000-0001-5118-256X

Review Details

---

Reviewer has chosen not to be Anonymous

Summary of paper in a few sentences:

This article introduces a blockchain-based approach for writing and publishing scholarly articles.
With this approach, the authors allow democratic creation of citations,
and democratic publication of new versions,
where all previous versions of an article are maintained.
Furthermore, any third-party can comment on any version,
where comments are immutable and nonretractable.

Reasons to accept:

The authors prove that their blockchain-based model can be used to solve problems
in the scholarly publications domain related to
trust, tamperproofing, democratic collaboration, and citation counting.
As such, this work is valuable for the future of scholarly publication.

Next to that, the article is very well written and easy to understand.
Their implementation is available on GitHub,
and appears to be functional after trying this out myself locally.

Reasons to reject:

The cost analysis in section 5.3 concludes that Smart Papers are more cost-effective
than the services offered by big publishers.
While publishers have a cost of $3,500-$4,000 per article,
the authors claim that the cost with Smart Papers is ~2$.
While I agree that the Smart Papers approach is likely to be a lot cheaper,
the authors comparison is unfortunately a bit unfair.
This is because the calculation of the authors only includes the cost for Ethereum transactions,
and it is missing a lot of other aspects such as costs for Web hosting, typesetting, printing, ...
One could argue that some of these aspects may not be applicable for Smart Papers,
but they are likely to come up in some other form (e.g., Web hosting -> IPFS hosting),
so it is important to discuss these.

Another (minor) issue I have with this work,
is the assumption on citations being seen as a representation of trust.
While this may be true in most cases, there are concrete cases where this is not true.
For example, one may cite an article that one does _not_ agree with,
as a way to explain the mistakes of another work.
For this reason, reducing a citations graph to simple counters is insufficient.
A more elaborate solution could involve tracking citations of different types (agrees with, disagrees with, ...)
This is however a problem that is much bigger than this work,
so coming up with a full solution for this is definitely out-of-scope for this work.
However, I would expect at least a discussion of how different kinds of citations
could be tracked and analyzed in this model.

Nanopublication comments:

Further comments:

Minor comments:
* "RDFA" -> "RDFa"
* Figure 4 is missing axis labels. I assume the Y axis represents gas?
* https://rinkeby.aragon.org//soton.aragonid.eth/ returns TypeError: invalid media type

By Allan Third https://orcid.org/0000-0002-0386-1936

Review Details

---

Reviewer has chosen not to be Anonymous

Summary of paper in a few sentences:

This very clear paper describes the use of a set of Ethereum smart contracts for managing collaboration in paper authoring and publication, and for smart citation counting. It uses a DAO governance framework to handle the policies and, e.g., voting processes, required to ensure decision-making processes and outcomes. The approach is evaluated with respect to cost and performance.

Reasons to accept:

* It's an important area, where current issues have real consequences
* It's timely, and contributes to the ongoing rethinking of the scientific communication process
* The approach described is very clear and simple to follow, with flexibility clearly designed-in
* It addresses multiple perspectives in the scholarly publication process

Reasons to reject:

* Are speed and cost particularly relevant metrics for evaluation? On the timescales of paper authoring, it's nice to know that it won't take 15 minutes to publish, but it wouldn't (unless you're working *very* close to a deadline!) matter much if it did. Cost is more interesting, but, as acknowledged within the paper, the absolute results (the ones that matter to budgets) largely depend on the real-world exchange rate for ether, which is highly variable.
* It's not clear how to reproduce the results; the instructions for setting up and interacting with the software are easy to follow, but not the input to, and process for, reproducing the experiments described (cf. score for Data availability)

Nanopublication comments:

Further comments:

Thanks for this paper, I enjoyed reading it a lot. As mentioned above, I have some concerns about the evaluation. My suggestion would be that instead of speed and cost, evaluation could perhaps be in terms of completeness against a set of best practice guidelines (e.g., the Vancouver protocols on co-authorship) or against possible "attacks"/abuses. To some extent, the latter is discussed in the paper with comparison to the motivating examples, but a more rigorous approach would be a little more convincing (or an argument as to the rigor of the chosen examples). One question that comes to mind is whether new opportunities to game the publication/citation system are possible with Smart Papers, or whether there are unintended consequences (less academic attribution because of a per-citation cost?). This is not a major point, however: I think it's an interesting idea and approach, and I do believe the paper has defended its claims.

There seems to be an assumption, in discussion of citations, that citations are positive - that the content of a citation will support the claim of the citing paper. Citations are more sophisticated than this, and I'd be curious to see how semantic citations could be handled in Smart Papers. Perhaps a different count for each main possible reason for citing, with the total count computed from those by each Smart Paper's own contracts?

For related work, I don't know if you're aware of Cristina Iulia-Bucur in Amsterdam - she's working with Tobias Kuhn on decentralised scholarly authoring with nanopublications. Just FYI if you weren't.

Some more specific comments:
Abstract, line 17. "facilitate trustworthy decentralised computation of citations." would sound a bit better.
p1, l41: needs a comma after "sovereignty over"
p2, l2-4: lower case after (i), (ii), etc.
p2, l5: "an often-overlooked"
p2, l7: "reviews and comments on research"
p2, l18: Neither of these are actually questions. How about "How can releases and their attribution agreements be managed in a trusted way?" and "How can we avoid malicious/accidental...."
p2, l33: "available for analysis, and by"
p3, l24/25, and later: Why is Bob listed first? Alice, Bob, Charlie, Diane, etc., are named to give an alphabetical sequence, but mostly when mentioned as a pair, this paper says "Bob and Alice". I'm sure it's not the intent, but it reads as the male name being prioritised.
p5, l21: "agreed to the release of a particular"
p7, l25: citation meaning "trust" - it might mean other things, and it'd be worth discussing that.
p8, l45: It's not clear how the listed properties contribute to a difference between smart contracts and UML modelling. I'm not sure it matters - I think it's sufficient that all the smart contract features you wanted to model seem to be able to be represented.
p9, l27: "the Aragon framework"
p10, l42: "an Ethereum blockchain"/"an Ethereum-compatible blockchain", "the main Ethereum blockchain" - something like that, instead of "Ethereum's Blockchain"
p12, l25: "The Ethereum blockchain"
p13, l22: "the University of Southampton"
p13, l25: unnecessary space between "apps" and footnote number
p14, l1: Are new members granted ACLs (i.e., lists, which I'd interpret as "granted ACLs to manage") or granted permissions *in* an ACL?
p14, l10: "that is seamless"
p14, l28: "need a POSIX-compliant"
p16, l41: I agree with the overall point, but the costs with publishers do cover other services (e.g., typesetting) than are handled by Smart Papers

By Reza Samavi https://orcid.org/0000-0001-6768-0168

Review Details

---

Reviewer has chosen not to be Anonymous

Summary of paper in a few sentences:

This paper is a proposal for implementation and evaluation of Smart Papers. The Smart Papers implementation model and extends the existing blockchain-based model that the scientific community proposed to decentralise and make more transparent different aspects of scholarly communications. The novelty of the model is on complementing the current models for decentralised publishing and informetrics calculation, and analyse cost and performance implications.

Reasons to accept:

The paper is very well written. The idea of addressing potential collusion in the scientific collaborative environment is new and interesting. Having a model for verifying the so-called objective measures for quantifying scientific contributions can be helpful if the quantification itself is reliable (please see my further comments).

Reasons to reject:

NA

Nanopublication comments:

Further comments:

There are three areas of concerns that I would like the authors to address:

A) A philosophical view on the limitations of informetrics for scientific publications: I understand that justifying whether the metrics are justified or not is not the focus of this paper rather making the process transparent and supporting non-repudiation. However, I think it is important for the authors to acknowledge the limitations of the current metrics (#of citation etc.). For a thorough investigation of these limitations, please see: http://www.labri.fr/perso/gimbert/lawrence-current-biology-2007.pdf

B) the content: There are a number of questions that the authors should consider to answer:
Questions:

- Quorum: what if author contributions are not equally distributed? Can weights be applied to the votes cast by authors?
- Is there a minimum number of authors required for an effective quorum (3, more than 3)?
- This application seems more suitable for a mid to large number of authors, especially distributed authors who aren’t in constant communication, or “one-time” collaboration (trust not established yet between authors). Are there rules for tie-breaking votes?
- The paper mentions that this application can be applied to public and/or private blockchain networks. Would the lack of tokens (i.e. transaction fees) in a private network affect the way a paper is evolved over time (more updates/transactions since there is no fee)? How would a private blockchain allow for public/open reviews and comments (reviewers/commenters would need to be identified, brought in to the network).
-If the paper contains intellectual property or must follow a non-disclosure agreement, can the smart contracts delay publication until a specific date? If the paper is not meant to be public for a period of time, is the paper's meta-data publicly available on the Ethereum blockchain (since it’s a public network)? Can the research objects be linked to patents?

- It could be beneficial if the application can track acknowledgements, i.e. financial support. These organizations can then directly track research results they funded.
- Why was IPFS chosen as the distributed file store? There are alternatives such as Swarm that were designed to work with Ethereum. A comparative analysis would help.

C) Writing issues:
P4- Trusty URI should be referenced and explained in its first appearance
P6- Reference is needed: Interactions with documents are registered using the Web Annotations vocabulary.
P7- scientometric should be referenced and explained in its first appearance
P8- Section 4 the first two paragraphs are almost redundant
P9 - transition is a noun and cannot be used as a verb. you can change it to makes a transition...
P10- replace actors in Fig. 2 with the agent names
P12- IPFS should be expanded in its first appearance
P12- Add a reference after ...on-chain storage.
P13- ---we show this..., *this* is vague.
P14- ned -->need
P15- it's --> it is

This review was written together with Andrew Sutton.