Reviewer has chosen not to be AnonymousOverall Impression:
AcceptTechnical Quality of the paper:
Clear noveltyData availability:
Not all used and produced data are FAIR and openly available in established data repositories; authors need to fix thisLength of the manuscript:
The length of this manuscript is about right
Summary of paper in a few sentences:
This very clear paper describes the use of a set of Ethereum smart contracts for managing collaboration in paper authoring and publication, and for smart citation counting. It uses a DAO governance framework to handle the policies and, e.g., voting processes, required to ensure decision-making processes and outcomes. The approach is evaluated with respect to cost and performance.
Reasons to accept:
* It's an important area, where current issues have real consequences
* It's timely, and contributes to the ongoing rethinking of the scientific communication process
* The approach described is very clear and simple to follow, with flexibility clearly designed-in
* It addresses multiple perspectives in the scholarly publication process
Reasons to reject:
* Are speed and cost particularly relevant metrics for evaluation? On the timescales of paper authoring, it's nice to know that it won't take 15 minutes to publish, but it wouldn't (unless you're working *very* close to a deadline!) matter much if it did. Cost is more interesting, but, as acknowledged within the paper, the absolute results (the ones that matter to budgets) largely depend on the real-world exchange rate for ether, which is highly variable.
* It's not clear how to reproduce the results; the instructions for setting up and interacting with the software are easy to follow, but not the input to, and process for, reproducing the experiments described (cf. score for Data availability)
Thanks for this paper, I enjoyed reading it a lot. As mentioned above, I have some concerns about the evaluation. My suggestion would be that instead of speed and cost, evaluation could perhaps be in terms of completeness against a set of best practice guidelines (e.g., the Vancouver protocols on co-authorship) or against possible "attacks"/abuses. To some extent, the latter is discussed in the paper with comparison to the motivating examples, but a more rigorous approach would be a little more convincing (or an argument as to the rigor of the chosen examples). One question that comes to mind is whether new opportunities to game the publication/citation system are possible with Smart Papers, or whether there are unintended consequences (less academic attribution because of a per-citation cost?). This is not a major point, however: I think it's an interesting idea and approach, and I do believe the paper has defended its claims.
There seems to be an assumption, in discussion of citations, that citations are positive - that the content of a citation will support the claim of the citing paper. Citations are more sophisticated than this, and I'd be curious to see how semantic citations could be handled in Smart Papers. Perhaps a different count for each main possible reason for citing, with the total count computed from those by each Smart Paper's own contracts?
For related work, I don't know if you're aware of Cristina Iulia-Bucur in Amsterdam - she's working with Tobias Kuhn on decentralised scholarly authoring with nanopublications. Just FYI if you weren't.
Some more specific comments:
Abstract, line 17. "facilitate trustworthy decentralised computation of citations." would sound a bit better.
p1, l41: needs a comma after "sovereignty over"
p2, l2-4: lower case after (i), (ii), etc.
p2, l5: "an often-overlooked"
p2, l7: "reviews and comments on research"
p2, l18: Neither of these are actually questions. How about "How can releases and their attribution agreements be managed in a trusted way?" and "How can we avoid malicious/accidental...."
p2, l33: "available for analysis, and by"
p3, l24/25, and later: Why is Bob listed first? Alice, Bob, Charlie, Diane, etc., are named to give an alphabetical sequence, but mostly when mentioned as a pair, this paper says "Bob and Alice". I'm sure it's not the intent, but it reads as the male name being prioritised.
p5, l21: "agreed to the release of a particular"
p7, l25: citation meaning "trust" - it might mean other things, and it'd be worth discussing that.
p8, l45: It's not clear how the listed properties contribute to a difference between smart contracts and UML modelling. I'm not sure it matters - I think it's sufficient that all the smart contract features you wanted to model seem to be able to be represented.
p9, l27: "the Aragon framework"
p10, l42: "an Ethereum blockchain"/"an Ethereum-compatible blockchain", "the main Ethereum blockchain" - something like that, instead of "Ethereum's Blockchain"
p12, l25: "The Ethereum blockchain"
p13, l22: "the University of Southampton"
p13, l25: unnecessary space between "apps" and footnote number
p14, l1: Are new members granted ACLs (i.e., lists, which I'd interpret as "granted ACLs to manage") or granted permissions *in* an ACL?
p14, l10: "that is seamless"
p14, l28: "need a POSIX-compliant"
p16, l41: I agree with the overall point, but the costs with publishers do cover other services (e.g., typesetting) than are handled by Smart Papers
Meta-Review by Editor
Submitted by Tobias Kuhn on
The points raised by all of the reviewer, in particular:
- extending the calculation of costs (as suggested by Reviewer 1)
- critically assessing the correspondence between citations and trust (Reviewer 1)
- documenting reproducibility (Reviewer 2)
- commenting on the limitations of the work (Reviewer 3)
Ruben Verborgh (https://orcid.org/0000-0002-8596-222X)
ORCID as uint256
Submitted by Tobias Kuhn on
I found something that I think is a small error in your paper. In Figure 1, you model ORCID identifiers as "uint256". However, ORCID identifiers are not purely numeric, as the last character can be an "X" (see e.g. the ORCID for the last author).
Checked the camera-ready; all
Submitted by Ruben Verborgh on
Checked the camera-ready; all fine with me.
Re: ORCID as uint256
Submitted by Tobias Kuhn on
I learned that the last "digit" of ORCID identifiers is just a checksum that therefore doesn't need to be represented in this context. So, my issue raised above is not valid.