By Victor de Boer https://orcid.org/0000-0001-9079-039X

Review Details

---

Reviewer has chosen not to be Anonymous

Summary of paper in a few sentences:

This paper presents the HOBBIT benchmarking platform. HOBBIT is a technical infrastructure that allows various Linked Data benchmarking experiments. The design is based on a combination of elicited user requirements and requirements derived from the FAIR principles. The infrastructure is based on various independent components implemented as (Docker) containers. The platform features containers for coordinating and monitoring experiments and storing data, as well as basic UI components. Metadata about experiments on the platform are made available through a SPARQL endpoint.

The paper describes the usage of the platform in a number of public challenges. The platform was evaluated through two experiments that show the scalability of the platform and the deployability on singular machines as well as clusters.

Reasons to accept:

- Well written paper, easy to follow
- The paper clearly is more of a 'resource paper' rather than a research paper and the main contribution is the description of the platform and its design decisions. The benchmarking platform itself is very valuable to the larger (Linked) Data science community and as such the paper would be very relevant for the journal..

Reasons to reject:

- While it is great to see the use of the use of the FAIR principles, it is not entirely clear how FAIR the resulting data is. For example, the provenance metadata is now quite limited, with only time stamps and hardware details. I would argue that this might not be 'detailed' provenance as stated in R 1.2. The paper would be improved by a discussion on the extent to which the FAIR principles are adhered to and how reusable the data actually is.

- The method of Requirements analysis (an important part of the design proces) is now only briefly addressed in Section 3, with a pointer to an unpublished project deliverable (ref [34]), of which the future availability is unclear. As these requirements play an important role in the core topic of this paper, I would like to see a bit more detail in this paper about the process of eliciting these requirements. What is the profile of the participants, how were they contacted and what were generally the questions. How were then the answers aggregated into the user requirements.

- From the User requirements it is unclear what type of experiments the platform should support. Are these linking/reasoning/query benchmarks?

Nanopublication comments:

Further comments:

- U8: Please elaborate on the "execution of challenges?" What does this exactly entail? What challenges does the platform support or not support. HOw is this support organized for the managers and participants of such a challenge.

Minor issues
- 3.3 F3 "they. describe" -> they describe
- Table 1: Fair Benchmarking -> FAIR Benchmarking
- There is a remaining comment to add a number for the RAKI project in the acknowledgements

By Laura Rettig https://orcid.org/0000-0002-9765-0549

Review Details

---

Reviewer has chosen not to be Anonymous

Summary of paper in a few sentences:

This article presents Hobbit, a platform that is used for benchmarking systems that solve various tasks related to Linked Data in both local and distributed systems. The platform architecture was defined based on a set of requirements that were defined with the help of numerous experts, as well as the requirements defined in FAIR, such that the platform supports FAIR data processing (in some respects requiring adherence of the benchmarked systems to FAIR principles). The paper thus clearly explains the reasoning between certain design choices (such as decoupling into containers for platform stability). The platform's functionality is evaluated on two LD tasks, but moreover, it has demonstrated real-world usefulness by use in different challenges.

Reasons to accept:

A1 Sufficient novelty to previous publications relating to the platform.
A2 The platform and its source code are openly available to test and view.
A3 The authors define an ontology that both allows for clear functionality on the platform and ensures FAIR principles.
A4 The description of the motivation behind the components of the platform as well as of the workflow of using it as a resource for further research are clearly and understandably described in this article.
A5 The authors evaluate their platform on two common LD tasks with different configurations, showing that their platform works as expected.
A6 The quality and functionality of the platform has been established through use in numerous real-world challenges and it has a large number of registered users already.

Reasons to reject:

R1 The comparison with related work, i.e. existing frameworks on pp2-3, is somewhat weak and Table 1 could be explained in more detail.
R2 It is unclear where the limitations lie, i.e., what cannot be done on the platform.
R3 As stated in section 6, the platform controller being a single node creates a bottleneck under realistic loads.

Nanopublication comments:

Further comments:

- In Table 1, why are only 4 frameworks compared when the text mentions others? I understand that some such as Peel are more difficult to compare given their specific limitations, yet I think it would be nice to mention anyway. Also, the inclusion of "Manual Revision" at this stage seems odd given it is not supported by any of the compared frameworks.
- U5 is not addressed explicitly in section 4 but I believe it to be implicitly fulfilled.
- Section 4.3.2, are Data generators responsible for synthetic data sets? How are real data sets loaded into the benchmarking system? I suppose this is trivial but it would be nice to point out.
- Direction of messages in Figure 4 is nearly impossible to recognize, please enlarge the arrowheads slightly.
- With respect to the conclusion, what further extensions are intended in the future?

Typos and minor errors spotted:
- p2 l7: missing closing parenthesis
- p2 l8: "can be installed deployed locally"
- p3 Table 1: capitalize FAIR
- p4 l5: I suppose Ux should be U?
- p4 l34: was build -> built
- p7 l7: "implemented a system" unclear
- p7 l9 and l20: benchmark's, l12 and p10 l8: system's, l23: experiment's, p8 l38: browser's
- p9 l22: two triple stores
- p9 l28: R1.1
- p9 l40: cannot
- p11 l20: "computation the KPIs"
- p13 l42: "the average length... has a length of..."
- p14 l22: than on the single
- p15 l27: per document