By Lance Downing https://orcid.org/0000-0002-2133-9244

Review Details

---

Reviewer has chosen not to be Anonymous

Summary of paper in a few sentences:

A position paper on best practices in translational research.

Reasons to accept:

Timely and appropriate set of best practices supported by examples and historical relevance. Generally well written with reasonable use of figures.

Reasons to reject:

Not very novel, however seems reasonable given the format of a position paper.

Nanopublication comments:

Further comments:

Reviewer has chosen to be Anonymous

Summary of paper in a few sentences:

Paper aims to give listing of ten commandments translational teams should follow when considering data for projects and how best to manage that data. THey stress the importance of consistency and planning BEFORE a study actually begins, which will enable easier analysis at the end of the study.

Reasons to accept:

This type of information is extremely useful, especially to newly formed teams. Having worked in the field for years, I frequently encounter resistance to several of these topics from PI's, and it is often challenging to perform data analysis when some of these steps are not thought of at the beginning of a study. Having something to reference would be very helpful.

Reasons to reject:

It is unclear who the intended audience is. The author seems to imply the commandments would be useful for new teams, however much of the language is in acronyms which a new team might not know. The audience should be better defined, and that should be thought of when explaining each commandment. Along those lines, including examples for some of the commandments (i.e. commandment 1 would be nice to include or reference a work package). The author mentions this is for translational research, yet the focus seems to be mainly on clinical data. Basic science data needs to be mentioned/included more to make it more relevant to the entire translational research spectrum, and not just clinical research which is how it largely reads at the moment.

Explain acronyms...there are many, and some are not defined.

Commandment 1 - work package is strange phrasing, as Ive never heard that term, and I suspect many others have not either. SOP (standard operating protocol) might be a better choice. A reference should be included for Horizon 2020. It would be nice to include what a WP should include in this commandment as well. Examples would be helpful for people forming new teams.

Commandment 2 - What is EDC? SInce reference NCATS earlier, would be nice to include REDCap in references here as well. How does this commandment apply to EMR or basic science data? Can you extract data directly from machines to ease this problem?

Commandment 3 - in addition to a codebook, it would be good to consider restricting the data types for some variables to diminish errors (such as gender should not be recorded as numeric). Also good to include plausible values for some variables to diminish errors as well. This is in addition to having a codebook. What is eCRF? ANd how does that apply to basic science research/teams?

Commandment 4 - would be good to mention places that one can share data as well as how to go about the sharing process as it is not always straightforward.

Commandment 5 - Would be good to list, or reference, specific imaging tools mentioned in the last sentence.

Commandment 6 - Table 1 would benefit from a better description of the software...such as what each tool actually does, or what specific field (analytic area) do the resources apply to. Potentially along the lines of why someone would want to visit each of the websites (i.e. what can one use i2b2 to help them with?)

Commandment 7 - should also mention importance of dissemination of these tools so that others know they actually exist, and can easily find

Commandment 8 - it would be good to include an example of how to apply/follow the FAIR guidelines

One additional thing to consider is that it is often a good idea to include in the team/data management plan to consult with someone that will be doing the downstream data analysis from the acquired data. This is to ensure that the fields collected will be in a format needed for easier access, and that the statistician/bioinformatician won't have to perform magic before analysis can be performed.

Nanopublication comments:

Further comments:

Reviewer has chosen to be Anonymous

Summary of paper in a few sentences:

The paper outlines several suggestions for planning data management processes based on the author's experience in prior projects.
These mostly seem to revolve around clinical trials of translational research products. The suggestions are all reasonable, though not sure they are novel for those familiar with such work or clear/compelling for those who are not.

Reasons to accept:

Principles described are all reasonable and important for successful data management and collection processes that could easily be overlooked by PIs only focused on the end results/overriding hypothesis. The "dirty work" of gluing together the study/data infrastructure is indeed important to ensure projects are successful.

Reasons to reject:

Hard to parse out who target audience is for the writing here.
Suggestions are largely just "plan ahead," which is a generally good suggestion, but often not specific enough to be useful to those who need it. May help some translational research managers formulate a checklist of elements to consider on predictable data management issues, but limited background or examples/illustrations provided on the consequences means that many of the people who would understand how to apply the suggestions (data managers) are likely those who would already know about them. Whereas those that writer is probably trying to convince (research PIs) are not getting enough background or illustration to convince them to commit the extra resources and planning that are appropriately suggested.

Excessive use of abbreviations without definition (e.g., WP, FTE, EDC, PCCM, eCRF, BFO, OBO, RO, GDPR, HIPAA, EHR, DICOM, ARX) is just a superficial example of how the writing is going to miss people at different ends of the translational research spectrum, and largely be "preaching to the choir" when it does hit.

May help to reduce the number of suggestions (not artificially constrained to the "ten commandments" framework), so have more room to make individual key sections on HOW to achieve the general suggestions made.
For example, reference to GDPR and HIPAA and how should have good data processor agreements and proper patient consent explanation. A reader who does not already know, will have no greater clarity on how to do any of this. Could have an example of what a typical, but poor, patient consent statement would be vs. a good one.
Commandment 10: Think about sustainability. A reader can "think" about it and not actually be able to do anything about it. Some concrete suggestions on example sustainability plans would help. (e.g., data repositories as suggested, or operational hosting plans, institutional core resources, etc.)

Nanopublication comments:

Further comments:

By Mark Musen https://orcid.org/0000-0003-3325-793X

Review Details

---

Reviewer has chosen not to be Anonymous

Summary of paper in a few sentences:

This position paper offers a set of desiderata for the data stewardship of projects in translational research—or really any research actually. These principles are derived from the authors personal experience in several large research projects.

Reasons to accept:

The paper is accessible, written in a style that will grab attention and that communicates its ideas well. The topic is important and is not well covered in the translational science literature.

Reasons to reject:

The paper does not reference current literature on data stewardship and it does not seem to recognize that data stewardship is already a scholarly discipline.

Nanopublication comments:

Further comments:

The paper is generally well written, but needs copyediting for English usage. "Learnings" is not a concrete noun in English. "Up front" is two words. "Institutes" are transient, whereas "institutions" have durability. It is not clear that "replacement" refers specifically to a person.

It is not obvious to me that any of the commandments will necessarily become irrelevant with time as the author suggests. The ideas may become practiced more widely, but scientists will still need to be aware of these principles. It is also not obvious why these principles apply only to translational research. They would seem to have much wider applicability.

The author needs to define all the acronyms in the paper: CRF, eCRF, PCCM, BFO, OBO, RO, GDPR, HIPAA, EHR

The reference to "ontologies for translational research" in Commandment 3 mentions BFO, OBO, and RO. OBO is not an ontology, but rather a library of ontologies. BFO and RO are ontologies sometimes used in the construction of other ontologies, but they themselves do not specify terms that are relevant for the creation of standardized metadata.

What was the method by which the entries in Table 1 were created? How can the reader know that this list is in any way complete? Many of the descriptions in this table do not give the reader a sense of what the tool actually does.

Although these commandments are quite reasonable, how do we know that there are only ten such commandments? A review of the literature on data stewardship might reveal others worth including as well—although I appreciate the attraction of having exactly ten.

Reviewer has chosen to be Anonymous

Summary of paper in a few sentences:

Important and relevant topic, but this manuscript takes an overly simplistic and incomplete approach to development of such a list. The manuscript makes statements that are naïve to the complexities of the issues involved, for example, “at the end of the project, data should be shared with the whole world.” This is a nice idea, and could be useful to investigators, but is underdeveloped in its current form.

Reasons to accept:

This is a nice idea, and could be useful to investigators, but is underdeveloped in its current form.

Reasons to reject:

This is a nice idea, and could be useful to investigators, but is underdeveloped in its current form. For example, statements made about some ‘commandments’ are overly simplistic and naïve to the complexities of the issues involved.

For example, the statement that “at the end of the project, data should be shared with the whole world” is overly simplistic and frankly naive. What about PHI and PII? How do you balance public access to data with the need to incentivize investigators to (painstakingly) collect data for many years? How do you protect participant privacy when multiple datasets with their information may be released? Another example is the statement “if the party performing the data integration receives data that is not properly de-identified, it should be destroyed immediately because of the privacy risk.” Again, this is overly simplistic.

In addition, there are inaccuracies in the list of freely available software table. For example, the REDCap reference points to its implementation at University of Minnesota, rather than the REDCap consortium webpage itself. This suggests that the author is making lists/recommendations for tools that he/she does not fully understand or has adequately investigated.

Although the author refers to the “valley of death” (based on a published Nature article), reliance on the “Ten Commandments” structure and the religious overtones it entails is overdone and potentially offensive. Would suggest changing to “Ten Recommendations” and leaving out all the religious references.

Nanopublication comments:

Further comments:

The manuscript would benefit from substantial copy-editing. For example, there is no such word as “productized.”