Reviewer has chosen to be Anonymous

Summary of paper in a few sentences:

A review of the role of data scientists in 'team science', i.e. the conducting of large studies by large interdisciplinary teams. The paper identifies the importance of the data scientists in such teams and states that this will increase over time. As such it is important to consider now how that role should be defined and how it should interact with existing academic systems, especially with the criteria defining individual career progression for data scientists.

Reasons to accept:

This paper makes some important points concerning the mechanics of collaboration when data scientists become part of large research teams. The author uses examples of her own experience working in such teams to illustrate the important role that data scientists can play in such teams, as well as the challenges to the team and the individual. With the large scale hiring of new roles in data-oriented positions (at least in my own university), and the subsequent pool of data scientists looking for collaborations, this is a timely issue to address.

Reasons to reject:

None. However, the paper skims or misses several issues which are detailed below in further comments. It would benefit from addressing these in more detail.

Nanopublication comments:

Further comments:

Major issues:

I feel that the author lets other team members off the hook rather easily. For example, take the statement: ‘Thus, today, the modern scientist cannot be expected to know all that is necessary about the data generated or the implications of how to address the research question raised.’ In my experience, many scientists convince themselves that they could not possibly understand data and statistics. Of course, this opens opportunities for data scientists, but it also leads to the quasi-consultancy relationships that are then criticised later in the paper. It also abrogates the responsibility co-authors have to adequately check each other’s work. A major challenge in bringing data scientists into teams is avoiding a balkanisation where the data and analysis is the responsibility of one individual and the collection of that data another, with too little common knowledge. Indeed, the article touches on this area: ‘When data scientists dig deep as collaborators, gaps in methodology can easily be identified.’ The flip side of this is that (in my own experience) too much time is wasted because some researchers lack the statistical literacy to adequately explain all of the constraints or peculiarities of the data collection process, until they are finally uncovered by the frustrated analyst.

On the issue of authorship, credit and evaluation (e.g. for promotion), the author focuses her attention on how institutions can appropriately change their metrics for promotion that relate to author-order. Perhaps this is a system-specific issue as I am not aware of any *explicit* metrics based on such factors in promotion within our university (though of course such things are relevant). If criteria are explicit and/or strictly quantitative then this should be an area of frequent review. However, I suggest that the author has neglected an equally important side of the equation: namely the responsibility or otherwise of the data scientist to demand the credit they deserve and need. For instance, how does this issue intersect with relative seniorities between colleagues, and with gender and ethnic biases? Considering why data scientists consistently occupying the middle author slots is as important as re-weighting the importance of those slots. This may mean also revisiting the previous point: are data scientists joining teams as full intellectual contributors or as consultants? Are both sides in agreement about this role a priori, or are they in unspoken disagreement? Could this be solved with a more explicit conversation about roles at the start of the collaboration, as in ‘this is what I want/need to get out the project’?

Minor issues:

References are sometimes oddly formatted, for instance: 'This information directly informed how we data scientists designed the study and developed our statistical models. 7' - the reference superscript is outside the sentence.

It would be good to have a citation for the use of ‘team science’ as a previously recognised term near the start of the introduction.

‘Further, individual labs have become more specialized, as they have consequently been generating and/or handling more specific types of data for which a unique skill set is required. Thus, today, the modern scientist cannot be expected to know all that is necessary about the data generated…’ - This seems a bit vague, and the second sentence possibly contradicts the first - if most data is being generated in specialised labs with skill sets focused on that type of data, then it seems that most researchers would know how to deal with the data their lab produces (but potentially miss other techniques or know-how that they could translate from other areas)

Some examples are needed for the statement: ‘Conferences that discuss new tools and strategies for successful cross-disciplinary (including multi-, inter-, and transdisciplinary) engagement have been established.’

‘In addition, research articles in scientific journals that describe methods for cross-disciplinary collaboration are appearing (e.g.,See Börner et al., 2010 and Bennett and Galdlin, 2010)’ - these references are now 7 years old; are there no more recent examples?

“For example, as part of a group working with investigators to study the comparative effectiveness of HIV anti-retroviral agents on cardiovascular disease, team members with expertise in treating HIV noted the importance of adjusting for potential confounders like CD4 count, viral load, and cholesterol levels. Further, they emphasized that while the literature implicated single agents as having a role in cardiovascular disease, in practice, therapy was prescribed and taken in combinations, making the entire combined therapy a more pertinent focus. In addition, as we observed in the data, patients switched their combinations with great frequency.” - this section seems to move awkwardly from third to first person. I suggest working in the first person immediately to let the reader know this is personally experienced example

By Olivia Woolley-Meza https://orcid.org/0000-0003-4517-2765

Review Details

---

Reviewer has chosen not to be Anonymous

Summary of paper in a few sentences:

The author argues for the importance of multi-, trans- and interdisciplinary work where the data scientist is well integrated in a team with the necessary domain specific knowledge. The specific case she discusses is data-science in biomedical research. The main points she makes are, first, that domain specific knowledge helps the data scientist select relevant models and produce interpretable and usable results. Second, that new technical gaps can be exposed through applied data-science work especially when the non-data scientists in the team push the data scientist to detect shortcomings in the existing approaches. This leads to advancements in "pure" data-science research. Further, the author suggests some of the institution level frameworks necessary to sustain and encourage this activity. Namely, the appropriate metrics need to be designed to evaluate data science work that is primarily team and application driven, and career tracks opened up for researchers that work principally in this environment.

Reasons to accept:

I think this is an important and urgent topic, with general applicability to the work of data-scientists. The commentary is well-written and an interesting read.

Reasons to reject:

However, I think that in some places the general thread of the argument is lost, and some critical points are touched upon superficially. I make some suggestions below. Once these points are addressed this will be a piece well-worth publishing.

Nanopublication comments:

Further comments:

I would recommend that the title make specific reference to Biomedical work since this is the only domain that is explicitly discussed. I would also suggest that in the conclusion the author points to other specific subject domains where well-integrated team data-science is important and should be fostered by academic institutions.

The author uses the idea of relevance of a model as a central argument for why integrating a data-scientist in the team is good. Another point is the appropriate clinical interpretation. Specifically: "Without the context provided by the HIV experts, the data scientists would have developed a less relevant model, and possibly provided misleading findings. Thus, integration of the data scientist into the team environment enabled the data scientists to arrive at an approach that yielded clinically appropriate interpretation. The iterative team-based process also more generally ensures that products from the collaboration will be relevant. "
These are very important points, but also controversial. It is easy to make the opposite argument that the involvement in the team leads to "group-think", and narrowed/biased model selection according to conventional understanding, ruling out better and more innovative approaches. The author should address how to avoid/minimize this problem or why it is not an issue in the setting she is discussing.

In the example of exposing gaps in existing methods, the author does not provide a reference for the missing data imputation method developed. Either a reference is required and/or an explanation of how the method developed is different from the standard, and how the input of domain experts made this so.

It is unclear to me where the line is drawn between an external "consultant" data scientist and one who is really part of the team. I think spelling this out more clearly early on in the argument would make it clearer.

Finally, in regards for how to evaluate contribution to team work, and the overall performance of a researcher who primarily engages in this work, the author says :"The authors developed an excellent and systematic approach for how evaluation and promotion could better recognize the intellectual leadership of the team scientist.[8] They further described how to evaluate contributions to publications, grants, and research programs, in order to summarize overall scholarship in a way that appropriately weights contributions to team-based science."
What is the main idea behind this systematic approach? The innovation? A short explanation of the specific method suggested is really left wanting.

By Brian Davis https://orcid.org/0000-0002-5759-2655

Review Details

---

Reviewer has chosen not to be Anonymous

Summary of paper in a few sentences:

This position paper in essence advocates for the importance of team science but more importantly in order to encourage the activity in must be recognised and rewarded in academic/research staff performance review. This is valid point but the paper could be more convincing with respect to structuring its arguments. I like the topic but it needs some more work.

Reasons to accept:

Well written
Team science is an important issue and I will a publication that is not concerned with a technical experiment but is concerned with scholarship with respect research skills and professional development of the research scientist (data scientist)
Relevant in that this is paper is concerned with the (academic/research) practitioner of data science.
I argue that perspectives on the an emerging field such as this should be encouraged.
Good related work.

Reasons to reject:

Referencing is little confusing. Either footnotes/citations?
There is what appears to be a quote from Dr. Julie Segre. This should be surrounded by "". Its not clear to me if the author is referring to her or someone else. The later is not acceptable for quote in a publication.
The is not well structured. There are no sections.
The author states the problem - but does not go into detail about possible causes and or/what methods are needed to verify explore this problems wrt to credit/promotion further. What can be found in related literature from other disciplines. How does one claim credit for scientific team work in other disciplines such as physics. How can team science skills be measured? What does the literature say with respect to other disciplines. I cant imagine that this is not a new problem but rather needs some detective work in other disciplines under the scholarship of research/academic practice

Nanopublication comments:

Further comments: