By Stefano Balietti https://orcid.org/0000-0002-6056-6765

Review Details

---

Reviewer has chosen not to be Anonymous

Summary of paper in a few sentences (summary of changes and improvements for second round reviews):

The paper has clearly improved from previous round. The literature review has been expanded, and it is now a fairly comprehensive overview of smartphones-based studies, specially on the effects of nudges on participants.
The authors also reduced the part on the results of the pilot, which still remains the least interesting part of an overall interesting and inspiring paper.

Reasons to accept:

Good overview of methods and papers to conduct social science research with smartphones
Inspiring
Important research topic

Reasons to reject:

I don't think it should rejected, but it could be improved on these two points:
- The part on the pilot is still of limited use to other researchers, maybe it could be shrunk even more
- The pilot is run with a software which is different from what is currently being developed, and described in the Supplementary Material. This is a bit confusing, we would like to know more about the potential of the new software!

Nanopublication comments:

Further comments:

I enjoyed reading it, and I hope to see this type of research quickly moving from position papers to actual research papers. Some comments that should be addressed:
- The procedure for recruitment of subject is not clearly described: how were subjects actually found? This is important to clarify because it increases reproducibility, and permits the reader to judge whether interference was actually taken care of or not.
- Many readers might not be familiar with the Gaussian Process choice models used to generate the landscapes of interactions in Fig.2. Even if references are correctly provided, a brief description of the method would be very useful.
- There is no reference to IRB approval in the main text (I might have missed in the supplementary material), but the number should be there.
- A few typos: emacipatory, Furuthermore, "tool place"

By Victor de Boer https://orcid.org/0000-0001-9079-039X

Review Details

---

Reviewer has chosen not to be Anonymous

Summary of paper in a few sentences (summary of changes and improvements for second round reviews):

his is a revised and resubmitted version of an earlier research submission by the same authors. This new version of the paper is presented as a position paper rather than a research paper. In my previous research I indicated that indeed I could not find a research contribution. This version emphasizes the methodogical aspects a bit more and identifies specific opportunities for using smartphones in experimental studies to monitor and affect behaviour change. The authors have expanded the literature research and place their ideas in this context. The ideas are still supported by an experimental 'pilot' study. The pilot study has not been updated so my specific remarks regarding this experiment stand.

Reasons to accept:

- Interesting pilot study
- Potential for clear methodological guidelines for this type of interventions in living labs sessions

Reasons to reject:

With respect to the methodological contributions. The authors claim that their approach allows for experimental setups, in contrast to previous study. If I understand this claim correctly, this is where the novelty lies. (p2 "Most of these studies however do not implement an experimental design. ") I am not at all convinced of this point. Especially in the health domain, but also in other domains. These types of experimental studies with devices / smartphones with bespoke applications in living lab type setups have been done before and I dont see how the suggestions of the authors push beyond the state of the art. One recent example can be found here [1]. but there are likely many others. In fact, the platform on which the experiment is done already existed..
Also, the identification of different types of interventions is not really novel or well worked-out. The "AI-based emancipatory" intervention sounds interesting but is really under-explained.

Many ideas and open questions are posed throughout the paper without details or follow-ups. Example are:

- p6 "This intervention could be tested against a “non-emancipatory" AI-based decision-making assistant which would also provide the user with all necessary information and answer their questions, but which would then suggest to participants an “optimal" decision solution based on participants’ stated preferences."
- "We suggest to study multiple experimental interventions" -> I do not understand why this would be stated in a research or position paper. This would indeed be an interesting study, but in my view, a position paper should not just point out interesting research directions.
- I have similar concerns about the qustions phraased in p5 "But how could we encourage
behavioural change in a more emacipatory, empowering way? Should the goal not be to empower individuals in their capability to make informed decision that are right for them and the society overall?" -> Rather than posing open questions, the paper could identify possible answers, pros/cons or identify challenges and solution directions.

The paper does not present a clear conclusion to the points raised but rather ends with a discussion section regarding the pilot study. No clear position is identified, lessons learned or a comprehensive methodology presented.

[1] Mollee, J. S. (2018). Moving forward: Supporting physical activity behavior change through intelligent technology

Nanopublication comments:

Further comments:

- Large sections (1, 2), can do with some structuring, they are very long fields of text. Some structuring might make the paper easier to read.

- The paper starts ("In 2015 the United Nations") with an introduction which has no section header or number. Section 1 is not 'state of the art'. This is an unusual layout decision, I would suggest having a standard Introduction header.

By Evangelos Pournaras https://orcid.org/0000-0003-3900-2057

Review Details

---

Reviewer has chosen not to be Anonymous

Summary of paper in a few sentences (summary of changes and improvements for second round reviews):

This paper discusses the use of smart phones to study environmental behavior in field and living lab experiments. The authors provide an overview of state of the art and illustrate a small-scale pilot study with some inconclusive results.

Overall, although the paper is very interesting and discusses some very timely methodological and epistemological aspects of big data and IoT technologies, its contributions remain questionable. I would like to elaborate below on these.

Reasons to accept:

1. Very interesting topic and perspective
2. Some nice overview of state of the art (though not always relevant and well positioned in the paper)
3. Valuable methodological discussions with references to theory on conducting field tests

Reasons to reject:

1. Highly controversial and (self-)conflicting statements.
2. Inconclusive results
3. Not good positioning of related work (though as mentioned before a great outlook)
4. Unclear contributions as well as paper objective: stepping on two boats - position paper vs. research paper without addressing properly each one.

Nanopublication comments:

Further comments:

This paper is split into a positioning/review part and an experimental part. I have the following major comments for each of these two parts:

1. The authors claim that a big data approach without introducing rigorous experimental design may not be relevant to study causality. This is an important statement with significant value and I sympathize with authors addressing this important issue. However, by reading this paper the authors themselves seem to undermine their statement with several remarks they make throughout the paper. For instance, in page 4 & 5 they list several data that "should be collected", "should be obtained", "should be able to record data", "could be potentially collected", etc. but I am questioning this...Why "should" they be? It sounds like the big data approach, there is not clear link of how a data collection process should be connected with causality studies and what requirements such data collection processes should fulfill to provide causal evidence (lessons learnt from their field tests as well as related work could be integrated here). This would be a paper contribution, and I encourage the authors to move to this direction.

2. Following the above, the authors mention: "The challenge is to find a way [...] users' everyday life". Fair comment, but is not this an utopia in the context of all these "should be collected" data? Given that nowadays inference can be performed even if we share zero data, e.g. social networks, friends of friends, etc. it makes sense here the authors to discuss some way to eliminate inference, e.g. privacy mechanisms (differential privacy), informational self-determination, accountability in data sharing with blockchain, etc. The same principle holds for nudging: who nudges and for what purpose? How do we make sure that nudges serve sustainability and not only corporate profits for instance? Moreover sustainability is multi-faceted, how do we encounter for rebound effects? I am not expecting the authors to address all these issues, but in the context of this discussion these aspects are relevant.

3. I would prefer the authors to discuss the related work in a narrower scope around the main positioning, i.e. experimental design in data collection. I would like to point to some very relevant work here in case authors find it useful:

- ASSET project (http://asset-consumerism.eu), Johannes Klinglmayr, Bernhard Bergmair, Maria Klaffenböck, Leander B. Hörmann, Evangelos Pournaras, Sustainable Consumerism via Context-Aware Shopping, International Journal of Distributed Systems and Technologies, 2017
- Sensing & mining urban qualities: Danielle Griego, Varin Buff, Eric Hayoz, Izabela Moise, Evangelos Pournaras, Sensing and Mining Urban Qualities in Smart Cities, in the proceedings of the 31st IEEE International Conference on Advanced Information Networking and Applications-AINA-2017, Taiwan, March 2017
- Authors also mention "Our idea here is to design [...] society overall." discussing about options and altenratives. This is indeed a very exciting research pathway, however, there is significant ongoing work here as well, in the area of multiagent systems and reinforcement learning. I think the following work captures some relevant social/computational dilemmas in the area of energy and transport sharing: Peter Pilgerstorfer and Evangelos Pournaras, Self-adaptive Learning in Decentralized Combinatorial Optimization-A Design Paradigm for Sharing Economies, in the Proceedings of the 12th International Symposium on Software Engineering for Adapt\
ive and Self-managing Systems-SEAMS-2017, Buenos Aires, May 2017

4. The experimental methodology of the field test seems impossible to scale it up at large scale: several administrative actions and high level of moderation is required. What would you do and what would you change to scale up the approach to thousands of people? Finally, on page 8, 2nd paragraph it is impossible to understand what all the statistical test mean and what purpose they serve, it seems that all numbers are packed in a paragraph. Please explain the rationalle of the measurements.

Some other minor comments:

- relation between distance and preference to higher CO2 transport means is misleading: it is unlikely in principle that someone will travel abroad by bike, for instance.
- The plots are not B&W friendly, please improve.
- Please add subcaptions in Figure 2 for an easier readability.