By Victor de Boer https://orcid.org/0000-0001-9079-039X

Review Details

---

Reviewer has chosen not to be Anonymous

Summary of paper in a few sentences:

This paper presents a pilot study on the possible role that smartphone applications can have in behaviour change, specifically to encourage environmentally-friendly behaviour. The pilot study asks 20 people over two weeks to report behaviour. In the second week, two random groups receive either individual or group-based feedback and the main research question is what the effect of a) using the application and b) each of these conditions is. The statistical analysis and visualizations performed appear reasonable and could be informative in a larger scale experiment.

Reasons to accept:

- Interesting idea and application setup
- Well-described and clearly presented, with appropriate discussion on the limitations of the study
- Some valuable lessons learned for similar studies

Reasons to reject:

- This is a pilot study only, on which no conclusions can be based
- No new methodological insights beyond the specific case
- no new concepts or ideas introduced

Nanopublication comments:

Further comments:

This paper presents a pilot study on the possible role that smartphone applications can have in behaviour change, specifically to encourage environmentally-friendly behaviour. The pilot study asks 20 people over two weeks to report behaviour. In the second week, two random groups receive either individual or group-based feedback and the main research question is what the effect of a) using the application and b) each of these conditions is. The statistical analysis and visualizations performed appear reasonable and could be informative in a larger scale experiment.

Where this is an interesting idea, the authors acknowledge that the study is very limited to measure behavioural change. The main issues being the small sample size, the short-term naturea and the fact that there is no control group. This makes it impossible to draw any specific conclusions from the study, which the authors state on multiple occasions. The question then remains what the contribution of this paper is other then presenting a nice idea, and I cannot find it.

A possible contribution could be methodological. However, for that there is simply too little progress beyond the state of the art. On many occasions, authors point our lessons learned for their 'real study'. While these points are interesting (for example the barcode and meter scanning) , there is a lack of generalizations of these remarks and as such do not contribute new structured methodological insights.

The paper also does not introduce new concepts or ideas, as also explained in the description of related work. Using smartphone applications for behaviour change has been studied before, and for the field of Data Science I could not identify new concepts, nor do the authors state their contributions or positions explicitly.

Therefore, I do not think the paper in its current form should be accepted fo this journal. However, if the proposed larger scale and improved study is performed, I am sure that interesting insights could be gaind and this pilot study could be an interesting part of that paper.

A few remarks about specific parts of the paper and the study:
- The study requires people to spend 5-12 minutes per day to enter information. While the paid participants can be expected to do this, it seems unreasonabe to assume that the general public will do this voluntarily. This is addresed a little bit in the limitations section, but this is potentially a serious problem. The authors could elaborate on how much is automatically detectable using the smartphones sensors, what things would still need to be entered manually and whether this can be reasonably expected.
- It is unclear how much of the effects over time are because of increased awareness rather than the treatments. Again, here a control group will provide more insight.
I have some issues with some of the questions. For example, the "Did you use a lamp" question. I think everybody in the western world uses a lamp at least once a day, and I dont think this is informative. More informative are a) the duration is more important (did you turn off the lamp) or b) the type (old-style lamp vs energy efficient makes a huge difference). The same holds for some of the other questions as well.
- the paper is generally written in a clear and understandable language. Some minor typos and grammar issues:
- p2: cannot necessary -> cannot necessarily
- p3 "In this study the nudges" -> In this study, the nudges
- p4 "In the pilot study notifications" -> In the pilot study, notifications
- p5: "in the second weak" -> week

By Tobias Kuhn https://orcid.org/0000-0002-1267-0234

Review Details

---

Reviewer has chosen not to be Anonymous

Summary of paper in a few sentences:

This paper argues for a new mobile phone based method to raise awareness on sustainable behavior and an experimental design to test such approaches. The design and implementation of a pilot study is presented, which uncovered several interesting avenues of research questions to follow in the future but did not lead by itself to conclusive results.

Reasons to accept:

- very important problem
- interesting methods to increase awareness about sustainable behavior
- promising and interesting definition of "treatments" for the experiment

Reasons to reject:

- As a position paper, it is lacking an explicit and well-motivated description of a novel view or methodology (we can only find fragments thereof)
- It is also not really a research paper either (which it doesn't claim to be): it doesn't have a clearly formulated research question for which an answer is sought by means of experimentation (even if that answer turns out to be not conclusive)
- The overall approach is insufficiently explained on a general level
- The lack of a control group in the experiment makes it difficult to gain insights from the negative/non-conclusive results
- Insufficient coverage of related work

Nanopublication comments:

Further comments:

This position paper presents a pilot study of a very promising type on a very promising method to raise awareness about sustainable behavior. Unfortunately, the paper focuses too much on the particular pilot study and gives too little information about the overall position, vision, novelty, and methodology. I think there is an interesting vision behind the work presented in the paper, and I would encourage the authors to motivate and explain this vision in detail, which could then lead to a convincing and valuable position paper.

The pilot study was quite small but large enough so it could have given significant results, but unfortunately it did not. Such negative/non-conclusive results can be very interesting in their own right, but unfortunately the value of the given results are also diminished by the lack of a proper control group or null treatment in the study design.

The paper is also lacking details on related work that would point out in what way exactly the presented work/position is novel. For example, we read about related work that "most of these studies do not implement an experimental design", but we are not told about the few existing studies that *do* implement an experimental design and how they differ from the presented work.

In summary, I think the paper is neither a good position paper nor a good research paper, and therefore I suggest the rejection of this work at this point. However, I encourage the authors to improve the above-mentioned shortcomings and re-submit it in the form of a proper position paper and/or proper research paper.

Below are some more detailed comments.

- The section "Methodological Approach" does not actually introduce a methodology or approach, but jumps right to implementation details (like which software was used)

- Study results are mixed into the method description (like "Moreover, the pilot study showed that ...")

- "collecting electric meter data through image taking": This type of data collection should be explained first.

- "it was attempted to maintain the non-interference assumption": how was this done?

- "It shows how abstract climate change concerns remains for most people when it comes to their day-to-day life.": Not necessarily. They might be fully rational (in the economic sense) and fully aware actors in a public goods dilemma. This would also be a quite obvious possible answer to the question in the last section "why are people not more environmentally friendly, even if they clearly think that sustainability and climate change is important?".

- Figure 1: I would find it more helpful to show an aggregation of all users, instead of plots for each individual user.

- As the authors write all the non-conclusive results were "rather to be expected", it is not clear why the authors set up the pilot study in this way. And what specifically can we learn from these expected non-conclusive results?

- "The choice is furthermore influenced by climate change attitudes." This direct causal relation does not follow from the observed correlation. For example, people with high awareness of climate change might tend to move to cities, where in turn biking tends to be a more feasible option for transportation.

- I am not a native speaker, but I think that the way how "allow to" is used in the paper (e.g. in "which would allow to implement") is not admissible in English (but rather "allow us to" or "allow for"). But again, I might be wrong.

- "intension" > "intention"