Reviewer has chosen not to be AnonymousOverall Impression:
UndecidedTechnical Quality of the paper:
Lack of noveltyData availability:
With exceptions that are admissible according to the data availability guidelines, all used and produced data are FAIR and openly available in established data repositoriesLength of the manuscript:
The length of this manuscript is about right
Summary of paper in a few sentences (summary of changes and improvements for
second round reviews):
This paper discusses the use of smart phones to study environmental behavior in field and living lab experiments. The authors provide an overview of state of the art and illustrate a small-scale pilot study with some inconclusive results.
Overall, although the paper is very interesting and discusses some very timely methodological and epistemological aspects of big data and IoT technologies, its contributions remain questionable. I would like to elaborate below on these.
Reasons to accept:
1. Very interesting topic and perspective
2. Some nice overview of state of the art (though not always relevant and well positioned in the paper)
3. Valuable methodological discussions with references to theory on conducting field tests
Reasons to reject:
1. Highly controversial and (self-)conflicting statements.
2. Inconclusive results
3. Not good positioning of related work (though as mentioned before a great outlook)
4. Unclear contributions as well as paper objective: stepping on two boats - position paper vs. research paper without addressing properly each one.
This paper is split into a positioning/review part and an experimental part. I have the following major comments for each of these two parts:
1. The authors claim that a big data approach without introducing rigorous experimental design may not be relevant to study causality. This is an important statement with significant value and I sympathize with authors addressing this important issue. However, by reading this paper the authors themselves seem to undermine their statement with several remarks they make throughout the paper. For instance, in page 4 & 5 they list several data that "should be collected", "should be obtained", "should be able to record data", "could be potentially collected", etc. but I am questioning this...Why "should" they be? It sounds like the big data approach, there is not clear link of how a data collection process should be connected with causality studies and what requirements such data collection processes should fulfill to provide causal evidence (lessons learnt from their field tests as well as related work could be integrated here). This would be a paper contribution, and I encourage the authors to move to this direction.
2. Following the above, the authors mention: "The challenge is to find a way [...] users' everyday life". Fair comment, but is not this an utopia in the context of all these "should be collected" data? Given that nowadays inference can be performed even if we share zero data, e.g. social networks, friends of friends, etc. it makes sense here the authors to discuss some way to eliminate inference, e.g. privacy mechanisms (differential privacy), informational self-determination, accountability in data sharing with blockchain, etc. The same principle holds for nudging: who nudges and for what purpose? How do we make sure that nudges serve sustainability and not only corporate profits for instance? Moreover sustainability is multi-faceted, how do we encounter for rebound effects? I am not expecting the authors to address all these issues, but in the context of this discussion these aspects are relevant.
3. I would prefer the authors to discuss the related work in a narrower scope around the main positioning, i.e. experimental design in data collection. I would like to point to some very relevant work here in case authors find it useful:
- ASSET project (http://asset-consumerism.eu), Johannes Klinglmayr, Bernhard Bergmair, Maria Klaffenböck, Leander B. Hörmann, Evangelos Pournaras, Sustainable Consumerism via Context-Aware Shopping, International Journal of Distributed Systems and Technologies, 2017
- Sensing & mining urban qualities: Danielle Griego, Varin Buff, Eric Hayoz, Izabela Moise, Evangelos Pournaras, Sensing and Mining Urban Qualities in Smart Cities, in the proceedings of the 31st IEEE International Conference on Advanced Information Networking and Applications-AINA-2017, Taiwan, March 2017
- Authors also mention "Our idea here is to design [...] society overall." discussing about options and altenratives. This is indeed a very exciting research pathway, however, there is significant ongoing work here as well, in the area of multiagent systems and reinforcement learning. I think the following work captures some relevant social/computational dilemmas in the area of energy and transport sharing: Peter Pilgerstorfer and Evangelos Pournaras, Self-adaptive Learning in Decentralized Combinatorial Optimization-A Design Paradigm for Sharing Economies, in the Proceedings of the 12th International Symposium on Software Engineering for Adapt\
ive and Self-managing Systems-SEAMS-2017, Buenos Aires, May 2017
4. The experimental methodology of the field test seems impossible to scale it up at large scale: several administrative actions and high level of moderation is required. What would you do and what would you change to scale up the approach to thousands of people? Finally, on page 8, 2nd paragraph it is impossible to understand what all the statistical test mean and what purpose they serve, it seems that all numbers are packed in a paragraph. Please explain the rationalle of the measurements.
Some other minor comments:
- relation between distance and preference to higher CO2 transport means is misleading: it is unlikely in principle that someone will travel abroad by bike, for instance.
- The plots are not B&W friendly, please improve.
- Please add subcaptions in Figure 2 for an easier readability.