Reviewer has chosen not to be AnonymousOverall Impression:
UndecidedTechnical Quality of the paper:
Incomplete or inappropriateNovelty:
Limited noveltyData availability:
All used and produced data (if any) are FAIR and openly available in established data repositoriesLength of the manuscript:
The length of this manuscript is about right
Summary of paper in a few sentences:
This submission presents a framework to model and to predict user engagement with mobile applications. The framework is evaluated by using a data set of app usage of one particular app focusing on waste recycling. With the successful evaluation, the authors aim to provide evidence that it is possible to predict when users of a mobile application will get disengaged.
Reasons to accept:
The focused topic of modeling and predicting user engagement for mobile applications is timely and relevant for the research communities in Data Science and Human-Computer Interaction. Overall, the presented approach seems to be novel and well-suited. Furthermore, also the results of the evaluation are promising.
Reasons to reject:
I have strong doubts regarding the used data set and features. The used waste recycling app is described only briefly. The authors do not argue, why this is a common mobile application. I would recommend discussing this with consideration of the results presented by Müller et al. . I would question that it is common for mobile applications that gamification aspects (here earning points) are directly connected to providing monetary benefits. Here, it is particularly interesting that the granted points can only be used at local shops. Thereby user’s location becomes an obvious feature for disengagement. Additionally, using the zip code and the geolocation provides only redundant information. In general, the list of features and calculated variables is fuzzy. The authors claim to use 7 features but present only 6 in a list. Also, it reminds unclear how they combined the features to the 122 variables.
The authors do not describe if the application triggered any notifications. However, Sahami Shirazi et al. describe notifications as an essential element for engaging with mobile applications . Hence, I wonder why the authors did not use the number of notifications or the reaction to notifications as a feature. To be able to understand user engagement or disengagement with the waste recycling app, it would be helpful, if the authors would also publish the application or provide at least a reference to the application.
While the authors motivate their work in the introduction very general, also looking on specific application domains such as health (reference  in the submission), the authors discuss the limitation of the used data set only briefly at the end of the paper.
 Hendrik Müller, Jennifer Gove, and John Webb. 2012. Understanding tablet use: a multi-method exploration. In Proceedings of the 14th international conference on Human-computer interaction with mobile devices and services (MobileHCI '12). ACM, New York, NY, USA, 1-10. DOI: https://doi.org/10.1145/2371574.2371576
 Alireza Sahami Shirazi, Niels Henze, Tilman Dingler, Martin Pielot, Dominik Weber, and Albrecht Schmidt. 2014. Large-scale assessment of mobile notifications. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '14). ACM, New York, NY, USA, 3055-3064. DOI=http://dx.doi.org/10.1145/2556288.2557189
As described the used features and the data set look more specific than general to me. Hence, the submission would be more substantial if the authors would make less general claims and focus particularly on comparable mobile applications. Furthermore, publishing not only the data set but also the application or providing a reference to the application would improve the validity.
Submitted by Bin Liu on
This paper studies the user engagement in mobile apps. This is an interesting problem with important practical implications. The paper investigates the predictability of when mobile app users get disengaged with apps and shows that it can achieves the engagement prediction with a good level of accuracy. It applies different prediction models and also show clustering further facilitate the prediction. The paper is interesting, but there are some limits.
First, it only shows the predictability while doesnot show much details about the prediction itself, namely, what features would lead to the prediction. As a result, the practical implication would be limited. Also the features used in the perdition might not provide useful indication of engagement management without further investigation, such as significance test, etc.
Second, the prediction just applies some standard models without much technical novelties (also given the practical implication can be limited given the current status of the paper (ie, lacking of details about the prediction model); it would be helpful also give more details of technical barrier of the problem and solutions.
Third, more features would be helpful, in particular some features can explain use engagement such as version updates, similar apps in the market. App usage feature might just related to what to predict in this paper.
Meta-Review by Editor
Submitted by Tobias Kuhn on
We encourage you to revise the manuscript, based on the 2 reviews and 1 comment made.
Jodi Schneider (http://orcid.org/0000-0002-5098-5667)