Modelling and Predicting User Engagement in Mobile Applications

Tracking #: 608-1588

Authors:

	Name	ORCID
	Eduardo Barbaro	https://orcid.org/0000-0001-7878-2514
	Eoin Martino Grua	https://orcid.org/0000-0002-5471-4338
	Ivano Malavolta	https://orcid.org/0000-0001-5773-8346
	Mirjana Stercevic	https://orcid.org/0000-0002-9847-2959
	Esther Weusthof	https://orcid.org/0000-0002-1940-0800
	Jeroen van den Hoven	https://orcid.org/0000-0002-4529-137X

Responsible editor:

Jodi Schneider

Submission Type:

Research Paper

Abstract:

The mobile ecosystem is dramatically growing towards an unprecedented scale, with an extremely crowded market and fierce competition among app developers. Today, keeping users engaged with a mobile app is key for its success since users can remain active consumers of services and/or producers of new contents. However, users may abandon a mobile app at any time due to various reasons, e.g. the success of competing apps, decrease of interest in the provided services, etc. In this context, predicting when a user may get disengaged from an app is an invaluable resource for developers, creating the opportunity to apply intervention strategies aiming at recovering from disengagement (e.g. sending push notifications with new contents). In this study, we aim at providing evidence that predicting when mobile app users get disengaged is possible with a good level of accuracy. Specifically, we propose, apply, and evaluate a framework to model and predict User Engagement (UE) in mobile applications via different numerical models. The proposed framework is composed of an optimized agglomerative hierarchical clustering model coupled to (i) a Cox proportional hazards, (ii) a negative binomial, (iii) a random forest, and (iv) a boosted-tree model. The proposed framework is empirically validated by means of a year-long observational dataset collected from a real deployment of a waste recycling app. Our results show that in this context the optimized clustering model classifies users adequately and improves UE predictability for all numerical models. Also, the highest levels of prediction accuracy and robustness are obtained by applying either the random forest classifier or the boosted-tree algorithm.

Manuscript:

ds-paper-608.pdf

Supplementary Files (optional):

ds-supplementary-608-939.pdf

Previous Version:

Modelling and Predicting User Engagement in Mobile Applications

Data repository URLs:

http://s2group.cs.vu.nl/files/DataScienceReplicationPackage.zip

Date of Submission:

Friday, August 23, 2019

Date of Decision:

Friday, September 20, 2019

Nanopublication URLs:

Decision:

Solicited Reviews:

Review #1 submitted on 28/Aug/2019

By Viktoria Spaiser ORCID logo

https://orcid.org/0000-0002-5892-245X

Review Details

Reviewer has chosen not to be Anonymous

Overall Impression: Good
Suggested Decision: Accept
Technical Quality of the paper: Good
Presentation: Good
Reviewer`s confidence: Medium
Significance: Moderate significance
Background: Comprehensive
Novelty: Unable to judge
Data availability: All used and produced data (if any) are FAIR and openly available in established data repositories
Length of the manuscript: The length of this manuscript is about right

Summary of paper in a few sentences (summary of changes and improvements for second round reviews):

The paper tests four models (Cox Proportional Hazard Model, Negative Binomial Model, Random Forest Model, XRBoost Model) to predict mobile phone applications user engagement. The goal is to provide evidence that it is possible to predict user engagement fairly accurately, though the wider goal is to facilitate intervention to re-engage users or increase user engagement. The models are tested using data from user engagement with a waste recycling app. The paper shows that random forest and XGBoost models are best suited to make accurate predictions in terms of identifying engaged and disengaged users. The negative binomial model is the least accurate model in terms of predicting the number of user activities before disengagement.

Reasons to accept:

Earlier comments and raised issues have been sufficiently addressed

Reasons to reject:

No reason to reject

Nanopublication comments:

Further comments:

Review #2 submitted on 03/Sep/2019

By Lars Lischke ORCID logo

https://orcid.org/0000-0002-9650-4945

Review Details

Reviewer has chosen not to be Anonymous

Overall Impression: Weak
Suggested Decision: Undecided
Technical Quality of the paper: Average
Presentation: Average
Reviewer`s confidence: Medium
Significance: High significance
Background: Reasonable
Novelty: Limited novelty
Data availability: All used and produced data (if any) are FAIR and openly available in established data repositories
Length of the manuscript: The length of this manuscript is about right

Summary of paper in a few sentences (summary of changes and improvements for second round reviews):

In the revised version, the authors extended the motivation for this work and clarified the used feature set. Furthermore, the authors provided more details in the discussion and highlighted the limitations of the used data set.

Reasons to accept:

The general topic is still relevant for the research communities in Data Science and Human-Computer Interaction.

Reasons to reject:

I acknowledge that the authors highlight the limitations of the used data set. However, the limitations are not reflected in the narrative of the submission. In the introduction, the authors motivate their work with an eHealth scenario. However, it reminds unclear if the results can be transferred to this scenario. I still see the issue that this particular app provides only a benefit for users in a specific geographical area. Thereby the location of the user becomes an obvious feature. Beyond, naming the limitations, it would be helpful to discuss where the results can be applied.

Nanopublication comments:

Further comments:

RESPONSE TO REVIEWERS

Dear Editors and Reviewers,

We would like to thank you for the useful comments that permitted us to greatly improve the manuscript. In the current version of the work, we have seriously taken into account each comment and have improved the paper accordingly. In the attached document, we explain how we addressed the editor’s and each reviewers’ comments. For easing the reviewing work, the major changes are marked with blue text in the revised version of the manuscript.

2 Comments

Meta-Review by Editor

Submitted by Tobias Kuhn on Fri, 09/20/2019 - 01:02

Please take into account the comments of reviewer 2, regarding discussing limitations and where the results can be applied.

Jodi Schneider (http://orcid.org/0000-0002-5098-5667)

Reply to Reviewer 2

Submitted by Eduardo Barbaro on Mon, 10/07/2019 - 09:09

Dear Editors,

We have uploaded the camera-ready files as well as the reply addressing the remarks raised by Reviewer 2.

Best regards,

Eduardo Barbaro