By Margherita Martorana https://orcid.org/0000-0001-8004-0464

Review Details

---

Reviewer has chosen not to be Anonymous

Summary of paper in a few sentences:

This paper aims to describe a new tool for de-identification of PII in specific types of data files (e.g. DDP), in the context of Instagram as data provider. The paper introduce this new software explaining the benefit that could arise from it for new researchers, and I found that overall the content of this manuscript is indeed very interesting. Nevertheless, there are various grammar errors and I found the paper quite hard to read. I think that multiple sentences are long, and difficult to understand. Moreover, I think that the sections "Evaluation" and "Results" are in need of further work: some results are presented in the "evaluation" section, and I found the "result" to be lacking of some quantitative results and statistics.

Reasons to accept:

The software presented has clear benefits in the field of data science, and I believe could be used by various researchers.

Reasons to reject:

Overall, the manuscript is hard to read. I believe that it needs some more work in the format and syntax.

Nanopublication comments:

Further comments:

Reviewer has chosen to be Anonymous

Summary of paper in a few sentences:

The paper presents a pice of software for de-identifying data download packages. The proposed solution combines a number of heuristics, blurring faces in pictures, replaces usernames/phone numbers etc. by pseudonyms (to preserve some utility), etc.

Reasons to accept:

Useful piece of software, basic but ok ideas.

Reasons to reject:

Not really research. Very basic. Limited evaluation

Nanopublication comments:

Further comments:

Overall, I enjoyed reading the paper and I find it to be a nice contribution to the community.

My main concerns with the paper are the following:
- (automatically) redacting documents/data is not a new problem. No related work on this is provided. In a slightly different context, solutions for detecting/removing sensitive info about to be sent on a network were proposed in the past.
- the utility aspects are not covered. Are the resulting DDP still useful. To assess this, the authors should explain what data scientist typically need from DDP and why they collect them in the first place. Also, the general pipeline for collecting DDP is not clear (users do it and then pass it to the researchers I guess; it should be explained that the de-identification would be done on the user side). Also, it is not clear how the utility is preserved for studies that focus on the *interaction* between users. It should be made clearer how interaction/relation information is preserved
- the privacy goals are not clear. whose privacy should be protected? that of the DDP owner or that of the mentioned individuals? If it's the owner, I'm afraid a simple Google (image) search would still re-identify the original data (if it's public) from the redacted data. For instance, a google image search fed with the picture with blurred face would probably return the original image
- Sections 2.1 and 2.2 lack references to (official) documentation
- The dataset is relatively small. Also, except for some data, the annotation was done by a single annotator; this is not very robust.
- Not sure Section 4.4 is needed for the audience of a journal on Data Science.

Reviewer has chosen to be Anonymous

Summary of paper in a few sentences:

The submitted manuscript presents a new tool for anonymizing data download packages (DDPs) released by online service providers (data controllers) in the context of the GDPR. The main motivation behind this tool is to enable researchers to make use of these DDPs for scientific purposes. The provided tool is tested with 11 participants creating fake Instagram profiles and actively using them for about a week.

Reasons to accept:

- The work is overall well presented and detailed, with a clear motivation and a sound approach
- The empirical results of the improved script are excellent
- Both the scripts and dataset are made open source

Reasons to reject:

- Limited fit with the journal scope
- Missing relevant related work
- Tool tailored to a specific application (instagram)
- The method used to hide faces in images/videos previously shown to be prone to re-identification attacks

Nanopublication comments:

Further comments:

I overall liked reading this paper which targets a timely problem with DDP data sharing. The results are overall very good, especially those of the improved script, which is unfortunately specific to one application, namely instagram (but there is a clear trade-off between application specificity and accuracy of the de-identification). Only the results of face de-identification in videos still need to be improved.

On the downside, it is unclear how the submitted manuscript relates to data science and if the proposed de-identification method relates to the journal's aims and scope.

Second, the authors missed several highly related work/software that propose anonymization tools for research data:
https://arx.deidentifier.org
https://amnesia.openaire.eu
https://cran.r-project.org/web/packages/sdcMicro/index.html
I encourage the authors to cite them and position their work with respect to them.

Third, as mentioned already, the tool, in particular the improved script, is tailored to a specific application, which limits its scope and impact. It would be good to discuss how the proposed improvements could also apply in other contexts, such as other social networks or applications.

Fourth, the method used for de-identifying faces in media (photos and videos), blurring, has been shown to be prone to re-identification attacks (using deep learning) by McPherson et al.: https://arxiv.org/abs/1609.00408. As a consequence, I would use more robust methods for de-identifying faces in images and videos. This could be a key novel contribution of the paper, which currently lacks strong technical contributions.

Besides, I encouraged the authors to proof-read their paper and correct the numerous typos (incl. in subsections' titles).