By Stefano Bennati https://orcid.org/0000-0001-7603-8564

Review Details

---

Reviewer has chosen not to be Anonymous

Summary of paper in a few sentences:

The paper describes a technique for prediction of software bugs based on fuzzy logic, to be used in situations where the training datasets contains a small number of positive labels.

Although being interesting and relevant work, the paper raises many questions about soundness of methodology and results, thus I recommend rejection.
Specifically, there are too many unjustified statements and modeling choices. Without proper justification, i.e. citing previous work or showing convincing evidence, it is impossible to determine if the claims are true and if results are novel.

Reasons to accept:

None

Reasons to reject:

A large field of literature is neither described nor compared to the proposed method, which weakens the paper significantly (see the survey paper D’Ambros et al. 2012. Evaluating defect prediction approaches: a benchmark and an extensive comparison).
The technique uses a classic evaluation metric: the ROC curve. This choice over a more sophisticated metric, e.g. effort-aware, is not justified.
The authors state that some attributes of the dataset have been removed as they were weakly relevant, yet no proof of this has been presented.
Furthermore, the results of the simulations are not compared to any other existing technique, thus it is impossible to evaluate the quality of the proposed method.

Nanopublication comments:

Further comments:

There are several grammatical errors in the text, so a thorough proof reading of the paper by a native English speaker is recommended.

The introduction talks about how other authors used a specific dataset and lists its attributes.
Although being a useful information, it is not presented appropriately: this information should be moved to either the related work section or to the methods section, the database is not named, specific work relying on this database should be explicitly cited, the attribute names are not informative enough by themselves.

Figure 2 is a screenshot showing several plots, which are not readable and miss labels and captions. This figure should be rethought in order to convey more information with less charts.

Figure1 and the description of the layers can be improved.

Table 2 is not explained.

Figures 3 to 6 need titles, labels, legends and a baseline for comparison.

Overall the paper does not look curated and gives the impression of being put together in a hurry.

Reviewer has chosen to be Anonymous

Summary of paper in a few sentences:

This article describes a machine learning process to increase the accuracy of a binary classifier on data with very uneven distribution of classes, also referred as the unbalanced class classification problem. This topic is being thoroughly researched in the field of machine learning, meaning that there are several related articles in literature.

Reasons to accept:

No evident reason to accept. The revisions that are needed for this article demand a lot of time to implement.

Reasons to reject:

Although the topic is popular, there is no comparison against other machine learning algorithms or benchmarks. The explanation of the reasons behind the addition of a cost weighting process needs to be more precise and clear. The absence of comparison with state of the art algorithms and the lack of a detailed and comprehensive explanation of the modification to the ANFIS model, make the contribution of the paper vague. The usage of the English language in the paper is not appropriate. Several syntactic, spelling and grammar mistakes, make the text difficult to read and understand. A lot of technical terms and abbreviations remain unexplained, leaving their understanding at the discretion of the user. There are unreferenced or unexplained strong arguments, e.g.:
- Large number of inputs unnecessarily over fits the model.
Such arguments need to be referenced with several references, because there are several debates about them in modern data science, and may also be rejected for certain fields i.e. image recognition classification. No equation styling is used. The graphs are of very low quality, most of them are raster images, making it difficult to zoom and analyse. This paper, at its current state is not suitable for a journal publication. The presentation quality is very low and demands a lot of concentration, knowledge and dedication from the reader to follow the contents of the article. Improvements in plots, references and especially writing are necessary. More experiments need to be done for comparison, like the authors did in their previous ANFIS paper. If the increase in performance is significant, then the contribution of this paper will be clearer. Finally, I would like to see the evaluation for train, dev/validation and test set for this algorithm, to be able to determine whether the algorithm overfits/underfits. Other evaluation metrics can be shown as well in a table format or other chars.

Nanopublication comments:

Further comments: