Review Details

Reviewer has chosen to be *Anonymous*

**Overall Impression:** Good

**Suggested Decision: ** Accept

**Technical Quality of the paper:** Good

**Presentation:** Good

**Reviewer`s confidence:** Medium

**Significance:** Low significance

**Background:** Comprehensive

**Novelty:** Limited novelty

**Data availability:** All used and produced data (if any) are FAIR and openly available in established data repositories

**Length of the manuscript:** The length of this manuscript is about right

**Summary of paper in a few sentences: **

The paper introduces the Unstable Population Indicator (UPI) as a measure for quantifying data drift, a critical aspect for models that are trained on different samples than those encountered during production. UPI is defined as a flexible and robust implementation of Jeffrey's divergence, a symmetric and discretized version of the Kullback-Leibler divergence, designed to handle various data types, including continuous, discrete, ordinal, and nominal. It answers the problem of bins with zero count by adding a small quantity to each bin. The authors highlight the importance of measuring data drift in both target and feature variables, emphasizing that the choice of a cut-off value for distinguishing stable and unstable populations should be case-dependent. Numerical experiments demonstrate UPI's effectiveness in controlled scenarios, and the paper provides a Python package for practical implementation, offering improved flexibility and performance over existing measures.

**Reasons to accept: **

The paper provides a solution to the "zero count bin" problem arising in the PSI calculation. Although there may be other ways to resolve this issue, the paper provides an alternative to PSI by adding a measured small value to each bin. This analytical solution provides a solution which is applicable for large enough population sizes.

**Reasons to reject: **

None

**Nanopublication comments: **

**Further comments: **

Here are some notes for this paper. The following items should be reviewed and addressed in order to clarify the message of the paper:

1. Note how the definition of PSI does not take into account any order in the bins, nor distances between the bins, which makes the measure equally suitable for categorical/nominal data, but interestingly this is rarely done. Besides, many posts and papers suggest the same, uninformed cut-off values for the PSI as a distinction between stable and shifted or drifted data sets. What counts as an important shift in your data should be strongly use case dependent and investigated on a per-feature basis.

The text above is not clear to me. What is the significance of order in the bins for calculation of PSI or the resulting PSI values and their interpretations?

2. What exactly the uninformed cut-off values? Do you mean .1,.25 etc.? Why are they uninformed? [11] suggests that PSI has a distribution and one can use {95th, 99th, 99.9th etc} percentiles of PSI as cut-off instead of fixed values .1,.25 etc.? That may be mentioned here.

3. In section 2.3, a discussion of PSI and its relation to $\chi^2$ tests is included however, it is not clear why there is a relationship. It may be illuminating to include that PSI has an asymptotic $\chi^2$ distribution with $B-1$ degrees of freedom where $B$ is the number of bins used in the calculation of PSI

4. $$UPI = \sum_{bins, i} (f_{1,i} − f_{0,i}) · ln(\frac{f_{1,i} + \frac{1}{ntot}}{f_{0,i} + \frac{1}{ntot}})$$

The formulation of UPI introduces an addition of a small fraction based on total count of the both dataset i.e. $ntot =$ number of observation in the base and target datasets.

The impact of this addition for small population is not discussed. No warning is provided to use UPI with small population sizes.

5. It is not clear how high dimensional UPI is connected with the rest of the paper

6. On page 4, the footnote does not list the sources explicitly

## 2 Comments

## meta-review by editor

Submitted by Tobias Kuhn on

The authors of the paper extend the Population Stability Index (PSI) to a more flexible Unstable Population Indicator (UPI), which solves the problem of a zero bin size for categorical data in PSI by adding a fraction of the population to each bin. The authors have also released a python package for UPI, which makes it easily accessible by other researchers in the field. They present a comprehensive discussion around the statistical properties of UPI and how it compares to PSI and the well-known Kullback-Leibler divergence. While the modification presented in the publication is simple, it is clearly written and comprehensively evaluated, and it cleverly preserves Jeffrey’s divergence and is a smarter approach than adding a constant to the bin size (which can bias bins unexpectedly). The addition of the python package makes it an easily usable metric.

Gargi Datta (https://orcid.org/0000-0002-1314-7824)

## Nanopublication

Submitted by Tobias Kuhn on

Please also take into account the reactions on the nanopublication, which you can find here: http://ds.kpxl.org/RAagramW3zuY74wddL8L7yWtJHXOMuCscs3HUKNb8YL50

And get in touch with us if you need help with that.

Regards,

Tobias