How does Khiops handle datasets with imbalanced classes?

[lucaurelien]

This discussion is based on a question received via our contact form:

“I’m contacting you about Khiops for a binary classification question with very unbalanced classes.
What precautions should be taken when subsampling the training sample supplied to Khiops?
In this case, the ratio between classes is very different between the train and test samples.
What impact does this have on the calculation of scores and performance for the test sample?”

[lucaurelien]

Khiops is robust to class imbalance, so rebalancing techniques are generally not required.

However, class imbalance often necessitates collecting large amounts of data to gather sufficient information about the minority class. This can result in very large datasets and significantly longer training times, potentially exceeding reasonable limits. In such cases, rebalancing the dataset can be beneficial.

The standard approach is to retain all examples from the minority class and undersample the majority class. This strategy reduces computation time while preserving critical information. Never oversample the minority class, as duplicate instances lead to significant overfitting.

Best practices for rebalancing:

• Retain all individuals in the minority class.
• Undersample the majority class only as much as necessary to fit within the available computational resources, ensuring sufficient information is retained for training. In most cases, having up to 100 times more examples than the minority class is sufficient.

Impact on model performance and scores:

• Score ranking: The ranking of scores (used for ranking or thresholding) on a non-rebalanced test set remains consistent. Metrics such as ROC curves and AUC are minimally impacted.
• Predicted probabilities are completely skewed and unusable because the training set has an artificially higher representation of the minority class. This is typically not an issue, as most applications prioritize score rankings over probabilities.
• Calibration: If accurate probability estimates are required, apply a calibration method to adjust the predicted probabilities.