Contributing new metrics

Metrics in REP are more complicated than those of scikit-learn.

The typical usage of metrics of scikit-learn looks like:

from sklearn.metrics import roc_auc_score
proba = estimator.predict_proba(X)
print roc_auc_score(y, proba[:, 1], sample_weight=sample_weight)

Which is fine for many metrics and many situations, but we want to turn metric in ready-to-use object, which can be passed i.e. to factory to compute classification quality for different classifiers.

#Metric-function For many metrics it is convenient to define them as functions. REP API requires them to have interface

def function(y, proba, sample_weight=None):
    # return something!

For instance, this way you can use roc_auc_score

my_metric = lambda y, proba, sample_weight=None: roc_auc_score(y, proba[:, 1], sample_weight=sample_weight)
# compute value for each estimator
report.compute_metric(my_metric)
# compute 
report.learning_curve(my_metric)

Metric-estimator

However in many situations this is insufficient. Some metrics require additional information (query_id for ranking metrics, uniform variables for uniform metrics and so on). This information may be obtained from X - data used to estimate quality.

Let's remind that (contrary to sklearn) we support inputs with named columns (via pandas.DataFrame), so metric should keep the name of columns it requires. This information is to be saved during fitting. So, our metrics turns into an estimator, which should implement the following simple interface:

class AbstractMetric(BaseEstimator):
    def fit(self, X, y, sample_weight=None):
        """
        If metrics needs some initial heavy computations,
        this can be done here.
        interface is the same as for
        """
        pass

    def __call__(self, y, proba, sample_weight):
        """
        Compute value of metrics
        :param proba: numpy.array of shape [n_samples, n_classes]
            with predicted probabilities (typically returned by predict_proba)
        Events should be passed in the same order, as to method fit
        """
        pass

The information about dataset is passed during fit, before any __call__. Later __call__ may be used many times, but on the same data (so, y and sample_weight arguments are typically not used in call, since we already have any necessary information).

This strategy allows one to make some heavy precomputations during fit and reuse this information later, which is useful when comparing many classifiers, and specially useful when building learning curves.

Example

# raw usage of metric function
from rep.report.metrics import RocAuc
m = RocAuc().fit(X, y, sample_weight)
print m(y, classifier.predict_proba(), sample_weight=sample_weight) 

When used inside report:

report.learning_curve(RocAuc())
report.learning_curve(LogLoss())
# passing name of column with query_ids, this columns will be saved during fitting 
report.learning_curve(NDCG(query_column='query_id'))
# metric from hep_ml requires additional variables to compute non-uniformity
report.learning_curve(BinBasedCVM(uniform_variables=['mass']))