wrapper for gridsearchCV results

[aspitarl]

I was interested in visualizing the results of a multidimensional gridsearch with xarray and wrote a minimal example converting the results to a DataArray. I'm not sure if this is relevant to sklearn-xarray, as it seems like the package is more centered around the representation of the training data as an xarray object, but I wanted to post it in case there was interest in incorporating this in the package. Or maybe I've missed it and you already have this functionality. This example is just an initial idea and obviously doesn't cover all bases.

import numpy as np
import pandas as pd
import xarray as xr

from sklearn.datasets import load_boston
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
from sklearn.ensemble import RandomForestRegressor
from sklearn.pipeline import Pipeline

data = load_boston()

X_train, X_test, y_train, y_test = train_test_split(data['data'], data['target'])

#%%
#Form the pipeline

pipe = Pipeline([
    ('scaler', StandardScaler()),
    ('reduce_dim', PCA()),
    ('regressor', RandomForestRegressor())
])

pipe = pipe.fit(X_train, y_train)
pipe.score(X_test, y_test)

params = {
    'reduce_dim__n_components': np.arange(1,11),
    'regressor__n_estimators': [int(n) for n in np.linspace(10,100,10)],
    'regressor__min_samples_split': [2,4,8,16]
}

gridsearch = GridSearchCV(pipe, params, verbose=1)
gridsearch.fit(X_train, y_train)


#%%
# Form a datarray of the mean test score 

results = gridsearch.cv_results_
results_coords = {param : results['param_' + param].data for param in params.keys()}


# Make a multindex of the 1D result data. Probably a way to do this natively in xarray
mi = pd.MultiIndex.from_arrays(list(results_coords.values()), names = params.keys())

da = xr.DataArray(results['mean_test_score'], coords = {'mi': mi}, dims = ['mi'], name='mts', attrs=dict(long_name = 'Mean Test Score'))
da = da.unstack()
#%%

da.plot(col = 'regressor__min_samples_split')

Results:

[phausamann]

Hi @aspitarl, that sounds like a nice feature. I'm personally a big fan of using properly annotated DataArrays or Datasets for plotting.

If you would be interested in contributing, I think it would be great to have a utility function that takes a GridSearchCV instances an creates a DataArray that can be used for plotting. Something like:

from sklearn_xarray.utils import gridsearch_results_dataarray
da = gridsearch_results_dataarray(gridsearch)
da.plot(...)

not really sure about the name though...

Let me know if you would be interested!

[phausamann]

@aspitarl just realized that this would probably make more sense in the model_selection module. And wrap_gridsearch_results might be a better name for the function, but if you have a better idea go ahead.