Run BaseSearchCV instances which are not scikit-learn builtins

[NicolasHug]

Description

I have a custom implementation of Successive Halving scikit-learn/scikit-learn#13900 which inherits from BaseSearchCV. When I use run_model_on_task I'm getting the following warning:

Warning! Using subclass BaseSearchCV other than {GridSearchCV, RandomizedSearchCV}. Should implement param check.

I tried to look at the code but couldn't find what those param checks were. It seems that none of the metric are computed on that estimator.

Any help appreciated. Thanks!

Versions

OpenML 0.11.0dev

[janvanrijn]

Hi Nicolas, this basically means that we could not find the standardized way where the param_grid was stored, so we review this on a case by case basis.

We identified them for Grid Search and Random Search. If you point me towards the attribute name for where the param grid is stored for SH, it should be a one liner to add this to the OpenML lib.

[NicolasHug]

Thanks Jan,

The code currently is

            if isinstance(model, sklearn.model_selection.GridSearchCV):
                param_distributions = model.param_grid
            elif isinstance(model, sklearn.model_selection.RandomizedSearchCV):
                param_distributions = model.param_distributions
            else:
                if hasattr(model, 'param_distributions'):
                    param_distributions = model.param_distributions
                else:
                    raise AttributeError('Using subclass BaseSearchCV other than '
                                         '{GridSearchCV, RandomizedSearchCV}. '
                                         'Could not find attribute '
                                         'param_distributions.')
                print('Warning! Using subclass BaseSearchCV other than '
                      '{GridSearchCV, RandomizedSearchCV}. '
                      'Should implement param check. ')

which means that:

1. you can automatically find param_distribution, but not param_grid
2. the warning is raised iff the object isn't an instance of GridSearchCV or RandomizedSearchCV (regardless of whether the parameter was found)

To fix 1. I think we could replace

                if hasattr(model, 'param_distributions'):
                    param_distributions = model.param_distributions

by e.g.

param_distributions = getattr(model, 'param_distributions', None) or getattr(model, 'param_grid', None)
if param_distribution is None:
	... error

But my problem is more related to 2.: I'm getting a warning and I don't know how I should fix it ;)