deephyper.hpo.CBO

deephyper.hpo.CBO#

class deephyper.hpo.CBO(problem, random_state: int | None = None, log_dir: str = '.', verbose: int = 0, stopper: Stopper | None = None, checkpoint_history_to_csv: bool = True, solution_selection: Literal['argmax_obs', 'argmax_est'] | SolutionSelection | None = None, checkpoint_restart: bool = False, surrogate_model='ET', surrogate_model_kwargs: SurrogateModelKwargs | dict | None = None, acq_func: str = 'UCBd', acq_func_kwargs: AcqFuncKwargs | dict | None = None, acq_optimizer: str = 'mixedga', acq_optimizer_kwargs: AcqOptimizerKwargs | dict | None = None, multi_point_strategy: str = 'cl_max', n_initial_points: int | None = None, initial_point_generator: str = 'random', initial_points: List[Dict[str, Any]] | None = None, moo_lower_bounds=None, moo_scalarization_strategy: str = 'Chebyshev', moo_scalarization_weight=None, objective_scaler='minmax')[source]#

Bases: Search

Centralized Bayesian Optimisation Search.

It follows a manager-workers architecture where the manager runs the Bayesian optimization loop and workers execute parallel evaluations of the black-box function.

Single-Objective	Multi-Objectives	Failures
✅	✅	✅

Example Usage:

>>> search = CBO(problem)
>>> results = search.search(evaluator, max_evals=100, timeout=120)

Parameters:

problem (HpProblem) – Hyperparameter problem describing the search space to explore.
random_state (int, optional) – Random seed. Defaults to None.
log_dir (str, optional) – Log directory where search’s results are saved. Defaults to ".".
verbose (int, optional) – Indicate the verbosity level of the search. Defaults to 0.
stopper (Stopper, optional) – a stopper to leverage multi-fidelity when evaluating the function. Defaults to None which does not use any stopper.
checkpoint_history_to_csv (bool, optional) – wether the results from progressively collected evaluations should be checkpointed regularly to disc as a csv. Defaults to True.
solution_selection (Literal["argmax_obs", "argmax_est"] | SolutionSelection, optional) – the solution selection strategy. It can be a string where "argmax_obs" would select the argmax of observed objective values, and "argmax_est" would select the argmax of estimated objective values (through a predictive model).
surrogate_model (str | sklearn.base.RegressorMixin, optional) –
Surrogate model used by the Bayesian optimization. Can be a value in ["RF", "GP", "ET", "GBRT", "DUMMY"] or a sklearn regressor. Defaults to "ET".
- "ET" :
  is for Extremely Randomized Trees which is the best compromise between speed and quality when performing a lot of parallel evaluations, i.e., reaching more than hundreds of evaluations.
- "GP" :
  is for Gaussian-Process which is the best choice when maximizing the quality of iteration but quickly slow down when reaching hundreds of evaluations, also it does not support constrained search space.
- "RF" :
  is for Random-Forest, slower than extremely randomized trees but with better mean estimate and worse epistemic uncertainty quantification capabilities.
- "GBRT" :
  is for Gradient-Boosting Regression Tree, it has better mean estimate than other tree-based method worse uncertainty quantification capabilities and slower than "RF".
surrogate_model_kwargs (dict, optional) – keyword-arguments to pass to the surrogate model. Defaults to None. See the description of these arguments in the module deephyper.skopt.learning.
acq_func (str, optional) – Acquisition function used by the Bayesian optimization. Can be a value in ["UCB", "EI", "PI", "gp_hedge"]. Defaults to "UCB".
acq_func_kwargs (dict, optional) –
A dictionnary of parameters for the acquisition function:
- "kappa" (float)
  Manage the exploration/exploitation tradeoff for the "UCB" acquisition function. Defaults to 1.96 which corresponds to 95% of the confidence interval.
- "xi" (float)
  Manage the exploration/exploitation tradeoff of "EI" and "PI" acquisition function. Defaults to 0.001.
- "scheduler" (dict, callable, optional)
  a function to manage the value of kappa, xi with iterations. Defaults to None which does not use any scheduler. The periodic exponential decay scheduler can be used with scheduler={"type": "periodic-exp-decay", "period": 30, "rate": 0.1}. The scheduler can also be a callable function with signature scheduler(i, eta_0, **kwargs) where i is the current iteration, eta_0 is the initial value of [kappa, xi] and kwargs are other fixed parameters of the function. Instead of fixing the decay "rate" the final kappa or xi can be used {"type": "periodic-exp-decay", "period": 25, "kappa_final": 1.96}.
acq_optimizer (str, optional) –
Method used to minimze the acquisition function. Can be a value in ["sampling", "lbfgs", "ga", "mixedga"]. Defaults to "auto".
- "sampling"
  the optimization is performed via sampling where the number of samples is controlled by acq_optimizer_kwargs={"n_points": 10_000}.
- "lbfgs":
  the optimization is performed via gradient-descent. It is only compatible with surrogate_model="GP".
- "mixedga":
  the optimization is performed via a Mixed Genetic Algorithm. It is made for mixed-integer search space (with continuous, discrete and categorical variables). It can increase the cost of BO iterations. In order to amortize this cost we can use acq_optimizer_kwargs={"acq_optimizer_freq": 2} with a value > 1.
- "ga":
  the optimization is performed via a continuous Genetic Algorithm. It is made for surrogate models without gradients (e.g., trees, forest) and for search space that contains mostly continuous variables. Its cost can also be amortized using acq_optimizer_freq > 1.
acq_optimizer_kwargs (dict, optional) –
A dictionnary of parameters for the acquisition function optimizer:
- "acq_optimizer_freq" (int)
  Frequency of optimization calls for the acquisition function. Defaults to 10, using optimizer every 10 surrogate model updates.
- "n_points" (int)
  The number of configurations sampled from the search space to infer each batch of new evaluated configurations.
- "filter_duplicated" (bool)
  Force the optimizer to sample unique points until the search space is “exhausted” in the sens that no new unique points can be found given the sampling size n_points. Defaults to True.
- "n_jobs" (int)
  Number of parallel processes used when possible. Defaults to 1.
- "filter_failures" (str)
  Replace objective of failed configurations by "min" or "mean". If "ignore" is passed then failed configurations will be filtered-out and not passed to the surrogate model. For multiple objectives, failure of any single objective will lead to treating that configuration as failed and each of these multiple objective will be replaced by their individual "min" or "mean" of past configurations. Defaults to "min" to replace failed configurations objectives by the running min of all objectives.
- "max_total_failures" (int)
  Maximum number of failed configurations (i.e., returning “F” as objective value) allowed for the entire search when filter_failures is not equal to "ignore". If set to -1 it allows for infinite number of failed configurations. Defaults to 100.
multi_point_strategy (str, optional) – Definition of the constant value use for the Liar strategy. Can be a value in ["cl_min", "cl_mean", "cl_max", "qUCB", "qUCBd"]. All "cl_..." strategies follow the constant-liar scheme, where if \(N\) new points are requested, the surrogate model is re-fitted \(N-1\) times with lies (respectively, the minimum, mean and maximum objective found so far; for multiple objectives, these are the minimum, mean and maximum of the individual objectives) to infer the acquisition function. Constant-Liar strategy have poor scalability because of this repeated re-fitting. The "qUCB" strategy is much more efficient by sampling a new \(\kappa\) value for each new requested point without re-fitting the model.
n_initial_points (int, optional) – Number of collected objectives required before fitting the surrogate-model. Defaults to None that will use 2 * N + 1 where N is the number of parameters in the problem.
initial_point_generator (str, optional) – Sets an initial points generator. Can be either ["random", "sobol", "halton", "hammersly", "lhs", "grid"]. Defaults to "random".
initial_points (List[Dict], optional) – A list of initial points to evaluate where each point is a dictionnary where keys are names of hyperparameters and values their corresponding choice. Defaults to None for them to be generated randomly from the search space.
moo_lower_bounds (list, optional) – List of lower bounds on the interesting range of objective values. Must be the same length as the number of obejctives. Defaults to None, i.e., no bounds. Can bound only a single objective by providing None for all other values. For example, moo_lower_bounds=[None, 0.5, None] will explore all tradeoffs for the objectives at index 0 and 2, but only consider scores for objective 1 that exceed 0.5.
moo_scalarization_strategy (str, optional) – Scalarization strategy used in multiobjective optimization. Can be a value in ["Linear", "Chebyshev", "AugChebyshev", "PBI", "Quadratic"]. Defaults to "Chebyshev". Typically, randomized methods should be used to capture entire Pareto front, unless there is a known target solution a priori. Additional details on each scalarization can be found in deephyper.skopt.moo.
moo_scalarization_weight (list, optional) – Scalarization weights to be used in multiobjective optimization with length equal to the number of objective functions. Defaults to None for randomized weights. Only set if you want to fix the scalarization weights for a multiobjective HPS.
scheduler (dict, callable, optional) – a function to manage the value of kappa, xi with iterations. Defaults to None which does not use any scheduler. The periodic exponential decay scheduler can be used with scheduler={"type": "periodic-exp-decay", "period": 30, "rate": 0.1}. The scheduler can also be a callable function with signature scheduler(i, eta_0, **kwargs) where i is the current iteration, eta_0 is the initial value of [kappa, xi] and kwargs are other fixed parameters of the function. Instead of fixing the decay "rate" the final kappa or xi can be used {"type": "periodic-exp-decay", "period": 25, "kappa_final": 1.96}.
objective_scaler (str, optional) – a way to map the objective space to some other support for example to normalize it. Defaults to "auto" which automatically set it to “identity” for any surrogate model except “RF” which will use “quantile-uniform”.

Methods

`ask`	Ask the search for new configurations to evaluate.
`check_evaluator`	Check if the input is a callable, an evaluator or else.
`dump_jobs_done_to_csv`	Dump jobs completed to CSV in log_dir.
`fit_generative_model`	Fits a generative model for sampling during BO.
`fit_search_space`	Apply prior-guided transfer learning based on a DataFrame of results.
`fit_surrogate`	Fit the surrogate model of the search from a checkpointed Dataframe.
`get_params`	Get parameters used for the search object.
`reload_checkpoint`
`save_params`	Save the search parameters to a JSON file in the log folder.
`search`	Execute the search algorithm.
`tell`	Tell the search the results of the evaluations.

Attributes

search_id

The identifier of the search used by the evaluator.

ask(n: int = 1) → List[Dict]#

Ask the search for new configurations to evaluate.

Parameters:: n (int, optional) – The number of configurations to ask. Defaults to 1.
Returns:: a list of hyperparameter configurations to evaluate.
Return type:: List[Dict]

check_evaluator(evaluator)#

Check if the input is a callable, an evaluator or else.

dump_jobs_done_to_csv(flush: bool = False)#

Dump jobs completed to CSV in log_dir.

Parameters:: flush (bool, optional) – Force the dumping if set to True. Defaults to False.

fit_generative_model(df: str | DataFrame, q: float = 0.9)[source]#

Fits a generative model for sampling during BO.

Learn the distribution of hyperparameters for the top-(1-q)x100% configurations and sample from this distribution. It can be used for transfer learning. For multiobjective problems, this function computes the top-(1-q)x100% configurations in terms of their ranking with respect to pareto efficiency: all points on the first non-dominated pareto front have rank 1 and in general, points on the k’th non-dominated front have rank k.

Example Usage:

>>> search = CBO(problem)
>>> search.fit_surrogate("results.csv")
>>> search.search(evaluator, max_evals=100)

Parameters:

df (str | DataFrame) – a dataframe or path to CSV from a previous search.
q (float) – the quantile defined the set of top configurations used to bias the search. Defaults to 0.90 which select the top-10% configurations from df.

Returns:

the generative model.

Return type:

model

fit_search_space(df: str | DataFrame, fac_numerical: float = 0.125, fac_categorical: int = 10)[source]#

Apply prior-guided transfer learning based on a DataFrame of results.

Example Usage:

>>> search = CBO(problem)
>>> search.fit_surrogate("results.csv")
>>> search.search(evaluator, max_evals=100)

Parameters:

df (str | DataFrame) – a checkpoint from a previous search.
fac_numerical (float) – the factor used to compute the sigma of a truncated normal distribution based on sigma = max(1.0, (upper - lower) * fac_numerical). A small large factor increase exploration while a small factor increase exploitation around the best-configuration from the df parameter.
fac_categorical (float) – the weight given to a categorical feature part of the best configuration. A large weight > 1 increase exploitation while a small factor close to 1 increase exploration.

fit_surrogate(df: str | DataFrame)[source]#

Fit the surrogate model of the search from a checkpointed Dataframe.

Parameters:: df (str|DataFrame) – a checkpoint from a previous search.

Example Usage:

>>> search = CBO(problem)
>>> search.fit_surrogate("results.csv")
>>> search.search(evaluator, max_evals=100)

get_params() → dict[str, Any]#

Get parameters used for the search object.

Returns:: A dictionary of the search parameters.

save_params(filename: str = 'params.json')#

Save the search parameters to a JSON file in the log folder.

Parameters:: filename – Name of JSON file where search parameters are saved. Default is params.json.

search(evaluator, max_evals: int = -1, timeout: int | float | None = None, max_evals_strict: bool = False) → DataFrame#

Execute the search algorithm.

Parameters:

evaluator – object describing the evaluation process.
max_evals (int, optional) – The maximum number of evaluations of the run function to perform before stopping the search. Defaults to -1, will run indefinitely.
timeout (int, optional) – The time budget (in seconds) of the search before stopping. Defaults to None, will not impose a time budget.
max_evals_strict (bool, optional) – If True the search will not spawn more than max_evals jobs. Defaults to False.

Returns:

A pandas DataFrame containing the evaluations performed or None if the

search could not evaluate any configuration.

This DataFrame contains the following columns: - p:HYPERPARAMETER_NAME: for each hyperparameter of the problem. - objective: for single objective optimization. - objective_0, objective_1, …: for multi-objective optimization. - job_id: the identifier of the job. - job_status: the status of the job at the end of the search. - m:METADATA_NAME: for each metadata of the problem. Some metadata are always

present like m:timestamp_submit and m:timestamp_gather which are the timestamps of the submission and gathering of the job.

Return type:

pd.DataFrame

property search_id#: The identifier of the search used by the evaluator.

Tell the search the results of the evaluations.

Parameters:: results (list[tuple[dict[str, Optional[str | int | float]], str | int | float]]) – a dictionary containing the results of the evaluations.

deephyper.hpo.CBO

Contents

deephyper.hpo.CBO#