Black-Box Optimization

Note

Go to the end to download the full example code.

Black-Box Optimization#

Author(s): Romain Egele, Brett Eiffert.

In this tutorial, we introduce you to the notion of black-box optimization (Wikipedia) (a.k.a., derivative-free optimization) with DeepHyper.

Black-box optimization is a field of optimization research where an objective function \(f(x) = y \in \mathbb{R}\) is optimized only based on input-output observations \(\{ (x_1,y_1), \ldots, (x_n, y_n) \}\).

Let’s start by installing DeepHyper!

%%bash
pip install deephyper

Then, we can import it and check the installed version:

import deephyper
print(deephyper.__version__)

0.9.3

Optimization Problem#

The optimization problem is based on two components:

  1. The black-box function that we want to optimize.

  2. The search space or domain of input variables over which we want to optimize.

Black-Box Function#

DeepHyper is developed to optimize black-box functions. Here, we define the function \(f(x) = - x ^ 2\) that we want to maximise (the maximum being \(f(x=0) = 0\) on \(I_x = [-10;10]\)). The black-box function f takes as input a job that follows a dictionary interface from which we retrieve the variables of interest.

def f(job):
    return -job.parameters["x"] ** 2

Search Space of Input Variables#

In this example, we have only one variable \(x\) for the black-box functin \(f\). We empirically decide to optimize this variable $x$ on the interval \(I_x = [-10;10]\). To do so we use the deephyper.hpo.HpProblem from DeepHyper and add a real hyperparameter by using a tuple of two floats.

from deephyper.hpo import HpProblem


problem = HpProblem()

# Define the variable you want to optimize
problem.add_hyperparameter((-10.0, 10.0), "x")

problem

Configuration space object:
  Hyperparameters:
    x, Type: UniformFloat, Range: [-10.0, 10.0], Default: 0.0

Evaluator Interface#

DeepHyper uses an API called deephyper.evaluator.Evaluator to distribute the computation of black-box functions and adapt to different backends (e.g., threads, processes, MPI, Ray). An Evaluator object wraps the black-box function f that we want to optimize. Then a method parameter is used to select the backend and method_kwargs defines some available options of this backend.

Hint

The method="thread" provides parallel computation only if the black-box is releasing the global interpretor lock (GIL). Therefore, if you want parallelism in Jupyter notebooks you should use the Ray evaluator (method="ray") after installing Ray with pip install ray.

It is possible to define callbacks to extend the behaviour of Evaluator each time a function-evaluation is launched or completed. In this example we use the deephyper.evaluator.callback.TqdmCallback to follow the completed evaluations and the evolution of the objective with a progress-bar.

from deephyper.evaluator import Evaluator
from deephyper.evaluator.callback import TqdmCallback


# define the evaluator to distribute the computation
evaluator = Evaluator.create(
    f,
    method="thread",
    method_kwargs={
        "num_workers": 4,
        "callbacks": [TqdmCallback()]
    },
)

print(f"Evaluator has {evaluator.num_workers} available worker{'' if evaluator.num_workers == 1 else 's'}")

Evaluator has 4 available workers

Search Algorithm#

The next step is to define the search algorithm that we want to use. Here, we choose deephyper.hpo.CBO (Centralized Bayesian Optimization) which is a sampling based Bayesian optimization strategy. This algorithm has the advantage of being asynchronous thanks to a constant liar strategy which is crutial to keep a good utilization of the resources when the number of available workers increases.

from deephyper.hpo import CBO

# define your search
search = CBO(
    problem,
    evaluator,
    acq_optimizer="ga",
)

WARNING:root:Results file already exists, it will be renamed to /Users/romainegele/Documents/DeepHyper/deephyper/examples/examples_bbo/results_20250324-091742.csv

Then, we can execute the search for a given number of iterations by using the search.search(max_evals=...). It is also possible to use the timeout parameter if one needs a specific time budget (e.g., restricted computational time in machine learning competitions, allocation time in HPC).

results = search.search(max_evals=100)

  0%|          | 0/100 [00:00<?, ?it/s]
  1%|          | 1/100 [00:00<00:00, 4563.99it/s, failures=0, objective=-19.1]
  2%|▏         | 2/100 [00:00<00:00, 3184.74it/s, failures=0, objective=-1.19]
  3%|▎         | 3/100 [00:00<00:00, 4212.56it/s, failures=0, objective=-1.19]
  4%|▍         | 4/100 [00:00<00:00, 5166.99it/s, failures=0, objective=-1.19]
  5%|▌         | 5/100 [00:00<00:18,  5.24it/s, failures=0, objective=-1.19]
  5%|▌         | 5/100 [00:00<00:18,  5.24it/s, failures=0, objective=-1.19]
  6%|▌         | 6/100 [00:00<00:17,  5.24it/s, failures=0, objective=-1.19]
  7%|▋         | 7/100 [00:00<00:17,  5.24it/s, failures=0, objective=-1.19]
  8%|▊         | 8/100 [00:00<00:17,  5.24it/s, failures=0, objective=-1.19]
  9%|▉         | 9/100 [00:01<00:18,  4.97it/s, failures=0, objective=-1.19]
  9%|▉         | 9/100 [00:01<00:18,  4.97it/s, failures=0, objective=-1.19]
 10%|█         | 10/100 [00:01<00:18,  4.97it/s, failures=0, objective=-1.19]
 11%|█         | 11/100 [00:01<00:17,  4.97it/s, failures=0, objective=-1.19]
 12%|█▏        | 12/100 [00:01<00:17,  4.97it/s, failures=0, objective=-1.19]min_impurity_decrease updated: 0.001

 13%|█▎        | 13/100 [00:02<00:18,  4.72it/s, failures=0, objective=-1.19]
 13%|█▎        | 13/100 [00:02<00:18,  4.72it/s, failures=0, objective=-0.983]
 14%|█▍        | 14/100 [00:02<00:18,  4.72it/s, failures=0, objective=-0.746]
 15%|█▌        | 15/100 [00:02<00:18,  4.72it/s, failures=0, objective=-0.746]
 16%|█▌        | 16/100 [00:02<00:17,  4.72it/s, failures=0, objective=-0.278]
 17%|█▋        | 17/100 [00:03<00:17,  4.73it/s, failures=0, objective=-0.278]
 17%|█▋        | 17/100 [00:03<00:17,  4.73it/s, failures=0, objective=-0.278]
 18%|█▊        | 18/100 [00:03<00:17,  4.73it/s, failures=0, objective=-0.278]
 19%|█▉        | 19/100 [00:03<00:17,  4.73it/s, failures=0, objective=-0.278]
 20%|██        | 20/100 [00:03<00:16,  4.73it/s, failures=0, objective=-0.278]
 21%|██        | 21/100 [00:04<00:17,  4.59it/s, failures=0, objective=-0.278]
 21%|██        | 21/100 [00:04<00:17,  4.59it/s, failures=0, objective=-0.173]
 22%|██▏       | 22/100 [00:04<00:17,  4.59it/s, failures=0, objective=-0.0164]
 23%|██▎       | 23/100 [00:04<00:16,  4.59it/s, failures=0, objective=-0.0164]
 24%|██▍       | 24/100 [00:04<00:16,  4.59it/s, failures=0, objective=-0.0164]
 25%|██▌       | 25/100 [00:05<00:16,  4.51it/s, failures=0, objective=-0.0164]
 25%|██▌       | 25/100 [00:05<00:16,  4.51it/s, failures=0, objective=-0.0164]
 26%|██▌       | 26/100 [00:05<00:16,  4.51it/s, failures=0, objective=-0.0164]
 27%|██▋       | 27/100 [00:05<00:16,  4.51it/s, failures=0, objective=-0.0164]
 28%|██▊       | 28/100 [00:05<00:15,  4.51it/s, failures=0, objective=-0.0164]
 29%|██▉       | 29/100 [00:06<00:15,  4.57it/s, failures=0, objective=-0.0164]
 29%|██▉       | 29/100 [00:06<00:15,  4.57it/s, failures=0, objective=-0.0147]
 30%|███       | 30/100 [00:06<00:15,  4.57it/s, failures=0, objective=-0.0146]
 31%|███       | 31/100 [00:06<00:15,  4.57it/s, failures=0, objective=-0.0146]
 32%|███▏      | 32/100 [00:06<00:14,  4.57it/s, failures=0, objective=-4.82e-5]
 33%|███▎      | 33/100 [00:07<00:14,  4.50it/s, failures=0, objective=-4.82e-5]
 33%|███▎      | 33/100 [00:07<00:14,  4.50it/s, failures=0, objective=-4.82e-5]
 34%|███▍      | 34/100 [00:07<00:14,  4.50it/s, failures=0, objective=-4.82e-5]
 35%|███▌      | 35/100 [00:07<00:14,  4.50it/s, failures=0, objective=-4.82e-5]
 36%|███▌      | 36/100 [00:07<00:14,  4.50it/s, failures=0, objective=-4.82e-5]
 37%|███▋      | 37/100 [00:07<00:13,  4.58it/s, failures=0, objective=-4.82e-5]
 37%|███▋      | 37/100 [00:07<00:13,  4.58it/s, failures=0, objective=-4.82e-5]
 38%|███▊      | 38/100 [00:07<00:13,  4.58it/s, failures=0, objective=-4.82e-5]
 39%|███▉      | 39/100 [00:07<00:13,  4.58it/s, failures=0, objective=-4.82e-5]
 40%|████      | 40/100 [00:07<00:13,  4.58it/s, failures=0, objective=-4.82e-5]
 41%|████      | 41/100 [00:08<00:13,  4.51it/s, failures=0, objective=-4.82e-5]
 41%|████      | 41/100 [00:08<00:13,  4.51it/s, failures=0, objective=-4.82e-5]
 42%|████▏     | 42/100 [00:08<00:12,  4.51it/s, failures=0, objective=-4.82e-5]
 43%|████▎     | 43/100 [00:08<00:12,  4.51it/s, failures=0, objective=-4.82e-5]
 44%|████▍     | 44/100 [00:08<00:12,  4.51it/s, failures=0, objective=-4.82e-5]
 45%|████▌     | 45/100 [00:09<00:12,  4.54it/s, failures=0, objective=-4.82e-5]
 45%|████▌     | 45/100 [00:09<00:12,  4.54it/s, failures=0, objective=-4.82e-5]
 46%|████▌     | 46/100 [00:09<00:11,  4.54it/s, failures=0, objective=-4.82e-5]
 47%|████▋     | 47/100 [00:09<00:11,  4.54it/s, failures=0, objective=-4.82e-5]
 48%|████▊     | 48/100 [00:09<00:11,  4.54it/s, failures=0, objective=-4.82e-5]
 49%|████▉     | 49/100 [00:10<00:11,  4.50it/s, failures=0, objective=-4.82e-5]
 49%|████▉     | 49/100 [00:10<00:11,  4.50it/s, failures=0, objective=-4.82e-5]
 50%|█████     | 50/100 [00:10<00:11,  4.50it/s, failures=0, objective=-4.82e-5]
 51%|█████     | 51/100 [00:10<00:10,  4.50it/s, failures=0, objective=-4.82e-5]
 52%|█████▏    | 52/100 [00:10<00:10,  4.50it/s, failures=0, objective=-4.82e-5]
 53%|█████▎    | 53/100 [00:11<00:10,  4.56it/s, failures=0, objective=-4.82e-5]
 53%|█████▎    | 53/100 [00:11<00:10,  4.56it/s, failures=0, objective=-4.82e-5]
 54%|█████▍    | 54/100 [00:11<00:10,  4.56it/s, failures=0, objective=-4.82e-5]
 55%|█████▌    | 55/100 [00:11<00:09,  4.56it/s, failures=0, objective=-4.82e-5]
 56%|█████▌    | 56/100 [00:11<00:09,  4.56it/s, failures=0, objective=-4.82e-5]
 57%|█████▋    | 57/100 [00:12<00:09,  4.62it/s, failures=0, objective=-4.82e-5]
 57%|█████▋    | 57/100 [00:12<00:09,  4.62it/s, failures=0, objective=-4.82e-5]
 58%|█████▊    | 58/100 [00:12<00:09,  4.62it/s, failures=0, objective=-4.82e-5]
 59%|█████▉    | 59/100 [00:12<00:08,  4.62it/s, failures=0, objective=-4.82e-5]
 60%|██████    | 60/100 [00:12<00:08,  4.62it/s, failures=0, objective=-4.82e-5]min_impurity_decrease updated: 0.0002

 61%|██████    | 61/100 [00:13<00:08,  4.54it/s, failures=0, objective=-4.82e-5]
 61%|██████    | 61/100 [00:13<00:08,  4.54it/s, failures=0, objective=-4.82e-5]
 62%|██████▏   | 62/100 [00:13<00:08,  4.54it/s, failures=0, objective=-4.82e-5]
 63%|██████▎   | 63/100 [00:13<00:08,  4.54it/s, failures=0, objective=-4.82e-5]
 64%|██████▍   | 64/100 [00:13<00:07,  4.54it/s, failures=0, objective=-4.82e-5]
 65%|██████▌   | 65/100 [00:14<00:07,  4.60it/s, failures=0, objective=-4.82e-5]
 65%|██████▌   | 65/100 [00:14<00:07,  4.60it/s, failures=0, objective=-4.82e-5]
 66%|██████▌   | 66/100 [00:14<00:07,  4.60it/s, failures=0, objective=-4.82e-5]
 67%|██████▋   | 67/100 [00:14<00:07,  4.60it/s, failures=0, objective=-4.82e-5]
 68%|██████▊   | 68/100 [00:14<00:06,  4.60it/s, failures=0, objective=-4.82e-5]
 69%|██████▉   | 69/100 [00:15<00:06,  4.53it/s, failures=0, objective=-4.82e-5]
 69%|██████▉   | 69/100 [00:15<00:06,  4.53it/s, failures=0, objective=-1.85e-5]
 70%|███████   | 70/100 [00:15<00:06,  4.53it/s, failures=0, objective=-1.85e-5]
 71%|███████   | 71/100 [00:15<00:06,  4.53it/s, failures=0, objective=-1.85e-5]
 72%|███████▏  | 72/100 [00:15<00:06,  4.53it/s, failures=0, objective=-1.85e-5]
 73%|███████▎  | 73/100 [00:15<00:05,  4.58it/s, failures=0, objective=-1.85e-5]
 73%|███████▎  | 73/100 [00:15<00:05,  4.58it/s, failures=0, objective=-1.85e-5]
 74%|███████▍  | 74/100 [00:15<00:05,  4.58it/s, failures=0, objective=-1.85e-5]
 75%|███████▌  | 75/100 [00:15<00:05,  4.58it/s, failures=0, objective=-1.85e-5]
 76%|███████▌  | 76/100 [00:15<00:05,  4.58it/s, failures=0, objective=-1.85e-5]
 77%|███████▋  | 77/100 [00:16<00:05,  4.50it/s, failures=0, objective=-1.85e-5]
 77%|███████▋  | 77/100 [00:16<00:05,  4.50it/s, failures=0, objective=-1.85e-5]
 78%|███████▊  | 78/100 [00:16<00:04,  4.50it/s, failures=0, objective=-1.85e-5]
 79%|███████▉  | 79/100 [00:16<00:04,  4.50it/s, failures=0, objective=-1.85e-5]
 80%|████████  | 80/100 [00:16<00:04,  4.50it/s, failures=0, objective=-1.85e-5]
 81%|████████  | 81/100 [00:17<00:04,  4.57it/s, failures=0, objective=-1.85e-5]
 81%|████████  | 81/100 [00:17<00:04,  4.57it/s, failures=0, objective=-1.85e-5]
 82%|████████▏ | 82/100 [00:17<00:03,  4.57it/s, failures=0, objective=-1.85e-5]
 83%|████████▎ | 83/100 [00:17<00:03,  4.57it/s, failures=0, objective=-1.85e-5]
 84%|████████▍ | 84/100 [00:17<00:03,  4.57it/s, failures=0, objective=-1.85e-5]min_impurity_decrease updated: 4e-05

 85%|████████▌ | 85/100 [00:18<00:03,  4.49it/s, failures=0, objective=-1.85e-5]
 85%|████████▌ | 85/100 [00:18<00:03,  4.49it/s, failures=0, objective=-1.85e-5]
 86%|████████▌ | 86/100 [00:18<00:03,  4.49it/s, failures=0, objective=-1.85e-5]
 87%|████████▋ | 87/100 [00:18<00:02,  4.49it/s, failures=0, objective=-1.85e-5]
 88%|████████▊ | 88/100 [00:18<00:02,  4.49it/s, failures=0, objective=-1.85e-5]
 89%|████████▉ | 89/100 [00:19<00:02,  4.57it/s, failures=0, objective=-1.85e-5]
 89%|████████▉ | 89/100 [00:19<00:02,  4.57it/s, failures=0, objective=-1.85e-5]
 90%|█████████ | 90/100 [00:19<00:02,  4.57it/s, failures=0, objective=-1.85e-5]
 91%|█████████ | 91/100 [00:19<00:01,  4.57it/s, failures=0, objective=-1.85e-5]
 92%|█████████▏| 92/100 [00:19<00:01,  4.57it/s, failures=0, objective=-1.85e-5]
 93%|█████████▎| 93/100 [00:20<00:01,  4.50it/s, failures=0, objective=-1.85e-5]
 93%|█████████▎| 93/100 [00:20<00:01,  4.50it/s, failures=0, objective=-1.85e-5]
 94%|█████████▍| 94/100 [00:20<00:01,  4.50it/s, failures=0, objective=-1.85e-5]
 95%|█████████▌| 95/100 [00:20<00:01,  4.50it/s, failures=0, objective=-1.85e-5]
 96%|█████████▌| 96/100 [00:20<00:00,  4.50it/s, failures=0, objective=-1.85e-5]
 97%|█████████▋| 97/100 [00:21<00:00,  4.57it/s, failures=0, objective=-1.85e-5]
 97%|█████████▋| 97/100 [00:21<00:00,  4.57it/s, failures=0, objective=-1.85e-5]
 98%|█████████▊| 98/100 [00:21<00:00,  4.57it/s, failures=0, objective=-1.85e-5]
 99%|█████████▉| 99/100 [00:21<00:00,  4.57it/s, failures=0, objective=-1.85e-5]
100%|██████████| 100/100 [00:21<00:00,  4.57it/s, failures=0, objective=-1.85e-5]

Finally, let us visualize the results. The search(...) returns a DataFrame also saved locally under results.csv (in case of crash we don’t want to lose the possibly expensive evaluations already performed).

The DataFrame contains as columns:

  1. the optimized hyperparameters: such as \(x\) with name p:x.

  2. the objective maximised which directly match the results of the \(f\) function in our example.

  3. the job_id of each evaluated function (increased incrementally following the order of created evaluations).

  4. the time of creation/collection of each task timestamp_submit and timestamp_gather respectively (in secondes, since the creation of the Evaluator).

results
p:x objective job_id job_status m:timestamp_submit m:timestamp_gather
0 -4.375545 -19.145392 0 DONE 0.008249 0.008734
1 1.090036 -1.188179 1 DONE 0.008265 0.015338
2 9.009590 -81.172715 2 DONE 0.008272 0.015476
3 8.568972 -73.427281 3 DONE 0.008276 0.015541
4 2.919459 -8.523240 5 DONE 0.969010 0.969412
... ... ... ... ... ... ...
95 0.120686 -0.014565 93 DONE 20.369269 20.369850
96 0.127925 -0.016365 99 DONE 21.212798 21.213076
97 0.135909 -0.018471 96 DONE 21.212777 21.213264
98 0.128395 -0.016485 98 DONE 21.212795 21.213325
99 0.134130 -0.017991 97 DONE 21.212791 21.213377

100 rows × 6 columns



To get the parameters at the observed maximum value we can use the deephyper.analysis.hpo.parameters_at_max():

from deephyper.analysis.hpo import parameters_at_max


parameters, objective = parameters_at_max(results)
print("\nOptimum values")
print("x:", parameters["x"])
print("objective:", objective)

Optimum values
x: -0.0042995960267848
objective: -1.848652599354423e-05

We can also plot the evolution of the objective to verify that we converge correctly toward \(0\).

import matplotlib.pyplot as plt
from deephyper.analysis.hpo import plot_search_trajectory_single_objective_hpo


WIDTH_PLOTS = 8
HEIGHT_PLOTS = WIDTH_PLOTS / 1.618

fig, ax = plt.subplots(figsize=(WIDTH_PLOTS, HEIGHT_PLOTS))
plot_search_trajectory_single_objective_hpo(results, mode="min", ax=ax)
_ = plt.title("Search Trajectory")
_ = plt.yscale("log")
Search Trajectory

Total running time of the script: (0 minutes 23.380 seconds)

Gallery generated by Sphinx-Gallery

Contents