Black-Box Optimization

Black-Box Optimization#

Author(s): Romain Egele, Brett Eiffert.

In this tutorial, we introduce you to the notion of black-box optimization (Wikipedia) (a.k.a., derivative-free optimization) with DeepHyper.

Black-box optimization is a field of optimization research where an objective function $f(x) = y \in \mathbb{R}$ is optimized only based on input-output observations $\{ (x_1,y_1), \ldots, (x_n, y_n) \}$.

Let’s start by installing DeepHyper!

%%bash
pip install deephyper

Then, we can import it and check the installed version:

import deephyper
print(deephyper.__version__)

0.10.2

Optimization Problem#

The optimization problem is based on two components:

The black-box function that we want to optimize.
The search space or domain of input variables over which we want to optimize.

Black-Box Function#

DeepHyper is developed to optimize black-box functions. Here, we define the function $f(x) = - x ^ 2$ that we want to maximise (the maximum being $f(x=0) = 0$ on $I_x = [-10;10]$). The black-box function f takes as input a job that follows a dictionary interface from which we retrieve the variables of interest.

def f(job):
    return -job.parameters["x"] ** 2

Search Space of Input Variables#

In this example, we have only one variable $x$ for the black-box functin $f$. We empirically decide to optimize this variable $x$ on the interval $I_x = [-10;10]$. To do so we use the deephyper.hpo.HpProblem from DeepHyper and add a real hyperparameter by using a tuple of two floats.

from deephyper.hpo import HpProblem

problem = HpProblem()

# Define the variable you want to optimize
problem.add_hyperparameter((-10.0, 10.0), "x")

problem

Configuration space object:
  Hyperparameters:
    x, Type: UniformFloat, Range: [-10.0, 10.0], Default: 0.0

Evaluator Interface#

DeepHyper uses an API called deephyper.evaluator.Evaluator to distribute the computation of black-box functions and adapt to different backends (e.g., threads, processes, MPI, Ray). An Evaluator object wraps the black-box function f that we want to optimize. Then a method parameter is used to select the backend and method_kwargs defines some available options of this backend.

Hint

The method="thread" provides parallel computation only if the black-box is releasing the global interpretor lock (GIL). Therefore, if you want parallelism in Jupyter notebooks you should use the Ray evaluator (method="ray") after installing Ray with pip install ray.

It is possible to define callbacks to extend the behaviour of Evaluator each time a function-evaluation is launched or completed. In this example we use the deephyper.evaluator.callback.TqdmCallback to follow the completed evaluations and the evolution of the objective with a progress-bar.

from deephyper.evaluator import Evaluator
from deephyper.evaluator.callback import TqdmCallback


# define the evaluator to distribute the computation
evaluator = Evaluator.create(
    f,
    method="thread",
    method_kwargs={
        "num_workers": 4,
        "callbacks": [TqdmCallback()]
    },
)

print(f"Evaluator has {evaluator.num_workers} available worker{'' if evaluator.num_workers == 1 else 's'}")

Evaluator has 4 available workers

Search Algorithm#

The next step is to define the search algorithm that we want to use. Here, we choose deephyper.hpo.CBO (Centralized Bayesian Optimization) which is a sampling based Bayesian optimization strategy. This algorithm has the advantage of being asynchronous thanks to a constant liar strategy which is crutial to keep a good utilization of the resources when the number of available workers increases.

from deephyper.hpo import CBO

# define your search
def create_search():
    search = CBO(
        problem,
        acq_optimizer="ga",
    )
    return search

Then, to execute the search we provide two interfaces.

Ask and Tell Interface#

The first interface is the classic configurations = search.ask(...) and search.tell(configurations_with_objective). In this case, you need to manage the computation of objectives yourself. This interface is more flexible. However, asynchronous parallel evaluations are not managed for you.

Here is an example:

search = create_search()
max_evals = 100
for i in range(max_evals):
    config = search.ask(1)[0]
    y = -config["x"] ** 2
    print(f"[{i=:03d}] >>> f(x={config['x']:.3f}) = {y:.3f}")
    search.tell([(config, y)])

[i=000] >>> f(x=-0.150) = -0.022
[i=001] >>> f(x=-9.819) = -96.421
[i=002] >>> f(x=3.473) = -12.065
[i=003] >>> f(x=-1.518) = -2.305
[i=004] >>> f(x=-3.703) = -13.709
[i=005] >>> f(x=-0.002) = -0.000
[i=006] >>> f(x=-0.501) = -0.251
[i=007] >>> f(x=-0.878) = -0.771
[i=008] >>> f(x=0.194) = -0.038
[i=009] >>> f(x=-0.610) = -0.372
[i=010] >>> f(x=-0.814) = -0.662
[i=011] >>> f(x=-0.195) = -0.038
[i=012] >>> f(x=-0.089) = -0.008
[i=013] >>> f(x=-0.433) = -0.188
[i=014] >>> f(x=1.639) = -2.685
[i=015] >>> f(x=-1.606) = -2.580
[i=016] >>> f(x=0.245) = -0.060
[i=017] >>> f(x=-0.028) = -0.001
[i=018] >>> f(x=-0.062) = -0.004
[i=019] >>> f(x=0.003) = -0.000
[i=020] >>> f(x=0.082) = -0.007
[i=021] >>> f(x=-0.060) = -0.004
[i=022] >>> f(x=0.091) = -0.008
[i=023] >>> f(x=0.097) = -0.009
[i=024] >>> f(x=0.394) = -0.155
[i=025] >>> f(x=0.668) = -0.446
[i=026] >>> f(x=0.085) = -0.007
[i=027] >>> f(x=0.002) = -0.000
[i=028] >>> f(x=-0.085) = -0.007
[i=029] >>> f(x=-0.099) = -0.010
[i=030] >>> f(x=-0.064) = -0.004
[i=031] >>> f(x=-0.050) = -0.003
[i=032] >>> f(x=-0.056) = -0.003
[i=033] >>> f(x=-0.017) = -0.000
[i=034] >>> f(x=0.443) = -0.196
[i=035] >>> f(x=-0.216) = -0.046
[i=036] >>> f(x=0.187) = -0.035
[i=037] >>> f(x=-0.080) = -0.006
[i=038] >>> f(x=-0.042) = -0.002
[i=039] >>> f(x=0.084) = -0.007
[i=040] >>> f(x=-0.010) = -0.000
[i=041] >>> f(x=0.021) = -0.000
[i=042] >>> f(x=0.021) = -0.000
[i=043] >>> f(x=0.022) = -0.000
[i=044] >>> f(x=-0.305) = -0.093
[i=045] >>> f(x=0.262) = -0.069
[i=046] >>> f(x=-0.007) = -0.000
[i=047] >>> f(x=0.023) = -0.001
[i=048] >>> f(x=0.031) = -0.001
[i=049] >>> f(x=0.030) = -0.001
[i=050] >>> f(x=0.026) = -0.001
[i=051] >>> f(x=0.022) = -0.000
[i=052] >>> f(x=0.016) = -0.000
[i=053] >>> f(x=0.020) = -0.000
[i=054] >>> f(x=0.287) = -0.082
[i=055] >>> f(x=-0.123) = -0.015
[i=056] >>> f(x=0.102) = -0.010
[i=057] >>> f(x=0.015) = -0.000
[i=058] >>> f(x=-0.008) = -0.000
[i=059] >>> f(x=-0.007) = -0.000
[i=060] >>> f(x=-0.006) = -0.000
[i=061] >>> f(x=-0.019) = -0.000
[i=062] >>> f(x=-0.013) = -0.000
[i=063] >>> f(x=-0.008) = -0.000
[i=064] >>> f(x=0.132) = -0.017
[i=065] >>> f(x=0.016) = -0.000
[i=066] >>> f(x=0.016) = -0.000
[i=067] >>> f(x=0.013) = -0.000
[i=068] >>> f(x=0.015) = -0.000
[i=069] >>> f(x=-0.011) = -0.000
[i=070] >>> f(x=-0.006) = -0.000
[i=071] >>> f(x=0.008) = -0.000
[i=072] >>> f(x=0.012) = -0.000
[i=073] >>> f(x=0.014) = -0.000
[i=074] >>> f(x=0.033) = -0.001
[i=075] >>> f(x=0.041) = -0.002
[i=076] >>> f(x=0.004) = -0.000
[i=077] >>> f(x=0.010) = -0.000
[i=078] >>> f(x=0.007) = -0.000
[i=079] >>> f(x=0.009) = -0.000
[i=080] >>> f(x=0.007) = -0.000
[i=081] >>> f(x=0.005) = -0.000
[i=082] >>> f(x=0.004) = -0.000
[i=083] >>> f(x=0.005) = -0.000
[i=084] >>> f(x=0.050) = -0.002
[i=085] >>> f(x=-0.010) = -0.000
[i=086] >>> f(x=0.011) = -0.000
[i=087] >>> f(x=0.000) = -0.000
[i=088] >>> f(x=-0.010) = -0.000
[i=089] >>> f(x=-0.011) = -0.000
[i=090] >>> f(x=-0.010) = -0.000
[i=091] >>> f(x=-0.009) = -0.000
[i=092] >>> f(x=0.003) = -0.000
[i=093] >>> f(x=-0.008) = -0.000
[i=094] >>> f(x=0.063) = -0.004
[i=095] >>> f(x=-0.005) = -0.000
[i=096] >>> f(x=-0.007) = -0.000
[i=097] >>> f(x=0.003) = -0.000
[i=098] >>> f(x=-0.001) = -0.000
[i=099] >>> f(x=-0.001) = -0.000

Search Interface#

The second interface is results = search.search(evaluator, ...) that is binded to the evaluator and manages the loop for asynchronous parallel evaluations for you. We can execute the search for a given number of iterations by using the search.search(evaluator, max_evals=...). It is also possible to use the timeout parameter if one needs a specific time budget (e.g., restricted computational time in machine learning competitions, allocation time in HPC).

search = create_search()
results = search.search(evaluator, max_evals)

  0%|          | 0/100 [00:00<?, ?it/s]
  1%|          | 1/100 [00:00<00:00, 5370.43it/s, failures=0, objective=-0.51]
  2%|▏         | 2/100 [00:00<00:00, 4888.47it/s, failures=0, objective=-0.51]
  3%|▎         | 3/100 [00:00<00:00, 5745.62it/s, failures=0, objective=-0.51]
  4%|▍         | 4/100 [00:00<00:00, 6708.20it/s, failures=0, objective=-0.51]
  5%|▌         | 5/100 [00:00<00:16,  5.72it/s, failures=0, objective=-0.51]
  5%|▌         | 5/100 [00:00<00:16,  5.72it/s, failures=0, objective=-0.51]
  6%|▌         | 6/100 [00:00<00:16,  5.72it/s, failures=0, objective=-0.51]
  7%|▋         | 7/100 [00:00<00:16,  5.72it/s, failures=0, objective=-0.51]
  8%|▊         | 8/100 [00:00<00:16,  5.72it/s, failures=0, objective=-0.51]
  9%|▉         | 9/100 [00:01<00:17,  5.34it/s, failures=0, objective=-0.51]
  9%|▉         | 9/100 [00:01<00:17,  5.34it/s, failures=0, objective=-0.51]
 10%|█         | 10/100 [00:01<00:16,  5.34it/s, failures=0, objective=-0.51]
 11%|█         | 11/100 [00:01<00:16,  5.34it/s, failures=0, objective=-0.51]
 12%|█▏        | 12/100 [00:01<00:16,  5.34it/s, failures=0, objective=-0.51]
 13%|█▎        | 13/100 [00:02<00:16,  5.18it/s, failures=0, objective=-0.51]
 13%|█▎        | 13/100 [00:02<00:16,  5.18it/s, failures=0, objective=-0.26]
 14%|█▍        | 14/100 [00:02<00:16,  5.18it/s, failures=0, objective=-0.0695]
 15%|█▌        | 15/100 [00:02<00:16,  5.18it/s, failures=0, objective=-0.000646]
 16%|█▌        | 16/100 [00:02<00:16,  5.18it/s, failures=0, objective=-0.000646]
 17%|█▋        | 17/100 [00:03<00:16,  5.14it/s, failures=0, objective=-0.000646]
 17%|█▋        | 17/100 [00:03<00:16,  5.14it/s, failures=0, objective=-0.000646]
 18%|█▊        | 18/100 [00:03<00:15,  5.14it/s, failures=0, objective=-0.000646]
 19%|█▉        | 19/100 [00:03<00:15,  5.14it/s, failures=0, objective=-0.000646]
 20%|██        | 20/100 [00:03<00:15,  5.14it/s, failures=0, objective=-0.000646]
 21%|██        | 21/100 [00:04<00:15,  5.13it/s, failures=0, objective=-0.000646]
 21%|██        | 21/100 [00:04<00:15,  5.13it/s, failures=0, objective=-0.000646]
 22%|██▏       | 22/100 [00:04<00:15,  5.13it/s, failures=0, objective=-0.000646]
 23%|██▎       | 23/100 [00:04<00:15,  5.13it/s, failures=0, objective=-0.000406]
 24%|██▍       | 24/100 [00:04<00:14,  5.13it/s, failures=0, objective=-0.000406]
 25%|██▌       | 25/100 [00:04<00:14,  5.12it/s, failures=0, objective=-0.000406]
 25%|██▌       | 25/100 [00:04<00:14,  5.12it/s, failures=0, objective=-0.000406]
 26%|██▌       | 26/100 [00:04<00:14,  5.12it/s, failures=0, objective=-0.000406]
 27%|██▋       | 27/100 [00:04<00:14,  5.12it/s, failures=0, objective=-0.000406]
 28%|██▊       | 28/100 [00:04<00:14,  5.12it/s, failures=0, objective=-0.000406]
 29%|██▉       | 29/100 [00:05<00:13,  5.11it/s, failures=0, objective=-0.000406]
 29%|██▉       | 29/100 [00:05<00:13,  5.11it/s, failures=0, objective=-0.000406]
 30%|███       | 30/100 [00:05<00:13,  5.11it/s, failures=0, objective=-0.000406]
 31%|███       | 31/100 [00:05<00:13,  5.11it/s, failures=0, objective=-0.000406]
 32%|███▏      | 32/100 [00:06<00:13,  5.02it/s, failures=0, objective=-0.000406]
 32%|███▏      | 32/100 [00:06<00:13,  5.02it/s, failures=0, objective=-0.000406]
 33%|███▎      | 33/100 [00:06<00:13,  5.14it/s, failures=0, objective=-0.000406]
 33%|███▎      | 33/100 [00:06<00:13,  5.14it/s, failures=0, objective=-0.000406]
 34%|███▍      | 34/100 [00:06<00:12,  5.14it/s, failures=0, objective=-0.000406]
 35%|███▌      | 35/100 [00:06<00:12,  5.14it/s, failures=0, objective=-0.000406]
 36%|███▌      | 36/100 [00:07<00:12,  5.00it/s, failures=0, objective=-0.000406]
 36%|███▌      | 36/100 [00:07<00:12,  5.00it/s, failures=0, objective=-0.000406]
 37%|███▋      | 37/100 [00:07<00:12,  5.14it/s, failures=0, objective=-0.000406]
 37%|███▋      | 37/100 [00:07<00:12,  5.14it/s, failures=0, objective=-0.000406]
 38%|███▊      | 38/100 [00:07<00:12,  5.14it/s, failures=0, objective=-0.000406]
 39%|███▉      | 39/100 [00:07<00:11,  5.14it/s, failures=0, objective=-0.000406]
 40%|████      | 40/100 [00:07<00:12,  4.80it/s, failures=0, objective=-0.000406]
 40%|████      | 40/100 [00:07<00:12,  4.80it/s, failures=0, objective=-0.000406]
 41%|████      | 41/100 [00:08<00:11,  4.98it/s, failures=0, objective=-0.000406]
 41%|████      | 41/100 [00:08<00:11,  4.98it/s, failures=0, objective=-0.000406]
 42%|████▏     | 42/100 [00:08<00:11,  4.98it/s, failures=0, objective=-0.000406]
 43%|████▎     | 43/100 [00:08<00:11,  4.98it/s, failures=0, objective=-0.000406]
 44%|████▍     | 44/100 [00:08<00:11,  4.87it/s, failures=0, objective=-0.000406]
 44%|████▍     | 44/100 [00:08<00:11,  4.87it/s, failures=0, objective=-0.000406]
 45%|████▌     | 45/100 [00:08<00:10,  5.05it/s, failures=0, objective=-0.000406]
 45%|████▌     | 45/100 [00:08<00:10,  5.05it/s, failures=0, objective=-0.000406]
 46%|████▌     | 46/100 [00:08<00:10,  5.05it/s, failures=0, objective=-0.000406]
 47%|████▋     | 47/100 [00:08<00:10,  5.05it/s, failures=0, objective=-0.000406]
 48%|████▊     | 48/100 [00:09<00:10,  4.92it/s, failures=0, objective=-0.000406]
 48%|████▊     | 48/100 [00:09<00:10,  4.92it/s, failures=0, objective=-0.000406]
 49%|████▉     | 49/100 [00:09<00:09,  5.10it/s, failures=0, objective=-0.000406]
 49%|████▉     | 49/100 [00:09<00:09,  5.10it/s, failures=0, objective=-0.000229]
 50%|█████     | 50/100 [00:09<00:09,  5.10it/s, failures=0, objective=-0.000229]
 51%|█████     | 51/100 [00:09<00:09,  5.10it/s, failures=0, objective=-0.000229]
 52%|█████▏    | 52/100 [00:10<00:09,  4.95it/s, failures=0, objective=-0.000229]
 52%|█████▏    | 52/100 [00:10<00:09,  4.95it/s, failures=0, objective=-0.000229]
 53%|█████▎    | 53/100 [00:10<00:09,  5.12it/s, failures=0, objective=-0.000229]
 53%|█████▎    | 53/100 [00:10<00:09,  5.12it/s, failures=0, objective=-0.000229]
 54%|█████▍    | 54/100 [00:10<00:08,  5.12it/s, failures=0, objective=-0.000229]
 55%|█████▌    | 55/100 [00:10<00:08,  5.12it/s, failures=0, objective=-0.000229]
 56%|█████▌    | 56/100 [00:11<00:08,  4.95it/s, failures=0, objective=-0.000229]
 56%|█████▌    | 56/100 [00:11<00:08,  4.95it/s, failures=0, objective=-0.000229]
 57%|█████▋    | 57/100 [00:11<00:08,  5.12it/s, failures=0, objective=-0.000229]
 57%|█████▋    | 57/100 [00:11<00:08,  5.12it/s, failures=0, objective=-1.36e-7]
 58%|█████▊    | 58/100 [00:11<00:08,  5.12it/s, failures=0, objective=-1.36e-7]
 59%|█████▉    | 59/100 [00:11<00:08,  5.12it/s, failures=0, objective=-1.36e-7]
 60%|██████    | 60/100 [00:11<00:08,  4.93it/s, failures=0, objective=-1.36e-7]
 60%|██████    | 60/100 [00:11<00:08,  4.93it/s, failures=0, objective=-1.36e-7]
 61%|██████    | 61/100 [00:12<00:07,  5.12it/s, failures=0, objective=-1.36e-7]
 61%|██████    | 61/100 [00:12<00:07,  5.12it/s, failures=0, objective=-1.36e-7]
 62%|██████▏   | 62/100 [00:12<00:07,  5.12it/s, failures=0, objective=-1.36e-7]
 63%|██████▎   | 63/100 [00:12<00:07,  5.12it/s, failures=0, objective=-1.36e-7]
 64%|██████▍   | 64/100 [00:12<00:07,  4.95it/s, failures=0, objective=-1.36e-7]
 64%|██████▍   | 64/100 [00:12<00:07,  4.95it/s, failures=0, objective=-1.36e-7]
 65%|██████▌   | 65/100 [00:12<00:06,  5.12it/s, failures=0, objective=-1.36e-7]
 65%|██████▌   | 65/100 [00:12<00:06,  5.12it/s, failures=0, objective=-1.36e-7]
 66%|██████▌   | 66/100 [00:12<00:06,  5.12it/s, failures=0, objective=-1.36e-7]
 67%|██████▋   | 67/100 [00:12<00:06,  5.12it/s, failures=0, objective=-1.36e-7]
 68%|██████▊   | 68/100 [00:13<00:06,  4.94it/s, failures=0, objective=-1.36e-7]
 68%|██████▊   | 68/100 [00:13<00:06,  4.94it/s, failures=0, objective=-1.36e-7]
 69%|██████▉   | 69/100 [00:13<00:06,  5.11it/s, failures=0, objective=-1.36e-7]
 69%|██████▉   | 69/100 [00:13<00:06,  5.11it/s, failures=0, objective=-1.36e-7]
 70%|███████   | 70/100 [00:13<00:05,  5.11it/s, failures=0, objective=-1.36e-7]
 71%|███████   | 71/100 [00:13<00:05,  5.11it/s, failures=0, objective=-1.36e-7]
 72%|███████▏  | 72/100 [00:14<00:05,  4.91it/s, failures=0, objective=-1.36e-7]
 72%|███████▏  | 72/100 [00:14<00:05,  4.91it/s, failures=0, objective=-1.36e-7]
 73%|███████▎  | 73/100 [00:14<00:05,  4.82it/s, failures=0, objective=-1.36e-7]
 73%|███████▎  | 73/100 [00:14<00:05,  4.82it/s, failures=0, objective=-1.36e-7]
 74%|███████▍  | 74/100 [00:14<00:05,  4.82it/s, failures=0, objective=-1.36e-7]
 75%|███████▌  | 75/100 [00:14<00:05,  4.82it/s, failures=0, objective=-1.36e-7]
 76%|███████▌  | 76/100 [00:15<00:05,  4.77it/s, failures=0, objective=-1.36e-7]
 76%|███████▌  | 76/100 [00:15<00:05,  4.77it/s, failures=0, objective=-1.36e-7]
 77%|███████▋  | 77/100 [00:15<00:04,  4.95it/s, failures=0, objective=-1.36e-7]
 77%|███████▋  | 77/100 [00:15<00:04,  4.95it/s, failures=0, objective=-1.36e-7]
 78%|███████▊  | 78/100 [00:15<00:04,  4.95it/s, failures=0, objective=-1.36e-7]
 79%|███████▉  | 79/100 [00:15<00:04,  4.95it/s, failures=0, objective=-1.36e-7]
 80%|████████  | 80/100 [00:15<00:04,  4.85it/s, failures=0, objective=-1.36e-7]
 80%|████████  | 80/100 [00:15<00:04,  4.85it/s, failures=0, objective=-1.36e-7]
 81%|████████  | 81/100 [00:16<00:03,  5.04it/s, failures=0, objective=-1.36e-7]
 81%|████████  | 81/100 [00:16<00:03,  5.04it/s, failures=0, objective=-1.36e-7]
 82%|████████▏ | 82/100 [00:16<00:03,  5.04it/s, failures=0, objective=-1.36e-7]
 83%|████████▎ | 83/100 [00:16<00:03,  5.04it/s, failures=0, objective=-1.36e-7]
 84%|████████▍ | 84/100 [00:16<00:03,  4.88it/s, failures=0, objective=-1.36e-7]
 84%|████████▍ | 84/100 [00:16<00:03,  4.88it/s, failures=0, objective=-1.36e-7]
 85%|████████▌ | 85/100 [00:16<00:02,  5.06it/s, failures=0, objective=-1.36e-7]
 85%|████████▌ | 85/100 [00:16<00:02,  5.06it/s, failures=0, objective=-1.36e-7]
 86%|████████▌ | 86/100 [00:16<00:02,  5.06it/s, failures=0, objective=-1.36e-7]
 87%|████████▋ | 87/100 [00:16<00:02,  5.06it/s, failures=0, objective=-1.36e-7]
 88%|████████▊ | 88/100 [00:17<00:02,  4.90it/s, failures=0, objective=-1.36e-7]
 88%|████████▊ | 88/100 [00:17<00:02,  4.90it/s, failures=0, objective=-1.36e-7]
 89%|████████▉ | 89/100 [00:17<00:02,  5.08it/s, failures=0, objective=-1.36e-7]
 89%|████████▉ | 89/100 [00:17<00:02,  5.08it/s, failures=0, objective=-1.36e-7]
 90%|█████████ | 90/100 [00:17<00:01,  5.08it/s, failures=0, objective=-1.36e-7]
 91%|█████████ | 91/100 [00:17<00:01,  5.08it/s, failures=0, objective=-1.36e-7]
 92%|█████████▏| 92/100 [00:18<00:01,  4.91it/s, failures=0, objective=-1.36e-7]
 92%|█████████▏| 92/100 [00:18<00:01,  4.91it/s, failures=0, objective=-1.36e-7]
 93%|█████████▎| 93/100 [00:18<00:01,  5.09it/s, failures=0, objective=-1.36e-7]
 93%|█████████▎| 93/100 [00:18<00:01,  5.09it/s, failures=0, objective=-1.36e-7]
 94%|█████████▍| 94/100 [00:18<00:01,  5.09it/s, failures=0, objective=-1.36e-7]
 95%|█████████▌| 95/100 [00:18<00:00,  5.09it/s, failures=0, objective=-1.36e-7]
 96%|█████████▌| 96/100 [00:19<00:00,  4.92it/s, failures=0, objective=-1.36e-7]
 96%|█████████▌| 96/100 [00:19<00:00,  4.92it/s, failures=0, objective=-1.36e-7]
 97%|█████████▋| 97/100 [00:19<00:00,  5.09it/s, failures=0, objective=-1.36e-7]
 97%|█████████▋| 97/100 [00:19<00:00,  5.09it/s, failures=0, objective=-1.36e-7]
 98%|█████████▊| 98/100 [00:19<00:00,  5.09it/s, failures=0, objective=-1.36e-7]
 99%|█████████▉| 99/100 [00:19<00:00,  5.09it/s, failures=0, objective=-1.36e-7]
100%|██████████| 100/100 [00:19<00:00,  4.91it/s, failures=0, objective=-1.36e-7]
100%|██████████| 100/100 [00:19<00:00,  4.91it/s, failures=0, objective=-1.36e-7]
100%|██████████| 100/100 [00:19<00:00,  5.02it/s, failures=0, objective=-1.36e-7]

Finally, let us visualize the results. The search(...) returns a DataFrame also saved locally under results.csv (in case of crash we don’t want to lose the possibly expensive evaluations already performed).

The DataFrame contains as columns:

the optimized hyperparameters: such as $x$ with name p:x.
the objective maximised which directly match the results of the $f$ function in our example.
the job_id of each evaluated function (increased incrementally following the order of created evaluations).
the time of creation/collection of each task timestamp_submit and timestamp_gather respectively (in secondes, since the creation of the Evaluator).

results

	p:x	objective	job_id	job_status	m:timestamp_submit	m:timestamp_gather
0	0.714013	-0.509815	0	DONE	16.055601	16.056108
1	-7.299946	-53.289215	2	DONE	16.055627	16.062735
2	2.997893	-8.987361	1	DONE	16.055620	16.062853
3	8.294617	-68.800670	3	DONE	16.055632	16.062946
4	5.378610	-28.929448	4	DONE	16.936808	16.937233
...	...	...	...	...	...	...
98	-0.260893	-0.068065	96	DONE	35.183628	35.345498
99	-0.330581	-0.109284	99	DONE	35.345077	35.986678
100	-0.267424	-0.071516	102	DONE	35.986451	36.149026
101	-0.047429	-0.002249	100	DONE	35.986431	36.149059
102	-0.062054	-0.003851	101	DONE	35.986446	36.149077

103 rows × 6 columns

To get the parameters at the observed maximum value we can use the deephyper.analysis.hpo.parameters_at_max():

from deephyper.analysis.hpo import parameters_at_max


parameters, objective = parameters_at_max(results)
print("\nOptimum values")
print("x:", parameters["x"])
print("objective:", objective)

Optimum values
x: -0.00036876827022958025
objective: -1.3599003712811672e-07

We can also plot the evolution of the objective to verify that we converge correctly toward $0$.

import matplotlib.pyplot as plt
from deephyper.analysis.hpo import plot_search_trajectory_single_objective_hpo


WIDTH_PLOTS = 8
HEIGHT_PLOTS = WIDTH_PLOTS / 1.618

fig, ax = plt.subplots(figsize=(WIDTH_PLOTS, HEIGHT_PLOTS))
plot_search_trajectory_single_objective_hpo(results, mode="min", ax=ax)
_ = plt.title("Search Trajectory")
_ = plt.yscale("log")

Total running time of the script: (0 minutes 37.458 seconds)

Gallery generated by Sphinx-Gallery