1. DeepHyper 101

Open In Colab

In this tutorial, we present the basics of DeepHyper.

Let us start with installing DeepHyper!

[ ]:
!pip install deephyper

Warning

By design asyncio does not allow nested event loops. Jupyter is using Tornado which already starts an event loop. Therefore the following patch is required to run DeepHyper in a Jupyter notebook.

[ ]:
!pip install nest_asyncio

import nest_asyncio
nest_asyncio.apply()
Requirement already satisfied: nest_asyncio in /usr/local/lib/python3.7/dist-packages (1.5.1)

1.1. Optimization Problem

In the definition of our optimization problem we have two components:

  1. black-box function that we want to optimize

  2. the search space of input variables

1.1.1. Black-Box Function

DeepHyper is developed to optimize black-box functions. Here, we define the function \(f(x) = - x ^ 2\) that we want to maximise (the maximum being \(f(x=0) = 0\) on \(I_x = [-10;10]\)). The black-box function f takes as input a config dictionnary from which we retrieve the variables of interest.

[11]:
def f(config):
    return - config["x"]**2

1.1.2. Search Space of Input Variables

In this example, we have only one variable \(x\) for the black-box functin \(f\). We empirically decide to optimize this variable \(x\) on the interval \(I_x = [-10;10]\). To do so we use the HpProblem from DeepHyper and add a real hyperparameter by using a tuple of two floats.

[ ]:
from deephyper.problem import HpProblem


problem = HpProblem()

# define the variable you want to optimize
problem.add_hyperparameter((-10.0, 10.0), "x")

problem
Configuration space object:
  Hyperparameters:
    x, Type: UniformFloat, Range: [-10.0, 10.0], Default: 0.0

1.2. Evaluator Interface

DeepHyper uses an API called Evaluator to distribute the computation of black-box functions and adapt to different backends (e.g., threads, processes, MPI, Ray). An Evaluator object wraps the black-box function f that we want to optimize. Then a method parameter is used to select the backend and method_kwargs defines some available options of this backend. For Jupyter notebooks we use the Ray evaluator (method="ray").

Depending on available resources (num_cpus) available and the quantity of resources used by each computation of the black-box function (num_cpus_per_task) the number of available workers is automatically computed.

It is possible to define callbacks to extend the behaviour of Evaluator each time a function-evaluation is launched or completed. In this example we use the LoggerCallback to follow the completed evaluations and the evolution of the objective.

[14]:
from deephyper.evaluator import Evaluator
from deephyper.evaluator.callback import LoggerCallback


# define the evaluator to distribute the computation
evaluator = Evaluator.create(
    f,
    method="ray",
    method_kwargs={
        "num_cpus": 1,
        "num_cpus_per_task": 0.25,
        "callbacks": [LoggerCallback()]
    },
)

print(f"Evaluator has {evaluator.num_workers} available worker{'' if evaluator.num_workers == 1 else 's'}")
Evaluator has 4 available workers

1.3. Search Algorithm

The next step is to define the search algorithm that we want to use. Here, we choose AMBS (Asynchronous Model-Based Search) which is a sampling based Bayesian optimization strategy. This algorithm has the advantage of being asynchronous thanks to a constant liar strategy which is crutial to keep a good utilization of the resources when the number of available workers increases.

[15]:
from deephyper.search.hps import AMBS


# define your search
search = AMBS(problem, evaluator)

Then, we can execute the search for a given number of iterations by using the search.search(max_evals=...). It is also possible to use the timeout parameter if one needs a specific time budget (e.g., restricted computational time in machine learning competitions, allocation time in HPC).

[16]:
results = search.search(max_evals=100)
[00001] -- best objective: -48.01665 -- received objective: -48.01665
[00002] -- best objective: -32.48806 -- received objective: -32.48806
[00003] -- best objective: -6.95366 -- received objective: -6.95366
[00004] -- best objective: -0.02131 -- received objective: -0.02131
[00005] -- best objective: -0.02131 -- received objective: -6.22327
[00006] -- best objective: -0.02131 -- received objective: -6.66244
[00007] -- best objective: -0.02131 -- received objective: -23.36765
[00008] -- best objective: -0.02131 -- received objective: -18.95907
[00009] -- best objective: -0.02131 -- received objective: -16.40159
[00010] -- best objective: -0.02131 -- received objective: -9.62992
[00011] -- best objective: -0.02131 -- received objective: -17.61847
[00012] -- best objective: -0.02131 -- received objective: -24.02175
[00013] -- best objective: -0.02131 -- received objective: -8.78387
[00014] -- best objective: -0.02131 -- received objective: -1.66473
[00015] -- best objective: -0.02131 -- received objective: -2.46526
[00016] -- best objective: -0.02131 -- received objective: -1.18740
[00017] -- best objective: -0.02131 -- received objective: -1.16130
[00018] -- best objective: -0.02131 -- received objective: -3.40520
[00019] -- best objective: -0.02131 -- received objective: -4.64195
[00020] -- best objective: -0.02131 -- received objective: -1.43801
[00021] -- best objective: -0.02131 -- received objective: -3.21256
[00022] -- best objective: -0.02131 -- received objective: -0.44638
[00023] -- best objective: -0.02131 -- received objective: -0.19772
[00024] -- best objective: -0.02131 -- received objective: -0.79561
[00025] -- best objective: -0.02131 -- received objective: -0.17115
[00026] -- best objective: -0.02131 -- received objective: -0.08435
[00027] -- best objective: -0.02131 -- received objective: -0.10716
[00028] -- best objective: -0.02131 -- received objective: -0.48718
[00029] -- best objective: -0.02131 -- received objective: -0.10139
[00030] -- best objective: -0.02131 -- received objective: -4.23256
[00031] -- best objective: -0.02131 -- received objective: -4.00711
[00032] -- best objective: -0.02131 -- received objective: -5.20448
[00033] -- best objective: -0.02131 -- received objective: -3.99205
[00034] -- best objective: -0.02131 -- received objective: -0.05532
[00035] -- best objective: -0.02131 -- received objective: -0.33852
[00036] -- best objective: -0.02131 -- received objective: -0.26087
[00037] -- best objective: -0.02131 -- received objective: -0.25922
[00038] -- best objective: -0.01225 -- received objective: -0.01225
[00039] -- best objective: -0.00311 -- received objective: -0.00311
[00040] -- best objective: -0.00311 -- received objective: -0.07740
[00041] -- best objective: -0.00311 -- received objective: -0.00392
[00042] -- best objective: -0.00311 -- received objective: -0.02743
[00043] -- best objective: -0.00311 -- received objective: -0.05363
[00044] -- best objective: -0.00311 -- received objective: -0.04416
[00045] -- best objective: -0.00311 -- received objective: -0.03211
[00046] -- best objective: -0.00311 -- received objective: -0.01239
[00047] -- best objective: -0.00311 -- received objective: -0.01307
[00048] -- best objective: -0.00311 -- received objective: -0.01898
[00049] -- best objective: -0.00311 -- received objective: -0.01255
[00050] -- best objective: -0.00059 -- received objective: -0.00059
[00051] -- best objective: -0.00059 -- received objective: -0.00727
[00052] -- best objective: -0.00059 -- received objective: -0.00359
[00053] -- best objective: -0.00059 -- received objective: -0.00070
[00054] -- best objective: -0.00059 -- received objective: -0.00156
[00055] -- best objective: -0.00059 -- received objective: -0.00462
[00056] -- best objective: -0.00059 -- received objective: -0.00095
[00057] -- best objective: -0.00059 -- received objective: -0.00112
[00058] -- best objective: -0.00059 -- received objective: -0.00076
[00059] -- best objective: -0.00059 -- received objective: -0.00159
[00060] -- best objective: -0.00018 -- received objective: -0.00018
[00061] -- best objective: -0.00018 -- received objective: -0.00023
[00062] -- best objective: -0.00018 -- received objective: -0.00039
[00063] -- best objective: -0.00018 -- received objective: -0.00168
[00064] -- best objective: -0.00018 -- received objective: -0.00033
[00065] -- best objective: -0.00018 -- received objective: -0.00027
[00066] -- best objective: -0.00018 -- received objective: -1.65799
[00067] -- best objective: -0.00018 -- received objective: -1.51467
[00068] -- best objective: -0.00018 -- received objective: -0.00018
[00069] -- best objective: -0.00018 -- received objective: -1.49407
[00070] -- best objective: -0.00018 -- received objective: -0.04112
[00071] -- best objective: -0.00018 -- received objective: -0.03010
[00072] -- best objective: -0.00018 -- received objective: -0.03879
[00073] -- best objective: -0.00018 -- received objective: -0.77481
[00074] -- best objective: -0.00018 -- received objective: -0.95751
[00075] -- best objective: -0.00018 -- received objective: -1.20611
[00076] -- best objective: -0.00018 -- received objective: -0.69324
[00077] -- best objective: -0.00018 -- received objective: -0.69422
[00078] -- best objective: -0.00018 -- received objective: -0.00609
[00079] -- best objective: -0.00018 -- received objective: -0.00586
[00080] -- best objective: -0.00018 -- received objective: -0.00867
[00081] -- best objective: -0.00018 -- received objective: -0.00582
[00082] -- best objective: -0.00018 -- received objective: -0.00164
[00083] -- best objective: -0.00018 -- received objective: -0.00098
[00084] -- best objective: -0.00018 -- received objective: -0.00130
[00085] -- best objective: -0.00018 -- received objective: -0.00773
[00086] -- best objective: -0.00018 -- received objective: -0.00796
[00087] -- best objective: -0.00018 -- received objective: -0.00952
[00088] -- best objective: -0.00018 -- received objective: -0.00863
[00089] -- best objective: -0.00000 -- received objective: -0.00000
[00090] -- best objective: -0.00000 -- received objective: -0.00017
[00091] -- best objective: -0.00000 -- received objective: -0.00004
[00092] -- best objective: -0.00000 -- received objective: -0.00009
[00093] -- best objective: -0.00000 -- received objective: -0.00005
[00094] -- best objective: -0.00000 -- received objective: -0.00031
[00095] -- best objective: -0.00000 -- received objective: -0.00005
[00096] -- best objective: -0.00000 -- received objective: -0.00000
[00097] -- best objective: -0.00000 -- received objective: -0.00031
[00098] -- best objective: -0.00000 -- received objective: -0.00009
[00099] -- best objective: -0.00000 -- received objective: -0.00002
[00100] -- best objective: -0.00000 -- received objective: -0.00001

Finally, let us visualize the results. The search(...) returns a DataFrame also saved locally under results.csv (in case of crash we don’t want to loose the possibly expensive evaluations already performed).

The DataFrame contains as columns: 1. the optimized hyperparameters: such as x in our case. 2. the id of each evaluated function (increased incrementally following the order of created evaluations). 3. the objective maximised which directly match the results of the \(f\)-function in our example. 4. the time of termination of each task elapsed_sec (in secondes, since the creation of the Evaluator) and the duration (in secondes).

[17]:
results
[17]:
x id objective elapsed_sec duration
0 6.929404 1 -48.016645 18.691893 0.015289
1 -5.699830 2 -32.488058 19.265710 0.589085
2 -0.145986 3 -0.021312 19.267566 0.590926
3 -2.636978 4 -6.953655 19.266957 0.590303
4 2.494648 5 -6.223271 20.655589 1.391588
... ... ... ... ... ...
93 -0.007204 94 -0.000052 76.526612 1.072505
94 -0.017661 96 -0.000312 76.526468 1.072291
95 -0.017504 97 -0.000306 78.039819 1.515123
96 0.001946 95 -0.000004 76.528772 1.074623
97 0.004225 99 -0.000018 79.006809 0.969238

98 rows × 5 columns

We can also plot the evolution of the objective to verify that we converge correctly toward \(0\).

[21]:
import matplotlib.pyplot as plt

plt.plot(results.objective)

plt.xlabel("Iterations")
plt.ylabel("$y = f(x)$")

plt.show()
../../../_images/tutorials_tutorials_colab_DeepHyper_101_17_0.png