DeepHyper 101
Contents
1. DeepHyper 101¶
In this tutorial, we present the basics of DeepHyper.
Let us start with installing DeepHyper!
[ ]:
!pip install deephyper
!pip install ray==1.9.2 -I
Warning
By design asyncio does not allow nested event loops. Jupyter is using Tornado which already starts an event loop. Therefore the following patch is required to run DeepHyper in a Jupyter notebook.
[ ]:
!pip install nest_asyncio
import nest_asyncio
nest_asyncio.apply()
Requirement already satisfied: nest_asyncio in /usr/local/lib/python3.7/dist-packages (1.5.1)
1.1. Optimization Problem¶
In the definition of our optimization problem we have two components:
black-box function that we want to optimize
the search space of input variables
1.1.1. Black-Box Function¶
DeepHyper is developed to optimize black-box functions. Here, we define the function \(f(x) = - x ^ 2\) that we want to maximise (the maximum being \(f(x=0) = 0\) on \(I_x = [-10;10]\)). The black-box function f takes as input a config dictionnary from which we retrieve the variables of interest.
[11]:
def f(config):
return - config["x"]**2
1.1.2. Search Space of Input Variables¶
In this example, we have only one variable \(x\) for the black-box functin \(f\). We empirically decide to optimize this variable \(x\) on the interval \(I_x = [-10;10]\). To do so we use the HpProblem from DeepHyper and add a real hyperparameter by using a tuple of two floats.
[ ]:
from deephyper.problem import HpProblem
problem = HpProblem()
# define the variable you want to optimize
problem.add_hyperparameter((-10.0, 10.0), "x")
problem
Configuration space object:
Hyperparameters:
x, Type: UniformFloat, Range: [-10.0, 10.0], Default: 0.0
1.2. Evaluator Interface¶
DeepHyper uses an API called Evaluator to distribute the computation of black-box functions and adapt to different backends (e.g., threads, processes, MPI, Ray). An Evaluator object wraps the black-box function f that we want to optimize. Then a method parameter is used to select the backend and method_kwargs defines some available options of this backend. For Jupyter notebooks we use the Ray evaluator (method="ray").
Depending on available resources (num_cpus) available and the quantity of resources used by each computation of the black-box function (num_cpus_per_task) the number of available workers is automatically computed.
It is possible to define callbacks to extend the behaviour of Evaluator each time a function-evaluation is launched or completed. In this example we use the LoggerCallback to follow the completed evaluations and the evolution of the objective.
[14]:
from deephyper.evaluator import Evaluator
from deephyper.evaluator.callback import LoggerCallback
# define the evaluator to distribute the computation
evaluator = Evaluator.create(
f,
method="ray",
method_kwargs={
"num_cpus": 1,
"num_cpus_per_task": 0.25,
"callbacks": [LoggerCallback()]
},
)
print(f"Evaluator has {evaluator.num_workers} available worker{'' if evaluator.num_workers == 1 else 's'}")
Evaluator has 4 available workers
1.3. Search Algorithm¶
The next step is to define the search algorithm that we want to use. Here, we choose AMBS (Asynchronous Model-Based Search) which is a sampling based Bayesian optimization strategy. This algorithm has the advantage of being asynchronous thanks to a constant liar strategy which is crutial to keep a good utilization of the resources when the number of available workers increases.
[15]:
from deephyper.search.hps import AMBS
# define your search
search = AMBS(problem, evaluator)
Then, we can execute the search for a given number of iterations by using the search.search(max_evals=...). It is also possible to use the timeout parameter if one needs a specific time budget (e.g., restricted computational time in machine learning competitions, allocation time in HPC).
[16]:
results = search.search(max_evals=100)
[00001] -- best objective: -48.01665 -- received objective: -48.01665
[00002] -- best objective: -32.48806 -- received objective: -32.48806
[00003] -- best objective: -6.95366 -- received objective: -6.95366
[00004] -- best objective: -0.02131 -- received objective: -0.02131
[00005] -- best objective: -0.02131 -- received objective: -6.22327
[00006] -- best objective: -0.02131 -- received objective: -6.66244
[00007] -- best objective: -0.02131 -- received objective: -23.36765
[00008] -- best objective: -0.02131 -- received objective: -18.95907
[00009] -- best objective: -0.02131 -- received objective: -16.40159
[00010] -- best objective: -0.02131 -- received objective: -9.62992
[00011] -- best objective: -0.02131 -- received objective: -17.61847
[00012] -- best objective: -0.02131 -- received objective: -24.02175
[00013] -- best objective: -0.02131 -- received objective: -8.78387
[00014] -- best objective: -0.02131 -- received objective: -1.66473
[00015] -- best objective: -0.02131 -- received objective: -2.46526
[00016] -- best objective: -0.02131 -- received objective: -1.18740
[00017] -- best objective: -0.02131 -- received objective: -1.16130
[00018] -- best objective: -0.02131 -- received objective: -3.40520
[00019] -- best objective: -0.02131 -- received objective: -4.64195
[00020] -- best objective: -0.02131 -- received objective: -1.43801
[00021] -- best objective: -0.02131 -- received objective: -3.21256
[00022] -- best objective: -0.02131 -- received objective: -0.44638
[00023] -- best objective: -0.02131 -- received objective: -0.19772
[00024] -- best objective: -0.02131 -- received objective: -0.79561
[00025] -- best objective: -0.02131 -- received objective: -0.17115
[00026] -- best objective: -0.02131 -- received objective: -0.08435
[00027] -- best objective: -0.02131 -- received objective: -0.10716
[00028] -- best objective: -0.02131 -- received objective: -0.48718
[00029] -- best objective: -0.02131 -- received objective: -0.10139
[00030] -- best objective: -0.02131 -- received objective: -4.23256
[00031] -- best objective: -0.02131 -- received objective: -4.00711
[00032] -- best objective: -0.02131 -- received objective: -5.20448
[00033] -- best objective: -0.02131 -- received objective: -3.99205
[00034] -- best objective: -0.02131 -- received objective: -0.05532
[00035] -- best objective: -0.02131 -- received objective: -0.33852
[00036] -- best objective: -0.02131 -- received objective: -0.26087
[00037] -- best objective: -0.02131 -- received objective: -0.25922
[00038] -- best objective: -0.01225 -- received objective: -0.01225
[00039] -- best objective: -0.00311 -- received objective: -0.00311
[00040] -- best objective: -0.00311 -- received objective: -0.07740
[00041] -- best objective: -0.00311 -- received objective: -0.00392
[00042] -- best objective: -0.00311 -- received objective: -0.02743
[00043] -- best objective: -0.00311 -- received objective: -0.05363
[00044] -- best objective: -0.00311 -- received objective: -0.04416
[00045] -- best objective: -0.00311 -- received objective: -0.03211
[00046] -- best objective: -0.00311 -- received objective: -0.01239
[00047] -- best objective: -0.00311 -- received objective: -0.01307
[00048] -- best objective: -0.00311 -- received objective: -0.01898
[00049] -- best objective: -0.00311 -- received objective: -0.01255
[00050] -- best objective: -0.00059 -- received objective: -0.00059
[00051] -- best objective: -0.00059 -- received objective: -0.00727
[00052] -- best objective: -0.00059 -- received objective: -0.00359
[00053] -- best objective: -0.00059 -- received objective: -0.00070
[00054] -- best objective: -0.00059 -- received objective: -0.00156
[00055] -- best objective: -0.00059 -- received objective: -0.00462
[00056] -- best objective: -0.00059 -- received objective: -0.00095
[00057] -- best objective: -0.00059 -- received objective: -0.00112
[00058] -- best objective: -0.00059 -- received objective: -0.00076
[00059] -- best objective: -0.00059 -- received objective: -0.00159
[00060] -- best objective: -0.00018 -- received objective: -0.00018
[00061] -- best objective: -0.00018 -- received objective: -0.00023
[00062] -- best objective: -0.00018 -- received objective: -0.00039
[00063] -- best objective: -0.00018 -- received objective: -0.00168
[00064] -- best objective: -0.00018 -- received objective: -0.00033
[00065] -- best objective: -0.00018 -- received objective: -0.00027
[00066] -- best objective: -0.00018 -- received objective: -1.65799
[00067] -- best objective: -0.00018 -- received objective: -1.51467
[00068] -- best objective: -0.00018 -- received objective: -0.00018
[00069] -- best objective: -0.00018 -- received objective: -1.49407
[00070] -- best objective: -0.00018 -- received objective: -0.04112
[00071] -- best objective: -0.00018 -- received objective: -0.03010
[00072] -- best objective: -0.00018 -- received objective: -0.03879
[00073] -- best objective: -0.00018 -- received objective: -0.77481
[00074] -- best objective: -0.00018 -- received objective: -0.95751
[00075] -- best objective: -0.00018 -- received objective: -1.20611
[00076] -- best objective: -0.00018 -- received objective: -0.69324
[00077] -- best objective: -0.00018 -- received objective: -0.69422
[00078] -- best objective: -0.00018 -- received objective: -0.00609
[00079] -- best objective: -0.00018 -- received objective: -0.00586
[00080] -- best objective: -0.00018 -- received objective: -0.00867
[00081] -- best objective: -0.00018 -- received objective: -0.00582
[00082] -- best objective: -0.00018 -- received objective: -0.00164
[00083] -- best objective: -0.00018 -- received objective: -0.00098
[00084] -- best objective: -0.00018 -- received objective: -0.00130
[00085] -- best objective: -0.00018 -- received objective: -0.00773
[00086] -- best objective: -0.00018 -- received objective: -0.00796
[00087] -- best objective: -0.00018 -- received objective: -0.00952
[00088] -- best objective: -0.00018 -- received objective: -0.00863
[00089] -- best objective: -0.00000 -- received objective: -0.00000
[00090] -- best objective: -0.00000 -- received objective: -0.00017
[00091] -- best objective: -0.00000 -- received objective: -0.00004
[00092] -- best objective: -0.00000 -- received objective: -0.00009
[00093] -- best objective: -0.00000 -- received objective: -0.00005
[00094] -- best objective: -0.00000 -- received objective: -0.00031
[00095] -- best objective: -0.00000 -- received objective: -0.00005
[00096] -- best objective: -0.00000 -- received objective: -0.00000
[00097] -- best objective: -0.00000 -- received objective: -0.00031
[00098] -- best objective: -0.00000 -- received objective: -0.00009
[00099] -- best objective: -0.00000 -- received objective: -0.00002
[00100] -- best objective: -0.00000 -- received objective: -0.00001
Finally, let us visualize the results. The search(...) returns a DataFrame also saved locally under results.csv (in case of crash we don’t want to loose the possibly expensive evaluations already performed).
The DataFrame contains as columns: 1. the optimized hyperparameters: such as x in our case. 2. the id of each evaluated function (increased incrementally following the order of created evaluations). 3. the objective maximised which directly match the results of the \(f\)-function in our example. 4. the time of termination of each task elapsed_sec (in secondes, since the creation of the Evaluator) and the duration (in secondes).
[17]:
results
[17]:
| x | id | objective | elapsed_sec | duration | |
|---|---|---|---|---|---|
| 0 | 6.929404 | 1 | -48.016645 | 18.691893 | 0.015289 |
| 1 | -5.699830 | 2 | -32.488058 | 19.265710 | 0.589085 |
| 2 | -0.145986 | 3 | -0.021312 | 19.267566 | 0.590926 |
| 3 | -2.636978 | 4 | -6.953655 | 19.266957 | 0.590303 |
| 4 | 2.494648 | 5 | -6.223271 | 20.655589 | 1.391588 |
| ... | ... | ... | ... | ... | ... |
| 93 | -0.007204 | 94 | -0.000052 | 76.526612 | 1.072505 |
| 94 | -0.017661 | 96 | -0.000312 | 76.526468 | 1.072291 |
| 95 | -0.017504 | 97 | -0.000306 | 78.039819 | 1.515123 |
| 96 | 0.001946 | 95 | -0.000004 | 76.528772 | 1.074623 |
| 97 | 0.004225 | 99 | -0.000018 | 79.006809 | 0.969238 |
98 rows × 5 columns
We can also plot the evolution of the objective to verify that we converge correctly toward \(0\).
[21]:
import matplotlib.pyplot as plt
plt.plot(results.objective)
plt.xlabel("Iterations")
plt.ylabel("$y = f(x)$")
plt.show()
