1. DeepHyper 101#
In this tutorial, we present the basics of DeepHyper.
Let us start with installing DeepHyper!
[1]:
try:
import deephyper
print(deephyper.__version__)
except (ImportError, ModuleNotFoundError):
!pip install deephyper
0.6.0
1.1. Optimization Problem#
In the definition of our optimization problem we have two components:
black-box function that we want to optimize
the search space of input variables
1.1.1. Black-Box Function#
DeepHyper is developed to optimize black-box functions. Here, we define the function \(f(x) = - x ^ 2\) that we want to maximise (the maximum being \(f(x=0) = 0\) on \(I_x = [-10;10]\)). The black-box function f
takes as input a config
dictionary from which we retrieve the variables of interest.
[2]:
def f(job):
return -job.parameters["x"] ** 2
1.1.2. Search Space of Input Variables#
In this example, we have only one variable \(x\) for the black-box functin \(f\). We empirically decide to optimize this variable \(x\) on the interval \(I_x = [-10;10]\). To do so we use the HpProblem
from DeepHyper and add a real hyperparameter by using a tuple
of two floats
.
[3]:
from deephyper.problem import HpProblem
problem = HpProblem()
# Define the variable you want to optimize
problem.add_hyperparameter((-10.0, 10.0), "x")
problem
[3]:
Configuration space object:
Hyperparameters:
x, Type: UniformFloat, Range: [-10.0, 10.0], Default: 0.0
1.2. Evaluator Interface#
DeepHyper uses an API called Evaluator
to distribute the computation of black-box functions and adapt to different backends (e.g., threads, processes, MPI, Ray). An Evaluator
object wraps the black-box function f
that we want to optimize. Then a method
parameter is used to select the backend and method_kwargs
defines some available options of this backend.
Tip
The method="thread"
provides parallel computation only if the black-box is releasing the global interpretor lock (GIL). Therefore, if you want parallelism in Jupyter notebooks you should use the Ray evaluator (method="ray"
) after installing Ray with pip install ray
.
It is possible to define callbacks to extend the behaviour of Evaluator
each time a function-evaluation is launched or completed. In this example we use the TqdmCallback
to follow the completed evaluations and the evolution of the objective with a progress-bar.
[4]:
from deephyper.evaluator import Evaluator
from deephyper.evaluator.callback import TqdmCallback
# define the evaluator to distribute the computation
evaluator = Evaluator.create(
f,
method="thread",
method_kwargs={
"num_workers": 4,
"callbacks": [TqdmCallback()]
},
)
print(f"Evaluator has {evaluator.num_workers} available worker{'' if evaluator.num_workers == 1 else 's'}")
Evaluator has 4 available workers
/Users/romainegele/Documents/Argonne/deephyper/deephyper/evaluator/_evaluator.py:127: UserWarning: Applying nest-asyncio patch for IPython Shell!
warnings.warn(
1.3. Search Algorithm#
The next step is to define the search algorithm that we want to use. Here, we choose CBO
(Centralized Bayesian Optimization) which is a sampling based Bayesian optimization strategy. This algorithm has the advantage of being asynchronous thanks to a constant liar strategy which is crutial to keep a good utilization of the resources when the number of available workers increases.
[5]:
from deephyper.search.hps import CBO
# define your search
search = CBO(
problem,
evaluator,
acq_func="UCB", # Acquisition function to Upper Confidence Bound
multi_point_strategy="qUCB", # Fast Multi-point strategy with q-Upper Confidence Bound
n_jobs=2, # Number of threads to fit surrogate models in parallel
)
Then, we can execute the search for a given number of iterations by using the search.search(max_evals=...)
. It is also possible to use the timeout
parameter if one needs a specific time budget (e.g., restricted computational time in machine learning competitions, allocation time in HPC).
[6]:
results = search.search(max_evals=100)
Finally, let us visualize the results. The search(...)
returns a DataFrame also saved locally under results.csv
(in case of crash we don’t want to lose the possibly expensive evaluations already performed).
The DataFrame contains as columns: 1. the optimized hyperparameters: such as x
in our case. 2. the id
of each evaluated function (increased incrementally following the order of created evaluations). 3. the objective
maximised which directly match the results of the \(f\)-function in our example. 4. the objective
maximised which directly match the results of the \(f\)-function in our example. 5. the time of creation/collection of each task timestamp_submit and
timestamp_gather respectively (in secondes, since the creation of the Evaluator).
[7]:
results
[7]:
p:x | objective | job_id | m:timestamp_submit | m:timestamp_gather | |
---|---|---|---|---|---|
0 | 3.241574 | -1.050780e+01 | 3 | 0.088059 | 0.088764 |
1 | 3.401464 | -1.156996e+01 | 2 | 0.088030 | 0.094780 |
2 | 9.761872 | -9.529415e+01 | 0 | 0.087993 | 0.094980 |
3 | -2.094117 | -4.385325e+00 | 1 | 0.088019 | 0.095260 |
4 | 4.381632 | -1.919870e+01 | 7 | 0.114489 | 0.115110 |
... | ... | ... | ... | ... | ... |
95 | -0.004540 | -2.060865e-05 | 95 | 3.757070 | 3.758207 |
96 | -0.000577 | -3.328177e-07 | 97 | 3.979956 | 3.980287 |
97 | -0.003078 | -9.475395e-06 | 96 | 3.979945 | 3.980701 |
98 | -0.000577 | -3.328177e-07 | 99 | 3.979969 | 3.980882 |
99 | -0.000577 | -3.328177e-07 | 98 | 3.979963 | 3.981050 |
100 rows × 5 columns
We can also plot the evolution of the objective to verify that we converge correctly toward \(0\).
[8]:
from deephyper.analysis.hpo import plot_search_trajectory_single_objective_hpo
fig, ax = plot_search_trajectory_single_objective_hpo(results)

[ ]: