Noisy Black-Box Optimization

Noisy Black-Box Optimization#

Author(s): Romain Egele.

In this tutorial, we show you how to manage noisy black-box optimization (Wikipedia) (a.k.a., derivative-free optimization) with DeepHyper.

Black-box optimization is a field of optimization research where an objective function $f(x) = y \in \mathbb{R}$ is optimized only based on input-output observations $\{ (x_1,y_1), \ldots, (x_n, y_n) \}$.

Let’s start by installing DeepHyper!

%%bash
pip install deephyper

Optimization Problem#

The optimization problem consists of two components:

The black-box function that we aim to optimize.
The search space (or domain) of input variables over which the optimization is performed.

Black-Box Function#

In DeepHyper, black-box optimization is performed on user-defined functions that can be noisy or stochastic. Below, we define a noisy black-box function f that depends on a single variable $x$ in the domain $I_x = [-10, 10]$.

The noisy black-box function is defined as:

\[f(x) = \text{Binomial}(n=1, p(x))\]

where the probability of success is:

\[p(x) = \frac{100 - x^2}{100}.\]

This means that for each evaluation, f(x) returns a random binary value (0 or 1) with probability p(x) of success. The maximum expected value of $f(x)$ is obtained at $x = 0$, where $p(0) = 1$.

The function f takes as input a job, which behaves like a dictionary. The variable of interest x is accessed via job.parameters[“x”].

import numpy as np


def f(job):
    p = (100 - job.parameters["x"] ** 2) / 100
    obs = np.random.binomial(n=1, p=p)
    return obs

Search Space of Input Variables#

In this example, we have only one variable $x$ for the black-box functin $f$. We empirically decide to optimize this variable $x$ on the interval $I_x = [-10;10]$. To do so we use the deephyper.hpo.HpProblem from DeepHyper and add a real hyperparameter by using a tuple of two floats.

from deephyper.hpo import HpProblem

problem = HpProblem()

# Define the variable you want to optimize
problem.add_hyperparameter((-10.0, 10.0), "x")

problem

Configuration space object:
  Hyperparameters:
    x, Type: UniformFloat, Range: [-10.0, 10.0], Default: 0.0

Evaluator Interface#

DeepHyper uses an API called deephyper.evaluator.Evaluator to distribute the computation of black-box functions and adapt to different backends (e.g., threads, processes, MPI, Ray). An Evaluator object wraps the black-box function f that we want to optimize. Then a method parameter is used to select the backend and method_kwargs defines some available options of this backend.

Hint

The method="thread" provides parallel computation only if the black-box is releasing the global interpretor lock (GIL). Therefore, if you want parallelism in Jupyter notebooks you should use the Ray evaluator (method="ray") after installing Ray with pip install ray.

It is possible to define callbacks to extend the behaviour of Evaluator each time a function-evaluation is launched or completed. In this example we use the deephyper.evaluator.callback.TqdmCallback to follow the completed evaluations and the evolution of the objective with a progress-bar.

from deephyper.evaluator import Evaluator
from deephyper.evaluator.callback import TqdmCallback


# define the evaluator to distribute the computation
evaluator = Evaluator.create(
    f,
    method="thread",
    method_kwargs={
        "num_workers": 1,
        "callbacks": [TqdmCallback()]
    },
)

print(f"Evaluator has {evaluator.num_workers} available worker{'' if evaluator.num_workers == 1 else 's'}")

Evaluator has 1 available worker

Search Algorithm#

The next step is to define the search algorithm that we want to use. Here, we choose deephyper.hpo.CBO (Centralized Bayesian Optimization) which is a sampling based Bayesian optimization strategy. This algorithm has the advantage of being asynchronous which is crutial to keep a good utilization of the resources when the number of available workers increases. We also choose, how to optimize the acquisition function of the Bayesian optimization with "ga" (i.e., continuous Genetic Algorithm).

Then, we setup a solution selection method. Here we use deephyper.hpo.ArgMaxEstSelection, that will select the optimum based on the estimated maximum of a surrogate model. The model_grid_search=True activates the auto-tuning of the surrogate model every 100 observations by default. The noisy_objective=True sets the default configuration of the surrogate model for a noisy objective.

from deephyper.hpo import CBO, ArgMaxEstSelection


def create_search():
    search = CBO(
        problem,
        acq_optimizer="ga",
        solution_selection=ArgMaxEstSelection(
            problem,
            model_grid_search=True,
            noisy_objective=True,
        ),
    )
    return search

max_evals = 300
search = create_search()
results = search.search(evaluator, max_evals)

Results file already exists, it will be renamed to /Users/rp5/Documents/DeepHyper/deephyper/examples/examples_bbo/results_20251021-154659.csv

  0%|          | 0/300 [00:00<?, ?it/s]
  0%|          | 1/300 [00:00<00:00, 6250.83it/s, failures=0, objective=1]
  1%|          | 2/300 [00:00<01:27,  3.40it/s, failures=0, objective=1]
  1%|          | 2/300 [00:00<01:27,  3.40it/s, failures=0, objective=1]
  1%|          | 3/300 [00:01<02:01,  2.45it/s, failures=0, objective=1]
  1%|          | 3/300 [00:01<02:01,  2.45it/s, failures=0, objective=1]
  1%|▏         | 4/300 [00:01<02:38,  1.87it/s, failures=0, objective=1]
  1%|▏         | 4/300 [00:01<02:38,  1.87it/s, failures=0, objective=1]
  2%|▏         | 5/300 [00:02<02:54,  1.70it/s, failures=0, objective=1]
  2%|▏         | 5/300 [00:02<02:54,  1.70it/s, failures=0, objective=1]
  2%|▏         | 6/300 [00:03<03:03,  1.60it/s, failures=0, objective=1]
  2%|▏         | 6/300 [00:03<03:03,  1.60it/s, failures=0, objective=1]
  2%|▏         | 7/300 [00:03<03:09,  1.54it/s, failures=0, objective=1]
  2%|▏         | 7/300 [00:03<03:09,  1.54it/s, failures=0, objective=1]
  3%|▎         | 8/300 [00:04<03:18,  1.47it/s, failures=0, objective=1]
  3%|▎         | 8/300 [00:04<03:18,  1.47it/s, failures=0, objective=1]
  3%|▎         | 9/300 [00:05<03:20,  1.45it/s, failures=0, objective=1]
  3%|▎         | 9/300 [00:05<03:20,  1.45it/s, failures=0, objective=1]
  3%|▎         | 10/300 [00:06<03:20,  1.44it/s, failures=0, objective=1]
  3%|▎         | 10/300 [00:06<03:20,  1.44it/s, failures=0, objective=1]
  4%|▎         | 11/300 [00:06<03:20,  1.44it/s, failures=0, objective=1]
  4%|▎         | 11/300 [00:06<03:20,  1.44it/s, failures=0, objective=1]
  4%|▍         | 12/300 [00:07<03:24,  1.41it/s, failures=0, objective=1]
  4%|▍         | 12/300 [00:07<03:24,  1.41it/s, failures=0, objective=1]
  4%|▍         | 13/300 [00:08<03:22,  1.41it/s, failures=0, objective=1]
  4%|▍         | 13/300 [00:08<03:22,  1.41it/s, failures=0, objective=1]
  5%|▍         | 14/300 [00:08<03:21,  1.42it/s, failures=0, objective=1]
  5%|▍         | 14/300 [00:08<03:21,  1.42it/s, failures=0, objective=1]
  5%|▌         | 15/300 [00:09<03:25,  1.38it/s, failures=0, objective=1]
  5%|▌         | 15/300 [00:09<03:25,  1.38it/s, failures=0, objective=1]
  5%|▌         | 16/300 [00:10<03:23,  1.40it/s, failures=0, objective=1]
  5%|▌         | 16/300 [00:10<03:23,  1.40it/s, failures=0, objective=1]
  6%|▌         | 17/300 [00:11<03:21,  1.40it/s, failures=0, objective=1]
  6%|▌         | 17/300 [00:11<03:21,  1.40it/s, failures=0, objective=1]
  6%|▌         | 18/300 [00:11<03:20,  1.40it/s, failures=0, objective=1]
  6%|▌         | 18/300 [00:11<03:20,  1.40it/s, failures=0, objective=1]
  6%|▋         | 19/300 [00:12<03:23,  1.38it/s, failures=0, objective=1]
  6%|▋         | 19/300 [00:12<03:23,  1.38it/s, failures=0, objective=1]
  7%|▋         | 20/300 [00:13<03:21,  1.39it/s, failures=0, objective=1]
  7%|▋         | 20/300 [00:13<03:21,  1.39it/s, failures=0, objective=1]
  7%|▋         | 21/300 [00:14<03:19,  1.40it/s, failures=0, objective=1]
  7%|▋         | 21/300 [00:14<03:19,  1.40it/s, failures=0, objective=1]
  7%|▋         | 22/300 [00:14<03:16,  1.42it/s, failures=0, objective=1]
  7%|▋         | 22/300 [00:14<03:16,  1.42it/s, failures=0, objective=1]
  8%|▊         | 23/300 [00:15<03:20,  1.38it/s, failures=0, objective=1]
  8%|▊         | 23/300 [00:15<03:20,  1.38it/s, failures=0, objective=1]
  8%|▊         | 24/300 [00:16<03:18,  1.39it/s, failures=0, objective=1]
  8%|▊         | 24/300 [00:16<03:18,  1.39it/s, failures=0, objective=1]
  8%|▊         | 25/300 [00:16<03:15,  1.40it/s, failures=0, objective=1]
  8%|▊         | 25/300 [00:16<03:15,  1.40it/s, failures=0, objective=1]
  9%|▊         | 26/300 [00:17<03:14,  1.41it/s, failures=0, objective=1]
  9%|▊         | 26/300 [00:17<03:14,  1.41it/s, failures=0, objective=1]
  9%|▉         | 27/300 [00:18<03:18,  1.38it/s, failures=0, objective=1]
  9%|▉         | 27/300 [00:18<03:18,  1.38it/s, failures=0, objective=1]
  9%|▉         | 28/300 [00:19<03:15,  1.39it/s, failures=0, objective=1]
  9%|▉         | 28/300 [00:19<03:15,  1.39it/s, failures=0, objective=1]
 10%|▉         | 29/300 [00:19<03:12,  1.40it/s, failures=0, objective=1]
 10%|▉         | 29/300 [00:19<03:12,  1.40it/s, failures=0, objective=1]
 10%|█         | 30/300 [00:20<03:10,  1.42it/s, failures=0, objective=1]
 10%|█         | 30/300 [00:20<03:10,  1.42it/s, failures=0, objective=1]
 10%|█         | 31/300 [00:21<03:14,  1.38it/s, failures=0, objective=1]
 10%|█         | 31/300 [00:21<03:14,  1.38it/s, failures=0, objective=1]
 11%|█         | 32/300 [00:21<03:11,  1.40it/s, failures=0, objective=1]
 11%|█         | 32/300 [00:21<03:11,  1.40it/s, failures=0, objective=1]
 11%|█         | 33/300 [00:22<03:09,  1.41it/s, failures=0, objective=1]
 11%|█         | 33/300 [00:22<03:09,  1.41it/s, failures=0, objective=1]
 11%|█▏        | 34/300 [00:23<03:09,  1.41it/s, failures=0, objective=1]
 11%|█▏        | 34/300 [00:23<03:09,  1.41it/s, failures=0, objective=1]
 12%|█▏        | 35/300 [00:24<03:12,  1.38it/s, failures=0, objective=1]
 12%|█▏        | 35/300 [00:24<03:12,  1.38it/s, failures=0, objective=1]
 12%|█▏        | 36/300 [00:24<03:10,  1.39it/s, failures=0, objective=1]
 12%|█▏        | 36/300 [00:24<03:10,  1.39it/s, failures=0, objective=1]
 12%|█▏        | 37/300 [00:25<03:09,  1.39it/s, failures=0, objective=1]
 12%|█▏        | 37/300 [00:25<03:09,  1.39it/s, failures=0, objective=1]
 13%|█▎        | 38/300 [00:26<03:07,  1.40it/s, failures=0, objective=1]
 13%|█▎        | 38/300 [00:26<03:07,  1.40it/s, failures=0, objective=1]
 13%|█▎        | 39/300 [00:26<03:10,  1.37it/s, failures=0, objective=1]
 13%|█▎        | 39/300 [00:26<03:10,  1.37it/s, failures=0, objective=1]
 13%|█▎        | 40/300 [00:27<03:06,  1.39it/s, failures=0, objective=1]
 13%|█▎        | 40/300 [00:27<03:06,  1.39it/s, failures=0, objective=1]
 14%|█▎        | 41/300 [00:28<03:04,  1.40it/s, failures=0, objective=1]
 14%|█▎        | 41/300 [00:28<03:04,  1.40it/s, failures=0, objective=1]
 14%|█▍        | 42/300 [00:29<03:03,  1.41it/s, failures=0, objective=1]
 14%|█▍        | 42/300 [00:29<03:03,  1.41it/s, failures=0, objective=1]
 14%|█▍        | 43/300 [00:29<03:05,  1.38it/s, failures=0, objective=1]
 14%|█▍        | 43/300 [00:29<03:05,  1.38it/s, failures=0, objective=1]
 15%|█▍        | 44/300 [00:30<03:03,  1.39it/s, failures=0, objective=1]
 15%|█▍        | 44/300 [00:30<03:03,  1.39it/s, failures=0, objective=1]
 15%|█▌        | 45/300 [00:31<03:02,  1.40it/s, failures=0, objective=1]
 15%|█▌        | 45/300 [00:31<03:02,  1.40it/s, failures=0, objective=1]
 15%|█▌        | 46/300 [00:31<03:00,  1.40it/s, failures=0, objective=1]
 15%|█▌        | 46/300 [00:31<03:00,  1.40it/s, failures=0, objective=1]
 16%|█▌        | 47/300 [00:32<03:02,  1.38it/s, failures=0, objective=1]
 16%|█▌        | 47/300 [00:32<03:02,  1.38it/s, failures=0, objective=1]
 16%|█▌        | 48/300 [00:33<03:01,  1.38it/s, failures=0, objective=1]
 16%|█▌        | 48/300 [00:33<03:01,  1.38it/s, failures=0, objective=1]
 16%|█▋        | 49/300 [00:34<03:00,  1.39it/s, failures=0, objective=1]
 16%|█▋        | 49/300 [00:34<03:00,  1.39it/s, failures=0, objective=1]
 17%|█▋        | 50/300 [00:34<02:57,  1.41it/s, failures=0, objective=1]
 17%|█▋        | 50/300 [00:34<02:57,  1.41it/s, failures=0, objective=1]
 17%|█▋        | 51/300 [00:35<03:01,  1.37it/s, failures=0, objective=1]
 17%|█▋        | 51/300 [00:35<03:01,  1.37it/s, failures=0, objective=1]
 17%|█▋        | 52/300 [00:36<02:58,  1.39it/s, failures=0, objective=1]
 17%|█▋        | 52/300 [00:36<02:58,  1.39it/s, failures=0, objective=1]
 18%|█▊        | 53/300 [00:36<02:56,  1.40it/s, failures=0, objective=1]
 18%|█▊        | 53/300 [00:36<02:56,  1.40it/s, failures=0, objective=1]
 18%|█▊        | 54/300 [00:37<02:55,  1.40it/s, failures=0, objective=1]
 18%|█▊        | 54/300 [00:37<02:55,  1.40it/s, failures=0, objective=1]
 18%|█▊        | 55/300 [00:38<02:57,  1.38it/s, failures=0, objective=1]
 18%|█▊        | 55/300 [00:38<02:57,  1.38it/s, failures=0, objective=1]
 19%|█▊        | 56/300 [00:39<02:55,  1.39it/s, failures=0, objective=1]
 19%|█▊        | 56/300 [00:39<02:55,  1.39it/s, failures=0, objective=1]
 19%|█▉        | 57/300 [00:39<02:53,  1.40it/s, failures=0, objective=1]
 19%|█▉        | 57/300 [00:39<02:53,  1.40it/s, failures=0, objective=1]
 19%|█▉        | 58/300 [00:40<02:56,  1.37it/s, failures=0, objective=1]
 19%|█▉        | 58/300 [00:40<02:56,  1.37it/s, failures=0, objective=1]
 20%|█▉        | 59/300 [00:41<02:53,  1.39it/s, failures=0, objective=1]
 20%|█▉        | 59/300 [00:41<02:53,  1.39it/s, failures=0, objective=1]
 20%|██        | 60/300 [00:42<02:51,  1.40it/s, failures=0, objective=1]
 20%|██        | 60/300 [00:42<02:51,  1.40it/s, failures=0, objective=1]
 20%|██        | 61/300 [00:42<02:49,  1.41it/s, failures=0, objective=1]
 20%|██        | 61/300 [00:42<02:49,  1.41it/s, failures=0, objective=1]
 21%|██        | 62/300 [00:43<02:54,  1.37it/s, failures=0, objective=1]
 21%|██        | 62/300 [00:43<02:54,  1.37it/s, failures=0, objective=1]
 21%|██        | 63/300 [00:44<02:51,  1.38it/s, failures=0, objective=1]
 21%|██        | 63/300 [00:44<02:51,  1.38it/s, failures=0, objective=1]
 21%|██▏       | 64/300 [00:44<02:49,  1.40it/s, failures=0, objective=1]
 21%|██▏       | 64/300 [00:44<02:49,  1.40it/s, failures=0, objective=1]
 22%|██▏       | 65/300 [00:45<02:46,  1.41it/s, failures=0, objective=1]
 22%|██▏       | 65/300 [00:45<02:46,  1.41it/s, failures=0, objective=1]
 22%|██▏       | 66/300 [00:46<02:51,  1.37it/s, failures=0, objective=1]
 22%|██▏       | 66/300 [00:46<02:51,  1.37it/s, failures=0, objective=1]
 22%|██▏       | 67/300 [00:47<02:49,  1.38it/s, failures=0, objective=1]
 22%|██▏       | 67/300 [00:47<02:49,  1.38it/s, failures=0, objective=1]
 23%|██▎       | 68/300 [00:47<02:46,  1.39it/s, failures=0, objective=1]
 23%|██▎       | 68/300 [00:47<02:46,  1.39it/s, failures=0, objective=1]
 23%|██▎       | 69/300 [00:48<02:45,  1.39it/s, failures=0, objective=1]
 23%|██▎       | 69/300 [00:48<02:45,  1.39it/s, failures=0, objective=1]
 23%|██▎       | 70/300 [00:49<02:50,  1.35it/s, failures=0, objective=1]
 23%|██▎       | 70/300 [00:49<02:50,  1.35it/s, failures=0, objective=1]
 24%|██▎       | 71/300 [00:50<02:46,  1.38it/s, failures=0, objective=1]
 24%|██▎       | 71/300 [00:50<02:46,  1.38it/s, failures=0, objective=1]
 24%|██▍       | 72/300 [00:50<02:43,  1.39it/s, failures=0, objective=1]
 24%|██▍       | 72/300 [00:50<02:43,  1.39it/s, failures=0, objective=1]
 24%|██▍       | 73/300 [00:51<02:45,  1.37it/s, failures=0, objective=1]
 24%|██▍       | 73/300 [00:51<02:45,  1.37it/s, failures=0, objective=1]
 25%|██▍       | 74/300 [00:52<02:43,  1.39it/s, failures=0, objective=1]
 25%|██▍       | 74/300 [00:52<02:43,  1.39it/s, failures=0, objective=1]
 25%|██▌       | 75/300 [00:52<02:41,  1.39it/s, failures=0, objective=1]
 25%|██▌       | 75/300 [00:52<02:41,  1.39it/s, failures=0, objective=1]
 25%|██▌       | 76/300 [00:53<02:40,  1.40it/s, failures=0, objective=1]
 25%|██▌       | 76/300 [00:53<02:40,  1.40it/s, failures=0, objective=1]
 26%|██▌       | 77/300 [00:54<02:43,  1.37it/s, failures=0, objective=1]
 26%|██▌       | 77/300 [00:54<02:43,  1.37it/s, failures=0, objective=1]
 26%|██▌       | 78/300 [00:55<02:40,  1.38it/s, failures=0, objective=1]
 26%|██▌       | 78/300 [00:55<02:40,  1.38it/s, failures=0, objective=1]
 26%|██▋       | 79/300 [00:55<02:38,  1.40it/s, failures=0, objective=1]
 26%|██▋       | 79/300 [00:55<02:38,  1.40it/s, failures=0, objective=1]
 27%|██▋       | 80/300 [00:56<02:36,  1.40it/s, failures=0, objective=1]
 27%|██▋       | 80/300 [00:56<02:36,  1.40it/s, failures=0, objective=1]
 27%|██▋       | 81/300 [00:57<02:39,  1.37it/s, failures=0, objective=1]
 27%|██▋       | 81/300 [00:57<02:39,  1.37it/s, failures=0, objective=1]
 27%|██▋       | 82/300 [00:57<02:36,  1.39it/s, failures=0, objective=1]
 27%|██▋       | 82/300 [00:57<02:36,  1.39it/s, failures=0, objective=1]
 28%|██▊       | 83/300 [00:58<02:35,  1.40it/s, failures=0, objective=1]
 28%|██▊       | 83/300 [00:58<02:35,  1.40it/s, failures=0, objective=1]
 28%|██▊       | 84/300 [00:59<02:34,  1.40it/s, failures=0, objective=1]
 28%|██▊       | 84/300 [00:59<02:34,  1.40it/s, failures=0, objective=1]
 28%|██▊       | 85/300 [01:00<02:36,  1.37it/s, failures=0, objective=1]
 28%|██▊       | 85/300 [01:00<02:36,  1.37it/s, failures=0, objective=1]
 29%|██▊       | 86/300 [01:00<02:34,  1.38it/s, failures=0, objective=1]
 29%|██▊       | 86/300 [01:00<02:34,  1.38it/s, failures=0, objective=1]
 29%|██▉       | 87/300 [01:01<02:34,  1.38it/s, failures=0, objective=1]
 29%|██▉       | 87/300 [01:01<02:34,  1.38it/s, failures=0, objective=1]
 29%|██▉       | 88/300 [01:02<02:32,  1.39it/s, failures=0, objective=1]
 29%|██▉       | 88/300 [01:02<02:32,  1.39it/s, failures=0, objective=1]
 30%|██▉       | 89/300 [01:03<02:34,  1.36it/s, failures=0, objective=1]
 30%|██▉       | 89/300 [01:03<02:34,  1.36it/s, failures=0, objective=1]
 30%|███       | 90/300 [01:03<02:31,  1.38it/s, failures=0, objective=1]
 30%|███       | 90/300 [01:03<02:31,  1.38it/s, failures=0, objective=1]
 30%|███       | 91/300 [01:04<02:30,  1.39it/s, failures=0, objective=1]
 30%|███       | 91/300 [01:04<02:30,  1.39it/s, failures=0, objective=1]
 31%|███       | 92/300 [01:05<02:32,  1.37it/s, failures=0, objective=1]
 31%|███       | 92/300 [01:05<02:32,  1.37it/s, failures=0, objective=1]
 31%|███       | 93/300 [01:05<02:30,  1.38it/s, failures=0, objective=1]
 31%|███       | 93/300 [01:05<02:30,  1.38it/s, failures=0, objective=1]
 31%|███▏      | 94/300 [01:06<02:28,  1.39it/s, failures=0, objective=1]
 31%|███▏      | 94/300 [01:06<02:28,  1.39it/s, failures=0, objective=1]
 32%|███▏      | 95/300 [01:07<02:26,  1.40it/s, failures=0, objective=1]
 32%|███▏      | 95/300 [01:07<02:26,  1.40it/s, failures=0, objective=1]
 32%|███▏      | 96/300 [01:08<02:28,  1.37it/s, failures=0, objective=1]
 32%|███▏      | 96/300 [01:08<02:28,  1.37it/s, failures=0, objective=1]
 32%|███▏      | 97/300 [01:08<02:26,  1.38it/s, failures=0, objective=1]
 32%|███▏      | 97/300 [01:08<02:26,  1.38it/s, failures=0, objective=1]
 33%|███▎      | 98/300 [01:09<02:25,  1.39it/s, failures=0, objective=1]
 33%|███▎      | 98/300 [01:09<02:25,  1.39it/s, failures=0, objective=1]
 33%|███▎      | 99/300 [01:10<02:23,  1.40it/s, failures=0, objective=1]
 33%|███▎      | 99/300 [01:10<02:23,  1.40it/s, failures=0, objective=1]
 33%|███▎      | 100/300 [01:10<02:27,  1.36it/s, failures=0, objective=1]
 33%|███▎      | 100/300 [01:10<02:27,  1.36it/s, failures=0, objective=1]Tuning selection model...
Tuned model parameters: {'bootstrap': False, 'max_depth': None, 'min_samples_leaf': 1, 'min_samples_split': 16, 'n_estimators': 100, 'splitter': 'random'}
Tuned model scores: {'r2': 0.616854946090871, 'r2_upper_bound': 0.563968797128797, 'y_std_corr': {'statistic': 0.5640805681606513, 'pvalue': 9.876830200563737e-10}}

 34%|███▎      | 101/300 [01:20<10:42,  3.23s/it, failures=0, objective=1]
 34%|███▎      | 101/300 [01:20<10:42,  3.23s/it, failures=0, objective=1]
 34%|███▍      | 102/300 [01:20<08:09,  2.47s/it, failures=0, objective=1]
 34%|███▍      | 102/300 [01:20<08:09,  2.47s/it, failures=0, objective=1]
 34%|███▍      | 103/300 [01:21<06:26,  1.96s/it, failures=0, objective=1]
 34%|███▍      | 103/300 [01:21<06:26,  1.96s/it, failures=0, objective=1]
 35%|███▍      | 104/300 [01:22<05:10,  1.59s/it, failures=0, objective=1]
 35%|███▍      | 104/300 [01:22<05:10,  1.59s/it, failures=0, objective=1]
 35%|███▌      | 105/300 [01:22<04:17,  1.32s/it, failures=0, objective=1]
 35%|███▌      | 105/300 [01:22<04:17,  1.32s/it, failures=0, objective=1]
 35%|███▌      | 106/300 [01:23<03:39,  1.13s/it, failures=0, objective=1]
 35%|███▌      | 106/300 [01:23<03:39,  1.13s/it, failures=0, objective=1]
 36%|███▌      | 107/300 [01:24<03:16,  1.02s/it, failures=0, objective=1]
 36%|███▌      | 107/300 [01:24<03:16,  1.02s/it, failures=0, objective=1]
 36%|███▌      | 108/300 [01:25<02:57,  1.08it/s, failures=0, objective=1]
 36%|███▌      | 108/300 [01:25<02:57,  1.08it/s, failures=0, objective=1]
 36%|███▋      | 109/300 [01:25<02:44,  1.16it/s, failures=0, objective=1]
 36%|███▋      | 109/300 [01:25<02:44,  1.16it/s, failures=0, objective=1]
 37%|███▋      | 110/300 [01:26<02:37,  1.21it/s, failures=0, objective=1]
 37%|███▋      | 110/300 [01:26<02:37,  1.21it/s, failures=0, objective=1]
 37%|███▋      | 111/300 [01:27<02:28,  1.27it/s, failures=0, objective=1]
 37%|███▋      | 111/300 [01:27<02:28,  1.27it/s, failures=0, objective=1]
 37%|███▋      | 112/300 [01:27<02:23,  1.31it/s, failures=0, objective=1]
 37%|███▋      | 112/300 [01:27<02:23,  1.31it/s, failures=0, objective=1]
 38%|███▊      | 113/300 [01:28<02:18,  1.35it/s, failures=0, objective=1]
 38%|███▊      | 113/300 [01:28<02:18,  1.35it/s, failures=0, objective=1]
 38%|███▊      | 114/300 [01:29<02:19,  1.33it/s, failures=0, objective=1]
 38%|███▊      | 114/300 [01:29<02:19,  1.33it/s, failures=0, objective=1]
 38%|███▊      | 115/300 [01:30<02:15,  1.36it/s, failures=0, objective=1]
 38%|███▊      | 115/300 [01:30<02:15,  1.36it/s, failures=0, objective=1]
 39%|███▊      | 116/300 [01:30<02:12,  1.38it/s, failures=0, objective=1]
 39%|███▊      | 116/300 [01:30<02:12,  1.38it/s, failures=0, objective=1]
 39%|███▉      | 117/300 [01:31<02:11,  1.39it/s, failures=0, objective=1]
 39%|███▉      | 117/300 [01:31<02:11,  1.39it/s, failures=0, objective=1]
 39%|███▉      | 118/300 [01:32<02:14,  1.35it/s, failures=0, objective=1]
 39%|███▉      | 118/300 [01:32<02:14,  1.35it/s, failures=0, objective=1]
 40%|███▉      | 119/300 [01:32<02:12,  1.37it/s, failures=0, objective=1]
 40%|███▉      | 119/300 [01:32<02:12,  1.37it/s, failures=0, objective=1]
 40%|████      | 120/300 [01:33<02:10,  1.38it/s, failures=0, objective=1]
 40%|████      | 120/300 [01:33<02:10,  1.38it/s, failures=0, objective=1]
 40%|████      | 121/300 [01:34<02:07,  1.40it/s, failures=0, objective=1]
 40%|████      | 121/300 [01:34<02:07,  1.40it/s, failures=0, objective=1]
 41%|████      | 122/300 [01:35<02:09,  1.37it/s, failures=0, objective=1]
 41%|████      | 122/300 [01:35<02:09,  1.37it/s, failures=0, objective=1]
 41%|████      | 123/300 [01:35<02:07,  1.39it/s, failures=0, objective=1]
 41%|████      | 123/300 [01:35<02:07,  1.39it/s, failures=0, objective=1]
 41%|████▏     | 124/300 [01:36<02:05,  1.40it/s, failures=0, objective=1]
 41%|████▏     | 124/300 [01:36<02:05,  1.40it/s, failures=0, objective=1]
 42%|████▏     | 125/300 [01:37<02:08,  1.37it/s, failures=0, objective=1]
 42%|████▏     | 125/300 [01:37<02:08,  1.37it/s, failures=0, objective=1]
 42%|████▏     | 126/300 [01:38<02:05,  1.39it/s, failures=0, objective=1]
 42%|████▏     | 126/300 [01:38<02:05,  1.39it/s, failures=0, objective=1]
 42%|████▏     | 127/300 [01:38<02:03,  1.40it/s, failures=0, objective=1]
 42%|████▏     | 127/300 [01:38<02:03,  1.40it/s, failures=0, objective=1]
 43%|████▎     | 128/300 [01:39<02:02,  1.41it/s, failures=0, objective=1]
 43%|████▎     | 128/300 [01:39<02:02,  1.41it/s, failures=0, objective=1]
 43%|████▎     | 129/300 [01:40<02:04,  1.38it/s, failures=0, objective=1]
 43%|████▎     | 129/300 [01:40<02:04,  1.38it/s, failures=0, objective=1]
 43%|████▎     | 130/300 [01:40<02:02,  1.39it/s, failures=0, objective=1]
 43%|████▎     | 130/300 [01:40<02:02,  1.39it/s, failures=0, objective=1]
 44%|████▎     | 131/300 [01:41<02:00,  1.40it/s, failures=0, objective=1]
 44%|████▎     | 131/300 [01:41<02:00,  1.40it/s, failures=0, objective=1]
 44%|████▍     | 132/300 [01:42<01:58,  1.41it/s, failures=0, objective=1]
 44%|████▍     | 132/300 [01:42<01:58,  1.41it/s, failures=0, objective=1]
 44%|████▍     | 133/300 [01:43<02:01,  1.38it/s, failures=0, objective=1]
 44%|████▍     | 133/300 [01:43<02:01,  1.38it/s, failures=0, objective=1]
 45%|████▍     | 134/300 [01:43<01:59,  1.39it/s, failures=0, objective=1]
 45%|████▍     | 134/300 [01:43<01:59,  1.39it/s, failures=0, objective=1]
 45%|████▌     | 135/300 [01:44<01:58,  1.40it/s, failures=0, objective=1]
 45%|████▌     | 135/300 [01:44<01:58,  1.40it/s, failures=0, objective=1]
 45%|████▌     | 136/300 [01:45<01:56,  1.40it/s, failures=0, objective=1]
 45%|████▌     | 136/300 [01:45<01:56,  1.40it/s, failures=0, objective=1]
 46%|████▌     | 137/300 [01:45<01:58,  1.38it/s, failures=0, objective=1]
 46%|████▌     | 137/300 [01:45<01:58,  1.38it/s, failures=0, objective=1]
 46%|████▌     | 138/300 [01:46<01:56,  1.39it/s, failures=0, objective=1]
 46%|████▌     | 138/300 [01:46<01:56,  1.39it/s, failures=0, objective=1]
 46%|████▋     | 139/300 [01:47<01:55,  1.40it/s, failures=0, objective=1]
 46%|████▋     | 139/300 [01:47<01:55,  1.40it/s, failures=0, objective=1]
 47%|████▋     | 140/300 [01:48<01:56,  1.38it/s, failures=0, objective=1]
 47%|████▋     | 140/300 [01:48<01:56,  1.38it/s, failures=0, objective=1]
 47%|████▋     | 141/300 [01:48<01:53,  1.40it/s, failures=0, objective=1]
 47%|████▋     | 141/300 [01:48<01:53,  1.40it/s, failures=0, objective=1]
 47%|████▋     | 142/300 [01:49<01:52,  1.40it/s, failures=0, objective=1]
 47%|████▋     | 142/300 [01:49<01:52,  1.40it/s, failures=0, objective=1]
 48%|████▊     | 143/300 [01:50<01:51,  1.41it/s, failures=0, objective=1]
 48%|████▊     | 143/300 [01:50<01:51,  1.41it/s, failures=0, objective=1]
 48%|████▊     | 144/300 [01:50<01:52,  1.38it/s, failures=0, objective=1]
 48%|████▊     | 144/300 [01:50<01:52,  1.38it/s, failures=0, objective=1]
 48%|████▊     | 145/300 [01:51<01:51,  1.39it/s, failures=0, objective=1]
 48%|████▊     | 145/300 [01:51<01:51,  1.39it/s, failures=0, objective=1]
 49%|████▊     | 146/300 [01:52<01:49,  1.40it/s, failures=0, objective=1]
 49%|████▊     | 146/300 [01:52<01:49,  1.40it/s, failures=0, objective=1]
 49%|████▉     | 147/300 [01:53<01:51,  1.38it/s, failures=0, objective=1]
 49%|████▉     | 147/300 [01:53<01:51,  1.38it/s, failures=0, objective=1]
 49%|████▉     | 148/300 [01:53<01:49,  1.39it/s, failures=0, objective=1]
 49%|████▉     | 148/300 [01:53<01:49,  1.39it/s, failures=0, objective=1]
 50%|████▉     | 149/300 [01:54<01:47,  1.40it/s, failures=0, objective=1]
 50%|████▉     | 149/300 [01:54<01:47,  1.40it/s, failures=0, objective=1]
 50%|█████     | 150/300 [01:55<01:46,  1.41it/s, failures=0, objective=1]
 50%|█████     | 150/300 [01:55<01:46,  1.41it/s, failures=0, objective=1]
 50%|█████     | 151/300 [01:55<01:47,  1.38it/s, failures=0, objective=1]
 50%|█████     | 151/300 [01:55<01:47,  1.38it/s, failures=0, objective=1]
 51%|█████     | 152/300 [01:56<01:46,  1.39it/s, failures=0, objective=1]
 51%|█████     | 152/300 [01:56<01:46,  1.39it/s, failures=0, objective=1]
 51%|█████     | 153/300 [01:57<01:45,  1.40it/s, failures=0, objective=1]
 51%|█████     | 153/300 [01:57<01:45,  1.40it/s, failures=0, objective=1]
 51%|█████▏    | 154/300 [01:58<01:44,  1.40it/s, failures=0, objective=1]
 51%|█████▏    | 154/300 [01:58<01:44,  1.40it/s, failures=0, objective=1]
 52%|█████▏    | 155/300 [01:58<01:45,  1.37it/s, failures=0, objective=1]
 52%|█████▏    | 155/300 [01:58<01:45,  1.37it/s, failures=0, objective=1]
 52%|█████▏    | 156/300 [01:59<01:43,  1.39it/s, failures=0, objective=1]
 52%|█████▏    | 156/300 [01:59<01:43,  1.39it/s, failures=0, objective=1]
 52%|█████▏    | 157/300 [02:00<01:41,  1.40it/s, failures=0, objective=1]
 52%|█████▏    | 157/300 [02:00<01:41,  1.40it/s, failures=0, objective=1]
 53%|█████▎    | 158/300 [02:00<01:41,  1.40it/s, failures=0, objective=1]
 53%|█████▎    | 158/300 [02:00<01:41,  1.40it/s, failures=0, objective=1]
 53%|█████▎    | 159/300 [02:01<01:42,  1.37it/s, failures=0, objective=1]
 53%|█████▎    | 159/300 [02:01<01:42,  1.37it/s, failures=0, objective=1]
 53%|█████▎    | 160/300 [02:02<01:40,  1.39it/s, failures=0, objective=1]
 53%|█████▎    | 160/300 [02:02<01:40,  1.39it/s, failures=0, objective=1]
 54%|█████▎    | 161/300 [02:03<01:39,  1.40it/s, failures=0, objective=1]
 54%|█████▎    | 161/300 [02:03<01:39,  1.40it/s, failures=0, objective=1]
 54%|█████▍    | 162/300 [02:03<01:37,  1.41it/s, failures=0, objective=1]
 54%|█████▍    | 162/300 [02:03<01:37,  1.41it/s, failures=0, objective=1]
 54%|█████▍    | 163/300 [02:04<01:39,  1.38it/s, failures=0, objective=1]
 54%|█████▍    | 163/300 [02:04<01:39,  1.38it/s, failures=0, objective=1]
 55%|█████▍    | 164/300 [02:05<01:36,  1.40it/s, failures=0, objective=1]
 55%|█████▍    | 164/300 [02:05<01:36,  1.40it/s, failures=0, objective=1]
 55%|█████▌    | 165/300 [02:05<01:35,  1.41it/s, failures=0, objective=1]
 55%|█████▌    | 165/300 [02:05<01:35,  1.41it/s, failures=0, objective=1]
 55%|█████▌    | 166/300 [02:06<01:37,  1.37it/s, failures=0, objective=1]
 55%|█████▌    | 166/300 [02:06<01:37,  1.37it/s, failures=0, objective=1]
 56%|█████▌    | 167/300 [02:07<01:35,  1.39it/s, failures=0, objective=1]
 56%|█████▌    | 167/300 [02:07<01:35,  1.39it/s, failures=0, objective=1]
 56%|█████▌    | 168/300 [02:08<01:34,  1.40it/s, failures=0, objective=1]
 56%|█████▌    | 168/300 [02:08<01:34,  1.40it/s, failures=0, objective=1]
 56%|█████▋    | 169/300 [02:08<01:32,  1.41it/s, failures=0, objective=1]
 56%|█████▋    | 169/300 [02:08<01:32,  1.41it/s, failures=0, objective=1]
 57%|█████▋    | 170/300 [02:09<01:33,  1.38it/s, failures=0, objective=1]
 57%|█████▋    | 170/300 [02:09<01:33,  1.38it/s, failures=0, objective=1]
 57%|█████▋    | 171/300 [02:10<01:32,  1.40it/s, failures=0, objective=1]
 57%|█████▋    | 171/300 [02:10<01:32,  1.40it/s, failures=0, objective=1]
 57%|█████▋    | 172/300 [02:10<01:31,  1.40it/s, failures=0, objective=1]
 57%|█████▋    | 172/300 [02:10<01:31,  1.40it/s, failures=0, objective=1]
 58%|█████▊    | 173/300 [02:11<01:30,  1.41it/s, failures=0, objective=1]
 58%|█████▊    | 173/300 [02:11<01:30,  1.41it/s, failures=0, objective=1]
 58%|█████▊    | 174/300 [02:12<01:31,  1.38it/s, failures=0, objective=1]
 58%|█████▊    | 174/300 [02:12<01:31,  1.38it/s, failures=0, objective=1]
 58%|█████▊    | 175/300 [02:13<01:29,  1.40it/s, failures=0, objective=1]
 58%|█████▊    | 175/300 [02:13<01:29,  1.40it/s, failures=0, objective=1]
 59%|█████▊    | 176/300 [02:13<01:27,  1.41it/s, failures=0, objective=1]
 59%|█████▊    | 176/300 [02:13<01:27,  1.41it/s, failures=0, objective=1]
 59%|█████▉    | 177/300 [02:14<01:27,  1.41it/s, failures=0, objective=1]
 59%|█████▉    | 177/300 [02:14<01:27,  1.41it/s, failures=0, objective=1]
 59%|█████▉    | 178/300 [02:15<01:27,  1.39it/s, failures=0, objective=1]
 59%|█████▉    | 178/300 [02:15<01:27,  1.39it/s, failures=0, objective=1]
 60%|█████▉    | 179/300 [02:15<01:26,  1.40it/s, failures=0, objective=1]
 60%|█████▉    | 179/300 [02:15<01:26,  1.40it/s, failures=0, objective=1]
 60%|██████    | 180/300 [02:16<01:25,  1.41it/s, failures=0, objective=1]
 60%|██████    | 180/300 [02:16<01:25,  1.41it/s, failures=0, objective=1]
 60%|██████    | 181/300 [02:17<01:26,  1.38it/s, failures=0, objective=1]
 60%|██████    | 181/300 [02:17<01:26,  1.38it/s, failures=0, objective=1]
 61%|██████    | 182/300 [02:18<01:24,  1.40it/s, failures=0, objective=1]
 61%|██████    | 182/300 [02:18<01:24,  1.40it/s, failures=0, objective=1]
 61%|██████    | 183/300 [02:18<01:23,  1.41it/s, failures=0, objective=1]
 61%|██████    | 183/300 [02:18<01:23,  1.41it/s, failures=0, objective=1]
 61%|██████▏   | 184/300 [02:19<01:21,  1.42it/s, failures=0, objective=1]
 61%|██████▏   | 184/300 [02:19<01:21,  1.42it/s, failures=0, objective=1]
 62%|██████▏   | 185/300 [02:20<01:23,  1.38it/s, failures=0, objective=1]
 62%|██████▏   | 185/300 [02:20<01:23,  1.38it/s, failures=0, objective=1]
 62%|██████▏   | 186/300 [02:21<01:21,  1.39it/s, failures=0, objective=1]
 62%|██████▏   | 186/300 [02:21<01:21,  1.39it/s, failures=0, objective=1]
 62%|██████▏   | 187/300 [02:21<01:20,  1.40it/s, failures=0, objective=1]
 62%|██████▏   | 187/300 [02:21<01:20,  1.40it/s, failures=0, objective=1]
 63%|██████▎   | 188/300 [02:22<01:19,  1.40it/s, failures=0, objective=1]
 63%|██████▎   | 188/300 [02:22<01:19,  1.40it/s, failures=0, objective=1]
 63%|██████▎   | 189/300 [02:23<01:20,  1.38it/s, failures=0, objective=1]
 63%|██████▎   | 189/300 [02:23<01:20,  1.38it/s, failures=0, objective=1]
 63%|██████▎   | 190/300 [02:23<01:18,  1.39it/s, failures=0, objective=1]
 63%|██████▎   | 190/300 [02:23<01:18,  1.39it/s, failures=0, objective=1]
 64%|██████▎   | 191/300 [02:24<01:17,  1.41it/s, failures=0, objective=1]
 64%|██████▎   | 191/300 [02:24<01:17,  1.41it/s, failures=0, objective=1]
 64%|██████▍   | 192/300 [02:25<01:16,  1.41it/s, failures=0, objective=1]
 64%|██████▍   | 192/300 [02:25<01:16,  1.41it/s, failures=0, objective=1]
 64%|██████▍   | 193/300 [02:26<01:17,  1.38it/s, failures=0, objective=1]
 64%|██████▍   | 193/300 [02:26<01:17,  1.38it/s, failures=0, objective=1]
 65%|██████▍   | 194/300 [02:26<01:15,  1.40it/s, failures=0, objective=1]
 65%|██████▍   | 194/300 [02:26<01:15,  1.40it/s, failures=0, objective=1]
 65%|██████▌   | 195/300 [02:27<01:14,  1.41it/s, failures=0, objective=1]
 65%|██████▌   | 195/300 [02:27<01:14,  1.41it/s, failures=0, objective=1]
 65%|██████▌   | 196/300 [02:28<01:14,  1.39it/s, failures=0, objective=1]
 65%|██████▌   | 196/300 [02:28<01:14,  1.39it/s, failures=0, objective=1]
 66%|██████▌   | 197/300 [02:28<01:13,  1.40it/s, failures=0, objective=1]
 66%|██████▌   | 197/300 [02:28<01:13,  1.40it/s, failures=0, objective=1]
 66%|██████▌   | 198/300 [02:29<01:12,  1.41it/s, failures=0, objective=1]
 66%|██████▌   | 198/300 [02:29<01:12,  1.41it/s, failures=0, objective=1]
 66%|██████▋   | 199/300 [02:30<01:11,  1.41it/s, failures=0, objective=1]
 66%|██████▋   | 199/300 [02:30<01:11,  1.41it/s, failures=0, objective=1]
 67%|██████▋   | 200/300 [02:31<01:12,  1.38it/s, failures=0, objective=1]
 67%|██████▋   | 200/300 [02:31<01:12,  1.38it/s, failures=0, objective=1]Tuning selection model...
Tuned model parameters: {'bootstrap': True, 'max_depth': None, 'min_samples_leaf': 1, 'min_samples_split': 16, 'n_estimators': 100, 'splitter': 'random'}
Tuned model scores: {'r2': 0.5364447759757613, 'r2_upper_bound': 0.5281183982243103, 'y_std_corr': {'statistic': 0.5169270064934015, 'pvalue': 4.6412103962294715e-15}}

 67%|██████▋   | 201/300 [02:44<07:24,  4.49s/it, failures=0, objective=1]
 67%|██████▋   | 201/300 [02:44<07:24,  4.49s/it, failures=0, objective=1]
 67%|██████▋   | 202/300 [02:45<05:29,  3.36s/it, failures=0, objective=1]
 67%|██████▋   | 202/300 [02:45<05:29,  3.36s/it, failures=0, objective=1]
 68%|██████▊   | 203/300 [02:45<04:10,  2.58s/it, failures=0, objective=1]
 68%|██████▊   | 203/300 [02:45<04:10,  2.58s/it, failures=0, objective=1]
 68%|██████▊   | 204/300 [02:46<03:14,  2.02s/it, failures=0, objective=1]
 68%|██████▊   | 204/300 [02:46<03:14,  2.02s/it, failures=0, objective=1]
 68%|██████▊   | 205/300 [02:47<02:35,  1.63s/it, failures=0, objective=1]
 68%|██████▊   | 205/300 [02:47<02:35,  1.63s/it, failures=0, objective=1]
 69%|██████▊   | 206/300 [02:48<02:09,  1.37s/it, failures=0, objective=1]
 69%|██████▊   | 206/300 [02:48<02:09,  1.37s/it, failures=0, objective=1]
 69%|██████▉   | 207/300 [02:48<01:49,  1.18s/it, failures=0, objective=1]
 69%|██████▉   | 207/300 [02:48<01:49,  1.18s/it, failures=0, objective=1]
 69%|██████▉   | 208/300 [02:49<01:35,  1.04s/it, failures=0, objective=1]
 69%|██████▉   | 208/300 [02:49<01:35,  1.04s/it, failures=0, objective=1]
 70%|██████▉   | 209/300 [02:50<01:25,  1.06it/s, failures=0, objective=1]
 70%|██████▉   | 209/300 [02:50<01:25,  1.06it/s, failures=0, objective=1]
 70%|███████   | 210/300 [02:50<01:20,  1.12it/s, failures=0, objective=1]
 70%|███████   | 210/300 [02:50<01:20,  1.12it/s, failures=0, objective=1]
 70%|███████   | 211/300 [02:51<01:14,  1.19it/s, failures=0, objective=1]
 70%|███████   | 211/300 [02:51<01:14,  1.19it/s, failures=0, objective=1]
 71%|███████   | 212/300 [02:52<01:10,  1.25it/s, failures=0, objective=1]
 71%|███████   | 212/300 [02:52<01:10,  1.25it/s, failures=0, objective=1]
 71%|███████   | 213/300 [02:53<01:07,  1.29it/s, failures=0, objective=1]
 71%|███████   | 213/300 [02:53<01:07,  1.29it/s, failures=0, objective=1]
 71%|███████▏  | 214/300 [02:53<01:06,  1.29it/s, failures=0, objective=1]
 71%|███████▏  | 214/300 [02:53<01:06,  1.29it/s, failures=0, objective=1]
 72%|███████▏  | 215/300 [02:54<01:04,  1.32it/s, failures=0, objective=1]
 72%|███████▏  | 215/300 [02:54<01:04,  1.32it/s, failures=0, objective=1]
 72%|███████▏  | 216/300 [02:55<01:02,  1.34it/s, failures=0, objective=1]
 72%|███████▏  | 216/300 [02:55<01:02,  1.34it/s, failures=0, objective=1]
 72%|███████▏  | 217/300 [02:56<01:01,  1.36it/s, failures=0, objective=1]
 72%|███████▏  | 217/300 [02:56<01:01,  1.36it/s, failures=0, objective=1]
 73%|███████▎  | 218/300 [02:56<01:01,  1.34it/s, failures=0, objective=1]
 73%|███████▎  | 218/300 [02:56<01:01,  1.34it/s, failures=0, objective=1]
 73%|███████▎  | 219/300 [02:57<00:59,  1.36it/s, failures=0, objective=1]
 73%|███████▎  | 219/300 [02:57<00:59,  1.36it/s, failures=0, objective=1]
 73%|███████▎  | 220/300 [02:58<00:58,  1.37it/s, failures=0, objective=1]
 73%|███████▎  | 220/300 [02:58<00:58,  1.37it/s, failures=0, objective=1]
 74%|███████▎  | 221/300 [02:58<00:57,  1.38it/s, failures=0, objective=1]
 74%|███████▎  | 221/300 [02:58<00:57,  1.38it/s, failures=0, objective=1]
 74%|███████▍  | 222/300 [02:59<00:57,  1.35it/s, failures=0, objective=1]
 74%|███████▍  | 222/300 [02:59<00:57,  1.35it/s, failures=0, objective=1]
 74%|███████▍  | 223/300 [03:00<00:56,  1.36it/s, failures=0, objective=1]
 74%|███████▍  | 223/300 [03:00<00:56,  1.36it/s, failures=0, objective=1]
 75%|███████▍  | 224/300 [03:01<00:55,  1.37it/s, failures=0, objective=1]
 75%|███████▍  | 224/300 [03:01<00:55,  1.37it/s, failures=0, objective=1]
 75%|███████▌  | 225/300 [03:01<00:54,  1.38it/s, failures=0, objective=1]
 75%|███████▌  | 225/300 [03:01<00:54,  1.38it/s, failures=0, objective=1]
 75%|███████▌  | 226/300 [03:02<00:54,  1.35it/s, failures=0, objective=1]
 75%|███████▌  | 226/300 [03:02<00:54,  1.35it/s, failures=0, objective=1]
 76%|███████▌  | 227/300 [03:03<00:53,  1.37it/s, failures=0, objective=1]
 76%|███████▌  | 227/300 [03:03<00:53,  1.37it/s, failures=0, objective=1]
 76%|███████▌  | 228/300 [03:04<00:52,  1.38it/s, failures=0, objective=1]
 76%|███████▌  | 228/300 [03:04<00:52,  1.38it/s, failures=0, objective=1]
 76%|███████▋  | 229/300 [03:04<00:52,  1.36it/s, failures=0, objective=1]
 76%|███████▋  | 229/300 [03:04<00:52,  1.36it/s, failures=0, objective=1]
 77%|███████▋  | 230/300 [03:05<00:51,  1.37it/s, failures=0, objective=1]
 77%|███████▋  | 230/300 [03:05<00:51,  1.37it/s, failures=0, objective=1]
 77%|███████▋  | 231/300 [03:06<00:50,  1.38it/s, failures=0, objective=1]
 77%|███████▋  | 231/300 [03:06<00:50,  1.38it/s, failures=0, objective=1]
 77%|███████▋  | 232/300 [03:06<00:49,  1.38it/s, failures=0, objective=1]
 77%|███████▋  | 232/300 [03:06<00:49,  1.38it/s, failures=0, objective=1]
 78%|███████▊  | 233/300 [03:07<00:49,  1.36it/s, failures=0, objective=1]
 78%|███████▊  | 233/300 [03:07<00:49,  1.36it/s, failures=0, objective=1]
 78%|███████▊  | 234/300 [03:08<00:48,  1.37it/s, failures=0, objective=1]
 78%|███████▊  | 234/300 [03:08<00:48,  1.37it/s, failures=0, objective=1]
 78%|███████▊  | 235/300 [03:09<00:47,  1.38it/s, failures=0, objective=1]
 78%|███████▊  | 235/300 [03:09<00:47,  1.38it/s, failures=0, objective=1]
 79%|███████▊  | 236/300 [03:09<00:46,  1.38it/s, failures=0, objective=1]
 79%|███████▊  | 236/300 [03:09<00:46,  1.38it/s, failures=0, objective=1]
 79%|███████▉  | 237/300 [03:10<00:46,  1.35it/s, failures=0, objective=1]
 79%|███████▉  | 237/300 [03:10<00:46,  1.35it/s, failures=0, objective=1]
 79%|███████▉  | 238/300 [03:11<00:45,  1.37it/s, failures=0, objective=1]
 79%|███████▉  | 238/300 [03:11<00:45,  1.37it/s, failures=0, objective=1]
 80%|███████▉  | 239/300 [03:12<00:44,  1.38it/s, failures=0, objective=1]
 80%|███████▉  | 239/300 [03:12<00:44,  1.38it/s, failures=0, objective=1]
 80%|████████  | 240/300 [03:12<00:43,  1.38it/s, failures=0, objective=1]
 80%|████████  | 240/300 [03:12<00:43,  1.38it/s, failures=0, objective=1]
 80%|████████  | 241/300 [03:13<00:43,  1.36it/s, failures=0, objective=1]
 80%|████████  | 241/300 [03:13<00:43,  1.36it/s, failures=0, objective=1]
 81%|████████  | 242/300 [03:14<00:42,  1.37it/s, failures=0, objective=1]
 81%|████████  | 242/300 [03:14<00:42,  1.37it/s, failures=0, objective=1]
 81%|████████  | 243/300 [03:14<00:41,  1.38it/s, failures=0, objective=1]
 81%|████████  | 243/300 [03:14<00:41,  1.38it/s, failures=0, objective=1]
 81%|████████▏ | 244/300 [03:15<00:40,  1.38it/s, failures=0, objective=1]
 81%|████████▏ | 244/300 [03:15<00:40,  1.38it/s, failures=0, objective=1]
 82%|████████▏ | 245/300 [03:16<00:40,  1.35it/s, failures=0, objective=1]
 82%|████████▏ | 245/300 [03:16<00:40,  1.35it/s, failures=0, objective=1]
 82%|████████▏ | 246/300 [03:17<00:39,  1.37it/s, failures=0, objective=1]
 82%|████████▏ | 246/300 [03:17<00:39,  1.37it/s, failures=0, objective=1]
 82%|████████▏ | 247/300 [03:17<00:38,  1.38it/s, failures=0, objective=1]
 82%|████████▏ | 247/300 [03:17<00:38,  1.38it/s, failures=0, objective=1]
 83%|████████▎ | 248/300 [03:18<00:37,  1.39it/s, failures=0, objective=1]
 83%|████████▎ | 248/300 [03:18<00:37,  1.39it/s, failures=0, objective=1]
 83%|████████▎ | 249/300 [03:19<00:37,  1.36it/s, failures=0, objective=1]
 83%|████████▎ | 249/300 [03:19<00:37,  1.36it/s, failures=0, objective=1]
 83%|████████▎ | 250/300 [03:20<00:36,  1.37it/s, failures=0, objective=1]
 83%|████████▎ | 250/300 [03:20<00:36,  1.37it/s, failures=0, objective=1]
 84%|████████▎ | 251/300 [03:20<00:35,  1.38it/s, failures=0, objective=1]
 84%|████████▎ | 251/300 [03:20<00:35,  1.38it/s, failures=0, objective=1]
 84%|████████▍ | 252/300 [03:21<00:35,  1.36it/s, failures=0, objective=1]
 84%|████████▍ | 252/300 [03:21<00:35,  1.36it/s, failures=0, objective=1]
 84%|████████▍ | 253/300 [03:22<00:34,  1.37it/s, failures=0, objective=1]
 84%|████████▍ | 253/300 [03:22<00:34,  1.37it/s, failures=0, objective=1]
 85%|████████▍ | 254/300 [03:22<00:33,  1.38it/s, failures=0, objective=1]
 85%|████████▍ | 254/300 [03:22<00:33,  1.38it/s, failures=0, objective=1]
 85%|████████▌ | 255/300 [03:23<00:32,  1.39it/s, failures=0, objective=1]
 85%|████████▌ | 255/300 [03:23<00:32,  1.39it/s, failures=0, objective=1]
 85%|████████▌ | 256/300 [03:24<00:32,  1.36it/s, failures=0, objective=1]
 85%|████████▌ | 256/300 [03:24<00:32,  1.36it/s, failures=0, objective=1]
 86%|████████▌ | 257/300 [03:25<00:31,  1.37it/s, failures=0, objective=1]
 86%|████████▌ | 257/300 [03:25<00:31,  1.37it/s, failures=0, objective=1]
 86%|████████▌ | 258/300 [03:25<00:30,  1.38it/s, failures=0, objective=1]
 86%|████████▌ | 258/300 [03:25<00:30,  1.38it/s, failures=0, objective=1]
 86%|████████▋ | 259/300 [03:26<00:29,  1.38it/s, failures=0, objective=1]
 86%|████████▋ | 259/300 [03:26<00:29,  1.38it/s, failures=0, objective=1]
 87%|████████▋ | 260/300 [03:27<00:29,  1.36it/s, failures=0, objective=1]
 87%|████████▋ | 260/300 [03:27<00:29,  1.36it/s, failures=0, objective=1]
 87%|████████▋ | 261/300 [03:28<00:28,  1.37it/s, failures=0, objective=1]
 87%|████████▋ | 261/300 [03:28<00:28,  1.37it/s, failures=0, objective=1]
 87%|████████▋ | 262/300 [03:28<00:27,  1.38it/s, failures=0, objective=1]
 87%|████████▋ | 262/300 [03:28<00:27,  1.38it/s, failures=0, objective=1]
 88%|████████▊ | 263/300 [03:29<00:26,  1.39it/s, failures=0, objective=1]
 88%|████████▊ | 263/300 [03:29<00:26,  1.39it/s, failures=0, objective=1]
 88%|████████▊ | 264/300 [03:30<00:26,  1.36it/s, failures=0, objective=1]
 88%|████████▊ | 264/300 [03:30<00:26,  1.36it/s, failures=0, objective=1]
 88%|████████▊ | 265/300 [03:31<00:25,  1.37it/s, failures=0, objective=1]
 88%|████████▊ | 265/300 [03:31<00:25,  1.37it/s, failures=0, objective=1]
 89%|████████▊ | 266/300 [03:31<00:24,  1.38it/s, failures=0, objective=1]
 89%|████████▊ | 266/300 [03:31<00:24,  1.38it/s, failures=0, objective=1]
 89%|████████▉ | 267/300 [03:32<00:23,  1.38it/s, failures=0, objective=1]
 89%|████████▉ | 267/300 [03:32<00:23,  1.38it/s, failures=0, objective=1]
 89%|████████▉ | 268/300 [03:33<00:23,  1.36it/s, failures=0, objective=1]
 89%|████████▉ | 268/300 [03:33<00:23,  1.36it/s, failures=0, objective=1]
 90%|████████▉ | 269/300 [03:33<00:22,  1.36it/s, failures=0, objective=1]
 90%|████████▉ | 269/300 [03:33<00:22,  1.36it/s, failures=0, objective=1]
 90%|█████████ | 270/300 [03:34<00:22,  1.36it/s, failures=0, objective=1]
 90%|█████████ | 270/300 [03:34<00:22,  1.36it/s, failures=0, objective=1]
 90%|█████████ | 271/300 [03:35<00:21,  1.37it/s, failures=0, objective=1]
 90%|█████████ | 271/300 [03:35<00:21,  1.37it/s, failures=0, objective=1]
 91%|█████████ | 272/300 [03:36<00:20,  1.34it/s, failures=0, objective=1]
 91%|█████████ | 272/300 [03:36<00:20,  1.34it/s, failures=0, objective=1]
 91%|█████████ | 273/300 [03:36<00:19,  1.36it/s, failures=0, objective=1]
 91%|█████████ | 273/300 [03:36<00:19,  1.36it/s, failures=0, objective=1]
 91%|█████████▏| 274/300 [03:37<00:18,  1.37it/s, failures=0, objective=1]
 91%|█████████▏| 274/300 [03:37<00:18,  1.37it/s, failures=0, objective=1]
 92%|█████████▏| 275/300 [03:38<00:17,  1.39it/s, failures=0, objective=1]
 92%|█████████▏| 275/300 [03:38<00:17,  1.39it/s, failures=0, objective=1]
 92%|█████████▏| 276/300 [03:39<00:17,  1.36it/s, failures=0, objective=1]
 92%|█████████▏| 276/300 [03:39<00:17,  1.36it/s, failures=0, objective=1]
 92%|█████████▏| 277/300 [03:39<00:16,  1.37it/s, failures=0, objective=1]
 92%|█████████▏| 277/300 [03:39<00:16,  1.37it/s, failures=0, objective=1]
 93%|█████████▎| 278/300 [03:40<00:15,  1.38it/s, failures=0, objective=1]
 93%|█████████▎| 278/300 [03:40<00:15,  1.38it/s, failures=0, objective=1]
 93%|█████████▎| 279/300 [03:41<00:15,  1.35it/s, failures=0, objective=1]
 93%|█████████▎| 279/300 [03:41<00:15,  1.35it/s, failures=0, objective=1]
 93%|█████████▎| 280/300 [03:42<00:14,  1.36it/s, failures=0, objective=1]
 93%|█████████▎| 280/300 [03:42<00:14,  1.36it/s, failures=0, objective=1]
 94%|█████████▎| 281/300 [03:42<00:13,  1.37it/s, failures=0, objective=1]
 94%|█████████▎| 281/300 [03:42<00:13,  1.37it/s, failures=0, objective=1]
 94%|█████████▍| 282/300 [03:43<00:13,  1.38it/s, failures=0, objective=1]
 94%|█████████▍| 282/300 [03:43<00:13,  1.38it/s, failures=0, objective=1]
 94%|█████████▍| 283/300 [03:44<00:12,  1.36it/s, failures=0, objective=1]
 94%|█████████▍| 283/300 [03:44<00:12,  1.36it/s, failures=0, objective=1]
 95%|█████████▍| 284/300 [03:44<00:11,  1.37it/s, failures=0, objective=1]
 95%|█████████▍| 284/300 [03:44<00:11,  1.37it/s, failures=0, objective=1]
 95%|█████████▌| 285/300 [03:45<00:10,  1.37it/s, failures=0, objective=1]
 95%|█████████▌| 285/300 [03:45<00:10,  1.37it/s, failures=0, objective=1]
 95%|█████████▌| 286/300 [03:46<00:10,  1.39it/s, failures=0, objective=1]
 95%|█████████▌| 286/300 [03:46<00:10,  1.39it/s, failures=0, objective=1]
 96%|█████████▌| 287/300 [03:47<00:09,  1.36it/s, failures=0, objective=1]
 96%|█████████▌| 287/300 [03:47<00:09,  1.36it/s, failures=0, objective=1]
 96%|█████████▌| 288/300 [03:47<00:08,  1.37it/s, failures=0, objective=1]
 96%|█████████▌| 288/300 [03:47<00:08,  1.37it/s, failures=0, objective=1]
 96%|█████████▋| 289/300 [03:48<00:07,  1.38it/s, failures=0, objective=1]
 96%|█████████▋| 289/300 [03:48<00:07,  1.38it/s, failures=0, objective=1]
 97%|█████████▋| 290/300 [03:49<00:07,  1.36it/s, failures=0, objective=1]
 97%|█████████▋| 290/300 [03:49<00:07,  1.36it/s, failures=0, objective=1]
 97%|█████████▋| 291/300 [03:50<00:06,  1.37it/s, failures=0, objective=1]
 97%|█████████▋| 291/300 [03:50<00:06,  1.37it/s, failures=0, objective=1]
 97%|█████████▋| 292/300 [03:50<00:05,  1.38it/s, failures=0, objective=1]
 97%|█████████▋| 292/300 [03:50<00:05,  1.38it/s, failures=0, objective=1]
 98%|█████████▊| 293/300 [03:51<00:05,  1.39it/s, failures=0, objective=1]
 98%|█████████▊| 293/300 [03:51<00:05,  1.39it/s, failures=0, objective=1]
 98%|█████████▊| 294/300 [03:52<00:04,  1.35it/s, failures=0, objective=1]
 98%|█████████▊| 294/300 [03:52<00:04,  1.35it/s, failures=0, objective=1]
 98%|█████████▊| 295/300 [03:52<00:03,  1.37it/s, failures=0, objective=1]
 98%|█████████▊| 295/300 [03:52<00:03,  1.37it/s, failures=0, objective=1]
 99%|█████████▊| 296/300 [03:53<00:02,  1.38it/s, failures=0, objective=1]
 99%|█████████▊| 296/300 [03:53<00:02,  1.38it/s, failures=0, objective=1]
 99%|█████████▉| 297/300 [03:54<00:02,  1.39it/s, failures=0, objective=1]
 99%|█████████▉| 297/300 [03:54<00:02,  1.39it/s, failures=0, objective=1]
 99%|█████████▉| 298/300 [03:55<00:01,  1.36it/s, failures=0, objective=1]
 99%|█████████▉| 298/300 [03:55<00:01,  1.36it/s, failures=0, objective=1]
100%|█████████▉| 299/300 [03:55<00:00,  1.38it/s, failures=0, objective=1]
100%|█████████▉| 299/300 [03:55<00:00,  1.38it/s, failures=0, objective=1]
100%|██████████| 300/300 [03:56<00:00,  1.39it/s, failures=0, objective=1]
100%|██████████| 300/300 [03:56<00:00,  1.39it/s, failures=0, objective=1]
100%|██████████| 300/300 [03:56<00:00,  1.27it/s, failures=0, objective=1]
Tuning selection model...
Tuned model parameters: {'bootstrap': True, 'max_depth': None, 'min_samples_leaf': 1, 'min_samples_split': 16, 'n_estimators': 100, 'splitter': 'random'}
Tuned model scores: {'r2': 0.5758640399372144, 'r2_upper_bound': 0.5824243393398307, 'y_std_corr': {'statistic': 0.5507077171258088, 'pvalue': 3.4360452071374125e-25}}

Finally, let us visualize the results. The search(...) returns a DataFrame also saved locally under results.csv (in case of crash we don’t want to lose the possibly expensive evaluations already performed).

The DataFrame contains the usual columns:

the optimized hyperparameters: such as $x$ with name p:x.
the objective maximised which directly match the results of the $f$ function in our example.
the job_id of each evaluated function (increased incrementally following the order of created evaluations).
the time of creation/collection of each task timestamp_submit and timestamp_gather respectively (in secondes, since the creation of the Evaluator).

In addition, it now also contains the new columns: 1. the estimated solution parameter sol.p:x. 2. the estimated solution objective sol.objective. 3. the estimated solution objective aleatoric uncertainty sol.objective_std_al. 4. the estimated solution objective epistemic uncertainty sol.objective_std_ep.

results

	p:x	objective	job_id	job_status	m:timestamp_submit	m:timestamp_gather	sol.p:x	sol.objective	sol.objective_std_al	sol.objective_std_ep
0	-5.897543	1.0	0	DONE	0.008010	0.008471	3.787579	1.0000	0.000000	0.000000
1	1.173746	1.0	1	DONE	0.601424	0.601710	4.478960	1.0000	0.000000	0.000000
2	-8.123233	0.0	2	DONE	1.170570	1.170761	-3.484530	0.6400	0.382971	0.289367
3	-5.765876	0.0	3	DONE	1.917559	1.917737	4.345440	0.4575	0.425000	0.259940
4	-0.429996	1.0	4	DONE	2.610857	2.611031	-8.594116	0.6120	0.438178	0.213204
...	...	...	...	...	...	...	...	...	...	...
295	1.884849	1.0	295	DONE	233.669751	233.669944	0.867834	1.0000	0.000000	0.000000
296	-5.175182	1.0	296	DONE	234.377051	234.377236	0.194228	1.0000	0.000000	0.000000
297	3.121228	1.0	297	DONE	235.143700	235.143892	-0.569485	1.0000	0.000000	0.000000
298	-8.950916	0.0	298	DONE	235.844466	235.844659	1.215950	1.0000	0.000000	0.000000
299	4.389502	0.0	299	DONE	236.546905	236.547096	0.284153	1.0000	0.000000	0.000000

300 rows × 10 columns

To get the parameters at the observed maximum value we can use the deephyper.analysis.hpo.parameters_at_max(): We make sure to select the right column and prefix for parameters. Also, we prefer to select the solution amoung the n_last=20 rows to avoid selecting noisy observations at the beginning.

from deephyper.analysis.hpo import parameters_at_max


parameters, objective = parameters_at_max(results, column="sol.objective", prefix="sol.p:", n_last=20)
print("\nEstimated Optimum values")
print("x:", parameters["x"])
print("objective:", objective)

Estimated Optimum values
x: 0.28415316302867666
objective: 1.0

We can also plot the evolution of the estimated solution value of $x$ to verify that we converge correctly toward $x=0$.

import matplotlib.pyplot as plt
from deephyper.analysis.hpo import plot_search_trajectory_single_objective_hpo


WIDTH_PLOTS = 8
HEIGHT_PLOTS = WIDTH_PLOTS / 1.618

fig, ax = plt.subplots(figsize=(WIDTH_PLOTS, HEIGHT_PLOTS))
plot_search_trajectory_single_objective_hpo(results, column="sol.p:x", mode="max", ax=ax)
_ = ax.set_ylabel(r"Estimated solution $x$")
_ = ax.set_ylim(-10, 10)
_ = plt.title("Search Trajectory")

Total running time of the script: (4 minutes 13.660 seconds)

Gallery generated by Sphinx-Gallery