Constrained Black-Box Optimization with Custom Sampler

Note

Go to the end to download the full example code.

Constrained Black-Box Optimization with Custom Sampler#

Author(s): Romain Egele.

This tutorial demonstrates how to solve constrained black-box optimization using DeepHyper, focusing on how to encode structural constraints directly in the sampling strategy of the search algorithm.

Black-box optimization aims to optimize an unknown function \(f(x) = y \in \mathbb{R}\) using only input–output evaluations \(\{(x_1, y_1), \ldots, (x_n, y_n)\}\). No analytical gradients or structural properties of \(f\) are required.

In constrained settings, the search is further restricted to parameters that satisfy one or more feasibility rules. Constraints can significantly reshape the search space and modify the behavior of the optimizer.

Problem Setting#

In this example, we consider a discrete, ordered search space of dimension \(N\). Each variable \(x_i\) must satisfy the monotonicity constraint

\[x_0 < x_1 < \cdots < x_{N-1}.\]

Each \(x_i\) is bounded between \(i\) and \(m - N + i\). This could tipically represent the layer indexes to drop in Depth pruning of Large language models. The objective is to maximize the sum

\[f(x) = \sum_{i=0}^{N-1} x_i.\]

Since the optimal strategy is to push every variable as high as possible while respecting monotonicity, the theoretical optimum is:

\[\sum_{i=0}^{N-1} (m - i).\]

DeepHyper offers several ways to incorporate constraints:

  1. Custom sampler (this tutorial): constraints are enforced directly when generating new candidate points.

  2. Rejection sampling: see Constrained Black-Box Optimization with Rejection Sampling.

  3. Learning to avoid failures (CBO auto-handles failed evaluations): see this tutorial and also Learn to Avoid Failures with Bayesian Optimization.

  4. Multi-objective approach where the constraint becomes an additional objective (tutorial forthcoming).

import matplotlib.pyplot as plt
import matplotlib.cm as cm
import matplotlib.colors as colors
import numpy as np
import pandas as pd
from numpy.random import Generator
from deephyper.analysis.hpo import (
    plot_search_trajectory_single_objective_hpo,
    parameters_at_max,
    filter_failed_objectives,
)
from deephyper.hpo import HpProblem, CBO

Custom Sampler#

Because every \(x_i\) must be strictly larger than \(x_{i-1}\), the usual independent sampling over each variable would frequently violate the constraint.

Instead, we implement a custom sampler to ensure that:

  • all generated samples satisfy \(x_i < x_{i+1}\) by construction;

  • the sampler focuses on the feasible region, avoiding wasted evaluations.

n = 10
m = 32

print("optimum:", sum([m - i - 1 for i in range(n)]))

pb = HpProblem()
for i in range(n):
    pb.add((i, m - n + i), f"x{i}")
print(pb)


def sampling_fn(size: int) -> list[dict]:
    if n > m:
        raise ValueError(f"Cannot sample {n} items from {m} elements.")

    rng = np.random.default_rng(None)
    indexes = np.arange(m)

    def sample_one():
        vals = np.sort(rng.choice(indexes, size=n, replace=False)).tolist()
        return {k: v for k, v in zip(pb.hyperparameter_names, vals)}

    return [sample_one() for _ in range(size)]


pb.set_sampling_fn(sampling_fn)

optimum: 265
Configuration space object:
  Hyperparameters:
    x0, Type: UniformInteger, Range: [0, 22], Default: 11
    x1, Type: UniformInteger, Range: [1, 23], Default: 12
    x2, Type: UniformInteger, Range: [2, 24], Default: 13
    x3, Type: UniformInteger, Range: [3, 25], Default: 14
    x4, Type: UniformInteger, Range: [4, 26], Default: 15
    x5, Type: UniformInteger, Range: [5, 27], Default: 16
    x6, Type: UniformInteger, Range: [6, 28], Default: 17
    x7, Type: UniformInteger, Range: [7, 29], Default: 18
    x8, Type: UniformInteger, Range: [8, 30], Default: 19
    x9, Type: UniformInteger, Range: [9, 31], Default: 20

Constraint Function#

Although the sampler already generates feasible points, we explicitly define a constraint_fn. This allows DeepHyper to properly handle failed trials (e.g., from manually constructed parameter sets or mutation-based acquisition optimizers). Not only that, this will help report non-feasible points using "F_constraint" in the objective function so that CBO learns to avoid them.

def constraint_fn(df: pd.DataFrame) -> pd.Series:
    accept = pd.Series(np.ones((len(df)), dtype=bool))
    for i in range(n - 1):
        accept = accept & (df[f"x{i}"] < df[f"x{i + 1}"])
    return accept


pb.set_constraint_fn(constraint_fn)


def f(job):
    """Objective function: maximize sum(x_i)."""
    df = pd.DataFrame([job.parameters])
    accept = constraint_fn(df)
    if all(accept):
        return sum(job.parameters.values())
    else:
        return "F_constraint"

Bayesian Optimization with Mixed-GA Acquisition Optimization#

We run a Centralized Bayesian Optimization (CBO) search using:

  • Ensemble of Trees surrogate model ("ET").

  • A mixed genetic algorithm ("mixedga") to optimize the acquisition function.

  • A periodically decaying scheduler on the exploration parameter kappa.

This setup is well suited for discrete, irregularly constrained spaces.

The search runs for max_evals=300 iterations.

search = CBO(
    pb,
    surrogate_model="ET",
    surrogate_model_kwargs={"max_features": "sqrt"},
    acq_optimizer="mixedga",
    acq_optimizer_kwargs={
        "n_points": 1_000,
        "acq_optimizer_freq": 2,
        "filter_failures": "mean",
    },
    acq_func_kwargs={
        # Exploration/Exploitation mechanism
        "kappa": 10.0,
        "scheduler": {
            "type": "periodic-exp-decay",
            "period": 20,
            "kappa_final": 0.1,
        },
    },
    verbose=1,
)
results = search.search(f, max_evals=300)

Results file already exists, it will be renamed to /Users/rp5/Documents/DeepHyper/deephyper/examples/examples_bbo/results_20260112-103739.csv

  0%|          | 0/300 [00:00<?, ?it/s]
  0%|          | 1/300 [00:00<00:00, 8208.03it/s, failures=0, objective=177]
  1%|          | 2/300 [00:00<00:01, 170.85it/s, failures=0, objective=177]
  1%|          | 3/300 [00:00<00:02, 141.79it/s, failures=0, objective=177]
  1%|▏         | 4/300 [00:00<00:02, 131.18it/s, failures=0, objective=177]
  2%|▏         | 5/300 [00:00<00:02, 125.53it/s, failures=0, objective=177]
  2%|▏         | 6/300 [00:00<00:02, 122.13it/s, failures=0, objective=197]
  2%|▏         | 7/300 [00:00<00:02, 119.48it/s, failures=0, objective=215]
  3%|▎         | 8/300 [00:00<00:02, 117.92it/s, failures=0, objective=215]
  3%|▎         | 9/300 [00:00<00:02, 116.96it/s, failures=0, objective=215]
  3%|▎         | 10/300 [00:00<00:02, 116.30it/s, failures=0, objective=215]
  4%|▎         | 11/300 [00:00<00:02, 115.75it/s, failures=0, objective=215]
  4%|▍         | 12/300 [00:00<00:02, 115.30it/s, failures=0, objective=215]
  4%|▍         | 12/300 [00:00<00:02, 115.30it/s, failures=0, objective=215]
  4%|▍         | 13/300 [00:00<00:02, 115.30it/s, failures=0, objective=215]
  5%|▍         | 14/300 [00:00<00:02, 115.30it/s, failures=0, objective=215]
  5%|▌         | 15/300 [00:00<00:02, 115.30it/s, failures=0, objective=215]
  5%|▌         | 16/300 [00:00<00:02, 115.30it/s, failures=0, objective=215]
  6%|▌         | 17/300 [00:00<00:02, 115.30it/s, failures=0, objective=215]
  6%|▌         | 18/300 [00:00<00:02, 115.30it/s, failures=0, objective=215]
  6%|▋         | 19/300 [00:00<00:02, 115.30it/s, failures=0, objective=215]
  7%|▋         | 20/300 [00:00<00:02, 115.30it/s, failures=0, objective=215]
  7%|▋         | 21/300 [00:00<00:02, 115.30it/s, failures=0, objective=215]
  7%|▋         | 22/300 [00:00<00:02, 115.30it/s, failures=0, objective=228]
  8%|▊         | 23/300 [00:00<00:02, 115.30it/s, failures=0, objective=228]
  8%|▊         | 24/300 [00:00<00:10, 25.71it/s, failures=0, objective=228]
  8%|▊         | 24/300 [00:00<00:10, 25.71it/s, failures=0, objective=228]
  8%|▊         | 25/300 [00:00<00:10, 25.71it/s, failures=0, objective=228]
  9%|▊         | 26/300 [00:01<00:10, 25.71it/s, failures=0, objective=228]
  9%|▉         | 27/300 [00:01<00:10, 25.71it/s, failures=0, objective=228]
  9%|▉         | 28/300 [00:01<00:10, 25.71it/s, failures=0, objective=228]
 10%|▉         | 29/300 [00:01<00:10, 25.71it/s, failures=0, objective=228]
 10%|█         | 30/300 [00:02<00:25, 10.47it/s, failures=0, objective=228]
 10%|█         | 30/300 [00:02<00:25, 10.47it/s, failures=0, objective=228]
 10%|█         | 31/300 [00:02<00:25, 10.47it/s, failures=0, objective=228]
 11%|█         | 32/300 [00:02<00:25, 10.47it/s, failures=0, objective=228]
 11%|█         | 33/300 [00:02<00:25, 10.47it/s, failures=0, objective=228]
 11%|█▏        | 34/300 [00:02<00:30,  8.67it/s, failures=0, objective=228]
 11%|█▏        | 34/300 [00:02<00:30,  8.67it/s, failures=0, objective=228]
 12%|█▏        | 35/300 [00:02<00:30,  8.67it/s, failures=0, objective=228]
 12%|█▏        | 36/300 [00:03<00:30,  8.67it/s, failures=0, objective=228]
 12%|█▏        | 37/300 [00:03<00:29,  8.89it/s, failures=0, objective=228]
 12%|█▏        | 37/300 [00:03<00:29,  8.89it/s, failures=0, objective=228]
 13%|█▎        | 38/300 [00:03<00:29,  8.89it/s, failures=0, objective=238]
 13%|█▎        | 39/300 [00:03<00:30,  8.58it/s, failures=0, objective=238]
 13%|█▎        | 39/300 [00:03<00:30,  8.58it/s, failures=0, objective=238]
 13%|█▎        | 40/300 [00:03<00:30,  8.58it/s, failures=0, objective=238]
 14%|█▎        | 41/300 [00:03<00:31,  8.18it/s, failures=0, objective=238]
 14%|█▎        | 41/300 [00:03<00:31,  8.18it/s, failures=0, objective=238]
 14%|█▍        | 42/300 [00:04<00:31,  8.18it/s, failures=0, objective=238]
 14%|█▍        | 43/300 [00:04<00:32,  7.82it/s, failures=0, objective=238]
 14%|█▍        | 43/300 [00:04<00:32,  7.82it/s, failures=0, objective=238]
 15%|█▍        | 44/300 [00:04<00:32,  7.82it/s, failures=0, objective=238]
 15%|█▌        | 45/300 [00:04<00:37,  6.85it/s, failures=0, objective=238]
 15%|█▌        | 45/300 [00:04<00:37,  6.85it/s, failures=0, objective=238]
 15%|█▌        | 46/300 [00:04<00:44,  5.65it/s, failures=0, objective=238]
 15%|█▌        | 46/300 [00:04<00:44,  5.65it/s, failures=0, objective=238]
 16%|█▌        | 47/300 [00:04<00:44,  5.65it/s, failures=0, objective=238]
 16%|█▌        | 48/300 [00:05<00:49,  5.07it/s, failures=0, objective=238]
 16%|█▌        | 48/300 [00:05<00:49,  5.07it/s, failures=0, objective=238]
 16%|█▋        | 49/300 [00:05<00:49,  5.07it/s, failures=0, objective=238]
 17%|█▋        | 50/300 [00:06<01:02,  3.97it/s, failures=0, objective=238]
 17%|█▋        | 50/300 [00:06<01:02,  3.97it/s, failures=0, objective=238]
 17%|█▋        | 51/300 [00:06<01:02,  3.97it/s, failures=0, objective=238]
 17%|█▋        | 52/300 [00:06<01:02,  4.00it/s, failures=0, objective=238]
 17%|█▋        | 52/300 [00:06<01:02,  4.00it/s, failures=0, objective=238]
 18%|█▊        | 53/300 [00:06<01:01,  4.00it/s, failures=0, objective=238]
 18%|█▊        | 54/300 [00:07<01:01,  4.03it/s, failures=0, objective=238]
 18%|█▊        | 54/300 [00:07<01:01,  4.03it/s, failures=0, objective=238]
 18%|█▊        | 55/300 [00:07<01:00,  4.03it/s, failures=0, objective=238]
 19%|█▊        | 56/300 [00:07<00:53,  4.53it/s, failures=0, objective=238]
 19%|█▊        | 56/300 [00:07<00:53,  4.53it/s, failures=0, objective=243]
 19%|█▉        | 57/300 [00:07<00:53,  4.53it/s, failures=0, objective=243]
 19%|█▉        | 58/300 [00:07<00:50,  4.75it/s, failures=0, objective=243]
 19%|█▉        | 58/300 [00:07<00:50,  4.75it/s, failures=0, objective=243]
 20%|█▉        | 59/300 [00:07<00:50,  4.75it/s, failures=0, objective=243]
 20%|██        | 60/300 [00:08<00:44,  5.37it/s, failures=0, objective=243]
 20%|██        | 60/300 [00:08<00:44,  5.37it/s, failures=0, objective=243]
 20%|██        | 61/300 [00:08<00:44,  5.37it/s, failures=0, objective=243]
 21%|██        | 62/300 [00:08<00:44,  5.39it/s, failures=0, objective=243]
 21%|██        | 62/300 [00:08<00:44,  5.39it/s, failures=0, objective=243]
 21%|██        | 63/300 [00:08<00:43,  5.39it/s, failures=0, objective=243]
 21%|██▏       | 64/300 [00:09<00:55,  4.22it/s, failures=0, objective=243]
 21%|██▏       | 64/300 [00:09<00:55,  4.22it/s, failures=0, objective=243]
 22%|██▏       | 65/300 [00:09<00:55,  4.22it/s, failures=0, objective=243]
 22%|██▏       | 66/300 [00:09<01:02,  3.74it/s, failures=0, objective=243]
 22%|██▏       | 66/300 [00:09<01:02,  3.74it/s, failures=0, objective=243]
 22%|██▏       | 67/300 [00:09<01:02,  3.74it/s, failures=0, objective=243]
 23%|██▎       | 68/300 [00:10<01:01,  3.79it/s, failures=0, objective=243]
 23%|██▎       | 68/300 [00:10<01:01,  3.79it/s, failures=0, objective=243]
 23%|██▎       | 69/300 [00:10<01:00,  3.79it/s, failures=0, objective=243]
 23%|██▎       | 70/300 [00:11<01:20,  2.85it/s, failures=0, objective=243]
 23%|██▎       | 70/300 [00:11<01:20,  2.85it/s, failures=0, objective=243]
 24%|██▎       | 71/300 [00:11<01:20,  2.85it/s, failures=0, objective=243]
 24%|██▍       | 72/300 [00:12<01:37,  2.33it/s, failures=0, objective=243]
 24%|██▍       | 72/300 [00:12<01:37,  2.33it/s, failures=0, objective=243]
 24%|██▍       | 73/300 [00:12<01:37,  2.33it/s, failures=0, objective=243]
 25%|██▍       | 74/300 [00:13<01:26,  2.62it/s, failures=0, objective=243]
 25%|██▍       | 74/300 [00:13<01:26,  2.62it/s, failures=0, objective=243]
 25%|██▌       | 75/300 [00:13<01:26,  2.62it/s, failures=0, objective=243]
 25%|██▌       | 76/300 [00:14<01:34,  2.38it/s, failures=0, objective=243]
 25%|██▌       | 76/300 [00:14<01:34,  2.38it/s, failures=0, objective=243]
 26%|██▌       | 77/300 [00:14<01:33,  2.38it/s, failures=0, objective=243]
 26%|██▌       | 78/300 [00:15<01:34,  2.34it/s, failures=0, objective=243]
 26%|██▌       | 78/300 [00:15<01:34,  2.34it/s, failures=0, objective=243]
 26%|██▋       | 79/300 [00:15<01:34,  2.34it/s, failures=0, objective=243]
 27%|██▋       | 80/300 [00:15<01:26,  2.54it/s, failures=0, objective=243]
 27%|██▋       | 80/300 [00:15<01:26,  2.54it/s, failures=0, objective=243]
 27%|██▋       | 81/300 [00:15<01:26,  2.54it/s, failures=0, objective=243]
 27%|██▋       | 82/300 [00:16<01:31,  2.37it/s, failures=0, objective=243]
 27%|██▋       | 82/300 [00:16<01:31,  2.37it/s, failures=0, objective=243]
 28%|██▊       | 83/300 [00:16<01:31,  2.37it/s, failures=0, objective=243]
 28%|██▊       | 84/300 [00:17<01:20,  2.68it/s, failures=0, objective=243]
 28%|██▊       | 84/300 [00:17<01:20,  2.68it/s, failures=0, objective=243]
 28%|██▊       | 85/300 [00:17<01:20,  2.68it/s, failures=0, objective=243]
 29%|██▊       | 86/300 [00:17<01:16,  2.81it/s, failures=0, objective=243]
 29%|██▊       | 86/300 [00:17<01:16,  2.81it/s, failures=0, objective=243]
 29%|██▉       | 87/300 [00:17<01:15,  2.81it/s, failures=0, objective=243]
 29%|██▉       | 88/300 [00:18<01:23,  2.54it/s, failures=0, objective=243]
 29%|██▉       | 88/300 [00:18<01:23,  2.54it/s, failures=0, objective=243]
 30%|██▉       | 89/300 [00:18<01:22,  2.54it/s, failures=0, objective=243]
 30%|███       | 90/300 [00:19<01:28,  2.37it/s, failures=0, objective=243]
 30%|███       | 90/300 [00:19<01:28,  2.37it/s, failures=0, objective=243]
 30%|███       | 91/300 [00:19<01:28,  2.37it/s, failures=0, objective=243]
 31%|███       | 92/300 [00:20<01:24,  2.46it/s, failures=0, objective=243]
 31%|███       | 92/300 [00:20<01:24,  2.46it/s, failures=0, objective=243]
 31%|███       | 93/300 [00:20<01:24,  2.46it/s, failures=0, objective=243]
 31%|███▏      | 94/300 [00:21<01:27,  2.36it/s, failures=0, objective=243]
 31%|███▏      | 94/300 [00:21<01:27,  2.36it/s, failures=0, objective=243]
 32%|███▏      | 95/300 [00:21<01:26,  2.36it/s, failures=0, objective=244]
 32%|███▏      | 96/300 [00:22<01:23,  2.45it/s, failures=0, objective=244]
 32%|███▏      | 96/300 [00:22<01:23,  2.45it/s, failures=0, objective=244]
 32%|███▏      | 97/300 [00:22<01:22,  2.45it/s, failures=0, objective=244]/Users/rp5/Documents/DeepHyper/deephyper/src/deephyper/skopt/optimizer/optimizer.py:820: UserWarning: The objective has been evaluated at this point before.
  warnings.warn("The objective has been evaluated at this point before.")

 33%|███▎      | 98/300 [00:23<01:23,  2.42it/s, failures=0, objective=244]
 33%|███▎      | 98/300 [00:23<01:23,  2.42it/s, failures=0, objective=244]
 33%|███▎      | 99/300 [00:23<01:23,  2.42it/s, failures=0, objective=244]/Users/rp5/Documents/DeepHyper/deephyper/src/deephyper/skopt/optimizer/optimizer.py:820: UserWarning: The objective has been evaluated at this point before.
  warnings.warn("The objective has been evaluated at this point before.")

 33%|███▎      | 100/300 [00:23<01:22,  2.43it/s, failures=0, objective=244]
 33%|███▎      | 100/300 [00:23<01:22,  2.43it/s, failures=0, objective=244]
 34%|███▎      | 101/300 [00:23<01:21,  2.43it/s, failures=0, objective=244]
 34%|███▍      | 102/300 [00:24<01:19,  2.49it/s, failures=0, objective=244]
 34%|███▍      | 102/300 [00:24<01:19,  2.49it/s, failures=0, objective=246]
 34%|███▍      | 103/300 [00:24<01:19,  2.49it/s, failures=0, objective=246]
 35%|███▍      | 104/300 [00:25<01:13,  2.65it/s, failures=0, objective=246]
 35%|███▍      | 104/300 [00:25<01:13,  2.65it/s, failures=0, objective=246]
 35%|███▌      | 105/300 [00:25<01:13,  2.65it/s, failures=0, objective=246]
 35%|███▌      | 106/300 [00:26<01:11,  2.71it/s, failures=0, objective=246]
 35%|███▌      | 106/300 [00:26<01:11,  2.71it/s, failures=0, objective=246]
 36%|███▌      | 107/300 [00:26<01:11,  2.71it/s, failures=0, objective=246]
 36%|███▌      | 108/300 [00:26<01:10,  2.71it/s, failures=0, objective=246]
 36%|███▌      | 108/300 [00:26<01:10,  2.71it/s, failures=0, objective=246]
 36%|███▋      | 109/300 [00:26<01:10,  2.71it/s, failures=0, objective=246]
 37%|███▋      | 110/300 [00:27<01:09,  2.72it/s, failures=0, objective=246]
 37%|███▋      | 110/300 [00:27<01:09,  2.72it/s, failures=0, objective=246]
 37%|███▋      | 111/300 [00:27<01:09,  2.72it/s, failures=0, objective=246]
 37%|███▋      | 112/300 [00:28<01:15,  2.47it/s, failures=0, objective=246]
 37%|███▋      | 112/300 [00:28<01:15,  2.47it/s, failures=0, objective=249]
 38%|███▊      | 113/300 [00:28<01:15,  2.47it/s, failures=0, objective=249]
 38%|███▊      | 114/300 [00:29<01:19,  2.34it/s, failures=0, objective=249]
 38%|███▊      | 114/300 [00:29<01:19,  2.34it/s, failures=0, objective=249]
 38%|███▊      | 115/300 [00:29<01:18,  2.34it/s, failures=0, objective=249]
 39%|███▊      | 116/300 [00:30<01:22,  2.23it/s, failures=0, objective=249]
 39%|███▊      | 116/300 [00:30<01:22,  2.23it/s, failures=0, objective=249]
 39%|███▉      | 117/300 [00:30<01:21,  2.23it/s, failures=0, objective=249]
 39%|███▉      | 118/300 [00:31<01:19,  2.28it/s, failures=0, objective=249]
 39%|███▉      | 118/300 [00:31<01:19,  2.28it/s, failures=0, objective=249]
 40%|███▉      | 119/300 [00:31<01:19,  2.28it/s, failures=0, objective=249]
 40%|████      | 120/300 [00:32<01:22,  2.18it/s, failures=0, objective=249]
 40%|████      | 120/300 [00:32<01:22,  2.18it/s, failures=0, objective=250]
 40%|████      | 121/300 [00:32<01:22,  2.18it/s, failures=0, objective=250]
 41%|████      | 122/300 [00:33<01:20,  2.20it/s, failures=0, objective=250]
 41%|████      | 122/300 [00:33<01:20,  2.20it/s, failures=0, objective=252]
 41%|████      | 123/300 [00:33<01:20,  2.20it/s, failures=0, objective=252]
 41%|████▏     | 124/300 [00:33<01:14,  2.36it/s, failures=0, objective=252]
 41%|████▏     | 124/300 [00:33<01:14,  2.36it/s, failures=0, objective=252]
 42%|████▏     | 125/300 [00:33<01:14,  2.36it/s, failures=0, objective=252]
 42%|████▏     | 126/300 [00:34<01:08,  2.54it/s, failures=0, objective=252]
 42%|████▏     | 126/300 [00:34<01:08,  2.54it/s, failures=0, objective=252]
 42%|████▏     | 127/300 [00:34<01:08,  2.54it/s, failures=0, objective=252]
 43%|████▎     | 128/300 [00:35<01:01,  2.79it/s, failures=0, objective=252]
 43%|████▎     | 128/300 [00:35<01:01,  2.79it/s, failures=0, objective=252]
 43%|████▎     | 129/300 [00:35<01:01,  2.79it/s, failures=0, objective=252]
 43%|████▎     | 130/300 [00:35<01:04,  2.62it/s, failures=0, objective=252]
 43%|████▎     | 130/300 [00:35<01:04,  2.62it/s, failures=0, objective=252]
 44%|████▎     | 131/300 [00:35<01:04,  2.62it/s, failures=0, objective=252]
 44%|████▍     | 132/300 [00:36<01:06,  2.52it/s, failures=0, objective=252]
 44%|████▍     | 132/300 [00:36<01:06,  2.52it/s, failures=0, objective=252]
 44%|████▍     | 133/300 [00:36<01:06,  2.52it/s, failures=0, objective=252]
 45%|████▍     | 134/300 [00:37<01:07,  2.47it/s, failures=0, objective=252]
 45%|████▍     | 134/300 [00:37<01:07,  2.47it/s, failures=0, objective=253]
 45%|████▌     | 135/300 [00:37<01:06,  2.47it/s, failures=0, objective=253]/Users/rp5/Documents/DeepHyper/deephyper/src/deephyper/skopt/optimizer/optimizer.py:820: UserWarning: The objective has been evaluated at this point before.
  warnings.warn("The objective has been evaluated at this point before.")

 45%|████▌     | 136/300 [00:38<01:07,  2.44it/s, failures=0, objective=253]
 45%|████▌     | 136/300 [00:38<01:07,  2.44it/s, failures=0, objective=253]
 46%|████▌     | 137/300 [00:38<01:06,  2.44it/s, failures=0, objective=253]
 46%|████▌     | 138/300 [00:39<01:12,  2.22it/s, failures=0, objective=253]
 46%|████▌     | 138/300 [00:39<01:12,  2.22it/s, failures=0, objective=255]
 46%|████▋     | 139/300 [00:39<01:12,  2.22it/s, failures=0, objective=255]
 47%|████▋     | 140/300 [00:40<01:10,  2.27it/s, failures=0, objective=255]
 47%|████▋     | 140/300 [00:40<01:10,  2.27it/s, failures=0, objective=257]
 47%|████▋     | 141/300 [00:40<01:10,  2.27it/s, failures=0, objective=257]/Users/rp5/Documents/DeepHyper/deephyper/src/deephyper/skopt/optimizer/optimizer.py:820: UserWarning: The objective has been evaluated at this point before.
  warnings.warn("The objective has been evaluated at this point before.")

 47%|████▋     | 142/300 [00:41<01:13,  2.14it/s, failures=0, objective=257]
 47%|████▋     | 142/300 [00:41<01:13,  2.14it/s, failures=0, objective=257]
 48%|████▊     | 143/300 [00:41<01:13,  2.14it/s, failures=0, objective=257]
 48%|████▊     | 144/300 [00:42<01:08,  2.28it/s, failures=0, objective=257]
 48%|████▊     | 144/300 [00:42<01:08,  2.28it/s, failures=0, objective=257]
 48%|████▊     | 145/300 [00:42<01:07,  2.28it/s, failures=0, objective=257]
 49%|████▊     | 146/300 [00:42<01:03,  2.43it/s, failures=0, objective=257]
 49%|████▊     | 146/300 [00:42<01:03,  2.43it/s, failures=0, objective=257]
 49%|████▉     | 147/300 [00:42<01:03,  2.43it/s, failures=0, objective=257]
 49%|████▉     | 148/300 [00:43<01:04,  2.34it/s, failures=0, objective=257]
 49%|████▉     | 148/300 [00:43<01:04,  2.34it/s, failures=0, objective=257]
 50%|████▉     | 149/300 [00:43<01:04,  2.34it/s, failures=0, objective=257]
 50%|█████     | 150/300 [00:45<01:12,  2.07it/s, failures=0, objective=257]
 50%|█████     | 150/300 [00:45<01:12,  2.07it/s, failures=0, objective=257]
 50%|█████     | 151/300 [00:45<01:11,  2.07it/s, failures=0, objective=257]
 51%|█████     | 152/300 [00:45<01:10,  2.11it/s, failures=0, objective=257]
 51%|█████     | 152/300 [00:45<01:10,  2.11it/s, failures=0, objective=258]
 51%|█████     | 153/300 [00:45<01:09,  2.11it/s, failures=0, objective=258]
 51%|█████▏    | 154/300 [00:46<01:09,  2.10it/s, failures=0, objective=258]
 51%|█████▏    | 154/300 [00:46<01:09,  2.10it/s, failures=0, objective=258]
 52%|█████▏    | 155/300 [00:46<01:09,  2.10it/s, failures=0, objective=258]
 52%|█████▏    | 156/300 [00:47<01:08,  2.11it/s, failures=0, objective=258]
 52%|█████▏    | 156/300 [00:47<01:08,  2.11it/s, failures=0, objective=259]
 52%|█████▏    | 157/300 [00:47<01:07,  2.11it/s, failures=0, objective=259]/Users/rp5/Documents/DeepHyper/deephyper/src/deephyper/skopt/optimizer/optimizer.py:820: UserWarning: The objective has been evaluated at this point before.
  warnings.warn("The objective has been evaluated at this point before.")

 53%|█████▎    | 158/300 [00:48<01:09,  2.04it/s, failures=0, objective=259]
 53%|█████▎    | 158/300 [00:48<01:09,  2.04it/s, failures=0, objective=259]
 53%|█████▎    | 159/300 [00:48<01:09,  2.04it/s, failures=0, objective=259]/Users/rp5/Documents/DeepHyper/deephyper/src/deephyper/skopt/optimizer/optimizer.py:820: UserWarning: The objective has been evaluated at this point before.
  warnings.warn("The objective has been evaluated at this point before.")

 53%|█████▎    | 160/300 [00:49<01:10,  1.99it/s, failures=0, objective=259]
 53%|█████▎    | 160/300 [00:49<01:10,  1.99it/s, failures=0, objective=259]
 54%|█████▎    | 161/300 [00:50<01:09,  1.99it/s, failures=0, objective=259]/Users/rp5/Documents/DeepHyper/deephyper/src/deephyper/skopt/optimizer/optimizer.py:820: UserWarning: The objective has been evaluated at this point before.
  warnings.warn("The objective has been evaluated at this point before.")

 54%|█████▍    | 162/300 [00:50<01:08,  2.00it/s, failures=0, objective=259]
 54%|█████▍    | 162/300 [00:50<01:08,  2.00it/s, failures=0, objective=259]
 54%|█████▍    | 163/300 [00:50<01:08,  2.00it/s, failures=0, objective=259]
 55%|█████▍    | 164/300 [00:51<01:03,  2.13it/s, failures=0, objective=259]
 55%|█████▍    | 164/300 [00:51<01:03,  2.13it/s, failures=0, objective=259]
 55%|█████▌    | 165/300 [00:51<01:03,  2.13it/s, failures=0, objective=259]
 55%|█████▌    | 166/300 [00:52<00:57,  2.35it/s, failures=0, objective=259]
 55%|█████▌    | 166/300 [00:52<00:57,  2.35it/s, failures=0, objective=259]
 56%|█████▌    | 167/300 [00:52<00:56,  2.35it/s, failures=0, objective=259]
 56%|█████▌    | 168/300 [00:53<00:58,  2.27it/s, failures=0, objective=259]
 56%|█████▌    | 168/300 [00:53<00:58,  2.27it/s, failures=0, objective=259]
 56%|█████▋    | 169/300 [00:53<00:57,  2.27it/s, failures=0, objective=259]
 57%|█████▋    | 170/300 [00:53<00:46,  2.82it/s, failures=0, objective=259]
 57%|█████▋    | 170/300 [00:53<00:46,  2.82it/s, failures=0, objective=259]
 57%|█████▋    | 171/300 [00:53<00:45,  2.82it/s, failures=0, objective=259]
 57%|█████▋    | 172/300 [00:54<00:50,  2.52it/s, failures=0, objective=259]
 57%|█████▋    | 172/300 [00:54<00:50,  2.52it/s, failures=0, objective=259]
 58%|█████▊    | 173/300 [00:54<00:50,  2.52it/s, failures=0, objective=259]
 58%|█████▊    | 174/300 [00:55<00:54,  2.29it/s, failures=0, objective=259]
 58%|█████▊    | 174/300 [00:55<00:54,  2.29it/s, failures=0, objective=259]
 58%|█████▊    | 175/300 [00:55<00:54,  2.29it/s, failures=0, objective=259]
 59%|█████▊    | 176/300 [00:56<00:56,  2.20it/s, failures=0, objective=259]
 59%|█████▊    | 176/300 [00:56<00:56,  2.20it/s, failures=0, objective=259]
 59%|█████▉    | 177/300 [00:56<00:55,  2.20it/s, failures=0, objective=259]
 59%|█████▉    | 178/300 [00:57<00:56,  2.14it/s, failures=0, objective=259]
 59%|█████▉    | 178/300 [00:57<00:56,  2.14it/s, failures=0, objective=260]
 60%|█████▉    | 179/300 [00:57<00:56,  2.14it/s, failures=0, objective=260]/Users/rp5/Documents/DeepHyper/deephyper/src/deephyper/skopt/optimizer/optimizer.py:820: UserWarning: The objective has been evaluated at this point before.
  warnings.warn("The objective has been evaluated at this point before.")

 60%|██████    | 180/300 [00:58<00:58,  2.04it/s, failures=0, objective=260]
 60%|██████    | 180/300 [00:58<00:58,  2.04it/s, failures=0, objective=260]
 60%|██████    | 181/300 [00:58<00:58,  2.04it/s, failures=0, objective=260]
 61%|██████    | 182/300 [00:59<00:59,  1.98it/s, failures=0, objective=260]
 61%|██████    | 182/300 [00:59<00:59,  1.98it/s, failures=0, objective=261]
 61%|██████    | 183/300 [00:59<00:59,  1.98it/s, failures=0, objective=261]
 61%|██████▏   | 184/300 [01:00<00:56,  2.04it/s, failures=0, objective=261]
 61%|██████▏   | 184/300 [01:00<00:56,  2.04it/s, failures=0, objective=261]
 62%|██████▏   | 185/300 [01:00<00:56,  2.04it/s, failures=0, objective=261]
 62%|██████▏   | 186/300 [01:01<00:50,  2.27it/s, failures=0, objective=261]
 62%|██████▏   | 186/300 [01:01<00:50,  2.27it/s, failures=0, objective=261]
 62%|██████▏   | 187/300 [01:01<00:49,  2.27it/s, failures=0, objective=261]
 63%|██████▎   | 188/300 [01:02<00:52,  2.13it/s, failures=0, objective=261]
 63%|██████▎   | 188/300 [01:02<00:52,  2.13it/s, failures=0, objective=261]
 63%|██████▎   | 189/300 [01:02<00:52,  2.13it/s, failures=0, objective=261]
 63%|██████▎   | 190/300 [01:03<00:48,  2.28it/s, failures=0, objective=261]
 63%|██████▎   | 190/300 [01:03<00:48,  2.28it/s, failures=0, objective=261]
 64%|██████▎   | 191/300 [01:03<00:47,  2.28it/s, failures=0, objective=261]
 64%|██████▍   | 192/300 [01:04<00:51,  2.10it/s, failures=0, objective=261]
 64%|██████▍   | 192/300 [01:04<00:51,  2.10it/s, failures=0, objective=261]
 64%|██████▍   | 193/300 [01:04<00:51,  2.10it/s, failures=0, objective=261]/Users/rp5/Documents/DeepHyper/deephyper/src/deephyper/skopt/optimizer/optimizer.py:820: UserWarning: The objective has been evaluated at this point before.
  warnings.warn("The objective has been evaluated at this point before.")

 65%|██████▍   | 194/300 [01:05<00:52,  2.00it/s, failures=0, objective=261]
 65%|██████▍   | 194/300 [01:05<00:52,  2.00it/s, failures=0, objective=261]
 65%|██████▌   | 195/300 [01:05<00:52,  2.00it/s, failures=0, objective=261]/Users/rp5/Documents/DeepHyper/deephyper/src/deephyper/skopt/optimizer/optimizer.py:820: UserWarning: The objective has been evaluated at this point before.
  warnings.warn("The objective has been evaluated at this point before.")

 65%|██████▌   | 196/300 [01:06<00:53,  1.93it/s, failures=0, objective=261]
 65%|██████▌   | 196/300 [01:06<00:53,  1.93it/s, failures=0, objective=261]
 66%|██████▌   | 197/300 [01:06<00:53,  1.93it/s, failures=0, objective=261]/Users/rp5/Documents/DeepHyper/deephyper/src/deephyper/skopt/optimizer/optimizer.py:820: UserWarning: The objective has been evaluated at this point before.
  warnings.warn("The objective has been evaluated at this point before.")

 66%|██████▌   | 198/300 [01:07<00:53,  1.92it/s, failures=0, objective=261]
 66%|██████▌   | 198/300 [01:07<00:53,  1.92it/s, failures=0, objective=261]
 66%|██████▋   | 199/300 [01:07<00:52,  1.92it/s, failures=0, objective=261]/Users/rp5/Documents/DeepHyper/deephyper/src/deephyper/skopt/optimizer/optimizer.py:820: UserWarning: The objective has been evaluated at this point before.
  warnings.warn("The objective has been evaluated at this point before.")

 67%|██████▋   | 200/300 [01:08<00:52,  1.90it/s, failures=0, objective=261]
 67%|██████▋   | 200/300 [01:08<00:52,  1.90it/s, failures=0, objective=261]
 67%|██████▋   | 201/300 [01:08<00:52,  1.90it/s, failures=0, objective=261]/Users/rp5/Documents/DeepHyper/deephyper/src/deephyper/skopt/optimizer/optimizer.py:820: UserWarning: The objective has been evaluated at this point before.
  warnings.warn("The objective has been evaluated at this point before.")

 67%|██████▋   | 202/300 [01:09<00:52,  1.88it/s, failures=0, objective=261]
 67%|██████▋   | 202/300 [01:09<00:52,  1.88it/s, failures=0, objective=261]
 68%|██████▊   | 203/300 [01:09<00:51,  1.88it/s, failures=0, objective=261]
 68%|██████▊   | 204/300 [01:11<00:56,  1.71it/s, failures=0, objective=261]
 68%|██████▊   | 204/300 [01:11<00:56,  1.71it/s, failures=0, objective=261]
 68%|██████▊   | 205/300 [01:11<00:55,  1.71it/s, failures=0, objective=261]
 69%|██████▊   | 206/300 [01:11<00:46,  2.02it/s, failures=0, objective=261]
 69%|██████▊   | 206/300 [01:11<00:46,  2.02it/s, failures=0, objective=261]
 69%|██████▉   | 207/300 [01:11<00:46,  2.02it/s, failures=0, objective=261]
 69%|██████▉   | 208/300 [01:12<00:37,  2.44it/s, failures=0, objective=261]
 69%|██████▉   | 208/300 [01:12<00:37,  2.44it/s, failures=0, objective=261]
 70%|██████▉   | 209/300 [01:12<00:37,  2.44it/s, failures=0, objective=261]
 70%|███████   | 210/300 [01:13<00:42,  2.10it/s, failures=0, objective=261]
 70%|███████   | 210/300 [01:13<00:42,  2.10it/s, failures=0, objective=261]
 70%|███████   | 211/300 [01:13<00:42,  2.10it/s, failures=0, objective=261]
 71%|███████   | 212/300 [01:14<00:41,  2.11it/s, failures=0, objective=261]
 71%|███████   | 212/300 [01:14<00:41,  2.11it/s, failures=0, objective=261]
 71%|███████   | 213/300 [01:14<00:41,  2.11it/s, failures=0, objective=261]
 71%|███████▏  | 214/300 [01:14<00:32,  2.62it/s, failures=0, objective=261]
 71%|███████▏  | 214/300 [01:14<00:32,  2.62it/s, failures=0, objective=261]
 72%|███████▏  | 215/300 [01:14<00:32,  2.62it/s, failures=0, objective=261]
 72%|███████▏  | 216/300 [01:15<00:25,  3.23it/s, failures=0, objective=261]
 72%|███████▏  | 216/300 [01:15<00:25,  3.23it/s, failures=0, objective=261]
 72%|███████▏  | 217/300 [01:15<00:25,  3.23it/s, failures=0, objective=261]/Users/rp5/Documents/DeepHyper/deephyper/src/deephyper/skopt/optimizer/optimizer.py:820: UserWarning: The objective has been evaluated at this point before.
  warnings.warn("The objective has been evaluated at this point before.")

 73%|███████▎  | 218/300 [01:16<00:30,  2.66it/s, failures=0, objective=261]
 73%|███████▎  | 218/300 [01:16<00:30,  2.66it/s, failures=0, objective=261]
 73%|███████▎  | 219/300 [01:16<00:30,  2.66it/s, failures=0, objective=261]/Users/rp5/Documents/DeepHyper/deephyper/src/deephyper/skopt/optimizer/optimizer.py:820: UserWarning: The objective has been evaluated at this point before.
  warnings.warn("The objective has been evaluated at this point before.")

 73%|███████▎  | 220/300 [01:17<00:32,  2.49it/s, failures=0, objective=261]
 73%|███████▎  | 220/300 [01:17<00:32,  2.49it/s, failures=0, objective=261]
 74%|███████▎  | 221/300 [01:17<00:31,  2.49it/s, failures=0, objective=261]/Users/rp5/Documents/DeepHyper/deephyper/src/deephyper/skopt/optimizer/optimizer.py:820: UserWarning: The objective has been evaluated at this point before.
  warnings.warn("The objective has been evaluated at this point before.")

 74%|███████▍  | 222/300 [01:17<00:32,  2.43it/s, failures=0, objective=261]
 74%|███████▍  | 222/300 [01:17<00:32,  2.43it/s, failures=0, objective=261]
 74%|███████▍  | 223/300 [01:17<00:31,  2.43it/s, failures=0, objective=261]
 75%|███████▍  | 224/300 [01:18<00:27,  2.76it/s, failures=0, objective=261]
 75%|███████▍  | 224/300 [01:18<00:27,  2.76it/s, failures=0, objective=261]
 75%|███████▌  | 225/300 [01:18<00:27,  2.76it/s, failures=0, objective=261]
 75%|███████▌  | 226/300 [01:19<00:32,  2.31it/s, failures=0, objective=261]
 75%|███████▌  | 226/300 [01:19<00:32,  2.31it/s, failures=0, objective=261]
 76%|███████▌  | 227/300 [01:19<00:31,  2.31it/s, failures=0, objective=261]
 76%|███████▌  | 228/300 [01:20<00:32,  2.23it/s, failures=0, objective=261]
 76%|███████▌  | 228/300 [01:20<00:32,  2.23it/s, failures=0, objective=261]
 76%|███████▋  | 229/300 [01:20<00:31,  2.23it/s, failures=0, objective=261]
 77%|███████▋  | 230/300 [01:21<00:29,  2.34it/s, failures=0, objective=261]
 77%|███████▋  | 230/300 [01:21<00:29,  2.34it/s, failures=0, objective=261]
 77%|███████▋  | 231/300 [01:21<00:29,  2.34it/s, failures=0, objective=261]
 77%|███████▋  | 232/300 [01:22<00:28,  2.37it/s, failures=0, objective=261]
 77%|███████▋  | 232/300 [01:22<00:28,  2.37it/s, failures=0, objective=261]
 78%|███████▊  | 233/300 [01:22<00:28,  2.37it/s, failures=0, objective=261]/Users/rp5/Documents/DeepHyper/deephyper/src/deephyper/skopt/optimizer/optimizer.py:820: UserWarning: The objective has been evaluated at this point before.
  warnings.warn("The objective has been evaluated at this point before.")

 78%|███████▊  | 234/300 [01:23<00:30,  2.13it/s, failures=0, objective=261]
 78%|███████▊  | 234/300 [01:23<00:30,  2.13it/s, failures=0, objective=261]
 78%|███████▊  | 235/300 [01:23<00:30,  2.13it/s, failures=0, objective=261]/Users/rp5/Documents/DeepHyper/deephyper/src/deephyper/skopt/optimizer/optimizer.py:820: UserWarning: The objective has been evaluated at this point before.
  warnings.warn("The objective has been evaluated at this point before.")

 79%|███████▊  | 236/300 [01:24<00:30,  2.07it/s, failures=0, objective=261]
 79%|███████▊  | 236/300 [01:24<00:30,  2.07it/s, failures=0, objective=261]
 79%|███████▉  | 237/300 [01:24<00:30,  2.07it/s, failures=0, objective=261]/Users/rp5/Documents/DeepHyper/deephyper/src/deephyper/skopt/optimizer/optimizer.py:820: UserWarning: The objective has been evaluated at this point before.
  warnings.warn("The objective has been evaluated at this point before.")

 79%|███████▉  | 238/300 [01:25<00:31,  1.96it/s, failures=0, objective=261]
 79%|███████▉  | 238/300 [01:25<00:31,  1.96it/s, failures=0, objective=261]
 80%|███████▉  | 239/300 [01:25<00:31,  1.96it/s, failures=0, objective=261]
 80%|████████  | 240/300 [01:26<00:30,  2.00it/s, failures=0, objective=261]
 80%|████████  | 240/300 [01:26<00:30,  2.00it/s, failures=0, objective=262]
 80%|████████  | 241/300 [01:26<00:29,  2.00it/s, failures=0, objective=262]/Users/rp5/Documents/DeepHyper/deephyper/src/deephyper/skopt/optimizer/optimizer.py:820: UserWarning: The objective has been evaluated at this point before.
  warnings.warn("The objective has been evaluated at this point before.")

 81%|████████  | 242/300 [01:27<00:28,  2.01it/s, failures=0, objective=262]
 81%|████████  | 242/300 [01:27<00:28,  2.01it/s, failures=0, objective=262]
 81%|████████  | 243/300 [01:27<00:28,  2.01it/s, failures=0, objective=262]
 81%|████████▏ | 244/300 [01:28<00:27,  2.03it/s, failures=0, objective=262]
 81%|████████▏ | 244/300 [01:28<00:27,  2.03it/s, failures=0, objective=262]
 82%|████████▏ | 245/300 [01:28<00:27,  2.03it/s, failures=0, objective=262]
 82%|████████▏ | 246/300 [01:29<00:24,  2.20it/s, failures=0, objective=262]
 82%|████████▏ | 246/300 [01:29<00:24,  2.20it/s, failures=0, objective=262]
 82%|████████▏ | 247/300 [01:29<00:24,  2.20it/s, failures=0, objective=262]
 83%|████████▎ | 248/300 [01:30<00:23,  2.22it/s, failures=0, objective=262]
 83%|████████▎ | 248/300 [01:30<00:23,  2.22it/s, failures=0, objective=262]
 83%|████████▎ | 249/300 [01:30<00:23,  2.22it/s, failures=0, objective=262]
 83%|████████▎ | 250/300 [01:30<00:22,  2.20it/s, failures=0, objective=262]
 83%|████████▎ | 250/300 [01:30<00:22,  2.20it/s, failures=0, objective=262]
 84%|████████▎ | 251/300 [01:30<00:22,  2.20it/s, failures=0, objective=262]
 84%|████████▍ | 252/300 [01:31<00:22,  2.13it/s, failures=0, objective=262]
 84%|████████▍ | 252/300 [01:31<00:22,  2.13it/s, failures=0, objective=262]
 84%|████████▍ | 253/300 [01:31<00:22,  2.13it/s, failures=0, objective=262]/Users/rp5/Documents/DeepHyper/deephyper/src/deephyper/skopt/optimizer/optimizer.py:820: UserWarning: The objective has been evaluated at this point before.
  warnings.warn("The objective has been evaluated at this point before.")

 85%|████████▍ | 254/300 [01:32<00:21,  2.11it/s, failures=0, objective=262]
 85%|████████▍ | 254/300 [01:32<00:21,  2.11it/s, failures=0, objective=262]
 85%|████████▌ | 255/300 [01:32<00:21,  2.11it/s, failures=0, objective=262]/Users/rp5/Documents/DeepHyper/deephyper/src/deephyper/skopt/optimizer/optimizer.py:820: UserWarning: The objective has been evaluated at this point before.
  warnings.warn("The objective has been evaluated at this point before.")

 85%|████████▌ | 256/300 [01:34<00:24,  1.82it/s, failures=0, objective=262]
 85%|████████▌ | 256/300 [01:34<00:24,  1.82it/s, failures=0, objective=262]
 86%|████████▌ | 257/300 [01:34<00:23,  1.82it/s, failures=0, objective=262]/Users/rp5/Documents/DeepHyper/deephyper/src/deephyper/skopt/optimizer/optimizer.py:820: UserWarning: The objective has been evaluated at this point before.
  warnings.warn("The objective has been evaluated at this point before.")

 86%|████████▌ | 258/300 [01:35<00:21,  1.96it/s, failures=0, objective=262]
 86%|████████▌ | 258/300 [01:35<00:21,  1.96it/s, failures=0, objective=262]
 86%|████████▋ | 259/300 [01:35<00:20,  1.96it/s, failures=0, objective=262]
 87%|████████▋ | 260/300 [01:36<00:21,  1.85it/s, failures=0, objective=262]
 87%|████████▋ | 260/300 [01:36<00:21,  1.85it/s, failures=0, objective=263]
 87%|████████▋ | 261/300 [01:36<00:21,  1.85it/s, failures=0, objective=263]/Users/rp5/Documents/DeepHyper/deephyper/src/deephyper/skopt/optimizer/optimizer.py:820: UserWarning: The objective has been evaluated at this point before.
  warnings.warn("The objective has been evaluated at this point before.")

 87%|████████▋ | 262/300 [01:37<00:20,  1.83it/s, failures=0, objective=263]
 87%|████████▋ | 262/300 [01:37<00:20,  1.83it/s, failures=0, objective=263]
 88%|████████▊ | 263/300 [01:37<00:20,  1.83it/s, failures=0, objective=263]
 88%|████████▊ | 264/300 [01:38<00:17,  2.02it/s, failures=0, objective=263]
 88%|████████▊ | 264/300 [01:38<00:17,  2.02it/s, failures=0, objective=263]
 88%|████████▊ | 265/300 [01:38<00:17,  2.02it/s, failures=0, objective=263]
 89%|████████▊ | 266/300 [01:39<00:15,  2.19it/s, failures=0, objective=263]
 89%|████████▊ | 266/300 [01:39<00:15,  2.19it/s, failures=0, objective=263]
 89%|████████▉ | 267/300 [01:39<00:15,  2.19it/s, failures=0, objective=263]
 89%|████████▉ | 268/300 [01:39<00:13,  2.38it/s, failures=0, objective=263]
 89%|████████▉ | 268/300 [01:39<00:13,  2.38it/s, failures=0, objective=263]
 90%|████████▉ | 269/300 [01:39<00:13,  2.38it/s, failures=0, objective=263]
 90%|█████████ | 270/300 [01:40<00:12,  2.46it/s, failures=0, objective=263]
 90%|█████████ | 270/300 [01:40<00:12,  2.46it/s, failures=0, objective=263]
 90%|█████████ | 271/300 [01:40<00:11,  2.46it/s, failures=0, objective=263]
 91%|█████████ | 272/300 [01:41<00:12,  2.28it/s, failures=0, objective=263]
 91%|█████████ | 272/300 [01:41<00:12,  2.28it/s, failures=0, objective=263]
 91%|█████████ | 273/300 [01:41<00:11,  2.28it/s, failures=0, objective=263]/Users/rp5/Documents/DeepHyper/deephyper/src/deephyper/skopt/optimizer/optimizer.py:820: UserWarning: The objective has been evaluated at this point before.
  warnings.warn("The objective has been evaluated at this point before.")

 91%|█████████▏| 274/300 [01:42<00:12,  2.04it/s, failures=0, objective=263]
 91%|█████████▏| 274/300 [01:42<00:12,  2.04it/s, failures=0, objective=263]
 92%|█████████▏| 275/300 [01:42<00:12,  2.04it/s, failures=0, objective=263]/Users/rp5/Documents/DeepHyper/deephyper/src/deephyper/skopt/optimizer/optimizer.py:820: UserWarning: The objective has been evaluated at this point before.
  warnings.warn("The objective has been evaluated at this point before.")

 92%|█████████▏| 276/300 [01:43<00:12,  1.88it/s, failures=0, objective=263]
 92%|█████████▏| 276/300 [01:43<00:12,  1.88it/s, failures=0, objective=263]
 92%|█████████▏| 277/300 [01:43<00:12,  1.88it/s, failures=0, objective=263]/Users/rp5/Documents/DeepHyper/deephyper/src/deephyper/skopt/optimizer/optimizer.py:820: UserWarning: The objective has been evaluated at this point before.
  warnings.warn("The objective has been evaluated at this point before.")

 93%|█████████▎| 278/300 [01:45<00:13,  1.68it/s, failures=0, objective=263]
 93%|█████████▎| 278/300 [01:45<00:13,  1.68it/s, failures=0, objective=263]
 93%|█████████▎| 279/300 [01:45<00:12,  1.68it/s, failures=0, objective=263]/Users/rp5/Documents/DeepHyper/deephyper/src/deephyper/skopt/optimizer/optimizer.py:820: UserWarning: The objective has been evaluated at this point before.
  warnings.warn("The objective has been evaluated at this point before.")

 93%|█████████▎| 280/300 [01:46<00:11,  1.76it/s, failures=0, objective=263]
 93%|█████████▎| 280/300 [01:46<00:11,  1.76it/s, failures=0, objective=263]
 94%|█████████▎| 281/300 [01:46<00:10,  1.76it/s, failures=0, objective=263]/Users/rp5/Documents/DeepHyper/deephyper/src/deephyper/skopt/optimizer/optimizer.py:820: UserWarning: The objective has been evaluated at this point before.
  warnings.warn("The objective has been evaluated at this point before.")

 94%|█████████▍| 282/300 [01:48<00:11,  1.53it/s, failures=0, objective=263]
 94%|█████████▍| 282/300 [01:48<00:11,  1.53it/s, failures=0, objective=263]
 94%|█████████▍| 283/300 [01:48<00:11,  1.53it/s, failures=0, objective=263]
 95%|█████████▍| 284/300 [01:49<00:09,  1.67it/s, failures=0, objective=263]
 95%|█████████▍| 284/300 [01:49<00:09,  1.67it/s, failures=0, objective=263]
 95%|█████████▌| 285/300 [01:49<00:09,  1.67it/s, failures=0, objective=263]
 95%|█████████▌| 286/300 [01:49<00:07,  1.84it/s, failures=0, objective=263]
 95%|█████████▌| 286/300 [01:49<00:07,  1.84it/s, failures=0, objective=263]
 96%|█████████▌| 287/300 [01:49<00:07,  1.84it/s, failures=0, objective=263]
 96%|█████████▌| 288/300 [01:51<00:06,  1.84it/s, failures=0, objective=263]
 96%|█████████▌| 288/300 [01:51<00:06,  1.84it/s, failures=0, objective=263]
 96%|█████████▋| 289/300 [01:51<00:05,  1.84it/s, failures=0, objective=263]
 97%|█████████▋| 290/300 [01:51<00:05,  1.95it/s, failures=0, objective=263]
 97%|█████████▋| 290/300 [01:51<00:05,  1.95it/s, failures=0, objective=263]
 97%|█████████▋| 291/300 [01:51<00:04,  1.95it/s, failures=0, objective=263]
 97%|█████████▋| 292/300 [01:53<00:04,  1.81it/s, failures=0, objective=263]
 97%|█████████▋| 292/300 [01:53<00:04,  1.81it/s, failures=0, objective=263]
 98%|█████████▊| 293/300 [01:53<00:03,  1.81it/s, failures=0, objective=263]/Users/rp5/Documents/DeepHyper/deephyper/src/deephyper/skopt/optimizer/optimizer.py:820: UserWarning: The objective has been evaluated at this point before.
  warnings.warn("The objective has been evaluated at this point before.")

 98%|█████████▊| 294/300 [01:54<00:03,  1.71it/s, failures=0, objective=263]
 98%|█████████▊| 294/300 [01:54<00:03,  1.71it/s, failures=0, objective=263]
 98%|█████████▊| 295/300 [01:54<00:02,  1.71it/s, failures=0, objective=263]/Users/rp5/Documents/DeepHyper/deephyper/src/deephyper/skopt/optimizer/optimizer.py:820: UserWarning: The objective has been evaluated at this point before.
  warnings.warn("The objective has been evaluated at this point before.")

 99%|█████████▊| 296/300 [01:56<00:02,  1.58it/s, failures=0, objective=263]
 99%|█████████▊| 296/300 [01:56<00:02,  1.58it/s, failures=0, objective=263]
 99%|█████████▉| 297/300 [01:56<00:01,  1.58it/s, failures=0, objective=263]/Users/rp5/Documents/DeepHyper/deephyper/src/deephyper/skopt/optimizer/optimizer.py:820: UserWarning: The objective has been evaluated at this point before.
  warnings.warn("The objective has been evaluated at this point before.")

 99%|█████████▉| 298/300 [01:57<00:01,  1.63it/s, failures=0, objective=263]
 99%|█████████▉| 298/300 [01:57<00:01,  1.63it/s, failures=0, objective=263]
100%|█████████▉| 299/300 [01:57<00:00,  1.63it/s, failures=0, objective=263]/Users/rp5/Documents/DeepHyper/deephyper/src/deephyper/skopt/optimizer/optimizer.py:820: UserWarning: The objective has been evaluated at this point before.
  warnings.warn("The objective has been evaluated at this point before.")

100%|██████████| 300/300 [01:58<00:00,  1.67it/s, failures=0, objective=263]
100%|██████████| 300/300 [01:58<00:00,  1.67it/s, failures=0, objective=263]
100%|██████████| 300/300 [01:58<00:00,  2.54it/s, failures=0, objective=263]

results
p:x0 p:x1 p:x2 p:x3 p:x4 p:x5 p:x6 p:x7 p:x8 p:x9 objective job_id job_status m:timestamp_submit m:timestamp_gather
0 3 6 12 16 17 18 21 26 27 31 177.0 0 DONE 0.013970 0.014726
1 2 4 5 7 13 19 22 25 28 30 155.0 1 DONE 0.030732 0.031577
2 3 4 6 7 8 12 14 19 20 26 119.0 2 DONE 0.040470 0.041046
3 5 6 7 9 10 16 18 19 21 25 136.0 3 DONE 0.049747 0.050385
4 3 5 6 11 13 16 18 23 25 30 150.0 4 DONE 0.059143 0.059724
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
295 20 23 24 25 26 27 28 29 30 31 263.0 295 DONE 116.029499 116.030171
296 11 16 19 22 23 24 25 27 28 30 225.0 296 DONE 116.052359 116.053000
297 20 23 24 25 26 27 28 29 30 31 263.0 297 DONE 117.169291 117.169883
298 12 17 18 19 23 26 27 29 30 31 232.0 298 DONE 117.188699 117.189303
299 20 23 24 25 26 27 28 29 30 31 263.0 299 DONE 118.302572 118.303206

300 rows × 15 columns



Extracting the Best Parameters#

To recover the parameters corresponding to the best observed objective value, we can use deephyper.analysis.hpo.parameters_at_max().

parameters, objective = parameters_at_max(results)
print("\nOptimum values")
for i in range(n):
    print(f"x{i}: {parameters[f'x{i}']:.3f}")
print("objective:", objective)

Optimum values
x0: 20.000
x1: 23.000
x2: 24.000
x3: 25.000
x4: 26.000
x5: 27.000
x6: 28.000
x7: 29.000
x8: 30.000
x9: 31.000
objective: 263.0

Visualization#

We conclude with:

  • a search trajectomakery plot showing the best objective value over time, where the periodic exploration schedule is clearly visible;

  • a feasible-space evaluation plot showing all sampled curves \(i \mapsto x_i\) (each curve is one evaluation), colored by objective value.

These visualizations confirm that the optimizer progressively learns the structure of the monotonic constraint and approaches the theoretical optimum.

WIDTH_PLOTS = 8
HEIGHT_PLOTS = WIDTH_PLOTS / 1.618

fig, ax = plt.subplots(figsize=(WIDTH_PLOTS, HEIGHT_PLOTS))
plot_search_trajectory_single_objective_hpo(results, mode="max", ax=ax)
_ = plt.title("Search Trajectory")
Search Trajectory

Visualizing the Feasible Region and Evaluations#

We now plot all evaluated points in the (x, y) plane, color-coded by objective value, along with the constraint boundary x + y = 10.

results, _ = filter_failed_objectives(results)

p_columns = [col for col in results.columns if col.startswith("p:")]

# Create a normalizer over the objective range
obj_vals = results["objective"]
norm = colors.Normalize(vmin=obj_vals.min(), vmax=obj_vals.max())

# Choose a colormap (viridis is a good default)
cmap = plt.get_cmap("viridis")

fig, ax = plt.subplots(figsize=(WIDTH_PLOTS, HEIGHT_PLOTS))

for i, row in results.iterrows():
    x_values = row[p_columns].values
    y_values = np.arange(n)
    obj_value = row["objective"]

    color = cmap(norm(obj_value))  # map objective → color
    ax.plot(x_values, y_values, color=color, alpha=0.9)

# Optionally add a colorbar
sm = cm.ScalarMappable(norm=norm, cmap=cmap)
cbar = fig.colorbar(sm, ax=ax)
cbar.set_label("Objective value")
ax.grid()
ax.set_ylim(0, n - 1)
ax.set_xlim(0, m)
ax.set_ylabel(r"$i$")
ax.set_xlabel(r"$x_i$")
ax.set_yticks(list(range(n)), [str(i) for i in range(n)])
ax.set_xticks(list(range(0, m, 2)), [str(i) for i in range(0, m, 2)])
plt.show()
plot constrained black box optimization chained sampler

Total running time of the script: (2 minutes 0.512 seconds)

Gallery generated by Sphinx-Gallery

Contents