Constrained Black-Box Optimization with Custom Chained Sampler

Note

Go to the end to download the full example code.

Constrained Black-Box Optimization with Custom Chained Sampler#

Author(s): Romain Egele.

This tutorial demonstrates how to solve constrained black-box optimization using DeepHyper, focusing on how to encode structural constraints directly in the sampling strategy of the search algorithm.

Black-box optimization aims to optimize an unknown function \(f(x) = y \in \mathbb{R}\) using only input–output evaluations \(\{(x_1, y_1), \ldots, (x_n, y_n)\}\). No analytical gradients or structural properties of \(f\) are required.

In constrained settings, the search is further restricted to parameters that satisfy one or more feasibility rules. Constraints can significantly reshape the search space and modify the behavior of the optimizer.

Problem Setting#

In this example, we consider a discrete, ordered search space of dimension \(N\). Each variable \(x_i\) must satisfy the monotonicity constraint

\[x_0 < x_1 < \cdots < x_{N-1}.\]

Each \(x_i\) is bounded between \(i\) and \(m - N + i\). This could tipically represent the layer indexes to drop in Depth pruning of Large language models. The objective is to maximize the sum

\[f(x) = \sum_{i=0}^{N-1} x_i.\]

Since the optimal strategy is to push every variable as high as possible while respecting monotonicity, the theoretical optimum is:

\[\sum_{i=0}^{N-1} (m - i).\]

DeepHyper offers several ways to incorporate constraints:

  1. Custom chained sampler (this tutorial): constraints are enforced directly when generating new candidate points.

  2. Rejection sampling: see Constrained Black-Box Optimization with Rejection Sampling.

  3. Learning to avoid failures (CBO auto-handles failed evaluations): see this tutorial and also Learn to Avoid Failures with Bayesian Optimization.

  4. Multi-objective approach where the constraint becomes an additional objective (tutorial forthcoming).

import matplotlib.pyplot as plt
import matplotlib.cm as cm
import matplotlib.colors as colors
import numpy as np
import pandas as pd

from deephyper.analysis.hpo import (
    plot_search_trajectory_single_objective_hpo,
    parameters_at_max,
    filter_failed_objectives,
)
from deephyper.hpo import HpProblem, CBO

Custom Chained Sampler#

Because every \(x_i\) must be strictly larger than \(x_{i-1}\), the usual independent sampling over each variable would frequently violate the constraint.

Instead, we implement a chained sampler: each \(x_k\) is sampled conditionally so that enough “room” remains for future variables. This ensures that:

  • all generated samples satisfy \(x_i < x_{i+1}\) by construction;

  • the sampler focuses on the feasible region, avoiding wasted evaluations.

n = 10
m = 32

print("optimum:", sum([m - i - 1 for i in range(n)]))

pb = HpProblem()
for i in range(n):
    pb.add((i, m - n + i - 1), f"x{i}")


def sampling_fn(size: int) -> list[dict]:
    def sample_chain():
        # Chain the sampling
        vals = []
        lo = 0
        for k in range(n):
            low = max(k, lo + 1 if k > 0 else 0)
            high = m - n + k
            v = np.random.randint(low, high - (n - 1 - k))  # keep room for future vars
            vals.append(v)
            lo = v
        return {k: v for k, v in zip(pb.hyperparameter_names, vals)}

    return [sample_chain() for _ in range(size)]


pb.set_sampling_fn(sampling_fn)

optimum: 265

Constraint Function#

Although the sampler already generates feasible points, we explicitly define a constraint_fn. This allows DeepHyper to properly handle failed trials (e.g., from manually constructed parameter sets or mutation-based acquisition optimizers). Not only that, this will help report non-feasible points using "F_constraint" in the objective function so that CBO learns to avoid them.

def constraint_fn(df: pd.DataFrame) -> pd.Series:
    accept = pd.Series(np.ones((len(df)), dtype=bool))
    for i in range(n - 1):
        accept = accept & (df[f"x{i}"] < df[f"x{i + 1}"])
    return accept


pb.set_constraint_fn(constraint_fn)


def f(job):
    """Objective function: maximize sum(x_i)."""
    df = pd.DataFrame([job.parameters])
    accept = constraint_fn(df)
    if all(accept):
        return sum(job.parameters.values())
    else:
        return "F_constraint"

Bayesian Optimization with Mixed-GA Acquisition Optimization#

We run a Centralized Bayesian Optimization (CBO) search using:

  • Ensemble of Trees surrogate model ("ET").

  • A mixed genetic algorithm ("mixedga") to optimize the acquisition function.

  • A periodically decaying scheduler on the exploration parameter kappa.

This setup is well suited for discrete, irregularly constrained spaces.

The search runs for max_evals=300 iterations.

search = CBO(
    pb,
    surrogate_model="ET",
    surrogate_model_kwargs={"max_features": "sqrt"},
    acq_optimizer="mixedga",
    acq_optimizer_kwargs={
        "n_points": 1_000,
        "acq_optimizer_freq": 2,
        "filter_failures": "mean",
    },
    acq_func_kwargs={
        # Exploration/Exploitation mechanism
        "kappa": 10.0,
        "scheduler": {
            "type": "periodic-exp-decay",
            "period": 20,
            "kappa_final": 0.1,
        },
    },
    verbose=1,
)
results = search.search(f, max_evals=300)

  0%|          | 0/300 [00:00<?, ?it/s]
  0%|          | 1/300 [00:00<00:00, 7002.18it/s, failures=0, objective=191]
  1%|          | 2/300 [00:00<00:02, 125.11it/s, failures=0, objective=191]
  1%|          | 3/300 [00:00<00:02, 99.83it/s, failures=0, objective=198]
  1%|▏         | 4/300 [00:00<00:03, 90.11it/s, failures=0, objective=198]
  2%|▏         | 5/300 [00:00<00:03, 86.22it/s, failures=0, objective=198]
  2%|▏         | 6/300 [00:00<00:03, 83.64it/s, failures=0, objective=201]
  2%|▏         | 7/300 [00:00<00:03, 82.16it/s, failures=0, objective=201]
  3%|▎         | 8/300 [00:00<00:03, 80.89it/s, failures=0, objective=201]
  3%|▎         | 9/300 [00:00<00:03, 79.96it/s, failures=0, objective=201]
  3%|▎         | 9/300 [00:00<00:03, 79.96it/s, failures=0, objective=202]
  3%|▎         | 10/300 [00:00<00:03, 79.96it/s, failures=0, objective=202]
  4%|▎         | 11/300 [00:00<00:03, 79.96it/s, failures=0, objective=202]
  4%|▍         | 12/300 [00:00<00:03, 79.96it/s, failures=0, objective=202]
  4%|▍         | 13/300 [00:00<00:03, 79.96it/s, failures=0, objective=202]
  5%|▍         | 14/300 [00:00<00:03, 79.96it/s, failures=0, objective=202]
  5%|▌         | 15/300 [00:00<00:03, 79.96it/s, failures=0, objective=202]
  5%|▌         | 16/300 [00:00<00:03, 79.96it/s, failures=0, objective=202]
  6%|▌         | 17/300 [00:00<00:03, 75.88it/s, failures=0, objective=202]
  6%|▌         | 17/300 [00:00<00:03, 75.88it/s, failures=0, objective=205]
  6%|▌         | 18/300 [00:00<00:03, 75.88it/s, failures=0, objective=205]
  6%|▋         | 19/300 [00:00<00:03, 75.88it/s, failures=0, objective=205]
  7%|▋         | 20/300 [00:00<00:03, 75.88it/s, failures=0, objective=205]
  7%|▋         | 21/300 [00:00<00:03, 75.88it/s, failures=0, objective=205]
  7%|▋         | 22/300 [00:00<00:03, 75.88it/s, failures=0, objective=205]
  8%|▊         | 23/300 [00:00<00:03, 75.88it/s, failures=0, objective=205]
  8%|▊         | 24/300 [00:01<00:03, 75.88it/s, failures=0, objective=205]
  8%|▊         | 25/300 [00:01<00:16, 16.22it/s, failures=0, objective=205]
  8%|▊         | 25/300 [00:01<00:16, 16.22it/s, failures=0, objective=205]
  9%|▊         | 26/300 [00:01<00:16, 16.22it/s, failures=0, objective=205]
  9%|▉         | 27/300 [00:01<00:16, 16.22it/s, failures=0, objective=205]
  9%|▉         | 28/300 [00:01<00:16, 16.22it/s, failures=0, objective=205]
 10%|▉         | 29/300 [00:01<00:16, 16.22it/s, failures=0, objective=205]
 10%|█         | 30/300 [00:02<00:26, 10.07it/s, failures=0, objective=205]
 10%|█         | 30/300 [00:02<00:26, 10.07it/s, failures=0, objective=205]
 10%|█         | 31/300 [00:02<00:26, 10.07it/s, failures=0, objective=205]
 11%|█         | 32/300 [00:02<00:26, 10.07it/s, failures=0, objective=205]
 11%|█         | 33/300 [00:02<00:26, 10.16it/s, failures=0, objective=205]
 11%|█         | 33/300 [00:02<00:26, 10.16it/s, failures=0, objective=205]
 11%|█▏        | 34/300 [00:02<00:26, 10.16it/s, failures=0, objective=205]
 12%|█▏        | 35/300 [00:02<00:26, 10.16it/s, failures=0, objective=206]
 12%|█▏        | 36/300 [00:02<00:30,  8.61it/s, failures=0, objective=206]
 12%|█▏        | 36/300 [00:02<00:30,  8.61it/s, failures=0, objective=207]
 12%|█▏        | 37/300 [00:03<00:30,  8.61it/s, failures=0, objective=207]
 13%|█▎        | 38/300 [00:03<00:32,  7.97it/s, failures=0, objective=207]
 13%|█▎        | 38/300 [00:03<00:32,  7.97it/s, failures=0, objective=207]
 13%|█▎        | 39/300 [00:03<00:32,  7.97it/s, failures=0, objective=208]
 13%|█▎        | 40/300 [00:03<00:33,  7.77it/s, failures=0, objective=208]
 13%|█▎        | 40/300 [00:03<00:33,  7.77it/s, failures=0, objective=208]
 14%|█▎        | 41/300 [00:03<00:33,  7.77it/s, failures=0, objective=208]
 14%|█▍        | 42/300 [00:03<00:33,  7.61it/s, failures=0, objective=208]
 14%|█▍        | 42/300 [00:03<00:33,  7.61it/s, failures=0, objective=208]
 14%|█▍        | 43/300 [00:03<00:33,  7.61it/s, failures=0, objective=208]
 15%|█▍        | 44/300 [00:04<00:34,  7.34it/s, failures=0, objective=208]
 15%|█▍        | 44/300 [00:04<00:34,  7.34it/s, failures=0, objective=208]
 15%|█▌        | 45/300 [00:04<00:34,  7.34it/s, failures=0, objective=208]
 15%|█▌        | 46/300 [00:04<00:36,  6.93it/s, failures=0, objective=208]
 15%|█▌        | 46/300 [00:04<00:36,  6.93it/s, failures=0, objective=208]
 16%|█▌        | 47/300 [00:04<00:36,  6.93it/s, failures=0, objective=208]
 16%|█▌        | 48/300 [00:04<00:38,  6.63it/s, failures=0, objective=208]
 16%|█▌        | 48/300 [00:04<00:38,  6.63it/s, failures=0, objective=208]
 16%|█▋        | 49/300 [00:04<00:37,  6.63it/s, failures=0, objective=208]
 17%|█▋        | 50/300 [00:05<00:43,  5.73it/s, failures=0, objective=208]
 17%|█▋        | 50/300 [00:05<00:43,  5.73it/s, failures=0, objective=208]
 17%|█▋        | 51/300 [00:05<00:43,  5.73it/s, failures=0, objective=208]
 17%|█▋        | 52/300 [00:05<00:44,  5.59it/s, failures=0, objective=208]
 17%|█▋        | 52/300 [00:05<00:44,  5.59it/s, failures=0, objective=208]
 18%|█▊        | 53/300 [00:05<00:44,  5.59it/s, failures=0, objective=208]
 18%|█▊        | 54/300 [00:06<00:46,  5.30it/s, failures=0, objective=208]
 18%|█▊        | 54/300 [00:06<00:46,  5.30it/s, failures=0, objective=208]
 18%|█▊        | 55/300 [00:06<00:46,  5.30it/s, failures=0, objective=208]
 19%|█▊        | 56/300 [00:06<00:46,  5.27it/s, failures=0, objective=208]
 19%|█▊        | 56/300 [00:06<00:46,  5.27it/s, failures=0, objective=209]
 19%|█▉        | 57/300 [00:06<00:46,  5.27it/s, failures=0, objective=209]
 19%|█▉        | 58/300 [00:06<00:45,  5.36it/s, failures=0, objective=209]
 19%|█▉        | 58/300 [00:06<00:45,  5.36it/s, failures=0, objective=211]
 20%|█▉        | 59/300 [00:06<00:44,  5.36it/s, failures=0, objective=211]
 20%|██        | 60/300 [00:07<00:49,  4.84it/s, failures=0, objective=211]
 20%|██        | 60/300 [00:07<00:49,  4.84it/s, failures=0, objective=211]
 20%|██        | 61/300 [00:07<00:49,  4.84it/s, failures=0, objective=211]
 21%|██        | 62/300 [00:07<00:49,  4.76it/s, failures=0, objective=211]
 21%|██        | 62/300 [00:07<00:49,  4.76it/s, failures=0, objective=211]
 21%|██        | 63/300 [00:07<00:49,  4.76it/s, failures=0, objective=211]
 21%|██▏       | 64/300 [00:08<00:50,  4.65it/s, failures=0, objective=211]
 21%|██▏       | 64/300 [00:08<00:50,  4.65it/s, failures=0, objective=211]
 22%|██▏       | 65/300 [00:08<00:50,  4.65it/s, failures=0, objective=211]
 22%|██▏       | 66/300 [00:08<00:51,  4.51it/s, failures=0, objective=211]
 22%|██▏       | 66/300 [00:08<00:51,  4.51it/s, failures=0, objective=211]
 22%|██▏       | 67/300 [00:08<00:51,  4.51it/s, failures=0, objective=211]
 23%|██▎       | 68/300 [00:09<01:04,  3.57it/s, failures=0, objective=211]
 23%|██▎       | 68/300 [00:09<01:04,  3.57it/s, failures=0, objective=211]
 23%|██▎       | 69/300 [00:09<01:04,  3.57it/s, failures=0, objective=211]
 23%|██▎       | 70/300 [00:10<01:17,  2.96it/s, failures=0, objective=211]
 23%|██▎       | 70/300 [00:10<01:17,  2.96it/s, failures=0, objective=211]
 24%|██▎       | 71/300 [00:10<01:17,  2.96it/s, failures=0, objective=211]
 24%|██▍       | 72/300 [00:11<01:12,  3.16it/s, failures=0, objective=211]
 24%|██▍       | 72/300 [00:11<01:12,  3.16it/s, failures=0, objective=212]
 24%|██▍       | 73/300 [00:11<01:11,  3.16it/s, failures=0, objective=212]
 25%|██▍       | 74/300 [00:11<01:06,  3.39it/s, failures=0, objective=212]
 25%|██▍       | 74/300 [00:11<01:06,  3.39it/s, failures=0, objective=215]
 25%|██▌       | 75/300 [00:11<01:06,  3.39it/s, failures=0, objective=215]
 25%|██▌       | 76/300 [00:12<01:04,  3.48it/s, failures=0, objective=215]
 25%|██▌       | 76/300 [00:12<01:04,  3.48it/s, failures=0, objective=216]
 26%|██▌       | 77/300 [00:12<01:04,  3.48it/s, failures=0, objective=216]
 26%|██▌       | 78/300 [00:12<01:04,  3.47it/s, failures=0, objective=216]
 26%|██▌       | 78/300 [00:12<01:04,  3.47it/s, failures=0, objective=220]
 26%|██▋       | 79/300 [00:12<01:03,  3.47it/s, failures=0, objective=220]
 27%|██▋       | 80/300 [00:13<01:02,  3.54it/s, failures=0, objective=220]
 27%|██▋       | 80/300 [00:13<01:02,  3.54it/s, failures=0, objective=223]
 27%|██▋       | 81/300 [00:13<01:01,  3.54it/s, failures=0, objective=223]/Users/rp5/Documents/DeepHyper/deephyper/src/deephyper/skopt/optimizer/optimizer.py:820: UserWarning: The objective has been evaluated at this point before.
  warnings.warn("The objective has been evaluated at this point before.")

 27%|██▋       | 82/300 [00:13<01:01,  3.52it/s, failures=0, objective=223]
 27%|██▋       | 82/300 [00:13<01:01,  3.52it/s, failures=0, objective=223]
 28%|██▊       | 83/300 [00:13<01:01,  3.52it/s, failures=0, objective=223]
 28%|██▊       | 84/300 [00:14<00:58,  3.71it/s, failures=0, objective=223]
 28%|██▊       | 84/300 [00:14<00:58,  3.71it/s, failures=0, objective=223]
 28%|██▊       | 85/300 [00:14<00:58,  3.71it/s, failures=0, objective=223]
 29%|██▊       | 86/300 [00:15<01:13,  2.92it/s, failures=0, objective=223]
 29%|██▊       | 86/300 [00:15<01:13,  2.92it/s, failures=0, objective=223]
 29%|██▉       | 87/300 [00:15<01:12,  2.92it/s, failures=0, objective=223]
 29%|██▉       | 88/300 [00:15<01:09,  3.06it/s, failures=0, objective=223]
 29%|██▉       | 88/300 [00:15<01:09,  3.06it/s, failures=0, objective=223]
 30%|██▉       | 89/300 [00:15<01:08,  3.06it/s, failures=0, objective=223]
 30%|███       | 90/300 [00:16<01:06,  3.14it/s, failures=0, objective=223]
 30%|███       | 90/300 [00:16<01:06,  3.14it/s, failures=0, objective=223]
 30%|███       | 91/300 [00:16<01:06,  3.14it/s, failures=0, objective=223]
 31%|███       | 92/300 [00:17<01:05,  3.18it/s, failures=0, objective=223]
 31%|███       | 92/300 [00:17<01:05,  3.18it/s, failures=0, objective=225]
 31%|███       | 93/300 [00:17<01:05,  3.18it/s, failures=0, objective=225]
 31%|███▏      | 94/300 [00:17<01:07,  3.06it/s, failures=0, objective=225]
 31%|███▏      | 94/300 [00:17<01:07,  3.06it/s, failures=0, objective=225]
 32%|███▏      | 95/300 [00:17<01:07,  3.06it/s, failures=0, objective=225]
 32%|███▏      | 96/300 [00:18<01:04,  3.14it/s, failures=0, objective=225]
 32%|███▏      | 96/300 [00:18<01:04,  3.14it/s, failures=0, objective=225]
 32%|███▏      | 97/300 [00:18<01:04,  3.14it/s, failures=0, objective=225]
 33%|███▎      | 98/300 [00:19<01:03,  3.20it/s, failures=0, objective=225]
 33%|███▎      | 98/300 [00:19<01:03,  3.20it/s, failures=0, objective=225]
 33%|███▎      | 99/300 [00:19<01:02,  3.20it/s, failures=0, objective=225]
 33%|███▎      | 100/300 [00:19<01:02,  3.20it/s, failures=0, objective=225]
 33%|███▎      | 100/300 [00:19<01:02,  3.20it/s, failures=0, objective=226]
 34%|███▎      | 101/300 [00:19<01:02,  3.20it/s, failures=0, objective=226]
 34%|███▍      | 102/300 [00:20<01:02,  3.18it/s, failures=0, objective=226]
 34%|███▍      | 102/300 [00:20<01:02,  3.18it/s, failures=0, objective=226]
 34%|███▍      | 103/300 [00:20<01:01,  3.18it/s, failures=0, objective=226]
 35%|███▍      | 104/300 [00:20<01:00,  3.23it/s, failures=0, objective=226]
 35%|███▍      | 104/300 [00:20<01:00,  3.23it/s, failures=0, objective=226]
 35%|███▌      | 105/300 [00:20<01:00,  3.23it/s, failures=0, objective=226]
 35%|███▌      | 106/300 [00:21<01:07,  2.89it/s, failures=0, objective=226]
 35%|███▌      | 106/300 [00:21<01:07,  2.89it/s, failures=0, objective=226]
 36%|███▌      | 107/300 [00:21<01:06,  2.89it/s, failures=0, objective=226]
 36%|███▌      | 108/300 [00:22<01:02,  3.06it/s, failures=0, objective=226]
 36%|███▌      | 108/300 [00:22<01:02,  3.06it/s, failures=0, objective=226]
 36%|███▋      | 109/300 [00:22<01:02,  3.06it/s, failures=0, objective=226]
 37%|███▋      | 110/300 [00:22<01:00,  3.15it/s, failures=0, objective=226]
 37%|███▋      | 110/300 [00:22<01:00,  3.15it/s, failures=0, objective=226]
 37%|███▋      | 111/300 [00:22<00:59,  3.15it/s, failures=0, objective=226]
 37%|███▋      | 112/300 [00:23<01:01,  3.05it/s, failures=0, objective=226]
 37%|███▋      | 112/300 [00:23<01:01,  3.05it/s, failures=0, objective=226]
 38%|███▊      | 113/300 [00:23<01:01,  3.05it/s, failures=0, objective=226]
 38%|███▊      | 114/300 [00:24<01:02,  2.96it/s, failures=0, objective=226]
 38%|███▊      | 114/300 [00:24<01:02,  2.96it/s, failures=0, objective=227]
 38%|███▊      | 115/300 [00:24<01:02,  2.96it/s, failures=0, objective=227]
 39%|███▊      | 116/300 [00:25<01:03,  2.91it/s, failures=0, objective=227]
 39%|███▊      | 116/300 [00:25<01:03,  2.91it/s, failures=0, objective=227]
 39%|███▉      | 117/300 [00:25<01:02,  2.91it/s, failures=0, objective=227]
 39%|███▉      | 118/300 [00:25<01:03,  2.86it/s, failures=0, objective=227]
 39%|███▉      | 118/300 [00:25<01:03,  2.86it/s, failures=0, objective=228]
 40%|███▉      | 119/300 [00:25<01:03,  2.86it/s, failures=0, objective=228]
 40%|████      | 120/300 [00:26<01:03,  2.85it/s, failures=0, objective=228]
 40%|████      | 120/300 [00:26<01:03,  2.85it/s, failures=0, objective=228]
 40%|████      | 121/300 [00:26<01:02,  2.85it/s, failures=0, objective=228]/Users/rp5/Documents/DeepHyper/deephyper/src/deephyper/skopt/optimizer/optimizer.py:820: UserWarning: The objective has been evaluated at this point before.
  warnings.warn("The objective has been evaluated at this point before.")

 41%|████      | 122/300 [00:27<01:02,  2.87it/s, failures=0, objective=228]
 41%|████      | 122/300 [00:27<01:02,  2.87it/s, failures=0, objective=228]
 41%|████      | 123/300 [00:27<01:01,  2.87it/s, failures=0, objective=228]
 41%|████▏     | 124/300 [00:27<01:02,  2.84it/s, failures=0, objective=228]
 41%|████▏     | 124/300 [00:27<01:02,  2.84it/s, failures=0, objective=228]
 42%|████▏     | 125/300 [00:27<01:01,  2.84it/s, failures=0, objective=228]
 42%|████▏     | 126/300 [00:28<01:05,  2.67it/s, failures=0, objective=228]
 42%|████▏     | 126/300 [00:28<01:05,  2.67it/s, failures=0, objective=228]
 42%|████▏     | 127/300 [00:28<01:04,  2.67it/s, failures=0, objective=228]
 43%|████▎     | 128/300 [00:29<01:04,  2.67it/s, failures=0, objective=228]
 43%|████▎     | 128/300 [00:29<01:04,  2.67it/s, failures=0, objective=228]
 43%|████▎     | 129/300 [00:29<01:03,  2.67it/s, failures=0, objective=228]
 43%|████▎     | 130/300 [00:30<01:13,  2.31it/s, failures=0, objective=228]
 43%|████▎     | 130/300 [00:30<01:13,  2.31it/s, failures=0, objective=228]
 44%|████▎     | 131/300 [00:30<01:13,  2.31it/s, failures=0, objective=228]
 44%|████▍     | 132/300 [00:31<01:08,  2.45it/s, failures=0, objective=228]
 44%|████▍     | 132/300 [00:31<01:08,  2.45it/s, failures=0, objective=228]
 44%|████▍     | 133/300 [00:31<01:08,  2.45it/s, failures=0, objective=228]
 45%|████▍     | 134/300 [00:32<01:06,  2.49it/s, failures=0, objective=228]
 45%|████▍     | 134/300 [00:32<01:06,  2.49it/s, failures=0, objective=228]
 45%|████▌     | 135/300 [00:32<01:06,  2.49it/s, failures=0, objective=228]
 45%|████▌     | 136/300 [00:32<01:04,  2.54it/s, failures=0, objective=228]
 45%|████▌     | 136/300 [00:32<01:04,  2.54it/s, failures=0, objective=229]
 46%|████▌     | 137/300 [00:32<01:04,  2.54it/s, failures=0, objective=229]
 46%|████▌     | 138/300 [00:33<01:04,  2.52it/s, failures=0, objective=229]
 46%|████▌     | 138/300 [00:33<01:04,  2.52it/s, failures=0, objective=229]
 46%|████▋     | 139/300 [00:33<01:03,  2.52it/s, failures=0, objective=229]/Users/rp5/Documents/DeepHyper/deephyper/src/deephyper/skopt/optimizer/optimizer.py:820: UserWarning: The objective has been evaluated at this point before.
  warnings.warn("The objective has been evaluated at this point before.")

 47%|████▋     | 140/300 [00:34<01:02,  2.56it/s, failures=0, objective=229]
 47%|████▋     | 140/300 [00:34<01:02,  2.56it/s, failures=0, objective=229]
 47%|████▋     | 141/300 [00:34<01:02,  2.56it/s, failures=0, objective=229]/Users/rp5/Documents/DeepHyper/deephyper/src/deephyper/skopt/optimizer/optimizer.py:820: UserWarning: The objective has been evaluated at this point before.
  warnings.warn("The objective has been evaluated at this point before.")

 47%|████▋     | 142/300 [00:35<01:01,  2.57it/s, failures=0, objective=229]
 47%|████▋     | 142/300 [00:35<01:01,  2.57it/s, failures=0, objective=229]
 48%|████▊     | 143/300 [00:35<01:01,  2.57it/s, failures=0, objective=229]
 48%|████▊     | 144/300 [00:35<00:57,  2.71it/s, failures=0, objective=229]
 48%|████▊     | 144/300 [00:35<00:57,  2.71it/s, failures=0, objective=229]
 48%|████▊     | 145/300 [00:35<00:57,  2.71it/s, failures=0, objective=229]
 49%|████▊     | 146/300 [00:37<01:12,  2.13it/s, failures=0, objective=229]
 49%|████▊     | 146/300 [00:37<01:12,  2.13it/s, failures=0, objective=229]
 49%|████▉     | 147/300 [00:37<01:11,  2.13it/s, failures=0, objective=229]
 49%|████▉     | 148/300 [00:37<01:05,  2.32it/s, failures=0, objective=229]
 49%|████▉     | 148/300 [00:37<01:05,  2.32it/s, failures=0, objective=229]
 50%|████▉     | 149/300 [00:37<01:05,  2.32it/s, failures=0, objective=229]
 50%|█████     | 150/300 [00:38<01:06,  2.25it/s, failures=0, objective=229]
 50%|█████     | 150/300 [00:38<01:06,  2.25it/s, failures=0, objective=229]
 50%|█████     | 151/300 [00:38<01:06,  2.25it/s, failures=0, objective=229]
 51%|█████     | 152/300 [00:39<01:03,  2.35it/s, failures=0, objective=229]
 51%|█████     | 152/300 [00:39<01:03,  2.35it/s, failures=0, objective=229]
 51%|█████     | 153/300 [00:39<01:02,  2.35it/s, failures=0, objective=229]
 51%|█████▏    | 154/300 [00:40<01:01,  2.38it/s, failures=0, objective=229]
 51%|█████▏    | 154/300 [00:40<01:01,  2.38it/s, failures=0, objective=229]
 52%|█████▏    | 155/300 [00:40<01:00,  2.38it/s, failures=0, objective=229]
 52%|█████▏    | 156/300 [00:41<00:58,  2.45it/s, failures=0, objective=229]
 52%|█████▏    | 156/300 [00:41<00:58,  2.45it/s, failures=0, objective=230]
 52%|█████▏    | 157/300 [00:41<00:58,  2.45it/s, failures=0, objective=230]
 53%|█████▎    | 158/300 [00:41<00:57,  2.49it/s, failures=0, objective=230]
 53%|█████▎    | 158/300 [00:41<00:57,  2.49it/s, failures=0, objective=231]
 53%|█████▎    | 159/300 [00:41<00:56,  2.49it/s, failures=0, objective=231]
 53%|█████▎    | 160/300 [00:42<01:00,  2.32it/s, failures=0, objective=231]
 53%|█████▎    | 160/300 [00:42<01:00,  2.32it/s, failures=0, objective=232]
 54%|█████▎    | 161/300 [00:42<01:00,  2.32it/s, failures=0, objective=232]/Users/rp5/Documents/DeepHyper/deephyper/src/deephyper/skopt/optimizer/optimizer.py:820: UserWarning: The objective has been evaluated at this point before.
  warnings.warn("The objective has been evaluated at this point before.")

 54%|█████▍    | 162/300 [00:43<01:00,  2.28it/s, failures=0, objective=232]
 54%|█████▍    | 162/300 [00:43<01:00,  2.28it/s, failures=0, objective=232]
 54%|█████▍    | 163/300 [00:43<01:00,  2.28it/s, failures=0, objective=232]
 55%|█████▍    | 164/300 [00:44<00:52,  2.59it/s, failures=0, objective=232]
 55%|█████▍    | 164/300 [00:44<00:52,  2.59it/s, failures=0, objective=232]
 55%|█████▌    | 165/300 [00:44<00:52,  2.59it/s, failures=0, objective=232]
 55%|█████▌    | 166/300 [00:45<00:48,  2.76it/s, failures=0, objective=232]
 55%|█████▌    | 166/300 [00:45<00:48,  2.76it/s, failures=0, objective=232]
 56%|█████▌    | 167/300 [00:45<00:48,  2.76it/s, failures=0, objective=232]
 56%|█████▌    | 168/300 [00:46<00:58,  2.26it/s, failures=0, objective=232]
 56%|█████▌    | 168/300 [00:46<00:58,  2.26it/s, failures=0, objective=232]
 56%|█████▋    | 169/300 [00:46<00:57,  2.26it/s, failures=0, objective=232]
 57%|█████▋    | 170/300 [00:47<00:57,  2.27it/s, failures=0, objective=232]
 57%|█████▋    | 170/300 [00:47<00:57,  2.27it/s, failures=0, objective=232]
 57%|█████▋    | 171/300 [00:47<00:56,  2.27it/s, failures=0, objective=232]
 57%|█████▋    | 172/300 [00:48<01:00,  2.10it/s, failures=0, objective=232]
 57%|█████▋    | 172/300 [00:48<01:00,  2.10it/s, failures=0, objective=232]
 58%|█████▊    | 173/300 [00:48<01:00,  2.10it/s, failures=0, objective=232]
 58%|█████▊    | 174/300 [00:49<01:00,  2.07it/s, failures=0, objective=232]
 58%|█████▊    | 174/300 [00:49<01:00,  2.07it/s, failures=0, objective=232]
 58%|█████▊    | 175/300 [00:49<01:00,  2.07it/s, failures=0, objective=232]
 59%|█████▊    | 176/300 [00:50<00:58,  2.10it/s, failures=0, objective=232]
 59%|█████▊    | 176/300 [00:50<00:58,  2.10it/s, failures=0, objective=235]
 59%|█████▉    | 177/300 [00:50<00:58,  2.10it/s, failures=0, objective=235]
 59%|█████▉    | 178/300 [00:51<00:58,  2.08it/s, failures=0, objective=235]
 59%|█████▉    | 178/300 [00:51<00:58,  2.08it/s, failures=0, objective=239]
 60%|█████▉    | 179/300 [00:51<00:58,  2.08it/s, failures=0, objective=239]/Users/rp5/Documents/DeepHyper/deephyper/src/deephyper/skopt/optimizer/optimizer.py:820: UserWarning: The objective has been evaluated at this point before.
  warnings.warn("The objective has been evaluated at this point before.")

 60%|██████    | 180/300 [00:52<00:58,  2.04it/s, failures=0, objective=239]
 60%|██████    | 180/300 [00:52<00:58,  2.04it/s, failures=0, objective=239]
 60%|██████    | 181/300 [00:52<00:58,  2.04it/s, failures=0, objective=239]
 61%|██████    | 182/300 [00:53<00:56,  2.08it/s, failures=0, objective=239]
 61%|██████    | 182/300 [00:53<00:56,  2.08it/s, failures=0, objective=240]
 61%|██████    | 183/300 [00:53<00:56,  2.08it/s, failures=0, objective=240]
 61%|██████▏   | 184/300 [00:53<00:49,  2.35it/s, failures=0, objective=240]
 61%|██████▏   | 184/300 [00:53<00:49,  2.35it/s, failures=0, objective=240]
 62%|██████▏   | 185/300 [00:53<00:48,  2.35it/s, failures=0, objective=240]
 62%|██████▏   | 186/300 [00:54<00:49,  2.29it/s, failures=0, objective=240]
 62%|██████▏   | 186/300 [00:54<00:49,  2.29it/s, failures=0, objective=240]
 62%|██████▏   | 187/300 [00:54<00:49,  2.29it/s, failures=0, objective=240]
 63%|██████▎   | 188/300 [00:55<00:50,  2.23it/s, failures=0, objective=240]
 63%|██████▎   | 188/300 [00:55<00:50,  2.23it/s, failures=0, objective=240]
 63%|██████▎   | 189/300 [00:55<00:49,  2.23it/s, failures=0, objective=240]
 63%|██████▎   | 190/300 [00:56<00:54,  2.03it/s, failures=0, objective=240]
 63%|██████▎   | 190/300 [00:56<00:54,  2.03it/s, failures=0, objective=240]
 64%|██████▎   | 191/300 [00:56<00:53,  2.03it/s, failures=0, objective=240]
 64%|██████▍   | 192/300 [00:57<00:51,  2.09it/s, failures=0, objective=240]
 64%|██████▍   | 192/300 [00:57<00:51,  2.09it/s, failures=0, objective=240]
 64%|██████▍   | 193/300 [00:57<00:51,  2.09it/s, failures=0, objective=240]
 65%|██████▍   | 194/300 [00:58<00:50,  2.11it/s, failures=0, objective=240]
 65%|██████▍   | 194/300 [00:58<00:50,  2.11it/s, failures=0, objective=240]
 65%|██████▌   | 195/300 [00:58<00:49,  2.11it/s, failures=0, objective=240]
 65%|██████▌   | 196/300 [00:59<00:49,  2.08it/s, failures=0, objective=240]
 65%|██████▌   | 196/300 [00:59<00:49,  2.08it/s, failures=0, objective=241]
 66%|██████▌   | 197/300 [00:59<00:49,  2.08it/s, failures=0, objective=241]/Users/rp5/Documents/DeepHyper/deephyper/src/deephyper/skopt/optimizer/optimizer.py:820: UserWarning: The objective has been evaluated at this point before.
  warnings.warn("The objective has been evaluated at this point before.")

 66%|██████▌   | 198/300 [01:00<00:49,  2.05it/s, failures=0, objective=241]
 66%|██████▌   | 198/300 [01:00<00:49,  2.05it/s, failures=0, objective=241]
 66%|██████▋   | 199/300 [01:00<00:49,  2.05it/s, failures=0, objective=241]
 67%|██████▋   | 200/300 [01:01<00:47,  2.10it/s, failures=0, objective=241]
 67%|██████▋   | 200/300 [01:01<00:47,  2.10it/s, failures=0, objective=242]
 67%|██████▋   | 201/300 [01:01<00:47,  2.10it/s, failures=0, objective=242]/Users/rp5/Documents/DeepHyper/deephyper/src/deephyper/skopt/optimizer/optimizer.py:820: UserWarning: The objective has been evaluated at this point before.
  warnings.warn("The objective has been evaluated at this point before.")

 67%|██████▋   | 202/300 [01:02<00:50,  1.96it/s, failures=0, objective=242]
 67%|██████▋   | 202/300 [01:02<00:50,  1.96it/s, failures=0, objective=242]
 68%|██████▊   | 203/300 [01:02<00:49,  1.96it/s, failures=0, objective=242]
 68%|██████▊   | 204/300 [01:03<00:48,  1.97it/s, failures=0, objective=242]
 68%|██████▊   | 204/300 [01:03<00:48,  1.97it/s, failures=0, objective=242]
 68%|██████▊   | 205/300 [01:03<00:48,  1.97it/s, failures=0, objective=242]
 69%|██████▊   | 206/300 [01:04<00:49,  1.91it/s, failures=0, objective=242]
 69%|██████▊   | 206/300 [01:04<00:49,  1.91it/s, failures=0, objective=242]
 69%|██████▉   | 207/300 [01:04<00:48,  1.91it/s, failures=0, objective=242]
 69%|██████▉   | 208/300 [01:05<00:47,  1.93it/s, failures=0, objective=242]
 69%|██████▉   | 208/300 [01:05<00:47,  1.93it/s, failures=0, objective=242]
 70%|██████▉   | 209/300 [01:05<00:47,  1.93it/s, failures=0, objective=242]
 70%|███████   | 210/300 [01:06<00:44,  2.03it/s, failures=0, objective=242]
 70%|███████   | 210/300 [01:06<00:44,  2.03it/s, failures=0, objective=242]
 70%|███████   | 211/300 [01:06<00:43,  2.03it/s, failures=0, objective=242]
 71%|███████   | 212/300 [01:07<00:44,  2.00it/s, failures=0, objective=242]
 71%|███████   | 212/300 [01:07<00:44,  2.00it/s, failures=0, objective=244]
 71%|███████   | 213/300 [01:07<00:43,  2.00it/s, failures=0, objective=244]
 71%|███████▏  | 214/300 [01:08<00:43,  1.96it/s, failures=0, objective=244]
 71%|███████▏  | 214/300 [01:08<00:43,  1.96it/s, failures=0, objective=244]
 72%|███████▏  | 215/300 [01:08<00:43,  1.96it/s, failures=0, objective=244]
 72%|███████▏  | 216/300 [01:09<00:42,  2.00it/s, failures=0, objective=244]
 72%|███████▏  | 216/300 [01:09<00:42,  2.00it/s, failures=0, objective=244]
 72%|███████▏  | 217/300 [01:09<00:41,  2.00it/s, failures=0, objective=244]/Users/rp5/Documents/DeepHyper/deephyper/src/deephyper/skopt/optimizer/optimizer.py:820: UserWarning: The objective has been evaluated at this point before.
  warnings.warn("The objective has been evaluated at this point before.")

 73%|███████▎  | 218/300 [01:10<00:40,  2.00it/s, failures=0, objective=244]
 73%|███████▎  | 218/300 [01:10<00:40,  2.00it/s, failures=0, objective=244]
 73%|███████▎  | 219/300 [01:10<00:40,  2.00it/s, failures=0, objective=244]
 73%|███████▎  | 220/300 [01:11<00:40,  1.99it/s, failures=0, objective=244]
 73%|███████▎  | 220/300 [01:11<00:40,  1.99it/s, failures=0, objective=245]
 74%|███████▎  | 221/300 [01:11<00:39,  1.99it/s, failures=0, objective=245]/Users/rp5/Documents/DeepHyper/deephyper/src/deephyper/skopt/optimizer/optimizer.py:820: UserWarning: The objective has been evaluated at this point before.
  warnings.warn("The objective has been evaluated at this point before.")

 74%|███████▍  | 222/300 [01:12<00:39,  1.96it/s, failures=0, objective=245]
 74%|███████▍  | 222/300 [01:12<00:39,  1.96it/s, failures=0, objective=245]
 74%|███████▍  | 223/300 [01:12<00:39,  1.96it/s, failures=0, objective=245]
 75%|███████▍  | 224/300 [01:13<00:36,  2.08it/s, failures=0, objective=245]
 75%|███████▍  | 224/300 [01:13<00:36,  2.08it/s, failures=0, objective=245]
 75%|███████▌  | 225/300 [01:13<00:36,  2.08it/s, failures=0, objective=245]
 75%|███████▌  | 226/300 [01:14<00:38,  1.93it/s, failures=0, objective=245]
 75%|███████▌  | 226/300 [01:14<00:38,  1.93it/s, failures=0, objective=245]
 76%|███████▌  | 227/300 [01:14<00:37,  1.93it/s, failures=0, objective=245]
 76%|███████▌  | 228/300 [01:15<00:34,  2.08it/s, failures=0, objective=245]
 76%|███████▌  | 228/300 [01:15<00:34,  2.08it/s, failures=0, objective=245]
 76%|███████▋  | 229/300 [01:15<00:34,  2.08it/s, failures=0, objective=245]
 77%|███████▋  | 230/300 [01:16<00:31,  2.24it/s, failures=0, objective=245]
 77%|███████▋  | 230/300 [01:16<00:31,  2.24it/s, failures=0, objective=245]
 77%|███████▋  | 231/300 [01:16<00:30,  2.24it/s, failures=0, objective=245]
 77%|███████▋  | 232/300 [01:17<00:34,  1.97it/s, failures=0, objective=245]
 77%|███████▋  | 232/300 [01:17<00:34,  1.97it/s, failures=0, objective=245]
 78%|███████▊  | 233/300 [01:17<00:34,  1.97it/s, failures=0, objective=245]
 78%|███████▊  | 234/300 [01:18<00:34,  1.93it/s, failures=0, objective=245]
 78%|███████▊  | 234/300 [01:18<00:34,  1.93it/s, failures=0, objective=245]
 78%|███████▊  | 235/300 [01:18<00:33,  1.93it/s, failures=0, objective=245]
 79%|███████▊  | 236/300 [01:19<00:34,  1.87it/s, failures=0, objective=245]
 79%|███████▊  | 236/300 [01:19<00:34,  1.87it/s, failures=0, objective=245]
 79%|███████▉  | 237/300 [01:19<00:33,  1.87it/s, failures=0, objective=245]/Users/rp5/Documents/DeepHyper/deephyper/src/deephyper/skopt/optimizer/optimizer.py:820: UserWarning: The objective has been evaluated at this point before.
  warnings.warn("The objective has been evaluated at this point before.")

 79%|███████▉  | 238/300 [01:21<00:33,  1.83it/s, failures=0, objective=245]
 79%|███████▉  | 238/300 [01:21<00:33,  1.83it/s, failures=0, objective=245]
 80%|███████▉  | 239/300 [01:21<00:33,  1.83it/s, failures=0, objective=245]/Users/rp5/Documents/DeepHyper/deephyper/src/deephyper/skopt/optimizer/optimizer.py:820: UserWarning: The objective has been evaluated at this point before.
  warnings.warn("The objective has been evaluated at this point before.")

 80%|████████  | 240/300 [01:22<00:32,  1.87it/s, failures=0, objective=245]
 80%|████████  | 240/300 [01:22<00:32,  1.87it/s, failures=0, objective=245]
 80%|████████  | 241/300 [01:22<00:31,  1.87it/s, failures=0, objective=245]/Users/rp5/Documents/DeepHyper/deephyper/src/deephyper/skopt/optimizer/optimizer.py:820: UserWarning: The objective has been evaluated at this point before.
  warnings.warn("The objective has been evaluated at this point before.")

 81%|████████  | 242/300 [01:23<00:30,  1.87it/s, failures=0, objective=245]
 81%|████████  | 242/300 [01:23<00:30,  1.87it/s, failures=0, objective=245]
 81%|████████  | 243/300 [01:23<00:30,  1.87it/s, failures=0, objective=245]
 81%|████████▏ | 244/300 [01:23<00:27,  2.02it/s, failures=0, objective=245]
 81%|████████▏ | 244/300 [01:23<00:27,  2.02it/s, failures=0, objective=245]
 82%|████████▏ | 245/300 [01:23<00:27,  2.02it/s, failures=0, objective=245]
 82%|████████▏ | 246/300 [01:24<00:23,  2.29it/s, failures=0, objective=245]
 82%|████████▏ | 246/300 [01:24<00:23,  2.29it/s, failures=0, objective=245]
 82%|████████▏ | 247/300 [01:24<00:23,  2.29it/s, failures=0, objective=245]
 83%|████████▎ | 248/300 [01:25<00:21,  2.42it/s, failures=0, objective=245]
 83%|████████▎ | 248/300 [01:25<00:21,  2.42it/s, failures=0, objective=245]
 83%|████████▎ | 249/300 [01:25<00:21,  2.42it/s, failures=0, objective=245]
 83%|████████▎ | 250/300 [01:26<00:23,  2.09it/s, failures=0, objective=245]
 83%|████████▎ | 250/300 [01:26<00:23,  2.09it/s, failures=0, objective=245]
 84%|████████▎ | 251/300 [01:26<00:23,  2.09it/s, failures=0, objective=245]
 84%|████████▍ | 252/300 [01:27<00:24,  1.97it/s, failures=0, objective=245]
 84%|████████▍ | 252/300 [01:27<00:24,  1.97it/s, failures=0, objective=245]
 84%|████████▍ | 253/300 [01:27<00:23,  1.97it/s, failures=0, objective=245]
 85%|████████▍ | 254/300 [01:28<00:22,  2.01it/s, failures=0, objective=245]
 85%|████████▍ | 254/300 [01:28<00:22,  2.01it/s, failures=0, objective=245]
 85%|████████▌ | 255/300 [01:28<00:22,  2.01it/s, failures=0, objective=245]
 85%|████████▌ | 256/300 [01:29<00:21,  2.04it/s, failures=0, objective=245]
 85%|████████▌ | 256/300 [01:29<00:21,  2.04it/s, failures=0, objective=246]
 86%|████████▌ | 257/300 [01:29<00:21,  2.04it/s, failures=0, objective=246]/Users/rp5/Documents/DeepHyper/deephyper/src/deephyper/skopt/optimizer/optimizer.py:820: UserWarning: The objective has been evaluated at this point before.
  warnings.warn("The objective has been evaluated at this point before.")

 86%|████████▌ | 258/300 [01:30<00:21,  1.98it/s, failures=0, objective=246]
 86%|████████▌ | 258/300 [01:30<00:21,  1.98it/s, failures=0, objective=246]
 86%|████████▋ | 259/300 [01:30<00:20,  1.98it/s, failures=0, objective=246]/Users/rp5/Documents/DeepHyper/deephyper/src/deephyper/skopt/optimizer/optimizer.py:820: UserWarning: The objective has been evaluated at this point before.
  warnings.warn("The objective has been evaluated at this point before.")

 87%|████████▋ | 260/300 [01:31<00:20,  1.98it/s, failures=0, objective=246]
 87%|████████▋ | 260/300 [01:31<00:20,  1.98it/s, failures=0, objective=246]
 87%|████████▋ | 261/300 [01:31<00:19,  1.98it/s, failures=0, objective=246]/Users/rp5/Documents/DeepHyper/deephyper/src/deephyper/skopt/optimizer/optimizer.py:820: UserWarning: The objective has been evaluated at this point before.
  warnings.warn("The objective has been evaluated at this point before.")

 87%|████████▋ | 262/300 [01:32<00:19,  1.96it/s, failures=0, objective=246]
 87%|████████▋ | 262/300 [01:32<00:19,  1.96it/s, failures=0, objective=246]
 88%|████████▊ | 263/300 [01:32<00:18,  1.96it/s, failures=0, objective=246]
 88%|████████▊ | 264/300 [01:33<00:17,  2.08it/s, failures=0, objective=246]
 88%|████████▊ | 264/300 [01:33<00:17,  2.08it/s, failures=0, objective=246]
 88%|████████▊ | 265/300 [01:33<00:16,  2.08it/s, failures=0, objective=246]
 89%|████████▊ | 266/300 [01:34<00:17,  1.99it/s, failures=0, objective=246]
 89%|████████▊ | 266/300 [01:34<00:17,  1.99it/s, failures=0, objective=246]
 89%|████████▉ | 267/300 [01:34<00:16,  1.99it/s, failures=0, objective=246]
 89%|████████▉ | 268/300 [01:35<00:16,  1.95it/s, failures=0, objective=246]
 89%|████████▉ | 268/300 [01:35<00:16,  1.95it/s, failures=0, objective=246]
 90%|████████▉ | 269/300 [01:35<00:15,  1.95it/s, failures=0, objective=246]
 90%|█████████ | 270/300 [01:36<00:16,  1.80it/s, failures=0, objective=246]
 90%|█████████ | 270/300 [01:36<00:16,  1.80it/s, failures=0, objective=246]
 90%|█████████ | 271/300 [01:37<00:16,  1.80it/s, failures=0, objective=246]
 91%|█████████ | 272/300 [01:38<00:15,  1.82it/s, failures=0, objective=246]
 91%|█████████ | 272/300 [01:38<00:15,  1.82it/s, failures=0, objective=246]
 91%|█████████ | 273/300 [01:38<00:14,  1.82it/s, failures=0, objective=246]
 91%|█████████▏| 274/300 [01:39<00:14,  1.79it/s, failures=0, objective=246]
 91%|█████████▏| 274/300 [01:39<00:14,  1.79it/s, failures=0, objective=247]
 92%|█████████▏| 275/300 [01:39<00:13,  1.79it/s, failures=0, objective=247]
 92%|█████████▏| 276/300 [01:40<00:13,  1.83it/s, failures=0, objective=247]
 92%|█████████▏| 276/300 [01:40<00:13,  1.83it/s, failures=0, objective=248]
 92%|█████████▏| 277/300 [01:40<00:12,  1.83it/s, failures=0, objective=248]/Users/rp5/Documents/DeepHyper/deephyper/src/deephyper/skopt/optimizer/optimizer.py:820: UserWarning: The objective has been evaluated at this point before.
  warnings.warn("The objective has been evaluated at this point before.")

 93%|█████████▎| 278/300 [01:41<00:12,  1.77it/s, failures=0, objective=248]
 93%|█████████▎| 278/300 [01:41<00:12,  1.77it/s, failures=0, objective=248]
 93%|█████████▎| 279/300 [01:41<00:11,  1.77it/s, failures=0, objective=248]/Users/rp5/Documents/DeepHyper/deephyper/src/deephyper/skopt/optimizer/optimizer.py:820: UserWarning: The objective has been evaluated at this point before.
  warnings.warn("The objective has been evaluated at this point before.")

 93%|█████████▎| 280/300 [01:42<00:12,  1.66it/s, failures=0, objective=248]
 93%|█████████▎| 280/300 [01:42<00:12,  1.66it/s, failures=0, objective=248]
 94%|█████████▎| 281/300 [01:42<00:11,  1.66it/s, failures=0, objective=248]/Users/rp5/Documents/DeepHyper/deephyper/src/deephyper/skopt/optimizer/optimizer.py:820: UserWarning: The objective has been evaluated at this point before.
  warnings.warn("The objective has been evaluated at this point before.")

 94%|█████████▍| 282/300 [01:44<00:10,  1.64it/s, failures=0, objective=248]
 94%|█████████▍| 282/300 [01:44<00:10,  1.64it/s, failures=0, objective=248]
 94%|█████████▍| 283/300 [01:44<00:10,  1.64it/s, failures=0, objective=248]
 95%|█████████▍| 284/300 [01:44<00:08,  1.96it/s, failures=0, objective=248]
 95%|█████████▍| 284/300 [01:44<00:08,  1.96it/s, failures=0, objective=248]
 95%|█████████▌| 285/300 [01:44<00:07,  1.96it/s, failures=0, objective=248]
 95%|█████████▌| 286/300 [01:45<00:06,  2.29it/s, failures=0, objective=248]
 95%|█████████▌| 286/300 [01:45<00:06,  2.29it/s, failures=0, objective=248]
 96%|█████████▌| 287/300 [01:45<00:05,  2.29it/s, failures=0, objective=248]
 96%|█████████▌| 288/300 [01:46<00:05,  2.24it/s, failures=0, objective=248]
 96%|█████████▌| 288/300 [01:46<00:05,  2.24it/s, failures=0, objective=248]
 96%|█████████▋| 289/300 [01:46<00:04,  2.24it/s, failures=0, objective=248]
 97%|█████████▋| 290/300 [01:47<00:04,  2.09it/s, failures=0, objective=248]
 97%|█████████▋| 290/300 [01:47<00:04,  2.09it/s, failures=0, objective=248]
 97%|█████████▋| 291/300 [01:47<00:04,  2.09it/s, failures=0, objective=248]
 97%|█████████▋| 292/300 [01:48<00:03,  2.04it/s, failures=0, objective=248]
 97%|█████████▋| 292/300 [01:48<00:03,  2.04it/s, failures=0, objective=248]
 98%|█████████▊| 293/300 [01:48<00:03,  2.04it/s, failures=0, objective=248]
 98%|█████████▊| 294/300 [01:49<00:02,  2.06it/s, failures=0, objective=248]
 98%|█████████▊| 294/300 [01:49<00:02,  2.06it/s, failures=0, objective=248]
 98%|█████████▊| 295/300 [01:49<00:02,  2.06it/s, failures=0, objective=248]
 99%|█████████▊| 296/300 [01:50<00:02,  1.92it/s, failures=0, objective=248]
 99%|█████████▊| 296/300 [01:50<00:02,  1.92it/s, failures=0, objective=252]
 99%|█████████▉| 297/300 [01:50<00:01,  1.92it/s, failures=0, objective=252]
 99%|█████████▉| 298/300 [01:51<00:01,  1.88it/s, failures=0, objective=252]
 99%|█████████▉| 298/300 [01:51<00:01,  1.88it/s, failures=0, objective=253]
100%|█████████▉| 299/300 [01:51<00:00,  1.88it/s, failures=0, objective=253]/Users/rp5/Documents/DeepHyper/deephyper/src/deephyper/skopt/optimizer/optimizer.py:820: UserWarning: The objective has been evaluated at this point before.
  warnings.warn("The objective has been evaluated at this point before.")

100%|██████████| 300/300 [01:52<00:00,  1.75it/s, failures=0, objective=253]
100%|██████████| 300/300 [01:52<00:00,  1.75it/s, failures=0, objective=253]
100%|██████████| 300/300 [01:52<00:00,  2.66it/s, failures=0, objective=253]

results
p:x0 p:x1 p:x2 p:x3 p:x4 p:x5 p:x6 p:x7 p:x8 p:x9 objective job_id job_status m:timestamp_submit m:timestamp_gather
0 6 12 15 17 19 20 24 25 26 27 191.0 0 DONE 0.015556 0.016203
1 1 7 13 16 19 21 23 24 27 30 181.0 1 DONE 0.035443 0.036249
2 10 14 15 17 18 20 24 25 27 28 198.0 2 DONE 0.049745 0.050325
3 11 13 14 16 18 19 24 26 28 29 198.0 3 DONE 0.064089 0.064661
4 1 14 16 18 20 22 23 25 26 27 192.0 4 DONE 0.077750 0.078269
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
295 19 21 23 24 25 26 27 28 29 30 252.0 295 DONE 110.451081 110.451661
296 12 14 16 17 20 22 24 26 28 30 209.0 296 DONE 110.473861 110.474404
297 20 21 23 24 25 26 27 28 29 30 253.0 297 DONE 111.577984 111.578579
298 12 14 16 18 19 22 24 26 27 29 207.0 298 DONE 111.600968 111.601481
299 20 21 23 24 25 26 27 28 29 30 253.0 299 DONE 112.907730 112.908327

300 rows × 15 columns



Extracting the Best Parameters#

To recover the parameters corresponding to the best observed objective value, we can use deephyper.analysis.hpo.parameters_at_max().

parameters, objective = parameters_at_max(results)
print("\nOptimum values")
for i in range(n):
    print(f"x{i}: {parameters[f'x{i}']:.3f}")
print("objective:", objective)

Optimum values
x0: 20.000
x1: 21.000
x2: 23.000
x3: 24.000
x4: 25.000
x5: 26.000
x6: 27.000
x7: 28.000
x8: 29.000
x9: 30.000
objective: 253.0

Visualization#

We conclude with:

  • a search trajectomakery plot showing the best objective value over time, where the periodic exploration schedule is clearly visible;

  • a feasible-space evaluation plot showing all sampled curves \(i \mapsto x_i\) (each curve is one evaluation), colored by objective value.

These visualizations confirm that the optimizer progressively learns the structure of the monotonic constraint and approaches the theoretical optimum.

WIDTH_PLOTS = 8
HEIGHT_PLOTS = WIDTH_PLOTS / 1.618

fig, ax = plt.subplots(figsize=(WIDTH_PLOTS, HEIGHT_PLOTS))
plot_search_trajectory_single_objective_hpo(results, mode="max", ax=ax)
_ = plt.title("Search Trajectory")
Search Trajectory

Visualizing the Feasible Region and Evaluations#

We now plot all evaluated points in the (x, y) plane, color-coded by objective value, along with the constraint boundary x + y = 10.

results, _ = filter_failed_objectives(results)

p_columns = [col for col in results.columns if col.startswith("p:")]

# Create a normalizer over the objective range
obj_vals = results["objective"]
norm = colors.Normalize(vmin=obj_vals.min(), vmax=obj_vals.max())

# Choose a colormap (viridis is a good default)
cmap = plt.get_cmap("viridis")

fig, ax = plt.subplots(figsize=(WIDTH_PLOTS, HEIGHT_PLOTS))

for i, row in results.iterrows():
    x_values = row[p_columns].values
    y_values = np.arange(n)
    obj_value = row["objective"]

    color = cmap(norm(obj_value))  # map objective → color
    ax.plot(x_values, y_values, color=color, alpha=0.9)

# Optionally add a colorbar
sm = cm.ScalarMappable(norm=norm, cmap=cmap)
cbar = fig.colorbar(sm, ax=ax)
cbar.set_label("Objective value")
ax.grid()
ax.set_ylim(0, n - 1)
ax.set_xlim(0, m)
ax.set_ylabel(r"$i$")
ax.set_xlabel(r"$x_i$")
ax.set_yticks(list(range(n)), [str(i) for i in range(n)])
ax.set_xticks(list(range(0, m, 2)), [str(i) for i in range(0, m, 2)])
plot constrained black box optimization chained sampler
[<matplotlib.axis.XTick object at 0x12c978a50>, <matplotlib.axis.XTick object at 0x12c9782d0>, <matplotlib.axis.XTick object at 0x14a26ec10>, <matplotlib.axis.XTick object at 0x14a26efd0>, <matplotlib.axis.XTick object at 0x14a26f390>, <matplotlib.axis.XTick object at 0x14a26f750>, <matplotlib.axis.XTick object at 0x14a26fb10>, <matplotlib.axis.XTick object at 0x14a26fed0>, <matplotlib.axis.XTick object at 0x14a2bc2d0>, <matplotlib.axis.XTick object at 0x14a2bc690>, <matplotlib.axis.XTick object at 0x14a2bca50>, <matplotlib.axis.XTick object at 0x14a2bce10>, <matplotlib.axis.XTick object at 0x14a2bd1d0>, <matplotlib.axis.XTick object at 0x14a2bd590>, <matplotlib.axis.XTick object at 0x14a2bd950>, <matplotlib.axis.XTick object at 0x14a2bdd10>]

Total running time of the script: (1 minutes 54.009 seconds)

Gallery generated by Sphinx-Gallery

Contents