Mutli-Objective Black-Box Optimization

Note

Go to the end to download the full example code.

Mutli-Objective Black-Box Optimization#

In this tutorial, we will explore how to run black-box multi-objective optimization (MOO). In this setting, the goal is to resolve the following problem:

\[\text{max}_x (f_0(x), f_1(x), ..., f_n(x))\]

where \(x\) is the set of optimized variables and \(f_i\) are the different objectives. In DeepHyper, we use scalarization to transform such multi-objective problem into a single-objective problem:

\[\text{max}_x s_w((f_0(x), f_1(x), ..., f_n(x)))\]

where \(w\) is a set of weights which manages the trade-off between objectives and \(s_w : \mathbb{R}^n \rightarrow \mathbb{R}\). The weight vector \(w\) is randomized and re-sampled for each new batch of suggestion from the optimizer.

We will look at the DTLZ benchmark suite, a classic in multi-objective optimization (MOO) litterature. This benchmark exibit some characteristic cases of MOO. By default, this tutorial is loading the DTLZ-II benchmark which exibit a Pareto-Front with a concave shape.

Installation and imports#

Installing dependencies with the pip installation is recommended. It requires Python >= 3.10.

%%bash
pip install deephyper
pip install "git+https://github.com/deephyper/benchmark.git@main#egg=deephyper-benchmark"
Code (Import statements)
import matplotlib.pyplot as plt

from deephyper.hpo import CBO
from deephyper_benchmark.benchmarks.dtlz import DTLZBenchmark

WIDTH_PLOTS = 8
HEIGHT_PLOTS = WIDTH_PLOTS / 1.618

n_objectives = 2
bench = DTLZBenchmark(nobj=n_objectives)

We can display the variable search space of the benchmark we just loaded:

bench.problem

Configuration space object:
  Hyperparameters:
    x0, Type: UniformFloat, Range: [0.0, 1.0], Default: 0.5
    x1, Type: UniformFloat, Range: [0.0, 1.0], Default: 0.5
    x2, Type: UniformFloat, Range: [0.0, 1.0], Default: 0.5
    x3, Type: UniformFloat, Range: [0.0, 1.0], Default: 0.5
    x4, Type: UniformFloat, Range: [0.0, 1.0], Default: 0.5

To define a black-box for multi-objective optimization it is very similar to single-objective optimization at the difference that the objective can now be a list of values. A first possibility is:

def run(job):
    ...
    return objective_0, objective_1, ..., objective_n

which just returns the objectives to optimize as a tuple. If additionnal metadata are interesting to gather for each evaluation it is also possible to return them by following this format:

def run(job):
    ...
    return {
        "objective": [objective_0, objective_1, ..., objective_n],
        "metadata": {
            "flops": ...,
            "memory_footprint": ...,
            "duration": ...,
         }
     }

each of the metadata needs to be JSON serializable and will be returned in the final results with a column name formatted as m:metadata_key such as m:duration.

For the search algorithm, we use the centralized Bayesian Optimization search (CBO). Search algorithm

The arguments specific to multi-objective optimization are:

  • moo_scalarization_strategy is used to specify the scalarization strategy. Chebyshev scalarizationis capable of generating a diverse set of solutions for non-convex problems.

  • moo_scalarization_weight argument is used to specify the weight of objectives in the scalarization. "random" is used to generate a random weight vector at each iteration.

search = CBO(
    bench.problem,
    acq_optimizer="sampling",
    moo_scalarization_strategy="AugChebyshev",
    moo_scalarization_weight="random",
    verbose=1,
)

Results file already exists, it will be renamed to /Users/rp5/Documents/DeepHyper/deephyper/examples/examples_bbo/results_20250818-134730.csv

Launch the search for a given number of evaluations other stopping criteria can be used (e.g. timeout, early-stopping/convergence)

results = search.search(bench.run_function, max_evals=500)

  0%|          | 0/500 [00:00<?, ?it/s]
  0%|          | 1/500 [00:00<00:00, 3731.59it/s, failures=0, hvi=0]
  0%|          | 2/500 [00:00<00:04, 113.17it/s, failures=0, hvi=0]
  1%|          | 3/500 [00:00<00:05, 97.61it/s, failures=0, hvi=0.433]
  1%|          | 4/500 [00:00<00:05, 92.30it/s, failures=0, hvi=0.433]
  1%|          | 5/500 [00:00<00:05, 90.20it/s, failures=0, hvi=0.504]
  1%|          | 6/500 [00:00<00:05, 88.47it/s, failures=0, hvi=0.628]
  1%|▏         | 7/500 [00:00<00:05, 87.65it/s, failures=0, hvi=0.732]
  2%|▏         | 8/500 [00:00<00:05, 86.98it/s, failures=0, hvi=0.756]
  2%|▏         | 9/500 [00:00<00:05, 86.61it/s, failures=0, hvi=0.756]
  2%|▏         | 9/500 [00:00<00:05, 86.61it/s, failures=0, hvi=0.756]
  2%|▏         | 10/500 [00:00<00:05, 86.61it/s, failures=0, hvi=0.934]
  2%|▏         | 11/500 [00:00<00:05, 86.61it/s, failures=0, hvi=0.942]
  2%|▏         | 12/500 [00:00<00:05, 86.61it/s, failures=0, hvi=0.947]
  3%|▎         | 13/500 [00:00<00:05, 86.61it/s, failures=0, hvi=0.947]
  3%|▎         | 14/500 [00:00<00:05, 86.61it/s, failures=0, hvi=1.02]
  3%|▎         | 15/500 [00:00<00:05, 86.61it/s, failures=0, hvi=1.07]
  3%|▎         | 16/500 [00:00<00:05, 86.61it/s, failures=0, hvi=1.09]
  3%|▎         | 17/500 [00:00<00:05, 86.61it/s, failures=0, hvi=1.1]
  4%|▎         | 18/500 [00:00<00:07, 66.09it/s, failures=0, hvi=1.1]
  4%|▎         | 18/500 [00:00<00:07, 66.09it/s, failures=0, hvi=1.11]
  4%|▍         | 19/500 [00:00<00:07, 66.09it/s, failures=0, hvi=1.11]
  4%|▍         | 20/500 [00:00<00:07, 66.09it/s, failures=0, hvi=1.12]
  4%|▍         | 21/500 [00:00<00:07, 66.09it/s, failures=0, hvi=1.15]
  4%|▍         | 22/500 [00:00<00:07, 66.09it/s, failures=0, hvi=1.15]
  5%|▍         | 23/500 [00:00<00:07, 66.09it/s, failures=0, hvi=1.15]
  5%|▍         | 24/500 [00:00<00:07, 66.09it/s, failures=0, hvi=1.15]
  5%|▌         | 25/500 [00:00<00:08, 59.37it/s, failures=0, hvi=1.15]
  5%|▌         | 25/500 [00:00<00:08, 59.37it/s, failures=0, hvi=1.15]
  5%|▌         | 26/500 [00:00<00:07, 59.37it/s, failures=0, hvi=1.25]
  5%|▌         | 27/500 [00:00<00:07, 59.37it/s, failures=0, hvi=1.27]
  6%|▌         | 28/500 [00:00<00:07, 59.37it/s, failures=0, hvi=1.27]
  6%|▌         | 29/500 [00:00<00:07, 59.37it/s, failures=0, hvi=1.27]
  6%|▌         | 30/500 [00:00<00:07, 59.37it/s, failures=0, hvi=1.27]
  6%|▌         | 31/500 [00:00<00:07, 59.37it/s, failures=0, hvi=1.28]
  6%|▋         | 32/500 [00:00<00:08, 55.35it/s, failures=0, hvi=1.28]
  6%|▋         | 32/500 [00:00<00:08, 55.35it/s, failures=0, hvi=1.32]
  7%|▋         | 33/500 [00:00<00:08, 55.35it/s, failures=0, hvi=1.35]
  7%|▋         | 34/500 [00:00<00:08, 55.35it/s, failures=0, hvi=1.35]
  7%|▋         | 35/500 [00:00<00:08, 55.35it/s, failures=0, hvi=1.35]
  7%|▋         | 36/500 [00:00<00:08, 55.35it/s, failures=0, hvi=1.35]
  7%|▋         | 37/500 [00:00<00:08, 55.35it/s, failures=0, hvi=1.35]
  8%|▊         | 38/500 [00:00<00:08, 53.18it/s, failures=0, hvi=1.35]
  8%|▊         | 38/500 [00:00<00:08, 53.18it/s, failures=0, hvi=1.36]
  8%|▊         | 39/500 [00:00<00:08, 53.18it/s, failures=0, hvi=1.36]
  8%|▊         | 40/500 [00:00<00:08, 53.18it/s, failures=0, hvi=1.36]
  8%|▊         | 41/500 [00:00<00:08, 53.18it/s, failures=0, hvi=1.36]
  8%|▊         | 42/500 [00:00<00:08, 53.18it/s, failures=0, hvi=1.36]
  9%|▊         | 43/500 [00:00<00:08, 53.18it/s, failures=0, hvi=1.36]
  9%|▉         | 44/500 [00:00<00:08, 51.01it/s, failures=0, hvi=1.36]
  9%|▉         | 44/500 [00:00<00:08, 51.01it/s, failures=0, hvi=1.36]
  9%|▉         | 45/500 [00:00<00:08, 51.01it/s, failures=0, hvi=1.36]
  9%|▉         | 46/500 [00:00<00:08, 51.01it/s, failures=0, hvi=1.36]
  9%|▉         | 47/500 [00:00<00:08, 51.01it/s, failures=0, hvi=1.36]
 10%|▉         | 48/500 [00:00<00:08, 51.01it/s, failures=0, hvi=1.36]
 10%|▉         | 49/500 [00:00<00:08, 51.01it/s, failures=0, hvi=1.36]
 10%|█         | 50/500 [00:00<00:10, 44.94it/s, failures=0, hvi=1.36]
 10%|█         | 50/500 [00:00<00:10, 44.94it/s, failures=0, hvi=1.39]
 10%|█         | 51/500 [00:00<00:09, 44.94it/s, failures=0, hvi=1.45]
 10%|█         | 52/500 [00:01<00:09, 44.94it/s, failures=0, hvi=1.45]
 11%|█         | 53/500 [00:01<00:09, 44.94it/s, failures=0, hvi=1.45]
 11%|█         | 54/500 [00:01<00:09, 44.94it/s, failures=0, hvi=1.45]
 11%|█         | 55/500 [00:01<00:09, 44.88it/s, failures=0, hvi=1.45]
 11%|█         | 55/500 [00:01<00:09, 44.88it/s, failures=0, hvi=1.45]
 11%|█         | 56/500 [00:01<00:09, 44.88it/s, failures=0, hvi=1.46]
 11%|█▏        | 57/500 [00:01<00:09, 44.88it/s, failures=0, hvi=1.46]
 12%|█▏        | 58/500 [00:01<00:09, 44.88it/s, failures=0, hvi=1.46]
 12%|█▏        | 59/500 [00:01<00:09, 44.88it/s, failures=0, hvi=1.47]
 12%|█▏        | 60/500 [00:01<00:09, 44.45it/s, failures=0, hvi=1.47]
 12%|█▏        | 60/500 [00:01<00:09, 44.45it/s, failures=0, hvi=1.47]
 12%|█▏        | 61/500 [00:01<00:09, 44.45it/s, failures=0, hvi=1.47]
 12%|█▏        | 62/500 [00:01<00:09, 44.45it/s, failures=0, hvi=1.47]
 13%|█▎        | 63/500 [00:01<00:09, 44.45it/s, failures=0, hvi=1.47]
 13%|█▎        | 64/500 [00:01<00:09, 44.45it/s, failures=0, hvi=1.47]
 13%|█▎        | 65/500 [00:01<00:09, 43.78it/s, failures=0, hvi=1.47]
 13%|█▎        | 65/500 [00:01<00:09, 43.78it/s, failures=0, hvi=1.47]
 13%|█▎        | 66/500 [00:01<00:09, 43.78it/s, failures=0, hvi=1.49]
 13%|█▎        | 67/500 [00:01<00:09, 43.78it/s, failures=0, hvi=1.49]
 14%|█▎        | 68/500 [00:01<00:09, 43.78it/s, failures=0, hvi=1.49]
 14%|█▍        | 69/500 [00:01<00:09, 43.78it/s, failures=0, hvi=1.49]
 14%|█▍        | 70/500 [00:01<00:09, 43.28it/s, failures=0, hvi=1.49]
 14%|█▍        | 70/500 [00:01<00:09, 43.28it/s, failures=0, hvi=1.49]
 14%|█▍        | 71/500 [00:01<00:09, 43.28it/s, failures=0, hvi=1.5]
 14%|█▍        | 72/500 [00:01<00:09, 43.28it/s, failures=0, hvi=1.5]
 15%|█▍        | 73/500 [00:01<00:09, 43.28it/s, failures=0, hvi=1.5]
 15%|█▍        | 74/500 [00:01<00:09, 43.28it/s, failures=0, hvi=1.5]
 15%|█▌        | 75/500 [00:01<00:09, 42.86it/s, failures=0, hvi=1.5]
 15%|█▌        | 75/500 [00:01<00:09, 42.86it/s, failures=0, hvi=1.5]
 15%|█▌        | 76/500 [00:01<00:09, 42.86it/s, failures=0, hvi=1.5]
 15%|█▌        | 77/500 [00:01<00:09, 42.86it/s, failures=0, hvi=1.51]
 16%|█▌        | 78/500 [00:01<00:09, 42.86it/s, failures=0, hvi=1.51]
 16%|█▌        | 79/500 [00:01<00:09, 42.86it/s, failures=0, hvi=1.51]
 16%|█▌        | 80/500 [00:01<00:09, 42.35it/s, failures=0, hvi=1.51]
 16%|█▌        | 80/500 [00:01<00:09, 42.35it/s, failures=0, hvi=1.51]
 16%|█▌        | 81/500 [00:01<00:09, 42.35it/s, failures=0, hvi=1.51]
 16%|█▋        | 82/500 [00:01<00:09, 42.35it/s, failures=0, hvi=1.51]
 17%|█▋        | 83/500 [00:01<00:09, 42.35it/s, failures=0, hvi=1.51]
 17%|█▋        | 84/500 [00:01<00:09, 42.35it/s, failures=0, hvi=1.51]
 17%|█▋        | 85/500 [00:01<00:09, 42.10it/s, failures=0, hvi=1.51]
 17%|█▋        | 85/500 [00:01<00:09, 42.10it/s, failures=0, hvi=1.51]
 17%|█▋        | 86/500 [00:01<00:09, 42.10it/s, failures=0, hvi=1.51]
 17%|█▋        | 87/500 [00:01<00:09, 42.10it/s, failures=0, hvi=1.51]
 18%|█▊        | 88/500 [00:01<00:09, 42.10it/s, failures=0, hvi=1.51]
 18%|█▊        | 89/500 [00:01<00:09, 42.10it/s, failures=0, hvi=1.53]
 18%|█▊        | 90/500 [00:01<00:09, 41.66it/s, failures=0, hvi=1.53]
 18%|█▊        | 90/500 [00:01<00:09, 41.66it/s, failures=0, hvi=1.53]
 18%|█▊        | 91/500 [00:01<00:09, 41.66it/s, failures=0, hvi=1.53]
 18%|█▊        | 92/500 [00:01<00:09, 41.66it/s, failures=0, hvi=1.53]
 19%|█▊        | 93/500 [00:01<00:09, 41.66it/s, failures=0, hvi=1.53]
 19%|█▉        | 94/500 [00:02<00:09, 41.66it/s, failures=0, hvi=1.53]
 19%|█▉        | 95/500 [00:02<00:09, 41.19it/s, failures=0, hvi=1.53]
 19%|█▉        | 95/500 [00:02<00:09, 41.19it/s, failures=0, hvi=1.53]
 19%|█▉        | 96/500 [00:02<00:09, 41.19it/s, failures=0, hvi=1.53]
 19%|█▉        | 97/500 [00:02<00:09, 41.19it/s, failures=0, hvi=1.54]
 20%|█▉        | 98/500 [00:02<00:09, 41.19it/s, failures=0, hvi=1.54]
 20%|█▉        | 99/500 [00:02<00:09, 41.19it/s, failures=0, hvi=1.54]
 20%|██        | 100/500 [00:02<00:09, 40.67it/s, failures=0, hvi=1.54]
 20%|██        | 100/500 [00:02<00:09, 40.67it/s, failures=0, hvi=1.54]
 20%|██        | 101/500 [00:02<00:09, 40.67it/s, failures=0, hvi=1.54]
 20%|██        | 102/500 [00:02<00:09, 40.67it/s, failures=0, hvi=1.54]
 21%|██        | 103/500 [00:02<00:09, 40.67it/s, failures=0, hvi=1.54]
 21%|██        | 104/500 [00:02<00:09, 40.67it/s, failures=0, hvi=1.54]
 21%|██        | 105/500 [00:02<00:09, 40.29it/s, failures=0, hvi=1.54]
 21%|██        | 105/500 [00:02<00:09, 40.29it/s, failures=0, hvi=1.54]
 21%|██        | 106/500 [00:02<00:09, 40.29it/s, failures=0, hvi=1.55]
 21%|██▏       | 107/500 [00:02<00:09, 40.29it/s, failures=0, hvi=1.55]
 22%|██▏       | 108/500 [00:02<00:09, 40.29it/s, failures=0, hvi=1.55]
 22%|██▏       | 109/500 [00:02<00:09, 40.29it/s, failures=0, hvi=1.55]
 22%|██▏       | 110/500 [00:02<00:09, 39.86it/s, failures=0, hvi=1.55]
 22%|██▏       | 110/500 [00:02<00:09, 39.86it/s, failures=0, hvi=1.55]
 22%|██▏       | 111/500 [00:02<00:09, 39.86it/s, failures=0, hvi=1.56]
 22%|██▏       | 112/500 [00:02<00:09, 39.86it/s, failures=0, hvi=1.56]
 23%|██▎       | 113/500 [00:02<00:09, 39.86it/s, failures=0, hvi=1.56]
 23%|██▎       | 114/500 [00:02<00:09, 39.58it/s, failures=0, hvi=1.56]
 23%|██▎       | 114/500 [00:02<00:09, 39.58it/s, failures=0, hvi=1.56]
 23%|██▎       | 115/500 [00:02<00:09, 39.58it/s, failures=0, hvi=1.56]
 23%|██▎       | 116/500 [00:02<00:09, 39.58it/s, failures=0, hvi=1.56]
 23%|██▎       | 117/500 [00:02<00:09, 39.58it/s, failures=0, hvi=1.56]
 24%|██▎       | 118/500 [00:02<00:10, 35.97it/s, failures=0, hvi=1.56]
 24%|██▎       | 118/500 [00:02<00:10, 35.97it/s, failures=0, hvi=1.56]
 24%|██▍       | 119/500 [00:02<00:10, 35.97it/s, failures=0, hvi=1.56]
 24%|██▍       | 120/500 [00:02<00:10, 35.97it/s, failures=0, hvi=1.56]
 24%|██▍       | 121/500 [00:02<00:10, 35.97it/s, failures=0, hvi=1.56]
 24%|██▍       | 122/500 [00:02<00:10, 36.67it/s, failures=0, hvi=1.56]
 24%|██▍       | 122/500 [00:02<00:10, 36.67it/s, failures=0, hvi=1.58]
 25%|██▍       | 123/500 [00:02<00:10, 36.67it/s, failures=0, hvi=1.58]
 25%|██▍       | 124/500 [00:02<00:10, 36.67it/s, failures=0, hvi=1.59]
 25%|██▌       | 125/500 [00:02<00:10, 36.67it/s, failures=0, hvi=1.6]
 25%|██▌       | 126/500 [00:02<00:10, 37.12it/s, failures=0, hvi=1.6]
 25%|██▌       | 126/500 [00:02<00:10, 37.12it/s, failures=0, hvi=1.6]
 25%|██▌       | 127/500 [00:02<00:10, 37.12it/s, failures=0, hvi=1.6]
 26%|██▌       | 128/500 [00:02<00:10, 37.12it/s, failures=0, hvi=1.6]
 26%|██▌       | 129/500 [00:02<00:09, 37.12it/s, failures=0, hvi=1.6]
 26%|██▌       | 130/500 [00:02<00:09, 37.41it/s, failures=0, hvi=1.6]
 26%|██▌       | 130/500 [00:02<00:09, 37.41it/s, failures=0, hvi=1.6]
 26%|██▌       | 131/500 [00:02<00:09, 37.41it/s, failures=0, hvi=1.61]
 26%|██▋       | 132/500 [00:03<00:09, 37.41it/s, failures=0, hvi=1.61]
 27%|██▋       | 133/500 [00:03<00:09, 37.41it/s, failures=0, hvi=1.61]
 27%|██▋       | 134/500 [00:03<00:09, 37.57it/s, failures=0, hvi=1.61]
 27%|██▋       | 134/500 [00:03<00:09, 37.57it/s, failures=0, hvi=1.61]
 27%|██▋       | 135/500 [00:03<00:09, 37.57it/s, failures=0, hvi=1.61]
 27%|██▋       | 136/500 [00:03<00:09, 37.57it/s, failures=0, hvi=1.61]
 27%|██▋       | 137/500 [00:03<00:09, 37.57it/s, failures=0, hvi=1.61]
 28%|██▊       | 138/500 [00:03<00:09, 37.60it/s, failures=0, hvi=1.61]
 28%|██▊       | 138/500 [00:03<00:09, 37.60it/s, failures=0, hvi=1.61]
 28%|██▊       | 139/500 [00:03<00:09, 37.60it/s, failures=0, hvi=1.61]
 28%|██▊       | 140/500 [00:03<00:09, 37.60it/s, failures=0, hvi=1.62]
 28%|██▊       | 141/500 [00:03<00:09, 37.60it/s, failures=0, hvi=1.62]
 28%|██▊       | 142/500 [00:03<00:09, 37.60it/s, failures=0, hvi=1.62]
 28%|██▊       | 142/500 [00:03<00:09, 37.60it/s, failures=0, hvi=1.62]
 29%|██▊       | 143/500 [00:03<00:09, 37.60it/s, failures=0, hvi=1.62]
 29%|██▉       | 144/500 [00:03<00:09, 37.60it/s, failures=0, hvi=1.62]
 29%|██▉       | 145/500 [00:03<00:09, 37.60it/s, failures=0, hvi=1.62]
 29%|██▉       | 146/500 [00:03<00:09, 37.58it/s, failures=0, hvi=1.62]
 29%|██▉       | 146/500 [00:03<00:09, 37.58it/s, failures=0, hvi=1.62]
 29%|██▉       | 147/500 [00:03<00:09, 37.58it/s, failures=0, hvi=1.62]
 30%|██▉       | 148/500 [00:03<00:09, 37.58it/s, failures=0, hvi=1.62]
 30%|██▉       | 149/500 [00:03<00:09, 37.58it/s, failures=0, hvi=1.62]
 30%|███       | 150/500 [00:03<00:09, 37.31it/s, failures=0, hvi=1.62]
 30%|███       | 150/500 [00:03<00:09, 37.31it/s, failures=0, hvi=1.62]
 30%|███       | 151/500 [00:03<00:09, 37.31it/s, failures=0, hvi=1.62]
 30%|███       | 152/500 [00:03<00:09, 37.31it/s, failures=0, hvi=1.62]
 31%|███       | 153/500 [00:03<00:09, 37.31it/s, failures=0, hvi=1.62]
 31%|███       | 154/500 [00:03<00:09, 37.16it/s, failures=0, hvi=1.62]
 31%|███       | 154/500 [00:03<00:09, 37.16it/s, failures=0, hvi=1.62]
 31%|███       | 155/500 [00:03<00:09, 37.16it/s, failures=0, hvi=1.62]
 31%|███       | 156/500 [00:03<00:09, 37.16it/s, failures=0, hvi=1.62]
 31%|███▏      | 157/500 [00:03<00:09, 37.16it/s, failures=0, hvi=1.63]
 32%|███▏      | 158/500 [00:03<00:09, 36.94it/s, failures=0, hvi=1.63]
 32%|███▏      | 158/500 [00:03<00:09, 36.94it/s, failures=0, hvi=1.63]
 32%|███▏      | 159/500 [00:03<00:09, 36.94it/s, failures=0, hvi=1.63]
 32%|███▏      | 160/500 [00:03<00:09, 36.94it/s, failures=0, hvi=1.63]
 32%|███▏      | 161/500 [00:03<00:09, 36.94it/s, failures=0, hvi=1.63]
 32%|███▏      | 162/500 [00:03<00:09, 36.85it/s, failures=0, hvi=1.63]
 32%|███▏      | 162/500 [00:03<00:09, 36.85it/s, failures=0, hvi=1.63]
 33%|███▎      | 163/500 [00:03<00:09, 36.85it/s, failures=0, hvi=1.69]
 33%|███▎      | 164/500 [00:03<00:09, 36.85it/s, failures=0, hvi=1.69]
 33%|███▎      | 165/500 [00:03<00:09, 36.85it/s, failures=0, hvi=1.76]
 33%|███▎      | 166/500 [00:03<00:09, 36.71it/s, failures=0, hvi=1.76]
 33%|███▎      | 166/500 [00:03<00:09, 36.71it/s, failures=0, hvi=1.76]
 33%|███▎      | 167/500 [00:03<00:09, 36.71it/s, failures=0, hvi=1.76]
 34%|███▎      | 168/500 [00:03<00:09, 36.71it/s, failures=0, hvi=1.76]
 34%|███▍      | 169/500 [00:04<00:09, 36.71it/s, failures=0, hvi=1.76]
 34%|███▍      | 170/500 [00:04<00:09, 36.61it/s, failures=0, hvi=1.76]
 34%|███▍      | 170/500 [00:04<00:09, 36.61it/s, failures=0, hvi=1.77]
 34%|███▍      | 171/500 [00:04<00:08, 36.61it/s, failures=0, hvi=1.77]
 34%|███▍      | 172/500 [00:04<00:08, 36.61it/s, failures=0, hvi=1.77]
 35%|███▍      | 173/500 [00:04<00:08, 36.61it/s, failures=0, hvi=1.77]
 35%|███▍      | 174/500 [00:04<00:08, 36.45it/s, failures=0, hvi=1.77]
 35%|███▍      | 174/500 [00:04<00:08, 36.45it/s, failures=0, hvi=1.77]
 35%|███▌      | 175/500 [00:04<00:08, 36.45it/s, failures=0, hvi=1.77]
 35%|███▌      | 176/500 [00:04<00:08, 36.45it/s, failures=0, hvi=1.77]
 35%|███▌      | 177/500 [00:04<00:08, 36.45it/s, failures=0, hvi=1.77]
 36%|███▌      | 178/500 [00:04<00:08, 36.28it/s, failures=0, hvi=1.77]
 36%|███▌      | 178/500 [00:04<00:08, 36.28it/s, failures=0, hvi=1.78]
 36%|███▌      | 179/500 [00:04<00:08, 36.28it/s, failures=0, hvi=1.78]
 36%|███▌      | 180/500 [00:04<00:08, 36.28it/s, failures=0, hvi=1.78]
 36%|███▌      | 181/500 [00:04<00:08, 36.28it/s, failures=0, hvi=1.78]
 36%|███▋      | 182/500 [00:04<00:09, 33.05it/s, failures=0, hvi=1.78]
 36%|███▋      | 182/500 [00:04<00:09, 33.05it/s, failures=0, hvi=1.78]
 37%|███▋      | 183/500 [00:04<00:09, 33.05it/s, failures=0, hvi=1.78]
 37%|███▋      | 184/500 [00:04<00:09, 33.05it/s, failures=0, hvi=1.78]
 37%|███▋      | 185/500 [00:04<00:09, 33.05it/s, failures=0, hvi=1.78]
 37%|███▋      | 186/500 [00:04<00:09, 33.83it/s, failures=0, hvi=1.78]
 37%|███▋      | 186/500 [00:04<00:09, 33.83it/s, failures=0, hvi=1.78]
 37%|███▋      | 187/500 [00:04<00:09, 33.83it/s, failures=0, hvi=1.78]
 38%|███▊      | 188/500 [00:04<00:09, 33.83it/s, failures=0, hvi=1.78]
 38%|███▊      | 189/500 [00:04<00:09, 33.83it/s, failures=0, hvi=1.78]
 38%|███▊      | 190/500 [00:04<00:09, 34.37it/s, failures=0, hvi=1.78]
 38%|███▊      | 190/500 [00:04<00:09, 34.37it/s, failures=0, hvi=1.78]
 38%|███▊      | 191/500 [00:04<00:08, 34.37it/s, failures=0, hvi=1.78]
 38%|███▊      | 192/500 [00:04<00:08, 34.37it/s, failures=0, hvi=1.78]
 39%|███▊      | 193/500 [00:04<00:08, 34.37it/s, failures=0, hvi=1.78]
 39%|███▉      | 194/500 [00:04<00:08, 34.65it/s, failures=0, hvi=1.78]
 39%|███▉      | 194/500 [00:04<00:08, 34.65it/s, failures=0, hvi=1.78]
 39%|███▉      | 195/500 [00:04<00:08, 34.65it/s, failures=0, hvi=1.78]
 39%|███▉      | 196/500 [00:04<00:08, 34.65it/s, failures=0, hvi=1.78]
 39%|███▉      | 197/500 [00:04<00:08, 34.65it/s, failures=0, hvi=1.78]
 40%|███▉      | 198/500 [00:04<00:08, 34.79it/s, failures=0, hvi=1.78]
 40%|███▉      | 198/500 [00:04<00:08, 34.79it/s, failures=0, hvi=1.78]
 40%|███▉      | 199/500 [00:04<00:08, 34.79it/s, failures=0, hvi=1.78]
 40%|████      | 200/500 [00:04<00:08, 34.79it/s, failures=0, hvi=1.78]
 40%|████      | 201/500 [00:04<00:08, 34.79it/s, failures=0, hvi=1.79]
 40%|████      | 202/500 [00:04<00:08, 34.85it/s, failures=0, hvi=1.79]
 40%|████      | 202/500 [00:04<00:08, 34.85it/s, failures=0, hvi=1.79]
 41%|████      | 203/500 [00:05<00:08, 34.85it/s, failures=0, hvi=1.79]
 41%|████      | 204/500 [00:05<00:08, 34.85it/s, failures=0, hvi=1.79]
 41%|████      | 205/500 [00:05<00:08, 34.85it/s, failures=0, hvi=1.79]
 41%|████      | 206/500 [00:05<00:08, 34.82it/s, failures=0, hvi=1.79]
 41%|████      | 206/500 [00:05<00:08, 34.82it/s, failures=0, hvi=1.79]
 41%|████▏     | 207/500 [00:05<00:08, 34.82it/s, failures=0, hvi=1.79]
 42%|████▏     | 208/500 [00:05<00:08, 34.82it/s, failures=0, hvi=1.79]
 42%|████▏     | 209/500 [00:05<00:08, 34.82it/s, failures=0, hvi=1.79]
 42%|████▏     | 210/500 [00:05<00:08, 34.79it/s, failures=0, hvi=1.79]
 42%|████▏     | 210/500 [00:05<00:08, 34.79it/s, failures=0, hvi=1.79]
 42%|████▏     | 211/500 [00:05<00:08, 34.79it/s, failures=0, hvi=1.79]
 42%|████▏     | 212/500 [00:05<00:08, 34.79it/s, failures=0, hvi=1.79]
 43%|████▎     | 213/500 [00:05<00:08, 34.79it/s, failures=0, hvi=1.79]
 43%|████▎     | 214/500 [00:05<00:08, 34.76it/s, failures=0, hvi=1.79]
 43%|████▎     | 214/500 [00:05<00:08, 34.76it/s, failures=0, hvi=1.79]
 43%|████▎     | 215/500 [00:05<00:08, 34.76it/s, failures=0, hvi=1.79]
 43%|████▎     | 216/500 [00:05<00:08, 34.76it/s, failures=0, hvi=1.79]
 43%|████▎     | 217/500 [00:05<00:08, 34.76it/s, failures=0, hvi=1.79]
 44%|████▎     | 218/500 [00:05<00:08, 34.73it/s, failures=0, hvi=1.79]
 44%|████▎     | 218/500 [00:05<00:08, 34.73it/s, failures=0, hvi=1.79]
 44%|████▍     | 219/500 [00:05<00:08, 34.73it/s, failures=0, hvi=1.79]
 44%|████▍     | 220/500 [00:05<00:08, 34.73it/s, failures=0, hvi=1.79]
 44%|████▍     | 221/500 [00:05<00:08, 34.73it/s, failures=0, hvi=1.79]
 44%|████▍     | 222/500 [00:05<00:08, 34.73it/s, failures=0, hvi=1.79]
 44%|████▍     | 222/500 [00:05<00:08, 34.73it/s, failures=0, hvi=1.79]
 45%|████▍     | 223/500 [00:05<00:07, 34.73it/s, failures=0, hvi=1.79]
 45%|████▍     | 224/500 [00:05<00:07, 34.73it/s, failures=0, hvi=1.79]
 45%|████▌     | 225/500 [00:05<00:07, 34.73it/s, failures=0, hvi=1.79]
 45%|████▌     | 226/500 [00:05<00:07, 34.60it/s, failures=0, hvi=1.79]
 45%|████▌     | 226/500 [00:05<00:07, 34.60it/s, failures=0, hvi=1.79]
 45%|████▌     | 227/500 [00:05<00:07, 34.60it/s, failures=0, hvi=1.79]
 46%|████▌     | 228/500 [00:05<00:07, 34.60it/s, failures=0, hvi=1.8]
 46%|████▌     | 229/500 [00:05<00:07, 34.60it/s, failures=0, hvi=1.8]
 46%|████▌     | 230/500 [00:05<00:07, 34.40it/s, failures=0, hvi=1.8]
 46%|████▌     | 230/500 [00:05<00:07, 34.40it/s, failures=0, hvi=1.8]
 46%|████▌     | 231/500 [00:05<00:07, 34.40it/s, failures=0, hvi=1.8]
 46%|████▋     | 232/500 [00:05<00:07, 34.40it/s, failures=0, hvi=1.8]
 47%|████▋     | 233/500 [00:05<00:07, 34.40it/s, failures=0, hvi=1.8]
 47%|████▋     | 234/500 [00:05<00:07, 34.25it/s, failures=0, hvi=1.8]
 47%|████▋     | 234/500 [00:05<00:07, 34.25it/s, failures=0, hvi=1.8]
 47%|████▋     | 235/500 [00:05<00:07, 34.25it/s, failures=0, hvi=1.8]
 47%|████▋     | 236/500 [00:05<00:07, 34.25it/s, failures=0, hvi=1.8]
 47%|████▋     | 237/500 [00:06<00:07, 34.25it/s, failures=0, hvi=1.8]
 48%|████▊     | 238/500 [00:06<00:07, 34.16it/s, failures=0, hvi=1.8]
 48%|████▊     | 238/500 [00:06<00:07, 34.16it/s, failures=0, hvi=1.8]
 48%|████▊     | 239/500 [00:06<00:07, 34.16it/s, failures=0, hvi=1.8]
 48%|████▊     | 240/500 [00:06<00:07, 34.16it/s, failures=0, hvi=1.8]
 48%|████▊     | 241/500 [00:06<00:07, 34.16it/s, failures=0, hvi=1.8]
 48%|████▊     | 242/500 [00:06<00:07, 34.06it/s, failures=0, hvi=1.8]
 48%|████▊     | 242/500 [00:06<00:07, 34.06it/s, failures=0, hvi=1.8]
 49%|████▊     | 243/500 [00:06<00:07, 34.06it/s, failures=0, hvi=1.8]
 49%|████▉     | 244/500 [00:06<00:07, 34.06it/s, failures=0, hvi=1.8]
 49%|████▉     | 245/500 [00:06<00:07, 34.06it/s, failures=0, hvi=1.8]
 49%|████▉     | 246/500 [00:06<00:07, 33.98it/s, failures=0, hvi=1.8]
 49%|████▉     | 246/500 [00:06<00:07, 33.98it/s, failures=0, hvi=1.8]
 49%|████▉     | 247/500 [00:06<00:07, 33.98it/s, failures=0, hvi=1.8]
 50%|████▉     | 248/500 [00:06<00:07, 33.98it/s, failures=0, hvi=1.8]
 50%|████▉     | 249/500 [00:06<00:07, 33.98it/s, failures=0, hvi=1.8]
 50%|█████     | 250/500 [00:06<00:08, 31.08it/s, failures=0, hvi=1.8]
 50%|█████     | 250/500 [00:06<00:08, 31.08it/s, failures=0, hvi=1.8]
 50%|█████     | 251/500 [00:06<00:08, 31.08it/s, failures=0, hvi=1.81]
 50%|█████     | 252/500 [00:06<00:07, 31.08it/s, failures=0, hvi=1.81]
 51%|█████     | 253/500 [00:06<00:07, 31.08it/s, failures=0, hvi=1.81]
 51%|█████     | 254/500 [00:06<00:07, 31.81it/s, failures=0, hvi=1.81]
 51%|█████     | 254/500 [00:06<00:07, 31.81it/s, failures=0, hvi=1.81]
 51%|█████     | 255/500 [00:06<00:07, 31.81it/s, failures=0, hvi=1.81]
 51%|█████     | 256/500 [00:06<00:07, 31.81it/s, failures=0, hvi=1.81]
 51%|█████▏    | 257/500 [00:06<00:07, 31.81it/s, failures=0, hvi=1.81]
 52%|█████▏    | 258/500 [00:06<00:07, 32.32it/s, failures=0, hvi=1.81]
 52%|█████▏    | 258/500 [00:06<00:07, 32.32it/s, failures=0, hvi=1.81]
 52%|█████▏    | 259/500 [00:06<00:07, 32.32it/s, failures=0, hvi=1.81]
 52%|█████▏    | 260/500 [00:06<00:07, 32.32it/s, failures=0, hvi=1.81]
 52%|█████▏    | 261/500 [00:06<00:07, 32.32it/s, failures=0, hvi=1.81]
 52%|█████▏    | 262/500 [00:06<00:07, 32.55it/s, failures=0, hvi=1.81]
 52%|█████▏    | 262/500 [00:06<00:07, 32.55it/s, failures=0, hvi=1.81]
 53%|█████▎    | 263/500 [00:06<00:07, 32.55it/s, failures=0, hvi=1.81]
 53%|█████▎    | 264/500 [00:06<00:07, 32.55it/s, failures=0, hvi=1.81]
 53%|█████▎    | 265/500 [00:06<00:07, 32.55it/s, failures=0, hvi=1.81]
 53%|█████▎    | 266/500 [00:06<00:07, 32.75it/s, failures=0, hvi=1.81]
 53%|█████▎    | 266/500 [00:06<00:07, 32.75it/s, failures=0, hvi=1.81]
 53%|█████▎    | 267/500 [00:06<00:07, 32.75it/s, failures=0, hvi=1.81]
 54%|█████▎    | 268/500 [00:06<00:07, 32.75it/s, failures=0, hvi=1.81]
 54%|█████▍    | 269/500 [00:06<00:07, 32.75it/s, failures=0, hvi=1.81]
 54%|█████▍    | 270/500 [00:07<00:06, 32.88it/s, failures=0, hvi=1.81]
 54%|█████▍    | 270/500 [00:07<00:06, 32.88it/s, failures=0, hvi=1.81]
 54%|█████▍    | 271/500 [00:07<00:06, 32.88it/s, failures=0, hvi=1.81]
 54%|█████▍    | 272/500 [00:07<00:06, 32.88it/s, failures=0, hvi=1.81]
 55%|█████▍    | 273/500 [00:07<00:06, 32.88it/s, failures=0, hvi=1.81]
 55%|█████▍    | 274/500 [00:07<00:06, 33.01it/s, failures=0, hvi=1.81]
 55%|█████▍    | 274/500 [00:07<00:06, 33.01it/s, failures=0, hvi=1.81]
 55%|█████▌    | 275/500 [00:07<00:06, 33.01it/s, failures=0, hvi=1.81]
 55%|█████▌    | 276/500 [00:07<00:06, 33.01it/s, failures=0, hvi=1.81]
 55%|█████▌    | 277/500 [00:07<00:06, 33.01it/s, failures=0, hvi=1.81]
 56%|█████▌    | 278/500 [00:07<00:06, 33.01it/s, failures=0, hvi=1.81]
 56%|█████▌    | 278/500 [00:07<00:06, 33.01it/s, failures=0, hvi=1.81]
 56%|█████▌    | 279/500 [00:07<00:06, 33.01it/s, failures=0, hvi=1.81]
 56%|█████▌    | 280/500 [00:07<00:06, 33.01it/s, failures=0, hvi=1.81]
 56%|█████▌    | 281/500 [00:07<00:06, 33.01it/s, failures=0, hvi=1.81]
 56%|█████▋    | 282/500 [00:07<00:06, 33.01it/s, failures=0, hvi=1.81]
 56%|█████▋    | 282/500 [00:07<00:06, 33.01it/s, failures=0, hvi=1.81]
 57%|█████▋    | 283/500 [00:07<00:06, 33.01it/s, failures=0, hvi=1.81]
 57%|█████▋    | 284/500 [00:07<00:06, 33.01it/s, failures=0, hvi=1.81]
 57%|█████▋    | 285/500 [00:07<00:06, 33.01it/s, failures=0, hvi=1.81]
 57%|█████▋    | 286/500 [00:07<00:06, 32.91it/s, failures=0, hvi=1.81]
 57%|█████▋    | 286/500 [00:07<00:06, 32.91it/s, failures=0, hvi=1.81]
 57%|█████▋    | 287/500 [00:07<00:06, 32.91it/s, failures=0, hvi=1.81]
 58%|█████▊    | 288/500 [00:07<00:06, 32.91it/s, failures=0, hvi=1.81]
 58%|█████▊    | 289/500 [00:07<00:06, 32.91it/s, failures=0, hvi=1.81]
 58%|█████▊    | 290/500 [00:07<00:06, 32.90it/s, failures=0, hvi=1.81]
 58%|█████▊    | 290/500 [00:07<00:06, 32.90it/s, failures=0, hvi=1.81]
 58%|█████▊    | 291/500 [00:07<00:06, 32.90it/s, failures=0, hvi=1.81]
 58%|█████▊    | 292/500 [00:07<00:06, 32.90it/s, failures=0, hvi=1.81]
 59%|█████▊    | 293/500 [00:07<00:06, 32.90it/s, failures=0, hvi=1.81]
 59%|█████▉    | 294/500 [00:07<00:06, 32.85it/s, failures=0, hvi=1.81]
 59%|█████▉    | 294/500 [00:07<00:06, 32.85it/s, failures=0, hvi=1.81]
 59%|█████▉    | 295/500 [00:07<00:06, 32.85it/s, failures=0, hvi=1.81]
 59%|█████▉    | 296/500 [00:07<00:06, 32.85it/s, failures=0, hvi=1.81]
 59%|█████▉    | 297/500 [00:07<00:06, 32.85it/s, failures=0, hvi=1.81]
 60%|█████▉    | 298/500 [00:07<00:06, 32.70it/s, failures=0, hvi=1.81]
 60%|█████▉    | 298/500 [00:07<00:06, 32.70it/s, failures=0, hvi=1.81]
 60%|█████▉    | 299/500 [00:07<00:06, 32.70it/s, failures=0, hvi=1.81]
 60%|██████    | 300/500 [00:07<00:06, 32.70it/s, failures=0, hvi=1.81]
 60%|██████    | 301/500 [00:07<00:06, 32.70it/s, failures=0, hvi=1.81]
 60%|██████    | 302/500 [00:07<00:06, 32.66it/s, failures=0, hvi=1.81]
 60%|██████    | 302/500 [00:07<00:06, 32.66it/s, failures=0, hvi=1.81]
 61%|██████    | 303/500 [00:08<00:06, 32.66it/s, failures=0, hvi=1.89]
 61%|██████    | 304/500 [00:08<00:06, 32.66it/s, failures=0, hvi=1.89]
 61%|██████    | 305/500 [00:08<00:05, 32.66it/s, failures=0, hvi=1.89]
 61%|██████    | 306/500 [00:08<00:05, 32.59it/s, failures=0, hvi=1.89]
 61%|██████    | 306/500 [00:08<00:05, 32.59it/s, failures=0, hvi=1.89]
 61%|██████▏   | 307/500 [00:08<00:05, 32.59it/s, failures=0, hvi=1.89]
 62%|██████▏   | 308/500 [00:08<00:05, 32.59it/s, failures=0, hvi=1.89]
 62%|██████▏   | 309/500 [00:08<00:05, 32.59it/s, failures=0, hvi=1.89]
 62%|██████▏   | 310/500 [00:08<00:05, 32.56it/s, failures=0, hvi=1.89]
 62%|██████▏   | 310/500 [00:08<00:05, 32.56it/s, failures=0, hvi=1.89]
 62%|██████▏   | 311/500 [00:08<00:05, 32.56it/s, failures=0, hvi=1.89]
 62%|██████▏   | 312/500 [00:08<00:05, 32.56it/s, failures=0, hvi=1.89]
 63%|██████▎   | 313/500 [00:08<00:05, 32.56it/s, failures=0, hvi=1.89]
 63%|██████▎   | 314/500 [00:08<00:06, 29.85it/s, failures=0, hvi=1.89]
 63%|██████▎   | 314/500 [00:08<00:06, 29.85it/s, failures=0, hvi=1.89]
 63%|██████▎   | 315/500 [00:08<00:06, 29.85it/s, failures=0, hvi=1.89]
 63%|██████▎   | 316/500 [00:08<00:06, 29.85it/s, failures=0, hvi=1.89]
 63%|██████▎   | 317/500 [00:08<00:06, 29.85it/s, failures=0, hvi=1.89]
 64%|██████▎   | 318/500 [00:08<00:05, 30.46it/s, failures=0, hvi=1.89]
 64%|██████▎   | 318/500 [00:08<00:05, 30.46it/s, failures=0, hvi=1.89]
 64%|██████▍   | 319/500 [00:08<00:05, 30.46it/s, failures=0, hvi=1.89]
 64%|██████▍   | 320/500 [00:08<00:05, 30.46it/s, failures=0, hvi=1.89]
 64%|██████▍   | 321/500 [00:08<00:05, 30.46it/s, failures=0, hvi=1.89]
 64%|██████▍   | 322/500 [00:08<00:05, 30.89it/s, failures=0, hvi=1.89]
 64%|██████▍   | 322/500 [00:08<00:05, 30.89it/s, failures=0, hvi=1.89]
 65%|██████▍   | 323/500 [00:08<00:05, 30.89it/s, failures=0, hvi=1.89]
 65%|██████▍   | 324/500 [00:08<00:05, 30.89it/s, failures=0, hvi=1.89]
 65%|██████▌   | 325/500 [00:08<00:05, 30.89it/s, failures=0, hvi=1.89]
 65%|██████▌   | 326/500 [00:08<00:05, 31.20it/s, failures=0, hvi=1.89]
 65%|██████▌   | 326/500 [00:08<00:05, 31.20it/s, failures=0, hvi=1.89]
 65%|██████▌   | 327/500 [00:08<00:05, 31.20it/s, failures=0, hvi=1.89]
 66%|██████▌   | 328/500 [00:08<00:05, 31.20it/s, failures=0, hvi=1.89]
 66%|██████▌   | 329/500 [00:08<00:05, 31.20it/s, failures=0, hvi=1.89]
 66%|██████▌   | 330/500 [00:08<00:05, 31.38it/s, failures=0, hvi=1.89]
 66%|██████▌   | 330/500 [00:08<00:05, 31.38it/s, failures=0, hvi=1.89]
 66%|██████▌   | 331/500 [00:08<00:05, 31.38it/s, failures=0, hvi=1.89]
 66%|██████▋   | 332/500 [00:08<00:05, 31.38it/s, failures=0, hvi=1.89]
 67%|██████▋   | 333/500 [00:09<00:05, 31.38it/s, failures=0, hvi=1.89]
 67%|██████▋   | 334/500 [00:09<00:05, 31.34it/s, failures=0, hvi=1.89]
 67%|██████▋   | 334/500 [00:09<00:05, 31.34it/s, failures=0, hvi=1.89]
 67%|██████▋   | 335/500 [00:09<00:05, 31.34it/s, failures=0, hvi=1.89]
 67%|██████▋   | 336/500 [00:09<00:05, 31.34it/s, failures=0, hvi=1.89]
 67%|██████▋   | 337/500 [00:09<00:05, 31.34it/s, failures=0, hvi=1.89]
 68%|██████▊   | 338/500 [00:09<00:05, 31.44it/s, failures=0, hvi=1.89]
 68%|██████▊   | 338/500 [00:09<00:05, 31.44it/s, failures=0, hvi=1.89]
 68%|██████▊   | 339/500 [00:09<00:05, 31.44it/s, failures=0, hvi=1.89]
 68%|██████▊   | 340/500 [00:09<00:05, 31.44it/s, failures=0, hvi=1.89]
 68%|██████▊   | 341/500 [00:09<00:05, 31.44it/s, failures=0, hvi=1.89]
 68%|██████▊   | 342/500 [00:09<00:05, 31.52it/s, failures=0, hvi=1.89]
 68%|██████▊   | 342/500 [00:09<00:05, 31.52it/s, failures=0, hvi=1.89]
 69%|██████▊   | 343/500 [00:09<00:04, 31.52it/s, failures=0, hvi=1.89]
 69%|██████▉   | 344/500 [00:09<00:04, 31.52it/s, failures=0, hvi=1.89]
 69%|██████▉   | 345/500 [00:09<00:04, 31.52it/s, failures=0, hvi=1.89]
 69%|██████▉   | 346/500 [00:09<00:04, 31.34it/s, failures=0, hvi=1.89]
 69%|██████▉   | 346/500 [00:09<00:04, 31.34it/s, failures=0, hvi=1.89]
 69%|██████▉   | 347/500 [00:09<00:04, 31.34it/s, failures=0, hvi=1.89]
 70%|██████▉   | 348/500 [00:09<00:04, 31.34it/s, failures=0, hvi=1.89]
 70%|██████▉   | 349/500 [00:09<00:04, 31.34it/s, failures=0, hvi=1.89]
 70%|███████   | 350/500 [00:09<00:04, 31.32it/s, failures=0, hvi=1.89]
 70%|███████   | 350/500 [00:09<00:04, 31.32it/s, failures=0, hvi=1.89]
 70%|███████   | 351/500 [00:09<00:04, 31.32it/s, failures=0, hvi=1.89]
 70%|███████   | 352/500 [00:09<00:04, 31.32it/s, failures=0, hvi=1.89]
 71%|███████   | 353/500 [00:09<00:04, 31.32it/s, failures=0, hvi=1.89]
 71%|███████   | 354/500 [00:09<00:04, 31.36it/s, failures=0, hvi=1.89]
 71%|███████   | 354/500 [00:09<00:04, 31.36it/s, failures=0, hvi=1.89]
 71%|███████   | 355/500 [00:09<00:04, 31.36it/s, failures=0, hvi=1.89]
 71%|███████   | 356/500 [00:09<00:04, 31.36it/s, failures=0, hvi=1.89]
 71%|███████▏  | 357/500 [00:09<00:04, 31.36it/s, failures=0, hvi=1.89]
 72%|███████▏  | 358/500 [00:09<00:04, 31.34it/s, failures=0, hvi=1.89]
 72%|███████▏  | 358/500 [00:09<00:04, 31.34it/s, failures=0, hvi=1.89]
 72%|███████▏  | 359/500 [00:09<00:04, 31.34it/s, failures=0, hvi=1.89]
 72%|███████▏  | 360/500 [00:09<00:04, 31.34it/s, failures=0, hvi=1.89]
 72%|███████▏  | 361/500 [00:09<00:04, 31.34it/s, failures=0, hvi=1.89]
 72%|███████▏  | 362/500 [00:09<00:04, 31.24it/s, failures=0, hvi=1.89]
 72%|███████▏  | 362/500 [00:09<00:04, 31.24it/s, failures=0, hvi=1.89]
 73%|███████▎  | 363/500 [00:09<00:04, 31.24it/s, failures=0, hvi=1.89]
 73%|███████▎  | 364/500 [00:09<00:04, 31.24it/s, failures=0, hvi=1.89]
 73%|███████▎  | 365/500 [00:10<00:04, 31.24it/s, failures=0, hvi=1.89]
 73%|███████▎  | 366/500 [00:10<00:04, 31.13it/s, failures=0, hvi=1.89]
 73%|███████▎  | 366/500 [00:10<00:04, 31.13it/s, failures=0, hvi=1.89]
 73%|███████▎  | 367/500 [00:10<00:04, 31.13it/s, failures=0, hvi=1.89]
 74%|███████▎  | 368/500 [00:10<00:04, 31.13it/s, failures=0, hvi=1.89]
 74%|███████▍  | 369/500 [00:10<00:04, 31.13it/s, failures=0, hvi=1.89]
 74%|███████▍  | 370/500 [00:10<00:04, 31.06it/s, failures=0, hvi=1.89]
 74%|███████▍  | 370/500 [00:10<00:04, 31.06it/s, failures=0, hvi=1.89]
 74%|███████▍  | 371/500 [00:10<00:04, 31.06it/s, failures=0, hvi=1.89]
 74%|███████▍  | 372/500 [00:10<00:04, 31.06it/s, failures=0, hvi=1.89]
 75%|███████▍  | 373/500 [00:10<00:04, 31.06it/s, failures=0, hvi=1.89]
 75%|███████▍  | 374/500 [00:10<00:04, 31.05it/s, failures=0, hvi=1.89]
 75%|███████▍  | 374/500 [00:10<00:04, 31.05it/s, failures=0, hvi=1.89]
 75%|███████▌  | 375/500 [00:10<00:04, 31.05it/s, failures=0, hvi=1.89]
 75%|███████▌  | 376/500 [00:10<00:03, 31.05it/s, failures=0, hvi=1.89]
 75%|███████▌  | 377/500 [00:10<00:03, 31.05it/s, failures=0, hvi=1.89]
 76%|███████▌  | 378/500 [00:10<00:03, 30.96it/s, failures=0, hvi=1.89]
 76%|███████▌  | 378/500 [00:10<00:03, 30.96it/s, failures=0, hvi=1.89]
 76%|███████▌  | 379/500 [00:10<00:03, 30.96it/s, failures=0, hvi=1.89]
 76%|███████▌  | 380/500 [00:10<00:03, 30.96it/s, failures=0, hvi=1.89]
 76%|███████▌  | 381/500 [00:10<00:03, 30.96it/s, failures=0, hvi=1.89]
 76%|███████▋  | 382/500 [00:10<00:04, 28.54it/s, failures=0, hvi=1.89]
 76%|███████▋  | 382/500 [00:10<00:04, 28.54it/s, failures=0, hvi=1.89]
 77%|███████▋  | 383/500 [00:10<00:04, 28.54it/s, failures=0, hvi=1.89]
 77%|███████▋  | 384/500 [00:10<00:04, 28.54it/s, failures=0, hvi=1.89]
 77%|███████▋  | 385/500 [00:10<00:04, 28.54it/s, failures=0, hvi=1.89]
 77%|███████▋  | 386/500 [00:10<00:03, 29.19it/s, failures=0, hvi=1.89]
 77%|███████▋  | 386/500 [00:10<00:03, 29.19it/s, failures=0, hvi=1.89]
 77%|███████▋  | 387/500 [00:10<00:03, 29.19it/s, failures=0, hvi=1.89]
 78%|███████▊  | 388/500 [00:10<00:03, 29.19it/s, failures=0, hvi=1.89]
 78%|███████▊  | 389/500 [00:10<00:03, 29.19it/s, failures=0, hvi=1.89]
 78%|███████▊  | 390/500 [00:10<00:03, 29.67it/s, failures=0, hvi=1.89]
 78%|███████▊  | 390/500 [00:10<00:03, 29.67it/s, failures=0, hvi=1.89]
 78%|███████▊  | 391/500 [00:10<00:03, 29.67it/s, failures=0, hvi=1.89]
 78%|███████▊  | 392/500 [00:10<00:03, 29.67it/s, failures=0, hvi=1.89]
 79%|███████▊  | 393/500 [00:10<00:03, 29.67it/s, failures=0, hvi=1.89]
 79%|███████▉  | 394/500 [00:11<00:03, 29.95it/s, failures=0, hvi=1.89]
 79%|███████▉  | 394/500 [00:11<00:03, 29.95it/s, failures=0, hvi=1.89]
 79%|███████▉  | 395/500 [00:11<00:03, 29.95it/s, failures=0, hvi=1.89]
 79%|███████▉  | 396/500 [00:11<00:03, 29.95it/s, failures=0, hvi=1.9]
 79%|███████▉  | 397/500 [00:11<00:03, 29.95it/s, failures=0, hvi=1.9]
 80%|███████▉  | 398/500 [00:11<00:03, 30.08it/s, failures=0, hvi=1.9]
 80%|███████▉  | 398/500 [00:11<00:03, 30.08it/s, failures=0, hvi=1.9]
 80%|███████▉  | 399/500 [00:11<00:03, 30.08it/s, failures=0, hvi=1.9]
 80%|████████  | 400/500 [00:11<00:03, 30.08it/s, failures=0, hvi=1.9]
 80%|████████  | 401/500 [00:11<00:03, 30.08it/s, failures=0, hvi=1.9]
 80%|████████  | 402/500 [00:11<00:03, 30.17it/s, failures=0, hvi=1.9]
 80%|████████  | 402/500 [00:11<00:03, 30.17it/s, failures=0, hvi=1.9]
 81%|████████  | 403/500 [00:11<00:03, 30.17it/s, failures=0, hvi=1.9]
 81%|████████  | 404/500 [00:11<00:03, 30.17it/s, failures=0, hvi=1.9]
 81%|████████  | 405/500 [00:11<00:03, 30.17it/s, failures=0, hvi=1.9]
 81%|████████  | 406/500 [00:11<00:03, 30.20it/s, failures=0, hvi=1.9]
 81%|████████  | 406/500 [00:11<00:03, 30.20it/s, failures=0, hvi=1.9]
 81%|████████▏ | 407/500 [00:11<00:03, 30.20it/s, failures=0, hvi=1.91]
 82%|████████▏ | 408/500 [00:11<00:03, 30.20it/s, failures=0, hvi=1.91]
 82%|████████▏ | 409/500 [00:11<00:03, 30.20it/s, failures=0, hvi=1.91]
 82%|████████▏ | 410/500 [00:11<00:02, 30.19it/s, failures=0, hvi=1.91]
 82%|████████▏ | 410/500 [00:11<00:02, 30.19it/s, failures=0, hvi=1.91]
 82%|████████▏ | 411/500 [00:11<00:02, 30.19it/s, failures=0, hvi=1.91]
 82%|████████▏ | 412/500 [00:11<00:02, 30.19it/s, failures=0, hvi=1.91]
 83%|████████▎ | 413/500 [00:11<00:02, 30.19it/s, failures=0, hvi=1.91]
 83%|████████▎ | 414/500 [00:11<00:02, 30.16it/s, failures=0, hvi=1.91]
 83%|████████▎ | 414/500 [00:11<00:02, 30.16it/s, failures=0, hvi=1.91]
 83%|████████▎ | 415/500 [00:11<00:02, 30.16it/s, failures=0, hvi=1.91]
 83%|████████▎ | 416/500 [00:11<00:02, 30.16it/s, failures=0, hvi=1.91]
 83%|████████▎ | 417/500 [00:11<00:02, 30.16it/s, failures=0, hvi=1.91]
 84%|████████▎ | 418/500 [00:11<00:02, 30.16it/s, failures=0, hvi=1.91]
 84%|████████▎ | 418/500 [00:11<00:02, 30.16it/s, failures=0, hvi=1.91]
 84%|████████▍ | 419/500 [00:11<00:02, 30.16it/s, failures=0, hvi=1.91]
 84%|████████▍ | 420/500 [00:11<00:02, 30.16it/s, failures=0, hvi=1.91]
 84%|████████▍ | 421/500 [00:11<00:02, 30.16it/s, failures=0, hvi=1.91]
 84%|████████▍ | 422/500 [00:11<00:02, 30.13it/s, failures=0, hvi=1.91]
 84%|████████▍ | 422/500 [00:11<00:02, 30.13it/s, failures=0, hvi=1.91]
 85%|████████▍ | 423/500 [00:11<00:02, 30.13it/s, failures=0, hvi=1.91]
 85%|████████▍ | 424/500 [00:11<00:02, 30.13it/s, failures=0, hvi=1.91]
 85%|████████▌ | 425/500 [00:12<00:02, 30.13it/s, failures=0, hvi=1.91]
 85%|████████▌ | 426/500 [00:12<00:02, 30.09it/s, failures=0, hvi=1.91]
 85%|████████▌ | 426/500 [00:12<00:02, 30.09it/s, failures=0, hvi=1.91]
 85%|████████▌ | 427/500 [00:12<00:02, 30.09it/s, failures=0, hvi=1.91]
 86%|████████▌ | 428/500 [00:12<00:02, 30.09it/s, failures=0, hvi=1.91]
 86%|████████▌ | 429/500 [00:12<00:02, 30.09it/s, failures=0, hvi=1.91]
 86%|████████▌ | 430/500 [00:12<00:02, 29.99it/s, failures=0, hvi=1.91]
 86%|████████▌ | 430/500 [00:12<00:02, 29.99it/s, failures=0, hvi=1.91]
 86%|████████▌ | 431/500 [00:12<00:02, 29.99it/s, failures=0, hvi=1.91]
 86%|████████▋ | 432/500 [00:12<00:02, 29.99it/s, failures=0, hvi=1.91]
 87%|████████▋ | 433/500 [00:12<00:02, 29.95it/s, failures=0, hvi=1.91]
 87%|████████▋ | 433/500 [00:12<00:02, 29.95it/s, failures=0, hvi=1.91]
 87%|████████▋ | 434/500 [00:12<00:02, 29.95it/s, failures=0, hvi=1.91]
 87%|████████▋ | 435/500 [00:12<00:02, 29.95it/s, failures=0, hvi=1.91]
 87%|████████▋ | 436/500 [00:12<00:02, 29.85it/s, failures=0, hvi=1.91]
 87%|████████▋ | 436/500 [00:12<00:02, 29.85it/s, failures=0, hvi=1.91]
 87%|████████▋ | 437/500 [00:12<00:02, 29.85it/s, failures=0, hvi=1.91]
 88%|████████▊ | 438/500 [00:12<00:02, 29.85it/s, failures=0, hvi=1.91]
 88%|████████▊ | 439/500 [00:12<00:02, 29.70it/s, failures=0, hvi=1.91]
 88%|████████▊ | 439/500 [00:12<00:02, 29.70it/s, failures=0, hvi=1.91]
 88%|████████▊ | 440/500 [00:12<00:02, 29.70it/s, failures=0, hvi=1.91]
 88%|████████▊ | 441/500 [00:12<00:01, 29.70it/s, failures=0, hvi=1.91]
 88%|████████▊ | 442/500 [00:12<00:01, 29.69it/s, failures=0, hvi=1.91]
 88%|████████▊ | 442/500 [00:12<00:01, 29.69it/s, failures=0, hvi=1.91]
 89%|████████▊ | 443/500 [00:12<00:01, 29.69it/s, failures=0, hvi=1.91]
 89%|████████▉ | 444/500 [00:12<00:01, 29.69it/s, failures=0, hvi=1.91]
 89%|████████▉ | 445/500 [00:12<00:01, 29.68it/s, failures=0, hvi=1.91]
 89%|████████▉ | 445/500 [00:12<00:01, 29.68it/s, failures=0, hvi=1.91]
 89%|████████▉ | 446/500 [00:12<00:01, 29.68it/s, failures=0, hvi=1.91]
 89%|████████▉ | 447/500 [00:12<00:01, 29.68it/s, failures=0, hvi=1.91]
 90%|████████▉ | 448/500 [00:12<00:01, 26.82it/s, failures=0, hvi=1.91]
 90%|████████▉ | 448/500 [00:12<00:01, 26.82it/s, failures=0, hvi=1.91]
 90%|████████▉ | 449/500 [00:12<00:01, 26.82it/s, failures=0, hvi=1.91]
 90%|█████████ | 450/500 [00:12<00:01, 26.82it/s, failures=0, hvi=1.91]
 90%|█████████ | 451/500 [00:12<00:01, 27.50it/s, failures=0, hvi=1.91]
 90%|█████████ | 451/500 [00:12<00:01, 27.50it/s, failures=0, hvi=1.91]
 90%|█████████ | 452/500 [00:12<00:01, 27.50it/s, failures=0, hvi=1.91]
 91%|█████████ | 453/500 [00:13<00:01, 27.50it/s, failures=0, hvi=1.91]
 91%|█████████ | 454/500 [00:13<00:01, 27.89it/s, failures=0, hvi=1.91]
 91%|█████████ | 454/500 [00:13<00:01, 27.89it/s, failures=0, hvi=1.91]
 91%|█████████ | 455/500 [00:13<00:01, 27.89it/s, failures=0, hvi=1.91]
 91%|█████████ | 456/500 [00:13<00:01, 27.89it/s, failures=0, hvi=1.91]
 91%|█████████▏| 457/500 [00:13<00:01, 28.27it/s, failures=0, hvi=1.91]
 91%|█████████▏| 457/500 [00:13<00:01, 28.27it/s, failures=0, hvi=1.91]
 92%|█████████▏| 458/500 [00:13<00:01, 28.27it/s, failures=0, hvi=1.91]
 92%|█████████▏| 459/500 [00:13<00:01, 28.27it/s, failures=0, hvi=1.91]
 92%|█████████▏| 460/500 [00:13<00:01, 28.55it/s, failures=0, hvi=1.91]
 92%|█████████▏| 460/500 [00:13<00:01, 28.55it/s, failures=0, hvi=1.91]
 92%|█████████▏| 461/500 [00:13<00:01, 28.55it/s, failures=0, hvi=1.91]
 92%|█████████▏| 462/500 [00:13<00:01, 28.55it/s, failures=0, hvi=1.91]
 93%|█████████▎| 463/500 [00:13<00:01, 28.76it/s, failures=0, hvi=1.91]
 93%|█████████▎| 463/500 [00:13<00:01, 28.76it/s, failures=0, hvi=1.91]
 93%|█████████▎| 464/500 [00:13<00:01, 28.76it/s, failures=0, hvi=1.91]
 93%|█████████▎| 465/500 [00:13<00:01, 28.76it/s, failures=0, hvi=1.91]
 93%|█████████▎| 466/500 [00:13<00:01, 28.89it/s, failures=0, hvi=1.91]
 93%|█████████▎| 466/500 [00:13<00:01, 28.89it/s, failures=0, hvi=1.91]
 93%|█████████▎| 467/500 [00:13<00:01, 28.89it/s, failures=0, hvi=1.91]
 94%|█████████▎| 468/500 [00:13<00:01, 28.89it/s, failures=0, hvi=1.91]
 94%|█████████▍| 469/500 [00:13<00:01, 28.93it/s, failures=0, hvi=1.91]
 94%|█████████▍| 469/500 [00:13<00:01, 28.93it/s, failures=0, hvi=1.91]
 94%|█████████▍| 470/500 [00:13<00:01, 28.93it/s, failures=0, hvi=1.91]
 94%|█████████▍| 471/500 [00:13<00:01, 28.93it/s, failures=0, hvi=1.91]
 94%|█████████▍| 472/500 [00:13<00:00, 28.95it/s, failures=0, hvi=1.91]
 94%|█████████▍| 472/500 [00:13<00:00, 28.95it/s, failures=0, hvi=1.91]
 95%|█████████▍| 473/500 [00:13<00:00, 28.95it/s, failures=0, hvi=1.91]
 95%|█████████▍| 474/500 [00:13<00:00, 28.95it/s, failures=0, hvi=1.91]
 95%|█████████▌| 475/500 [00:13<00:00, 29.04it/s, failures=0, hvi=1.91]
 95%|█████████▌| 475/500 [00:13<00:00, 29.04it/s, failures=0, hvi=1.91]
 95%|█████████▌| 476/500 [00:13<00:00, 29.04it/s, failures=0, hvi=1.91]
 95%|█████████▌| 477/500 [00:13<00:00, 29.04it/s, failures=0, hvi=1.91]
 96%|█████████▌| 478/500 [00:13<00:00, 28.95it/s, failures=0, hvi=1.91]
 96%|█████████▌| 478/500 [00:13<00:00, 28.95it/s, failures=0, hvi=1.91]
 96%|█████████▌| 479/500 [00:13<00:00, 28.95it/s, failures=0, hvi=1.91]
 96%|█████████▌| 480/500 [00:13<00:00, 28.95it/s, failures=0, hvi=1.91]
 96%|█████████▌| 481/500 [00:13<00:00, 28.96it/s, failures=0, hvi=1.91]
 96%|█████████▌| 481/500 [00:13<00:00, 28.96it/s, failures=0, hvi=1.91]
 96%|█████████▋| 482/500 [00:14<00:00, 28.96it/s, failures=0, hvi=1.91]
 97%|█████████▋| 483/500 [00:14<00:00, 28.96it/s, failures=0, hvi=1.91]
 97%|█████████▋| 484/500 [00:14<00:00, 28.86it/s, failures=0, hvi=1.91]
 97%|█████████▋| 484/500 [00:14<00:00, 28.86it/s, failures=0, hvi=1.91]
 97%|█████████▋| 485/500 [00:14<00:00, 28.86it/s, failures=0, hvi=1.91]
 97%|█████████▋| 486/500 [00:14<00:00, 28.86it/s, failures=0, hvi=1.91]
 97%|█████████▋| 487/500 [00:14<00:00, 28.78it/s, failures=0, hvi=1.91]
 97%|█████████▋| 487/500 [00:14<00:00, 28.78it/s, failures=0, hvi=1.91]
 98%|█████████▊| 488/500 [00:14<00:00, 28.78it/s, failures=0, hvi=1.91]
 98%|█████████▊| 489/500 [00:14<00:00, 28.78it/s, failures=0, hvi=1.91]
 98%|█████████▊| 490/500 [00:14<00:00, 28.76it/s, failures=0, hvi=1.91]
 98%|█████████▊| 490/500 [00:14<00:00, 28.76it/s, failures=0, hvi=1.91]
 98%|█████████▊| 491/500 [00:14<00:00, 28.76it/s, failures=0, hvi=1.91]
 98%|█████████▊| 492/500 [00:14<00:00, 28.76it/s, failures=0, hvi=1.91]
 99%|█████████▊| 493/500 [00:14<00:00, 28.76it/s, failures=0, hvi=1.91]
 99%|█████████▊| 493/500 [00:14<00:00, 28.76it/s, failures=0, hvi=1.91]
 99%|█████████▉| 494/500 [00:14<00:00, 28.76it/s, failures=0, hvi=1.91]
 99%|█████████▉| 495/500 [00:14<00:00, 28.76it/s, failures=0, hvi=1.91]
 99%|█████████▉| 496/500 [00:14<00:00, 28.73it/s, failures=0, hvi=1.91]
 99%|█████████▉| 496/500 [00:14<00:00, 28.73it/s, failures=0, hvi=1.91]
 99%|█████████▉| 497/500 [00:14<00:00, 28.73it/s, failures=0, hvi=1.91]
100%|█████████▉| 498/500 [00:14<00:00, 28.73it/s, failures=0, hvi=1.91]
100%|█████████▉| 499/500 [00:14<00:00, 28.61it/s, failures=0, hvi=1.91]
100%|█████████▉| 499/500 [00:14<00:00, 28.61it/s, failures=0, hvi=1.91]
100%|██████████| 500/500 [00:14<00:00, 28.61it/s, failures=0, hvi=1.91]
100%|██████████| 500/500 [00:14<00:00, 34.15it/s, failures=0, hvi=1.91]

A Pandas table of results is returned by the search and also saved at ./results.csv. An other location can be specified by using CBO(..., log_dir=...).

results
p:x0 p:x1 p:x2 p:x3 p:x4 objective_0 objective_1 job_id job_status m:timestamp_submit m:timestamp_start m:timestamp_end m:timestamp_gather pareto_efficient
0 0.132669 0.013103 0.000849 0.028166 0.477660 -1.672363 -0.353647 0 DONE 0.009166 1.755518e+09 1.755518e+09 0.009453 False
1 0.445254 0.587523 0.731304 0.203790 0.529300 -0.879828 -0.740171 1 DONE 0.030374 1.755518e+09 1.755518e+09 0.030762 False
2 0.992770 0.923032 0.260705 0.623351 0.312814 -0.014610 -1.286390 2 DONE 0.043637 1.755518e+09 1.755518e+09 0.043849 False
3 0.433051 0.933044 0.576779 0.958234 0.830414 -1.175913 -0.951372 3 DONE 0.056278 1.755518e+09 1.755518e+09 0.056461 False
4 0.992771 0.156779 0.224604 0.442812 0.136333 -0.015093 -1.329082 4 DONE 0.068399 1.755518e+09 1.755518e+09 0.068564 False
... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
495 0.799012 0.598050 0.108476 0.492106 0.515866 -0.361171 -1.105727 495 DONE 14.512292 1.755518e+09 1.755518e+09 14.512488 False
496 0.079906 0.509656 0.656180 0.393031 0.398593 -1.037981 -0.130973 496 DONE 14.547404 1.755518e+09 1.755518e+09 14.547589 False
497 0.692482 0.404241 0.645611 0.424970 0.516073 -0.481323 -0.917695 497 DONE 14.582559 1.755518e+09 1.755518e+09 14.582775 False
498 0.354439 0.456215 0.494466 0.542073 0.537370 -0.853318 -0.531134 498 DONE 14.618182 1.755518e+09 1.755518e+09 14.618367 True
499 0.798545 0.551645 0.450832 0.392065 0.334940 -0.324875 -0.992144 499 DONE 14.652979 1.755518e+09 1.755518e+09 14.653161 False

500 rows × 14 columns



In this table we retrieve:

  • columns starting by p: which are the optimized variables.

  • the objective_{i} are the objectives returned by the black-box function.

  • the job_id is the identifier of the executed evaluations.

  • columns starting by m: are metadata returned by the black-box function.

  • pareto_efficient is a column only returned for MOO which specify if the evaluation is part of the set of optimal solutions.

Let us use this table to visualize evaluated objectives. The estimated optimal solutions will be colored in red.

Code (Plot evaluated objectives)
fig, ax = plt.subplots(figsize=(WIDTH_PLOTS, HEIGHT_PLOTS), tight_layout=True)
_ = ax.plot(
    -results[~results["pareto_efficient"]]["objective_0"],
    -results[~results["pareto_efficient"]]["objective_1"],
    "o",
    color="blue",
    alpha=0.7,
    label="Non Pareto-Efficient",
)
_ = ax.plot(
    -results[results["pareto_efficient"]]["objective_0"],
    -results[results["pareto_efficient"]]["objective_1"],
    "o",
    color="red",
    alpha=0.7,
    label="Pareto-Efficient",
)
_ = ax.grid()
_ = ax.legend()
_ = ax.set_xlabel("Objective 0")
_ = ax.set_ylabel("Objective 1")
plot multi objective optimization

Let us look the evolution of the hypervolume indicator. This metric should increase over time.

Code (Plot hypervolume)
scorer = bench.scorer
hvi = scorer.hypervolume(results[["objective_0", "objective_1"]].values)
x = list(range(1, len(hvi)+1))
fig, ax = plt.subplots(figsize=(WIDTH_PLOTS, HEIGHT_PLOTS), tight_layout=True)
_ = ax.plot(x, hvi)
_ = ax.grid()
_ = ax.set_xlabel("Evaluations")
_ = ax.set_ylabel("Hypervolume Indicator")
plot multi objective optimization

Total running time of the script: (0 minutes 15.918 seconds)

Gallery generated by Sphinx-Gallery

Contents