Mutli-Objective Black-Box Optimization

Note

Go to the end to download the full example code.

Mutli-Objective Black-Box Optimization#

In this tutorial, we will explore how to run black-box multi-objective optimization (MOO). In this setting, the goal is to resolve the following problem:

\[\text{max}_x (f_0(x), f_1(x), ..., f_n(x))\]

where \(x\) is the set of optimized variables and \(f_i\) are the different objectives. In DeepHyper, we use scalarization to transform such multi-objective problem into a single-objective problem:

\[\text{max}_x s_w((f_0(x), f_1(x), ..., f_n(x)))\]

where \(w\) is a set of weights which manages the trade-off between objectives and \(s_w : \mathbb{R}^n \rightarrow \mathbb{R}\). The weight vector \(w\) is randomized and re-sampled for each new batch of suggestion from the optimizer.

We will look at the DTLZ benchmark suite, a classic in multi-objective optimization (MOO) litterature. This benchmark exibit some characteristic cases of MOO. By default, this tutorial is loading the DTLZ-II benchmark which exibit a Pareto-Front with a concave shape.

Installation and imports#

Installing dependencies with the pip installation is recommended. It requires Python >= 3.10.

%%bash
pip install deephyper
pip install -e "git+https://github.com/deephyper/benchmark.git@main#egg=deephyper-benchmark"
Code (Import statements)
import matplotlib.pyplot as plt

from deephyper.hpo import CBO
from deephyper_benchmark.benchmarks.dtlz import DTLZBenchmark

WIDTH_PLOTS = 8
HEIGHT_PLOTS = WIDTH_PLOTS / 1.618

n_objectives = 2
bench = DTLZBenchmark(nobj=n_objectives)

We can display the variable search space of the benchmark we just loaded:

bench.problem

Configuration space object:
  Hyperparameters:
    x0, Type: UniformFloat, Range: [0.0, 1.0], Default: 0.5
    x1, Type: UniformFloat, Range: [0.0, 1.0], Default: 0.5
    x2, Type: UniformFloat, Range: [0.0, 1.0], Default: 0.5
    x3, Type: UniformFloat, Range: [0.0, 1.0], Default: 0.5
    x4, Type: UniformFloat, Range: [0.0, 1.0], Default: 0.5

To define a black-box for multi-objective optimization it is very similar to single-objective optimization at the difference that the objective can now be a list of values. A first possibility is:

def run(job):
    ...
    return objective_0, objective_1, ..., objective_n

which just returns the objectives to optimize as a tuple. If additionnal metadata are interesting to gather for each evaluation it is also possible to return them by following this format:

def run(job):
    ...
    return {
        "objective": [objective_0, objective_1, ..., objective_n],
        "metadata": {
            "flops": ...,
            "memory_footprint": ...,
            "duration": ...,
         }
     }

each of the metadata needs to be JSON serializable and will be returned in the final results with a column name formatted as m:metadata_key such as m:duration.

For the search algorithm, we use the centralized Bayesian Optimization search (CBO). Search algorithm

The arguments specific to multi-objective optimization are:

  • moo_scalarization_strategy is used to specify the scalarization strategy. Chebyshev scalarizationis capable of generating a diverse set of solutions for non-convex problems.

  • moo_scalarization_weight argument is used to specify the weight of objectives in the scalarization. "random" is used to generate a random weight vector at each iteration.

search = CBO(
    bench.problem,
    bench.run_function,
    acq_optimizer="sampling",
    moo_scalarization_strategy="AugChebyshev",
    moo_scalarization_weight="random",
    verbose=1,
)

WARNING:root:Results file already exists, it will be renamed to /Users/romainegele/Documents/DeepHyper/deephyper/examples/examples_bbo/results_20250326-104325.csv

Launch the search for a given number of evaluations other stopping criteria can be used (e.g. timeout, early-stopping/convergence)

results = search.search(max_evals=500)

  0%|          | 0/500 [00:00<?, ?it/s]
  0%|          | 1/500 [00:00<00:00, 2532.79it/s, failures=0, hvi=0]
  0%|          | 2/500 [00:00<00:05, 89.76it/s, failures=0, hvi=0]
  1%|          | 3/500 [00:00<00:06, 74.18it/s, failures=0, hvi=0.121]
  1%|          | 4/500 [00:00<00:07, 69.66it/s, failures=0, hvi=0.167]
  1%|          | 5/500 [00:00<00:07, 67.26it/s, failures=0, hvi=0.189]
  1%|          | 6/500 [00:00<00:07, 65.69it/s, failures=0, hvi=0.189]
  1%|▏         | 7/500 [00:00<00:07, 64.71it/s, failures=0, hvi=0.189]
  1%|▏         | 7/500 [00:00<00:07, 64.71it/s, failures=0, hvi=1.04]
  2%|▏         | 8/500 [00:00<00:07, 64.71it/s, failures=0, hvi=1.07]
  2%|▏         | 9/500 [00:00<00:07, 64.71it/s, failures=0, hvi=1.07]
  2%|▏         | 10/500 [00:00<00:07, 64.71it/s, failures=0, hvi=1.07]
  2%|▏         | 11/500 [00:00<00:07, 64.71it/s, failures=0, hvi=1.07]
  2%|▏         | 12/500 [00:00<00:07, 64.71it/s, failures=0, hvi=1.07]
  3%|▎         | 13/500 [00:00<00:07, 64.71it/s, failures=0, hvi=1.26]
  3%|▎         | 14/500 [00:00<00:09, 52.26it/s, failures=0, hvi=1.26]
  3%|▎         | 14/500 [00:00<00:09, 52.26it/s, failures=0, hvi=1.26]
  3%|▎         | 15/500 [00:00<00:09, 52.26it/s, failures=0, hvi=1.26]
  3%|▎         | 16/500 [00:00<00:09, 52.26it/s, failures=0, hvi=1.26]
  3%|▎         | 17/500 [00:00<00:09, 52.26it/s, failures=0, hvi=1.26]
  4%|▎         | 18/500 [00:00<00:09, 52.26it/s, failures=0, hvi=1.26]
  4%|▍         | 19/500 [00:00<00:09, 52.26it/s, failures=0, hvi=1.27]
  4%|▍         | 20/500 [00:00<00:13, 34.71it/s, failures=0, hvi=1.27]
  4%|▍         | 20/500 [00:00<00:13, 34.71it/s, failures=0, hvi=1.3]
  4%|▍         | 21/500 [00:00<00:13, 34.71it/s, failures=0, hvi=1.31]
  4%|▍         | 22/500 [00:00<00:13, 34.71it/s, failures=0, hvi=1.31]
  5%|▍         | 23/500 [00:00<00:13, 34.71it/s, failures=0, hvi=1.34]
  5%|▍         | 24/500 [00:00<00:13, 34.71it/s, failures=0, hvi=1.39]
  5%|▌         | 25/500 [00:00<00:13, 35.74it/s, failures=0, hvi=1.39]
  5%|▌         | 25/500 [00:00<00:13, 35.74it/s, failures=0, hvi=1.39]
  5%|▌         | 26/500 [00:00<00:13, 35.74it/s, failures=0, hvi=1.42]
  5%|▌         | 27/500 [00:00<00:13, 35.74it/s, failures=0, hvi=1.42]
  6%|▌         | 28/500 [00:00<00:13, 35.74it/s, failures=0, hvi=1.44]
  6%|▌         | 29/500 [00:00<00:12, 36.37it/s, failures=0, hvi=1.44]
  6%|▌         | 29/500 [00:00<00:12, 36.37it/s, failures=0, hvi=1.44]
  6%|▌         | 30/500 [00:00<00:12, 36.37it/s, failures=0, hvi=1.44]
  6%|▌         | 31/500 [00:00<00:12, 36.37it/s, failures=0, hvi=1.44]
  6%|▋         | 32/500 [00:00<00:12, 36.37it/s, failures=0, hvi=1.47]
  7%|▋         | 33/500 [00:00<00:12, 36.64it/s, failures=0, hvi=1.47]
  7%|▋         | 33/500 [00:00<00:12, 36.64it/s, failures=0, hvi=1.47]
  7%|▋         | 34/500 [00:00<00:12, 36.64it/s, failures=0, hvi=1.47]
  7%|▋         | 35/500 [00:00<00:12, 36.64it/s, failures=0, hvi=1.47]
  7%|▋         | 36/500 [00:00<00:12, 36.64it/s, failures=0, hvi=1.47]
  7%|▋         | 37/500 [00:00<00:12, 36.72it/s, failures=0, hvi=1.47]
  7%|▋         | 37/500 [00:00<00:12, 36.72it/s, failures=0, hvi=1.47]
  8%|▊         | 38/500 [00:00<00:12, 36.72it/s, failures=0, hvi=1.47]
  8%|▊         | 39/500 [00:01<00:12, 36.72it/s, failures=0, hvi=1.47]
  8%|▊         | 40/500 [00:01<00:12, 36.72it/s, failures=0, hvi=1.47]
  8%|▊         | 41/500 [00:01<00:12, 36.92it/s, failures=0, hvi=1.47]
  8%|▊         | 41/500 [00:01<00:12, 36.92it/s, failures=0, hvi=1.47]
  8%|▊         | 42/500 [00:01<00:12, 36.92it/s, failures=0, hvi=1.47]
  9%|▊         | 43/500 [00:01<00:12, 36.92it/s, failures=0, hvi=1.47]
  9%|▉         | 44/500 [00:01<00:12, 36.92it/s, failures=0, hvi=1.47]
  9%|▉         | 45/500 [00:01<00:12, 36.66it/s, failures=0, hvi=1.47]
  9%|▉         | 45/500 [00:01<00:12, 36.66it/s, failures=0, hvi=1.47]
  9%|▉         | 46/500 [00:01<00:12, 36.66it/s, failures=0, hvi=1.48]
  9%|▉         | 47/500 [00:01<00:12, 36.66it/s, failures=0, hvi=1.48]
 10%|▉         | 48/500 [00:01<00:12, 36.66it/s, failures=0, hvi=1.48]
 10%|▉         | 49/500 [00:01<00:12, 36.55it/s, failures=0, hvi=1.48]
 10%|▉         | 49/500 [00:01<00:12, 36.55it/s, failures=0, hvi=1.48]
 10%|█         | 50/500 [00:01<00:12, 36.55it/s, failures=0, hvi=1.48]
 10%|█         | 51/500 [00:01<00:12, 36.55it/s, failures=0, hvi=1.48]
 10%|█         | 52/500 [00:01<00:12, 36.55it/s, failures=0, hvi=1.48]
 11%|█         | 53/500 [00:01<00:12, 36.58it/s, failures=0, hvi=1.48]
 11%|█         | 53/500 [00:01<00:12, 36.58it/s, failures=0, hvi=1.48]
 11%|█         | 54/500 [00:01<00:12, 36.58it/s, failures=0, hvi=1.48]
 11%|█         | 55/500 [00:01<00:12, 36.58it/s, failures=0, hvi=1.48]
 11%|█         | 56/500 [00:01<00:12, 36.58it/s, failures=0, hvi=1.48]
 11%|█▏        | 57/500 [00:01<00:12, 36.59it/s, failures=0, hvi=1.48]
 11%|█▏        | 57/500 [00:01<00:12, 36.59it/s, failures=0, hvi=1.48]
 12%|█▏        | 58/500 [00:01<00:12, 36.59it/s, failures=0, hvi=1.48]
 12%|█▏        | 59/500 [00:01<00:12, 36.59it/s, failures=0, hvi=1.48]
 12%|█▏        | 60/500 [00:01<00:12, 36.59it/s, failures=0, hvi=1.48]
 12%|█▏        | 61/500 [00:01<00:11, 36.74it/s, failures=0, hvi=1.48]
 12%|█▏        | 61/500 [00:01<00:11, 36.74it/s, failures=0, hvi=1.48]
 12%|█▏        | 62/500 [00:01<00:11, 36.74it/s, failures=0, hvi=1.48]
 13%|█▎        | 63/500 [00:01<00:11, 36.74it/s, failures=0, hvi=1.49]
 13%|█▎        | 64/500 [00:01<00:11, 36.74it/s, failures=0, hvi=1.49]
 13%|█▎        | 65/500 [00:01<00:11, 36.61it/s, failures=0, hvi=1.49]
 13%|█▎        | 65/500 [00:01<00:11, 36.61it/s, failures=0, hvi=1.49]
 13%|█▎        | 66/500 [00:01<00:11, 36.61it/s, failures=0, hvi=1.49]
 13%|█▎        | 67/500 [00:01<00:11, 36.61it/s, failures=0, hvi=1.49]
 14%|█▎        | 68/500 [00:01<00:11, 36.61it/s, failures=0, hvi=1.5]
 14%|█▍        | 69/500 [00:01<00:11, 36.26it/s, failures=0, hvi=1.5]
 14%|█▍        | 69/500 [00:01<00:11, 36.26it/s, failures=0, hvi=1.5]
 14%|█▍        | 70/500 [00:01<00:11, 36.26it/s, failures=0, hvi=1.5]
 14%|█▍        | 71/500 [00:01<00:11, 36.26it/s, failures=0, hvi=1.5]
 14%|█▍        | 72/500 [00:01<00:11, 36.26it/s, failures=0, hvi=1.5]
 15%|█▍        | 73/500 [00:01<00:12, 35.33it/s, failures=0, hvi=1.5]
 15%|█▍        | 73/500 [00:01<00:12, 35.33it/s, failures=0, hvi=1.5]
 15%|█▍        | 74/500 [00:01<00:12, 35.33it/s, failures=0, hvi=1.5]
 15%|█▌        | 75/500 [00:02<00:12, 35.33it/s, failures=0, hvi=1.5]
 15%|█▌        | 76/500 [00:02<00:12, 35.33it/s, failures=0, hvi=1.5]
 15%|█▌        | 77/500 [00:02<00:12, 34.06it/s, failures=0, hvi=1.5]
 15%|█▌        | 77/500 [00:02<00:12, 34.06it/s, failures=0, hvi=1.51]
 16%|█▌        | 78/500 [00:02<00:12, 34.06it/s, failures=0, hvi=1.51]
 16%|█▌        | 79/500 [00:02<00:12, 34.06it/s, failures=0, hvi=1.51]
 16%|█▌        | 80/500 [00:02<00:12, 34.06it/s, failures=0, hvi=1.51]
 16%|█▌        | 81/500 [00:02<00:12, 33.11it/s, failures=0, hvi=1.51]
 16%|█▌        | 81/500 [00:02<00:12, 33.11it/s, failures=0, hvi=1.51]
 16%|█▋        | 82/500 [00:02<00:12, 33.11it/s, failures=0, hvi=1.51]
 17%|█▋        | 83/500 [00:02<00:12, 33.11it/s, failures=0, hvi=1.51]
 17%|█▋        | 84/500 [00:02<00:12, 33.11it/s, failures=0, hvi=1.51]
 17%|█▋        | 85/500 [00:02<00:15, 26.81it/s, failures=0, hvi=1.51]
 17%|█▋        | 85/500 [00:02<00:15, 26.81it/s, failures=0, hvi=1.51]
 17%|█▋        | 86/500 [00:02<00:15, 26.81it/s, failures=0, hvi=1.51]
 17%|█▋        | 87/500 [00:02<00:15, 26.81it/s, failures=0, hvi=1.53]
 18%|█▊        | 88/500 [00:02<00:15, 26.81it/s, failures=0, hvi=1.54]
 18%|█▊        | 89/500 [00:02<00:14, 27.99it/s, failures=0, hvi=1.54]
 18%|█▊        | 89/500 [00:02<00:14, 27.99it/s, failures=0, hvi=1.54]
 18%|█▊        | 90/500 [00:02<00:14, 27.99it/s, failures=0, hvi=1.54]
 18%|█▊        | 91/500 [00:02<00:14, 27.99it/s, failures=0, hvi=1.54]
 18%|█▊        | 92/500 [00:02<00:14, 27.99it/s, failures=0, hvi=1.54]
 19%|█▊        | 93/500 [00:02<00:14, 28.79it/s, failures=0, hvi=1.54]
 19%|█▊        | 93/500 [00:02<00:14, 28.79it/s, failures=0, hvi=1.54]
 19%|█▉        | 94/500 [00:02<00:14, 28.79it/s, failures=0, hvi=1.54]
 19%|█▉        | 95/500 [00:02<00:14, 28.79it/s, failures=0, hvi=1.6]
 19%|█▉        | 96/500 [00:02<00:14, 28.79it/s, failures=0, hvi=1.61]
 19%|█▉        | 97/500 [00:02<00:13, 29.28it/s, failures=0, hvi=1.61]
 19%|█▉        | 97/500 [00:02<00:13, 29.28it/s, failures=0, hvi=1.61]
 20%|█▉        | 98/500 [00:02<00:13, 29.28it/s, failures=0, hvi=1.61]
 20%|█▉        | 99/500 [00:02<00:13, 29.28it/s, failures=0, hvi=1.62]
 20%|██        | 100/500 [00:02<00:13, 29.28it/s, failures=0, hvi=1.62]
 20%|██        | 101/500 [00:02<00:13, 29.19it/s, failures=0, hvi=1.62]
 20%|██        | 101/500 [00:02<00:13, 29.19it/s, failures=0, hvi=1.64]
 20%|██        | 102/500 [00:02<00:13, 29.19it/s, failures=0, hvi=1.64]
 21%|██        | 103/500 [00:03<00:13, 29.19it/s, failures=0, hvi=1.64]
 21%|██        | 104/500 [00:03<00:13, 29.24it/s, failures=0, hvi=1.64]
 21%|██        | 104/500 [00:03<00:13, 29.24it/s, failures=0, hvi=1.64]
 21%|██        | 105/500 [00:03<00:13, 29.24it/s, failures=0, hvi=1.64]
 21%|██        | 106/500 [00:03<00:13, 29.24it/s, failures=0, hvi=1.64]
 21%|██▏       | 107/500 [00:03<00:13, 29.31it/s, failures=0, hvi=1.64]
 21%|██▏       | 107/500 [00:03<00:13, 29.31it/s, failures=0, hvi=1.64]
 22%|██▏       | 108/500 [00:03<00:13, 29.31it/s, failures=0, hvi=1.64]
 22%|██▏       | 109/500 [00:03<00:13, 29.31it/s, failures=0, hvi=1.65]
 22%|██▏       | 110/500 [00:03<00:13, 29.29it/s, failures=0, hvi=1.65]
 22%|██▏       | 110/500 [00:03<00:13, 29.29it/s, failures=0, hvi=1.65]
 22%|██▏       | 111/500 [00:03<00:13, 29.29it/s, failures=0, hvi=1.65]
 22%|██▏       | 112/500 [00:03<00:13, 29.29it/s, failures=0, hvi=1.65]
 23%|██▎       | 113/500 [00:03<00:13, 29.30it/s, failures=0, hvi=1.65]
 23%|██▎       | 113/500 [00:03<00:13, 29.30it/s, failures=0, hvi=1.65]
 23%|██▎       | 114/500 [00:03<00:13, 29.30it/s, failures=0, hvi=1.65]
 23%|██▎       | 115/500 [00:03<00:13, 29.30it/s, failures=0, hvi=1.83]
 23%|██▎       | 116/500 [00:03<00:13, 29.28it/s, failures=0, hvi=1.83]
 23%|██▎       | 116/500 [00:03<00:13, 29.28it/s, failures=0, hvi=1.83]
 23%|██▎       | 117/500 [00:03<00:13, 29.28it/s, failures=0, hvi=1.83]
 24%|██▎       | 118/500 [00:03<00:13, 29.28it/s, failures=0, hvi=1.83]
 24%|██▍       | 119/500 [00:03<00:13, 29.14it/s, failures=0, hvi=1.83]
 24%|██▍       | 119/500 [00:03<00:13, 29.14it/s, failures=0, hvi=1.83]
 24%|██▍       | 120/500 [00:03<00:13, 29.14it/s, failures=0, hvi=1.83]
 24%|██▍       | 121/500 [00:03<00:13, 29.14it/s, failures=0, hvi=1.83]
 24%|██▍       | 122/500 [00:03<00:13, 28.98it/s, failures=0, hvi=1.83]
 24%|██▍       | 122/500 [00:03<00:13, 28.98it/s, failures=0, hvi=1.83]
 25%|██▍       | 123/500 [00:03<00:13, 28.98it/s, failures=0, hvi=1.83]
 25%|██▍       | 124/500 [00:03<00:12, 28.98it/s, failures=0, hvi=1.83]
 25%|██▌       | 125/500 [00:03<00:13, 28.74it/s, failures=0, hvi=1.83]
 25%|██▌       | 125/500 [00:03<00:13, 28.74it/s, failures=0, hvi=1.83]
 25%|██▌       | 126/500 [00:03<00:13, 28.74it/s, failures=0, hvi=1.83]
 25%|██▌       | 127/500 [00:03<00:12, 28.74it/s, failures=0, hvi=1.83]
 26%|██▌       | 128/500 [00:03<00:13, 28.48it/s, failures=0, hvi=1.83]
 26%|██▌       | 128/500 [00:03<00:13, 28.48it/s, failures=0, hvi=1.84]
 26%|██▌       | 129/500 [00:03<00:13, 28.48it/s, failures=0, hvi=1.84]
 26%|██▌       | 130/500 [00:03<00:12, 28.48it/s, failures=0, hvi=1.84]
 26%|██▌       | 131/500 [00:04<00:13, 28.28it/s, failures=0, hvi=1.84]
 26%|██▌       | 131/500 [00:04<00:13, 28.28it/s, failures=0, hvi=1.84]
 26%|██▋       | 132/500 [00:04<00:13, 28.28it/s, failures=0, hvi=1.85]
 27%|██▋       | 133/500 [00:04<00:12, 28.28it/s, failures=0, hvi=1.85]
 27%|██▋       | 134/500 [00:04<00:12, 28.23it/s, failures=0, hvi=1.85]
 27%|██▋       | 134/500 [00:04<00:12, 28.23it/s, failures=0, hvi=1.85]
 27%|██▋       | 135/500 [00:04<00:12, 28.23it/s, failures=0, hvi=1.85]
 27%|██▋       | 136/500 [00:04<00:12, 28.23it/s, failures=0, hvi=1.85]
 27%|██▋       | 137/500 [00:04<00:12, 28.20it/s, failures=0, hvi=1.85]
 27%|██▋       | 137/500 [00:04<00:12, 28.20it/s, failures=0, hvi=1.85]
 28%|██▊       | 138/500 [00:04<00:12, 28.20it/s, failures=0, hvi=1.85]
 28%|██▊       | 139/500 [00:04<00:12, 28.20it/s, failures=0, hvi=1.86]
 28%|██▊       | 140/500 [00:04<00:12, 28.17it/s, failures=0, hvi=1.86]
 28%|██▊       | 140/500 [00:04<00:12, 28.17it/s, failures=0, hvi=1.86]
 28%|██▊       | 141/500 [00:04<00:12, 28.17it/s, failures=0, hvi=1.86]
 28%|██▊       | 142/500 [00:04<00:12, 28.17it/s, failures=0, hvi=1.86]
 29%|██▊       | 143/500 [00:04<00:12, 28.00it/s, failures=0, hvi=1.86]
 29%|██▊       | 143/500 [00:04<00:12, 28.00it/s, failures=0, hvi=1.86]
 29%|██▉       | 144/500 [00:04<00:12, 28.00it/s, failures=0, hvi=1.86]
 29%|██▉       | 145/500 [00:04<00:12, 28.00it/s, failures=0, hvi=1.86]
 29%|██▉       | 146/500 [00:04<00:12, 27.95it/s, failures=0, hvi=1.86]
 29%|██▉       | 146/500 [00:04<00:12, 27.95it/s, failures=0, hvi=1.86]
 29%|██▉       | 147/500 [00:04<00:12, 27.95it/s, failures=0, hvi=1.86]
 30%|██▉       | 148/500 [00:04<00:12, 27.95it/s, failures=0, hvi=1.86]
 30%|██▉       | 149/500 [00:04<00:12, 27.88it/s, failures=0, hvi=1.86]
 30%|██▉       | 149/500 [00:04<00:12, 27.88it/s, failures=0, hvi=1.86]
 30%|███       | 150/500 [00:04<00:12, 27.88it/s, failures=0, hvi=1.87]
 30%|███       | 151/500 [00:04<00:12, 27.88it/s, failures=0, hvi=1.87]
 30%|███       | 152/500 [00:04<00:12, 27.67it/s, failures=0, hvi=1.87]
 30%|███       | 152/500 [00:04<00:12, 27.67it/s, failures=0, hvi=1.87]
 31%|███       | 153/500 [00:04<00:12, 27.67it/s, failures=0, hvi=1.87]
 31%|███       | 154/500 [00:04<00:12, 27.67it/s, failures=0, hvi=1.87]
 31%|███       | 155/500 [00:04<00:12, 27.61it/s, failures=0, hvi=1.87]
 31%|███       | 155/500 [00:04<00:12, 27.61it/s, failures=0, hvi=1.87]
 31%|███       | 156/500 [00:04<00:12, 27.61it/s, failures=0, hvi=1.87]
 31%|███▏      | 157/500 [00:04<00:12, 27.61it/s, failures=0, hvi=1.87]
 32%|███▏      | 158/500 [00:04<00:13, 26.05it/s, failures=0, hvi=1.87]
 32%|███▏      | 158/500 [00:04<00:13, 26.05it/s, failures=0, hvi=1.87]
 32%|███▏      | 159/500 [00:05<00:13, 26.05it/s, failures=0, hvi=1.87]
 32%|███▏      | 160/500 [00:05<00:13, 26.05it/s, failures=0, hvi=1.87]
 32%|███▏      | 161/500 [00:05<00:12, 26.16it/s, failures=0, hvi=1.87]
 32%|███▏      | 161/500 [00:05<00:12, 26.16it/s, failures=0, hvi=1.87]
 32%|███▏      | 162/500 [00:05<00:12, 26.16it/s, failures=0, hvi=1.87]
 33%|███▎      | 163/500 [00:05<00:12, 26.16it/s, failures=0, hvi=1.87]
 33%|███▎      | 164/500 [00:05<00:12, 26.44it/s, failures=0, hvi=1.87]
 33%|███▎      | 164/500 [00:05<00:12, 26.44it/s, failures=0, hvi=1.87]
 33%|███▎      | 165/500 [00:05<00:12, 26.44it/s, failures=0, hvi=1.87]
 33%|███▎      | 166/500 [00:05<00:12, 26.44it/s, failures=0, hvi=1.87]
 33%|███▎      | 167/500 [00:05<00:15, 21.68it/s, failures=0, hvi=1.87]
 33%|███▎      | 167/500 [00:05<00:15, 21.68it/s, failures=0, hvi=1.87]
 34%|███▎      | 168/500 [00:05<00:15, 21.68it/s, failures=0, hvi=1.87]
 34%|███▍      | 169/500 [00:05<00:15, 21.68it/s, failures=0, hvi=1.87]
 34%|███▍      | 170/500 [00:05<00:14, 23.03it/s, failures=0, hvi=1.87]
 34%|███▍      | 170/500 [00:05<00:14, 23.03it/s, failures=0, hvi=1.87]
 34%|███▍      | 171/500 [00:05<00:14, 23.03it/s, failures=0, hvi=1.87]
 34%|███▍      | 172/500 [00:05<00:14, 23.03it/s, failures=0, hvi=1.89]
 35%|███▍      | 173/500 [00:05<00:13, 24.08it/s, failures=0, hvi=1.89]
 35%|███▍      | 173/500 [00:05<00:13, 24.08it/s, failures=0, hvi=1.89]
 35%|███▍      | 174/500 [00:05<00:13, 24.08it/s, failures=0, hvi=1.89]
 35%|███▌      | 175/500 [00:05<00:13, 24.08it/s, failures=0, hvi=1.89]
 35%|███▌      | 176/500 [00:05<00:13, 24.83it/s, failures=0, hvi=1.89]
 35%|███▌      | 176/500 [00:05<00:13, 24.83it/s, failures=0, hvi=1.89]
 35%|███▌      | 177/500 [00:05<00:13, 24.83it/s, failures=0, hvi=1.89]
 36%|███▌      | 178/500 [00:05<00:12, 24.83it/s, failures=0, hvi=1.89]
 36%|███▌      | 179/500 [00:05<00:12, 25.37it/s, failures=0, hvi=1.89]
 36%|███▌      | 179/500 [00:05<00:12, 25.37it/s, failures=0, hvi=1.89]
 36%|███▌      | 180/500 [00:05<00:12, 25.37it/s, failures=0, hvi=1.89]
 36%|███▌      | 181/500 [00:05<00:12, 25.37it/s, failures=0, hvi=1.89]
 36%|███▋      | 182/500 [00:05<00:12, 25.72it/s, failures=0, hvi=1.89]
 36%|███▋      | 182/500 [00:05<00:12, 25.72it/s, failures=0, hvi=1.89]
 37%|███▋      | 183/500 [00:06<00:12, 25.72it/s, failures=0, hvi=1.89]
 37%|███▋      | 184/500 [00:06<00:12, 25.72it/s, failures=0, hvi=1.89]
 37%|███▋      | 185/500 [00:06<00:12, 26.01it/s, failures=0, hvi=1.89]
 37%|███▋      | 185/500 [00:06<00:12, 26.01it/s, failures=0, hvi=1.89]
 37%|███▋      | 186/500 [00:06<00:12, 26.01it/s, failures=0, hvi=1.89]
 37%|███▋      | 187/500 [00:06<00:12, 26.01it/s, failures=0, hvi=1.89]
 38%|███▊      | 188/500 [00:06<00:11, 26.16it/s, failures=0, hvi=1.89]
 38%|███▊      | 188/500 [00:06<00:11, 26.16it/s, failures=0, hvi=1.89]
 38%|███▊      | 189/500 [00:06<00:11, 26.16it/s, failures=0, hvi=1.89]
 38%|███▊      | 190/500 [00:06<00:11, 26.16it/s, failures=0, hvi=1.89]
 38%|███▊      | 191/500 [00:06<00:11, 26.23it/s, failures=0, hvi=1.89]
 38%|███▊      | 191/500 [00:06<00:11, 26.23it/s, failures=0, hvi=1.89]
 38%|███▊      | 192/500 [00:06<00:11, 26.23it/s, failures=0, hvi=1.89]
 39%|███▊      | 193/500 [00:06<00:11, 26.23it/s, failures=0, hvi=1.89]
 39%|███▉      | 194/500 [00:06<00:11, 26.29it/s, failures=0, hvi=1.89]
 39%|███▉      | 194/500 [00:06<00:11, 26.29it/s, failures=0, hvi=1.89]
 39%|███▉      | 195/500 [00:06<00:11, 26.29it/s, failures=0, hvi=1.89]
 39%|███▉      | 196/500 [00:06<00:11, 26.29it/s, failures=0, hvi=1.89]
 39%|███▉      | 197/500 [00:06<00:11, 26.35it/s, failures=0, hvi=1.89]
 39%|███▉      | 197/500 [00:06<00:11, 26.35it/s, failures=0, hvi=1.89]
 40%|███▉      | 198/500 [00:06<00:11, 26.35it/s, failures=0, hvi=1.89]
 40%|███▉      | 199/500 [00:06<00:11, 26.35it/s, failures=0, hvi=1.89]
 40%|████      | 200/500 [00:06<00:11, 26.41it/s, failures=0, hvi=1.89]
 40%|████      | 200/500 [00:06<00:11, 26.41it/s, failures=0, hvi=1.89]
 40%|████      | 201/500 [00:06<00:11, 26.41it/s, failures=0, hvi=1.89]
 40%|████      | 202/500 [00:06<00:11, 26.41it/s, failures=0, hvi=1.89]
 41%|████      | 203/500 [00:06<00:11, 26.39it/s, failures=0, hvi=1.89]
 41%|████      | 203/500 [00:06<00:11, 26.39it/s, failures=0, hvi=1.89]
 41%|████      | 204/500 [00:06<00:11, 26.39it/s, failures=0, hvi=1.89]
 41%|████      | 205/500 [00:06<00:11, 26.39it/s, failures=0, hvi=1.89]
 41%|████      | 206/500 [00:06<00:11, 26.34it/s, failures=0, hvi=1.89]
 41%|████      | 206/500 [00:06<00:11, 26.34it/s, failures=0, hvi=1.89]
 41%|████▏     | 207/500 [00:06<00:11, 26.34it/s, failures=0, hvi=1.89]
 42%|████▏     | 208/500 [00:06<00:11, 26.34it/s, failures=0, hvi=1.89]
 42%|████▏     | 209/500 [00:07<00:11, 26.29it/s, failures=0, hvi=1.89]
 42%|████▏     | 209/500 [00:07<00:11, 26.29it/s, failures=0, hvi=1.89]
 42%|████▏     | 210/500 [00:07<00:11, 26.29it/s, failures=0, hvi=1.89]
 42%|████▏     | 211/500 [00:07<00:10, 26.29it/s, failures=0, hvi=1.89]
 42%|████▏     | 212/500 [00:07<00:10, 26.28it/s, failures=0, hvi=1.89]
 42%|████▏     | 212/500 [00:07<00:10, 26.28it/s, failures=0, hvi=1.89]
 43%|████▎     | 213/500 [00:07<00:10, 26.28it/s, failures=0, hvi=1.89]
 43%|████▎     | 214/500 [00:07<00:10, 26.28it/s, failures=0, hvi=1.89]
 43%|████▎     | 215/500 [00:07<00:10, 25.97it/s, failures=0, hvi=1.89]
 43%|████▎     | 215/500 [00:07<00:10, 25.97it/s, failures=0, hvi=1.89]
 43%|████▎     | 216/500 [00:07<00:10, 25.97it/s, failures=0, hvi=1.89]
 43%|████▎     | 217/500 [00:07<00:10, 25.97it/s, failures=0, hvi=1.89]
 44%|████▎     | 218/500 [00:07<00:10, 25.87it/s, failures=0, hvi=1.89]
 44%|████▎     | 218/500 [00:07<00:10, 25.87it/s, failures=0, hvi=1.89]
 44%|████▍     | 219/500 [00:07<00:10, 25.87it/s, failures=0, hvi=1.89]
 44%|████▍     | 220/500 [00:07<00:10, 25.87it/s, failures=0, hvi=1.89]
 44%|████▍     | 221/500 [00:07<00:10, 25.87it/s, failures=0, hvi=1.89]
 44%|████▍     | 221/500 [00:07<00:10, 25.87it/s, failures=0, hvi=1.89]
 44%|████▍     | 222/500 [00:07<00:10, 25.87it/s, failures=0, hvi=1.89]
 45%|████▍     | 223/500 [00:07<00:10, 25.87it/s, failures=0, hvi=1.89]
 45%|████▍     | 224/500 [00:07<00:10, 25.94it/s, failures=0, hvi=1.89]
 45%|████▍     | 224/500 [00:07<00:10, 25.94it/s, failures=0, hvi=1.89]
 45%|████▌     | 225/500 [00:07<00:10, 25.94it/s, failures=0, hvi=1.89]
 45%|████▌     | 226/500 [00:07<00:10, 25.94it/s, failures=0, hvi=1.89]
 45%|████▌     | 227/500 [00:07<00:10, 25.91it/s, failures=0, hvi=1.89]
 45%|████▌     | 227/500 [00:07<00:10, 25.91it/s, failures=0, hvi=1.89]
 46%|████▌     | 228/500 [00:07<00:10, 25.91it/s, failures=0, hvi=1.89]
 46%|████▌     | 229/500 [00:07<00:10, 25.91it/s, failures=0, hvi=1.89]
 46%|████▌     | 230/500 [00:07<00:10, 25.85it/s, failures=0, hvi=1.89]
 46%|████▌     | 230/500 [00:07<00:10, 25.85it/s, failures=0, hvi=1.89]
 46%|████▌     | 231/500 [00:07<00:10, 25.85it/s, failures=0, hvi=1.89]
 46%|████▋     | 232/500 [00:07<00:10, 25.85it/s, failures=0, hvi=1.89]
 47%|████▋     | 233/500 [00:08<00:12, 21.06it/s, failures=0, hvi=1.89]
 47%|████▋     | 233/500 [00:08<00:12, 21.06it/s, failures=0, hvi=1.89]
 47%|████▋     | 234/500 [00:08<00:12, 21.06it/s, failures=0, hvi=1.9]
 47%|████▋     | 235/500 [00:08<00:12, 21.06it/s, failures=0, hvi=1.9]
 47%|████▋     | 236/500 [00:08<00:11, 22.28it/s, failures=0, hvi=1.9]
 47%|████▋     | 236/500 [00:08<00:11, 22.28it/s, failures=0, hvi=1.9]
 47%|████▋     | 237/500 [00:08<00:11, 22.28it/s, failures=0, hvi=1.9]
 48%|████▊     | 238/500 [00:08<00:11, 22.28it/s, failures=0, hvi=1.9]
 48%|████▊     | 239/500 [00:08<00:11, 23.20it/s, failures=0, hvi=1.9]
 48%|████▊     | 239/500 [00:08<00:11, 23.20it/s, failures=0, hvi=1.9]
 48%|████▊     | 240/500 [00:08<00:11, 23.20it/s, failures=0, hvi=1.9]
 48%|████▊     | 241/500 [00:08<00:11, 23.20it/s, failures=0, hvi=1.9]
 48%|████▊     | 242/500 [00:08<00:10, 23.91it/s, failures=0, hvi=1.9]
 48%|████▊     | 242/500 [00:08<00:10, 23.91it/s, failures=0, hvi=1.9]
 49%|████▊     | 243/500 [00:08<00:10, 23.91it/s, failures=0, hvi=1.9]
 49%|████▉     | 244/500 [00:08<00:10, 23.91it/s, failures=0, hvi=1.9]
 49%|████▉     | 245/500 [00:08<00:10, 24.31it/s, failures=0, hvi=1.9]
 49%|████▉     | 245/500 [00:08<00:10, 24.31it/s, failures=0, hvi=1.9]
 49%|████▉     | 246/500 [00:08<00:10, 24.31it/s, failures=0, hvi=1.9]
 49%|████▉     | 247/500 [00:08<00:10, 24.31it/s, failures=0, hvi=1.9]
 50%|████▉     | 248/500 [00:08<00:10, 24.69it/s, failures=0, hvi=1.9]
 50%|████▉     | 248/500 [00:08<00:10, 24.69it/s, failures=0, hvi=1.9]
 50%|████▉     | 249/500 [00:08<00:10, 24.69it/s, failures=0, hvi=1.9]
 50%|█████     | 250/500 [00:08<00:10, 24.69it/s, failures=0, hvi=1.9]
 50%|█████     | 251/500 [00:08<00:10, 24.90it/s, failures=0, hvi=1.9]
 50%|█████     | 251/500 [00:08<00:10, 24.90it/s, failures=0, hvi=1.9]
 50%|█████     | 252/500 [00:08<00:09, 24.90it/s, failures=0, hvi=1.9]
 51%|█████     | 253/500 [00:08<00:09, 24.90it/s, failures=0, hvi=1.9]
 51%|█████     | 254/500 [00:08<00:09, 25.01it/s, failures=0, hvi=1.9]
 51%|█████     | 254/500 [00:08<00:09, 25.01it/s, failures=0, hvi=1.9]
 51%|█████     | 255/500 [00:08<00:09, 25.01it/s, failures=0, hvi=1.9]
 51%|█████     | 256/500 [00:08<00:09, 25.01it/s, failures=0, hvi=1.9]
 51%|█████▏    | 257/500 [00:08<00:09, 25.09it/s, failures=0, hvi=1.9]
 51%|█████▏    | 257/500 [00:08<00:09, 25.09it/s, failures=0, hvi=1.9]
 52%|█████▏    | 258/500 [00:08<00:09, 25.09it/s, failures=0, hvi=1.9]
 52%|█████▏    | 259/500 [00:09<00:09, 25.09it/s, failures=0, hvi=1.9]
 52%|█████▏    | 260/500 [00:09<00:09, 25.18it/s, failures=0, hvi=1.9]
 52%|█████▏    | 260/500 [00:09<00:09, 25.18it/s, failures=0, hvi=1.9]
 52%|█████▏    | 261/500 [00:09<00:09, 25.18it/s, failures=0, hvi=1.9]
 52%|█████▏    | 262/500 [00:09<00:09, 25.18it/s, failures=0, hvi=1.9]
 53%|█████▎    | 263/500 [00:09<00:09, 25.17it/s, failures=0, hvi=1.9]
 53%|█████▎    | 263/500 [00:09<00:09, 25.17it/s, failures=0, hvi=1.9]
 53%|█████▎    | 264/500 [00:09<00:09, 25.17it/s, failures=0, hvi=1.9]
 53%|█████▎    | 265/500 [00:09<00:09, 25.17it/s, failures=0, hvi=1.9]
 53%|█████▎    | 266/500 [00:09<00:09, 25.05it/s, failures=0, hvi=1.9]
 53%|█████▎    | 266/500 [00:09<00:09, 25.05it/s, failures=0, hvi=1.9]
 53%|█████▎    | 267/500 [00:09<00:09, 25.05it/s, failures=0, hvi=1.9]
 54%|█████▎    | 268/500 [00:09<00:09, 25.05it/s, failures=0, hvi=1.9]
 54%|█████▍    | 269/500 [00:09<00:09, 24.87it/s, failures=0, hvi=1.9]
 54%|█████▍    | 269/500 [00:09<00:09, 24.87it/s, failures=0, hvi=1.9]
 54%|█████▍    | 270/500 [00:09<00:09, 24.87it/s, failures=0, hvi=1.9]
 54%|█████▍    | 271/500 [00:09<00:09, 24.87it/s, failures=0, hvi=1.91]
 54%|█████▍    | 272/500 [00:09<00:09, 24.99it/s, failures=0, hvi=1.91]
 54%|█████▍    | 272/500 [00:09<00:09, 24.99it/s, failures=0, hvi=1.91]
 55%|█████▍    | 273/500 [00:09<00:09, 24.99it/s, failures=0, hvi=1.91]
 55%|█████▍    | 274/500 [00:09<00:09, 24.99it/s, failures=0, hvi=1.91]
 55%|█████▌    | 275/500 [00:09<00:08, 25.00it/s, failures=0, hvi=1.91]
 55%|█████▌    | 275/500 [00:09<00:08, 25.00it/s, failures=0, hvi=1.91]
 55%|█████▌    | 276/500 [00:09<00:08, 25.00it/s, failures=0, hvi=1.91]
 55%|█████▌    | 277/500 [00:09<00:08, 25.00it/s, failures=0, hvi=1.91]
 56%|█████▌    | 278/500 [00:09<00:08, 25.02it/s, failures=0, hvi=1.91]
 56%|█████▌    | 278/500 [00:09<00:08, 25.02it/s, failures=0, hvi=1.91]
 56%|█████▌    | 279/500 [00:09<00:08, 25.02it/s, failures=0, hvi=1.91]
 56%|█████▌    | 280/500 [00:09<00:08, 25.02it/s, failures=0, hvi=1.91]
 56%|█████▌    | 281/500 [00:09<00:08, 25.01it/s, failures=0, hvi=1.91]
 56%|█████▌    | 281/500 [00:09<00:08, 25.01it/s, failures=0, hvi=1.91]
 56%|█████▋    | 282/500 [00:09<00:08, 25.01it/s, failures=0, hvi=1.91]
 57%|█████▋    | 283/500 [00:09<00:08, 25.01it/s, failures=0, hvi=1.91]
 57%|█████▋    | 284/500 [00:10<00:08, 25.05it/s, failures=0, hvi=1.91]
 57%|█████▋    | 284/500 [00:10<00:08, 25.05it/s, failures=0, hvi=1.91]
 57%|█████▋    | 285/500 [00:10<00:08, 25.05it/s, failures=0, hvi=1.91]
 57%|█████▋    | 286/500 [00:10<00:08, 25.05it/s, failures=0, hvi=1.91]
 57%|█████▋    | 287/500 [00:10<00:08, 25.00it/s, failures=0, hvi=1.91]
 57%|█████▋    | 287/500 [00:10<00:08, 25.00it/s, failures=0, hvi=1.91]
 58%|█████▊    | 288/500 [00:10<00:08, 25.00it/s, failures=0, hvi=1.91]
 58%|█████▊    | 289/500 [00:10<00:08, 25.00it/s, failures=0, hvi=1.91]
 58%|█████▊    | 290/500 [00:10<00:08, 24.90it/s, failures=0, hvi=1.91]
 58%|█████▊    | 290/500 [00:10<00:08, 24.90it/s, failures=0, hvi=1.91]
 58%|█████▊    | 291/500 [00:10<00:08, 24.90it/s, failures=0, hvi=1.91]
 58%|█████▊    | 292/500 [00:10<00:08, 24.90it/s, failures=0, hvi=1.91]
 59%|█████▊    | 293/500 [00:10<00:08, 24.85it/s, failures=0, hvi=1.91]
 59%|█████▊    | 293/500 [00:10<00:08, 24.85it/s, failures=0, hvi=1.91]
 59%|█████▉    | 294/500 [00:10<00:08, 24.85it/s, failures=0, hvi=1.91]
 59%|█████▉    | 295/500 [00:10<00:08, 24.85it/s, failures=0, hvi=1.91]
 59%|█████▉    | 296/500 [00:10<00:08, 24.83it/s, failures=0, hvi=1.91]
 59%|█████▉    | 296/500 [00:10<00:08, 24.83it/s, failures=0, hvi=1.91]
 59%|█████▉    | 297/500 [00:10<00:08, 24.83it/s, failures=0, hvi=1.91]
 60%|█████▉    | 298/500 [00:10<00:08, 24.83it/s, failures=0, hvi=1.91]
 60%|█████▉    | 299/500 [00:10<00:08, 24.82it/s, failures=0, hvi=1.91]
 60%|█████▉    | 299/500 [00:10<00:08, 24.82it/s, failures=0, hvi=1.91]
 60%|██████    | 300/500 [00:10<00:08, 24.82it/s, failures=0, hvi=1.91]
 60%|██████    | 301/500 [00:10<00:08, 24.82it/s, failures=0, hvi=1.91]
 60%|██████    | 302/500 [00:10<00:08, 24.74it/s, failures=0, hvi=1.91]
 60%|██████    | 302/500 [00:10<00:08, 24.74it/s, failures=0, hvi=1.91]
 61%|██████    | 303/500 [00:10<00:07, 24.74it/s, failures=0, hvi=1.91]
 61%|██████    | 304/500 [00:10<00:07, 24.74it/s, failures=0, hvi=1.91]
 61%|██████    | 305/500 [00:10<00:07, 24.70it/s, failures=0, hvi=1.91]
 61%|██████    | 305/500 [00:10<00:07, 24.70it/s, failures=0, hvi=1.91]
 61%|██████    | 306/500 [00:10<00:07, 24.70it/s, failures=0, hvi=1.91]
 61%|██████▏   | 307/500 [00:10<00:07, 24.70it/s, failures=0, hvi=1.91]
 62%|██████▏   | 308/500 [00:11<00:07, 24.69it/s, failures=0, hvi=1.91]
 62%|██████▏   | 308/500 [00:11<00:07, 24.69it/s, failures=0, hvi=1.91]
 62%|██████▏   | 309/500 [00:11<00:07, 24.69it/s, failures=0, hvi=1.91]
 62%|██████▏   | 310/500 [00:11<00:07, 24.69it/s, failures=0, hvi=1.91]
 62%|██████▏   | 311/500 [00:11<00:07, 24.70it/s, failures=0, hvi=1.91]
 62%|██████▏   | 311/500 [00:11<00:07, 24.70it/s, failures=0, hvi=1.91]
 62%|██████▏   | 312/500 [00:11<00:07, 24.70it/s, failures=0, hvi=1.91]
 63%|██████▎   | 313/500 [00:11<00:07, 24.70it/s, failures=0, hvi=1.92]
 63%|██████▎   | 314/500 [00:11<00:07, 24.66it/s, failures=0, hvi=1.92]
 63%|██████▎   | 314/500 [00:11<00:07, 24.66it/s, failures=0, hvi=1.92]
 63%|██████▎   | 315/500 [00:11<00:07, 24.66it/s, failures=0, hvi=1.92]
 63%|██████▎   | 316/500 [00:11<00:07, 24.66it/s, failures=0, hvi=1.92]
 63%|██████▎   | 317/500 [00:11<00:09, 20.04it/s, failures=0, hvi=1.92]
 63%|██████▎   | 317/500 [00:11<00:09, 20.04it/s, failures=0, hvi=1.92]
 64%|██████▎   | 318/500 [00:11<00:09, 20.04it/s, failures=0, hvi=1.92]
 64%|██████▍   | 319/500 [00:11<00:09, 20.04it/s, failures=0, hvi=1.92]
 64%|██████▍   | 320/500 [00:11<00:08, 21.14it/s, failures=0, hvi=1.92]
 64%|██████▍   | 320/500 [00:11<00:08, 21.14it/s, failures=0, hvi=1.92]
 64%|██████▍   | 321/500 [00:11<00:08, 21.14it/s, failures=0, hvi=1.92]
 64%|██████▍   | 322/500 [00:11<00:08, 21.14it/s, failures=0, hvi=1.92]
 65%|██████▍   | 323/500 [00:11<00:08, 21.97it/s, failures=0, hvi=1.92]
 65%|██████▍   | 323/500 [00:11<00:08, 21.97it/s, failures=0, hvi=1.92]
 65%|██████▍   | 324/500 [00:11<00:08, 21.97it/s, failures=0, hvi=1.92]
 65%|██████▌   | 325/500 [00:11<00:07, 21.97it/s, failures=0, hvi=1.92]
 65%|██████▌   | 326/500 [00:11<00:07, 22.64it/s, failures=0, hvi=1.92]
 65%|██████▌   | 326/500 [00:11<00:07, 22.64it/s, failures=0, hvi=1.92]
 65%|██████▌   | 327/500 [00:11<00:07, 22.64it/s, failures=0, hvi=1.92]
 66%|██████▌   | 328/500 [00:11<00:07, 22.64it/s, failures=0, hvi=1.92]
 66%|██████▌   | 329/500 [00:11<00:07, 23.06it/s, failures=0, hvi=1.92]
 66%|██████▌   | 329/500 [00:11<00:07, 23.06it/s, failures=0, hvi=1.92]
 66%|██████▌   | 330/500 [00:12<00:07, 23.06it/s, failures=0, hvi=1.92]
 66%|██████▌   | 331/500 [00:12<00:07, 23.06it/s, failures=0, hvi=1.92]
 66%|██████▋   | 332/500 [00:12<00:07, 23.43it/s, failures=0, hvi=1.92]
 66%|██████▋   | 332/500 [00:12<00:07, 23.43it/s, failures=0, hvi=1.92]
 67%|██████▋   | 333/500 [00:12<00:07, 23.43it/s, failures=0, hvi=1.92]
 67%|██████▋   | 334/500 [00:12<00:07, 23.43it/s, failures=0, hvi=1.92]
 67%|██████▋   | 335/500 [00:12<00:06, 23.65it/s, failures=0, hvi=1.92]
 67%|██████▋   | 335/500 [00:12<00:06, 23.65it/s, failures=0, hvi=1.92]
 67%|██████▋   | 336/500 [00:12<00:06, 23.65it/s, failures=0, hvi=1.92]
 67%|██████▋   | 337/500 [00:12<00:06, 23.65it/s, failures=0, hvi=1.92]
 68%|██████▊   | 338/500 [00:12<00:06, 23.77it/s, failures=0, hvi=1.92]
 68%|██████▊   | 338/500 [00:12<00:06, 23.77it/s, failures=0, hvi=1.92]
 68%|██████▊   | 339/500 [00:12<00:06, 23.77it/s, failures=0, hvi=1.92]
 68%|██████▊   | 340/500 [00:12<00:06, 23.77it/s, failures=0, hvi=1.92]
 68%|██████▊   | 341/500 [00:12<00:06, 23.79it/s, failures=0, hvi=1.92]
 68%|██████▊   | 341/500 [00:12<00:06, 23.79it/s, failures=0, hvi=1.93]
 68%|██████▊   | 342/500 [00:12<00:06, 23.79it/s, failures=0, hvi=1.93]
 69%|██████▊   | 343/500 [00:12<00:06, 23.79it/s, failures=0, hvi=1.93]
 69%|██████▉   | 344/500 [00:12<00:06, 23.91it/s, failures=0, hvi=1.93]
 69%|██████▉   | 344/500 [00:12<00:06, 23.91it/s, failures=0, hvi=1.93]
 69%|██████▉   | 345/500 [00:12<00:06, 23.91it/s, failures=0, hvi=1.93]
 69%|██████▉   | 346/500 [00:12<00:06, 23.91it/s, failures=0, hvi=1.93]
 69%|██████▉   | 347/500 [00:12<00:06, 23.92it/s, failures=0, hvi=1.93]
 69%|██████▉   | 347/500 [00:12<00:06, 23.92it/s, failures=0, hvi=1.93]
 70%|██████▉   | 348/500 [00:12<00:06, 23.92it/s, failures=0, hvi=1.93]
 70%|██████▉   | 349/500 [00:12<00:06, 23.92it/s, failures=0, hvi=1.93]
 70%|███████   | 350/500 [00:12<00:06, 23.87it/s, failures=0, hvi=1.93]
 70%|███████   | 350/500 [00:12<00:06, 23.87it/s, failures=0, hvi=1.93]
 70%|███████   | 351/500 [00:12<00:06, 23.87it/s, failures=0, hvi=1.93]
 70%|███████   | 352/500 [00:12<00:06, 23.87it/s, failures=0, hvi=1.93]
 71%|███████   | 353/500 [00:12<00:06, 23.81it/s, failures=0, hvi=1.93]
 71%|███████   | 353/500 [00:12<00:06, 23.81it/s, failures=0, hvi=1.93]
 71%|███████   | 354/500 [00:13<00:06, 23.81it/s, failures=0, hvi=1.93]
 71%|███████   | 355/500 [00:13<00:06, 23.81it/s, failures=0, hvi=1.93]
 71%|███████   | 356/500 [00:13<00:06, 23.85it/s, failures=0, hvi=1.93]
 71%|███████   | 356/500 [00:13<00:06, 23.85it/s, failures=0, hvi=1.93]
 71%|███████▏  | 357/500 [00:13<00:05, 23.85it/s, failures=0, hvi=1.93]
 72%|███████▏  | 358/500 [00:13<00:05, 23.85it/s, failures=0, hvi=1.93]
 72%|███████▏  | 359/500 [00:13<00:05, 23.67it/s, failures=0, hvi=1.93]
 72%|███████▏  | 359/500 [00:13<00:05, 23.67it/s, failures=0, hvi=1.93]
 72%|███████▏  | 360/500 [00:13<00:05, 23.67it/s, failures=0, hvi=1.93]
 72%|███████▏  | 361/500 [00:13<00:05, 23.67it/s, failures=0, hvi=1.93]
 72%|███████▏  | 362/500 [00:13<00:06, 22.81it/s, failures=0, hvi=1.93]
 72%|███████▏  | 362/500 [00:13<00:06, 22.81it/s, failures=0, hvi=1.93]
 73%|███████▎  | 363/500 [00:13<00:06, 22.81it/s, failures=0, hvi=1.93]
 73%|███████▎  | 364/500 [00:13<00:05, 22.81it/s, failures=0, hvi=1.93]
 73%|███████▎  | 365/500 [00:13<00:05, 22.91it/s, failures=0, hvi=1.93]
 73%|███████▎  | 365/500 [00:13<00:05, 22.91it/s, failures=0, hvi=1.93]
 73%|███████▎  | 366/500 [00:13<00:05, 22.91it/s, failures=0, hvi=1.93]
 73%|███████▎  | 367/500 [00:13<00:05, 22.91it/s, failures=0, hvi=1.93]
 74%|███████▎  | 368/500 [00:13<00:05, 22.91it/s, failures=0, hvi=1.93]
 74%|███████▎  | 368/500 [00:13<00:05, 22.91it/s, failures=0, hvi=1.93]
 74%|███████▍  | 369/500 [00:13<00:05, 22.91it/s, failures=0, hvi=1.93]
 74%|███████▍  | 370/500 [00:13<00:05, 22.91it/s, failures=0, hvi=1.93]
 74%|███████▍  | 371/500 [00:13<00:05, 23.03it/s, failures=0, hvi=1.93]
 74%|███████▍  | 371/500 [00:13<00:05, 23.03it/s, failures=0, hvi=1.93]
 74%|███████▍  | 372/500 [00:13<00:05, 23.03it/s, failures=0, hvi=1.93]
 75%|███████▍  | 373/500 [00:13<00:05, 23.03it/s, failures=0, hvi=1.93]
 75%|███████▍  | 374/500 [00:13<00:05, 23.13it/s, failures=0, hvi=1.93]
 75%|███████▍  | 374/500 [00:13<00:05, 23.13it/s, failures=0, hvi=1.93]
 75%|███████▌  | 375/500 [00:13<00:05, 23.13it/s, failures=0, hvi=1.93]
 75%|███████▌  | 376/500 [00:13<00:05, 23.13it/s, failures=0, hvi=1.93]
 75%|███████▌  | 377/500 [00:14<00:05, 23.22it/s, failures=0, hvi=1.93]
 75%|███████▌  | 377/500 [00:14<00:05, 23.22it/s, failures=0, hvi=1.93]
 76%|███████▌  | 378/500 [00:14<00:05, 23.22it/s, failures=0, hvi=1.93]
 76%|███████▌  | 379/500 [00:14<00:05, 23.22it/s, failures=0, hvi=1.93]
 76%|███████▌  | 380/500 [00:14<00:05, 23.34it/s, failures=0, hvi=1.93]
 76%|███████▌  | 380/500 [00:14<00:05, 23.34it/s, failures=0, hvi=1.93]
 76%|███████▌  | 381/500 [00:14<00:05, 23.34it/s, failures=0, hvi=1.93]
 76%|███████▋  | 382/500 [00:14<00:05, 23.34it/s, failures=0, hvi=1.93]
 77%|███████▋  | 383/500 [00:14<00:05, 23.29it/s, failures=0, hvi=1.93]
 77%|███████▋  | 383/500 [00:14<00:05, 23.29it/s, failures=0, hvi=1.93]
 77%|███████▋  | 384/500 [00:14<00:04, 23.29it/s, failures=0, hvi=1.93]
 77%|███████▋  | 385/500 [00:14<00:04, 23.29it/s, failures=0, hvi=1.93]
 77%|███████▋  | 386/500 [00:14<00:05, 19.38it/s, failures=0, hvi=1.93]
 77%|███████▋  | 386/500 [00:14<00:05, 19.38it/s, failures=0, hvi=1.93]
 77%|███████▋  | 387/500 [00:14<00:05, 19.38it/s, failures=0, hvi=1.93]
 78%|███████▊  | 388/500 [00:14<00:05, 19.38it/s, failures=0, hvi=1.93]
 78%|███████▊  | 389/500 [00:14<00:05, 20.45it/s, failures=0, hvi=1.93]
 78%|███████▊  | 389/500 [00:14<00:05, 20.45it/s, failures=0, hvi=1.93]
 78%|███████▊  | 390/500 [00:14<00:05, 20.45it/s, failures=0, hvi=1.93]
 78%|███████▊  | 391/500 [00:14<00:05, 20.45it/s, failures=0, hvi=1.93]
 78%|███████▊  | 392/500 [00:14<00:05, 21.24it/s, failures=0, hvi=1.93]
 78%|███████▊  | 392/500 [00:14<00:05, 21.24it/s, failures=0, hvi=1.93]
 79%|███████▊  | 393/500 [00:14<00:05, 21.24it/s, failures=0, hvi=1.93]
 79%|███████▉  | 394/500 [00:14<00:04, 21.24it/s, failures=0, hvi=1.93]
 79%|███████▉  | 395/500 [00:14<00:04, 21.77it/s, failures=0, hvi=1.93]
 79%|███████▉  | 395/500 [00:14<00:04, 21.77it/s, failures=0, hvi=1.93]
 79%|███████▉  | 396/500 [00:14<00:04, 21.77it/s, failures=0, hvi=1.93]
 79%|███████▉  | 397/500 [00:14<00:04, 21.77it/s, failures=0, hvi=1.93]
 80%|███████▉  | 398/500 [00:14<00:04, 22.12it/s, failures=0, hvi=1.93]
 80%|███████▉  | 398/500 [00:14<00:04, 22.12it/s, failures=0, hvi=1.93]
 80%|███████▉  | 399/500 [00:15<00:04, 22.12it/s, failures=0, hvi=1.93]
 80%|████████  | 400/500 [00:15<00:04, 22.12it/s, failures=0, hvi=1.93]
 80%|████████  | 401/500 [00:15<00:04, 22.46it/s, failures=0, hvi=1.93]
 80%|████████  | 401/500 [00:15<00:04, 22.46it/s, failures=0, hvi=1.93]
 80%|████████  | 402/500 [00:15<00:04, 22.46it/s, failures=0, hvi=1.93]
 81%|████████  | 403/500 [00:15<00:04, 22.46it/s, failures=0, hvi=1.93]
 81%|████████  | 404/500 [00:15<00:04, 22.66it/s, failures=0, hvi=1.93]
 81%|████████  | 404/500 [00:15<00:04, 22.66it/s, failures=0, hvi=1.93]
 81%|████████  | 405/500 [00:15<00:04, 22.66it/s, failures=0, hvi=1.93]
 81%|████████  | 406/500 [00:15<00:04, 22.66it/s, failures=0, hvi=1.93]
 81%|████████▏ | 407/500 [00:15<00:04, 22.79it/s, failures=0, hvi=1.93]
 81%|████████▏ | 407/500 [00:15<00:04, 22.79it/s, failures=0, hvi=1.93]
 82%|████████▏ | 408/500 [00:15<00:04, 22.79it/s, failures=0, hvi=1.93]
 82%|████████▏ | 409/500 [00:15<00:03, 22.79it/s, failures=0, hvi=1.93]
 82%|████████▏ | 410/500 [00:15<00:03, 22.86it/s, failures=0, hvi=1.93]
 82%|████████▏ | 410/500 [00:15<00:03, 22.86it/s, failures=0, hvi=1.93]
 82%|████████▏ | 411/500 [00:15<00:03, 22.86it/s, failures=0, hvi=1.94]
 82%|████████▏ | 412/500 [00:15<00:03, 22.86it/s, failures=0, hvi=1.94]
 83%|████████▎ | 413/500 [00:15<00:03, 22.95it/s, failures=0, hvi=1.94]
 83%|████████▎ | 413/500 [00:15<00:03, 22.95it/s, failures=0, hvi=1.94]
 83%|████████▎ | 414/500 [00:15<00:03, 22.95it/s, failures=0, hvi=1.94]
 83%|████████▎ | 415/500 [00:15<00:03, 22.95it/s, failures=0, hvi=1.94]
 83%|████████▎ | 416/500 [00:15<00:03, 22.81it/s, failures=0, hvi=1.94]
 83%|████████▎ | 416/500 [00:15<00:03, 22.81it/s, failures=0, hvi=1.94]
 83%|████████▎ | 417/500 [00:15<00:03, 22.81it/s, failures=0, hvi=1.94]
 84%|████████▎ | 418/500 [00:15<00:03, 22.81it/s, failures=0, hvi=1.94]
 84%|████████▍ | 419/500 [00:15<00:03, 22.80it/s, failures=0, hvi=1.94]
 84%|████████▍ | 419/500 [00:15<00:03, 22.80it/s, failures=0, hvi=1.94]
 84%|████████▍ | 420/500 [00:15<00:03, 22.80it/s, failures=0, hvi=1.94]
 84%|████████▍ | 421/500 [00:15<00:03, 22.80it/s, failures=0, hvi=1.94]
 84%|████████▍ | 422/500 [00:16<00:03, 22.80it/s, failures=0, hvi=1.94]
 84%|████████▍ | 422/500 [00:16<00:03, 22.80it/s, failures=0, hvi=1.94]
 85%|████████▍ | 423/500 [00:16<00:03, 22.80it/s, failures=0, hvi=1.94]
 85%|████████▍ | 424/500 [00:16<00:03, 22.80it/s, failures=0, hvi=2.07]
 85%|████████▌ | 425/500 [00:16<00:03, 22.83it/s, failures=0, hvi=2.07]
 85%|████████▌ | 425/500 [00:16<00:03, 22.83it/s, failures=0, hvi=2.07]
 85%|████████▌ | 426/500 [00:16<00:03, 22.83it/s, failures=0, hvi=2.07]
 85%|████████▌ | 427/500 [00:16<00:03, 22.83it/s, failures=0, hvi=2.07]
 86%|████████▌ | 428/500 [00:16<00:03, 22.72it/s, failures=0, hvi=2.07]
 86%|████████▌ | 428/500 [00:16<00:03, 22.72it/s, failures=0, hvi=2.07]
 86%|████████▌ | 429/500 [00:16<00:03, 22.72it/s, failures=0, hvi=2.07]
 86%|████████▌ | 430/500 [00:16<00:03, 22.72it/s, failures=0, hvi=2.07]
 86%|████████▌ | 431/500 [00:16<00:03, 22.73it/s, failures=0, hvi=2.07]
 86%|████████▌ | 431/500 [00:16<00:03, 22.73it/s, failures=0, hvi=2.07]
 86%|████████▋ | 432/500 [00:16<00:02, 22.73it/s, failures=0, hvi=2.07]
 87%|████████▋ | 433/500 [00:16<00:02, 22.73it/s, failures=0, hvi=2.07]
 87%|████████▋ | 434/500 [00:16<00:02, 22.76it/s, failures=0, hvi=2.07]
 87%|████████▋ | 434/500 [00:16<00:02, 22.76it/s, failures=0, hvi=2.07]
 87%|████████▋ | 435/500 [00:16<00:02, 22.76it/s, failures=0, hvi=2.07]
 87%|████████▋ | 436/500 [00:16<00:02, 22.76it/s, failures=0, hvi=2.07]
 87%|████████▋ | 437/500 [00:16<00:02, 22.73it/s, failures=0, hvi=2.07]
 87%|████████▋ | 437/500 [00:16<00:02, 22.73it/s, failures=0, hvi=2.07]
 88%|████████▊ | 438/500 [00:16<00:02, 22.73it/s, failures=0, hvi=2.07]
 88%|████████▊ | 439/500 [00:16<00:02, 22.73it/s, failures=0, hvi=2.07]
 88%|████████▊ | 440/500 [00:16<00:02, 22.67it/s, failures=0, hvi=2.07]
 88%|████████▊ | 440/500 [00:16<00:02, 22.67it/s, failures=0, hvi=2.07]
 88%|████████▊ | 441/500 [00:16<00:02, 22.67it/s, failures=0, hvi=2.07]
 88%|████████▊ | 442/500 [00:16<00:02, 22.67it/s, failures=0, hvi=2.07]
 89%|████████▊ | 443/500 [00:16<00:02, 22.61it/s, failures=0, hvi=2.07]
 89%|████████▊ | 443/500 [00:16<00:02, 22.61it/s, failures=0, hvi=2.07]
 89%|████████▉ | 444/500 [00:17<00:02, 22.61it/s, failures=0, hvi=2.07]
 89%|████████▉ | 445/500 [00:17<00:02, 22.61it/s, failures=0, hvi=2.07]
 89%|████████▉ | 446/500 [00:17<00:02, 22.62it/s, failures=0, hvi=2.07]
 89%|████████▉ | 446/500 [00:17<00:02, 22.62it/s, failures=0, hvi=2.07]
 89%|████████▉ | 447/500 [00:17<00:02, 22.62it/s, failures=0, hvi=2.07]
 90%|████████▉ | 448/500 [00:17<00:02, 22.62it/s, failures=0, hvi=2.07]
 90%|████████▉ | 449/500 [00:17<00:02, 22.59it/s, failures=0, hvi=2.07]
 90%|████████▉ | 449/500 [00:17<00:02, 22.59it/s, failures=0, hvi=2.07]
 90%|█████████ | 450/500 [00:17<00:02, 22.59it/s, failures=0, hvi=2.07]
 90%|█████████ | 451/500 [00:17<00:02, 22.59it/s, failures=0, hvi=2.07]
 90%|█████████ | 452/500 [00:17<00:02, 22.50it/s, failures=0, hvi=2.07]
 90%|█████████ | 452/500 [00:17<00:02, 22.50it/s, failures=0, hvi=2.07]
 91%|█████████ | 453/500 [00:17<00:02, 22.50it/s, failures=0, hvi=2.07]
 91%|█████████ | 454/500 [00:17<00:02, 22.50it/s, failures=0, hvi=2.07]
 91%|█████████ | 455/500 [00:17<00:02, 22.45it/s, failures=0, hvi=2.07]
 91%|█████████ | 455/500 [00:17<00:02, 22.45it/s, failures=0, hvi=2.07]
 91%|█████████ | 456/500 [00:17<00:01, 22.45it/s, failures=0, hvi=2.07]
 91%|█████████▏| 457/500 [00:17<00:01, 22.45it/s, failures=0, hvi=2.07]
 92%|█████████▏| 458/500 [00:17<00:01, 22.35it/s, failures=0, hvi=2.07]
 92%|█████████▏| 458/500 [00:17<00:01, 22.35it/s, failures=0, hvi=2.07]
 92%|█████████▏| 459/500 [00:17<00:01, 22.35it/s, failures=0, hvi=2.07]
 92%|█████████▏| 460/500 [00:17<00:01, 22.35it/s, failures=0, hvi=2.07]
 92%|█████████▏| 461/500 [00:17<00:01, 22.14it/s, failures=0, hvi=2.07]
 92%|█████████▏| 461/500 [00:17<00:01, 22.14it/s, failures=0, hvi=2.07]
 92%|█████████▏| 462/500 [00:17<00:01, 22.14it/s, failures=0, hvi=2.07]
 93%|█████████▎| 463/500 [00:17<00:01, 22.14it/s, failures=0, hvi=2.07]
 93%|█████████▎| 464/500 [00:17<00:01, 22.00it/s, failures=0, hvi=2.07]
 93%|█████████▎| 464/500 [00:17<00:01, 22.00it/s, failures=0, hvi=2.07]
 93%|█████████▎| 465/500 [00:17<00:01, 22.00it/s, failures=0, hvi=2.07]
 93%|█████████▎| 466/500 [00:18<00:01, 22.00it/s, failures=0, hvi=2.07]
 93%|█████████▎| 467/500 [00:18<00:01, 21.79it/s, failures=0, hvi=2.07]
 93%|█████████▎| 467/500 [00:18<00:01, 21.79it/s, failures=0, hvi=2.08]
 94%|█████████▎| 468/500 [00:18<00:01, 21.79it/s, failures=0, hvi=2.08]
 94%|█████████▍| 469/500 [00:18<00:01, 21.79it/s, failures=0, hvi=2.08]
 94%|█████████▍| 470/500 [00:18<00:01, 18.35it/s, failures=0, hvi=2.08]
 94%|█████████▍| 470/500 [00:18<00:01, 18.35it/s, failures=0, hvi=2.08]
 94%|█████████▍| 471/500 [00:18<00:01, 18.35it/s, failures=0, hvi=2.08]
 94%|█████████▍| 472/500 [00:18<00:01, 18.35it/s, failures=0, hvi=2.08]
 95%|█████████▍| 473/500 [00:18<00:01, 19.37it/s, failures=0, hvi=2.08]
 95%|█████████▍| 473/500 [00:18<00:01, 19.37it/s, failures=0, hvi=2.08]
 95%|█████████▍| 474/500 [00:18<00:01, 19.37it/s, failures=0, hvi=2.08]
 95%|█████████▌| 475/500 [00:18<00:01, 19.37it/s, failures=0, hvi=2.08]
 95%|█████████▌| 476/500 [00:18<00:01, 20.15it/s, failures=0, hvi=2.08]
 95%|█████████▌| 476/500 [00:18<00:01, 20.15it/s, failures=0, hvi=2.08]
 95%|█████████▌| 477/500 [00:18<00:01, 20.15it/s, failures=0, hvi=2.08]
 96%|█████████▌| 478/500 [00:18<00:01, 20.15it/s, failures=0, hvi=2.08]
 96%|█████████▌| 479/500 [00:18<00:01, 20.73it/s, failures=0, hvi=2.08]
 96%|█████████▌| 479/500 [00:18<00:01, 20.73it/s, failures=0, hvi=2.08]
 96%|█████████▌| 480/500 [00:18<00:00, 20.73it/s, failures=0, hvi=2.08]
 96%|█████████▌| 481/500 [00:18<00:00, 20.73it/s, failures=0, hvi=2.08]
 96%|█████████▋| 482/500 [00:18<00:00, 21.15it/s, failures=0, hvi=2.08]
 96%|█████████▋| 482/500 [00:18<00:00, 21.15it/s, failures=0, hvi=2.08]
 97%|█████████▋| 483/500 [00:18<00:00, 21.15it/s, failures=0, hvi=2.08]
 97%|█████████▋| 484/500 [00:18<00:00, 21.15it/s, failures=0, hvi=2.08]
 97%|█████████▋| 485/500 [00:18<00:00, 21.37it/s, failures=0, hvi=2.08]
 97%|█████████▋| 485/500 [00:18<00:00, 21.37it/s, failures=0, hvi=2.08]
 97%|█████████▋| 486/500 [00:19<00:00, 21.37it/s, failures=0, hvi=2.08]
 97%|█████████▋| 487/500 [00:19<00:00, 21.37it/s, failures=0, hvi=2.08]
 98%|█████████▊| 488/500 [00:19<00:00, 21.58it/s, failures=0, hvi=2.08]
 98%|█████████▊| 488/500 [00:19<00:00, 21.58it/s, failures=0, hvi=2.08]
 98%|█████████▊| 489/500 [00:19<00:00, 21.58it/s, failures=0, hvi=2.08]
 98%|█████████▊| 490/500 [00:19<00:00, 21.58it/s, failures=0, hvi=2.08]
 98%|█████████▊| 491/500 [00:19<00:00, 21.39it/s, failures=0, hvi=2.08]
 98%|█████████▊| 491/500 [00:19<00:00, 21.39it/s, failures=0, hvi=2.08]
 98%|█████████▊| 492/500 [00:19<00:00, 21.39it/s, failures=0, hvi=2.08]
 99%|█████████▊| 493/500 [00:19<00:00, 21.39it/s, failures=0, hvi=2.08]
 99%|█████████▉| 494/500 [00:19<00:00, 21.52it/s, failures=0, hvi=2.08]
 99%|█████████▉| 494/500 [00:19<00:00, 21.52it/s, failures=0, hvi=2.08]
 99%|█████████▉| 495/500 [00:19<00:00, 21.52it/s, failures=0, hvi=2.08]
 99%|█████████▉| 496/500 [00:19<00:00, 21.52it/s, failures=0, hvi=2.08]
 99%|█████████▉| 497/500 [00:19<00:00, 21.57it/s, failures=0, hvi=2.08]
 99%|█████████▉| 497/500 [00:19<00:00, 21.57it/s, failures=0, hvi=2.08]
100%|█████████▉| 498/500 [00:19<00:00, 21.57it/s, failures=0, hvi=2.08]
100%|█████████▉| 499/500 [00:19<00:00, 21.57it/s, failures=0, hvi=2.08]
100%|██████████| 500/500 [00:19<00:00, 21.69it/s, failures=0, hvi=2.08]
100%|██████████| 500/500 [00:19<00:00, 21.69it/s, failures=0, hvi=2.08]
100%|██████████| 500/500 [00:19<00:00, 25.45it/s, failures=0, hvi=2.08]

A Pandas table of results is returned by the search and also saved at ./results.csv. An other location can be specified by using CBO(..., log_dir=...).

results
p:x0 p:x1 p:x2 p:x3 p:x4 objective_0 objective_1 job_id job_status m:timestamp_submit m:timestamp_start m:timestamp_end m:timestamp_gather pareto_efficient
0 0.970527 0.489247 0.559856 0.611608 0.394239 -0.047545 -1.026239 0 DONE 0.012426 1.742982e+09 1.742982e+09 0.012786 False
1 0.089371 0.788825 0.361672 0.049895 0.558958 -1.295751 -0.183106 1 DONE 0.042940 1.742982e+09 1.742982e+09 0.043424 False
2 0.179558 0.941069 0.887661 0.666855 0.854480 -1.439117 -0.417018 2 DONE 0.061388 1.742982e+09 1.742982e+09 0.061619 False
3 0.517007 0.333782 0.722270 0.272979 0.868425 -0.869801 -0.917562 3 DONE 0.078366 1.742982e+09 1.742982e+09 0.078584 False
4 0.001609 0.077751 0.835900 0.437460 0.658283 -1.320084 -0.003336 4 DONE 0.095299 1.742982e+09 1.742982e+09 0.095504 False
... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
495 0.564694 0.466094 0.566032 0.558083 0.485525 -0.637469 -0.782241 495 DONE 19.484580 1.742982e+09 1.742982e+09 19.484810 True
496 0.195629 0.480185 0.589754 0.514997 0.500252 -0.961423 -0.305103 496 DONE 19.530415 1.742982e+09 1.742982e+09 19.530644 True
497 0.208287 0.587116 0.429743 0.460374 0.586788 -0.967434 -0.328321 497 DONE 19.576269 1.742982e+09 1.742982e+09 19.576496 False
498 0.719470 0.479758 0.537997 0.530006 0.556089 -0.429048 -0.909809 498 DONE 19.621704 1.742982e+09 1.742982e+09 19.621928 False
499 0.445290 0.580477 0.300647 0.440964 0.498338 -0.803224 -0.675806 499 DONE 19.666949 1.742982e+09 1.742982e+09 19.667172 False

500 rows × 14 columns



In this table we retrieve:

  • columns starting by p: which are the optimized variables.

  • the objective_{i} are the objectives returned by the black-box function.

  • the job_id is the identifier of the executed evaluations.

  • columns starting by m: are metadata returned by the black-box function.

  • pareto_efficient is a column only returned for MOO which specify if the evaluation is part of the set of optimal solutions.

Let us use this table to visualize evaluated objectives. The estimated optimal solutions will be colored in red.

Code (Plot evaluated objectives)
fig, ax = plt.subplots(figsize=(WIDTH_PLOTS, HEIGHT_PLOTS), tight_layout=True)
_ = ax.plot(
    -results[~results["pareto_efficient"]]["objective_0"],
    -results[~results["pareto_efficient"]]["objective_1"],
    "o",
    color="blue",
    alpha=0.7,
    label="Non Pareto-Efficient",
)
_ = ax.plot(
    -results[results["pareto_efficient"]]["objective_0"],
    -results[results["pareto_efficient"]]["objective_1"],
    "o",
    color="red",
    alpha=0.7,
    label="Pareto-Efficient",
)
_ = ax.grid()
_ = ax.legend()
_ = ax.set_xlabel("Objective 0")
_ = ax.set_ylabel("Objective 1")
plot multi objective optimization

Let us look the evolution of the hypervolume indicator. This metric should increase over time.

Code (Plot hypervolume)
scorer = bench.scorer
hvi = scorer.hypervolume(results[["objective_0", "objective_1"]].values)
x = list(range(1, len(hvi)+1))
fig, ax = plt.subplots(figsize=(WIDTH_PLOTS, HEIGHT_PLOTS), tight_layout=True)
_ = ax.plot(x, hvi)
_ = ax.grid()
_ = ax.set_xlabel("Evaluations")
_ = ax.set_ylabel("Hypervolume Indicator")
plot multi objective optimization

Total running time of the script: (0 minutes 21.313 seconds)

Gallery generated by Sphinx-Gallery

Contents