.. DO NOT EDIT.
.. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY.
.. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE:
.. "examples/examples_bbo/plot_multi_objective_optimization.py"
.. LINE NUMBERS ARE GIVEN BELOW.

.. only:: html

    .. note::
        :class: sphx-glr-download-link-note

        :ref:`Go to the end <sphx_glr_download_examples_examples_bbo_plot_multi_objective_optimization.py>`
        to download the full example code.

.. rst-class:: sphx-glr-example-title

.. _sphx_glr_examples_examples_bbo_plot_multi_objective_optimization.py:


Mutli-Objective Black-Box Optimization
======================================

In this tutorial, we will explore how to run black-box multi-objective optimization (MOO). In this setting, the goal is to resolve the following problem:

.. math::
   \text{max}_x (f_0(x), f_1(x), ..., f_n(x))

where :math:`x` is the set of optimized variables and :math:`f_i` are the different objectives. In DeepHyper, we use scalarization to transform such multi-objective problem into a single-objective problem:

.. math::
   \text{max}_x s_w((f_0(x), f_1(x), ..., f_n(x)))

where :math:`w` is a set of weights which manages the trade-off between objectives and :math:`s_w : \mathbb{R}^n \rightarrow \mathbb{R}`. The weight vector :math:`w` is randomized and re-sampled for each new batch of suggestion from the optimizer.

We will look at the DTLZ benchmark suite, a classic in multi-objective optimization (MOO) litterature. This benchmark exibit some characteristic cases of MOO. By default, this tutorial is loading the DTLZ-II benchmark which exibit a Pareto-Front with a concave shape.

.. GENERATED FROM PYTHON SOURCE LINES 20-30

Installation and imports
------------------------

Installing dependencies with the :ref:`pip installation <install-pip>` is recommended. It requires **Python >= 3.10**.

.. code-block:: bash

    %%bash
    pip install deephyper
    pip install "git+https://github.com/deephyper/benchmark.git@main#egg=deephyper-benchmark"

.. GENERATED FROM PYTHON SOURCE LINES 30-43

.. dropdown:: Code (Import statements)

    .. code-block:: Python


        import matplotlib.pyplot as plt

        from deephyper.hpo import CBO
        from deephyper_benchmark.benchmarks.dtlz import DTLZBenchmark

        WIDTH_PLOTS = 8
        HEIGHT_PLOTS = WIDTH_PLOTS / 1.618

        n_objectives = 2
        bench = DTLZBenchmark(nobj=n_objectives)








.. GENERATED FROM PYTHON SOURCE LINES 44-45

We can display the variable search space of the benchmark we just loaded:

.. GENERATED FROM PYTHON SOURCE LINES 45-48

.. code-block:: Python


    bench.problem





.. rst-class:: sphx-glr-script-out

 .. code-block:: none


    Configuration space object:
      Hyperparameters:
        x0, Type: UniformFloat, Range: [0.0, 1.0], Default: 0.5
        x1, Type: UniformFloat, Range: [0.0, 1.0], Default: 0.5
        x2, Type: UniformFloat, Range: [0.0, 1.0], Default: 0.5
        x3, Type: UniformFloat, Range: [0.0, 1.0], Default: 0.5
        x4, Type: UniformFloat, Range: [0.0, 1.0], Default: 0.5




.. GENERATED FROM PYTHON SOURCE LINES 49-73

To define a black-box for multi-objective optimization it is very similar to single-objective optimization at the difference that the ``objective`` can now be a list of values. A first possibility is:

.. code-block:: python

    def run(job):
        ...
        return objective_0, objective_1, ..., objective_n

which just returns the objectives to optimize as a tuple. If additionnal metadata are interesting to gather for each evaluation it is also possible to return them by following this format:

.. code-block:: python

    def run(job):
        ...
        return {
            "objective": [objective_0, objective_1, ..., objective_n],
            "metadata": {
                "flops": ...,
                "memory_footprint": ...,
                "duration": ...,
             }
         }

each of the metadata needs to be JSON serializable and will be returned in the final results with a column name formatted as ``m:metadata_key`` such as ``m:duration``.

.. GENERATED FROM PYTHON SOURCE LINES 75-84

For the search algorithm, we use the centralized Bayesian Optimization search (CBO).
Search algorithm

The arguments specific to multi-objective optimization are:

- ``moo_scalarization_strategy`` is used to specify the scalarization strategy. 
  Chebyshev  scalarizationis capable of generating a diverse set of solutions for non-convex problems.
- ``moo_scalarization_weight`` argument is used to specify the weight of objectives in the scalarization.
  ``"random"`` is used to generate a random weight vector at each iteration.

.. GENERATED FROM PYTHON SOURCE LINES 84-93

.. code-block:: Python


    search = CBO(
        bench.problem,
        acq_optimizer="sampling",
        moo_scalarization_strategy="AugChebyshev",
        moo_scalarization_weight="random",
        verbose=1,
    )





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    Results file already exists, it will be renamed to /Users/rp5/Documents/DeepHyper/deephyper/examples/examples_bbo/results_20250818-134730.csv




.. GENERATED FROM PYTHON SOURCE LINES 94-96

Launch the search for a given number of evaluations
other stopping criteria can be used (e.g. timeout, early-stopping/convergence)

.. GENERATED FROM PYTHON SOURCE LINES 96-98

.. code-block:: Python

    results = search.search(bench.run_function, max_evals=500)





.. rst-class:: sphx-glr-script-out

 .. code-block:: none


      0%|          | 0/500 [00:00<?, ?it/s]
      0%|          | 1/500 [00:00<00:00, 3731.59it/s, failures=0, hvi=0]
      0%|          | 2/500 [00:00<00:04, 113.17it/s, failures=0, hvi=0] 
      1%|          | 3/500 [00:00<00:05, 97.61it/s, failures=0, hvi=0.433]
      1%|          | 4/500 [00:00<00:05, 92.30it/s, failures=0, hvi=0.433]
      1%|          | 5/500 [00:00<00:05, 90.20it/s, failures=0, hvi=0.504]
      1%|          | 6/500 [00:00<00:05, 88.47it/s, failures=0, hvi=0.628]
      1%|▏         | 7/500 [00:00<00:05, 87.65it/s, failures=0, hvi=0.732]
      2%|▏         | 8/500 [00:00<00:05, 86.98it/s, failures=0, hvi=0.756]
      2%|▏         | 9/500 [00:00<00:05, 86.61it/s, failures=0, hvi=0.756]
      2%|▏         | 9/500 [00:00<00:05, 86.61it/s, failures=0, hvi=0.756]
      2%|▏         | 10/500 [00:00<00:05, 86.61it/s, failures=0, hvi=0.934]
      2%|▏         | 11/500 [00:00<00:05, 86.61it/s, failures=0, hvi=0.942]
      2%|▏         | 12/500 [00:00<00:05, 86.61it/s, failures=0, hvi=0.947]
      3%|▎         | 13/500 [00:00<00:05, 86.61it/s, failures=0, hvi=0.947]
      3%|▎         | 14/500 [00:00<00:05, 86.61it/s, failures=0, hvi=1.02] 
      3%|▎         | 15/500 [00:00<00:05, 86.61it/s, failures=0, hvi=1.07]
      3%|▎         | 16/500 [00:00<00:05, 86.61it/s, failures=0, hvi=1.09]
      3%|▎         | 17/500 [00:00<00:05, 86.61it/s, failures=0, hvi=1.1] 
      4%|▎         | 18/500 [00:00<00:07, 66.09it/s, failures=0, hvi=1.1]
      4%|▎         | 18/500 [00:00<00:07, 66.09it/s, failures=0, hvi=1.11]
      4%|▍         | 19/500 [00:00<00:07, 66.09it/s, failures=0, hvi=1.11]
      4%|▍         | 20/500 [00:00<00:07, 66.09it/s, failures=0, hvi=1.12]
      4%|▍         | 21/500 [00:00<00:07, 66.09it/s, failures=0, hvi=1.15]
      4%|▍         | 22/500 [00:00<00:07, 66.09it/s, failures=0, hvi=1.15]
      5%|▍         | 23/500 [00:00<00:07, 66.09it/s, failures=0, hvi=1.15]
      5%|▍         | 24/500 [00:00<00:07, 66.09it/s, failures=0, hvi=1.15]
      5%|▌         | 25/500 [00:00<00:08, 59.37it/s, failures=0, hvi=1.15]
      5%|▌         | 25/500 [00:00<00:08, 59.37it/s, failures=0, hvi=1.15]
      5%|▌         | 26/500 [00:00<00:07, 59.37it/s, failures=0, hvi=1.25]
      5%|▌         | 27/500 [00:00<00:07, 59.37it/s, failures=0, hvi=1.27]
      6%|▌         | 28/500 [00:00<00:07, 59.37it/s, failures=0, hvi=1.27]
      6%|▌         | 29/500 [00:00<00:07, 59.37it/s, failures=0, hvi=1.27]
      6%|▌         | 30/500 [00:00<00:07, 59.37it/s, failures=0, hvi=1.27]
      6%|▌         | 31/500 [00:00<00:07, 59.37it/s, failures=0, hvi=1.28]
      6%|▋         | 32/500 [00:00<00:08, 55.35it/s, failures=0, hvi=1.28]
      6%|▋         | 32/500 [00:00<00:08, 55.35it/s, failures=0, hvi=1.32]
      7%|▋         | 33/500 [00:00<00:08, 55.35it/s, failures=0, hvi=1.35]
      7%|▋         | 34/500 [00:00<00:08, 55.35it/s, failures=0, hvi=1.35]
      7%|▋         | 35/500 [00:00<00:08, 55.35it/s, failures=0, hvi=1.35]
      7%|▋         | 36/500 [00:00<00:08, 55.35it/s, failures=0, hvi=1.35]
      7%|▋         | 37/500 [00:00<00:08, 55.35it/s, failures=0, hvi=1.35]
      8%|▊         | 38/500 [00:00<00:08, 53.18it/s, failures=0, hvi=1.35]
      8%|▊         | 38/500 [00:00<00:08, 53.18it/s, failures=0, hvi=1.36]
      8%|▊         | 39/500 [00:00<00:08, 53.18it/s, failures=0, hvi=1.36]
      8%|▊         | 40/500 [00:00<00:08, 53.18it/s, failures=0, hvi=1.36]
      8%|▊         | 41/500 [00:00<00:08, 53.18it/s, failures=0, hvi=1.36]
      8%|▊         | 42/500 [00:00<00:08, 53.18it/s, failures=0, hvi=1.36]
      9%|▊         | 43/500 [00:00<00:08, 53.18it/s, failures=0, hvi=1.36]
      9%|▉         | 44/500 [00:00<00:08, 51.01it/s, failures=0, hvi=1.36]
      9%|▉         | 44/500 [00:00<00:08, 51.01it/s, failures=0, hvi=1.36]
      9%|▉         | 45/500 [00:00<00:08, 51.01it/s, failures=0, hvi=1.36]
      9%|▉         | 46/500 [00:00<00:08, 51.01it/s, failures=0, hvi=1.36]
      9%|▉         | 47/500 [00:00<00:08, 51.01it/s, failures=0, hvi=1.36]
     10%|▉         | 48/500 [00:00<00:08, 51.01it/s, failures=0, hvi=1.36]
     10%|▉         | 49/500 [00:00<00:08, 51.01it/s, failures=0, hvi=1.36]
     10%|█         | 50/500 [00:00<00:10, 44.94it/s, failures=0, hvi=1.36]
     10%|█         | 50/500 [00:00<00:10, 44.94it/s, failures=0, hvi=1.39]
     10%|█         | 51/500 [00:00<00:09, 44.94it/s, failures=0, hvi=1.45]
     10%|█         | 52/500 [00:01<00:09, 44.94it/s, failures=0, hvi=1.45]
     11%|█         | 53/500 [00:01<00:09, 44.94it/s, failures=0, hvi=1.45]
     11%|█         | 54/500 [00:01<00:09, 44.94it/s, failures=0, hvi=1.45]
     11%|█         | 55/500 [00:01<00:09, 44.88it/s, failures=0, hvi=1.45]
     11%|█         | 55/500 [00:01<00:09, 44.88it/s, failures=0, hvi=1.45]
     11%|█         | 56/500 [00:01<00:09, 44.88it/s, failures=0, hvi=1.46]
     11%|█▏        | 57/500 [00:01<00:09, 44.88it/s, failures=0, hvi=1.46]
     12%|█▏        | 58/500 [00:01<00:09, 44.88it/s, failures=0, hvi=1.46]
     12%|█▏        | 59/500 [00:01<00:09, 44.88it/s, failures=0, hvi=1.47]
     12%|█▏        | 60/500 [00:01<00:09, 44.45it/s, failures=0, hvi=1.47]
     12%|█▏        | 60/500 [00:01<00:09, 44.45it/s, failures=0, hvi=1.47]
     12%|█▏        | 61/500 [00:01<00:09, 44.45it/s, failures=0, hvi=1.47]
     12%|█▏        | 62/500 [00:01<00:09, 44.45it/s, failures=0, hvi=1.47]
     13%|█▎        | 63/500 [00:01<00:09, 44.45it/s, failures=0, hvi=1.47]
     13%|█▎        | 64/500 [00:01<00:09, 44.45it/s, failures=0, hvi=1.47]
     13%|█▎        | 65/500 [00:01<00:09, 43.78it/s, failures=0, hvi=1.47]
     13%|█▎        | 65/500 [00:01<00:09, 43.78it/s, failures=0, hvi=1.47]
     13%|█▎        | 66/500 [00:01<00:09, 43.78it/s, failures=0, hvi=1.49]
     13%|█▎        | 67/500 [00:01<00:09, 43.78it/s, failures=0, hvi=1.49]
     14%|█▎        | 68/500 [00:01<00:09, 43.78it/s, failures=0, hvi=1.49]
     14%|█▍        | 69/500 [00:01<00:09, 43.78it/s, failures=0, hvi=1.49]
     14%|█▍        | 70/500 [00:01<00:09, 43.28it/s, failures=0, hvi=1.49]
     14%|█▍        | 70/500 [00:01<00:09, 43.28it/s, failures=0, hvi=1.49]
     14%|█▍        | 71/500 [00:01<00:09, 43.28it/s, failures=0, hvi=1.5] 
     14%|█▍        | 72/500 [00:01<00:09, 43.28it/s, failures=0, hvi=1.5]
     15%|█▍        | 73/500 [00:01<00:09, 43.28it/s, failures=0, hvi=1.5]
     15%|█▍        | 74/500 [00:01<00:09, 43.28it/s, failures=0, hvi=1.5]
     15%|█▌        | 75/500 [00:01<00:09, 42.86it/s, failures=0, hvi=1.5]
     15%|█▌        | 75/500 [00:01<00:09, 42.86it/s, failures=0, hvi=1.5]
     15%|█▌        | 76/500 [00:01<00:09, 42.86it/s, failures=0, hvi=1.5]
     15%|█▌        | 77/500 [00:01<00:09, 42.86it/s, failures=0, hvi=1.51]
     16%|█▌        | 78/500 [00:01<00:09, 42.86it/s, failures=0, hvi=1.51]
     16%|█▌        | 79/500 [00:01<00:09, 42.86it/s, failures=0, hvi=1.51]
     16%|█▌        | 80/500 [00:01<00:09, 42.35it/s, failures=0, hvi=1.51]
     16%|█▌        | 80/500 [00:01<00:09, 42.35it/s, failures=0, hvi=1.51]
     16%|█▌        | 81/500 [00:01<00:09, 42.35it/s, failures=0, hvi=1.51]
     16%|█▋        | 82/500 [00:01<00:09, 42.35it/s, failures=0, hvi=1.51]
     17%|█▋        | 83/500 [00:01<00:09, 42.35it/s, failures=0, hvi=1.51]
     17%|█▋        | 84/500 [00:01<00:09, 42.35it/s, failures=0, hvi=1.51]
     17%|█▋        | 85/500 [00:01<00:09, 42.10it/s, failures=0, hvi=1.51]
     17%|█▋        | 85/500 [00:01<00:09, 42.10it/s, failures=0, hvi=1.51]
     17%|█▋        | 86/500 [00:01<00:09, 42.10it/s, failures=0, hvi=1.51]
     17%|█▋        | 87/500 [00:01<00:09, 42.10it/s, failures=0, hvi=1.51]
     18%|█▊        | 88/500 [00:01<00:09, 42.10it/s, failures=0, hvi=1.51]
     18%|█▊        | 89/500 [00:01<00:09, 42.10it/s, failures=0, hvi=1.53]
     18%|█▊        | 90/500 [00:01<00:09, 41.66it/s, failures=0, hvi=1.53]
     18%|█▊        | 90/500 [00:01<00:09, 41.66it/s, failures=0, hvi=1.53]
     18%|█▊        | 91/500 [00:01<00:09, 41.66it/s, failures=0, hvi=1.53]
     18%|█▊        | 92/500 [00:01<00:09, 41.66it/s, failures=0, hvi=1.53]
     19%|█▊        | 93/500 [00:01<00:09, 41.66it/s, failures=0, hvi=1.53]
     19%|█▉        | 94/500 [00:02<00:09, 41.66it/s, failures=0, hvi=1.53]
     19%|█▉        | 95/500 [00:02<00:09, 41.19it/s, failures=0, hvi=1.53]
     19%|█▉        | 95/500 [00:02<00:09, 41.19it/s, failures=0, hvi=1.53]
     19%|█▉        | 96/500 [00:02<00:09, 41.19it/s, failures=0, hvi=1.53]
     19%|█▉        | 97/500 [00:02<00:09, 41.19it/s, failures=0, hvi=1.54]
     20%|█▉        | 98/500 [00:02<00:09, 41.19it/s, failures=0, hvi=1.54]
     20%|█▉        | 99/500 [00:02<00:09, 41.19it/s, failures=0, hvi=1.54]
     20%|██        | 100/500 [00:02<00:09, 40.67it/s, failures=0, hvi=1.54]
     20%|██        | 100/500 [00:02<00:09, 40.67it/s, failures=0, hvi=1.54]
     20%|██        | 101/500 [00:02<00:09, 40.67it/s, failures=0, hvi=1.54]
     20%|██        | 102/500 [00:02<00:09, 40.67it/s, failures=0, hvi=1.54]
     21%|██        | 103/500 [00:02<00:09, 40.67it/s, failures=0, hvi=1.54]
     21%|██        | 104/500 [00:02<00:09, 40.67it/s, failures=0, hvi=1.54]
     21%|██        | 105/500 [00:02<00:09, 40.29it/s, failures=0, hvi=1.54]
     21%|██        | 105/500 [00:02<00:09, 40.29it/s, failures=0, hvi=1.54]
     21%|██        | 106/500 [00:02<00:09, 40.29it/s, failures=0, hvi=1.55]
     21%|██▏       | 107/500 [00:02<00:09, 40.29it/s, failures=0, hvi=1.55]
     22%|██▏       | 108/500 [00:02<00:09, 40.29it/s, failures=0, hvi=1.55]
     22%|██▏       | 109/500 [00:02<00:09, 40.29it/s, failures=0, hvi=1.55]
     22%|██▏       | 110/500 [00:02<00:09, 39.86it/s, failures=0, hvi=1.55]
     22%|██▏       | 110/500 [00:02<00:09, 39.86it/s, failures=0, hvi=1.55]
     22%|██▏       | 111/500 [00:02<00:09, 39.86it/s, failures=0, hvi=1.56]
     22%|██▏       | 112/500 [00:02<00:09, 39.86it/s, failures=0, hvi=1.56]
     23%|██▎       | 113/500 [00:02<00:09, 39.86it/s, failures=0, hvi=1.56]
     23%|██▎       | 114/500 [00:02<00:09, 39.58it/s, failures=0, hvi=1.56]
     23%|██▎       | 114/500 [00:02<00:09, 39.58it/s, failures=0, hvi=1.56]
     23%|██▎       | 115/500 [00:02<00:09, 39.58it/s, failures=0, hvi=1.56]
     23%|██▎       | 116/500 [00:02<00:09, 39.58it/s, failures=0, hvi=1.56]
     23%|██▎       | 117/500 [00:02<00:09, 39.58it/s, failures=0, hvi=1.56]
     24%|██▎       | 118/500 [00:02<00:10, 35.97it/s, failures=0, hvi=1.56]
     24%|██▎       | 118/500 [00:02<00:10, 35.97it/s, failures=0, hvi=1.56]
     24%|██▍       | 119/500 [00:02<00:10, 35.97it/s, failures=0, hvi=1.56]
     24%|██▍       | 120/500 [00:02<00:10, 35.97it/s, failures=0, hvi=1.56]
     24%|██▍       | 121/500 [00:02<00:10, 35.97it/s, failures=0, hvi=1.56]
     24%|██▍       | 122/500 [00:02<00:10, 36.67it/s, failures=0, hvi=1.56]
     24%|██▍       | 122/500 [00:02<00:10, 36.67it/s, failures=0, hvi=1.58]
     25%|██▍       | 123/500 [00:02<00:10, 36.67it/s, failures=0, hvi=1.58]
     25%|██▍       | 124/500 [00:02<00:10, 36.67it/s, failures=0, hvi=1.59]
     25%|██▌       | 125/500 [00:02<00:10, 36.67it/s, failures=0, hvi=1.6] 
     25%|██▌       | 126/500 [00:02<00:10, 37.12it/s, failures=0, hvi=1.6]
     25%|██▌       | 126/500 [00:02<00:10, 37.12it/s, failures=0, hvi=1.6]
     25%|██▌       | 127/500 [00:02<00:10, 37.12it/s, failures=0, hvi=1.6]
     26%|██▌       | 128/500 [00:02<00:10, 37.12it/s, failures=0, hvi=1.6]
     26%|██▌       | 129/500 [00:02<00:09, 37.12it/s, failures=0, hvi=1.6]
     26%|██▌       | 130/500 [00:02<00:09, 37.41it/s, failures=0, hvi=1.6]
     26%|██▌       | 130/500 [00:02<00:09, 37.41it/s, failures=0, hvi=1.6]
     26%|██▌       | 131/500 [00:02<00:09, 37.41it/s, failures=0, hvi=1.61]
     26%|██▋       | 132/500 [00:03<00:09, 37.41it/s, failures=0, hvi=1.61]
     27%|██▋       | 133/500 [00:03<00:09, 37.41it/s, failures=0, hvi=1.61]
     27%|██▋       | 134/500 [00:03<00:09, 37.57it/s, failures=0, hvi=1.61]
     27%|██▋       | 134/500 [00:03<00:09, 37.57it/s, failures=0, hvi=1.61]
     27%|██▋       | 135/500 [00:03<00:09, 37.57it/s, failures=0, hvi=1.61]
     27%|██▋       | 136/500 [00:03<00:09, 37.57it/s, failures=0, hvi=1.61]
     27%|██▋       | 137/500 [00:03<00:09, 37.57it/s, failures=0, hvi=1.61]
     28%|██▊       | 138/500 [00:03<00:09, 37.60it/s, failures=0, hvi=1.61]
     28%|██▊       | 138/500 [00:03<00:09, 37.60it/s, failures=0, hvi=1.61]
     28%|██▊       | 139/500 [00:03<00:09, 37.60it/s, failures=0, hvi=1.61]
     28%|██▊       | 140/500 [00:03<00:09, 37.60it/s, failures=0, hvi=1.62]
     28%|██▊       | 141/500 [00:03<00:09, 37.60it/s, failures=0, hvi=1.62]
     28%|██▊       | 142/500 [00:03<00:09, 37.60it/s, failures=0, hvi=1.62]
     28%|██▊       | 142/500 [00:03<00:09, 37.60it/s, failures=0, hvi=1.62]
     29%|██▊       | 143/500 [00:03<00:09, 37.60it/s, failures=0, hvi=1.62]
     29%|██▉       | 144/500 [00:03<00:09, 37.60it/s, failures=0, hvi=1.62]
     29%|██▉       | 145/500 [00:03<00:09, 37.60it/s, failures=0, hvi=1.62]
     29%|██▉       | 146/500 [00:03<00:09, 37.58it/s, failures=0, hvi=1.62]
     29%|██▉       | 146/500 [00:03<00:09, 37.58it/s, failures=0, hvi=1.62]
     29%|██▉       | 147/500 [00:03<00:09, 37.58it/s, failures=0, hvi=1.62]
     30%|██▉       | 148/500 [00:03<00:09, 37.58it/s, failures=0, hvi=1.62]
     30%|██▉       | 149/500 [00:03<00:09, 37.58it/s, failures=0, hvi=1.62]
     30%|███       | 150/500 [00:03<00:09, 37.31it/s, failures=0, hvi=1.62]
     30%|███       | 150/500 [00:03<00:09, 37.31it/s, failures=0, hvi=1.62]
     30%|███       | 151/500 [00:03<00:09, 37.31it/s, failures=0, hvi=1.62]
     30%|███       | 152/500 [00:03<00:09, 37.31it/s, failures=0, hvi=1.62]
     31%|███       | 153/500 [00:03<00:09, 37.31it/s, failures=0, hvi=1.62]
     31%|███       | 154/500 [00:03<00:09, 37.16it/s, failures=0, hvi=1.62]
     31%|███       | 154/500 [00:03<00:09, 37.16it/s, failures=0, hvi=1.62]
     31%|███       | 155/500 [00:03<00:09, 37.16it/s, failures=0, hvi=1.62]
     31%|███       | 156/500 [00:03<00:09, 37.16it/s, failures=0, hvi=1.62]
     31%|███▏      | 157/500 [00:03<00:09, 37.16it/s, failures=0, hvi=1.63]
     32%|███▏      | 158/500 [00:03<00:09, 36.94it/s, failures=0, hvi=1.63]
     32%|███▏      | 158/500 [00:03<00:09, 36.94it/s, failures=0, hvi=1.63]
     32%|███▏      | 159/500 [00:03<00:09, 36.94it/s, failures=0, hvi=1.63]
     32%|███▏      | 160/500 [00:03<00:09, 36.94it/s, failures=0, hvi=1.63]
     32%|███▏      | 161/500 [00:03<00:09, 36.94it/s, failures=0, hvi=1.63]
     32%|███▏      | 162/500 [00:03<00:09, 36.85it/s, failures=0, hvi=1.63]
     32%|███▏      | 162/500 [00:03<00:09, 36.85it/s, failures=0, hvi=1.63]
     33%|███▎      | 163/500 [00:03<00:09, 36.85it/s, failures=0, hvi=1.69]
     33%|███▎      | 164/500 [00:03<00:09, 36.85it/s, failures=0, hvi=1.69]
     33%|███▎      | 165/500 [00:03<00:09, 36.85it/s, failures=0, hvi=1.76]
     33%|███▎      | 166/500 [00:03<00:09, 36.71it/s, failures=0, hvi=1.76]
     33%|███▎      | 166/500 [00:03<00:09, 36.71it/s, failures=0, hvi=1.76]
     33%|███▎      | 167/500 [00:03<00:09, 36.71it/s, failures=0, hvi=1.76]
     34%|███▎      | 168/500 [00:03<00:09, 36.71it/s, failures=0, hvi=1.76]
     34%|███▍      | 169/500 [00:04<00:09, 36.71it/s, failures=0, hvi=1.76]
     34%|███▍      | 170/500 [00:04<00:09, 36.61it/s, failures=0, hvi=1.76]
     34%|███▍      | 170/500 [00:04<00:09, 36.61it/s, failures=0, hvi=1.77]
     34%|███▍      | 171/500 [00:04<00:08, 36.61it/s, failures=0, hvi=1.77]
     34%|███▍      | 172/500 [00:04<00:08, 36.61it/s, failures=0, hvi=1.77]
     35%|███▍      | 173/500 [00:04<00:08, 36.61it/s, failures=0, hvi=1.77]
     35%|███▍      | 174/500 [00:04<00:08, 36.45it/s, failures=0, hvi=1.77]
     35%|███▍      | 174/500 [00:04<00:08, 36.45it/s, failures=0, hvi=1.77]
     35%|███▌      | 175/500 [00:04<00:08, 36.45it/s, failures=0, hvi=1.77]
     35%|███▌      | 176/500 [00:04<00:08, 36.45it/s, failures=0, hvi=1.77]
     35%|███▌      | 177/500 [00:04<00:08, 36.45it/s, failures=0, hvi=1.77]
     36%|███▌      | 178/500 [00:04<00:08, 36.28it/s, failures=0, hvi=1.77]
     36%|███▌      | 178/500 [00:04<00:08, 36.28it/s, failures=0, hvi=1.78]
     36%|███▌      | 179/500 [00:04<00:08, 36.28it/s, failures=0, hvi=1.78]
     36%|███▌      | 180/500 [00:04<00:08, 36.28it/s, failures=0, hvi=1.78]
     36%|███▌      | 181/500 [00:04<00:08, 36.28it/s, failures=0, hvi=1.78]
     36%|███▋      | 182/500 [00:04<00:09, 33.05it/s, failures=0, hvi=1.78]
     36%|███▋      | 182/500 [00:04<00:09, 33.05it/s, failures=0, hvi=1.78]
     37%|███▋      | 183/500 [00:04<00:09, 33.05it/s, failures=0, hvi=1.78]
     37%|███▋      | 184/500 [00:04<00:09, 33.05it/s, failures=0, hvi=1.78]
     37%|███▋      | 185/500 [00:04<00:09, 33.05it/s, failures=0, hvi=1.78]
     37%|███▋      | 186/500 [00:04<00:09, 33.83it/s, failures=0, hvi=1.78]
     37%|███▋      | 186/500 [00:04<00:09, 33.83it/s, failures=0, hvi=1.78]
     37%|███▋      | 187/500 [00:04<00:09, 33.83it/s, failures=0, hvi=1.78]
     38%|███▊      | 188/500 [00:04<00:09, 33.83it/s, failures=0, hvi=1.78]
     38%|███▊      | 189/500 [00:04<00:09, 33.83it/s, failures=0, hvi=1.78]
     38%|███▊      | 190/500 [00:04<00:09, 34.37it/s, failures=0, hvi=1.78]
     38%|███▊      | 190/500 [00:04<00:09, 34.37it/s, failures=0, hvi=1.78]
     38%|███▊      | 191/500 [00:04<00:08, 34.37it/s, failures=0, hvi=1.78]
     38%|███▊      | 192/500 [00:04<00:08, 34.37it/s, failures=0, hvi=1.78]
     39%|███▊      | 193/500 [00:04<00:08, 34.37it/s, failures=0, hvi=1.78]
     39%|███▉      | 194/500 [00:04<00:08, 34.65it/s, failures=0, hvi=1.78]
     39%|███▉      | 194/500 [00:04<00:08, 34.65it/s, failures=0, hvi=1.78]
     39%|███▉      | 195/500 [00:04<00:08, 34.65it/s, failures=0, hvi=1.78]
     39%|███▉      | 196/500 [00:04<00:08, 34.65it/s, failures=0, hvi=1.78]
     39%|███▉      | 197/500 [00:04<00:08, 34.65it/s, failures=0, hvi=1.78]
     40%|███▉      | 198/500 [00:04<00:08, 34.79it/s, failures=0, hvi=1.78]
     40%|███▉      | 198/500 [00:04<00:08, 34.79it/s, failures=0, hvi=1.78]
     40%|███▉      | 199/500 [00:04<00:08, 34.79it/s, failures=0, hvi=1.78]
     40%|████      | 200/500 [00:04<00:08, 34.79it/s, failures=0, hvi=1.78]
     40%|████      | 201/500 [00:04<00:08, 34.79it/s, failures=0, hvi=1.79]
     40%|████      | 202/500 [00:04<00:08, 34.85it/s, failures=0, hvi=1.79]
     40%|████      | 202/500 [00:04<00:08, 34.85it/s, failures=0, hvi=1.79]
     41%|████      | 203/500 [00:05<00:08, 34.85it/s, failures=0, hvi=1.79]
     41%|████      | 204/500 [00:05<00:08, 34.85it/s, failures=0, hvi=1.79]
     41%|████      | 205/500 [00:05<00:08, 34.85it/s, failures=0, hvi=1.79]
     41%|████      | 206/500 [00:05<00:08, 34.82it/s, failures=0, hvi=1.79]
     41%|████      | 206/500 [00:05<00:08, 34.82it/s, failures=0, hvi=1.79]
     41%|████▏     | 207/500 [00:05<00:08, 34.82it/s, failures=0, hvi=1.79]
     42%|████▏     | 208/500 [00:05<00:08, 34.82it/s, failures=0, hvi=1.79]
     42%|████▏     | 209/500 [00:05<00:08, 34.82it/s, failures=0, hvi=1.79]
     42%|████▏     | 210/500 [00:05<00:08, 34.79it/s, failures=0, hvi=1.79]
     42%|████▏     | 210/500 [00:05<00:08, 34.79it/s, failures=0, hvi=1.79]
     42%|████▏     | 211/500 [00:05<00:08, 34.79it/s, failures=0, hvi=1.79]
     42%|████▏     | 212/500 [00:05<00:08, 34.79it/s, failures=0, hvi=1.79]
     43%|████▎     | 213/500 [00:05<00:08, 34.79it/s, failures=0, hvi=1.79]
     43%|████▎     | 214/500 [00:05<00:08, 34.76it/s, failures=0, hvi=1.79]
     43%|████▎     | 214/500 [00:05<00:08, 34.76it/s, failures=0, hvi=1.79]
     43%|████▎     | 215/500 [00:05<00:08, 34.76it/s, failures=0, hvi=1.79]
     43%|████▎     | 216/500 [00:05<00:08, 34.76it/s, failures=0, hvi=1.79]
     43%|████▎     | 217/500 [00:05<00:08, 34.76it/s, failures=0, hvi=1.79]
     44%|████▎     | 218/500 [00:05<00:08, 34.73it/s, failures=0, hvi=1.79]
     44%|████▎     | 218/500 [00:05<00:08, 34.73it/s, failures=0, hvi=1.79]
     44%|████▍     | 219/500 [00:05<00:08, 34.73it/s, failures=0, hvi=1.79]
     44%|████▍     | 220/500 [00:05<00:08, 34.73it/s, failures=0, hvi=1.79]
     44%|████▍     | 221/500 [00:05<00:08, 34.73it/s, failures=0, hvi=1.79]
     44%|████▍     | 222/500 [00:05<00:08, 34.73it/s, failures=0, hvi=1.79]
     44%|████▍     | 222/500 [00:05<00:08, 34.73it/s, failures=0, hvi=1.79]
     45%|████▍     | 223/500 [00:05<00:07, 34.73it/s, failures=0, hvi=1.79]
     45%|████▍     | 224/500 [00:05<00:07, 34.73it/s, failures=0, hvi=1.79]
     45%|████▌     | 225/500 [00:05<00:07, 34.73it/s, failures=0, hvi=1.79]
     45%|████▌     | 226/500 [00:05<00:07, 34.60it/s, failures=0, hvi=1.79]
     45%|████▌     | 226/500 [00:05<00:07, 34.60it/s, failures=0, hvi=1.79]
     45%|████▌     | 227/500 [00:05<00:07, 34.60it/s, failures=0, hvi=1.79]
     46%|████▌     | 228/500 [00:05<00:07, 34.60it/s, failures=0, hvi=1.8] 
     46%|████▌     | 229/500 [00:05<00:07, 34.60it/s, failures=0, hvi=1.8]
     46%|████▌     | 230/500 [00:05<00:07, 34.40it/s, failures=0, hvi=1.8]
     46%|████▌     | 230/500 [00:05<00:07, 34.40it/s, failures=0, hvi=1.8]
     46%|████▌     | 231/500 [00:05<00:07, 34.40it/s, failures=0, hvi=1.8]
     46%|████▋     | 232/500 [00:05<00:07, 34.40it/s, failures=0, hvi=1.8]
     47%|████▋     | 233/500 [00:05<00:07, 34.40it/s, failures=0, hvi=1.8]
     47%|████▋     | 234/500 [00:05<00:07, 34.25it/s, failures=0, hvi=1.8]
     47%|████▋     | 234/500 [00:05<00:07, 34.25it/s, failures=0, hvi=1.8]
     47%|████▋     | 235/500 [00:05<00:07, 34.25it/s, failures=0, hvi=1.8]
     47%|████▋     | 236/500 [00:05<00:07, 34.25it/s, failures=0, hvi=1.8]
     47%|████▋     | 237/500 [00:06<00:07, 34.25it/s, failures=0, hvi=1.8]
     48%|████▊     | 238/500 [00:06<00:07, 34.16it/s, failures=0, hvi=1.8]
     48%|████▊     | 238/500 [00:06<00:07, 34.16it/s, failures=0, hvi=1.8]
     48%|████▊     | 239/500 [00:06<00:07, 34.16it/s, failures=0, hvi=1.8]
     48%|████▊     | 240/500 [00:06<00:07, 34.16it/s, failures=0, hvi=1.8]
     48%|████▊     | 241/500 [00:06<00:07, 34.16it/s, failures=0, hvi=1.8]
     48%|████▊     | 242/500 [00:06<00:07, 34.06it/s, failures=0, hvi=1.8]
     48%|████▊     | 242/500 [00:06<00:07, 34.06it/s, failures=0, hvi=1.8]
     49%|████▊     | 243/500 [00:06<00:07, 34.06it/s, failures=0, hvi=1.8]
     49%|████▉     | 244/500 [00:06<00:07, 34.06it/s, failures=0, hvi=1.8]
     49%|████▉     | 245/500 [00:06<00:07, 34.06it/s, failures=0, hvi=1.8]
     49%|████▉     | 246/500 [00:06<00:07, 33.98it/s, failures=0, hvi=1.8]
     49%|████▉     | 246/500 [00:06<00:07, 33.98it/s, failures=0, hvi=1.8]
     49%|████▉     | 247/500 [00:06<00:07, 33.98it/s, failures=0, hvi=1.8]
     50%|████▉     | 248/500 [00:06<00:07, 33.98it/s, failures=0, hvi=1.8]
     50%|████▉     | 249/500 [00:06<00:07, 33.98it/s, failures=0, hvi=1.8]
     50%|█████     | 250/500 [00:06<00:08, 31.08it/s, failures=0, hvi=1.8]
     50%|█████     | 250/500 [00:06<00:08, 31.08it/s, failures=0, hvi=1.8]
     50%|█████     | 251/500 [00:06<00:08, 31.08it/s, failures=0, hvi=1.81]
     50%|█████     | 252/500 [00:06<00:07, 31.08it/s, failures=0, hvi=1.81]
     51%|█████     | 253/500 [00:06<00:07, 31.08it/s, failures=0, hvi=1.81]
     51%|█████     | 254/500 [00:06<00:07, 31.81it/s, failures=0, hvi=1.81]
     51%|█████     | 254/500 [00:06<00:07, 31.81it/s, failures=0, hvi=1.81]
     51%|█████     | 255/500 [00:06<00:07, 31.81it/s, failures=0, hvi=1.81]
     51%|█████     | 256/500 [00:06<00:07, 31.81it/s, failures=0, hvi=1.81]
     51%|█████▏    | 257/500 [00:06<00:07, 31.81it/s, failures=0, hvi=1.81]
     52%|█████▏    | 258/500 [00:06<00:07, 32.32it/s, failures=0, hvi=1.81]
     52%|█████▏    | 258/500 [00:06<00:07, 32.32it/s, failures=0, hvi=1.81]
     52%|█████▏    | 259/500 [00:06<00:07, 32.32it/s, failures=0, hvi=1.81]
     52%|█████▏    | 260/500 [00:06<00:07, 32.32it/s, failures=0, hvi=1.81]
     52%|█████▏    | 261/500 [00:06<00:07, 32.32it/s, failures=0, hvi=1.81]
     52%|█████▏    | 262/500 [00:06<00:07, 32.55it/s, failures=0, hvi=1.81]
     52%|█████▏    | 262/500 [00:06<00:07, 32.55it/s, failures=0, hvi=1.81]
     53%|█████▎    | 263/500 [00:06<00:07, 32.55it/s, failures=0, hvi=1.81]
     53%|█████▎    | 264/500 [00:06<00:07, 32.55it/s, failures=0, hvi=1.81]
     53%|█████▎    | 265/500 [00:06<00:07, 32.55it/s, failures=0, hvi=1.81]
     53%|█████▎    | 266/500 [00:06<00:07, 32.75it/s, failures=0, hvi=1.81]
     53%|█████▎    | 266/500 [00:06<00:07, 32.75it/s, failures=0, hvi=1.81]
     53%|█████▎    | 267/500 [00:06<00:07, 32.75it/s, failures=0, hvi=1.81]
     54%|█████▎    | 268/500 [00:06<00:07, 32.75it/s, failures=0, hvi=1.81]
     54%|█████▍    | 269/500 [00:06<00:07, 32.75it/s, failures=0, hvi=1.81]
     54%|█████▍    | 270/500 [00:07<00:06, 32.88it/s, failures=0, hvi=1.81]
     54%|█████▍    | 270/500 [00:07<00:06, 32.88it/s, failures=0, hvi=1.81]
     54%|█████▍    | 271/500 [00:07<00:06, 32.88it/s, failures=0, hvi=1.81]
     54%|█████▍    | 272/500 [00:07<00:06, 32.88it/s, failures=0, hvi=1.81]
     55%|█████▍    | 273/500 [00:07<00:06, 32.88it/s, failures=0, hvi=1.81]
     55%|█████▍    | 274/500 [00:07<00:06, 33.01it/s, failures=0, hvi=1.81]
     55%|█████▍    | 274/500 [00:07<00:06, 33.01it/s, failures=0, hvi=1.81]
     55%|█████▌    | 275/500 [00:07<00:06, 33.01it/s, failures=0, hvi=1.81]
     55%|█████▌    | 276/500 [00:07<00:06, 33.01it/s, failures=0, hvi=1.81]
     55%|█████▌    | 277/500 [00:07<00:06, 33.01it/s, failures=0, hvi=1.81]
     56%|█████▌    | 278/500 [00:07<00:06, 33.01it/s, failures=0, hvi=1.81]
     56%|█████▌    | 278/500 [00:07<00:06, 33.01it/s, failures=0, hvi=1.81]
     56%|█████▌    | 279/500 [00:07<00:06, 33.01it/s, failures=0, hvi=1.81]
     56%|█████▌    | 280/500 [00:07<00:06, 33.01it/s, failures=0, hvi=1.81]
     56%|█████▌    | 281/500 [00:07<00:06, 33.01it/s, failures=0, hvi=1.81]
     56%|█████▋    | 282/500 [00:07<00:06, 33.01it/s, failures=0, hvi=1.81]
     56%|█████▋    | 282/500 [00:07<00:06, 33.01it/s, failures=0, hvi=1.81]
     57%|█████▋    | 283/500 [00:07<00:06, 33.01it/s, failures=0, hvi=1.81]
     57%|█████▋    | 284/500 [00:07<00:06, 33.01it/s, failures=0, hvi=1.81]
     57%|█████▋    | 285/500 [00:07<00:06, 33.01it/s, failures=0, hvi=1.81]
     57%|█████▋    | 286/500 [00:07<00:06, 32.91it/s, failures=0, hvi=1.81]
     57%|█████▋    | 286/500 [00:07<00:06, 32.91it/s, failures=0, hvi=1.81]
     57%|█████▋    | 287/500 [00:07<00:06, 32.91it/s, failures=0, hvi=1.81]
     58%|█████▊    | 288/500 [00:07<00:06, 32.91it/s, failures=0, hvi=1.81]
     58%|█████▊    | 289/500 [00:07<00:06, 32.91it/s, failures=0, hvi=1.81]
     58%|█████▊    | 290/500 [00:07<00:06, 32.90it/s, failures=0, hvi=1.81]
     58%|█████▊    | 290/500 [00:07<00:06, 32.90it/s, failures=0, hvi=1.81]
     58%|█████▊    | 291/500 [00:07<00:06, 32.90it/s, failures=0, hvi=1.81]
     58%|█████▊    | 292/500 [00:07<00:06, 32.90it/s, failures=0, hvi=1.81]
     59%|█████▊    | 293/500 [00:07<00:06, 32.90it/s, failures=0, hvi=1.81]
     59%|█████▉    | 294/500 [00:07<00:06, 32.85it/s, failures=0, hvi=1.81]
     59%|█████▉    | 294/500 [00:07<00:06, 32.85it/s, failures=0, hvi=1.81]
     59%|█████▉    | 295/500 [00:07<00:06, 32.85it/s, failures=0, hvi=1.81]
     59%|█████▉    | 296/500 [00:07<00:06, 32.85it/s, failures=0, hvi=1.81]
     59%|█████▉    | 297/500 [00:07<00:06, 32.85it/s, failures=0, hvi=1.81]
     60%|█████▉    | 298/500 [00:07<00:06, 32.70it/s, failures=0, hvi=1.81]
     60%|█████▉    | 298/500 [00:07<00:06, 32.70it/s, failures=0, hvi=1.81]
     60%|█████▉    | 299/500 [00:07<00:06, 32.70it/s, failures=0, hvi=1.81]
     60%|██████    | 300/500 [00:07<00:06, 32.70it/s, failures=0, hvi=1.81]
     60%|██████    | 301/500 [00:07<00:06, 32.70it/s, failures=0, hvi=1.81]
     60%|██████    | 302/500 [00:07<00:06, 32.66it/s, failures=0, hvi=1.81]
     60%|██████    | 302/500 [00:07<00:06, 32.66it/s, failures=0, hvi=1.81]
     61%|██████    | 303/500 [00:08<00:06, 32.66it/s, failures=0, hvi=1.89]
     61%|██████    | 304/500 [00:08<00:06, 32.66it/s, failures=0, hvi=1.89]
     61%|██████    | 305/500 [00:08<00:05, 32.66it/s, failures=0, hvi=1.89]
     61%|██████    | 306/500 [00:08<00:05, 32.59it/s, failures=0, hvi=1.89]
     61%|██████    | 306/500 [00:08<00:05, 32.59it/s, failures=0, hvi=1.89]
     61%|██████▏   | 307/500 [00:08<00:05, 32.59it/s, failures=0, hvi=1.89]
     62%|██████▏   | 308/500 [00:08<00:05, 32.59it/s, failures=0, hvi=1.89]
     62%|██████▏   | 309/500 [00:08<00:05, 32.59it/s, failures=0, hvi=1.89]
     62%|██████▏   | 310/500 [00:08<00:05, 32.56it/s, failures=0, hvi=1.89]
     62%|██████▏   | 310/500 [00:08<00:05, 32.56it/s, failures=0, hvi=1.89]
     62%|██████▏   | 311/500 [00:08<00:05, 32.56it/s, failures=0, hvi=1.89]
     62%|██████▏   | 312/500 [00:08<00:05, 32.56it/s, failures=0, hvi=1.89]
     63%|██████▎   | 313/500 [00:08<00:05, 32.56it/s, failures=0, hvi=1.89]
     63%|██████▎   | 314/500 [00:08<00:06, 29.85it/s, failures=0, hvi=1.89]
     63%|██████▎   | 314/500 [00:08<00:06, 29.85it/s, failures=0, hvi=1.89]
     63%|██████▎   | 315/500 [00:08<00:06, 29.85it/s, failures=0, hvi=1.89]
     63%|██████▎   | 316/500 [00:08<00:06, 29.85it/s, failures=0, hvi=1.89]
     63%|██████▎   | 317/500 [00:08<00:06, 29.85it/s, failures=0, hvi=1.89]
     64%|██████▎   | 318/500 [00:08<00:05, 30.46it/s, failures=0, hvi=1.89]
     64%|██████▎   | 318/500 [00:08<00:05, 30.46it/s, failures=0, hvi=1.89]
     64%|██████▍   | 319/500 [00:08<00:05, 30.46it/s, failures=0, hvi=1.89]
     64%|██████▍   | 320/500 [00:08<00:05, 30.46it/s, failures=0, hvi=1.89]
     64%|██████▍   | 321/500 [00:08<00:05, 30.46it/s, failures=0, hvi=1.89]
     64%|██████▍   | 322/500 [00:08<00:05, 30.89it/s, failures=0, hvi=1.89]
     64%|██████▍   | 322/500 [00:08<00:05, 30.89it/s, failures=0, hvi=1.89]
     65%|██████▍   | 323/500 [00:08<00:05, 30.89it/s, failures=0, hvi=1.89]
     65%|██████▍   | 324/500 [00:08<00:05, 30.89it/s, failures=0, hvi=1.89]
     65%|██████▌   | 325/500 [00:08<00:05, 30.89it/s, failures=0, hvi=1.89]
     65%|██████▌   | 326/500 [00:08<00:05, 31.20it/s, failures=0, hvi=1.89]
     65%|██████▌   | 326/500 [00:08<00:05, 31.20it/s, failures=0, hvi=1.89]
     65%|██████▌   | 327/500 [00:08<00:05, 31.20it/s, failures=0, hvi=1.89]
     66%|██████▌   | 328/500 [00:08<00:05, 31.20it/s, failures=0, hvi=1.89]
     66%|██████▌   | 329/500 [00:08<00:05, 31.20it/s, failures=0, hvi=1.89]
     66%|██████▌   | 330/500 [00:08<00:05, 31.38it/s, failures=0, hvi=1.89]
     66%|██████▌   | 330/500 [00:08<00:05, 31.38it/s, failures=0, hvi=1.89]
     66%|██████▌   | 331/500 [00:08<00:05, 31.38it/s, failures=0, hvi=1.89]
     66%|██████▋   | 332/500 [00:08<00:05, 31.38it/s, failures=0, hvi=1.89]
     67%|██████▋   | 333/500 [00:09<00:05, 31.38it/s, failures=0, hvi=1.89]
     67%|██████▋   | 334/500 [00:09<00:05, 31.34it/s, failures=0, hvi=1.89]
     67%|██████▋   | 334/500 [00:09<00:05, 31.34it/s, failures=0, hvi=1.89]
     67%|██████▋   | 335/500 [00:09<00:05, 31.34it/s, failures=0, hvi=1.89]
     67%|██████▋   | 336/500 [00:09<00:05, 31.34it/s, failures=0, hvi=1.89]
     67%|██████▋   | 337/500 [00:09<00:05, 31.34it/s, failures=0, hvi=1.89]
     68%|██████▊   | 338/500 [00:09<00:05, 31.44it/s, failures=0, hvi=1.89]
     68%|██████▊   | 338/500 [00:09<00:05, 31.44it/s, failures=0, hvi=1.89]
     68%|██████▊   | 339/500 [00:09<00:05, 31.44it/s, failures=0, hvi=1.89]
     68%|██████▊   | 340/500 [00:09<00:05, 31.44it/s, failures=0, hvi=1.89]
     68%|██████▊   | 341/500 [00:09<00:05, 31.44it/s, failures=0, hvi=1.89]
     68%|██████▊   | 342/500 [00:09<00:05, 31.52it/s, failures=0, hvi=1.89]
     68%|██████▊   | 342/500 [00:09<00:05, 31.52it/s, failures=0, hvi=1.89]
     69%|██████▊   | 343/500 [00:09<00:04, 31.52it/s, failures=0, hvi=1.89]
     69%|██████▉   | 344/500 [00:09<00:04, 31.52it/s, failures=0, hvi=1.89]
     69%|██████▉   | 345/500 [00:09<00:04, 31.52it/s, failures=0, hvi=1.89]
     69%|██████▉   | 346/500 [00:09<00:04, 31.34it/s, failures=0, hvi=1.89]
     69%|██████▉   | 346/500 [00:09<00:04, 31.34it/s, failures=0, hvi=1.89]
     69%|██████▉   | 347/500 [00:09<00:04, 31.34it/s, failures=0, hvi=1.89]
     70%|██████▉   | 348/500 [00:09<00:04, 31.34it/s, failures=0, hvi=1.89]
     70%|██████▉   | 349/500 [00:09<00:04, 31.34it/s, failures=0, hvi=1.89]
     70%|███████   | 350/500 [00:09<00:04, 31.32it/s, failures=0, hvi=1.89]
     70%|███████   | 350/500 [00:09<00:04, 31.32it/s, failures=0, hvi=1.89]
     70%|███████   | 351/500 [00:09<00:04, 31.32it/s, failures=0, hvi=1.89]
     70%|███████   | 352/500 [00:09<00:04, 31.32it/s, failures=0, hvi=1.89]
     71%|███████   | 353/500 [00:09<00:04, 31.32it/s, failures=0, hvi=1.89]
     71%|███████   | 354/500 [00:09<00:04, 31.36it/s, failures=0, hvi=1.89]
     71%|███████   | 354/500 [00:09<00:04, 31.36it/s, failures=0, hvi=1.89]
     71%|███████   | 355/500 [00:09<00:04, 31.36it/s, failures=0, hvi=1.89]
     71%|███████   | 356/500 [00:09<00:04, 31.36it/s, failures=0, hvi=1.89]
     71%|███████▏  | 357/500 [00:09<00:04, 31.36it/s, failures=0, hvi=1.89]
     72%|███████▏  | 358/500 [00:09<00:04, 31.34it/s, failures=0, hvi=1.89]
     72%|███████▏  | 358/500 [00:09<00:04, 31.34it/s, failures=0, hvi=1.89]
     72%|███████▏  | 359/500 [00:09<00:04, 31.34it/s, failures=0, hvi=1.89]
     72%|███████▏  | 360/500 [00:09<00:04, 31.34it/s, failures=0, hvi=1.89]
     72%|███████▏  | 361/500 [00:09<00:04, 31.34it/s, failures=0, hvi=1.89]
     72%|███████▏  | 362/500 [00:09<00:04, 31.24it/s, failures=0, hvi=1.89]
     72%|███████▏  | 362/500 [00:09<00:04, 31.24it/s, failures=0, hvi=1.89]
     73%|███████▎  | 363/500 [00:09<00:04, 31.24it/s, failures=0, hvi=1.89]
     73%|███████▎  | 364/500 [00:09<00:04, 31.24it/s, failures=0, hvi=1.89]
     73%|███████▎  | 365/500 [00:10<00:04, 31.24it/s, failures=0, hvi=1.89]
     73%|███████▎  | 366/500 [00:10<00:04, 31.13it/s, failures=0, hvi=1.89]
     73%|███████▎  | 366/500 [00:10<00:04, 31.13it/s, failures=0, hvi=1.89]
     73%|███████▎  | 367/500 [00:10<00:04, 31.13it/s, failures=0, hvi=1.89]
     74%|███████▎  | 368/500 [00:10<00:04, 31.13it/s, failures=0, hvi=1.89]
     74%|███████▍  | 369/500 [00:10<00:04, 31.13it/s, failures=0, hvi=1.89]
     74%|███████▍  | 370/500 [00:10<00:04, 31.06it/s, failures=0, hvi=1.89]
     74%|███████▍  | 370/500 [00:10<00:04, 31.06it/s, failures=0, hvi=1.89]
     74%|███████▍  | 371/500 [00:10<00:04, 31.06it/s, failures=0, hvi=1.89]
     74%|███████▍  | 372/500 [00:10<00:04, 31.06it/s, failures=0, hvi=1.89]
     75%|███████▍  | 373/500 [00:10<00:04, 31.06it/s, failures=0, hvi=1.89]
     75%|███████▍  | 374/500 [00:10<00:04, 31.05it/s, failures=0, hvi=1.89]
     75%|███████▍  | 374/500 [00:10<00:04, 31.05it/s, failures=0, hvi=1.89]
     75%|███████▌  | 375/500 [00:10<00:04, 31.05it/s, failures=0, hvi=1.89]
     75%|███████▌  | 376/500 [00:10<00:03, 31.05it/s, failures=0, hvi=1.89]
     75%|███████▌  | 377/500 [00:10<00:03, 31.05it/s, failures=0, hvi=1.89]
     76%|███████▌  | 378/500 [00:10<00:03, 30.96it/s, failures=0, hvi=1.89]
     76%|███████▌  | 378/500 [00:10<00:03, 30.96it/s, failures=0, hvi=1.89]
     76%|███████▌  | 379/500 [00:10<00:03, 30.96it/s, failures=0, hvi=1.89]
     76%|███████▌  | 380/500 [00:10<00:03, 30.96it/s, failures=0, hvi=1.89]
     76%|███████▌  | 381/500 [00:10<00:03, 30.96it/s, failures=0, hvi=1.89]
     76%|███████▋  | 382/500 [00:10<00:04, 28.54it/s, failures=0, hvi=1.89]
     76%|███████▋  | 382/500 [00:10<00:04, 28.54it/s, failures=0, hvi=1.89]
     77%|███████▋  | 383/500 [00:10<00:04, 28.54it/s, failures=0, hvi=1.89]
     77%|███████▋  | 384/500 [00:10<00:04, 28.54it/s, failures=0, hvi=1.89]
     77%|███████▋  | 385/500 [00:10<00:04, 28.54it/s, failures=0, hvi=1.89]
     77%|███████▋  | 386/500 [00:10<00:03, 29.19it/s, failures=0, hvi=1.89]
     77%|███████▋  | 386/500 [00:10<00:03, 29.19it/s, failures=0, hvi=1.89]
     77%|███████▋  | 387/500 [00:10<00:03, 29.19it/s, failures=0, hvi=1.89]
     78%|███████▊  | 388/500 [00:10<00:03, 29.19it/s, failures=0, hvi=1.89]
     78%|███████▊  | 389/500 [00:10<00:03, 29.19it/s, failures=0, hvi=1.89]
     78%|███████▊  | 390/500 [00:10<00:03, 29.67it/s, failures=0, hvi=1.89]
     78%|███████▊  | 390/500 [00:10<00:03, 29.67it/s, failures=0, hvi=1.89]
     78%|███████▊  | 391/500 [00:10<00:03, 29.67it/s, failures=0, hvi=1.89]
     78%|███████▊  | 392/500 [00:10<00:03, 29.67it/s, failures=0, hvi=1.89]
     79%|███████▊  | 393/500 [00:10<00:03, 29.67it/s, failures=0, hvi=1.89]
     79%|███████▉  | 394/500 [00:11<00:03, 29.95it/s, failures=0, hvi=1.89]
     79%|███████▉  | 394/500 [00:11<00:03, 29.95it/s, failures=0, hvi=1.89]
     79%|███████▉  | 395/500 [00:11<00:03, 29.95it/s, failures=0, hvi=1.89]
     79%|███████▉  | 396/500 [00:11<00:03, 29.95it/s, failures=0, hvi=1.9] 
     79%|███████▉  | 397/500 [00:11<00:03, 29.95it/s, failures=0, hvi=1.9]
     80%|███████▉  | 398/500 [00:11<00:03, 30.08it/s, failures=0, hvi=1.9]
     80%|███████▉  | 398/500 [00:11<00:03, 30.08it/s, failures=0, hvi=1.9]
     80%|███████▉  | 399/500 [00:11<00:03, 30.08it/s, failures=0, hvi=1.9]
     80%|████████  | 400/500 [00:11<00:03, 30.08it/s, failures=0, hvi=1.9]
     80%|████████  | 401/500 [00:11<00:03, 30.08it/s, failures=0, hvi=1.9]
     80%|████████  | 402/500 [00:11<00:03, 30.17it/s, failures=0, hvi=1.9]
     80%|████████  | 402/500 [00:11<00:03, 30.17it/s, failures=0, hvi=1.9]
     81%|████████  | 403/500 [00:11<00:03, 30.17it/s, failures=0, hvi=1.9]
     81%|████████  | 404/500 [00:11<00:03, 30.17it/s, failures=0, hvi=1.9]
     81%|████████  | 405/500 [00:11<00:03, 30.17it/s, failures=0, hvi=1.9]
     81%|████████  | 406/500 [00:11<00:03, 30.20it/s, failures=0, hvi=1.9]
     81%|████████  | 406/500 [00:11<00:03, 30.20it/s, failures=0, hvi=1.9]
     81%|████████▏ | 407/500 [00:11<00:03, 30.20it/s, failures=0, hvi=1.91]
     82%|████████▏ | 408/500 [00:11<00:03, 30.20it/s, failures=0, hvi=1.91]
     82%|████████▏ | 409/500 [00:11<00:03, 30.20it/s, failures=0, hvi=1.91]
     82%|████████▏ | 410/500 [00:11<00:02, 30.19it/s, failures=0, hvi=1.91]
     82%|████████▏ | 410/500 [00:11<00:02, 30.19it/s, failures=0, hvi=1.91]
     82%|████████▏ | 411/500 [00:11<00:02, 30.19it/s, failures=0, hvi=1.91]
     82%|████████▏ | 412/500 [00:11<00:02, 30.19it/s, failures=0, hvi=1.91]
     83%|████████▎ | 413/500 [00:11<00:02, 30.19it/s, failures=0, hvi=1.91]
     83%|████████▎ | 414/500 [00:11<00:02, 30.16it/s, failures=0, hvi=1.91]
     83%|████████▎ | 414/500 [00:11<00:02, 30.16it/s, failures=0, hvi=1.91]
     83%|████████▎ | 415/500 [00:11<00:02, 30.16it/s, failures=0, hvi=1.91]
     83%|████████▎ | 416/500 [00:11<00:02, 30.16it/s, failures=0, hvi=1.91]
     83%|████████▎ | 417/500 [00:11<00:02, 30.16it/s, failures=0, hvi=1.91]
     84%|████████▎ | 418/500 [00:11<00:02, 30.16it/s, failures=0, hvi=1.91]
     84%|████████▎ | 418/500 [00:11<00:02, 30.16it/s, failures=0, hvi=1.91]
     84%|████████▍ | 419/500 [00:11<00:02, 30.16it/s, failures=0, hvi=1.91]
     84%|████████▍ | 420/500 [00:11<00:02, 30.16it/s, failures=0, hvi=1.91]
     84%|████████▍ | 421/500 [00:11<00:02, 30.16it/s, failures=0, hvi=1.91]
     84%|████████▍ | 422/500 [00:11<00:02, 30.13it/s, failures=0, hvi=1.91]
     84%|████████▍ | 422/500 [00:11<00:02, 30.13it/s, failures=0, hvi=1.91]
     85%|████████▍ | 423/500 [00:11<00:02, 30.13it/s, failures=0, hvi=1.91]
     85%|████████▍ | 424/500 [00:11<00:02, 30.13it/s, failures=0, hvi=1.91]
     85%|████████▌ | 425/500 [00:12<00:02, 30.13it/s, failures=0, hvi=1.91]
     85%|████████▌ | 426/500 [00:12<00:02, 30.09it/s, failures=0, hvi=1.91]
     85%|████████▌ | 426/500 [00:12<00:02, 30.09it/s, failures=0, hvi=1.91]
     85%|████████▌ | 427/500 [00:12<00:02, 30.09it/s, failures=0, hvi=1.91]
     86%|████████▌ | 428/500 [00:12<00:02, 30.09it/s, failures=0, hvi=1.91]
     86%|████████▌ | 429/500 [00:12<00:02, 30.09it/s, failures=0, hvi=1.91]
     86%|████████▌ | 430/500 [00:12<00:02, 29.99it/s, failures=0, hvi=1.91]
     86%|████████▌ | 430/500 [00:12<00:02, 29.99it/s, failures=0, hvi=1.91]
     86%|████████▌ | 431/500 [00:12<00:02, 29.99it/s, failures=0, hvi=1.91]
     86%|████████▋ | 432/500 [00:12<00:02, 29.99it/s, failures=0, hvi=1.91]
     87%|████████▋ | 433/500 [00:12<00:02, 29.95it/s, failures=0, hvi=1.91]
     87%|████████▋ | 433/500 [00:12<00:02, 29.95it/s, failures=0, hvi=1.91]
     87%|████████▋ | 434/500 [00:12<00:02, 29.95it/s, failures=0, hvi=1.91]
     87%|████████▋ | 435/500 [00:12<00:02, 29.95it/s, failures=0, hvi=1.91]
     87%|████████▋ | 436/500 [00:12<00:02, 29.85it/s, failures=0, hvi=1.91]
     87%|████████▋ | 436/500 [00:12<00:02, 29.85it/s, failures=0, hvi=1.91]
     87%|████████▋ | 437/500 [00:12<00:02, 29.85it/s, failures=0, hvi=1.91]
     88%|████████▊ | 438/500 [00:12<00:02, 29.85it/s, failures=0, hvi=1.91]
     88%|████████▊ | 439/500 [00:12<00:02, 29.70it/s, failures=0, hvi=1.91]
     88%|████████▊ | 439/500 [00:12<00:02, 29.70it/s, failures=0, hvi=1.91]
     88%|████████▊ | 440/500 [00:12<00:02, 29.70it/s, failures=0, hvi=1.91]
     88%|████████▊ | 441/500 [00:12<00:01, 29.70it/s, failures=0, hvi=1.91]
     88%|████████▊ | 442/500 [00:12<00:01, 29.69it/s, failures=0, hvi=1.91]
     88%|████████▊ | 442/500 [00:12<00:01, 29.69it/s, failures=0, hvi=1.91]
     89%|████████▊ | 443/500 [00:12<00:01, 29.69it/s, failures=0, hvi=1.91]
     89%|████████▉ | 444/500 [00:12<00:01, 29.69it/s, failures=0, hvi=1.91]
     89%|████████▉ | 445/500 [00:12<00:01, 29.68it/s, failures=0, hvi=1.91]
     89%|████████▉ | 445/500 [00:12<00:01, 29.68it/s, failures=0, hvi=1.91]
     89%|████████▉ | 446/500 [00:12<00:01, 29.68it/s, failures=0, hvi=1.91]
     89%|████████▉ | 447/500 [00:12<00:01, 29.68it/s, failures=0, hvi=1.91]
     90%|████████▉ | 448/500 [00:12<00:01, 26.82it/s, failures=0, hvi=1.91]
     90%|████████▉ | 448/500 [00:12<00:01, 26.82it/s, failures=0, hvi=1.91]
     90%|████████▉ | 449/500 [00:12<00:01, 26.82it/s, failures=0, hvi=1.91]
     90%|█████████ | 450/500 [00:12<00:01, 26.82it/s, failures=0, hvi=1.91]
     90%|█████████ | 451/500 [00:12<00:01, 27.50it/s, failures=0, hvi=1.91]
     90%|█████████ | 451/500 [00:12<00:01, 27.50it/s, failures=0, hvi=1.91]
     90%|█████████ | 452/500 [00:12<00:01, 27.50it/s, failures=0, hvi=1.91]
     91%|█████████ | 453/500 [00:13<00:01, 27.50it/s, failures=0, hvi=1.91]
     91%|█████████ | 454/500 [00:13<00:01, 27.89it/s, failures=0, hvi=1.91]
     91%|█████████ | 454/500 [00:13<00:01, 27.89it/s, failures=0, hvi=1.91]
     91%|█████████ | 455/500 [00:13<00:01, 27.89it/s, failures=0, hvi=1.91]
     91%|█████████ | 456/500 [00:13<00:01, 27.89it/s, failures=0, hvi=1.91]
     91%|█████████▏| 457/500 [00:13<00:01, 28.27it/s, failures=0, hvi=1.91]
     91%|█████████▏| 457/500 [00:13<00:01, 28.27it/s, failures=0, hvi=1.91]
     92%|█████████▏| 458/500 [00:13<00:01, 28.27it/s, failures=0, hvi=1.91]
     92%|█████████▏| 459/500 [00:13<00:01, 28.27it/s, failures=0, hvi=1.91]
     92%|█████████▏| 460/500 [00:13<00:01, 28.55it/s, failures=0, hvi=1.91]
     92%|█████████▏| 460/500 [00:13<00:01, 28.55it/s, failures=0, hvi=1.91]
     92%|█████████▏| 461/500 [00:13<00:01, 28.55it/s, failures=0, hvi=1.91]
     92%|█████████▏| 462/500 [00:13<00:01, 28.55it/s, failures=0, hvi=1.91]
     93%|█████████▎| 463/500 [00:13<00:01, 28.76it/s, failures=0, hvi=1.91]
     93%|█████████▎| 463/500 [00:13<00:01, 28.76it/s, failures=0, hvi=1.91]
     93%|█████████▎| 464/500 [00:13<00:01, 28.76it/s, failures=0, hvi=1.91]
     93%|█████████▎| 465/500 [00:13<00:01, 28.76it/s, failures=0, hvi=1.91]
     93%|█████████▎| 466/500 [00:13<00:01, 28.89it/s, failures=0, hvi=1.91]
     93%|█████████▎| 466/500 [00:13<00:01, 28.89it/s, failures=0, hvi=1.91]
     93%|█████████▎| 467/500 [00:13<00:01, 28.89it/s, failures=0, hvi=1.91]
     94%|█████████▎| 468/500 [00:13<00:01, 28.89it/s, failures=0, hvi=1.91]
     94%|█████████▍| 469/500 [00:13<00:01, 28.93it/s, failures=0, hvi=1.91]
     94%|█████████▍| 469/500 [00:13<00:01, 28.93it/s, failures=0, hvi=1.91]
     94%|█████████▍| 470/500 [00:13<00:01, 28.93it/s, failures=0, hvi=1.91]
     94%|█████████▍| 471/500 [00:13<00:01, 28.93it/s, failures=0, hvi=1.91]
     94%|█████████▍| 472/500 [00:13<00:00, 28.95it/s, failures=0, hvi=1.91]
     94%|█████████▍| 472/500 [00:13<00:00, 28.95it/s, failures=0, hvi=1.91]
     95%|█████████▍| 473/500 [00:13<00:00, 28.95it/s, failures=0, hvi=1.91]
     95%|█████████▍| 474/500 [00:13<00:00, 28.95it/s, failures=0, hvi=1.91]
     95%|█████████▌| 475/500 [00:13<00:00, 29.04it/s, failures=0, hvi=1.91]
     95%|█████████▌| 475/500 [00:13<00:00, 29.04it/s, failures=0, hvi=1.91]
     95%|█████████▌| 476/500 [00:13<00:00, 29.04it/s, failures=0, hvi=1.91]
     95%|█████████▌| 477/500 [00:13<00:00, 29.04it/s, failures=0, hvi=1.91]
     96%|█████████▌| 478/500 [00:13<00:00, 28.95it/s, failures=0, hvi=1.91]
     96%|█████████▌| 478/500 [00:13<00:00, 28.95it/s, failures=0, hvi=1.91]
     96%|█████████▌| 479/500 [00:13<00:00, 28.95it/s, failures=0, hvi=1.91]
     96%|█████████▌| 480/500 [00:13<00:00, 28.95it/s, failures=0, hvi=1.91]
     96%|█████████▌| 481/500 [00:13<00:00, 28.96it/s, failures=0, hvi=1.91]
     96%|█████████▌| 481/500 [00:13<00:00, 28.96it/s, failures=0, hvi=1.91]
     96%|█████████▋| 482/500 [00:14<00:00, 28.96it/s, failures=0, hvi=1.91]
     97%|█████████▋| 483/500 [00:14<00:00, 28.96it/s, failures=0, hvi=1.91]
     97%|█████████▋| 484/500 [00:14<00:00, 28.86it/s, failures=0, hvi=1.91]
     97%|█████████▋| 484/500 [00:14<00:00, 28.86it/s, failures=0, hvi=1.91]
     97%|█████████▋| 485/500 [00:14<00:00, 28.86it/s, failures=0, hvi=1.91]
     97%|█████████▋| 486/500 [00:14<00:00, 28.86it/s, failures=0, hvi=1.91]
     97%|█████████▋| 487/500 [00:14<00:00, 28.78it/s, failures=0, hvi=1.91]
     97%|█████████▋| 487/500 [00:14<00:00, 28.78it/s, failures=0, hvi=1.91]
     98%|█████████▊| 488/500 [00:14<00:00, 28.78it/s, failures=0, hvi=1.91]
     98%|█████████▊| 489/500 [00:14<00:00, 28.78it/s, failures=0, hvi=1.91]
     98%|█████████▊| 490/500 [00:14<00:00, 28.76it/s, failures=0, hvi=1.91]
     98%|█████████▊| 490/500 [00:14<00:00, 28.76it/s, failures=0, hvi=1.91]
     98%|█████████▊| 491/500 [00:14<00:00, 28.76it/s, failures=0, hvi=1.91]
     98%|█████████▊| 492/500 [00:14<00:00, 28.76it/s, failures=0, hvi=1.91]
     99%|█████████▊| 493/500 [00:14<00:00, 28.76it/s, failures=0, hvi=1.91]
     99%|█████████▊| 493/500 [00:14<00:00, 28.76it/s, failures=0, hvi=1.91]
     99%|█████████▉| 494/500 [00:14<00:00, 28.76it/s, failures=0, hvi=1.91]
     99%|█████████▉| 495/500 [00:14<00:00, 28.76it/s, failures=0, hvi=1.91]
     99%|█████████▉| 496/500 [00:14<00:00, 28.73it/s, failures=0, hvi=1.91]
     99%|█████████▉| 496/500 [00:14<00:00, 28.73it/s, failures=0, hvi=1.91]
     99%|█████████▉| 497/500 [00:14<00:00, 28.73it/s, failures=0, hvi=1.91]
    100%|█████████▉| 498/500 [00:14<00:00, 28.73it/s, failures=0, hvi=1.91]
    100%|█████████▉| 499/500 [00:14<00:00, 28.61it/s, failures=0, hvi=1.91]
    100%|█████████▉| 499/500 [00:14<00:00, 28.61it/s, failures=0, hvi=1.91]
    100%|██████████| 500/500 [00:14<00:00, 28.61it/s, failures=0, hvi=1.91]
    100%|██████████| 500/500 [00:14<00:00, 34.15it/s, failures=0, hvi=1.91]




.. GENERATED FROM PYTHON SOURCE LINES 99-100

A Pandas table of results is returned by the search and also saved at ``./results.csv``. An other location can be specified by using ``CBO(..., log_dir=...)``.

.. GENERATED FROM PYTHON SOURCE LINES 100-103

.. code-block:: Python


    results






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>p:x0</th>
          <th>p:x1</th>
          <th>p:x2</th>
          <th>p:x3</th>
          <th>p:x4</th>
          <th>objective_0</th>
          <th>objective_1</th>
          <th>job_id</th>
          <th>job_status</th>
          <th>m:timestamp_submit</th>
          <th>m:timestamp_start</th>
          <th>m:timestamp_end</th>
          <th>m:timestamp_gather</th>
          <th>pareto_efficient</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>0.132669</td>
          <td>0.013103</td>
          <td>0.000849</td>
          <td>0.028166</td>
          <td>0.477660</td>
          <td>-1.672363</td>
          <td>-0.353647</td>
          <td>0</td>
          <td>DONE</td>
          <td>0.009166</td>
          <td>1.755518e+09</td>
          <td>1.755518e+09</td>
          <td>0.009453</td>
          <td>False</td>
        </tr>
        <tr>
          <th>1</th>
          <td>0.445254</td>
          <td>0.587523</td>
          <td>0.731304</td>
          <td>0.203790</td>
          <td>0.529300</td>
          <td>-0.879828</td>
          <td>-0.740171</td>
          <td>1</td>
          <td>DONE</td>
          <td>0.030374</td>
          <td>1.755518e+09</td>
          <td>1.755518e+09</td>
          <td>0.030762</td>
          <td>False</td>
        </tr>
        <tr>
          <th>2</th>
          <td>0.992770</td>
          <td>0.923032</td>
          <td>0.260705</td>
          <td>0.623351</td>
          <td>0.312814</td>
          <td>-0.014610</td>
          <td>-1.286390</td>
          <td>2</td>
          <td>DONE</td>
          <td>0.043637</td>
          <td>1.755518e+09</td>
          <td>1.755518e+09</td>
          <td>0.043849</td>
          <td>False</td>
        </tr>
        <tr>
          <th>3</th>
          <td>0.433051</td>
          <td>0.933044</td>
          <td>0.576779</td>
          <td>0.958234</td>
          <td>0.830414</td>
          <td>-1.175913</td>
          <td>-0.951372</td>
          <td>3</td>
          <td>DONE</td>
          <td>0.056278</td>
          <td>1.755518e+09</td>
          <td>1.755518e+09</td>
          <td>0.056461</td>
          <td>False</td>
        </tr>
        <tr>
          <th>4</th>
          <td>0.992771</td>
          <td>0.156779</td>
          <td>0.224604</td>
          <td>0.442812</td>
          <td>0.136333</td>
          <td>-0.015093</td>
          <td>-1.329082</td>
          <td>4</td>
          <td>DONE</td>
          <td>0.068399</td>
          <td>1.755518e+09</td>
          <td>1.755518e+09</td>
          <td>0.068564</td>
          <td>False</td>
        </tr>
        <tr>
          <th>...</th>
          <td>...</td>
          <td>...</td>
          <td>...</td>
          <td>...</td>
          <td>...</td>
          <td>...</td>
          <td>...</td>
          <td>...</td>
          <td>...</td>
          <td>...</td>
          <td>...</td>
          <td>...</td>
          <td>...</td>
          <td>...</td>
        </tr>
        <tr>
          <th>495</th>
          <td>0.799012</td>
          <td>0.598050</td>
          <td>0.108476</td>
          <td>0.492106</td>
          <td>0.515866</td>
          <td>-0.361171</td>
          <td>-1.105727</td>
          <td>495</td>
          <td>DONE</td>
          <td>14.512292</td>
          <td>1.755518e+09</td>
          <td>1.755518e+09</td>
          <td>14.512488</td>
          <td>False</td>
        </tr>
        <tr>
          <th>496</th>
          <td>0.079906</td>
          <td>0.509656</td>
          <td>0.656180</td>
          <td>0.393031</td>
          <td>0.398593</td>
          <td>-1.037981</td>
          <td>-0.130973</td>
          <td>496</td>
          <td>DONE</td>
          <td>14.547404</td>
          <td>1.755518e+09</td>
          <td>1.755518e+09</td>
          <td>14.547589</td>
          <td>False</td>
        </tr>
        <tr>
          <th>497</th>
          <td>0.692482</td>
          <td>0.404241</td>
          <td>0.645611</td>
          <td>0.424970</td>
          <td>0.516073</td>
          <td>-0.481323</td>
          <td>-0.917695</td>
          <td>497</td>
          <td>DONE</td>
          <td>14.582559</td>
          <td>1.755518e+09</td>
          <td>1.755518e+09</td>
          <td>14.582775</td>
          <td>False</td>
        </tr>
        <tr>
          <th>498</th>
          <td>0.354439</td>
          <td>0.456215</td>
          <td>0.494466</td>
          <td>0.542073</td>
          <td>0.537370</td>
          <td>-0.853318</td>
          <td>-0.531134</td>
          <td>498</td>
          <td>DONE</td>
          <td>14.618182</td>
          <td>1.755518e+09</td>
          <td>1.755518e+09</td>
          <td>14.618367</td>
          <td>True</td>
        </tr>
        <tr>
          <th>499</th>
          <td>0.798545</td>
          <td>0.551645</td>
          <td>0.450832</td>
          <td>0.392065</td>
          <td>0.334940</td>
          <td>-0.324875</td>
          <td>-0.992144</td>
          <td>499</td>
          <td>DONE</td>
          <td>14.652979</td>
          <td>1.755518e+09</td>
          <td>1.755518e+09</td>
          <td>14.653161</td>
          <td>False</td>
        </tr>
      </tbody>
    </table>
    <p>500 rows × 14 columns</p>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 104-112

In this table we retrieve:

- columns starting by ``p:`` which are the optimized variables.
- the ``objective_{i}`` are the objectives returned by the black-box function.
- the ``job_id`` is the identifier of the executed evaluations.
- columns starting by ``m:`` are metadata returned by the black-box function.
- ``pareto_efficient`` is a column only returned for MOO which specify if the evaluation is part of the set of optimal solutions.


.. GENERATED FROM PYTHON SOURCE LINES 114-116

Let us use this table to visualize evaluated objectives.
The estimated optimal solutions will be colored in red.

.. GENERATED FROM PYTHON SOURCE LINES 116-140

.. dropdown:: Code (Plot evaluated objectives)

    .. code-block:: Python


        fig, ax = plt.subplots(figsize=(WIDTH_PLOTS, HEIGHT_PLOTS), tight_layout=True)
        _ = ax.plot(
            -results[~results["pareto_efficient"]]["objective_0"],
            -results[~results["pareto_efficient"]]["objective_1"],
            "o",
            color="blue",
            alpha=0.7,
            label="Non Pareto-Efficient",
        )
        _ = ax.plot(
            -results[results["pareto_efficient"]]["objective_0"],
            -results[results["pareto_efficient"]]["objective_1"],
            "o",
            color="red",
            alpha=0.7,
            label="Pareto-Efficient",
        )
        _ = ax.grid()
        _ = ax.legend()
        _ = ax.set_xlabel("Objective 0")
        _ = ax.set_ylabel("Objective 1")




.. image-sg:: /examples/examples_bbo/images/sphx_glr_plot_multi_objective_optimization_001.png
   :alt: plot multi objective optimization
   :srcset: /examples/examples_bbo/images/sphx_glr_plot_multi_objective_optimization_001.png
   :class: sphx-glr-single-img





.. GENERATED FROM PYTHON SOURCE LINES 141-143

Let us look the evolution of the hypervolume indicator.
This metric should increase over time.

.. GENERATED FROM PYTHON SOURCE LINES 143-152

.. dropdown:: Code (Plot hypervolume)

    .. code-block:: Python


        scorer = bench.scorer
        hvi = scorer.hypervolume(results[["objective_0", "objective_1"]].values)
        x = list(range(1, len(hvi)+1))
        fig, ax = plt.subplots(figsize=(WIDTH_PLOTS, HEIGHT_PLOTS), tight_layout=True)
        _ = ax.plot(x, hvi)
        _ = ax.grid()
        _ = ax.set_xlabel("Evaluations")
        _ = ax.set_ylabel("Hypervolume Indicator")


.. image-sg:: /examples/examples_bbo/images/sphx_glr_plot_multi_objective_optimization_002.png
   :alt: plot multi objective optimization
   :srcset: /examples/examples_bbo/images/sphx_glr_plot_multi_objective_optimization_002.png
   :class: sphx-glr-single-img






.. rst-class:: sphx-glr-timing

   **Total running time of the script:** (0 minutes 15.918 seconds)


.. _sphx_glr_download_examples_examples_bbo_plot_multi_objective_optimization.py:

.. only:: html

  .. container:: sphx-glr-footer sphx-glr-footer-example

    .. container:: sphx-glr-download sphx-glr-download-jupyter

      :download:`Download Jupyter notebook: plot_multi_objective_optimization.ipynb <plot_multi_objective_optimization.ipynb>`

    .. container:: sphx-glr-download sphx-glr-download-python

      :download:`Download Python source code: plot_multi_objective_optimization.py <plot_multi_objective_optimization.py>`

    .. container:: sphx-glr-download sphx-glr-download-zip

      :download:`Download zipped: plot_multi_objective_optimization.zip <plot_multi_objective_optimization.zip>`


.. only:: html

 .. rst-class:: sphx-glr-signature

    `Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_