Notify Failures in Hyperparameter optimization

Note

Go to the end to download the full example code.

Notify Failures in Hyperparameter optimization#

Author(s): Romain Egele.

In this example, you will learn how to handle failures in black-box optimization. In many application of black-box optimization such as software auto-tuning (where we minimize the run-time of a software application) some configurations can create run-time errors and therefore no scalar objective is returned. A default choice could be to return in this case the worst case objective if known. Other possibilites are to ignore these configurations or to replace them with the running average or minimum objective.

Code (Import statements)
import matplotlib.pyplot as plt
import numpy as np

from deephyper.hpo import HpProblem
from deephyper.hpo import CBO
from deephyper.evaluator import Evaluator
from deephyper.evaluator.callback import TqdmCallback

WIDTH_PLOTS = 8
HEIGHT_PLOTS = WIDTH_PLOTS / 1.618

To illustrate such a use-case we define a run-function that will fail when its input parameter p_failure is greater than 0.5. To notify deephyper about a failure, we return a “string” value with "F" as prefix such as:

def run(job) -> float:
    if job.parameters["p_failure"] > 0.5:
        try:
            raise ValueError("Some example exception")
        except ValueError:
            # Notify the search about the failure
            return "F_value_error"
    else:
        # Returns a regular objective value that is maximized
        return sum(job.parameters[k] for k in job.parameters if "x" in k)

Then, we define the corresponding hyperparameter problem where x{i} are the value to maximize and p_parameter is a value that impact the appearance of failures.

problem = HpProblem()
problem.add_hyperparameter((0.0, 1.0), "p_failure")
for i in range(10):
    problem.add_hyperparameter((0.0, 1.0), f"x{i}")
problem

Configuration space object:
  Hyperparameters:
    p_failure, Type: UniformFloat, Range: [0.0, 1.0], Default: 0.5
    x0, Type: UniformFloat, Range: [0.0, 1.0], Default: 0.5
    x1, Type: UniformFloat, Range: [0.0, 1.0], Default: 0.5
    x2, Type: UniformFloat, Range: [0.0, 1.0], Default: 0.5
    x3, Type: UniformFloat, Range: [0.0, 1.0], Default: 0.5
    x4, Type: UniformFloat, Range: [0.0, 1.0], Default: 0.5
    x5, Type: UniformFloat, Range: [0.0, 1.0], Default: 0.5
    x6, Type: UniformFloat, Range: [0.0, 1.0], Default: 0.5
    x7, Type: UniformFloat, Range: [0.0, 1.0], Default: 0.5
    x8, Type: UniformFloat, Range: [0.0, 1.0], Default: 0.5
    x9, Type: UniformFloat, Range: [0.0, 1.0], Default: 0.5

We use the centralized Bayesian optimization (CBO) for the search. The deephyper.hpo.CBO has a parameter filter_failures. We will compare:

  • filter_failures="ignore": filters-out failed configurations.

  • filter_failures="mean": replaces failures with the running average of non-failed objectives.

  • filter_failures="min": replaces failures with the running minimum of non-failed objectives.

results = {}
max_evals = 100

for failure_strategy in ["ignore", "mean", "min"]:

    evaluator = Evaluator.create(
        run,
        method="thread",
        method_kwargs={
            "callbacks": [TqdmCallback(f"Failure Strategy: {failure_strategy}")]}
    )

    search = CBO(
        problem,
        evaluator,
        acq_optimizer="ga",
        acq_optimizer_kwargs=dict(
            filter_duplicated=False,
            filter_failures=failure_strategy,
        ),
        log_dir=f"search_{failure_strategy}",
        random_state=42,
    )

    results[failure_strategy] = search.search(max_evals)

  0%|          | 0/100 [00:00<?, ?it/s]
Failure Strategy: ignore:   0%|          | 0/100 [00:00<?, ?it/s]
Failure Strategy: ignore:   1%|          | 1/100 [00:00<00:00, 4064.25it/s, failures=0, objective=6.63]
Failure Strategy: ignore:   2%|▏         | 2/100 [00:00<00:00, 221.58it/s, failures=1, objective=6.63]
Failure Strategy: ignore:   3%|▎         | 3/100 [00:00<00:00, 178.68it/s, failures=2, objective=6.63]
Failure Strategy: ignore:   4%|▍         | 4/100 [00:00<00:00, 155.01it/s, failures=2, objective=6.63]
Failure Strategy: ignore:   5%|▌         | 5/100 [00:00<00:00, 152.67it/s, failures=2, objective=6.63]
Failure Strategy: ignore:   6%|▌         | 6/100 [00:00<00:00, 148.48it/s, failures=3, objective=6.63]
Failure Strategy: ignore:   7%|▋         | 7/100 [00:00<00:00, 147.04it/s, failures=4, objective=6.63]
Failure Strategy: ignore:   8%|▊         | 8/100 [00:00<00:00, 146.54it/s, failures=4, objective=6.63]
Failure Strategy: ignore:   9%|▉         | 9/100 [00:00<00:00, 146.53it/s, failures=5, objective=6.63]
Failure Strategy: ignore:  10%|█         | 10/100 [00:00<00:00, 146.72it/s, failures=5, objective=6.63]
Failure Strategy: ignore:  11%|█         | 11/100 [00:00<00:00, 146.13it/s, failures=5, objective=6.63]
Failure Strategy: ignore:  12%|█▏        | 12/100 [00:00<00:00, 145.38it/s, failures=6, objective=6.63]
Failure Strategy: ignore:  13%|█▎        | 13/100 [00:00<00:00, 145.46it/s, failures=6, objective=6.63]
Failure Strategy: ignore:  14%|█▍        | 14/100 [00:00<00:00, 145.38it/s, failures=7, objective=6.63]
Failure Strategy: ignore:  15%|█▌        | 15/100 [00:00<00:00, 145.50it/s, failures=7, objective=6.63]
Failure Strategy: ignore:  15%|█▌        | 15/100 [00:00<00:00, 145.50it/s, failures=8, objective=6.63]
Failure Strategy: ignore:  16%|█▌        | 16/100 [00:00<00:00, 145.50it/s, failures=9, objective=6.63]
Failure Strategy: ignore:  17%|█▋        | 17/100 [00:00<00:00, 145.50it/s, failures=10, objective=6.63]
Failure Strategy: ignore:  18%|█▊        | 18/100 [00:00<00:00, 145.50it/s, failures=11, objective=6.63]
Failure Strategy: ignore:  19%|█▉        | 19/100 [00:00<00:00, 145.50it/s, failures=12, objective=6.63]
Failure Strategy: ignore:  20%|██        | 20/100 [00:00<00:00, 145.50it/s, failures=13, objective=6.63]
Failure Strategy: ignore:  21%|██        | 21/100 [00:00<00:00, 145.50it/s, failures=14, objective=6.63]
Failure Strategy: ignore:  22%|██▏       | 22/100 [00:00<00:00, 145.50it/s, failures=15, objective=6.63]
Failure Strategy: ignore:  23%|██▎       | 23/100 [00:00<00:00, 145.50it/s, failures=16, objective=6.63]
Failure Strategy: ignore:  24%|██▍       | 24/100 [00:00<00:00, 145.50it/s, failures=16, objective=6.63]
Failure Strategy: ignore:  25%|██▌       | 25/100 [00:00<00:00, 145.50it/s, failures=16, objective=6.63]
Failure Strategy: ignore:  26%|██▌       | 26/100 [00:00<00:00, 145.50it/s, failures=17, objective=6.63]
Failure Strategy: ignore:  27%|██▋       | 27/100 [00:00<00:00, 145.50it/s, failures=17, objective=6.63]
Failure Strategy: ignore:  28%|██▊       | 28/100 [00:00<00:00, 145.50it/s, failures=18, objective=6.63]
Failure Strategy: ignore:  29%|██▉       | 29/100 [00:00<00:00, 145.50it/s, failures=19, objective=6.63]
Failure Strategy: ignore:  30%|███       | 30/100 [00:00<00:00, 142.10it/s, failures=19, objective=6.63]
Failure Strategy: ignore:  30%|███       | 30/100 [00:00<00:00, 142.10it/s, failures=20, objective=6.63]
Failure Strategy: ignore:  31%|███       | 31/100 [00:00<00:00, 142.10it/s, failures=20, objective=6.63]
Failure Strategy: ignore:  32%|███▏      | 32/100 [00:00<00:00, 142.10it/s, failures=21, objective=6.63]
Failure Strategy: ignore:  33%|███▎      | 33/100 [00:00<00:00, 142.10it/s, failures=21, objective=6.63]
Failure Strategy: ignore:  34%|███▍      | 34/100 [00:00<00:00, 142.10it/s, failures=21, objective=6.63]
Failure Strategy: ignore:  35%|███▌      | 35/100 [00:00<00:00, 142.10it/s, failures=21, objective=6.63]
Failure Strategy: ignore:  36%|███▌      | 36/100 [00:00<00:00, 142.10it/s, failures=21, objective=6.63]
Failure Strategy: ignore:  37%|███▋      | 37/100 [00:00<00:00, 142.10it/s, failures=22, objective=6.63]
Failure Strategy: ignore:  38%|███▊      | 38/100 [00:00<00:00, 142.10it/s, failures=22, objective=6.63]
Failure Strategy: ignore:  39%|███▉      | 39/100 [00:00<00:00, 142.10it/s, failures=22, objective=6.87]
Failure Strategy: ignore:  40%|████      | 40/100 [00:00<00:00, 142.10it/s, failures=22, objective=6.87]
Failure Strategy: ignore:  41%|████      | 41/100 [00:00<00:00, 142.10it/s, failures=22, objective=6.87]
Failure Strategy: ignore:  42%|████▏     | 42/100 [00:00<00:00, 142.10it/s, failures=22, objective=6.87]
Failure Strategy: ignore:  43%|████▎     | 43/100 [00:00<00:00, 142.10it/s, failures=23, objective=6.87]
Failure Strategy: ignore:  44%|████▍     | 44/100 [00:00<00:00, 142.10it/s, failures=23, objective=6.87]
Failure Strategy: ignore:  45%|████▌     | 45/100 [00:00<00:00, 141.04it/s, failures=23, objective=6.87]
Failure Strategy: ignore:  45%|████▌     | 45/100 [00:00<00:00, 141.04it/s, failures=24, objective=6.87]
Failure Strategy: ignore:  46%|████▌     | 46/100 [00:00<00:00, 141.04it/s, failures=24, objective=6.87]
Failure Strategy: ignore:  47%|████▋     | 47/100 [00:00<00:00, 141.04it/s, failures=25, objective=6.87]
Failure Strategy: ignore:  48%|████▊     | 48/100 [00:00<00:00, 141.04it/s, failures=25, objective=6.87]
Failure Strategy: ignore:  49%|████▉     | 49/100 [00:00<00:00, 141.04it/s, failures=26, objective=6.87]
Failure Strategy: ignore:  50%|█████     | 50/100 [00:00<00:00, 141.04it/s, failures=27, objective=6.87]
Failure Strategy: ignore:  51%|█████     | 51/100 [00:00<00:00, 141.04it/s, failures=28, objective=6.87]
Failure Strategy: ignore:  52%|█████▏    | 52/100 [00:00<00:00, 141.04it/s, failures=29, objective=6.87]
Failure Strategy: ignore:  53%|█████▎    | 53/100 [00:00<00:00, 141.04it/s, failures=30, objective=6.87]
Failure Strategy: ignore:  54%|█████▍    | 54/100 [00:00<00:00, 141.04it/s, failures=31, objective=6.87]
Failure Strategy: ignore:  55%|█████▌    | 55/100 [00:00<00:00, 141.04it/s, failures=32, objective=6.87]
Failure Strategy: ignore:  56%|█████▌    | 56/100 [00:00<00:00, 141.04it/s, failures=33, objective=6.87]
Failure Strategy: ignore:  57%|█████▋    | 57/100 [00:00<00:00, 141.04it/s, failures=34, objective=6.87]
Failure Strategy: ignore:  58%|█████▊    | 58/100 [00:00<00:00, 141.04it/s, failures=35, objective=6.87]
Failure Strategy: ignore:  59%|█████▉    | 59/100 [00:00<00:00, 141.04it/s, failures=36, objective=6.87]
Failure Strategy: ignore:  60%|██████    | 60/100 [00:00<00:00, 103.18it/s, failures=36, objective=6.87]
Failure Strategy: ignore:  60%|██████    | 60/100 [00:00<00:00, 103.18it/s, failures=37, objective=6.87]
Failure Strategy: ignore:  61%|██████    | 61/100 [00:00<00:00, 103.18it/s, failures=38, objective=6.87]
Failure Strategy: ignore:  62%|██████▏   | 62/100 [00:00<00:00, 103.18it/s, failures=39, objective=6.87]
Failure Strategy: ignore:  63%|██████▎   | 63/100 [00:00<00:00, 103.18it/s, failures=40, objective=6.87]
Failure Strategy: ignore:  64%|██████▍   | 64/100 [00:00<00:00, 103.18it/s, failures=41, objective=6.87]
Failure Strategy: ignore:  65%|██████▌   | 65/100 [00:00<00:00, 103.18it/s, failures=42, objective=6.87]
Failure Strategy: ignore:  66%|██████▌   | 66/100 [00:00<00:00, 103.18it/s, failures=43, objective=6.87]
Failure Strategy: ignore:  67%|██████▋   | 67/100 [00:00<00:00, 103.18it/s, failures=44, objective=6.87]
Failure Strategy: ignore:  68%|██████▊   | 68/100 [00:00<00:00, 103.18it/s, failures=45, objective=6.87]
Failure Strategy: ignore:  69%|██████▉   | 69/100 [00:00<00:00, 103.18it/s, failures=46, objective=6.87]
Failure Strategy: ignore:  70%|███████   | 70/100 [00:00<00:00, 103.18it/s, failures=47, objective=6.87]
Failure Strategy: ignore:  71%|███████   | 71/100 [00:00<00:00, 103.18it/s, failures=48, objective=6.87]
Failure Strategy: ignore:  72%|███████▏  | 72/100 [00:00<00:00, 103.18it/s, failures=49, objective=6.87]
Failure Strategy: ignore:  73%|███████▎  | 73/100 [00:00<00:00, 103.18it/s, failures=50, objective=6.87]
Failure Strategy: ignore:  74%|███████▍  | 74/100 [00:00<00:00, 103.18it/s, failures=51, objective=6.87]
Failure Strategy: ignore:  75%|███████▌  | 75/100 [00:00<00:00, 103.18it/s, failures=52, objective=6.87]
Failure Strategy: ignore:  76%|███████▌  | 76/100 [00:00<00:00, 103.18it/s, failures=53, objective=6.87]
Failure Strategy: ignore:  77%|███████▋  | 77/100 [00:00<00:00, 103.18it/s, failures=54, objective=6.87]
Failure Strategy: ignore:  78%|███████▊  | 78/100 [00:00<00:00, 103.18it/s, failures=55, objective=6.87]
Failure Strategy: ignore:  79%|███████▉  | 79/100 [00:00<00:00, 103.18it/s, failures=56, objective=6.87]
Failure Strategy: ignore:  80%|████████  | 80/100 [00:00<00:00, 103.18it/s, failures=57, objective=6.87]
Failure Strategy: ignore:  81%|████████  | 81/100 [00:00<00:00, 103.18it/s, failures=58, objective=6.87]
Failure Strategy: ignore:  82%|████████▏ | 82/100 [00:00<00:00, 103.18it/s, failures=59, objective=6.87]
Failure Strategy: ignore:  83%|████████▎ | 83/100 [00:00<00:00, 103.18it/s, failures=60, objective=6.87]
Failure Strategy: ignore:  84%|████████▍ | 84/100 [00:00<00:00, 103.18it/s, failures=61, objective=6.87]
Failure Strategy: ignore:  85%|████████▌ | 85/100 [00:00<00:00, 103.18it/s, failures=62, objective=6.87]
Failure Strategy: ignore:  86%|████████▌ | 86/100 [00:00<00:00, 103.18it/s, failures=63, objective=6.87]
Failure Strategy: ignore:  87%|████████▋ | 87/100 [00:00<00:00, 103.18it/s, failures=64, objective=6.87]
Failure Strategy: ignore:  88%|████████▊ | 88/100 [00:00<00:00, 103.18it/s, failures=65, objective=6.87]
Failure Strategy: ignore:  89%|████████▉ | 89/100 [00:00<00:00, 103.18it/s, failures=66, objective=6.87]
Failure Strategy: ignore:  90%|█████████ | 90/100 [00:00<00:00, 103.18it/s, failures=67, objective=6.87]
Failure Strategy: ignore:  91%|█████████ | 91/100 [00:00<00:00, 103.18it/s, failures=68, objective=6.87]
Failure Strategy: ignore:  92%|█████████▏| 92/100 [00:00<00:00, 103.18it/s, failures=69, objective=6.87]
Failure Strategy: ignore:  93%|█████████▎| 93/100 [00:00<00:00, 103.18it/s, failures=70, objective=6.87]
Failure Strategy: ignore:  94%|█████████▍| 94/100 [00:00<00:00, 103.18it/s, failures=71, objective=6.87]
Failure Strategy: ignore:  95%|█████████▌| 95/100 [00:00<00:00, 103.18it/s, failures=72, objective=6.87]
Failure Strategy: ignore:  96%|█████████▌| 96/100 [00:00<00:00, 103.18it/s, failures=73, objective=6.87]
Failure Strategy: ignore:  97%|█████████▋| 97/100 [00:00<00:00, 103.18it/s, failures=74, objective=6.87]
Failure Strategy: ignore:  98%|█████████▊| 98/100 [00:00<00:00, 103.18it/s, failures=75, objective=6.87]
Failure Strategy: ignore:  99%|█████████▉| 99/100 [00:00<00:00, 103.18it/s, failures=76, objective=6.87]
Failure Strategy: ignore: 100%|██████████| 100/100 [00:00<00:00, 103.18it/s, failures=77, objective=6.87]
Failure Strategy: ignore: 100%|██████████| 100/100 [00:00<00:00, 184.62it/s, failures=77, objective=6.87]

  0%|          | 0/100 [00:00<?, ?it/s]
Failure Strategy: mean:   0%|          | 0/100 [00:00<?, ?it/s]
Failure Strategy: mean:   1%|          | 1/100 [00:00<00:00, 17848.10it/s, failures=0, objective=6.63]
Failure Strategy: mean:   2%|▏         | 2/100 [00:00<00:00, 275.94it/s, failures=1, objective=6.63]
Failure Strategy: mean:   3%|▎         | 3/100 [00:00<00:00, 211.20it/s, failures=2, objective=6.63]
Failure Strategy: mean:   4%|▍         | 4/100 [00:00<00:00, 187.76it/s, failures=2, objective=6.63]
Failure Strategy: mean:   5%|▌         | 5/100 [00:00<00:00, 172.29it/s, failures=2, objective=6.63]
Failure Strategy: mean:   6%|▌         | 6/100 [00:00<00:00, 165.16it/s, failures=3, objective=6.63]
Failure Strategy: mean:   7%|▋         | 7/100 [00:00<00:00, 158.41it/s, failures=4, objective=6.63]
Failure Strategy: mean:   8%|▊         | 8/100 [00:00<00:00, 156.14it/s, failures=4, objective=6.63]
Failure Strategy: mean:   9%|▉         | 9/100 [00:00<00:00, 153.83it/s, failures=5, objective=6.63]
Failure Strategy: mean:  10%|█         | 10/100 [00:00<00:00, 151.26it/s, failures=5, objective=6.63]
Failure Strategy: mean:  11%|█         | 11/100 [00:00<00:00, 148.79it/s, failures=5, objective=6.63]
Failure Strategy: mean:  12%|█▏        | 12/100 [00:00<00:00, 147.87it/s, failures=6, objective=6.63]
Failure Strategy: mean:  13%|█▎        | 13/100 [00:00<00:00, 146.69it/s, failures=6, objective=6.63]
Failure Strategy: mean:  14%|█▍        | 14/100 [00:00<00:00, 145.90it/s, failures=7, objective=6.63]
Failure Strategy: mean:  15%|█▌        | 15/100 [00:00<00:00, 145.30it/s, failures=7, objective=6.63]
Failure Strategy: mean:  15%|█▌        | 15/100 [00:00<00:00, 145.30it/s, failures=8, objective=6.63]
Failure Strategy: mean:  16%|█▌        | 16/100 [00:00<00:00, 145.30it/s, failures=9, objective=6.63]
Failure Strategy: mean:  17%|█▋        | 17/100 [00:00<00:00, 145.30it/s, failures=10, objective=6.63]
Failure Strategy: mean:  18%|█▊        | 18/100 [00:00<00:00, 145.30it/s, failures=11, objective=6.63]
Failure Strategy: mean:  19%|█▉        | 19/100 [00:00<00:00, 145.30it/s, failures=12, objective=6.63]
Failure Strategy: mean:  20%|██        | 20/100 [00:00<00:00, 145.30it/s, failures=13, objective=6.63]
Failure Strategy: mean:  21%|██        | 21/100 [00:00<00:00, 145.30it/s, failures=14, objective=6.63]
Failure Strategy: mean:  22%|██▏       | 22/100 [00:00<00:00, 145.30it/s, failures=15, objective=6.63]
Failure Strategy: mean:  23%|██▎       | 23/100 [00:00<00:00, 145.30it/s, failures=16, objective=6.63]
Failure Strategy: mean:  24%|██▍       | 24/100 [00:00<00:00, 145.30it/s, failures=16, objective=6.63]
Failure Strategy: mean:  25%|██▌       | 25/100 [00:00<00:00, 145.30it/s, failures=16, objective=6.63]
Failure Strategy: mean:  26%|██▌       | 26/100 [00:00<00:00, 145.30it/s, failures=17, objective=6.63]
Failure Strategy: mean:  27%|██▋       | 27/100 [00:00<00:00, 145.30it/s, failures=17, objective=6.63]
Failure Strategy: mean:  28%|██▊       | 28/100 [00:00<00:00, 145.30it/s, failures=18, objective=6.63]
Failure Strategy: mean:  29%|██▉       | 29/100 [00:00<00:00, 145.30it/s, failures=19, objective=6.63]
Failure Strategy: mean:  30%|███       | 30/100 [00:00<00:00, 143.30it/s, failures=19, objective=6.63]
Failure Strategy: mean:  30%|███       | 30/100 [00:00<00:00, 143.30it/s, failures=20, objective=6.63]
Failure Strategy: mean:  31%|███       | 31/100 [00:00<00:00, 143.30it/s, failures=20, objective=6.63]
Failure Strategy: mean:  32%|███▏      | 32/100 [00:00<00:00, 143.30it/s, failures=21, objective=6.63]
Failure Strategy: mean:  33%|███▎      | 33/100 [00:00<00:00, 143.30it/s, failures=21, objective=6.63]
Failure Strategy: mean:  34%|███▍      | 34/100 [00:00<00:00, 143.30it/s, failures=21, objective=6.63]
Failure Strategy: mean:  35%|███▌      | 35/100 [00:00<00:00, 143.30it/s, failures=21, objective=6.63]
Failure Strategy: mean:  36%|███▌      | 36/100 [00:00<00:00, 143.30it/s, failures=21, objective=6.63]
Failure Strategy: mean:  37%|███▋      | 37/100 [00:00<00:00, 143.30it/s, failures=22, objective=6.63]
Failure Strategy: mean:  38%|███▊      | 38/100 [00:00<00:00, 143.30it/s, failures=22, objective=6.63]
Failure Strategy: mean:  39%|███▉      | 39/100 [00:00<00:00, 143.30it/s, failures=22, objective=6.87]
Failure Strategy: mean:  40%|████      | 40/100 [00:00<00:00, 143.30it/s, failures=22, objective=6.87]
Failure Strategy: mean:  41%|████      | 41/100 [00:00<00:00, 143.30it/s, failures=22, objective=6.87]
Failure Strategy: mean:  42%|████▏     | 42/100 [00:00<00:00, 143.30it/s, failures=22, objective=6.87]
Failure Strategy: mean:  43%|████▎     | 43/100 [00:00<00:00, 143.30it/s, failures=23, objective=6.87]
Failure Strategy: mean:  44%|████▍     | 44/100 [00:00<00:00, 143.30it/s, failures=23, objective=6.87]
Failure Strategy: mean:  45%|████▌     | 45/100 [00:00<00:00, 110.23it/s, failures=23, objective=6.87]
Failure Strategy: mean:  45%|████▌     | 45/100 [00:00<00:00, 110.23it/s, failures=24, objective=6.87]
Failure Strategy: mean:  46%|████▌     | 46/100 [00:00<00:00, 110.23it/s, failures=24, objective=6.87]
Failure Strategy: mean:  47%|████▋     | 47/100 [00:00<00:00, 110.23it/s, failures=25, objective=6.87]
Failure Strategy: mean:  48%|████▊     | 48/100 [00:00<00:00, 110.23it/s, failures=25, objective=6.87]
Failure Strategy: mean:  49%|████▉     | 49/100 [00:00<00:00, 110.23it/s, failures=25, objective=6.87]
Failure Strategy: mean:  50%|█████     | 50/100 [00:00<00:00, 110.23it/s, failures=25, objective=7.36]
Failure Strategy: mean:  51%|█████     | 51/100 [00:00<00:00, 110.23it/s, failures=25, objective=8.13]
Failure Strategy: mean:  52%|█████▏    | 52/100 [00:01<00:00, 110.23it/s, failures=25, objective=8.13]
Failure Strategy: mean:  53%|█████▎    | 53/100 [00:01<00:00, 110.23it/s, failures=26, objective=8.13]
Failure Strategy: mean:  54%|█████▍    | 54/100 [00:01<00:00, 110.23it/s, failures=26, objective=8.13]
Failure Strategy: mean:  55%|█████▌    | 55/100 [00:01<00:00, 110.23it/s, failures=26, objective=8.13]
Failure Strategy: mean:  56%|█████▌    | 56/100 [00:02<00:00, 110.23it/s, failures=26, objective=8.13]
Failure Strategy: mean:  57%|█████▋    | 57/100 [00:02<00:02, 16.47it/s, failures=26, objective=8.13]
Failure Strategy: mean:  57%|█████▋    | 57/100 [00:02<00:02, 16.47it/s, failures=26, objective=8.13]
Failure Strategy: mean:  58%|█████▊    | 58/100 [00:02<00:02, 16.47it/s, failures=26, objective=8.13]
Failure Strategy: mean:  59%|█████▉    | 59/100 [00:02<00:02, 16.47it/s, failures=26, objective=8.13]
Failure Strategy: mean:  60%|██████    | 60/100 [00:02<00:02, 16.47it/s, failures=26, objective=8.13]
Failure Strategy: mean:  61%|██████    | 61/100 [00:02<00:02, 16.47it/s, failures=26, objective=8.13]
Failure Strategy: mean:  62%|██████▏   | 62/100 [00:03<00:02, 16.47it/s, failures=26, objective=8.13]
Failure Strategy: mean:  63%|██████▎   | 63/100 [00:03<00:02, 16.47it/s, failures=26, objective=8.13]
Failure Strategy: mean:  64%|██████▍   | 64/100 [00:03<00:02, 16.47it/s, failures=26, objective=8.68]
Failure Strategy: mean:  65%|██████▌   | 65/100 [00:03<00:03, 11.03it/s, failures=26, objective=8.68]
Failure Strategy: mean:  65%|██████▌   | 65/100 [00:03<00:03, 11.03it/s, failures=26, objective=8.68]
Failure Strategy: mean:  66%|██████▌   | 66/100 [00:03<00:03, 11.03it/s, failures=26, objective=8.68]
Failure Strategy: mean:  67%|██████▋   | 67/100 [00:04<00:02, 11.03it/s, failures=26, objective=8.94]
Failure Strategy: mean:  68%|██████▊   | 68/100 [00:04<00:02, 11.03it/s, failures=26, objective=8.94]
Failure Strategy: mean:  69%|██████▉   | 69/100 [00:04<00:02, 11.03it/s, failures=26, objective=8.94]
Failure Strategy: mean:  70%|███████   | 70/100 [00:04<00:03,  8.82it/s, failures=26, objective=8.94]
Failure Strategy: mean:  70%|███████   | 70/100 [00:04<00:03,  8.82it/s, failures=26, objective=9.31]
Failure Strategy: mean:  71%|███████   | 71/100 [00:05<00:03,  8.82it/s, failures=26, objective=9.31]
Failure Strategy: mean:  72%|███████▏  | 72/100 [00:05<00:03,  8.82it/s, failures=26, objective=9.31]
Failure Strategy: mean:  73%|███████▎  | 73/100 [00:05<00:03,  8.82it/s, failures=27, objective=9.31]
Failure Strategy: mean:  74%|███████▍  | 74/100 [00:05<00:03,  7.77it/s, failures=27, objective=9.31]
Failure Strategy: mean:  74%|███████▍  | 74/100 [00:05<00:03,  7.77it/s, failures=27, objective=9.31]
Failure Strategy: mean:  75%|███████▌  | 75/100 [00:05<00:03,  7.77it/s, failures=27, objective=9.31]
Failure Strategy: mean:  76%|███████▌  | 76/100 [00:06<00:03,  7.77it/s, failures=28, objective=9.31]
Failure Strategy: mean:  77%|███████▋  | 77/100 [00:06<00:03,  6.74it/s, failures=28, objective=9.31]
Failure Strategy: mean:  77%|███████▋  | 77/100 [00:06<00:03,  6.74it/s, failures=28, objective=9.31]
Failure Strategy: mean:  78%|███████▊  | 78/100 [00:06<00:03,  6.74it/s, failures=28, objective=9.31]
Failure Strategy: mean:  79%|███████▉  | 79/100 [00:06<00:03,  6.31it/s, failures=28, objective=9.31]
Failure Strategy: mean:  79%|███████▉  | 79/100 [00:06<00:03,  6.31it/s, failures=28, objective=9.31]
Failure Strategy: mean:  80%|████████  | 80/100 [00:07<00:03,  6.31it/s, failures=28, objective=9.31]
Failure Strategy: mean:  81%|████████  | 81/100 [00:07<00:03,  6.01it/s, failures=28, objective=9.31]
Failure Strategy: mean:  81%|████████  | 81/100 [00:07<00:03,  6.01it/s, failures=28, objective=9.33]
Failure Strategy: mean:  82%|████████▏ | 82/100 [00:07<00:02,  6.01it/s, failures=28, objective=9.49]
Failure Strategy: mean:  83%|████████▎ | 83/100 [00:07<00:03,  5.51it/s, failures=28, objective=9.49]
Failure Strategy: mean:  83%|████████▎ | 83/100 [00:07<00:03,  5.51it/s, failures=28, objective=9.49]
Failure Strategy: mean:  84%|████████▍ | 84/100 [00:08<00:03,  5.19it/s, failures=28, objective=9.49]
Failure Strategy: mean:  84%|████████▍ | 84/100 [00:08<00:03,  5.19it/s, failures=28, objective=9.49]
Failure Strategy: mean:  85%|████████▌ | 85/100 [00:08<00:02,  5.16it/s, failures=28, objective=9.49]
Failure Strategy: mean:  85%|████████▌ | 85/100 [00:08<00:02,  5.16it/s, failures=28, objective=9.49]
Failure Strategy: mean:  86%|████████▌ | 86/100 [00:08<00:02,  5.02it/s, failures=28, objective=9.49]
Failure Strategy: mean:  86%|████████▌ | 86/100 [00:08<00:02,  5.02it/s, failures=28, objective=9.49]
Failure Strategy: mean:  87%|████████▋ | 87/100 [00:08<00:02,  4.94it/s, failures=28, objective=9.49]
Failure Strategy: mean:  87%|████████▋ | 87/100 [00:08<00:02,  4.94it/s, failures=28, objective=9.49]
Failure Strategy: mean:  88%|████████▊ | 88/100 [00:08<00:02,  4.91it/s, failures=28, objective=9.49]
Failure Strategy: mean:  88%|████████▊ | 88/100 [00:08<00:02,  4.91it/s, failures=28, objective=9.7]
Failure Strategy: mean:  89%|████████▉ | 89/100 [00:09<00:02,  4.79it/s, failures=28, objective=9.7]
Failure Strategy: mean:  89%|████████▉ | 89/100 [00:09<00:02,  4.79it/s, failures=28, objective=9.7]
Failure Strategy: mean:  90%|█████████ | 90/100 [00:09<00:02,  4.59it/s, failures=28, objective=9.7]
Failure Strategy: mean:  90%|█████████ | 90/100 [00:09<00:02,  4.59it/s, failures=28, objective=9.7]
Failure Strategy: mean:  91%|█████████ | 91/100 [00:09<00:01,  4.55it/s, failures=28, objective=9.7]
Failure Strategy: mean:  91%|█████████ | 91/100 [00:09<00:01,  4.55it/s, failures=28, objective=9.7]
Failure Strategy: mean:  92%|█████████▏| 92/100 [00:09<00:01,  4.34it/s, failures=28, objective=9.7]
Failure Strategy: mean:  92%|█████████▏| 92/100 [00:09<00:01,  4.34it/s, failures=28, objective=9.7]
Failure Strategy: mean:  93%|█████████▎| 93/100 [00:10<00:01,  4.21it/s, failures=28, objective=9.7]
Failure Strategy: mean:  93%|█████████▎| 93/100 [00:10<00:01,  4.21it/s, failures=28, objective=9.7]
Failure Strategy: mean:  94%|█████████▍| 94/100 [00:10<00:01,  4.23it/s, failures=28, objective=9.7]
Failure Strategy: mean:  94%|█████████▍| 94/100 [00:10<00:01,  4.23it/s, failures=28, objective=9.7]
Failure Strategy: mean:  95%|█████████▌| 95/100 [00:10<00:01,  4.25it/s, failures=28, objective=9.7]
Failure Strategy: mean:  95%|█████████▌| 95/100 [00:10<00:01,  4.25it/s, failures=28, objective=9.7]
Failure Strategy: mean:  96%|█████████▌| 96/100 [00:10<00:01,  3.98it/s, failures=28, objective=9.7]
Failure Strategy: mean:  96%|█████████▌| 96/100 [00:10<00:01,  3.98it/s, failures=28, objective=9.7]
Failure Strategy: mean:  97%|█████████▋| 97/100 [00:11<00:00,  3.80it/s, failures=28, objective=9.7]
Failure Strategy: mean:  97%|█████████▋| 97/100 [00:11<00:00,  3.80it/s, failures=28, objective=9.7]
Failure Strategy: mean:  98%|█████████▊| 98/100 [00:11<00:00,  3.95it/s, failures=28, objective=9.7]
Failure Strategy: mean:  98%|█████████▊| 98/100 [00:11<00:00,  3.95it/s, failures=29, objective=9.7]
Failure Strategy: mean:  99%|█████████▉| 99/100 [00:11<00:00,  3.73it/s, failures=29, objective=9.7]
Failure Strategy: mean:  99%|█████████▉| 99/100 [00:11<00:00,  3.73it/s, failures=29, objective=9.74]
Failure Strategy: mean: 100%|██████████| 100/100 [00:11<00:00,  3.71it/s, failures=29, objective=9.74]
Failure Strategy: mean: 100%|██████████| 100/100 [00:11<00:00,  3.71it/s, failures=29, objective=9.74]
Failure Strategy: mean: 100%|██████████| 100/100 [00:11<00:00,  8.34it/s, failures=29, objective=9.74]

  0%|          | 0/100 [00:00<?, ?it/s]
Failure Strategy: min:   0%|          | 0/100 [00:00<?, ?it/s]
Failure Strategy: min:   1%|          | 1/100 [00:00<00:00, 16710.37it/s, failures=0, objective=6.63]
Failure Strategy: min:   2%|▏         | 2/100 [00:00<00:00, 253.36it/s, failures=1, objective=6.63]
Failure Strategy: min:   3%|▎         | 3/100 [00:00<00:00, 196.72it/s, failures=2, objective=6.63]
Failure Strategy: min:   4%|▍         | 4/100 [00:00<00:00, 170.24it/s, failures=2, objective=6.63]
Failure Strategy: min:   5%|▌         | 5/100 [00:00<00:00, 156.10it/s, failures=2, objective=6.63]
Failure Strategy: min:   6%|▌         | 6/100 [00:00<00:00, 139.38it/s, failures=3, objective=6.63]
Failure Strategy: min:   7%|▋         | 7/100 [00:00<00:00, 138.42it/s, failures=4, objective=6.63]
Failure Strategy: min:   8%|▊         | 8/100 [00:00<00:00, 136.98it/s, failures=4, objective=6.63]
Failure Strategy: min:   9%|▉         | 9/100 [00:00<00:00, 136.24it/s, failures=5, objective=6.63]
Failure Strategy: min:  10%|█         | 10/100 [00:00<00:00, 135.39it/s, failures=5, objective=6.63]
Failure Strategy: min:  11%|█         | 11/100 [00:00<00:00, 135.49it/s, failures=5, objective=6.63]
Failure Strategy: min:  12%|█▏        | 12/100 [00:00<00:00, 135.09it/s, failures=6, objective=6.63]
Failure Strategy: min:  13%|█▎        | 13/100 [00:00<00:00, 134.94it/s, failures=6, objective=6.63]
Failure Strategy: min:  14%|█▍        | 14/100 [00:00<00:00, 134.44it/s, failures=6, objective=6.63]
Failure Strategy: min:  14%|█▍        | 14/100 [00:00<00:00, 134.44it/s, failures=7, objective=6.63]
Failure Strategy: min:  15%|█▌        | 15/100 [00:00<00:00, 134.44it/s, failures=8, objective=6.63]
Failure Strategy: min:  16%|█▌        | 16/100 [00:00<00:00, 134.44it/s, failures=9, objective=6.63]
Failure Strategy: min:  17%|█▋        | 17/100 [00:00<00:00, 134.44it/s, failures=10, objective=6.63]
Failure Strategy: min:  18%|█▊        | 18/100 [00:00<00:00, 134.44it/s, failures=11, objective=6.63]
Failure Strategy: min:  19%|█▉        | 19/100 [00:00<00:00, 134.44it/s, failures=12, objective=6.63]
Failure Strategy: min:  20%|██        | 20/100 [00:00<00:00, 134.44it/s, failures=13, objective=6.63]
Failure Strategy: min:  21%|██        | 21/100 [00:00<00:00, 134.44it/s, failures=14, objective=6.63]
Failure Strategy: min:  22%|██▏       | 22/100 [00:00<00:00, 134.44it/s, failures=15, objective=6.63]
Failure Strategy: min:  23%|██▎       | 23/100 [00:00<00:00, 134.44it/s, failures=16, objective=6.63]
Failure Strategy: min:  24%|██▍       | 24/100 [00:00<00:00, 134.44it/s, failures=16, objective=6.63]
Failure Strategy: min:  25%|██▌       | 25/100 [00:00<00:00, 134.44it/s, failures=16, objective=6.63]
Failure Strategy: min:  26%|██▌       | 26/100 [00:00<00:00, 134.44it/s, failures=17, objective=6.63]
Failure Strategy: min:  27%|██▋       | 27/100 [00:00<00:00, 134.44it/s, failures=17, objective=6.63]
Failure Strategy: min:  28%|██▊       | 28/100 [00:00<00:00, 133.95it/s, failures=17, objective=6.63]
Failure Strategy: min:  28%|██▊       | 28/100 [00:00<00:00, 133.95it/s, failures=18, objective=6.63]
Failure Strategy: min:  29%|██▉       | 29/100 [00:00<00:00, 133.95it/s, failures=19, objective=6.63]
Failure Strategy: min:  30%|███       | 30/100 [00:00<00:00, 133.95it/s, failures=20, objective=6.63]
Failure Strategy: min:  31%|███       | 31/100 [00:00<00:00, 133.95it/s, failures=20, objective=6.63]
Failure Strategy: min:  32%|███▏      | 32/100 [00:00<00:00, 133.95it/s, failures=21, objective=6.63]
Failure Strategy: min:  33%|███▎      | 33/100 [00:00<00:00, 133.95it/s, failures=21, objective=6.63]
Failure Strategy: min:  34%|███▍      | 34/100 [00:00<00:00, 133.95it/s, failures=21, objective=6.63]
Failure Strategy: min:  35%|███▌      | 35/100 [00:00<00:00, 133.95it/s, failures=21, objective=6.63]
Failure Strategy: min:  36%|███▌      | 36/100 [00:00<00:00, 133.95it/s, failures=21, objective=6.63]
Failure Strategy: min:  37%|███▋      | 37/100 [00:00<00:00, 133.95it/s, failures=22, objective=6.63]
Failure Strategy: min:  38%|███▊      | 38/100 [00:00<00:00, 133.95it/s, failures=22, objective=6.63]
Failure Strategy: min:  39%|███▉      | 39/100 [00:00<00:00, 133.95it/s, failures=22, objective=6.87]
Failure Strategy: min:  40%|████      | 40/100 [00:00<00:00, 133.95it/s, failures=22, objective=6.87]
Failure Strategy: min:  41%|████      | 41/100 [00:00<00:00, 133.95it/s, failures=22, objective=6.87]
Failure Strategy: min:  42%|████▏     | 42/100 [00:00<00:00, 135.07it/s, failures=22, objective=6.87]
Failure Strategy: min:  42%|████▏     | 42/100 [00:00<00:00, 135.07it/s, failures=22, objective=6.87]
Failure Strategy: min:  43%|████▎     | 43/100 [00:00<00:00, 135.07it/s, failures=23, objective=6.87]
Failure Strategy: min:  44%|████▍     | 44/100 [00:00<00:00, 135.07it/s, failures=23, objective=6.87]
Failure Strategy: min:  45%|████▌     | 45/100 [00:00<00:00, 135.07it/s, failures=24, objective=6.87]
Failure Strategy: min:  46%|████▌     | 46/100 [00:00<00:00, 135.07it/s, failures=24, objective=6.87]
Failure Strategy: min:  47%|████▋     | 47/100 [00:00<00:00, 135.07it/s, failures=25, objective=6.87]
Failure Strategy: min:  48%|████▊     | 48/100 [00:00<00:00, 135.07it/s, failures=25, objective=6.87]
Failure Strategy: min:  49%|████▉     | 49/100 [00:00<00:00, 135.07it/s, failures=25, objective=6.87]
Failure Strategy: min:  50%|█████     | 50/100 [00:00<00:00, 135.07it/s, failures=25, objective=6.87]
Failure Strategy: min:  51%|█████     | 51/100 [00:00<00:00, 135.07it/s, failures=25, objective=6.87]
Failure Strategy: min:  52%|█████▏    | 52/100 [00:01<00:00, 135.07it/s, failures=25, objective=7.85]
Failure Strategy: min:  53%|█████▎    | 53/100 [00:01<00:00, 135.07it/s, failures=25, objective=7.85]
Failure Strategy: min:  54%|█████▍    | 54/100 [00:01<00:00, 135.07it/s, failures=25, objective=7.85]
Failure Strategy: min:  55%|█████▌    | 55/100 [00:01<00:00, 135.07it/s, failures=25, objective=7.85]
Failure Strategy: min:  56%|█████▌    | 56/100 [00:01<00:02, 20.96it/s, failures=25, objective=7.85]
Failure Strategy: min:  56%|█████▌    | 56/100 [00:01<00:02, 20.96it/s, failures=25, objective=7.85]
Failure Strategy: min:  57%|█████▋    | 57/100 [00:02<00:02, 20.96it/s, failures=25, objective=8.01]
Failure Strategy: min:  58%|█████▊    | 58/100 [00:02<00:02, 20.96it/s, failures=25, objective=8.01]
Failure Strategy: min:  59%|█████▉    | 59/100 [00:02<00:01, 20.96it/s, failures=25, objective=8.01]
Failure Strategy: min:  60%|██████    | 60/100 [00:02<00:01, 20.96it/s, failures=25, objective=8.01]
Failure Strategy: min:  61%|██████    | 61/100 [00:02<00:01, 20.96it/s, failures=25, objective=8.01]
Failure Strategy: min:  62%|██████▏   | 62/100 [00:02<00:01, 20.96it/s, failures=25, objective=8.01]
Failure Strategy: min:  63%|██████▎   | 63/100 [00:03<00:01, 20.96it/s, failures=25, objective=8.01]
Failure Strategy: min:  64%|██████▍   | 64/100 [00:03<00:01, 20.96it/s, failures=25, objective=8.01]
Failure Strategy: min:  65%|██████▌   | 65/100 [00:03<00:02, 11.81it/s, failures=25, objective=8.01]
Failure Strategy: min:  65%|██████▌   | 65/100 [00:03<00:02, 11.81it/s, failures=26, objective=8.01]
Failure Strategy: min:  66%|██████▌   | 66/100 [00:03<00:02, 11.81it/s, failures=27, objective=8.01]
Failure Strategy: min:  67%|██████▋   | 67/100 [00:03<00:02, 11.81it/s, failures=28, objective=8.01]
Failure Strategy: min:  68%|██████▊   | 68/100 [00:04<00:02, 11.81it/s, failures=28, objective=8.2]
Failure Strategy: min:  69%|██████▉   | 69/100 [00:04<00:02, 11.81it/s, failures=28, objective=8.2]
Failure Strategy: min:  70%|███████   | 70/100 [00:04<00:02, 11.81it/s, failures=28, objective=8.65]
Failure Strategy: min:  71%|███████   | 71/100 [00:04<00:03,  9.44it/s, failures=28, objective=8.65]
Failure Strategy: min:  71%|███████   | 71/100 [00:04<00:03,  9.44it/s, failures=28, objective=8.89]
Failure Strategy: min:  72%|███████▏  | 72/100 [00:04<00:02,  9.44it/s, failures=28, objective=8.89]
Failure Strategy: min:  73%|███████▎  | 73/100 [00:05<00:02,  9.44it/s, failures=28, objective=8.89]
Failure Strategy: min:  74%|███████▍  | 74/100 [00:05<00:02,  9.44it/s, failures=28, objective=8.89]
Failure Strategy: min:  75%|███████▌  | 75/100 [00:05<00:03,  8.17it/s, failures=28, objective=8.89]
Failure Strategy: min:  75%|███████▌  | 75/100 [00:05<00:03,  8.17it/s, failures=29, objective=8.89]
Failure Strategy: min:  76%|███████▌  | 76/100 [00:05<00:02,  8.17it/s, failures=29, objective=8.89]
Failure Strategy: min:  77%|███████▋  | 77/100 [00:05<00:02,  8.17it/s, failures=29, objective=9.07]
Failure Strategy: min:  78%|███████▊  | 78/100 [00:06<00:03,  7.16it/s, failures=29, objective=9.07]
Failure Strategy: min:  78%|███████▊  | 78/100 [00:06<00:03,  7.16it/s, failures=30, objective=9.07]
Failure Strategy: min:  79%|███████▉  | 79/100 [00:06<00:02,  7.16it/s, failures=30, objective=9.07]
Failure Strategy: min:  80%|████████  | 80/100 [00:06<00:02,  7.16it/s, failures=30, objective=9.07]
Failure Strategy: min:  81%|████████  | 81/100 [00:06<00:03,  6.25it/s, failures=30, objective=9.07]
Failure Strategy: min:  81%|████████  | 81/100 [00:06<00:03,  6.25it/s, failures=30, objective=9.07]
Failure Strategy: min:  82%|████████▏ | 82/100 [00:07<00:02,  6.25it/s, failures=30, objective=9.3]
Failure Strategy: min:  83%|████████▎ | 83/100 [00:07<00:02,  5.96it/s, failures=30, objective=9.3]
Failure Strategy: min:  83%|████████▎ | 83/100 [00:07<00:02,  5.96it/s, failures=30, objective=9.3]
Failure Strategy: min:  84%|████████▍ | 84/100 [00:07<00:02,  5.96it/s, failures=30, objective=9.3]
Failure Strategy: min:  85%|████████▌ | 85/100 [00:07<00:02,  5.66it/s, failures=30, objective=9.3]
Failure Strategy: min:  85%|████████▌ | 85/100 [00:07<00:02,  5.66it/s, failures=31, objective=9.3]
Failure Strategy: min:  86%|████████▌ | 86/100 [00:08<00:02,  5.58it/s, failures=31, objective=9.3]
Failure Strategy: min:  86%|████████▌ | 86/100 [00:08<00:02,  5.58it/s, failures=31, objective=9.3]
Failure Strategy: min:  87%|████████▋ | 87/100 [00:08<00:02,  5.40it/s, failures=31, objective=9.3]
Failure Strategy: min:  87%|████████▋ | 87/100 [00:08<00:02,  5.40it/s, failures=31, objective=9.3]
Failure Strategy: min:  88%|████████▊ | 88/100 [00:08<00:02,  4.51it/s, failures=31, objective=9.3]
Failure Strategy: min:  88%|████████▊ | 88/100 [00:08<00:02,  4.51it/s, failures=31, objective=9.3]
Failure Strategy: min:  89%|████████▉ | 89/100 [00:09<00:02,  4.10it/s, failures=31, objective=9.3]
Failure Strategy: min:  89%|████████▉ | 89/100 [00:09<00:02,  4.10it/s, failures=32, objective=9.3]
Failure Strategy: min:  90%|█████████ | 90/100 [00:09<00:02,  4.15it/s, failures=32, objective=9.3]
Failure Strategy: min:  90%|█████████ | 90/100 [00:09<00:02,  4.15it/s, failures=32, objective=9.3]
Failure Strategy: min:  91%|█████████ | 91/100 [00:09<00:02,  3.96it/s, failures=32, objective=9.3]
Failure Strategy: min:  91%|█████████ | 91/100 [00:09<00:02,  3.96it/s, failures=32, objective=9.3]
Failure Strategy: min:  92%|█████████▏| 92/100 [00:09<00:02,  3.92it/s, failures=32, objective=9.3]
Failure Strategy: min:  92%|█████████▏| 92/100 [00:09<00:02,  3.92it/s, failures=32, objective=9.3]
Failure Strategy: min:  93%|█████████▎| 93/100 [00:10<00:01,  3.99it/s, failures=32, objective=9.3]
Failure Strategy: min:  93%|█████████▎| 93/100 [00:10<00:01,  3.99it/s, failures=32, objective=9.3]
Failure Strategy: min:  94%|█████████▍| 94/100 [00:10<00:01,  3.93it/s, failures=32, objective=9.3]
Failure Strategy: min:  94%|█████████▍| 94/100 [00:10<00:01,  3.93it/s, failures=32, objective=9.3]
Failure Strategy: min:  95%|█████████▌| 95/100 [00:10<00:01,  4.13it/s, failures=32, objective=9.3]
Failure Strategy: min:  95%|█████████▌| 95/100 [00:10<00:01,  4.13it/s, failures=33, objective=9.3]
Failure Strategy: min:  96%|█████████▌| 96/100 [00:10<00:00,  4.29it/s, failures=33, objective=9.3]
Failure Strategy: min:  96%|█████████▌| 96/100 [00:10<00:00,  4.29it/s, failures=33, objective=9.3]
Failure Strategy: min:  97%|█████████▋| 97/100 [00:11<00:00,  3.82it/s, failures=33, objective=9.3]
Failure Strategy: min:  97%|█████████▋| 97/100 [00:11<00:00,  3.82it/s, failures=34, objective=9.3]
Failure Strategy: min:  98%|█████████▊| 98/100 [00:11<00:00,  3.70it/s, failures=34, objective=9.3]
Failure Strategy: min:  98%|█████████▊| 98/100 [00:11<00:00,  3.70it/s, failures=34, objective=9.3]
Failure Strategy: min:  99%|█████████▉| 99/100 [00:11<00:00,  3.70it/s, failures=34, objective=9.3]
Failure Strategy: min:  99%|█████████▉| 99/100 [00:11<00:00,  3.70it/s, failures=34, objective=9.3]
Failure Strategy: min: 100%|██████████| 100/100 [00:11<00:00,  3.76it/s, failures=34, objective=9.3]
Failure Strategy: min: 100%|██████████| 100/100 [00:11<00:00,  3.76it/s, failures=34, objective=9.3]
Failure Strategy: min: 100%|██████████| 100/100 [00:11<00:00,  8.41it/s, failures=34, objective=9.3]

Finally we plot the collected results. We can see that the "mean" and min strategy have much less failures than "ignore". In addition, we observe that they return significantly better objectives.

Code (Plot results with failures)
fig, axes = plt.subplots(nrows=3, ncols=1, figsize=(WIDTH_PLOTS, HEIGHT_PLOTS), tight_layout=True, sharex=True, sharey=True)
for i, (failure_strategy, df) in enumerate(results.items()):

    if df.objective.dtype != np.float64:
        x = np.arange(len(df))
        mask_failed = np.where(df.objective.str.startswith("F"))[0]
        mask_success = np.where(~df.objective.str.startswith("F"))[0]
        x_success, x_failed = x[mask_success], x[mask_failed]
        y_success = df["objective"][mask_success].astype(float)

    axes[i].scatter(x_success, y_success, label="success")
    axes[i].scatter(x_failed, np.zeros(x_failed.shape), marker="v", color="red", label="failure")

    axes[i].set_ylabel("Objective")
    axes[i].legend(title=f"Strategy: {failure_strategy}")
    axes[i].grid()
axes[i].set_xlabel("Evaluations")
plot notify failures hpo
Text(0.5, 29.00000000000002, 'Evaluations')

Total running time of the script: (0 minutes 26.470 seconds)

Gallery generated by Sphinx-Gallery