Hyperparameter Optimized Ensemble of Random Decision Trees with Uncertainty for Classification

Author(s): Romain Egele.

In this tutorial, you will learn about how to use hyperparameter optimization to generate an ensemble of Scikit-Learn models that can be used for uncertainty quantification.

Installation and imports

Installing dependencies with the pip installation is recommended. It requires Python >= 3.10.

%%bash
pip install "deephyper[ray]"

Code (Import statements)

import pathlib
import pickle
import os

import matplotlib.pyplot as plt
import numpy as np

from sklearn.calibration import CalibrationDisplay
from sklearn.datasets import make_moons
from sklearn.metrics import log_loss, accuracy_score
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier

WIDTH_PLOTS = 8
HEIGHT_PLOTS = WIDTH_PLOTS / 1.618

Synthetic data generation

For the data, we use the sklearn.datasets.make_moons() functionality from Scikit-Learn to have a synthetic binary-classification problem with two moons. The input data \(x\) are two dimensionnal and the target data \(y\) are binary values. We randomly flip 10% of the labels to generate artificial noise that should later be estimated by what we call “aleatoric uncertainty” (a.k.a., intrinsic random noise).

Code (Loading synthetic data)

def flip_binary_labels(y, ratio, random_state=None):
    """Increase the variance of P(Y|X) by ``ratio``"""
    y_flipped = np.zeros(np.shape(y))
    y_flipped[:] = y[:]
    rs = np.random.RandomState(random_state)
    idx = np.arange(len(y_flipped))
    idx = rs.choice(idx, size=int(ratio * len(y_flipped)), replace=False)
    y_flipped[idx] = 1 - y_flipped[idx]
    return y_flipped


def load_data(noise=0.1, n=1_000, ratio_flipped=0.1, test_size=0.33, valid_size=0.33, random_state=42):
    rng = np.random.RandomState(random_state)
    max_int = np.iinfo(np.int32).max

    test_size = int(test_size * n)
    valid_size = int(valid_size * n)

    X, y = make_moons(n_samples=n, noise=noise, shuffle=True, random_state=rng.randint(max_int))
    X = X - np.mean(X, axis=0)

    y = flip_binary_labels(y, ratio=ratio_flipped, random_state=rng.randint(max_int))
    y = y.astype(np.int64)

    train_X, test_X, train_y, test_y = train_test_split(
        X,
        y,
        test_size=test_size,
        random_state=rng.randint(max_int),
        stratify=y,
    )

    train_X, valid_X, train_y, valid_y = train_test_split(
        train_X,
        train_y,
        test_size=valid_size,
        random_state=rng.randint(max_int),
        stratify=train_y,
    )

    return (train_X, train_y), (valid_X, valid_y), (test_X, test_y)

(x, y), (vx, vy), (tx, ty) = load_data()

_ = plt.subplots(figsize=(WIDTH_PLOTS, HEIGHT_PLOTS), tight_layout=True)
_ = plt.scatter(
    x[:, 0].reshape(-1), x[:, 1].reshape(-1), c=y, label="train", alpha=0.8
)
_ = plt.scatter(
    vx[:, 0].reshape(-1),
    vx[:, 1].reshape(-1),
    c=vy,
    marker="s",
    label="valid",
    alpha=0.8,
)
_ = plt.ylabel("$x1$", fontsize=12)
_ = plt.xlabel("$x0$", fontsize=12)
_ = plt.legend(loc="upper center", ncol=3, fontsize=12)

Training a Decision Tree

We focus on the class of random decision tree models. We define a function that trains and evaluate a random decision tree from given parameters job.parameters. These parameters will be optimized in the next steps by DeepHyper.

The score we minimize with respect to hyperparameters \(\theta\) is the validation log loss (a.k.a., binary cross entropy) as we want to have calibrated uncertainty estimates of \(P(Y|X=x)\) and \(1-P(Y|X=x)\):

\[L_\text{BCE}(x, y;\theta) = y \cdot \log\left(p(y|x;\theta)\right) + (1 - y) \cdot \log\left(1 - p(y|x\theta)\right)\]

where \(p(y|x;\theta)\) is the predited probability of a tree with hyperparameters \(\theta\).

Code (Plot decision boundary)

def plot_decision_boundary_decision_tree(dataset, labels, model, steps=1000, color_map="viridis", ax=None):
    color_map = plt.get_cmap(color_map)
    # Define region of interest by data limits
    xmin, xmax = dataset[:, 0].min() - 1, dataset[:, 0].max() + 1
    ymin, ymax = dataset[:, 1].min() - 1, dataset[:, 1].max() + 1
    x_span = np.linspace(xmin, xmax, steps)
    y_span = np.linspace(ymin, ymax, steps)
    xx, yy = np.meshgrid(x_span, y_span)

    # Make predictions across region of interest
    labels_predicted = model.predict_proba(np.c_[xx.ravel(), yy.ravel()])

    # Plot decision boundary in region of interest
    z = labels_predicted[:, 1].reshape(xx.shape)

    ax.contourf(xx, yy, z, cmap=color_map, alpha=0.5)

    # Get predicted labels on training data and plot
    ax.scatter(
        dataset[:, 0],
        dataset[:, 1],
        c=labels,
        # cmap=color_map,
        lw=0,
    )

The run function takes a job object as input suggested by DeepHyper. We use it to pass the job.parameters and create the decision tree model. Then, we fit the model on the data on compute its log-loss score on the validation dataset. In case of unexpected error we return a special value F_fit so that our hyperparameter optimization can learn to avoid these unexepected failures. We checkpoint the model on disk as model_*.pkl files. Finally, we return all of our scores, the "objective" is the value maximized by DeepHyper. Other scores are returned as metadata for further analysis (e.g., overfitting, underfitting, etc.).

hpo_dir = "hpo_sklearn_classification"
model_checkpoint_dir = os.path.join(hpo_dir, "models")


def run(job, model_checkpoint_dir=".", verbose=True, show_plots=False):

    (x, y), (vx, vy), (tx, ty) = load_data()

    model = DecisionTreeClassifier(**job.parameters)

    try:
        model.fit(x, y)
        vy_pred_proba = model.predict_proba(vx)
        val_cce = log_loss(vy, vy_pred_proba)
    except:
        return "F_fit"

    # Saving the model
    with open(os.path.join(model_checkpoint_dir, f"model_{job.id}.pkl"), "wb") as f:
        pickle.dump(model, f)

    if verbose:
        print(f"{job.id}: {val_cce=:.3f}")

    if show_plots:
        fig, axes = plt.subplots(nrows=2, ncols=1, figsize=(WIDTH_PLOTS, HEIGHT_PLOTS*2), tight_layout=True)
        plot_decision_boundary_decision_tree(tx, ty, model, steps=1000, color_map="viridis", ax=axes[0])
        disp = CalibrationDisplay.from_predictions(ty, model.predict_proba(tx)[:, 1], ax=axes[1])

    test_cce = log_loss(ty, model.predict_proba(tx))
    test_acc = accuracy_score(ty, model.predict(tx))

    # The score is negated for maximization
    # The score is -Categorical Cross Entropy/LogLoss
    return {
        "objective": -val_cce,
        "metadata": {"test_cce": test_cce, "test_acc": test_acc},
    }

It is important to note that we did not fix the random state of the random decision tree. The hyperparameter optimization takes into consideration the fact that the observed objective is noisy and of course this can be tuned. For example, as the default surrogate model of DeepHyper is itself a randomized forest, increasing the number of samples in leaf nodes would have the effect of averaging out the prediction of the surrogate.

Also, the point of ensembling randomized decision trees is to build a model with lower variance (i.e., variability of the score when fitting it) than its base estimators.

Hyperparameter search space

We define the hyperparameter search space for decision trees. This tells to DeepHyper the hyperparameter values it can use for the optimization. To define these hyperparameters we look at the DecisionTreeClassifier API Reference.

from deephyper.hpo import HpProblem


def create_hpo_problem():

    problem = HpProblem()

    problem.add_hyperparameter(["gini", "entropy", "log_loss"], "criterion")
    problem.add_hyperparameter(["best", "random"], "splitter")
    problem.add_hyperparameter((10, 1000, "log-uniform"), "max_depth", default_value=1000)
    problem.add_hyperparameter((2, 20), "min_samples_split", default_value=2)
    problem.add_hyperparameter((1, 20), "min_samples_leaf", default_value=1)
    problem.add_hyperparameter((0.0, 0.5), "min_weight_fraction_leaf", default_value=0.0)

    return problem

problem = create_hpo_problem()
problem

Configuration space object:
  Hyperparameters:
    criterion, Type: Categorical, Choices: {gini, entropy, log_loss}, Default: gini
    max_depth, Type: UniformInteger, Range: [10, 1000], Default: 1000, on log-scale
    min_samples_leaf, Type: UniformInteger, Range: [1, 20], Default: 1
    min_samples_split, Type: UniformInteger, Range: [2, 20], Default: 2
    min_weight_fraction_leaf, Type: UniformFloat, Range: [0.0, 0.5], Default: 0.0
    splitter, Type: Categorical, Choices: {best, random}, Default: best

Evaluation of the baseline

We previously defined default_value=... for each hyperparameter. These values corresponds to the default hyperparameters used in Scikit-Learn. We now test them to have a base performance.

from deephyper.evaluator import RunningJob


def evaluate_decision_tree(problem):

    model_checkpoint_dir = "models_sklearn_test"
    pathlib.Path(model_checkpoint_dir).mkdir(parents=True, exist_ok=True)

    default_parameters = problem.default_configuration
    print(f"{default_parameters=}")

    output = run(
        RunningJob(id="test", parameters=default_parameters),
        model_checkpoint_dir=model_checkpoint_dir,
        show_plots=True,
    )
    return output

baseline_output = evaluate_decision_tree(problem)
baseline_output

default_parameters={'criterion': 'gini', 'max_depth': 1000, 'min_samples_leaf': 1, 'min_samples_split': 2, 'min_weight_fraction_leaf': 0.0, 'splitter': 'best'}
0.0: val_cce=7.100

{'objective': -7.099507485735197, 'metadata': {'test_cce': 6.553391525294028, 'test_acc': 0.8181818181818182}}

The accuracy is great, but the uncertainty is not well calibrated.

Hyperparameter Optimization

In DeepHyper, instead of just performing sequential Bayesian optimization we provide asynchronous parallelisation for Bayesian optimization (and other methods). This allows to execute multiple evaluation function in parallel to collect observations of objectives faster.

In this example, we will focus on using centralized Bayesian optimization (CBO). In this setting, we have one main process that runs the Bayesian optimization algorithm and we have multiple worker processes that run evaluation functions. The class we use for this is deephyper.hpo.CBO.

Let us start by explaining import configuration parameters of deephyper.hpo.CBO:

• initial_points: is a list of initial hyperparameter configurations to test, we add the baseline hyperparameters as we want to be at least better than this configuration.

• surrogate_model_*: are parameters related to the surrogate model we use, here "ET" is an alias for the Extremely Randomized Trees regression model.

• multi_point_strategy: is the strategy we use for parallel suggestion of hyperparameters, here we use the qUCBd that will sample for each new parallel configuration a different \(\kappa^j_i\) value from an exponential with mean \(\kappa_i\) where \(j\) is the index in the current generated parallel batch and \(i\) is the iteration of the Bayesian optimization loop. UCB corresponds to the Upper Confidence Bound acquisition function. Finally the "d" postfix in qUCBd means that we will only consider the epistemic component of the uncertainty returned by the surrogate model.

• acq_optimizer_*: are parameters related to optimization of the previously defined acquisition function.

• kappa and scheduler: are the parameters that define the schedule of \(\kappa^j_i\) previously mentionned.

• objective_scaler: is a parameter that can be used to rescale the observed objectives (e.g., identity, min-max, log).

search_kwargs = {
    "initial_points": [problem.default_configuration],
    "n_initial_points": 2 * len(problem) + 1,  # Number of initial random points
    "surrogate_model": "ET",  # Use Extra Trees as surrogate model
    "surrogate_model_kwargs": {
        "n_estimators": 50,  # Relatively small number of trees in the surrogate to make it "fast"
        "min_samples_split": 8,  # Larger number to avoid small leaf nodes (smoothing the objective response)
    },
    "multi_point_strategy": "qUCBd",  # Multi-point strategy for asynchronous batch generations (explained later)
    "acq_optimizer": "sampling",  # Use random sampling for the acquisition function optimizer
    "filter_duplicated": False,  # Deactivate filtration of duplicated new points
    "kappa": 10.0,  # Initial value of exploration-exploitation parameter for the acquisition function
    "scheduler": {  # Scheduler for the exploration-exploitation parameter "kappa"
        "type": "periodic-exp-decay",  # Periodic exponential decay
        "period": 50,  # Period over which the decay is applied. It is useful to escape local solutions.
        "kappa_final": 0.001,  # Value of kappa at the end of each "period"
    },
    "objective_scaler": "identity",
    "random_state": 42,  # Random seed
}

Then we can run the optimization.

from deephyper.hpo import CBO
from deephyper.evaluator import Evaluator
from deephyper.evaluator.callback import TqdmCallback


def run_hpo(problem):

    pathlib.Path(model_checkpoint_dir).mkdir(parents=True, exist_ok=True)

    evaluator = Evaluator.create(
        run,
        method="ray",
        method_kwargs={
            "num_cpus_per_task": 1,
            "run_function_kwargs": {
                "model_checkpoint_dir": model_checkpoint_dir,
                "verbose": False,
            },
            "callbacks": [TqdmCallback()]
        },
    )
    search = CBO(
        problem,
        evaluator,
        log_dir=hpo_dir,
        **search_kwargs,
    )

    results = search.search(max_evals=1_000)

    return results

results = run_hpo(problem)

2025-03-11 09:53:29,922 INFO worker.py:1841 -- Started a local Ray instance.
WARNING:root:Results file already exists, it will be renamed to /Users/romainegele/Documents/DeepHyper/deephyper/examples/examples_uq/hpo_sklearn_classification/results_20250311-095331.csv

  0%|          | 0/1000 [00:00<?, ?it/s]
  0%|          | 1/1000 [00:00<00:00, 3039.35it/s, failures=0, objective=-0.525]
  0%|          | 2/1000 [00:00<00:15, 64.19it/s, failures=0, objective=-0.525]
  0%|          | 3/1000 [00:00<00:11, 84.16it/s, failures=0, objective=-0.525]
  0%|          | 4/1000 [00:00<00:13, 76.03it/s, failures=0, objective=-0.525]
  0%|          | 5/1000 [00:00<00:11, 86.96it/s, failures=0, objective=-0.501]
  1%|          | 6/1000 [00:00<00:10, 97.26it/s, failures=0, objective=-0.501]
  1%|          | 7/1000 [00:00<00:11, 87.88it/s, failures=0, objective=-0.501]
  1%|          | 8/1000 [00:00<00:10, 96.72it/s, failures=0, objective=-0.501]
  1%|          | 9/1000 [00:00<00:09, 106.06it/s, failures=0, objective=-0.501]
  1%|          | 10/1000 [00:00<00:08, 114.55it/s, failures=0, objective=-0.501]
  1%|          | 11/1000 [00:00<00:08, 121.85it/s, failures=0, objective=-0.482]
  1%|          | 12/1000 [00:00<00:09, 108.03it/s, failures=0, objective=-0.482]
  1%|          | 12/1000 [00:00<00:09, 108.03it/s, failures=0, objective=-0.482]
  1%|▏         | 13/1000 [00:00<00:09, 108.03it/s, failures=0, objective=-0.482]
  1%|▏         | 14/1000 [00:00<00:09, 108.03it/s, failures=0, objective=-0.482]
  2%|▏         | 15/1000 [00:00<00:09, 108.03it/s, failures=0, objective=-0.482]
  2%|▏         | 16/1000 [00:00<00:09, 108.03it/s, failures=0, objective=-0.482]
  2%|▏         | 17/1000 [00:00<00:09, 108.03it/s, failures=0, objective=-0.482]
  2%|▏         | 18/1000 [00:00<00:09, 108.03it/s, failures=0, objective=-0.482]
  2%|▏         | 19/1000 [00:00<00:09, 108.03it/s, failures=0, objective=-0.482]
  2%|▏         | 20/1000 [00:00<00:09, 108.03it/s, failures=0, objective=-0.482]
  2%|▏         | 21/1000 [00:00<00:09, 108.03it/s, failures=0, objective=-0.482]
  2%|▏         | 22/1000 [00:00<00:09, 108.03it/s, failures=0, objective=-0.482]
  2%|▏         | 23/1000 [00:00<00:15, 64.15it/s, failures=0, objective=-0.482]
  2%|▏         | 23/1000 [00:00<00:15, 64.15it/s, failures=0, objective=-0.482]
  2%|▏         | 24/1000 [00:00<00:15, 64.15it/s, failures=0, objective=-0.48]
  2%|▎         | 25/1000 [00:00<00:15, 64.15it/s, failures=0, objective=-0.48]
  3%|▎         | 26/1000 [00:00<00:15, 64.15it/s, failures=0, objective=-0.48]
  3%|▎         | 27/1000 [00:00<00:15, 64.15it/s, failures=0, objective=-0.48]
  3%|▎         | 28/1000 [00:00<00:15, 64.15it/s, failures=0, objective=-0.48]
  3%|▎         | 29/1000 [00:00<00:15, 64.15it/s, failures=0, objective=-0.48]
  3%|▎         | 30/1000 [00:00<00:15, 64.15it/s, failures=0, objective=-0.48]
  3%|▎         | 31/1000 [00:00<00:19, 49.75it/s, failures=0, objective=-0.48]
  3%|▎         | 31/1000 [00:00<00:19, 49.75it/s, failures=0, objective=-0.48]
  3%|▎         | 32/1000 [00:00<00:19, 49.75it/s, failures=0, objective=-0.48]
  3%|▎         | 33/1000 [00:00<00:19, 49.75it/s, failures=0, objective=-0.48]
  3%|▎         | 34/1000 [00:00<00:19, 49.75it/s, failures=0, objective=-0.48]
  4%|▎         | 35/1000 [00:00<00:19, 49.75it/s, failures=0, objective=-0.477]
  4%|▎         | 36/1000 [00:00<00:19, 49.75it/s, failures=0, objective=-0.477]
  4%|▎         | 37/1000 [00:00<00:19, 49.54it/s, failures=0, objective=-0.477]
  4%|▎         | 37/1000 [00:00<00:19, 49.54it/s, failures=0, objective=-0.477]
  4%|▍         | 38/1000 [00:00<00:19, 49.54it/s, failures=0, objective=-0.477]
  4%|▍         | 39/1000 [00:00<00:19, 49.54it/s, failures=0, objective=-0.477]
  4%|▍         | 40/1000 [00:00<00:19, 49.54it/s, failures=0, objective=-0.477]
  4%|▍         | 41/1000 [00:00<00:19, 49.54it/s, failures=0, objective=-0.477]
  4%|▍         | 42/1000 [00:00<00:19, 49.54it/s, failures=0, objective=-0.477]
  4%|▍         | 43/1000 [00:00<00:19, 49.13it/s, failures=0, objective=-0.477]
  4%|▍         | 43/1000 [00:00<00:19, 49.13it/s, failures=0, objective=-0.477]
  4%|▍         | 44/1000 [00:00<00:19, 49.13it/s, failures=0, objective=-0.477]
  4%|▍         | 45/1000 [00:00<00:19, 49.13it/s, failures=0, objective=-0.477]
  5%|▍         | 46/1000 [00:00<00:19, 49.13it/s, failures=0, objective=-0.477]
  5%|▍         | 47/1000 [00:00<00:19, 49.13it/s, failures=0, objective=-0.477]
  5%|▍         | 48/1000 [00:00<00:19, 49.13it/s, failures=0, objective=-0.477]
  5%|▍         | 49/1000 [00:00<00:19, 49.07it/s, failures=0, objective=-0.477]
  5%|▍         | 49/1000 [00:00<00:19, 49.07it/s, failures=0, objective=-0.477]
  5%|▌         | 50/1000 [00:00<00:19, 49.07it/s, failures=0, objective=-0.477]
  5%|▌         | 51/1000 [00:01<00:19, 49.07it/s, failures=0, objective=-0.477]
  5%|▌         | 52/1000 [00:01<00:19, 49.07it/s, failures=0, objective=-0.477]
  5%|▌         | 53/1000 [00:01<00:19, 49.07it/s, failures=0, objective=-0.477]
  5%|▌         | 54/1000 [00:01<00:19, 49.07it/s, failures=0, objective=-0.477]
  6%|▌         | 55/1000 [00:01<00:19, 49.42it/s, failures=0, objective=-0.477]
  6%|▌         | 55/1000 [00:01<00:19, 49.42it/s, failures=0, objective=-0.477]
  6%|▌         | 56/1000 [00:01<00:19, 49.42it/s, failures=0, objective=-0.477]
  6%|▌         | 57/1000 [00:01<00:19, 49.42it/s, failures=0, objective=-0.477]
  6%|▌         | 58/1000 [00:01<00:19, 49.42it/s, failures=0, objective=-0.477]
  6%|▌         | 59/1000 [00:01<00:19, 49.42it/s, failures=0, objective=-0.477]
  6%|▌         | 60/1000 [00:01<00:19, 49.42it/s, failures=0, objective=-0.477]
  6%|▌         | 61/1000 [00:01<00:24, 38.98it/s, failures=0, objective=-0.477]
  6%|▌         | 61/1000 [00:01<00:24, 38.98it/s, failures=0, objective=-0.477]
  6%|▌         | 62/1000 [00:01<00:24, 38.98it/s, failures=0, objective=-0.477]
  6%|▋         | 63/1000 [00:01<00:24, 38.98it/s, failures=0, objective=-0.477]
  6%|▋         | 64/1000 [00:01<00:24, 38.98it/s, failures=0, objective=-0.477]
  6%|▋         | 65/1000 [00:01<00:23, 38.98it/s, failures=0, objective=-0.477]
  7%|▋         | 66/1000 [00:01<00:23, 39.56it/s, failures=0, objective=-0.477]
  7%|▋         | 66/1000 [00:01<00:23, 39.56it/s, failures=0, objective=-0.477]
  7%|▋         | 67/1000 [00:01<00:23, 39.56it/s, failures=0, objective=-0.429]
  7%|▋         | 68/1000 [00:01<00:23, 39.56it/s, failures=0, objective=-0.429]
  7%|▋         | 69/1000 [00:01<00:23, 39.56it/s, failures=0, objective=-0.429]
  7%|▋         | 70/1000 [00:01<00:23, 39.56it/s, failures=0, objective=-0.429]
  7%|▋         | 71/1000 [00:01<00:23, 39.92it/s, failures=0, objective=-0.429]
  7%|▋         | 71/1000 [00:01<00:23, 39.92it/s, failures=0, objective=-0.429]
  7%|▋         | 72/1000 [00:01<00:23, 39.92it/s, failures=0, objective=-0.429]
  7%|▋         | 73/1000 [00:01<00:23, 39.92it/s, failures=0, objective=-0.429]
  7%|▋         | 74/1000 [00:01<00:23, 39.92it/s, failures=0, objective=-0.429]
  8%|▊         | 75/1000 [00:01<00:23, 39.92it/s, failures=0, objective=-0.429]
  8%|▊         | 76/1000 [00:01<00:23, 39.95it/s, failures=0, objective=-0.429]
  8%|▊         | 76/1000 [00:01<00:23, 39.95it/s, failures=0, objective=-0.429]
  8%|▊         | 77/1000 [00:01<00:23, 39.95it/s, failures=0, objective=-0.429]
  8%|▊         | 78/1000 [00:01<00:23, 39.95it/s, failures=0, objective=-0.429]
  8%|▊         | 79/1000 [00:01<00:23, 39.95it/s, failures=0, objective=-0.429]
  8%|▊         | 80/1000 [00:01<00:23, 39.95it/s, failures=0, objective=-0.429]
  8%|▊         | 81/1000 [00:01<00:22, 39.96it/s, failures=0, objective=-0.429]
  8%|▊         | 81/1000 [00:01<00:22, 39.96it/s, failures=0, objective=-0.429]
  8%|▊         | 82/1000 [00:01<00:22, 39.96it/s, failures=0, objective=-0.429]
  8%|▊         | 83/1000 [00:01<00:22, 39.96it/s, failures=0, objective=-0.429]
  8%|▊         | 84/1000 [00:01<00:22, 39.96it/s, failures=0, objective=-0.429]
  8%|▊         | 85/1000 [00:01<00:22, 39.96it/s, failures=0, objective=-0.429]
  9%|▊         | 86/1000 [00:01<00:22, 39.90it/s, failures=0, objective=-0.429]
  9%|▊         | 86/1000 [00:01<00:22, 39.90it/s, failures=0, objective=-0.429]
  9%|▊         | 87/1000 [00:01<00:22, 39.90it/s, failures=0, objective=-0.429]
  9%|▉         | 88/1000 [00:01<00:22, 39.90it/s, failures=0, objective=-0.429]
  9%|▉         | 89/1000 [00:01<00:22, 39.90it/s, failures=0, objective=-0.429]
  9%|▉         | 90/1000 [00:01<00:22, 39.90it/s, failures=0, objective=-0.429]
  9%|▉         | 91/1000 [00:02<00:22, 39.89it/s, failures=0, objective=-0.429]
  9%|▉         | 91/1000 [00:02<00:22, 39.89it/s, failures=0, objective=-0.429]
  9%|▉         | 92/1000 [00:02<00:22, 39.89it/s, failures=0, objective=-0.429]
  9%|▉         | 93/1000 [00:02<00:22, 39.89it/s, failures=0, objective=-0.429]
  9%|▉         | 94/1000 [00:02<00:22, 39.89it/s, failures=0, objective=-0.429]
 10%|▉         | 95/1000 [00:02<00:22, 39.89it/s, failures=0, objective=-0.429]
 10%|▉         | 96/1000 [00:02<00:22, 39.81it/s, failures=0, objective=-0.429]
 10%|▉         | 96/1000 [00:02<00:22, 39.81it/s, failures=0, objective=-0.429]
 10%|▉         | 97/1000 [00:02<00:22, 39.81it/s, failures=0, objective=-0.429]
 10%|▉         | 98/1000 [00:02<00:22, 39.81it/s, failures=0, objective=-0.429]
 10%|▉         | 99/1000 [00:02<00:22, 39.81it/s, failures=0, objective=-0.429]
 10%|█         | 100/1000 [00:02<00:22, 39.81it/s, failures=0, objective=-0.429]
 10%|█         | 101/1000 [00:02<00:22, 39.92it/s, failures=0, objective=-0.429]
 10%|█         | 101/1000 [00:02<00:22, 39.92it/s, failures=0, objective=-0.429]
 10%|█         | 102/1000 [00:02<00:22, 39.92it/s, failures=0, objective=-0.429]
 10%|█         | 103/1000 [00:02<00:22, 39.92it/s, failures=0, objective=-0.429]
 10%|█         | 104/1000 [00:02<00:22, 39.92it/s, failures=0, objective=-0.429]
 10%|█         | 105/1000 [00:02<00:22, 39.92it/s, failures=0, objective=-0.429]
 11%|█         | 106/1000 [00:02<00:22, 39.56it/s, failures=0, objective=-0.429]
 11%|█         | 106/1000 [00:02<00:22, 39.56it/s, failures=0, objective=-0.429]
 11%|█         | 107/1000 [00:02<00:22, 39.56it/s, failures=0, objective=-0.429]
 11%|█         | 108/1000 [00:02<00:22, 39.56it/s, failures=0, objective=-0.429]
 11%|█         | 109/1000 [00:02<00:22, 39.56it/s, failures=0, objective=-0.429]
 11%|█         | 110/1000 [00:02<00:22, 39.56it/s, failures=0, objective=-0.429]
 11%|█         | 111/1000 [00:02<00:22, 39.68it/s, failures=0, objective=-0.429]
 11%|█         | 111/1000 [00:02<00:22, 39.68it/s, failures=0, objective=-0.429]
 11%|█         | 112/1000 [00:02<00:22, 39.68it/s, failures=0, objective=-0.429]
 11%|█▏        | 113/1000 [00:02<00:22, 39.68it/s, failures=0, objective=-0.429]
 11%|█▏        | 114/1000 [00:02<00:22, 39.68it/s, failures=0, objective=-0.429]
 12%|█▏        | 115/1000 [00:02<00:22, 39.68it/s, failures=0, objective=-0.429]
 12%|█▏        | 116/1000 [00:02<00:22, 39.57it/s, failures=0, objective=-0.429]
 12%|█▏        | 116/1000 [00:02<00:22, 39.57it/s, failures=0, objective=-0.429]
 12%|█▏        | 117/1000 [00:02<00:22, 39.57it/s, failures=0, objective=-0.429]
 12%|█▏        | 118/1000 [00:02<00:22, 39.57it/s, failures=0, objective=-0.42]
 12%|█▏        | 119/1000 [00:02<00:22, 39.57it/s, failures=0, objective=-0.42]
 12%|█▏        | 120/1000 [00:02<00:22, 39.57it/s, failures=0, objective=-0.42]
 12%|█▏        | 121/1000 [00:02<00:22, 39.44it/s, failures=0, objective=-0.42]
 12%|█▏        | 121/1000 [00:02<00:22, 39.44it/s, failures=0, objective=-0.369]
 12%|█▏        | 122/1000 [00:02<00:22, 39.44it/s, failures=0, objective=-0.369]
 12%|█▏        | 123/1000 [00:02<00:22, 39.44it/s, failures=0, objective=-0.369]
 12%|█▏        | 124/1000 [00:02<00:22, 39.44it/s, failures=0, objective=-0.369]
 12%|█▎        | 125/1000 [00:02<00:22, 39.44it/s, failures=0, objective=-0.369]
 13%|█▎        | 126/1000 [00:02<00:22, 39.28it/s, failures=0, objective=-0.369]
 13%|█▎        | 126/1000 [00:02<00:22, 39.28it/s, failures=0, objective=-0.369]
 13%|█▎        | 127/1000 [00:02<00:22, 39.28it/s, failures=0, objective=-0.369]
 13%|█▎        | 128/1000 [00:02<00:22, 39.28it/s, failures=0, objective=-0.369]
 13%|█▎        | 129/1000 [00:02<00:22, 39.28it/s, failures=0, objective=-0.369]
 13%|█▎        | 130/1000 [00:02<00:22, 39.28it/s, failures=0, objective=-0.369]
 13%|█▎        | 131/1000 [00:03<00:22, 39.07it/s, failures=0, objective=-0.369]
 13%|█▎        | 131/1000 [00:03<00:22, 39.07it/s, failures=0, objective=-0.369]
 13%|█▎        | 132/1000 [00:03<00:22, 39.07it/s, failures=0, objective=-0.369]
 13%|█▎        | 133/1000 [00:03<00:22, 39.07it/s, failures=0, objective=-0.369]
 13%|█▎        | 134/1000 [00:03<00:22, 39.07it/s, failures=0, objective=-0.369]
 14%|█▎        | 135/1000 [00:03<00:22, 39.07it/s, failures=0, objective=-0.369]
 14%|█▎        | 136/1000 [00:03<00:22, 38.73it/s, failures=0, objective=-0.369]
 14%|█▎        | 136/1000 [00:03<00:22, 38.73it/s, failures=0, objective=-0.369]
 14%|█▎        | 137/1000 [00:03<00:22, 38.73it/s, failures=0, objective=-0.369]
 14%|█▍        | 138/1000 [00:03<00:22, 38.73it/s, failures=0, objective=-0.369]
 14%|█▍        | 139/1000 [00:03<00:22, 38.73it/s, failures=0, objective=-0.369]
 14%|█▍        | 140/1000 [00:03<00:22, 38.73it/s, failures=0, objective=-0.369]
 14%|█▍        | 141/1000 [00:03<00:22, 38.64it/s, failures=0, objective=-0.369]
 14%|█▍        | 141/1000 [00:03<00:22, 38.64it/s, failures=0, objective=-0.369]
 14%|█▍        | 142/1000 [00:03<00:22, 38.64it/s, failures=0, objective=-0.369]
 14%|█▍        | 143/1000 [00:03<00:22, 38.64it/s, failures=0, objective=-0.369]
 14%|█▍        | 144/1000 [00:03<00:22, 38.64it/s, failures=0, objective=-0.369]
 14%|█▍        | 145/1000 [00:03<00:22, 38.64it/s, failures=0, objective=-0.369]
 15%|█▍        | 146/1000 [00:03<00:22, 38.31it/s, failures=0, objective=-0.369]
 15%|█▍        | 146/1000 [00:03<00:22, 38.31it/s, failures=0, objective=-0.369]
 15%|█▍        | 147/1000 [00:03<00:22, 38.31it/s, failures=0, objective=-0.369]
 15%|█▍        | 148/1000 [00:03<00:22, 38.31it/s, failures=0, objective=-0.369]
 15%|█▍        | 149/1000 [00:03<00:22, 38.31it/s, failures=0, objective=-0.369]
 15%|█▌        | 150/1000 [00:03<00:22, 38.31it/s, failures=0, objective=-0.369]
 15%|█▌        | 151/1000 [00:03<00:22, 38.10it/s, failures=0, objective=-0.369]
 15%|█▌        | 151/1000 [00:03<00:22, 38.10it/s, failures=0, objective=-0.369]
 15%|█▌        | 152/1000 [00:03<00:22, 38.10it/s, failures=0, objective=-0.369]
 15%|█▌        | 153/1000 [00:03<00:22, 38.10it/s, failures=0, objective=-0.369]
 15%|█▌        | 154/1000 [00:03<00:22, 38.10it/s, failures=0, objective=-0.369]
 16%|█▌        | 155/1000 [00:03<00:22, 38.10it/s, failures=0, objective=-0.369]
 16%|█▌        | 156/1000 [00:03<00:22, 37.83it/s, failures=0, objective=-0.369]
 16%|█▌        | 156/1000 [00:03<00:22, 37.83it/s, failures=0, objective=-0.369]
 16%|█▌        | 157/1000 [00:03<00:22, 37.83it/s, failures=0, objective=-0.369]
 16%|█▌        | 158/1000 [00:03<00:22, 37.83it/s, failures=0, objective=-0.369]
 16%|█▌        | 159/1000 [00:03<00:22, 37.83it/s, failures=0, objective=-0.369]
 16%|█▌        | 160/1000 [00:03<00:22, 37.83it/s, failures=0, objective=-0.369]
 16%|█▌        | 161/1000 [00:03<00:22, 37.78it/s, failures=0, objective=-0.369]
 16%|█▌        | 161/1000 [00:03<00:22, 37.78it/s, failures=0, objective=-0.369]
 16%|█▌        | 162/1000 [00:03<00:22, 37.78it/s, failures=0, objective=-0.369]
 16%|█▋        | 163/1000 [00:03<00:22, 37.78it/s, failures=0, objective=-0.369]
 16%|█▋        | 164/1000 [00:03<00:22, 37.78it/s, failures=0, objective=-0.369]
 16%|█▋        | 165/1000 [00:03<00:22, 37.78it/s, failures=0, objective=-0.369]
 17%|█▋        | 166/1000 [00:03<00:22, 37.42it/s, failures=0, objective=-0.369]
 17%|█▋        | 166/1000 [00:03<00:22, 37.42it/s, failures=0, objective=-0.369]
 17%|█▋        | 167/1000 [00:03<00:22, 37.42it/s, failures=0, objective=-0.369]
 17%|█▋        | 168/1000 [00:03<00:22, 37.42it/s, failures=0, objective=-0.369]
 17%|█▋        | 169/1000 [00:03<00:22, 37.42it/s, failures=0, objective=-0.369]
 17%|█▋        | 170/1000 [00:03<00:22, 37.42it/s, failures=0, objective=-0.369]
 17%|█▋        | 171/1000 [00:04<00:22, 37.36it/s, failures=0, objective=-0.369]
 17%|█▋        | 171/1000 [00:04<00:22, 37.36it/s, failures=0, objective=-0.369]
 17%|█▋        | 172/1000 [00:04<00:22, 37.36it/s, failures=0, objective=-0.369]
 17%|█▋        | 173/1000 [00:04<00:22, 37.36it/s, failures=0, objective=-0.369]
 17%|█▋        | 174/1000 [00:04<00:22, 37.36it/s, failures=0, objective=-0.369]
 18%|█▊        | 175/1000 [00:04<00:22, 37.36it/s, failures=0, objective=-0.369]
 18%|█▊        | 176/1000 [00:04<00:22, 37.44it/s, failures=0, objective=-0.369]
 18%|█▊        | 176/1000 [00:04<00:22, 37.44it/s, failures=0, objective=-0.369]
 18%|█▊        | 177/1000 [00:04<00:21, 37.44it/s, failures=0, objective=-0.369]
 18%|█▊        | 178/1000 [00:04<00:21, 37.44it/s, failures=0, objective=-0.369]
 18%|█▊        | 179/1000 [00:04<00:21, 37.44it/s, failures=0, objective=-0.369]
 18%|█▊        | 180/1000 [00:04<00:21, 37.44it/s, failures=0, objective=-0.369]
 18%|█▊        | 181/1000 [00:04<00:21, 37.38it/s, failures=0, objective=-0.369]
 18%|█▊        | 181/1000 [00:04<00:21, 37.38it/s, failures=0, objective=-0.369]
 18%|█▊        | 182/1000 [00:04<00:21, 37.38it/s, failures=0, objective=-0.369]
 18%|█▊        | 183/1000 [00:04<00:21, 37.38it/s, failures=0, objective=-0.369]
 18%|█▊        | 184/1000 [00:04<00:21, 37.38it/s, failures=0, objective=-0.369]
 18%|█▊        | 185/1000 [00:04<00:21, 37.38it/s, failures=0, objective=-0.369]
 19%|█▊        | 186/1000 [00:04<00:21, 37.27it/s, failures=0, objective=-0.369]
 19%|█▊        | 186/1000 [00:04<00:21, 37.27it/s, failures=0, objective=-0.369]
 19%|█▊        | 187/1000 [00:04<00:21, 37.27it/s, failures=0, objective=-0.369]
 19%|█▉        | 188/1000 [00:04<00:21, 37.27it/s, failures=0, objective=-0.369]
 19%|█▉        | 189/1000 [00:04<00:21, 37.27it/s, failures=0, objective=-0.369]
 19%|█▉        | 190/1000 [00:04<00:21, 37.27it/s, failures=0, objective=-0.369]
 19%|█▉        | 191/1000 [00:04<00:22, 36.58it/s, failures=0, objective=-0.369]
 19%|█▉        | 191/1000 [00:04<00:22, 36.58it/s, failures=0, objective=-0.369]
 19%|█▉        | 192/1000 [00:04<00:22, 36.58it/s, failures=0, objective=-0.369]
 19%|█▉        | 193/1000 [00:04<00:22, 36.58it/s, failures=0, objective=-0.369]
 19%|█▉        | 194/1000 [00:04<00:22, 36.58it/s, failures=0, objective=-0.369]
 20%|█▉        | 195/1000 [00:04<00:22, 36.58it/s, failures=0, objective=-0.369]
 20%|█▉        | 196/1000 [00:04<00:22, 36.53it/s, failures=0, objective=-0.369]
 20%|█▉        | 196/1000 [00:04<00:22, 36.53it/s, failures=0, objective=-0.369]
 20%|█▉        | 197/1000 [00:04<00:21, 36.53it/s, failures=0, objective=-0.369]
 20%|█▉        | 198/1000 [00:04<00:21, 36.53it/s, failures=0, objective=-0.369]
 20%|█▉        | 199/1000 [00:04<00:21, 36.53it/s, failures=0, objective=-0.369]
 20%|██        | 200/1000 [00:04<00:21, 36.53it/s, failures=0, objective=-0.369]
 20%|██        | 201/1000 [00:04<00:21, 36.58it/s, failures=0, objective=-0.369]
 20%|██        | 201/1000 [00:04<00:21, 36.58it/s, failures=0, objective=-0.369]
 20%|██        | 202/1000 [00:04<00:21, 36.58it/s, failures=0, objective=-0.369]
 20%|██        | 203/1000 [00:04<00:21, 36.58it/s, failures=0, objective=-0.369]
 20%|██        | 204/1000 [00:04<00:21, 36.58it/s, failures=0, objective=-0.369]
 20%|██        | 205/1000 [00:04<00:21, 36.58it/s, failures=0, objective=-0.369]
 21%|██        | 206/1000 [00:05<00:21, 36.60it/s, failures=0, objective=-0.369]
 21%|██        | 206/1000 [00:05<00:21, 36.60it/s, failures=0, objective=-0.369]
 21%|██        | 207/1000 [00:05<00:21, 36.60it/s, failures=0, objective=-0.369]
 21%|██        | 208/1000 [00:05<00:21, 36.60it/s, failures=0, objective=-0.369]
 21%|██        | 209/1000 [00:05<00:21, 36.60it/s, failures=0, objective=-0.369]
 21%|██        | 210/1000 [00:05<00:21, 36.60it/s, failures=0, objective=-0.369]
 21%|██        | 211/1000 [00:05<00:21, 36.17it/s, failures=0, objective=-0.369]
 21%|██        | 211/1000 [00:05<00:21, 36.17it/s, failures=0, objective=-0.369]
 21%|██        | 212/1000 [00:05<00:21, 36.17it/s, failures=0, objective=-0.369]
 21%|██▏       | 213/1000 [00:05<00:21, 36.17it/s, failures=0, objective=-0.369]
 21%|██▏       | 214/1000 [00:05<00:21, 36.17it/s, failures=0, objective=-0.369]
 22%|██▏       | 215/1000 [00:05<00:21, 36.17it/s, failures=0, objective=-0.369]
 22%|██▏       | 216/1000 [00:05<00:21, 36.36it/s, failures=0, objective=-0.369]
 22%|██▏       | 216/1000 [00:05<00:21, 36.36it/s, failures=0, objective=-0.369]
 22%|██▏       | 217/1000 [00:05<00:21, 36.36it/s, failures=0, objective=-0.369]
 22%|██▏       | 218/1000 [00:05<00:21, 36.36it/s, failures=0, objective=-0.369]
 22%|██▏       | 219/1000 [00:05<00:21, 36.36it/s, failures=0, objective=-0.369]
 22%|██▏       | 220/1000 [00:05<00:21, 36.36it/s, failures=0, objective=-0.369]
 22%|██▏       | 221/1000 [00:05<00:21, 35.76it/s, failures=0, objective=-0.369]
 22%|██▏       | 221/1000 [00:05<00:21, 35.76it/s, failures=0, objective=-0.369]
 22%|██▏       | 222/1000 [00:05<00:21, 35.76it/s, failures=0, objective=-0.369]
 22%|██▏       | 223/1000 [00:05<00:21, 35.76it/s, failures=0, objective=-0.369]
 22%|██▏       | 224/1000 [00:05<00:21, 35.76it/s, failures=0, objective=-0.369]
 22%|██▎       | 225/1000 [00:05<00:21, 35.76it/s, failures=0, objective=-0.369]
 23%|██▎       | 226/1000 [00:05<00:21, 35.80it/s, failures=0, objective=-0.369]
 23%|██▎       | 226/1000 [00:05<00:21, 35.80it/s, failures=0, objective=-0.369]
 23%|██▎       | 227/1000 [00:05<00:21, 35.80it/s, failures=0, objective=-0.369]
 23%|██▎       | 228/1000 [00:05<00:21, 35.80it/s, failures=0, objective=-0.369]
 23%|██▎       | 229/1000 [00:05<00:21, 35.80it/s, failures=0, objective=-0.369]
 23%|██▎       | 230/1000 [00:05<00:21, 35.80it/s, failures=0, objective=-0.369]
 23%|██▎       | 231/1000 [00:05<00:21, 35.88it/s, failures=0, objective=-0.369]
 23%|██▎       | 231/1000 [00:05<00:21, 35.88it/s, failures=0, objective=-0.369]
 23%|██▎       | 232/1000 [00:05<00:21, 35.88it/s, failures=0, objective=-0.369]
 23%|██▎       | 233/1000 [00:05<00:21, 35.88it/s, failures=0, objective=-0.369]
 23%|██▎       | 234/1000 [00:05<00:21, 35.88it/s, failures=0, objective=-0.369]
 24%|██▎       | 235/1000 [00:05<00:21, 35.88it/s, failures=0, objective=-0.369]
 24%|██▎       | 236/1000 [00:05<00:21, 35.88it/s, failures=0, objective=-0.369]
 24%|██▎       | 236/1000 [00:05<00:21, 35.88it/s, failures=0, objective=-0.369]
 24%|██▎       | 237/1000 [00:05<00:21, 35.88it/s, failures=0, objective=-0.369]
 24%|██▍       | 238/1000 [00:05<00:21, 35.88it/s, failures=0, objective=-0.369]
 24%|██▍       | 239/1000 [00:05<00:21, 35.88it/s, failures=0, objective=-0.369]
 24%|██▍       | 240/1000 [00:05<00:21, 35.88it/s, failures=0, objective=-0.369]
 24%|██▍       | 241/1000 [00:06<00:21, 35.83it/s, failures=0, objective=-0.369]
 24%|██▍       | 241/1000 [00:06<00:21, 35.83it/s, failures=0, objective=-0.369]
 24%|██▍       | 242/1000 [00:06<00:21, 35.83it/s, failures=0, objective=-0.369]
 24%|██▍       | 243/1000 [00:06<00:21, 35.83it/s, failures=0, objective=-0.369]
 24%|██▍       | 244/1000 [00:06<00:21, 35.83it/s, failures=0, objective=-0.369]
 24%|██▍       | 245/1000 [00:06<00:21, 35.83it/s, failures=0, objective=-0.369]
 25%|██▍       | 246/1000 [00:06<00:21, 35.68it/s, failures=0, objective=-0.369]
 25%|██▍       | 246/1000 [00:06<00:21, 35.68it/s, failures=0, objective=-0.369]
 25%|██▍       | 247/1000 [00:06<00:21, 35.68it/s, failures=0, objective=-0.369]
 25%|██▍       | 248/1000 [00:06<00:21, 35.68it/s, failures=0, objective=-0.369]
 25%|██▍       | 249/1000 [00:06<00:21, 35.68it/s, failures=0, objective=-0.369]
 25%|██▌       | 250/1000 [00:06<00:21, 35.68it/s, failures=0, objective=-0.369]
 25%|██▌       | 251/1000 [00:06<00:21, 35.59it/s, failures=0, objective=-0.369]
 25%|██▌       | 251/1000 [00:06<00:21, 35.59it/s, failures=0, objective=-0.369]
 25%|██▌       | 252/1000 [00:06<00:21, 35.59it/s, failures=0, objective=-0.369]
 25%|██▌       | 253/1000 [00:06<00:20, 35.59it/s, failures=0, objective=-0.369]
 25%|██▌       | 254/1000 [00:06<00:20, 35.59it/s, failures=0, objective=-0.369]
 26%|██▌       | 255/1000 [00:06<00:20, 35.59it/s, failures=0, objective=-0.369]
 26%|██▌       | 256/1000 [00:06<00:20, 35.47it/s, failures=0, objective=-0.369]
 26%|██▌       | 256/1000 [00:06<00:20, 35.47it/s, failures=0, objective=-0.369]
 26%|██▌       | 257/1000 [00:06<00:20, 35.47it/s, failures=0, objective=-0.369]
 26%|██▌       | 258/1000 [00:06<00:20, 35.47it/s, failures=0, objective=-0.369]
 26%|██▌       | 259/1000 [00:06<00:20, 35.47it/s, failures=0, objective=-0.369]
 26%|██▌       | 260/1000 [00:06<00:20, 35.47it/s, failures=0, objective=-0.369]
 26%|██▌       | 261/1000 [00:06<00:20, 35.45it/s, failures=0, objective=-0.369]
 26%|██▌       | 261/1000 [00:06<00:20, 35.45it/s, failures=0, objective=-0.369]
 26%|██▌       | 262/1000 [00:06<00:20, 35.45it/s, failures=0, objective=-0.369]
 26%|██▋       | 263/1000 [00:06<00:20, 35.45it/s, failures=0, objective=-0.369]
 26%|██▋       | 264/1000 [00:06<00:20, 35.45it/s, failures=0, objective=-0.369]
 26%|██▋       | 265/1000 [00:06<00:20, 35.45it/s, failures=0, objective=-0.369]
 27%|██▋       | 266/1000 [00:06<00:20, 35.37it/s, failures=0, objective=-0.369]
 27%|██▋       | 266/1000 [00:06<00:20, 35.37it/s, failures=0, objective=-0.369]
 27%|██▋       | 267/1000 [00:06<00:20, 35.37it/s, failures=0, objective=-0.369]
 27%|██▋       | 268/1000 [00:06<00:20, 35.37it/s, failures=0, objective=-0.369]
 27%|██▋       | 269/1000 [00:06<00:20, 35.37it/s, failures=0, objective=-0.369]
 27%|██▋       | 270/1000 [00:06<00:20, 35.37it/s, failures=0, objective=-0.369]
 27%|██▋       | 271/1000 [00:06<00:20, 35.52it/s, failures=0, objective=-0.369]
 27%|██▋       | 271/1000 [00:06<00:20, 35.52it/s, failures=0, objective=-0.369]
 27%|██▋       | 272/1000 [00:06<00:20, 35.52it/s, failures=0, objective=-0.369]
 27%|██▋       | 273/1000 [00:06<00:20, 35.52it/s, failures=0, objective=-0.369]
 27%|██▋       | 274/1000 [00:06<00:20, 35.52it/s, failures=0, objective=-0.369]
 28%|██▊       | 275/1000 [00:06<00:20, 35.52it/s, failures=0, objective=-0.369]
 28%|██▊       | 276/1000 [00:07<00:20, 35.59it/s, failures=0, objective=-0.369]
 28%|██▊       | 276/1000 [00:07<00:20, 35.59it/s, failures=0, objective=-0.369]
 28%|██▊       | 277/1000 [00:07<00:20, 35.59it/s, failures=0, objective=-0.369]
 28%|██▊       | 278/1000 [00:07<00:20, 35.59it/s, failures=0, objective=-0.369]
 28%|██▊       | 279/1000 [00:07<00:20, 35.59it/s, failures=0, objective=-0.369]
 28%|██▊       | 280/1000 [00:07<00:20, 35.59it/s, failures=0, objective=-0.369]
 28%|██▊       | 281/1000 [00:07<00:20, 35.44it/s, failures=0, objective=-0.369]
 28%|██▊       | 281/1000 [00:07<00:20, 35.44it/s, failures=0, objective=-0.369]
 28%|██▊       | 282/1000 [00:07<00:20, 35.44it/s, failures=0, objective=-0.369]
 28%|██▊       | 283/1000 [00:07<00:20, 35.44it/s, failures=0, objective=-0.369]
 28%|██▊       | 284/1000 [00:07<00:20, 35.44it/s, failures=0, objective=-0.369]
 28%|██▊       | 285/1000 [00:07<00:20, 35.44it/s, failures=0, objective=-0.369]
 29%|██▊       | 286/1000 [00:07<00:20, 35.22it/s, failures=0, objective=-0.369]
 29%|██▊       | 286/1000 [00:07<00:20, 35.22it/s, failures=0, objective=-0.369]
 29%|██▊       | 287/1000 [00:07<00:20, 35.22it/s, failures=0, objective=-0.369]
 29%|██▉       | 288/1000 [00:07<00:20, 35.22it/s, failures=0, objective=-0.369]
 29%|██▉       | 289/1000 [00:07<00:20, 35.22it/s, failures=0, objective=-0.369]
 29%|██▉       | 290/1000 [00:07<00:20, 35.22it/s, failures=0, objective=-0.369]
 29%|██▉       | 291/1000 [00:07<00:20, 35.00it/s, failures=0, objective=-0.369]
 29%|██▉       | 291/1000 [00:07<00:20, 35.00it/s, failures=0, objective=-0.369]
 29%|██▉       | 292/1000 [00:07<00:20, 35.00it/s, failures=0, objective=-0.369]
 29%|██▉       | 293/1000 [00:07<00:20, 35.00it/s, failures=0, objective=-0.369]
 29%|██▉       | 294/1000 [00:07<00:20, 35.00it/s, failures=0, objective=-0.369]
 30%|██▉       | 295/1000 [00:07<00:20, 35.00it/s, failures=0, objective=-0.369]
 30%|██▉       | 296/1000 [00:07<00:20, 34.91it/s, failures=0, objective=-0.369]
 30%|██▉       | 296/1000 [00:07<00:20, 34.91it/s, failures=0, objective=-0.369]
 30%|██▉       | 297/1000 [00:07<00:20, 34.91it/s, failures=0, objective=-0.369]
 30%|██▉       | 298/1000 [00:07<00:20, 34.91it/s, failures=0, objective=-0.369]
 30%|██▉       | 299/1000 [00:07<00:20, 34.91it/s, failures=0, objective=-0.369]
 30%|███       | 300/1000 [00:07<00:20, 34.91it/s, failures=0, objective=-0.369]
 30%|███       | 301/1000 [00:07<00:20, 34.90it/s, failures=0, objective=-0.369]
 30%|███       | 301/1000 [00:07<00:20, 34.90it/s, failures=0, objective=-0.369]
 30%|███       | 302/1000 [00:07<00:19, 34.90it/s, failures=0, objective=-0.369]
 30%|███       | 303/1000 [00:07<00:19, 34.90it/s, failures=0, objective=-0.369]
 30%|███       | 304/1000 [00:07<00:19, 34.90it/s, failures=0, objective=-0.369]
 30%|███       | 305/1000 [00:07<00:19, 34.90it/s, failures=0, objective=-0.369]
 31%|███       | 306/1000 [00:07<00:19, 34.89it/s, failures=0, objective=-0.369]
 31%|███       | 306/1000 [00:07<00:19, 34.89it/s, failures=0, objective=-0.369]
 31%|███       | 307/1000 [00:07<00:19, 34.89it/s, failures=0, objective=-0.369]
 31%|███       | 308/1000 [00:07<00:19, 34.89it/s, failures=0, objective=-0.369]
 31%|███       | 309/1000 [00:07<00:19, 34.89it/s, failures=0, objective=-0.369]
 31%|███       | 310/1000 [00:07<00:19, 34.89it/s, failures=0, objective=-0.369]
 31%|███       | 311/1000 [00:08<00:19, 34.93it/s, failures=0, objective=-0.369]
 31%|███       | 311/1000 [00:08<00:19, 34.93it/s, failures=0, objective=-0.369]
 31%|███       | 312/1000 [00:08<00:19, 34.93it/s, failures=0, objective=-0.369]
 31%|███▏      | 313/1000 [00:08<00:19, 34.93it/s, failures=0, objective=-0.369]
 31%|███▏      | 314/1000 [00:08<00:19, 34.93it/s, failures=0, objective=-0.369]
 32%|███▏      | 315/1000 [00:08<00:19, 34.93it/s, failures=0, objective=-0.369]
 32%|███▏      | 316/1000 [00:08<00:19, 35.02it/s, failures=0, objective=-0.369]
 32%|███▏      | 316/1000 [00:08<00:19, 35.02it/s, failures=0, objective=-0.369]
 32%|███▏      | 317/1000 [00:08<00:19, 35.02it/s, failures=0, objective=-0.369]
 32%|███▏      | 318/1000 [00:08<00:19, 35.02it/s, failures=0, objective=-0.369]
 32%|███▏      | 319/1000 [00:08<00:19, 35.02it/s, failures=0, objective=-0.369]
 32%|███▏      | 320/1000 [00:08<00:19, 35.02it/s, failures=0, objective=-0.369]
 32%|███▏      | 321/1000 [00:08<00:19, 35.08it/s, failures=0, objective=-0.369]
 32%|███▏      | 321/1000 [00:08<00:19, 35.08it/s, failures=0, objective=-0.369]
 32%|███▏      | 322/1000 [00:08<00:19, 35.08it/s, failures=0, objective=-0.369]
 32%|███▏      | 323/1000 [00:08<00:19, 35.08it/s, failures=0, objective=-0.369]
 32%|███▏      | 324/1000 [00:08<00:19, 35.08it/s, failures=0, objective=-0.369]
 32%|███▎      | 325/1000 [00:08<00:19, 35.08it/s, failures=0, objective=-0.369]
 33%|███▎      | 326/1000 [00:08<00:19, 34.79it/s, failures=0, objective=-0.369]
 33%|███▎      | 326/1000 [00:08<00:19, 34.79it/s, failures=0, objective=-0.369]
 33%|███▎      | 327/1000 [00:08<00:19, 34.79it/s, failures=0, objective=-0.369]
 33%|███▎      | 328/1000 [00:08<00:19, 34.79it/s, failures=0, objective=-0.369]
 33%|███▎      | 329/1000 [00:08<00:19, 34.79it/s, failures=0, objective=-0.369]
 33%|███▎      | 330/1000 [00:08<00:19, 34.79it/s, failures=0, objective=-0.369]
 33%|███▎      | 331/1000 [00:08<00:19, 34.70it/s, failures=0, objective=-0.369]
 33%|███▎      | 331/1000 [00:08<00:19, 34.70it/s, failures=0, objective=-0.369]
 33%|███▎      | 332/1000 [00:08<00:19, 34.70it/s, failures=0, objective=-0.369]
 33%|███▎      | 333/1000 [00:08<00:19, 34.70it/s, failures=0, objective=-0.369]
 33%|███▎      | 334/1000 [00:08<00:19, 34.70it/s, failures=0, objective=-0.369]
 34%|███▎      | 335/1000 [00:08<00:19, 34.70it/s, failures=0, objective=-0.369]
 34%|███▎      | 336/1000 [00:08<00:19, 34.59it/s, failures=0, objective=-0.369]
 34%|███▎      | 336/1000 [00:08<00:19, 34.59it/s, failures=0, objective=-0.369]
 34%|███▎      | 337/1000 [00:08<00:19, 34.59it/s, failures=0, objective=-0.369]
 34%|███▍      | 338/1000 [00:08<00:19, 34.59it/s, failures=0, objective=-0.369]
 34%|███▍      | 339/1000 [00:08<00:19, 34.59it/s, failures=0, objective=-0.369]
 34%|███▍      | 340/1000 [00:08<00:19, 34.59it/s, failures=0, objective=-0.369]
 34%|███▍      | 341/1000 [00:08<00:19, 34.58it/s, failures=0, objective=-0.369]
 34%|███▍      | 341/1000 [00:08<00:19, 34.58it/s, failures=0, objective=-0.369]
 34%|███▍      | 342/1000 [00:08<00:19, 34.58it/s, failures=0, objective=-0.369]
 34%|███▍      | 343/1000 [00:08<00:19, 34.58it/s, failures=0, objective=-0.369]
 34%|███▍      | 344/1000 [00:08<00:18, 34.58it/s, failures=0, objective=-0.369]
 34%|███▍      | 345/1000 [00:08<00:18, 34.58it/s, failures=0, objective=-0.369]
 35%|███▍      | 346/1000 [00:09<00:18, 34.72it/s, failures=0, objective=-0.369]
 35%|███▍      | 346/1000 [00:09<00:18, 34.72it/s, failures=0, objective=-0.369]
 35%|███▍      | 347/1000 [00:09<00:18, 34.72it/s, failures=0, objective=-0.369]
 35%|███▍      | 348/1000 [00:09<00:18, 34.72it/s, failures=0, objective=-0.369]
 35%|███▍      | 349/1000 [00:09<00:18, 34.72it/s, failures=0, objective=-0.369]
 35%|███▌      | 350/1000 [00:09<00:18, 34.72it/s, failures=0, objective=-0.369]
 35%|███▌      | 351/1000 [00:09<00:18, 34.79it/s, failures=0, objective=-0.369]
 35%|███▌      | 351/1000 [00:09<00:18, 34.79it/s, failures=0, objective=-0.369]
 35%|███▌      | 352/1000 [00:09<00:18, 34.79it/s, failures=0, objective=-0.369]
 35%|███▌      | 353/1000 [00:09<00:18, 34.79it/s, failures=0, objective=-0.369]
 35%|███▌      | 354/1000 [00:09<00:18, 34.79it/s, failures=0, objective=-0.369]
 36%|███▌      | 355/1000 [00:09<00:18, 34.79it/s, failures=0, objective=-0.369]
 36%|███▌      | 356/1000 [00:09<00:18, 34.42it/s, failures=0, objective=-0.369]
 36%|███▌      | 356/1000 [00:09<00:18, 34.42it/s, failures=0, objective=-0.369]
 36%|███▌      | 357/1000 [00:09<00:18, 34.42it/s, failures=0, objective=-0.369]
 36%|███▌      | 358/1000 [00:09<00:18, 34.42it/s, failures=0, objective=-0.369]
 36%|███▌      | 359/1000 [00:09<00:18, 34.42it/s, failures=0, objective=-0.369]
 36%|███▌      | 360/1000 [00:09<00:18, 34.42it/s, failures=0, objective=-0.369]
 36%|███▌      | 361/1000 [00:09<00:18, 34.06it/s, failures=0, objective=-0.369]
 36%|███▌      | 361/1000 [00:09<00:18, 34.06it/s, failures=0, objective=-0.369]
 36%|███▌      | 362/1000 [00:09<00:18, 34.06it/s, failures=0, objective=-0.369]
 36%|███▋      | 363/1000 [00:09<00:18, 34.06it/s, failures=0, objective=-0.369]
 36%|███▋      | 364/1000 [00:09<00:18, 34.06it/s, failures=0, objective=-0.369]
 36%|███▋      | 365/1000 [00:09<00:18, 34.06it/s, failures=0, objective=-0.369]
 37%|███▋      | 366/1000 [00:09<00:18, 34.00it/s, failures=0, objective=-0.369]
 37%|███▋      | 366/1000 [00:09<00:18, 34.00it/s, failures=0, objective=-0.369]
 37%|███▋      | 367/1000 [00:09<00:18, 34.00it/s, failures=0, objective=-0.369]
 37%|███▋      | 368/1000 [00:09<00:18, 34.00it/s, failures=0, objective=-0.369]
 37%|███▋      | 369/1000 [00:09<00:18, 34.00it/s, failures=0, objective=-0.369]
 37%|███▋      | 370/1000 [00:09<00:18, 34.00it/s, failures=0, objective=-0.369]
 37%|███▋      | 371/1000 [00:09<00:18, 34.08it/s, failures=0, objective=-0.369]
 37%|███▋      | 371/1000 [00:09<00:18, 34.08it/s, failures=0, objective=-0.369]
 37%|███▋      | 372/1000 [00:09<00:18, 34.08it/s, failures=0, objective=-0.369]
 37%|███▋      | 373/1000 [00:09<00:18, 34.08it/s, failures=0, objective=-0.369]
 37%|███▋      | 374/1000 [00:09<00:18, 34.08it/s, failures=0, objective=-0.369]
 38%|███▊      | 375/1000 [00:09<00:18, 34.08it/s, failures=0, objective=-0.369]
 38%|███▊      | 376/1000 [00:09<00:18, 34.22it/s, failures=0, objective=-0.369]
 38%|███▊      | 376/1000 [00:09<00:18, 34.22it/s, failures=0, objective=-0.369]
 38%|███▊      | 377/1000 [00:09<00:18, 34.22it/s, failures=0, objective=-0.369]
 38%|███▊      | 378/1000 [00:09<00:18, 34.22it/s, failures=0, objective=-0.369]
 38%|███▊      | 379/1000 [00:09<00:18, 34.22it/s, failures=0, objective=-0.369]
 38%|███▊      | 380/1000 [00:09<00:18, 34.22it/s, failures=0, objective=-0.369]
 38%|███▊      | 381/1000 [00:10<00:18, 34.19it/s, failures=0, objective=-0.369]
 38%|███▊      | 381/1000 [00:10<00:18, 34.19it/s, failures=0, objective=-0.369]
 38%|███▊      | 382/1000 [00:10<00:18, 34.19it/s, failures=0, objective=-0.369]
 38%|███▊      | 383/1000 [00:10<00:18, 34.19it/s, failures=0, objective=-0.369]
 38%|███▊      | 384/1000 [00:10<00:18, 34.19it/s, failures=0, objective=-0.369]
 38%|███▊      | 385/1000 [00:10<00:17, 34.19it/s, failures=0, objective=-0.369]
 39%|███▊      | 386/1000 [00:10<00:21, 28.39it/s, failures=0, objective=-0.369]
 39%|███▊      | 386/1000 [00:10<00:21, 28.39it/s, failures=0, objective=-0.369]
 39%|███▊      | 387/1000 [00:10<00:21, 28.39it/s, failures=0, objective=-0.369]
 39%|███▉      | 388/1000 [00:10<00:21, 28.39it/s, failures=0, objective=-0.369]
 39%|███▉      | 389/1000 [00:10<00:21, 28.39it/s, failures=0, objective=-0.369]
 39%|███▉      | 390/1000 [00:10<00:21, 28.39it/s, failures=0, objective=-0.369]
 39%|███▉      | 391/1000 [00:10<00:20, 29.84it/s, failures=0, objective=-0.369]
 39%|███▉      | 391/1000 [00:10<00:20, 29.84it/s, failures=0, objective=-0.369]
 39%|███▉      | 392/1000 [00:10<00:20, 29.84it/s, failures=0, objective=-0.369]
 39%|███▉      | 393/1000 [00:10<00:20, 29.84it/s, failures=0, objective=-0.369]
 39%|███▉      | 394/1000 [00:10<00:20, 29.84it/s, failures=0, objective=-0.369]
 40%|███▉      | 395/1000 [00:10<00:20, 29.84it/s, failures=0, objective=-0.369]
 40%|███▉      | 396/1000 [00:10<00:19, 30.91it/s, failures=0, objective=-0.369]
 40%|███▉      | 396/1000 [00:10<00:19, 30.91it/s, failures=0, objective=-0.369]
 40%|███▉      | 397/1000 [00:10<00:19, 30.91it/s, failures=0, objective=-0.369]
 40%|███▉      | 398/1000 [00:10<00:19, 30.91it/s, failures=0, objective=-0.369]
 40%|███▉      | 399/1000 [00:10<00:19, 30.91it/s, failures=0, objective=-0.369]
 40%|████      | 400/1000 [00:10<00:19, 30.91it/s, failures=0, objective=-0.369]
 40%|████      | 401/1000 [00:10<00:19, 31.36it/s, failures=0, objective=-0.369]
 40%|████      | 401/1000 [00:10<00:19, 31.36it/s, failures=0, objective=-0.369]
 40%|████      | 402/1000 [00:10<00:19, 31.36it/s, failures=0, objective=-0.369]
 40%|████      | 403/1000 [00:10<00:19, 31.36it/s, failures=0, objective=-0.369]
 40%|████      | 404/1000 [00:10<00:19, 31.36it/s, failures=0, objective=-0.369]
 40%|████      | 405/1000 [00:10<00:18, 31.36it/s, failures=0, objective=-0.369]
 41%|████      | 406/1000 [00:10<00:18, 32.01it/s, failures=0, objective=-0.369]
 41%|████      | 406/1000 [00:10<00:18, 32.01it/s, failures=0, objective=-0.369]
 41%|████      | 407/1000 [00:10<00:18, 32.01it/s, failures=0, objective=-0.369]
 41%|████      | 408/1000 [00:10<00:18, 32.01it/s, failures=0, objective=-0.369]
 41%|████      | 409/1000 [00:10<00:18, 32.01it/s, failures=0, objective=-0.369]
 41%|████      | 410/1000 [00:10<00:18, 32.01it/s, failures=0, objective=-0.369]
 41%|████      | 411/1000 [00:11<00:18, 32.52it/s, failures=0, objective=-0.369]
 41%|████      | 411/1000 [00:11<00:18, 32.52it/s, failures=0, objective=-0.369]
 41%|████      | 412/1000 [00:11<00:18, 32.52it/s, failures=0, objective=-0.369]
 41%|████▏     | 413/1000 [00:11<00:18, 32.52it/s, failures=0, objective=-0.369]
 41%|████▏     | 414/1000 [00:11<00:18, 32.52it/s, failures=0, objective=-0.369]
 42%|████▏     | 415/1000 [00:11<00:17, 32.52it/s, failures=0, objective=-0.369]
 42%|████▏     | 416/1000 [00:11<00:17, 32.91it/s, failures=0, objective=-0.369]
 42%|████▏     | 416/1000 [00:11<00:17, 32.91it/s, failures=0, objective=-0.369]
 42%|████▏     | 417/1000 [00:11<00:17, 32.91it/s, failures=0, objective=-0.369]
 42%|████▏     | 418/1000 [00:11<00:17, 32.91it/s, failures=0, objective=-0.369]
 42%|████▏     | 419/1000 [00:11<00:17, 32.91it/s, failures=0, objective=-0.369]
 42%|████▏     | 420/1000 [00:11<00:17, 32.91it/s, failures=0, objective=-0.369]
 42%|████▏     | 421/1000 [00:11<00:17, 33.16it/s, failures=0, objective=-0.369]
 42%|████▏     | 421/1000 [00:11<00:17, 33.16it/s, failures=0, objective=-0.369]
 42%|████▏     | 422/1000 [00:11<00:17, 33.16it/s, failures=0, objective=-0.369]
 42%|████▏     | 423/1000 [00:11<00:17, 33.16it/s, failures=0, objective=-0.369]
 42%|████▏     | 424/1000 [00:11<00:17, 33.16it/s, failures=0, objective=-0.369]
 42%|████▎     | 425/1000 [00:11<00:17, 33.16it/s, failures=0, objective=-0.369]
 43%|████▎     | 426/1000 [00:11<00:17, 33.02it/s, failures=0, objective=-0.369]
 43%|████▎     | 426/1000 [00:11<00:17, 33.02it/s, failures=0, objective=-0.369]
 43%|████▎     | 427/1000 [00:11<00:17, 33.02it/s, failures=0, objective=-0.369]
 43%|████▎     | 428/1000 [00:11<00:17, 33.02it/s, failures=0, objective=-0.369]
 43%|████▎     | 429/1000 [00:11<00:17, 33.02it/s, failures=0, objective=-0.369]
 43%|████▎     | 430/1000 [00:11<00:17, 33.02it/s, failures=0, objective=-0.369]
 43%|████▎     | 431/1000 [00:11<00:17, 32.81it/s, failures=0, objective=-0.369]
 43%|████▎     | 431/1000 [00:11<00:17, 32.81it/s, failures=0, objective=-0.369]
 43%|████▎     | 432/1000 [00:11<00:17, 32.81it/s, failures=0, objective=-0.369]
 43%|████▎     | 433/1000 [00:11<00:17, 32.81it/s, failures=0, objective=-0.369]
 43%|████▎     | 434/1000 [00:11<00:17, 32.81it/s, failures=0, objective=-0.369]
 44%|████▎     | 435/1000 [00:11<00:17, 32.81it/s, failures=0, objective=-0.369]
 44%|████▎     | 436/1000 [00:11<00:17, 33.01it/s, failures=0, objective=-0.369]
 44%|████▎     | 436/1000 [00:11<00:17, 33.01it/s, failures=0, objective=-0.369]
 44%|████▎     | 437/1000 [00:11<00:17, 33.01it/s, failures=0, objective=-0.369]
 44%|████▍     | 438/1000 [00:11<00:17, 33.01it/s, failures=0, objective=-0.369]
 44%|████▍     | 439/1000 [00:11<00:16, 33.01it/s, failures=0, objective=-0.369]
 44%|████▍     | 440/1000 [00:11<00:16, 33.01it/s, failures=0, objective=-0.369]
 44%|████▍     | 441/1000 [00:11<00:16, 33.06it/s, failures=0, objective=-0.369]
 44%|████▍     | 441/1000 [00:11<00:16, 33.06it/s, failures=0, objective=-0.369]
 44%|████▍     | 442/1000 [00:11<00:16, 33.06it/s, failures=0, objective=-0.369]
 44%|████▍     | 443/1000 [00:11<00:16, 33.06it/s, failures=0, objective=-0.369]
 44%|████▍     | 444/1000 [00:11<00:16, 33.06it/s, failures=0, objective=-0.369]
 44%|████▍     | 445/1000 [00:11<00:16, 33.06it/s, failures=0, objective=-0.369]
 45%|████▍     | 446/1000 [00:12<00:16, 33.14it/s, failures=0, objective=-0.369]
 45%|████▍     | 446/1000 [00:12<00:16, 33.14it/s, failures=0, objective=-0.369]
 45%|████▍     | 447/1000 [00:12<00:16, 33.14it/s, failures=0, objective=-0.369]
 45%|████▍     | 448/1000 [00:12<00:16, 33.14it/s, failures=0, objective=-0.369]
 45%|████▍     | 449/1000 [00:12<00:16, 33.14it/s, failures=0, objective=-0.369]
 45%|████▌     | 450/1000 [00:12<00:16, 33.14it/s, failures=0, objective=-0.369]
 45%|████▌     | 451/1000 [00:12<00:16, 33.30it/s, failures=0, objective=-0.369]
 45%|████▌     | 451/1000 [00:12<00:16, 33.30it/s, failures=0, objective=-0.369]
 45%|████▌     | 452/1000 [00:12<00:16, 33.30it/s, failures=0, objective=-0.369]
 45%|████▌     | 453/1000 [00:12<00:16, 33.30it/s, failures=0, objective=-0.369]
 45%|████▌     | 454/1000 [00:12<00:16, 33.30it/s, failures=0, objective=-0.369]
 46%|████▌     | 455/1000 [00:12<00:16, 33.30it/s, failures=0, objective=-0.369]
 46%|████▌     | 456/1000 [00:12<00:16, 33.31it/s, failures=0, objective=-0.369]
 46%|████▌     | 456/1000 [00:12<00:16, 33.31it/s, failures=0, objective=-0.369]
 46%|████▌     | 457/1000 [00:12<00:16, 33.31it/s, failures=0, objective=-0.369]
 46%|████▌     | 458/1000 [00:12<00:16, 33.31it/s, failures=0, objective=-0.369]
 46%|████▌     | 459/1000 [00:12<00:16, 33.31it/s, failures=0, objective=-0.369]
 46%|████▌     | 460/1000 [00:12<00:16, 33.31it/s, failures=0, objective=-0.369]
 46%|████▌     | 461/1000 [00:12<00:16, 33.13it/s, failures=0, objective=-0.369]
 46%|████▌     | 461/1000 [00:12<00:16, 33.13it/s, failures=0, objective=-0.369]
 46%|████▌     | 462/1000 [00:12<00:16, 33.13it/s, failures=0, objective=-0.369]
 46%|████▋     | 463/1000 [00:12<00:16, 33.13it/s, failures=0, objective=-0.369]
 46%|████▋     | 464/1000 [00:12<00:16, 33.13it/s, failures=0, objective=-0.369]
 46%|████▋     | 465/1000 [00:12<00:16, 33.13it/s, failures=0, objective=-0.369]
 47%|████▋     | 466/1000 [00:12<00:16, 33.14it/s, failures=0, objective=-0.369]
 47%|████▋     | 466/1000 [00:12<00:16, 33.14it/s, failures=0, objective=-0.369]
 47%|████▋     | 467/1000 [00:12<00:16, 33.14it/s, failures=0, objective=-0.369]
 47%|████▋     | 468/1000 [00:12<00:16, 33.14it/s, failures=0, objective=-0.369]
 47%|████▋     | 469/1000 [00:12<00:16, 33.14it/s, failures=0, objective=-0.369]
 47%|████▋     | 470/1000 [00:12<00:15, 33.14it/s, failures=0, objective=-0.369]
 47%|████▋     | 471/1000 [00:12<00:15, 33.31it/s, failures=0, objective=-0.369]
 47%|████▋     | 471/1000 [00:12<00:15, 33.31it/s, failures=0, objective=-0.369]
 47%|████▋     | 472/1000 [00:12<00:15, 33.31it/s, failures=0, objective=-0.369]
 47%|████▋     | 473/1000 [00:12<00:15, 33.31it/s, failures=0, objective=-0.369]
 47%|████▋     | 474/1000 [00:12<00:15, 33.31it/s, failures=0, objective=-0.369]
 48%|████▊     | 475/1000 [00:12<00:15, 33.31it/s, failures=0, objective=-0.369]
 48%|████▊     | 476/1000 [00:13<00:15, 33.48it/s, failures=0, objective=-0.369]
 48%|████▊     | 476/1000 [00:13<00:15, 33.48it/s, failures=0, objective=-0.369]
 48%|████▊     | 477/1000 [00:13<00:15, 33.48it/s, failures=0, objective=-0.369]
 48%|████▊     | 478/1000 [00:13<00:15, 33.48it/s, failures=0, objective=-0.369]
 48%|████▊     | 479/1000 [00:13<00:15, 33.48it/s, failures=0, objective=-0.369]
 48%|████▊     | 480/1000 [00:13<00:15, 33.48it/s, failures=0, objective=-0.369]
 48%|████▊     | 481/1000 [00:13<00:15, 33.32it/s, failures=0, objective=-0.369]
 48%|████▊     | 481/1000 [00:13<00:15, 33.32it/s, failures=0, objective=-0.369]
 48%|████▊     | 482/1000 [00:13<00:15, 33.32it/s, failures=0, objective=-0.369]
 48%|████▊     | 483/1000 [00:13<00:15, 33.32it/s, failures=0, objective=-0.369]
 48%|████▊     | 484/1000 [00:13<00:15, 33.32it/s, failures=0, objective=-0.369]
 48%|████▊     | 485/1000 [00:13<00:15, 33.32it/s, failures=0, objective=-0.369]
 49%|████▊     | 486/1000 [00:13<00:15, 33.29it/s, failures=0, objective=-0.369]
 49%|████▊     | 486/1000 [00:13<00:15, 33.29it/s, failures=0, objective=-0.369]
 49%|████▊     | 487/1000 [00:13<00:15, 33.29it/s, failures=0, objective=-0.369]
 49%|████▉     | 488/1000 [00:13<00:15, 33.29it/s, failures=0, objective=-0.369]
 49%|████▉     | 489/1000 [00:13<00:15, 33.29it/s, failures=0, objective=-0.369]
 49%|████▉     | 490/1000 [00:13<00:15, 33.29it/s, failures=0, objective=-0.369]
 49%|████▉     | 491/1000 [00:13<00:15, 33.13it/s, failures=0, objective=-0.369]
 49%|████▉     | 491/1000 [00:13<00:15, 33.13it/s, failures=0, objective=-0.369]
 49%|████▉     | 492/1000 [00:13<00:15, 33.13it/s, failures=0, objective=-0.369]
 49%|████▉     | 493/1000 [00:13<00:15, 33.13it/s, failures=0, objective=-0.369]
 49%|████▉     | 494/1000 [00:13<00:15, 33.13it/s, failures=0, objective=-0.369]
 50%|████▉     | 495/1000 [00:13<00:15, 33.13it/s, failures=0, objective=-0.369]
 50%|████▉     | 496/1000 [00:13<00:15, 33.00it/s, failures=0, objective=-0.369]
 50%|████▉     | 496/1000 [00:13<00:15, 33.00it/s, failures=0, objective=-0.369]
 50%|████▉     | 497/1000 [00:13<00:15, 33.00it/s, failures=0, objective=-0.369]
 50%|████▉     | 498/1000 [00:13<00:15, 33.00it/s, failures=0, objective=-0.369]
 50%|████▉     | 499/1000 [00:13<00:15, 33.00it/s, failures=0, objective=-0.369]
 50%|█████     | 500/1000 [00:13<00:15, 33.00it/s, failures=0, objective=-0.369]
 50%|█████     | 501/1000 [00:13<00:15, 32.92it/s, failures=0, objective=-0.369]
 50%|█████     | 501/1000 [00:13<00:15, 32.92it/s, failures=0, objective=-0.369]
 50%|█████     | 502/1000 [00:13<00:15, 32.92it/s, failures=0, objective=-0.369]
 50%|█████     | 503/1000 [00:13<00:15, 32.92it/s, failures=0, objective=-0.369]
 50%|█████     | 504/1000 [00:13<00:15, 32.92it/s, failures=0, objective=-0.369]
 50%|█████     | 505/1000 [00:13<00:15, 32.92it/s, failures=0, objective=-0.369]
 51%|█████     | 506/1000 [00:13<00:15, 32.89it/s, failures=0, objective=-0.369]
 51%|█████     | 506/1000 [00:13<00:15, 32.89it/s, failures=0, objective=-0.369]
 51%|█████     | 507/1000 [00:13<00:14, 32.89it/s, failures=0, objective=-0.369]
 51%|█████     | 508/1000 [00:13<00:14, 32.89it/s, failures=0, objective=-0.369]
 51%|█████     | 509/1000 [00:13<00:14, 32.89it/s, failures=0, objective=-0.369]
 51%|█████     | 510/1000 [00:13<00:14, 32.89it/s, failures=0, objective=-0.369]
 51%|█████     | 511/1000 [00:14<00:14, 32.90it/s, failures=0, objective=-0.369]
 51%|█████     | 511/1000 [00:14<00:14, 32.90it/s, failures=0, objective=-0.369]
 51%|█████     | 512/1000 [00:14<00:14, 32.90it/s, failures=0, objective=-0.369]
 51%|█████▏    | 513/1000 [00:14<00:14, 32.90it/s, failures=0, objective=-0.369]
 51%|█████▏    | 514/1000 [00:14<00:14, 32.90it/s, failures=0, objective=-0.369]
 52%|█████▏    | 515/1000 [00:14<00:14, 32.90it/s, failures=0, objective=-0.369]
 52%|█████▏    | 516/1000 [00:14<00:14, 32.97it/s, failures=0, objective=-0.369]
 52%|█████▏    | 516/1000 [00:14<00:14, 32.97it/s, failures=0, objective=-0.369]
 52%|█████▏    | 517/1000 [00:14<00:14, 32.97it/s, failures=0, objective=-0.369]
 52%|█████▏    | 518/1000 [00:14<00:14, 32.97it/s, failures=0, objective=-0.369]
 52%|█████▏    | 519/1000 [00:14<00:14, 32.97it/s, failures=0, objective=-0.369]
 52%|█████▏    | 520/1000 [00:14<00:14, 32.97it/s, failures=0, objective=-0.369]
 52%|█████▏    | 521/1000 [00:14<00:14, 32.98it/s, failures=0, objective=-0.369]
 52%|█████▏    | 521/1000 [00:14<00:14, 32.98it/s, failures=0, objective=-0.369]
 52%|█████▏    | 522/1000 [00:14<00:14, 32.98it/s, failures=0, objective=-0.369]
 52%|█████▏    | 523/1000 [00:14<00:14, 32.98it/s, failures=0, objective=-0.369]
 52%|█████▏    | 524/1000 [00:14<00:14, 32.98it/s, failures=0, objective=-0.369]
 52%|█████▎    | 525/1000 [00:14<00:14, 32.98it/s, failures=0, objective=-0.369]
 53%|█████▎    | 526/1000 [00:14<00:14, 32.93it/s, failures=0, objective=-0.369]
 53%|█████▎    | 526/1000 [00:14<00:14, 32.93it/s, failures=0, objective=-0.369]
 53%|█████▎    | 527/1000 [00:14<00:14, 32.93it/s, failures=0, objective=-0.369]
 53%|█████▎    | 528/1000 [00:14<00:14, 32.93it/s, failures=0, objective=-0.369]
 53%|█████▎    | 529/1000 [00:14<00:14, 32.93it/s, failures=0, objective=-0.369]
 53%|█████▎    | 530/1000 [00:14<00:14, 32.93it/s, failures=0, objective=-0.369]
 53%|█████▎    | 531/1000 [00:14<00:14, 32.93it/s, failures=0, objective=-0.369]
 53%|█████▎    | 531/1000 [00:14<00:14, 32.93it/s, failures=0, objective=-0.369]
 53%|█████▎    | 532/1000 [00:14<00:14, 32.93it/s, failures=0, objective=-0.369]
 53%|█████▎    | 533/1000 [00:14<00:14, 32.93it/s, failures=0, objective=-0.369]
 53%|█████▎    | 534/1000 [00:14<00:14, 32.93it/s, failures=0, objective=-0.369]
 54%|█████▎    | 535/1000 [00:14<00:14, 32.93it/s, failures=0, objective=-0.369]
 54%|█████▎    | 536/1000 [00:14<00:14, 32.79it/s, failures=0, objective=-0.369]
 54%|█████▎    | 536/1000 [00:14<00:14, 32.79it/s, failures=0, objective=-0.369]
 54%|█████▎    | 537/1000 [00:14<00:14, 32.79it/s, failures=0, objective=-0.369]
 54%|█████▍    | 538/1000 [00:14<00:14, 32.79it/s, failures=0, objective=-0.369]
 54%|█████▍    | 539/1000 [00:14<00:14, 32.79it/s, failures=0, objective=-0.369]
 54%|█████▍    | 540/1000 [00:14<00:14, 32.79it/s, failures=0, objective=-0.369]
 54%|█████▍    | 541/1000 [00:14<00:14, 32.75it/s, failures=0, objective=-0.369]
 54%|█████▍    | 541/1000 [00:14<00:14, 32.75it/s, failures=0, objective=-0.369]
 54%|█████▍    | 542/1000 [00:14<00:13, 32.75it/s, failures=0, objective=-0.369]
 54%|█████▍    | 543/1000 [00:14<00:13, 32.75it/s, failures=0, objective=-0.369]
 54%|█████▍    | 544/1000 [00:14<00:13, 32.75it/s, failures=0, objective=-0.369]
 55%|█████▍    | 545/1000 [00:15<00:13, 32.75it/s, failures=0, objective=-0.369]
 55%|█████▍    | 546/1000 [00:15<00:13, 32.67it/s, failures=0, objective=-0.369]
 55%|█████▍    | 546/1000 [00:15<00:13, 32.67it/s, failures=0, objective=-0.369]
 55%|█████▍    | 547/1000 [00:15<00:13, 32.67it/s, failures=0, objective=-0.369]
 55%|█████▍    | 548/1000 [00:15<00:13, 32.67it/s, failures=0, objective=-0.369]
 55%|█████▍    | 549/1000 [00:15<00:13, 32.67it/s, failures=0, objective=-0.369]
 55%|█████▌    | 550/1000 [00:15<00:13, 32.67it/s, failures=0, objective=-0.369]
 55%|█████▌    | 551/1000 [00:15<00:13, 32.71it/s, failures=0, objective=-0.369]
 55%|█████▌    | 551/1000 [00:15<00:13, 32.71it/s, failures=0, objective=-0.369]
 55%|█████▌    | 552/1000 [00:15<00:13, 32.71it/s, failures=0, objective=-0.369]
 55%|█████▌    | 553/1000 [00:15<00:13, 32.71it/s, failures=0, objective=-0.369]
 55%|█████▌    | 554/1000 [00:15<00:13, 32.71it/s, failures=0, objective=-0.369]
 56%|█████▌    | 555/1000 [00:15<00:13, 32.71it/s, failures=0, objective=-0.369]
 56%|█████▌    | 556/1000 [00:15<00:13, 32.59it/s, failures=0, objective=-0.369]
 56%|█████▌    | 556/1000 [00:15<00:13, 32.59it/s, failures=0, objective=-0.369]
 56%|█████▌    | 557/1000 [00:15<00:13, 32.59it/s, failures=0, objective=-0.369]
 56%|█████▌    | 558/1000 [00:15<00:13, 32.59it/s, failures=0, objective=-0.369]
 56%|█████▌    | 559/1000 [00:15<00:13, 32.59it/s, failures=0, objective=-0.369]
 56%|█████▌    | 560/1000 [00:15<00:13, 32.59it/s, failures=0, objective=-0.369]
 56%|█████▌    | 561/1000 [00:15<00:13, 32.74it/s, failures=0, objective=-0.369]
 56%|█████▌    | 561/1000 [00:15<00:13, 32.74it/s, failures=0, objective=-0.369]
 56%|█████▌    | 562/1000 [00:15<00:13, 32.74it/s, failures=0, objective=-0.369]
 56%|█████▋    | 563/1000 [00:15<00:13, 32.74it/s, failures=0, objective=-0.369]
 56%|█████▋    | 564/1000 [00:15<00:13, 32.74it/s, failures=0, objective=-0.369]
 56%|█████▋    | 565/1000 [00:15<00:13, 32.74it/s, failures=0, objective=-0.369]
 57%|█████▋    | 566/1000 [00:15<00:13, 32.76it/s, failures=0, objective=-0.369]
 57%|█████▋    | 566/1000 [00:15<00:13, 32.76it/s, failures=0, objective=-0.369]
 57%|█████▋    | 567/1000 [00:15<00:13, 32.76it/s, failures=0, objective=-0.369]
 57%|█████▋    | 568/1000 [00:15<00:13, 32.76it/s, failures=0, objective=-0.369]
 57%|█████▋    | 569/1000 [00:15<00:13, 32.76it/s, failures=0, objective=-0.369]
 57%|█████▋    | 570/1000 [00:15<00:13, 32.76it/s, failures=0, objective=-0.369]
 57%|█████▋    | 571/1000 [00:15<00:13, 32.73it/s, failures=0, objective=-0.369]
 57%|█████▋    | 571/1000 [00:15<00:13, 32.73it/s, failures=0, objective=-0.369]
 57%|█████▋    | 572/1000 [00:15<00:13, 32.73it/s, failures=0, objective=-0.369]
 57%|█████▋    | 573/1000 [00:15<00:13, 32.73it/s, failures=0, objective=-0.369]
 57%|█████▋    | 574/1000 [00:15<00:13, 32.73it/s, failures=0, objective=-0.369]
 57%|█████▊    | 575/1000 [00:15<00:12, 32.73it/s, failures=0, objective=-0.369]
 58%|█████▊    | 576/1000 [00:16<00:12, 32.77it/s, failures=0, objective=-0.369]
 58%|█████▊    | 576/1000 [00:16<00:12, 32.77it/s, failures=0, objective=-0.369]
 58%|█████▊    | 577/1000 [00:16<00:12, 32.77it/s, failures=0, objective=-0.369]
 58%|█████▊    | 578/1000 [00:16<00:12, 32.77it/s, failures=0, objective=-0.369]
 58%|█████▊    | 579/1000 [00:16<00:12, 32.77it/s, failures=0, objective=-0.369]
 58%|█████▊    | 580/1000 [00:16<00:12, 32.77it/s, failures=0, objective=-0.369]
 58%|█████▊    | 581/1000 [00:16<00:12, 32.62it/s, failures=0, objective=-0.369]
 58%|█████▊    | 581/1000 [00:16<00:12, 32.62it/s, failures=0, objective=-0.369]
 58%|█████▊    | 582/1000 [00:16<00:12, 32.62it/s, failures=0, objective=-0.369]
 58%|█████▊    | 583/1000 [00:16<00:12, 32.62it/s, failures=0, objective=-0.369]
 58%|█████▊    | 584/1000 [00:16<00:12, 32.62it/s, failures=0, objective=-0.369]
 58%|█████▊    | 585/1000 [00:16<00:12, 32.62it/s, failures=0, objective=-0.369]
 59%|█████▊    | 586/1000 [00:16<00:12, 32.48it/s, failures=0, objective=-0.369]
 59%|█████▊    | 586/1000 [00:16<00:12, 32.48it/s, failures=0, objective=-0.369]
 59%|█████▊    | 587/1000 [00:16<00:12, 32.48it/s, failures=0, objective=-0.369]
 59%|█████▉    | 588/1000 [00:16<00:12, 32.48it/s, failures=0, objective=-0.369]
 59%|█████▉    | 589/1000 [00:16<00:12, 32.48it/s, failures=0, objective=-0.369]
 59%|█████▉    | 590/1000 [00:16<00:12, 32.48it/s, failures=0, objective=-0.369]
 59%|█████▉    | 591/1000 [00:16<00:12, 32.33it/s, failures=0, objective=-0.369]
 59%|█████▉    | 591/1000 [00:16<00:12, 32.33it/s, failures=0, objective=-0.369]
 59%|█████▉    | 592/1000 [00:16<00:12, 32.33it/s, failures=0, objective=-0.369]
 59%|█████▉    | 593/1000 [00:16<00:12, 32.33it/s, failures=0, objective=-0.369]
 59%|█████▉    | 594/1000 [00:16<00:12, 32.33it/s, failures=0, objective=-0.369]
 60%|█████▉    | 595/1000 [00:16<00:12, 32.33it/s, failures=0, objective=-0.369]
 60%|█████▉    | 596/1000 [00:16<00:12, 32.23it/s, failures=0, objective=-0.369]
 60%|█████▉    | 596/1000 [00:16<00:12, 32.23it/s, failures=0, objective=-0.369]
 60%|█████▉    | 597/1000 [00:16<00:12, 32.23it/s, failures=0, objective=-0.369]
 60%|█████▉    | 598/1000 [00:16<00:12, 32.23it/s, failures=0, objective=-0.369]
 60%|█████▉    | 599/1000 [00:16<00:12, 32.23it/s, failures=0, objective=-0.369]
 60%|██████    | 600/1000 [00:16<00:12, 32.23it/s, failures=0, objective=-0.369]
 60%|██████    | 601/1000 [00:16<00:12, 32.10it/s, failures=0, objective=-0.369]
 60%|██████    | 601/1000 [00:16<00:12, 32.10it/s, failures=0, objective=-0.369]
 60%|██████    | 602/1000 [00:16<00:12, 32.10it/s, failures=0, objective=-0.369]
 60%|██████    | 603/1000 [00:16<00:12, 32.10it/s, failures=0, objective=-0.369]
 60%|██████    | 604/1000 [00:16<00:12, 32.10it/s, failures=0, objective=-0.369]
 60%|██████    | 605/1000 [00:16<00:12, 32.10it/s, failures=0, objective=-0.369]
 61%|██████    | 606/1000 [00:16<00:12, 32.15it/s, failures=0, objective=-0.369]
 61%|██████    | 606/1000 [00:16<00:12, 32.15it/s, failures=0, objective=-0.369]
 61%|██████    | 607/1000 [00:16<00:12, 32.15it/s, failures=0, objective=-0.369]
 61%|██████    | 608/1000 [00:17<00:12, 32.15it/s, failures=0, objective=-0.369]
 61%|██████    | 609/1000 [00:17<00:12, 32.15it/s, failures=0, objective=-0.369]
 61%|██████    | 610/1000 [00:17<00:12, 32.15it/s, failures=0, objective=-0.369]
 61%|██████    | 611/1000 [00:17<00:12, 32.02it/s, failures=0, objective=-0.369]
 61%|██████    | 611/1000 [00:17<00:12, 32.02it/s, failures=0, objective=-0.369]
 61%|██████    | 612/1000 [00:17<00:12, 32.02it/s, failures=0, objective=-0.369]
 61%|██████▏   | 613/1000 [00:17<00:12, 32.02it/s, failures=0, objective=-0.369]
 61%|██████▏   | 614/1000 [00:17<00:12, 32.02it/s, failures=0, objective=-0.369]
 62%|██████▏   | 615/1000 [00:17<00:12, 32.02it/s, failures=0, objective=-0.369]
 62%|██████▏   | 616/1000 [00:17<00:12, 31.83it/s, failures=0, objective=-0.369]
 62%|██████▏   | 616/1000 [00:17<00:12, 31.83it/s, failures=0, objective=-0.369]
 62%|██████▏   | 617/1000 [00:17<00:12, 31.83it/s, failures=0, objective=-0.369]
 62%|██████▏   | 618/1000 [00:17<00:12, 31.83it/s, failures=0, objective=-0.369]
 62%|██████▏   | 619/1000 [00:17<00:11, 31.83it/s, failures=0, objective=-0.369]
 62%|██████▏   | 620/1000 [00:17<00:11, 31.83it/s, failures=0, objective=-0.369]
 62%|██████▏   | 621/1000 [00:17<00:11, 31.83it/s, failures=0, objective=-0.369]
 62%|██████▏   | 621/1000 [00:17<00:11, 31.83it/s, failures=0, objective=-0.369]
 62%|██████▏   | 622/1000 [00:17<00:11, 31.83it/s, failures=0, objective=-0.369]
 62%|██████▏   | 623/1000 [00:17<00:11, 31.83it/s, failures=0, objective=-0.369]
 62%|██████▏   | 624/1000 [00:17<00:11, 31.83it/s, failures=0, objective=-0.369]
 62%|██████▎   | 625/1000 [00:17<00:11, 31.83it/s, failures=0, objective=-0.369]
 63%|██████▎   | 626/1000 [00:17<00:11, 31.85it/s, failures=0, objective=-0.369]
 63%|██████▎   | 626/1000 [00:17<00:11, 31.85it/s, failures=0, objective=-0.369]
 63%|██████▎   | 627/1000 [00:17<00:11, 31.85it/s, failures=0, objective=-0.369]
 63%|██████▎   | 628/1000 [00:17<00:11, 31.85it/s, failures=0, objective=-0.369]
 63%|██████▎   | 629/1000 [00:17<00:11, 31.85it/s, failures=0, objective=-0.369]
 63%|██████▎   | 630/1000 [00:17<00:11, 31.85it/s, failures=0, objective=-0.369]
 63%|██████▎   | 631/1000 [00:17<00:11, 31.89it/s, failures=0, objective=-0.369]
 63%|██████▎   | 631/1000 [00:17<00:11, 31.89it/s, failures=0, objective=-0.369]
 63%|██████▎   | 632/1000 [00:17<00:11, 31.89it/s, failures=0, objective=-0.369]
 63%|██████▎   | 633/1000 [00:17<00:11, 31.89it/s, failures=0, objective=-0.369]
 63%|██████▎   | 634/1000 [00:17<00:11, 31.89it/s, failures=0, objective=-0.369]
 64%|██████▎   | 635/1000 [00:17<00:11, 31.89it/s, failures=0, objective=-0.369]
 64%|██████▎   | 636/1000 [00:17<00:11, 31.86it/s, failures=0, objective=-0.369]
 64%|██████▎   | 636/1000 [00:17<00:11, 31.86it/s, failures=0, objective=-0.369]
 64%|██████▎   | 637/1000 [00:17<00:11, 31.86it/s, failures=0, objective=-0.369]
 64%|██████▍   | 638/1000 [00:17<00:11, 31.86it/s, failures=0, objective=-0.369]
 64%|██████▍   | 639/1000 [00:17<00:11, 31.86it/s, failures=0, objective=-0.369]
 64%|██████▍   | 640/1000 [00:17<00:11, 31.86it/s, failures=0, objective=-0.369]
 64%|██████▍   | 641/1000 [00:18<00:11, 31.83it/s, failures=0, objective=-0.369]
 64%|██████▍   | 641/1000 [00:18<00:11, 31.83it/s, failures=0, objective=-0.369]
 64%|██████▍   | 642/1000 [00:18<00:11, 31.83it/s, failures=0, objective=-0.369]
 64%|██████▍   | 643/1000 [00:18<00:11, 31.83it/s, failures=0, objective=-0.369]
 64%|██████▍   | 644/1000 [00:18<00:11, 31.83it/s, failures=0, objective=-0.369]
 64%|██████▍   | 645/1000 [00:18<00:11, 31.83it/s, failures=0, objective=-0.369]
 65%|██████▍   | 646/1000 [00:18<00:11, 31.72it/s, failures=0, objective=-0.369]
 65%|██████▍   | 646/1000 [00:18<00:11, 31.72it/s, failures=0, objective=-0.369]
 65%|██████▍   | 647/1000 [00:18<00:11, 31.72it/s, failures=0, objective=-0.369]
 65%|██████▍   | 648/1000 [00:18<00:11, 31.72it/s, failures=0, objective=-0.369]
 65%|██████▍   | 649/1000 [00:18<00:11, 31.72it/s, failures=0, objective=-0.369]
 65%|██████▌   | 650/1000 [00:18<00:11, 31.72it/s, failures=0, objective=-0.369]
 65%|██████▌   | 651/1000 [00:18<00:11, 31.58it/s, failures=0, objective=-0.369]
 65%|██████▌   | 651/1000 [00:18<00:11, 31.58it/s, failures=0, objective=-0.369]
 65%|██████▌   | 652/1000 [00:18<00:11, 31.58it/s, failures=0, objective=-0.369]
 65%|██████▌   | 653/1000 [00:18<00:10, 31.58it/s, failures=0, objective=-0.369]
 65%|██████▌   | 654/1000 [00:18<00:10, 31.58it/s, failures=0, objective=-0.369]
 66%|██████▌   | 655/1000 [00:18<00:10, 31.58it/s, failures=0, objective=-0.369]
 66%|██████▌   | 656/1000 [00:18<00:10, 31.70it/s, failures=0, objective=-0.369]
 66%|██████▌   | 656/1000 [00:18<00:10, 31.70it/s, failures=0, objective=-0.362]
 66%|██████▌   | 657/1000 [00:18<00:10, 31.70it/s, failures=0, objective=-0.362]
 66%|██████▌   | 658/1000 [00:18<00:10, 31.70it/s, failures=0, objective=-0.362]
 66%|██████▌   | 659/1000 [00:18<00:10, 31.70it/s, failures=0, objective=-0.362]
 66%|██████▌   | 660/1000 [00:18<00:10, 31.70it/s, failures=0, objective=-0.362]
 66%|██████▌   | 661/1000 [00:18<00:10, 31.49it/s, failures=0, objective=-0.362]
 66%|██████▌   | 661/1000 [00:18<00:10, 31.49it/s, failures=0, objective=-0.362]
 66%|██████▌   | 662/1000 [00:18<00:10, 31.49it/s, failures=0, objective=-0.362]
 66%|██████▋   | 663/1000 [00:18<00:10, 31.49it/s, failures=0, objective=-0.362]
 66%|██████▋   | 664/1000 [00:18<00:10, 31.49it/s, failures=0, objective=-0.362]
 66%|██████▋   | 665/1000 [00:18<00:10, 31.49it/s, failures=0, objective=-0.362]
 67%|██████▋   | 666/1000 [00:18<00:10, 31.57it/s, failures=0, objective=-0.362]
 67%|██████▋   | 666/1000 [00:18<00:10, 31.57it/s, failures=0, objective=-0.362]
 67%|██████▋   | 667/1000 [00:18<00:10, 31.57it/s, failures=0, objective=-0.362]
 67%|██████▋   | 668/1000 [00:18<00:10, 31.57it/s, failures=0, objective=-0.362]
 67%|██████▋   | 669/1000 [00:18<00:10, 31.57it/s, failures=0, objective=-0.362]
 67%|██████▋   | 670/1000 [00:18<00:10, 31.57it/s, failures=0, objective=-0.362]
 67%|██████▋   | 671/1000 [00:19<00:10, 31.50it/s, failures=0, objective=-0.362]
 67%|██████▋   | 671/1000 [00:19<00:10, 31.50it/s, failures=0, objective=-0.362]
 67%|██████▋   | 672/1000 [00:19<00:10, 31.50it/s, failures=0, objective=-0.362]
 67%|██████▋   | 673/1000 [00:19<00:10, 31.50it/s, failures=0, objective=-0.362]
 67%|██████▋   | 674/1000 [00:19<00:10, 31.50it/s, failures=0, objective=-0.362]
 68%|██████▊   | 675/1000 [00:19<00:10, 31.50it/s, failures=0, objective=-0.362]
 68%|██████▊   | 676/1000 [00:19<00:10, 31.44it/s, failures=0, objective=-0.362]
 68%|██████▊   | 676/1000 [00:19<00:10, 31.44it/s, failures=0, objective=-0.362]
 68%|██████▊   | 677/1000 [00:19<00:10, 31.44it/s, failures=0, objective=-0.362]
 68%|██████▊   | 678/1000 [00:19<00:10, 31.44it/s, failures=0, objective=-0.362]
 68%|██████▊   | 679/1000 [00:19<00:10, 31.44it/s, failures=0, objective=-0.362]
 68%|██████▊   | 680/1000 [00:19<00:10, 31.44it/s, failures=0, objective=-0.362]
 68%|██████▊   | 681/1000 [00:19<00:10, 31.57it/s, failures=0, objective=-0.362]
 68%|██████▊   | 681/1000 [00:19<00:10, 31.57it/s, failures=0, objective=-0.362]
 68%|██████▊   | 682/1000 [00:19<00:10, 31.57it/s, failures=0, objective=-0.362]
 68%|██████▊   | 683/1000 [00:19<00:10, 31.57it/s, failures=0, objective=-0.362]
 68%|██████▊   | 684/1000 [00:19<00:10, 31.57it/s, failures=0, objective=-0.362]
 68%|██████▊   | 685/1000 [00:19<00:09, 31.57it/s, failures=0, objective=-0.362]
 69%|██████▊   | 686/1000 [00:19<00:10, 31.33it/s, failures=0, objective=-0.362]
 69%|██████▊   | 686/1000 [00:19<00:10, 31.33it/s, failures=0, objective=-0.362]
 69%|██████▊   | 687/1000 [00:19<00:09, 31.33it/s, failures=0, objective=-0.362]
 69%|██████▉   | 688/1000 [00:19<00:09, 31.33it/s, failures=0, objective=-0.362]
 69%|██████▉   | 689/1000 [00:19<00:09, 31.33it/s, failures=0, objective=-0.362]
 69%|██████▉   | 690/1000 [00:19<00:09, 31.33it/s, failures=0, objective=-0.362]
 69%|██████▉   | 691/1000 [00:19<00:09, 31.31it/s, failures=0, objective=-0.362]
 69%|██████▉   | 691/1000 [00:19<00:09, 31.31it/s, failures=0, objective=-0.362]
 69%|██████▉   | 692/1000 [00:19<00:09, 31.31it/s, failures=0, objective=-0.362]
 69%|██████▉   | 693/1000 [00:19<00:09, 31.31it/s, failures=0, objective=-0.362]
 69%|██████▉   | 694/1000 [00:19<00:09, 31.31it/s, failures=0, objective=-0.362]
 70%|██████▉   | 695/1000 [00:19<00:09, 31.31it/s, failures=0, objective=-0.362]
 70%|██████▉   | 696/1000 [00:19<00:09, 31.31it/s, failures=0, objective=-0.362]
 70%|██████▉   | 696/1000 [00:19<00:09, 31.31it/s, failures=0, objective=-0.362]
 70%|██████▉   | 697/1000 [00:19<00:09, 31.31it/s, failures=0, objective=-0.362]
 70%|██████▉   | 698/1000 [00:19<00:09, 31.31it/s, failures=0, objective=-0.362]
 70%|██████▉   | 699/1000 [00:19<00:09, 31.31it/s, failures=0, objective=-0.362]
 70%|███████   | 700/1000 [00:19<00:09, 31.31it/s, failures=0, objective=-0.362]
 70%|███████   | 701/1000 [00:20<00:09, 31.26it/s, failures=0, objective=-0.362]
 70%|███████   | 701/1000 [00:20<00:09, 31.26it/s, failures=0, objective=-0.362]
 70%|███████   | 702/1000 [00:20<00:09, 31.26it/s, failures=0, objective=-0.362]
 70%|███████   | 703/1000 [00:20<00:09, 31.26it/s, failures=0, objective=-0.362]
 70%|███████   | 704/1000 [00:20<00:09, 31.26it/s, failures=0, objective=-0.362]
 70%|███████   | 705/1000 [00:20<00:09, 31.26it/s, failures=0, objective=-0.362]
 71%|███████   | 706/1000 [00:20<00:09, 31.27it/s, failures=0, objective=-0.362]
 71%|███████   | 706/1000 [00:20<00:09, 31.27it/s, failures=0, objective=-0.362]
 71%|███████   | 707/1000 [00:20<00:09, 31.27it/s, failures=0, objective=-0.362]
 71%|███████   | 708/1000 [00:20<00:09, 31.27it/s, failures=0, objective=-0.362]
 71%|███████   | 709/1000 [00:20<00:09, 31.27it/s, failures=0, objective=-0.362]
 71%|███████   | 710/1000 [00:20<00:09, 31.27it/s, failures=0, objective=-0.362]
 71%|███████   | 711/1000 [00:20<00:09, 31.37it/s, failures=0, objective=-0.362]
 71%|███████   | 711/1000 [00:20<00:09, 31.37it/s, failures=0, objective=-0.354]
 71%|███████   | 712/1000 [00:20<00:09, 31.37it/s, failures=0, objective=-0.354]
 71%|███████▏  | 713/1000 [00:20<00:09, 31.37it/s, failures=0, objective=-0.354]
 71%|███████▏  | 714/1000 [00:20<00:09, 31.37it/s, failures=0, objective=-0.354]
 72%|███████▏  | 715/1000 [00:20<00:09, 31.37it/s, failures=0, objective=-0.354]
 72%|███████▏  | 716/1000 [00:20<00:09, 30.13it/s, failures=0, objective=-0.354]
 72%|███████▏  | 716/1000 [00:20<00:09, 30.13it/s, failures=0, objective=-0.354]
 72%|███████▏  | 717/1000 [00:20<00:09, 30.13it/s, failures=0, objective=-0.354]
 72%|███████▏  | 718/1000 [00:20<00:09, 30.13it/s, failures=0, objective=-0.354]
 72%|███████▏  | 719/1000 [00:20<00:09, 30.13it/s, failures=0, objective=-0.354]
 72%|███████▏  | 720/1000 [00:20<00:09, 30.13it/s, failures=0, objective=-0.354]
 72%|███████▏  | 721/1000 [00:20<00:10, 26.12it/s, failures=0, objective=-0.354]
 72%|███████▏  | 721/1000 [00:20<00:10, 26.12it/s, failures=0, objective=-0.354]
 72%|███████▏  | 722/1000 [00:20<00:10, 26.12it/s, failures=0, objective=-0.354]
 72%|███████▏  | 723/1000 [00:20<00:10, 26.12it/s, failures=0, objective=-0.354]
 72%|███████▏  | 724/1000 [00:20<00:10, 26.12it/s, failures=0, objective=-0.354]
 72%|███████▎  | 725/1000 [00:20<00:10, 26.12it/s, failures=0, objective=-0.354]
 73%|███████▎  | 726/1000 [00:20<00:10, 27.37it/s, failures=0, objective=-0.354]
 73%|███████▎  | 726/1000 [00:20<00:10, 27.37it/s, failures=0, objective=-0.354]
 73%|███████▎  | 727/1000 [00:20<00:09, 27.37it/s, failures=0, objective=-0.354]
 73%|███████▎  | 728/1000 [00:20<00:09, 27.37it/s, failures=0, objective=-0.354]
 73%|███████▎  | 729/1000 [00:20<00:09, 27.37it/s, failures=0, objective=-0.354]
 73%|███████▎  | 730/1000 [00:20<00:09, 27.37it/s, failures=0, objective=-0.354]
 73%|███████▎  | 731/1000 [00:21<00:09, 28.34it/s, failures=0, objective=-0.354]
 73%|███████▎  | 731/1000 [00:21<00:09, 28.34it/s, failures=0, objective=-0.354]
 73%|███████▎  | 732/1000 [00:21<00:09, 28.34it/s, failures=0, objective=-0.354]
 73%|███████▎  | 733/1000 [00:21<00:09, 28.34it/s, failures=0, objective=-0.354]
 73%|███████▎  | 734/1000 [00:21<00:09, 28.34it/s, failures=0, objective=-0.354]
 74%|███████▎  | 735/1000 [00:21<00:09, 28.34it/s, failures=0, objective=-0.354]
 74%|███████▎  | 736/1000 [00:21<00:09, 29.08it/s, failures=0, objective=-0.354]
 74%|███████▎  | 736/1000 [00:21<00:09, 29.08it/s, failures=0, objective=-0.354]
 74%|███████▎  | 737/1000 [00:21<00:09, 29.08it/s, failures=0, objective=-0.354]
 74%|███████▍  | 738/1000 [00:21<00:09, 29.08it/s, failures=0, objective=-0.354]
 74%|███████▍  | 739/1000 [00:21<00:08, 29.08it/s, failures=0, objective=-0.354]
 74%|███████▍  | 740/1000 [00:21<00:08, 29.08it/s, failures=0, objective=-0.354]
 74%|███████▍  | 741/1000 [00:21<00:08, 29.57it/s, failures=0, objective=-0.354]
 74%|███████▍  | 741/1000 [00:21<00:08, 29.57it/s, failures=0, objective=-0.354]
 74%|███████▍  | 742/1000 [00:21<00:08, 29.57it/s, failures=0, objective=-0.354]
 74%|███████▍  | 743/1000 [00:21<00:08, 29.57it/s, failures=0, objective=-0.354]
 74%|███████▍  | 744/1000 [00:21<00:08, 29.57it/s, failures=0, objective=-0.354]
 74%|███████▍  | 745/1000 [00:21<00:08, 29.57it/s, failures=0, objective=-0.354]
 75%|███████▍  | 746/1000 [00:21<00:08, 29.99it/s, failures=0, objective=-0.354]
 75%|███████▍  | 746/1000 [00:21<00:08, 29.99it/s, failures=0, objective=-0.354]
 75%|███████▍  | 747/1000 [00:21<00:08, 29.99it/s, failures=0, objective=-0.354]
 75%|███████▍  | 748/1000 [00:21<00:08, 29.99it/s, failures=0, objective=-0.354]
 75%|███████▍  | 749/1000 [00:21<00:08, 29.99it/s, failures=0, objective=-0.354]
 75%|███████▌  | 750/1000 [00:21<00:08, 29.99it/s, failures=0, objective=-0.354]
 75%|███████▌  | 751/1000 [00:21<00:08, 30.20it/s, failures=0, objective=-0.354]
 75%|███████▌  | 751/1000 [00:21<00:08, 30.20it/s, failures=0, objective=-0.354]
 75%|███████▌  | 752/1000 [00:21<00:08, 30.20it/s, failures=0, objective=-0.354]
 75%|███████▌  | 753/1000 [00:21<00:08, 30.20it/s, failures=0, objective=-0.354]
 75%|███████▌  | 754/1000 [00:21<00:08, 30.20it/s, failures=0, objective=-0.354]
 76%|███████▌  | 755/1000 [00:21<00:08, 30.20it/s, failures=0, objective=-0.354]
 76%|███████▌  | 756/1000 [00:21<00:08, 30.50it/s, failures=0, objective=-0.354]
 76%|███████▌  | 756/1000 [00:21<00:08, 30.50it/s, failures=0, objective=-0.354]
 76%|███████▌  | 757/1000 [00:21<00:07, 30.50it/s, failures=0, objective=-0.354]
 76%|███████▌  | 758/1000 [00:21<00:07, 30.50it/s, failures=0, objective=-0.354]
 76%|███████▌  | 759/1000 [00:21<00:07, 30.50it/s, failures=0, objective=-0.354]
 76%|███████▌  | 760/1000 [00:21<00:07, 30.50it/s, failures=0, objective=-0.354]
 76%|███████▌  | 761/1000 [00:22<00:07, 30.53it/s, failures=0, objective=-0.354]
 76%|███████▌  | 761/1000 [00:22<00:07, 30.53it/s, failures=0, objective=-0.354]
 76%|███████▌  | 762/1000 [00:22<00:07, 30.53it/s, failures=0, objective=-0.354]
 76%|███████▋  | 763/1000 [00:22<00:07, 30.53it/s, failures=0, objective=-0.354]
 76%|███████▋  | 764/1000 [00:22<00:07, 30.53it/s, failures=0, objective=-0.354]
 76%|███████▋  | 765/1000 [00:22<00:07, 30.53it/s, failures=0, objective=-0.354]
 77%|███████▋  | 766/1000 [00:22<00:07, 30.51it/s, failures=0, objective=-0.354]
 77%|███████▋  | 766/1000 [00:22<00:07, 30.51it/s, failures=0, objective=-0.354]
 77%|███████▋  | 767/1000 [00:22<00:07, 30.51it/s, failures=0, objective=-0.354]
 77%|███████▋  | 768/1000 [00:22<00:07, 30.51it/s, failures=0, objective=-0.354]
 77%|███████▋  | 769/1000 [00:22<00:07, 30.51it/s, failures=0, objective=-0.354]
 77%|███████▋  | 770/1000 [00:22<00:07, 30.51it/s, failures=0, objective=-0.354]
 77%|███████▋  | 771/1000 [00:22<00:07, 30.51it/s, failures=0, objective=-0.354]
 77%|███████▋  | 771/1000 [00:22<00:07, 30.51it/s, failures=0, objective=-0.354]
 77%|███████▋  | 772/1000 [00:22<00:07, 30.51it/s, failures=0, objective=-0.354]
 77%|███████▋  | 773/1000 [00:22<00:07, 30.51it/s, failures=0, objective=-0.354]
 77%|███████▋  | 774/1000 [00:22<00:07, 30.51it/s, failures=0, objective=-0.354]
 78%|███████▊  | 775/1000 [00:22<00:07, 30.51it/s, failures=0, objective=-0.354]
 78%|███████▊  | 776/1000 [00:22<00:07, 30.51it/s, failures=0, objective=-0.354]
 78%|███████▊  | 776/1000 [00:22<00:07, 30.51it/s, failures=0, objective=-0.354]
 78%|███████▊  | 777/1000 [00:22<00:07, 30.51it/s, failures=0, objective=-0.354]
 78%|███████▊  | 778/1000 [00:22<00:07, 30.51it/s, failures=0, objective=-0.354]
 78%|███████▊  | 779/1000 [00:22<00:07, 30.51it/s, failures=0, objective=-0.354]
 78%|███████▊  | 780/1000 [00:22<00:07, 30.51it/s, failures=0, objective=-0.354]
 78%|███████▊  | 781/1000 [00:22<00:07, 30.47it/s, failures=0, objective=-0.354]
 78%|███████▊  | 781/1000 [00:22<00:07, 30.47it/s, failures=0, objective=-0.354]
 78%|███████▊  | 782/1000 [00:22<00:07, 30.47it/s, failures=0, objective=-0.354]
 78%|███████▊  | 783/1000 [00:22<00:07, 30.47it/s, failures=0, objective=-0.354]
 78%|███████▊  | 784/1000 [00:22<00:07, 30.47it/s, failures=0, objective=-0.354]
 78%|███████▊  | 785/1000 [00:22<00:07, 30.47it/s, failures=0, objective=-0.354]
 79%|███████▊  | 786/1000 [00:22<00:07, 30.46it/s, failures=0, objective=-0.354]
 79%|███████▊  | 786/1000 [00:22<00:07, 30.46it/s, failures=0, objective=-0.354]
 79%|███████▊  | 787/1000 [00:22<00:06, 30.46it/s, failures=0, objective=-0.354]
 79%|███████▉  | 788/1000 [00:22<00:06, 30.46it/s, failures=0, objective=-0.354]
 79%|███████▉  | 789/1000 [00:22<00:06, 30.46it/s, failures=0, objective=-0.354]
 79%|███████▉  | 790/1000 [00:22<00:06, 30.46it/s, failures=0, objective=-0.354]
 79%|███████▉  | 791/1000 [00:23<00:06, 30.43it/s, failures=0, objective=-0.354]
 79%|███████▉  | 791/1000 [00:23<00:06, 30.43it/s, failures=0, objective=-0.354]
 79%|███████▉  | 792/1000 [00:23<00:06, 30.43it/s, failures=0, objective=-0.354]
 79%|███████▉  | 793/1000 [00:23<00:06, 30.43it/s, failures=0, objective=-0.354]
 79%|███████▉  | 794/1000 [00:23<00:06, 30.43it/s, failures=0, objective=-0.354]
 80%|███████▉  | 795/1000 [00:23<00:06, 30.43it/s, failures=0, objective=-0.354]
 80%|███████▉  | 796/1000 [00:23<00:06, 30.41it/s, failures=0, objective=-0.354]
 80%|███████▉  | 796/1000 [00:23<00:06, 30.41it/s, failures=0, objective=-0.354]
 80%|███████▉  | 797/1000 [00:23<00:06, 30.41it/s, failures=0, objective=-0.354]
 80%|███████▉  | 798/1000 [00:23<00:06, 30.41it/s, failures=0, objective=-0.354]
 80%|███████▉  | 799/1000 [00:23<00:06, 30.41it/s, failures=0, objective=-0.354]
 80%|████████  | 800/1000 [00:23<00:06, 30.41it/s, failures=0, objective=-0.354]
 80%|████████  | 801/1000 [00:23<00:06, 30.47it/s, failures=0, objective=-0.354]
 80%|████████  | 801/1000 [00:23<00:06, 30.47it/s, failures=0, objective=-0.354]
 80%|████████  | 802/1000 [00:23<00:06, 30.47it/s, failures=0, objective=-0.354]
 80%|████████  | 803/1000 [00:23<00:06, 30.47it/s, failures=0, objective=-0.354]
 80%|████████  | 804/1000 [00:23<00:06, 30.47it/s, failures=0, objective=-0.354]
 80%|████████  | 805/1000 [00:23<00:06, 30.47it/s, failures=0, objective=-0.354]
 81%|████████  | 806/1000 [00:23<00:06, 30.32it/s, failures=0, objective=-0.354]
 81%|████████  | 806/1000 [00:23<00:06, 30.32it/s, failures=0, objective=-0.354]
 81%|████████  | 807/1000 [00:23<00:06, 30.32it/s, failures=0, objective=-0.354]
 81%|████████  | 808/1000 [00:23<00:06, 30.32it/s, failures=0, objective=-0.354]
 81%|████████  | 809/1000 [00:23<00:06, 30.32it/s, failures=0, objective=-0.354]
 81%|████████  | 810/1000 [00:23<00:06, 30.32it/s, failures=0, objective=-0.354]
 81%|████████  | 811/1000 [00:23<00:06, 30.00it/s, failures=0, objective=-0.354]
 81%|████████  | 811/1000 [00:23<00:06, 30.00it/s, failures=0, objective=-0.354]
 81%|████████  | 812/1000 [00:23<00:06, 30.00it/s, failures=0, objective=-0.354]
 81%|████████▏ | 813/1000 [00:23<00:06, 30.00it/s, failures=0, objective=-0.354]
 81%|████████▏ | 814/1000 [00:23<00:06, 30.00it/s, failures=0, objective=-0.354]
 82%|████████▏ | 815/1000 [00:23<00:06, 30.00it/s, failures=0, objective=-0.354]
 82%|████████▏ | 816/1000 [00:23<00:06, 29.59it/s, failures=0, objective=-0.354]
 82%|████████▏ | 816/1000 [00:23<00:06, 29.59it/s, failures=0, objective=-0.354]
 82%|████████▏ | 817/1000 [00:23<00:06, 29.59it/s, failures=0, objective=-0.354]
 82%|████████▏ | 818/1000 [00:23<00:06, 29.59it/s, failures=0, objective=-0.354]
 82%|████████▏ | 819/1000 [00:23<00:06, 29.59it/s, failures=0, objective=-0.354]
 82%|████████▏ | 820/1000 [00:23<00:06, 29.59it/s, failures=0, objective=-0.354]
 82%|████████▏ | 821/1000 [00:24<00:06, 29.52it/s, failures=0, objective=-0.354]
 82%|████████▏ | 821/1000 [00:24<00:06, 29.52it/s, failures=0, objective=-0.354]
 82%|████████▏ | 822/1000 [00:24<00:06, 29.52it/s, failures=0, objective=-0.354]
 82%|████████▏ | 823/1000 [00:24<00:05, 29.52it/s, failures=0, objective=-0.354]
 82%|████████▏ | 824/1000 [00:24<00:05, 29.52it/s, failures=0, objective=-0.354]
 82%|████████▎ | 825/1000 [00:24<00:05, 29.52it/s, failures=0, objective=-0.354]
 83%|████████▎ | 826/1000 [00:24<00:05, 29.50it/s, failures=0, objective=-0.354]
 83%|████████▎ | 826/1000 [00:24<00:05, 29.50it/s, failures=0, objective=-0.354]
 83%|████████▎ | 827/1000 [00:24<00:05, 29.50it/s, failures=0, objective=-0.354]
 83%|████████▎ | 828/1000 [00:24<00:05, 29.50it/s, failures=0, objective=-0.354]
 83%|████████▎ | 829/1000 [00:24<00:05, 29.50it/s, failures=0, objective=-0.354]
 83%|████████▎ | 830/1000 [00:24<00:05, 29.50it/s, failures=0, objective=-0.354]
 83%|████████▎ | 831/1000 [00:24<00:05, 29.50it/s, failures=0, objective=-0.354]
 83%|████████▎ | 831/1000 [00:24<00:05, 29.50it/s, failures=0, objective=-0.354]
 83%|████████▎ | 832/1000 [00:24<00:05, 29.50it/s, failures=0, objective=-0.354]
 83%|████████▎ | 833/1000 [00:24<00:05, 29.50it/s, failures=0, objective=-0.354]
 83%|████████▎ | 834/1000 [00:24<00:05, 29.50it/s, failures=0, objective=-0.354]
 84%|████████▎ | 835/1000 [00:24<00:05, 29.50it/s, failures=0, objective=-0.354]
 84%|████████▎ | 836/1000 [00:24<00:05, 29.43it/s, failures=0, objective=-0.354]
 84%|████████▎ | 836/1000 [00:24<00:05, 29.43it/s, failures=0, objective=-0.354]
 84%|████████▎ | 837/1000 [00:24<00:05, 29.43it/s, failures=0, objective=-0.354]
 84%|████████▍ | 838/1000 [00:24<00:05, 29.43it/s, failures=0, objective=-0.354]
 84%|████████▍ | 839/1000 [00:24<00:05, 29.43it/s, failures=0, objective=-0.354]
 84%|████████▍ | 840/1000 [00:24<00:05, 29.43it/s, failures=0, objective=-0.354]
 84%|████████▍ | 841/1000 [00:24<00:05, 29.28it/s, failures=0, objective=-0.354]
 84%|████████▍ | 841/1000 [00:24<00:05, 29.28it/s, failures=0, objective=-0.354]
 84%|████████▍ | 842/1000 [00:24<00:05, 29.28it/s, failures=0, objective=-0.354]
 84%|████████▍ | 843/1000 [00:24<00:05, 29.28it/s, failures=0, objective=-0.354]
 84%|████████▍ | 844/1000 [00:24<00:05, 29.28it/s, failures=0, objective=-0.354]
 84%|████████▍ | 845/1000 [00:24<00:05, 29.28it/s, failures=0, objective=-0.354]
 85%|████████▍ | 846/1000 [00:24<00:05, 29.29it/s, failures=0, objective=-0.354]
 85%|████████▍ | 846/1000 [00:24<00:05, 29.29it/s, failures=0, objective=-0.354]
 85%|████████▍ | 847/1000 [00:24<00:05, 29.29it/s, failures=0, objective=-0.354]
 85%|████████▍ | 848/1000 [00:24<00:05, 29.29it/s, failures=0, objective=-0.354]
 85%|████████▍ | 849/1000 [00:24<00:05, 29.29it/s, failures=0, objective=-0.354]
 85%|████████▌ | 850/1000 [00:24<00:05, 29.29it/s, failures=0, objective=-0.354]
 85%|████████▌ | 851/1000 [00:25<00:05, 29.30it/s, failures=0, objective=-0.354]
 85%|████████▌ | 851/1000 [00:25<00:05, 29.30it/s, failures=0, objective=-0.354]
 85%|████████▌ | 852/1000 [00:25<00:05, 29.30it/s, failures=0, objective=-0.354]
 85%|████████▌ | 853/1000 [00:25<00:05, 29.30it/s, failures=0, objective=-0.354]
 85%|████████▌ | 854/1000 [00:25<00:04, 29.30it/s, failures=0, objective=-0.354]
 86%|████████▌ | 855/1000 [00:25<00:04, 29.30it/s, failures=0, objective=-0.354]
 86%|████████▌ | 856/1000 [00:25<00:04, 29.37it/s, failures=0, objective=-0.354]
 86%|████████▌ | 856/1000 [00:25<00:04, 29.37it/s, failures=0, objective=-0.354]
 86%|████████▌ | 857/1000 [00:25<00:04, 29.37it/s, failures=0, objective=-0.354]
 86%|████████▌ | 858/1000 [00:25<00:04, 29.37it/s, failures=0, objective=-0.354]
 86%|████████▌ | 859/1000 [00:25<00:04, 29.37it/s, failures=0, objective=-0.354]
 86%|████████▌ | 860/1000 [00:25<00:04, 29.37it/s, failures=0, objective=-0.354]
 86%|████████▌ | 861/1000 [00:25<00:04, 29.44it/s, failures=0, objective=-0.354]
 86%|████████▌ | 861/1000 [00:25<00:04, 29.44it/s, failures=0, objective=-0.354]
 86%|████████▌ | 862/1000 [00:25<00:04, 29.44it/s, failures=0, objective=-0.354]
 86%|████████▋ | 863/1000 [00:25<00:04, 29.44it/s, failures=0, objective=-0.354]
 86%|████████▋ | 864/1000 [00:25<00:04, 29.44it/s, failures=0, objective=-0.354]
 86%|████████▋ | 865/1000 [00:25<00:04, 29.44it/s, failures=0, objective=-0.354]
 87%|████████▋ | 866/1000 [00:25<00:04, 29.21it/s, failures=0, objective=-0.354]
 87%|████████▋ | 866/1000 [00:25<00:04, 29.21it/s, failures=0, objective=-0.354]
 87%|████████▋ | 867/1000 [00:25<00:04, 29.21it/s, failures=0, objective=-0.354]
 87%|████████▋ | 868/1000 [00:25<00:04, 29.21it/s, failures=0, objective=-0.354]
 87%|████████▋ | 869/1000 [00:25<00:04, 29.21it/s, failures=0, objective=-0.354]
 87%|████████▋ | 870/1000 [00:25<00:04, 29.21it/s, failures=0, objective=-0.354]
 87%|████████▋ | 871/1000 [00:25<00:04, 29.16it/s, failures=0, objective=-0.354]
 87%|████████▋ | 871/1000 [00:25<00:04, 29.16it/s, failures=0, objective=-0.354]
 87%|████████▋ | 872/1000 [00:25<00:04, 29.16it/s, failures=0, objective=-0.354]
 87%|████████▋ | 873/1000 [00:25<00:04, 29.16it/s, failures=0, objective=-0.354]
 87%|████████▋ | 874/1000 [00:25<00:04, 29.16it/s, failures=0, objective=-0.354]
 88%|████████▊ | 875/1000 [00:25<00:04, 29.16it/s, failures=0, objective=-0.354]
 88%|████████▊ | 876/1000 [00:25<00:04, 29.15it/s, failures=0, objective=-0.354]
 88%|████████▊ | 876/1000 [00:25<00:04, 29.15it/s, failures=0, objective=-0.354]
 88%|████████▊ | 877/1000 [00:25<00:04, 29.15it/s, failures=0, objective=-0.354]
 88%|████████▊ | 878/1000 [00:25<00:04, 29.15it/s, failures=0, objective=-0.354]
 88%|████████▊ | 879/1000 [00:25<00:04, 29.15it/s, failures=0, objective=-0.354]
 88%|████████▊ | 880/1000 [00:25<00:04, 29.15it/s, failures=0, objective=-0.354]
 88%|████████▊ | 881/1000 [00:26<00:04, 29.03it/s, failures=0, objective=-0.354]
 88%|████████▊ | 881/1000 [00:26<00:04, 29.03it/s, failures=0, objective=-0.354]
 88%|████████▊ | 882/1000 [00:26<00:04, 29.03it/s, failures=0, objective=-0.354]
 88%|████████▊ | 883/1000 [00:26<00:04, 29.03it/s, failures=0, objective=-0.354]
 88%|████████▊ | 884/1000 [00:26<00:03, 29.03it/s, failures=0, objective=-0.354]
 88%|████████▊ | 885/1000 [00:26<00:03, 29.03it/s, failures=0, objective=-0.354]
 89%|████████▊ | 886/1000 [00:26<00:03, 28.84it/s, failures=0, objective=-0.354]
 89%|████████▊ | 886/1000 [00:26<00:03, 28.84it/s, failures=0, objective=-0.354]
 89%|████████▊ | 887/1000 [00:26<00:03, 28.84it/s, failures=0, objective=-0.354]
 89%|████████▉ | 888/1000 [00:26<00:03, 28.84it/s, failures=0, objective=-0.354]
 89%|████████▉ | 889/1000 [00:26<00:03, 28.84it/s, failures=0, objective=-0.354]
 89%|████████▉ | 890/1000 [00:26<00:03, 28.84it/s, failures=0, objective=-0.354]
 89%|████████▉ | 891/1000 [00:26<00:03, 28.85it/s, failures=0, objective=-0.354]
 89%|████████▉ | 891/1000 [00:26<00:03, 28.85it/s, failures=0, objective=-0.354]
 89%|████████▉ | 892/1000 [00:26<00:03, 28.85it/s, failures=0, objective=-0.354]
 89%|████████▉ | 893/1000 [00:26<00:03, 28.85it/s, failures=0, objective=-0.354]
 89%|████████▉ | 894/1000 [00:26<00:03, 28.85it/s, failures=0, objective=-0.354]
 90%|████████▉ | 895/1000 [00:26<00:03, 28.85it/s, failures=0, objective=-0.354]
 90%|████████▉ | 896/1000 [00:26<00:03, 28.83it/s, failures=0, objective=-0.354]
 90%|████████▉ | 896/1000 [00:26<00:03, 28.83it/s, failures=0, objective=-0.354]
 90%|████████▉ | 897/1000 [00:26<00:03, 28.83it/s, failures=0, objective=-0.354]
 90%|████████▉ | 898/1000 [00:26<00:03, 28.83it/s, failures=0, objective=-0.354]
 90%|████████▉ | 899/1000 [00:26<00:03, 28.83it/s, failures=0, objective=-0.354]
 90%|█████████ | 900/1000 [00:26<00:03, 28.83it/s, failures=0, objective=-0.354]
 90%|█████████ | 901/1000 [00:26<00:03, 28.72it/s, failures=0, objective=-0.354]
 90%|█████████ | 901/1000 [00:26<00:03, 28.72it/s, failures=0, objective=-0.354]
 90%|█████████ | 902/1000 [00:26<00:03, 28.72it/s, failures=0, objective=-0.354]
 90%|█████████ | 903/1000 [00:26<00:03, 28.72it/s, failures=0, objective=-0.354]
 90%|█████████ | 904/1000 [00:26<00:03, 28.72it/s, failures=0, objective=-0.354]
 90%|█████████ | 905/1000 [00:26<00:03, 28.72it/s, failures=0, objective=-0.354]
 91%|█████████ | 906/1000 [00:26<00:03, 28.76it/s, failures=0, objective=-0.354]
 91%|█████████ | 906/1000 [00:26<00:03, 28.76it/s, failures=0, objective=-0.354]
 91%|█████████ | 907/1000 [00:26<00:03, 28.76it/s, failures=0, objective=-0.354]
 91%|█████████ | 908/1000 [00:26<00:03, 28.76it/s, failures=0, objective=-0.354]
 91%|█████████ | 909/1000 [00:26<00:03, 28.76it/s, failures=0, objective=-0.354]
 91%|█████████ | 910/1000 [00:26<00:03, 28.76it/s, failures=0, objective=-0.354]
 91%|█████████ | 911/1000 [00:27<00:03, 28.84it/s, failures=0, objective=-0.354]
 91%|█████████ | 911/1000 [00:27<00:03, 28.84it/s, failures=0, objective=-0.354]
 91%|█████████ | 912/1000 [00:27<00:03, 28.84it/s, failures=0, objective=-0.354]
 91%|█████████▏| 913/1000 [00:27<00:03, 28.84it/s, failures=0, objective=-0.354]
 91%|█████████▏| 914/1000 [00:27<00:02, 28.84it/s, failures=0, objective=-0.354]
 92%|█████████▏| 915/1000 [00:27<00:02, 28.84it/s, failures=0, objective=-0.354]
 92%|█████████▏| 916/1000 [00:27<00:02, 28.69it/s, failures=0, objective=-0.354]
 92%|█████████▏| 916/1000 [00:27<00:02, 28.69it/s, failures=0, objective=-0.354]
 92%|█████████▏| 917/1000 [00:27<00:02, 28.69it/s, failures=0, objective=-0.354]
 92%|█████████▏| 918/1000 [00:27<00:02, 28.69it/s, failures=0, objective=-0.354]
 92%|█████████▏| 919/1000 [00:27<00:02, 28.69it/s, failures=0, objective=-0.354]
 92%|█████████▏| 920/1000 [00:27<00:02, 28.69it/s, failures=0, objective=-0.354]
 92%|█████████▏| 921/1000 [00:27<00:02, 28.51it/s, failures=0, objective=-0.354]
 92%|█████████▏| 921/1000 [00:27<00:02, 28.51it/s, failures=0, objective=-0.354]
 92%|█████████▏| 922/1000 [00:27<00:02, 28.51it/s, failures=0, objective=-0.354]
 92%|█████████▏| 923/1000 [00:27<00:02, 28.51it/s, failures=0, objective=-0.354]
 92%|█████████▏| 924/1000 [00:27<00:02, 28.51it/s, failures=0, objective=-0.354]
 92%|█████████▎| 925/1000 [00:27<00:02, 28.51it/s, failures=0, objective=-0.354]
 93%|█████████▎| 926/1000 [00:27<00:02, 28.53it/s, failures=0, objective=-0.354]
 93%|█████████▎| 926/1000 [00:27<00:02, 28.53it/s, failures=0, objective=-0.354]
 93%|█████████▎| 927/1000 [00:27<00:02, 28.53it/s, failures=0, objective=-0.354]
 93%|█████████▎| 928/1000 [00:27<00:02, 28.53it/s, failures=0, objective=-0.354]
 93%|█████████▎| 929/1000 [00:27<00:02, 28.53it/s, failures=0, objective=-0.354]
 93%|█████████▎| 930/1000 [00:27<00:02, 28.53it/s, failures=0, objective=-0.354]
 93%|█████████▎| 931/1000 [00:27<00:02, 28.64it/s, failures=0, objective=-0.354]
 93%|█████████▎| 931/1000 [00:27<00:02, 28.64it/s, failures=0, objective=-0.354]
 93%|█████████▎| 932/1000 [00:27<00:02, 28.64it/s, failures=0, objective=-0.354]
 93%|█████████▎| 933/1000 [00:27<00:02, 28.64it/s, failures=0, objective=-0.354]
 93%|█████████▎| 934/1000 [00:27<00:02, 28.64it/s, failures=0, objective=-0.354]
 94%|█████████▎| 935/1000 [00:27<00:02, 28.64it/s, failures=0, objective=-0.354]
 94%|█████████▎| 936/1000 [00:28<00:02, 28.58it/s, failures=0, objective=-0.354]
 94%|█████████▎| 936/1000 [00:28<00:02, 28.58it/s, failures=0, objective=-0.354]
 94%|█████████▎| 937/1000 [00:28<00:02, 28.58it/s, failures=0, objective=-0.354]
 94%|█████████▍| 938/1000 [00:28<00:02, 28.58it/s, failures=0, objective=-0.354]
 94%|█████████▍| 939/1000 [00:28<00:02, 28.58it/s, failures=0, objective=-0.354]
 94%|█████████▍| 940/1000 [00:28<00:02, 28.58it/s, failures=0, objective=-0.354]
 94%|█████████▍| 941/1000 [00:28<00:02, 28.57it/s, failures=0, objective=-0.354]
 94%|█████████▍| 941/1000 [00:28<00:02, 28.57it/s, failures=0, objective=-0.354]
 94%|█████████▍| 942/1000 [00:28<00:02, 28.57it/s, failures=0, objective=-0.354]
 94%|█████████▍| 943/1000 [00:28<00:01, 28.57it/s, failures=0, objective=-0.354]
 94%|█████████▍| 944/1000 [00:28<00:01, 28.57it/s, failures=0, objective=-0.354]
 94%|█████████▍| 945/1000 [00:28<00:01, 28.57it/s, failures=0, objective=-0.354]
 95%|█████████▍| 946/1000 [00:28<00:01, 28.50it/s, failures=0, objective=-0.354]
 95%|█████████▍| 946/1000 [00:28<00:01, 28.50it/s, failures=0, objective=-0.354]
 95%|█████████▍| 947/1000 [00:28<00:01, 28.50it/s, failures=0, objective=-0.354]
 95%|█████████▍| 948/1000 [00:28<00:01, 28.50it/s, failures=0, objective=-0.354]
 95%|█████████▍| 949/1000 [00:28<00:01, 28.50it/s, failures=0, objective=-0.354]
 95%|█████████▌| 950/1000 [00:28<00:01, 28.50it/s, failures=0, objective=-0.354]
 95%|█████████▌| 951/1000 [00:28<00:01, 28.22it/s, failures=0, objective=-0.354]
 95%|█████████▌| 951/1000 [00:28<00:01, 28.22it/s, failures=0, objective=-0.354]
 95%|█████████▌| 952/1000 [00:28<00:01, 28.22it/s, failures=0, objective=-0.354]
 95%|█████████▌| 953/1000 [00:28<00:01, 28.22it/s, failures=0, objective=-0.354]
 95%|█████████▌| 954/1000 [00:28<00:01, 28.22it/s, failures=0, objective=-0.354]
 96%|█████████▌| 955/1000 [00:28<00:01, 28.22it/s, failures=0, objective=-0.354]
 96%|█████████▌| 956/1000 [00:28<00:01, 28.06it/s, failures=0, objective=-0.354]
 96%|█████████▌| 956/1000 [00:28<00:01, 28.06it/s, failures=0, objective=-0.354]
 96%|█████████▌| 957/1000 [00:28<00:01, 28.06it/s, failures=0, objective=-0.354]
 96%|█████████▌| 958/1000 [00:28<00:01, 28.06it/s, failures=0, objective=-0.354]
 96%|█████████▌| 959/1000 [00:28<00:01, 28.06it/s, failures=0, objective=-0.354]
 96%|█████████▌| 960/1000 [00:28<00:01, 28.06it/s, failures=0, objective=-0.354]
 96%|█████████▌| 961/1000 [00:28<00:01, 27.99it/s, failures=0, objective=-0.354]
 96%|█████████▌| 961/1000 [00:28<00:01, 27.99it/s, failures=0, objective=-0.354]
 96%|█████████▌| 962/1000 [00:28<00:01, 27.99it/s, failures=0, objective=-0.354]
 96%|█████████▋| 963/1000 [00:28<00:01, 27.99it/s, failures=0, objective=-0.354]
 96%|█████████▋| 964/1000 [00:28<00:01, 27.99it/s, failures=0, objective=-0.354]
 96%|█████████▋| 965/1000 [00:28<00:01, 27.99it/s, failures=0, objective=-0.354]
 97%|█████████▋| 966/1000 [00:29<00:01, 28.10it/s, failures=0, objective=-0.354]
 97%|█████████▋| 966/1000 [00:29<00:01, 28.10it/s, failures=0, objective=-0.354]
 97%|█████████▋| 967/1000 [00:29<00:01, 28.10it/s, failures=0, objective=-0.354]
 97%|█████████▋| 968/1000 [00:29<00:01, 28.10it/s, failures=0, objective=-0.354]
 97%|█████████▋| 969/1000 [00:29<00:01, 28.10it/s, failures=0, objective=-0.354]
 97%|█████████▋| 970/1000 [00:29<00:01, 28.10it/s, failures=0, objective=-0.354]
 97%|█████████▋| 971/1000 [00:29<00:01, 27.98it/s, failures=0, objective=-0.354]
 97%|█████████▋| 971/1000 [00:29<00:01, 27.98it/s, failures=0, objective=-0.354]
 97%|█████████▋| 972/1000 [00:29<00:01, 27.98it/s, failures=0, objective=-0.354]
 97%|█████████▋| 973/1000 [00:29<00:00, 27.98it/s, failures=0, objective=-0.354]
 97%|█████████▋| 974/1000 [00:29<00:00, 27.98it/s, failures=0, objective=-0.354]
 98%|█████████▊| 975/1000 [00:29<00:00, 27.98it/s, failures=0, objective=-0.354]
 98%|█████████▊| 976/1000 [00:29<00:00, 28.07it/s, failures=0, objective=-0.354]
 98%|█████████▊| 976/1000 [00:29<00:00, 28.07it/s, failures=0, objective=-0.354]
 98%|█████████▊| 977/1000 [00:29<00:00, 28.07it/s, failures=0, objective=-0.354]
 98%|█████████▊| 978/1000 [00:29<00:00, 28.07it/s, failures=0, objective=-0.354]
 98%|█████████▊| 979/1000 [00:29<00:00, 28.07it/s, failures=0, objective=-0.354]
 98%|█████████▊| 980/1000 [00:29<00:00, 28.07it/s, failures=0, objective=-0.354]
 98%|█████████▊| 981/1000 [00:29<00:00, 28.03it/s, failures=0, objective=-0.354]
 98%|█████████▊| 981/1000 [00:29<00:00, 28.03it/s, failures=0, objective=-0.354]
 98%|█████████▊| 982/1000 [00:29<00:00, 28.03it/s, failures=0, objective=-0.354]
 98%|█████████▊| 983/1000 [00:29<00:00, 28.03it/s, failures=0, objective=-0.354]
 98%|█████████▊| 984/1000 [00:29<00:00, 28.03it/s, failures=0, objective=-0.354]
 98%|█████████▊| 985/1000 [00:29<00:00, 28.03it/s, failures=0, objective=-0.354]
 99%|█████████▊| 986/1000 [00:29<00:00, 28.18it/s, failures=0, objective=-0.354]
 99%|█████████▊| 986/1000 [00:29<00:00, 28.18it/s, failures=0, objective=-0.354]
 99%|█████████▊| 987/1000 [00:29<00:00, 28.18it/s, failures=0, objective=-0.354]
 99%|█████████▉| 988/1000 [00:29<00:00, 28.18it/s, failures=0, objective=-0.354]
 99%|█████████▉| 989/1000 [00:29<00:00, 28.18it/s, failures=0, objective=-0.354]
 99%|█████████▉| 990/1000 [00:29<00:00, 28.18it/s, failures=0, objective=-0.354]
 99%|█████████▉| 991/1000 [00:29<00:00, 28.26it/s, failures=0, objective=-0.354]
 99%|█████████▉| 991/1000 [00:29<00:00, 28.26it/s, failures=0, objective=-0.354]
 99%|█████████▉| 992/1000 [00:29<00:00, 28.26it/s, failures=0, objective=-0.354]
 99%|█████████▉| 993/1000 [00:29<00:00, 28.26it/s, failures=0, objective=-0.354]
 99%|█████████▉| 994/1000 [00:29<00:00, 28.26it/s, failures=0, objective=-0.354]
100%|█████████▉| 995/1000 [00:29<00:00, 28.26it/s, failures=0, objective=-0.354]
100%|█████████▉| 996/1000 [00:30<00:00, 28.26it/s, failures=0, objective=-0.354]
100%|█████████▉| 996/1000 [00:30<00:00, 28.26it/s, failures=0, objective=-0.354]
100%|█████████▉| 997/1000 [00:30<00:00, 28.26it/s, failures=0, objective=-0.354]
100%|█████████▉| 998/1000 [00:30<00:00, 28.26it/s, failures=0, objective=-0.354]
100%|█████████▉| 999/1000 [00:30<00:00, 28.26it/s, failures=0, objective=-0.354]
100%|██████████| 1000/1000 [00:30<00:00, 28.26it/s, failures=0, objective=-0.354]
1001it [00:30, 31.97it/s, failures=0, objective=-0.354]
1001it [00:30, 31.97it/s, failures=0, objective=-0.354]
1002it [00:30, 31.97it/s, failures=0, objective=-0.354]
1003it [00:30, 31.97it/s, failures=0, objective=-0.354]
1004it [00:30, 31.97it/s, failures=0, objective=-0.354]
1005it [00:30, 31.97it/s, failures=0, objective=-0.354]

Analysis of the results

The results of the HPO is a dataframe. The columns starting with p: are the hyperparameters. The columns starting with m: are the metadata. There are also special columns: objective, job_id and job_status.

results

	p:criterion	p:max_depth	p:min_samples_leaf	p:min_samples_split	p:min_weight_fraction_leaf	p:splitter	objective	job_id	job_status	m:timestamp_submit	m:test_cce	m:test_acc	m:timestamp_gather
0	entropy	13	18	17	0.358516	best	-0.525434	6	DONE	0.936074	0.540779	0.763636	2.604752
1	entropy	473	1	15	0.399777	random	-0.693145	1	DONE	0.929979	0.693145	0.503030	2.646949
2	log_loss	535	5	5	0.451926	random	-0.693145	5	DONE	0.934895	0.693145	0.503030	2.653171
3	entropy	122	9	20	0.439568	random	-0.693145	10	DONE	2.638798	0.693145	0.503030	2.669706
4	log_loss	584	6	8	0.219015	random	-0.500981	2	DONE	0.931257	0.547766	0.766667	2.674750
...	...	...	...	...	...	...	...	...	...	...	...	...	...
1000	gini	11	18	12	0.264603	best	-0.477218	1000	DONE	32.770154	0.521591	0.763636	32.889597
1001	gini	13	17	12	0.263007	best	-0.477218	1001	DONE	32.771561	0.521591	0.763636	32.892539
1002	gini	13	17	12	0.263007	best	-0.477218	1002	DONE	32.772853	0.521591	0.763636	32.895129
1003	gini	13	17	12	0.263007	best	-0.477218	1004	DONE	32.775232	0.521591	0.763636	32.897887
1004	gini	13	17	12	0.263007	best	-0.477218	1003	DONE	32.774098	0.521591	0.763636	32.900907

1005 rows × 13 columns

Evolution of the objective

We use deephyper.analysis.hpo.plot_search_trajectory_single_objective_hpo() to look at the evolution of the objective during the search.

Code (Plot search trajectory)

from deephyper.analysis.hpo import plot_search_trajectory_single_objective_hpo


_, ax = plt.subplots(figsize=(WIDTH_PLOTS, HEIGHT_PLOTS), tight_layout=True)
_ = plot_search_trajectory_single_objective_hpo(results, mode="min", ax=ax)
ax.axhline(-baseline_output["objective"], linestyle="--", color="red", label="baseline")
ax.set_yscale("log")

The dashed red horizontal line corresponds to the baseline performance.

Worker utilization

We use deephyper.analysis.hpo.plot_worker_utilization() to look at the number of active workers over the search.

Code (Plot worker utilization)

from deephyper.analysis.hpo import plot_worker_utilization

_, ax = plt.subplots(figsize=(WIDTH_PLOTS, HEIGHT_PLOTS), tight_layout=True)
_ = plot_worker_utilization(results, ax=ax)

Best decision tree

Then, we look indivudualy at the performance of the top 5 models by using deephyper.analysis.hpo.parameters_from_row():

from deephyper.analysis.hpo import parameters_from_row


topk_rows = results.nlargest(5, "objective").reset_index(drop=True)

for i, row in topk_rows.iterrows():
    parameters = parameters_from_row(row)
    obj = row["objective"]
    print(f"Top-{i+1} -> {obj=:.3f}: {parameters}")
    print()

Top-1 -> obj=-0.354: {'criterion': 'gini', 'max_depth': 11, 'min_samples_leaf': 19, 'min_samples_split': 11, 'min_weight_fraction_leaf': 0.0433992433267581, 'splitter': 'random'}

Top-2 -> obj=-0.359: {'criterion': 'gini', 'max_depth': 18, 'min_samples_leaf': 20, 'min_samples_split': 13, 'min_weight_fraction_leaf': 0.0378149227185198, 'splitter': 'random'}

Top-3 -> obj=-0.362: {'criterion': 'entropy', 'max_depth': 18, 'min_samples_leaf': 2, 'min_samples_split': 2, 'min_weight_fraction_leaf': 0.1026436806137768, 'splitter': 'random'}

Top-4 -> obj=-0.369: {'criterion': 'entropy', 'max_depth': 152, 'min_samples_leaf': 1, 'min_samples_split': 2, 'min_weight_fraction_leaf': 0.0511377033250129, 'splitter': 'random'}

Top-5 -> obj=-0.370: {'criterion': 'entropy', 'max_depth': 28, 'min_samples_leaf': 2, 'min_samples_split': 4, 'min_weight_fraction_leaf': 0.0996880486124737, 'splitter': 'random'}

If we just plot the decision boundary and calibration plots of the best model we can observe a significant improvement over the baseline with log-loss values around 0.338 when it was previously around 6.

best_job = topk_rows.iloc[0]
hpo_dir = "hpo_sklearn_classification"
model_checkpoint_dir = os.path.join(hpo_dir, "models")
with open(os.path.join(model_checkpoint_dir, f"model_0.{best_job.job_id}.pkl"), "rb") as f:
    best_model = pickle.load(f)

Code (Plot decision boundary and calibration)

fig, axes = plt.subplots(nrows=2, ncols=1, figsize=(WIDTH_PLOTS, HEIGHT_PLOTS*2), tight_layout=True)
plot_decision_boundary_decision_tree(tx, ty, best_model, steps=1000, color_map="viridis", ax=axes[0])
disp = CalibrationDisplay.from_predictions(ty, best_model.predict_proba(tx)[:, 1], ax=axes[1])

1005it [00:30, 32.77it/s, failures=0, objective=-0.354]

Ensemble of decision trees

We now move to ensembling checkpointed models and we start by importing utilities from deephyper.ensemble and deephyper.predictor.

from deephyper.ensemble import EnsemblePredictor
from deephyper.ensemble.aggregator import MixedCategoricalAggregator
from deephyper.ensemble.loss import CategoricalCrossEntropy
from deephyper.ensemble.selector import GreedySelector, TopKSelector
from deephyper.predictor.sklearn import SklearnPredictorFileLoader

Code (Plot decision boundary and uncertainty)

def plot_decision_boundary_and_uncertainty(
    dataset, labels, model, steps=1000, color_map="viridis", s=5
):

    fig, axs = plt.subplots(
        3, sharex="all", sharey="all", figsize=(WIDTH_PLOTS, HEIGHT_PLOTS * 2), tight_layout=True,
    )

    # Define region of interest by data limits
    xmin, xmax = dataset[:, 0].min() - 1, dataset[:, 0].max() + 1
    ymin, ymax = dataset[:, 1].min() - 1, dataset[:, 1].max() + 1
    x_span = np.linspace(xmin, xmax, steps)
    y_span = np.linspace(ymin, ymax, steps)
    xx, yy = np.meshgrid(x_span, y_span)

    # Make predictions across region of interest
    y_pred = model.predict(np.c_[xx.ravel(), yy.ravel()].astype(np.float32))
    y_pred_proba = y_pred["loc"]
    y_pred_aleatoric = y_pred["uncertainty_aleatoric"]
    y_pred_epistemic = y_pred["uncertainty_epistemic"]

    # Plot decision boundary in region of interest

    # 1. MODE
    color_map = plt.get_cmap("viridis")
    z = y_pred_proba[:, 1].reshape(xx.shape)

    cont = axs[0].contourf(xx, yy, z, cmap=color_map, vmin=0, vmax=1, alpha=0.5)

    # Get predicted labels on training data and plot
    axs[0].scatter(
        dataset[:, 0],
        dataset[:, 1],
        c=labels,
        cmap=color_map,
        s=s,
        lw=0,
    )
    plt.colorbar(cont, ax=axs[0], label="Probability of class 1")

    # 2. ALEATORIC
    color_map = plt.get_cmap("plasma")
    z = y_pred_aleatoric.reshape(xx.shape)

    cont = axs[1].contourf(xx, yy, z, cmap=color_map, vmin=0, vmax=0.69, alpha=0.5)

    # Get predicted labels on training data and plot
    axs[1].scatter(
        dataset[:, 0],
        dataset[:, 1],
        c=labels,
        cmap=color_map,
        s=s,
        lw=0,
    )
    plt.colorbar(cont, ax=axs[1], label="Aleatoric uncertainty")

    # 3. EPISTEMIC
    z = y_pred_epistemic.reshape(xx.shape)

    cont = axs[2].contourf(xx, yy, z, cmap=color_map, vmin=0, vmax=0.69, alpha=0.5)

    # Get predicted labels on training data and plot
    axs[2].scatter(
        dataset[:, 0],
        dataset[:, 1],
        c=labels,
        cmap=color_map,
        s=s,
        lw=0,
    )
    plt.colorbar(cont, ax=axs[2], label="Epistemic uncertainty")

We define a function that will create an ensemble with TopK or Greedy selection strategies. This function also has a parameter k that sets the number of unique member in the ensemble.

def create_ensemble_from_checkpoints(ensemble_selector: str = "topk", k=50):

    # 0. Load data
    _, (vx, vy), _ = load_data()

    # !1.3 SKLEARN EXAMPLE
    predictor_files = SklearnPredictorFileLoader.find_predictor_files(
        model_checkpoint_dir
    )
    predictor_loaders = [SklearnPredictorFileLoader(f) for f in predictor_files]
    predictors = [p.load() for p in predictor_loaders]

    # 2. Build an ensemble
    ensemble = EnsemblePredictor(
        predictors=predictors,
        aggregator=MixedCategoricalAggregator(
            uncertainty_method="entropy",
            decomposed_uncertainty=True,
        ),
        # You can specify parallel backends for the evaluation of the ensemble
        evaluator={
            "method": "ray",
            "method_kwargs": {"num_cpus_per_task": 1},
        },
    )
    y_predictors = ensemble.predictions_from_predictors(
        vx, predictors=ensemble.predictors
    )

    # Use TopK or Greedy/Caruana
    if ensemble_selector == "topk":
        selector = TopKSelector(
            loss_func=CategoricalCrossEntropy(),
            k=k,
        )
    elif ensemble_selector == "greedy":
        selector = GreedySelector(
            loss_func=CategoricalCrossEntropy(),
            aggregator=MixedCategoricalAggregator(),
            k=k,
            k_init=5,
            max_it=100,
            early_stopping=False,
            bagging=True,
            eps_tol=1e-5,
        )
    else:
        raise ValueError(f"Unknown ensemble_selector: {ensemble_selector}")

    selected_predictors_indexes, selected_predictors_weights = selector.select(
        vy, y_predictors
    )
    print(f"{selected_predictors_indexes=}")
    print(f"{selected_predictors_weights=}")

    ensemble.predictors = [ensemble.predictors[i] for i in selected_predictors_indexes]
    ensemble.weights = selected_predictors_weights

    return ensemble

We start by testing the Topk strategy.

ensemble = create_ensemble_from_checkpoints("topk")
ty_pred = ensemble.predict(tx)["loc"]
cce = log_loss(ty, ty_pred)
acc = accuracy_score(ty, np.argmax(ty_pred, axis=1))
print(f"Test scores: {cce=:.3f}, {acc=:.3f}")

selected_predictors_indexes=[680, 923, 413, 483, 487, 915, 69, 800, 333, 601, 903, 42, 897, 445, 105, 977, 633, 421, 63, 891, 300, 257, 535, 357, 807, 683, 737, 909, 401, 185, 72, 641, 12, 349, 1001, 227, 926, 39, 693, 805, 146, 381, 684, 972, 36, 392, 324, 186, 802, 477]
selected_predictors_weights=[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]
Test scores: cce=0.409, acc=0.870

Code (Plot decision boundary and uncertainty for ensemble)

plot_decision_boundary_and_uncertainty(tx, ty, ensemble, steps=1000, color_map="viridis")

Code (Plot calibration for ensemble)

fig, ax = plt.subplots(figsize=(WIDTH_PLOTS, HEIGHT_PLOTS), tight_layout=True)
disp = CalibrationDisplay.from_predictions(ty, ty_pred[:, 1], ax=ax)

We do the same for the Greedy strategy.

ensemble = create_ensemble_from_checkpoints("greedy")
ty_pred = ensemble.predict(tx)["loc"]
cce = log_loss(ty, ty_pred)
acc = accuracy_score(ty, np.argmax(ty_pred, axis=1))
print(f"Test scores: {cce=:.3f}, {acc=:.3f}")

selected_predictors_indexes=[34, 58, 65, 147, 182, 225, 413, 434, 483, 487, 563, 576, 640, 680, 879, 882, 903, 921, 923, 986]
selected_predictors_weights=[0.05714285714285714, 0.0380952380952381, 0.3333333333333333, 0.01904761904761905, 0.02857142857142857, 0.01904761904761905, 0.009523809523809525, 0.01904761904761905, 0.047619047619047616, 0.009523809523809525, 0.01904761904761905, 0.01904761904761905, 0.009523809523809525, 0.009523809523809525, 0.0380952380952381, 0.05714285714285714, 0.12380952380952381, 0.12380952380952381, 0.009523809523809525, 0.009523809523809525]
Test scores: cce=0.366, acc=0.879

Code (Plot decision boundary and uncertainty for ensemble)

plot_decision_boundary_and_uncertainty(tx, ty, ensemble, steps=1000, color_map="viridis")

Code (Plot calibration for ensemble)

fig, ax = plt.subplots(figsize=(WIDTH_PLOTS, HEIGHT_PLOTS), tight_layout=True)
disp = CalibrationDisplay.from_predictions(ty, ty_pred[:, 1], ax=ax)

In conclusion, the improvement over the default hyperparameters is significant.

For CCE, we improved from about 6 to 0.4.

For Accuracy, we improved from 0.82 to 0.87.

Not only that we have disentangled uncertainty estimates. The epistemic uncertainty is informative of locations where we are missing data and the aleatoric uncertainty is informative of the noise level in the labels.