.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "examples/examples_bbo/plot_notify_failures_hpo.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note :ref:`Go to the end ` to download the full example code. .. rst-class:: sphx-glr-example-title .. _sphx_glr_examples_examples_bbo_plot_notify_failures_hpo.py: Learn to Avoid Failures with Bayesian Optimization ================================================== **Author(s)**: Romain Egele. In this example, you will learn how to handle failures in black-box optimization. In many application of black-box optimization such as software auto-tuning (where we minimize the run-time of a software application) some configurations can create run-time errors and therefore no scalar objective is returned. A default choice could be to return in this case the worst case objective if known. Other possibilites are to ignore these configurations or to replace them with the running average or minimum objective. .. GENERATED FROM PYTHON SOURCE LINES 17-30 .. dropdown:: Code (Import statements) .. code-block:: Python import matplotlib.pyplot as plt import numpy as np from deephyper.hpo import HpProblem from deephyper.hpo import CBO from deephyper.evaluator import Evaluator from deephyper.evaluator.callback import TqdmCallback WIDTH_PLOTS = 8 HEIGHT_PLOTS = WIDTH_PLOTS / 1.618 .. GENERATED FROM PYTHON SOURCE LINES 31-34 To illustrate such a use-case we define a ``run``-function that will fail when its input parameter ``p_failure`` is greater than 0.5. To notify deephyper about a failure, we return a "string" value with ``"F"`` as prefix such as: .. GENERATED FROM PYTHON SOURCE LINES 34-46 .. code-block:: Python def run(job) -> float: if job.parameters["p_failure"] > 0.5: try: raise ValueError("Some example exception") except ValueError: # Notify the search about the failure return "F_value_error" else: # Returns a regular objective value that is maximized return sum(job.parameters[k] for k in job.parameters if "x" in k) .. GENERATED FROM PYTHON SOURCE LINES 47-49 Then, we define the corresponding hyperparameter problem where ``x{i}`` are the value to maximize and ``p_parameter`` is a value that impact the appearance of failures. .. GENERATED FROM PYTHON SOURCE LINES 49-55 .. code-block:: Python problem = HpProblem() problem.add_hyperparameter((0.0, 1.0), "p_failure") for i in range(10): problem.add_hyperparameter((0.0, 1.0), f"x{i}") problem .. rst-class:: sphx-glr-script-out .. code-block:: none Configuration space object: Hyperparameters: p_failure, Type: UniformFloat, Range: [0.0, 1.0], Default: 0.5 x0, Type: UniformFloat, Range: [0.0, 1.0], Default: 0.5 x1, Type: UniformFloat, Range: [0.0, 1.0], Default: 0.5 x2, Type: UniformFloat, Range: [0.0, 1.0], Default: 0.5 x3, Type: UniformFloat, Range: [0.0, 1.0], Default: 0.5 x4, Type: UniformFloat, Range: [0.0, 1.0], Default: 0.5 x5, Type: UniformFloat, Range: [0.0, 1.0], Default: 0.5 x6, Type: UniformFloat, Range: [0.0, 1.0], Default: 0.5 x7, Type: UniformFloat, Range: [0.0, 1.0], Default: 0.5 x8, Type: UniformFloat, Range: [0.0, 1.0], Default: 0.5 x9, Type: UniformFloat, Range: [0.0, 1.0], Default: 0.5 .. GENERATED FROM PYTHON SOURCE LINES 56-63 We use the centralized Bayesian optimization (CBO) for the search. The :class:`deephyper.hpo.CBO` has a parameter ``filter_failures``. We will compare: - ``filter_failures="ignore"``: filters-out failed configurations. - ``filter_failures="mean"``: replaces failures with the running average of non-failed objectives. - ``filter_failures="min"``: replaces failures with the running minimum of non-failed objectives. .. GENERATED FROM PYTHON SOURCE LINES 63-67 .. code-block:: Python results = {} max_evals = 100 .. GENERATED FROM PYTHON SOURCE LINES 68-90 .. code-block:: Python for failure_strategy in ["ignore", "mean", "min"]: evaluator = Evaluator.create( run, method="thread", method_kwargs={ "callbacks": [TqdmCallback(f"Failure Strategy: {failure_strategy}")]} ) search = CBO( problem, acq_optimizer="ga", acq_optimizer_kwargs=dict( filter_duplicated=False, filter_failures=failure_strategy, ), log_dir=f"search_{failure_strategy}", random_state=42, ) results[failure_strategy] = search.search(evaluator, max_evals) .. rst-class:: sphx-glr-script-out .. code-block:: none 0%| | 0/100 [00:00` .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: plot_notify_failures_hpo.py ` .. container:: sphx-glr-download sphx-glr-download-zip :download:`Download zipped: plot_notify_failures_hpo.zip ` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_