.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "examples/examples_bbo/plot_transfer_learning_for_hpo.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note :ref:`Go to the end ` to download the full example code. .. rst-class:: sphx-glr-example-title .. _sphx_glr_examples_examples_bbo_plot_transfer_learning_for_hpo.py: Using Transfer Learning to Speed-up Bayesian Optimization ========================================================= **Author(s)**: Romain Egele. In this example, we demonstrate how to leverage transfer learning for hyperparameter optimization. Imagine you are working on multiple related tasks, such as optimizing the hyperparameters of neural networks for various datasets. It's reasonable to expect that similar hyperparameter configurations might perform well across these datasets, even if some minor adjustments are needed to fine-tune performance. By conducting a thorough (and potentially expensive) search on one task, you can reuse the resulting hyperparameter set to guide and accelerate optimization for subsequent tasks. This approach reduces computational costs while maintaining high performance. To illustrate, we will use a simple and computationally inexpensive example: minimizing the function :math:`f(x) = \sum_{i=0}^ {n-1}`. Here, the difficulty of the problem is defined by the number of variables :math:`n`. We will start by optimizing the small problem where :math:`n=5`. Then, we will apply transfer learning to optimize a larger problem where :math:`n=10`, comparing the results with and without transfer learning to highlight the benefits. Let's begin by defining the run-functions for both the small-scale and large-scale problems: .. GENERATED FROM PYTHON SOURCE LINES 18-32 .. dropdown:: Code (Import statements) .. code-block:: Python import functools import matplotlib.pyplot as plt from deephyper.analysis.hpo import plot_search_trajectory_single_objective_hpo from deephyper.evaluator import Evaluator from deephyper.evaluator.callback import TqdmCallback from deephyper.hpo import CBO, HpProblem WIDTH_PLOTS = 8 HEIGHT_PLOTS = WIDTH_PLOTS / 1.618 .. GENERATED FROM PYTHON SOURCE LINES 33-44 .. code-block:: Python def run(job, N: int) -> float: # Definition of the function to minimize y = sum([job.parameters[f"x{i}"] ** 2 for i in range(N)]) return -y # Use the `-` sign to perform minimization n_small = 5 n_large = 10 run_small = functools.partial(run, N=n_small) run_large = functools.partial(run, N=n_large) .. GENERATED FROM PYTHON SOURCE LINES 45-46 Then, we can define the hyperparameter problem space based on :math:`n` .. GENERATED FROM PYTHON SOURCE LINES 46-53 .. code-block:: Python def create_problem(n): problem = HpProblem() for i in range(n): problem.add_hyperparameter((-10.0, 10.0), f"x{i}") return problem .. GENERATED FROM PYTHON SOURCE LINES 54-56 .. code-block:: Python problem_small = create_problem(n_small) .. GENERATED FROM PYTHON SOURCE LINES 57-59 .. code-block:: Python problem_large = create_problem(n_large) .. GENERATED FROM PYTHON SOURCE LINES 60-61 We define the parameters of the search: .. GENERATED FROM PYTHON SOURCE LINES 61-66 .. code-block:: Python search_kwargs = { "acq_optimizer": "ga", # Optimizing the acquisition function with countinuous genetic algorithm "random_state": 42, } .. GENERATED FROM PYTHON SOURCE LINES 67-69 We create a dictionnary that will store the results of each experiment and also fix the number of evaluation of the search to 200. .. GENERATED FROM PYTHON SOURCE LINES 69-72 .. code-block:: Python results = {} max_evals = 200 .. GENERATED FROM PYTHON SOURCE LINES 73-74 Then, we run the search for each problem. We start with the small problem: .. GENERATED FROM PYTHON SOURCE LINES 74-86 .. code-block:: Python evaluator_small = Evaluator.create( run_small, method="thread", method_kwargs={"callbacks": [TqdmCallback("HPO - Small Problem")]}, ) search_small = CBO( problem_small, **search_kwargs, ) results_small = search_small.search(evaluator_small, max_evals) .. rst-class:: sphx-glr-script-out .. code-block:: none 0%| | 0/200 [00:00 .. rst-class:: sphx-glr-timing **Total running time of the script:** (1 minutes 51.170 seconds) .. _sphx_glr_download_examples_examples_bbo_plot_transfer_learning_for_hpo.py: .. only:: html .. container:: sphx-glr-footer sphx-glr-footer-example .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: plot_transfer_learning_for_hpo.ipynb ` .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: plot_transfer_learning_for_hpo.py ` .. container:: sphx-glr-download sphx-glr-download-zip :download:`Download zipped: plot_transfer_learning_for_hpo.zip ` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_