.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "examples/examples_hpo/plot_hpo_text_classification_with_stopper.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note :ref:`Go to the end ` to download the full example code. .. rst-class:: sphx-glr-example-title .. _sphx_glr_examples_examples_hpo_plot_hpo_text_classification_with_stopper.py: Hyperparameter Optimization for Text Classification with Early Discarding ========================================================================= **Author(s)**: Romain Egele, Brett Eiffert. In this example, we will edit the DeepHyper Hyperparameter Search for Text Classification example to use the :mod:`deephyper.stopper` module. The Stopper class is used to check if training per job/evaluation can be ended early and save run time if the stopper algorithm determines that no more training is needed. Read more about the Stopper class `here `_ **Reference**: This example is based on materials from the Pytorch Documentation: `Text classification with the torchtext library `_ .. GENERATED FROM PYTHON SOURCE LINES 16-20 .. code-block:: bash %%bash pip install deephyper ray numpy==1.26.4 torch torchtext==0.17.2 torchdata==0.7.1 'portalocker>=2.0.0' .. GENERATED FROM PYTHON SOURCE LINES 23-27 Imports ------- All imports used in the tutorial are declared at the top of the file. .. GENERATED FROM PYTHON SOURCE LINES 27-47 .. dropdown:: Code (Imports) .. code-block:: Python from deephyper.evaluator import RunningJob import ray import json from functools import partial import torch from torchtext.data.utils import get_tokenizer from torchtext.data.functional import to_map_style_dataset from torchtext.vocab import build_vocab_from_iterator from torchtext.datasets import AG_NEWS from torch.utils.data import DataLoader from torch.utils.data.dataset import random_split from torch import nn .. GENERATED FROM PYTHON SOURCE LINES 48-51 .. note:: The following can be used to detect if **CUDA** devices are available on the current host. Therefore, this notebook will automatically adapt the parallel execution based on the resources available locally. However, it will not be the case if many compute nodes are requested. .. GENERATED FROM PYTHON SOURCE LINES 53-54 If GPU is available, this code will enabled the tutorial to use the GPU for pytorch operations. .. GENERATED FROM PYTHON SOURCE LINES 55-60 .. dropdown:: Code (Code to check if using CPU or GPU) .. code-block:: Python is_gpu_available = torch.cuda.is_available() n_gpus = torch.cuda.device_count() .. GENERATED FROM PYTHON SOURCE LINES 61-66 The dataset ----------- The torchtext library provides a few raw dataset iterators, which yield the raw text strings. For example, the :code:`AG_NEWS` dataset iterators yield the raw data as a tuple of label and text. It has four labels (1 : World 2 : Sports 3 : Business 4 : Sci/Tec). .. GENERATED FROM PYTHON SOURCE LINES 66-84 .. dropdown:: Code (Loading the data) .. code-block:: Python def load_data(train_ratio, fast=False): train_iter, test_iter = AG_NEWS() train_dataset = to_map_style_dataset(train_iter) test_dataset = to_map_style_dataset(test_iter) num_train = int(len(train_dataset) * train_ratio) split_train, split_valid = \ random_split(train_dataset, [num_train, len(train_dataset) - num_train]) ## downsample if fast: split_train, _ = random_split(split_train, [int(len(split_train)*.05), int(len(split_train)*.95)]) split_valid, _ = random_split(split_valid, [int(len(split_valid)*.05), int(len(split_valid)*.95)]) test_dataset, _ = random_split(test_dataset, [int(len(test_dataset)*.05), int(len(test_dataset)*.95)]) return split_train, split_valid, test_dataset .. GENERATED FROM PYTHON SOURCE LINES 85-93 Preprocessing pipelines and Batch generation -------------------------------------------- Here is an example for typical NLP data processing with tokenizer and vocabulary. The first step is to build a vocabulary with the raw training dataset. Here we use built in factory function :code:`build_vocab_from_iterator` which accepts iterator that yield list or iterator of tokens. Users can also pass any special symbols to be added to the vocabulary. The vocabulary block converts a list of tokens into integers. .. GENERATED FROM PYTHON SOURCE LINES 95-99 .. code-block:: python vocab(['here', 'is', 'an', 'example']) >>> [475, 21, 30, 5286] .. GENERATED FROM PYTHON SOURCE LINES 101-102 The text pipeline converts a text string into a list of integers based on the lookup table defined in the vocabulary. The label pipeline converts the label into integers. For example, .. GENERATED FROM PYTHON SOURCE LINES 104-110 .. code-block:: python text_pipeline('here is the an example') >>> [475, 21, 2, 30, 5286] label_pipeline('10') >>> 9 .. GENERATED FROM PYTHON SOURCE LINES 110-141 .. dropdown:: Code (Code to tokenize and build vocabulary for text processing) .. code-block:: Python train_iter = AG_NEWS(split='train') num_class = 4 tokenizer = get_tokenizer('basic_english') def yield_tokens(data_iter): for _, text in data_iter: yield tokenizer(text) vocab = build_vocab_from_iterator(yield_tokens(train_iter), specials=[""]) vocab.set_default_index(vocab[""]) vocab_size = len(vocab) text_pipeline = lambda x: vocab(tokenizer(x)) label_pipeline = lambda x: int(x) - 1 def collate_batch(batch, device): label_list, text_list, offsets = [], [], [0] for (_label, _text) in batch: label_list.append(label_pipeline(_label)) processed_text = torch.tensor(text_pipeline(_text), dtype=torch.int64) text_list.append(processed_text) offsets.append(processed_text.size(0)) label_list = torch.tensor(label_list, dtype=torch.int64) offsets = torch.tensor(offsets[:-1]).cumsum(dim=0) text_list = torch.cat(text_list) return label_list.to(device), text_list.to(device), offsets.to(device) .. GENERATED FROM PYTHON SOURCE LINES 142-144 .. note:: The :code:`collate_fn` function works on a batch of samples generated from :code:`DataLoader`. The input to :code:`collate_fn` is a batch of data with the batch size in :code:`DataLoader`, and :code:`collate_fn` processes them according to the data processing pipelines declared previously. .. GENERATED FROM PYTHON SOURCE LINES 146-150 Define the model ---------------- The model is composed of the `nn.EmbeddingBag `_ layer plus a linear layer for the classification purpose. .. GENERATED FROM PYTHON SOURCE LINES 150-170 .. dropdown:: Code (Defining the Text Classification model) .. code-block:: Python class TextClassificationModel(nn.Module): def __init__(self, vocab_size, embed_dim, num_class): super().__init__() self.embedding = nn.EmbeddingBag(vocab_size, embed_dim, sparse=False) self.fc = nn.Linear(embed_dim, num_class) self.init_weights() def init_weights(self): initrange = 0.5 self.embedding.weight.data.uniform_(-initrange, initrange) self.fc.weight.data.uniform_(-initrange, initrange) self.fc.bias.data.zero_() def forward(self, text, offsets): embedded = self.embedding(text, offsets) return self.fc(embedded) .. GENERATED FROM PYTHON SOURCE LINES 171-173 Define functions to train the model and evaluate results. --------------------------------------------------------- .. GENERATED FROM PYTHON SOURCE LINES 173-197 .. dropdown:: Code (Define the training and evaluation of the Text Classification model) .. code-block:: Python def train(model, criterion, optimizer, dataloader): model.train() for _, (label, text, offsets) in enumerate(dataloader): optimizer.zero_grad() predicted_label = model(text, offsets) loss = criterion(predicted_label, label) loss.backward() torch.nn.utils.clip_grad_norm_(model.parameters(), 0.1) optimizer.step() def evaluate(model, dataloader): model.eval() total_acc, total_count = 0, 0 with torch.no_grad(): for _, (label, text, offsets) in enumerate(dataloader): predicted_label = model(text, offsets) total_acc += (predicted_label.argmax(1) == label).sum().item() total_count += label.size(0) return total_acc/total_count .. GENERATED FROM PYTHON SOURCE LINES 198-213 Define the run-function ----------------------- The run-function defines how the objective that we want to maximize is computed. It takes a :code:`config` dictionary as input and often returns a scalar value that we want to maximize. The :code:`config` contains a sample value of hyperparameters that we want to tune. In this example we will search for: * :code:`num_epochs` (default value: :code:`10`) * :code:`batch_size` (default value: :code:`64`) * :code:`learning_rate` (default value: :code:`5`) A hyperparameter value can be accessed easily in the dictionary through the corresponding key, for example :code:`config["units"]`. When a Stopper is defined and set as a parameter in a search (below :code:`CBO()``), the run function must invoke methods :code:`job.record()` and :code:`job.stopped()`. :code:`job.record()` tells the Stopper which values to watch so it knows to stop and then :code:`job.stopped()` is a state the stopper uses to exit the specific job in the search earlier than expected. .. GENERATED FROM PYTHON SOURCE LINES 213-245 .. code-block:: Python def get_run(train_ratio=0.95): def run(job: RunningJob): device = torch.device("cuda" if torch.cuda.is_available() else "cpu") embed_dim = 64 num_epochs = 100 collate_fn = partial(collate_batch, device=device) split_train, split_valid, _ = load_data(train_ratio, fast=True) # set fast=false for longer running, more accurate example train_dataloader = DataLoader(split_train, batch_size=int(job["batch_size"]), shuffle=True, collate_fn=collate_fn) valid_dataloader = DataLoader(split_valid, batch_size=int(job["batch_size"]), shuffle=True, collate_fn=collate_fn) model = TextClassificationModel(vocab_size, int(embed_dim), num_class).to(device) criterion = torch.nn.CrossEntropyLoss() optimizer = torch.optim.SGD(model.parameters(), lr=job["learning_rate"]) accu_list = [] for i in range(1, num_epochs + 1): train(model, criterion, optimizer, train_dataloader) accu_list.append(evaluate(model, valid_dataloader)) job.record(budget = i + 1, objective=evaluate(model, valid_dataloader)) if job.stopped(): break accu_test = evaluate(model, valid_dataloader) return {"objective": accu_test, "metadata": {"index_stopped": i, "accu_list": accu_list}} return run .. GENERATED FROM PYTHON SOURCE LINES 246-247 We create two versions of :code:`run`, one quicker to evaluate for the search, with a small training dataset, and another one, for performance evaluation, which uses a normal training/validation ratio. .. GENERATED FROM PYTHON SOURCE LINES 249-252 .. code-block:: Python quick_run = get_run(train_ratio=0.3) perf_run = get_run(train_ratio=0.95) .. GENERATED FROM PYTHON SOURCE LINES 253-256 .. note:: The objective maximised by DeepHyper is the scalar value returned by the :code:`run`-function. In this tutorial it corresponds to the validation accuracy of the model after training. .. GENERATED FROM PYTHON SOURCE LINES 258-268 Define the Hyperparameter optimization problem ---------------------------------------------- Hyperparameter ranges are defined using the following syntax: * Discrete integer ranges are generated from a tuple :code:`(lower: int, upper: int)` * Continuous prarameters are generated from a tuple :code:`(lower: float, upper: float)` * Categorical or nonordinal hyperparameter ranges can be given as a list of possible values :code:`[val1, val2, ...]` We provide the default configuration of hyperparameters as a starting point of the problem. .. GENERATED FROM PYTHON SOURCE LINES 270-280 .. code-block:: Python from deephyper.hpo import HpProblem problem = HpProblem() # Discrete and Real hyperparameters (sampled with log-uniform) problem.add_hyperparameter((8, 512, "log-uniform"), "batch_size", default_value=64) problem.add_hyperparameter((0.1, 10, "log-uniform"), "learning_rate", default_value=5) problem .. rst-class:: sphx-glr-script-out .. code-block:: none Configuration space object: Hyperparameters: batch_size, Type: UniformInteger, Range: [8, 512], Default: 64, on log-scale learning_rate, Type: UniformFloat, Range: [0.1, 10.0], Default: 5.0, on log-scale .. GENERATED FROM PYTHON SOURCE LINES 281-285 Evaluate a default configuration -------------------------------- We evaluate the performance of the default set of hyperparameters provided in the Pytorch tutorial. .. GENERATED FROM PYTHON SOURCE LINES 285-305 .. dropdown:: Code (Imports) .. code-block:: Python #We launch the Ray run-time and execute the `run` function #with the default configuration if is_gpu_available: if not(ray.is_initialized()): ray.init(num_cpus=n_gpus, num_gpus=n_gpus, log_to_driver=False) run_default = ray.remote(num_cpus=1, num_gpus=1)(perf_run) objective_default = ray.get(run_default.remote(RunningJob(parameters=problem.default_configuration))) else: if not(ray.is_initialized()): ray.init(num_cpus=1, log_to_driver=False) run_default = perf_run objective_default = run_default(RunningJob(parameters=problem.default_configuration)) print(problem.default_configuration) print(f"Accuracy Default Configuration: {objective_default["objective"]:.3f}") .. rst-class:: sphx-glr-script-out .. code-block:: none 2025-10-21 16:04:22,173 INFO worker.py:1843 -- Started a local Ray instance. View the dashboard at http://127.0.0.1:8265 {'batch_size': 64, 'learning_rate': 5.0} Accuracy Default Configuration: 0.887 .. GENERATED FROM PYTHON SOURCE LINES 306-312 Define the evaluator object --------------------------- The :code:`Evaluator` object allows to change the parallelization backend used by DeepHyper. It is a standalone object which schedules the execution of remote tasks. All evaluators needs a :code:`run_function` to be instantiated. Then a keyword :code:`method` defines the backend (e.g., :code:`"ray"`) and the :code:`method_kwargs` corresponds to keyword arguments of this chosen :code:`method`. .. GENERATED FROM PYTHON SOURCE LINES 314-317 .. code-block:: python evaluator = Evaluator.create(run_function, method, method_kwargs) .. GENERATED FROM PYTHON SOURCE LINES 319-322 Once created the :code:`evaluator.num_workers` gives access to the number of available parallel workers. Finally, to submit and collect tasks to the evaluator one just needs to use the following interface: .. GENERATED FROM PYTHON SOURCE LINES 324-331 .. code-block:: python configs = [...] evaluator.submit(configs) ... tasks_done = evaluator.get("BATCH", size=1) # For asynchronous tasks_done = evaluator.get("ALL") # For batch synchronous .. GENERATED FROM PYTHON SOURCE LINES 333-334 .. warning:: Each `Evaluator` saves its own state, therefore it is crucial to create a new evaluator when launching a fresh search. .. GENERATED FROM PYTHON SOURCE LINES 334-367 .. dropdown:: Code (Imports) .. code-block:: Python from deephyper.evaluator import Evaluator from deephyper.evaluator.callback import TqdmCallback def get_evaluator(run_function): # Default arguments for Ray: 1 worker and 1 worker per evaluation method_kwargs = { "num_cpus": 1, "num_cpus_per_task": 1, "callbacks": [TqdmCallback()] } # If GPU devices are detected then it will create 'n_gpus' workers # and use 1 worker for each evaluation if is_gpu_available: method_kwargs["num_cpus"] = n_gpus method_kwargs["num_gpus"] = n_gpus method_kwargs["num_cpus_per_task"] = 1 method_kwargs["num_gpus_per_task"] = 1 evaluator = Evaluator.create( run_function, method="ray", method_kwargs=method_kwargs ) print(f"Created new evaluator with {evaluator.num_workers} worker{'s' if evaluator.num_workers > 1 else ''} and config: {method_kwargs}", ) return evaluator evaluator = get_evaluator(quick_run) .. rst-class:: sphx-glr-script-out .. code-block:: none Created new evaluator with 1 worker and config: {'num_cpus': 1, 'num_cpus_per_task': 1, 'callbacks': []} .. GENERATED FROM PYTHON SOURCE LINES 368-374 Define and run the Centralized Bayesian Optimization search (CBO) ----------------------------------------------------------------- We create the CBO using the :code:`problem` and :code:`evaluator` defined above. A Stopper is also defined and passed as an argument to the CBO. This Stopper controls the :code:`job.observe()` and :code:`job.stopped()` functions. .. GENERATED FROM PYTHON SOURCE LINES 376-379 .. code-block:: Python from deephyper.hpo import CBO from deephyper.stopper import SuccessiveHalvingStopper .. GENERATED FROM PYTHON SOURCE LINES 380-381 Instantiate the search with the problem and a specific evaluator .. GENERATED FROM PYTHON SOURCE LINES 381-384 .. code-block:: Python stopper = SuccessiveHalvingStopper(min_steps=1, max_steps=100) search = CBO(problem, stopper=stopper, log_dir="stopper-log-files") .. rst-class:: sphx-glr-script-out .. code-block:: none Results file already exists, it will be renamed to /Users/rp5/Documents/DeepHyper/deephyper/examples/examples_hpo/stopper-log-files/results_20251021-160501.csv .. GENERATED FROM PYTHON SOURCE LINES 385-390 .. note:: All DeepHyper's search algorithm have two stopping criteria: * :code:`max_evals (int)`: Defines the maximum number of evaluations that we want to perform. Default to :code:`-1` for an infinite number. * :code:`timeout (int)`: Defines a time budget (in seconds) before stopping the search. Default to :code:`None` for an infinite time budget. .. GENERATED FROM PYTHON SOURCE LINES 392-394 .. code-block:: Python results = search.search(evaluator, max_evals=30) .. rst-class:: sphx-glr-script-out .. code-block:: none 0%| | 0/30 [00:00
p:batch_size p:learning_rate objective job_id job_status m:timestamp_submit m:index_stopped m:accu_list m:timestamp_gather
0 46 0.282405 0.729286 0 DONE 1.081796 99 [0.27214285714285713, 0.29642857142857143, 0.3... 30.520860
1 10 0.263392 0.786667 1 DONE 30.545998 99 [0.2757142857142857, 0.30738095238095237, 0.34... 91.025548
2 29 0.620113 0.807619 2 DONE 91.036948 99 [0.3061904761904762, 0.3404761904761905, 0.369... 127.161049
3 313 1.944931 0.275952 3 DONE 127.173890 1 [0.27595238095238095] 127.940275
4 14 0.874725 0.810000 4 DONE 127.951657 99 [0.33214285714285713, 0.3995238095238095, 0.40... 183.347690
5 29 0.965256 0.282857 5 DONE 183.612584 1 [0.28285714285714286] 184.573359
6 11 1.211957 0.377381 6 DONE 184.835052 2 [0.3385714285714286, 0.3773809523809524] 186.740075
7 20 0.858054 0.300952 7 DONE 187.125586 1 [0.30095238095238097] 188.157044
8 13 0.854774 0.336429 8 DONE 188.556284 2 [0.31976190476190475, 0.3364285714285714] 190.540544
9 14 0.875480 0.293333 9 DONE 190.792234 1 [0.29333333333333333] 191.928877
10 56 0.575649 0.285000 10 DONE 192.176605 1 [0.285] 193.011374
11 29 0.623690 0.288095 11 DONE 193.268442 1 [0.28809523809523807] 194.197651
12 15 0.106875 0.270952 12 DONE 194.455415 1 [0.27095238095238094] 195.550734
13 10 0.281804 0.267857 13 DONE 195.816266 1 [0.26785714285714285] 197.110387
14 37 0.257322 0.271905 14 DONE 197.359947 1 [0.2719047619047619] 198.260534
15 33 0.381022 0.297143 15 DONE 198.506771 1 [0.29714285714285715] 199.443259
16 66 0.281230 0.258095 16 DONE 199.790292 1 [0.2580952380952381] 200.614023
17 29 0.580171 0.508571 17 DONE 200.865638 8 [0.3171428571428571, 0.38857142857142857, 0.41... 204.302500
18 45 0.269436 0.279048 18 DONE 204.555375 1 [0.27904761904761904] 205.415226
19 46 0.285466 0.260000 19 DONE 205.678262 1 [0.26] 206.542666
20 8 0.255445 0.326190 20 DONE 206.853032 2 [0.30023809523809525, 0.3261904761904762] 209.160815
21 14 0.871431 0.801429 21 DONE 209.422959 80 [0.31642857142857145, 0.39880952380952384, 0.4... 254.199654
22 14 0.874070 0.375238 22 DONE 254.579880 2 [0.3057142857142857, 0.37523809523809526] 256.296301
23 13 0.265414 0.260476 23 DONE 256.555508 1 [0.2604761904761905] 257.700605
24 11 0.263654 0.258333 24 DONE 257.965976 1 [0.25833333333333336] 259.181543
25 14 0.871357 0.317381 25 DONE 259.455943 2 [0.3088095238095238, 0.3173809523809524] 261.115557
26 40 0.282374 0.274762 26 DONE 261.374624 1 [0.27476190476190476] 262.256141
27 10 0.264670 0.285238 27 DONE 262.512628 1 [0.28523809523809524] 263.792388
28 27 0.619722 0.266190 28 DONE 264.053499 1 [0.2661904761904762] 264.994080
29 29 0.619863 0.278571 29 DONE 265.421543 1 [0.2785714285714286] 266.385228


.. GENERATED FROM PYTHON SOURCE LINES 410-416 Visualizing the Stopper ----------------------- This graph shows the same information as described above but in a visual form. Each of the 30 jobs and the rate at which they learned against the validation dataset is shown here. As shown above, not all job lines will show 100 epochs because the Stopper determined the jobs did not need to run the full time to converge on a solution. .. GENERATED FROM PYTHON SOURCE LINES 418-434 .. code-block:: Python import numpy as np import matplotlib.pyplot as plt i = 0 for row in results.iterrows(): y = row[1]["m:accu_list"] x = np.arange(i+1, i+1+len(y)) plt.plot(x, y, label=row[1]["job_id"]) i += len(y) plt.xlabel('Epoch') plt.ylabel('Validation accuracy') plt.title("Validation Accuracies during training") plt.show() .. image-sg:: /examples/examples_hpo/images/sphx_glr_plot_hpo_text_classification_with_stopper_001.png :alt: Validation Accuracies during training :srcset: /examples/examples_hpo/images/sphx_glr_plot_hpo_text_classification_with_stopper_001.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 435-439 Evaluate the best configuration ------------------------------- Now that the search is over, let us print the best configuration found during this run and evaluate it on the full training dataset. .. GENERATED FROM PYTHON SOURCE LINES 441-443 Show the job with best configuration and compare this with the graph above. The result of the comparison should be intuitive - the job with the best objective in the graph should match :code:`i_max`. .. GENERATED FROM PYTHON SOURCE LINES 443-446 .. code-block:: Python i_max = results.objective.argmax() i_max .. rst-class:: sphx-glr-script-out .. code-block:: none 4 .. GENERATED FROM PYTHON SOURCE LINES 447-456 .. code-block:: Python best_config = results.iloc[i_max][:-3].to_dict() best_config = {k[2:]: v for k, v in best_config.items() if k.startswith("p:")} print(f"The default configuration has an accuracy of {objective_default["objective"]:.3f}. \n" f"The best configuration found by DeepHyper has an accuracy {results['objective'].iloc[i_max]:.3f}, \n" f"finished after {results['m:timestamp_gather'].iloc[i_max]:.2f} seconds of search.\n") print(json.dumps(best_config, indent=4)) .. rst-class:: sphx-glr-script-out .. code-block:: none The default configuration has an accuracy of 0.887. The best configuration found by DeepHyper has an accuracy 0.810, finished after 183.35 seconds of search. { "batch_size": 14, "learning_rate": 0.8747246682101409 } .. GENERATED FROM PYTHON SOURCE LINES 457-459 .. code-block:: Python objective_best = perf_run(RunningJob(parameters=best_config)) print(f"Accuracy Best Configuration: {objective_best["objective"]:.3f}") .. rst-class:: sphx-glr-script-out .. code-block:: none Accuracy Best Configuration: 0.867 .. rst-class:: sphx-glr-timing **Total running time of the script:** (7 minutes 49.357 seconds) .. _sphx_glr_download_examples_examples_hpo_plot_hpo_text_classification_with_stopper.py: .. only:: html .. container:: sphx-glr-footer sphx-glr-footer-example .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: plot_hpo_text_classification_with_stopper.ipynb ` .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: plot_hpo_text_classification_with_stopper.py ` .. container:: sphx-glr-download sphx-glr-download-zip :download:`Download zipped: plot_hpo_text_classification_with_stopper.zip ` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_