.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "examples/examples_hpo/plot_hpo_text_classification.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note :ref:`Go to the end ` to download the full example code. .. rst-class:: sphx-glr-example-title .. _sphx_glr_examples_examples_hpo_plot_hpo_text_classification.py: Hyperparameter search for text classification ============================================= **Author(s)**: Romain Egele, Brett Eiffert. In this tutorial we present how to use hyperparameter optimization on a text classification analysis example from the Pytorch documentation. **Reference**: This tutorial is based on materials from the Pytorch Documentation: `Text classification with the torchtext library `_ .. GENERATED FROM PYTHON SOURCE LINES 15-19 .. code-block:: bash %%bash pip install deephyper ray numpy==1.26.4 torch torchtext==0.17.2 torchdata==0.7.1 'portalocker>=2.0.0' .. GENERATED FROM PYTHON SOURCE LINES 22-26 Imports ------- All imports used in the tutorial are declared at the top of the file. .. GENERATED FROM PYTHON SOURCE LINES 26-44 .. dropdown:: Code (Imports) .. code-block:: Python import ray import json from functools import partial import torch from torchtext.data.utils import get_tokenizer from torchtext.data.functional import to_map_style_dataset from torchtext.vocab import build_vocab_from_iterator from torchtext.datasets import AG_NEWS from torch.utils.data import DataLoader from torch.utils.data.dataset import random_split from torch import nn .. GENERATED FROM PYTHON SOURCE LINES 45-48 .. note:: The following can be used to detect if **CUDA** devices are available on the current host. Therefore, this notebook will automatically adapt the parallel execution based on the ressources available locally. However, it will not be the case if many compute nodes are requested. .. GENERATED FROM PYTHON SOURCE LINES 50-51 If GPU is available, this code will enabled the tutorial to use the GPU for pytorch operations. .. GENERATED FROM PYTHON SOURCE LINES 52-57 .. dropdown:: Code (Code to check if using CPU or GPU) .. code-block:: Python is_gpu_available = torch.cuda.is_available() n_gpus = torch.cuda.device_count() .. GENERATED FROM PYTHON SOURCE LINES 58-63 The dataset ----------- The torchtext library provides a few raw dataset iterators, which yield the raw text strings. For example, the :code:`AG_NEWS` dataset iterators yield the raw data as a tuple of label and text. It has four labels (1 : World 2 : Sports 3 : Business 4 : Sci/Tec). .. GENERATED FROM PYTHON SOURCE LINES 63-81 .. dropdown:: Code (Loading the data) .. code-block:: Python def load_data(train_ratio, fast=False): train_iter, test_iter = AG_NEWS() train_dataset = to_map_style_dataset(train_iter) test_dataset = to_map_style_dataset(test_iter) num_train = int(len(train_dataset) * train_ratio) split_train, split_valid = \ random_split(train_dataset, [num_train, len(train_dataset) - num_train]) ## downsample if fast: split_train, _ = random_split(split_train, [int(len(split_train)*.05), int(len(split_train)*.95)]) split_valid, _ = random_split(split_valid, [int(len(split_valid)*.05), int(len(split_valid)*.95)]) test_dataset, _ = random_split(test_dataset, [int(len(test_dataset)*.05), int(len(test_dataset)*.95)]) return split_train, split_valid, test_dataset .. GENERATED FROM PYTHON SOURCE LINES 82-90 Preprocessing pipelines and Batch generation -------------------------------------------- Here is an example for typical NLP data processing with tokenizer and vocabulary. The first step is to build a vocabulary with the raw training dataset. Here we use built in factory function :code:`build_vocab_from_iterator` which accepts iterator that yield list or iterator of tokens. Users can also pass any special symbols to be added to the vocabulary. The vocabulary block converts a list of tokens into integers. .. GENERATED FROM PYTHON SOURCE LINES 92-96 .. code-block:: python vocab(['here', 'is', 'an', 'example']) >>> [475, 21, 30, 5286] .. GENERATED FROM PYTHON SOURCE LINES 98-99 The text pipeline converts a text string into a list of integers based on the lookup table defined in the vocabulary. The label pipeline converts the label into integers. For example, .. GENERATED FROM PYTHON SOURCE LINES 101-107 .. code-block:: python text_pipeline('here is the an example') >>> [475, 21, 2, 30, 5286] label_pipeline('10') >>> 9 .. GENERATED FROM PYTHON SOURCE LINES 107-138 .. dropdown:: Code (Code to tokenize and build vocabulary for text processing) .. code-block:: Python train_iter = AG_NEWS(split='train') num_class = 4 tokenizer = get_tokenizer('basic_english') def yield_tokens(data_iter): for _, text in data_iter: yield tokenizer(text) vocab = build_vocab_from_iterator(yield_tokens(train_iter), specials=[""]) vocab.set_default_index(vocab[""]) vocab_size = len(vocab) text_pipeline = lambda x: vocab(tokenizer(x)) label_pipeline = lambda x: int(x) - 1 def collate_batch(batch, device): label_list, text_list, offsets = [], [], [0] for (_label, _text) in batch: label_list.append(label_pipeline(_label)) processed_text = torch.tensor(text_pipeline(_text), dtype=torch.int64) text_list.append(processed_text) offsets.append(processed_text.size(0)) label_list = torch.tensor(label_list, dtype=torch.int64) offsets = torch.tensor(offsets[:-1]).cumsum(dim=0) text_list = torch.cat(text_list) return label_list.to(device), text_list.to(device), offsets.to(device) .. GENERATED FROM PYTHON SOURCE LINES 139-141 .. note:: The :code:`collate_fn` function works on a batch of samples generated from :code:`DataLoader`. The input to :code:`collate_fn` is a batch of data with the batch size in :code:`DataLoader`, and :code:`collate_fn` processes them according to the data processing pipelines declared previously. .. GENERATED FROM PYTHON SOURCE LINES 143-147 Define the model ---------------- The model is composed of the `nn.EmbeddingBag `_ layer plus a linear layer for the classification purpose. .. GENERATED FROM PYTHON SOURCE LINES 147-167 .. dropdown:: Code (Defining the Text Classification model) .. code-block:: Python class TextClassificationModel(nn.Module): def __init__(self, vocab_size, embed_dim, num_class): super().__init__() self.embedding = nn.EmbeddingBag(vocab_size, embed_dim, sparse=False) self.fc = nn.Linear(embed_dim, num_class) self.init_weights() def init_weights(self): initrange = 0.5 self.embedding.weight.data.uniform_(-initrange, initrange) self.fc.weight.data.uniform_(-initrange, initrange) self.fc.bias.data.zero_() def forward(self, text, offsets): embedded = self.embedding(text, offsets) return self.fc(embedded) .. GENERATED FROM PYTHON SOURCE LINES 168-170 Define functions to train the model and evaluate results. --------------------------------------------------------- .. GENERATED FROM PYTHON SOURCE LINES 170-194 .. dropdown:: Code (Define the training and evaluation of the Text Classification model) .. code-block:: Python def train(model, criterion, optimizer, dataloader): model.train() for _, (label, text, offsets) in enumerate(dataloader): optimizer.zero_grad() predicted_label = model(text, offsets) loss = criterion(predicted_label, label) loss.backward() torch.nn.utils.clip_grad_norm_(model.parameters(), 0.1) optimizer.step() def evaluate(model, dataloader): model.eval() total_acc, total_count = 0, 0 with torch.no_grad(): for _, (label, text, offsets) in enumerate(dataloader): predicted_label = model(text, offsets) total_acc += (predicted_label.argmax(1) == label).sum().item() total_count += label.size(0) return total_acc/total_count .. GENERATED FROM PYTHON SOURCE LINES 195-205 Define the run-function ----------------------- The run-function defines how the objective that we want to maximize is computed. It takes a :code:`config` dictionary as input and often returns a scalar value that we want to maximize. The :code:`config` contains a sample value of hyperparameters that we want to tune. In this example we will search for: * :code:`num_epochs` (default value: :code:`10`) * :code:`batch_size` (default value: :code:`64`) * :code:`learning_rate` (default value: :code:`5`) A hyperparameter value can be acessed easily in the dictionary through the corresponding key, for example :code:`config["units"]`. .. GENERATED FROM PYTHON SOURCE LINES 205-232 .. dropdown:: Code (Run the Text Classification model) .. code-block:: Python def get_run(train_ratio=0.95): def run(config: dict): device = torch.device("cuda" if torch.cuda.is_available() else "cpu") embed_dim = 64 collate_fn = partial(collate_batch, device=device) split_train, split_valid, _ = load_data(train_ratio, fast=True) # set fast=false for longer running, more accurate example train_dataloader = DataLoader(split_train, batch_size=int(config["batch_size"]), shuffle=True, collate_fn=collate_fn) valid_dataloader = DataLoader(split_valid, batch_size=int(config["batch_size"]), shuffle=True, collate_fn=collate_fn) model = TextClassificationModel(vocab_size, int(embed_dim), num_class).to(device) criterion = torch.nn.CrossEntropyLoss() optimizer = torch.optim.SGD(model.parameters(), lr=config["learning_rate"]) for _ in range(1, int(config["num_epochs"]) + 1): train(model, criterion, optimizer, train_dataloader) accu_test = evaluate(model, valid_dataloader) return accu_test return run .. GENERATED FROM PYTHON SOURCE LINES 233-234 We create two versions of :code:`run`, one quicker to evaluate for the search, with a small training dataset, and another one, for performance evaluation, which uses a normal training/validation ratio. .. GENERATED FROM PYTHON SOURCE LINES 236-239 .. code-block:: Python quick_run = get_run(train_ratio=0.3) perf_run = get_run(train_ratio=0.95) .. GENERATED FROM PYTHON SOURCE LINES 240-243 .. note:: The objective maximised by DeepHyper is the scalar value returned by the :code:`run`-function. In this tutorial it corresponds to the validation accuracy of the model after training. .. GENERATED FROM PYTHON SOURCE LINES 245-255 Define the Hyperparameter optimization problem ---------------------------------------------- Hyperparameter ranges are defined using the following syntax: * Discrete integer ranges are generated from a tuple :code:`(lower: int, upper: int)` * Continuous prarameters are generated from a tuple :code:`(lower: float, upper: float)` * Categorical or nonordinal hyperparameter ranges can be given as a list of possible values :code:`[val1, val2, ...]` We provide the default configuration of hyperparameters as a starting point of the problem. .. GENERATED FROM PYTHON SOURCE LINES 257-270 .. code-block:: Python from deephyper.hpo import HpProblem problem = HpProblem() # Discrete hyperparameter (sampled with uniform prior) problem.add_hyperparameter((5, 20), "num_epochs", default_value=10) # Discrete and Real hyperparameters (sampled with log-uniform) problem.add_hyperparameter((8, 512, "log-uniform"), "batch_size", default_value=64) problem.add_hyperparameter((0.1, 10, "log-uniform"), "learning_rate", default_value=5) problem .. rst-class:: sphx-glr-script-out .. code-block:: none Configuration space object: Hyperparameters: batch_size, Type: UniformInteger, Range: [8, 512], Default: 64, on log-scale learning_rate, Type: UniformFloat, Range: [0.1, 10.0], Default: 5.0, on log-scale num_epochs, Type: UniformInteger, Range: [5, 20], Default: 10 .. GENERATED FROM PYTHON SOURCE LINES 271-275 Evaluate a default configuration -------------------------------- We evaluate the performance of the default set of hyperparameters provided in the Pytorch tutorial. .. GENERATED FROM PYTHON SOURCE LINES 275-292 .. code-block:: Python #We launch the Ray run-time and execute the `run` function #with the default configuration if is_gpu_available: if not(ray.is_initialized()): ray.init(num_cpus=n_gpus, num_gpus=n_gpus, log_to_driver=False) run_default = ray.remote(num_cpus=1, num_gpus=1)(perf_run) objective_default = ray.get(run_default.remote(problem.default_configuration)) else: if not(ray.is_initialized()): ray.init(num_cpus=1, log_to_driver=False) run_default = perf_run objective_default = run_default(problem.default_configuration) print(f"Accuracy Default Configuration: {objective_default:.3f}") .. rst-class:: sphx-glr-script-out .. code-block:: none 2025-08-18 14:44:20,921 INFO worker.py:1843 -- Started a local Ray instance. View the dashboard at http://127.0.0.1:8265 Accuracy Default Configuration: 0.863 .. GENERATED FROM PYTHON SOURCE LINES 293-299 Define the evaluator object --------------------------- The :code:`Evaluator` object allows to change the parallelization backend used by DeepHyper. It is a standalone object which schedules the execution of remote tasks. All evaluators needs a :code:`run_function` to be instantiated. Then a keyword :code:`method` defines the backend (e.g., :code:`"ray"`) and the :code:`method_kwargs` corresponds to keyword arguments of this chosen :code:`method`. .. GENERATED FROM PYTHON SOURCE LINES 301-304 .. code-block:: python evaluator = Evaluator.create(run_function, method, method_kwargs) .. GENERATED FROM PYTHON SOURCE LINES 306-309 Once created the :code:`evaluator.num_workers` gives access to the number of available parallel workers. Finally, to submit and collect tasks to the evaluator one just needs to use the following interface: .. GENERATED FROM PYTHON SOURCE LINES 311-318 .. code-block:: python configs = [...] evaluator.submit(configs) ... tasks_done = evaluator.get("BATCH", size=1) # For asynchronous tasks_done = evaluator.get("ALL") # For batch synchronous .. GENERATED FROM PYTHON SOURCE LINES 320-321 .. warning:: Each `Evaluator` saves its own state, therefore it is crucial to create a new evaluator when launching a fresh search. .. GENERATED FROM PYTHON SOURCE LINES 323-353 .. code-block:: Python from deephyper.evaluator import Evaluator from deephyper.evaluator.callback import TqdmCallback def get_evaluator(run_function): # Default arguments for Ray: 1 worker and 1 worker per evaluation method_kwargs = { "num_cpus": 1, "num_cpus_per_task": 1, "callbacks": [TqdmCallback()] } # If GPU devices are detected then it will create 'n_gpus' workers # and use 1 worker for each evaluation if is_gpu_available: method_kwargs["num_cpus"] = n_gpus method_kwargs["num_gpus"] = n_gpus method_kwargs["num_cpus_per_task"] = 1 method_kwargs["num_gpus_per_task"] = 1 evaluator = Evaluator.create( run_function, method="ray", method_kwargs=method_kwargs ) print(f"Created new evaluator with {evaluator.num_workers} worker{'s' if evaluator.num_workers > 1 else ''} and config: {method_kwargs}", ) return evaluator evaluator = get_evaluator(quick_run) .. rst-class:: sphx-glr-script-out .. code-block:: none Created new evaluator with 1 worker and config: {'num_cpus': 1, 'num_cpus_per_task': 1, 'callbacks': []} .. GENERATED FROM PYTHON SOURCE LINES 354-358 Define and run the Centralized Bayesian Optimization search (CBO) ----------------------------------------------------------------- We create the CBO using the :code:`problem` and :code:`evaluator` defined above. .. GENERATED FROM PYTHON SOURCE LINES 360-362 .. code-block:: Python from deephyper.hpo import CBO .. GENERATED FROM PYTHON SOURCE LINES 363-364 Instanciate the search with the problem and a specific evaluator .. GENERATED FROM PYTHON SOURCE LINES 364-366 .. code-block:: Python search = CBO(problem) .. rst-class:: sphx-glr-script-out .. code-block:: none Results file already exists, it will be renamed to /Users/rp5/Documents/DeepHyper/deephyper/examples/examples_hpo/results_20250818-144426.csv .. GENERATED FROM PYTHON SOURCE LINES 367-372 .. note:: All DeepHyper's search algorithm have two stopping criteria: * :code:`max_evals (int)`: Defines the maximum number of evaluations that we want to perform. Default to :code:`-1` for an infinite number. * :code:`timeout (int)`: Defines a time budget (in seconds) before stopping the search. Default to :code:`None` for an infinite time budget. .. GENERATED FROM PYTHON SOURCE LINES 374-376 .. code-block:: Python results = search.search(evaluator, max_evals=30) .. rst-class:: sphx-glr-script-out .. code-block:: none 0%| | 0/30 [00:00
p:batch_size p:learning_rate p:num_epochs objective job_id job_status m:timestamp_submit m:timestamp_gather
0 245 1.631048 9 0.385000 0 DONE 0.806100 3.504088
1 56 1.112981 16 0.601667 1 DONE 3.526653 5.828167
2 23 0.143010 16 0.394524 2 DONE 5.839790 10.173068
3 264 5.976338 17 0.690000 3 DONE 10.184379 11.913512
4 136 1.760105 8 0.425714 4 DONE 11.925383 13.314167
5 82 2.249590 20 0.739524 5 DONE 13.325365 15.923063
6 53 0.913211 14 0.563095 6 DONE 15.934397 18.385043
7 283 3.305364 19 0.590714 7 DONE 18.615469 20.076355
8 17 2.536505 20 0.816429 8 DONE 20.309002 26.365440
9 11 2.086957 20 0.815714 9 DONE 26.609885 35.309791
10 9 8.421344 20 0.820238 10 DONE 35.544787 45.785644
11 16 0.911853 20 0.746905 11 DONE 46.023974 52.367163
12 9 4.070022 20 0.803571 12 DONE 52.705372 63.077662
13 11 2.122347 20 0.809286 13 DONE 63.321248 71.979538
14 16 6.979859 20 0.807143 14 DONE 72.221648 78.707475
15 15 1.276222 20 0.770476 15 DONE 78.946452 85.686414
16 9 1.361241 20 0.800476 16 DONE 85.925403 96.158468
17 11 2.886584 20 0.807619 17 DONE 96.397284 105.057524
18 36 0.523676 20 0.582857 18 DONE 105.291716 108.786150
19 10 0.765039 20 0.746905 19 DONE 109.025325 118.416570
20 22 7.267154 20 0.818095 20 DONE 118.738970 123.678590
21 20 9.786512 20 0.806190 21 DONE 123.911904 129.223925
22 19 8.426698 20 0.805714 22 DONE 129.460490 135.063884
23 17 6.601954 20 0.816190 23 DONE 135.300321 141.337653
24 14 8.959608 20 0.798571 24 DONE 141.570983 148.691888
25 9 8.727873 20 0.815952 25 DONE 148.930730 159.216285
26 19 6.263893 20 0.799762 26 DONE 159.453678 165.011176
27 21 8.539551 20 0.814524 27 DONE 165.337578 170.484569
28 22 1.172272 20 0.751429 28 DONE 170.722989 175.696918
29 30 5.805169 20 0.796190 29 DONE 175.931295 179.869023


.. GENERATED FROM PYTHON SOURCE LINES 388-392 Evaluate the best configuration ------------------------------- Now that the search is over, let us print the best configuration found during this run and evaluate it on the full training dataset. .. GENERATED FROM PYTHON SOURCE LINES 394-404 .. code-block:: Python i_max = results.objective.argmax() best_config = results.iloc[i_max][:-3].to_dict() best_config = {k[2:]: v for k, v in best_config.items() if k.startswith("p:")} print(f"The default configuration has an accuracy of {objective_default:.3f}. \n" f"The best configuration found by DeepHyper has an accuracy {results['objective'].iloc[i_max]:.3f}, \n" f"finished after {results['m:timestamp_gather'].iloc[i_max]:.2f} secondes of search.\n") print(json.dumps(best_config, indent=4)) .. rst-class:: sphx-glr-script-out .. code-block:: none The default configuration has an accuracy of 0.863. The best configuration found by DeepHyper has an accuracy 0.820, finished after 45.79 secondes of search. { "batch_size": 9, "learning_rate": 8.421343891942513, "num_epochs": 20 } .. GENERATED FROM PYTHON SOURCE LINES 405-407 .. code-block:: Python objective_best = perf_run(best_config) print(f"Accuracy Best Configuration: {objective_best:.3f}") .. rst-class:: sphx-glr-script-out .. code-block:: none Accuracy Best Configuration: 0.807 .. rst-class:: sphx-glr-timing **Total running time of the script:** (3 minutes 47.673 seconds) .. _sphx_glr_download_examples_examples_hpo_plot_hpo_text_classification.py: .. only:: html .. container:: sphx-glr-footer sphx-glr-footer-example .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: plot_hpo_text_classification.ipynb ` .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: plot_hpo_text_classification.py ` .. container:: sphx-glr-download sphx-glr-download-zip :download:`Download zipped: plot_hpo_text_classification.zip ` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_