"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"from torchtext.datasets import AG_NEWS\n",
"\n",
"def load_data(train_ratio):\n",
" train_iter, test_iter = AG_NEWS()\n",
" train_dataset = to_map_style_dataset(train_iter)\n",
" test_dataset = to_map_style_dataset(test_iter)\n",
" num_train = int(len(train_dataset) * train_ratio)\n",
" split_train, split_valid = \\\n",
" random_split(train_dataset, [num_train, len(train_dataset) - num_train])\n",
"\n",
" return split_train, split_valid, test_dataset"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Preprocessing pipelines and Batch generation\n",
"\n",
"Here is an example for typical NLP data processing with tokenizer and vocabulary. The first step is to build a vocabulary with the raw training dataset. Here we use built in\n",
"factory function `build_vocab_from_iterator` which accepts iterator that yield list or iterator of tokens. Users can also pass any special symbols to be added to the\n",
"vocabulary.\n",
"\n",
"The vocabulary block converts a list of tokens into integers.\n",
"\n",
"```\n",
"vocab(['here', 'is', 'an', 'example'])\n",
">>> [475, 21, 30, 5286]\n",
"```\n",
"\n",
"The text pipeline converts a text string into a list of integers based on the lookup table defined in the vocabulary. The label pipeline converts the label into integers. For example,\n",
"\n",
"```\n",
"text_pipeline('here is the an example')\n",
">>> [475, 21, 2, 30, 5286]\n",
"label_pipeline('10')\n",
">>> 9\n",
"```"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/Users/romainegele/miniforge3/envs/dh-env-test/lib/python3.9/site-packages/torch/utils/data/datapipes/utils/common.py:24: UserWarning: Lambda function is not supported for pickle, please use regular python function or functools.partial instead.\n",
" warnings.warn(\n"
]
}
],
"source": [
"train_iter = AG_NEWS(split='train')\n",
"num_class = 4\n",
"\n",
"tokenizer = get_tokenizer('basic_english')\n",
"\n",
"def yield_tokens(data_iter):\n",
" for _, text in data_iter:\n",
" yield tokenizer(text)\n",
"\n",
"vocab = build_vocab_from_iterator(yield_tokens(train_iter), specials=[\"
\"])\n",
"vocab.set_default_index(vocab[\"\"])\n",
"vocab_size = len(vocab)\n",
"\n",
"text_pipeline = lambda x: vocab(tokenizer(x))\n",
"label_pipeline = lambda x: int(x) - 1\n",
"\n",
"\n",
"def collate_batch(batch, device):\n",
" label_list, text_list, offsets = [], [], [0]\n",
" for (_label, _text) in batch:\n",
" label_list.append(label_pipeline(_label))\n",
" processed_text = torch.tensor(text_pipeline(_text), dtype=torch.int64)\n",
" text_list.append(processed_text)\n",
" offsets.append(processed_text.size(0))\n",
" label_list = torch.tensor(label_list, dtype=torch.int64)\n",
" offsets = torch.tensor(offsets[:-1]).cumsum(dim=0)\n",
" text_list = torch.cat(text_list)\n",
" return label_list.to(device), text_list.to(device), offsets.to(device)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
" \n",
"Note \n",
"
\n",
"\n",
"The `collate_fn` function works on a batch of samples generated from `DataLoader`. The input to `collate_fn` is a batch of data with the batch size in `DataLoader`, and `collate_fn` processes them according to the data processing pipelines declared previously.\n",
" \n",
"
\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Define the model\n",
"\n",
"The model is composed of the [nn.EmbeddingBag](https://pytorch.org/docs/stable/nn.html?highlight=embeddingbag#torch.nn.EmbeddingBag) layer plus a linear layer for the classification purpose."
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"class TextClassificationModel(nn.Module):\n",
"\n",
" def __init__(self, vocab_size, embed_dim, num_class):\n",
" super(TextClassificationModel, self).__init__()\n",
" self.embedding = nn.EmbeddingBag(vocab_size, embed_dim, sparse=True)\n",
" self.fc = nn.Linear(embed_dim, num_class)\n",
" self.init_weights()\n",
"\n",
" def init_weights(self):\n",
" initrange = 0.5\n",
" self.embedding.weight.data.uniform_(-initrange, initrange)\n",
" self.fc.weight.data.uniform_(-initrange, initrange)\n",
" self.fc.bias.data.zero_()\n",
"\n",
" def forward(self, text, offsets):\n",
" embedded = self.embedding(text, offsets)\n",
" return self.fc(embedded)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Define functions to train the model and evaluate results.\n",
"---------------------------------------------------------\n",
"\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"def train(model, criterion, optimizer, dataloader):\n",
" model.train()\n",
"\n",
" for _, (label, text, offsets) in enumerate(dataloader):\n",
" optimizer.zero_grad()\n",
" predicted_label = model(text, offsets)\n",
" loss = criterion(predicted_label, label)\n",
" loss.backward()\n",
" torch.nn.utils.clip_grad_norm_(model.parameters(), 0.1)\n",
" optimizer.step()\n",
"\n",
"def evaluate(model, dataloader):\n",
" model.eval()\n",
" total_acc, total_count = 0, 0\n",
"\n",
" with torch.no_grad():\n",
" for _, (label, text, offsets) in enumerate(dataloader):\n",
" predicted_label = model(text, offsets)\n",
" total_acc += (predicted_label.argmax(1) == label).sum().item()\n",
" total_count += label.size(0)\n",
" return total_acc/total_count"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Define the run-function\n",
"\n",
"The run-function defines how the objective that we want to maximize is computed. It takes a `config` dictionary as input and often returns a scalar value that we want to maximize. The `config` contains a sample value of hyperparameters that we want to tune. In this example we will search for:\n",
"\n",
"* `num_epochs` (default value: `10`)\n",
"* `batch_size` (default value: `64`)\n",
"* `learning_rate` (default value: `5`)\n",
"\n",
"A hyperparameter value can be acessed easily in the dictionary through the corresponding key, for example `config[\"units\"]`."
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"def get_run(train_ratio=0.95):\n",
" def run(config: dict):\n",
" device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
"\n",
" embed_dim = 64\n",
" \n",
" collate_fn = partial(collate_batch, device=device)\n",
" split_train, split_valid, _ = load_data(train_ratio)\n",
" train_dataloader = DataLoader(split_train, batch_size=int(config[\"batch_size\"]),\n",
" shuffle=True, collate_fn=collate_fn)\n",
" valid_dataloader = DataLoader(split_valid, batch_size=int(config[\"batch_size\"]),\n",
" shuffle=True, collate_fn=collate_fn)\n",
"\n",
" model = TextClassificationModel(vocab_size, int(embed_dim), num_class).to(device)\n",
" \n",
" criterion = torch.nn.CrossEntropyLoss()\n",
" optimizer = torch.optim.SGD(model.parameters(), lr=config[\"learning_rate\"])\n",
"\n",
" for _ in range(1, int(config[\"num_epochs\"]) + 1):\n",
" train(model, criterion, optimizer, train_dataloader)\n",
" \n",
" accu_test = evaluate(model, valid_dataloader)\n",
" return accu_test\n",
" return run"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We create two versions of `run`, one quicker to evaluate for the seacrh, with a small training dataset, and another one, for performance evaluation, which uses a normal training/validation ratio."
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [],
"source": [
"quick_run = get_run(train_ratio=0.3)\n",
"perf_run = get_run(train_ratio=0.95)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
" \n",
"Note \n",
"
\n",
"\n",
"The objective maximised by DeepHyper is the scalar value returned by the `run`-function.\n",
" \n",
"
\n",
"\n",
"In this tutorial it corresponds to the validation accuracy of the model after training."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Define the Hyperparameter optimization problem\n",
"\n",
"Hyperparameter ranges are defined using the following syntax:\n",
"\n",
"* Discrete integer ranges are generated from a tuple `(lower: int, upper: int)`\n",
"* Continuous prarameters are generated from a tuple `(lower: float, upper: float)`\n",
"* Categorical or nonordinal hyperparameter ranges can be given as a list of possible values `[val1, val2, ...]`\n",
"\n",
"We provide the default configuration of hyperparameters as a starting point of the problem."
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"Configuration space object:\n",
" Hyperparameters:\n",
" batch_size, Type: UniformInteger, Range: [8, 512], Default: 64, on log-scale\n",
" learning_rate, Type: UniformFloat, Range: [0.1, 10.0], Default: 5.0, on log-scale\n",
" num_epochs, Type: UniformInteger, Range: [5, 20], Default: 10"
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from deephyper.problem import HpProblem\n",
"\n",
"problem = HpProblem()\n",
"\n",
"# Discrete hyperparameter (sampled with uniform prior)\n",
"problem.add_hyperparameter((5, 20), \"num_epochs\", default_value=10)\n",
"\n",
"# Discrete and Real hyperparameters (sampled with log-uniform)\n",
"problem.add_hyperparameter((8, 512, \"log-uniform\"), \"batch_size\", default_value=64)\n",
"problem.add_hyperparameter((0.1, 10, \"log-uniform\"), \"learning_rate\", default_value=5)\n",
"\n",
"problem"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Evaluate a default configuration\n",
"\n",
"We evaluate the performance of the default set of hyperparameters provided in the Pytorch tutorial."
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/Users/romainegele/miniforge3/envs/dh-env-test/lib/python3.9/site-packages/torch/utils/data/datapipes/utils/common.py:24: UserWarning: Lambda function is not supported for pickle, please use regular python function or functools.partial instead.\n",
" warnings.warn(\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Accuracy Default Configuration: 0.902\n"
]
}
],
"source": [
"# We launch the Ray run-time and execute the `run` function\n",
"# with the default configuration\n",
"\n",
"if is_gpu_available:\n",
" if not(ray.is_initialized()):\n",
" ray.init(num_cpus=n_gpus, num_gpus=n_gpus, log_to_driver=False)\n",
" \n",
" run_default = ray.remote(num_cpus=1, num_gpus=1)(perf_run)\n",
" objective_default = ray.get(run_default.remote(problem.default_configuration))\n",
"else:\n",
" if not(ray.is_initialized()):\n",
" ray.init(num_cpus=1, log_to_driver=False)\n",
" run_default = perf_run\n",
" objective_default = run_default(problem.default_configuration)\n",
"\n",
"print(f\"Accuracy Default Configuration: {objective_default:.3f}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Define the evaluator object\n",
"\n",
"The `Evaluator` object allows to change the parallelization backend used by DeepHyper. \n",
"It is a standalone object which schedules the execution of remote tasks. All evaluators needs a `run_function` to be instantiated. \n",
"Then a keyword `method` defines the backend (e.g., `\"ray\"`) and the `method_kwargs` corresponds to keyword arguments of this chosen `method`.\n",
"\n",
"```python\n",
"evaluator = Evaluator.create(run_function, method, method_kwargs)\n",
"```\n",
"\n",
"Once created the `evaluator.num_workers` gives access to the number of available parallel workers.\n",
"\n",
"Finally, to submit and collect tasks to the evaluator one just needs to use the following interface:\n",
"\n",
"```python\n",
"configs = [...]\n",
"evaluator.submit(configs)\n",
"...\n",
"tasks_done = evaluator.get(\"BATCH\", size=1) # For asynchronous\n",
"tasks_done = evaluator.get(\"ALL\") # For batch synchronous\n",
"```\n",
"\n",
"\n",
"\n",
"Warning\n",
"\n",
"Each `Evaluator` saves its own state, therefore it is crucial to create a new evaluator when launching a fresh search.\n",
" \n",
"
\n"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Created new evaluator with 1 worker and config: {'num_cpus': 1, 'num_cpus_per_task': 1, 'callbacks': []}\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"/Users/romainegele/Documents/Argonne/deephyper/deephyper/evaluator/_evaluator.py:99: UserWarning: Applying nest-asyncio patch for IPython Shell!\n",
" warnings.warn(\n"
]
}
],
"source": [
"from deephyper.evaluator import Evaluator\n",
"from deephyper.evaluator.callback import TqdmCallback\n",
"\n",
"def get_evaluator(run_function):\n",
" # Default arguments for Ray: 1 worker and 1 worker per evaluation\n",
" method_kwargs = {\n",
" \"num_cpus\": 1, \n",
" \"num_cpus_per_task\": 1,\n",
" \"callbacks\": [TqdmCallback()]\n",
" }\n",
"\n",
" # If GPU devices are detected then it will create 'n_gpus' workers\n",
" # and use 1 worker for each evaluation\n",
" if is_gpu_available:\n",
" method_kwargs[\"num_cpus\"] = n_gpus\n",
" method_kwargs[\"num_gpus\"] = n_gpus\n",
" method_kwargs[\"num_cpus_per_task\"] = 1\n",
" method_kwargs[\"num_gpus_per_task\"] = 1\n",
"\n",
" evaluator = Evaluator.create(\n",
" run_function, \n",
" method=\"ray\", \n",
" method_kwargs=method_kwargs\n",
" )\n",
" print(f\"Created new evaluator with {evaluator.num_workers} worker{'s' if evaluator.num_workers > 1 else ''} and config: {method_kwargs}\", )\n",
" \n",
" return evaluator\n",
"\n",
"evaluator_1 = get_evaluator(quick_run)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Define and run the Centralized Bayesian Optimization search (CBO)\n",
"\n",
"We create the CBO using the `problem` and `evaluator` defined above."
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [],
"source": [
"from deephyper.search.hps import CBO\n",
"# Uncomment the following line to show the arguments of CBO.\n",
"# CBO?"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [],
"source": [
"# Instanciate the search with the problem and a specific evaluator\n",
"search = CBO(problem, evaluator_1)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
" \n",
"
Note\n",
" \n",
"All DeepHyper's search algorithm have two stopping criteria:\n",
"
\n",
" -
`max_evals (int)`: Defines the maximum number of evaluations that we want to perform. Default to -1 for an infinite number. \n",
" -
`timeout (int)`: Defines a time budget (in seconds) before stopping the search. Default to None for an infinite time budget. \n",
"
\n",
" \n",
"
\n",
"\n",
"
\n",
" \n",
" \n",
" | \n",
" batch_size | \n",
" learning_rate | \n",
" num_epochs | \n",
" job_id | \n",
" objective | \n",
" timestamp_submit | \n",
" timestamp_gather | \n",
"
\n",
" \n",
" \n",
" \n",
" | 0 | \n",
" 13 | \n",
" 0.396163 | \n",
" 10 | \n",
" 1 | \n",
" 0.876310 | \n",
" 0.167126 | \n",
" 21.861172 | \n",
"
\n",
" \n",
" | 1 | \n",
" 23 | \n",
" 0.176352 | \n",
" 13 | \n",
" 2 | \n",
" 0.836810 | \n",
" 21.893403 | \n",
" 46.585527 | \n",
"
\n",
" \n",
" | 2 | \n",
" 85 | \n",
" 0.615317 | \n",
" 9 | \n",
" 3 | \n",
" 0.849250 | \n",
" 46.603550 | \n",
" 62.330578 | \n",
"
\n",
" \n",
" | 3 | \n",
" 9 | \n",
" 4.647854 | \n",
" 15 | \n",
" 4 | \n",
" 0.888524 | \n",
" 62.348272 | \n",
" 96.357171 | \n",
"
\n",
" \n",
" | 4 | \n",
" 22 | \n",
" 1.862895 | \n",
" 7 | \n",
" 5 | \n",
" 0.892131 | \n",
" 96.375062 | \n",
" 111.058422 | \n",
"
\n",
" \n",
" | 5 | \n",
" 13 | \n",
" 8.390420 | \n",
" 7 | \n",
" 6 | \n",
" 0.892155 | \n",
" 111.077491 | \n",
" 127.247439 | \n",
"
\n",
" \n",
" | 6 | \n",
" 56 | \n",
" 5.709700 | \n",
" 7 | \n",
" 7 | \n",
" 0.888726 | \n",
" 127.265297 | \n",
" 140.431804 | \n",
"
\n",
" \n",
" | 7 | \n",
" 47 | \n",
" 1.073937 | \n",
" 17 | \n",
" 8 | \n",
" 0.891607 | \n",
" 140.449749 | \n",
" 168.683609 | \n",
"
\n",
" \n",
" | 8 | \n",
" 116 | \n",
" 0.356077 | \n",
" 8 | \n",
" 9 | \n",
" 0.757940 | \n",
" 168.787206 | \n",
" 183.260858 | \n",
"
\n",
" \n",
" | 9 | \n",
" 39 | \n",
" 3.127017 | \n",
" 7 | \n",
" 10 | \n",
" 0.891643 | \n",
" 183.278615 | \n",
" 196.925297 | \n",
"
\n",
" \n",
" | 10 | \n",
" 512 | \n",
" 0.348708 | \n",
" 6 | \n",
" 11 | \n",
" 0.478179 | \n",
" 197.162700 | \n",
" 207.922961 | \n",
"
\n",
" \n",
" | 11 | \n",
" 512 | \n",
" 3.726319 | \n",
" 17 | \n",
" 12 | \n",
" 0.879036 | \n",
" 208.166723 | \n",
" 232.997138 | \n",
"
\n",
" \n",
" | 12 | \n",
" 512 | \n",
" 8.765370 | \n",
" 6 | \n",
" 13 | \n",
" 0.864667 | \n",
" 233.320723 | \n",
" 244.213859 | \n",
"
\n",
" \n",
" | 13 | \n",
" 512 | \n",
" 1.910815 | \n",
" 5 | \n",
" 14 | \n",
" 0.737810 | \n",
" 244.466962 | \n",
" 254.037378 | \n",
"
\n",
" \n",
" | 14 | \n",
" 512 | \n",
" 0.313972 | \n",
" 6 | \n",
" 15 | \n",
" 0.471179 | \n",
" 254.293386 | \n",
" 265.038085 | \n",
"
\n",
" \n",
" | 15 | \n",
" 512 | \n",
" 0.390254 | \n",
" 17 | \n",
" 16 | \n",
" 0.659024 | \n",
" 265.296111 | \n",
" 291.446844 | \n",
"
\n",
" \n",
" | 16 | \n",
" 512 | \n",
" 0.314117 | \n",
" 6 | \n",
" 17 | \n",
" 0.463964 | \n",
" 291.793700 | \n",
" 302.993824 | \n",
"
\n",
" \n",
" | 17 | \n",
" 512 | \n",
" 0.406197 | \n",
" 6 | \n",
" 18 | \n",
" 0.495940 | \n",
" 303.264247 | \n",
" 314.411603 | \n",
"
\n",
" \n",
" | 18 | \n",
" 512 | \n",
" 0.320737 | \n",
" 6 | \n",
" 19 | \n",
" 0.476452 | \n",
" 314.679307 | \n",
" 325.618635 | \n",
"
\n",
" \n",
" | 19 | \n",
" 512 | \n",
" 0.310132 | \n",
" 6 | \n",
" 20 | \n",
" 0.455762 | \n",
" 325.959387 | \n",
" 336.946107 | \n",
"
\n",
" \n",
" | 20 | \n",
" 512 | \n",
" 0.277178 | \n",
" 6 | \n",
" 21 | \n",
" 0.438250 | \n",
" 337.212328 | \n",
" 348.157166 | \n",
"
\n",
" \n",
" | 21 | \n",
" 512 | \n",
" 0.276917 | \n",
" 6 | \n",
" 22 | \n",
" 0.449714 | \n",
" 348.422956 | \n",
" 359.220447 | \n",
"
\n",
" \n",
" | 22 | \n",
" 512 | \n",
" 0.273978 | \n",
" 6 | \n",
" 23 | \n",
" 0.453071 | \n",
" 359.485135 | \n",
" 370.697273 | \n",
"
\n",
" \n",
" | 23 | \n",
" 512 | \n",
" 0.298666 | \n",
" 6 | \n",
" 24 | \n",
" 0.450143 | \n",
" 371.048602 | \n",
" 382.304554 | \n",
"
\n",
" \n",
" | 24 | \n",
" 512 | \n",
" 0.280337 | \n",
" 6 | \n",
" 25 | \n",
" 0.463952 | \n",
" 382.571858 | \n",
" 393.747633 | \n",
"
\n",
" \n",
" | 25 | \n",
" 512 | \n",
" 0.292463 | \n",
" 6 | \n",
" 26 | \n",
" 0.469250 | \n",
" 394.017767 | \n",
" 405.152517 | \n",
"
\n",
" \n",
" | 26 | \n",
" 512 | \n",
" 0.273828 | \n",
" 6 | \n",
" 27 | \n",
" 0.459488 | \n",
" 405.424290 | \n",
" 416.641872 | \n",
"
\n",
" \n",
" | 27 | \n",
" 512 | \n",
" 0.306901 | \n",
" 6 | \n",
" 28 | \n",
" 0.461964 | \n",
" 416.986107 | \n",
" 428.063347 | \n",
"
\n",
" \n",
" | 28 | \n",
" 512 | \n",
" 0.281089 | \n",
" 6 | \n",
" 29 | \n",
" 0.458131 | \n",
" 428.335923 | \n",
" 439.447637 | \n",
"
\n",
" \n",
" | 29 | \n",
" 512 | \n",
" 0.278833 | \n",
" 6 | \n",
" 30 | \n",
" 0.436702 | \n",
" 439.720846 | \n",
" 450.642993 | \n",
"
\n",
" \n",
"
\n",
"