deephyper.problem

deephyper.problem

This module provides tools to define hyperparameter and neural architecture search problems. Some features of this module are based on the ConfigSpace project.

class deephyper.problem.HpProblem(config_space=None)[source]

Bases: object

Class to define an hyperparameter problem.

>>> from deephyper.problem import HpProblem
>>> problem = HpProblem()
Parameters

config_space (ConfigurationSpace, optional) – In case the HpProblem is defined from a ConfigurationSpace.

add_condition(condition)[source]

Add a condition to the HpProblem.

>>> from deephyper.problem import HpProblem
>>> import ConfigSpace as cs
>>> problem = HpProblem()
>>> x = problem.add_hyperparameter((0.0, 10.0), "x")
>>> y = problem.add_hyperparameter((1e-4, 1.0), "y")
>>> problem.add_condition(cs.LessThanCondition(y, x, 1.0))
Parameters

condition – A ConfigSpace condition.

add_conditions(conditions: list) → None[source]

Add a list of condition to the HpProblem.

Parameters

conditions (list) – A list of ConfigSpace conditions.

add_forbidden_clause(clause)[source]

Add a forbidden clause to the HpProblem.

For example if we want to optimize \(\frac{1}{x}\) where \(x\) cannot be equal to 0:

>>> from deephyper.problem import HpProblem
>>> import ConfigSpace as cs
>>> problem = HpProblem()
>>> x = problem.add_hyperparameter((0.0, 10.0), "x")
>>> problem.add_forbidden_clause(cs.ForbiddenEqualsClause(x, 0.0))
Parameters

clause – a ConfigSpace forbidden clause.

add_hyperparameter(value, name: Optional[str] = None, default_value=None) → ConfigSpace.hyperparameters.Hyperparameter[source]

Add an hyperparameter to the HpProblem.

Hyperparameters can be added to a HpProblem with a short syntax:

>>> problem.add_hyperparameter((0, 10), "discrete", default_value=5)
>>> problem.add_hyperparameter((0.0, 10.0), "real", default_value=5.0)
>>> problem.add_hyperparameter([0, 10], "categorical", default_value=0)

Sampling distributions can be provided:

>>> problem.add_hyperparameter((0.0, 10.0, "log-uniform"), "real", default_value=5.0)

It is also possible to use ConfigSpace Hyperparameters:

>>> import ConfigSpace.hyperparameters as csh
>>> csh_hp = csh.UniformIntegerHyperparameter(
...     name='uni_int', lower=10, upper=100, log=False)
>>> problem.add_hyperparameter(csh_hp)
Parameters

  • value (tuple or list or ConfigSpace.Hyperparameter) – a valid hyperparametr description.

  • name (str) – The name of the hyperparameter to add.

  • default_value (float or int or str) – A default value for the corresponding hyperparameter.

Returns

a ConfigSpace Hyperparameter object corresponding to the (value, name, default_value).

Return type

ConfigSpace.Hyperparameter

add_hyperparameters(hp_list)[source]

Add a list of hyperparameters. It can be useful when a list of ConfigSpace.Hyperparameter are defined and we need to add them to the HpProblem.

Parameters

hp_list (ConfigSpace.Hyperparameter) – a list of ConfigSpace hyperparameters.

Returns

The list of added hyperparameters.

Return type

list

add_starting_point(**parameters)[source]

Add starting points to the HpProblem. It is useful when a good-baseline is known to help initialize the search at a given location of the search space.

>>> from deephyper.problem import HpProblem
>>> problem = HpProblem()
>>> x = problem.add_hyperparameter((0.0, 10.0), "x")
>>> problem.add_starting_point(x=1.0)
property space

The wrapped ConfigSpace object.

property starting_point

Starting point(s) of the search space.

Returns

list of starting points where each point is a list of values. Values are indexed in the same order as the order of creation of space’s dimensions.

Return type

list

property starting_point_asdict

Starting point(s) of the search space.

Returns

list of starting points where each point is a dict of values. Each key are correspnding to dimensions of the space.

Return type

list(dict)

class deephyper.problem.NaProblem[source]

Bases: object

A Neural Architecture Problem specification for Neural Architecture Search.

>>> from deephyper.problem import NaProblem
>>> from deephyper.benchmark.nas.linearReg.load_data import load_data
>>> from deephyper.nas.preprocessing import minmaxstdscaler
>>> from deepspace.tabular import OneLayerSpace
>>> def create_search_space(input_shape, output_shape, **kwargs):
...     return OneLayerSpace()(input_shape, output_shape, **kwargs)
>>> Problem = NaProblem()
>>> Problem.load_data(load_data)
>>> Problem.preprocessing(minmaxstdscaler)
>>> Problem.search_space(create_search_space)
>>> Problem.hyperparameters(
...     batch_size=100,
...     learning_rate=0.1,
...     optimizer='adam',
...     num_epochs=10,
...     callbacks=dict(
...        EarlyStopping=dict(
...             monitor='val_r2',
...             mode='max',
...             verbose=0,
...             patience=5
...         )
...     )
... )
>>> Problem.loss('mse')
>>> Problem.metrics(['r2'])
>>> Problem.objective('val_r2__last')
add_hyperparameter(value, name: Optional[str] = None, default_value=None) → ConfigSpace.hyperparameters.Hyperparameter[source]

Add hyperparameters to search the neural architecture search problem.

>>> Problem.hyperparameters(
...     batch_size=problem.add_hyperparameter((32, 256), "batch_size")
...     )
Parameters

  • value – a hyperparameter description.

  • name – a name of the defined hyperparameter, the same as the current key.

  • default_value (Optional) – a default value of the hyperparameter.

Returns

the defined hyperparameter.

Return type

Hyperparameter

build_search_space(seed=None)[source]

Build and return a search space object using the infered data shapes after loading data.

Returns

A search space instance.

Return type

KSearchSpace

extract_hp_values(config)[source]

Extract the value of hyperparameters present in config based on the defined hyperparameters in the current NaProblem

gen_config(arch_seq: list, hp_values: list) → dict[source]

Generate a dict configuration from the arch_seq and hp_values passed.

Parameters

  • arch_seq (list) – a valid embedding of a neural network described by the search space of the current NaProblem.

  • hp_values (list) – a valid list of hyperparameters corresponding to the defined hyperparameters of the current NaProblem.

get_keras_model(arch_seq: list) → keras.engine.training.Model[source]

Get a keras model object from a set of decisions in the current search space. :param arch_seq: a list of int of floats describing a choice of operations for the search space defined in the current problem. :type arch_seq: list

hyperparameters(**kwargs)[source]

Define hyperparameters used to evaluate generated architectures.

Hyperparameters can be defined such as:

Problem.hyperparameters(
    batch_size=256,
    learning_rate=0.01,
    optimizer="adam",
    num_epochs=20,
    verbose=0,
    callbacks=dict(...),
)
load_data(func: callable, **kwargs)[source]

Define the function loading the data. .. code-block:: python

Problem.load_data(load_data, load_data_kwargs)

This load_data callable can follow two different interfaces: Numpy arrays or generators.

  1. Numpy arrays:

In the case of Numpy arrays, the callable passed to Problem.load_data(...) has to return the following tuple: (X_train, y_train), (X_valid, y_valid). In the most simple case where the model takes a single input, each of these elements is a Numpy array. Generally, X_train and y_train have to be of the same length (i.e., same array.shape[0]) which is also the case for X_valid and y_valid. Similarly, the shape of the elements of X_train and X_valid which is also the case for y_train and y_valid. An example load_data function can be

import numpy as np

def load_data(N=100):
    X = np.zeros((N, 1))
    y = np.zeros((N,1))
    return (X, y), (X, y)

It is also possible for the model to take several inputs. In fact, experimentaly it can be notices that separating some inputs with different inputs can significantly help the learning of the model. Also, sometimes different inputs may be of the “types” for example two molecular fingerprints. In this case, it can be very interesting to share the weights of the model to process these two inputs. In the case of multi-inputs models the load_data function will also return (X_train, y_train), (X_valid, y_valid) bu where X_train and X_valid are two lists of Numpy arrays. For example, the following is correct:

import numpy as np

def load_data(N=100):
    X = np.zeros((N, 1))
    y = np.zeros((N,1))
    return ([X, X], y), ([X, X], y)

  1. Generators:

Returning generators with a single input:

def load_data(N=100):

    tX, ty = np.zeros((N,1)), np.zeros((N,1))
    vX, vy = np.zeros((N,1)), np.zeros((N,1))

    def train_gen():
        for x, y in zip(tX, ty):
            yield ({"input_0": x}, y)

    def valid_gen():
        for x, y in zip(vX, vy):
            yield ({"input_0": x}, y)

    res = {
        "train_gen": train_gen,
        "train_size": N,
        "valid_gen": valid_gen,
        "valid_size": N,
        "types": ({"input_0": tf.float64}, tf.float64),
        "shapes": ({"input_0": (1, )}, (1, ))
        }
    return res

Returning generators with multiple inputs:

def load_data(N=100):

    tX0, tX1, ty = np.zeros((N,1)), np.zeros((N,1)), np.zeros((N,1)),
    vX0, vX1, vy = np.zeros((N,1)), np.zeros((N,1)), np.zeros((N,1)),

    def train_gen():
        for x0, x1, y in zip(tX0, tX1, ty):
            yield ({
                "input_0": x0,
                "input_1": x1
                }, y)

    def valid_gen():
        for x0, x1, y in zip(vX0, vX1, vy):
            yield ({
                "input_0": x0,
                "input_1": x1
            }, y)

    res = {
        "train_gen": train_gen,
        "train_size": N,
        "valid_gen": valid_gen,
        "valid_size": N,
        "types": ({"input_0": tf.float64, "input_1": tf.float64}, tf.float64),
        "shapes": ({"input_0": (5, ), "input_1": (5, )}, (1, ))
        }

    return res
Parameters

func (callable) – the load data function.

loss(loss, loss_weights=None, class_weights=None)[source]

Define the loss used to train generated architectures.

It can be a str corresponding to a Keras loss function:

problem.loss("categorical_crossentropy")

A custom loss function can also be defined:

def NLL(y, rv_y):
    return -rv_y.log_prob(y)

problem.loss(NLL)

The loss can be automatically searched:

problem.loss(
    problem.add_hyperparameter(
        ["mae", "mse", "huber_loss", "log_cosh", "mape", "msle"], "loss"
    )
)

It is possible to define a different loss for each output:

problem.loss(
    loss={"output_0": "mse", "output_1": "mse"},
    loss_weights={"output_0": 0.0, "output_1": 1.0},
)
Parameters

  • loss (str or callable orlist) – a string indicating a specific loss function.

  • loss_weights (list) – Optional.

  • class_weights (dict) – Optional.

metrics(metrics)[source]

Define a list of metrics for the training of generated architectures.

A list of metrics can be defined to be monitored or used as an objective. It can be a keyword or a callable. For example, if it is a keyword:

problem.metrics(["acc"])

In case you need multiple metrics:

problem.metrics["mae", "mse"]

In case you want to use a custom metric:

def sparse_perplexity(y_true, y_pred):
    cross_entropy = tf.keras.losses.sparse_categorical_crossentropy(y_true, y_pred)
    perplexity = tf.pow(2.0, cross_entropy)
    return perplexity

problem.metrics([sparse_perplexity])
Parameters

metrics (list(str or callable) or dict) – If str the metric should be defined in Keras or in DeepHyper. If callable it should take 2 arguments (y_pred, y_true) which are a prediction and a true value respectively.

objective(objective)[source]

Define the objective you want to maximize for the search.

If you want to use the validation accuracy at the last epoch:

problem.objective("val_acc")

Note

Be sure to define acc in the problem.metrics(["acc"]).

It can accept some prefix and suffix such as __min, __max, __last:

problem.objective("-val_acc__max")

It can be a callable:

def myobjective(history: dict) -> float:
    return history["val_acc"][-1]

problem.objective(myobjective)
Parameters

objective (str or callable) – The objective will be maximized. If objective is str then it should be either ‘loss’ or a defined metric. You can use the 'val_' prefix when you want to select the objective on the validation set. You can use one of ['__min', '__max', '__last'] which respectively means you want to select the min, max or last value among all epochs. Using ‘__last’ will save a consequent compute time because the evaluation will not compute metrics on validation set for all epochs but the last. If objective is callable it should return a scalar value (i.e. float) and it will take a dict parameter. The dict will contain keys corresponding to loss and metrics such as ['loss', 'val_loss', 'r2', 'val_r2], it will also contains 'n_parameters' which corresponds to the number of trainable parameters of the current model, 'training_time' which corresponds to the time required to train the model, 'predict_time' which corresponds to the time required to make a prediction over the whole validation set. If this callable has a '__last' suffix then the evaluation will only compute validation loss/metrics for the last epoch. If this callable has contains ‘with_pred’ in its name then the dict will have two other keys ['y_pred', 'y_true'] where 'y_pred corresponds to prediction of the model on validation set and 'y_true' corresponds to real prediction.

Raises

WrongProblemObjective – raised when the objective is of a wrong definition.

preprocessing(func: callable)[source]

Define how to preprocess your data.

Parameters

func (callable) – a function which returns a preprocessing scikit-learn Pipeline.

search_space(space_class, **kwargs)[source]

Set a search space for neural architecture search.

Parameters

func (callable) – an object which has to be a callable and which is returning a Structure instance.

Raises

  • SearchSpaceBuilderIsNotCallable – raised when the func parameter is not a callable.

  • SearchSpaceBuilderMissingParameter – raised when either of (input_shape, output_shape) are missing parameters of func.

  • SearchSpaceBuilderMissingDefaultParameter – raised when either of (input_shape, output_shape) are missing a default value.

property space

previous

deephyper.nas.node

next

deephyper.search