deephyper.problem.NaProblem

class deephyper.problem.NaProblem

Bases: object

A Neural Architecture Problem specification for Neural Architecture Search.

>>> from deephyper.problem import NaProblem
>>> from deephyper.nas.preprocessing import minmaxstdscaler
>>> from deepspace.tabular import OneLayerSpace
>>> Problem = NaProblem()
>>> Problem.load_data(load_data)
>>> Problem.preprocessing(minmaxstdscaler)
>>> Problem.search_space(OneLayerSpace)
>>> Problem.hyperparameters(
...     batch_size=100,
...     learning_rate=0.1,
...     optimizer='adam',
...     num_epochs=10,
...     callbacks=dict(
...        EarlyStopping=dict(
...             monitor='val_r2',
...             mode='max',
...             verbose=0,
...             patience=5
...         )
...     )
... )
>>> Problem.loss('mse')
>>> Problem.metrics(['r2'])
>>> Problem.objective('val_r2__last')

Methods

add_hyperparameter	Add hyperparameters to search the neural architecture search problem.
augment	meta private:
build_search_space	Build and return a search space object using the infered data shapes after loading data.
check_objective	meta private:
extract_hp_values	Extract the value of hyperparameters present in config based on the defined hyperparameters in the current NaProblem
gen_config	Generate a dict configuration from the arch_seq and hp_values passed.
get_keras_model	Get a keras model object from a set of decisions in the current search space.
hyperparameters	Define hyperparameters used to evaluate generated architectures.
load_data	Define the function loading the data.
loss	Define the loss used to train generated architectures.
metrics	Define a list of metrics for the training of generated architectures.
objective	Define the objective you want to maximize for the search.
preprocessing	Define how to preprocess your data.
search_space	Set a search space for neural architecture search.

Attributes

default_hp_configuration	The default configuration as a dictionnary.
hyperparameter_names	The list of hyperparameters names.
space

add_hyperparameter(value, name: str = None, default_value=None) → ConfigSpace.hyperparameters.Hyperparameter

Add hyperparameters to search the neural architecture search problem.

>>> Problem.hyperparameters(
...     batch_size=problem.add_hyperparameter((32, 256), "batch_size")
...     )

Parameters:

• value – a hyperparameter description.

• name – a name of the defined hyperparameter, the same as the current key.

• default_value (Optional) – a default value of the hyperparameter.

Returns:

the defined hyperparameter.

Return type:

Hyperparameter

build_search_space(seed=None)

Build and return a search space object using the infered data shapes after loading data.

Returns:

A search space instance.

Return type:

KSearchSpace

property default_hp_configuration

The default configuration as a dictionnary.

extract_hp_values(config)

Extract the value of hyperparameters present in config based on the defined hyperparameters in the current NaProblem

gen_config(arch_seq: list, hp_values: list) → dict

Generate a dict configuration from the arch_seq and hp_values passed.

Parameters:

• arch_seq (list) – a valid embedding of a neural network described by the search space of the current NaProblem.

• hp_values (list) – a valid list of hyperparameters corresponding to the defined hyperparameters of the current NaProblem.

get_keras_model(arch_seq: list) → tensorflow.keras.Model

Get a keras model object from a set of decisions in the current search space. :param arch_seq: a list of int of floats describing a choice of operations for the search space defined in the current problem. :type arch_seq: list

property hyperparameter_names

The list of hyperparameters names.

hyperparameters(**kwargs)

Define hyperparameters used to evaluate generated architectures.

Hyperparameters can be defined such as:

Problem.hyperparameters(
    batch_size=256,
    learning_rate=0.01,
    optimizer="adam",
    num_epochs=20,
    verbose=0,
    callbacks=dict(...),
)

load_data(func: callable, **kwargs)

Define the function loading the data. .. code-block:: python

Problem.load_data(load_data, load_data_kwargs)

This load_data callable can follow two different interfaces: Numpy arrays or generators.

1. Numpy arrays:

In the case of Numpy arrays, the callable passed to Problem.load_data(...) has to return the following tuple: (X_train, y_train), (X_valid, y_valid). In the most simple case where the model takes a single input, each of these elements is a Numpy array. Generally, X_train and y_train have to be of the same length (i.e., same array.shape[0]) which is also the case for X_valid and y_valid. Similarly, the shape of the elements of X_train and X_valid which is also the case for y_train and y_valid. An example load_data function can be

import numpy as np

def load_data(N=100):
    X = np.zeros((N, 1))
    y = np.zeros((N,1))
    return (X, y), (X, y)

It is also possible for the model to take several inputs. In fact, experimentaly it can be notices that separating some inputs with different inputs can significantly help the learning of the model. Also, sometimes different inputs may be of the “types” for example two molecular fingerprints. In this case, it can be very interesting to share the weights of the model to process these two inputs. In the case of multi-inputs models the load_data function will also return (X_train, y_train), (X_valid, y_valid) bu where X_train and X_valid are two lists of Numpy arrays. For example, the following is correct:

import numpy as np

def load_data(N=100):
    X = np.zeros((N, 1))
    y = np.zeros((N,1))
    return ([X, X], y), ([X, X], y)

2. Generators:

Returning generators with a single input:

def load_data(N=100):

    tX, ty = np.zeros((N,1)), np.zeros((N,1))
    vX, vy = np.zeros((N,1)), np.zeros((N,1))

    def train_gen():
        for x, y in zip(tX, ty):
            yield ({"input_0": x}, y)

    def valid_gen():
        for x, y in zip(vX, vy):
            yield ({"input_0": x}, y)

    res = {
        "train_gen": train_gen,
        "train_size": N,
        "valid_gen": valid_gen,
        "valid_size": N,
        "types": ({"input_0": tf.float64}, tf.float64),
        "shapes": ({"input_0": (1, )}, (1, ))
        }
    return res

Returning generators with multiple inputs:

def load_data(N=100):

    tX0, tX1, ty = np.zeros((N,1)), np.zeros((N,1)), np.zeros((N,1)),
    vX0, vX1, vy = np.zeros((N,1)), np.zeros((N,1)), np.zeros((N,1)),

    def train_gen():
        for x0, x1, y in zip(tX0, tX1, ty):
            yield ({
                "input_0": x0,
                "input_1": x1
                }, y)

    def valid_gen():
        for x0, x1, y in zip(vX0, vX1, vy):
            yield ({
                "input_0": x0,
                "input_1": x1
            }, y)

    res = {
        "train_gen": train_gen,
        "train_size": N,
        "valid_gen": valid_gen,
        "valid_size": N,
        "types": ({"input_0": tf.float64, "input_1": tf.float64}, tf.float64),
        "shapes": ({"input_0": (5, ), "input_1": (5, )}, (1, ))
        }

    return res

Parameters:

func (callable) – the load data function.

loss(loss, loss_weights=None, class_weights=None)

Define the loss used to train generated architectures.

It can be a str corresponding to a Keras loss function:

problem.loss("categorical_crossentropy")

A custom loss function can also be defined:

def NLL(y, rv_y):
    return -rv_y.log_prob(y)

problem.loss(NLL)

The loss can be automatically searched:

problem.loss(
    problem.add_hyperparameter(
        ["mae", "mse", "huber_loss", "log_cosh", "mape", "msle"], "loss"
    )
)

It is possible to define a different loss for each output:

problem.loss(
    loss={"output_0": "mse", "output_1": "mse"},
    loss_weights={"output_0": 0.0, "output_1": 1.0},
)

Parameters:

• loss (str or callable orlist) – a string indicating a specific loss function.

• loss_weights (list) – Optional.

• class_weights (dict) – Optional.

metrics(metrics=None)

Define a list of metrics for the training of generated architectures.

A list of metrics can be defined to be monitored or used as an objective. It can be a keyword or a callable. For example, if it is a keyword:

problem.metrics(["acc"])

In case you need multiple metrics:

problem.metrics["mae", "mse"]

In case you want to use a custom metric:

def sparse_perplexity(y_true, y_pred):
    cross_entropy = tf.keras.losses.sparse_categorical_crossentropy(y_true, y_pred)
    perplexity = tf.pow(2.0, cross_entropy)
    return perplexity

problem.metrics([sparse_perplexity])

Parameters:

metrics (list(str or callable) or dict) – If str the metric should be defined in Keras or in DeepHyper. If callable it should take 2 arguments (y_pred, y_true) which are a prediction and a true value respectively.

objective(objective)

Define the objective you want to maximize for the search.

If you want to use the validation accuracy at the last epoch:

problem.objective("val_acc")

Note

Be sure to define acc in the problem.metrics(["acc"]).

It can accept some prefix and suffix such as __min, __max, __last:

problem.objective("-val_acc__max")

It can be a callable:

def myobjective(history: dict) -> float:
    return history["val_acc"][-1]

problem.objective(myobjective)

Parameters:

objective (str or callable) – The objective will be maximized. If objective is str then it should be either ‘loss’ or a defined metric. You can use the 'val_' prefix when you want to select the objective on the validation set. You can use one of ['__min', '__max', '__last'] which respectively means you want to select the min, max or last value among all epochs. Using ‘__last’ will save a consequent compute time because the evaluation will not compute metrics on validation set for all epochs but the last. If objective is callable it should return a scalar value (i.e. float) and it will take a dict parameter. The dict will contain keys corresponding to loss and metrics such as ['loss', 'val_loss', 'r2', 'val_r2], it will also contains 'n_parameters' which corresponds to the number of trainable parameters of the current model, 'training_time' which corresponds to the time required to train the model, 'predict_time' which corresponds to the time required to make a prediction over the whole validation set. If this callable has a '__last' suffix then the evaluation will only compute validation loss/metrics for the last epoch. If this callable has contains ‘with_pred’ in its name then the dict will have two other keys ['y_pred', 'y_true'] where 'y_pred corresponds to prediction of the model on validation set and 'y_true' corresponds to real prediction.

Raises:

WrongProblemObjective – raised when the objective is of a wrong definition.

preprocessing(func: callable)

Define how to preprocess your data.

Parameters:

func (callable) – a function which returns a preprocessing scikit-learn Pipeline.

search_space(space_class, **kwargs)

Set a search space for neural architecture search.

Parameters:

space_class (KSearchSpace) – an object of type KSearchSpace which has to implement the build() method.

Raises:

SearchSpaceBuilderMissingParameter – raised when either of (input_shape, output_shape) are missing parameters of func.