Neural Architecture Search (NAS)

Algorithms

Regularized Evolution

class deephyper.search.nas.regevo.RegularizedEvolution(problem, run, evaluator, population_size=100, sample_size=10, **kwargs)[source]

Regularized evolution.

https://arxiv.org/abs/1802.01548

Parameters
  • problem (str) – Module path to the Problem instance you want to use for the search (e.g. deephyper.benchmark.nas.linearReg.Problem).

  • run (str) – Module path to the run function you want to use for the search (e.g. deephyper.nas.run.quick).

  • evaluator (str) – value in [‘balsam’, ‘subprocess’, ‘processPool’, ‘threadPool’].

  • population_size (int, optional) – the number of individuals to keep in the population. Defaults to 100.

  • sample_size (int, optional) – the number of individuals that should participate in each tournament. Defaults to 10.

copy_mutate_arch(parent_arch: list)dict[source]

# ! Time performance is critical because called sequentialy

Parameters

parent_arch (list(int)) – [description]

Returns

[description]

Return type

dict

If you want to run the aging evolution search:

bash
deephyper nas regevo --problem deephyper.benchmark.nas.mnist1D.Problem

Asynchronous Model Based Neural Architecture Search (AMBNAS)

class deephyper.search.nas.ambs.AMBNeuralArchitectureSearch(problem, run, evaluator, surrogate_model='RF', liar_strategy='cl_max', acq_func='gp_hedge', n_jobs=- 1, **kwargs)[source]

Asynchronous Model-Based Search.

Parameters
  • problem (str) – python attribute import of the NaProblem instance (e.g. mypackage.mymodule.myproblem).

  • run (str) – python attribute import of the run function (e.g. mypackage.mymodule.myrunfunction).

  • evaluator (str) – the name of the evaluator to use.

  • surrogate_model (str, optional) – Choices are [“RF”, “ET”, “GBRT”, “DUMMY”, “GP”]. RF is Random Forest, ET is Extra Trees, GBRT is Gradient Boosting Regression Trees, DUMMY is random, GP is Gaussian process. Defaults to “RF”.

  • liar_strategy (str, optional) – [“cl_max”, “cl_min”, “cl_mean”]. Defaults to “cl_max”.

  • acq_func (str, optional) – Acquisition function, choices are [“gp_hedge”, “LCB”, “EI”, “PI”]. Defaults to “gp_hedge”.

  • n_jobs (int, optional) – Number of parallel jobs to distribute the surrogate model (learner). Defaults to -1, means as many as the number of logical cores.

NAS Benchmarks

Neural Architecture Search models are following a generic workflow. This is why you don’t have to define the function which runs the model yourself if this pipeline corresponds to your needs. Here is the generic training pipeline:
  • load data

  • preprocess data

  • build tensor graph of model (using Structure interface)

  • train model

  • return reward

The basic generic function which is used in our package to run a model for NAS is

deephyper.nas.run.alpha.run(config)[source]

All the items are linked by a problem definition.

Neural Architecture Search Benchmarks deephyper.benchmark.nas

Name

Type

Description

ackleyReg

Regression

Generation of points in N dimensions corresponding to y=f(x) where f is https://www.sfu.ca/~ssurjano/ackley.html

cifar10

Classification

https://www.cs.toronto.edu/~kriz/cifar.html

dixonpriceReg

Regression

https://www.sfu.ca/~ssurjano/dixonpr.html

levyReg

Regression

Generation of points in N dimensions corresponding to y=f(x) where f is https://www.sfu.ca/~ssurjano/levy.html

linearReg

Regression

Generation of points in N dimensions corresponding to y=x

mnistNas

Classification

http://yann.lecun.com/exdb/mnist/

polynome2Reg

Regression

Generation of points in N dimensions corresponding to y=sum(x_i^2)

saddleReg

Regression

https://en.wikipedia.org/wiki/Saddle_point

NasBench-101

NasBench-101 is a database of trained neural networks corresponding to a specific cell-based search space for convolution neural networks (Link to the paper). To use this benchmark you need to install the nasbench package available at github.com/google-research/nasbench.

Then, to download the full data set use:

sh download_full.sh

Or, if you want to Download the dataset only with models evaluated for 108 epochs:

sh download_only108.sh

An example usage with the regularized evolution search if you used download_only108.sh:

python -m deephyper.search.nas.regevo --problem deephyper.benchmark.nas.nasbench101.problem.Problem --evaluator threadPool --run deephyper.benchmark.nas.nasbench101.run_only108.run --max-evals 1000

Or, if you used download_full.sh:

python -m deephyper.search.nas.regevo --problem deephyper.benchmark.nas.nasbench101.problem.Problem --evaluator threadPool --run deephyper.benchmark.nas.nasbench101.run_full.run --max-evals 1000

Environments

Problem

class deephyper.benchmark.problem.NaProblem(seed=None, **kwargs)[source]

A Neural Architecture Problem specification for Neural Architecture Search.

>>> from deephyper.problem import NaProblem
>>> from deephyper.benchmark.nas.linearReg.load_data import load_data
>>> from deephyper.nas.preprocessing import minmaxstdscaler
>>> from deepspace.tabular import OneLayerFactory
>>> def create_search_space(input_shape, output_shape, **kwargs):
...     return OneLayerFactory()(input_shape, output_shape, **kwargs)
>>> Problem = NaProblem()
>>> Problem.load_data(load_data)
>>> Problem.preprocessing(minmaxstdscaler)
>>> Problem.search_space(create_search_space)
>>> Problem.hyperparameters(
...     batch_size=100,
...     learning_rate=0.1,
...     optimizer='adam',
...     num_epochs=10,
...     callbacks=dict(
...        EarlyStopping=dict(
...             monitor='val_r2',
...             mode='max',
...             verbose=0,
...             patience=5
...         )
...     )
... )
>>> Problem.loss('mse')
>>> Problem.metrics(['r2'])
>>> Problem.objective('val_r2__last')
>>> Problem.post_training(
...        num_epochs=1000,
...        metrics=['r2'],
...        callbacks=dict(
...         ModelCheckpoint={
...             'monitor': 'val_r2',
...             'mode': 'max',
...             'save_best_only': True,
...             'verbose': 1
...         },
...         EarlyStopping={
...             'monitor': 'val_r2',
...             'mode': 'max',
...             'verbose': 1,
...             'patience': 10
...         },
...         TensorBoard={
...             'log_dir':'tb_logs',
...             'histogram_freq':1,
...             'batch_size':64,
...             'write_graph':True,
...             'write_grads':True,
...             'write_images':True,
...             'update_freq':'epoch'
...         })
... )
Parameters

regression (bool) – if True the problem is defined as a regression problem, if False the problem is defined as a classification problem.

build_search_space()[source]

Build and return a search space object using the infered data shapes after loading data.

Returns

A search space instance.

Return type

NxSearchSpace

get_keras_model(arch_seq: list) → tensorflow.python.keras.engine.training.Model[source]

Get a keras model object from a set of decisions in the current search space.

Parameters

arch_seq (list) – a list of int of floats describing a choice of operations for the search space defined in the current problem.

hyperparameters(**kwargs)[source]

Define hyperparameters used to evaluate generated search_spaces.

loss(loss, weights=None)[source]

Define the loss used to train generated search_spaces.

Parameters
  • loss (str|callable|list) – a string indicating a specific loss function.

  • weights (list) – Optional.

metrics(metrics)[source]

Define a list of metrics for the training of generated search_spaces.

Parameters

metrics (list(str|callable)|dict) – If str the metric should be defined in Keras or in DeepHyper. If callable it should take 2 arguments (y_pred, y_true) which are a prediction and a true value respectively.

objective(objective)[source]

Define the objective you want to maximize for the search.

Parameters

objective (str|callable) – The objective will be maximized. If objective is str then it should be either ‘loss’ or a defined metric. You can use the 'val_' prefix when you want to select the objective on the validation set. You can use one of ['__min', '__max', '__last'] which respectively means you want to select the min, max or last value among all epochs. Using ‘__last’ will save a consequent compute time because the evaluation will not compute metrics on validation set for all epochs but the last. If objective is callable it should return a scalar value (i.e. float) and it will take a dict parameter. The dict will contain keys corresponding to loss and metrics such as ['loss', 'val_loss', 'r2', 'val_r2], it will also contains 'n_parameters' which corresponds to the number of trainable parameters of the current model, 'training_time' which corresponds to the time required to train the model, 'predict_time' which corresponds to the time required to make a prediction over the whole validation set. If this callable has a '__last' suffix then the evaluation will only compute validation loss/metrics for the last epoch. If this callable has contains ‘with_pred’ in its name then the dict will have two other keys ['y_pred', 'y_true'] where 'y_pred corresponds to prediction of the model on validation set and 'y_true' corresponds to real prediction.

Raises

WrongProblemObjective – raised when the objective is of a wrong definition.

post_training(num_epochs: int, metrics: list, callbacks: dict, repeat: int = 1)[source]

Choose settings to run a post-training.

Parameters
  • num_epochs (int) – the number of post-training epochs.

  • metrics (list) – list of post-training metrics.

  • callbacks (dict) –

    dict of keras.callbacks such as,

    • ModelCheckpoint (dict): tensorflow.keras.callbacks.ModelCheckpoint settings.

      • 'filepath': string, path to save the model file.

      • monitor: quantity to monitor.

      • verbose: verbosity mode, 0 or 1.

      • save_best_only: if save_best_only=True, the latest best model according to the quantity monitored will not be overwritten.

      • save_weights_only: if True, then only the model’s weights will be saved (model.save_weights(filepath)), else the full model is saved (model.save(filepath)).

      • mode: one of {auto, min, max}. If save_best_only=True, the decision to overwrite the current save file is made based on either the maximization or the minimization of the monitored quantity. For val_acc, this should be max, for val_loss this should be min, etc. In auto mode, the direction is automatically inferred from the name of the monitored quantity.

      • period: Interval (number of epochs) between checkpoints.

    • EarlyStopping (dict): tensorflow.keras.callbacks.EarlyStopping settings.

      • monitor: quantity to be monitored.

      • min_delta: minimum change in the monitored quantity to qualify as an improvement, i.e. an absolute change of less than min_delta, will count as no improvement.

      • patience: number of epochs with no improvement after which training will be stopped.

      • verbose: verbosity mode.

      • mode: one of {'auto', 'min', 'max'}. In min mode, training will stop when the quantity monitored has stopped decreasing; in max mode it will stop when the quantity monitored has stopped increasing; in auto mode, the direction is automatically inferred from the name of the monitored quantity.

      • baseline: Baseline value for the monitored quantity to reach. Training will stop if the model doesn’t show improvement over the baseline. restore_best_weights: whether to restore model weights from the epoch with the best value of the monitored quantity. If False, the model weights obtained at the last step of training are used.

  • repeat (int) – Number of times to repeat the training. Default to 1.

preprocessing(func: callable)[source]

Define how to preprocess your data.

Parameters

func (callable) – a function which returns a preprocessing scikit-learn Pipeline.

search_space(func: callable, **kwargs)[source]

Set a search space for neural architecture search.

Parameters

func (callable) – an object which has to be a callable and which is returning a Structure instance.

Raises

Search space

Nodes

class deephyper.nas.space.node.Node(name='', *args, **kwargs)[source]

This class represents a node of a graph

Parameters

name (str) – node name.

VariableNode

class deephyper.nas.space.node.VariableNode(name='')[source]

This class represents a node of a graph where you have a set of possible operations. It means the agent will have to act to choose one of these operations.

>>> import tensorflow as tf
>>> from deephyper.nas.space.node import VariableNode
>>> vnode = VariableNode("VNode1")
>>> from deephyper.nas.space.op.op1d import Dense
>>> vnode.add_op(Dense(
... units=10,
... activation=tf.nn.relu))
>>> vnode.num_ops
1
>>> vnode.add_op(Dense(
... units=1000,
... activation=tf.nn.tanh))
>>> vnode.num_ops
2
>>> vnode.set_op(0)
>>> vnode.op.units
10
Parameters

name (str) – node name.

denormalize(index)[source]

Denormalize a normalized index to get an absolute indexes. Useful when you want to compare the number of different search_spaces.

Parameters

indexes (float|int) – a normalized index.

Returns

An absolute indexes corresponding to the operation choosen with the relative index of index.

Return type

int

ConstantNode

class deephyper.nas.space.node.ConstantNode(op=None, name='', *args, **kwargs)[source]

A ConstantNode represents a node with a fixed operation. It means the agent will not make any new decision for this node. The common use case for this node is to add a tensor in the graph.

>>> import tensorflow as tf
>>> from deephyper.nas.space.node import ConstantNode
>>> from deephyper.nas.space.op.op1d import Dense
>>> cnode = ConstantNode(op=Dense(units=100, activation=tf.nn.relu), name='CNode1')
>>> cnode.op
Dense_100_relu
Parameters
  • op (Operation, optional) – [description]. Defaults to None.

  • name (str, optional) – [description]. Defaults to ‘’.

MirrorNode

class deephyper.nas.space.node.MirrorNode(node)[source]

A MirrorNode is a node which reuse an other, it enable the reuse of keras layers. This node will not add operations to choose.

Parameters

node (Node) – The targeted node to mirror.

>>> from deephyper.nas.space.node import VariableNode, MirrorNode
>>> from deephyper.nas.space.op.op1d import Dense
>>> vnode = VariableNode()
>>> vnode.add_op(Dense(10))
>>> vnode.add_op(Dense(20))
>>> mnode = MirrorNode(vnode)
>>> vnode.set_op(0)
>>> vnode.op
Dense_10
>>> mnode.op
Dense_10

MimeNode

class deephyper.nas.space.node.MimeNode(node, name='')[source]

A MimeNode is a node which reuse an the choice made for an VariableNode, it enable the definition of a Cell based search_space. This node reuse the operation from the mimed VariableNode but only the choice made.

Parameters

node (VariableNode) – the VariableNode to mime.

>>> from deephyper.nas.space.node import VariableNode, MimeNode
>>> from deephyper.nas.space.op.op1d import Dense
>>> vnode = VariableNode()
>>> vnode.add_op(Dense(10))
>>> vnode.add_op(Dense(20))
>>> mnode = MimeNode(vnode)
>>> mnode.add_op(Dense(30))
>>> mnode.add_op(Dense(40))
>>> vnode.set_op(0)
>>> vnode.op
Dense_10
>>> mnode.op
Dense_30

Search Space

class deephyper.nas.space.KSearchSpace(input_shape, output_shape, batch_size=None, *args, **kwargs)[source]

A KSearchSpace represents a search space of neural networks.

>>> from tensorflow.keras.utils import plot_model
>>> from deephyper.nas.space import KSearchSpace
>>> from deephyper.nas.space.node import VariableNode, ConstantNode
>>> from deephyper.nas.space.op.op1d import Dense
>>> struct = KSearchSpace((5, ), (1, ))
>>> vnode = VariableNode()
>>> struct.connect(struct.input_nodes[0], vnode)
>>> vnode.add_op(Dense(10))
>>> vnode.add_op(Dense(20))
>>> output_node = ConstantNode(op=Dense(1))
>>> struct.connect(vnode, output_node)
>>> struct.set_ops([0])
>>> model = struct.create_model()
Parameters
  • input_shape (list(tuple(int))) – list of shapes of all inputs.

  • output_shape (tuple(int)) – shape of output.

  • batch_size (list(tuple(int))) – batch size of the input layer. If input_shape is defining a list of inputs, batch_size should also define a list of inputs.

Raises

InputShapeOfWrongType – [description]

create_model()[source]

Create the tensors corresponding to the search_space.

Returns

A keras.Model for the current search_space with the corresponding set of operations.

set_ops(indexes)[source]

Set the operations for each node of each cell of the search_space.

Parameters

indexes (list) – element of list can be float in [0, 1] or int.

Raises

WrongSequenceToSetOperations – raised when ‘indexes’ is of a wrong length.

set_output_node(graph, output_nodes)[source]

Set the output node of the search_space.

Parameters
  • graph (nx.DiGraph) – graph of the search_space.

  • output_nodes (Node) – nodes of the current search_space without successors.

Returns

output node of the search_space.

Return type

Node

class deephyper.nas.space.AutoKSearchSpace(input_shape, output_shape, regression: bool, *args, **kwargs)[source]

An AutoKSearchSpace represents a search space of neural networks.

Parameters
  • input_shape (list(tuple(int))) – list of shapes of all inputs.

  • output_shape (tuple(int)) – shape of output.

  • regression (bool) – if True the output will be a simple tf.keras.layers.Dense(output_shape[0]) layer as the output layer. if False the output will be tf.keras.layers.Dense(output_shape[0], activation='softmax').

Raises

InputShapeOfWrongType – [description]

create_model()[source]

Create the tensors corresponding to the search_space.

Returns

The output tensor.

set_output_node(graph, output_nodes)[source]

Set the output node of the search_space.

Parameters
  • graph (nx.DiGraph) – graph of the search_space.

  • output_nodes (Node) – nodes of the current search_space without successors.

Returns

output node of the search_space.

Return type

Node

Training Pipeline

deephyper.nas.run.alpha.run(config)[source]

Trainer

class deephyper.nas.trainer.train_valid.TrainerTrainValid(config, model)[source]
evaluate(dataset='train')[source]

Evaluate the performance of your model for the same configuration.

Parameters

dataset (str, optional) – must be “train” or “valid”. If “train” then metrics will be evaluated on the training dataset. If “valid” then metrics will be evaluated on the “validation” dataset. Defaults to ‘train’.

Returns

a list of scalar values corresponding do config loss & metrics.

Return type

list

predict(dataset: str = 'valid', keep_normalize: bool = False)tuple[source]

[summary]

Parameters
  • dataset (str, optional) – ‘valid’ or ‘train’. Defaults to ‘valid’.

  • keep_normalize (bool, optional) – if False then the preprocessing will be reversed after prediction. if True nothing will be reversed. Defaults to False.

Raises

DeephyperRuntimeError – [description]

Returns

(y_true, y_pred)

Return type

tuple

train(num_epochs: int = None, with_pred: bool = False, last_only: bool = False)[source]

Train the model.

Parameters
  • num_epochs (int, optional) – override the num_epochs passed to init the Trainer. Defaults to None, will use the num_epochs passed to init the Trainer.

  • with_pred (bool, optional) – will compute a prediction after the training and will add (‘y_true’, ‘y_pred’) to the output history. Defaults to False, will skip it (use it to save compute time).

  • last_only (bool, optional) – will compute metrics after the last epoch only. Defaults to False, will compute metrics after each training epoch (use it to save compute time).

Raises

DeephyperRuntimeError – raised when the num_epochs < 0.

Returns

a dictionnary corresponding to the training.

Return type

dict

Post-Training

The module deephyper.post.train aims to run post-training using an already defined Problem and the results of a finished search (e.g. list of best search_spaces).

Create a post-training Balsam application:

bash
balsam app --name POST --exe "$(which python) -m deephyper.post.train"

Collect a list of 50 best search_spaces:

bash
deephyper-analytics json best -n 50 -p /projects/datascience/regele/experiments/cfd/cls_mlp_turb/exp_0/cls_mlp_turb_exp_0_2019-05-25_15.json

Create a Balsam job to run your post-training using the previously created list of best search_spaces:

bash
balsam job --name post_cls_0 --workflow post_cls_0 --app POST --args '--evaluator balsam --problem cfdpb.cls_mlp_turbulence.problem_0.Problem --fbest /projects/datascience/regele/cfdpb/cfdpb/cls_mlp_turbulence/exp/exp_0/best_archs.json'

Submit a Theta job:

bash
balsam submit-launch -n 8 -q debug-cache-quad -t 60 -A datascience --job-mode mpi --wf-filter post_cls_0