DeepHyper: Scalable Neural Architecture and Hyperparameter Search for Deep Neural Networks
Contents
DeepHyper: Scalable Neural Architecture and Hyperparameter Search for Deep Neural Networks#

DeepHyper is a distributed machine learning (AutoML) package for automating the development of deep neural networks for scientific applications. It can run on a single laptop as well as on 1,000 of nodes.
It comprises different tools such as:
Optimizing hyper-parameters for a given black-box function.
Neural architecture search to discover high-performing deep neural network with variable operations and connections.
Automated machine learning, to easily experiment many learning algorithms from Scikit-Learn.
DeepHyper provides an infrastructure that targets experimental research in NAS and HPS methods, scalability, and portability across diverse supercomputers. It comprises three main modules:
deephyper.problem
: Tools for defining neural architecture and hyper-parameter search problems.deephyper.evaluator
: A simple interface to dispatch model evaluation tasks. Implementations range from subprocess for laptop experiments to ray for large-scale runs on HPC systems.deephyper.search
: Search methods for NAS and HPS. By extending the generic Search class, one can easily add new NAS or HPS methods to DeepHyper.
DeepHyper installation requires Python >= 3.7.
Quick Start#
The black-box function named run
is defined by taking an input dictionnary named config
which contains the different variables to optimize. Then the run-function is binded to an Evaluator
in charge of distributing the computation of multiple evaluations. Finally, a Bayesian search named CBO
is created and executed to find the values of config which maximize the return value of run(config)
.
def run(config: dict):
return -config["x"]**2
# Necessary IF statement otherwise it will enter in a infinite loop
# when loading the 'run' function from a subprocess
if __name__ == "__main__":
from deephyper.problem import HpProblem
from deephyper.search.hps import CBO
from deephyper.evaluator import Evaluator
# define the variable you want to optimize
problem = HpProblem()
problem.add_hyperparameter((-10.0, 10.0), "x")
# define the evaluator to distribute the computation
evaluator = Evaluator.create(
run,
method="subprocess",
method_kwargs={
"num_workers": 2,
},
)
# define your search and execute it
search = CBO(problem, evaluator)
results = search.search(max_evals=100)
print(results)
Which outputs the following where the best x
found is clearly around 0
.
x job_id objective timestamp_submit timestamp_gather
0 -7.744105 1 -5.997117e+01 0.011047 0.037649
1 -9.058254 2 -8.205196e+01 0.011054 0.056398
2 -1.959750 3 -3.840621e+00 0.049750 0.073166
3 -5.150553 4 -2.652819e+01 0.065681 0.089355
4 -6.697095 5 -4.485108e+01 0.082465 0.158050
.. ... ... ... ... ...
95 -0.034096 96 -1.162566e-03 26.479630 26.795639
96 -0.034204 97 -1.169901e-03 26.789255 27.155481
97 -0.037873 98 -1.434366e-03 27.148506 27.466934
98 -0.000073 99 -5.387088e-09 27.460253 27.774704
99 0.697162 100 -4.860350e-01 27.768153 28.142431
Table of Contents#
Get Started
API Reference
- Core
- deephyper.core.cli
- deephyper.core.exceptions
- deephyper.core.exceptions.DeephyperError
- deephyper.core.exceptions.DeephyperRuntimeError
- deephyper.core.exceptions.MissingRequirementError
- deephyper.core.exceptions.RunFunctionError
- deephyper.core.exceptions.SearchTerminationError
- deephyper.core.exceptions.loading
- deephyper.core.exceptions.nas
- deephyper.core.exceptions.nas.DeephyperError
- deephyper.core.exceptions.nas.NASError
- deephyper.core.exceptions.nas.space
- deephyper.core.exceptions.nas.space.InputShapeOfWrongType
- deephyper.core.exceptions.nas.space.NASError
- deephyper.core.exceptions.nas.space.NodeAlreadyAdded
- deephyper.core.exceptions.nas.space.StructureHasACycle
- deephyper.core.exceptions.nas.space.WrongOutputShape
- deephyper.core.exceptions.nas.space.WrongSequenceToSetOperations
- deephyper.core.exceptions.problem
- deephyper.core.exceptions.problem.DeephyperError
- deephyper.core.exceptions.problem.NaProblemError
- deephyper.core.exceptions.problem.ProblemLoadDataIsNotCallable
- deephyper.core.exceptions.problem.ProblemPreprocessingIsNotCallable
- deephyper.core.exceptions.problem.SearchSpaceBuilderIsNotCallable
- deephyper.core.exceptions.problem.SearchSpaceBuilderMissingDefaultParameter
- deephyper.core.exceptions.problem.SearchSpaceBuilderMissingParameter
- deephyper.core.exceptions.problem.SpaceDimNameOfWrongType
- deephyper.core.exceptions.problem.WrongProblemObjective
- deephyper.core.parser
- deephyper.core.utils
- Ensemble
- Evaluator
- deephyper.evaluator.distributed
- deephyper.evaluator.parse_subprocess_result
- deephyper.evaluator.profile
- deephyper.evaluator.queued
- deephyper.evaluator.to_json
- deephyper.evaluator.Evaluator
- deephyper.evaluator.Job
- deephyper.evaluator.MPICommEvaluator
- deephyper.evaluator.ProcessPoolEvaluator
- deephyper.evaluator.RayEvaluator
- deephyper.evaluator.SerialEvaluator
- deephyper.evaluator.SubprocessEvaluator
- deephyper.evaluator.ThreadPoolEvaluator
- deephyper.evaluator.callback
- Keras
- deephyper.keras.callbacks
- deephyper.keras.callbacks.import_callback
- deephyper.keras.callbacks.CSVExtendedLogger
- deephyper.keras.callbacks.LearningRateScheduleCallback
- deephyper.keras.callbacks.LearningRateWarmupCallback
- deephyper.keras.callbacks.StopIfUnfeasible
- deephyper.keras.callbacks.TimeStopping
- deephyper.keras.callbacks.csv_extended_logger
- deephyper.keras.callbacks.learning_rate_warmup
- deephyper.keras.callbacks.stop_if_unfeasible
- deephyper.keras.callbacks.stop_on_timeout
- deephyper.keras.callbacks.time_stopping
- deephyper.keras.callbacks.utils
- deephyper.keras.layers
- deephyper.keras.layers.AttentionCOS
- deephyper.keras.layers.AttentionConst
- deephyper.keras.layers.AttentionGAT
- deephyper.keras.layers.AttentionGCN
- deephyper.keras.layers.AttentionGenLinear
- deephyper.keras.layers.AttentionLinear
- deephyper.keras.layers.AttentionSymGAT
- deephyper.keras.layers.GlobalAttentionPool
- deephyper.keras.layers.GlobalAttentionSumPool
- deephyper.keras.layers.GlobalAvgPool
- deephyper.keras.layers.GlobalMaxPool
- deephyper.keras.layers.GlobalSumPool
- deephyper.keras.layers.MessagePasserNNM
- deephyper.keras.layers.MessagePassing
- deephyper.keras.layers.Padding
- deephyper.keras.layers.SparseMPNN
- deephyper.keras.layers.UpdateFuncGRU
- deephyper.keras.layers.UpdateFuncMLP
- deephyper.keras.utils
- deephyper.keras.callbacks
- NAS
- deephyper.nas.KSearchSpace
- deephyper.nas.NxSearchSpace
- deephyper.nas.losses
- deephyper.nas.lr_scheduler
- deephyper.nas.metrics
- deephyper.nas.metrics.acc
- deephyper.nas.metrics.load_attr
- deephyper.nas.metrics.mae
- deephyper.nas.metrics.mse
- deephyper.nas.metrics.r2
- deephyper.nas.metrics.rmse
- deephyper.nas.metrics.selectMetric
- deephyper.nas.metrics.sparse_perplexity
- deephyper.nas.metrics.tfp_mae
- deephyper.nas.metrics.tfp_mse
- deephyper.nas.metrics.tfp_r2
- deephyper.nas.metrics.tfp_rmse
- deephyper.nas.metrics.to_tfp
- deephyper.nas.metrics.OrderedDict
- deephyper.nas.node
- deephyper.nas.operation
- deephyper.nas.operation.operation
- deephyper.nas.operation.AddByPadding
- deephyper.nas.operation.AddByProjecting
- deephyper.nas.operation.Concatenate
- deephyper.nas.operation.Connect
- deephyper.nas.operation.Identity
- deephyper.nas.operation.Operation
- deephyper.nas.operation.Tensor
- deephyper.nas.operation.Zero
- deephyper.nas.preprocessing
- deephyper.nas.run
- deephyper.nas.spacelib
- deephyper.nas.trainer
- Problem
- deephyper.problem.Categorical
- deephyper.problem.Float
- deephyper.problem.Integer
- deephyper.problem.AndConjunction
- deephyper.problem.Beta
- deephyper.problem.BetaFloatHyperparameter
- deephyper.problem.BetaIntegerHyperparameter
- deephyper.problem.CategoricalHyperparameter
- deephyper.problem.Configuration
- deephyper.problem.ConfigurationSpace
- deephyper.problem.Constant
- deephyper.problem.Distribution
- deephyper.problem.EqualsCondition
- deephyper.problem.ForbiddenAndConjunction
- deephyper.problem.ForbiddenEqualsClause
- deephyper.problem.ForbiddenEqualsRelation
- deephyper.problem.ForbiddenGreaterThanRelation
- deephyper.problem.ForbiddenInClause
- deephyper.problem.ForbiddenLessThanRelation
- deephyper.problem.GreaterThanCondition
- deephyper.problem.HpProblem
- deephyper.problem.InCondition
- deephyper.problem.LessThanCondition
- deephyper.problem.NaProblem
- deephyper.problem.Normal
- deephyper.problem.NormalFloatHyperparameter
- deephyper.problem.NormalIntegerHyperparameter
- deephyper.problem.NotEqualsCondition
- deephyper.problem.OrConjunction
- deephyper.problem.OrdinalHyperparameter
- deephyper.problem.UnParametrizedHyperparameter
- deephyper.problem.Uniform
- deephyper.problem.UniformFloatHyperparameter
- deephyper.problem.UniformIntegerHyperparameter
- Search
- Sklearn