Evaluator Interface

The goal off the evaluator module is to have a set of objects which can helps us to run our task on different environments and with different system settings/properties.

evaluator diag

Evaluator

class deephyper.evaluator.evaluate.Evaluator(run_function, cache_key=None, encoder=<class 'deephyper.evaluator.evaluate.Encoder'>, seed=None, num_workers=None, **kwargs)[source]

The goal off the evaluator module is to have a set of objects which can helps us to run our task on different environments and with different system settings/properties.

Parameters
  • run_function (Callable) – the function to execute, it must be a callable and should not be defined in the __main__ module.

  • cache_key (Callable, str, optional) – A way of defining how to use cached results. When an evaluation is done a corresponding uuid is generated, if an new evaluation as a already known uuid then the past result will be re-used. You have different ways to define how to generate an uuid. If None then the whole parameter dict (corresponding to the evaluation) will be serialized and used as a uuid (it may raise an exception if it’s not serializable, please use the encoder parameter to use a custom json.JSONEncoder). If callable then the parameter dict will be passed to the callable which must return an uuid. If ‘uuid’ then the uuid.uuid4() will be used for every new evaluation. Defaults to None.

Raises

BalsamEvaluator

class deephyper.evaluator._balsam.BalsamEvaluator(run_function, cache_key=None, num_nodes_master=1, num_nodes_per_eval=1, num_ranks_per_node=1, num_evals_per_node=1, num_threads_per_rank=128, num_threads_per_node=None, **kwargs)[source]

Evaluator using Balsam software.

Documentation to Balsam : https://balsam.readthedocs.io This class helps us to run task on HPC systems with more flexibility and ease of use.

Parameters
  • run_function (func) – takes one parameter of type dict and returns a scalar value.

  • cache_key (func) – takes one parameter of type dict and returns a hashable type, used as the key for caching evaluations. Multiple inputs that map to the same hashable key will only be evaluated once. If None, then cache_key defaults to a lossless (identity) encoding of the input dict.

  • num_nodes_per_eval (int) –

RayEvaluator

class deephyper.evaluator._ray_evaluator.RayEvaluator(run_function, cache_key=None, redis_address=None, **kwargs)[source]

The RayEvaluator relies on the Ray (https://ray.readthedocs.io) package. Ray is a fast and simple framework for building and running distributed applications.

Parameters

redis_address (str, optional) – The “IP:PORT” redis address for the RAY-driver to connect on the RAY-head.

SubprocessEvaluator

class deephyper.evaluator._subprocess.SubprocessEvaluator(run_function, cache_key=None, **kwargs)[source]

Evaluator using subprocess.

The SubprocessEvaluator use the subprocess package. The generated processes have a fresh memory independant from their parent process. All the imports are going to be repeated.

Parameters
  • run_function (func) – takes one parameter of type dict and returns a scalar value.

  • cache_key (func) – takes one parameter of type dict and returns a hashable type, used as the key for caching evaluations. Multiple inputs that map to the same hashable key will only be evaluated once. If None, then cache_key defaults to a lossless (identity) encoding of the input dict.

ProcessPoolEvaluator

class deephyper.evaluator._processPool.ProcessPoolEvaluator(run_function, cache_key=None, **kwargs)[source]

Evaluator using ProcessPoolExecutor.

The ProcessPoolEvaluator use the concurrent.futures.ProcessPoolExecutor class. The processes doesn’t share memory but they are forked from the mother process so imports done before are done repeated. Be carefull if your run_function is loading an package such as tensorflow it can hang.

Parameters
  • run_function (func) – takes one parameter of type dict and returns a scalar value.

  • cache_key (func) – takes one parameter of type dict and returns a hashable type, used as the key for caching evaluations. Multiple inputs that map to the same hashable key will only be evaluated once. If None, then cache_key defaults to a lossless (identity) encoding of the input dict.

ThreadPoolEvaluator

class deephyper.evaluator._threadPool.ThreadPoolEvaluator(run_function, cache_key=None, **kwargs)[source]

Evaluator using ThreadPoolExecutor.

The ThreadPoolEvaluator use the concurrent.futures.ThreadPoolExecutor class. The processes share memory and they are forked from the mother process so imports done before are done repeated. Be carefull if your run_function is loading an package such as tensorflow it can hang. If your run_function is very fast this evaluator can be faster than ProcessPoolEvaluator.

Parameters
  • run_function (func) – takes one parameter of type dict and returns a scalar value.

  • cache_key (func) – takes one parameter of type dict and returns a hashable type, used as the key for caching evaluations. Multiple inputs that map to the same hashable key will only be evaluated once. If None, then cache_key defaults to a lossless (identity) encoding of the input dict.

Warning

For ThreadPoolEvaluator, note that this does not mean that they are executed on different CPUs. Python threads will NOT make your program faster if it already uses 100 % CPU time. Python threads are used in cases where the execution of a task involves some waiting. One example would be interaction with a service hosted on another computer, such as a webserver. Threading allows python to execute other code while waiting; this is easily simulated with the sleep function. (from: https://en.wikibooks.org/wiki/Python_Programming/Threading)