deephyper.evaluator

Evaluator subpackage.

This package a common interface to execute asynchronous parallel tasks with different backends and system properties.

This interface is used by search algorithm to perform black-box optimization (the black-box being represented by the run-function). An Evaluator, when instanciated, is bound to a run-function which takes as first argument a job and optionally has other keyword-arguments. In it’s most basic form the return value is a float. The standard way of creating an Evaluator is to use deephyper.evaluator.Evaluator.create().

An example run-function is:

def run(job: RunningJob) -> float | str | dict:

    config = job.parameters
    y = config["x"]**2

    return y

The return value of the run-function respect the following standards (but the feature is not necessarily supported by all search algorithms, such as multi-objective optimization):

# float for single objective optimization
return 42.0
# str with "F" prefix for failed evaluation
return "F_out_of_memory"
# dict
return {"objective": 42.0}
# dict with additional information
return {"objective": 42.0, "metadata": {"num_epochs_trained": 25, "num_parameters": 420000}}
# dict with reserved keywords (when @profile decorator is used)
return {"objective": 42.0, "metadata": {"timestamp_start": ..., "timestamp_end": ...}"
# tuple of float for multi-objective optimization, appears as "objective_0" and "objective_1"
return 42.0, 0.42

Functions

parse_subprocess_result	Utility to parse a result from a subprocess of the format "DH-OUTPUT:...".
profile	Decorator to use on a run_function to profile its execution-time and peak memory usage.
queued	Decorator transforming an Evaluator into a Queued{Evaluator}.
to_json

Classes

Evaluator	This class manages the execution of asynchronous parallel calls of a Python function.
HPOJob	Represents the execution of a run_function for HPO by the Evaluator.
Job	Represents the execution of a generic run_function by the Evaluator.
JobStatus	Represents the execution status of a job.
LokyEvaluator	This evaluator uses the ProcessPoolExecutor from loky as backend.
MPICommEvaluator	This evaluator uses the mpi4py library as backend.
ProcessPoolEvaluator	This evaluator uses the ProcessPoolExecutor as backend.
RayEvaluator	This evaluator uses the ray library as backend.
RunningJob	A RunningJob is an adapted Job object that is passed to the run-function as input.
SerialEvaluator	This evaluator uses Python AsyncIO as backend.
ThreadPoolEvaluator	This evaluator uses the ThreadPoolExecutor as backend.

Exceptions

MaximumJobsSpawnReached

Raised when the maximum number of jobs spawned by the evaluator is reached.

callback	The callback module contains sub-classes of the Callback class.
mpi	Submodule from where MPI should always be imported within the package.
storage	This subpackage provides an interface to implement new storage clients.
utils	Utilies for evaluator module.