deephyper.evaluator

deephyper.evaluator#

This evaluator subpackage provides a common interface to execute isolated tasks with different parallel backends and system properties. This interface is used by search algorithm to perform black-box optimization (the black-box being represented by the run-function). An Evaluator, when instanciated, is bound to a run-function which takes as first argument a dictionnary and optionally has other keyword-arguments. The run-function has to return a Python serializable value (under pickle protocol). In it’s most basic form the return value is a float.

An example run-function is:

def run(job: RunningJob) -> Union[float, str, Dict]:

    config = job.parameters
    y = config["x"]**2

    return y

The return value of the run-function respect the following standards (but the feature is not necessarily supported by all search algorithms, such as multi-objective optimization):

# float for single objective optimization
return 42.0
# str with "F" prefix for failed evaluation
return "F_out_of_memory"
# dict
return {"objective": 42.0}
# dict with additional information
return {"objective": 42.0, "metadata": {"num_epochs_trained": 25, "num_parameters": 420000}}
# dict with reserved keywords (when @profile decorator is used)
return {"objective": 42.0, "metadata": {"timestamp_start": ..., "timestamp_end": ...}"
# tuple of float for multi-objective optimization (will appear as "objective_0" and "objective_1" in the resulting dataframe)
return 42.0, 0.42

Functions

parse_subprocess_result

Utility to parse a result from a subprocess of the format "DH-OUTPUT:...".

profile

Decorator to use on a run_function to profile its execution-time and peak memory usage.

queued

Decorator transforming an Evaluator into a Queued{Evaluator}.

to_json

Classes

Evaluator

This Evaluator class asynchronously manages a series of Job objects to help execute given HPS or NAS tasks on various environments with differing system settings and properties.

HPOJob

Job

Represents an evaluation executed by the Evaluator class.

MPICommEvaluator

This evaluator uses the mpi4py library as backend.

ProcessPoolEvaluator

This evaluator uses the ProcessPoolExecutor as backend.

RayEvaluator

This evaluator uses the ray library as backend.

RunningJob

A RunningJob is adapted Job object that is passed to the run-function as input.

SerialEvaluator

This evaluator run evaluations one after the other (not parallel).

ThreadPoolEvaluator

This evaluator uses the ThreadPoolExecutor as backend.

callback

The callback module contains sub-classes of the Callback class used to trigger custom actions on the start and completion of jobs by the Evaluator.

storage

This subpackage provides an interface to implement new storage clients.