This evaluator subpackage provides a common interface to execute isolated tasks with different parallel backends and system properties. This interface is used by search algorithm to perform black-box optimization (the black-box being represented by the run-function). An Evaluator, when instanciated, is bound to a run-function which takes as first argument a dictionnary and optionally has other keyword-arguments. The run-function has to return a Python serializable value (under pickle protocol). In it’s most basic form the return value is a float.

An example run-function is:

def run(job: RunningJob) -> Union[float, str, Dict]:

    config = job.parameters
    y = config["x"]**2

    return y

The return value of the run-function respect the following standards (but the feature is not necessarily supported by all search algorithms, such as multi-objective optimization):

# float for single objective optimization
return 42.0
# str with "F" prefix for failed evaluation
return "F_out_of_memory"
# dict
return {"objective": 42.0}
# dict with additional information
return {"objective": 42.0, "metadata": {"num_epochs_trained": 25, "num_parameters": 420000}}
# dict with reserved keywords (when @profile decorator is used)
return {"objective": 42.0, "metadata": {"timestamp_start": ..., "timestamp_end": ...}"
# tuple of float for multi-objective optimization (will appear as "objective_0" and "objective_1" in the resulting dataframe)
return 42.0, 0.42



Decorator transforming an Evaluator into a Distributed{Evaluator}.


Utility to parse a result from a subprocess of the format "DH-OUTPUT:...".


Decorator to use on a run_function to profile its execution-time and peak memory usage.


Decorator transforming an Evaluator into a Queued{Evaluator}.




This Evaluator class asynchronously manages a series of Job objects to help execute given HPS or NAS tasks on various environments with differing system settings and properties.


Represents an evaluation executed by the Evaluator class.


This evaluator uses the mpi4py library as backend.


This evaluator uses the ProcessPoolExecutor as backend.


This evaluator uses the ray library as backend.


A RunningJob is adapted Job object that is passed to the run-function as input.


This evaluator run evaluations one after the other (not parallel).


This evaluator uses the ThreadPoolExecutor as backend.


The callback module contains sub-classes of the Callback class used to trigger custom actions on the start and completion of jobs by the Evaluator.

This subpackage provides an interface to implement new storage clients.