deephyper.evaluator#
This evaluator subpackage provides a common interface to execute isolated tasks with different parallel backends and system properties. This interface is used by search algorithm to perform black-box optimization (the black-box being represented by the run
-function).
An Evaluator
, when instanciated, is bound to a run
-function which takes as first argument a dictionnary and optionally has other keyword-arguments. The run
-function has to return a Python serializable value (under pickle
protocol). In it’s most basic form the return value is a float
.
An example run
-function is:
def run(job: RunningJob) -> Union[float, str, Dict]:
config = job.parameters
y = config["x"]**2
return y
The return value of the run
-function respect the following standards (but the feature is not necessarily supported by all search algorithms, such as multi-objective optimization):
# float for single objective optimization
return 42.0
# str with "F" prefix for failed evaluation
return "F_out_of_memory"
# dict
return {"objective": 42.0}
# dict with additional information
return {"objective": 42.0, "metadata": {"num_epochs_trained": 25, "num_parameters": 420000}}
# dict with reserved keywords (when @profile decorator is used)
return {"objective": 42.0, "metadata": {"timestamp_start": ..., "timestamp_end": ...}"
# tuple of float for multi-objective optimization (will appear as "objective_0" and "objective_1" in the resulting dataframe)
return 42.0, 0.42
Functions
Utility to parse a result from a subprocess of the format "DH-OUTPUT:...". |
|
Decorator to use on a |
|
Decorator transforming an Evaluator into a |
|
Classes
This |
|
Represents an evaluation executed by the |
|
This evaluator uses the |
|
This evaluator uses the |
|
This evaluator uses the |
|
A RunningJob is adapted Job object that is passed to the run-function as input. |
|
This evaluator run evaluations one after the other (not parallel). |
|
This evaluator uses the |