deephyper.evaluator.MPICommEvaluator

class deephyper.evaluator.MPICommEvaluator(run_function: Callable, num_workers: Optional[int] = None, callbacks=None, run_function_kwargs=None, storage: Optional[deephyper.evaluator.storage._storage.Storage] = None, search_id: Optional[Hashable] = None, comm=None, root=0, abort_on_exit=False, wait_on_exit=True, cancel_jobs_on_exit=True)

Bases: deephyper.evaluator._evaluator.Evaluator

This evaluator uses the mpi4py library as backend.

This evaluator consider an already existing MPI-context (with running processes), therefore it has less overhead than MPIPoolEvaluator which spawn processes dynamically.

Parameters

• run_function (callable) – functions to be executed by the Evaluator.

• num_workers (int, optional) – Number of parallel Ray-workers used to compute the run_function. Defaults to None which consider 1 rank as a worker (minus the master rank).

• callbacks (list, optional) – A list of callbacks to trigger custom actions at the creation or completion of jobs. Defaults to None.

• run_function_kwargs (dict, optional) – Keyword-arguments to pass to the run_function. Defaults to None.

• comm (optional) – A MPI communicator, if None it will use MPI.COMM_WORLD. Defaults to None.

• rank (int, optional) – The rank of the master process. Defaults to 0.

• abort_on_exit (bool) – If True then it will call comm.Abort() to force all MPI processes to finish when closing the Evaluator (i.e., exiting the current with block).

Methods

convert_for_csv	Convert an input value to an accepted format to be saved as a value of a CSV file (e.g., a list becomes it’s str representation).
create	Create evaluator with a specific backend and configuration.
decode	Decode the key following a JSON format to return a dict.
dump_evals	Dump evaluations to a CSV file.
execute	Execute the received job.
gather	Collect the completed tasks from the evaluator in batches of one or more.
set_timeout	Set a timeout for the Evaluator.
submit	Send configurations to be evaluated by available workers.
to_json	Returns a json version of the evaluator.

Attributes

FAIL_RETURN_VALUE
NEST_ASYNCIO_PATCHED
PYTHON_EXE

convert_for_csv(val)

Convert an input value to an accepted format to be saved as a value of a CSV file (e.g., a list becomes it’s str representation).

Parameters

val (Any) – The input value to convert.

Returns

The converted value.

Return type

Any

static create(run_function, method='serial', method_kwargs={})

Create evaluator with a specific backend and configuration.

Parameters

• run_function (function) – the function to execute in parallel.

• method (str, optional) – the backend to use in ["serial", "thread", "process", "ray", "mpicomm"]. Defaults to "serial".

• method_kwargs (dict, optional) – configuration dictionnary of the corresponding backend. Keys corresponds to the keyword arguments of the corresponding implementation. Defaults to “{}”.

Raises

ValueError – if the method is not acceptable.

Returns

the Evaluator with the corresponding backend and configuration.

Return type

Evaluator

decode(key)

Decode the key following a JSON format to return a dict.

dump_evals(saved_keys=None, log_dir: str = '.', filename='results.csv')

Dump evaluations to a CSV file.

Parameters

• saved_keys (list|callable) – If None the whole job.config will be added as row of the CSV file. If a list filtered keys will be added as a row of the CSV file. If a callable the output dictionnary will be added as a row of the CSV file.

• log_dir (str) – directory where to dump the CSV file.

• filename (str) – name of the file where to write the data.

async execute(job: deephyper.evaluator._job.Job) → deephyper.evaluator._job.Job

Execute the received job. To be implemented with a specific backend.

Parameters

job (Job) – the Job to be executed.

gather(type, size=1)

Collect the completed tasks from the evaluator in batches of one or more.

Parameters

• type (str) –

  Options:

  "ALL"

  Block until all jobs submitted to the evaluator are completed.

  "BATCH"

  Specify a minimum batch size of jobs to collect from the evaluator. The method will block until at least size evaluations are completed.

• size (int, optional) – The minimum batch size that we want to collect from the evaluator. Defaults to 1.

Raises

Exception – Raised when a gather operation other than “ALL” or “BATCH” is provided.

Returns

A batch of completed jobs that is at minimum the given size.

Return type

List[Job]

set_timeout(timeout)

Set a timeout for the Evaluator. It will create task with a “time budget” and will kill the the task if this budget is exhausted.

submit(configs: List[Dict])

Send configurations to be evaluated by available workers.

Parameters

configs (List[Dict]) – A list of dict which will be passed to the run function to be executed.

to_json()

Returns a json version of the evaluator.