deephyper.evaluator.RayEvaluator#
- class deephyper.evaluator.RayEvaluator(run_function: Callable, callbacks: list | None = None, run_function_kwargs: dict | None = None, storage: Storage | None = None, search_id: Hashable | None = None, address: str | None = None, password: str | None = None, num_cpus: int | None = None, num_gpus: int | None = None, include_dashboard: bool = False, num_cpus_per_task: float = 1, num_gpus_per_task: float | None = None, ray_kwargs: dict | None = None, num_workers: int | None = None)[source]#
Bases:
Evaluator
This evaluator uses the
ray
library as backend.- Parameters:
run_function (callable) – functions to be executed by the
Evaluator
.callbacks (list, optional) – A list of callbacks to trigger custom actions at the creation or completion of jobs. Defaults to None.
run_function_kwargs (dict, optional) – Static keyword arguments to pass to the
run_function
when executed.storage (Storage, optional) – Storage used by the evaluator. Defaults to
MemoryStorage
.search_id (Hashable, optional) – The id of the search to use in the corresponding storage. If
None
it will create a new search identifier when initializing the search.address (str, optional) – address of the Ray-head. Defaults to None, if no Ray-head was started.
password (str, optional) – password to connect ot the Ray-head. Defaults to None, if the default Ray-password is used.
num_cpus (int, optional) – number of CPUs available in the Ray-cluster. Defaults to None, if the Ray-cluster was already started it will be automatically computed.
num_gpus (int, optional) – number of GPUs available in the Ray-cluster. Defaults to None, if the Ray-cluster was already started it will be automatically computed.
num_cpus_per_task (float, optional) – number of CPUs used per remote task. Defaults to 1.
num_gpus_per_task (float, optional) – number of GPUs used per remote task. Defaults to None.
ray_kwargs (dict, optional) – other ray keyword arguments passed to
ray.init(...)
. Defaults to {}.num_workers (int, optional) – number of workers available to compute remote-tasks in parallel. Defaults to
None
, or if it is-1
it is automatically computed based withnum_workers = int(num_cpus // num_cpus_per_task)
.
Methods
Convert an input value to an accepted format to be saved as a value of a CSV file (e.g., a list becomes it's str representation).
Create evaluator with a specific backend and configuration.
Decode the key following a JSON format to return a dict.
Dump evaluations to a CSV file.
Execute the received job.
Collect the completed tasks from the evaluator in batches of one or more.
Set a timeout for the Evaluator.
Send configurations to be evaluated by available workers.
Returns a json version of the evaluator.
Attributes
FAIL_RETURN_VALUE
NEST_ASYNCIO_PATCHED
PYTHON_EXE
- convert_for_csv(val)#
Convert an input value to an accepted format to be saved as a value of a CSV file (e.g., a list becomes it’s str representation).
- Parameters:
val (Any) – The input value to convert.
- Returns:
The converted value.
- Return type:
Any
- static create(run_function, method='serial', method_kwargs={})#
Create evaluator with a specific backend and configuration.
- Parameters:
run_function (function) – the function to execute in parallel.
method (str, optional) – the backend to use in
["serial", "thread", "process", "ray", "mpicomm"]
. Defaults to"serial"
.method_kwargs (dict, optional) – configuration dictionnary of the corresponding backend. Keys corresponds to the keyword arguments of the corresponding implementation. Defaults to “{}”.
- Raises:
ValueError – if the
method is
not acceptable.- Returns:
the
Evaluator
with the corresponding backend and configuration.- Return type:
- decode(key)#
Decode the key following a JSON format to return a dict.
- dump_evals(saved_keys=None, log_dir: str = '.', filename='results.csv')#
Dump evaluations to a CSV file.
- Parameters:
saved_keys (list|callable) – If
None
the wholejob.config
will be added as row of the CSV file. If alist
filtered keys will be added as a row of the CSV file. If acallable
the output dictionnary will be added as a row of the CSV file.log_dir (str) – directory where to dump the CSV file.
filename (str) – name of the file where to write the data.
- async execute(job: Job) Job [source]#
Execute the received job. To be implemented with a specific backend.
- Parameters:
job (Job) – the
Job
to be executed.
- gather(type, size=1)#
Collect the completed tasks from the evaluator in batches of one or more.
- Parameters:
type (str) –
- Options:
"ALL"
Block until all jobs submitted to the evaluator are completed.
"BATCH"
Specify a minimum batch size of jobs to collect from the evaluator. The method will block until at least
size
evaluations are completed.
size (int, optional) – The minimum batch size that we want to collect from the evaluator. Defaults to 1.
- Raises:
Exception – Raised when a gather operation other than “ALL” or “BATCH” is provided.
- Returns:
A batch of completed jobs that is at minimum the given size.
- Return type:
List[Job]
- set_timeout(timeout)#
Set a timeout for the Evaluator. It will create task with a “time budget” and will kill the the task if this budget is exhausted.
- submit(configs: List[Dict])#
Send configurations to be evaluated by available workers.
- Parameters:
configs (List[Dict]) – A list of dict which will be passed to the run function to be executed.
- to_json()#
Returns a json version of the evaluator.