.. DO NOT EDIT.
.. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY.
.. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE:
.. "examples/examples_parallelism/plot_queued_evaluator_mpi.py"
.. LINE NUMBERS ARE GIVEN BELOW.

.. only:: html

    .. note::
        :class: sphx-glr-download-link-note

        :ref:`Go to the end <sphx_glr_download_examples_examples_parallelism_plot_queued_evaluator_mpi.py>`
        to download the full example code.

.. rst-class:: sphx-glr-example-title

.. _sphx_glr_examples_examples_parallelism_plot_queued_evaluator_mpi.py:


Managing Queued Compute Resources Using Evaluator and MPI Executables
=====================================================================

**Author(s)**: Romain Egele, Brett Eiffert.

In this example, you will learn how to use a queued ProcessPoolEvaluator with an mpi run function.

.. GENERATED FROM PYTHON SOURCE LINES 11-25

.. dropdown:: Code (Import statements)

    .. code-block:: Python


        import subprocess

        from deephyper.evaluator import (
            LokyEvaluator,
            parse_subprocess_result,
            profile,
            queued,
        )
        from deephyper.evaluator.callback import TqdmCallback
        from deephyper.hpo import CBO, HpProblem









.. GENERATED FROM PYTHON SOURCE LINES 26-30

Run function using MPI
----------------

Used by the main function to fork and manage mpi processes within the evaluator.

.. GENERATED FROM PYTHON SOURCE LINES 30-43

.. code-block:: Python

    @profile
    def run_mpi_exe(job, dequed=None):
        x = job.parameters["x"]

        # The format of the print "DH-OUTPUT:(.+)\n" is strict if you use parse_suprocess_result
        command = f"mpirun -np {len(dequed)} echo DH-OUTPUT:{x}\n"
        completed_process = subprocess.run(command.split(), capture_output=True)
        objective = parse_subprocess_result(completed_process)

        # In the results.csv a new `m:dequed` will show the passed dequed values.
        return {"objective": objective, "metadata": {}}









.. GENERATED FROM PYTHON SOURCE LINES 44-48

Setup
-----

The problem is defined with basic hyperparameters for a straightforward example.

.. GENERATED FROM PYTHON SOURCE LINES 48-52

.. code-block:: Python


    problem = HpProblem()
    problem.add_hyperparameter((0.0, 10.0), "x")





.. rst-class:: sphx-glr-script-out

 .. code-block:: none


    UniformFloatHyperparameter(name='x', default_value=5.0, meta=None, size=inf, lower=0.0, upper=10.0, log=False)



.. GENERATED FROM PYTHON SOURCE LINES 53-56

Variables used for selecting the number of workers to execute the pool of mpi workers.
These are defined for show and can be run in a multi-node setup or on a single node or local machine.
Number of processes spawned = num_nodes / num_nodes_per_task

.. GENERATED FROM PYTHON SOURCE LINES 56-65

.. code-block:: Python


    # Local machine or single node
    num_nodes = 1
    num_nodes_per_task = 1

    # Multi-node setup (uncomment to try)
    # num_nodes = 10
    # num_nodes_per_task = 2








.. GENERATED FROM PYTHON SOURCE LINES 66-74

Parallel Processing
-------------------

We define a main function which sets up an mpi enabled evaluator object to be used to evaluate the model in parallel. Tasks are spawned in the run_mpi_exe function that was defined earlier and queued in a ``LokyEvaluator``.

Using the evaluator (``LokyEvaluator``), the search is performed for a user defined number of iterations (50).
``LokyEvaluator`` was chosen over other deephyper evaluators ``ProcessPoolEvaluator`` and ``ThreadPoolEvaluator`` due to the preference of running MPI processes and the necessity of argument based process spawning required by notebook-style runtimes.
 To read more about the evaluator backend options and how to choose the best on for a specific use case, go to (coming soon). 

.. GENERATED FROM PYTHON SOURCE LINES 74-92

.. code-block:: Python


    def main():
        evaluator = queued(LokyEvaluator)(
            run_function=run_mpi_exe,
            num_workers=num_nodes // num_nodes_per_task,
            callbacks=[TqdmCallback()],
            queue=[node_id for node_id in range(num_nodes)],
            queue_pop_per_task=num_nodes_per_task,
        )

        print(f"Evaluator uses {evaluator.num_workers} workers")

        search = CBO(problem, log_dir="log_queued_evaluator")
    
        search.search(evaluator, max_evals=50)

    if __name__ == "__main__":
        main()




.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    Evaluator uses 1 workers
    Results file already exists, it will be renamed to /Users/rp5/Documents/DeepHyper/deephyper/examples/examples_parallelism/log_queued_evaluator/results_20250818-151837.csv

      0%|          | 0/50 [00:00<?, ?it/s]
      2%|▏         | 1/50 [00:00<00:00, 23301.69it/s, failures=0, objective=8.34]
      4%|▍         | 2/50 [00:00<00:01, 24.16it/s, failures=0, objective=8.34]   
      6%|▌         | 3/50 [00:00<00:02, 20.60it/s, failures=0, objective=8.34]
      6%|▌         | 3/50 [00:00<00:02, 20.60it/s, failures=0, objective=8.34]
      8%|▊         | 4/50 [00:00<00:02, 20.60it/s, failures=0, objective=9.9] 
     10%|█         | 5/50 [00:00<00:02, 20.60it/s, failures=0, objective=9.9]
     12%|█▏        | 6/50 [00:01<00:08,  5.00it/s, failures=0, objective=9.9]
     12%|█▏        | 6/50 [00:01<00:08,  5.00it/s, failures=0, objective=9.9]
     14%|█▍        | 7/50 [00:01<00:08,  5.00it/s, failures=0, objective=9.9]
     16%|█▌        | 8/50 [00:01<00:09,  4.37it/s, failures=0, objective=9.9]
     16%|█▌        | 8/50 [00:01<00:09,  4.37it/s, failures=0, objective=9.9]
     18%|█▊        | 9/50 [00:01<00:09,  4.17it/s, failures=0, objective=9.9]
     18%|█▊        | 9/50 [00:01<00:09,  4.17it/s, failures=0, objective=9.9]
     20%|██        | 10/50 [00:02<00:09,  4.02it/s, failures=0, objective=9.9]
     20%|██        | 10/50 [00:02<00:09,  4.02it/s, failures=0, objective=9.9]
     22%|██▏       | 11/50 [00:02<00:09,  3.90it/s, failures=0, objective=9.9]
     22%|██▏       | 11/50 [00:02<00:09,  3.90it/s, failures=0, objective=9.9]
     24%|██▍       | 12/50 [00:02<00:10,  3.60it/s, failures=0, objective=9.9]
     24%|██▍       | 12/50 [00:02<00:10,  3.60it/s, failures=0, objective=9.9]
     26%|██▌       | 13/50 [00:03<00:10,  3.58it/s, failures=0, objective=9.9]
     26%|██▌       | 13/50 [00:03<00:10,  3.58it/s, failures=0, objective=9.9]
     28%|██▊       | 14/50 [00:03<00:10,  3.56it/s, failures=0, objective=9.9]
     28%|██▊       | 14/50 [00:03<00:10,  3.56it/s, failures=0, objective=9.9]
     30%|███       | 15/50 [00:03<00:09,  3.56it/s, failures=0, objective=9.9]
     30%|███       | 15/50 [00:03<00:09,  3.56it/s, failures=0, objective=9.9]
     32%|███▏      | 16/50 [00:03<00:09,  3.54it/s, failures=0, objective=9.9]
     32%|███▏      | 16/50 [00:03<00:09,  3.54it/s, failures=0, objective=9.9]
     34%|███▍      | 17/50 [00:04<00:09,  3.55it/s, failures=0, objective=9.9]
     34%|███▍      | 17/50 [00:04<00:09,  3.55it/s, failures=0, objective=9.9]
     36%|███▌      | 18/50 [00:04<00:09,  3.34it/s, failures=0, objective=9.9]
     36%|███▌      | 18/50 [00:04<00:09,  3.34it/s, failures=0, objective=9.96]
     38%|███▊      | 19/50 [00:04<00:09,  3.40it/s, failures=0, objective=9.96]
     38%|███▊      | 19/50 [00:04<00:09,  3.40it/s, failures=0, objective=9.96]
     40%|████      | 20/50 [00:05<00:08,  3.42it/s, failures=0, objective=9.96]
     40%|████      | 20/50 [00:05<00:08,  3.42it/s, failures=0, objective=9.96]
     42%|████▏     | 21/50 [00:05<00:08,  3.46it/s, failures=0, objective=9.96]
     42%|████▏     | 21/50 [00:05<00:08,  3.46it/s, failures=0, objective=9.97]
     44%|████▍     | 22/50 [00:05<00:07,  3.50it/s, failures=0, objective=9.97]
     44%|████▍     | 22/50 [00:05<00:07,  3.50it/s, failures=0, objective=9.97]
     46%|████▌     | 23/50 [00:05<00:07,  3.52it/s, failures=0, objective=9.97]
     46%|████▌     | 23/50 [00:05<00:07,  3.52it/s, failures=0, objective=9.97]
     48%|████▊     | 24/50 [00:06<00:07,  3.34it/s, failures=0, objective=9.97]
     48%|████▊     | 24/50 [00:06<00:07,  3.34it/s, failures=0, objective=9.97]
     50%|█████     | 25/50 [00:06<00:07,  3.40it/s, failures=0, objective=9.97]
     50%|█████     | 25/50 [00:06<00:07,  3.40it/s, failures=0, objective=9.97]
     52%|█████▏    | 26/50 [00:06<00:06,  3.45it/s, failures=0, objective=9.97]
     52%|█████▏    | 26/50 [00:06<00:06,  3.45it/s, failures=0, objective=9.97]
     54%|█████▍    | 27/50 [00:07<00:06,  3.50it/s, failures=0, objective=9.97]
     54%|█████▍    | 27/50 [00:07<00:06,  3.50it/s, failures=0, objective=9.97]
     56%|█████▌    | 28/50 [00:07<00:06,  3.34it/s, failures=0, objective=9.97]
     56%|█████▌    | 28/50 [00:07<00:06,  3.34it/s, failures=0, objective=10]  
     58%|█████▊    | 29/50 [00:07<00:06,  3.22it/s, failures=0, objective=10]
     58%|█████▊    | 29/50 [00:07<00:06,  3.22it/s, failures=0, objective=10]
     60%|██████    | 30/50 [00:08<00:06,  3.30it/s, failures=0, objective=10]
     60%|██████    | 30/50 [00:08<00:06,  3.30it/s, failures=0, objective=10]
     62%|██████▏   | 31/50 [00:08<00:05,  3.35it/s, failures=0, objective=10]
     62%|██████▏   | 31/50 [00:08<00:05,  3.35it/s, failures=0, objective=10]
     64%|██████▍   | 32/50 [00:08<00:05,  3.40it/s, failures=0, objective=10]
     64%|██████▍   | 32/50 [00:08<00:05,  3.40it/s, failures=0, objective=10]
     66%|██████▌   | 33/50 [00:08<00:04,  3.45it/s, failures=0, objective=10]
     66%|██████▌   | 33/50 [00:08<00:04,  3.45it/s, failures=0, objective=10]
     68%|██████▊   | 34/50 [00:09<00:04,  3.48it/s, failures=0, objective=10]
     68%|██████▊   | 34/50 [00:09<00:04,  3.48it/s, failures=0, objective=10]
     70%|███████   | 35/50 [00:09<00:04,  3.31it/s, failures=0, objective=10]
     70%|███████   | 35/50 [00:09<00:04,  3.31it/s, failures=0, objective=10]
     72%|███████▏  | 36/50 [00:09<00:04,  3.38it/s, failures=0, objective=10]
     72%|███████▏  | 36/50 [00:09<00:04,  3.38it/s, failures=0, objective=10]
     74%|███████▍  | 37/50 [00:10<00:03,  3.44it/s, failures=0, objective=10]
     74%|███████▍  | 37/50 [00:10<00:03,  3.44it/s, failures=0, objective=10]
     76%|███████▌  | 38/50 [00:10<00:03,  3.47it/s, failures=0, objective=10]
     76%|███████▌  | 38/50 [00:10<00:03,  3.47it/s, failures=0, objective=10]
     78%|███████▊  | 39/50 [00:10<00:03,  3.51it/s, failures=0, objective=10]
     78%|███████▊  | 39/50 [00:10<00:03,  3.51it/s, failures=0, objective=10]
     80%|████████  | 40/50 [00:10<00:02,  3.52it/s, failures=0, objective=10]
     80%|████████  | 40/50 [00:10<00:02,  3.52it/s, failures=0, objective=10]
     82%|████████▏ | 41/50 [00:11<00:02,  3.33it/s, failures=0, objective=10]
     82%|████████▏ | 41/50 [00:11<00:02,  3.33it/s, failures=0, objective=10]
     84%|████████▍ | 42/50 [00:11<00:02,  3.39it/s, failures=0, objective=10]
     84%|████████▍ | 42/50 [00:11<00:02,  3.39it/s, failures=0, objective=10]
     86%|████████▌ | 43/50 [00:11<00:02,  3.45it/s, failures=0, objective=10]
     86%|████████▌ | 43/50 [00:11<00:02,  3.45it/s, failures=0, objective=10]
     88%|████████▊ | 44/50 [00:12<00:01,  3.48it/s, failures=0, objective=10]
     88%|████████▊ | 44/50 [00:12<00:01,  3.48it/s, failures=0, objective=10]
     90%|█████████ | 45/50 [00:12<00:01,  3.52it/s, failures=0, objective=10]
     90%|█████████ | 45/50 [00:12<00:01,  3.52it/s, failures=0, objective=10]
     92%|█████████▏| 46/50 [00:12<00:01,  3.33it/s, failures=0, objective=10]
     92%|█████████▏| 46/50 [00:12<00:01,  3.33it/s, failures=0, objective=10]
     94%|█████████▍| 47/50 [00:13<00:00,  3.41it/s, failures=0, objective=10]
     94%|█████████▍| 47/50 [00:13<00:00,  3.41it/s, failures=0, objective=10]
     96%|█████████▌| 48/50 [00:13<00:00,  3.46it/s, failures=0, objective=10]
     96%|█████████▌| 48/50 [00:13<00:00,  3.46it/s, failures=0, objective=10]
     98%|█████████▊| 49/50 [00:13<00:00,  3.48it/s, failures=0, objective=10]
     98%|█████████▊| 49/50 [00:13<00:00,  3.48it/s, failures=0, objective=10]
    100%|██████████| 50/50 [00:13<00:00,  3.52it/s, failures=0, objective=10]
    100%|██████████| 50/50 [00:13<00:00,  3.52it/s, failures=0, objective=10]
    100%|██████████| 50/50 [00:13<00:00,  3.60it/s, failures=0, objective=10]





.. rst-class:: sphx-glr-timing

   **Total running time of the script:** (0 minutes 17.525 seconds)


.. _sphx_glr_download_examples_examples_parallelism_plot_queued_evaluator_mpi.py:

.. only:: html

  .. container:: sphx-glr-footer sphx-glr-footer-example

    .. container:: sphx-glr-download sphx-glr-download-jupyter

      :download:`Download Jupyter notebook: plot_queued_evaluator_mpi.ipynb <plot_queued_evaluator_mpi.ipynb>`

    .. container:: sphx-glr-download sphx-glr-download-python

      :download:`Download Python source code: plot_queued_evaluator_mpi.py <plot_queued_evaluator_mpi.py>`

    .. container:: sphx-glr-download sphx-glr-download-zip

      :download:`Download zipped: plot_queued_evaluator_mpi.zip <plot_queued_evaluator_mpi.zip>`


.. only:: html

 .. rst-class:: sphx-glr-signature

    `Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_