From Serial to Parallel Evaluations#

Author(s): Romain Egele.

This example demonstrates the advantages of parallel evaluations over serial evaluations. We start by defining an artificial black-box run-function by using the Ackley function:

Ackley Function in 2D

We will use the time.sleep function to simulate a budget of 2 secondes of execution in average which helps illustrate the advantage of parallel evaluations. The @profile decorator is useful to collect starting/ending time of the run-function execution which help us know exactly when we are inside the black-box. When using this decorator, the run-function will return a dictionnary with 2 new keys "timestamp_start" and "timestamp_end". The run-function is defined in a separate module because of the “multiprocessing” backend that we are using in this example.

"""Set of Black-Box functions useful to build examples.
"""
import time
import numpy as np
from deephyper.evaluator import profile


def ackley(x, a=20, b=0.2, c=2 * np.pi):
    d = len(x)
    s1 = np.sum(x**2)
    s2 = np.sum(np.cos(c * x))
    term1 = -a * np.exp(-b * np.sqrt(s1 / d))
    term2 = -np.exp(s2 / d)
    y = term1 + term2 + a + np.exp(1)
    return y


@profile
def run_ackley(config, sleep_loc=2, sleep_scale=0.5):

    # to simulate the computation of an expensive black-box
    if sleep_loc > 0:
        t_sleep = np.random.normal(loc=sleep_loc, scale=sleep_scale)
        t_sleep = max(t_sleep, 0)
        time.sleep(t_sleep)

    x = np.array([config[k] for k in config if "x" in k])
    x = np.asarray_chkfinite(x)  # ValueError if any NaN or Inf
    return -ackley(x)  # maximisation is performed

After defining the black-box we can continue with the definition of our main script:

import black_box_util as black_box

Then we define the variable(s) we want to optimize. For this problem we optimize Ackley in a 2-dimensional search space, the true minimul is located at (0, 0).

from deephyper.problem import HpProblem


nb_dim = 2
problem = HpProblem()
for i in range(nb_dim):
    problem.add_hyperparameter((-32.768, 32.768), f"x{i}")
problem

Out:

Configuration space object:
  Hyperparameters:
    x0, Type: UniformFloat, Range: [-32.768, 32.768], Default: 0.0
    x1, Type: UniformFloat, Range: [-32.768, 32.768], Default: 0.0

Then we define serial search by creation a "serial"-evaluator and we execute the search with a fixed time-budget of 2 minutes (i.e., 120 secondes).

if __name__ == "__main__":
    from deephyper.evaluator import Evaluator
    from deephyper.evaluator.callback import TqdmCallback
    from deephyper.search.hps import CBO

    # we give a budget of 2 minutes for each search
    timeout = 120
    serial_evaluator = Evaluator.create(
        black_box.run_ackley,
        method="serial",
        method_kwargs={"callbacks": [TqdmCallback()]},
    )

    results = {}
    serial_search = CBO(problem, serial_evaluator, random_state=42)
    results["serial"] = serial_search.search(timeout=timeout)

Out:

0it [00:00, ?it/s]
1it [00:00, 3269.14it/s, objective=-21.2]
2it [00:02,  1.17s/it, objective=-21.2]
2it [00:02,  1.17s/it, objective=-19.9]
3it [00:04,  1.52s/it, objective=-19.9]
3it [00:04,  1.52s/it, objective=-19.9]
4it [00:06,  1.90s/it, objective=-19.9]
4it [00:06,  1.90s/it, objective=-19.9]
5it [00:09,  2.06s/it, objective=-19.9]
5it [00:09,  2.06s/it, objective=-19.9]
6it [00:10,  1.92s/it, objective=-19.9]
6it [00:10,  1.92s/it, objective=-19.9]
7it [00:12,  1.84s/it, objective=-19.9]
7it [00:12,  1.84s/it, objective=-19.9]
8it [00:14,  1.85s/it, objective=-19.9]
8it [00:14,  1.85s/it, objective=-19.9]
9it [00:16,  2.04s/it, objective=-19.9]
9it [00:16,  2.04s/it, objective=-19.9]
10it [00:19,  2.12s/it, objective=-19.9]
10it [00:19,  2.12s/it, objective=-11.3]
11it [00:21,  2.17s/it, objective=-11.3]
11it [00:21,  2.17s/it, objective=-9.86]
12it [00:23,  2.20s/it, objective=-9.86]
12it [00:23,  2.20s/it, objective=-9.86]
13it [00:24,  1.79s/it, objective=-9.86]
13it [00:24,  1.79s/it, objective=-9.86]
14it [00:26,  1.71s/it, objective=-9.86]
14it [00:26,  1.71s/it, objective=-9.86]
15it [00:28,  2.00s/it, objective=-9.86]
15it [00:28,  2.00s/it, objective=-9.86]
16it [00:30,  1.90s/it, objective=-9.86]
16it [00:30,  1.90s/it, objective=-3.64]
17it [00:33,  2.16s/it, objective=-3.64]
17it [00:33,  2.16s/it, objective=-2.66]
18it [00:36,  2.46s/it, objective=-2.66]
18it [00:36,  2.46s/it, objective=-2.66]
19it [00:38,  2.46s/it, objective=-2.66]
19it [00:38,  2.46s/it, objective=-2.66]
20it [00:40,  2.36s/it, objective=-2.66]
20it [00:40,  2.36s/it, objective=-2.66]
21it [00:43,  2.40s/it, objective=-2.66]
21it [00:43,  2.40s/it, objective=-2.66]
22it [00:46,  2.44s/it, objective=-2.66]
22it [00:46,  2.44s/it, objective=-2.66]
23it [00:47,  2.22s/it, objective=-2.66]
23it [00:47,  2.22s/it, objective=-2.66]
24it [00:49,  2.19s/it, objective=-2.66]
24it [00:49,  2.19s/it, objective=-2.66]
25it [00:51,  2.07s/it, objective=-2.66]
25it [00:51,  2.07s/it, objective=-2.66]
26it [00:53,  2.12s/it, objective=-2.66]
26it [00:53,  2.12s/it, objective=-2.66]
27it [00:56,  2.19s/it, objective=-2.66]
27it [00:56,  2.19s/it, objective=-2.66]
28it [00:59,  2.56s/it, objective=-2.66]
28it [00:59,  2.56s/it, objective=-2.66]
29it [01:00,  2.13s/it, objective=-2.66]
29it [01:00,  2.13s/it, objective=-2.66]
30it [01:02,  2.12s/it, objective=-2.66]
30it [01:02,  2.12s/it, objective=-2.66]
31it [01:04,  2.10s/it, objective=-2.66]
31it [01:04,  2.10s/it, objective=-2.66]
32it [01:07,  2.16s/it, objective=-2.66]
32it [01:07,  2.16s/it, objective=-2.66]
33it [01:08,  1.89s/it, objective=-2.66]
33it [01:08,  1.89s/it, objective=-2.66]
34it [01:10,  1.99s/it, objective=-2.66]
34it [01:10,  1.99s/it, objective=-2.66]
35it [01:12,  2.00s/it, objective=-2.66]
35it [01:12,  2.00s/it, objective=-2.66]
36it [01:15,  2.25s/it, objective=-2.66]
36it [01:15,  2.25s/it, objective=-2.66]
37it [01:17,  2.27s/it, objective=-2.66]
37it [01:17,  2.27s/it, objective=-2.66]
38it [01:19,  2.17s/it, objective=-2.66]
38it [01:19,  2.17s/it, objective=-2.66]
39it [01:22,  2.31s/it, objective=-2.66]
39it [01:22,  2.31s/it, objective=-2.66]
40it [01:24,  2.27s/it, objective=-2.66]
40it [01:24,  2.27s/it, objective=-2.66]
41it [01:27,  2.35s/it, objective=-2.66]
41it [01:27,  2.35s/it, objective=-2.66]
42it [01:29,  2.43s/it, objective=-2.66]
42it [01:29,  2.43s/it, objective=-2.66]
43it [01:31,  2.15s/it, objective=-2.66]
43it [01:31,  2.15s/it, objective=-2.66]
44it [01:33,  2.28s/it, objective=-2.66]
44it [01:33,  2.28s/it, objective=-2.66]
45it [01:36,  2.30s/it, objective=-2.66]
45it [01:36,  2.30s/it, objective=-2.66]
46it [01:38,  2.41s/it, objective=-2.66]
46it [01:38,  2.41s/it, objective=-2.66]
47it [01:40,  2.27s/it, objective=-2.66]
47it [01:40,  2.27s/it, objective=-2.66]
48it [01:43,  2.31s/it, objective=-2.66]
48it [01:43,  2.31s/it, objective=-2.66]
49it [01:45,  2.25s/it, objective=-2.66]
49it [01:45,  2.25s/it, objective=-2.66]
50it [01:48,  2.58s/it, objective=-2.66]
50it [01:48,  2.58s/it, objective=-2.66]
51it [01:50,  2.42s/it, objective=-2.66]
51it [01:50,  2.42s/it, objective=-2.66]
52it [01:52,  2.17s/it, objective=-2.66]
52it [01:52,  2.17s/it, objective=-2.66]
53it [01:55,  2.34s/it, objective=-2.66]
53it [01:55,  2.34s/it, objective=-2.66]
54it [01:57,  2.49s/it, objective=-2.66]
54it [01:57,  2.49s/it, objective=-2.66]

After, executing the serial-search for 2 minutes we can create a parallel search which uses the "process"-evaluator and defines 5 parallel workers. The search is also executed for 2 minutes.

if __name__ == "__main__":
    parallel_evaluator = Evaluator.create(
        black_box.run_ackley,
        method="process",
        method_kwargs={"num_workers": 5, "callbacks": [TqdmCallback()]},
    )

    parallel_search = CBO(problem, parallel_evaluator, random_state=42)
    results["parallel"] = parallel_search.search(timeout=timeout)

Out:

0it [00:00, ?it/s]

1it [00:00, 6990.51it/s, objective=-21.5]

2it [00:00, 18.17it/s, objective=-21.5]

2it [00:00, 18.17it/s, objective=-21.2]

3it [00:00, 18.17it/s, objective=-21.2]

4it [00:00,  6.14it/s, objective=-21.2]

4it [00:00,  6.14it/s, objective=-20.6]

5it [00:01,  3.00it/s, objective=-20.6]

5it [00:01,  3.00it/s, objective=-20.6]

6it [00:02,  1.59it/s, objective=-20.6]

6it [00:02,  1.59it/s, objective=-20.6]

7it [00:02,  1.91it/s, objective=-20.6]

7it [00:02,  1.91it/s, objective=-20.6]

8it [00:03,  1.91it/s, objective=-19.9]

9it [00:03,  2.58it/s, objective=-19.9]

9it [00:03,  2.58it/s, objective=-19.9]

10it [00:04,  1.94it/s, objective=-19.9]

10it [00:04,  1.94it/s, objective=-19.9]

11it [00:04,  1.82it/s, objective=-19.9]

11it [00:04,  1.82it/s, objective=-19.9]

12it [00:05,  1.88it/s, objective=-19.9]

12it [00:05,  1.88it/s, objective=-19.9]

13it [00:05,  1.91it/s, objective=-19.9]

13it [00:05,  1.91it/s, objective=-11.3]

14it [00:06,  2.24it/s, objective=-11.3]

14it [00:06,  2.24it/s, objective=-11.3]

15it [00:07,  1.64it/s, objective=-11.3]

15it [00:07,  1.64it/s, objective=-11.3]

16it [00:07,  1.98it/s, objective=-11.3]

16it [00:07,  1.98it/s, objective=-11.3]

17it [00:08,  1.56it/s, objective=-11.3]

17it [00:08,  1.56it/s, objective=-11.3]

18it [00:08,  1.82it/s, objective=-11.3]

18it [00:08,  1.82it/s, objective=-10.8]

19it [00:09,  2.15it/s, objective=-10.8]

19it [00:09,  2.15it/s, objective=-10.8]

20it [00:09,  1.73it/s, objective=-10.8]

20it [00:09,  1.73it/s, objective=-10.7]

21it [00:10,  1.35it/s, objective=-10.7]

21it [00:10,  1.35it/s, objective=-10.5]

22it [00:11,  1.67it/s, objective=-10.5]

22it [00:11,  1.67it/s, objective=-10.5]

23it [00:11,  1.99it/s, objective=-10.5]

23it [00:11,  1.99it/s, objective=-6.17]

24it [00:12,  1.62it/s, objective=-6.17]

24it [00:12,  1.62it/s, objective=-6.17]

25it [00:13,  1.59it/s, objective=-6.17]

25it [00:13,  1.59it/s, objective=-6.17]

26it [00:13,  1.47it/s, objective=-6.17]

26it [00:13,  1.47it/s, objective=-6.17]

27it [00:14,  1.61it/s, objective=-6.17]

27it [00:14,  1.61it/s, objective=-6.17]

28it [00:14,  1.84it/s, objective=-6.17]

28it [00:14,  1.84it/s, objective=-6.17]

29it [00:15,  1.50it/s, objective=-6.17]

29it [00:15,  1.50it/s, objective=-5.19]

30it [00:16,  1.39it/s, objective=-5.19]

30it [00:16,  1.39it/s, objective=-5.19]

31it [00:16,  1.63it/s, objective=-5.19]

31it [00:16,  1.63it/s, objective=-5.19]

32it [00:17,  1.93it/s, objective=-5.19]

32it [00:17,  1.93it/s, objective=-5.19]

33it [00:18,  1.31it/s, objective=-5.19]

33it [00:18,  1.31it/s, objective=-5.19]

34it [00:18,  1.54it/s, objective=-5.19]

34it [00:18,  1.54it/s, objective=-5.19]

35it [00:19,  1.47it/s, objective=-5.19]

35it [00:19,  1.47it/s, objective=-5.19]

36it [00:19,  1.77it/s, objective=-5.19]

36it [00:19,  1.77it/s, objective=-5.19]

37it [00:20,  1.98it/s, objective=-5.19]

37it [00:20,  1.98it/s, objective=-5.19]

38it [00:21,  1.27it/s, objective=-5.19]

38it [00:21,  1.27it/s, objective=-5.19]

39it [00:22,  1.54it/s, objective=-5.19]

39it [00:22,  1.54it/s, objective=-5.19]

40it [00:22,  1.38it/s, objective=-5.19]

40it [00:22,  1.38it/s, objective=-5.19]

41it [00:23,  1.61it/s, objective=-5.19]

41it [00:23,  1.61it/s, objective=-5.19]

42it [00:23,  1.86it/s, objective=-5.19]

42it [00:23,  1.86it/s, objective=-5.19]

43it [00:24,  1.60it/s, objective=-5.19]

43it [00:24,  1.60it/s, objective=-5.19]

44it [00:24,  1.82it/s, objective=-5.19]

44it [00:24,  1.82it/s, objective=-5.19]

45it [00:25,  1.41it/s, objective=-5.19]

45it [00:25,  1.41it/s, objective=-5.19]

46it [00:26,  1.42it/s, objective=-5.19]

46it [00:26,  1.42it/s, objective=-5.19]

47it [00:27,  1.65it/s, objective=-5.19]

47it [00:27,  1.65it/s, objective=-5.19]

48it [00:27,  1.56it/s, objective=-5.19]

48it [00:27,  1.56it/s, objective=-4.97]

49it [00:28,  1.85it/s, objective=-4.97]

49it [00:28,  1.85it/s, objective=-3.23]

50it [00:28,  2.03it/s, objective=-3.23]

50it [00:28,  2.03it/s, objective=-3.23]

51it [00:29,  1.36it/s, objective=-3.23]

51it [00:29,  1.36it/s, objective=-2.7]

52it [00:30,  1.63it/s, objective=-2.7]

52it [00:30,  1.63it/s, objective=-2.7]

53it [00:30,  1.53it/s, objective=-2.7]

53it [00:30,  1.53it/s, objective=-2.7]

54it [00:31,  1.75it/s, objective=-2.7]

54it [00:31,  1.75it/s, objective=-2.69]

55it [00:31,  1.75it/s, objective=-2.01]

56it [00:32,  1.73it/s, objective=-2.01]

56it [00:32,  1.73it/s, objective=-2.01]

57it [00:32,  1.82it/s, objective=-2.01]

57it [00:32,  1.82it/s, objective=-2.01]

58it [00:33,  1.79it/s, objective=-2.01]

58it [00:33,  1.79it/s, objective=-2.01]

59it [00:33,  1.88it/s, objective=-2.01]

59it [00:33,  1.88it/s, objective=-2.01]

60it [00:34,  1.77it/s, objective=-2.01]

60it [00:34,  1.77it/s, objective=-2.01]

61it [00:34,  1.95it/s, objective=-2.01]

61it [00:34,  1.95it/s, objective=-2.01]

62it [00:35,  2.21it/s, objective=-2.01]

62it [00:35,  2.21it/s, objective=-2.01]

63it [00:35,  2.45it/s, objective=-2.01]

63it [00:35,  2.45it/s, objective=-2.01]

64it [00:36,  1.47it/s, objective=-2.01]

64it [00:36,  1.47it/s, objective=-2.01]

65it [00:37,  1.46it/s, objective=-2.01]

65it [00:37,  1.46it/s, objective=-2.01]

66it [00:37,  1.75it/s, objective=-2.01]

66it [00:37,  1.75it/s, objective=-2.01]

67it [00:38,  1.94it/s, objective=-2.01]

67it [00:38,  1.94it/s, objective=-2.01]

68it [00:38,  2.06it/s, objective=-2.01]

68it [00:38,  2.06it/s, objective=-2.01]

69it [00:39,  1.91it/s, objective=-2.01]

69it [00:39,  1.91it/s, objective=-2.01]

70it [00:40,  1.19it/s, objective=-2.01]

70it [00:40,  1.19it/s, objective=-2.01]

71it [00:41,  1.46it/s, objective=-2.01]

71it [00:41,  1.46it/s, objective=-2.01]

72it [00:41,  1.75it/s, objective=-2.01]

72it [00:41,  1.75it/s, objective=-2.01]

73it [00:41,  1.89it/s, objective=-2.01]

73it [00:41,  1.89it/s, objective=-2.01]

74it [00:42,  2.04it/s, objective=-2.01]

74it [00:42,  2.04it/s, objective=-2.01]

75it [00:44,  1.02it/s, objective=-2.01]

75it [00:44,  1.02it/s, objective=-2.01]

76it [00:44,  1.27it/s, objective=-2.01]

76it [00:44,  1.27it/s, objective=-2.01]

77it [00:45,  1.49it/s, objective=-2.01]

77it [00:45,  1.49it/s, objective=-2.01]

78it [00:45,  1.49it/s, objective=-2.01]

79it [00:45,  1.49it/s, objective=-2.01]

80it [00:47,  1.42it/s, objective=-2.01]

80it [00:47,  1.42it/s, objective=-2.01]

81it [00:47,  1.50it/s, objective=-2.01]

81it [00:47,  1.50it/s, objective=-2.01]

82it [00:48,  1.72it/s, objective=-2.01]

82it [00:48,  1.72it/s, objective=-2.01]

83it [00:48,  1.70it/s, objective=-2.01]

83it [00:48,  1.70it/s, objective=-2.01]

84it [00:49,  1.87it/s, objective=-2.01]

84it [00:49,  1.87it/s, objective=-2.01]

85it [00:50,  1.47it/s, objective=-2.01]

85it [00:50,  1.47it/s, objective=-2.01]

86it [00:51,  1.36it/s, objective=-2.01]

86it [00:51,  1.36it/s, objective=-2.01]

87it [00:51,  1.58it/s, objective=-2.01]

87it [00:51,  1.58it/s, objective=-2.01]

88it [00:51,  1.70it/s, objective=-2.01]

88it [00:51,  1.70it/s, objective=-2.01]

89it [00:52,  1.96it/s, objective=-2.01]

89it [00:52,  1.96it/s, objective=-2.01]

90it [00:52,  2.21it/s, objective=-2.01]

90it [00:52,  2.21it/s, objective=-2.01]

91it [00:54,  1.31it/s, objective=-2.01]

91it [00:54,  1.31it/s, objective=-2.01]

92it [00:54,  1.48it/s, objective=-2.01]

92it [00:54,  1.48it/s, objective=-2.01]

93it [00:54,  1.77it/s, objective=-2.01]

93it [00:54,  1.77it/s, objective=-2.01]

94it [00:55,  1.94it/s, objective=-2.01]

94it [00:55,  1.94it/s, objective=-2.01]

95it [00:56,  1.66it/s, objective=-2.01]

95it [00:56,  1.66it/s, objective=-2.01]

96it [00:57,  1.39it/s, objective=-2.01]

96it [00:57,  1.39it/s, objective=-2.01]

97it [00:57,  1.54it/s, objective=-2.01]

97it [00:57,  1.54it/s, objective=-2.01]

98it [00:57,  1.74it/s, objective=-2.01]

98it [00:57,  1.74it/s, objective=-2.01]

99it [00:58,  2.02it/s, objective=-2.01]

99it [00:58,  2.02it/s, objective=-2.01]

100it [00:59,  1.50it/s, objective=-2.01]

100it [00:59,  1.50it/s, objective=-2.01]

101it [00:59,  1.70it/s, objective=-2.01]

101it [00:59,  1.70it/s, objective=-2.01]

102it [01:00,  1.63it/s, objective=-2.01]

102it [01:00,  1.63it/s, objective=-2.01]

103it [01:00,  1.91it/s, objective=-2.01]

103it [01:00,  1.91it/s, objective=-2.01]

104it [01:01,  1.56it/s, objective=-2.01]

104it [01:01,  1.56it/s, objective=-2.01]

105it [01:01,  1.83it/s, objective=-2.01]

105it [01:01,  1.83it/s, objective=-2.01]

106it [01:03,  1.23it/s, objective=-2.01]

106it [01:03,  1.23it/s, objective=-2.01]

107it [01:03,  1.39it/s, objective=-2.01]

107it [01:03,  1.39it/s, objective=-2.01]

108it [01:04,  1.55it/s, objective=-2.01]

108it [01:04,  1.55it/s, objective=-.378]

109it [01:04,  1.83it/s, objective=-.378]

109it [01:04,  1.83it/s, objective=-.378]

110it [01:04,  1.83it/s, objective=-.378]

111it [01:06,  1.47it/s, objective=-.378]

111it [01:06,  1.47it/s, objective=-.378]

112it [01:07,  1.37it/s, objective=-.378]

112it [01:07,  1.37it/s, objective=-.378]

113it [01:07,  1.60it/s, objective=-.378]

113it [01:07,  1.60it/s, objective=-.378]

114it [01:07,  1.78it/s, objective=-.378]

114it [01:07,  1.78it/s, objective=-.0826]

115it [01:08,  2.02it/s, objective=-.0826]

115it [01:08,  2.02it/s, objective=-.0826]

116it [01:09,  1.33it/s, objective=-.0826]

116it [01:09,  1.33it/s, objective=-.0826]

117it [01:10,  1.46it/s, objective=-.0826]

117it [01:10,  1.46it/s, objective=-.0826]

118it [01:10,  1.74it/s, objective=-.0826]

118it [01:10,  1.74it/s, objective=-.0826]

119it [01:10,  2.00it/s, objective=-.0826]

119it [01:10,  2.00it/s, objective=-.0826]

120it [01:10,  2.00it/s, objective=-.0826]

121it [01:12,  1.41it/s, objective=-.0826]

121it [01:12,  1.41it/s, objective=-.0826]

122it [01:13,  1.61it/s, objective=-.0826]

122it [01:13,  1.61it/s, objective=-.0826]

123it [01:14,  1.31it/s, objective=-.0826]

123it [01:14,  1.31it/s, objective=-.0826]

124it [01:14,  1.51it/s, objective=-.0826]

124it [01:14,  1.51it/s, objective=-.0826]

125it [01:14,  1.51it/s, objective=-.0826]

126it [01:15,  1.51it/s, objective=-.0826]

126it [01:15,  1.51it/s, objective=-.0826]

127it [01:16,  1.72it/s, objective=-.0826]

127it [01:16,  1.72it/s, objective=-.0826]

128it [01:17,  1.53it/s, objective=-.0826]

128it [01:17,  1.53it/s, objective=-.0826]

129it [01:17,  1.44it/s, objective=-.0826]

129it [01:17,  1.44it/s, objective=-.0826]

130it [01:18,  1.70it/s, objective=-.0826]

130it [01:18,  1.70it/s, objective=-.0826]

131it [01:19,  1.42it/s, objective=-.0826]

131it [01:19,  1.42it/s, objective=-.0826]

132it [01:19,  1.57it/s, objective=-.0826]

132it [01:19,  1.57it/s, objective=-.0826]

133it [01:20,  1.74it/s, objective=-.0826]

133it [01:20,  1.74it/s, objective=-.0826]

134it [01:20,  1.91it/s, objective=-.0826]

134it [01:20,  1.91it/s, objective=-.0826]

135it [01:20,  2.17it/s, objective=-.0826]

135it [01:20,  2.17it/s, objective=-.0826]

136it [01:22,  1.31it/s, objective=-.0826]

136it [01:22,  1.31it/s, objective=-.0826]

137it [01:22,  1.57it/s, objective=-.0826]

137it [01:22,  1.57it/s, objective=-.0826]

138it [01:23,  1.77it/s, objective=-.0826]

138it [01:23,  1.77it/s, objective=-.0826]

139it [01:23,  1.72it/s, objective=-.0826]

139it [01:23,  1.72it/s, objective=-.0826]

140it [01:24,  1.72it/s, objective=-.0826]

140it [01:24,  1.72it/s, objective=-.0826]

141it [01:24,  1.65it/s, objective=-.0826]

141it [01:24,  1.65it/s, objective=-.0826]

142it [01:25,  1.66it/s, objective=-.0826]

142it [01:25,  1.66it/s, objective=-.0826]

143it [01:26,  1.60it/s, objective=-.0826]

143it [01:26,  1.60it/s, objective=-.0826]

144it [01:26,  1.80it/s, objective=-.0826]

144it [01:26,  1.80it/s, objective=-.0826]

145it [01:27,  1.71it/s, objective=-.0826]

145it [01:27,  1.71it/s, objective=-.0826]

146it [01:27,  1.98it/s, objective=-.0826]

146it [01:27,  1.98it/s, objective=-.0826]

147it [01:28,  1.82it/s, objective=-.0826]

147it [01:28,  1.82it/s, objective=-.0826]

148it [01:29,  1.41it/s, objective=-.0826]

148it [01:29,  1.41it/s, objective=-.0826]

149it [01:29,  1.70it/s, objective=-.0826]

149it [01:29,  1.70it/s, objective=-.0826]

150it [01:30,  1.69it/s, objective=-.0826]

150it [01:30,  1.69it/s, objective=-.0826]

151it [01:30,  1.71it/s, objective=-.0826]

151it [01:30,  1.71it/s, objective=-.0826]

152it [01:31,  1.95it/s, objective=-.0826]

152it [01:31,  1.95it/s, objective=-.0826]

153it [01:33,  1.09it/s, objective=-.0826]

153it [01:33,  1.09it/s, objective=-.0826]

154it [01:33,  1.30it/s, objective=-.0826]

154it [01:33,  1.30it/s, objective=-.0826]

155it [01:33,  1.30it/s, objective=-.0826]

156it [01:33,  1.30it/s, objective=-.0826]

157it [01:34,  2.19it/s, objective=-.0826]

157it [01:34,  2.19it/s, objective=-.0826]

158it [01:36,  1.28it/s, objective=-.0826]

158it [01:36,  1.28it/s, objective=-.0826]

159it [01:36,  1.48it/s, objective=-.0826]

159it [01:36,  1.48it/s, objective=-.0826]

160it [01:36,  1.61it/s, objective=-.0826]

160it [01:36,  1.61it/s, objective=-.0826]

161it [01:37,  1.77it/s, objective=-.0826]

161it [01:37,  1.77it/s, objective=-.0826]

162it [01:37,  2.01it/s, objective=-.0826]

162it [01:37,  2.01it/s, objective=-.0826]

163it [01:38,  1.57it/s, objective=-.0826]

163it [01:38,  1.57it/s, objective=-.0826]

164it [01:39,  1.27it/s, objective=-.0826]

164it [01:39,  1.27it/s, objective=-.0826]

165it [01:40,  1.47it/s, objective=-.0826]

165it [01:40,  1.47it/s, objective=-.0826]

166it [01:40,  1.72it/s, objective=-.0826]

166it [01:40,  1.72it/s, objective=-.0826]

167it [01:40,  1.90it/s, objective=-.0826]

167it [01:40,  1.90it/s, objective=-.0826]

168it [01:41,  2.04it/s, objective=-.0826]

168it [01:41,  2.04it/s, objective=-.0826]

169it [01:43,  1.02s/it, objective=-.0826]

169it [01:43,  1.02s/it, objective=-.0826]

170it [01:43,  1.22it/s, objective=-.0826]

170it [01:43,  1.22it/s, objective=-.0826]

171it [01:43,  1.22it/s, objective=-.0826]

172it [01:44,  1.75it/s, objective=-.0826]

172it [01:44,  1.75it/s, objective=-.0826]

173it [01:44,  1.96it/s, objective=-.0826]

173it [01:44,  1.96it/s, objective=-.0826]

174it [01:45,  1.54it/s, objective=-.0826]

174it [01:45,  1.54it/s, objective=-.0826]

175it [01:46,  1.30it/s, objective=-.0826]

175it [01:46,  1.30it/s, objective=-.0826]

176it [01:47,  1.56it/s, objective=-.0826]

176it [01:47,  1.56it/s, objective=-.0826]

177it [01:47,  1.70it/s, objective=-.0826]

177it [01:47,  1.70it/s, objective=-.0826]

178it [01:48,  1.74it/s, objective=-.0826]

178it [01:48,  1.74it/s, objective=-.0826]

179it [01:48,  2.00it/s, objective=-.0826]

179it [01:48,  2.00it/s, objective=-.0826]

180it [01:49,  1.75it/s, objective=-.0826]

180it [01:49,  1.75it/s, objective=-.0826]

181it [01:50,  1.55it/s, objective=-.0826]

181it [01:50,  1.55it/s, objective=-.0826]

182it [01:50,  1.53it/s, objective=-.0826]

182it [01:50,  1.53it/s, objective=-.0826]

183it [01:51,  1.60it/s, objective=-.0826]

183it [01:51,  1.60it/s, objective=-.0826]

184it [01:52,  1.58it/s, objective=-.0826]

184it [01:52,  1.58it/s, objective=-.0826]

185it [01:52,  1.43it/s, objective=-.0826]

185it [01:52,  1.43it/s, objective=-.0826]

186it [01:53,  1.66it/s, objective=-.0826]

186it [01:53,  1.66it/s, objective=-.0826]

187it [01:53,  1.66it/s, objective=-.0826]

187it [01:53,  1.66it/s, objective=-.0826]

188it [01:54,  1.84it/s, objective=-.0826]

188it [01:54,  1.84it/s, objective=-.0826]

189it [01:55,  1.23it/s, objective=-.0826]

189it [01:55,  1.23it/s, objective=-.0826]

190it [01:56,  1.48it/s, objective=-.0826]

190it [01:56,  1.48it/s, objective=-.0826]

191it [01:56,  1.48it/s, objective=-.0826]

192it [01:57,  1.65it/s, objective=-.0826]

192it [01:57,  1.65it/s, objective=-.0826]

193it [01:57,  1.80it/s, objective=-.0826]

193it [01:57,  1.80it/s, objective=-.0826]

Finally, we plot the results from the collected DataFrame. The execution time is used as the x-axis which help-us vizualise the advantages of the parallel search.

if __name__ == "__main__":
    import matplotlib.pyplot as plt

    plt.figure()

    for strategy, df in results.items():
        plt.scatter(df.timestamp_end, df.objective, label=strategy)
        plt.plot(df.timestamp_end, df.objective.cummax())

    plt.xlabel("Time (sec.)")
    plt.ylabel("Objective")
    plt.grid()
    plt.legend()
    plt.show()
plot from serial to parallel hyperparameter search

Total running time of the script: ( 4 minutes 1.933 seconds)

Gallery generated by Sphinx-Gallery