.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "examples/examples_uq/plot_hpo_tree_ensemble_uq_classification_sklearn.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note :ref:`Go to the end ` to download the full example code. .. rst-class:: sphx-glr-example-title .. _sphx_glr_examples_examples_uq_plot_hpo_tree_ensemble_uq_classification_sklearn.py: Hyperparameter Optimized Ensemble of Random Decision Trees with Uncertainty for Classification ============================================================================================== **Author(s)**: Romain Egele. In this tutorial, you will learn about how to use hyperparameter optimization to generate an ensemble of `Scikit-Learn `_ models that can be used for uncertainty quantification. .. GENERATED FROM PYTHON SOURCE LINES 11-20 Installation and imports ------------------------ Installing dependencies with the :ref:`pip installation ` is recommended. It requires **Python >= 3.10**. .. code-block:: bash %%bash pip install "deephyper[ray]" .. GENERATED FROM PYTHON SOURCE LINES 22-40 .. dropdown:: Code (Import statements) .. code-block:: Python import pathlib import pickle import os import matplotlib.pyplot as plt import numpy as np from sklearn.calibration import CalibrationDisplay from sklearn.datasets import make_moons from sklearn.metrics import log_loss, accuracy_score from sklearn.model_selection import train_test_split from sklearn.tree import DecisionTreeClassifier WIDTH_PLOTS = 8 HEIGHT_PLOTS = WIDTH_PLOTS / 1.618 .. GENERATED FROM PYTHON SOURCE LINES 41-47 Synthetic data generation ------------------------- For the data, we use the :func:`sklearn.datasets.make_moons` functionality from Scikit-Learn to have a synthetic binary-classification problem with two moons. The input data :math:`x` are two dimensionnal and the target data :math:`y` are binary values. We randomly flip 10% of the labels to generate artificial noise that should later be estimated by what we call "aleatoric uncertainty" (a.k.a., intrinsic random noise). .. GENERATED FROM PYTHON SOURCE LINES 47-109 .. dropdown:: Code (Loading synthetic data) .. code-block:: Python def flip_binary_labels(y, ratio, random_state=None): """Increase the variance of P(Y|X) by ``ratio``""" y_flipped = np.zeros(np.shape(y)) y_flipped[:] = y[:] rs = np.random.RandomState(random_state) idx = np.arange(len(y_flipped)) idx = rs.choice(idx, size=int(ratio * len(y_flipped)), replace=False) y_flipped[idx] = 1 - y_flipped[idx] return y_flipped def load_data(noise=0.1, n=1_000, ratio_flipped=0.1, test_size=0.33, valid_size=0.33, random_state=42): rng = np.random.RandomState(random_state) max_int = np.iinfo(np.int32).max test_size = int(test_size * n) valid_size = int(valid_size * n) X, y = make_moons(n_samples=n, noise=noise, shuffle=True, random_state=rng.randint(max_int)) X = X - np.mean(X, axis=0) y = flip_binary_labels(y, ratio=ratio_flipped, random_state=rng.randint(max_int)) y = y.astype(np.int64) train_X, test_X, train_y, test_y = train_test_split( X, y, test_size=test_size, random_state=rng.randint(max_int), stratify=y, ) train_X, valid_X, train_y, valid_y = train_test_split( train_X, train_y, test_size=valid_size, random_state=rng.randint(max_int), stratify=train_y, ) return (train_X, train_y), (valid_X, valid_y), (test_X, test_y) (x, y), (vx, vy), (tx, ty) = load_data() _ = plt.subplots(figsize=(WIDTH_PLOTS, HEIGHT_PLOTS), tight_layout=True) _ = plt.scatter( x[:, 0].reshape(-1), x[:, 1].reshape(-1), c=y, label="train", alpha=0.8 ) _ = plt.scatter( vx[:, 0].reshape(-1), vx[:, 1].reshape(-1), c=vy, marker="s", label="valid", alpha=0.8, ) _ = plt.ylabel("$x1$", fontsize=12) _ = plt.xlabel("$x0$", fontsize=12) _ = plt.legend(loc="upper center", ncol=3, fontsize=12) .. image-sg:: /examples/examples_uq/images/sphx_glr_plot_hpo_tree_ensemble_uq_classification_sklearn_001.png :alt: plot hpo tree ensemble uq classification sklearn :srcset: /examples/examples_uq/images/sphx_glr_plot_hpo_tree_ensemble_uq_classification_sklearn_001.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 110-124 Training a Decision Tree ------------------------ We focus on the class of random decision tree models. We define a function that trains and evaluate a random decision tree from given parameters ``job.parameters``. These parameters will be optimized in the next steps by DeepHyper. The score we minimize with respect to hyperparameters :math:`\theta` is the validation log loss (a.k.a., binary cross entropy) as we want to have calibrated uncertainty estimates of :math:`P(Y|X=x)` and :math:`1-P(Y|X=x)`: .. math:: L_\text{BCE}(x, y;\theta) = y \cdot \log\left(p(y|x;\theta)\right) + (1 - y) \cdot \log\left(1 - p(y|x\theta)\right) where :math:`p(y|x;\theta)` is the predited probability of a tree with hyperparameters :math:`\theta`. .. GENERATED FROM PYTHON SOURCE LINES 124-152 .. dropdown:: Code (Plot decision boundary) .. code-block:: Python def plot_decision_boundary_decision_tree(dataset, labels, model, steps=1000, color_map="viridis", ax=None): color_map = plt.get_cmap(color_map) # Define region of interest by data limits xmin, xmax = dataset[:, 0].min() - 1, dataset[:, 0].max() + 1 ymin, ymax = dataset[:, 1].min() - 1, dataset[:, 1].max() + 1 x_span = np.linspace(xmin, xmax, steps) y_span = np.linspace(ymin, ymax, steps) xx, yy = np.meshgrid(x_span, y_span) # Make predictions across region of interest labels_predicted = model.predict_proba(np.c_[xx.ravel(), yy.ravel()]) # Plot decision boundary in region of interest z = labels_predicted[:, 1].reshape(xx.shape) ax.contourf(xx, yy, z, cmap=color_map, alpha=0.5) # Get predicted labels on training data and plot ax.scatter( dataset[:, 0], dataset[:, 1], c=labels, # cmap=color_map, lw=0, ) .. GENERATED FROM PYTHON SOURCE LINES 153-159 The ``run`` function takes a ``job`` object as input suggested by DeepHyper. We use it to pass the ``job.parameters`` and create the decision tree ``model``. Then, we fit the model on the data on compute its log-loss score on the validation dataset. In case of unexpected error we return a special value ``F_fit`` so that our hyperparameter optimization can learn to avoid these unexepected failures. We checkpoint the model on disk as ``model_*.pkl`` files. Finally, we return all of our scores, the ``"objective"`` is the value maximized by DeepHyper. Other scores are returned as metadata for further analysis (e.g., overfitting, underfitting, etc.). .. GENERATED FROM PYTHON SOURCE LINES 159-198 .. code-block:: Python hpo_dir = "hpo_sklearn_classification" model_checkpoint_dir = os.path.join(hpo_dir, "models") def run(job, model_checkpoint_dir=".", verbose=True, show_plots=False): (x, y), (vx, vy), (tx, ty) = load_data() model = DecisionTreeClassifier(**job.parameters) try: model.fit(x, y) vy_pred_proba = model.predict_proba(vx) val_cce = log_loss(vy, vy_pred_proba) except: return "F_fit" # Saving the model with open(os.path.join(model_checkpoint_dir, f"model_{job.id}.pkl"), "wb") as f: pickle.dump(model, f) if verbose: print(f"{job.id}: {val_cce=:.3f}") if show_plots: fig, axes = plt.subplots(nrows=2, ncols=1, figsize=(WIDTH_PLOTS, HEIGHT_PLOTS*2), tight_layout=True) plot_decision_boundary_decision_tree(tx, ty, model, steps=1000, color_map="viridis", ax=axes[0]) disp = CalibrationDisplay.from_predictions(ty, model.predict_proba(tx)[:, 1], ax=axes[1]) test_cce = log_loss(ty, model.predict_proba(tx)) test_acc = accuracy_score(ty, model.predict(tx)) # The score is negated for maximization # The score is -Categorical Cross Entropy/LogLoss return { "objective": -val_cce, "metadata": {"test_cce": test_cce, "test_acc": test_acc}, } .. GENERATED FROM PYTHON SOURCE LINES 199-204 It is important to note that we did not fix the random state of the random decision tree. The hyperparameter optimization takes into consideration the fact that the observed objective is noisy and of course this can be tuned. For example, as the default surrogate model of DeepHyper is itself a randomized forest, increasing the number of samples in leaf nodes would have the effect of averaging out the prediction of the surrogate. Also, the point of ensembling randomized decision trees is to build a model with lower variance (i.e., variability of the score when fitting it) than its base estimators. .. GENERATED FROM PYTHON SOURCE LINES 206-212 Hyperparameter search space --------------------------- We define the hyperparameter search space for decision trees. This tells to DeepHyper the hyperparameter values it can use for the optimization. To define these hyperparameters we look at the `DecisionTreeClassifier API Reference `_. .. GENERATED FROM PYTHON SOURCE LINES 212-231 .. code-block:: Python from deephyper.hpo import HpProblem def create_hpo_problem(): problem = HpProblem() problem.add_hyperparameter(["gini", "entropy", "log_loss"], "criterion") problem.add_hyperparameter(["best", "random"], "splitter") problem.add_hyperparameter((10, 1000, "log-uniform"), "max_depth", default_value=1000) problem.add_hyperparameter((2, 20), "min_samples_split", default_value=2) problem.add_hyperparameter((1, 20), "min_samples_leaf", default_value=1) problem.add_hyperparameter((0.0, 0.5), "min_weight_fraction_leaf", default_value=0.0) return problem problem = create_hpo_problem() problem .. rst-class:: sphx-glr-script-out .. code-block:: none Configuration space object: Hyperparameters: criterion, Type: Categorical, Choices: {gini, entropy, log_loss}, Default: gini max_depth, Type: UniformInteger, Range: [10, 1000], Default: 1000, on log-scale min_samples_leaf, Type: UniformInteger, Range: [1, 20], Default: 1 min_samples_split, Type: UniformInteger, Range: [2, 20], Default: 2 min_weight_fraction_leaf, Type: UniformFloat, Range: [0.0, 0.5], Default: 0.0 splitter, Type: Categorical, Choices: {best, random}, Default: best .. GENERATED FROM PYTHON SOURCE LINES 232-236 Evaluation of the baseline -------------------------- We previously defined ``default_value=...`` for each hyperparameter. These values corresponds to the default hyperparameters used in Scikit-Learn. We now test them to have a base performance. .. GENERATED FROM PYTHON SOURCE LINES 236-257 .. code-block:: Python from deephyper.evaluator import RunningJob def evaluate_decision_tree(problem): model_checkpoint_dir = "models_sklearn_test" pathlib.Path(model_checkpoint_dir).mkdir(parents=True, exist_ok=True) default_parameters = problem.default_configuration print(f"{default_parameters=}") output = run( RunningJob(id="test", parameters=default_parameters), model_checkpoint_dir=model_checkpoint_dir, show_plots=True, ) return output baseline_output = evaluate_decision_tree(problem) baseline_output .. image-sg:: /examples/examples_uq/images/sphx_glr_plot_hpo_tree_ensemble_uq_classification_sklearn_002.png :alt: plot hpo tree ensemble uq classification sklearn :srcset: /examples/examples_uq/images/sphx_glr_plot_hpo_tree_ensemble_uq_classification_sklearn_002.png :class: sphx-glr-single-img .. rst-class:: sphx-glr-script-out .. code-block:: none default_parameters={'criterion': 'gini', 'max_depth': 1000, 'min_samples_leaf': 1, 'min_samples_split': 2, 'min_weight_fraction_leaf': 0.0, 'splitter': 'best'} 0.0: val_cce=6.772 {'objective': -6.771837909470495, 'metadata': {'test_cce': 6.33494514111756, 'test_acc': 0.8242424242424242}} .. GENERATED FROM PYTHON SOURCE LINES 258-259 The accuracy is great, but the uncertainty is not well calibrated. .. GENERATED FROM PYTHON SOURCE LINES 261-280 Hyperparameter Optimization --------------------------- In DeepHyper, instead of just performing sequential Bayesian optimization we provide asynchronous parallelisation for Bayesian optimization (and other methods). This allows to execute multiple evaluation function in parallel to collect observations of objectives faster. In this example, we will focus on using centralized Bayesian optimization (CBO). In this setting, we have one main process that runs the Bayesian optimization algorithm and we have multiple worker processes that run evaluation functions. The class we use for this is :class:`deephyper.hpo.CBO`. Let us start by explaining import configuration parameters of :class:`deephyper.hpo.CBO`: - ``initial_points``: is a list of initial hyperparameter configurations to test, we add the baseline hyperparameters as we want to be at least better than this configuration. - ``surrogate_model_*``: are parameters related to the surrogate model we use, here ``"ET"`` is an alias for the Extremely Randomized Trees regression model. - ``multi_point_strategy``: is the strategy we use for parallel suggestion of hyperparameters, here we use the ``qUCBd`` that will sample for each new parallel configuration a different :math:`\kappa^j_i` value from an exponential with mean :math:`\kappa_i` where :math:`j` is the index in the current generated parallel batch and :math:`i` is the iteration of the Bayesian optimization loop. ``UCB`` corresponds to the Upper Confidence Bound acquisition function. Finally the ``"d"`` postfix in ``qUCBd`` means that we will only consider the epistemic component of the uncertainty returned by the surrogate model. - ``acq_optimizer_*``: are parameters related to optimization of the previously defined acquisition function. - ``kappa`` and ``scheduler``: are the parameters that define the schedule of :math:`\kappa^j_i` previously mentionned. - ``objective_scaler``: is a parameter that can be used to rescale the observed objectives (e.g., identity, min-max, log). .. GENERATED FROM PYTHON SOURCE LINES 280-306 .. code-block:: Python search_kwargs = { "initial_points": [problem.default_configuration], "n_initial_points": 2 * len(problem) + 1, # Number of initial random points "surrogate_model": "ET", # Use Extra Trees as surrogate model "surrogate_model_kwargs": { "n_estimators": 50, # Relatively small number of trees in the surrogate to make it "fast" "min_samples_split": 8, # Larger number to avoid small leaf nodes (smoothing the objective response) }, "multi_point_strategy": "qUCBd", # Multi-point strategy for asynchronous batch generations (explained later) "acq_optimizer": "sampling", # Use random sampling for the acquisition function optimizer "acq_optimizer_kwargs": { "filter_duplicated": False, # Deactivate filtration of duplicated new points }, "acq_func": "UCBd", "acq_func_kwargs": { "kappa": 10.0, # Initial value of exploration-exploitation parameter for the acquisition function "scheduler": { # Scheduler for the exploration-exploitation parameter "kappa" "type": "periodic-exp-decay", # Periodic exponential decay "period": 50, # Period over which the decay is applied. It is useful to escape local solutions. "kappa_final": 0.001, # Value of kappa at the end of each "period" }, }, "objective_scaler": "identity", "random_state": 42, # Random seed } .. GENERATED FROM PYTHON SOURCE LINES 307-308 Then we can run the optimization. .. GENERATED FROM PYTHON SOURCE LINES 308-342 .. code-block:: Python from deephyper.hpo import CBO from deephyper.evaluator import Evaluator from deephyper.evaluator.callback import TqdmCallback def run_hpo(problem): pathlib.Path(model_checkpoint_dir).mkdir(parents=True, exist_ok=True) evaluator = Evaluator.create( run, method="ray", method_kwargs={ "num_cpus_per_task": 1, "run_function_kwargs": { "model_checkpoint_dir": model_checkpoint_dir, "verbose": False, }, "callbacks": [TqdmCallback()] }, ) search = CBO( problem, log_dir=hpo_dir, **search_kwargs, ) results = search.search(evaluator, max_evals=1_000) return results results = run_hpo(problem) .. rst-class:: sphx-glr-script-out .. code-block:: none 2025-08-18 15:31:34,448 INFO worker.py:1852 -- Started a local Ray instance. 0%| | 0/1000 [00:00

	p:criterion	p:max_depth	p:min_samples_leaf	p:min_samples_split	p:min_weight_fraction_leaf	p:splitter	objective	job_id	job_status	m:timestamp_submit	m:test_cce	m:test_acc	m:timestamp_gather
0	entropy	473	1	15	0.399777	random	-0.652100	1	DONE	0.857375	0.668201	0.615152	2.527716
1	log_loss	584	6	8	0.219015	random	-0.524039	2	DONE	0.858437	0.546033	0.757576	2.557154
2	log_loss	40	11	8	0.263310	best	-0.482095	3	DONE	0.859354	0.517822	0.763636	2.562505
3	entropy	122	9	20	0.439568	random	-0.693145	8	DONE	2.552939	0.693145	0.503030	2.577085
4	gini	799	13	13	0.357857	best	-0.525434	4	DONE	0.860122	0.540779	0.763636	2.581751
...	...	...	...	...	...	...	...	...	...	...	...	...	...
1000	gini	171	14	2	0.078259	random	-0.448314	998	DONE	32.196486	0.509838	0.772727	32.344678
1001	gini	56	14	3	0.067282	random	-0.492451	1001	DONE	32.331540	0.517744	0.748485	32.424286
1002	gini	323	12	7	0.076291	random	-0.499077	1002	DONE	32.332477	0.544741	0.727273	32.426498
1003	gini	323	12	7	0.076291	random	-0.719315	1003	DONE	32.333362	0.498796	0.772727	32.428591
1004	gini	323	12	7	0.076291	random	-0.505116	1004	DONE	32.334272	0.531597	0.760606	32.430668

1005 rows × 13 columns

.. GENERATED FROM PYTHON SOURCE LINES 354-358 Evolution of the objective ~~~~~~~~~~~~~~~~~~~~~~~~~~ We use :func:`deephyper.analysis.hpo.plot_search_trajectory_single_objective_hpo` to look at the evolution of the objective during the search. .. GENERATED FROM PYTHON SOURCE LINES 358-368 .. dropdown:: Code (Plot search trajectory) .. code-block:: Python from deephyper.analysis.hpo import plot_search_trajectory_single_objective_hpo _, ax = plt.subplots(figsize=(WIDTH_PLOTS, HEIGHT_PLOTS), tight_layout=True) _ = plot_search_trajectory_single_objective_hpo(results, mode="min", ax=ax) ax.axhline(-baseline_output["objective"], linestyle="--", color="red", label="baseline") ax.set_yscale("log") .. image-sg:: /examples/examples_uq/images/sphx_glr_plot_hpo_tree_ensemble_uq_classification_sklearn_003.png :alt: plot hpo tree ensemble uq classification sklearn :srcset: /examples/examples_uq/images/sphx_glr_plot_hpo_tree_ensemble_uq_classification_sklearn_003.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 369-370 The dashed red horizontal line corresponds to the baseline performance. .. GENERATED FROM PYTHON SOURCE LINES 372-376 Worker utilization ~~~~~~~~~~~~~~~~~~ We use :func:`deephyper.analysis.hpo.plot_worker_utilization` to look at the number of active workers over the search. .. GENERATED FROM PYTHON SOURCE LINES 376-383 .. dropdown:: Code (Plot worker utilization) .. code-block:: Python from deephyper.analysis.hpo import plot_worker_utilization _, ax = plt.subplots(figsize=(WIDTH_PLOTS, HEIGHT_PLOTS), tight_layout=True) _ = plot_worker_utilization(results, ax=ax) .. image-sg:: /examples/examples_uq/images/sphx_glr_plot_hpo_tree_ensemble_uq_classification_sklearn_004.png :alt: plot hpo tree ensemble uq classification sklearn :srcset: /examples/examples_uq/images/sphx_glr_plot_hpo_tree_ensemble_uq_classification_sklearn_004.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 384-388 Best decision tree ~~~~~~~~~~~~~~~~~~~ Then, we look indivudualy at the performance of the top 5 models by using :func:`deephyper.analysis.hpo.parameters_from_row`: .. GENERATED FROM PYTHON SOURCE LINES 388-399 .. code-block:: Python from deephyper.analysis.hpo import parameters_from_row topk_rows = results.nlargest(5, "objective").reset_index(drop=True) for i, row in topk_rows.iterrows(): parameters = parameters_from_row(row) obj = row["objective"] print(f"Top-{i+1} -> {obj=:.3f}: {parameters}") print() .. rst-class:: sphx-glr-script-out .. code-block:: none Top-1 -> obj=-0.342: {'criterion': 'gini', 'max_depth': 391, 'min_samples_leaf': 12, 'min_samples_split': 9, 'min_weight_fraction_leaf': 0.0799892256322516, 'splitter': 'random'} Top-2 -> obj=-0.344: {'criterion': 'log_loss', 'max_depth': 44, 'min_samples_leaf': 18, 'min_samples_split': 9, 'min_weight_fraction_leaf': 0.03020412119569555, 'splitter': 'random'} Top-3 -> obj=-0.345: {'criterion': 'gini', 'max_depth': 174, 'min_samples_leaf': 16, 'min_samples_split': 6, 'min_weight_fraction_leaf': 0.06825880170388439, 'splitter': 'random'} Top-4 -> obj=-0.348: {'criterion': 'log_loss', 'max_depth': 24, 'min_samples_leaf': 18, 'min_samples_split': 10, 'min_weight_fraction_leaf': 0.034806060105093746, 'splitter': 'random'} Top-5 -> obj=-0.349: {'criterion': 'log_loss', 'max_depth': 347, 'min_samples_leaf': 16, 'min_samples_split': 6, 'min_weight_fraction_leaf': 0.07115076987342352, 'splitter': 'random'} .. GENERATED FROM PYTHON SOURCE LINES 400-403 If we just plot the decision boundary and calibration plots of the best model we can observe a significant improvement over the baseline with log-loss values around 0.338 when it was previously around 6. .. GENERATED FROM PYTHON SOURCE LINES 403-410 .. code-block:: Python best_job = topk_rows.iloc[0] hpo_dir = "hpo_sklearn_classification" model_checkpoint_dir = os.path.join(hpo_dir, "models") with open(os.path.join(model_checkpoint_dir, f"model_0.{best_job.job_id}.pkl"), "rb") as f: best_model = pickle.load(f) .. GENERATED FROM PYTHON SOURCE LINES 411-417 .. dropdown:: Code (Plot decision boundary and calibration) .. code-block:: Python fig, axes = plt.subplots(nrows=2, ncols=1, figsize=(WIDTH_PLOTS, HEIGHT_PLOTS*2), tight_layout=True) plot_decision_boundary_decision_tree(tx, ty, best_model, steps=1000, color_map="viridis", ax=axes[0]) disp = CalibrationDisplay.from_predictions(ty, best_model.predict_proba(tx)[:, 1], ax=axes[1]) .. image-sg:: /examples/examples_uq/images/sphx_glr_plot_hpo_tree_ensemble_uq_classification_sklearn_005.png :alt: plot hpo tree ensemble uq classification sklearn :srcset: /examples/examples_uq/images/sphx_glr_plot_hpo_tree_ensemble_uq_classification_sklearn_005.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 418-423 Ensemble of decision trees -------------------------- We now move to ensembling checkpointed models and we start by importing utilities from :mod:`deephyper.ensemble` and :mod:`deephyper.predictor`. .. GENERATED FROM PYTHON SOURCE LINES 423-429 .. code-block:: Python from deephyper.ensemble import EnsemblePredictor from deephyper.ensemble.aggregator import MixedCategoricalAggregator from deephyper.ensemble.loss import CategoricalCrossEntropy from deephyper.ensemble.selector import GreedySelector, TopKSelector from deephyper.predictor.sklearn import SklearnPredictorFileLoader .. GENERATED FROM PYTHON SOURCE LINES 430-506 .. dropdown:: Code (Plot decision boundary and uncertainty) .. code-block:: Python def plot_decision_boundary_and_uncertainty( dataset, labels, model, steps=1000, color_map="viridis", s=5 ): fig, axs = plt.subplots( 3, sharex="all", sharey="all", figsize=(WIDTH_PLOTS, HEIGHT_PLOTS * 2), tight_layout=True, ) # Define region of interest by data limits xmin, xmax = dataset[:, 0].min() - 1, dataset[:, 0].max() + 1 ymin, ymax = dataset[:, 1].min() - 1, dataset[:, 1].max() + 1 x_span = np.linspace(xmin, xmax, steps) y_span = np.linspace(ymin, ymax, steps) xx, yy = np.meshgrid(x_span, y_span) # Make predictions across region of interest y_pred = model.predict(np.c_[xx.ravel(), yy.ravel()].astype(np.float32)) y_pred_proba = y_pred["loc"] y_pred_aleatoric = y_pred["uncertainty_aleatoric"] y_pred_epistemic = y_pred["uncertainty_epistemic"] # Plot decision boundary in region of interest # 1. MODE color_map = plt.get_cmap("viridis") z = y_pred_proba[:, 1].reshape(xx.shape) cont = axs[0].contourf(xx, yy, z, cmap=color_map, vmin=0, vmax=1, alpha=0.5) # Get predicted labels on training data and plot axs[0].scatter( dataset[:, 0], dataset[:, 1], c=labels, cmap=color_map, s=s, lw=0, ) plt.colorbar(cont, ax=axs[0], label="Probability of class 1") # 2. ALEATORIC color_map = plt.get_cmap("plasma") z = y_pred_aleatoric.reshape(xx.shape) cont = axs[1].contourf(xx, yy, z, cmap=color_map, vmin=0, vmax=0.69, alpha=0.5) # Get predicted labels on training data and plot axs[1].scatter( dataset[:, 0], dataset[:, 1], c=labels, cmap=color_map, s=s, lw=0, ) plt.colorbar(cont, ax=axs[1], label="Aleatoric uncertainty") # 3. EPISTEMIC z = y_pred_epistemic.reshape(xx.shape) cont = axs[2].contourf(xx, yy, z, cmap=color_map, vmin=0, vmax=0.69, alpha=0.5) # Get predicted labels on training data and plot axs[2].scatter( dataset[:, 0], dataset[:, 1], c=labels, cmap=color_map, s=s, lw=0, ) plt.colorbar(cont, ax=axs[2], label="Epistemic uncertainty") .. GENERATED FROM PYTHON SOURCE LINES 507-509 We define a function that will create an ensemble with TopK or Greedy selection strategies. This function also has a parameter ``k`` that sets the number of unique member in the ensemble. .. GENERATED FROM PYTHON SOURCE LINES 509-569 .. code-block:: Python def create_ensemble_from_checkpoints(ensemble_selector: str = "topk", k=50): # 0. Load data _, (vx, vy), _ = load_data() # !1.3 SKLEARN EXAMPLE predictor_files = SklearnPredictorFileLoader.find_predictor_files( model_checkpoint_dir ) predictor_loaders = [SklearnPredictorFileLoader(f) for f in predictor_files] predictors = [p.load() for p in predictor_loaders] # 2. Build an ensemble ensemble = EnsemblePredictor( predictors=predictors, aggregator=MixedCategoricalAggregator( uncertainty_method="entropy", decomposed_uncertainty=True, ), # You can specify parallel backends for the evaluation of the ensemble evaluator={ "method": "ray", "method_kwargs": {"num_cpus_per_task": 1}, }, ) y_predictors = ensemble.predictions_from_predictors( vx, predictors=ensemble.predictors ) # Use TopK or Greedy/Caruana if ensemble_selector == "topk": selector = TopKSelector( loss_func=CategoricalCrossEntropy(), k=k, ) elif ensemble_selector == "greedy": selector = GreedySelector( loss_func=CategoricalCrossEntropy(), aggregator=MixedCategoricalAggregator(), k=k, k_init=5, max_it=100, early_stopping=False, bagging=True, eps_tol=1e-5, ) else: raise ValueError(f"Unknown ensemble_selector: {ensemble_selector}") selected_predictors_indexes, selected_predictors_weights = selector.select( vy, y_predictors ) print(f"{selected_predictors_indexes=}") print(f"{selected_predictors_weights=}") ensemble.predictors = [ensemble.predictors[i] for i in selected_predictors_indexes] ensemble.weights = selected_predictors_weights return ensemble .. GENERATED FROM PYTHON SOURCE LINES 570-571 We start by testing the Topk strategy. .. GENERATED FROM PYTHON SOURCE LINES 571-577 .. code-block:: Python ensemble = create_ensemble_from_checkpoints("topk") ty_pred = ensemble.predict(tx)["loc"] cce = log_loss(ty, ty_pred) acc = accuracy_score(ty, np.argmax(ty_pred, axis=1)) print(f"Test scores: {cce=:.3f}, {acc=:.3f}") .. rst-class:: sphx-glr-script-out .. code-block:: none selected_predictors_indexes=[791, 346, 960, 725, 698, 303, 128, 192, 547, 757, 555, 872, 452, 374, 910, 310, 278, 527, 682, 147, 595, 268, 284, 600, 336, 743, 362, 9, 732, 70, 752, 773, 49, 733, 164, 389, 308, 453, 451, 941, 155, 391, 484, 962, 82, 561, 997, 655, 316, 948] selected_predictors_weights=[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] Test scores: cce=0.386, acc=0.879 .. GENERATED FROM PYTHON SOURCE LINES 578-583 .. dropdown:: Code (Plot decision boundary and uncertainty for ensemble) .. code-block:: Python plot_decision_boundary_and_uncertainty(tx, ty, ensemble, steps=1000, color_map="viridis") .. image-sg:: /examples/examples_uq/images/sphx_glr_plot_hpo_tree_ensemble_uq_classification_sklearn_006.png :alt: plot hpo tree ensemble uq classification sklearn :srcset: /examples/examples_uq/images/sphx_glr_plot_hpo_tree_ensemble_uq_classification_sklearn_006.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 585-590 .. dropdown:: Code (Plot calibration for ensemble) .. code-block:: Python fig, ax = plt.subplots(figsize=(WIDTH_PLOTS, HEIGHT_PLOTS), tight_layout=True) disp = CalibrationDisplay.from_predictions(ty, ty_pred[:, 1], ax=ax) .. image-sg:: /examples/examples_uq/images/sphx_glr_plot_hpo_tree_ensemble_uq_classification_sklearn_007.png :alt: plot hpo tree ensemble uq classification sklearn :srcset: /examples/examples_uq/images/sphx_glr_plot_hpo_tree_ensemble_uq_classification_sklearn_007.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 591-592 We do the same for the Greedy strategy. .. GENERATED FROM PYTHON SOURCE LINES 592-598 .. code-block:: Python ensemble = create_ensemble_from_checkpoints("greedy") ty_pred = ensemble.predict(tx)["loc"] cce = log_loss(ty, ty_pred) acc = accuracy_score(ty, np.argmax(ty_pred, axis=1)) print(f"Test scores: {cce=:.3f}, {acc=:.3f}") .. rst-class:: sphx-glr-script-out .. code-block:: none selected_predictors_indexes=[80, 284, 327, 346, 453, 555, 698, 716, 725, 743, 759, 791, 910, 960, 968] selected_predictors_weights=[0.047619047619047616, 0.06666666666666667, 0.13333333333333333, 0.1523809523809524, 0.009523809523809525, 0.14285714285714285, 0.009523809523809525, 0.01904761904761905, 0.009523809523809525, 0.0380952380952381, 0.02857142857142857, 0.17142857142857143, 0.009523809523809525, 0.09523809523809523, 0.06666666666666667] Test scores: cce=0.368, acc=0.879 .. GENERATED FROM PYTHON SOURCE LINES 599-603 .. dropdown:: Code (Plot decision boundary and uncertainty for ensemble) .. code-block:: Python plot_decision_boundary_and_uncertainty(tx, ty, ensemble, steps=1000, color_map="viridis") .. image-sg:: /examples/examples_uq/images/sphx_glr_plot_hpo_tree_ensemble_uq_classification_sklearn_008.png :alt: plot hpo tree ensemble uq classification sklearn :srcset: /examples/examples_uq/images/sphx_glr_plot_hpo_tree_ensemble_uq_classification_sklearn_008.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 604-609 .. dropdown:: Code (Plot calibration for ensemble) .. code-block:: Python fig, ax = plt.subplots(figsize=(WIDTH_PLOTS, HEIGHT_PLOTS), tight_layout=True) disp = CalibrationDisplay.from_predictions(ty, ty_pred[:, 1], ax=ax) .. image-sg:: /examples/examples_uq/images/sphx_glr_plot_hpo_tree_ensemble_uq_classification_sklearn_009.png :alt: plot hpo tree ensemble uq classification sklearn :srcset: /examples/examples_uq/images/sphx_glr_plot_hpo_tree_ensemble_uq_classification_sklearn_009.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 610-617 In conclusion, the improvement over the default hyperparameters is significant. For CCE, we improved from about 6 to 0.4. For Accuracy, we improved from 0.82 to 0.87. Not only that we have disentangled uncertainty estimates. The epistemic uncertainty is informative of locations where we are missing data and the aleatoric uncertainty is informative of the noise level in the labels. .. rst-class:: sphx-glr-timing **Total running time of the script:** (0 minutes 42.666 seconds) .. _sphx_glr_download_examples_examples_uq_plot_hpo_tree_ensemble_uq_classification_sklearn.py: .. only:: html .. container:: sphx-glr-footer sphx-glr-footer-example .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: plot_hpo_tree_ensemble_uq_classification_sklearn.ipynb ` .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: plot_hpo_tree_ensemble_uq_classification_sklearn.py ` .. container:: sphx-glr-download sphx-glr-download-zip :download:`Download zipped: plot_hpo_tree_ensemble_uq_classification_sklearn.zip ` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_