目录
目录

注意

点击 这里 下载完整示例代码

多目标NAS与Ax

创建日期：2022年8月19日 | 最后更新：2024年7月31日 | 最后验证：2024年11月5日

作者: David Eriksson, Max Balandat, 和Meta的Adaptive Experimentation团队。

在这个教程中，我们将展示如何使用 Ax 在流行的MNIST数据集上运行多目标神经网络架构搜索（NAS）以训练一个简单的神经网络模型。虽然底层的方法通常用于更复杂的模型和更大的数据集，但我们选择了一个可以在笔记本电脑上不到20分钟内端到端运行的教程。

在许多NAS应用程序中，多个感兴趣的目标之间存在自然的权衡。例如，在设备上部署模型时，我们可能希望最大化模型性能（例如准确性），同时最小化像功耗、推理延迟或模型大小等竞争指标，以满足部署约束。通常，通过接受稍微降低的模型性能，我们可以大幅减少计算需求或预测延迟。探索此类权衡的有效原则方法是可扩展和可持续AI的关键推动因素，并且在Meta有许多成功的应用 - 例如，请参阅我们的案例研究关于自然语言理解模型。

在我们的示例中，我们将调整两个隐藏层的宽度、学习率、dropout 概率、批量大小以及训练轮数。目标是在模型性能（验证集上的准确率）和模型规模（模型参数的数量）之间进行权衡。

本教程使用了以下 PyTorch 库：

  • PyTorch Lightning （指定模型和训练循环）

  • TorchX （用于远程/异步运行训练任务）

  • BoTorch (用于驱动Ax算法的贝叶斯优化库)

定义TorchX应用

我们的目标是优化在 mnist_train_nas.py 中定义的PyTorch Lightning训练任务。 为了使用TorchX实现这一点，我们编写一个辅助函数，该函数接收训练任务的架构和超参数值，并创建具有适当设置的 TorchX AppDef。

from pathlib import Path

import torchx

from torchx import specs
from torchx.components import utils


def trainer(
    log_path: str,
    hidden_size_1: int,
    hidden_size_2: int,
    learning_rate: float,
    epochs: int,
    dropout: float,
    batch_size: int,
    trial_idx: int = -1,
) -> specs.AppDef:

    # define the log path so we can pass it to the TorchX ``AppDef``
    if trial_idx >= 0:
        log_path = Path(log_path).joinpath(str(trial_idx)).absolute().as_posix()

    return utils.python(
        # command line arguments to the training script
        "--log_path",
        log_path,
        "--hidden_size_1",
        str(hidden_size_1),
        "--hidden_size_2",
        str(hidden_size_2),
        "--learning_rate",
        str(learning_rate),
        "--epochs",
        str(epochs),
        "--dropout",
        str(dropout),
        "--batch_size",
        str(batch_size),
        # other config options
        name="trainer",
        script="mnist_train_nas.py",
        image=torchx.version.TORCHX_IMAGE,
    )

设置Runner

Ax’s Runner 抽象允许编写对接各种后端的接口。 Ax 已经内置了用于 TorchX 的 Runner，因此我们只需要进行配置。为了本教程的目的，我们将以完全异步的方式在本地运行任务。

为了在集群上运行它们，您可以指定不同的TorchX调度器并相应地调整配置。 例如，如果您有一个Kubernetes集群，只需将调度器从 local_cwd 更改为 kubernetes。

import tempfile
from ax.runners.torchx import TorchXRunner

# Make a temporary dir to log our results into
log_dir = tempfile.mkdtemp()

ax_runner = TorchXRunner(
    tracker_base="/tmp/",
    component=trainer,
    # NOTE: To launch this job on a cluster instead of locally you can
    # specify a different scheduler and adjust arguments appropriately.
    scheduler="local_cwd",
    component_const_params={"log_path": log_dir},
    cfg={},
)

设置 SearchSpace

首先，我们定义我们的搜索空间。Ax 支持整数和浮点类型的范围参数，以及可以具有非数值类型（如字符串）的选择参数。我们将隐藏层大小、学习率、Dropout 和训练轮数作为范围参数进行调优，并将批量大小作为有序选择参数进行调优，以确保其为 2 的幂次方。

from ax.core import (
    ChoiceParameter,
    ParameterType,
    RangeParameter,
    SearchSpace,
)

parameters = [
    # NOTE: In a real-world setting, hidden_size_1 and hidden_size_2
    # should probably be powers of 2, but in our simple example this
    # would mean that ``num_params`` can't take on that many values, which
    # in turn makes the Pareto frontier look pretty weird.
    RangeParameter(
        name="hidden_size_1",
        lower=16,
        upper=128,
        parameter_type=ParameterType.INT,
        log_scale=True,
    ),
    RangeParameter(
        name="hidden_size_2",
        lower=16,
        upper=128,
        parameter_type=ParameterType.INT,
        log_scale=True,
    ),
    RangeParameter(
        name="learning_rate",
        lower=1e-4,
        upper=1e-2,
        parameter_type=ParameterType.FLOAT,
        log_scale=True,
    ),
    RangeParameter(
        name="epochs",
        lower=1,
        upper=4,
        parameter_type=ParameterType.INT,
    ),
    RangeParameter(
        name="dropout",
        lower=0.0,
        upper=0.5,
        parameter_type=ParameterType.FLOAT,
    ),
    ChoiceParameter(  # NOTE: ``ChoiceParameters`` don't require log-scale
        name="batch_size",
        values=[32, 64, 128, 256],
        parameter_type=ParameterType.INT,
        is_ordered=True,
        sort_values=True,
    ),
]

search_space = SearchSpace(
    parameters=parameters,
    # NOTE: In practice, it may make sense to add a constraint
    # hidden_size_2 <= hidden_size_1
    parameter_constraints=[],
)

设置指标

Ax 有 Metric 的概念，它定义了结果的属性以及如何获取这些结果的观测值。 这允许例如编码如何从某些分布式执行后端获取数据，并在传递给 Ax 作为输入之前进行后处理。

在本教程中，我们将使用 多目标优化 以最大化验证准确率并最小化模型参数数量为目标。后者代表了模型延迟的一个简单代理， 对于小型机器学习模型来说，准确估计延迟是很困难的（在实际应用中，我们会在设备上运行模型时对延迟进行基准测试）。

在我们的示例中，TorchX 会在本地以完全异步的方式运行训练任务，并根据试验索引将结果写入 log_dir（参见上方的 trainer() 函数）。我们将定义一个了解该日志目录的度量类。通过继承 TensorboardCurveMetric 我们可以免费获得读取和解析 TensorBoard 日志的逻辑。

from ax.metrics.tensorboard import TensorboardMetric
from tensorboard.backend.event_processing import plugin_event_multiplexer as event_multiplexer

class MyTensorboardMetric(TensorboardMetric):

    # NOTE: We need to tell the new TensorBoard metric how to get the id /
    # file handle for the TensorBoard logs from a trial. In this case
    # our convention is to just save a separate file per trial in
    # the prespecified log dir.
    def _get_event_multiplexer_for_trial(self, trial):
        mul = event_multiplexer.EventMultiplexer(max_reload_threads=20)
        mul.AddRunsFromDirectory(Path(log_dir).joinpath(str(trial.index)).as_posix(), None)
        mul.Reload()

        return mul

    # This indicates whether the metric is queryable while the trial is
    # still running. We don't use this in the current tutorial, but Ax
    # utilizes this to implement trial-level early-stopping functionality.
    @classmethod
    def is_available_while_running(cls):
        return False

现在我们可以实例化准确率和模型参数数量的指标。这里 curve_name 是 TensorBoard 日志中指标的名称，而 name 是 Ax 内部使用的指标名称。我们还指定 lower_is_better 来表示这两个指标的有利方向。

val_acc = MyTensorboardMetric(
    name="val_acc",
    tag="val_acc",
    lower_is_better=False,
)
model_num_params = MyTensorboardMetric(
    name="num_params",
    tag="num_params",
    lower_is_better=True,
)

设置 OptimizationConfig

告诉Ax它应该优化什么的方法是通过一个 OptimizationConfig。 在这里我们使用一个 MultiObjectiveOptimizationConfig，因为我们 将进行多目标优化。

此外，Ax 还支持通过指定目标阈值来对不同的指标施加约束，这可以限定我们希望探索的结果空间中的感兴趣区域。对于这个示例，我们将约束验证准确率至少为 0.94（94%），并且模型参数数量最多为 80,000。

from ax.core import MultiObjective, Objective, ObjectiveThreshold
from ax.core.optimization_config import MultiObjectiveOptimizationConfig


opt_config = MultiObjectiveOptimizationConfig(
    objective=MultiObjective(
        objectives=[
            Objective(metric=val_acc, minimize=False),
            Objective(metric=model_num_params, minimize=True),
        ],
    ),
    objective_thresholds=[
        ObjectiveThreshold(metric=val_acc, bound=0.94, relative=False),
        ObjectiveThreshold(metric=model_num_params, bound=80_000, relative=False),
    ],
)

创建Ax实验

在Ax中，实验对象是存储有关问题设置的所有信息的对象。

from ax.core import Experiment

experiment = Experiment(
    name="torchx_mnist",
    search_space=search_space,
    optimization_config=opt_config,
    runner=ax_runner,
)

选择生成策略

一个 GenerationStrategy 是优化方式的抽象表示。虽然这可以自定义（如果您想这样做，请参见 此教程）， 但在大多数情况下，Ax 可以根据搜索空间、优化配置以及我们想要运行的试验总数 自动确定适当策略。

通常，Ax 会在开始基于模型的贝叶斯优化策略之前，评估一定数量的随机配置。

total_trials = 48  # total evaluation budget

from ax.modelbridge.dispatch_utils import choose_generation_strategy

gs = choose_generation_strategy(
    search_space=experiment.search_space,
    optimization_config=experiment.optimization_config,
    num_trials=total_trials,
  )

[INFO 01-02 21:56:00] ax.modelbridge.dispatch_utils: Using Models.BOTORCH_MODULAR since there is at least one ordered parameter and there are no unordered categorical parameters.
[INFO 01-02 21:56:00] ax.modelbridge.dispatch_utils: Calculating the number of remaining initialization trials based on num_initialization_trials=None max_initialization_trials=None num_tunable_parameters=6 num_trials=48 use_batch_trials=False
[INFO 01-02 21:56:00] ax.modelbridge.dispatch_utils: calculated num_initialization_trials=9
[INFO 01-02 21:56:00] ax.modelbridge.dispatch_utils: num_completed_initialization_trials=0 num_remaining_initialization_trials=9
[INFO 01-02 21:56:00] ax.modelbridge.dispatch_utils: `verbose`, `disable_progbar`, and `jit_compile` are not yet supported when using `choose_generation_strategy` with ModularBoTorchModel, dropping these arguments.
[INFO 01-02 21:56:00] ax.modelbridge.dispatch_utils: Using Bayesian Optimization generation strategy: GenerationStrategy(name='Sobol+BoTorch', steps=[Sobol for 9 trials, BoTorch for subsequent trials]). Iterations after 9 will take longer to generate due to model-fitting.

调度器配置

The Scheduler acts as the loop control for the optimization. It communicates with the backend to launch trials, check their status, and retrieve results. In the case of this tutorial, it is simply reading and parsing the locally saved logs. In a remote execution setting, it would call APIs. The following illustration from the Ax Scheduler tutorial summarizes how the Scheduler interacts with external systems used to run trial evaluations:

../_static/img/ax_scheduler_illustration.png

The Scheduler requires the Experiment and the GenerationStrategy. A set of options can be passed in via SchedulerOptions. Here, we configure the number of total evaluations as well as max_pending_trials, the maximum number of trials that should run concurrently. In our local setting, this is the number of training jobs running as individual processes, while in a remote execution setting, this would be the number of machines you want to use in parallel.

from ax.service.scheduler import Scheduler, SchedulerOptions

scheduler = Scheduler(
    experiment=experiment,
    generation_strategy=gs,
    options=SchedulerOptions(
        total_trials=total_trials, max_pending_trials=4
    ),
)

[WARNING 01-02 21:56:00] ax.service.utils.with_db_settings_base: Ax currently requires a sqlalchemy version below 2.0. This will be addressed in a future release. Disabling SQL storage in Ax for now, if you would like to use SQL storage please install Ax with mysql extras via `pip install ax-platform[mysql]`.
[INFO 01-02 21:56:00] Scheduler: `Scheduler` requires experiment to have immutable search space and optimization config. Setting property immutable_search_space_and_opt_config to `True` on experiment.

运行优化

现在所有配置都已完成，我们可以让 Ax 以完全自动的方式运行优化过程。调度器会定期检查日志，以获取当前所有正在运行试验的状态，如果某个试验完成，调度器将更新该试验在实验中的状态，并获取贝叶斯优化算法所需的观测数据。

scheduler.run_all_trials()

[INFO 01-02 21:56:00] Scheduler: Fetching data for newly completed trials: [].
/usr/local/lib/python3.10/dist-packages/ax/modelbridge/cross_validation.py:463: UserWarning:

Encountered exception in computing model fit quality: RandomModelBridge does not support prediction.

[INFO 01-02 21:56:00] Scheduler: Running trials [0]...
/usr/local/lib/python3.10/dist-packages/ax/modelbridge/cross_validation.py:463: UserWarning:

Encountered exception in computing model fit quality: RandomModelBridge does not support prediction.

[INFO 01-02 21:56:01] Scheduler: Running trials [1]...
/usr/local/lib/python3.10/dist-packages/ax/modelbridge/cross_validation.py:463: UserWarning:

Encountered exception in computing model fit quality: RandomModelBridge does not support prediction.

[INFO 01-02 21:56:02] Scheduler: Running trials [2]...
/usr/local/lib/python3.10/dist-packages/ax/modelbridge/cross_validation.py:463: UserWarning:

Encountered exception in computing model fit quality: RandomModelBridge does not support prediction.

[INFO 01-02 21:56:03] Scheduler: Running trials [3]...
[INFO 01-02 21:56:04] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:56:04] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 4).
[INFO 01-02 21:56:05] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:56:05] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 4).
[INFO 01-02 21:56:07] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:56:07] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 4).
[INFO 01-02 21:56:09] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:56:09] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 4).
[INFO 01-02 21:56:12] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:56:12] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 4).
[INFO 01-02 21:56:17] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:56:17] Scheduler: Waiting for completed trials (for 7 sec, currently running trials: 4).
[INFO 01-02 21:56:25] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:56:25] Scheduler: Waiting for completed trials (for 11 sec, currently running trials: 4).
[INFO 01-02 21:56:36] Scheduler: Fetching data for newly completed trials: [1, 3].
[INFO 01-02 21:56:36] Scheduler: Retrieved COMPLETED trials: [1, 3].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/modelbridge/cross_validation.py:463: UserWarning:

Encountered exception in computing model fit quality: RandomModelBridge does not support prediction.

[INFO 01-02 21:56:36] Scheduler: Running trials [4]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/modelbridge/cross_validation.py:463: UserWarning:

Encountered exception in computing model fit quality: RandomModelBridge does not support prediction.

[INFO 01-02 21:56:37] Scheduler: Running trials [5]...
[INFO 01-02 21:56:38] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:56:38] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 4).
[INFO 01-02 21:56:39] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:56:39] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 4).
[INFO 01-02 21:56:41] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:56:41] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 4).
[INFO 01-02 21:56:43] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:56:43] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 4).
[INFO 01-02 21:56:47] Scheduler: Fetching data for newly completed trials: [2].
[INFO 01-02 21:56:47] Scheduler: Retrieved COMPLETED trials: [2].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/modelbridge/cross_validation.py:463: UserWarning:

Encountered exception in computing model fit quality: RandomModelBridge does not support prediction.

[INFO 01-02 21:56:47] Scheduler: Running trials [6]...
[INFO 01-02 21:56:48] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:56:48] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 4).
[INFO 01-02 21:56:49] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:56:49] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 4).
[INFO 01-02 21:56:50] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:56:50] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 4).
[INFO 01-02 21:56:52] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:56:52] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 4).
[INFO 01-02 21:56:56] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:56:56] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 4).
[INFO 01-02 21:57:01] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:57:01] Scheduler: Waiting for completed trials (for 7 sec, currently running trials: 4).
[INFO 01-02 21:57:08] Scheduler: Fetching data for newly completed trials: [4].
[INFO 01-02 21:57:08] Scheduler: Retrieved COMPLETED trials: [4].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/modelbridge/cross_validation.py:463: UserWarning:

Encountered exception in computing model fit quality: RandomModelBridge does not support prediction.

[INFO 01-02 21:57:09] Scheduler: Running trials [7]...
[INFO 01-02 21:57:10] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:57:10] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 4).
[INFO 01-02 21:57:11] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:57:11] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 4).
[INFO 01-02 21:57:12] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:57:12] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 4).
[INFO 01-02 21:57:14] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:57:14] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 4).
[INFO 01-02 21:57:18] Scheduler: Fetching data for newly completed trials: [6].
[INFO 01-02 21:57:18] Scheduler: Retrieved COMPLETED trials: [6].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/modelbridge/cross_validation.py:463: UserWarning:

Encountered exception in computing model fit quality: RandomModelBridge does not support prediction.

[INFO 01-02 21:57:18] Scheduler: Running trials [8]...
[INFO 01-02 21:57:19] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:57:19] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 4).
[INFO 01-02 21:57:20] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:57:20] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 4).
[INFO 01-02 21:57:21] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:57:21] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 4).
[INFO 01-02 21:57:24] Scheduler: Fetching data for newly completed trials: [0, 5].
[INFO 01-02 21:57:24] Scheduler: Retrieved COMPLETED trials: [0, 5].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 21:57:27] Scheduler: Running trials [9]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 21:57:33] Scheduler: Running trials [10]...
[INFO 01-02 21:57:34] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:57:34] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 4).
[INFO 01-02 21:57:35] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:57:35] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 4).
[INFO 01-02 21:57:37] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:57:37] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 4).
[INFO 01-02 21:57:39] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:57:39] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 4).
[INFO 01-02 21:57:42] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:57:42] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 4).
[INFO 01-02 21:57:47] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:57:47] Scheduler: Waiting for completed trials (for 7 sec, currently running trials: 4).
[INFO 01-02 21:57:55] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:57:55] Scheduler: Waiting for completed trials (for 11 sec, currently running trials: 4).
[INFO 01-02 21:58:06] Scheduler: Fetching data for newly completed trials: [10].
[INFO 01-02 21:58:06] Scheduler: Retrieved COMPLETED trials: [10].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 21:58:14] Scheduler: Running trials [11]...
[INFO 01-02 21:58:15] Scheduler: Fetching data for newly completed trials: 8 - 9.
[INFO 01-02 21:58:15] Scheduler: Retrieved COMPLETED trials: 8 - 9.
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 21:58:23] Scheduler: Running trials [12]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 21:58:30] Scheduler: Running trials [13]...
[INFO 01-02 21:58:31] Scheduler: Fetching data for newly completed trials: [7].
[INFO 01-02 21:58:31] Scheduler: Retrieved COMPLETED trials: [7].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 21:58:33] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:58:33] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 01-02 21:58:34] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:58:34] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 01-02 21:58:35] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:58:35] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 01-02 21:58:38] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:58:38] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 01-02 21:58:41] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:58:41] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 3).
[INFO 01-02 21:58:46] Scheduler: Fetching data for newly completed trials: 11 - 12.
[INFO 01-02 21:58:46] Scheduler: Retrieved COMPLETED trials: 11 - 12.
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 21:58:51] Scheduler: Running trials [14]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 21:58:58] Scheduler: Running trials [15]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 21:59:04] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 01-02 21:59:04] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:59:04] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 01-02 21:59:05] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:59:05] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 01-02 21:59:06] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:59:06] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 01-02 21:59:09] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:59:09] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 01-02 21:59:12] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:59:12] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 3).
[INFO 01-02 21:59:17] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:59:17] Scheduler: Waiting for completed trials (for 7 sec, currently running trials: 3).
[INFO 01-02 21:59:25] Scheduler: Fetching data for newly completed trials: [13].
[INFO 01-02 21:59:25] Scheduler: Retrieved COMPLETED trials: [13].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 21:59:31] Scheduler: Running trials [16]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 21:59:36] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 01-02 21:59:36] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:59:36] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 01-02 21:59:37] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:59:37] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 01-02 21:59:39] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:59:39] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 01-02 21:59:41] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:59:41] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 01-02 21:59:44] Scheduler: Fetching data for newly completed trials: [15].
[INFO 01-02 21:59:44] Scheduler: Retrieved COMPLETED trials: [15].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 21:59:52] Scheduler: Running trials [17]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 21:59:57] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 01-02 21:59:57] Scheduler: Fetching data for newly completed trials: [14].
[INFO 01-02 21:59:57] Scheduler: Retrieved COMPLETED trials: [14].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:00:05] Scheduler: Running trials [18]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:00:11] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 01-02 22:00:11] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:00:11] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 01-02 22:00:12] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:00:12] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 01-02 22:00:13] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:00:13] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 01-02 22:00:16] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:00:16] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 01-02 22:00:19] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:00:19] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 3).
[INFO 01-02 22:00:24] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:00:24] Scheduler: Waiting for completed trials (for 7 sec, currently running trials: 3).
[INFO 01-02 22:00:32] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:00:32] Scheduler: Waiting for completed trials (for 11 sec, currently running trials: 3).
[INFO 01-02 22:00:43] Scheduler: Fetching data for newly completed trials: [16].
[INFO 01-02 22:00:43] Scheduler: Retrieved COMPLETED trials: [16].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:00:54] Scheduler: Running trials [19]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:01:00] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 01-02 22:01:00] Scheduler: Fetching data for newly completed trials: [17].
[INFO 01-02 22:01:00] Scheduler: Retrieved COMPLETED trials: [17].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:01:07] Scheduler: Running trials [20]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:01:12] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 01-02 22:01:12] Scheduler: Fetching data for newly completed trials: [18].
[INFO 01-02 22:01:12] Scheduler: Retrieved COMPLETED trials: [18].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:01:24] Scheduler: Running trials [21]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:01:30] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 01-02 22:01:30] Scheduler: Fetching data for newly completed trials: [19].
[INFO 01-02 22:01:30] Scheduler: Retrieved COMPLETED trials: [19].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:01:47] Scheduler: Running trials [22]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:01:53] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 01-02 22:01:53] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:01:53] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 01-02 22:01:54] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:01:54] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 01-02 22:01:56] Scheduler: Fetching data for newly completed trials: [21].
[INFO 01-02 22:01:56] Scheduler: Retrieved COMPLETED trials: [21].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:02:13] Scheduler: Running trials [23]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:02:16] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 01-02 22:02:16] Scheduler: Fetching data for newly completed trials: [20, 22].
[INFO 01-02 22:02:16] Scheduler: Retrieved COMPLETED trials: [20, 22].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:02:29] Scheduler: Running trials [24]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:02:44] Scheduler: Running trials [25]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:02:53] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 01-02 22:02:53] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:02:53] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 01-02 22:02:54] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:02:54] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 01-02 22:02:56] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:02:56] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 01-02 22:02:58] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:02:58] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 01-02 22:03:02] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:03:02] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 3).
[INFO 01-02 22:03:07] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:03:07] Scheduler: Waiting for completed trials (for 7 sec, currently running trials: 3).
[INFO 01-02 22:03:14] Scheduler: Fetching data for newly completed trials: [23].
[INFO 01-02 22:03:14] Scheduler: Retrieved COMPLETED trials: [23].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:03:24] Scheduler: Running trials [26]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:03:28] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 01-02 22:03:28] Scheduler: Fetching data for newly completed trials: [24].
[INFO 01-02 22:03:28] Scheduler: Retrieved COMPLETED trials: [24].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:03:44] Scheduler: Running trials [27]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:03:52] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 01-02 22:03:52] Scheduler: Fetching data for newly completed trials: [25].
[INFO 01-02 22:03:52] Scheduler: Retrieved COMPLETED trials: [25].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:04:14] Scheduler: Running trials [28]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:04:25] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 01-02 22:04:25] Scheduler: Fetching data for newly completed trials: [26].
[INFO 01-02 22:04:25] Scheduler: Retrieved COMPLETED trials: [26].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:04:44] Scheduler: Running trials [29]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:04:55] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 01-02 22:04:55] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:04:55] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 01-02 22:04:56] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:04:56] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 01-02 22:04:57] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:04:57] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 01-02 22:04:59] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:04:59] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 01-02 22:05:03] Scheduler: Fetching data for newly completed trials: [27].
[INFO 01-02 22:05:03] Scheduler: Retrieved COMPLETED trials: [27].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:05:18] Scheduler: Running trials [30]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:05:27] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 01-02 22:05:27] Scheduler: Fetching data for newly completed trials: [28].
[INFO 01-02 22:05:27] Scheduler: Retrieved COMPLETED trials: [28].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:05:37] Scheduler: Running trials [31]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:05:45] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 01-02 22:05:45] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:05:45] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 01-02 22:05:46] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:05:46] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 01-02 22:05:47] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:05:47] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 01-02 22:05:50] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:05:50] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 01-02 22:05:53] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:05:53] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 3).
[INFO 01-02 22:05:58] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:05:58] Scheduler: Waiting for completed trials (for 7 sec, currently running trials: 3).
[INFO 01-02 22:06:06] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:06:06] Scheduler: Waiting for completed trials (for 11 sec, currently running trials: 3).
[INFO 01-02 22:06:17] Scheduler: Fetching data for newly completed trials: 29 - 30.
[INFO 01-02 22:06:17] Scheduler: Retrieved COMPLETED trials: 29 - 30.
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:06:26] Scheduler: Running trials [32]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:06:35] Scheduler: Running trials [33]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:06:43] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 01-02 22:06:43] Scheduler: Fetching data for newly completed trials: [31].
[INFO 01-02 22:06:43] Scheduler: Retrieved COMPLETED trials: [31].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:06:55] Scheduler: Running trials [34]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:07:04] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 01-02 22:07:04] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:07:04] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 01-02 22:07:05] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:07:05] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 01-02 22:07:06] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:07:06] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 01-02 22:07:09] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:07:09] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 01-02 22:07:12] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:07:12] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 3).
[INFO 01-02 22:07:17] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:07:17] Scheduler: Waiting for completed trials (for 7 sec, currently running trials: 3).
[INFO 01-02 22:07:25] Scheduler: Fetching data for newly completed trials: [32].
[INFO 01-02 22:07:25] Scheduler: Retrieved COMPLETED trials: [32].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:07:38] Scheduler: Running trials [35]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:07:48] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 01-02 22:07:48] Scheduler: Fetching data for newly completed trials: [33].
[INFO 01-02 22:07:48] Scheduler: Retrieved COMPLETED trials: [33].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:07:59] Scheduler: Running trials [36]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:08:08] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 01-02 22:08:08] Scheduler: Fetching data for newly completed trials: [34].
[INFO 01-02 22:08:08] Scheduler: Retrieved COMPLETED trials: [34].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:08:17] Scheduler: Running trials [37]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:08:24] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 01-02 22:08:24] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:08:24] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 01-02 22:08:25] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:08:25] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 01-02 22:08:26] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:08:26] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 01-02 22:08:29] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:08:29] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 01-02 22:08:32] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:08:32] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 3).
[INFO 01-02 22:08:37] Scheduler: Fetching data for newly completed trials: [35].
[INFO 01-02 22:08:37] Scheduler: Retrieved COMPLETED trials: [35].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:08:47] Scheduler: Running trials [38]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:08:54] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 01-02 22:08:54] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:08:54] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 01-02 22:08:55] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:08:55] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 01-02 22:08:56] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:08:56] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 01-02 22:08:58] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:08:58] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 01-02 22:09:02] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:09:02] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 3).
[INFO 01-02 22:09:07] Scheduler: Fetching data for newly completed trials: [36].
[INFO 01-02 22:09:07] Scheduler: Retrieved COMPLETED trials: [36].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:09:18] Scheduler: Running trials [39]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:09:23] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 01-02 22:09:23] Scheduler: Fetching data for newly completed trials: [37].
[INFO 01-02 22:09:23] Scheduler: Retrieved COMPLETED trials: [37].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:09:36] Scheduler: Running trials [40]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:09:43] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 01-02 22:09:43] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:09:43] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 01-02 22:09:44] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:09:44] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 01-02 22:09:46] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:09:46] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 01-02 22:09:48] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:09:48] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 01-02 22:09:51] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:09:51] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 3).
[INFO 01-02 22:09:56] Scheduler: Fetching data for newly completed trials: [38].
[INFO 01-02 22:09:56] Scheduler: Retrieved COMPLETED trials: [38].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:10:08] Scheduler: Running trials [41]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:10:16] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 01-02 22:10:16] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:10:16] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 01-02 22:10:17] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:10:17] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 01-02 22:10:18] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:10:18] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 01-02 22:10:21] Scheduler: Fetching data for newly completed trials: [39].
[INFO 01-02 22:10:21] Scheduler: Retrieved COMPLETED trials: [39].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:10:40] Scheduler: Running trials [42]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:10:55] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 01-02 22:10:55] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:10:55] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 01-02 22:10:56] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:10:56] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 01-02 22:10:57] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:10:57] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 01-02 22:10:59] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:10:59] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 01-02 22:11:03] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:11:03] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 3).
[INFO 01-02 22:11:08] Scheduler: Fetching data for newly completed trials: 40 - 41.
[INFO 01-02 22:11:08] Scheduler: Retrieved COMPLETED trials: 40 - 41.
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:11:15] Scheduler: Running trials [43]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:11:24] Scheduler: Running trials [44]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:11:29] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 01-02 22:11:29] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:11:29] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 01-02 22:11:30] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:11:30] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 01-02 22:11:31] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:11:31] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 01-02 22:11:34] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:11:34] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 01-02 22:11:37] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:11:37] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 3).
[INFO 01-02 22:11:42] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:11:42] Scheduler: Waiting for completed trials (for 7 sec, currently running trials: 3).
[INFO 01-02 22:11:50] Scheduler: Fetching data for newly completed trials: [42].
[INFO 01-02 22:11:50] Scheduler: Retrieved COMPLETED trials: [42].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:12:01] Scheduler: Running trials [45]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:12:07] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 01-02 22:12:07] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:12:07] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 01-02 22:12:08] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:12:08] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 01-02 22:12:10] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:12:10] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 01-02 22:12:12] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:12:12] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 01-02 22:12:16] Scheduler: Fetching data for newly completed trials: [43].
[INFO 01-02 22:12:16] Scheduler: Retrieved COMPLETED trials: [43].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:12:27] Scheduler: Running trials [46]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:12:31] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 01-02 22:12:31] Scheduler: Fetching data for newly completed trials: [44].
[INFO 01-02 22:12:31] Scheduler: Retrieved COMPLETED trials: [44].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:12:40] Scheduler: Running trials [47]...
[INFO 01-02 22:12:40] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:12:40] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 01-02 22:12:41] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:12:41] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 01-02 22:12:43] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:12:43] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 01-02 22:12:45] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:12:45] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 01-02 22:12:49] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:12:49] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 3).
[INFO 01-02 22:12:54] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:12:54] Scheduler: Waiting for completed trials (for 7 sec, currently running trials: 3).
[INFO 01-02 22:13:01] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:13:01] Scheduler: Waiting for completed trials (for 11 sec, currently running trials: 3).
[INFO 01-02 22:13:13] Scheduler: Fetching data for newly completed trials: [45].
[INFO 01-02 22:13:13] Scheduler: Retrieved COMPLETED trials: [45].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:13:13] Scheduler: Done submitting trials, waiting for remaining 2 running trials...
[INFO 01-02 22:13:13] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:13:13] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 2).
[INFO 01-02 22:13:14] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:13:14] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 2).
[INFO 01-02 22:13:15] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:13:15] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 2).
[INFO 01-02 22:13:18] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:13:18] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 2).
[INFO 01-02 22:13:21] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:13:21] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 2).
[INFO 01-02 22:13:26] Scheduler: Fetching data for newly completed trials: [46].
[INFO 01-02 22:13:26] Scheduler: Retrieved COMPLETED trials: [46].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:13:26] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:13:26] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 1).
[INFO 01-02 22:13:27] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:13:27] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 1).
[INFO 01-02 22:13:29] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:13:29] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 1).
[INFO 01-02 22:13:31] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:13:31] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 1).
[INFO 01-02 22:13:34] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:13:34] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 1).
[INFO 01-02 22:13:39] Scheduler: Fetching data for newly completed trials: [47].
[INFO 01-02 22:13:39] Scheduler: Retrieved COMPLETED trials: [47].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.


OptimizationResult()

评估结果

我们现在可以使用 Ax 提供的辅助函数和可视化工具来检查优化的结果。

首先，我们生成一个数据框，以总结实验的结果。此数据框中的每一行对应一个试验（即运行的一个训练任务），并包含该试验的状态、评估的参数配置以及观察到的度量值。这为优化过程提供了一种简便的方式来进行合理性检查。

from ax.service.utils.report_utils import exp_to_df

df = exp_to_df(experiment)
df.head(10)

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[WARNING 01-02 22:13:39] ax.service.utils.report_utils: Column reason missing for all trials. Not appending column.
trial_index arm_name trial_status generation_method is_feasible num_params val_acc hidden_size_1 hidden_size_2 learning_rate 训练周期 丢弃 batch_size
0 0 0_0 COMPLETED Sobol 假的 16810.0 0.892440 19 66 0.003182 4 0.190970 32
1 1 1_0 COMPLETED Sobol 假的 40280.0 0.916779 50 18 0.000614 2 0.425028 128
2 2 2_0 COMPLETED Sobol 假的 112720.0 0.963429 127 96 0.001233 3 0.334354 256
3 3 3_0 COMPLETED Sobol 假的 30653.0 0.905993 37 35 0.000234 1 0.037933 64
4 4 4_0 COMPLETED Sobol 假的 26056.0 0.917627 28 108 0.000339 1 0.062996 32
5 5 5_0 COMPLETED Sobol 真 71312.0 0.951953 87 32 0.006412 3 0.297078 128
6 6 6_0 COMPLETED Sobol 假的 50051.0 0.862612 59 55 0.000116 2 0.458433 256
7 7 7_0 COMPLETED Sobol 假的 19530.0 0.936613 24 21 0.002236 4 0.161959 64
8 8 8_0 COMPLETED Sobol 假的 18080.0 0.905648 20 80 0.000163 3 0.388082 64
9 9 9_0 COMPLETED BoTorch 真 46403.0 0.952467 53 77 0.002655 3 0.263433 256


我们还可以可视化验证准确率与模型参数数量之间的权衡帕累托前沿。

提示

Ax 使用 Plotly 生成交互式图表，这些图表允许您进行缩放、裁剪或悬停以查看图表组件的详细信息。试试看，如果您想了解更多，请参阅 可视化教程 。

最终的优化结果如下面的图所示，其中颜色对应每个试验的迭代次数。我们看到我们的方法成功地探索了权衡，并找到了具有高验证准确率的大模型以及验证准确率相对较低的小模型。

from ax.service.utils.report_utils import _pareto_frontier_scatter_2d_plotly

_pareto_frontier_scatter_2d_plotly(experiment)

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[WARNING 01-02 22:13:40] ax.service.utils.report_utils: Column reason missing for all trials. Not appending column.


为了更好地理解我们的替代模型从黑箱目标中学习到了什么，我们可以查看留一法交叉验证的结果。由于我们的模型是高斯过程，它们不仅提供点预测，还提供这些预测的不确定性估计。一个好的模型意味着预测均值（图中的点）接近45度线，并且置信区间以预期的频率覆盖45度线（在这里我们使用95%的置信区间，因此我们期望它们在95%的情况下包含真实观测值）。

如下面的图表所示，模型大小（num_params）指标比验证准确率（val_acc）指标更容易建模。

from ax.modelbridge.cross_validation import compute_diagnostics, cross_validate
from ax.plot.diagnostic import interact_cross_validation_plotly
from ax.utils.notebook.plotting import init_notebook_plotting, render

cv = cross_validate(model=gs.model)  # The surrogate model is stored on the ``GenerationStrategy``
compute_diagnostics(cv)

interact_cross_validation_plotly(cv)


我们也可以绘制等高线图，以更好地理解不同的目标函数如何依赖于两个输入参数。在下面的图中，我们展示了模型预测的验证准确率作为两个隐藏层大小的函数。随着隐藏层大小的增加，验证准确率明显提高。

from ax.plot.contour import interact_contour_plotly

interact_contour_plotly(model=gs.model, metric_name="val_acc")


同样，我们在下面的图中展示了模型参数数量随隐藏层大小的变化情况，并看到它也随着隐藏层大小的增加而增加（对hidden_size_1 的依赖更大）。

interact_contour_plotly(model=gs.model, metric_name="num_params")


致谢

我们感谢TorchX团队（特别是Kiuk Chung和Tristan Rice）在将TorchX与Ax集成方面提供的帮助。

脚本总运行时间: ( 17 分钟 54.688 秒)

通过 Sphinx-Gallery 生成的画廊

文档

访问 PyTorch 的全面开发人员文档

查看文档

教程

获取面向初学者和高级开发人员的深入教程

查看教程

资源

查找开发资源并解答您的问题

查看资源