注意

单击此处下载完整的示例代码

带 Ax 的多目标 NAS¶

创建时间： 2022 年 8 月 19 日 |上次更新时间： 2024 年 7 月 31 日 |上次验证： Nov 05， 2024

作者： David Eriksson， Max Balandat，以及 Meta 的 Adaptive Experimentation 团队。

在本教程中，我们将展示如何使用 Ax 运行用于简单神经的多目标神经结构搜索（NAS） network 模型。虽然底层的方法通常用于更复杂的模型，并且较大的数据集，我们选择易于运行的教程在不到 20 分钟的时间内在笔记本电脑上端到端完成。

在许多 NAS 应用程序中，多个 NAS 之间存在自然的权衡感兴趣的目标。例如，在设备上部署模型时我们可能希望最大限度地提高模型性能（例如，准确性），而同时最大限度地减少功耗等竞争指标，推理延迟或模型大小，以满足部署约束。通常，我们可能能够减少计算需求或预测的延迟，通过接受最低限度的降低模型性能。探索此类权衡的原则方法高效地是可扩展和可持续 AI 的关键推动因素，并且具有 Meta 的许多成功应用 - 例如，请参阅我们关于自然语言理解模型的案例研究。

在我们的示例中，我们将调整两个隐藏层的宽度，学习率、 dropout probability （随机辍学概率）、batch size （批量大小）和训练 epoch 数。目标是权衡性能（验证集上的准确率）和模型大小（ model 参数）。

本教程使用以下 PyTorch 库：

PyTorch Lightning（指定模型和训练循环）
TorchX（用于远程/异步运行训练作业）
BoTorch （为 Ax 算法提供支持的贝叶斯优化库）

定义 TorchX 应用程序¶

我们的目标是优化 mnist_train_nas.py 中定义的 PyTorch Lightning 训练作业。要使用 TorchX 执行此作，我们编写一个辅助函数，该函数接收训练的架构和超参数的值 job 并创建具有适当设置的 TorchX AppDef。

from pathlib import Path

import torchx

from torchx import specs
from torchx.components import utils


def trainer(
    log_path: str,
    hidden_size_1: int,
    hidden_size_2: int,
    learning_rate: float,
    epochs: int,
    dropout: float,
    batch_size: int,
    trial_idx: int = -1,
) -> specs.AppDef:

    # define the log path so we can pass it to the TorchX ``AppDef``
    if trial_idx >= 0:
        log_path = Path(log_path).joinpath(str(trial_idx)).absolute().as_posix()

    return utils.python(
        # command line arguments to the training script
        "--log_path",
        log_path,
        "--hidden_size_1",
        str(hidden_size_1),
        "--hidden_size_2",
        str(hidden_size_2),
        "--learning_rate",
        str(learning_rate),
        "--epochs",
        str(epochs),
        "--dropout",
        str(dropout),
        "--batch_size",
        str(batch_size),
        # other config options
        name="trainer",
        script="mnist_train_nas.py",
        image=torchx.version.TORCHX_IMAGE,
    )

设置 Runner¶

Ax 的 Runner 抽象允许将接口写入各种后端。 Ax 已经附带了 TorchX 的 Runner，所以我们只需要配置它。在本教程中，我们将在本地运行作业以完全异步的方式。

为了在集群上启动它们，您可以改为指定不同的 TorchX 调度器，并适当调整配置。例如，如果您有一个 Kubernetes 集群，则只需将 scheduler 从到）。local_cwdkubernetes

import tempfile
from ax.runners.torchx import TorchXRunner

# Make a temporary dir to log our results into
log_dir = tempfile.mkdtemp()

ax_runner = TorchXRunner(
    tracker_base="/tmp/",
    component=trainer,
    # NOTE: To launch this job on a cluster instead of locally you can
    # specify a different scheduler and adjust arguments appropriately.
    scheduler="local_cwd",
    component_const_params={"log_path": log_dir},
    cfg={},
)

设置`SearchSpace`¶

首先，我们定义我们的搜索空间。Ax 支持两个范围参数类型 integer 和 float 以及 choice 参数，这些参数可以具有非数字类型，例如字符串。我们将调整隐藏的大小、学习率、辍学和数量 epochs 作为范围参数，并将批量大小调整为有序选择参数强制其为 2 的幂。

from ax.core import (
    ChoiceParameter,
    ParameterType,
    RangeParameter,
    SearchSpace,
)

parameters = [
    # NOTE: In a real-world setting, hidden_size_1 and hidden_size_2
    # should probably be powers of 2, but in our simple example this
    # would mean that ``num_params`` can't take on that many values, which
    # in turn makes the Pareto frontier look pretty weird.
    RangeParameter(
        name="hidden_size_1",
        lower=16,
        upper=128,
        parameter_type=ParameterType.INT,
        log_scale=True,
    ),
    RangeParameter(
        name="hidden_size_2",
        lower=16,
        upper=128,
        parameter_type=ParameterType.INT,
        log_scale=True,
    ),
    RangeParameter(
        name="learning_rate",
        lower=1e-4,
        upper=1e-2,
        parameter_type=ParameterType.FLOAT,
        log_scale=True,
    ),
    RangeParameter(
        name="epochs",
        lower=1,
        upper=4,
        parameter_type=ParameterType.INT,
    ),
    RangeParameter(
        name="dropout",
        lower=0.0,
        upper=0.5,
        parameter_type=ParameterType.FLOAT,
    ),
    ChoiceParameter(  # NOTE: ``ChoiceParameters`` don't require log-scale
        name="batch_size",
        values=[32, 64, 128, 256],
        parameter_type=ParameterType.INT,
        is_ordered=True,
        sort_values=True,
    ),
]

search_space = SearchSpace(
    parameters=parameters,
    # NOTE: In practice, it may make sense to add a constraint
    # hidden_size_2 <= hidden_size_1
    parameter_constraints=[],
)

设置量度¶

Ax 有一个 Metric 的概念，它定义了结果的属性以及如何获得观察对于这些结果。这允许编码，例如如何从中获取数据一些分布式执行后端和后处理后作为 Ax.

在本教程中，我们将使用多目标优化，目标是最大限度地提高验证精度和最小化模型参数的数量。后者表示一个简单的代理模型延迟，对于小型 ML 来说，很难准确估计模型（在实际应用程序中，我们会对延迟进行基准测试，而在设备上运行模型）。

在我们的示例中，TorchX 将以完全异步的方式运行训练作业 fashion 并将结果写入基于 Trial 的 index 的 intent （请参阅上面的函数）。我们将定义一个指标类。通过子类化 TensorboardCurveMetric，我们可以免费获得读取和解析 TensorBoard 日志的逻辑。log_dirtrainer()

from ax.metrics.tensorboard import TensorboardMetric
from tensorboard.backend.event_processing import plugin_event_multiplexer as event_multiplexer

class MyTensorboardMetric(TensorboardMetric):

    # NOTE: We need to tell the new TensorBoard metric how to get the id /
    # file handle for the TensorBoard logs from a trial. In this case
    # our convention is to just save a separate file per trial in
    # the prespecified log dir.
    def _get_event_multiplexer_for_trial(self, trial):
        mul = event_multiplexer.EventMultiplexer(max_reload_threads=20)
        mul.AddRunsFromDirectory(Path(log_dir).joinpath(str(trial.index)).as_posix(), None)
        mul.Reload()

        return mul

    # This indicates whether the metric is queryable while the trial is
    # still running. We don't use this in the current tutorial, but Ax
    # utilizes this to implement trial-level early-stopping functionality.
    @classmethod
    def is_available_while_running(cls):
        return False

现在我们可以实例化准确性和 model 参数。这里curve_name是 TensorBoard 日志，而 name 是内部使用的指标名称由 Ax.我们还指定了 lower_is_better 以表示有利方向。

val_acc = MyTensorboardMetric(
    name="val_acc",
    tag="val_acc",
    lower_is_better=False,
)
model_num_params = MyTensorboardMetric(
    name="num_params",
    tag="num_params",
    lower_is_better=True,
)

设置`OptimizationConfig`¶

告诉 Ax 应该优化什么的方法是通过 OptimizationConfig。在这里，我们随心所欲地使用 a 正在执行多目标优化。MultiObjectiveOptimizationConfig

此外，Ax 支持将约束放在不同的指标，通过指定目标阈值来限制区域对我们想要探索的结果空间感兴趣。对于这个示例中，我们将验证精度限制为至少 0.94 （94%）且模型参数的数量最多为 80,000。

from ax.core import MultiObjective, Objective, ObjectiveThreshold
from ax.core.optimization_config import MultiObjectiveOptimizationConfig


opt_config = MultiObjectiveOptimizationConfig(
    objective=MultiObjective(
        objectives=[
            Objective(metric=val_acc, minimize=False),
            Objective(metric=model_num_params, minimize=True),
        ],
    ),
    objective_thresholds=[
        ObjectiveThreshold(metric=val_acc, bound=0.94, relative=False),
        ObjectiveThreshold(metric=model_num_params, bound=80_000, relative=False),
    ],
)

创建 Ax 实验¶

在 Ax 中，Experiment 对象是存储有关问题的所有信息的对象设置。

from ax.core import Experiment

experiment = Experiment(
    name="torchx_mnist",
    search_space=search_space,
    optimization_config=opt_config,
    runner=ax_runner,
)

选择生成策略¶

GenerationStrategy 是我们希望如何执行优化。虽然这可以自定义（如果您想这样做，请参阅本教程），但在大多数情况下，Ax 可以自动确定适当的策略根据 Search Space、Optimization Config 和 Total Number 我们想要运行的试验。

通常，Ax 选择评估许多随机配置在开始基于模型的贝叶斯优化策略之前。

total_trials = 48  # total evaluation budget

from ax.modelbridge.dispatch_utils import choose_generation_strategy

gs = choose_generation_strategy(
    search_space=experiment.search_space,
    optimization_config=experiment.optimization_config,
    num_trials=total_trials,
  )

[INFO 01-02 21:56:00] ax.modelbridge.dispatch_utils: Using Models.BOTORCH_MODULAR since there is at least one ordered parameter and there are no unordered categorical parameters.
[INFO 01-02 21:56:00] ax.modelbridge.dispatch_utils: Calculating the number of remaining initialization trials based on num_initialization_trials=None max_initialization_trials=None num_tunable_parameters=6 num_trials=48 use_batch_trials=False
[INFO 01-02 21:56:00] ax.modelbridge.dispatch_utils: calculated num_initialization_trials=9
[INFO 01-02 21:56:00] ax.modelbridge.dispatch_utils: num_completed_initialization_trials=0 num_remaining_initialization_trials=9
[INFO 01-02 21:56:00] ax.modelbridge.dispatch_utils: `verbose`, `disable_progbar`, and `jit_compile` are not yet supported when using `choose_generation_strategy` with ModularBoTorchModel, dropping these arguments.
[INFO 01-02 21:56:00] ax.modelbridge.dispatch_utils: Using Bayesian Optimization generation strategy: GenerationStrategy(name='Sobol+BoTorch', steps=[Sobol for 9 trials, BoTorch for subsequent trials]). Iterations after 9 will take longer to generate due to model-fitting.

配置调度程序¶

它充当优化的 loop control。它与后端通信以启动 Trial 并检查其状态，并检索结果。在本教程中，它只是读取并解析本地保存的日志。在远程执行设置中，它将调用 API。Ax Scheduler 教程中的下图总结了 Scheduler 如何与用于运行的外部系统交互试验评估：Scheduler

../_static/img/ax_scheduler_illustration.png

需要和。可以通过 .在这里，我们配置总评估数以及，应同时运行的最大 Trial 数。在我们的 local 设置，这是作为 individual 运行的训练作业的数量 processes 的 s，而在 Remote Execution 设置中，这将是 number 要并行使用的计算机。SchedulerExperimentGenerationStrategySchedulerOptionsmax_pending_trials

from ax.service.scheduler import Scheduler, SchedulerOptions

scheduler = Scheduler(
    experiment=experiment,
    generation_strategy=gs,
    options=SchedulerOptions(
        total_trials=total_trials, max_pending_trials=4
    ),
)

[WARNING 01-02 21:56:00] ax.service.utils.with_db_settings_base: Ax currently requires a sqlalchemy version below 2.0. This will be addressed in a future release. Disabling SQL storage in Ax for now, if you would like to use SQL storage please install Ax with mysql extras via `pip install ax-platform[mysql]`.
[INFO 01-02 21:56:00] Scheduler: `Scheduler` requires experiment to have immutable search space and optimization config. Setting property immutable_search_space_and_opt_config to `True` on experiment.

运行优化¶

现在一切都已配置完毕，我们可以让 Ax 运行优化以完全自动化的方式。Scheduler 将定期检查当前正在运行的 Trial 的状态日志，如果 Trial 完成时，调度程序将在 experiment 并获取 Bayesian 所需的观测值优化算法。

scheduler.run_all_trials()

[INFO 01-02 21:56:00] Scheduler: Fetching data for newly completed trials: [].
/usr/local/lib/python3.10/dist-packages/ax/modelbridge/cross_validation.py:463: UserWarning:

Encountered exception in computing model fit quality: RandomModelBridge does not support prediction.

[INFO 01-02 21:56:00] Scheduler: Running trials [0]...
/usr/local/lib/python3.10/dist-packages/ax/modelbridge/cross_validation.py:463: UserWarning:

Encountered exception in computing model fit quality: RandomModelBridge does not support prediction.

[INFO 01-02 21:56:01] Scheduler: Running trials [1]...
/usr/local/lib/python3.10/dist-packages/ax/modelbridge/cross_validation.py:463: UserWarning:

Encountered exception in computing model fit quality: RandomModelBridge does not support prediction.

[INFO 01-02 21:56:02] Scheduler: Running trials [2]...
/usr/local/lib/python3.10/dist-packages/ax/modelbridge/cross_validation.py:463: UserWarning:

Encountered exception in computing model fit quality: RandomModelBridge does not support prediction.

[INFO 01-02 21:56:03] Scheduler: Running trials [3]...
[INFO 01-02 21:56:04] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:56:04] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 4).
[INFO 01-02 21:56:05] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:56:05] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 4).
[INFO 01-02 21:56:07] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:56:07] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 4).
[INFO 01-02 21:56:09] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:56:09] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 4).
[INFO 01-02 21:56:12] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:56:12] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 4).
[INFO 01-02 21:56:17] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:56:17] Scheduler: Waiting for completed trials (for 7 sec, currently running trials: 4).
[INFO 01-02 21:56:25] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:56:25] Scheduler: Waiting for completed trials (for 11 sec, currently running trials: 4).
[INFO 01-02 21:56:36] Scheduler: Fetching data for newly completed trials: [1, 3].
[INFO 01-02 21:56:36] Scheduler: Retrieved COMPLETED trials: [1, 3].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/modelbridge/cross_validation.py:463: UserWarning:

Encountered exception in computing model fit quality: RandomModelBridge does not support prediction.

[INFO 01-02 21:56:36] Scheduler: Running trials [4]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/modelbridge/cross_validation.py:463: UserWarning:

Encountered exception in computing model fit quality: RandomModelBridge does not support prediction.

[INFO 01-02 21:56:37] Scheduler: Running trials [5]...
[INFO 01-02 21:56:38] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:56:38] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 4).
[INFO 01-02 21:56:39] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:56:39] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 4).
[INFO 01-02 21:56:41] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:56:41] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 4).
[INFO 01-02 21:56:43] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:56:43] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 4).
[INFO 01-02 21:56:47] Scheduler: Fetching data for newly completed trials: [2].
[INFO 01-02 21:56:47] Scheduler: Retrieved COMPLETED trials: [2].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/modelbridge/cross_validation.py:463: UserWarning:

Encountered exception in computing model fit quality: RandomModelBridge does not support prediction.

[INFO 01-02 21:56:47] Scheduler: Running trials [6]...
[INFO 01-02 21:56:48] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:56:48] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 4).
[INFO 01-02 21:56:49] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:56:49] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 4).
[INFO 01-02 21:56:50] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:56:50] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 4).
[INFO 01-02 21:56:52] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:56:52] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 4).
[INFO 01-02 21:56:56] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:56:56] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 4).
[INFO 01-02 21:57:01] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:57:01] Scheduler: Waiting for completed trials (for 7 sec, currently running trials: 4).
[INFO 01-02 21:57:08] Scheduler: Fetching data for newly completed trials: [4].
[INFO 01-02 21:57:08] Scheduler: Retrieved COMPLETED trials: [4].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/modelbridge/cross_validation.py:463: UserWarning:

Encountered exception in computing model fit quality: RandomModelBridge does not support prediction.

[INFO 01-02 21:57:09] Scheduler: Running trials [7]...
[INFO 01-02 21:57:10] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:57:10] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 4).
[INFO 01-02 21:57:11] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:57:11] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 4).
[INFO 01-02 21:57:12] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:57:12] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 4).
[INFO 01-02 21:57:14] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:57:14] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 4).
[INFO 01-02 21:57:18] Scheduler: Fetching data for newly completed trials: [6].
[INFO 01-02 21:57:18] Scheduler: Retrieved COMPLETED trials: [6].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/modelbridge/cross_validation.py:463: UserWarning:

Encountered exception in computing model fit quality: RandomModelBridge does not support prediction.

[INFO 01-02 21:57:18] Scheduler: Running trials [8]...
[INFO 01-02 21:57:19] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:57:19] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 4).
[INFO 01-02 21:57:20] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:57:20] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 4).
[INFO 01-02 21:57:21] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:57:21] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 4).
[INFO 01-02 21:57:24] Scheduler: Fetching data for newly completed trials: [0, 5].
[INFO 01-02 21:57:24] Scheduler: Retrieved COMPLETED trials: [0, 5].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 21:57:27] Scheduler: Running trials [9]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 21:57:33] Scheduler: Running trials [10]...
[INFO 01-02 21:57:34] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:57:34] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 4).
[INFO 01-02 21:57:35] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:57:35] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 4).
[INFO 01-02 21:57:37] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:57:37] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 4).
[INFO 01-02 21:57:39] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:57:39] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 4).
[INFO 01-02 21:57:42] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:57:42] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 4).
[INFO 01-02 21:57:47] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:57:47] Scheduler: Waiting for completed trials (for 7 sec, currently running trials: 4).
[INFO 01-02 21:57:55] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:57:55] Scheduler: Waiting for completed trials (for 11 sec, currently running trials: 4).
[INFO 01-02 21:58:06] Scheduler: Fetching data for newly completed trials: [10].
[INFO 01-02 21:58:06] Scheduler: Retrieved COMPLETED trials: [10].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 21:58:14] Scheduler: Running trials [11]...
[INFO 01-02 21:58:15] Scheduler: Fetching data for newly completed trials: 8 - 9.
[INFO 01-02 21:58:15] Scheduler: Retrieved COMPLETED trials: 8 - 9.
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 21:58:23] Scheduler: Running trials [12]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 21:58:30] Scheduler: Running trials [13]...
[INFO 01-02 21:58:31] Scheduler: Fetching data for newly completed trials: [7].
[INFO 01-02 21:58:31] Scheduler: Retrieved COMPLETED trials: [7].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 21:58:33] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:58:33] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 01-02 21:58:34] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:58:34] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 01-02 21:58:35] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:58:35] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 01-02 21:58:38] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:58:38] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 01-02 21:58:41] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:58:41] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 3).
[INFO 01-02 21:58:46] Scheduler: Fetching data for newly completed trials: 11 - 12.
[INFO 01-02 21:58:46] Scheduler: Retrieved COMPLETED trials: 11 - 12.
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 21:58:51] Scheduler: Running trials [14]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 21:58:58] Scheduler: Running trials [15]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 21:59:04] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 01-02 21:59:04] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:59:04] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 01-02 21:59:05] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:59:05] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 01-02 21:59:06] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:59:06] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 01-02 21:59:09] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:59:09] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 01-02 21:59:12] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:59:12] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 3).
[INFO 01-02 21:59:17] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:59:17] Scheduler: Waiting for completed trials (for 7 sec, currently running trials: 3).
[INFO 01-02 21:59:25] Scheduler: Fetching data for newly completed trials: [13].
[INFO 01-02 21:59:25] Scheduler: Retrieved COMPLETED trials: [13].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 21:59:31] Scheduler: Running trials [16]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 21:59:36] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 01-02 21:59:36] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:59:36] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 01-02 21:59:37] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:59:37] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 01-02 21:59:39] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:59:39] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 01-02 21:59:41] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:59:41] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 01-02 21:59:44] Scheduler: Fetching data for newly completed trials: [15].
[INFO 01-02 21:59:44] Scheduler: Retrieved COMPLETED trials: [15].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 21:59:52] Scheduler: Running trials [17]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 21:59:57] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 01-02 21:59:57] Scheduler: Fetching data for newly completed trials: [14].
[INFO 01-02 21:59:57] Scheduler: Retrieved COMPLETED trials: [14].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:00:05] Scheduler: Running trials [18]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:00:11] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 01-02 22:00:11] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:00:11] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 01-02 22:00:12] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:00:12] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 01-02 22:00:13] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:00:13] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 01-02 22:00:16] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:00:16] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 01-02 22:00:19] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:00:19] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 3).
[INFO 01-02 22:00:24] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:00:24] Scheduler: Waiting for completed trials (for 7 sec, currently running trials: 3).
[INFO 01-02 22:00:32] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:00:32] Scheduler: Waiting for completed trials (for 11 sec, currently running trials: 3).
[INFO 01-02 22:00:43] Scheduler: Fetching data for newly completed trials: [16].
[INFO 01-02 22:00:43] Scheduler: Retrieved COMPLETED trials: [16].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:00:54] Scheduler: Running trials [19]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:01:00] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 01-02 22:01:00] Scheduler: Fetching data for newly completed trials: [17].
[INFO 01-02 22:01:00] Scheduler: Retrieved COMPLETED trials: [17].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:01:07] Scheduler: Running trials [20]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:01:12] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 01-02 22:01:12] Scheduler: Fetching data for newly completed trials: [18].
[INFO 01-02 22:01:12] Scheduler: Retrieved COMPLETED trials: [18].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:01:24] Scheduler: Running trials [21]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:01:30] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 01-02 22:01:30] Scheduler: Fetching data for newly completed trials: [19].
[INFO 01-02 22:01:30] Scheduler: Retrieved COMPLETED trials: [19].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:01:47] Scheduler: Running trials [22]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:01:53] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 01-02 22:01:53] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:01:53] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 01-02 22:01:54] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:01:54] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 01-02 22:01:56] Scheduler: Fetching data for newly completed trials: [21].
[INFO 01-02 22:01:56] Scheduler: Retrieved COMPLETED trials: [21].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:02:13] Scheduler: Running trials [23]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:02:16] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 01-02 22:02:16] Scheduler: Fetching data for newly completed trials: [20, 22].
[INFO 01-02 22:02:16] Scheduler: Retrieved COMPLETED trials: [20, 22].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:02:29] Scheduler: Running trials [24]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:02:44] Scheduler: Running trials [25]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:02:53] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 01-02 22:02:53] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:02:53] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 01-02 22:02:54] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:02:54] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 01-02 22:02:56] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:02:56] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 01-02 22:02:58] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:02:58] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 01-02 22:03:02] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:03:02] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 3).
[INFO 01-02 22:03:07] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:03:07] Scheduler: Waiting for completed trials (for 7 sec, currently running trials: 3).
[INFO 01-02 22:03:14] Scheduler: Fetching data for newly completed trials: [23].
[INFO 01-02 22:03:14] Scheduler: Retrieved COMPLETED trials: [23].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:03:24] Scheduler: Running trials [26]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:03:28] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 01-02 22:03:28] Scheduler: Fetching data for newly completed trials: [24].
[INFO 01-02 22:03:28] Scheduler: Retrieved COMPLETED trials: [24].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:03:44] Scheduler: Running trials [27]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:03:52] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 01-02 22:03:52] Scheduler: Fetching data for newly completed trials: [25].
[INFO 01-02 22:03:52] Scheduler: Retrieved COMPLETED trials: [25].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:04:14] Scheduler: Running trials [28]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:04:25] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 01-02 22:04:25] Scheduler: Fetching data for newly completed trials: [26].
[INFO 01-02 22:04:25] Scheduler: Retrieved COMPLETED trials: [26].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:04:44] Scheduler: Running trials [29]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:04:55] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 01-02 22:04:55] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:04:55] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 01-02 22:04:56] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:04:56] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 01-02 22:04:57] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:04:57] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 01-02 22:04:59] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:04:59] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 01-02 22:05:03] Scheduler: Fetching data for newly completed trials: [27].
[INFO 01-02 22:05:03] Scheduler: Retrieved COMPLETED trials: [27].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:05:18] Scheduler: Running trials [30]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:05:27] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 01-02 22:05:27] Scheduler: Fetching data for newly completed trials: [28].
[INFO 01-02 22:05:27] Scheduler: Retrieved COMPLETED trials: [28].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:05:37] Scheduler: Running trials [31]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:05:45] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 01-02 22:05:45] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:05:45] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 01-02 22:05:46] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:05:46] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 01-02 22:05:47] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:05:47] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 01-02 22:05:50] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:05:50] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 01-02 22:05:53] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:05:53] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 3).
[INFO 01-02 22:05:58] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:05:58] Scheduler: Waiting for completed trials (for 7 sec, currently running trials: 3).
[INFO 01-02 22:06:06] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:06:06] Scheduler: Waiting for completed trials (for 11 sec, currently running trials: 3).
[INFO 01-02 22:06:17] Scheduler: Fetching data for newly completed trials: 29 - 30.
[INFO 01-02 22:06:17] Scheduler: Retrieved COMPLETED trials: 29 - 30.
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:06:26] Scheduler: Running trials [32]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:06:35] Scheduler: Running trials [33]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:06:43] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 01-02 22:06:43] Scheduler: Fetching data for newly completed trials: [31].
[INFO 01-02 22:06:43] Scheduler: Retrieved COMPLETED trials: [31].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:06:55] Scheduler: Running trials [34]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:07:04] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 01-02 22:07:04] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:07:04] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 01-02 22:07:05] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:07:05] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 01-02 22:07:06] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:07:06] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 01-02 22:07:09] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:07:09] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 01-02 22:07:12] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:07:12] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 3).
[INFO 01-02 22:07:17] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:07:17] Scheduler: Waiting for completed trials (for 7 sec, currently running trials: 3).
[INFO 01-02 22:07:25] Scheduler: Fetching data for newly completed trials: [32].
[INFO 01-02 22:07:25] Scheduler: Retrieved COMPLETED trials: [32].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:07:38] Scheduler: Running trials [35]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:07:48] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 01-02 22:07:48] Scheduler: Fetching data for newly completed trials: [33].
[INFO 01-02 22:07:48] Scheduler: Retrieved COMPLETED trials: [33].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:07:59] Scheduler: Running trials [36]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:08:08] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 01-02 22:08:08] Scheduler: Fetching data for newly completed trials: [34].
[INFO 01-02 22:08:08] Scheduler: Retrieved COMPLETED trials: [34].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:08:17] Scheduler: Running trials [37]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:08:24] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 01-02 22:08:24] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:08:24] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 01-02 22:08:25] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:08:25] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 01-02 22:08:26] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:08:26] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 01-02 22:08:29] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:08:29] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 01-02 22:08:32] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:08:32] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 3).
[INFO 01-02 22:08:37] Scheduler: Fetching data for newly completed trials: [35].
[INFO 01-02 22:08:37] Scheduler: Retrieved COMPLETED trials: [35].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:08:47] Scheduler: Running trials [38]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:08:54] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 01-02 22:08:54] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:08:54] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 01-02 22:08:55] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:08:55] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 01-02 22:08:56] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:08:56] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 01-02 22:08:58] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:08:58] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 01-02 22:09:02] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:09:02] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 3).
[INFO 01-02 22:09:07] Scheduler: Fetching data for newly completed trials: [36].
[INFO 01-02 22:09:07] Scheduler: Retrieved COMPLETED trials: [36].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:09:18] Scheduler: Running trials [39]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:09:23] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 01-02 22:09:23] Scheduler: Fetching data for newly completed trials: [37].
[INFO 01-02 22:09:23] Scheduler: Retrieved COMPLETED trials: [37].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:09:36] Scheduler: Running trials [40]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:09:43] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 01-02 22:09:43] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:09:43] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 01-02 22:09:44] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:09:44] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 01-02 22:09:46] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:09:46] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 01-02 22:09:48] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:09:48] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 01-02 22:09:51] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:09:51] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 3).
[INFO 01-02 22:09:56] Scheduler: Fetching data for newly completed trials: [38].
[INFO 01-02 22:09:56] Scheduler: Retrieved COMPLETED trials: [38].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:10:08] Scheduler: Running trials [41]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:10:16] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 01-02 22:10:16] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:10:16] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 01-02 22:10:17] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:10:17] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 01-02 22:10:18] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:10:18] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 01-02 22:10:21] Scheduler: Fetching data for newly completed trials: [39].
[INFO 01-02 22:10:21] Scheduler: Retrieved COMPLETED trials: [39].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:10:40] Scheduler: Running trials [42]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:10:55] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 01-02 22:10:55] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:10:55] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 01-02 22:10:56] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:10:56] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 01-02 22:10:57] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:10:57] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 01-02 22:10:59] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:10:59] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 01-02 22:11:03] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:11:03] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 3).
[INFO 01-02 22:11:08] Scheduler: Fetching data for newly completed trials: 40 - 41.
[INFO 01-02 22:11:08] Scheduler: Retrieved COMPLETED trials: 40 - 41.
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:11:15] Scheduler: Running trials [43]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:11:24] Scheduler: Running trials [44]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:11:29] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 01-02 22:11:29] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:11:29] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 01-02 22:11:30] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:11:30] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 01-02 22:11:31] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:11:31] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 01-02 22:11:34] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:11:34] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 01-02 22:11:37] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:11:37] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 3).
[INFO 01-02 22:11:42] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:11:42] Scheduler: Waiting for completed trials (for 7 sec, currently running trials: 3).
[INFO 01-02 22:11:50] Scheduler: Fetching data for newly completed trials: [42].
[INFO 01-02 22:11:50] Scheduler: Retrieved COMPLETED trials: [42].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:12:01] Scheduler: Running trials [45]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:12:07] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 01-02 22:12:07] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:12:07] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 01-02 22:12:08] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:12:08] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 01-02 22:12:10] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:12:10] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 01-02 22:12:12] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:12:12] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 01-02 22:12:16] Scheduler: Fetching data for newly completed trials: [43].
[INFO 01-02 22:12:16] Scheduler: Retrieved COMPLETED trials: [43].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:12:27] Scheduler: Running trials [46]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:12:31] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 01-02 22:12:31] Scheduler: Fetching data for newly completed trials: [44].
[INFO 01-02 22:12:31] Scheduler: Retrieved COMPLETED trials: [44].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:12:40] Scheduler: Running trials [47]...
[INFO 01-02 22:12:40] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:12:40] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 01-02 22:12:41] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:12:41] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 01-02 22:12:43] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:12:43] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 01-02 22:12:45] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:12:45] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 01-02 22:12:49] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:12:49] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 3).
[INFO 01-02 22:12:54] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:12:54] Scheduler: Waiting for completed trials (for 7 sec, currently running trials: 3).
[INFO 01-02 22:13:01] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:13:01] Scheduler: Waiting for completed trials (for 11 sec, currently running trials: 3).
[INFO 01-02 22:13:13] Scheduler: Fetching data for newly completed trials: [45].
[INFO 01-02 22:13:13] Scheduler: Retrieved COMPLETED trials: [45].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:13:13] Scheduler: Done submitting trials, waiting for remaining 2 running trials...
[INFO 01-02 22:13:13] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:13:13] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 2).
[INFO 01-02 22:13:14] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:13:14] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 2).
[INFO 01-02 22:13:15] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:13:15] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 2).
[INFO 01-02 22:13:18] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:13:18] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 2).
[INFO 01-02 22:13:21] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:13:21] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 2).
[INFO 01-02 22:13:26] Scheduler: Fetching data for newly completed trials: [46].
[INFO 01-02 22:13:26] Scheduler: Retrieved COMPLETED trials: [46].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:13:26] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:13:26] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 1).
[INFO 01-02 22:13:27] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:13:27] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 1).
[INFO 01-02 22:13:29] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:13:29] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 1).
[INFO 01-02 22:13:31] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:13:31] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 1).
[INFO 01-02 22:13:34] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:13:34] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 1).
[INFO 01-02 22:13:39] Scheduler: Fetching data for newly completed trials: [47].
[INFO 01-02 22:13:39] Scheduler: Retrieved COMPLETED trials: [47].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.


OptimizationResult()

评估结果¶

我们现在可以使用 helper 检查优化结果 Ax 附带的函数和可视化。

首先，我们生成一个带有结果摘要的 DataFrame 的实验。此 DataFrame 中的每一行对应于一个 trial（即已运行的训练作业），并包含信息在 Trial 的 status 上，参数 configuration 为评估的指标值以及观察到的指标值。这提供了一种健全性检查优化的简单方法。

from ax.service.utils.report_utils import exp_to_df

df = exp_to_df(experiment)
df.head(10)

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[WARNING 01-02 22:13:39] ax.service.utils.report_utils: Column reason missing for all trials. Not appending column.

	trial_index	arm_name	trial_status	generation_method	is_feasible	num_params	val_acc	hidden_size_1	hidden_size_2	learning_rate	时代	辍学	batch_size
0	0	0_0	完成	索博尔	假	16810.0	0.892440	19	66	0.003182	4	0.190970	32
1	1	1_0	完成	索博尔	假	40280.0	0.916779	50	18	0.000614	2	0.425028	128
2	2	2_0	完成	索博尔	假	112720.0	0.963429	127	96	0.001233	3	0.334354	256
3	3	3_0	完成	索博尔	假	30653.0	0.905993	37	35	0.000234	1	0.037933	64
4	4	4_0	完成	索博尔	假	26056.0	0.917627	28	108	0.000339	1	0.062996	32
5	5	5_0	完成	索博尔	真	71312.0	0.951953	87	32	0.006412	3	0.297078	128
6	6	6_0	完成	索博尔	假	50051.0	0.862612	59	55	0.000116	2	0.458433	256
7	7	7_0	完成	索博尔	假	19530.0	0.936613	24	21	0.002236	4	0.161959	64
8	8	8_0	完成	索博尔	假	18080.0	0.905648	20	80	0.000163	3	0.388082	64
9	9	9_0	完成	BoTorch 手电筒	真	46403.0	0.952467	53	77	0.002655	3	0.263433	256

我们还可以可视化验证精度和模型参数的数量。

提示

Ax 使用 Plotly 生成交互式绘图，这允许您执行缩放、裁剪或悬停等作以查看详细信息图的组成部分。试试看，如果你想了解更多，可以看看可视化教程）。

最终的优化结果如下图所示，其中颜色对应于每个试验的迭代编号。我们看到，我们的方法能够成功地探索权衡并找到具有高验证度的两个大型模型精度以及相对较低的小型模型验证准确性。

from ax.service.utils.report_utils import _pareto_frontier_scatter_2d_plotly

_pareto_frontier_scatter_2d_plotly(experiment)

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[WARNING 01-02 22:13:40] ax.service.utils.report_utils: Column reason missing for all trials. Not appending column.

为了更好地了解我们的代理模型所学到的内容黑盒目标，我们可以看看留一法交叉验证结果。由于我们的模型是高斯过程，它们不仅提供点预测，还提供不确定性估计关于这些预测。一个好的模型意味着预测的均值（图中的点）靠近 45 度线，并且置信区间以预期频率覆盖 45 度线（这里我们使用 95% 置信区间，因此我们期望它们包含 95% 的时间是真实的观察结果）。

如下图所示，模型大小（）指标为比验证准确性（）指标更容易建模。num_paramsval_acc

from ax.modelbridge.cross_validation import compute_diagnostics, cross_validate
from ax.plot.diagnostic import interact_cross_validation_plotly
from ax.utils.notebook.plotting import init_notebook_plotting, render

cv = cross_validate(model=gs.model)  # The surrogate model is stored on the ``GenerationStrategy``
compute_diagnostics(cv)

interact_cross_validation_plotly(cv)

我们还可以制作等值线图，以更好地了解目标取决于两个 Importing 参数。在下图中，我们将模型预测的验证精度显示为一个函数两个隐藏的尺寸。验证精度明显提高随着隐藏大小的增加。

from ax.plot.contour import interact_contour_plotly

interact_contour_plotly(model=gs.model, metric_name="val_acc")

同样，我们将模型参数的数量显示为下图中隐藏的大小，可以看到它也增加了作为隐藏大小的函数（对的依赖性要大得多）。hidden_size_1

interact_contour_plotly(model=gs.model, metric_name="num_params")

确认¶

我们感谢 TorchX 团队（特别是 Kiuk Chung 和 Tristan Rice）感谢他们在将 TorchX 与 Ax 集成方面的帮助。

脚本总运行时间：（17 分 54.688 秒）

由 Sphinx-Gallery 生成的图库

带 Ax 的多目标 NAS¶

定义 TorchX 应用程序¶

设置 Runner¶

设置`SearchSpace`¶

设置量度¶

设置`OptimizationConfig`¶

创建 Ax 实验¶

选择生成策略¶

配置调度程序¶

运行优化¶

评估结果¶

确认¶

文档

教程

资源

带 Ax 的多目标 NAS¶

定义 TorchX 应用程序¶

设置 Runner¶

设置SearchSpace¶

设置量度¶

设置OptimizationConfig¶

创建 Ax 实验¶

选择生成策略¶

配置调度程序¶

运行优化¶

评估结果¶

确认¶

文档

教程

资源

设置`SearchSpace`¶

设置`OptimizationConfig`¶