Help request. Most Auto* models fail but AutoRNN, AutoLSTM, and TFT work

[SaintRod]

Hi, all:

I am reaching out for assistance. I can't seem to get most of the Auto* models to work. The only Auto* models that work are AutoRNN and AutoLSTM.

In particular, I'm trying to get the AutoTFT model to work, but have been thus far unable. There are several warnings and errors while running .fit but the main issue is RuntimeError: No best trial found for the given metric: loss. This means that no trial has reported this metric, or all values reported for this metric are NaN. To not ignore NaN values, you can set the 'filter_nan_and_inf' arg to False. I receive the same error message no matter what I try. I tried adding filter_nan_and_inf to config but then in the stdout of running .fit I see TypeError: Trainer.__init__() got an unexpected keyword argument 'filter_nan_and_inf'... so where does filter_nan_and_inf go? I've seen an issue about this but the answer wasn't clear [to me anyway].

Interestingly the non-auto version of TFT works. I copied the TFT specification below from an example in the neuralforecast documentation. I've tried AutoTFT without passing anything to the config argument too, but the result is the same error.

Since AutoRNN, AutoLSTM, and TFT work I assume things were installed properly.

Does anyone have any ideas?

Thank you!

# Fails
# AutoTFT model specification
# Results in an error
autotft = AutoTFT(
    h = forecast_horizon_static,
    alias = "AutoTFT",
    cpus = 14,
    verbose = True,
    num_samples = 20,
    loss = MQLoss(level = [95]),
    config = dict(
        start_padding_enabled=True,
        enable_progress_bar = False,
        input_size = 12,
        filter_nan_and_inf = True,
        scaler_type = 'robust'
    )
)

# collect models
list_of_models = [autotft]

# using NeuralForecast
fcst = NeuralForecast(
    models = list_of_models,
    freq = "MS",
)

# training
fcst.fit(df = df_train_pandas)

# result of .fit
WARNING experiment_analysis.py:558 -- Could not find best trial. Did you pass the correct `metric` parameter?

TypeError: Trainer.__init__() got an unexpected keyword argument 'filter_nan_and_inf'

RuntimeError: No best trial found for the given metric: loss. This means that no trial has reported this metric, or all values reported for this metric are NaN. To not ignore NaN values, you can set the `filter_nan_and_inf` arg to False.

# Succeeds
autornn = AutoRNN(
    h = forecast_horizon_static,
    alias = "AutoRNN",
    cpus = num_cores,
    num_samples=num_samples,
    loss = MQLoss(level = [95])
)

autolstm = AutoLSTM(
    h = forecast_horizon_static,
    alias = "AutoLSTM",
    cpus = num_cores,
    num_samples=num_samples,
    loss = MQLoss(level = [95])
)

tft = TFT(
    h = forecast_horizon_static,
    input_size = 48,
    hidden_size = 20,
    loss = MQLoss(level = [95]),
    learning_rate = 0.005,
    max_steps = num_samples,
    val_check_steps = 10,
    early_stop_patience_steps = -1,
    scaler_type = 'robust',
    windows_batch_size = None,
    enable_progress_bar = False,
    num_workers_loader = num_cores,
    start_padding_enabled = True,
    alias = "TFT"
)

# collect models
list_of_models = [autornn, autolstm, tft]

# using NeuralForecast
fcst = NeuralForecast(
    models = list_of_models,
    freq = "MS",
)

# training
fcst.fit(df = df_train_pandas)

[jmoralez]

Hey @SaintRod. What is the filter_nan_and_inf argument supposed to do? That's not an argument of the neuralforecast models

[SaintRod]

It's a TFT arg I think. Found it online. We can close this discussion as I no longer support this team/work