Skip to content

Commit

Permalink
minor wording updates
Browse files Browse the repository at this point in the history
  • Loading branch information
wgifford committed Nov 14, 2024
1 parent 0e9586b commit f872d1c
Showing 1 changed file with 12 additions and 15 deletions.
Original file line number Diff line number Diff line change
@@ -1,17 +1,14 @@
# NOTE TO FOUNDTS LEADERBOARD TEAM


# Note to FoundTS leaderboard team

- [FoundTS-Leaderboard](https://arxiv.org/abs/2410.11802) provides a general benchmarking framework for zero-shot and few-shot forecasting using time-series foundation models.
- Thanks to the authors of the paper for providing this benchmarking system which is very essential in the current literature space.
- However, we observe that the TTM numbers reported as part of the benchmarking are not computed correctly. Hence we are sharing the scripts here to rightly run the TTM models on the benchmarked datasets (ETTH1, ETTH2, ETTM1, ETTM2, Weather, Traffic, Electricity, Zafnoo, Exchange). Please note that, Solar dataset is used as part of the TTM pretraining. Hence TTM's results on Solar is extremely good and should not be considered for Zero-shot evaluation. Likewise, Weather and Electricity is part of TimesFM pretraining data and Traffic is part of both TimesFM and Moirai. Hence, we should exclude these results in zero/few-shot comparisons.
- Since TTM requires atleast 512 time-points in the context length, we ignore short-context length datasets such as ILI, NASDAQ, NN5, Wike2000 where TTM cannot be applied currently.
- We thank the authors of the paper for providing this benchmarking system which is very essential in the current literature space.
- However, we observe that the TTM numbers reported as part of the benchmarking are not computed correctly. Hence, we are sharing the scripts here to correctly run the TTM models on the benchmarked datasets (ETTH1, ETTH2, ETTM1, ETTM2, Weather, Traffic, Electricity, Zafnoo, Exchange). Please note that, Solar dataset is used as part of the TTM pretraining. Hence TTM's results on Solar is extremely good and should not be considered for Zero-shot evaluation. Likewise, Weather and Electricity is part of TimesFM pretraining data and Traffic is part of both TimesFM and Moirai. Hence, we should exclude these results in zero/few-shot comparisons.
- Since TTM requires at least 512 time-points in the context length, we ignore short-context length datasets such as ILI, NASDAQ, NN5, Wike2000 where TTM cannot be applied currently.
- We request the authors to update their benchmarking/leaderboard table accordingly.
- As per the below zeroshot results, TTM outperform most of the datasets by a good margin.
- As per the below zeroshot results, TTM outperforms other approaches on most of the datasets by a good margin.



# Updated Zeroshot FoundTS Table with correct TTM numbers
# Updated zeroshot FoundTS table with correct TTM numbers

![fig1](foundts_zeroshot_corrected_view.jpg)
![fig2](foundts_zeroshot_corrected_view_avg.jpg)
Expand All @@ -20,7 +17,7 @@
# Steps to reproduce

## Fetching the data
The evaluation data can be downloaded from any of the previous time-series github repos like autoformer or timesnet or informer or foundTS repo. [Sample download link](https://drive.google.com/drive/folders/1vE0ONyqPlym2JaaAoEe0XNDR8FS_d322). The ETT datasets can also be downloaded from [ETT-Github-Repository](https://github.com/zhouhaoyi/ETDataset).
The evaluation data can be downloaded from any of the previous time-series github repos like Autoformer, TimesNet, Informer, or FoundTS repo. [Sample download link](https://drive.google.com/drive/folders/1vE0ONyqPlym2JaaAoEe0XNDR8FS_d322). The ETT datasets can also be downloaded from [ETT-Github-Repository](https://github.com/zhouhaoyi/ETDataset).

Download and save the datasets in a directory. For example, in `data_root_path`.

Expand All @@ -30,15 +27,15 @@ CSVs of each data should reside in location `data_root_path/$dataset_name/$datas

## Running the scripts

1. In terminal, run `sh ttm-a-1536.sh`
1. In a terminal, run `sh ttm-a-1536.sh`
2. Run `summarize_results.py`. For example,

```
```bash
sh ttm-a-1536.sh data_root_path/
python summarize_results.py -rd=results-ttm-a/
```

It will run all benchmarking and dump the results. The dumped results are available in the CSV files.
The above commands will run all benchmarking and dump the results. The dumped results are available as CSV files.

## Different TTM Models:
- TTM-B1: Refers to TTM results with 512 context length and frequency prefix tuning enabled: `ttm-b1-512.sh`
Expand All @@ -48,7 +45,7 @@ It will run all benchmarking and dump the results. The dumped results are availa

## Few-shot results

We request the FoundTS authors to follow the same approach mentioned in the script for few-shot results also. We couldn't report the few-shot results as TTM's sampling technqiue is different from the sampling technique used in the FoundTS method and we are facing challenges to replicate the exact sampling approach followed in FoundTS.
We also request that the FoundTS authors follow the same approach mentioned in the script for few-shot results. We couldn't report the few-shot results as TTM's sampling technique is different from the sampling technique used in the FoundTS method and we are facing challenges to replicate the exact sampling approach followed in FoundTS.

## Disclaimer:
Please note that, the intent of this file is to report and update the correct TTM numbers in the FoundTS benchmarking paper and leaderboard. MSE results reported in the above tables of other models are directly taken from the FoundTS benchmarking paper and not computed by us.
Please note that the intent of this file is to report and provide updates to the TTM numbers in the FoundTS benchmarking paper and leaderboard. MSE results reported in the above tables of other models are directly taken from the FoundTS benchmarking paper and not computed by us.

0 comments on commit f872d1c

Please sign in to comment.