Skip to content

Commit

Permalink
minor changes
Browse files Browse the repository at this point in the history
  • Loading branch information
vijaye12 committed Nov 14, 2024
1 parent da567f0 commit 0e9586b
Showing 1 changed file with 8 additions and 3 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

- [FoundTS-Leaderboard](https://arxiv.org/abs/2410.11802) provides a general benchmarking framework for zero-shot and few-shot forecasting using time-series foundation models.
- Thanks to the authors of the paper for providing this benchmarking system which is very essential in the current literature space.
- However, we observe that the TTM numbers reported as part of the benchmarking is not computed correctly. Hence we are sharing the scripts here to rightly run the TTM models on the benchmarked datasets (ETTH1, ETTH2, ETTM1, ETTM2, Weather, Traffic, Electricity, Zafnoo, Exchange). Please note that, Solar dataset is used as part of the TTM pretraining. Hence TTM's results on Solar is extremely good and should not be considered for Zero-shot evaluation. Likewise, Weather and Electricity is part of TimesFM pretraining data and Traffic is part of both TimesFM and Moirai. Hence, we should exclude these results in zero/few-shot comparisons.
- However, we observe that the TTM numbers reported as part of the benchmarking are not computed correctly. Hence we are sharing the scripts here to rightly run the TTM models on the benchmarked datasets (ETTH1, ETTH2, ETTM1, ETTM2, Weather, Traffic, Electricity, Zafnoo, Exchange). Please note that, Solar dataset is used as part of the TTM pretraining. Hence TTM's results on Solar is extremely good and should not be considered for Zero-shot evaluation. Likewise, Weather and Electricity is part of TimesFM pretraining data and Traffic is part of both TimesFM and Moirai. Hence, we should exclude these results in zero/few-shot comparisons.
- Since TTM requires atleast 512 time-points in the context length, we ignore short-context length datasets such as ILI, NASDAQ, NN5, Wike2000 where TTM cannot be applied currently.
- We request the authors to update their benchmarking/leaderboard table accordingly.
- As per the below zeroshot results, TTM outperform most of the datasets by a good margin.
Expand Down Expand Up @@ -43,7 +43,12 @@ It will run all benchmarking and dump the results. The dumped results are availa
## Different TTM Models:
- TTM-B1: Refers to TTM results with 512 context length and frequency prefix tuning enabled: `ttm-b1-512.sh`
- TTM-B2: Refers to TTM results with 512 context length: `ttm-b1-512.sh`
- TTM-E: Refers to TTM results with 1024 context length: `ttm-b1-512.sh`
- TTM-A: Refers to TTM results with 1536 context length: `ttm-b1-512.sh`
- TTM-E: Refers to TTM results with 1024 context length: `ttm-e-512.sh`
- TTM-A: Refers to TTM results with 1536 context length: `ttm-a-512.sh`

## Few-shot results

We request the FoundTS authors to follow the same approach mentioned in the script for few-shot results also. We couldn't report the few-shot results as TTM's sampling technqiue is different from the sampling technique used in the FoundTS method and we are facing challenges to replicate the exact sampling approach followed in FoundTS.

## Disclaimer:
Please note that, the intent of this file is to report and update the correct TTM numbers in the FoundTS benchmarking paper and leaderboard. MSE results reported in the above tables of other models are directly taken from the FoundTS benchmarking paper and not computed by us.

0 comments on commit 0e9586b

Please sign in to comment.