# Time Series Foundation Model - TEMPO: Prompt-based Generative Pre-trained Transformer for Time Series Forecasting
[](https://arxiv.org/pdf/2310.04948)
[](https://huggingface.co/Melady/TEMPO)
[](https://opensource.org/licenses/Apache-2.0)
</div>
<div align="center"><img src=https://raw.githubusercontent.com/idevede/TEMPO_for_pip/main/tempo/pics/TEMPO_logo.png width=80% /></div>
The official code for [["TEMPO: Prompt-based Generative Pre-trained Transformer for Time Series Forecasting (ICLR 2024)"]](https://arxiv.org/pdf/2310.04948).
TEMPO is one of the very first open source **Time Series Foundation Models** for forecasting task v1.0 version.
<div align="center"><img src=https://raw.githubusercontent.com/idevede/TEMPO_for_pip/main/tempo/pics/TEMPO.png width=80% /></div>
## 🚀 News
- **Oct 2024**: 🚀 We've streamlined our code structure, enabling users to download the pre-trained model and perform zero-shot inference with a single line of code! Check out our [demo](./run_TEMPO_demo.py) for more details. Our model's download count on HuggingFace is now trackable!
- **Jun 2024**: 🚀 We added demos for reproducing zero-shot experiments in [Colab](https://colab.research.google.com/drive/11qGpT7H1JMaTlMlm9WtHFZ3_cJz7p-og?usp=sharing). We also added the demo of building the customer dataset and directly do the inference via our pre-trained foundation model: [Colab](https://colab.research.google.com/drive/1ZpWbK0L6mq1pav2yDqOuORo4rHbv80-A?usp=sharing)
- **May 2024**: 🚀 TEMPO has launched a GUI-based online [demo](https://4171a8a7484b3e9148.gradio.live/), allowing users to directly interact with our foundation model!
- **May 2024**: 🚀 TEMPO published the 80M pretrained foundation model in [HuggingFace](https://huggingface.co/Melady/TEMPO)!
- **May 2024**: 🧪 We added the code for pretraining and inference TEMPO models. You can find a pre-training script demo in [this folder](./scripts/etth2.sh). We also added [a script](./scripts/etth2_test.sh) for the inference demo.
- **Mar 2024**: 📈 Released [TETS dataset](https://drive.google.com/file/d/1Hu2KFj0kp4kIIpjbss2ciLCV_KiBreoJ/view?usp=drive_link) from [S&P 500](https://www.spglobal.com/spdji/en/indices/equity/sp-500/#overview) used in multimodal experiments in TEMPO.
- **Mar 2024**: 🧪 TEMPO published the project [code](https://github.com/DC-research/TEMPO) and the pre-trained checkpoint [online](https://drive.google.com/file/d/11Ho_seP9NGh-lQCyBkvQhAQFy_3XVwKp/view?usp=drive_link)!
- **Jan 2024**: 🚀 TEMPO [paper](https://openreview.net/pdf?id=YH5w12OUuU) get accepted by ICLR!
- **Oct 2023**: 🚀 TEMPO [paper](https://arxiv.org/pdf/2310.04948) released on Arxiv!
## Build the environment
```
conda create -n tempo python=3.8
```
```
conda activate tempo
```
```
pip install -r requirements.txt
```
## Script Demo
A streamlining example showing how to perform forecasting using TEMPO:
```python
# Third-party library imports
import numpy as np
import torch
from numpy.random import choice
# Local imports
from models.TEMPO import TEMPO
model = TEMPO.load_pretrained_model(
device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu'),
repo_id = "Melady/TEMPO",
filename = "TEMPO-80M_v1.pth",
cache_dir = "./checkpoints/TEMPO_checkpoints"
)
input_data = np.random.rand(336) # Random input data
with torch.no_grad():
predicted_values = model.predict(input_data, pred_length=96)
print("Predicted values:")
print(predicted_values)
```
## Demos
### 1. Reproducing zero-shot experiments on ETTh2:
Please try to reproduc the zero-shot experiments on ETTh2 [[here on Colab]](https://colab.research.google.com/drive/11qGpT7H1JMaTlMlm9WtHFZ3_cJz7p-og?usp=sharing).
### 2. Zero-shot experiments on customer dataset:
We use the following Colab page to show the demo of building the customer dataset and directly do the inference via our pre-trained foundation model: [[Colab]](https://colab.research.google.com/drive/1ZpWbK0L6mq1pav2yDqOuORo4rHbv80-A?usp=sharing)
### 3. Online demo:
Please try our foundation model demo [[here]](https://4171a8a7484b3e9148.gradio.live).
<div align="center"><img src=https://raw.githubusercontent.com/idevede/TEMPO_for_pip/main/tempo/pics/TEMPO_demo.jpg width=80% /></div>
## Practice on your end
We also updated our models on HuggingFace: [[Melady/TEMPO]](https://huggingface.co/Melady/TEMPO).
### Get Data
Download the data from [[Google Drive]](https://drive.google.com/drive/folders/13Cg1KYOlzM5C7K8gK8NfC-F3EYxkM3D2?usp=sharing) or [[Baidu Drive]](https://pan.baidu.com/s/1r3KhGd0Q9PJIUZdfEYoymg?pwd=i9iy), and place the downloaded data in the folder`./dataset`. You can also download the STL results from [[Google Drive]](https://drive.google.com/file/d/1gWliIGDDSi2itUAvYaRgACru18j753Kw/view?usp=sharing), and place the downloaded data in the folder`./stl`.
### Run TEMPO
### Pre-Training Stage
```
bash [ecl, etth1, etth2, ettm1, ettm2, traffic, weather].sh
```
### Test/ Inference Stage
After training, we can test TEMPO model under the zero-shot setting:
```
bash [ecl, etth1, etth2, ettm1, ettm2, traffic, weather]_test.sh
```
<div align="center"><img src=https://raw.githubusercontent.com/idevede/TEMPO_for_pip/main/tempo/pics/results.jpg width=90% /></div>
## Pre-trained Models
You can download the pre-trained model from [[Google Drive]](https://drive.google.com/file/d/11Ho_seP9NGh-lQCyBkvQhAQFy_3XVwKp/view?usp=drive_link) and then run the test script for fun.
## TETS dataset
Here is the prompts use to generate the coresponding textual informaton of time series via [[OPENAI ChatGPT-3.5 API]](https://platform.openai.com/docs/guides/text-generation)
<div align="center"><img src=https://raw.githubusercontent.com/idevede/TEMPO_for_pip/main/tempo/pics/TETS_prompt.png width=80% /></div>
The time series data are come from [[S&P 500]](https://www.spglobal.com/spdji/en/indices/equity/sp-500/#overview). Here is the EBITDA case for one company from the dataset:
<div align="center"><img src=https://raw.githubusercontent.com/idevede/TEMPO_for_pip/main/tempo/pics/Company1_ebitda_summary.png width=80% /></div>
Example of generated contextual information for the Company marked above:
<div align="center"><img src=https://raw.githubusercontent.com/idevede/TEMPO_for_pip/main/tempo/pics/Company1_ebitda_summary_words.jpg width=80% /></div>
You can download the processed data with text embedding from GPT2 from: [[TETS]](https://drive.google.com/file/d/1Hu2KFj0kp4kIIpjbss2ciLCV_KiBreoJ/view?usp=drive_link
).
## Contact
Feel free to connect DefuCao@USC.EDU / YanLiu.CS@USC.EDU if you’re interested in applying TEMPO to your real-world application.
## Cite our work
```
@inproceedings{
cao2024tempo,
title={{TEMPO}: Prompt-based Generative Pre-trained Transformer for Time Series Forecasting},
author={Defu Cao and Furong Jia and Sercan O Arik and Tomas Pfister and Yixiang Zheng and Wen Ye and Yan Liu},
booktitle={The Twelfth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=YH5w12OUuU}
}
```
```
@article{
Jia_Wang_Zheng_Cao_Liu_2024,
title={GPT4MTS: Prompt-based Large Language Model for Multimodal Time-series Forecasting},
volume={38},
url={https://ojs.aaai.org/index.php/AAAI/article/view/30383},
DOI={10.1609/aaai.v38i21.30383},
number={21},
journal={Proceedings of the AAAI Conference on Artificial Intelligence},
author={Jia, Furong and Wang, Kevin and Zheng, Yixiang and Cao, Defu and Liu, Yan},
year={2024}, month={Mar.}, pages={23343-23351}
}
```
Raw data
{
"_id": null,
"home_page": "https://github.com/idevede/TEMPO_for_pip",
"name": "tempodev",
"maintainer": null,
"docs_url": null,
"requires_python": "<4,>=3.7.2",
"maintainer_email": null,
"keywords": "time-series, ml, llm",
"author": "Defu Cao",
"author_email": "defucao@usc.edu",
"download_url": "https://files.pythonhosted.org/packages/f3/55/da1ec4a4f12f44d7ca52bd2ec5078c03d22b02ecf338e60e0b19c3e9d69e/tempodev-0.0.9.tar.gz",
"platform": null,
"description": "# Time Series Foundation Model - TEMPO: Prompt-based Generative Pre-trained Transformer for Time Series Forecasting\n\n[](https://arxiv.org/pdf/2310.04948)\n[](https://huggingface.co/Melady/TEMPO)\n[](https://opensource.org/licenses/Apache-2.0)\n</div>\n\n<div align=\"center\"><img src=https://raw.githubusercontent.com/idevede/TEMPO_for_pip/main/tempo/pics/TEMPO_logo.png width=80% /></div>\n\nThe official code for [[\"TEMPO: Prompt-based Generative Pre-trained Transformer for Time Series Forecasting (ICLR 2024)\"]](https://arxiv.org/pdf/2310.04948).\n\nTEMPO is one of the very first open source **Time Series Foundation Models** for forecasting task v1.0 version.\n\n<div align=\"center\"><img src=https://raw.githubusercontent.com/idevede/TEMPO_for_pip/main/tempo/pics/TEMPO.png width=80% /></div>\n\n\n## \ud83d\ude80 News\n\n\n- **Oct 2024**: \ud83d\ude80 We've streamlined our code structure, enabling users to download the pre-trained model and perform zero-shot inference with a single line of code! Check out our [demo](./run_TEMPO_demo.py) for more details. Our model's download count on HuggingFace is now trackable!\n\n- **Jun 2024**: \ud83d\ude80 We added demos for reproducing zero-shot experiments in [Colab](https://colab.research.google.com/drive/11qGpT7H1JMaTlMlm9WtHFZ3_cJz7p-og?usp=sharing). We also added the demo of building the customer dataset and directly do the inference via our pre-trained foundation model: [Colab](https://colab.research.google.com/drive/1ZpWbK0L6mq1pav2yDqOuORo4rHbv80-A?usp=sharing)\n- **May 2024**: \ud83d\ude80 TEMPO has launched a GUI-based online [demo](https://4171a8a7484b3e9148.gradio.live/), allowing users to directly interact with our foundation model!\n- **May 2024**: \ud83d\ude80 TEMPO published the 80M pretrained foundation model in [HuggingFace](https://huggingface.co/Melady/TEMPO)!\n- **May 2024**: \ud83e\uddea We added the code for pretraining and inference TEMPO models. You can find a pre-training script demo in [this folder](./scripts/etth2.sh). We also added [a script](./scripts/etth2_test.sh) for the inference demo.\n\n- **Mar 2024**: \ud83d\udcc8 Released [TETS dataset](https://drive.google.com/file/d/1Hu2KFj0kp4kIIpjbss2ciLCV_KiBreoJ/view?usp=drive_link) from [S&P 500](https://www.spglobal.com/spdji/en/indices/equity/sp-500/#overview) used in multimodal experiments in TEMPO. \n- **Mar 2024**: \ud83e\uddea TEMPO published the project [code](https://github.com/DC-research/TEMPO) and the pre-trained checkpoint [online](https://drive.google.com/file/d/11Ho_seP9NGh-lQCyBkvQhAQFy_3XVwKp/view?usp=drive_link)! \n- **Jan 2024**: \ud83d\ude80 TEMPO [paper](https://openreview.net/pdf?id=YH5w12OUuU) get accepted by ICLR!\n- **Oct 2023**: \ud83d\ude80 TEMPO [paper](https://arxiv.org/pdf/2310.04948) released on Arxiv!\n\n## Build the environment\n\n```\nconda create -n tempo python=3.8\n```\n```\nconda activate tempo\n```\n```\npip install -r requirements.txt\n```\n\n## Script Demo\n\nA streamlining example showing how to perform forecasting using TEMPO:\n\n```python\n# Third-party library imports\nimport numpy as np\nimport torch\nfrom numpy.random import choice\n# Local imports\nfrom models.TEMPO import TEMPO\n\n\nmodel = TEMPO.load_pretrained_model(\n device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu'),\n repo_id = \"Melady/TEMPO\",\n filename = \"TEMPO-80M_v1.pth\",\n cache_dir = \"./checkpoints/TEMPO_checkpoints\" \n)\n\ninput_data = np.random.rand(336) # Random input data\nwith torch.no_grad():\n predicted_values = model.predict(input_data, pred_length=96)\nprint(\"Predicted values:\")\nprint(predicted_values)\n\n```\n\n## Demos\n\n### 1. Reproducing zero-shot experiments on ETTh2:\n\nPlease try to reproduc the zero-shot experiments on ETTh2 [[here on Colab]](https://colab.research.google.com/drive/11qGpT7H1JMaTlMlm9WtHFZ3_cJz7p-og?usp=sharing).\n\n### 2. Zero-shot experiments on customer dataset:\n\nWe use the following Colab page to show the demo of building the customer dataset and directly do the inference via our pre-trained foundation model: [[Colab]](https://colab.research.google.com/drive/1ZpWbK0L6mq1pav2yDqOuORo4rHbv80-A?usp=sharing)\n\n### 3. Online demo:\n\nPlease try our foundation model demo [[here]](https://4171a8a7484b3e9148.gradio.live).\n\n<div align=\"center\"><img src=https://raw.githubusercontent.com/idevede/TEMPO_for_pip/main/tempo/pics/TEMPO_demo.jpg width=80% /></div>\n\n## Practice on your end\n\nWe also updated our models on HuggingFace: [[Melady/TEMPO]](https://huggingface.co/Melady/TEMPO).\n\n\n\n### Get Data\n\n Download the data from [[Google Drive]](https://drive.google.com/drive/folders/13Cg1KYOlzM5C7K8gK8NfC-F3EYxkM3D2?usp=sharing) or [[Baidu Drive]](https://pan.baidu.com/s/1r3KhGd0Q9PJIUZdfEYoymg?pwd=i9iy), and place the downloaded data in the folder`./dataset`. You can also download the STL results from [[Google Drive]](https://drive.google.com/file/d/1gWliIGDDSi2itUAvYaRgACru18j753Kw/view?usp=sharing), and place the downloaded data in the folder`./stl`.\n\n### Run TEMPO\n\n### Pre-Training Stage\n```\nbash [ecl, etth1, etth2, ettm1, ettm2, traffic, weather].sh\n```\n\n### Test/ Inference Stage\n\nAfter training, we can test TEMPO model under the zero-shot setting:\n\n```\nbash [ecl, etth1, etth2, ettm1, ettm2, traffic, weather]_test.sh\n```\n\n<div align=\"center\"><img src=https://raw.githubusercontent.com/idevede/TEMPO_for_pip/main/tempo/pics/results.jpg width=90% /></div>\n\n\n## Pre-trained Models\n\nYou can download the pre-trained model from [[Google Drive]](https://drive.google.com/file/d/11Ho_seP9NGh-lQCyBkvQhAQFy_3XVwKp/view?usp=drive_link) and then run the test script for fun.\n\n## TETS dataset\n\nHere is the prompts use to generate the coresponding textual informaton of time series via [[OPENAI ChatGPT-3.5 API]](https://platform.openai.com/docs/guides/text-generation)\n\n<div align=\"center\"><img src=https://raw.githubusercontent.com/idevede/TEMPO_for_pip/main/tempo/pics/TETS_prompt.png width=80% /></div>\n\nThe time series data are come from [[S&P 500]](https://www.spglobal.com/spdji/en/indices/equity/sp-500/#overview). Here is the EBITDA case for one company from the dataset:\n\n\n<div align=\"center\"><img src=https://raw.githubusercontent.com/idevede/TEMPO_for_pip/main/tempo/pics/Company1_ebitda_summary.png width=80% /></div>\n\nExample of generated contextual information for the Company marked above:\n\n<div align=\"center\"><img src=https://raw.githubusercontent.com/idevede/TEMPO_for_pip/main/tempo/pics/Company1_ebitda_summary_words.jpg width=80% /></div>\n\n\n\n\nYou can download the processed data with text embedding from GPT2 from: [[TETS]](https://drive.google.com/file/d/1Hu2KFj0kp4kIIpjbss2ciLCV_KiBreoJ/view?usp=drive_link\n).\n\n## Contact\nFeel free to connect DefuCao@USC.EDU / YanLiu.CS@USC.EDU if you\u2019re interested in applying TEMPO to your real-world application.\n\n## Cite our work\n```\n@inproceedings{\ncao2024tempo,\ntitle={{TEMPO}: Prompt-based Generative Pre-trained Transformer for Time Series Forecasting},\nauthor={Defu Cao and Furong Jia and Sercan O Arik and Tomas Pfister and Yixiang Zheng and Wen Ye and Yan Liu},\nbooktitle={The Twelfth International Conference on Learning Representations},\nyear={2024},\nurl={https://openreview.net/forum?id=YH5w12OUuU}\n}\n```\n\n```\n@article{\n Jia_Wang_Zheng_Cao_Liu_2024, \n title={GPT4MTS: Prompt-based Large Language Model for Multimodal Time-series Forecasting}, \n volume={38}, \n url={https://ojs.aaai.org/index.php/AAAI/article/view/30383}, \n DOI={10.1609/aaai.v38i21.30383}, \n number={21}, \n journal={Proceedings of the AAAI Conference on Artificial Intelligence}, \n author={Jia, Furong and Wang, Kevin and Zheng, Yixiang and Cao, Defu and Liu, Yan}, \n year={2024}, month={Mar.}, pages={23343-23351} \n }\n```\n\n\n",
"bugtrack_url": null,
"license": null,
"summary": "Time Series Foundation Model - TEMPO: Prompt-based Generative Pre-trained Transformer for Time Series Forecasting",
"version": "0.0.9",
"project_urls": {
"Bug Reports": "https://github.com/idevede/TEMPO_for_pip/issues",
"Homepage": "https://github.com/idevede/TEMPO_for_pip",
"Source": "https://github.com/idevede/TEMPO_for_pip"
},
"split_keywords": [
"time-series",
" ml",
" llm"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "e48336e19fa0cea729a76b1330a6715c02f7cef84dd5baf3edee63d07057b908",
"md5": "87cf129bbcf4820bfb0841111a56b0d3",
"sha256": "2e07fb8960024bc10f437a5cb3b06d472fe56975ff6150617ffb74f0f1f140fd"
},
"downloads": -1,
"filename": "tempodev-0.0.9-py3-none-any.whl",
"has_sig": false,
"md5_digest": "87cf129bbcf4820bfb0841111a56b0d3",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4,>=3.7.2",
"size": 116321,
"upload_time": "2024-11-24T13:29:56",
"upload_time_iso_8601": "2024-11-24T13:29:56.264088Z",
"url": "https://files.pythonhosted.org/packages/e4/83/36e19fa0cea729a76b1330a6715c02f7cef84dd5baf3edee63d07057b908/tempodev-0.0.9-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "f355da1ec4a4f12f44d7ca52bd2ec5078c03d22b02ecf338e60e0b19c3e9d69e",
"md5": "616cb30069dcfcbb24e53b6a20715f29",
"sha256": "6b0618ba23917c0308494982ed258a541f9f539c8c0f93d7c9d5472439f6cc6d"
},
"downloads": -1,
"filename": "tempodev-0.0.9.tar.gz",
"has_sig": false,
"md5_digest": "616cb30069dcfcbb24e53b6a20715f29",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4,>=3.7.2",
"size": 92577,
"upload_time": "2024-11-24T13:29:58",
"upload_time_iso_8601": "2024-11-24T13:29:58.163042Z",
"url": "https://files.pythonhosted.org/packages/f3/55/da1ec4a4f12f44d7ca52bd2ec5078c03d22b02ecf338e60e0b19c3e9d69e/tempodev-0.0.9.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-11-24 13:29:58",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "idevede",
"github_project": "TEMPO_for_pip",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [
{
"name": "reformer_pytorch",
"specs": []
},
{
"name": "patool",
"specs": []
},
{
"name": "transformers",
"specs": []
},
{
"name": "pandas",
"specs": []
},
{
"name": "scikit-learn",
"specs": []
},
{
"name": "matplotlib",
"specs": []
},
{
"name": "statsmodels",
"specs": []
},
{
"name": "numpy",
"specs": []
},
{
"name": "torch",
"specs": [
[
"==",
"1.13.0"
]
]
},
{
"name": "torchvision",
"specs": [
[
"==",
"0.14.0"
]
]
},
{
"name": "torchaudio",
"specs": [
[
"==",
"0.13.0"
]
]
},
{
"name": "peft",
"specs": []
},
{
"name": "huggingface_hub",
"specs": []
},
{
"name": "omegaconf",
"specs": []
}
],
"lcname": "tempodev"
}
JSON
