<div align="center">
# ChatTime: A Multimodal Time Series Foundation Model
[](https://arxiv.org/abs/2412.11376)
[](https://huggingface.co/collections/ChengsenWang/chattime-datasets-6731b504efecc8a6e439741c)
[](https://huggingface.co/collections/ChengsenWang/chattime-models-6731b650cb98bc7842713fde)
</div>
## ✨ Introduction
Human experts typically integrate numerical and textual multimodal information to analyze time series. However, most traditional deep learning predictors rely solely on unimodal numerical data, using a fixed-length window for training and prediction on a single dataset, and cannot adapt to different scenarios. The powered pre-trained large language model has introduced new opportunities for time series analysis. Yet, existing methods are either inefficient in training, incapable of handling textual information, or lack zero-shot forecasting capability. In this paper, we innovatively model time series as a foreign language and construct ChatTime, a unified framework for time series and text processing. As an out-of-the-box multimodal time series foundation model, ChatTime provides zero-shot forecasting capability and supports bimodal input/output for both time series and text. We design a series of experiments to verify the superior performance of ChatTime across multiple tasks and scenarios, and create four multimodal datasets to address data gaps. The experimental results demonstrate the potential and utility of ChatTime. Specifically, in the unimodal zero-shot forecasting task, ChatTime achieves 99.9% of the previous state-of-the-art using only 4% of the pre-training data. In the multimodal tasks of context-guided forecasting and time series question answering, ChatTime outperforms the previous state-of-the-art by 3.7% and 36.6%, respectively.
## 📪 How to install
```bash
pip install chat-time==0.1.0
```
## 📈 Usage
We present three minimal examples showing how to perform the multimodal time series analysis using the ChatTime model. The code and corresponding results are available in the [notebook](./demo.ipynb).
### Zero-Shot Time Series Forecasting
```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from model.model import ChatTime
dataset = "Traffic"
hist_len = 120
pred_len = 24
model_path = "ChengsenWang/ChatTime-1-7B-Chat"
df = pd.read_csv(f"./dataset/{dataset}.csv")
hist_data = np.array(df["Hist"].apply(eval).values.tolist())[:, -hist_len:][0]
pred_data = np.array(df["Pred"].apply(eval).values.tolist())[:, :pred_len][0]
model = ChatTime(hist_len=hist_len, pred_len=pred_len, model_path=model_path)
out = model.predict(hist_data)
hist_x = np.linspace(0, hist_len-1, hist_len)
pred_x = np.linspace(hist_len, hist_len+pred_len-1, pred_len)
plt.figure(figsize=(8, 2), dpi=500)
plt.plot(hist_x, hist_data, color='#000000')
plt.plot(pred_x, pred_data, color='#000000', label='true')
plt.plot(pred_x, out, color='#FF7F0E', label='pred')
plt.axvline(hist_len, color='red')
plt.legend(loc="upper left")
plt.show()
```
### Context-Guided Time Series Forecasting
```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from chat_time.model.model import ChatTime
dataset = "PTF"
hist_len = 120
pred_len = 24
model_path = "ChengsenWang/ChatTime-1-7B-Chat"
df = pd.read_csv(f"./dataset/{dataset}.csv")
hist_data = np.array(df["Hist"].apply(eval).values.tolist())[:, -hist_len:][0]
pred_data = np.array(df["Pred"].apply(eval).values.tolist())[:, :pred_len][0]
context = df["Text"].values[0]
model = ChatTime(hist_len=hist_len, pred_len=pred_len, model_path=model_path)
out_text = model.predict(hist_data, context)
out = model.predict(hist_data)
hist_x = np.linspace(0, hist_len-1, hist_len)
pred_x = np.linspace(hist_len, hist_len+pred_len-1, pred_len)
plt.figure(figsize=(8, 2), dpi=500)
plt.plot(hist_x, hist_data, color='#000000')
plt.plot(pred_x, pred_data, color='#000000', label='true')
plt.plot(pred_x, out_text, color='#FF7F0E', label='pred_text')
plt.plot(pred_x, out, color='#1F77B4', label='pred')
plt.axvline(hist_len, color='red')
plt.legend(loc="upper left")
plt.show()
```
### Time Series Question Answering
```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from chat_time.model.model import ChatTime
dataset = "TSQA"
model_path = "ChengsenWang/ChatTime-1-7B-Chat"
df = pd.read_csv(f"./dataset/{dataset}.csv")
series = np.array(df["Series"].apply(eval).values.tolist())[0]
question = df["Question"].values[0]
answer = df["Answer"].values[0]
model = ChatTime(model_path=model_path)
out = model.analyze(question, series)
plt.figure(figsize=(8, 2), dpi=500)
plt.plot(series, color='#000000')
plt.show()
print(question)
print(f"\n{out} / {answer}\n")
```
### Continuous Pre-Training and Instruction Fine-Tuning
The code for Continuous Pre-Training and Instruction Fine-Tuning for ChatTime is located within the [training](./training/) folder, while the model weights are available at [ChengsenWang/ChatTime-1-7B-Base](https://huggingface.co/ChengsenWang/ChatTime-1-7B-Base) and [ChengsenWang/ChatTime-1-7B-Chat](https://huggingface.co/ChengsenWang/ChatTime-1-7B-Chat), respectively.
## :floppy_disk: Datasets
Refer to following repositories for instructions on downloading and utilizing the datasets.
- The datasets used in the ChatTime paper for Continuous Pre-Training and Instruction Fine-Tuning are available in the HuggingFace repositories: [ChengsenWang/ChatTime-1-Pretrain-1M](https://huggingface.co/datasets/ChengsenWang/ChatTime-1-Pretrain-1M) and [ChengsenWang/ChatTime-1-Finetune-100K](https://huggingface.co/datasets/ChengsenWang/ChatTime-1-Finetune-100K).
- Unimodal zero-shot forecasting datasets can be accessed via the previous [Google Drive](https://drive.google.com/drive/folders/1S7u4exc5NkKRWfdgqYBZ-VqSz9XfrEDV?usp=sharing), while multimodal datasets for context-guided forecasting and time series question-answering tasks are available on HuggingFace at [ChengsenWang/CGTSF](https://huggingface.co/datasets/ChengsenWang/CGTSF) and [ChengsenWang/TSQA](https://huggingface.co/datasets/ChengsenWang/TSQA).
## 📝 Citation
If you find this repo or our work useful for your research, please consider citing the [paper](https://arxiv.org/abs/2412.11376):
```tex
@inproceedings{
author = {Chengsen Wang and Qi Qi and Jingyu Wang and Haifeng Sun and Zirui Zhuang and Jinming Wu and Lei Zhang and Jianxin Liao},
title = {ChatTime: A Unified Multimodal Time Series Foundation Model Bridging Numerical and Textual Data},
booktitle = {AAAI Conference on Artificial Intelligence},
year = {2025},
}
```
## 📪 Contact
If you have any question, please contact [cswang@bupt.edu.cn]().
Raw data
{
"_id": null,
"home_page": null,
"name": "chat-time",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.10",
"maintainer_email": "cswang <cswang@bupt.edu.cn>",
"keywords": "Deep Learning, PyTorch, Time Series Forecasting, Transformer",
"author": "Qi Qi, Jingyu Wang, Haifeng Sun, Zirui Zhuang, Jinming Wu, Lei Zhang, Jianxin Liao",
"author_email": "Chengsen Wang <cswang@bupt.edu.cn>",
"download_url": "https://files.pythonhosted.org/packages/02/fd/1953951f05680ed99db5dd56f354848b6ff0ddee8d0bff79c4453dc9ac65/chat_time-0.2.0.tar.gz",
"platform": null,
"description": "<div align=\"center\">\n\n# ChatTime: A Multimodal Time Series Foundation Model\n\n[](https://arxiv.org/abs/2412.11376)\n[](https://huggingface.co/collections/ChengsenWang/chattime-datasets-6731b504efecc8a6e439741c)\n[](https://huggingface.co/collections/ChengsenWang/chattime-models-6731b650cb98bc7842713fde)\n\n\n\n\n</div>\n\n## \u2728 Introduction\n\nHuman experts typically integrate numerical and textual multimodal information to analyze time series. However, most traditional deep learning predictors rely solely on unimodal numerical data, using a fixed-length window for training and prediction on a single dataset, and cannot adapt to different scenarios. The powered pre-trained large language model has introduced new opportunities for time series analysis. Yet, existing methods are either inefficient in training, incapable of handling textual information, or lack zero-shot forecasting capability. In this paper, we innovatively model time series as a foreign language and construct ChatTime, a unified framework for time series and text processing. As an out-of-the-box multimodal time series foundation model, ChatTime provides zero-shot forecasting capability and supports bimodal input/output for both time series and text. We design a series of experiments to verify the superior performance of ChatTime across multiple tasks and scenarios, and create four multimodal datasets to address data gaps. The experimental results demonstrate the potential and utility of ChatTime. Specifically, in the unimodal zero-shot forecasting task, ChatTime achieves 99.9% of the previous state-of-the-art using only 4% of the pre-training data. In the multimodal tasks of context-guided forecasting and time series question answering, ChatTime outperforms the previous state-of-the-art by 3.7% and 36.6%, respectively.\n\n\n\n## \ud83d\udcea How to install\n\n```bash\npip install chat-time==0.1.0\n```\n## \ud83d\udcc8 Usage\n\nWe present three minimal examples showing how to perform the multimodal time series analysis using the ChatTime model. The code and corresponding results are available in the [notebook](./demo.ipynb).\n\n### Zero-Shot Time Series Forecasting\n\n```python\n\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nfrom model.model import ChatTime\n\ndataset = \"Traffic\"\nhist_len = 120\npred_len = 24\nmodel_path = \"ChengsenWang/ChatTime-1-7B-Chat\"\n\ndf = pd.read_csv(f\"./dataset/{dataset}.csv\")\nhist_data = np.array(df[\"Hist\"].apply(eval).values.tolist())[:, -hist_len:][0]\npred_data = np.array(df[\"Pred\"].apply(eval).values.tolist())[:, :pred_len][0]\n\nmodel = ChatTime(hist_len=hist_len, pred_len=pred_len, model_path=model_path)\n\nout = model.predict(hist_data)\n\nhist_x = np.linspace(0, hist_len-1, hist_len)\npred_x = np.linspace(hist_len, hist_len+pred_len-1, pred_len)\n\nplt.figure(figsize=(8, 2), dpi=500)\nplt.plot(hist_x, hist_data, color='#000000')\nplt.plot(pred_x, pred_data, color='#000000', label='true')\nplt.plot(pred_x, out, color='#FF7F0E', label='pred')\nplt.axvline(hist_len, color='red')\nplt.legend(loc=\"upper left\")\nplt.show()\n\n```\n\n### Context-Guided Time Series Forecasting\n\n```python\n\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nfrom chat_time.model.model import ChatTime\n\ndataset = \"PTF\"\nhist_len = 120\npred_len = 24\nmodel_path = \"ChengsenWang/ChatTime-1-7B-Chat\"\n\ndf = pd.read_csv(f\"./dataset/{dataset}.csv\")\nhist_data = np.array(df[\"Hist\"].apply(eval).values.tolist())[:, -hist_len:][0]\npred_data = np.array(df[\"Pred\"].apply(eval).values.tolist())[:, :pred_len][0]\ncontext = df[\"Text\"].values[0]\n\nmodel = ChatTime(hist_len=hist_len, pred_len=pred_len, model_path=model_path)\n\nout_text = model.predict(hist_data, context)\nout = model.predict(hist_data)\n\nhist_x = np.linspace(0, hist_len-1, hist_len)\npred_x = np.linspace(hist_len, hist_len+pred_len-1, pred_len)\n\nplt.figure(figsize=(8, 2), dpi=500)\nplt.plot(hist_x, hist_data, color='#000000')\nplt.plot(pred_x, pred_data, color='#000000', label='true')\nplt.plot(pred_x, out_text, color='#FF7F0E', label='pred_text')\nplt.plot(pred_x, out, color='#1F77B4', label='pred')\nplt.axvline(hist_len, color='red')\nplt.legend(loc=\"upper left\")\nplt.show()\n\n```\n\n### Time Series Question Answering\n\n```python\n\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nfrom chat_time.model.model import ChatTime\n\ndataset = \"TSQA\"\nmodel_path = \"ChengsenWang/ChatTime-1-7B-Chat\"\n\ndf = pd.read_csv(f\"./dataset/{dataset}.csv\")\nseries = np.array(df[\"Series\"].apply(eval).values.tolist())[0]\nquestion = df[\"Question\"].values[0]\nanswer = df[\"Answer\"].values[0]\n\nmodel = ChatTime(model_path=model_path)\n\nout = model.analyze(question, series)\n\nplt.figure(figsize=(8, 2), dpi=500)\nplt.plot(series, color='#000000')\nplt.show()\n\nprint(question)\nprint(f\"\\n{out} / {answer}\\n\")\n\n```\n\n### Continuous Pre-Training and Instruction Fine-Tuning\n\nThe code for Continuous Pre-Training and Instruction Fine-Tuning for ChatTime is located within the [training](./training/) folder, while the model weights are available at [ChengsenWang/ChatTime-1-7B-Base](https://huggingface.co/ChengsenWang/ChatTime-1-7B-Base) and [ChengsenWang/ChatTime-1-7B-Chat](https://huggingface.co/ChengsenWang/ChatTime-1-7B-Chat), respectively.\n\n## :floppy_disk: Datasets\n\nRefer to following repositories for instructions on downloading and utilizing the datasets.\n\n- The datasets used in the ChatTime paper for Continuous Pre-Training and Instruction Fine-Tuning are available in the HuggingFace repositories: [ChengsenWang/ChatTime-1-Pretrain-1M](https://huggingface.co/datasets/ChengsenWang/ChatTime-1-Pretrain-1M) and [ChengsenWang/ChatTime-1-Finetune-100K](https://huggingface.co/datasets/ChengsenWang/ChatTime-1-Finetune-100K).\n- Unimodal zero-shot forecasting datasets can be accessed via the previous [Google Drive](https://drive.google.com/drive/folders/1S7u4exc5NkKRWfdgqYBZ-VqSz9XfrEDV?usp=sharing), while multimodal datasets for context-guided forecasting and time series question-answering tasks are available on HuggingFace at [ChengsenWang/CGTSF](https://huggingface.co/datasets/ChengsenWang/CGTSF) and [ChengsenWang/TSQA](https://huggingface.co/datasets/ChengsenWang/TSQA).\n\n## \ud83d\udcdd Citation\n\nIf you find this repo or our work useful for your research, please consider citing the [paper](https://arxiv.org/abs/2412.11376):\n\n```tex\n@inproceedings{\n author = {Chengsen Wang and Qi Qi and Jingyu Wang and Haifeng Sun and Zirui Zhuang and Jinming Wu and Lei Zhang and Jianxin Liao},\n title = {ChatTime: A Unified Multimodal Time Series Foundation Model Bridging Numerical and Textual Data},\n booktitle = {AAAI Conference on Artificial Intelligence},\n year = {2025},\n}\n```\n\n## \ud83d\udcea Contact\n\nIf you have any question, please contact [cswang@bupt.edu.cn]().\n",
"bugtrack_url": null,
"license": null,
"summary": "ChatTime: A Multimodal Time Series Foundation Model",
"version": "0.2.0",
"project_urls": null,
"split_keywords": [
"deep learning",
" pytorch",
" time series forecasting",
" transformer"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "940f754f414344d6af32a322208c148c092bcafe26db4e23c67379e51f13aaa8",
"md5": "1e51776c4cca0fb5e10ea0775c95cf2c",
"sha256": "53553fee6bc4086d9b0b24057cd07194c48041434a9f0ad98a868986add88434"
},
"downloads": -1,
"filename": "chat_time-0.2.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "1e51776c4cca0fb5e10ea0775c95cf2c",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.10",
"size": 3473,
"upload_time": "2025-02-16T16:01:52",
"upload_time_iso_8601": "2025-02-16T16:01:52.846082Z",
"url": "https://files.pythonhosted.org/packages/94/0f/754f414344d6af32a322208c148c092bcafe26db4e23c67379e51f13aaa8/chat_time-0.2.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "02fd1953951f05680ed99db5dd56f354848b6ff0ddee8d0bff79c4453dc9ac65",
"md5": "072625316e930fd49d5bc71e020eff63",
"sha256": "51ffe219a5b0d40658edc3e1806a61c52c0e69d4ff2f8893902fb9c10b0eb7f4"
},
"downloads": -1,
"filename": "chat_time-0.2.0.tar.gz",
"has_sig": false,
"md5_digest": "072625316e930fd49d5bc71e020eff63",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.10",
"size": 5986,
"upload_time": "2025-02-16T16:01:54",
"upload_time_iso_8601": "2025-02-16T16:01:54.789716Z",
"url": "https://files.pythonhosted.org/packages/02/fd/1953951f05680ed99db5dd56f354848b6ff0ddee8d0bff79c4453dc9ac65/chat_time-0.2.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-02-16 16:01:54",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "chat-time"
}
JSON