# PFeed: Data Pipeline for Algo-Trading, Getting and Storing Real-Time and Historical Data Made Easy.
[](https://x.com/pfund_ai)
[](https://pypi.org/project/pfeed)
<!-- [](https://jupyterbook.org) -->
[](https://python-poetry.org/)
[MinIO]: https://min.io/
[PFund]: https://github.com/PFund-Software-Ltd/pfund
[Polars]: https://github.com/pola-rs/polars
[Dask]: https://www.dask.org/
[Spark]: https://spark.apache.org/docs/latest/api/python/index.html
[PyTrade.org]: https://pytrade.org
[Yahoo Finance]: https://github.com/ranaroussi/yfinance
[Bybit]: https://public.bybit.com
[Binance]: https://data.binance.vision
[OKX]: https://www.okx.com/data-download
[Databento]: https://databento.com/
[Polygon]: https://polygon.io/
[FirstRate Data]: https://firstratedata.com
## Problem
Starting algo-trading requires reliable, clean data. However, the time-consuming and mundane tasks of data cleaning and storage often discourage traders from embarking on their algo-trading journey.
## Solution
By leveraging modern data engineering tools, `pfeed` handles the tedious data work and **outputs backtesting-ready data**, allowing traders to focus on strategy development.
---
PFeed (/piː fiːd/) is a data pipeline for algorithmic trading, serving as a bridge between raw data sources and traders. It enables you to **download historical data**, **stream real-time data**, and **store cleaned data** in a **local data lake for quantitative analysis**, by automating the processes of data collection, cleaning, transformation, and storage.
## Core Features
- [x] Download or stream reliable, validated and **clean data** for research, backtesting, or live trading
- [x] Get historical data (**dataframe**) or live data in standardized formats by just calling a **single** function
- [x] **Own your data** by storing them locally using [MinIO], with the option to connect to the cloud
- [x] Interact with different kinds of data (including TradFi, CeFi and DeFi) using a **unified interface**
---
<details>
<summary>Table of Contents</summary>
- [Installation](#installation)
- [Quick Start](#quick-start)
- [Get Historical Data in Dataframe](#1-get-historical-data-in-dataframe-no-storage)
- [Download Historical Data on Command Line](#2-download-historical-data-on-the-command-line-interface-cli)
- [Download Historical Data in Python](#3-download-historical-data-in-python)
- [Supported Data Sources](#supported-data-sources)
- [Related Projects](#related-projects)
- [Disclaimer](#disclaimer)
</details>
## Installation
> For more installation options, please refer to the [documentation](https://pfeed-docs.pfund.ai/installation).
```bash
# [RECOMMENDED]: Full Features, choose this if you do not care about the package size
pip install -U "pfeed[all]"
# Minimal Features, only supports getting, downloading and streaming data
pip install -U "pfeed[core]"
```
## Quick Start
### 1. Get Historical Data in Dataframe (No storage)
Get [Bybit]'s data in dataframe, e.g. 1-minute data (data is downloaded on the fly if not stored locally)
```python
import pfeed as pe
feed = pe.BybitFeed(data_tool='polars')
df = feed.get_historical_data(
'BTC_USDT_PERP',
resolution='1minute', # 'raw' or '1tick'/'1t' or '2second'/'2s' etc.
start_date='2024-03-01',
end_date='2024-03-01',
)
```
Printing the first few rows of `df`:
| | ts | product | resolution | open | high | low | close | volume |
|---:|:--------------------|:--------------|:-------------|--------:|--------:|--------:|--------:|---------:|
| 0 | 2024-03-01 00:00:00 | BTC_USDT_PERP | 1m | 61184.1 | 61244.5 | 61175.8 | 61244.5 | 159.142 |
| 1 | 2024-03-01 00:01:00 | BTC_USDT_PERP | 1m | 61245.3 | 61276.5 | 61200.7 | 61232.2 | 227.242 |
| 2 | 2024-03-01 00:02:00 | BTC_USDT_PERP | 1m | 61232.2 | 61249 | 61180 | 61184.2 | 91.446 |
> By using pfeed, you are just a few lines of code away from getting a standardized dataframe, how convenient!
### 2. Download Historical Data on the Command Line Interface (CLI)
> For more CLI commands, please refer to the [documentation](https://pfeed-docs.pfund.ai/cli-commands).
```bash
# download data, default data type (dtype) is 'raw' data
pfeed download -d BYBIT -p BTC_USDT_PERP --start-date 2024-03-01 --end-date 2024-03-08
# download multiple products BTC_USDT_PERP and ETH_USDT_PERP as minute data and store them locally
pfeed download -d BYBIT -p BTC_USDT_PERP -p ETH_USDT_PERP --dtypes minute --use-minio
```
### 3. Download Historical Data in Python
```python
import pfeed as pe
# compared to the CLI approach, this approach is more convenient for downloading multiple products
pe.download(
data_source='bybit',
pdts=[
'BTC_USDT_PERP',
'ETH_USDT_PERP',
'BCH_USDT_PERP',
],
dtypes=['raw'], # data types, e.g. 'raw', 'tick', 'second', 'minute' etc.
start_date='2024-03-01',
end_date='2024-03-08',
use_minio=False,
)
```
## Supported Data Sources
| Data Source | Get Historical Data | Download Historical Data | Get Live Data | Stream Live Data |
| -------------------- | ------------------- | ------------------------ | --------------| ---------------- |
| [Yahoo Finance] | 🟢 | ⚪ | ⚪ | ⚪ |
| [Bybit] | 🟢 | 🟢 | 🟡 | 🔴 |
| *Interactive Brokers | 🔴 | ⚪ | 🔴 | 🔴 |
| *[FirstRate Data] | 🔴 | 🔴 | ⚪ | ⚪ |
| *[Databento] | 🔴 | 🔴 | 🔴 | 🔴 |
| *[Polygon] | 🔴 | 🔴 | 🔴 | 🔴 |
| [Binance] | 🔴 | 🔴 | 🔴 | 🔴 |
| [OKX] | 🔴 | 🔴 | 🔴 | 🔴 |
🟢 = finished \
🟡 = in progress \
🔴 = todo \
⚪ = not applicable \
\* = paid data
## Related Projects
- [PFund] — A Complete Algo-Trading Framework for Machine Learning, TradFi, CeFi and DeFi ready. Supports Vectorized and Event-Driven Backtesting, Paper and Live Trading
- [PyTrade.org] - A curated list of Python libraries and resources for algorithmic trading.
## Disclaimer
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES, OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF, OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
This framework is intended for educational and research purposes only. It should not be used for real trading without understanding the risks involved. Trading in financial markets involves significant risk, and there is always the potential for loss. Your trading results may vary. No representation is being made that any account will or is likely to achieve profits or losses similar to those discussed on this platform.
The developers of this framework are not responsible for any financial losses incurred from using this software. This includes but not limited to losses resulting from inaccuracies in any financial data output by PFeed. Users should conduct their due diligence, verify the accuracy of any data produced by PFeed, and consult with a professional financial advisor before engaging in real trading activities.
Raw data
{
"_id": null,
"home_page": "https://pfund.ai",
"name": "pfeed",
"maintainer": null,
"docs_url": null,
"requires_python": "<4.0,>=3.10",
"maintainer_email": null,
"keywords": "trading, algo-trading, data pipeline, ETL, data lake, data warehouse, data integration, historical data, live data, data streaming",
"author": "Stephen Yau",
"author_email": "softwareentrepreneer+pfeed@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/42/5e/b281f29dd8028f1130ad627c6742bf205334b8e877474d25dfbb386e3536/pfeed-0.0.2.tar.gz",
"platform": null,
"description": "# PFeed: Data Pipeline for Algo-Trading, Getting and Storing Real-Time and Historical Data Made Easy.\n\n[](https://x.com/pfund_ai)\n\n\n[](https://pypi.org/project/pfeed)\n\n<!-- [](https://jupyterbook.org) -->\n[](https://python-poetry.org/)\n\n[MinIO]: https://min.io/\n[PFund]: https://github.com/PFund-Software-Ltd/pfund\n[Polars]: https://github.com/pola-rs/polars\n[Dask]: https://www.dask.org/\n[Spark]: https://spark.apache.org/docs/latest/api/python/index.html\n[PyTrade.org]: https://pytrade.org\n[Yahoo Finance]: https://github.com/ranaroussi/yfinance\n[Bybit]: https://public.bybit.com\n[Binance]: https://data.binance.vision\n[OKX]: https://www.okx.com/data-download\n[Databento]: https://databento.com/\n[Polygon]: https://polygon.io/\n[FirstRate Data]: https://firstratedata.com\n\n## Problem\nStarting algo-trading requires reliable, clean data. However, the time-consuming and mundane tasks of data cleaning and storage often discourage traders from embarking on their algo-trading journey.\n\n## Solution\nBy leveraging modern data engineering tools, `pfeed` handles the tedious data work and **outputs backtesting-ready data**, allowing traders to focus on strategy development.\n\n---\nPFeed (/pi\u02d0 fi\u02d0d/) is a data pipeline for algorithmic trading, serving as a bridge between raw data sources and traders. It enables you to **download historical data**, **stream real-time data**, and **store cleaned data** in a **local data lake for quantitative analysis**, by automating the processes of data collection, cleaning, transformation, and storage.\n\n## Core Features\n- [x] Download or stream reliable, validated and **clean data** for research, backtesting, or live trading\n- [x] Get historical data (**dataframe**) or live data in standardized formats by just calling a **single** function\n- [x] **Own your data** by storing them locally using [MinIO], with the option to connect to the cloud\n- [x] Interact with different kinds of data (including TradFi, CeFi and DeFi) using a **unified interface**\n\n---\n\n<details>\n<summary>Table of Contents</summary>\n\n- [Installation](#installation)\n- [Quick Start](#quick-start)\n - [Get Historical Data in Dataframe](#1-get-historical-data-in-dataframe-no-storage)\n - [Download Historical Data on Command Line](#2-download-historical-data-on-the-command-line-interface-cli)\n - [Download Historical Data in Python](#3-download-historical-data-in-python)\n- [Supported Data Sources](#supported-data-sources)\n- [Related Projects](#related-projects)\n- [Disclaimer](#disclaimer)\n\n</details>\n\n\n\n## Installation\n> For more installation options, please refer to the [documentation](https://pfeed-docs.pfund.ai/installation).\n```bash\n# [RECOMMENDED]: Full Features, choose this if you do not care about the package size\npip install -U \"pfeed[all]\"\n\n# Minimal Features, only supports getting, downloading and streaming data\npip install -U \"pfeed[core]\"\n```\n\n\n\n## Quick Start\n### 1. Get Historical Data in Dataframe (No storage)\nGet [Bybit]'s data in dataframe, e.g. 1-minute data (data is downloaded on the fly if not stored locally)\n\n```python\nimport pfeed as pe\n\nfeed = pe.BybitFeed(data_tool='polars')\n\ndf = feed.get_historical_data(\n 'BTC_USDT_PERP',\n resolution='1minute', # 'raw' or '1tick'/'1t' or '2second'/'2s' etc.\n start_date='2024-03-01',\n end_date='2024-03-01',\n)\n```\n\nPrinting the first few rows of `df`:\n| | ts | product | resolution | open | high | low | close | volume |\n|---:|:--------------------|:--------------|:-------------|--------:|--------:|--------:|--------:|---------:|\n| 0 | 2024-03-01 00:00:00 | BTC_USDT_PERP | 1m | 61184.1 | 61244.5 | 61175.8 | 61244.5 | 159.142 |\n| 1 | 2024-03-01 00:01:00 | BTC_USDT_PERP | 1m | 61245.3 | 61276.5 | 61200.7 | 61232.2 | 227.242 |\n| 2 | 2024-03-01 00:02:00 | BTC_USDT_PERP | 1m | 61232.2 | 61249 | 61180 | 61184.2 | 91.446 |\n\n> By using pfeed, you are just a few lines of code away from getting a standardized dataframe, how convenient!\n\n### 2. Download Historical Data on the Command Line Interface (CLI)\n> For more CLI commands, please refer to the [documentation](https://pfeed-docs.pfund.ai/cli-commands).\n```bash\n# download data, default data type (dtype) is 'raw' data\npfeed download -d BYBIT -p BTC_USDT_PERP --start-date 2024-03-01 --end-date 2024-03-08\n\n# download multiple products BTC_USDT_PERP and ETH_USDT_PERP as minute data and store them locally\npfeed download -d BYBIT -p BTC_USDT_PERP -p ETH_USDT_PERP --dtypes minute --use-minio\n```\n\n### 3. Download Historical Data in Python\n```python\nimport pfeed as pe\n\n# compared to the CLI approach, this approach is more convenient for downloading multiple products\npe.download(\n data_source='bybit',\n pdts=[\n 'BTC_USDT_PERP',\n 'ETH_USDT_PERP',\n 'BCH_USDT_PERP',\n ],\n dtypes=['raw'], # data types, e.g. 'raw', 'tick', 'second', 'minute' etc.\n start_date='2024-03-01',\n end_date='2024-03-08',\n use_minio=False,\n)\n```\n\n\n\n## Supported Data Sources\n| Data Source | Get Historical Data | Download Historical Data | Get Live Data | Stream Live Data |\n| -------------------- | ------------------- | ------------------------ | --------------| ---------------- |\n| [Yahoo Finance] | \ud83d\udfe2 | \u26aa | \u26aa | \u26aa |\n| [Bybit] | \ud83d\udfe2 | \ud83d\udfe2 | \ud83d\udfe1 | \ud83d\udd34 |\n| *Interactive Brokers | \ud83d\udd34 | \u26aa | \ud83d\udd34 | \ud83d\udd34 |\n| *[FirstRate Data] | \ud83d\udd34 | \ud83d\udd34 | \u26aa | \u26aa |\n| *[Databento] | \ud83d\udd34 | \ud83d\udd34 | \ud83d\udd34 | \ud83d\udd34 |\n| *[Polygon] | \ud83d\udd34 | \ud83d\udd34 | \ud83d\udd34 | \ud83d\udd34 |\n| [Binance] | \ud83d\udd34 | \ud83d\udd34 | \ud83d\udd34 | \ud83d\udd34 |\n| [OKX] | \ud83d\udd34 | \ud83d\udd34 | \ud83d\udd34 | \ud83d\udd34 |\n\n\ud83d\udfe2 = finished \\\n\ud83d\udfe1 = in progress \\\n\ud83d\udd34 = todo \\\n\u26aa = not applicable \\\n\\* = paid data\n\n\n\n## Related Projects\n- [PFund] \u2014 A Complete Algo-Trading Framework for Machine Learning, TradFi, CeFi and DeFi ready. Supports Vectorized and Event-Driven Backtesting, Paper and Live Trading\n- [PyTrade.org] - A curated list of Python libraries and resources for algorithmic trading.\n\n\n\n## Disclaimer\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES, OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF, OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.\n\nThis framework is intended for educational and research purposes only. It should not be used for real trading without understanding the risks involved. Trading in financial markets involves significant risk, and there is always the potential for loss. Your trading results may vary. No representation is being made that any account will or is likely to achieve profits or losses similar to those discussed on this platform.\n\nThe developers of this framework are not responsible for any financial losses incurred from using this software. This includes but not limited to losses resulting from inaccuracies in any financial data output by PFeed. Users should conduct their due diligence, verify the accuracy of any data produced by PFeed, and consult with a professional financial advisor before engaging in real trading activities.\n",
"bugtrack_url": null,
"license": "Apache-2.0",
"summary": "Data pipeline for algo-trading, getting and storing both real-time and historical data made easy.",
"version": "0.0.2",
"project_urls": {
"Documentation": "https://pfeed-docs.pfund.ai",
"Homepage": "https://pfund.ai",
"Repository": "https://github.com/PFund-Software-Ltd/pfeed"
},
"split_keywords": [
"trading",
" algo-trading",
" data pipeline",
" etl",
" data lake",
" data warehouse",
" data integration",
" historical data",
" live data",
" data streaming"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "43ed81c9d64f843e3aefbca0e7b94f831c38a9d7b4a943d51a7021d73829b8dc",
"md5": "434f4f0e2881e9cfc772b168b7c55b03",
"sha256": "a41a5f70b107e099e4f358415ebc02c2d5a7a4a5a9774dfbf013b78f7b82a5be"
},
"downloads": -1,
"filename": "pfeed-0.0.2-py3-none-any.whl",
"has_sig": false,
"md5_digest": "434f4f0e2881e9cfc772b168b7c55b03",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4.0,>=3.10",
"size": 54983,
"upload_time": "2024-10-12T05:59:46",
"upload_time_iso_8601": "2024-10-12T05:59:46.528104Z",
"url": "https://files.pythonhosted.org/packages/43/ed/81c9d64f843e3aefbca0e7b94f831c38a9d7b4a943d51a7021d73829b8dc/pfeed-0.0.2-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "425eb281f29dd8028f1130ad627c6742bf205334b8e877474d25dfbb386e3536",
"md5": "2b7509a29d52cdc7bcabdc0d6a82b526",
"sha256": "6f9e741320492676202fd05fc4e401688cef44b16f818532e8b0cdfadeb4eb51"
},
"downloads": -1,
"filename": "pfeed-0.0.2.tar.gz",
"has_sig": false,
"md5_digest": "2b7509a29d52cdc7bcabdc0d6a82b526",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4.0,>=3.10",
"size": 134271,
"upload_time": "2024-10-12T05:59:48",
"upload_time_iso_8601": "2024-10-12T05:59:48.587475Z",
"url": "https://files.pythonhosted.org/packages/42/5e/b281f29dd8028f1130ad627c6742bf205334b8e877474d25dfbb386e3536/pfeed-0.0.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-10-12 05:59:48",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "PFund-Software-Ltd",
"github_project": "pfeed",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "pfeed"
}
JSON
