# bc-utils
[Barchart.com](https://www.barchart.com) allows registered users to download historic futures contract prices in CSV format. Individual contracts must be downloaded separately, which is laborious and slow. This script automates the process.
## Quickstart
```
import os
import logging
from bcutils.bc_utils import get_barchart_downloads, create_bc_session
logging.basicConfig(level=logging.INFO)
CONTRACTS = {
"AUD" : {"code": "A6", "cycle": "HMUZ", "exchange": "CME"},
"GOLD" : {"code": "GC", "cycle": "GJMQVZ", "exchange": "COMEX"},
}
session = create_bc_session(
config_obj=dict(
barchart_username="user@domain.com",
barchart_password="s3cr3t_321",
)
)
get_barchart_downloads(
session,
contract_map=CONTRACTS,
save_dir=os.getcwd(),
start_year=2020,
end_year=2021
)
```
The code above would:
* for the CME Australian Dollar future, get OHLCV price data for the Mar, Jun, Sep and Dec 2020 contracts
* download in CSV format
* save with filenames like Hour_AUD_20200300.csv, Day_AUD_20200600.csv into the specified directory
* for COMEX Gold, get Feb, Apr, Jun, Aug, Oct, and Dec data, with filenames like Hour_GOLD_20200200.csv etc
Features:
* Designed to be run once a day by a scheduler
* the script skips contracts already downloaded
* by default gets 120 days of data per contract, override possible per instrument
* dry run mode to check setup
* allows updates to previously downloaded files
* you must be a registered user. Paid subscribers get 250 downloads a day, otherwise 5
## For pysystemtrade users
This project was originally created to make it easier to populate [pysystemtrade](https://github.com/robcarver17/pysystemtrade) (PST) with futures prices from Barchart, so setup is straightforward. Steps:
1. Clone the bc-utils repo, or your own fork. The remaining steps assume the location `~/bc-utils`
2. Have a look at `~/bc-utils/bcutils/config.py`. This file already contains config items for many futures instruments, with their matching PST symbols. For example,
```python
CONTRACT_MAP = {
...
"GOLD": {"code": "GC", "cycle": "GJMQVZ", "exchange": "COMEX"},
...
}
```
indicates that instrument with PST code **GOLD** has the Barchart symbol **GC**, months **GJMQVZ**, and exchange **COMEX**. It also contains date config for various Futures exchanges. For example,
```python
EXCHANGES = {
...
"COMEX": {"tick_date": "2008-05-04", "eod_date": "1978-02-27"},
...
}
```
That indicates that futures instruments with exchange **COMEX** have daily price data from 27 Feb 1978, and hourly from 4 May 2008. Those date attributes are [provided by Barchart](https://www.barchart.com/solutions/data/market), and turn out to be inaccurate. Previous versions of this library would waste valuable allowance by attempting to download data that was not there. Newer versions handle this much better. If you use this library to download prices that are not in the config file, please consider contributing with a PR.
3. Have a look at the sample snippets in `~bc-utils/sample/pst.py`. There are examples for use with an external config file. Use the sample config `~bc-utils/sample/private_config_sample.yaml`, copy and rename to the top level of the `~bc-utils` dir. Update with your credentials and save path etc
Alternatively, paste the contents of the sample config into your PST private config, and do something like
```python
def download_with_pst_config():
config = load_config("<path to your PST private config>")
get_barchart_downloads(
create_bc_session(config),
instr_list=config['barchart_download_list'],
start_year=config['barchart_start_year'],
end_year=config['barchart_end_year'],
save_dir=config['barchart_path'],
do_daily=config['barchart_do_daily'],
dry_run=config['barchart_dry_run'],
)
```
4. add bc-utils to your crontab
```
00 08 * * 1-7 . $HOME/.profile; cd ~/bc-utils; python3 bcutils/sample/pst.py >> $ECHO_PATH/barchart_download.txt 2>&1
```
You could also add another entry to run the updater once a week.
5. To import the prices into PST:
```python
from sysdata.config.production_config import get_production_config
from syscore.fileutils import resolve_path_and_filename_for_package
from sysdata.csv.csv_futures_contract_prices import ConfigCsvFuturesPrices
from sysinit.futures.contract_prices_from_csv_to_arctic import (
init_db_with_csv_futures_contract_prices,
)
BARCHART_CONFIG = ConfigCsvFuturesPrices(
input_date_index_name="Time",
input_skiprows=0,
input_skipfooter=0,
input_date_format="%Y-%m-%dT%H:%M:%S%z",
input_column_mapping=dict(
OPEN="Open", HIGH="High", LOW="Low", FINAL="Close", VOLUME="Volume"
),
)
# assuming bc-utils config pasted into private
datapath = resolve_path_and_filename_for_package(
get_production_config().get_element_or_default("barchart_path", None)
)
init_db_with_csv_futures_contract_prices(datapath, csv_config=BARCHART_CONFIG)
```
Raw data
{
"_id": null,
"home_page": "https://github.com/bug-or-feature/bc-utils",
"name": "bc-utils",
"maintainer": null,
"docs_url": null,
"requires_python": "<4.0,>=3.8.1",
"maintainer_email": null,
"keywords": "trading",
"author": "Andy Geach",
"author_email": "andy@bugorfeature.net",
"download_url": "https://files.pythonhosted.org/packages/f2/bf/11e91b34ac4c42bb9e8e6e9f5b91b70d328b54eae3be1394953370a577fe/bc_utils-0.1.6.tar.gz",
"platform": null,
"description": "# bc-utils\n\n[Barchart.com](https://www.barchart.com) allows registered users to download historic futures contract prices in CSV format. Individual contracts must be downloaded separately, which is laborious and slow. This script automates the process.\n\n## Quickstart\n\n```\nimport os\nimport logging\nfrom bcutils.bc_utils import get_barchart_downloads, create_bc_session\n\nlogging.basicConfig(level=logging.INFO)\n\nCONTRACTS = {\n \"AUD\" : {\"code\": \"A6\", \"cycle\": \"HMUZ\", \"exchange\": \"CME\"},\n \"GOLD\" : {\"code\": \"GC\", \"cycle\": \"GJMQVZ\", \"exchange\": \"COMEX\"},\n}\n\nsession = create_bc_session(\n config_obj=dict(\n barchart_username=\"user@domain.com\",\n barchart_password=\"s3cr3t_321\",\n )\n)\n\nget_barchart_downloads(\n session,\n contract_map=CONTRACTS,\n save_dir=os.getcwd(),\n start_year=2020,\n end_year=2021\n)\n```\n\nThe code above would: \n* for the CME Australian Dollar future, get OHLCV price data for the Mar, Jun, Sep and Dec 2020 contracts\n* download in CSV format\n* save with filenames like Hour_AUD_20200300.csv, Day_AUD_20200600.csv into the specified directory\n* for COMEX Gold, get Feb, Apr, Jun, Aug, Oct, and Dec data, with filenames like Hour_GOLD_20200200.csv etc\n\nFeatures:\n* Designed to be run once a day by a scheduler\n* the script skips contracts already downloaded\n* by default gets 120 days of data per contract, override possible per instrument\n* dry run mode to check setup\n* allows updates to previously downloaded files\n* you must be a registered user. Paid subscribers get 250 downloads a day, otherwise 5\n\n## For pysystemtrade users\n\nThis project was originally created to make it easier to populate [pysystemtrade](https://github.com/robcarver17/pysystemtrade) (PST) with futures prices from Barchart, so setup is straightforward. Steps:\n\n1. Clone the bc-utils repo, or your own fork. The remaining steps assume the location `~/bc-utils`\n\n2. Have a look at `~/bc-utils/bcutils/config.py`. This file already contains config items for many futures instruments, with their matching PST symbols. For example, \n```python\nCONTRACT_MAP = {\n ...\n \"GOLD\": {\"code\": \"GC\", \"cycle\": \"GJMQVZ\", \"exchange\": \"COMEX\"},\n ...\n}\n```\nindicates that instrument with PST code **GOLD** has the Barchart symbol **GC**, months **GJMQVZ**, and exchange **COMEX**. It also contains date config for various Futures exchanges. For example,\n\n```python\nEXCHANGES = {\n ...\n \"COMEX\": {\"tick_date\": \"2008-05-04\", \"eod_date\": \"1978-02-27\"},\n ...\n}\n```\n\nThat indicates that futures instruments with exchange **COMEX** have daily price data from 27 Feb 1978, and hourly from 4 May 2008. Those date attributes are [provided by Barchart](https://www.barchart.com/solutions/data/market), and turn out to be inaccurate. Previous versions of this library would waste valuable allowance by attempting to download data that was not there. Newer versions handle this much better. If you use this library to download prices that are not in the config file, please consider contributing with a PR.\n\n3. Have a look at the sample snippets in `~bc-utils/sample/pst.py`. There are examples for use with an external config file. Use the sample config `~bc-utils/sample/private_config_sample.yaml`, copy and rename to the top level of the `~bc-utils` dir. Update with your credentials and save path etc\n\nAlternatively, paste the contents of the sample config into your PST private config, and do something like\n\n```python\ndef download_with_pst_config():\n config = load_config(\"<path to your PST private config>\")\n get_barchart_downloads(\n create_bc_session(config),\n instr_list=config['barchart_download_list'],\n start_year=config['barchart_start_year'],\n end_year=config['barchart_end_year'],\n save_dir=config['barchart_path'],\n do_daily=config['barchart_do_daily'],\n dry_run=config['barchart_dry_run'],\n )\n```\n\n4. add bc-utils to your crontab\n```\n00 08 * * 1-7 . $HOME/.profile; cd ~/bc-utils; python3 bcutils/sample/pst.py >> $ECHO_PATH/barchart_download.txt 2>&1\n```\nYou could also add another entry to run the updater once a week.\n\n5. To import the prices into PST:\n\n```python\nfrom sysdata.config.production_config import get_production_config\nfrom syscore.fileutils import resolve_path_and_filename_for_package\nfrom sysdata.csv.csv_futures_contract_prices import ConfigCsvFuturesPrices\nfrom sysinit.futures.contract_prices_from_csv_to_arctic import (\n init_db_with_csv_futures_contract_prices,\n)\n\nBARCHART_CONFIG = ConfigCsvFuturesPrices(\n input_date_index_name=\"Time\",\n input_skiprows=0,\n input_skipfooter=0,\n input_date_format=\"%Y-%m-%dT%H:%M:%S%z\",\n input_column_mapping=dict(\n OPEN=\"Open\", HIGH=\"High\", LOW=\"Low\", FINAL=\"Close\", VOLUME=\"Volume\"\n ),\n)\n\n# assuming bc-utils config pasted into private\ndatapath = resolve_path_and_filename_for_package(\n get_production_config().get_element_or_default(\"barchart_path\", None)\n)\n\ninit_db_with_csv_futures_contract_prices(datapath, csv_config=BARCHART_CONFIG)\n```\n",
"bugtrack_url": null,
"license": "BSD-3-Clause",
"summary": "Python utility automation scripts for Barchart.com",
"version": "0.1.6",
"project_urls": {
"Homepage": "https://github.com/bug-or-feature/bc-utils",
"Issues": "https://github.com/bug-or-feature/bc-utils/issues",
"Repository": "https://github.com/bug-or-feature/bc-utils"
},
"split_keywords": [
"trading"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "4c20ee20913fb6c0dd94434e4a271c025986af5e33f51ec52e2fe023edf9faf0",
"md5": "20bba8bdb44abe4314782464ceefc594",
"sha256": "ed19eeb13cb9105e5d2ed96eba5a48f3b8ad1e001f775310b98b18228132c86c"
},
"downloads": -1,
"filename": "bc_utils-0.1.6-py3-none-any.whl",
"has_sig": false,
"md5_digest": "20bba8bdb44abe4314782464ceefc594",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4.0,>=3.8.1",
"size": 30381,
"upload_time": "2024-06-13T08:54:52",
"upload_time_iso_8601": "2024-06-13T08:54:52.151550Z",
"url": "https://files.pythonhosted.org/packages/4c/20/ee20913fb6c0dd94434e4a271c025986af5e33f51ec52e2fe023edf9faf0/bc_utils-0.1.6-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "f2bf11e91b34ac4c42bb9e8e6e9f5b91b70d328b54eae3be1394953370a577fe",
"md5": "4dcbb77f72cbff70c7997dcab88bb49c",
"sha256": "603818f35acd702a5c4f153361635aac3daf26e7a2f6f519c5b4a00f32be4879"
},
"downloads": -1,
"filename": "bc_utils-0.1.6.tar.gz",
"has_sig": false,
"md5_digest": "4dcbb77f72cbff70c7997dcab88bb49c",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4.0,>=3.8.1",
"size": 30267,
"upload_time": "2024-06-13T08:54:53",
"upload_time_iso_8601": "2024-06-13T08:54:53.364580Z",
"url": "https://files.pythonhosted.org/packages/f2/bf/11e91b34ac4c42bb9e8e6e9f5b91b70d328b54eae3be1394953370a577fe/bc_utils-0.1.6.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-06-13 08:54:53",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "bug-or-feature",
"github_project": "bc-utils",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "bc-utils"
}
JSON
