| Name | nemdata JSON |
| Version |
0.3.4
JSON |
| download |
| home_page | |
| Summary | Simple CLI for downloading data for Australia's NEM from AEMO. |
| upload_time | 2023-02-09 06:35:11 |
| maintainer | |
| docs_url | None |
| author | Adam Green |
| requires_python | >=3.10,<4.0 |
| license | MIT |
| keywords |
|
| VCS |
|
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# nem-data
A simple & opinionated Python command line tool to access Australian National Energy Market (NEM) data provided by the Australian Energy Market Operator (AEMO). It features:
- access to the NEMDE dataset as well as the predispatch, unit-scada, trading-price, demand and interconnectors tables from MMSDM,
- a cache to not re-download data that already exists in `~/nem-data/data`,
- adds `interval-start` and `interval-end` columns.
It is designed for use by researchers & data scientists - this tool supports my personal research work. It is not designed for production use - see [NEMOSIS](https://github.com/UNSW-CEEM/NEMOSIS) for a production grade package.
See [A Hackers Guide to AEMO & NEM Data](https://adgefficiency.com/hackers-aemo/) for more context on the data provided by AEMO.
## Setup
```bash
$ pip install nemdata
```
## Use
### CLI
```shell-session
$ nemdata --help
Usage: nemdata [OPTIONS]
Downloads NEM data from AEMO.
Options:
-t, --table TEXT Available tables: nemde, dispatch-price,
predispatch, unit-scada, trading-price, demand,
interconnectors.
-s, --start TEXT Start date (YYYY-MM or YYYY-MM-DD for NEMDE).
-e, --end TEXT End date (incusive) (YYYY-MM or YYYY-MM-DD for
NEMDE).
--dry-run / --no-dry-run Whether to save downloaded data to disk.
--help Show this message and exit.
```
Download NEMDE data for the first three days in January 2018:
```shell-session
$ nemdata -t nemde --start 2018-01-01 --end 2018-01-03
```
Download trading price data from MMSDM for January to March 2018:
```shell-session
$ nemdata -t trading-price -s 2018-01 -e 2018-03
```
### Python
Download trading price data from MMSDM for January to Feburary 2020:
```python
import nemdata
data = nemdata.download(start="2020-01", end="2020-02", table="trading-price")
```
Load this data back into a pandas DataFrame:
```python
data = nemdata.load()['trading-price']
```
At this point, `data` will have the trading price for all regions.
## Data
Downloaded into into `$HOME/nem-data/data/`:
```shell-session
$ nemdata -t trading-price -s 2020-01 -e 2020-02
$ tree ~/nem-data
/Users/adam/nem-data
└── data
└── trading-price
├── 2020-01
│ ├── PUBLIC_DVD_TRADINGPRICE_202001010000.CSV
│ ├── clean.csv
│ ├── clean.parquet
│ └── raw.zip
└── 2020-02
├── PUBLIC_DVD_TRADINGPRICE_202002010000.CSV
├── clean.csv
├── clean.parquet
└── raw.zip
```
A few things happen during data processing:
- rows of the raw CSV are removed to create a rectangular, single table CSV,
- `interval-start` and `interval-end` timezone aware datetime columns are added,
- when using `nemdata.loader.loader` for `trading-price`, all data is resampled to a 5 minute frequency (both before and after the 30 to 5 minute settlement interval change).
Raw data
{
"_id": null,
"home_page": "",
"name": "nemdata",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.10,<4.0",
"maintainer_email": "",
"keywords": "",
"author": "Adam Green",
"author_email": "adam.green@adgefficiency.com",
"download_url": "https://files.pythonhosted.org/packages/b1/13/4f311b4e2e0691eb5bde2e207f2876646dcd2ed752a3b6e021f5ead29942/nemdata-0.3.4.tar.gz",
"platform": null,
"description": "# nem-data\n\nA simple & opinionated Python command line tool to access Australian National Energy Market (NEM) data provided by the Australian Energy Market Operator (AEMO). It features:\n\n- access to the NEMDE dataset as well as the predispatch, unit-scada, trading-price, demand and interconnectors tables from MMSDM,\n- a cache to not re-download data that already exists in `~/nem-data/data`,\n- adds `interval-start` and `interval-end` columns.\n\nIt is designed for use by researchers & data scientists - this tool supports my personal research work. It is not designed for production use - see [NEMOSIS](https://github.com/UNSW-CEEM/NEMOSIS) for a production grade package.\n\nSee [A Hackers Guide to AEMO & NEM Data](https://adgefficiency.com/hackers-aemo/) for more context on the data provided by AEMO.\n\n\n## Setup\n\n```bash\n$ pip install nemdata\n```\n\n\n## Use\n\n### CLI\n\n```shell-session\n$ nemdata --help\nUsage: nemdata [OPTIONS]\n\n Downloads NEM data from AEMO.\n\nOptions:\n -t, --table TEXT Available tables: nemde, dispatch-price,\n predispatch, unit-scada, trading-price, demand,\n interconnectors.\n -s, --start TEXT Start date (YYYY-MM or YYYY-MM-DD for NEMDE).\n -e, --end TEXT End date (incusive) (YYYY-MM or YYYY-MM-DD for\n NEMDE).\n --dry-run / --no-dry-run Whether to save downloaded data to disk.\n --help Show this message and exit.\n```\n\nDownload NEMDE data for the first three days in January 2018:\n\n```shell-session\n$ nemdata -t nemde --start 2018-01-01 --end 2018-01-03\n```\n\nDownload trading price data from MMSDM for January to March 2018:\n\n```shell-session\n$ nemdata -t trading-price -s 2018-01 -e 2018-03\n```\n\n### Python\n\nDownload trading price data from MMSDM for January to Feburary 2020:\n\n```python\nimport nemdata\n\ndata = nemdata.download(start=\"2020-01\", end=\"2020-02\", table=\"trading-price\")\n```\n\nLoad this data back into a pandas DataFrame:\n\n```python\ndata = nemdata.load()['trading-price']\n```\n\nAt this point, `data` will have the trading price for all regions.\n\n\n## Data\n\nDownloaded into into `$HOME/nem-data/data/`:\n\n```shell-session\n$ nemdata -t trading-price -s 2020-01 -e 2020-02\n$ tree ~/nem-data\n/Users/adam/nem-data\n\u2514\u2500\u2500 data\n \u2514\u2500\u2500 trading-price\n \u251c\u2500\u2500 2020-01\n \u2502\u00a0\u00a0 \u251c\u2500\u2500 PUBLIC_DVD_TRADINGPRICE_202001010000.CSV\n \u2502\u00a0\u00a0 \u251c\u2500\u2500 clean.csv\n \u2502\u00a0\u00a0 \u251c\u2500\u2500 clean.parquet\n \u2502\u00a0\u00a0 \u2514\u2500\u2500 raw.zip\n \u2514\u2500\u2500 2020-02\n \u251c\u2500\u2500 PUBLIC_DVD_TRADINGPRICE_202002010000.CSV\n \u251c\u2500\u2500 clean.csv\n \u251c\u2500\u2500 clean.parquet\n \u2514\u2500\u2500 raw.zip\n```\n\nA few things happen during data processing:\n\n- rows of the raw CSV are removed to create a rectangular, single table CSV,\n- `interval-start` and `interval-end` timezone aware datetime columns are added,\n- when using `nemdata.loader.loader` for `trading-price`, all data is resampled to a 5 minute frequency (both before and after the 30 to 5 minute settlement interval change).\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Simple CLI for downloading data for Australia's NEM from AEMO.",
"version": "0.3.4",
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "75f3cfeed308b4c7deeee6d17f94b2855c614e88e4247bcf585ff9fdcad8f2f5",
"md5": "ff1414a458fa27cfaae6a81b37e31fad",
"sha256": "a67eb4f9d96112d5549e6be9946d3e2e34f3fd796077d36de77b4ef5bfe4f049"
},
"downloads": -1,
"filename": "nemdata-0.3.4-py3-none-any.whl",
"has_sig": false,
"md5_digest": "ff1414a458fa27cfaae6a81b37e31fad",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.10,<4.0",
"size": 10558,
"upload_time": "2023-02-09T06:35:10",
"upload_time_iso_8601": "2023-02-09T06:35:10.631620Z",
"url": "https://files.pythonhosted.org/packages/75/f3/cfeed308b4c7deeee6d17f94b2855c614e88e4247bcf585ff9fdcad8f2f5/nemdata-0.3.4-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "b1134f311b4e2e0691eb5bde2e207f2876646dcd2ed752a3b6e021f5ead29942",
"md5": "a3c44839da64ffc52d72402cc9eb92f0",
"sha256": "9cbd0b5137b63a84c383730ba37e6f1ba71aa6aad5f5328163ba8ee3f44ec80a"
},
"downloads": -1,
"filename": "nemdata-0.3.4.tar.gz",
"has_sig": false,
"md5_digest": "a3c44839da64ffc52d72402cc9eb92f0",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.10,<4.0",
"size": 9066,
"upload_time": "2023-02-09T06:35:11",
"upload_time_iso_8601": "2023-02-09T06:35:11.971820Z",
"url": "https://files.pythonhosted.org/packages/b1/13/4f311b4e2e0691eb5bde2e207f2876646dcd2ed752a3b6e021f5ead29942/nemdata-0.3.4.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-02-09 06:35:11",
"github": false,
"gitlab": false,
"bitbucket": false,
"lcname": "nemdata"
}
