# pvdaq_access
Detailed documentaion is found in [pvdaq_access Pages][3]
## Overview
Data from public PV Sites is available through the [OEDI Data Lake][1]. This data is refreshed monthly where possible, and is stored in daily CSV or Parquet files for each system. This softeware module will allow a user to downlaod all the data to their local system from the OEDI Data Lake.
### Solar Bounty Data Prize Datasets
Part of the data archives in PVDAQ contains the winning data sets from the 2023 Solar Data Bounty Prize systems. This storage architecture is a bit different, therefore a new function has been added to facilitate this download.
When the software runs it will ask via a console prompt if the system you are downloading is one of the prize sites. If it is, it will extract data from those archives instead of the PVDAQ basic archives.
## Usage
This is an exectuable python packge. It requires the passing of parameters at the start.
* system : Followed by an integer of the system unique ID.
* path : Followed by a string that targets the local computer location/path you wish the data to be stored into.
* parquet : A flag that changes the data type from default CSV to Parquet file for download.
System IDs for PVDAQ basic sites can be found in the metadata file for Available System Information on the [OEDI site][1]</br>
System IDs for the Prize sites can be found on their repository folders within the PVDAQ Data Lake or from information on the [PVDAQ interactive website][2]
## Infomation on running the program
When the software starts the user may be prompted for additional information. The software will prompt the user to identify if this is a Solar Data Bounty Prize request or one of the
PVDAQ basic sites. If the this is one of the PVDAQ basic sites the software will further ask, after the download of files, if the user wants to concatenate all the files into a single time-series
file for analysis.
If the this is one of the PVDAQ basic sites the software will further ask, after the download of files, if the user wants to concatenate all the files into a single time-series file for analysis.
### Processing
The PVDAQ basic data will be downloaded as the series of daily files. At this point you can then concatenate the files together using the built in function to create a full time-series for the system.
We decided to maintain the daily files for download, due to the size of the final file. In some cases, this could be quite large and to take an extreme amount of time to download.
For the Solar Data Bounty Prize files, these are curated files that have been broken down into a series of hardware specific and in some cases, yearly files. To concatenate these files could be
difficult to do without prepration since some systems have many years of data and thousands of columns.
For these larger Data Prize systems we advise assuring the local system being downloaded to has enough space and that there are not any bandwidth network issues to address. In some cases
you could be downloading hundreds of gigabytes of data. Plan accordingly.
</code>
[1]:https://data.openei.org/submissions/4568
[2]:https://openei.org/wiki/PVDAQ
[3]:https://nrel.github.io/pvdaq_access/
Raw data
{
"_id": null,
"home_page": "https://github.com/NREL/pvdaq_access",
"name": "pvdaq-access",
"maintainer": "",
"docs_url": null,
"requires_python": "",
"maintainer_email": "",
"keywords": "photovoltaic data datasets performance power production weatherdata PVFleets SolarDataPrize",
"author": "Robert White",
"author_email": "robert.white@nrel.gov",
"download_url": "https://files.pythonhosted.org/packages/85/c0/66ef866cf284ea42e521f85019a6f486c85a6c07fcbff39cff329fcf248a/pvdaq_access-0.1.2.tar.gz",
"platform": null,
"description": "\n# pvdaq_access\nDetailed documentaion is found in [pvdaq_access Pages][3]\n## Overview\nData from public PV Sites is available through the [OEDI Data Lake][1]. This data is refreshed monthly where possible, and is stored in daily CSV or Parquet files for each system. This softeware module will allow a user to downlaod all the data to their local system from the OEDI Data Lake.\n\n### Solar Bounty Data Prize Datasets\nPart of the data archives in PVDAQ contains the winning data sets from the 2023 Solar Data Bounty Prize systems. This storage architecture is a bit different, therefore a new function has been added to facilitate this download.\n\nWhen the software runs it will ask via a console prompt if the system you are downloading is one of the prize sites. If it is, it will extract data from those archives instead of the PVDAQ basic archives.\n\n## Usage\nThis is an exectuable python packge. It requires the passing of parameters at the start.\n* system : Followed by an integer of the system unique ID. \n* path : Followed by a string that targets the local computer location/path you wish the data to be stored into.\n* parquet : A flag that changes the data type from default CSV to Parquet file for download.\n\nSystem IDs for PVDAQ basic sites can be found in the metadata file for Available System Information on the [OEDI site][1]</br>\nSystem IDs for the Prize sites can be found on their repository folders within the PVDAQ Data Lake or from information on the [PVDAQ interactive website][2] \n\n## Infomation on running the program\nWhen the software starts the user may be prompted for additional information. The software will prompt the user to identify if this is a Solar Data Bounty Prize request or one of the \nPVDAQ basic sites. If the this is one of the PVDAQ basic sites the software will further ask, after the download of files, if the user wants to concatenate all the files into a single time-series \nfile for analysis. \n\nIf the this is one of the PVDAQ basic sites the software will further ask, after the download of files, if the user wants to concatenate all the files into a single time-series file for analysis. \n\n### Processing\nThe PVDAQ basic data will be downloaded as the series of daily files. At this point you can then concatenate the files together using the built in function to create a full time-series for the system. \nWe decided to maintain the daily files for download, due to the size of the final file. In some cases, this could be quite large and to take an extreme amount of time to download.\n\nFor the Solar Data Bounty Prize files, these are curated files that have been broken down into a series of hardware specific and in some cases, yearly files. To concatenate these files could be \ndifficult to do without prepration since some systems have many years of data and thousands of columns. \n\nFor these larger Data Prize systems we advise assuring the local system being downloaded to has enough space and that there are not any bandwidth network issues to address. In some cases\nyou could be downloading hundreds of gigabytes of data. Plan accordingly.\n\n</code>\n[1]:https://data.openei.org/submissions/4568\n[2]:https://openei.org/wiki/PVDAQ\n[3]:https://nrel.github.io/pvdaq_access/\n",
"bugtrack_url": null,
"license": "BSD License (3 clause)",
"summary": "Functions to interface with PVDAQ datasets",
"version": "0.1.2",
"project_urls": {
"Homepage": "https://github.com/NREL/pvdaq_access"
},
"split_keywords": [
"photovoltaic",
"data",
"datasets",
"performance",
"power",
"production",
"weatherdata",
"pvfleets",
"solardataprize"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "72cd8552442236690340c5db371398aa83b5b1c987b64eb3b487c7d5ad166d23",
"md5": "97177fabbb0088b89d02240fc86d8f27",
"sha256": "393d502a5fc0f3ccde05c0e42c4d2223e2be37eba9e2ae9a1cfda24d738c5129"
},
"downloads": -1,
"filename": "pvdaq_access-0.1.2-py2.py3-none-any.whl",
"has_sig": false,
"md5_digest": "97177fabbb0088b89d02240fc86d8f27",
"packagetype": "bdist_wheel",
"python_version": "py2.py3",
"requires_python": null,
"size": 5182,
"upload_time": "2024-03-09T14:30:08",
"upload_time_iso_8601": "2024-03-09T14:30:08.190583Z",
"url": "https://files.pythonhosted.org/packages/72/cd/8552442236690340c5db371398aa83b5b1c987b64eb3b487c7d5ad166d23/pvdaq_access-0.1.2-py2.py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "85c066ef866cf284ea42e521f85019a6f486c85a6c07fcbff39cff329fcf248a",
"md5": "34eea9d10127190fc252d83144c7aee3",
"sha256": "3f50a70e66b68942d33af422cb1c7f1ed481ed972cf8f8530c8e3719e4b538ca"
},
"downloads": -1,
"filename": "pvdaq_access-0.1.2.tar.gz",
"has_sig": false,
"md5_digest": "34eea9d10127190fc252d83144c7aee3",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 25159,
"upload_time": "2024-03-09T14:30:09",
"upload_time_iso_8601": "2024-03-09T14:30:09.224428Z",
"url": "https://files.pythonhosted.org/packages/85/c0/66ef866cf284ea42e521f85019a6f486c85a6c07fcbff39cff329fcf248a/pvdaq_access-0.1.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-03-09 14:30:09",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "NREL",
"github_project": "pvdaq_access",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [
{
"name": "boto3",
"specs": []
},
{
"name": "botocore",
"specs": []
},
{
"name": "jmespath",
"specs": []
},
{
"name": "numpy",
"specs": []
},
{
"name": "pandas",
"specs": []
},
{
"name": "python-dateutil",
"specs": []
},
{
"name": "pytz",
"specs": []
},
{
"name": "s3transfer",
"specs": [
[
"==",
"0.6.0"
]
]
},
{
"name": "six",
"specs": []
},
{
"name": "urllib3",
"specs": []
}
],
"lcname": "pvdaq-access"
}
JSON
