======================
Hydrobot
======================
.. image:: https://img.shields.io/pypi/v/hydrobot.svg
:target: https://pypi.python.org/pypi/hydrobot
.. image:: https://readthedocs.org/projects/hydrobot/badge/?version=latest
:target: https://hydrobot.readthedocs.io/en/latest/?version=latest
:alt: Documentation Status
Python Package providing a suite of processing tools and utilities for Hilltop hydrological data.
* Free software: GNU General Public License v3
* Documentation: https://hydrobot.readthedocs.io.
Features
--------
* Processes data downloaded from Hilltop Server
* Uses annalist to record all changes to data
* Capable of various automated processing techniques, including:
* Clipping data
* Removing spikes based on FBEWMA smoothing
* Identifying and removing 'flatlining' data, where an instrument repeats it's last collected data point (NOTE: It's unclear if this actually happening.)
* Identifying gaps and gap lengths and closing small gaps
* Aggregating check data from various sources.
* Quality coding data based on NEMS standards
* Plotting data, including:
* Processed data with quality codes
* Comparing raw data to processed data
* Showing all changes to the data
* Visualizing check points from various sources.
Usage (Alpha)
-------------
The Alpha release of Hydrobot supports a "hybrid" workflow. This means that some external tools are still required to do a full processing. Importantly, the hybrid workflow relies on some R scripts to obtain check data from sources other than Hilltop. Further processing using Hilltop manager is also supported.
NOTE: Hydrobot 0.5.2 supports only Water Temperature processing at the moment, but more measurements will be supported in patches as the processing progresses.
Initial Setup (Repeat for each release)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#. Install a Python 3.11 interpreter. Note that 3.12 is not supported just yet.
#. In your favourite shell, create a new virtual environment using this python interpreter and name it "hydrobot0.5.2". It's important that this is stored somewhere locally. I suggest creating a folder for virtual environments in your home folder::
python -m venv path/to/venv/hydrobot0.5.2/
#. Activate this virtual environment. In powershell this should be something like::
./path/to/venv/hydrobot0.5.2/Scripts/Activate.ps1
#. If you're sure your venv is active (ensure with `which python` and confirm that you're using the interpreter in your venv folder), install the latest version of Hydrobot using pip::
pip install hydrobot
#. Record which version of dependencies you have installed. The following pip freeze records which dependencies are
installed by the hydrobot install process for if auditing/reprocessing is required later::
pip freeze > dependencies.txt
#. Navigate to the processing folder that you create as part of the Processing Steps below::
cd //ares/hydro/processing/whatever/watertemp/site/30X/
Processing Steps
^^^^^^^^^^^^^^^^
#. Open Logsheet Loader. Fill it as normal, and note the start date of your processing period (i.e. end date of the previous period).
#. Navigate to the data source and site folder, and create your processing folder.
#. Copy all the processing files in this folder into your processing folder::
\\ares\Environmental Data Validation\Water Temperature\Documents\Hydrobot_0.5.2_Files\
//ares/Environmental\ Data\ Validation/Water\ Temperature/Documents/Hydrobot_0.5.2_Files/
#. In your processing folder, open the `config.yaml` file and change the fields `site`, `from_date`, `to_date`, `analyst_name`. Feel free to mess with the other values once you get the hang of it. No one will die.
#. Run the R script. I'm not an R guy so I'm not sure how to do this other than to open it in R studio, highlighting all the code, and hitting `Ctrl+Enter`. This should create a bunch of `.csv` files containing the check data from various sources. This is a good reasource for perusal during processing, but will be imbibed by hydrobot to for QC encoding.
#. Make sure your virtual environment is set up and active. Ensure with `which python` and confirm that your python interpreter is running from your venv folder.
#. Run the hydrobot processing script::
python processing_script.py
#. If all goes well, the processing script will open a browser tab showing a diagnostic dash for your site. Use this to identify issues in the site.
#. Use your python skills to solve some issues, like removing erroneous check data points or deleting chunks of data. More extensive documentation on actually using hydrobot will follow in future releases.
#. Open the resulting processed.xml in manager, and copy it over to a hts file.
#. Open the WaterTemp_check_data.csv outputed from the R file in a spreadsheet (sorry) and copy into hts file.
#. Happy processing!
Credits
-------
This package was created with Cookiecutter_ and the `audreyr/cookiecutter-pypackage`_ project template. Furthermore,
Sam is a real champ with the coding and whatnot. Thanks Sam.
Aww thanks Nic. You also da man <3
.. _Cookiecutter: https://github.com/audreyr/cookiecutter
.. _`audreyr/cookiecutter-pypackage`: https://github.com/audreyr/cookiecutter-pypackage
Raw data
{
"_id": null,
"home_page": null,
"name": "hydrobot",
"maintainer": null,
"docs_url": null,
"requires_python": "==3.11.*",
"maintainer_email": null,
"keywords": "hydrology, automation, pandas, hilltop, hilltop-py, HorizonsRC",
"author": null,
"author_email": "Nic Mostert <nicolas.mostert@horizons.govt.nz>, Sam Irvine <sam.irvine@horizons.govt.nz>",
"download_url": "https://files.pythonhosted.org/packages/80/a8/ee7b9e69f4c1617206f324ac7c496a2bbf08e0ff6c72a54b908bc7637a46/hydrobot-0.5.2.tar.gz",
"platform": null,
"description": "======================\nHydrobot\n======================\n\n\n.. image:: https://img.shields.io/pypi/v/hydrobot.svg\n :target: https://pypi.python.org/pypi/hydrobot\n\n.. image:: https://readthedocs.org/projects/hydrobot/badge/?version=latest\n :target: https://hydrobot.readthedocs.io/en/latest/?version=latest\n :alt: Documentation Status\n\nPython Package providing a suite of processing tools and utilities for Hilltop hydrological data.\n\n\n* Free software: GNU General Public License v3\n* Documentation: https://hydrobot.readthedocs.io.\n\n\nFeatures\n--------\n\n* Processes data downloaded from Hilltop Server\n* Uses annalist to record all changes to data\n* Capable of various automated processing techniques, including:\n\n * Clipping data\n * Removing spikes based on FBEWMA smoothing\n * Identifying and removing 'flatlining' data, where an instrument repeats it's last collected data point (NOTE: It's unclear if this actually happening.)\n * Identifying gaps and gap lengths and closing small gaps\n * Aggregating check data from various sources.\n * Quality coding data based on NEMS standards\n\n* Plotting data, including:\n\n * Processed data with quality codes\n * Comparing raw data to processed data\n * Showing all changes to the data\n * Visualizing check points from various sources.\n\nUsage (Alpha)\n-------------\n\nThe Alpha release of Hydrobot supports a \"hybrid\" workflow. This means that some external tools are still required to do a full processing. Importantly, the hybrid workflow relies on some R scripts to obtain check data from sources other than Hilltop. Further processing using Hilltop manager is also supported.\n\nNOTE: Hydrobot 0.5.2 supports only Water Temperature processing at the moment, but more measurements will be supported in patches as the processing progresses.\n\nInitial Setup (Repeat for each release)\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n#. Install a Python 3.11 interpreter. Note that 3.12 is not supported just yet.\n#. In your favourite shell, create a new virtual environment using this python interpreter and name it \"hydrobot0.5.2\". It's important that this is stored somewhere locally. I suggest creating a folder for virtual environments in your home folder::\n\n python -m venv path/to/venv/hydrobot0.5.2/\n\n#. Activate this virtual environment. In powershell this should be something like::\n\n ./path/to/venv/hydrobot0.5.2/Scripts/Activate.ps1\n\n#. If you're sure your venv is active (ensure with `which python` and confirm that you're using the interpreter in your venv folder), install the latest version of Hydrobot using pip::\n\n pip install hydrobot\n\n#. Record which version of dependencies you have installed. The following pip freeze records which dependencies are\ninstalled by the hydrobot install process for if auditing/reprocessing is required later::\n\n pip freeze > dependencies.txt\n\n#. Navigate to the processing folder that you create as part of the Processing Steps below::\n\n cd //ares/hydro/processing/whatever/watertemp/site/30X/\n\n\nProcessing Steps\n^^^^^^^^^^^^^^^^\n\n#. Open Logsheet Loader. Fill it as normal, and note the start date of your processing period (i.e. end date of the previous period).\n#. Navigate to the data source and site folder, and create your processing folder.\n#. Copy all the processing files in this folder into your processing folder::\n\n \\\\ares\\Environmental Data Validation\\Water Temperature\\Documents\\Hydrobot_0.5.2_Files\\\n\n //ares/Environmental\\ Data\\ Validation/Water\\ Temperature/Documents/Hydrobot_0.5.2_Files/\n\n#. In your processing folder, open the `config.yaml` file and change the fields `site`, `from_date`, `to_date`, `analyst_name`. Feel free to mess with the other values once you get the hang of it. No one will die.\n\n#. Run the R script. I'm not an R guy so I'm not sure how to do this other than to open it in R studio, highlighting all the code, and hitting `Ctrl+Enter`. This should create a bunch of `.csv` files containing the check data from various sources. This is a good reasource for perusal during processing, but will be imbibed by hydrobot to for QC encoding.\n\n#. Make sure your virtual environment is set up and active. Ensure with `which python` and confirm that your python interpreter is running from your venv folder.\n\n#. Run the hydrobot processing script::\n\n python processing_script.py\n\n#. If all goes well, the processing script will open a browser tab showing a diagnostic dash for your site. Use this to identify issues in the site.\n\n#. Use your python skills to solve some issues, like removing erroneous check data points or deleting chunks of data. More extensive documentation on actually using hydrobot will follow in future releases.\n\n#. Open the resulting processed.xml in manager, and copy it over to a hts file.\n\n#. Open the WaterTemp_check_data.csv outputed from the R file in a spreadsheet (sorry) and copy into hts file.\n\n#. Happy processing!\n\nCredits\n-------\n\nThis package was created with Cookiecutter_ and the `audreyr/cookiecutter-pypackage`_ project template. Furthermore,\nSam is a real champ with the coding and whatnot. Thanks Sam.\n\nAww thanks Nic. You also da man <3\n\n.. _Cookiecutter: https://github.com/audreyr/cookiecutter\n.. _`audreyr/cookiecutter-pypackage`: https://github.com/audreyr/cookiecutter-pypackage\n",
"bugtrack_url": null,
"license": "GNU General Public License v3",
"summary": "A suite of processing tools for Hilltop hydrological data.",
"version": "0.5.2",
"project_urls": {
"Documentation": "https://hydrobot.readthedocs.io",
"Homepage": "https://github.com/HorizonsRC/hydrobot",
"Issues": "https://github.com/HorizonsRC/hydrobot/issues",
"Package": "https://pypi.org/project/hydrobot"
},
"split_keywords": [
"hydrology",
" automation",
" pandas",
" hilltop",
" hilltop-py",
" horizonsrc"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "dc2fa77611a8c6e983b8b112bcdaba30887a5b13d7fe8e5f4d5804e325de5d27",
"md5": "8965745087dda600667d49a6a56a5420",
"sha256": "53a9e10ba503aa708e1034d386bffb4aca1dadc885adcd3d4927e3b46a10f70a"
},
"downloads": -1,
"filename": "hydrobot-0.5.2-py3-none-any.whl",
"has_sig": false,
"md5_digest": "8965745087dda600667d49a6a56a5420",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "==3.11.*",
"size": 36488,
"upload_time": "2024-04-10T21:29:13",
"upload_time_iso_8601": "2024-04-10T21:29:13.058785Z",
"url": "https://files.pythonhosted.org/packages/dc/2f/a77611a8c6e983b8b112bcdaba30887a5b13d7fe8e5f4d5804e325de5d27/hydrobot-0.5.2-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "80a8ee7b9e69f4c1617206f324ac7c496a2bbf08e0ff6c72a54b908bc7637a46",
"md5": "df6eb93b69d9dbcf11e6e6fd0dba2335",
"sha256": "e8923615facb6c23664cd91656531febe8f008bf62d3ddc05f2bb649d2c5e855"
},
"downloads": -1,
"filename": "hydrobot-0.5.2.tar.gz",
"has_sig": false,
"md5_digest": "df6eb93b69d9dbcf11e6e6fd0dba2335",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "==3.11.*",
"size": 80426,
"upload_time": "2024-04-10T21:29:18",
"upload_time_iso_8601": "2024-04-10T21:29:18.777755Z",
"url": "https://files.pythonhosted.org/packages/80/a8/ee7b9e69f4c1617206f324ac7c496a2bbf08e0ff6c72a54b908bc7637a46/hydrobot-0.5.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-04-10 21:29:18",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "HorizonsRC",
"github_project": "hydrobot",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "hydrobot"
}
JSON
