======================
Hydrobot
======================
.. image:: https://img.shields.io/pypi/v/hydrobot.svg
:target: https://pypi.python.org/pypi/hydrobot
.. image:: https://readthedocs.org/projects/hydrobot/badge/?version=latest
:target: https://hydrobot.readthedocs.io/en/latest/?version=latest
:alt: Documentation Status
Python Package providing a suite of processing tools and utilities for Hilltop
hydrological data.
* Free software: GNU General Public License v3
* Documentation: https://hydrobot.readthedocs.io.
Features
--------
* Processes data downloaded from Hilltop Server
* Uses annalist to record all changes to data
* Capable of various automated processing techniques, including:
* Clipping data
* Removing spikes based on FBEWMA smoothing
* Identifying and removing 'flatlining' data, where an instrument repeats
it's last collected data point (NOTE: It's unclear if this actually
happening.)
* Identifying gaps and gap lengths and closing small gaps
* Aggregating check data from various sources.
* Quality coding data based on NEMS standards
* Plotting data, including:
* Processed data with quality codes
* Comparing raw data to processed data
* Showing all changes to the data
* Visualizing check points from various sources.
Usage (Alpha)
-------------
The Alpha release of Hydrobot supports a "hybrid" workflow. This means that
some external tools are still required to do a full processing. Importantly,
the hybrid workflow relies on some R scripts to obtain check data from sources
other than Hilltop. Further processing using Hilltop manager is also supported.
NOTE: Hydrobot 0.7.8 does not support all NEMs data sources currently,
but more measurements will be supported in patches as the processing
progresses.
Initial Setup (Repeat for each release)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#. Install a Python 3.11 interpreter (3.12 is not supported just yet).
#. In your favourite shell (if you don't know what that is, use powershell -
it's already installed on windows), create a new virtual environment using
this python interpreter and name it "hydrobot0.7.8". It's important that
this is stored somewhere locally. For example, it could be stored in a
"Hydrobot" folder in the C: drive, which would need the command::
python -m venv C:/Hydrobot/hydrobot0.7.8/
cd C:/Hydrobot/hydrobot0.7.8/
#. Activate this virtual environment. In powershell this should be something
like::
C:/Hydrobot/hydrobot0.7.8/Scripts/Activate.ps1
#. With your venv active, install the latest version of Hydrobot using pip::
pip install hydrobot
#. Record which version of dependencies you have installed. The following pip
freeze records which dependencies are installed by the hydrobot install
process for if auditing/reprocessing is required later::
pip freeze > dependencies.txt
Processing Steps
^^^^^^^^^^^^^^^^
#. Open Logsheet Loader. Fill it as normal, and note the start date of your
processing period (i.e. end date of the previous period).
#. Navigate to the data source and site folder, and create your batch folder.
#. Copy all the hydrobot processing template files from the documents folder
into your new batch folder, then rename with batch number and location code.
The location of e.g. the water temperature template is::
\\ares\Environmental Data Validation\Water Temperature\Documents\Hydrobot_template\
#. In your processing folder, open the `config.yaml` file and fill the fields
`site`, `from_date`, `to_date`, `analyst_name`. Adjust the other values if
desired - default values should work for most situations, but each site can
have it's own idiosyncrasies.
#. Run the R script. I'm not an R guy so I'm not sure how to do this other than
to open it in R studio, highlighting all the code, and hitting `Ctrl+Enter`.
This should create a bunch of `.csv` files containing the check data from
various sources. This is a good resource for perusal during processing, but
will be imbibed by hydrobot to for QC encoding.
#. Make sure your virtual environment is set up (see initial setup
instructions) and activate it. To activate, in your shell type the location
of the "Activate.ps1" script in the venv/Scripts folder, e.g.::
C:/Hydrobot/hydrobot0.7.8/Scripts/Activate.ps1
You can ensure it is active by typing `gcm python` and confirm that your
python interpreter (under "Source") is running from your venv folder.
#. Navigate to your batch folder in your shell. To navigate to the batch
folder, use the following "cd" command with your specific details::
cd "\\ares\Environmental Data Validation\measurement\site\batch\"
e.g. if you are doing water temperature for teachers college in batch 400,
your cd command would be::
cd "\\ares\Environmental Data Validation\Water Temperature\Manawatu at Teachers College\400"
#. Run the hydrobot processing script using streamlit (name changes slightly by
data source)::
streamlit run wt_script.py
#. If all goes well, the processing script will open a browser tab showing a
diagnostic dash for your site. Use this to identify issues in the site.
#. Optionally, modify the python script to solve some issues, like removing
erroneous check data points or deleting chunks of data, then rerun the
script.
#. Open the resulting processed.xml in manager, and copy it over to the hts
file found in the batch folder.
#. Modify the data in hilltop as needed
#. Open the WaterTemp_check_data.csv output by the R file in a spreadsheet and
copy into the hts batch file.
#. Copy to provisional automation when complete.
Credits
-------
This package was created with Cookiecutter_ and the `audreyr/cookiecutter-pypackage`_ project template. Furthermore,
Sam is a real champ with the coding and whatnot. Thanks Sam.
Aww thanks Nic. You also da man <3
.. _Cookiecutter: https://github.com/audreyr/cookiecutter
.. _`audreyr/cookiecutter-pypackage`: https://github.com/audreyr/cookiecutter-pypackage
Raw data
{
"_id": null,
"home_page": null,
"name": "hydrobot",
"maintainer": null,
"docs_url": null,
"requires_python": "==3.11.*",
"maintainer_email": null,
"keywords": "hydrology, automation, pandas, hilltop, hilltop-py, HorizonsRC",
"author": null,
"author_email": "Nic Mostert <nicolas.mostert@horizons.govt.nz>, Sam Irvine <sam.irvine@horizons.govt.nz>",
"download_url": "https://files.pythonhosted.org/packages/bc/43/d5d39111dd07c5ae0ee5858a894f1593914178ddf04c007647cea7de85c3/hydrobot-0.7.8.tar.gz",
"platform": null,
"description": "======================\r\nHydrobot\r\n======================\r\n\r\n\r\n.. image:: https://img.shields.io/pypi/v/hydrobot.svg\r\n :target: https://pypi.python.org/pypi/hydrobot\r\n\r\n.. image:: https://readthedocs.org/projects/hydrobot/badge/?version=latest\r\n :target: https://hydrobot.readthedocs.io/en/latest/?version=latest\r\n :alt: Documentation Status\r\n\r\nPython Package providing a suite of processing tools and utilities for Hilltop\r\nhydrological data.\r\n\r\n\r\n* Free software: GNU General Public License v3\r\n* Documentation: https://hydrobot.readthedocs.io.\r\n\r\n\r\nFeatures\r\n--------\r\n\r\n* Processes data downloaded from Hilltop Server\r\n* Uses annalist to record all changes to data\r\n* Capable of various automated processing techniques, including:\r\n\r\n * Clipping data\r\n * Removing spikes based on FBEWMA smoothing\r\n * Identifying and removing 'flatlining' data, where an instrument repeats\r\n it's last collected data point (NOTE: It's unclear if this actually\r\n happening.)\r\n * Identifying gaps and gap lengths and closing small gaps\r\n * Aggregating check data from various sources.\r\n * Quality coding data based on NEMS standards\r\n\r\n* Plotting data, including:\r\n\r\n * Processed data with quality codes\r\n * Comparing raw data to processed data\r\n * Showing all changes to the data\r\n * Visualizing check points from various sources.\r\n\r\nUsage (Alpha)\r\n-------------\r\n\r\nThe Alpha release of Hydrobot supports a \"hybrid\" workflow. This means that\r\nsome external tools are still required to do a full processing. Importantly,\r\nthe hybrid workflow relies on some R scripts to obtain check data from sources\r\nother than Hilltop. Further processing using Hilltop manager is also supported.\r\n\r\nNOTE: Hydrobot 0.7.8 does not support all NEMs data sources currently,\r\nbut more measurements will be supported in patches as the processing\r\nprogresses.\r\n\r\nInitial Setup (Repeat for each release)\r\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n\r\n#. Install a Python 3.11 interpreter (3.12 is not supported just yet).\r\n\r\n#. In your favourite shell (if you don't know what that is, use powershell -\r\n it's already installed on windows), create a new virtual environment using\r\n this python interpreter and name it \"hydrobot0.7.8\". It's important that\r\n this is stored somewhere locally. For example, it could be stored in a\r\n \"Hydrobot\" folder in the C: drive, which would need the command::\r\n\r\n python -m venv C:/Hydrobot/hydrobot0.7.8/\r\n cd C:/Hydrobot/hydrobot0.7.8/\r\n\r\n#. Activate this virtual environment. In powershell this should be something\r\n like::\r\n\r\n C:/Hydrobot/hydrobot0.7.8/Scripts/Activate.ps1\r\n\r\n#. With your venv active, install the latest version of Hydrobot using pip::\r\n\r\n pip install hydrobot\r\n\r\n#. Record which version of dependencies you have installed. The following pip\r\n freeze records which dependencies are installed by the hydrobot install\r\n process for if auditing/reprocessing is required later::\r\n\r\n pip freeze > dependencies.txt\r\n\r\n\r\nProcessing Steps\r\n^^^^^^^^^^^^^^^^\r\n\r\n#. Open Logsheet Loader. Fill it as normal, and note the start date of your\r\n processing period (i.e. end date of the previous period).\r\n\r\n#. Navigate to the data source and site folder, and create your batch folder.\r\n\r\n#. Copy all the hydrobot processing template files from the documents folder\r\n into your new batch folder, then rename with batch number and location code.\r\n The location of e.g. the water temperature template is::\r\n\r\n \\\\ares\\Environmental Data Validation\\Water Temperature\\Documents\\Hydrobot_template\\\r\n\r\n#. In your processing folder, open the `config.yaml` file and fill the fields\r\n `site`, `from_date`, `to_date`, `analyst_name`. Adjust the other values if\r\n desired - default values should work for most situations, but each site can\r\n have it's own idiosyncrasies.\r\n\r\n#. Run the R script. I'm not an R guy so I'm not sure how to do this other than\r\n to open it in R studio, highlighting all the code, and hitting `Ctrl+Enter`.\r\n This should create a bunch of `.csv` files containing the check data from\r\n various sources. This is a good resource for perusal during processing, but\r\n will be imbibed by hydrobot to for QC encoding.\r\n\r\n#. Make sure your virtual environment is set up (see initial setup\r\n instructions) and activate it. To activate, in your shell type the location\r\n of the \"Activate.ps1\" script in the venv/Scripts folder, e.g.::\r\n\r\n C:/Hydrobot/hydrobot0.7.8/Scripts/Activate.ps1\r\n\r\n You can ensure it is active by typing `gcm python` and confirm that your\r\n python interpreter (under \"Source\") is running from your venv folder.\r\n\r\n#. Navigate to your batch folder in your shell. To navigate to the batch\r\n folder, use the following \"cd\" command with your specific details::\r\n\r\n cd \"\\\\ares\\Environmental Data Validation\\measurement\\site\\batch\\\"\r\n\r\n e.g. if you are doing water temperature for teachers college in batch 400,\r\n your cd command would be::\r\n\r\n cd \"\\\\ares\\Environmental Data Validation\\Water Temperature\\Manawatu at Teachers College\\400\"\r\n\r\n#. Run the hydrobot processing script using streamlit (name changes slightly by\r\n data source)::\r\n\r\n streamlit run wt_script.py\r\n\r\n#. If all goes well, the processing script will open a browser tab showing a\r\n diagnostic dash for your site. Use this to identify issues in the site.\r\n\r\n#. Optionally, modify the python script to solve some issues, like removing\r\n erroneous check data points or deleting chunks of data, then rerun the\r\n script.\r\n\r\n#. Open the resulting processed.xml in manager, and copy it over to the hts\r\n file found in the batch folder.\r\n\r\n#. Modify the data in hilltop as needed\r\n\r\n#. Open the WaterTemp_check_data.csv output by the R file in a spreadsheet and\r\n copy into the hts batch file.\r\n\r\n#. Copy to provisional automation when complete.\r\n\r\nCredits\r\n-------\r\n\r\nThis package was created with Cookiecutter_ and the `audreyr/cookiecutter-pypackage`_ project template. Furthermore,\r\nSam is a real champ with the coding and whatnot. Thanks Sam.\r\n\r\nAww thanks Nic. You also da man <3\r\n\r\n.. _Cookiecutter: https://github.com/audreyr/cookiecutter\r\n.. _`audreyr/cookiecutter-pypackage`: https://github.com/audreyr/cookiecutter-pypackage\r\n",
"bugtrack_url": null,
"license": "GNU General Public License v3",
"summary": "A suite of processing tools for Hilltop hydrological data.",
"version": "0.7.8",
"project_urls": {
"Documentation": "https://hydrobot.readthedocs.io",
"Homepage": "https://github.com/HorizonsRC/hydrobot",
"Issues": "https://github.com/HorizonsRC/hydrobot/issues",
"Package": "https://pypi.org/project/hydrobot"
},
"split_keywords": [
"hydrology",
" automation",
" pandas",
" hilltop",
" hilltop-py",
" horizonsrc"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "1cbf072d4b061816aa3e90ad50376647cf084a42fe19e2b5b4fe6d6812545d0d",
"md5": "feb0863715a18c40a9f7eeb1edf08285",
"sha256": "3888b344ce79fa33c5060688c105971e018bcf1ee6a6012bcd00b9850a2f6c38"
},
"downloads": -1,
"filename": "hydrobot-0.7.8-py3-none-any.whl",
"has_sig": false,
"md5_digest": "feb0863715a18c40a9f7eeb1edf08285",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "==3.11.*",
"size": 67164,
"upload_time": "2024-10-30T20:21:43",
"upload_time_iso_8601": "2024-10-30T20:21:43.650313Z",
"url": "https://files.pythonhosted.org/packages/1c/bf/072d4b061816aa3e90ad50376647cf084a42fe19e2b5b4fe6d6812545d0d/hydrobot-0.7.8-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "bc43d5d39111dd07c5ae0ee5858a894f1593914178ddf04c007647cea7de85c3",
"md5": "11a2f4c435c1a44cf5a56007a67fb3b5",
"sha256": "db76a6b2b0f68f65656f8f28c5babc55f1ad12bbdec185735d80bf7f005c65b7"
},
"downloads": -1,
"filename": "hydrobot-0.7.8.tar.gz",
"has_sig": false,
"md5_digest": "11a2f4c435c1a44cf5a56007a67fb3b5",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "==3.11.*",
"size": 149261,
"upload_time": "2024-10-30T20:21:46",
"upload_time_iso_8601": "2024-10-30T20:21:46.042356Z",
"url": "https://files.pythonhosted.org/packages/bc/43/d5d39111dd07c5ae0ee5858a894f1593914178ddf04c007647cea7de85c3/hydrobot-0.7.8.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-10-30 20:21:46",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "HorizonsRC",
"github_project": "hydrobot",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "hydrobot"
}
JSON
