nz-flow-naturalisation
==================================
This git repository contains project code for the flow naturalisation procedure. The procedure has several modules for performing different tasks that ultimately combine for the naturalisation.
The base class (FlowNat) initialises the tool with a from_date, to_date, min_gaugings, input_sites, and output_path. This sets up and prepares a lot of datasets for the successive modules.
Modules:
- Querying and/or estimating flow at the input_sites
- Catchment delineation above the input_sites
- Selecting the upstream water abstraction sites from the catchment delineation
- Querying and Estimating water usage when the usage doesn't exist
- Flow naturalisation
Input parameters
----------------
The base class (FlowNat) initialises the tool with a from_date, to_date, min_gaugings, input_sites, rec_data_code, and output_path. This sets up and prepares a lot of datasets for the successive modules. If all of those input parameters are defined at initialisation, then all of the successive modules/methods will not require any other input.
.. Example scripts
.. ---------------
.. An example script can be found here: https://github.com/Data-to-Knowledge/FlowNaturalisation/blob/master/flownat/tests/utest_ash_2019-07-19.py.
.. The output_path will need to be modified for the specific user.
Background and package dependencies
-----------------------------------
The modules use several python packages for their procedures.
The catchment delineation module uses the python package gistools which has a catchment delineation function. This functions uses the REC stream network version 2 and the associated catchments for determining the catchments above specific points. The flow locations are used to delineate the upstream catchments. The upstream catchments are then used to select the WAPs that are within each catchment.
Not all flow locations have a continuous record from a recorder. Consequently, the flow sites with only gaugings need to be correlated to flow sites with (nearly) continuous recorders. This is done via the hydrolm package that uses ordinary least squares regressions of one or two recorders. The F statistic is used to determine the best regression.
Water usage data also needs to be estimated when it doesn't already exist. This was done by grouping the consents by catchment and use type and estimating the ratio of usage to allocation by month. These ratios were then applied at all consents without existing water usage data. This analysis was performed on a monthly scale.
General methodology
-------------------
Introduction
~~~~~~~~~~~~~~~
River flows are affected by many factors. These include climate, geology, vegetation, water abstractions, and others. To be able to appropriately allocate water for abstraction, river flows need to be estimated without the influence of existing water abstractions for the consideration of other important values associated with surface waters (e.g. environmental, cultural, recreational, etc.). The process of estimating a new flow series by removing the effects of upstream water abstractions is called naturalisation.
The naturalisation of river flows is the process of ‘adding back’ the surface water abstractions and hydraulically connected groundwater abstractions to a flow record. Naturalisation is a process to recreate the flow record if the upstream water abstractions did not take place.
The goal of this memo is to describe a new tool to accurately and efficiently naturalise surface water flows in Canterbury. This tool has been developed in Python as an installable package from pypi. The github repository can be found here: https://github.com/mullenkamp/nz-flow-naturalisation
Requirements
~~~~~~~~~~~~~
The requirements for the tool includes:
- Datasets available to Tethys
- Consistent with existing Environment Canterbury and Environment Southland methods
- Relatively simple
- Work for the entire record of any recorder or gauging site with sufficient data
- Reproducible
- Produce results quickly (a few minutes)
Methods and assumptions
~~~~~~~~~~~~~~~~~~~~~~~
The methods can be broken down into five modules:
- Querying and/or estimating flow at the input sites
- Catchment delineation above the input sites
- Selecting the upstream water abstraction sites from the catchment delineation
- Querying and Estimating water usage when the usage doesn't exist
- Flow naturalisation calculation
This method is a daily lumped model, which means all inputs and outputs are daily time series and assume that there is no lag time between flow and abstractions. The model also assumes that all abstractions that affect the downstream flow only come from the upstream delineated surface water catchment. Both of these assumptions are oversimplifications. The consequence of the first assumption is that stream depletion effects will occur sooner and end sooner than in real life. The second assumption will affect the seasonality lags due to groundwater storage and depletion from outside the catchment area.
Querying and/or estimating flow at the input sites
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
One of the input data required for the naturalisation is continuous flow records at particular locations prior to naturalisation. This usually comes in two forms: recorded continuous flow at a site with a recorder, or a site where manual flow measurements are taken but do not have a recorder.
If the site has a recorder, then the tool simply pulls the data out from the Tethys system and uses it directly as long as the flow record covers the requested date period for the naturalisation. If it does not cover the requested period, then manual flows for that site is used.
If manual flows for sites are used then a log-log regression to flow records with sufficient length and a maximum distance (default is 50 km) is used. The best correlation is selected based on the highest F value and a new continuous flow record is estimated from the correlation.
All flows (and other time series data used in this naturalisation) are daily means. Currently, the regressions use the full range of flows.
Catchment delineation above the input sites
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The catchments upstream of each flow site needs to be delineated to determine the water abstraction points (WAPs) that affect the flow record. Currently, the stream and associated catchments layer used for this delineation is the River Environmental Classification (REC) river network and watersheds.
The nearest river segments to flow sites are identified, then upstream river network delineation is performed and the catchments are extracted and merged to have a single catchment above each flow site. The delineation is at the network resolution of the REC dataset, which means that the delineation will likely not be directly at the site. Rather, it will be slightly downstream of the site and consequently the catchment will be of a slightly larger area than it would be realistically. The benefit of using a pregenerated river network and catchments is the significant reduction in the processing time and complexity of the tool.
Selecting the upstream water abstraction sites from the catchment delineation
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Once the catchments have been delineated above all flow sites, then the WAPs that are considered stream depleting are selected and assigned to each catchment/flow site. The WAPs and the associated consents have already been preassigned whether they are stream depleting and by how much using an ECan implementation of the Theis method for in Smith, 2015. Both the WAPs and the associated stream depletion rates are extracted from the regional council's systems.
Querying and Estimating water usage when the usage doesn't exist
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
This is probably the most complicated module of the entire naturalisation process and this section will not go into exhaustive detail about it’s implementation. But generally, this involves querying the existing usage data associated with all WAPs/consents found in the prior module, then estimating the usage where it does not exist.
This module primarily uses the nz-allo-usage-tools python package by processing allocation and water usage obtained from the regional councils. This tool pulls from a summary of the data specifically for stream depleting consents that are considered "consumptive". Given the limitations of the source databases, any consents that have "temporary wavers" or have conditions that are shared between multiple consents (e.g. non-concurrence) are ignored (because the info is not readily available). The minimum flow restrictions were accounted for in the consented rates and volumes. If a consent was restricted from taking water for some period of the year, that volume was deducted from the values used in the processing. These minimum flow restricted rates and volumes for the consents are split proportionally across their WAPs (if there are more than one WAP on a consent).
First, all of the existing water usage data for the upstream WAPs are extracted. Given that the raw water usage data has not had much quality controls, three filters are used to ensure that the usage values are “realistic”. The first are the usage/allocation ratios at the daily, monthly, and yearly scales. The defaults are 2, 3, and 2 respectively. These were found to be generous enough to retain real-looking usage values and exclude erroneous ones. Though it is still possible to get false positives and negatives using these filters.
Once the data has been filtered, abstraction/allocation ratios were calculated and lumped by month of the year, catchment, and use type. These ratios were then applied to the WAPs that did not have usage data to estimate the usage data by month. The results of the querying and estimating of the usage data is that all consented WAPs that are considered both stream depleting and consumptive have usage data.
If usage data already exists, then the daily values are used, if not then the monthly estimates are resampled to daily using the pchip interpolation method in Python. This method is used to make the result more smooth and consequently more natural looking.
Flow naturalisation calculation
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Once the usage has been estimated for all upstream catchments and the flow record has also been estimated, then its a simple matter of adding the two daily time series together to get the final naturalised flow record estimate.
Potential improvements
----------------------
Regression options
~~~~~~~~~~~~~~~~~~~
Currently, the flow regressions use the entire flow record for the correlation to create a continuous flow record. This may be useful for some purposes, but other purposes (e.g. estimating low flow stats) using the entire record may not be appropriate. Additional options to constrain the flow range of the regression (e.g. 1.5 * median) could be useful.
Extend flow recorder data
~~~~~~~~~~~~~~~~~~~~~~~~~
At the moment, if the requested from and to dates extend beyond the record of a flow recorder site, then it uses that gaugings to create a new longer record. It would probably be better to extend the existing flow recorder record rather than using gaugings.
This option is already possible in hydrolm (which is used in the package). It just needs to be integrated into this package.
Finer resolution catchment delineation
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Currently, the catchment delineation uses the REC v2.4 and consequently it's resolution. If finer resolution delineation is required, then a different dataset is needed. At the same time, I don't want to increase the run time significantly higher than it already is.
There are a couple options:
We could pre-generate all catchments above all flow measurement sites. This would still require a rivers layer and associated DEM.
This already used as a shortcut in the existing implementation, but it uses the REC. Combining the OSM waterways and the LINZ 8m DEM could do this.
The other alternative if any arbitrary point along a river needs to be delineated on-the-fly would be to do something similar to the first option, but only create an OSM-like network that extends all the way up to all 2nd order streams. Using only the streams layer, all WAPs upstream can be selected based on a nearest neighbor query to the stream network rather than having catchments. The only downside of not using polygon catchments is that there is a possibility (though VERY slim) that WAPs might be in a different catchment even if a different river is technically closer. I do think in practice this won't be an issue if we use all 2nd order and higher streams.
Transient abstractions
~~~~~~~~~~~~~~~~~~~~~~~
Currently abstractions regardless of the distance to the streams are instantaneous abstractions to the stream on that daily. This is of course not true. Lag times would need to be assigned based on the distance to the stream to be more appropriate.
Graphical user interface
~~~~~~~~~~~~~~~~~~~~~~~~~
A GUI on top of the tool would make using it easier for non-programmers.
Raw data
{
"_id": null,
"home_page": null,
"name": "nz-flow-naturalisation",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.9",
"maintainer_email": null,
"keywords": "flow, naturalisation",
"author": null,
"author_email": "Mike Kittridge <mullenkamp1@gmail.com>",
"download_url": "https://files.pythonhosted.org/packages/59/d4/f723a064e508acf7c316a0e150e12bbf7ec942028f1becd0d147235f6f03/nz_flow_naturalisation-0.2.0.tar.gz",
"platform": null,
"description": "nz-flow-naturalisation\n==================================\n\nThis git repository contains project code for the flow naturalisation procedure. The procedure has several modules for performing different tasks that ultimately combine for the naturalisation.\n\nThe base class (FlowNat) initialises the tool with a from_date, to_date, min_gaugings, input_sites, and output_path. This sets up and prepares a lot of datasets for the successive modules.\n\nModules:\n - Querying and/or estimating flow at the input_sites\n - Catchment delineation above the input_sites\n - Selecting the upstream water abstraction sites from the catchment delineation\n - Querying and Estimating water usage when the usage doesn't exist\n - Flow naturalisation\n\nInput parameters\n----------------\nThe base class (FlowNat) initialises the tool with a from_date, to_date, min_gaugings, input_sites, rec_data_code, and output_path. This sets up and prepares a lot of datasets for the successive modules. If all of those input parameters are defined at initialisation, then all of the successive modules/methods will not require any other input.\n\n.. Example scripts\n.. ---------------\n.. An example script can be found here: https://github.com/Data-to-Knowledge/FlowNaturalisation/blob/master/flownat/tests/utest_ash_2019-07-19.py.\n.. The output_path will need to be modified for the specific user.\n\nBackground and package dependencies\n-----------------------------------\nThe modules use several python packages for their procedures.\n\nThe catchment delineation module uses the python package gistools which has a catchment delineation function. This functions uses the REC stream network version 2 and the associated catchments for determining the catchments above specific points. The flow locations are used to delineate the upstream catchments. The upstream catchments are then used to select the WAPs that are within each catchment.\n\nNot all flow locations have a continuous record from a recorder. Consequently, the flow sites with only gaugings need to be correlated to flow sites with (nearly) continuous recorders. This is done via the hydrolm package that uses ordinary least squares regressions of one or two recorders. The F statistic is used to determine the best regression.\n\nWater usage data also needs to be estimated when it doesn't already exist. This was done by grouping the consents by catchment and use type and estimating the ratio of usage to allocation by month. These ratios were then applied at all consents without existing water usage data. This analysis was performed on a monthly scale.\n\nGeneral methodology\n-------------------\n\nIntroduction\n~~~~~~~~~~~~~~~\nRiver flows are affected by many factors. These include climate, geology, vegetation, water abstractions, and others. To be able to appropriately allocate water for abstraction, river flows need to be estimated without the influence of existing water abstractions for the consideration of other important values associated with surface waters (e.g. environmental, cultural, recreational, etc.). The process of estimating a new flow series by removing the effects of upstream water abstractions is called naturalisation.\n\nThe naturalisation of river flows is the process of \u2018adding back\u2019 the surface water abstractions and hydraulically connected groundwater abstractions to a flow record. Naturalisation is a process to recreate the flow record if the upstream water abstractions did not take place.\nThe goal of this memo is to describe a new tool to accurately and efficiently naturalise surface water flows in Canterbury. This tool has been developed in Python as an installable package from pypi. The github repository can be found here: https://github.com/mullenkamp/nz-flow-naturalisation\n\nRequirements\n~~~~~~~~~~~~~\nThe requirements for the tool includes:\n\n- Datasets available to Tethys\n- Consistent with existing Environment Canterbury and Environment Southland methods\n- Relatively simple\n- Work for the entire record of any recorder or gauging site with sufficient data\n- Reproducible\n- Produce results quickly (a few minutes)\n\nMethods and assumptions\n~~~~~~~~~~~~~~~~~~~~~~~\nThe methods can be broken down into five modules:\n\n- Querying and/or estimating flow at the input sites\n- Catchment delineation above the input sites\n- Selecting the upstream water abstraction sites from the catchment delineation\n- Querying and Estimating water usage when the usage doesn't exist\n- Flow naturalisation calculation\n\nThis method is a daily lumped model, which means all inputs and outputs are daily time series and assume that there is no lag time between flow and abstractions. The model also assumes that all abstractions that affect the downstream flow only come from the upstream delineated surface water catchment. Both of these assumptions are oversimplifications. The consequence of the first assumption is that stream depletion effects will occur sooner and end sooner than in real life. The second assumption will affect the seasonality lags due to groundwater storage and depletion from outside the catchment area.\n\nQuerying and/or estimating flow at the input sites\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nOne of the input data required for the naturalisation is continuous flow records at particular locations prior to naturalisation. This usually comes in two forms: recorded continuous flow at a site with a recorder, or a site where manual flow measurements are taken but do not have a recorder.\n\nIf the site has a recorder, then the tool simply pulls the data out from the Tethys system and uses it directly as long as the flow record covers the requested date period for the naturalisation. If it does not cover the requested period, then manual flows for that site is used.\nIf manual flows for sites are used then a log-log regression to flow records with sufficient length and a maximum distance (default is 50 km) is used. The best correlation is selected based on the highest F value and a new continuous flow record is estimated from the correlation.\nAll flows (and other time series data used in this naturalisation) are daily means. Currently, the regressions use the full range of flows.\n\nCatchment delineation above the input sites\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nThe catchments upstream of each flow site needs to be delineated to determine the water abstraction points (WAPs) that affect the flow record. Currently, the stream and associated catchments layer used for this delineation is the River Environmental Classification (REC) river network and watersheds.\n\nThe nearest river segments to flow sites are identified, then upstream river network delineation is performed and the catchments are extracted and merged to have a single catchment above each flow site. The delineation is at the network resolution of the REC dataset, which means that the delineation will likely not be directly at the site. Rather, it will be slightly downstream of the site and consequently the catchment will be of a slightly larger area than it would be realistically. The benefit of using a pregenerated river network and catchments is the significant reduction in the processing time and complexity of the tool.\n\nSelecting the upstream water abstraction sites from the catchment delineation\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nOnce the catchments have been delineated above all flow sites, then the WAPs that are considered stream depleting are selected and assigned to each catchment/flow site. The WAPs and the associated consents have already been preassigned whether they are stream depleting and by how much using an ECan implementation of the Theis method for in Smith, 2015. Both the WAPs and the associated stream depletion rates are extracted from the regional council's systems.\n\nQuerying and Estimating water usage when the usage doesn't exist\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nThis is probably the most complicated module of the entire naturalisation process and this section will not go into exhaustive detail about it\u2019s implementation. But generally, this involves querying the existing usage data associated with all WAPs/consents found in the prior module, then estimating the usage where it does not exist.\n\nThis module primarily uses the nz-allo-usage-tools python package by processing allocation and water usage obtained from the regional councils. This tool pulls from a summary of the data specifically for stream depleting consents that are considered \"consumptive\". Given the limitations of the source databases, any consents that have \"temporary wavers\" or have conditions that are shared between multiple consents (e.g. non-concurrence) are ignored (because the info is not readily available). The minimum flow restrictions were accounted for in the consented rates and volumes. If a consent was restricted from taking water for some period of the year, that volume was deducted from the values used in the processing. These minimum flow restricted rates and volumes for the consents are split proportionally across their WAPs (if there are more than one WAP on a consent).\n\nFirst, all of the existing water usage data for the upstream WAPs are extracted. Given that the raw water usage data has not had much quality controls, three filters are used to ensure that the usage values are \u201crealistic\u201d. The first are the usage/allocation ratios at the daily, monthly, and yearly scales. The defaults are 2, 3, and 2 respectively. These were found to be generous enough to retain real-looking usage values and exclude erroneous ones. Though it is still possible to get false positives and negatives using these filters.\nOnce the data has been filtered, abstraction/allocation ratios were calculated and lumped by month of the year, catchment, and use type. These ratios were then applied to the WAPs that did not have usage data to estimate the usage data by month. The results of the querying and estimating of the usage data is that all consented WAPs that are considered both stream depleting and consumptive have usage data.\nIf usage data already exists, then the daily values are used, if not then the monthly estimates are resampled to daily using the pchip interpolation method in Python. This method is used to make the result more smooth and consequently more natural looking.\n\nFlow naturalisation calculation\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nOnce the usage has been estimated for all upstream catchments and the flow record has also been estimated, then its a simple matter of adding the two daily time series together to get the final naturalised flow record estimate.\n\nPotential improvements\n----------------------\n\nRegression options\n~~~~~~~~~~~~~~~~~~~\nCurrently, the flow regressions use the entire flow record for the correlation to create a continuous flow record. This may be useful for some purposes, but other purposes (e.g. estimating low flow stats) using the entire record may not be appropriate. Additional options to constrain the flow range of the regression (e.g. 1.5 * median) could be useful.\n\nExtend flow recorder data\n~~~~~~~~~~~~~~~~~~~~~~~~~\nAt the moment, if the requested from and to dates extend beyond the record of a flow recorder site, then it uses that gaugings to create a new longer record. It would probably be better to extend the existing flow recorder record rather than using gaugings.\n\nThis option is already possible in hydrolm (which is used in the package). It just needs to be integrated into this package.\n\nFiner resolution catchment delineation\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\nCurrently, the catchment delineation uses the REC v2.4 and consequently it's resolution. If finer resolution delineation is required, then a different dataset is needed. At the same time, I don't want to increase the run time significantly higher than it already is.\n\nThere are a couple options:\n\nWe could pre-generate all catchments above all flow measurement sites. This would still require a rivers layer and associated DEM.\nThis already used as a shortcut in the existing implementation, but it uses the REC. Combining the OSM waterways and the LINZ 8m DEM could do this.\n\nThe other alternative if any arbitrary point along a river needs to be delineated on-the-fly would be to do something similar to the first option, but only create an OSM-like network that extends all the way up to all 2nd order streams. Using only the streams layer, all WAPs upstream can be selected based on a nearest neighbor query to the stream network rather than having catchments. The only downside of not using polygon catchments is that there is a possibility (though VERY slim) that WAPs might be in a different catchment even if a different river is technically closer. I do think in practice this won't be an issue if we use all 2nd order and higher streams.\n\nTransient abstractions\n~~~~~~~~~~~~~~~~~~~~~~~\nCurrently abstractions regardless of the distance to the streams are instantaneous abstractions to the stream on that daily. This is of course not true. Lag times would need to be assigned based on the distance to the stream to be more appropriate.\n\nGraphical user interface\n~~~~~~~~~~~~~~~~~~~~~~~~~\nA GUI on top of the tool would make using it easier for non-programmers.\n",
"bugtrack_url": null,
"license": "Apache-2.0",
"summary": "Tools to perform flow naturalisation in New Zealand",
"version": "0.2.0",
"project_urls": {
"Homepage": "https://github.com/mullenkamp/nz-flow-naturalisation"
},
"split_keywords": [
"flow",
" naturalisation"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "a916805ee4a136688cd5e0dcf074c09c99b24693548cf0aac83f72c9effe215b",
"md5": "29f685d8c9b22faf8bc5ee935e82f9d1",
"sha256": "3b181f7b0437f6af5521f7d534a2263eb6efdab09080458f27b8119ef4e1c85f"
},
"downloads": -1,
"filename": "nz_flow_naturalisation-0.2.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "29f685d8c9b22faf8bc5ee935e82f9d1",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9",
"size": 28506,
"upload_time": "2024-12-11T03:53:11",
"upload_time_iso_8601": "2024-12-11T03:53:11.842919Z",
"url": "https://files.pythonhosted.org/packages/a9/16/805ee4a136688cd5e0dcf074c09c99b24693548cf0aac83f72c9effe215b/nz_flow_naturalisation-0.2.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "59d4f723a064e508acf7c316a0e150e12bbf7ec942028f1becd0d147235f6f03",
"md5": "2f7246220f49769b0d40da89a6cf9e81",
"sha256": "798e5810e5c6a17d4ae0d7432a4c6db39996132a19787c5795e1ef21ea67ae73"
},
"downloads": -1,
"filename": "nz_flow_naturalisation-0.2.0.tar.gz",
"has_sig": false,
"md5_digest": "2f7246220f49769b0d40da89a6cf9e81",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9",
"size": 21983,
"upload_time": "2024-12-11T03:53:08",
"upload_time_iso_8601": "2024-12-11T03:53:08.490056Z",
"url": "https://files.pythonhosted.org/packages/59/d4/f723a064e508acf7c316a0e150e12bbf7ec942028f1becd0d147235f6f03/nz_flow_naturalisation-0.2.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-12-11 03:53:08",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "mullenkamp",
"github_project": "nz-flow-naturalisation",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "nz-flow-naturalisation"
}
JSON
