### Context
* Created: April 2023
* Author: Renier Kramer, renier.kramer@hdsr.nl
* Maintainer: Roger de Crook, roger.de.crook@hdsr.nl
* Python version: >=3.7
[hkvfewspy]: https://github.com/HKV-products-services/hkvfewspy
[fewspy]: https://github.com/d2hydro/fewspy
[MIT]: https://github.com/hdsr-mid/hdsr_fewspy/blob/main/LICENSE.txt
[Deltares FEWS PI]: https://publicwiki.deltares.nl/display/FEWSDOC/FEWS+PI+REST+Web+Service
[issues page]: https://github.com/hdsr-mid/hdsr_fewspy/issues
[github personal token]: https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token
[releases]: https://pypi.org/project/hdsr-fewspy/#history
[flag details]: https://publicwiki.deltares.nl/display/FEWSDOC/D+Time+Series+Flags
[user and auth settings]: https://github.com/hdsr-mid/hdsr_fewspy_auth
### Description
A python project to request data (locations, time-series, etc.) from FEWS-WIS (waterquantity) and FEWS-EFCIS (waterquality). \
This project only works in HDSR's internal network, so within the VDI or virtual machine within the network. \
It combines the best from two existing fewspy projects: [fewspy] and [hkvfewspy]. On top of that it adds
client-side authentication, authorisation, and throttling.
The latter is to minimize request load on HDSR's internal FEWS instances. \
For detailed info on requesting FEWS APIs visit the [Deltares FEWS wiki][Deltares FEWS PI].
Hdsr_fewspy API supports 9 different API calls that can return 6 different output formats:
1. xml_file_in_download_dir: The xml response is written to a .xml file in your download_dir
2. json_file_in_download_dir: The json response is written to a .json file in your download_dir
3. csv_file_in_download_dir: The json response is converted to csv and written to a .csv file in your download_dir
4. xml_response_in_memory: the xml response is returned memory meaning you get a list with one or more responses
5. json_response_in_memory: the json response is returned memory meaning you get a list with one or more responses
6. pandas_dataframe_in_memory: the json response is converted to a pandas dataframe meaning you get one dataframe
API call | Supported outputs | Notes |
------------------------------|-------------------|---------------------------------------------------------------------------------------------------------|
1 get_parameters | 4, 5, 6 | Returns 1 object (xml/json response or dataframe) |
2 get_filters | 4, 5 | Returns 1 object (xml/json response) |
3 get_locations | 4, 5 | Returns 1 object (xml/json response) |
4 get_qualifiers | 4, 6 | Returns 1 object (xml response or dataframe) |
5 get_timezone_id | 4, 5 | Returns 1 object (xml/json response) |
6 get_samples | 4 | Returns 1 object (xml response) |
7 get_time_series_single | 4, 5, 6 | Returns 1 dataframe or a list >=1 xml/json responses |
8 get_time_series_multi | 1, 2, 3 | Returns a list with downloaded files (1 .csv or >=1 .xml/.json per unique location_parameter_qualifier) |
9 get_time_series_statistics | 4, 5 | Returns 1 object (xml/json response) |
###### DefaultPiSettingsChoices:
Several predefined pi_settings exists for point data and for area (e.g. averaged all points within an area). We mainly distinguish three levels of data:
- raw: raw measurements from field stations. Contains valid and invalid data but lags little time with field measurements.
- work: this data is being validated by a HDSR person (data validator CAW). This data might change every day.
- validated: data without invalid data. Note that this data is published months after the actual measurement.
The names of the pi_settings are (see class DefaultPiSettingsChoices):
- wis_production_point_raw
- wis_production_point_work
- wis_production_point_validated
- wis_production_area_soilmoisture
- wis_production_area_precipitation_wiwb
- wis_production_area_precipitation_radarcorrection
- wis_production_area_evaporation_wiwb_satdata
- wis_production_area_evaporation_waterwatch
- efcis_production_point_fysische_chemie
- efcis_production_point_biologie
### Usage
Below you find 10 examples for the 9 different requests. In hdsr_fewspy/examples/ you also find code to download
discharge (point), soil moisture (area), evaporation (area), and precipitation (area) time-series.
#### Preparation
1. Only once needed: ensure you have a github account with a GITHUB_PERSONAL_ACCESS_TOKEN. Read [github_personal_access_token](#github_personal_access_token) below to
know what to do with this token.
2. Only once needed: ensure your github username is registered in [user and auth settings] permissions.csv file. If
not, ask the maintainer of hdsr_fewspy to fix this.
3. You can create a hdsr_fewspy API in two ways (the first dominates the second). We advise to use the second:
- with API argument 'github_personal_access_token', thus ```api = hdsr_fewspy.Api(<github_personal_access_token>)```.
- or with API argument 'secrets_env_path' (defaults to 'G:/secrets.env'), thus ```api = hdsr_fewspy.Api(<your_secrets_env_path>)```.
The .env file must have a row:
```
GITHUB_PERSONAL_ACCESS_TOKEN=<your_github_token>
```
4. Only once per project: install hdsr_fewspy dependency:
```
pip install hdsr-fewspy
# or
conda install hdsr-fewspy --channel hdsr-mid
```
5. Example simple 'create API instance':
```
import hdsr_fewspy
api = hdsr_fewspy.Api()
```
6. Example sophisticated 'create API instance':
```
import hdsr_fewspy
# Optionally, you can specify several API arguments:
# github_email: str,
# github_personal_access_token: str
# secrets_env_path: str or pathlib.Path
# pi_settings: hdsr_fewspy.PiSettings
# output_directory_root: str or pathlib.path
# option 1
# For example in case of pi_settings, you can use predefined settings, see topic 'DefaultPiSettingsChoices' above
# To list all DefaultPiSettingsChoices:
hdsr_fewspy.DefaultPiSettingsChoices.get_all(
# For fews-wis api, use a DefaultPiSettingsChoices that starts with 'wis_', for example:
api = hdsr_fewspy.Api(pi_settings=hdsr_fewspy.DefaultPiSettingsChoices.wis_production_point_work)
# For fews-efcis api, use a DefaultPiSettingsChoices that starts with 'efics_', for example:
api = hdsr_fewspy.Api(pi_settings=hdsr_fewspy.DefaultPiSettingsChoices.efcis_production_point_biologie)
# option 2
# Or create your own pi_settings:
custom_settings = hdsr_fewspy.PiSettings(
settings_name="does not matter blabla",
document_version=1.25,
ssl_verify=True,
domain="localhost",
port=8080,
service="FewsWebServices",
filter_id="INTERNAL-API",
module_instance_ids="WerkFilter",
time_zone=hdsr_fewspy.TimeZoneChoices.eu_amsterdam.value, # = 1.0 (only affects get_time_series dataframe and csv)
)
api = hdsr_fewspy.Api(pi_settings=custom_settings)
# option 3
# In case you want to download responses to file, then you need to specify an output_directory_root
# The files will be downloaded in a subdir: output_directory_root/hdsr_fewspy_<datetime>/<files_will_be_downloaded_here>
api = hdsr_fewspy.Api(output_directory_root=<path_to_a_dir>)
```
#### Examples API calls
###### Below you see 9 examples using api option 3 above.
###### Moreover, in hdsr_fewspy/examples/ you find examples for point and area downloads.
1. get_parameters
```
df = api.get_parameters(output_choice=hdsr_fewspy.OutputChoices.pandas_dataframe_in_memory)
# id name parameter_type unit display_unit uses_datum parameter_group
# ---------------------------------------------------------------------------------------------------------
# BG.b.0 Oppervlaktebegroeiing [%] - noneq instantaneous % % False Begroeiingsgraad
# BG.fd.0 Flab en draadwieren [%] - noneq instantaneous % % False Begroeiingsgraad
# BG.ka.0 Algen-/kroosbedekking [%] - noneq instantaneous % % False Begroeiingsgraad
# ...etc...
```
2. get_filters
```
response = api.get_filters(output_choice=hdsr_fewspy.OutputChoices.json_response_in_memory)
response.json()
# {
# "version": "1.25",
# "filters": [
# {
# "id": "INTERNAL-API",
# "name": "INTERNAL-API",
# "child": [{"id": "INTERNAL-API.RUWMET", "name": "Ruwe metingen (punt)"
# ...etc...
```
3. get_locations
```
gdf = get_locations(output_choice=hdsr_fewspy.OutputChoices.pandas_dataframe_in_memory, show_attributes=True)
# location_id description short_name lat lon x y z parent_location_id geometry attributes
# -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
# beg_062 BEGROEIINGSMEETPUNT beg_062-LR_13_xruim 52.58907339700342 5.1718081593021505 140251.0 460118.0 0.0 NaN POINT (140251.000 460118.000) [{'name': 'LOC_NAME', 'type': 'text', 'id': 'LOC_NAME', 'value': 'beg_062 ...etc...]
# beg_084 BEGROEIINGSMEETPUNT beg_084-LR_17_xruim 52.06261306007392 5.12600382893812 137088.0 452734.0 0.0 NaN POINT (137088.000 452734.000) [{'name': 'LOC_NAME', 'type': 'text', 'id': 'LOC_NAME', 'value': 'beg_084 ...etc...]
# beg_102 BEGROEIINGSMEETPUNT beg_102-KR_9_xruim 52.07249358678008 5.215531005948442 143230.0 453815.0 0.0 NaN POINT (143230.000 453815.000) [{'name': 'LOC_NAME', 'type': 'text', 'id': 'LOC_NAME', 'value': 'beg_102 ...etc...]
# ...etc...
```
4. get_qualifiers
```
df = fixture_api_sa_no_download_dir.get_qualifiers(output_choice=hdsr_fewspy.OutputChoices.pandas_dataframe_in_memory)
# id name group_id
# -----------------------------------
# ergkrap erg krap (max 10%) None
# krap krap (max 30%) None
# normaal normaal (max 50%) None
# ruim ruim (max 70%) None
# ...etc...
```
5. get_timezone_id
```
response = api.get_timezone_id(output_choice=hdsr_fewspy.OutputChoices.json_response_in_memory)
# verify response
assert response.text == "GMT"
assert TimeZoneChoices.get_tz_float(value="GMT") == TimeZoneChoices.gmt.value == 0.0
```
6. get_samples
```
# Note: only FEWS-EFCIS contains sample data
api = hdsr_fewspy.Api(pi_settings=hdsr_fewspy.DefaultPiSettingsChoices.efcis_production_point_biologie)
response = api.get_samples(output_choice=hdsr_fewspy.OutputChoices.xml_response_in_memory)
print(response.text)
# <Samples xmlns="http://www.wldelft.nl/fews/PI" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:fs="http://www.wldelft.nl/fews/fs" xmlns:qualifierId="http://www.wldelft.nl/fews/qualifierId" xsi:schemaLocation="http://www.wldelft.nl/fews/PI https://fewsdocs.deltares.nl/schemas/version1.0/pi-schemas/pi_samples.xsd" version="1.34">
# <timeZone>0.0</timeZone>
# <sample id="2020_00090e">
# <header>
# <moduleInstanceId>Import_EFCIS_dump</moduleInstanceId>
# <locationId>20924</locationId>
# <sampleDate date="2020-01-09" time="09:57:00"/>
# <lat>51.98814213752072</lat>
# <lon>5.243994665879352</lon>
# <x>145163.0</x>
# <y>444426.0</y>
# </header>
# <properties>
# <string key="activiteit" value="Automatische FTP import"/>
# <string key="namespace" value="NL14"/>
# </properties>
# <table>
# <row parameterId="WNS1042" qualifierId:q1="WNS1042" qualifierId:q2="GH_34" qualifierId:q3="HH_492" qualifierId:q4="ACO_39" qualifierId:q5="MCO_39" qualifierId:q6="PC_1569" flag="0" value="0.71" unit="mg/l"/>
# <row parameterId="WNS1078" qualifierId:q1="WNS1078" qualifierId:q2="GH_34" qualifierId:q3="HH_492" qualifierId:q4="ACO_39" qualifierId:q5="MCO_39" qualifierId:q6="PC_1582" flag="0" value="0.01" detection="<" unit="ug/l"/>
# <row parameterId="WNS1085" qualifierId:q1="WNS1085" qualifierId:q2="GH_34" qualifierId:q3="HH_492" qualifierId:q4="ACO_39" qualifierId:q5="MCO_39" qualifierId:q6="PC_1584" flag="0" value="0.01" detection="<" unit="ug/l"/>
# <row parameterId="WNS1230" qualifierId:q1="WNS1230" qualifierId:q2="GH_16" qualifierId:q3="HH_492" qualifierId:q4="ACO_39" qualifierId:q5="MCO_39" qualifierId:q6="PC_1721" flag="0" value="0" unit="DIMSLS"/>
# ...
# <row parameterId="WNS742" qualifierId:q1="WNS742" qualifierId:q2="GH_34" qualifierId:q3="HH_492" qualifierId:q4="ACO_39" qualifierId:q5="MCO_39" qualifierId:q6="PC_1248" flag="0" value="3" unit="ug/l"/>
# <row parameterId="WNS814" qualifierId:q1="WNS814" qualifierId:q2="GH_34" qualifierId:q3="HH_492" qualifierId:q4="ACO_39" qualifierId:q5="MCO_39" qualifierId:q6="PC_1257" flag="0" value="0.01" detection="<" unit="ug/l"/>
# <row parameterId="WNS8372" qualifierId:q1="WNS8372" qualifierId:q2="GH_20" qualifierId:q3="HH_492" qualifierId:q4="ACO_39" qualifierId:q5="MCO_39" qualifierId:q6="PC_1725" flag="0" value="5" detection="<" unit="%"/>
# <row parameterId="WNS9756" qualifierId:q1="WNS9756" qualifierId:q2="GH_191" qualifierId:q3="HH_492" qualifierId:q4="ACO_39" qualifierId:q5="MCO_39" flag="0" value="100" unit="-"/>
# </table>
# </sample>
# </Samples>
```
7. get_time_series_single
[click here for more info on flags][flag details]
```
# Single means: use max 1 location_id and/or parameter_id and/or qualifier_id. One large call can result in multiple
# small calls and therefore multiple responses. If your output_choice is json/xml in memory, then you get a list with
# >=1 responses. Arguments 'flag_threshold' and 'drop_missing_values' have no effect.
responses = api.get_time_series_single(
location_id = "OW433001",
parameter_id = "H.G.0",
start_time = "2012-01-01T00:00:00Z", # or as datetime.datetime(year=2012, month=1, day=1)
end_time = "2012-01-02T00:00:00Z", # or as datetime.datetime(year=2012, month=1, day=2)
output_choice = hdsr_fewspy.OutputChoices.xml_response_in_memory
)
print(responses[0].text)
# <?xml version="1.0" encoding="UTF-8"?>
# <TimeSeries xmlns="http://www.wldelft.nl/fews/PI" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews/PI http://fews.wldelft.nl/schemas/version1.0/pi-schemas/pi_timeseries.xsd" version="1.25" xmlns:fs="http://www.wldelft.nl/fews/fs">
# <timeZone>0.0</timeZone>
# <series>
# <header>
# <type>instantaneous</type>
# <moduleInstanceId>WerkFilter</moduleInstanceId>
# <locationId>OW433001</locationId>
# <parameterId>H.G.0</parameterId>
# <timeStep unit="nonequidistant"/>
# <startDate date="2012-01-01" time="00:00:00"/>
# <endDate date="2012-01-02" time="00:00:00"/>
# <missVal>-999.0</missVal>
# <stationName>HAANWIJKERSLUIS_4330-w_Leidsche Rijn</stationName>
# <lat>52.08992726570302 asdf renier</lat>
# <lon>4.9547458967486095</lon>
# <x>125362.0</x>
# <y>455829.0</y>
# <z>-0.18</z>
# <units>mNAP</units>
# </header>
# <event date="2012-01-01" time="00:15:00" value="-0.35" flag="0" fs:PRIMAIR="OK" fs:VISUEEL="OK"/>
# <event date="2012-01-01" time="00:45:00" value="-0.36" flag="0" fs:PRIMAIR="OK" fs:VISUEEL="OK"/>
# <event date="2012-01-01" time="02:30:00" value="-0.37" flag="0" fs:PRIMAIR="OK" fs:VISUEEL="OK"/>
# <event date="2012-01-01" time="02:31:17" value="-0.38" flag="0" fs:PRIMAIR="OK" fs:VISUEEL="OK"/>
# <event date="2012-01-01" time="03:15:00" value="-0.39" flag="0" fs:PRIMAIR="OK" fs:VISUEEL="OK"/>
# ...etc..
# If your output_choice is dataframe, then all responses are collected in one dataframe. Arguments 'flag_threshold'
# and 'drop_missing_values' do have effect.
# Note that argument only_value_and_flag=True returns timeseries with {timestamp, value, flag}.
# This only affects output_choices dataframe and csv.
# only_value_and_flag=False returns also fields like {comment, etc}.
df = api.get_time_series_single(
location_id = "OW433001",
parameter_id = "H.G.0",
qualifier_id = "", # defaults to ""
start_time = "2012-01-01T00:00:00Z", # or as datetime.datetime(year=2012, month=1, day=1)
end_time = "2012-01-02T00:00:00Z", # or as datetime.datetime(year=2012, month=1, day=2)
drop_missing_values = True,
flag_threshold = 6, # all flags 6 and higher are removed from dataframe
thinning = None, # [integer: ms/pixel] defaults to None. See Deltares wiki (link above)
omit_empty_time_series = True, # [bool] defaults to True
only_value_and_flag = True, # [bool] defaults to True
output_choice = hdsr_fewspy.OutputChoices.pandas_dataframe_in_memory,
)
```
8. get_time_series_multi
[click here for more info on flags][flag details]
```
# Multi means: use >=1 location_id and/or parameter_id and/or qualifier_id. The api call below results in 4 unique
# location_parameter_qualifier combinations: OW433001_hg0, OW433001_hgd, OW433002_hg0, OW433002_hgd. Per unique
# combination we do >=1 requests which therefore result in >=1 responses. If output_choice is xml/json to file, then
# each response results in a file. Arguments 'flag_threshold' and 'drop_missing_values' have no effect.
from datetime import datetime
list_with_donwloaded_csv_filepaths = api.get_time_series_multi(
location_ids = ["OW433001", "OW433002"]
parameter_ids = ["H.G.0", "H.G.d"],
start_time = "2012-01-01T00:00:00Z", # or as datetime.datetime(year=2012, month=1, day=1)
end_time = "2012-01-02T00:00:00Z", # or as datetime.datetime(year=2012, month=1, day=2)
output_choice = hdsr_fewspy.OutputChoices.xml_file_in_download_dir,
)
# This api call accepts same arguments as get_time_series_single
print(list_with_donwloaded_csv_filepaths)
# <output_directory_root>/hdsr_fewspy_<datetime>/gettimeseriesmulti_ow433001_hg0_20120101t000000z_20120102t000000z_0.json
# <output_directory_root>/hdsr_fewspy_<datetime>/gettimeseriesmulti_ow433002_hg0_20120101t000000z_20120102t000000z_0.json
# <output_directory_root>/hdsr_fewspy_<datetime>/gettimeseriesmulti_ow433002_hg0_20120101t000000z_20120102t000000z_1.json
# If output_choice is csv to file, then all responses per unique combi are grouped in one csv file. Arguments
# 'flag_threshold' and 'drop_missing_values' do have effect.
list_with_donwloaded_csv_filepaths = api.get_time_series_multi(
location_ids = ["OW433001", "OW433002"]
parameter_ids = ["H.G.0", "H.G.d"],
start_time = "2012-01-01T00:00:00Z", # or as datetime.datetime(year=2012, month=1, day=1)
end_time = "2012-01-02T00:00:00Z", # or as datetime.datetime(year=2012, month=1, day=2)
output_choice = hdsr_fewspy.OutputChoices.csv_file_in_download_dir,
)
print(list_with_donwloaded_csv_filepaths)
# <output_directory_root>/hdsr_fewspy_<datetime>/gettimeseriesmulti_ow433001_hg0_20120101t000000z_20120102t000000z.csv
# <output_directory_root>/hdsr_fewspy_<datetime>/gettimeseriesmulti_ow433002_hg0_20120101t000000z_20120102t000000z.csv
```
9. get_time_series_statistics
```
from datetime import datetime
response = api.get_time_series_statistics(
location_id = "OW433001",
parameter_id = "H.G.0",
start_time = "2012-01-01T00:00:00Z", # or as datetime.datetime(year=2012, month=1, day=1)
end_time = "2012-01-02T00:00:00Z", # or as datetime.datetime(year=2012, month=1, day=2)
output_choice = hdsr_fewspy.OutputChoices.json_response_in_memory
)
print(response.text)
# {
# "timeSeries": [
# {
# "header": {
# "endDate": {"date": "2012-01-02", "time": "00:00:00"},
# "firstValueTime": {"date": "2012-01-01", "time": "00:15:00"},
# "lastValueTime": {"date": "2012-01-02", "time": "00:00:00"},
# "lat": "52.08992726570302",
# "locationId": "OW433001",
# "lon": "4.9547458967486095",
# "maxValue": "-0.28",
# "minValue": "-0.44",
# "missVal": "-999.0",
# "moduleInstanceId": "WerkFilter",
# "parameterId": "H.G.0",
# "startDate": {"date": "2012-01-01", "time": "00:00:00"},
# "stationName": "HAANWIJKERSLUIS_4330-w_Leidsche " "Rijn",
# "timeStep": {"unit": "nonequidistant"},
# "type": "instantaneous",
# "units": "mNAP",
# "valueCount": "102",
# "x": "125362.0",
# "y": "455829.0",
# "z": "-0.18",
# }
# }
# ]
# }
```
#### GITHUB_PERSONAL_ACCESS_TOKEN
A github personal token (a long hash) has to be created once and updated when it expires. You can have maximum 1 token.
This token is related to your github user account, so you don't need a token per repo/organisation/etc.
You can [create a token yourself][github personal token]. In short:
1. Login github.com with your account (user + password)
2. Ensure you have at least read-permission for the hdsr-mid repo(s) you want to interact with. To verify, browse to
the specific repo. If you can open it, then you have at least read-permission. If not, please contact
roger.de.crook@hdsr.nl to get access.
3. Create a token:
1. On github.com, go to your profile settings (click your icon right upper corner and 'settings' in the dropdown).
2. Click 'developer settings' (left lower corner).
3. Click 'Personal access tokens' and then 'Tokens (classic)'.
4. Click 'Generate new token' and then 'Generate new token (classic)'.
4. We recommend setting an expiry date of max 1 year (for safety reasons).
5. Create a file (Do not share this file with others!) on your personal HDSR drive 'G:/secrets.env' and add a line:
GITHUB_PERSONAL_ACCESS_TOKEN=<your_token>
### Development
For development, we request a FEWS SA webservice instance. The responses should not vary over time. Therefore, we use
FEWS stand alone that serves as a reference (D:\FEWS_202202_Peilevaluatie6 on Reken01).
1. Start the (reference) FEWS stand alone
2. Start PiWebservice
a. click with mouse on left side of screen
b. press F12
c. select 'Embedded servers'
d. select 'start embedded tomcat web services'
Now in the log panel you should see something like: "successfully started PiWebservice with test page: blabla"
3. Activate your conda environment and run tests with command 'pytest'
### License
[MIT]
### Releases
[Release history][releases]
### Contributions
All contributions, bug reports, documentation improvements, enhancements and ideas are welcome on the [issues page].
### Test Coverage (release 1.16)
```
---------- coverage: platform win32, python 3.12.0-final-0 -----------
Name Stmts Miss Cover
-------------------------------------------------------------------------------------
hdsr_fewspy\_version.py 1 0 100%
hdsr_fewspy\api.py 121 18 85%
hdsr_fewspy\api_calls\base.py 110 14 87%
hdsr_fewspy\api_calls\get_filters.py 25 0 100%
hdsr_fewspy\api_calls\get_locations.py 44 2 95%
hdsr_fewspy\api_calls\get_parameters.py 40 1 98%
hdsr_fewspy\api_calls\get_qualifiers.py 50 16 68%
hdsr_fewspy\api_calls\get_samples.py 41 4 90%
hdsr_fewspy\api_calls\get_timezone_id.py 26 1 96%
hdsr_fewspy\api_calls\time_series\base.py 161 15 91%
hdsr_fewspy\api_calls\time_series\get_time_series_multi.py 82 6 93%
hdsr_fewspy\api_calls\time_series\get_time_series_single.py 37 1 97%
hdsr_fewspy\api_calls\time_series\get_time_series_statistics.py 24 2 92%
hdsr_fewspy\constants\choices.py 134 5 96%
hdsr_fewspy\constants\custom_types.py 2 0 100%
hdsr_fewspy\constants\github.py 8 0 100%
hdsr_fewspy\constants\paths.py 9 0 100%
hdsr_fewspy\constants\pi_settings.py 76 6 92%
hdsr_fewspy\constants\request_settings.py 12 0 100%
hdsr_fewspy\converters\download.py 83 4 95%
hdsr_fewspy\converters\json_to_df_time_series.py 151 7 95%
hdsr_fewspy\converters\manager.py 31 2 94%
hdsr_fewspy\converters\utils.py 45 11 76%
hdsr_fewspy\converters\xml_to_python_obj.py 105 26 75%
hdsr_fewspy\date_frequency.py 47 2 96%
hdsr_fewspy\examples\area.py 30 30 0%
hdsr_fewspy\examples\point.py 22 22 0%
hdsr_fewspy\exceptions.py 14 0 100%
hdsr_fewspy\permissions.py 69 5 93%
hdsr_fewspy\retry_session.py 77 15 81%
hdsr_fewspy\secrets.py 40 5 88%
setup.py 16 16 0%
-------------------------------------------------------------------------------------
TOTAL 1733 236 86%
```
### Conda general tips
#### Build conda environment (on Windows) from any directory using environment.yml:
Note1: prefix is not set in the environment.yml as then conda does not handle it very well
Note2: env_directory can be anywhere, it does not have to be in your code project
```
> conda env create --prefix <env_directory><env_name> --file <path_to_project>/environment.yml
# example: conda env create --prefix C:/Users/xxx/.conda/envs/project_xx --file C:/Users/code_projects/xx/environment.yml
> conda info --envs # verify that <env_name> (project_xx) is in this list
```
#### Start the application from any directory:
```
> conda activate <env_name>
# At any location:
> (<env_name>) python <path_to_project>/main.py
```
#### Test the application:
```
> conda activate <env_name>
> cd <path_to_project>
> pytest # make sure pytest is installed (conda install pytest)
```
#### List all conda environments on your machine:
```
At any location:
> conda info --envs
```
#### Delete a conda environment:
```
Get directory where environment is located
> conda info --envs
Remove the enviroment
> conda env remove --name <env_name>
Finally, remove the left-over directory by hand
```
#### Write dependencies to environment.yml:
The goal is to keep the .yml as short as possible (not include sub-dependencies), yet make the environment
reproducible. Why? If you do 'conda install matplotlib' you also install sub-dependencies like pyqt, qt
icu, and sip. You should not include these sub-dependencies in your .yml as:
- including sub-dependencies result in an unnecessary strict environment (difficult to solve when conflicting)
- sub-dependencies will be installed when dependencies are being installed
```
> conda activate <conda_env_name>
Recommended:
> conda env export --from-history --no-builds | findstr -v "prefix" > --file <path_to_project>/environment_new.yml
Alternative:
> conda env export --no-builds | findstr -v "prefix" > --file <path_to_project>/environment_new.yml
--from-history:
Only include packages that you have explicitly asked for, as opposed to including every package in the
environment. This flag works regardless how you created the environment (through CMD or Anaconda Navigator).
--no-builds:
By default, the YAML includes platform-specific build constraints. If you transfer across platforms (e.g.
win32 to 64) omit the build info with '--no-builds'.
```
#### Pip and Conda:
If a package is not available on all conda channels, but available as pip package, one can install pip as a dependency.
Note that mixing packages from conda and pip is always a potential problem: conda calls pip, but pip does not know
how to satisfy missing dependencies with packages from Anaconda repositories.
```
> conda activate <env_name>
> conda install pip
> pip install <pip_package>
```
The environment.yml might look like:
```
channels:
- defaults
dependencies:
- <a conda package>=<version>
- pip
- pip:
- <a pip package>==<version>
```
You can also write a requirements.txt file:
```
> pip list --format=freeze > <path_to_project>/requirements.txt
```
Raw data
{
"_id": null,
"home_page": "https://github.com/hdsr-mid/hdsr_fewspy",
"name": "hdsr-fewspy",
"maintainer": "Roger de Crook",
"docs_url": null,
"requires_python": ">=3.7",
"maintainer_email": "roger.de.crook@hdsr.nl",
"keywords": "hdsr, fews, api, fewspy, wis",
"author": "Renier Kramer",
"author_email": "renier.kramer@hdsr.nl",
"download_url": "https://files.pythonhosted.org/packages/e1/4d/7f5153a811323ee53a34c199bc2b9bf793a7e54bf541da72fd3291bd10a8/hdsr_fewspy-1.17.tar.gz",
"platform": null,
"description": "### Context\r\n* Created: April 2023\r\n* Author: Renier Kramer, renier.kramer@hdsr.nl\r\n* Maintainer: Roger de Crook, roger.de.crook@hdsr.nl\r\n* Python version: >=3.7\r\n\r\n[hkvfewspy]: https://github.com/HKV-products-services/hkvfewspy\r\n[fewspy]: https://github.com/d2hydro/fewspy\r\n[MIT]: https://github.com/hdsr-mid/hdsr_fewspy/blob/main/LICENSE.txt\r\n[Deltares FEWS PI]: https://publicwiki.deltares.nl/display/FEWSDOC/FEWS+PI+REST+Web+Service\r\n[issues page]: https://github.com/hdsr-mid/hdsr_fewspy/issues\r\n[github personal token]: https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token\r\n[releases]: https://pypi.org/project/hdsr-fewspy/#history\r\n[flag details]: https://publicwiki.deltares.nl/display/FEWSDOC/D+Time+Series+Flags\r\n[user and auth settings]: https://github.com/hdsr-mid/hdsr_fewspy_auth \r\n\r\n\r\n### Description\r\nA python project to request data (locations, time-series, etc.) from FEWS-WIS (waterquantity) and FEWS-EFCIS (waterquality). \\\r\nThis project only works in HDSR's internal network, so within the VDI or virtual machine within the network. \\\r\nIt combines the best from two existing fewspy projects: [fewspy] and [hkvfewspy]. On top of that it adds \r\nclient-side authentication, authorisation, and throttling. \r\nThe latter is to minimize request load on HDSR's internal FEWS instances. \\\r\nFor detailed info on requesting FEWS APIs visit the [Deltares FEWS wiki][Deltares FEWS PI]. \r\n\r\nHdsr_fewspy API supports 9 different API calls that can return 6 different output formats: \r\n1. xml_file_in_download_dir: The xml response is written to a .xml file in your download_dir\r\n2. json_file_in_download_dir: The json response is written to a .json file in your download_dir\r\n3. csv_file_in_download_dir: The json response is converted to csv and written to a .csv file in your download_dir\r\n4. xml_response_in_memory: the xml response is returned memory meaning you get a list with one or more responses \r\n5. json_response_in_memory: the json response is returned memory meaning you get a list with one or more responses \r\n6. pandas_dataframe_in_memory: the json response is converted to a pandas dataframe meaning you get one dataframe \r\n\r\nAPI call | Supported outputs | Notes |\r\n------------------------------|-------------------|---------------------------------------------------------------------------------------------------------|\r\n1 get_parameters | 4, 5, 6 | Returns 1 object (xml/json response or dataframe) |\r\n2 get_filters | 4, 5 | Returns 1 object (xml/json response) |\r\n3 get_locations | 4, 5 | Returns 1 object (xml/json response) |\r\n4 get_qualifiers | 4, 6 | Returns 1 object (xml response or dataframe) |\r\n5 get_timezone_id | 4, 5 | Returns 1 object (xml/json response) |\r\n6 get_samples | 4 | Returns 1 object (xml response) |\r\n7 get_time_series_single | 4, 5, 6 | Returns 1 dataframe or a list >=1 xml/json responses |\r\n8 get_time_series_multi | 1, 2, 3 | Returns a list with downloaded files (1 .csv or >=1 .xml/.json per unique location_parameter_qualifier) |\r\n9 get_time_series_statistics | 4, 5 | Returns 1 object (xml/json response) |\r\n\r\n###### DefaultPiSettingsChoices:\r\nSeveral predefined pi_settings exists for point data and for area (e.g. averaged all points within an area). We mainly distinguish three levels of data:\r\n- raw: raw measurements from field stations. Contains valid and invalid data but lags little time with field measurements. \r\n- work: this data is being validated by a HDSR person (data validator CAW). This data might change every day. \r\n- validated: data without invalid data. Note that this data is published months after the actual measurement.\r\n\r\nThe names of the pi_settings are (see class DefaultPiSettingsChoices):\r\n- wis_production_point_raw\r\n- wis_production_point_work\r\n- wis_production_point_validated\r\n- wis_production_area_soilmoisture\r\n- wis_production_area_precipitation_wiwb\r\n- wis_production_area_precipitation_radarcorrection\r\n- wis_production_area_evaporation_wiwb_satdata\r\n- wis_production_area_evaporation_waterwatch\r\n- efcis_production_point_fysische_chemie\r\n- efcis_production_point_biologie\r\n\r\n\r\n### Usage\r\nBelow you find 10 examples for the 9 different requests. In hdsr_fewspy/examples/ you also find code to download \r\ndischarge (point), soil moisture (area), evaporation (area), and precipitation (area) time-series. \r\n\r\n#### Preparation\r\n\r\n1. Only once needed: ensure you have a github account with a GITHUB_PERSONAL_ACCESS_TOKEN. Read [github_personal_access_token](#github_personal_access_token) below to\r\n know what to do with this token.\r\n2. Only once needed: ensure your github username is registered in [user and auth settings] permissions.csv file. If \r\n not, ask the maintainer of hdsr_fewspy to fix this.\r\n3. You can create a hdsr_fewspy API in two ways (the first dominates the second). We advise to use the second: \r\n - with API argument 'github_personal_access_token', thus ```api = hdsr_fewspy.Api(<github_personal_access_token>)```.\r\n - or with API argument 'secrets_env_path' (defaults to 'G:/secrets.env'), thus ```api = hdsr_fewspy.Api(<your_secrets_env_path>)```. \r\n The .env file must have a row:\r\n```\r\nGITHUB_PERSONAL_ACCESS_TOKEN=<your_github_token>\r\n```\r\n4. Only once per project: install hdsr_fewspy dependency:\r\n```\r\npip install hdsr-fewspy \r\n# or \r\nconda install hdsr-fewspy --channel hdsr-mid\r\n```\r\n5. Example simple 'create API instance':\r\n```\r\nimport hdsr_fewspy\r\n\r\napi = hdsr_fewspy.Api()\r\n```\r\n6. Example sophisticated 'create API instance':\r\n```\r\nimport hdsr_fewspy\r\n\r\n# Optionally, you can specify several API arguments:\r\n# github_email: str, \r\n# github_personal_access_token: str\r\n# secrets_env_path: str or pathlib.Path\r\n# pi_settings: hdsr_fewspy.PiSettings \r\n# output_directory_root: str or pathlib.path\r\n\r\n# option 1\r\n# For example in case of pi_settings, you can use predefined settings, see topic 'DefaultPiSettingsChoices' above\r\n# To list all DefaultPiSettingsChoices:\r\nhdsr_fewspy.DefaultPiSettingsChoices.get_all( \r\n# For fews-wis api, use a DefaultPiSettingsChoices that starts with 'wis_', for example:\r\napi = hdsr_fewspy.Api(pi_settings=hdsr_fewspy.DefaultPiSettingsChoices.wis_production_point_work)\r\n# For fews-efcis api, use a DefaultPiSettingsChoices that starts with 'efics_', for example:\r\napi = hdsr_fewspy.Api(pi_settings=hdsr_fewspy.DefaultPiSettingsChoices.efcis_production_point_biologie)\r\n\r\n# option 2\r\n# Or create your own pi_settings:\r\ncustom_settings = hdsr_fewspy.PiSettings(\r\n settings_name=\"does not matter blabla\", \r\n document_version=1.25,\r\n ssl_verify=True,\r\n domain=\"localhost\",\r\n port=8080,\r\n service=\"FewsWebServices\",\r\n filter_id=\"INTERNAL-API\",\r\n module_instance_ids=\"WerkFilter\",\r\n time_zone=hdsr_fewspy.TimeZoneChoices.eu_amsterdam.value, # = 1.0 (only affects get_time_series dataframe and csv)\r\n)\r\napi = hdsr_fewspy.Api(pi_settings=custom_settings)\r\n\r\n# option 3\r\n# In case you want to download responses to file, then you need to specify an output_directory_root \r\n# The files will be downloaded in a subdir: output_directory_root/hdsr_fewspy_<datetime>/<files_will_be_downloaded_here>\r\napi = hdsr_fewspy.Api(output_directory_root=<path_to_a_dir>)\r\n```\r\n\r\n\r\n#### Examples API calls\r\n###### Below you see 9 examples using api option 3 above.\r\n###### Moreover, in hdsr_fewspy/examples/ you find examples for point and area downloads.\r\n\r\n1. get_parameters\r\n```\r\ndf = api.get_parameters(output_choice=hdsr_fewspy.OutputChoices.pandas_dataframe_in_memory)\r\n\r\n# id name parameter_type unit display_unit uses_datum parameter_group \r\n# --------------------------------------------------------------------------------------------------------- \r\n# BG.b.0 Oppervlaktebegroeiing [%] - noneq instantaneous % % False Begroeiingsgraad \r\n# BG.fd.0 Flab en draadwieren [%] - noneq instantaneous % % False Begroeiingsgraad \r\n# BG.ka.0 Algen-/kroosbedekking [%] - noneq instantaneous % % False Begroeiingsgraad \r\n# ...etc...\r\n```\r\n2. get_filters\r\n```\r\nresponse = api.get_filters(output_choice=hdsr_fewspy.OutputChoices.json_response_in_memory)\r\nresponse.json() \r\n\r\n# {\r\n# \"version\": \"1.25\",\r\n# \"filters\": [\r\n# {\r\n# \"id\": \"INTERNAL-API\",\r\n# \"name\": \"INTERNAL-API\",\r\n# \"child\": [{\"id\": \"INTERNAL-API.RUWMET\", \"name\": \"Ruwe metingen (punt)\"\r\n# ...etc...\r\n```\r\n3. get_locations\r\n```\r\ngdf = get_locations(output_choice=hdsr_fewspy.OutputChoices.pandas_dataframe_in_memory, show_attributes=True)\r\n\r\n# location_id description short_name lat lon x y z parent_location_id geometry attributes\r\n# ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- \r\n# beg_062 BEGROEIINGSMEETPUNT beg_062-LR_13_xruim 52.58907339700342 5.1718081593021505 140251.0 460118.0 0.0 NaN POINT (140251.000 460118.000) [{'name': 'LOC_NAME', 'type': 'text', 'id': 'LOC_NAME', 'value': 'beg_062 ...etc...]\r\n# beg_084 BEGROEIINGSMEETPUNT beg_084-LR_17_xruim 52.06261306007392 5.12600382893812 137088.0 452734.0 0.0 NaN POINT (137088.000 452734.000) [{'name': 'LOC_NAME', 'type': 'text', 'id': 'LOC_NAME', 'value': 'beg_084 ...etc...]\r\n# beg_102 BEGROEIINGSMEETPUNT beg_102-KR_9_xruim 52.07249358678008 5.215531005948442 143230.0 453815.0 0.0 NaN POINT (143230.000 453815.000) [{'name': 'LOC_NAME', 'type': 'text', 'id': 'LOC_NAME', 'value': 'beg_102 ...etc...]\r\n# ...etc...\r\n```\r\n4. get_qualifiers\r\n```\r\ndf = fixture_api_sa_no_download_dir.get_qualifiers(output_choice=hdsr_fewspy.OutputChoices.pandas_dataframe_in_memory)\r\n \r\n# id name group_id\r\n# -----------------------------------\r\n# ergkrap erg krap (max 10%) None\r\n# krap krap (max 30%) None\r\n# normaal normaal (max 50%) None\r\n# ruim ruim (max 70%) None\r\n# ...etc...\r\n```\r\n5. get_timezone_id\r\n```\r\nresponse = api.get_timezone_id(output_choice=hdsr_fewspy.OutputChoices.json_response_in_memory)\r\n\r\n# verify response\r\nassert response.text == \"GMT\"\r\nassert TimeZoneChoices.get_tz_float(value=\"GMT\") == TimeZoneChoices.gmt.value == 0.0\r\n```\r\n6. get_samples\r\n```\r\n# Note: only FEWS-EFCIS contains sample data\r\napi = hdsr_fewspy.Api(pi_settings=hdsr_fewspy.DefaultPiSettingsChoices.efcis_production_point_biologie)\r\nresponse = api.get_samples(output_choice=hdsr_fewspy.OutputChoices.xml_response_in_memory)\r\n\r\nprint(response.text)\r\n# <Samples xmlns=\"http://www.wldelft.nl/fews/PI\" xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" xmlns:fs=\"http://www.wldelft.nl/fews/fs\" xmlns:qualifierId=\"http://www.wldelft.nl/fews/qualifierId\" xsi:schemaLocation=\"http://www.wldelft.nl/fews/PI https://fewsdocs.deltares.nl/schemas/version1.0/pi-schemas/pi_samples.xsd\" version=\"1.34\">\r\n# <timeZone>0.0</timeZone>\r\n# <sample id=\"2020_00090e\">\r\n# <header>\r\n# <moduleInstanceId>Import_EFCIS_dump</moduleInstanceId>\r\n# <locationId>20924</locationId>\r\n# <sampleDate date=\"2020-01-09\" time=\"09:57:00\"/>\r\n# <lat>51.98814213752072</lat>\r\n# <lon>5.243994665879352</lon>\r\n# <x>145163.0</x>\r\n# <y>444426.0</y>\r\n# </header>\r\n# <properties>\r\n# <string key=\"activiteit\" value=\"Automatische FTP import\"/>\r\n# <string key=\"namespace\" value=\"NL14\"/>\r\n# </properties>\r\n# <table>\r\n# <row parameterId=\"WNS1042\" qualifierId:q1=\"WNS1042\" qualifierId:q2=\"GH_34\" qualifierId:q3=\"HH_492\" qualifierId:q4=\"ACO_39\" qualifierId:q5=\"MCO_39\" qualifierId:q6=\"PC_1569\" flag=\"0\" value=\"0.71\" unit=\"mg/l\"/>\r\n# <row parameterId=\"WNS1078\" qualifierId:q1=\"WNS1078\" qualifierId:q2=\"GH_34\" qualifierId:q3=\"HH_492\" qualifierId:q4=\"ACO_39\" qualifierId:q5=\"MCO_39\" qualifierId:q6=\"PC_1582\" flag=\"0\" value=\"0.01\" detection=\"<\" unit=\"ug/l\"/>\r\n# <row parameterId=\"WNS1085\" qualifierId:q1=\"WNS1085\" qualifierId:q2=\"GH_34\" qualifierId:q3=\"HH_492\" qualifierId:q4=\"ACO_39\" qualifierId:q5=\"MCO_39\" qualifierId:q6=\"PC_1584\" flag=\"0\" value=\"0.01\" detection=\"<\" unit=\"ug/l\"/>\r\n# <row parameterId=\"WNS1230\" qualifierId:q1=\"WNS1230\" qualifierId:q2=\"GH_16\" qualifierId:q3=\"HH_492\" qualifierId:q4=\"ACO_39\" qualifierId:q5=\"MCO_39\" qualifierId:q6=\"PC_1721\" flag=\"0\" value=\"0\" unit=\"DIMSLS\"/>\r\n# ...\r\n# <row parameterId=\"WNS742\" qualifierId:q1=\"WNS742\" qualifierId:q2=\"GH_34\" qualifierId:q3=\"HH_492\" qualifierId:q4=\"ACO_39\" qualifierId:q5=\"MCO_39\" qualifierId:q6=\"PC_1248\" flag=\"0\" value=\"3\" unit=\"ug/l\"/>\r\n# <row parameterId=\"WNS814\" qualifierId:q1=\"WNS814\" qualifierId:q2=\"GH_34\" qualifierId:q3=\"HH_492\" qualifierId:q4=\"ACO_39\" qualifierId:q5=\"MCO_39\" qualifierId:q6=\"PC_1257\" flag=\"0\" value=\"0.01\" detection=\"<\" unit=\"ug/l\"/>\r\n# <row parameterId=\"WNS8372\" qualifierId:q1=\"WNS8372\" qualifierId:q2=\"GH_20\" qualifierId:q3=\"HH_492\" qualifierId:q4=\"ACO_39\" qualifierId:q5=\"MCO_39\" qualifierId:q6=\"PC_1725\" flag=\"0\" value=\"5\" detection=\"<\" unit=\"%\"/>\r\n# <row parameterId=\"WNS9756\" qualifierId:q1=\"WNS9756\" qualifierId:q2=\"GH_191\" qualifierId:q3=\"HH_492\" qualifierId:q4=\"ACO_39\" qualifierId:q5=\"MCO_39\" flag=\"0\" value=\"100\" unit=\"-\"/>\r\n# </table>\r\n# </sample>\r\n# </Samples>\r\n```\r\n7. get_time_series_single\r\n\r\n[click here for more info on flags][flag details]\r\n```\r\n# Single means: use max 1 location_id and/or parameter_id and/or qualifier_id. One large call can result in multiple \r\n# small calls and therefore multiple responses. If your output_choice is json/xml in memory, then you get a list with \r\n# >=1 responses. Arguments 'flag_threshold' and 'drop_missing_values' have no effect.\r\n \r\n\r\nresponses = api.get_time_series_single(\r\n location_id = \"OW433001\",\r\n parameter_id = \"H.G.0\",\r\n start_time = \"2012-01-01T00:00:00Z\", # or as datetime.datetime(year=2012, month=1, day=1)\r\n end_time = \"2012-01-02T00:00:00Z\", # or as datetime.datetime(year=2012, month=1, day=2)\r\n output_choice = hdsr_fewspy.OutputChoices.xml_response_in_memory\r\n)\r\n\r\nprint(responses[0].text)\r\n# <?xml version=\"1.0\" encoding=\"UTF-8\"?>\r\n# <TimeSeries xmlns=\"http://www.wldelft.nl/fews/PI\" xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" xsi:schemaLocation=\"http://www.wldelft.nl/fews/PI http://fews.wldelft.nl/schemas/version1.0/pi-schemas/pi_timeseries.xsd\" version=\"1.25\" xmlns:fs=\"http://www.wldelft.nl/fews/fs\">\r\n# <timeZone>0.0</timeZone>\r\n# <series>\r\n# <header>\r\n# <type>instantaneous</type>\r\n# <moduleInstanceId>WerkFilter</moduleInstanceId>\r\n# <locationId>OW433001</locationId>\r\n# <parameterId>H.G.0</parameterId>\r\n# <timeStep unit=\"nonequidistant\"/>\r\n# <startDate date=\"2012-01-01\" time=\"00:00:00\"/>\r\n# <endDate date=\"2012-01-02\" time=\"00:00:00\"/>\r\n# <missVal>-999.0</missVal>\r\n# <stationName>HAANWIJKERSLUIS_4330-w_Leidsche Rijn</stationName>\r\n# <lat>52.08992726570302 asdf renier</lat>\r\n# <lon>4.9547458967486095</lon>\r\n# <x>125362.0</x>\r\n# <y>455829.0</y>\r\n# <z>-0.18</z>\r\n# <units>mNAP</units>\r\n# </header>\r\n# <event date=\"2012-01-01\" time=\"00:15:00\" value=\"-0.35\" flag=\"0\" fs:PRIMAIR=\"OK\" fs:VISUEEL=\"OK\"/>\r\n# <event date=\"2012-01-01\" time=\"00:45:00\" value=\"-0.36\" flag=\"0\" fs:PRIMAIR=\"OK\" fs:VISUEEL=\"OK\"/>\r\n# <event date=\"2012-01-01\" time=\"02:30:00\" value=\"-0.37\" flag=\"0\" fs:PRIMAIR=\"OK\" fs:VISUEEL=\"OK\"/>\r\n# <event date=\"2012-01-01\" time=\"02:31:17\" value=\"-0.38\" flag=\"0\" fs:PRIMAIR=\"OK\" fs:VISUEEL=\"OK\"/>\r\n# <event date=\"2012-01-01\" time=\"03:15:00\" value=\"-0.39\" flag=\"0\" fs:PRIMAIR=\"OK\" fs:VISUEEL=\"OK\"/>\r\n# ...etc..\r\n\r\n# If your output_choice is dataframe, then all responses are collected in one dataframe. Arguments 'flag_threshold' \r\n# and 'drop_missing_values' do have effect.\r\n\r\n# Note that argument only_value_and_flag=True returns timeseries with {timestamp, value, flag}. \r\n# This only affects output_choices dataframe and csv. \r\n# only_value_and_flag=False returns also fields like {comment, etc}.\r\n\r\ndf = api.get_time_series_single(\r\n location_id = \"OW433001\",\r\n parameter_id = \"H.G.0\",\r\n qualifier_id = \"\", # defaults to \"\"\r\n start_time = \"2012-01-01T00:00:00Z\", # or as datetime.datetime(year=2012, month=1, day=1)\r\n end_time = \"2012-01-02T00:00:00Z\", # or as datetime.datetime(year=2012, month=1, day=2)\r\n drop_missing_values = True,\r\n flag_threshold = 6, # all flags 6 and higher are removed from dataframe\r\n thinning = None, # [integer: ms/pixel] defaults to None. See Deltares wiki (link above)\r\n omit_empty_time_series = True, # [bool] defaults to True\r\n only_value_and_flag = True, # [bool] defaults to True \r\n output_choice = hdsr_fewspy.OutputChoices.pandas_dataframe_in_memory, \r\n)\r\n```\r\n8. get_time_series_multi\r\n\r\n[click here for more info on flags][flag details]\r\n```\r\n# Multi means: use >=1 location_id and/or parameter_id and/or qualifier_id. The api call below results in 4 unique \r\n# location_parameter_qualifier combinations: OW433001_hg0, OW433001_hgd, OW433002_hg0, OW433002_hgd. Per unique \r\n# combination we do >=1 requests which therefore result in >=1 responses. If output_choice is xml/json to file, then \r\n# each response results in a file. Arguments 'flag_threshold' and 'drop_missing_values' have no effect. \r\n\r\nfrom datetime import datetime\r\nlist_with_donwloaded_csv_filepaths = api.get_time_series_multi(\r\n location_ids = [\"OW433001\", \"OW433002\"]\r\n parameter_ids = [\"H.G.0\", \"H.G.d\"],\r\n start_time = \"2012-01-01T00:00:00Z\", # or as datetime.datetime(year=2012, month=1, day=1)\r\n end_time = \"2012-01-02T00:00:00Z\", # or as datetime.datetime(year=2012, month=1, day=2)\r\n output_choice = hdsr_fewspy.OutputChoices.xml_file_in_download_dir,\r\n)\r\n# This api call accepts same arguments as get_time_series_single\r\n\r\nprint(list_with_donwloaded_csv_filepaths)\r\n# <output_directory_root>/hdsr_fewspy_<datetime>/gettimeseriesmulti_ow433001_hg0_20120101t000000z_20120102t000000z_0.json\r\n# <output_directory_root>/hdsr_fewspy_<datetime>/gettimeseriesmulti_ow433002_hg0_20120101t000000z_20120102t000000z_0.json\r\n# <output_directory_root>/hdsr_fewspy_<datetime>/gettimeseriesmulti_ow433002_hg0_20120101t000000z_20120102t000000z_1.json\r\n\r\n# If output_choice is csv to file, then all responses per unique combi are grouped in one csv file. Arguments \r\n# 'flag_threshold' and 'drop_missing_values' do have effect.\r\n \r\nlist_with_donwloaded_csv_filepaths = api.get_time_series_multi(\r\n location_ids = [\"OW433001\", \"OW433002\"]\r\n parameter_ids = [\"H.G.0\", \"H.G.d\"],\r\n start_time = \"2012-01-01T00:00:00Z\", # or as datetime.datetime(year=2012, month=1, day=1)\r\n end_time = \"2012-01-02T00:00:00Z\", # or as datetime.datetime(year=2012, month=1, day=2)\r\n output_choice = hdsr_fewspy.OutputChoices.csv_file_in_download_dir,\r\n)\r\n\r\nprint(list_with_donwloaded_csv_filepaths) \r\n# <output_directory_root>/hdsr_fewspy_<datetime>/gettimeseriesmulti_ow433001_hg0_20120101t000000z_20120102t000000z.csv\r\n# <output_directory_root>/hdsr_fewspy_<datetime>/gettimeseriesmulti_ow433002_hg0_20120101t000000z_20120102t000000z.csv\r\n```\r\n9. get_time_series_statistics\r\n```\r\nfrom datetime import datetime\r\nresponse = api.get_time_series_statistics(\r\n location_id = \"OW433001\",\r\n parameter_id = \"H.G.0\",\r\n start_time = \"2012-01-01T00:00:00Z\", # or as datetime.datetime(year=2012, month=1, day=1)\r\n end_time = \"2012-01-02T00:00:00Z\", # or as datetime.datetime(year=2012, month=1, day=2)\r\n output_choice = hdsr_fewspy.OutputChoices.json_response_in_memory\r\n)\r\n\r\nprint(response.text)\r\n# {\r\n# \"timeSeries\": [\r\n# {\r\n# \"header\": {\r\n# \"endDate\": {\"date\": \"2012-01-02\", \"time\": \"00:00:00\"},\r\n# \"firstValueTime\": {\"date\": \"2012-01-01\", \"time\": \"00:15:00\"},\r\n# \"lastValueTime\": {\"date\": \"2012-01-02\", \"time\": \"00:00:00\"},\r\n# \"lat\": \"52.08992726570302\",\r\n# \"locationId\": \"OW433001\",\r\n# \"lon\": \"4.9547458967486095\",\r\n# \"maxValue\": \"-0.28\",\r\n# \"minValue\": \"-0.44\",\r\n# \"missVal\": \"-999.0\",\r\n# \"moduleInstanceId\": \"WerkFilter\",\r\n# \"parameterId\": \"H.G.0\",\r\n# \"startDate\": {\"date\": \"2012-01-01\", \"time\": \"00:00:00\"},\r\n# \"stationName\": \"HAANWIJKERSLUIS_4330-w_Leidsche \" \"Rijn\",\r\n# \"timeStep\": {\"unit\": \"nonequidistant\"},\r\n# \"type\": \"instantaneous\",\r\n# \"units\": \"mNAP\",\r\n# \"valueCount\": \"102\",\r\n# \"x\": \"125362.0\",\r\n# \"y\": \"455829.0\",\r\n# \"z\": \"-0.18\",\r\n# }\r\n# }\r\n# ]\r\n# } \r\n```\r\n\r\n#### GITHUB_PERSONAL_ACCESS_TOKEN\r\nA github personal token (a long hash) has to be created once and updated when it expires. You can have maximum 1 token.\r\nThis token is related to your github user account, so you don't need a token per repo/organisation/etc. \r\nYou can [create a token yourself][github personal token]. In short:\r\n1. Login github.com with your account (user + password)\r\n2. Ensure you have at least read-permission for the hdsr-mid repo(s) you want to interact with. To verify, browse to \r\n the specific repo. If you can open it, then you have at least read-permission. If not, please contact \r\n roger.de.crook@hdsr.nl to get access.\r\n3. Create a token:\r\n 1. On github.com, go to your profile settings (click your icon right upper corner and 'settings' in the dropdown).\r\n 2. Click 'developer settings' (left lower corner).\r\n 3. Click 'Personal access tokens' and then 'Tokens (classic)'.\r\n 4. Click 'Generate new token' and then 'Generate new token (classic)'.\r\n4. We recommend setting an expiry date of max 1 year (for safety reasons).\r\n5. Create a file (Do not share this file with others!) on your personal HDSR drive 'G:/secrets.env' and add a line: \r\n GITHUB_PERSONAL_ACCESS_TOKEN=<your_token>\r\n \r\n\r\n### Development\r\nFor development, we request a FEWS SA webservice instance. The responses should not vary over time. Therefore, we use \r\nFEWS stand alone that serves as a reference (D:\\FEWS_202202_Peilevaluatie6 on Reken01). \r\n1. Start the (reference) FEWS stand alone\r\n2. Start PiWebservice \r\n a. click with mouse on left side of screen \r\n b. press F12\r\n c. select 'Embedded servers'\r\n d. select 'start embedded tomcat web services'\r\n Now in the log panel you should see something like: \"successfully started PiWebservice with test page: blabla\"\r\n3. Activate your conda environment and run tests with command 'pytest'\r\n\r\n\r\n### License \r\n[MIT]\r\n\r\n### Releases\r\n[Release history][releases]\r\n\r\n### Contributions\r\nAll contributions, bug reports, documentation improvements, enhancements and ideas are welcome on the [issues page].\r\n\r\n### Test Coverage (release 1.16)\r\n```\r\n---------- coverage: platform win32, python 3.12.0-final-0 -----------\r\nName Stmts Miss Cover\r\n-------------------------------------------------------------------------------------\r\nhdsr_fewspy\\_version.py 1 0 100%\r\nhdsr_fewspy\\api.py 121 18 85%\r\nhdsr_fewspy\\api_calls\\base.py 110 14 87%\r\nhdsr_fewspy\\api_calls\\get_filters.py 25 0 100%\r\nhdsr_fewspy\\api_calls\\get_locations.py 44 2 95%\r\nhdsr_fewspy\\api_calls\\get_parameters.py 40 1 98%\r\nhdsr_fewspy\\api_calls\\get_qualifiers.py 50 16 68%\r\nhdsr_fewspy\\api_calls\\get_samples.py 41 4 90%\r\nhdsr_fewspy\\api_calls\\get_timezone_id.py 26 1 96%\r\nhdsr_fewspy\\api_calls\\time_series\\base.py 161 15 91%\r\nhdsr_fewspy\\api_calls\\time_series\\get_time_series_multi.py 82 6 93%\r\nhdsr_fewspy\\api_calls\\time_series\\get_time_series_single.py 37 1 97%\r\nhdsr_fewspy\\api_calls\\time_series\\get_time_series_statistics.py 24 2 92%\r\nhdsr_fewspy\\constants\\choices.py 134 5 96%\r\nhdsr_fewspy\\constants\\custom_types.py 2 0 100%\r\nhdsr_fewspy\\constants\\github.py 8 0 100%\r\nhdsr_fewspy\\constants\\paths.py 9 0 100%\r\nhdsr_fewspy\\constants\\pi_settings.py 76 6 92%\r\nhdsr_fewspy\\constants\\request_settings.py 12 0 100%\r\nhdsr_fewspy\\converters\\download.py 83 4 95%\r\nhdsr_fewspy\\converters\\json_to_df_time_series.py 151 7 95%\r\nhdsr_fewspy\\converters\\manager.py 31 2 94%\r\nhdsr_fewspy\\converters\\utils.py 45 11 76%\r\nhdsr_fewspy\\converters\\xml_to_python_obj.py 105 26 75%\r\nhdsr_fewspy\\date_frequency.py 47 2 96%\r\nhdsr_fewspy\\examples\\area.py 30 30 0%\r\nhdsr_fewspy\\examples\\point.py 22 22 0%\r\nhdsr_fewspy\\exceptions.py 14 0 100%\r\nhdsr_fewspy\\permissions.py 69 5 93%\r\nhdsr_fewspy\\retry_session.py 77 15 81%\r\nhdsr_fewspy\\secrets.py 40 5 88%\r\nsetup.py 16 16 0%\r\n-------------------------------------------------------------------------------------\r\nTOTAL 1733 236 86%\r\n```\r\n\r\n### Conda general tips\r\n#### Build conda environment (on Windows) from any directory using environment.yml:\r\nNote1: prefix is not set in the environment.yml as then conda does not handle it very well\r\nNote2: env_directory can be anywhere, it does not have to be in your code project\r\n```\r\n> conda env create --prefix <env_directory><env_name> --file <path_to_project>/environment.yml\r\n# example: conda env create --prefix C:/Users/xxx/.conda/envs/project_xx --file C:/Users/code_projects/xx/environment.yml\r\n> conda info --envs # verify that <env_name> (project_xx) is in this list \r\n```\r\n#### Start the application from any directory:\r\n```\r\n> conda activate <env_name>\r\n# At any location:\r\n> (<env_name>) python <path_to_project>/main.py\r\n```\r\n#### Test the application:\r\n```\r\n> conda activate <env_name>\r\n> cd <path_to_project>\r\n> pytest # make sure pytest is installed (conda install pytest)\r\n```\r\n#### List all conda environments on your machine:\r\n```\r\nAt any location:\r\n> conda info --envs\r\n```\r\n#### Delete a conda environment:\r\n```\r\nGet directory where environment is located \r\n> conda info --envs\r\nRemove the enviroment\r\n> conda env remove --name <env_name>\r\nFinally, remove the left-over directory by hand\r\n```\r\n#### Write dependencies to environment.yml:\r\nThe goal is to keep the .yml as short as possible (not include sub-dependencies), yet make the environment \r\nreproducible. Why? If you do 'conda install matplotlib' you also install sub-dependencies like pyqt, qt \r\nicu, and sip. You should not include these sub-dependencies in your .yml as:\r\n- including sub-dependencies result in an unnecessary strict environment (difficult to solve when conflicting)\r\n- sub-dependencies will be installed when dependencies are being installed\r\n```\r\n> conda activate <conda_env_name>\r\n\r\nRecommended:\r\n> conda env export --from-history --no-builds | findstr -v \"prefix\" > --file <path_to_project>/environment_new.yml \r\n\r\nAlternative:\r\n> conda env export --no-builds | findstr -v \"prefix\" > --file <path_to_project>/environment_new.yml \r\n\r\n--from-history: \r\n Only include packages that you have explicitly asked for, as opposed to including every package in the \r\n environment. This flag works regardless how you created the environment (through CMD or Anaconda Navigator).\r\n--no-builds:\r\n By default, the YAML includes platform-specific build constraints. If you transfer across platforms (e.g. \r\n win32 to 64) omit the build info with '--no-builds'.\r\n```\r\n#### Pip and Conda:\r\nIf a package is not available on all conda channels, but available as pip package, one can install pip as a dependency.\r\nNote that mixing packages from conda and pip is always a potential problem: conda calls pip, but pip does not know \r\nhow to satisfy missing dependencies with packages from Anaconda repositories. \r\n```\r\n> conda activate <env_name>\r\n> conda install pip\r\n> pip install <pip_package>\r\n```\r\nThe environment.yml might look like:\r\n```\r\nchannels:\r\n - defaults\r\ndependencies:\r\n - <a conda package>=<version>\r\n - pip\r\n - pip:\r\n - <a pip package>==<version>\r\n```\r\nYou can also write a requirements.txt file:\r\n```\r\n> pip list --format=freeze > <path_to_project>/requirements.txt\r\n```\r\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "A python project to request data (locations, time-series, etc.) from a HDSR FEWS PiWebService",
"version": "1.17",
"project_urls": {
"Download": "https://github.com/hdsr-mid/hdsr_fewspy/archive/v1.17.tar.gz",
"Homepage": "https://github.com/hdsr-mid/hdsr_fewspy"
},
"split_keywords": [
"hdsr",
" fews",
" api",
" fewspy",
" wis"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "e14d7f5153a811323ee53a34c199bc2b9bf793a7e54bf541da72fd3291bd10a8",
"md5": "2ed5dbd019ada6bad301fdb97c7fdcf2",
"sha256": "4fe6047d0233a84f3e06ef1a73fdcc8c70ee089ef063cd29a6a47f77b8600f89"
},
"downloads": -1,
"filename": "hdsr_fewspy-1.17.tar.gz",
"has_sig": false,
"md5_digest": "2ed5dbd019ada6bad301fdb97c7fdcf2",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7",
"size": 66273,
"upload_time": "2024-05-05T16:54:51",
"upload_time_iso_8601": "2024-05-05T16:54:51.351171Z",
"url": "https://files.pythonhosted.org/packages/e1/4d/7f5153a811323ee53a34c199bc2b9bf793a7e54bf541da72fd3291bd10a8/hdsr_fewspy-1.17.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-05-05 16:54:51",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "hdsr-mid",
"github_project": "hdsr_fewspy",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "hdsr-fewspy"
}
JSON
