# Elwood
An open source dataset transformation, standardization, and normalization python library.
# Usage
To use start using Elwood, simply run:
`pip install elwood`
Now you are able to use any of the dataset transformation, standardization, or normalization functions exposed through this library. To start, simply include `from elwood import elwood` in your python file.
## Standardization
`elwood.process(args)`
Given an arbitrary dataset containing geospatial data (with columns and rows) with arbitrary non-standard format, and given some annotations/dictionary about the dataset, Elwood can standardize. Standardization means creating an output dataset with stable and predictable columns. The data can be normalized, regridded, scaled, and resolved using GADM to standard country names, as well as resolve the latitude,longitude of the event/measurement. A usual standard output will contain the following columns: `timestamp`, `country`, `admin1`, `admin2`, `admin3`, `lat`, `lng`, alongside other measurements/events/features of interest (additional columns to the right of the standard ones) contained within the input dataset.
#TODO document standardization further
## Transformation
The transformation functions include geographical extent clipping (latitude/longitude), geographical regridding (gridded data such as NetCDF or GeoTIFF), temporal clipping, and temporal scaling.
### Geospatial Clipping
`elwood.clip_geo(dataframe, geo_columns, polygons_list)`
This function takes a pandas dataframe, a geo_columns dict of the column names for latitude and longitude, ex:
`{'lat_column': 'latitude', 'lon_column': 'longitude'}`, and a list containing lists of objects representing the polygons to clip the data to. ex:
```
[
[
{
"lat": 11.0,
"lng": 42.0
},
{
"lat": 11.0,
"lng": 43.0
},
{
"lat": 12.0,
"lng": 43.0
},
{
"lat": 12.0,
"lng": 42.0
}
],
...
]
```
### Geospatial regridding
`elwood.regrid_dataframe_geo(dataframe, geo_columns, scale_multi)`
This function takes a dataframe and regrids it's geography by some scale multiplier that is provided. This multiplier will be used to divide the current geographical scale in order to make a more coarse grained resolution dataset. The dataframe must have a detectable geographical scale, meaning each lat/lon represents a point in the middle of a gridded cell for the data provided. Lat and lon and determined by the geo_columns passed in: a dict of the column names ex: `{'lat_column': 'my_latitude', 'lon_column': 'my_longitude'}`
### Temporal Clipping
`elwood.clip_dataframe_time(dataframe, time_column, time_ranges)`
This function will produce a dataframe that only includes rows with `time_column` values contained within `time_ranges`. The time_ranges argument is a list of objects containing a start and end time. ex: `[{"start": datetime, "end": datetime}, ...]`
### Temporal Scaling
`elwood.rescale_dataframe_time(dataframe, time_column, time_bucket, aggregation_function_list)`
This function will produce a dataframe who's rows are the aggregated data based on some time bucket and some aggregation function list provided. The `time_column` is the name of the column containing targeted time values for rescaling. The `time_bucket` is some DateOffset, Timedelta or str representing the desired time granularity, ex. `'M', 'A', '2H'`. The `aggregation_function_list` is a list of aggregation functions to apply to the data. ex. `['sum']` or `['sum', 'min', 'max']`
## 0 to 1 Normalization
`elwood.normalize_features(dataframe, output_file)`
This function expects a dataframe with a "feature" column and a "value" column, or long data. Each entry for a feature has its own feature/value row.
This function returns a dataframe in which all numerical values under the "value" column for each "feature" have been 0 to 1 scaled.
Optionally you may specify an `output_file` name to generate a parquet file of the dataframe.
# History
0.1.4
Added new regridding functionality, updated existing numpy and pandas based regridding.
0.1.4
0.1.2
0.1.1
0.1.0
Raw data
{
"_id": null,
"home_page": "https://github.com/jataware/elwood",
"name": "elwood",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": "",
"keywords": "elwood",
"author": "Brandon Rose, Powell Fendley",
"author_email": "info@jataware.com",
"download_url": "",
"platform": null,
"description": "# Elwood\nAn open source dataset transformation, standardization, and normalization python library.\n\n# Usage\n\nTo use start using Elwood, simply run:\n\n`pip install elwood`\n\nNow you are able to use any of the dataset transformation, standardization, or normalization functions exposed through this library. To start, simply include `from elwood import elwood` in your python file. \n\n## Standardization\n`elwood.process(args)`\n\nGiven an arbitrary dataset containing geospatial data (with columns and rows) with arbitrary non-standard format, and given some annotations/dictionary about the dataset, Elwood can standardize. Standardization means creating an output dataset with stable and predictable columns. The data can be normalized, regridded, scaled, and resolved using GADM to standard country names, as well as resolve the latitude,longitude of the event/measurement. A usual standard output will contain the following columns: `timestamp`, `country`, `admin1`, `admin2`, `admin3`, `lat`, `lng`, alongside other measurements/events/features of interest (additional columns to the right of the standard ones) contained within the input dataset.\n\n#TODO document standardization further\n\n## Transformation\n\nThe transformation functions include geographical extent clipping (latitude/longitude), geographical regridding (gridded data such as NetCDF or GeoTIFF), temporal clipping, and temporal scaling. \n\n### Geospatial Clipping\n\n`elwood.clip_geo(dataframe, geo_columns, polygons_list)`\n\nThis function takes a pandas dataframe, a geo_columns dict of the column names for latitude and longitude, ex:\n`{'lat_column': 'latitude', 'lon_column': 'longitude'}`, and a list containing lists of objects representing the polygons to clip the data to. ex: \n```\n[\n [\n {\n \"lat\": 11.0,\n \"lng\": 42.0\n },\n {\n \"lat\": 11.0,\n \"lng\": 43.0\n },\n {\n \"lat\": 12.0,\n \"lng\": 43.0\n },\n {\n \"lat\": 12.0,\n \"lng\": 42.0\n }\n ],\n ...\n]\n```\n### Geospatial regridding\n\n`elwood.regrid_dataframe_geo(dataframe, geo_columns, scale_multi)`\n\nThis function takes a dataframe and regrids it's geography by some scale multiplier that is provided. This multiplier will be used to divide the current geographical scale in order to make a more coarse grained resolution dataset. The dataframe must have a detectable geographical scale, meaning each lat/lon represents a point in the middle of a gridded cell for the data provided. Lat and lon and determined by the geo_columns passed in: a dict of the column names ex: `{'lat_column': 'my_latitude', 'lon_column': 'my_longitude'}`\n\n### Temporal Clipping\n`elwood.clip_dataframe_time(dataframe, time_column, time_ranges)`\n\nThis function will produce a dataframe that only includes rows with `time_column` values contained within `time_ranges`. The time_ranges argument is a list of objects containing a start and end time. ex: `[{\"start\": datetime, \"end\": datetime}, ...]`\n\n### Temporal Scaling\n`elwood.rescale_dataframe_time(dataframe, time_column, time_bucket, aggregation_function_list)`\n\nThis function will produce a dataframe who's rows are the aggregated data based on some time bucket and some aggregation function list provided. The `time_column` is the name of the column containing targeted time values for rescaling. The `time_bucket` is some DateOffset, Timedelta or str representing the desired time granularity, ex. `'M', 'A', '2H'`. The `aggregation_function_list` is a list of aggregation functions to apply to the data. ex. `['sum']` or `['sum', 'min', 'max']`\n\n## 0 to 1 Normalization\n\n`elwood.normalize_features(dataframe, output_file)`\n\nThis function expects a dataframe with a \"feature\" column and a \"value\" column, or long data. Each entry for a feature has its own feature/value row.\nThis function returns a dataframe in which all numerical values under the \"value\" column for each \"feature\" have been 0 to 1 scaled.\nOptionally you may specify an `output_file` name to generate a parquet file of the dataframe.\n\n\n# History\n\n0.1.4\nAdded new regridding functionality, updated existing numpy and pandas based regridding.\n\n0.1.4\n\n0.1.2\n\n0.1.1\n\n0.1.0\n\n\n",
"bugtrack_url": null,
"license": "MIT license",
"summary": "An open source dataset transformation, standardization, and normalization python library.",
"version": "0.1.4",
"project_urls": {
"Homepage": "https://github.com/jataware/elwood"
},
"split_keywords": [
"elwood"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "6fb9d638840a020594ae984d638c6222a77c4aa77617c02a0afe846f7034ef52",
"md5": "2887b8d77d3dc45e1ae8cb31dfa3061d",
"sha256": "4541e92580a60f1f08eedc4fa29b3f273acff8211fd72f44ea79702a890ef2a6"
},
"downloads": -1,
"filename": "elwood-0.1.4-py2.py3-none-any.whl",
"has_sig": false,
"md5_digest": "2887b8d77d3dc45e1ae8cb31dfa3061d",
"packagetype": "bdist_wheel",
"python_version": "py2.py3",
"requires_python": ">=3.8",
"size": 67142,
"upload_time": "2023-08-07T23:53:15",
"upload_time_iso_8601": "2023-08-07T23:53:15.585389Z",
"url": "https://files.pythonhosted.org/packages/6f/b9/d638840a020594ae984d638c6222a77c4aa77617c02a0afe846f7034ef52/elwood-0.1.4-py2.py3-none-any.whl",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-08-07 23:53:15",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "jataware",
"github_project": "elwood",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"requirements": [],
"tox": true,
"lcname": "elwood"
}
JSON
