========
insitupy
========
.. image:: https://img.shields.io/pypi/v/insitupy.svg
:target: https://pypi.python.org/pypi/insitupy
Manage reading and analyzing raw files of insitu data. The goal is to get
raw insitu data files into manageable classes with helpful functions and provide
access to the data as GeoPandas Arrays.
The first application of this will be for SnowEx pit data.
THIS IS A WORK IN PROGRESS (use at your own risk as outlined in the MIT license). We
fully welcome any contribution and ideas. Snow science works better together!
* Free software: MIT license
Features
--------
* Parsing of raw insitu data files with a variable number of header lines
* Reading csv data files into a pandas DataFrame, and parsing metadata into usable format
* Flexible user-defined variables
* Reading both pit and point data
* Parsing of date and location data
Definitions
-----------
Types of variables
~~~~~~~~~~~~~~~~~~
Insitupy uses two types of variable definitions to parse CSV information:
1. `primary variables` - These are the data that expected to be found in the
data columns. Think of them as the column headers that describe the data. In
the `example file <example_file_>` below it would be the last header row
indicated by the '#'.
2. `metadata variables` - These are the data that are expected to be found in
the header lines. These are assumed to start with a '#' sign by default. All
data above the last row with the '#' is assumed to be additional information
that describes the data.
Variable definitions
~~~~~~~~~~~~~~~~~~~~
Both variables types are defined the same way, in a separate yaml file.
A standard single variable definition looks like this:
.. code-block:: yaml
TOTAL_DEPTH: # <- YAML root key
code: total_depth
description: Total depth of measurement
map_from:
- total_snow_depth
- hs
match_on_code: true
auto_remap: true
* code: The string that will be used to reference this variable within the insitupy code
* description: A description of the variable
* map_from: A list of strings that will be used to match the entry as primary or metadata variables
* match_on_code: If true, the variable will be matched if the `code` values is found
in the data, not just the `map_from` values
* auto_remap: If true, the variable will be remapped to the `code` value
Overriding variables
~~~~~~~~~~~~~~~~~~~~
One can override a variable by defining the same `code` as in the default supplied
files within insitupy. The user definition always has precedence over the internal
ones.
Example
-------
I want to read in a file that looks like this:
::
# Location,East River
# Date/Local Standard Time,2020-04-27T08:45
# Latitude,38.92524
# Longitude,-106.97112
# Flags,random flag
# Top (cm),Bottom (cm),Density (kg/m3)
95.0,85.0,401.0
85.0,75.0,449.0
75.0,65.0,472.0
Parsing data
~~~~~~~~~~~~
The metadata, variable and data parsers are configured with a lot of defaults
for working with SnowEx data within insitupy, so the simple approach is
.. code-block:: python
from insitupy.campaigns.snowex import SnowExProfileData
my_data = SnowExProfileDataCollection.from_csv(
"./some_data.csv",
# Don't fail when there are unknown variables in the header
allow_map_failure=True
)
# Inspect the data
print(my_data.profiles[0].df)
# Look at the parsed metadata
print(my_data.profiles[0].metadata)
Defining your own variables
~~~~~~~~~~~~~~~~~~~~~~~~~~~
If you want to try your hand at defining variables yourself, you can do
as follows.
A user custom `metadata` YAML file:
::
LATITUDE:
auto_remap: true
code: latitude
description: Latitude
map_from:
- lat
- latitude
match_on_code: true
LONGITUDE:
auto_remap: true
code: longitude
description: Longitude
map_from:
- long
- lon
- longitude
DATETIME:
auto_remap: true
code: datetime
description: Combined date and time
map_from:
- Date/Local Standard Time
- date/local_standard_time
- datetime
- "date&time"
- date/time
- date/local_time
match_on_code: true
SITE_NAME:
auto_remap: true
code: site_name
description: Name of campaign site
map_from:
- location
match_on_code: true
and a separate primary variable YAML file:
::
BOTTOM_DEPTH:
auto_remap: true
code: bottom_depth
description: Lower edge of measurement
map_from:
- bottom
- bottom_depth
match_on_code: true
DENSITY:
auto_remap: true
code: density
description: measured snow density
map_from:
- density
- density_mean
match_on_code: true
DEPTH:
auto_remap: true
code: depth
description: top or center depth of measurement
map_from:
- depth
- top
match_on_code: true
LAYER_THICKNESS:
auto_remap: true
code: layer_thickness
description: thickness of layer
map_from: null
match_on_code: true
Save the two files to your local hard drive.
They will be used as arguments in Python code with the next step.
.. important::
LAYER_THICKNESS, DEPTH, and BOTTOM_DEPTH are required variables
for reading in **profile** data
Then use the new definitions and read in the file:
.. code-block:: python
from insitupy.campaigns.snowex import SnowExProfileData
my_data = SnowExProfileDataCollection.from_csv(
"./some_data.csv",
# Don't fail when there are unknown variables in the header
allow_map_failure=True,
# Use the files YOU defined here
primary_variable_files="/path/to/saved/primaryvariables.yaml",
metadata_variable_files="/path/to/saved/metadatavariables.yaml",
)
print(my_data.profiles[0].df)
print(my_data.profiles[0].metadata)
Credits
-------
This package was created with Cookiecutter_ and the `audreyr/cookiecutter-pypackage`_ project template.
.. _Cookiecutter: https://github.com/audreyr/cookiecutter
.. _`audreyr/cookiecutter-pypackage`: https://github.com/audreyr/cookiecutter-pypackage
=======
History
=======
0.1.0 (2024-03-27)
------------------
* First release on PyPI.
Raw data
{
"_id": null,
"home_page": "https://github.com/m3works/insitupy",
"name": "insitupy",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.6",
"maintainer_email": null,
"keywords": "insitupy",
"author": "M3 Works LLC",
"author_email": "info@m3works.io",
"download_url": "https://files.pythonhosted.org/packages/23/0c/460a236812298b47ca6716c29a5155a272a220222f14641c265d2e402a4f/insitupy-0.4.2.tar.gz",
"platform": null,
"description": "========\ninsitupy\n========\n\n\n.. image:: https://img.shields.io/pypi/v/insitupy.svg\n :target: https://pypi.python.org/pypi/insitupy\n\n\nManage reading and analyzing raw files of insitu data. The goal is to get\nraw insitu data files into manageable classes with helpful functions and provide\naccess to the data as GeoPandas Arrays.\n\nThe first application of this will be for SnowEx pit data.\n\nTHIS IS A WORK IN PROGRESS (use at your own risk as outlined in the MIT license). We\nfully welcome any contribution and ideas. Snow science works better together!\n\n\n* Free software: MIT license\n\n\nFeatures\n--------\n\n* Parsing of raw insitu data files with a variable number of header lines\n* Reading csv data files into a pandas DataFrame, and parsing metadata into usable format\n* Flexible user-defined variables\n* Reading both pit and point data\n* Parsing of date and location data\n\nDefinitions\n-----------\n\nTypes of variables\n~~~~~~~~~~~~~~~~~~\nInsitupy uses two types of variable definitions to parse CSV information:\n\n1. `primary variables` - These are the data that expected to be found in the\ndata columns. Think of them as the column headers that describe the data. In\nthe `example file <example_file_>` below it would be the last header row\nindicated by the '#'.\n2. `metadata variables` - These are the data that are expected to be found in\nthe header lines. These are assumed to start with a '#' sign by default. All\ndata above the last row with the '#' is assumed to be additional information\nthat describes the data.\n\nVariable definitions\n~~~~~~~~~~~~~~~~~~~~\nBoth variables types are defined the same way, in a separate yaml file.\nA standard single variable definition looks like this:\n\n.. code-block:: yaml\n\n TOTAL_DEPTH: # <- YAML root key\n code: total_depth\n description: Total depth of measurement\n map_from:\n - total_snow_depth\n - hs\n match_on_code: true\n auto_remap: true\n\n* code: The string that will be used to reference this variable within the insitupy code\n* description: A description of the variable\n* map_from: A list of strings that will be used to match the entry as primary or metadata variables\n* match_on_code: If true, the variable will be matched if the `code` values is found\n in the data, not just the `map_from` values\n* auto_remap: If true, the variable will be remapped to the `code` value\n\nOverriding variables\n~~~~~~~~~~~~~~~~~~~~\nOne can override a variable by defining the same `code` as in the default supplied\nfiles within insitupy. The user definition always has precedence over the internal\nones.\n\nExample\n-------\nI want to read in a file that looks like this:\n\n::\n\n # Location,East River\n # Date/Local Standard Time,2020-04-27T08:45\n # Latitude,38.92524\n # Longitude,-106.97112\n # Flags,random flag\n # Top (cm),Bottom (cm),Density (kg/m3)\n 95.0,85.0,401.0\n 85.0,75.0,449.0\n 75.0,65.0,472.0\n\nParsing data\n~~~~~~~~~~~~\nThe metadata, variable and data parsers are configured with a lot of defaults\nfor working with SnowEx data within insitupy, so the simple approach is\n\n.. code-block:: python\n\n from insitupy.campaigns.snowex import SnowExProfileData\n my_data = SnowExProfileDataCollection.from_csv(\n \"./some_data.csv\",\n # Don't fail when there are unknown variables in the header\n allow_map_failure=True\n )\n # Inspect the data\n print(my_data.profiles[0].df)\n # Look at the parsed metadata\n print(my_data.profiles[0].metadata)\n\nDefining your own variables\n~~~~~~~~~~~~~~~~~~~~~~~~~~~\nIf you want to try your hand at defining variables yourself, you can do\nas follows.\n\nA user custom `metadata` YAML file:\n\n::\n\n LATITUDE:\n auto_remap: true\n code: latitude\n description: Latitude\n map_from:\n - lat\n - latitude\n match_on_code: true\n LONGITUDE:\n auto_remap: true\n code: longitude\n description: Longitude\n map_from:\n - long\n - lon\n - longitude\n DATETIME:\n auto_remap: true\n code: datetime\n description: Combined date and time\n map_from:\n - Date/Local Standard Time\n - date/local_standard_time\n - datetime\n - \"date&time\"\n - date/time\n - date/local_time\n match_on_code: true\n SITE_NAME:\n auto_remap: true\n code: site_name\n description: Name of campaign site\n map_from:\n - location\n match_on_code: true\n\nand a separate primary variable YAML file:\n\n::\n\n BOTTOM_DEPTH:\n auto_remap: true\n code: bottom_depth\n description: Lower edge of measurement\n map_from:\n - bottom\n - bottom_depth\n match_on_code: true\n DENSITY:\n auto_remap: true\n code: density\n description: measured snow density\n map_from:\n - density\n - density_mean\n match_on_code: true\n DEPTH:\n auto_remap: true\n code: depth\n description: top or center depth of measurement\n map_from:\n - depth\n - top\n match_on_code: true\n LAYER_THICKNESS:\n auto_remap: true\n code: layer_thickness\n description: thickness of layer\n map_from: null\n match_on_code: true\n\nSave the two files to your local hard drive.\nThey will be used as arguments in Python code with the next step.\n\n.. important::\n\n LAYER_THICKNESS, DEPTH, and BOTTOM_DEPTH are required variables\n for reading in **profile** data\n\nThen use the new definitions and read in the file:\n\n.. code-block:: python\n\n from insitupy.campaigns.snowex import SnowExProfileData\n my_data = SnowExProfileDataCollection.from_csv(\n \"./some_data.csv\",\n # Don't fail when there are unknown variables in the header\n allow_map_failure=True,\n # Use the files YOU defined here\n primary_variable_files=\"/path/to/saved/primaryvariables.yaml\",\n metadata_variable_files=\"/path/to/saved/metadatavariables.yaml\",\n )\n print(my_data.profiles[0].df)\n print(my_data.profiles[0].metadata)\n\n\nCredits\n-------\n\nThis package was created with Cookiecutter_ and the `audreyr/cookiecutter-pypackage`_ project template.\n\n.. _Cookiecutter: https://github.com/audreyr/cookiecutter\n.. _`audreyr/cookiecutter-pypackage`: https://github.com/audreyr/cookiecutter-pypackage\n\n\n=======\nHistory\n=======\n\n0.1.0 (2024-03-27)\n------------------\n\n* First release on PyPI.\n",
"bugtrack_url": null,
"license": "MIT license",
"summary": "Weave insitu data together",
"version": "0.4.2",
"project_urls": {
"Homepage": "https://github.com/m3works/insitupy"
},
"split_keywords": [
"insitupy"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "3f25b07ab2a813652ac14260459f894256ab58c5b9279a971e8e427a237594ee",
"md5": "c332b1181a04dba3626a2620239a0bea",
"sha256": "af1cd89eece971a58c8b535f3fe7d1d740c8909dfa6dfceabe0f01095768924e"
},
"downloads": -1,
"filename": "insitupy-0.4.2-py2.py3-none-any.whl",
"has_sig": false,
"md5_digest": "c332b1181a04dba3626a2620239a0bea",
"packagetype": "bdist_wheel",
"python_version": "py2.py3",
"requires_python": ">=3.6",
"size": 31639,
"upload_time": "2025-07-15T16:40:57",
"upload_time_iso_8601": "2025-07-15T16:40:57.635760Z",
"url": "https://files.pythonhosted.org/packages/3f/25/b07ab2a813652ac14260459f894256ab58c5b9279a971e8e427a237594ee/insitupy-0.4.2-py2.py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "230c460a236812298b47ca6716c29a5155a272a220222f14641c265d2e402a4f",
"md5": "de935162d6d06f52b3a4a6d56e559ea7",
"sha256": "ac2452724df950f58afb8f9af68c789c9e7fd7479646912e4898101773b53201"
},
"downloads": -1,
"filename": "insitupy-0.4.2.tar.gz",
"has_sig": false,
"md5_digest": "de935162d6d06f52b3a4a6d56e559ea7",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.6",
"size": 41342,
"upload_time": "2025-07-15T16:40:58",
"upload_time_iso_8601": "2025-07-15T16:40:58.850444Z",
"url": "https://files.pythonhosted.org/packages/23/0c/460a236812298b47ca6716c29a5155a272a220222f14641c265d2e402a4f/insitupy-0.4.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-07-15 16:40:58",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "m3works",
"github_project": "insitupy",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "insitupy"
}