| Name | psense-common JSON |
| Version |
0.4.11
JSON |
| download |
| home_page | |
| Summary | PercuSense Common Modules |
| upload_time | 2023-11-09 21:06:43 |
| maintainer | |
| docs_url | None |
| author | PercuSense |
| requires_python | >3.9,<3.13 |
| license | |
| keywords |
|
| VCS |
|
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# PSENSE-COMMON
[](https://pypi.org/project/psense-common/)
[](./LICENSE)
Common Modules used by internal PercuSense python applications. This project follows semantic versioning (breaking.major.minor release).
## Getting Started
### General Instructions
Steps to try this out yourself:
1. Install the `psense-common` library:
#### From PyPi
```bash
$ pip install psense-common
```
#### Local Installation
```bash
$ git clone git@bitbucket.org:psense/psense-common.git
$ python setup.py install
```
2. Import the modules you need in your script.
```python
from psense_common import PSenseParser)
```
3. Access module methods directly.
```python
from psense_common import PSenseParser
filename = '/path/to/data/file'
parser = PSenseParser()
parser.identify_file_source(filename)
if parser.source:
parser.load_rawfile(filename)
print(parser.data) # this is of type pandas.DataFrame
```
### AWS
`psense_common/psense_aws_itfc.py` introduces the class `PSenseAWSInterface`, which provides an interface to the Amazon DynamoDB datastore. Currently, data is stored on a per-record basis (NoSQL) for Experiment, Sensor, and Event data ("Experiments", "SensorData", and "Calibration" -- not fully supported, will be changed to "EventData" in migration from 0.x.x to 1.x.x).
The class contains query and put operations but does not provide provisions for deleting items from the datastore.
#### Credentials
At initialization, the class will create an AWS session that is persisted. The session object allows the user to authenticate just a single time (rather than for every communication).
In generating the session, bot3 will attempt to load a profile (if not provided, will try "PShield") that contains the necessary credentials for authentication/authorization to our dynamodb instances. If the profile doesn't exist or cannot be loaded, the class reverts to the environment default AWS credentials in the 'us-east-1' region.
`config` and `credential` files for aws are stored in the user's home directory: `~/.aws/`. In Windows, the comparable location is `C:\Users\[username]\.aws\`
#### Usage
**Get Sensor Data**
```python
from psense_common import PSenseAWSInterface
import pytz
aws = PSenseAWSInterface(debugmode=True)
aws.set_query_config(req_size=7200, query_count=2)
experiments = ['[my experiment id 1]',
'[my experiment id 2]']
for expid in experiments:
if not aws.verif_experiment(expid):
print('invalid experiment id skipping {}'.format(expid))
pass
count, data = aws.get_sensor_data()
data.index = data.index.tz_localize(pytz.utc).tz_convert(localtz)
print(data)
```
**Add Experiment and real-time Sensor data**
```python
from psense_common import (PSenseAWSInterface, PSenseParser)
# user variables
filename = '[vfp600 gamry file].txt'
expid = 'experimentid'
# initialize classes + helper func
parser = PSenseParser(debugmode=True)
aws = PSenseAWSInterface(debugmode=True)
aws.set_query_config(req_size=7200, query_count=2)
def tail(fin):
"Listen for new lines added to file."
while True:
where = fin.tell()
line = fin.readline()
if not line:
time.sleep(SLEEP_INTERVAL)
fin.seek(where)
else:
yield line
# identify the type of file we are parsing
if not parser.identify_file_source(filename):
print('unknown file type. exiting')
assert False
# confirm that the experiment id is valid
if not aws.verif_experiment(expid):
print('invalid experiment id skipping {}'.format(expid))
assert False
# add experiment to database
add_experiment_success, if_fail_reason = aws.add_experiment()
if if_fail_reason == 'error':
print('Aborted. Error occurred in communication with AWS.')
assert False
elif if_fail_reason == 'exists':
print('Experiment already exists in database. Must delete existing experiment sensor data before uploading new values.')
# "tail" the file and send each record to ddb
with open(filename, 'r') as fin:
# skip to the end of the file before beginning tail
fin.seek(0, os.SEEK_END)
for line in tail(fin):
row = parser.parse_record(line.strip())
aws.add_sensordata(*row)
```
*Future notes: Sensor and Event data should remain as a blob-store -- we expect schema to change depending on the type of sensor (or event). For example, sensors with 3 working electrodes should contain more properties than sensors with a single signal.*
### Experiment ID Formatting
All sensor data is associated with a particular experiment ID. Experiment IDs should be unique and follow the PercuSense naming scheme.
#### Usage
Decoding an experiment:
```python
from psense_common import (psense_format)
experiment = '[my experiment id]'
validate_name = psense_format.psense_exp_title()
validate_name.decode(experiment)
```
Generating an experiment id through console IO (must provide 3-digit device id):
```python
from psense_common import (psense_format)
device_id = 'P01'
experiment = psense_format.setup_new_experiment(device_id)
```
### Data Parsing
The PSenseParser class will parse data from flat-file into Pandas DataFrame. Output object will contain columns appropriate for PercuSense analysis scripts (and PercuSense Data Viewer web application). The parser works on individual records (live streaming) as well as full files (retrospective analysis).
Supported formats:
```bash
BWII 2-channel
BWII 3-Channel ("BWII-MINI")
PSHIELD
GAMRY VFP600
GAMRY EXPLAIN
DATA VIEWER single-channel
DATA VIEWER 2-channel
DATA VIEWER 3-channel
CH Instruments (txt)
```
TODO: documentation pending
Check `demo/parser.py` for some example usage.
See above sections (General Instructions, AWS) for basic usage.
### PercuSense Data Filter
TODO: documentation pending
## Development
### Tests
```bash
$ python setup.py test
```
.. or with code coverage (`pip install --upgrade coverage`):
```bash
$ coverage run --source=psense_common/ setup.py test
$ coverage report -m
```
### Publishing
Bitbucket has been configured to run tests and publish directly to pypi after code is merged to master. See `bitbucket-pipelines.yml` for configuration details.
Manual publishing (not recommended):
```
$ rm -rf dist
$ python setup.py sdist
$ twine upload dist/*
```
Raw data
{
"_id": null,
"home_page": "",
"name": "psense-common",
"maintainer": "",
"docs_url": null,
"requires_python": ">3.9,<3.13",
"maintainer_email": "",
"keywords": "",
"author": "PercuSense",
"author_email": "info@percusense.com",
"download_url": "https://files.pythonhosted.org/packages/fa/ab/ecb8a6167eb79268b9ef64d98c39cba4622bc169be7bfd289561ef408437/psense_common-0.4.11.tar.gz",
"platform": null,
"description": "# PSENSE-COMMON\n\n[](https://pypi.org/project/psense-common/)\n\n[](./LICENSE)\n\n\nCommon Modules used by internal PercuSense python applications. This project follows semantic versioning (breaking.major.minor release).\n\n## Getting Started\n\n### General Instructions\n\nSteps to try this out yourself:\n\n1. Install the `psense-common` library:\n\n#### From PyPi\n\n```bash\n$ pip install psense-common\n```\n\n#### Local Installation\n\n```bash\n$ git clone git@bitbucket.org:psense/psense-common.git\n$ python setup.py install\n```\n\n2. Import the modules you need in your script.\n\n```python\nfrom psense_common import PSenseParser)\n```\n\n3. Access module methods directly.\n\n```python\nfrom psense_common import PSenseParser\n\nfilename = '/path/to/data/file'\nparser = PSenseParser()\nparser.identify_file_source(filename)\n\nif parser.source:\n parser.load_rawfile(filename)\n print(parser.data) # this is of type pandas.DataFrame\n```\n\n### AWS\n\n`psense_common/psense_aws_itfc.py` introduces the class `PSenseAWSInterface`, which provides an interface to the Amazon DynamoDB datastore. Currently, data is stored on a per-record basis (NoSQL) for Experiment, Sensor, and Event data (\"Experiments\", \"SensorData\", and \"Calibration\" -- not fully supported, will be changed to \"EventData\" in migration from 0.x.x to 1.x.x).\n\nThe class contains query and put operations but does not provide provisions for deleting items from the datastore.\n\n#### Credentials\n\nAt initialization, the class will create an AWS session that is persisted. The session object allows the user to authenticate just a single time (rather than for every communication).\n\nIn generating the session, bot3 will attempt to load a profile (if not provided, will try \"PShield\") that contains the necessary credentials for authentication/authorization to our dynamodb instances. If the profile doesn't exist or cannot be loaded, the class reverts to the environment default AWS credentials in the 'us-east-1' region.\n\n`config` and `credential` files for aws are stored in the user's home directory: `~/.aws/`. In Windows, the comparable location is `C:\\Users\\[username]\\.aws\\`\n\n#### Usage\n\n**Get Sensor Data**\n\n```python\nfrom psense_common import PSenseAWSInterface\nimport pytz\n\naws = PSenseAWSInterface(debugmode=True)\naws.set_query_config(req_size=7200, query_count=2)\n\nexperiments = ['[my experiment id 1]',\n '[my experiment id 2]']\n\nfor expid in experiments:\n if not aws.verif_experiment(expid):\n print('invalid experiment id skipping {}'.format(expid))\n pass\n\n count, data = aws.get_sensor_data()\n data.index = data.index.tz_localize(pytz.utc).tz_convert(localtz)\n\n print(data)\n```\n\n**Add Experiment and real-time Sensor data**\n\n```python\nfrom psense_common import (PSenseAWSInterface, PSenseParser)\n\n# user variables\nfilename = '[vfp600 gamry file].txt'\nexpid = 'experimentid'\n\n# initialize classes + helper func\nparser = PSenseParser(debugmode=True)\naws = PSenseAWSInterface(debugmode=True)\naws.set_query_config(req_size=7200, query_count=2)\n\ndef tail(fin):\n \"Listen for new lines added to file.\"\n while True:\n where = fin.tell() \n line = fin.readline()\n if not line:\n time.sleep(SLEEP_INTERVAL)\n fin.seek(where)\n else:\n yield line\n\n# identify the type of file we are parsing\nif not parser.identify_file_source(filename):\n print('unknown file type. exiting')\n assert False\n\n\n# confirm that the experiment id is valid\nif not aws.verif_experiment(expid):\n print('invalid experiment id skipping {}'.format(expid))\n assert False\n\n\n# add experiment to database\nadd_experiment_success, if_fail_reason = aws.add_experiment()\nif if_fail_reason == 'error':\n print('Aborted. Error occurred in communication with AWS.')\n assert False\nelif if_fail_reason == 'exists':\n print('Experiment already exists in database. Must delete existing experiment sensor data before uploading new values.')\n\n\n# \"tail\" the file and send each record to ddb\nwith open(filename, 'r') as fin:\n # skip to the end of the file before beginning tail\n fin.seek(0, os.SEEK_END)\n\n for line in tail(fin):\n row = parser.parse_record(line.strip())\n aws.add_sensordata(*row)\n```\n\n*Future notes: Sensor and Event data should remain as a blob-store -- we expect schema to change depending on the type of sensor (or event). For example, sensors with 3 working electrodes should contain more properties than sensors with a single signal.*\n\n### Experiment ID Formatting\n\nAll sensor data is associated with a particular experiment ID. Experiment IDs should be unique and follow the PercuSense naming scheme.\n\n#### Usage\n\nDecoding an experiment:\n\n```python\nfrom psense_common import (psense_format)\n\nexperiment = '[my experiment id]'\nvalidate_name = psense_format.psense_exp_title()\nvalidate_name.decode(experiment)\n```\n\nGenerating an experiment id through console IO (must provide 3-digit device id):\n\n```python\nfrom psense_common import (psense_format)\n\ndevice_id = 'P01'\nexperiment = psense_format.setup_new_experiment(device_id)\n```\n\n### Data Parsing\n\nThe PSenseParser class will parse data from flat-file into Pandas DataFrame. Output object will contain columns appropriate for PercuSense analysis scripts (and PercuSense Data Viewer web application). The parser works on individual records (live streaming) as well as full files (retrospective analysis).\n\nSupported formats:\n\n```bash\nBWII 2-channel\nBWII 3-Channel (\"BWII-MINI\")\nPSHIELD\nGAMRY VFP600\nGAMRY EXPLAIN\nDATA VIEWER single-channel\nDATA VIEWER 2-channel\nDATA VIEWER 3-channel\nCH Instruments (txt)\n```\n\nTODO: documentation pending\n\nCheck `demo/parser.py` for some example usage.\n\nSee above sections (General Instructions, AWS) for basic usage.\n\n### PercuSense Data Filter\n\nTODO: documentation pending\n\n## Development\n\n### Tests\n\n```bash\n$ python setup.py test\n```\n\n.. or with code coverage (`pip install --upgrade coverage`):\n```bash\n$ coverage run --source=psense_common/ setup.py test\n$ coverage report -m\n```\n\n### Publishing\n\nBitbucket has been configured to run tests and publish directly to pypi after code is merged to master. See `bitbucket-pipelines.yml` for configuration details.\n\nManual publishing (not recommended):\n```\n$ rm -rf dist\n$ python setup.py sdist\n$ twine upload dist/*\n```\n",
"bugtrack_url": null,
"license": "",
"summary": "PercuSense Common Modules",
"version": "0.4.11",
"project_urls": null,
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "c705d851b2e5fe1e123e9dab64fa688ad080e3a70eff1ec309ac221c2f63f3a6",
"md5": "8d545244d386eceabc95f3c38ad15489",
"sha256": "598553ada2d20d7818929900564fd877fba4cb7f3ac0ccb5590ea30359d80f26"
},
"downloads": -1,
"filename": "psense_common-0.4.11-py3-none-any.whl",
"has_sig": false,
"md5_digest": "8d545244d386eceabc95f3c38ad15489",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">3.9,<3.13",
"size": 30151,
"upload_time": "2023-11-09T21:06:41",
"upload_time_iso_8601": "2023-11-09T21:06:41.517231Z",
"url": "https://files.pythonhosted.org/packages/c7/05/d851b2e5fe1e123e9dab64fa688ad080e3a70eff1ec309ac221c2f63f3a6/psense_common-0.4.11-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "faabecb8a6167eb79268b9ef64d98c39cba4622bc169be7bfd289561ef408437",
"md5": "42558c40aa03e7879621980254f4bd4d",
"sha256": "8604d55f43b9ef60e543ae2179f2a9fc642b4a69a024389f449a78b4df77d0c2"
},
"downloads": -1,
"filename": "psense_common-0.4.11.tar.gz",
"has_sig": false,
"md5_digest": "42558c40aa03e7879621980254f4bd4d",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">3.9,<3.13",
"size": 29279,
"upload_time": "2023-11-09T21:06:43",
"upload_time_iso_8601": "2023-11-09T21:06:43.291539Z",
"url": "https://files.pythonhosted.org/packages/fa/ab/ecb8a6167eb79268b9ef64d98c39cba4622bc169be7bfd289561ef408437/psense_common-0.4.11.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-11-09 21:06:43",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "psense-common"
}
