# Python Dataware Toolkit
A Python toolkit for managing, retrieving, and processing data.
## Installation
You can install the toolkit with:
```bash
$ pip3 install pydtk
```
If you want to install the toolkit with extra feature (e.g. support for PointCloud and ROS),
you can install it with extra dependencies as follows:
```bash
$ pip3 install pydtk[pointcloud,ros]
```
Some PyDTK models require additional packages.
Please refer the following table and install them manually with command `pip install ...`.
| PyDTK model | Required packages |
| --- | --- |
| rosbag.* | ros_numpy (https://github.com/eric-wieser/ros_numpy.git) |
| pointcloud.PCDModel | pypcd (https://github.com/klintan/pypcd.git) |
## Usage
By using Pydtk, you can load a variety of types of data with a unified interface as shown below.
1. Load DBHandler for retrieving metadata
```python
from pydtk.db import DBHandler
# Initialize handler (This will read all the metadata from DB on initialization)
handler = DBHandler(
db_class='meta',
db_host='./examples/example_db',
base_dir_path='./test'
)
```
2. Read metadata from db with data selection.
```python
# Select by timestamp
handler.read(pql='start_timestamp > 1420000000 and end_timestamp < 1500000000')
print(handler.data)
# Select by record-id
handler.read(pql='record_id == regex("test.*")')
print(handler.data)
```
3. Load data from files based on metadata.
```python
from pydtk.io import BaseFileReader, NoModelMatchedError
reader = BaseFileReader()
try:
for sample in handler:
print('loading content "{0}" from file "{1}"'.format(sample['contents'], sample['path']))
try:
timestamps, data, columns = reader.read(sample)
assert print(data)
except NoModelMatchedError as e:
print(str(e))
continue
except EOFError:
pass
```
## Documentation
For more information about this toolkit, please refer the [document](https://dataware-tools.github.io/pydtk/).
## Setup for contribution
To improve this toolkit, firstly clone this repository and then
run the following command to prepare the environment.
```bash
$ git clone git@github.com:dataware-tools/pydtk.git --recurse-submodules
$ poetry install
```
Make sure that [poetry](https://python-poetry.org/) is installed before executing the command.
If you want to install the toolkit with extra feature (e.g. support for ROS),
please specify it with `-E` option.
Example (installation with `pointcloud` and `ros` extras):
```bash
$ poetry install -E pointcloud -E ros
```
Raw data
{
"_id": null,
"home_page": "https://github.com/dataware-tools/python-toolkit.git",
"name": "pydtk",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.8,<3.11",
"maintainer_email": "",
"keywords": "toolkit,data,dataware,metadata",
"author": "Yusuke Adachi",
"author_email": "adachi.yusuke@hdwlab.co.jp",
"download_url": "https://files.pythonhosted.org/packages/34/da/4193b839ce60435ddc182996c2ecdf509c8d37baca48d8fb55eed46fc25f/pydtk-0.3.1.tar.gz",
"platform": null,
"description": "# Python Dataware Toolkit\n\nA Python toolkit for managing, retrieving, and processing data.\n\n## Installation\nYou can install the toolkit with:\n```bash\n$ pip3 install pydtk\n\n```\n\nIf you want to install the toolkit with extra feature (e.g. support for PointCloud and ROS),\nyou can install it with extra dependencies as follows:\n```bash\n$ pip3 install pydtk[pointcloud,ros]\n\n```\n\nSome PyDTK models require additional packages. \nPlease refer the following table and install them manually with command `pip install ...`. \n\n| PyDTK model | Required packages |\n| --- | --- |\n| rosbag.* | ros_numpy (https://github.com/eric-wieser/ros_numpy.git) |\n| pointcloud.PCDModel | pypcd (https://github.com/klintan/pypcd.git) |\n\n\n## Usage\n\nBy using Pydtk, you can load a variety of types of data with a unified interface as shown below.\n\n1. Load DBHandler for retrieving metadata\n```python\nfrom pydtk.db import DBHandler\n\n# Initialize handler (This will read all the metadata from DB on initialization)\nhandler = DBHandler(\n db_class='meta',\n db_host='./examples/example_db',\n base_dir_path='./test'\n)\n\n```\n\n2. Read metadata from db with data selection.\n```python\n# Select by timestamp\nhandler.read(pql='start_timestamp > 1420000000 and end_timestamp < 1500000000')\nprint(handler.data)\n\n# Select by record-id\nhandler.read(pql='record_id == regex(\"test.*\")')\nprint(handler.data)\n\n```\n\n3. Load data from files based on metadata.\n```python\nfrom pydtk.io import BaseFileReader, NoModelMatchedError\n\nreader = BaseFileReader()\n\ntry:\n for sample in handler:\n print('loading content \"{0}\" from file \"{1}\"'.format(sample['contents'], sample['path']))\n try:\n timestamps, data, columns = reader.read(sample)\n assert print(data)\n except NoModelMatchedError as e:\n print(str(e))\n continue\nexcept EOFError:\n pass\n```\n\n\n## Documentation\nFor more information about this toolkit, please refer the [document](https://dataware-tools.github.io/pydtk/).\n\n\n## Setup for contribution\nTo improve this toolkit, firstly clone this repository and then\nrun the following command to prepare the environment.\n\n```bash\n$ git clone git@github.com:dataware-tools/pydtk.git --recurse-submodules\n$ poetry install\n\n```\n\nMake sure that [poetry](https://python-poetry.org/) is installed before executing the command.\n\nIf you want to install the toolkit with extra feature (e.g. support for ROS),\nplease specify it with `-E` option. \nExample (installation with `pointcloud` and `ros` extras):\n```bash\n$ poetry install -E pointcloud -E ros\n\n```\n",
"bugtrack_url": null,
"license": "Apache-2.0",
"summary": "A Python toolkit for managing, retrieving and processing data.",
"version": "0.3.1",
"project_urls": {
"Homepage": "https://github.com/dataware-tools/python-toolkit.git",
"Repository": "https://github.com/dataware-tools/python-toolkit.git"
},
"split_keywords": [
"toolkit",
"data",
"dataware",
"metadata"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "f12ba7fb4cfebce0f05ab62ecc1bbef48338396228fa9f5b3c65450d0e20a83d",
"md5": "778582f5726bbacf55f27f6307819bbf",
"sha256": "eb5ea59cff51f04b08e62657d38aaa238d1d4aab8a657b1b8e6646fcab384eb7"
},
"downloads": -1,
"filename": "pydtk-0.3.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "778582f5726bbacf55f27f6307819bbf",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8,<3.11",
"size": 163312,
"upload_time": "2023-06-28T01:26:15",
"upload_time_iso_8601": "2023-06-28T01:26:15.349534Z",
"url": "https://files.pythonhosted.org/packages/f1/2b/a7fb4cfebce0f05ab62ecc1bbef48338396228fa9f5b3c65450d0e20a83d/pydtk-0.3.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "34da4193b839ce60435ddc182996c2ecdf509c8d37baca48d8fb55eed46fc25f",
"md5": "f5f317ada2e3d182ea0dad0075779ae7",
"sha256": "a16c9dfa16f5e7b54761f8382291d3c183b7d296e6d2552fc0331f8d69d160f0"
},
"downloads": -1,
"filename": "pydtk-0.3.1.tar.gz",
"has_sig": false,
"md5_digest": "f5f317ada2e3d182ea0dad0075779ae7",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8,<3.11",
"size": 102054,
"upload_time": "2023-06-28T01:26:17",
"upload_time_iso_8601": "2023-06-28T01:26:17.059942Z",
"url": "https://files.pythonhosted.org/packages/34/da/4193b839ce60435ddc182996c2ecdf509c8d37baca48d8fb55eed46fc25f/pydtk-0.3.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-06-28 01:26:17",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "dataware-tools",
"github_project": "python-toolkit",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"tox": true,
"lcname": "pydtk"
}
JSON
