# mcap2timescale
mcap2timescale transforms `.mcap` and `.bag` (ROS bag) files into a Timescale database. You can now query any robotics data using SQL without writing ETLs.
## Hosted solution
We will soon offer a hosted solution with a number of perks:
* Managed ingestion, database, and infrastructure for integrations (e.g. S3, Grafana).
* Reverse Timescale to `.mcap` and `.bag` generator.
* Vector search for imagery, audio, and other unstructured data.
* Web interface to track and share data with your team.
## Running yourself
1. Set up a Timescale database. Take note of all connection details (host, port, username, password, and database name).
2. Install this package.
```
pip install mcap2timescale
```
3. Run the job against any file. The database connection arguments are optional.
```
mcap2timescale /path/to/file.mcap \
--host localhost \
--port 5432 \
--user postgres \
--password password \
--name postgres
```
4. Connect to your database and run any query.
## Design considerations
* __Timescale Hypertables__: this project creates hypertables with Timescale. Due to the sheer quantity of messages, it is critical make use of Timescale's compression features.
* __No ROS Dependency:__ this project runs without any ROS dependencies. It is a pain to install ROS simply to extract data from ROS bag or MCAP files. We leverage the `rosbags` project to accomplish this by dynamically loading message schemas at runtime.
* __Transform MCAP to ROS bag__: this project converts all MCAP files into ROS bags before transforming the data to avoid needing to rewrite the ingestion flow.
Raw data
{
"_id": null,
"home_page": "https://github.com/SensorSurf/mcap2timescale",
"name": "mcap2timescale",
"maintainer": "",
"docs_url": null,
"requires_python": "",
"maintainer_email": "",
"keywords": "ros,ros1,rosbag,mcap,timescale,etl,timeseries,database,etl",
"author": "SensorSurf",
"author_email": "support@sensorsurf.com",
"download_url": "https://files.pythonhosted.org/packages/2f/dc/b50db5eedaad7314436e798e7006ecf8845555b790ccc8a246a3e416a8dd/mcap2timescale-0.1.2.tar.gz",
"platform": null,
"description": "# mcap2timescale\n\nmcap2timescale transforms `.mcap` and `.bag` (ROS bag) files into a Timescale database. You can now query any robotics data using SQL without writing ETLs.\n\n## Hosted solution\n\nWe will soon offer a hosted solution with a number of perks:\n* Managed ingestion, database, and infrastructure for integrations (e.g. S3, Grafana).\n* Reverse Timescale to `.mcap` and `.bag` generator.\n* Vector search for imagery, audio, and other unstructured data.\n* Web interface to track and share data with your team.\n\n## Running yourself\n\n1. Set up a Timescale database. Take note of all connection details (host, port, username, password, and database name).\n\n2. Install this package.\n```\npip install mcap2timescale\n```\n\n3. Run the job against any file. The database connection arguments are optional.\n```\nmcap2timescale /path/to/file.mcap \\\n --host localhost \\\n --port 5432 \\\n --user postgres \\\n --password password \\\n --name postgres\n```\n\n4. Connect to your database and run any query.\n\n## Design considerations\n\n* __Timescale Hypertables__: this project creates hypertables with Timescale. Due to the sheer quantity of messages, it is critical make use of Timescale's compression features.\n* __No ROS Dependency:__ this project runs without any ROS dependencies. It is a pain to install ROS simply to extract data from ROS bag or MCAP files. We leverage the `rosbags` project to accomplish this by dynamically loading message schemas at runtime.\n* __Transform MCAP to ROS bag__: this project converts all MCAP files into ROS bags before transforming the data to avoid needing to rewrite the ingestion flow.\n\n\n",
"bugtrack_url": null,
"license": "",
"summary": "Transform mcap (or rosbag) files into a Timescale database",
"version": "0.1.2",
"project_urls": {
"Bug Reports": "https://github.com/SensorSurf/mcap2timescale/issues",
"Homepage": "https://github.com/SensorSurf/mcap2timescale",
"Source": "https://github.com/SensorSurf/mcap2timescale"
},
"split_keywords": [
"ros",
"ros1",
"rosbag",
"mcap",
"timescale",
"etl",
"timeseries",
"database",
"etl"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "d4024a48a8e4db3b28a7fe2a508cae24ac755df73a66dcf54262efd617849173",
"md5": "e04629b8be7c3edcbd992d2d30ef7f83",
"sha256": "a4e8d0e02ddb4d378add9bc784d8142c87b8c63a248d74411205185965077459"
},
"downloads": -1,
"filename": "mcap2timescale-0.1.2-py3-none-any.whl",
"has_sig": false,
"md5_digest": "e04629b8be7c3edcbd992d2d30ef7f83",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 6951,
"upload_time": "2023-06-06T03:07:00",
"upload_time_iso_8601": "2023-06-06T03:07:00.742495Z",
"url": "https://files.pythonhosted.org/packages/d4/02/4a48a8e4db3b28a7fe2a508cae24ac755df73a66dcf54262efd617849173/mcap2timescale-0.1.2-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "2fdcb50db5eedaad7314436e798e7006ecf8845555b790ccc8a246a3e416a8dd",
"md5": "5825b03c9b220614f4556462abb1fa14",
"sha256": "a3444a9225cd9ddec7edfa2464708f0cc5aa499f8054cd8f8066c79423d6b37d"
},
"downloads": -1,
"filename": "mcap2timescale-0.1.2.tar.gz",
"has_sig": false,
"md5_digest": "5825b03c9b220614f4556462abb1fa14",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 6021,
"upload_time": "2023-06-06T03:07:02",
"upload_time_iso_8601": "2023-06-06T03:07:02.117814Z",
"url": "https://files.pythonhosted.org/packages/2f/dc/b50db5eedaad7314436e798e7006ecf8845555b790ccc8a246a3e416a8dd/mcap2timescale-0.1.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-06-06 03:07:02",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "SensorSurf",
"github_project": "mcap2timescale",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"requirements": [],
"lcname": "mcap2timescale"
}
JSON
