| Name | chroma-migrate JSON |
| Version |
0.0.7
JSON |
| download |
| home_page | |
| Summary | A tool for migrating to chroma versions >= 0.4.0 |
| upload_time | 2023-08-04 03:20:52 |
| maintainer | |
| docs_url | None |
| author | |
| requires_python | >=3.7 |
| license | |
| keywords |
chroma
migrate
|
| VCS |
|
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
## Chroma Migrate
Schema and data format changes are a necessary evil of evolving software. We take changes seriously and make them infrequently and only when necessary.
Chroma's commitment is whenever schema or data format change, we will provide a seamless and easy-to-use migration tool to move to the new schema/format.
Specifically we will announce schema changes on:
- Discord ([#migrations channel](https://discord.com/channels/1073293645303795742/1129286514845691975))
- Github (here)
- Email listserv [Sign up](https://airtable.com/shrHaErIs1j9F97BE)
We will aim to provide:
- a description of the change and the rationale for the change.
- a CLI migration tool you can run
- a video walkthrough of using the tool
### Migration Log
#### Migration from >0.4.0 to 0.4.0 - July 17, 2023
We are migrating:
- `metadata store`: where metadata is stored
- `index on disk`: how indexes are stored on disk
`Metadata Store`: Previously Chroma used underlying storage engines `DuckDB` for the `in-memory` version of Chroma, and `Clickhouse` for the `single-node server` version of Chroma. These decisions were made when Chroma was addressing more batch analytical workloads and are no longer the best choice for users. The new metadata store for the `in-memory` and `single-node server` version of Chroma will be `sqlite`. (The distributed version of Chroma (forthcoming), will use a different distributed metadata store.)
`Index store`: Previously Chroma saved the **entire** index on every write. This because painfully slow when the collection grew to a reasonable amount of embeddings. The new index store saves *only the change* and should scale seamlessly!
Here are the 9-possible migration paths, and any notes, if applicable.
| From 👇 ➡️ To 👉 | Persistent Chroma | Local Chroma Server | Remote Chroma Server |
| -------- | -------- | -------- | -------- |
| Persistent Chroma | ✅ | ✅ | 1️⃣ |
| Local Chroma Server| ✅| 2️⃣| 1️⃣|
| Remote Chroma Server| ✅| ✅| 1️⃣ 2️⃣|
1️⃣ - Make sure to configure any auth headers correctly
2️⃣ - Run both the existing version of Chroma and the new `0.4.0` version of Chroma at same time. Run the new version on a new port if local.
[Embed video here]()
##### How to use the migration tool
1. `pip` install this utility. `pip install chroma_migrate`
1. Running the CLI. In your terminal run:
```
chroma_migrate
```
2. Choose whether the data you want to migrate is locally on disk (duckdb) on clickhouse instance used by chroma, or directly from another chroma server
3. Choose where you want to write the new data to.
### Developing Locally
Run python main.py to test locally
Raw data
{
"_id": null,
"home_page": "",
"name": "chroma-migrate",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.7",
"maintainer_email": "",
"keywords": "chroma,migrate",
"author": "",
"author_email": "Jeff Huber <jeff@trychroma.com>, Hammad Bashir <hammad@trychroma.com>",
"download_url": "https://files.pythonhosted.org/packages/84/8a/652378d58855c985ba932d3333631958beadf1cbd890542c047512225b3c/chroma_migrate-0.0.7.tar.gz",
"platform": null,
"description": "## Chroma Migrate\n\nSchema and data format changes are a necessary evil of evolving software. We take changes seriously and make them infrequently and only when necessary.\n\nChroma's commitment is whenever schema or data format change, we will provide a seamless and easy-to-use migration tool to move to the new schema/format. \n\nSpecifically we will announce schema changes on:\n- Discord ([#migrations channel](https://discord.com/channels/1073293645303795742/1129286514845691975))\n- Github (here)\n- Email listserv [Sign up](https://airtable.com/shrHaErIs1j9F97BE)\n\nWe will aim to provide:\n- a description of the change and the rationale for the change.\n- a CLI migration tool you can run\n- a video walkthrough of using the tool\n\n### Migration Log\n\n#### Migration from >0.4.0 to 0.4.0 - July 17, 2023\n\nWe are migrating:\n- `metadata store`: where metadata is stored\n- `index on disk`: how indexes are stored on disk\n\n`Metadata Store`: Previously Chroma used underlying storage engines `DuckDB` for the `in-memory` version of Chroma, and `Clickhouse` for the `single-node server` version of Chroma. These decisions were made when Chroma was addressing more batch analytical workloads and are no longer the best choice for users. The new metadata store for the `in-memory` and `single-node server` version of Chroma will be `sqlite`. (The distributed version of Chroma (forthcoming), will use a different distributed metadata store.)\n\n`Index store`: Previously Chroma saved the **entire** index on every write. This because painfully slow when the collection grew to a reasonable amount of embeddings. The new index store saves *only the change* and should scale seamlessly! \n\nHere are the 9-possible migration paths, and any notes, if applicable.\n\n| From \ud83d\udc47 \u27a1\ufe0f To \ud83d\udc49 | Persistent Chroma | Local Chroma Server | Remote Chroma Server |\n| -------- | -------- | -------- | -------- |\n| Persistent Chroma | \u2705 | \u2705 | 1\ufe0f\u20e3 |\n| Local Chroma Server| \u2705| 2\ufe0f\u20e3| 1\ufe0f\u20e3|\n| Remote Chroma Server| \u2705| \u2705| 1\ufe0f\u20e3 2\ufe0f\u20e3|\n\n1\ufe0f\u20e3 - Make sure to configure any auth headers correctly\n\n2\ufe0f\u20e3 - Run both the existing version of Chroma and the new `0.4.0` version of Chroma at same time. Run the new version on a new port if local.\n\n[Embed video here]()\n\n##### How to use the migration tool\n\n1. `pip` install this utility. `pip install chroma_migrate`\n\n1. Running the CLI. In your terminal run:\n\n```\nchroma_migrate\n```\n\n2. Choose whether the data you want to migrate is locally on disk (duckdb) on clickhouse instance used by chroma, or directly from another chroma server\n\n3. Choose where you want to write the new data to. \n\n### Developing Locally\nRun python main.py to test locally\n",
"bugtrack_url": null,
"license": "",
"summary": "A tool for migrating to chroma versions >= 0.4.0",
"version": "0.0.7",
"project_urls": null,
"split_keywords": [
"chroma",
"migrate"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "9aaf6348764d965691a78a4695e0fae310e8b469e35875033d53ab77582bba70",
"md5": "ad3ff4912f68fa9a129216768e4ff5f8",
"sha256": "59646a4c3b23242f8c8d1bd907f5ec6d122512bcf36a8da0b8e2fa1ed74a76b8"
},
"downloads": -1,
"filename": "chroma_migrate-0.0.7-py3-none-any.whl",
"has_sig": false,
"md5_digest": "ad3ff4912f68fa9a129216768e4ff5f8",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7",
"size": 12990,
"upload_time": "2023-08-04T03:20:50",
"upload_time_iso_8601": "2023-08-04T03:20:50.414813Z",
"url": "https://files.pythonhosted.org/packages/9a/af/6348764d965691a78a4695e0fae310e8b469e35875033d53ab77582bba70/chroma_migrate-0.0.7-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "848a652378d58855c985ba932d3333631958beadf1cbd890542c047512225b3c",
"md5": "7290527dd8f9fa42dbd1a66f50a95acf",
"sha256": "ae402998b2ca0aa76e5ce6ec8a6c3f13053c7ae07a72e17df3a2b320c9aa3177"
},
"downloads": -1,
"filename": "chroma_migrate-0.0.7.tar.gz",
"has_sig": false,
"md5_digest": "7290527dd8f9fa42dbd1a66f50a95acf",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7",
"size": 340010,
"upload_time": "2023-08-04T03:20:52",
"upload_time_iso_8601": "2023-08-04T03:20:52.268878Z",
"url": "https://files.pythonhosted.org/packages/84/8a/652378d58855c985ba932d3333631958beadf1cbd890542c047512225b3c/chroma_migrate-0.0.7.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-08-04 03:20:52",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "chroma-migrate"
}
