# AwaDB - AI Native Database for embedding vectors
Easily Use - No boring database schema definition. No need to pay attention to vector indexing details.
Realtime Search - Lock free realtime index keeps new data fresh with millisecond level latency. No wait no manual operation.
Stability - AwaDB builds upon over 4 years experience at JD.com running production workloads at scale using a system called [Vearch](https://github.com/vearch/vearch), combined with best-of-breed ideas and practices from the community.
## Run awadb locally on Mac OSX or Linux
First install awadb:
```bash
pip3 install awadb'
```
Then use as below:
```bash
import awadb
# 1. Initialize awadb client!
awadb_client = awadb.Client()
# 2. Create table
awadb_client.Create("test_llm1")
# 3. Add sentences, the sentence is embedded with SentenceTransformer by default
# You can also embed the sentences all by yourself with OpenAI or other LLMs
awadb_client.Add([{'embedding_text':'The man is happy'}, {'source' : 'pic1'}])
awadb_client.Add([{'embedding_text':'The man is very happy'}, {'source' : 'pic2'}])
awadb_client.Add([{'embedding_text':'The cat is happy'}, {'source' : 'pic3'}])
awadb_client.Add([{'embedding_text':'The man is eating'}, {'source':'pic4'}])
# 4. Search the most Top3 sentences by the specified query
query = "The man is happy"
results = awadb_client.Search(query, 3)
# Output the results
print(results)
```
Here the text is embedded by SentenceTransformer which is supported by [Hugging Face](https://huggingface.co)
More detailed python local library usage you can read [here](https://ljeagle.github.io/awadb/)
## Run AwaDB as a service
If you are on the Windows platform or want a awadb service, you can download and deploy the awadb docker.
The installation of awadb docker please see [here](https://github.com/awa-ai/awadb/tree/main/docs/deploy.md)
- Python Usage
First, Install gRPC and awadb service python client as below:
```bash
pip3 install grpcio
pip3 install awadb-client
```
A simple example as below:
```bash
# Import the package and module
from awadb_client import Awa
# Initialize awadb client
client = Awa()
# Add dict with vector to table 'example1'
client.add("example1", {'name':'david', 'feature':[1.3, 2.5, 1.9]})
client.add("example1", {'name':'jim', 'feature':[1.1, 1.4, 2.3]})
# Search
results = client.search("example1", [1.0, 2.0, 3.0])
# Output results
print(results)
# '_id' is the primary key of each document
# It can be specified clearly when adding documents
# Here no field '_id' is specified, it is generated by the awadb server
db_name: "default"
table_name: "example1"
results {
total: 2
msg: "Success"
result_items {
score: 0.860000074
fields {
name: "_id"
value: "64ddb69d-6038-4311-9118-605686d758d9"
}
fields {
name: "name"
value: "jim"
}
}
result_items {
score: 1.55
fields {
name: "_id"
value: "f9f3035b-faaf-48d4-a947-801416c005b3"
}
fields {
name: "name"
value: "david"
}
}
}
result_code: SUCCESS
```
More python sdk for service is [here](https://ljeagle.github.io/awadb/)
More detailed quick start examples you can find [here](https://github.com/awa-ai/awadb/blob/main/tests/test_awadb_client.py)
- RESTful Usage
```bash
# add documents to table 'test' of db 'default', no need to create table first
curl -H "Content-Type: application/json" -X POST -d '{"db":"default", "table":"test", "docs":[{"_id":1, "name":"lj", "age":23 "f":[1,0]},{"_id":2, "name":"david", "age":32, "f":[1,2]}]}' http://localhost:8080/add
# search documents by the vector field 'f' of the value '[1, 1]'
curl -H "Content-Type: application/json" -X POST -d '{"db":"default", "table":"test", "vector_query":{"f":[1, 1]}}' http://localhost:8080/search
```
More detailed RESTful API is [here](https://github.com/awa-ai/awadb/tree/main/docs/restful_tutorial.md)
## What are the Embeddings?
Any unstructured data(image/text/audio/video) can be transferred to vectors which are generally understanded by computers through AI(LLMs or other deep neural networks).
For example, "The man is happy"-this sentence can be transferred to a 384-dimension vector(a list of numbers `[0.23, 1.98, ....]`) by SentenceTransformer language model. This process is called embedding.
More detailed information about embeddings can be read from [OpenAI](https://platform.openai.com/docs/guides/embeddings/what-are-embeddings)
Awadb uses [Sentence Transformers](https://huggingface.co/sentence-transformers) to embed the sentence by default, while you can also use OpenAI or other LLMs to do the embeddings according to your needs.
## Combined with LLMs(here use LLaMa and ChatGLM) By LangChain
Examples of combining LLaMa or quantized Alpaca with llama.cpp to do local knowledge database please see [here](./examples/llama.cpp)
Examples of combining ChatGLM to do local knowledge database please see [here](./examples/chatglm)
## Get involved
- [Issues and PR](https://github.com/awa-ai/awadb/issues)
- [Roadmap and Contribution](https://github.com/awa-ai/awadb/blob/main/ROADMAP.md)
## License
[Apache 2.0](./LICENSE)
## Community
Join the AwaDB community to share any problem, suggestion, or discussion with us:
- [Discord](https://discord.gg/GP7QxRrDjB)
- [Slack](https://awadbhq.slack.com)
- [Reddit](https://www.reddit.com/r/Awadb/)
Raw data
{
"_id": null,
"home_page": "https://github.com/awa-ai/awadb",
"name": "awadb",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.7",
"maintainer_email": "",
"keywords": "AI Native SearchEngine,RAG,Embedding vectors,Multi-model search",
"author": "Vincent",
"author_email": "awadb.vincent@gmail.com",
"download_url": "",
"platform": null,
"description": "# AwaDB - AI Native Database for embedding vectors\n\nEasily Use - No boring database schema definition. No need to pay attention to vector indexing details. \n\nRealtime Search - Lock free realtime index keeps new data fresh with millisecond level latency. No wait no manual operation. \n\nStability - AwaDB builds upon over 4 years experience at JD.com running production workloads at scale using a system called [Vearch](https://github.com/vearch/vearch), combined with best-of-breed ideas and practices from the community.\n\n## Run awadb locally on Mac OSX or Linux\n\nFirst install awadb:\n```bash\npip3 install awadb'\n```\n\nThen use as below:\n```bash\nimport awadb\n# 1. Initialize awadb client!\nawadb_client = awadb.Client()\n\n# 2. Create table\nawadb_client.Create(\"test_llm1\") \n\n# 3. Add sentences, the sentence is embedded with SentenceTransformer by default\n# You can also embed the sentences all by yourself with OpenAI or other LLMs\nawadb_client.Add([{'embedding_text':'The man is happy'}, {'source' : 'pic1'}])\nawadb_client.Add([{'embedding_text':'The man is very happy'}, {'source' : 'pic2'}])\nawadb_client.Add([{'embedding_text':'The cat is happy'}, {'source' : 'pic3'}])\nawadb_client.Add([{'embedding_text':'The man is eating'}, {'source':'pic4'}])\n\n# 4. Search the most Top3 sentences by the specified query\nquery = \"The man is happy\"\nresults = awadb_client.Search(query, 3)\n\n# Output the results\nprint(results)\n```\nHere the text is embedded by SentenceTransformer which is supported by [Hugging Face](https://huggingface.co) \nMore detailed python local library usage you can read [here](https://ljeagle.github.io/awadb/)\n\n## Run AwaDB as a service \nIf you are on the Windows platform or want a awadb service, you can download and deploy the awadb docker.\nThe installation of awadb docker please see [here](https://github.com/awa-ai/awadb/tree/main/docs/deploy.md)\n\n- Python Usage\n\nFirst, Install gRPC and awadb service python client as below:\n\n```bash\npip3 install grpcio\npip3 install awadb-client\n```\n\nA simple example as below:\n\n```bash\n# Import the package and module\nfrom awadb_client import Awa\n\n# Initialize awadb client\nclient = Awa()\n\n# Add dict with vector to table 'example1'\nclient.add(\"example1\", {'name':'david', 'feature':[1.3, 2.5, 1.9]})\nclient.add(\"example1\", {'name':'jim', 'feature':[1.1, 1.4, 2.3]})\n\n# Search\nresults = client.search(\"example1\", [1.0, 2.0, 3.0])\n\n# Output results\nprint(results)\n\n# '_id' is the primary key of each document\n# It can be specified clearly when adding documents\n# Here no field '_id' is specified, it is generated by the awadb server \ndb_name: \"default\"\ntable_name: \"example1\"\nresults {\n total: 2\n msg: \"Success\"\n result_items {\n score: 0.860000074\n fields {\n name: \"_id\" \n value: \"64ddb69d-6038-4311-9118-605686d758d9\"\n }\n fields {\n name: \"name\"\n value: \"jim\"\n }\n }\n result_items {\n score: 1.55\n fields {\n name: \"_id\"\n value: \"f9f3035b-faaf-48d4-a947-801416c005b3\"\n }\n fields {\n name: \"name\"\n value: \"david\"\n }\n }\n}\nresult_code: SUCCESS\n```\nMore python sdk for service is [here](https://ljeagle.github.io/awadb/) \nMore detailed quick start examples you can find [here](https://github.com/awa-ai/awadb/blob/main/tests/test_awadb_client.py)\n\n- RESTful Usage\n```bash\n# add documents to table 'test' of db 'default', no need to create table first\ncurl -H \"Content-Type: application/json\" -X POST -d '{\"db\":\"default\", \"table\":\"test\", \"docs\":[{\"_id\":1, \"name\":\"lj\", \"age\":23 \"f\":[1,0]},{\"_id\":2, \"name\":\"david\", \"age\":32, \"f\":[1,2]}]}' http://localhost:8080/add\n\n# search documents by the vector field 'f' of the value '[1, 1]'\ncurl -H \"Content-Type: application/json\" -X POST -d '{\"db\":\"default\", \"table\":\"test\", \"vector_query\":{\"f\":[1, 1]}}' http://localhost:8080/search\n```\nMore detailed RESTful API is [here](https://github.com/awa-ai/awadb/tree/main/docs/restful_tutorial.md)\n\n\n## What are the Embeddings?\n\nAny unstructured data(image/text/audio/video) can be transferred to vectors which are generally understanded by computers through AI(LLMs or other deep neural networks). \n \nFor example, \"The man is happy\"-this sentence can be transferred to a 384-dimension vector(a list of numbers `[0.23, 1.98, ....]`) by SentenceTransformer language model. This process is called embedding.\n\nMore detailed information about embeddings can be read from [OpenAI](https://platform.openai.com/docs/guides/embeddings/what-are-embeddings)\n\nAwadb uses [Sentence Transformers](https://huggingface.co/sentence-transformers) to embed the sentence by default, while you can also use OpenAI or other LLMs to do the embeddings according to your needs.\n\n\n## Combined with LLMs(here use LLaMa and ChatGLM) By LangChain\nExamples of combining LLaMa or quantized Alpaca with llama.cpp to do local knowledge database please see [here](./examples/llama.cpp) \nExamples of combining ChatGLM to do local knowledge database please see [here](./examples/chatglm) \n\n\n## Get involved\n\n- [Issues and PR](https://github.com/awa-ai/awadb/issues) \n- [Roadmap and Contribution](https://github.com/awa-ai/awadb/blob/main/ROADMAP.md)\n\n\n## License\n\n[Apache 2.0](./LICENSE)\n\n\n## Community\n\nJoin the AwaDB community to share any problem, suggestion, or discussion with us:\n\n- [Discord](https://discord.gg/GP7QxRrDjB)\n- [Slack](https://awadbhq.slack.com)\n- [Reddit](https://www.reddit.com/r/Awadb/)\n\n\n",
"bugtrack_url": null,
"license": "Apache 2.0",
"summary": "Local lightweight AI Native database for RAG, incluing embedding vectors and text search for LLM generation",
"version": "0.3.13",
"project_urls": {
"Homepage": "https://github.com/awa-ai/awadb"
},
"split_keywords": [
"ai native searchengine",
"rag",
"embedding vectors",
"multi-model search"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "59592548feeb3348c7ddbf99777ef8a78b26a735a6002cdaf78c4104268cdb22",
"md5": "071c76cc957954789f36493eaed6b435",
"sha256": "b9636a0c0c0b9973364ef841683e45124977c9f80f39b8b4a2455cf4b500cba9"
},
"downloads": -1,
"filename": "awadb-0.3.13-cp310-cp310-macosx_13_0_x86_64.whl",
"has_sig": false,
"md5_digest": "071c76cc957954789f36493eaed6b435",
"packagetype": "bdist_wheel",
"python_version": "cp310",
"requires_python": ">=3.7",
"size": 2214183,
"upload_time": "2023-12-16T09:07:48",
"upload_time_iso_8601": "2023-12-16T09:07:48.508124Z",
"url": "https://files.pythonhosted.org/packages/59/59/2548feeb3348c7ddbf99777ef8a78b26a735a6002cdaf78c4104268cdb22/awadb-0.3.13-cp310-cp310-macosx_13_0_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "36bd2b5d197f6b5a669f2aa04da96ccb764b876b9a7d93dcf92404c91d33646c",
"md5": "583a28504aadbe25630e7dbe84f72f66",
"sha256": "982d6af59cad5143947948dc55f76d668dc9f1b5a2b6c413437226dc0d331859"
},
"downloads": -1,
"filename": "awadb-0.3.13-cp310-cp310-manylinux1_x86_64.whl",
"has_sig": false,
"md5_digest": "583a28504aadbe25630e7dbe84f72f66",
"packagetype": "bdist_wheel",
"python_version": "cp310",
"requires_python": ">=3.7",
"size": 2509154,
"upload_time": "2023-12-16T09:03:53",
"upload_time_iso_8601": "2023-12-16T09:03:53.286582Z",
"url": "https://files.pythonhosted.org/packages/36/bd/2b5d197f6b5a669f2aa04da96ccb764b876b9a7d93dcf92404c91d33646c/awadb-0.3.13-cp310-cp310-manylinux1_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "5b460684e78622704dd40237648b4a9e0b3d9631651da2702a7f4c76c60796ab",
"md5": "8e0d9c4c6e7651e4b202aa21b61e0629",
"sha256": "875541d8a9bca0e42c9b2985dc664163e9913f596ec70404f30ade49b963e2e5"
},
"downloads": -1,
"filename": "awadb-0.3.13-cp311-cp311-macosx_13_0_x86_64.whl",
"has_sig": false,
"md5_digest": "8e0d9c4c6e7651e4b202aa21b61e0629",
"packagetype": "bdist_wheel",
"python_version": "cp311",
"requires_python": ">=3.7",
"size": 2214515,
"upload_time": "2023-12-16T09:08:47",
"upload_time_iso_8601": "2023-12-16T09:08:47.340817Z",
"url": "https://files.pythonhosted.org/packages/5b/46/0684e78622704dd40237648b4a9e0b3d9631651da2702a7f4c76c60796ab/awadb-0.3.13-cp311-cp311-macosx_13_0_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "d6a4bd0fcb0c393a4ff76c07705d77d91bf52ba935768e3dc23b4ae7dde6ee93",
"md5": "df913fb84870041958c6385d68b82cbe",
"sha256": "60a4676c3404c90f8354291a85ca152d908588bf53cf8f4aef8f73baaf5d3b76"
},
"downloads": -1,
"filename": "awadb-0.3.13-cp311-cp311-manylinux1_x86_64.whl",
"has_sig": false,
"md5_digest": "df913fb84870041958c6385d68b82cbe",
"packagetype": "bdist_wheel",
"python_version": "cp311",
"requires_python": ">=3.7",
"size": 2512322,
"upload_time": "2023-12-16T09:04:57",
"upload_time_iso_8601": "2023-12-16T09:04:57.820397Z",
"url": "https://files.pythonhosted.org/packages/d6/a4/bd0fcb0c393a4ff76c07705d77d91bf52ba935768e3dc23b4ae7dde6ee93/awadb-0.3.13-cp311-cp311-manylinux1_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "7e9f00c60229576847436b7b8e863572b6202ef1cacd78e0f2b959fab208f587",
"md5": "044ace4472b3d5a78b011489629bf057",
"sha256": "26f8ebb7c877264f32b0c1f496c9e87bbef420242efae366b2bddd97ba720461"
},
"downloads": -1,
"filename": "awadb-0.3.13-cp37-cp37m-macosx_13_0_x86_64.whl",
"has_sig": false,
"md5_digest": "044ace4472b3d5a78b011489629bf057",
"packagetype": "bdist_wheel",
"python_version": "cp37",
"requires_python": ">=3.7",
"size": 2203578,
"upload_time": "2023-12-16T09:10:08",
"upload_time_iso_8601": "2023-12-16T09:10:08.103086Z",
"url": "https://files.pythonhosted.org/packages/7e/9f/00c60229576847436b7b8e863572b6202ef1cacd78e0f2b959fab208f587/awadb-0.3.13-cp37-cp37m-macosx_13_0_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "6ae134de4242950f28a93ac44ed072c054e3b99382fd58fd426c6feb9a7ad062",
"md5": "7c5998481d6cd02895a08d9dc7f9ab43",
"sha256": "6e251927de268b5074dcf437a2ea31f6d9cdfc54176a021d306a0a6c1550a897"
},
"downloads": -1,
"filename": "awadb-0.3.13-cp37-cp37m-manylinux1_x86_64.whl",
"has_sig": false,
"md5_digest": "7c5998481d6cd02895a08d9dc7f9ab43",
"packagetype": "bdist_wheel",
"python_version": "cp37",
"requires_python": ">=3.7",
"size": 2509951,
"upload_time": "2023-12-16T09:07:00",
"upload_time_iso_8601": "2023-12-16T09:07:00.221793Z",
"url": "https://files.pythonhosted.org/packages/6a/e1/34de4242950f28a93ac44ed072c054e3b99382fd58fd426c6feb9a7ad062/awadb-0.3.13-cp37-cp37m-manylinux1_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "3249a114a84c7903a3d9050e742267ee95bb3058eab409c2873fd2a69fee7749",
"md5": "53a5acb9f5e54c0e9fc875d88f8b629b",
"sha256": "a5990b59ce43ecaaf993cc3eb1f25cc753ace6eb09c884d7a9dbda6325c81fa3"
},
"downloads": -1,
"filename": "awadb-0.3.13-cp38-cp38-macosx_13_0_x86_64.whl",
"has_sig": false,
"md5_digest": "53a5acb9f5e54c0e9fc875d88f8b629b",
"packagetype": "bdist_wheel",
"python_version": "cp38",
"requires_python": ">=3.7",
"size": 2213929,
"upload_time": "2023-12-16T09:10:25",
"upload_time_iso_8601": "2023-12-16T09:10:25.683992Z",
"url": "https://files.pythonhosted.org/packages/32/49/a114a84c7903a3d9050e742267ee95bb3058eab409c2873fd2a69fee7749/awadb-0.3.13-cp38-cp38-macosx_13_0_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "a4b1e29c01d347e35b64cca97982382cfdb93031f117a29f8fd841ce4ed6d8d0",
"md5": "f06f526599bd1c456a565e2e2a02cc50",
"sha256": "afa364ca60f3375eb7cdd137286b392e3e60905a45c54d56b7337c2861253716"
},
"downloads": -1,
"filename": "awadb-0.3.13-cp38-cp38-manylinux1_x86_64.whl",
"has_sig": false,
"md5_digest": "f06f526599bd1c456a565e2e2a02cc50",
"packagetype": "bdist_wheel",
"python_version": "cp38",
"requires_python": ">=3.7",
"size": 2509963,
"upload_time": "2023-12-16T09:06:21",
"upload_time_iso_8601": "2023-12-16T09:06:21.492267Z",
"url": "https://files.pythonhosted.org/packages/a4/b1/e29c01d347e35b64cca97982382cfdb93031f117a29f8fd841ce4ed6d8d0/awadb-0.3.13-cp38-cp38-manylinux1_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "1aabe29fe2bf8f280253faeb681b4b722676d93568fad91391190edacec382c8",
"md5": "4f814fd29df1dbbecf8d99d430791ca9",
"sha256": "2a0a79342d268ee13133f061ff6a3a6bd0afaeac101fd8c7751b8cc1905cbaf7"
},
"downloads": -1,
"filename": "awadb-0.3.13-cp39-cp39-macosx_13_0_x86_64.whl",
"has_sig": false,
"md5_digest": "4f814fd29df1dbbecf8d99d430791ca9",
"packagetype": "bdist_wheel",
"python_version": "cp39",
"requires_python": ">=3.7",
"size": 2140368,
"upload_time": "2023-12-16T09:10:44",
"upload_time_iso_8601": "2023-12-16T09:10:44.870882Z",
"url": "https://files.pythonhosted.org/packages/1a/ab/e29fe2bf8f280253faeb681b4b722676d93568fad91391190edacec382c8/awadb-0.3.13-cp39-cp39-macosx_13_0_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "8c0a6322977cc2b3fb4b2f0776d794179744d186b900113288291e1fd9d6d9fc",
"md5": "76d99ce659ba63c9fc4cdbc61dc5e055",
"sha256": "7b8da7adebd1ae831a92466630fc51067fb6643889152c18da28260a94915d6d"
},
"downloads": -1,
"filename": "awadb-0.3.13-cp39-cp39-manylinux1_x86_64.whl",
"has_sig": false,
"md5_digest": "76d99ce659ba63c9fc4cdbc61dc5e055",
"packagetype": "bdist_wheel",
"python_version": "cp39",
"requires_python": ">=3.7",
"size": 2506878,
"upload_time": "2023-12-16T09:05:36",
"upload_time_iso_8601": "2023-12-16T09:05:36.155758Z",
"url": "https://files.pythonhosted.org/packages/8c/0a/6322977cc2b3fb4b2f0776d794179744d186b900113288291e1fd9d6d9fc/awadb-0.3.13-cp39-cp39-manylinux1_x86_64.whl",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-12-16 09:07:48",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "awa-ai",
"github_project": "awadb",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"requirements": [],
"lcname": "awadb"
}
JSON
