| Name | superlinked JSON |
| Version |
17.7.1
JSON |
| download |
| home_page | None |
| Summary | The Superlinked vector computing library |
| upload_time | 2024-12-16 12:44:19 |
| maintainer | None |
| docs_url | None |
| author | Superlinked Release |
| requires_python | <=3.12.3,>=3.10 |
| license | Apache-2.0 |
| keywords |
|
| VCS |
|
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
<div align="center">
<picture>
<source
srcset="https://cdn.prod.website-files.com/65dce6831bf9f730421e2915/66ef0317ed8616151ee1d451_superlinked_logo_white.png"
media="(prefers-color-scheme: dark)"
/>
<source
srcset="https://cdn.prod.website-files.com/65dce6831bf9f730421e2915/65dce6831bf9f730421e2929_superlinked_logo.svg"
media="(prefers-color-scheme: light), (prefers-color-scheme: no-preference)"
/>
<img width=400 src="https://cdn.prod.website-files.com/65dce6831bf9f730421e2915/66ef0317ed8616151ee1d451_superlinked_logo_white.png" />
</picture>
</div>
<div align="center">
</div>
<div align="center">
[Experiment in a notebook](#experiment-in-a-notebook) | [Run in production](#run-in-production) | [Use-cases](#use-cases) | [Supported VDBs](#supported-vdbs) | [Resources](#resources)
</div>
#### Why use Superlinked
Improve your vector search relevance by encoding your metadata together with your data into your vector embeddings.
#### What is Superlinked
Superlinked is a framework AND a self-hostable REST API server that helps you make better vectors, that sits between your data, vector database and backend services.
#### How does it work
Superlinked makes it easy to construct custom data & query embedding models from pre-trained encoders, see the <b>feature</b> and <b>use-case</b> notebooks below for examples.
If you like what we do, give us a star! ⭐
Visit [Superlinked](https://superlinked.com/) for more information about the company behind this product and our other initiatives.
## Features
- Embed structured and unstructured data ([Text](https://colab.research.google.com/github/superlinked/superlinked/blob/main/notebook/feature/text_embedding.ipynb) | [Number](https://colab.research.google.com/github/superlinked/superlinked/blob/main/notebook/feature/number_embedding_minmax.ipynb) | [Category](https://colab.research.google.com/github/superlinked/superlinked/blob/main/notebook/feature/categorical_embedding.ipynb) | [Time](https://colab.research.google.com/github/superlinked/superlinked/blob/main/notebook/feature/recency_embedding.ipynb) | [Event](https://colab.research.google.com/github/superlinked/superlinked/blob/main/notebook/feature/event_effects.ipynb))
- Combine encoders to build a custom model ([notebook](https://colab.research.google.com/github/superlinked/superlinked/blob/main/notebook/feature/combine_multiple_embeddings.ipynb))
- Add a custom encoder ([notebook](https://colab.research.google.com/github/superlinked/superlinked/blob/main/notebook/feature/custom_space.ipynb))
- Update your vectors with behavioral events & relationships ([notebook](https://colab.research.google.com/github/superlinked/superlinked/blob/main/notebook/feature/event_effects.ipynb))
- Use query-time weights ([notebook](https://colab.research.google.com/github/superlinked/superlinked/blob/main/notebook/feature/query_time_weights.ipynb))
- Query with natural language ([notebook](https://colab.research.google.com/github/superlinked/superlinked/blob/main/notebook/feature/natural_language_querying.ipynb))
- Filter your results ([notebook](https://colab.research.google.com/github/superlinked/superlinked/blob/main/notebook/feature/hard_filtering.ipynb))
- Export vectors for analysis ([notebook](https://colab.research.google.com/github/superlinked/superlinked/blob/main/notebook/feature/vector_sampler.ipynb))
You can check a full list of our [features](https://github.com/superlinked/superlinked/tree/main/notebook/feature) or head to our [reference](#reference) section for more information.
## Use-cases
Dive deeper with our notebooks into how each use-case benefits from the Superlinked framework.
- **RAG**: [HR Knowledgebase](https://colab.research.google.com/github/superlinked/superlinked/blob/main/notebook/rag_hr_knowledgebase.ipynb)
- **Semantic Search**: [Movies](https://colab.research.google.com/github/superlinked/superlinked/blob/main/notebook/semantic_search_netflix_titles.ipynb), [Business News](https://colab.research.google.com/github/superlinked/superlinked/blob/main/notebook/semantic_search_news.ipynb), [Product Images & Descriptions](https://colab.research.google.com/github/superlinked/superlinked/blob/main/notebook/image_search_e_commerce.ipynb)
- **Recommendation Systems**: [E-commerce](https://colab.research.google.com/github/superlinked/superlinked/blob/main/notebook/recommendations_e_commerce.ipynb)
- **Analytics**: [User Acquisition](https://colab.research.google.com/github/superlinked/superlinked/blob/main/notebook/analytics_user_acquisition.ipynb), [Keyword expansion](https://colab.research.google.com/github/superlinked/superlinked/blob/main/notebook/analytics_keyword_expansion_ads.ipynb)
You can check a full list of examples [here](https://github.com/superlinked/superlinked/tree/main/notebook).
## Experiment in a notebook
Example on combining Text with Numerical encoders to get correct results with LLMs.
#### Install the superlinked library
```
%pip install superlinked
```
#### Run the example:
>First run will take slightly longer as it has to download the embedding model.
```python
import json
import os
from superlinked import framework as sl
class Product(sl.Schema):
id: sl.IdField
description: sl.String
rating: sl.Integer
product = Product()
description_space = sl.TextSimilaritySpace(
text=product.description, model="Alibaba-NLP/gte-large-en-v1.5"
)
rating_maximizer_space = sl.NumberSpace(
number=product.rating, min_value=1, max_value=5, mode=sl.Mode.MAXIMUM
)
index = sl.Index([description_space, rating_maximizer_space], fields=[product.rating])
# fill this with your API key - this will drive param extraction
openai_config = sl.OpenAIClientConfig(
api_key=os.environ["OPEN_AI_API_KEY"], model="gpt-4o"
)
# it is possible now to add descriptions to a `Param` to aid the parsing of information from natural language queries.
text_similar_param = sl.Param(
"query_text",
description="The text in the user's query that refers to product descriptions.",
)
# Define your query using dynamic parameters for query text and weights.
# we will have our LLM fill them based on our natural language query
query = (
sl.Query(
index,
weights={
description_space: sl.Param("description_weight"),
rating_maximizer_space: sl.Param("rating_maximizer_weight"),
},
)
.find(product)
.similar(
description_space,
text_similar_param,
)
.limit(sl.Param("limit"))
.with_natural_query(sl.Param("natural_query"), openai_config)
)
# Run the app.
source = sl.InMemorySource(product)
executor = sl.InMemoryExecutor(sources=[source], indices=[index])
app = executor.run()
# Download dataset.
data = [
{"id": 1, "description": "Budget toothbrush in black color.", "rating": 1},
{"id": 2, "description": "High-end toothbrush created with no compromises.", "rating": 5},
{"id": 3, "description": "A toothbrush created for the smart 21st century man.", "rating": 3},
]
# Ingest data to the framework.
source.put(data)
result = app.query(query, natural_query="best toothbrushes", limit=1)
# examine the extracted parameters from your query
print(json.dumps(result.knn_params, indent=2))
# the result is the 5 star rated product
result.to_pandas()
```
## Run in production
[Superlinked Server](https://github.com/superlinked/superlinked/tree/main/server) allows you to leverage the power of Superlinked in deployable projects. With a single script, you can deploy a Superlinked-powered app instance that creates REST endpoints and connects to external Vector Databases. This makes it an ideal solution for those seeking an easy-to-deploy environment for their Superlinked projects.
If your are interested in learning more about running at scale, [Book a demo](https://links.superlinked.com/sl-repo-readme-form) for an early access to our managed cloud.
### Supported VDBs
We have started partnering with vector database providers to allow you to use Superlinked with your VDB of choice. If you are unsure, which VDB to chose, check-out our [Vector DB Comparison](https://superlinked.com/vector-db-comparison/).
- [Redis](https://github.com/superlinked/superlinked/tree/main/server/docs/redis/redis.md)
- [MongoDB](https://github.com/superlinked/superlinked/tree/main/server/docs/mongodb/mongodb.md)
Missing your favorite VDB? [Tell us which vector database we should support next!](https://github.com/superlinked/superlinked/discussions/41)
## Reference
1. Describe your data using Python classes with the [@schema](https://github.com/superlinked/superlinked/blob/main/framework/reference/common/schema/schema.md) decorator.
2. Describe your vector embeddings from building blocks with [Spaces](https://github.com/superlinked/superlinked/blob/main/framework/reference/dsl/space/index.md).
3. Combine your embeddings into a queryable [Index](https://github.com/superlinked/superlinked/blob/main/framework/reference/dsl/index/index.m.md).
4. Define your search with dynamic parameters and weights as a [Query](https://github.com/superlinked/superlinked/blob/main/framework/reference/dsl/query/query.md).
5. Load your data using a [Source](https://github.com/superlinked/superlinked/blob/main/framework/reference/dsl/source/index.md).
6. Define your transformations with a [Parser](https://github.com/superlinked/superlinked/blob/main/framework/reference/common/parser) (e.g.: from [`pd.DataFrame`](https://github.com/superlinked/superlinked/blob/main/framework/reference/common/parser/dataframe_parser.md)).
7. Run your configuration with an [Executor](https://github.com/superlinked/superlinked/blob/main/framework/reference/dsl/executor/in_memory/in_memory_executor.md).
You can check all references [here](https://github.com/superlinked/superlinked/tree/main/framework/reference).
## Logging
Contextual information is automatically included in log messages, such as the process ID and package scope. Personally Identifiable Information (PII) is filtered out by default but can be exposed with the `SUPERLINKED_EXPOSE_PII` environment variable to `true`.
## Resources
- [Vector DB Comparison](https://superlinked.com/vector-db-comparison/): Open-source collaborative comparison of vector databases by Superlinked.
- [Vector Hub](https://superlinked.com/vectorhub/): VectorHub is a free and open-sourced learning hub for people interested in adding vector retrieval to their ML stack
## Support
If you encounter any challenges during your experiments, feel free to create an [issue](https://github.com/superlinked/superlinked/issues/new?assignees=kembala&labels=bug&projects=&template=bug_report.md&title=), request a [feature](https://github.com/superlinked/superlinked/issues/new?assignees=kembala&labels=enhancement&projects=&template=feature_request.md&title=) or to [start a discussion](https://github.com/superlinked/superlinked/discussions/new/choose).
Make sure to group your feedback in separate issues and discussions by topic. Thank you for your feedback!
Raw data
{
"_id": null,
"home_page": null,
"name": "superlinked",
"maintainer": null,
"docs_url": null,
"requires_python": "<=3.12.3,>=3.10",
"maintainer_email": null,
"keywords": null,
"author": "Superlinked Release",
"author_email": "release@superlinked.com",
"download_url": "https://files.pythonhosted.org/packages/9b/90/867ce189ec955fa305f0e9bc948ca45fc99e5c9760a9a77916c69751b11d/superlinked-17.7.1.tar.gz",
"platform": null,
"description": "<div align=\"center\">\n<picture>\n <source\n srcset=\"https://cdn.prod.website-files.com/65dce6831bf9f730421e2915/66ef0317ed8616151ee1d451_superlinked_logo_white.png\"\n media=\"(prefers-color-scheme: dark)\"\n />\n <source\n srcset=\"https://cdn.prod.website-files.com/65dce6831bf9f730421e2915/65dce6831bf9f730421e2929_superlinked_logo.svg\"\n media=\"(prefers-color-scheme: light), (prefers-color-scheme: no-preference)\"\n />\n <img width=400 src=\"https://cdn.prod.website-files.com/65dce6831bf9f730421e2915/66ef0317ed8616151ee1d451_superlinked_logo_white.png\" />\n</picture>\n\n</div> \n\n<div align=\"center\">\n\n\n\n \n\n\n</div>\n\n<div align=\"center\">\n\n[Experiment in a notebook](#experiment-in-a-notebook) | [Run in production](#run-in-production) | [Use-cases](#use-cases) | [Supported VDBs](#supported-vdbs) | [Resources](#resources)\n\n</div>\n\n#### Why use Superlinked\nImprove your vector search relevance by encoding your metadata together with your data into your vector embeddings.\n\n#### What is Superlinked\nSuperlinked is a framework AND a self-hostable REST API server that helps you make better vectors, that sits between your data, vector database and backend services.\n\n#### How does it work\nSuperlinked makes it easy to construct custom data & query embedding models from pre-trained encoders, see the <b>feature</b> and <b>use-case</b> notebooks below for examples.\n\nIf you like what we do, give us a star! \u2b50\n\n\n\nVisit [Superlinked](https://superlinked.com/) for more information about the company behind this product and our other initiatives.\n\n## Features\n\n- Embed structured and unstructured data ([Text](https://colab.research.google.com/github/superlinked/superlinked/blob/main/notebook/feature/text_embedding.ipynb) | [Number](https://colab.research.google.com/github/superlinked/superlinked/blob/main/notebook/feature/number_embedding_minmax.ipynb) | [Category](https://colab.research.google.com/github/superlinked/superlinked/blob/main/notebook/feature/categorical_embedding.ipynb) | [Time](https://colab.research.google.com/github/superlinked/superlinked/blob/main/notebook/feature/recency_embedding.ipynb) | [Event](https://colab.research.google.com/github/superlinked/superlinked/blob/main/notebook/feature/event_effects.ipynb))\n- Combine encoders to build a custom model ([notebook](https://colab.research.google.com/github/superlinked/superlinked/blob/main/notebook/feature/combine_multiple_embeddings.ipynb))\n- Add a custom encoder ([notebook](https://colab.research.google.com/github/superlinked/superlinked/blob/main/notebook/feature/custom_space.ipynb))\n- Update your vectors with behavioral events & relationships ([notebook](https://colab.research.google.com/github/superlinked/superlinked/blob/main/notebook/feature/event_effects.ipynb))\n- Use query-time weights ([notebook](https://colab.research.google.com/github/superlinked/superlinked/blob/main/notebook/feature/query_time_weights.ipynb))\n- Query with natural language ([notebook](https://colab.research.google.com/github/superlinked/superlinked/blob/main/notebook/feature/natural_language_querying.ipynb))\n- Filter your results ([notebook](https://colab.research.google.com/github/superlinked/superlinked/blob/main/notebook/feature/hard_filtering.ipynb))\n- Export vectors for analysis ([notebook](https://colab.research.google.com/github/superlinked/superlinked/blob/main/notebook/feature/vector_sampler.ipynb))\n\nYou can check a full list of our [features](https://github.com/superlinked/superlinked/tree/main/notebook/feature) or head to our [reference](#reference) section for more information.\n\n## Use-cases\n\nDive deeper with our notebooks into how each use-case benefits from the Superlinked framework.\n\n- **RAG**: [HR Knowledgebase](https://colab.research.google.com/github/superlinked/superlinked/blob/main/notebook/rag_hr_knowledgebase.ipynb)\n- **Semantic Search**: [Movies](https://colab.research.google.com/github/superlinked/superlinked/blob/main/notebook/semantic_search_netflix_titles.ipynb), [Business News](https://colab.research.google.com/github/superlinked/superlinked/blob/main/notebook/semantic_search_news.ipynb), [Product Images & Descriptions](https://colab.research.google.com/github/superlinked/superlinked/blob/main/notebook/image_search_e_commerce.ipynb)\n- **Recommendation Systems**: [E-commerce](https://colab.research.google.com/github/superlinked/superlinked/blob/main/notebook/recommendations_e_commerce.ipynb)\n- **Analytics**: [User Acquisition](https://colab.research.google.com/github/superlinked/superlinked/blob/main/notebook/analytics_user_acquisition.ipynb), [Keyword expansion](https://colab.research.google.com/github/superlinked/superlinked/blob/main/notebook/analytics_keyword_expansion_ads.ipynb)\n\nYou can check a full list of examples [here](https://github.com/superlinked/superlinked/tree/main/notebook).\n\n## Experiment in a notebook\n\nExample on combining Text with Numerical encoders to get correct results with LLMs.\n\n#### Install the superlinked library\n```\n%pip install superlinked\n```\n\n#### Run the example:\n\n>First run will take slightly longer as it has to download the embedding model. \n\n```python\nimport json\nimport os\n\nfrom superlinked import framework as sl\n\n\nclass Product(sl.Schema):\n id: sl.IdField\n description: sl.String\n rating: sl.Integer\n\n\nproduct = Product()\n\ndescription_space = sl.TextSimilaritySpace(\n text=product.description, model=\"Alibaba-NLP/gte-large-en-v1.5\"\n)\nrating_maximizer_space = sl.NumberSpace(\n number=product.rating, min_value=1, max_value=5, mode=sl.Mode.MAXIMUM\n)\nindex = sl.Index([description_space, rating_maximizer_space], fields=[product.rating])\n\n# fill this with your API key - this will drive param extraction\nopenai_config = sl.OpenAIClientConfig(\n api_key=os.environ[\"OPEN_AI_API_KEY\"], model=\"gpt-4o\"\n)\n\n# it is possible now to add descriptions to a `Param` to aid the parsing of information from natural language queries.\ntext_similar_param = sl.Param(\n \"query_text\",\n description=\"The text in the user's query that refers to product descriptions.\",\n)\n\n# Define your query using dynamic parameters for query text and weights.\n# we will have our LLM fill them based on our natural language query\nquery = (\n sl.Query(\n index,\n weights={\n description_space: sl.Param(\"description_weight\"),\n rating_maximizer_space: sl.Param(\"rating_maximizer_weight\"),\n },\n )\n .find(product)\n .similar(\n description_space,\n text_similar_param,\n )\n .limit(sl.Param(\"limit\"))\n .with_natural_query(sl.Param(\"natural_query\"), openai_config)\n)\n\n# Run the app.\nsource = sl.InMemorySource(product)\nexecutor = sl.InMemoryExecutor(sources=[source], indices=[index])\napp = executor.run()\n\n# Download dataset.\ndata = [\n {\"id\": 1, \"description\": \"Budget toothbrush in black color.\", \"rating\": 1},\n {\"id\": 2, \"description\": \"High-end toothbrush created with no compromises.\", \"rating\": 5},\n {\"id\": 3, \"description\": \"A toothbrush created for the smart 21st century man.\", \"rating\": 3},\n]\n\n# Ingest data to the framework.\nsource.put(data)\n\nresult = app.query(query, natural_query=\"best toothbrushes\", limit=1)\n\n# examine the extracted parameters from your query\nprint(json.dumps(result.knn_params, indent=2))\n# the result is the 5 star rated product\nresult.to_pandas()\n```\n\n## Run in production\n\n[Superlinked Server](https://github.com/superlinked/superlinked/tree/main/server) allows you to leverage the power of Superlinked in deployable projects. With a single script, you can deploy a Superlinked-powered app instance that creates REST endpoints and connects to external Vector Databases. This makes it an ideal solution for those seeking an easy-to-deploy environment for their Superlinked projects.\n\nIf your are interested in learning more about running at scale, [Book a demo](https://links.superlinked.com/sl-repo-readme-form) for an early access to our managed cloud.\n\n### Supported VDBs\n\nWe have started partnering with vector database providers to allow you to use Superlinked with your VDB of choice. If you are unsure, which VDB to chose, check-out our [Vector DB Comparison](https://superlinked.com/vector-db-comparison/).\n\n- [Redis](https://github.com/superlinked/superlinked/tree/main/server/docs/redis/redis.md)\n- [MongoDB](https://github.com/superlinked/superlinked/tree/main/server/docs/mongodb/mongodb.md)\n\nMissing your favorite VDB? [Tell us which vector database we should support next!](https://github.com/superlinked/superlinked/discussions/41)\n\n## Reference\n\n1. Describe your data using Python classes with the [@schema](https://github.com/superlinked/superlinked/blob/main/framework/reference/common/schema/schema.md) decorator.\n2. Describe your vector embeddings from building blocks with [Spaces](https://github.com/superlinked/superlinked/blob/main/framework/reference/dsl/space/index.md).\n3. Combine your embeddings into a queryable [Index](https://github.com/superlinked/superlinked/blob/main/framework/reference/dsl/index/index.m.md).\n4. Define your search with dynamic parameters and weights as a [Query](https://github.com/superlinked/superlinked/blob/main/framework/reference/dsl/query/query.md).\n5. Load your data using a [Source](https://github.com/superlinked/superlinked/blob/main/framework/reference/dsl/source/index.md).\n6. Define your transformations with a [Parser](https://github.com/superlinked/superlinked/blob/main/framework/reference/common/parser) (e.g.: from [`pd.DataFrame`](https://github.com/superlinked/superlinked/blob/main/framework/reference/common/parser/dataframe_parser.md)). \n7. Run your configuration with an [Executor](https://github.com/superlinked/superlinked/blob/main/framework/reference/dsl/executor/in_memory/in_memory_executor.md).\n\nYou can check all references [here](https://github.com/superlinked/superlinked/tree/main/framework/reference).\n\n## Logging\n\nContextual information is automatically included in log messages, such as the process ID and package scope. Personally Identifiable Information (PII) is filtered out by default but can be exposed with the `SUPERLINKED_EXPOSE_PII` environment variable to `true`.\n\n## Resources\n\n- [Vector DB Comparison](https://superlinked.com/vector-db-comparison/): Open-source collaborative comparison of vector databases by Superlinked.\n- [Vector Hub](https://superlinked.com/vectorhub/): VectorHub is a free and open-sourced learning hub for people interested in adding vector retrieval to their ML stack\n\n## Support\n\nIf you encounter any challenges during your experiments, feel free to create an [issue](https://github.com/superlinked/superlinked/issues/new?assignees=kembala&labels=bug&projects=&template=bug_report.md&title=), request a [feature](https://github.com/superlinked/superlinked/issues/new?assignees=kembala&labels=enhancement&projects=&template=feature_request.md&title=) or to [start a discussion](https://github.com/superlinked/superlinked/discussions/new/choose).\nMake sure to group your feedback in separate issues and discussions by topic. Thank you for your feedback!\n\n",
"bugtrack_url": null,
"license": "Apache-2.0",
"summary": "The Superlinked vector computing library",
"version": "17.7.1",
"project_urls": null,
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "4b87f05c0215a0f27ee43a50f2e2e5d325b7e6910d1dad7e86519c3ebaf16d8d",
"md5": "240b266faaced4b0aa6838255bdcf476",
"sha256": "838a1fd716101bbb8e64160f86ad9b8ebb90b62f02d746b208e99cec1057bec6"
},
"downloads": -1,
"filename": "superlinked-17.7.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "240b266faaced4b0aa6838255bdcf476",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<=3.12.3,>=3.10",
"size": 446119,
"upload_time": "2024-12-16T12:44:17",
"upload_time_iso_8601": "2024-12-16T12:44:17.387646Z",
"url": "https://files.pythonhosted.org/packages/4b/87/f05c0215a0f27ee43a50f2e2e5d325b7e6910d1dad7e86519c3ebaf16d8d/superlinked-17.7.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "9b90867ce189ec955fa305f0e9bc948ca45fc99e5c9760a9a77916c69751b11d",
"md5": "4c6d91e980b7df4cbc50b3049b23e6cd",
"sha256": "4ede8216722c40bc97ced819b29cb8ce6ee3cd61104d4d4ec5ade405677b276e"
},
"downloads": -1,
"filename": "superlinked-17.7.1.tar.gz",
"has_sig": false,
"md5_digest": "4c6d91e980b7df4cbc50b3049b23e6cd",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<=3.12.3,>=3.10",
"size": 185591,
"upload_time": "2024-12-16T12:44:19",
"upload_time_iso_8601": "2024-12-16T12:44:19.614206Z",
"url": "https://files.pythonhosted.org/packages/9b/90/867ce189ec955fa305f0e9bc948ca45fc99e5c9760a9a77916c69751b11d/superlinked-17.7.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-12-16 12:44:19",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "superlinked"
}
