# SQLVector
A flexible and efficient Retrieval-Augmented Generation (RAG) library that uses SQL databases as vector stores through SQLAlchemy. Supports multiple backends including DuckDB with HNSW indexing and SQLite with VSS (Vector Similarity Search).
[](https://www.python.org/downloads/)
[](https://opensource.org/licenses/MIT)
[](https://github.com/dinedal/sqlvector/actions/workflows/tests.yml)
## Features
- **Multiple Backend Support**: DuckDB with HNSW indexing, SQLite with VSS extension
- **Async and Sync Interfaces**: Both `SQLRAG` (async) and `SyncSQLRAG` interfaces
- **Flexible Embedding Providers**: Bring your own embeddings or use the default provider
- **Batch Operations**: Efficient batch loading and querying
- **Multiple Similarity Functions**: Cosine, Euclidean, Inner Product similarity
- **Metadata Filtering**: Query with metadata filters and complex conditions
- **SQLAlchemy Integration**: Works with any SQLAlchemy-supported database
- **Export Capabilities**: Export to Polars DataFrames (DuckDB) or dictionaries
## Installation
### Basic Installation
```bash
pip install sqlvector
```
### With DuckDB Support
```bash
pip install "sqlvector[duckdb]"
```
### With SQLite Async Support
```bash
# For async SQLite support
pip install sqlvector aiosqlite sqlalchemy
```
### With Custom Embedding Providers
If you want to use custom embedding providers (e.g., with Sentence Transformers), install the required dependencies:
```bash
# For Sentence Transformers based embeddings
pip install sqlvector transformers sentence-transformers torch
```
### With Test Dependencies
```bash
pip install "sqlvector[test]"
```
### Development Installation
```bash
git clone https://github.com/dinedal/sqlvector.git
cd sqlvector
pip install -e ".[duckdb,test]"
```
## Quick Start
### Using Custom Embeddings
```python
import asyncio
from sqlalchemy.ext.asyncio import create_async_engine
from sqlvector import SQLRAG, EmbeddingProvider
from sentence_transformers import SentenceTransformer
from typing import List
import torch
import numpy as np
class CustomEmbeddingProvider(EmbeddingProvider):
def __init__(self, batch_size=4, max_seq_length=512, use_gpu=False):
MODEL_NAME = "sentence-transformers/all-MiniLM-L6-v2"
device = "cpu"
if use_gpu:
if torch.cuda.is_available():
device = "cuda"
elif torch.backends.mps.is_available():
device = "mps"
else:
raise RuntimeError(
"No suitable GPU found. Please check your PyTorch installation."
)
self.model = SentenceTransformer(
MODEL_NAME,
device=device,
)
# We can reduce the max_seq_length from the default for faster encoding
self.model.max_seq_length = max_seq_length
self.batch_size = batch_size
super().__init__()
async def embed(self, text: str) -> List[float]:
return self.model.encode(text).tolist()
async def embed_batch(self, texts: List[str]) -> List[List[float]]:
return self.model.encode(texts, batch_size=self.batch_size).tolist()
def similarity(self, vec1: List[float], vec2: List[float]) -> float:
return self.model.similarity(
np.array(vec1, dtype=np.float32),
np.array(vec2, dtype=np.float32),
).item()
async def main():
# Create async engine
engine = create_async_engine("sqlite+aiosqlite:///example.db")
# Initialize with custom embedding provider
rag = SQLRAG(
engine=engine,
embedding_provider=CustomEmbeddingProvider(use_gpu=True)
)
# Create tables
await rag.create_tables()
# Load documents
documents = [
{
"content": "The quick brown fox jumps over the lazy dog.",
"metadata": {"source": "example", "category": "animals"}
},
{
"content": "Machine learning is a subset of artificial intelligence.",
"metadata": {"source": "textbook", "category": "technology"}
}
]
document_ids = await rag.load_documents(documents)
print(f"Loaded {len(document_ids)} documents")
# Query similar documents
results = await rag.query("artificial intelligence", top_k=5)
for result in results:
print(f"Content: {result['content']}")
print(f"Similarity: {result['similarity']:.3f}")
print(f"Metadata: {result.get('metadata', {})}")
await engine.dispose()
if __name__ == "__main__":
asyncio.run(main())
```
### Using Synchronous Interface
```python
from sqlalchemy import create_engine
from sqlvector import SyncSQLRAG
# Create sync engine
engine = create_engine("sqlite:///example.db")
# Initialize RAG
rag = SyncSQLRAG(engine=engine)
# Create tables
rag.create_tables()
# Load and query documents
documents = [{"content": "Your content here", "metadata": {"key": "value"}}]
document_ids = rag.load_documents(documents)
results = rag.query("search term", top_k=5)
```
## Backend Configuration
### DuckDB with HNSW Indexing
```python
from sqlvector.backends.duckdb import DuckDBConfig, DuckDBRAG
config = DuckDBConfig(
connection_string="duckdb:///rag.duckdb",
embedding_dim=384,
use_hnsw=True, # Enable HNSW indexing
hnsw_config={
"max_elements": 100000,
"ef_construction": 200,
"ef": 100,
"M": 16
}
)
rag = DuckDBRAG(config=config)
```
### SQLite with VSS Extension
```python
from sqlvector.backends.sqlite import SQLiteConfig, SQLiteRAG
config = SQLiteConfig(
connection_string="sqlite:///rag.db",
embedding_dim=384,
use_vss=True, # Enable VSS extension
vss_version="v0.1.2"
)
rag = SQLiteRAG(config=config)
```
## Advanced Features
### Metadata Filtering
```python
# Query with metadata filters
results = await rag.query_with_filters(
filters={"category": "technology", "year": 2025},
query_text="machine learning",
top_k=10
)
```
### Batch Operations
```python
# Batch document loading
documents = [{"content": f"Document {i}", "metadata": {"id": i}} for i in range(100)]
document_ids = await rag.load_documents(documents, batch_size=10)
# Batch querying
queries = ["query1", "query2", "query3"]
batch_results = await rag.query_batch(queries, top_k=5)
```
### Export Data
```python
# For DuckDB backend - export to Polars DataFrame
df = rag.export_to_polars(include_embeddings=True)
# For SQLite backend - export to dictionary
data = rag.export_documents(include_embeddings=False)
```
## Architecture
SQLVector uses a protocol-based architecture that allows for flexible backend implementations:
- **Protocols**: Define interfaces for backends (`RAGSystemProtocol`, `DocumentLoaderProtocol`, `DocumentQuerierProtocol`)
- **Backends**: Pluggable database backends with specific optimizations
- **Embedding Service**: Handles text-to-vector conversion with pluggable providers
- **Models**: SQLAlchemy models for documents and embeddings
## Performance Considerations
- **DuckDB**: Best for analytical workloads, supports HNSW indexing for fast similarity search
- **SQLite**: Lightweight option with VSS extension for vector similarity
- **Batch Size**: Tune batch sizes based on your hardware and document sizes
- **Embedding Dimensions**: Lower dimensions generally provide faster search at the cost of accuracy
## Testing
Run the test suite:
```bash
# Run all tests
pytest
# Run specific backend tests
pytest tests/test_duckdb.py
pytest tests/test_sqlite.py
# Run with coverage
pytest --cov=sqlvector tests/
```
## Contributing
We welcome contributions! Please see [CONTRIBUTING.md](CONTRIBUTING.md) for details on:
- Development setup
- Code style guidelines
- Testing requirements
- Pull request process
## Examples
Check out the [examples/](examples/) directory for more detailed examples:
- Basic usage with different backends
- Custom embedding providers
- Advanced querying techniques
- Performance benchmarks
## License
This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
## Citation
If you use SQLVector in your research, please cite:
```bibtex
@software{sqlvector,
title = {SQLVector: SQL-based Retrieval-Augmented Generation},
year = {2025},
url = {https://github.com/dinedal/sqlvector}
}
```
## Acknowledgments
- Built on [SQLAlchemy](https://www.sqlalchemy.org/) for database abstraction
- DuckDB backend uses [DuckDB](https://duckdb.org/) with VSS extension
- SQLite backend uses [sqlite-vss](https://github.com/asg017/sqlite-vss) for vector similarity
## Support
- =� [Documentation](https://github.com/dinedal/sqlvector/wiki)
- = [Issue Tracker](https://github.com/dinedal/sqlvector/issues)
- =� [Discussions](https://github.com/dinedal/sqlvector/discussions)
Raw data
{
"_id": null,
"home_page": null,
"name": "sqlvector",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.12",
"maintainer_email": null,
"keywords": "rag, vector-search, sqlalchemy, duckdb, sqlite, embeddings, retrieval-augmented-generation",
"author": null,
"author_email": "Paul Bergeron <paul.d.bergeron@gmail.com>",
"download_url": "https://files.pythonhosted.org/packages/95/03/01cdcba222b63fc0085295817dd78cc4a423c7b0dd6874d2eb042e4f6db7/sqlvector-0.1.2.tar.gz",
"platform": null,
"description": "# SQLVector\n\nA flexible and efficient Retrieval-Augmented Generation (RAG) library that uses SQL databases as vector stores through SQLAlchemy. Supports multiple backends including DuckDB with HNSW indexing and SQLite with VSS (Vector Similarity Search).\n\n[](https://www.python.org/downloads/)\n[](https://opensource.org/licenses/MIT)\n[](https://github.com/dinedal/sqlvector/actions/workflows/tests.yml)\n\n## Features\n\n- **Multiple Backend Support**: DuckDB with HNSW indexing, SQLite with VSS extension\n- **Async and Sync Interfaces**: Both `SQLRAG` (async) and `SyncSQLRAG` interfaces\n- **Flexible Embedding Providers**: Bring your own embeddings or use the default provider\n- **Batch Operations**: Efficient batch loading and querying\n- **Multiple Similarity Functions**: Cosine, Euclidean, Inner Product similarity\n- **Metadata Filtering**: Query with metadata filters and complex conditions\n- **SQLAlchemy Integration**: Works with any SQLAlchemy-supported database\n- **Export Capabilities**: Export to Polars DataFrames (DuckDB) or dictionaries\n\n## Installation\n\n### Basic Installation\n\n```bash\npip install sqlvector\n```\n\n### With DuckDB Support\n\n```bash\npip install \"sqlvector[duckdb]\"\n```\n\n### With SQLite Async Support\n\n```bash\n# For async SQLite support\npip install sqlvector aiosqlite sqlalchemy\n```\n\n### With Custom Embedding Providers\n\nIf you want to use custom embedding providers (e.g., with Sentence Transformers), install the required dependencies:\n\n```bash\n# For Sentence Transformers based embeddings\npip install sqlvector transformers sentence-transformers torch\n```\n\n### With Test Dependencies\n\n```bash\npip install \"sqlvector[test]\"\n```\n\n### Development Installation\n\n```bash\ngit clone https://github.com/dinedal/sqlvector.git\ncd sqlvector\npip install -e \".[duckdb,test]\"\n```\n\n## Quick Start\n\n### Using Custom Embeddings\n\n```python\nimport asyncio\nfrom sqlalchemy.ext.asyncio import create_async_engine\nfrom sqlvector import SQLRAG, EmbeddingProvider\nfrom sentence_transformers import SentenceTransformer\nfrom typing import List\nimport torch\nimport numpy as np\n\nclass CustomEmbeddingProvider(EmbeddingProvider):\n def __init__(self, batch_size=4, max_seq_length=512, use_gpu=False):\n MODEL_NAME = \"sentence-transformers/all-MiniLM-L6-v2\"\n \n device = \"cpu\"\n if use_gpu:\n if torch.cuda.is_available():\n device = \"cuda\"\n elif torch.backends.mps.is_available():\n device = \"mps\"\n else:\n raise RuntimeError(\n \"No suitable GPU found. Please check your PyTorch installation.\"\n )\n \n self.model = SentenceTransformer(\n MODEL_NAME,\n device=device,\n )\n # We can reduce the max_seq_length from the default for faster encoding\n self.model.max_seq_length = max_seq_length\n self.batch_size = batch_size\n \n super().__init__()\n \n async def embed(self, text: str) -> List[float]:\n return self.model.encode(text).tolist()\n \n async def embed_batch(self, texts: List[str]) -> List[List[float]]:\n return self.model.encode(texts, batch_size=self.batch_size).tolist()\n \n def similarity(self, vec1: List[float], vec2: List[float]) -> float:\n return self.model.similarity(\n np.array(vec1, dtype=np.float32),\n np.array(vec2, dtype=np.float32),\n ).item()\n\nasync def main():\n # Create async engine\n engine = create_async_engine(\"sqlite+aiosqlite:///example.db\")\n \n # Initialize with custom embedding provider\n rag = SQLRAG(\n engine=engine, \n embedding_provider=CustomEmbeddingProvider(use_gpu=True)\n )\n \n # Create tables\n await rag.create_tables()\n \n # Load documents\n documents = [\n {\n \"content\": \"The quick brown fox jumps over the lazy dog.\",\n \"metadata\": {\"source\": \"example\", \"category\": \"animals\"}\n },\n {\n \"content\": \"Machine learning is a subset of artificial intelligence.\",\n \"metadata\": {\"source\": \"textbook\", \"category\": \"technology\"}\n }\n ]\n \n document_ids = await rag.load_documents(documents)\n print(f\"Loaded {len(document_ids)} documents\")\n \n # Query similar documents\n results = await rag.query(\"artificial intelligence\", top_k=5)\n \n for result in results:\n print(f\"Content: {result['content']}\")\n print(f\"Similarity: {result['similarity']:.3f}\")\n print(f\"Metadata: {result.get('metadata', {})}\")\n \n await engine.dispose()\n\nif __name__ == \"__main__\":\n asyncio.run(main())\n```\n\n### Using Synchronous Interface\n\n```python\nfrom sqlalchemy import create_engine\nfrom sqlvector import SyncSQLRAG\n\n# Create sync engine\nengine = create_engine(\"sqlite:///example.db\")\n\n# Initialize RAG\nrag = SyncSQLRAG(engine=engine)\n\n# Create tables\nrag.create_tables()\n\n# Load and query documents\ndocuments = [{\"content\": \"Your content here\", \"metadata\": {\"key\": \"value\"}}]\ndocument_ids = rag.load_documents(documents)\n\nresults = rag.query(\"search term\", top_k=5)\n```\n\n## Backend Configuration\n\n### DuckDB with HNSW Indexing\n\n```python\nfrom sqlvector.backends.duckdb import DuckDBConfig, DuckDBRAG\n\nconfig = DuckDBConfig(\n connection_string=\"duckdb:///rag.duckdb\",\n embedding_dim=384,\n use_hnsw=True, # Enable HNSW indexing\n hnsw_config={\n \"max_elements\": 100000,\n \"ef_construction\": 200,\n \"ef\": 100,\n \"M\": 16\n }\n)\n\nrag = DuckDBRAG(config=config)\n```\n\n### SQLite with VSS Extension\n\n```python\nfrom sqlvector.backends.sqlite import SQLiteConfig, SQLiteRAG\n\nconfig = SQLiteConfig(\n connection_string=\"sqlite:///rag.db\",\n embedding_dim=384,\n use_vss=True, # Enable VSS extension\n vss_version=\"v0.1.2\"\n)\n\nrag = SQLiteRAG(config=config)\n```\n\n## Advanced Features\n\n### Metadata Filtering\n\n```python\n# Query with metadata filters\nresults = await rag.query_with_filters(\n filters={\"category\": \"technology\", \"year\": 2025},\n query_text=\"machine learning\",\n top_k=10\n)\n```\n\n### Batch Operations\n\n```python\n# Batch document loading\ndocuments = [{\"content\": f\"Document {i}\", \"metadata\": {\"id\": i}} for i in range(100)]\ndocument_ids = await rag.load_documents(documents, batch_size=10)\n\n# Batch querying\nqueries = [\"query1\", \"query2\", \"query3\"]\nbatch_results = await rag.query_batch(queries, top_k=5)\n```\n\n### Export Data\n\n```python\n# For DuckDB backend - export to Polars DataFrame\ndf = rag.export_to_polars(include_embeddings=True)\n\n# For SQLite backend - export to dictionary\ndata = rag.export_documents(include_embeddings=False)\n```\n\n## Architecture\n\nSQLVector uses a protocol-based architecture that allows for flexible backend implementations:\n\n- **Protocols**: Define interfaces for backends (`RAGSystemProtocol`, `DocumentLoaderProtocol`, `DocumentQuerierProtocol`)\n- **Backends**: Pluggable database backends with specific optimizations\n- **Embedding Service**: Handles text-to-vector conversion with pluggable providers\n- **Models**: SQLAlchemy models for documents and embeddings\n\n## Performance Considerations\n\n- **DuckDB**: Best for analytical workloads, supports HNSW indexing for fast similarity search\n- **SQLite**: Lightweight option with VSS extension for vector similarity\n- **Batch Size**: Tune batch sizes based on your hardware and document sizes\n- **Embedding Dimensions**: Lower dimensions generally provide faster search at the cost of accuracy\n\n## Testing\n\nRun the test suite:\n\n```bash\n# Run all tests\npytest\n\n# Run specific backend tests\npytest tests/test_duckdb.py\npytest tests/test_sqlite.py\n\n# Run with coverage\npytest --cov=sqlvector tests/\n```\n\n## Contributing\n\nWe welcome contributions! Please see [CONTRIBUTING.md](CONTRIBUTING.md) for details on:\n\n- Development setup\n- Code style guidelines\n- Testing requirements\n- Pull request process\n\n## Examples\n\nCheck out the [examples/](examples/) directory for more detailed examples:\n\n- Basic usage with different backends\n- Custom embedding providers\n- Advanced querying techniques\n- Performance benchmarks\n\n## License\n\nThis project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.\n\n## Citation\n\nIf you use SQLVector in your research, please cite:\n\n```bibtex\n@software{sqlvector,\n title = {SQLVector: SQL-based Retrieval-Augmented Generation},\n year = {2025},\n url = {https://github.com/dinedal/sqlvector}\n}\n```\n\n## Acknowledgments\n\n- Built on [SQLAlchemy](https://www.sqlalchemy.org/) for database abstraction\n- DuckDB backend uses [DuckDB](https://duckdb.org/) with VSS extension\n- SQLite backend uses [sqlite-vss](https://github.com/asg017/sqlite-vss) for vector similarity\n\n## Support\n\n- =\ufffd [Documentation](https://github.com/dinedal/sqlvector/wiki)\n- =\u001b [Issue Tracker](https://github.com/dinedal/sqlvector/issues)\n- =\ufffd [Discussions](https://github.com/dinedal/sqlvector/discussions)\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "A flexible RAG library using SQL databases as vector stores through SQLAlchemy",
"version": "0.1.2",
"project_urls": {
"Changelog": "https://github.com/dinedal/sql-rag/blob/main/CHANGELOG.md",
"Documentation": "https://github.com/dinedal/sql-rag/wiki",
"Issues": "https://github.com/dinedal/sql-rag/issues",
"Repository": "https://github.com/dinedal/sql-rag"
},
"split_keywords": [
"rag",
" vector-search",
" sqlalchemy",
" duckdb",
" sqlite",
" embeddings",
" retrieval-augmented-generation"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "0a9c06ec2acd517faa01bb7ebe99c9b3c315a6b06fc3e801e9b0b70e090f3ff5",
"md5": "3999e25ea8cbc5191978b9e821decce5",
"sha256": "d3a8a5a5387780804b889ecf32c95e686187bffce6559b6900d992a7db316e9f"
},
"downloads": -1,
"filename": "sqlvector-0.1.2-py3-none-any.whl",
"has_sig": false,
"md5_digest": "3999e25ea8cbc5191978b9e821decce5",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.12",
"size": 65332,
"upload_time": "2025-09-05T03:53:27",
"upload_time_iso_8601": "2025-09-05T03:53:27.399400Z",
"url": "https://files.pythonhosted.org/packages/0a/9c/06ec2acd517faa01bb7ebe99c9b3c315a6b06fc3e801e9b0b70e090f3ff5/sqlvector-0.1.2-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "950301cdcba222b63fc0085295817dd78cc4a423c7b0dd6874d2eb042e4f6db7",
"md5": "6087b17bdcb839fde45b275c73305abb",
"sha256": "d070042c90961405c09398b402b569d122176f2dea619f1b35e1a8cfdea0fb81"
},
"downloads": -1,
"filename": "sqlvector-0.1.2.tar.gz",
"has_sig": false,
"md5_digest": "6087b17bdcb839fde45b275c73305abb",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.12",
"size": 78600,
"upload_time": "2025-09-05T03:53:29",
"upload_time_iso_8601": "2025-09-05T03:53:29.223068Z",
"url": "https://files.pythonhosted.org/packages/95/03/01cdcba222b63fc0085295817dd78cc4a423c7b0dd6874d2eb042e4f6db7/sqlvector-0.1.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-09-05 03:53:29",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "dinedal",
"github_project": "sql-rag",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "sqlvector"
}
JSON
