# 🚀 d-vecDB
[](https://rustup.rs/)
[](LICENSE)
[](#-performance-benchmarks)
**A high-performance, production-ready vector database written in Rust**
d-vecDB is a modern vector database designed for AI applications, semantic search, and similarity matching. Built from the ground up in Rust, it delivers exceptional performance, memory safety, and concurrent processing capabilities.
---
## 🎯 **Key Features**
### ⚡ **Ultra-High Performance**
- **Sub-microsecond vector operations** (28-76ns per distance calculation)
- **HNSW indexing** with O(log N) search complexity
- **Concurrent processing** with Rust's fearless concurrency
- **Memory-mapped storage** for efficient large dataset handling
### 🏗️ **Production Architecture**
- **gRPC & REST APIs** for universal client compatibility
- **Write-Ahead Logging (WAL)** for ACID durability and crash recovery
- **Multi-threaded indexing** and query processing
- **Comprehensive error handling** and observability
### 🔧 **Developer Experience**
- **Type-safe APIs** with Protocol Buffers
- **Rich metadata support** with JSON field storage
- **Comprehensive benchmarking** suite with HTML reports
- **CLI tools** for database management
### 📊 **Enterprise Ready**
- **Horizontal scaling** capabilities
- **Monitoring integration** with Prometheus metrics
- **Flexible deployment** (standalone, containerized, embedded)
- **Cross-platform support** (Linux, macOS, Windows)
---
## 📈 **Benchmark Results**
*Tested on macOS Darwin 24.6.0 with optimized release builds*
### **Distance Calculations**
| Operation | Latency | Throughput |
|-----------|---------|------------|
| **Dot Product** | 28.3 ns | 35.4M ops/sec |
| **Euclidean Distance** | 30.6 ns | 32.7M ops/sec |
| **Cosine Similarity** | 76.1 ns | 13.1M ops/sec |
### **HNSW Index Operations**
| Operation | Performance | Scale |
|-----------|-------------|--------|
| **Vector Insertion** | 7,108 vectors/sec | 1,000 vectors benchmark |
| **Vector Search** | 13,150 queries/sec | 5,000 vector dataset |
| **With Metadata** | 2,560 inserts/sec | Rich JSON metadata |
### **Performance Projections on Higher-End Hardware**
Based on our benchmark results, here are conservative performance extrapolations for production hardware:
#### **High-End Server (32-core AMD EPYC, 128GB RAM, NVMe)**
| Operation | Current (Mac) | Projected (Server) | Improvement |
|-----------|---------------|-------------------|-------------|
| **Distance Calculations** | 35M ops/sec | **150M+ ops/sec** | 4.3x |
| **Vector Insertion** | 7K vectors/sec | **50K+ vectors/sec** | 7x |
| **Vector Search** | 13K queries/sec | **100K+ queries/sec** | 7.7x |
| **Concurrent Queries** | Single-threaded | **500K+ queries/sec** | 38x |
#### **Optimized Cloud Instance (16-core, 64GB RAM, SSD)**
| Operation | Current (Mac) | Projected (Cloud) | Improvement |
|-----------|---------------|-------------------|-------------|
| **Distance Calculations** | 35M ops/sec | **80M+ ops/sec** | 2.3x |
| **Vector Insertion** | 7K vectors/sec | **25K+ vectors/sec** | 3.6x |
| **Vector Search** | 13K queries/sec | **45K+ queries/sec** | 3.5x |
| **Concurrent Queries** | Single-threaded | **180K+ queries/sec** | 14x |
*Projections based on CPU core scaling, memory bandwidth improvements, and storage I/O optimizations*
---
## 🏗️ **Architecture**
```
┌─────────────────────────────────────────────────────────────┐
│ 🎯 d-vecDB Stack │
├─────────────────────────────────────────────────────────────┤
│ CLI Tool │ Client SDKs │ REST + gRPC APIs │
│ (Management) │ (Rust/Python) │ (Universal Access) │
├─────────────────────────────────────────────────────────────┤
│ Vector Store Engine │
│ (Indexing + Storage + Querying) │
├─────────────────────────────────────────────────────────────┤
│ HNSW Index │ WAL Storage │ Memory Mapping │
│ (O(log N)) │ (Durability) │ (Performance) │
└─────────────────────────────────────────────────────────────┘
```
### **Core Components**
- **🔍 HNSW Index**: Hierarchical Navigable Small World graphs for approximate nearest neighbor search
- **💾 Storage Engine**: Memory-mapped files with write-ahead logging for durability
- **🌐 API Layer**: Both REST (HTTP/JSON) and gRPC (Protocol Buffers) interfaces
- **📊 Monitoring**: Built-in Prometheus metrics and comprehensive logging
- **🔧 CLI Tools**: Database management, collection operations, and administrative tasks
---
## 🚀 **Quick Start**
### **Installation**
**Option 1: Install from PyPI (Recommended)**
```bash
# Install d-vecDB with Python client
pip install d-vecdb
# Or install with development extras
pip install d-vecdb[dev,docs,examples]
```
**Option 2: Install from Source**
```bash
# Clone the repository
git clone https://github.com/rdmurugan/d-vecDB.git
cd d-vecDB
# Quick install using script
./scripts/install.sh
# Or manual installation
pip install .
```
**Option 3: For Development**
```bash
# Clone and setup development environment
git clone https://github.com/rdmurugan/d-vecDB.git
cd d-vecDB
# Install in development mode with all extras
./scripts/install.sh dev
# Build Rust server components
./scripts/build-server.sh
```
**Option 4: Using Virtual Environment**
```bash
# Create isolated environment
./scripts/install.sh venv
source venv/bin/activate # Linux/macOS
# venv\Scripts\activate # Windows
```
### **Start the Server**
```bash
# Start with default configuration
./target/release/vectordb-server --config config.toml
# Or with custom settings
./target/release/vectordb-server \
--host 0.0.0.0 \
--port 8080 \
--data-dir /path/to/data \
--log-level info
```
### **Basic Usage**
```bash
# Create a collection
curl -X POST http://localhost:8080/collections \
-H "Content-Type: application/json" \
-d '{
"name": "documents",
"dimension": 128,
"distance_metric": "cosine"
}'
# Insert vectors
curl -X POST http://localhost:8080/collections/documents/vectors \
-H "Content-Type: application/json" \
-d '{
"id": "doc1",
"data": [0.1, 0.2, 0.3, ...],
"metadata": {"title": "Example Document"}
}'
# Search for similar vectors
curl -X POST http://localhost:8080/collections/documents/search \
-H "Content-Type: application/json" \
-d '{
"query_vector": [0.1, 0.2, 0.3, ...],
"limit": 10
}'
```
---
## 🛠️ **Development Setup**
### **Prerequisites**
- **Rust** 1.70+ ([Install Rust](https://rustup.rs/))
- **Protocol Buffers** compiler (`protoc`)
- **Git** for version control
### **Build Instructions**
```bash
# Development build
cargo build
# Optimized release build
cargo build --release
# Run all tests
cargo test
# Run benchmarks
cargo bench --package vectordb-common
# Generate documentation
cargo doc --open
```
### **Project Structure**
```
d-vecDB/
├── common/ # Core types, distance functions, utilities
├── index/ # HNSW indexing implementation
├── storage/ # WAL, memory-mapping, persistence
├── vectorstore/ # Main vector store engine
├── server/ # REST & gRPC API servers
├── python-client/ # 🐍 Official Python client library
├── client/ # Additional client SDKs and libraries
├── cli/ # Command-line tools
├── proto/ # Protocol Buffer definitions
└── benchmarks/ # Performance testing suite
```
---
## 📚 **Client Libraries**
d-vecDB provides official client libraries for multiple programming languages:
### 🐍 **Python Client**
[](https://www.python.org/downloads/)
**Full-featured Python client with async support, NumPy integration, and type safety.**
- 🔄 **Sync & Async**: Both synchronous and asynchronous clients
- ⚡ **High Performance**: Concurrent batch operations (1000+ vectors/sec)
- 🧮 **NumPy Native**: Direct NumPy array support
- 🔒 **Type Safe**: Pydantic models with validation
- 🌐 **Multi-Protocol**: REST and gRPC support
```bash
# Install from PyPI
pip install vectordb-client
# Quick usage
from vectordb_client import VectorDBClient
import numpy as np
client = VectorDBClient()
client.create_collection_simple("docs", 384, "cosine")
client.insert_simple("docs", "doc_1", np.random.random(384))
results = client.search_simple("docs", np.random.random(384), limit=5)
```
**📖 [Complete Python Documentation →](python-client/README.md)**
### 🦀 **Rust Client** *(Native)*
Direct access to the native Rust API for maximum performance.
### 🌐 **HTTP/REST API**
Language-agnostic REST API with OpenAPI specification.
**📖 [API Documentation →](docs/api.md)**
### 🚧 **Coming Soon**
- **JavaScript/TypeScript** client
- **Go** client
- **Java** client
- **C++** bindings
---
## 📊 **Comprehensive Benchmarking**
### **Running Benchmarks**
```bash
# Core performance benchmarks
cargo bench --package vectordb-common
# Generate HTML reports
cargo bench --package vectordb-common
open target/criterion/report/index.html
# Custom benchmark suite
./scripts/run-comprehensive-benchmarks.sh
```
### **Benchmark Categories**
1. **🧮 Distance Calculations**: Core mathematical operations (cosine, euclidean, dot product)
2. **🗂️ Index Operations**: Vector insertion, search, and maintenance
3. **💾 Storage Performance**: WAL writes, memory-mapped reads, persistence
4. **🌐 API Throughput**: REST and gRPC endpoint performance
5. **📈 Scaling Tests**: Performance under load with varying dataset sizes
### **Hardware Optimization Guide**
#### **For Maximum Insertion Throughput:**
- **CPU**: High core count (32+ cores) for parallel indexing
- **RAM**: Large memory pool (128GB+) for index caching
- **Storage**: NVMe SSDs for fast WAL writes
#### **For Maximum Query Performance:**
- **CPU**: High single-thread performance with many cores
- **RAM**: Fast memory (DDR4-3200+) for index traversal
- **Network**: High bandwidth for concurrent client connections
#### **For Large Scale Deployments:**
- **Distributed Setup**: Multiple nodes with load balancing
- **Storage Tiering**: Hot data in memory, warm data on SSD
- **Monitoring**: Comprehensive metrics and alerting
---
## 🔧 **Configuration**
### **Server Configuration**
```toml
# config.toml
[server]
host = "0.0.0.0"
port = 8080
grpc_port = 9090
workers = 8
[storage]
data_dir = "./data"
wal_sync_interval = "1s"
memory_map_size = "1GB"
[index]
hnsw_max_connections = 16
hnsw_ef_construction = 200
hnsw_max_layer = 16
[monitoring]
enable_metrics = true
prometheus_port = 9091
log_level = "info"
```
### **Performance Tuning**
```toml
[performance]
# Optimize for insertion throughput
batch_size = 1000
insert_workers = 16
# Optimize for query latency
query_cache_size = "500MB"
prefetch_enabled = true
# Memory management
gc_interval = "30s"
memory_limit = "8GB"
```
---
## 🌐 **API Reference**
### **REST API**
| Endpoint | Method | Description |
|----------|--------|-------------|
| `/collections` | POST | Create collection |
| `/collections/{name}` | GET | Get collection info |
| `/collections/{name}/vectors` | POST | Insert vectors |
| `/collections/{name}/search` | POST | Search vectors |
| `/collections/{name}/vectors/{id}` | DELETE | Delete vector |
| `/stats` | GET | Server statistics |
| `/health` | GET | Health check |
### **gRPC Services**
```protobuf
service VectorDb {
rpc CreateCollection(CreateCollectionRequest) returns (CreateCollectionResponse);
rpc Insert(InsertRequest) returns (InsertResponse);
rpc BatchInsert(BatchInsertRequest) returns (BatchInsertResponse);
rpc Query(QueryRequest) returns (QueryResponse);
rpc Delete(DeleteRequest) returns (DeleteResponse);
rpc GetStats(GetStatsRequest) returns (GetStatsResponse);
}
```
### **Client SDKs**
```rust
// Rust Client
use vectordb_client::VectorDbClient;
let client = VectorDbClient::new("http://localhost:8080").await?;
// Create collection
client.create_collection("documents", 128, DistanceMetric::Cosine).await?;
// Insert vector
client.insert("documents", "doc1", vec![0.1, 0.2, 0.3], metadata).await?;
// Search
let results = client.search("documents", query_vector, 10).await?;
```
```python
# Python Client (Coming Soon)
import vectordb
client = vectordb.Client("http://localhost:8080")
client.create_collection("documents", 128, "cosine")
client.insert("documents", "doc1", [0.1, 0.2, 0.3], {"title": "Example"})
results = client.search("documents", query_vector, limit=10)
```
---
## 🔍 **Use Cases**
### **🤖 AI & Machine Learning**
- **Embedding storage** for transformer models (BERT, GPT, etc.)
- **Recommendation engines** with user/item similarity
- **Content-based filtering** and personalization
### **🔍 Search & Discovery**
- **Semantic search** in documents and knowledge bases
- **Image/video similarity** search and retrieval
- **Product recommendation** in e-commerce platforms
### **📊 Data Analytics**
- **Anomaly detection** in high-dimensional data
- **Clustering and classification** of complex datasets
- **Feature matching** in computer vision applications
### **🏢 Enterprise Applications**
- **Document similarity** in legal and compliance systems
- **Fraud detection** through pattern matching
- **Customer segmentation** and behavioral analysis
---
## 🚦 **Production Deployment**
### **Docker Deployment**
```dockerfile
FROM rust:1.70 as builder
COPY . .
RUN cargo build --release
FROM debian:bookworm-slim
COPY --from=builder /target/release/vectordb-server /usr/local/bin/
EXPOSE 8080 9090 9091
CMD ["vectordb-server", "--config", "/etc/vectordb/config.toml"]
```
```bash
# Build and run
docker build -t d-vecdb .
docker run -p 8080:8080 -p 9090:9090 -v ./data:/data d-vecdb
```
### **Kubernetes Deployment**
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: d-vecdb
spec:
replicas: 3
selector:
matchLabels:
app: d-vecdb
template:
metadata:
labels:
app: d-vecdb
spec:
containers:
- name: d-vecdb
image: d-vecdb:latest
ports:
- containerPort: 8080
- containerPort: 9090
resources:
requests:
memory: "2Gi"
cpu: "500m"
limits:
memory: "8Gi"
cpu: "4"
```
### **Monitoring Integration**
```yaml
# Prometheus configuration
- job_name: 'd-vecdb'
static_configs:
- targets: ['d-vecdb:9091']
scrape_interval: 15s
metrics_path: /metrics
```
---
## 📈 **Performance Comparison**
### **vs. Traditional Vector Databases**
| Feature | d-vecDB | Pinecone | Weaviate | Qdrant |
|---------|-------------|----------|----------|--------|
| **Language** | Rust | Python/C++ | Go | Rust |
| **Memory Safety** | ✅ Zero-cost | ❌ Manual | ❌ GC Overhead | ✅ Zero-cost |
| **Concurrency** | ✅ Native | ⚠️ Limited | ⚠️ GC Pauses | ✅ Native |
| **Deployment** | ✅ Single Binary | ❌ Cloud Only | ⚠️ Complex | ✅ Flexible |
| **Performance** | ✅ 35M ops/sec | ⚠️ Network Bound | ⚠️ GC Limited | ✅ Comparable |
### **Scaling Characteristics**
| Dataset Size | Query Latency | Memory Usage | Throughput |
|-------------|---------------|--------------|------------|
| **1K vectors** | <100µs | <10MB | 50K+ qps |
| **100K vectors** | <500µs | <500MB | 25K+ qps |
| **1M vectors** | <2ms | <2GB | 15K+ qps |
| **10M vectors** | <10ms | <8GB | 8K+ qps |
---
## 🤝 **Contributing**
We welcome contributions! Please see our [Contributing Guide](CONTRIBUTING.md) for details.
### **Development Workflow**
```bash
# Fork and clone the repository
git clone https://github.com/your-username/d-vecDB.git
cd d-vecDB
# Create a feature branch
git checkout -b feature/amazing-feature
# Make changes and test
cargo test
cargo clippy
cargo fmt
# Submit a pull request
git push origin feature/amazing-feature
```
### **Areas for Contribution**
- 🚀 Performance optimizations and SIMD implementations
- 🌐 Additional client SDK languages (Python, JavaScript, Java)
- 📊 Advanced indexing algorithms (IVF, PQ, LSH)
- 🔧 Operational tools and monitoring dashboards
- 📚 Documentation and example applications
---
## 📄 **License**
This project is licensed under the d-vecDB Enterprise License - see the [LICENSE](LICENSE) file for details.
**For Enterprise Use**: Commercial usage requires a separate enterprise license. Contact durai@infinidatum.com for licensing terms.
---
## 🆘 **Support**
- **📧 Email**: durai@infinidatum.com
- **💬 Discord**: [d-vecDB Community](https://discord.gg/d-vecdb)
- **🐛 Issues**: [GitHub Issues](https://github.com/rdmurugan/d-vecDB/issues)
- **📚 Documentation**: [docs.d-vecdb.com](https://docs.d-vecdb.com)
---
## 🙏 **Acknowledgments**
- Built with ❤️ in Rust
- Inspired by modern vector database architectures
- Powered by the amazing Rust ecosystem
- Community-driven development
---
**⚡ Ready to build the future of AI-powered applications? Get started with d-vecDB today!**
Raw data
{
"_id": null,
"home_page": "https://github.com/rdmurugan/d-vecDB",
"name": "d-vecdb",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": "vector database, similarity search, machine learning, embeddings, HNSW, rust",
"author": "Durai",
"author_email": "Durai <durai@infinidatum.com>",
"download_url": "https://files.pythonhosted.org/packages/be/31/5985bec279e9eff37acc455622582798d3f408ede7bd388f4a5761d52c33/d_vecdb-0.1.1.tar.gz",
"platform": null,
"description": "# \ud83d\ude80 d-vecDB\n\n[](https://rustup.rs/)\n[](LICENSE)\n[](#-performance-benchmarks)\n\n**A high-performance, production-ready vector database written in Rust**\n\nd-vecDB is a modern vector database designed for AI applications, semantic search, and similarity matching. Built from the ground up in Rust, it delivers exceptional performance, memory safety, and concurrent processing capabilities.\n\n---\n\n## \ud83c\udfaf **Key Features**\n\n### \u26a1 **Ultra-High Performance**\n- **Sub-microsecond vector operations** (28-76ns per distance calculation)\n- **HNSW indexing** with O(log N) search complexity\n- **Concurrent processing** with Rust's fearless concurrency\n- **Memory-mapped storage** for efficient large dataset handling\n\n### \ud83c\udfd7\ufe0f **Production Architecture**\n- **gRPC & REST APIs** for universal client compatibility\n- **Write-Ahead Logging (WAL)** for ACID durability and crash recovery \n- **Multi-threaded indexing** and query processing\n- **Comprehensive error handling** and observability\n\n### \ud83d\udd27 **Developer Experience**\n- **Type-safe APIs** with Protocol Buffers\n- **Rich metadata support** with JSON field storage\n- **Comprehensive benchmarking** suite with HTML reports\n- **CLI tools** for database management\n\n### \ud83d\udcca **Enterprise Ready**\n- **Horizontal scaling** capabilities\n- **Monitoring integration** with Prometheus metrics\n- **Flexible deployment** (standalone, containerized, embedded)\n- **Cross-platform support** (Linux, macOS, Windows)\n\n---\n\n## \ud83d\udcc8 **Benchmark Results**\n\n*Tested on macOS Darwin 24.6.0 with optimized release builds*\n\n### **Distance Calculations**\n| Operation | Latency | Throughput |\n|-----------|---------|------------|\n| **Dot Product** | 28.3 ns | 35.4M ops/sec |\n| **Euclidean Distance** | 30.6 ns | 32.7M ops/sec |\n| **Cosine Similarity** | 76.1 ns | 13.1M ops/sec |\n\n### **HNSW Index Operations** \n| Operation | Performance | Scale |\n|-----------|-------------|--------|\n| **Vector Insertion** | 7,108 vectors/sec | 1,000 vectors benchmark |\n| **Vector Search** | 13,150 queries/sec | 5,000 vector dataset |\n| **With Metadata** | 2,560 inserts/sec | Rich JSON metadata |\n\n### **Performance Projections on Higher-End Hardware**\n\nBased on our benchmark results, here are conservative performance extrapolations for production hardware:\n\n#### **High-End Server (32-core AMD EPYC, 128GB RAM, NVMe)**\n| Operation | Current (Mac) | Projected (Server) | Improvement |\n|-----------|---------------|-------------------|-------------|\n| **Distance Calculations** | 35M ops/sec | **150M+ ops/sec** | 4.3x |\n| **Vector Insertion** | 7K vectors/sec | **50K+ vectors/sec** | 7x |\n| **Vector Search** | 13K queries/sec | **100K+ queries/sec** | 7.7x |\n| **Concurrent Queries** | Single-threaded | **500K+ queries/sec** | 38x |\n\n#### **Optimized Cloud Instance (16-core, 64GB RAM, SSD)**\n| Operation | Current (Mac) | Projected (Cloud) | Improvement |\n|-----------|---------------|-------------------|-------------|\n| **Distance Calculations** | 35M ops/sec | **80M+ ops/sec** | 2.3x |\n| **Vector Insertion** | 7K vectors/sec | **25K+ vectors/sec** | 3.6x |\n| **Vector Search** | 13K queries/sec | **45K+ queries/sec** | 3.5x |\n| **Concurrent Queries** | Single-threaded | **180K+ queries/sec** | 14x |\n\n*Projections based on CPU core scaling, memory bandwidth improvements, and storage I/O optimizations*\n\n---\n\n## \ud83c\udfd7\ufe0f **Architecture**\n\n```\n\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n\u2502 \ud83c\udfaf d-vecDB Stack \u2502\n\u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524\n\u2502 CLI Tool \u2502 Client SDKs \u2502 REST + gRPC APIs \u2502\n\u2502 (Management) \u2502 (Rust/Python) \u2502 (Universal Access) \u2502\n\u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524\n\u2502 Vector Store Engine \u2502\n\u2502 (Indexing + Storage + Querying) \u2502\n\u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524\n\u2502 HNSW Index \u2502 WAL Storage \u2502 Memory Mapping \u2502\n\u2502 (O(log N)) \u2502 (Durability) \u2502 (Performance) \u2502\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n```\n\n### **Core Components**\n\n- **\ud83d\udd0d HNSW Index**: Hierarchical Navigable Small World graphs for approximate nearest neighbor search\n- **\ud83d\udcbe Storage Engine**: Memory-mapped files with write-ahead logging for durability\n- **\ud83c\udf10 API Layer**: Both REST (HTTP/JSON) and gRPC (Protocol Buffers) interfaces\n- **\ud83d\udcca Monitoring**: Built-in Prometheus metrics and comprehensive logging\n- **\ud83d\udd27 CLI Tools**: Database management, collection operations, and administrative tasks\n\n---\n\n## \ud83d\ude80 **Quick Start**\n\n### **Installation**\n\n**Option 1: Install from PyPI (Recommended)**\n```bash\n# Install d-vecDB with Python client\npip install d-vecdb\n\n# Or install with development extras\npip install d-vecdb[dev,docs,examples]\n```\n\n**Option 2: Install from Source**\n```bash\n# Clone the repository\ngit clone https://github.com/rdmurugan/d-vecDB.git\ncd d-vecDB\n\n# Quick install using script\n./scripts/install.sh\n\n# Or manual installation\npip install .\n```\n\n**Option 3: For Development**\n```bash\n# Clone and setup development environment\ngit clone https://github.com/rdmurugan/d-vecDB.git\ncd d-vecDB\n\n# Install in development mode with all extras\n./scripts/install.sh dev\n\n# Build Rust server components\n./scripts/build-server.sh\n```\n\n**Option 4: Using Virtual Environment**\n```bash\n# Create isolated environment\n./scripts/install.sh venv\nsource venv/bin/activate # Linux/macOS\n# venv\\Scripts\\activate # Windows\n```\n\n### **Start the Server**\n\n```bash\n# Start with default configuration\n./target/release/vectordb-server --config config.toml\n\n# Or with custom settings\n./target/release/vectordb-server \\\n --host 0.0.0.0 \\\n --port 8080 \\\n --data-dir /path/to/data \\\n --log-level info\n```\n\n### **Basic Usage**\n\n```bash\n# Create a collection\ncurl -X POST http://localhost:8080/collections \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"name\": \"documents\",\n \"dimension\": 128,\n \"distance_metric\": \"cosine\"\n }'\n\n# Insert vectors\ncurl -X POST http://localhost:8080/collections/documents/vectors \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"id\": \"doc1\",\n \"data\": [0.1, 0.2, 0.3, ...],\n \"metadata\": {\"title\": \"Example Document\"}\n }'\n\n# Search for similar vectors\ncurl -X POST http://localhost:8080/collections/documents/search \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"query_vector\": [0.1, 0.2, 0.3, ...],\n \"limit\": 10\n }'\n```\n\n---\n\n## \ud83d\udee0\ufe0f **Development Setup**\n\n### **Prerequisites**\n- **Rust** 1.70+ ([Install Rust](https://rustup.rs/))\n- **Protocol Buffers** compiler (`protoc`)\n- **Git** for version control\n\n### **Build Instructions**\n\n```bash\n# Development build\ncargo build\n\n# Optimized release build \ncargo build --release\n\n# Run all tests\ncargo test\n\n# Run benchmarks\ncargo bench --package vectordb-common\n\n# Generate documentation\ncargo doc --open\n```\n\n### **Project Structure**\n\n```\nd-vecDB/\n\u251c\u2500\u2500 common/ # Core types, distance functions, utilities\n\u251c\u2500\u2500 index/ # HNSW indexing implementation\n\u251c\u2500\u2500 storage/ # WAL, memory-mapping, persistence\n\u251c\u2500\u2500 vectorstore/ # Main vector store engine\n\u251c\u2500\u2500 server/ # REST & gRPC API servers\n\u251c\u2500\u2500 python-client/ # \ud83d\udc0d Official Python client library\n\u251c\u2500\u2500 client/ # Additional client SDKs and libraries\n\u251c\u2500\u2500 cli/ # Command-line tools\n\u251c\u2500\u2500 proto/ # Protocol Buffer definitions\n\u2514\u2500\u2500 benchmarks/ # Performance testing suite\n```\n\n---\n\n## \ud83d\udcda **Client Libraries**\n\nd-vecDB provides official client libraries for multiple programming languages:\n\n### \ud83d\udc0d **Python Client**\n[](https://www.python.org/downloads/)\n\n**Full-featured Python client with async support, NumPy integration, and type safety.**\n\n- \ud83d\udd04 **Sync & Async**: Both synchronous and asynchronous clients\n- \u26a1 **High Performance**: Concurrent batch operations (1000+ vectors/sec)\n- \ud83e\uddee **NumPy Native**: Direct NumPy array support\n- \ud83d\udd12 **Type Safe**: Pydantic models with validation\n- \ud83c\udf10 **Multi-Protocol**: REST and gRPC support\n\n```bash\n# Install from PyPI\npip install vectordb-client\n\n# Quick usage\nfrom vectordb_client import VectorDBClient\nimport numpy as np\n\nclient = VectorDBClient()\nclient.create_collection_simple(\"docs\", 384, \"cosine\")\nclient.insert_simple(\"docs\", \"doc_1\", np.random.random(384))\nresults = client.search_simple(\"docs\", np.random.random(384), limit=5)\n```\n\n**\ud83d\udcd6 [Complete Python Documentation \u2192](python-client/README.md)**\n\n### \ud83e\udd80 **Rust Client** *(Native)*\n\nDirect access to the native Rust API for maximum performance.\n\n### \ud83c\udf10 **HTTP/REST API**\nLanguage-agnostic REST API with OpenAPI specification.\n\n**\ud83d\udcd6 [API Documentation \u2192](docs/api.md)**\n\n### \ud83d\udea7 **Coming Soon**\n- **JavaScript/TypeScript** client\n- **Go** client \n- **Java** client\n- **C++** bindings\n\n---\n\n## \ud83d\udcca **Comprehensive Benchmarking**\n\n### **Running Benchmarks**\n\n```bash\n# Core performance benchmarks\ncargo bench --package vectordb-common\n\n# Generate HTML reports\ncargo bench --package vectordb-common\nopen target/criterion/report/index.html\n\n# Custom benchmark suite\n./scripts/run-comprehensive-benchmarks.sh\n```\n\n### **Benchmark Categories**\n\n1. **\ud83e\uddee Distance Calculations**: Core mathematical operations (cosine, euclidean, dot product)\n2. **\ud83d\uddc2\ufe0f Index Operations**: Vector insertion, search, and maintenance \n3. **\ud83d\udcbe Storage Performance**: WAL writes, memory-mapped reads, persistence\n4. **\ud83c\udf10 API Throughput**: REST and gRPC endpoint performance\n5. **\ud83d\udcc8 Scaling Tests**: Performance under load with varying dataset sizes\n\n### **Hardware Optimization Guide**\n\n#### **For Maximum Insertion Throughput:**\n- **CPU**: High core count (32+ cores) for parallel indexing\n- **RAM**: Large memory pool (128GB+) for index caching \n- **Storage**: NVMe SSDs for fast WAL writes\n\n#### **For Maximum Query Performance:**\n- **CPU**: High single-thread performance with many cores\n- **RAM**: Fast memory (DDR4-3200+) for index traversal\n- **Network**: High bandwidth for concurrent client connections\n\n#### **For Large Scale Deployments:**\n- **Distributed Setup**: Multiple nodes with load balancing\n- **Storage Tiering**: Hot data in memory, warm data on SSD\n- **Monitoring**: Comprehensive metrics and alerting\n\n---\n\n## \ud83d\udd27 **Configuration**\n\n### **Server Configuration**\n\n```toml\n# config.toml\n[server]\nhost = \"0.0.0.0\"\nport = 8080\ngrpc_port = 9090\nworkers = 8\n\n[storage]\ndata_dir = \"./data\"\nwal_sync_interval = \"1s\"\nmemory_map_size = \"1GB\"\n\n[index]\nhnsw_max_connections = 16\nhnsw_ef_construction = 200\nhnsw_max_layer = 16\n\n[monitoring]\nenable_metrics = true\nprometheus_port = 9091\nlog_level = \"info\"\n```\n\n### **Performance Tuning**\n\n```toml\n[performance]\n# Optimize for insertion throughput\nbatch_size = 1000\ninsert_workers = 16\n\n# Optimize for query latency \nquery_cache_size = \"500MB\"\nprefetch_enabled = true\n\n# Memory management\ngc_interval = \"30s\"\nmemory_limit = \"8GB\"\n```\n\n---\n\n## \ud83c\udf10 **API Reference**\n\n### **REST API**\n\n| Endpoint | Method | Description |\n|----------|--------|-------------|\n| `/collections` | POST | Create collection |\n| `/collections/{name}` | GET | Get collection info |\n| `/collections/{name}/vectors` | POST | Insert vectors |\n| `/collections/{name}/search` | POST | Search vectors |\n| `/collections/{name}/vectors/{id}` | DELETE | Delete vector |\n| `/stats` | GET | Server statistics |\n| `/health` | GET | Health check |\n\n### **gRPC Services**\n\n```protobuf\nservice VectorDb {\n rpc CreateCollection(CreateCollectionRequest) returns (CreateCollectionResponse);\n rpc Insert(InsertRequest) returns (InsertResponse);\n rpc BatchInsert(BatchInsertRequest) returns (BatchInsertResponse);\n rpc Query(QueryRequest) returns (QueryResponse);\n rpc Delete(DeleteRequest) returns (DeleteResponse);\n rpc GetStats(GetStatsRequest) returns (GetStatsResponse);\n}\n```\n\n### **Client SDKs**\n\n```rust\n// Rust Client\nuse vectordb_client::VectorDbClient;\n\nlet client = VectorDbClient::new(\"http://localhost:8080\").await?;\n\n// Create collection\nclient.create_collection(\"documents\", 128, DistanceMetric::Cosine).await?;\n\n// Insert vector\nclient.insert(\"documents\", \"doc1\", vec![0.1, 0.2, 0.3], metadata).await?;\n\n// Search\nlet results = client.search(\"documents\", query_vector, 10).await?;\n```\n\n```python\n# Python Client (Coming Soon)\nimport vectordb\n\nclient = vectordb.Client(\"http://localhost:8080\")\nclient.create_collection(\"documents\", 128, \"cosine\")\nclient.insert(\"documents\", \"doc1\", [0.1, 0.2, 0.3], {\"title\": \"Example\"})\nresults = client.search(\"documents\", query_vector, limit=10)\n```\n\n---\n\n## \ud83d\udd0d **Use Cases**\n\n### **\ud83e\udd16 AI & Machine Learning**\n- **Embedding storage** for transformer models (BERT, GPT, etc.)\n- **Recommendation engines** with user/item similarity\n- **Content-based filtering** and personalization\n\n### **\ud83d\udd0d Search & Discovery** \n- **Semantic search** in documents and knowledge bases\n- **Image/video similarity** search and retrieval\n- **Product recommendation** in e-commerce platforms\n\n### **\ud83d\udcca Data Analytics**\n- **Anomaly detection** in high-dimensional data\n- **Clustering and classification** of complex datasets \n- **Feature matching** in computer vision applications\n\n### **\ud83c\udfe2 Enterprise Applications**\n- **Document similarity** in legal and compliance systems\n- **Fraud detection** through pattern matching\n- **Customer segmentation** and behavioral analysis\n\n---\n\n## \ud83d\udea6 **Production Deployment**\n\n### **Docker Deployment**\n\n```dockerfile\nFROM rust:1.70 as builder\nCOPY . .\nRUN cargo build --release\n\nFROM debian:bookworm-slim\nCOPY --from=builder /target/release/vectordb-server /usr/local/bin/\nEXPOSE 8080 9090 9091\nCMD [\"vectordb-server\", \"--config\", \"/etc/vectordb/config.toml\"]\n```\n\n```bash\n# Build and run\ndocker build -t d-vecdb .\ndocker run -p 8080:8080 -p 9090:9090 -v ./data:/data d-vecdb\n```\n\n### **Kubernetes Deployment**\n\n```yaml\napiVersion: apps/v1\nkind: Deployment\nmetadata:\n name: d-vecdb\nspec:\n replicas: 3\n selector:\n matchLabels:\n app: d-vecdb\n template:\n metadata:\n labels:\n app: d-vecdb\n spec:\n containers:\n - name: d-vecdb\n image: d-vecdb:latest\n ports:\n - containerPort: 8080\n - containerPort: 9090\n resources:\n requests:\n memory: \"2Gi\"\n cpu: \"500m\"\n limits:\n memory: \"8Gi\"\n cpu: \"4\"\n```\n\n### **Monitoring Integration**\n\n```yaml\n# Prometheus configuration\n- job_name: 'd-vecdb'\n static_configs:\n - targets: ['d-vecdb:9091']\n scrape_interval: 15s\n metrics_path: /metrics\n```\n\n---\n\n## \ud83d\udcc8 **Performance Comparison**\n\n### **vs. Traditional Vector Databases**\n\n| Feature | d-vecDB | Pinecone | Weaviate | Qdrant |\n|---------|-------------|----------|----------|--------|\n| **Language** | Rust | Python/C++ | Go | Rust |\n| **Memory Safety** | \u2705 Zero-cost | \u274c Manual | \u274c GC Overhead | \u2705 Zero-cost |\n| **Concurrency** | \u2705 Native | \u26a0\ufe0f Limited | \u26a0\ufe0f GC Pauses | \u2705 Native |\n| **Deployment** | \u2705 Single Binary | \u274c Cloud Only | \u26a0\ufe0f Complex | \u2705 Flexible |\n| **Performance** | \u2705 35M ops/sec | \u26a0\ufe0f Network Bound | \u26a0\ufe0f GC Limited | \u2705 Comparable |\n\n### **Scaling Characteristics**\n\n| Dataset Size | Query Latency | Memory Usage | Throughput |\n|-------------|---------------|--------------|------------|\n| **1K vectors** | <100\u00b5s | <10MB | 50K+ qps |\n| **100K vectors** | <500\u00b5s | <500MB | 25K+ qps |\n| **1M vectors** | <2ms | <2GB | 15K+ qps |\n| **10M vectors** | <10ms | <8GB | 8K+ qps |\n\n---\n\n## \ud83e\udd1d **Contributing**\n\nWe welcome contributions! Please see our [Contributing Guide](CONTRIBUTING.md) for details.\n\n### **Development Workflow**\n\n```bash\n# Fork and clone the repository\ngit clone https://github.com/your-username/d-vecDB.git\ncd d-vecDB\n\n# Create a feature branch\ngit checkout -b feature/amazing-feature\n\n# Make changes and test\ncargo test\ncargo clippy\ncargo fmt\n\n# Submit a pull request\ngit push origin feature/amazing-feature\n```\n\n### **Areas for Contribution**\n- \ud83d\ude80 Performance optimizations and SIMD implementations\n- \ud83c\udf10 Additional client SDK languages (Python, JavaScript, Java)\n- \ud83d\udcca Advanced indexing algorithms (IVF, PQ, LSH)\n- \ud83d\udd27 Operational tools and monitoring dashboards\n- \ud83d\udcda Documentation and example applications\n\n---\n\n## \ud83d\udcc4 **License**\n\nThis project is licensed under the d-vecDB Enterprise License - see the [LICENSE](LICENSE) file for details.\n\n**For Enterprise Use**: Commercial usage requires a separate enterprise license. Contact durai@infinidatum.com for licensing terms.\n\n---\n\n## \ud83c\udd98 **Support**\n\n- **\ud83d\udce7 Email**: durai@infinidatum.com\n- **\ud83d\udcac Discord**: [d-vecDB Community](https://discord.gg/d-vecdb)\n- **\ud83d\udc1b Issues**: [GitHub Issues](https://github.com/rdmurugan/d-vecDB/issues)\n- **\ud83d\udcda Documentation**: [docs.d-vecdb.com](https://docs.d-vecdb.com)\n\n---\n\n## \ud83d\ude4f **Acknowledgments**\n\n- Built with \u2764\ufe0f in Rust\n- Inspired by modern vector database architectures\n- Powered by the amazing Rust ecosystem\n- Community-driven development\n\n---\n\n**\u26a1 Ready to build the future of AI-powered applications? Get started with d-vecDB today!**\n",
"bugtrack_url": null,
"license": null,
"summary": "High-performance vector database written in Rust with Python client",
"version": "0.1.1",
"project_urls": {
"Bug Reports": "https://github.com/rdmurugan/d-vecDB/issues",
"Documentation": "https://github.com/rdmurugan/d-vecDB#readme",
"Homepage": "https://github.com/rdmurugan/d-vecDB",
"Repository": "https://github.com/rdmurugan/d-vecDB"
},
"split_keywords": [
"vector database",
" similarity search",
" machine learning",
" embeddings",
" hnsw",
" rust"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "f4b7954cd9a7c0bf15610c09f1567d8d73832d0f9f277cad56180c7f0f991187",
"md5": "35f041cc9a928a5fdf691437045f3a7a",
"sha256": "7e4d9f18da584718667e1febe3e15d2532b35deddf8161c3f7f39f688c06126e"
},
"downloads": -1,
"filename": "d_vecdb-0.1.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "35f041cc9a928a5fdf691437045f3a7a",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 68216,
"upload_time": "2025-09-02T17:19:21",
"upload_time_iso_8601": "2025-09-02T17:19:21.640967Z",
"url": "https://files.pythonhosted.org/packages/f4/b7/954cd9a7c0bf15610c09f1567d8d73832d0f9f277cad56180c7f0f991187/d_vecdb-0.1.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "be315985bec279e9eff37acc455622582798d3f408ede7bd388f4a5761d52c33",
"md5": "b6db34bf855c5dd90eafbb6ed081a7e1",
"sha256": "a02545f056ae3c1ebc80f183b57e6a5c53324a4b8eec80975c7cf426437a6508"
},
"downloads": -1,
"filename": "d_vecdb-0.1.1.tar.gz",
"has_sig": false,
"md5_digest": "b6db34bf855c5dd90eafbb6ed081a7e1",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 67049,
"upload_time": "2025-09-02T17:19:22",
"upload_time_iso_8601": "2025-09-02T17:19:22.876347Z",
"url": "https://files.pythonhosted.org/packages/be/31/5985bec279e9eff37acc455622582798d3f408ede7bd388f4a5761d52c33/d_vecdb-0.1.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-09-02 17:19:22",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "rdmurugan",
"github_project": "d-vecDB",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "d-vecdb"
}
JSON
