# 🔍 ctx-miner
🚀 A Python library for context-aware conversational AI using graph databases. Built on top of [Graphiti Core](https://github.com/getzep/graphiti), ctx-miner provides an easy-to-use interface for storing, retrieving, and analyzing conversational context using graph-based knowledge representation.
## ✨ Features
- 🕸️ **Graph-based Context Storage**: Store conversations as interconnected knowledge graphs
- 🔎 **Semantic Search**: Find relevant context using hybrid search (semantic + BM25)
- 🤖 **Multiple LLM Support**: Configurable LLM providers (OpenAI, etc.)
- 🧮 **Flexible Embedding Models**: Support for various embedding providers
- 🗄️ **FalkorDB Integration**: Redis-based graph database backend
- ⚡ **Async/Await Support**: Built for high-performance async applications
- 🔒 **Type Safety**: Full type hints and Pydantic models
## 📦 Installation
### 🔧 Using uv (Recommended)
```bash
uv add ctx-miner
```
### 🐍 Using pip
```bash
pip install ctx-miner
```
### 💻 Development Installation
```bash
git clone https://github.com/hienhayho/ctx-miner.git
cd ctx-miner
uv sync
```
## 🚀 Quick Start
### 1. 🔐 Environment Setup
Run `falkor-db` with docker:
```bash
docker run \
-it -d \
--restart always \
-p 6379:6379 \
-p 3000:3000 \
--name falkordb \
falkordb/falkordb:latest
```
Create a `.env` file with your configuration:
```env
OPENAI_API_KEY=your_openai_api_key
FALKORDB_HOST=localhost
FALKORDB_PORT=6379
FALKORDB_USERNAME=
FALKORDB_PASSWORD=
```
### 2. 💡 Quick Usage
```python
import asyncio
from ctx_miner import CtxMiner
from ctx_miner.core.schemas import CtxMinerEpisode, CtxMinerMessage
from ctx_miner.utils.helpers import load_config
async def main():
# Configure the library
config = load_config(group_id="demo_conversation", auto_build_indices=True)
# Initialize CtxMiner
miner = CtxMiner(config=config)
try:
await miner.initialize()
# Create a conversation episode
episode = CtxMinerEpisode(
messages=[
CtxMinerMessage(role="user", content="Hello! I need help with my internet plan."),
CtxMinerMessage(role="assistant", content="Hi! I'd be happy to help you with your internet plan. What specific questions do you have?"),
CtxMinerMessage(role="user", content="What's the fastest speed you offer?"),
CtxMinerMessage(role="assistant", content="Our fastest plan is Super200 with 200 Mbps download speed.")
]
)
# Add episode to the knowledge graph
episode_uuid = await miner.add_episode(episode)
print(f"Added episode: {episode_uuid}")
# Search for relevant context
results = await miner.search_context(
query="internet speed plans",
limit=5
)
print(f"Found {len(results)} relevant contexts:")
for result in results:
print(f"- {result['fact']}")
# Get statistics
stats = await miner.get_stats()
print(f"Database contains {stats['episode_count']} episodes")
finally:
await miner.close()
if __name__ == "__main__":
asyncio.run(main())
```
## ⚙️ Configuration
### 🎛️ CtxMinerConfig
The main configuration object that combines all settings:
```python
from ctx_miner.core.schemas import CtxMinerConfig, FalkorDBConfig, CtxMinerLLMConfig, EmbeddingConfig
config = CtxMinerConfig(
falkordb_config=FalkorDBConfig(
host="localhost",
port=6379,
database="my_app",
username="", # Optional
password="" # Optional
),
llm_config=CtxMinerLLMConfig(
provider="openai",
model="gpt-4o-mini",
temperature=0.0,
max_tokens=8192
),
embedding_config=EmbeddingConfig(
provider="openai",
model="text-embedding-3-small",
dimensions=1536
),
group_id="default", # Logical grouping for conversations
auto_build_indices=True # Automatically create database indices
)
```
### 🌍 Environment Variables
| Variable | Description | Default |
|----------|-------------|---------|
| `OPENAI_API_KEY` | OpenAI API key | Required |
| `FALKORDB_HOST` | FalkorDB host | localhost |
| `FALKORDB_PORT` | FalkorDB port | 6379 |
| `FALKORDB_USERNAME` | FalkorDB username | (empty) |
| `FALKORDB_PASSWORD` | FalkorDB password | (empty) |
## 📚 Core Concepts
### 📝 Episodes
Episodes are the fundamental units of conversation stored in the graph. Each episode contains a sequence of messages and gets processed to extract entities and relationships.
```python
from ctx_miner.core.schemas import CtxMinerEpisode, CtxMinerMessage
episode = CtxMinerEpisode(
messages=[
CtxMinerMessage(role="user", content="What are your business hours?"),
CtxMinerMessage(role="assistant", content="We're open Monday-Friday, 9 AM to 6 PM EST.")
]
)
```
### 🔍 Search and Retrieval
ctx-miner provides multiple search methods:
1. **Context Search**: Find relevant facts and relationships
2. **Node Search**: Search for specific entities
3. **Graph-based Reranking**: Use graph distance for improved relevance
```python
# Basic context search
results = await miner.search_context("business hours", limit=10)
# Search with center node reranking
results = await miner.search_context(
query="customer support",
center_node_uuid="some-node-uuid",
limit=5
)
# Direct node search
nodes = await miner.search_nodes("customer", limit=5)
```
## 🎯 Advanced Usage
### 📦 Batch Processing
```python
# Add multiple episodes efficiently
episodes = [
CtxMinerEpisode(messages=[...]),
CtxMinerEpisode(messages=[...]),
# ... more episodes
]
uuids = await miner.add_episodes(episodes)
print(f"Added {len(uuids)} episodes")
```
### 🗂️ Episode Management
```python
# List episodes
episodes = await miner.list_episodes(limit=50, offset=0)
# Get specific episode
episode_data = await miner.get_episode(episode_uuid)
# Delete episode
success = await miner.delete_episode(episode_uuid)
# Clear all episodes in group
await miner.clear_all()
```
### 🔧 Custom Search Configuration
```python
from graphiti_core.search.search_config_recipes import NODE_HYBRID_SEARCH_RRF
# Use custom search configuration
custom_config = NODE_HYBRID_SEARCH_RRF.model_copy(deep=True)
custom_config.limit = 20
results = await miner.search_nodes(
query="customer preferences",
search_config=custom_config
)
```
## 📋 Requirements
- 🐍 Python 3.8+
- 🗄️ FalkorDB instance (Redis with graph capabilities)
- 🔑 OpenAI API key (or other supported LLM provider)
## 📚 Dependencies
- 🕸️ `graphiti-core[falkordb]`: Graph-based knowledge management
- 📝 `loguru`: Structured logging
- ✅ `pydantic`: Data validation and serialization
- 📊 `tqdm`: Progress bars for batch operations
- 🔐 `python-dotenv`: Environment variable management
## 📄 License
[Add your license information here]
## 🤝 Contributing
[Add contribution guidelines here]
## 💬 Support
For issues and questions:
- 📁 Check the [examples](./examples/) directory
- 📖 Review the [documentation](./docs/)
- 🐛 Open an issue on GitHub
Raw data
{
"_id": null,
"home_page": null,
"name": "ctx-miner",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.11",
"maintainer_email": null,
"keywords": "context-mining, conversational-ai, falkordb, graph-database, graphiti",
"author": null,
"author_email": "hienhayho <hienhayho3002@gmail.com>",
"download_url": "https://files.pythonhosted.org/packages/1a/67/7ee5e793abf15e00ad77e96ac6f11ef8760048e7cebc7deacca8ac8ee2b4/ctx_miner-0.1.0.tar.gz",
"platform": null,
"description": "# \ud83d\udd0d ctx-miner\n\n\ud83d\ude80 A Python library for context-aware conversational AI using graph databases. Built on top of [Graphiti Core](https://github.com/getzep/graphiti), ctx-miner provides an easy-to-use interface for storing, retrieving, and analyzing conversational context using graph-based knowledge representation.\n\n## \u2728 Features\n\n- \ud83d\udd78\ufe0f **Graph-based Context Storage**: Store conversations as interconnected knowledge graphs\n- \ud83d\udd0e **Semantic Search**: Find relevant context using hybrid search (semantic + BM25)\n- \ud83e\udd16 **Multiple LLM Support**: Configurable LLM providers (OpenAI, etc.)\n- \ud83e\uddee **Flexible Embedding Models**: Support for various embedding providers\n- \ud83d\uddc4\ufe0f **FalkorDB Integration**: Redis-based graph database backend\n- \u26a1 **Async/Await Support**: Built for high-performance async applications\n- \ud83d\udd12 **Type Safety**: Full type hints and Pydantic models\n\n## \ud83d\udce6 Installation\n\n### \ud83d\udd27 Using uv (Recommended)\n\n```bash\nuv add ctx-miner\n```\n\n### \ud83d\udc0d Using pip\n\n```bash\npip install ctx-miner\n```\n\n### \ud83d\udcbb Development Installation\n\n```bash\ngit clone https://github.com/hienhayho/ctx-miner.git\ncd ctx-miner\nuv sync\n```\n\n## \ud83d\ude80 Quick Start\n\n### 1. \ud83d\udd10 Environment Setup\n\nRun `falkor-db` with docker:\n\n```bash\ndocker run \\\n -it -d \\\n --restart always \\\n -p 6379:6379 \\\n -p 3000:3000 \\\n --name falkordb \\\n falkordb/falkordb:latest\n```\n\nCreate a `.env` file with your configuration:\n\n```env\nOPENAI_API_KEY=your_openai_api_key\nFALKORDB_HOST=localhost\nFALKORDB_PORT=6379\nFALKORDB_USERNAME=\nFALKORDB_PASSWORD=\n```\n\n### 2. \ud83d\udca1 Quick Usage\n\n```python\nimport asyncio\nfrom ctx_miner import CtxMiner\nfrom ctx_miner.core.schemas import CtxMinerEpisode, CtxMinerMessage\nfrom ctx_miner.utils.helpers import load_config\n\nasync def main():\n # Configure the library\n config = load_config(group_id=\"demo_conversation\", auto_build_indices=True)\n \n # Initialize CtxMiner\n miner = CtxMiner(config=config)\n \n try:\n await miner.initialize()\n # Create a conversation episode\n episode = CtxMinerEpisode(\n messages=[\n CtxMinerMessage(role=\"user\", content=\"Hello! I need help with my internet plan.\"),\n CtxMinerMessage(role=\"assistant\", content=\"Hi! I'd be happy to help you with your internet plan. What specific questions do you have?\"),\n CtxMinerMessage(role=\"user\", content=\"What's the fastest speed you offer?\"),\n CtxMinerMessage(role=\"assistant\", content=\"Our fastest plan is Super200 with 200 Mbps download speed.\")\n ]\n )\n \n # Add episode to the knowledge graph\n episode_uuid = await miner.add_episode(episode)\n print(f\"Added episode: {episode_uuid}\")\n \n # Search for relevant context\n results = await miner.search_context(\n query=\"internet speed plans\",\n limit=5\n )\n \n print(f\"Found {len(results)} relevant contexts:\")\n for result in results:\n print(f\"- {result['fact']}\")\n \n # Get statistics\n stats = await miner.get_stats()\n print(f\"Database contains {stats['episode_count']} episodes\")\n \n finally:\n await miner.close()\n\nif __name__ == \"__main__\":\n asyncio.run(main())\n```\n\n## \u2699\ufe0f Configuration\n\n### \ud83c\udf9b\ufe0f CtxMinerConfig\n\nThe main configuration object that combines all settings:\n\n```python\nfrom ctx_miner.core.schemas import CtxMinerConfig, FalkorDBConfig, CtxMinerLLMConfig, EmbeddingConfig\n\nconfig = CtxMinerConfig(\n falkordb_config=FalkorDBConfig(\n host=\"localhost\",\n port=6379,\n database=\"my_app\",\n username=\"\", # Optional\n password=\"\" # Optional\n ),\n llm_config=CtxMinerLLMConfig(\n provider=\"openai\",\n model=\"gpt-4o-mini\",\n temperature=0.0,\n max_tokens=8192\n ),\n embedding_config=EmbeddingConfig(\n provider=\"openai\",\n model=\"text-embedding-3-small\",\n dimensions=1536\n ),\n group_id=\"default\", # Logical grouping for conversations\n auto_build_indices=True # Automatically create database indices\n)\n```\n\n### \ud83c\udf0d Environment Variables\n\n| Variable | Description | Default |\n|----------|-------------|---------|\n| `OPENAI_API_KEY` | OpenAI API key | Required |\n| `FALKORDB_HOST` | FalkorDB host | localhost |\n| `FALKORDB_PORT` | FalkorDB port | 6379 |\n| `FALKORDB_USERNAME` | FalkorDB username | (empty) |\n| `FALKORDB_PASSWORD` | FalkorDB password | (empty) |\n\n## \ud83d\udcda Core Concepts\n\n### \ud83d\udcdd Episodes\n\nEpisodes are the fundamental units of conversation stored in the graph. Each episode contains a sequence of messages and gets processed to extract entities and relationships.\n\n```python\nfrom ctx_miner.core.schemas import CtxMinerEpisode, CtxMinerMessage\n\nepisode = CtxMinerEpisode(\n messages=[\n CtxMinerMessage(role=\"user\", content=\"What are your business hours?\"),\n CtxMinerMessage(role=\"assistant\", content=\"We're open Monday-Friday, 9 AM to 6 PM EST.\")\n ]\n)\n```\n\n### \ud83d\udd0d Search and Retrieval\n\nctx-miner provides multiple search methods:\n\n1. **Context Search**: Find relevant facts and relationships\n2. **Node Search**: Search for specific entities\n3. **Graph-based Reranking**: Use graph distance for improved relevance\n\n```python\n# Basic context search\nresults = await miner.search_context(\"business hours\", limit=10)\n\n# Search with center node reranking\nresults = await miner.search_context(\n query=\"customer support\",\n center_node_uuid=\"some-node-uuid\",\n limit=5\n)\n\n# Direct node search\nnodes = await miner.search_nodes(\"customer\", limit=5)\n```\n\n## \ud83c\udfaf Advanced Usage\n\n### \ud83d\udce6 Batch Processing\n\n```python\n# Add multiple episodes efficiently\nepisodes = [\n CtxMinerEpisode(messages=[...]),\n CtxMinerEpisode(messages=[...]),\n # ... more episodes\n]\n\nuuids = await miner.add_episodes(episodes)\nprint(f\"Added {len(uuids)} episodes\")\n```\n\n### \ud83d\uddc2\ufe0f Episode Management\n\n```python\n# List episodes\nepisodes = await miner.list_episodes(limit=50, offset=0)\n\n# Get specific episode\nepisode_data = await miner.get_episode(episode_uuid)\n\n# Delete episode\nsuccess = await miner.delete_episode(episode_uuid)\n\n# Clear all episodes in group\nawait miner.clear_all()\n```\n\n### \ud83d\udd27 Custom Search Configuration\n\n```python\nfrom graphiti_core.search.search_config_recipes import NODE_HYBRID_SEARCH_RRF\n\n# Use custom search configuration\ncustom_config = NODE_HYBRID_SEARCH_RRF.model_copy(deep=True)\ncustom_config.limit = 20\n\nresults = await miner.search_nodes(\n query=\"customer preferences\",\n search_config=custom_config\n)\n```\n\n## \ud83d\udccb Requirements\n\n- \ud83d\udc0d Python 3.8+\n- \ud83d\uddc4\ufe0f FalkorDB instance (Redis with graph capabilities)\n- \ud83d\udd11 OpenAI API key (or other supported LLM provider)\n\n## \ud83d\udcda Dependencies\n\n- \ud83d\udd78\ufe0f `graphiti-core[falkordb]`: Graph-based knowledge management\n- \ud83d\udcdd `loguru`: Structured logging\n- \u2705 `pydantic`: Data validation and serialization\n- \ud83d\udcca `tqdm`: Progress bars for batch operations\n- \ud83d\udd10 `python-dotenv`: Environment variable management\n\n## \ud83d\udcc4 License\n\n[Add your license information here]\n\n## \ud83e\udd1d Contributing\n\n[Add contribution guidelines here]\n\n## \ud83d\udcac Support\n\nFor issues and questions:\n\n- \ud83d\udcc1 Check the [examples](./examples/) directory\n- \ud83d\udcd6 Review the [documentation](./docs/)\n- \ud83d\udc1b Open an issue on GitHub\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "A library for context-aware conversational AI using graph databases",
"version": "0.1.0",
"project_urls": null,
"split_keywords": [
"context-mining",
" conversational-ai",
" falkordb",
" graph-database",
" graphiti"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "1ba70632482ed9ebf7f19645d96e98d28604b6b01f43ba96081773daec5a3de9",
"md5": "8ee1b978d03a72f4b6ecd28ca5f87a51",
"sha256": "b31a4feabe1aa9035ea2a89fe650f7c2c5330b64152fa5fd3bd3f246793d7ff6"
},
"downloads": -1,
"filename": "ctx_miner-0.1.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "8ee1b978d03a72f4b6ecd28ca5f87a51",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.11",
"size": 15793,
"upload_time": "2025-08-02T17:36:24",
"upload_time_iso_8601": "2025-08-02T17:36:24.288407Z",
"url": "https://files.pythonhosted.org/packages/1b/a7/0632482ed9ebf7f19645d96e98d28604b6b01f43ba96081773daec5a3de9/ctx_miner-0.1.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "1a677ee5e793abf15e00ad77e96ac6f11ef8760048e7cebc7deacca8ac8ee2b4",
"md5": "86e983cc9f19e1155f4ccd0736e6ab95",
"sha256": "656be90e17a565d0faa372d87adac8783e9000cf2572363fc2dc3a8becf41ac3"
},
"downloads": -1,
"filename": "ctx_miner-0.1.0.tar.gz",
"has_sig": false,
"md5_digest": "86e983cc9f19e1155f4ccd0736e6ab95",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.11",
"size": 58066,
"upload_time": "2025-08-02T17:36:26",
"upload_time_iso_8601": "2025-08-02T17:36:26.051886Z",
"url": "https://files.pythonhosted.org/packages/1a/67/7ee5e793abf15e00ad77e96ac6f11ef8760048e7cebc7deacca8ac8ee2b4/ctx_miner-0.1.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-08-02 17:36:26",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "ctx-miner"
}
JSON