# Snowflake Data Exchange Agent
[](http://www.apache.org/licenses/LICENSE-2.0.txt)
[](https://www.python.org/downloads/)
A REST API service for database migrations and data validation. Supports multiple databases including Snowflake, PostgreSQL, and SQL Server with queue-based task processing.
## Quick Start
```bash
# Install
pip install snowflake-data-exchange-agent
# Run
data-exchange-agent --port 8080
# Test
curl http://localhost:8080/health
```
## Installation
### From PyPI (Production)
```bash
pip install snowflake-data-exchange-agent
```
### Requirements & Dependencies
**Python Version**: 3.10, 3.11, or 3.12 (3.13 not yet supported)
**Available dependency groups**:
- `development`: Testing and development tools (pytest, ruff, etc.)
- `all`: Includes all development dependencies
**Core dependencies include**:
- Snowflake Connector for Python
- PySpark for data processing
- Flask + Waitress for REST API
- PostgreSQL support (psycopg2-binary)
- AWS SDK (boto3)
## Configuration
Create `src/data_exchange_agent/configuration.toml`:
```toml
# API settings
[api_configuration]
key = "your-api-key"
host = "0.0.0.0"
port = 5001
workers = 4
# Database connections
[connection.postgresql]
driver_name = "postgresql"
username = "user"
password = "password"
host = "localhost"
port = 5432
database = "mydb"
# Task queue
[task_queue]
type = "sqlite"
database_path = "~/.data_exchange_agent/tasks.db"
```
For Snowflake, create `~/.snowflake/config.toml`:
```toml
[connections.default]
account = "your_account.region"
user = "your_username"
password = "your_password"
warehouse = "COMPUTE_WH"
database = "PRODUCTION_DB"
```
## API Usage
### Command Line
```bash
# Basic usage
data-exchange-agent
# Production settings
data-exchange-agent --workers 8 --port 8080
# Debug mode
data-exchange-agent --debug --port 5001
```
### Health Check
```http
GET /health
```
```json
{
"status": "healthy",
"version": "0.0.18",
"database_connections": {
"snowflake": "connected"
}
}
```
### Task Management
```http
# Start processing
GET /handle_tasks
# Stop processing
GET /stop
# Get status
GET /get_handling_tasks_status
# Task count
GET /get_tasks_count
```
### Add Task
```http
POST /tasks
Content-Type: application/json
```
```json
{
"task_type": "data_extraction",
"source_config": {
"database": "postgresql",
"query": "SELECT * FROM users"
},
"destination_config": {
"type": "snowflake_stage",
"stage": "@data_stage/users/"
}
}
```
## Development
### Setup
```bash
git clone https://github.com/snowflakedb/migrations-data-validation.git
cd migrations-data-validation/data-exchange-agent
pip install -e .[development]
```
### Testing
```bash
# Run all tests
pytest
# With coverage
pytest --cov=src/data_exchange_agent
# Run specific test types
pytest tests/unit/ # Unit tests only
pytest -m "not integration" # Non-integration tests
```
### Code Quality
```bash
# Format code
ruff format .
# Lint code
ruff check .
# Auto-fix linting issues
ruff check --fix .
```
## 🤝 Contributing
We welcome contributions! See our [Contributing Guide](../CONTRIBUTING.md) for details on how to collaborate, set up your development environment, and submit PRs.
---
## 📄 License
This project is licensed under the Apache License 2.0. See the [LICENSE](../LICENSE) file for details.
## 🆘 Support
- **Documentation**: [Full documentation](https://github.com/snowflakedb/migrations-data-validation)
- **Issues**: [GitHub Issues](https://github.com/snowflakedb/migrations-data-validation/issues)
---
**Developed with ❄️ by Snowflake**
Raw data
{
"_id": null,
"home_page": null,
"name": "snowflake-data-exchange-agent",
"maintainer": null,
"docs_url": null,
"requires_python": "<3.13,>=3.10",
"maintainer_email": null,
"keywords": "Snowflake, analytics, cloud, data, data-analysis, data-analytics, data-engineering, data-management, data-processing, data-science, data-visualization, data-warehouse, database",
"author": null,
"author_email": "\"Snowflake, Inc.\" <snowflake-python-libraries-dl@snowflake.com>",
"download_url": "https://files.pythonhosted.org/packages/71/bd/42d8bd995eeecb5f73b2ed4f3c48e2e831167169b1476cc87ba155d7a320/snowflake_data_exchange_agent-0.0.19.tar.gz",
"platform": null,
"description": "# Snowflake Data Exchange Agent\n\n[](http://www.apache.org/licenses/LICENSE-2.0.txt)\n[](https://www.python.org/downloads/)\n\nA REST API service for database migrations and data validation. Supports multiple databases including Snowflake, PostgreSQL, and SQL Server with queue-based task processing.\n\n## Quick Start\n\n```bash\n# Install\npip install snowflake-data-exchange-agent\n\n# Run\ndata-exchange-agent --port 8080\n\n# Test\ncurl http://localhost:8080/health\n```\n\n## Installation\n\n### From PyPI (Production)\n```bash\npip install snowflake-data-exchange-agent\n```\n\n### Requirements & Dependencies\n\n**Python Version**: 3.10, 3.11, or 3.12 (3.13 not yet supported)\n\n**Available dependency groups**:\n- `development`: Testing and development tools (pytest, ruff, etc.)\n- `all`: Includes all development dependencies\n\n**Core dependencies include**:\n- Snowflake Connector for Python\n- PySpark for data processing\n- Flask + Waitress for REST API\n- PostgreSQL support (psycopg2-binary)\n- AWS SDK (boto3)\n\n## Configuration\n\nCreate `src/data_exchange_agent/configuration.toml`:\n\n```toml\n# API settings\n[api_configuration]\nkey = \"your-api-key\"\nhost = \"0.0.0.0\"\nport = 5001\nworkers = 4\n\n# Database connections\n[connection.postgresql]\ndriver_name = \"postgresql\"\nusername = \"user\"\npassword = \"password\"\nhost = \"localhost\"\nport = 5432\ndatabase = \"mydb\"\n\n# Task queue\n[task_queue]\ntype = \"sqlite\"\ndatabase_path = \"~/.data_exchange_agent/tasks.db\"\n```\n\nFor Snowflake, create `~/.snowflake/config.toml`:\n\n```toml\n[connections.default]\naccount = \"your_account.region\"\nuser = \"your_username\"\npassword = \"your_password\"\nwarehouse = \"COMPUTE_WH\"\ndatabase = \"PRODUCTION_DB\"\n```\n\n## API Usage\n\n### Command Line\n```bash\n# Basic usage\ndata-exchange-agent\n\n# Production settings\ndata-exchange-agent --workers 8 --port 8080\n\n# Debug mode\ndata-exchange-agent --debug --port 5001\n```\n\n### Health Check\n```http\nGET /health\n```\n```json\n{\n \"status\": \"healthy\",\n \"version\": \"0.0.18\",\n \"database_connections\": {\n \"snowflake\": \"connected\"\n }\n}\n```\n\n### Task Management\n```http\n# Start processing\nGET /handle_tasks\n\n# Stop processing\nGET /stop\n\n# Get status\nGET /get_handling_tasks_status\n\n# Task count\nGET /get_tasks_count\n```\n\n### Add Task\n```http\nPOST /tasks\nContent-Type: application/json\n```\n```json\n{\n \"task_type\": \"data_extraction\",\n \"source_config\": {\n \"database\": \"postgresql\",\n \"query\": \"SELECT * FROM users\"\n },\n \"destination_config\": {\n \"type\": \"snowflake_stage\",\n \"stage\": \"@data_stage/users/\"\n }\n}\n```\n\n## Development\n\n### Setup\n```bash\ngit clone https://github.com/snowflakedb/migrations-data-validation.git\ncd migrations-data-validation/data-exchange-agent\npip install -e .[development]\n```\n\n### Testing\n```bash\n# Run all tests\npytest\n\n# With coverage\npytest --cov=src/data_exchange_agent\n\n# Run specific test types\npytest tests/unit/ # Unit tests only\npytest -m \"not integration\" # Non-integration tests\n```\n\n### Code Quality\n```bash\n# Format code\nruff format .\n\n# Lint code\nruff check .\n\n# Auto-fix linting issues\nruff check --fix .\n```\n\n## \ud83e\udd1d Contributing\n\nWe welcome contributions! See our [Contributing Guide](../CONTRIBUTING.md) for details on how to collaborate, set up your development environment, and submit PRs.\n\n---\n\n## \ud83d\udcc4 License\n\nThis project is licensed under the Apache License 2.0. See the [LICENSE](../LICENSE) file for details.\n\n## \ud83c\udd98 Support\n\n- **Documentation**: [Full documentation](https://github.com/snowflakedb/migrations-data-validation)\n- **Issues**: [GitHub Issues](https://github.com/snowflakedb/migrations-data-validation/issues)\n\n---\n\n**Developed with \u2744\ufe0f by Snowflake**\n",
"bugtrack_url": null,
"license": "Apache License, Version 2.0",
"summary": "Data exchange agent for migrations and validation",
"version": "0.0.19",
"project_urls": {
"Bug Tracker": "https://github.com/snowflakedb/migrations-data-validation/issues",
"Source code": "https://github.com/snowflakedb/migrations-data-validation/",
"homepage": "https://www.snowflake.com/"
},
"split_keywords": [
"snowflake",
" analytics",
" cloud",
" data",
" data-analysis",
" data-analytics",
" data-engineering",
" data-management",
" data-processing",
" data-science",
" data-visualization",
" data-warehouse",
" database"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "ddc38a3425b4d661fae81375a214db3eceab5a8adca19119dd236300f3bf6fb5",
"md5": "0a2903bab52613b4697b4c3eb1d5ebf0",
"sha256": "dae3f6e92cbfdbeb2418587082b11ca6bbc76fcbcdc4b4045f96637c6faccb7d"
},
"downloads": -1,
"filename": "snowflake_data_exchange_agent-0.0.19-py3-none-any.whl",
"has_sig": false,
"md5_digest": "0a2903bab52613b4697b4c3eb1d5ebf0",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<3.13,>=3.10",
"size": 55380,
"upload_time": "2025-10-14T21:36:28",
"upload_time_iso_8601": "2025-10-14T21:36:28.974527Z",
"url": "https://files.pythonhosted.org/packages/dd/c3/8a3425b4d661fae81375a214db3eceab5a8adca19119dd236300f3bf6fb5/snowflake_data_exchange_agent-0.0.19-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "71bd42d8bd995eeecb5f73b2ed4f3c48e2e831167169b1476cc87ba155d7a320",
"md5": "7e9a5fbfd299883bca6445c3c58d2520",
"sha256": "2419baa37f64e3af3e0738bfb70a800be7ce6f1ce80c9aa77e303314f574b1c4"
},
"downloads": -1,
"filename": "snowflake_data_exchange_agent-0.0.19.tar.gz",
"has_sig": false,
"md5_digest": "7e9a5fbfd299883bca6445c3c58d2520",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<3.13,>=3.10",
"size": 83863,
"upload_time": "2025-10-14T21:36:30",
"upload_time_iso_8601": "2025-10-14T21:36:30.113738Z",
"url": "https://files.pythonhosted.org/packages/71/bd/42d8bd995eeecb5f73b2ed4f3c48e2e831167169b1476cc87ba155d7a320/snowflake_data_exchange_agent-0.0.19.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-10-14 21:36:30",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "snowflakedb",
"github_project": "migrations-data-validation",
"github_not_found": true,
"lcname": "snowflake-data-exchange-agent"
}
JSON
