# RethinkPort 🚢
A comprehensive tool for migrating RethinkDB databases to MySQL, designed to work with official RethinkDB dump files.
**Port your RethinkDB data to MySQL.**
## Overview
This tool was created to address the lack of reliable migration tools for moving from RethinkDB to MySQL. It handles the complete migration pipeline including:
- Schema inference and optimization
- Data type conversion (including RethinkDB-specific types)
- Index and constraint migration
- Foreign key dependency management
- Batch processing for large datasets
## Features
- ✅ **Official RethinkDB dump support** - Works with `rethinkdb dump` output
- ✅ **Smart schema inference** - Analyzes data to create optimal MySQL schemas
- ✅ **Data type conversion** - Handles RethinkDB TIME objects, JSON, arrays
- ✅ **Index migration** - Preserves indexes and primary keys from .info files
- ✅ **Dependency management** - Processes tables in correct order for foreign keys
- ✅ **Batch processing** - Efficient handling of large datasets
- ✅ **Progress tracking** - Real-time migration statistics
- ✅ **Error handling** - Comprehensive logging and error reporting
## Installation
### Prerequisites
- Python 3.9+
- MySQL 5.7+ or MariaDB 10.2+
- RethinkDB (for creating dumps)
### Install from PyPI
```bash
pip install rethinkport
```
### Install from Source
```bash
git clone https://github.com/aoamusat/rethinkport.git
cd rethinkport
pip install -e .
```
## Usage
### Step 1: Create RethinkDB Dump
First, create a dump of your RethinkDB database:
```bash
rethinkdb dump -c <host:port> -f my_database_dump.tar.gz
```
### Step 2: Extract the Dump
```bash
tar -xzf my_database_dump.tar.gz
```
This creates a directory structure like:
```
rethinkdb_dump_<date>/
└── <database_name>/
├── Table1.info
├── Table1.json
├── Table2.info
├── Table2.json
└── ...
```
### Step 3: Run the Migration
#### Basic Usage
```bash
rethinkport /path/to/rethinkdb_dump_<date>/<database_name>/
```
#### With Configuration File
```bash
rethinkport /path/to/dump/ --config config.json
```
#### Command Line Options
```bash
rethinkport --help
usage: rethinkport [-h] [--config CONFIG] [--host HOST] [--port PORT]
[--user USER] [--password PASSWORD] [--database DATABASE]
[--dry-run] [--batch-size BATCH_SIZE] [--table-order TABLE_ORDER]
dump_path
RethinkPort 🚢 - Port your RethinkDB data to MySQL
positional arguments:
dump_path Path to extracted RethinkDB dump directory
optional arguments:
-h, --help show this help message and exit
--config CONFIG Configuration file path
--host HOST MySQL host (default: localhost)
--port PORT MySQL port (default: 3306)
--user USER MySQL username (default: root)
--password PASSWORD MySQL password
--database DATABASE MySQL database name
--dry-run Show what would be migrated without executing
--batch-size BATCH_SIZE
Batch size for data insertion (default: 1000)
--table-order TABLE_ORDER
Custom table processing order (JSON file)
```
## Configuration
### Environment Variables
```bash
export MYSQL_HOST=localhost
export MYSQL_PORT=3306
export MYSQL_USER=root
export MYSQL_PASSWORD=your_password
export MYSQL_DATABASE=your_database
```
### Configuration File
Create a `config.json` file:
```json
{
"mysql": {
"host": "localhost",
"port": 3306,
"user": "root",
"password": "your_password",
"database": "your_database",
"charset": "utf8mb4"
},
"migration": {
"batch_size": 1000,
"table_order": [
"Users",
"Products",
"Orders"
]
}
}
```
## Data Type Mapping
| RethinkDB Type | MySQL Type | Notes |
|----------------|------------|-------|
| String (short) | VARCHAR(255) | Auto-sized based on data |
| String (long) | TEXT | For strings > 2000 chars |
| Number (int) | INT/BIGINT | Based on value range |
| Number (float) | DECIMAL(15,4) | Preserves precision |
| Boolean | BOOLEAN | Direct mapping |
| Object | JSON | RethinkDB objects → MySQL JSON |
| Array | JSON | RethinkDB arrays → MySQL JSON |
| TIME | DATETIME | Converts epoch_time to MySQL datetime |
| UUID | VARCHAR(255) | For primary/foreign keys |
## Examples
### Basic Migration
```bash
# Dump RethinkDB
rethinkdb dump -c localhost:28015 -f myapp_dump.tar.gz
# Extract
tar -xzf myapp_dump.tar.gz
# Migrate
rethinkport rethinkdb_dump_2024_01_15/myapp/
```
### Advanced Migration with Custom Configuration
```bash
rethinkport \
rethinkdb_dump_2024_01_15/myapp/ \
--config production_config.json \
--batch-size 5000 \
--dry-run
```
## Troubleshooting
### Common Issues
1. **Connection Errors**
```
Error: Failed to connect to MySQL
```
- Check MySQL credentials and connection
- Ensure MySQL server is running
- Verify database exists
2. **Large Dataset Timeouts**
```
Error: MySQL server has gone away
```
- Increase `max_allowed_packet` in MySQL
- Reduce `--batch-size`
- Check MySQL timeout settings
3. **Schema Conflicts**
```
Error: Table already exists
```
- Tool drops existing tables by default
- Check MySQL user permissions
- Verify database name
### Performance Tips
- Use SSD storage for better I/O performance
- Increase MySQL `innodb_buffer_pool_size`
- Adjust `--batch-size` based on available memory
- Run migration during low-traffic periods
## Contributing
1. Fork the repository
2. Create a feature branch (`git checkout -b feature/amazing-feature`)
3. Commit your changes (`git commit -m 'Add amazing feature'`)
4. Push to the branch (`git push origin feature/amazing-feature`)
5. Open a Pull Request
### Development Setup
```bash
git clone https://github.com/aoamusat/rethinkport.git
cd rethinkport
pip install -e ".[dev]"
pytest
```
## License
This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
## Acknowledgments
- Built to solve real-world RethinkDB migration challenges
- Inspired by the need for reliable database migration tools
- Thanks to the RethinkDB and MySQL communities
## Support
If you encounter issues or have questions:
1. Check the [Issues](https://github.com/aoamusat/rethinkport/issues) page
2. Create a new issue with detailed information
3. Include your RethinkDB/MySQL versions and error logs
---
**Note**: This tool was created because existing migration solutions were insufficient for complex RethinkDB to MySQL migrations. It's designed to handle real-world scenarios with large datasets, complex schemas, and foreign key relationships.
Raw data
{
"_id": null,
"home_page": "https://github.com/aoamusat/rethinkport",
"name": "rethinkport",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.9",
"maintainer_email": null,
"keywords": "rethinkdb, mysql, migration, database, port, converter",
"author": "Akeem Amusat",
"author_email": "Akeem Amusat <hello@a4m.dev>",
"download_url": "https://files.pythonhosted.org/packages/b2/bd/c5e3863432d9e771abfdfaf6deb12b1da7b8db251d39df8cdce9dc11530c/rethinkport-1.0.3.tar.gz",
"platform": null,
"description": "# RethinkPort \ud83d\udea2\n\nA comprehensive tool for migrating RethinkDB databases to MySQL, designed to work with official RethinkDB dump files.\n\n**Port your RethinkDB data to MySQL.**\n\n## Overview\n\nThis tool was created to address the lack of reliable migration tools for moving from RethinkDB to MySQL. It handles the complete migration pipeline including:\n\n- Schema inference and optimization\n- Data type conversion (including RethinkDB-specific types)\n- Index and constraint migration\n- Foreign key dependency management\n- Batch processing for large datasets\n\n## Features\n\n- \u2705 **Official RethinkDB dump support** - Works with `rethinkdb dump` output\n- \u2705 **Smart schema inference** - Analyzes data to create optimal MySQL schemas\n- \u2705 **Data type conversion** - Handles RethinkDB TIME objects, JSON, arrays\n- \u2705 **Index migration** - Preserves indexes and primary keys from .info files\n- \u2705 **Dependency management** - Processes tables in correct order for foreign keys\n- \u2705 **Batch processing** - Efficient handling of large datasets\n- \u2705 **Progress tracking** - Real-time migration statistics\n- \u2705 **Error handling** - Comprehensive logging and error reporting\n\n## Installation\n\n### Prerequisites\n\n- Python 3.9+\n- MySQL 5.7+ or MariaDB 10.2+\n- RethinkDB (for creating dumps)\n\n### Install from PyPI\n\n```bash\npip install rethinkport\n```\n\n### Install from Source\n\n```bash\ngit clone https://github.com/aoamusat/rethinkport.git\ncd rethinkport\npip install -e .\n```\n\n## Usage\n\n### Step 1: Create RethinkDB Dump\n\nFirst, create a dump of your RethinkDB database:\n\n```bash\nrethinkdb dump -c <host:port> -f my_database_dump.tar.gz\n```\n\n### Step 2: Extract the Dump\n\n```bash\ntar -xzf my_database_dump.tar.gz\n```\n\nThis creates a directory structure like:\n```\nrethinkdb_dump_<date>/\n\u2514\u2500\u2500 <database_name>/\n \u251c\u2500\u2500 Table1.info\n \u251c\u2500\u2500 Table1.json\n \u251c\u2500\u2500 Table2.info\n \u251c\u2500\u2500 Table2.json\n \u2514\u2500\u2500 ...\n```\n\n### Step 3: Run the Migration\n\n#### Basic Usage\n\n```bash\nrethinkport /path/to/rethinkdb_dump_<date>/<database_name>/\n```\n\n#### With Configuration File\n\n```bash\nrethinkport /path/to/dump/ --config config.json\n```\n\n#### Command Line Options\n\n```bash\nrethinkport --help\n\nusage: rethinkport [-h] [--config CONFIG] [--host HOST] [--port PORT]\n [--user USER] [--password PASSWORD] [--database DATABASE]\n [--dry-run] [--batch-size BATCH_SIZE] [--table-order TABLE_ORDER]\n dump_path\n\nRethinkPort \ud83d\udea2 - Port your RethinkDB data to MySQL\n\npositional arguments:\n dump_path Path to extracted RethinkDB dump directory\n\noptional arguments:\n -h, --help show this help message and exit\n --config CONFIG Configuration file path\n --host HOST MySQL host (default: localhost)\n --port PORT MySQL port (default: 3306)\n --user USER MySQL username (default: root)\n --password PASSWORD MySQL password\n --database DATABASE MySQL database name\n --dry-run Show what would be migrated without executing\n --batch-size BATCH_SIZE\n Batch size for data insertion (default: 1000)\n --table-order TABLE_ORDER\n Custom table processing order (JSON file)\n```\n\n## Configuration\n\n### Environment Variables\n\n```bash\nexport MYSQL_HOST=localhost\nexport MYSQL_PORT=3306\nexport MYSQL_USER=root\nexport MYSQL_PASSWORD=your_password\nexport MYSQL_DATABASE=your_database\n```\n\n### Configuration File\n\nCreate a `config.json` file:\n\n```json\n{\n \"mysql\": {\n \"host\": \"localhost\",\n \"port\": 3306,\n \"user\": \"root\",\n \"password\": \"your_password\",\n \"database\": \"your_database\",\n \"charset\": \"utf8mb4\"\n },\n \"migration\": {\n \"batch_size\": 1000,\n \"table_order\": [\n \"Users\",\n \"Products\",\n \"Orders\"\n ]\n }\n}\n```\n\n## Data Type Mapping\n\n| RethinkDB Type | MySQL Type | Notes |\n|----------------|------------|-------|\n| String (short) | VARCHAR(255) | Auto-sized based on data |\n| String (long) | TEXT | For strings > 2000 chars |\n| Number (int) | INT/BIGINT | Based on value range |\n| Number (float) | DECIMAL(15,4) | Preserves precision |\n| Boolean | BOOLEAN | Direct mapping |\n| Object | JSON | RethinkDB objects \u2192 MySQL JSON |\n| Array | JSON | RethinkDB arrays \u2192 MySQL JSON |\n| TIME | DATETIME | Converts epoch_time to MySQL datetime |\n| UUID | VARCHAR(255) | For primary/foreign keys |\n\n## Examples\n\n### Basic Migration\n\n```bash\n# Dump RethinkDB\nrethinkdb dump -c localhost:28015 -f myapp_dump.tar.gz\n\n# Extract\ntar -xzf myapp_dump.tar.gz\n\n# Migrate\nrethinkport rethinkdb_dump_2024_01_15/myapp/\n```\n\n### Advanced Migration with Custom Configuration\n\n```bash\nrethinkport \\\n rethinkdb_dump_2024_01_15/myapp/ \\\n --config production_config.json \\\n --batch-size 5000 \\\n --dry-run\n```\n\n## Troubleshooting\n\n### Common Issues\n\n1. **Connection Errors**\n ```\n Error: Failed to connect to MySQL\n ```\n - Check MySQL credentials and connection\n - Ensure MySQL server is running\n - Verify database exists\n\n2. **Large Dataset Timeouts**\n ```\n Error: MySQL server has gone away\n ```\n - Increase `max_allowed_packet` in MySQL\n - Reduce `--batch-size`\n - Check MySQL timeout settings\n\n3. **Schema Conflicts**\n ```\n Error: Table already exists\n ```\n - Tool drops existing tables by default\n - Check MySQL user permissions\n - Verify database name\n\n### Performance Tips\n\n- Use SSD storage for better I/O performance\n- Increase MySQL `innodb_buffer_pool_size`\n- Adjust `--batch-size` based on available memory\n- Run migration during low-traffic periods\n\n## Contributing\n\n1. Fork the repository\n2. Create a feature branch (`git checkout -b feature/amazing-feature`)\n3. Commit your changes (`git commit -m 'Add amazing feature'`)\n4. Push to the branch (`git push origin feature/amazing-feature`)\n5. Open a Pull Request\n\n### Development Setup\n\n```bash\ngit clone https://github.com/aoamusat/rethinkport.git\ncd rethinkport\npip install -e \".[dev]\"\npytest\n```\n\n## License\n\nThis project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.\n\n## Acknowledgments\n\n- Built to solve real-world RethinkDB migration challenges\n- Inspired by the need for reliable database migration tools\n- Thanks to the RethinkDB and MySQL communities\n\n## Support\n\nIf you encounter issues or have questions:\n\n1. Check the [Issues](https://github.com/aoamusat/rethinkport/issues) page\n2. Create a new issue with detailed information\n3. Include your RethinkDB/MySQL versions and error logs\n\n---\n\n**Note**: This tool was created because existing migration solutions were insufficient for complex RethinkDB to MySQL migrations. It's designed to handle real-world scenarios with large datasets, complex schemas, and foreign key relationships.\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Port your RethinkDB data to MySQL",
"version": "1.0.3",
"project_urls": {
"Bug Reports": "https://github.com/aoamusat/rethinkport/issues",
"Documentation": "https://github.com/aoamusat/rethinkport#readme",
"Homepage": "https://github.com/aoamusat/rethinkport",
"Source": "https://github.com/aoamusat/rethinkport"
},
"split_keywords": [
"rethinkdb",
" mysql",
" migration",
" database",
" port",
" converter"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "b4691617fd0c92bb2a1d7cd44fe8b746d8b4909f827b9506b490304f3b6c322a",
"md5": "5aefc99132ed5854beb720b06f9f7f19",
"sha256": "34b881b376381ea1bea2027e3db78245ec469c2b5ad2b0e93ef5c80cb496bad5"
},
"downloads": -1,
"filename": "rethinkport-1.0.3-py3-none-any.whl",
"has_sig": false,
"md5_digest": "5aefc99132ed5854beb720b06f9f7f19",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9",
"size": 12956,
"upload_time": "2025-07-28T11:52:16",
"upload_time_iso_8601": "2025-07-28T11:52:16.727442Z",
"url": "https://files.pythonhosted.org/packages/b4/69/1617fd0c92bb2a1d7cd44fe8b746d8b4909f827b9506b490304f3b6c322a/rethinkport-1.0.3-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "b2bdc5e3863432d9e771abfdfaf6deb12b1da7b8db251d39df8cdce9dc11530c",
"md5": "5fc9f54dff2a3abf81bbaf7233be03fc",
"sha256": "fa4a61ddef1bcecb0534d07acf0b223dfc59decb812b6bcd53d04cb8fb905044"
},
"downloads": -1,
"filename": "rethinkport-1.0.3.tar.gz",
"has_sig": false,
"md5_digest": "5fc9f54dff2a3abf81bbaf7233be03fc",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9",
"size": 19984,
"upload_time": "2025-07-28T11:52:17",
"upload_time_iso_8601": "2025-07-28T11:52:17.718364Z",
"url": "https://files.pythonhosted.org/packages/b2/bd/c5e3863432d9e771abfdfaf6deb12b1da7b8db251d39df8cdce9dc11530c/rethinkport-1.0.3.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-07-28 11:52:17",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "aoamusat",
"github_project": "rethinkport",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [
{
"name": "pymysql",
"specs": [
[
">=",
"1.0.2"
]
]
},
{
"name": "click",
"specs": [
[
">=",
"8.0.0"
]
]
},
{
"name": "tqdm",
"specs": [
[
">=",
"4.64.0"
]
]
},
{
"name": "colorama",
"specs": [
[
">=",
"0.4.4"
]
]
}
],
"lcname": "rethinkport"
}
JSON
