# Pytopspeed Modernized
A modernized Python library for converting Clarion TopSpeed database files (.phd, .mod, .tps, .phz) to SQLite databases and back.
[](https://www.python.org/downloads/)
[](https://opensource.org/licenses/MIT)
[](tests/)
## 🚀 Features
- **Multi-format Support**: Convert .phd, .mod, .tps, and .phz files
- **Combined Conversion**: Merge multiple TopSpeed files into a single SQLite database
- **Reverse Conversion**: Convert SQLite databases back to TopSpeed files
- **PHZ Support**: Handle zip archives containing TopSpeed files
- **Progress Tracking**: Real-time progress reporting and detailed logging
- **Data Integrity**: Preserve all data types and relationships
- **CLI Interface**: Easy-to-use command-line tools
- **Python API**: Programmatic access to all functionality
- **Comprehensive Testing**: 88+ unit tests and integration tests
## 📋 Supported File Formats
| Format | Description | Support |
|--------|-------------|---------|
| `.phd` | Clarion TopSpeed database files | ✅ Full |
| `.mod` | Clarion TopSpeed model files | ✅ Full |
| `.tps` | Clarion TopSpeed files | ✅ Full |
| `.phz` | Zip archives containing TopSpeed files | ✅ Full |
## 🛠️ Quick Start
### Installation
#### Option 1: Install from PyPI (Recommended)
```bash
# Install directly from PyPI
pip install pytopspeed-modernized
```
#### Option 2: Install from Source
```bash
# Clone the repository
git clone https://github.com/gregeasley/pytopspeed_modernized
cd pytopspeed_modernized
# Create conda environment (optional)
conda create -n pytopspeed_modernized python=3.11
conda activate pytopspeed_modernized
# Install in development mode
pip install -e .
```
### Basic Usage
```bash
# Convert a single .phd file to SQLite
python pytopspeed.py convert assets/TxWells.PHD output.sqlite
# Convert multiple files to a combined database
python pytopspeed.py convert assets/TxWells.PHD assets/TxWells.mod combined.sqlite
# Convert a .phz file (zip archive)
python pytopspeed.py convert assets/TxWells.phz output.sqlite
# List contents of a .phz file
python pytopspeed.py list assets/TxWells.phz
# Convert SQLite back to TopSpeed files
python pytopspeed.py reverse input.sqlite output_directory/
```
### Python API
```python
from converter.sqlite_converter import SqliteConverter
# Single file conversion
converter = SqliteConverter()
results = converter.convert('input.phd', 'output.sqlite')
# Multiple file conversion
results = converter.convert_multiple(['file1.phd', 'file2.mod'], 'combined.sqlite')
# PHZ file conversion
from converter.phz_converter import PhzConverter
phz_converter = PhzConverter()
results = phz_converter.convert_phz('input.phz', 'output.sqlite')
```
## 📊 Performance
Based on testing with `TxWells.PHD` and `TxWells.mod`:
- **Single file conversion**: ~1,300 records/second
- **Combined conversion**: ~1,650 records/second
- **Reverse conversion**: ~50,000 records/second
- **Memory efficient**: Configurable batch processing
- **Progress tracking**: Real-time progress reporting
## 🔧 Command Line Interface
### Convert Command
```bash
python pytopspeed.py convert [OPTIONS] INPUT_FILES... OUTPUT_FILE
```
**Options:**
- `--batch-size BATCH_SIZE` - Number of records to process in each batch (default: 1000)
- `-v, --verbose` - Enable verbose logging
- `-q, --quiet` - Suppress progress output
### Reverse Command
```bash
python pytopspeed.py reverse [OPTIONS] INPUT_FILE OUTPUT_DIRECTORY
```
### List Command
```bash
python pytopspeed.py list [OPTIONS] PHZ_FILE
```
## 📚 Documentation
Comprehensive documentation is available in the `docs/` directory:
- **[Installation Guide](docs/INSTALLATION.md)** - Detailed installation instructions
- **[API Documentation](docs/API.md)** - Complete API reference
- **[Troubleshooting Guide](docs/TROUBLESHOOTING.md)** - Common issues and solutions
- **[Developer Documentation](docs/DEVELOPER.md)** - Development and contribution guidelines
## 🧪 Testing
```bash
# Run all unit tests
python -m pytest tests/unit/ -v
# Run integration tests
python -m pytest tests/integration/ -v
# Run with coverage
python -m pytest tests/ --cov=src --cov-report=html
```
**Test Results:**
- ✅ **88 unit tests** - All passing
- ✅ **Integration tests** - End-to-end conversion testing
- ✅ **Performance tests** - Benchmarking and optimization
- ✅ **Error handling tests** - Robust error handling validation
## 📖 Examples
Working examples are available in the `examples/` directory:
- **Basic conversion** - Single file conversion
- **Combined conversion** - Multiple file conversion
- **PHZ handling** - Zip archive processing
- **Reverse conversion** - SQLite to TopSpeed
- **Round-trip conversion** - Complete conversion cycle
- **Custom progress tracking** - Advanced progress monitoring
- **Error handling** - Comprehensive error handling patterns
## 🏗️ Architecture
```
TopSpeed Files → Parser → Schema Mapper → SQLite Converter → SQLite Database
↓ ↓ ↓ ↓
.phd/.mod Modernized Type Mapping Data Migration
.tps/.phz pytopspeed Field Names Batch Processing
```
### Key Components
- **TopSpeed Parser** - Modernized parser for reading TopSpeed files
- **Schema Mapper** - Maps TopSpeed schemas to SQLite
- **SQLite Converter** - Handles data migration and conversion
- **PHZ Converter** - Processes zip archives containing TopSpeed files
- **Reverse Converter** - Converts SQLite back to TopSpeed files
- **CLI Interface** - Command-line tools for easy usage
## 🔄 Data Type Conversion
| TopSpeed Type | SQLite Type | Notes |
|---------------|-------------|-------|
| BYTE | INTEGER | 8-bit unsigned integer |
| SHORT | INTEGER | 16-bit signed integer |
| LONG | INTEGER | 32-bit signed integer |
| DATE | TEXT | Format: YYYY-MM-DD |
| TIME | TEXT | Format: HH:MM:SS |
| STRING | TEXT | Variable length text |
| DECIMAL | REAL | Floating point number |
| MEMO | BLOB | Binary large object |
| BLOB | BLOB | Binary large object |
## 🎯 Key Features
### Table Name Prefixing
When converting multiple files, tables are automatically prefixed to avoid conflicts:
- **.phd files** → `phd_` prefix (e.g., `phd_OWNER`, `phd_CLASS`)
- **.mod files** → `mod_` prefix (e.g., `mod_DEPRECIATION`, `mod_MODID`)
- **.tps files** → `tps_` prefix
- **Other files** → `file_N_` prefix
### Column Name Sanitization
Column names are automatically sanitized for SQLite compatibility:
- **Prefix removal**: `TIT:PROJ_DESCR` → `PROJ_DESCR`
- **Special characters**: `.` → `_`
- **Numeric prefixes**: `123FIELD` → `_123FIELD`
- **Reserved words**: `ORDER` → `ORDER_TABLE`
### Error Handling
- **Graceful degradation** - Continue processing despite individual table errors
- **Detailed logging** - Comprehensive error reporting and debugging information
- **Data preservation** - Ensure data integrity even with parsing issues
- **Recovery mechanisms** - Automatic handling of common issues
## 🤝 Contributing
We welcome contributions! Please see our [Developer Documentation](docs/DEVELOPER.md) for:
- Development setup instructions
- Code style guidelines
- Testing requirements
- Contribution process
### Development Setup
```bash
# Clone and setup development environment
git clone https://github.com/gregeasley/pytopspeed_modernized
cd pytopspeed_modernized
conda create -n pytopspeed_modernized_dev python=3.11
conda activate pytopspeed_modernized_dev
pip install -e .[dev]
# Run tests
python -m pytest tests/ -v
```
## 📄 License
This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
## 🙏 Acknowledgments
- Based on the original [pytopspeed library](https://github.com/dylangiles/pytopspeed/)
- Modernized for Python 3.11 and construct 2.10+
- Enhanced with SQLite conversion and reverse conversion capabilities
- Comprehensive testing and documentation
## 📞 Support
- **Documentation**: See the `docs/` directory for comprehensive guides
- **Examples**: Check the `examples/` directory for working code
- **Issues**: Open an issue in the project repository
- **Discussions**: Use the project's discussion forum for questions
---
**Ready to convert your TopSpeed files?** Start with the [Installation Guide](docs/INSTALLATION.md) and try the [examples](examples/)!
Raw data
{
"_id": null,
"home_page": "https://github.com/gregeasley/pytopspeed_modernized",
"name": "pytopspeed-modernized",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": "Greg Easley <greg@easley.dev>",
"keywords": "topspeed, clarion, database, sqlite, conversion, migration, legacy",
"author": "Greg Easley",
"author_email": "Greg Easley <greg@easley.dev>",
"download_url": "https://files.pythonhosted.org/packages/29/6d/556157c20e135bdbc8df695148840d688598f9cac04cb21349446b03cd24/pytopspeed_modernized-1.0.2.tar.gz",
"platform": null,
"description": "# Pytopspeed Modernized\r\n\r\nA modernized Python library for converting Clarion TopSpeed database files (.phd, .mod, .tps, .phz) to SQLite databases and back.\r\n\r\n[](https://www.python.org/downloads/)\r\n[](https://opensource.org/licenses/MIT)\r\n[](tests/)\r\n\r\n## \ud83d\ude80 Features\r\n\r\n- **Multi-format Support**: Convert .phd, .mod, .tps, and .phz files\r\n- **Combined Conversion**: Merge multiple TopSpeed files into a single SQLite database\r\n- **Reverse Conversion**: Convert SQLite databases back to TopSpeed files\r\n- **PHZ Support**: Handle zip archives containing TopSpeed files\r\n- **Progress Tracking**: Real-time progress reporting and detailed logging\r\n- **Data Integrity**: Preserve all data types and relationships\r\n- **CLI Interface**: Easy-to-use command-line tools\r\n- **Python API**: Programmatic access to all functionality\r\n- **Comprehensive Testing**: 88+ unit tests and integration tests\r\n\r\n## \ud83d\udccb Supported File Formats\r\n\r\n| Format | Description | Support |\r\n|--------|-------------|---------|\r\n| `.phd` | Clarion TopSpeed database files | \u2705 Full |\r\n| `.mod` | Clarion TopSpeed model files | \u2705 Full |\r\n| `.tps` | Clarion TopSpeed files | \u2705 Full |\r\n| `.phz` | Zip archives containing TopSpeed files | \u2705 Full |\r\n\r\n## \ud83d\udee0\ufe0f Quick Start\r\n\r\n### Installation\r\n\r\n#### Option 1: Install from PyPI (Recommended)\r\n```bash\r\n# Install directly from PyPI\r\npip install pytopspeed-modernized\r\n```\r\n\r\n#### Option 2: Install from Source\r\n```bash\r\n# Clone the repository\r\ngit clone https://github.com/gregeasley/pytopspeed_modernized\r\ncd pytopspeed_modernized\r\n\r\n# Create conda environment (optional)\r\nconda create -n pytopspeed_modernized python=3.11\r\nconda activate pytopspeed_modernized\r\n\r\n# Install in development mode\r\npip install -e .\r\n```\r\n\r\n### Basic Usage\r\n\r\n```bash\r\n# Convert a single .phd file to SQLite\r\npython pytopspeed.py convert assets/TxWells.PHD output.sqlite\r\n\r\n# Convert multiple files to a combined database\r\npython pytopspeed.py convert assets/TxWells.PHD assets/TxWells.mod combined.sqlite\r\n\r\n# Convert a .phz file (zip archive)\r\npython pytopspeed.py convert assets/TxWells.phz output.sqlite\r\n\r\n# List contents of a .phz file\r\npython pytopspeed.py list assets/TxWells.phz\r\n\r\n# Convert SQLite back to TopSpeed files\r\npython pytopspeed.py reverse input.sqlite output_directory/\r\n```\r\n\r\n### Python API\r\n\r\n```python\r\nfrom converter.sqlite_converter import SqliteConverter\r\n\r\n# Single file conversion\r\nconverter = SqliteConverter()\r\nresults = converter.convert('input.phd', 'output.sqlite')\r\n\r\n# Multiple file conversion\r\nresults = converter.convert_multiple(['file1.phd', 'file2.mod'], 'combined.sqlite')\r\n\r\n# PHZ file conversion\r\nfrom converter.phz_converter import PhzConverter\r\nphz_converter = PhzConverter()\r\nresults = phz_converter.convert_phz('input.phz', 'output.sqlite')\r\n```\r\n\r\n## \ud83d\udcca Performance\r\n\r\nBased on testing with `TxWells.PHD` and `TxWells.mod`:\r\n\r\n- **Single file conversion**: ~1,300 records/second\r\n- **Combined conversion**: ~1,650 records/second \r\n- **Reverse conversion**: ~50,000 records/second\r\n- **Memory efficient**: Configurable batch processing\r\n- **Progress tracking**: Real-time progress reporting\r\n\r\n## \ud83d\udd27 Command Line Interface\r\n\r\n### Convert Command\r\n\r\n```bash\r\npython pytopspeed.py convert [OPTIONS] INPUT_FILES... OUTPUT_FILE\r\n```\r\n\r\n**Options:**\r\n- `--batch-size BATCH_SIZE` - Number of records to process in each batch (default: 1000)\r\n- `-v, --verbose` - Enable verbose logging\r\n- `-q, --quiet` - Suppress progress output\r\n\r\n### Reverse Command\r\n\r\n```bash\r\npython pytopspeed.py reverse [OPTIONS] INPUT_FILE OUTPUT_DIRECTORY\r\n```\r\n\r\n### List Command\r\n\r\n```bash\r\npython pytopspeed.py list [OPTIONS] PHZ_FILE\r\n```\r\n\r\n## \ud83d\udcda Documentation\r\n\r\nComprehensive documentation is available in the `docs/` directory:\r\n\r\n- **[Installation Guide](docs/INSTALLATION.md)** - Detailed installation instructions\r\n- **[API Documentation](docs/API.md)** - Complete API reference\r\n- **[Troubleshooting Guide](docs/TROUBLESHOOTING.md)** - Common issues and solutions\r\n- **[Developer Documentation](docs/DEVELOPER.md)** - Development and contribution guidelines\r\n\r\n## \ud83e\uddea Testing\r\n\r\n```bash\r\n# Run all unit tests\r\npython -m pytest tests/unit/ -v\r\n\r\n# Run integration tests\r\npython -m pytest tests/integration/ -v\r\n\r\n# Run with coverage\r\npython -m pytest tests/ --cov=src --cov-report=html\r\n```\r\n\r\n**Test Results:**\r\n- \u2705 **88 unit tests** - All passing\r\n- \u2705 **Integration tests** - End-to-end conversion testing\r\n- \u2705 **Performance tests** - Benchmarking and optimization\r\n- \u2705 **Error handling tests** - Robust error handling validation\r\n\r\n## \ud83d\udcd6 Examples\r\n\r\nWorking examples are available in the `examples/` directory:\r\n\r\n- **Basic conversion** - Single file conversion\r\n- **Combined conversion** - Multiple file conversion\r\n- **PHZ handling** - Zip archive processing\r\n- **Reverse conversion** - SQLite to TopSpeed\r\n- **Round-trip conversion** - Complete conversion cycle\r\n- **Custom progress tracking** - Advanced progress monitoring\r\n- **Error handling** - Comprehensive error handling patterns\r\n\r\n## \ud83c\udfd7\ufe0f Architecture\r\n\r\n```\r\nTopSpeed Files \u2192 Parser \u2192 Schema Mapper \u2192 SQLite Converter \u2192 SQLite Database\r\n \u2193 \u2193 \u2193 \u2193\r\n .phd/.mod Modernized Type Mapping Data Migration\r\n .tps/.phz pytopspeed Field Names Batch Processing\r\n```\r\n\r\n### Key Components\r\n\r\n- **TopSpeed Parser** - Modernized parser for reading TopSpeed files\r\n- **Schema Mapper** - Maps TopSpeed schemas to SQLite\r\n- **SQLite Converter** - Handles data migration and conversion\r\n- **PHZ Converter** - Processes zip archives containing TopSpeed files\r\n- **Reverse Converter** - Converts SQLite back to TopSpeed files\r\n- **CLI Interface** - Command-line tools for easy usage\r\n\r\n## \ud83d\udd04 Data Type Conversion\r\n\r\n| TopSpeed Type | SQLite Type | Notes |\r\n|---------------|-------------|-------|\r\n| BYTE | INTEGER | 8-bit unsigned integer |\r\n| SHORT | INTEGER | 16-bit signed integer |\r\n| LONG | INTEGER | 32-bit signed integer |\r\n| DATE | TEXT | Format: YYYY-MM-DD |\r\n| TIME | TEXT | Format: HH:MM:SS |\r\n| STRING | TEXT | Variable length text |\r\n| DECIMAL | REAL | Floating point number |\r\n| MEMO | BLOB | Binary large object |\r\n| BLOB | BLOB | Binary large object |\r\n\r\n## \ud83c\udfaf Key Features\r\n\r\n### Table Name Prefixing\r\n\r\nWhen converting multiple files, tables are automatically prefixed to avoid conflicts:\r\n\r\n- **.phd files** \u2192 `phd_` prefix (e.g., `phd_OWNER`, `phd_CLASS`)\r\n- **.mod files** \u2192 `mod_` prefix (e.g., `mod_DEPRECIATION`, `mod_MODID`)\r\n- **.tps files** \u2192 `tps_` prefix\r\n- **Other files** \u2192 `file_N_` prefix\r\n\r\n### Column Name Sanitization\r\n\r\nColumn names are automatically sanitized for SQLite compatibility:\r\n\r\n- **Prefix removal**: `TIT:PROJ_DESCR` \u2192 `PROJ_DESCR`\r\n- **Special characters**: `.` \u2192 `_`\r\n- **Numeric prefixes**: `123FIELD` \u2192 `_123FIELD`\r\n- **Reserved words**: `ORDER` \u2192 `ORDER_TABLE`\r\n\r\n### Error Handling\r\n\r\n- **Graceful degradation** - Continue processing despite individual table errors\r\n- **Detailed logging** - Comprehensive error reporting and debugging information\r\n- **Data preservation** - Ensure data integrity even with parsing issues\r\n- **Recovery mechanisms** - Automatic handling of common issues\r\n\r\n## \ud83e\udd1d Contributing\r\n\r\nWe welcome contributions! Please see our [Developer Documentation](docs/DEVELOPER.md) for:\r\n\r\n- Development setup instructions\r\n- Code style guidelines\r\n- Testing requirements\r\n- Contribution process\r\n\r\n### Development Setup\r\n\r\n```bash\r\n# Clone and setup development environment\r\ngit clone https://github.com/gregeasley/pytopspeed_modernized\r\ncd pytopspeed_modernized\r\nconda create -n pytopspeed_modernized_dev python=3.11\r\nconda activate pytopspeed_modernized_dev\r\npip install -e .[dev]\r\n\r\n# Run tests\r\npython -m pytest tests/ -v\r\n```\r\n\r\n## \ud83d\udcc4 License\r\n\r\nThis project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.\r\n\r\n## \ud83d\ude4f Acknowledgments\r\n\r\n- Based on the original [pytopspeed library](https://github.com/dylangiles/pytopspeed/)\r\n- Modernized for Python 3.11 and construct 2.10+\r\n- Enhanced with SQLite conversion and reverse conversion capabilities\r\n- Comprehensive testing and documentation\r\n\r\n## \ud83d\udcde Support\r\n\r\n- **Documentation**: See the `docs/` directory for comprehensive guides\r\n- **Examples**: Check the `examples/` directory for working code\r\n- **Issues**: Open an issue in the project repository\r\n- **Discussions**: Use the project's discussion forum for questions\r\n\r\n---\r\n\r\n**Ready to convert your TopSpeed files?** Start with the [Installation Guide](docs/INSTALLATION.md) and try the [examples](examples/)!\r\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Modernized pytopspeed library for converting TopSpeed database files to SQLite",
"version": "1.0.2",
"project_urls": {
"Bug Tracker": "https://github.com/gregeasley/pytopspeed_modernized/issues",
"Documentation": "https://github.com/gregeasley/pytopspeed_modernized/blob/master/docs/README.md",
"Homepage": "https://github.com/gregeasley/pytopspeed_modernized",
"Repository": "https://github.com/gregeasley/pytopspeed_modernized"
},
"split_keywords": [
"topspeed",
" clarion",
" database",
" sqlite",
" conversion",
" migration",
" legacy"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "674d1876437cb1ce71c937ea5f0d09598a7a1a5175995726970aa4f2e5803ddd",
"md5": "274f754dffd2721cd5b3b08282854119",
"sha256": "d8a728fe06323219e00230fcb73a31f4b0d92f0bf96b895f7fd6f6bfadedd226"
},
"downloads": -1,
"filename": "pytopspeed_modernized-1.0.2-py2.py3-none-any.whl",
"has_sig": false,
"md5_digest": "274f754dffd2721cd5b3b08282854119",
"packagetype": "bdist_wheel",
"python_version": "py2.py3",
"requires_python": ">=3.8",
"size": 59164,
"upload_time": "2025-09-05T20:52:45",
"upload_time_iso_8601": "2025-09-05T20:52:45.716789Z",
"url": "https://files.pythonhosted.org/packages/67/4d/1876437cb1ce71c937ea5f0d09598a7a1a5175995726970aa4f2e5803ddd/pytopspeed_modernized-1.0.2-py2.py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "296d556157c20e135bdbc8df695148840d688598f9cac04cb21349446b03cd24",
"md5": "bab2e47554cda160623bbbaf5c094aae",
"sha256": "0922d97cc3291355fc7c45b0c6c3b6b0ea9554e55eccfd6e5fca45bad6fd8989"
},
"downloads": -1,
"filename": "pytopspeed_modernized-1.0.2.tar.gz",
"has_sig": false,
"md5_digest": "bab2e47554cda160623bbbaf5c094aae",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 681613,
"upload_time": "2025-09-05T20:52:47",
"upload_time_iso_8601": "2025-09-05T20:52:47.483970Z",
"url": "https://files.pythonhosted.org/packages/29/6d/556157c20e135bdbc8df695148840d688598f9cac04cb21349446b03cd24/pytopspeed_modernized-1.0.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-09-05 20:52:47",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "gregeasley",
"github_project": "pytopspeed_modernized",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [
{
"name": "construct",
"specs": [
[
">=",
"2.10.0"
]
]
},
{
"name": "click",
"specs": [
[
">=",
"8.0.0"
]
]
},
{
"name": "pytest",
"specs": [
[
">=",
"7.0.0"
]
]
},
{
"name": "pytest-cov",
"specs": [
[
">=",
"4.0.0"
]
]
},
{
"name": "pandas",
"specs": [
[
">=",
"1.5.0"
]
]
}
],
"lcname": "pytopspeed-modernized"
}
JSON
