pytopspeed-modernized

Name	pytopspeed-modernized JSON
Version	1.1.3 JSON
	download
home_page	https://github.com/gregeasley/pytopspeed_modernized
Summary	Modernized pytopspeed library for converting TopSpeed database files to SQLite
upload_time	2025-09-12 22:40:39
maintainer	None
docs_url	None
author	Greg Easley
requires_python	>=3.8
license	MIT
keywords	topspeed clarion database sqlite conversion migration legacy
VCS
bugtrack_url
requirements	construct click pytest pytest-cov pandas
Travis-CI	No Travis.
coveralls test coverage	No coveralls.

            # Pytopspeed Modernized

A modernized Python library for converting Clarion TopSpeed database files (.phd, .mod, .tps, .phz) to SQLite databases and back.

[![Python 3.8+](https://img.shields.io/badge/python-3.8+-blue.svg)](https://www.python.org/downloads/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![Tests](https://img.shields.io/badge/tests-354%20passing-brightgreen.svg)](tests/)

## 🚀 Features

- **Multi-format Support**: Convert .phd, .mod, .tps, and .phz files
- **Multidimensional Arrays**: Advanced handling of TopSpeed array fields with JSON storage
- **Enterprise Resilience**: Memory management, adaptive batch sizing, and error recovery for large databases
- **Combined Conversion**: Merge multiple TopSpeed files into a single SQLite database
- **Reverse Conversion**: Convert SQLite databases back to TopSpeed files
- **PHZ Support**: Handle zip archives containing TopSpeed files
- **Progress Tracking**: Real-time progress reporting and detailed logging
- **Data Integrity**: Preserve all data types, relationships, and null vs zero distinctions
- **CLI Interface**: Easy-to-use command-line tools
- **Python API**: Programmatic access to all functionality
- **Comprehensive Testing**: 354 unit tests, integration tests, and performance tests with 100% pass rate

## 📋 Supported File Formats

| Format | Description | Support | Converter Class |
|--------|-------------|---------|-----------------|
| `.phd` | Clarion TopSpeed database files | ✅ Full | `SqliteConverter` |
| `.mod` | Clarion TopSpeed model files | ✅ Full | `SqliteConverter` |
| `.tps` | Clarion TopSpeed files | ✅ Full | `SqliteConverter` |
| `.phz` | Zip archives containing TopSpeed files | ✅ Full | `PhzConverter` |

## 🔧 File Types and Usage

### Single TopSpeed Files (.phd, .mod, .tps)
Use `SqliteConverter` for individual TopSpeed files:

```python
from converter.sqlite_converter import SqliteConverter

converter = SqliteConverter()
result = converter.convert('input.phd', 'output.sqlite')
```

### Multiple TopSpeed Files (Combined Database)
Use `SqliteConverter.convert_multiple()` to combine multiple files into one SQLite database:

```python
from converter.sqlite_converter import SqliteConverter

converter = SqliteConverter()
result = converter.convert_multiple(
    ['file1.phd', 'file2.mod', 'file3.tps'], 
    'combined.sqlite'
)
```

### PHZ Files (Zip Archives)
Use `PhzConverter` for .phz files (zip archives containing TopSpeed files):

```python
from converter.phz_converter import PhzConverter

converter = PhzConverter()
result = converter.convert_phz('input.phz', 'output.sqlite')
```

### Reverse Conversion (SQLite to TopSpeed)
Use `ReverseConverter` to convert SQLite databases back to TopSpeed files:

```python
from converter.reverse_converter import ReverseConverter

converter = ReverseConverter()
result = converter.convert_sqlite_to_topspeed('input.sqlite', 'output_directory/')
```

## 🔄 Multidimensional Array Handling

Pytopspeed Modernized includes advanced support for TopSpeed multidimensional arrays, automatically detecting and converting array fields to JSON format in SQLite.

### Array Detection

The system automatically detects two types of arrays:

1. **Single-Field Arrays**: Large fields containing multiple elements (e.g., 96-byte `DAT:PROD1` with 12 elements)
2. **Multi-Field Arrays**: Multiple small fields forming an array (e.g., `CUM:PROD1`, `CUM:PROD2`, etc.)

### Example: MONHIST Table

```python
# TopSpeed structure
DAT:PROD1    # 96-byte field with 12 DOUBLE elements
DAT:PROD2    # 96-byte field with 12 DOUBLE elements
DAT:PROD3    # 96-byte field with 12 DOUBLE elements

# SQLite result
PROD1        # JSON: [1.5, 2.3, 0.0, null, ...]
PROD2        # JSON: [0.8, 1.2, 0.0, null, ...]
PROD3        # JSON: [2.1, 1.8, 0.0, null, ...]
```

### Data Type Preservation

- **Zero vs NULL**: Distinguishes between actual zero values (`0.0`) and missing data (`null`)
- **Boolean Arrays**: Converts `BYTE` arrays to proper boolean values (`true`/`false`)
- **Numeric Arrays**: Preserves `DOUBLE`, `LONG`, `SHORT` precision
- **String Arrays**: Maintains text encoding and length

### Querying Array Data

```sql
-- Query array elements
SELECT 
    LSE_ID,
    json_extract(PROD1, '$[0]') as PROD1_Month1,
    json_extract(PROD1, '$[1]') as PROD1_Month2
FROM MONHIST;

-- Filter by array content
SELECT * FROM MONHIST 
WHERE json_extract(PROD1, '$[0]') > 100.0;

-- Count non-null elements
SELECT LSE_ID,
       json_array_length(PROD1) as PROD1_Count
FROM MONHIST;
```

## 🛡️ Enterprise Resilience Features

### Memory Management
- **Configurable Memory Limits**: 200MB - 2GB based on database size
- **Automatic Cleanup**: Garbage collection every 1,000 records
- **Memory Monitoring**: Real-time memory usage tracking with psutil
- **Streaming Processing**: Handle databases larger than available RAM

### Adaptive Batch Sizing
- **Dynamic Optimization**: Batch sizes automatically adjust based on table characteristics
  - Small records (< 100B): 100-400 records per batch
  - Medium records (100B-1KB): 25-100 records per batch
  - Large records (1KB-5KB): 10-25 records per batch
  - Very large records (> 5KB): 5-10 records per batch
- **Complex Table Handling**: Smaller batches for tables with many fields
- **Memory-Aware**: Batch sizes adapt to available memory

### Predefined Configurations
```python
from converter.resilience_config import get_resilience_config

# Small databases (< 10MB)
config = get_resilience_config('small')  # 200MB limit, 200 batch size

# Medium databases (10MB - 1GB)  
config = get_resilience_config('medium')  # 500MB limit, 100 batch size

# Large databases (1GB - 10GB)
config = get_resilience_config('large')  # 1GB limit, 50 batch size, parallel processing

# Enterprise databases (> 10GB)
config = get_resilience_config('enterprise')  # 2GB limit, 25 batch size, full features
```

### Error Recovery
- **Partial Conversion**: Save progress even if conversion is interrupted
- **Graceful Degradation**: Continue processing despite individual record failures
- **Detailed Logging**: Comprehensive error reporting for troubleshooting
- **Resume Capability**: Restart from checkpoints for enterprise configurations

### Performance Optimization
- **SQLite Tuning**: WAL mode, optimized cache sizes, memory temp storage
- **Parallel Processing**: Multi-threaded conversion for large databases
- **Progress Tracking**: Real-time progress reporting for long operations
- **Resource Monitoring**: Prevent system overload with configurable limits

### Scalability
- **Tested Limits**: Successfully handles databases with millions of records
- **Large Tables**: FORCAST table with 4,370 records (2,528 bytes each)
- **Memory Efficiency**: 60-80% reduction in memory usage with adaptive batching
- **Enterprise Ready**: Production-tested with databases > 10GB

## 🛠️ Quick Start

### Installation

#### Option 1: Install from PyPI (Recommended)
```bash
# Install directly from PyPI
pip install pytopspeed-modernized
```

#### Option 2: Install from Source
```bash
# Clone the repository
git clone https://github.com/gregeasley/pytopspeed_modernized
cd pytopspeed_modernized

# Create conda environment (optional)
conda create -n pytopspeed_modernized python=3.11
conda activate pytopspeed_modernized

# Install in development mode
pip install -e .
```

### Basic Usage

```bash
# Convert a single .phd file to SQLite
python pytopspeed.py convert assets/TxWells.PHD output.sqlite

# Convert multiple files to a combined database
python pytopspeed.py convert assets/TxWells.PHD assets/TxWells.mod combined.sqlite

# Convert a .phz file (zip archive)
python pytopspeed.py convert assets/TxWells.phz output.sqlite

# List contents of a .phz file
python pytopspeed.py list assets/TxWells.phz

# Convert SQLite back to TopSpeed files
python pytopspeed.py reverse input.sqlite output_directory/
```

### Python API Examples

#### Single TopSpeed File Conversion
```python
from converter.sqlite_converter import SqliteConverter

# Convert a single .phd, .mod, or .tps file
converter = SqliteConverter()
results = converter.convert('input.phd', 'output.sqlite')
print(f"Success: {results['success']}, Records: {results['total_records']}")
```

#### Multiple Files to Combined Database
```python
from converter.sqlite_converter import SqliteConverter

# Combine multiple TopSpeed files into one SQLite database
converter = SqliteConverter()
results = converter.convert_multiple(
    ['file1.phd', 'file2.mod'], 
    'combined.sqlite'
)
print(f"Files processed: {results['files_processed']}")
```

#### PHZ File Conversion (Zip Archives)
```python
from converter.phz_converter import PhzConverter

# Convert .phz files (zip archives containing TopSpeed files)
phz_converter = PhzConverter()
results = phz_converter.convert_phz('input.phz', 'output.sqlite')
print(f"Extracted files: {results['extracted_files']}")
```

#### Reverse Conversion (SQLite to TopSpeed)
```python
from converter.reverse_converter import ReverseConverter

# Convert SQLite database back to TopSpeed files
reverse_converter = ReverseConverter()
results = reverse_converter.convert_sqlite_to_topspeed(
    'input.sqlite', 
    'output_directory/'
)
print(f"Generated files: {results['generated_files']}")
```

## 🚨 Common Issues and Solutions

### Wrong Converter for File Type
**Problem**: Using `SqliteConverter.convert()` with a `.phz` file
```python
# ❌ WRONG - This will fail
converter = SqliteConverter()
result = converter.convert('input.phz', 'output.sqlite')  # Error: 'TPS' object has no attribute 'tables'
```

**Solution**: Use `PhzConverter.convert_phz()` for `.phz` files
```python
# ✅ CORRECT
from converter.phz_converter import PhzConverter
converter = PhzConverter()
result = converter.convert_phz('input.phz', 'output.sqlite')
```

### File Not Found
**Problem**: File path doesn't exist
```python
# ❌ WRONG - File doesn't exist
result = converter.convert('nonexistent.phd', 'output.sqlite')
```

**Solution**: Check file exists before conversion
```python
import os
if os.path.exists('input.phd'):
    result = converter.convert('input.phd', 'output.sqlite')
else:
    print("File not found!")
```

### Import Errors
**Problem**: Import path issues
```python
# ❌ WRONG - Incorrect import path
from sqlite_converter import SqliteConverter  # ModuleNotFoundError
```

**Solution**: Use correct import path
```python
# ✅ CORRECT
from converter.sqlite_converter import SqliteConverter
```

## 📊 Performance

Based on testing with `TxWells.PHD` and `TxWells.mod`:

- **Single file conversion**: ~1,300 records/second
- **Combined conversion**: ~1,650 records/second  
- **Reverse conversion**: ~50,000 records/second
- **Memory efficient**: Configurable batch processing
- **Progress tracking**: Real-time progress reporting

## 🔧 Command Line Interface

### Convert Command

```bash
python pytopspeed.py convert [OPTIONS] INPUT_FILES... OUTPUT_FILE
```

**Options:**
- `--batch-size BATCH_SIZE` - Number of records to process in each batch (default: 1000)
- `-v, --verbose` - Enable verbose logging
- `-q, --quiet` - Suppress progress output

### Reverse Command

```bash
python pytopspeed.py reverse [OPTIONS] INPUT_FILE OUTPUT_DIRECTORY
```

### List Command

```bash
python pytopspeed.py list [OPTIONS] PHZ_FILE
```

## 📚 Documentation

Comprehensive documentation is available in the `docs/` directory:

- **[Installation Guide](docs/INSTALLATION.md)** - Detailed installation instructions
- **[API Documentation](docs/API.md)** - Complete API reference
- **[Troubleshooting Guide](docs/TROUBLESHOOTING.md)** - Common issues and solutions
- **[Developer Documentation](docs/DEVELOPER.md)** - Development and contribution guidelines

## 🧪 Testing

### Comprehensive Test Suite

```bash
# Run all resilience tests
python tests/run_resilience_tests.py

# Run specific test types
python tests/run_resilience_tests.py unit
python tests/run_resilience_tests.py integration
python tests/run_resilience_tests.py performance

# Run with coverage
python tests/run_resilience_tests.py -c

# Run with pytest directly
python -m pytest tests/unit/ -v
python -m pytest tests/integration/ -v
python -m pytest tests/performance/ --run-performance
```

### Test Coverage

**Unit Tests (70+ tests):**
- ✅ **ResilienceEnhancer** - Memory management, adaptive batch sizing, data extraction
- ✅ **ResilienceConfig** - Configuration management and validation
- ✅ **SQLite Converter Enhancements** - Enhanced table definition parsing
- ✅ **Error Handling** - Robust error recovery and fallback mechanisms

**Integration Tests (15+ tests):**
- ✅ **End-to-End Scenarios** - Complete conversion workflows
- ✅ **Configuration Selection** - Auto-detection based on database size
- ✅ **Component Integration** - Cross-component interaction validation
- ✅ **Performance Integration** - Resource usage under realistic conditions

**Performance Tests (12+ tests):**
- ✅ **Memory Performance** - Memory usage patterns and cleanup efficiency
- ✅ **Processing Performance** - Speed and throughput under various loads
- ✅ **Scalability Performance** - Performance with increasing data sizes
- ✅ **Concurrent Performance** - Multi-threaded operation testing

**Test Results:**
- ✅ **354 total tests** - All passing with 100% pass rate
- ✅ **95%+ code coverage** - Comprehensive test coverage
- ✅ **Performance benchmarks** - Validated scalability characteristics
- ✅ **Memory efficiency** - Tested memory usage patterns

## 📖 Examples

Working examples are available in the `examples/` directory:

- **Basic conversion** - Single file conversion
- **Combined conversion** - Multiple file conversion
- **PHZ handling** - Zip archive processing
- **Reverse conversion** - SQLite to TopSpeed
- **Round-trip conversion** - Complete conversion cycle
- **Custom progress tracking** - Advanced progress monitoring
- **Error handling** - Comprehensive error handling patterns

## 🏗️ Architecture

```
TopSpeed Files → Parser → Schema Mapper → SQLite Converter → SQLite Database
     ↓              ↓           ↓              ↓
   .phd/.mod    Modernized   Type Mapping   Data Migration
   .tps/.phz    pytopspeed   Field Names    Batch Processing
```

### Key Components

- **TopSpeed Parser** - Modernized parser for reading TopSpeed files
- **Schema Mapper** - Maps TopSpeed schemas to SQLite
- **SQLite Converter** - Handles data migration and conversion
- **PHZ Converter** - Processes zip archives containing TopSpeed files
- **Reverse Converter** - Converts SQLite back to TopSpeed files
- **CLI Interface** - Command-line tools for easy usage

## 🔄 Data Type Conversion

| TopSpeed Type | SQLite Type | Notes |
|---------------|-------------|-------|
| BYTE | INTEGER | 8-bit unsigned integer |
| SHORT | INTEGER | 16-bit signed integer |
| LONG | INTEGER | 32-bit signed integer |
| DATE | TEXT | Format: YYYY-MM-DD |
| TIME | TEXT | Format: HH:MM:SS |
| STRING | TEXT | Variable length text |
| DECIMAL | REAL | Floating point number |
| MEMO | BLOB | Binary large object |
| BLOB | BLOB | Binary large object |

## 🎯 Key Features

### Table Name Prefixing

When converting multiple files, tables are automatically prefixed to avoid conflicts:

- **.phd files** → `phd_` prefix (e.g., `phd_OWNER`, `phd_CLASS`)
- **.mod files** → `mod_` prefix (e.g., `mod_DEPRECIATION`, `mod_MODID`)
- **.tps files** → `tps_` prefix
- **Other files** → `file_N_` prefix

### Column Name Sanitization

Column names are automatically sanitized for SQLite compatibility:

- **Prefix removal**: `TIT:PROJ_DESCR` → `PROJ_DESCR`
- **Special characters**: `.` → `_`
- **Numeric prefixes**: `123FIELD` → `_123FIELD`
- **Reserved words**: `ORDER` → `ORDER_TABLE`

### Error Handling

- **Graceful degradation** - Continue processing despite individual table errors
- **Detailed logging** - Comprehensive error reporting and debugging information
- **Data preservation** - Ensure data integrity even with parsing issues
- **Recovery mechanisms** - Automatic handling of common issues

## 🤝 Contributing

We welcome contributions! Please see our [Developer Documentation](docs/DEVELOPER.md) for:

- Development setup instructions
- Code style guidelines
- Testing requirements
- Contribution process

### Development Setup

```bash
# Clone and setup development environment
git clone https://github.com/gregeasley/pytopspeed_modernized
cd pytopspeed_modernized
conda create -n pytopspeed_modernized_dev python=3.11
conda activate pytopspeed_modernized_dev
pip install -e .[dev]

# Run tests
python -m pytest tests/ -v
```

## 📄 License

This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.

## 🙏 Acknowledgments

- Based on the original [pytopspeed library](https://github.com/dylangiles/pytopspeed/)
- Modernized for Python 3.11 and construct 2.10+
- Enhanced with SQLite conversion and reverse conversion capabilities
- Comprehensive testing and documentation

## 📞 Support

- **Documentation**: See the `docs/` directory for comprehensive guides
- **Examples**: Check the `examples/` directory for working code
- **Issues**: Open an issue in the project repository
- **Discussions**: Use the project's discussion forum for questions

---

**Ready to convert your TopSpeed files?** Start with the [Installation Guide](docs/INSTALLATION.md) and try the [examples](examples/)!

            

Raw data

            {
    "_id": null,
    "home_page": "https://github.com/gregeasley/pytopspeed_modernized",
    "name": "pytopspeed-modernized",
    "maintainer": null,
    "docs_url": null,
    "requires_python": ">=3.8",
    "maintainer_email": "Greg Easley <greg@easley.dev>",
    "keywords": "topspeed, clarion, database, sqlite, conversion, migration, legacy",
    "author": "Greg Easley",
    "author_email": "Greg Easley <greg@easley.dev>",
    "download_url": "https://files.pythonhosted.org/packages/86/e8/f73cc96743f2d6eea24e1ac04fd80c34b11841e62660bb0136ef35948c75/pytopspeed_modernized-1.1.3.tar.gz",
    "platform": null,
    "description": "# Pytopspeed Modernized\r\n\r\nA modernized Python library for converting Clarion TopSpeed database files (.phd, .mod, .tps, .phz) to SQLite databases and back.\r\n\r\n[![Python 3.8+](https://img.shields.io/badge/python-3.8+-blue.svg)](https://www.python.org/downloads/)\r\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\r\n[![Tests](https://img.shields.io/badge/tests-354%20passing-brightgreen.svg)](tests/)\r\n\r\n## \ud83d\ude80 Features\r\n\r\n- **Multi-format Support**: Convert .phd, .mod, .tps, and .phz files\r\n- **Multidimensional Arrays**: Advanced handling of TopSpeed array fields with JSON storage\r\n- **Enterprise Resilience**: Memory management, adaptive batch sizing, and error recovery for large databases\r\n- **Combined Conversion**: Merge multiple TopSpeed files into a single SQLite database\r\n- **Reverse Conversion**: Convert SQLite databases back to TopSpeed files\r\n- **PHZ Support**: Handle zip archives containing TopSpeed files\r\n- **Progress Tracking**: Real-time progress reporting and detailed logging\r\n- **Data Integrity**: Preserve all data types, relationships, and null vs zero distinctions\r\n- **CLI Interface**: Easy-to-use command-line tools\r\n- **Python API**: Programmatic access to all functionality\r\n- **Comprehensive Testing**: 354 unit tests, integration tests, and performance tests with 100% pass rate\r\n\r\n## \ud83d\udccb Supported File Formats\r\n\r\n| Format | Description | Support | Converter Class |\r\n|--------|-------------|---------|-----------------|\r\n| `.phd` | Clarion TopSpeed database files | \u2705 Full | `SqliteConverter` |\r\n| `.mod` | Clarion TopSpeed model files | \u2705 Full | `SqliteConverter` |\r\n| `.tps` | Clarion TopSpeed files | \u2705 Full | `SqliteConverter` |\r\n| `.phz` | Zip archives containing TopSpeed files | \u2705 Full | `PhzConverter` |\r\n\r\n## \ud83d\udd27 File Types and Usage\r\n\r\n### Single TopSpeed Files (.phd, .mod, .tps)\r\nUse `SqliteConverter` for individual TopSpeed files:\r\n\r\n```python\r\nfrom converter.sqlite_converter import SqliteConverter\r\n\r\nconverter = SqliteConverter()\r\nresult = converter.convert('input.phd', 'output.sqlite')\r\n```\r\n\r\n### Multiple TopSpeed Files (Combined Database)\r\nUse `SqliteConverter.convert_multiple()` to combine multiple files into one SQLite database:\r\n\r\n```python\r\nfrom converter.sqlite_converter import SqliteConverter\r\n\r\nconverter = SqliteConverter()\r\nresult = converter.convert_multiple(\r\n    ['file1.phd', 'file2.mod', 'file3.tps'], \r\n    'combined.sqlite'\r\n)\r\n```\r\n\r\n### PHZ Files (Zip Archives)\r\nUse `PhzConverter` for .phz files (zip archives containing TopSpeed files):\r\n\r\n```python\r\nfrom converter.phz_converter import PhzConverter\r\n\r\nconverter = PhzConverter()\r\nresult = converter.convert_phz('input.phz', 'output.sqlite')\r\n```\r\n\r\n### Reverse Conversion (SQLite to TopSpeed)\r\nUse `ReverseConverter` to convert SQLite databases back to TopSpeed files:\r\n\r\n```python\r\nfrom converter.reverse_converter import ReverseConverter\r\n\r\nconverter = ReverseConverter()\r\nresult = converter.convert_sqlite_to_topspeed('input.sqlite', 'output_directory/')\r\n```\r\n\r\n## \ud83d\udd04 Multidimensional Array Handling\r\n\r\nPytopspeed Modernized includes advanced support for TopSpeed multidimensional arrays, automatically detecting and converting array fields to JSON format in SQLite.\r\n\r\n### Array Detection\r\n\r\nThe system automatically detects two types of arrays:\r\n\r\n1. **Single-Field Arrays**: Large fields containing multiple elements (e.g., 96-byte `DAT:PROD1` with 12 elements)\r\n2. **Multi-Field Arrays**: Multiple small fields forming an array (e.g., `CUM:PROD1`, `CUM:PROD2`, etc.)\r\n\r\n### Example: MONHIST Table\r\n\r\n```python\r\n# TopSpeed structure\r\nDAT:PROD1    # 96-byte field with 12 DOUBLE elements\r\nDAT:PROD2    # 96-byte field with 12 DOUBLE elements\r\nDAT:PROD3    # 96-byte field with 12 DOUBLE elements\r\n\r\n# SQLite result\r\nPROD1        # JSON: [1.5, 2.3, 0.0, null, ...]\r\nPROD2        # JSON: [0.8, 1.2, 0.0, null, ...]\r\nPROD3        # JSON: [2.1, 1.8, 0.0, null, ...]\r\n```\r\n\r\n### Data Type Preservation\r\n\r\n- **Zero vs NULL**: Distinguishes between actual zero values (`0.0`) and missing data (`null`)\r\n- **Boolean Arrays**: Converts `BYTE` arrays to proper boolean values (`true`/`false`)\r\n- **Numeric Arrays**: Preserves `DOUBLE`, `LONG`, `SHORT` precision\r\n- **String Arrays**: Maintains text encoding and length\r\n\r\n### Querying Array Data\r\n\r\n```sql\r\n-- Query array elements\r\nSELECT \r\n    LSE_ID,\r\n    json_extract(PROD1, '$[0]') as PROD1_Month1,\r\n    json_extract(PROD1, '$[1]') as PROD1_Month2\r\nFROM MONHIST;\r\n\r\n-- Filter by array content\r\nSELECT * FROM MONHIST \r\nWHERE json_extract(PROD1, '$[0]') > 100.0;\r\n\r\n-- Count non-null elements\r\nSELECT LSE_ID,\r\n       json_array_length(PROD1) as PROD1_Count\r\nFROM MONHIST;\r\n```\r\n\r\n## \ud83d\udee1\ufe0f Enterprise Resilience Features\r\n\r\n### Memory Management\r\n- **Configurable Memory Limits**: 200MB - 2GB based on database size\r\n- **Automatic Cleanup**: Garbage collection every 1,000 records\r\n- **Memory Monitoring**: Real-time memory usage tracking with psutil\r\n- **Streaming Processing**: Handle databases larger than available RAM\r\n\r\n### Adaptive Batch Sizing\r\n- **Dynamic Optimization**: Batch sizes automatically adjust based on table characteristics\r\n  - Small records (< 100B): 100-400 records per batch\r\n  - Medium records (100B-1KB): 25-100 records per batch\r\n  - Large records (1KB-5KB): 10-25 records per batch\r\n  - Very large records (> 5KB): 5-10 records per batch\r\n- **Complex Table Handling**: Smaller batches for tables with many fields\r\n- **Memory-Aware**: Batch sizes adapt to available memory\r\n\r\n### Predefined Configurations\r\n```python\r\nfrom converter.resilience_config import get_resilience_config\r\n\r\n# Small databases (< 10MB)\r\nconfig = get_resilience_config('small')  # 200MB limit, 200 batch size\r\n\r\n# Medium databases (10MB - 1GB)  \r\nconfig = get_resilience_config('medium')  # 500MB limit, 100 batch size\r\n\r\n# Large databases (1GB - 10GB)\r\nconfig = get_resilience_config('large')  # 1GB limit, 50 batch size, parallel processing\r\n\r\n# Enterprise databases (> 10GB)\r\nconfig = get_resilience_config('enterprise')  # 2GB limit, 25 batch size, full features\r\n```\r\n\r\n### Error Recovery\r\n- **Partial Conversion**: Save progress even if conversion is interrupted\r\n- **Graceful Degradation**: Continue processing despite individual record failures\r\n- **Detailed Logging**: Comprehensive error reporting for troubleshooting\r\n- **Resume Capability**: Restart from checkpoints for enterprise configurations\r\n\r\n### Performance Optimization\r\n- **SQLite Tuning**: WAL mode, optimized cache sizes, memory temp storage\r\n- **Parallel Processing**: Multi-threaded conversion for large databases\r\n- **Progress Tracking**: Real-time progress reporting for long operations\r\n- **Resource Monitoring**: Prevent system overload with configurable limits\r\n\r\n### Scalability\r\n- **Tested Limits**: Successfully handles databases with millions of records\r\n- **Large Tables**: FORCAST table with 4,370 records (2,528 bytes each)\r\n- **Memory Efficiency**: 60-80% reduction in memory usage with adaptive batching\r\n- **Enterprise Ready**: Production-tested with databases > 10GB\r\n\r\n## \ud83d\udee0\ufe0f Quick Start\r\n\r\n### Installation\r\n\r\n#### Option 1: Install from PyPI (Recommended)\r\n```bash\r\n# Install directly from PyPI\r\npip install pytopspeed-modernized\r\n```\r\n\r\n#### Option 2: Install from Source\r\n```bash\r\n# Clone the repository\r\ngit clone https://github.com/gregeasley/pytopspeed_modernized\r\ncd pytopspeed_modernized\r\n\r\n# Create conda environment (optional)\r\nconda create -n pytopspeed_modernized python=3.11\r\nconda activate pytopspeed_modernized\r\n\r\n# Install in development mode\r\npip install -e .\r\n```\r\n\r\n### Basic Usage\r\n\r\n```bash\r\n# Convert a single .phd file to SQLite\r\npython pytopspeed.py convert assets/TxWells.PHD output.sqlite\r\n\r\n# Convert multiple files to a combined database\r\npython pytopspeed.py convert assets/TxWells.PHD assets/TxWells.mod combined.sqlite\r\n\r\n# Convert a .phz file (zip archive)\r\npython pytopspeed.py convert assets/TxWells.phz output.sqlite\r\n\r\n# List contents of a .phz file\r\npython pytopspeed.py list assets/TxWells.phz\r\n\r\n# Convert SQLite back to TopSpeed files\r\npython pytopspeed.py reverse input.sqlite output_directory/\r\n```\r\n\r\n### Python API Examples\r\n\r\n#### Single TopSpeed File Conversion\r\n```python\r\nfrom converter.sqlite_converter import SqliteConverter\r\n\r\n# Convert a single .phd, .mod, or .tps file\r\nconverter = SqliteConverter()\r\nresults = converter.convert('input.phd', 'output.sqlite')\r\nprint(f\"Success: {results['success']}, Records: {results['total_records']}\")\r\n```\r\n\r\n#### Multiple Files to Combined Database\r\n```python\r\nfrom converter.sqlite_converter import SqliteConverter\r\n\r\n# Combine multiple TopSpeed files into one SQLite database\r\nconverter = SqliteConverter()\r\nresults = converter.convert_multiple(\r\n    ['file1.phd', 'file2.mod'], \r\n    'combined.sqlite'\r\n)\r\nprint(f\"Files processed: {results['files_processed']}\")\r\n```\r\n\r\n#### PHZ File Conversion (Zip Archives)\r\n```python\r\nfrom converter.phz_converter import PhzConverter\r\n\r\n# Convert .phz files (zip archives containing TopSpeed files)\r\nphz_converter = PhzConverter()\r\nresults = phz_converter.convert_phz('input.phz', 'output.sqlite')\r\nprint(f\"Extracted files: {results['extracted_files']}\")\r\n```\r\n\r\n#### Reverse Conversion (SQLite to TopSpeed)\r\n```python\r\nfrom converter.reverse_converter import ReverseConverter\r\n\r\n# Convert SQLite database back to TopSpeed files\r\nreverse_converter = ReverseConverter()\r\nresults = reverse_converter.convert_sqlite_to_topspeed(\r\n    'input.sqlite', \r\n    'output_directory/'\r\n)\r\nprint(f\"Generated files: {results['generated_files']}\")\r\n```\r\n\r\n## \ud83d\udea8 Common Issues and Solutions\r\n\r\n### Wrong Converter for File Type\r\n**Problem**: Using `SqliteConverter.convert()` with a `.phz` file\r\n```python\r\n# \u274c WRONG - This will fail\r\nconverter = SqliteConverter()\r\nresult = converter.convert('input.phz', 'output.sqlite')  # Error: 'TPS' object has no attribute 'tables'\r\n```\r\n\r\n**Solution**: Use `PhzConverter.convert_phz()` for `.phz` files\r\n```python\r\n# \u2705 CORRECT\r\nfrom converter.phz_converter import PhzConverter\r\nconverter = PhzConverter()\r\nresult = converter.convert_phz('input.phz', 'output.sqlite')\r\n```\r\n\r\n### File Not Found\r\n**Problem**: File path doesn't exist\r\n```python\r\n# \u274c WRONG - File doesn't exist\r\nresult = converter.convert('nonexistent.phd', 'output.sqlite')\r\n```\r\n\r\n**Solution**: Check file exists before conversion\r\n```python\r\nimport os\r\nif os.path.exists('input.phd'):\r\n    result = converter.convert('input.phd', 'output.sqlite')\r\nelse:\r\n    print(\"File not found!\")\r\n```\r\n\r\n### Import Errors\r\n**Problem**: Import path issues\r\n```python\r\n# \u274c WRONG - Incorrect import path\r\nfrom sqlite_converter import SqliteConverter  # ModuleNotFoundError\r\n```\r\n\r\n**Solution**: Use correct import path\r\n```python\r\n# \u2705 CORRECT\r\nfrom converter.sqlite_converter import SqliteConverter\r\n```\r\n\r\n## \ud83d\udcca Performance\r\n\r\nBased on testing with `TxWells.PHD` and `TxWells.mod`:\r\n\r\n- **Single file conversion**: ~1,300 records/second\r\n- **Combined conversion**: ~1,650 records/second  \r\n- **Reverse conversion**: ~50,000 records/second\r\n- **Memory efficient**: Configurable batch processing\r\n- **Progress tracking**: Real-time progress reporting\r\n\r\n## \ud83d\udd27 Command Line Interface\r\n\r\n### Convert Command\r\n\r\n```bash\r\npython pytopspeed.py convert [OPTIONS] INPUT_FILES... OUTPUT_FILE\r\n```\r\n\r\n**Options:**\r\n- `--batch-size BATCH_SIZE` - Number of records to process in each batch (default: 1000)\r\n- `-v, --verbose` - Enable verbose logging\r\n- `-q, --quiet` - Suppress progress output\r\n\r\n### Reverse Command\r\n\r\n```bash\r\npython pytopspeed.py reverse [OPTIONS] INPUT_FILE OUTPUT_DIRECTORY\r\n```\r\n\r\n### List Command\r\n\r\n```bash\r\npython pytopspeed.py list [OPTIONS] PHZ_FILE\r\n```\r\n\r\n## \ud83d\udcda Documentation\r\n\r\nComprehensive documentation is available in the `docs/` directory:\r\n\r\n- **[Installation Guide](docs/INSTALLATION.md)** - Detailed installation instructions\r\n- **[API Documentation](docs/API.md)** - Complete API reference\r\n- **[Troubleshooting Guide](docs/TROUBLESHOOTING.md)** - Common issues and solutions\r\n- **[Developer Documentation](docs/DEVELOPER.md)** - Development and contribution guidelines\r\n\r\n## \ud83e\uddea Testing\r\n\r\n### Comprehensive Test Suite\r\n\r\n```bash\r\n# Run all resilience tests\r\npython tests/run_resilience_tests.py\r\n\r\n# Run specific test types\r\npython tests/run_resilience_tests.py unit\r\npython tests/run_resilience_tests.py integration\r\npython tests/run_resilience_tests.py performance\r\n\r\n# Run with coverage\r\npython tests/run_resilience_tests.py -c\r\n\r\n# Run with pytest directly\r\npython -m pytest tests/unit/ -v\r\npython -m pytest tests/integration/ -v\r\npython -m pytest tests/performance/ --run-performance\r\n```\r\n\r\n### Test Coverage\r\n\r\n**Unit Tests (70+ tests):**\r\n- \u2705 **ResilienceEnhancer** - Memory management, adaptive batch sizing, data extraction\r\n- \u2705 **ResilienceConfig** - Configuration management and validation\r\n- \u2705 **SQLite Converter Enhancements** - Enhanced table definition parsing\r\n- \u2705 **Error Handling** - Robust error recovery and fallback mechanisms\r\n\r\n**Integration Tests (15+ tests):**\r\n- \u2705 **End-to-End Scenarios** - Complete conversion workflows\r\n- \u2705 **Configuration Selection** - Auto-detection based on database size\r\n- \u2705 **Component Integration** - Cross-component interaction validation\r\n- \u2705 **Performance Integration** - Resource usage under realistic conditions\r\n\r\n**Performance Tests (12+ tests):**\r\n- \u2705 **Memory Performance** - Memory usage patterns and cleanup efficiency\r\n- \u2705 **Processing Performance** - Speed and throughput under various loads\r\n- \u2705 **Scalability Performance** - Performance with increasing data sizes\r\n- \u2705 **Concurrent Performance** - Multi-threaded operation testing\r\n\r\n**Test Results:**\r\n- \u2705 **354 total tests** - All passing with 100% pass rate\r\n- \u2705 **95%+ code coverage** - Comprehensive test coverage\r\n- \u2705 **Performance benchmarks** - Validated scalability characteristics\r\n- \u2705 **Memory efficiency** - Tested memory usage patterns\r\n\r\n## \ud83d\udcd6 Examples\r\n\r\nWorking examples are available in the `examples/` directory:\r\n\r\n- **Basic conversion** - Single file conversion\r\n- **Combined conversion** - Multiple file conversion\r\n- **PHZ handling** - Zip archive processing\r\n- **Reverse conversion** - SQLite to TopSpeed\r\n- **Round-trip conversion** - Complete conversion cycle\r\n- **Custom progress tracking** - Advanced progress monitoring\r\n- **Error handling** - Comprehensive error handling patterns\r\n\r\n## \ud83c\udfd7\ufe0f Architecture\r\n\r\n```\r\nTopSpeed Files \u2192 Parser \u2192 Schema Mapper \u2192 SQLite Converter \u2192 SQLite Database\r\n     \u2193              \u2193           \u2193              \u2193\r\n   .phd/.mod    Modernized   Type Mapping   Data Migration\r\n   .tps/.phz    pytopspeed   Field Names    Batch Processing\r\n```\r\n\r\n### Key Components\r\n\r\n- **TopSpeed Parser** - Modernized parser for reading TopSpeed files\r\n- **Schema Mapper** - Maps TopSpeed schemas to SQLite\r\n- **SQLite Converter** - Handles data migration and conversion\r\n- **PHZ Converter** - Processes zip archives containing TopSpeed files\r\n- **Reverse Converter** - Converts SQLite back to TopSpeed files\r\n- **CLI Interface** - Command-line tools for easy usage\r\n\r\n## \ud83d\udd04 Data Type Conversion\r\n\r\n| TopSpeed Type | SQLite Type | Notes |\r\n|---------------|-------------|-------|\r\n| BYTE | INTEGER | 8-bit unsigned integer |\r\n| SHORT | INTEGER | 16-bit signed integer |\r\n| LONG | INTEGER | 32-bit signed integer |\r\n| DATE | TEXT | Format: YYYY-MM-DD |\r\n| TIME | TEXT | Format: HH:MM:SS |\r\n| STRING | TEXT | Variable length text |\r\n| DECIMAL | REAL | Floating point number |\r\n| MEMO | BLOB | Binary large object |\r\n| BLOB | BLOB | Binary large object |\r\n\r\n## \ud83c\udfaf Key Features\r\n\r\n### Table Name Prefixing\r\n\r\nWhen converting multiple files, tables are automatically prefixed to avoid conflicts:\r\n\r\n- **.phd files** \u2192 `phd_` prefix (e.g., `phd_OWNER`, `phd_CLASS`)\r\n- **.mod files** \u2192 `mod_` prefix (e.g., `mod_DEPRECIATION`, `mod_MODID`)\r\n- **.tps files** \u2192 `tps_` prefix\r\n- **Other files** \u2192 `file_N_` prefix\r\n\r\n### Column Name Sanitization\r\n\r\nColumn names are automatically sanitized for SQLite compatibility:\r\n\r\n- **Prefix removal**: `TIT:PROJ_DESCR` \u2192 `PROJ_DESCR`\r\n- **Special characters**: `.` \u2192 `_`\r\n- **Numeric prefixes**: `123FIELD` \u2192 `_123FIELD`\r\n- **Reserved words**: `ORDER` \u2192 `ORDER_TABLE`\r\n\r\n### Error Handling\r\n\r\n- **Graceful degradation** - Continue processing despite individual table errors\r\n- **Detailed logging** - Comprehensive error reporting and debugging information\r\n- **Data preservation** - Ensure data integrity even with parsing issues\r\n- **Recovery mechanisms** - Automatic handling of common issues\r\n\r\n## \ud83e\udd1d Contributing\r\n\r\nWe welcome contributions! Please see our [Developer Documentation](docs/DEVELOPER.md) for:\r\n\r\n- Development setup instructions\r\n- Code style guidelines\r\n- Testing requirements\r\n- Contribution process\r\n\r\n### Development Setup\r\n\r\n```bash\r\n# Clone and setup development environment\r\ngit clone https://github.com/gregeasley/pytopspeed_modernized\r\ncd pytopspeed_modernized\r\nconda create -n pytopspeed_modernized_dev python=3.11\r\nconda activate pytopspeed_modernized_dev\r\npip install -e .[dev]\r\n\r\n# Run tests\r\npython -m pytest tests/ -v\r\n```\r\n\r\n## \ud83d\udcc4 License\r\n\r\nThis project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.\r\n\r\n## \ud83d\ude4f Acknowledgments\r\n\r\n- Based on the original [pytopspeed library](https://github.com/dylangiles/pytopspeed/)\r\n- Modernized for Python 3.11 and construct 2.10+\r\n- Enhanced with SQLite conversion and reverse conversion capabilities\r\n- Comprehensive testing and documentation\r\n\r\n## \ud83d\udcde Support\r\n\r\n- **Documentation**: See the `docs/` directory for comprehensive guides\r\n- **Examples**: Check the `examples/` directory for working code\r\n- **Issues**: Open an issue in the project repository\r\n- **Discussions**: Use the project's discussion forum for questions\r\n\r\n---\r\n\r\n**Ready to convert your TopSpeed files?** Start with the [Installation Guide](docs/INSTALLATION.md) and try the [examples](examples/)!\r\n",
    "bugtrack_url": null,
    "license": "MIT",
    "summary": "Modernized pytopspeed library for converting TopSpeed database files to SQLite",
    "version": "1.1.3",
    "project_urls": {
        "Bug Tracker": "https://github.com/gregeasley/pytopspeed_modernized/issues",
        "Documentation": "https://github.com/gregeasley/pytopspeed_modernized/blob/master/docs/README.md",
        "Homepage": "https://github.com/gregeasley/pytopspeed_modernized",
        "Repository": "https://github.com/gregeasley/pytopspeed_modernized"
    },
    "split_keywords": [
        "topspeed",
        " clarion",
        " database",
        " sqlite",
        " conversion",
        " migration",
        " legacy"
    ],
    "urls": [
        {
            "comment_text": null,
            "digests": {
                "blake2b_256": "98dbd8c94188551b2da232cb4a60b6bcd9bac61b54da81390553379b12deb49c",
                "md5": "14211c9be1674533085f13f614b0eca9",
                "sha256": "84039fbe87809fe852d23bea4da1a0c0c2f77dc42848c18fc0f60fa5f092e598"
            },
            "downloads": -1,
            "filename": "pytopspeed_modernized-1.1.3-py2.py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "14211c9be1674533085f13f614b0eca9",
            "packagetype": "bdist_wheel",
            "python_version": "py2.py3",
            "requires_python": ">=3.8",
            "size": 85796,
            "upload_time": "2025-09-12T22:40:37",
            "upload_time_iso_8601": "2025-09-12T22:40:37.961611Z",
            "url": "https://files.pythonhosted.org/packages/98/db/d8c94188551b2da232cb4a60b6bcd9bac61b54da81390553379b12deb49c/pytopspeed_modernized-1.1.3-py2.py3-none-any.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": null,
            "digests": {
                "blake2b_256": "86e8f73cc96743f2d6eea24e1ac04fd80c34b11841e62660bb0136ef35948c75",
                "md5": "cf34452fec64b5ac4eb725739793fe21",
                "sha256": "d6b9d63b4ebe846b7bbd07d2b5de111d42bc4a630feca3834becd42c67691f06"
            },
            "downloads": -1,
            "filename": "pytopspeed_modernized-1.1.3.tar.gz",
            "has_sig": false,
            "md5_digest": "cf34452fec64b5ac4eb725739793fe21",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": ">=3.8",
            "size": 727522,
            "upload_time": "2025-09-12T22:40:39",
            "upload_time_iso_8601": "2025-09-12T22:40:39.415245Z",
            "url": "https://files.pythonhosted.org/packages/86/e8/f73cc96743f2d6eea24e1ac04fd80c34b11841e62660bb0136ef35948c75/pytopspeed_modernized-1.1.3.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2025-09-12 22:40:39",
    "github": true,
    "gitlab": false,
    "bitbucket": false,
    "codeberg": false,
    "github_user": "gregeasley",
    "github_project": "pytopspeed_modernized",
    "travis_ci": false,
    "coveralls": false,
    "github_actions": true,
    "requirements": [
        {
            "name": "construct",
            "specs": [
                [
                    ">=",
                    "2.10.0"
                ]
            ]
        },
        {
            "name": "click",
            "specs": [
                [
                    ">=",
                    "8.0.0"
                ]
            ]
        },
        {
            "name": "pytest",
            "specs": [
                [
                    ">=",
                    "7.0.0"
                ]
            ]
        },
        {
            "name": "pytest-cov",
            "specs": [
                [
                    ">=",
                    "4.0.0"
                ]
            ]
        },
        {
            "name": "pandas",
            "specs": [
                [
                    ">=",
                    "1.5.0"
                ]
            ]
        }
    ],
    "lcname": "pytopspeed-modernized"
}