# PyOghma_ML
PyOghma_ML is a Python-based machine learning pipeline designed to integrate seamlessly with OghmaNano, a powerful platform for simulating and analyzing optoelectronic devices. This library provides tools for training machine learning models, generating predictions, and producing detailed reports based on experimental and simulated data.
The primary goal of PyOghma_ML is to streamline the process of leveraging machine learning for device characterization and prediction. By automating tasks such as data preprocessing, model training, and result generation, PyOghma_ML enables researchers to focus on interpreting results and advancing their research.
Key features of PyOghma_ML include:
- **Training Machine Learning Models**: Train deep learning models using simulated data to predict device characteristics.
- **Prediction and Analysis**: Use trained models to predict device performance based on experimental data.
- **Customizable Model Settings**: Fine-tune model architectures, learning rates, and other hyperparameters to suit specific datasets.
- **Report Generation**: Automatically generate LaTeX-based reports and CSV files summarizing predictions and model performance.
- **Integration with OghmaNano**: Designed to work directly with OghmaNano's data formats, ensuring compatibility and ease of use.
Whether you are a researcher working on photovoltaics, LEDs, or other optoelectronic devices, PyOghma_ML provides a robust and flexible framework for incorporating machine learning into your workflow.
---
## Installation
Before installing PyOghma_ML, ensure that OghmaNano is installed. You can download OghmaNano from [oghma-nano.com](https://www.oghma-nano.com).
To install PyOghma_ML, use pip:
```bash
# For production use
python -m pip install PyOghma-ML
# For development (from source)
git clone https://github.com/CaiWilliams/PyOghma_ML.git
cd PyOghma_ML
pip install -e .
# With optional dependencies
pip install PyOghma-ML[dev,docs,gpu]
```
> **Modern Installation**: PyOghma_ML uses modern Python packaging with `pyproject.toml`. For pip versions >= 21.0, all dependencies are automatically handled.
---
## Usage and Examples
This guide assumes familiarity with OghmaNano and its machine learning features. If you are new to OghmaNano, refer to the [OghmaNano User Manual](https://www.oghma-nano.com/docs/man/understanding_oghma_nano.pdf) before proceeding.
For detailed API documentation, visit the [PyOghma_ML Documentation](https://caiwilliams.github.io/PyOghma_ML/PyOghma_ML/index.html).
---
### Machine Learning Inputs
Machine learning networks for training or prediction are defined through the machine learning window in OghmaNano. Once the dataset is generated and vectors are built and normalized, the required files for the pipeline are created.
#### Required Files:
- **`nets.json`**: Defines the networks, their inputs, and outputs.
- **`vectors.csv`**: Contains normalized parameter values for each device.
- **`min_max.csv`**: Specifies the minimum, maximum, and range for each parameter.
> **Note**: Only `nets.json` can be modified after dataset and vector generation. Exercise caution when editing it outside OghmaNano.
---
### Training with `Training.py` and `SubTraining.py`
#### `Training.py`
This script orchestrates the training of multiple neural network models across different device configurations and network architectures. It features improved error handling, progress reporting, and organized configuration management.
**Key Features:**
- Automatic model existence checking to avoid retraining
- Enhanced error handling with detailed status reporting
- Progress tracking with clear status indicators (✓/✗)
- Support for multiple material systems and network types
- Configurable training parameters for different scenarios
```python
# Example: Basic usage of the improved Training.py
import os
from Training import process_dirs
# Configuration
sim_dir_og = os.path.join('/', 'media', 'cai', 'Big', 'Simulated_Data', 'opkm', 'Networks')
# Train different network types with enhanced error handling
print("Starting training for Point networks...")
process_dirs(['Point_pm6y12'], sim_dir_og, 'Point', 10e6)
print("Starting training for Difference networks...")
process_dirs(['Difference_pm6y12'], sim_dir_og, 'Difference', 10e6)
print("Starting training for Residual networks...")
process_dirs(['Residual_pm6y12'], sim_dir_og, 'Residual', 10e6)
```
**Improved Features in the Updated Script:**
- **Error Handling**: Comprehensive try/catch blocks with detailed error reporting
- **Progress Tracking**: Clear status messages with ✓/✗ indicators for success/failure
- **File Validation**: Checks for directory existence and valid JSON configurations
- **Subprocess Management**: Proper error handling for SubTraining.py execution
- **Organized Configuration**: Well-commented sections for different training scenarios
```python
# Training.py - Enhanced version with comprehensive error handling
"""
Training Script for PyOghma ML Models
This script orchestrates the training of multiple neural network models across
different device configurations and network architectures.
"""
def process_dirs(dirs, sim_dir_og, network_type, train_size, extra_arg="1"):
"""
Process training directories and train missing models.
Args:
dirs (list[str]): List of directory names to process
sim_dir_og (str): Base simulation directory path
network_type (str): Network architecture type
train_size (int): Number of training samples
extra_arg (str): Additional argument for SubTraining.py
"""
for d in dirs:
sim_dir = os.path.join(sim_dir_og, d, 'training')
print(f'Processing simulation directory: {sim_dir}')
# Enhanced error handling and validation
if not os.path.exists(sim_dir):
print(f"Warning: Directory {sim_dir} does not exist. Skipping...")
continue
# Process networks with detailed progress reporting
# (Full implementation available in the actual script)
# Usage with improved configuration management
if __name__ == "__main__":
sim_dir_og = os.path.join('/', 'media', 'cai', 'Big', 'Simulated_Data', 'opkm', 'Networks')
print("Starting training for Point networks...")
process_dirs(['Point_pm6y12'], sim_dir_og, 'Point', 10e6)
print("Training completed!")
```
#### `SubTraining.py`
This script handles individual network training with optimized configurations for different network types. The updated version includes enhanced documentation, improved argument parsing, and better model configuration management.
**Key Features:**
- **Network-Specific Optimization**: Tailored hyperparameters for each network type
- **Flexible Configuration**: Command-line arguments for all training parameters
- **Model Settings**: Optimized layer architectures and learning rates
- **Error Handling**: Comprehensive error reporting and validation
```python
# Example usage (called automatically by Training.py)
python SubTraining.py /path/to/sim_dir 0 Residual 10000000 4
# Arguments:
# 1. sim_dir: Simulation directory path
# 2. idx: Network index to train
# 3. type: Network type (Residual/Difference/Point)
# 4. limit: Training data limit
# 5. inputs: Number of input features
```
**Network Type Configurations:**
| Network Type | Layers | Nodes | Learning Rate | Batch Size | Epochs |
|-------------|---------|-------|---------------|------------|---------|
| Residual | 4 | 128 | 8e-5 | 16384 | 1024 |
| Difference | 4 | 256 | 1e-5 | 16384 | 1024 |
| Point | 4 | 256 | 1e-4 | 1024 | 4096 |
```python
# SubTraining.py - Network-specific training with optimized configurations
"""
Individual Network Training Script
Handles training of specific networks with optimized hyperparameters
for different network types (Residual, Difference, Point).
"""
from argparse import ArgumentParser
import PyOghma_ML_Private as OML
from PyOghma_ML_Private.Training import Model_Settings
def configure_model_settings(network_type, limit, inputs):
"""Configure model settings based on network type."""
if network_type == 'Residual':
return Model_Settings(
initializer='he_normal',
activation='silu',
layer_nodes=[128, 128, 128, 128],
dropout=[0.05, 0.05, 0.05, 0.05],
inital_learning_rate=8e-5,
batch_size=16384,
epochs=1024,
patience=16,
decay_rate=6e-1,
permutations_limit=int(limit),
inputs=int(inputs)
)
# Additional configurations for Difference and Point networks...
def main():
parser = ArgumentParser(description="Train individual neural networks")
parser.add_argument("sim_dir", help="Simulation directory path")
parser.add_argument("idx", type=int, help="Network index")
parser.add_argument("type", help="Network type")
parser.add_argument("limit", help="Training data limit")
parser.add_argument("inputs", help="Number of inputs")
args = parser.parse_args()
# Configure and train with enhanced error handling
m = configure_model_settings(args.type, args.limit, args.inputs)
A = OML.Networks.initialise(args.sim_dir, network_type=args.type, model_settings=m)
A.train(args.idx)
if __name__ == "__main__":
main()
```
---
### Predictions with `Predict.py` and `SubPredict.py`
#### `Predict.py`
This script orchestrates the prediction pipeline for experimental data using trained neural network models. The updated version features improved error handling, better device mapping, and enhanced progress reporting.
**Key Features:**
- **Multi-Device Support**: Handles various device types with automatic material mapping
- **Experiment Flexibility**: Supports multiple experiment types (J-V, Suns-Voc, etc.)
- **Enhanced Error Handling**: Comprehensive error checking and detailed status reporting
- **Progress Tracking**: Clear progress indicators and completion status
- **Modular Design**: Well-organized helper functions for maintainability
```python
# Example: Using the improved Predict.py
from Predict import map_device_to_material, get_experimental, get_illumination_files
# Configuration
device_types = ["PM6Y12_ZnO", "PM6BTPeC9"] # Multiple device types supported
experiment_type = "light_dark_jv" # or "sunsvoc"
# Automatic device-to-material mapping
for device_type in device_types:
net_mat = map_device_to_material(device_type)
print(f"Device {device_type} mapped to material: {net_mat}")
# Get experimental files with error handling
try:
exp_files = get_experimental(device_type, base_exp_dir, experiment_type)
print(f"Found {len(exp_files)} experimental files")
except FileNotFoundError as e:
print(f"Error: {e}")
```
**Improved Features:**
- **Helper Functions**: Modular design with `get_experimental()`, `map_device_to_material()`, and `get_illumination_files()`
- **Error Handling**: File existence checking and detailed error reporting
- **Status Reporting**: Progress indicators (✓/✗) for each processing step
- **Flexible Configuration**: Easy-to-modify device types and experiment settings
```python
# filepath: /media/cai/Big/PyOghma_ml/PyOghma_ML/Predict.py
import os
import subprocess
from natsort import natsorted
import json
def get_experimental(device_type, directory, exp):
"""
Get sorted list of experimental data files for a given device type and experiment.
"""
exp_dir = os.path.join(directory, device_type, exp)
files = os.listdir(exp_dir)
if exp == "light_dark_jv":
ending = 'am15.dat'
elif exp == "sunsvoc":
ending = '0000000uIllu_IV.dat'
else:
ending = ''
files = [os.path.join(exp_dir, f) for f in files if f.endswith(ending)]
files = natsorted(files)
return files
if __name__ == "__main__":
# Define simulation and experimental data directories
sim_dir_og = os.path.join('/', 'media', 'cai', 'Big', 'Simulated_Data', 'opkm', 'Networks_Extended')
base_exp_dir = os.path.join('/', 'media', 'cai', 'Big', 'Experimental_Data', 'Data_From_Chen')
device_type = ["PM6Y12_ZnO"] # Can include "PM6BTPeC9", "W108-2", etc.
exp = "light_dark_jv" # or "sunsvoc"
res_dir = os.path.join(os.getcwd(), 'Results')
for idx in device_type:
# Map device type to network material
match idx:
case 'PM6BTPeC9'| 'W108-2':
net_mat = 'pm6ec9'
case 'PM6Y12_ZnO':
net_mat = 'pm6y12'
case 'PM6Y12_PEIZn':
net_mat = 'pm6y12_peizn'
case _:
raise ValueError(f"Unknown device type: {idx}")
files = get_experimental(idx, base_exp_dir, exp)
for jdx in files:
name = os.path.basename(jdx).split('.')[0]
# List and filter simulation directories for the current device type
sim_dirs = [kdx for kdx in os.listdir(sim_dir_og) if kdx.startswith(('Residual_')) and net_mat in kdx]
sim_dirs = natsorted(sim_dirs)
sim_dirs = [os.path.join(sim_dir_og, kdx) for kdx in sim_dirs]
for kdx in sim_dirs:
training = os.path.join(kdx, 'training')
conversion = os.path.join(kdx, 'conversion')
network_type = os.path.basename(kdx).split('_')[0]
extra_info = os.path.basename(kdx).split('_')[-1]
# Check if result already exists before processing
out_file = os.path.join(res_dir, network_type, name + '.csv')
if os.path.isfile(out_file):
print(f"File {name}.csv already exists. Skipping processing.")
continue
# Run SubPredict with appropriate arguments
subprocess.run(["python", "SubPredict.py", training, conversion, jdx, res_dir, network_type, "JV", "Deibel", extra_info])
```
#### `SubPredict.py`
This script handles individual prediction tasks using trained neural network models. The updated version features enhanced documentation, improved error handling, and better argument parsing for more reliable predictions.
**Key Features:**
- **Enhanced Documentation**: Comprehensive argument descriptions and usage examples
- **Robust Error Handling**: Try/catch blocks with detailed error reporting
- **Progress Tracking**: Status messages for each prediction step
- **Flexible Output**: Configurable output directory structure
- **Network Type Support**: Handles different prediction methods for various network types
```python
# Example usage (automatically called by Predict.py)
python SubPredict.py /path/to/sim_dir /path/to/abs_dir /path/to/exp.dat /path/to/results Residual JV Deibel
# Enhanced argument parsing with detailed help
python SubPredict.py --help
```
**Improved Features:**
- **Comprehensive Error Handling**: Detailed error messages for debugging
- **Progress Reporting**: Step-by-step status updates with ✓/✗ indicators
- **Enhanced Documentation**: Built-in help text and usage examples
- **Output Organization**: Automatic directory creation and file management
- **Network-Specific Logic**: Optimized prediction methods for different network types
**Prediction Workflow:**
1. **Data Loading**: Load and standardize experimental data with validation
2. **Network Initialization**: Initialize appropriate network type with error checking
3. **Prediction Generation**: Execute predictions based on network architecture
4. **Report Building**: Generate comprehensive output reports
5. **File Management**: Save results with organized directory structure
```python
# SubPredict.py - Enhanced individual prediction script
"""
Individual Prediction Script
Performs predictions on single experimental files with comprehensive
error handling and progress reporting.
"""
def main():
parser = ArgumentParser(description="Individual prediction script")
# Enhanced argument parsing with detailed help
parser.add_argument("sim_dir", help="Simulation directory path")
parser.add_argument("abs_dir", help="Absolute data directory")
parser.add_argument("exp_dir", help="Experimental data file")
# ... additional arguments
args = parser.parse_args()
try:
print(f"Starting prediction for: {os.path.basename(args.exp_dir)}")
# Load and standardize experimental data
Exp = OML.Input.experiment(args.exp_dir, args.exp_typ, args.Source_Lab)
Exp.standardise_inputs()
print(f"✓ Loaded experimental data: {len(Exp.x)} data points")
# Initialize network and generate predictions
N = OML.Networks.initialise(args.sim_dir, network_type=args.net_typ)
if args.net_typ in ['Difference', 'Residual']:
N.predict(args.abs_dir, Exp)
else:
N.predict(Exp)
print("✓ Predictions generated successfully")
# Build and save report
O = OML.Output(N, Exp, abs_dir=args.abs_dir)
O.build_report()
O.save_report(output_path)
print(f"✓ Results saved to: {output_path}.csv")
except Exception as e:
print(f"✗ Error during prediction: {e}")
raise
if __name__ == "__main__":
main()
```
---
### Supported Device Types and Materials
The current implementation supports the following device types and their corresponding network materials:
| Device Type | Network Material | Description |
|-------------------|------------------|--------------------------------|
| PM6BTPeC9 | pm6ec9 | PM6:BTP-eC9 based devices |
| W108-2 | pm6ec9 | W108-2 devices |
| PM6Y12_ZnO | pm6y12 | PM6:Y12 with ZnO devices |
| PM6Y12_PEIZn | pm6y12_peizn | PM6:Y12 with PEI-Zn devices |
### Supported Experiments
| Experiment Type | File Extension | Description |
|-------------------|---------------------|--------------------------------|
| light_dark_jv | am15.dat | Light and dark JV measurements |
| sunsvoc | 0000000uIllu_IV.dat| Suns-Voc measurements |
### Supported Laboratories
| Laboratory | Label |
|------------------|---------|
| OPKM | Deibel |
| HSP | Shoaee |
| Herzig Group | Herzig |
| OghmaNano | Oghma |
> **Note**: For new laboratories or characteristics, custom input functions may need to be written. Contact support for assistance.
---
## Recent Improvements (v0.2.0)
PyOghma_ML has been significantly enhanced with improved documentation, error handling, and code organization:
### **Enhanced Documentation**
- **Comprehensive Module Docstrings**: All scripts now include detailed module-level documentation
- **Function Documentation**: Improved docstrings with parameter types, return values, and examples
- **Usage Examples**: Clear examples and configuration guidance throughout
### **Improved Error Handling**
- **Robust Validation**: File existence checking and directory validation
- **Detailed Error Messages**: Clear error reporting with troubleshooting information
- **Graceful Failure**: Scripts continue processing when individual tasks fail
- **Status Indicators**: Visual progress tracking with ✓/✗ status markers
### **Code Structure Enhancements**
- **Modular Design**: Refactored scripts into well-organized functions
- **Helper Functions**: Separated concerns for better maintainability
- **Configuration Management**: Organized and documented configuration sections
- **Modern Python**: Updated to use modern Python packaging with `pyproject.toml`
### **User Experience Improvements**
- **Progress Tracking**: Real-time status updates during training and prediction
- **Better Logging**: Informative console output with clear progress indicators
- **Organized Output**: Structured result directories and file naming
- **Enhanced CLI**: Improved command-line argument parsing with detailed help
---
### Example Workflow
#### 1. **Training Networks** (Enhanced with Error Handling)
```python
# Use the improved Training.py with automatic error handling
from Training import process_dirs
# The script now provides detailed status reporting
print("=== Starting Neural Network Training ===")
# Train different network types with enhanced monitoring
configurations = [
(['Point_pm6y12'], 'Point', 10e6),
(['Difference_pm6y12'], 'Difference', 10e6),
(['Residual_pm6y12'], 'Residual', 10e6)
]
sim_dir_og = '/path/to/simulation/data'
for dirs, network_type, train_size in configurations:
print(f"\n--- Training {network_type} Networks ---")
try:
process_dirs(dirs, sim_dir_og, network_type, train_size)
print(f"✓ {network_type} training completed successfully")
except Exception as e:
print(f"✗ {network_type} training failed: {e}")
```
#### 2. **Generate Predictions** (Improved Pipeline)
```python
# Use the enhanced Predict.py with better device mapping
from Predict import map_device_to_material, get_experimental
# Configuration with multiple device types
device_types = ["PM6Y12_ZnO", "PM6BTPeC9", "W108-2"]
experiment_type = "light_dark_jv"
print("=== Starting Prediction Pipeline ===")
for device_type in device_types:
print(f"\n--- Processing {device_type} ---")
try:
# Automatic device-to-material mapping
net_mat = map_device_to_material(device_type)
print(f"✓ Mapped {device_type} to material: {net_mat}")
# Load experimental files with validation
exp_files = get_experimental(device_type, base_exp_dir, experiment_type)
print(f"✓ Found {len(exp_files)} experimental files")
# Process predictions with progress tracking
# (Detailed processing handled automatically)
except Exception as e:
print(f"✗ Error processing {device_type}: {e}")
continue
print("✓ Prediction pipeline completed")
```
#### 3. **Network Type Optimization**
| Network Type | Architecture | Use Case | Key Features |
|-------------|--------------|----------|--------------|
| **Residual** | 4×128 nodes | Residual prediction | Fast training, efficient memory |
| **Difference** | 4×256 nodes | Differential analysis | High accuracy, robust to noise |
| **Point** | 4×256 nodes | Direct prediction | Minimal data requirements |
#### 4. **Testing Data Loading**
```python
# Use the enhanced LoadingFileTest.py for validation
from LoadingFileTest import main
# Test data loading with multiple optical densities
print("=== Testing Data Loading Functionality ===")
# The script provides comprehensive testing:
# - Multiple OD values (0.0, 1.0, 2.0, 3.0)
# - Error handling for missing files
# - Visual validation with enhanced plots
# - Detailed progress reporting
main() # Automatically runs all tests with status indicators
```
---
## Requirements and Dependencies
PyOghma_ML requires Python 3.9+ and includes the following key dependencies:
### **Core Dependencies**
- **Scientific Computing**: NumPy ≥1.24.0, SciPy ≥1.11.0, Pandas ≥2.1.0
- **Machine Learning**: TensorFlow ≥2.15.0, Keras ≥3.0.0, Keras-Tuner ≥1.4.0
- **Visualization**: Matplotlib ≥3.7.0
- **Data Processing**: h5py ≥3.9.0, ujson ≥5.8.0, natsort ≥8.4.0
- **Templating**: Jinja2 ≥3.1.0
### **Optional Dependencies**
```bash
# Development tools
pip install PyOghma-ML[dev] # pytest, black, flake8, mypy
# Documentation generation
pip install PyOghma-ML[docs] # sphinx, themes, parsers
# GPU acceleration
pip install PyOghma-ML[gpu] # tensorflow with CUDA support
```
### **System Requirements**
- **Python**: 3.9, 3.10, 3.11, or 3.12
- **Operating System**: Cross-platform (Windows, macOS, Linux)
- **Memory**: Minimum 8GB RAM (16GB recommended for large datasets)
- **Storage**: 1GB+ for models and results
---
## Authors
- [@CaiWilliams](https://github.com/CaiWilliams)
Raw data
{
"_id": null,
"home_page": null,
"name": "PyOghma-ML",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.9",
"maintainer_email": null,
"keywords": "machine learning, neural networks, oghma, photovoltaics, solar cells",
"author": null,
"author_email": "Cai Williams <cai.williams@physik.tu-chemnitz.de>",
"download_url": "https://files.pythonhosted.org/packages/7a/07/05e3f7e5c696d4b934440c88193587b5dcb65db4eecc874b8efe3d222c5a/pyoghma_ml-0.2.0.tar.gz",
"platform": null,
"description": "# PyOghma_ML\n\nPyOghma_ML is a Python-based machine learning pipeline designed to integrate seamlessly with OghmaNano, a powerful platform for simulating and analyzing optoelectronic devices. This library provides tools for training machine learning models, generating predictions, and producing detailed reports based on experimental and simulated data. \n\nThe primary goal of PyOghma_ML is to streamline the process of leveraging machine learning for device characterization and prediction. By automating tasks such as data preprocessing, model training, and result generation, PyOghma_ML enables researchers to focus on interpreting results and advancing their research.\n\nKey features of PyOghma_ML include:\n- **Training Machine Learning Models**: Train deep learning models using simulated data to predict device characteristics.\n- **Prediction and Analysis**: Use trained models to predict device performance based on experimental data.\n- **Customizable Model Settings**: Fine-tune model architectures, learning rates, and other hyperparameters to suit specific datasets.\n- **Report Generation**: Automatically generate LaTeX-based reports and CSV files summarizing predictions and model performance.\n- **Integration with OghmaNano**: Designed to work directly with OghmaNano's data formats, ensuring compatibility and ease of use.\n\nWhether you are a researcher working on photovoltaics, LEDs, or other optoelectronic devices, PyOghma_ML provides a robust and flexible framework for incorporating machine learning into your workflow.\n\n---\n\n## Installation\n\nBefore installing PyOghma_ML, ensure that OghmaNano is installed. You can download OghmaNano from [oghma-nano.com](https://www.oghma-nano.com).\n\nTo install PyOghma_ML, use pip:\n\n```bash\n# For production use\npython -m pip install PyOghma-ML\n\n# For development (from source)\ngit clone https://github.com/CaiWilliams/PyOghma_ML.git\ncd PyOghma_ML\npip install -e .\n\n# With optional dependencies\npip install PyOghma-ML[dev,docs,gpu]\n```\n\n> **Modern Installation**: PyOghma_ML uses modern Python packaging with `pyproject.toml`. For pip versions >= 21.0, all dependencies are automatically handled.\n\n---\n\n## Usage and Examples\n\nThis guide assumes familiarity with OghmaNano and its machine learning features. If you are new to OghmaNano, refer to the [OghmaNano User Manual](https://www.oghma-nano.com/docs/man/understanding_oghma_nano.pdf) before proceeding.\n\nFor detailed API documentation, visit the [PyOghma_ML Documentation](https://caiwilliams.github.io/PyOghma_ML/PyOghma_ML/index.html).\n\n---\n\n### Machine Learning Inputs\n\nMachine learning networks for training or prediction are defined through the machine learning window in OghmaNano. Once the dataset is generated and vectors are built and normalized, the required files for the pipeline are created.\n\n#### Required Files:\n- **`nets.json`**: Defines the networks, their inputs, and outputs.\n- **`vectors.csv`**: Contains normalized parameter values for each device.\n- **`min_max.csv`**: Specifies the minimum, maximum, and range for each parameter.\n\n> **Note**: Only `nets.json` can be modified after dataset and vector generation. Exercise caution when editing it outside OghmaNano.\n\n---\n\n### Training with `Training.py` and `SubTraining.py`\n\n#### `Training.py`\n\nThis script orchestrates the training of multiple neural network models across different device configurations and network architectures. It features improved error handling, progress reporting, and organized configuration management.\n\n**Key Features:**\n- Automatic model existence checking to avoid retraining\n- Enhanced error handling with detailed status reporting\n- Progress tracking with clear status indicators (\u2713/\u2717)\n- Support for multiple material systems and network types\n- Configurable training parameters for different scenarios\n\n```python\n# Example: Basic usage of the improved Training.py\nimport os\nfrom Training import process_dirs\n\n# Configuration\nsim_dir_og = os.path.join('/', 'media', 'cai', 'Big', 'Simulated_Data', 'opkm', 'Networks')\n\n# Train different network types with enhanced error handling\nprint(\"Starting training for Point networks...\")\nprocess_dirs(['Point_pm6y12'], sim_dir_og, 'Point', 10e6)\n\nprint(\"Starting training for Difference networks...\")\nprocess_dirs(['Difference_pm6y12'], sim_dir_og, 'Difference', 10e6)\n\nprint(\"Starting training for Residual networks...\")\nprocess_dirs(['Residual_pm6y12'], sim_dir_og, 'Residual', 10e6)\n```\n\n**Improved Features in the Updated Script:**\n- **Error Handling**: Comprehensive try/catch blocks with detailed error reporting\n- **Progress Tracking**: Clear status messages with \u2713/\u2717 indicators for success/failure\n- **File Validation**: Checks for directory existence and valid JSON configurations\n- **Subprocess Management**: Proper error handling for SubTraining.py execution\n- **Organized Configuration**: Well-commented sections for different training scenarios\n\n```python\n# Training.py - Enhanced version with comprehensive error handling\n\"\"\"\nTraining Script for PyOghma ML Models\n\nThis script orchestrates the training of multiple neural network models across\ndifferent device configurations and network architectures.\n\"\"\"\n\ndef process_dirs(dirs, sim_dir_og, network_type, train_size, extra_arg=\"1\"):\n \"\"\"\n Process training directories and train missing models.\n \n Args:\n dirs (list[str]): List of directory names to process\n sim_dir_og (str): Base simulation directory path \n network_type (str): Network architecture type\n train_size (int): Number of training samples\n extra_arg (str): Additional argument for SubTraining.py\n \"\"\"\n for d in dirs:\n sim_dir = os.path.join(sim_dir_og, d, 'training')\n print(f'Processing simulation directory: {sim_dir}')\n \n # Enhanced error handling and validation\n if not os.path.exists(sim_dir):\n print(f\"Warning: Directory {sim_dir} does not exist. Skipping...\")\n continue\n \n # Process networks with detailed progress reporting\n # (Full implementation available in the actual script)\n\n# Usage with improved configuration management\nif __name__ == \"__main__\":\n sim_dir_og = os.path.join('/', 'media', 'cai', 'Big', 'Simulated_Data', 'opkm', 'Networks')\n \n print(\"Starting training for Point networks...\")\n process_dirs(['Point_pm6y12'], sim_dir_og, 'Point', 10e6)\n \n print(\"Training completed!\")\n```\n\n#### `SubTraining.py`\n\nThis script handles individual network training with optimized configurations for different network types. The updated version includes enhanced documentation, improved argument parsing, and better model configuration management.\n\n**Key Features:**\n- **Network-Specific Optimization**: Tailored hyperparameters for each network type\n- **Flexible Configuration**: Command-line arguments for all training parameters\n- **Model Settings**: Optimized layer architectures and learning rates\n- **Error Handling**: Comprehensive error reporting and validation\n\n```python\n# Example usage (called automatically by Training.py)\npython SubTraining.py /path/to/sim_dir 0 Residual 10000000 4\n\n# Arguments:\n# 1. sim_dir: Simulation directory path\n# 2. idx: Network index to train\n# 3. type: Network type (Residual/Difference/Point)\n# 4. limit: Training data limit\n# 5. inputs: Number of input features\n```\n\n**Network Type Configurations:**\n\n| Network Type | Layers | Nodes | Learning Rate | Batch Size | Epochs |\n|-------------|---------|-------|---------------|------------|---------|\n| Residual | 4 | 128 | 8e-5 | 16384 | 1024 |\n| Difference | 4 | 256 | 1e-5 | 16384 | 1024 |\n| Point | 4 | 256 | 1e-4 | 1024 | 4096 |\n\n```python\n# SubTraining.py - Network-specific training with optimized configurations\n\"\"\"\nIndividual Network Training Script\n\nHandles training of specific networks with optimized hyperparameters\nfor different network types (Residual, Difference, Point).\n\"\"\"\n\nfrom argparse import ArgumentParser\nimport PyOghma_ML_Private as OML\nfrom PyOghma_ML_Private.Training import Model_Settings\n\ndef configure_model_settings(network_type, limit, inputs):\n \"\"\"Configure model settings based on network type.\"\"\"\n if network_type == 'Residual':\n return Model_Settings(\n initializer='he_normal',\n activation='silu',\n layer_nodes=[128, 128, 128, 128],\n dropout=[0.05, 0.05, 0.05, 0.05],\n inital_learning_rate=8e-5,\n batch_size=16384,\n epochs=1024,\n patience=16,\n decay_rate=6e-1,\n permutations_limit=int(limit),\n inputs=int(inputs)\n )\n # Additional configurations for Difference and Point networks...\n \ndef main():\n parser = ArgumentParser(description=\"Train individual neural networks\")\n parser.add_argument(\"sim_dir\", help=\"Simulation directory path\")\n parser.add_argument(\"idx\", type=int, help=\"Network index\")\n parser.add_argument(\"type\", help=\"Network type\")\n parser.add_argument(\"limit\", help=\"Training data limit\")\n parser.add_argument(\"inputs\", help=\"Number of inputs\")\n \n args = parser.parse_args()\n \n # Configure and train with enhanced error handling\n m = configure_model_settings(args.type, args.limit, args.inputs)\n A = OML.Networks.initialise(args.sim_dir, network_type=args.type, model_settings=m)\n A.train(args.idx)\n\nif __name__ == \"__main__\":\n main()\n```\n\n---\n\n### Predictions with `Predict.py` and `SubPredict.py`\n\n#### `Predict.py`\n\nThis script orchestrates the prediction pipeline for experimental data using trained neural network models. The updated version features improved error handling, better device mapping, and enhanced progress reporting.\n\n**Key Features:**\n- **Multi-Device Support**: Handles various device types with automatic material mapping\n- **Experiment Flexibility**: Supports multiple experiment types (J-V, Suns-Voc, etc.)\n- **Enhanced Error Handling**: Comprehensive error checking and detailed status reporting\n- **Progress Tracking**: Clear progress indicators and completion status\n- **Modular Design**: Well-organized helper functions for maintainability\n\n```python\n# Example: Using the improved Predict.py\nfrom Predict import map_device_to_material, get_experimental, get_illumination_files\n\n# Configuration\ndevice_types = [\"PM6Y12_ZnO\", \"PM6BTPeC9\"] # Multiple device types supported\nexperiment_type = \"light_dark_jv\" # or \"sunsvoc\"\n\n# Automatic device-to-material mapping\nfor device_type in device_types:\n net_mat = map_device_to_material(device_type)\n print(f\"Device {device_type} mapped to material: {net_mat}\")\n \n # Get experimental files with error handling\n try:\n exp_files = get_experimental(device_type, base_exp_dir, experiment_type)\n print(f\"Found {len(exp_files)} experimental files\")\n except FileNotFoundError as e:\n print(f\"Error: {e}\")\n```\n\n**Improved Features:**\n- **Helper Functions**: Modular design with `get_experimental()`, `map_device_to_material()`, and `get_illumination_files()`\n- **Error Handling**: File existence checking and detailed error reporting\n- **Status Reporting**: Progress indicators (\u2713/\u2717) for each processing step\n- **Flexible Configuration**: Easy-to-modify device types and experiment settings\n\n```python\n# filepath: /media/cai/Big/PyOghma_ml/PyOghma_ML/Predict.py\nimport os\nimport subprocess\nfrom natsort import natsorted\nimport json\n\ndef get_experimental(device_type, directory, exp):\n \"\"\"\n Get sorted list of experimental data files for a given device type and experiment.\n \"\"\"\n exp_dir = os.path.join(directory, device_type, exp)\n files = os.listdir(exp_dir)\n if exp == \"light_dark_jv\":\n ending = 'am15.dat'\n elif exp == \"sunsvoc\":\n ending = '0000000uIllu_IV.dat'\n else:\n ending = ''\n files = [os.path.join(exp_dir, f) for f in files if f.endswith(ending)]\n files = natsorted(files)\n return files\n\nif __name__ == \"__main__\":\n # Define simulation and experimental data directories\n sim_dir_og = os.path.join('/', 'media', 'cai', 'Big', 'Simulated_Data', 'opkm', 'Networks_Extended')\n base_exp_dir = os.path.join('/', 'media', 'cai', 'Big', 'Experimental_Data', 'Data_From_Chen')\n device_type = [\"PM6Y12_ZnO\"] # Can include \"PM6BTPeC9\", \"W108-2\", etc.\n exp = \"light_dark_jv\" # or \"sunsvoc\"\n res_dir = os.path.join(os.getcwd(), 'Results')\n\n for idx in device_type:\n # Map device type to network material\n match idx:\n case 'PM6BTPeC9'| 'W108-2':\n net_mat = 'pm6ec9'\n case 'PM6Y12_ZnO':\n net_mat = 'pm6y12'\n case 'PM6Y12_PEIZn':\n net_mat = 'pm6y12_peizn'\n case _:\n raise ValueError(f\"Unknown device type: {idx}\")\n\n files = get_experimental(idx, base_exp_dir, exp)\n\n for jdx in files:\n name = os.path.basename(jdx).split('.')[0]\n # List and filter simulation directories for the current device type\n sim_dirs = [kdx for kdx in os.listdir(sim_dir_og) if kdx.startswith(('Residual_')) and net_mat in kdx]\n sim_dirs = natsorted(sim_dirs)\n sim_dirs = [os.path.join(sim_dir_og, kdx) for kdx in sim_dirs]\n \n for kdx in sim_dirs:\n training = os.path.join(kdx, 'training')\n conversion = os.path.join(kdx, 'conversion')\n network_type = os.path.basename(kdx).split('_')[0]\n extra_info = os.path.basename(kdx).split('_')[-1]\n \n # Check if result already exists before processing\n out_file = os.path.join(res_dir, network_type, name + '.csv')\n if os.path.isfile(out_file):\n print(f\"File {name}.csv already exists. Skipping processing.\")\n continue\n\n # Run SubPredict with appropriate arguments\n subprocess.run([\"python\", \"SubPredict.py\", training, conversion, jdx, res_dir, network_type, \"JV\", \"Deibel\", extra_info])\n```\n\n#### `SubPredict.py`\n\nThis script handles individual prediction tasks using trained neural network models. The updated version features enhanced documentation, improved error handling, and better argument parsing for more reliable predictions.\n\n**Key Features:**\n- **Enhanced Documentation**: Comprehensive argument descriptions and usage examples\n- **Robust Error Handling**: Try/catch blocks with detailed error reporting\n- **Progress Tracking**: Status messages for each prediction step\n- **Flexible Output**: Configurable output directory structure\n- **Network Type Support**: Handles different prediction methods for various network types\n\n```python\n# Example usage (automatically called by Predict.py)\npython SubPredict.py /path/to/sim_dir /path/to/abs_dir /path/to/exp.dat /path/to/results Residual JV Deibel\n\n# Enhanced argument parsing with detailed help\npython SubPredict.py --help\n```\n\n**Improved Features:**\n- **Comprehensive Error Handling**: Detailed error messages for debugging\n- **Progress Reporting**: Step-by-step status updates with \u2713/\u2717 indicators\n- **Enhanced Documentation**: Built-in help text and usage examples\n- **Output Organization**: Automatic directory creation and file management\n- **Network-Specific Logic**: Optimized prediction methods for different network types\n\n**Prediction Workflow:**\n1. **Data Loading**: Load and standardize experimental data with validation\n2. **Network Initialization**: Initialize appropriate network type with error checking\n3. **Prediction Generation**: Execute predictions based on network architecture\n4. **Report Building**: Generate comprehensive output reports\n5. **File Management**: Save results with organized directory structure\n\n```python\n# SubPredict.py - Enhanced individual prediction script\n\"\"\"\nIndividual Prediction Script\n\nPerforms predictions on single experimental files with comprehensive\nerror handling and progress reporting.\n\"\"\"\n\ndef main():\n parser = ArgumentParser(description=\"Individual prediction script\")\n # Enhanced argument parsing with detailed help\n parser.add_argument(\"sim_dir\", help=\"Simulation directory path\")\n parser.add_argument(\"abs_dir\", help=\"Absolute data directory\")\n parser.add_argument(\"exp_dir\", help=\"Experimental data file\")\n # ... additional arguments\n \n args = parser.parse_args()\n \n try:\n print(f\"Starting prediction for: {os.path.basename(args.exp_dir)}\")\n \n # Load and standardize experimental data\n Exp = OML.Input.experiment(args.exp_dir, args.exp_typ, args.Source_Lab)\n Exp.standardise_inputs()\n print(f\"\u2713 Loaded experimental data: {len(Exp.x)} data points\")\n \n # Initialize network and generate predictions\n N = OML.Networks.initialise(args.sim_dir, network_type=args.net_typ)\n if args.net_typ in ['Difference', 'Residual']:\n N.predict(args.abs_dir, Exp)\n else:\n N.predict(Exp)\n print(\"\u2713 Predictions generated successfully\")\n \n # Build and save report\n O = OML.Output(N, Exp, abs_dir=args.abs_dir)\n O.build_report()\n O.save_report(output_path)\n print(f\"\u2713 Results saved to: {output_path}.csv\")\n \n except Exception as e:\n print(f\"\u2717 Error during prediction: {e}\")\n raise\n\nif __name__ == \"__main__\":\n main()\n```\n\n---\n\n### Supported Device Types and Materials\n\nThe current implementation supports the following device types and their corresponding network materials:\n\n| Device Type | Network Material | Description |\n|-------------------|------------------|--------------------------------|\n| PM6BTPeC9 | pm6ec9 | PM6:BTP-eC9 based devices |\n| W108-2 | pm6ec9 | W108-2 devices |\n| PM6Y12_ZnO | pm6y12 | PM6:Y12 with ZnO devices |\n| PM6Y12_PEIZn | pm6y12_peizn | PM6:Y12 with PEI-Zn devices |\n\n### Supported Experiments\n\n| Experiment Type | File Extension | Description |\n|-------------------|---------------------|--------------------------------|\n| light_dark_jv | am15.dat | Light and dark JV measurements |\n| sunsvoc | 0000000uIllu_IV.dat| Suns-Voc measurements |\n\n### Supported Laboratories\n\n| Laboratory | Label |\n|------------------|---------|\n| OPKM | Deibel |\n| HSP | Shoaee |\n| Herzig Group | Herzig |\n| OghmaNano | Oghma |\n\n> **Note**: For new laboratories or characteristics, custom input functions may need to be written. Contact support for assistance.\n\n---\n\n## Recent Improvements (v0.2.0)\n\nPyOghma_ML has been significantly enhanced with improved documentation, error handling, and code organization:\n\n### **Enhanced Documentation**\n- **Comprehensive Module Docstrings**: All scripts now include detailed module-level documentation\n- **Function Documentation**: Improved docstrings with parameter types, return values, and examples\n- **Usage Examples**: Clear examples and configuration guidance throughout\n\n### **Improved Error Handling**\n- **Robust Validation**: File existence checking and directory validation\n- **Detailed Error Messages**: Clear error reporting with troubleshooting information\n- **Graceful Failure**: Scripts continue processing when individual tasks fail\n- **Status Indicators**: Visual progress tracking with \u2713/\u2717 status markers\n\n### **Code Structure Enhancements**\n- **Modular Design**: Refactored scripts into well-organized functions\n- **Helper Functions**: Separated concerns for better maintainability\n- **Configuration Management**: Organized and documented configuration sections\n- **Modern Python**: Updated to use modern Python packaging with `pyproject.toml`\n\n### **User Experience Improvements**\n- **Progress Tracking**: Real-time status updates during training and prediction\n- **Better Logging**: Informative console output with clear progress indicators\n- **Organized Output**: Structured result directories and file naming\n- **Enhanced CLI**: Improved command-line argument parsing with detailed help\n\n---\n\n### Example Workflow\n\n#### 1. **Training Networks** (Enhanced with Error Handling)\n\n```python\n# Use the improved Training.py with automatic error handling\nfrom Training import process_dirs\n\n# The script now provides detailed status reporting\nprint(\"=== Starting Neural Network Training ===\")\n\n# Train different network types with enhanced monitoring\nconfigurations = [\n (['Point_pm6y12'], 'Point', 10e6),\n (['Difference_pm6y12'], 'Difference', 10e6),\n (['Residual_pm6y12'], 'Residual', 10e6)\n]\n\nsim_dir_og = '/path/to/simulation/data'\n\nfor dirs, network_type, train_size in configurations:\n print(f\"\\n--- Training {network_type} Networks ---\")\n try:\n process_dirs(dirs, sim_dir_og, network_type, train_size)\n print(f\"\u2713 {network_type} training completed successfully\")\n except Exception as e:\n print(f\"\u2717 {network_type} training failed: {e}\")\n```\n\n#### 2. **Generate Predictions** (Improved Pipeline)\n\n```python\n# Use the enhanced Predict.py with better device mapping\nfrom Predict import map_device_to_material, get_experimental\n\n# Configuration with multiple device types\ndevice_types = [\"PM6Y12_ZnO\", \"PM6BTPeC9\", \"W108-2\"]\nexperiment_type = \"light_dark_jv\"\n\nprint(\"=== Starting Prediction Pipeline ===\")\n\nfor device_type in device_types:\n print(f\"\\n--- Processing {device_type} ---\")\n \n try:\n # Automatic device-to-material mapping\n net_mat = map_device_to_material(device_type)\n print(f\"\u2713 Mapped {device_type} to material: {net_mat}\")\n \n # Load experimental files with validation\n exp_files = get_experimental(device_type, base_exp_dir, experiment_type)\n print(f\"\u2713 Found {len(exp_files)} experimental files\")\n \n # Process predictions with progress tracking\n # (Detailed processing handled automatically)\n \n except Exception as e:\n print(f\"\u2717 Error processing {device_type}: {e}\")\n continue\n\nprint(\"\u2713 Prediction pipeline completed\")\n```\n\n#### 3. **Network Type Optimization**\n\n| Network Type | Architecture | Use Case | Key Features |\n|-------------|--------------|----------|--------------|\n| **Residual** | 4\u00d7128 nodes | Residual prediction | Fast training, efficient memory |\n| **Difference** | 4\u00d7256 nodes | Differential analysis | High accuracy, robust to noise |\n| **Point** | 4\u00d7256 nodes | Direct prediction | Minimal data requirements |\n\n#### 4. **Testing Data Loading**\n\n```python\n# Use the enhanced LoadingFileTest.py for validation\nfrom LoadingFileTest import main\n\n# Test data loading with multiple optical densities\nprint(\"=== Testing Data Loading Functionality ===\")\n\n# The script provides comprehensive testing:\n# - Multiple OD values (0.0, 1.0, 2.0, 3.0)\n# - Error handling for missing files\n# - Visual validation with enhanced plots\n# - Detailed progress reporting\n\nmain() # Automatically runs all tests with status indicators\n```\n\n---\n\n## Requirements and Dependencies\n\nPyOghma_ML requires Python 3.9+ and includes the following key dependencies:\n\n### **Core Dependencies**\n- **Scientific Computing**: NumPy \u22651.24.0, SciPy \u22651.11.0, Pandas \u22652.1.0\n- **Machine Learning**: TensorFlow \u22652.15.0, Keras \u22653.0.0, Keras-Tuner \u22651.4.0\n- **Visualization**: Matplotlib \u22653.7.0\n- **Data Processing**: h5py \u22653.9.0, ujson \u22655.8.0, natsort \u22658.4.0\n- **Templating**: Jinja2 \u22653.1.0\n\n### **Optional Dependencies**\n```bash\n# Development tools\npip install PyOghma-ML[dev] # pytest, black, flake8, mypy\n\n# Documentation generation\npip install PyOghma-ML[docs] # sphinx, themes, parsers\n\n# GPU acceleration\npip install PyOghma-ML[gpu] # tensorflow with CUDA support\n```\n\n### **System Requirements**\n- **Python**: 3.9, 3.10, 3.11, or 3.12\n- **Operating System**: Cross-platform (Windows, macOS, Linux)\n- **Memory**: Minimum 8GB RAM (16GB recommended for large datasets)\n- **Storage**: 1GB+ for models and results\n\n---\n\n## Authors\n\n- [@CaiWilliams](https://github.com/CaiWilliams)\n\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "A Python Pipeline for Machine Learning with Oghma",
"version": "0.2.0",
"project_urls": {
"Homepage": "https://github.com/CaiWilliams/PyOghma_ML",
"Issues": "https://github.com/CaiWilliams/PyOghma_ML/issues"
},
"split_keywords": [
"machine learning",
" neural networks",
" oghma",
" photovoltaics",
" solar cells"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "cc21d3d595ab40e445473a34988004f3923e03ad506c5887b0b5464f0f6f7f70",
"md5": "71934db2b76d7ed46272194223873148",
"sha256": "77050d6557fcc56d1468eedc625d1de1c1d1144ded9f2af0863b9ebfc11d166a"
},
"downloads": -1,
"filename": "pyoghma_ml-0.2.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "71934db2b76d7ed46272194223873148",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9",
"size": 58061,
"upload_time": "2025-07-19T16:19:00",
"upload_time_iso_8601": "2025-07-19T16:19:00.857140Z",
"url": "https://files.pythonhosted.org/packages/cc/21/d3d595ab40e445473a34988004f3923e03ad506c5887b0b5464f0f6f7f70/pyoghma_ml-0.2.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "7a0705e3f7e5c696d4b934440c88193587b5dcb65db4eecc874b8efe3d222c5a",
"md5": "a807f2db629c7789a0fbeb41e30579be",
"sha256": "7a7dd7dfe5712bf89d39c03525b4637166c0003abf8b743678aadce4619494c1"
},
"downloads": -1,
"filename": "pyoghma_ml-0.2.0.tar.gz",
"has_sig": false,
"md5_digest": "a807f2db629c7789a0fbeb41e30579be",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9",
"size": 795552,
"upload_time": "2025-07-19T16:19:02",
"upload_time_iso_8601": "2025-07-19T16:19:02.827267Z",
"url": "https://files.pythonhosted.org/packages/7a/07/05e3f7e5c696d4b934440c88193587b5dcb65db4eecc874b8efe3d222c5a/pyoghma_ml-0.2.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-07-19 16:19:02",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "CaiWilliams",
"github_project": "PyOghma_ML",
"github_not_found": true,
"lcname": "pyoghma-ml"
}
JSON
