MergeSourceFile

Name	MergeSourceFile JSON
Version	1.2.0 JSON
	download
home_page	None
Summary	A Python tool to process SQL*Plus scripts with Jinja2 template support, resolving file inclusions (@, @@) and variable substitutions (DEFINE/UNDEFINE)
upload_time	2025-10-20 21:36:56
maintainer	None
docs_url	None
author	Alejandro G.
requires_python	>=3.8
license	MIT
keywords	sqlplus sql oracle script-processor file-merger
VCS
bugtrack_url
requirements	No requirements were recorded.
Travis-CI
coveralls test coverage	No coveralls.

            # MergeSourceFile 

[![CircleCI](https://dl.circleci.com/status-badge/img/gh/alegorico/MergeSourceFile/tree/main.svg?style=svg)](https://dl.circleci.com/status-badge/redirect/gh/alegorico/MergeSourceFile/tree/main)
[![codecov](https://codecov.io/gh/alegorico/MergeSourceFile/branch/main/graph/badge.svg)](https://codecov.io/gh/alegorico/MergeSourceFile)

A Python tool to process SQL*Plus scripts with Jinja2 template support, resolving file inclusions and variable substitutions.

## Description

This is a Python project that includes a script capable of processing SQL*Plus scripts with Jinja2 template support. The program resolves file inclusions referenced through `@` and `@@`, performs variable substitutions defined with `DEFINE`, supports variable removal with `UNDEFINE`, allows variable redefinition throughout the script, and **now includes Jinja2 template processing** with custom filters and multiple processing strategies.

## Features

- **File Inclusion Resolution**: Processes `@` and `@@` directives to include external SQL files
- **Variable Substitution**: Handles `DEFINE` and `UNDEFINE` commands for variable management
- **Variable Redefinition**: Supports redefining variables throughout the script
- **🆕 Jinja2 Template Processing**: Full Jinja2 template support with variables, conditionals, loops, and filters
- **🆕 Custom Jinja2 Filters**: `sql_escape` for SQL injection protection and `strftime` for date formatting
- **🆕 Multiple Processing Orders**: Choose between `default`, `jinja2_first`, or `includes_last` processing strategies
- **🆕 Dynamic File Inclusion**: Use Jinja2 variables to determine which files to include
- **Tree Display**: Shows the inclusion hierarchy in a tree structure
- **Verbose Mode**: Detailed logging for debugging and understanding the processing flow

## Installation

```bash
pip install MergeSourceFile
```

## What's New in v1.2.0

- ✨ **TOML Configuration Support**: New `--config` / `-c` parameter to read settings from a TOML file
- 🔧 **Configuration File**: Centralized configuration in `config.toml` instead of command-line parameters
- ⚠️ **Deprecation Warning**: Traditional command-line parameters now show a deprecation warning
- 🔒 **Mutual Exclusivity**: Config file and command-line parameters cannot be used together
- 📋 **Backward Compatibility**: All existing command-line parameters continue to work
- 🧪 **Comprehensive Testing**: 11 new tests added (67 total tests passing)

## What's New in v1.1.1

- 🐛 **DEFINE Bug Fixes**: Critical fix for DEFINE statements without quotes (e.g., `DEFINE VAR = value`)
- 🔧 **Enhanced DEFINE Support**: Improved regex to handle decimal values, hyphens, and complex alphanumeric values
- 📊 **Better Error Reporting**: Verbose mode now shows ignored DEFINE statements with line numbers
- 🪟 **Windows Compatibility**: Fixed Unicode encoding issues for full Windows support
- ✅ **Robust Testing**: 17 new tests added, 56/56 tests passing including full CLI integration

## 🚨 Important: Command-Line Parameters Deprecation Notice

**Starting from the next major version, command-line parameters will be deprecated in favor of TOML configuration files.**

We strongly recommend migrating to the new TOML configuration format:
- ✅ **Use**: `mergesourcefile --config config.toml` (Recommended)
- ⚠️ **Deprecated**: `mergesourcefile --input file.sql --output out.sql` (Will be removed in future versions)

See the [TOML Configuration](#toml-configuration-recommended) section below for migration instructions.

## What's New in v1.1.0

- ✨ **Jinja2 Template Support**: Full integration with Jinja2 templating engine
- 🔧 **Custom Filters**: Added `sql_escape` and `strftime` filters for enhanced functionality
- 🔀 **Processing Orders**: Three different processing strategies for complex scenarios
- 🎯 **Dynamic Inclusion**: Use Jinja2 variables to conditionally include files
- 📋 **Enhanced CLI**: New command-line options for Jinja2 functionality
- 🧪 **Comprehensive Testing**: 20+ new tests ensuring reliability

## TOML Configuration (Recommended)

### Why Use TOML Configuration?

TOML configuration files provide several advantages:
- **Version Control Friendly**: Store your configuration in the repository
- **Maintainable**: Easier to manage complex configurations
- **Reusable**: Use the same configuration across different environments
- **Future-Proof**: Command-line parameters will be deprecated in future versions

### Quick Start with TOML

1. **Create a configuration file** (e.g., `config.toml`):

```toml
[mergesourcefile]
input = "main.sql"
output = "merged.sql"
skip_var = false
verbose = false
jinja2 = false
processing_order = "default"
```

2. **Run with the configuration file**:

```bash
mergesourcefile --config config.toml
# or using short form:
mergesourcefile -c config.toml
```

### TOML Configuration Options

All command-line options are available in TOML format:

```toml
[mergesourcefile]
# Required fields
input = "input.sql"              # Input file to process
output = "output.sql"            # Output file for results

# Optional fields (with default values)
skip_var = false                 # Skip variable substitution
verbose = false                  # Enable verbose mode
jinja2 = false                   # Enable Jinja2 processing
jinja2_vars = "vars.json"        # JSON file with Jinja2 variables
processing_order = "default"     # Processing order: default, jinja2_first, includes_last
```

### Example Configurations

**Basic Processing**:
```toml
[mergesourcefile]
input = "main.sql"
output = "merged.sql"
```

**With Jinja2 Templates**:
```toml
[mergesourcefile]
input = "template.sql"
output = "generated.sql"
jinja2 = true
jinja2_vars = "production_vars.json"
processing_order = "jinja2_first"
```

**Verbose Mode for Debugging**:
```toml
[mergesourcefile]
input = "debug.sql"
output = "debug_output.sql"
verbose = true
skip_var = false
```

### Migration from Command-Line Parameters

If you're currently using command-line parameters, migration is simple:

**Before (Deprecated)**:
```bash
mergesourcefile --input main.sql --output merged.sql --verbose --jinja2 --jinja2-vars vars.json
```

**After (Recommended)**:

Create `config.toml`:
```toml
[mergesourcefile]
input = "main.sql"
output = "merged.sql"
verbose = true
jinja2 = true
jinja2_vars = "vars.json"
```

Run:
```bash
mergesourcefile --config config.toml
```

### Configuration File Location

Place your TOML configuration file:
- In the root of your project directory
- Or anywhere else and reference it with the full path: `mergesourcefile --config /path/to/config.toml`

See `config.example.toml` in the repository for a complete example with all available options.

## Usage

### Recommended: TOML Configuration File

```bash
mergesourcefile --config config.toml
```

See the [TOML Configuration](#toml-configuration-recommended) section for detailed information.

### Legacy: Command Line (Deprecated)

⚠️ **Warning**: Command-line parameters will be deprecated in future versions. Please migrate to TOML configuration.

```bash
mergesourcefile --input input.sql --output output.sql
```

### Options (Legacy Command-Line)

- `--config, -c`: **RECOMMENDED** - Load configuration from TOML file
- `--input, -i`: Input SQL*Plus file to process (deprecated, use TOML config)
- `--output, -o`: Output file where the result will be written (deprecated, use TOML config)
- `--skip-var, -sv`: Skip variable substitution, only resolve file inclusions (deprecated, use TOML config)
- `--verbose, -v`: Enable verbose mode for detailed processing information (deprecated, use TOML config)
- `--jinja2`: Enable Jinja2 template processing (deprecated, use TOML config)
- `--jinja2-vars`: JSON string with variables for Jinja2 template processing (deprecated, use TOML config)
- `--processing-order`: Choose processing order: `default`, `jinja2_first`, or `includes_last` (deprecated, use TOML config)

### TOML Configuration File Format

Create a `config.toml` file in your project directory:

```toml
[mergesourcefile]
input = "main.sql"
output = "merged.sql"

# Optional parameters
skip_var = false        # Set to true to skip variable substitution
verbose = false         # Set to true for detailed processing information
jinja2 = false          # Set to true to enable Jinja2 template processing
jinja2_vars = ""        # Path to JSON file with Jinja2 variables
processing_order = "default"  # Options: default, jinja2_first, includes_last
```

Example with Jinja2 support:

```toml
[mergesourcefile]
input = "template.sql"
output = "output.sql"
jinja2 = true
jinja2_vars = "vars.json"
processing_order = "jinja2_first"
verbose = true
```

### Examples

1. **Recommended: Use TOML configuration**:
   ```bash
   mergesourcefile --config config.toml
   ```

2. **Legacy: Process a SQL file with full processing** (deprecated):
   ```bash
   mergesourcefile -i main.sql -o merged.sql
   ```

3. **Legacy: Process only file inclusions, skip variable substitution** (deprecated):
   ```bash
   mergesourcefile -i main.sql -o merged.sql --skip-var
   ```

4. **Legacy: Process with verbose output** (deprecated):
   ```bash
   mergesourcefile -i main.sql -o merged.sql --verbose
   ```

5. **Legacy: Process with Jinja2 template support** (deprecated):
   ```bash
   mergesourcefile -i template.sql -o merged.sql --jinja2
   ```

6. **Legacy: Process with Jinja2 variables** (deprecated):
   ```bash
   mergesourcefile -i template.sql -o merged.sql --jinja2 --jinja2-vars vars.json
   ```

7. **Legacy: Process with Jinja2-first processing order** (deprecated):
   ```bash
   mergesourcefile -i template.sql -o merged.sql --jinja2 --processing-order jinja2_first
   ```

## How It Works

### File Inclusion

- `@filename`: Includes a file relative to the original base path
- `@@filename`: Includes a file relative to the current file's directory

### Variable Substitution

#### DEFINE Syntax (Enhanced in v1.1.1)
- `DEFINE varname = 'quoted value';`: Defines with quoted value (supports spaces)
- `DEFINE varname = unquoted_value;`: Defines with unquoted value (no spaces)
- `DEFINE varname = 3.14;`: Supports decimal values
- `DEFINE varname = ABC-123;`: Supports hyphenated values
- `DEFINE varname = '';`: Supports empty string values

#### Variable Usage
- `&varname`: References a variable for substitution
- `&varname..`: Variable concatenation with period
- `UNDEFINE varname;`: Removes a variable definition

#### Error Handling (v1.1.1)
- Invalid DEFINE syntax is ignored and reported in verbose mode
- Example: `DEFINE var = ;` will be skipped with a warning
- Variables must be defined before use or an error is thrown

### 🆕 Jinja2 Template Processing

#### Basic Template Syntax
- `{{ variable }}`: Variable substitution
- `{% if condition %}...{% endif %}`: Conditional blocks
- `{% for item in list %}...{% endfor %}`: Loop blocks
- `{# comment #}`: Template comments

#### Custom Filters
- `sql_escape`: Escapes single quotes for SQL safety
  ```sql
  SELECT * FROM users WHERE name = '{{ user_name | sql_escape }}';
  ```
- `strftime`: Formats datetime objects
  ```sql
  -- Generated on {{ now() | strftime('%Y-%m-%d %H:%M:%S') }}
  ```

#### Processing Orders
1. **default**: File Inclusions → Jinja2 Templates → SQL Variables
2. **jinja2_first**: Jinja2 Templates → File Inclusions → SQL Variables
3. **includes_last**: SQL Variables → Jinja2 Templates → File Inclusions

#### Dynamic File Inclusion Example
```sql
-- Using jinja2_first order to dynamically determine which files to include
{% if environment == 'production' %}
@prod_config.sql
{% else %}
@dev_config.sql
{% endif %}
```

## Complete Example

### Input Template (`template.sql`)
```sql
{# This is a Jinja2 comment #}
-- Database setup for {{ environment | upper }} environment
-- Generated on {{ now() | strftime('%Y-%m-%d %H:%M:%S') }}

{% if environment == 'production' %}
@production_settings.sql
{% else %}
@development_settings.sql
{% endif %}

DEFINE db_name = '{{ database_name }}';
DEFINE table_prefix = '{{ table_prefix }}';

CREATE TABLE &table_prefix._users (
    id NUMBER PRIMARY KEY,
    name VARCHAR2(100) NOT NULL,
    email VARCHAR2(255) UNIQUE,
    created_date DATE DEFAULT SYSDATE
);

{% for table in additional_tables %}
CREATE TABLE &table_prefix._{{ table.name }} (
    id NUMBER PRIMARY KEY,
    {% for column in table.columns -%}
    {{ column.name }} {{ column.type }}{% if not loop.last %},{% endif %}
    {% endfor %}
);
{% endfor %}

-- Insert sample data with escaped values
INSERT INTO &table_prefix._users (name, email) 
VALUES ('{{ sample_user | sql_escape }}', '{{ sample_email | sql_escape }}');
```

### Command
```bash
mergesourcefile -i template.sql -o output.sql --jinja2 --processing-order jinja2_first --jinja2-vars '{
  "environment": "production",
  "database_name": "MYAPP_DB",
  "table_prefix": "APP",
  "sample_user": "John O'\''Brien",
  "sample_email": "john@example.com",
  "additional_tables": [
    {
      "name": "products",
      "columns": [
        {"name": "title", "type": "VARCHAR2(200)"},
        {"name": "price", "type": "NUMBER(10,2)"}
      ]
    }
  ]
}'
```

## Migration from v1.0.x

If you're upgrading from a previous version, your existing scripts will continue to work without any changes. The new Jinja2 functionality is **completely optional** and requires explicit activation with the `--jinja2` flag.

### Backward Compatibility
- All existing command-line options work exactly as before
- File inclusion (`@`, `@@`) behavior is unchanged
- Variable substitution (`DEFINE`, `UNDEFINE`) works as expected
- No breaking changes to existing functionality

### Gradual Adoption
You can gradually adopt Jinja2 features:
1. Start with simple variable substitution: `{{ variable }}`
2. Add conditional logic: `{% if condition %}`
3. Use loops for repetitive structures: `{% for item in list %}`
4. Apply custom filters: `{{ value | sql_escape }}`
5. Experiment with processing orders for complex scenarios

## Migration from v1.1.x to v1.2.0

### Migrating to TOML Configuration

The new TOML configuration file approach offers a cleaner, more maintainable way to manage your MergeSourceFile settings. While command-line parameters are still supported, they will be deprecated in future versions.

#### Step 1: Create a TOML Configuration File

Instead of:
```bash
mergesourcefile -i main.sql -o output.sql --verbose --skip-var
```

Create a `config.toml` file:
```toml
[mergesourcefile]
input = "main.sql"
output = "output.sql"
verbose = true
skip_var = true
```

Then run:
```bash
mergesourcefile --config config.toml
```

#### Step 2: Migrating Jinja2 Configurations

For projects using Jinja2, instead of:
```bash
mergesourcefile -i template.sql -o output.sql --jinja2 --jinja2-vars vars.json --processing-order jinja2_first
```

Use a TOML config:
```toml
[mergesourcefile]
input = "template.sql"
output = "output.sql"
jinja2 = true
jinja2_vars = "vars.json"
processing_order = "jinja2_first"
```

#### Benefits of TOML Configuration

1. **Version Control Friendly**: Configuration files can be committed to your repository
2. **Project-Specific Settings**: Each project can have its own `config.toml`
3. **Cleaner Scripts**: Simplifies build scripts and CI/CD pipelines
4. **Self-Documenting**: Configuration files are easier to read and understand
5. **No Shell Escaping**: Avoid issues with special characters in command-line parameters

#### Deprecation Timeline

- **v1.2.0**: TOML configuration introduced, deprecation warning added for command-line parameters
- **v1.3.0** (planned): Increased warning severity
- **v2.0.0** (future): Command-line parameters may be removed entirely

## Best Practices

### When to Use Each Processing Order

- **default**: Best for most use cases where Jinja2 templates don't need to generate file inclusion directives
- **jinja2_first**: Use when Jinja2 templates need to conditionally determine which files to include
- **includes_last**: Use when you need SQL variables to be processed before Jinja2 templates and file inclusions

### Security Considerations

Always use the `sql_escape` filter when inserting user-provided data:
```sql
-- ❌ Vulnerable to SQL injection
SELECT * FROM users WHERE name = '{{ user_input }}';

-- ✅ Safe with sql_escape filter
SELECT * FROM users WHERE name = '{{ user_input | sql_escape }}';
```

### Performance Tips

- Use `--skip-var` if you don't need SQL variable processing
- For large projects, consider splitting templates into smaller, focused files
- Use Jinja2 comments `{# comment #}` instead of SQL comments for template-specific notes

## Platform Compatibility

### Operating Systems
- ✅ **Linux**: Full support with all features
- ✅ **macOS**: Full support with all features  
- ✅ **Windows**: Full support with enhanced compatibility (v1.1.1)
  - Fixed Unicode encoding issues for CLI operations
  - All 56 tests pass successfully on Windows systems
  - Proper error codes and file path handling

### Python Versions
- Python 3.8+
- Tested with Python 3.9, 3.10, 3.11, 3.12, 3.14

### Character Encoding
- Primary support: UTF-8 (recommended)
- Windows compatibility: ASCII-safe output for CLI operations
- All text files should use UTF-8 encoding for best results

## Troubleshooting

### Common Issues

1. **DEFINE syntax errors** (Fixed in v1.1.1):
   - ✅ `DEFINE VAR = value` now works correctly (was broken in v1.1.0)
   - ✅ Both quoted and unquoted DEFINE values supported
   - Use verbose mode (`--verbose`) to see ignored invalid DEFINE statements

2. **Jinja2 syntax errors**: Ensure proper template syntax with matching braces and tags
3. **Variable not found**: Check that all variables are provided via `--jinja2-vars`
4. **File inclusion issues**: Verify file paths and choose appropriate processing order
5. **Encoding problems** (Fixed in v1.1.1): 
   - ✅ Windows encoding issues resolved
   - Ensure all files use consistent encoding (UTF-8 recommended)
   - CLI now works properly on all Windows systems

### Windows-Specific Issues (Resolved in v1.1.1)
- ✅ **Unicode character display**: Fixed issues with special characters in CLI output
- ✅ **File path resolution**: Enhanced path handling for nested file inclusions
- ✅ **Exit codes**: CLI now returns proper error codes (1 for errors, 0 for success)

### Debug Mode

Use `--verbose` flag to see detailed processing information:
```bash
mergesourcefile -i template.sql -o output.sql --jinja2 --verbose
```

## License

This project is licensed under the MIT License.  
You are free to use, copy, modify, and distribute this software, provided that the copyright notice and this permission are included.  
The software is provided "as is", without warranty of any kind.

## Author

Alejandro G.

## Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

            

Raw data

            {
    "_id": null,
    "home_page": null,
    "name": "MergeSourceFile",
    "maintainer": null,
    "docs_url": null,
    "requires_python": ">=3.8",
    "maintainer_email": null,
    "keywords": "sqlplus, sql, oracle, script-processor, file-merger",
    "author": "Alejandro G.",
    "author_email": null,
    "download_url": "https://files.pythonhosted.org/packages/d1/b0/b7b0bd49a4eaed5a0463ce59fa672434b985a3cb71f7b2000cb2f6f73a91/mergesourcefile-1.2.0.tar.gz",
    "platform": null,
    "description": "# MergeSourceFile \r\n\r\n[![CircleCI](https://dl.circleci.com/status-badge/img/gh/alegorico/MergeSourceFile/tree/main.svg?style=svg)](https://dl.circleci.com/status-badge/redirect/gh/alegorico/MergeSourceFile/tree/main)\r\n[![codecov](https://codecov.io/gh/alegorico/MergeSourceFile/branch/main/graph/badge.svg)](https://codecov.io/gh/alegorico/MergeSourceFile)\r\n\r\nA Python tool to process SQL*Plus scripts with Jinja2 template support, resolving file inclusions and variable substitutions.\r\n\r\n## Description\r\n\r\nThis is a Python project that includes a script capable of processing SQL*Plus scripts with Jinja2 template support. The program resolves file inclusions referenced through `@` and `@@`, performs variable substitutions defined with `DEFINE`, supports variable removal with `UNDEFINE`, allows variable redefinition throughout the script, and **now includes Jinja2 template processing** with custom filters and multiple processing strategies.\r\n\r\n## Features\r\n\r\n- **File Inclusion Resolution**: Processes `@` and `@@` directives to include external SQL files\r\n- **Variable Substitution**: Handles `DEFINE` and `UNDEFINE` commands for variable management\r\n- **Variable Redefinition**: Supports redefining variables throughout the script\r\n- **\ud83c\udd95 Jinja2 Template Processing**: Full Jinja2 template support with variables, conditionals, loops, and filters\r\n- **\ud83c\udd95 Custom Jinja2 Filters**: `sql_escape` for SQL injection protection and `strftime` for date formatting\r\n- **\ud83c\udd95 Multiple Processing Orders**: Choose between `default`, `jinja2_first`, or `includes_last` processing strategies\r\n- **\ud83c\udd95 Dynamic File Inclusion**: Use Jinja2 variables to determine which files to include\r\n- **Tree Display**: Shows the inclusion hierarchy in a tree structure\r\n- **Verbose Mode**: Detailed logging for debugging and understanding the processing flow\r\n\r\n## Installation\r\n\r\n```bash\r\npip install MergeSourceFile\r\n```\r\n\r\n## What's New in v1.2.0\r\n\r\n- \u2728 **TOML Configuration Support**: New `--config` / `-c` parameter to read settings from a TOML file\r\n- \ud83d\udd27 **Configuration File**: Centralized configuration in `config.toml` instead of command-line parameters\r\n- \u26a0\ufe0f **Deprecation Warning**: Traditional command-line parameters now show a deprecation warning\r\n- \ud83d\udd12 **Mutual Exclusivity**: Config file and command-line parameters cannot be used together\r\n- \ud83d\udccb **Backward Compatibility**: All existing command-line parameters continue to work\r\n- \ud83e\uddea **Comprehensive Testing**: 11 new tests added (67 total tests passing)\r\n\r\n## What's New in v1.1.1\r\n\r\n- \ud83d\udc1b **DEFINE Bug Fixes**: Critical fix for DEFINE statements without quotes (e.g., `DEFINE VAR = value`)\r\n- \ud83d\udd27 **Enhanced DEFINE Support**: Improved regex to handle decimal values, hyphens, and complex alphanumeric values\r\n- \ud83d\udcca **Better Error Reporting**: Verbose mode now shows ignored DEFINE statements with line numbers\r\n- \ud83e\ude9f **Windows Compatibility**: Fixed Unicode encoding issues for full Windows support\r\n- \u2705 **Robust Testing**: 17 new tests added, 56/56 tests passing including full CLI integration\r\n\r\n## \ud83d\udea8 Important: Command-Line Parameters Deprecation Notice\r\n\r\n**Starting from the next major version, command-line parameters will be deprecated in favor of TOML configuration files.**\r\n\r\nWe strongly recommend migrating to the new TOML configuration format:\r\n- \u2705 **Use**: `mergesourcefile --config config.toml` (Recommended)\r\n- \u26a0\ufe0f **Deprecated**: `mergesourcefile --input file.sql --output out.sql` (Will be removed in future versions)\r\n\r\nSee the [TOML Configuration](#toml-configuration-recommended) section below for migration instructions.\r\n\r\n## What's New in v1.1.0\r\n\r\n- \u2728 **Jinja2 Template Support**: Full integration with Jinja2 templating engine\r\n- \ud83d\udd27 **Custom Filters**: Added `sql_escape` and `strftime` filters for enhanced functionality\r\n- \ud83d\udd00 **Processing Orders**: Three different processing strategies for complex scenarios\r\n- \ud83c\udfaf **Dynamic Inclusion**: Use Jinja2 variables to conditionally include files\r\n- \ud83d\udccb **Enhanced CLI**: New command-line options for Jinja2 functionality\r\n- \ud83e\uddea **Comprehensive Testing**: 20+ new tests ensuring reliability\r\n\r\n## TOML Configuration (Recommended)\r\n\r\n### Why Use TOML Configuration?\r\n\r\nTOML configuration files provide several advantages:\r\n- **Version Control Friendly**: Store your configuration in the repository\r\n- **Maintainable**: Easier to manage complex configurations\r\n- **Reusable**: Use the same configuration across different environments\r\n- **Future-Proof**: Command-line parameters will be deprecated in future versions\r\n\r\n### Quick Start with TOML\r\n\r\n1. **Create a configuration file** (e.g., `config.toml`):\r\n\r\n```toml\r\n[mergesourcefile]\r\ninput = \"main.sql\"\r\noutput = \"merged.sql\"\r\nskip_var = false\r\nverbose = false\r\njinja2 = false\r\nprocessing_order = \"default\"\r\n```\r\n\r\n2. **Run with the configuration file**:\r\n\r\n```bash\r\nmergesourcefile --config config.toml\r\n# or using short form:\r\nmergesourcefile -c config.toml\r\n```\r\n\r\n### TOML Configuration Options\r\n\r\nAll command-line options are available in TOML format:\r\n\r\n```toml\r\n[mergesourcefile]\r\n# Required fields\r\ninput = \"input.sql\"              # Input file to process\r\noutput = \"output.sql\"            # Output file for results\r\n\r\n# Optional fields (with default values)\r\nskip_var = false                 # Skip variable substitution\r\nverbose = false                  # Enable verbose mode\r\njinja2 = false                   # Enable Jinja2 processing\r\njinja2_vars = \"vars.json\"        # JSON file with Jinja2 variables\r\nprocessing_order = \"default\"     # Processing order: default, jinja2_first, includes_last\r\n```\r\n\r\n### Example Configurations\r\n\r\n**Basic Processing**:\r\n```toml\r\n[mergesourcefile]\r\ninput = \"main.sql\"\r\noutput = \"merged.sql\"\r\n```\r\n\r\n**With Jinja2 Templates**:\r\n```toml\r\n[mergesourcefile]\r\ninput = \"template.sql\"\r\noutput = \"generated.sql\"\r\njinja2 = true\r\njinja2_vars = \"production_vars.json\"\r\nprocessing_order = \"jinja2_first\"\r\n```\r\n\r\n**Verbose Mode for Debugging**:\r\n```toml\r\n[mergesourcefile]\r\ninput = \"debug.sql\"\r\noutput = \"debug_output.sql\"\r\nverbose = true\r\nskip_var = false\r\n```\r\n\r\n### Migration from Command-Line Parameters\r\n\r\nIf you're currently using command-line parameters, migration is simple:\r\n\r\n**Before (Deprecated)**:\r\n```bash\r\nmergesourcefile --input main.sql --output merged.sql --verbose --jinja2 --jinja2-vars vars.json\r\n```\r\n\r\n**After (Recommended)**:\r\n\r\nCreate `config.toml`:\r\n```toml\r\n[mergesourcefile]\r\ninput = \"main.sql\"\r\noutput = \"merged.sql\"\r\nverbose = true\r\njinja2 = true\r\njinja2_vars = \"vars.json\"\r\n```\r\n\r\nRun:\r\n```bash\r\nmergesourcefile --config config.toml\r\n```\r\n\r\n### Configuration File Location\r\n\r\nPlace your TOML configuration file:\r\n- In the root of your project directory\r\n- Or anywhere else and reference it with the full path: `mergesourcefile --config /path/to/config.toml`\r\n\r\nSee `config.example.toml` in the repository for a complete example with all available options.\r\n\r\n## Usage\r\n\r\n### Recommended: TOML Configuration File\r\n\r\n```bash\r\nmergesourcefile --config config.toml\r\n```\r\n\r\nSee the [TOML Configuration](#toml-configuration-recommended) section for detailed information.\r\n\r\n### Legacy: Command Line (Deprecated)\r\n\r\n\u26a0\ufe0f **Warning**: Command-line parameters will be deprecated in future versions. Please migrate to TOML configuration.\r\n\r\n```bash\r\nmergesourcefile --input input.sql --output output.sql\r\n```\r\n\r\n### Options (Legacy Command-Line)\r\n\r\n- `--config, -c`: **RECOMMENDED** - Load configuration from TOML file\r\n- `--input, -i`: Input SQL*Plus file to process (deprecated, use TOML config)\r\n- `--output, -o`: Output file where the result will be written (deprecated, use TOML config)\r\n- `--skip-var, -sv`: Skip variable substitution, only resolve file inclusions (deprecated, use TOML config)\r\n- `--verbose, -v`: Enable verbose mode for detailed processing information (deprecated, use TOML config)\r\n- `--jinja2`: Enable Jinja2 template processing (deprecated, use TOML config)\r\n- `--jinja2-vars`: JSON string with variables for Jinja2 template processing (deprecated, use TOML config)\r\n- `--processing-order`: Choose processing order: `default`, `jinja2_first`, or `includes_last` (deprecated, use TOML config)\r\n\r\n### TOML Configuration File Format\r\n\r\nCreate a `config.toml` file in your project directory:\r\n\r\n```toml\r\n[mergesourcefile]\r\ninput = \"main.sql\"\r\noutput = \"merged.sql\"\r\n\r\n# Optional parameters\r\nskip_var = false        # Set to true to skip variable substitution\r\nverbose = false         # Set to true for detailed processing information\r\njinja2 = false          # Set to true to enable Jinja2 template processing\r\njinja2_vars = \"\"        # Path to JSON file with Jinja2 variables\r\nprocessing_order = \"default\"  # Options: default, jinja2_first, includes_last\r\n```\r\n\r\nExample with Jinja2 support:\r\n\r\n```toml\r\n[mergesourcefile]\r\ninput = \"template.sql\"\r\noutput = \"output.sql\"\r\njinja2 = true\r\njinja2_vars = \"vars.json\"\r\nprocessing_order = \"jinja2_first\"\r\nverbose = true\r\n```\r\n\r\n### Examples\r\n\r\n1. **Recommended: Use TOML configuration**:\r\n   ```bash\r\n   mergesourcefile --config config.toml\r\n   ```\r\n\r\n2. **Legacy: Process a SQL file with full processing** (deprecated):\r\n   ```bash\r\n   mergesourcefile -i main.sql -o merged.sql\r\n   ```\r\n\r\n3. **Legacy: Process only file inclusions, skip variable substitution** (deprecated):\r\n   ```bash\r\n   mergesourcefile -i main.sql -o merged.sql --skip-var\r\n   ```\r\n\r\n4. **Legacy: Process with verbose output** (deprecated):\r\n   ```bash\r\n   mergesourcefile -i main.sql -o merged.sql --verbose\r\n   ```\r\n\r\n5. **Legacy: Process with Jinja2 template support** (deprecated):\r\n   ```bash\r\n   mergesourcefile -i template.sql -o merged.sql --jinja2\r\n   ```\r\n\r\n6. **Legacy: Process with Jinja2 variables** (deprecated):\r\n   ```bash\r\n   mergesourcefile -i template.sql -o merged.sql --jinja2 --jinja2-vars vars.json\r\n   ```\r\n\r\n7. **Legacy: Process with Jinja2-first processing order** (deprecated):\r\n   ```bash\r\n   mergesourcefile -i template.sql -o merged.sql --jinja2 --processing-order jinja2_first\r\n   ```\r\n\r\n## How It Works\r\n\r\n### File Inclusion\r\n\r\n- `@filename`: Includes a file relative to the original base path\r\n- `@@filename`: Includes a file relative to the current file's directory\r\n\r\n### Variable Substitution\r\n\r\n#### DEFINE Syntax (Enhanced in v1.1.1)\r\n- `DEFINE varname = 'quoted value';`: Defines with quoted value (supports spaces)\r\n- `DEFINE varname = unquoted_value;`: Defines with unquoted value (no spaces)\r\n- `DEFINE varname = 3.14;`: Supports decimal values\r\n- `DEFINE varname = ABC-123;`: Supports hyphenated values\r\n- `DEFINE varname = '';`: Supports empty string values\r\n\r\n#### Variable Usage\r\n- `&varname`: References a variable for substitution\r\n- `&varname..`: Variable concatenation with period\r\n- `UNDEFINE varname;`: Removes a variable definition\r\n\r\n#### Error Handling (v1.1.1)\r\n- Invalid DEFINE syntax is ignored and reported in verbose mode\r\n- Example: `DEFINE var = ;` will be skipped with a warning\r\n- Variables must be defined before use or an error is thrown\r\n\r\n### \ud83c\udd95 Jinja2 Template Processing\r\n\r\n#### Basic Template Syntax\r\n- `{{ variable }}`: Variable substitution\r\n- `{% if condition %}...{% endif %}`: Conditional blocks\r\n- `{% for item in list %}...{% endfor %}`: Loop blocks\r\n- `{# comment #}`: Template comments\r\n\r\n#### Custom Filters\r\n- `sql_escape`: Escapes single quotes for SQL safety\r\n  ```sql\r\n  SELECT * FROM users WHERE name = '{{ user_name | sql_escape }}';\r\n  ```\r\n- `strftime`: Formats datetime objects\r\n  ```sql\r\n  -- Generated on {{ now() | strftime('%Y-%m-%d %H:%M:%S') }}\r\n  ```\r\n\r\n#### Processing Orders\r\n1. **default**: File Inclusions \u2192 Jinja2 Templates \u2192 SQL Variables\r\n2. **jinja2_first**: Jinja2 Templates \u2192 File Inclusions \u2192 SQL Variables\r\n3. **includes_last**: SQL Variables \u2192 Jinja2 Templates \u2192 File Inclusions\r\n\r\n#### Dynamic File Inclusion Example\r\n```sql\r\n-- Using jinja2_first order to dynamically determine which files to include\r\n{% if environment == 'production' %}\r\n@prod_config.sql\r\n{% else %}\r\n@dev_config.sql\r\n{% endif %}\r\n```\r\n\r\n## Complete Example\r\n\r\n### Input Template (`template.sql`)\r\n```sql\r\n{# This is a Jinja2 comment #}\r\n-- Database setup for {{ environment | upper }} environment\r\n-- Generated on {{ now() | strftime('%Y-%m-%d %H:%M:%S') }}\r\n\r\n{% if environment == 'production' %}\r\n@production_settings.sql\r\n{% else %}\r\n@development_settings.sql\r\n{% endif %}\r\n\r\nDEFINE db_name = '{{ database_name }}';\r\nDEFINE table_prefix = '{{ table_prefix }}';\r\n\r\nCREATE TABLE &table_prefix._users (\r\n    id NUMBER PRIMARY KEY,\r\n    name VARCHAR2(100) NOT NULL,\r\n    email VARCHAR2(255) UNIQUE,\r\n    created_date DATE DEFAULT SYSDATE\r\n);\r\n\r\n{% for table in additional_tables %}\r\nCREATE TABLE &table_prefix._{{ table.name }} (\r\n    id NUMBER PRIMARY KEY,\r\n    {% for column in table.columns -%}\r\n    {{ column.name }} {{ column.type }}{% if not loop.last %},{% endif %}\r\n    {% endfor %}\r\n);\r\n{% endfor %}\r\n\r\n-- Insert sample data with escaped values\r\nINSERT INTO &table_prefix._users (name, email) \r\nVALUES ('{{ sample_user | sql_escape }}', '{{ sample_email | sql_escape }}');\r\n```\r\n\r\n### Command\r\n```bash\r\nmergesourcefile -i template.sql -o output.sql --jinja2 --processing-order jinja2_first --jinja2-vars '{\r\n  \"environment\": \"production\",\r\n  \"database_name\": \"MYAPP_DB\",\r\n  \"table_prefix\": \"APP\",\r\n  \"sample_user\": \"John O'\\''Brien\",\r\n  \"sample_email\": \"john@example.com\",\r\n  \"additional_tables\": [\r\n    {\r\n      \"name\": \"products\",\r\n      \"columns\": [\r\n        {\"name\": \"title\", \"type\": \"VARCHAR2(200)\"},\r\n        {\"name\": \"price\", \"type\": \"NUMBER(10,2)\"}\r\n      ]\r\n    }\r\n  ]\r\n}'\r\n```\r\n\r\n## Migration from v1.0.x\r\n\r\nIf you're upgrading from a previous version, your existing scripts will continue to work without any changes. The new Jinja2 functionality is **completely optional** and requires explicit activation with the `--jinja2` flag.\r\n\r\n### Backward Compatibility\r\n- All existing command-line options work exactly as before\r\n- File inclusion (`@`, `@@`) behavior is unchanged\r\n- Variable substitution (`DEFINE`, `UNDEFINE`) works as expected\r\n- No breaking changes to existing functionality\r\n\r\n### Gradual Adoption\r\nYou can gradually adopt Jinja2 features:\r\n1. Start with simple variable substitution: `{{ variable }}`\r\n2. Add conditional logic: `{% if condition %}`\r\n3. Use loops for repetitive structures: `{% for item in list %}`\r\n4. Apply custom filters: `{{ value | sql_escape }}`\r\n5. Experiment with processing orders for complex scenarios\r\n\r\n## Migration from v1.1.x to v1.2.0\r\n\r\n### Migrating to TOML Configuration\r\n\r\nThe new TOML configuration file approach offers a cleaner, more maintainable way to manage your MergeSourceFile settings. While command-line parameters are still supported, they will be deprecated in future versions.\r\n\r\n#### Step 1: Create a TOML Configuration File\r\n\r\nInstead of:\r\n```bash\r\nmergesourcefile -i main.sql -o output.sql --verbose --skip-var\r\n```\r\n\r\nCreate a `config.toml` file:\r\n```toml\r\n[mergesourcefile]\r\ninput = \"main.sql\"\r\noutput = \"output.sql\"\r\nverbose = true\r\nskip_var = true\r\n```\r\n\r\nThen run:\r\n```bash\r\nmergesourcefile --config config.toml\r\n```\r\n\r\n#### Step 2: Migrating Jinja2 Configurations\r\n\r\nFor projects using Jinja2, instead of:\r\n```bash\r\nmergesourcefile -i template.sql -o output.sql --jinja2 --jinja2-vars vars.json --processing-order jinja2_first\r\n```\r\n\r\nUse a TOML config:\r\n```toml\r\n[mergesourcefile]\r\ninput = \"template.sql\"\r\noutput = \"output.sql\"\r\njinja2 = true\r\njinja2_vars = \"vars.json\"\r\nprocessing_order = \"jinja2_first\"\r\n```\r\n\r\n#### Benefits of TOML Configuration\r\n\r\n1. **Version Control Friendly**: Configuration files can be committed to your repository\r\n2. **Project-Specific Settings**: Each project can have its own `config.toml`\r\n3. **Cleaner Scripts**: Simplifies build scripts and CI/CD pipelines\r\n4. **Self-Documenting**: Configuration files are easier to read and understand\r\n5. **No Shell Escaping**: Avoid issues with special characters in command-line parameters\r\n\r\n#### Deprecation Timeline\r\n\r\n- **v1.2.0**: TOML configuration introduced, deprecation warning added for command-line parameters\r\n- **v1.3.0** (planned): Increased warning severity\r\n- **v2.0.0** (future): Command-line parameters may be removed entirely\r\n\r\n## Best Practices\r\n\r\n### When to Use Each Processing Order\r\n\r\n- **default**: Best for most use cases where Jinja2 templates don't need to generate file inclusion directives\r\n- **jinja2_first**: Use when Jinja2 templates need to conditionally determine which files to include\r\n- **includes_last**: Use when you need SQL variables to be processed before Jinja2 templates and file inclusions\r\n\r\n### Security Considerations\r\n\r\nAlways use the `sql_escape` filter when inserting user-provided data:\r\n```sql\r\n-- \u274c Vulnerable to SQL injection\r\nSELECT * FROM users WHERE name = '{{ user_input }}';\r\n\r\n-- \u2705 Safe with sql_escape filter\r\nSELECT * FROM users WHERE name = '{{ user_input | sql_escape }}';\r\n```\r\n\r\n### Performance Tips\r\n\r\n- Use `--skip-var` if you don't need SQL variable processing\r\n- For large projects, consider splitting templates into smaller, focused files\r\n- Use Jinja2 comments `{# comment #}` instead of SQL comments for template-specific notes\r\n\r\n## Platform Compatibility\r\n\r\n### Operating Systems\r\n- \u2705 **Linux**: Full support with all features\r\n- \u2705 **macOS**: Full support with all features  \r\n- \u2705 **Windows**: Full support with enhanced compatibility (v1.1.1)\r\n  - Fixed Unicode encoding issues for CLI operations\r\n  - All 56 tests pass successfully on Windows systems\r\n  - Proper error codes and file path handling\r\n\r\n### Python Versions\r\n- Python 3.8+\r\n- Tested with Python 3.9, 3.10, 3.11, 3.12, 3.14\r\n\r\n### Character Encoding\r\n- Primary support: UTF-8 (recommended)\r\n- Windows compatibility: ASCII-safe output for CLI operations\r\n- All text files should use UTF-8 encoding for best results\r\n\r\n## Troubleshooting\r\n\r\n### Common Issues\r\n\r\n1. **DEFINE syntax errors** (Fixed in v1.1.1):\r\n   - \u2705 `DEFINE VAR = value` now works correctly (was broken in v1.1.0)\r\n   - \u2705 Both quoted and unquoted DEFINE values supported\r\n   - Use verbose mode (`--verbose`) to see ignored invalid DEFINE statements\r\n\r\n2. **Jinja2 syntax errors**: Ensure proper template syntax with matching braces and tags\r\n3. **Variable not found**: Check that all variables are provided via `--jinja2-vars`\r\n4. **File inclusion issues**: Verify file paths and choose appropriate processing order\r\n5. **Encoding problems** (Fixed in v1.1.1): \r\n   - \u2705 Windows encoding issues resolved\r\n   - Ensure all files use consistent encoding (UTF-8 recommended)\r\n   - CLI now works properly on all Windows systems\r\n\r\n### Windows-Specific Issues (Resolved in v1.1.1)\r\n- \u2705 **Unicode character display**: Fixed issues with special characters in CLI output\r\n- \u2705 **File path resolution**: Enhanced path handling for nested file inclusions\r\n- \u2705 **Exit codes**: CLI now returns proper error codes (1 for errors, 0 for success)\r\n\r\n### Debug Mode\r\n\r\nUse `--verbose` flag to see detailed processing information:\r\n```bash\r\nmergesourcefile -i template.sql -o output.sql --jinja2 --verbose\r\n```\r\n\r\n## License\r\n\r\nThis project is licensed under the MIT License.  \r\nYou are free to use, copy, modify, and distribute this software, provided that the copyright notice and this permission are included.  \r\nThe software is provided \"as is\", without warranty of any kind.\r\n\r\n## Author\r\n\r\nAlejandro G.\r\n\r\n## Contributing\r\n\r\nContributions are welcome! Please feel free to submit a Pull Request.\r\n",
    "bugtrack_url": null,
    "license": "MIT",
    "summary": "A Python tool to process SQL*Plus scripts with Jinja2 template support, resolving file inclusions (@, @@) and variable substitutions (DEFINE/UNDEFINE)",
    "version": "1.2.0",
    "project_urls": {
        "Homepage": "https://github.com/alegorico/MergeSourceFile",
        "Issues": "https://github.com/alegorico/MergeSourceFile/issues",
        "Repository": "https://github.com/alegorico/MergeSourceFile"
    },
    "split_keywords": [
        "sqlplus",
        " sql",
        " oracle",
        " script-processor",
        " file-merger"
    ],
    "urls": [
        {
            "comment_text": null,
            "digests": {
                "blake2b_256": "75944c1e7630f3d7b99902f84c4b6212a7c61b273f2855aaa073092b40354f68",
                "md5": "24bcecb96cc8abcc3eef6c407f1561fe",
                "sha256": "55f4d7b17cd35304920210f26c044ce81c2b14bf4a67e86ed8042dbaa186d0e2"
            },
            "downloads": -1,
            "filename": "mergesourcefile-1.2.0-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "24bcecb96cc8abcc3eef6c407f1561fe",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": ">=3.8",
            "size": 16529,
            "upload_time": "2025-10-20T21:36:54",
            "upload_time_iso_8601": "2025-10-20T21:36:54.855996Z",
            "url": "https://files.pythonhosted.org/packages/75/94/4c1e7630f3d7b99902f84c4b6212a7c61b273f2855aaa073092b40354f68/mergesourcefile-1.2.0-py3-none-any.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": null,
            "digests": {
                "blake2b_256": "d1b0b7b0bd49a4eaed5a0463ce59fa672434b985a3cb71f7b2000cb2f6f73a91",
                "md5": "09c40fba54d9669d6e850735b9e145c8",
                "sha256": "58b462f2c083d00eb27e4e90e88b515ee1beecae2d6ed75959eb50a2e97b8ee0"
            },
            "downloads": -1,
            "filename": "mergesourcefile-1.2.0.tar.gz",
            "has_sig": false,
            "md5_digest": "09c40fba54d9669d6e850735b9e145c8",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": ">=3.8",
            "size": 31494,
            "upload_time": "2025-10-20T21:36:56",
            "upload_time_iso_8601": "2025-10-20T21:36:56.812579Z",
            "url": "https://files.pythonhosted.org/packages/d1/b0/b7b0bd49a4eaed5a0463ce59fa672434b985a3cb71f7b2000cb2f6f73a91/mergesourcefile-1.2.0.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2025-10-20 21:36:56",
    "github": true,
    "gitlab": false,
    "bitbucket": false,
    "codeberg": false,
    "github_user": "alegorico",
    "github_project": "MergeSourceFile",
    "travis_ci": true,
    "coveralls": false,
    "github_actions": false,
    "circle": true,
    "lcname": "mergesourcefile"
}