# Spawn Parallel Instances
Spawn Parallel Instances is a versatile tool designed to facilitate running multiple instances of various types of scripts with customizable environment variables and arguments. By leveraging its command-line options, you can efficiently manage and execute scripts in parallel, making it an essential tool for developers and system administrators.
## Features
- Run multiple instances of a script concurrently.
- Load environment variables from a YAML file.
- Pass arguments to each process via a json file
- Specify the interpreter to use.
- Log output for each process with unique identifiers.
## Installation
You can install the package using pip:
```sh
pip install spawn_parallel_instances
```
## Usage
To use the concurrent executor, run the following command:
```sh
spawn_parallel_instances <script_name> --processes <num_processes> --env_file <env_file> --interpreter <interpreter> --script-type <script_type>
```
## Arguments
### Command-Line Arguments
The `ScriptRunner` accepts the following command-line arguments:
1. **script** (positional, required):
- **Description:** The script to run.
- **Type:** `str`
- **Example:** `my_script.py` or `my_script.js`
2. **--processes** (optional):
- **Description:** Number of processes to spawn.
- **Type:** `int`
- **Default:** `1`
- **Example:** `--processes 4`
3. **--env-file** (optional):
- **Description:** YAML file containing environment variables.
- **Type:** `str`
- **Default:** `env_variables.yml`
- **Example:** `--env-file config.yml`
4. **--script-type** (optional):
- **Description:** Script type (either `python`, `nodejs`, `go`).
- **Type:** `str`
- **Choices:** `["python", "nodejs", "go"]`
- **Default:** `python`
- **Example:** `--script-type nodejs`
5. **--interpreter** (optional):
- **Description:** Interpreter to use (Python interpreter or node).
- **Type:** `str`
- **Default:** If not specified, defaults to `python3.11` for Python scripts or `node` for Node.js scripts or `go run` for go.
- **Example:** `--interpreter python3.8` or `--interpreter node`
6. **--script-args-file** (optional):
- **Description:** JSON file containing script arguments for each process.
- **Type:** `str`
- **Default:** `""`
- **Example:** `--script-args-file args.json`
7. **--script-args** (optional):
- **Description:** Arguments to pass to each script instance, comma-separated for each instance.
- **Type:** `str`
- **Default:** `""`
- **Example:** `--script-args "arg1,arg2,arg3"`
### Environment Variables
Environment variables should be defined in a YAML file. For example:
```yaml
FOO: bar
BAZ: qux
```
The above YAML file can be used with the --env_file argument to set environment variables for the script.
### Arguments
The JSON file specified with `--script-args-file` should contain a list of argument sets, one for each process instance. The number of argument sets in the JSON file must match the number of processes specified.
#### Json file format
```json
[
["arg1", "arg2", "arg3"],
["arg4", "arg5", "arg6"],
["arg7", "arg8", "arg9"]
]
```
### Logging
Each process will log its output to a file with a unique identifier. The log files are named based on the script name, process number, and a unique UUID.
### Example
Assuming you have a script called [`tests/sample/test_script.py`](tests/sample/test_script.py) and an environment file [`tests/sample/config.yml`](tests/sample/config.yml), you can run:
```sh
spawn_parallel_instances tests/sample/test_script.py --processes 3 --env_file tests/sample/config.yml --script-type python --interpreter python3.9
```
## Use Cases
### Running a Python Script with Default Settings:
```bash
spawn_parallel_instances my_script.py
```
### Running a Node.js Script with 4 Processes:
```bash
spawn_parallel_instances my_script.js --script-type nodejs --processes 4
```
### Running a Go Script with Environment Variables:
```bash
spawn_parallel_instances my_script.go --script-type go --env-file my_env.yml
```
### Running a Python Script with Specific Interpreter and Arguments:
```bash
spawn_parallel_instances my_script.py --interpreter python3.9 --script-args-file args.json
```
### Running a Python Script with Direct Arguments:
```bash
spawn_parallel_instances my_script.py --script-args "arg1,arg2,arg3,arg4"
```
## Troubleshooting
If you encounter issues while running ScriptRunner, consider the following tips:
### Interpreter Not Found:
Ensure that the interpreter specified in --interpreter or derived from --script-type is correctly installed and available in your PATH.
### Incorrect JSON Format:
Verify that the JSON file used for --script-args-file is correctly formatted as a list of lists. Each inner list should correspond to the arguments for one process instance.
### Environment Variables Not Loaded:
Check if the YAML file specified with --env-file exists and is correctly formatted. Ensure it contains valid key-value pairs.
### Script Execution Errors:
Review the log files generated for each process to identify any specific errors related to script execution.
## Contributing
Contributions are welcome! Please follow these steps:
- Fork the repository.
- Create a new branch for your feature or bug fix.
- Make your changes.
- Write tests for your changes.
- Ensure all tests pass.
- Submit a pull request.
### Development
To set up a development environment:
#### Clone the repository:
```sh
git clone https://github.com/stenzr/spawn_parallel_instances
cd spawn_parallel_instances
```
#### Create a virtual environment and activate it:
```sh
Copy code
python3.9 -m venv venv
source venv/bin/activate
```
#### Install the package in editable mode with development dependencies:
```sh
Copy code
pip install -e .[test]
```
## License
This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
Raw data
{
"_id": null,
"home_page": "https://github.com/stenzr/spawn_parallel_instances",
"name": "spawn-parallel-instances",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.6",
"maintainer_email": null,
"keywords": null,
"author": "Rohit Kumar",
"author_email": "rkrohitkr01@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/69/01/01bf7b616e85e8778a9df80a82d40fe874a8429c4065c11d9901c8396006/spawn_parallel_instances-1.1.0.tar.gz",
"platform": null,
"description": "# Spawn Parallel Instances\n\nSpawn Parallel Instances is a versatile tool designed to facilitate running multiple instances of various types of scripts with customizable environment variables and arguments. By leveraging its command-line options, you can efficiently manage and execute scripts in parallel, making it an essential tool for developers and system administrators.\n\n## Features\n\n- Run multiple instances of a script concurrently.\n- Load environment variables from a YAML file.\n- Pass arguments to each process via a json file\n- Specify the interpreter to use.\n- Log output for each process with unique identifiers.\n\n## Installation\n\nYou can install the package using pip:\n\n```sh\npip install spawn_parallel_instances\n```\n\n## Usage\n\nTo use the concurrent executor, run the following command:\n\n```sh\nspawn_parallel_instances <script_name> --processes <num_processes> --env_file <env_file> --interpreter <interpreter> --script-type <script_type>\n```\n\n## Arguments\n\n### Command-Line Arguments\n\nThe `ScriptRunner` accepts the following command-line arguments:\n\n1. **script** (positional, required):\n - **Description:** The script to run.\n - **Type:** `str`\n - **Example:** `my_script.py` or `my_script.js`\n\n2. **--processes** (optional):\n - **Description:** Number of processes to spawn.\n - **Type:** `int`\n - **Default:** `1`\n - **Example:** `--processes 4`\n\n3. **--env-file** (optional):\n - **Description:** YAML file containing environment variables.\n - **Type:** `str`\n - **Default:** `env_variables.yml`\n - **Example:** `--env-file config.yml`\n\n4. **--script-type** (optional):\n - **Description:** Script type (either `python`, `nodejs`, `go`).\n - **Type:** `str`\n - **Choices:** `[\"python\", \"nodejs\", \"go\"]`\n - **Default:** `python`\n - **Example:** `--script-type nodejs`\n\n5. **--interpreter** (optional):\n - **Description:** Interpreter to use (Python interpreter or node).\n - **Type:** `str`\n - **Default:** If not specified, defaults to `python3.11` for Python scripts or `node` for Node.js scripts or `go run` for go.\n - **Example:** `--interpreter python3.8` or `--interpreter node`\n\n6. **--script-args-file** (optional):\n - **Description:** JSON file containing script arguments for each process.\n - **Type:** `str`\n - **Default:** `\"\"`\n - **Example:** `--script-args-file args.json`\n\n7. **--script-args** (optional):\n - **Description:** Arguments to pass to each script instance, comma-separated for each instance.\n - **Type:** `str`\n - **Default:** `\"\"`\n - **Example:** `--script-args \"arg1,arg2,arg3\"`\n\n### Environment Variables\n\nEnvironment variables should be defined in a YAML file. For example:\n\n```yaml\nFOO: bar\nBAZ: qux\n```\n\nThe above YAML file can be used with the --env_file argument to set environment variables for the script.\n\n### Arguments\n\nThe JSON file specified with `--script-args-file` should contain a list of argument sets, one for each process instance. The number of argument sets in the JSON file must match the number of processes specified.\n\n#### Json file format\n\n```json\n[\n [\"arg1\", \"arg2\", \"arg3\"],\n [\"arg4\", \"arg5\", \"arg6\"],\n [\"arg7\", \"arg8\", \"arg9\"]\n]\n```\n\n### Logging\nEach process will log its output to a file with a unique identifier. The log files are named based on the script name, process number, and a unique UUID.\n\n### Example\nAssuming you have a script called [`tests/sample/test_script.py`](tests/sample/test_script.py) and an environment file [`tests/sample/config.yml`](tests/sample/config.yml), you can run:\n\n```sh\nspawn_parallel_instances tests/sample/test_script.py --processes 3 --env_file tests/sample/config.yml --script-type python --interpreter python3.9\n```\n## Use Cases\n\n### Running a Python Script with Default Settings:\n\n```bash\nspawn_parallel_instances my_script.py\n```\n\n### Running a Node.js Script with 4 Processes:\n\n```bash\nspawn_parallel_instances my_script.js --script-type nodejs --processes 4\n```\n\n### Running a Go Script with Environment Variables:\n\n```bash\nspawn_parallel_instances my_script.go --script-type go --env-file my_env.yml\n```\n\n### Running a Python Script with Specific Interpreter and Arguments:\n\n```bash\nspawn_parallel_instances my_script.py --interpreter python3.9 --script-args-file args.json\n```\n### Running a Python Script with Direct Arguments:\n\n```bash\nspawn_parallel_instances my_script.py --script-args \"arg1,arg2,arg3,arg4\"\n```\n\n## Troubleshooting\n\nIf you encounter issues while running ScriptRunner, consider the following tips:\n\n### Interpreter Not Found:\n\nEnsure that the interpreter specified in --interpreter or derived from --script-type is correctly installed and available in your PATH.\n\n### Incorrect JSON Format:\n\nVerify that the JSON file used for --script-args-file is correctly formatted as a list of lists. Each inner list should correspond to the arguments for one process instance.\n\n### Environment Variables Not Loaded:\n\nCheck if the YAML file specified with --env-file exists and is correctly formatted. Ensure it contains valid key-value pairs.\n\n### Script Execution Errors:\n\nReview the log files generated for each process to identify any specific errors related to script execution.\n\n\n## Contributing\nContributions are welcome! Please follow these steps:\n\n- Fork the repository.\n- Create a new branch for your feature or bug fix.\n- Make your changes.\n- Write tests for your changes.\n- Ensure all tests pass.\n- Submit a pull request.\n\n### Development\nTo set up a development environment:\n\n#### Clone the repository:\n\n```sh\ngit clone https://github.com/stenzr/spawn_parallel_instances\ncd spawn_parallel_instances\n```\n\n#### Create a virtual environment and activate it:\n\n```sh\nCopy code\npython3.9 -m venv venv\nsource venv/bin/activate\n```\n\n#### Install the package in editable mode with development dependencies:\n\n```sh\nCopy code\npip install -e .[test]\n```\n\n## License\n\nThis project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.\n\n\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "A package to run multiple instances of the same script.",
"version": "1.1.0",
"project_urls": {
"Homepage": "https://github.com/stenzr/spawn_parallel_instances"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "a884e0c9fdd0926a8f27d320e84f74f64d0e8fd3cc73d64cb51f25c7c9c46bde",
"md5": "dacb9a8492977e8f1980213ce8863564",
"sha256": "a1c65e533a3d50ed559912ab32e1d6888b90d319d4b9675da4695d3bf0330026"
},
"downloads": -1,
"filename": "spawn_parallel_instances-1.1.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "dacb9a8492977e8f1980213ce8863564",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.6",
"size": 7425,
"upload_time": "2024-07-28T09:21:46",
"upload_time_iso_8601": "2024-07-28T09:21:46.346872Z",
"url": "https://files.pythonhosted.org/packages/a8/84/e0c9fdd0926a8f27d320e84f74f64d0e8fd3cc73d64cb51f25c7c9c46bde/spawn_parallel_instances-1.1.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "690101bf7b616e85e8778a9df80a82d40fe874a8429c4065c11d9901c8396006",
"md5": "dccd85c1384dd44a5727ed7500d87734",
"sha256": "b4526ddecc4d6df6a81ad60248e2d1171723f9c95337006f771cf73e89d16285"
},
"downloads": -1,
"filename": "spawn_parallel_instances-1.1.0.tar.gz",
"has_sig": false,
"md5_digest": "dccd85c1384dd44a5727ed7500d87734",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.6",
"size": 7004,
"upload_time": "2024-07-28T09:21:47",
"upload_time_iso_8601": "2024-07-28T09:21:47.461256Z",
"url": "https://files.pythonhosted.org/packages/69/01/01bf7b616e85e8778a9df80a82d40fe874a8429c4065c11d9901c8396006/spawn_parallel_instances-1.1.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-07-28 09:21:47",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "stenzr",
"github_project": "spawn_parallel_instances",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [],
"lcname": "spawn-parallel-instances"
}
JSON
