<!-- These are examples of badges you might want to add to your README:
please update the URLs accordingly
[](https://cirrus-ci.com/github/<USER>/batchframe)
[](https://batchframe.readthedocs.io/en/stable/)
[](https://coveralls.io/r/<USER>/batchframe)
[](https://pypi.org/project/batchframe/)
[](https://anaconda.org/conda-forge/batchframe)
[](https://pepy.tech/project/batchframe)
[](https://twitter.com/batchframe)
-->
[](https://pyscaffold.org/)
#  Batchframe
> A framework for small, repeated tasks.
This CLI tool/framework aims to provide out-of-the-box functionality for many common tasks one might have when building python scripts that do a simple task repeatedly.
Features include:
- Automatic capture of logs to files.
- Type-safe capture of CLI parameters.
- Ability to pause execution and inspect objects in the python shell.
- Colorful visualization of progress and similar statistics.
- Retry logic with backoff.
- Dependency injection.
- Pseudo-parallelism with AsyncIO.
- Fully-typed, class-based configuration.
- Saving of failed inputs for future re-execution.
## Features in Depth
### Automatic Capture of Logs to Files
Batchframe will save the logs of the current run under `OUTPUT_DIR/current_datetime/`,
where `OUTPUT_DIR` defaults to `batchframe_outputs`, but can be changed with the `-d` flag.
### Type-safe Capture of CLI Parameters
Usually any non-trivial python program requires some user input, for example, a path to a file that should be read.
Argv alone works for very simple cases, but very quickly one needs to start using [argparse](https://docs.python.org/3/library/argparse.html) to handle the complexity of user input.
The tool is as versatile as it gets, but is often too verbose for workloads batchframe is intended for.
We abstract this complexity away by providing a generic type called `BatchframeParam[T]`, where `T` is the type variable.
All one needs to do is to annotate the desired input with this type inside any constructor, and Batchframe will automatically ask for it when running.
When the required parameters are provided, they will be cast and injected automatically, as long as the class itself has an `@inject` annotation.
For example, let's say you want a `str` and an optional `datetime` parameter in your service.
You'd write the constructor like so:
```python
from batchframe import BatchframeParam, Service, inject
from datetime import datetime
@inject
class MyService(Service):
def __init__(self, file_path: BatchframeParam[str], search_from: BatchframeParam[datetime] = datetime.now()):
# Do some stuff here
```
You would then provide these values like so: `... -p file_path=./here.txt -p search_from 2024-01-03`.
This is also useful for overriding values in the `Configuration` class.
Currently, the list of supported injectable types is limited, but we're constantly adding more!
### Ability to Pause Execution and Inspect Objects in the Python Shell
Batchframe features a "pause shell" that allows the user to interrupt execution (Ctrl-C) and access all parts of running system through a fully-featured ipython shell.
This shell is also activated when a fatal error occurs, giving the user a chance to save the execution progress.
Execution can be completely stopped, while saving all failed/unprocessed work items by calling `self.stop()` inside the pause shell.
### Dependency Injection
**Keep in mind that this API is currently experimental and subject to change.**
Batchframe uses [kink](https://github.com/kodemore/kink) under the hood to automatically resolve dependencies between classes and inject configuration parameters.
In order for your class to be included in the DI system, decorate it with the `@inject` decorator like this:
```python
from batchframe import inject
from batchframe.models.service import Service
@inject()
class MyService(Service):
pass
```
Batchframe automatically "aliases" all parent classes with the decorated class if they are not already set.
This means that `MyService` will be injected where ever `Service` is requested.
This is the same as using the decorator like so: `@inject(alias=Service)` and is sometimes required to be done manually.
### Fully-typed, Class-based Configuration
**Keep in mind that this API is currently experimental and subject to change.**
Instead of plain text files, Batchframe uses typed python dataclasses for configuration.
In theory, this makes configuration easier to work with, avoids duplication and improves flexibility.
Since there are still some kinks to work out with the API,
please refer to the `package_module` directory under `examples` for the latest working implementation of this system.
## Usage
### CLI
Run `batchframe exec PATH_TO_MODULE --params param1=value1...`
where `PATH_TO_MODULE` is one of the following:
- A single python file containing all the necessary classes.
- A single python file in a directory-style project that imports all the necessary classes (usually your service file does this naturally).
- A directory containing an `__init__.py` file that imports all the necessary classes.
If you are using a directory-style project, supply the name of the desired configuration file with the `-c` flag.
This will automatically alias the built-in Batchframe `Configuration` class.
You should not include configuration files in `__init__.py` or the file you're pointing batchframe to.
See the `examples` directory for inspiration.
<!-- pyscaffold-notes -->
## Development
This project uses [pipenv](https://pipenv.pypa.io/en/latest/) to make the management of dependencies in dev environments easier.
To create a virtual environment with all of the required dependencies, run `pipenv sync -d`.
When adding new runtime dependencies to `setup.cfg`, run `pipenv install && pipenv lock`.
When adding new dev dependencies to `setup.cfg`, you have to also add them to pipenv by running `pipenv install --dev DEPENDENCY`
Activate the virtual environment in your terminal with `pipenv shell`.
## Releasing
This project has dev and prod releases on TestPyPi and PyPi respectively.
Packages are built in the GitLab pipeline.
## Planned features/improvements
- Import entire directories without \_\_init__.py
- Support iterables for BatchframeParam
- Publish via the [trusted publisher workflow](https://docs.pypi.org/trusted-publishers/using-a-publisher/#gitlab-cicd).
- Add reasons for failed work items.
- Extract parameter descriptions from pydoc.
- Auto-generate UI.
- Have an actual multi-threading/multi-processing executor.
### Debugging
You can find some debugging examples in the `.vscode/launch.json` file.
As the name suggests, these work out-of-the-box with Visual Studio Code.
### Known Bugs
- Updating the number of failed items doesn't always work. Looks like a race condition or a bug with the rich library.
## Note
This project has been set up using PyScaffold 4.5. For details and usage
information on PyScaffold see https://pyscaffold.org/.
Raw data
{
"_id": null,
"home_page": "https://gitlab.com/Dzeri96/batchframe",
"name": "batchframe",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": null,
"author": "Dzeri96",
"author_email": "dzeri96@proton.me",
"download_url": "https://files.pythonhosted.org/packages/e6/43/2ad8e0c6740359d9a3b3e897a101aedab1dc5e0d1ee3b652281ffbac11db/batchframe-0.0.1a10.tar.gz",
"platform": "any",
"description": "<!-- These are examples of badges you might want to add to your README:\n please update the URLs accordingly\n\n[](https://cirrus-ci.com/github/<USER>/batchframe)\n[](https://batchframe.readthedocs.io/en/stable/)\n[](https://coveralls.io/r/<USER>/batchframe)\n[](https://pypi.org/project/batchframe/)\n[](https://anaconda.org/conda-forge/batchframe)\n[](https://pepy.tech/project/batchframe)\n[](https://twitter.com/batchframe)\n-->\n\n[](https://pyscaffold.org/)\n\n\n\n#  Batchframe\n\n> A framework for small, repeated tasks.\n\nThis CLI tool/framework aims to provide out-of-the-box functionality for many common tasks one might have when building python scripts that do a simple task repeatedly.\nFeatures include:\n- Automatic capture of logs to files.\n- Type-safe capture of CLI parameters.\n- Ability to pause execution and inspect objects in the python shell.\n- Colorful visualization of progress and similar statistics.\n- Retry logic with backoff.\n- Dependency injection.\n- Pseudo-parallelism with AsyncIO.\n- Fully-typed, class-based configuration.\n- Saving of failed inputs for future re-execution.\n\n## Features in Depth\n\n### Automatic Capture of Logs to Files\nBatchframe will save the logs of the current run under `OUTPUT_DIR/current_datetime/`,\nwhere `OUTPUT_DIR` defaults to `batchframe_outputs`, but can be changed with the `-d` flag.\n\n### Type-safe Capture of CLI Parameters\nUsually any non-trivial python program requires some user input, for example, a path to a file that should be read.\nArgv alone works for very simple cases, but very quickly one needs to start using [argparse](https://docs.python.org/3/library/argparse.html) to handle the complexity of user input.\nThe tool is as versatile as it gets, but is often too verbose for workloads batchframe is intended for.\n\nWe abstract this complexity away by providing a generic type called `BatchframeParam[T]`, where `T` is the type variable.\nAll one needs to do is to annotate the desired input with this type inside any constructor, and Batchframe will automatically ask for it when running.\nWhen the required parameters are provided, they will be cast and injected automatically, as long as the class itself has an `@inject` annotation.\n\nFor example, let's say you want a `str` and an optional `datetime` parameter in your service.\nYou'd write the constructor like so:\n```python\nfrom batchframe import BatchframeParam, Service, inject\nfrom datetime import datetime\n\n@inject\nclass MyService(Service):\n def __init__(self, file_path: BatchframeParam[str], search_from: BatchframeParam[datetime] = datetime.now()):\n # Do some stuff here\n```\nYou would then provide these values like so: `... -p file_path=./here.txt -p search_from 2024-01-03`.\n\nThis is also useful for overriding values in the `Configuration` class.\n\nCurrently, the list of supported injectable types is limited, but we're constantly adding more!\n\n### Ability to Pause Execution and Inspect Objects in the Python Shell\nBatchframe features a \"pause shell\" that allows the user to interrupt execution (Ctrl-C) and access all parts of running system through a fully-featured ipython shell.\nThis shell is also activated when a fatal error occurs, giving the user a chance to save the execution progress.\n\nExecution can be completely stopped, while saving all failed/unprocessed work items by calling `self.stop()` inside the pause shell.\n\n### Dependency Injection\n**Keep in mind that this API is currently experimental and subject to change.**\n\nBatchframe uses [kink](https://github.com/kodemore/kink) under the hood to automatically resolve dependencies between classes and inject configuration parameters.\nIn order for your class to be included in the DI system, decorate it with the `@inject` decorator like this:\n```python\nfrom batchframe import inject\nfrom batchframe.models.service import Service\n\n@inject()\nclass MyService(Service):\n pass\n```\nBatchframe automatically \"aliases\" all parent classes with the decorated class if they are not already set.\nThis means that `MyService` will be injected where ever `Service` is requested.\n\nThis is the same as using the decorator like so: `@inject(alias=Service)` and is sometimes required to be done manually.\n\n### Fully-typed, Class-based Configuration\n**Keep in mind that this API is currently experimental and subject to change.**\n\nInstead of plain text files, Batchframe uses typed python dataclasses for configuration.\nIn theory, this makes configuration easier to work with, avoids duplication and improves flexibility.\n\nSince there are still some kinks to work out with the API,\nplease refer to the `package_module` directory under `examples` for the latest working implementation of this system.\n\n## Usage\n\n### CLI\nRun `batchframe exec PATH_TO_MODULE --params param1=value1...`\nwhere `PATH_TO_MODULE` is one of the following:\n- A single python file containing all the necessary classes.\n- A single python file in a directory-style project that imports all the necessary classes (usually your service file does this naturally).\n- A directory containing an `__init__.py` file that imports all the necessary classes.\n\nIf you are using a directory-style project, supply the name of the desired configuration file with the `-c` flag.\nThis will automatically alias the built-in Batchframe `Configuration` class.\nYou should not include configuration files in `__init__.py` or the file you're pointing batchframe to. \n\nSee the `examples` directory for inspiration.\n\n<!-- pyscaffold-notes -->\n\n## Development\nThis project uses [pipenv](https://pipenv.pypa.io/en/latest/) to make the management of dependencies in dev environments easier.\nTo create a virtual environment with all of the required dependencies, run `pipenv sync -d`.\nWhen adding new runtime dependencies to `setup.cfg`, run `pipenv install && pipenv lock`.\nWhen adding new dev dependencies to `setup.cfg`, you have to also add them to pipenv by running `pipenv install --dev DEPENDENCY`\nActivate the virtual environment in your terminal with `pipenv shell`.\n\n## Releasing\nThis project has dev and prod releases on TestPyPi and PyPi respectively.\nPackages are built in the GitLab pipeline.\n\n## Planned features/improvements\n- Import entire directories without \\_\\_init__.py\n- Support iterables for BatchframeParam\n- Publish via the [trusted publisher workflow](https://docs.pypi.org/trusted-publishers/using-a-publisher/#gitlab-cicd).\n- Add reasons for failed work items.\n- Extract parameter descriptions from pydoc.\n- Auto-generate UI.\n- Have an actual multi-threading/multi-processing executor.\n\n### Debugging\nYou can find some debugging examples in the `.vscode/launch.json` file.\nAs the name suggests, these work out-of-the-box with Visual Studio Code.\n\n### Known Bugs\n- Updating the number of failed items doesn't always work. Looks like a race condition or a bug with the rich library.\n\n## Note\n\nThis project has been set up using PyScaffold 4.5. For details and usage\ninformation on PyScaffold see https://pyscaffold.org/.\n",
"bugtrack_url": null,
"license": "GPL-3.0-or-later",
"summary": "Batteries-included framework for running repeatable tasks.",
"version": "0.0.1a10",
"project_urls": {
"Homepage": "https://gitlab.com/Dzeri96/batchframe",
"Source": "https://gitlab.com/Dzeri96/batchframe"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "acd478c737d23f6fefe415a8e131d9a20620874aac32747908b12cba8f017e26",
"md5": "f261b0a7e7138943802ea4d04ee57d2b",
"sha256": "1818bb797059ff24df15cf950ec76bb69f2348c85b2065e161a9365f3da04896"
},
"downloads": -1,
"filename": "batchframe-0.0.1a10-py3-none-any.whl",
"has_sig": false,
"md5_digest": "f261b0a7e7138943802ea4d04ee57d2b",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 30123,
"upload_time": "2024-09-15T12:31:28",
"upload_time_iso_8601": "2024-09-15T12:31:28.082539Z",
"url": "https://files.pythonhosted.org/packages/ac/d4/78c737d23f6fefe415a8e131d9a20620874aac32747908b12cba8f017e26/batchframe-0.0.1a10-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "e6432ad8e0c6740359d9a3b3e897a101aedab1dc5e0d1ee3b652281ffbac11db",
"md5": "b8d75ea2840f1e6fe6f3c6b37adb3cbb",
"sha256": "edb83c4fa10023a4a5bd8506ee4cbfd821b03054e3fdbebabf8e3c355bc72f92"
},
"downloads": -1,
"filename": "batchframe-0.0.1a10.tar.gz",
"has_sig": false,
"md5_digest": "b8d75ea2840f1e6fe6f3c6b37adb3cbb",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 31207,
"upload_time": "2024-09-15T12:31:29",
"upload_time_iso_8601": "2024-09-15T12:31:29.707824Z",
"url": "https://files.pythonhosted.org/packages/e6/43/2ad8e0c6740359d9a3b3e897a101aedab1dc5e0d1ee3b652281ffbac11db/batchframe-0.0.1a10.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-09-15 12:31:29",
"github": false,
"gitlab": true,
"bitbucket": false,
"codeberg": false,
"gitlab_user": "Dzeri96",
"gitlab_project": "batchframe",
"lcname": "batchframe"
}
JSON