batchframe


Namebatchframe JSON
Version 0.0.1a13 PyPI version JSON
download
home_pagehttps://gitlab.com/Dzeri96/batchframe
SummaryBatteries-included framework for running repeatable tasks.
upload_time2024-10-13 17:58:52
maintainerNone
docs_urlNone
authorDzeri96
requires_python>=3.8
licenseGPL-3.0-or-later
keywords
VCS
bugtrack_url
requirements No requirements were recorded.
Travis-CI No Travis.
coveralls test coverage No coveralls.
            <!-- These are examples of badges you might want to add to your README:
     please update the URLs accordingly

[![Built Status](https://api.cirrus-ci.com/github/<USER>/batchframe.svg?branch=main)](https://cirrus-ci.com/github/<USER>/batchframe)
[![ReadTheDocs](https://readthedocs.org/projects/batchframe/badge/?version=latest)](https://batchframe.readthedocs.io/en/stable/)
[![Coveralls](https://img.shields.io/coveralls/github/<USER>/batchframe/main.svg)](https://coveralls.io/r/<USER>/batchframe)
[![PyPI-Server](https://img.shields.io/pypi/v/batchframe.svg)](https://pypi.org/project/batchframe/)
[![Conda-Forge](https://img.shields.io/conda/vn/conda-forge/batchframe.svg)](https://anaconda.org/conda-forge/batchframe)
[![Monthly Downloads](https://pepy.tech/badge/batchframe/month)](https://pepy.tech/project/batchframe)
[![Twitter](https://img.shields.io/twitter/url/http/shields.io.svg?style=social&label=Twitter)](https://twitter.com/batchframe)
-->

[![Project generated with PyScaffold](https://img.shields.io/badge/-PyScaffold-005CA0?logo=pyscaffold)](https://pyscaffold.org/)
![PyPI - Version](https://img.shields.io/pypi/v/batchframe)
![PyPI - Python Version](https://img.shields.io/pypi/pyversions/batchframe)

# ![logo](./assets/Batchframe_Logo_Small.png) Batchframe

> A python framework for decomposable, repetitive tasks.

This CLI tool/framework aims to provide out-of-the-box functionality for many common tasks one might have when building python scripts that do a simple task repeatedly.
It works on the "Don't Call Us, We'll Call You" principle.
You implement a few abstract classes and call the CLI tool on your project.
It runs your code and abstracts away much of the boilerplate you'd need to write.

Features include:
- Automatic capture of logs to files.
- Type-safe capture of CLI parameters.
- Ability to pause execution and inspect objects in the python shell.
- Colorful visualization of progress and similar statistics.
- Retry logic with backoff.
- Dependency injection.
- Pseudo-parallelism with AsyncIO.
- Fully-typed, class-based configuration.
- Saving of failed inputs for future re-execution.

![Execution example](./assets/example_execution.webm)

## Installation
```
pip install batchframe
```

## Features in Depth

### Automatic Capture of Logs to Files
Batchframe will save the logs of the current run under `OUTPUT_DIR/current_datetime/`,
where `OUTPUT_DIR` defaults to `batchframe_outputs`, but can be changed with the `-d` flag.

### Type-safe Capture of CLI Parameters
Usually any non-trivial python program requires some user input, for example, a path to a file that should be read.
Argv alone works for very simple cases, but very quickly one needs to start using [argparse](https://docs.python.org/3/library/argparse.html) to handle the complexity of user input.
The tool is as versatile as it gets, but is often too verbose for workloads batchframe is intended for.

We abstract this complexity away by providing a generic type called `BatchframeParam[T]`, where `T` is the type variable.
All one needs to do is to annotate the desired input with this type inside any constructor, and Batchframe will automatically ask for it when running.
When the required parameters are provided, they will be cast and injected automatically, as long as the class itself has an `@inject` annotation.

For example, let's say you want a `str` and an optional `datetime` parameter in your service.
You'd write the constructor like so:
```python
from batchframe import BatchframeParam, Service, inject
from datetime import datetime

@inject
class MyService(Service):
     def __init__(self, file_path: BatchframeParam[str], search_from: BatchframeParam[datetime] = datetime.now()):
          # Do some stuff here
```
You would then provide these values like so: `... -p file_path=./here.txt -p search_from 2024-01-03`.

This is also useful for overriding values in the `Configuration` class.

Currently, the list of supported injectable types is limited, but we're constantly adding more!

### Ability to Pause Execution and Inspect Objects in the Python Shell
Batchframe features a "pause shell" that allows the user to interrupt execution (Ctrl-C) and access all parts of running system through a fully-featured ipython shell.
This shell is also activated when a fatal error occurs, giving the user a chance to save the execution progress.

Execution can be completely stopped, while saving all failed/unprocessed work items by calling `self.stop()` inside the pause shell.

### Dependency Injection
**Keep in mind that this API is currently experimental and subject to change.**

Batchframe uses [kink](https://github.com/kodemore/kink) under the hood to automatically resolve dependencies between classes and inject configuration parameters.
In order for your class to be included in the DI system, decorate it with the `@inject` decorator like this:
```python
from batchframe import inject
from batchframe.models.service import Service

@inject()
class MyService(Service):
     pass
```
Batchframe automatically "aliases" all parent classes with the decorated class if they are not already set.
This means that `MyService` will be injected where ever `Service` is requested.

This is the same as using the decorator like so: `@inject(alias=Service)` and is sometimes required to be done manually.

### Fully-typed, Class-based Configuration
**Keep in mind that this API is currently experimental and subject to change.**

Instead of plain text files, Batchframe uses typed python dataclasses for configuration.
In theory, this makes configuration easier to work with, avoids duplication and improves flexibility.

Since there are still some kinks to work out with the API,
please refer to the `package_module` directory under `examples` for the latest working implementation of this system.

## Usage

### Writing a Project for Batchframe
Each Batchframe project consists of at least these two parts:
- A `Source` class, which provides the input data.
- A `Service` class, containing the execution logic.

Once you have implemented these two abstract classes provided by Batchframe,
its internal executor iterates through `Source` and gives the output to the `Service`.
This setup is extremely simple for the user and yet allows Batchframe to abstract away all of the repetitive tasks
associated with tools that work through a large number of inputs.

As an example, here's a very simple `Source`, which just emits numbers from one to ten:
```python
from batchframe import Source, inject
import logging

@inject
class TestSource(Source[int]):
    _curr_count = 0

    def __len__(self) -> int:
        return 10
    
    def next(self) -> int:
        self._curr_count += 1
        if self._curr_count <= 10:
            return self._curr_count
        else:
            raise StopIteration()

    def persist_failed(self, failed: list[int]):
        logging.error(f'Oh no! {failed} did not compute!')
```
A `Source` has just three methods that need to be implemented: `__len__`, `next` and `persist_failed`.
With these three functions, you can implement any finite source of task inputs,
for example reading from a database, a filesystem, a dataframe...
You also have the ability to save any inputs that caused issues during processing.

Continuing with the example, here's a very simple `Service` that just takes an input of any `int`,
multiplies it by itself, sleeps to simulate work, and logs the output:
```python
from batchframe import Service, inject
import asyncio
import logging

@inject
class TestService(Service):

    source: Source
    _exceptions_to_retry_for = {ValueError}
    
    @property
    def exceptions_to_retry_for(self) -> set[type[Exception]]:
        return self._exceptions_to_retry_for

    def __init__(self, source: Source) -> None:
        self.source = source

    async def process(self, input: int):
        await asyncio.sleep(0.1)
        logging.info(input*input)
        
    def finish(self):
        logging.info("I could do some cleanup here...")

```
A `Service` also has just three methods to implement: `__init__`, `process` and `finish`,
with the addition of a mandatory property `exceptions_to_retry_for`, written as a method.

Let's focus on the `process` method, as this is where the core of your project lies.
Here, you can implement essentially any processing logic that should be executed on the output of `Source`,
such as calling an API, doing transformations, saving to disk and so on.
As the method is a coroutine, using libraries that utilize `asyncio` is recommended for performance reasons.
The `process` method gets called multiple times "at the same time" in the event loop.

The `finish` method is just there to allow you do to some cleanup, for example closing of output streams.

The `exceptions_to_retry_for` is a list of exceptions that,
when thrown in `process` cause Batchframe to save the causing input and retry calling `process` at a later time.

#### Dependency Injection

You might have noticed the `@inject` annotation on both classes.
This tells Batchframe to include these two classes in its dependency injection process,
meaning that you just need to point the CLI at the right python script or directory,
and everything will be picked up automatically.
This also means that Batchframe will auto-inject any arguments into the constructor.
For example, you could ask the user for the path of a file to be read in `Source` by writing its constructor like so:
```python
from batchframe import Service, inject, BatchframeParam

@inject
class TestSource(Source[int]):

    def __init__(self, file_path: BatchframeParam[str]):
        # Load file here
     
     # Other mandatory methods...
```
Essentially any python class can be injected, meaning you can encapsulate logic into different services and call them
from any other class you want.

For more information on CLI parameter injection, refer to the "Type-safe Capture of CLI Parameters" section above.

### Configuration
Batchframe does away with envfiles and uses Python's  [dataclasses](https://docs.python.org/3/library/dataclasses.html) as vessels for config data.
Let's take a simplified project where we have two environments as an example.
The files on disk would look something like this:
```
my_simple_project/
├─ config/
│  ├─ config_dev.py
│  ├─ config_prod.py
│  ├─ abstract_custom_config.py
├─ source.py
├─ service.py
```

The class in `abstract_custom_config.py` would contain common properties for all implementations in the `config` directory.
For example something like this:
```python
from batchframe import Configuration
from dataclasses import dataclass
from abc import ABCMeta

@dataclass()
class AbstractCustomConfig(Configuration, metaclass=ABCMeta):
    env: str = "PLACEHOLDER"
```

The config implementations would look something this:
```python
from dataclasses import dataclass
from .abstract_custom_config import AbstractCustomConfig
from batchframe import inject

# file: config_dev.py
@inject
@dataclass()
class CustomDevConfig(AbstractCustomConfig):
     env: str = "DEV"

# file: config_prod.py
@inject
@dataclass()
class CustomProdConfig(AbstractCustomConfig):
     env: str = "PROD"
```
You have to point batchframe to which configuration you want to use with the `-c` flag.
For example `batchframe exec -c config_dev ...`.
Keep in mind that batchframe always looks in the `config` directory.
This might be changed in the future. 

Finally, one would use one of these configurations like so:

```python
from .config.abstract_custom_config import AbstractCustomConfig

@inject
class TestSource(Source[int]):

    def __init__(self, config: AbstractCustomConfig):
        # Use config here...
     
     # Other mandatory methods...
```

#### Static Configuration Parameters
The default Batchframe `Configuration` includes so-called static parameters.
These are injected by the runtime and are not meant to be overriden by the caller.
Static parameters are denoted by a leading underscore and include things like `_current_run_output_dir`,
which is built from the current run time and the directory supplied by the `-d` flag (`batchframe_outputs` by default).

We strongly suggest looking at the `examples` directory and calling Bachframe on those projects to see everything in action!

### CLI
Run `batchframe exec PATH_TO_MODULE --params param1=value1...`
where `PATH_TO_MODULE` is one of the following:
- A single python file containing all the necessary classes.
- A single python file in a directory-style project that imports all the necessary classes (usually your service file does this naturally).
- A directory containing an `__init__.py` file that imports all the necessary classes.

If you are using a directory-style project, supply the name of the desired configuration file with the `-c` flag.
This will automatically alias the built-in Batchframe `Configuration` class.
You should not include configuration files in `__init__.py` or the file you're pointing batchframe to. 

See the `examples` directory for inspiration.

<!-- pyscaffold-notes -->

## Development
This project uses [pipenv](https://pipenv.pypa.io/en/latest/) to make the management of dependencies in dev environments easier.
To create a virtual environment with all of the required dependencies, run `pipenv sync -d`.
When adding new runtime dependencies to `setup.cfg`, run `pipenv install && pipenv lock`.
When adding new dev dependencies to `setup.cfg`, you have to also add them to pipenv by running `pipenv install --dev DEPENDENCY`
Activate the virtual environment in your terminal with `pipenv shell`.

## Releasing
This project has dev and prod releases on TestPyPi and PyPi respectively.
Packages are built in the GitLab pipeline.

## Planned features/improvements
- Import entire directories without \_\_init__.py
- Support iterables for BatchframeParam
- Publish via the [trusted publisher workflow](https://docs.pypi.org/trusted-publishers/using-a-publisher/#gitlab-cicd).
- Add reasons for failed work items.
- Extract parameter descriptions from pydoc.
- Auto-generate UI.
- Have an actual multi-threading/multi-processing executor.
- Have default implementation of exceptions_to_retry_for
- Prevent shell autocompletion breaking with parameters
- `batchframe init` - create a template module-based project
- Explore using protocols instead of abstract classes?

### Debugging
You can find some debugging examples in the `.vscode/launch.json` file.
As the name suggests, these work out-of-the-box with Visual Studio Code.

### Known Bugs
- Updating the number of failed items doesn't always work. Looks like a race condition or a bug with the rich library.

## Note

This project has been set up using PyScaffold 4.5. For details and usage
information on PyScaffold see https://pyscaffold.org/.

            

Raw data

            {
    "_id": null,
    "home_page": "https://gitlab.com/Dzeri96/batchframe",
    "name": "batchframe",
    "maintainer": null,
    "docs_url": null,
    "requires_python": ">=3.8",
    "maintainer_email": null,
    "keywords": null,
    "author": "Dzeri96",
    "author_email": "dzeri96@proton.me",
    "download_url": "https://files.pythonhosted.org/packages/9a/a6/2ff6b6916273eea90aa287b1ada87fb02d8b78934a2fa85009a505afce75/batchframe-0.0.1a13.tar.gz",
    "platform": "any",
    "description": "<!-- These are examples of badges you might want to add to your README:\n     please update the URLs accordingly\n\n[![Built Status](https://api.cirrus-ci.com/github/<USER>/batchframe.svg?branch=main)](https://cirrus-ci.com/github/<USER>/batchframe)\n[![ReadTheDocs](https://readthedocs.org/projects/batchframe/badge/?version=latest)](https://batchframe.readthedocs.io/en/stable/)\n[![Coveralls](https://img.shields.io/coveralls/github/<USER>/batchframe/main.svg)](https://coveralls.io/r/<USER>/batchframe)\n[![PyPI-Server](https://img.shields.io/pypi/v/batchframe.svg)](https://pypi.org/project/batchframe/)\n[![Conda-Forge](https://img.shields.io/conda/vn/conda-forge/batchframe.svg)](https://anaconda.org/conda-forge/batchframe)\n[![Monthly Downloads](https://pepy.tech/badge/batchframe/month)](https://pepy.tech/project/batchframe)\n[![Twitter](https://img.shields.io/twitter/url/http/shields.io.svg?style=social&label=Twitter)](https://twitter.com/batchframe)\n-->\n\n[![Project generated with PyScaffold](https://img.shields.io/badge/-PyScaffold-005CA0?logo=pyscaffold)](https://pyscaffold.org/)\n![PyPI - Version](https://img.shields.io/pypi/v/batchframe)\n![PyPI - Python Version](https://img.shields.io/pypi/pyversions/batchframe)\n\n# ![logo](./assets/Batchframe_Logo_Small.png) Batchframe\n\n> A python framework for decomposable, repetitive tasks.\n\nThis CLI tool/framework aims to provide out-of-the-box functionality for many common tasks one might have when building python scripts that do a simple task repeatedly.\nIt works on the \"Don't Call Us, We'll Call You\" principle.\nYou implement a few abstract classes and call the CLI tool on your project.\nIt runs your code and abstracts away much of the boilerplate you'd need to write.\n\nFeatures include:\n- Automatic capture of logs to files.\n- Type-safe capture of CLI parameters.\n- Ability to pause execution and inspect objects in the python shell.\n- Colorful visualization of progress and similar statistics.\n- Retry logic with backoff.\n- Dependency injection.\n- Pseudo-parallelism with AsyncIO.\n- Fully-typed, class-based configuration.\n- Saving of failed inputs for future re-execution.\n\n![Execution example](./assets/example_execution.webm)\n\n## Installation\n```\npip install batchframe\n```\n\n## Features in Depth\n\n### Automatic Capture of Logs to Files\nBatchframe will save the logs of the current run under `OUTPUT_DIR/current_datetime/`,\nwhere `OUTPUT_DIR` defaults to `batchframe_outputs`, but can be changed with the `-d` flag.\n\n### Type-safe Capture of CLI Parameters\nUsually any non-trivial python program requires some user input, for example, a path to a file that should be read.\nArgv alone works for very simple cases, but very quickly one needs to start using [argparse](https://docs.python.org/3/library/argparse.html) to handle the complexity of user input.\nThe tool is as versatile as it gets, but is often too verbose for workloads batchframe is intended for.\n\nWe abstract this complexity away by providing a generic type called `BatchframeParam[T]`, where `T` is the type variable.\nAll one needs to do is to annotate the desired input with this type inside any constructor, and Batchframe will automatically ask for it when running.\nWhen the required parameters are provided, they will be cast and injected automatically, as long as the class itself has an `@inject` annotation.\n\nFor example, let's say you want a `str` and an optional `datetime` parameter in your service.\nYou'd write the constructor like so:\n```python\nfrom batchframe import BatchframeParam, Service, inject\nfrom datetime import datetime\n\n@inject\nclass MyService(Service):\n     def __init__(self, file_path: BatchframeParam[str], search_from: BatchframeParam[datetime] = datetime.now()):\n          # Do some stuff here\n```\nYou would then provide these values like so: `... -p file_path=./here.txt -p search_from 2024-01-03`.\n\nThis is also useful for overriding values in the `Configuration` class.\n\nCurrently, the list of supported injectable types is limited, but we're constantly adding more!\n\n### Ability to Pause Execution and Inspect Objects in the Python Shell\nBatchframe features a \"pause shell\" that allows the user to interrupt execution (Ctrl-C) and access all parts of running system through a fully-featured ipython shell.\nThis shell is also activated when a fatal error occurs, giving the user a chance to save the execution progress.\n\nExecution can be completely stopped, while saving all failed/unprocessed work items by calling `self.stop()` inside the pause shell.\n\n### Dependency Injection\n**Keep in mind that this API is currently experimental and subject to change.**\n\nBatchframe uses [kink](https://github.com/kodemore/kink) under the hood to automatically resolve dependencies between classes and inject configuration parameters.\nIn order for your class to be included in the DI system, decorate it with the `@inject` decorator like this:\n```python\nfrom batchframe import inject\nfrom batchframe.models.service import Service\n\n@inject()\nclass MyService(Service):\n     pass\n```\nBatchframe automatically \"aliases\" all parent classes with the decorated class if they are not already set.\nThis means that `MyService` will be injected where ever `Service` is requested.\n\nThis is the same as using the decorator like so: `@inject(alias=Service)` and is sometimes required to be done manually.\n\n### Fully-typed, Class-based Configuration\n**Keep in mind that this API is currently experimental and subject to change.**\n\nInstead of plain text files, Batchframe uses typed python dataclasses for configuration.\nIn theory, this makes configuration easier to work with, avoids duplication and improves flexibility.\n\nSince there are still some kinks to work out with the API,\nplease refer to the `package_module` directory under `examples` for the latest working implementation of this system.\n\n## Usage\n\n### Writing a Project for Batchframe\nEach Batchframe project consists of at least these two parts:\n- A `Source` class, which provides the input data.\n- A `Service` class, containing the execution logic.\n\nOnce you have implemented these two abstract classes provided by Batchframe,\nits internal executor iterates through `Source` and gives the output to the `Service`.\nThis setup is extremely simple for the user and yet allows Batchframe to abstract away all of the repetitive tasks\nassociated with tools that work through a large number of inputs.\n\nAs an example, here's a very simple `Source`, which just emits numbers from one to ten:\n```python\nfrom batchframe import Source, inject\nimport logging\n\n@inject\nclass TestSource(Source[int]):\n    _curr_count = 0\n\n    def __len__(self) -> int:\n        return 10\n    \n    def next(self) -> int:\n        self._curr_count += 1\n        if self._curr_count <= 10:\n            return self._curr_count\n        else:\n            raise StopIteration()\n\n    def persist_failed(self, failed: list[int]):\n        logging.error(f'Oh no! {failed} did not compute!')\n```\nA `Source` has just three methods that need to be implemented: `__len__`, `next` and `persist_failed`.\nWith these three functions, you can implement any finite source of task inputs,\nfor example reading from a database, a filesystem, a dataframe...\nYou also have the ability to save any inputs that caused issues during processing.\n\nContinuing with the example, here's a very simple `Service` that just takes an input of any `int`,\nmultiplies it by itself, sleeps to simulate work, and logs the output:\n```python\nfrom batchframe import Service, inject\nimport asyncio\nimport logging\n\n@inject\nclass TestService(Service):\n\n    source: Source\n    _exceptions_to_retry_for = {ValueError}\n    \n    @property\n    def exceptions_to_retry_for(self) -> set[type[Exception]]:\n        return self._exceptions_to_retry_for\n\n    def __init__(self, source: Source) -> None:\n        self.source = source\n\n    async def process(self, input: int):\n        await asyncio.sleep(0.1)\n        logging.info(input*input)\n        \n    def finish(self):\n        logging.info(\"I could do some cleanup here...\")\n\n```\nA `Service` also has just three methods to implement: `__init__`, `process` and `finish`,\nwith the addition of a mandatory property `exceptions_to_retry_for`, written as a method.\n\nLet's focus on the `process` method, as this is where the core of your project lies.\nHere, you can implement essentially any processing logic that should be executed on the output of `Source`,\nsuch as calling an API, doing transformations, saving to disk and so on.\nAs the method is a coroutine, using libraries that utilize `asyncio` is recommended for performance reasons.\nThe `process` method gets called multiple times \"at the same time\" in the event loop.\n\nThe `finish` method is just there to allow you do to some cleanup, for example closing of output streams.\n\nThe `exceptions_to_retry_for` is a list of exceptions that,\nwhen thrown in `process` cause Batchframe to save the causing input and retry calling `process` at a later time.\n\n#### Dependency Injection\n\nYou might have noticed the `@inject` annotation on both classes.\nThis tells Batchframe to include these two classes in its dependency injection process,\nmeaning that you just need to point the CLI at the right python script or directory,\nand everything will be picked up automatically.\nThis also means that Batchframe will auto-inject any arguments into the constructor.\nFor example, you could ask the user for the path of a file to be read in `Source` by writing its constructor like so:\n```python\nfrom batchframe import Service, inject, BatchframeParam\n\n@inject\nclass TestSource(Source[int]):\n\n    def __init__(self, file_path: BatchframeParam[str]):\n        # Load file here\n     \n     # Other mandatory methods...\n```\nEssentially any python class can be injected, meaning you can encapsulate logic into different services and call them\nfrom any other class you want.\n\nFor more information on CLI parameter injection, refer to the \"Type-safe Capture of CLI Parameters\" section above.\n\n### Configuration\nBatchframe does away with envfiles and uses Python's  [dataclasses](https://docs.python.org/3/library/dataclasses.html) as vessels for config data.\nLet's take a simplified project where we have two environments as an example.\nThe files on disk would look something like this:\n```\nmy_simple_project/\n\u251c\u2500 config/\n\u2502  \u251c\u2500 config_dev.py\n\u2502  \u251c\u2500 config_prod.py\n\u2502  \u251c\u2500 abstract_custom_config.py\n\u251c\u2500 source.py\n\u251c\u2500 service.py\n```\n\nThe class in `abstract_custom_config.py` would contain common properties for all implementations in the `config` directory.\nFor example something like this:\n```python\nfrom batchframe import Configuration\nfrom dataclasses import dataclass\nfrom abc import ABCMeta\n\n@dataclass()\nclass AbstractCustomConfig(Configuration, metaclass=ABCMeta):\n    env: str = \"PLACEHOLDER\"\n```\n\nThe config implementations would look something this:\n```python\nfrom dataclasses import dataclass\nfrom .abstract_custom_config import AbstractCustomConfig\nfrom batchframe import inject\n\n# file: config_dev.py\n@inject\n@dataclass()\nclass CustomDevConfig(AbstractCustomConfig):\n     env: str = \"DEV\"\n\n# file: config_prod.py\n@inject\n@dataclass()\nclass CustomProdConfig(AbstractCustomConfig):\n     env: str = \"PROD\"\n```\nYou have to point batchframe to which configuration you want to use with the `-c` flag.\nFor example `batchframe exec -c config_dev ...`.\nKeep in mind that batchframe always looks in the `config` directory.\nThis might be changed in the future. \n\nFinally, one would use one of these configurations like so:\n\n```python\nfrom .config.abstract_custom_config import AbstractCustomConfig\n\n@inject\nclass TestSource(Source[int]):\n\n    def __init__(self, config: AbstractCustomConfig):\n        # Use config here...\n     \n     # Other mandatory methods...\n```\n\n#### Static Configuration Parameters\nThe default Batchframe `Configuration` includes so-called static parameters.\nThese are injected by the runtime and are not meant to be overriden by the caller.\nStatic parameters are denoted by a leading underscore and include things like `_current_run_output_dir`,\nwhich is built from the current run time and the directory supplied by the `-d` flag (`batchframe_outputs` by default).\n\nWe strongly suggest looking at the `examples` directory and calling Bachframe on those projects to see everything in action!\n\n### CLI\nRun `batchframe exec PATH_TO_MODULE --params param1=value1...`\nwhere `PATH_TO_MODULE` is one of the following:\n- A single python file containing all the necessary classes.\n- A single python file in a directory-style project that imports all the necessary classes (usually your service file does this naturally).\n- A directory containing an `__init__.py` file that imports all the necessary classes.\n\nIf you are using a directory-style project, supply the name of the desired configuration file with the `-c` flag.\nThis will automatically alias the built-in Batchframe `Configuration` class.\nYou should not include configuration files in `__init__.py` or the file you're pointing batchframe to. \n\nSee the `examples` directory for inspiration.\n\n<!-- pyscaffold-notes -->\n\n## Development\nThis project uses [pipenv](https://pipenv.pypa.io/en/latest/) to make the management of dependencies in dev environments easier.\nTo create a virtual environment with all of the required dependencies, run `pipenv sync -d`.\nWhen adding new runtime dependencies to `setup.cfg`, run `pipenv install && pipenv lock`.\nWhen adding new dev dependencies to `setup.cfg`, you have to also add them to pipenv by running `pipenv install --dev DEPENDENCY`\nActivate the virtual environment in your terminal with `pipenv shell`.\n\n## Releasing\nThis project has dev and prod releases on TestPyPi and PyPi respectively.\nPackages are built in the GitLab pipeline.\n\n## Planned features/improvements\n- Import entire directories without \\_\\_init__.py\n- Support iterables for BatchframeParam\n- Publish via the [trusted publisher workflow](https://docs.pypi.org/trusted-publishers/using-a-publisher/#gitlab-cicd).\n- Add reasons for failed work items.\n- Extract parameter descriptions from pydoc.\n- Auto-generate UI.\n- Have an actual multi-threading/multi-processing executor.\n- Have default implementation of exceptions_to_retry_for\n- Prevent shell autocompletion breaking with parameters\n- `batchframe init` - create a template module-based project\n- Explore using protocols instead of abstract classes?\n\n### Debugging\nYou can find some debugging examples in the `.vscode/launch.json` file.\nAs the name suggests, these work out-of-the-box with Visual Studio Code.\n\n### Known Bugs\n- Updating the number of failed items doesn't always work. Looks like a race condition or a bug with the rich library.\n\n## Note\n\nThis project has been set up using PyScaffold 4.5. For details and usage\ninformation on PyScaffold see https://pyscaffold.org/.\n",
    "bugtrack_url": null,
    "license": "GPL-3.0-or-later",
    "summary": "Batteries-included framework for running repeatable tasks.",
    "version": "0.0.1a13",
    "project_urls": {
        "Homepage": "https://gitlab.com/Dzeri96/batchframe",
        "Source": "https://gitlab.com/Dzeri96/batchframe"
    },
    "split_keywords": [],
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "879ab0bb4321d80fe4ba69ecf0189b3e9d05581cc098699132471bb3066528de",
                "md5": "6785a3ed304bb7a263c7e00b099c490c",
                "sha256": "40972823b3d270abda5579583aa013642362829a556fb06ca019eb8ee87d77e3"
            },
            "downloads": -1,
            "filename": "batchframe-0.0.1a13-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "6785a3ed304bb7a263c7e00b099c490c",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": ">=3.8",
            "size": 33032,
            "upload_time": "2024-10-13T17:58:51",
            "upload_time_iso_8601": "2024-10-13T17:58:51.036671Z",
            "url": "https://files.pythonhosted.org/packages/87/9a/b0bb4321d80fe4ba69ecf0189b3e9d05581cc098699132471bb3066528de/batchframe-0.0.1a13-py3-none-any.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "9aa62ff6b6916273eea90aa287b1ada87fb02d8b78934a2fa85009a505afce75",
                "md5": "8deff3ae4db611fbb4b1342cc27509c7",
                "sha256": "e6f968dc355354c97ad2140cc2ba5bf6a76feda18bd3371e2953ebb632471f73"
            },
            "downloads": -1,
            "filename": "batchframe-0.0.1a13.tar.gz",
            "has_sig": false,
            "md5_digest": "8deff3ae4db611fbb4b1342cc27509c7",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": ">=3.8",
            "size": 36180,
            "upload_time": "2024-10-13T17:58:52",
            "upload_time_iso_8601": "2024-10-13T17:58:52.494061Z",
            "url": "https://files.pythonhosted.org/packages/9a/a6/2ff6b6916273eea90aa287b1ada87fb02d8b78934a2fa85009a505afce75/batchframe-0.0.1a13.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2024-10-13 17:58:52",
    "github": false,
    "gitlab": true,
    "bitbucket": false,
    "codeberg": false,
    "gitlab_user": "Dzeri96",
    "gitlab_project": "batchframe",
    "lcname": "batchframe"
}
        
Elapsed time: 0.33292s