ewms-pilot


Nameewms-pilot JSON
Version 0.26.8 PyPI version JSON
download
home_pageNone
SummaryEWMS's Pilot: A Job Pilot for Processing EWMS Events/Tasks
upload_time2024-10-29 21:23:10
maintainerNone
docs_urlNone
authorNone
requires_python<3.13,>=3.10
licenseMIT License Copyright (c) 2022 Observation Management Service Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
keywords pilot observation management service event workflow management service ewms message passing mq task
VCS
bugtrack_url
requirements No requirements were recorded.
Travis-CI No Travis.
coveralls test coverage No coveralls.
            <!--- Top of README Badges (automated) --->
[![PyPI](https://img.shields.io/pypi/v/ewms-pilot)](https://pypi.org/project/ewms-pilot/) [![GitHub release (latest by date including pre-releases)](https://img.shields.io/github/v/release/Observation-Management-Service/ewms-pilot?include_prereleases)](https://github.com/Observation-Management-Service/ewms-pilot/) [![Versions](https://img.shields.io/pypi/pyversions/ewms-pilot.svg)](https://pypi.org/project/ewms-pilot) [![PyPI - License](https://img.shields.io/pypi/l/ewms-pilot)](https://github.com/Observation-Management-Service/ewms-pilot/blob/main/LICENSE) [![GitHub issues](https://img.shields.io/github/issues/Observation-Management-Service/ewms-pilot)](https://github.com/Observation-Management-Service/ewms-pilot/issues?q=is%3Aissue+sort%3Aupdated-desc+is%3Aopen) [![GitHub pull requests](https://img.shields.io/github/issues-pr/Observation-Management-Service/ewms-pilot)](https://github.com/Observation-Management-Service/ewms-pilot/pulls?q=is%3Apr+sort%3Aupdated-desc+is%3Aopen)
<!--- End of README Badges (automated) --->

# ewms-pilot

An Event-Task Pilot for EWMS

The EWMS Pilot is a non-user-facing wrapper for task container instances in the Event Workflow Management System (EWMS), running on an HTCondor Execution Point (EP). The pilot:

- **Triggers task instances** for each inbound event.
- **Interfaces with EWMS events** as input/output files.
- **Isolates [task containers](#task-container)** from one another.
- **Provides fault tolerance** for failed tasks, CPUs, etc.

The following outlines what users need to know to operate within EWMS.

## Overview

The Pilot is designed to be invisible to users. However, some key details are necessary for running a [task container](#task-container):

### Task Container Overview

A **[task container](#task-container)** is created for each inbound event, it is defined by its image, arguments, and environment variables. See the [WMS docs](https://github.com/Observation-Management-Service/ewms-workflow-management-service#the-task-container) for information on setting these within EWMS.

#### Event I/O

An **input event** is provided to the task container as a file. The task container creates an **output event** by writing to a predetermined location.

The pilot provides the filepaths to the input and output files in two ways:

1. By replacing the placeholder strings, `{{INFILE}}` and `{{OUTFILE}}`, in the container's arguments at runtime.
2. By setting the task container's environment variables: `EWMS_TASK_INFILE` and `EWMS_TASK_OUTFILE`.

The files' extensions are configured by the pilot's environment variables, `EWMS_PILOT_INFILE_EXT` and `EWMS_PILOT_OUTFILE_EXT`: by default, these are `.in` and `.out`, respectively.

No other event or [message](#message-queue) handling is required by the task container.

### The Init Container

An **init container** is an optional, user-supplied image used to set up the environment, wait for conditions, or perform other preparatory actions before running task containers. It is configured using the `EWMS_PILOT_INIT_IMAGE`, `EWMS_PILOT_INIT_ARGS`, and `EWMS_PILOT_INIT_ENV_JSON` environment variables.

### File I/O

Task containers (and [init containers](#the-init-container)) can interact with external files in two ways:

#### Inter-Task Files

To transfer files between task containers, a shared directory is available to all task containers and the init container.

The pilot provides the filepath to the "data hub" in two ways:

1. By replacing the placeholder string, `{{DATA_HUB}}`, in the container's arguments at runtime.
2. By setting the task container's environment variable: `EWMS_TASK_DATA_HUB_DIR`.

**Note**:

- The data hub directory is writable, but there is no protection against race conditions for parallelized tasks.

#### External Files

Externally-mounted directories are supported in EWMS. See the [WMS documentation](https://github.com/Observation-Management-Service/ewms-workflow-management-service#task-file-io) for more details.

## EWMS Glossary Applied to the Pilot

### Workflow

_Does not exist within the Pilot._ _[Compare to WMS.](https://github.com/Observation-Management-Service/ewms-workflow-management-service#workflow)_

### Message Queue

The **message queue** is abstracted from the task container and can be ignored. _[Compare to WMS.](https://github.com/Observation-Management-Service/ewms-workflow-management-service#message-queue)_

#### Event

An **event** is an object transferred via [event I/O](#event-io). _[Compare to WMS.](https://github.com/Observation-Management-Service/ewms-workflow-management-service#event)_

### Task

In the context of the Pilot, the **task** is the runtime instance of the task image (a [task container](#task-container)) applied to an inbound event, possibly producing outbound events. _[Compare to WMS.](https://github.com/Observation-Management-Service/ewms-workflow-management-service#task)_

#### Task Container

The **task container** is an instance of a task image and is nearly synonymous with [task](#task).

### Task Directive

_Does not exist within the Pilot._ _[Compare to WMS.](https://github.com/Observation-Management-Service/ewms-workflow-management-service#task-directive)_

### Taskforce

_Does not exist within the Pilot._ _[Compare to WMS.](https://github.com/Observation-Management-Service/ewms-workflow-management-service#taskforce)_

            

Raw data

            {
    "_id": null,
    "home_page": null,
    "name": "ewms-pilot",
    "maintainer": null,
    "docs_url": null,
    "requires_python": "<3.13,>=3.10",
    "maintainer_email": null,
    "keywords": "pilot, Observation Management Service, Event Workflow Management Service, EWMS, message passing, MQ, task",
    "author": null,
    "author_email": "WIPAC Developers <developers@icecube.wisc.edu>",
    "download_url": "https://files.pythonhosted.org/packages/3e/22/98941cfd6c5fd2e2840cfb903205aca608065a79057b1f906545b9b4d663/ewms_pilot-0.26.8.tar.gz",
    "platform": null,
    "description": "<!--- Top of README Badges (automated) --->\n[![PyPI](https://img.shields.io/pypi/v/ewms-pilot)](https://pypi.org/project/ewms-pilot/) [![GitHub release (latest by date including pre-releases)](https://img.shields.io/github/v/release/Observation-Management-Service/ewms-pilot?include_prereleases)](https://github.com/Observation-Management-Service/ewms-pilot/) [![Versions](https://img.shields.io/pypi/pyversions/ewms-pilot.svg)](https://pypi.org/project/ewms-pilot) [![PyPI - License](https://img.shields.io/pypi/l/ewms-pilot)](https://github.com/Observation-Management-Service/ewms-pilot/blob/main/LICENSE) [![GitHub issues](https://img.shields.io/github/issues/Observation-Management-Service/ewms-pilot)](https://github.com/Observation-Management-Service/ewms-pilot/issues?q=is%3Aissue+sort%3Aupdated-desc+is%3Aopen) [![GitHub pull requests](https://img.shields.io/github/issues-pr/Observation-Management-Service/ewms-pilot)](https://github.com/Observation-Management-Service/ewms-pilot/pulls?q=is%3Apr+sort%3Aupdated-desc+is%3Aopen)\n<!--- End of README Badges (automated) --->\n\n# ewms-pilot\n\nAn Event-Task Pilot for EWMS\n\nThe EWMS Pilot is a non-user-facing wrapper for task container instances in the Event Workflow Management System (EWMS), running on an HTCondor Execution Point (EP). The pilot:\n\n- **Triggers task instances** for each inbound event.\n- **Interfaces with EWMS events** as input/output files.\n- **Isolates [task containers](#task-container)** from one another.\n- **Provides fault tolerance** for failed tasks, CPUs, etc.\n\nThe following outlines what users need to know to operate within EWMS.\n\n## Overview\n\nThe Pilot is designed to be invisible to users. However, some key details are necessary for running a [task container](#task-container):\n\n### Task Container Overview\n\nA **[task container](#task-container)** is created for each inbound event, it is defined by its image, arguments, and environment variables. See the [WMS docs](https://github.com/Observation-Management-Service/ewms-workflow-management-service#the-task-container) for information on setting these within EWMS.\n\n#### Event I/O\n\nAn **input event** is provided to the task container as a file. The task container creates an **output event** by writing to a predetermined location.\n\nThe pilot provides the filepaths to the input and output files in two ways:\n\n1. By replacing the placeholder strings, `{{INFILE}}` and `{{OUTFILE}}`, in the container's arguments at runtime.\n2. By setting the task container's environment variables: `EWMS_TASK_INFILE` and `EWMS_TASK_OUTFILE`.\n\nThe files' extensions are configured by the pilot's environment variables, `EWMS_PILOT_INFILE_EXT` and `EWMS_PILOT_OUTFILE_EXT`: by default, these are `.in` and `.out`, respectively.\n\nNo other event or [message](#message-queue) handling is required by the task container.\n\n### The Init Container\n\nAn **init container** is an optional, user-supplied image used to set up the environment, wait for conditions, or perform other preparatory actions before running task containers. It is configured using the `EWMS_PILOT_INIT_IMAGE`, `EWMS_PILOT_INIT_ARGS`, and `EWMS_PILOT_INIT_ENV_JSON` environment variables.\n\n### File I/O\n\nTask containers (and [init containers](#the-init-container)) can interact with external files in two ways:\n\n#### Inter-Task Files\n\nTo transfer files between task containers, a shared directory is available to all task containers and the init container.\n\nThe pilot provides the filepath to the \"data hub\" in two ways:\n\n1. By replacing the placeholder string, `{{DATA_HUB}}`, in the container's arguments at runtime.\n2. By setting the task container's environment variable: `EWMS_TASK_DATA_HUB_DIR`.\n\n**Note**:\n\n- The data hub directory is writable, but there is no protection against race conditions for parallelized tasks.\n\n#### External Files\n\nExternally-mounted directories are supported in EWMS. See the [WMS documentation](https://github.com/Observation-Management-Service/ewms-workflow-management-service#task-file-io) for more details.\n\n## EWMS Glossary Applied to the Pilot\n\n### Workflow\n\n_Does not exist within the Pilot._ _[Compare to WMS.](https://github.com/Observation-Management-Service/ewms-workflow-management-service#workflow)_\n\n### Message Queue\n\nThe **message queue** is abstracted from the task container and can be ignored. _[Compare to WMS.](https://github.com/Observation-Management-Service/ewms-workflow-management-service#message-queue)_\n\n#### Event\n\nAn **event** is an object transferred via [event I/O](#event-io). _[Compare to WMS.](https://github.com/Observation-Management-Service/ewms-workflow-management-service#event)_\n\n### Task\n\nIn the context of the Pilot, the **task** is the runtime instance of the task image (a [task container](#task-container)) applied to an inbound event, possibly producing outbound events. _[Compare to WMS.](https://github.com/Observation-Management-Service/ewms-workflow-management-service#task)_\n\n#### Task Container\n\nThe **task container** is an instance of a task image and is nearly synonymous with [task](#task).\n\n### Task Directive\n\n_Does not exist within the Pilot._ _[Compare to WMS.](https://github.com/Observation-Management-Service/ewms-workflow-management-service#task-directive)_\n\n### Taskforce\n\n_Does not exist within the Pilot._ _[Compare to WMS.](https://github.com/Observation-Management-Service/ewms-workflow-management-service#taskforce)_\n",
    "bugtrack_url": null,
    "license": "MIT License  Copyright (c) 2022 Observation Management Service  Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \"Software\"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:  The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.  THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ",
    "summary": "EWMS's Pilot: A Job Pilot for Processing EWMS Events/Tasks",
    "version": "0.26.8",
    "project_urls": {
        "Homepage": "https://pypi.org/project/ewms-pilot/",
        "Source": "https://github.com/Observation-Management-Service/ewms-pilot",
        "Tracker": "https://github.com/Observation-Management-Service/ewms-pilot/issues"
    },
    "split_keywords": [
        "pilot",
        " observation management service",
        " event workflow management service",
        " ewms",
        " message passing",
        " mq",
        " task"
    ],
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "331b80bf59bd42e5c36c4e65ebc67bae18ac962da7ae45fb7ce8045757024c16",
                "md5": "c6103f18ab9c7843c1fcfdc48e1bb1bc",
                "sha256": "9b5b94d268ede9ab42b7aaab1c79ffe718b1793418ce80e8ca52362f08b872f1"
            },
            "downloads": -1,
            "filename": "ewms_pilot-0.26.8-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "c6103f18ab9c7843c1fcfdc48e1bb1bc",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": "<3.13,>=3.10",
            "size": 27598,
            "upload_time": "2024-10-29T21:23:09",
            "upload_time_iso_8601": "2024-10-29T21:23:09.368778Z",
            "url": "https://files.pythonhosted.org/packages/33/1b/80bf59bd42e5c36c4e65ebc67bae18ac962da7ae45fb7ce8045757024c16/ewms_pilot-0.26.8-py3-none-any.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "3e2298941cfd6c5fd2e2840cfb903205aca608065a79057b1f906545b9b4d663",
                "md5": "fe71c11a39b58650f9944f13598c493f",
                "sha256": "aaa55d4ef71822d29dc59034e8e39c888d44cd2338187b84b36309cff657995b"
            },
            "downloads": -1,
            "filename": "ewms_pilot-0.26.8.tar.gz",
            "has_sig": false,
            "md5_digest": "fe71c11a39b58650f9944f13598c493f",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": "<3.13,>=3.10",
            "size": 30364,
            "upload_time": "2024-10-29T21:23:10",
            "upload_time_iso_8601": "2024-10-29T21:23:10.646584Z",
            "url": "https://files.pythonhosted.org/packages/3e/22/98941cfd6c5fd2e2840cfb903205aca608065a79057b1f906545b9b4d663/ewms_pilot-0.26.8.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2024-10-29 21:23:10",
    "github": true,
    "gitlab": false,
    "bitbucket": false,
    "codeberg": false,
    "github_user": "Observation-Management-Service",
    "github_project": "ewms-pilot",
    "travis_ci": false,
    "coveralls": false,
    "github_actions": true,
    "lcname": "ewms-pilot"
}
        
Elapsed time: 2.99087s