# packtivity
[](https://zenodo.org/badge/latestdoi/53696818)
[](https://coveralls.io/github/diana-hep/packtivity)
[](http://packtivity.readthedocs.io/en/latest/?badge=latest)
[](https://pypi.python.org/pypi/packtivity)
This package aims to collect implementations of both synchronous and asynchronous execution of preserved, but parametrized scientific computational tasks that come with batteries included, i.e. with a full specification of their software dependencies. In that sense they are *packaged activities* -- packtivities.
This package provides tools to validate and execute data processing tasks that are written according to the "packtivity" JSON schemas defined in https://github.com/diana-hep/yadage-schemas.
Packtivities define
* the software environment
* parametrized process descriptions (what programs to run within these environment) and
* produces human and machine readable outputs (as JSON) of the resulting data fragments.
At run-time they are paired with a concrete set of parameters supplied as JSON documents and and external storage/state to actually execute these tasks.
## Packtivity in Yadage
This package is used by https://github.com/lukasheinrich/yadage to execute the individual steps of yadage workflows.
## Example Packtivity spec
This packtivity spec is part of a number of yadage workflow and runs the Delphes detector simulation on a HepMC file and outputs events in the LHCO and ROOT file formats. This packtivity is (stored in a public location)[https://github.com/lukasheinrich/yadage-workflows/blob/master/phenochain/delphes.yml] from which it can be later retrieved:
process:
process_type: 'string-interpolated-cmd'
cmd: 'DelphesHepMC {delphes_card} {outputroot} {inputhepmc} && root2lhco {outputroot} {outputlhco}'
publisher:
publisher_type: 'frompar-pub'
outputmap:
lhcofile: outputlhco
rootfile: outputroot
environment:
environment_type: 'docker-encapsulated'
image: lukasheinrich/root-delphes
## Usage
You can run the packtivity in a synchronous way by specifying the spec (can point to GitHub), all necessary parameters and attaching an external state (via the `--read` and `--write` flags).
packtivity-run -t from-github/phenochain delphes.yml \
-p inputhepmc="$PWD/pythia/output.hepmc" \
-p outputroot="'{workdir}/output.root'" \
-p outputlhco="'{workdir}/output.lhco'" \
-p delphes_card=delphes/cards/delphes_card_ATLAS.tcl \
--read pythia --write outdir
## Asynchronous Backends
In order to facilitate usage of distributed resources, a number of Asynchronous
backends can be specified. Here is an example for IPython Parallel clusters
packtivity-run -b ipcluster --asyncwait \
-t from-github/phenochain delphes.yml \
-p inputhepmc="$PWD/pythia/output.hepmc" \
-p outputroot="'{workdir}/output.root'" \
-p outputlhco="'{workdir}/output.lhco'" \
-p delphes_card=delphes/cards/delphes_card_ATLAS.tcl \
--read pythia --write outdir
You can replacing the `--asyncwait` with `--async` flag in order to get a JSONable proxy representation with which to later on check on the job status. By default the proxy information is written to `proxy.json` (customizable via the `-x` flag):
packtivity-run -b celery --async \
-t from-github/phenochain delphes.yml \
-p inputhepmc="$PWD/pythia/output.hepmc" \
-p outputroot="'{workdir}/output.root'" \
-p outputlhco="'{workdir}/output.lhco'" \
-p delphes_card=delphes/cards/delphes_card_ATLAS.tcl \
--read pythia --write outdir
And at a later point in time you can check via:
packtivity-checkproxy proxy.json
## External Backends
Users can implement their own backends to handle the JSON documents describing the packtivities. It can be enabled
by using the `fromenv` backend and setting an environment variable specifying the module holding the backend and proxy
classes. The format of the environment variable is `module:backendclass:proxyclass`. E.g.:
export PACKTIVITY_ASYNCBACKEND="externalbackend:ExternalBackend:ExternalProxy"
Raw data
{
"_id": null,
"home_page": "https://github.com/yadage/packtivity",
"name": "packtivity",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": "",
"keywords": "",
"author": "Lukas Heinrich",
"author_email": "lukas.heinrich@cern.ch",
"download_url": "https://files.pythonhosted.org/packages/b6/03/be3d6c135420ddddf613da4abd771df735a5a269166d0acf76461bc2dc70/packtivity-0.16.2.tar.gz",
"platform": null,
"description": "# packtivity\n\n[](https://zenodo.org/badge/latestdoi/53696818)\n[](https://coveralls.io/github/diana-hep/packtivity)\n[](http://packtivity.readthedocs.io/en/latest/?badge=latest)\n[](https://pypi.python.org/pypi/packtivity)\n\nThis package aims to collect implementations of both synchronous and asynchronous execution of preserved, but parametrized scientific computational tasks that come with batteries included, i.e. with a full specification of their software dependencies. In that sense they are *packaged activities* -- packtivities.\n\nThis package provides tools to validate and execute data processing tasks that are written according to the \"packtivity\" JSON schemas defined in https://github.com/diana-hep/yadage-schemas.\n\nPacktivities define\n\n* the software environment\n* parametrized process descriptions (what programs to run within these environment) and\n* produces human and machine readable outputs (as JSON) of the resulting data fragments.\n\nAt run-time they are paired with a concrete set of parameters supplied as JSON documents and and external storage/state to actually execute these tasks.\n\n## Packtivity in Yadage\n\nThis package is used by https://github.com/lukasheinrich/yadage to execute the individual steps of yadage workflows.\n\n## Example Packtivity spec\n\nThis packtivity spec is part of a number of yadage workflow and runs the Delphes detector simulation on a HepMC file and outputs events in the LHCO and ROOT file formats. This packtivity is (stored in a public location)[https://github.com/lukasheinrich/yadage-workflows/blob/master/phenochain/delphes.yml] from which it can be later retrieved:\n\n process:\n process_type: 'string-interpolated-cmd'\n cmd: 'DelphesHepMC {delphes_card} {outputroot} {inputhepmc} && root2lhco {outputroot} {outputlhco}'\n publisher:\n publisher_type: 'frompar-pub'\n outputmap:\n lhcofile: outputlhco\n rootfile: outputroot\n environment:\n environment_type: 'docker-encapsulated'\n image: lukasheinrich/root-delphes\n\n## Usage\n\nYou can run the packtivity in a synchronous way by specifying the spec (can point to GitHub), all necessary parameters and attaching an external state (via the `--read` and `--write` flags).\n\n packtivity-run -t from-github/phenochain delphes.yml \\\n -p inputhepmc=\"$PWD/pythia/output.hepmc\" \\\n -p outputroot=\"'{workdir}/output.root'\" \\\n -p outputlhco=\"'{workdir}/output.lhco'\" \\\n -p delphes_card=delphes/cards/delphes_card_ATLAS.tcl \\\n --read pythia --write outdir\n\n## Asynchronous Backends\n\nIn order to facilitate usage of distributed resources, a number of Asynchronous\nbackends can be specified. Here is an example for IPython Parallel clusters\n\n packtivity-run -b ipcluster --asyncwait \\\n -t from-github/phenochain delphes.yml \\\n -p inputhepmc=\"$PWD/pythia/output.hepmc\" \\\n -p outputroot=\"'{workdir}/output.root'\" \\\n -p outputlhco=\"'{workdir}/output.lhco'\" \\\n -p delphes_card=delphes/cards/delphes_card_ATLAS.tcl \\\n --read pythia --write outdir\n\nYou can replacing the `--asyncwait` with `--async` flag in order to get a JSONable proxy representation with which to later on check on the job status. By default the proxy information is written to `proxy.json` (customizable via the `-x` flag):\n\n packtivity-run -b celery --async \\\n -t from-github/phenochain delphes.yml \\\n -p inputhepmc=\"$PWD/pythia/output.hepmc\" \\\n -p outputroot=\"'{workdir}/output.root'\" \\\n -p outputlhco=\"'{workdir}/output.lhco'\" \\\n -p delphes_card=delphes/cards/delphes_card_ATLAS.tcl \\\n --read pythia --write outdir\n\nAnd at a later point in time you can check via:\n\n packtivity-checkproxy proxy.json\n\n## External Backends\n\nUsers can implement their own backends to handle the JSON documents describing the packtivities. It can be enabled\nby using the `fromenv` backend and setting an environment variable specifying the module holding the backend and proxy\nclasses. The format of the environment variable is `module:backendclass:proxyclass`. E.g.:\n\n export PACKTIVITY_ASYNCBACKEND=\"externalbackend:ExternalBackend:ExternalProxy\"\n",
"bugtrack_url": null,
"license": "",
"summary": "packtivity - general purpose schema + bindings for PROV activities",
"version": "0.16.2",
"project_urls": {
"Homepage": "https://github.com/yadage/packtivity"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "0ec0da8e93a5c0efa4c015adaea96433077af9b36ae1b2015aa65e19acd6e475",
"md5": "b74a41e5effa76f82283a38b76f07f2f",
"sha256": "3bc53aac52753ddee347196fc364eac1928ecacfb3a6e9d5a915813d64d8d0f4"
},
"downloads": -1,
"filename": "packtivity-0.16.2-py3-none-any.whl",
"has_sig": false,
"md5_digest": "b74a41e5effa76f82283a38b76f07f2f",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 36314,
"upload_time": "2023-09-19T00:25:13",
"upload_time_iso_8601": "2023-09-19T00:25:13.114004Z",
"url": "https://files.pythonhosted.org/packages/0e/c0/da8e93a5c0efa4c015adaea96433077af9b36ae1b2015aa65e19acd6e475/packtivity-0.16.2-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "b603be3d6c135420ddddf613da4abd771df735a5a269166d0acf76461bc2dc70",
"md5": "2048e4207c1de8b4d1e6c55ec3dc3e9a",
"sha256": "1171e7f7f933a312b2d558e2b614e1634ebb96e7602822b23056d22e8728a316"
},
"downloads": -1,
"filename": "packtivity-0.16.2.tar.gz",
"has_sig": false,
"md5_digest": "2048e4207c1de8b4d1e6c55ec3dc3e9a",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 34768,
"upload_time": "2023-09-19T00:25:14",
"upload_time_iso_8601": "2023-09-19T00:25:14.973284Z",
"url": "https://files.pythonhosted.org/packages/b6/03/be3d6c135420ddddf613da4abd771df735a5a269166d0acf76461bc2dc70/packtivity-0.16.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-09-19 00:25:14",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "yadage",
"github_project": "packtivity",
"travis_ci": true,
"coveralls": false,
"github_actions": true,
"lcname": "packtivity"
}
JSON
