# Asimov
Asimov is a workflow management and automation platform for scientific analyses.
[Documentation](https://asimov.docs.ligo.org/asimov) · [Releases](https://git.ligo.org/asimov/asimov/-/releases) · [Issue Tracker](https://git.ligo.org/asimov/asimov/-/issues)
[](https://anaconda.org/conda-forge/ligo-asimov)
[](https://conda.anaconda.org/conda-forge)
[](https://git.ligo.org/asimov/asimov/-/commits/infrastructure-updates)
Asimov was developed to manage and automate the parameter estimation analyses used by the LIGO, Virgo, and KAGRA collaborations to analyse gravitational wave signals, but it aims to provide tools which can be used for other workflows.
Asimov has been used to organise and run the major catalogue analyses from the third observing run, O3, but it's designed to be flexible enough to allow new pipelines and analyses to be added to the framework.
## Branch notes
These notes relate to in-development features on this branch, and what's described here is only expected to be relevant during development.
More generally useful documentation will move to the main documentation before moving to production.
### Starting the logging server
Run in ``asimov`` directory:
```
export FLASK_APP=server
flask run
```
## Features
### Job monitoring and management
Asimov is able to interact with high throughput job management tools, and can submit jobs to clusters, monitor them for problems, and initiate post-processing tasks.
### Uniform pipeline interface
Asimov provides an API layer which allows a single configuration to be deployed to numerous different analysis pipelines.
Current gravitational wave pipelines which are supported are ``lalinference``, ``bayeswave``, ``RIFT``, and ``bilby``.
### Centralised configuration
Asimov records all ongoing, completed, and scheduled analyses, allowing jobs, configurations, and results to be found easily.
### Reporting overview
Asimov can provide both machine-readible and human-friendly reports of all jobs it is monitoring, while collating relevant log files and outputs.
### Results management
Your results are important, and Asimov provides special tools to help manage the outputs of analyses as well as ensuring their veracity.
## Do I need Asimov?
Asimov makes setting-up and running parameter estimation jobs easier.
It can generate configuration files for several parameter estimation pipelines, and handle submitting these to a cluster.
Whether you're setting-up a preliminary analysis for a single gravitational wave event, or analysing hundreds of events for a catalog, Asimov can help.
## Installing Asimov
Asimov is written in Python, and is available on ``pypi``.
It can be installed by running
```
$ pip install asimov
```
It is also available on conda, and can be installed by running
```
$ conda install -c conda-forge ligo-asimov
```
Asimov also requires that you have `git` installed on your machine, and that you've set it up by running:
```
$ git config --global user.email "you@example.com"
$ git config --global user.name "Your Name"
```
## Get started
Asimov supports a variety of different ways of running, but the simplest way, running a workflow on a local machine, can be set up with a single command.
We start by setting up a project, which is a directory which keeps all of the analyses and the required metadata together.
A project can include just a single event or a whole selection of events, for example if you're producing a catalogue.
First create a directory to store your project:
```
$ mkdir my-new-project
$ cd my-new-project
```
and then get asimov to set things up
```
$ asimov init "Test project"
```
where you can replace `"Test project"` with the name you want to give your project.
A project will be set-up in your current working directory.
In order to start setting up analyses we next need to download some default settings.
An analysis is a pipeline run, and asimov supports `bayeswave`, `bilby` in the default installation.
We'll download the default configurations for jobs which are going to be run on the LIGO data grid.
We do this using the `asimov apply` command, which pulls-in data from a file either locally or online.
```
$ asimov apply -f https://git.ligo.org/asimov/data/-/raw/main/defaults/production-pe.yaml
```
and the load default priors the same way:
```
$ asimov apply -f https://git.ligo.org/asimov/data/-/raw/main/defaults/production-pe-priors.yaml
```
Now you can add an existing event, by downloading the event data settings using the `asimov apply` function, for example, to add GW150914 to the project you can run
```
$ asimov apply -f https://git.ligo.org/asimov/data/-/raw/main/events/gwtc-2-1/GW150914_095045.yaml
```
This downloads the settings which were used for analysing GW150914 for the GWTC-2.1 catalogue paper, and stores them in the ledger file in the project (`.asimov/ledger.yml`).
Many analyses can be run on a single event (these are called "productions" in asimov parlence).
We can add some pre-configured analyses by downloading some analysis configuration settings.
```
$ asimov apply -f https://git.ligo.org/asimov/data/-/raw/main/analyses/production-default.yaml -e GW150914_095045
```
Note that if you omit the `-e` argument asimov will ask which event the analyses should be applied to.
You can now build and submit your jobs to the cluster.
First use `asimov manage build` to create the configuration files for each analysis:
```
$ asimov manage build
```
These will be added to the git repositories in the `checkouts` directory inside the project directory.
You then submit the jobs to the cluster by running
```
$ asimov manage submit
```
It will normally take a long time for a parameter estimation job to finish, but you can get asimov to check up on it by running
```
$ asimov monitor
```
If the job is finished `asimov` will start post-processing using `PESummary`, and if it's fallen over it will attempt to fix the problem and resubmit it to the cluster.
If the post-processing has finished it will move the results to a read-only directory.
You can also set the asimov monitor to automatically check the status of the job every 15 minutes by running
```
$ asimov start
```
which will automatically start any post-processing, and start any jobs once their dependencies are met.
You can stop this automatic monitoring process any time by running
```
$ asimov stop
```
For a full description of the workflow management process see the documentation.
## I want to help develop new features, or add a new pipeline
Great! We're always looking for help with developing asimov!
Please take a look at our [contributors' guide](CONTRIBUTING.rst) to get started!
## Roadmap
### Gravitic pipelines
While Asimov already supports a large number of pre-existing pipelines, and provides a straightforward interface for adding new pipelines, we also intend to support pipelines constructed using [gravitic](https://github.com/transientlunatic/gravitic), allowing experimental tools to be used without constructing an entire new pipeline, while also allowing asimov to manage the training of machine learning algorithms.
### Workflow replication, extension and duplication
Asimov will allow an existing workflow to be duplicated, in a similar way to a ``git clone``, and then extended, with new jobs gaining access to the completed jobs in the workflow.
It will also allow entire workflows to be re-run, providing a straightforward way to replicate results, or make minor modifications.
## Authors
Asimov is made by the LIGO, Virgo, and KAGRA collaborations.
The primary maintainer of the project is Daniel Williams.
Its development is supported by the Science and Technology Facilities Council.
Raw data
{
"_id": null,
"home_page": "https://git.ligo.org/asimov/asimov",
"name": "asimov",
"maintainer": "",
"docs_url": null,
"requires_python": "",
"maintainer_email": "",
"keywords": "pe,ligo,asimov",
"author": "Daniel Williams",
"author_email": "daniel.williams@ligo.org",
"download_url": "https://files.pythonhosted.org/packages/2a/77/af0cf8a1169f31e647044584515b345a5de9ada782f7fa6c7f1f9c8a69f5/asimov-0.5.6.tar.gz",
"platform": null,
"description": "# Asimov\n\nAsimov is a workflow management and automation platform for scientific analyses.\n\n[Documentation](https://asimov.docs.ligo.org/asimov) \u00b7 [Releases](https://git.ligo.org/asimov/asimov/-/releases) \u00b7 [Issue Tracker](https://git.ligo.org/asimov/asimov/-/issues)\n\n[](https://anaconda.org/conda-forge/ligo-asimov) \n[](https://conda.anaconda.org/conda-forge)\n\n[](https://git.ligo.org/asimov/asimov/-/commits/infrastructure-updates)\n\nAsimov was developed to manage and automate the parameter estimation analyses used by the LIGO, Virgo, and KAGRA collaborations to analyse gravitational wave signals, but it aims to provide tools which can be used for other workflows.\n\nAsimov has been used to organise and run the major catalogue analyses from the third observing run, O3, but it's designed to be flexible enough to allow new pipelines and analyses to be added to the framework.\n\n## Branch notes\n\nThese notes relate to in-development features on this branch, and what's described here is only expected to be relevant during development.\nMore generally useful documentation will move to the main documentation before moving to production.\n\n### Starting the logging server\n\nRun in ``asimov`` directory:\n\n```\nexport FLASK_APP=server\nflask run\n```\n\n## Features\n\n### Job monitoring and management\n\nAsimov is able to interact with high throughput job management tools, and can submit jobs to clusters, monitor them for problems, and initiate post-processing tasks.\n\n### Uniform pipeline interface\n\nAsimov provides an API layer which allows a single configuration to be deployed to numerous different analysis pipelines.\nCurrent gravitational wave pipelines which are supported are ``lalinference``, ``bayeswave``, ``RIFT``, and ``bilby``.\n\n### Centralised configuration\n\nAsimov records all ongoing, completed, and scheduled analyses, allowing jobs, configurations, and results to be found easily.\n\n### Reporting overview\n\nAsimov can provide both machine-readible and human-friendly reports of all jobs it is monitoring, while collating relevant log files and outputs.\n\n### Results management\n\nYour results are important, and Asimov provides special tools to help manage the outputs of analyses as well as ensuring their veracity.\n\n## Do I need Asimov?\n\nAsimov makes setting-up and running parameter estimation jobs easier.\nIt can generate configuration files for several parameter estimation pipelines, and handle submitting these to a cluster.\nWhether you're setting-up a preliminary analysis for a single gravitational wave event, or analysing hundreds of events for a catalog, Asimov can help.\n\n## Installing Asimov\n\nAsimov is written in Python, and is available on ``pypi``. \nIt can be installed by running\n```\n$ pip install asimov\n```\nIt is also available on conda, and can be installed by running\n```\n$ conda install -c conda-forge ligo-asimov\n```\n\nAsimov also requires that you have `git` installed on your machine, and that you've set it up by running:\n```\n$ git config --global user.email \"you@example.com\"\n$ git config --global user.name \"Your Name\"\n```\n\n## Get started\n\nAsimov supports a variety of different ways of running, but the simplest way, running a workflow on a local machine, can be set up with a single command.\n\nWe start by setting up a project, which is a directory which keeps all of the analyses and the required metadata together.\nA project can include just a single event or a whole selection of events, for example if you're producing a catalogue.\nFirst create a directory to store your project:\n```\n$ mkdir my-new-project\n$ cd my-new-project\n```\nand then get asimov to set things up\n```\n$ asimov init \"Test project\"\n```\nwhere you can replace `\"Test project\"` with the name you want to give your project.\nA project will be set-up in your current working directory.\n\nIn order to start setting up analyses we next need to download some default settings.\n\nAn analysis is a pipeline run, and asimov supports `bayeswave`, `bilby` in the default installation.\n\nWe'll download the default configurations for jobs which are going to be run on the LIGO data grid.\nWe do this using the `asimov apply` command, which pulls-in data from a file either locally or online.\n\n```\n$ asimov apply -f https://git.ligo.org/asimov/data/-/raw/main/defaults/production-pe.yaml\n```\n\nand the load default priors the same way:\n```\n$ asimov apply -f https://git.ligo.org/asimov/data/-/raw/main/defaults/production-pe-priors.yaml\n```\n\nNow you can add an existing event, by downloading the event data settings using the `asimov apply` function, for example, to add GW150914 to the project you can run\n\n```\n$ asimov apply -f https://git.ligo.org/asimov/data/-/raw/main/events/gwtc-2-1/GW150914_095045.yaml\n```\n\nThis downloads the settings which were used for analysing GW150914 for the GWTC-2.1 catalogue paper, and stores them in the ledger file in the project (`.asimov/ledger.yml`).\n\nMany analyses can be run on a single event (these are called \"productions\" in asimov parlence).\nWe can add some pre-configured analyses by downloading some analysis configuration settings.\n\n```\n$ asimov apply -f https://git.ligo.org/asimov/data/-/raw/main/analyses/production-default.yaml -e GW150914_095045\n```\nNote that if you omit the `-e` argument asimov will ask which event the analyses should be applied to.\n\n\nYou can now build and submit your jobs to the cluster.\nFirst use `asimov manage build` to create the configuration files for each analysis:\n```\n$ asimov manage build\n```\nThese will be added to the git repositories in the `checkouts` directory inside the project directory.\nYou then submit the jobs to the cluster by running\n```\n$ asimov manage submit\n```\n\nIt will normally take a long time for a parameter estimation job to finish, but you can get asimov to check up on it by running\n```\n$ asimov monitor\n```\nIf the job is finished `asimov` will start post-processing using `PESummary`, and if it's fallen over it will attempt to fix the problem and resubmit it to the cluster.\nIf the post-processing has finished it will move the results to a read-only directory.\n\nYou can also set the asimov monitor to automatically check the status of the job every 15 minutes by running \n```\n$ asimov start\n```\nwhich will automatically start any post-processing, and start any jobs once their dependencies are met.\nYou can stop this automatic monitoring process any time by running\n```\n$ asimov stop\n```\n\nFor a full description of the workflow management process see the documentation.\n\n\n## I want to help develop new features, or add a new pipeline\n\nGreat! We're always looking for help with developing asimov!\nPlease take a look at our [contributors' guide](CONTRIBUTING.rst) to get started!\n\n\n## Roadmap\n\n### Gravitic pipelines\n\nWhile Asimov already supports a large number of pre-existing pipelines, and provides a straightforward interface for adding new pipelines, we also intend to support pipelines constructed using [gravitic](https://github.com/transientlunatic/gravitic), allowing experimental tools to be used without constructing an entire new pipeline, while also allowing asimov to manage the training of machine learning algorithms.\n\n\n### Workflow replication, extension and duplication\n\nAsimov will allow an existing workflow to be duplicated, in a similar way to a ``git clone``, and then extended, with new jobs gaining access to the completed jobs in the workflow.\nIt will also allow entire workflows to be re-run, providing a straightforward way to replicate results, or make minor modifications.\n\n\n## Authors\n\nAsimov is made by the LIGO, Virgo, and KAGRA collaborations.\nThe primary maintainer of the project is Daniel Williams.\nIts development is supported by the Science and Technology Facilities Council.\n",
"bugtrack_url": null,
"license": "MIT license",
"summary": "A Python package for managing and interacting with data analysis jobs.",
"version": "0.5.6",
"project_urls": {
"Homepage": "https://git.ligo.org/asimov/asimov"
},
"split_keywords": [
"pe",
"ligo",
"asimov"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "e05001c6e7c73f94978d3eacc438fa23eb955b92f8df844bae0993acb2cf07fc",
"md5": "555c735f751ddcb1dbe0fb8312de7207",
"sha256": "f4f33d3174936432937939cf5adb92d822c7f2d966e86f2fb58e62b324f1578c"
},
"downloads": -1,
"filename": "asimov-0.5.6-py2.py3-none-any.whl",
"has_sig": false,
"md5_digest": "555c735f751ddcb1dbe0fb8312de7207",
"packagetype": "bdist_wheel",
"python_version": "py2.py3",
"requires_python": null,
"size": 95967,
"upload_time": "2024-01-19T14:23:54",
"upload_time_iso_8601": "2024-01-19T14:23:54.970741Z",
"url": "https://files.pythonhosted.org/packages/e0/50/01c6e7c73f94978d3eacc438fa23eb955b92f8df844bae0993acb2cf07fc/asimov-0.5.6-py2.py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "ef3292910ca24b9b0c7262625c1ce6c5122b1c835f8972a05cfbfec997fb047d",
"md5": "24efc52720c21622580239949f572434",
"sha256": "2c0c3a20ee225293cec574a19c8c1bfbd2b9288f8add9e55d97d55f7e753c97f"
},
"downloads": -1,
"filename": "asimov-0.5.6-py3-none-any.whl",
"has_sig": false,
"md5_digest": "24efc52720c21622580239949f572434",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 95949,
"upload_time": "2024-01-19T14:21:06",
"upload_time_iso_8601": "2024-01-19T14:21:06.446699Z",
"url": "https://files.pythonhosted.org/packages/ef/32/92910ca24b9b0c7262625c1ce6c5122b1c835f8972a05cfbfec997fb047d/asimov-0.5.6-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "2a77af0cf8a1169f31e647044584515b345a5de9ada782f7fa6c7f1f9c8a69f5",
"md5": "2c2ac0a3c8ccbf1193ae706b084c1979",
"sha256": "f08569ca3da55bc93344402dfb8125b022344b459dcee66e442d15801f01dd0f"
},
"downloads": -1,
"filename": "asimov-0.5.6.tar.gz",
"has_sig": false,
"md5_digest": "2c2ac0a3c8ccbf1193ae706b084c1979",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 217963,
"upload_time": "2024-01-19T14:21:08",
"upload_time_iso_8601": "2024-01-19T14:21:08.924779Z",
"url": "https://files.pythonhosted.org/packages/2a/77/af0cf8a1169f31e647044584515b345a5de9ada782f7fa6c7f1f9c8a69f5/asimov-0.5.6.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-01-19 14:21:08",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "asimov"
}
JSON