# Storm
Storm is a Python DSL to generate a graph of task dependencies.
## Install
For development, it's best to install in *editable* mode in a virtual environment.
```
> git clone git@github.com/ConductorTechnologies/cwstorm.git
> cd cwstorm
# Create a virtual environment
> python3 -m venv cwstorm.venv
> . cwstorm.venv/bin/activate
# Install in editable mode
> pip install -e .
# Optionally install the black formatter and some other dev tools
> pip install -r requirements.txt
```
## Quick Start CLI
Use the `storm` CLI command to serialize one of the example jobs. The pretty option generates human readable JSON output.
```
> storm serialize -x simple_qt -f pretty -s default ~/Desktop/simple_qt.json
> cat ~/Desktop/simple_qt.json
```
If using the `default` serializer and you want `JSON` output, you can omit the `--serializer` and `--format` options.
```
> storm serialize -x simple_qt ~/Desktop/simple_qt.json
```
## Visualize
You can visualize graphs on the [web app](https://inst-tag-assign.vercel.app/chart). In the upper left, you'll see a menu containing some presets and an upload button. Upload the file you just made. `~/Desktop/simple_qt.json` or make a new one.
### Command-line interface
The command-line interface has one subcommand, `serialize`. It creates the argument parser dynamically based on the available examples and serializers.
```
storm serialize --help
```
### Examples
The `ass_comp_normal.py` example is a more complex script. It builds a graph of tasks that uploads assets, generates ass files, renders them, adds optical motion blur, makes a quicktime, and notifies people.
## DSL Structure
A graph consists of dag nodes with attributes. Look through the examples folder to familiarize yourself with the API.
Inheritance hierarchy is as follows:
### Node
The base class. Responsible for generating getters and setters for different attributes. The idea is to have a consistent language to build the DAG, add options to nodes, and to quickly see how the graph will look. See the ATTRS lists in Job, Task, and Cmd. The types of attributes that can be added to nodes are:
* int
* str
* dict
* Cmd
* list of int
* list of str
* list of Cmd
Setters return self, which allows for chaining.
Getters and setters are generated are as follows for Attribute name `atr` of Node `n`:
`int`
* Setter: `n.atr(value) -> self`
* Getter `n.atr() -> int`
`str`
* Setter: `n.atr(value) -> self`
* Getter `n.atr() -> str`
`dict`
* Setter: `n.atr({"key": "VAL", ...}) -> self`
* Updater `n.update_atr({"key2": "VAL2", ...}) -> self`
* Getter `n.atr() -> dict`
`Cmd`
* Setter: `n.atr(Cmd(*args)) -> self`
* Getter `n.atr() -> Cmd`
`list:int`
* Setter: `n.atr(*args) -> self`
* Extender `n.push_atr(*args) -> self`
* Getter `n.atr() -> list of int`
`list:str`
* Setter: `n.atr(*args) -> self`
* Extender `n.push_atr(*args) -> self`
* Getter `n.atr() -> list of str`
`list:Cmd`
* Setter: `n.atr(Cmd(*args), Cmd(*args), ...) -> self`
* Extender `n.push_atr(Cmd(*args), Cmd(*args), ...) -> self`
* Getter `n.atr() -> list of Cmd`
`list:dict`
* Setter: `n.atr({"key": "VAL", ...}, {"key": "VAL", ...}, ...) -> self`
* Extender `n.push_atr({"key": "VAL", ...}, {"key": "VAL", ...}, ...) -> self`
* Getter `n.atr() -> list of dict`
### DagNode(Node)
Any node used for building the DAG - Currently Job, Task, and Upload. DagNode has methods to add children and to manage serialization of the hierarchy.
### Task(DagNode)
Tasks contain commands. They may be added to other Tasks as children or to the Job. A task may be the child of many parents.
### Upload(DagNode)
Uploads contain lists of filepaths. They can be added anywhere a Task can be added. The only difference is their set of properties.
### Job(DagNode)
A job is like a Task, but there can be only one and it cannot have parents. Think of it as a container for all other tasks and uploads.
### Cmd(Node)
Commands currently exist in the "commands" or "cleanup" attributes of tasks. Lists of commands in a task run in serial.
## Changelog
## Version:0.1.1 -- 23 Jan 2024
* Schema tweaks
* * Remove environment from job
* * Remove cleanup from all nodes
* * Add `upload` node type
* * Add `lifecycle` property to tasks
## Unreleased:
* 0.0.1-beta.1
* Initial CICD setup
--
Raw data
{
"_id": null,
"home_page": "https://github.com/ConductorTechnologies/cwstorm",
"name": "cwstorm",
"maintainer": "",
"docs_url": null,
"requires_python": "",
"maintainer_email": "",
"keywords": "",
"author": "conductor",
"author_email": "info@conductortech.com",
"download_url": "",
"platform": null,
"description": "# Storm\n\nStorm is a Python DSL to generate a graph of task dependencies.\n\n## Install\n\nFor development, it's best to install in *editable* mode in a virtual environment. \n\n\n```\n> git clone git@github.com/ConductorTechnologies/cwstorm.git\n> cd cwstorm\n\n# Create a virtual environment\n> python3 -m venv cwstorm.venv\n> . cwstorm.venv/bin/activate\n\n# Install in editable mode\n> pip install -e .\n\n# Optionally install the black formatter and some other dev tools\n> pip install -r requirements.txt\n```\n\n## Quick Start CLI\n\nUse the `storm` CLI command to serialize one of the example jobs. The pretty option generates human readable JSON output. \n\n\n```\n> storm serialize -x simple_qt -f pretty -s default ~/Desktop/simple_qt.json\n> cat ~/Desktop/simple_qt.json\n```\n\nIf using the `default` serializer and you want `JSON` output, you can omit the `--serializer` and `--format` options.\n\n```\n> storm serialize -x simple_qt ~/Desktop/simple_qt.json\n```\n\n## Visualize\n\nYou can visualize graphs on the [web app](https://inst-tag-assign.vercel.app/chart). In the upper left, you'll see a menu containing some presets and an upload button. Upload the file you just made. `~/Desktop/simple_qt.json` or make a new one. \n\n### Command-line interface\n\nThe command-line interface has one subcommand, `serialize`. It creates the argument parser dynamically based on the available examples and serializers.\n\n```\nstorm serialize --help\n```\n\n### Examples\n\nThe `ass_comp_normal.py` example is a more complex script. It builds a graph of tasks that uploads assets, generates ass files, renders them, adds optical motion blur, makes a quicktime, and notifies people.\n\n## DSL Structure\n\nA graph consists of dag nodes with attributes. Look through the examples folder to familiarize yourself with the API.\n\nInheritance hierarchy is as follows:\n\n### Node \nThe base class. Responsible for generating getters and setters for different attributes. The idea is to have a consistent language to build the DAG, add options to nodes, and to quickly see how the graph will look. See the ATTRS lists in Job, Task, and Cmd. The types of attributes that can be added to nodes are:\n * int\n * str\n * dict\n * Cmd\n * list of int\n * list of str\n * list of Cmd\n\nSetters return self, which allows for chaining.\n\nGetters and setters are generated are as follows for Attribute name `atr` of Node `n`:\n\n`int` \n\n* Setter: `n.atr(value) -> self`\n* Getter `n.atr() -> int`\n\n\n`str` \n\n* Setter: `n.atr(value) -> self`\n* Getter `n.atr() -> str`\n\n\n`dict` \n\n* Setter: `n.atr({\"key\": \"VAL\", ...}) -> self`\n* Updater `n.update_atr({\"key2\": \"VAL2\", ...}) -> self`\n* Getter `n.atr() -> dict`\n\n`Cmd`\n\n* Setter: `n.atr(Cmd(*args)) -> self`\n* Getter `n.atr() -> Cmd`\n\n`list:int`\n\n* Setter: `n.atr(*args) -> self`\n* Extender `n.push_atr(*args) -> self`\n* Getter `n.atr() -> list of int`\n\n`list:str`\n\n* Setter: `n.atr(*args) -> self`\n* Extender `n.push_atr(*args) -> self`\n* Getter `n.atr() -> list of str`\n\n\n`list:Cmd`\n\n* Setter: `n.atr(Cmd(*args), Cmd(*args), ...) -> self`\n* Extender `n.push_atr(Cmd(*args), Cmd(*args), ...) -> self`\n* Getter `n.atr() -> list of Cmd`\n\n`list:dict`\n\n* Setter: `n.atr({\"key\": \"VAL\", ...}, {\"key\": \"VAL\", ...}, ...) -> self`\n* Extender `n.push_atr({\"key\": \"VAL\", ...}, {\"key\": \"VAL\", ...}, ...) -> self`\n* Getter `n.atr() -> list of dict`\n\n### DagNode(Node)\n\nAny node used for building the DAG - Currently Job, Task, and Upload. DagNode has methods to add children and to manage serialization of the hierarchy.\n\n### Task(DagNode)\nTasks contain commands. They may be added to other Tasks as children or to the Job. A task may be the child of many parents.\n\n### Upload(DagNode)\nUploads contain lists of filepaths. They can be added anywhere a Task can be added. The only difference is their set of properties.\n\n### Job(DagNode)\n\nA job is like a Task, but there can be only one and it cannot have parents. Think of it as a container for all other tasks and uploads.\n\n### Cmd(Node)\nCommands currently exist in the \"commands\" or \"cleanup\" attributes of tasks. Lists of commands in a task run in serial.\n\n## Changelog\n\n## Version:0.1.1 -- 23 Jan 2024\n\n* Schema tweaks\n* * Remove environment from job\n* * Remove cleanup from all nodes\n* * Add `upload` node type\n* * Add `lifecycle` property to tasks\n\n## Unreleased:\n\n* 0.0.1-beta.1\n * Initial CICD setup\n\n\n\n--\n",
"bugtrack_url": null,
"license": "",
"summary": "Core functionality for Conductor's client tools",
"version": "0.1.1",
"project_urls": {
"Homepage": "https://github.com/ConductorTechnologies/cwstorm"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "099ed138370b5a3e6ecb7223873a547b9a5081e9fb305ba10f0672c962057a94",
"md5": "752245b288e690459d854acd7b53bd60",
"sha256": "9ec7a03db1fe23db7ea4f27de60112fdf89b0e1c506dcc9c0aa1fc45902cf253"
},
"downloads": -1,
"filename": "cwstorm-0.1.1-py2.py3-none-any.whl",
"has_sig": false,
"md5_digest": "752245b288e690459d854acd7b53bd60",
"packagetype": "bdist_wheel",
"python_version": "py2.py3",
"requires_python": null,
"size": 22075,
"upload_time": "2024-01-23T14:33:08",
"upload_time_iso_8601": "2024-01-23T14:33:08.399597Z",
"url": "https://files.pythonhosted.org/packages/09/9e/d138370b5a3e6ecb7223873a547b9a5081e9fb305ba10f0672c962057a94/cwstorm-0.1.1-py2.py3-none-any.whl",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-01-23 14:33:08",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "ConductorTechnologies",
"github_project": "cwstorm",
"github_not_found": true,
"lcname": "cwstorm"
}
JSON
