## DAWGIE
### Data and Algorithm Work-flow Generation, Introspection, and Execution
The DAWGIE software accomplishes:
1. Data anonymity is required by the framework because the implementation DAWGIE is independent of the Algorithm Engine element of the framework. The framework allows DAWGIE to identify the author of the data through naming and version even though DAWGIE as no knowledge of the data itself. Because the language of the framework and further implementations is Python, DAWGIE forces a further requirement on the data that it be "pickle-able". The additional requirement was accepted by the framework and pushed to the Algorithm Engine element. DAWGIE uses a lazy load implementation -- loaded only when an Algorithm Engine sub-element requests data from a specific author -- to remove routing and, thus, requiring knowledge about the data itself.
1. Data persistence has two independent implementations. In the case of small and simple data author relationships, the Python shelve module is used for persistence. As the number of unique authors grows, the relations grow faster (non-linear) and requires a more sophisticated persistence tool. In the latter case, postgresql is used for persistence. Both implementations are the design making them quickly interchangeable. Neither actually store the data blobs themselves. They -- shelve and postgresql -- only store the author's unique ID -- name and version -- and then reference the blob on disk. Using the relational DB proprieties of shelve (Python dictionaries) and postgresql (relational DB) any request for data from a specific author can quickly be resolved. Creating unique names for all of the data blobs is achieved using {md5 hash}_{sha1 hash}. These names also allow the persistence to recognize if the data is already known.
1. For pipeline management, DAWGIE implements a worker farm, a work flow Abstract Syntax Tree (AST), and signaling from data persistence to execute the tasks within the framework Algorithm Engine element. The data persistence implementation signals the pipeline management element when new data has been authored. The manager searches the AST for all tasks that depend on the author and schedules all tasks that depend on the author starting with the earliest dependent. The new data signals can be generated at the end any task. When a task moves from being scheduled to executing, the foreman of the worker farm passes the task to a waiting worker on a compute node. The worker then loads the data via data persistence for the task and begins executing the task. Upon completion of the task, the worker saves the data via data persistence and notifies the foreman it is ready for another task. In this fashion, DAWGIE walks the minimum and complete nodes in the AST that depend on any new data that has been generated. The pipeline management offers periodic tasks as well that treat a temporal event as new data.
### Organization
DAWGIE is configured and controlled by ENV variables and command line arguments. The command line arguments override the ENV variables. While this section cover nearly all of the command line options, please use the --help switch to see all of the available options.
#### Access to Running DAWGIE
<dl>
<dt>DAWGIE_DB_PORT --context-db-port</dt>
<dd>The database access port. See the specific database implemntation being used for detailed definition of this parameter.</dd>
<dt>DAWGIE_FARM_PORT --context-farm-port</dt>
<dd>The access port that workers in the farm will use to communicate with the DAWGIE server.</dd>
<dt>DAWGIE_FE_PORT --port</dt>
<dd>The web display port. All subsequent ports are computed from this one.</dd>
<dt>DAWGIE_LOG_PORT --context-log-port</dt>
<dd>Port number for distributed workers to log messages through the DAWGIE server.</dd>
</dl>
#### Algorithm Engine (AE)
<dl>
<dt>DAWGIE_AE_BASE_PATH --context-ae-dir</dt>
<dd>The complete path to the AE source code. It is the first directory to start walking down and checking all of the sub-directories and including those that are packages that implement the necessary factories to be identified as AE packages.</dd>
<dt>DAAWGIE_AE_BASE_PACKAGE --context-ae-pkg</dt>
<dd>Because the AE code may be intermixed with non-AE code and, therefore, may be a subset of code, need to know the packaage prefix.</dd>
</dl>
** Example **
If all of the Python code starts in foo and the AE code starts in foo/bar/ae. Then `DAWGIE_AE_BASE_PATH` should be 'foo/bar/ae' and `DAWGIE_AE_BASE_PACKAGE` should be 'foo.bar.ae'.
#### Data
<dl>
<dt>DAWGIE_DATA_DBSTOR --context-data-dbs</dt>
<dd>The location for DAWGIE to store the data generated by the AE known as StateVectors. This area should be vast enough to hold all of the data genreated by the AE over all time.</dd>
<dt>DAWGIE_DATA_LOGDIR --context-data-log</dt>
<dd>The location for DAWGIE to write its log files.</dd>
<dt>DAWGIE_DATA_STAGED --context-data-stg</dt>
<dd>The location for DAWGIE to store temporary data for the AE. It should be sized to fit the expected AE use and DAWGIE will clean out the staging area. However, when there are hiccups in the </dd>
</dl>
#### Database
DAWGIE supports two styles of databases. It supports Python shelve for tiny applications and then POSTGRES for a much larger and scalable system.
##### Postgresql
<dl>
<dt>DAWGIE_DB_HOST --context-db-host</dt>
<dd>The IP hostname of the POSTGRESQL server.</dd>
<dt>DAWGIE_DB_IMPL --context-db-impl</dt>
<dd>Must be 'post'</dd>
<dt>DAWGIE_DB_NAME --context-db-name</dt>
<dd>The name of the database to use.</dd>
<dt>DAWGIE_DB_PATH --context-db-path</dt>
<dd>THe username:password for the database named with DAWGIE_DB_NAME.</dd>
<dt>DAWGIE_DB_PORT --context-db-port</dt>
<dd>The IP port number of the POSTGRESQL server. When DAWGIE_DB_IMPL is 'post', this value defaults to 5432 because POSTGRESQL is independent of DAWGIE.</dd>
</dl>
##### Shelve
<dl>
<dt>DAWGIE_DB_HOST --context-db-host</dt>
<dd>The IP hostname of the machine running DAWGIE.</dd>
<dt>DAWGIE_DB_IMPL --context-db-impl</dt>
<dd>Must be 'shelve'</dd>
<dt>DAWGIE_DB_NAME --context-db-name</dt>
<dd>The name of the database to use.</dd>
<dt>DAWGIE_DB_PATH --context-db-path</dt>
<dd>The directory path on DAWGIE_DB_HOST to write the shelve files.</dd>
<dt>DAWGIE_DB_PORT --context-db-port</dt>
<dd>The IP port number of the DAWGIE DB interface. In is automatically computed from the general port number where DAWGIE is being served (see --port)</dd>
</dl>
##### Tools
<dl>
<dt>DAWGIE_DB_POST2SHELVE_PREFIX</dt>
<dd>Used when converting POSTGRESQL to shelve for development of new AE modules.</dd>
<dt>DAWGIE_DB_ROTATE_PATH --context-db-rotate-path</dt>
<dd>Allows the data to backed up with every new run ID</dd>
<dt>DAWGIE_DB_COPY_PATH --context-db-copy-path</dt>
<dd>Temporary working space for database work.</dd>
<dt>DAWGIE_DB_ROTATES --context-db-rotate</dt>
<dd>The number of database backups to preserve.</dd>
</dl>
#### Souce Code
The source code is then organized by language:
- Bash : utilities for simpler access to the Python
- Python : implementation of DAWGIE
The Python has view key packages:
- dawgie.db : the database interface
- dawgie.de : the display engine that allows user requests to render state vectors to meaningful images
- dawgie.fe : the [front-end](http://mentor.jpl.nasa.gov:8080) that we see and interact with
- dawgie.pl : the actual pipeline code that exercises the algorithm engine
- dawgie.tools : a tool box used by the pipeline and adminsitrators (mostly)
### Documentation
[Fundamental Brochure](https://github.jpl.nasa.gov/pages/niessner/DAWGIE/Notebook/Fundamentals-Brochure.slides.html) is a sales brochure used for SOYA 2018 contest.
[Fundamental Developer Overiew](https://github.jpl.nasa.gov/pages/niessner/DAWGIE/Notebook/Fundamentals-Developer-Overview.slides.html) is a mix of sales and development. It frames the problem and the solution provided. It then proceeds to a high level description of how to use the tool using Gamma et al *Design Patterns*. Armed with the patterns being used, a developer should be able to move to the HOW TO slides connecting the minutiae in those slides with the highest view in these.
[Fundamental Magic](https://github.jpl.nasa.gov/pages/niessner/DAWGIE/Notebook/Fundamentals-Magic.slides.html) is a manager level explanation of what the pipeline does and how it can help development.
[Fundamental How To](https://github.jpl.nasa.gov/pages/niessner/DAWGIE/Notebook/Fundamentals-HOWTO.slides.html) is a beginner course on working with DAWGIE.
[Fundamental Administration](https://github.jpl.nasa.gov/pages/niessner/DAWGIE/Notebook/Fundamentals-Admin.slides.html) is a starter on how to adminsiter DAWGIE.
### Installation
1. `python3 Python/setup.py build install`
1. `bash install.sh`
### Use
Raw data
{
"_id": null,
"home_page": "https://github.com/al-niessner/DAWGIE",
"name": "dawgie",
"maintainer": null,
"docs_url": null,
"requires_python": null,
"maintainer_email": null,
"keywords": "adaptive pipeline",
"author": "Al Niessner",
"author_email": "Al.Niessner@jpl.nasa.gov",
"download_url": "https://files.pythonhosted.org/packages/49/62/5780806163ba6554a12ee1eb02716fdc56bdf10649efbfcf2fa400aeb839/dawgie-1.4.4.tar.gz",
"platform": null,
"description": "## DAWGIE\n\n### Data and Algorithm Work-flow Generation, Introspection, and Execution\n\nThe DAWGIE software accomplishes:\n\n1. Data anonymity is required by the framework because the implementation DAWGIE is independent of the Algorithm Engine element of the framework. The framework allows DAWGIE to identify the author of the data through naming and version even though DAWGIE as no knowledge of the data itself. Because the language of the framework and further implementations is Python, DAWGIE forces a further requirement on the data that it be \"pickle-able\". The additional requirement was accepted by the framework and pushed to the Algorithm Engine element. DAWGIE uses a lazy load implementation -- loaded only when an Algorithm Engine sub-element requests data from a specific author -- to remove routing and, thus, requiring knowledge about the data itself.\n1. Data persistence has two independent implementations. In the case of small and simple data author relationships, the Python shelve module is used for persistence. As the number of unique authors grows, the relations grow faster (non-linear) and requires a more sophisticated persistence tool. In the latter case, postgresql is used for persistence. Both implementations are the design making them quickly interchangeable. Neither actually store the data blobs themselves. They -- shelve and postgresql -- only store the author's unique ID -- name and version -- and then reference the blob on disk. Using the relational DB proprieties of shelve (Python dictionaries) and postgresql (relational DB) any request for data from a specific author can quickly be resolved. Creating unique names for all of the data blobs is achieved using {md5 hash}_{sha1 hash}. These names also allow the persistence to recognize if the data is already known.\n1. For pipeline management, DAWGIE implements a worker farm, a work flow Abstract Syntax Tree (AST), and signaling from data persistence to execute the tasks within the framework Algorithm Engine element. The data persistence implementation signals the pipeline management element when new data has been authored. The manager searches the AST for all tasks that depend on the author and schedules all tasks that depend on the author starting with the earliest dependent. The new data signals can be generated at the end any task. When a task moves from being scheduled to executing, the foreman of the worker farm passes the task to a waiting worker on a compute node. The worker then loads the data via data persistence for the task and begins executing the task. Upon completion of the task, the worker saves the data via data persistence and notifies the foreman it is ready for another task. In this fashion, DAWGIE walks the minimum and complete nodes in the AST that depend on any new data that has been generated. The pipeline management offers periodic tasks as well that treat a temporal event as new data.\n\n### Organization\n\nDAWGIE is configured and controlled by ENV variables and command line arguments. The command line arguments override the ENV variables. While this section cover nearly all of the command line options, please use the --help switch to see all of the available options.\n\n#### Access to Running DAWGIE\n\n<dl>\n <dt>DAWGIE_DB_PORT --context-db-port</dt>\n <dd>The database access port. See the specific database implemntation being used for detailed definition of this parameter.</dd>\n <dt>DAWGIE_FARM_PORT --context-farm-port</dt>\n <dd>The access port that workers in the farm will use to communicate with the DAWGIE server.</dd>\n <dt>DAWGIE_FE_PORT --port</dt>\n <dd>The web display port. All subsequent ports are computed from this one.</dd>\n <dt>DAWGIE_LOG_PORT --context-log-port</dt>\n <dd>Port number for distributed workers to log messages through the DAWGIE server.</dd>\n</dl>\n\n#### Algorithm Engine (AE)\n\n<dl>\n <dt>DAWGIE_AE_BASE_PATH --context-ae-dir</dt>\n <dd>The complete path to the AE source code. It is the first directory to start walking down and checking all of the sub-directories and including those that are packages that implement the necessary factories to be identified as AE packages.</dd>\n <dt>DAAWGIE_AE_BASE_PACKAGE --context-ae-pkg</dt>\n <dd>Because the AE code may be intermixed with non-AE code and, therefore, may be a subset of code, need to know the packaage prefix.</dd>\n</dl>\n\n** Example **\n\nIf all of the Python code starts in foo and the AE code starts in foo/bar/ae. Then `DAWGIE_AE_BASE_PATH` should be 'foo/bar/ae' and `DAWGIE_AE_BASE_PACKAGE` should be 'foo.bar.ae'.\n\n#### Data\n\n<dl>\n <dt>DAWGIE_DATA_DBSTOR --context-data-dbs</dt>\n <dd>The location for DAWGIE to store the data generated by the AE known as StateVectors. This area should be vast enough to hold all of the data genreated by the AE over all time.</dd>\n <dt>DAWGIE_DATA_LOGDIR --context-data-log</dt>\n <dd>The location for DAWGIE to write its log files.</dd>\n <dt>DAWGIE_DATA_STAGED --context-data-stg</dt>\n <dd>The location for DAWGIE to store temporary data for the AE. It should be sized to fit the expected AE use and DAWGIE will clean out the staging area. However, when there are hiccups in the </dd>\n</dl>\n\n#### Database\n\nDAWGIE supports two styles of databases. It supports Python shelve for tiny applications and then POSTGRES for a much larger and scalable system.\n\n##### Postgresql\n\n<dl>\n <dt>DAWGIE_DB_HOST --context-db-host</dt>\n <dd>The IP hostname of the POSTGRESQL server.</dd>\n <dt>DAWGIE_DB_IMPL --context-db-impl</dt>\n <dd>Must be 'post'</dd>\n <dt>DAWGIE_DB_NAME --context-db-name</dt>\n <dd>The name of the database to use.</dd>\n <dt>DAWGIE_DB_PATH --context-db-path</dt>\n <dd>THe username:password for the database named with DAWGIE_DB_NAME.</dd>\n <dt>DAWGIE_DB_PORT --context-db-port</dt>\n <dd>The IP port number of the POSTGRESQL server. When DAWGIE_DB_IMPL is 'post', this value defaults to 5432 because POSTGRESQL is independent of DAWGIE.</dd>\n</dl>\n\n##### Shelve\n\n<dl>\n <dt>DAWGIE_DB_HOST --context-db-host</dt>\n <dd>The IP hostname of the machine running DAWGIE.</dd>\n <dt>DAWGIE_DB_IMPL --context-db-impl</dt>\n <dd>Must be 'shelve'</dd>\n <dt>DAWGIE_DB_NAME --context-db-name</dt>\n <dd>The name of the database to use.</dd>\n <dt>DAWGIE_DB_PATH --context-db-path</dt>\n <dd>The directory path on DAWGIE_DB_HOST to write the shelve files.</dd>\n <dt>DAWGIE_DB_PORT --context-db-port</dt>\n <dd>The IP port number of the DAWGIE DB interface. In is automatically computed from the general port number where DAWGIE is being served (see --port)</dd>\n</dl>\n\n##### Tools\n\n<dl>\n <dt>DAWGIE_DB_POST2SHELVE_PREFIX</dt>\n <dd>Used when converting POSTGRESQL to shelve for development of new AE modules.</dd>\n <dt>DAWGIE_DB_ROTATE_PATH --context-db-rotate-path</dt>\n <dd>Allows the data to backed up with every new run ID</dd>\n <dt>DAWGIE_DB_COPY_PATH --context-db-copy-path</dt>\n <dd>Temporary working space for database work.</dd>\n <dt>DAWGIE_DB_ROTATES --context-db-rotate</dt>\n <dd>The number of database backups to preserve.</dd>\n</dl>\n\n\n#### Souce Code\n\nThe source code is then organized by language:\n- Bash : utilities for simpler access to the Python\n- Python : implementation of DAWGIE\n\nThe Python has view key packages:\n- dawgie.db : the database interface\n- dawgie.de : the display engine that allows user requests to render state vectors to meaningful images\n- dawgie.fe : the [front-end](http://mentor.jpl.nasa.gov:8080) that we see and interact with\n- dawgie.pl : the actual pipeline code that exercises the algorithm engine\n- dawgie.tools : a tool box used by the pipeline and adminsitrators (mostly)\n\n### Documentation\n\n[Fundamental Brochure](https://github.jpl.nasa.gov/pages/niessner/DAWGIE/Notebook/Fundamentals-Brochure.slides.html) is a sales brochure used for SOYA 2018 contest.\n\n[Fundamental Developer Overiew](https://github.jpl.nasa.gov/pages/niessner/DAWGIE/Notebook/Fundamentals-Developer-Overview.slides.html) is a mix of sales and development. It frames the problem and the solution provided. It then proceeds to a high level description of how to use the tool using Gamma et al *Design Patterns*. Armed with the patterns being used, a developer should be able to move to the HOW TO slides connecting the minutiae in those slides with the highest view in these.\n\n[Fundamental Magic](https://github.jpl.nasa.gov/pages/niessner/DAWGIE/Notebook/Fundamentals-Magic.slides.html) is a manager level explanation of what the pipeline does and how it can help development.\n\n[Fundamental How To](https://github.jpl.nasa.gov/pages/niessner/DAWGIE/Notebook/Fundamentals-HOWTO.slides.html) is a beginner course on working with DAWGIE.\n\n[Fundamental Administration](https://github.jpl.nasa.gov/pages/niessner/DAWGIE/Notebook/Fundamentals-Admin.slides.html) is a starter on how to adminsiter DAWGIE.\n### Installation\n\n1. `python3 Python/setup.py build install`\n1. `bash install.sh`\n\n### Use\n\n",
"bugtrack_url": null,
"license": "see LICENSE file for details",
"summary": "Data and Algorithm Work-flow Generation, Introspection, and Execution (DAWGIE)",
"version": "1.4.4",
"project_urls": {
"Homepage": "https://github.com/al-niessner/DAWGIE"
},
"split_keywords": [
"adaptive",
"pipeline"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "72046a23453c425f67753678ead6028ffd73ce5e72e58864b4818df805c94de0",
"md5": "0d22c2bc92b99e7bfd6be8ad2d61ea5c",
"sha256": "1aa391dcad52cdcbc10c5d70d2e0e4907822d678e50c646d1e1486c63ac3fccc"
},
"downloads": -1,
"filename": "dawgie-1.4.4-py3-none-any.whl",
"has_sig": false,
"md5_digest": "0d22c2bc92b99e7bfd6be8ad2d61ea5c",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 826789,
"upload_time": "2024-10-07T17:52:44",
"upload_time_iso_8601": "2024-10-07T17:52:44.329389Z",
"url": "https://files.pythonhosted.org/packages/72/04/6a23453c425f67753678ead6028ffd73ce5e72e58864b4818df805c94de0/dawgie-1.4.4-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "49625780806163ba6554a12ee1eb02716fdc56bdf10649efbfcf2fa400aeb839",
"md5": "3fecfe70edf5f7e52a717db7c1928892",
"sha256": "a4e6cd764aee8dcc42297a954b769a49022b46ee9193c36dc90e58c6771a49f3"
},
"downloads": -1,
"filename": "dawgie-1.4.4.tar.gz",
"has_sig": false,
"md5_digest": "3fecfe70edf5f7e52a717db7c1928892",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 8265,
"upload_time": "2024-10-07T17:52:46",
"upload_time_iso_8601": "2024-10-07T17:52:46.514110Z",
"url": "https://files.pythonhosted.org/packages/49/62/5780806163ba6554a12ee1eb02716fdc56bdf10649efbfcf2fa400aeb839/dawgie-1.4.4.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-10-07 17:52:46",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "al-niessner",
"github_project": "DAWGIE",
"travis_ci": true,
"coveralls": false,
"github_actions": false,
"circle": true,
"lcname": "dawgie"
}
JSON
