# MADAM
MADAM is the Multi Agent Digital Asset Manager.
It provides a three-tier architecture platform to handle workflow processing in a distributed environment.
It uses Docker swarm to dispatch processes in a cluster of machines.
It is a free (as freedom) software written in Python.
## Documentation
[Link to the documentation](https://m5231.gitlab.io/documentation/)
## Support
If you find this project useful and want to contribute, please submit issues, merge requests. If you use it regularly,
you can help by the author by a financial support.
<script src="https://liberapay.com/vit/widgets/button.js"></script>
<noscript><a href="https://liberapay.com/vit/donate"><img alt="Donate using Liberapay" src="https://liberapay.com/assets/widgets/donate.svg"></a></noscript>
## Requirements
You will need [Camunda Modeler 4.11+](https://github.com/camunda/camunda-modeler/releases) to easily create
Zeebe BPMN XML workflows for MADAM.
## Licensing
MADAM is licensed under the [Gnu Public License Version 3](https://www.gnu.org/licenses/gpl-3.0.en.html).
Camunda Modeler is licensed under [the MIT License (MIT)](https://mit-license.org/).
At its core, MADAM use [adhesive-zebe](https://github.com/vtexier/adhesive), a BPMN workflow python engine able to
execute Zeebe BPMN XML workflows. It is a fork of [adhesive](https://github.com/germaniumhq/adhesive) under
the original adhesive license that is [GNU Affero General Public License v3.0](https://www.gnu.org/licenses/agpl-3.0.en.html)
## System environment setup
1. [Install Docker](https://docs.docker.com/engine/install/).
2. [Configure userns-remap](https://docs.docker.com/engine/security/userns-remap/) to map container user `root` to a
host non-root user.
3. Configure the dev station as a [Docker Swarm Manager](https://docs.docker.com/engine/swarm/).
4. Install a [Postgresql](https://www.postgresql.org/download/) database server.
_You can use the Ansible playbook provided to install PostgreSQL locally with Docker,
after configuring `hosts.yaml`:_
make environment
### Python environment setup
* It requires Python 3.8+.
* [Pyenv](https://github.com/pyenv/pyenv) should be used to choose the right version of Python, without breaking the
default Python of the Operating System.
* A Python virtual environment should be created in a `.venv` folder.
```bash
pyenv install 3.8.0
pyenv shell 3.8.0
python -m venv .venv
source .venv/bin/activate`
```
### Installation/Update
From PyPI:
In a Python virtualenv:
pip install -U madam-mam
In your user install directory:
pip install --user -U madam-mam
You should have a the `madam` cli command available:
madam
or
madam --help
will display command usage.
To have bash completion, you can type:
_MADAM_COMPLETE=source_bash madam > madam-complete.sh
sudo cp madam-complete.sh /etc/bash_completion.d/.
For another shell, replace `source_bash` by `source_zsh` or `source_fish`
### Development environment
Install [Poetry](https://python-poetry.org/) with the custom installer:
curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python
Install Python dependencies:
poetry install --no-root
Copy `bin/pre-commit.sh` for pre-commmit git hooks:
cp bin/pre-commit.sh .git/hooks/pre-commit
You can use the madam-cli dev command:
./bin/madam-cli
Get `bin/madam-cli` bash shell completion:
_MADAM_CLI_COMPLETE=source_bash bin/madam-cli > madam-cli-complete.sh
sudo cp madam-cli-complete.sh /etc/bash_completion.d/.
For another shell, replace `source_bash` by `source_zsh` or `source_fish`
### Configuration
Make a copy of the environment config example file:
cp .env.example .env
Edit `.env` to suit your needs, then:
export $(grep -v '^#' .env | xargs -d '\n')
Make a copy of the Ansible inventory example file:
cp hosts.yaml.example hosts.yaml
Edit `hosts.yaml` to suit your needs.
Make a copy of the MADAM config example file:
cp madam.yaml.example madam.yaml
Edit `madam.yaml` to suit your needs.
Make a copy of the MADAM config example file for the test environment:
cp madam_tests.yaml.example madam_tests.yaml
Edit `madam_tests.yaml` to suit your needs.
Make a copy of the MADAM config example file for the local deploy:
cp madam_deploy.yaml.example madam_deploy.yaml
Edit `madam_deploy.yaml` to suit your needs.
## Check static type and code quality
make check
## Run tests
Run all [pytest](https://docs.pytest.org) tests with:
make tests
Run only some tests by using `bin/tests.sh`:
bin/tests.sh tests/domains/test_workflows.py::test_create
## Database setup
Set `DATABASE_URL` and `DATABASE_URL_TESTS` environment variable in `.env` file:
DATABASE_URL=postgresql://postgres:xxxxx@hostname:5432/madam?sslmode=allow
DATABASE_URL_TESTS=postgresql://postgres:xxxxx@hostname:5432/madam_tests?sslmode=allow
### Migrations scripts
Add/Edit scripts in `resources/migrations` directory:
# version.name.[rollback].sql
00001.init_tables.sql
00001.init_tables.rollback.sql
### Migrate commands
make databases
make databases_rollback
make databases_list
## Deployment
### Set and tag project version in Git
./bin/release.sh 1.0.0
### Build MADAM python package and Docker image
Build python wheel and docker image:
make build
The wheel package will be build in the `dist` directory.
Push docker image on private registry:
docker image tag madam:[version] registry_hostname:registry_port/madam:[version]
docker push registry_hostname:registry_port/madam:[version]
### Deploy MADAM as local docker container
First you need to have docker installed locally, and the node declared as a swarm manager.
By default, docker swarm services will run as root, so files will be created as root user.
You can change the user and group for the agents by setting the UID and GID in the docker.user parameter in madam.yml conbfig file:
docker:
base_url: "unix://var/run/docker.sock"
user: "1000:1000" # Here the agents will run as the 1000 user and 1000 group
networks: []
To deploy MADAM container on localhost:
make deploy
### Publish to PyPI and Docker Hub
Configure poetry with PyPI Access Token:
poetry config pypi-token.pypi your-api-token
Publish the Python package on PyPI:
make publish
Raw data
{
"_id": null,
"home_page": "",
"name": "madam-mam",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.8,<4.0",
"maintainer_email": "",
"keywords": "BPMN,workflow,media,assets,management,MAM,DAM,docker,swarm",
"author": "Vincent Texier",
"author_email": "vit@free.fr",
"download_url": "https://files.pythonhosted.org/packages/9d/52/b03ee1a1021b0f7170f5176d6c251e2eda6c3fc369942582fb5d0d2c812b/madam_mam-0.7.3.tar.gz",
"platform": null,
"description": "# MADAM\n\nMADAM is the Multi Agent Digital Asset Manager.\n\nIt provides a three-tier architecture platform to handle workflow processing in a distributed environment.\n\nIt uses Docker swarm to dispatch processes in a cluster of machines.\n\nIt is a free (as freedom) software written in Python.\n\n## Documentation\n\n[Link to the documentation](https://m5231.gitlab.io/documentation/)\n\n## Support\n\nIf you find this project useful and want to contribute, please submit issues, merge requests. If you use it regularly,\nyou can help by the author by a financial support.\n\n<script src=\"https://liberapay.com/vit/widgets/button.js\"></script>\n<noscript><a href=\"https://liberapay.com/vit/donate\"><img alt=\"Donate using Liberapay\" src=\"https://liberapay.com/assets/widgets/donate.svg\"></a></noscript>\n\n## Requirements\n\nYou will need [Camunda Modeler 4.11+](https://github.com/camunda/camunda-modeler/releases) to easily create\nZeebe BPMN XML workflows for MADAM.\n\n## Licensing\n\nMADAM is licensed under the [Gnu Public License Version 3](https://www.gnu.org/licenses/gpl-3.0.en.html).\n\nCamunda Modeler is licensed under [the MIT License (MIT)](https://mit-license.org/).\n\nAt its core, MADAM use [adhesive-zebe](https://github.com/vtexier/adhesive), a BPMN workflow python engine able to\nexecute Zeebe BPMN XML workflows. It is a fork of [adhesive](https://github.com/germaniumhq/adhesive) under\nthe original adhesive license that is [GNU Affero General Public License v3.0](https://www.gnu.org/licenses/agpl-3.0.en.html)\n\n## System environment setup\n\n1. [Install Docker](https://docs.docker.com/engine/install/).\n\n2. [Configure userns-remap](https://docs.docker.com/engine/security/userns-remap/) to map container user `root` to a\n host non-root user.\n\n3. Configure the dev station as a [Docker Swarm Manager](https://docs.docker.com/engine/swarm/).\n\n4. Install a [Postgresql](https://www.postgresql.org/download/) database server.\n \n_You can use the Ansible playbook provided to install PostgreSQL locally with Docker,\nafter configuring `hosts.yaml`:_\n\n make environment\n\n### Python environment setup\n\n* It requires Python 3.8+.\n\n* [Pyenv](https://github.com/pyenv/pyenv) should be used to choose the right version of Python, without breaking the\n default Python of the Operating System.\n\n* A Python virtual environment should be created in a `.venv` folder.\n\n```bash\n pyenv install 3.8.0\n pyenv shell 3.8.0\n python -m venv .venv \n source .venv/bin/activate`\n```\n\n### Installation/Update\n\nFrom PyPI:\n\nIn a Python virtualenv:\n\n pip install -U madam-mam\n\nIn your user install directory:\n\n pip install --user -U madam-mam\n\nYou should have a the `madam` cli command available:\n\n madam\n\nor\n\n madam --help\n\nwill display command usage.\n\nTo have bash completion, you can type:\n\n _MADAM_COMPLETE=source_bash madam > madam-complete.sh\n sudo cp madam-complete.sh /etc/bash_completion.d/.\n\nFor another shell, replace `source_bash` by `source_zsh` or `source_fish`\n\n### Development environment\n\nInstall [Poetry](https://python-poetry.org/) with the custom installer:\n\n curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python\n\nInstall Python dependencies:\n\n poetry install --no-root\n\nCopy `bin/pre-commit.sh` for pre-commmit git hooks:\n\n cp bin/pre-commit.sh .git/hooks/pre-commit\n\nYou can use the madam-cli dev command:\n\n ./bin/madam-cli\n\nGet `bin/madam-cli` bash shell completion:\n\n _MADAM_CLI_COMPLETE=source_bash bin/madam-cli > madam-cli-complete.sh\n sudo cp madam-cli-complete.sh /etc/bash_completion.d/.\n\nFor another shell, replace `source_bash` by `source_zsh` or `source_fish`\n\n### Configuration\n\nMake a copy of the environment config example file:\n\n cp .env.example .env\n\nEdit `.env` to suit your needs, then:\n\n export $(grep -v '^#' .env | xargs -d '\\n')\n\nMake a copy of the Ansible inventory example file:\n\n cp hosts.yaml.example hosts.yaml\n\nEdit `hosts.yaml` to suit your needs.\n\nMake a copy of the MADAM config example file:\n\n cp madam.yaml.example madam.yaml\n\nEdit `madam.yaml` to suit your needs.\n\nMake a copy of the MADAM config example file for the test environment:\n\n cp madam_tests.yaml.example madam_tests.yaml\n\nEdit `madam_tests.yaml` to suit your needs.\n\nMake a copy of the MADAM config example file for the local deploy:\n\n cp madam_deploy.yaml.example madam_deploy.yaml\n\nEdit `madam_deploy.yaml` to suit your needs.\n\n## Check static type and code quality\n\n make check\n\n## Run tests\n\nRun all [pytest](https://docs.pytest.org) tests with:\n\n make tests\n\nRun only some tests by using `bin/tests.sh`:\n\n bin/tests.sh tests/domains/test_workflows.py::test_create\n\n## Database setup\n\nSet `DATABASE_URL` and `DATABASE_URL_TESTS` environment variable in `.env` file:\n\n DATABASE_URL=postgresql://postgres:xxxxx@hostname:5432/madam?sslmode=allow\n DATABASE_URL_TESTS=postgresql://postgres:xxxxx@hostname:5432/madam_tests?sslmode=allow\n\n### Migrations scripts\n\nAdd/Edit scripts in `resources/migrations` directory:\n\n # version.name.[rollback].sql\n 00001.init_tables.sql\n 00001.init_tables.rollback.sql\n\n### Migrate commands\n\n make databases\n make databases_rollback\n make databases_list\n\n## Deployment\n\n### Set and tag project version in Git\n\n ./bin/release.sh 1.0.0\n\n### Build MADAM python package and Docker image\n\nBuild python wheel and docker image:\n\n make build\n\nThe wheel package will be build in the `dist` directory.\n\nPush docker image on private registry:\n\n docker image tag madam:[version] registry_hostname:registry_port/madam:[version]\n docker push registry_hostname:registry_port/madam:[version]\n\n### Deploy MADAM as local docker container\n\nFirst you need to have docker installed locally, and the node declared as a swarm manager.\n\nBy default, docker swarm services will run as root, so files will be created as root user.\n\nYou can change the user and group for the agents by setting the UID and GID in the docker.user parameter in madam.yml conbfig file:\n\n docker:\n base_url: \"unix://var/run/docker.sock\"\n user: \"1000:1000\" # Here the agents will run as the 1000 user and 1000 group\n networks: []\n\nTo deploy MADAM container on localhost:\n\n make deploy\n\n### Publish to PyPI and Docker Hub\n\nConfigure poetry with PyPI Access Token:\n\n poetry config pypi-token.pypi your-api-token\n\nPublish the Python package on PyPI:\n\n make publish\n\n",
"bugtrack_url": null,
"license": "GPL-3.0-only",
"summary": "MADAM (TM) Multi Agent Digital Asset Manager - a MAM server for Docker Swarm to handle higly distributed media processes",
"version": "0.7.3",
"project_urls": null,
"split_keywords": [
"bpmn",
"workflow",
"media",
"assets",
"management",
"mam",
"dam",
"docker",
"swarm"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "dd302cbfad09105165724fb6248326e39f02176f67f6c7697e3e7bf73b6476cc",
"md5": "04373f2083fc7345468c7fb6b147184c",
"sha256": "aaff210805da99c3556ee37f6f3ef6d98b7a0f4fa83b81a1e7e930a6bab88b89"
},
"downloads": -1,
"filename": "madam_mam-0.7.3-py3-none-any.whl",
"has_sig": false,
"md5_digest": "04373f2083fc7345468c7fb6b147184c",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8,<4.0",
"size": 146150,
"upload_time": "2023-08-31T13:34:23",
"upload_time_iso_8601": "2023-08-31T13:34:23.529882Z",
"url": "https://files.pythonhosted.org/packages/dd/30/2cbfad09105165724fb6248326e39f02176f67f6c7697e3e7bf73b6476cc/madam_mam-0.7.3-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "9d52b03ee1a1021b0f7170f5176d6c251e2eda6c3fc369942582fb5d0d2c812b",
"md5": "ace1e3853fe1cd4a3a57f5b58401395d",
"sha256": "655939a438896b7bd1cf5669bb7854fbba9b145c504d09b5ed97b5121ee5f551"
},
"downloads": -1,
"filename": "madam_mam-0.7.3.tar.gz",
"has_sig": false,
"md5_digest": "ace1e3853fe1cd4a3a57f5b58401395d",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8,<4.0",
"size": 65981,
"upload_time": "2023-08-31T13:34:25",
"upload_time_iso_8601": "2023-08-31T13:34:25.965251Z",
"url": "https://files.pythonhosted.org/packages/9d/52/b03ee1a1021b0f7170f5176d6c251e2eda6c3fc369942582fb5d0d2c812b/madam_mam-0.7.3.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-08-31 13:34:25",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "madam-mam"
}
JSON