yacfg


Nameyacfg JSON
Version 0.8.3 PyPI version JSON
download
home_pagehttps://github.com/rh-messaging-qe/yacfg
SummaryTemplate based configuration generator
upload_time2021-02-23 17:54:47
maintainerDominik Lenoch
docs_urlNone
authorZdenek Kraus
requires_python
licenseApache-2.0
keywords
VCS
bugtrack_url
requirements No requirements were recorded.
Travis-CI No Travis.
coveralls test coverage No coveralls.
            # yacfg - YAML Configurator

This tool can generate a set of configuration files mainly needed for
AMQ Broker, but it is not limited to only generating files for one product.

It has a user facing Command Line Tool for quick and easy command line usage.
Furthermore, it is possible to use its API in your python code.

## Getting started

* python 3.5+ or python2.7
* current requirements from setup.py (runtime requirements only)
* python virtualenv recommended (install via system package manager
or `pip install --user virtualenv`)

for contributors:
* requirements from requirements.txt (there are Dev and QA requirements as well)

### From git

```bash
git clone git@github.com:rh-messaging-qe/yacfg.git
python -m virtualenv -p python3 venv3
source venv3/bin/activate
./setup.py install
yacfg --help
```

### From PyPI

```bash
python -m virtualenv -p python3 venv3
source venv3/bin/activate
pip install yacfg
yacfg --help
```

## User (CLI) guide

```bash
yacfg --help

yacfg --list-profiles
yacfg --list-templates

# perform a generation of a default profile
yacfg --profile artemis/2.5.0/default.yaml.jinja2
# also save result to [OUTDIR] directory
yacfg --profile [PROFILE] --output [OUTDIR]
```

## Customization

Quickest way to customize data is to use hot-variables, basically variables
that the profile itself provides for tuning. Next step is to write (modify) custom
profile with completely custom values.
If that does not satisfy your needs, then a custom template might be required.

### Profile tuning

Simply export tuning values from profile you want to tune and change those you
need to change. Then supply the custom tuning file(s) when generating the profile.

```bash
yacfg --profile [PROFILE] --export-tuning my_values.yaml
vim my_values.yaml
yacfg --profile [PROFILE] --tune my_values.yaml

# multiple tuning files can be overlaid
# they are updated in sequence, only values present are overwritten
yacfg --profile [PROFILE] --tune my_values.yaml --tune machine_specific.yaml \
       --tune logging_debug.yaml --output [OUTDIR]
```

## Custom profiles

Write your own, or simply export an existing profile and modify that.

You can export dynamic version with includes of some modules, that would still
 work. Either you can use imports from package, or your own local files.

Or you can export completely rendered profile file without any includes or
variables and modify that as you like.


```bash
# export profile with dynamic includes still active jinja2 files
yacfg --profile [PROFILE] --new-profile my_new_profile.yaml.jinja2
# export completely generated profile without any jinja2 fields, plain yaml
yacfg --profile [PROFILE] --new-profile-static my_new_profile.yaml
vim my_new_profile.yaml
yacfg --profile my_new_profile.yaml
```

Profile is just another jinja2 file that enables customization of profile data
 -- that is tuning. Becuase of that we recommend to keep the extension `.yaml.jinja2`
 unless it is static profile withou any jinja2 capabilities, in that case it could
 be named `.yaml`. That way we can run yaml lint against static profiles and verify
 that they are correct.

 All profiles have to be used to generate files without any tuning. That means,
 if they are tune-able, they have to contain all default values in `_defaults` section.
 That section is also used for tuning, so any variable in there will be exported as tuning.

## Custom templates

The last resort is to export a template and modify that. But remember a template,
or more correctly a template set is a directory containing a set of main
templates that subsequently generate_via_tuning_files a new file.

Of course feel free to write your own templates. Especially when you need to
generate_via_tuning_files files for something that is not packaged.

Just remember for a template set to be identified the directory must contain
a file named '_template' and then main templates ending with '.jinja2'.

```bash
yacfg --template [TEMPLATE] --new-template my_new_template
vim my_new_template/[MAIN_TEMPLATES].jinja2
yacfg --template my_new_template --profile [PROFILE]

```

## API guide

Direct use of API is to use `generate()` nearly the same as the CLI.
With option to use tuning values directly.

Tuning data will be overlaid in order of appearance, using python
dict.update(), so values that will appear later will overwrite previous
values. We recommend that tuning values are always flat, because update
is not recursive. The same applies for data from tuning files as well
as the directly provided data.

Data application order:

- profile defaults
- data from tuning files (in order of appearance) `tuning_files_list`
- data provided directly (in order of appearance) `tuning_data_list`


```python
import yacfg

# generating only broker.xml config using default values from profile,
# no tuning, writing output to a target path
yacfg.generate(
    profile='artemis/2.5.0/default.yaml.jinja2',
    output_filter=['broker.xml'],
    output_path='/opt/artemis-2.5.0-i0/etc/',
)

# using both files and direct values, and writing generated configs to
# a target directory
yacfg.generate(
    profile='artemis/2.5.0/default.yaml.jinja2',
    tuning_files_list=[
        'my_values.yaml',
        'machine_specific.yaml',
        'logging_debug.yaml'
    ],
    tuning_data_list=[
        {'name': 'custom name', 'config': 'option_a'},
        {'address': '10.0.0.1'},
        {'LOG_LEVEL': 'debug'},
    ],
    output_path='/opt/artemis-2.5.0-i0/etc/',
)

# just get generated data for further processing, using just tuning files
data = yacfg.generate(
    profile='artemis/2.5.0/default.yaml.jinja2',
    tuning_files_list=[
        'my_values.yaml',
        'machine_specific.yaml',
        'logging_debug.yaml'
    ],
)
print(data['broker.xml'])
```

## Batch configurations

In case you have multiple services to configure in your environment,
that you probably will have at some point, there is a tool for that
as well. The tool is called yacfg-batch. It has only yaml input and
it uses yacfg to generate configurations as you are already used to.

Input yaml file defines all services you need to generate, what
profiles to use, and what tuning to provide to `yacfg`.
It allows you to configure defaults and common for services.

### Batch profile file

As said it is YAML. It has two special sections: `_default` and `_common`.
As the name suggests, `_default` values are used when values are not
defined per specific section. Where `_common` is added to the values
of all sections. The important thing here is that `_default` has lower
priority than `_common` and that has lower priority than specific section
values.

Every section has 4 values: `profile`, `template`, `tuning_files`,
 and `tuning`. As the name suggests, `profile` defines what generation profile
 to select, and it directly correlates with `yacfg`'s `--profile`.
 `template` defines what generation template to use
 (overrides one in the profile if defined), and it directly correlates with
 `--template` from `yacfg`. `tuning_files` option is a list of tuning
 files to use, when combining defaults, commons, and specific values,
 tuning_files list is concatenated. Finally `tuning` is a map of
 specific tuning values, correlates with `--opt` of `yacfg`. When combining
 defaults, commons, and specifics, it will be updated over using python
 dict.update() and it will work only on first level, so it is recommended
 to use flat values for tuning only.

Example:
```yaml

_default:
    profile: artemis/2.5.0/default.yaml.jinja2
    tuning_files:
      - defaults/broker_default.yaml

_common:
    tuning_files:
      - common/security.yaml
      - common/logging.yaml
    tuning_values:
      LOG_LEVEL_ALL: INFO

brokerA/opt/artemis/etc:
    pass: true

brokerB/opt/artemis/etc:
    profile: artemis/2.5.0/AIOBasic.yaml.jinja2
    tuning_files:
      - brokerB/queues.yaml

---

_default:
    profile: amq_broker/7.2.0/default.yaml.jinja2
    tuning_files:
      - defaults/amq_broker_default.yaml

brokerC/opt/amq/etc:
    tuning:
      LOG_LEVEL_ALL: DEBUG

```

As you can see, `yacfg-batch` supports multiple sections, in single
batch profile file, that allows you to generate multiple groups using
separated `_default` and `_common` sections for that.

### executing batch

When you have defined all tuning files you need, and in the root of this
batch configuration you have your batch profile file, you can now simply
run `yacfg-batch`:

```bash

yacfg-batch --input [batch_profile_file] --output [output_path]
```

You can use multiple input files and all of those will be generated
consecutively. In the output path, new subdirectories will be created
for every item you configure (every section), section key will be used
for that subdirectory. If the section name resembles a path, whole
path will be created. For example for `brokerA/opt/artemis/etc`
the configuration will be generated into
`[output_path]/brokerA/opt/artemis/etc/`.

## Documentation
Formatted documentation can be viewed at [rh-messaging-qe.github.io/yacfg/](https://rh-messaging-qe.github.io/yacfg/).


## Contributing

If you find a bug or room for improvement, submit either a ticket or PR.

## Contributors

_Alphabetically ordered_

* Michal Tóth <mtoth@redhat.com>
* Otavio Piske <opiske@redhat.com>
* Sean Davey <sdavey@redhat.com>
* Zdenek Kraus <zkraus@redhat.com> (maintainer)

## License

Copyright 2018-2020 Red Hat Inc.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

   http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

## Acknowledgments

* [jinja2](http://jinja.pocoo.org/docs/2.10/) -- awesome templating engine
* [yaml](http://yaml.org/) -- very convenient user readable format
* [learn_yaml](https://learnxinyminutes.com/docs/yaml/) -- great YAML cheat sheet
* [pyyaml](https://github.com/yaml/pyyaml) -- python YAML parser
* [jq](https://stedolan.github.io/jq/) -- great tool for working with structured data (JSON)
* [yq](https://yq.readthedocs.io/en/latest/) -- YAML variant of jq
* [github templates examples](https://github.com/stevemao/github-issue-templates/tree/master/simple) -- Nice set of ISSUE_TEMPLATE.md and PULL_REQUESTS_TEMPLATE.md examples
* [contributing example](https://gist.github.com/PurpleBooth/b24679402957c63ec426) -- example/template of CONTRIBUTING.md
* [Fedora Project code-of-conduct](https://docs.fedoraproject.org/en-US/project/code-of-conduct/) -- the inspiration for CODE_OF_CONDUCT.md



            

Raw data

            {
    "_id": null,
    "home_page": "https://github.com/rh-messaging-qe/yacfg",
    "name": "yacfg",
    "maintainer": "Dominik Lenoch",
    "docs_url": null,
    "requires_python": "",
    "maintainer_email": "dlenoch@redhat.com",
    "keywords": "",
    "author": "Zdenek Kraus",
    "author_email": "zkraus@redhat.com",
    "download_url": "https://files.pythonhosted.org/packages/f6/fa/b546bb113a3f8e575e1dbfa2ee66bc29f5b37f4e5bc7de2839c75780fd32/yacfg-0.8.3.tar.gz",
    "platform": "",
    "description": "# yacfg - YAML Configurator\n\nThis tool can generate a set of configuration files mainly needed for\nAMQ Broker, but it is not limited to only generating files for one product.\n\nIt has a user facing Command Line Tool for quick and easy command line usage.\nFurthermore, it is possible to use its API in your python code.\n\n## Getting started\n\n* python 3.5+ or python2.7\n* current requirements from setup.py (runtime requirements only)\n* python virtualenv recommended (install via system package manager\nor `pip install --user virtualenv`)\n\nfor contributors:\n* requirements from requirements.txt (there are Dev and QA requirements as well)\n\n### From git\n\n```bash\ngit clone git@github.com:rh-messaging-qe/yacfg.git\npython -m virtualenv -p python3 venv3\nsource venv3/bin/activate\n./setup.py install\nyacfg --help\n```\n\n### From PyPI\n\n```bash\npython -m virtualenv -p python3 venv3\nsource venv3/bin/activate\npip install yacfg\nyacfg --help\n```\n\n## User (CLI) guide\n\n```bash\nyacfg --help\n\nyacfg --list-profiles\nyacfg --list-templates\n\n# perform a generation of a default profile\nyacfg --profile artemis/2.5.0/default.yaml.jinja2\n# also save result to [OUTDIR] directory\nyacfg --profile [PROFILE] --output [OUTDIR]\n```\n\n## Customization\n\nQuickest way to customize data is to use hot-variables, basically variables\nthat the profile itself provides for tuning. Next step is to write (modify) custom\nprofile with completely custom values.\nIf that does not satisfy your needs, then a custom template might be required.\n\n### Profile tuning\n\nSimply export tuning values from profile you want to tune and change those you\nneed to change. Then supply the custom tuning file(s) when generating the profile.\n\n```bash\nyacfg --profile [PROFILE] --export-tuning my_values.yaml\nvim my_values.yaml\nyacfg --profile [PROFILE] --tune my_values.yaml\n\n# multiple tuning files can be overlaid\n# they are updated in sequence, only values present are overwritten\nyacfg --profile [PROFILE] --tune my_values.yaml --tune machine_specific.yaml \\\n       --tune logging_debug.yaml --output [OUTDIR]\n```\n\n## Custom profiles\n\nWrite your own, or simply export an existing profile and modify that.\n\nYou can export dynamic version with includes of some modules, that would still\n work. Either you can use imports from package, or your own local files.\n\nOr you can export completely rendered profile file without any includes or\nvariables and modify that as you like.\n\n\n```bash\n# export profile with dynamic includes still active jinja2 files\nyacfg --profile [PROFILE] --new-profile my_new_profile.yaml.jinja2\n# export completely generated profile without any jinja2 fields, plain yaml\nyacfg --profile [PROFILE] --new-profile-static my_new_profile.yaml\nvim my_new_profile.yaml\nyacfg --profile my_new_profile.yaml\n```\n\nProfile is just another jinja2 file that enables customization of profile data\n -- that is tuning. Becuase of that we recommend to keep the extension `.yaml.jinja2`\n unless it is static profile withou any jinja2 capabilities, in that case it could\n be named `.yaml`. That way we can run yaml lint against static profiles and verify\n that they are correct.\n\n All profiles have to be used to generate files without any tuning. That means,\n if they are tune-able, they have to contain all default values in `_defaults` section.\n That section is also used for tuning, so any variable in there will be exported as tuning.\n\n## Custom templates\n\nThe last resort is to export a template and modify that. But remember a template,\nor more correctly a template set is a directory containing a set of main\ntemplates that subsequently generate_via_tuning_files a new file.\n\nOf course feel free to write your own templates. Especially when you need to\ngenerate_via_tuning_files files for something that is not packaged.\n\nJust remember for a template set to be identified the directory must contain\na file named '_template' and then main templates ending with '.jinja2'.\n\n```bash\nyacfg --template [TEMPLATE] --new-template my_new_template\nvim my_new_template/[MAIN_TEMPLATES].jinja2\nyacfg --template my_new_template --profile [PROFILE]\n\n```\n\n## API guide\n\nDirect use of API is to use `generate()` nearly the same as the CLI.\nWith option to use tuning values directly.\n\nTuning data will be overlaid in order of appearance, using python\ndict.update(), so values that will appear later will overwrite previous\nvalues. We recommend that tuning values are always flat, because update\nis not recursive. The same applies for data from tuning files as well\nas the directly provided data.\n\nData application order:\n\n- profile defaults\n- data from tuning files (in order of appearance) `tuning_files_list`\n- data provided directly (in order of appearance) `tuning_data_list`\n\n\n```python\nimport yacfg\n\n# generating only broker.xml config using default values from profile,\n# no tuning, writing output to a target path\nyacfg.generate(\n    profile='artemis/2.5.0/default.yaml.jinja2',\n    output_filter=['broker.xml'],\n    output_path='/opt/artemis-2.5.0-i0/etc/',\n)\n\n# using both files and direct values, and writing generated configs to\n# a target directory\nyacfg.generate(\n    profile='artemis/2.5.0/default.yaml.jinja2',\n    tuning_files_list=[\n        'my_values.yaml',\n        'machine_specific.yaml',\n        'logging_debug.yaml'\n    ],\n    tuning_data_list=[\n        {'name': 'custom name', 'config': 'option_a'},\n        {'address': '10.0.0.1'},\n        {'LOG_LEVEL': 'debug'},\n    ],\n    output_path='/opt/artemis-2.5.0-i0/etc/',\n)\n\n# just get generated data for further processing, using just tuning files\ndata = yacfg.generate(\n    profile='artemis/2.5.0/default.yaml.jinja2',\n    tuning_files_list=[\n        'my_values.yaml',\n        'machine_specific.yaml',\n        'logging_debug.yaml'\n    ],\n)\nprint(data['broker.xml'])\n```\n\n## Batch configurations\n\nIn case you have multiple services to configure in your environment,\nthat you probably will have at some point, there is a tool for that\nas well. The tool is called yacfg-batch. It has only yaml input and\nit uses yacfg to generate configurations as you are already used to.\n\nInput yaml file defines all services you need to generate, what\nprofiles to use, and what tuning to provide to `yacfg`.\nIt allows you to configure defaults and common for services.\n\n### Batch profile file\n\nAs said it is YAML. It has two special sections: `_default` and `_common`.\nAs the name suggests, `_default` values are used when values are not\ndefined per specific section. Where `_common` is added to the values\nof all sections. The important thing here is that `_default` has lower\npriority than `_common` and that has lower priority than specific section\nvalues.\n\nEvery section has 4 values: `profile`, `template`, `tuning_files`,\n and `tuning`. As the name suggests, `profile` defines what generation profile\n to select, and it directly correlates with `yacfg`'s `--profile`.\n `template` defines what generation template to use\n (overrides one in the profile if defined), and it directly correlates with\n `--template` from `yacfg`. `tuning_files` option is a list of tuning\n files to use, when combining defaults, commons, and specific values,\n tuning_files list is concatenated. Finally `tuning` is a map of\n specific tuning values, correlates with `--opt` of `yacfg`. When combining\n defaults, commons, and specifics, it will be updated over using python\n dict.update() and it will work only on first level, so it is recommended\n to use flat values for tuning only.\n\nExample:\n```yaml\n\n_default:\n    profile: artemis/2.5.0/default.yaml.jinja2\n    tuning_files:\n      - defaults/broker_default.yaml\n\n_common:\n    tuning_files:\n      - common/security.yaml\n      - common/logging.yaml\n    tuning_values:\n      LOG_LEVEL_ALL: INFO\n\nbrokerA/opt/artemis/etc:\n    pass: true\n\nbrokerB/opt/artemis/etc:\n    profile: artemis/2.5.0/AIOBasic.yaml.jinja2\n    tuning_files:\n      - brokerB/queues.yaml\n\n---\n\n_default:\n    profile: amq_broker/7.2.0/default.yaml.jinja2\n    tuning_files:\n      - defaults/amq_broker_default.yaml\n\nbrokerC/opt/amq/etc:\n    tuning:\n      LOG_LEVEL_ALL: DEBUG\n\n```\n\nAs you can see, `yacfg-batch` supports multiple sections, in single\nbatch profile file, that allows you to generate multiple groups using\nseparated `_default` and `_common` sections for that.\n\n### executing batch\n\nWhen you have defined all tuning files you need, and in the root of this\nbatch configuration you have your batch profile file, you can now simply\nrun `yacfg-batch`:\n\n```bash\n\nyacfg-batch --input [batch_profile_file] --output [output_path]\n```\n\nYou can use multiple input files and all of those will be generated\nconsecutively. In the output path, new subdirectories will be created\nfor every item you configure (every section), section key will be used\nfor that subdirectory. If the section name resembles a path, whole\npath will be created. For example for `brokerA/opt/artemis/etc`\nthe configuration will be generated into\n`[output_path]/brokerA/opt/artemis/etc/`.\n\n## Documentation\nFormatted documentation can be viewed at [rh-messaging-qe.github.io/yacfg/](https://rh-messaging-qe.github.io/yacfg/).\n\n\n## Contributing\n\nIf you find a bug or room for improvement, submit either a ticket or PR.\n\n## Contributors\n\n_Alphabetically ordered_\n\n* Michal T\u00f3th <mtoth@redhat.com>\n* Otavio Piske <opiske@redhat.com>\n* Sean Davey <sdavey@redhat.com>\n* Zdenek Kraus <zkraus@redhat.com> (maintainer)\n\n## License\n\nCopyright 2018-2020 Red Hat Inc.\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this file except in compliance with the License.\nYou may obtain a copy of the License at\n\n   http://www.apache.org/licenses/LICENSE-2.0\n\nUnless required by applicable law or agreed to in writing, software\ndistributed under the License is distributed on an \"AS IS\" BASIS,\nWITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nSee the License for the specific language governing permissions and\nlimitations under the License.\n\n## Acknowledgments\n\n* [jinja2](http://jinja.pocoo.org/docs/2.10/) -- awesome templating engine\n* [yaml](http://yaml.org/) -- very convenient user readable format\n* [learn_yaml](https://learnxinyminutes.com/docs/yaml/) -- great YAML cheat sheet\n* [pyyaml](https://github.com/yaml/pyyaml) -- python YAML parser\n* [jq](https://stedolan.github.io/jq/) -- great tool for working with structured data (JSON)\n* [yq](https://yq.readthedocs.io/en/latest/) -- YAML variant of jq\n* [github templates examples](https://github.com/stevemao/github-issue-templates/tree/master/simple) -- Nice set of ISSUE_TEMPLATE.md and PULL_REQUESTS_TEMPLATE.md examples\n* [contributing example](https://gist.github.com/PurpleBooth/b24679402957c63ec426) -- example/template of CONTRIBUTING.md\n* [Fedora Project code-of-conduct](https://docs.fedoraproject.org/en-US/project/code-of-conduct/) -- the inspiration for CODE_OF_CONDUCT.md\n\n\n",
    "bugtrack_url": null,
    "license": "Apache-2.0",
    "summary": "Template based configuration generator",
    "version": "0.8.3",
    "split_keywords": [],
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "e9b7c43fdec2df421d7ffe0e42a71d97",
                "sha256": "5d542d88bd3415023153dda46898601401824a3637ee38386cde2988d39abf09"
            },
            "downloads": -1,
            "filename": "yacfg-0.8.3-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "e9b7c43fdec2df421d7ffe0e42a71d97",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": null,
            "size": 661667,
            "upload_time": "2021-02-23T17:54:45",
            "upload_time_iso_8601": "2021-02-23T17:54:45.608209Z",
            "url": "https://files.pythonhosted.org/packages/ed/12/834438474bebfb37537bd8ef8e8028f5f4705c0ba87b437f0a8d9848add5/yacfg-0.8.3-py3-none-any.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": "",
            "digests": {
                "md5": "6ef3cae576124c150498d6a17f60a514",
                "sha256": "f63e3fc03649296b807f49bb7c73bd168b83a642d8c971e703d825aee65e941a"
            },
            "downloads": -1,
            "filename": "yacfg-0.8.3.tar.gz",
            "has_sig": false,
            "md5_digest": "6ef3cae576124c150498d6a17f60a514",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 275348,
            "upload_time": "2021-02-23T17:54:47",
            "upload_time_iso_8601": "2021-02-23T17:54:47.065631Z",
            "url": "https://files.pythonhosted.org/packages/f6/fa/b546bb113a3f8e575e1dbfa2ee66bc29f5b37f4e5bc7de2839c75780fd32/yacfg-0.8.3.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2021-02-23 17:54:47",
    "github": true,
    "gitlab": false,
    "bitbucket": false,
    "github_user": null,
    "github_project": "rh-messaging-qe",
    "error": "Could not fetch GitHub repository",
    "lcname": "yacfg"
}
        
Elapsed time: 0.24845s