.. _yaml-to-mux-plugin:
YAML to Mux plugin
==================
`avocado_varianter_yaml_to_mux`
This plugin utilizes the ``multiplexation`` mechanism to
produce variants out of a yaml file. This section is example-based,
if you are interested in test parameters and/or ``multiplexation``
overview, please take a look at `test-parameter`.
As mentioned earlier, it inherits from the
`avocado_varianter_yaml_to_mux.mux.MuxPlugin`
and the only thing it implements is the argument parsing
to get some input and a custom ``yaml``
parser (which is also capable of parsing ``json``).
The YAML file is perfect for this task as it's easily read by
both, humans and machines. Let's start with an example (line
numbers at the first columns are for documentation purposes only,
they are not part of the multiplex file format):
.. code-block:: yaml
1 hw:
2 cpu: !mux
3 intel:
4 cpu_CFLAGS: '-march=core2'
5 amd:
6 cpu_CFLAGS: '-march=athlon64'
7 arm:
8 cpu_CFLAGS: '-mabi=apcs-gnu -march=armv8-a -mtune=arm8'
9 disk: !mux
10 scsi:
11 disk_type: 'scsi'
12 virtio:
13 disk_type: 'virtio'
14 distro: !mux
15 fedora:
16 init: 'systemd'
17 mint:
18 init: 'systemv'
19 env: !mux
20 debug:
21 opt_CFLAGS: '-O0 -g'
22 prod:
23 opt_CFLAGS: '-O2'
.. warning:: On some architectures misbehaving versions of CYaml
Python library were reported and Avocado always fails with
``unacceptable character #x0000: control characters are not
allowed``. To workaround this issue you need to either update
the PyYaml to the version which works properly, or you need
to remove the ``python2.7/site-packages/yaml/cyaml.py`` or
disable CYaml import in Avocado sources. For details check
out the `Github issue <https://github.com/avocado-framework/avocado/issues/1190>`_
There are couple of ``key=>value`` pairs (lines 4,6,8,11,13,...) and there are
named nodes which define scope (lines 1,2,3,5,7,9,...). There are also additional
flags (lines 2, 9, 14, 19) which modifies the behavior.
Nodes
-----
They define context of the ``key=>value`` pairs allowing us to easily identify
for what this values might be used for and also it makes possible to define
multiple values of the same keys with different scope.
Due to their purpose the YAML automatic type conversion for nodes names
is disabled, so the value of node name is always as written in the YAML
file (unlike values, where ``yes`` converts to ``True`` and such).
Nodes are organized in parent-child relationship and together they create
a tree. To view this structure use ``avocado variants --tree -m <file>``::
┗━━ run
┣━━ hw
┃ ┣━━ cpu
┃ ┃ ╠══ intel
┃ ┃ ╠══ amd
┃ ┃ ╚══ arm
┃ ┗━━ disk
┃ ╠══ scsi
┃ ╚══ virtio
┣━━ distro
┃ ╠══ fedora
┃ ╚══ mint
┗━━ env
╠══ debug
╚══ prod
You can see that ``hw`` has 2 children ``cpu`` and ``disk``. All parameters
defined in parent node are inherited to children and extended/overwritten by
their values up to the leaf nodes. The leaf nodes (``intel``, ``amd``, ``arm``,
``scsi``, ...) are the most important as after multiplexation they form the
parameters available in tests.
Keys and Values
---------------
Every value other than dict (4,6,8,11) is used as value of the antecedent
node.
Each node can define key/value pairs (lines 4,6,8,11,...). Additionally
each children node inherits values of it's parent and the result is called
node ``environment``.
Given the node structure below:
.. code-block:: yaml
devtools:
compiler: 'cc'
flags:
- '-O2'
debug: '-g'
fedora:
compiler: 'gcc'
flags:
- '-Wall'
osx:
compiler: 'clang'
flags:
- '-arch i386'
- '-arch x86_64'
And the rules defined as:
* Scalar values (Booleans, Numbers and Strings) are overwritten by walking from the root until the final node.
* Lists are appended (to the tail) whenever we walk from the root to the final node.
The environment created for the nodes ``fedora`` and ``osx`` are:
- Node ``//devtools/fedora`` environment ``compiler: 'gcc'``, ``flags: ['-O2', '-Wall']``
- Node ``//devtools/osx`` environment ``compiler: 'clang'``, ``flags: ['-O2', '-arch i386', '-arch x86_64']``
Note that due to different usage of key and values in environment we disabled
the automatic value conversion for keys while keeping it enabled for values.
This means that the key is always a string and the value can be YAML value,
eg. bool, list, custom type, or string. Please be aware that due to limitation
None type can be provided in yaml specifically as string 'null'.
Variants
--------
In the end all leaves are gathered and turned into parameters, more specifically into
``AvocadoParams``:
.. code-block:: yaml
setup:
graphic:
user: "guest"
password: "pass"
text:
user: "root"
password: "123456"
produces ``[graphic, text]``. In the test code you'll be able to query only
those leaves. Intermediary or root nodes are available.
The example above generates a single test execution with parameters separated
by path. But the most powerful multiplexer feature is that it can generate
multiple variants. To do that you need to tag a node whose children are
meant to be multiplexed. Effectively it returns only leaves of one child at the
time.In order to generate all possible variants multiplexer creates cartesian
product of all of these variants:
.. code-block:: yaml
cpu: !mux
intel:
amd:
arm:
fmt: !mux
qcow2:
raw:
Produces 6 variants::
/cpu/intel, /fmt/qcow2
/cpu/intel, /fmt/raw
...
/cpu/arm, /fmt/raw
The !mux evaluation is recursive so one variant can expand to multiple
ones:
.. code-block:: yaml
fmt: !mux
qcow: !mux
2:
2v3:
raw:
Results in::
/fmt/qcow2/2
/fmt/qcow2/2v3
/raw
.. _yaml-to-mux-resolution-order:
Resolution order
----------------
You can see that only leaves are part of the test parameters. It might happen
that some of these leaves contain different values of the same key. Then
you need to make sure your queries separate them by different paths. When
the path matches multiple results with different origin, an exception is raised
as it's impossible to guess which key was originally intended.
To avoid these problems it's recommended to use unique names in test parameters if
possible, to avoid the mentioned clashes. It also makes it easier to extend or mix
multiple YAML files for a test.
For multiplex YAML files that are part of a framework, contain default
configurations, or serve as plugin configurations and other advanced setups it is
possible and commonly desirable to use non-unique names. But always keep those points
in mind and provide sensible paths.
Multiplexer also supports default paths. By default it's ``/run/*`` but it can
be overridden by ``--mux-path``, which accepts multiple arguments. What it does
it splits leaves by the provided paths. Each query goes one by one through
those sub-trees and first one to hit the match returns the result. It might not
solve all problems, but it can help to combine existing YAML files with your
ones:
.. code-block:: yaml
qa: # large and complex read-only file, content injected into /qa
tests:
timeout: 10
...
my_variants: !mux # your YAML file injected into /my_variants
short:
timeout: 1
long:
timeout: 1000
You want to use an existing test which uses ``params.get('timeout', '*')``. Then you
can use ``--mux-path '/my_variants/*' '/qa/*'`` and it'll first look in your
variants. If no matches are found, then it would proceed to ``/qa/*``
Keep in mind that only slices defined in mux-path are taken into account for
relative paths (the ones starting with ``*``)
Injecting files
---------------
You can run any test with any YAML file by::
avocado run sleeptest.py --mux-yaml file.yaml
This puts the content of ``file.yaml`` into ``/run``
location, which as mentioned in previous section, is the default ``mux-path``
path. For most simple cases this is the expected behavior as your files
are available in the default path and you can safely use ``params.get(key)``.
When you need to put a file into a different location, for example
when you have two files and you don't want the content to be merged into
a single place becoming effectively a single blob, you can do that by
giving a name to your YAML file::
avocado run sleeptest.py --mux-yaml duration:duration.yaml
The content of ``duration.yaml`` is injected into ``/run/duration``. Still when
keys from other files don't clash, you can use ``params.get(key)`` and retrieve
from this location as it's in the default path, only extended by the
``duration`` intermediary node. Another benefit is you can merge or separate
multiple files by using the same or different name, or even a complex
(relative) path.
Last but not least, advanced users can inject the file into whatever location
they prefer by::
avocado run sleeptest.py --mux-yaml /my/variants/duration:duration.yaml
Simple ``params.get(key)`` won't look in this location, which might be the
intention of the test writer. There are several ways to access the values:
* absolute location ``params.get(key, '/my/variants/duration')``
* absolute location with wildcards ``params.get(key, '/my/*)``
(or ``/*/duration/*``...)
* set the mux-path ``avocado run ... --mux-path /my/*`` and use relative path
It's recommended to use the simple injection for single YAML files, relative
injection for multiple simple YAML files and the last option is for very
advanced setups when you either can't modify the YAML files and you need to
specify custom resolution order or you are specifying non-test parameters, for
example parameters for your plugin, which you need to separate from the test
parameters.
Special values
--------------
As you might have noticed, we are using mapping/dicts to define the structure
of the params. To avoid surprises we disallowed the smart typing of mapping
keys so:
.. code-block:: yaml
on: on
Won't become ``True: True``, but the key will be preserved as string
``on: True``.
You might also want to use dict as values in your params. This is also
supported but as we can't easily distinguish whether that value is
a value or a node (structure), you have to either embed it into another
object (list, ..) or you have to clearly state the type (yaml tag
``!!python/dict``). Even then the value won't be a standard dictionary,
but it'll be ``collections.OrderedDict`` and similarly to nodes
structure all keys are preserved as strings and no smart type detection
is used. Apart from that it should behave similarly as dict, only you
get the values ordered by the order they appear in the file.
Multiple files
--------------
You can provide multiple files. In such scenario final tree is a combination
of the provided files where later nodes with the same name override values of
the preceding corresponding node. New nodes are appended as new children:
.. code-block:: yaml
file-1.yaml:
debug:
CFLAGS: '-O0 -g'
prod:
CFLAGS: '-O2'
file-2.yaml:
prod:
CFLAGS: '-Os'
fast:
CFLAGS: '-Ofast'
results in:
.. code-block:: yaml
debug:
CFLAGS: '-O0 -g'
prod:
CFLAGS: '-Os' # overridden
fast:
CFLAGS: '-Ofast' # appended
It's also possible to include existing file into another a given node in another
file. This is done by the `!include : $path` directive:
.. code-block:: yaml
os:
fedora:
!include : fedora.yaml
gentoo:
!include : gentoo.yaml
.. warning:: Due to YAML nature, it's **mandatory** to put space between
`!include` and the colon (`:`) that must follow it.
The file location can be either absolute path or relative path to the YAML
file where the `!include` is called (even when it's nested).
Whole file is **merged** into the node where it's defined.
Advanced YAML tags
------------------
There are additional features related to YAML files. Most of them require values
separated by ``":"``. Again, in all such cases it's mandatory to add a white space
(``" "``) between the tag and the ``":"``, otherwise ``":"`` is part of the tag
name and the parsing fails.
!include
^^^^^^^^
Includes other file and injects it into the node it's specified in:
.. code-block:: yaml
my_other_file:
!include : other.yaml
The content of ``/my_other_file`` would be parsed from the ``other.yaml``. It's
the hardcoded equivalent of the ``-m $using:$path``.
Relative paths start from the original file's directory.
!using
^^^^^^
Prepends path to the node it's defined in:
.. code-block:: yaml
!using : /foo
bar:
!using : baz
``bar`` is put into ``baz`` becoming ``/baz/bar`` and everything is put into
``/foo``. So the final path of ``bar`` is ``/foo/baz/bar``.
!remove_node
^^^^^^^^^^^^
Removes node if it existed during the merge. It can be used to extend
incompatible YAML files:
.. code-block:: yaml
os:
fedora:
windows:
3.11:
95:
os:
!remove_node : windows
windows:
win3.11:
win95:
Removes the `windows` node from structure. It's different from `filter-out`
as it really removes the node (and all children) from the tree and
it can be replaced by you new structure as shown in the example. It removes
`windows` with all children and then replaces this structure with slightly
modified version.
As `!remove_node` is processed during merge, when you reverse the order,
windows is not removed and you end-up with `/windows/{win3.11,win95,3.11,95}`
nodes.
!remove_value
^^^^^^^^^^^^^
It's similar to `!remove_node`_ only with values.
!mux
^^^^
Children of this node will be multiplexed. This means that in first variant
it'll return leaves of the first child, in second the leaves of the second
child, etc. Example is in section `Variants`_
!filter-only
------------
Defines internal filters. They are inherited by children and evaluated
during multiplexation. It allows one to specify the only compatible branch
of the tree with the current variant, for example::
cpu:
arm:
!filter-only : /disk/virtio
disk:
virtio:
scsi:
will skip the ``[arm, scsi]`` variant and result only in ``[arm, virtio]``
_Note: It's possible to use ``!filter-only`` multiple times with the same
parent and all allowed variants will be included (unless they are
filtered-out by ``!filter-out``)_
_Note2: The evaluation order is 1. filter-out, 2. filter-only. This means when
you booth filter-out and filter-only a branch it won't take part in the
multiplexed variants._
!filter-out
-----------
Similarly to `!filter-only`_ only it skips the specified branches and leaves
the remaining ones. (in the same example the use of
``!filter-out : /disk/scsi`` results in the same behavior). The difference
is when a new disk type is introduced, ``!filter-only`` still allows just
the specified variants, while ``!filter-out`` only removes the specified
ones.
As for the speed optimization, currently Avocado is strongly optimized
towards fast ``!filter-out`` so it's highly recommended using them
rather than ``!filter-only``, which takes significantly longer to
process.
Complete example
----------------
Let's take a second look at the first example::
1 hw:
2 cpu: !mux
3 intel:
4 cpu_CFLAGS: '-march=core2'
5 amd:
6 cpu_CFLAGS: '-march=athlon64'
7 arm:
8 cpu_CFLAGS: '-mabi=apcs-gnu -march=armv8-a -mtune=arm8'
9 disk: !mux
10 scsi:
11 disk_type: 'scsi'
12 virtio:
13 disk_type: 'virtio'
14 distro: !mux
15 fedora:
16 init: 'systemd'
17 mint:
18 init: 'systemv'
19 env: !mux
20 debug:
21 opt_CFLAGS: '-O0 -g'
22 prod:
23 opt_CFLAGS: '-O2'
After filters are applied (simply removes non-matching variants), leaves
are gathered and all variants are generated::
$ avocado variants -m selftests/.data/mux-environment.yaml
Variants generated:
Variant 1: /hw/cpu/intel, /hw/disk/scsi, /distro/fedora, /env/debug
Variant 2: /hw/cpu/intel, /hw/disk/scsi, /distro/fedora, /env/prod
Variant 3: /hw/cpu/intel, /hw/disk/scsi, /distro/mint, /env/debug
Variant 4: /hw/cpu/intel, /hw/disk/scsi, /distro/mint, /env/prod
Variant 5: /hw/cpu/intel, /hw/disk/virtio, /distro/fedora, /env/debug
Variant 6: /hw/cpu/intel, /hw/disk/virtio, /distro/fedora, /env/prod
Variant 7: /hw/cpu/intel, /hw/disk/virtio, /distro/mint, /env/debug
Variant 8: /hw/cpu/intel, /hw/disk/virtio, /distro/mint, /env/prod
Variant 9: /hw/cpu/amd, /hw/disk/scsi, /distro/fedora, /env/debug
Variant 10: /hw/cpu/amd, /hw/disk/scsi, /distro/fedora, /env/prod
Variant 11: /hw/cpu/amd, /hw/disk/scsi, /distro/mint, /env/debug
Variant 12: /hw/cpu/amd, /hw/disk/scsi, /distro/mint, /env/prod
Variant 13: /hw/cpu/amd, /hw/disk/virtio, /distro/fedora, /env/debug
Variant 14: /hw/cpu/amd, /hw/disk/virtio, /distro/fedora, /env/prod
Variant 15: /hw/cpu/amd, /hw/disk/virtio, /distro/mint, /env/debug
Variant 16: /hw/cpu/amd, /hw/disk/virtio, /distro/mint, /env/prod
Variant 17: /hw/cpu/arm, /hw/disk/scsi, /distro/fedora, /env/debug
Variant 18: /hw/cpu/arm, /hw/disk/scsi, /distro/fedora, /env/prod
Variant 19: /hw/cpu/arm, /hw/disk/scsi, /distro/mint, /env/debug
Variant 20: /hw/cpu/arm, /hw/disk/scsi, /distro/mint, /env/prod
Variant 21: /hw/cpu/arm, /hw/disk/virtio, /distro/fedora, /env/debug
Variant 22: /hw/cpu/arm, /hw/disk/virtio, /distro/fedora, /env/prod
Variant 23: /hw/cpu/arm, /hw/disk/virtio, /distro/mint, /env/debug
Variant 24: /hw/cpu/arm, /hw/disk/virtio, /distro/mint, /env/prod
Where the first variant contains::
/hw/cpu/intel/ => cpu_CFLAGS: -march=core2
/hw/disk/ => disk_type: scsi
/distro/fedora/ => init: systemd
/env/debug/ => opt_CFLAGS: -O0 -g
The second one::
/hw/cpu/intel/ => cpu_CFLAGS: -march=core2
/hw/disk/ => disk_type: scsi
/distro/fedora/ => init: systemd
/env/prod/ => opt_CFLAGS: -O2
From this example you can see that querying for ``/env/debug`` works only in
the first variant, but returns nothing in the second variant. Keep this in mind
and when you use the ``!mux`` flag always query for the pre-mux path,
``/env/*`` in this example.
Injecting values
----------------
Beyond the values injected by YAML files specified it's also possible
inject values directly from command line to the final multiplex tree.
It's done by the argument ``--mux-inject``. The format of expected
value is ``[path:]key:node_value``.
.. warning:: When no path is specified to ``--mux-inject`` the parameter
is added under tree root ``/``. For example: running avocado passing
``--mux-inject my_key:my_value`` the parameter can be accessed calling
``self.params.get('my_key')``. If the test writer wants to put the injected
value in any other path location, like extending the ``/run`` path, it needs
to be informed on avocado run call. For example: ``--mux-inject
/run/:my_key:my_value`` makes possible to access the parameters
calling ``self.params.get('my_key', '/run')``
A test that gets parameters without a defined path, such as
``examples/tests/multiplextest.py``::
os_type = self.params.get('os_type', default='linux')
Running it::
$ avocado --show=test run -- examples/tests/multiplextest.py | grep os_type
PARAMS (key=os_type, path=*, default=linux) => 'linux'
Now, injecting a value, by default will put it in /, which is not in the
default list of paths searched for::
$ avocado --show=test run --mux-inject os_type:myos -- examples/tests/multiplextest.py | grep os_type
PARAMS (key=os_type, path=*, default=linux) => 'linux'
A path that is searched for by default is /run. To set the value to that path use::
$ avocado --show=test run --mux-inject /run:os_type:myos -- examples/tests/multiplextest.py | grep os_type
PARAMS (key=os_type, path=*, default=linux) => 'myos'
Or, add the / to the list of paths searched for by default::
$ avocado --show=test run --mux-inject os_type:myos --mux-path / -- examples/tests/multiplextest.py | grep os_type
PARAMS (key=os_type, path=*, default=linux) => 'myos'
.. warning:: By default, the values are parsed for the respective data types.
When not possible, it falls back to string. If you want to maintain some
value as string, enclose within quotes, properly escaped, and eclose that
again in quotes.
For example: a value of ``1`` is treated as integer, a value of ``1,2`` is
treated as list, a value of ``abc`` is treated as string, a value of
``1,2,5-10`` is treated as list of integers as ``1,2,-5``. If you want to
maintain this as string, provide the value as ``"\"1,2,5-10\""``
.. _mutliplexer:
Multiplexer
-----------
`avocado_varianter_yaml_to_mux.mux`
``Multiplexer`` or simply ``Mux`` is an abstract concept, which was
the basic idea behind the tree-like params structure with the support
to produce all possible variants. There is a core implementation of
basic building blocks that can be used when creating a custom plugin.
There is a demonstration version of plugin using this concept in
`avocado_varianter_yaml_to_mux`
which adds a parser and then
uses this multiplexer concept to define an Avocado plugin to produce
variants from ``yaml`` (or ``json``) files.
Multiplexer concept
^^^^^^^^^^^^^^^^^^^
As mentioned earlier, this is an in-core implementation of building
blocks intended for writing `varianter-plugins` based on a tree
with `Multiplex domains`_ defined. The available blocks are:
* `MuxTree`_ - Object which represents a part of the tree and handles
the multiplexation, which means producing all possible variants
from a tree-like object.
* `MuxPlugin`_ - Base class to build `varianter-plugins`
* ``MuxTreeNode`` - Inherits from `tree-node` and adds the support for
control flags (``MuxTreeNode.ctrl``) and multiplex domains
(``MuxTreeNode.multiplex``).
And some support classes and methods eg. for filtering and so on.
Multiplex domains
^^^^^^^^^^^^^^^^^
A default ``avocado-params`` tree with variables could look like this::
Multiplex tree representation:
┣━━ paths
┃ → tmp: /var/tmp
┃ → qemu: /usr/libexec/qemu-kvm
┗━━ environ
→ debug: False
The multiplexer wants to produce similar structure, but also to be able
to define not just one variant, but to define all possible combinations
and then report the slices as variants. We use the term
`Multiplex domains`_ to define that children of this node are not just
different paths, but they are different values and we only want one at
a time. In the representation we use double-line to visibly distinguish
between normal relation and multiplexed relation. Let's modify our
example a bit::
Multiplex tree representation:
┣━━ paths
┃ → tmp: /var/tmp
┃ → qemu: /usr/libexec/qemu-kvm
┗━━ environ
╠══ production
║ → debug: False
╚══ debug
→ debug: True
The difference is that ``environ`` is now a ``multiplex`` node and it's
children will be yielded one at a time producing two variants::
Variant 1:
┣━━ paths
┃ → tmp: /var/tmp
┃ → qemu: /usr/libexec/qemu-kvm
┗━━ environ
┗━━ production
→ debug: False
Variant 2:
┣━━ paths
┃ → tmp: /var/tmp
┃ → qemu: /usr/libexec/qemu-kvm
┗━━ environ
┗━━ debug
→ debug: False
Note that the ``multiplex`` is only about direct children, therefore
the number of leaves in variants might differ::
Multiplex tree representation:
┣━━ paths
┃ → tmp: /var/tmp
┃ → qemu: /usr/libexec/qemu-kvm
┗━━ environ
╠══ production
║ → debug: False
╚══ debug
┣━━ system
┃ → debug: False
┗━━ program
→ debug: True
Produces one variant with ``/paths`` and ``/environ/production`` and
other variant with ``/paths``, ``/environ/debug/system`` and
``/environ/debug/program``.
As mentioned earlier the power is not in producing one variant, but
in defining huge scenarios with all possible variants. By using
tree-structure with multiplex domains you can avoid most of the
ugly filters you might know from Jenkins sparse matrix jobs.
For comparison let's have a look at the same example in Avocado::
Multiplex tree representation:
┗━━ os
┣━━ distro
┃ ┗━━ redhat
┃ ╠══ fedora
┃ ║ ┣━━ version
┃ ║ ┃ ╠══ 20
┃ ║ ┃ ╚══ 21
┃ ║ ┗━━ flavor
┃ ║ ╠══ workstation
┃ ║ ╚══ cloud
┃ ╚══ rhel
┃ ╠══ 5
┃ ╚══ 6
┗━━ arch
╠══ i386
╚══ x86_64
Which produces::
Variant 1: /os/distro/redhat/fedora/version/20, /os/distro/redhat/fedora/flavor/workstation, /os/arch/i386
Variant 2: /os/distro/redhat/fedora/version/20, /os/distro/redhat/fedora/flavor/workstation, /os/arch/x86_64
Variant 3: /os/distro/redhat/fedora/version/20, /os/distro/redhat/fedora/flavor/cloud, /os/arch/i386
Variant 4: /os/distro/redhat/fedora/version/20, /os/distro/redhat/fedora/flavor/cloud, /os/arch/x86_64
Variant 5: /os/distro/redhat/fedora/version/21, /os/distro/redhat/fedora/flavor/workstation, /os/arch/i386
Variant 6: /os/distro/redhat/fedora/version/21, /os/distro/redhat/fedora/flavor/workstation, /os/arch/x86_64
Variant 7: /os/distro/redhat/fedora/version/21, /os/distro/redhat/fedora/flavor/cloud, /os/arch/i386
Variant 8: /os/distro/redhat/fedora/version/21, /os/distro/redhat/fedora/flavor/cloud, /os/arch/x86_64
Variant 9: /os/distro/redhat/rhel/5, /os/arch/i386
Variant 10: /os/distro/redhat/rhel/5, /os/arch/x86_64
Variant 11: /os/distro/redhat/rhel/6, /os/arch/i386
Variant 12: /os/distro/redhat/rhel/6, /os/arch/x86_64
Versus Jenkins sparse matrix::
os_version = fedora20 fedora21 rhel5 rhel6
os_flavor = none workstation cloud
arch = i386 x86_64
filter = ((os_version == "rhel5").implies(os_flavor == "none") &&
(os_version == "rhel6").implies(os_flavor == "none")) &&
!(os_version == "fedora20" && os_flavor == "none") &&
!(os_version == "fedora21" && os_flavor == "none")
Which is still relatively simple example, but it grows dramatically with
inner-dependencies.
MuxPlugin
^^^^^^^^^
`avocado_varianter_yaml_to_mux.mux.MuxPlugin`
Defines the full interface required by
`avocado.core.plugin_interfaces.Varianter`. The plugin writer
should inherit from this ``MuxPlugin``, then from the ``Varianter``
and call the::
self.initialize_mux(root, paths, debug)
Where:
* root - is the root of your params tree (compound of `tree-node` -like
nodes)
* paths - is the `parameter-paths` to be used in test with all variants
* debug - whether to use debug mode (requires the passed tree to be
compound of ``TreeNodeDebug``-like nodes which stores the origin
of the variant/value/environment as the value for listing purposes
and is __NOT__ intended for test execution.
This method must be called before the `varianter`'s second
stage. The `MuxPlugin`_'s code will take care of the rest.
MuxTree
^^^^^^^
This is the core feature where the hard work happens. It walks the tree
and remembers all leaf nodes or uses list of ``MuxTrees`` when another
multiplex domain is reached while searching for a leaf.
When it's asked to report variants, it combines one variant of each
remembered item (leaf node always stays the same, but ``MuxTree`` circles
through it's values) which recursively produces all possible variants
of different `multiplex domains`_.
Raw data
{
"_id": null,
"home_page": "http://avocado-framework.github.io/",
"name": "avocado-framework-plugin-varianter-yaml-to-mux",
"maintainer": null,
"docs_url": null,
"requires_python": null,
"maintainer_email": null,
"keywords": null,
"author": "Avocado Developers",
"author_email": "avocado-devel@redhat.com",
"download_url": "https://files.pythonhosted.org/packages/21/0d/c5419e0b4dbd5cce3f7b47ff6b696a65b2620bb93882e9a30d5df5b336be/avocado_framework_plugin_varianter_yaml_to_mux-105.0.tar.gz",
"platform": null,
"description": ".. _yaml-to-mux-plugin:\n\nYAML to Mux plugin\n==================\n\n`avocado_varianter_yaml_to_mux`\n\nThis plugin utilizes the ``multiplexation`` mechanism to\nproduce variants out of a yaml file. This section is example-based,\nif you are interested in test parameters and/or ``multiplexation``\noverview, please take a look at `test-parameter`.\n\nAs mentioned earlier, it inherits from the\n`avocado_varianter_yaml_to_mux.mux.MuxPlugin`\nand the only thing it implements is the argument parsing\nto get some input and a custom ``yaml``\nparser (which is also capable of parsing ``json``).\n\nThe YAML file is perfect for this task as it's easily read by\nboth, humans and machines. Let's start with an example (line\nnumbers at the first columns are for documentation purposes only,\nthey are not part of the multiplex file format):\n\n.. code-block:: yaml\n\n 1 hw:\n 2 cpu: !mux\n 3 intel:\n 4 cpu_CFLAGS: '-march=core2'\n 5 amd:\n 6 cpu_CFLAGS: '-march=athlon64'\n 7 arm:\n 8 cpu_CFLAGS: '-mabi=apcs-gnu -march=armv8-a -mtune=arm8'\n 9 disk: !mux\n 10 scsi:\n 11 disk_type: 'scsi'\n 12 virtio:\n 13 disk_type: 'virtio'\n 14 distro: !mux\n 15 fedora:\n 16 init: 'systemd'\n 17 mint:\n 18 init: 'systemv'\n 19 env: !mux\n 20 debug:\n 21 opt_CFLAGS: '-O0 -g'\n 22 prod:\n 23 opt_CFLAGS: '-O2'\n\n\n.. warning:: On some architectures misbehaving versions of CYaml\n Python library were reported and Avocado always fails with\n ``unacceptable character #x0000: control characters are not\n allowed``. To workaround this issue you need to either update\n the PyYaml to the version which works properly, or you need\n to remove the ``python2.7/site-packages/yaml/cyaml.py`` or\n disable CYaml import in Avocado sources. For details check\n out the `Github issue <https://github.com/avocado-framework/avocado/issues/1190>`_\n\nThere are couple of ``key=>value`` pairs (lines 4,6,8,11,13,...) and there are\nnamed nodes which define scope (lines 1,2,3,5,7,9,...). There are also additional\nflags (lines 2, 9, 14, 19) which modifies the behavior.\n\n\nNodes\n-----\n\nThey define context of the ``key=>value`` pairs allowing us to easily identify\nfor what this values might be used for and also it makes possible to define\nmultiple values of the same keys with different scope.\n\nDue to their purpose the YAML automatic type conversion for nodes names\nis disabled, so the value of node name is always as written in the YAML\nfile (unlike values, where ``yes`` converts to ``True`` and such).\n\nNodes are organized in parent-child relationship and together they create\na tree. To view this structure use ``avocado variants --tree -m <file>``::\n\n \u2517\u2501\u2501 run\n \u2523\u2501\u2501 hw\n \u2503 \u2523\u2501\u2501 cpu\n \u2503 \u2503 \u2560\u2550\u2550 intel\n \u2503 \u2503 \u2560\u2550\u2550 amd\n \u2503 \u2503 \u255a\u2550\u2550 arm\n \u2503 \u2517\u2501\u2501 disk\n \u2503 \u2560\u2550\u2550 scsi\n \u2503 \u255a\u2550\u2550 virtio\n \u2523\u2501\u2501 distro\n \u2503 \u2560\u2550\u2550 fedora\n \u2503 \u255a\u2550\u2550 mint\n \u2517\u2501\u2501 env\n \u2560\u2550\u2550 debug\n \u255a\u2550\u2550 prod\n\nYou can see that ``hw`` has 2 children ``cpu`` and ``disk``. All parameters\ndefined in parent node are inherited to children and extended/overwritten by\ntheir values up to the leaf nodes. The leaf nodes (``intel``, ``amd``, ``arm``,\n``scsi``, ...) are the most important as after multiplexation they form the\nparameters available in tests.\n\n\nKeys and Values\n---------------\n\nEvery value other than dict (4,6,8,11) is used as value of the antecedent\nnode.\n\nEach node can define key/value pairs (lines 4,6,8,11,...). Additionally\neach children node inherits values of it's parent and the result is called\nnode ``environment``.\n\nGiven the node structure below:\n\n.. code-block:: yaml\n\n devtools:\n compiler: 'cc'\n flags:\n - '-O2'\n debug: '-g'\n fedora:\n compiler: 'gcc'\n flags:\n - '-Wall'\n osx:\n compiler: 'clang'\n flags:\n - '-arch i386'\n - '-arch x86_64'\n\nAnd the rules defined as:\n\n* Scalar values (Booleans, Numbers and Strings) are overwritten by walking from the root until the final node.\n* Lists are appended (to the tail) whenever we walk from the root to the final node.\n\nThe environment created for the nodes ``fedora`` and ``osx`` are:\n\n- Node ``//devtools/fedora`` environment ``compiler: 'gcc'``, ``flags: ['-O2', '-Wall']``\n- Node ``//devtools/osx`` environment ``compiler: 'clang'``, ``flags: ['-O2', '-arch i386', '-arch x86_64']``\n\nNote that due to different usage of key and values in environment we disabled\nthe automatic value conversion for keys while keeping it enabled for values.\nThis means that the key is always a string and the value can be YAML value,\neg. bool, list, custom type, or string. Please be aware that due to limitation\nNone type can be provided in yaml specifically as string 'null'.\n\nVariants\n--------\n\nIn the end all leaves are gathered and turned into parameters, more specifically into\n``AvocadoParams``:\n\n.. code-block:: yaml\n\n setup:\n graphic:\n user: \"guest\"\n password: \"pass\"\n text:\n user: \"root\"\n password: \"123456\"\n\nproduces ``[graphic, text]``. In the test code you'll be able to query only\nthose leaves. Intermediary or root nodes are available.\n\nThe example above generates a single test execution with parameters separated\nby path. But the most powerful multiplexer feature is that it can generate\nmultiple variants. To do that you need to tag a node whose children are\nmeant to be multiplexed. Effectively it returns only leaves of one child at the\ntime.In order to generate all possible variants multiplexer creates cartesian\nproduct of all of these variants:\n\n.. code-block:: yaml\n\n cpu: !mux\n intel:\n amd:\n arm:\n fmt: !mux\n qcow2:\n raw:\n\nProduces 6 variants::\n\n /cpu/intel, /fmt/qcow2\n /cpu/intel, /fmt/raw\n ...\n /cpu/arm, /fmt/raw\n\nThe !mux evaluation is recursive so one variant can expand to multiple\nones:\n\n.. code-block:: yaml\n\n fmt: !mux\n qcow: !mux\n 2:\n 2v3:\n raw:\n\nResults in::\n\n /fmt/qcow2/2\n /fmt/qcow2/2v3\n /raw\n\n\n.. _yaml-to-mux-resolution-order:\n\nResolution order\n----------------\n\nYou can see that only leaves are part of the test parameters. It might happen\nthat some of these leaves contain different values of the same key. Then\nyou need to make sure your queries separate them by different paths. When\nthe path matches multiple results with different origin, an exception is raised\nas it's impossible to guess which key was originally intended.\n\nTo avoid these problems it's recommended to use unique names in test parameters if\npossible, to avoid the mentioned clashes. It also makes it easier to extend or mix\nmultiple YAML files for a test.\n\nFor multiplex YAML files that are part of a framework, contain default\nconfigurations, or serve as plugin configurations and other advanced setups it is\npossible and commonly desirable to use non-unique names. But always keep those points\nin mind and provide sensible paths.\n\nMultiplexer also supports default paths. By default it's ``/run/*`` but it can\nbe overridden by ``--mux-path``, which accepts multiple arguments. What it does\nit splits leaves by the provided paths. Each query goes one by one through\nthose sub-trees and first one to hit the match returns the result. It might not\nsolve all problems, but it can help to combine existing YAML files with your\nones:\n\n.. code-block:: yaml\n\n qa: # large and complex read-only file, content injected into /qa\n tests:\n timeout: 10\n ...\n my_variants: !mux # your YAML file injected into /my_variants\n short:\n timeout: 1\n long:\n timeout: 1000\n\nYou want to use an existing test which uses ``params.get('timeout', '*')``. Then you\ncan use ``--mux-path '/my_variants/*' '/qa/*'`` and it'll first look in your\nvariants. If no matches are found, then it would proceed to ``/qa/*``\n\nKeep in mind that only slices defined in mux-path are taken into account for\nrelative paths (the ones starting with ``*``)\n\n\nInjecting files\n---------------\n\nYou can run any test with any YAML file by::\n\n avocado run sleeptest.py --mux-yaml file.yaml\n\nThis puts the content of ``file.yaml`` into ``/run``\nlocation, which as mentioned in previous section, is the default ``mux-path``\npath. For most simple cases this is the expected behavior as your files\nare available in the default path and you can safely use ``params.get(key)``.\n\nWhen you need to put a file into a different location, for example\nwhen you have two files and you don't want the content to be merged into\na single place becoming effectively a single blob, you can do that by\ngiving a name to your YAML file::\n\n avocado run sleeptest.py --mux-yaml duration:duration.yaml\n\nThe content of ``duration.yaml`` is injected into ``/run/duration``. Still when\nkeys from other files don't clash, you can use ``params.get(key)`` and retrieve\nfrom this location as it's in the default path, only extended by the\n``duration`` intermediary node. Another benefit is you can merge or separate\nmultiple files by using the same or different name, or even a complex\n(relative) path.\n\nLast but not least, advanced users can inject the file into whatever location\nthey prefer by::\n\n avocado run sleeptest.py --mux-yaml /my/variants/duration:duration.yaml\n\nSimple ``params.get(key)`` won't look in this location, which might be the\nintention of the test writer. There are several ways to access the values:\n\n* absolute location ``params.get(key, '/my/variants/duration')``\n* absolute location with wildcards ``params.get(key, '/my/*)``\n (or ``/*/duration/*``...)\n* set the mux-path ``avocado run ... --mux-path /my/*`` and use relative path\n\nIt's recommended to use the simple injection for single YAML files, relative\ninjection for multiple simple YAML files and the last option is for very\nadvanced setups when you either can't modify the YAML files and you need to\nspecify custom resolution order or you are specifying non-test parameters, for\nexample parameters for your plugin, which you need to separate from the test\nparameters.\n\n\nSpecial values\n--------------\n\nAs you might have noticed, we are using mapping/dicts to define the structure\nof the params. To avoid surprises we disallowed the smart typing of mapping\nkeys so:\n\n.. code-block:: yaml\n\n on: on\n\nWon't become ``True: True``, but the key will be preserved as string\n``on: True``.\n\nYou might also want to use dict as values in your params. This is also\nsupported but as we can't easily distinguish whether that value is\na value or a node (structure), you have to either embed it into another\nobject (list, ..) or you have to clearly state the type (yaml tag\n``!!python/dict``). Even then the value won't be a standard dictionary,\nbut it'll be ``collections.OrderedDict`` and similarly to nodes\nstructure all keys are preserved as strings and no smart type detection\nis used. Apart from that it should behave similarly as dict, only you\nget the values ordered by the order they appear in the file.\n\nMultiple files\n--------------\n\nYou can provide multiple files. In such scenario final tree is a combination\nof the provided files where later nodes with the same name override values of\nthe preceding corresponding node. New nodes are appended as new children:\n\n.. code-block:: yaml\n\n file-1.yaml:\n debug:\n CFLAGS: '-O0 -g'\n prod:\n CFLAGS: '-O2'\n\n file-2.yaml:\n prod:\n CFLAGS: '-Os'\n fast:\n CFLAGS: '-Ofast'\n\nresults in:\n\n.. code-block:: yaml\n\n debug:\n CFLAGS: '-O0 -g'\n prod:\n CFLAGS: '-Os' # overridden\n fast:\n CFLAGS: '-Ofast' # appended\n\nIt's also possible to include existing file into another a given node in another\nfile. This is done by the `!include : $path` directive:\n\n.. code-block:: yaml\n\n os:\n fedora:\n !include : fedora.yaml\n gentoo:\n !include : gentoo.yaml\n\n.. warning:: Due to YAML nature, it's **mandatory** to put space between\n `!include` and the colon (`:`) that must follow it.\n\nThe file location can be either absolute path or relative path to the YAML\nfile where the `!include` is called (even when it's nested).\n\nWhole file is **merged** into the node where it's defined.\n\n\nAdvanced YAML tags\n------------------\n\nThere are additional features related to YAML files. Most of them require values\nseparated by ``\":\"``. Again, in all such cases it's mandatory to add a white space\n(``\" \"``) between the tag and the ``\":\"``, otherwise ``\":\"`` is part of the tag\nname and the parsing fails.\n\n!include\n^^^^^^^^\n\nIncludes other file and injects it into the node it's specified in:\n\n.. code-block:: yaml\n\n my_other_file:\n !include : other.yaml\n\nThe content of ``/my_other_file`` would be parsed from the ``other.yaml``. It's\nthe hardcoded equivalent of the ``-m $using:$path``.\n\nRelative paths start from the original file's directory.\n\n!using\n^^^^^^\n\nPrepends path to the node it's defined in:\n\n.. code-block:: yaml\n\n !using : /foo\n bar:\n !using : baz\n\n``bar`` is put into ``baz`` becoming ``/baz/bar`` and everything is put into\n``/foo``. So the final path of ``bar`` is ``/foo/baz/bar``.\n\n!remove_node\n^^^^^^^^^^^^\n\nRemoves node if it existed during the merge. It can be used to extend\nincompatible YAML files:\n\n.. code-block:: yaml\n\n os:\n fedora:\n windows:\n 3.11:\n 95:\n os:\n !remove_node : windows\n windows:\n win3.11:\n win95:\n\nRemoves the `windows` node from structure. It's different from `filter-out`\nas it really removes the node (and all children) from the tree and\nit can be replaced by you new structure as shown in the example. It removes\n`windows` with all children and then replaces this structure with slightly\nmodified version.\n\nAs `!remove_node` is processed during merge, when you reverse the order,\nwindows is not removed and you end-up with `/windows/{win3.11,win95,3.11,95}`\nnodes.\n\n!remove_value\n^^^^^^^^^^^^^\n\nIt's similar to `!remove_node`_ only with values.\n\n!mux\n^^^^\n\nChildren of this node will be multiplexed. This means that in first variant\nit'll return leaves of the first child, in second the leaves of the second\nchild, etc. Example is in section `Variants`_\n\n!filter-only\n------------\n\nDefines internal filters. They are inherited by children and evaluated\nduring multiplexation. It allows one to specify the only compatible branch\nof the tree with the current variant, for example::\n\n cpu:\n arm:\n !filter-only : /disk/virtio\n disk:\n virtio:\n scsi:\n\nwill skip the ``[arm, scsi]`` variant and result only in ``[arm, virtio]``\n\n_Note: It's possible to use ``!filter-only`` multiple times with the same\nparent and all allowed variants will be included (unless they are\nfiltered-out by ``!filter-out``)_\n\n_Note2: The evaluation order is 1. filter-out, 2. filter-only. This means when\nyou booth filter-out and filter-only a branch it won't take part in the\nmultiplexed variants._\n\n!filter-out\n-----------\n\nSimilarly to `!filter-only`_ only it skips the specified branches and leaves\nthe remaining ones. (in the same example the use of\n``!filter-out : /disk/scsi`` results in the same behavior). The difference\nis when a new disk type is introduced, ``!filter-only`` still allows just\nthe specified variants, while ``!filter-out`` only removes the specified\nones.\n\nAs for the speed optimization, currently Avocado is strongly optimized\ntowards fast ``!filter-out`` so it's highly recommended using them\nrather than ``!filter-only``, which takes significantly longer to\nprocess.\n\nComplete example\n----------------\n\nLet's take a second look at the first example::\n\n 1 hw:\n 2 cpu: !mux\n 3 intel:\n 4 cpu_CFLAGS: '-march=core2'\n 5 amd:\n 6 cpu_CFLAGS: '-march=athlon64'\n 7 arm:\n 8 cpu_CFLAGS: '-mabi=apcs-gnu -march=armv8-a -mtune=arm8'\n 9 disk: !mux\n 10 scsi:\n 11 disk_type: 'scsi'\n 12 virtio:\n 13 disk_type: 'virtio'\n 14 distro: !mux\n 15 fedora:\n 16 init: 'systemd'\n 17 mint:\n 18 init: 'systemv'\n 19 env: !mux\n 20 debug:\n 21 opt_CFLAGS: '-O0 -g'\n 22 prod:\n 23 opt_CFLAGS: '-O2'\n\nAfter filters are applied (simply removes non-matching variants), leaves\nare gathered and all variants are generated::\n\n $ avocado variants -m selftests/.data/mux-environment.yaml\n Variants generated:\n Variant 1: /hw/cpu/intel, /hw/disk/scsi, /distro/fedora, /env/debug\n Variant 2: /hw/cpu/intel, /hw/disk/scsi, /distro/fedora, /env/prod\n Variant 3: /hw/cpu/intel, /hw/disk/scsi, /distro/mint, /env/debug\n Variant 4: /hw/cpu/intel, /hw/disk/scsi, /distro/mint, /env/prod\n Variant 5: /hw/cpu/intel, /hw/disk/virtio, /distro/fedora, /env/debug\n Variant 6: /hw/cpu/intel, /hw/disk/virtio, /distro/fedora, /env/prod\n Variant 7: /hw/cpu/intel, /hw/disk/virtio, /distro/mint, /env/debug\n Variant 8: /hw/cpu/intel, /hw/disk/virtio, /distro/mint, /env/prod\n Variant 9: /hw/cpu/amd, /hw/disk/scsi, /distro/fedora, /env/debug\n Variant 10: /hw/cpu/amd, /hw/disk/scsi, /distro/fedora, /env/prod\n Variant 11: /hw/cpu/amd, /hw/disk/scsi, /distro/mint, /env/debug\n Variant 12: /hw/cpu/amd, /hw/disk/scsi, /distro/mint, /env/prod\n Variant 13: /hw/cpu/amd, /hw/disk/virtio, /distro/fedora, /env/debug\n Variant 14: /hw/cpu/amd, /hw/disk/virtio, /distro/fedora, /env/prod\n Variant 15: /hw/cpu/amd, /hw/disk/virtio, /distro/mint, /env/debug\n Variant 16: /hw/cpu/amd, /hw/disk/virtio, /distro/mint, /env/prod\n Variant 17: /hw/cpu/arm, /hw/disk/scsi, /distro/fedora, /env/debug\n Variant 18: /hw/cpu/arm, /hw/disk/scsi, /distro/fedora, /env/prod\n Variant 19: /hw/cpu/arm, /hw/disk/scsi, /distro/mint, /env/debug\n Variant 20: /hw/cpu/arm, /hw/disk/scsi, /distro/mint, /env/prod\n Variant 21: /hw/cpu/arm, /hw/disk/virtio, /distro/fedora, /env/debug\n Variant 22: /hw/cpu/arm, /hw/disk/virtio, /distro/fedora, /env/prod\n Variant 23: /hw/cpu/arm, /hw/disk/virtio, /distro/mint, /env/debug\n Variant 24: /hw/cpu/arm, /hw/disk/virtio, /distro/mint, /env/prod\n\nWhere the first variant contains::\n\n /hw/cpu/intel/ => cpu_CFLAGS: -march=core2\n /hw/disk/ => disk_type: scsi\n /distro/fedora/ => init: systemd\n /env/debug/ => opt_CFLAGS: -O0 -g\n\nThe second one::\n\n /hw/cpu/intel/ => cpu_CFLAGS: -march=core2\n /hw/disk/ => disk_type: scsi\n /distro/fedora/ => init: systemd\n /env/prod/ => opt_CFLAGS: -O2\n\nFrom this example you can see that querying for ``/env/debug`` works only in\nthe first variant, but returns nothing in the second variant. Keep this in mind\nand when you use the ``!mux`` flag always query for the pre-mux path,\n``/env/*`` in this example.\n\n\nInjecting values\n----------------\n\nBeyond the values injected by YAML files specified it's also possible\ninject values directly from command line to the final multiplex tree.\nIt's done by the argument ``--mux-inject``. The format of expected\nvalue is ``[path:]key:node_value``.\n\n.. warning:: When no path is specified to ``--mux-inject`` the parameter\n is added under tree root ``/``. For example: running avocado passing\n ``--mux-inject my_key:my_value`` the parameter can be accessed calling\n ``self.params.get('my_key')``. If the test writer wants to put the injected\n value in any other path location, like extending the ``/run`` path, it needs\n to be informed on avocado run call. For example: ``--mux-inject\n /run/:my_key:my_value`` makes possible to access the parameters\n calling ``self.params.get('my_key', '/run')``\n\n\nA test that gets parameters without a defined path, such as\n``examples/tests/multiplextest.py``::\n\n os_type = self.params.get('os_type', default='linux')\n\nRunning it::\n\n $ avocado --show=test run -- examples/tests/multiplextest.py | grep os_type\n PARAMS (key=os_type, path=*, default=linux) => 'linux'\n\nNow, injecting a value, by default will put it in /, which is not in the\ndefault list of paths searched for::\n\n $ avocado --show=test run --mux-inject os_type:myos -- examples/tests/multiplextest.py | grep os_type\n PARAMS (key=os_type, path=*, default=linux) => 'linux'\n\nA path that is searched for by default is /run. To set the value to that path use::\n\n $ avocado --show=test run --mux-inject /run:os_type:myos -- examples/tests/multiplextest.py | grep os_type\n PARAMS (key=os_type, path=*, default=linux) => 'myos'\n\nOr, add the / to the list of paths searched for by default::\n\n $ avocado --show=test run --mux-inject os_type:myos --mux-path / -- examples/tests/multiplextest.py | grep os_type\n PARAMS (key=os_type, path=*, default=linux) => 'myos'\n\n.. warning:: By default, the values are parsed for the respective data types.\n When not possible, it falls back to string. If you want to maintain some\n value as string, enclose within quotes, properly escaped, and eclose that\n again in quotes.\n For example: a value of ``1`` is treated as integer, a value of ``1,2`` is\n treated as list, a value of ``abc`` is treated as string, a value of\n ``1,2,5-10`` is treated as list of integers as ``1,2,-5``. If you want to\n maintain this as string, provide the value as ``\"\\\"1,2,5-10\\\"\"``\n\n.. _mutliplexer:\n\nMultiplexer\n-----------\n\n`avocado_varianter_yaml_to_mux.mux`\n\n``Multiplexer`` or simply ``Mux`` is an abstract concept, which was\nthe basic idea behind the tree-like params structure with the support\nto produce all possible variants. There is a core implementation of\nbasic building blocks that can be used when creating a custom plugin.\nThere is a demonstration version of plugin using this concept in\n`avocado_varianter_yaml_to_mux`\nwhich adds a parser and then\nuses this multiplexer concept to define an Avocado plugin to produce\nvariants from ``yaml`` (or ``json``) files.\n\n\nMultiplexer concept\n^^^^^^^^^^^^^^^^^^^\n\nAs mentioned earlier, this is an in-core implementation of building\nblocks intended for writing `varianter-plugins` based on a tree\nwith `Multiplex domains`_ defined. The available blocks are:\n\n* `MuxTree`_ - Object which represents a part of the tree and handles\n the multiplexation, which means producing all possible variants\n from a tree-like object.\n* `MuxPlugin`_ - Base class to build `varianter-plugins`\n* ``MuxTreeNode`` - Inherits from `tree-node` and adds the support for\n control flags (``MuxTreeNode.ctrl``) and multiplex domains\n (``MuxTreeNode.multiplex``).\n\nAnd some support classes and methods eg. for filtering and so on.\n\nMultiplex domains\n^^^^^^^^^^^^^^^^^\n\nA default ``avocado-params`` tree with variables could look like this::\n\n Multiplex tree representation:\n \u2523\u2501\u2501 paths\n \u2503 \u2192 tmp: /var/tmp\n \u2503 \u2192 qemu: /usr/libexec/qemu-kvm\n \u2517\u2501\u2501 environ\n \u2192 debug: False\n\nThe multiplexer wants to produce similar structure, but also to be able\nto define not just one variant, but to define all possible combinations\nand then report the slices as variants. We use the term\n`Multiplex domains`_ to define that children of this node are not just\ndifferent paths, but they are different values and we only want one at\na time. In the representation we use double-line to visibly distinguish\nbetween normal relation and multiplexed relation. Let's modify our\nexample a bit::\n\n Multiplex tree representation:\n \u2523\u2501\u2501 paths\n \u2503 \u2192 tmp: /var/tmp\n \u2503 \u2192 qemu: /usr/libexec/qemu-kvm\n \u2517\u2501\u2501 environ\n \u2560\u2550\u2550 production\n \u2551 \u2192 debug: False\n \u255a\u2550\u2550 debug\n \u2192 debug: True\n\nThe difference is that ``environ`` is now a ``multiplex`` node and it's\nchildren will be yielded one at a time producing two variants::\n\n Variant 1:\n \u2523\u2501\u2501 paths\n \u2503 \u2192 tmp: /var/tmp\n \u2503 \u2192 qemu: /usr/libexec/qemu-kvm\n \u2517\u2501\u2501 environ\n \u2517\u2501\u2501 production\n \u2192 debug: False\n Variant 2:\n \u2523\u2501\u2501 paths\n \u2503 \u2192 tmp: /var/tmp\n \u2503 \u2192 qemu: /usr/libexec/qemu-kvm\n \u2517\u2501\u2501 environ\n \u2517\u2501\u2501 debug\n \u2192 debug: False\n\nNote that the ``multiplex`` is only about direct children, therefore\nthe number of leaves in variants might differ::\n\n Multiplex tree representation:\n \u2523\u2501\u2501 paths\n \u2503 \u2192 tmp: /var/tmp\n \u2503 \u2192 qemu: /usr/libexec/qemu-kvm\n \u2517\u2501\u2501 environ\n \u2560\u2550\u2550 production\n \u2551 \u2192 debug: False\n \u255a\u2550\u2550 debug\n \u2523\u2501\u2501 system\n \u2503 \u2192 debug: False\n \u2517\u2501\u2501 program\n \u2192 debug: True\n\nProduces one variant with ``/paths`` and ``/environ/production`` and\nother variant with ``/paths``, ``/environ/debug/system`` and\n``/environ/debug/program``.\n\nAs mentioned earlier the power is not in producing one variant, but\nin defining huge scenarios with all possible variants. By using\ntree-structure with multiplex domains you can avoid most of the\nugly filters you might know from Jenkins sparse matrix jobs.\nFor comparison let's have a look at the same example in Avocado::\n\n Multiplex tree representation:\n \u2517\u2501\u2501 os\n \u2523\u2501\u2501 distro\n \u2503 \u2517\u2501\u2501 redhat\n \u2503 \u2560\u2550\u2550 fedora\n \u2503 \u2551 \u2523\u2501\u2501 version\n \u2503 \u2551 \u2503 \u2560\u2550\u2550 20\n \u2503 \u2551 \u2503 \u255a\u2550\u2550 21\n \u2503 \u2551 \u2517\u2501\u2501 flavor\n \u2503 \u2551 \u2560\u2550\u2550 workstation\n \u2503 \u2551 \u255a\u2550\u2550 cloud\n \u2503 \u255a\u2550\u2550 rhel\n \u2503 \u2560\u2550\u2550 5\n \u2503 \u255a\u2550\u2550 6\n \u2517\u2501\u2501 arch\n \u2560\u2550\u2550 i386\n \u255a\u2550\u2550 x86_64\n\nWhich produces::\n\n Variant 1: /os/distro/redhat/fedora/version/20, /os/distro/redhat/fedora/flavor/workstation, /os/arch/i386\n Variant 2: /os/distro/redhat/fedora/version/20, /os/distro/redhat/fedora/flavor/workstation, /os/arch/x86_64\n Variant 3: /os/distro/redhat/fedora/version/20, /os/distro/redhat/fedora/flavor/cloud, /os/arch/i386\n Variant 4: /os/distro/redhat/fedora/version/20, /os/distro/redhat/fedora/flavor/cloud, /os/arch/x86_64\n Variant 5: /os/distro/redhat/fedora/version/21, /os/distro/redhat/fedora/flavor/workstation, /os/arch/i386\n Variant 6: /os/distro/redhat/fedora/version/21, /os/distro/redhat/fedora/flavor/workstation, /os/arch/x86_64\n Variant 7: /os/distro/redhat/fedora/version/21, /os/distro/redhat/fedora/flavor/cloud, /os/arch/i386\n Variant 8: /os/distro/redhat/fedora/version/21, /os/distro/redhat/fedora/flavor/cloud, /os/arch/x86_64\n Variant 9: /os/distro/redhat/rhel/5, /os/arch/i386\n Variant 10: /os/distro/redhat/rhel/5, /os/arch/x86_64\n Variant 11: /os/distro/redhat/rhel/6, /os/arch/i386\n Variant 12: /os/distro/redhat/rhel/6, /os/arch/x86_64\n\nVersus Jenkins sparse matrix::\n\n os_version = fedora20 fedora21 rhel5 rhel6\n os_flavor = none workstation cloud\n arch = i386 x86_64\n\n filter = ((os_version == \"rhel5\").implies(os_flavor == \"none\") &&\n (os_version == \"rhel6\").implies(os_flavor == \"none\")) &&\n !(os_version == \"fedora20\" && os_flavor == \"none\") &&\n !(os_version == \"fedora21\" && os_flavor == \"none\")\n\nWhich is still relatively simple example, but it grows dramatically with\ninner-dependencies.\n\nMuxPlugin\n^^^^^^^^^\n\n`avocado_varianter_yaml_to_mux.mux.MuxPlugin`\n\nDefines the full interface required by\n`avocado.core.plugin_interfaces.Varianter`. The plugin writer\nshould inherit from this ``MuxPlugin``, then from the ``Varianter``\nand call the::\n\n self.initialize_mux(root, paths, debug)\n\nWhere:\n\n* root - is the root of your params tree (compound of `tree-node` -like\n nodes)\n* paths - is the `parameter-paths` to be used in test with all variants\n* debug - whether to use debug mode (requires the passed tree to be\n compound of ``TreeNodeDebug``-like nodes which stores the origin\n of the variant/value/environment as the value for listing purposes\n and is __NOT__ intended for test execution.\n\nThis method must be called before the `varianter`'s second\nstage. The `MuxPlugin`_'s code will take care of the rest.\n\nMuxTree\n^^^^^^^\n\nThis is the core feature where the hard work happens. It walks the tree\nand remembers all leaf nodes or uses list of ``MuxTrees`` when another\nmultiplex domain is reached while searching for a leaf.\n\nWhen it's asked to report variants, it combines one variant of each\nremembered item (leaf node always stays the same, but ``MuxTree`` circles\nthrough it's values) which recursively produces all possible variants\nof different `multiplex domains`_.\n",
"bugtrack_url": null,
"license": null,
"summary": "Avocado Varianter plugin to parse YAML file into variants",
"version": "105.0",
"project_urls": {
"Homepage": "http://avocado-framework.github.io/"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "948d58d3e25a7a042a8454d87590600aa539b3b62db330509ba31c4da8c32a5f",
"md5": "bc40668f0acd41e91be5bfec1a464259",
"sha256": "cf0bb68c4e70f16682474c673cb04448cdfd7a70ac8a46d806ef0a459b7cdf58"
},
"downloads": -1,
"filename": "avocado_framework_plugin_varianter_yaml_to_mux-105.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "bc40668f0acd41e91be5bfec1a464259",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 20267,
"upload_time": "2024-05-07T18:47:19",
"upload_time_iso_8601": "2024-05-07T18:47:19.116727Z",
"url": "https://files.pythonhosted.org/packages/94/8d/58d3e25a7a042a8454d87590600aa539b3b62db330509ba31c4da8c32a5f/avocado_framework_plugin_varianter_yaml_to_mux-105.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "210dc5419e0b4dbd5cce3f7b47ff6b696a65b2620bb93882e9a30d5df5b336be",
"md5": "078a88f63ca9714cfbfef2271dbf9bea",
"sha256": "64737272e1422c7cfc3b1d86690c56f447f8f88c6ba05673d64a584ee7715817"
},
"downloads": -1,
"filename": "avocado_framework_plugin_varianter_yaml_to_mux-105.0.tar.gz",
"has_sig": false,
"md5_digest": "078a88f63ca9714cfbfef2271dbf9bea",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 28853,
"upload_time": "2024-05-07T18:47:40",
"upload_time_iso_8601": "2024-05-07T18:47:40.103627Z",
"url": "https://files.pythonhosted.org/packages/21/0d/c5419e0b4dbd5cce3f7b47ff6b696a65b2620bb93882e9a30d5df5b336be/avocado_framework_plugin_varianter_yaml_to_mux-105.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-05-07 18:47:40",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "avocado-framework-plugin-varianter-yaml-to-mux"
}
JSON