sic2dc

Name	sic2dc JSON
Version	0.0.5 JSON
	download
home_page	None
Summary	Simple indented config to dict compare.
upload_time	2024-11-18 09:52:34
maintainer	None
docs_url	None
author	None
requires_python	>=3.11
license	MIT License Copyright (c) 2024 alexonishchenko Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
keywords	configuration difference diff compare config-compare network network-config
VCS
bugtrack_url
requirements	deepdiff pydantic pydantic_core ruamel.yaml ruamel.yaml.clib
Travis-CI	No Travis.
coveralls test coverage	No coveralls.

            # sic2dc
Simple indented config to dict compare.


# Summary
This is another configuration diff tool. It transforms indented configs into nested dictionaries, applies 'filters' to both dictionaries and then compares them. Applying filters helps skipping hidden/default lines or parts of the configuration which we are not interested in comparing.
Before transforing into dicts the configs can be cured (e.g. indentation may be added).
Also sic2dc can ignore "command" / "no command" in the same configuration section.


# Why
When comparing desired to operstate configurations one can face some difficulties that can be overcome by sic2dc.
 - desired state sections may be sorted in a way that doesn't match operstate but after applying it will give the same result.
 - device OS may sort sections in some odd way which is not always possible to replicate in desired state.
 - desired state config may have default lines which will be hidden in operstate configuration.
 - device OS can add lines to operstate config
 - device OS can alter incomming lines with unique values
 - desired state config may consist of multiple parts (e.g. arista configlets) which may override sections from each other
 - configuration syntax may need some treatment before comparison.


# Install

    pip install sic2dc

# Usage

## cli
```bash
# help
sic2dc  -h
usage: sic2dc [-h] -c1 -c2 -s [-f] [-c] [-g]

Simple indented config to dict compare.

options:
  -h, --help           show this help message and exit
  -c1, --config-1  relative path to the first config.
  -c2, --config-2  relative path to the second config.
  -s, --settings   relative path to settings yaml.
  -f, --filters    relative path to filters list yaml.
  -c, --cures      relative path to cures list yaml.
  -g, --no-color   disable color.
```

## cli example
```bash
sic2dc -c1 intended/sw1.cfg -c2 oper/sw1.cfg -s sic2dc/settings_arista_dcs.yml
```

```diff        
interface Port-Channel1
-   no shutdown
interface Ethernet1
+   no switchport
+ snmp-server engineID local 123
+ system l1
+   unsupported speed action error
+   unsupported error-correction action error
+ interface Ethernet3
+   shutdown
+   no switchport
- errdisable recovery interval 300
- router bfd
-   multihop interval 300 min-rx 300 multiplier 3
interface Port-Channel2
-   no shutdown
router bgp 66666
-   bgp default ipv4-unicast
```

The following options are required: c1, c2, settings. If no filters or cures are passed the configs are transformed and compared as they are. Filters and cures are yaml files with lists at the top level. Settings is a yaml with a dict.

## python
```python
from sic2dc import sic2dc
f1 = 'path_to_c1'
f2 = 'path_to_c2'
settings = {
    'indent_char': ' ',
    'indent': 3,
    'comments': ['^\s*[\!\#].*?$', '^\s*$'],
}

filters = []
cures = []

result = sic2dc(f1, f2, settings, filters=filters, cures=cures, color=True)

result['diff_dict']
result['diff_lines']
```        
## ansible filter
```python
"""filter_plugins file"""
from sic2dc import sic2dc
class FilterModule(object):
    def filters(self):
        return {'sic2dc': sic2dc}
```
```yaml
# playbook
# settings, filters and cures can be set as ansible vars of a host.
- set_fact:
    cfg_diff: "{{ f1 | sic2dc(f2, settings, filters, cures, False) }}"
- debug:
    msg: "{{ cfg_diff['diff_lines'] | join('\n') }}"
  when: cfg_diff['diff_dict']
- fail:
  when: cfg_diff['diff_dict']
```

# Concepts
## dicts
When a config is transformed into dict the lines of the config are trimmed and become dict keys.
```python
router bgp 1234    # spaces in the end of line
  router-id 1234
# transforms into
{'router bgp 1234': {'router-id 1234': {}}}
```

## path
Since we compare nested dicts **path** is used to define config parts of interest. Path is a list of regex patterns.
<br>path examples
```python
path = ['interface [Ee]thernet \S+'] # all ethernet interfaces
path = ['router bgp \d+', 'address-family .*'] # all address-families in 'router bgp' section. 
```
## whens
When applying filters **whens** are used to select more specific sections. Imagine we want to select all unused interfaces that exist in operstate and do not exist in desired state. So they should be 'shutdown' and they should be absent in destination.

```yaml
path: [^interface (Ethernet|Management).*]
when:
  - has_children: ['^shutdown$']
  - absent_in_destination: True
```
## examples
See [examples](https://github.com/alexonishchenko/sic2dc/tree/main/sic2dc/example) for filters/cures/settings examples

# Settings
Settings configure the following parameters (example for b4com switches).

```yaml
# enable/disable deleting of command/no command from both configs
ignore_cmd_nocmd: True

# indent char
indent_char: ' '

# number of indent_chars on single indentation level
indent: 1

# list of patterns for comment lines. they will be deleted from both configs.
comments:
  - '^\s*[\!\#].*?$'
  - '^\s*$'
  - '^\s*exit\s*$'
  - '^end\s*$'
```



# Filters
Filters mostly copy, delete or change sections in c1 and c2. A filter is defined by the following fileds:
 - **action** - str value e.g cp21, cp12, upd1, upd2, del1, del2
 - **when** - list of when conditions ('has_children', 'doesnt_have_chidren', 'absent_in_destination')
 - **path** - path to set config parts of interest.
 - **data** - dict of data used by filter (in udpate filters)


### cp21. Copy from c2 to c1
```yaml
# arista.desiredstate: copy unused interfaces from operstate
- action: cp21
  path: [^interface (Ethernet|Management).*]
  when:
    - has_children: ['^shutdown$']
    - absent_in_destination: True
```
### cp12. Copy from c1 to c2
Does the opposite.

### upd2. Update c2 with data
```yaml
# arista.operstate: add swprt mode access if swprt access vlan
#   in operstate config this is hidden
- action: upd2
  path: [^interface Eth.*]
  data:
    switchport mode access: {}
  when:
    - has_children: [switchport access vlan.*]
```
### upd1. Update c1 with data
Same as upd2

### del1. Delete section in c1
```yaml
# arista.desiredstate: delete errdisable default value
- action: del1
  path: [^errdisable recovery interval 300]
```

### del2. Delete section in c2
```yaml
# arista.operstate: delete snmp engine id
- action: del2
  path: ['snmp-server engineID .*']
```


# Cures
Cures are defined by **action** and its **kwargs**. They are applied to configs prior to dict transformation.
Currently the only cure supported is "enter_exit". It adds indentation by pattern.

## enter_exit
Find 'enter' pattern, add a single level of indentation to all following lines until 'exit' pattern is met.

```yaml
- action: enter_exit
  kwargs:
    enter_exits:
      - enter: ' address-family \S+\s.*$'
        exit: ' exit-address-family$'
```

The cure above transform example

        router bgp 1234
         address-family l2vpn evpn
         neighbor 10.10.10.7 activate
         neighbor 10.10.10.8 activate
         exit-address-family
        ->
        router bgp 1234
         address-family l2vpn evpn
          neighbor 10.10.10.7 activate
          neighbor 10.10.10.8 activate
          exit-address-family

# Api notes
## sic2dc

```python
def sic2dc(
        f1: str,
        f2: str,
        settings: dict,
        filters: list[dict] | None = None,
        cures: list[dict] | None = None,
        color: bool = False) -> dict:
    """
    Creates ConfigCompareBase object and compares f1 and f2.
    Returns ConfigCompareBase.diff_dict and ConfigCompareBase.dump() lines as dict
    Returns dict:
        'diff_dict': dict
        'diff_lines': str
    """
```

## ConfigCompareBase

```python
class ConfigCompareBase(...):
    def __init__(self, f1: str, f2: str, settings: CfgCmprSettings,
                 filters: list[dict] = None, cures: list[dict] = None):
        """
        1. Create cc object: read files, apply cures and create d1 and d2. 
        2. Apply filters to dicts
        3. Run comparison
        """

cc = ConfigCompareBase(...)

# uncured c1 and c2:
cc.c1_uncured
cc.c2_uncured

# cured c1 and c2
cc.c1
cc.c2

# unfiltered d1 and d2
cc.d1_unfiltered
cc.d2_unfiltered

# filtered d1 and d2
cc.d1
cc.d2

# dump
bw_diff_text = cc.dump(quiet=True, color=False)
color_diff_text = cc.dump(quiet=True, color=True)

# printout color diff
cc.dump(color=True)
```

            

Raw data

            {
    "_id": null,
    "home_page": null,
    "name": "sic2dc",
    "maintainer": null,
    "docs_url": null,
    "requires_python": ">=3.11",
    "maintainer_email": null,
    "keywords": "configuration, difference, diff, compare, config-compare, network, network-config",
    "author": null,
    "author_email": "Alexander Onishchenko <alexonishchenko@gmail.com>",
    "download_url": "https://files.pythonhosted.org/packages/d7/f7/97f02a7689b3804b5a2dbdc912f9f65c0af09091b74567ed7eaf79cab6e1/sic2dc-0.0.5.tar.gz",
    "platform": null,
    "description": "# sic2dc\nSimple indented config to dict compare.\n\n\n# Summary\nThis is another configuration diff tool. It transforms indented configs into nested dictionaries, applies 'filters' to both dictionaries and then compares them. Applying filters helps skipping hidden/default lines or parts of the configuration which we are not interested in comparing.\nBefore transforing into dicts the configs can be cured (e.g. indentation may be added).\nAlso sic2dc can ignore \"command\" / \"no command\" in the same configuration section.\n\n\n# Why\nWhen comparing desired to operstate configurations one can face some difficulties that can be overcome by sic2dc.\n - desired state sections may be sorted in a way that doesn't match operstate but after applying it will give the same result.\n - device OS may sort sections in some odd way which is not always possible to replicate in desired state.\n - desired state config may have default lines which will be hidden in operstate configuration.\n - device OS can add lines to operstate config\n - device OS can alter incomming lines with unique values\n - desired state config may consist of multiple parts (e.g. arista configlets) which may override sections from each other\n - configuration syntax may need some treatment before comparison.\n\n\n# Install\n\n    pip install sic2dc\n\n# Usage\n\n## cli\n```bash\n# help\nsic2dc  -h\nusage: sic2dc [-h] -c1 -c2 -s [-f] [-c] [-g]\n\nSimple indented config to dict compare.\n\noptions:\n  -h, --help           show this help message and exit\n  -c1, --config-1  relative path to the first config.\n  -c2, --config-2  relative path to the second config.\n  -s, --settings   relative path to settings yaml.\n  -f, --filters    relative path to filters list yaml.\n  -c, --cures      relative path to cures list yaml.\n  -g, --no-color   disable color.\n```\n\n## cli example\n```bash\nsic2dc -c1 intended/sw1.cfg -c2 oper/sw1.cfg -s sic2dc/settings_arista_dcs.yml\n```\n\n```diff        \ninterface Port-Channel1\n-   no shutdown\ninterface Ethernet1\n+   no switchport\n+ snmp-server engineID local 123\n+ system l1\n+   unsupported speed action error\n+   unsupported error-correction action error\n+ interface Ethernet3\n+   shutdown\n+   no switchport\n- errdisable recovery interval 300\n- router bfd\n-   multihop interval 300 min-rx 300 multiplier 3\ninterface Port-Channel2\n-   no shutdown\nrouter bgp 66666\n-   bgp default ipv4-unicast\n```\n\nThe following options are required: c1, c2, settings. If no filters or cures are passed the configs are transformed and compared as they are. Filters and cures are yaml files with lists at the top level. Settings is a yaml with a dict.\n\n## python\n```python\nfrom sic2dc import sic2dc\nf1 = 'path_to_c1'\nf2 = 'path_to_c2'\nsettings = {\n    'indent_char': ' ',\n    'indent': 3,\n    'comments': ['^\\s*[\\!\\#].*?$', '^\\s*$'],\n}\n\nfilters = []\ncures = []\n\nresult = sic2dc(f1, f2, settings, filters=filters, cures=cures, color=True)\n\nresult['diff_dict']\nresult['diff_lines']\n```        \n## ansible filter\n```python\n\"\"\"filter_plugins file\"\"\"\nfrom sic2dc import sic2dc\nclass FilterModule(object):\n    def filters(self):\n        return {'sic2dc': sic2dc}\n```\n```yaml\n# playbook\n# settings, filters and cures can be set as ansible vars of a host.\n- set_fact:\n    cfg_diff: \"{{ f1 | sic2dc(f2, settings, filters, cures, False) }}\"\n- debug:\n    msg: \"{{ cfg_diff['diff_lines'] | join('\\n') }}\"\n  when: cfg_diff['diff_dict']\n- fail:\n  when: cfg_diff['diff_dict']\n```\n\n# Concepts\n## dicts\nWhen a config is transformed into dict the lines of the config are trimmed and become dict keys.\n```python\nrouter bgp 1234    # spaces in the end of line\n  router-id 1234\n# transforms into\n{'router bgp 1234': {'router-id 1234': {}}}\n```\n\n## path\nSince we compare nested dicts **path** is used to define config parts of interest. Path is a list of regex patterns.\n<br>path examples\n```python\npath = ['interface [Ee]thernet \\S+'] # all ethernet interfaces\npath = ['router bgp \\d+', 'address-family .*'] # all address-families in 'router bgp' section. \n```\n## whens\nWhen applying filters **whens** are used to select more specific sections. Imagine we want to select all unused interfaces that exist in operstate and do not exist in desired state. So they should be 'shutdown' and they should be absent in destination.\n\n```yaml\npath: [^interface (Ethernet|Management).*]\nwhen:\n  - has_children: ['^shutdown$']\n  - absent_in_destination: True\n```\n## examples\nSee [examples](https://github.com/alexonishchenko/sic2dc/tree/main/sic2dc/example) for filters/cures/settings examples\n\n# Settings\nSettings configure the following parameters (example for b4com switches).\n\n```yaml\n# enable/disable deleting of command/no command from both configs\nignore_cmd_nocmd: True\n\n# indent char\nindent_char: ' '\n\n# number of indent_chars on single indentation level\nindent: 1\n\n# list of patterns for comment lines. they will be deleted from both configs.\ncomments:\n  - '^\\s*[\\!\\#].*?$'\n  - '^\\s*$'\n  - '^\\s*exit\\s*$'\n  - '^end\\s*$'\n```\n\n\n\n# Filters\nFilters mostly copy, delete or change sections in c1 and c2. A filter is defined by the following fileds:\n - **action** - str value e.g cp21, cp12, upd1, upd2, del1, del2\n - **when** - list of when conditions ('has_children', 'doesnt_have_chidren', 'absent_in_destination')\n - **path** - path to set config parts of interest.\n - **data** - dict of data used by filter (in udpate filters)\n\n\n### cp21. Copy from c2 to c1\n```yaml\n# arista.desiredstate: copy unused interfaces from operstate\n- action: cp21\n  path: [^interface (Ethernet|Management).*]\n  when:\n    - has_children: ['^shutdown$']\n    - absent_in_destination: True\n```\n### cp12. Copy from c1 to c2\nDoes the opposite.\n\n### upd2. Update c2 with data\n```yaml\n# arista.operstate: add swprt mode access if swprt access vlan\n#   in operstate config this is hidden\n- action: upd2\n  path: [^interface Eth.*]\n  data:\n    switchport mode access: {}\n  when:\n    - has_children: [switchport access vlan.*]\n```\n### upd1. Update c1 with data\nSame as upd2\n\n### del1. Delete section in c1\n```yaml\n# arista.desiredstate: delete errdisable default value\n- action: del1\n  path: [^errdisable recovery interval 300]\n```\n\n### del2. Delete section in c2\n```yaml\n# arista.operstate: delete snmp engine id\n- action: del2\n  path: ['snmp-server engineID .*']\n```\n\n\n# Cures\nCures are defined by **action** and its **kwargs**. They are applied to configs prior to dict transformation.\nCurrently the only cure supported is \"enter_exit\". It adds indentation by pattern.\n\n## enter_exit\nFind 'enter' pattern, add a single level of indentation to all following lines until 'exit' pattern is met.\n\n```yaml\n- action: enter_exit\n  kwargs:\n    enter_exits:\n      - enter: ' address-family \\S+\\s.*$'\n        exit: ' exit-address-family$'\n```\n\nThe cure above transform example\n\n        router bgp 1234\n         address-family l2vpn evpn\n         neighbor 10.10.10.7 activate\n         neighbor 10.10.10.8 activate\n         exit-address-family\n        ->\n        router bgp 1234\n         address-family l2vpn evpn\n          neighbor 10.10.10.7 activate\n          neighbor 10.10.10.8 activate\n          exit-address-family\n\n# Api notes\n## sic2dc\n\n```python\ndef sic2dc(\n        f1: str,\n        f2: str,\n        settings: dict,\n        filters: list[dict] | None = None,\n        cures: list[dict] | None = None,\n        color: bool = False) -> dict:\n    \"\"\"\n    Creates ConfigCompareBase object and compares f1 and f2.\n    Returns ConfigCompareBase.diff_dict and ConfigCompareBase.dump() lines as dict\n    Returns dict:\n        'diff_dict': dict\n        'diff_lines': str\n    \"\"\"\n```\n\n## ConfigCompareBase\n\n```python\nclass ConfigCompareBase(...):\n    def __init__(self, f1: str, f2: str, settings: CfgCmprSettings,\n                 filters: list[dict] = None, cures: list[dict] = None):\n        \"\"\"\n        1. Create cc object: read files, apply cures and create d1 and d2. \n        2. Apply filters to dicts\n        3. Run comparison\n        \"\"\"\n\ncc = ConfigCompareBase(...)\n\n# uncured c1 and c2:\ncc.c1_uncured\ncc.c2_uncured\n\n# cured c1 and c2\ncc.c1\ncc.c2\n\n# unfiltered d1 and d2\ncc.d1_unfiltered\ncc.d2_unfiltered\n\n# filtered d1 and d2\ncc.d1\ncc.d2\n\n# dump\nbw_diff_text = cc.dump(quiet=True, color=False)\ncolor_diff_text = cc.dump(quiet=True, color=True)\n\n# printout color diff\ncc.dump(color=True)\n```\n",
    "bugtrack_url": null,
    "license": "MIT License  Copyright (c) 2024 alexonishchenko  Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \"Software\"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:  The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.  THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ",
    "summary": "Simple indented config to dict compare.",
    "version": "0.0.5",
    "project_urls": {
        "homepage": "https://github.com/alexonishchenko/sic2dc",
        "repository": "https://github.com/alexonishchenko/sic2dc"
    },
    "split_keywords": [
        "configuration",
        " difference",
        " diff",
        " compare",
        " config-compare",
        " network",
        " network-config"
    ],
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "c69a3dd9e823116b7a026f20493024b7f57797f1c4f82f0017df0383b0dbb141",
                "md5": "df76f741e74a7098847b52db57ee4d4f",
                "sha256": "e5d9899a58b2a875e160c51d1971fc3ef147ddaeb10c182bc7ee51b762323577"
            },
            "downloads": -1,
            "filename": "sic2dc-0.0.5-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "df76f741e74a7098847b52db57ee4d4f",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": ">=3.11",
            "size": 21586,
            "upload_time": "2024-11-18T09:52:32",
            "upload_time_iso_8601": "2024-11-18T09:52:32.045278Z",
            "url": "https://files.pythonhosted.org/packages/c6/9a/3dd9e823116b7a026f20493024b7f57797f1c4f82f0017df0383b0dbb141/sic2dc-0.0.5-py3-none-any.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "d7f797f02a7689b3804b5a2dbdc912f9f65c0af09091b74567ed7eaf79cab6e1",
                "md5": "74d088a0a192e8d02b6a7e15b1257d2d",
                "sha256": "d7597b8787338b7993525b264ce0b7bfe4b77332671c3c073a852c3b50495217"
            },
            "downloads": -1,
            "filename": "sic2dc-0.0.5.tar.gz",
            "has_sig": false,
            "md5_digest": "74d088a0a192e8d02b6a7e15b1257d2d",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": ">=3.11",
            "size": 18540,
            "upload_time": "2024-11-18T09:52:34",
            "upload_time_iso_8601": "2024-11-18T09:52:34.141142Z",
            "url": "https://files.pythonhosted.org/packages/d7/f7/97f02a7689b3804b5a2dbdc912f9f65c0af09091b74567ed7eaf79cab6e1/sic2dc-0.0.5.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2024-11-18 09:52:34",
    "github": true,
    "gitlab": false,
    "bitbucket": false,
    "codeberg": false,
    "github_user": "alexonishchenko",
    "github_project": "sic2dc",
    "travis_ci": false,
    "coveralls": false,
    "github_actions": false,
    "requirements": [
        {
            "name": "deepdiff",
            "specs": []
        },
        {
            "name": "pydantic",
            "specs": []
        },
        {
            "name": "pydantic_core",
            "specs": []
        },
        {
            "name": "ruamel.yaml",
            "specs": []
        },
        {
            "name": "ruamel.yaml.clib",
            "specs": []
        }
    ],
    "lcname": "sic2dc"
}