logstash-pipeline-parser


Namelogstash-pipeline-parser JSON
Version 0.0.2 PyPI version JSON
download
home_page
SummaryParsing expression grammar and Abstract syntax tree for Logstash pipeline syntax.
upload_time2023-12-18 14:25:30
maintainer
docs_urlNone
author
requires_python<3.12.0,>=3.11.0
licenseMIT
keywords logstash config pipeline peg ast parser
VCS
bugtrack_url
requirements No requirements were recorded.
Travis-CI No Travis.
coveralls test coverage No coveralls.
            ## About The Project

### How Logstash Pipeline Works
> The Logstash event processing pipeline has three stages: inputs → filters → outputs.\
> Inputs generate events, filters modify them, and outputs ship them elsewhere.\
> Inputs and outputs support codecs that enable you to encode or decode the data as it enters or exits the pipeline without having to use a separate filter.

The pipeline configuration file is a custom format developed by the Logstash folks using [Treetop](https://cjheath.github.io/treetop/syntactic_recognition.html).
The grammar itself is described in the source file [grammar.treetop](https://github.com/elastic/logstash/tree/v8.11.1/logstash-core/lib/logstash/config/grammar.treetop) and compiled using Treetop into the custom [grammar.rb](https://github.com/elastic/logstash/blob/v8.11.1/logstash-core/lib/logstash/config/grammar.rb) parser.
That parser is then used to set up the pipeline from the Logstash configuration.

See also:
- [How Logstash Works](https://www.elastic.co/guide/en/logstash/current/pipeline.html)
- [Creating a Logstash pipeline](https://www.elastic.co/guide/en/logstash/current/configuration.html)


## Documentation

The latest documentation is at https://tomaskoutek.github.io/logstash-pipeline-parser/ and contains description of all methods, classes with individual examples and their results. 
Or you can display description directly with the help function.

For example:

```python
from logstash_pipeline_parser import Pipeline

help(Pipeline.parse)
```

## Getting Started

![graph](./doc/graphviz.png "Plugins tree")

### Installing

Install from [pip](https://pypi.org/project/logstash-pipeline-parser/):

```
pip install logstash-pipeline-parser
```

### Dependencies
- [pyparsing](https://github.com/pyparsing/pyparsing) for creating [PEG](https://en.wikipedia.org/wiki/Parsing_expression_grammar) parser

## Quick Example

For full documentation with examples please see [documentation](https://tomaskoutek.github.io/logstash-pipeline-parser/).

Let's try pipeline with two input plugins: 

```python
from logstash_pipeline_parser import Pipeline

data = r"""
       input {
         beats {
           host => "0.0.0.0"
           port => 5044
           client_inactivity_timeout => 3600
           include_codec_tag => true
           enrich => [source_metadata, ssl_peer_metadata]
           ssl => true
           ssl_key => "/some/path/my.key"
           id => "input_beats"
         }
         
         udp {
           port => 5045
           host => "0.0.0.0"
         }
       }
"""

pipeline = Pipeline(data)
ast = pipeline.parse()
```

will produce [Abstract syntax tree](https://en.wikipedia.org/wiki/Abstract_syntax_tree):

```python
from ipaddress import IPv4Address
from pathlib import Path

[
    ["input", [
        ["beats", [
            ["host", [IPv4Address("0.0.0.0")]], 
            ["port", [5044]], 
            ["client_inactivity_timeout", [3600]], 
            ["include_codec_tag", [True]], 
            ["enrich", [
                ["source_metadata", "ssl_peer_metadata"]
            ]], 
            ["ssl", [True]], 
            ["ssl_key", [Path("/some/path/my.key")]], 
            ["id", ["input_beats"]]
        ]], 
        ["udp", [
            ["port", [5045]], 
            ["host", [IPv4Address("0.0.0.0")]]
        ]]
    ]]
]

```
Of course, it is possible to parse all kinds of plugins, conditions and data types.
You can also search in the pipeline. The searched key can also contain the wildcard `*`, for example **"output.\*.hosts"** will return
(if the pipeline definition contains them):

- `("output.elasticsearch.hosts", [["127.0.0.1:9200","127.0.0.2:9200"]])`
- `("output.logstash.hosts", ["127.0.0.1:9801"])`


```python
results = pipeline.search("input.*.port")

print(list(results))
# [
#   ("input.beats.port", [5044]), 
#   ("input.udp.port", [5045])
# ]
```

The `search` method returns a generator, so we can easily iterate:

```python
for key, value in pipeline.search("*.port"):
    print(f"key: {key}, value: {value[0]}")

# key: input.beats.port, value: 5044
# key: input.udp.sub.port, value: 5045
```

The return value can be any element from the tree (integer, string, field, plugin,...):

```python
results = pipeline.search("input.beats.enrich")

print(list(results))
# [
#   ("input.beats.enrich", [["source_metadata", "ssl_peer_metadata"]])
# ]
```


## License

Distributed under the MIT License. See LICENSE for more information.

            

Raw data

            {
    "_id": null,
    "home_page": "",
    "name": "logstash-pipeline-parser",
    "maintainer": "",
    "docs_url": null,
    "requires_python": "<3.12.0,>=3.11.0",
    "maintainer_email": "",
    "keywords": "logstash,config,pipeline,peg,ast,parser",
    "author": "",
    "author_email": "Tomas Koutek <66636b6f6666@gmail.com>",
    "download_url": "https://files.pythonhosted.org/packages/b5/ab/5ae19cbe10f8e385a03d06bdfd1bd498f9efb94b8b52907c86dedc4c5e83/logstash-pipeline-parser-0.0.2.tar.gz",
    "platform": null,
    "description": "## About The Project\r\n\r\n### How Logstash Pipeline Works\r\n> The Logstash event processing pipeline has three stages: inputs \u2192 filters \u2192 outputs.\\\r\n> Inputs generate events, filters modify them, and outputs ship them elsewhere.\\\r\n> Inputs and outputs support codecs that enable you to encode or decode the data as it enters or exits the pipeline without having to use a separate filter.\r\n\r\nThe pipeline configuration file is a custom format developed by the Logstash folks using [Treetop](https://cjheath.github.io/treetop/syntactic_recognition.html).\r\nThe grammar itself is described in the source file [grammar.treetop](https://github.com/elastic/logstash/tree/v8.11.1/logstash-core/lib/logstash/config/grammar.treetop) and compiled using Treetop into the custom [grammar.rb](https://github.com/elastic/logstash/blob/v8.11.1/logstash-core/lib/logstash/config/grammar.rb) parser.\r\nThat parser is then used to set up the pipeline from the Logstash configuration.\r\n\r\nSee also:\r\n- [How Logstash Works](https://www.elastic.co/guide/en/logstash/current/pipeline.html)\r\n- [Creating a Logstash pipeline](https://www.elastic.co/guide/en/logstash/current/configuration.html)\r\n\r\n\r\n## Documentation\r\n\r\nThe latest documentation is at https://tomaskoutek.github.io/logstash-pipeline-parser/ and contains description of all methods, classes with individual examples and their results. \r\nOr you can display description directly with the help function.\r\n\r\nFor example:\r\n\r\n```python\r\nfrom logstash_pipeline_parser import Pipeline\r\n\r\nhelp(Pipeline.parse)\r\n```\r\n\r\n## Getting Started\r\n\r\n![graph](./doc/graphviz.png \"Plugins tree\")\r\n\r\n### Installing\r\n\r\nInstall from [pip](https://pypi.org/project/logstash-pipeline-parser/):\r\n\r\n```\r\npip install logstash-pipeline-parser\r\n```\r\n\r\n### Dependencies\r\n- [pyparsing](https://github.com/pyparsing/pyparsing) for creating [PEG](https://en.wikipedia.org/wiki/Parsing_expression_grammar) parser\r\n\r\n## Quick Example\r\n\r\nFor full documentation with examples please see [documentation](https://tomaskoutek.github.io/logstash-pipeline-parser/).\r\n\r\nLet's try pipeline with two input plugins: \r\n\r\n```python\r\nfrom logstash_pipeline_parser import Pipeline\r\n\r\ndata = r\"\"\"\r\n       input {\r\n         beats {\r\n           host => \"0.0.0.0\"\r\n           port => 5044\r\n           client_inactivity_timeout => 3600\r\n           include_codec_tag => true\r\n           enrich => [source_metadata, ssl_peer_metadata]\r\n           ssl => true\r\n           ssl_key => \"/some/path/my.key\"\r\n           id => \"input_beats\"\r\n         }\r\n         \r\n         udp {\r\n           port => 5045\r\n           host => \"0.0.0.0\"\r\n         }\r\n       }\r\n\"\"\"\r\n\r\npipeline = Pipeline(data)\r\nast = pipeline.parse()\r\n```\r\n\r\nwill produce [Abstract syntax tree](https://en.wikipedia.org/wiki/Abstract_syntax_tree):\r\n\r\n```python\r\nfrom ipaddress import IPv4Address\r\nfrom pathlib import Path\r\n\r\n[\r\n    [\"input\", [\r\n        [\"beats\", [\r\n            [\"host\", [IPv4Address(\"0.0.0.0\")]], \r\n            [\"port\", [5044]], \r\n            [\"client_inactivity_timeout\", [3600]], \r\n            [\"include_codec_tag\", [True]], \r\n            [\"enrich\", [\r\n                [\"source_metadata\", \"ssl_peer_metadata\"]\r\n            ]], \r\n            [\"ssl\", [True]], \r\n            [\"ssl_key\", [Path(\"/some/path/my.key\")]], \r\n            [\"id\", [\"input_beats\"]]\r\n        ]], \r\n        [\"udp\", [\r\n            [\"port\", [5045]], \r\n            [\"host\", [IPv4Address(\"0.0.0.0\")]]\r\n        ]]\r\n    ]]\r\n]\r\n\r\n```\r\nOf course, it is possible to parse all kinds of plugins, conditions and data types.\r\nYou can also search in the pipeline. The searched key can also contain the wildcard `*`, for example **\"output.\\*.hosts\"** will return\r\n(if the pipeline definition contains them):\r\n\r\n- `(\"output.elasticsearch.hosts\", [[\"127.0.0.1:9200\",\"127.0.0.2:9200\"]])`\r\n- `(\"output.logstash.hosts\", [\"127.0.0.1:9801\"])`\r\n\r\n\r\n```python\r\nresults = pipeline.search(\"input.*.port\")\r\n\r\nprint(list(results))\r\n# [\r\n#   (\"input.beats.port\", [5044]), \r\n#   (\"input.udp.port\", [5045])\r\n# ]\r\n```\r\n\r\nThe `search` method returns a generator, so we can easily iterate:\r\n\r\n```python\r\nfor key, value in pipeline.search(\"*.port\"):\r\n    print(f\"key: {key}, value: {value[0]}\")\r\n\r\n# key: input.beats.port, value: 5044\r\n# key: input.udp.sub.port, value: 5045\r\n```\r\n\r\nThe return value can be any element from the tree (integer, string, field, plugin,...):\r\n\r\n```python\r\nresults = pipeline.search(\"input.beats.enrich\")\r\n\r\nprint(list(results))\r\n# [\r\n#   (\"input.beats.enrich\", [[\"source_metadata\", \"ssl_peer_metadata\"]])\r\n# ]\r\n```\r\n\r\n\r\n## License\r\n\r\nDistributed under the MIT License. See LICENSE for more information.\r\n",
    "bugtrack_url": null,
    "license": "MIT",
    "summary": "Parsing expression grammar and Abstract syntax tree for Logstash pipeline syntax.",
    "version": "0.0.2",
    "project_urls": {
        "Documentation": "https://tomaskoutek.github.io/logstash-pipeline-parser/",
        "Source": "https://github.com/TomasKoutek/logstash-pipeline-parser",
        "Tracker": "https://github.com/TomasKoutek/logstash-pipeline-parser/issues"
    },
    "split_keywords": [
        "logstash",
        "config",
        "pipeline",
        "peg",
        "ast",
        "parser"
    ],
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "1f501fbadcee0b0bf546cf9c4e98bc20d23eb476729bfa956dab415eb53ed4c9",
                "md5": "73b51b61f2e902d4601961b89a7405c0",
                "sha256": "40e77d8a8505e401186d2f857a1cbc8080f3c578c17128db72fb59a717661965"
            },
            "downloads": -1,
            "filename": "logstash_pipeline_parser-0.0.2-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "73b51b61f2e902d4601961b89a7405c0",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": "<3.12.0,>=3.11.0",
            "size": 10822,
            "upload_time": "2023-12-18T14:25:20",
            "upload_time_iso_8601": "2023-12-18T14:25:20.345033Z",
            "url": "https://files.pythonhosted.org/packages/1f/50/1fbadcee0b0bf546cf9c4e98bc20d23eb476729bfa956dab415eb53ed4c9/logstash_pipeline_parser-0.0.2-py3-none-any.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "b5ab5ae19cbe10f8e385a03d06bdfd1bd498f9efb94b8b52907c86dedc4c5e83",
                "md5": "dc2f0a69a1205c1d15f2b21509dd7e63",
                "sha256": "e07d21accd782e50eb83f248821a2a150d99f1a3211524a41153e07ecbab1bbc"
            },
            "downloads": -1,
            "filename": "logstash-pipeline-parser-0.0.2.tar.gz",
            "has_sig": false,
            "md5_digest": "dc2f0a69a1205c1d15f2b21509dd7e63",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": "<3.12.0,>=3.11.0",
            "size": 17646,
            "upload_time": "2023-12-18T14:25:30",
            "upload_time_iso_8601": "2023-12-18T14:25:30.774841Z",
            "url": "https://files.pythonhosted.org/packages/b5/ab/5ae19cbe10f8e385a03d06bdfd1bd498f9efb94b8b52907c86dedc4c5e83/logstash-pipeline-parser-0.0.2.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2023-12-18 14:25:30",
    "github": true,
    "gitlab": false,
    "bitbucket": false,
    "codeberg": false,
    "github_user": "TomasKoutek",
    "github_project": "logstash-pipeline-parser",
    "travis_ci": false,
    "coveralls": false,
    "github_actions": true,
    "requirements": [],
    "lcname": "logstash-pipeline-parser"
}
        
Elapsed time: 0.18319s