| Name | logstash-pipeline-parser JSON |
| Version |
0.0.2
JSON |
| download |
| home_page | |
| Summary | Parsing expression grammar and Abstract syntax tree for Logstash pipeline syntax. |
| upload_time | 2023-12-18 14:25:30 |
| maintainer | |
| docs_url | None |
| author | |
| requires_python | <3.12.0,>=3.11.0 |
| license | MIT |
| keywords |
logstash
config
pipeline
peg
ast
parser
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
## About The Project
### How Logstash Pipeline Works
> The Logstash event processing pipeline has three stages: inputs → filters → outputs.\
> Inputs generate events, filters modify them, and outputs ship them elsewhere.\
> Inputs and outputs support codecs that enable you to encode or decode the data as it enters or exits the pipeline without having to use a separate filter.
The pipeline configuration file is a custom format developed by the Logstash folks using [Treetop](https://cjheath.github.io/treetop/syntactic_recognition.html).
The grammar itself is described in the source file [grammar.treetop](https://github.com/elastic/logstash/tree/v8.11.1/logstash-core/lib/logstash/config/grammar.treetop) and compiled using Treetop into the custom [grammar.rb](https://github.com/elastic/logstash/blob/v8.11.1/logstash-core/lib/logstash/config/grammar.rb) parser.
That parser is then used to set up the pipeline from the Logstash configuration.
See also:
- [How Logstash Works](https://www.elastic.co/guide/en/logstash/current/pipeline.html)
- [Creating a Logstash pipeline](https://www.elastic.co/guide/en/logstash/current/configuration.html)
## Documentation
The latest documentation is at https://tomaskoutek.github.io/logstash-pipeline-parser/ and contains description of all methods, classes with individual examples and their results.
Or you can display description directly with the help function.
For example:
```python
from logstash_pipeline_parser import Pipeline
help(Pipeline.parse)
```
## Getting Started
### Installing
Install from [pip](https://pypi.org/project/logstash-pipeline-parser/):
```
pip install logstash-pipeline-parser
```
### Dependencies
- [pyparsing](https://github.com/pyparsing/pyparsing) for creating [PEG](https://en.wikipedia.org/wiki/Parsing_expression_grammar) parser
## Quick Example
For full documentation with examples please see [documentation](https://tomaskoutek.github.io/logstash-pipeline-parser/).
Let's try pipeline with two input plugins:
```python
from logstash_pipeline_parser import Pipeline
data = r"""
input {
beats {
host => "0.0.0.0"
port => 5044
client_inactivity_timeout => 3600
include_codec_tag => true
enrich => [source_metadata, ssl_peer_metadata]
ssl => true
ssl_key => "/some/path/my.key"
id => "input_beats"
}
udp {
port => 5045
host => "0.0.0.0"
}
}
"""
pipeline = Pipeline(data)
ast = pipeline.parse()
```
will produce [Abstract syntax tree](https://en.wikipedia.org/wiki/Abstract_syntax_tree):
```python
from ipaddress import IPv4Address
from pathlib import Path
[
["input", [
["beats", [
["host", [IPv4Address("0.0.0.0")]],
["port", [5044]],
["client_inactivity_timeout", [3600]],
["include_codec_tag", [True]],
["enrich", [
["source_metadata", "ssl_peer_metadata"]
]],
["ssl", [True]],
["ssl_key", [Path("/some/path/my.key")]],
["id", ["input_beats"]]
]],
["udp", [
["port", [5045]],
["host", [IPv4Address("0.0.0.0")]]
]]
]]
]
```
Of course, it is possible to parse all kinds of plugins, conditions and data types.
You can also search in the pipeline. The searched key can also contain the wildcard `*`, for example **"output.\*.hosts"** will return
(if the pipeline definition contains them):
- `("output.elasticsearch.hosts", [["127.0.0.1:9200","127.0.0.2:9200"]])`
- `("output.logstash.hosts", ["127.0.0.1:9801"])`
```python
results = pipeline.search("input.*.port")
print(list(results))
# [
# ("input.beats.port", [5044]),
# ("input.udp.port", [5045])
# ]
```
The `search` method returns a generator, so we can easily iterate:
```python
for key, value in pipeline.search("*.port"):
print(f"key: {key}, value: {value[0]}")
# key: input.beats.port, value: 5044
# key: input.udp.sub.port, value: 5045
```
The return value can be any element from the tree (integer, string, field, plugin,...):
```python
results = pipeline.search("input.beats.enrich")
print(list(results))
# [
# ("input.beats.enrich", [["source_metadata", "ssl_peer_metadata"]])
# ]
```
## License
Distributed under the MIT License. See LICENSE for more information.
Raw data
{
"_id": null,
"home_page": "",
"name": "logstash-pipeline-parser",
"maintainer": "",
"docs_url": null,
"requires_python": "<3.12.0,>=3.11.0",
"maintainer_email": "",
"keywords": "logstash,config,pipeline,peg,ast,parser",
"author": "",
"author_email": "Tomas Koutek <66636b6f6666@gmail.com>",
"download_url": "https://files.pythonhosted.org/packages/b5/ab/5ae19cbe10f8e385a03d06bdfd1bd498f9efb94b8b52907c86dedc4c5e83/logstash-pipeline-parser-0.0.2.tar.gz",
"platform": null,
"description": "## About The Project\r\n\r\n### How Logstash Pipeline Works\r\n> The Logstash event processing pipeline has three stages: inputs \u2192 filters \u2192 outputs.\\\r\n> Inputs generate events, filters modify them, and outputs ship them elsewhere.\\\r\n> Inputs and outputs support codecs that enable you to encode or decode the data as it enters or exits the pipeline without having to use a separate filter.\r\n\r\nThe pipeline configuration file is a custom format developed by the Logstash folks using [Treetop](https://cjheath.github.io/treetop/syntactic_recognition.html).\r\nThe grammar itself is described in the source file [grammar.treetop](https://github.com/elastic/logstash/tree/v8.11.1/logstash-core/lib/logstash/config/grammar.treetop) and compiled using Treetop into the custom [grammar.rb](https://github.com/elastic/logstash/blob/v8.11.1/logstash-core/lib/logstash/config/grammar.rb) parser.\r\nThat parser is then used to set up the pipeline from the Logstash configuration.\r\n\r\nSee also:\r\n- [How Logstash Works](https://www.elastic.co/guide/en/logstash/current/pipeline.html)\r\n- [Creating a Logstash pipeline](https://www.elastic.co/guide/en/logstash/current/configuration.html)\r\n\r\n\r\n## Documentation\r\n\r\nThe latest documentation is at https://tomaskoutek.github.io/logstash-pipeline-parser/ and contains description of all methods, classes with individual examples and their results. \r\nOr you can display description directly with the help function.\r\n\r\nFor example:\r\n\r\n```python\r\nfrom logstash_pipeline_parser import Pipeline\r\n\r\nhelp(Pipeline.parse)\r\n```\r\n\r\n## Getting Started\r\n\r\n![graph](./doc/graphviz.png \"Plugins tree\")\r\n\r\n### Installing\r\n\r\nInstall from [pip](https://pypi.org/project/logstash-pipeline-parser/):\r\n\r\n```\r\npip install logstash-pipeline-parser\r\n```\r\n\r\n### Dependencies\r\n- [pyparsing](https://github.com/pyparsing/pyparsing) for creating [PEG](https://en.wikipedia.org/wiki/Parsing_expression_grammar) parser\r\n\r\n## Quick Example\r\n\r\nFor full documentation with examples please see [documentation](https://tomaskoutek.github.io/logstash-pipeline-parser/).\r\n\r\nLet's try pipeline with two input plugins: \r\n\r\n```python\r\nfrom logstash_pipeline_parser import Pipeline\r\n\r\ndata = r\"\"\"\r\n input {\r\n beats {\r\n host => \"0.0.0.0\"\r\n port => 5044\r\n client_inactivity_timeout => 3600\r\n include_codec_tag => true\r\n enrich => [source_metadata, ssl_peer_metadata]\r\n ssl => true\r\n ssl_key => \"/some/path/my.key\"\r\n id => \"input_beats\"\r\n }\r\n \r\n udp {\r\n port => 5045\r\n host => \"0.0.0.0\"\r\n }\r\n }\r\n\"\"\"\r\n\r\npipeline = Pipeline(data)\r\nast = pipeline.parse()\r\n```\r\n\r\nwill produce [Abstract syntax tree](https://en.wikipedia.org/wiki/Abstract_syntax_tree):\r\n\r\n```python\r\nfrom ipaddress import IPv4Address\r\nfrom pathlib import Path\r\n\r\n[\r\n [\"input\", [\r\n [\"beats\", [\r\n [\"host\", [IPv4Address(\"0.0.0.0\")]], \r\n [\"port\", [5044]], \r\n [\"client_inactivity_timeout\", [3600]], \r\n [\"include_codec_tag\", [True]], \r\n [\"enrich\", [\r\n [\"source_metadata\", \"ssl_peer_metadata\"]\r\n ]], \r\n [\"ssl\", [True]], \r\n [\"ssl_key\", [Path(\"/some/path/my.key\")]], \r\n [\"id\", [\"input_beats\"]]\r\n ]], \r\n [\"udp\", [\r\n [\"port\", [5045]], \r\n [\"host\", [IPv4Address(\"0.0.0.0\")]]\r\n ]]\r\n ]]\r\n]\r\n\r\n```\r\nOf course, it is possible to parse all kinds of plugins, conditions and data types.\r\nYou can also search in the pipeline. The searched key can also contain the wildcard `*`, for example **\"output.\\*.hosts\"** will return\r\n(if the pipeline definition contains them):\r\n\r\n- `(\"output.elasticsearch.hosts\", [[\"127.0.0.1:9200\",\"127.0.0.2:9200\"]])`\r\n- `(\"output.logstash.hosts\", [\"127.0.0.1:9801\"])`\r\n\r\n\r\n```python\r\nresults = pipeline.search(\"input.*.port\")\r\n\r\nprint(list(results))\r\n# [\r\n# (\"input.beats.port\", [5044]), \r\n# (\"input.udp.port\", [5045])\r\n# ]\r\n```\r\n\r\nThe `search` method returns a generator, so we can easily iterate:\r\n\r\n```python\r\nfor key, value in pipeline.search(\"*.port\"):\r\n print(f\"key: {key}, value: {value[0]}\")\r\n\r\n# key: input.beats.port, value: 5044\r\n# key: input.udp.sub.port, value: 5045\r\n```\r\n\r\nThe return value can be any element from the tree (integer, string, field, plugin,...):\r\n\r\n```python\r\nresults = pipeline.search(\"input.beats.enrich\")\r\n\r\nprint(list(results))\r\n# [\r\n# (\"input.beats.enrich\", [[\"source_metadata\", \"ssl_peer_metadata\"]])\r\n# ]\r\n```\r\n\r\n\r\n## License\r\n\r\nDistributed under the MIT License. See LICENSE for more information.\r\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Parsing expression grammar and Abstract syntax tree for Logstash pipeline syntax.",
"version": "0.0.2",
"project_urls": {
"Documentation": "https://tomaskoutek.github.io/logstash-pipeline-parser/",
"Source": "https://github.com/TomasKoutek/logstash-pipeline-parser",
"Tracker": "https://github.com/TomasKoutek/logstash-pipeline-parser/issues"
},
"split_keywords": [
"logstash",
"config",
"pipeline",
"peg",
"ast",
"parser"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "1f501fbadcee0b0bf546cf9c4e98bc20d23eb476729bfa956dab415eb53ed4c9",
"md5": "73b51b61f2e902d4601961b89a7405c0",
"sha256": "40e77d8a8505e401186d2f857a1cbc8080f3c578c17128db72fb59a717661965"
},
"downloads": -1,
"filename": "logstash_pipeline_parser-0.0.2-py3-none-any.whl",
"has_sig": false,
"md5_digest": "73b51b61f2e902d4601961b89a7405c0",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<3.12.0,>=3.11.0",
"size": 10822,
"upload_time": "2023-12-18T14:25:20",
"upload_time_iso_8601": "2023-12-18T14:25:20.345033Z",
"url": "https://files.pythonhosted.org/packages/1f/50/1fbadcee0b0bf546cf9c4e98bc20d23eb476729bfa956dab415eb53ed4c9/logstash_pipeline_parser-0.0.2-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "b5ab5ae19cbe10f8e385a03d06bdfd1bd498f9efb94b8b52907c86dedc4c5e83",
"md5": "dc2f0a69a1205c1d15f2b21509dd7e63",
"sha256": "e07d21accd782e50eb83f248821a2a150d99f1a3211524a41153e07ecbab1bbc"
},
"downloads": -1,
"filename": "logstash-pipeline-parser-0.0.2.tar.gz",
"has_sig": false,
"md5_digest": "dc2f0a69a1205c1d15f2b21509dd7e63",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<3.12.0,>=3.11.0",
"size": 17646,
"upload_time": "2023-12-18T14:25:30",
"upload_time_iso_8601": "2023-12-18T14:25:30.774841Z",
"url": "https://files.pythonhosted.org/packages/b5/ab/5ae19cbe10f8e385a03d06bdfd1bd498f9efb94b8b52907c86dedc4c5e83/logstash-pipeline-parser-0.0.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-12-18 14:25:30",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "TomasKoutek",
"github_project": "logstash-pipeline-parser",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [],
"lcname": "logstash-pipeline-parser"
}
