| Name | splunk-data-sender JSON |
| Version |
0.5.0
JSON |
| download |
| home_page | None |
| Summary | A Python connector that sends your data to Splunk via HEC |
| upload_time | 2024-08-05 22:05:19 |
| maintainer | None |
| docs_url | None |
| author | None |
| requires_python | >=3.8 |
| license | None |
| keywords |
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# Splunk Data Sender
[](https://app.codacy.com/manual/andrea.salvatori92/splunk-data-sender?utm_source=github.com&utm_medium=referral&utm_content=Sonic0/splunk-data-sender&utm_campaign=Badge_Grade_Dashboard)
**Splunk Data Sender is a very simple and minimal Python package for sending logged events to an installation of Splunk Enterprise throw REST API**
*This logger requires the destination Splunk Enterprise server to have enabled and configured the [Splunk HTTP Event Collector](http://dev.splunk.com/view/event-collector/SP-CAAAE6M).*
## Installation
Pip:
pip install splunk-data-sender
Manual:
python3 setup.py install
## Usage
from splunk_data_sender import SplunkSender
Then create a SplunkSender instance with your preferred configuration. Now you can use the two methods **send_data()**
and **check_acks()** to, respectively, send logs and check messages acks(Requires useAck enabled in HEC).
Example:
```python
import logging
from splunk_data_sender import SplunkSender
splunk_conf = {
'endpoint': 'localhost',
'port': '8000',
'token': '851A5E58-4EF1-7291-F947-F614A76ACB21',
'index': 'main',
'channel': '16c70678-e516-44a9-854d-d139929e6869', # GUID
'api_version': '1.0',
# 'hostname': 'hostname', # manually set a hostname parameter, defaults to socket.gethostname()
# 'source': 'source', # manually set a source, defaults to the log record.pathname
# 'source_type': '_json', # manually set a source_type, defaults to 'generic_single_line'
'allow_overrides': True, # Whether to look for one of the Splunk built-in parameters(index, host, ecc)
'verify': False, # turn SSL verification on or off, defaults to True
# 'timeout': 60, # timeout for waiting on a 200 OK from Splunk server, defaults to 60s
# 'retry_count': 5, # Number of retry attempts on a failed/erroring connection, defaults to 5
# 'retry_backoff': 2.0, # Backoff factor, default options will retry for 1 min, defaults to 2.0
'enable_debug': True # turn on debug mode; prints module activity to stdout, defaults to False
}
splunk = SplunkSender(**splunk_conf)
is_alive = splunk.get_health()
logging.info(is_alive)
if not is_alive:
raise Exception("HEC not alive")
# The first payload is a quote to the italian theme song of Hello!Spank
txt_record = "Hello! Splunk resta con me, Hello! Splunk non te ne andare, Caro Splunk! gioca con me, siamo amici io e te."
json_record = { # this record will be parsed as normal text due to default "sourcetype" conf param
"source": "spacecraft Discovery 1",
"host": "HAL9000",
# "sourcetype": "_json", # source type without underscore to allow the override of this built-in parameter
"index": "main",
"event": {"message": "I am afraid I can't do that Dave.", "severity": "ERROR"},
"rack": "42",
"os": "Linux, obvious",
"arch": "x64"
}
payloads = [txt_record, json_record]
splunk_res = splunk.send_data(payloads)
logging.info(splunk_res)
ack_id = splunk_res.get('ackId')
splunk_ack_res = splunk.send_acks(ack_id)
logging.info(splunk_ack_res)
```
### Batching support
You may want to enable batching support, to avoid sending a API call to Splunk every time send\_data() is called. \
To enable the feature, use the `max_buf_size` parameter:
```python
from splunk_data_sender import SplunkSender
splunk_conf = {
'endpoint': 'localhost',
'port': '8000',
'token': '851A5E58-4EF1-7291-F947-F614A76ACB21',
'index': 'main',
'source_type': '_json',
# Enable internal buffering, up to 2000 events
'max_buf_size': 2000,
}
splunk = SplunkSender(**splunk_conf)
```
Then, simply call send\_data() in your loop. API calls will be performed when enough events are gathered. \
Call flush\_buffer() at the end of your code to send remaining events, or if you want to flush the buffer:
```python
for event in generate_event():
# event may be a dict, a string, or a list of events
# the event or list of events will be appended to the internal buffer
# If the buffer holds more than max_buf_size items (2000 in our example),
# then an API call will be made and the buffer will be reset.
# If such call is made, send_data returns its result.
# If no call is made, send_data() returns None
splunk.send_data(event)
# We finished processing our stuff, we must commit any remaining events to Splunk
splunk.flush_buffer()
```
## Configuration parameters notes
### "source_type"
If this param is sets to "_json" (and "allow_overrides" too), not built-in params will be added inside a **"fields"** key described below.
Refer to the [official Splunk documentation](https://docs.splunk.com/Documentation/Splunk/8.0.5/Data/Listofpretrainedsourcetypes)
for more information about source types.
### "allow_overrides"
If this param is set to "True", whether to look for one of the Splunk built-in parameters
(time, source, host, index) it will override the autocompleted parameters.<br>
For example, a json record with "time"=1486683865.000 will simulates a payload in the past for Splunk.
### Overriding parameters without json
By default, Splunk built-in parameters can only be overwritten from json records.
However, you may need to send non-json records while keeping this ability.
To that purpose, you may use the `event\_formatter` parameter. It takes a function in charge
of formatting events before sending them to Splunk.
Once set, you should call `send_data()` with events formatted as dict.
The library will process the events as if they were json objects, will override the parameters
accordingly, and will then call your function to reformat the resulting event.
A sample code follows:
```python
from splunk_data_sender import SplunkSender
def myEventFormatter(event):
# Transforms the event in any way you like
# event is of the same kind of what you sent to send_data()
# This sample function will return the event as XML, because why not
import xmltodict
return xmltodict.unparse({'event': event})
splunk_conf = {
'endpoint': 'localhost',
'port': '8000',
'token': '851A5E58-4EF1-7291-F947-F614A76ACB21',
'index': 'main',
'allow_overrides': True,
'source_type': 'my_source_type',
'event_formatter': myEventFormatter
}
splunk = SplunkSender(**splunk_conf)
# We can still send non-dict events. The formatter callback must be able to handle whatever
kind of data you use.
txt_record = "Hello! Splunk resta con me, Hello! Splunk non te ne andare, Caro Splunk! gioca con me, siamo amici io e te."
# Built-in parameters will be overridden.
# The resulting dict will be received by the callback function and transformed to XML
json_record = {
"source": "spacecraft Discovery 1",
"host": "HAL9000",
"index": "main",
"event": {"message": "I am afraid I can't do that Dave.", "severity": "ERROR"},
"rack": "42",
"os": "Linux, obvious",
"arch": "x64"
}
payloads = [txt_record, json_record]
splunk_res = splunk.send_data(payloads)
```
## Notes for JSON source type event
Refer to the [official Splunk documentation](https://docs.splunk.com/Documentation/Splunk/8.0.5/Data/IFXandHEC)
for more information about the use of JSON source type.
### Nested JSON inside the "event" property
In some cases, events have nested json which contains the custom fields to be indexed. In this case you have to set "sourcetype" = "_json".
### "fields"
Fields for indexing that do not occur in the event payload itself.<br>
You can use this parameter when you do not want particular fields to be included in the event data,
but you need additional metadata for indexing and searching.<br>
In the above example, "rack", "os" and "arch" will included inside "fields" key.<br>
Using this method is also typically faster than the nested JSON method.<br>
Be aware that you must send HEC requests containing the fields property to the _/collector/event_ endpoint. Otherwise, they will not be indexed.
## Retry Logic
This library uses the built-in retry logic from urllib3 (a retry counter and a backoff factor).
Should the defaults not be desireable, you can find more information about how to best configure these
settings in the [urllib3 documentation](https://github.com/kennethreitz/requests/blob/b2289cd2d5d21bd31cf4a818a4e0ff6951b2317a/requests/packages/urllib3/util/retry.py#L104).
## Contributing
Feel free to contribute an issue or pull request:
1. Check for existing issues and PRs
2. Fork the repo, and clone it locally
3. Create a new branch for your contribution
4. Push to your fork and submit a pull request
## License
This project is licensed under the terms of the [MIT license](http://opensource.org/licenses/MIT).
Raw data
{
"_id": null,
"home_page": null,
"name": "splunk-data-sender",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": null,
"author": null,
"author_email": "Andrea Salvatori <16443598+Sonic0@users.noreply.github.com>",
"download_url": "https://files.pythonhosted.org/packages/27/56/896ce2890ec804c1dd45312750afdf9899241c3f0a205e1b7b7444d406c4/splunk_data_sender-0.5.0.tar.gz",
"platform": null,
"description": "# Splunk Data Sender\n\n[](https://app.codacy.com/manual/andrea.salvatori92/splunk-data-sender?utm_source=github.com&utm_medium=referral&utm_content=Sonic0/splunk-data-sender&utm_campaign=Badge_Grade_Dashboard)\n\n**Splunk Data Sender is a very simple and minimal Python package for sending logged events to an installation of Splunk Enterprise throw REST API**\n\n*This logger requires the destination Splunk Enterprise server to have enabled and configured the [Splunk HTTP Event Collector](http://dev.splunk.com/view/event-collector/SP-CAAAE6M).*\n\n## Installation\nPip:\n\n pip install splunk-data-sender\n\n\nManual:\n\n python3 setup.py install\n\n## Usage\n\n from splunk_data_sender import SplunkSender\n\nThen create a SplunkSender instance with your preferred configuration. Now you can use the two methods **send_data()**\nand **check_acks()** to, respectively, send logs and check messages acks(Requires useAck enabled in HEC).\n\nExample:\n\n```python\nimport logging\nfrom splunk_data_sender import SplunkSender\n\n\nsplunk_conf = {\n 'endpoint': 'localhost',\n 'port': '8000',\n 'token': '851A5E58-4EF1-7291-F947-F614A76ACB21',\n 'index': 'main',\n 'channel': '16c70678-e516-44a9-854d-d139929e6869', # GUID\n 'api_version': '1.0',\n # 'hostname': 'hostname', # manually set a hostname parameter, defaults to socket.gethostname()\n # 'source': 'source', # manually set a source, defaults to the log record.pathname\n # 'source_type': '_json', # manually set a source_type, defaults to 'generic_single_line'\n 'allow_overrides': True, # Whether to look for one of the Splunk built-in parameters(index, host, ecc)\n 'verify': False, # turn SSL verification on or off, defaults to True\n # 'timeout': 60, # timeout for waiting on a 200 OK from Splunk server, defaults to 60s\n # 'retry_count': 5, # Number of retry attempts on a failed/erroring connection, defaults to 5\n # 'retry_backoff': 2.0, # Backoff factor, default options will retry for 1 min, defaults to 2.0\n 'enable_debug': True # turn on debug mode; prints module activity to stdout, defaults to False\n}\n\nsplunk = SplunkSender(**splunk_conf)\n\nis_alive = splunk.get_health()\nlogging.info(is_alive)\nif not is_alive:\n raise Exception(\"HEC not alive\")\n\n# The first payload is a quote to the italian theme song of Hello!Spank\ntxt_record = \"Hello! Splunk resta con me, Hello! Splunk non te ne andare, Caro Splunk! gioca con me, siamo amici io e te.\"\njson_record = { # this record will be parsed as normal text due to default \"sourcetype\" conf param\n \"source\": \"spacecraft Discovery 1\",\n \"host\": \"HAL9000\",\n # \"sourcetype\": \"_json\", # source type without underscore to allow the override of this built-in parameter\n \"index\": \"main\",\n \"event\": {\"message\": \"I am afraid I can't do that Dave.\", \"severity\": \"ERROR\"},\n \"rack\": \"42\",\n \"os\": \"Linux, obvious\",\n \"arch\": \"x64\"\n }\npayloads = [txt_record, json_record]\n\nsplunk_res = splunk.send_data(payloads)\nlogging.info(splunk_res)\n\nack_id = splunk_res.get('ackId')\nsplunk_ack_res = splunk.send_acks(ack_id)\nlogging.info(splunk_ack_res)\n```\n\n### Batching support\n\nYou may want to enable batching support, to avoid sending a API call to Splunk every time send\\_data() is called. \\\nTo enable the feature, use the `max_buf_size` parameter:\n```python\nfrom splunk_data_sender import SplunkSender\n\nsplunk_conf = {\n 'endpoint': 'localhost',\n 'port': '8000',\n 'token': '851A5E58-4EF1-7291-F947-F614A76ACB21',\n 'index': 'main',\n 'source_type': '_json',\n\n # Enable internal buffering, up to 2000 events\n 'max_buf_size': 2000,\n}\n\nsplunk = SplunkSender(**splunk_conf)\n```\n\nThen, simply call send\\_data() in your loop. API calls will be performed when enough events are gathered. \\\nCall flush\\_buffer() at the end of your code to send remaining events, or if you want to flush the buffer:\n\n```python\nfor event in generate_event():\n\t# event may be a dict, a string, or a list of events\n\t# the event or list of events will be appended to the internal buffer\n\t# If the buffer holds more than max_buf_size items (2000 in our example),\n\t# then an API call will be made and the buffer will be reset.\n\t# If such call is made, send_data returns its result.\n\t# If no call is made, send_data() returns None\n\tsplunk.send_data(event)\n\n# We finished processing our stuff, we must commit any remaining events to Splunk\nsplunk.flush_buffer()\n```\n\n## Configuration parameters notes\n### \"source_type\"\nIf this param is sets to \"_json\" (and \"allow_overrides\" too), not built-in params will be added inside a **\"fields\"** key described below.\nRefer to the [official Splunk documentation](https://docs.splunk.com/Documentation/Splunk/8.0.5/Data/Listofpretrainedsourcetypes) \nfor more information about source types. \n\n### \"allow_overrides\"\nIf this param is set to \"True\", whether to look for one of the Splunk built-in parameters \n(time, source, host, index) it will override the autocompleted parameters.<br>\nFor example, a json record with \"time\"=1486683865.000 will simulates a payload in the past for Splunk.\n\n### Overriding parameters without json\nBy default, Splunk built-in parameters can only be overwritten from json records.\nHowever, you may need to send non-json records while keeping this ability.\nTo that purpose, you may use the `event\\_formatter` parameter. It takes a function in charge\nof formatting events before sending them to Splunk.\nOnce set, you should call `send_data()` with events formatted as dict.\nThe library will process the events as if they were json objects, will override the parameters\naccordingly, and will then call your function to reformat the resulting event.\n\nA sample code follows:\n```python\nfrom splunk_data_sender import SplunkSender\n\n\ndef myEventFormatter(event):\n # Transforms the event in any way you like\n # event is of the same kind of what you sent to send_data()\n # This sample function will return the event as XML, because why not\n import xmltodict\n return xmltodict.unparse({'event': event})\n\nsplunk_conf = {\n 'endpoint': 'localhost',\n 'port': '8000',\n 'token': '851A5E58-4EF1-7291-F947-F614A76ACB21',\n 'index': 'main',\n 'allow_overrides': True,\n 'source_type': 'my_source_type',\n 'event_formatter': myEventFormatter\n}\n\nsplunk = SplunkSender(**splunk_conf)\n\n# We can still send non-dict events. The formatter callback must be able to handle whatever\nkind of data you use.\ntxt_record = \"Hello! Splunk resta con me, Hello! Splunk non te ne andare, Caro Splunk! gioca con me, siamo amici io e te.\"\n\n# Built-in parameters will be overridden.\n# The resulting dict will be received by the callback function and transformed to XML\njson_record = {\n \"source\": \"spacecraft Discovery 1\",\n \"host\": \"HAL9000\",\n \"index\": \"main\",\n \"event\": {\"message\": \"I am afraid I can't do that Dave.\", \"severity\": \"ERROR\"},\n \"rack\": \"42\",\n \"os\": \"Linux, obvious\",\n \"arch\": \"x64\"\n}\npayloads = [txt_record, json_record]\nsplunk_res = splunk.send_data(payloads)\n```\n\n## Notes for JSON source type event \nRefer to the [official Splunk documentation](https://docs.splunk.com/Documentation/Splunk/8.0.5/Data/IFXandHEC) \nfor more information about the use of JSON source type.\n\n### Nested JSON inside the \"event\" property\nIn some cases, events have nested json which contains the custom fields to be indexed. In this case you have to set \"sourcetype\" = \"_json\".\n\n### \"fields\"\nFields for indexing that do not occur in the event payload itself.<br>\nYou can use this parameter when you do not want particular fields to be included in the event data, \nbut you need additional metadata for indexing and searching.<br>\nIn the above example, \"rack\", \"os\" and \"arch\" will included inside \"fields\" key.<br>\n\nUsing this method is also typically faster than the nested JSON method.<br>\nBe aware that you must send HEC requests containing the fields property to the _/collector/event_ endpoint. Otherwise, they will not be indexed.\n\n## Retry Logic\n\nThis library uses the built-in retry logic from urllib3 (a retry counter and a backoff factor). \nShould the defaults not be desireable, you can find more information about how to best configure these\nsettings in the [urllib3 documentation](https://github.com/kennethreitz/requests/blob/b2289cd2d5d21bd31cf4a818a4e0ff6951b2317a/requests/packages/urllib3/util/retry.py#L104).\n\n## Contributing\n\nFeel free to contribute an issue or pull request:\n \n1. Check for existing issues and PRs\n2. Fork the repo, and clone it locally\n3. Create a new branch for your contribution\n4. Push to your fork and submit a pull request\n\n## License\n\nThis project is licensed under the terms of the [MIT license](http://opensource.org/licenses/MIT).\n",
"bugtrack_url": null,
"license": null,
"summary": "A Python connector that sends your data to Splunk via HEC",
"version": "0.5.0",
"project_urls": {
"Homepage": "https://github.com/Sonic0/splunk-data-sender",
"Issues": "https://github.com/Sonic0/splunk-data-sender/issues"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "8666aa3b99848a71300f715ab03125552a600043fa19199504413c109bb36ddd",
"md5": "8a38198ffd804034de2a26d3aac2b39c",
"sha256": "380780b89ae0109edfddbb955635d746f8e1dc0bc32f71279e68a1ff68defe06"
},
"downloads": -1,
"filename": "splunk_data_sender-0.5.0-py2.py3-none-any.whl",
"has_sig": false,
"md5_digest": "8a38198ffd804034de2a26d3aac2b39c",
"packagetype": "bdist_wheel",
"python_version": "py2.py3",
"requires_python": ">=3.8",
"size": 12873,
"upload_time": "2024-08-05T22:05:17",
"upload_time_iso_8601": "2024-08-05T22:05:17.762413Z",
"url": "https://files.pythonhosted.org/packages/86/66/aa3b99848a71300f715ab03125552a600043fa19199504413c109bb36ddd/splunk_data_sender-0.5.0-py2.py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "2756896ce2890ec804c1dd45312750afdf9899241c3f0a205e1b7b7444d406c4",
"md5": "ac6dfca9b78e7e714f2851f6059bea8f",
"sha256": "d1391dacb2d00add9d6fa38bfc872d7b50c3a901384c38cafa4e15e4f51ed44f"
},
"downloads": -1,
"filename": "splunk_data_sender-0.5.0.tar.gz",
"has_sig": false,
"md5_digest": "ac6dfca9b78e7e714f2851f6059bea8f",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 15185,
"upload_time": "2024-08-05T22:05:19",
"upload_time_iso_8601": "2024-08-05T22:05:19.364916Z",
"url": "https://files.pythonhosted.org/packages/27/56/896ce2890ec804c1dd45312750afdf9899241c3f0a205e1b7b7444d406c4/splunk_data_sender-0.5.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-08-05 22:05:19",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "Sonic0",
"github_project": "splunk-data-sender",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "splunk-data-sender"
}
