==============
gocept.logging
==============
.. image:: https://github.com/gocept/gocept.logging/workflows/tests/badge.svg
:target: https://github.com/gocept/gocept.logging/actions?query=workflow%3Atests
.. image:: https://coveralls.io/repos/github/gocept/gocept.logging/badge.svg
:target: https://coveralls.io/github/gocept/gocept.logging
.. contents::
:depth: 2
This package provides infrastructure for semi-structured log messages.
This means appending easily parseable information after the free-text log
message to facilitate analysis of the logs later on. The ``logging`` module of
the Python standard library already has support for this, via the ``extra``
parameter. gocept.logging provides a ``Formatter`` that extracts these
``extra`` values, formats them as ``key=value`` pairs and appends them to the
message::
>>> import gocept.logging
>>> import logging
>>> import sys
>>> handler = logging.StreamHandler(sys.stdout)
>>> handler.setFormatter(gocept.logging.SyslogKeyValueFormatter())
>>> log = logging.getLogger('example')
>>> log.addHandler(handler)
>>> log.warning('Hello, world!', extra={'foo': 'bar'})
Aug 24 12:10:08 localhost example: Hello, world! foo=bar
This package is tested to be compatible with Python version 2.7 and 3.3.
Advanced usage
==============
If you have ``extra`` values that you always want to pass to your log messages
(e.g things like the current user, session id, ...) you can wrap your logger
with an `LoggerAdapter`_ that prefills these values. gocept.logging provides
one that allows both stacking adapters and overriding the prefilled values::
>>> from gocept.logging.adapter import StaticDefaults
>>> import logging
>>> log = logging.getLogger('advanced')
>>> log = StaticDefaults(log, {'foo': 'bar', 'qux': 'baz'})
>>> log = StaticDefaults(log, {'blam': 'splat'})
>>> log.warning('Hello, world!', extra={'foo': 'override'})
# yields {'foo': 'override', 'qux': 'baz', 'blam': 'splat'}
.. _`LoggerAdapter`: http://docs.python.org/2/library/logging.html#loggeradapter-objects
Testing support
---------------
To help inspecting the ``extra`` values, gocept.logging comes with a
specialized handler for testing::
>>> import gocept.logging
>>> import logging
>>> log = logging.getLogger('testing')
>>> handler = gocept.logging.TestingHandler()
>>> log.addHandler(handler)
>>> log.warning('Hello, world!', extra={'foo': 'bar'})
>>> handler.messages[0].extra['foo']
'bar'
The TestingHandler records each log message as a namedtuple of type
``gocept.logging.testing.LogMessage`` so you an easily access all parts of the
message.
Example configuration
=====================
Creating semi-structured log messages is the first half of the issue, while
analysing them is the second half. We use `logstash`_ for that purpose.
The recommended setup is::
application -> syslogd on localhost -> logstash on central host (via UDP syslog input)
For development you might want to leave out the middle man and configure the
application to send log messags via syslog protocol directly to logstash.
.. _`logstash`: http://logstash.net/
Setup with ini file
-------------------
If you have a paste.ini for your application, you might use something like
this::
[loggers]
keys = root
[handlers]
keys = console, syslog
[formatters]
keys = generic, keyvalue
[logger_root]
level = INFO
handlers = console, syslog
[handler_console]
class = StreamHandler
args = (sys.stderr,)
level = NOTSET
formatter = generic
[formatter_generic]
format = %(asctime)s %(levelname)-5.5s %(name)s: %(message)s
[handler_syslog]
class = logging.handlers.SysLogHandler
args = ()
formatter = keyvalue
[formatter_keyvalue]
class = gocept.logging.SyslogKeyValueFormatter
Setup with ZConfig
------------------
If you have a Zope application, you might use something like this::
<eventlog>
<logfile>
formatter zope.exceptions.log.Formatter
format %(asctime)s %(levelname)-5.5s %(name)s: %(message)s
path STDOUT
</logfile>
<syslog>
formatter gocept.logging.SyslogKeyValueFormatter
</syslog>
</eventlog>
syslogd configuration
---------------------
rsyslog::
$EscapeControlCharactersOnReceive off
$MaxMessageSize 64k
user.* @localhost:5140
The first two lines are to support tracebacks, which are multiline and might
take up some space. The last line tells rsyslogd to forward all messages of the
``user`` facility (which is what stdlib ``logging`` uses by default) via syslog
UDP protocol to localhost port 5140 (where logstash might be listening).
logstash configuration
----------------------
::
input {
tcp {
host => "localhost"
port => 5140
type => syslog
}
udp {
host => "localhost"
port => 5140
type => syslog
}
}
filter {
grok {
type => "syslog"
pattern => [ "(?m)<%{POSINT:syslog_pri}>%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" ]
}
syslog_pri {
type => "syslog"
}
date {
type => "syslog"
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
mutate {
type => "syslog"
exclude_tags => "_grokparsefailure"
replace => [ "@source_host", "%{syslog_hostname}" ]
replace => [ "@message", "%{syslog_program}: %{syslog_message}" ]
}
mutate {
type => "syslog"
remove => [ "syslog_hostname", "syslog_timestamp" ]
}
kv {
exclude_tags => "_grokparsefailure"
type => "syslog"
}
}
output {
elasticsearch { embedded => true }
}
Additional features
===================
ArgumentParser
--------------
The provided ``gocept.logging.ArgumentParser`` provides you with the ability to
set a ``logging`` level in you runscripts.::
from gocept.logging import ArgumentParser
parser = ArgumentParser()
# Optionally set a custom log format, defaults to ``logging.BASIC_FORMAT``
parser.LOG_FORMAT = 'LOG:%(message)s'
# add your arguments with parser.add_argument() here
options = parser.parse_args()
Use ``your_run_script --help`` to see a help message about the arguments you
can pass to set logging level.
Known bugs
==========
If you log messages as unicode, e.g. ``log.info(u'foo')``, the SyslogHandler
will (incorrectly) prepend a byte-order mark, which confuses the logstash
parser, resulting in "_grokparsefailure". This is a `known bug`_ in the Python
standard library that has been fixed in Python-2.7.4.
.. _`known bug`: http://bugs.python.org/issue14452
=========================
Developing gocept.logging
=========================
:Author:
`gocept <http://gocept.com/>`_ <mail@gocept.com>
:PyPI page:
http://pypi.python.org/pypi/gocept.logging/
:Issues:
`report by e-mail <mail@gocept.com>`_
:Source code:
https://github.com/gocept/gocept.logging
:Current change log:
https://raw.githubusercontent.com/gocept/gocept.logging/master/CHANGES.rst
=============================
Change log for gocept.logging
=============================
1.0 (2023-07-18)
================
- Drop support for Python 2.7, 3.3, 3.4, 3.5, 3.6.
- Add support for Python 3.7, 3.8 3.9, 3.10, 3.11.
- Add an alias ``.warn`` for ``.warning`` for ``StaticDefaults``. (#1)
0.8.1 (2017-01-09)
==================
- Fix `setup.py` to use relative paths.
0.8 (2016-03-17)
================
- Declare compatibility with PyPy and PyPy3.
0.7 (2015-09-29)
================
- Declare Python 3.5 compatibility.
0.6 (2015-09-17)
================
- Declare Python 3.4 compatibility.
- ``ArgumentParser.parse_args()`` now stores the computed log level on the
``log_level`` attribute of the return value.
0.5 (2014-02-07)
================
- Allow to change log format for the ``ArgumentParser``
0.4 (2013-09-24)
================
- Handle non-string log messages properly.
0.3 (2013-09-04)
================
- Added sepcialized ``argparse.ArgumentParser`` which enables user to set the
``logging`` level by default..
0.2 (2013-08-24)
================
- Add timestamp and hostname to syslog messages,
this allows plugging SyslogKeyValueFormatter directly into logstash
without an intermediary syslogd.
0.1 (2013-08-16)
================
- initial release
Raw data
{
"_id": null,
"home_page": "https://github.com/gocept/gocept.logging",
"name": "gocept.logging",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.7",
"maintainer_email": "",
"keywords": "",
"author": "gocept <mail@gocept.com>",
"author_email": "mail@gocept.com",
"download_url": "https://files.pythonhosted.org/packages/97/14/ed87355e03abb6e686d8fa719719e1e7087945bda77cfddba2e825495e02/gocept.logging-1.0.tar.gz",
"platform": null,
"description": "==============\ngocept.logging\n==============\n\n.. image:: https://github.com/gocept/gocept.logging/workflows/tests/badge.svg\n :target: https://github.com/gocept/gocept.logging/actions?query=workflow%3Atests\n.. image:: https://coveralls.io/repos/github/gocept/gocept.logging/badge.svg\n :target: https://coveralls.io/github/gocept/gocept.logging\n\n.. contents::\n :depth: 2\n\nThis package provides infrastructure for semi-structured log messages.\n\nThis means appending easily parseable information after the free-text log\nmessage to facilitate analysis of the logs later on. The ``logging`` module of\nthe Python standard library already has support for this, via the ``extra``\nparameter. gocept.logging provides a ``Formatter`` that extracts these\n``extra`` values, formats them as ``key=value`` pairs and appends them to the\nmessage::\n\n >>> import gocept.logging\n >>> import logging\n >>> import sys\n\n >>> handler = logging.StreamHandler(sys.stdout)\n >>> handler.setFormatter(gocept.logging.SyslogKeyValueFormatter())\n >>> log = logging.getLogger('example')\n >>> log.addHandler(handler)\n >>> log.warning('Hello, world!', extra={'foo': 'bar'})\n Aug 24 12:10:08 localhost example: Hello, world! foo=bar\n\nThis package is tested to be compatible with Python version 2.7 and 3.3.\n\n\nAdvanced usage\n==============\n\nIf you have ``extra`` values that you always want to pass to your log messages\n(e.g things like the current user, session id, ...) you can wrap your logger\nwith an `LoggerAdapter`_ that prefills these values. gocept.logging provides\none that allows both stacking adapters and overriding the prefilled values::\n\n >>> from gocept.logging.adapter import StaticDefaults\n >>> import logging\n\n >>> log = logging.getLogger('advanced')\n >>> log = StaticDefaults(log, {'foo': 'bar', 'qux': 'baz'})\n >>> log = StaticDefaults(log, {'blam': 'splat'})\n >>> log.warning('Hello, world!', extra={'foo': 'override'})\n # yields {'foo': 'override', 'qux': 'baz', 'blam': 'splat'}\n\n.. _`LoggerAdapter`: http://docs.python.org/2/library/logging.html#loggeradapter-objects\n\n\nTesting support\n---------------\n\nTo help inspecting the ``extra`` values, gocept.logging comes with a\nspecialized handler for testing::\n\n >>> import gocept.logging\n >>> import logging\n\n >>> log = logging.getLogger('testing')\n >>> handler = gocept.logging.TestingHandler()\n >>> log.addHandler(handler)\n >>> log.warning('Hello, world!', extra={'foo': 'bar'})\n >>> handler.messages[0].extra['foo']\n 'bar'\n\nThe TestingHandler records each log message as a namedtuple of type\n``gocept.logging.testing.LogMessage`` so you an easily access all parts of the\nmessage.\n\n\nExample configuration\n=====================\n\nCreating semi-structured log messages is the first half of the issue, while\nanalysing them is the second half. We use `logstash`_ for that purpose.\n\nThe recommended setup is::\n\n application -> syslogd on localhost -> logstash on central host (via UDP syslog input)\n\nFor development you might want to leave out the middle man and configure the\napplication to send log messags via syslog protocol directly to logstash.\n\n\n.. _`logstash`: http://logstash.net/\n\n\nSetup with ini file\n-------------------\n\nIf you have a paste.ini for your application, you might use something like\nthis::\n\n [loggers]\n keys = root\n\n [handlers]\n keys = console, syslog\n\n [formatters]\n keys = generic, keyvalue\n\n [logger_root]\n level = INFO\n handlers = console, syslog\n\n [handler_console]\n class = StreamHandler\n args = (sys.stderr,)\n level = NOTSET\n formatter = generic\n\n [formatter_generic]\n format = %(asctime)s %(levelname)-5.5s %(name)s: %(message)s\n\n [handler_syslog]\n class = logging.handlers.SysLogHandler\n args = ()\n formatter = keyvalue\n\n [formatter_keyvalue]\n class = gocept.logging.SyslogKeyValueFormatter\n\n\nSetup with ZConfig\n------------------\n\nIf you have a Zope application, you might use something like this::\n\n <eventlog>\n <logfile>\n formatter zope.exceptions.log.Formatter\n format %(asctime)s %(levelname)-5.5s %(name)s: %(message)s\n path STDOUT\n </logfile>\n <syslog>\n formatter gocept.logging.SyslogKeyValueFormatter\n </syslog>\n </eventlog>\n\n\nsyslogd configuration\n---------------------\n\nrsyslog::\n\n $EscapeControlCharactersOnReceive off\n $MaxMessageSize 64k\n user.* @localhost:5140\n\nThe first two lines are to support tracebacks, which are multiline and might\ntake up some space. The last line tells rsyslogd to forward all messages of the\n``user`` facility (which is what stdlib ``logging`` uses by default) via syslog\nUDP protocol to localhost port 5140 (where logstash might be listening).\n\n\nlogstash configuration\n----------------------\n\n::\n\n input {\n tcp {\n host => \"localhost\"\n port => 5140\n type => syslog\n }\n udp {\n host => \"localhost\"\n port => 5140\n type => syslog\n }\n }\n\n filter {\n grok {\n type => \"syslog\"\n pattern => [ \"(?m)<%{POSINT:syslog_pri}>%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\\[%{POSINT:syslog_pid}\\])?: %{GREEDYDATA:syslog_message}\" ]\n }\n syslog_pri {\n type => \"syslog\"\n }\n date {\n type => \"syslog\"\n match => [ \"syslog_timestamp\", \"MMM d HH:mm:ss\", \"MMM dd HH:mm:ss\" ]\n }\n mutate {\n type => \"syslog\"\n exclude_tags => \"_grokparsefailure\"\n replace => [ \"@source_host\", \"%{syslog_hostname}\" ]\n replace => [ \"@message\", \"%{syslog_program}: %{syslog_message}\" ]\n }\n mutate {\n type => \"syslog\"\n remove => [ \"syslog_hostname\", \"syslog_timestamp\" ]\n }\n kv {\n exclude_tags => \"_grokparsefailure\"\n type => \"syslog\"\n }\n }\n\n output {\n elasticsearch { embedded => true }\n }\n\n\nAdditional features\n===================\n\nArgumentParser\n--------------\n\nThe provided ``gocept.logging.ArgumentParser`` provides you with the ability to\nset a ``logging`` level in you runscripts.::\n\n from gocept.logging import ArgumentParser\n parser = ArgumentParser()\n # Optionally set a custom log format, defaults to ``logging.BASIC_FORMAT``\n parser.LOG_FORMAT = 'LOG:%(message)s'\n # add your arguments with parser.add_argument() here\n options = parser.parse_args()\n\nUse ``your_run_script --help`` to see a help message about the arguments you\ncan pass to set logging level.\n\n\nKnown bugs\n==========\n\nIf you log messages as unicode, e.g. ``log.info(u'foo')``, the SyslogHandler\nwill (incorrectly) prepend a byte-order mark, which confuses the logstash\nparser, resulting in \"_grokparsefailure\". This is a `known bug`_ in the Python\nstandard library that has been fixed in Python-2.7.4.\n\n.. _`known bug`: http://bugs.python.org/issue14452\n\n\n=========================\nDeveloping gocept.logging\n=========================\n\n:Author:\n `gocept <http://gocept.com/>`_ <mail@gocept.com>\n\n:PyPI page:\n http://pypi.python.org/pypi/gocept.logging/\n\n:Issues:\n `report by e-mail <mail@gocept.com>`_\n\n:Source code:\n https://github.com/gocept/gocept.logging\n\n:Current change log:\n https://raw.githubusercontent.com/gocept/gocept.logging/master/CHANGES.rst\n\n\n=============================\nChange log for gocept.logging\n=============================\n\n1.0 (2023-07-18)\n================\n\n- Drop support for Python 2.7, 3.3, 3.4, 3.5, 3.6.\n\n- Add support for Python 3.7, 3.8 3.9, 3.10, 3.11.\n\n- Add an alias ``.warn`` for ``.warning`` for ``StaticDefaults``. (#1)\n\n\n0.8.1 (2017-01-09)\n==================\n\n- Fix `setup.py` to use relative paths.\n\n\n0.8 (2016-03-17)\n================\n\n- Declare compatibility with PyPy and PyPy3.\n\n\n0.7 (2015-09-29)\n================\n\n- Declare Python 3.5 compatibility.\n\n\n0.6 (2015-09-17)\n================\n\n- Declare Python 3.4 compatibility.\n\n- ``ArgumentParser.parse_args()`` now stores the computed log level on the\n ``log_level`` attribute of the return value.\n\n0.5 (2014-02-07)\n================\n\n- Allow to change log format for the ``ArgumentParser``\n\n\n0.4 (2013-09-24)\n================\n\n- Handle non-string log messages properly.\n\n\n0.3 (2013-09-04)\n================\n\n- Added sepcialized ``argparse.ArgumentParser`` which enables user to set the\n ``logging`` level by default..\n\n\n0.2 (2013-08-24)\n================\n\n- Add timestamp and hostname to syslog messages,\n this allows plugging SyslogKeyValueFormatter directly into logstash\n without an intermediary syslogd.\n\n\n0.1 (2013-08-16)\n================\n\n- initial release\n",
"bugtrack_url": null,
"license": "ZPL 2.1",
"summary": "Infrastructure for semi-structured log messages.",
"version": "1.0",
"project_urls": {
"Homepage": "https://github.com/gocept/gocept.logging"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "a165a49acaea764e1a8719e44011a8d4fb4fb31e21843298452deb5db3c3a10f",
"md5": "45b73781fc42c557ef30ba0ce1ab6e34",
"sha256": "9376bb115d0cec5e772d9516800b6b0bad429ff78b2909eaa99e96babbe8a316"
},
"downloads": -1,
"filename": "gocept.logging-1.0-py2.py3-none-any.whl",
"has_sig": false,
"md5_digest": "45b73781fc42c557ef30ba0ce1ab6e34",
"packagetype": "bdist_wheel",
"python_version": "py2.py3",
"requires_python": ">=3.7",
"size": 11513,
"upload_time": "2023-07-18T05:43:28",
"upload_time_iso_8601": "2023-07-18T05:43:28.309447Z",
"url": "https://files.pythonhosted.org/packages/a1/65/a49acaea764e1a8719e44011a8d4fb4fb31e21843298452deb5db3c3a10f/gocept.logging-1.0-py2.py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "9714ed87355e03abb6e686d8fa719719e1e7087945bda77cfddba2e825495e02",
"md5": "b999e5d3e183fdf4a026ff86c43ddfe7",
"sha256": "b77fa80c24d981f2ba4e4852adb21825b1e5be02744ca5291a2674b8de99091f"
},
"downloads": -1,
"filename": "gocept.logging-1.0.tar.gz",
"has_sig": false,
"md5_digest": "b999e5d3e183fdf4a026ff86c43ddfe7",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7",
"size": 12948,
"upload_time": "2023-07-18T05:43:29",
"upload_time_iso_8601": "2023-07-18T05:43:29.873010Z",
"url": "https://files.pythonhosted.org/packages/97/14/ed87355e03abb6e686d8fa719719e1e7087945bda77cfddba2e825495e02/gocept.logging-1.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-07-18 05:43:29",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "gocept",
"github_project": "gocept.logging",
"travis_ci": false,
"coveralls": true,
"github_actions": true,
"tox": true,
"lcname": "gocept.logging"
}
JSON
