Python logging library to emit JSON logs in a SAP CloudFoundry environment.
===========================================================================
.. image:: https://api.reuse.software/badge/github.com/SAP/cf-python-logging-support
:target: https://api.reuse.software/info/github.com/SAP/cf-python-logging-support
This is a collection of support libraries for Python applications running on Cloud Foundry that
serve two main purposes: provide (a) means to emit structured application log messages and (b)
instrument web applications of your application stack to collect request metrics.
For details on the concepts and log formats, please look at the sibling project for `java logging
support <https://github.com/SAP/cf-java-logging-support>`__.
Features
-----------
1. Lightweight, no dependencies. Support of Python 2.7 & 3.5.
2. Compatible with the Python **logging** module. Minimal configuration needed.
3. Emits JSON logs (`format
details <https://github.com/SAP/cf-java-logging-support/tree/master/cf-java-logging-support-core/beats>`__).
4. Supports **correlation-id**.
5. Supports request instrumentation. Built in support for:
* `Flask 0.1x <http://flask.pocoo.org/>`__
* `Sanic 0.5.x <https://github.com/channelcat/sanic>`__
* `Falcon <https://falconframework.org/>`__
* `Django <https://www.djangoproject.com/>`__
* Extensible to support others
6. Includes CF-specific information (space id, app id, etc.) to logs.
7. Supports adding extra properties to JSON log object.
Installation
------------
Install the package with pip:
::
pip install sap_cf_logging
Usage
-----
Setting up your application
~~~~~~~~~~~~~~~~~~~~~~~~~~~
Logging library needs to be initialized. Depending on you application type, different initialization
is used. You should usually do this in your application entrypoint.
For CLI applications you just need to call ``cf_logging.init()`` *once* to configure the library.
The library will try to configure future loggers to emit logs in JSON format.
If you are using one of the supported frameworks, check the `Configuration <#configuration>`__
section to see how to configure it.
**Setting up the CloudFoundry environment**
In order for your logs to appear in the Kibana dashboard, you have to create an **application-logs**
service instance and bind it to your application.
Configuration
~~~~~~~~~~~~~
After installation use the following guide to configure the Python cf logging library.
Flask
^^^^^
First import the ``cf_logging`` library and setup Flask logging on the application.
.. code:: python
from sap.cf_logging import flask_logging
app = flask.Flask(__name__)
flask_logging.init(app, logging.INFO)
Next use Python’s logging library
.. code:: python
@app.route('/')
def root_route():
logger = logging.getLogger('my.logger')
logger.info('Hi')
return 'ok'
Note the logs generated by the application
Sanic
^^^^^
.. code:: python
import sanic
import logging
from sanic.response import HTTPResponse
from sap.cf_logging import sanic_logging
from sap.cf_logging.core.constants import REQUEST_KEY
app = sanic.Sanic('test.cf_logging')
sanic_logging.init(app)
@app.route('/')
async def two(request):
extra = {REQUEST_KEY: request}
logging.getLogger('my.logger').debug('Hi', extra = extra)
return HTTPResponse(body='ok')
**Note**: With Sanic you need to pass the request with an ``extra`` parameter in the logging API.
This is needed in order to get the *correlation_id* generated at the beginning of the request or
fetched from the HTTP headers.
Falcon
^^^^^^
.. code:: python
import falcon
from sap.cf_logging import falcon_logging
from sap.cf_logging.core.constants import REQUEST_KEY
class Resource:
def on_get(self, req, resp):
extra = {REQUEST_KEY: req}
logging.getLogger('my.logger').log('Resource requested', extra=extra)
resp.media = {'name': 'Cloud Foundry'}
app = falcon.API(middleware=[
falcon_logging.LoggingMiddleware()
])
app.add_route('/resource', Resource())
falcon_logging.init(app)
Django
^^^^^^
.. code:: bash
django-admin startproject example
.. code:: python
# example/settings.py
MIDDLEWARES = [
# ...,
'sap.cf_logging.django_logging.LoggingMiddleware'
]
# example/wsgi.py
# ...
from sap.cf_logging import django_logging
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "sap_logtesting.settings")
django_logging.init()
# ...
Create a new app
.. code:: bash
python manage.py startapp example_app
.. code:: python
# example_app/views.py
import logging
from django.http import HttpResponse
from sap.cf_logging.core.constants import REQUEST_KEY
def index(request):
extra = {REQUEST_KEY: request}
logger = logging.getLogger('my.logger')
logger.info("Resource requested", extra=extra)
return HttpResponse("ok")
# example_app/urls.py
from django.conf.urls import url
from . import views
urlpatterns = [
url('^$', views.index)
]
# example/urls.py
from django.contrib import admin
from django.conf.urls import url, include
urlpatterns = [
url('admin/', admin.site.urls),
url('example/', include('example_app.urls'))
]
General
^^^^^^^
.. code:: python
import logging
from sap import cf_logging
cf_logging.init()
logger = logging.getLogger("cli.logger")
logger.info('hi')
**Notes**: All loggers set up and created before the initialization of the Cloud Foundry logging library will
be left untouched. When using Flask and Sanic with the logging library before and
after request middleware is attached, and it will capture response times for each request.
Custom Fields
"""""""""""""
To use custom fields. Pass a dictionary property custom_fields to the initialize method:
.. code:: python
import logging
from sap import cf_logging
cf_logging.init(custom_fields={"foo": "default", "bar": None})
Here we mark the two fields: foo and bar as custom_fields. Logging with:
.. code:: python
logging.getLogger('my.logger').debug('Hi')
The property foo will be output as a custom field with a value "default". The property bar will not be logged, as it does not have a value.
To log bar, provide a value when logging:
.. code:: python
logging.getLogger('my.logger').debug('Hi', extra={"bar": "new_value"})
It is also possible to log foo with a different value:
.. code:: python
logging.getLogger('my.logger').debug('Hi', extra={"foo": "hello"})
Setting and getting correlation ID
""""""""""""""""""""""""""""""""""
When using cf_logging in a web application you don't need to set the correlation ID, because the logging library will fetch it from the HTTP headers and set it.
For non web applications you could set the correlation ID manually, so that the log entries can be filtered later on based on the ``correlation_id`` log property.
In this case the correlation ID is kept in a thread local variable and each thread should set its own correlation ID.
Setting and getting the correlation_id can be done via:
.. code:: python
cf_logging.FRAMEWORK.context.get_correlation_id()
cf_logging.FRAMEWORK.context.set_correlation_id(value)
If you need to get the correlation ID in a web application, take into account the framework you are using.
In async frameworks like Sanic and Falcon the context is stored into the request object and you need to provide the request to the call:
.. code:: python
cf_logging.FRAMEWORK.context.get_correlation_id(request)
Logging sensitive data
^^^^^^^^^^^^^^^^^^^^^^
The logging library does not log sensitive fields by default. Those fields are replaced with 'redacted' instead of their original content.
The following fields are considered sensitive data: ``remote_ip``, ``remote_host``, ``remote_port``, ``x_forwarded_for``, ``remote_user``, ``referer``.
Logging of all or some of these fields can be activated by setting the following environment variables:
+-----------------------------------+-----------+------------------------------------------------------------------------+
| Environment variable | Value | Enables sensitive field |
+===================================+===========+========================================================================+
| ``LOG_SENSITIVE_CONNECTION_DATA`` | true | ``remote_ip``, ``remote_host``, ``remote_port``, ``x_forwarded_for`` |
+-----------------------------------+-----------+------------------------------------------------------------------------+
| ``LOG_REMOTE_USER`` | true | ``remote_user`` |
+-----------------------------------+-----------+------------------------------------------------------------------------+
| ``LOG_REFERER`` | true | ``referer`` |
+-----------------------------------+-----------+------------------------------------------------------------------------+
This behavior matches the corresponding mechanism in the `CF Java Logging Support library <https://github.com/SAP/cf-java-logging-support/wiki/Overview#logging-sensitive-user-data>`__.
Examples
~~~~~~~~
For more examples please see the tests within the ``./tests/`` directory.
Requirements
------------
No external requirements are needed to run the package.
Limitations
-----------
NA
Known Issues
------------
NA
How to obtain support
---------------------
Please open an issue on the github page.
Contributing
------------
Please create a pull request and briefly describe the nature of the change. Please submit a test
case along with your pull request.
To-Do (upcoming changes)
------------------------
NA
Changelog
---------
See `CHANGELOG file <https://github.com/SAP/cf-python-logging-support/blob/master/CHANGELOG.md>`__.
License
-------
Copyright (c) 2017-2021 SAP SE or an SAP affiliate company and cf-python-logging-support contributors. Please see our `LICENSE file <https://github.com/SAP/cf-python-logging-support/blob/master/LICENSE>`__ for copyright and license information. Detailed information including third-party components and their licensing/copyright information is available `via the REUSE tool <https://api.reuse.software/info/github.com/SAP/cf-python-logging-support>`__.
Raw data
{
"_id": null,
"home_page": "https://github.com/SAP/cf-python-logging-support",
"name": "sap-cf-logging",
"maintainer": null,
"docs_url": null,
"requires_python": null,
"maintainer_email": null,
"keywords": null,
"author": "SAP",
"author_email": null,
"download_url": "https://files.pythonhosted.org/packages/57/1f/c848ad1f0e14c34d00a22dfe0f735c1ae824ae1d478c10d5b2b74a44273d/sap_cf_logging-4.2.7.tar.gz",
"platform": "any",
"description": "\r\nPython logging library to emit JSON logs in a SAP CloudFoundry environment.\r\n===========================================================================\r\n\r\n.. image:: https://api.reuse.software/badge/github.com/SAP/cf-python-logging-support\r\n :target: https://api.reuse.software/info/github.com/SAP/cf-python-logging-support\r\n\r\nThis is a collection of support libraries for Python applications running on Cloud Foundry that\r\nserve two main purposes: provide (a) means to emit structured application log messages and (b)\r\ninstrument web applications of your application stack to collect request metrics.\r\n\r\nFor details on the concepts and log formats, please look at the sibling project for `java logging\r\nsupport <https://github.com/SAP/cf-java-logging-support>`__.\r\n\r\n\r\nFeatures\r\n-----------\r\n\r\n1. Lightweight, no dependencies. Support of Python 2.7 & 3.5.\r\n2. Compatible with the Python **logging** module. Minimal configuration needed.\r\n3. Emits JSON logs (`format\r\n details <https://github.com/SAP/cf-java-logging-support/tree/master/cf-java-logging-support-core/beats>`__).\r\n4. Supports **correlation-id**.\r\n5. Supports request instrumentation. Built in support for:\r\n\r\n * `Flask 0.1x <http://flask.pocoo.org/>`__\r\n * `Sanic 0.5.x <https://github.com/channelcat/sanic>`__\r\n * `Falcon <https://falconframework.org/>`__\r\n * `Django <https://www.djangoproject.com/>`__\r\n * Extensible to support others\r\n\r\n6. Includes CF-specific information (space id, app id, etc.) to logs.\r\n7. Supports adding extra properties to JSON log object.\r\n\r\nInstallation\r\n------------\r\n\r\nInstall the package with pip:\r\n\r\n::\r\n\r\n pip install sap_cf_logging\r\n\r\nUsage\r\n-----\r\n\r\nSetting up your application\r\n~~~~~~~~~~~~~~~~~~~~~~~~~~~\r\n\r\nLogging library needs to be initialized. Depending on you application type, different initialization\r\nis used. You should usually do this in your application entrypoint.\r\n\r\nFor CLI applications you just need to call ``cf_logging.init()`` *once* to configure the library.\r\nThe library will try to configure future loggers to emit logs in JSON format.\r\n\r\nIf you are using one of the supported frameworks, check the `Configuration <#configuration>`__\r\nsection to see how to configure it.\r\n\r\n**Setting up the CloudFoundry environment**\r\n\r\nIn order for your logs to appear in the Kibana dashboard, you have to create an **application-logs**\r\nservice instance and bind it to your application.\r\n\r\n\r\nConfiguration\r\n~~~~~~~~~~~~~\r\n\r\nAfter installation use the following guide to configure the Python cf logging library.\r\n\r\nFlask\r\n^^^^^\r\n\r\nFirst import the ``cf_logging`` library and setup Flask logging on the application.\r\n\r\n.. code:: python\r\n\r\n from sap.cf_logging import flask_logging\r\n\r\n app = flask.Flask(__name__)\r\n flask_logging.init(app, logging.INFO)\r\n\r\nNext use Python\u2019s logging library\r\n\r\n.. code:: python\r\n\r\n @app.route('/')\r\n def root_route():\r\n logger = logging.getLogger('my.logger')\r\n logger.info('Hi')\r\n\r\n return 'ok'\r\n\r\nNote the logs generated by the application\r\n\r\nSanic\r\n^^^^^\r\n\r\n.. code:: python\r\n\r\n import sanic\r\n import logging\r\n\r\n from sanic.response import HTTPResponse\r\n from sap.cf_logging import sanic_logging\r\n from sap.cf_logging.core.constants import REQUEST_KEY\r\n\r\n app = sanic.Sanic('test.cf_logging')\r\n sanic_logging.init(app)\r\n\r\n @app.route('/')\r\n async def two(request):\r\n extra = {REQUEST_KEY: request}\r\n logging.getLogger('my.logger').debug('Hi', extra = extra)\r\n return HTTPResponse(body='ok')\r\n\r\n**Note**: With Sanic you need to pass the request with an ``extra`` parameter in the logging API.\r\nThis is needed in order to get the *correlation_id* generated at the beginning of the request or\r\nfetched from the HTTP headers.\r\n\r\nFalcon\r\n^^^^^^\r\n\r\n.. code:: python\r\n\r\n\r\n import falcon\r\n from sap.cf_logging import falcon_logging\r\n from sap.cf_logging.core.constants import REQUEST_KEY\r\n\r\n\r\n class Resource:\r\n def on_get(self, req, resp):\r\n extra = {REQUEST_KEY: req}\r\n logging.getLogger('my.logger').log('Resource requested', extra=extra)\r\n resp.media = {'name': 'Cloud Foundry'}\r\n\r\n\r\n app = falcon.API(middleware=[\r\n falcon_logging.LoggingMiddleware()\r\n ])\r\n app.add_route('/resource', Resource())\r\n falcon_logging.init(app)\r\n\r\nDjango\r\n^^^^^^\r\n\r\n.. code:: bash\r\n\r\n django-admin startproject example\r\n\r\n.. code:: python\r\n\r\n # example/settings.py\r\n\r\n MIDDLEWARES = [\r\n # ...,\r\n 'sap.cf_logging.django_logging.LoggingMiddleware'\r\n ]\r\n\r\n # example/wsgi.py\r\n\r\n # ...\r\n from sap.cf_logging import django_logging\r\n\r\n os.environ.setdefault(\"DJANGO_SETTINGS_MODULE\", \"sap_logtesting.settings\")\r\n django_logging.init()\r\n\r\n # ...\r\n\r\nCreate a new app\r\n\r\n.. code:: bash\r\n\r\n python manage.py startapp example_app\r\n\r\n.. code:: python\r\n\r\n # example_app/views.py\r\n\r\n import logging\r\n\r\n from django.http import HttpResponse\r\n from sap.cf_logging.core.constants import REQUEST_KEY\r\n\r\n def index(request):\r\n extra = {REQUEST_KEY: request}\r\n logger = logging.getLogger('my.logger')\r\n logger.info(\"Resource requested\", extra=extra)\r\n return HttpResponse(\"ok\")\r\n\r\n # example_app/urls.py\r\n\r\n from django.conf.urls import url\r\n\r\n from . import views\r\n\r\n urlpatterns = [\r\n url('^$', views.index)\r\n ]\r\n\r\n # example/urls.py\r\n\r\n from django.contrib import admin\r\n from django.conf.urls import url, include\r\n\r\n urlpatterns = [\r\n url('admin/', admin.site.urls),\r\n url('example/', include('example_app.urls'))\r\n ]\r\n\r\nGeneral\r\n^^^^^^^\r\n\r\n.. code:: python\r\n\r\n import logging\r\n from sap import cf_logging\r\n\r\n cf_logging.init()\r\n\r\n logger = logging.getLogger(\"cli.logger\")\r\n logger.info('hi')\r\n\r\n**Notes**: All loggers set up and created before the initialization of the Cloud Foundry logging library will\r\nbe left untouched. When using Flask and Sanic with the logging library before and\r\nafter request middleware is attached, and it will capture response times for each request.\r\n\r\n\r\nCustom Fields\r\n\"\"\"\"\"\"\"\"\"\"\"\"\"\r\n\r\nTo use custom fields. Pass a dictionary property custom_fields to the initialize method:\r\n\r\n.. code:: python\r\n\r\n import logging\r\n from sap import cf_logging\r\n cf_logging.init(custom_fields={\"foo\": \"default\", \"bar\": None})\r\n\r\nHere we mark the two fields: foo and bar as custom_fields. Logging with:\r\n\r\n.. code:: python\r\n\r\n logging.getLogger('my.logger').debug('Hi')\r\n\r\nThe property foo will be output as a custom field with a value \"default\". The property bar will not be logged, as it does not have a value.\r\n\r\nTo log bar, provide a value when logging:\r\n\r\n.. code:: python\r\n\r\n logging.getLogger('my.logger').debug('Hi', extra={\"bar\": \"new_value\"})\r\n\r\nIt is also possible to log foo with a different value:\r\n\r\n.. code:: python\r\n\r\n logging.getLogger('my.logger').debug('Hi', extra={\"foo\": \"hello\"})\r\n\r\n\r\nSetting and getting correlation ID\r\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\r\n\r\nWhen using cf_logging in a web application you don't need to set the correlation ID, because the logging library will fetch it from the HTTP headers and set it.\r\nFor non web applications you could set the correlation ID manually, so that the log entries can be filtered later on based on the ``correlation_id`` log property.\r\nIn this case the correlation ID is kept in a thread local variable and each thread should set its own correlation ID.\r\n\r\nSetting and getting the correlation_id can be done via:\r\n\r\n.. code:: python\r\n\r\n cf_logging.FRAMEWORK.context.get_correlation_id()\r\n cf_logging.FRAMEWORK.context.set_correlation_id(value)\r\n\r\nIf you need to get the correlation ID in a web application, take into account the framework you are using.\r\nIn async frameworks like Sanic and Falcon the context is stored into the request object and you need to provide the request to the call:\r\n\r\n.. code:: python\r\n\r\n cf_logging.FRAMEWORK.context.get_correlation_id(request)\r\n\r\n\r\nLogging sensitive data\r\n^^^^^^^^^^^^^^^^^^^^^^\r\n\r\nThe logging library does not log sensitive fields by default. Those fields are replaced with 'redacted' instead of their original content.\r\nThe following fields are considered sensitive data: ``remote_ip``, ``remote_host``, ``remote_port``, ``x_forwarded_for``, ``remote_user``, ``referer``.\r\nLogging of all or some of these fields can be activated by setting the following environment variables:\r\n\r\n+-----------------------------------+-----------+------------------------------------------------------------------------+\r\n| Environment variable | Value | Enables sensitive field |\r\n+===================================+===========+========================================================================+\r\n| ``LOG_SENSITIVE_CONNECTION_DATA`` | true | ``remote_ip``, ``remote_host``, ``remote_port``, ``x_forwarded_for`` |\r\n+-----------------------------------+-----------+------------------------------------------------------------------------+\r\n| ``LOG_REMOTE_USER`` | true | ``remote_user`` |\r\n+-----------------------------------+-----------+------------------------------------------------------------------------+\r\n| ``LOG_REFERER`` | true | ``referer`` |\r\n+-----------------------------------+-----------+------------------------------------------------------------------------+\r\n\r\nThis behavior matches the corresponding mechanism in the `CF Java Logging Support library <https://github.com/SAP/cf-java-logging-support/wiki/Overview#logging-sensitive-user-data>`__.\r\n\r\nExamples\r\n~~~~~~~~\r\n\r\nFor more examples please see the tests within the ``./tests/`` directory.\r\n\r\nRequirements\r\n------------\r\n\r\nNo external requirements are needed to run the package.\r\n\r\nLimitations\r\n-----------\r\n\r\nNA\r\n\r\nKnown Issues\r\n------------\r\n\r\nNA\r\n\r\nHow to obtain support\r\n---------------------\r\n\r\nPlease open an issue on the github page.\r\n\r\nContributing\r\n------------\r\n\r\nPlease create a pull request and briefly describe the nature of the change. Please submit a test\r\ncase along with your pull request.\r\n\r\nTo-Do (upcoming changes)\r\n------------------------\r\n\r\nNA\r\n\r\nChangelog\r\n---------\r\n\r\nSee `CHANGELOG file <https://github.com/SAP/cf-python-logging-support/blob/master/CHANGELOG.md>`__.\r\n\r\nLicense\r\n-------\r\n\r\nCopyright (c) 2017-2021 SAP SE or an SAP affiliate company and cf-python-logging-support contributors. Please see our `LICENSE file <https://github.com/SAP/cf-python-logging-support/blob/master/LICENSE>`__ for copyright and license information. Detailed information including third-party components and their licensing/copyright information is available `via the REUSE tool <https://api.reuse.software/info/github.com/SAP/cf-python-logging-support>`__.\r\n\r\n",
"bugtrack_url": null,
"license": "Apache License, Version 2.0",
"summary": "Python logging library to emit JSON logs in a SAP CloudFoundry environment",
"version": "4.2.7",
"project_urls": {
"Homepage": "https://github.com/SAP/cf-python-logging-support"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "1bbf7e1ba9626dabbb0f4c1a9e4a6ca1e9e25ceecb521752dabfdc51947eb2d1",
"md5": "f93333d1436df4e7f21e48ad656de0ea",
"sha256": "21e3247e180317c58c206888e8f795928fd8f7d38ab2dd94aaf08812843d9471"
},
"downloads": -1,
"filename": "sap_cf_logging-4.2.7-py2.py3-none-any.whl",
"has_sig": false,
"md5_digest": "f93333d1436df4e7f21e48ad656de0ea",
"packagetype": "bdist_wheel",
"python_version": "py2.py3",
"requires_python": null,
"size": 31128,
"upload_time": "2024-06-28T08:50:38",
"upload_time_iso_8601": "2024-06-28T08:50:38.935399Z",
"url": "https://files.pythonhosted.org/packages/1b/bf/7e1ba9626dabbb0f4c1a9e4a6ca1e9e25ceecb521752dabfdc51947eb2d1/sap_cf_logging-4.2.7-py2.py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "571fc848ad1f0e14c34d00a22dfe0f735c1ae824ae1d478c10d5b2b74a44273d",
"md5": "ced0da88b6e3fc7a31ec68e85abd5284",
"sha256": "e24c2611da5cf380714c3c9dcb8440870444379b7fc42971122b030311eed247"
},
"downloads": -1,
"filename": "sap_cf_logging-4.2.7.tar.gz",
"has_sig": false,
"md5_digest": "ced0da88b6e3fc7a31ec68e85abd5284",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 28515,
"upload_time": "2024-06-28T08:50:40",
"upload_time_iso_8601": "2024-06-28T08:50:40.518314Z",
"url": "https://files.pythonhosted.org/packages/57/1f/c848ad1f0e14c34d00a22dfe0f735c1ae824ae1d478c10d5b2b74a44273d/sap_cf_logging-4.2.7.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-06-28 08:50:40",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "SAP",
"github_project": "cf-python-logging-support",
"travis_ci": true,
"coveralls": true,
"github_actions": false,
"tox": true,
"lcname": "sap-cf-logging"
}
JSON
