# multiprocessing-logging
[](https://pypi.python.org/pypi/multiprocessing-logging/)
[](https://pypi.python.org/pypi/multiprocessing-logging/)
When using the `multiprocessing` module, logging becomes less useful since
sub-processes should log to individual files/streams or there's the risk of
records becoming garbled.
This simple module implements a `Handler` that when set on the root
`Logger` will handle tunneling the records to the main process so that
they are handled correctly.
It's currently tested in Linux and Python 2.7 & 3.6+.
Pypy3 hangs on the tests so I don't recommend using it.
Pypy appears to be working, recently.
Only works on POSIX systems and only Linux is supported. It does not work on Windows.
# Origin
This library was taken verbatim from a [StackOverflow post](http://stackoverflow.com/questions/641420/how-should-i-log-while-using-multiprocessing-in-python)
and extracted into a module so that I wouldn't have to copy the code in every
project.
Later, several improvements have been contributed.
# Usage
Before you start logging but after you configure the logging framework (maybe with `logging.basicConfig(...)`), do the following:
```py
import multiprocessing_logging
multiprocessing_logging.install_mp_handler()
```
and that's it.
## With multiprocessing.Pool
When using a Pool, make sure `install_mp_handler` is called before the Pool is instantiated, for example:
```py
import logging
from multiprocessing import Pool
from multiprocessing_logging import install_mp_handler
logging.basicConfig(...)
install_mp_handler()
pool = Pool(...)
```
# Problems
The approach of this module relies on
[fork](https://docs.python.org/3.9/library/multiprocessing.html#multiprocessing.set_start_method)
being used to create new processes. This start method
[is basically unsafe when also using threads](https://bugs.python.org/issue37429),
as this module does.
The consequence is that there's a low probability of the application hanging
when creating new processes.
As a palliative, don't continuously create new processes. Instead, create a
Pool once and reuse it.
Raw data
{
"_id": null,
"home_page": "https://github.com/jruere/multiprocessing-logging",
"name": "multiprocessing-logging",
"maintainer": "",
"docs_url": null,
"requires_python": "",
"maintainer_email": "",
"keywords": "multiprocessing logging logger handler",
"author": "Javier Ruere",
"author_email": "javier@ruere.com.ar",
"download_url": "",
"platform": "POSIX",
"description": "# multiprocessing-logging\n\n[](https://pypi.python.org/pypi/multiprocessing-logging/)\n[](https://pypi.python.org/pypi/multiprocessing-logging/)\n\n\nWhen using the `multiprocessing` module, logging becomes less useful since\nsub-processes should log to individual files/streams or there's the risk of\nrecords becoming garbled.\n\nThis simple module implements a `Handler` that when set on the root\n`Logger` will handle tunneling the records to the main process so that\nthey are handled correctly.\n\nIt's currently tested in Linux and Python 2.7 & 3.6+.\n\nPypy3 hangs on the tests so I don't recommend using it.\n\nPypy appears to be working, recently.\n\nOnly works on POSIX systems and only Linux is supported. It does not work on Windows.\n\n# Origin\n\nThis library was taken verbatim from a [StackOverflow post](http://stackoverflow.com/questions/641420/how-should-i-log-while-using-multiprocessing-in-python)\nand extracted into a module so that I wouldn't have to copy the code in every\nproject.\n\nLater, several improvements have been contributed.\n\n# Usage\n\nBefore you start logging but after you configure the logging framework (maybe with `logging.basicConfig(...)`), do the following:\n\n```py\nimport multiprocessing_logging\n\nmultiprocessing_logging.install_mp_handler()\n```\n\nand that's it.\n\n## With multiprocessing.Pool\n\nWhen using a Pool, make sure `install_mp_handler` is called before the Pool is instantiated, for example:\n\n```py\nimport logging\nfrom multiprocessing import Pool\nfrom multiprocessing_logging import install_mp_handler\n\nlogging.basicConfig(...)\ninstall_mp_handler()\npool = Pool(...)\n```\n\n# Problems\nThe approach of this module relies on\n[fork](https://docs.python.org/3.9/library/multiprocessing.html#multiprocessing.set_start_method)\nbeing used to create new processes. This start method\n[is basically unsafe when also using threads](https://bugs.python.org/issue37429),\nas this module does.\n\nThe consequence is that there's a low probability of the application hanging\nwhen creating new processes.\n\nAs a palliative, don't continuously create new processes. Instead, create a\nPool once and reuse it.\n",
"bugtrack_url": null,
"license": "LGPLv3",
"summary": "Logger for multiprocessing applications",
"version": "0.3.4",
"project_urls": {
"Homepage": "https://github.com/jruere/multiprocessing-logging"
},
"split_keywords": [
"multiprocessing",
"logging",
"logger",
"handler"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "9efe32bd864bcb604b0607924a4cf618ed267a0ef21ac9c3e255109256046e1f",
"md5": "f79b62a9ba38cd45b95038dbf33e1b67",
"sha256": "8a5be02b02edbd6fa6e3e89499af7680db69db9e2d8707fcd28d445fa248f23e"
},
"downloads": -1,
"filename": "multiprocessing_logging-0.3.4-py2.py3-none-any.whl",
"has_sig": false,
"md5_digest": "f79b62a9ba38cd45b95038dbf33e1b67",
"packagetype": "bdist_wheel",
"python_version": "py2.py3",
"requires_python": null,
"size": 8770,
"upload_time": "2023-02-05T17:06:20",
"upload_time_iso_8601": "2023-02-05T17:06:20.632611Z",
"url": "https://files.pythonhosted.org/packages/9e/fe/32bd864bcb604b0607924a4cf618ed267a0ef21ac9c3e255109256046e1f/multiprocessing_logging-0.3.4-py2.py3-none-any.whl",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-02-05 17:06:20",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "jruere",
"github_project": "multiprocessing-logging",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "multiprocessing-logging"
}
JSON
