# offlog 0.1
`offlog` is a non-blocking file-append service for Unix systems. It consists of a server
program and Python client library that interact via a Unix domain socket, allowing
applications to offload file appends to the server and mitigating the risk of
application latency caused by blocking file I/O. It has a focus on performance,
specifically on minimising overhead for the client.
Both client and server are written in Python, but the protocol is not Python-specific so
other clients could be written as well. This package has no dependencies outside the
Python standard library.
## install
Install to your virtualenv or other Python environment using `pip`
```bash
pip install offlog
```
Or build from source
```bash
git clone https://github.com/chrisjbillington/offlog
cd offlog
pip install setuptools wheel
pip install .
```
Or build a wheel
```
git clone https://github.com/chrisjbillington/offlog
cd offlog
pip install setuptools wheel build
python -m build .
```
## example
Example:
```python
# example client code example.py
from offlog import Logger
logger = Logger("my app", '/tmp/myapp.log')
logger.info("Hello world, params are %s %d", "foo", 7)
try:
1/0
except Exception:
logger.exception("Got an error")
```
Start the the server in one terminal, and then run the client in another. You'll see the
following:
```shell
$ python -m offlog
[2023-07-03 10:12:21.895 offlog INFO] This is offlog server
[2023-07-03 10:12:21.895 offlog INFO] Listening on socket /tmp/offlog.sock
[2023-07-03 10:12:54.796 offlog INFO] Client 0 connected
[2023-07-03 10:12:54.796 offlog INFO] Client 0 access confirmed for /tmp/myapp.log
[2023-07-03 10:12:54.796 offlog INFO] New client 0 (total: 1) for /tmp/myapp.log
[2023-07-03 10:12:54.799 offlog INFO] Client 0 disconnected
[2023-07-03 10:12:54.799 offlog INFO] Client 0 done (remaining: 0) with /tmp/myapp.log
[2023-07-03 10:12:54.799 offlog INFO] Closed /tmp/myapp.log
```
```shell
$ python example.py
[2023-07-03 10:12:54.796 my app INFO] Hello world, params are foo 7
[2023-07-03 10:12:54.796 my app ERROR] Got an error
Traceback (most recent call last):
File "/path/to/example.py", line 6, in <module>
1/0
~^~
ZeroDivisionError: division by zero
```
And the contents of `/tmp/myapp.log` will be:
```shell
$ cat /tmp/myapp.log
[2023-07-03 10:12:54.796 my app INFO] Hello world, params are foo 7
[2023-07-03 10:12:54.796 my app ERROR] Got an error
Traceback (most recent call last):
File "/path/to/example.py", line 6, in <module>
1/0
~^~
ZeroDivisionError: division by zero
```
## Details and notes
The `Logger()` class by default outputs `DEBUG` and higher to file, `INFO` to stdout,
and `ERROR` and higher to stderr. These are configurable. Formatting and exception
options are similar to the Python standard library.
The focus of this library is more on low-overhead file appends than the `Logger()`
class, which is very basic. It is easily subclassable if you want to change some of its
behaviour, or you could feasibly plug the `FileProxy()` object that sends data to the
`offlog` server into some other logging framework. You may also use the `FileProxy`
object directly to proxy file appends, including of binary data, to a file.
Despite being offloaded to another process, all file appends are still reliable, in the
sense that all data will eventually be written if the server and client shut down in
normal ways. This includes if the client encounters a Python exception, but does not
include if the client, e.g. segfaults or is terminated via SIGTERM and Python's default
SIGTERM handler (which exits immediately without doing any normal Python cleanup).
The server does flush files after every write, so when things are running normally, logs
are written in real-time. The only reason they might be delayed is when the system is
under load and messages are getting backed up - in which case clients will not retry
sending queued data until the next time they have new data to send, or until program
shutdown, whichever comes first.
There is no facility for global look up of loggers. Pass them around yourself. They are
not thread-safe, but if you instantiate one per thread pointing to the same file, they
will both talk to server which will interleave their appends. But their data won't be
guaranteed to be in the same order or even have the same boundaries as they were fired
off in your application.
Instantiating a FileProxy is blocking - the server will verify it can open the file and
we wait for it to confirm. Writing data is not ever blocking, and for now data that
fails to send because the server is too busy will just be locally buffered indefinitely
(you can still run out of memory). Unsent data will be retried before sending new data
on subsequent calls to `write()`. At shutdown, all unsent data will attempted to be
flushed to the server, this is also blocking.
If the server is shut down or encounters an exception writing data on the client's
behalf, it will terminate the connection with the client. Upon the next attempt to write
the client will get a BrokenPipeError and read the exception message from the server,
raising it as an exception in the client code.
## Client documentation
### Logger
```python
offlog.Logger(
name=None,
filepath=None,
file_level=DEBUG,
stdout_level=INFO,
stderr_level=WARNING,
local_file=False,
offlog_socket_path='/tmp/offlog.sock',
offlog_timeout=5000
)
```
Logging object to log to file, `stdout` and `stderr`, with optional proxying of file
writes via a offlog server.
Records with level `file_level` and above will be written to the given file. Records
with level `stdout_level` and above, up to but not including `stderr_level` will be
written to stdout, or with no upper limit if `stdout_level` is None. Records with level
`stderr_level` and above will be written to stderr. Any of these can be set to None to
disaable writing to that stream. `filepath=None` will also disable file logging.
if `local_file` is `True`, then an ordinary file will be opened for writing. Otherwise
`offlog_socket_path` is used to connect to a running offlog server, which will open the
file for us, writes will be proxied through it. Any blocking operations communicating
with the server (such as the initial file open, and flushing data at shutdown) will be
subject to a communications timeout of `offlog_timeout` in milliseconds, default 5000.
UTF-8 encoding is assumed throughout.
```python
offlog.Logger.close()
```
Close the file. Possibly blocking. Idempotent.
Logging methods work similarly to the Python standard library:
```python
offlog.Logger.log(level, msg, *args, exc_info=False):
offlog.Logger.debug(msg, *args, exc_info=False):
offlog.Logger.info(msg, *args, exc_info=False):
offlog.Logger.warning(msg, *args, exc_info=False):
offlog.Logger.error(msg, *args, exc_info=False):
offlog.Logger.exception(msg, *args, exc_info=True)
offlog.Logger.critical(msg, *args, exc_info=False)
```
### ProxyFile
```python
offlog.ProxyFile(
filepath,
sock_path='/tmp/offlog.sock',
timeout=5000,
)
```
Object to proxy appending file writes via a running offlog server.
```python
offlog.ProxyFile.write( data):
```
Send as much as we can without blocking. If sending would block, queue unsent data for
later. On `BrokenPipeError`, check if the server sent us an error and raise it if so.
This method will always attempt to re-send previously-queued data before attempting to
send new data.
```python
offlog.ProxyFile.close():
```
Close the socket. Attempt to send all queued unsent data to the server and
cleanly close the connection to it, raising exceptions if anything goes wrong
## Server documentation
```shell
$ python -m offlog -h
usage: python -m offlog [-h] [-n] [-s SOCKET_FILE] [-l LOGFILE]
offlog file-append service
options:
-h, --help show this help message and exit
-n, --notify Notify systemd when the server has started up and is
listening.
-s SOCKET_FILE, --socket-file SOCKET_FILE
Path of the Unix socket the server binds to. Default:
/tmp/offlog.sock
-l LOGFILE, --logfile LOGFILE
Path of the (optional) log file for the offlog server
itself.
```
You may wish to start the service as a systemd service. Here is an example unit file:
```ini
# offlog.service
[Unit]
Description=offlog file-append service
After=network.target
[Service]
Type=notify
ExecStart=/path/to/your/.venv/bin/python -u -m offlog --notify
Restart=always
RestartSec=5s
User=<user_to_run_as>
[Install]
WantedBy=multi-user.target
```
Note: the server will run with permissions of the specific user, which means all file
operations will be performed as that user. The server's unix socket file will be created
with default access permissions of that user, and thus only users will permission to
access that socket will be able to communicate with the server. This means by default,
clients will not be able to access an offlog server running as root in order to write to
files they would not have otherwise had permission to write to.
To avoid race conditions in systemd starting other services that depend on the offlog
server, ensure you use the `--notify` flag to the offlog server, which will notify
systemd when the server's socket is bound and it is ready to accept clients. Mark other
units that require the offlog server by adding `Requires` and `After` lines to the
`[Unit]` section of their service files:
```ini
[Unit]
Description=Some other service that needs an offlog server
...
Requires=offlog.service
After=offlog.service
```
The server shuts down cleanly upon `SIGINT` (i.e. pressing ctrl-C when run from a
terminal) or `SIGTERM` (i.e. what systemd will send by default if you run `systemctl
stop`, or at shutdown). It will close all client connections, and write any data clients
have already sent to files. Running clients will get an exception upon their next
attempt to write data.
protocol
========
TODO - document the protocol
Raw data
{
"_id": null,
"home_page": "https://github.com/chrisjbillington/offlog",
"name": "offlog",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.7",
"maintainer_email": null,
"keywords": "logging unix",
"author": "Christopher Billington",
"author_email": "chrisjbillington@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/ae/4b/24f6ed3f7b12c523945f410b01f532fa9bc2505c17295f5fe5f5b4870db1/offlog-0.3.1.tar.gz",
"platform": null,
"description": "# offlog 0.1\n\n\n`offlog` is a non-blocking file-append service for Unix systems. It consists of a server\nprogram and Python client library that interact via a Unix domain socket, allowing\napplications to offload file appends to the server and mitigating the risk of\napplication latency caused by blocking file I/O. It has a focus on performance,\nspecifically on minimising overhead for the client. \n\nBoth client and server are written in Python, but the protocol is not Python-specific so\nother clients could be written as well. This package has no dependencies outside the\nPython standard library.\n\n## install\n\nInstall to your virtualenv or other Python environment using `pip`\n```bash\npip install offlog\n```\n\nOr build from source\n```bash\ngit clone https://github.com/chrisjbillington/offlog\ncd offlog\npip install setuptools wheel\npip install .\n```\n\nOr build a wheel\n```\ngit clone https://github.com/chrisjbillington/offlog\ncd offlog\npip install setuptools wheel build\npython -m build .\n```\n\n## example\n\nExample:\n```python\n# example client code example.py\nfrom offlog import Logger\nlogger = Logger(\"my app\", '/tmp/myapp.log')\nlogger.info(\"Hello world, params are %s %d\", \"foo\", 7)\ntry:\n 1/0\nexcept Exception:\n logger.exception(\"Got an error\")\n```\n\nStart the the server in one terminal, and then run the client in another. You'll see the\nfollowing:\n\n```shell\n$ python -m offlog\n[2023-07-03 10:12:21.895 offlog INFO] This is offlog server\n[2023-07-03 10:12:21.895 offlog INFO] Listening on socket /tmp/offlog.sock\n[2023-07-03 10:12:54.796 offlog INFO] Client 0 connected\n[2023-07-03 10:12:54.796 offlog INFO] Client 0 access confirmed for /tmp/myapp.log\n[2023-07-03 10:12:54.796 offlog INFO] New client 0 (total: 1) for /tmp/myapp.log\n[2023-07-03 10:12:54.799 offlog INFO] Client 0 disconnected\n[2023-07-03 10:12:54.799 offlog INFO] Client 0 done (remaining: 0) with /tmp/myapp.log\n[2023-07-03 10:12:54.799 offlog INFO] Closed /tmp/myapp.log\n```\n\n```shell\n$ python example.py\n[2023-07-03 10:12:54.796 my app INFO] Hello world, params are foo 7\n[2023-07-03 10:12:54.796 my app ERROR] Got an error\nTraceback (most recent call last):\n File \"/path/to/example.py\", line 6, in <module>\n 1/0\n ~^~\nZeroDivisionError: division by zero\n```\n\nAnd the contents of `/tmp/myapp.log` will be:\n```shell\n$ cat /tmp/myapp.log\n[2023-07-03 10:12:54.796 my app INFO] Hello world, params are foo 7\n[2023-07-03 10:12:54.796 my app ERROR] Got an error\nTraceback (most recent call last):\n File \"/path/to/example.py\", line 6, in <module>\n 1/0\n ~^~\nZeroDivisionError: division by zero\n```\n\n## Details and notes\n\n\nThe `Logger()` class by default outputs `DEBUG` and higher to file, `INFO` to stdout,\nand `ERROR` and higher to stderr. These are configurable. Formatting and exception\noptions are similar to the Python standard library.\n\nThe focus of this library is more on low-overhead file appends than the `Logger()`\nclass, which is very basic. It is easily subclassable if you want to change some of its\nbehaviour, or you could feasibly plug the `FileProxy()` object that sends data to the\n`offlog` server into some other logging framework. You may also use the `FileProxy`\nobject directly to proxy file appends, including of binary data, to a file.\n\nDespite being offloaded to another process, all file appends are still reliable, in the\nsense that all data will eventually be written if the server and client shut down in\nnormal ways. This includes if the client encounters a Python exception, but does not\ninclude if the client, e.g. segfaults or is terminated via SIGTERM and Python's default\nSIGTERM handler (which exits immediately without doing any normal Python cleanup).\n\nThe server does flush files after every write, so when things are running normally, logs\nare written in real-time. The only reason they might be delayed is when the system is\nunder load and messages are getting backed up - in which case clients will not retry\nsending queued data until the next time they have new data to send, or until program\nshutdown, whichever comes first.\n\nThere is no facility for global look up of loggers. Pass them around yourself. They are\nnot thread-safe, but if you instantiate one per thread pointing to the same file, they\nwill both talk to server which will interleave their appends. But their data won't be\nguaranteed to be in the same order or even have the same boundaries as they were fired\noff in your application.\n\nInstantiating a FileProxy is blocking - the server will verify it can open the file and\nwe wait for it to confirm. Writing data is not ever blocking, and for now data that\nfails to send because the server is too busy will just be locally buffered indefinitely\n(you can still run out of memory). Unsent data will be retried before sending new data\non subsequent calls to `write()`. At shutdown, all unsent data will attempted to be\nflushed to the server, this is also blocking.\n\nIf the server is shut down or encounters an exception writing data on the client's\nbehalf, it will terminate the connection with the client. Upon the next attempt to write\nthe client will get a BrokenPipeError and read the exception message from the server,\nraising it as an exception in the client code.\n\n## Client documentation\n\n### Logger\n\n```python\nofflog.Logger(\n name=None,\n filepath=None,\n file_level=DEBUG,\n stdout_level=INFO,\n stderr_level=WARNING,\n local_file=False,\n offlog_socket_path='/tmp/offlog.sock',\n offlog_timeout=5000\n)\n```\n\nLogging object to log to file, `stdout` and `stderr`, with optional proxying of file\nwrites via a offlog server.\n\nRecords with level `file_level` and above will be written to the given file. Records\nwith level `stdout_level` and above, up to but not including `stderr_level` will be\nwritten to stdout, or with no upper limit if `stdout_level` is None. Records with level\n`stderr_level` and above will be written to stderr. Any of these can be set to None to\ndisaable writing to that stream. `filepath=None` will also disable file logging.\n\nif `local_file` is `True`, then an ordinary file will be opened for writing. Otherwise\n`offlog_socket_path` is used to connect to a running offlog server, which will open the\nfile for us, writes will be proxied through it. Any blocking operations communicating\nwith the server (such as the initial file open, and flushing data at shutdown) will be\nsubject to a communications timeout of `offlog_timeout` in milliseconds, default 5000.\n\nUTF-8 encoding is assumed throughout.\n\n```python\nofflog.Logger.close()\n```\nClose the file. Possibly blocking. Idempotent.\n\nLogging methods work similarly to the Python standard library:\n\n```python\nofflog.Logger.log(level, msg, *args, exc_info=False):\n\nofflog.Logger.debug(msg, *args, exc_info=False):\n\nofflog.Logger.info(msg, *args, exc_info=False):\n\nofflog.Logger.warning(msg, *args, exc_info=False):\n\nofflog.Logger.error(msg, *args, exc_info=False):\n\nofflog.Logger.exception(msg, *args, exc_info=True)\n\nofflog.Logger.critical(msg, *args, exc_info=False)\n```\n\n\n### ProxyFile\n\n```python\nofflog.ProxyFile(\n filepath,\n sock_path='/tmp/offlog.sock',\n timeout=5000,\n)\n```\n\nObject to proxy appending file writes via a running offlog server.\n\n```python\nofflog.ProxyFile.write( data):\n```\nSend as much as we can without blocking. If sending would block, queue unsent data for\nlater. On `BrokenPipeError`, check if the server sent us an error and raise it if so.\nThis method will always attempt to re-send previously-queued data before attempting to\nsend new data.\n\n```python\nofflog.ProxyFile.close():\n```\nClose the socket. Attempt to send all queued unsent data to the server and\ncleanly close the connection to it, raising exceptions if anything goes wrong\n\n## Server documentation\n\n```shell\n$ python -m offlog -h\nusage: python -m offlog [-h] [-n] [-s SOCKET_FILE] [-l LOGFILE]\n\nofflog file-append service\n\noptions:\n -h, --help show this help message and exit\n -n, --notify Notify systemd when the server has started up and is\n listening.\n -s SOCKET_FILE, --socket-file SOCKET_FILE\n Path of the Unix socket the server binds to. Default:\n /tmp/offlog.sock\n -l LOGFILE, --logfile LOGFILE\n Path of the (optional) log file for the offlog server\n itself.\n```\n\nYou may wish to start the service as a systemd service. Here is an example unit file:\n```ini\n# offlog.service\n[Unit]\nDescription=offlog file-append service\nAfter=network.target\n\n[Service]\nType=notify\nExecStart=/path/to/your/.venv/bin/python -u -m offlog --notify\nRestart=always\nRestartSec=5s\nUser=<user_to_run_as>\n\n[Install]\nWantedBy=multi-user.target\n```\n\nNote: the server will run with permissions of the specific user, which means all file\noperations will be performed as that user. The server's unix socket file will be created\nwith default access permissions of that user, and thus only users will permission to\naccess that socket will be able to communicate with the server. This means by default,\nclients will not be able to access an offlog server running as root in order to write to\nfiles they would not have otherwise had permission to write to.\n\nTo avoid race conditions in systemd starting other services that depend on the offlog\nserver, ensure you use the `--notify` flag to the offlog server, which will notify\nsystemd when the server's socket is bound and it is ready to accept clients. Mark other\nunits that require the offlog server by adding `Requires` and `After` lines to the\n`[Unit]` section of their service files:\n\n```ini\n[Unit]\nDescription=Some other service that needs an offlog server\n...\nRequires=offlog.service\nAfter=offlog.service\n```\n\nThe server shuts down cleanly upon `SIGINT` (i.e. pressing ctrl-C when run from a\nterminal) or `SIGTERM` (i.e. what systemd will send by default if you run `systemctl\nstop`, or at shutdown). It will close all client connections, and write any data clients\nhave already sent to files. Running clients will get an exception upon their next\nattempt to write data.\n\nprotocol\n========\n\nTODO - document the protocol\n\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "A non-blocking file-append service for Unix systems",
"version": "0.3.1",
"project_urls": {
"Download": "https://pypi.org/project/offlog/",
"Homepage": "https://github.com/chrisjbillington/offlog",
"Source Code": "https://github.com/chrisjbillington/offlog",
"Tracker": "https://github.com/chrisjbillington/offlog/issues"
},
"split_keywords": [
"logging",
"unix"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "e46e11c981be92e324e1e83149a717ee45df61c7a9412dbb57eab0f1095cafbb",
"md5": "89f839c27b58d53877e0d95d5a427e4b",
"sha256": "a7f0bc5c4eedd99629ac26c9c412674a7104c72271a5eba08076af93801511da"
},
"downloads": -1,
"filename": "offlog-0.3.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "89f839c27b58d53877e0d95d5a427e4b",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7",
"size": 16216,
"upload_time": "2024-12-20T00:21:46",
"upload_time_iso_8601": "2024-12-20T00:21:46.357386Z",
"url": "https://files.pythonhosted.org/packages/e4/6e/11c981be92e324e1e83149a717ee45df61c7a9412dbb57eab0f1095cafbb/offlog-0.3.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "ae4b24f6ed3f7b12c523945f410b01f532fa9bc2505c17295f5fe5f5b4870db1",
"md5": "9a78b532c4e5b5f4a070a201b760aba3",
"sha256": "d9f7732d215ee6b226d1bf7761e0de69606b739345305be7e4b771fb0af1312a"
},
"downloads": -1,
"filename": "offlog-0.3.1.tar.gz",
"has_sig": false,
"md5_digest": "9a78b532c4e5b5f4a070a201b760aba3",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7",
"size": 16917,
"upload_time": "2024-12-20T00:21:50",
"upload_time_iso_8601": "2024-12-20T00:21:50.358320Z",
"url": "https://files.pythonhosted.org/packages/ae/4b/24f6ed3f7b12c523945f410b01f532fa9bc2505c17295f5fe5f5b4870db1/offlog-0.3.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-12-20 00:21:50",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "chrisjbillington",
"github_project": "offlog",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "offlog"
}
JSON
