gocept.runner
=============
Create stand alone programs with full a Zope3 runtime environment.
These programs can be used either as one time run scripts or can be
automatically called after a defined period of time.
.. contents::
Creating runners
================
The ``gocept.runner`` package allows it to *easily* create small, long running
scripts which interact with the ZODB. The scripts have the full component
architecture set up when they're run.
Runners are defined with the appmain decorator:
>>> import logging
>>> import gocept.runner
>>> work_count = 0
>>> @gocept.runner.appmain(ticks=0.1)
... def worker():
... import zope.app.appsetup.product
... log = logging.getLogger('test')
... log.info("Working")
... log.info(sorted(
... zope.app.appsetup.product.getProductConfiguration('test').items()))
... global work_count
... work_count += 1
... if work_count >= 3:
... return gocept.runner.Exit
The decorated worker takes two arguments now:
1. The name of an object in the root which will be set as site or None for the
root.
2. The path to a configuration file (zope.conf)
Create a simple zope.conf:
>>> import os.path
>>> import tempfile
>>> zodb_path = tempfile.mkdtemp()
>>> site_zcml = os.path.join(
... os.path.dirname(__file__), 'ftesting.zcml')
>>> fd, zope_conf = tempfile.mkstemp()
>>> zope_conf_file = os.fdopen(fd, 'w')
>>> _ = zope_conf_file.write('''\
... site-definition %s
... <zodb>
... <filestorage>
... path %s/Data.fs
... </filestorage>
... </zodb>
... <product-config test>
... foo bar
... test-principal zope.mgr
... </product-config>
... <eventlog>
... <logfile>
... formatter zope.exceptions.log.Formatter
... path STDOUT
... </logfile>
... </eventlog>
... ''' % (site_zcml, zodb_path))
>>> zope_conf_file.close()
So call the worker:
>>> worker(None, zope_conf)
------
... INFO test Working
------
... INFO test [('foo', 'bar'), ('test-principal', 'zope.mgr')]
------
... INFO test Working
------
... INFO test [('foo', 'bar'), ('test-principal', 'zope.mgr')]
------
... INFO test Working
------
... INFO test [('foo', 'bar'), ('test-principal', 'zope.mgr')]
Signals
-------
The worker-procss can be terminated by SIGTERM and SIGHUP in a sane way. Write
a script to a temporary file:
>>> import sys
>>> runner_path = os.path.abspath(
... os.path.join(os.path.dirname(__file__), '..', '..'))
>>> fd, script_name = tempfile.mkstemp(suffix='.py')
>>> exchange_fd, exchange_file_name = tempfile.mkstemp()
>>> script = os.fdopen(fd, 'w')
>>> _ = script.write("""\
... import sys
... sys.path[0:0] = %s
... sys.path.insert(0, '%s')
... import gocept.runner
...
... f = open('%s', 'w')
...
... @gocept.runner.appmain(ticks=0.1)
... def worker():
... f.write("Working.\\n")
... f.flush()
...
... worker(None, '%s')
... """ % (sys.path, runner_path, exchange_file_name, zope_conf))
>>> script.close()
Call the script and wait for it to produce some output:
>>> import signal
>>> import subprocess
>>> import time
>>> exchange = os.fdopen(exchange_fd, 'r+')
>>> proc = subprocess.Popen(
... [sys.executable, script_name],
... stdout=subprocess.PIPE,
... text=True)
>>> while not exchange.read():
... time.sleep(0.1)
... _ = exchange.seek(0, 0)
>>> _ = exchange.seek(0, 0)
>>> print(exchange.read())
Working...
Okay, now kill it:
>>> os.kill(proc.pid, signal.SIGTERM)
Wait for the process to really finish and get the output. The runner logs that
it was terminated:
>>> stdout, stderr = proc.communicate()
>>> print(stdout)
------...INFO gocept.runner.runner Received signal 15, terminating...
This also works with SIGHUP:
>>> _ = exchange.truncate(0)
>>> proc = subprocess.Popen(
... [sys.executable, script_name],
... stdout=subprocess.PIPE,
... text=True)
>>> while not exchange.read():
... time.sleep(0.1)
... _ = exchange.seek(0, 0)
>>> _ = exchange.seek(0, 0)
>>> print(exchange.read())
Working...
Okay, now kill it:
>>> os.kill(proc.pid, signal.SIGHUP)
>>> stdout, stderr = proc.communicate()
>>> print(stdout)
------...INFO gocept.runner.runner Received signal 1, terminating...
Clean up:
>>> exchange.close()
>>> os.remove(script_name)
>>> os.remove(exchange_file_name)
Setting the principal
---------------------
It is also prossible to create a main loop which runs in an interaction:
>>> def get_principal():
... return 'zope.mgr'
>>> import zope.security.management
>>> work_count = 0
>>> def interaction_worker():
... global work_count
... work_count += 1
... if work_count >= 3:
... raise SystemExit(1)
... log = logging.getLogger('test')
... interaction = zope.security.management.getInteraction()
... principal = interaction.participations[0].principal
... log.info("Working as %s" % principal.id)
>>> worker = gocept.runner.appmain(ticks=0.1, principal=get_principal)(
... interaction_worker)
Call the worker now:
>>> worker(None, zope_conf)
------
... INFO test Working as zope.mgr
------
... INFO test Working as zope.mgr
After the worker is run there is no interaction:
>>> zope.security.management.queryInteraction() is None
True
It's quite common to read the principal from zope.conf. Therefore there is a
helper which makes this task easier:
>>> work_count = 0
>>> worker = gocept.runner.appmain(
... ticks=0.1,
... principal=gocept.runner.from_config('test', 'test-principal'))(
... interaction_worker)
>>> worker(None, zope_conf)
------
... INFO test Working as zope.mgr
------
... INFO test Working as zope.mgr
Subsites
--------
It is possible to directly work on sites inside the root. The site must already
exist of course, otherwise there will be an error:
>>> worker('a-site', zope_conf)
Traceback (most recent call last):
...
KeyError: 'a-site'
Clean up:
>>> import shutil
>>> shutil.rmtree(zodb_path)
>>> os.remove(zope_conf)
Run once commands
=================
It is often needed to run a command once (or via cron) with the full component
architecture. Usually ``zopectl run`` is used for this.
>>> import logging
>>> import gocept.runner
>>> import zope.component.hooks
>>> @gocept.runner.once()
... def worker():
... log = logging.getLogger('test')
... log.info("Working.")
... site = zope.component.hooks.getSite()
... if hasattr(site, 'store'):
... log.info("Having attribute store.")
... site.store = True
Define a Zope environment:
>>> import os.path
>>> import tempfile
>>> zodb_path = tempfile.mkdtemp()
>>> site_zcml = os.path.join(
... os.path.dirname(__file__), 'ftesting.zcml')
>>> fd, zope_conf = tempfile.mkstemp()
>>> zope_conf_file = os.fdopen(fd, 'w')
>>> _ = zope_conf_file.write('''\
... site-definition %s
... <zodb>
... <filestorage>
... path %s/Data.fs
... </filestorage>
... </zodb>
... <product-config test>
... foo bar
... </product-config>
... <eventlog>
... <logfile>
... formatter zope.exceptions.log.Formatter
... path STDOUT
... </logfile>
... </eventlog>
... ''' % (site_zcml, zodb_path))
>>> zope_conf_file.close()
So call the worker for the first time. It will be terminated after one call:
>>> worker(None, zope_conf)
------
... INFO test Working.
Calling it a second time indicates that a property was changed in the first
run:
>>> worker(None, zope_conf)
------
... INFO test Working.
------
... INFO test Having attribute store.
...
Runner details
==============
Main loop
---------
The main loop loops until it encounters a KeyboardInterrupt or a SystemExit
exception and calls the worker once a second.
Define a worker function which exits when it is called the 3rd time:
>>> work_count = 0
>>> def worker():
... print("Working")
... global work_count
... work_count += 1
... if work_count >= 3:
... raise SystemExit(1)
Call the main loop:
>>> import gocept.runner.runner
>>> gocept.runner.runner.MainLoop(getRootFolder(), 0.1, worker)()
Working
Working
Working
>>> work_count
3
During the loop the site is set:
>>> import zope.component.hooks
>>> zope.component.hooks.getSite() is None
True
>>> def worker():
... print(zope.component.hooks.getSite())
... raise SystemExit(1)
>>> gocept.runner.runner.MainLoop(getRootFolder(), 0.1, worker)()
<zope.site.folder.Folder object at 0x...>
After the loop, no site is set again:
>>> zope.component.hooks.getSite() is None
True
When the worker passes without error a transaction is committed:
>>> work_count = 0
>>> def worker():
... print("Working")
... global work_count
... work_count += 1
... if work_count >= 2:
... raise SystemExit(1)
... site = zope.component.hooks.getSite()
... site.worker_done = 1
>>> gocept.runner.runner.MainLoop(getRootFolder(), 0.1, worker)()
Working
Working
We have set the attribute ``worker_done`` now:
>>> getRootFolder().worker_done
1
When the worker produces an error, the transaction is aborted:
>>> work_count = 0
>>> def worker():
... global work_count
... work_count += 1
... print("Working")
... site = zope.component.hooks.getSite()
... site.worker_done += 1
... if work_count < 3:
... raise ValueError('hurz')
... raise SystemExit(1)
>>> gocept.runner.runner.MainLoop(getRootFolder(), 0.1, worker)()
Working
Working
Working
We still have the attribute ``worker_done`` set to 1:b
>>> getRootFolder().worker_done
1
Controlling sleep time
----------------------
The worker function can control the sleep time.
Register a log handler so we can observe this:
>>> from io import StringIO
>>> import logging
>>> log = StringIO()
>>> log_handler = logging.StreamHandler(log)
>>> logging.root.addHandler(log_handler)
>>> old_log_level = logging.root.level
>>> logging.root.setLevel(logging.DEBUG)
>>> work_count = 0
>>> def worker():
... global work_count
... work_count += 1
... new_sleep = work_count * 0.1
... if work_count == 3:
... print("Will sleep default")
... return None
... if work_count > 3:
... raise SystemExit(1)
... print("Will sleep %s" % new_sleep)
... return new_sleep
>>> gocept.runner.runner.MainLoop(getRootFolder(), 0.15, worker)()
Will sleep 0.1
Will sleep 0.2
Will sleep default
The real sleep values are in the log:
>>> print(log.getvalue())
new transaction
commit
Sleeping 0.1 seconds
new transaction
commit
Sleeping 0.2 seconds
new transaction
commit
Sleeping 0.15 seconds
new transaction
abort...
When an error occours within the worker, the default sleep time will be used:
>>> _ = log.seek(0)
>>> _ = log.truncate()
>>> work_count = 0
>>> def worker():
... global work_count
... work_count += 1
... if work_count == 1:
... new_sleep = 0.1
... elif work_count == 2:
... print("Failing")
... raise Exception("Fail!")
... elif work_count == 3:
... print("Will sleep default")
... return None
... elif work_count > 3:
... return gocept.runner.Exit
... print("Will sleep %s" % new_sleep)
... return new_sleep
>>> gocept.runner.runner.MainLoop(getRootFolder(), 0.15, worker)()
Will sleep 0.1
Failing
Will sleep default
The real sleep values are in the log:
>>> print(log.getvalue())
new transaction
commit
Sleeping 0.1 seconds
new transaction
Error in worker: Exception('Fail!'...)
Traceback (most recent call last):
...
Exception: Fail!
abort
Sleeping 0.15 seconds
new transaction
commit
Sleeping 0.15 seconds
new transaction
commit...
Restore old log handler:
>>> logging.root.removeHandler(log_handler)
>>> log.close()
>>> logging.root.setLevel(old_log_level)
Multiple commit helper
======================
Sometimes you have several actions and want to commit in between each step.
``gocept.runner`` provides a decorator ``transaction_per_item`` that simplifies
writing this down, for example::
@gocept.runner.transaction_per_item
def multiple_steps():
for item in source:
yield lambda: process(item)
This will call each ``process(item)`` in turn and commit after each. If an
exception is raised by a step, the transaction is aborted and the loop
continues with the next step. ZODB ConflictErrors are ignored likewise.
Changes
=======
3.0 (2023-07-20)
----------------
- Drop support for Python 2.7.
- Add support for Python 3.9, 3.10, 3.11.
2.2 (2020-09-14)
----------------
- Get rid of `ZODB3` dependency.
2.1 (2019-12-02)
----------------
- Add support for Python 3.7 and 3.8.
- Migrate to Github.
2.0 (2018-03-14)
----------------
- Backwards incompatible change: No longer depend on ``zope.app.server``,
and no longer superfluously use its `ZConfig` schema, which superfluously
requires an `<accesslog>` section that does not make any sense in our context
anyway. So `<accesslog>` is no longer allowed in the ZConfing config file
used by this package.
1.0 (2016-01-12)
----------------
- No longer depend on ``zope.app.component``.
0.7.1 (2015-01-28)
------------------
- Fix faulty release 0.7.0.
0.7.0 (2015-01-28)
------------------
- An error message is correctly logged when the worker produces an exception
which cannot be represented in us-ascii. This used to cause an exception in
the logging module. The cause of those exceptions where quite hard to track.
- Removed dependency on zope.app.security which was neither declared nor
necessary.
- Moved project home to <https://bitbucket.org/gocept/gocept.runner/>.
0.6.0 (2011-12-01)
------------------
- Added transaction_per_item decorator.
- Use stdlib's doctest instead of zope.testing.
0.5.3 (2010-04-14)
------------------
- Use log.error/log.warning instead of log.exception. Conflict errors are
logged as warning now, because they are not really a problem.
0.5.2 (2010-04-14)
------------------
- Convert logged exceptions to str.
0.5.1 (2009-10-13)
------------------
- Declared dependencies correctly.
0.5 (2009-09-21)
++++++++++++++++
- Does no longer use ``zope.app.twisted`` but ``zope.app.server`` instead.
0.4 (2009-09-03)
++++++++++++++++
- The principal set by appmain/once can be computed by a function now.
0.3.2 (2009-05-21)
------------------
- Fixed handling of subsites in appmain.
0.3.1 (2009-05-21)
------------------
- Declared namespace package.
0.3 (2009-04-15)
++++++++++++++++
- When a worker fails the default sleep time (instead of the last one) will be
used.
0.2 (2009-04-09)
++++++++++++++++
- Added a clean way to exit the runner (by returning gocept.runner.Exit).
0.1 (2009-04-07)
++++++++++++++++
- first public release
Raw data
{
"_id": null,
"home_page": "https://github.com/gocept/gocept.runner",
"name": "gocept.runner",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.7",
"maintainer_email": "",
"keywords": "zope3 mainloop",
"author": "gocept gmbh & co. kg",
"author_email": "mail@gocept.com",
"download_url": "https://files.pythonhosted.org/packages/28/af/a189ca7ed1171da022543aa8cbc529e6af8b516ccee55e87cd5f8c94b768/gocept.runner-3.0.tar.gz",
"platform": null,
"description": "gocept.runner\n=============\n\nCreate stand alone programs with full a Zope3 runtime environment.\nThese programs can be used either as one time run scripts or can be\nautomatically called after a defined period of time.\n\n.. contents::\n\n\nCreating runners\n================\n\nThe ``gocept.runner`` package allows it to *easily* create small, long running\nscripts which interact with the ZODB. The scripts have the full component\narchitecture set up when they're run.\n\nRunners are defined with the appmain decorator:\n\n>>> import logging\n>>> import gocept.runner\n>>> work_count = 0\n>>> @gocept.runner.appmain(ticks=0.1)\n... def worker():\n... import zope.app.appsetup.product\n... log = logging.getLogger('test')\n... log.info(\"Working\")\n... log.info(sorted(\n... zope.app.appsetup.product.getProductConfiguration('test').items()))\n... global work_count\n... work_count += 1\n... if work_count >= 3:\n... return gocept.runner.Exit\n\n\nThe decorated worker takes two arguments now:\n\n1. The name of an object in the root which will be set as site or None for the\n root.\n2. The path to a configuration file (zope.conf)\n\nCreate a simple zope.conf:\n\n>>> import os.path\n>>> import tempfile\n>>> zodb_path = tempfile.mkdtemp()\n>>> site_zcml = os.path.join(\n... os.path.dirname(__file__), 'ftesting.zcml')\n>>> fd, zope_conf = tempfile.mkstemp()\n>>> zope_conf_file = os.fdopen(fd, 'w')\n>>> _ = zope_conf_file.write('''\\\n... site-definition %s\n... <zodb>\n... <filestorage>\n... path %s/Data.fs\n... </filestorage>\n... </zodb>\n... <product-config test>\n... foo bar\n... test-principal zope.mgr\n... </product-config>\n... <eventlog>\n... <logfile>\n... formatter zope.exceptions.log.Formatter\n... path STDOUT\n... </logfile>\n... </eventlog>\n... ''' % (site_zcml, zodb_path))\n>>> zope_conf_file.close()\n\n\nSo call the worker:\n\n>>> worker(None, zope_conf)\n------\n... INFO test Working\n------\n... INFO test [('foo', 'bar'), ('test-principal', 'zope.mgr')]\n------\n... INFO test Working\n------\n... INFO test [('foo', 'bar'), ('test-principal', 'zope.mgr')]\n------\n... INFO test Working\n------\n... INFO test [('foo', 'bar'), ('test-principal', 'zope.mgr')]\n\n\n\nSignals\n-------\n\nThe worker-procss can be terminated by SIGTERM and SIGHUP in a sane way. Write\na script to a temporary file:\n\n>>> import sys\n>>> runner_path = os.path.abspath(\n... os.path.join(os.path.dirname(__file__), '..', '..'))\n>>> fd, script_name = tempfile.mkstemp(suffix='.py')\n>>> exchange_fd, exchange_file_name = tempfile.mkstemp()\n>>> script = os.fdopen(fd, 'w')\n>>> _ = script.write(\"\"\"\\\n... import sys\n... sys.path[0:0] = %s\n... sys.path.insert(0, '%s')\n... import gocept.runner\n...\n... f = open('%s', 'w')\n...\n... @gocept.runner.appmain(ticks=0.1)\n... def worker():\n... f.write(\"Working.\\\\n\")\n... f.flush()\n...\n... worker(None, '%s')\n... \"\"\" % (sys.path, runner_path, exchange_file_name, zope_conf))\n>>> script.close()\n\n\nCall the script and wait for it to produce some output:\n\n>>> import signal\n>>> import subprocess\n>>> import time\n>>> exchange = os.fdopen(exchange_fd, 'r+')\n>>> proc = subprocess.Popen(\n... [sys.executable, script_name],\n... stdout=subprocess.PIPE,\n... text=True)\n>>> while not exchange.read():\n... time.sleep(0.1)\n... _ = exchange.seek(0, 0)\n>>> _ = exchange.seek(0, 0)\n>>> print(exchange.read())\nWorking...\n\n\nOkay, now kill it:\n\n>>> os.kill(proc.pid, signal.SIGTERM)\n\nWait for the process to really finish and get the output. The runner logs that\nit was terminated:\n\n>>> stdout, stderr = proc.communicate()\n>>> print(stdout)\n------...INFO gocept.runner.runner Received signal 15, terminating...\n\n\nThis also works with SIGHUP:\n\n>>> _ = exchange.truncate(0)\n>>> proc = subprocess.Popen(\n... [sys.executable, script_name],\n... stdout=subprocess.PIPE,\n... text=True)\n>>> while not exchange.read():\n... time.sleep(0.1)\n... _ = exchange.seek(0, 0)\n>>> _ = exchange.seek(0, 0)\n>>> print(exchange.read())\nWorking...\n\nOkay, now kill it:\n\n>>> os.kill(proc.pid, signal.SIGHUP)\n>>> stdout, stderr = proc.communicate()\n>>> print(stdout)\n------...INFO gocept.runner.runner Received signal 1, terminating...\n\n\nClean up:\n>>> exchange.close()\n>>> os.remove(script_name)\n>>> os.remove(exchange_file_name)\n\n\nSetting the principal\n---------------------\n\nIt is also prossible to create a main loop which runs in an interaction:\n\n>>> def get_principal():\n... return 'zope.mgr'\n\n>>> import zope.security.management\n>>> work_count = 0\n>>> def interaction_worker():\n... global work_count\n... work_count += 1\n... if work_count >= 3:\n... raise SystemExit(1)\n... log = logging.getLogger('test')\n... interaction = zope.security.management.getInteraction()\n... principal = interaction.participations[0].principal\n... log.info(\"Working as %s\" % principal.id)\n>>> worker = gocept.runner.appmain(ticks=0.1, principal=get_principal)(\n... interaction_worker)\n\nCall the worker now:\n\n>>> worker(None, zope_conf)\n------\n... INFO test Working as zope.mgr\n------\n... INFO test Working as zope.mgr\n\n\nAfter the worker is run there is no interaction:\n\n>>> zope.security.management.queryInteraction() is None\nTrue\n\nIt's quite common to read the principal from zope.conf. Therefore there is a\nhelper which makes this task easier:\n\n>>> work_count = 0\n>>> worker = gocept.runner.appmain(\n... ticks=0.1,\n... principal=gocept.runner.from_config('test', 'test-principal'))(\n... interaction_worker)\n>>> worker(None, zope_conf)\n------\n... INFO test Working as zope.mgr\n------\n... INFO test Working as zope.mgr\n\n\nSubsites\n--------\n\nIt is possible to directly work on sites inside the root. The site must already\nexist of course, otherwise there will be an error:\n\n>>> worker('a-site', zope_conf)\nTraceback (most recent call last):\n ...\nKeyError: 'a-site'\n\n\nClean up:\n\n>>> import shutil\n>>> shutil.rmtree(zodb_path)\n>>> os.remove(zope_conf)\n\n\nRun once commands\n=================\n\nIt is often needed to run a command once (or via cron) with the full component\narchitecture. Usually ``zopectl run`` is used for this.\n\n>>> import logging\n>>> import gocept.runner\n>>> import zope.component.hooks\n>>> @gocept.runner.once()\n... def worker():\n... log = logging.getLogger('test')\n... log.info(\"Working.\")\n... site = zope.component.hooks.getSite()\n... if hasattr(site, 'store'):\n... log.info(\"Having attribute store.\")\n... site.store = True\n\n\nDefine a Zope environment:\n\n>>> import os.path\n>>> import tempfile\n>>> zodb_path = tempfile.mkdtemp()\n>>> site_zcml = os.path.join(\n... os.path.dirname(__file__), 'ftesting.zcml')\n>>> fd, zope_conf = tempfile.mkstemp()\n>>> zope_conf_file = os.fdopen(fd, 'w')\n>>> _ = zope_conf_file.write('''\\\n... site-definition %s\n... <zodb>\n... <filestorage>\n... path %s/Data.fs\n... </filestorage>\n... </zodb>\n... <product-config test>\n... foo bar\n... </product-config>\n... <eventlog>\n... <logfile>\n... formatter zope.exceptions.log.Formatter\n... path STDOUT\n... </logfile>\n... </eventlog>\n... ''' % (site_zcml, zodb_path))\n>>> zope_conf_file.close()\n\n\nSo call the worker for the first time. It will be terminated after one call:\n\n>>> worker(None, zope_conf)\n------\n... INFO test Working.\n\n\nCalling it a second time indicates that a property was changed in the first\nrun:\n\n>>> worker(None, zope_conf)\n------\n... INFO test Working.\n------\n... INFO test Having attribute store.\n...\n\n\nRunner details\n==============\n\nMain loop\n---------\n\nThe main loop loops until it encounters a KeyboardInterrupt or a SystemExit\nexception and calls the worker once a second.\n\nDefine a worker function which exits when it is called the 3rd time:\n\n>>> work_count = 0\n>>> def worker():\n... print(\"Working\")\n... global work_count\n... work_count += 1\n... if work_count >= 3:\n... raise SystemExit(1)\n\n\nCall the main loop:\n\n>>> import gocept.runner.runner\n>>> gocept.runner.runner.MainLoop(getRootFolder(), 0.1, worker)()\nWorking\nWorking\nWorking\n>>> work_count\n3\n\n\nDuring the loop the site is set:\n\n>>> import zope.component.hooks\n>>> zope.component.hooks.getSite() is None\nTrue\n>>> def worker():\n... print(zope.component.hooks.getSite())\n... raise SystemExit(1)\n>>> gocept.runner.runner.MainLoop(getRootFolder(), 0.1, worker)()\n<zope.site.folder.Folder object at 0x...>\n\n\n\nAfter the loop, no site is set again:\n\n>>> zope.component.hooks.getSite() is None\nTrue\n\n\nWhen the worker passes without error a transaction is committed:\n\n>>> work_count = 0\n>>> def worker():\n... print(\"Working\")\n... global work_count\n... work_count += 1\n... if work_count >= 2:\n... raise SystemExit(1)\n... site = zope.component.hooks.getSite()\n... site.worker_done = 1\n>>> gocept.runner.runner.MainLoop(getRootFolder(), 0.1, worker)()\nWorking\nWorking\n\nWe have set the attribute ``worker_done`` now:\n\n>>> getRootFolder().worker_done\n1\n\n\nWhen the worker produces an error, the transaction is aborted:\n\n>>> work_count = 0\n>>> def worker():\n... global work_count\n... work_count += 1\n... print(\"Working\")\n... site = zope.component.hooks.getSite()\n... site.worker_done += 1\n... if work_count < 3:\n... raise ValueError('hurz')\n... raise SystemExit(1)\n>>> gocept.runner.runner.MainLoop(getRootFolder(), 0.1, worker)()\nWorking\nWorking\nWorking\n\n\nWe still have the attribute ``worker_done`` set to 1:b\n\n>>> getRootFolder().worker_done\n1\n\n\nControlling sleep time\n----------------------\n\nThe worker function can control the sleep time.\n\nRegister a log handler so we can observe this:\n\n>>> from io import StringIO\n>>> import logging\n>>> log = StringIO()\n>>> log_handler = logging.StreamHandler(log)\n>>> logging.root.addHandler(log_handler)\n>>> old_log_level = logging.root.level\n>>> logging.root.setLevel(logging.DEBUG)\n\n\n>>> work_count = 0\n>>> def worker():\n... global work_count\n... work_count += 1\n... new_sleep = work_count * 0.1\n... if work_count == 3:\n... print(\"Will sleep default\")\n... return None\n... if work_count > 3:\n... raise SystemExit(1)\n... print(\"Will sleep %s\" % new_sleep)\n... return new_sleep\n>>> gocept.runner.runner.MainLoop(getRootFolder(), 0.15, worker)()\nWill sleep 0.1\nWill sleep 0.2\nWill sleep default\n\nThe real sleep values are in the log:\n\n>>> print(log.getvalue())\nnew transaction\ncommit\nSleeping 0.1 seconds\nnew transaction\ncommit\nSleeping 0.2 seconds\nnew transaction\ncommit\nSleeping 0.15 seconds\nnew transaction\nabort...\n\nWhen an error occours within the worker, the default sleep time will be used:\n\n>>> _ = log.seek(0)\n>>> _ = log.truncate()\n>>> work_count = 0\n>>> def worker():\n... global work_count\n... work_count += 1\n... if work_count == 1:\n... new_sleep = 0.1\n... elif work_count == 2:\n... print(\"Failing\")\n... raise Exception(\"Fail!\")\n... elif work_count == 3:\n... print(\"Will sleep default\")\n... return None\n... elif work_count > 3:\n... return gocept.runner.Exit\n... print(\"Will sleep %s\" % new_sleep)\n... return new_sleep\n>>> gocept.runner.runner.MainLoop(getRootFolder(), 0.15, worker)()\nWill sleep 0.1\nFailing\nWill sleep default\n\nThe real sleep values are in the log:\n\n>>> print(log.getvalue())\nnew transaction\ncommit\nSleeping 0.1 seconds\nnew transaction\nError in worker: Exception('Fail!'...)\nTraceback (most recent call last):\n ...\nException: Fail!\nabort\nSleeping 0.15 seconds\nnew transaction\ncommit\nSleeping 0.15 seconds\nnew transaction\ncommit...\n\nRestore old log handler:\n\n>>> logging.root.removeHandler(log_handler)\n>>> log.close()\n>>> logging.root.setLevel(old_log_level)\n\n\nMultiple commit helper\n======================\n\n\nSometimes you have several actions and want to commit in between each step.\n``gocept.runner`` provides a decorator ``transaction_per_item`` that simplifies\nwriting this down, for example::\n\n @gocept.runner.transaction_per_item\n def multiple_steps():\n for item in source:\n yield lambda: process(item)\n\nThis will call each ``process(item)`` in turn and commit after each. If an\nexception is raised by a step, the transaction is aborted and the loop\ncontinues with the next step. ZODB ConflictErrors are ignored likewise.\n\n\nChanges\n=======\n\n3.0 (2023-07-20)\n----------------\n\n- Drop support for Python 2.7.\n\n- Add support for Python 3.9, 3.10, 3.11.\n\n\n2.2 (2020-09-14)\n----------------\n\n- Get rid of `ZODB3` dependency.\n\n\n2.1 (2019-12-02)\n----------------\n\n- Add support for Python 3.7 and 3.8.\n\n- Migrate to Github.\n\n\n2.0 (2018-03-14)\n----------------\n\n- Backwards incompatible change: No longer depend on ``zope.app.server``,\n and no longer superfluously use its `ZConfig` schema, which superfluously\n requires an `<accesslog>` section that does not make any sense in our context\n anyway. So `<accesslog>` is no longer allowed in the ZConfing config file\n used by this package.\n\n\n1.0 (2016-01-12)\n----------------\n\n- No longer depend on ``zope.app.component``.\n\n\n0.7.1 (2015-01-28)\n------------------\n\n- Fix faulty release 0.7.0.\n\n\n0.7.0 (2015-01-28)\n------------------\n\n- An error message is correctly logged when the worker produces an exception\n which cannot be represented in us-ascii. This used to cause an exception in\n the logging module. The cause of those exceptions where quite hard to track.\n\n- Removed dependency on zope.app.security which was neither declared nor\n necessary.\n\n- Moved project home to <https://bitbucket.org/gocept/gocept.runner/>.\n\n\n0.6.0 (2011-12-01)\n------------------\n\n- Added transaction_per_item decorator.\n- Use stdlib's doctest instead of zope.testing.\n\n\n0.5.3 (2010-04-14)\n------------------\n\n- Use log.error/log.warning instead of log.exception. Conflict errors are\n logged as warning now, because they are not really a problem.\n\n0.5.2 (2010-04-14)\n------------------\n\n- Convert logged exceptions to str.\n\n\n0.5.1 (2009-10-13)\n------------------\n\n- Declared dependencies correctly.\n\n\n0.5 (2009-09-21)\n++++++++++++++++\n\n- Does no longer use ``zope.app.twisted`` but ``zope.app.server`` instead.\n\n\n0.4 (2009-09-03)\n++++++++++++++++\n\n- The principal set by appmain/once can be computed by a function now.\n\n0.3.2 (2009-05-21)\n------------------\n\n- Fixed handling of subsites in appmain.\n\n0.3.1 (2009-05-21)\n------------------\n\n- Declared namespace package.\n\n0.3 (2009-04-15)\n++++++++++++++++\n\n- When a worker fails the default sleep time (instead of the last one) will be\n used.\n\n0.2 (2009-04-09)\n++++++++++++++++\n\n- Added a clean way to exit the runner (by returning gocept.runner.Exit).\n\n0.1 (2009-04-07)\n++++++++++++++++\n\n- first public release\n",
"bugtrack_url": null,
"license": "ZPL 2.1",
"summary": "Create stand alone programs with full Zope3 runtime environment",
"version": "3.0",
"project_urls": {
"Homepage": "https://github.com/gocept/gocept.runner"
},
"split_keywords": [
"zope3",
"mainloop"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "362f290cc4c59ae51ce0f266619d0f6998659ca3209c9462c7c5ade218fef9a9",
"md5": "da82a6b055d56b9c3c787a50263ed6cd",
"sha256": "d2103f45b13b17f20b5fb92bc8c4f2f12dd4dbeeadab167628dd34107c55c1e8"
},
"downloads": -1,
"filename": "gocept.runner-3.0-py2.py3-none-any.whl",
"has_sig": false,
"md5_digest": "da82a6b055d56b9c3c787a50263ed6cd",
"packagetype": "bdist_wheel",
"python_version": "py2.py3",
"requires_python": ">=3.7",
"size": 18185,
"upload_time": "2023-07-20T05:57:47",
"upload_time_iso_8601": "2023-07-20T05:57:47.573862Z",
"url": "https://files.pythonhosted.org/packages/36/2f/290cc4c59ae51ce0f266619d0f6998659ca3209c9462c7c5ade218fef9a9/gocept.runner-3.0-py2.py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "28afa189ca7ed1171da022543aa8cbc529e6af8b516ccee55e87cd5f8c94b768",
"md5": "08a112953181d75d23018b4ea802acf6",
"sha256": "0980e11d41ca6d5fa86e715b68e0097df63c62741ad22475b35d7ddfba5b8e7f"
},
"downloads": -1,
"filename": "gocept.runner-3.0.tar.gz",
"has_sig": false,
"md5_digest": "08a112953181d75d23018b4ea802acf6",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7",
"size": 18073,
"upload_time": "2023-07-20T05:57:49",
"upload_time_iso_8601": "2023-07-20T05:57:49.437231Z",
"url": "https://files.pythonhosted.org/packages/28/af/a189ca7ed1171da022543aa8cbc529e6af8b516ccee55e87cd5f8c94b768/gocept.runner-3.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-07-20 05:57:49",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "gocept",
"github_project": "gocept.runner",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"tox": true,
"lcname": "gocept.runner"
}
JSON
