collective.purgebyid
====================
.. image:: https://img.shields.io/pypi/v/collective.purgebyid.svg
:target: https://pypi.python.org/pypi/collective.purgebyid/
:alt: Latest Version
.. image:: https://img.shields.io/pypi/pyversions/collective.purgebyid.svg?style=plastic
:target: https://pypi.python.org/pypi/collective.purgebyid/
:alt: Supported - Python Versions
.. image:: https://img.shields.io/pypi/dm/collective.purgebyid.svg
:target: https://pypi.python.org/pypi/collective.purgebyid/
:alt: Number of PyPI downloads
.. image:: https://img.shields.io/pypi/l/collective.purgebyid.svg
:target: https://pypi.python.org/pypi/collective.purgebyid/
:alt: License
.. image:: https://github.com/collective/collective.purgebyid/actions/workflows/tests.yml/badge.svg
:target: https://github.com/collective/collective.purgebyid/actions
:alt: Tests
.. image:: https://coveralls.io/repos/github/collective/collective.purgebyid/badge.svg?branch=master
:target: https://coveralls.io/github/collective/collective.purgebyid?branch=master
:alt: Coverage
collective.purgbyid is a new method for cache invalidation of Plone
based web sites. It uses the idea of adding an extra header, called
X-Ids-Involved, which contains the uuids of the objects involved in the
construction of the resources. For example, an image contains just its
uuid::
% wget -S http://localhost:8080/Plone/image01
...
X-Ids-Involved: #c8d7c0bc2b794325b916d990de91d7ee#
Other pages may be more complicated. Then a new purge rewrite rule adds
a custom URL to the set of URLs to purge: the "purge by id" custom URL
is in the form /@@purgebyid/<UUID> where UUID is the object's uuid to be
purged.
Last, Varnish is configured so that, when an URL /@@purgebyid/<UUID> is
purged, it will ban all the objects that match an X-Ids-Involved header
of the right type (i.e. containing the uuid of the resource to purge).
This means that when a resources is purged, it is enough to purge also
it /@@purgebyid/<UUID> URL because it will be Varnish responsibily to
also catch all of the occurrencies of the resources whenever the URL
which is used to access it.
Varnish without xkey varnish module
-----------------------------------
Without the xkey module, the way to purge a resource is to ban all objects
which have the X-Ids-Involved header with the id of the resource to be purged.
For a better understanding of the differences between the two approaches (ban vs. purge), please read:
https://varnish-cache.org/docs/trunk/users-guide/purging.html
Config example::
sub vcl_recv {
if (req.method == "PURGE") {
if (!client.ip ~ purge) {
return (synth(405, "This IP is not allowed to send PURGE requests."));
}
if (req.url ~ "^/@@purgebyid/") {
ban("obj.http.x-ids-involved ~ #" + regsub(req.url, "^/@@purgebyid/", "") + "#");
return(synth(200, "Ban added"));
}
return(purge);
}
}
sub vcl_deliver {
unset resp.http.x-ids-involved;
}
Varnish with xkey varnish module
--------------------------------
By default, Varnish uses the URL as the hash key for purging, but with
the xkey module (https://github.com/varnish/varnish-modules/blob/master/src/vmod_xkey.vcc)
there comes a secondary hash for doing so. Cached objects
being tagged can be specifically purged for a more targeted cache control.
To have xkey working, it is mandatory to provide a special HTTP header called
"Xkey" which contains all the tags (separated by white-space). Few additional codes in
the `vcl_backend_response` transforms the X-Ids-Involved header into an XKey.
Config example::
import xkey;
sub vcl_recv {
if (req.method == "PURGE") {
if (!client.ip ~ purge) {
return (synth(405, "This IP is not allowed to send PURGE requests."));
}
if (req.url ~ "^/@@purgebyid/") {
set req.http.n-gone = xkey.purge(regsub(req.url, "^/@@purgebyid/", ""));
# or: set req.http.n-gone = xkey.softpurge(regsub(req.url, "^/@@purgebyid/", ""));
return (synth(200, "Invalidated "+req.http.n-gone+" objects"));
}
}
return(purge);
}
sub vcl_backend_response {
if (beresp.http.x-ids-involved) {
set beresp.http.xkey = regsuball(beresp.http.x-ids-involved, "#", " ");
}
}
sub vcl_deliver {
unset resp.http.x-ids-involved;
unset resp.http.xkey;
}
A softpurge differs from a regular purge in that it resets an
object's TTL but keeps it available for grace mode and conditional
requests for the remainder of its configured grace and keep time.
How does it work? How to extend it?
-----------------------------------
During the publishing process all involved IDs (UUIDs and custom IDs) are collected
(by subscribing to IPubAfterTraversal).
Important are the adapters for IInvolvedID, which are responsible for collecting IDs for their given context.
The base implementation looks for the UUIDs, but may be specialized for your custom content types.
Apart from the adapter approach, there is the inline approach. You may call the following methods
from collective.purgebyid.api:
* mark_involved_objects(request, objs, stoponfirst=False)
* mark_involved(request, single_id)
Whereas the first method uses internally the adapters for IInvolvedIDs for the given objects,
the second one allows setting any arbitrary IDs.
These methods combined might be used in your views, whenever a certain object or part is being rendered.
Additionally, there is a utility browser view "purgebyid", that can be used in your template as follows:
.. code-block:: xml
<body tal:define="purgeutils nocall:context/@@purgebyid">
...
<tal:image tal:define="image python:context.get_image()" tal:condition="python: image">
<tal:mark-involved tal:define="dummy python:purgeutils.mark(image)" />
<!-- put image rendering here -->
...
</tal:image>
...
</body>
Alternatively, you can again set arbitrary IDs:
.. code-block:: xml
<tal:mark-involved tal:define="dummy python:purgeutils.mark('my_custom_id')" />
After having collected all IDs a ITransform adapter puts the expected `X-Ids-Involved` header in
the HTTP response header.
When Plone sends a purge request to the configured Cache Proxy, it sends additionally a specialized
request for handling objects with tags.
References
----------
* Blog post https://www.biodec.com/it/blog/migliorare-la-gestione-del-purge-caching-in-plone-collective-purgebyid (italian language)
Contributors
============
Mauro Amico, Author
Changelog
=========
1.2.1 (2022-12-08)
------------------
- plone 6.0 / python 3.10 support
[mamico]
- avoid marking requests with the UUID of the plonesite
[mamico]
1.2.0 (2022-08-04)
------------------
- collective.xkey backports. Add utility browser view.
[pgrunewald, mamico]
- fix p.a.multilingual IUUID adapter inconsistency
[mamico]
1.1.2 (2021-11-22)
------------------
- Remove unused importDependencies (for pip install compatibility).
[cekk]
1.1.1 (2019-06-05)
------------------
- Python 3 support
[mamico]
1.1.0 (2018-05-14)
------------------
- moved headers mutator from PubSuccess event to plone.transformchain.
fix missing header using p.a.caching's ramcache operations #2
[mamico]
- added IIDinvolved adapter for easy implements "involved id" extractors
[mamico]
- manage resourcedirectory, because previously all resources were marked as "involved" by
navigation root
[mamico]
- fix issue where IUUID-adaptation did not have default value
[datakurre]
1.0.0 (2016-01-14)
------------------
- use zope.annotation on request
[mamico]
- unused generic setup profile removed
[mamico]
1.0.0a1 (2013-09-11)
--------------------
- Package created using templer
[Mauro Amico]
Raw data
{
"_id": null,
"home_page": "https://github.com/collective/collective.purgebyid",
"name": "collective.purgebyid",
"maintainer": "",
"docs_url": null,
"requires_python": "",
"maintainer_email": "",
"keywords": "plone caching",
"author": "Mauro Amico",
"author_email": "mauro.amico@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/c1/8c/5f0851b7b397c936517f1fddfe4fd135bde21958bd51311123c6b695fbd0/collective.purgebyid-1.2.1.tar.gz",
"platform": null,
"description": "collective.purgebyid\n====================\n\n.. image:: https://img.shields.io/pypi/v/collective.purgebyid.svg\n :target: https://pypi.python.org/pypi/collective.purgebyid/\n :alt: Latest Version\n\n.. image:: https://img.shields.io/pypi/pyversions/collective.purgebyid.svg?style=plastic\n :target: https://pypi.python.org/pypi/collective.purgebyid/\n :alt: Supported - Python Versions\n\n.. image:: https://img.shields.io/pypi/dm/collective.purgebyid.svg\n :target: https://pypi.python.org/pypi/collective.purgebyid/\n :alt: Number of PyPI downloads\n \n.. image:: https://img.shields.io/pypi/l/collective.purgebyid.svg\n :target: https://pypi.python.org/pypi/collective.purgebyid/\n :alt: License\n\n.. image:: https://github.com/collective/collective.purgebyid/actions/workflows/tests.yml/badge.svg\n :target: https://github.com/collective/collective.purgebyid/actions\n :alt: Tests\n\n.. image:: https://coveralls.io/repos/github/collective/collective.purgebyid/badge.svg?branch=master\n :target: https://coveralls.io/github/collective/collective.purgebyid?branch=master\n :alt: Coverage\n\ncollective.purgbyid is a new method for cache invalidation of Plone\nbased web sites. It uses the idea of adding an extra header, called\nX-Ids-Involved, which contains the uuids of the objects involved in the\nconstruction of the resources. For example, an image contains just its\nuuid::\n\n % wget -S http://localhost:8080/Plone/image01\n ...\n X-Ids-Involved: #c8d7c0bc2b794325b916d990de91d7ee#\n\nOther pages may be more complicated. Then a new purge rewrite rule adds\na custom URL to the set of URLs to purge: the \"purge by id\" custom URL\nis in the form /@@purgebyid/<UUID> where UUID is the object's uuid to be\npurged.\n\nLast, Varnish is configured so that, when an URL /@@purgebyid/<UUID> is\npurged, it will ban all the objects that match an X-Ids-Involved header\nof the right type (i.e. containing the uuid of the resource to purge).\nThis means that when a resources is purged, it is enough to purge also\nit /@@purgebyid/<UUID> URL because it will be Varnish responsibily to\nalso catch all of the occurrencies of the resources whenever the URL\nwhich is used to access it.\n\nVarnish without xkey varnish module\n-----------------------------------\n\nWithout the xkey module, the way to purge a resource is to ban all objects\nwhich have the X-Ids-Involved header with the id of the resource to be purged.\n\nFor a better understanding of the differences between the two approaches (ban vs. purge), please read:\nhttps://varnish-cache.org/docs/trunk/users-guide/purging.html\n\nConfig example::\n\n sub vcl_recv {\n if (req.method == \"PURGE\") {\n if (!client.ip ~ purge) {\n return (synth(405, \"This IP is not allowed to send PURGE requests.\"));\n }\n if (req.url ~ \"^/@@purgebyid/\") {\n ban(\"obj.http.x-ids-involved ~ #\" + regsub(req.url, \"^/@@purgebyid/\", \"\") + \"#\");\n return(synth(200, \"Ban added\"));\n }\n return(purge);\n }\n }\n\n sub vcl_deliver {\n unset resp.http.x-ids-involved;\n }\n\n\nVarnish with xkey varnish module\n--------------------------------\n\nBy default, Varnish uses the URL as the hash key for purging, but with\nthe xkey module (https://github.com/varnish/varnish-modules/blob/master/src/vmod_xkey.vcc)\nthere comes a secondary hash for doing so. Cached objects\nbeing tagged can be specifically purged for a more targeted cache control.\n\nTo have xkey working, it is mandatory to provide a special HTTP header called\n\"Xkey\" which contains all the tags (separated by white-space). Few additional codes in\nthe `vcl_backend_response` transforms the X-Ids-Involved header into an XKey.\n\nConfig example::\n\n import xkey;\n\n sub vcl_recv {\n if (req.method == \"PURGE\") {\n if (!client.ip ~ purge) {\n return (synth(405, \"This IP is not allowed to send PURGE requests.\"));\n }\n if (req.url ~ \"^/@@purgebyid/\") {\n set req.http.n-gone = xkey.purge(regsub(req.url, \"^/@@purgebyid/\", \"\"));\n # or: set req.http.n-gone = xkey.softpurge(regsub(req.url, \"^/@@purgebyid/\", \"\"));\n return (synth(200, \"Invalidated \"+req.http.n-gone+\" objects\"));\n }\n }\n return(purge);\n }\n\n sub vcl_backend_response {\n if (beresp.http.x-ids-involved) {\n set beresp.http.xkey = regsuball(beresp.http.x-ids-involved, \"#\", \" \");\n }\n }\n\n sub vcl_deliver {\n unset resp.http.x-ids-involved;\n unset resp.http.xkey;\n }\n\n\nA softpurge differs from a regular purge in that it resets an\nobject's TTL but keeps it available for grace mode and conditional\nrequests for the remainder of its configured grace and keep time.\n\nHow does it work? How to extend it?\n-----------------------------------\n\nDuring the publishing process all involved IDs (UUIDs and custom IDs) are collected\n(by subscribing to IPubAfterTraversal).\n\nImportant are the adapters for IInvolvedID, which are responsible for collecting IDs for their given context.\nThe base implementation looks for the UUIDs, but may be specialized for your custom content types.\n\nApart from the adapter approach, there is the inline approach. You may call the following methods\nfrom collective.purgebyid.api:\n\n* mark_involved_objects(request, objs, stoponfirst=False)\n* mark_involved(request, single_id)\n\nWhereas the first method uses internally the adapters for IInvolvedIDs for the given objects,\nthe second one allows setting any arbitrary IDs.\nThese methods combined might be used in your views, whenever a certain object or part is being rendered.\n\nAdditionally, there is a utility browser view \"purgebyid\", that can be used in your template as follows:\n\n.. code-block:: xml\n\n <body tal:define=\"purgeutils nocall:context/@@purgebyid\">\n ...\n <tal:image tal:define=\"image python:context.get_image()\" tal:condition=\"python: image\">\n\n <tal:mark-involved tal:define=\"dummy python:purgeutils.mark(image)\" />\n <!-- put image rendering here -->\n ...\n\n </tal:image>\n ...\n </body>\n\nAlternatively, you can again set arbitrary IDs:\n\n.. code-block:: xml\n\n <tal:mark-involved tal:define=\"dummy python:purgeutils.mark('my_custom_id')\" />\n\nAfter having collected all IDs a ITransform adapter puts the expected `X-Ids-Involved` header in\nthe HTTP response header.\n\nWhen Plone sends a purge request to the configured Cache Proxy, it sends additionally a specialized\nrequest for handling objects with tags.\n\n\nReferences\n----------\n\n* Blog post https://www.biodec.com/it/blog/migliorare-la-gestione-del-purge-caching-in-plone-collective-purgebyid (italian language)\n\nContributors\n============\n\nMauro Amico, Author\n\nChangelog\n=========\n\n1.2.1 (2022-12-08)\n------------------\n\n- plone 6.0 / python 3.10 support\n [mamico]\n\n- avoid marking requests with the UUID of the plonesite\n [mamico]\n\n1.2.0 (2022-08-04)\n------------------\n\n- collective.xkey backports. Add utility browser view.\n [pgrunewald, mamico]\n\n- fix p.a.multilingual IUUID adapter inconsistency\n [mamico]\n\n1.1.2 (2021-11-22)\n------------------\n\n- Remove unused importDependencies (for pip install compatibility).\n [cekk]\n\n1.1.1 (2019-06-05)\n------------------\n\n- Python 3 support \n [mamico]\n\n\n1.1.0 (2018-05-14)\n------------------\n\n- moved headers mutator from PubSuccess event to plone.transformchain.\n fix missing header using p.a.caching's ramcache operations #2\n [mamico]\n- added IIDinvolved adapter for easy implements \"involved id\" extractors\n [mamico]\n- manage resourcedirectory, because previously all resources were marked as \"involved\" by\n navigation root\n [mamico]\n- fix issue where IUUID-adaptation did not have default value\n [datakurre]\n\n\n1.0.0 (2016-01-14)\n------------------\n\n- use zope.annotation on request\n [mamico]\n- unused generic setup profile removed\n [mamico]\n\n1.0.0a1 (2013-09-11)\n--------------------\n\n- Package created using templer\n [Mauro Amico]\n\n",
"bugtrack_url": null,
"license": "gpl",
"summary": "p.a.caching add-on for a better purging policy",
"version": "1.2.1",
"split_keywords": [
"plone",
"caching"
],
"urls": [
{
"comment_text": "",
"digests": {
"md5": "42e283438a0810dcf24b50e8f077d2d1",
"sha256": "2b8f4ed089046a41c1f2174dbd5f7554b35bad9dbc88f65e4383c44a719e5d49"
},
"downloads": -1,
"filename": "collective.purgebyid-1.2.1-py2.py3-none-any.whl",
"has_sig": false,
"md5_digest": "42e283438a0810dcf24b50e8f077d2d1",
"packagetype": "bdist_wheel",
"python_version": "py2.py3",
"requires_python": null,
"size": 21530,
"upload_time": "2022-12-08T01:21:12",
"upload_time_iso_8601": "2022-12-08T01:21:12.670322Z",
"url": "https://files.pythonhosted.org/packages/68/4e/38e842c6c37bcea652fcdeab4cd34cb0cd1720a167ce7bb5996bc63870b2/collective.purgebyid-1.2.1-py2.py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"md5": "e4b90c6fdf2da20dc3013d05cab7b1d5",
"sha256": "805906fafd14a2e77ea6c27d19f5e5e42387dcea9b5d7fa6a0567e0c104b98f4"
},
"downloads": -1,
"filename": "collective.purgebyid-1.2.1.tar.gz",
"has_sig": false,
"md5_digest": "e4b90c6fdf2da20dc3013d05cab7b1d5",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 24961,
"upload_time": "2022-12-08T01:21:14",
"upload_time_iso_8601": "2022-12-08T01:21:14.237152Z",
"url": "https://files.pythonhosted.org/packages/c1/8c/5f0851b7b397c936517f1fddfe4fd135bde21958bd51311123c6b695fbd0/collective.purgebyid-1.2.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2022-12-08 01:21:14",
"github": true,
"gitlab": false,
"bitbucket": false,
"github_user": "collective",
"github_project": "collective.purgebyid",
"travis_ci": false,
"coveralls": true,
"github_actions": true,
"tox": true,
"lcname": "collective.purgebyid"
}
JSON
