=============================================================
zodbupdate - Update existing databases to match your software
=============================================================
This package provides a tool that automatically identifies and updates
references from persistent objects to classes that are in the process of being
moved from one module to another and/or being renamed.
If a class is being moved or renamed, you need to update all references from
your database to the new name before finally deleting the old code.
This tool looks through all current objects of your database,
identifies moved/renamed classes and `touches` objects accordingly. It
creates transactions that contains the update of your database (one
transaction every 100,000 records).
Having run this tool, you are then free to delete the old code.
.. contents::
Usage
=====
Installing the egg of this tool provides a console script `zodbupdate` which
you can call giving either a FileStorage filename or a configuration file
defining a storage::
$ zodbupdate -f Data.fs
$ zodbupdate -c zodb.conf
Detailed usage information is available:
$ zodbupdate -h
Custom software/eggs
--------------------
It is important to install this egg in an interpreter/environment where your
software is installed as well. If you're using a regular Python installation
or virtualenv, just installing the package using easy_install should be fine.
If you are using buildout, installing can be done using the egg recipe with
this configuration::
[buildout]
parts += zodbupdate
[zodbupdate]
recipe = zc.recipe.egg
eggs = zodbupdate
<list additional eggs here>
If you do not install `zodbupdate` together with the necessary software it
will report missing classes and not touch your database.
Non-FileStorage configurations
------------------------------
You can configure any storage known to your ZODB installation by providing a
ZConfig configuration file (similar to zope.conf). For example you can connect
to a ZEO server by providing a config file `zeo.conf`::
<zeoclient>
server 127.0.0.1:8100
storage 1
</zeoclient>
And then running `zodbupdate` using:
$ zodbupdate -c zeo.conf
Pre-defined rename rules
------------------------
Rename rules can be defined using an entry point called ``zodbupdate``::
setup(...
entry_points = """
[zodbupdate]
renames = mypackage.mymodule:rename_dict
""")
These can also be defined in python::
setup(...
entry_points={
'zodbupdate': ['renames = mypackage.mymodule:rename_dict'],
})
Those entry points must points to dictionaries that map old class
names to new class names::
rename_dict = {
'mypackage.mymodule ClassName':
'otherpackage.othermodule OtherClass'}
As soon as you have rules defined, you can already remove the old
import location mentioned in them.
Packing
-------
The option ``--pack`` will pack the storage on success. (You tell your
users to use that option. If they never pack their storage, it is a good
occasion).
Converting to Python 3
----------------------
``zodbupdate`` can be used to migrate a database created with a Python
2 application to be usable with the same application in Python 3. To
accomplish this, you need to:
1. Stop your application. Nothing should be written to the database
while the migration is running.
2. Update your Python 2 application to use the latest ZODB version. It
will not work with ZODB 3.
3. With Python 2, run ``zodbupdate --pack --convert-py3``.
If you use a Data.fs we recommend you to use the ``-f`` option to
specify your database. After the conversion the magic header of the
database will be updated so that you will be able to open the database
with Python 3.
If you use a different storage (like RelStorage), be sure you will be
connecting to it using your Python 3 application after the
migration. You will still be able to connect to your database and use
your application with Python 2 without errors, but then you will need
to convert it again to Python 3.
While the pack is not required, it is highly recommended.
The conversion will take care of the following tasks:
- Updating stored Python datetime, date and time objects to use
Python 3 bytes,
- Updating ZODB references to use Python 3 bytes.
- Optionally convert stored strings to either unicode or bytes pending
your configuration.
If your application expect to use bytes in Python 3, they must be
stored as such in the database, and all other strings must be stored
as unicode string, if they contain other characters than ascii characters.
When using ``--convert-py3``, ``zodbupdate`` will load a set of
decoders from the entry points::
setup(...
entry_points = """
[zodbupdate.decode]
decodes = mypackage.mymodule:decode_dict
""")
Decoders are dictionaries that specifies as keys attributes on
Persistent classes that must either be encoded as bytes (if the value
is ``binary``) or decoded to unicode using value as encoding (for
instance ``utf-8`` here)::
decode_dict = {
'mypackage.mymodule ClassName attribute': 'binary',
'otherpackage.othermodule OtherClass other_attribute': 'utf-8'}
Please note that for the moment only attributes on Persistent classes
are supported.
Please also note that these conversion rules are _only_ selected for the
class that is referenced in the pickle, rules for superclasses are _not_
applied. This means that you have to push down annotation rules to all
the subclasses of a superclass that has a field that needs this annotation.
Converting to Python 3 from within Python 3
-------------------------------------------
``zodbupdate`` can also be run from within Python 3 to convert a database
created with Python 2 to be usable in Python 3. However this works
slightly differently than when running the conversion using Python 2.
In Python 3 you must specify a default encoding to use while unpickling strings:
``zodbupdate --pack --convert-py3 --encoding utf-8``.
For each string in the database, zodbupdate will convert it as follows:
1. If it's an attribute configured explicitly via a decoder as described
above, it will be decoded or encoded as specified there.
2. Otherwise the value will be decoded using the encoding specified
on the command line.
3. If there is an error while decoding using the encoding specified
on the command line, the value will be stored as bytes.
Problems and solutions
----------------------
Your Data.fs has POSKey errors
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
If you call `zodbupdate` with ``-f`` and the path to your Data.fs,
records triggering those errors will be ignored.
You have another error
~~~~~~~~~~~~~~~~~~~~~~
We recommend to run zodbupdate with ``-v -d`` to get the
maximum of information.
If you are working on big storages, you can use the option ``-o`` to
re-run `zodbupdate` at a failing record you previously encountered
afterward.
Changes
=======
2.0 (2023-02-09)
----------------
- Add support for Python 3.9, 3.10, 3.11.
- Drop support for Python 2.7, 3.5, 3.6.
- Test with history-free and history-preserving RelStorage. Note that
history-preserving RelStorage requires RelStorage 3.3 or above, and
Python 2.7 or Python 3.6 and above.
(`#30 <https://github.com/zopefoundation/zodbupdate/issues/30>`_)
1.5 (2020-07-28)
----------------
- Fixed incompatibility with ZODB 5.6
(`#35 <https://github.com/zopefoundation/zodbupdate/issues/35>`_)
- Added support for history-free RelStorage
(`#28 <https://github.com/zopefoundation/zodbupdate/issues/28>`_)
- Support zope.interface >= 5 in tests.
(`issue 32 <https://github.com/zopefoundation/zodbupdate/issues/32>`_)
1.4 (2019-08-23)
----------------
- Fail with explanation when opening a Python 2 ZODB with --dry-run on Python 3
(`#22 <https://github.com/zopefoundation/zodbupdate/issues/22>`_)
1.3 (2019-07-30)
----------------
- Support converting sets.Set() objects from ancient Python 2 versions.
(`issue 23 <https://github.com/zopefoundation/zodbupdate/issues/23>`_)
- Convert set objects to ``builtins.set`` without relying on ZODB.broken.rebuild.
(`issue 25 <https://github.com/zopefoundation/zodbupdate/pull/25>`_)
1.2 (2019-05-09)
----------------
- Enable fallback encodings for Python 3 conversion for old/grown ZODBs using
the new command line option ``--encoding-fallback``.
(`#15 <https://github.com/zopefoundation/zodbupdate/pull/15>`_)
- Switch to use `argparse` as `optparse` is deprecated.
- Add ability to run the Python 3 migration with a default encoding for
``str`` objects.
(`#14 <https://github.com/zopefoundation/zodbupdate/pull/14>`_)
- Fix updating records that reference a broken interface
when the interface's top-level module is missing.
- Fixed skipping of blob records so that oids in references to blobs
are still converted.
- Add support for Python 3.8a3.
- Drop support for Python 3.4.
1.1 (2018-10-05)
----------------
- Skip records for ZODB.blob when migrating database to Python 3 to not break
references to blobfiles.
- When migrating databases to Python 3, do not fail when converting
attributes containing None.
- Fix tests on Python 2 with ZODB >= 5.4.0, which now uses pickle
protocol 3.
- Fix `is_broken` check for old-style class instances.
- Add support for Python 3.7.
- Drop PyPy support.
1.0 (2018-02-13)
----------------
- Support Python 2.7 and 3.4, 3.5 and 3.6 and pypy 3. Drop any older
version of Python.
- The option to the select the pickler (``--pickler``) has been
removed. This was only useful if you had extension classes with
Python 2.5 or less.
- Added an option to convert a database to Python 3.
0.5 (2010-10-07)
----------------
- More debug logging shows now the currently processed OID
(that is helpful to determine which object misses the factory).
- Support for missing factories have been improved: an error used to
occur if a pickle needed an update and contained a reference to a
missing class (not instance of this class). This case is now fixed.
- Python 2.4 is no longer supported. Please stick to version 0.3 if
you need Python 2.4 support.
0.4 (2010-07-14)
----------------
- Add an option to debug broken records.
- Add an option to skip records.
- Add an option to use Python unPickler instead of C one. This let you
debug records. As well Python unPickler let you update old ExtensionClass
records who had a special hack in the past.
- Broken interfaces are well supported now (if you did alsoProvides with them).
0.3 (2010-02-02)
----------------
- Unplickle and re-pickle the code to rename references to moved classes.
This make the script works on database created with older versions of
ZODB.
- If you are working directly with a FileStorage, POSKeyError are reported
but non-fatal.
- Remove superfluous code that tried to prevent commits when no changes
happened: ZODB does this all by itself already.
0.2 (2009-06-23)
----------------
- Add option to store the rename rules into a file.
- Don't commit transactions that have no changes.
- Load rename rules from entry points ``zodbupdate``.
- Compatibility with Python 2.4
- Rename from ``zodbupgrade`` to ``zodbupdate``.
- Add 'verbose' option.
- Improve logging.
- Suppress duplicate log messages (e.g. if the same class is missing in
multiple objects).
- Improve the updating process: rewrite pickle opcodes instead of blindly
touching a class. This also allows updating pickles that can't be unpickled
due to missing classes.
0.1 (2009-06-08)
----------------
- First release.
Raw data
{
"_id": null,
"home_page": "https://github.com/zopefoundation/zodbupdate/",
"name": "zodbupdate",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.7",
"maintainer_email": "",
"keywords": "zodb update upgrade migrate data pickle",
"author": "Zope Developers",
"author_email": "zope-dev@zope.dev",
"download_url": "https://files.pythonhosted.org/packages/32/1c/09b5a7aac793de9091e92c6393df1e7abe064d7238feb3c5d0cc6df29fb6/zodbupdate-2.0.tar.gz",
"platform": null,
"description": "=============================================================\nzodbupdate - Update existing databases to match your software\n=============================================================\n\nThis package provides a tool that automatically identifies and updates\nreferences from persistent objects to classes that are in the process of being\nmoved from one module to another and/or being renamed.\n\nIf a class is being moved or renamed, you need to update all references from\nyour database to the new name before finally deleting the old code.\n\nThis tool looks through all current objects of your database,\nidentifies moved/renamed classes and `touches` objects accordingly. It\ncreates transactions that contains the update of your database (one\ntransaction every 100,000 records).\n\nHaving run this tool, you are then free to delete the old code.\n\n.. contents::\n\nUsage\n=====\n\nInstalling the egg of this tool provides a console script `zodbupdate` which\nyou can call giving either a FileStorage filename or a configuration file\ndefining a storage::\n\n $ zodbupdate -f Data.fs\n $ zodbupdate -c zodb.conf\n\nDetailed usage information is available:\n\n $ zodbupdate -h\n\nCustom software/eggs\n--------------------\n\nIt is important to install this egg in an interpreter/environment where your\nsoftware is installed as well. If you're using a regular Python installation\nor virtualenv, just installing the package using easy_install should be fine.\n\nIf you are using buildout, installing can be done using the egg recipe with\nthis configuration::\n\n [buildout]\n parts += zodbupdate\n\n [zodbupdate]\n recipe = zc.recipe.egg\n eggs = zodbupdate\n <list additional eggs here>\n\nIf you do not install `zodbupdate` together with the necessary software it\nwill report missing classes and not touch your database.\n\nNon-FileStorage configurations\n------------------------------\n\nYou can configure any storage known to your ZODB installation by providing a\nZConfig configuration file (similar to zope.conf). For example you can connect\nto a ZEO server by providing a config file `zeo.conf`::\n\n <zeoclient>\n server 127.0.0.1:8100\n storage 1\n </zeoclient>\n\nAnd then running `zodbupdate` using:\n\n $ zodbupdate -c zeo.conf\n\n\nPre-defined rename rules\n------------------------\n\nRename rules can be defined using an entry point called ``zodbupdate``::\n\n setup(...\n entry_points = \"\"\"\n [zodbupdate]\n renames = mypackage.mymodule:rename_dict\n \"\"\")\n\nThese can also be defined in python::\n\n setup(...\n entry_points={\n 'zodbupdate': ['renames = mypackage.mymodule:rename_dict'],\n })\n\nThose entry points must points to dictionaries that map old class\nnames to new class names::\n\n rename_dict = {\n 'mypackage.mymodule ClassName':\n 'otherpackage.othermodule OtherClass'}\n\nAs soon as you have rules defined, you can already remove the old\nimport location mentioned in them.\n\n\nPacking\n-------\n\nThe option ``--pack`` will pack the storage on success. (You tell your\nusers to use that option. If they never pack their storage, it is a good\noccasion).\n\n\nConverting to Python 3\n----------------------\n\n``zodbupdate`` can be used to migrate a database created with a Python\n2 application to be usable with the same application in Python 3. To\naccomplish this, you need to:\n\n1. Stop your application. Nothing should be written to the database\n while the migration is running.\n\n2. Update your Python 2 application to use the latest ZODB version. It\n will not work with ZODB 3.\n\n3. With Python 2, run ``zodbupdate --pack --convert-py3``.\n\nIf you use a Data.fs we recommend you to use the ``-f`` option to\nspecify your database. After the conversion the magic header of the\ndatabase will be updated so that you will be able to open the database\nwith Python 3.\n\nIf you use a different storage (like RelStorage), be sure you will be\nconnecting to it using your Python 3 application after the\nmigration. You will still be able to connect to your database and use\nyour application with Python 2 without errors, but then you will need\nto convert it again to Python 3.\n\nWhile the pack is not required, it is highly recommended.\n\nThe conversion will take care of the following tasks:\n\n- Updating stored Python datetime, date and time objects to use\n Python 3 bytes,\n\n- Updating ZODB references to use Python 3 bytes.\n\n- Optionally convert stored strings to either unicode or bytes pending\n your configuration.\n\nIf your application expect to use bytes in Python 3, they must be\nstored as such in the database, and all other strings must be stored\nas unicode string, if they contain other characters than ascii characters.\n\nWhen using ``--convert-py3``, ``zodbupdate`` will load a set of\ndecoders from the entry points::\n\n setup(...\n entry_points = \"\"\"\n [zodbupdate.decode]\n decodes = mypackage.mymodule:decode_dict\n \"\"\")\n\nDecoders are dictionaries that specifies as keys attributes on\nPersistent classes that must either be encoded as bytes (if the value\nis ``binary``) or decoded to unicode using value as encoding (for\ninstance ``utf-8`` here)::\n\n decode_dict = {\n 'mypackage.mymodule ClassName attribute': 'binary',\n 'otherpackage.othermodule OtherClass other_attribute': 'utf-8'}\n\nPlease note that for the moment only attributes on Persistent classes\nare supported.\n\nPlease also note that these conversion rules are _only_ selected for the \nclass that is referenced in the pickle, rules for superclasses are _not_ \napplied. This means that you have to push down annotation rules to all \nthe subclasses of a superclass that has a field that needs this annotation.\n\nConverting to Python 3 from within Python 3\n-------------------------------------------\n\n``zodbupdate`` can also be run from within Python 3 to convert a database\ncreated with Python 2 to be usable in Python 3. However this works\nslightly differently than when running the conversion using Python 2.\nIn Python 3 you must specify a default encoding to use while unpickling strings:\n``zodbupdate --pack --convert-py3 --encoding utf-8``.\n\nFor each string in the database, zodbupdate will convert it as follows:\n\n1. If it's an attribute configured explicitly via a decoder as described\n above, it will be decoded or encoded as specified there.\n2. Otherwise the value will be decoded using the encoding specified\n on the command line.\n3. If there is an error while decoding using the encoding specified\n on the command line, the value will be stored as bytes.\n\nProblems and solutions\n----------------------\n\nYour Data.fs has POSKey errors\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nIf you call `zodbupdate` with ``-f`` and the path to your Data.fs,\nrecords triggering those errors will be ignored.\n\nYou have another error\n~~~~~~~~~~~~~~~~~~~~~~\n\nWe recommend to run zodbupdate with ``-v -d`` to get the\nmaximum of information.\n\nIf you are working on big storages, you can use the option ``-o`` to\nre-run `zodbupdate` at a failing record you previously encountered\nafterward.\n\nChanges\n=======\n\n2.0 (2023-02-09)\n----------------\n\n- Add support for Python 3.9, 3.10, 3.11.\n\n- Drop support for Python 2.7, 3.5, 3.6.\n\n- Test with history-free and history-preserving RelStorage. Note that\n history-preserving RelStorage requires RelStorage 3.3 or above, and\n Python 2.7 or Python 3.6 and above.\n (`#30 <https://github.com/zopefoundation/zodbupdate/issues/30>`_)\n\n\n1.5 (2020-07-28)\n----------------\n\n- Fixed incompatibility with ZODB 5.6\n (`#35 <https://github.com/zopefoundation/zodbupdate/issues/35>`_)\n\n- Added support for history-free RelStorage\n (`#28 <https://github.com/zopefoundation/zodbupdate/issues/28>`_)\n\n- Support zope.interface >= 5 in tests.\n (`issue 32 <https://github.com/zopefoundation/zodbupdate/issues/32>`_)\n\n\n1.4 (2019-08-23)\n----------------\n\n- Fail with explanation when opening a Python 2 ZODB with --dry-run on Python 3\n (`#22 <https://github.com/zopefoundation/zodbupdate/issues/22>`_)\n\n\n1.3 (2019-07-30)\n----------------\n\n- Support converting sets.Set() objects from ancient Python 2 versions.\n (`issue 23 <https://github.com/zopefoundation/zodbupdate/issues/23>`_)\n\n- Convert set objects to ``builtins.set`` without relying on ZODB.broken.rebuild.\n (`issue 25 <https://github.com/zopefoundation/zodbupdate/pull/25>`_)\n\n\n1.2 (2019-05-09)\n----------------\n\n- Enable fallback encodings for Python 3 conversion for old/grown ZODBs using\n the new command line option ``--encoding-fallback``.\n (`#15 <https://github.com/zopefoundation/zodbupdate/pull/15>`_)\n\n- Switch to use `argparse` as `optparse` is deprecated.\n\n- Add ability to run the Python 3 migration with a default encoding for\n ``str`` objects.\n (`#14 <https://github.com/zopefoundation/zodbupdate/pull/14>`_)\n\n- Fix updating records that reference a broken interface\n when the interface's top-level module is missing.\n\n- Fixed skipping of blob records so that oids in references to blobs\n are still converted.\n\n- Add support for Python 3.8a3.\n\n- Drop support for Python 3.4.\n\n\n1.1 (2018-10-05)\n----------------\n\n- Skip records for ZODB.blob when migrating database to Python 3 to not break\n references to blobfiles.\n\n- When migrating databases to Python 3, do not fail when converting\n attributes containing None.\n\n- Fix tests on Python 2 with ZODB >= 5.4.0, which now uses pickle\n protocol 3.\n\n- Fix `is_broken` check for old-style class instances.\n\n- Add support for Python 3.7.\n\n- Drop PyPy support.\n\n\n1.0 (2018-02-13)\n----------------\n\n- Support Python 2.7 and 3.4, 3.5 and 3.6 and pypy 3. Drop any older\n version of Python.\n\n- The option to the select the pickler (``--pickler``) has been\n removed. This was only useful if you had extension classes with\n Python 2.5 or less.\n\n- Added an option to convert a database to Python 3.\n\n0.5 (2010-10-07)\n----------------\n\n- More debug logging shows now the currently processed OID\n (that is helpful to determine which object misses the factory).\n\n- Support for missing factories have been improved: an error used to\n occur if a pickle needed an update and contained a reference to a\n missing class (not instance of this class). This case is now fixed.\n\n- Python 2.4 is no longer supported. Please stick to version 0.3 if\n you need Python 2.4 support.\n\n\n\n0.4 (2010-07-14)\n----------------\n\n- Add an option to debug broken records.\n\n- Add an option to skip records.\n\n- Add an option to use Python unPickler instead of C one. This let you\n debug records. As well Python unPickler let you update old ExtensionClass\n records who had a special hack in the past.\n\n- Broken interfaces are well supported now (if you did alsoProvides with them).\n\n\n0.3 (2010-02-02)\n----------------\n\n- Unplickle and re-pickle the code to rename references to moved classes.\n This make the script works on database created with older versions of\n ZODB.\n\n- If you are working directly with a FileStorage, POSKeyError are reported\n but non-fatal.\n\n- Remove superfluous code that tried to prevent commits when no changes\n happened: ZODB does this all by itself already.\n\n0.2 (2009-06-23)\n----------------\n\n- Add option to store the rename rules into a file.\n\n- Don't commit transactions that have no changes.\n\n- Load rename rules from entry points ``zodbupdate``.\n\n- Compatibility with Python 2.4\n\n- Rename from ``zodbupgrade`` to ``zodbupdate``.\n\n- Add 'verbose' option.\n\n- Improve logging.\n\n- Suppress duplicate log messages (e.g. if the same class is missing in\n multiple objects).\n\n- Improve the updating process: rewrite pickle opcodes instead of blindly\n touching a class. This also allows updating pickles that can't be unpickled\n due to missing classes.\n\n0.1 (2009-06-08)\n----------------\n\n- First release.\n",
"bugtrack_url": null,
"license": "ZPL 2.1",
"summary": "Update ZODB class references for moved or renamed classes.",
"version": "2.0",
"split_keywords": [
"zodb",
"update",
"upgrade",
"migrate",
"data",
"pickle"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "414c56b3e0767a1c12647a1b0e7c1b1cdf1de1fe5fceffb1e7d2587d122e8131",
"md5": "8882b2901a8f4da19c8f90dc2916e4a6",
"sha256": "432487816b864e2699eeac89b0b8c3ed3c214bb640bc8c5297b1cd9b6179e504"
},
"downloads": -1,
"filename": "zodbupdate-2.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "8882b2901a8f4da19c8f90dc2916e4a6",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7",
"size": 24312,
"upload_time": "2023-02-09T10:33:00",
"upload_time_iso_8601": "2023-02-09T10:33:00.266483Z",
"url": "https://files.pythonhosted.org/packages/41/4c/56b3e0767a1c12647a1b0e7c1b1cdf1de1fe5fceffb1e7d2587d122e8131/zodbupdate-2.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "321c09b5a7aac793de9091e92c6393df1e7abe064d7238feb3c5d0cc6df29fb6",
"md5": "3c26942806c094d964045cffb3053496",
"sha256": "026f8a0ab194f254898b172d4263a7650698f17f61bf73b33b9da90e36764ddd"
},
"downloads": -1,
"filename": "zodbupdate-2.0.tar.gz",
"has_sig": false,
"md5_digest": "3c26942806c094d964045cffb3053496",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7",
"size": 27072,
"upload_time": "2023-02-09T10:33:02",
"upload_time_iso_8601": "2023-02-09T10:33:02.422405Z",
"url": "https://files.pythonhosted.org/packages/32/1c/09b5a7aac793de9091e92c6393df1e7abe064d7238feb3c5d0cc6df29fb6/zodbupdate-2.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-02-09 10:33:02",
"github": true,
"gitlab": false,
"bitbucket": false,
"github_user": "zopefoundation",
"github_project": "zodbupdate",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"tox": true,
"lcname": "zodbupdate"
}
JSON
