Easy Zope backup/restore recipe for buildout
********************************************
.. image:: https://github.com/collective/collective.recipe.backup/workflows/tests/badge.svg
:target: https://github.com/collective/collective.recipe.backup/actions?query=workflow%3Atests
.. contents::
Introduction
============
This recipe is mostly a wrapper around the ``bin/repozo`` script in
your Zope buildout. It requires that this script is already made
available. If this is not the case, you will get an error like this
when you run one of the scripts: ``bin/repozo: No such file or
directory``. This should be there when you
are using ``plone.recipe.zeoserver``. If this is
not the case, the easiest way of getting a ``bin/repozo`` script is to
add a new section in your ``buildout.cfg`` (do not forget to add it in the
``parts`` directive)::
[repozo]
recipe = zc.recipe.egg
eggs = ZODB
# or this for an older version:
# eggs = ZODB3
scripts = repozo
dependent-scripts = true
``bin/repozo`` is a Zope script to make backups of your ``Data.fs``.
Looking up the settings can be a chore. And you have to pick a
directory where to put the backups. This recipe provides **sensible
defaults** for your common backup tasks. Making backups a piece of
cake is important!
- ``bin/backup`` makes an incremental backup.
- ``bin/restore`` restores the latest backup created by the backup script.
- ``bin/snapshotbackup`` makes a full snapshot backup, separate from the
regular backups. Handy right before a big change in the site.
- ``bin/snapshotrestore`` restores the latest full snapshot backup.
- ``bin/zipbackup`` makes a zip backup. This zips the Data.fs and the
blobstorage, handy for copying production data to your local
machine, especially the blobstorage with its many files.
Actually, zipping the Data.fs is standard,
and we do not zip the blobstorage,
because most files in there are already compressed.
But we do combine the blobs in one tar archive.
*Note*: the Data.fs and blobstorage (or other storages) are *not* combined
in one file; you need to download multiple files. Enable this
script by using the ``enable_zipbackup`` option.
- ``bin/ziprestore`` restores the latest zipbackup.
Compatibility
=============
The recipe is tested with Python 2.7, 3.6, 3.7, 3.8, 3.9.
In Plone terms it works fine on Plone 4, 5 and 6.
Note that the integration with ``plone.recipe.zope2instance`` is not tested on Python 3.
It would pulls in too many dependencies, like Zope and ZODB.
Development
===========
- Code repository: https://github.com/collective/collective.recipe.backup
- Issue tracker: https://github.com/collective/collective.recipe.backup/issues
- Obvious fixes, like fixing typos, are fine on master.
For larger changes or if you are unsure, please create a branch or a pull request.
- The code comes with a ``buildout.cfg``. Please bootstrap the
buildout and run the created ``bin/test`` to see if the tests still
pass. Please try to add tests if you add code.
To run the tests for all supported Python versions, run ``tox``.
- The long description of this package (as shown on PyPI), used to
contain a big file with lots of test code that showed how to use the
recipe. This grew too large, so we left it out. It is probably
still good reading if you are wondering about the effect some
options have. See ``src/collective/recipe/backup/tests/*.rst``.
- We are tested on GitHub Actions:
https://github.com/collective/collective.recipe.backup/actions
- Questions and comments to https://community.plone.org or to
`Maurits van Rees <mailto:maurits@vanrees.org>`_.
Example usage
=============
The simplest way to use this recipe is to add a part in ``buildout.cfg`` like this::
[buildout]
parts = backup
[backup]
recipe = collective.recipe.backup
You can set lots of extra options, but the recipe authors like to
think they have created sane defaults, so this single line stating the
recipe name should be enough in most cases.
Running the buildout adds the ``backup``,
``snapshotbackup``, ``zipbackup``, ``restore``, ``snapshotrestore``
and ``ziprestore`` scripts to the ``bin/`` directory of the buildout.
Some are not added by default, others can be switched off.
Backed up data
==============
Which data does this recipe backup?
- The Zope Object DataBase (ZODB) filestorage, by default located at
``var/filestorage/Data.fs``.
- The blobstorage (since version 2.0) if your buildout uses it, by
default located at ``var/blobstorage``.
Data that is *not* backed up
============================
Which data does this recipe *not* backup? Everything else of course,
but specifically:
- Data stored in ``RelStorage`` will *not* be backed up. (You could
still use this recipe to back up the filesystem blobstorage,
possibly with the ``only_blobs`` option.)
- Other data stored in SQL, perhaps via SQLAlchemy, will *not* be
backed up.
- It does *not* create a backup of your entire buildout directory.
Is your backup backed up?
=========================
Note that the backups are by default created in the ``var`` directory
of the buildout, so if you accidentally remove the entire buildout,
you also lose your backups. It should be standard practice to use the
``location`` option to specify a backup location in for example the
home directory of the user. You should also arrange to copy that
backup to a different machine/country/continent/planet.
Backup
======
Calling ``bin/backup`` results in a normal incremental repozo backup
that creates a backup of the ``Data.fs`` in ``var/backups``. When you
have a blob storage it is by default backed up to
``var/blobstoragebackups``.
Snapshots
=========
A quick backup just before updating the production server is a good
idea. But you may not want to interfere with the regular backup
regime. For that, the ``bin/snapshotbackup`` is great. It places a
full backup in, by default, ``var/snapshotbackups``.
Zipbackups
==========
For quickly grabbing the current state of a production database so you
can download it to your development laptop, you want a full and zipped
backup. The zipped part is important for the blobstorage, because you
do not want to use ``scp`` to recursively copy over all those blob
files: downloading one tarball is faster.
You can use the ``bin/zipbackup`` script for this. This script
overrides a few settings, ignoring whatever is set in the buildout
config section:
- ``archive_blob`` is turned on.
- ``keep`` is set to 1 to avoid keeping lots of needless backups.
- ``keep_blob_days`` is ignored because it is a full backup.
The script places a full backup in, by default, ``var/zipbackups`` and
it puts a tarball of the blobstorage in ``var/blobstoragezips``.
This script is not created by default.
You can enable it by setting the ``enable_zipbackup`` option to true.
Also, if ``backup_blobs`` is false, the scripts are useless, so we do not create them, even when you have enabled them explicitly.
Restore
=======
Calling ``bin/restore`` restores the very latest normal incremental
``repozo`` backup and restores the blobstorage if you have that.
You can restore the very latest snapshotbackup with ``bin/snapshotrestore``.
You can restore the zipbackup with ``bin/ziprestore``.
You can also restore the backup as of a certain date. Just pass a date argument.
According to ``repozo``: specify UTC (not local) time.
The format is ``yyyy-mm-dd[-hh[-mm[-ss]]]``.
So as a simple example, restore to 25 december 1972::
bin/restore 1972-12-25
or to that same date, at 2,03 seconds past 1::
bin/restore 1972-12-25-01-02-03
Since version 2.3 this also works for restoring blobs.
We restore the directory from the first backup at or before the specified date.
(Note that before version 4.0 we restored the directory from the first backup after the specified date,
which should be fine as long as you did not do a database pack in between.)
Since version 2.0, the restore scripts ask for confirmation before
starting the restore, as this is a potentially dangerous command.
("Oops, I have restored the live site but I meant to restore the test
site.") You need to explicitly type 'yes'::
This will replace the filestorage (Data.fs).
This will replace the blobstorage.
Are you sure? (yes/No)?
Note that for large filestorages and blobstorages **it may take long to restore**.
You should do a test restore and check how long it takes.
Seconds? Minutes? Hours?
Is that time acceptable or should you take other measures?
Names of created scripts
========================
A backup part will normally be called ``[backup]``, leading to a
``bin/backup`` and ``bin/snapshotbackup``. Should you name your part
something else, the script names will also be different, as will the created
``var/`` directories (since version 1.2)::
[buildout]
parts = plonebackup
[plonebackup]
recipe = collective.recipe.backup
enable_zipbackup = true
That buildout snippet will create these scripts::
bin/plonebackup
bin/plonebackup-full
bin/plonebackup-zip
bin/plonebackup-snapshot
bin/plonebackup-restore
bin/plonebackup-ziprestore
bin/plonebackup-snapshotrestore
Supported options
=================
The recipe supports the following options, none of which are needed by
default. The most common ones to change are ``location`` and
``blobbackuplocation``, as those allow you to place your backups in
some system-wide directory like ``/var/zopebackups/instancename/`` and
``/var/zopebackups/instancename-blobs/``.
.. Note: keep this in alphabetical order please.
``archive_blob``
Use ``tar`` archiving functionality. ``false`` by default. Set it to ``true``
and backup/restore will be done with ``tar`` command. Note that ``tar``
command must be available on machine if this option is set to ``true``.
This option also works with snapshot backup/restore commands. As this
counts as a full backup ``keep_blob_days`` is ignored.
See the ``compress_blob`` option if you want to compress the archive.
``alternative_restore_source``
You can restore from an alternative source. Use case: first make
a backup of your production site, then go to the testing or
staging server and restore the production data there. See
`Alternative restore source`_
``alternative_restore_sources``
Backwards compatibility spelling for ``alternative_restore_source``.
This will no longer work in version 7.
``backup_blobs``
Backup the blob storage. Default is ``True`` on Python 2.6 (Plone
4) and higher, and ``False`` otherwise. This requires the
``blob_storage`` location to be set. If no ``blob_storage``
location has been set and we cannot find one by looking in the
other buildout parts, we quit with an error (since version 2.22).
If ``backup_blobs`` is false, ``enable_zipbackup`` cannot be true,
because the ``zipbackup`` script is not useful then.
``blob_storage``
Location of the directory where the blobs (binary large objects)
are stored. This is used in Plone 4 and higher, or on Plone 3 if
you use ``plone.app.blob``. This option is ignored if backup_blobs is
``false``. The location is not set by default. When there is a part
using ``plone.recipe.zeoserver``, ``plone.recipe.zope2instance`` or
``plone.recipe.zope2zeoserver``, we check if that has a
blob-storage option and use that as default. Note that we pick
the first one that has this option and we do not care about
shared-blob settings, so there are probably corner cases where we
do not make the best decision here. Use this option to override
it in that case.
``blob-storage``
Alternative spelling for the preferred ``blob_storage``, as
``plone.recipe.zope2instance`` spells it as ``blob-storage`` and we are
using underscores in all the other options. Pick one.
``blob_timestamps``
New in version 4.0. Default is true (this was false before version 4.2).
If false, we create ``blobstorage.0``.
The next time, we rotate this to ``blobstorage.1`` and create a new ``blobstorage.0``.
With ``blob_timestamps = true``, we create stable directories that we do not rotate.
They get a timestamp, the same timestamp that the ZODB filestorage backup gets.
For example: ``blobstorage.1972-12-25-01-02-03``.
Or with ``archive_blob = true``: ``blobstorage.1972-12-25-01-02-03.tar``.
Because the filename is unpredictable, since version 4.1 we create a ``latest`` symlink
to the most recent backup.
Blob timestamps are not used with zipbackup, because this only keeps 1 backup,
which means there is no confusion about which filestorage backup it belongs to.
``blobbackuplocation``
Directory where the blob storage will be backed up to. Defaults
to ``var/blobstoragebackups`` inside the buildout directory.
``blobsnapshotlocation``
Directory where the blob storage snapshots will be created.
Defaults to ``var/blobstoragesnapshots`` inside the buildout
directory.
``blobziplocation``
Directory where the blob storage zipbackups will be created.
Defaults to ``var/blobstoragezips`` inside the buildout
directory.
``compress_blob``
New in version 4.0. Default is false.
This is only used when the ``archive_blob`` option is true.
When switched on, it will compress the archive,
resulting in a ``.tar.gz`` instead of a ``tar`` file.
When restoring, we always look for both compressed and normal archives.
We used to always compress them, but in most cases it hardly decreases the size
and it takes a long time anyway. I have seen archiving take 15 seconds,
and compressing take an additional 45 seconds.
The result was an archive of 5.0 GB instead of 5.1 GB.
``datafs``
In case the ``Data.fs`` isn't in the default ``var/filestorage/Data.fs``
location, this option can overwrite it.
``debug``
In rare cases when you want to know exactly what's going on, set debug to
``true`` to get debug level logging of the recipe itself. ``repozo`` is also run
with ``--verbose`` if this option is enabled.
``enable_snapshotrestore``
Having a ``snapshotrestore`` script is very useful in development
environments, but can be harmful in a production buildout. The
script restores the latest snapshot directly to your filestorage
and it used to do this without asking any questions whatsoever
(this has been changed to require an explicit ``yes`` as answer).
If you don't want a ``snapshotrestore`` script, set this option to false.
``enable_zipbackup``
Create ``zipbackup`` and ``ziprestore`` scripts. Default: false.
If ``backup_blobs`` is not on, these scripts are always disabled,
because they are not useful then.
``full``
By default, incremental backups are made. If this option is set to ``true``,
``bin/backup`` will always make a full backup.
``incremental_blobs``
New in version 4.0. Default is false.
When switched on, it will use the ``--listed-incremental`` option of ``tar``.
Note: this only works with the GNU version of ``tar``.
On Mac you may need to install this with ``brew install gnu-tar`` and change your ``PATH`` according to the instructions.
It will create a metadata or `snapshot file <https://www.gnu.org/software/tar/manual/html_node/Incremental-Dumps.html>`_
so that a second call to the backup script will create a second tarball with only the differences.
For some reason, all directories always end up in the second tarball,
even when there are no changes; this may depend on the used file system.
This option is ignored when the ``archive_blob`` option is false.
This option *requires* the ``blob_timestamps`` option to be true,
because it needs the tarball names to be stable, instead of getting rotated.
If you have explicitly set ``blob_timestamps`` to false, buildout will exit with an error.
Note that the ``latest`` symlink to the most recent backup is not created with ``incremental_blobs`` true.
For large blobstorages it may take long to restore, so do test it out.
But that is wise in all cases.
Essentially, this feature seems to trade off storage space reduction with restore time.
``keep``
Number of full backups to keep. Defaults to ``2``, which means that the
current and the previous full backup are kept. Older backups are removed,
including their incremental backups. Set it to ``0`` to keep all backups.
``keep_blob_days``
Number of *days* of blob backups to keep. Defaults to ``14``, so
two weeks. This is **only** used for partial (full=False)
backups, so this is what gets used normally when you do a
``bin/backup``. This option has been added in 2.2. For full
backups (snapshots) we just use the ``keep`` option. Recommended
is to keep these values in sync with how often you do a ``zeopack`` on
the ``Data.fs``, according to the formula ``keep *
days_between_zeopacks = keep_blob_days``. The default matches one
zeopack per seven days (``2*7=14``).
Since version 4.0, this option is ignored unless ``only_blobs`` is true.
Instead, we remove the blob backups that have no matching filestorage backup.
``location``
Location where backups are stored. Defaults to ``var/backups`` inside the
buildout directory.
``locationprefix``
Location of the folder where all other backup and snapshot folders will
be created. Defaults to ``var/``.
Note that this does not influence where we look for a source filestorage or blobstorage.
``only_blobs``
Only backup the blobstorage, not the ``Data.fs`` filestorage. False
by default. May be a useful option if for example you want to
create one ``bin/filestoragebackup`` script and one
``bin/blobstoragebackup`` script, using ``only_blobs`` in one and
``backup_blobs`` in the other.
``post_command``
Command to execute after the backup has finished. One use case
would be to unmount the remote file system that you mounted
earlier using the ``pre_command``. See that ``pre_command`` above for
more info.
``pre_command``
Command to execute before starting the backup. One use case would
be to mount a remote file system using NFS or sshfs and put the
backup there. Any output will be printed. If you do not like
that, you can always redirect output somewhere else (``mycommand >
/dev/null`` on Unix). Refer to your local Unix guru for more
information. If the command fails, the backup script quits with
an error. You can specify multiple commands.
``quick``
Call ``repozo`` with the ``--quick`` option. This option was
introduced to ``collective.recipe.backup`` in version 2.19, with
**default value true**. Due to all the checksums that the repozo
default non-quick behavior does, an amount of data is read that is
three to four times as much as is in the actual filestorage. With
the quick option it could easily be just a few kilobytes.
Theoretically the quick option is less safe, but it looks like it
can only go wrong when someone edits the ``.dat`` file in the
repository or removes a ``.deltafs`` file.
The ``quick`` option only influences the created ``bin/backup``
script. It has no effect on the snapshot or restore scripts.
The repozo help says about this option: "Verify via md5 checksum
only the last incremental written. This significantly reduces the
disk i/o at the (theoretical) cost of inconsistency. This is a
probabilistic way of determining whether a full backup is
necessary."
``rsync_options``
Add extra options to the default ``rsync -a`` command. Default is no
extra parameters. This can be useful for example when you want to restore
a backup from a symlinked directory, in which case
``rsync_options = --no-l -k`` does the trick.
``rsync_hard_links_on_first_copy``
When using ``rsync``, the blob files for the first backup are copied
and then subsequent backups make use of hard links from this initial
copy, to save time and disk space.
Enable this option to also use hard links for the initial copy to further reduce
disk usage.
This is safe for ZODB blobs, since they are not modified in place.
The ``blob_storage`` and the backup folder ``blobbackuplocation``
have to be in the same partition for hard links to be possible.
``snapshotlocation``
Location where snapshot backups of the filestorage are stored. Defaults to
``var/snapshotbackups`` inside the buildout directory.
``use_rsync``
Use ``rsync`` with hard links for backing up the blobs. Default is
true. ``rsync`` is probably not available on all machines though, and
I guess hard links will not work on Windows. When you set this to
false, we fall back to a simple copy (``shutil.copytree`` from
Python in fact).
``ziplocation``
Location where zip backups of the filestorage are stored. Defaults to
``var/zipbackups`` inside the buildout directory.
An example buildout snippet using various options, would look like this::
[backup]
recipe = collective.recipe.backup
location = ${buildout:directory}/myproject
keep = 2
datafs = subfolder/myproject.fs
full = true
debug = true
snapshotlocation = snap/my
enable_snapshotrestore = true
pre_command = echo 'Can I have a backup?'
post_command =
echo 'Thanks a lot for the backup.'
echo 'We are done.'
Paths in directories or files can use relative (``../``) paths, and
``~`` (home dir) and ``$BACKUP``-style environment variables are
expanded.
Cron job integration
====================
``bin/backup`` is of course ideal to put in your cronjob instead of a whole
``bin/repozo ....`` line. But you don't want the "INFO" level logging that you
get, as you'll get that in your mailbox. In your cronjob, just add ``-q`` or
``--quiet``, and ``bin/backup`` will shut up unless there's a problem.
This option ignores the debug variable, if set to true in buildout.
Speaking of cron jobs? Take a look at `zc.recipe.usercrontab
<http://pypi.python.org/pypi/z3c.recipe.usercrontab>`_ if you want to handle
cronjobs from within your buildout. For example::
[backupcronjob]
recipe = z3c.recipe.usercrontab
times = 0 12 * * *
command = ${buildout:directory}/bin/backup
Blob storage
============
Added in version 2.0.
We can backup the blob storage. Plone 4 uses a blob storage to store
files (Binary Large OBjects) on the file system. In Plone 3 this is
optional. When this is used, it should be backed up of course. You
must specify the source blob_storage directory where Plone (or Zope)
stores its blobs. As indicated earlier, when we do not set it
specifically, we try to get the location from other parts, for example
the ``plone.recipe.zope2instance`` recipe::
[buildout]
parts = instance backup
[instance]
recipe = plone.recipe.zope2instance
user = admin:admin
blob-storage = ${buildout:directory}/var/somewhere
[backup]
recipe = collective.recipe.backup
If needed, we can tell buildout that we *only* want to backup blobs or
specifically do *not* want to backup the blobs. Specifying this using
the ``backup_blobs`` and ``only_blobs`` options might be useful in
case you want to separate this into several scripts::
[buildout]
newest = false
parts = filebackup blobbackup
[filebackup]
recipe = collective.recipe.backup
backup_blobs = false
[blobbackup]
recipe = collective.recipe.backup
blob_storage = ${buildout:directory}/var/blobstorage
only_blobs = true
With this setup ``bin/filebackup`` now only backs up the filestorage
and ``bin/blobbackup`` only backs up the blobstorage.
New in version 4.0: you may want to specify ``blob_timestamps = true``.
Then we create stable directories that we do not rotate.
For example: ``blobstorage.1972-12-25-01-02-03`` instead of ``blobstorage.0``.
rsync
=====
By default we use ``rsync`` to create backups. We create hard links
with this tool, to save disk space and still have incremental backups.
This probably requires a unixy (Linux, Mac OS X) operating system.
It is based on this article by Mike Rubel:
http://www.mikerubel.org/computers/rsync_snapshots/
We have not tried this on Windows. Reports are welcome, but best is
probably to set the ``use_rsync = false`` option in the backup part.
Then we simply copy the blobstorage directory.
Alternative restore source
==========================
Added in version 2.17.
Changed in version 5: only one source is supported.
You can restore from an alternative source. Use case: first make a
backup of your production site, then go to the testing or staging
server and restore the production data there.
In the ``alternative_restore_source`` option you can define different
filestorage and blobstorage backup source directories using this
syntax::
alternative_restore_source =
storagename datafs1_backup [blobdir1_backup]
The storagename must be ``Data`` (or ``1``) for the standard ``Data.fs``
and optionally its blobstorage.
The result is a ``bin/altrestore`` script.
This will work for a standard buildout with a single filestorage and
blobstorage::
[backup]
recipe = collective.recipe.backup
alternative_restore_source =
Data /path/to/production/var/backups /path/to/production/var/blobstoragebackups
The above configuration uses ``repozo`` to restore the Data.fs from
the ``/path/to/production/var/backups`` repository to the standard
``var/filestorage/Data.fs`` location. It copies the most recent
blobstorage backup from
``/path/to/production/var/blobstoragebackups/`` to the standard
``var/blobstorage`` location.
Calling the script with a specific date is supported just like the
normal restore script::
bin/altrestore 2000-12-31-23-59
The recipe will fail if the alternative source does not match the
standard filestorage and blobstorage. For
example, you get an error when the ``alternative_restore_source`` is
missing the ``Data`` key, when it has an extra key, when a
key has no paths, when a key has an extra or missing blobstorage.
During install of the recipe, so during the ``bin/buildout`` run, it
does not check if the sources exist: you might have the production
backups on a different server and need to setup a remote shared
directory, or you copy the data over manually.
Note that the script takes the ``archive_blob`` and ``use_rsync`` options
into account. So if the alternative restore source contains a blob
backup that was made with ``archive_blob = true``, you need an
``altrestore`` script that also uses this setting.
Contributors
************
collective.recipe.backup is basically a port of ye olde instancemanager_'s
backup functionality. That backup functionality was coded mostly by Reinout
van Rees and Maurits van Rees, both from `Zest software`_
Creating the buildout recipe was done by Reinout_ with some fixes by Maurits_, who is now the main developer and maintainer.
The snapshotrestore script was added by Nejc Zupan (niteoweb_).
The fullbackup script was added by `Tom 'Spanky' Kapanka`_.
Archive blob backups feature added by `Matej Cotman`_ (niteoweb_).
Sponsorship
===========
Work on collective.recipe.backup has been made possible by Ghent University, or UGent.
See https://www.ugent.be.
Ghent University is a top 100 university and one of the major universities in Belgium.
.. Links used above.
.. _Zest software: http://zestsoftware.nl/
.. _Reinout: http://reinout.vanrees.org/
.. _Maurits: http://maurits.vanrees.org/
.. _instancemanager: https://pypi.python.org/pypi/instancemanager
.. _`Tom 'Spanky' Kapanka`: https://github.com/spanktar
.. _`Sylvain Bouchard`: https://github.com/bouchardsyl
.. _`Matej Cotman`: https://github.com/matejc
.. _niteoweb: http://www.niteoweb.com
Change history
**************
.. NOTE: You should *NOT* be adding new change log entries to this file, this
file is managed by towncrier. You *may* edit previous change logs to
fix problems like typo corrections or such.
To add a new change log entry, please see the notes from the ``pip`` project at
https://pip.pypa.io/en/latest/development/#adding-a-news-entry
.. towncrier release notes start
5.0.0 (2023-06-12)
==================
Bug fixes:
- Final release, no changes since previous alpha. [maurits]
5.0.0a3 (2023-03-15)
====================
Breaking changes:
- Removed ``additional_filestorages`` option.
Removed support in ``alternative_restore_sources`` for multiple storages: only the standard single filestorage plus blobstorage is supported.
Renamed ``alternative_restore_sources`` to ``alternative_restore_source``, keeping the old alias for now.
[maurits] (`Issue #64 <https://github.com/collective/collective.recipe.backup/issues/64>`_)
- Removed ``gzip`` option.
From now on, we always call repozo with the gzip option, you can no longer turn this off.
[maurits] (`Issue #67 <https://github.com/collective/collective.recipe.backup/issues/67>`_)
- Removed option ``gzip_blob``.
This was a backwards compatibility alias for ``archive_blob``.
You should use that now it you want to create a tar archive.
Additionally use the ``compress_blob`` option if you are really sure you want to compress the archive.
[maurits] (`Issue #68 <https://github.com/collective/collective.recipe.backup/issues/68>`_)
- Removed option ``quick``.
From now on, we always call repozo with the quick option for normal backups, you can no longer turn this off.
For full backups (snapshotbackups and zipbackups), the quick option is not used.
[maurits] (`Issue #69 <https://github.com/collective/collective.recipe.backup/issues/69>`_)
Bug fixes:
- Deprecate setting blob_timestamps to false.
See `issue 65 <https://github.com/collective/collective.recipe.backup/issues/65>`_.
[maurits] (`Issue #65 <https://github.com/collective/collective.recipe.backup/issues/65>`_)
5.0.0a2 (2023-02-01)
====================
Bug fixes:
- Add missing changelog entries for release 5.0.0a1. [maurits]
5.0.0a1 (2022-11-08)
====================
Breaking changes:
- Drop support for Python 2.7-3.7, add support for 3.10.
[maurits] (`Issue #62 <https://github.com/collective/collective.recipe.backup/issues/62>`_)
- No longer support the ``enable_fullbackup`` option and the ``fullbackup`` script.
A full backup is automatically made by the first call of ``bin/backup`` after you have done a zeopack.
If you still want such a script, look at ``bin/snapshotbackup`` script or use the `full` option, optionally in a second buildout section.
[maurits] (`Issue #66 <https://github.com/collective/collective.recipe.backup/issues/66>`_)
New features:
- Added an option to use hard links for the first backup copy
[ale-rt] (`Issue #57 <https://github.com/collective/collective.recipe.backup/issues/57>`_)
- Add support for Python 3.11. [maurits] (`Issue #311 <https://github.com/collective/collective.recipe.backup/issues/311>`_)
Bug fixes:
- Drop code-analysis, add other QA checks: isort, black, flake8.
[maurits] (`Issue #63 <https://github.com/collective/collective.recipe.backup/issues/63>`_)
4.2.0 (2021-08-30)
==================
New features:
- Changed default of ``blob_timestamps`` to true.
We might remove the previous behavior (creating ``blobstorage.0``, ``blobstorage.1`` etcetera) completely in version 5.
[maurits] (`Issue #61 <https://github.com/collective/collective.recipe.backup/issues/61>`_)
Bug fixes:
- Switch the test infrastructure to GitHub Actions and tox.
Also test Python 3.8 and 3.9. [ale-rt, maurits] (`Issue #58 <https://github.com/collective/collective.recipe.backup/issues/58>`_)
- Fixed ``latest`` symlink when using options ``blob_timestamps`` and ``no_rsync``.
Previously it pointed to for example ``blobstorage.2021-01-02-03-04-05/blobstorage`` instead of the directory above it.
[maurits] (`Issue #61 <https://github.com/collective/collective.recipe.backup/issues/61>`_)
- When creating a zipbackup, do not create a blob timestamp.
We always only keep one zipbackup, so there is no need for timestamps to know which filestorage backup it belongs to.
[maurits] (`Issue #61 <https://github.com/collective/collective.recipe.backup/issues/61>`_)
4.1.1 (2019-10-07)
==================
Bug fixes:
- When using timestamps, use correct filestorage backup location for blob timestamp of snapshot and zip backup.
[maurits] (`Issue #55 <https://github.com/collective/collective.recipe.backup/issues/55>`_)
4.1.0 (2019-04-10)
==================
New features:
- Create symlink to latest timestamped blobstorage backup.
When ``blob_timestamps`` is false, ``blobstorage.0`` is a stable filename,
but with ``blob_timestamps`` true, such a stable name was missing.
[maurits] (`Issue #48 <https://github.com/collective/collective.recipe.backup/issues/48>`_)
- Dropped official Python 2.6 support, added Python 3.7 support.
No code change, but we are no longer testing 2.6.
[maurits] (`Issue #49 <https://github.com/collective/collective.recipe.backup/issues/49>`_)
Bug fixes:
- Updated package versions in test buildout.
Improved code quality according to flake8 and with black.
[maurits] (`Issue #49 <https://github.com/collective/collective.recipe.backup/issues/49>`_)
- Removed already disabled support for sitepackage_safe_scripts. [maurits] (`Issue #51 <https://github.com/collective/collective.recipe.backup/issues/51>`_)
4.0.1 (2018-04-19)
==================
Bug fixes:
- With archive_blob true, still allow restoring non archives. We prefer
archives then, but if no proper archive is found, we look for non archives.
[maurits] (`Issue #44
<https://github.com/collective/collective.recipe.backup/issues/44>`_)
- Sort os.listdir, for improved functioning on Mac. [maurits] (`Issue #45
<https://github.com/collective/collective.recipe.backup/issues/45>`_)
4.0 (2017-12-22)
================
- Updated readme and added sponsorship note. [maurits]
4.0b5 (2017-11-17)
==================
- Added ``incremental_blobs`` option.
This creates tarballs with only the changes compared to the previous blob backups.
This option is ignored when the ``archive_blob`` option is false.
[maurits]
- Improved code quality, reducing complexity. [maurits]
- Refactored the main functions to not have so much code duplication.
The normal, full, snapshot and zip backups had almost the same code.
This made it hard to add new options.
[maurits]
4.0b4 (2017-08-18)
==================
- Test Python 3.6 (and 2.6 and 2.7) on Travis from now on. [maurits]
- Ignore the zope2instance recipe integration tests on Python 3.
They would need a compatible ``mailinglogger`` package.
See `issue #31 <https://github.com/collective/collective.recipe.backup/issues/31>`_. [maurits]
- Tests: use cleaner way to check the mock repozo output.
Share this setup between tests.
This makes the output order the same on Python 2 and 3.
See `issue #31 <https://github.com/collective/collective.recipe.backup/issues/31>`_. [maurits]
4.0b3 (2017-07-05)
==================
- Added basic Python 3 support. We do not test with it yet,
but you should not get NameErrors anymore.
See `issue #31 <https://github.com/collective/collective.recipe.backup/issues/31>`_. [maurits]
4.0b2 (2017-06-26)
==================
- No longer create the ``fullbackup`` script by default.
You can still enable it by setting ``enable_fullbackup`` to ``true``.
[maurits]
- Without explicit ``blob-storage`` option, default to ``var/blobstorage``.
Take the ``var`` option from zeoserver/client recipes into account.
Fixes `issue #27 <https://github.com/collective/collective.recipe.backup/issues/27>`_.
[maurits]
- Do not create hidden backup ``.0`` when blob_storage ends with a slash.
Fixes `issue #26 <https://github.com/collective/collective.recipe.backup/issues/26>`_.
[maurits]
4.0b1 (2017-05-31)
==================
- Make custom backup locations relative to the ``locationprefix`` option or the ``var`` directory.
Until now, the ``locationprefix`` option was only used if you did not set custom locations.
Custom location would be relative to the buildout directory.
Now they are relative to the ``locationprefix`` option, with the ``var`` directory as default.
So if you used a relative path, your backups may end up in a different path.
Absolute paths are not affected: they ignore the locationprefix.
[maurits]
- When log level is DEBUG, show time stamps in the log. [maurits]
- Added ``compress_blob`` option. Default is false.
This is only used when the ``archive_blob`` option is true.
When switched on, it will compress the archive,
resulting in a ``.tar.gz`` instead of a ``tar`` file.
When restoring, we always look for both compressed and normal archives.
We used to always compress them, but in most cases it hardly decreases the size
and it takes a long time anyway. I have seen archiving take 15 seconds,
and compressing take an additional 45 seconds.
The result was an archive of 5.0 GB instead of 5.1 GB.
[maurits]
- Renamed ``gzip_blob`` option to ``archive_blob``.
Kept the old name as alias for backwards compatibility.
This makes room for letting this create an archive without zipping it.
[maurits]
- Automatically remove old blobs backups that have no corresponding filestorage backup.
We compare the timestamp of the oldest filestorage backup with the timestamps of the
blob backups. This can be the name, if you use ``blob_timestamps = true``,
or the modification date of the blob backup.
This means that the ``keep_blob_days`` option is ignored, unless you use ``only_blobs = true``.
[maurits]
- When backing up a blobstorage, use the timestamp of the latest filestorage backup.
If a blob backup with that name is already there, then there were no database changes,
so we do not make a backup.
This is only done when you use the new ``blob_timestamps = true`` option.
[maurits]
- When restoring to a specific date, find the first blob backup at or before
the specified date. Otherwise fail. The repozo script does the same.
We used to pick the first blob backup *after* the specified date,
because we assumed that the user would specify the exact date that is
in the filestorage backup.
Note that the timestamp of the filestorage and blobstorage backups may be
a few seconds apart, unless you use the ``blob_timestamps == true`` option.
In the new situation, the user should pick the date of the blob backup
or slightly later.
[maurits]
- Added ``blob_timestamps`` option. Default is false.
By default we create ``blobstorage.0``.
The next time, we rotate this to ``blobstorage.1`` and create a new ``blobstorage.0``.
With ``blob_timestamps = true``, we create stable directories that we do not rotate.
They get a timestamp, the same timestamp that the ZODB filestorage backup gets.
For example: ``blobstorage.1972-12-25-01-02-03``.
[maurits]
- When restoring, first run checks for all filestorages and blobstorages.
When one of the backups is missing, we quit with an error.
This avoids restoring a filestorage and then getting into trouble
due to a missing blobstorage backup. [maurits]
3.1 (2017-02-24)
================
- Add a ``locationprefix`` option to configure a folder where all other
backup and snapshot folders will be created [erral]
- Only claim compatibility with Python 2.6 and 2.7. [maurits]
- Updated test buildout to use most recent versions. [maurits]
3.0.0 (2015-12-31)
==================
- Refactored the init and install methods of this recipe. During the
init phase we were reading the buildout configuration, but during
this phase the configuration is still being build. So differences
could occur, especially in the order of execution of parts. This
was not good. Most code is now moved from the init to the install
(and update) method. This has less possible problems. Downside:
some configuration errors are caught later.
[maurits]
- Read ``zeo-var``, ``var``, ``file-storage`` from buildout sections.
Update default backup and Data.fs locations based on this.
[maurits]
2.22 (2015-12-30)
=================
- Do not accept ``backup_blobs`` false and ``enable_zipbackup`` true.
The zipbackup script is useless without blobs.
[maurits]
- Set default ``backup_blobs`` to true on Python 2.6 (Plone 4) and
higher. Otherwise false. If no ``blob_storage`` can be found, we
quit with an error.
[maurits]
- Accept ``true``, ``yes``, ``on``, ``1``, in lower, upper or mixed
case as true value. Treat all other values in the buildout options
as false.
[maurits]
- Find plone.recipe.zope2instance recipes also when they are not
completely lower case. The zope2instance recipe itself works fine
when it has mixed case, so we should accept this too.
[maurits]
2.21 (2015-10-06)
=================
- When restoring, create ``var/filestorage`` if needed.
Fixes #23.
[maurits]
2.20 (2014-11-11)
=================
- Add ``enable_fullbackup`` option. Default: true, so no change
compared to previous version.
[maurits]
- Create backup/snapshot/zipbackup directories only when needed.
Running the backup script should not create the snapshot
directories.
[maurits]
- Add zipbackup and ziprestore scripts when ``enable_zipbackup = true``.
[maurits]
2.19 (2014-06-16)
=================
- Call repozo with ``--quick`` when making an incremental backup.
This is a lot faster. Theoretically it lead to inconsistency if
someone is messing in your backup directory. You can return to the
previous behavior by specifying ``quick = false`` in the backup
recipe part in your buildout config.
[maurits]
- check and create folders now happens after pre_commands is run
[@djay]
2.18 (2014-04-29)
=================
- Add ``rsync_options`` option. These are added to the default
``rsync -a`` command. Default is no extra parameters. This can be
useful for example when you want to restore a backup from a
symlinked directory, in which case ``rsync_options = --no-l -k``
does the trick.
[fiterbek]
2.17 (2014-02-07)
=================
- Add ``alternative_restore_sources`` option. This creates a
``bin/altrestore`` script that restores from an alternative backup
location, specified by that option. You can use this to restore a
backup of the production data to your testing or staging server.
[maurits]
- When checking if the backup script will be able to create a path,
remove all created directories. Until now, only the final directory
was removed, and not any created parent directories.
[maurits]
- Testing: split the single big doctest file into multiple files, to
make the automated tests less dependent on one another, making it
easier to change them and add new ones.
[maurits]
- No longer test with Python 2.4, because Travis does not support it
out of the box. Should still work fine.
[maurits]
2.16 (2014-01-14)
=================
- Do not create blob backup dirs when not backing up blobs.
Do not create filestorage backup dirs when not backing up filestorage.
Fixes https://github.com/collective/collective.recipe.backup/issues/17
[maurits]
2.15 (2013-09-16)
=================
- Restore compatibility with Python 2.4 (Plone 3).
[maurits]
2.14 (2013-09-09)
=================
- Archive blob backups with buildout option ``gzip_blob``.
[matejc]
2.13 (2013-07-15)
=================
- When printing that we halt the execution due to an error running
repozo, actually halt the execution.
[maurits]
2.12 (2013-06-28)
=================
- Backup directories are now created when we launch ``backup`` or
``fullbackup`` or ``snapshotbackup`` scripts, no more during
initialization.
[bsuttor]
2.11 (2013-05-06)
=================
- Print the names of filestorages and blobstorages that will be
restored. Issue #8.
[maurits]
- Added a new command-line argument : ``--no-prompt`` disables user
input when restoring a backup or snapshot. Useful for shell scripts.
[bouchardsyl]
- Fixed command-line behavior with many arguments and not only a date.
[bouchardsyl]
2.10 (2013-03-30)
=================
- Added ``fullbackup`` script that defaults to ``full=true``. This
could have been handled by making a new part, but it seemed like
overkill to have to generate a complete new set of backup scripts,
just to get one for full.
[spanky]
2.9 (2013-03-06)
================
- Fixed possible KeyError: ``blob_snapshot_location``.
[gforcada]
2.8 (2012-11-13)
================
- Fixed possible KeyError: ``blob_backup_location``.
https://github.com/collective/collective.recipe.backup/issues/3
[maurits]
2.7 (2012-09-27)
================
- additional_filestorages improved: blob support and custom location.
[mamico]
2.6 (2012-08-29)
================
- Added pre_command and post_command options. See the documentation.
[maurits]
2.5 (2012-08-08)
================
- Moved code to github:
https://github.com/collective/collective.recipe.backup
[maurits]
2.4 (2011-12-20)
================
- Fixed silly indentation error that prevented old blob backups from
being deleted when older than ``keep_blob_days`` days.
[maurits]
2.3 (2011-10-05)
================
- Quit the rest of the backup or restore when a repozo call gives an
error. Main use case: when restoring to a specific date repozo will
quit with an error when no files can be found, so we should also not
try to restore blobs then.
[maurits]
- Allow restoring the blobs to the specified date as well.
[maurits]
2.2 (2011-09-14)
================
- Refactored script generation to make a split between initialization
code and script arguments. This restores compatibility with
zc.buildout 1.5 for system pythons. Actually we no longer create so
called 'site package safe scripts' but just normal scripts that work
for all zc.buildout versions.
[maurits]
- Added option ``keep_blob_days``, which by default specifies that
only for partial backups we keep 14 days of backups. See the
documentation.
[maurits]
- Remove old blob backups when doing a snapshot backup.
[maurits]
2.1 (2011-09-01)
================
- Raise an error when the four backup location options
(blobbackuplocation, blobsnapshotlocation, location and
snapshotlocation) are not four distinct locations (or empty
strings).
[maurits]
- Fixed possible TypeError: 'Option values must be strings'.
Found by Alex Clark, thanks.
[maurits]
2.0 (2011-08-26)
================
- Backup and restore blobs, using rsync.
[maurits]
- Ask if the user is sure before doing a restore.
[maurits]
1.7 (2010-12-10)
================
- Fix generated repozo commands to work also
when recipe is configured to have a non **Data.fs**
main db plus additional filestorages.
e.g.:
datafs= var/filestorage/main.fs
additional = catalog
[hplocher]
1.6 (2010-09-21)
================
- Added the option enable_snapshotrestore so that the creation of the
script can be removed. Backwards compatible, if you don't specify it
the script will still be created. Rationale: you may not want this
script in a production buildout where mistakenly using
snapshotrestore instead of snapshotbackup could hurt.
[fredvd]
1.5 (2010-09-08)
================
- Fix: when running buildout with a config in a separate directory
(like ``bin/buildout -c conf/prod.cfg``) the default backup
directories are no longer created inside that separate directory.
If you previously manually specified one of the location,
snapshotlocation, or datafs parameters to work around this, you can
probably remove those lines. So: slightly saner defaults.
[maurits]
1.4 (2010-08-06)
================
- Added documentation about how to get the required bin/repozo script
in your buildout if for some reason you do not have it yet (like on
Plone 4 when you do not have a zeo setup).
Thanks to Vincent Fretin for the extra buildout lines.
[maurits]
1.3 (2009-12-08)
================
- Added snapshotrestore script. [Nejc Zupan]
1.2 (2009-10-26)
================
- The part name is now reflected in the created scripts and var/ directories.
Originally bin/backup, bin/snapshotbackup, bin/restore and var/backups
plus var/snapshotbackups were hardcoded. Those are still there when you
name your part ``[backup]``. With a part named ``[NAME]``, you get
bin/NAME, bin/NAME-snapshot, bin/NAME-restore and var/NAMEs plus
var/NAME-snapshots. Request by aclark for plone.org. [reinout]
1.1 (2009-08-21)
================
- Run the cleanup script (removing too old backups that we no longer
want to keep) for additional file storages as well.
Fixes https://bugs.launchpad.net/collective.buildout/+bug/408224
[maurits]
- Moved everything into a src/ subdirectory to ease testing on buildbot (which
would grab all egss in the eggs/ dir that buildbot's mechanism creates.
[reinout]
1.0 (2009-02-06)
================
- Quote all paths and arguments so that it works on paths that contain
spaces (specially on Windows). [sidnei]
0.9 (2008-12-05)
================
- Windows path compatibility fix. [Juan A. Diaz]
0.8 (2008-09-23)
================
- Changed the default for gzipping to True. Adding ``gzip = true`` to all our
server deployment configs gets tired pretty quickly, so doing it by default
is the best default. Stuff like this needs to be changed **before** a 1.0
release :-) [reinout]
- Backup of additional databases (if you have configured them) now takes place
before the backup of the main database (same with restore). [reinout]
0.7 (2008-09-19)
================
- Added $BACKUP-style enviroment variable subsitution in addition to the tilde
expansion offered by 0.6. [reinout, idea by Fred van Dijk]
0.6 (2008-09-19)
================
- Fixed the test setup so both bin/test and python setup.py test
work. [reinout+maurits]
- Added support for ~ in path names. And fixed a bug at the same time that
would occur if you call the backup script from a different location than
your buildout directory in combination with a non-absolute backup
location. [reinout]
0.5 (2008-09-18)
================
- Added support for additional_filestorages option, needed for for instance a
split-out catalog.fs. [reinout]
- Test setup fixes. [reinout+maurits]
0.4 (2008-08-19)
================
- Allowed the user to make the script more quiet (say in a cronjob)
by using 'bin/backup -q' (or --quiet). [maurits]
- Refactored initialization template so it is easier to change. [maurits]
0.3.1 (2008-07-04)
==================
- Added 'gzip' option, including changes to the cleanup functionality that
treats .fsz also as a full backup like .fs. [reinout]
- Fixed typo: repoze is now repozo everywhere... [reinout]
0.2 (2008-07-03)
================
- Extra tests and documentation change for 'keep': the default is to keep 2
backups instead of all backups. [reinout]
- If debug=true, then repozo is also run in --verbose mode. [reinout]
0.1 (2008-07-03)
================
- Added bin/restore. [reinout]
- Added snapshot backups. [reinout]
- Enabled cleaning up of older backups. [reinout]
- First working version that runs repozo and that creates a backup dir if
needed. [reinout]
- Started project based on zopeskel template. [reinout]
Raw data
{
"_id": null,
"home_page": "https://github.com/collective/collective.recipe.backup",
"name": "collective.recipe.backup",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": "",
"keywords": "buildout backup repozo zope",
"author": "Reinout van Rees, Maurits van Rees",
"author_email": "m.van.rees@zestsoftware.nl",
"download_url": "https://files.pythonhosted.org/packages/a8/9b/747858a73b1cda6967fc9db18360da5ec8d703552be1de3244678747967c/collective.recipe.backup-5.0.0.tar.gz",
"platform": null,
"description": "Easy Zope backup/restore recipe for buildout\n********************************************\n\n.. image:: https://github.com/collective/collective.recipe.backup/workflows/tests/badge.svg\n :target: https://github.com/collective/collective.recipe.backup/actions?query=workflow%3Atests\n\n.. contents::\n\n\nIntroduction\n============\n\nThis recipe is mostly a wrapper around the ``bin/repozo`` script in\nyour Zope buildout. It requires that this script is already made\navailable. If this is not the case, you will get an error like this\nwhen you run one of the scripts: ``bin/repozo: No such file or\ndirectory``. This should be there when you\nare using ``plone.recipe.zeoserver``. If this is\nnot the case, the easiest way of getting a ``bin/repozo`` script is to\nadd a new section in your ``buildout.cfg`` (do not forget to add it in the\n``parts`` directive)::\n\n [repozo]\n recipe = zc.recipe.egg\n eggs = ZODB\n # or this for an older version:\n # eggs = ZODB3\n scripts = repozo\n dependent-scripts = true\n\n``bin/repozo`` is a Zope script to make backups of your ``Data.fs``.\nLooking up the settings can be a chore. And you have to pick a\ndirectory where to put the backups. This recipe provides **sensible\ndefaults** for your common backup tasks. Making backups a piece of\ncake is important!\n\n- ``bin/backup`` makes an incremental backup.\n\n- ``bin/restore`` restores the latest backup created by the backup script.\n\n- ``bin/snapshotbackup`` makes a full snapshot backup, separate from the\n regular backups. Handy right before a big change in the site.\n\n- ``bin/snapshotrestore`` restores the latest full snapshot backup.\n\n- ``bin/zipbackup`` makes a zip backup. This zips the Data.fs and the\n blobstorage, handy for copying production data to your local\n machine, especially the blobstorage with its many files.\n Actually, zipping the Data.fs is standard,\n and we do not zip the blobstorage,\n because most files in there are already compressed.\n But we do combine the blobs in one tar archive.\n *Note*: the Data.fs and blobstorage (or other storages) are *not* combined\n in one file; you need to download multiple files. Enable this\n script by using the ``enable_zipbackup`` option.\n\n- ``bin/ziprestore`` restores the latest zipbackup.\n\n\nCompatibility\n=============\n\nThe recipe is tested with Python 2.7, 3.6, 3.7, 3.8, 3.9.\nIn Plone terms it works fine on Plone 4, 5 and 6.\n\nNote that the integration with ``plone.recipe.zope2instance`` is not tested on Python 3.\nIt would pulls in too many dependencies, like Zope and ZODB.\n\n\nDevelopment\n===========\n\n- Code repository: https://github.com/collective/collective.recipe.backup\n\n- Issue tracker: https://github.com/collective/collective.recipe.backup/issues\n\n- Obvious fixes, like fixing typos, are fine on master.\n For larger changes or if you are unsure, please create a branch or a pull request.\n\n- The code comes with a ``buildout.cfg``. Please bootstrap the\n buildout and run the created ``bin/test`` to see if the tests still\n pass. Please try to add tests if you add code.\n To run the tests for all supported Python versions, run ``tox``.\n\n- The long description of this package (as shown on PyPI), used to\n contain a big file with lots of test code that showed how to use the\n recipe. This grew too large, so we left it out. It is probably\n still good reading if you are wondering about the effect some\n options have. See ``src/collective/recipe/backup/tests/*.rst``.\n\n- We are tested on GitHub Actions:\n https://github.com/collective/collective.recipe.backup/actions\n\n- Questions and comments to https://community.plone.org or to\n `Maurits van Rees <mailto:maurits@vanrees.org>`_.\n\n\nExample usage\n=============\n\nThe simplest way to use this recipe is to add a part in ``buildout.cfg`` like this::\n\n [buildout]\n parts = backup\n\n [backup]\n recipe = collective.recipe.backup\n\nYou can set lots of extra options, but the recipe authors like to\nthink they have created sane defaults, so this single line stating the\nrecipe name should be enough in most cases.\n\nRunning the buildout adds the ``backup``,\n``snapshotbackup``, ``zipbackup``, ``restore``, ``snapshotrestore``\nand ``ziprestore`` scripts to the ``bin/`` directory of the buildout.\nSome are not added by default, others can be switched off.\n\n\nBacked up data\n==============\n\nWhich data does this recipe backup?\n\n- The Zope Object DataBase (ZODB) filestorage, by default located at\n ``var/filestorage/Data.fs``.\n\n- The blobstorage (since version 2.0) if your buildout uses it, by\n default located at ``var/blobstorage``.\n\n\nData that is *not* backed up\n============================\n\nWhich data does this recipe *not* backup? Everything else of course,\nbut specifically:\n\n- Data stored in ``RelStorage`` will *not* be backed up. (You could\n still use this recipe to back up the filesystem blobstorage,\n possibly with the ``only_blobs`` option.)\n\n- Other data stored in SQL, perhaps via SQLAlchemy, will *not* be\n backed up.\n\n- It does *not* create a backup of your entire buildout directory.\n\n\nIs your backup backed up?\n=========================\n\nNote that the backups are by default created in the ``var`` directory\nof the buildout, so if you accidentally remove the entire buildout,\nyou also lose your backups. It should be standard practice to use the\n``location`` option to specify a backup location in for example the\nhome directory of the user. You should also arrange to copy that\nbackup to a different machine/country/continent/planet.\n\n\nBackup\n======\n\nCalling ``bin/backup`` results in a normal incremental repozo backup\nthat creates a backup of the ``Data.fs`` in ``var/backups``. When you\nhave a blob storage it is by default backed up to\n``var/blobstoragebackups``.\n\n\nSnapshots\n=========\n\nA quick backup just before updating the production server is a good\nidea. But you may not want to interfere with the regular backup\nregime. For that, the ``bin/snapshotbackup`` is great. It places a\nfull backup in, by default, ``var/snapshotbackups``.\n\n\nZipbackups\n==========\n\nFor quickly grabbing the current state of a production database so you\ncan download it to your development laptop, you want a full and zipped\nbackup. The zipped part is important for the blobstorage, because you\ndo not want to use ``scp`` to recursively copy over all those blob\nfiles: downloading one tarball is faster.\n\nYou can use the ``bin/zipbackup`` script for this. This script\noverrides a few settings, ignoring whatever is set in the buildout\nconfig section:\n\n- ``archive_blob`` is turned on.\n\n- ``keep`` is set to 1 to avoid keeping lots of needless backups.\n\n- ``keep_blob_days`` is ignored because it is a full backup.\n\nThe script places a full backup in, by default, ``var/zipbackups`` and\nit puts a tarball of the blobstorage in ``var/blobstoragezips``.\n\nThis script is not created by default.\nYou can enable it by setting the ``enable_zipbackup`` option to true.\nAlso, if ``backup_blobs`` is false, the scripts are useless, so we do not create them, even when you have enabled them explicitly.\n\n\nRestore\n=======\n\nCalling ``bin/restore`` restores the very latest normal incremental\n``repozo`` backup and restores the blobstorage if you have that.\n\nYou can restore the very latest snapshotbackup with ``bin/snapshotrestore``.\n\nYou can restore the zipbackup with ``bin/ziprestore``.\n\nYou can also restore the backup as of a certain date. Just pass a date argument.\nAccording to ``repozo``: specify UTC (not local) time.\nThe format is ``yyyy-mm-dd[-hh[-mm[-ss]]]``.\nSo as a simple example, restore to 25 december 1972::\n\n bin/restore 1972-12-25\n\nor to that same date, at 2,03 seconds past 1::\n\n bin/restore 1972-12-25-01-02-03\n\nSince version 2.3 this also works for restoring blobs.\nWe restore the directory from the first backup at or before the specified date.\n(Note that before version 4.0 we restored the directory from the first backup after the specified date,\nwhich should be fine as long as you did not do a database pack in between.)\n\nSince version 2.0, the restore scripts ask for confirmation before\nstarting the restore, as this is a potentially dangerous command.\n(\"Oops, I have restored the live site but I meant to restore the test\nsite.\") You need to explicitly type 'yes'::\n\n This will replace the filestorage (Data.fs).\n This will replace the blobstorage.\n Are you sure? (yes/No)?\n\nNote that for large filestorages and blobstorages **it may take long to restore**.\nYou should do a test restore and check how long it takes.\nSeconds? Minutes? Hours?\nIs that time acceptable or should you take other measures?\n\nNames of created scripts\n========================\n\nA backup part will normally be called ``[backup]``, leading to a\n``bin/backup`` and ``bin/snapshotbackup``. Should you name your part\nsomething else, the script names will also be different, as will the created\n``var/`` directories (since version 1.2)::\n\n [buildout]\n parts = plonebackup\n\n [plonebackup]\n recipe = collective.recipe.backup\n enable_zipbackup = true\n\nThat buildout snippet will create these scripts::\n\n bin/plonebackup\n bin/plonebackup-full\n bin/plonebackup-zip\n bin/plonebackup-snapshot\n bin/plonebackup-restore\n bin/plonebackup-ziprestore\n bin/plonebackup-snapshotrestore\n\n\nSupported options\n=================\n\nThe recipe supports the following options, none of which are needed by\ndefault. The most common ones to change are ``location`` and\n``blobbackuplocation``, as those allow you to place your backups in\nsome system-wide directory like ``/var/zopebackups/instancename/`` and\n``/var/zopebackups/instancename-blobs/``.\n\n.. Note: keep this in alphabetical order please.\n\n``archive_blob``\n Use ``tar`` archiving functionality. ``false`` by default. Set it to ``true``\n and backup/restore will be done with ``tar`` command. Note that ``tar``\n command must be available on machine if this option is set to ``true``.\n This option also works with snapshot backup/restore commands. As this\n counts as a full backup ``keep_blob_days`` is ignored.\n See the ``compress_blob`` option if you want to compress the archive.\n\n``alternative_restore_source``\n You can restore from an alternative source. Use case: first make\n a backup of your production site, then go to the testing or\n staging server and restore the production data there. See\n `Alternative restore source`_\n\n``alternative_restore_sources``\n Backwards compatibility spelling for ``alternative_restore_source``.\n This will no longer work in version 7.\n\n``backup_blobs``\n Backup the blob storage. Default is ``True`` on Python 2.6 (Plone\n 4) and higher, and ``False`` otherwise. This requires the\n ``blob_storage`` location to be set. If no ``blob_storage``\n location has been set and we cannot find one by looking in the\n other buildout parts, we quit with an error (since version 2.22).\n If ``backup_blobs`` is false, ``enable_zipbackup`` cannot be true,\n because the ``zipbackup`` script is not useful then.\n\n``blob_storage``\n Location of the directory where the blobs (binary large objects)\n are stored. This is used in Plone 4 and higher, or on Plone 3 if\n you use ``plone.app.blob``. This option is ignored if backup_blobs is\n ``false``. The location is not set by default. When there is a part\n using ``plone.recipe.zeoserver``, ``plone.recipe.zope2instance`` or\n ``plone.recipe.zope2zeoserver``, we check if that has a\n blob-storage option and use that as default. Note that we pick\n the first one that has this option and we do not care about\n shared-blob settings, so there are probably corner cases where we\n do not make the best decision here. Use this option to override\n it in that case.\n\n``blob-storage``\n Alternative spelling for the preferred ``blob_storage``, as\n ``plone.recipe.zope2instance`` spells it as ``blob-storage`` and we are\n using underscores in all the other options. Pick one.\n\n``blob_timestamps``\n New in version 4.0. Default is true (this was false before version 4.2).\n If false, we create ``blobstorage.0``.\n The next time, we rotate this to ``blobstorage.1`` and create a new ``blobstorage.0``.\n With ``blob_timestamps = true``, we create stable directories that we do not rotate.\n They get a timestamp, the same timestamp that the ZODB filestorage backup gets.\n For example: ``blobstorage.1972-12-25-01-02-03``.\n Or with ``archive_blob = true``: ``blobstorage.1972-12-25-01-02-03.tar``.\n Because the filename is unpredictable, since version 4.1 we create a ``latest`` symlink\n to the most recent backup.\n Blob timestamps are not used with zipbackup, because this only keeps 1 backup,\n which means there is no confusion about which filestorage backup it belongs to.\n\n``blobbackuplocation``\n Directory where the blob storage will be backed up to. Defaults\n to ``var/blobstoragebackups`` inside the buildout directory.\n\n``blobsnapshotlocation``\n Directory where the blob storage snapshots will be created.\n Defaults to ``var/blobstoragesnapshots`` inside the buildout\n directory.\n\n``blobziplocation``\n Directory where the blob storage zipbackups will be created.\n Defaults to ``var/blobstoragezips`` inside the buildout\n directory.\n\n``compress_blob``\n New in version 4.0. Default is false.\n This is only used when the ``archive_blob`` option is true.\n When switched on, it will compress the archive,\n resulting in a ``.tar.gz`` instead of a ``tar`` file.\n When restoring, we always look for both compressed and normal archives.\n We used to always compress them, but in most cases it hardly decreases the size\n and it takes a long time anyway. I have seen archiving take 15 seconds,\n and compressing take an additional 45 seconds.\n The result was an archive of 5.0 GB instead of 5.1 GB.\n\n``datafs``\n In case the ``Data.fs`` isn't in the default ``var/filestorage/Data.fs``\n location, this option can overwrite it.\n\n``debug``\n In rare cases when you want to know exactly what's going on, set debug to\n ``true`` to get debug level logging of the recipe itself. ``repozo`` is also run\n with ``--verbose`` if this option is enabled.\n\n``enable_snapshotrestore``\n Having a ``snapshotrestore`` script is very useful in development\n environments, but can be harmful in a production buildout. The\n script restores the latest snapshot directly to your filestorage\n and it used to do this without asking any questions whatsoever\n (this has been changed to require an explicit ``yes`` as answer).\n If you don't want a ``snapshotrestore`` script, set this option to false.\n\n``enable_zipbackup``\n Create ``zipbackup`` and ``ziprestore`` scripts. Default: false.\n If ``backup_blobs`` is not on, these scripts are always disabled,\n because they are not useful then.\n\n``full``\n By default, incremental backups are made. If this option is set to ``true``,\n ``bin/backup`` will always make a full backup.\n\n``incremental_blobs``\n New in version 4.0. Default is false.\n When switched on, it will use the ``--listed-incremental`` option of ``tar``.\n Note: this only works with the GNU version of ``tar``.\n On Mac you may need to install this with ``brew install gnu-tar`` and change your ``PATH`` according to the instructions.\n It will create a metadata or `snapshot file <https://www.gnu.org/software/tar/manual/html_node/Incremental-Dumps.html>`_\n so that a second call to the backup script will create a second tarball with only the differences.\n For some reason, all directories always end up in the second tarball,\n even when there are no changes; this may depend on the used file system.\n This option is ignored when the ``archive_blob`` option is false.\n This option *requires* the ``blob_timestamps`` option to be true,\n because it needs the tarball names to be stable, instead of getting rotated.\n If you have explicitly set ``blob_timestamps`` to false, buildout will exit with an error.\n Note that the ``latest`` symlink to the most recent backup is not created with ``incremental_blobs`` true.\n For large blobstorages it may take long to restore, so do test it out.\n But that is wise in all cases.\n Essentially, this feature seems to trade off storage space reduction with restore time.\n\n``keep``\n Number of full backups to keep. Defaults to ``2``, which means that the\n current and the previous full backup are kept. Older backups are removed,\n including their incremental backups. Set it to ``0`` to keep all backups.\n\n``keep_blob_days``\n Number of *days* of blob backups to keep. Defaults to ``14``, so\n two weeks. This is **only** used for partial (full=False)\n backups, so this is what gets used normally when you do a\n ``bin/backup``. This option has been added in 2.2. For full\n backups (snapshots) we just use the ``keep`` option. Recommended\n is to keep these values in sync with how often you do a ``zeopack`` on\n the ``Data.fs``, according to the formula ``keep *\n days_between_zeopacks = keep_blob_days``. The default matches one\n zeopack per seven days (``2*7=14``).\n Since version 4.0, this option is ignored unless ``only_blobs`` is true.\n Instead, we remove the blob backups that have no matching filestorage backup.\n\n``location``\n Location where backups are stored. Defaults to ``var/backups`` inside the\n buildout directory.\n\n``locationprefix``\n Location of the folder where all other backup and snapshot folders will\n be created. Defaults to ``var/``.\n Note that this does not influence where we look for a source filestorage or blobstorage.\n\n``only_blobs``\n Only backup the blobstorage, not the ``Data.fs`` filestorage. False\n by default. May be a useful option if for example you want to\n create one ``bin/filestoragebackup`` script and one\n ``bin/blobstoragebackup`` script, using ``only_blobs`` in one and\n ``backup_blobs`` in the other.\n\n``post_command``\n Command to execute after the backup has finished. One use case\n would be to unmount the remote file system that you mounted\n earlier using the ``pre_command``. See that ``pre_command`` above for\n more info.\n\n``pre_command``\n Command to execute before starting the backup. One use case would\n be to mount a remote file system using NFS or sshfs and put the\n backup there. Any output will be printed. If you do not like\n that, you can always redirect output somewhere else (``mycommand >\n /dev/null`` on Unix). Refer to your local Unix guru for more\n information. If the command fails, the backup script quits with\n an error. You can specify multiple commands.\n\n``quick``\n Call ``repozo`` with the ``--quick`` option. This option was\n introduced to ``collective.recipe.backup`` in version 2.19, with\n **default value true**. Due to all the checksums that the repozo\n default non-quick behavior does, an amount of data is read that is\n three to four times as much as is in the actual filestorage. With\n the quick option it could easily be just a few kilobytes.\n Theoretically the quick option is less safe, but it looks like it\n can only go wrong when someone edits the ``.dat`` file in the\n repository or removes a ``.deltafs`` file.\n\n The ``quick`` option only influences the created ``bin/backup``\n script. It has no effect on the snapshot or restore scripts.\n\n The repozo help says about this option: \"Verify via md5 checksum\n only the last incremental written. This significantly reduces the\n disk i/o at the (theoretical) cost of inconsistency. This is a\n probabilistic way of determining whether a full backup is\n necessary.\"\n\n``rsync_options``\n Add extra options to the default ``rsync -a`` command. Default is no\n extra parameters. This can be useful for example when you want to restore\n a backup from a symlinked directory, in which case\n ``rsync_options = --no-l -k`` does the trick.\n\n``rsync_hard_links_on_first_copy``\n When using ``rsync``, the blob files for the first backup are copied\n and then subsequent backups make use of hard links from this initial\n copy, to save time and disk space.\n Enable this option to also use hard links for the initial copy to further reduce\n disk usage.\n This is safe for ZODB blobs, since they are not modified in place.\n The ``blob_storage`` and the backup folder ``blobbackuplocation``\n have to be in the same partition for hard links to be possible.\n\n``snapshotlocation``\n Location where snapshot backups of the filestorage are stored. Defaults to\n ``var/snapshotbackups`` inside the buildout directory.\n\n``use_rsync``\n Use ``rsync`` with hard links for backing up the blobs. Default is\n true. ``rsync`` is probably not available on all machines though, and\n I guess hard links will not work on Windows. When you set this to\n false, we fall back to a simple copy (``shutil.copytree`` from\n Python in fact).\n\n``ziplocation``\n Location where zip backups of the filestorage are stored. Defaults to\n ``var/zipbackups`` inside the buildout directory.\n\n\nAn example buildout snippet using various options, would look like this::\n\n [backup]\n recipe = collective.recipe.backup\n location = ${buildout:directory}/myproject\n keep = 2\n datafs = subfolder/myproject.fs\n full = true\n debug = true\n snapshotlocation = snap/my\n enable_snapshotrestore = true\n pre_command = echo 'Can I have a backup?'\n post_command =\n echo 'Thanks a lot for the backup.'\n echo 'We are done.'\n\nPaths in directories or files can use relative (``../``) paths, and\n``~`` (home dir) and ``$BACKUP``-style environment variables are\nexpanded.\n\n\nCron job integration\n====================\n\n``bin/backup`` is of course ideal to put in your cronjob instead of a whole\n``bin/repozo ....`` line. But you don't want the \"INFO\" level logging that you\nget, as you'll get that in your mailbox. In your cronjob, just add ``-q`` or\n``--quiet``, and ``bin/backup`` will shut up unless there's a problem.\nThis option ignores the debug variable, if set to true in buildout.\n\nSpeaking of cron jobs? Take a look at `zc.recipe.usercrontab\n<http://pypi.python.org/pypi/z3c.recipe.usercrontab>`_ if you want to handle\ncronjobs from within your buildout. For example::\n\n [backupcronjob]\n recipe = z3c.recipe.usercrontab\n times = 0 12 * * *\n command = ${buildout:directory}/bin/backup\n\n\nBlob storage\n============\n\nAdded in version 2.0.\n\nWe can backup the blob storage. Plone 4 uses a blob storage to store\nfiles (Binary Large OBjects) on the file system. In Plone 3 this is\noptional. When this is used, it should be backed up of course. You\nmust specify the source blob_storage directory where Plone (or Zope)\nstores its blobs. As indicated earlier, when we do not set it\nspecifically, we try to get the location from other parts, for example\nthe ``plone.recipe.zope2instance`` recipe::\n\n [buildout]\n parts = instance backup\n\n [instance]\n recipe = plone.recipe.zope2instance\n user = admin:admin\n blob-storage = ${buildout:directory}/var/somewhere\n\n [backup]\n recipe = collective.recipe.backup\n\nIf needed, we can tell buildout that we *only* want to backup blobs or\nspecifically do *not* want to backup the blobs. Specifying this using\nthe ``backup_blobs`` and ``only_blobs`` options might be useful in\ncase you want to separate this into several scripts::\n\n [buildout]\n newest = false\n parts = filebackup blobbackup\n\n [filebackup]\n recipe = collective.recipe.backup\n backup_blobs = false\n\n [blobbackup]\n recipe = collective.recipe.backup\n blob_storage = ${buildout:directory}/var/blobstorage\n only_blobs = true\n\nWith this setup ``bin/filebackup`` now only backs up the filestorage\nand ``bin/blobbackup`` only backs up the blobstorage.\n\nNew in version 4.0: you may want to specify ``blob_timestamps = true``.\nThen we create stable directories that we do not rotate.\nFor example: ``blobstorage.1972-12-25-01-02-03`` instead of ``blobstorage.0``.\n\n\nrsync\n=====\n\nBy default we use ``rsync`` to create backups. We create hard links\nwith this tool, to save disk space and still have incremental backups.\nThis probably requires a unixy (Linux, Mac OS X) operating system.\nIt is based on this article by Mike Rubel:\nhttp://www.mikerubel.org/computers/rsync_snapshots/\n\nWe have not tried this on Windows. Reports are welcome, but best is\nprobably to set the ``use_rsync = false`` option in the backup part.\nThen we simply copy the blobstorage directory.\n\n\nAlternative restore source\n==========================\n\nAdded in version 2.17.\nChanged in version 5: only one source is supported.\n\nYou can restore from an alternative source. Use case: first make a\nbackup of your production site, then go to the testing or staging\nserver and restore the production data there.\n\nIn the ``alternative_restore_source`` option you can define different\nfilestorage and blobstorage backup source directories using this\nsyntax::\n\n alternative_restore_source =\n storagename datafs1_backup [blobdir1_backup]\n\nThe storagename must be ``Data`` (or ``1``) for the standard ``Data.fs``\nand optionally its blobstorage.\n\nThe result is a ``bin/altrestore`` script.\n\nThis will work for a standard buildout with a single filestorage and\nblobstorage::\n\n [backup]\n recipe = collective.recipe.backup\n alternative_restore_source =\n Data /path/to/production/var/backups /path/to/production/var/blobstoragebackups\n\nThe above configuration uses ``repozo`` to restore the Data.fs from\nthe ``/path/to/production/var/backups`` repository to the standard\n``var/filestorage/Data.fs`` location. It copies the most recent\nblobstorage backup from\n``/path/to/production/var/blobstoragebackups/`` to the standard\n``var/blobstorage`` location.\n\nCalling the script with a specific date is supported just like the\nnormal restore script::\n\n bin/altrestore 2000-12-31-23-59\n\nThe recipe will fail if the alternative source does not match the\nstandard filestorage and blobstorage. For\nexample, you get an error when the ``alternative_restore_source`` is\nmissing the ``Data`` key, when it has an extra key, when a\nkey has no paths, when a key has an extra or missing blobstorage.\n\nDuring install of the recipe, so during the ``bin/buildout`` run, it\ndoes not check if the sources exist: you might have the production\nbackups on a different server and need to setup a remote shared\ndirectory, or you copy the data over manually.\n\nNote that the script takes the ``archive_blob`` and ``use_rsync`` options\ninto account. So if the alternative restore source contains a blob\nbackup that was made with ``archive_blob = true``, you need an\n``altrestore`` script that also uses this setting.\n\nContributors\n************\n\ncollective.recipe.backup is basically a port of ye olde instancemanager_'s\nbackup functionality. That backup functionality was coded mostly by Reinout\nvan Rees and Maurits van Rees, both from `Zest software`_\n\nCreating the buildout recipe was done by Reinout_ with some fixes by Maurits_, who is now the main developer and maintainer.\n\nThe snapshotrestore script was added by Nejc Zupan (niteoweb_).\n\nThe fullbackup script was added by `Tom 'Spanky' Kapanka`_.\n\nArchive blob backups feature added by `Matej Cotman`_ (niteoweb_).\n\n\nSponsorship\n===========\n\nWork on collective.recipe.backup has been made possible by Ghent University, or UGent.\nSee https://www.ugent.be.\nGhent University is a top 100 university and one of the major universities in Belgium.\n\n.. Links used above.\n\n.. _Zest software: http://zestsoftware.nl/\n\n.. _Reinout: http://reinout.vanrees.org/\n\n.. _Maurits: http://maurits.vanrees.org/\n\n.. _instancemanager: https://pypi.python.org/pypi/instancemanager\n\n.. _`Tom 'Spanky' Kapanka`: https://github.com/spanktar\n\n.. _`Sylvain Bouchard`: https://github.com/bouchardsyl\n\n.. _`Matej Cotman`: https://github.com/matejc\n\n.. _niteoweb: http://www.niteoweb.com\n\nChange history\n**************\n\n.. NOTE: You should *NOT* be adding new change log entries to this file, this\n file is managed by towncrier. You *may* edit previous change logs to\n fix problems like typo corrections or such.\n\n To add a new change log entry, please see the notes from the ``pip`` project at\n https://pip.pypa.io/en/latest/development/#adding-a-news-entry\n\n.. towncrier release notes start\n\n5.0.0 (2023-06-12)\n==================\n\nBug fixes:\n\n\n- Final release, no changes since previous alpha. [maurits]\n\n\n5.0.0a3 (2023-03-15)\n====================\n\nBreaking changes:\n\n\n- Removed ``additional_filestorages`` option.\n Removed support in ``alternative_restore_sources`` for multiple storages: only the standard single filestorage plus blobstorage is supported.\n Renamed ``alternative_restore_sources`` to ``alternative_restore_source``, keeping the old alias for now.\n [maurits] (`Issue #64 <https://github.com/collective/collective.recipe.backup/issues/64>`_)\n- Removed ``gzip`` option.\n From now on, we always call repozo with the gzip option, you can no longer turn this off.\n [maurits] (`Issue #67 <https://github.com/collective/collective.recipe.backup/issues/67>`_)\n- Removed option ``gzip_blob``.\n This was a backwards compatibility alias for ``archive_blob``.\n You should use that now it you want to create a tar archive.\n Additionally use the ``compress_blob`` option if you are really sure you want to compress the archive.\n [maurits] (`Issue #68 <https://github.com/collective/collective.recipe.backup/issues/68>`_)\n- Removed option ``quick``.\n From now on, we always call repozo with the quick option for normal backups, you can no longer turn this off.\n For full backups (snapshotbackups and zipbackups), the quick option is not used.\n [maurits] (`Issue #69 <https://github.com/collective/collective.recipe.backup/issues/69>`_)\n\n\nBug fixes:\n\n\n- Deprecate setting blob_timestamps to false.\n See `issue 65 <https://github.com/collective/collective.recipe.backup/issues/65>`_.\n [maurits] (`Issue #65 <https://github.com/collective/collective.recipe.backup/issues/65>`_)\n\n\n5.0.0a2 (2023-02-01)\n====================\n\nBug fixes:\n\n- Add missing changelog entries for release 5.0.0a1. [maurits]\n\n\n5.0.0a1 (2022-11-08)\n====================\n\nBreaking changes:\n\n\n- Drop support for Python 2.7-3.7, add support for 3.10.\n [maurits] (`Issue #62 <https://github.com/collective/collective.recipe.backup/issues/62>`_)\n- No longer support the ``enable_fullbackup`` option and the ``fullbackup`` script.\n A full backup is automatically made by the first call of ``bin/backup`` after you have done a zeopack.\n If you still want such a script, look at ``bin/snapshotbackup`` script or use the `full` option, optionally in a second buildout section.\n [maurits] (`Issue #66 <https://github.com/collective/collective.recipe.backup/issues/66>`_)\n\n\nNew features:\n\n\n- Added an option to use hard links for the first backup copy\n [ale-rt] (`Issue #57 <https://github.com/collective/collective.recipe.backup/issues/57>`_)\n- Add support for Python 3.11. [maurits] (`Issue #311 <https://github.com/collective/collective.recipe.backup/issues/311>`_)\n\n\nBug fixes:\n\n\n- Drop code-analysis, add other QA checks: isort, black, flake8.\n [maurits] (`Issue #63 <https://github.com/collective/collective.recipe.backup/issues/63>`_)\n\n\n4.2.0 (2021-08-30)\n==================\n\nNew features:\n\n\n- Changed default of ``blob_timestamps`` to true.\n We might remove the previous behavior (creating ``blobstorage.0``, ``blobstorage.1`` etcetera) completely in version 5.\n [maurits] (`Issue #61 <https://github.com/collective/collective.recipe.backup/issues/61>`_)\n\n\nBug fixes:\n\n\n- Switch the test infrastructure to GitHub Actions and tox.\n Also test Python 3.8 and 3.9. [ale-rt, maurits] (`Issue #58 <https://github.com/collective/collective.recipe.backup/issues/58>`_)\n- Fixed ``latest`` symlink when using options ``blob_timestamps`` and ``no_rsync``.\n Previously it pointed to for example ``blobstorage.2021-01-02-03-04-05/blobstorage`` instead of the directory above it.\n [maurits] (`Issue #61 <https://github.com/collective/collective.recipe.backup/issues/61>`_)\n- When creating a zipbackup, do not create a blob timestamp.\n We always only keep one zipbackup, so there is no need for timestamps to know which filestorage backup it belongs to.\n [maurits] (`Issue #61 <https://github.com/collective/collective.recipe.backup/issues/61>`_)\n\n\n4.1.1 (2019-10-07)\n==================\n\nBug fixes:\n\n\n- When using timestamps, use correct filestorage backup location for blob timestamp of snapshot and zip backup.\n [maurits] (`Issue #55 <https://github.com/collective/collective.recipe.backup/issues/55>`_)\n\n\n4.1.0 (2019-04-10)\n==================\n\nNew features:\n\n\n- Create symlink to latest timestamped blobstorage backup.\n When ``blob_timestamps`` is false, ``blobstorage.0`` is a stable filename,\n but with ``blob_timestamps`` true, such a stable name was missing.\n [maurits] (`Issue #48 <https://github.com/collective/collective.recipe.backup/issues/48>`_)\n- Dropped official Python 2.6 support, added Python 3.7 support.\n No code change, but we are no longer testing 2.6.\n [maurits] (`Issue #49 <https://github.com/collective/collective.recipe.backup/issues/49>`_)\n\n\nBug fixes:\n\n\n- Updated package versions in test buildout.\n Improved code quality according to flake8 and with black.\n [maurits] (`Issue #49 <https://github.com/collective/collective.recipe.backup/issues/49>`_)\n- Removed already disabled support for sitepackage_safe_scripts. [maurits] (`Issue #51 <https://github.com/collective/collective.recipe.backup/issues/51>`_)\n\n\n4.0.1 (2018-04-19)\n==================\n\nBug fixes:\n\n\n- With archive_blob true, still allow restoring non archives. We prefer\n archives then, but if no proper archive is found, we look for non archives.\n [maurits] (`Issue #44\n <https://github.com/collective/collective.recipe.backup/issues/44>`_)\n- Sort os.listdir, for improved functioning on Mac. [maurits] (`Issue #45\n <https://github.com/collective/collective.recipe.backup/issues/45>`_)\n\n\n4.0 (2017-12-22)\n================\n\n- Updated readme and added sponsorship note. [maurits]\n\n\n4.0b5 (2017-11-17)\n==================\n\n- Added ``incremental_blobs`` option.\n This creates tarballs with only the changes compared to the previous blob backups.\n This option is ignored when the ``archive_blob`` option is false.\n [maurits]\n\n- Improved code quality, reducing complexity. [maurits]\n\n- Refactored the main functions to not have so much code duplication.\n The normal, full, snapshot and zip backups had almost the same code.\n This made it hard to add new options.\n [maurits]\n\n\n4.0b4 (2017-08-18)\n==================\n\n- Test Python 3.6 (and 2.6 and 2.7) on Travis from now on. [maurits]\n\n- Ignore the zope2instance recipe integration tests on Python 3.\n They would need a compatible ``mailinglogger`` package.\n See `issue #31 <https://github.com/collective/collective.recipe.backup/issues/31>`_. [maurits]\n\n- Tests: use cleaner way to check the mock repozo output.\n Share this setup between tests.\n This makes the output order the same on Python 2 and 3.\n See `issue #31 <https://github.com/collective/collective.recipe.backup/issues/31>`_. [maurits]\n\n\n4.0b3 (2017-07-05)\n==================\n\n- Added basic Python 3 support. We do not test with it yet,\n but you should not get NameErrors anymore.\n See `issue #31 <https://github.com/collective/collective.recipe.backup/issues/31>`_. [maurits]\n\n\n4.0b2 (2017-06-26)\n==================\n\n- No longer create the ``fullbackup`` script by default.\n You can still enable it by setting ``enable_fullbackup`` to ``true``.\n [maurits]\n\n- Without explicit ``blob-storage`` option, default to ``var/blobstorage``.\n Take the ``var`` option from zeoserver/client recipes into account.\n Fixes `issue #27 <https://github.com/collective/collective.recipe.backup/issues/27>`_.\n [maurits]\n\n- Do not create hidden backup ``.0`` when blob_storage ends with a slash.\n Fixes `issue #26 <https://github.com/collective/collective.recipe.backup/issues/26>`_.\n [maurits]\n\n\n4.0b1 (2017-05-31)\n==================\n\n- Make custom backup locations relative to the ``locationprefix`` option or the ``var`` directory.\n Until now, the ``locationprefix`` option was only used if you did not set custom locations.\n Custom location would be relative to the buildout directory.\n Now they are relative to the ``locationprefix`` option, with the ``var`` directory as default.\n So if you used a relative path, your backups may end up in a different path.\n Absolute paths are not affected: they ignore the locationprefix.\n [maurits]\n\n- When log level is DEBUG, show time stamps in the log. [maurits]\n\n- Added ``compress_blob`` option. Default is false.\n This is only used when the ``archive_blob`` option is true.\n When switched on, it will compress the archive,\n resulting in a ``.tar.gz`` instead of a ``tar`` file.\n When restoring, we always look for both compressed and normal archives.\n We used to always compress them, but in most cases it hardly decreases the size\n and it takes a long time anyway. I have seen archiving take 15 seconds,\n and compressing take an additional 45 seconds.\n The result was an archive of 5.0 GB instead of 5.1 GB.\n [maurits]\n\n- Renamed ``gzip_blob`` option to ``archive_blob``.\n Kept the old name as alias for backwards compatibility.\n This makes room for letting this create an archive without zipping it.\n [maurits]\n\n- Automatically remove old blobs backups that have no corresponding filestorage backup.\n We compare the timestamp of the oldest filestorage backup with the timestamps of the\n blob backups. This can be the name, if you use ``blob_timestamps = true``,\n or the modification date of the blob backup.\n This means that the ``keep_blob_days`` option is ignored, unless you use ``only_blobs = true``.\n [maurits]\n\n- When backing up a blobstorage, use the timestamp of the latest filestorage backup.\n If a blob backup with that name is already there, then there were no database changes,\n so we do not make a backup.\n This is only done when you use the new ``blob_timestamps = true`` option.\n [maurits]\n\n- When restoring to a specific date, find the first blob backup at or before\n the specified date. Otherwise fail. The repozo script does the same.\n We used to pick the first blob backup *after* the specified date,\n because we assumed that the user would specify the exact date that is\n in the filestorage backup.\n Note that the timestamp of the filestorage and blobstorage backups may be\n a few seconds apart, unless you use the ``blob_timestamps == true`` option.\n In the new situation, the user should pick the date of the blob backup\n or slightly later.\n [maurits]\n\n- Added ``blob_timestamps`` option. Default is false.\n By default we create ``blobstorage.0``.\n The next time, we rotate this to ``blobstorage.1`` and create a new ``blobstorage.0``.\n With ``blob_timestamps = true``, we create stable directories that we do not rotate.\n They get a timestamp, the same timestamp that the ZODB filestorage backup gets.\n For example: ``blobstorage.1972-12-25-01-02-03``.\n [maurits]\n\n- When restoring, first run checks for all filestorages and blobstorages.\n When one of the backups is missing, we quit with an error.\n This avoids restoring a filestorage and then getting into trouble\n due to a missing blobstorage backup. [maurits]\n\n\n3.1 (2017-02-24)\n================\n\n- Add a ``locationprefix`` option to configure a folder where all other\n backup and snapshot folders will be created [erral]\n\n- Only claim compatibility with Python 2.6 and 2.7. [maurits]\n\n- Updated test buildout to use most recent versions. [maurits]\n\n\n3.0.0 (2015-12-31)\n==================\n\n- Refactored the init and install methods of this recipe. During the\n init phase we were reading the buildout configuration, but during\n this phase the configuration is still being build. So differences\n could occur, especially in the order of execution of parts. This\n was not good. Most code is now moved from the init to the install\n (and update) method. This has less possible problems. Downside:\n some configuration errors are caught later.\n [maurits]\n\n- Read ``zeo-var``, ``var``, ``file-storage`` from buildout sections.\n Update default backup and Data.fs locations based on this.\n [maurits]\n\n\n2.22 (2015-12-30)\n=================\n\n- Do not accept ``backup_blobs`` false and ``enable_zipbackup`` true.\n The zipbackup script is useless without blobs.\n [maurits]\n\n- Set default ``backup_blobs`` to true on Python 2.6 (Plone 4) and\n higher. Otherwise false. If no ``blob_storage`` can be found, we\n quit with an error.\n [maurits]\n\n- Accept ``true``, ``yes``, ``on``, ``1``, in lower, upper or mixed\n case as true value. Treat all other values in the buildout options\n as false.\n [maurits]\n\n- Find plone.recipe.zope2instance recipes also when they are not\n completely lower case. The zope2instance recipe itself works fine\n when it has mixed case, so we should accept this too.\n [maurits]\n\n\n2.21 (2015-10-06)\n=================\n\n- When restoring, create ``var/filestorage`` if needed.\n Fixes #23.\n [maurits]\n\n\n2.20 (2014-11-11)\n=================\n\n- Add ``enable_fullbackup`` option. Default: true, so no change\n compared to previous version.\n [maurits]\n\n- Create backup/snapshot/zipbackup directories only when needed.\n Running the backup script should not create the snapshot\n directories.\n [maurits]\n\n- Add zipbackup and ziprestore scripts when ``enable_zipbackup = true``.\n [maurits]\n\n\n2.19 (2014-06-16)\n=================\n\n- Call repozo with ``--quick`` when making an incremental backup.\n This is a lot faster. Theoretically it lead to inconsistency if\n someone is messing in your backup directory. You can return to the\n previous behavior by specifying ``quick = false`` in the backup\n recipe part in your buildout config.\n [maurits]\n\n- check and create folders now happens after pre_commands is run\n [@djay]\n\n\n2.18 (2014-04-29)\n=================\n\n- Add ``rsync_options`` option. These are added to the default\n ``rsync -a`` command. Default is no extra parameters. This can be\n useful for example when you want to restore a backup from a\n symlinked directory, in which case ``rsync_options = --no-l -k``\n does the trick.\n [fiterbek]\n\n\n\n2.17 (2014-02-07)\n=================\n\n- Add ``alternative_restore_sources`` option. This creates a\n ``bin/altrestore`` script that restores from an alternative backup\n location, specified by that option. You can use this to restore a\n backup of the production data to your testing or staging server.\n [maurits]\n\n- When checking if the backup script will be able to create a path,\n remove all created directories. Until now, only the final directory\n was removed, and not any created parent directories.\n [maurits]\n\n- Testing: split the single big doctest file into multiple files, to\n make the automated tests less dependent on one another, making it\n easier to change them and add new ones.\n [maurits]\n\n- No longer test with Python 2.4, because Travis does not support it\n out of the box. Should still work fine.\n [maurits]\n\n\n2.16 (2014-01-14)\n=================\n\n- Do not create blob backup dirs when not backing up blobs.\n Do not create filestorage backup dirs when not backing up filestorage.\n Fixes https://github.com/collective/collective.recipe.backup/issues/17\n [maurits]\n\n\n2.15 (2013-09-16)\n=================\n\n- Restore compatibility with Python 2.4 (Plone 3).\n [maurits]\n\n\n2.14 (2013-09-09)\n=================\n\n- Archive blob backups with buildout option ``gzip_blob``.\n [matejc]\n\n\n2.13 (2013-07-15)\n=================\n\n- When printing that we halt the execution due to an error running\n repozo, actually halt the execution.\n [maurits]\n\n\n2.12 (2013-06-28)\n=================\n\n- Backup directories are now created when we launch ``backup`` or\n ``fullbackup`` or ``snapshotbackup`` scripts, no more during\n initialization.\n [bsuttor]\n\n\n2.11 (2013-05-06)\n=================\n\n- Print the names of filestorages and blobstorages that will be\n restored. Issue #8.\n [maurits]\n\n- Added a new command-line argument : ``--no-prompt`` disables user\n input when restoring a backup or snapshot. Useful for shell scripts.\n [bouchardsyl]\n\n- Fixed command-line behavior with many arguments and not only a date.\n [bouchardsyl]\n\n\n2.10 (2013-03-30)\n=================\n\n- Added ``fullbackup`` script that defaults to ``full=true``. This\n could have been handled by making a new part, but it seemed like\n overkill to have to generate a complete new set of backup scripts,\n just to get one for full.\n [spanky]\n\n\n2.9 (2013-03-06)\n================\n\n- Fixed possible KeyError: ``blob_snapshot_location``.\n [gforcada]\n\n\n\n2.8 (2012-11-13)\n================\n\n- Fixed possible KeyError: ``blob_backup_location``.\n https://github.com/collective/collective.recipe.backup/issues/3\n [maurits]\n\n\n2.7 (2012-09-27)\n================\n\n- additional_filestorages improved: blob support and custom location.\n [mamico]\n\n\n2.6 (2012-08-29)\n================\n\n- Added pre_command and post_command options. See the documentation.\n [maurits]\n\n\n2.5 (2012-08-08)\n================\n\n- Moved code to github:\n https://github.com/collective/collective.recipe.backup\n [maurits]\n\n\n2.4 (2011-12-20)\n================\n\n- Fixed silly indentation error that prevented old blob backups from\n being deleted when older than ``keep_blob_days`` days.\n [maurits]\n\n\n2.3 (2011-10-05)\n================\n\n- Quit the rest of the backup or restore when a repozo call gives an\n error. Main use case: when restoring to a specific date repozo will\n quit with an error when no files can be found, so we should also not\n try to restore blobs then.\n [maurits]\n\n- Allow restoring the blobs to the specified date as well.\n [maurits]\n\n\n2.2 (2011-09-14)\n================\n\n- Refactored script generation to make a split between initialization\n code and script arguments. This restores compatibility with\n zc.buildout 1.5 for system pythons. Actually we no longer create so\n called 'site package safe scripts' but just normal scripts that work\n for all zc.buildout versions.\n [maurits]\n\n- Added option ``keep_blob_days``, which by default specifies that\n only for partial backups we keep 14 days of backups. See the\n documentation.\n [maurits]\n\n- Remove old blob backups when doing a snapshot backup.\n [maurits]\n\n\n2.1 (2011-09-01)\n================\n\n- Raise an error when the four backup location options\n (blobbackuplocation, blobsnapshotlocation, location and\n snapshotlocation) are not four distinct locations (or empty\n strings).\n [maurits]\n\n- Fixed possible TypeError: 'Option values must be strings'.\n Found by Alex Clark, thanks.\n [maurits]\n\n\n2.0 (2011-08-26)\n================\n\n- Backup and restore blobs, using rsync.\n [maurits]\n\n- Ask if the user is sure before doing a restore.\n [maurits]\n\n\n1.7 (2010-12-10)\n================\n\n- Fix generated repozo commands to work also\n when recipe is configured to have a non **Data.fs**\n main db plus additional filestorages.\n e.g.:\n datafs= var/filestorage/main.fs\n additional = catalog\n [hplocher]\n\n\n1.6 (2010-09-21)\n================\n\n- Added the option enable_snapshotrestore so that the creation of the\n script can be removed. Backwards compatible, if you don't specify it\n the script will still be created. Rationale: you may not want this\n script in a production buildout where mistakenly using\n snapshotrestore instead of snapshotbackup could hurt.\n [fredvd]\n\n\n1.5 (2010-09-08)\n================\n\n- Fix: when running buildout with a config in a separate directory\n (like ``bin/buildout -c conf/prod.cfg``) the default backup\n directories are no longer created inside that separate directory.\n If you previously manually specified one of the location,\n snapshotlocation, or datafs parameters to work around this, you can\n probably remove those lines. So: slightly saner defaults.\n [maurits]\n\n\n1.4 (2010-08-06)\n================\n\n- Added documentation about how to get the required bin/repozo script\n in your buildout if for some reason you do not have it yet (like on\n Plone 4 when you do not have a zeo setup).\n Thanks to Vincent Fretin for the extra buildout lines.\n [maurits]\n\n\n1.3 (2009-12-08)\n================\n\n- Added snapshotrestore script. [Nejc Zupan]\n\n\n1.2 (2009-10-26)\n================\n\n- The part name is now reflected in the created scripts and var/ directories.\n Originally bin/backup, bin/snapshotbackup, bin/restore and var/backups\n plus var/snapshotbackups were hardcoded. Those are still there when you\n name your part ``[backup]``. With a part named ``[NAME]``, you get\n bin/NAME, bin/NAME-snapshot, bin/NAME-restore and var/NAMEs plus\n var/NAME-snapshots. Request by aclark for plone.org. [reinout]\n\n\n1.1 (2009-08-21)\n================\n\n- Run the cleanup script (removing too old backups that we no longer\n want to keep) for additional file storages as well.\n Fixes https://bugs.launchpad.net/collective.buildout/+bug/408224\n [maurits]\n\n- Moved everything into a src/ subdirectory to ease testing on buildbot (which\n would grab all egss in the eggs/ dir that buildbot's mechanism creates.\n [reinout]\n\n\n1.0 (2009-02-06)\n================\n\n- Quote all paths and arguments so that it works on paths that contain\n spaces (specially on Windows). [sidnei]\n\n\n0.9 (2008-12-05)\n================\n\n- Windows path compatibility fix. [Juan A. Diaz]\n\n\n0.8 (2008-09-23)\n================\n\n- Changed the default for gzipping to True. Adding ``gzip = true`` to all our\n server deployment configs gets tired pretty quickly, so doing it by default\n is the best default. Stuff like this needs to be changed **before** a 1.0\n release :-) [reinout]\n\n- Backup of additional databases (if you have configured them) now takes place\n before the backup of the main database (same with restore). [reinout]\n\n\n0.7 (2008-09-19)\n================\n\n- Added $BACKUP-style enviroment variable subsitution in addition to the tilde\n expansion offered by 0.6. [reinout, idea by Fred van Dijk]\n\n\n0.6 (2008-09-19)\n================\n\n- Fixed the test setup so both bin/test and python setup.py test\n work. [reinout+maurits]\n\n- Added support for ~ in path names. And fixed a bug at the same time that\n would occur if you call the backup script from a different location than\n your buildout directory in combination with a non-absolute backup\n location. [reinout]\n\n\n0.5 (2008-09-18)\n================\n\n- Added support for additional_filestorages option, needed for for instance a\n split-out catalog.fs. [reinout]\n\n- Test setup fixes. [reinout+maurits]\n\n\n0.4 (2008-08-19)\n================\n\n- Allowed the user to make the script more quiet (say in a cronjob)\n by using 'bin/backup -q' (or --quiet). [maurits]\n\n- Refactored initialization template so it is easier to change. [maurits]\n\n\n0.3.1 (2008-07-04)\n==================\n\n- Added 'gzip' option, including changes to the cleanup functionality that\n treats .fsz also as a full backup like .fs. [reinout]\n\n- Fixed typo: repoze is now repozo everywhere... [reinout]\n\n\n0.2 (2008-07-03)\n================\n\n- Extra tests and documentation change for 'keep': the default is to keep 2\n backups instead of all backups. [reinout]\n\n- If debug=true, then repozo is also run in --verbose mode. [reinout]\n\n\n0.1 (2008-07-03)\n================\n\n- Added bin/restore. [reinout]\n\n- Added snapshot backups. [reinout]\n\n- Enabled cleaning up of older backups. [reinout]\n\n- First working version that runs repozo and that creates a backup dir if\n needed. [reinout]\n\n- Started project based on zopeskel template. [reinout]\n",
"bugtrack_url": null,
"license": "GPL",
"summary": "bin/backup script: sensible defaults around bin/repozo",
"version": "5.0.0",
"project_urls": {
"Homepage": "https://github.com/collective/collective.recipe.backup"
},
"split_keywords": [
"buildout",
"backup",
"repozo",
"zope"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "e5bcbf321a130ba5ab61468694adaaf0c6252ca28f92ddb03722b1d7e1584b4a",
"md5": "d071238c387b5329230741f7f83e28d5",
"sha256": "5f451373e574ff6db59af43ef5259d182045485debfae6bce960cbcfecbb3508"
},
"downloads": -1,
"filename": "collective.recipe.backup-5.0.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "d071238c387b5329230741f7f83e28d5",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 81687,
"upload_time": "2023-06-12T17:45:40",
"upload_time_iso_8601": "2023-06-12T17:45:40.542510Z",
"url": "https://files.pythonhosted.org/packages/e5/bc/bf321a130ba5ab61468694adaaf0c6252ca28f92ddb03722b1d7e1584b4a/collective.recipe.backup-5.0.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "a89b747858a73b1cda6967fc9db18360da5ec8d703552be1de3244678747967c",
"md5": "5681fe1c427b50dc16672511871597c2",
"sha256": "baed1efdadd235b644552caae77f173295d6933b59a70f6dbcf6527b3d3a84d7"
},
"downloads": -1,
"filename": "collective.recipe.backup-5.0.0.tar.gz",
"has_sig": false,
"md5_digest": "5681fe1c427b50dc16672511871597c2",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 99705,
"upload_time": "2023-06-12T17:45:42",
"upload_time_iso_8601": "2023-06-12T17:45:42.515762Z",
"url": "https://files.pythonhosted.org/packages/a8/9b/747858a73b1cda6967fc9db18360da5ec8d703552be1de3244678747967c/collective.recipe.backup-5.0.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-06-12 17:45:42",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "collective",
"github_project": "collective.recipe.backup",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [],
"tox": true,
"lcname": "collective.recipe.backup"
}
JSON
