jaraco.mongodb


Namejaraco.mongodb JSON
Version 12.0.0 PyPI version JSON
download
home_pagehttps://github.com/jaraco/jaraco.mongodb
SummaryRoutines and classes supporting MongoDB environments
upload_time2023-09-01 20:19:54
maintainer
docs_urlNone
authorJason R. Coombs
requires_python>=3.8
license
keywords
VCS
bugtrack_url
requirements No requirements were recorded.
Travis-CI No Travis.
coveralls test coverage
            .. image:: https://img.shields.io/pypi/v/jaraco.mongodb.svg
   :target: https://pypi.org/project/jaraco.mongodb

.. image:: https://img.shields.io/pypi/pyversions/jaraco.mongodb.svg

.. image:: https://github.com/PROJECT_PATH/workflows/tests/badge.svg
   :target: https://github.com/PROJECT_PATH/actions?query=workflow%3A%22tests%22
   :alt: tests

.. image:: https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/charliermarsh/ruff/main/assets/badge/v2.json
    :target: https://github.com/astral-sh/ruff
    :alt: Ruff

.. image:: https://img.shields.io/badge/code%20style-black-000000.svg
   :target: https://github.com/psf/black
   :alt: Code style: Black

.. image:: https://readthedocs.org/projects/jaracomongodb/badge/?version=latest
   :target: https://jaracomongodb.readthedocs.io/en/latest/?badge=latest

.. image:: https://img.shields.io/badge/skeleton-2023-informational
   :target: https://blog.jaraco.com/skeleton

Migration Manager
=================

``jaraco.mongodb.migration`` implements the Migration Manager as featured
at the `MongoWorld 2016 <https://www.mongodb.com/world16>`_ presentation
`From the Polls to the Trolls
<https://combinatronics.com/yougov/mongoworld-2016/merged/index.html#/>`_.
Use it to load documents of various schema versions into a target version that
your application expects.

sessions
========

``jaraco.mongodb.sessions`` implements a CherryPy Sessions store backed by
MongoDB.

By default, the session store will handle sessions with any objects that can
be inserted into a MongoDB collection naturally.

To support richer objects, one may configure the codec to use ``jaraco.modb``.

fields
======

``jaraco.mongodb.fields`` provides two functions, encode and decode, which
take arbitrary unicode text and transform it into values suitable as keys
on older versions of MongoDB by backslash-escaping the values.

monitor-index-creation
======================

To monitor an ongoing index operation in a server, simply invoke:

    python -m jaraco.mongodb.monitor-index-creation mongodb://host/db

move-gridfs
===========

To move files from one gridfs collection to another, invoke:

    python -m jaraco.mongodb.move-gridfs --help

And follow the usage for moving all or some gridfs files and
optionally deleting the files after.

oplog
=====

This package provides an ``oplog`` module, which is based on the
`mongooplog-alt <https://github.com/asivokon/mongooplog-alt/>`_ project,
which itself is a Python remake of `official mongooplog utility
<https://docs.mongodb.com/manual/reference/program/mongooplog/>`_,
shipped with MongoDB starting from version 2.2 and deprecated in 3.2.
It reads oplog of a remote
server, and applies operations to the local server. This can be used to keep
independed replica set loosly synced in much the same way as Replica Sets
are synced, and may
be useful in various backup and migration scenarios.

``oplog`` implements basic functionality of the official utility and
adds following features:

* tailable oplog reader: runs forever polling new oplog event which is extremly
  useful for keeping two independent replica sets in almost real-time sync.

* option to sync only selected databases/collections.

* option to exclude one or more namespaces (i.e. dbs or collections) from
  being synced.

* ability to "rename" dbs/collections on fly, i.e. destination namespaces can
  differ from the original ones. This feature works on mongodb 1.8 and later.
  Official utility only supports version 2.2.x and higher.

* save last processed timestamp to file, resume from saved point later.


Invoke the command as a module script: ``python -m jaraco.mongodb.oplog``.

Command-line options
--------------------

Usage is as follows::

    $ python -m jaraco.mongodb.oplog  --help
    usage: oplog.py [--help] [--source host[:port]] [--oplogns OPLOGNS]
                    [--dest host[:port]] [-w WINDOW] [-f] [--ns [NS [NS ...]]]
                    [-x [EXCLUDE [EXCLUDE ...]]]
                    [--rename [ns_old=ns_new [ns_old=ns_new ...]]] [--dry-run]
                    [--resume-file FILENAME] [-s SECONDS] [-l LOG_LEVEL]

    optional arguments:
      --help                show usage information
      --source host[:port]  Hostname of the mongod server from which oplog
                            operations are going to be pulled. Called "--from" in
                            mongooplog.
      --oplogns OPLOGNS     Source namespace for oplog
      --dest host[:port]    Hostname of the mongod server (or replica set as <set
                            name>/s1,s2) to which oplog operations are going to be
                            applied. Default is "localhost". Called "--host" in
                            mongooplog.
      -w WINDOW, --window WINDOW
                            Time window to query, like "3 days" or "24:00" (24
                            hours, 0 minutes).
      -f, --follow          Wait for new data in oplog. Makes the utility polling
                            oplog forever (until interrupted). New data is going
                            to be applied immediately with at most one second
                            delay.
      --ns [NS [NS ...]]    Process only these namespaces, ignoring all others.
                            Space separated list of strings in form of ``dname``
                            or ``dbname.collection``. May be specified multiple
                            times.
      -x [EXCLUDE [EXCLUDE ...]], --exclude [EXCLUDE [EXCLUDE ...]]
                            List of space separated namespaces which should be
                            ignored. Can be in form of ``dname`` or
                            ``dbname.collection``. May be specified multiple
                            times.
      --rename [ns_old=ns_new [ns_old=ns_new ...]]
                            Rename database(s) and/or collection(s). Operations on
                            namespace ``ns_old`` from the source server will be
                            applied to namespace ``ns_new`` on the destination
                            server. May be specified multiple times.
      --dry-run             Suppress application of ops.
      --resume-file FILENAME
                            Read from and write to this file the last processed
                            timestamp.
      -l LOG_LEVEL, --log-level LOG_LEVEL
                            Set log level (DEBUG, INFO, WARNING, ERROR)

Example usages
--------------

Consider the following sample usage::

    python -m jaraco.mongodb.oplog --source prod.example.com:28000 --dest dev.example.com:28500 -f --exclude logdb data.transactions --seconds 600

This command is going to take operations from the last 10 minutes from prod,
and apply them to dev. Database ``logdb`` and collection ``transactions`` of
``data`` database will be omitted. After operations for the last minutes will
be applied, command will wait for new changes to come, keep running until
Ctrl+C or other termination signal recieved.

The tool provides a ``--dry-run`` option and when logging at the DEBUG level will
emit the oplog entries. Combine these to use the tool as an oplog cat tool::

    $ python -m jaraco.mongodb.oplog --dry-run -s 0 -f --source prod.example.com --ns survey_tabs -l DEBUG


Testing
-------

Tests for ``oplog`` are written in javascript using test harness
which is used for testing MongoDB iteself. You can run the oplog suite with::

    mongo tests/oplog.js

Tests produce alot of output. Succesful execution ends with line like this::

    ReplSetTest stopSet *** Shut down repl set - test worked ****

These tests are run as part of the continuous integration and release acceptance
tests in Travis.

            

Raw data

            {
    "_id": null,
    "home_page": "https://github.com/jaraco/jaraco.mongodb",
    "name": "jaraco.mongodb",
    "maintainer": "",
    "docs_url": null,
    "requires_python": ">=3.8",
    "maintainer_email": "",
    "keywords": "",
    "author": "Jason R. Coombs",
    "author_email": "jaraco@jaraco.com",
    "download_url": "https://files.pythonhosted.org/packages/f5/8d/c8757e74eb699bb1fe004c02ac53d0eb7c1c9f8e6cee687782b355ebf1c4/jaraco.mongodb-12.0.0.tar.gz",
    "platform": null,
    "description": ".. image:: https://img.shields.io/pypi/v/jaraco.mongodb.svg\n   :target: https://pypi.org/project/jaraco.mongodb\n\n.. image:: https://img.shields.io/pypi/pyversions/jaraco.mongodb.svg\n\n.. image:: https://github.com/PROJECT_PATH/workflows/tests/badge.svg\n   :target: https://github.com/PROJECT_PATH/actions?query=workflow%3A%22tests%22\n   :alt: tests\n\n.. image:: https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/charliermarsh/ruff/main/assets/badge/v2.json\n    :target: https://github.com/astral-sh/ruff\n    :alt: Ruff\n\n.. image:: https://img.shields.io/badge/code%20style-black-000000.svg\n   :target: https://github.com/psf/black\n   :alt: Code style: Black\n\n.. image:: https://readthedocs.org/projects/jaracomongodb/badge/?version=latest\n   :target: https://jaracomongodb.readthedocs.io/en/latest/?badge=latest\n\n.. image:: https://img.shields.io/badge/skeleton-2023-informational\n   :target: https://blog.jaraco.com/skeleton\n\nMigration Manager\n=================\n\n``jaraco.mongodb.migration`` implements the Migration Manager as featured\nat the `MongoWorld 2016 <https://www.mongodb.com/world16>`_ presentation\n`From the Polls to the Trolls\n<https://combinatronics.com/yougov/mongoworld-2016/merged/index.html#/>`_.\nUse it to load documents of various schema versions into a target version that\nyour application expects.\n\nsessions\n========\n\n``jaraco.mongodb.sessions`` implements a CherryPy Sessions store backed by\nMongoDB.\n\nBy default, the session store will handle sessions with any objects that can\nbe inserted into a MongoDB collection naturally.\n\nTo support richer objects, one may configure the codec to use ``jaraco.modb``.\n\nfields\n======\n\n``jaraco.mongodb.fields`` provides two functions, encode and decode, which\ntake arbitrary unicode text and transform it into values suitable as keys\non older versions of MongoDB by backslash-escaping the values.\n\nmonitor-index-creation\n======================\n\nTo monitor an ongoing index operation in a server, simply invoke:\n\n    python -m jaraco.mongodb.monitor-index-creation mongodb://host/db\n\nmove-gridfs\n===========\n\nTo move files from one gridfs collection to another, invoke:\n\n    python -m jaraco.mongodb.move-gridfs --help\n\nAnd follow the usage for moving all or some gridfs files and\noptionally deleting the files after.\n\noplog\n=====\n\nThis package provides an ``oplog`` module, which is based on the\n`mongooplog-alt <https://github.com/asivokon/mongooplog-alt/>`_ project,\nwhich itself is a Python remake of `official mongooplog utility\n<https://docs.mongodb.com/manual/reference/program/mongooplog/>`_,\nshipped with MongoDB starting from version 2.2 and deprecated in 3.2.\nIt reads oplog of a remote\nserver, and applies operations to the local server. This can be used to keep\nindepended replica set loosly synced in much the same way as Replica Sets\nare synced, and may\nbe useful in various backup and migration scenarios.\n\n``oplog`` implements basic functionality of the official utility and\nadds following features:\n\n* tailable oplog reader: runs forever polling new oplog event which is extremly\n  useful for keeping two independent replica sets in almost real-time sync.\n\n* option to sync only selected databases/collections.\n\n* option to exclude one or more namespaces (i.e. dbs or collections) from\n  being synced.\n\n* ability to \"rename\" dbs/collections on fly, i.e. destination namespaces can\n  differ from the original ones. This feature works on mongodb 1.8 and later.\n  Official utility only supports version 2.2.x and higher.\n\n* save last processed timestamp to file, resume from saved point later.\n\n\nInvoke the command as a module script: ``python -m jaraco.mongodb.oplog``.\n\nCommand-line options\n--------------------\n\nUsage is as follows::\n\n    $ python -m jaraco.mongodb.oplog  --help\n    usage: oplog.py [--help] [--source host[:port]] [--oplogns OPLOGNS]\n                    [--dest host[:port]] [-w WINDOW] [-f] [--ns [NS [NS ...]]]\n                    [-x [EXCLUDE [EXCLUDE ...]]]\n                    [--rename [ns_old=ns_new [ns_old=ns_new ...]]] [--dry-run]\n                    [--resume-file FILENAME] [-s SECONDS] [-l LOG_LEVEL]\n\n    optional arguments:\n      --help                show usage information\n      --source host[:port]  Hostname of the mongod server from which oplog\n                            operations are going to be pulled. Called \"--from\" in\n                            mongooplog.\n      --oplogns OPLOGNS     Source namespace for oplog\n      --dest host[:port]    Hostname of the mongod server (or replica set as <set\n                            name>/s1,s2) to which oplog operations are going to be\n                            applied. Default is \"localhost\". Called \"--host\" in\n                            mongooplog.\n      -w WINDOW, --window WINDOW\n                            Time window to query, like \"3 days\" or \"24:00\" (24\n                            hours, 0 minutes).\n      -f, --follow          Wait for new data in oplog. Makes the utility polling\n                            oplog forever (until interrupted). New data is going\n                            to be applied immediately with at most one second\n                            delay.\n      --ns [NS [NS ...]]    Process only these namespaces, ignoring all others.\n                            Space separated list of strings in form of ``dname``\n                            or ``dbname.collection``. May be specified multiple\n                            times.\n      -x [EXCLUDE [EXCLUDE ...]], --exclude [EXCLUDE [EXCLUDE ...]]\n                            List of space separated namespaces which should be\n                            ignored. Can be in form of ``dname`` or\n                            ``dbname.collection``. May be specified multiple\n                            times.\n      --rename [ns_old=ns_new [ns_old=ns_new ...]]\n                            Rename database(s) and/or collection(s). Operations on\n                            namespace ``ns_old`` from the source server will be\n                            applied to namespace ``ns_new`` on the destination\n                            server. May be specified multiple times.\n      --dry-run             Suppress application of ops.\n      --resume-file FILENAME\n                            Read from and write to this file the last processed\n                            timestamp.\n      -l LOG_LEVEL, --log-level LOG_LEVEL\n                            Set log level (DEBUG, INFO, WARNING, ERROR)\n\nExample usages\n--------------\n\nConsider the following sample usage::\n\n    python -m jaraco.mongodb.oplog --source prod.example.com:28000 --dest dev.example.com:28500 -f --exclude logdb data.transactions --seconds 600\n\nThis command is going to take operations from the last 10 minutes from prod,\nand apply them to dev. Database ``logdb`` and collection ``transactions`` of\n``data`` database will be omitted. After operations for the last minutes will\nbe applied, command will wait for new changes to come, keep running until\nCtrl+C or other termination signal recieved.\n\nThe tool provides a ``--dry-run`` option and when logging at the DEBUG level will\nemit the oplog entries. Combine these to use the tool as an oplog cat tool::\n\n    $ python -m jaraco.mongodb.oplog --dry-run -s 0 -f --source prod.example.com --ns survey_tabs -l DEBUG\n\n\nTesting\n-------\n\nTests for ``oplog`` are written in javascript using test harness\nwhich is used for testing MongoDB iteself. You can run the oplog suite with::\n\n    mongo tests/oplog.js\n\nTests produce alot of output. Succesful execution ends with line like this::\n\n    ReplSetTest stopSet *** Shut down repl set - test worked ****\n\nThese tests are run as part of the continuous integration and release acceptance\ntests in Travis.\n",
    "bugtrack_url": null,
    "license": "",
    "summary": "Routines and classes supporting MongoDB environments",
    "version": "12.0.0",
    "project_urls": {
        "Homepage": "https://github.com/jaraco/jaraco.mongodb"
    },
    "split_keywords": [],
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "72aadd487231f94fe5dd285d0beb62c175de9c6fffaaa002e45c659c5518e787",
                "md5": "ff5e4ab21de7ce133ab9d7be4e2f3a10",
                "sha256": "a11ea10e846178ea64e54d46ce3d7b8f1c856faabb2b23e7fd639f4d6e6a93c8"
            },
            "downloads": -1,
            "filename": "jaraco.mongodb-12.0.0-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "ff5e4ab21de7ce133ab9d7be4e2f3a10",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": ">=3.8",
            "size": 38612,
            "upload_time": "2023-09-01T20:19:52",
            "upload_time_iso_8601": "2023-09-01T20:19:52.776989Z",
            "url": "https://files.pythonhosted.org/packages/72/aa/dd487231f94fe5dd285d0beb62c175de9c6fffaaa002e45c659c5518e787/jaraco.mongodb-12.0.0-py3-none-any.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "f58dc8757e74eb699bb1fe004c02ac53d0eb7c1c9f8e6cee687782b355ebf1c4",
                "md5": "ad2c7271a52af6f57a080b55bca526c1",
                "sha256": "63d94dca0d7eab8a6ab74c7bc628d18eb93d46a8d1f9d5a0684f236c9440b23c"
            },
            "downloads": -1,
            "filename": "jaraco.mongodb-12.0.0.tar.gz",
            "has_sig": false,
            "md5_digest": "ad2c7271a52af6f57a080b55bca526c1",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": ">=3.8",
            "size": 43443,
            "upload_time": "2023-09-01T20:19:54",
            "upload_time_iso_8601": "2023-09-01T20:19:54.570978Z",
            "url": "https://files.pythonhosted.org/packages/f5/8d/c8757e74eb699bb1fe004c02ac53d0eb7c1c9f8e6cee687782b355ebf1c4/jaraco.mongodb-12.0.0.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2023-09-01 20:19:54",
    "github": true,
    "gitlab": false,
    "bitbucket": false,
    "codeberg": false,
    "github_user": "jaraco",
    "github_project": "jaraco.mongodb",
    "travis_ci": false,
    "coveralls": true,
    "github_actions": true,
    "tox": true,
    "lcname": "jaraco.mongodb"
}
        
Elapsed time: 0.10827s