redismirror


Nameredismirror JSON
Version 1.0.1 PyPI version JSON
download
home_pagehttps://github.com/alivx/redis-mirror
SummaryMirror Redis Traffic to another redis node
upload_time2023-03-23 22:38:32
maintainer
docs_urlNone
authorAli Saleh Baker
requires_python
licenseMIT
keywords traffic mirror redis migration cli
VCS
bugtrack_url
requirements No requirements were recorded.
Travis-CI No Travis.
coveralls test coverage No coveralls.
            # Mirror Redis Traffic to another redis node
<img src="https://raw.githubusercontent.com/alivx/redis-mirror/master/Generator/redis-mirror-logo.jpg" alt="logo" style="zoom:50%;" />


# redis-mirror
Realtime Redis Traffic Mirror to another instance, this script reads the STDOUT from the `redis-cli monitor` command and mirrors the keys to another instance.

## Use Case/Note
In some production/development cases, you need to mirror Redis traffic to another node to do some investigation or debugging.
* This script does not set `TTL` to mirrored key since you need it for debugging.
* The script support all command since it simply dump and restore the key as is.



## TO DO:
1. Add TTL as an option in mirrored redis instance, plus add an option to expand the origin `TTL`.
2. Support cluster to a single redis instance or vice versa.
3. Add an option to dump all keys name to the file for further analysis.
4. Add an option to get all keys from the source and migrate the keys to another redis instance.

## Option

```
redismirror --help
Usage: redismirror [OPTIONS]

Options:
  --shost TEXT     Source redis host/IP.
  --sport INTEGER  Source redis port.
  --sdb INTEGER    Source redis DB.
  --sauth TEXT     Source redis auth info.
  --dhost TEXT     Destination redis host/IP.
  --dport INTEGER  Destination redis port.
  --ddb INTEGER    Destination redis DB.
  --dauth TEXT     Destination redis auth info.
  --limit INTEGER  Stop mirror process at limit X.
  --replace        Replace key if exists.
  --help           Show this message and exit.
```


## Useages
```Bash
redis-cli monitor | redismirror  --sport 6379 --shost localhost  --dhost 127.0.0.1 --dport 6379

#Exmaple 2
redis-cli monitor |  redismirror  --shost localhost --dport 6377  --limit 100
```

## Exmaple output:
```
$ redis-cli monitor | redismirror  --sport 6379 --shost localhost  --dhost 127.0.0.1 --dport 6378 --replace  --ttl --ttle 10 --limit 10

onnected to Redis: Host: localhost, Port: 6379, DB: 0
Connected to Redis: Host: 127.0.0.1, Port: 6378, DB: 0
✔  Mirrored key | Key: e2657ecd-6ee0-4182-9128-239be76cbba5_1679609769_json, TTL: 10, Status: OK - Command SET
☀  Skipping Key -> DUMP, Key -> e2657ecd-6ee0-4182-9128-239be76cbba5_1679609769_json
✔  Mirrored key | Key: e2657ecd-6ee0-4182-9128-239be76cbba5_1679609769_image_base64, TTL: 10, Status: OK - Command SET
☀  Skipping Key -> TTL, Key -> e2657ecd-6ee0-4182-9128-239be76cbba5_1679609769_json
✔  Mirrored key | Key: e2657ecd-6ee0-4182-9128-239be76cbba5_key1, TTL: 10, Status: OK - Command MSET
✔  Mirrored key | Key: e2657ecd-6ee0-4182-9128-239be76cbba5_setnx_key, TTL: 10, Status: OK - Command SETNX
✔  Mirrored key | Key: e2657ecd-6ee0-4182-9128-239be76cbba5_setex_key, TTL: 20, Status: OK - Command SETEX
☀  Skipping Key -> DUMP, Key -> e2657ecd-6ee0-4182-9128-239be76cbba5_1679609769_image_base64
✔  Mirrored key | Key: e2657ecd-6ee0-4182-9128-239be76cbba5_psetex_key, TTL: 20, Status: OK - Command PSETEX
☀  Skipping Key -> TTL, Key -> e2657ecd-6ee0-4182-9128-239be76cbba5_1679609769_image_base64
✔  Mirrored key | Key: e2657ecd-6ee0-4182-9128-239be76cbba5_getset_key, TTL: 10, Status: OK - Command GETSET
✔  Mirrored key | Key: e2657ecd-6ee0-4182-9128-239be76cbba5_hash, TTL: 10, Status: OK - Command HSET
✔  Mirrored key | Key: e2657ecd-6ee0-4182-9128-239be76cbba5_hash_multi, TTL: 10, Status: OK - Command HSET
✔  Mirrored key | Key: e2657ecd-6ee0-4182-9128-239be76cbba5_hash_setnx, TTL: 10, Status: OK - Command HSETNX
Limit reached: 10
```

## Installation using pypi
```
pip install redismirror
```

## Installation

```
$ pip install -r requirements.txt

$ pip install setup.py
```

## Development

This project includes a number of helpers in the `Makefile` to streamline common development tasks.

### Environment Setup

The following demonstrates setting up and working with a development environment:

```
### create a virtualenv for development

$ make virtualenv
Or
$ python3 -m venv env
then
$ source env/bin/activate
$ pip install -r requirements.txt

$ pip install setup.py

### run redismirror cli application

$ redismirror --help


### run pytest / coverage

$ make test
```


### Releasing to PyPi

Before releasing to PyPi, you must configure your login credentials:

**~/.pypirc**:

```
[pypi]
username = YOUR_USERNAME
password = YOUR_PASSWORD
```

Then use the included helper function via the `Makefile`:

```
$ make dist

$ make dist-upload

```

## Deployments

### Docker

Included is a basic `Dockerfile` for building and distributing `Redis Mirror `,
and can be built with the included `make` helper:

```
$ make docker

$ docker run -it redismirror --help
```



## Extra
To Generate sample data for your test using the below command:
```Bash
cd tests/Generator/;bash SampleDataInserter.sh

# Or Using python script
# Make sure to update config in the same file
python fillRedis.py
```


The following commands are also not logged:

```
skip_commands_list = [
    'FLUSHDB', 'INFO', 'FLUSHALL', 'AUTH', 'QUIT', 'SELECT', 'CLIENT', 'ROLE',
    'BGREWRITEAOF', 'TIME', 'ECHO', 'CONFIG', 'MONITOR', 'SYNC', 'SHUTDOWN',
    'DBSIZE', 'DEBUG', 'COMMAND', 'SCRIPT', 'SAVE', 'OBJECT', 'SLAVEOF',
    'KEYS', 'BGSAVE', 'SCAN', 'DUMP', 'SLOWLOG', 'TTL', 'PING', 'LASTSAVE'
]
```

Cost of running MONITOR
Because MONITOR streams back all commands, its use comes at a cost. The following (totally unscientific) benchmark numbers illustrate what the cost of running MONITOR can be.

Benchmark result without MONITOR running:


```Bash
$ src/redis-benchmark -c 10 -n 100000 -q
PING_INLINE: 101936.80 requests per second
PING_BULK: 102880.66 requests per second
SET: 95419.85 requests per second
GET: 104275.29 requests per second
INCR: 93283.58 requests per second
```
Benchmark results with MONITOR running (redis-cli monitor > /dev/null):
```Bash
$ src/redis-benchmark -c 10 -n 100000 -q
PING_INLINE: 58479.53 requests per second
PING_BULK: 59136.61 requests per second
SET: 41823.50 requests per second
GET: 45330.91 requests per second
INCR: 41771.09 requests per second
```
In this particular case, running a single MONITOR client can reduce the throughput by more than 50%. Running more MONITOR clients will reduce throughput even more.


License
-------

GNU GENERAL PUBLIC LICENSE

Author Information
------------------

The tool was originally developed by [Ali Saleh Baker](https://www.linkedin.com/in/alivx/).



            

Raw data

            {
    "_id": null,
    "home_page": "https://github.com/alivx/redis-mirror",
    "name": "redismirror",
    "maintainer": "",
    "docs_url": null,
    "requires_python": "",
    "maintainer_email": "",
    "keywords": "traffic,mirror,redis,migration,cli",
    "author": "Ali Saleh Baker",
    "author_email": "alivxlive@gmail.com",
    "download_url": "https://files.pythonhosted.org/packages/96/3e/eed5f2a5b83c2c16d93319694387bc28e3b9295ffb97ced6e814eca7eb92/redismirror-1.0.1.tar.gz",
    "platform": null,
    "description": "# Mirror Redis Traffic to another redis node\n<img src=\"https://raw.githubusercontent.com/alivx/redis-mirror/master/Generator/redis-mirror-logo.jpg\" alt=\"logo\" style=\"zoom:50%;\" />\n\n\n# redis-mirror\nRealtime Redis Traffic Mirror to another instance, this script reads the STDOUT from the `redis-cli monitor` command and mirrors the keys to another instance.\n\n## Use Case/Note\nIn some production/development cases, you need to mirror Redis traffic to another node to do some investigation or debugging.\n* This script does not set `TTL` to mirrored key since you need it for debugging.\n* The script support all command since it simply dump and restore the key as is.\n\n\n\n## TO DO:\n1. Add TTL as an option in mirrored redis instance, plus add an option to expand the origin `TTL`.\n2. Support cluster to a single redis instance or vice versa.\n3. Add an option to dump all keys name to the file for further analysis.\n4. Add an option to get all keys from the source and migrate the keys to another redis instance.\n\n## Option\n\n```\nredismirror --help\nUsage: redismirror [OPTIONS]\n\nOptions:\n  --shost TEXT     Source redis host/IP.\n  --sport INTEGER  Source redis port.\n  --sdb INTEGER    Source redis DB.\n  --sauth TEXT     Source redis auth info.\n  --dhost TEXT     Destination redis host/IP.\n  --dport INTEGER  Destination redis port.\n  --ddb INTEGER    Destination redis DB.\n  --dauth TEXT     Destination redis auth info.\n  --limit INTEGER  Stop mirror process at limit X.\n  --replace        Replace key if exists.\n  --help           Show this message and exit.\n```\n\n\n## Useages\n```Bash\nredis-cli monitor | redismirror  --sport 6379 --shost localhost  --dhost 127.0.0.1 --dport 6379\n\n#Exmaple 2\nredis-cli monitor |  redismirror  --shost localhost --dport 6377  --limit 100\n```\n\n## Exmaple output:\n```\n$ redis-cli monitor | redismirror  --sport 6379 --shost localhost  --dhost 127.0.0.1 --dport 6378 --replace  --ttl --ttle 10 --limit 10\n\nonnected to Redis: Host: localhost, Port: 6379, DB: 0\nConnected to Redis: Host: 127.0.0.1, Port: 6378, DB: 0\n\u2714  Mirrored key | Key: e2657ecd-6ee0-4182-9128-239be76cbba5_1679609769_json, TTL: 10, Status: OK - Command SET\n\u2600  Skipping Key -> DUMP, Key -> e2657ecd-6ee0-4182-9128-239be76cbba5_1679609769_json\n\u2714  Mirrored key | Key: e2657ecd-6ee0-4182-9128-239be76cbba5_1679609769_image_base64, TTL: 10, Status: OK - Command SET\n\u2600  Skipping Key -> TTL, Key -> e2657ecd-6ee0-4182-9128-239be76cbba5_1679609769_json\n\u2714  Mirrored key | Key: e2657ecd-6ee0-4182-9128-239be76cbba5_key1, TTL: 10, Status: OK - Command MSET\n\u2714  Mirrored key | Key: e2657ecd-6ee0-4182-9128-239be76cbba5_setnx_key, TTL: 10, Status: OK - Command SETNX\n\u2714  Mirrored key | Key: e2657ecd-6ee0-4182-9128-239be76cbba5_setex_key, TTL: 20, Status: OK - Command SETEX\n\u2600  Skipping Key -> DUMP, Key -> e2657ecd-6ee0-4182-9128-239be76cbba5_1679609769_image_base64\n\u2714  Mirrored key | Key: e2657ecd-6ee0-4182-9128-239be76cbba5_psetex_key, TTL: 20, Status: OK - Command PSETEX\n\u2600  Skipping Key -> TTL, Key -> e2657ecd-6ee0-4182-9128-239be76cbba5_1679609769_image_base64\n\u2714  Mirrored key | Key: e2657ecd-6ee0-4182-9128-239be76cbba5_getset_key, TTL: 10, Status: OK - Command GETSET\n\u2714  Mirrored key | Key: e2657ecd-6ee0-4182-9128-239be76cbba5_hash, TTL: 10, Status: OK - Command HSET\n\u2714  Mirrored key | Key: e2657ecd-6ee0-4182-9128-239be76cbba5_hash_multi, TTL: 10, Status: OK - Command HSET\n\u2714  Mirrored key | Key: e2657ecd-6ee0-4182-9128-239be76cbba5_hash_setnx, TTL: 10, Status: OK - Command HSETNX\nLimit reached: 10\n```\n\n## Installation using pypi\n```\npip install redismirror\n```\n\n## Installation\n\n```\n$ pip install -r requirements.txt\n\n$ pip install setup.py\n```\n\n## Development\n\nThis project includes a number of helpers in the `Makefile` to streamline common development tasks.\n\n### Environment Setup\n\nThe following demonstrates setting up and working with a development environment:\n\n```\n### create a virtualenv for development\n\n$ make virtualenv\nOr\n$ python3 -m venv env\nthen\n$ source env/bin/activate\n$ pip install -r requirements.txt\n\n$ pip install setup.py\n\n### run redismirror cli application\n\n$ redismirror --help\n\n\n### run pytest / coverage\n\n$ make test\n```\n\n\n### Releasing to PyPi\n\nBefore releasing to PyPi, you must configure your login credentials:\n\n**~/.pypirc**:\n\n```\n[pypi]\nusername = YOUR_USERNAME\npassword = YOUR_PASSWORD\n```\n\nThen use the included helper function via the `Makefile`:\n\n```\n$ make dist\n\n$ make dist-upload\n\n```\n\n## Deployments\n\n### Docker\n\nIncluded is a basic `Dockerfile` for building and distributing `Redis Mirror `,\nand can be built with the included `make` helper:\n\n```\n$ make docker\n\n$ docker run -it redismirror --help\n```\n\n\n\n## Extra\nTo Generate sample data for your test using the below command:\n```Bash\ncd tests/Generator/;bash SampleDataInserter.sh\n\n# Or Using python script\n# Make sure to update config in the same file\npython fillRedis.py\n```\n\n\nThe following commands are also not logged:\n\n```\nskip_commands_list = [\n    'FLUSHDB', 'INFO', 'FLUSHALL', 'AUTH', 'QUIT', 'SELECT', 'CLIENT', 'ROLE',\n    'BGREWRITEAOF', 'TIME', 'ECHO', 'CONFIG', 'MONITOR', 'SYNC', 'SHUTDOWN',\n    'DBSIZE', 'DEBUG', 'COMMAND', 'SCRIPT', 'SAVE', 'OBJECT', 'SLAVEOF',\n    'KEYS', 'BGSAVE', 'SCAN', 'DUMP', 'SLOWLOG', 'TTL', 'PING', 'LASTSAVE'\n]\n```\n\nCost of running MONITOR\nBecause MONITOR streams back all commands, its use comes at a cost. The following (totally unscientific) benchmark numbers illustrate what the cost of running MONITOR can be.\n\nBenchmark result without MONITOR running:\n\n\n```Bash\n$ src/redis-benchmark -c 10 -n 100000 -q\nPING_INLINE: 101936.80 requests per second\nPING_BULK: 102880.66 requests per second\nSET: 95419.85 requests per second\nGET: 104275.29 requests per second\nINCR: 93283.58 requests per second\n```\nBenchmark results with MONITOR running (redis-cli monitor > /dev/null):\n```Bash\n$ src/redis-benchmark -c 10 -n 100000 -q\nPING_INLINE: 58479.53 requests per second\nPING_BULK: 59136.61 requests per second\nSET: 41823.50 requests per second\nGET: 45330.91 requests per second\nINCR: 41771.09 requests per second\n```\nIn this particular case, running a single MONITOR client can reduce the throughput by more than 50%. Running more MONITOR clients will reduce throughput even more.\n\n\nLicense\n-------\n\nGNU GENERAL PUBLIC LICENSE\n\nAuthor Information\n------------------\n\nThe tool was originally developed by [Ali Saleh Baker](https://www.linkedin.com/in/alivx/).\n\n\n",
    "bugtrack_url": null,
    "license": "MIT",
    "summary": "Mirror Redis Traffic to another redis node",
    "version": "1.0.1",
    "split_keywords": [
        "traffic",
        "mirror",
        "redis",
        "migration",
        "cli"
    ],
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "25254a23014b7f443c2f134856f5d3fa3d6f041ee6d6ce1f17f2b249bab3ef8b",
                "md5": "80abc6c97dd60fcf1c99457790de85a3",
                "sha256": "10ddbcd996c8917924c9e829a40cefe2d3550f0dbd150d1ddb72804a39966c4d"
            },
            "downloads": -1,
            "filename": "redismirror-1.0.1-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "80abc6c97dd60fcf1c99457790de85a3",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": null,
            "size": 18135,
            "upload_time": "2023-03-23T22:38:30",
            "upload_time_iso_8601": "2023-03-23T22:38:30.132009Z",
            "url": "https://files.pythonhosted.org/packages/25/25/4a23014b7f443c2f134856f5d3fa3d6f041ee6d6ce1f17f2b249bab3ef8b/redismirror-1.0.1-py3-none-any.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "963eeed5f2a5b83c2c16d93319694387bc28e3b9295ffb97ced6e814eca7eb92",
                "md5": "3d2cbe28ec35fa16673661cef576c92d",
                "sha256": "1a31787edd11b01b19ade88f3923013df34c275a5c74dd76fd69a47a04e18524"
            },
            "downloads": -1,
            "filename": "redismirror-1.0.1.tar.gz",
            "has_sig": false,
            "md5_digest": "3d2cbe28ec35fa16673661cef576c92d",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 18146,
            "upload_time": "2023-03-23T22:38:32",
            "upload_time_iso_8601": "2023-03-23T22:38:32.449778Z",
            "url": "https://files.pythonhosted.org/packages/96/3e/eed5f2a5b83c2c16d93319694387bc28e3b9295ffb97ced6e814eca7eb92/redismirror-1.0.1.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2023-03-23 22:38:32",
    "github": true,
    "gitlab": false,
    "bitbucket": false,
    "github_user": "alivx",
    "github_project": "redis-mirror",
    "travis_ci": false,
    "coveralls": false,
    "github_actions": true,
    "requirements": [],
    "lcname": "redismirror"
}
        
Elapsed time: 0.05022s