txredisapi
==========
[](http://travis-ci.org/IlyaSkriblovsky/txredisapi)
*For the latest source code, see <http://github.com/IlyaSkriblovsky/txredisapi>*
``txredisapi`` is a non-blocking client driver for the [redis](http://redis.io)
database, written in Python. It uses [Twisted](http://twistedmatrix.com) for
the asynchronous communication with redis.
It started as a fork of the original
[redis protocol for twisted](http://pypi.python.org/pypi/txredis/), and evolved
into a more robust, reliable, and complete solution for applications like web
servers. These types of applications often need a fault-tolerant pool of
connections with multiple redis servers, making it possible to easily develop
and maintain distributed systems.
Most of the [redis commands](http://redis.io/commands) are supported, as well
as other features such as silent reconnection, connection pools, and automatic
sharding.
This driver is distributed as part of the [cyclone](http://cyclone.io) web
framework.
### Changelog ###
See [CHANGELOG.md](CHANGELOG.md)
### Features ###
- Connection Pools
- Lazy Connections
- Automatic Sharding
- Automatic Reconnection
- Connection using Redis Sentinel
- Publish/Subscribe (PubSub)
- Transactions
- Unix Socket Connections
Install
-------
Bear in mind that ``txredisapi.py`` is pure-python, in a single file.
Thus, there's absolutely no need to install it. Instead, just copy it to your
project directory and start using.
Latest source code is at <https://github.com/IlyaSkriblovsky/txredisapi>.
If you have [cyclone](http://cyclone.io), you probably already have it too.
Try the following:
$ python
>>> import cyclone.redis
>>> cyclone.redis.version
'1.0'
However, if you really really insist in installing, get it from pypi:
pip install txredisapi
### Unit Tests ###
[Twisted Trial](http://twistedmatrix.com/trac/wiki/TwistedTrial) unit tests
are available. Just start redis, and run ``trial ./tests``.
If *unix sockets* are disabled in redis, it will silently skip those tests.
Make sure you run `redis-cli flushall` to clean up redis after the tests.
Usage
-----
First thing to do is choose what type of connection you want. The driver
supports single connection, connection pools, sharded connections (with
automatic distribution based on a built-in consistent hashing algorithm),
sharded connection pools, and all of these different types can be *lazy*,
which is explained later (because I'm lazy now).
Basically, you want normal connections for simple batch clients that connect
to redis, execute a couple of commands and disconnect - like crawlers, etc.
Example:
#!/usr/bin/env python
# coding: utf-8
import txredisapi as redis
from twisted.internet import defer
from twisted.internet import reactor
@defer.inlineCallbacks
def main():
rc = yield redis.Connection()
print rc
yield rc.set("foo", "bar")
v = yield rc.get("foo")
print "foo:", repr(v)
yield rc.disconnect()
if __name__ == "__main__":
main().addCallback(lambda ign: reactor.stop())
reactor.run()
Easily switch between ``redis.Connection()`` and ``redis.ConnectionPool()``
with absolutely no changes to the logic of your program.
These are all the supported methods for connecting to Redis::
Connection(host, port, dbid, reconnect, charset)
lazyConnection(host, port, dbid, reconnect, charset)
ConnectionPool(host, port, dbid, poolsize, reconnect, charset)
lazyConnectionPool(host, port, dbid, poolsize, reconnect, charset)
ShardedConnection(hosts, dbid, reconnect, charset)
lazyShardedConnection(hosts, dbid, reconnect, charset)
ShardedConnectionPool(hosts, dbid, poolsize, reconnect, charset)
lazyShardedConnectionPool(hosts, dbid, poolsize, reconnect, charset)
UnixConnection(path, dbid, reconnect, charset)
lazyUnixConnection(path, dbid, reconnect, charset)
UnixConnectionPool(unix, dbid, poolsize, reconnect, charset)
lazyUnixConnectionPool(unix, dbid, poolsize, reconnect, charset)
ShardedUnixConnection(paths, dbid, reconnect, charset)
lazyShardedUnixConnection(paths, dbid, reconnect, charset)
ShardedUnixConnectionPool(paths, dbid, poolsize, reconnect, charset)
lazyShardedUnixConnectionPool(paths, dbid, poolsize, reconnect, charset)
The arguments are:
- host: the IP address or hostname of the redis server. [default: localhost]
- port: port number of the redis server. [default: 6379]
- path: path of redis server's socket [default: /tmp/redis.sock]
- dbid: database id of redis server. [default: 0]
- poolsize: how many connections to make. [default: 10]
- reconnect: auto-reconnect if connection is lost. [default: True]
- charset: string encoding. Do not decode/encode strings if None.
[default: utf-8]
- hosts (for sharded): list of ``host:port`` pairs. [default: None]
- paths (for sharded): list of ``pathnames``. [default: None]
- password: password for the redis server. [default: None]
- ssl_context_factory: Either a boolean indicating wether to use SSL/TLS or a specific `ClientContextFactory`. [default: False]
### Connection Handlers ###
All connection methods return a connection handler object at some point.
Normal connections (not lazy) return a deferred, which is fired with the
connection handler after the connection is established.
In case of connection pools, it will only fire the callback after all
connections are set up, and ready.
Connection handler is the client interface with redis. It accepts all the
commands supported by redis, such as ``get``, ``set``, etc. It is the ``rc``
object in the example below.
Connection handlers will automatically select one of the available connections
in connection pools, and automatically reconnect to redis when necessary.
If the connection with redis is lost, all commands will raise the
``ConnectionError`` exception, to indicate that there's no active connection.
However, if the ``reconnect`` argument was set to ``True`` during the
initialization, it will continuosly try to reconnect, in background.
Example:
#!/usr/bin/env python
# coding: utf-8
import txredisapi as redis
from twisted.internet import defer
from twisted.internet import reactor
def sleep(n):
d = defer.Deferred()
reactor.callLater(5, lambda *ign: d.callback(None))
return d
@defer.inlineCallbacks
def main():
rc = yield redis.ConnectionPool()
print rc
# set
yield rc.set("foo", "bar")
# sleep, so you can kill redis
print "sleeping for 5s, kill redis now..."
yield sleep(5)
try:
v = yield rc.get("foo")
print "foo:", v
yield rc.disconnect()
except redis.ConnectionError, e:
print str(e)
if __name__ == "__main__":
main().addCallback(lambda ign: reactor.stop())
reactor.run()
### Lazy Connections ###
This type of connection will immediately return the connection handler object,
even before the connection is made.
It will start the connection, (or connections, in case of connection pools) in
background, and automatically reconnect if necessary.
You want lazy connections when you're writing servers, like web servers, or
any other type of server that should not wait for the redis connection during
the initialization of the program.
The example below is a web application, which will expose redis set, get and
delete commands over HTTP.
If the database connection is down (either because redis is not running, or
whatever reason), the web application will start normally. If connection is
lost during the operation, nothing will change.
When there's no connection, all commands will fail, therefore the web
application will respond with HTTP 503 (Service Unavailable). It will resume to
normal once the connection with redis is re-established.
Try killing redis server after the application is running, and make a couple
of requests. Then, start redis again and give it another try.
Example:
#!/usr/bin/env python
# coding: utf-8
import sys
import cyclone.web
import cyclone.redis
from twisted.internet import defer
from twisted.internet import reactor
from twisted.python import log
class Application(cyclone.web.Application):
def __init__(self):
handlers = [ (r"/text/(.+)", TextHandler) ]
RedisMixin.setup()
cyclone.web.Application.__init__(self, handlers, debug=True)
class RedisMixin(object):
redis_conn = None
@classmethod
def setup(self):
RedisMixin.redis_conn = cyclone.redis.lazyConnectionPool()
# Provide GET, SET and DELETE redis operations via HTTP
class TextHandler(cyclone.web.RequestHandler, RedisMixin):
@defer.inlineCallbacks
def get(self, key):
try:
value = yield self.redis_conn.get(key)
except Exception, e:
log.msg("Redis failed to get('%s'): %s" % (key, str(e)))
raise cyclone.web.HTTPError(503)
self.set_header("Content-Type", "text/plain")
self.write("%s=%s\r\n" % (key, value))
@defer.inlineCallbacks
def post(self, key):
value = self.get_argument("value")
try:
yield self.redis_conn.set(key, value)
except Exception, e:
log.msg("Redis failed to set('%s', '%s'): %s" % (key, value, str(e)))
raise cyclone.web.HTTPError(503)
self.set_header("Content-Type", "text/plain")
self.write("%s=%s\r\n" % (key, value))
@defer.inlineCallbacks
def delete(self, key):
try:
n = yield self.redis_conn.delete(key)
except Exception, e:
log.msg("Redis failed to del('%s'): %s" % (key, str(e)))
raise cyclone.web.HTTPError(503)
self.set_header("Content-Type", "text/plain")
self.write("DEL %s=%d\r\n" % (key, n))
def main():
log.startLogging(sys.stdout)
reactor.listenTCP(8888, Application(), interface="127.0.0.1")
reactor.run()
if __name__ == "__main__":
main()
This is the server running in one terminal::
$ ./helloworld.py
2012-02-17 15:40:25-0500 [-] Log opened.
2012-02-17 15:40:25-0500 [-] Starting factory <redis.Factory instance at 0x1012f0560>
2012-02-17 15:40:25-0500 [-] __main__.Application starting on 8888
2012-02-17 15:40:25-0500 [-] Starting factory <__main__.Application instance at 0x100f42290>
2012-02-17 15:40:53-0500 [RedisProtocol,client] 200 POST /text/foo (127.0.0.1) 1.20ms
2012-02-17 15:41:01-0500 [RedisProtocol,client] 200 GET /text/foo (127.0.0.1) 0.97ms
2012-02-17 15:41:09-0500 [RedisProtocol,client] 200 DELETE /text/foo (127.0.0.1) 0.65ms
(killed redis-server)
2012-02-17 15:48:48-0500 [HTTPConnection,0,127.0.0.1] Redis failed to get('foo'): Not connected
2012-02-17 15:48:48-0500 [HTTPConnection,0,127.0.0.1] 503 GET /text/foo (127.0.0.1) 2.99ms
And these are the requests, from ``curl`` in another terminal.
Set:
$ curl -D - -d "value=bar" http://localhost:8888/text/foo
HTTP/1.1 200 OK
Content-Length: 9
Content-Type: text/plain
foo=bar
Get:
$ curl -D - http://localhost:8888/text/foo
HTTP/1.1 200 OK
Content-Length: 9
Etag: "b63729aa7fa0e438eed735880951dcc21d733676"
Content-Type: text/plain
foo=bar
Delete:
$ curl -D - -X DELETE http://localhost:8888/text/foo
HTTP/1.1 200 OK
Content-Length: 11
Content-Type: text/plain
DEL foo=1
When redis is not running:
$ curl -D - http://localhost:8888/text/foo
HTTP/1.1 503 Service Unavailable
Content-Length: 89
Content-Type: text/html; charset=UTF-8
<html><title>503: Service Unavailable</title>
<body>503: Service Unavailable</body></html>
### Sharded Connections ###
They can be normal, or lazy connections. They can be sharded connection pools.
Not all commands are supported on sharded connections.
If the command you're trying to run is not supported on sharded connections,
the connection handler will raise the ``NotImplementedError`` exception.
Simple example with automatic sharding of keys between two redis servers:
#!/usr/bin/env python
# coding: utf-8
import txredisapi as redis
from twisted.internet import defer
from twisted.internet import reactor
@defer.inlineCallbacks
def main():
rc = yield redis.ShardedConnection(["localhost:6379", "localhost:6380"])
print rc
print "Supported methods on sharded connections:", rc.ShardedMethods
keys = []
for x in xrange(100):
key = "foo%02d" % x
yield rc.set(key, "bar%02d" % x)
keys.append(key)
# yey! mget is supported!
response = yield rc.mget(keys)
for val in response:
print val
yield rc.disconnect()
if __name__ == "__main__":
main().addCallback(lambda ign: reactor.stop())
reactor.run()
### Transactions ###
For obvious reasons, transactions are NOT supported on sharded connections.
But they work pretty good on normal or lazy connections, and connection pools.
NOTE: redis uses the following methods for transactions:
- WATCH: synchronization
- MULTI: start the transaction
- EXEC: commit the transaction
- DISCARD: you got it.
Because ``exec`` is a reserved word in Python, the command to commit is
``commit``.
Example:
#!/usr/bin/env python
# coding: utf-8
import txredisapi as redis
from twisted.internet import defer
from twisted.internet import reactor
@defer.inlineCallbacks
def main():
rc = yield redis.ConnectionPool()
# Remove the keys
yield rc.delete(["a1", "a2", "a3"])
# Start transaction
t = yield rc.multi()
# These will return "QUEUED" - even t.get(key)
yield t.set("a1", "1")
yield t.set("a2", "2")
yield t.set("a3", "3")
yield t.get("a1")
# Try to call get() while in a transaction.
# It will fail if it's not a connection pool, or if all connections
# in the pool are in a transaction.
# Note that it's rc.get(), not the transaction object t.get().
try:
v = yield rc.get("foo")
print "foo=", v
except Exception, e:
print "can't get foo:", e
# Commit, and get all responses from transaction.
r = yield t.commit()
print "commit=", repr(r)
yield rc.disconnect()
if __name__ == "__main__":
main().addCallback(lambda ign: reactor.stop())
reactor.run()
A "COUNTER" example, using WATCH/MULTI:
#!/usr/bin/env python
# coding: utf-8
import txredisapi as redis
from twisted.internet import defer
from twisted.internet import reactor
@defer.inlineCallbacks
def main():
rc = yield redis.ConnectionPool()
# Reset keys
yield rc.set("a1", 0)
# Synchronize and start transaction
t = yield rc.watch("a1")
# Load previous value
a1 = yield t.get("a1")
# start the transactional pipeline
yield t.multi()
# modify and retrieve the new a1 value
yield t.set("a1", a1 + 1)
yield t.get("a1")
print "simulating concurrency, this will abort the transaction"
yield rc.set("a1", 2)
try:
r = yield t.commit()
print "commit=", repr(r)
except redis.WatchError, e:
a1 = yield rc.get("a1")
print "transaction has failed."
print "current a1 value: ", a1
yield rc.disconnect()
if __name__ == "__main__":
main().addCallback(lambda ign: reactor.stop())
reactor.run()
Calling ``commit`` will cause it to return a list with the return of all
commands executed in the transaction. ``discard``, on the other hand, will
normally return just an ``OK``.
### Pipelining ###
txredisapi automatically [pipelines](http://redis.io/topics/pipelining) all commands
by sending next commands without waiting for the previous one to receive reply from
server. This works even on single connections and increases performance by reducing
number of round-trip delays and. There are two exceptions, though:
- no commands will be sent after blocking `blpop`, `brpop` or `brpoplpush` until
response is received;
- transaction by `multi`/`commit` are also blocking connection making all other
commands to wait until transaction is executed.
When you need to load tons of data to Redis it might be more effective to sent
commands in batches grouping them together offline to save on TCP packets and network
stack overhead. You can do this using `pipeline` method to explicitly accumulate
commands and send them to server in a single batch. Be careful to not accumulate too
many commands: unreasonable batch size may eat up unexpected amount of memory on both
client and server side. Group commands in batches of, for example, 10k commands instead
of sending all your data at once. The speed will be nearly the same, but the additional
memory used will be at max the amount needed to queue this 10k commands
To send commands in a batch:
#!/usr/bin/env python
# coding: utf-8
import txredisapi as redis
from twisted.internet import defer
from twisted.internet import reactor
@defer.inlineCallbacks
def main():
rc = yield redis.ConnectionPool()
# Start grouping commands
pipeline = yield rc.pipeline()
pipeline.set("foo", 123)
pipeline.set("bar", 987)
pipeline.get("foo")
pipeline.get("bar")
# Write those 2 sets and 2 gets to redis all at once, and wait
# for all replies before continuing.
results = yield pipeline.execute_pipeline()
print "foo:", results[2] # should be 123
print "bar:", results[3] # should be 987
yield rc.disconnect()
if __name__ == "__main__":
main().addCallback(lambda ign: reactor.stop())
reactor.run()
### Authentication ###
This is how to authenticate::
#!/usr/bin/env python
import txredisapi
from twisted.internet import defer
from twisted.internet import reactor
@defer.inlineCallbacks
def main():
redis = yield txredisapi.Connection(password="foobared")
yield redis.set("foo", "bar")
print (yield redis.get("foo"))
reactor.stop()
if __name__ == "__main__":
main()
reactor.run()
### Connection using Redis Sentinel ###
`txredisapi` can discover Redis master and slaves addresses using
[Redis Sentinel](http://redis.io/topics/sentinel) and automatically failover
in case of server failure.
#!/usr/bin/env python
from twisted.internet.task import react
import txredisapi
@defer.inlineCallbacks
def main(reactor):
sentinel = txredisapi.Sentinel([("sentinel-a", 26379), ("sentinel-b", 26379), ("sentinel-c", 26379)])
redis = sentinel.master_for("service_name")
yield redis.set("foo", "bar")
print (yield redis.get("foo"))
yield redis.disconnect()
yield sentinel.disconnect()
react(main)
Usual connection arguments like `dbid=N` or `poolsize=N` can be specified in
`master_for()` call. Use `sentinel.slave_for()` to connect to one of the slaves
instead of master.
Add `min_other_sentinels=N` to `Sentinel` constructor call to make it obey information
only from sentinels that currently connected to specified number of other sentinels
to minimize a risk of split-brain in case of network partitioning.
Credits
=======
Thanks to (in no particular order):
- Alexandre Fiori
- Author of txredisapi
- Gleicon Moraes
- Bug fixes, testing, and [RestMQ](http://github.com/gleicon/restmq>).
- For writing the Consistent Hashing algorithm used for sharding.
- Dorian Raymer and Ludovico Magnocavallo
- Authors of the original *redis protocol for twisted*.
- Vanderson Mota
- Initial pypi setup, and patches.
- Jeethu Rao
- Contributed with test cases, and other ideas like support for travis-ci
- Jeremy Archer
- Minor bugfixes.
- Christoph Tavan (@ctavan)
- Idea and test case for nested multi bulk replies, minor command enhancements.
- dgvncsz0f
- WATCH/UNWATCH commands
- Ilia Glazkov
- Free connection selection algorithm for pools.
- Non-unicode charset fixes.
- SCAN commands
- Matt Pizzimenti (mjpizz)
- pipelining support
- Nickolai Novik (jettify)
- update of SET command
- Evgeny Tataurov (etataurov)
- Ability to use hiredis protocol parser
- Ilya Skriblovsky (IlyaSkriblovsky)
- Sentinel support
Raw data
{
"_id": null,
"home_page": "http://github.com/IlyaSkriblovsky/txredisapi",
"name": "txredisapi",
"maintainer": "",
"docs_url": null,
"requires_python": "",
"maintainer_email": "",
"keywords": "",
"author": "Alexandre Fiori",
"author_email": "fiorix@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/80/9a/3ecb0d5fb8fc37c977673c3f2b43ff799ed46517549da768edb4fb5c750e/txredisapi-1.4.10.tar.gz",
"platform": null,
"description": "txredisapi\n==========\n\n[](http://travis-ci.org/IlyaSkriblovsky/txredisapi)\n\n\n*For the latest source code, see <http://github.com/IlyaSkriblovsky/txredisapi>*\n\n\n``txredisapi`` is a non-blocking client driver for the [redis](http://redis.io)\ndatabase, written in Python. It uses [Twisted](http://twistedmatrix.com) for\nthe asynchronous communication with redis.\n\nIt started as a fork of the original\n[redis protocol for twisted](http://pypi.python.org/pypi/txredis/), and evolved\ninto a more robust, reliable, and complete solution for applications like web\nservers. These types of applications often need a fault-tolerant pool of\nconnections with multiple redis servers, making it possible to easily develop\nand maintain distributed systems.\n\nMost of the [redis commands](http://redis.io/commands) are supported, as well\nas other features such as silent reconnection, connection pools, and automatic\nsharding.\n\nThis driver is distributed as part of the [cyclone](http://cyclone.io) web\nframework.\n\n### Changelog ###\n\nSee [CHANGELOG.md](CHANGELOG.md)\n\n### Features ###\n\n- Connection Pools\n- Lazy Connections\n- Automatic Sharding\n- Automatic Reconnection\n- Connection using Redis Sentinel\n- Publish/Subscribe (PubSub)\n- Transactions\n- Unix Socket Connections\n\n\nInstall\n-------\n\nBear in mind that ``txredisapi.py`` is pure-python, in a single file.\nThus, there's absolutely no need to install it. Instead, just copy it to your\nproject directory and start using.\n\nLatest source code is at <https://github.com/IlyaSkriblovsky/txredisapi>.\n\nIf you have [cyclone](http://cyclone.io), you probably already have it too.\nTry the following:\n\n $ python\n >>> import cyclone.redis\n >>> cyclone.redis.version\n '1.0'\n\nHowever, if you really really insist in installing, get it from pypi:\n\n pip install txredisapi\n\n\n### Unit Tests ###\n\n[Twisted Trial](http://twistedmatrix.com/trac/wiki/TwistedTrial) unit tests\nare available. Just start redis, and run ``trial ./tests``.\nIf *unix sockets* are disabled in redis, it will silently skip those tests.\n\nMake sure you run `redis-cli flushall` to clean up redis after the tests.\n\nUsage\n-----\n\nFirst thing to do is choose what type of connection you want. The driver\nsupports single connection, connection pools, sharded connections (with\nautomatic distribution based on a built-in consistent hashing algorithm),\nsharded connection pools, and all of these different types can be *lazy*,\nwhich is explained later (because I'm lazy now).\n\nBasically, you want normal connections for simple batch clients that connect\nto redis, execute a couple of commands and disconnect - like crawlers, etc.\n\nExample:\n\n #!/usr/bin/env python\n # coding: utf-8\n\n import txredisapi as redis\n\n from twisted.internet import defer\n from twisted.internet import reactor\n\n\n @defer.inlineCallbacks\n def main():\n rc = yield redis.Connection()\n print rc\n\n yield rc.set(\"foo\", \"bar\")\n v = yield rc.get(\"foo\")\n print \"foo:\", repr(v)\n\n yield rc.disconnect()\n\n\n if __name__ == \"__main__\":\n main().addCallback(lambda ign: reactor.stop())\n reactor.run()\n\n\nEasily switch between ``redis.Connection()`` and ``redis.ConnectionPool()``\nwith absolutely no changes to the logic of your program.\n\nThese are all the supported methods for connecting to Redis::\n\n Connection(host, port, dbid, reconnect, charset)\n lazyConnection(host, port, dbid, reconnect, charset)\n\n ConnectionPool(host, port, dbid, poolsize, reconnect, charset)\n lazyConnectionPool(host, port, dbid, poolsize, reconnect, charset)\n\n ShardedConnection(hosts, dbid, reconnect, charset)\n lazyShardedConnection(hosts, dbid, reconnect, charset)\n\n ShardedConnectionPool(hosts, dbid, poolsize, reconnect, charset)\n lazyShardedConnectionPool(hosts, dbid, poolsize, reconnect, charset)\n\n UnixConnection(path, dbid, reconnect, charset)\n lazyUnixConnection(path, dbid, reconnect, charset)\n\n UnixConnectionPool(unix, dbid, poolsize, reconnect, charset)\n lazyUnixConnectionPool(unix, dbid, poolsize, reconnect, charset)\n\n ShardedUnixConnection(paths, dbid, reconnect, charset)\n lazyShardedUnixConnection(paths, dbid, reconnect, charset)\n\n ShardedUnixConnectionPool(paths, dbid, poolsize, reconnect, charset)\n lazyShardedUnixConnectionPool(paths, dbid, poolsize, reconnect, charset)\n\n\nThe arguments are:\n\n- host: the IP address or hostname of the redis server. [default: localhost]\n- port: port number of the redis server. [default: 6379]\n- path: path of redis server's socket [default: /tmp/redis.sock]\n- dbid: database id of redis server. [default: 0]\n- poolsize: how many connections to make. [default: 10]\n- reconnect: auto-reconnect if connection is lost. [default: True]\n- charset: string encoding. Do not decode/encode strings if None.\n [default: utf-8]\n- hosts (for sharded): list of ``host:port`` pairs. [default: None]\n- paths (for sharded): list of ``pathnames``. [default: None]\n- password: password for the redis server. [default: None]\n- ssl_context_factory: Either a boolean indicating wether to use SSL/TLS or a specific `ClientContextFactory`. [default: False]\n\n\n### Connection Handlers ###\n\nAll connection methods return a connection handler object at some point.\n\nNormal connections (not lazy) return a deferred, which is fired with the\nconnection handler after the connection is established.\n\nIn case of connection pools, it will only fire the callback after all\nconnections are set up, and ready.\n\nConnection handler is the client interface with redis. It accepts all the\ncommands supported by redis, such as ``get``, ``set``, etc. It is the ``rc``\nobject in the example below.\n\nConnection handlers will automatically select one of the available connections\nin connection pools, and automatically reconnect to redis when necessary.\n\nIf the connection with redis is lost, all commands will raise the\n``ConnectionError`` exception, to indicate that there's no active connection.\nHowever, if the ``reconnect`` argument was set to ``True`` during the\ninitialization, it will continuosly try to reconnect, in background.\n\nExample:\n\n #!/usr/bin/env python\n # coding: utf-8\n\n import txredisapi as redis\n\n from twisted.internet import defer\n from twisted.internet import reactor\n\n\n def sleep(n):\n d = defer.Deferred()\n reactor.callLater(5, lambda *ign: d.callback(None))\n return d\n\n\n @defer.inlineCallbacks\n def main():\n rc = yield redis.ConnectionPool()\n print rc\n\n # set\n yield rc.set(\"foo\", \"bar\")\n\n # sleep, so you can kill redis\n print \"sleeping for 5s, kill redis now...\"\n yield sleep(5)\n\n try:\n v = yield rc.get(\"foo\")\n print \"foo:\", v\n\n yield rc.disconnect()\n except redis.ConnectionError, e:\n print str(e)\n\n\n if __name__ == \"__main__\":\n main().addCallback(lambda ign: reactor.stop())\n reactor.run()\n\n\n### Lazy Connections ###\n\nThis type of connection will immediately return the connection handler object,\neven before the connection is made.\n\nIt will start the connection, (or connections, in case of connection pools) in\nbackground, and automatically reconnect if necessary.\n\nYou want lazy connections when you're writing servers, like web servers, or\nany other type of server that should not wait for the redis connection during\nthe initialization of the program.\n\nThe example below is a web application, which will expose redis set, get and\ndelete commands over HTTP.\n\nIf the database connection is down (either because redis is not running, or\nwhatever reason), the web application will start normally. If connection is\nlost during the operation, nothing will change.\n\nWhen there's no connection, all commands will fail, therefore the web\napplication will respond with HTTP 503 (Service Unavailable). It will resume to\nnormal once the connection with redis is re-established.\n\nTry killing redis server after the application is running, and make a couple\nof requests. Then, start redis again and give it another try.\n\nExample:\n\n #!/usr/bin/env python\n # coding: utf-8\n\n import sys\n\n import cyclone.web\n import cyclone.redis\n from twisted.internet import defer\n from twisted.internet import reactor\n from twisted.python import log\n\n\n class Application(cyclone.web.Application):\n def __init__(self):\n handlers = [ (r\"/text/(.+)\", TextHandler) ]\n\n RedisMixin.setup()\n cyclone.web.Application.__init__(self, handlers, debug=True)\n\n\n class RedisMixin(object):\n redis_conn = None\n\n @classmethod\n def setup(self):\n RedisMixin.redis_conn = cyclone.redis.lazyConnectionPool()\n\n\n # Provide GET, SET and DELETE redis operations via HTTP\n class TextHandler(cyclone.web.RequestHandler, RedisMixin):\n @defer.inlineCallbacks\n def get(self, key):\n try:\n value = yield self.redis_conn.get(key)\n except Exception, e:\n log.msg(\"Redis failed to get('%s'): %s\" % (key, str(e)))\n raise cyclone.web.HTTPError(503)\n\n self.set_header(\"Content-Type\", \"text/plain\")\n self.write(\"%s=%s\\r\\n\" % (key, value))\n\n @defer.inlineCallbacks\n def post(self, key):\n value = self.get_argument(\"value\")\n try:\n yield self.redis_conn.set(key, value)\n except Exception, e:\n log.msg(\"Redis failed to set('%s', '%s'): %s\" % (key, value, str(e)))\n raise cyclone.web.HTTPError(503)\n\n self.set_header(\"Content-Type\", \"text/plain\")\n self.write(\"%s=%s\\r\\n\" % (key, value))\n\n @defer.inlineCallbacks\n def delete(self, key):\n try:\n n = yield self.redis_conn.delete(key)\n except Exception, e:\n log.msg(\"Redis failed to del('%s'): %s\" % (key, str(e)))\n raise cyclone.web.HTTPError(503)\n\n self.set_header(\"Content-Type\", \"text/plain\")\n self.write(\"DEL %s=%d\\r\\n\" % (key, n))\n\n\n def main():\n log.startLogging(sys.stdout)\n reactor.listenTCP(8888, Application(), interface=\"127.0.0.1\")\n reactor.run()\n\n\n if __name__ == \"__main__\":\n main()\n\n\nThis is the server running in one terminal::\n\n $ ./helloworld.py\n 2012-02-17 15:40:25-0500 [-] Log opened.\n 2012-02-17 15:40:25-0500 [-] Starting factory <redis.Factory instance at 0x1012f0560>\n 2012-02-17 15:40:25-0500 [-] __main__.Application starting on 8888\n 2012-02-17 15:40:25-0500 [-] Starting factory <__main__.Application instance at 0x100f42290>\n 2012-02-17 15:40:53-0500 [RedisProtocol,client] 200 POST /text/foo (127.0.0.1) 1.20ms\n 2012-02-17 15:41:01-0500 [RedisProtocol,client] 200 GET /text/foo (127.0.0.1) 0.97ms\n 2012-02-17 15:41:09-0500 [RedisProtocol,client] 200 DELETE /text/foo (127.0.0.1) 0.65ms\n (killed redis-server)\n 2012-02-17 15:48:48-0500 [HTTPConnection,0,127.0.0.1] Redis failed to get('foo'): Not connected\n 2012-02-17 15:48:48-0500 [HTTPConnection,0,127.0.0.1] 503 GET /text/foo (127.0.0.1) 2.99ms\n\n\nAnd these are the requests, from ``curl`` in another terminal.\n\nSet:\n\n $ curl -D - -d \"value=bar\" http://localhost:8888/text/foo\n HTTP/1.1 200 OK\n Content-Length: 9\n Content-Type: text/plain\n\n foo=bar\n\nGet:\n\n $ curl -D - http://localhost:8888/text/foo\n HTTP/1.1 200 OK\n Content-Length: 9\n Etag: \"b63729aa7fa0e438eed735880951dcc21d733676\"\n Content-Type: text/plain\n\n foo=bar\n\nDelete:\n\n $ curl -D - -X DELETE http://localhost:8888/text/foo\n HTTP/1.1 200 OK\n Content-Length: 11\n Content-Type: text/plain\n\n DEL foo=1\n\nWhen redis is not running:\n\n $ curl -D - http://localhost:8888/text/foo\n HTTP/1.1 503 Service Unavailable\n Content-Length: 89\n Content-Type: text/html; charset=UTF-8\n\n <html><title>503: Service Unavailable</title>\n <body>503: Service Unavailable</body></html>\n\n\n### Sharded Connections ###\n\nThey can be normal, or lazy connections. They can be sharded connection pools.\nNot all commands are supported on sharded connections.\n\nIf the command you're trying to run is not supported on sharded connections,\nthe connection handler will raise the ``NotImplementedError`` exception.\n\nSimple example with automatic sharding of keys between two redis servers:\n\n #!/usr/bin/env python\n # coding: utf-8\n\n import txredisapi as redis\n\n from twisted.internet import defer\n from twisted.internet import reactor\n\n\n @defer.inlineCallbacks\n def main():\n rc = yield redis.ShardedConnection([\"localhost:6379\", \"localhost:6380\"])\n print rc\n print \"Supported methods on sharded connections:\", rc.ShardedMethods\n\n keys = []\n for x in xrange(100):\n key = \"foo%02d\" % x\n yield rc.set(key, \"bar%02d\" % x)\n keys.append(key)\n\n # yey! mget is supported!\n response = yield rc.mget(keys)\n for val in response:\n print val\n\n yield rc.disconnect()\n\n\n if __name__ == \"__main__\":\n main().addCallback(lambda ign: reactor.stop())\n reactor.run()\n\n\n### Transactions ###\n\nFor obvious reasons, transactions are NOT supported on sharded connections.\nBut they work pretty good on normal or lazy connections, and connection pools.\n\nNOTE: redis uses the following methods for transactions:\n\n- WATCH: synchronization\n- MULTI: start the transaction\n- EXEC: commit the transaction\n- DISCARD: you got it.\n\nBecause ``exec`` is a reserved word in Python, the command to commit is\n``commit``.\n\nExample:\n\n #!/usr/bin/env python\n # coding: utf-8\n\n import txredisapi as redis\n\n from twisted.internet import defer\n from twisted.internet import reactor\n\n\n @defer.inlineCallbacks\n def main():\n rc = yield redis.ConnectionPool()\n\n # Remove the keys\n yield rc.delete([\"a1\", \"a2\", \"a3\"])\n\n # Start transaction\n t = yield rc.multi()\n\n # These will return \"QUEUED\" - even t.get(key)\n yield t.set(\"a1\", \"1\")\n yield t.set(\"a2\", \"2\")\n yield t.set(\"a3\", \"3\")\n yield t.get(\"a1\")\n\n # Try to call get() while in a transaction.\n # It will fail if it's not a connection pool, or if all connections\n # in the pool are in a transaction.\n # Note that it's rc.get(), not the transaction object t.get().\n try:\n v = yield rc.get(\"foo\")\n print \"foo=\", v\n except Exception, e:\n print \"can't get foo:\", e\n\n # Commit, and get all responses from transaction.\n r = yield t.commit()\n print \"commit=\", repr(r)\n\n yield rc.disconnect()\n\n\n if __name__ == \"__main__\":\n main().addCallback(lambda ign: reactor.stop())\n reactor.run()\n\nA \"COUNTER\" example, using WATCH/MULTI:\n\n #!/usr/bin/env python\n # coding: utf-8\n\n import txredisapi as redis\n\n from twisted.internet import defer\n from twisted.internet import reactor\n\n\n @defer.inlineCallbacks\n def main():\n rc = yield redis.ConnectionPool()\n\n # Reset keys\n yield rc.set(\"a1\", 0)\n\n # Synchronize and start transaction\n t = yield rc.watch(\"a1\")\n\n # Load previous value\n a1 = yield t.get(\"a1\")\n\n # start the transactional pipeline\n yield t.multi()\n\n # modify and retrieve the new a1 value\n yield t.set(\"a1\", a1 + 1)\n yield t.get(\"a1\")\n\n print \"simulating concurrency, this will abort the transaction\"\n yield rc.set(\"a1\", 2)\n\n try:\n r = yield t.commit()\n print \"commit=\", repr(r)\n except redis.WatchError, e:\n a1 = yield rc.get(\"a1\")\n print \"transaction has failed.\"\n print \"current a1 value: \", a1\n\n yield rc.disconnect()\n\n\n if __name__ == \"__main__\":\n main().addCallback(lambda ign: reactor.stop())\n reactor.run()\n\n\nCalling ``commit`` will cause it to return a list with the return of all\ncommands executed in the transaction. ``discard``, on the other hand, will\nnormally return just an ``OK``.\n\n### Pipelining ###\n\ntxredisapi automatically [pipelines](http://redis.io/topics/pipelining) all commands\nby sending next commands without waiting for the previous one to receive reply from\nserver. This works even on single connections and increases performance by reducing\nnumber of round-trip delays and. There are two exceptions, though:\n - no commands will be sent after blocking `blpop`, `brpop` or `brpoplpush` until\n response is received;\n - transaction by `multi`/`commit` are also blocking connection making all other\n commands to wait until transaction is executed.\n\nWhen you need to load tons of data to Redis it might be more effective to sent\ncommands in batches grouping them together offline to save on TCP packets and network\nstack overhead. You can do this using `pipeline` method to explicitly accumulate\ncommands and send them to server in a single batch. Be careful to not accumulate too\nmany commands: unreasonable batch size may eat up unexpected amount of memory on both\nclient and server side. Group commands in batches of, for example, 10k commands instead\nof sending all your data at once. The speed will be nearly the same, but the additional\nmemory used will be at max the amount needed to queue this 10k commands\n\nTo send commands in a batch:\n\n #!/usr/bin/env python\n # coding: utf-8\n\n import txredisapi as redis\n\n from twisted.internet import defer\n from twisted.internet import reactor\n\n @defer.inlineCallbacks\n def main():\n rc = yield redis.ConnectionPool()\n\n # Start grouping commands\n pipeline = yield rc.pipeline()\n\n pipeline.set(\"foo\", 123)\n pipeline.set(\"bar\", 987)\n pipeline.get(\"foo\")\n pipeline.get(\"bar\")\n\n # Write those 2 sets and 2 gets to redis all at once, and wait\n # for all replies before continuing.\n results = yield pipeline.execute_pipeline()\n\n print \"foo:\", results[2] # should be 123\n print \"bar:\", results[3] # should be 987\n\n yield rc.disconnect()\n\n if __name__ == \"__main__\":\n main().addCallback(lambda ign: reactor.stop())\n reactor.run()\n\n### Authentication ###\n\nThis is how to authenticate::\n\n #!/usr/bin/env python\n\n import txredisapi\n from twisted.internet import defer\n from twisted.internet import reactor\n\n\n @defer.inlineCallbacks\n def main():\n redis = yield txredisapi.Connection(password=\"foobared\")\n yield redis.set(\"foo\", \"bar\")\n print (yield redis.get(\"foo\"))\n reactor.stop()\n\n\n if __name__ == \"__main__\":\n main()\n reactor.run()\n \n### Connection using Redis Sentinel ###\n\n`txredisapi` can discover Redis master and slaves addresses using \n[Redis Sentinel](http://redis.io/topics/sentinel) and automatically failover\nin case of server failure.\n\n #!/usr/bin/env python\n \n from twisted.internet.task import react\n import txredisapi\n \n @defer.inlineCallbacks\n def main(reactor):\n sentinel = txredisapi.Sentinel([(\"sentinel-a\", 26379), (\"sentinel-b\", 26379), (\"sentinel-c\", 26379)])\n redis = sentinel.master_for(\"service_name\")\n yield redis.set(\"foo\", \"bar\")\n print (yield redis.get(\"foo\"))\n yield redis.disconnect()\n yield sentinel.disconnect()\n \n react(main)\n \nUsual connection arguments like `dbid=N` or `poolsize=N` can be specified in\n`master_for()` call. Use `sentinel.slave_for()` to connect to one of the slaves \ninstead of master.\n\nAdd `min_other_sentinels=N` to `Sentinel` constructor call to make it obey information\nonly from sentinels that currently connected to specified number of other sentinels\nto minimize a risk of split-brain in case of network partitioning.\n\n\nCredits\n=======\nThanks to (in no particular order):\n\n- Alexandre Fiori\n\n - Author of txredisapi\n\n- Gleicon Moraes\n\n - Bug fixes, testing, and [RestMQ](http://github.com/gleicon/restmq>).\n - For writing the Consistent Hashing algorithm used for sharding.\n\n- Dorian Raymer and Ludovico Magnocavallo\n\n - Authors of the original *redis protocol for twisted*.\n\n- Vanderson Mota\n\n - Initial pypi setup, and patches.\n\n- Jeethu Rao\n\n - Contributed with test cases, and other ideas like support for travis-ci\n\n- Jeremy Archer\n\n - Minor bugfixes.\n\n- Christoph Tavan (@ctavan)\n\n - Idea and test case for nested multi bulk replies, minor command enhancements.\n\n- dgvncsz0f\n\n - WATCH/UNWATCH commands\n\n- Ilia Glazkov\n\n - Free connection selection algorithm for pools.\n - Non-unicode charset fixes.\n - SCAN commands\n\n- Matt Pizzimenti (mjpizz)\n\n - pipelining support\n\n- Nickolai Novik (jettify)\n\n - update of SET command\n\n- Evgeny Tataurov (etataurov)\n\n - Ability to use hiredis protocol parser\n\n- Ilya Skriblovsky (IlyaSkriblovsky)\n\n - Sentinel support\n",
"bugtrack_url": null,
"license": "http://www.apache.org/licenses/LICENSE-2.0",
"summary": "non-blocking redis client for python",
"version": "1.4.10",
"project_urls": {
"Homepage": "http://github.com/IlyaSkriblovsky/txredisapi"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "b8a17a737e5bb288099c67e5c02e7928700882fd81882d7c223e1f714401f3bb",
"md5": "8d3c61ab10bf95091cdfbaecadfe2d06",
"sha256": "0a6ea77f27f8cf092f907654f08302a97b48fa35f24e0ad99dfb74115f018161"
},
"downloads": -1,
"filename": "txredisapi-1.4.10-py3-none-any.whl",
"has_sig": false,
"md5_digest": "8d3c61ab10bf95091cdfbaecadfe2d06",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 31003,
"upload_time": "2023-07-07T05:34:35",
"upload_time_iso_8601": "2023-07-07T05:34:35.362784Z",
"url": "https://files.pythonhosted.org/packages/b8/a1/7a737e5bb288099c67e5c02e7928700882fd81882d7c223e1f714401f3bb/txredisapi-1.4.10-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "809a3ecb0d5fb8fc37c977673c3f2b43ff799ed46517549da768edb4fb5c750e",
"md5": "878c127a289a8f62a2b4aff914780eca",
"sha256": "7609a6af6ff4619a3189c0adfb86aeda789afba69eb59fc1e19ac0199e725395"
},
"downloads": -1,
"filename": "txredisapi-1.4.10.tar.gz",
"has_sig": false,
"md5_digest": "878c127a289a8f62a2b4aff914780eca",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 31217,
"upload_time": "2023-07-07T05:34:37",
"upload_time_iso_8601": "2023-07-07T05:34:37.307547Z",
"url": "https://files.pythonhosted.org/packages/80/9a/3ecb0d5fb8fc37c977673c3f2b43ff799ed46517549da768edb4fb5c750e/txredisapi-1.4.10.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-07-07 05:34:37",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "IlyaSkriblovsky",
"github_project": "txredisapi",
"travis_ci": true,
"coveralls": false,
"github_actions": false,
"lcname": "txredisapi"
}
JSON
