Note: as of redis-helper v0.4.0, version 3.0 of redis-py is in use,
which has backwards incompatible changes withe redis-py 2.x. See
https://github.com/redis/redis-py/tree/70ef9ec68f9163c86d4cace2941e2f0ae4ce8525#upgrading-from-redis-py-2x-to-30
About
-----
Install redis-helper, create an instance of ``redis_helper.Collection``
(**the args/kwargs define the model**) and use the ``add``, ``get``,
``update``, ``delete``, and ``find`` methods to:
- quickly store/retrieve/modify Python dicts in Redis
- filter through indexed fields with simple/flexible find arguments
- power real-time dashboards with metrics at a variety of time ranges
- super-charge event logging and system debugging
- build FAST prototypes and simulators
- greatly simplify data access patterns throughout application
See the `request logging demo <https://asciinema.org/a/101422?t=1:10>`__
and `urls
demo <https://asciinema.org/a/75kl95ty9vg2jl93pfz9fbs9q?t=1:00>`__ (with
``unique_field`` defined). The
`examples <https://github.com/kenjyco/redis-helper/tree/master/examples>`__
they reference are **short** and **easy to read**.
The `redis-helper project <https://github.com/kenjyco/redis-helper>`__
evolved from a `reference Python
project <https://github.com/kenjyco/beu/tree/4aea6146fc5f01df3e344b9fadddf28b795dac89>`__
that would be **easy to teach** and follow many practical best practices
and useful patterns. Main purpose was to have something that was super
**easy to configure** (a single ``~/.config/redis-helper/settings.ini``
file for multiple application environments) that did cool things with
`Redis <http://redis.io/topics/data-types-intro>`__.
The `redis-helper package <https://pypi.python.org/pypi/redis-helper>`__
provides a ``Collection`` class that was designed to be **easy to
interact with** in the shell (for exploration, experimentation, and
debugging). Most methods on a ``Collection`` help **minimize typing**
(passing multiple arguments in a single delimited string when
appropriate) and do “the most reasonable thing” whenever possible.
The first time that ``redis_helper`` is imported, the sample
`settings.ini <https://github.com/kenjyco/redis-helper/blob/master/redis_helper/settings.ini>`__
file will be copied to the ``~/.config/redis-helper`` directory.
::
[default]
image_version = 6-alpine
[dev]
container_name = redis-helper
port = 6379
rm = False
redis_url = redis://localhost:6379/1
[test]
container_name = redis-helper-test
port = 6380
rm = True
redis_url = redis://localhost:6380/9
If docker is installed to your system and your user has permission to
use it, the `bg-helper docker
tools <https://github.com/kenjyco/bg-helper#helper-functions-in-bg_helpertools-that-use-docker-if-it-is-installed>`__
will be used to start a redis container for development or running
tests, if Redis is not already installed locally.
(Optionally) install Redis and start server locally
---------------------------------------------------
::
% sudo apt-get install -y redis-server
or
% brew install redis
% brew services start redis
Install redis-helper
--------------------
- install latest tag/release of `redis-helper
package <https://pypi.python.org/pypi/redis-helper>`__
::
% pip3 install redis-helper
- or, install latest commit on master of `redis-helper
project <https://github.com/kenjyco/redis-helper>`__
::
% pip3 install git+git://github.com/kenjyco/redis-helper
Intro
-----
`Redis <http://redis.io/topics/data-types-intro>`__ is a fast in-memory
**data structure server**, where each stored object is referenced by a
key name. Objects in Redis correspond to one of several basic types,
each having their own set of specialized commands to perform operations.
The `redis Python package <https://github.com/andymccurdy/redis-py>`__
provides the
`StrictRedis <https://redis-py.readthedocs.org/en/latest/#redis.StrictRedis>`__
class, which contains methods that correspond to all of the Redis server
commands.
When initializing Collection objects, you must specify the “namespace”
and “name” of the collection (which are used to create the internally
used ``_base_key`` property). All Redis keys associated with a
Collection will have a name pattern that starts with the ``_base_key``.
.. code:: python
import redis_helper as rh
request_logs = rh.Collection(
'log',
'request',
index_fields='status, uri, host',
json_fields='request, response, headers'
)
urls = rh.Collection(
'web',
'url',
unique_field='name',
index_fields='domain, _type'
)
notes = rh.Collection(
'input',
'note',
index_fields='topic, tag',
insert_ts=True
)
sample = rh.Collection(
'ns',
'sample',
unique_field='name',
index_fields='status',
json_fields='data',
rx_name='\S{4,6}',
rx_status='(active|inactive|cancelled)',
rx_aws='[a-z]+\-[0-9a-f]+',
insert_ts=True
)
uses_sample = rh.Collection(
'ns',
'uses_sample',
index_fields='z',
rx_thing='\S{4,6}',
reference_fields='thing--ns:sample'
)
- a ``unique_field`` can be specified on a collection if items in the
collection should not contain duplicate values for that particular
field
- the ``unique_field`` cannot also be included in ``json_fields`` or
``pickle_fields``
- if you specify a ``unique_field``, that field must exist on each
item you add to the collection
- use ``index_fields`` to specify which fields you will want to filter
on when using the ``find`` method
- the values for data fields being indexed MUST be simple strings or
numbers
- the values for data fields being indexed SHOULD NOT be long
strings, as the values themselves are part of the index keys
- use ``json_fields`` to specify which fields should be JSON encoded
before insertion to Redis (using the very fast
`ujson <https://pypi.python.org/pypi/ujson>`__ library)
- use ``rx_{field}`` to specify a regular expression for any field with
strict rules for validation
- use ``reference_fields`` to specify fields that reference the
``unique_field`` of another collection
- uses field–basekey combos
- use ``pickle_fields`` to specify which fields should be pickled
before insertion to Redis
- set ``insert_ts=True`` to create an additional index to store insert
times
- only do this if you are storing items that you are likely to
update and also likely to want to know the original insert time
- each time an object is updated, the score associated with the
``hash_id`` (at the ``_ts_zset_key``) is updated to the current
timestamp
- the score associated with the ``hash_id`` (at the
``_in_zset_key``) is never updated
Essentially, you can store a Python
`dict <https://docs.python.org/3/tutorial/datastructures.html#dictionaries>`__
in a Redis `hash <https://redis.io/topics/data-types#hashes>`__ and
index some of the fields in Redis
`sets <https://redis.io/topics/data-types#sets>`__. The collection’s
``_ts_zset_key`` is the Redis key name for the `sorted
set <https://redis.io/topics/data-types#sorted-sets>`__ containing the
``hash_id`` of every hash in the collection (with the ``score`` being a
``utc_float`` corresponding to the UTC time the ``hash_id`` was added or
modified).
- if ``insert_ts=True`` was passed in when initializing the
``Collection`` (or sub-class), then the collection will also define
``self.in_zset_key`` to be the Redis key name for the sorted set (for
``hash_id`` and ``utc_float`` of insert time)
.. code:: python
request_logs.add(
method='get',
status=400,
host='blah.net',
uri='/info',
request={'x': 50, 'y': 100},
response={'error': 'bad request'},
)
urls.add(
name='redis-helper github',
url='https://github.com/kenjyco/redis-helper',
domain='github.com',
_type='repo',
)
The ``get`` method is a wrapper to `hash
commands <http://redis.io/commands#hash>`__ ``hget``, ``hmget``, or
``hgetall``. The actual hash command that gets called is determined by
the number of fields requested.
- a Python dict is typically returned from ``get``
- if ``item_format`` is specified, a string will be returned matching
that format instead
.. code:: python
request_logs.get('log:request:1')
request_logs.get('log:request:1', 'host,status')
request_logs.get('log:request:1', item_format='{status} for {host}{uri}')
request_logs.get_by_position(0, item_format='{status} for {host}{uri}')
urls.get_by_position(-1, 'domain,url')
urls.get_by_unique_value('redis-helper github', item_format='{url} points to a {_type}')
- the ``get_by_position`` and ``get_by_unique_value`` methods are
wrappers to ``get``
- the ``get_by_unique_value`` method is only useful if a
``unique_field`` was set on the Collection
The ``find`` method allows you to return data for items in the
collection that match some set of search criteria. Multiple search terms
(i.e. ``index_field:value`` pairs) maybe be passed in the ``terms``
parameter, as long as they are separated by one of ``,`` ``;`` ``|``.
Any fields specified in the ``get_fields`` parameter are passed along to
the ``get`` method (when the actual fetching takes place).
- when using ``terms``, all terms that include the same field will be
treatead like an “or” (union of related sets), then the intersection
of different sets will be computed
- see the Redis `set commands <https://redis.io/commands#set>`__ and
`sorted set commands <https://redis.io/commands#sorted_set>`__
There are many options for specifying time ranges in the ``find`` method
including:
- ``since`` and ``until`` when specifying ``num:unit`` strings
(i.e. 15:seconds, 1.5:weeks, etc)
- ``start_ts`` and ``end_ts`` when specifying timestamps with a form
between ``YYYY`` and ``YYYY-MM-DD HH:MM:SS.f``
- ``start`` and ``end`` when specifying a ``utc_float``
- for ``since``, ``until``, ``start_ts``, and ``end_ts``, multiple
values may be passed in the string, as long as they are separated by
one of ``,`` ``;`` ``|``.
- when multiple time ranges are specified, the ``find`` method will
determine all reasonable combinations and return a result-set per
combination (instead of returning a list of items, returns a dict
of list of items)
If ``count=True`` is specified, the number of results matching the
search criteria are returned instead of the actual results
- if there are multiple time ranges specified, counts will be returned
for each combination
.. code:: python
request_logs.find('status:400, host:blah.net', get_fields='uri,error')
request_logs.find(since='1:hr, 30:min', until='15:min, 5:min')
request_logs.find(count=True, since='1:hr, 30:min', until='15:min, 5:min')
urls.find(count=True, since='1:hr, 30:min, 10:min, 5:min, 1:min')
urls.find(start_ts='2017-02-03', end_ts='2017-02-03 7:15:00')
urls.find(start_ts='2017-02-03', item_format='{_ts} -> {_id}')
The ``update`` method allows you to change values for some fields
(modifying the ``unique_field``, when it is specified, is not allowed).
- every time a field is modified for a particular ``hash_id``, the
previous value and score (timestamp) are stored in a Redis hash
- the ``old_data_for_hash_id`` or ``old_data_for_unique_value`` methods
can be used to retrieve the history of all changes for a ``hash_id``
.. code:: python
urls.update('web:url:1', _type='fancy', notes='this is a fancy url')
urls.old_data_for_hash_id('web:url:1')
urls.old_data_for_unique_value('redis-helper github')
The ``load_ref_data`` option on ``get``, ``get_by_unique_value``, or
``find`` methods allow you to load the referenced data object from the
other collection (where ``reference_fields`` are specified)
.. code:: python
In [1]: sample.add(name='hello', aws='ami-0ad5743816d822b81', status='active')
Out[1]: 'ns:sample:1'
In [2]: uses_sample.add(thing='hello', z=500, y=True)
Out[2]: 'ns:uses_sample:1'
In [3]: uses_sample.get('ns:uses_sample:1')
Out[3]: {'thing': 'hello', 'z': 500, 'y': True}
In [4]: uses_sample.get('ns:uses_sample:1', load_ref_data=True)
Out[4]:
{'thing': {'name': 'hello',
'aws': 'ami-0ad5743816d822b81',
'status': 'active',
'_id': 'ns:sample:1',
'_ts': 20201028210044.875},
'z': 500,
'y': True}
In [5]: uses_sample.add(thing='byebye', z=100, y=True)
Out[5]: 'ns:uses_sample:2'
In [6]: uses_sample.get('ns:uses_sample:2', load_ref_data=True)
Out[6]: {'thing': 'byebye', 'z': 100, 'y': True}
Tip
---
There may be times where you want to use redis-helper (if it’s already
installed), but don’t want to make it an explicit requirement of your
project. In cases like this you can do the following:
::
try:
import redis_helper as rh
from redis import ConnectionError as RedisConnectionError
except (ImportError, ModuleNotFoundError):
SomeCollection = None
else:
try:
SomeCollection = rh.Collection(
...
)
except RedisConnectionError:
SomeCollection = None
Then in whatever function, you can just do:
::
def some_func():
if SomeCollection is None:
return
# Do stuff with SomeCollection
Local development setup
-----------------------
::
% git clone https://github.com/kenjyco/redis-helper
% cd redis-helper
% ./dev-setup.bash
The
`dev-setup.bash <https://github.com/kenjyco/redis-helper/blob/master/dev-setup.bash>`__
script will create a virtual environment in the ``./venv`` directory
with extra dependencies (ipython, pdbpp, pytest), then copy
``settings.ini`` to the ``~/.config/redis-helper`` directory.
Running tests in development setup
----------------------------------
The
`setup.cfg <https://github.com/kenjyco/redis-helper/blob/master/setup.cfg>`__
file contains the options for ``py.test``, currently ``-vsx -rs --pdb``.
The ``-vsx -rs --pdb`` options will run tests in a verbose manner and
output the reason why tests were skipped (if any were skipped). If there
are any failing tests, ``py.test`` will stop on the first failure and
drop you into a `pdb++ <https://pypi.python.org/pypi/pdbpp/>`__ debugger
session.
See the `debugging
section <https://github.com/kenjyco/redis-helper#settings-environments-testing-and-debugging>`__
of the README for tips on using the debugger and setting breakpoints (in
the actual `project
code <https://github.com/kenjyco/redis-helper/tree/master/redis_helper>`__,
or in the `test
code <https://github.com/kenjyco/redis-helper/tree/master/tests>`__).
::
% venv/bin/py.test
or
::
% venv/bin/python3 setup.py test
..
Note: This option requires ``setuptools`` to be installed.
Usage
-----
The ``rh-download-examples``, ``rh-download-scripts``, ``rh-notes``, and
``rh-shell`` scripts are provided.
::
$ venv/bin/rh-download-examples --help
Usage: rh-download-examples [OPTIONS] [DIRECTORY]
Download redis-helper example files from github
Options:
--help Show this message and exit.
$ venv/bin/rh-download-scripts --help
Usage: rh-download-scripts [OPTIONS] [DIRECTORY]
Download redis-helper script files from github
Options:
--help Show this message and exit.
$ venv/bin/rh-notes --help
Usage: rh-notes [OPTIONS] [TOPIC]
Prompt user to enter notes (about a topic) until finished; or review notes
Options:
-c, --ch TEXT string appended to the topic (default "> ")
-s, --shell Start an ipython shell to inspect the notes collection
--help Show this message and exit.
$ venv/bin/rh-shell --help
Usage: rh-shell [OPTIONS]
Interactively select a Collection model and start ipython shell
Options:
--help Show this message and exit.
.. code:: python
>>> import redis_helper as rh
>>> collection = rh.Collection(..., index_fields='field1, field3')
>>> hash_id = collection.add(field1='', field2='', field3='', ...)
>>> collection.add(...)
>>> collection.add(...)
>>> collection.update(hash_id, field1='', field4='', ...)
>>> change_history = collection.old_data_for_hash_id(hash_id)
>>> data = collection.get(hash_id)
>>> some_data = collection.get(hash_id, 'field1, field3')
>>> results = collection.find(...)
>>> results2 = collection.find('field1:val, field3:val', ...)
>>> results3 = collection.find(..., get_fields='field2, field4')
>>> counts = collection.find(count=True, ...)
>>> top_indexed = collection.index_field_info()
>>> collection.delete(hash_id, ...)
Basics - Part 1 (request logging demo)
--------------------------------------
`Demo <https://asciinema.org/a/101422?t=1:10>`__ bookmarks:
- `1:10 <https://asciinema.org/a/101422?t=1:10>`__ is when the
``ipython`` session is started with
``venv/bin/ipython -i request_logs.py``
- `3:14 <https://asciinema.org/a/101422?t=3:14>`__ is when a second
``ipython`` session is started (in a separate tmux pane) to simulate
a steady stream of requests with
``slow_trickle_requests(randomsleep=True, show=True)``
- `4:22 <https://asciinema.org/a/101422?t=4:22>`__ is when the
``index_field_info`` method is used to get the latest counts of top
indexed items
- `6:11 <https://asciinema.org/a/101422?t=6:11>`__ is when
``slow_trickle_requests(.001)`` is run to simulate a large quick
burst in traffic
- `7:00 <https://asciinema.org/a/101422?t=7:00>`__ is when multiple
values are passed in the ``since`` argument of ``find``\ …
``request_logs.find(count=True, since='5:min, 1:min, 30:sec')``
- `8:37 <https://asciinema.org/a/101422?t=8:37>`__ is when ``get`` and
``get_by_position`` methods are used with a variety of arguments to
change the structure of what’s returned
- `10:33 <https://asciinema.org/a/101422?t=10:33>`__ is when the
``redis_helper.ADMIN_TIMEZONE`` is changed at run time from
``America/Chicago`` to ``Europe/London``
- `11:27 <https://asciinema.org/a/101422?t=11:27>`__ is when ``find``
is used with a variety of arguments to change the structure of what’s
returned
- `14:30 <https://asciinema.org/a/101422?t=14:30>`__ is when ``find``
is used with multiple search terms and multiple ``since`` values…
``request_logs.find('host:dogs.com, uri:/breeds', count=True, since='5:min, 1:min, 10:sec')``
- `15:54 <https://asciinema.org/a/101422?t=15:54>`__ is when the
``update`` method is used to modify data and change history is
retrieved with the ``old_data_for_hash_id`` method
The first demo walks through the following:
- creating a virtual environment, installing redis-helper, and
downloading example files
::
$ python3 -m venv venv
$ venv/bin/pip3 install redis-helper ipython
$ venv/bin/rh-download-examples
$ cat ~/.config/redis-helper/settings.ini
$ venv/bin/ipython -i request_logs.py
- using the sample ``Collection`` defined in
`request_logs.py <https://github.com/kenjyco/redis-helper/blob/master/examples/request_logs.py>`__
to
- show values of some properties on a ``Collection``
- ``redis_helper.Collection._base_key``
- ``redis_helper.Collection.now_pretty``
- ``redis_helper.Collection.now_utc_float``
- ``redis_helper.Collection.keyspace``
- ``redis_helper.Collection.size``
- ``redis_helper.Collection.first``
- ``redis_helper.Collection.last``
- show values of some settings from ``redis_helper``
- ``redis_helper.APP_ENV``
- ``redis_helper.REDIS_URL``
- ``redis_helper.REDIS``
- ``redis_helper.SETTINGS_FILE``
- ``redis_helper.ADMIN_TIMEZONE``
- show output from some methods on a ``Collection``
- ``redis_helper.Collection.index_field_info()``
- ``redis_helper.Collection.find()``
- ``redis_helper.Collection.find(count=True)``
- ``redis_helper.Collection.find(count=True, since='30:sec')``
- ``redis_helper.Collection.find(since='30:sec')``
- ``redis_helper.Collection.find(since='30:sec', admin_fmt=True)``
- ``redis_helper.Collection.find(count=True, since='5:min, 1:min, 30:sec')``
- ``redis_helper.Collection.find('index_field:value')``
- ``redis_helper.Collection.find('index_field:value', all_fields=True, limit=2)``
- ``redis_helper.Collection.find('index_field:value', all_fields=True, limit=2, admin_fmt=True, item_format='{_ts} -> {_id}')``
- ``redis_helper.Collection.find('index_field:value', get_fields='field1, field2', include_meta=False)``
- ``redis_helper.Collection.find('index_field1:value1, index_field2:value2', count=True)``
- ``redis_helper.Collection.find('index_field1:value1, index_field2:value2', count=True, since='5:min, 1:min, 10:sec')``
- ``redis_helper.Collection.get(hash_id)``
- ``redis_helper.Collection.get(hash_id, 'field1,field2,field3')``
- ``redis_helper.Collection.get(hash_id, include_meta=True)``
- ``redis_helper.Collection.get(hash_id, include_meta=True, fields='field1, field2')``
- ``redis_helper.Collection.get(hash_id, include_meta=True, item_format='{_ts} -> {_id}')``
- ``redis_helper.Collection.get_by_position(0)``
- ``redis_helper.Collection.get_by_position(0, include_meta=True, admin_fmt=True)``
- ``redis_helper.Collection.update(hash_id, field1='value1', field2='value2')``
- ``redis_helper.Collection.old_data_for_hash_id(hash_id)``
Basics - Part 2 (urls demo, with unique field)
----------------------------------------------
`Demo <https://asciinema.org/a/75kl95ty9vg2jl93pfz9fbs9q?t=1:00>`__
bookmarks:
- ``TODO``
The second demo walks through the following:
- using the sample ``Collection`` defined in
`urls.py <https://github.com/kenjyco/redis-helper/blob/master/examples/urls.py>`__
to
- ``TODO``
Settings, environments, testing, and debugging
----------------------------------------------
To trigger a debugger session at a specific place in the `project
code <https://github.com/kenjyco/redis-helper/tree/master/redis_helper>`__,
insert the following, one line above where you want to inspect
::
import pdb; pdb.set_trace()
To start the debugger inside `test
code <https://github.com/kenjyco/redis-helper/tree/master/tests>`__, use
::
pytest.set_trace()
- use ``(l)ist`` to list context lines
- use ``(n)ext`` to move on to the next statement
- use ``(s)tep`` to step into a function
- use ``(c)ontinue`` to continue to next break point
(i.e. ``set_trace()`` lines in your code)
- use ``sticky`` to toggle sticky mode (to constantly show the
currently executing code as you move through with the debugger)
- use ``pp`` to pretty print a variable or statement
If the redis server at ``redis_url`` (in the **test section** of
``~/.config/redis-server/settings.ini``) is not running or is not empty,
redis server tests will be skipped.
Use the ``APP_ENV`` environment variable to specify which section of the
``settings.ini`` file your settings will be loaded from. Any settings in
the ``default`` section can be overwritten if explicity set in another
section.
- if no ``APP_ENV`` is explicitly set, ``dev`` is assumed
- the ``APP_ENV`` setting is overwritten to be ``test`` no matter what
was set when calling ``py.test`` tests
Raw data
{
"_id": null,
"home_page": "https://github.com/kenjyco/redis-helper",
"name": "redis-helper",
"maintainer": "",
"docs_url": null,
"requires_python": "",
"maintainer_email": "",
"keywords": "redis,cli,command-line,dictionary,data,database,secondary index,model,prototype,event logging,dashboard,easy modeling,helper,kenjyco",
"author": "Ken",
"author_email": "kenjyco@gmail.com",
"download_url": "https://github.com/kenjyco/redis-helper/tarball/v0.4.8",
"platform": null,
"description": " Note: as of redis-helper v0.4.0, version 3.0 of redis-py is in use,\n which has backwards incompatible changes withe redis-py 2.x. See\n https://github.com/redis/redis-py/tree/70ef9ec68f9163c86d4cace2941e2f0ae4ce8525#upgrading-from-redis-py-2x-to-30\n\nAbout\n-----\n\nInstall redis-helper, create an instance of ``redis_helper.Collection``\n(**the args/kwargs define the model**) and use the ``add``, ``get``,\n``update``, ``delete``, and ``find`` methods to:\n\n- quickly store/retrieve/modify Python dicts in Redis\n- filter through indexed fields with simple/flexible find arguments\n- power real-time dashboards with metrics at a variety of time ranges\n- super-charge event logging and system debugging\n- build FAST prototypes and simulators\n- greatly simplify data access patterns throughout application\n\nSee the `request logging demo <https://asciinema.org/a/101422?t=1:10>`__\nand `urls\ndemo <https://asciinema.org/a/75kl95ty9vg2jl93pfz9fbs9q?t=1:00>`__ (with\n``unique_field`` defined). The\n`examples <https://github.com/kenjyco/redis-helper/tree/master/examples>`__\nthey reference are **short** and **easy to read**.\n\nThe `redis-helper project <https://github.com/kenjyco/redis-helper>`__\nevolved from a `reference Python\nproject <https://github.com/kenjyco/beu/tree/4aea6146fc5f01df3e344b9fadddf28b795dac89>`__\nthat would be **easy to teach** and follow many practical best practices\nand useful patterns. Main purpose was to have something that was super\n**easy to configure** (a single ``~/.config/redis-helper/settings.ini``\nfile for multiple application environments) that did cool things with\n`Redis <http://redis.io/topics/data-types-intro>`__.\n\nThe `redis-helper package <https://pypi.python.org/pypi/redis-helper>`__\nprovides a ``Collection`` class that was designed to be **easy to\ninteract with** in the shell (for exploration, experimentation, and\ndebugging). Most methods on a ``Collection`` help **minimize typing**\n(passing multiple arguments in a single delimited string when\nappropriate) and do \u201cthe most reasonable thing\u201d whenever possible.\n\nThe first time that ``redis_helper`` is imported, the sample\n`settings.ini <https://github.com/kenjyco/redis-helper/blob/master/redis_helper/settings.ini>`__\nfile will be copied to the ``~/.config/redis-helper`` directory.\n\n::\n\n [default]\n image_version = 6-alpine\n\n [dev]\n container_name = redis-helper\n port = 6379\n rm = False\n redis_url = redis://localhost:6379/1\n\n [test]\n container_name = redis-helper-test\n port = 6380\n rm = True\n redis_url = redis://localhost:6380/9\n\nIf docker is installed to your system and your user has permission to\nuse it, the `bg-helper docker\ntools <https://github.com/kenjyco/bg-helper#helper-functions-in-bg_helpertools-that-use-docker-if-it-is-installed>`__\nwill be used to start a redis container for development or running\ntests, if Redis is not already installed locally.\n\n(Optionally) install Redis and start server locally\n---------------------------------------------------\n\n::\n\n % sudo apt-get install -y redis-server\n\n or\n\n % brew install redis\n % brew services start redis\n\nInstall redis-helper\n--------------------\n\n- install latest tag/release of `redis-helper\n package <https://pypi.python.org/pypi/redis-helper>`__\n\n ::\n\n % pip3 install redis-helper\n\n- or, install latest commit on master of `redis-helper\n project <https://github.com/kenjyco/redis-helper>`__\n\n ::\n\n % pip3 install git+git://github.com/kenjyco/redis-helper\n\nIntro\n-----\n\n`Redis <http://redis.io/topics/data-types-intro>`__ is a fast in-memory\n**data structure server**, where each stored object is referenced by a\nkey name. Objects in Redis correspond to one of several basic types,\neach having their own set of specialized commands to perform operations.\nThe `redis Python package <https://github.com/andymccurdy/redis-py>`__\nprovides the\n`StrictRedis <https://redis-py.readthedocs.org/en/latest/#redis.StrictRedis>`__\nclass, which contains methods that correspond to all of the Redis server\ncommands.\n\nWhen initializing Collection objects, you must specify the \u201cnamespace\u201d\nand \u201cname\u201d of the collection (which are used to create the internally\nused ``_base_key`` property). All Redis keys associated with a\nCollection will have a name pattern that starts with the ``_base_key``.\n\n.. code:: python\n\n import redis_helper as rh\n\n\n request_logs = rh.Collection(\n 'log',\n 'request',\n index_fields='status, uri, host',\n json_fields='request, response, headers'\n )\n\n urls = rh.Collection(\n 'web',\n 'url',\n unique_field='name',\n index_fields='domain, _type'\n )\n\n notes = rh.Collection(\n 'input',\n 'note',\n index_fields='topic, tag',\n insert_ts=True\n )\n\n sample = rh.Collection(\n 'ns',\n 'sample',\n unique_field='name',\n index_fields='status',\n json_fields='data',\n rx_name='\\S{4,6}',\n rx_status='(active|inactive|cancelled)',\n rx_aws='[a-z]+\\-[0-9a-f]+',\n insert_ts=True\n )\n\n uses_sample = rh.Collection(\n 'ns',\n 'uses_sample',\n index_fields='z',\n rx_thing='\\S{4,6}',\n reference_fields='thing--ns:sample'\n )\n\n- a ``unique_field`` can be specified on a collection if items in the\n collection should not contain duplicate values for that particular\n field\n\n - the ``unique_field`` cannot also be included in ``json_fields`` or\n ``pickle_fields``\n - if you specify a ``unique_field``, that field must exist on each\n item you add to the collection\n\n- use ``index_fields`` to specify which fields you will want to filter\n on when using the ``find`` method\n\n - the values for data fields being indexed MUST be simple strings or\n numbers\n - the values for data fields being indexed SHOULD NOT be long\n strings, as the values themselves are part of the index keys\n\n- use ``json_fields`` to specify which fields should be JSON encoded\n before insertion to Redis (using the very fast\n `ujson <https://pypi.python.org/pypi/ujson>`__ library)\n- use ``rx_{field}`` to specify a regular expression for any field with\n strict rules for validation\n- use ``reference_fields`` to specify fields that reference the\n ``unique_field`` of another collection\n\n - uses field\u2013basekey combos\n\n- use ``pickle_fields`` to specify which fields should be pickled\n before insertion to Redis\n- set ``insert_ts=True`` to create an additional index to store insert\n times\n\n - only do this if you are storing items that you are likely to\n update and also likely to want to know the original insert time\n\n - each time an object is updated, the score associated with the\n ``hash_id`` (at the ``_ts_zset_key``) is updated to the current\n timestamp\n - the score associated with the ``hash_id`` (at the\n ``_in_zset_key``) is never updated\n\nEssentially, you can store a Python\n`dict <https://docs.python.org/3/tutorial/datastructures.html#dictionaries>`__\nin a Redis `hash <https://redis.io/topics/data-types#hashes>`__ and\nindex some of the fields in Redis\n`sets <https://redis.io/topics/data-types#sets>`__. The collection\u2019s\n``_ts_zset_key`` is the Redis key name for the `sorted\nset <https://redis.io/topics/data-types#sorted-sets>`__ containing the\n``hash_id`` of every hash in the collection (with the ``score`` being a\n``utc_float`` corresponding to the UTC time the ``hash_id`` was added or\nmodified).\n\n- if ``insert_ts=True`` was passed in when initializing the\n ``Collection`` (or sub-class), then the collection will also define\n ``self.in_zset_key`` to be the Redis key name for the sorted set (for\n ``hash_id`` and ``utc_float`` of insert time)\n\n.. code:: python\n\n request_logs.add(\n method='get',\n status=400,\n host='blah.net',\n uri='/info',\n request={'x': 50, 'y': 100},\n response={'error': 'bad request'},\n )\n\n urls.add(\n name='redis-helper github',\n url='https://github.com/kenjyco/redis-helper',\n domain='github.com',\n _type='repo',\n )\n\nThe ``get`` method is a wrapper to `hash\ncommands <http://redis.io/commands#hash>`__ ``hget``, ``hmget``, or\n``hgetall``. The actual hash command that gets called is determined by\nthe number of fields requested.\n\n- a Python dict is typically returned from ``get``\n- if ``item_format`` is specified, a string will be returned matching\n that format instead\n\n.. code:: python\n\n request_logs.get('log:request:1')\n request_logs.get('log:request:1', 'host,status')\n request_logs.get('log:request:1', item_format='{status} for {host}{uri}')\n request_logs.get_by_position(0, item_format='{status} for {host}{uri}')\n urls.get_by_position(-1, 'domain,url')\n urls.get_by_unique_value('redis-helper github', item_format='{url} points to a {_type}')\n\n- the ``get_by_position`` and ``get_by_unique_value`` methods are\n wrappers to ``get``\n\n - the ``get_by_unique_value`` method is only useful if a\n ``unique_field`` was set on the Collection\n\nThe ``find`` method allows you to return data for items in the\ncollection that match some set of search criteria. Multiple search terms\n(i.e. ``index_field:value`` pairs) maybe be passed in the ``terms``\nparameter, as long as they are separated by one of ``,`` ``;`` ``|``.\nAny fields specified in the ``get_fields`` parameter are passed along to\nthe ``get`` method (when the actual fetching takes place).\n\n- when using ``terms``, all terms that include the same field will be\n treatead like an \u201cor\u201d (union of related sets), then the intersection\n of different sets will be computed\n- see the Redis `set commands <https://redis.io/commands#set>`__ and\n `sorted set commands <https://redis.io/commands#sorted_set>`__\n\nThere are many options for specifying time ranges in the ``find`` method\nincluding:\n\n- ``since`` and ``until`` when specifying ``num:unit`` strings\n (i.e.\u00a015:seconds, 1.5:weeks, etc)\n- ``start_ts`` and ``end_ts`` when specifying timestamps with a form\n between ``YYYY`` and ``YYYY-MM-DD HH:MM:SS.f``\n- ``start`` and ``end`` when specifying a ``utc_float``\n- for ``since``, ``until``, ``start_ts``, and ``end_ts``, multiple\n values may be passed in the string, as long as they are separated by\n one of ``,`` ``;`` ``|``.\n\n - when multiple time ranges are specified, the ``find`` method will\n determine all reasonable combinations and return a result-set per\n combination (instead of returning a list of items, returns a dict\n of list of items)\n\nIf ``count=True`` is specified, the number of results matching the\nsearch criteria are returned instead of the actual results\n\n- if there are multiple time ranges specified, counts will be returned\n for each combination\n\n.. code:: python\n\n request_logs.find('status:400, host:blah.net', get_fields='uri,error')\n request_logs.find(since='1:hr, 30:min', until='15:min, 5:min')\n request_logs.find(count=True, since='1:hr, 30:min', until='15:min, 5:min')\n urls.find(count=True, since='1:hr, 30:min, 10:min, 5:min, 1:min')\n urls.find(start_ts='2017-02-03', end_ts='2017-02-03 7:15:00')\n urls.find(start_ts='2017-02-03', item_format='{_ts} -> {_id}')\n\nThe ``update`` method allows you to change values for some fields\n(modifying the ``unique_field``, when it is specified, is not allowed).\n\n- every time a field is modified for a particular ``hash_id``, the\n previous value and score (timestamp) are stored in a Redis hash\n- the ``old_data_for_hash_id`` or ``old_data_for_unique_value`` methods\n can be used to retrieve the history of all changes for a ``hash_id``\n\n.. code:: python\n\n urls.update('web:url:1', _type='fancy', notes='this is a fancy url')\n urls.old_data_for_hash_id('web:url:1')\n urls.old_data_for_unique_value('redis-helper github')\n\nThe ``load_ref_data`` option on ``get``, ``get_by_unique_value``, or\n``find`` methods allow you to load the referenced data object from the\nother collection (where ``reference_fields`` are specified)\n\n.. code:: python\n\n In [1]: sample.add(name='hello', aws='ami-0ad5743816d822b81', status='active')\n Out[1]: 'ns:sample:1'\n\n In [2]: uses_sample.add(thing='hello', z=500, y=True)\n Out[2]: 'ns:uses_sample:1'\n\n In [3]: uses_sample.get('ns:uses_sample:1')\n Out[3]: {'thing': 'hello', 'z': 500, 'y': True}\n\n In [4]: uses_sample.get('ns:uses_sample:1', load_ref_data=True)\n Out[4]:\n {'thing': {'name': 'hello',\n 'aws': 'ami-0ad5743816d822b81',\n 'status': 'active',\n '_id': 'ns:sample:1',\n '_ts': 20201028210044.875},\n 'z': 500,\n 'y': True}\n\n In [5]: uses_sample.add(thing='byebye', z=100, y=True)\n Out[5]: 'ns:uses_sample:2'\n\n In [6]: uses_sample.get('ns:uses_sample:2', load_ref_data=True)\n Out[6]: {'thing': 'byebye', 'z': 100, 'y': True}\n\nTip\n---\n\nThere may be times where you want to use redis-helper (if it\u2019s already\ninstalled), but don\u2019t want to make it an explicit requirement of your\nproject. In cases like this you can do the following:\n\n::\n\n try:\n import redis_helper as rh\n from redis import ConnectionError as RedisConnectionError\n except (ImportError, ModuleNotFoundError):\n SomeCollection = None\n else:\n try:\n SomeCollection = rh.Collection(\n ...\n )\n except RedisConnectionError:\n SomeCollection = None\n\nThen in whatever function, you can just do:\n\n::\n\n def some_func():\n if SomeCollection is None:\n return\n\n # Do stuff with SomeCollection\n\nLocal development setup\n-----------------------\n\n::\n\n % git clone https://github.com/kenjyco/redis-helper\n % cd redis-helper\n % ./dev-setup.bash\n\nThe\n`dev-setup.bash <https://github.com/kenjyco/redis-helper/blob/master/dev-setup.bash>`__\nscript will create a virtual environment in the ``./venv`` directory\nwith extra dependencies (ipython, pdbpp, pytest), then copy\n``settings.ini`` to the ``~/.config/redis-helper`` directory.\n\nRunning tests in development setup\n----------------------------------\n\nThe\n`setup.cfg <https://github.com/kenjyco/redis-helper/blob/master/setup.cfg>`__\nfile contains the options for ``py.test``, currently ``-vsx -rs --pdb``.\n\nThe ``-vsx -rs --pdb`` options will run tests in a verbose manner and\noutput the reason why tests were skipped (if any were skipped). If there\nare any failing tests, ``py.test`` will stop on the first failure and\ndrop you into a `pdb++ <https://pypi.python.org/pypi/pdbpp/>`__ debugger\nsession.\n\nSee the `debugging\nsection <https://github.com/kenjyco/redis-helper#settings-environments-testing-and-debugging>`__\nof the README for tips on using the debugger and setting breakpoints (in\nthe actual `project\ncode <https://github.com/kenjyco/redis-helper/tree/master/redis_helper>`__,\nor in the `test\ncode <https://github.com/kenjyco/redis-helper/tree/master/tests>`__).\n\n::\n\n % venv/bin/py.test\n\nor\n\n::\n\n % venv/bin/python3 setup.py test\n\n..\n\n Note: This option requires ``setuptools`` to be installed.\n\nUsage\n-----\n\nThe ``rh-download-examples``, ``rh-download-scripts``, ``rh-notes``, and\n``rh-shell`` scripts are provided.\n\n::\n\n $ venv/bin/rh-download-examples --help\n Usage: rh-download-examples [OPTIONS] [DIRECTORY]\n\n Download redis-helper example files from github\n\n Options:\n --help Show this message and exit.\n\n $ venv/bin/rh-download-scripts --help\n Usage: rh-download-scripts [OPTIONS] [DIRECTORY]\n\n Download redis-helper script files from github\n\n Options:\n --help Show this message and exit.\n\n $ venv/bin/rh-notes --help\n Usage: rh-notes [OPTIONS] [TOPIC]\n\n Prompt user to enter notes (about a topic) until finished; or review notes\n\n Options:\n -c, --ch TEXT string appended to the topic (default \"> \")\n -s, --shell Start an ipython shell to inspect the notes collection\n --help Show this message and exit.\n\n $ venv/bin/rh-shell --help\n Usage: rh-shell [OPTIONS]\n\n Interactively select a Collection model and start ipython shell\n\n Options:\n --help Show this message and exit.\n\n.. code:: python\n\n >>> import redis_helper as rh\n >>> collection = rh.Collection(..., index_fields='field1, field3')\n >>> hash_id = collection.add(field1='', field2='', field3='', ...)\n >>> collection.add(...)\n >>> collection.add(...)\n >>> collection.update(hash_id, field1='', field4='', ...)\n >>> change_history = collection.old_data_for_hash_id(hash_id)\n >>> data = collection.get(hash_id)\n >>> some_data = collection.get(hash_id, 'field1, field3')\n >>> results = collection.find(...)\n >>> results2 = collection.find('field1:val, field3:val', ...)\n >>> results3 = collection.find(..., get_fields='field2, field4')\n >>> counts = collection.find(count=True, ...)\n >>> top_indexed = collection.index_field_info()\n >>> collection.delete(hash_id, ...)\n\nBasics - Part 1 (request logging demo)\n--------------------------------------\n\n`Demo <https://asciinema.org/a/101422?t=1:10>`__ bookmarks:\n\n- `1:10 <https://asciinema.org/a/101422?t=1:10>`__ is when the\n ``ipython`` session is started with\n ``venv/bin/ipython -i request_logs.py``\n- `3:14 <https://asciinema.org/a/101422?t=3:14>`__ is when a second\n ``ipython`` session is started (in a separate tmux pane) to simulate\n a steady stream of requests with\n ``slow_trickle_requests(randomsleep=True, show=True)``\n- `4:22 <https://asciinema.org/a/101422?t=4:22>`__ is when the\n ``index_field_info`` method is used to get the latest counts of top\n indexed items\n- `6:11 <https://asciinema.org/a/101422?t=6:11>`__ is when\n ``slow_trickle_requests(.001)`` is run to simulate a large quick\n burst in traffic\n- `7:00 <https://asciinema.org/a/101422?t=7:00>`__ is when multiple\n values are passed in the ``since`` argument of ``find``\\ \u2026\n ``request_logs.find(count=True, since='5:min, 1:min, 30:sec')``\n- `8:37 <https://asciinema.org/a/101422?t=8:37>`__ is when ``get`` and\n ``get_by_position`` methods are used with a variety of arguments to\n change the structure of what\u2019s returned\n- `10:33 <https://asciinema.org/a/101422?t=10:33>`__ is when the\n ``redis_helper.ADMIN_TIMEZONE`` is changed at run time from\n ``America/Chicago`` to ``Europe/London``\n- `11:27 <https://asciinema.org/a/101422?t=11:27>`__ is when ``find``\n is used with a variety of arguments to change the structure of what\u2019s\n returned\n- `14:30 <https://asciinema.org/a/101422?t=14:30>`__ is when ``find``\n is used with multiple search terms and multiple ``since`` values\u2026\n ``request_logs.find('host:dogs.com, uri:/breeds', count=True, since='5:min, 1:min, 10:sec')``\n- `15:54 <https://asciinema.org/a/101422?t=15:54>`__ is when the\n ``update`` method is used to modify data and change history is\n retrieved with the ``old_data_for_hash_id`` method\n\nThe first demo walks through the following:\n\n- creating a virtual environment, installing redis-helper, and\n downloading example files\n\n ::\n\n $ python3 -m venv venv\n $ venv/bin/pip3 install redis-helper ipython\n $ venv/bin/rh-download-examples\n $ cat ~/.config/redis-helper/settings.ini\n $ venv/bin/ipython -i request_logs.py\n\n- using the sample ``Collection`` defined in\n `request_logs.py <https://github.com/kenjyco/redis-helper/blob/master/examples/request_logs.py>`__\n to\n\n - show values of some properties on a ``Collection``\n\n - ``redis_helper.Collection._base_key``\n - ``redis_helper.Collection.now_pretty``\n - ``redis_helper.Collection.now_utc_float``\n - ``redis_helper.Collection.keyspace``\n - ``redis_helper.Collection.size``\n - ``redis_helper.Collection.first``\n - ``redis_helper.Collection.last``\n\n - show values of some settings from ``redis_helper``\n\n - ``redis_helper.APP_ENV``\n - ``redis_helper.REDIS_URL``\n - ``redis_helper.REDIS``\n - ``redis_helper.SETTINGS_FILE``\n - ``redis_helper.ADMIN_TIMEZONE``\n\n - show output from some methods on a ``Collection``\n\n - ``redis_helper.Collection.index_field_info()``\n - ``redis_helper.Collection.find()``\n - ``redis_helper.Collection.find(count=True)``\n - ``redis_helper.Collection.find(count=True, since='30:sec')``\n - ``redis_helper.Collection.find(since='30:sec')``\n - ``redis_helper.Collection.find(since='30:sec', admin_fmt=True)``\n - ``redis_helper.Collection.find(count=True, since='5:min, 1:min, 30:sec')``\n - ``redis_helper.Collection.find('index_field:value')``\n - ``redis_helper.Collection.find('index_field:value', all_fields=True, limit=2)``\n - ``redis_helper.Collection.find('index_field:value', all_fields=True, limit=2, admin_fmt=True, item_format='{_ts} -> {_id}')``\n - ``redis_helper.Collection.find('index_field:value', get_fields='field1, field2', include_meta=False)``\n - ``redis_helper.Collection.find('index_field1:value1, index_field2:value2', count=True)``\n - ``redis_helper.Collection.find('index_field1:value1, index_field2:value2', count=True, since='5:min, 1:min, 10:sec')``\n - ``redis_helper.Collection.get(hash_id)``\n - ``redis_helper.Collection.get(hash_id, 'field1,field2,field3')``\n - ``redis_helper.Collection.get(hash_id, include_meta=True)``\n - ``redis_helper.Collection.get(hash_id, include_meta=True, fields='field1, field2')``\n - ``redis_helper.Collection.get(hash_id, include_meta=True, item_format='{_ts} -> {_id}')``\n - ``redis_helper.Collection.get_by_position(0)``\n - ``redis_helper.Collection.get_by_position(0, include_meta=True, admin_fmt=True)``\n - ``redis_helper.Collection.update(hash_id, field1='value1', field2='value2')``\n - ``redis_helper.Collection.old_data_for_hash_id(hash_id)``\n\nBasics - Part 2 (urls demo, with unique field)\n----------------------------------------------\n\n`Demo <https://asciinema.org/a/75kl95ty9vg2jl93pfz9fbs9q?t=1:00>`__\nbookmarks:\n\n- ``TODO``\n\nThe second demo walks through the following:\n\n- using the sample ``Collection`` defined in\n `urls.py <https://github.com/kenjyco/redis-helper/blob/master/examples/urls.py>`__\n to\n\n - ``TODO``\n\nSettings, environments, testing, and debugging\n----------------------------------------------\n\nTo trigger a debugger session at a specific place in the `project\ncode <https://github.com/kenjyco/redis-helper/tree/master/redis_helper>`__,\ninsert the following, one line above where you want to inspect\n\n::\n\n import pdb; pdb.set_trace()\n\nTo start the debugger inside `test\ncode <https://github.com/kenjyco/redis-helper/tree/master/tests>`__, use\n\n::\n\n pytest.set_trace()\n\n- use ``(l)ist`` to list context lines\n- use ``(n)ext`` to move on to the next statement\n- use ``(s)tep`` to step into a function\n- use ``(c)ontinue`` to continue to next break point\n (i.e.\u00a0``set_trace()`` lines in your code)\n- use ``sticky`` to toggle sticky mode (to constantly show the\n currently executing code as you move through with the debugger)\n- use ``pp`` to pretty print a variable or statement\n\nIf the redis server at ``redis_url`` (in the **test section** of\n``~/.config/redis-server/settings.ini``) is not running or is not empty,\nredis server tests will be skipped.\n\nUse the ``APP_ENV`` environment variable to specify which section of the\n``settings.ini`` file your settings will be loaded from. Any settings in\nthe ``default`` section can be overwritten if explicity set in another\nsection.\n\n- if no ``APP_ENV`` is explicitly set, ``dev`` is assumed\n- the ``APP_ENV`` setting is overwritten to be ``test`` no matter what\n was set when calling ``py.test`` tests\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Easily store, index, and modify Python dicts in Redis (with flexible searching)",
"version": "0.4.8",
"project_urls": {
"Download": "https://github.com/kenjyco/redis-helper/tarball/v0.4.8",
"Homepage": "https://github.com/kenjyco/redis-helper"
},
"split_keywords": [
"redis",
"cli",
"command-line",
"dictionary",
"data",
"database",
"secondary index",
"model",
"prototype",
"event logging",
"dashboard",
"easy modeling",
"helper",
"kenjyco"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "3ed9c78cac6f91a4d3f1071af7728eee47d9ece8ececd0b85e7fa8247627b069",
"md5": "729972274e7214b34ed67a74c04baaee",
"sha256": "63f632f7e8c559e914c38f20867c60d4be8eee9ed55f65d787a61b45edeaed63"
},
"downloads": -1,
"filename": "redis_helper-0.4.8-py3-none-any.whl",
"has_sig": false,
"md5_digest": "729972274e7214b34ed67a74c04baaee",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 30730,
"upload_time": "2024-01-11T02:48:25",
"upload_time_iso_8601": "2024-01-11T02:48:25.847812Z",
"url": "https://files.pythonhosted.org/packages/3e/d9/c78cac6f91a4d3f1071af7728eee47d9ece8ececd0b85e7fa8247627b069/redis_helper-0.4.8-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-01-11 02:48:25",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "kenjyco",
"github_project": "redis-helper",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"requirements": [],
"lcname": "redis-helper"
}
JSON
