Rom - the Redis object mapper for Python
Copyright 2013-2023 Josiah Carlson
Released under the LGPL license version 2.1 and version 3 (you can choose
which you'd like to be bound under).
Sponsorships Available
======================
Don't like LGPL? Sponsor the project and get almost any license you want.
This project has been partly sponsored by structd.com . Historically,
rom has been used to help support the delivery of millions of food orders for
chownow.com, and has been used as the primary backend and prototype for several
startups that have made it all the way to series A.
Thank you to our sponsors and those who have consumed our services.
You are welcome for the good service.
Your company link here.
Documentation
=============
Updated documentation can be found: https://josiahcarlson.github.io/rom/
What
====
Rom is a package whose purpose is to offer active-record style data modeling
within Redis from Python, similar to the semantics of Django ORM, SQLAlchemy,
Google's Appengine datastore, and others.
Why
===
I was building a personal project, wanted to use Redis to store some of my
data, but didn't want to hack it poorly. I looked at the existing Redis object
mappers available in Python, but didn't like the features and functionality
offered.
What is available
=================
Data types:
* Strings (2.x: str/unicode, 3.3+: str), ints, floats, decimals, booleans
* datetime.datetime, datetime.date, datetime.time
* Json columns (for nested structures)
* OneToMany and ManyToOne columns (for model references)
* Non-rom ForeignModel reference support
Indexes:
* Numeric range fetches, searches, and ordering
* Full-word text search (find me entries with col X having words A and B)
* Prefix matching (can be used for prefix-based autocomplete)
* Suffix matching (can be used for suffix-based autocomplete)
* Pattern matching on string-based columns
* All indexing except Geo indexing is available when using Redis 2.6.0 and
later
* Geo indexing available with Redis 3.2.0 and later
Other features:
* Per-thread entity cache (to minimize round-trips, easy saving of all
entities)
* The ability to cache query results and get the key for any other use (see:
``Query.cached_result()``)
Getting started
===============
1. Make sure you have Python 2.6, 2.7, or 3.3+ installed
2. Make sure that you have Andy McCurdy's Redis client library installed:
https://github.com/andymccurdy/redis-py/ or
https://pypi.python.org/pypi/redis
3. Make sure that you have the Python 2 and 3 compatibility library, 'six'
installed: https://pypi.python.org/pypi/six
4. (optional) Make sure that you have the hiredis library installed for Python
5. Make sure that you have a Redis server installed and available remotely
6. Update the Redis connection settings for ``rom`` via
``rom.util.set_connection_settings()`` (other connection update options,
including per-model connections, can be read about in the ``rom.util``
documentation)::
import redis
from rom import util
util.set_connection_settings(host='myhost', db=7)
.. warning:: If you forget to update the connection function, rom will attempt
to connect to localhost:6379 .
7. Create a model::
import rom
# All models to be handled by rom must derived from rom.Model
class User(rom.Model):
email = rom.String(required=True, unique=True, suffix=True)
salt = rom.String()
hash = rom.String()
created_at = rom.Float(default=time.time)
8. Create an instance of the model and save it::
PASSES = 32768
def gen_hash(password, salt=None):
salt = salt or os.urandom(16)
comp = salt + password
out = sha256(comp).digest()
for i in xrange(PASSES-1):
out = sha256(out + comp).digest()
return salt, out
user = User(email='user@host.com')
user.salt, user.hash = gen_hash(password)
user.save()
# session.commit() or session.flush() works too
9. Load and use the object later::
user = User.get_by(email='user@host.com')
at_gmail = User.query.endswith(email='@gmail.com').all()
Lua support
===========
From version 0.25.0 and on, rom assumes that you are using Redis version 2.6
or later, which supports server-side Lua scripting. This allows for the
support of multiple unique column constraints without annoying race conditions
and retries. This also allows for the support of prefix, suffix, and pattern
matching on certain column types.
If you are using a version of Redis prior to 2.6, you should upgrade Redis. If
you are unable or unwilling to upgrade Redis, but you still wish to use rom,
you should call ``rom._disable_lua_writes()``, which will prevent you from
using features that require Lua scripting support.
Expiring models/TTLs
====================
There is a series of feature requests/bug reports/pull requests to add the
ability for rom to automatically delete and/or expire entity data stored in
Redis. This is a request that has been made (as of January 2016) 6 different
times.
Long story short: rom stores a bunch of data in secondary structures to make
querying fast. When a model "expires", that data doesn't get deleted. To
delete that data, you have to run a cleanup function that literally has to
scan over every entity in order to determine if the model had been expired. That
is a huge waste and is the antithesis of good design.
Instead, if you create a new ``expire_at`` float column with ``index=True``,
the column can store when the entity is to expire. Then to expire the data, you
can use: ``Model.query.filter(expire_at=(0, time.time())).limit(10)`` to (for
example) get up to the 10 oldest entities that need to be expired.
Now, I know what you are thinking. You are thinking, "but I wish the data would
just go away on its own." And I don't disagree. But for that to happen, Redis
needs to grow Lua-script triggers, or you need to run a separate daemon to
periodically clean up left-over data. But ... if you need to run a separate
daemon to clean up left-over data by scanning all of your rom entities,
wouldn't it just be better/faster in every way to keep an explicit column and do
it efficiently? I think so, and you should too.
Raw data
{
"_id": null,
"home_page": "https://github.com/josiahcarlson/rom",
"name": "rom",
"maintainer": "",
"docs_url": null,
"requires_python": "",
"maintainer_email": "",
"keywords": "",
"author": "Josiah Carlson",
"author_email": "josiah.carlson@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/04/cd/5930d891be647cc853c26ef3330bdbd5cc5c4e57593a32cb92a977c10806/rom-1.1.1.tar.gz",
"platform": null,
"description": "Rom - the Redis object mapper for Python\n\nCopyright 2013-2023 Josiah Carlson\n\nReleased under the LGPL license version 2.1 and version 3 (you can choose\nwhich you'd like to be bound under).\n\nSponsorships Available\n======================\n\nDon't like LGPL? Sponsor the project and get almost any license you want.\n\nThis project has been partly sponsored by structd.com . Historically,\nrom has been used to help support the delivery of millions of food orders for\nchownow.com, and has been used as the primary backend and prototype for several\nstartups that have made it all the way to series A.\n\nThank you to our sponsors and those who have consumed our services.\n\nYou are welcome for the good service.\n\nYour company link here.\n\nDocumentation\n=============\n\nUpdated documentation can be found: https://josiahcarlson.github.io/rom/\n\nWhat\n====\n\nRom is a package whose purpose is to offer active-record style data modeling\nwithin Redis from Python, similar to the semantics of Django ORM, SQLAlchemy,\nGoogle's Appengine datastore, and others.\n\nWhy\n===\n\nI was building a personal project, wanted to use Redis to store some of my\ndata, but didn't want to hack it poorly. I looked at the existing Redis object\nmappers available in Python, but didn't like the features and functionality\noffered.\n\nWhat is available\n=================\n\nData types:\n\n* Strings (2.x: str/unicode, 3.3+: str), ints, floats, decimals, booleans\n* datetime.datetime, datetime.date, datetime.time\n* Json columns (for nested structures)\n* OneToMany and ManyToOne columns (for model references)\n* Non-rom ForeignModel reference support\n\nIndexes:\n\n* Numeric range fetches, searches, and ordering\n* Full-word text search (find me entries with col X having words A and B)\n* Prefix matching (can be used for prefix-based autocomplete)\n* Suffix matching (can be used for suffix-based autocomplete)\n* Pattern matching on string-based columns\n* All indexing except Geo indexing is available when using Redis 2.6.0 and\n later\n* Geo indexing available with Redis 3.2.0 and later\n\nOther features:\n\n* Per-thread entity cache (to minimize round-trips, easy saving of all\n entities)\n* The ability to cache query results and get the key for any other use (see:\n ``Query.cached_result()``)\n\nGetting started\n===============\n\n1. Make sure you have Python 2.6, 2.7, or 3.3+ installed\n2. Make sure that you have Andy McCurdy's Redis client library installed:\n https://github.com/andymccurdy/redis-py/ or\n https://pypi.python.org/pypi/redis\n3. Make sure that you have the Python 2 and 3 compatibility library, 'six'\n installed: https://pypi.python.org/pypi/six\n4. (optional) Make sure that you have the hiredis library installed for Python\n5. Make sure that you have a Redis server installed and available remotely\n6. Update the Redis connection settings for ``rom`` via\n ``rom.util.set_connection_settings()`` (other connection update options,\n including per-model connections, can be read about in the ``rom.util``\n documentation)::\n\n import redis\n from rom import util\n\n util.set_connection_settings(host='myhost', db=7)\n\n.. warning:: If you forget to update the connection function, rom will attempt\n to connect to localhost:6379 .\n\n7. Create a model::\n\n import rom\n\n # All models to be handled by rom must derived from rom.Model\n class User(rom.Model):\n email = rom.String(required=True, unique=True, suffix=True)\n salt = rom.String()\n hash = rom.String()\n created_at = rom.Float(default=time.time)\n\n8. Create an instance of the model and save it::\n\n PASSES = 32768\n def gen_hash(password, salt=None):\n salt = salt or os.urandom(16)\n comp = salt + password\n out = sha256(comp).digest()\n for i in xrange(PASSES-1):\n out = sha256(out + comp).digest()\n return salt, out\n\n user = User(email='user@host.com')\n user.salt, user.hash = gen_hash(password)\n user.save()\n # session.commit() or session.flush() works too\n\n9. Load and use the object later::\n\n user = User.get_by(email='user@host.com')\n at_gmail = User.query.endswith(email='@gmail.com').all()\n\nLua support\n===========\n\nFrom version 0.25.0 and on, rom assumes that you are using Redis version 2.6\nor later, which supports server-side Lua scripting. This allows for the\nsupport of multiple unique column constraints without annoying race conditions\nand retries. This also allows for the support of prefix, suffix, and pattern\nmatching on certain column types.\n\nIf you are using a version of Redis prior to 2.6, you should upgrade Redis. If\nyou are unable or unwilling to upgrade Redis, but you still wish to use rom,\nyou should call ``rom._disable_lua_writes()``, which will prevent you from\nusing features that require Lua scripting support.\n\nExpiring models/TTLs\n====================\n\nThere is a series of feature requests/bug reports/pull requests to add the\nability for rom to automatically delete and/or expire entity data stored in\nRedis. This is a request that has been made (as of January 2016) 6 different\ntimes.\n\nLong story short: rom stores a bunch of data in secondary structures to make\nquerying fast. When a model \"expires\", that data doesn't get deleted. To\ndelete that data, you have to run a cleanup function that literally has to\nscan over every entity in order to determine if the model had been expired. That\nis a huge waste and is the antithesis of good design.\n\nInstead, if you create a new ``expire_at`` float column with ``index=True``,\nthe column can store when the entity is to expire. Then to expire the data, you\ncan use: ``Model.query.filter(expire_at=(0, time.time())).limit(10)`` to (for\nexample) get up to the 10 oldest entities that need to be expired.\n\nNow, I know what you are thinking. You are thinking, \"but I wish the data would\njust go away on its own.\" And I don't disagree. But for that to happen, Redis\nneeds to grow Lua-script triggers, or you need to run a separate daemon to\nperiodically clean up left-over data. But ... if you need to run a separate\ndaemon to clean up left-over data by scanning all of your rom entities,\nwouldn't it just be better/faster in every way to keep an explicit column and do\nit efficiently? I think so, and you should too.",
"bugtrack_url": null,
"license": "GNU LGPL v2.1",
"summary": "A Redis object mapper for Python",
"version": "1.1.1",
"project_urls": {
"Homepage": "https://github.com/josiahcarlson/rom"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "04cd5930d891be647cc853c26ef3330bdbd5cc5c4e57593a32cb92a977c10806",
"md5": "ad184bd10fb900869a2cb74b2cb10150",
"sha256": "7642211a52edfde00d3bc109ca4acf51a6f9c0974a875482bdf28d95258da1ec"
},
"downloads": -1,
"filename": "rom-1.1.1.tar.gz",
"has_sig": false,
"md5_digest": "ad184bd10fb900869a2cb74b2cb10150",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 71147,
"upload_time": "2023-08-11T00:04:34",
"upload_time_iso_8601": "2023-08-11T00:04:34.531913Z",
"url": "https://files.pythonhosted.org/packages/04/cd/5930d891be647cc853c26ef3330bdbd5cc5c4e57593a32cb92a977c10806/rom-1.1.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-08-11 00:04:34",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "josiahcarlson",
"github_project": "rom",
"travis_ci": true,
"coveralls": false,
"github_actions": false,
"requirements": [],
"tox": true,
"lcname": "rom"
}
JSON
