MeerKAT Science Data Processor Telescope State
==============================================
This is a client package that allows connection to a database that
stores telescope state information for the Science Data Processor of the
MeerKAT radio telescope. This database is colloquially known as *telstate*.
The telescope state is a key-value store. There are three types of keys:
immutables (aka *attributes*)
Stores a single value that is not allowed to change once set.
mutables (aka *sensors*)
Stores multiple timestamped values organised into an ordered set.
indexed
Stores a dictionary of key-value pairs, each of which behaves like an
immutable. This is useful to avoid the main key-space becoming too large.
It also supports some patterns like incrementally storing values but
fetching all values in a single operation. Furthermore, it allows more
general keys than just strings.
The keys are strings and the values (and the sub-keys of indexed keys) are
Python objects serialised via MessagePack_, which has been extended to support
tuples, complex numbers and NumPy arrays. Older versions of the database stored
the values as pickles, and the package warns the user if that's the case. Keys
can be retrieved from the telstate object using attribute syntax or dict
syntax.
.. _MessagePack: http://www.msgpack.org/
Databases can be accessed via one of two backends: a Redis client backend
that allows shared access to an actual Redis server over the network (or a
simulated server via fakeredis) and a simplified in-memory backend for
stand-alone access. Both backends support loading and saving a Redis snapshot
in the form of an RDB dump file.
It is possible to have multiple *views* on the same database (one per telstate
instance). A view is defined as a list of *prefixes* acting as namespaces that
group keys. When reading from the database, each prefix is prepended to the key
in turn until a match is found. When writing to the database, the first prefix
is prepended to the key. The first prefix therefore serves as the primary
namespace while the rest are supplementary read-only namespaces.
.. warning::
**WARNING**: The standard warning about Python pickles applies. Never
retrieve data from an untrusted telstate database with values encoded as
pickles, or connect to such a database over an untrusted network. Pickle
support is disabled by default, but can be enabled for trusted databases
by setting the environment variable KATSDPTELSTATE_ALLOW_PICKLE=1.
Getting Started
---------------
The simplest way to test out `katsdptelstate` is to use the in-memory backend.
If you want to run a real Redis server you will need to install Redis (version
4.0 or newer) on a suitable machine on the network. For example, do this:
- macOS: ``brew install redis``
- Ubuntu: ``apt-get install redis-server``
Then ``pip install katsdptelstate`` and run a local ``redis-server``. If you
also want to load RDB files, do ``pip install katsdptelstate[rdb]``.
A Simple Example
----------------
.. code:: python
import time
import katsdptelstate
# Connect to an actual Redis server via an endpoint or an URL
telstate = katsdptelstate.TelescopeState('localhost:6379')
telstate = katsdptelstate.TelescopeState('redis://localhost')
# Or use the in-memory backend (useful for testing)
telstate = katsdptelstate.TelescopeState()
# Load RDB file into Redis if katsdptelstate is installed with [rdb] option
telstate.load_from_file('dump.rdb')
# Attribute / dict style access returns the latest value
telstate.add('n_chans', 32768)
print(telstate.n_chans) # -> 32768
print(telstate['n_chans']) # -> 32768
# List all keys (attributes and sensors)
print(telstate.keys()) # -> ['n_chans']
# Sensors are timestamped underneath
st = time.time()
telstate.add('n_chans', 4096)
et = time.time()
telstate.add('n_chans', 16384)
# Time ranges can be used and are really fast
telstate.get_range('n_chans', st=st, et=et) # -> [(4096, 1556112474.453495)]
# Add an item 10 seconds back
telstate.add('n_chans', 1024, ts=time.time() - 10)
# Attributes cannot be changed (only deleted)
telstate.add('no_change', 1234, immutable=True)
# Adding it again is OK as long as the value doesn't change
telstate.add('no_change', 1234, immutable=True)
# Simpler notation for setting attributes
telstate['no_change'] = 1234
# Will raise katsdptelstate.ImmutableKeyError
telstate['no_change'] = 456
# Create a new view with namespace 'ns' and standard underscore separator
view = telstate.view('ns')
# Insert a new attribute in this namespace and retrieve it
view['x'] = 1
print(view['x']) # -> 1
print(view.prefixes) # -> ('ns_', '')
print(view.keys()) # -> ['n_chans', 'no_change', 'ns_x']
Asynchronous interface
----------------------
There is also an interface that works with asyncio. Use
``katsdptelstate.aio.TelescopeState`` instead of
``katsdptelstate.TelescopeState``. Functions that interact with the database are now
coroutines. Python 3.6+ is required.
There are a few differences from the synchronous version, partly necessary due
to the nature of asyncio and partly to streamline and modernise the code:
- The constructor only takes a backend, not an endpoint. See below for an
example of how to construct a redis backend.
- There is currently no support for reading or writing RDB files; you'll need
to create a synchronous telescope state client that connects to the same
storage.
- There is no support for attribute-style access.
- Item-style access is supported for read (``await ts.get('key')``), but not
for write. Use ``await ts.set('key', 'value')`` instead to set immutable
keys.
- Instead of ``key in ts``, use ``await ts.exists(key)``.
- The ``wait_key`` and ``wait_indexed`` methods do not take a timeout or a
cancellation future. They can be used with asyncio's cancellation machinery.
The `async-timeout`_ package is useful for timeouts.
- The backend should be closed when no longer needed to avoid warnings.
.. _async-timeout: https://pypi.org/project/async-timeout/
Example
^^^^^^^
.. code:: python
from katsdptelstate.aio import TelescopeState
from katsdptelstate.aio.redis import RedisBackend
# Create a connection to localhost redis server
backend = await RedisBackend.from_url('redis://localhost')
ts = TelescopeState(backend)
# Store and retrieve some data
await ts.set('key', 'value')
print(await ts.get('key'))
# Close the connections (do not try to use ts after this)
ts.backend.close()
await ts.backend.wait_closed()
History
=======
0.14 (2024-02-28)
-----------------
* Improvements to internal type hints (#133)
* Get lupa version (for testing) from katsdpdockerbase (#134)
* Fix variables referenced before use in exception path (#135)
* Improve error message when opening old files containing pickles (#136)
0.13 (2023-03-15)
-----------------
* Upgrade to redis-py>=4.2 and fakeredis>=2, fix test warnings (#132)
* This now requires at least Python 3.7 and no more aioredis
0.12 (2023-03-13)
-----------------
* Switch to aioredis 2.x (#124)
* Add async (and synchronous) `RedisBackend.from_url` constructor (#125, #126)
* Switch from nose to pytest (#129)
* Remove `Endpoint.multicast_subscribe` and `netifaces` dependency (#130)
* Fix Lua script for `set_indexed` so that `wait_indexed` actually works (#127)
* Make aio `wait_key` more robust (#128)
* General cleanup (#122, #123)
0.11 (2021-05-07)
-----------------
* Add asynchronous RDBWriter class (#108)
* Use a transaction for get_range instead of Lua: faster on server side (#110)
* Multiplex aio pubsubs over pool's standard connection (#113)
* Require hiredis for speed and aioredis<2 for compatibility (#114, #118)
* Improve `wait_keys` responsiveness for `MemoryBackend` (#111, #116)
* Avoid blocking the Redis server on telstate.clear (#112)
* Update Docker image to use Redis 6.x instead of 4.x (#109)
* Support older HMSET Redis command in unit tests (#107)
* Requirements cleanup (#115, #117, #119, #120)
0.10 (2020-05-25)
-----------------
* Remove Python 2 support. At least Python 3.5 is required.
* Remove support for old versions of redis-py (#100)
* Use redis-py health checks to improve robustness (#99)
* Add "indexed" keys (#98)
* Add an asyncio interface (#103)
* No longer throw InvalidKeyError when setting a key that shadows a method (#102)
* Add type annotations for mypy (#101)
0.9 (2020-05-25)
----------------
* Deprecate Python 2 support: this is the last release that will support Python 2 (#94)
* Remove ``get_message`` and ``send_message``, which were never used (#89)
* Publish the documentation on https://katsdptelstate.readthedocs.io (#90)
* Disable pickles by default for security (#92)
0.8 (2019-05-06)
----------------
* The default encoding is now msgpack; warn on loading pickles (#75, #79)
* The default backend is now in-memory (#76)
* Add the ability to dump in-memory backend to an RDB file (#77)
* Construct from RDB file-like objects and Redis URLs (#80, #82)
* Report keys and prefixes to the user as strings (#73)
* Add IPython tab completion (#83)
* RDB reader and writer cleanup (#85, #86)
0.7 (2019-02-12)
----------------
* Introduce encodings and add msgpack encoding as alternative to pickle (#64, #65)
* Introduce backends and add in-memory backend as alternative to redis (#71, #72)
* Simplify setting attributes via `__setitem__` (#68)
* Let keys be bytes internally, but allow specification as unicode strings (#63)
* The GitHub repository is now public as well
0.6 (2018-05-10)
----------------
* Initial release of katsdptelstate
Raw data
{
"_id": null,
"home_page": "https://github.com/ska-sa/katsdptelstate",
"name": "katsdptelstate",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.7",
"maintainer_email": "",
"keywords": "meerkat sarao",
"author": "MeerKAT SDP team",
"author_email": "sdpdev+katsdptelstate@sarao.ac.za",
"download_url": "https://files.pythonhosted.org/packages/f6/09/0a063dac1234d7c5a99290fc9e59a8d08c3751907fd864abfbf614f4dba4/katsdptelstate-0.14.tar.gz",
"platform": "OS Independent",
"description": "MeerKAT Science Data Processor Telescope State\n==============================================\n\nThis is a client package that allows connection to a database that\nstores telescope state information for the Science Data Processor of the\nMeerKAT radio telescope. This database is colloquially known as *telstate*.\n\nThe telescope state is a key-value store. There are three types of keys:\n\nimmutables (aka *attributes*)\n Stores a single value that is not allowed to change once set.\n\nmutables (aka *sensors*)\n Stores multiple timestamped values organised into an ordered set.\n\nindexed\n Stores a dictionary of key-value pairs, each of which behaves like an\n immutable. This is useful to avoid the main key-space becoming too large.\n It also supports some patterns like incrementally storing values but\n fetching all values in a single operation. Furthermore, it allows more\n general keys than just strings.\n\nThe keys are strings and the values (and the sub-keys of indexed keys) are\nPython objects serialised via MessagePack_, which has been extended to support\ntuples, complex numbers and NumPy arrays. Older versions of the database stored\nthe values as pickles, and the package warns the user if that's the case. Keys\ncan be retrieved from the telstate object using attribute syntax or dict\nsyntax.\n\n.. _MessagePack: http://www.msgpack.org/\n\nDatabases can be accessed via one of two backends: a Redis client backend\nthat allows shared access to an actual Redis server over the network (or a\nsimulated server via fakeredis) and a simplified in-memory backend for\nstand-alone access. Both backends support loading and saving a Redis snapshot\nin the form of an RDB dump file.\n\nIt is possible to have multiple *views* on the same database (one per telstate\ninstance). A view is defined as a list of *prefixes* acting as namespaces that\ngroup keys. When reading from the database, each prefix is prepended to the key\nin turn until a match is found. When writing to the database, the first prefix\nis prepended to the key. The first prefix therefore serves as the primary\nnamespace while the rest are supplementary read-only namespaces.\n\n.. warning::\n\n **WARNING**: The standard warning about Python pickles applies. Never\n retrieve data from an untrusted telstate database with values encoded as\n pickles, or connect to such a database over an untrusted network. Pickle\n support is disabled by default, but can be enabled for trusted databases\n by setting the environment variable KATSDPTELSTATE_ALLOW_PICKLE=1.\n\nGetting Started\n---------------\n\nThe simplest way to test out `katsdptelstate` is to use the in-memory backend.\nIf you want to run a real Redis server you will need to install Redis (version\n4.0 or newer) on a suitable machine on the network. For example, do this:\n\n- macOS: ``brew install redis``\n- Ubuntu: ``apt-get install redis-server``\n\nThen ``pip install katsdptelstate`` and run a local ``redis-server``. If you\nalso want to load RDB files, do ``pip install katsdptelstate[rdb]``.\n\nA Simple Example\n----------------\n\n.. code:: python\n\n import time\n import katsdptelstate\n\n # Connect to an actual Redis server via an endpoint or an URL\n telstate = katsdptelstate.TelescopeState('localhost:6379')\n telstate = katsdptelstate.TelescopeState('redis://localhost')\n # Or use the in-memory backend (useful for testing)\n telstate = katsdptelstate.TelescopeState()\n # Load RDB file into Redis if katsdptelstate is installed with [rdb] option\n telstate.load_from_file('dump.rdb')\n\n # Attribute / dict style access returns the latest value\n telstate.add('n_chans', 32768)\n print(telstate.n_chans) # -> 32768\n print(telstate['n_chans']) # -> 32768\n\n # List all keys (attributes and sensors)\n print(telstate.keys()) # -> ['n_chans']\n\n # Sensors are timestamped underneath\n st = time.time()\n telstate.add('n_chans', 4096)\n et = time.time()\n telstate.add('n_chans', 16384)\n # Time ranges can be used and are really fast\n telstate.get_range('n_chans', st=st, et=et) # -> [(4096, 1556112474.453495)]\n # Add an item 10 seconds back\n telstate.add('n_chans', 1024, ts=time.time() - 10)\n\n # Attributes cannot be changed (only deleted)\n telstate.add('no_change', 1234, immutable=True)\n # Adding it again is OK as long as the value doesn't change\n telstate.add('no_change', 1234, immutable=True)\n # Simpler notation for setting attributes\n telstate['no_change'] = 1234\n # Will raise katsdptelstate.ImmutableKeyError\n telstate['no_change'] = 456\n\n # Create a new view with namespace 'ns' and standard underscore separator\n view = telstate.view('ns')\n # Insert a new attribute in this namespace and retrieve it\n view['x'] = 1\n print(view['x']) # -> 1\n print(view.prefixes) # -> ('ns_', '')\n print(view.keys()) # -> ['n_chans', 'no_change', 'ns_x']\n\nAsynchronous interface\n----------------------\nThere is also an interface that works with asyncio. Use\n``katsdptelstate.aio.TelescopeState`` instead of\n``katsdptelstate.TelescopeState``. Functions that interact with the database are now\ncoroutines. Python 3.6+ is required.\n\nThere are a few differences from the synchronous version, partly necessary due\nto the nature of asyncio and partly to streamline and modernise the code:\n\n- The constructor only takes a backend, not an endpoint. See below for an\n example of how to construct a redis backend.\n- There is currently no support for reading or writing RDB files; you'll need\n to create a synchronous telescope state client that connects to the same\n storage.\n- There is no support for attribute-style access.\n- Item-style access is supported for read (``await ts.get('key')``), but not\n for write. Use ``await ts.set('key', 'value')`` instead to set immutable\n keys.\n- Instead of ``key in ts``, use ``await ts.exists(key)``.\n- The ``wait_key`` and ``wait_indexed`` methods do not take a timeout or a\n cancellation future. They can be used with asyncio's cancellation machinery.\n The `async-timeout`_ package is useful for timeouts.\n- The backend should be closed when no longer needed to avoid warnings.\n\n.. _async-timeout: https://pypi.org/project/async-timeout/\n\nExample\n^^^^^^^\n\n.. code:: python\n\n from katsdptelstate.aio import TelescopeState\n from katsdptelstate.aio.redis import RedisBackend\n\n # Create a connection to localhost redis server\n backend = await RedisBackend.from_url('redis://localhost')\n ts = TelescopeState(backend)\n\n # Store and retrieve some data\n await ts.set('key', 'value')\n print(await ts.get('key'))\n\n # Close the connections (do not try to use ts after this)\n ts.backend.close()\n await ts.backend.wait_closed()\n\n\nHistory\n=======\n\n0.14 (2024-02-28)\n-----------------\n* Improvements to internal type hints (#133)\n* Get lupa version (for testing) from katsdpdockerbase (#134)\n* Fix variables referenced before use in exception path (#135)\n* Improve error message when opening old files containing pickles (#136)\n\n0.13 (2023-03-15)\n-----------------\n* Upgrade to redis-py>=4.2 and fakeredis>=2, fix test warnings (#132)\n* This now requires at least Python 3.7 and no more aioredis\n\n0.12 (2023-03-13)\n-----------------\n* Switch to aioredis 2.x (#124)\n* Add async (and synchronous) `RedisBackend.from_url` constructor (#125, #126)\n* Switch from nose to pytest (#129)\n* Remove `Endpoint.multicast_subscribe` and `netifaces` dependency (#130)\n* Fix Lua script for `set_indexed` so that `wait_indexed` actually works (#127)\n* Make aio `wait_key` more robust (#128)\n* General cleanup (#122, #123)\n\n0.11 (2021-05-07)\n-----------------\n* Add asynchronous RDBWriter class (#108)\n* Use a transaction for get_range instead of Lua: faster on server side (#110)\n* Multiplex aio pubsubs over pool's standard connection (#113)\n* Require hiredis for speed and aioredis<2 for compatibility (#114, #118)\n* Improve `wait_keys` responsiveness for `MemoryBackend` (#111, #116)\n* Avoid blocking the Redis server on telstate.clear (#112)\n* Update Docker image to use Redis 6.x instead of 4.x (#109)\n* Support older HMSET Redis command in unit tests (#107)\n* Requirements cleanup (#115, #117, #119, #120)\n\n0.10 (2020-05-25)\n-----------------\n* Remove Python 2 support. At least Python 3.5 is required.\n* Remove support for old versions of redis-py (#100)\n* Use redis-py health checks to improve robustness (#99)\n* Add \"indexed\" keys (#98)\n* Add an asyncio interface (#103)\n* No longer throw InvalidKeyError when setting a key that shadows a method (#102)\n* Add type annotations for mypy (#101)\n\n0.9 (2020-05-25)\n----------------\n* Deprecate Python 2 support: this is the last release that will support Python 2 (#94)\n* Remove ``get_message`` and ``send_message``, which were never used (#89)\n* Publish the documentation on https://katsdptelstate.readthedocs.io (#90)\n* Disable pickles by default for security (#92)\n\n0.8 (2019-05-06)\n----------------\n* The default encoding is now msgpack; warn on loading pickles (#75, #79)\n* The default backend is now in-memory (#76)\n* Add the ability to dump in-memory backend to an RDB file (#77)\n* Construct from RDB file-like objects and Redis URLs (#80, #82)\n* Report keys and prefixes to the user as strings (#73)\n* Add IPython tab completion (#83)\n* RDB reader and writer cleanup (#85, #86)\n\n0.7 (2019-02-12)\n----------------\n* Introduce encodings and add msgpack encoding as alternative to pickle (#64, #65)\n* Introduce backends and add in-memory backend as alternative to redis (#71, #72)\n* Simplify setting attributes via `__setitem__` (#68)\n* Let keys be bytes internally, but allow specification as unicode strings (#63)\n* The GitHub repository is now public as well\n\n0.6 (2018-05-10)\n----------------\n* Initial release of katsdptelstate\n",
"bugtrack_url": null,
"license": "Modified BSD",
"summary": "Karoo Array Telescope - Telescope State Client",
"version": "0.14",
"project_urls": {
"Homepage": "https://github.com/ska-sa/katsdptelstate"
},
"split_keywords": [
"meerkat",
"sarao"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "d11f7fa2734168da0cf54370615a6f5472a8c298a79c8b24530f4e932307f78f",
"md5": "7de6ab05666866b47e4588add1cbd566",
"sha256": "56de4ec10bfeb1b662680efb0ccb1777a8643bf5016d66cbbcacf78fea0d3759"
},
"downloads": -1,
"filename": "katsdptelstate-0.14-py3-none-any.whl",
"has_sig": false,
"md5_digest": "7de6ab05666866b47e4588add1cbd566",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7",
"size": 83550,
"upload_time": "2024-02-28T07:15:50",
"upload_time_iso_8601": "2024-02-28T07:15:50.360196Z",
"url": "https://files.pythonhosted.org/packages/d1/1f/7fa2734168da0cf54370615a6f5472a8c298a79c8b24530f4e932307f78f/katsdptelstate-0.14-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "f6090a063dac1234d7c5a99290fc9e59a8d08c3751907fd864abfbf614f4dba4",
"md5": "9142fe2a4dcc6547ca4207f2b403a606",
"sha256": "2b0163f736878cbc4d59b285aab7599b5c02595c0bbd9289609ee2b7cc6cef79"
},
"downloads": -1,
"filename": "katsdptelstate-0.14.tar.gz",
"has_sig": false,
"md5_digest": "9142fe2a4dcc6547ca4207f2b403a606",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7",
"size": 71483,
"upload_time": "2024-02-28T07:15:25",
"upload_time_iso_8601": "2024-02-28T07:15:25.502121Z",
"url": "https://files.pythonhosted.org/packages/f6/09/0a063dac1234d7c5a99290fc9e59a8d08c3751907fd864abfbf614f4dba4/katsdptelstate-0.14.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-02-28 07:15:25",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "ska-sa",
"github_project": "katsdptelstate",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"requirements": [],
"lcname": "katsdptelstate"
}
JSON
