========================
The MoaT-Command library
========================
Rationale
=========
MoaT contains some components which require a possibly bidirectional stream
of asynchronous messaging, including request/reply interactions and data
streaming, possibly across system boundaries (but should be efficient on
same-system calls).
This library supports such interactions.
Prerequisites
=============
This library requires a reliable underlying transport for Python objects.
MoaT uses CBOR, but any reliable, non-reordering messsage stream that can
encode basic Python data structures (plus whatever objects you
send/receive) works.
The MoaT-Cmd library does not itself call the transport. Instead it affords
basic async methods to iterate on messages to send, and to feed incoming
lower-level data in.
Usage
=====
No transport
++++++++++++
.. code-block:: python
from moat.lib.codec import get_codec
from moat.lib.cmd import MsgEndpoint,MsgSender
class Called(MsgEndpoint):
async def handle_command(msg):
if msg.cmd[0] == "Start":
return "OK starting"
if msg.cmd[0] == "gimme data":
async with msg.stream_out("Start") as st:
for i in range(10):
await st.send(i+msg.kw["x"])
return "OK I'm done"
if msg.cmd[0] == "alive":
async with msg.stream_in("Start") as st:
async for data in st:
print("We got", data)
return "OK nice"
raise ValueError(f"Unknown: {msg !r}")
srv=Called()
client=MsgSender(srv)
res, = await client.cmd("Start")
assert res.startswith("OK")
async with client.cmd("gimme data",x=5).stream_in(5) as st:
async for nr, in st:
print(nr) # 5, 6, .. 14
assert st.a[0] == "OK I'm done"
async with client.cmd("alive").stream_out() as st:
for i in range(3):
await st.send(i)
assert st.a[0] == "OK nice"
Using a transport
+++++++++++++++++
TODO
API Specification
=================
TODO
Transport Specification
=======================
MoaT-Cmd messages are encoded with CBOR.
All MoaT-Cmd messages are non-empty lists whose first element is a
small(ish) integer.
A transport that enforces message boundaries MAY send each message without
the leading array mark byte(s). If this option is not used or not
available, messages that are not arrays MAY be used for out-of-band
communication.
MoaT-Cmd messaging is simple by design and consists of a command (sent from
A to B) followed by a reply (sent from B to A). Both directions may
independently indicate that more, possibly streamed, data will follow. The
first and (if streaming) last message of a streamed command or reply are
considered to be out-of-band.
There is no provision for messages that don't have a reply. On the other
hand, an "empty" reply is just three bytes and the sender isn't required to
wait for it.
The side opening a sub-channel uses a unique non-negative integer as
channel ID. Replies carry the ID's bitwise-negated value. Thus the ID
spaces of both directions are separate.
IDs are allocated when sending the first message on a sub-channel. They
MUST NOT be reused until final messages (stream bit off) have been
exchanged in both directions. Corollary: Exactly one final message MUST be
sent in both directions.
Message format
++++++++++++++
A Moat-Cmd message consist of a preferably-small signed integer, plus a
variable and usually non-empty amount of data.
The integer is interpreted as follows.
* Bit 0: if set, the message starts or continues a data stream; if clear,
the message is the final message for this subchannel and direction.
* Bit 1: Error/Warning.
If bit 0 is clear, the message denotes an error which terminates the channel.
Otherwise it is a warning or similar information, and SHOULD be attached
to the following command or reply.
All other bits contain the message ID, left-shifted by two bits. This
scheme allows for five concurrent messages per direction before encoding to
two bytes is required.
Negative integers signal that the ID has been allocated by that message's
recipient. They are inverted bit-wise, i.e. ``(-1-id)``. Thus an ID of zero
is legal. The bits described above are not affected by this inversion. Thus
a command with ID=1 (no streaming, no error) is sent with an initial
integer of 4; the reply uses -5.
Streaming
+++++++++
Data streams are inherently bidirectional. The command's semantics SHOULD
specify which side of a stream is supposed to send data. Error -2 will be
sent (once) if a streamed item is received that won't be handled.
Streaming may start when both sides have exchanged initial messages.
Sending a stream SHOULD NOT commence before the initial command has been
replied to.
Messages with both the streaming and error bits set carry out-of-band data
while the stream is open, e.g. advising the recipient of data loss.
Otherwise they MAY be delivered as warnings or similar out-of-band data.
Conceptally, these messages are attached to the command or reply that
immediately follows them.
For both directions, the initial and final message are assumed to be
out-of-band data. This also applies to warnings.
Flow Control
------------
For the most part: None. MoaT-Cmd is mostly used for monitoring events or
enumerating small data sets.
However, *if* a stream's recipient has limited buffer space and sends a
command that might trigger a nontrivial amount of messages, it MAY send a
specific warning (i.e. a message with both Error and Streaming bits set)
before its initial command or reply. This warning MUST consist of a single
non-negative integer that advises the sender of the number of streamed
messages it may transmit without acknowledgement.
During stream transmission, the recipient then MUST periodically send some
more (positive) integers to signal the availability of more buffer space.
It MUST send such a message if the counter is zero (after space becomes
available of course) and more messages are expected.
The initial flow control messages SHOULD be sent before the initial command
or reply, but MAY be deferred until later.
A receiver SHOULD start flow control sufficiently early, but that isn't
always feasible. It MUST notify the remote side (error -5, below) if an
incoming message gets dropped due to resource exhaustion; likewise, the API
is required to notify the local side.
Error handling
==============
The exact semantics of error messages are application specific.
Error messages with the streaming bit clear terminate the command.
They should be treated as fatal.
Error messages with the streaming bit set are either flow control
messages (see above) or warnings.
Well-Known Errors
+++++++++++++++++
* -1: Unspecified
The ``.stop()`` API method was called.
This message MAY be sent as a warning.
Usage: assume that a sender reads and transmits a block of ten
measurements each second. If a "stop" warning arrives, the sender should
complete the current block before terminating, while a "stop" error
forces the current transmission to end immediately.
* -2: Can't receive this stream
Sent if a command isn't prepared to receive a streamed request or reply
on this endpoint.
* -3: Cancel
The sender's or receiver's task is cancelled: the work is no longer
required / performed.
This message SHOULD NOT be transmitted as a warning;
that would be pointless.
* -4: No Commands
The sender on this side doesn't process commands at all.
* -5: Data loss
An incoming message was dropped due to resource exhaustion (full queue).
This message SHOULD be sent as a warning, but MAY be interpreted as a
hard error by its receiver.
* -6: Must stream
Sent if a command will not handle a non-streamed request or reply.
* -11 …: No Command
The command is not recognized.
The error number encodes the command's position for a hierarchical lookup
at the destination, i.e. if the command is ("foo","bahr","baz") and "foo"
doesn't know about "bahr", the error is -12.
TODO
Other errors are sent using MoaT's link object encapsulation, i.e. the
error type (either a proxy or the name of the exception) followed by its
argument list and keywords (if present).
Examples
++++++++
.. note::
Legend:
* D: direction / sign of message ID
* S: Streaming
* E: Error
Simple command:
= = = ====
S E D Data
= = = ====
- - + Hello
- - - You too
= = = ====
Simple command, error reply:
= = = ====
S E D Data
= = = ====
- - + Hello again
- * - Meh. you already said that
= = = ====
Receive a data stream:
= = = ====
S E D Data
= = = ====
* - + gimme some data
* - - OK here they are
* - - ONE
* - - TWO
* * - Missed some
* - - FIVE
- - + [ 'OopsError' ]
* - - SIX
- - - stopped
= = = ====
Transmit a data stream:
= = = ====
S E D Data
= = = ====
* - + I want to send some data
* - - OK send them
* - + FOO
- - - Nonono I don't want those after all
* - + BAR
- * + OK OK I'll stop
= = = ====
Receive with an error:
= = = ====
S E D Data
= = = ====
* - + gimme some more data
* - - OK here they are
* - - NINE
* - - TEN
- * - [ 'CrashedError', -42, 'Owch', {'mitigating': 'circumstances'} ]
- - + *sigh*
= = = ====
Bidirectional data stream:
= = = ====
S E D Data
= = = ====
* - + Let's talk
* - - OK
* - + *chat data* …
* - - *also chat data* …
- - + hanging up
- - - oh well
= = = ====
Data stream with flow control:
= = = ====
S E D Data
= = = ====
* * + 2
* - + gimme your data
* - - OK here they are
* - - A
* * + 1
* - - BB
* * + 1
* - - CCC
* - - DDDD
* * + 5
* - - EEEEE
* - - FFFFFF
* - - GGGGGGG
- - - that's all
- - + thx
= = = ====
Raw data
{
"_id": null,
"home_page": null,
"name": "moat-lib-cmd",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": "Matthias Urlichs <matthias@urlichs.de>",
"keywords": "MoaT",
"author": null,
"author_email": null,
"download_url": "https://files.pythonhosted.org/packages/5c/a6/6244e91446cb0edef78936e206a3a35d733575d0cc6ce27db1176dbd6235/moat_lib_cmd-0.3.7.tar.gz",
"platform": null,
"description": "========================\nThe MoaT-Command library\n========================\n\nRationale\n=========\n\nMoaT contains some components which require a possibly bidirectional stream\nof asynchronous messaging, including request/reply interactions and data\nstreaming, possibly across system boundaries (but should be efficient on\nsame-system calls).\n\nThis library supports such interactions.\n\nPrerequisites\n=============\n\nThis library requires a reliable underlying transport for Python objects.\nMoaT uses CBOR, but any reliable, non-reordering messsage stream that can\nencode basic Python data structures (plus whatever objects you\nsend/receive) works.\n\nThe MoaT-Cmd library does not itself call the transport. Instead it affords\nbasic async methods to iterate on messages to send, and to feed incoming\nlower-level data in.\n\n\nUsage\n=====\n\nNo transport\n++++++++++++\n\n.. code-block:: python\n\n from moat.lib.codec import get_codec\n from moat.lib.cmd import MsgEndpoint,MsgSender\n\n class Called(MsgEndpoint):\n async def handle_command(msg):\n if msg.cmd[0] == \"Start\":\n return \"OK starting\"\n\n if msg.cmd[0] == \"gimme data\":\n async with msg.stream_out(\"Start\") as st:\n for i in range(10):\n await st.send(i+msg.kw[\"x\"])\n return \"OK I'm done\"\n\n if msg.cmd[0] == \"alive\":\n async with msg.stream_in(\"Start\") as st:\n async for data in st:\n print(\"We got\", data)\n return \"OK nice\"\n\n raise ValueError(f\"Unknown: {msg !r}\")\n \n srv=Called()\n client=MsgSender(srv)\n\n res, = await client.cmd(\"Start\")\n assert res.startswith(\"OK\")\n\n async with client.cmd(\"gimme data\",x=5).stream_in(5) as st:\n async for nr, in st:\n print(nr) # 5, 6, .. 14\n assert st.a[0] == \"OK I'm done\"\n \n async with client.cmd(\"alive\").stream_out() as st:\n for i in range(3):\n await st.send(i)\n assert st.a[0] == \"OK nice\"\n\nUsing a transport\n+++++++++++++++++\n\nTODO\n\n\nAPI Specification\n=================\n\nTODO\n\n \nTransport Specification\n=======================\n\nMoaT-Cmd messages are encoded with CBOR.\n\nAll MoaT-Cmd messages are non-empty lists whose first element is a\nsmall(ish) integer.\n\nA transport that enforces message boundaries MAY send each message without\nthe leading array mark byte(s). If this option is not used or not\navailable, messages that are not arrays MAY be used for out-of-band\ncommunication.\n\nMoaT-Cmd messaging is simple by design and consists of a command (sent from\nA to B) followed by a reply (sent from B to A). Both directions may\nindependently indicate that more, possibly streamed, data will follow. The\nfirst and (if streaming) last message of a streamed command or reply are\nconsidered to be out-of-band.\n\nThere is no provision for messages that don't have a reply. On the other\nhand, an \"empty\" reply is just three bytes and the sender isn't required to\nwait for it.\n\nThe side opening a sub-channel uses a unique non-negative integer as\nchannel ID. Replies carry the ID's bitwise-negated value. Thus the ID\nspaces of both directions are separate.\n\nIDs are allocated when sending the first message on a sub-channel. They\nMUST NOT be reused until final messages (stream bit off) have been\nexchanged in both directions. Corollary: Exactly one final message MUST be\nsent in both directions.\n\n\nMessage format\n++++++++++++++\n\nA Moat-Cmd message consist of a preferably-small signed integer, plus a\nvariable and usually non-empty amount of data.\n\nThe integer is interpreted as follows.\n\n* Bit 0: if set, the message starts or continues a data stream; if clear,\n the message is the final message for this subchannel and direction.\n\n* Bit 1: Error/Warning.\n If bit 0 is clear, the message denotes an error which terminates the channel.\n Otherwise it is a warning or similar information, and SHOULD be attached\n to the following command or reply.\n\nAll other bits contain the message ID, left-shifted by two bits. This\nscheme allows for five concurrent messages per direction before encoding to\ntwo bytes is required.\n\nNegative integers signal that the ID has been allocated by that message's\nrecipient. They are inverted bit-wise, i.e. ``(-1-id)``. Thus an ID of zero\nis legal. The bits described above are not affected by this inversion. Thus\na command with ID=1 (no streaming, no error) is sent with an initial\ninteger of 4; the reply uses -5.\n\n\nStreaming\n+++++++++\n\nData streams are inherently bidirectional. The command's semantics SHOULD\nspecify which side of a stream is supposed to send data. Error -2 will be\nsent (once) if a streamed item is received that won't be handled.\n\nStreaming may start when both sides have exchanged initial messages.\nSending a stream SHOULD NOT commence before the initial command has been\nreplied to.\n\nMessages with both the streaming and error bits set carry out-of-band data\nwhile the stream is open, e.g. advising the recipient of data loss.\nOtherwise they MAY be delivered as warnings or similar out-of-band data.\nConceptally, these messages are attached to the command or reply that\nimmediately follows them.\n\nFor both directions, the initial and final message are assumed to be\nout-of-band data. This also applies to warnings.\n\nFlow Control\n------------\n\nFor the most part: None. MoaT-Cmd is mostly used for monitoring events or\nenumerating small data sets.\n\nHowever, *if* a stream's recipient has limited buffer space and sends a\ncommand that might trigger a nontrivial amount of messages, it MAY send a\nspecific warning (i.e. a message with both Error and Streaming bits set)\nbefore its initial command or reply. This warning MUST consist of a single\nnon-negative integer that advises the sender of the number of streamed\nmessages it may transmit without acknowledgement.\n\nDuring stream transmission, the recipient then MUST periodically send some\nmore (positive) integers to signal the availability of more buffer space.\nIt MUST send such a message if the counter is zero (after space becomes\navailable of course) and more messages are expected.\n\nThe initial flow control messages SHOULD be sent before the initial command\nor reply, but MAY be deferred until later.\n\nA receiver SHOULD start flow control sufficiently early, but that isn't\nalways feasible. It MUST notify the remote side (error -5, below) if an\nincoming message gets dropped due to resource exhaustion; likewise, the API\nis required to notify the local side.\n\nError handling\n==============\n\nThe exact semantics of error messages are application specific.\n\nError messages with the streaming bit clear terminate the command.\nThey should be treated as fatal.\n\nError messages with the streaming bit set are either flow control\nmessages (see above) or warnings.\n\n\nWell-Known Errors\n+++++++++++++++++\n\n* -1: Unspecified\n\n The ``.stop()`` API method was called.\n\n This message MAY be sent as a warning.\n\n Usage: assume that a sender reads and transmits a block of ten\n measurements each second. If a \"stop\" warning arrives, the sender should\n complete the current block before terminating, while a \"stop\" error\n forces the current transmission to end immediately.\n\n* -2: Can't receive this stream\n\n Sent if a command isn't prepared to receive a streamed request or reply\n on this endpoint.\n\n* -3: Cancel\n\n The sender's or receiver's task is cancelled: the work is no longer\n required / performed.\n\n This message SHOULD NOT be transmitted as a warning;\n that would be pointless.\n\n* -4: No Commands\n\n The sender on this side doesn't process commands at all.\n\n* -5: Data loss\n\n An incoming message was dropped due to resource exhaustion (full queue).\n\n This message SHOULD be sent as a warning, but MAY be interpreted as a\n hard error by its receiver.\n\n* -6: Must stream\n\n Sent if a command will not handle a non-streamed request or reply.\n\n\n* -11 \u2026: No Command\n\n The command is not recognized.\n\n The error number encodes the command's position for a hierarchical lookup\n at the destination, i.e. if the command is (\"foo\",\"bahr\",\"baz\") and \"foo\"\n doesn't know about \"bahr\", the error is -12.\n\n TODO\n\nOther errors are sent using MoaT's link object encapsulation, i.e. the\nerror type (either a proxy or the name of the exception) followed by its\nargument list and keywords (if present).\n\nExamples\n++++++++\n\n.. note::\n\n Legend:\n * D: direction / sign of message ID\n * S: Streaming\n * E: Error\n\nSimple command:\n\n= = = ====\nS E D Data\n= = = ====\n- - + Hello\n- - - You too\n= = = ====\n\nSimple command, error reply:\n\n= = = ====\nS E D Data\n= = = ====\n- - + Hello again\n- * - Meh. you already said that\n= = = ====\n\nReceive a data stream:\n\n= = = ====\nS E D Data\n= = = ====\n* - + gimme some data\n* - - OK here they are\n* - - ONE\n* - - TWO\n* * - Missed some\n* - - FIVE\n- - + [ 'OopsError' ]\n* - - SIX\n- - - stopped\n= = = ====\n\nTransmit a data stream:\n\n= = = ====\nS E D Data\n= = = ====\n* - + I want to send some data\n* - - OK send them\n* - + FOO\n- - - Nonono I don't want those after all\n* - + BAR\n- * + OK OK I'll stop\n= = = ====\n\nReceive with an error:\n\n= = = ====\nS E D Data\n= = = ====\n* - + gimme some more data\n* - - OK here they are\n* - - NINE\n* - - TEN\n- * - [ 'CrashedError', -42, 'Owch', {'mitigating': 'circumstances'} ]\n- - + *sigh*\n= = = ====\n\nBidirectional data stream:\n\n= = = ====\nS E D Data\n= = = ====\n* - + Let's talk\n* - - OK\n* - + *chat data* \u2026\n* - - *also chat data* \u2026\n- - + hanging up\n- - - oh well\n= = = ====\n\nData stream with flow control:\n\n= = = ====\nS E D Data\n= = = ====\n* * + 2\n* - + gimme your data\n* - - OK here they are\n* - - A\n* * + 1\n* - - BB\n* * + 1\n* - - CCC\n* - - DDDD\n* * + 5\n* - - EEEEE\n* - - FFFFFF\n* - - GGGGGGG\n- - - that's all\n- - + thx\n= = = ====\n\n",
"bugtrack_url": null,
"license": null,
"summary": "A simple command/stream multiplexer",
"version": "0.3.7",
"project_urls": {
"homepage": "https://m-o-a-t.org",
"repository": "https://github.com/M-o-a-T/moat"
},
"split_keywords": [
"moat"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "8b5c49ce2b339ae025e256f2bf57d719058aae727014936c32c250323c3ff655",
"md5": "ec1047d3dc19f5ab1abe93df84cd9c45",
"sha256": "ac727e4653eb161852c761f8d59cdd702ddeb53c160fb004be37974439677d5a"
},
"downloads": -1,
"filename": "moat_lib_cmd-0.3.7-py3-none-any.whl",
"has_sig": false,
"md5_digest": "ec1047d3dc19f5ab1abe93df84cd9c45",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 23507,
"upload_time": "2025-07-22T07:32:59",
"upload_time_iso_8601": "2025-07-22T07:32:59.146681Z",
"url": "https://files.pythonhosted.org/packages/8b/5c/49ce2b339ae025e256f2bf57d719058aae727014936c32c250323c3ff655/moat_lib_cmd-0.3.7-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "5ca66244e91446cb0edef78936e206a3a35d733575d0cc6ce27db1176dbd6235",
"md5": "860354dfc87a933ede03f43cc08e064c",
"sha256": "a08223aa35116297016266b85c4f089e5137e3ba6dcf9901f0f3cf19f7514162"
},
"downloads": -1,
"filename": "moat_lib_cmd-0.3.7.tar.gz",
"has_sig": false,
"md5_digest": "860354dfc87a933ede03f43cc08e064c",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 25943,
"upload_time": "2025-07-22T07:33:00",
"upload_time_iso_8601": "2025-07-22T07:33:00.463306Z",
"url": "https://files.pythonhosted.org/packages/5c/a6/6244e91446cb0edef78936e206a3a35d733575d0cc6ce27db1176dbd6235/moat_lib_cmd-0.3.7.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-07-22 07:33:00",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "M-o-a-T",
"github_project": "moat",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"requirements": [
{
"name": "anyio",
"specs": []
},
{
"name": "anyio_serial",
"specs": []
},
{
"name": "trio",
"specs": []
},
{
"name": "asyncclick",
"specs": []
},
{
"name": "asyncscope",
"specs": []
},
{
"name": "git",
"specs": []
},
{
"name": "msgpack",
"specs": []
},
{
"name": "simpleeval",
"specs": []
},
{
"name": "ruyaml",
"specs": []
},
{
"name": "cffi",
"specs": []
},
{
"name": "packaging",
"specs": []
},
{
"name": "pymodbus",
"specs": []
},
{
"name": "tomlkit",
"specs": []
}
],
"lcname": "moat-lib-cmd"
}
JSON
