| Name | cs.binary JSON |
| Version |
20240422
JSON |
| download |
| home_page | None |
| Summary | Facilities associated with binary data parsing and transcription. The classes in this module support easy parsing of binary data structures, returning instances with the binary data decoded into attributes and capable of transcribing themselves in binary form (trivially via `bytes(instance)` and also otherwise). |
| upload_time | 2024-04-22 06:31:05 |
| maintainer | None |
| docs_url | None |
| author | None |
| requires_python | None |
| license | GNU General Public License v3 or later (GPLv3+) |
| keywords |
python3
|
| VCS |
|
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
Facilities associated with binary data parsing and transcription.
The classes in this module support easy parsing of binary data
structures,
returning instances with the binary data decoded into attributes
and capable of transcribing themselves in binary form
(trivially via `bytes(instance)` and also otherwise).
*Latest release 20240422*:
New _BinaryMultiValue_Base.for_json() method returning a dict containing the fields.
Note: this module requires Python 3.6+ because various default
behaviours rely on `dict`s preserving their insert order.
See `cs.iso14496` for an ISO 14496 (eg MPEG4) parser
built using this module.
Terminology used below:
* buffer:
an instance of `cs.buffer.CornuCopyBuffer`,
which presents an iterable of bytes-like values
via various useful methods;
it also has a few factory methods to make one from a variety of sources
such as bytes, iterables, binary files, `mmap`ped files,
TCP data streams, etc.
* chunk:
a piece of binary data obeying the buffer protocol,
almost always a `bytes` instance or a `memoryview`,
but in principle also things like `bytearray`.
There are 5 main classes on which an implementor should base their data structures:
* `BinarySingleStruct`: a factory for classes based
on a `struct.struct` format string with a single value;
this builds a `namedtuple` subclass
* `BinaryMultiStruct`: a factory for classes based
on a `struct.struct` format string with multiple values;
this also builds a `namedtuple` subclass
* `BinarySingleValue`: a base class for subclasses
parsing and transcribing a single value
* `BinaryMultiValue`: a base class for subclasses
parsing and transcribing multiple values
with no variation
* `SimpleBinary`: a base class for subclasses
with custom `.parse` and `.transcribe` methods,
for structures with variable fields
All the classes derived from the above inherit all the methods
of `BinaryMixin`.
Amongst other things, this means that the binary transcription
can be had simply from `bytes(instance)`,
although there are more transcription methods provided
for when greater flexibility is desired.
It also means that all classes have `parse`* and `scan`* methods
for parsing binary data streams.
You can also instantiate objects directly;
there's no requirement for the source information to be binary.
There are several presupplied subclasses for common basic types
such as `UInt32BE` (an unsigned 32 bit big endian integer).
## Class `AbstractBinary(BinaryMixin)`
Abstract class for all `Binary`* implementations,
specifying the `parse` and `transcribe` methods
and providing the methods from `BinaryMixin`.
*Method `AbstractBinary.parse(bfr)`*:
Parse an instance of `cls` from the buffer `bfr`.
*Method `AbstractBinary.transcribe(self)`*:
Return or yield `bytes`, ASCII string, `None` or iterables
comprising the binary form of this instance.
This aims for maximum convenience
when transcribing a data structure.
This may be implemented as a generator, yielding parts of the structure.
This may be implemented as a normal function, returning:
* `None`: no bytes of data,
for example for an omitted or empty structure
* a `bytes`-like object: the full data bytes for the structure
* an ASCII compatible string:
this will be encoded with the `'ascii'` encoding to make `bytes`
* an iterable:
the components of the structure,
including substranscriptions which themselves
adhere to this protocol - they may be `None`, `bytes`-like objects,
ASCII compatible strings or iterables.
This supports directly returning or yielding the result of a field's
`.transcribe` method.
## Class `BinaryByteses(AbstractBinary)`
A list of `bytes` parsed directly from the native iteration of the buffer.
## Function `BinaryFixedBytes(class_name, length: int)`
Factory for an `AbstractBinary` subclass matching `length` bytes of data.
The bytes are saved as the attribute `.data`.
## Class `BinaryListValues(AbstractBinary)`
A list of values with a common parse specification,
such as sample or Boxes in an ISO14496 Box structure.
*Method `BinaryListValues.parse(bfr, count=None, *, end_offset=None, min_count=None, max_count=None, pt)`*:
Read values from `bfr`.
Return a `BinaryListValue` containing the values.
Parameters:
* `count`: optional count of values to read;
if specified, exactly this many values are expected.
* `end_offset`: an optional bounding end offset of the buffer.
* `min_count`: the least acceptable number of values.
* `max_count`: the most acceptable number of values.
* `pt`: a parse/transcribe specification
as accepted by the `pt_spec()` factory.
The values will be returned by its parse function.
*Method `BinaryListValues.transcribe(self)`*:
Transcribe all the values.
## Class `BinaryMixin`
Presupplied helper methods for binary objects.
Naming conventions:
- `parse`* methods parse a single instance from a buffer
- `scan`* methods are generators yielding successive instances from a buffer
*Method `BinaryMixin.__bytes__(self)`*:
The binary transcription as a single `bytes` object.
*Method `BinaryMixin.__len__(self)`*:
Compute the length by running a transcription and measuring it.
*Method `BinaryMixin.from_bytes(bs, **kw)`*:
Factory to parse an instance from the
bytes `bs` starting at `offset`.
Returns the new instance.
Raises `ValueError` if `bs` is not entirely consumed.
Raises `EOFError` if `bs` has insufficient data.
Keyword parameters are passed to the `.parse_bytes` method.
This relies on the `cls.parse` method for the parse.
*Method `BinaryMixin.load(f)`*:
Load an instance from the file `f`
which may be a filename or an open file as for `BinaryMixin.scan`.
Return the instance or `None` if the file is empty.
*Method `BinaryMixin.parse_bytes(bs, offset=0, length=None, **kw)`*:
Factory to parse an instance from the
bytes `bs` starting at `offset`.
Returns `(instance,offset)` being the new instance and the post offset.
Raises `EOFError` if `bs` has insufficient data.
The parameters `offset` and `length` are passed to the
`CornuCopyBuffer.from_bytes` factory.
Other keyword parameters are passed to the `.parse` method.
This relies on the `cls.parse` method for the parse.
*Method `BinaryMixin.save(self, f)`*:
Save this instance to the file `f`
which may be a filename or an open file.
Return the length of the transcription.
*Method `BinaryMixin.scan(bfr: cs.buffer.CornuCopyBuffer, count=None, *, min_count=None, max_count=None, with_offsets=False)`*:
Function to scan the buffer `bfr` for repeated instances of `cls`
until end of input and yield them.
Parameters:
* `bfr`: the buffer to scan, or any object suitable for `CornuCopyBuffer.promote`
* `count`: the required number of instances to scan,
equivalent to setting `min_count=count` and `max_count=count`
* `min_count`: the minimum number of instances to scan
* `max_count`: the maximum number of instances to scan
* `with_offsets`: optional flag, default `False`;
if true yield `(pre_offset,obj,post_offset)`, otherwise just `obj`
It is in error to specify both `count` and one of `min_count` or `max_count`.
Scanning stops after `max_count` instances (if specified).
If fewer than `min_count` instances (if specified) are scanned
a warning is issued.
This is to accomodate nonconformant streams
without raising exceptions.
Callers wanting to validate `max_count` may want to probe `bfr.at_eof()`
after return.
Callers not wanting a warning over `min_count` should not specify it,
and instead check the number of instances returned themselves.
*Method `BinaryMixin.scan_fspath(fspath: str, *, with_offsets=False, **kw)`*:
Open the file with filesystenm path `fspath` for read
and yield from `self.scan(..,**kw)` or
`self.scan_with_offsets(..,**kw)` according to the
`with_offsets` parameter.
*Deprecated; please just call `scan` with a filesystem pathname.
Parameters:
* `fspath`: the filesystem path of the file to scan
* `with_offsets`: optional flag, default `False`;
if true then scan with `scan_with_offsets` instead of
with `scan`
Other keyword parameters are passed to `scan` or
`scan_with_offsets`.
*Method `BinaryMixin.scan_with_offsets(bfr, count=None, min_count=None, max_count=None)`*:
Wrapper for `scan()` which yields `(pre_offset,instance,post_offset)`
indicating the start and end offsets of the yielded instances.
All parameters are as for `scan()`.
*Deprecated; please just call `scan` with the `with_offsets=True` parameter.
*Method `BinaryMixin.self_check(self)`*:
Internal self check. Returns `True` if passed.
If the structure has a `FIELD_TYPES` attribute, normally a
class attribute, then check the fields against it. The
`FIELD_TYPES` attribute is a mapping of `field_name` to
a specification of `required` and `types`. The specification
may take one of 2 forms:
* a tuple of `(required,types)`
* a single `type`; this is equivalent to `(True,(type,))`
Their meanings are as follows:
* `required`: a Boolean. If true, the field must be present
in the packet `field_map`, otherwise it need not be present.
* `types`: a tuple of acceptable field types
There are some special semantics involved here.
An implementation of a structure may choose to make some
fields plain instance attributes instead of binary objects
in the `field_map` mapping, particularly variable structures
such as a `cs.iso14496.BoxHeader`, whose `.length` may be parsed
directly from its binary form or computed from other fields
depending on the `box_size` value. Therefore, checking for
a field is first done via the `field_map` mapping, then by
`getattr`, and as such the acceptable `types` may include
nonstructure types such as `int`.
Here is the `cs.iso14496` `Box.FIELD_TYPES` definition as an example:
FIELD_TYPES = {
'header': BoxHeader,
'body': BoxBody,
'unparsed': list,
'offset': int,
'unparsed_offset': int,
'end_offset': int,
}
Note that `length` includes some nonstructure types,
and that it is written as a tuple of `(True,types)` because
it has more than one acceptable type.
*Method `BinaryMixin.transcribe_flat(self)`*:
Return a flat iterable of chunks transcribing this field.
*Method `BinaryMixin.transcribed_length(self)`*:
Compute the length by running a transcription and measuring it.
## Function `BinaryMultiStruct(class_name: str, struct_format: str, field_names: Union[str, List[str]])`
A class factory for `AbstractBinary` `namedtuple` subclasses
built around complex `struct` formats.
Parameters:
* `class_name`: name for the generated class
* `struct_format`: the `struct` format string
* `field_names`: field name list,
a space separated string or an interable of strings
Example:
# an "access point" record from the .ap file
Enigma2APInfo = BinaryMultiStruct('Enigma2APInfo', '>QQ', 'pts offset')
# a "cut" record from the .cuts file
Enigma2Cut = BinaryMultiStruct('Enigma2Cut', '>QL', 'pts type')
## Function `BinaryMultiValue(class_name, field_map, field_order=None)`
Construct a `SimpleBinary` subclass named `class_name`
whose fields are specified by the mapping `field_map`.
The `field_map` is a mapping of field name to buffer parsers and transcribers.
*Note*:
if `field_order` is not specified
it is constructed by iterating over `field_map`.
Prior to Python 3.6, `dict`s do not provide a reliable order
and should be accompanied by an explicit `field_order`.
From 3.6 onward a `dict` is enough and its insertion order
will dictate the default `field_order`.
For a fixed record structure
the default `.parse` and `.transcribe` methods will suffice;
they parse or transcribe each field in turn.
Subclasses with variable records should override
the `.parse` and `.transcribe` methods
accordingly.
The `field_map` is a mapping of field name
to a class returned by the `pt_spec()` function.
If the class has both `parse_value` and `transcribe_value` methods
then the value itself will be directly stored.
Otherwise the class it presumed to be more complex subclass
of `AbstractBinary` and the instance is stored.
Here is an example exhibiting various ways of defining each field:
* `n1`: defined with the *`_value` methods of `UInt8`,
which return or transcribe the `int` from an unsigned 8 bit value;
this stores a `BinarySingleValue` whose `.value` is an `int`
* `n2`: defined from the `UInt8` class,
which parses an unsigned 8 bit value;
this stores an `UInt8` instance
(also a `BinarySingleValue` whole `.value` is an `int`)
* `n3`: like `n2`
* `data1`: defined with the *`_value` methods of `BSData`,
which return or transcribe the data `bytes`
from a run length encoded data chunk;
this stores a `BinarySingleValue` whose `.value` is a `bytes`
* `data2`: defined from the `BSData` class
which parses a run length encoded data chunk;
this is a `BinarySingleValue` so we store its `bytes` value directly.
>>> class BMV(BinaryMultiValue("BMV", {
... 'n1': (UInt8.parse_value, UInt8.transcribe_value),
... 'n2': UInt8,
... 'n3': UInt8,
... 'nd': ('>H4s', 'short bs'),
... 'data1': (
... BSData.parse_value,
... BSData.transcribe_value,
... ),
... 'data2': BSData,
... })):
... pass
>>> BMV.FIELD_ORDER
['n1', 'n2', 'n3', 'nd', 'data1', 'data2']
>>> bmv = BMV.from_bytes(b'\x11\x22\x77\x81\x82zyxw\x02AB\x04DEFG')
>>> bmv.n1 #doctest: +ELLIPSIS
17
>>> bmv.n2
34
>>> bmv #doctest: +ELLIPSIS
BMV(n1=17, n2=34, n3=119, nd=nd_1_short__bs(short=33154, bs=b'zyxw'), data1=b'AB', data2=b'DEFG')
>>> bmv.nd #doctest: +ELLIPSIS
nd_1_short__bs(short=33154, bs=b'zyxw')
>>> bmv.nd.bs
b'zyxw'
>>> bytes(bmv.nd)
b'zyxw'
>>> bmv.data1
b'AB'
>>> bmv.data2
b'DEFG'
>>> bytes(bmv)
b'\x11"w\x81\x82zyxw\x02AB\x04DEFG'
>>> list(bmv.transcribe_flat())
[b'\x11', b'"', b'w', b'\x81\x82zyxw', b'\x02', b'AB', b'\x04', b'DEFG']
## Function `BinarySingleStruct(class_name, struct_format, field_name=None)`
A convenience wrapper for `BinaryMultiStruct`
for `struct_format`s with a single field.
Parameters:
* `class_name`: the class name for the generated class
* `struct_format`: the struct format string, specifying a
single struct field
* `field_name`: optional field name for the value,
default `'value'`
Example:
>>> UInt16BE = BinarySingleStruct('UInt16BE', '>H')
>>> UInt16BE.__name__
'UInt16BE'
>>> UInt16BE.format
'>H'
>>> UInt16BE.struct #doctest: +ELLIPSIS
<_struct.Struct object at ...>
>>> field = UInt16BE.from_bytes(bytes((2,3)))
>>> field
UInt16BE(value=515)
>>> field.value
515
## Class `BinarySingleValue(AbstractBinary)`
A representation of a single value as the attribute `.value`.
Subclasses must implement:
* `parse` or `parse_value`
* `transcribe` or `transcribe_value`
*Method `BinarySingleValue.parse(bfr)`*:
Parse an instance from `bfr`.
Subclasses must implement this method or `parse_value`.
*Method `BinarySingleValue.parse_value(bfr)`*:
Parse a value from `bfr` based on this class.
Subclasses must implement this method or `parse`.
*Method `BinarySingleValue.parse_value_from_bytes(bs, offset=0, length=None, **kw)`*:
Parse a value from the bytes `bs` based on this class.
Return `(value,offset)`.
*Method `BinarySingleValue.scan_values(bfr, **kw)`*:
Scan `bfr`, yield values.
*Method `BinarySingleValue.transcribe(self)`*:
Transcribe this instance as bytes.
Subclasses must implement this method or `transcribe_value`.
*Method `BinarySingleValue.transcribe_value(value)`*:
Transcribe `value` as bytes based on this class.
Subclasses must implement this method or `transcribe`.
## Class `BinaryUTF16NUL(BinarySingleValue)`
A NUL terminated UTF-16 string.
*Method `BinaryUTF16NUL.__init__(self, value, *, encoding)`*:
pylint: disable=super-init-not-called
*Method `BinaryUTF16NUL.parse(bfr, *, encoding)`*:
Parse the encoding and value and construct an instance.
*Method `BinaryUTF16NUL.parse_value(bfr, *, encoding)`*:
Read a NUL terminated UTF-16 string from `bfr`, return a `UTF16NULField`..
The mandatory parameter `encoding` specifies the UTF16 encoding to use
(`'utf_16_be'` or `'utf_16_le'`).
*Method `BinaryUTF16NUL.transcribe(self)`*:
Transcribe `self.value` in UTF-16 with a terminating NUL.
*Method `BinaryUTF16NUL.transcribe_value(value, encoding='utf-16')`*:
Transcribe `value` in UTF-16 with a terminating NUL.
## Class `BinaryUTF8NUL(BinarySingleValue)`
A NUL terminated UTF-8 string.
*Method `BinaryUTF8NUL.parse_value(bfr)`*:
Read a NUL terminated UTF-8 string from `bfr`, return field.
*Method `BinaryUTF8NUL.transcribe_value(s)`*:
Transcribe the `value` in UTF-8 with a terminating NUL.
## Class `BSData(BinarySingleValue)`
A run length encoded data chunk, with the length encoded as a `BSUInt`.
*Property `BSData.data`*:
An alias for the `.value` attribute.
*Property `BSData.data_offset`*:
The length of the length indicator,
useful for computing the location of the raw data.
*Method `BSData.data_offset_for(bs)`*:
Compute the `data_offset` which would obtain for the bytes `bs`.
*Method `BSData.parse_value(bfr)`*:
Parse the data from `bfr`.
*Method `BSData.transcribe_value(data)`*:
Transcribe the payload length and then the payload.
## Class `BSSFloat(BinarySingleValue)`
A float transcribed as a BSString of str(float).
*Method `BSSFloat.parse_value(bfr)`*:
Parse a BSSFloat from a buffer and return the float.
*Method `BSSFloat.transcribe_value(f)`*:
Transcribe a float.
## Class `BSString(BinarySingleValue)`
A run length encoded string, with the length encoded as a BSUInt.
*Method `BSString.parse_value(bfr, encoding='utf-8', errors='strict')`*:
Parse a run length encoded string from `bfr`.
*Method `BSString.transcribe_value(value: str, encoding='utf-8')`*:
Transcribe a string.
## Class `BSUInt(BinarySingleValue)`
A binary serialised unsigned `int`.
This uses a big endian byte encoding where continuation octets
have their high bit set. The bits contributing to the value
are in the low order 7 bits.
*Method `BSUInt.decode_bytes(data, offset=0)`*:
Decode an extensible byte serialised unsigned `int` from `data` at `offset`.
Return value and new offset.
Continuation octets have their high bit set.
The octets are big-endian.
If you just have a `bytes` instance, this is the go. If you're
reading from a stream you're better off with `parse` or `parse_value`.
Examples:
>>> BSUInt.decode_bytes(b'\0')
(0, 1)
Note: there is of course the usual `BinaryMixin.parse_bytes`
but that constructs a buffer to obtain the individual bytes;
this static method will be more performant
if all you are doing is reading this serialisation
and do not already have a buffer.
*Method `BSUInt.parse_value(bfr)`*:
Parse an extensible byte serialised unsigned `int` from a buffer.
Continuation octets have their high bit set.
The value is big-endian.
This is the go for reading from a stream. If you already have
a bare bytes instance then the `.decode_bytes` static method
is probably most efficient;
there is of course the usual `BinaryMixin.parse_bytes`
but that constructs a buffer to obtain the individual bytes.
*Method `BSUInt.transcribe_value(n)`*:
Encode an unsigned int as an entensible byte serialised octet
sequence for decode. Return the bytes object.
## Function `flatten(chunks)`
Flatten `chunks` into an iterable of `bytes` instances.
This exists to allow subclass methods to easily return
transcribeable things (having a `.transcribe` method), ASCII
strings or bytes or iterables or even `None`, in turn allowing
them simply to return their superclass' chunks iterators
directly instead of having to unpack them.
An example from the `cs.iso14496.METABoxBody` class:
def transcribe(self):
yield super().transcribe()
yield self.theHandler
yield self.boxes
The binary classes `flatten` the result of the `.transcribe`
method to obtain `bytes` insteances for the object's bnary
transcription.
## Class `Float64BE(Float64BE, AbstractBinary)`
An `AbstractBinary` `namedtuple` which parses and transcribes
the struct format `'>d'` and presents the attributes ('value',).
*Method `Float64BE.parse(bfr: cs.buffer.CornuCopyBuffer)`*:
Parse from `bfr` via `struct.unpack`.
*Method `Float64BE.parse_value(bfr)`*:
Parse a value from `bfr`, return the value.
*Method `Float64BE.transcribe(self)`*:
Transcribe via `struct.pack`.
*Method `Float64BE.transcribe_value(value)`*:
Transcribe a value back into bytes.
## Class `Float64LE(Float64LE, AbstractBinary)`
An `AbstractBinary` `namedtuple` which parses and transcribes
the struct format `'<d'` and presents the attributes ('value',).
*Method `Float64LE.parse(bfr: cs.buffer.CornuCopyBuffer)`*:
Parse from `bfr` via `struct.unpack`.
*Method `Float64LE.parse_value(bfr)`*:
Parse a value from `bfr`, return the value.
*Method `Float64LE.transcribe(self)`*:
Transcribe via `struct.pack`.
*Method `Float64LE.transcribe_value(value)`*:
Transcribe a value back into bytes.
## Class `Int16BE(Int16BE, AbstractBinary)`
An `AbstractBinary` `namedtuple` which parses and transcribes
the struct format `'>h'` and presents the attributes ('value',).
*Method `Int16BE.parse(bfr: cs.buffer.CornuCopyBuffer)`*:
Parse from `bfr` via `struct.unpack`.
*Method `Int16BE.parse_value(bfr)`*:
Parse a value from `bfr`, return the value.
*Method `Int16BE.transcribe(self)`*:
Transcribe via `struct.pack`.
*Method `Int16BE.transcribe_value(value)`*:
Transcribe a value back into bytes.
## Class `Int16LE(Int16LE, AbstractBinary)`
An `AbstractBinary` `namedtuple` which parses and transcribes
the struct format `'<h'` and presents the attributes ('value',).
*Method `Int16LE.parse(bfr: cs.buffer.CornuCopyBuffer)`*:
Parse from `bfr` via `struct.unpack`.
*Method `Int16LE.parse_value(bfr)`*:
Parse a value from `bfr`, return the value.
*Method `Int16LE.transcribe(self)`*:
Transcribe via `struct.pack`.
*Method `Int16LE.transcribe_value(value)`*:
Transcribe a value back into bytes.
## Class `Int32BE(Int32BE, AbstractBinary)`
An `AbstractBinary` `namedtuple` which parses and transcribes
the struct format `'>l'` and presents the attributes ('value',).
*Method `Int32BE.parse(bfr: cs.buffer.CornuCopyBuffer)`*:
Parse from `bfr` via `struct.unpack`.
*Method `Int32BE.parse_value(bfr)`*:
Parse a value from `bfr`, return the value.
*Method `Int32BE.transcribe(self)`*:
Transcribe via `struct.pack`.
*Method `Int32BE.transcribe_value(value)`*:
Transcribe a value back into bytes.
## Class `Int32LE(Int32LE, AbstractBinary)`
An `AbstractBinary` `namedtuple` which parses and transcribes
the struct format `'<l'` and presents the attributes ('value',).
*Method `Int32LE.parse(bfr: cs.buffer.CornuCopyBuffer)`*:
Parse from `bfr` via `struct.unpack`.
*Method `Int32LE.parse_value(bfr)`*:
Parse a value from `bfr`, return the value.
*Method `Int32LE.transcribe(self)`*:
Transcribe via `struct.pack`.
*Method `Int32LE.transcribe_value(value)`*:
Transcribe a value back into bytes.
## Function `pt_spec(pt, name=None)`
Convert a parse/transcribe specification `pt`
into an `AbstractBinary` subclass.
This is largely used to provide flexibility
in the specifications for the `BinaryMultiValue` factory
but can be used as a factory for other simple classes.
If the specification `pt` is a subclass of `AbstractBinary`
this is returned directly.
If `pt` is a 2-tuple of `str`
the values are presumed to be a format string for `struct.struct`
and filed names separated by spaces;
a new `BinaryMultiStruct` class is created from these and returned.
Otherwise two functions
`f_parse_value(bfr)` and `f_transcribe_value(value)`
are obtained and used to construct a new `BinarySingleValue` class
as follows:
If `pt` has `.parse_value` and `.transcribe_value` callable attributes,
use those for `f_parse_value` and `f_transcribe_value` respectively.
Otherwise, if `pt` is an `int`
define `f_parse_value` to obtain exactly that many bytes from a buffer
and `f_transcribe_value` to return those bytes directly.
Otherwise presume `pt` is a 2-tuple of `(f_parse_value,f_transcribe_value)`.
## Class `SimpleBinary(types.SimpleNamespace, AbstractBinary)`
Abstract binary class based on a `SimpleNamespace`,
thus providing a nice `__str__` and a keyword based `__init__`.
Implementors must still define `.parse` and `.transcribe`.
To constrain the arguments passed to `__init__`,
define an `__init__` which accepts specific keyword arguments
and pass through to `super().__init__()`. Example:
def __init__(self, *, field1=None, field2):
""" Accept only `field1` (optional)
and `field2` (mandatory).
"""
super().__init__(field1=field1, field2=field2)
## Class `UInt16BE(UInt16BE, AbstractBinary)`
An `AbstractBinary` `namedtuple` which parses and transcribes
the struct format `'>H'` and presents the attributes ('value',).
*Method `UInt16BE.parse(bfr: cs.buffer.CornuCopyBuffer)`*:
Parse from `bfr` via `struct.unpack`.
*Method `UInt16BE.parse_value(bfr)`*:
Parse a value from `bfr`, return the value.
*Method `UInt16BE.transcribe(self)`*:
Transcribe via `struct.pack`.
*Method `UInt16BE.transcribe_value(value)`*:
Transcribe a value back into bytes.
## Class `UInt16LE(UInt16LE, AbstractBinary)`
An `AbstractBinary` `namedtuple` which parses and transcribes
the struct format `'<H'` and presents the attributes ('value',).
*Method `UInt16LE.parse(bfr: cs.buffer.CornuCopyBuffer)`*:
Parse from `bfr` via `struct.unpack`.
*Method `UInt16LE.parse_value(bfr)`*:
Parse a value from `bfr`, return the value.
*Method `UInt16LE.transcribe(self)`*:
Transcribe via `struct.pack`.
*Method `UInt16LE.transcribe_value(value)`*:
Transcribe a value back into bytes.
## Class `UInt32BE(UInt32BE, AbstractBinary)`
An `AbstractBinary` `namedtuple` which parses and transcribes
the struct format `'>L'` and presents the attributes ('value',).
*Method `UInt32BE.parse(bfr: cs.buffer.CornuCopyBuffer)`*:
Parse from `bfr` via `struct.unpack`.
*Method `UInt32BE.parse_value(bfr)`*:
Parse a value from `bfr`, return the value.
*Method `UInt32BE.transcribe(self)`*:
Transcribe via `struct.pack`.
*Method `UInt32BE.transcribe_value(value)`*:
Transcribe a value back into bytes.
## Class `UInt32LE(UInt32LE, AbstractBinary)`
An `AbstractBinary` `namedtuple` which parses and transcribes
the struct format `'<L'` and presents the attributes ('value',).
*Method `UInt32LE.parse(bfr: cs.buffer.CornuCopyBuffer)`*:
Parse from `bfr` via `struct.unpack`.
*Method `UInt32LE.parse_value(bfr)`*:
Parse a value from `bfr`, return the value.
*Method `UInt32LE.transcribe(self)`*:
Transcribe via `struct.pack`.
*Method `UInt32LE.transcribe_value(value)`*:
Transcribe a value back into bytes.
## Class `UInt64BE(UInt64BE, AbstractBinary)`
An `AbstractBinary` `namedtuple` which parses and transcribes
the struct format `'>Q'` and presents the attributes ('value',).
*Method `UInt64BE.parse(bfr: cs.buffer.CornuCopyBuffer)`*:
Parse from `bfr` via `struct.unpack`.
*Method `UInt64BE.parse_value(bfr)`*:
Parse a value from `bfr`, return the value.
*Method `UInt64BE.transcribe(self)`*:
Transcribe via `struct.pack`.
*Method `UInt64BE.transcribe_value(value)`*:
Transcribe a value back into bytes.
## Class `UInt64LE(UInt64LE, AbstractBinary)`
An `AbstractBinary` `namedtuple` which parses and transcribes
the struct format `'<Q'` and presents the attributes ('value',).
*Method `UInt64LE.parse(bfr: cs.buffer.CornuCopyBuffer)`*:
Parse from `bfr` via `struct.unpack`.
*Method `UInt64LE.parse_value(bfr)`*:
Parse a value from `bfr`, return the value.
*Method `UInt64LE.transcribe(self)`*:
Transcribe via `struct.pack`.
*Method `UInt64LE.transcribe_value(value)`*:
Transcribe a value back into bytes.
## Class `UInt8(UInt8, AbstractBinary)`
An `AbstractBinary` `namedtuple` which parses and transcribes
the struct format `'B'` and presents the attributes ('value',).
*Method `UInt8.parse(bfr: cs.buffer.CornuCopyBuffer)`*:
Parse from `bfr` via `struct.unpack`.
*Method `UInt8.parse_value(bfr)`*:
Parse a value from `bfr`, return the value.
*Method `UInt8.transcribe(self)`*:
Transcribe via `struct.pack`.
*Method `UInt8.transcribe_value(value)`*:
Transcribe a value back into bytes.
# Release Log
*Release 20240422*:
New _BinaryMultiValue_Base.for_json() method returning a dict containing the fields.
*Release 20240316*:
Fixed release upload artifacts.
*Release 20240201*:
BREAKING CHANGE: drop the long deprecated PacketField related classes.
*Release 20231129*:
BinaryMultiStruct.parse: promote the buffer arguments to a CornuCopyBuffer.
*Release 20230401*:
* BinaryMixin.scan: `bfr` parameter may be any object acceptable to CornuCopyBuffer.promote.
* BinaryMixin.scan: accept new optional with_offsets parameter; deprecate scan_with_offsets and scan_fspathi in favour of scan.
*Release 20230212*:
* BinaryMixin: new load(file) and save(file) methods.
* BinaryMixin.scan: promote the bfr argument.
*Release 20221206*:
Documentation fix.
*Release 20220605*:
BinaryMixin: replace scan_file with scan_fspath, as the former left uncertainty about the amount of the file consumed.
*Release 20210316*:
* BSUInt: rename parse_bytes to decode_bytes, the former name conflicted with BinaryMixin.parse_bytes and broken the semantics.
* Minor refactors.
*Release 20210306*:
MAJOR RELEASE: The PacketField classes and friends were hard to use; this release supplied a suite of easier to use and more consistent Binary* classes, and ports most of those things based on the old scheme to the new scheme.
*Release 20200229*:
* ListField: replace transcribe method with transcribe_value method, aids external use.
* Add `.length` attribute to struct based packet classes providing the data length of the structure (struct.Struct.size).
* Packet: new `add_deferred_field` method to consume the raw data for a field for parsing later (done automatically if the attribute is accessed).
* New `@deferred_field` decorator for the parser for that stashed data.
*Release 20191230.3*:
Docstring tweak.
*Release 20191230.2*:
Documentation updates.
*Release 20191230.1*:
Docstring updates. Semantic changes were in the previous release.
*Release 20191230*:
* ListField: new __iter__ method.
* Packet: __str__: accept optional `skip_fields` parameter to omit some field names.
* Packet: new .add_from_value method to add a named field with a presupplied value.
* Packet: new remove_field(field_name) and pop_field() methods to remove fields.
* BytesesField: __iter__ yields the bytes values, transcribe=__iter__.
* PacketField: propagate keyword arguments through various methods, required for parameterised PacketFields.
* New UTF16NULField, a NUL terminated UTF16 string.
* PacketField: provide a default `.transcribe_value` method which makes a new instance and calls its `.transcribe` method.
* Documentation update and several minor changes.
*Release 20190220*:
* Packet.self_check: fields without a sanity check cause a warning, not a ValueError.
* New Float64BE, Float64LE and BSSFloat classes for IEEE floats and floats-as-strings.
* Additional module docstringage on subclassing Packet and PacketField.
* BSString: drop redundant from_buffer class method.
* PacketField.__init__: default to value=None if omitted.
*Release 20181231*:
flatten: do not yield zero length bytelike objects, can be misread as EOF on some streams.
*Release 20181108*:
* New PacketField.transcribe_value_flat convenience method to return a flat iterable of bytes-like objects.
* New PacketField.parse_buffer generator method to parse instances of the PacketField from a buffer until end of input.
* New PacketField.parse_buffer_values generator method to parse instances of the PacketField from a buffer and yield the `.value` attribute until end of input.
*Release 20180823*:
* Some bugfixes.
* Define PacketField.__eq__.
* BSUInt, BSData and BSString classes implementing the serialisations from cs.serialise.
* New PacketField.value_from_bytes class method.
* New PacketField.value_from_buffer method.
*Release 20180810.2*:
Documentation improvements.
*Release 20180810.1*:
Improve module description.
*Release 20180810*:
BytesesField.from_buffer: make use of the buffer's skipto method if discard_data is true.
*Release 20180805*:
* Packet: now an abstract class, new self_check method initially checking the
* PACKET_FIELDS class attribute against the instance, new methods get_field
* and set_field to fetch or replace existing fields, allow keyword arguments
* to initialise the Packet fields and document the dependency on keyword
* argument ordering.
* PacketField: __len__ computed directory from a transcribe, drop other __len__
* methods.
* EmptyField singleton to use as a placeholder for missing optional fields.
* BytesField: implement value_s and from_buffer.
* multi_struct_field: implement __len__ for generated class.
* flatten: treat memoryviews like bytes.
* Assorted docstrings and fixes.
*Release 20180801*:
Initial PyPI release.
Raw data
{
"_id": null,
"home_page": null,
"name": "cs.binary",
"maintainer": null,
"docs_url": null,
"requires_python": null,
"maintainer_email": null,
"keywords": "python3",
"author": null,
"author_email": "Cameron Simpson <cs@cskk.id.au>",
"download_url": "https://files.pythonhosted.org/packages/65/64/ca2a6141c5a6ed53bcaadbb7c0a7b02643017b4d4386e5d9e5e121f35db3/cs.binary-20240422.tar.gz",
"platform": null,
"description": "Facilities associated with binary data parsing and transcription.\nThe classes in this module support easy parsing of binary data\nstructures,\nreturning instances with the binary data decoded into attributes\nand capable of transcribing themselves in binary form\n(trivially via `bytes(instance)` and also otherwise).\n\n*Latest release 20240422*:\nNew _BinaryMultiValue_Base.for_json() method returning a dict containing the fields.\n\nNote: this module requires Python 3.6+ because various default\nbehaviours rely on `dict`s preserving their insert order.\n\nSee `cs.iso14496` for an ISO 14496 (eg MPEG4) parser\nbuilt using this module.\n\nTerminology used below:\n* buffer:\n an instance of `cs.buffer.CornuCopyBuffer`,\n which presents an iterable of bytes-like values\n via various useful methods;\n it also has a few factory methods to make one from a variety of sources\n such as bytes, iterables, binary files, `mmap`ped files,\n TCP data streams, etc.\n* chunk:\n a piece of binary data obeying the buffer protocol,\n almost always a `bytes` instance or a `memoryview`,\n but in principle also things like `bytearray`.\n\nThere are 5 main classes on which an implementor should base their data structures:\n* `BinarySingleStruct`: a factory for classes based\n on a `struct.struct` format string with a single value;\n this builds a `namedtuple` subclass\n* `BinaryMultiStruct`: a factory for classes based\n on a `struct.struct` format string with multiple values;\n this also builds a `namedtuple` subclass\n* `BinarySingleValue`: a base class for subclasses\n parsing and transcribing a single value\n* `BinaryMultiValue`: a base class for subclasses\n parsing and transcribing multiple values\n with no variation\n* `SimpleBinary`: a base class for subclasses\n with custom `.parse` and `.transcribe` methods,\n for structures with variable fields\n\nAll the classes derived from the above inherit all the methods\nof `BinaryMixin`.\nAmongst other things, this means that the binary transcription\ncan be had simply from `bytes(instance)`,\nalthough there are more transcription methods provided\nfor when greater flexibility is desired.\nIt also means that all classes have `parse`* and `scan`* methods\nfor parsing binary data streams.\n\nYou can also instantiate objects directly;\nthere's no requirement for the source information to be binary.\n\nThere are several presupplied subclasses for common basic types\nsuch as `UInt32BE` (an unsigned 32 bit big endian integer).\n\n## Class `AbstractBinary(BinaryMixin)`\n\nAbstract class for all `Binary`* implementations,\nspecifying the `parse` and `transcribe` methods\nand providing the methods from `BinaryMixin`.\n\n*Method `AbstractBinary.parse(bfr)`*:\nParse an instance of `cls` from the buffer `bfr`.\n\n*Method `AbstractBinary.transcribe(self)`*:\nReturn or yield `bytes`, ASCII string, `None` or iterables\ncomprising the binary form of this instance.\n\nThis aims for maximum convenience\nwhen transcribing a data structure.\n\nThis may be implemented as a generator, yielding parts of the structure.\n\nThis may be implemented as a normal function, returning:\n* `None`: no bytes of data,\n for example for an omitted or empty structure\n* a `bytes`-like object: the full data bytes for the structure\n* an ASCII compatible string:\n this will be encoded with the `'ascii'` encoding to make `bytes`\n* an iterable:\n the components of the structure,\n including substranscriptions which themselves\n adhere to this protocol - they may be `None`, `bytes`-like objects,\n ASCII compatible strings or iterables.\n This supports directly returning or yielding the result of a field's\n `.transcribe` method.\n\n## Class `BinaryByteses(AbstractBinary)`\n\nA list of `bytes` parsed directly from the native iteration of the buffer.\n\n## Function `BinaryFixedBytes(class_name, length: int)`\n\nFactory for an `AbstractBinary` subclass matching `length` bytes of data.\nThe bytes are saved as the attribute `.data`.\n\n## Class `BinaryListValues(AbstractBinary)`\n\nA list of values with a common parse specification,\nsuch as sample or Boxes in an ISO14496 Box structure.\n\n*Method `BinaryListValues.parse(bfr, count=None, *, end_offset=None, min_count=None, max_count=None, pt)`*:\nRead values from `bfr`.\nReturn a `BinaryListValue` containing the values.\n\nParameters:\n* `count`: optional count of values to read;\n if specified, exactly this many values are expected.\n* `end_offset`: an optional bounding end offset of the buffer.\n* `min_count`: the least acceptable number of values.\n* `max_count`: the most acceptable number of values.\n* `pt`: a parse/transcribe specification\n as accepted by the `pt_spec()` factory.\n The values will be returned by its parse function.\n\n*Method `BinaryListValues.transcribe(self)`*:\nTranscribe all the values.\n\n## Class `BinaryMixin`\n\nPresupplied helper methods for binary objects.\n\nNaming conventions:\n- `parse`* methods parse a single instance from a buffer\n- `scan`* methods are generators yielding successive instances from a buffer\n\n*Method `BinaryMixin.__bytes__(self)`*:\nThe binary transcription as a single `bytes` object.\n\n*Method `BinaryMixin.__len__(self)`*:\nCompute the length by running a transcription and measuring it.\n\n*Method `BinaryMixin.from_bytes(bs, **kw)`*:\nFactory to parse an instance from the\nbytes `bs` starting at `offset`.\nReturns the new instance.\n\nRaises `ValueError` if `bs` is not entirely consumed.\nRaises `EOFError` if `bs` has insufficient data.\n\nKeyword parameters are passed to the `.parse_bytes` method.\n\nThis relies on the `cls.parse` method for the parse.\n\n*Method `BinaryMixin.load(f)`*:\nLoad an instance from the file `f`\nwhich may be a filename or an open file as for `BinaryMixin.scan`.\nReturn the instance or `None` if the file is empty.\n\n*Method `BinaryMixin.parse_bytes(bs, offset=0, length=None, **kw)`*:\nFactory to parse an instance from the\nbytes `bs` starting at `offset`.\nReturns `(instance,offset)` being the new instance and the post offset.\n\nRaises `EOFError` if `bs` has insufficient data.\n\nThe parameters `offset` and `length` are passed to the\n`CornuCopyBuffer.from_bytes` factory.\n\nOther keyword parameters are passed to the `.parse` method.\n\nThis relies on the `cls.parse` method for the parse.\n\n*Method `BinaryMixin.save(self, f)`*:\nSave this instance to the file `f`\nwhich may be a filename or an open file.\nReturn the length of the transcription.\n\n*Method `BinaryMixin.scan(bfr: cs.buffer.CornuCopyBuffer, count=None, *, min_count=None, max_count=None, with_offsets=False)`*:\nFunction to scan the buffer `bfr` for repeated instances of `cls`\nuntil end of input and yield them.\n\nParameters:\n* `bfr`: the buffer to scan, or any object suitable for `CornuCopyBuffer.promote`\n* `count`: the required number of instances to scan,\n equivalent to setting `min_count=count` and `max_count=count`\n* `min_count`: the minimum number of instances to scan\n* `max_count`: the maximum number of instances to scan\n* `with_offsets`: optional flag, default `False`;\n if true yield `(pre_offset,obj,post_offset)`, otherwise just `obj`\nIt is in error to specify both `count` and one of `min_count` or `max_count`.\n\nScanning stops after `max_count` instances (if specified).\nIf fewer than `min_count` instances (if specified) are scanned\na warning is issued.\nThis is to accomodate nonconformant streams\nwithout raising exceptions.\nCallers wanting to validate `max_count` may want to probe `bfr.at_eof()`\nafter return.\nCallers not wanting a warning over `min_count` should not specify it,\nand instead check the number of instances returned themselves.\n\n*Method `BinaryMixin.scan_fspath(fspath: str, *, with_offsets=False, **kw)`*:\nOpen the file with filesystenm path `fspath` for read\nand yield from `self.scan(..,**kw)` or\n`self.scan_with_offsets(..,**kw)` according to the\n`with_offsets` parameter.\n\n*Deprecated; please just call `scan` with a filesystem pathname.\n\nParameters:\n* `fspath`: the filesystem path of the file to scan\n* `with_offsets`: optional flag, default `False`;\n if true then scan with `scan_with_offsets` instead of\n with `scan`\nOther keyword parameters are passed to `scan` or\n`scan_with_offsets`.\n\n*Method `BinaryMixin.scan_with_offsets(bfr, count=None, min_count=None, max_count=None)`*:\nWrapper for `scan()` which yields `(pre_offset,instance,post_offset)`\nindicating the start and end offsets of the yielded instances.\nAll parameters are as for `scan()`.\n\n*Deprecated; please just call `scan` with the `with_offsets=True` parameter.\n\n*Method `BinaryMixin.self_check(self)`*:\nInternal self check. Returns `True` if passed.\n\nIf the structure has a `FIELD_TYPES` attribute, normally a\nclass attribute, then check the fields against it. The\n`FIELD_TYPES` attribute is a mapping of `field_name` to\na specification of `required` and `types`. The specification\nmay take one of 2 forms:\n* a tuple of `(required,types)`\n* a single `type`; this is equivalent to `(True,(type,))`\nTheir meanings are as follows:\n* `required`: a Boolean. If true, the field must be present\n in the packet `field_map`, otherwise it need not be present.\n* `types`: a tuple of acceptable field types\n\nThere are some special semantics involved here.\n\nAn implementation of a structure may choose to make some\nfields plain instance attributes instead of binary objects\nin the `field_map` mapping, particularly variable structures\nsuch as a `cs.iso14496.BoxHeader`, whose `.length` may be parsed\ndirectly from its binary form or computed from other fields\ndepending on the `box_size` value. Therefore, checking for\na field is first done via the `field_map` mapping, then by\n`getattr`, and as such the acceptable `types` may include\nnonstructure types such as `int`.\n\nHere is the `cs.iso14496` `Box.FIELD_TYPES` definition as an example:\n\n FIELD_TYPES = {\n 'header': BoxHeader,\n 'body': BoxBody,\n 'unparsed': list,\n 'offset': int,\n 'unparsed_offset': int,\n 'end_offset': int,\n }\n\nNote that `length` includes some nonstructure types,\nand that it is written as a tuple of `(True,types)` because\nit has more than one acceptable type.\n\n*Method `BinaryMixin.transcribe_flat(self)`*:\nReturn a flat iterable of chunks transcribing this field.\n\n*Method `BinaryMixin.transcribed_length(self)`*:\nCompute the length by running a transcription and measuring it.\n\n## Function `BinaryMultiStruct(class_name: str, struct_format: str, field_names: Union[str, List[str]])`\n\nA class factory for `AbstractBinary` `namedtuple` subclasses\nbuilt around complex `struct` formats.\n\nParameters:\n* `class_name`: name for the generated class\n* `struct_format`: the `struct` format string\n* `field_names`: field name list,\n a space separated string or an interable of strings\n\nExample:\n\n # an \"access point\" record from the .ap file\n Enigma2APInfo = BinaryMultiStruct('Enigma2APInfo', '>QQ', 'pts offset')\n\n # a \"cut\" record from the .cuts file\n Enigma2Cut = BinaryMultiStruct('Enigma2Cut', '>QL', 'pts type')\n\n## Function `BinaryMultiValue(class_name, field_map, field_order=None)`\n\nConstruct a `SimpleBinary` subclass named `class_name`\nwhose fields are specified by the mapping `field_map`.\n\nThe `field_map` is a mapping of field name to buffer parsers and transcribers.\n\n*Note*:\nif `field_order` is not specified\nit is constructed by iterating over `field_map`.\nPrior to Python 3.6, `dict`s do not provide a reliable order\nand should be accompanied by an explicit `field_order`.\nFrom 3.6 onward a `dict` is enough and its insertion order\nwill dictate the default `field_order`.\n\nFor a fixed record structure\nthe default `.parse` and `.transcribe` methods will suffice;\nthey parse or transcribe each field in turn.\nSubclasses with variable records should override\nthe `.parse` and `.transcribe` methods\naccordingly.\n\nThe `field_map` is a mapping of field name\nto a class returned by the `pt_spec()` function.\n\nIf the class has both `parse_value` and `transcribe_value` methods\nthen the value itself will be directly stored.\nOtherwise the class it presumed to be more complex subclass\nof `AbstractBinary` and the instance is stored.\n\nHere is an example exhibiting various ways of defining each field:\n* `n1`: defined with the *`_value` methods of `UInt8`,\n which return or transcribe the `int` from an unsigned 8 bit value;\n this stores a `BinarySingleValue` whose `.value` is an `int`\n* `n2`: defined from the `UInt8` class,\n which parses an unsigned 8 bit value;\n this stores an `UInt8` instance\n (also a `BinarySingleValue` whole `.value` is an `int`)\n* `n3`: like `n2`\n* `data1`: defined with the *`_value` methods of `BSData`,\n which return or transcribe the data `bytes`\n from a run length encoded data chunk;\n this stores a `BinarySingleValue` whose `.value` is a `bytes`\n* `data2`: defined from the `BSData` class\n which parses a run length encoded data chunk;\n this is a `BinarySingleValue` so we store its `bytes` value directly.\n\n >>> class BMV(BinaryMultiValue(\"BMV\", {\n ... 'n1': (UInt8.parse_value, UInt8.transcribe_value),\n ... 'n2': UInt8,\n ... 'n3': UInt8,\n ... 'nd': ('>H4s', 'short bs'),\n ... 'data1': (\n ... BSData.parse_value,\n ... BSData.transcribe_value,\n ... ),\n ... 'data2': BSData,\n ... })):\n ... pass\n >>> BMV.FIELD_ORDER\n ['n1', 'n2', 'n3', 'nd', 'data1', 'data2']\n >>> bmv = BMV.from_bytes(b'\\x11\\x22\\x77\\x81\\x82zyxw\\x02AB\\x04DEFG')\n >>> bmv.n1 #doctest: +ELLIPSIS\n 17\n >>> bmv.n2\n 34\n >>> bmv #doctest: +ELLIPSIS\n BMV(n1=17, n2=34, n3=119, nd=nd_1_short__bs(short=33154, bs=b'zyxw'), data1=b'AB', data2=b'DEFG')\n >>> bmv.nd #doctest: +ELLIPSIS\n nd_1_short__bs(short=33154, bs=b'zyxw')\n >>> bmv.nd.bs\n b'zyxw'\n >>> bytes(bmv.nd)\n b'\u0081\u0082zyxw'\n >>> bmv.data1\n b'AB'\n >>> bmv.data2\n b'DEFG'\n >>> bytes(bmv)\n b'\\x11\"w\\x81\\x82zyxw\\x02AB\\x04DEFG'\n >>> list(bmv.transcribe_flat())\n [b'\\x11', b'\"', b'w', b'\\x81\\x82zyxw', b'\\x02', b'AB', b'\\x04', b'DEFG']\n\n## Function `BinarySingleStruct(class_name, struct_format, field_name=None)`\n\nA convenience wrapper for `BinaryMultiStruct`\nfor `struct_format`s with a single field.\n\nParameters:\n* `class_name`: the class name for the generated class\n* `struct_format`: the struct format string, specifying a\n single struct field\n* `field_name`: optional field name for the value,\n default `'value'`\n\nExample:\n\n >>> UInt16BE = BinarySingleStruct('UInt16BE', '>H')\n >>> UInt16BE.__name__\n 'UInt16BE'\n >>> UInt16BE.format\n '>H'\n >>> UInt16BE.struct #doctest: +ELLIPSIS\n <_struct.Struct object at ...>\n >>> field = UInt16BE.from_bytes(bytes((2,3)))\n >>> field\n UInt16BE(value=515)\n >>> field.value\n 515\n\n## Class `BinarySingleValue(AbstractBinary)`\n\nA representation of a single value as the attribute `.value`.\n\nSubclasses must implement:\n* `parse` or `parse_value`\n* `transcribe` or `transcribe_value`\n\n*Method `BinarySingleValue.parse(bfr)`*:\nParse an instance from `bfr`.\n\nSubclasses must implement this method or `parse_value`.\n\n*Method `BinarySingleValue.parse_value(bfr)`*:\nParse a value from `bfr` based on this class.\n\nSubclasses must implement this method or `parse`.\n\n*Method `BinarySingleValue.parse_value_from_bytes(bs, offset=0, length=None, **kw)`*:\nParse a value from the bytes `bs` based on this class.\nReturn `(value,offset)`.\n\n*Method `BinarySingleValue.scan_values(bfr, **kw)`*:\nScan `bfr`, yield values.\n\n*Method `BinarySingleValue.transcribe(self)`*:\nTranscribe this instance as bytes.\n\nSubclasses must implement this method or `transcribe_value`.\n\n*Method `BinarySingleValue.transcribe_value(value)`*:\nTranscribe `value` as bytes based on this class.\n\nSubclasses must implement this method or `transcribe`.\n\n## Class `BinaryUTF16NUL(BinarySingleValue)`\n\nA NUL terminated UTF-16 string.\n\n*Method `BinaryUTF16NUL.__init__(self, value, *, encoding)`*:\npylint: disable=super-init-not-called\n\n*Method `BinaryUTF16NUL.parse(bfr, *, encoding)`*:\nParse the encoding and value and construct an instance.\n\n*Method `BinaryUTF16NUL.parse_value(bfr, *, encoding)`*:\nRead a NUL terminated UTF-16 string from `bfr`, return a `UTF16NULField`..\nThe mandatory parameter `encoding` specifies the UTF16 encoding to use\n(`'utf_16_be'` or `'utf_16_le'`).\n\n*Method `BinaryUTF16NUL.transcribe(self)`*:\nTranscribe `self.value` in UTF-16 with a terminating NUL.\n\n*Method `BinaryUTF16NUL.transcribe_value(value, encoding='utf-16')`*:\nTranscribe `value` in UTF-16 with a terminating NUL.\n\n## Class `BinaryUTF8NUL(BinarySingleValue)`\n\nA NUL terminated UTF-8 string.\n\n*Method `BinaryUTF8NUL.parse_value(bfr)`*:\nRead a NUL terminated UTF-8 string from `bfr`, return field.\n\n*Method `BinaryUTF8NUL.transcribe_value(s)`*:\nTranscribe the `value` in UTF-8 with a terminating NUL.\n\n## Class `BSData(BinarySingleValue)`\n\nA run length encoded data chunk, with the length encoded as a `BSUInt`.\n\n*Property `BSData.data`*:\nAn alias for the `.value` attribute.\n\n*Property `BSData.data_offset`*:\nThe length of the length indicator,\nuseful for computing the location of the raw data.\n\n*Method `BSData.data_offset_for(bs)`*:\nCompute the `data_offset` which would obtain for the bytes `bs`.\n\n*Method `BSData.parse_value(bfr)`*:\nParse the data from `bfr`.\n\n*Method `BSData.transcribe_value(data)`*:\nTranscribe the payload length and then the payload.\n\n## Class `BSSFloat(BinarySingleValue)`\n\nA float transcribed as a BSString of str(float).\n\n*Method `BSSFloat.parse_value(bfr)`*:\nParse a BSSFloat from a buffer and return the float.\n\n*Method `BSSFloat.transcribe_value(f)`*:\nTranscribe a float.\n\n## Class `BSString(BinarySingleValue)`\n\nA run length encoded string, with the length encoded as a BSUInt.\n\n*Method `BSString.parse_value(bfr, encoding='utf-8', errors='strict')`*:\nParse a run length encoded string from `bfr`.\n\n*Method `BSString.transcribe_value(value: str, encoding='utf-8')`*:\nTranscribe a string.\n\n## Class `BSUInt(BinarySingleValue)`\n\nA binary serialised unsigned `int`.\n\nThis uses a big endian byte encoding where continuation octets\nhave their high bit set. The bits contributing to the value\nare in the low order 7 bits.\n\n*Method `BSUInt.decode_bytes(data, offset=0)`*:\nDecode an extensible byte serialised unsigned `int` from `data` at `offset`.\nReturn value and new offset.\n\nContinuation octets have their high bit set.\nThe octets are big-endian.\n\nIf you just have a `bytes` instance, this is the go. If you're\nreading from a stream you're better off with `parse` or `parse_value`.\n\nExamples:\n\n >>> BSUInt.decode_bytes(b'\\0')\n (0, 1)\n\nNote: there is of course the usual `BinaryMixin.parse_bytes`\nbut that constructs a buffer to obtain the individual bytes;\nthis static method will be more performant\nif all you are doing is reading this serialisation\nand do not already have a buffer.\n\n*Method `BSUInt.parse_value(bfr)`*:\nParse an extensible byte serialised unsigned `int` from a buffer.\n\nContinuation octets have their high bit set.\nThe value is big-endian.\n\nThis is the go for reading from a stream. If you already have\na bare bytes instance then the `.decode_bytes` static method\nis probably most efficient;\nthere is of course the usual `BinaryMixin.parse_bytes`\nbut that constructs a buffer to obtain the individual bytes.\n\n*Method `BSUInt.transcribe_value(n)`*:\nEncode an unsigned int as an entensible byte serialised octet\nsequence for decode. Return the bytes object.\n\n## Function `flatten(chunks)`\n\nFlatten `chunks` into an iterable of `bytes` instances.\n\nThis exists to allow subclass methods to easily return\ntranscribeable things (having a `.transcribe` method), ASCII\nstrings or bytes or iterables or even `None`, in turn allowing\nthem simply to return their superclass' chunks iterators\ndirectly instead of having to unpack them.\n\nAn example from the `cs.iso14496.METABoxBody` class:\n\n def transcribe(self):\n yield super().transcribe()\n yield self.theHandler\n yield self.boxes\n\nThe binary classes `flatten` the result of the `.transcribe`\nmethod to obtain `bytes` insteances for the object's bnary\ntranscription.\n\n## Class `Float64BE(Float64BE, AbstractBinary)`\n\nAn `AbstractBinary` `namedtuple` which parses and transcribes\nthe struct format `'>d'` and presents the attributes ('value',).\n\n*Method `Float64BE.parse(bfr: cs.buffer.CornuCopyBuffer)`*:\nParse from `bfr` via `struct.unpack`.\n\n*Method `Float64BE.parse_value(bfr)`*:\nParse a value from `bfr`, return the value.\n\n*Method `Float64BE.transcribe(self)`*:\nTranscribe via `struct.pack`.\n\n*Method `Float64BE.transcribe_value(value)`*:\nTranscribe a value back into bytes.\n\n## Class `Float64LE(Float64LE, AbstractBinary)`\n\nAn `AbstractBinary` `namedtuple` which parses and transcribes\nthe struct format `'<d'` and presents the attributes ('value',).\n\n*Method `Float64LE.parse(bfr: cs.buffer.CornuCopyBuffer)`*:\nParse from `bfr` via `struct.unpack`.\n\n*Method `Float64LE.parse_value(bfr)`*:\nParse a value from `bfr`, return the value.\n\n*Method `Float64LE.transcribe(self)`*:\nTranscribe via `struct.pack`.\n\n*Method `Float64LE.transcribe_value(value)`*:\nTranscribe a value back into bytes.\n\n## Class `Int16BE(Int16BE, AbstractBinary)`\n\nAn `AbstractBinary` `namedtuple` which parses and transcribes\nthe struct format `'>h'` and presents the attributes ('value',).\n\n*Method `Int16BE.parse(bfr: cs.buffer.CornuCopyBuffer)`*:\nParse from `bfr` via `struct.unpack`.\n\n*Method `Int16BE.parse_value(bfr)`*:\nParse a value from `bfr`, return the value.\n\n*Method `Int16BE.transcribe(self)`*:\nTranscribe via `struct.pack`.\n\n*Method `Int16BE.transcribe_value(value)`*:\nTranscribe a value back into bytes.\n\n## Class `Int16LE(Int16LE, AbstractBinary)`\n\nAn `AbstractBinary` `namedtuple` which parses and transcribes\nthe struct format `'<h'` and presents the attributes ('value',).\n\n*Method `Int16LE.parse(bfr: cs.buffer.CornuCopyBuffer)`*:\nParse from `bfr` via `struct.unpack`.\n\n*Method `Int16LE.parse_value(bfr)`*:\nParse a value from `bfr`, return the value.\n\n*Method `Int16LE.transcribe(self)`*:\nTranscribe via `struct.pack`.\n\n*Method `Int16LE.transcribe_value(value)`*:\nTranscribe a value back into bytes.\n\n## Class `Int32BE(Int32BE, AbstractBinary)`\n\nAn `AbstractBinary` `namedtuple` which parses and transcribes\nthe struct format `'>l'` and presents the attributes ('value',).\n\n*Method `Int32BE.parse(bfr: cs.buffer.CornuCopyBuffer)`*:\nParse from `bfr` via `struct.unpack`.\n\n*Method `Int32BE.parse_value(bfr)`*:\nParse a value from `bfr`, return the value.\n\n*Method `Int32BE.transcribe(self)`*:\nTranscribe via `struct.pack`.\n\n*Method `Int32BE.transcribe_value(value)`*:\nTranscribe a value back into bytes.\n\n## Class `Int32LE(Int32LE, AbstractBinary)`\n\nAn `AbstractBinary` `namedtuple` which parses and transcribes\nthe struct format `'<l'` and presents the attributes ('value',).\n\n*Method `Int32LE.parse(bfr: cs.buffer.CornuCopyBuffer)`*:\nParse from `bfr` via `struct.unpack`.\n\n*Method `Int32LE.parse_value(bfr)`*:\nParse a value from `bfr`, return the value.\n\n*Method `Int32LE.transcribe(self)`*:\nTranscribe via `struct.pack`.\n\n*Method `Int32LE.transcribe_value(value)`*:\nTranscribe a value back into bytes.\n\n## Function `pt_spec(pt, name=None)`\n\nConvert a parse/transcribe specification `pt`\ninto an `AbstractBinary` subclass.\n\nThis is largely used to provide flexibility\nin the specifications for the `BinaryMultiValue` factory\nbut can be used as a factory for other simple classes.\n\nIf the specification `pt` is a subclass of `AbstractBinary`\nthis is returned directly.\n\nIf `pt` is a 2-tuple of `str`\nthe values are presumed to be a format string for `struct.struct`\nand filed names separated by spaces;\na new `BinaryMultiStruct` class is created from these and returned.\n\nOtherwise two functions\n`f_parse_value(bfr)` and `f_transcribe_value(value)`\nare obtained and used to construct a new `BinarySingleValue` class\nas follows:\n\nIf `pt` has `.parse_value` and `.transcribe_value` callable attributes,\nuse those for `f_parse_value` and `f_transcribe_value` respectively.\n\nOtherwise, if `pt` is an `int`\ndefine `f_parse_value` to obtain exactly that many bytes from a buffer\nand `f_transcribe_value` to return those bytes directly.\n\nOtherwise presume `pt` is a 2-tuple of `(f_parse_value,f_transcribe_value)`.\n\n## Class `SimpleBinary(types.SimpleNamespace, AbstractBinary)`\n\nAbstract binary class based on a `SimpleNamespace`,\nthus providing a nice `__str__` and a keyword based `__init__`.\nImplementors must still define `.parse` and `.transcribe`.\n\nTo constrain the arguments passed to `__init__`,\ndefine an `__init__` which accepts specific keyword arguments\nand pass through to `super().__init__()`. Example:\n\n def __init__(self, *, field1=None, field2):\n \"\"\" Accept only `field1` (optional)\n and `field2` (mandatory).\n \"\"\"\n super().__init__(field1=field1, field2=field2)\n\n## Class `UInt16BE(UInt16BE, AbstractBinary)`\n\nAn `AbstractBinary` `namedtuple` which parses and transcribes\nthe struct format `'>H'` and presents the attributes ('value',).\n\n*Method `UInt16BE.parse(bfr: cs.buffer.CornuCopyBuffer)`*:\nParse from `bfr` via `struct.unpack`.\n\n*Method `UInt16BE.parse_value(bfr)`*:\nParse a value from `bfr`, return the value.\n\n*Method `UInt16BE.transcribe(self)`*:\nTranscribe via `struct.pack`.\n\n*Method `UInt16BE.transcribe_value(value)`*:\nTranscribe a value back into bytes.\n\n## Class `UInt16LE(UInt16LE, AbstractBinary)`\n\nAn `AbstractBinary` `namedtuple` which parses and transcribes\nthe struct format `'<H'` and presents the attributes ('value',).\n\n*Method `UInt16LE.parse(bfr: cs.buffer.CornuCopyBuffer)`*:\nParse from `bfr` via `struct.unpack`.\n\n*Method `UInt16LE.parse_value(bfr)`*:\nParse a value from `bfr`, return the value.\n\n*Method `UInt16LE.transcribe(self)`*:\nTranscribe via `struct.pack`.\n\n*Method `UInt16LE.transcribe_value(value)`*:\nTranscribe a value back into bytes.\n\n## Class `UInt32BE(UInt32BE, AbstractBinary)`\n\nAn `AbstractBinary` `namedtuple` which parses and transcribes\nthe struct format `'>L'` and presents the attributes ('value',).\n\n*Method `UInt32BE.parse(bfr: cs.buffer.CornuCopyBuffer)`*:\nParse from `bfr` via `struct.unpack`.\n\n*Method `UInt32BE.parse_value(bfr)`*:\nParse a value from `bfr`, return the value.\n\n*Method `UInt32BE.transcribe(self)`*:\nTranscribe via `struct.pack`.\n\n*Method `UInt32BE.transcribe_value(value)`*:\nTranscribe a value back into bytes.\n\n## Class `UInt32LE(UInt32LE, AbstractBinary)`\n\nAn `AbstractBinary` `namedtuple` which parses and transcribes\nthe struct format `'<L'` and presents the attributes ('value',).\n\n*Method `UInt32LE.parse(bfr: cs.buffer.CornuCopyBuffer)`*:\nParse from `bfr` via `struct.unpack`.\n\n*Method `UInt32LE.parse_value(bfr)`*:\nParse a value from `bfr`, return the value.\n\n*Method `UInt32LE.transcribe(self)`*:\nTranscribe via `struct.pack`.\n\n*Method `UInt32LE.transcribe_value(value)`*:\nTranscribe a value back into bytes.\n\n## Class `UInt64BE(UInt64BE, AbstractBinary)`\n\nAn `AbstractBinary` `namedtuple` which parses and transcribes\nthe struct format `'>Q'` and presents the attributes ('value',).\n\n*Method `UInt64BE.parse(bfr: cs.buffer.CornuCopyBuffer)`*:\nParse from `bfr` via `struct.unpack`.\n\n*Method `UInt64BE.parse_value(bfr)`*:\nParse a value from `bfr`, return the value.\n\n*Method `UInt64BE.transcribe(self)`*:\nTranscribe via `struct.pack`.\n\n*Method `UInt64BE.transcribe_value(value)`*:\nTranscribe a value back into bytes.\n\n## Class `UInt64LE(UInt64LE, AbstractBinary)`\n\nAn `AbstractBinary` `namedtuple` which parses and transcribes\nthe struct format `'<Q'` and presents the attributes ('value',).\n\n*Method `UInt64LE.parse(bfr: cs.buffer.CornuCopyBuffer)`*:\nParse from `bfr` via `struct.unpack`.\n\n*Method `UInt64LE.parse_value(bfr)`*:\nParse a value from `bfr`, return the value.\n\n*Method `UInt64LE.transcribe(self)`*:\nTranscribe via `struct.pack`.\n\n*Method `UInt64LE.transcribe_value(value)`*:\nTranscribe a value back into bytes.\n\n## Class `UInt8(UInt8, AbstractBinary)`\n\nAn `AbstractBinary` `namedtuple` which parses and transcribes\nthe struct format `'B'` and presents the attributes ('value',).\n\n*Method `UInt8.parse(bfr: cs.buffer.CornuCopyBuffer)`*:\nParse from `bfr` via `struct.unpack`.\n\n*Method `UInt8.parse_value(bfr)`*:\nParse a value from `bfr`, return the value.\n\n*Method `UInt8.transcribe(self)`*:\nTranscribe via `struct.pack`.\n\n*Method `UInt8.transcribe_value(value)`*:\nTranscribe a value back into bytes.\n\n# Release Log\n\n\n\n*Release 20240422*:\nNew _BinaryMultiValue_Base.for_json() method returning a dict containing the fields.\n\n*Release 20240316*:\nFixed release upload artifacts.\n\n*Release 20240201*:\nBREAKING CHANGE: drop the long deprecated PacketField related classes.\n\n*Release 20231129*:\nBinaryMultiStruct.parse: promote the buffer arguments to a CornuCopyBuffer.\n\n*Release 20230401*:\n* BinaryMixin.scan: `bfr` parameter may be any object acceptable to CornuCopyBuffer.promote.\n* BinaryMixin.scan: accept new optional with_offsets parameter; deprecate scan_with_offsets and scan_fspathi in favour of scan.\n\n*Release 20230212*:\n* BinaryMixin: new load(file) and save(file) methods.\n* BinaryMixin.scan: promote the bfr argument.\n\n*Release 20221206*:\nDocumentation fix.\n\n*Release 20220605*:\nBinaryMixin: replace scan_file with scan_fspath, as the former left uncertainty about the amount of the file consumed.\n\n*Release 20210316*:\n* BSUInt: rename parse_bytes to decode_bytes, the former name conflicted with BinaryMixin.parse_bytes and broken the semantics.\n* Minor refactors.\n\n*Release 20210306*:\nMAJOR RELEASE: The PacketField classes and friends were hard to use; this release supplied a suite of easier to use and more consistent Binary* classes, and ports most of those things based on the old scheme to the new scheme.\n\n*Release 20200229*:\n* ListField: replace transcribe method with transcribe_value method, aids external use.\n* Add `.length` attribute to struct based packet classes providing the data length of the structure (struct.Struct.size).\n* Packet: new `add_deferred_field` method to consume the raw data for a field for parsing later (done automatically if the attribute is accessed).\n* New `@deferred_field` decorator for the parser for that stashed data.\n\n*Release 20191230.3*:\nDocstring tweak.\n\n*Release 20191230.2*:\nDocumentation updates.\n\n*Release 20191230.1*:\nDocstring updates. Semantic changes were in the previous release.\n\n*Release 20191230*:\n* ListField: new __iter__ method.\n* Packet: __str__: accept optional `skip_fields` parameter to omit some field names.\n* Packet: new .add_from_value method to add a named field with a presupplied value.\n* Packet: new remove_field(field_name) and pop_field() methods to remove fields.\n* BytesesField: __iter__ yields the bytes values, transcribe=__iter__.\n* PacketField: propagate keyword arguments through various methods, required for parameterised PacketFields.\n* New UTF16NULField, a NUL terminated UTF16 string.\n* PacketField: provide a default `.transcribe_value` method which makes a new instance and calls its `.transcribe` method.\n* Documentation update and several minor changes.\n\n*Release 20190220*:\n* Packet.self_check: fields without a sanity check cause a warning, not a ValueError.\n* New Float64BE, Float64LE and BSSFloat classes for IEEE floats and floats-as-strings.\n* Additional module docstringage on subclassing Packet and PacketField.\n* BSString: drop redundant from_buffer class method.\n* PacketField.__init__: default to value=None if omitted.\n\n*Release 20181231*:\nflatten: do not yield zero length bytelike objects, can be misread as EOF on some streams.\n\n*Release 20181108*:\n* New PacketField.transcribe_value_flat convenience method to return a flat iterable of bytes-like objects.\n* New PacketField.parse_buffer generator method to parse instances of the PacketField from a buffer until end of input.\n* New PacketField.parse_buffer_values generator method to parse instances of the PacketField from a buffer and yield the `.value` attribute until end of input.\n\n*Release 20180823*:\n* Some bugfixes.\n* Define PacketField.__eq__.\n* BSUInt, BSData and BSString classes implementing the serialisations from cs.serialise.\n* New PacketField.value_from_bytes class method.\n* New PacketField.value_from_buffer method.\n\n*Release 20180810.2*:\nDocumentation improvements.\n\n*Release 20180810.1*:\nImprove module description.\n\n*Release 20180810*:\nBytesesField.from_buffer: make use of the buffer's skipto method if discard_data is true.\n\n*Release 20180805*:\n* Packet: now an abstract class, new self_check method initially checking the\n* PACKET_FIELDS class attribute against the instance, new methods get_field\n* and set_field to fetch or replace existing fields, allow keyword arguments\n* to initialise the Packet fields and document the dependency on keyword\n* argument ordering.\n* PacketField: __len__ computed directory from a transcribe, drop other __len__\n* methods.\n* EmptyField singleton to use as a placeholder for missing optional fields.\n* BytesField: implement value_s and from_buffer.\n* multi_struct_field: implement __len__ for generated class.\n* flatten: treat memoryviews like bytes.\n* Assorted docstrings and fixes.\n\n*Release 20180801*:\nInitial PyPI release.\n\n",
"bugtrack_url": null,
"license": "GNU General Public License v3 or later (GPLv3+)",
"summary": "Facilities associated with binary data parsing and transcription. The classes in this module support easy parsing of binary data structures, returning instances with the binary data decoded into attributes and capable of transcribing themselves in binary form (trivially via `bytes(instance)` and also otherwise).",
"version": "20240422",
"project_urls": {
"URL": "https://bitbucket.org/cameron_simpson/css/commits/all"
},
"split_keywords": [
"python3"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "531c8f7643db3576b3d54f468af7f3bd38a159683b63bb19ad07a9cc9aa3acc4",
"md5": "968a07ea7fec9acdfb88d661a43843ac",
"sha256": "efd0a9be8cf95cb07695b6c526774464f0188b0af146b40ecb1e84f92fc97607"
},
"downloads": -1,
"filename": "cs.binary-20240422-py3-none-any.whl",
"has_sig": false,
"md5_digest": "968a07ea7fec9acdfb88d661a43843ac",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 24312,
"upload_time": "2024-04-22T06:31:03",
"upload_time_iso_8601": "2024-04-22T06:31:03.018582Z",
"url": "https://files.pythonhosted.org/packages/53/1c/8f7643db3576b3d54f468af7f3bd38a159683b63bb19ad07a9cc9aa3acc4/cs.binary-20240422-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "6564ca2a6141c5a6ed53bcaadbb7c0a7b02643017b4d4386e5d9e5e121f35db3",
"md5": "9aff0d54def1ebf735b024afe3001e3f",
"sha256": "e152907a487bba28feca58144ab6641032f46b79aac95aac5a5b49bb9a5bd95a"
},
"downloads": -1,
"filename": "cs.binary-20240422.tar.gz",
"has_sig": false,
"md5_digest": "9aff0d54def1ebf735b024afe3001e3f",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 48522,
"upload_time": "2024-04-22T06:31:05",
"upload_time_iso_8601": "2024-04-22T06:31:05.860511Z",
"url": "https://files.pythonhosted.org/packages/65/64/ca2a6141c5a6ed53bcaadbb7c0a7b02643017b4d4386e5d9e5e121f35db3/cs.binary-20240422.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-04-22 06:31:05",
"github": false,
"gitlab": false,
"bitbucket": true,
"codeberg": false,
"bitbucket_user": "cameron_simpson",
"bitbucket_project": "css",
"lcname": "cs.binary"
}
