| Name | cs.buffer JSON |
| Version |
20240412
JSON |
| download |
| home_page | None |
| Summary | Facilities to do with buffers, particularly CornuCopyBuffer, an automatically refilling buffer to support parsing of data streams. |
| upload_time | 2024-04-12 05:09:34 |
| maintainer | None |
| docs_url | None |
| author | None |
| requires_python | None |
| license | GNU General Public License v3 or later (GPLv3+) |
| keywords |
python3
|
| VCS |
|
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
Facilities to do with buffers, particularly CornuCopyBuffer,
an automatically refilling buffer to support parsing of data streams.
*Latest release 20240412*:
CornuCopyBuffer.buf: return b"" if nothing is buffered.
## Class `CopyingIterator`
Wrapper for an iterator that copies every item retrieved to a callable.
*Method `CopyingIterator.__init__(self, it, copy_to)`*:
Initialise with the iterator `it` and the callable `copy_to`.
## Class `CornuCopyBuffer(cs.deco.Promotable)`
An automatically refilling buffer intended to support parsing
of data streams.
Its purpose is to aid binary parsers
which do not themselves need to handle sources specially;
`CornuCopyBuffer`s are trivially made from `bytes`,
iterables of `bytes` and file-like objects.
See `cs.binary` for convenient parsing classes
which work against `CornuCopyBuffer`s.
Attributes:
* `buf`: the first of any buffered leading chunks
buffer of unparsed data from the input, available
for direct inspection by parsers;
normally however parsers will use `.extend` and `.take`.
* `offset`: the logical offset of the buffer; this excludes
buffered data and unconsumed input data
*Note*: the initialiser may supply a cleanup function;
although this will be called via the buffer's `.__del__` method
a prudent user of a buffer should call the `.close()` method
when finished with the buffer to ensure prompt cleanup.
The primary methods supporting parsing of data streams are
`.extend()` and `take()`.
Calling `.extend(min_size)` arranges that the internal buffer
contains at least `min_size` bytes.
Calling `.take(size)` fetches exactly `size` bytes from the
internal buffer and the input source if necessary and returns
them, adjusting the internal buffer.
len(`CornuCopyBuffer`) returns the length of any buffered data.
bool(`CornuCopyBuffer`) tests whether len() > 0.
Indexing a `CornuCopyBuffer` accesses the buffered data only,
returning an individual byte's value (an `int`).
A `CornuCopyBuffer` is also iterable, yielding data in whatever
sizes come from its `input_data` source, preceeded by any
content in the internal buffer.
A `CornuCopyBuffer` also supports the file methods `.read`,
`.tell` and `.seek` supporting drop in use of the buffer in
many file contexts. Backward seeks are not supported. `.seek`
will take advantage of the `input_data`'s .seek method if it
has one, otherwise it will use consume the `input_data`
as required.
*Method `CornuCopyBuffer.__init__(self, input_data, buf=None, offset=0, seekable=None, copy_offsets=None, copy_chunks=None, close=None, progress=None)`*:
Prepare the buffer.
Parameters:
* `input_data`: an iterable of data chunks (`bytes`-like instances);
if your data source is a file see the `.from_file` factory;
if your data source is a file descriptor see the `.from_fd`
factory.
* `buf`: if not `None`, the initial state of the parse buffer
* `offset`: logical offset of the start of the buffer, default `0`
* `seekable`: whether `input_data` has a working `.seek` method;
the default is `None` meaning that it will be attempted on
the first skip or seek
* `copy_offsets`: if not `None`, a callable for parsers to
report pertinent offsets via the buffer's `.report_offset`
method
* `copy_chunks`: if not `None`, every fetched data chunk is
copied to this callable
The `input_data` is an iterable whose iterator may have
some optional additional properties:
* `seek`: if present, this is a seek method after the fashion
of `file.seek`; the buffer's `seek`, `skip` and `skipto`
methods will take advantage of this if available.
* `offset`: the current byte offset of the iterator; this
is used during the buffer initialisation to compute
`input_data_displacement`, the difference between the
buffer's logical offset and the input data iterable's logical offset;
if unavailable during initialisation this is presumed to
be `0`.
* `end_offset`: the end offset of the iterator if known.
* `close`: an optional callable
that may be provided for resource cleanup
when the user of the buffer calls its `.close()` method.
* `progress`: an optional `cs.Progress.progress` instance
to which to report data consumed from `input_data`;
any object supporting `+=` is acceptable
*Method `CornuCopyBuffer.__del__(self)`*:
Release resources when the object is deleted.
*Method `CornuCopyBuffer.__getitem__(self, index)`*:
Fetch from the internal buffer.
This does not consume data from the internal buffer.
Note that this is an expensive way to access the buffer,
particularly if `index` is a slice.
If `index` is a `slice`, slice the join of the internal subbuffers.
This is quite expensive
and it is probably better to `take` or `takev`
some data from the buffer.
Otherwise `index` should be an `int` and the corresponding
buffered byte is returned.
This is usually not a very useful method;
its primary use case is to probe the buffer to make a parsing decision
instead of taking a byte off and (possibly) pushing it back.
*Method `CornuCopyBuffer.__len__(self)`*:
The length is the length of the internal buffer: data available without a fetch.
*Method `CornuCopyBuffer.__next__(self)`*:
Fetch a data chunk from the buffer.
*Method `CornuCopyBuffer.as_fd(self, maxlength=Ellipsis)`*:
Create a pipe and dispatch a `Thread` to copy
up to `maxlength` bytes from `bfr` into it.
Return the file descriptor of the read end of the pipe.
The default `maxlength` is `Ellipsis`, meaning to copy all data.
Note that the thread preemptively consumes from the buffer.
This is useful for passing buffer data to subprocesses.
*Method `CornuCopyBuffer.at_eof(self)`*:
Test whether the buffer is at end of input.
*Warning*: this will fetch from the `input_data` if the buffer
is empty and so it may block.
*Method `CornuCopyBuffer.bounded(self, end_offset)`*:
Return a new `CornuCopyBuffer` operating on a bounded view
of this buffer.
This supports parsing of the buffer contents without risk
of consuming past a certain point, such as the known end
of a packet structure.
Parameters:
* `end_offset`: the ending offset of the new buffer.
Note that this is an absolute offset, not a length.
The new buffer starts with the same offset as `self` and
use of the new buffer affects `self`. After a flush both
buffers will again have the same offset and the data consumed
via the new buffer will also have been consumed from `self`.
Here is an example.
* Make a buffer `bfr` with 9 bytes of data in 3 chunks.
* Consume 2 bytes, advancing the offset to 2.
* Make a new bounded buffer `subbfr` extending to offset
5. Its inital offset is also 2.
* Iterate over it, yielding the remaining single byte chunk
from ``b'abc'`` and then the first 2 bytes of ``b'def'``.
The new buffer's offset is now 5.
* Try to take 2 more bytes from the new buffer - this fails.
* Flush the new buffer, synchronising with the original.
The original's offset is now also 5.
* Take 2 bytes from the original buffer, which succeeds.
Example:
>>> bfr = CornuCopyBuffer([b'abc', b'def', b'ghi'])
>>> bfr.offset
0
>>> bfr.take(2)
b'ab'
>>> bfr.offset
2
>>> subbfr = bfr.bounded(5)
>>> subbfr.offset
2
>>> for bs in subbfr:
... print(bs)
...
b'c'
b'de'
>>> subbfr.offset
5
>>> subbfr.take(2)
Traceback (most recent call last):
...
EOFError: insufficient input data, wanted 2 bytes but only found 0
>>> subbfr.flush()
>>> bfr.offset
5
>>> bfr.take(2)
b'fg'
*WARNING*: if the bounded buffer is not completely consumed
then it is critical to call the new `CornuCopyBuffer`'s `.flush`
method to push any unconsumed buffer back into this buffer.
Recommended practice is to always call `.flush` when finished
with the new buffer.
The `CornuCopyBuffer.subbuffer` method returns a context manager
which does this automatically.
Also, because the new buffer may buffer some of the unconsumed
data from this buffer, use of the original buffer should
be suspended.
*Property `CornuCopyBuffer.buf`*:
The first buffer, or `b''` if nothing is buffered.
*Method `CornuCopyBuffer.byte0(self)`*:
Consume the leading byte and return it as an `int` (`0`..`255`).
*Method `CornuCopyBuffer.close(self)`*:
Close the buffer.
This calls the `close` callable supplied
when the buffer was initialised, if any,
in order to release resources such as open file descriptors.
The callable will be called only on the first `close()` call.
*Note*: this does *not* prevent subsequent reads or iteration
from the buffer; it is only for resource cleanup,
though that cleanup might itself break iteration.
*Property `CornuCopyBuffer.end_offset`*:
Return the end offset of the input data (in buffer ordinates)
if known, otherwise `None`.
Note that this depends on the computation of the
`input_offset_displacement` which takes place at the buffer
initialisation, which in turn relies on the `input_data.offset`
attribute, which at initialisation is presumed to be 0 if missing.
*Method `CornuCopyBuffer.extend(self, min_size, short_ok=False)`*:
Extend the buffer to at least `min_size` bytes.
If `min_size` is `Ellipsis`, extend the buffer to consume all the input.
This should really only be used with bounded buffers
in order to avoid unconstrained memory consumption.
If there are insufficient data available then an `EOFError`
will be raised unless `short_ok` is true (default `False`)
in which case the updated buffer will be short.
*Method `CornuCopyBuffer.from_bytes(bs, offset=0, length=None, **kw)`*:
Return a `CornuCopyBuffer` fed from the supplied bytes `bs`
starting at `offset` and ending after `length`.
This is handy for callers parsing using buffers but handed bytes.
Parameters:
* `bs`: the bytes
* `offset`: a starting position for the data; the input
data will start this far into the bytes
* `length`: the maximium number of bytes to use; the input
data will be cropped this far past the starting point;
default: the number of bytes in `bs` after `offset`
Other keyword arguments are passed to the buffer constructor.
*Method `CornuCopyBuffer.from_fd(fd, readsize=None, offset=None, **kw)`*:
Return a new `CornuCopyBuffer` attached to an open file descriptor.
Internally this constructs a `SeekableFDIterator` for regular
files or an `FDIterator` for other files, which provides the
iteration that `CornuCopyBuffer` consumes, but also seek
support if the underlying file descriptor is seekable.
*Note*: a `SeekableFDIterator` makes an `os.dup` of the
supplied file descriptor, so the caller is responsible for
closing the original.
Parameters:
* `fd`: the operating system file descriptor
* `readsize`: an optional preferred read size
* `offset`: a starting position for the data; the file
descriptor will seek to this offset, and the buffer will
start with this offset
Other keyword arguments are passed to the buffer constructor.
*Method `CornuCopyBuffer.from_file(f, readsize=None, offset=None, **kw)`*:
Return a new `CornuCopyBuffer` attached to an open file.
Internally this constructs a `SeekableFileIterator`, which
provides the iteration that `CornuCopyBuffer` consumes
and also seek support if the underlying file is seekable.
Parameters:
* `f`: the file like object
* `readsize`: an optional preferred read size
* `offset`: a starting position for the data; the file
will seek to this offset, and the buffer will start with this
offset
Other keyword arguments are passed to the buffer constructor.
*Method `CornuCopyBuffer.from_filename(filename: str, offset=None, **kw)`*:
Open the file named `filename` and return a new `CornuCopyBuffer`.
If `offset` is provided, skip to that position in the file.
A negative offset skips to a position that far from the end of the file
as determined by its `Stat.st_size`.
Other keyword arguments are passed to the buffer constructor.
*Method `CornuCopyBuffer.from_mmap(fd, readsize=None, offset=None, **kw)`*:
Return a new `CornuCopyBuffer` attached to an mmap of an open
file descriptor.
Internally this constructs a `SeekableMMapIterator`, which
provides the iteration that `CornuCopyBuffer` consumes, but
also seek support.
*Note*: a `SeekableMMapIterator` makes an `os.dup` of the
supplied file descriptor, so the caller is responsible for
closing the original.
Parameters:
* `fd`: the operating system file descriptor
* `readsize`: an optional preferred read size
* `offset`: a starting position for the data; the file
descriptor will seek to this offset, and the buffer will
start with this offset
Other keyword arguments are passed to the buffer constructor.
*Method `CornuCopyBuffer.hint(self, size)`*:
Hint that the caller is seeking at least `size` bytes.
If the `input_data` iterator has a `hint` method, this is
passed to it.
*Method `CornuCopyBuffer.iter(self, maxlength)`*:
Yield chunks from the buffer
up to `maxlength` in total
or until EOF if `maxlength` is `Ellipsis`.
*Method `CornuCopyBuffer.next(self)`*:
Fetch a data chunk from the buffer.
*Method `CornuCopyBuffer.peek(self, size, short_ok=False)`*:
Examine the leading bytes of the buffer without consuming them,
a `take` followed by a `push`.
Returns the bytes.
*Method `CornuCopyBuffer.promote(obj)`*:
Promote `obj` to a `CornuCopyBuffer`,
used by the @cs.deco.promote` decorator.
Promotes:
* `int`: assumed to be a file descriptor of a file open for binary read
* `str`: assumed to be a filesystem pathname
* `bytes` and `bytes`like objects: data
* has a `.read1` or `.read` method: assume a file open for binary read
* iterable: assumed to be an iterable of `bytes`like objects
*Method `CornuCopyBuffer.push(self, bs)`*:
Push the chunk `bs` onto the front of the buffered data.
Rewinds the logical `.offset` by the length of `bs`.
*Method `CornuCopyBuffer.read(self, size, one_fetch=False)`*:
Compatibility method to allow using the buffer like a file.
Parameters:
* `size`: the desired data size
* `one_fetch`: do a single data fetch, default `False`
In `one_fetch` mode the read behaves like a POSIX file read,
returning up to to `size` bytes from a single I/O operation.
*Method `CornuCopyBuffer.read1(self, size)`*:
Shorthand method for `self.read(size,one_fetch=True)`.
*Method `CornuCopyBuffer.readline(self)`*:
Return a binary "line" from `self`, where a line is defined by
its ending `b'
'` delimiter.
The final line from a buffer might not have a trailing newline;
`b''` is returned at EOF.
Example:
>>> bfr = CornuCopyBuffer([b'abc', b'def
hij'])
>>> bfr.readline()
b'abcdef
'
>>> bfr.readline()
b'hij'
>>> bfr.readline()
b''
>>> bfr.readline()
b''
*Method `CornuCopyBuffer.report_offset(self, offset)`*:
Report a pertinent offset.
*Method `CornuCopyBuffer.seek(self, offset, whence=None, short_ok=False)`*:
Compatibility method to allow using the buffer like a file.
This returns the resulting absolute offset.
Parameters are as for `io.seek` except as noted below:
* `whence`: (default `os.SEEK_SET`). This method only supports
`os.SEEK_SET` and `os.SEEK_CUR`, and does not support seeking to a
lower offset than the current buffer offset.
* `short_ok`: (default `False`). If true, the seek may not reach
the target if there are insufficent `input_data` - the
position will be the end of the `input_data`, and the
`input_data` will have been consumed; the caller must check
the returned offset to check that it is as expected. If
false, a `ValueError` will be raised; however, note that the
`input_data` will still have been consumed.
*Method `CornuCopyBuffer.selfcheck(self, msg='')`*:
Integrity check for the buffer, useful during debugging.
*Method `CornuCopyBuffer.skip(self, toskip, copy_skip=None, short_ok=False)`*:
Advance position by `skip_to`. Return the new offset.
Parameters:
* `toskip`: the distance to advance
* `copy_skip`: callable to receive skipped data.
* `short_ok`: default `False`; if true then skip may return before
`skipto` bytes if there are insufficient `input_data`.
*Method `CornuCopyBuffer.skipto(self, new_offset, copy_skip=None, short_ok=False)`*:
Advance to position `new_offset`. Return the new offset.
Parameters:
* `new_offset`: the target offset.
* `copy_skip`: callable to receive skipped data.
* `short_ok`: default `False`; if true then skipto may return before
`new_offset` if there are insufficient `input_data`.
Return values:
* `buf`: the new state of `buf`
* `offset`: the final offset; this may be short if `short_ok`.
*Method `CornuCopyBuffer.subbuffer(self, end_offset)`*:
Context manager wrapper for `.bounded`
which calls the `.flush` method automatically
on exiting the context.
Example:
# avoid buffer overrun
with bfr.subbuffer(bfr.offset+128) as subbfr:
id3v1 = ID3V1Frame.parse(subbfr)
# ensure the whole buffer was consumed
assert subbfr.at_eof()
*Method `CornuCopyBuffer.tail_extend(self, size)`*:
Extend method for parsers reading "tail"-like chunk streams,
typically raw reads from a growing file.
This may read 0 bytes at EOF, but a future read may read
more bytes if the file grows.
Such an iterator can be obtained from
``cs.fileutils.read_from(..,tail_mode=True)``.
*Method `CornuCopyBuffer.take(self, size, short_ok=False)`*:
Return the next `size` bytes.
Other arguments are as for `.extend()`.
This is a thin wrapper for the `.takev` method.
*Method `CornuCopyBuffer.takev(self, size, short_ok=False)`*:
Return the next `size` bytes as a list of chunks
(because the internal buffering is also a list of chunks).
Other arguments are as for extend().
See `.take()` to get a flat chunk instead of a list.
*Method `CornuCopyBuffer.tell(self)`*:
Compatibility method to allow using the buffer like a file.
## Class `FDIterator(_Iterator)`
An iterator over the data of a file descriptor.
*Note*: the iterator works with an os.dup() of the file
descriptor so that it can close it with impunity; this requires
the caller to close their descriptor.
*Method `FDIterator.__init__(self, fd, offset=None, readsize=None, align=True)`*:
Initialise the iterator.
Parameters:
* `fd`: file descriptor
* `offset`: the initial logical offset, kept up to date by
iteration; the default is the current file position.
* `readsize`: a preferred read size; if omitted then
`DEFAULT_READSIZE` will be stored
* `align`: whether to align reads by default: if true then
the iterator will do a short read to bring the `offset`
into alignment with `readsize`; the default is `True`
*Method `FDIterator.__del__(self)`*:
Close the file descriptor.
*Method `FDIterator.close(self)`*:
Close the file descriptor.
## Class `FileIterator(_Iterator, SeekableIteratorMixin)`
An iterator over the data of a file object.
*Note*: the iterator closes the file on `__del__` or if its
`.close` method is called.
*Method `FileIterator.__init__(self, fp, offset=None, readsize=None, align=False)`*:
Initialise the iterator.
Parameters:
* `fp`: file object
* `offset`: the initial logical offset, kept up to date by
iteration; the default is 0.
* `readsize`: a preferred read size; if omitted then
`DEFAULT_READSIZE` will be stored
* `align`: whether to align reads by default: if true then
the iterator will do a short read to bring the `offset`
into alignment with `readsize`; the default is `False`
*Method `FileIterator.close(self)`*:
Detach from the file. Does *not* call `fp.close()`.
## Class `SeekableFDIterator(FDIterator, SeekableIteratorMixin)`
An iterator over the data of a seekable file descriptor.
*Note*: the iterator works with an `os.dup()` of the file
descriptor so that it can close it with impunity; this requires
the caller to close their descriptor.
*Property `SeekableFDIterator.end_offset`*:
The end offset of the file.
## Class `SeekableFileIterator(FileIterator)`
An iterator over the data of a seekable file object.
*Note*: the iterator closes the file on __del__ or if its
.close method is called.
*Method `SeekableFileIterator.__init__(self, fp, offset=None, **kw)`*:
Initialise the iterator.
Parameters:
* `fp`: file object
* `offset`: the initial logical offset, kept up to date by
iteration; the default is the current file position.
* `readsize`: a preferred read size; if omitted then
`DEFAULT_READSIZE` will be stored
* `align`: whether to align reads by default: if true then
the iterator will do a short read to bring the `offset`
into alignment with `readsize`; the default is `False`
*Method `SeekableFileIterator.seek(self, new_offset, mode=0)`*:
Move the logical file pointer.
WARNING: moves the underlying file's pointer.
## Class `SeekableIteratorMixin`
Mixin supplying a logical with a `seek` method.
*Method `SeekableIteratorMixin.seek(self, new_offset, mode=0)`*:
Move the logical offset.
## Class `SeekableMMapIterator(_Iterator, SeekableIteratorMixin)`
An iterator over the data of a mappable file descriptor.
*Note*: the iterator works with an `mmap` of an `os.dup()` of the
file descriptor so that it can close it with impunity; this
requires the caller to close their descriptor.
*Method `SeekableMMapIterator.__init__(self, fd, offset=None, readsize=None, align=True)`*:
Initialise the iterator.
Parameters:
* `offset`: the initial logical offset, kept up to date by
iteration; the default is the current file position.
* `readsize`: a preferred read size; if omitted then
`DEFAULT_READSIZE` will be stored
* `align`: whether to align reads by default: if true then
the iterator will do a short read to bring the `offset`
into alignment with `readsize`; the default is `True`
*Method `SeekableMMapIterator.close(self)`*:
Detach from the file descriptor and mmap and close.
*Property `SeekableMMapIterator.end_offset`*:
The end offset of the mmap memoryview.
# Release Log
*Release 20240412*:
CornuCopyBuffer.buf: return b"" if nothing is buffered.
*Release 20240316*:
Fixed release upload artifacts.
*Release 20240201*:
CornuCopyBuffer: read1() method shorthand for read(..,one_fetch=True).
*Release 20230401*:
* CornuCopyBuffer.promote: document accepting an iterable as an iterable of bytes.
* CornuCopyBuffer: new readline() method to return a binary line from the buffer.
* CornuCopyBuffer.promote: assume objects with a .read1 or .read are files.
*Release 20230212.2*:
* BREAKING: drop @chunky, superceded by @promote for CornuCopyBuffer parameters.
* CornuCopyBuffer: subclass Promotable.
*Release 20230212.1*:
Add missing requirement for cs.gimmicks.
*Release 20230212*:
CornuCopyBuffer: new promote() method to promote int,str,bytes, also assumes nonspecific iterables yield byteslike instances.
*Release 20211208*:
CornuCopyBuffer.__init__: bugfix for self.input_data when copy_chunks is not None.
*Release 20210316*:
* New CornuCopyBuffer.from_filename factory method.
* New CornuCopyBuffer.peek method to examine the leading bytes from the buffer.
*Release 20210306*:
* CornuCopyBuffer.from_file: improve the seekability test, handle files with no .tell().
* CornuCopyBuffer.from_bytes: bugfix: set the buffer.offset to the supplied offset.
* CornuCopyBuffer.from_bytes: queue a memoryview of the supplied bytes.
*Release 20201102*:
CornuCopyBuffer: new optional `progress` parameter for reporting data consumed from `input_data`.
*Release 20201021*:
* CornuCopyBuffer.from_file: changes to the test for a .seek method.
* CornuCopyBuffer.read: call extend with short_ok=True.
* CornuCopyBuffer.from_fd: record the fd as .fd, lets users os.fstat(bfr.fd).
* New CornuCopyBuffer.as_fd method to return a readable file descriptor fed from the buffer by a Thread, intended for feeding subprocesses.
* New CornuCopyBuffer.iter(maxlength) to return an iterator of up to maxlength bytes.
* CornuCopyBuffer.__init__: new "close" parameter to release resources; new CornuCopyBuffer.close method to call this.
* Some small fixes.
*Release 20200517*:
* CornuCopyBuffer.skip: bugfix sanity check.
* FileIterator: do not close the supplied file, just set self.fp=None.
* Improve EOFError message text.
*Release 20200328*:
* CornuCopyBuffer.takev: bugfix adjustment of buf.offset, was not always done.
* CornuCopyBuffer.__getitem__: add slice support, note how expensive it is to use.
*Release 20200229*:
* New CornuCopyBuffer.byte0() method consuming the next byte and returning it as an int.
* CornuCopyBuffer.takev: bugfix for size=0, logic refactor.
* CornuCopyBuffer: new .selfcheck method.
*Release 20200130*:
CornuCopyBuffer.skip: bugfix adjustment of skipto for already buffered data.
*Release 20191230.1*:
Docstring updates. Semantic changes were in the previous release.
*Release 20191230*:
* CornuCopyBuffer: accept a size of Ellipsis in .take and .extend methods, indicating "all the remaining data".
* CornuCopyBuffer: refactor the buffering, replacing .buf with .bufs as an array of chunks;
* this enables support for the new .push method and reduces memory copying.
*Release 20181231*:
Small bugfix.
*Release 20181108*:
New at_eof() method. Python 2 tweak to support incidental import by python 2 even if unused.
*Release 20180823*:
Better handling of seekable and unseekable input data. Tiny bugfix for from_bytes sanity check.
*Release 20180810*:
* Refactor SeekableFDIterator and SeekableFileIterator to subclass new SeekableIterator.
* New SeekableMMapIterator to process a memory mapped file descriptor, intended for large files.
* New CornuCopyBuffer.hint method to pass a length hint through to the input_data iterator
* if it has a `hint` method, causing it possibly to make a differently sized fetch.
* SeekableIterator: new __del__ method calling self.close() - subclasses must provide
* a .close, which should be safe to call multiple times.
* CornuCopyBuffer: add support for .offset and .end_offset optional attributes on the input_data iterator.
* _BoundedBufferIterator: add .offset property plumbed to the underlying buffer offset.
* New CornuCopyBuffer.from_mmap to make a mmap backed buffer so that large data can be returned without penalty.
* Assorted fixes and doc improvements.
*Release 20180805*:
Bugfixes for at_eof method and end_offset initialisation.
*Release 20180726.1*:
Improve docstrings and release with better long_description.
*Release 20180726*:
First PyPI release: CornuCopyBuffer and friends.
Raw data
{
"_id": null,
"home_page": null,
"name": "cs.buffer",
"maintainer": null,
"docs_url": null,
"requires_python": null,
"maintainer_email": null,
"keywords": "python3",
"author": null,
"author_email": "Cameron Simpson <cs@cskk.id.au>",
"download_url": "https://files.pythonhosted.org/packages/2a/e3/b965cd9e092cea85831c1762c1c5dcc524936fafa3ad0c84006ddd198ccf/cs.buffer-20240412.tar.gz",
"platform": null,
"description": "Facilities to do with buffers, particularly CornuCopyBuffer,\nan automatically refilling buffer to support parsing of data streams.\n\n*Latest release 20240412*:\nCornuCopyBuffer.buf: return b\"\" if nothing is buffered.\n\n## Class `CopyingIterator`\n\nWrapper for an iterator that copies every item retrieved to a callable.\n\n*Method `CopyingIterator.__init__(self, it, copy_to)`*:\nInitialise with the iterator `it` and the callable `copy_to`.\n\n## Class `CornuCopyBuffer(cs.deco.Promotable)`\n\nAn automatically refilling buffer intended to support parsing\nof data streams.\n\nIts purpose is to aid binary parsers\nwhich do not themselves need to handle sources specially;\n`CornuCopyBuffer`s are trivially made from `bytes`,\niterables of `bytes` and file-like objects.\nSee `cs.binary` for convenient parsing classes\nwhich work against `CornuCopyBuffer`s.\n\nAttributes:\n* `buf`: the first of any buffered leading chunks\n buffer of unparsed data from the input, available\n for direct inspection by parsers;\n normally however parsers will use `.extend` and `.take`.\n* `offset`: the logical offset of the buffer; this excludes\n buffered data and unconsumed input data\n\n*Note*: the initialiser may supply a cleanup function;\nalthough this will be called via the buffer's `.__del__` method\na prudent user of a buffer should call the `.close()` method\nwhen finished with the buffer to ensure prompt cleanup.\n\nThe primary methods supporting parsing of data streams are\n`.extend()` and `take()`.\nCalling `.extend(min_size)` arranges that the internal buffer\ncontains at least `min_size` bytes.\nCalling `.take(size)` fetches exactly `size` bytes from the\ninternal buffer and the input source if necessary and returns\nthem, adjusting the internal buffer.\n\nlen(`CornuCopyBuffer`) returns the length of any buffered data.\n\nbool(`CornuCopyBuffer`) tests whether len() > 0.\n\nIndexing a `CornuCopyBuffer` accesses the buffered data only,\nreturning an individual byte's value (an `int`).\n\nA `CornuCopyBuffer` is also iterable, yielding data in whatever\nsizes come from its `input_data` source, preceeded by any\ncontent in the internal buffer.\n\nA `CornuCopyBuffer` also supports the file methods `.read`,\n`.tell` and `.seek` supporting drop in use of the buffer in\nmany file contexts. Backward seeks are not supported. `.seek`\nwill take advantage of the `input_data`'s .seek method if it\nhas one, otherwise it will use consume the `input_data`\nas required.\n\n*Method `CornuCopyBuffer.__init__(self, input_data, buf=None, offset=0, seekable=None, copy_offsets=None, copy_chunks=None, close=None, progress=None)`*:\nPrepare the buffer.\n\nParameters:\n* `input_data`: an iterable of data chunks (`bytes`-like instances);\n if your data source is a file see the `.from_file` factory;\n if your data source is a file descriptor see the `.from_fd`\n factory.\n* `buf`: if not `None`, the initial state of the parse buffer\n* `offset`: logical offset of the start of the buffer, default `0`\n* `seekable`: whether `input_data` has a working `.seek` method;\n the default is `None` meaning that it will be attempted on\n the first skip or seek\n* `copy_offsets`: if not `None`, a callable for parsers to\n report pertinent offsets via the buffer's `.report_offset`\n method\n* `copy_chunks`: if not `None`, every fetched data chunk is\n copied to this callable\n\nThe `input_data` is an iterable whose iterator may have\nsome optional additional properties:\n* `seek`: if present, this is a seek method after the fashion\n of `file.seek`; the buffer's `seek`, `skip` and `skipto`\n methods will take advantage of this if available.\n* `offset`: the current byte offset of the iterator; this\n is used during the buffer initialisation to compute\n `input_data_displacement`, the difference between the\n buffer's logical offset and the input data iterable's logical offset;\n if unavailable during initialisation this is presumed to\n be `0`.\n* `end_offset`: the end offset of the iterator if known.\n* `close`: an optional callable\n that may be provided for resource cleanup\n when the user of the buffer calls its `.close()` method.\n* `progress`: an optional `cs.Progress.progress` instance\n to which to report data consumed from `input_data`;\n any object supporting `+=` is acceptable\n\n*Method `CornuCopyBuffer.__del__(self)`*:\nRelease resources when the object is deleted.\n\n*Method `CornuCopyBuffer.__getitem__(self, index)`*:\nFetch from the internal buffer.\nThis does not consume data from the internal buffer.\nNote that this is an expensive way to access the buffer,\nparticularly if `index` is a slice.\n\nIf `index` is a `slice`, slice the join of the internal subbuffers.\nThis is quite expensive\nand it is probably better to `take` or `takev`\nsome data from the buffer.\n\nOtherwise `index` should be an `int` and the corresponding\nbuffered byte is returned.\n\nThis is usually not a very useful method;\nits primary use case is to probe the buffer to make a parsing decision\ninstead of taking a byte off and (possibly) pushing it back.\n\n*Method `CornuCopyBuffer.__len__(self)`*:\nThe length is the length of the internal buffer: data available without a fetch.\n\n*Method `CornuCopyBuffer.__next__(self)`*:\nFetch a data chunk from the buffer.\n\n*Method `CornuCopyBuffer.as_fd(self, maxlength=Ellipsis)`*:\nCreate a pipe and dispatch a `Thread` to copy\nup to `maxlength` bytes from `bfr` into it.\nReturn the file descriptor of the read end of the pipe.\n\nThe default `maxlength` is `Ellipsis`, meaning to copy all data.\n\nNote that the thread preemptively consumes from the buffer.\n\nThis is useful for passing buffer data to subprocesses.\n\n*Method `CornuCopyBuffer.at_eof(self)`*:\nTest whether the buffer is at end of input.\n\n*Warning*: this will fetch from the `input_data` if the buffer\nis empty and so it may block.\n\n*Method `CornuCopyBuffer.bounded(self, end_offset)`*:\nReturn a new `CornuCopyBuffer` operating on a bounded view\nof this buffer.\n\nThis supports parsing of the buffer contents without risk\nof consuming past a certain point, such as the known end\nof a packet structure.\n\nParameters:\n* `end_offset`: the ending offset of the new buffer.\n Note that this is an absolute offset, not a length.\n\nThe new buffer starts with the same offset as `self` and\nuse of the new buffer affects `self`. After a flush both\nbuffers will again have the same offset and the data consumed\nvia the new buffer will also have been consumed from `self`.\n\nHere is an example.\n* Make a buffer `bfr` with 9 bytes of data in 3 chunks.\n* Consume 2 bytes, advancing the offset to 2.\n* Make a new bounded buffer `subbfr` extending to offset\n 5. Its inital offset is also 2.\n* Iterate over it, yielding the remaining single byte chunk\n from ``b'abc'`` and then the first 2 bytes of ``b'def'``.\n The new buffer's offset is now 5.\n* Try to take 2 more bytes from the new buffer - this fails.\n* Flush the new buffer, synchronising with the original.\n The original's offset is now also 5.\n* Take 2 bytes from the original buffer, which succeeds.\n\nExample:\n\n >>> bfr = CornuCopyBuffer([b'abc', b'def', b'ghi'])\n >>> bfr.offset\n 0\n >>> bfr.take(2)\n b'ab'\n >>> bfr.offset\n 2\n >>> subbfr = bfr.bounded(5)\n >>> subbfr.offset\n 2\n >>> for bs in subbfr:\n ... print(bs)\n ...\n b'c'\n b'de'\n >>> subbfr.offset\n 5\n >>> subbfr.take(2)\n Traceback (most recent call last):\n ...\n EOFError: insufficient input data, wanted 2 bytes but only found 0\n >>> subbfr.flush()\n >>> bfr.offset\n 5\n >>> bfr.take(2)\n b'fg'\n\n*WARNING*: if the bounded buffer is not completely consumed\nthen it is critical to call the new `CornuCopyBuffer`'s `.flush`\nmethod to push any unconsumed buffer back into this buffer.\nRecommended practice is to always call `.flush` when finished\nwith the new buffer.\nThe `CornuCopyBuffer.subbuffer` method returns a context manager\nwhich does this automatically.\n\nAlso, because the new buffer may buffer some of the unconsumed\ndata from this buffer, use of the original buffer should\nbe suspended.\n\n*Property `CornuCopyBuffer.buf`*:\nThe first buffer, or `b''` if nothing is buffered.\n\n*Method `CornuCopyBuffer.byte0(self)`*:\nConsume the leading byte and return it as an `int` (`0`..`255`).\n\n*Method `CornuCopyBuffer.close(self)`*:\nClose the buffer.\nThis calls the `close` callable supplied\nwhen the buffer was initialised, if any,\nin order to release resources such as open file descriptors.\nThe callable will be called only on the first `close()` call.\n\n*Note*: this does *not* prevent subsequent reads or iteration\nfrom the buffer; it is only for resource cleanup,\nthough that cleanup might itself break iteration.\n\n*Property `CornuCopyBuffer.end_offset`*:\nReturn the end offset of the input data (in buffer ordinates)\nif known, otherwise `None`.\n\nNote that this depends on the computation of the\n`input_offset_displacement` which takes place at the buffer\ninitialisation, which in turn relies on the `input_data.offset`\nattribute, which at initialisation is presumed to be 0 if missing.\n\n*Method `CornuCopyBuffer.extend(self, min_size, short_ok=False)`*:\nExtend the buffer to at least `min_size` bytes.\n\nIf `min_size` is `Ellipsis`, extend the buffer to consume all the input.\nThis should really only be used with bounded buffers\nin order to avoid unconstrained memory consumption.\n\nIf there are insufficient data available then an `EOFError`\nwill be raised unless `short_ok` is true (default `False`)\nin which case the updated buffer will be short.\n\n*Method `CornuCopyBuffer.from_bytes(bs, offset=0, length=None, **kw)`*:\nReturn a `CornuCopyBuffer` fed from the supplied bytes `bs`\nstarting at `offset` and ending after `length`.\n\nThis is handy for callers parsing using buffers but handed bytes.\n\nParameters:\n* `bs`: the bytes\n* `offset`: a starting position for the data; the input\n data will start this far into the bytes\n* `length`: the maximium number of bytes to use; the input\n data will be cropped this far past the starting point;\n default: the number of bytes in `bs` after `offset`\nOther keyword arguments are passed to the buffer constructor.\n\n*Method `CornuCopyBuffer.from_fd(fd, readsize=None, offset=None, **kw)`*:\nReturn a new `CornuCopyBuffer` attached to an open file descriptor.\n\nInternally this constructs a `SeekableFDIterator` for regular\nfiles or an `FDIterator` for other files, which provides the\niteration that `CornuCopyBuffer` consumes, but also seek\nsupport if the underlying file descriptor is seekable.\n\n*Note*: a `SeekableFDIterator` makes an `os.dup` of the\nsupplied file descriptor, so the caller is responsible for\nclosing the original.\n\nParameters:\n* `fd`: the operating system file descriptor\n* `readsize`: an optional preferred read size\n* `offset`: a starting position for the data; the file\n descriptor will seek to this offset, and the buffer will\n start with this offset\nOther keyword arguments are passed to the buffer constructor.\n\n*Method `CornuCopyBuffer.from_file(f, readsize=None, offset=None, **kw)`*:\nReturn a new `CornuCopyBuffer` attached to an open file.\n\nInternally this constructs a `SeekableFileIterator`, which\nprovides the iteration that `CornuCopyBuffer` consumes\nand also seek support if the underlying file is seekable.\n\nParameters:\n* `f`: the file like object\n* `readsize`: an optional preferred read size\n* `offset`: a starting position for the data; the file\n will seek to this offset, and the buffer will start with this\n offset\nOther keyword arguments are passed to the buffer constructor.\n\n*Method `CornuCopyBuffer.from_filename(filename: str, offset=None, **kw)`*:\nOpen the file named `filename` and return a new `CornuCopyBuffer`.\n\nIf `offset` is provided, skip to that position in the file.\nA negative offset skips to a position that far from the end of the file\nas determined by its `Stat.st_size`.\n\nOther keyword arguments are passed to the buffer constructor.\n\n*Method `CornuCopyBuffer.from_mmap(fd, readsize=None, offset=None, **kw)`*:\nReturn a new `CornuCopyBuffer` attached to an mmap of an open\nfile descriptor.\n\nInternally this constructs a `SeekableMMapIterator`, which\nprovides the iteration that `CornuCopyBuffer` consumes, but\nalso seek support.\n\n*Note*: a `SeekableMMapIterator` makes an `os.dup` of the\nsupplied file descriptor, so the caller is responsible for\nclosing the original.\n\nParameters:\n* `fd`: the operating system file descriptor\n* `readsize`: an optional preferred read size\n* `offset`: a starting position for the data; the file\n descriptor will seek to this offset, and the buffer will\n start with this offset\nOther keyword arguments are passed to the buffer constructor.\n\n*Method `CornuCopyBuffer.hint(self, size)`*:\nHint that the caller is seeking at least `size` bytes.\n\nIf the `input_data` iterator has a `hint` method, this is\npassed to it.\n\n*Method `CornuCopyBuffer.iter(self, maxlength)`*:\nYield chunks from the buffer\nup to `maxlength` in total\nor until EOF if `maxlength` is `Ellipsis`.\n\n*Method `CornuCopyBuffer.next(self)`*:\nFetch a data chunk from the buffer.\n\n*Method `CornuCopyBuffer.peek(self, size, short_ok=False)`*:\nExamine the leading bytes of the buffer without consuming them,\na `take` followed by a `push`.\nReturns the bytes.\n\n*Method `CornuCopyBuffer.promote(obj)`*:\nPromote `obj` to a `CornuCopyBuffer`,\nused by the @cs.deco.promote` decorator.\n\nPromotes:\n* `int`: assumed to be a file descriptor of a file open for binary read\n* `str`: assumed to be a filesystem pathname\n* `bytes` and `bytes`like objects: data\n* has a `.read1` or `.read` method: assume a file open for binary read\n* iterable: assumed to be an iterable of `bytes`like objects\n\n*Method `CornuCopyBuffer.push(self, bs)`*:\nPush the chunk `bs` onto the front of the buffered data.\nRewinds the logical `.offset` by the length of `bs`.\n\n*Method `CornuCopyBuffer.read(self, size, one_fetch=False)`*:\nCompatibility method to allow using the buffer like a file.\n\nParameters:\n* `size`: the desired data size\n* `one_fetch`: do a single data fetch, default `False`\n\nIn `one_fetch` mode the read behaves like a POSIX file read,\nreturning up to to `size` bytes from a single I/O operation.\n\n*Method `CornuCopyBuffer.read1(self, size)`*:\nShorthand method for `self.read(size,one_fetch=True)`.\n\n*Method `CornuCopyBuffer.readline(self)`*:\nReturn a binary \"line\" from `self`, where a line is defined by\n its ending `b'\n'` delimiter.\n The final line from a buffer might not have a trailing newline;\n `b''` is returned at EOF.\n\n Example:\n\n >>> bfr = CornuCopyBuffer([b'abc', b'def\nhij'])\n >>> bfr.readline()\n b'abcdef\n'\n >>> bfr.readline()\n b'hij'\n >>> bfr.readline()\n b''\n >>> bfr.readline()\n b''\n\n*Method `CornuCopyBuffer.report_offset(self, offset)`*:\nReport a pertinent offset.\n\n*Method `CornuCopyBuffer.seek(self, offset, whence=None, short_ok=False)`*:\nCompatibility method to allow using the buffer like a file.\nThis returns the resulting absolute offset.\n\nParameters are as for `io.seek` except as noted below:\n* `whence`: (default `os.SEEK_SET`). This method only supports\n `os.SEEK_SET` and `os.SEEK_CUR`, and does not support seeking to a\n lower offset than the current buffer offset.\n* `short_ok`: (default `False`). If true, the seek may not reach\n the target if there are insufficent `input_data` - the\n position will be the end of the `input_data`, and the\n `input_data` will have been consumed; the caller must check\n the returned offset to check that it is as expected. If\n false, a `ValueError` will be raised; however, note that the\n `input_data` will still have been consumed.\n\n*Method `CornuCopyBuffer.selfcheck(self, msg='')`*:\nIntegrity check for the buffer, useful during debugging.\n\n*Method `CornuCopyBuffer.skip(self, toskip, copy_skip=None, short_ok=False)`*:\nAdvance position by `skip_to`. Return the new offset.\n\nParameters:\n* `toskip`: the distance to advance\n* `copy_skip`: callable to receive skipped data.\n* `short_ok`: default `False`; if true then skip may return before\n `skipto` bytes if there are insufficient `input_data`.\n\n*Method `CornuCopyBuffer.skipto(self, new_offset, copy_skip=None, short_ok=False)`*:\nAdvance to position `new_offset`. Return the new offset.\n\nParameters:\n* `new_offset`: the target offset.\n* `copy_skip`: callable to receive skipped data.\n* `short_ok`: default `False`; if true then skipto may return before\n `new_offset` if there are insufficient `input_data`.\n\nReturn values:\n* `buf`: the new state of `buf`\n* `offset`: the final offset; this may be short if `short_ok`.\n\n*Method `CornuCopyBuffer.subbuffer(self, end_offset)`*:\nContext manager wrapper for `.bounded`\nwhich calls the `.flush` method automatically\non exiting the context.\n\nExample:\n\n # avoid buffer overrun\n with bfr.subbuffer(bfr.offset+128) as subbfr:\n id3v1 = ID3V1Frame.parse(subbfr)\n # ensure the whole buffer was consumed\n assert subbfr.at_eof()\n\n*Method `CornuCopyBuffer.tail_extend(self, size)`*:\nExtend method for parsers reading \"tail\"-like chunk streams,\ntypically raw reads from a growing file.\n\nThis may read 0 bytes at EOF, but a future read may read\nmore bytes if the file grows.\nSuch an iterator can be obtained from\n``cs.fileutils.read_from(..,tail_mode=True)``.\n\n*Method `CornuCopyBuffer.take(self, size, short_ok=False)`*:\nReturn the next `size` bytes.\nOther arguments are as for `.extend()`.\n\nThis is a thin wrapper for the `.takev` method.\n\n*Method `CornuCopyBuffer.takev(self, size, short_ok=False)`*:\nReturn the next `size` bytes as a list of chunks\n(because the internal buffering is also a list of chunks).\nOther arguments are as for extend().\n\nSee `.take()` to get a flat chunk instead of a list.\n\n*Method `CornuCopyBuffer.tell(self)`*:\nCompatibility method to allow using the buffer like a file.\n\n## Class `FDIterator(_Iterator)`\n\nAn iterator over the data of a file descriptor.\n\n*Note*: the iterator works with an os.dup() of the file\ndescriptor so that it can close it with impunity; this requires\nthe caller to close their descriptor.\n\n*Method `FDIterator.__init__(self, fd, offset=None, readsize=None, align=True)`*:\nInitialise the iterator.\n\nParameters:\n* `fd`: file descriptor\n* `offset`: the initial logical offset, kept up to date by\n iteration; the default is the current file position.\n* `readsize`: a preferred read size; if omitted then\n `DEFAULT_READSIZE` will be stored\n* `align`: whether to align reads by default: if true then\n the iterator will do a short read to bring the `offset`\n into alignment with `readsize`; the default is `True`\n\n*Method `FDIterator.__del__(self)`*:\nClose the file descriptor.\n\n*Method `FDIterator.close(self)`*:\nClose the file descriptor.\n\n## Class `FileIterator(_Iterator, SeekableIteratorMixin)`\n\nAn iterator over the data of a file object.\n\n*Note*: the iterator closes the file on `__del__` or if its\n`.close` method is called.\n\n*Method `FileIterator.__init__(self, fp, offset=None, readsize=None, align=False)`*:\nInitialise the iterator.\n\nParameters:\n* `fp`: file object\n* `offset`: the initial logical offset, kept up to date by\n iteration; the default is 0.\n* `readsize`: a preferred read size; if omitted then\n `DEFAULT_READSIZE` will be stored\n* `align`: whether to align reads by default: if true then\n the iterator will do a short read to bring the `offset`\n into alignment with `readsize`; the default is `False`\n\n*Method `FileIterator.close(self)`*:\nDetach from the file. Does *not* call `fp.close()`.\n\n## Class `SeekableFDIterator(FDIterator, SeekableIteratorMixin)`\n\nAn iterator over the data of a seekable file descriptor.\n\n*Note*: the iterator works with an `os.dup()` of the file\ndescriptor so that it can close it with impunity; this requires\nthe caller to close their descriptor.\n\n*Property `SeekableFDIterator.end_offset`*:\nThe end offset of the file.\n\n## Class `SeekableFileIterator(FileIterator)`\n\nAn iterator over the data of a seekable file object.\n\n*Note*: the iterator closes the file on __del__ or if its\n.close method is called.\n\n*Method `SeekableFileIterator.__init__(self, fp, offset=None, **kw)`*:\nInitialise the iterator.\n\nParameters:\n* `fp`: file object\n* `offset`: the initial logical offset, kept up to date by\n iteration; the default is the current file position.\n* `readsize`: a preferred read size; if omitted then\n `DEFAULT_READSIZE` will be stored\n* `align`: whether to align reads by default: if true then\n the iterator will do a short read to bring the `offset`\n into alignment with `readsize`; the default is `False`\n\n*Method `SeekableFileIterator.seek(self, new_offset, mode=0)`*:\nMove the logical file pointer.\n\nWARNING: moves the underlying file's pointer.\n\n## Class `SeekableIteratorMixin`\n\nMixin supplying a logical with a `seek` method.\n\n*Method `SeekableIteratorMixin.seek(self, new_offset, mode=0)`*:\nMove the logical offset.\n\n## Class `SeekableMMapIterator(_Iterator, SeekableIteratorMixin)`\n\nAn iterator over the data of a mappable file descriptor.\n\n*Note*: the iterator works with an `mmap` of an `os.dup()` of the\nfile descriptor so that it can close it with impunity; this\nrequires the caller to close their descriptor.\n\n*Method `SeekableMMapIterator.__init__(self, fd, offset=None, readsize=None, align=True)`*:\nInitialise the iterator.\n\nParameters:\n* `offset`: the initial logical offset, kept up to date by\n iteration; the default is the current file position.\n* `readsize`: a preferred read size; if omitted then\n `DEFAULT_READSIZE` will be stored\n* `align`: whether to align reads by default: if true then\n the iterator will do a short read to bring the `offset`\n into alignment with `readsize`; the default is `True`\n\n*Method `SeekableMMapIterator.close(self)`*:\nDetach from the file descriptor and mmap and close.\n\n*Property `SeekableMMapIterator.end_offset`*:\nThe end offset of the mmap memoryview.\n\n# Release Log\n\n\n\n*Release 20240412*:\nCornuCopyBuffer.buf: return b\"\" if nothing is buffered.\n\n*Release 20240316*:\nFixed release upload artifacts.\n\n*Release 20240201*:\nCornuCopyBuffer: read1() method shorthand for read(..,one_fetch=True).\n\n*Release 20230401*:\n* CornuCopyBuffer.promote: document accepting an iterable as an iterable of bytes.\n* CornuCopyBuffer: new readline() method to return a binary line from the buffer.\n* CornuCopyBuffer.promote: assume objects with a .read1 or .read are files.\n\n*Release 20230212.2*:\n* BREAKING: drop @chunky, superceded by @promote for CornuCopyBuffer parameters.\n* CornuCopyBuffer: subclass Promotable.\n\n*Release 20230212.1*:\nAdd missing requirement for cs.gimmicks.\n\n*Release 20230212*:\nCornuCopyBuffer: new promote() method to promote int,str,bytes, also assumes nonspecific iterables yield byteslike instances.\n\n*Release 20211208*:\nCornuCopyBuffer.__init__: bugfix for self.input_data when copy_chunks is not None.\n\n*Release 20210316*:\n* New CornuCopyBuffer.from_filename factory method.\n* New CornuCopyBuffer.peek method to examine the leading bytes from the buffer.\n\n*Release 20210306*:\n* CornuCopyBuffer.from_file: improve the seekability test, handle files with no .tell().\n* CornuCopyBuffer.from_bytes: bugfix: set the buffer.offset to the supplied offset.\n* CornuCopyBuffer.from_bytes: queue a memoryview of the supplied bytes.\n\n*Release 20201102*:\nCornuCopyBuffer: new optional `progress` parameter for reporting data consumed from `input_data`.\n\n*Release 20201021*:\n* CornuCopyBuffer.from_file: changes to the test for a .seek method.\n* CornuCopyBuffer.read: call extend with short_ok=True.\n* CornuCopyBuffer.from_fd: record the fd as .fd, lets users os.fstat(bfr.fd).\n* New CornuCopyBuffer.as_fd method to return a readable file descriptor fed from the buffer by a Thread, intended for feeding subprocesses.\n* New CornuCopyBuffer.iter(maxlength) to return an iterator of up to maxlength bytes.\n* CornuCopyBuffer.__init__: new \"close\" parameter to release resources; new CornuCopyBuffer.close method to call this.\n* Some small fixes.\n\n*Release 20200517*:\n* CornuCopyBuffer.skip: bugfix sanity check.\n* FileIterator: do not close the supplied file, just set self.fp=None.\n* Improve EOFError message text.\n\n*Release 20200328*:\n* CornuCopyBuffer.takev: bugfix adjustment of buf.offset, was not always done.\n* CornuCopyBuffer.__getitem__: add slice support, note how expensive it is to use.\n\n*Release 20200229*:\n* New CornuCopyBuffer.byte0() method consuming the next byte and returning it as an int.\n* CornuCopyBuffer.takev: bugfix for size=0, logic refactor.\n* CornuCopyBuffer: new .selfcheck method.\n\n*Release 20200130*:\nCornuCopyBuffer.skip: bugfix adjustment of skipto for already buffered data.\n\n*Release 20191230.1*:\nDocstring updates. Semantic changes were in the previous release.\n\n*Release 20191230*:\n* CornuCopyBuffer: accept a size of Ellipsis in .take and .extend methods, indicating \"all the remaining data\".\n* CornuCopyBuffer: refactor the buffering, replacing .buf with .bufs as an array of chunks;\n* this enables support for the new .push method and reduces memory copying.\n\n*Release 20181231*:\nSmall bugfix.\n\n*Release 20181108*:\nNew at_eof() method. Python 2 tweak to support incidental import by python 2 even if unused.\n\n*Release 20180823*:\nBetter handling of seekable and unseekable input data. Tiny bugfix for from_bytes sanity check.\n\n*Release 20180810*:\n* Refactor SeekableFDIterator and SeekableFileIterator to subclass new SeekableIterator.\n* New SeekableMMapIterator to process a memory mapped file descriptor, intended for large files.\n* New CornuCopyBuffer.hint method to pass a length hint through to the input_data iterator\n* if it has a `hint` method, causing it possibly to make a differently sized fetch.\n* SeekableIterator: new __del__ method calling self.close() - subclasses must provide\n* a .close, which should be safe to call multiple times.\n* CornuCopyBuffer: add support for .offset and .end_offset optional attributes on the input_data iterator.\n* _BoundedBufferIterator: add .offset property plumbed to the underlying buffer offset.\n* New CornuCopyBuffer.from_mmap to make a mmap backed buffer so that large data can be returned without penalty.\n* Assorted fixes and doc improvements.\n\n*Release 20180805*:\nBugfixes for at_eof method and end_offset initialisation.\n\n*Release 20180726.1*:\nImprove docstrings and release with better long_description.\n\n*Release 20180726*:\nFirst PyPI release: CornuCopyBuffer and friends.\n\n",
"bugtrack_url": null,
"license": "GNU General Public License v3 or later (GPLv3+)",
"summary": "Facilities to do with buffers, particularly CornuCopyBuffer, an automatically refilling buffer to support parsing of data streams.",
"version": "20240412",
"project_urls": {
"URL": "https://bitbucket.org/cameron_simpson/css/commits/all"
},
"split_keywords": [
"python3"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "79fba2f05165b4bf99c2af81d79a4bf7122c8a3404200557bdfbe989e5a9153c",
"md5": "538b92f234ccc0750fd7a2412c6bee86",
"sha256": "42f80f957b4318092dc969d5426d3ad39b214484b23584496e58feaa2e096dea"
},
"downloads": -1,
"filename": "cs.buffer-20240412-py3-none-any.whl",
"has_sig": false,
"md5_digest": "538b92f234ccc0750fd7a2412c6bee86",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 21174,
"upload_time": "2024-04-12T05:09:32",
"upload_time_iso_8601": "2024-04-12T05:09:32.562792Z",
"url": "https://files.pythonhosted.org/packages/79/fb/a2f05165b4bf99c2af81d79a4bf7122c8a3404200557bdfbe989e5a9153c/cs.buffer-20240412-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "2ae3b965cd9e092cea85831c1762c1c5dcc524936fafa3ad0c84006ddd198ccf",
"md5": "c0add03fd68a52a661c5fc74b453259b",
"sha256": "479962b0d74a94d65365511341c91fdf1740c859ffe4706fd419f5139d83a398"
},
"downloads": -1,
"filename": "cs.buffer-20240412.tar.gz",
"has_sig": false,
"md5_digest": "c0add03fd68a52a661c5fc74b453259b",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 34520,
"upload_time": "2024-04-12T05:09:34",
"upload_time_iso_8601": "2024-04-12T05:09:34.788372Z",
"url": "https://files.pythonhosted.org/packages/2a/e3/b965cd9e092cea85831c1762c1c5dcc524936fafa3ad0c84006ddd198ccf/cs.buffer-20240412.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-04-12 05:09:34",
"github": false,
"gitlab": false,
"bitbucket": true,
"codeberg": false,
"bitbucket_user": "cameron_simpson",
"bitbucket_project": "css",
"lcname": "cs.buffer"
}
