Sparse Virtual File
#################################################
Introduction
======================
Sometimes you don't need the whole file.
Sometimes you don't *want* the whole file.
Especially if it is huge and on some remote server.
But, you might know what parts of the file that you want and ``svfsc`` can help you store them locally so it looks
*as if* you have access to the complete file but with just the pieces of interest.
``svfsc`` is targeted at reading very large binary files such as TIFF, RP66V1, HDF5 where the structure is well known.
For example you might want to parse a TIFF file for its metadata or for a particular image tile or strip which is a tiny
fraction of the file itself.
``svfsc`` implements a *Sparse Virtual File*, a specialised in-memory cache where a particular file might not be
available but *parts of it can be obtained* without reading the whole file.
A Sparse Virtual File (SVF) is represented internally as a map of blocks of data with the key being their file
offsets.
Any write to an SVF will coalesce these blocks where possible.
There is no cache punting strategy implemented so an SVF always accumulates data.
A Sparse Virtual File System (SVFS) is an extension of this to provide a key/value store where the key is a file ID
and the value a Sparse Virtual File.
``svfsc`` is written in C++ with a Python interface.
It is thread safe in both domains.
A SVF might be used like this:
- The user requests some data (for example TIFF metadata) from a remote file using a Parser that knows the TIFF structure.
- The Parser consults the SVF, if the SVF has the data then the Parser parses it and gives the results to the user.
- If the SVF does *not* have the data then the Parser consults the SVF for what data is needed, then issues the appropriate GET request(s) to the remote server.
- That data is used to update the SVF, then the parser can use it and give the results to the user.
Here is a conceptual example of a ``SVF`` running on a local file system containing data from a single file.
.. code-block:: console
CLIENT SIDE | LOCAL FILE SYSTEM
.
/------\ /--------\ | /-------------\
| User | <--> | Parser | <-- read(fpos, len) --> | File System |
\------/ \--------/ | \-------------/
| .
| |
/-------\ .
| SVF | |
\-------/ .
Here is a conceptual example of an ``SVFS`` running with a remote file system.
.. code-block:: console
CLIENT SIDE | SERVER SIDE
.
/------\ /--------\ | /--------\
| User | <--> | Parser | <-- GET(fpos, len) --> | Server |
\------/ \--------/ | \--------/
| . |
| | |
/-------\ . /-------------\
| SVF | | | File System |
\-------/ . \-------------/
Example Python Usage
======================
Installation
------------
Install from pypi:
.. code-block:: console
$ pip install svfsc
Using a Single SVF
------------------
This shows the basic functionality: ``write()``, ``read()`` and ``need()``:
.. code-block:: python
import svfsc
# Construct a Sparse Virtual File
svf = svfsc.cSVF('Some file ID')
# Write six bytes at file position 14
svf.write(14, b'ABCDEF')
# Read from it
svf.read(16, 2) # Returns b'CD'
# What do I have to do to read 24 bytes from file position 8?
# This returns a tuple of pairs ((file_position, read_length), ...)
svf.need(8, 24) # Returns ((8, 6), (20, 4))
# Go and get the data from those file positions and write it to
# the SVF then you can read directly from the SVF.
The basic operation is to check if the ``SVF`` has data, if not then get it and write that data to the SVF.
Then read directly:
.. code-block:: python
if not svf.has_data(file_position, length):
for read_position, read_length in svf.need(file_position, length):
# Somehow get the data as a bytes object at (read_position, read_length)...
# This could be a GET request to a remote file.
# Then...
svf.write(read_position, data)
# Now read directly
svf.read(file_position, length)
A Sparse Virtual File System
-------------------------------------
The example above uses a single Sparse Virtual File, but you can also create a Sparse Virtual File System.
This is a key/value store where the key is some string and the value a ``SVF``:
.. code-block:: python
import svfsc
svfs = svfsc.cSVFS()
# Insert an empty SVF with a corresponding ID
ID = 'abc'
svfs.insert(ID)
# Write six bytes to that SVF at file position 14
svfs.write(ID, 14, b'ABCDEF')
# Read from the SVF
svfs.read(ID, 16, 2) # Returns b'CD'
# What do I have to do to read 24 bytes from file position 8
# from that SVF?
svfs.need(ID, 8, 24) # Returns ((8, 6), (20, 4))
Example C++ Usage
====================
``svfsc`` is written in C++ so can be used directly:
.. code-block:: c++
#include "svf.h"
// File modification time of 1672574430.0 (2023-01-01 12:00:30)
SVFS::SparseVirtualFile svf("Some file ID", 1672574430.0);
// Write six char at file position 14
svf.write(14, "ABCDEF", 6);
// Read from it
char read_buffer[2];
svf.read(16, 2, read_buffer);
// read_buffer now contains "CD"
// What do I have to do to read 24 bytes from file position 8?
// This returns a std::vector<std::pair<size_t, size_t>>
// as ((file_position, read_length), ...)
auto need = svf.need(8, 24);
// The following prints ((8, 6), (20, 4),)
std::cout << "(";
for (auto &val: need) {
std::cout << "(" << val.first << ", " << val.second << "),";
}
std::cout << ")" << std::endl;
.. note:: Naming conventions
On PyPi there is a preexisting `SVFS project <https://pypi.org/project/SVFS/>`_
(no relation, apparently abandoned since its release in 2012).
This project was renamed to ``svfsc``.
However there are many internal references in this project to ``SVF``, ``SVFS`` and variations thereof.
- The Cmake target is ``cppSVF``.
- The C++ code is in the namespace ``SVFS``, the important classes there are ``SVFS::SparseVirtualFile`` and ``SVFS::SparseVirtualFileSystem``.
- This `Python project on PyPi <https://pypi.org/project/svfsc/>`_ is named ``svfsc``. This can be installed by: ``pip install svfsc``.
- Access to the Python interface is done with: ``import svfsc``. The two important Python classes, equivalents of the C++ ones, are ``svfsc.cSVF`` and ``svfsc.cSVFS``
- Filenames often use ``svf`` and ``svfs`` in various ways.
Documentation
===============
Build the documentation from the ``docs`` directory or find it on readthedocs: https://svfsc.readthedocs.io/
Acknowledgments
===============
Many thanks to my employer `Paige.ai <https://paige.ai>`_ for allowing me to release this as FOSS software.
History
##################
0.4.0 (2024-02-11)
=====================
- Add counters for blocks/bytes erased and blocks/bytes punted and then their associated APIs.
- Use the ``SVFS_SVF_METHOD_SIZE_T_REGISTER`` macro in CPython to simplify registering CPython methods.
- Fix builds on Linux, mainly compiler flags.
- Move to -std=c++17 to exploit ``[[nodiscard]]``.
- Better alignment of compiler flags between CMakeLists.txt and setup.py
- Other minor fixes.
- Because of the extensive use of this in various projects this version 0.4 is moved to production status:
Development Status :: 5 - Production/Stable
0.3.0 (2024-01-06)
=====================
- Add ``need_many()``.
- Fix bug in ``lru_punt()``.
- Development Status :: 4 - Beta
0.2.2 (2023-12-28)
=====================
- Minor fixes.
- Development Status :: 4 - Beta
0.2.1 (2023-12-27)
=====================
- Include stub file.
- Development Status :: 4 - Beta
0.2.0 (2023-12-24)
=====================
- Add cache punting.
- Make C docstrings type parsable (good for Sphinx) and add a script that can create a mypy stub file.
- Development Status :: 4 - Beta
0.1.2 (2023-10-03)
=====================
- First release.
- Development Status :: 3 - Alpha
Raw data
{
"_id": null,
"home_page": "https://github.com/paulross/svfsc",
"name": "svfsc",
"maintainer": "",
"docs_url": null,
"requires_python": "",
"maintainer_email": "",
"keywords": "svf,svfs,svfsc",
"author": "Paul Ross",
"author_email": "apaulross@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/d1/d9/41615960bdef39dda85d80a3cd63c3cf75a92702384aec4a81bd7885d1fc/svfsc-0.4.0.tar.gz",
"platform": null,
"description": "\nSparse Virtual File\n#################################################\n\nIntroduction\n======================\n\nSometimes you don't need the whole file.\nSometimes you don't *want* the whole file.\nEspecially if it is huge and on some remote server.\nBut, you might know what parts of the file that you want and ``svfsc`` can help you store them locally so it looks\n*as if* you have access to the complete file but with just the pieces of interest.\n\n``svfsc`` is targeted at reading very large binary files such as TIFF, RP66V1, HDF5 where the structure is well known.\nFor example you might want to parse a TIFF file for its metadata or for a particular image tile or strip which is a tiny\nfraction of the file itself.\n\n``svfsc`` implements a *Sparse Virtual File*, a specialised in-memory cache where a particular file might not be\navailable but *parts of it can be obtained* without reading the whole file.\nA Sparse Virtual File (SVF) is represented internally as a map of blocks of data with the key being their file\noffsets.\nAny write to an SVF will coalesce these blocks where possible.\nThere is no cache punting strategy implemented so an SVF always accumulates data.\nA Sparse Virtual File System (SVFS) is an extension of this to provide a key/value store where the key is a file ID\nand the value a Sparse Virtual File.\n\n``svfsc`` is written in C++ with a Python interface.\nIt is thread safe in both domains.\n\nA SVF might be used like this:\n\n- The user requests some data (for example TIFF metadata) from a remote file using a Parser that knows the TIFF structure.\n- The Parser consults the SVF, if the SVF has the data then the Parser parses it and gives the results to the user.\n- If the SVF does *not* have the data then the Parser consults the SVF for what data is needed, then issues the appropriate GET request(s) to the remote server.\n- That data is used to update the SVF, then the parser can use it and give the results to the user.\n\nHere is a conceptual example of a ``SVF`` running on a local file system containing data from a single file.\n\n.. code-block:: console\n\n CLIENT SIDE | LOCAL FILE SYSTEM\n .\n /------\\ /--------\\ | /-------------\\\n | User | <--> | Parser | <-- read(fpos, len) --> | File System |\n \\------/ \\--------/ | \\-------------/\n | .\n | |\n /-------\\ .\n | SVF | |\n \\-------/ .\n\nHere is a conceptual example of an ``SVFS`` running with a remote file system.\n\n.. code-block:: console\n\n CLIENT SIDE | SERVER SIDE\n .\n /------\\ /--------\\ | /--------\\\n | User | <--> | Parser | <-- GET(fpos, len) --> | Server |\n \\------/ \\--------/ | \\--------/\n | . |\n | | |\n /-------\\ . /-------------\\\n | SVF | | | File System |\n \\-------/ . \\-------------/\n\nExample Python Usage\n======================\n\nInstallation\n------------\n\nInstall from pypi:\n\n.. code-block:: console\n\n $ pip install svfsc\n\nUsing a Single SVF\n------------------\n\nThis shows the basic functionality: ``write()``, ``read()`` and ``need()``:\n\n.. code-block:: python\n\n import svfsc\n\n # Construct a Sparse Virtual File\n svf = svfsc.cSVF('Some file ID')\n\n # Write six bytes at file position 14\n svf.write(14, b'ABCDEF')\n\n # Read from it\n svf.read(16, 2) # Returns b'CD'\n\n # What do I have to do to read 24 bytes from file position 8?\n # This returns a tuple of pairs ((file_position, read_length), ...)\n svf.need(8, 24) # Returns ((8, 6), (20, 4))\n # Go and get the data from those file positions and write it to\n # the SVF then you can read directly from the SVF.\n\nThe basic operation is to check if the ``SVF`` has data, if not then get it and write that data to the SVF.\nThen read directly:\n\n.. code-block:: python\n\n if not svf.has_data(file_position, length):\n for read_position, read_length in svf.need(file_position, length):\n # Somehow get the data as a bytes object at (read_position, read_length)...\n # This could be a GET request to a remote file.\n # Then...\n svf.write(read_position, data)\n # Now read directly\n svf.read(file_position, length)\n\nA Sparse Virtual File System\n-------------------------------------\n\nThe example above uses a single Sparse Virtual File, but you can also create a Sparse Virtual File System.\nThis is a key/value store where the key is some string and the value a ``SVF``:\n\n.. code-block:: python\n\n import svfsc\n\n svfs = svfsc.cSVFS()\n\n # Insert an empty SVF with a corresponding ID\n ID = 'abc'\n svfs.insert(ID)\n\n # Write six bytes to that SVF at file position 14\n svfs.write(ID, 14, b'ABCDEF')\n\n # Read from the SVF\n svfs.read(ID, 16, 2) # Returns b'CD'\n\n # What do I have to do to read 24 bytes from file position 8\n # from that SVF?\n svfs.need(ID, 8, 24) # Returns ((8, 6), (20, 4))\n\nExample C++ Usage\n====================\n\n``svfsc`` is written in C++ so can be used directly:\n\n.. code-block:: c++\n\n #include \"svf.h\"\n\n // File modification time of 1672574430.0 (2023-01-01 12:00:30)\n SVFS::SparseVirtualFile svf(\"Some file ID\", 1672574430.0);\n\n // Write six char at file position 14\n svf.write(14, \"ABCDEF\", 6);\n\n // Read from it\n char read_buffer[2];\n svf.read(16, 2, read_buffer);\n // read_buffer now contains \"CD\"\n\n // What do I have to do to read 24 bytes from file position 8?\n // This returns a std::vector<std::pair<size_t, size_t>>\n // as ((file_position, read_length), ...)\n auto need = svf.need(8, 24);\n\n // The following prints ((8, 6), (20, 4),)\n std::cout << \"(\";\n for (auto &val: need) {\n std::cout << \"(\" << val.first << \", \" << val.second << \"),\";\n }\n std::cout << \")\" << std::endl;\n\n.. note:: Naming conventions\n\n On PyPi there is a preexisting `SVFS project <https://pypi.org/project/SVFS/>`_\n (no relation, apparently abandoned since its release in 2012).\n This project was renamed to ``svfsc``.\n However there are many internal references in this project to ``SVF``, ``SVFS`` and variations thereof.\n\n - The Cmake target is ``cppSVF``.\n - The C++ code is in the namespace ``SVFS``, the important classes there are ``SVFS::SparseVirtualFile`` and ``SVFS::SparseVirtualFileSystem``.\n - This `Python project on PyPi <https://pypi.org/project/svfsc/>`_ is named ``svfsc``. This can be installed by: ``pip install svfsc``.\n - Access to the Python interface is done with: ``import svfsc``. The two important Python classes, equivalents of the C++ ones, are ``svfsc.cSVF`` and ``svfsc.cSVFS``\n - Filenames often use ``svf`` and ``svfs`` in various ways.\n\nDocumentation\n===============\n\nBuild the documentation from the ``docs`` directory or find it on readthedocs: https://svfsc.readthedocs.io/\n\nAcknowledgments\n===============\n\nMany thanks to my employer `Paige.ai <https://paige.ai>`_ for allowing me to release this as FOSS software.\n\n\nHistory\n##################\n\n0.4.0 (2024-02-11)\n=====================\n\n- Add counters for blocks/bytes erased and blocks/bytes punted and then their associated APIs.\n- Use the ``SVFS_SVF_METHOD_SIZE_T_REGISTER`` macro in CPython to simplify registering CPython methods.\n- Fix builds on Linux, mainly compiler flags.\n - Move to -std=c++17 to exploit ``[[nodiscard]]``.\n - Better alignment of compiler flags between CMakeLists.txt and setup.py\n- Other minor fixes.\n- Because of the extensive use of this in various projects this version 0.4 is moved to production status:\n Development Status :: 5 - Production/Stable\n\n0.3.0 (2024-01-06)\n=====================\n\n- Add ``need_many()``.\n- Fix bug in ``lru_punt()``.\n- Development Status :: 4 - Beta\n\n0.2.2 (2023-12-28)\n=====================\n\n- Minor fixes.\n- Development Status :: 4 - Beta\n\n0.2.1 (2023-12-27)\n=====================\n\n- Include stub file.\n- Development Status :: 4 - Beta\n\n0.2.0 (2023-12-24)\n=====================\n\n- Add cache punting.\n- Make C docstrings type parsable (good for Sphinx) and add a script that can create a mypy stub file.\n- Development Status :: 4 - Beta\n\n0.1.2 (2023-10-03)\n=====================\n\n- First release.\n- Development Status :: 3 - Alpha\n",
"bugtrack_url": null,
"license": "MIT License",
"summary": "Sparse Virtual File System Cache implemented in C++.",
"version": "0.4.0",
"project_urls": {
"Homepage": "https://github.com/paulross/svfsc"
},
"split_keywords": [
"svf",
"svfs",
"svfsc"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "7f7359099537e2951832f155b993c52e5b49dd4309a38895a095e9080290f0d2",
"md5": "afba6072d207e73aa74abedbbdcdf4ba",
"sha256": "2c3fdf39df5c261a14da70fe6df7f9e1659bf689376f9c8b51470450ceec19dd"
},
"downloads": -1,
"filename": "svfsc-0.4.0-cp310-cp310-macosx_10_9_universal2.whl",
"has_sig": false,
"md5_digest": "afba6072d207e73aa74abedbbdcdf4ba",
"packagetype": "bdist_wheel",
"python_version": "cp310",
"requires_python": null,
"size": 124605,
"upload_time": "2024-02-11T19:11:28",
"upload_time_iso_8601": "2024-02-11T19:11:28.010831Z",
"url": "https://files.pythonhosted.org/packages/7f/73/59099537e2951832f155b993c52e5b49dd4309a38895a095e9080290f0d2/svfsc-0.4.0-cp310-cp310-macosx_10_9_universal2.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "a798dcc5097f188b23daf104ba92dc691097bb4848f0bd24cc523678d156718b",
"md5": "c84b4409ae887417b798b0c24bd9fbb7",
"sha256": "e39f48e494a5f131f937498e3104d0967cf009704d97e100c8cfad9cd0896d29"
},
"downloads": -1,
"filename": "svfsc-0.4.0-cp311-cp311-macosx_10_9_universal2.whl",
"has_sig": false,
"md5_digest": "c84b4409ae887417b798b0c24bd9fbb7",
"packagetype": "bdist_wheel",
"python_version": "cp311",
"requires_python": null,
"size": 124724,
"upload_time": "2024-02-11T19:11:30",
"upload_time_iso_8601": "2024-02-11T19:11:30.135330Z",
"url": "https://files.pythonhosted.org/packages/a7/98/dcc5097f188b23daf104ba92dc691097bb4848f0bd24cc523678d156718b/svfsc-0.4.0-cp311-cp311-macosx_10_9_universal2.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "3fcd2ee8a865cb78d37b4a5fc2ddb10312cb180a252c3c8bcf86900e9ccfe6d0",
"md5": "ab836a205a3e173b7ce0dc3689b2dc4b",
"sha256": "2cb907f26925616e233239375212d3e34141f8b3c66742febfefee78b3330539"
},
"downloads": -1,
"filename": "svfsc-0.4.0-cp312-cp312-macosx_10_9_universal2.whl",
"has_sig": false,
"md5_digest": "ab836a205a3e173b7ce0dc3689b2dc4b",
"packagetype": "bdist_wheel",
"python_version": "cp312",
"requires_python": null,
"size": 124923,
"upload_time": "2024-02-11T19:11:31",
"upload_time_iso_8601": "2024-02-11T19:11:31.657579Z",
"url": "https://files.pythonhosted.org/packages/3f/cd/2ee8a865cb78d37b4a5fc2ddb10312cb180a252c3c8bcf86900e9ccfe6d0/svfsc-0.4.0-cp312-cp312-macosx_10_9_universal2.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "9000c42a0928e2ccd4b4f7d1f3c84ec6ccb14c5047e827e8bc1310434daab727",
"md5": "e7d0cebf42c10abde424e7002326755c",
"sha256": "700eaef1ac5da54031166222e8bdca1495efe94cfc1bdafd3cfd5b74d6b6f1b1"
},
"downloads": -1,
"filename": "svfsc-0.4.0-cp38-cp38-macosx_10_9_x86_64.whl",
"has_sig": false,
"md5_digest": "e7d0cebf42c10abde424e7002326755c",
"packagetype": "bdist_wheel",
"python_version": "cp38",
"requires_python": null,
"size": 65825,
"upload_time": "2024-02-11T19:11:33",
"upload_time_iso_8601": "2024-02-11T19:11:33.636243Z",
"url": "https://files.pythonhosted.org/packages/90/00/c42a0928e2ccd4b4f7d1f3c84ec6ccb14c5047e827e8bc1310434daab727/svfsc-0.4.0-cp38-cp38-macosx_10_9_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "b7a24daeb36c3ac1936e353483394e0c932b2866e55ef5bc16e4a7e8849da41b",
"md5": "918aaf885fb371d295ae9b2ad2f9cfaa",
"sha256": "05cc27af500e65588d339d064e77f629e620e74ddb98ac0042cf0301a16cc2ab"
},
"downloads": -1,
"filename": "svfsc-0.4.0-cp39-cp39-macosx_10_9_universal2.whl",
"has_sig": false,
"md5_digest": "918aaf885fb371d295ae9b2ad2f9cfaa",
"packagetype": "bdist_wheel",
"python_version": "cp39",
"requires_python": null,
"size": 124620,
"upload_time": "2024-02-11T19:11:35",
"upload_time_iso_8601": "2024-02-11T19:11:35.408648Z",
"url": "https://files.pythonhosted.org/packages/b7/a2/4daeb36c3ac1936e353483394e0c932b2866e55ef5bc16e4a7e8849da41b/svfsc-0.4.0-cp39-cp39-macosx_10_9_universal2.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "d1d941615960bdef39dda85d80a3cd63c3cf75a92702384aec4a81bd7885d1fc",
"md5": "a216aacbdd650aef97fb163fa45e7577",
"sha256": "a31398de82ef2f416237191d78d363b2b888e3708b96b39b6c95982dbf156e38"
},
"downloads": -1,
"filename": "svfsc-0.4.0.tar.gz",
"has_sig": false,
"md5_digest": "a216aacbdd650aef97fb163fa45e7577",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 47071,
"upload_time": "2024-02-11T19:11:36",
"upload_time_iso_8601": "2024-02-11T19:11:36.898248Z",
"url": "https://files.pythonhosted.org/packages/d1/d9/41615960bdef39dda85d80a3cd63c3cf75a92702384aec4a81bd7885d1fc/svfsc-0.4.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-02-11 19:11:36",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "paulross",
"github_project": "svfsc",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"requirements": [],
"lcname": "svfsc"
}
JSON
