# UniversalIO
UniversalIO is a library for Python developers looking to seamlessly integrate multiple different
file storage solutions.
The concept was born out of a need for a project to allow users to specify files which might be in the
Azure Blob Storage, Azure File Store, on a network drive, or on a SFTP server and then manipulate these
files. In addition, there was a need to write the same file back to multiple different locations. The
solution was to wrap the specific API for those storage solutions in an API that allowed the system to
operate on them consistently, as well as coordinating file transfers between them. The need for batch
operations led to the adoption of asyncio and its implementations for most of the backend, though
non-asynchronous methods are available as well.
## Operations Supported
- is_dir()
- is_file()
- exists()
- parent()
- child(child_name)
- basename()
- joinpath(*paths)
- remove()
- read(block_size)
- write(bytes)
- text(encoding)
- copy()
- move()
- mkdir(recursive)
- rmdir(recursive)
- crawl()
- detect_encoding()
- is_local_to(resource)
Asynchronous versions of most of the above exist with the suffix _async
## Storage Solutions Supported
- Local/network drives (via aiofiles)
- SFTP (via asyncssh)
- Azure Blob Storage (via azure.storage.blob.aio)
- HTTP servers (via aiohttp)
- Requires GET support for read access
- Requires PUT support for write access
- Requires DELETE support for delete
- Supports MKCOL for basic directory/collection creation
- Supports COPY, falls back to a GET and a PUT, for file copying
- Supports MOVE, falls back to a COPY and a DELETE, for file moving
## Planned Enhancements
- Some restructuring to match the pathlib API as much as possible
- Support for pulling credentials from TOML files in user's home directory
- Support for Azure File Storage
- Support for FTP and FTPS
- Support for direct HTTP access and HTML scraping
- Support for a batch file uploader/downloader
- Support for glob() matching and other searching options
- Support for pathlib-style with_name() to change the file name
- Support for touch()
- Support for writing text directly
- Support for HTTP authentication methods, including a custom method to support custom API calls
- More direct support for file copy/moving if the descriptor and connection is the same
- Considering support for SSH/SCP
- Considering support for relative file paths on local/network drives (challenging to understand what the root would be?)
- Considering support for chmod/etc (but not supported on Windows)
- Considering support for symbolic link creation (but complex on Windows)
Raw data
{
"_id": null,
"home_page": "https://github.com/turnbullerin/universalio",
"name": "universalio",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.7",
"maintainer_email": "",
"keywords": "",
"author": "Erin Turnbull",
"author_email": "erin.a.turnbull@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/52/a2/fa0a2ff70f35072c420942033f0b2bc85a579a8e253f59c836e53b612683/universalio-0.0.14.tar.gz",
"platform": null,
"description": "# UniversalIO\r\n\r\nUniversalIO is a library for Python developers looking to seamlessly integrate multiple different\r\nfile storage solutions. \r\n\r\nThe concept was born out of a need for a project to allow users to specify files which might be in the\r\nAzure Blob Storage, Azure File Store, on a network drive, or on a SFTP server and then manipulate these\r\nfiles. In addition, there was a need to write the same file back to multiple different locations. The\r\nsolution was to wrap the specific API for those storage solutions in an API that allowed the system to\r\noperate on them consistently, as well as coordinating file transfers between them. The need for batch\r\noperations led to the adoption of asyncio and its implementations for most of the backend, though \r\nnon-asynchronous methods are available as well. \r\n\r\n\r\n## Operations Supported\r\n\r\n- is_dir()\r\n- is_file()\r\n- exists()\r\n- parent()\r\n- child(child_name)\r\n- basename()\r\n- joinpath(*paths)\r\n- remove()\r\n- read(block_size)\r\n- write(bytes)\r\n- text(encoding)\r\n- copy()\r\n- move()\r\n- mkdir(recursive)\r\n- rmdir(recursive)\r\n- crawl()\r\n- detect_encoding()\r\n- is_local_to(resource)\r\n\r\nAsynchronous versions of most of the above exist with the suffix _async\r\n\r\n\r\n## Storage Solutions Supported\r\n\r\n- Local/network drives (via aiofiles)\r\n- SFTP (via asyncssh)\r\n- Azure Blob Storage (via azure.storage.blob.aio)\r\n- HTTP servers (via aiohttp)\r\n - Requires GET support for read access\r\n - Requires PUT support for write access\r\n - Requires DELETE support for delete\r\n - Supports MKCOL for basic directory/collection creation\r\n - Supports COPY, falls back to a GET and a PUT, for file copying\r\n - Supports MOVE, falls back to a COPY and a DELETE, for file moving\r\n\r\n\r\n## Planned Enhancements\r\n\r\n- Some restructuring to match the pathlib API as much as possible\r\n- Support for pulling credentials from TOML files in user's home directory\r\n- Support for Azure File Storage\r\n- Support for FTP and FTPS\r\n- Support for direct HTTP access and HTML scraping\r\n- Support for a batch file uploader/downloader\r\n- Support for glob() matching and other searching options\r\n- Support for pathlib-style with_name() to change the file name\r\n- Support for touch()\r\n- Support for writing text directly\r\n- Support for HTTP authentication methods, including a custom method to support custom API calls\r\n- More direct support for file copy/moving if the descriptor and connection is the same\r\n- Considering support for SSH/SCP\r\n- Considering support for relative file paths on local/network drives (challenging to understand what the root would be?)\r\n- Considering support for chmod/etc (but not supported on Windows)\r\n- Considering support for symbolic link creation (but complex on Windows)\r\n",
"bugtrack_url": null,
"license": "",
"summary": "Common API for asynchronous file management",
"version": "0.0.14",
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "bb18233cd7d1906ca8841056884cd93dfe718998b7e82b9236918da69f7a997d",
"md5": "c68a4497a727b7f8dd3d1f228dd90280",
"sha256": "3dca4114e4dcde0d24abc093e63ea974eb21ecc7332cc6a50e3bce753207ef99"
},
"downloads": -1,
"filename": "universalio-0.0.14-py3-none-any.whl",
"has_sig": false,
"md5_digest": "c68a4497a727b7f8dd3d1f228dd90280",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7",
"size": 21437,
"upload_time": "2023-04-17T19:58:29",
"upload_time_iso_8601": "2023-04-17T19:58:29.047771Z",
"url": "https://files.pythonhosted.org/packages/bb/18/233cd7d1906ca8841056884cd93dfe718998b7e82b9236918da69f7a997d/universalio-0.0.14-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "52a2fa0a2ff70f35072c420942033f0b2bc85a579a8e253f59c836e53b612683",
"md5": "cb0e824acf3efccb2c5e260f6295464c",
"sha256": "1e957df4b71e6e0a8514cd3c738a2f9eed53ff2eefcbef5bf3e0c102320f002d"
},
"downloads": -1,
"filename": "universalio-0.0.14.tar.gz",
"has_sig": false,
"md5_digest": "cb0e824acf3efccb2c5e260f6295464c",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7",
"size": 22042,
"upload_time": "2023-04-17T19:58:30",
"upload_time_iso_8601": "2023-04-17T19:58:30.333003Z",
"url": "https://files.pythonhosted.org/packages/52/a2/fa0a2ff70f35072c420942033f0b2bc85a579a8e253f59c836e53b612683/universalio-0.0.14.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-04-17 19:58:30",
"github": true,
"gitlab": false,
"bitbucket": false,
"github_user": "turnbullerin",
"github_project": "universalio",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "universalio"
}
JSON
