| Name | kvikio-cu11 JSON |
| Version |
24.12.1
JSON |
| download |
| home_page | None |
| Summary | KvikIO - GPUDirect Storage |
| upload_time | 2024-12-13 02:25:34 |
| maintainer | None |
| docs_url | None |
| author | NVIDIA Corporation |
| requires_python | >=3.10 |
| license | Apache 2.0 |
| keywords |
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# KvikIO: High Performance File IO
## Summary
KvikIO (pronounced "kuh-VICK-eye-oh", see [here](https://ordnet.dk/ddo_en/dict?query=kvik) for pronunciation of kvik) is a Python and C++ library for high performance file IO. It provides C++ and Python
bindings to [cuFile](https://docs.nvidia.com/gpudirect-storage/api-reference-guide/index.html),
which enables [GPUDirect Storage (GDS)](https://developer.nvidia.com/blog/gpudirect-storage/).
KvikIO also works efficiently when GDS isn't available and can read/write both host and device data seamlessly.
### Features
* Object oriented API of [cuFile](https://docs.nvidia.com/gpudirect-storage/api-reference-guide/index.html) with C++/Python exception handling.
* A Python [Zarr](https://zarr.readthedocs.io/en/stable/) backend for reading and writing GPU data to file seamlessly.
* Concurrent reads and writes using an internal thread pool.
* Non-blocking API.
* Transparently handles reads and writes to/from memory on both host and device.
* Provides Python bindings to [nvCOMP](https://github.com/NVIDIA/nvcomp).
### Documentation
* Python: <https://docs.rapids.ai/api/kvikio/nightly/>
* C++: <https://docs.rapids.ai/api/libkvikio/nightly/>
### Examples
#### Python
```python
import cupy
import kvikio
def main(path):
a = cupy.arange(100)
f = kvikio.CuFile(path, "w")
# Write whole array to file
f.write(a)
f.close()
b = cupy.empty_like(a)
f = kvikio.CuFile(path, "r")
# Read whole array from file
f.read(b)
assert all(a == b)
f.close()
# Use contexmanager
c = cupy.empty_like(a)
with kvikio.CuFile(path, "r") as f:
f.read(c)
assert all(a == c)
# Non-blocking read
d = cupy.empty_like(a)
with kvikio.CuFile(path, "r") as f:
future1 = f.pread(d[:50])
future2 = f.pread(d[50:], file_offset=d[:50].nbytes)
# Note: must wait for futures before exiting block
# at which point the file is closed.
future1.get() # Wait for first read
future2.get() # Wait for second read
assert all(a == d)
if __name__ == "__main__":
main("/tmp/kvikio-hello-world-file")
```
#### C++
```c++
#include <cstddef>
#include <future>
#include <cuda_runtime.h>
#include <kvikio/file_handle.hpp>
int main()
{
// Create two arrays `a` and `b`
constexpr std::size_t size = 100;
void *a = nullptr;
void *b = nullptr;
cudaMalloc(&a, size);
cudaMalloc(&b, size);
// Write `a` to file
kvikio::FileHandle fw("test-file", "w");
std::size_t written = fw.write(a, size);
fw.close();
// Read file into `b`
kvikio::FileHandle fr("test-file", "r");
std::size_t read = fr.read(b, size);
fr.close();
// Read file into `b` in parallel using 16 threads
kvikio::default_thread_pool::reset(16);
{
// FileHandles have RAII semantics
kvikio::FileHandle f("test-file", "r");
std::future<std::size_t> future = f.pread(b_dev, sizeof(a), 0); // Non-blocking
std::size_t read = future.get(); // Blocking
// Notice, `f` closes automatically on destruction.
}
}
```
Raw data
{
"_id": null,
"home_page": null,
"name": "kvikio-cu11",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.10",
"maintainer_email": null,
"keywords": null,
"author": "NVIDIA Corporation",
"author_email": null,
"download_url": "https://files.pythonhosted.org/packages/4b/c9/15d942a1b4e8fce93d6fe4f29eec1ef87c47cc06f821c87c1c2b7958a64a/kvikio_cu11-24.12.1.tar.gz",
"platform": null,
"description": "# KvikIO: High Performance File IO\n\n## Summary\n\nKvikIO (pronounced \"kuh-VICK-eye-oh\", see [here](https://ordnet.dk/ddo_en/dict?query=kvik) for pronunciation of kvik) is a Python and C++ library for high performance file IO. It provides C++ and Python\nbindings to [cuFile](https://docs.nvidia.com/gpudirect-storage/api-reference-guide/index.html),\nwhich enables [GPUDirect Storage (GDS)](https://developer.nvidia.com/blog/gpudirect-storage/).\nKvikIO also works efficiently when GDS isn't available and can read/write both host and device data seamlessly.\n\n\n### Features\n\n* Object oriented API of [cuFile](https://docs.nvidia.com/gpudirect-storage/api-reference-guide/index.html) with C++/Python exception handling.\n* A Python [Zarr](https://zarr.readthedocs.io/en/stable/) backend for reading and writing GPU data to file seamlessly.\n* Concurrent reads and writes using an internal thread pool.\n* Non-blocking API.\n* Transparently handles reads and writes to/from memory on both host and device.\n* Provides Python bindings to [nvCOMP](https://github.com/NVIDIA/nvcomp).\n\n\n### Documentation\n * Python: <https://docs.rapids.ai/api/kvikio/nightly/>\n * C++: <https://docs.rapids.ai/api/libkvikio/nightly/>\n\n\n### Examples\n\n#### Python\n```python\nimport cupy\nimport kvikio\n\ndef main(path):\n a = cupy.arange(100)\n f = kvikio.CuFile(path, \"w\")\n # Write whole array to file\n f.write(a)\n f.close()\n\n b = cupy.empty_like(a)\n f = kvikio.CuFile(path, \"r\")\n # Read whole array from file\n f.read(b)\n assert all(a == b)\n f.close()\n\n # Use contexmanager\n c = cupy.empty_like(a)\n with kvikio.CuFile(path, \"r\") as f:\n f.read(c)\n assert all(a == c)\n\n # Non-blocking read\n d = cupy.empty_like(a)\n with kvikio.CuFile(path, \"r\") as f:\n future1 = f.pread(d[:50])\n future2 = f.pread(d[50:], file_offset=d[:50].nbytes)\n # Note: must wait for futures before exiting block\n # at which point the file is closed.\n future1.get() # Wait for first read\n future2.get() # Wait for second read\n assert all(a == d)\n\n\nif __name__ == \"__main__\":\n main(\"/tmp/kvikio-hello-world-file\")\n```\n\n#### C++\n```c++\n#include <cstddef>\n#include <future>\n#include <cuda_runtime.h>\n#include <kvikio/file_handle.hpp>\n\nint main()\n{\n // Create two arrays `a` and `b`\n constexpr std::size_t size = 100;\n void *a = nullptr;\n void *b = nullptr;\n cudaMalloc(&a, size);\n cudaMalloc(&b, size);\n\n // Write `a` to file\n kvikio::FileHandle fw(\"test-file\", \"w\");\n std::size_t written = fw.write(a, size);\n fw.close();\n\n // Read file into `b`\n kvikio::FileHandle fr(\"test-file\", \"r\");\n std::size_t read = fr.read(b, size);\n fr.close();\n\n // Read file into `b` in parallel using 16 threads\n kvikio::default_thread_pool::reset(16);\n {\n // FileHandles have RAII semantics\n kvikio::FileHandle f(\"test-file\", \"r\");\n std::future<std::size_t> future = f.pread(b_dev, sizeof(a), 0); // Non-blocking\n std::size_t read = future.get(); // Blocking\n // Notice, `f` closes automatically on destruction.\n }\n}\n```\n",
"bugtrack_url": null,
"license": "Apache 2.0",
"summary": "KvikIO - GPUDirect Storage",
"version": "24.12.1",
"project_urls": {
"Homepage": "https://github.com/rapidsai/kvikio"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "4bc915d942a1b4e8fce93d6fe4f29eec1ef87c47cc06f821c87c1c2b7958a64a",
"md5": "c2c051644477764331215ff02185dcea",
"sha256": "dd9287bb96b4ba222326a6fad8efa47761f68979511326af3bc744df8e777533"
},
"downloads": -1,
"filename": "kvikio_cu11-24.12.1.tar.gz",
"has_sig": false,
"md5_digest": "c2c051644477764331215ff02185dcea",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.10",
"size": 2195,
"upload_time": "2024-12-13T02:25:34",
"upload_time_iso_8601": "2024-12-13T02:25:34.064628Z",
"url": "https://files.pythonhosted.org/packages/4b/c9/15d942a1b4e8fce93d6fe4f29eec1ef87c47cc06f821c87c1c2b7958a64a/kvikio_cu11-24.12.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-12-13 02:25:34",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "rapidsai",
"github_project": "kvikio",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "kvikio-cu11"
}
