# Backend.AI Storage Proxy
Backend.AI Storage Proxy is an RPC daemon to manage vfolders used in Backend.AI agent, with quota and
storage-specific optimization support.
## Package Structure
- `ai.backend.storage`
- `server`: The agent daemon which communicates between Backend.AI Manager
- `api.client`: The client-facing API to handle tus.io server-side protocol for uploads and ranged HTTP
queries for downloads.
- `api.manager`: The manager-facing (internal) API to provide abstraction of volumes and separation of
the hardware resources for volume and file operations.
- `vfs`
- The minimal fallback backend which only uses the standard Linux filesystem interfaces
- `xfs`
- XFS-optimized backend with a small daemon to manage XFS project IDs for quota limits
- `agent`: Implementation of `AbstractVolumeAgent` with XFS support
- `purestorage`
- PureStorage's FlashBlade-optimized backend with RapidFile Toolkit (formerly PureTools)
- `netapp`
- NetApp QTree integration backend based on the NetApp ONTAP REST API
- `weka`
- Weka.IO integration backend with Weka.IO V2 REST API
- `cephfs` (TODO)
- CephFS-optimized backend with quota limit support
## Installation
### Prerequisites
- Python 3.8 or higher with [pyenv](https://github.com/pyenv/pyenv)
and [pyenv-virtualenv](https://github.com/pyenv/pyenv-virtualenv) (optional but recommended)
### Installation Process
First, prepare the source clone of this agent:
```console
# git clone https://github.com/lablup/backend.ai-storage-proxy
```
From now on, let's assume all shell commands are executed inside the virtualenv.
Now install dependencies:
```console
# pip install -U -r requirements/dist.txt # for deployment
# pip install -U -r requirements/dev.txt # for development
```
Then, copy halfstack.toml to root of the project folder and edit to match your machine:
```console
# cp config/sample.toml storage-proxy.toml
```
When done, start storage server:
```console
# python -m ai.backend.storage.server
```
It will start Storage Proxy daemon bound at `127.0.0.1:6021` (client API) and
`127.0.0.1:6022` (manager API).
NOTE: Depending on the backend, the server may require to be run as root.
### Production Deployment
To get performance boosts by using OS-provided `sendfile()` syscall
for file transfers, SSL termination should be handled by reverse-proxies
such as nginx and the storage proxy daemon itself should be run without SSL.
## Filesystem Backends
### VFS
#### Prerequisites
- User account permission to access for the given directory
- Make sure a directory such as `/vfroot/vfs` a directory or you want to mount exists
### XFS
#### Prerequisites
- Local device mounted under `/vfroot`
- Native support for XFS filesystem
- Mounting XFS volume with an option `-o pquota` to enable project quota
- To turn on quotas on the root filesystem, the quota mount flags must be
set with the `rootflags=` boot parameter. Usually, this is not recommended.
- Access to root privilege
- Execution of `xfs_quota`, which performs quota-related commands, requires
the `root` privilege.
- Thus, you need to start the Storage-Proxy service by a `root` user or a
user with passwordless sudo access.
- If the root user starts the Storage-Proxy, the owner of every file created
is also root. In some situations, this would not be the desired setting.
In that case, it might be better to start the service with a regular user
with passwordless sudo privilege.
#### Creating virtual XFS device for testing
Create a virtual block device mounted to `lo` (loopback) if you are the only one
to use the storage for testing:
1. Create file with your desired size
```console
# dd if=/dev/zero of=xfs_test.img bs=1G count=100
```
2. Make file as XFS partition
```console
# mkfs.xfs xfs_test.img
```
3. Mount it to loopback
```console
# export LODEVICE=$(losetup -f)
# losetup $LODEVICE xfs_test.img
```
4. Create mount point and mount loopback device, with pquota option
```console
# mkdir -p /vfroot/xfs
# mount -o loop -o pquota $LODEVICE /vfroot/xfs
```
#### Note on operation
XFS keeps quota mapping information on two files: `/etc/projects` and
`/etc/projid`. If they are deleted or damaged in any way, per-directory quota
information will also be lost. So, it is crucial not to delete them
accidentally. If possible, it is a good idea to backup them to a different disk
or NFS.
### PureStorage FlashBlade
#### Prerequisites
- NFSv3 export mounted under `/vfroot`
- Purity API access
### CephFS
#### Prerequisites
- FUSE export mounted under `/vfroot`
### NetApp ONTAP
#### Prerequisites
- NFSv3 export mounted under `/vfroot`
- NetApp ONTAP API access
- native NetApp XCP or Dockerized NetApp XCP container
- To install NetApp XCP, please refer [NetApp XCP install guide](https://xcp.netapp.com/)
- Create Qtree in Volume explicitly using NetApp ONTAP Sysmgr GUI
#### Note on operation
The volume host of Backend.AI Storage proxy corresponds to Qtree of NetApp ONTAP, not NetApp ONTAP Volume.
Please DO NOT remove Backend.AI mapped qtree in NetApp ONTAP Sysmgr GUI. If not, you cannot access to NetApp ONTAP Volume through Backend.AI.
> NOTE:
> Qtree name in configuration file(`storage-proxy.toml`) must have the same name created in NetApp ONTAP Sysmgr.
### Weka.IO
#### Prerequisites
- Weka.IO agent installed and running
- Weka.IO filesystem mounted under local machine, with permission set to somewhat storage-proxy process can read and write
- Weka.IO REST API access (username/password/organization)
Raw data
{
"_id": null,
"home_page": "https://github.com/lablup/backend.ai",
"name": "backend.ai-storage-proxy",
"maintainer": null,
"docs_url": null,
"requires_python": "<3.14,>=3.13",
"maintainer_email": null,
"keywords": null,
"author": "Lablup Inc. and contributors",
"author_email": null,
"download_url": "https://files.pythonhosted.org/packages/e5/31/565ce3e582732eb7000d682fe78271f6bf751511c710ac44128f06cee60b/backend_ai_storage_proxy-25.11.0.tar.gz",
"platform": null,
"description": "# Backend.AI Storage Proxy\n\nBackend.AI Storage Proxy is an RPC daemon to manage vfolders used in Backend.AI agent, with quota and\nstorage-specific optimization support.\n\n## Package Structure\n\n- `ai.backend.storage`\n - `server`: The agent daemon which communicates between Backend.AI Manager\n - `api.client`: The client-facing API to handle tus.io server-side protocol for uploads and ranged HTTP\n queries for downloads.\n - `api.manager`: The manager-facing (internal) API to provide abstraction of volumes and separation of\n the hardware resources for volume and file operations.\n - `vfs`\n - The minimal fallback backend which only uses the standard Linux filesystem interfaces\n - `xfs`\n - XFS-optimized backend with a small daemon to manage XFS project IDs for quota limits\n - `agent`: Implementation of `AbstractVolumeAgent` with XFS support\n - `purestorage`\n - PureStorage's FlashBlade-optimized backend with RapidFile Toolkit (formerly PureTools)\n - `netapp`\n - NetApp QTree integration backend based on the NetApp ONTAP REST API\n - `weka`\n - Weka.IO integration backend with Weka.IO V2 REST API\n - `cephfs` (TODO)\n - CephFS-optimized backend with quota limit support\n\n## Installation\n\n### Prerequisites\n\n- Python 3.8 or higher with [pyenv](https://github.com/pyenv/pyenv)\n and [pyenv-virtualenv](https://github.com/pyenv/pyenv-virtualenv) (optional but recommended)\n\n### Installation Process\n\nFirst, prepare the source clone of this agent:\n\n```console\n# git clone https://github.com/lablup/backend.ai-storage-proxy\n```\n\nFrom now on, let's assume all shell commands are executed inside the virtualenv.\n\nNow install dependencies:\n\n```console\n# pip install -U -r requirements/dist.txt # for deployment\n# pip install -U -r requirements/dev.txt # for development\n```\n\nThen, copy halfstack.toml to root of the project folder and edit to match your machine:\n\n```console\n# cp config/sample.toml storage-proxy.toml\n```\n\nWhen done, start storage server:\n\n```console\n# python -m ai.backend.storage.server\n```\n\nIt will start Storage Proxy daemon bound at `127.0.0.1:6021` (client API) and\n`127.0.0.1:6022` (manager API).\n\nNOTE: Depending on the backend, the server may require to be run as root.\n\n### Production Deployment\n\nTo get performance boosts by using OS-provided `sendfile()` syscall\nfor file transfers, SSL termination should be handled by reverse-proxies\nsuch as nginx and the storage proxy daemon itself should be run without SSL.\n\n## Filesystem Backends\n\n### VFS\n\n#### Prerequisites\n\n- User account permission to access for the given directory\n - Make sure a directory such as `/vfroot/vfs` a directory or you want to mount exists\n\n### XFS\n\n#### Prerequisites\n\n- Local device mounted under `/vfroot`\n- Native support for XFS filesystem\n - Mounting XFS volume with an option `-o pquota` to enable project quota\n - To turn on quotas on the root filesystem, the quota mount flags must be\n set with the `rootflags=` boot parameter. Usually, this is not recommended.\n- Access to root privilege\n - Execution of `xfs_quota`, which performs quota-related commands, requires\n the `root` privilege.\n - Thus, you need to start the Storage-Proxy service by a `root` user or a\n user with passwordless sudo access.\n - If the root user starts the Storage-Proxy, the owner of every file created\n is also root. In some situations, this would not be the desired setting.\n In that case, it might be better to start the service with a regular user\n with passwordless sudo privilege.\n\n#### Creating virtual XFS device for testing\n\nCreate a virtual block device mounted to `lo` (loopback) if you are the only one\nto use the storage for testing:\n\n1. Create file with your desired size\n\n```console\n# dd if=/dev/zero of=xfs_test.img bs=1G count=100\n```\n\n2. Make file as XFS partition\n\n```console\n# mkfs.xfs xfs_test.img\n```\n\n3. Mount it to loopback\n\n```console\n# export LODEVICE=$(losetup -f)\n# losetup $LODEVICE xfs_test.img\n```\n\n4. Create mount point and mount loopback device, with pquota option\n\n```console\n# mkdir -p /vfroot/xfs\n# mount -o loop -o pquota $LODEVICE /vfroot/xfs\n```\n\n#### Note on operation\n\nXFS keeps quota mapping information on two files: `/etc/projects` and\n`/etc/projid`. If they are deleted or damaged in any way, per-directory quota\ninformation will also be lost. So, it is crucial not to delete them\naccidentally. If possible, it is a good idea to backup them to a different disk\nor NFS.\n\n### PureStorage FlashBlade\n\n#### Prerequisites\n\n- NFSv3 export mounted under `/vfroot`\n- Purity API access\n\n### CephFS\n\n#### Prerequisites\n\n- FUSE export mounted under `/vfroot`\n\n### NetApp ONTAP\n\n#### Prerequisites\n\n- NFSv3 export mounted under `/vfroot`\n- NetApp ONTAP API access\n- native NetApp XCP or Dockerized NetApp XCP container\n - To install NetApp XCP, please refer [NetApp XCP install guide](https://xcp.netapp.com/)\n- Create Qtree in Volume explicitly using NetApp ONTAP Sysmgr GUI\n\n#### Note on operation\n\nThe volume host of Backend.AI Storage proxy corresponds to Qtree of NetApp ONTAP, not NetApp ONTAP Volume. \nPlease DO NOT remove Backend.AI mapped qtree in NetApp ONTAP Sysmgr GUI. If not, you cannot access to NetApp ONTAP Volume through Backend.AI.\n\n> NOTE: \n> Qtree name in configuration file(`storage-proxy.toml`) must have the same name created in NetApp ONTAP Sysmgr.\n\n### Weka.IO\n\n#### Prerequisites\n\n- Weka.IO agent installed and running\n- Weka.IO filesystem mounted under local machine, with permission set to somewhat storage-proxy process can read and write\n- Weka.IO REST API access (username/password/organization)\n",
"bugtrack_url": null,
"license": "LGPLv3",
"summary": "Backend.AI Storage Proxy",
"version": "25.11.0",
"project_urls": {
"Documentation": "https://docs.backend.ai/",
"Homepage": "https://github.com/lablup/backend.ai",
"Source": "https://github.com/lablup/backend.ai"
},
"split_keywords": [],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "ad7d0f4c1da0e6b885e69a985dc22aacc30e8d8d9009795db9f19a36380359e3",
"md5": "718c2e008cd83963c0d7a28cdc083f13",
"sha256": "95ceb955da3c8bf5380238149cefc713d113213df0c49607a3e7794d7eb53cae"
},
"downloads": -1,
"filename": "backend_ai_storage_proxy-25.11.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "718c2e008cd83963c0d7a28cdc083f13",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<3.14,>=3.13",
"size": 102518,
"upload_time": "2025-07-09T04:55:33",
"upload_time_iso_8601": "2025-07-09T04:55:33.661734Z",
"url": "https://files.pythonhosted.org/packages/ad/7d/0f4c1da0e6b885e69a985dc22aacc30e8d8d9009795db9f19a36380359e3/backend_ai_storage_proxy-25.11.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "e531565ce3e582732eb7000d682fe78271f6bf751511c710ac44128f06cee60b",
"md5": "d13f1c31f606760e4a59d7661c490418",
"sha256": "6d389159a7734e9cf851879cf7fa8b7ec01f45f9537ceae7a5f3d1f75abdbdc8"
},
"downloads": -1,
"filename": "backend_ai_storage_proxy-25.11.0.tar.gz",
"has_sig": false,
"md5_digest": "d13f1c31f606760e4a59d7661c490418",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<3.14,>=3.13",
"size": 83096,
"upload_time": "2025-07-09T04:55:52",
"upload_time_iso_8601": "2025-07-09T04:55:52.537484Z",
"url": "https://files.pythonhosted.org/packages/e5/31/565ce3e582732eb7000d682fe78271f6bf751511c710ac44128f06cee60b/backend_ai_storage_proxy-25.11.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-07-09 04:55:52",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "lablup",
"github_project": "backend.ai",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [
{
"name": "aiodataloader",
"specs": [
[
"~=",
"0.4.2"
]
]
},
{
"name": "aiodocker",
"specs": [
[
"==",
"0.24.0"
]
]
},
{
"name": "aiofiles",
"specs": [
[
"~=",
"24.1.0"
]
]
},
{
"name": "aiohttp",
"specs": [
[
"~=",
"3.11.16"
]
]
},
{
"name": "aiohttp_cors",
"specs": [
[
"~=",
"0.8.1"
]
]
},
{
"name": "aiohttp_jinja2",
"specs": [
[
"~=",
"1.6"
]
]
},
{
"name": "aiohttp_sse",
"specs": [
[
">=",
"2.2"
]
]
},
{
"name": "aiodns",
"specs": [
[
"==",
"3.2"
]
]
},
{
"name": "aiomonitor",
"specs": [
[
"~=",
"0.7.0"
]
]
},
{
"name": "aioresponses",
"specs": [
[
">=",
"0.7.3"
]
]
},
{
"name": "aiosqlite",
"specs": [
[
"~=",
"0.21.0"
]
]
},
{
"name": "aiosignal",
"specs": [
[
"==",
"1.3.2"
]
]
},
{
"name": "aiotools",
"specs": [
[
"~=",
"1.9.0"
]
]
},
{
"name": "aiotusclient",
"specs": [
[
"~=",
"0.1.4"
]
]
},
{
"name": "alembic",
"specs": [
[
"~=",
"1.13.2"
]
]
},
{
"name": "appdirs",
"specs": [
[
"~=",
"1.4.4"
]
]
},
{
"name": "async_timeout",
"specs": [
[
"~=",
"4.0"
]
]
},
{
"name": "asyncpg",
"specs": [
[
">=",
"0.29.0"
]
]
},
{
"name": "asynctest",
"specs": [
[
">=",
"0.13.0"
]
]
},
{
"name": "asyncudp",
"specs": [
[
">=",
"0.11"
]
]
},
{
"name": "attrs",
"specs": [
[
">=",
"25.3"
]
]
},
{
"name": "bcrypt",
"specs": [
[
"~=",
"4.2.0"
]
]
},
{
"name": "boto3",
"specs": [
[
"~=",
"1.35"
]
]
},
{
"name": "cachetools",
"specs": [
[
"~=",
"5.5.0"
]
]
},
{
"name": "callosum",
"specs": [
[
"~=",
"1.0.3"
]
]
},
{
"name": "cattrs",
"specs": [
[
"~=",
"24.1.1"
]
]
},
{
"name": "click",
"specs": [
[
"~=",
"8.1.7"
]
]
},
{
"name": "coloredlogs",
"specs": [
[
"~=",
"15.0"
]
]
},
{
"name": "colorama",
"specs": [
[
">=",
"0.4.6"
]
]
},
{
"name": "cryptography",
"specs": [
[
">=",
"44.0.2"
]
]
},
{
"name": "dataclasses-json",
"specs": [
[
"~=",
"0.5.7"
]
]
},
{
"name": "faker",
"specs": [
[
"~=",
"24.7.1"
]
]
},
{
"name": "graphene",
"specs": [
[
"~=",
"3.3.0"
]
]
},
{
"name": "graypy",
"specs": [
[
"==",
"2.1.0"
]
]
},
{
"name": "humanize",
"specs": [
[
">=",
"3.1.0"
]
]
},
{
"name": "ifaddr",
"specs": [
[
"~=",
"0.2"
]
]
},
{
"name": "inquirer",
"specs": [
[
"~=",
"3.3.0"
]
]
},
{
"name": "janus",
"specs": [
[
"~=",
"2.0"
]
]
},
{
"name": "Jinja2",
"specs": [
[
"~=",
"3.1.6"
]
]
},
{
"name": "jupyter-client",
"specs": [
[
">=",
"8.6"
]
]
},
{
"name": "kubernetes",
"specs": [
[
"~=",
"10.0.0"
]
]
},
{
"name": "kubernetes-asyncio",
"specs": [
[
"~=",
"9.1.0"
]
]
},
{
"name": "lark",
"specs": [
[
"~=",
"1.1.5"
]
]
},
{
"name": "more-itertools",
"specs": [
[
"~=",
"10.5.0"
]
]
},
{
"name": "msgpack",
"specs": [
[
"~=",
"1.1.0"
]
]
},
{
"name": "multidict",
"specs": [
[
"~=",
"6.2.0"
]
]
},
{
"name": "namedlist",
"specs": [
[
"~=",
"1.8"
]
]
},
{
"name": "networkx",
"specs": [
[
"~=",
"3.3.0"
]
]
},
{
"name": "orjson",
"specs": [
[
"~=",
"3.10.16"
]
]
},
{
"name": "opentelemetry-api",
"specs": [
[
"~=",
"1.33.1"
]
]
},
{
"name": "opentelemetry-sdk",
"specs": [
[
"~=",
"1.33.1"
]
]
},
{
"name": "opentelemetry-exporter-otlp-proto-grpc",
"specs": [
[
"~=",
"1.33.1"
]
]
},
{
"name": "opentelemetry-instrumentation-aiohttp-client",
"specs": [
[
"~=",
"0.54b1"
]
]
},
{
"name": "opentelemetry-instrumentation-aiohttp-server",
"specs": [
[
"~=",
"0.54b1"
]
]
},
{
"name": "opentelemetry-instrumentation-logging",
"specs": [
[
"~=",
"0.54b1"
]
]
},
{
"name": "pexpect",
"specs": [
[
"~=",
"4.8"
]
]
},
{
"name": "prometheus-client",
"specs": [
[
"~=",
"0.21.1"
]
]
},
{
"name": "psutil",
"specs": [
[
"~=",
"7.0"
]
]
},
{
"name": "pycryptodome",
"specs": [
[
">=",
"3.20.0"
]
]
},
{
"name": "pyhumps",
"specs": [
[
"~=",
"3.8.0"
]
]
},
{
"name": "pyroscope-io",
"specs": [
[
"~=",
"0.8.8"
]
]
},
{
"name": "python-dateutil",
"specs": [
[
">=",
"2.9"
]
]
},
{
"name": "python-dotenv",
"specs": [
[
"~=",
"0.20.0"
]
]
},
{
"name": "python-json-logger",
"specs": [
[
"~=",
"3.2.0"
]
]
},
{
"name": "pyzmq",
"specs": [
[
"~=",
"26.4"
]
]
},
{
"name": "PyJWT",
"specs": [
[
"~=",
"2.10.1"
]
]
},
{
"name": "PyYAML",
"specs": [
[
"~=",
"6.0"
]
]
},
{
"name": "pydantic",
"specs": [
[
"~=",
"2.11.3"
]
]
},
{
"name": "packaging",
"specs": [
[
">=",
"24.1"
]
]
},
{
"name": "hiredis",
"specs": [
[
">=",
"3.0.0"
]
]
},
{
"name": "redis",
"specs": [
[
"==",
"4.5.5"
]
]
},
{
"name": "rich",
"specs": [
[
"~=",
"13.6"
]
]
},
{
"name": "ruamel.yaml",
"specs": [
[
"~=",
"0.18.10"
]
]
},
{
"name": "SQLAlchemy",
"specs": [
[
"~=",
"1.4.54"
]
]
},
{
"name": "setproctitle",
"specs": [
[
"~=",
"1.3.5"
]
]
},
{
"name": "setuptools",
"specs": [
[
"~=",
"80.0.0"
]
]
},
{
"name": "tabulate",
"specs": [
[
"~=",
"0.8.9"
]
]
},
{
"name": "temporenc",
"specs": [
[
"~=",
"0.1.0"
]
]
},
{
"name": "tenacity",
"specs": [
[
">=",
"9.0"
]
]
},
{
"name": "toml",
"specs": [
[
"~=",
"0.10.2"
]
]
},
{
"name": "tomli",
"specs": [
[
"~=",
"2.0.1"
]
]
},
{
"name": "tomlkit",
"specs": [
[
"~=",
"0.13.2"
]
]
},
{
"name": "tqdm",
"specs": [
[
"~=",
"4.67.1"
]
]
},
{
"name": "trafaret",
"specs": [
[
"~=",
"2.1"
]
]
},
{
"name": "treelib",
"specs": [
[
"~=",
"1.7.0"
]
]
},
{
"name": "typeguard",
"specs": [
[
"~=",
"4.3"
]
]
},
{
"name": "typing_extensions",
"specs": [
[
"~=",
"4.11"
]
]
},
{
"name": "textual",
"specs": [
[
"~=",
"0.79.1"
]
]
},
{
"name": "uvloop",
"specs": [
[
"~=",
"0.21"
]
]
},
{
"name": "valkey-glide",
"specs": [
[
"~=",
"2.0.1"
]
]
},
{
"name": "yarl",
"specs": [
[
"~=",
"1.19.0"
]
]
},
{
"name": "zipstream-new",
"specs": [
[
"~=",
"1.1.8"
]
]
},
{
"name": "pytest",
"specs": [
[
">=",
"8.3.3"
]
]
},
{
"name": "pytest-aiohttp",
"specs": [
[
"~=",
"1.0.5"
]
]
},
{
"name": "pytest-dependency",
"specs": [
[
">=",
"0.6.0"
]
]
},
{
"name": "types-six",
"specs": []
},
{
"name": "types-setuptools",
"specs": []
},
{
"name": "types-python-dateutil",
"specs": []
},
{
"name": "types-aiofiles",
"specs": []
},
{
"name": "types-cachetools",
"specs": []
},
{
"name": "types-Jinja2",
"specs": []
},
{
"name": "types-PyYAML",
"specs": []
},
{
"name": "types-redis",
"specs": []
},
{
"name": "types-tabulate",
"specs": []
},
{
"name": "types-toml",
"specs": []
},
{
"name": "backend.ai-krunner-alpine",
"specs": [
[
"==",
"5.4.0"
]
]
},
{
"name": "backend.ai-krunner-static-gnu",
"specs": [
[
"==",
"4.4.0"
]
]
},
{
"name": "etcd-client-py",
"specs": [
[
"~=",
"0.4.0"
]
]
}
],
"lcname": "backend.ai-storage-proxy"
}
