# PelicanFS
[](https://zenodo.org/doi/10.5281/zenodo.13376216)
## Overview
PelicanFS is a file system interface (fsspec) for the Pelican Platform. For more information about pelican, see our [main website](https://pelicanplatform.org) or [Github page](https://github.com/PelicanPlatform/pelican). For more information about fsspec, visit the [filesystem-spec](https://filesystem-spec.readthedocs.io/en/latest/index.html) page.
## Limitations
PelicanFS is built on top of the http fsspec implementation. As such, any functionality that isn’t available in the http implementation is also *not* available in PelicanFS.
### Installation
To install pelican, run:
```
pip install pelicanfs
```
To install from source, run:
```
git clone https://github.com/PelicanPlatform/pelicanfs.git
cd pelicanfs
pip install -e .
```
### Using PelicanFS
To use pelicanfs, first create a `PelicanFileSystem` and provide it with the pelican federation url. As an example using the OSDF federation
```python
from pelicanfs.core import PelicanFileSystem
pelfs = PelicanFileSystem("pelican://osg-htc.org")
```
Once `pelfs` is pointed at your federation's director, fsspec commands can be applied to Pelican namespaces. For example:
```python
hello_world = pelfs.cat('/ospool/uc-shared/public/OSG-Staff/validation/test.txt')
print(hello_world)
```
### Getting an FSMap
Sometimes various systems that interact with an fsspec want a key-value mapper rather than a url. To do that, call the `PelicanMap` function with the namespace path and a `PelicanFileSystem` object rather than using the fsspec `get_mapper` call. For example:
```python
from pelicanfs.core import PelicanFileSystem, PelicanMap
pelfs = PelicanFileSystem(“some-director-url”)
file1 = PelicanMap(“/namespace/file/1”, pelfs=pelfs)
file2 = PelicanMap(“/namespace/file/2”, pelfs=pelfs)
ds = xarray.open_mfdataset([file1,file2], engine='zarr')
```
### Specifying Endpoints
The following describes how to specify endpoints to get data from, rather than letting PelicanFS and the director determine the best cache. PelicanFS allows you to specify whether to read directly from the origin (bypassing data staging altogether) or to name a specific cache to stage data into.
**Note**
> If both direct reads and a specific cache are set, PelicanFS will use the specified cache and ignore the direct reads setting.
#### Enabling Direct Reads
Sometimes you might wish to read data directly from an origin rather than via a cache. To enable this at PelicanFileSystem creation, just pass in `direct_reads=True` to the constructor.
```python
pelfs = PelicanFileSystem("pelican://osg-htc.org", direct_reads=True)
```
#### Specifying a Cache
If you want to specify a specific cache to stage your data into (as opposed to the highest priority working cache), this can be done by passing in a cache URL during PelicanFileSystem construction via the `preferred_caches` variable:
```python
pelfs = PelicanFileSystem("pelican://osg-htc.org", preferred_caches=["https://cache.example.com"])
```
or
```python
pelfs = PelicanFileSystem("pelican://osg-htc.org", preferred_caches=["https://cache.example.com",
"https://cache2.example.com", "+"])
```
Note that the special cache value `"+"` indicates that the provided preferred caches should be prepended to the
list of caches from the director.
Raw data
{
"_id": null,
"home_page": "https://github.com/PelicanPlatform/pelicanfs",
"name": "pelicanfs",
"maintainer": null,
"docs_url": null,
"requires_python": "<4,>=3.7",
"maintainer_email": null,
"keywords": "pelican, fsspec",
"author": null,
"author_email": null,
"download_url": "https://files.pythonhosted.org/packages/d5/74/77015f570c17548ef5149294d8f1a44be3340e9474c43f9fb7186e5706d3/pelicanfs-1.0.2.tar.gz",
"platform": null,
"description": "# PelicanFS\n\n[](https://zenodo.org/doi/10.5281/zenodo.13376216)\n\n## Overview\n\nPelicanFS is a file system interface (fsspec) for the Pelican Platform. For more information about pelican, see our [main website](https://pelicanplatform.org) or [Github page](https://github.com/PelicanPlatform/pelican). For more information about fsspec, visit the [filesystem-spec](https://filesystem-spec.readthedocs.io/en/latest/index.html) page.\n\n\n## Limitations\n\nPelicanFS is built on top of the http fsspec implementation. As such, any functionality that isn\u2019t available in the http implementation is also *not* available in PelicanFS.\n\n### Installation\n\nTo install pelican, run:\n\n```\npip install pelicanfs\n```\n\nTo install from source, run:\n\n```\ngit clone https://github.com/PelicanPlatform/pelicanfs.git\ncd pelicanfs\npip install -e .\n```\n\n\n### Using PelicanFS\n\nTo use pelicanfs, first create a `PelicanFileSystem` and provide it with the pelican federation url. As an example using the OSDF federation\n\n```python\nfrom pelicanfs.core import PelicanFileSystem\n\npelfs = PelicanFileSystem(\"pelican://osg-htc.org\")\n```\n\nOnce `pelfs` is pointed at your federation's director, fsspec commands can be applied to Pelican namespaces. For example:\n\n```python\nhello_world = pelfs.cat('/ospool/uc-shared/public/OSG-Staff/validation/test.txt')\nprint(hello_world)\n```\n\n### Getting an FSMap\n\nSometimes various systems that interact with an fsspec want a key-value mapper rather than a url. To do that, call the `PelicanMap` function with the namespace path and a `PelicanFileSystem` object rather than using the fsspec `get_mapper` call. For example:\n\n```python\nfrom pelicanfs.core import PelicanFileSystem, PelicanMap\n\npelfs = PelicanFileSystem(\u201csome-director-url\u201d)\nfile1 = PelicanMap(\u201c/namespace/file/1\u201d, pelfs=pelfs)\nfile2 = PelicanMap(\u201c/namespace/file/2\u201d, pelfs=pelfs)\nds = xarray.open_mfdataset([file1,file2], engine='zarr')\n```\n\n### Specifying Endpoints\n\nThe following describes how to specify endpoints to get data from, rather than letting PelicanFS and the director determine the best cache. PelicanFS allows you to specify whether to read directly from the origin (bypassing data staging altogether) or to name a specific cache to stage data into. \n\n**Note**\n> If both direct reads and a specific cache are set, PelicanFS will use the specified cache and ignore the direct reads setting.\n\n\n#### Enabling Direct Reads\n\nSometimes you might wish to read data directly from an origin rather than via a cache. To enable this at PelicanFileSystem creation, just pass in `direct_reads=True` to the constructor.\n\n```python\npelfs = PelicanFileSystem(\"pelican://osg-htc.org\", direct_reads=True)\n```\n\n#### Specifying a Cache\n\nIf you want to specify a specific cache to stage your data into (as opposed to the highest priority working cache), this can be done by passing in a cache URL during PelicanFileSystem construction via the `preferred_caches` variable:\n\n```python\npelfs = PelicanFileSystem(\"pelican://osg-htc.org\", preferred_caches=[\"https://cache.example.com\"])\n```\n\nor\n\n```python\npelfs = PelicanFileSystem(\"pelican://osg-htc.org\", preferred_caches=[\"https://cache.example.com\",\n \"https://cache2.example.com\", \"+\"])\n```\n\nNote that the special cache value `\"+\"` indicates that the provided preferred caches should be prepended to the\nlist of caches from the director.\n",
"bugtrack_url": null,
"license": null,
"summary": "An FSSpec Implementation using the Pelican System",
"version": "1.0.2",
"project_urls": {
"Bug Reports": "https://github.com/PelicanPlatform/pelicanfs/issues",
"Homepage": "https://github.com/PelicanPlatform/pelicanfs",
"Pelican Source": "https://github.com/PelicanPlatform/pelican",
"Source": "https://github.com/PelicanPlatform/pelicanfs"
},
"split_keywords": [
"pelican",
" fsspec"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "9875c4db2e0bb487f21d577d798b6a377be8136b5c11e21f66208480c8359301",
"md5": "2b3ef77833469f2f5eafaefaa5106429",
"sha256": "314e610322c7ac8dea3dfe2c879c94a801d02a3504453670397f8ce323091380"
},
"downloads": -1,
"filename": "pelicanfs-1.0.2-py3-none-any.whl",
"has_sig": false,
"md5_digest": "2b3ef77833469f2f5eafaefaa5106429",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4,>=3.7",
"size": 14582,
"upload_time": "2024-08-26T19:16:22",
"upload_time_iso_8601": "2024-08-26T19:16:22.535752Z",
"url": "https://files.pythonhosted.org/packages/98/75/c4db2e0bb487f21d577d798b6a377be8136b5c11e21f66208480c8359301/pelicanfs-1.0.2-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "d57477015f570c17548ef5149294d8f1a44be3340e9474c43f9fb7186e5706d3",
"md5": "e0d44aefb9a9c994e99388dfcc5ad725",
"sha256": "0f72f8c754473f2fa8826cda62c2d91fd8f547146d217b1e37c2f0b03b280fa2"
},
"downloads": -1,
"filename": "pelicanfs-1.0.2.tar.gz",
"has_sig": false,
"md5_digest": "e0d44aefb9a9c994e99388dfcc5ad725",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4,>=3.7",
"size": 18667,
"upload_time": "2024-08-26T19:16:23",
"upload_time_iso_8601": "2024-08-26T19:16:23.544997Z",
"url": "https://files.pythonhosted.org/packages/d5/74/77015f570c17548ef5149294d8f1a44be3340e9474c43f9fb7186e5706d3/pelicanfs-1.0.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-08-26 19:16:23",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "PelicanPlatform",
"github_project": "pelicanfs",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "pelicanfs"
}
JSON
