# PelicanFS
## Overview
PelicanFS is a file system interface (fsspec) for the Pelican Platform. For more information about pelican, see our [main website](https://pelicanplatform.org) or [Github page](https://github.com/PelicanPlatform/pelican). For more information about fsspec, visit the [filesystem-spec](https://filesystem-spec.readthedocs.io/en/latest/index.html) page.
## Limitations
PelicanFS is built on top of the http fsspec implementation. As such, any functionality that isn’t available in the http implementation is also *not* available in PelicanFS.
### Installation
To install pelican, run:
```
pip install pelicanfs
```
To install from source, run:
```
git clone https://github.com/PelicanPlatform/pelicanfs.git
cd pelicanfs
pip install -e .
```
### Using PelicanFS
To use pelicanfs, first create a `PelicanFileSystem` and provide it with the url for the director of your data federation. As an example using the OSDF director
```python
from pelicanfs.core import PelicanFileSystem
pelfs = PelicanFileSystem("https://osdf-director.osg-htc.org/")
```
Once `pelfs` is pointed at your federation's director, fsspec commands can be applied to Pelican namespaces. For example:
```python
hello_world = pelfs.cat('/ospool/uc-shared/public/OSG-Staff/validation/test.txt')
print(hello_world)
```
### Getting an FSMap
Sometimes various systems that interact with an fsspec want a key-value mapper rather than a url. To do that, call the `PelicanMap` function with the namespace path and a `PelicanFileSystem` object rather than using the fsspec `get_mapper` call. For example
```python
from pelicanfs.core import PelicanFileSystem, PelicanMap
pelfs = PelicanFileSystem(“some-director-url”)
file1 = PelicanMap(“/namespace/file/1”, pelfs=pelfs)
file2 = PelicanMap(“/namespace/file/2”, pelfs=pelfs)
ds = xarray.open_mfdataset([file1,file2], engine='zarr')
```
Raw data
{
"_id": null,
"home_page": "https://github.com/PelicanPlatform/pelicanfs",
"name": "pelicanfs",
"maintainer": null,
"docs_url": null,
"requires_python": "<4,>=3.7",
"maintainer_email": null,
"keywords": "pelican, fsspec",
"author": null,
"author_email": null,
"download_url": "https://files.pythonhosted.org/packages/dc/b8/05b907db54d7c942ebd1f4f0daafba3cbe23423e035ea6364b41393c3ddd/pelicanfs-0.0.3.tar.gz",
"platform": null,
"description": "# PelicanFS\n\n## Overview\n\nPelicanFS is a file system interface (fsspec) for the Pelican Platform. For more information about pelican, see our [main website](https://pelicanplatform.org) or [Github page](https://github.com/PelicanPlatform/pelican). For more information about fsspec, visit the [filesystem-spec](https://filesystem-spec.readthedocs.io/en/latest/index.html) page.\n\n\n## Limitations\n\nPelicanFS is built on top of the http fsspec implementation. As such, any functionality that isn\u2019t available in the http implementation is also *not* available in PelicanFS.\n\n### Installation\n\nTo install pelican, run:\n\n```\npip install pelicanfs\n```\n\nTo install from source, run:\n\n```\ngit clone https://github.com/PelicanPlatform/pelicanfs.git\ncd pelicanfs\npip install -e .\n```\n\n\n### Using PelicanFS\n\nTo use pelicanfs, first create a `PelicanFileSystem` and provide it with the url for the director of your data federation. As an example using the OSDF director\n\n```python\nfrom pelicanfs.core import PelicanFileSystem\n\npelfs = PelicanFileSystem(\"https://osdf-director.osg-htc.org/\")\n```\n\nOnce `pelfs` is pointed at your federation's director, fsspec commands can be applied to Pelican namespaces. For example:\n\n```python\nhello_world = pelfs.cat('/ospool/uc-shared/public/OSG-Staff/validation/test.txt')\nprint(hello_world)\n```\n\n### Getting an FSMap\n\nSometimes various systems that interact with an fsspec want a key-value mapper rather than a url. To do that, call the `PelicanMap` function with the namespace path and a `PelicanFileSystem` object rather than using the fsspec `get_mapper` call. For example\n\n```python\nfrom pelicanfs.core import PelicanFileSystem, PelicanMap\n\npelfs = PelicanFileSystem(\u201csome-director-url\u201d)\nfile1 = PelicanMap(\u201c/namespace/file/1\u201d, pelfs=pelfs)\nfile2 = PelicanMap(\u201c/namespace/file/2\u201d, pelfs=pelfs)\nds = xarray.open_mfdataset([file1,file2], engine='zarr')\n```\n",
"bugtrack_url": null,
"license": null,
"summary": "An FSSpec Implementation using the Pelican System",
"version": "0.0.3",
"project_urls": {
"Bug Reports": "https://github.com/PelicanPlatform/pelicanfs/issues",
"Homepage": "https://github.com/PelicanPlatform/pelicanfs",
"Pelican Source": "https://github.com/PelicanPlatform/pelican",
"Source": "https://github.com/PelicanPlatform/pelicanfs"
},
"split_keywords": [
"pelican",
" fsspec"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "1e01d59cdf7885947a2c7abdf4f8cd5e67d1b32e1092d0bedffb3ec4398cf2ec",
"md5": "22d644f85d4b07e0bf1d7968da880c1f",
"sha256": "31c354f944f411292f795276211fdf32d30a27438615704c28e86058b68f89e4"
},
"downloads": -1,
"filename": "pelicanfs-0.0.3-py3-none-any.whl",
"has_sig": false,
"md5_digest": "22d644f85d4b07e0bf1d7968da880c1f",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4,>=3.7",
"size": 10201,
"upload_time": "2024-05-01T00:28:19",
"upload_time_iso_8601": "2024-05-01T00:28:19.764781Z",
"url": "https://files.pythonhosted.org/packages/1e/01/d59cdf7885947a2c7abdf4f8cd5e67d1b32e1092d0bedffb3ec4398cf2ec/pelicanfs-0.0.3-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "dcb805b907db54d7c942ebd1f4f0daafba3cbe23423e035ea6364b41393c3ddd",
"md5": "a1d32f113d5fc2cc19cbf34921452ca6",
"sha256": "50485e282279a09c578713d2177ceea2ce00c5a0b84369b3a2fde1cb349e1839"
},
"downloads": -1,
"filename": "pelicanfs-0.0.3.tar.gz",
"has_sig": false,
"md5_digest": "a1d32f113d5fc2cc19cbf34921452ca6",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4,>=3.7",
"size": 10578,
"upload_time": "2024-05-01T00:28:21",
"upload_time_iso_8601": "2024-05-01T00:28:21.389584Z",
"url": "https://files.pythonhosted.org/packages/dc/b8/05b907db54d7c942ebd1f4f0daafba3cbe23423e035ea6364b41393c3ddd/pelicanfs-0.0.3.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-05-01 00:28:21",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "PelicanPlatform",
"github_project": "pelicanfs",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "pelicanfs"
}
JSON
