GCSFS
=====
A Python filesystem abstraction of Google Cloud Storage (GCS) implemented as a `PyFilesystem2 <https://github.com/PyFilesystem/pyfilesystem2>`__ extension.
.. image:: https://img.shields.io/pypi/v/fs-gcsfs.svg
:target: https://pypi.org/project/fs-gcsfs/
.. image:: https://img.shields.io/pypi/pyversions/fs-gcsfs.svg
:target: https://pypi.org/project/fs-gcsfs/
.. image:: https://travis-ci.org/Othoz/gcsfs.svg?branch=master
:target: https://travis-ci.org/Othoz/gcsfs
.. image:: https://readthedocs.org/projects/fs-gcsfs/badge/?version=latest
:target: https://fs-gcsfs.readthedocs.io/en/latest/?badge=latest
With GCSFS, you can interact with `Google Cloud Storage <https://cloud.google.com/storage/>`__ as if it was a regular filesystem.
Apart from the nicer interface, this will highly decouple your code from the underlying storage mechanism: Exchanging the storage backend with an
`in-memory filesystem <https://pyfilesystem2.readthedocs.io/en/latest/reference/memoryfs.html>`__ for testing or any other
filesystem like `S3FS <https://github.com/pyfilesystem/s3fs>`__ becomes as easy as replacing ``gs://bucket_name`` with ``mem://`` or ``s3://bucket_name``.
For a full reference on all the PyFilesystem possibilities, take a look at the
`PyFilesystem Docs <https://pyfilesystem2.readthedocs.io/en/latest/index.html>`__!
Documentation
-------------
- `GCSFS Documentation <http://fs-gcsfs.readthedocs.io/en/latest/>`__
- `PyFilesystem Wiki <https://www.pyfilesystem.org>`__
- `PyFilesystem Reference <https://docs.pyfilesystem.org/en/latest/reference/base.html>`__
Installing
----------
Install the latest GCSFS version by running::
$ pip install fs-gcsfs
Or in case you are using conda::
$ conda install -c conda-forge fs-gcsfs
Examples
--------
Instantiating a filesystem on Google Cloud Storage (for a full reference visit the
`Documentation <http://fs-gcsfs.readthedocs.io/en/latest/index.html#reference>`__):
.. code-block:: python
from fs_gcsfs import GCSFS
gcsfs = GCSFS(bucket_name="mybucket")
Alternatively you can use a `FS URL <https://pyfilesystem2.readthedocs.io/en/latest/openers.html>`__ to open up a filesystem:
.. code-block:: python
from fs import open_fs
gcsfs = open_fs("gs://mybucket/root_path?project=test&api_endpoint=http%3A//localhost%3A8888&strict=False")
Supported query parameters are:
- `project` (str): Google Cloud project to use
- `api_endpoint` (str): URL-encoded endpoint that will be passed to the GCS client's `client_options <https://googleapis.dev/python/google-api-core/latest/client_options.html#google.api_core.client_options.ClientOptions>`__
- `strict` ("True" or "False"): Whether GCSFS will be opened in strict mode
You can use GCSFS like your local filesystem:
.. code-block:: python
>>> from fs_gcsfs import GCSFS
>>> gcsfs = GCSFS(bucket_name="mybucket")
>>> gcsfs.tree()
├── foo
│ ├── bar
│ │ ├── file1.txt
│ │ └── file2.csv
│ └── baz
│ └── file3.txt
└── file4.json
>>> gcsfs.listdir("foo")
["bar", "baz"]
>>> gcsfs.isdir("foo/bar")
True
Uploading a file is as easy as:
.. code-block:: python
from fs_gcsfs import GCSFS
gcsfs = GCSFS(bucket_name="mybucket")
with open("local/path/image.jpg", "rb") as local_file:
with gcsfs.open("path/on/bucket/image.jpg", "wb") as gcs_file:
gcs_file.write(local_file.read())
You can even sync an entire bucket on your local filesystem by using PyFilesystem's utility methods:
.. code-block:: python
from fs_gcsfs import GCSFS
from fs.osfs import OSFS
from fs.copy import copy_fs
gcsfs = GCSFS(bucket_name="mybucket")
local_fs = OSFS("local/path")
copy_fs(gcsfs, local_fs)
For exploring all the possibilities of GCSFS and other filesystems implementing the PyFilesystem interface, we recommend visiting the official
`PyFilesystem Docs <https://pyfilesystem2.readthedocs.io/en/latest/index.html>`__!
Development
-----------
To develop on this project make sure you have `pipenv <https://pipenv.readthedocs.io/en/latest/>`__ installed
and run the following from the root directory of the project::
$ pipenv install --dev --three
This will create a virtualenv with all packages and dev-packages installed.
Tests
-----
All CI tests run against an actual GCS bucket provided by `Othoz <http://othoz.com/>`__.
In order to run the tests against your own bucket,
make sure to set up a `Service Account <https://cloud.google.com/iam/docs/service-accounts>`__ with all necessary permissions:
- storage.objects.get
- storage.objects.list
- storage.objects.create
- storage.objects.update
- storage.objects.delete
All five permissions listed above are e.g. included in the `predefined Cloud Storage IAM Role <https://cloud.google.com/storage/docs/access-control/iam-roles>`__ ``roles/storage.objectAdmin``.
Expose your bucket name as an environment variable ``$TEST_BUCKET`` and run the tests via::
$ pipenv run pytest
Note that the tests mostly wait for I/O, therefore it makes sense to highly parallelize them with `xdist <https://github.com/pytest-dev/pytest-xdist>`__, e.g. by running the tests with::
$ pipenv run pytest -n 10
Credits
-------
Credits go to `S3FS <https://github.com/PyFilesystem/s3fs>`__ which was the main source of inspiration and shares a lot of code with GCSFS.
Raw data
{
"_id": null,
"home_page": "https://github.com/Othoz/gcsfs",
"name": "fs-gcsfs",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.5",
"maintainer_email": "",
"keywords": "pyfilesystem,filesystem,google,gcs,google cloud storage",
"author": "Othoz GmbH",
"author_email": "wiesner@othoz.com",
"download_url": "https://files.pythonhosted.org/packages/ed/93/68cb814940bf55478ea2205a4507cf6a96e2fa290ebf12e00e0b4e014398/fs-gcsfs-1.5.1.tar.gz",
"platform": null,
"description": "\nGCSFS\n=====\n\nA Python filesystem abstraction of Google Cloud Storage (GCS) implemented as a `PyFilesystem2 <https://github.com/PyFilesystem/pyfilesystem2>`__ extension.\n\n\n.. image:: https://img.shields.io/pypi/v/fs-gcsfs.svg\n :target: https://pypi.org/project/fs-gcsfs/\n\n.. image:: https://img.shields.io/pypi/pyversions/fs-gcsfs.svg\n :target: https://pypi.org/project/fs-gcsfs/\n\n.. image:: https://travis-ci.org/Othoz/gcsfs.svg?branch=master\n :target: https://travis-ci.org/Othoz/gcsfs\n\n.. image:: https://readthedocs.org/projects/fs-gcsfs/badge/?version=latest\n :target: https://fs-gcsfs.readthedocs.io/en/latest/?badge=latest\n\n\nWith GCSFS, you can interact with `Google Cloud Storage <https://cloud.google.com/storage/>`__ as if it was a regular filesystem.\n\nApart from the nicer interface, this will highly decouple your code from the underlying storage mechanism: Exchanging the storage backend with an\n`in-memory filesystem <https://pyfilesystem2.readthedocs.io/en/latest/reference/memoryfs.html>`__ for testing or any other\nfilesystem like `S3FS <https://github.com/pyfilesystem/s3fs>`__ becomes as easy as replacing ``gs://bucket_name`` with ``mem://`` or ``s3://bucket_name``.\n\nFor a full reference on all the PyFilesystem possibilities, take a look at the\n`PyFilesystem Docs <https://pyfilesystem2.readthedocs.io/en/latest/index.html>`__!\n\n\nDocumentation\n-------------\n\n- `GCSFS Documentation <http://fs-gcsfs.readthedocs.io/en/latest/>`__\n- `PyFilesystem Wiki <https://www.pyfilesystem.org>`__\n- `PyFilesystem Reference <https://docs.pyfilesystem.org/en/latest/reference/base.html>`__\n\n\nInstalling\n----------\n\nInstall the latest GCSFS version by running::\n\n $ pip install fs-gcsfs\n\nOr in case you are using conda::\n\n $ conda install -c conda-forge fs-gcsfs\n\n\nExamples\n--------\n\nInstantiating a filesystem on Google Cloud Storage (for a full reference visit the\n`Documentation <http://fs-gcsfs.readthedocs.io/en/latest/index.html#reference>`__):\n\n.. code-block:: python\n\n from fs_gcsfs import GCSFS\n gcsfs = GCSFS(bucket_name=\"mybucket\")\n\n\nAlternatively you can use a `FS URL <https://pyfilesystem2.readthedocs.io/en/latest/openers.html>`__ to open up a filesystem:\n\n.. code-block:: python\n\n from fs import open_fs\n gcsfs = open_fs(\"gs://mybucket/root_path?project=test&api_endpoint=http%3A//localhost%3A8888&strict=False\")\n\n\nSupported query parameters are:\n\n- `project` (str): Google Cloud project to use\n- `api_endpoint` (str): URL-encoded endpoint that will be passed to the GCS client's `client_options <https://googleapis.dev/python/google-api-core/latest/client_options.html#google.api_core.client_options.ClientOptions>`__\n- `strict` (\"True\" or \"False\"): Whether GCSFS will be opened in strict mode\n\n\nYou can use GCSFS like your local filesystem:\n\n.. code-block:: python\n\n >>> from fs_gcsfs import GCSFS\n >>> gcsfs = GCSFS(bucket_name=\"mybucket\")\n >>> gcsfs.tree()\n \u251c\u2500\u2500 foo\n \u2502 \u251c\u2500\u2500 bar\n \u2502 \u2502 \u251c\u2500\u2500 file1.txt\n \u2502 \u2502 \u2514\u2500\u2500 file2.csv\n \u2502 \u2514\u2500\u2500 baz\n \u2502 \u2514\u2500\u2500 file3.txt\n \u2514\u2500\u2500 file4.json\n >>> gcsfs.listdir(\"foo\")\n [\"bar\", \"baz\"]\n >>> gcsfs.isdir(\"foo/bar\")\n True\n\n\nUploading a file is as easy as:\n\n.. code-block:: python\n\n from fs_gcsfs import GCSFS\n gcsfs = GCSFS(bucket_name=\"mybucket\")\n with open(\"local/path/image.jpg\", \"rb\") as local_file:\n with gcsfs.open(\"path/on/bucket/image.jpg\", \"wb\") as gcs_file:\n gcs_file.write(local_file.read())\n\n\nYou can even sync an entire bucket on your local filesystem by using PyFilesystem's utility methods:\n\n.. code-block:: python\n\n from fs_gcsfs import GCSFS\n from fs.osfs import OSFS\n from fs.copy import copy_fs\n\n gcsfs = GCSFS(bucket_name=\"mybucket\")\n local_fs = OSFS(\"local/path\")\n\n copy_fs(gcsfs, local_fs)\n\n\nFor exploring all the possibilities of GCSFS and other filesystems implementing the PyFilesystem interface, we recommend visiting the official\n`PyFilesystem Docs <https://pyfilesystem2.readthedocs.io/en/latest/index.html>`__!\n\n\nDevelopment\n-----------\n\nTo develop on this project make sure you have `pipenv <https://pipenv.readthedocs.io/en/latest/>`__ installed\nand run the following from the root directory of the project::\n\n $ pipenv install --dev --three\n\nThis will create a virtualenv with all packages and dev-packages installed.\n\n\nTests\n-----\nAll CI tests run against an actual GCS bucket provided by `Othoz <http://othoz.com/>`__.\n\nIn order to run the tests against your own bucket,\nmake sure to set up a `Service Account <https://cloud.google.com/iam/docs/service-accounts>`__ with all necessary permissions:\n\n- storage.objects.get\n- storage.objects.list\n- storage.objects.create\n- storage.objects.update\n- storage.objects.delete\n\nAll five permissions listed above are e.g. included in the `predefined Cloud Storage IAM Role <https://cloud.google.com/storage/docs/access-control/iam-roles>`__ ``roles/storage.objectAdmin``.\n\nExpose your bucket name as an environment variable ``$TEST_BUCKET`` and run the tests via::\n\n $ pipenv run pytest\n\nNote that the tests mostly wait for I/O, therefore it makes sense to highly parallelize them with `xdist <https://github.com/pytest-dev/pytest-xdist>`__, e.g. by running the tests with::\n\n $ pipenv run pytest -n 10\n\n\nCredits\n-------\n\nCredits go to `S3FS <https://github.com/PyFilesystem/s3fs>`__ which was the main source of inspiration and shares a lot of code with GCSFS.\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "A PyFilesystem interface to Google Cloud Storage",
"version": "1.5.1",
"split_keywords": [
"pyfilesystem",
"filesystem",
"google",
"gcs",
"google cloud storage"
],
"urls": [
{
"comment_text": "",
"digests": {
"md5": "93b438cf3741fc682c199b1aacb7deeb",
"sha256": "35f52862723ed2214f92ad941527066d10a2261a95b1b991dbdacc162dab5045"
},
"downloads": -1,
"filename": "fs-gcsfs-1.5.1.tar.gz",
"has_sig": false,
"md5_digest": "93b438cf3741fc682c199b1aacb7deeb",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.5",
"size": 26892,
"upload_time": "2022-06-03T09:38:11",
"upload_time_iso_8601": "2022-06-03T09:38:11.245192Z",
"url": "https://files.pythonhosted.org/packages/ed/93/68cb814940bf55478ea2205a4507cf6a96e2fa290ebf12e00e0b4e014398/fs-gcsfs-1.5.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2022-06-03 09:38:11",
"github": true,
"gitlab": false,
"bitbucket": false,
"github_user": "Othoz",
"github_project": "gcsfs",
"travis_ci": true,
"coveralls": false,
"github_actions": false,
"lcname": "fs-gcsfs"
}
JSON
