# Universal Pathlib
[](https://pypi.org/project/universal_pathlib/)
[](https://pypi.org/project/universal_pathlib/)
[](https://github.com/fsspec/universal_pathlib/blob/main/LICENSE)
[](https://anaconda.org/conda-forge/universal_pathlib)
[](https://github.com/fsspec/universal_pathlib/actions/workflows/tests.yml)
[](https://github.com/fsspec/universal_pathlib/issues)
[](https://github.com/psf/black)
[](./CHANGELOG.md)
Universal Pathlib is a Python library that extends the [`pathlib.Path`][pathlib]
API to support a variety of backend filesystems via [`filesystem_spec`][fsspec].
[pathlib]: https://docs.python.org/3/library/pathlib.html
[fsspec]: https://filesystem-spec.readthedocs.io/en/latest/intro.html
## Installation
Install the latest version of `universal_pathlib` with pip or conda. Please note
that while this will install `fsspec` as a dependency, for some filesystems, you
have to install additional packages. For example, to use S3, you need to install
`s3fs`, or better depend on `fsspec[s3]`:
### PyPI
```bash
python -m pip install universal_pathlib
```
### conda
```bash
conda install -c conda-forge universal_pathlib
```
### Adding universal_pathlib to your project
Below is a `pyproject.toml` based example for adding `universal_pathlib` to your
project as a dependency if you want to use it with `s3` and `http` filesystems:
```toml
[project]
name = "myproject"
requires-python = ">=3.8"
dependencies = [
"universal_pathlib>=0.2.5",
"fsspec[s3,http]",
]
```
See [filesystem_spec/pyproject.toml][fsspec-pyproject-toml] for an overview of the available
fsspec extras.
[fsspec-pyproject-toml]:
https://github.com/fsspec/filesystem_spec/blob/master/pyproject.toml#L26
## Basic Usage
```pycon
# pip install universal_pathlib fsspec[s3]
>>> from upath import UPath
>>>
>>> s3path = UPath("s3://test_bucket") / "example.txt"
>>> s3path.name
example.txt
>>> s3path.stem
example
>>> s3path.suffix
.txt
>>> s3path.exists()
True
>>> s3path.read_text()
'Hello World'
```
For more examples, see the [example notebook here][example-notebook].
[example-notebook]: notebooks/examples.ipynb
### Currently supported filesystems (and protocols)
- `file:` and `local:` Local filesystem
- `memory:` Ephemeral filesystem in RAM
- `az:`, `adl:`, `abfs:` and `abfss:` Azure Storage _(requires `adlfs`)_
- `data:` RFC 2397 style data URLs _(requires `fsspec>=2023.12.2`)_
- `github:` GitHub repository filesystem
- `http:` and `https:` HTTP(S)-based filesystem
- `hdfs:` Hadoop distributed filesystem
- `gs:` and `gcs:` Google Cloud Storage _(requires `gcsfs`)_
- `s3:` and `s3a:` AWS S3 _(requires `s3fs` to be installed)_
- `sftp:` and `ssh:` SFTP and SSH filesystems _(requires `paramiko`)_
- `smb:` SMB filesystems _(requires `smbprotocol`)_
- `webdav`, `webdav+http:` and `webdav+https:` WebDAV-based filesystem on top of
HTTP(S) _(requires `webdav4[fsspec]`)_
It is likely, that other fsspec-compatible filesystems are supported through the
default implementation. But because they are not tested in the universal_pathlib
test-suite, correct behavior is not guaranteed. If you encounter any issues with
a specific filesystem using the default implementation, please open an issue. We
are happy to add support for other filesystems via custom UPath implementations.
And of course, contributions for new filesystems are welcome!
### Class hierarchy
The class hierarchy for `UPath` implementations and their relation to the stdlib
`pathlib` classes are visualized in the following diagram:
```mermaid
flowchart TB
subgraph s0[pathlib]
A---> B
A--> AP
A--> AW
B--> BP
AP---> BP
B--> BW
AW---> BW
end
subgraph s1[upath]
B ---> U
U --> UP
U --> UW
BP --> UP
BW --> UW
U --> UL
U --> US3
U --> UH
U -.-> UO
end
A(PurePath)
AP(PurePosixPath)
AW(PureWindowsPath)
B(Path)
BP(PosixPath)
BW(WindowsPath)
U(UPath)
UP(PosixUPath)
UW(WindowsUPath)
UL(FilePath)
US3(S3Path)
UH(HttpPath)
UO(...Path)
classDef np fill:#f7f7f7,stroke:#2166ac,stroke-width:2px,color:#333
classDef nu fill:#f7f7f7,stroke:#b2182b,stroke-width:2px,color:#333
class A,AP,AW,B,BP,BW,UP,UW np
class U,UL,US3,UH,UO nu
style UO stroke-dasharray: 3 3
style s0 fill:none,stroke:#07b,stroke-width:3px,stroke-dasharray:3,color:#07b
style s1 fill:none,stroke:#d02,stroke-width:3px,stroke-dasharray:3,color:#d02
```
When instantiating `UPath` the returned instance type is determined by the path,
or better said, the "protocol" that was provided to the constructor. The `UPath`
class will return a registered implementation for the protocol, if available. If
no specialized implementation can be found but the protocol is available through
`fsspec`, it will return a `UPath` instance and provide filesystem access with a
default implementation. Please note the default implementation can not guarantee
correct behavior for filesystems that are not tested in the test-suite.
### Local paths and url paths
If a local path is provided `UPath` will return a `PosixUPath` or `WindowsUPath`
instance. These two implementations are 100% compatible with the `PosixPath` and
`WindowsPath` classes of their specific Python version. They're tested against a
large subset of the CPython pathlib test-suite to ensure compatibility.
If a local urlpath is provided, i.e. a "file://" or "local://" URI, the returned
instance type will be a `FilePath` instance. This class is a subclass of `UPath`
that provides file access via `LocalFileSystem` from `fsspec`. You can use it to
ensure that all your local file access is done through `fsspec` as well.
### UPath public class API
The public class interface of `UPath` extends `pathlib.Path` via attributes that
simplify interaction with `filesystem_spec`. Think of the `UPath` class in terms
of the following code:
```python
from pathlib import Path
from typing import Any, Mapping
from fsspec import AbstractFileSystem
class UPath(Path):
# the real implementation is more complex, but this is the general idea
@property
def protocol(self) -> str:
"""The fsspec protocol for the path."""
@property
def storage_options(self) -> Mapping[str, Any]:
"""The fsspec storage options for the path."""
@property
def path(self) -> str:
"""The path that a fsspec filesystem can use."""
@property
def fs(self) -> AbstractFileSystem:
"""The cached fsspec filesystem instance for the path."""
```
These attributes are used to provide a public interface to move from the `UPath`
instance to more fsspec specific code:
```python
from upath import UPath
from fsspec import filesystem
p = UPath("s3://bucket/file.txt", anon=True)
fs = filesystem(p.protocol, **p.storage_options) # equivalent to p.fs
with fs.open(p.path) as f:
data = f.read()
```
## Advanced Usage
If you want to create your own UPath implementations, there are multiple ways to
customize your subclass behavior. Here are a few things to keep in mind when you
create your own UPath implementation:
### UPath's constructor, `upath.registry`, and subclassing
When instantiating `UPath(...)` the `UPath.__new__()` method determines the path
protocol and returns a registered implementation for the protocol, if available.
The registered implementations are mapped in the `upath.registry` module. When a
protocol is not registered, `universal_pathlib` checks if the protocol is mapped
to an `fsspec` filesystem. If so, it returns an instance of `UPath` and provides
filesystem access through the default implementation. The protocol is determined
by either looking at the URI scheme of the first argument to the constructor, or
by using the `protocol` keyword argument:
```python
from upath import UPath
from upath.implementations.cloud import S3Path
from upath.implementations.memory import MemoryPath
p0 = UPath("s3://bucket/file.txt")
assert p0.protocol == "s3"
assert type(p0) is S3Path
assert isinstance(p0, UPath)
p1 = UPath("/some/path/file.txt", protocol="memory")
assert p1.protocol == "memory"
assert type(p1) is MemoryPath
assert isinstance(p1, UPath)
# the ftp filesystem current has no custom UPath implementation and is not
# tested in the universal_pathlib test-suite. Therefore, the default UPath
# implementation is returned, and a warning is emitted on instantiation.
p2 = UPath("ftp://ftp.ncbi.nih.gov/snp/archive")
assert p2.protocol == "ftp"
assert type(p2) is UPath
```
This has some implications for custom UPath subclasses. We'll go through the two
main cases where you might want to create a custom UPath implementation:
#### Case 1: Custom filesystem works with default UPath implementation
Let's say you would like to add a new implementation of your "myproto" protocol.
You already built a custom AbstractFileSystem implementation for "myproto" which
you have registered through `fsspec.registry`. In some cases it is possible that
the custom filesystem class already works with `UPath`'s default implementation,
and you don't need to necessarily create a custom UPath implementation:
```python
import fsspec.registry
from fsspec.spec import AbstractFileSystem
class MyProtoFileSystem(AbstractFileSystem):
protocol = ("myproto",)
... # your custom implementation
fsspec.registry.register_implementation("myproto", MyProtoFileSystem)
from upath import UPath
p = UPath("myproto:///my/proto/path")
assert type(p) is UPath
assert p.protocol == "myproto"
assert isinstance(p.fs, MyProtoFileSystem)
```
#### Case 2: Custom filesystem requires a custom UPath implementation
Sometimes the default implementation isn't sufficient and some method(s) have to
be overridden to provide correct behavior. In this case, create a custom `UPath`
implementation:
```python
from upath import UPath
class MyProtoPath(UPath):
def mkdir(self, mode=0o777, parents=False, exist_ok=False):
something = {...: ...} # fixes to make MyProtoFileSystem.mkdir work
self.fs.mkdir(self.path, **something)
def path(self):
path = super().path
if path.startswith("/"):
return path[1:] # MyProtoFileSystem needs the path without "/"
return path
```
If you use your implementation directly via `MyProtoPath("myproto:///a/b")`, you
can use this implementation already as is. If you want a call to `UPath(...)` to
return your custom implementation when the detected protocol is `"myproto"`, you
need to register your implementation. The next section explains your options.
Also note: In case you develop a custom `UPath` implementation, please feel free
to open an issue to discuss integrating it in `universal_pathlib`.
#### Implementation registration dynamically from Python
You can register your custom UPath implementation dynamically from Python:
```python
# for example: mymodule/submodule.py
from upath import UPath
from upath.registry import register_implementation
class MyProtoPath(UPath):
... # your custom implementation
register_implementation("myproto", MyProtoPath)
```
#### Implementation registration on installation via entry points
If you distribute your implementation in your own Python package, you can inform
`universal_pathlib` about your implementation via the `entry_points` mechanism:
```
# pyproject.toml
[project.entry-points."universal_pathlib.implementations"]
myproto = "my_module.submodule:MyPath"
```
```
# setup.cfg
[options.entry_points]
universal_pathlib.implementations =
myproto = my_module.submodule:MyPath
```
Chose the method that fits your use-case best. If you have questions, open a new
issue in the `universal_pathlib` repository. We are happy to help you!
### Customization options for UPath subclasses
#### Filesystem access methods
Once you thoroughly test your custom UPath implementation, it's likely that some
methods need to be overridden to provide correct behavior compared to `stdlib`'s
`pathlib.Path` class. The most common issue is that for certain edge cases, your
implementation is not raising the same exceptions compared to the `pathlib.Path`
class. Or that the `UPath.path` property needs some prefix removed or added.
```python
class MyProtoPath(UPath):
@property
def path(self) -> str:
if p := self.path.startswith("/"):
p = p[1:]
return p
def mkdir(self, mode=0o777, parents=False, exist_ok=False):
if some_edge_case:
raise FileExistsError(str(self))
super().mkdir(mode=mode, parents=parents, exist_ok=exist_ok)
def is_file(self):
return self.fs.isfile(self.path, myproto_option=123)
```
#### Storage option parsing
It's possible that you might want to extract additional storage options from the
user provided arguments to you constructor. You can provide a custom classmethod
for `_parse_storage_options`:
```python
import os
class MyProtoPath(UPath):
@classmethod
def _parse_storage_options(
cls, urlpath: str, protocol: str, storage_options: Mapping[str, Any]
) -> dict[str, Any]:
if "SOME_VAR" in os.environ:
storage_options["some_var"] = os.environ["SOME_VAR"]
storage_options["my_proto_caching"] = True
storage_options["extra"] = get_setting_from_path(urlpath)
return storage_options
```
#### Fsspec filesystem instantiation
To have more control over fsspec filesystem instantiation you can write a custom
`_fs_factory` classmethod:
```python
class MyProtoPath(UPath):
@classmethod
def _fs_factory(
cls, urlpath: str, protocol: str, storage_options: Mapping[str, Any]
) -> AbstractFileSystem:
myfs = ... # custom code that creates a AbstractFileSystem instance
return myfs
```
#### Init argument parsing
In special cases you need to take more control over how the init args are parsed
for your custom subclass. You can override `__init__` or the `UPath` classmethod
`_transform_init_args`. The latter handles pickling of your custom subclass in a
better way in case you modify storage options or the protocol.
```python
class MyProtoPath(UPath):
@classmethod
def _transform_init_args(
cls,
args: tuple[str | os.PathLike, ...],
protocol: str,
storage_options: dict[str, Any],
) -> tuple[tuple[str | os.PathLike, ...], str, dict[str, Any]]:
# check the cloud, http or webdav implementations for examples
...
return args, protocol, storage_options
```
#### Stopping UPath's subclass dispatch mechanism
There are cases for which you want to disable the protocol dispatch mechanism of
the `UPath.__new__` constructor. For example if you want to extend the class API
of your `UPath` implementation, and use it as the base class for other, directly
instantiated subclasses. Together with other customization options this can be a
useful feature. Please be aware that in this case all protocols are handled with
the default implementation in UPath. Please always feel free to open an issue in
the issue tracker to discuss your use case. We're happy to help with finding the
most maintainable solution.
```python
class ExtraUPath(UPath):
_protocol_dispatch = False # disable the registry return an ExtraUPath
def some_extra_method(self) -> str:
return "hello world"
assert ExtraUPath("s3://bucket/file.txt").some_extra_method() == "hello world"
```
## Migration Guide
UPath's internal implementation is likely going to change with larger changes in
CPython's stdlib `pathlib` landing in the next Python versions (`3.13`, `3.14`).
To reduce the problems for user code, when these changes are landing in `UPath`,
there have been some significant changes in `v0.2.0`. This migration guide tries
to help migrating code that extensively relies on private implementation details
of the `UPath` class of versions `v0.1.x` to the new and better supported public
interface of `v0.2.0`
### migrating to `v0.2.0`
### _FSSpecAccessor subclasses with custom filesystem access methods
If you implemented a custom accessor subclass, it is now recommended to override
the corresponding `UPath` methods in your subclass directly:
```python
# OLD: v0.1.x
from upath.core import UPath, _FSSpecAccessor
class MyAccessor(_FSSpecAccessor):
def exists(self, path, **kwargs):
# custom code
return path.fs.exists(self._format_path(path), **kwargs)
def touch(self, path, **kwargs):
# custom
return path.fs.touch(self._format_path(path), **kwargs)
class MyPath(UPath):
_default_accessor = MyAccessor
# NEW: v0.2.0+
from upath import UPath
class MyPath(UPath):
def exists(self, *, follow_symlinks=True):
kwargs = {} # custom code
return self.fs.exists(self.path, **kwargs)
def touch(self, mode=0o666, exist_ok=True):
kwargs = {} # custom code
self.fs.touch(self.path, **kwargs)
```
### _FSSpecAccessor subclasses with custom `__init__` method
If you implemented a custom `__init__` method for your accessor subclass usually
the intention is to customize how the fsspec filesystem instance is created. You
have two options to recreate this with the new implementation. Chose one or both
dependent on the level of control you need.
```python
# OLD: v0.1.x
import fsspec
from upath.core import UPath, _FSSpecAccessor
class MyAccessor(_FSSpecAccessor):
def __init__(self, parsed_url: SplitResult | None, **kwargs: Any) -> None:
# custom code
protocol = ...
storage_options = ...
self._fs = fsspec.filesystem(protocol, storage_options)
class MyPath(UPath):
_default_accessor = MyAccessor
# NEW: v0.2.0+
from upath import UPath
class MyPath(UPath):
@classmethod
def _parse_storage_options(
cls, urlpath: str, protocol: str, storage_options: Mapping[str, Any]
) -> dict[str, Any]:
# custom code to change storage_options
storage_options = ...
return storage_options
@classmethod
def _fs_factory(
cls, urlpath: str, protocol: str, storage_options: Mapping[str, Any]
) -> AbstractFileSystem:
# custom code to instantiate fsspec filesystem
protocol = ...
storage_options = ... # note changes to storage_options here won't
# show up in MyPath().storage_options
return fsspec.filesystem(protocol, **storage_options)
```
### Access to `._accessor`
The `_accessor` attribute and the `_FSSpecAccessor` class is deprecated. In case
you need direct access to the underlying filesystem, just access `UPath().fs`.
```python
# OLD: v0.1.x
from upath.core import UPath
class MyPath(UPath):
def mkdir(self, mode=0o777, parents=False, exist_ok=False):
self._accessor.mkdir(...) # custom access to the underlying fs...
# NEW: v0.2.0+
from upath import UPath
class MyPath(UPath):
def mkdir(self, mode=0o777, parents=False, exist_ok=False):
self.fs.mkdir(...)
```
### Access to `._path`, `._kwargs`, `._drv`, `._root`, `._parts`
If you access one of the listed private attributes directly, move your code over
to the following public versions:
| _deprecated_ | `v0.2.0+` |
|:------------------|:--------------------------|
| `UPath()._path` | `UPath().path` |
| `UPath()._kwargs` | `UPath().storage_options` |
| `UPath()._drv` | `UPath().drive` |
| `UPath()._root` | `UPath().root` |
| `UPath()._parts` | `UPath().parts` |
### Access to `._url`
Be aware that the `._url` attribute will likely be deprecated once `UPath()` has
support for uri fragments and uri query parameters through a public api. In case
you are interested in contributing this functionality, please open an issue!
### Calling `_from_parts`, `_parse_args`, `_format_parsed_parts`
If your code is currently calling any of the three above listed classmethods, it
relies on functionality based on the implementation of `pathlib` in Python up to
`3.11`. In `universal_pathlib` we vendor code that allows the `UPath()` class to
be based on the `3.12` implementation of `pathlib.Path` alone. Usually, usage of
those classmethods occurs when copying some code of the internal implementations
of methods of the `UPath` `0.1.4` classes.
- To reproduce custom `_format_parsed_parts` methods in `v0.2.0`, try overriding
`UPath().path` and/or `UPath().with_segments()`.
- Custom `_from_parts` and `_parse_args` classmethods can now be implemented via
the `_transform_init_args` method or via more functionality in the new flavour
class. Please open an issue for discussion in case you have this use case.
### Custom `_URIFlavour` classes
The `_URIFlavour` class was removed from `universal_pathlib` and the new flavour
class for fsspec filesystem path operations now lives in `upath._flavour`. As of
now the internal FSSpecFlavour is experimental. In a future Python version, it's
likely that a flavour or flavour-like base class will become public, that allows
us to base our internal implementation on. Until then, if you find yourself in a
situation where a custom path flavour would solve your problem, please feel free
to open an issue for discussion. We're happy to find a maintainable solution.
### Using `.parse_parts()`, `.casefold()`, `.join_parsed_parts()` of `._flavour`
These methods of the `._flavour` attribute of `pathlib.Path()` and `UPath()` are
specific to `pathlib` of Python versions up to `3.11`. `UPath()` is now based on
the `3.12` implementation of `pathlib.Path`. Please refer to the implementations
of the `upath._flavour` submodule to see how you could avoid using them.
## Known issues solvable by installing newer upstream dependencies
Some issues in `UPath`'s behavior with specific fsspec filesystems are fixed via
installation of a newer version of its upstream dependencies. Below you can find
a list of known issues and their solutions. We attempt to keep this list updated
whenever we encounter more:
- **UPath().glob()**:
`fsspec` fixed glob behavior when handling `**` patterns in `fsspec>=2023.9.0`
- **GCSPath().mkdir()**:
a few mkdir quirks are solved by installing `gcsfs>=2022.7.1`
- **fsspec.filesystem(WebdavPath().protocol)**
the webdav protocol was added to fsspec in version `fsspec>=2022.5.0`
- **stat.S_ISDIR(HTTPPath().stat().st_mode)**
requires `fsspec>=2024.2.0` to correctly return `True` for directories
## Contributing
Contributions are very welcome.
To learn more, see the [Contributor Guide](CONTRIBUTING.rst).
## License
Distributed under the terms of the [MIT license](LICENSE),
*universal_pathlib* is free and open source software.
## Issues
If you encounter any problems, or if you create your own implementations and run
into limitations, please [file an issue][issues] with a detailed description. We
are always happy to help with any problems you might encounter.
[issues]: https://github.com/fsspec/universal_pathlib/issues
Raw data
{
"_id": null,
"home_page": null,
"name": "universal-pathlib",
"maintainer": "Norman Rzepka",
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": "Andreas Poehlmann <andreas@poehlmann.io>",
"keywords": "filesystem-spec, pathlib",
"author": null,
"author_email": "Andrew Fulton <andrewfulton9@gmail.com>",
"download_url": "https://files.pythonhosted.org/packages/24/67/6c31ba464eafda05c677628dd7859ed4904597a78694d9cc81b593c6bad2/universal_pathlib-0.2.5.tar.gz",
"platform": null,
"description": "# Universal Pathlib\n\n[](https://pypi.org/project/universal_pathlib/)\n[](https://pypi.org/project/universal_pathlib/)\n[](https://github.com/fsspec/universal_pathlib/blob/main/LICENSE)\n[](https://anaconda.org/conda-forge/universal_pathlib)\n\n[](https://github.com/fsspec/universal_pathlib/actions/workflows/tests.yml)\n[](https://github.com/fsspec/universal_pathlib/issues)\n[](https://github.com/psf/black)\n[](./CHANGELOG.md)\n\nUniversal Pathlib is a Python library that extends the [`pathlib.Path`][pathlib]\nAPI to support a variety of backend filesystems via [`filesystem_spec`][fsspec].\n\n[pathlib]: https://docs.python.org/3/library/pathlib.html\n[fsspec]: https://filesystem-spec.readthedocs.io/en/latest/intro.html\n\n\n## Installation\n\nInstall the latest version of `universal_pathlib` with pip or conda. Please note\nthat while this will install `fsspec` as a dependency, for some filesystems, you\nhave to install additional packages. For example, to use S3, you need to install\n`s3fs`, or better depend on `fsspec[s3]`:\n\n### PyPI\n\n```bash\npython -m pip install universal_pathlib\n```\n\n### conda\n\n```bash\nconda install -c conda-forge universal_pathlib\n```\n\n### Adding universal_pathlib to your project\n\nBelow is a `pyproject.toml` based example for adding `universal_pathlib` to your\nproject as a dependency if you want to use it with `s3` and `http` filesystems:\n\n```toml\n[project]\nname = \"myproject\"\nrequires-python = \">=3.8\"\ndependencies = [\n \"universal_pathlib>=0.2.5\",\n \"fsspec[s3,http]\",\n]\n```\n\nSee [filesystem_spec/pyproject.toml][fsspec-pyproject-toml] for an overview of the available\nfsspec extras.\n\n[fsspec-pyproject-toml]:\n https://github.com/fsspec/filesystem_spec/blob/master/pyproject.toml#L26\n\n\n## Basic Usage\n\n```pycon\n# pip install universal_pathlib fsspec[s3]\n>>> from upath import UPath\n>>>\n>>> s3path = UPath(\"s3://test_bucket\") / \"example.txt\"\n>>> s3path.name\nexample.txt\n>>> s3path.stem\nexample\n>>> s3path.suffix\n.txt\n>>> s3path.exists()\nTrue\n>>> s3path.read_text()\n'Hello World'\n```\n\nFor more examples, see the [example notebook here][example-notebook].\n\n[example-notebook]: notebooks/examples.ipynb\n\n### Currently supported filesystems (and protocols)\n\n- `file:` and `local:` Local filesystem\n- `memory:` Ephemeral filesystem in RAM\n- `az:`, `adl:`, `abfs:` and `abfss:` Azure Storage _(requires `adlfs`)_\n- `data:` RFC 2397 style data URLs _(requires `fsspec>=2023.12.2`)_\n- `github:` GitHub repository filesystem\n- `http:` and `https:` HTTP(S)-based filesystem\n- `hdfs:` Hadoop distributed filesystem\n- `gs:` and `gcs:` Google Cloud Storage _(requires `gcsfs`)_\n- `s3:` and `s3a:` AWS S3 _(requires `s3fs` to be installed)_\n- `sftp:` and `ssh:` SFTP and SSH filesystems _(requires `paramiko`)_\n- `smb:` SMB filesystems _(requires `smbprotocol`)_\n- `webdav`, `webdav+http:` and `webdav+https:` WebDAV-based filesystem on top of\n HTTP(S) _(requires `webdav4[fsspec]`)_\n\nIt is likely, that other fsspec-compatible filesystems are supported through the\ndefault implementation. But because they are not tested in the universal_pathlib\ntest-suite, correct behavior is not guaranteed. If you encounter any issues with\na specific filesystem using the default implementation, please open an issue. We\nare happy to add support for other filesystems via custom UPath implementations.\nAnd of course, contributions for new filesystems are welcome!\n\n### Class hierarchy\n\nThe class hierarchy for `UPath` implementations and their relation to the stdlib\n`pathlib` classes are visualized in the following diagram:\n\n```mermaid\nflowchart TB\n subgraph s0[pathlib]\n A---> B\n A--> AP\n A--> AW\n\n B--> BP\n AP---> BP\n B--> BW\n AW---> BW\n end\n subgraph s1[upath]\n B ---> U\n U --> UP\n U --> UW\n BP --> UP\n BW --> UW\n U --> UL\n U --> US3\n U --> UH\n U -.-> UO\n end\n\n A(PurePath)\n AP(PurePosixPath)\n AW(PureWindowsPath)\n B(Path)\n BP(PosixPath)\n BW(WindowsPath)\n\n U(UPath)\n UP(PosixUPath)\n UW(WindowsUPath)\n UL(FilePath)\n US3(S3Path)\n UH(HttpPath)\n UO(...Path)\n\n classDef np fill:#f7f7f7,stroke:#2166ac,stroke-width:2px,color:#333\n classDef nu fill:#f7f7f7,stroke:#b2182b,stroke-width:2px,color:#333\n\n class A,AP,AW,B,BP,BW,UP,UW np\n class U,UL,US3,UH,UO nu\n\n style UO stroke-dasharray: 3 3\n\n style s0 fill:none,stroke:#07b,stroke-width:3px,stroke-dasharray:3,color:#07b\n style s1 fill:none,stroke:#d02,stroke-width:3px,stroke-dasharray:3,color:#d02\n```\n\nWhen instantiating `UPath` the returned instance type is determined by the path,\nor better said, the \"protocol\" that was provided to the constructor. The `UPath`\nclass will return a registered implementation for the protocol, if available. If\nno specialized implementation can be found but the protocol is available through\n`fsspec`, it will return a `UPath` instance and provide filesystem access with a\ndefault implementation. Please note the default implementation can not guarantee\ncorrect behavior for filesystems that are not tested in the test-suite.\n\n### Local paths and url paths\n\nIf a local path is provided `UPath` will return a `PosixUPath` or `WindowsUPath`\ninstance. These two implementations are 100% compatible with the `PosixPath` and\n`WindowsPath` classes of their specific Python version. They're tested against a\nlarge subset of the CPython pathlib test-suite to ensure compatibility.\n\nIf a local urlpath is provided, i.e. a \"file://\" or \"local://\" URI, the returned\ninstance type will be a `FilePath` instance. This class is a subclass of `UPath`\nthat provides file access via `LocalFileSystem` from `fsspec`. You can use it to\nensure that all your local file access is done through `fsspec` as well.\n\n### UPath public class API\n\nThe public class interface of `UPath` extends `pathlib.Path` via attributes that\nsimplify interaction with `filesystem_spec`. Think of the `UPath` class in terms\nof the following code:\n\n```python\nfrom pathlib import Path\nfrom typing import Any, Mapping\nfrom fsspec import AbstractFileSystem\n\nclass UPath(Path):\n # the real implementation is more complex, but this is the general idea\n\n @property\n def protocol(self) -> str:\n \"\"\"The fsspec protocol for the path.\"\"\"\n\n @property\n def storage_options(self) -> Mapping[str, Any]:\n \"\"\"The fsspec storage options for the path.\"\"\"\n\n @property\n def path(self) -> str:\n \"\"\"The path that a fsspec filesystem can use.\"\"\"\n\n @property\n def fs(self) -> AbstractFileSystem:\n \"\"\"The cached fsspec filesystem instance for the path.\"\"\"\n\n```\n\nThese attributes are used to provide a public interface to move from the `UPath`\ninstance to more fsspec specific code:\n\n```python\nfrom upath import UPath\nfrom fsspec import filesystem\n\np = UPath(\"s3://bucket/file.txt\", anon=True)\n\nfs = filesystem(p.protocol, **p.storage_options) # equivalent to p.fs\n\nwith fs.open(p.path) as f:\n data = f.read()\n```\n\n## Advanced Usage\n\nIf you want to create your own UPath implementations, there are multiple ways to\ncustomize your subclass behavior. Here are a few things to keep in mind when you\ncreate your own UPath implementation:\n\n### UPath's constructor, `upath.registry`, and subclassing\n\nWhen instantiating `UPath(...)` the `UPath.__new__()` method determines the path\nprotocol and returns a registered implementation for the protocol, if available.\nThe registered implementations are mapped in the `upath.registry` module. When a\nprotocol is not registered, `universal_pathlib` checks if the protocol is mapped\nto an `fsspec` filesystem. If so, it returns an instance of `UPath` and provides\nfilesystem access through the default implementation. The protocol is determined\nby either looking at the URI scheme of the first argument to the constructor, or\nby using the `protocol` keyword argument:\n\n```python\nfrom upath import UPath\nfrom upath.implementations.cloud import S3Path\nfrom upath.implementations.memory import MemoryPath\n\np0 = UPath(\"s3://bucket/file.txt\")\nassert p0.protocol == \"s3\"\nassert type(p0) is S3Path\nassert isinstance(p0, UPath)\n\np1 = UPath(\"/some/path/file.txt\", protocol=\"memory\")\nassert p1.protocol == \"memory\"\nassert type(p1) is MemoryPath\nassert isinstance(p1, UPath)\n\n# the ftp filesystem current has no custom UPath implementation and is not\n# tested in the universal_pathlib test-suite. Therefore, the default UPath\n# implementation is returned, and a warning is emitted on instantiation.\np2 = UPath(\"ftp://ftp.ncbi.nih.gov/snp/archive\")\nassert p2.protocol == \"ftp\"\nassert type(p2) is UPath\n```\n\nThis has some implications for custom UPath subclasses. We'll go through the two\nmain cases where you might want to create a custom UPath implementation:\n\n#### Case 1: Custom filesystem works with default UPath implementation\n\nLet's say you would like to add a new implementation of your \"myproto\" protocol.\nYou already built a custom AbstractFileSystem implementation for \"myproto\" which\nyou have registered through `fsspec.registry`. In some cases it is possible that\nthe custom filesystem class already works with `UPath`'s default implementation,\nand you don't need to necessarily create a custom UPath implementation:\n\n```python\nimport fsspec.registry\nfrom fsspec.spec import AbstractFileSystem\n\nclass MyProtoFileSystem(AbstractFileSystem):\n protocol = (\"myproto\",)\n ... # your custom implementation\n\nfsspec.registry.register_implementation(\"myproto\", MyProtoFileSystem)\n\nfrom upath import UPath\n\np = UPath(\"myproto:///my/proto/path\")\nassert type(p) is UPath\nassert p.protocol == \"myproto\"\nassert isinstance(p.fs, MyProtoFileSystem)\n```\n\n#### Case 2: Custom filesystem requires a custom UPath implementation\n\nSometimes the default implementation isn't sufficient and some method(s) have to\nbe overridden to provide correct behavior. In this case, create a custom `UPath`\nimplementation:\n\n```python\nfrom upath import UPath\n\nclass MyProtoPath(UPath):\n\n def mkdir(self, mode=0o777, parents=False, exist_ok=False):\n something = {...: ...} # fixes to make MyProtoFileSystem.mkdir work\n self.fs.mkdir(self.path, **something)\n\n def path(self):\n path = super().path\n if path.startswith(\"/\"):\n return path[1:] # MyProtoFileSystem needs the path without \"/\"\n return path\n```\n\nIf you use your implementation directly via `MyProtoPath(\"myproto:///a/b\")`, you\ncan use this implementation already as is. If you want a call to `UPath(...)` to\nreturn your custom implementation when the detected protocol is `\"myproto\"`, you\nneed to register your implementation. The next section explains your options.\n\nAlso note: In case you develop a custom `UPath` implementation, please feel free\nto open an issue to discuss integrating it in `universal_pathlib`.\n\n#### Implementation registration dynamically from Python\n\nYou can register your custom UPath implementation dynamically from Python:\n\n```python\n# for example: mymodule/submodule.py\nfrom upath import UPath\nfrom upath.registry import register_implementation\n\nclass MyProtoPath(UPath):\n ... # your custom implementation\n\nregister_implementation(\"myproto\", MyProtoPath)\n```\n\n#### Implementation registration on installation via entry points\n\nIf you distribute your implementation in your own Python package, you can inform\n`universal_pathlib` about your implementation via the `entry_points` mechanism:\n\n```\n# pyproject.toml\n[project.entry-points.\"universal_pathlib.implementations\"]\nmyproto = \"my_module.submodule:MyPath\"\n```\n\n```\n# setup.cfg\n[options.entry_points]\nuniversal_pathlib.implementations =\n myproto = my_module.submodule:MyPath\n```\n\nChose the method that fits your use-case best. If you have questions, open a new\nissue in the `universal_pathlib` repository. We are happy to help you!\n\n### Customization options for UPath subclasses\n\n#### Filesystem access methods\n\nOnce you thoroughly test your custom UPath implementation, it's likely that some\nmethods need to be overridden to provide correct behavior compared to `stdlib`'s\n`pathlib.Path` class. The most common issue is that for certain edge cases, your\nimplementation is not raising the same exceptions compared to the `pathlib.Path`\nclass. Or that the `UPath.path` property needs some prefix removed or added.\n\n```python\nclass MyProtoPath(UPath):\n\n @property\n def path(self) -> str:\n if p := self.path.startswith(\"/\"):\n p = p[1:]\n return p\n\n def mkdir(self, mode=0o777, parents=False, exist_ok=False):\n if some_edge_case:\n raise FileExistsError(str(self))\n super().mkdir(mode=mode, parents=parents, exist_ok=exist_ok)\n\n def is_file(self):\n return self.fs.isfile(self.path, myproto_option=123)\n```\n\n#### Storage option parsing\n\nIt's possible that you might want to extract additional storage options from the\nuser provided arguments to you constructor. You can provide a custom classmethod\nfor `_parse_storage_options`:\n\n```python\nimport os\n\nclass MyProtoPath(UPath):\n\n @classmethod\n def _parse_storage_options(\n cls, urlpath: str, protocol: str, storage_options: Mapping[str, Any]\n ) -> dict[str, Any]:\n if \"SOME_VAR\" in os.environ:\n storage_options[\"some_var\"] = os.environ[\"SOME_VAR\"]\n storage_options[\"my_proto_caching\"] = True\n storage_options[\"extra\"] = get_setting_from_path(urlpath)\n return storage_options\n```\n\n#### Fsspec filesystem instantiation\n\nTo have more control over fsspec filesystem instantiation you can write a custom\n`_fs_factory` classmethod:\n\n```python\nclass MyProtoPath(UPath):\n\n @classmethod\n def _fs_factory(\n cls, urlpath: str, protocol: str, storage_options: Mapping[str, Any]\n ) -> AbstractFileSystem:\n myfs = ... # custom code that creates a AbstractFileSystem instance\n return myfs\n```\n\n#### Init argument parsing\n\nIn special cases you need to take more control over how the init args are parsed\nfor your custom subclass. You can override `__init__` or the `UPath` classmethod\n`_transform_init_args`. The latter handles pickling of your custom subclass in a\nbetter way in case you modify storage options or the protocol.\n\n```python\nclass MyProtoPath(UPath):\n\n @classmethod\n def _transform_init_args(\n cls,\n args: tuple[str | os.PathLike, ...],\n protocol: str,\n storage_options: dict[str, Any],\n ) -> tuple[tuple[str | os.PathLike, ...], str, dict[str, Any]]:\n # check the cloud, http or webdav implementations for examples\n ...\n return args, protocol, storage_options\n```\n\n#### Stopping UPath's subclass dispatch mechanism\n\nThere are cases for which you want to disable the protocol dispatch mechanism of\nthe `UPath.__new__` constructor. For example if you want to extend the class API\nof your `UPath` implementation, and use it as the base class for other, directly\ninstantiated subclasses. Together with other customization options this can be a\nuseful feature. Please be aware that in this case all protocols are handled with\nthe default implementation in UPath. Please always feel free to open an issue in\nthe issue tracker to discuss your use case. We're happy to help with finding the\nmost maintainable solution.\n\n```python\nclass ExtraUPath(UPath):\n _protocol_dispatch = False # disable the registry return an ExtraUPath\n\n def some_extra_method(self) -> str:\n return \"hello world\"\n\nassert ExtraUPath(\"s3://bucket/file.txt\").some_extra_method() == \"hello world\"\n```\n\n## Migration Guide\n\nUPath's internal implementation is likely going to change with larger changes in\nCPython's stdlib `pathlib` landing in the next Python versions (`3.13`, `3.14`).\nTo reduce the problems for user code, when these changes are landing in `UPath`,\nthere have been some significant changes in `v0.2.0`. This migration guide tries\nto help migrating code that extensively relies on private implementation details\nof the `UPath` class of versions `v0.1.x` to the new and better supported public\ninterface of `v0.2.0`\n\n### migrating to `v0.2.0`\n\n### _FSSpecAccessor subclasses with custom filesystem access methods\n\nIf you implemented a custom accessor subclass, it is now recommended to override\nthe corresponding `UPath` methods in your subclass directly:\n\n```python\n# OLD: v0.1.x\nfrom upath.core import UPath, _FSSpecAccessor\n\nclass MyAccessor(_FSSpecAccessor):\n def exists(self, path, **kwargs):\n # custom code\n return path.fs.exists(self._format_path(path), **kwargs)\n\n def touch(self, path, **kwargs):\n # custom\n return path.fs.touch(self._format_path(path), **kwargs)\n\nclass MyPath(UPath):\n _default_accessor = MyAccessor\n\n\n# NEW: v0.2.0+\nfrom upath import UPath\n\nclass MyPath(UPath):\n def exists(self, *, follow_symlinks=True):\n kwargs = {} # custom code\n return self.fs.exists(self.path, **kwargs)\n\n def touch(self, mode=0o666, exist_ok=True):\n kwargs = {} # custom code\n self.fs.touch(self.path, **kwargs)\n```\n\n### _FSSpecAccessor subclasses with custom `__init__` method\n\nIf you implemented a custom `__init__` method for your accessor subclass usually\nthe intention is to customize how the fsspec filesystem instance is created. You\nhave two options to recreate this with the new implementation. Chose one or both\ndependent on the level of control you need.\n\n```python\n# OLD: v0.1.x\nimport fsspec\nfrom upath.core import UPath, _FSSpecAccessor\n\nclass MyAccessor(_FSSpecAccessor):\n def __init__(self, parsed_url: SplitResult | None, **kwargs: Any) -> None:\n # custom code\n protocol = ...\n storage_options = ...\n self._fs = fsspec.filesystem(protocol, storage_options)\n\nclass MyPath(UPath):\n _default_accessor = MyAccessor\n\n\n# NEW: v0.2.0+\nfrom upath import UPath\n\nclass MyPath(UPath):\n @classmethod\n def _parse_storage_options(\n cls, urlpath: str, protocol: str, storage_options: Mapping[str, Any]\n ) -> dict[str, Any]:\n # custom code to change storage_options\n storage_options = ...\n return storage_options\n\n @classmethod\n def _fs_factory(\n cls, urlpath: str, protocol: str, storage_options: Mapping[str, Any]\n ) -> AbstractFileSystem:\n # custom code to instantiate fsspec filesystem\n protocol = ...\n storage_options = ... # note changes to storage_options here won't\n # show up in MyPath().storage_options\n return fsspec.filesystem(protocol, **storage_options)\n```\n\n### Access to `._accessor`\n\nThe `_accessor` attribute and the `_FSSpecAccessor` class is deprecated. In case\nyou need direct access to the underlying filesystem, just access `UPath().fs`.\n\n```python\n# OLD: v0.1.x\nfrom upath.core import UPath\n\nclass MyPath(UPath):\n def mkdir(self, mode=0o777, parents=False, exist_ok=False):\n self._accessor.mkdir(...) # custom access to the underlying fs...\n\n\n# NEW: v0.2.0+\nfrom upath import UPath\n\nclass MyPath(UPath):\n def mkdir(self, mode=0o777, parents=False, exist_ok=False):\n self.fs.mkdir(...)\n```\n\n### Access to `._path`, `._kwargs`, `._drv`, `._root`, `._parts`\n\nIf you access one of the listed private attributes directly, move your code over\nto the following public versions:\n\n| _deprecated_ | `v0.2.0+` |\n|:------------------|:--------------------------|\n| `UPath()._path` | `UPath().path` |\n| `UPath()._kwargs` | `UPath().storage_options` |\n| `UPath()._drv` | `UPath().drive` |\n| `UPath()._root` | `UPath().root` |\n| `UPath()._parts` | `UPath().parts` |\n\n### Access to `._url`\n\nBe aware that the `._url` attribute will likely be deprecated once `UPath()` has\nsupport for uri fragments and uri query parameters through a public api. In case\nyou are interested in contributing this functionality, please open an issue!\n\n### Calling `_from_parts`, `_parse_args`, `_format_parsed_parts`\n\nIf your code is currently calling any of the three above listed classmethods, it\nrelies on functionality based on the implementation of `pathlib` in Python up to\n`3.11`. In `universal_pathlib` we vendor code that allows the `UPath()` class to\nbe based on the `3.12` implementation of `pathlib.Path` alone. Usually, usage of\nthose classmethods occurs when copying some code of the internal implementations\nof methods of the `UPath` `0.1.4` classes.\n\n- To reproduce custom `_format_parsed_parts` methods in `v0.2.0`, try overriding\n `UPath().path` and/or `UPath().with_segments()`.\n- Custom `_from_parts` and `_parse_args` classmethods can now be implemented via\n the `_transform_init_args` method or via more functionality in the new flavour\n class. Please open an issue for discussion in case you have this use case.\n\n### Custom `_URIFlavour` classes\n\nThe `_URIFlavour` class was removed from `universal_pathlib` and the new flavour\nclass for fsspec filesystem path operations now lives in `upath._flavour`. As of\nnow the internal FSSpecFlavour is experimental. In a future Python version, it's\nlikely that a flavour or flavour-like base class will become public, that allows\nus to base our internal implementation on. Until then, if you find yourself in a\nsituation where a custom path flavour would solve your problem, please feel free\nto open an issue for discussion. We're happy to find a maintainable solution.\n\n### Using `.parse_parts()`, `.casefold()`, `.join_parsed_parts()` of `._flavour`\n\nThese methods of the `._flavour` attribute of `pathlib.Path()` and `UPath()` are\nspecific to `pathlib` of Python versions up to `3.11`. `UPath()` is now based on\nthe `3.12` implementation of `pathlib.Path`. Please refer to the implementations\nof the `upath._flavour` submodule to see how you could avoid using them.\n\n\n## Known issues solvable by installing newer upstream dependencies\n\nSome issues in `UPath`'s behavior with specific fsspec filesystems are fixed via\ninstallation of a newer version of its upstream dependencies. Below you can find\na list of known issues and their solutions. We attempt to keep this list updated\nwhenever we encounter more:\n\n- **UPath().glob()**:\n `fsspec` fixed glob behavior when handling `**` patterns in `fsspec>=2023.9.0`\n- **GCSPath().mkdir()**:\n a few mkdir quirks are solved by installing `gcsfs>=2022.7.1`\n- **fsspec.filesystem(WebdavPath().protocol)**\n the webdav protocol was added to fsspec in version `fsspec>=2022.5.0`\n- **stat.S_ISDIR(HTTPPath().stat().st_mode)**\n requires `fsspec>=2024.2.0` to correctly return `True` for directories\n\n\n## Contributing\n\nContributions are very welcome.\nTo learn more, see the [Contributor Guide](CONTRIBUTING.rst).\n\n\n## License\n\nDistributed under the terms of the [MIT license](LICENSE),\n*universal_pathlib* is free and open source software.\n\n\n## Issues\n\nIf you encounter any problems, or if you create your own implementations and run\ninto limitations, please [file an issue][issues] with a detailed description. We\nare always happy to help with any problems you might encounter.\n\n[issues]: https://github.com/fsspec/universal_pathlib/issues\n",
"bugtrack_url": null,
"license": "MIT License",
"summary": "pathlib api extended to use fsspec backends",
"version": "0.2.5",
"project_urls": {
"Changelog": "https://github.com/fsspec/universal_pathlib/blob/main/CHANGELOG.md",
"Homepage": "https://github.com/fsspec/universal_pathlib"
},
"split_keywords": [
"filesystem-spec",
" pathlib"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "95d9289d308f889aac33639703a60906e3a0f3ec97419b7ca5bedaddc77648fd",
"md5": "d5a8f33577a27608b8ff17793d9e3237",
"sha256": "a634f700eca827b4ad03bfa0267e51161560dd1de83b051cf0fccf39b3e56b32"
},
"downloads": -1,
"filename": "universal_pathlib-0.2.5-py3-none-any.whl",
"has_sig": false,
"md5_digest": "d5a8f33577a27608b8ff17793d9e3237",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 49892,
"upload_time": "2024-09-08T20:19:47",
"upload_time_iso_8601": "2024-09-08T20:19:47.282803Z",
"url": "https://files.pythonhosted.org/packages/95/d9/289d308f889aac33639703a60906e3a0f3ec97419b7ca5bedaddc77648fd/universal_pathlib-0.2.5-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "24676c31ba464eafda05c677628dd7859ed4904597a78694d9cc81b593c6bad2",
"md5": "e7d826a627e732cbd398bc62ab33142e",
"sha256": "ea5d4fb8178c2ab469cf4fa46d0ceb16ccb378da46dbbc28a8b9c1eebdccc655"
},
"downloads": -1,
"filename": "universal_pathlib-0.2.5.tar.gz",
"has_sig": false,
"md5_digest": "e7d826a627e732cbd398bc62ab33142e",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 174755,
"upload_time": "2024-09-08T20:19:48",
"upload_time_iso_8601": "2024-09-08T20:19:48.713915Z",
"url": "https://files.pythonhosted.org/packages/24/67/6c31ba464eafda05c677628dd7859ed4904597a78694d9cc81b593c6bad2/universal_pathlib-0.2.5.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-09-08 20:19:48",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "fsspec",
"github_project": "universal_pathlib",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "universal-pathlib"
}
JSON
