prefect-managedfiletransfer

Name	prefect-managedfiletransfer JSON
Version	0.2.0 JSON
	download
home_page	None
Summary	Upload and download files easily between Local/SFTP/Cloud storage using rclone
upload_time	2025-08-15 16:50:50
maintainer	None
docs_url	None
author	None
requires_python	>=3.10
license	None
keywords	prefect file copy move transfer sftp rclone
VCS
bugtrack_url
requirements	No requirements were recorded.
Travis-CI	No Travis.
coveralls test coverage	No coveralls.

# Prefect Managed File Transfer

Turn a prefect.io server into a managed file transfer solution. UI and Programatic creation of cron style jobs (aka Flows!) to upload and download files easily between servers. Support local, SFTP remotes plus any Cloud storage supported by rclone - so thats aws, azure, google, sharepoint, and [many more](https://rclone.org/overview/) out of the box.

Using prefect for managed file transfer means retries, logging, multi node and [high availability](https://docs.prefect.io/v3/advanced/self-hosted) come as standard - turning prefect into a reliable enterprise ready file transfer solution.

This package is not the fastest solution to move files around, but it prioritises reliability and ease of use, making it an excellent choice for replacing both quick cron job copy scripts and enterprise managed file transfer applicances.

Key features

- Copy and move files between almost any storage system easily.
- Reliable file moving with checksumming, file size checking etc.
- Smart and safe moving - settings to allow/block overwriting and to only copy files if they are new or changed.
- Unzip/Untar compressed folders after downloading them.
- Repath files as you move them.
- Complex filtering and ordering of files - by path, age, size etc. Pattern matching with regular expressions.
- Leverage Prefect.IO built in scheduling and orchestration capabilities:
- Transfer files on complex cron schedules
- notifications on success/failure - slack, email, etc
- Highly available server architecture - database server + multi-node workers and front ends.

Example use cases:

- Once per day SSH into my database server and copy the latest *.bkup file to a central storage location.
- Monitor a local network share directory for new files and automatically upload them to a cloud storage bucket.
- Schedule a weekly job to synchronize files between two remote servers.
- Move log files from a SSH available web server older than 30 days to a cold storage location.
- Copy file yyyy-MM-dd.zip from a remote server, where yyyy-MM-dd matches todays date, to a local directory and then unzip it.
- Download any file in an S3 bucket larger than 1GB and store it in a local directory.

### Installation

Install `prefect-managedfiletransfer` with `pip`. (Requires an installation of Python 3.10+.)

```bash
pip install prefect-managedfiletransfer
# or
uv add prefect-managedfiletransfer
```

We recommend using a Python virtual environment manager such as uv, pipenv, conda or virtualenv.

In one (venv) terminal start a prefect server with logs enabled

```bash
export PREFECT_LOGGING_LEVEL="INFO"
export PREFECT_LOGGING_EXTRA_LOGGERS="prefect_managedfiletransfer"
prefect server start
# OR uv run prefect server start
```

There are many ways to manage infrastructure and code with prefect - here we demonstate starting a local worker:

```bash
export PREFECT_API_URL=http://127.0.0.1:4200/api
# or perhaps export PREFECT_API_URL=http://host.docker.internal:4200/api
export PREFECT_LOGGING_EXTRA_LOGGERS="prefect_managedfiletransfer"
export PREFECT_LOGGING_LEVEL="INFO"
# [Optional] add all logs: export PREFECT_LOGGING_ROOT_LEVEL="INFO"

prefect worker start --pool 'default-pool' --type process

# OR add a worker with config to spawn containers that can talk to the server API:
PREFECT_API_URL=http://host.docker.internal:4200/api uv run prefect worker start --pool 'default-pool' --type=docker

```

Install the blocks using the prefect CLI

```bash
prefect block register -m prefect_managedfiletransfer
```

And then deploy the flows.

```bash
# deploy the flows to run locally
python -m prefect_managedfiletransfer.deploy --local

# OR deploy to run with a docker image - see deploy.py
python -m prefect_managedfiletransfer.deploy --docker

# or a version of the above using uv run:
uv run python -m prefect_managedfiletransfer.deploy --local
uv run python -m prefect_managedfiletransfer.deploy --docker
```

Visit the server UI http://localhost:4200.
1. Create 2 blocks, one source and one destination
2. On the deployments page start a `transfer_files_flow`. Configure your flow run to copy/move files between the 2 blocks.

### Installation via docker - TODO

```bash
# run prefect server in a container port-forwarded to your local machine’s 4200 port:
docker run -d -p 4200:4200 prefecthq/prefect:3-latest -- prefect server start --host 0.0.0.0
```

### Components

**Flows**

- transfer_files_flow - a fully featured flow for transferring files between different storage locations.
- upload_file_flow - a flow for uploading a file to a remote server. Supports pattern matching by date

**Blocks**

- ServerWithBasicAuthBlock - A block for connecting to a server using basic authentication.
- ServerWithPublicKeyAuthBlock - A block for connecting to a server using public key authentication.
- RCloneConfigFileBlock - A block for managing RClone configuration files.

**Tasks**

- list_remote_files_task - A task for listing files in a remote directory.
- download_file_task - A task for downloading a single file from a remote server.
- upload_file_task - A task for uploading a single file to a remote server.
- [TODO] delete_file_task

![Screenshot of transfer files flow](docs/img/transfer_files_screengrab.png)

### Feedback

If you encounter any bugs while using `prefect-managedfiletransfer`, feel free to open an issue in the [prefect-managedfiletransfer](https://github.com/ImperialCollegeLondon/prefect-managedfiletransfer) repository.

Feel free to star or watch [`prefect-managedfiletransfer`](https://github.com/ImperialCollegeLondon/prefect-managedfiletransfer) for updates too!

### Contributing

If you'd like to help contribute to fix an issue or add a feature to `prefect-managedfiletransfer`, please [propose changes through a pull request from a fork of the repository](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/creating-a-pull-request-from-a-fork).

Here are the steps:

1. [Fork the repository](https://docs.github.com/en/get-started/quickstart/fork-a-repo#forking-a-repository)
2. [Clone the forked repository](https://docs.github.com/en/get-started/quickstart/fork-a-repo#cloning-your-forked-repository)
3. Install the repository and its dependencies:
```
# install uv first, then
uv sync
```

You can also access all the prefect CLI tooling inside a uv managed venv
```
uv venv
source .venv/bin/activate
prefect server start
```

4. Make desired changes
5. Add tests
6. Insert an entry to [CHANGELOG.md](https://github.com/ImperialCollegeLondon/prefect-managedfiletransfer/blob/main/CHANGELOG.md)
7. Install `pre-commit` to perform quality checks prior to commit:
```
pre-commit install
```
8. use the build script to run all the checks and tests:

```
./build.sh
```
8. `git commit`, `git push`, and create a pull request

Raw data

            {
    "_id": null,
    "home_page": null,
    "name": "prefect-managedfiletransfer",
    "maintainer": null,
    "docs_url": null,
    "requires_python": ">=3.10",
    "maintainer_email": null,
    "keywords": "prefect, file, copy, move, transfer, SFTP, rclone",
    "author": null,
    "author_email": "Alastair Crabtree <a.crabtree@imperial.ac.uk>",
    "download_url": "https://files.pythonhosted.org/packages/30/ca/0a2b50105ce32e244b46a5add2f5bea87b3a6864f2795fea6d9c0d4a80bb/prefect_managedfiletransfer-0.2.0.tar.gz",
    "platform": null,
    "description": "# Prefect Managed File Transfer\n\n<p align=\"center\">\n    <!--- Insert a cover image here -->\n    <!--- <br> -->\n    <a href=\"https://pypi.python.org/pypi/prefect-managedfiletransfer/\" alt=\"PyPI version\">\n        <img alt=\"PyPI\" src=\"https://img.shields.io/pypi/v/prefect-managedfiletransfer?color=0052FF&labelColor=090422\"></a>\n    <a href=\"https://github.com/ImperialCollegeLondon/prefect-managedfiletransfer/\" alt=\"Stars\">\n        <img src=\"https://img.shields.io/github/stars/ImperialCollegeLondon/prefect-managedfiletransfer?color=0052FF&labelColor=090422\" /></a>\n    <a href=\"https://pypistats.org/packages/prefect-managedfiletransfer/\" alt=\"Downloads\">\n        <img src=\"https://img.shields.io/pypi/dm/prefect-managedfiletransfer?color=0052FF&labelColor=090422\" /></a>\n    <a href=\"https://github.com/ImperialCollegeLondon/prefect-managedfiletransfer/pulse\" alt=\"Activity\">\n        <img src=\"https://img.shields.io/github/commit-activity/m/ImperialCollegeLondon/prefect-managedfiletransfer?color=0052FF&labelColor=090422\" /></a>\n    <br>\n</p>\n\nTurn a prefect.io server into a managed file transfer solution. UI and Programatic creation of cron style jobs (aka Flows!) to upload and download files easily between servers. Support local, SFTP remotes plus any Cloud storage supported by rclone -  so thats aws, azure, google, sharepoint, and [many more](https://rclone.org/overview/) out of the box.\n\nUsing prefect for managed file transfer means retries, logging, multi node and [high availability](https://docs.prefect.io/v3/advanced/self-hosted) come as standard - turning prefect into a reliable enterprise ready file transfer solution. \n\nThis package is not the fastest solution to move files around, but it prioritises reliability and ease of use, making it an excellent choice for replacing both quick cron job copy scripts and enterprise managed file transfer applicances.\n\nKey features\n\n- Copy and move files between almost any storage system easily.\n- Reliable file moving with checksumming, file size checking etc.\n- Smart and safe moving - settings to allow/block overwriting and to only copy files if they are new or changed.\n- Unzip/Untar compressed folders after downloading them.\n- Repath files as you move them.\n- Complex filtering and ordering of files - by path, age, size etc. Pattern matching with regular expressions.\n- Leverage Prefect.IO built in scheduling and orchestration capabilities:\n    - Transfer files on complex cron schedules\n    - notifications on success/failure - slack, email, etc\n    - Highly available server architecture - database server + multi-node workers and front ends.\n\nExample use cases:\n\n- Once per day SSH into my database server and copy the latest *.bkup file to a central storage location.\n- Monitor a local network share directory for new files and automatically upload them to a cloud storage bucket.\n- Schedule a weekly job to synchronize files between two remote servers.\n- Move log files from a SSH available web server older than 30 days to a cold storage location.\n- Copy file yyyy-MM-dd.zip from a remote server, where yyyy-MM-dd matches todays date, to a local directory and then unzip it.\n- Download any file in an S3 bucket larger than 1GB and store it in a local directory.\n\n### Installation\n\nInstall `prefect-managedfiletransfer` with `pip`. (Requires an installation of Python 3.10+.)\n\n```bash\npip install prefect-managedfiletransfer\n# or \nuv add prefect-managedfiletransfer\n```\n\nWe recommend using a Python virtual environment manager such as uv, pipenv, conda or virtualenv.\n\nIn one (venv) terminal start a prefect server with logs enabled\n\n```bash\nexport PREFECT_LOGGING_LEVEL=\"INFO\"\nexport PREFECT_LOGGING_EXTRA_LOGGERS=\"prefect_managedfiletransfer\"\nprefect server start\n# OR uv run prefect server start\n```\n\nThere are many ways to manage infrastructure and code with prefect - here we demonstate starting a local worker:\n\n```bash\nexport PREFECT_API_URL=http://127.0.0.1:4200/api\n# or perhaps export PREFECT_API_URL=http://host.docker.internal:4200/api\nexport PREFECT_LOGGING_EXTRA_LOGGERS=\"prefect_managedfiletransfer\"\nexport PREFECT_LOGGING_LEVEL=\"INFO\"\n# [Optional] add all logs: export PREFECT_LOGGING_ROOT_LEVEL=\"INFO\"\n\n\nprefect worker start --pool 'default-pool' --type process\n\n# OR add a worker with config to spawn containers that can talk to the server API:\nPREFECT_API_URL=http://host.docker.internal:4200/api uv run prefect worker start --pool 'default-pool' --type=docker  \n\n```\n    \nInstall the blocks using the prefect CLI\n\n```bash\nprefect block register -m prefect_managedfiletransfer\n```\n\nAnd then deploy the flows. \n\n```bash\n# deploy the flows to run locally\npython -m prefect_managedfiletransfer.deploy --local\n\n# OR deploy to run with a docker image - see deploy.py\npython -m prefect_managedfiletransfer.deploy --docker\n\n# or a version of the above using uv run:\nuv run python -m prefect_managedfiletransfer.deploy --local\nuv run python -m prefect_managedfiletransfer.deploy --docker\n```\n\nVisit the server UI http://localhost:4200.\n1. Create 2 blocks, one source and one destination\n2. On the deployments page start a `transfer_files_flow`. Configure your flow run to copy/move files between the 2 blocks.\n\n### Installation via docker - TODO\n\n```bash\n# run prefect server in a container port-forwarded to your local machine\u2019s 4200 port:\ndocker run -d -p 4200:4200 prefecthq/prefect:3-latest -- prefect server start --host 0.0.0.0\n```\n\n### Components\n\n**Flows**\n\n- transfer_files_flow - a fully featured flow for transferring files between different storage locations.\n- upload_file_flow - a flow for uploading a file to a remote server. Supports pattern matching by date\n\n**Blocks**\n\n- ServerWithBasicAuthBlock - A block for connecting to a server using basic authentication.\n- ServerWithPublicKeyAuthBlock - A block for connecting to a server using public key authentication.\n- RCloneConfigFileBlock - A block for managing RClone configuration files.\n\n**Tasks**\n\n- list_remote_files_task - A task for listing files in a remote directory.\n- download_file_task - A task for downloading a single file from a remote server.\n- upload_file_task - A task for uploading a single file to a remote server.\n- [TODO] delete_file_task\n\n![Screenshot of transfer files flow](docs/img/transfer_files_screengrab.png)\n\n### Feedback\n\nIf you encounter any bugs while using `prefect-managedfiletransfer`, feel free to open an issue in the [prefect-managedfiletransfer](https://github.com/ImperialCollegeLondon/prefect-managedfiletransfer) repository.\n\n\nFeel free to star or watch [`prefect-managedfiletransfer`](https://github.com/ImperialCollegeLondon/prefect-managedfiletransfer) for updates too!\n\n### Contributing\n\nIf you'd like to help contribute to fix an issue or add a feature to `prefect-managedfiletransfer`, please [propose changes through a pull request from a fork of the repository](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/creating-a-pull-request-from-a-fork).\n\nHere are the steps:\n\n1. [Fork the repository](https://docs.github.com/en/get-started/quickstart/fork-a-repo#forking-a-repository)\n2. [Clone the forked repository](https://docs.github.com/en/get-started/quickstart/fork-a-repo#cloning-your-forked-repository)\n3. Install the repository and its dependencies:\n```\n# install uv first, then\nuv sync\n```\n\nYou can also access all the prefect CLI tooling inside a uv managed venv\n```\nuv venv\nsource .venv/bin/activate\nprefect server start\n```\n\n4. Make desired changes\n5. Add tests\n6. Insert an entry to [CHANGELOG.md](https://github.com/ImperialCollegeLondon/prefect-managedfiletransfer/blob/main/CHANGELOG.md)\n7. Install `pre-commit` to perform quality checks prior to commit:\n```\npre-commit install\n```\n8. use the build script to run all the checks and tests:\n\n```\n./build.sh\n```\n8. `git commit`, `git push`, and create a pull request\n",
    "bugtrack_url": null,
    "license": null,
    "summary": "Upload and download files easily between Local/SFTP/Cloud storage using rclone",
    "version": "0.2.0",
    "project_urls": {
        "Homepage": "https://github.com/ImperialCollegeLondon/prefect-managedfiletransfer"
    },
    "split_keywords": [
        "prefect",
        " file",
        " copy",
        " move",
        " transfer",
        " sftp",
        " rclone"
    ],
    "urls": [
        {
            "comment_text": null,
            "digests": {
                "blake2b_256": "1d3d43c85bab7846cfeade41dc8de5587cb9f810bf4e0c2fadfc95fe7b3a174d",
                "md5": "d231b761e82a89a94e225bae6fa08777",
                "sha256": "f8f542f4c8c49032464b8cd758aca6d85de74085689a6665e75678497acb0a24"
            },
            "downloads": -1,
            "filename": "prefect_managedfiletransfer-0.2.0-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "d231b761e82a89a94e225bae6fa08777",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": ">=3.10",
            "size": 71601716,
            "upload_time": "2025-08-15T16:50:45",
            "upload_time_iso_8601": "2025-08-15T16:50:45.245914Z",
            "url": "https://files.pythonhosted.org/packages/1d/3d/43c85bab7846cfeade41dc8de5587cb9f810bf4e0c2fadfc95fe7b3a174d/prefect_managedfiletransfer-0.2.0-py3-none-any.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": null,
            "digests": {
                "blake2b_256": "30ca0a2b50105ce32e244b46a5add2f5bea87b3a6864f2795fea6d9c0d4a80bb",
                "md5": "8c33f7156c2a113306cd2a0696d02ed9",
                "sha256": "d7765ffa6912c73610dbe91f91df65a75f301437b102900864238c99ef06db15"
            },
            "downloads": -1,
            "filename": "prefect_managedfiletransfer-0.2.0.tar.gz",
            "has_sig": false,
            "md5_digest": "8c33f7156c2a113306cd2a0696d02ed9",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": ">=3.10",
            "size": 71057395,
            "upload_time": "2025-08-15T16:50:50",
            "upload_time_iso_8601": "2025-08-15T16:50:50.082156Z",
            "url": "https://files.pythonhosted.org/packages/30/ca/0a2b50105ce32e244b46a5add2f5bea87b3a6864f2795fea6d9c0d4a80bb/prefect_managedfiletransfer-0.2.0.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2025-08-15 16:50:50",
    "github": true,
    "gitlab": false,
    "bitbucket": false,
    "codeberg": false,
    "github_user": "ImperialCollegeLondon",
    "github_project": "prefect-managedfiletransfer",
    "travis_ci": false,
    "coveralls": false,
    "github_actions": true,
    "lcname": "prefect-managedfiletransfer"
}

None