<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
# Airavata Managed File Transfers (MFT)
Apache Airavata MFT is a high-performance, multi-protocol data transfer engine to orchestrate data movement and operations across most cloud and On-premises storages. MFT aims to abstract the complexity of heterogenous storages by providing a unified and simple interface for users to seamlessly access and move data across any storage endpoint. To accomplish this goal, MFT provides simple but highly-performing tools to access most cloud and on-premise storages as seamlessly as they access local files in their workstations.
Apache Airavata MFT bundles easily deployable agents that auto determine optimum network path with additional multi-channel, parallel data paths to optimize the transfer performance to gain the maximum throughput between storage endpoints. MFT utilizes parallel Agents to transfer data between endpoints to gain the advantage of multiple network links.
# Try Airavata MFT
MFT requires Java 11+ and python3.10+ to install Airavata MFT in your environment. MFT currently supports Linux and MacOS operating systems. Contributions to support Windows are welcome!!.
### Download and Install
Following commands will download Airavata MFT into your machine and start the MFT service.
```
pip3 install airavata-mft-cli
mft init
```
> If the installer failed for M1 and M2 Macs complaining about grpcio installation. Follow the solution mentioned in [here](https://github.com/apache/airavata-mft/issues/71). You might have to uninstall already installed grpcio and grpcio-tools distributions first.
> For other common installation issues, please refer to the [troubleshooting section](https://github.com/apache/airavata-mft#common-issues).
To stop MFT after using
```
mft stop
```
### Registering Storages
First you need to register your storage endpoints into MFT in order to access them. Registering storage is an interactive process and you can easily register those without prior knowledge
```
mft storage add
```
This will ask the type of storage you need and credentials to access those storages. To list already added storages, you can run
```
mft storage list
```
### Accessing Data in Storages
In Airavata MFT, we provide a unified interface to access the data in any storage. Users can access data in storages just as they access data in their computers. MFT converts user queries into storage specific data representations (POSIX, Block, Objects, ..) internally
```
mft ls <storage name>
mft ls <storage name>/<resource path>
```
### Moving Data between Storages
Copying data between storages are simple as copying data between directories of local machine for users. MFT takes care of network path optimizations, parallel data path selections and selections or creations of suitable transfer agents.
```
mft cp <source storage name>/<path> <destination storage name>/<path>
```
MFT is capable of auto detecting directory copying and file copying based on the path given.
### Troubleshooting and Issue Reporting
This is our very first attempt release Airavata MFT for community usage and there might be lots of corner cases that we have not noticed. All the logs of MFT service are available in ```~/.mft/Standalone-Service-0.01/logs/airavata.log```. If you see any error while using MFT, please report that in our Github issue page and we will respond as soon as possible. We really appreciate your contribution as it will greatly help to improve the stability of the product.
#### Common issues
- Following error can be noticed if you have a python version which is less than 3.10
```
ERROR: Could not find a version that satisfies the requirement airavata-mft-cli (from versions: none)
ERROR: No matching distribution found for airavata-mft-cli
```
If the Error still occurs after installing the right python version, try creating a virtual environemnt
```
python3.10 -m venv venv
source venv/bin/activate
pip install airavata-mft-cli
```
Raw data
{
"_id": null,
"home_page": "https://github.com/apache/airavata-mft",
"name": "airavata-mft-cli",
"maintainer": "Apache Airavata Developers",
"docs_url": null,
"requires_python": "<4.0,>=3.10",
"maintainer_email": "dev@apache.airavata.org",
"keywords": "MFT, Data, High Thorught Data Transfer, Multi Protocol, S3, GCS, Azure, SCP, Google Drive, Swift, Box, DropBox",
"author": "Dimuthu Wannipurage",
"author_email": "dwannipu@iu.edu",
"download_url": "https://files.pythonhosted.org/packages/4a/ed/6a629e76bdcb33a50c5061c44d8950247bd61b3a6f32dde6742edf454491/airavata_mft_cli-0.1.24.tar.gz",
"platform": null,
"description": "<!--\nLicensed to the Apache Software Foundation (ASF) under one\nor more contributor license agreements. See the NOTICE file\ndistributed with this work for additional information\nregarding copyright ownership. The ASF licenses this file\nto you under the Apache License, Version 2.0 (the\n\"License\"); you may not use this file except in compliance\nwith the License. You may obtain a copy of the License at\n\n http://www.apache.org/licenses/LICENSE-2.0\n\nUnless required by applicable law or agreed to in writing,\nsoftware distributed under the License is distributed on an\n\"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\nKIND, either express or implied. See the License for the\nspecific language governing permissions and limitations\nunder the License.\n-->\n\n# Airavata Managed File Transfers (MFT)\n\nApache Airavata MFT is a high-performance, multi-protocol data transfer engine to orchestrate data movement and operations across most cloud and On-premises storages. MFT aims to abstract the complexity of heterogenous storages by providing a unified and simple interface for users to seamlessly access and move data across any storage endpoint. To accomplish this goal, MFT provides simple but highly-performing tools to access most cloud and on-premise storages as seamlessly as they access local files in their workstations.\n\nApache Airavata MFT bundles easily deployable agents that auto determine optimum network path with additional multi-channel, parallel data paths to optimize the transfer performance to gain the maximum throughput between storage endpoints. MFT utilizes parallel Agents to transfer data between endpoints to gain the advantage of multiple network links.\n\n# Try Airavata MFT\nMFT requires Java 11+ and python3.10+ to install Airavata MFT in your environment. MFT currently supports Linux and MacOS operating systems. Contributions to support Windows are welcome!!.\n\n### Download and Install\n\nFollowing commands will download Airavata MFT into your machine and start the MFT service.\n```\npip3 install airavata-mft-cli\nmft init\n```\n\n> If the installer failed for M1 and M2 Macs complaining about grpcio installation. Follow the solution mentioned in [here](https://github.com/apache/airavata-mft/issues/71). You might have to uninstall already installed grpcio and grpcio-tools distributions first.\n> For other common installation issues, please refer to the [troubleshooting section](https://github.com/apache/airavata-mft#common-issues).\n\nTo stop MFT after using\n\n```\nmft stop\n```\n\n\n### Registering Storages\n\nFirst you need to register your storage endpoints into MFT in order to access them. Registering storage is an interactive process and you can easily register those without prior knowledge\n\n```\nmft storage add\n```\n\nThis will ask the type of storage you need and credentials to access those storages. To list already added storages, you can run\n\n```\nmft storage list\n```\n### Accessing Data in Storages\n\nIn Airavata MFT, we provide a unified interface to access the data in any storage. Users can access data in storages just as they access data in their computers. MFT converts user queries into storage specific data representations (POSIX, Block, Objects, ..) internally\n\n```\nmft ls <storage name>\nmft ls <storage name>/<resource path>\n```\n\n### Moving Data between Storages\n\nCopying data between storages are simple as copying data between directories of local machine for users. MFT takes care of network path optimizations, parallel data path selections and selections or creations of suitable transfer agents.\n\n ```\n mft cp <source storage name>/<path> <destination storage name>/<path> \n ```\nMFT is capable of auto detecting directory copying and file copying based on the path given.\n\n### Troubleshooting and Issue Reporting\n\nThis is our very first attempt release Airavata MFT for community usage and there might be lots of corner cases that we have not noticed. All the logs of MFT service are available in ```~/.mft/Standalone-Service-0.01/logs/airavata.log```. If you see any error while using MFT, please report that in our Github issue page and we will respond as soon as possible. We really appreciate your contribution as it will greatly help to improve the stability of the product.\n\n#### Common issues\n\n- Following error can be noticed if you have a python version which is less than 3.10\n```\n ERROR: Could not find a version that satisfies the requirement airavata-mft-cli (from versions: none)\n ERROR: No matching distribution found for airavata-mft-cli\n```\nIf the Error still occurs after installing the right python version, try creating a virtual environemnt\n```\npython3.10 -m venv venv\nsource venv/bin/activate\npip install airavata-mft-cli\n```\n \n",
"bugtrack_url": null,
"license": "Apache-2.0",
"summary": "Command Line Client for Apache Airavata MFT data transfer software",
"version": "0.1.24",
"project_urls": {
"Homepage": "https://github.com/apache/airavata-mft"
},
"split_keywords": [
"mft",
" data",
" high thorught data transfer",
" multi protocol",
" s3",
" gcs",
" azure",
" scp",
" google drive",
" swift",
" box",
" dropbox"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "cc9dac469e57f9efe206c95e6d7aae5d0e17f00123ecf02fe60da8c876e5727f",
"md5": "049ce4f4a995565e98161d9dffbc94a3",
"sha256": "0700c639da1921b6cd1e3cb00470957d8aaba802d8e51a20668d80799ce7f9b7"
},
"downloads": -1,
"filename": "airavata_mft_cli-0.1.24-py3-none-any.whl",
"has_sig": false,
"md5_digest": "049ce4f4a995565e98161d9dffbc94a3",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4.0,>=3.10",
"size": 26978,
"upload_time": "2024-04-05T15:16:59",
"upload_time_iso_8601": "2024-04-05T15:16:59.807487Z",
"url": "https://files.pythonhosted.org/packages/cc/9d/ac469e57f9efe206c95e6d7aae5d0e17f00123ecf02fe60da8c876e5727f/airavata_mft_cli-0.1.24-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "4aed6a629e76bdcb33a50c5061c44d8950247bd61b3a6f32dde6742edf454491",
"md5": "adfcf1bb5d0cda1266a4b7ac4b6e2304",
"sha256": "f6d979f1928ade8ac83ce010a36c135b154fefca03c193f03e9d848a7a3265e6"
},
"downloads": -1,
"filename": "airavata_mft_cli-0.1.24.tar.gz",
"has_sig": false,
"md5_digest": "adfcf1bb5d0cda1266a4b7ac4b6e2304",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4.0,>=3.10",
"size": 16419,
"upload_time": "2024-04-05T15:17:01",
"upload_time_iso_8601": "2024-04-05T15:17:01.629785Z",
"url": "https://files.pythonhosted.org/packages/4a/ed/6a629e76bdcb33a50c5061c44d8950247bd61b3a6f32dde6742edf454491/airavata_mft_cli-0.1.24.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-04-05 15:17:01",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "apache",
"github_project": "airavata-mft",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "airavata-mft-cli"
}
JSON
