| Name | AptosConnector JSON |
| Version |
0.0.2
JSON |
| download |
| home_page | |
| Summary | A dataset validation and upload utility for Aptos |
| upload_time | 2024-03-05 18:26:46 |
| maintainer | |
| docs_url | None |
| author | |
| requires_python | >=3.8 |
| license | Copyright (c) 2023 Eta Compute Inc. Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. |
| keywords |
aptos
dataset
connector
upload
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# AptosConnector
## Introduction
This package is provides commandline tools that validate and upload User datasets to the Aptos Edge ML patform.
## Installation
AptosConnector is currently supported on:
* Linux
* Windows 10/11
* macOS
### Prerequisites
* `AWS CLI v2`. For more information consult the installation guide: https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html
* `Python >=3.7` installation (you can use `conda` or virtual environments `venv`)
* `git` (if you intend to clone AptosConnector directly from github)
### Installing by cloning from GitHub
Start your terminal or command line.
```bash
# If using python venv or conda activate your environment e.g.:
conda activate aptos_env
```
Clone and install python package:
```bash
cd ~/ # Will install in user folder
git clone git@github.com:Eta-Compute/AptosConnector.git
cd AptosConnector
pip install -e .
```
### Installing via `pip`
~~~
Note: This modality is not supported yet
~~~
## User Guide
Currently the following functions are supported:
* AptosConnector setup (one-time only)
* Dataset validation
* Dataset upload
### Dataset Prepartion
In order to prepare your dataset, please follow our guide: [Dataset preparation guide](docs/dataset_preparation.md)
### AptosConnector Setup
This one-time only procedure will configure AptosConnector's access to Aptos platform. You will need to provide your `Aptos Group ID`, `Aptos AWS Access Key ID` and `Aptos AWS Secret Access Key`:
```
(aptos_env) user@ubuntu: aptos_setup
Welcome to AptosConnector one-time setup wizzard.
We'll get you started in just a few simple steps!
--------------------------------------------------
AWS CLI installation verified.
--------------------------------------------------
Aptos Group ID: XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXX
Aptos AWS Access Key ID: XXXXXXXXXXXXXXXXXXXX
Aptos AWS Secret Access Key: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
--------------------------------------------------
Configuration successful!
Now you can use:
`aptos_validate` to check your dataset for errors and verify Aptos interoperability
`aptos_upload` to upload dataset to Aptos platform
```
Once configuration is done, you can proceed with dataset validation and upload
### Dataset Validation
Validates user dataset to endure conformity with Aptos dataset format. Any issues will be displayed to the user so that they can be correted prior to upload. If validation is passed succesfully your dataset will be signed and ready to upload. Note that you cannot upload the dataset without validating it first.
```
aptos_validate
usage: aptos_validate [-h] -d DATASET_PATH [--verbose]
```
Example usage:
```
(aptos_env) user@ubuntu:~$ aptos_validate -d ~/datasets/tf_test
AptosConnector (v0.0.1) - dataset validation utility
-------------------------------------------- Messages: ---------------------------------------------
WARNING: The dataset is missing the "thumbnail.jpg" file. Pick one image from the dataset, name it as "thumbnail.jpg" and place it inside the dataset root directory.
WARNING: There are 3 duplicate images (with same content, but different name) in your dataset. Check the "dataset validator log" file in the Dataset Analysis job to see details about those images.
WARNING: "dataset_infos.json" doesn't contain a valid entry for "num_examples" for split "train", 1222 images were found in the dataset root directory.
--------------------------------------------- Summary: ---------------------------------------------
No critical errors found
Creating dataset signature ...
Validation passed and signed: f3322dab7dccb61a66e611ee0dbaff1207b552b647bbb3d4066e6f953fc96ad1
```
### Dataset Upload
This utility will upload your dataset to Aptos.
```
(aptos_env) user@ubuntu:~$ aptos_upload -d ~/datasets/tf_test
AptosConnector (v0.0.1) - dataset validation utility
----------------------------------------------------------------------------------------------------
Veryfying dataset signature...
Done!
S3 access verified
Found 3693 files in the dataset: 505.73 MB
Uploading file: tfds/tf_test-validation.tfrecord-00000-of-00001
100%|███████████████████████████████████████████████████████████| 3693/3693 [02:25<00:00, 25.46it/s]
----------------------------------------------------------------------------------------------------
Upload complete. You can view your dataset at: https://aptos.training/datasets/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXX/tf_test
```
## Running tests
```
# install tox test runner:
python -m pip install --user tox
python -m tox --help
# install developer dependencies of the package
cd [...]/AptosConnector
pip install -e .[dev]
# run tests
tox
```
## Developed by:
<img src="https://etacompute.com/wp-content/uploads/2021/09/eta-logo.svg" width="200">
Raw data
{
"_id": null,
"home_page": "",
"name": "AptosConnector",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": "",
"keywords": "aptos,dataset,connector,upload",
"author": "",
"author_email": "Jeremi Wojcicki <jeremi@etacompute.com>, Vedin Klovo <vedin.klovo@etacompute.com>",
"download_url": "https://files.pythonhosted.org/packages/d8/3b/4ad8fce068a28b26de399eb7dca5e5a0bd67e7d73bb82f32d96156253e77/AptosConnector-0.0.2.tar.gz",
"platform": null,
"description": "# AptosConnector\n\n## Introduction\n\nThis package is provides commandline tools that validate and upload User datasets to the Aptos Edge ML patform.\n\n## Installation\n\nAptosConnector is currently supported on:\n* Linux\n* Windows 10/11\n* macOS\n\n### Prerequisites\n* `AWS CLI v2`. For more information consult the installation guide: https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html\n* `Python >=3.7` installation (you can use `conda` or virtual environments `venv`)\n* `git` (if you intend to clone AptosConnector directly from github)\n\n### Installing by cloning from GitHub\n\nStart your terminal or command line.\n```bash\n# If using python venv or conda activate your environment e.g.:\nconda activate aptos_env\n```\n\nClone and install python package:\n```bash\ncd ~/ # Will install in user folder\ngit clone git@github.com:Eta-Compute/AptosConnector.git\ncd AptosConnector\npip install -e .\n```\n\n### Installing via `pip`\n\n~~~\nNote: This modality is not supported yet\n~~~\n\n## User Guide\n\nCurrently the following functions are supported:\n* AptosConnector setup (one-time only)\n* Dataset validation\n* Dataset upload\n\n### Dataset Prepartion\n\nIn order to prepare your dataset, please follow our guide: [Dataset preparation guide](docs/dataset_preparation.md)\n\n### AptosConnector Setup\n\nThis one-time only procedure will configure AptosConnector's access to Aptos platform. You will need to provide your `Aptos Group ID`, `Aptos AWS Access Key ID` and `Aptos AWS Secret Access Key`:\n\n```\n(aptos_env) user@ubuntu: aptos_setup\nWelcome to AptosConnector one-time setup wizzard.\nWe'll get you started in just a few simple steps!\n--------------------------------------------------\nAWS CLI installation verified.\n--------------------------------------------------\nAptos Group ID: XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXX\nAptos AWS Access Key ID: XXXXXXXXXXXXXXXXXXXX\nAptos AWS Secret Access Key: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\n--------------------------------------------------\nConfiguration successful!\n\nNow you can use:\n `aptos_validate` to check your dataset for errors and verify Aptos interoperability\n `aptos_upload` to upload dataset to Aptos platform\n\n```\nOnce configuration is done, you can proceed with dataset validation and upload\n\n### Dataset Validation\nValidates user dataset to endure conformity with Aptos dataset format. Any issues will be displayed to the user so that they can be correted prior to upload. If validation is passed succesfully your dataset will be signed and ready to upload. Note that you cannot upload the dataset without validating it first.\n```\naptos_validate\nusage: aptos_validate [-h] -d DATASET_PATH [--verbose]\n```\nExample usage:\n\n```\n(aptos_env) user@ubuntu:~$ aptos_validate -d ~/datasets/tf_test\n AptosConnector (v0.0.1) - dataset validation utility\n\n-------------------------------------------- Messages: ---------------------------------------------\nWARNING: The dataset is missing the \"thumbnail.jpg\" file. Pick one image from the dataset, name it as \"thumbnail.jpg\" and place it inside the dataset root directory.\nWARNING: There are 3 duplicate images (with same content, but different name) in your dataset. Check the \"dataset validator log\" file in the Dataset Analysis job to see details about those images.\nWARNING: \"dataset_infos.json\" doesn't contain a valid entry for \"num_examples\" for split \"train\", 1222 images were found in the dataset root directory.\n\n--------------------------------------------- Summary: ---------------------------------------------\nNo critical errors found\nCreating dataset signature ...\nValidation passed and signed: f3322dab7dccb61a66e611ee0dbaff1207b552b647bbb3d4066e6f953fc96ad1\n```\n\n### Dataset Upload\n\nThis utility will upload your dataset to Aptos.\n\n```\n(aptos_env) user@ubuntu:~$ aptos_upload -d ~/datasets/tf_test\n AptosConnector (v0.0.1) - dataset validation utility\n\n----------------------------------------------------------------------------------------------------\nVeryfying dataset signature...\nDone!\nS3 access verified\nFound 3693 files in the dataset: 505.73 MB\n\nUploading file: tfds/tf_test-validation.tfrecord-00000-of-00001\n100%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 3693/3693 [02:25<00:00, 25.46it/s]\n----------------------------------------------------------------------------------------------------\nUpload complete. You can view your dataset at: https://aptos.training/datasets/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXX/tf_test\n```\n\n## Running tests\n\n```\n# install tox test runner:\npython -m pip install --user tox\npython -m tox --help\n\n# install developer dependencies of the package\ncd [...]/AptosConnector\npip install -e .[dev]\n\n# run tests\ntox\n```\n\n## Developed by:\n\n<img src=\"https://etacompute.com/wp-content/uploads/2021/09/eta-logo.svg\" width=\"200\">\n",
"bugtrack_url": null,
"license": "Copyright (c) 2023 Eta Compute Inc. Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \"Software\"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ",
"summary": "A dataset validation and upload utility for Aptos",
"version": "0.0.2",
"project_urls": {
"Bug Reports": "https://github.com/Eta-Compute/AptosConnector/issues",
"Homepage": "https://github.com/Eta-Compute/AptosConnector",
"Source": "https://github.com/Eta-Compute/AptosConnector"
},
"split_keywords": [
"aptos",
"dataset",
"connector",
"upload"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "2a7161979591184821d5d43a2d039cc687006212ae7e98569796bcf2c3f440b7",
"md5": "4864b013db4d14350151e876bea37be2",
"sha256": "644dfea35f4e237ea3bfc3c4b467fe6e598eada7718b3a9ff3fd03750eddca85"
},
"downloads": -1,
"filename": "AptosConnector-0.0.2-py3-none-any.whl",
"has_sig": false,
"md5_digest": "4864b013db4d14350151e876bea37be2",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 19359,
"upload_time": "2024-03-05T18:26:43",
"upload_time_iso_8601": "2024-03-05T18:26:43.130332Z",
"url": "https://files.pythonhosted.org/packages/2a/71/61979591184821d5d43a2d039cc687006212ae7e98569796bcf2c3f440b7/AptosConnector-0.0.2-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "d83b4ad8fce068a28b26de399eb7dca5e5a0bd67e7d73bb82f32d96156253e77",
"md5": "001c8da8a09260bf7bb76e41e42a3b70",
"sha256": "b766552c4b127fbe2b1488f413407220455408bc37d20eefeeb304cd0495ff0f"
},
"downloads": -1,
"filename": "AptosConnector-0.0.2.tar.gz",
"has_sig": false,
"md5_digest": "001c8da8a09260bf7bb76e41e42a3b70",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 6431623,
"upload_time": "2024-03-05T18:26:46",
"upload_time_iso_8601": "2024-03-05T18:26:46.527318Z",
"url": "https://files.pythonhosted.org/packages/d8/3b/4ad8fce068a28b26de399eb7dca5e5a0bd67e7d73bb82f32d96156253e77/AptosConnector-0.0.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-03-05 18:26:46",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "Eta-Compute",
"github_project": "AptosConnector",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"tox": true,
"lcname": "aptosconnector"
}
