# GofileIO Uploader
 
A python script to upload files or directories to Gofile.io
Built using `asyncio`, `aiohttp`, and `tqdm`
## Supports
- Gofile accounts
- Private and public directory uploads
- Parallel uploads
- Retries
- Progress bars
- Upload logging
- Skipping duplicate uploads
- Local configs
## Usage
1. `pip install GofileIOUploader`
```
usage: gofile-upload [-h] [-t TOKEN] [-z {na,eu}] [-f FOLDER] [-d]
[--debug-save-js-locally | --no-debug-save-js-locally]
[--rename-existing | --no-rename-existing]
[-c CONNECTIONS] [--timeout TIMEOUT]
[--public | --no-public] [--save | --no-save]
[--use-config | --no-use-config]
[--recurse-directories | --no-recurse-directories]
[--recurse-max RECURSE_MAX]
[--exclude-file-types EXCLUDE_FILE_TYPES]
[--only-file-types ONLY_FILE_TYPES] [-r RETRIES]
[--hash-pool-size HASH_POOL_SIZE]
[--log-level {debug,info,warning,error,critical}]
[--log-file LOG_FILE]
file
Gofile.io Uploader supporting parallel uploads
positional arguments:
file File or directory to look for files in to upload
optional arguments:
-h, --help show this help message and exit
-t TOKEN, --token TOKEN
API token for your account so that you can upload to a
specific account/folder. You can also set the
GOFILE_TOKEN environment variable for this
-z {na,eu}, --zone {na,eu}
Server zone to prefer uploading to
-f FOLDER, --folder FOLDER
Folder to upload files to overriding the directory
name if used
-d, --dry-run Don't create folders or upload files
--debug-save-js-locally, --no-debug-save-js-locally
Debug option to save the retrieved js file locally.
(default: False)
--rename-existing, --no-rename-existing
If a file is already found on the remote server but
the names differ, rename the file to its local name.
(default: True)
-c CONNECTIONS, --connections CONNECTIONS
Maximum parallel uploads to do at once. (default: 6)
--timeout TIMEOUT Number of seconds before aiohttp times out. If a
single upload exceed this time it will fail. This will
depend on internet speed but in best case scenario 5GB
requires 300s. (default: 600)
--public, --no-public
Make all files uploaded public. By default they are
private and not unsharable. (default: False)
--save, --no-save Don't save uploaded file urls to a
"gofile_upload_<unixtime>.csv" file. (default: True)
--use-config, --no-use-config
Whether to create and use a config file in
$HOME/.config/gofile_upload/config.json. (default:
True)
--recurse-directories, --no-recurse-directories
Whether to recursively iterate all directories and
search for files to upload if a directory is given as
the upload file
--recurse-max RECURSE_MAX
Maximum number of files before the program errors out
when using --recurse-directory feature. Put here as
safety feature.
--exclude-file-types EXCLUDE_FILE_TYPES
Exclude files ending with these extensions from being
uploaded. Comma separated values. Example: jpg,png
--only-file-types ONLY_FILE_TYPES
Only upload files ending with these extensions. Comma
separated values. Example: jpg,png
-r RETRIES, --retries RETRIES
How many times to retry a failed upload. (default: 3)
--hash-pool-size HASH_POOL_SIZE
How many md5 hashes to calculate in parallel.
(default: 4)
--log-level {debug,info,warning,error,critical}
Log level. (default: warning)
--log-file LOG_FILE Additional file to log information to. (default: None)
```
## Details
### Duplicate Files
If you try to upload a file and it already exists then the upload will be skipped. This comparison is based on MD5 sums,
This check is based on the account being used. You can upload the same file twice to an account if different directories were specified.
### History
Configs are stored in `$HOME/.config/gofile_upload/config.json` and all successful uploads and md5 sum hashes will be saved in there.
Each time you complete an upload a `gofile_upload_<timestamp>.csv` will be created with the items uploaded and the following metadata:
`filePath,filePathMD5,fileNameMD5,uploadSuccess,code,downloadPage,fileId,fileName,guestToken,md5,parentFolder`
### Local Configuration
A local configuration can be used when you specify the `--use-config` flag
This config is saved in `$HOME/.config/gofile-upload/config.json` and contains the following options:
```json
{
"token": "Optional[str]",
"zone": "Optional[str]",
"connections": "Optional[int]",
"public": "Optional[bool]",
"save": "Optional[bool]",
"retries": "Optional[int]",
"history": {
"md5_sums": {
"<filepath>": "<md5sum of file>"
},
"uploads": [
"<upload response>"
]
}
}
```
This config is loaded at runtime and combined with CLI options and defaults to make one config for the program.
The precedence for this is:
1. CLI Defaults
- connections: 6
- public: False
- retries: 3
- save: True
- debug_save_js_locally: False
- use_config: True
2. Local Config
3. CLI Options
The config will be saved when MD5 sums are calculated for files as well as when uploads are completed.
Configs that have a value of `None` will be omitted.
If you specify `--no-use-config` the local file will not be loaded and will not be saved to.
## Examples
Given
```
directory/
├── sample2.mkv
└── sample.mkv
```
**Upload single file anonymously**
The file will be private
`gofile-upload directory/sample.mkv`
**Upload single file to your account and make it public**
`gofile-upload --token 123 --public directory/sample.mkv`
**Upload single file to directory `foo` in your account and make it public**
`gofile-upload --token 123 --public --folder foo directory/sample.mkv`
**Upload directory to your account and make them public**
`gofile-upload --token 123 --public directory`
**Upload directory to directory `foo` in your account and make them public**
`gofile-upload --token 123 --public --folder foo directory`
# Development
## Optional Prerequesites
- tk `sudo pacman -S base-devel tk`
- [pyenv](https://github.com/pyenv/pyenv?tab=readme-ov-file#set-up-your-shell-environment-for-pyenv)
- [pyenv-virtualenv](https://github.com/pyenv/pyenv-virtualenv?tab=readme-ov-file#installing-as-a-pyenv-plugin)
```bash
pyenv install 3.9
```
## Setup
```bash
pip install -r requirements-dev.txt
pre-commit install
```
## Packaging
This package currently uses [just](https://github.com/casey/just which is a Makefile like utility.
You must install `just` first and then you can do things like `just build` or `just release` which depend on the `justfile` to take actions.
# Testing
**WARNING:** Tests will use a gofile account and are destructive (they will delete all created files).
Do not use your regular account for tests and be careful of running tests in the same environment if a `GOFILE_TOKEN` environment variable exists.
This packages uses [pytest](https://docs.pydantic.dev/latest/) and [pytest-asyncio](https://pytest-asyncio.readthedocs.io/en/latest/) for testing.
In order to omit different pytest async decorators, pytest has its configuration setup in `pyproject.toml` to
```
[tool.pytest.ini_options]
asyncio_mode = "auto"
```
In practice this is of little value because pytest-asyncio seems to be [a mess](https://github.com/pytest-dev/pytest-asyncio/issues/706) when working with fixtures at different scopes.
I've ended up setting all fixtures and tests to session based levels even when it does not make sense but at least this does work.
It also makes use of [pydantic](https://docs.pytest.org) in order to try and validate that certain server responses match the TypedDicts I've setup.
If there is a better way to do this please let me know or draft a PR.
You should set the gofile token in your environment so that new accounts are not created for each test.
This can be done via something like `export GOFILE_TOKEN=123` or your IDE.
To test you can run the following from the root directory:
```
(venv) [alex@xyz gofile-uploader]$ pytest src/gofile_uploader
============================ test session starts ============================
platform linux -- Python 3.9.18, pytest-8.2.2, pluggy-1.5.0
rootdir: /home/alex/PycharmProjects/gofile-uploader
configfile: pyproject.toml
plugins: asyncio-0.23.7
asyncio: mode=auto
collected 3 items
src/gofile_uploader/tests/test_api_servers.py .. [ 66%]
src/gofile_uploader/tests/test_api_wt.py . [100%]
============================= 3 passed in 1.05s =============================
```
## Coverage
Test coverage is also generated using [pytest-cov](https://github.com/pytest-dev/pytest-cov).
You should disable this when debugging.
# Improvements Wishlist
- [ ] Paid accounts support, I don't have a paid account so I can't test
- [ ] Add more tests
- [ ] Recursive directory upload support
# Thanks
- https://stackoverflow.com/questions/68690141/how-to-show-progress-on-aiohttp-post-with-both-form-data-and-file
- https://github.com/londarks/Unofficial-gofile.io-API-Documentation
- https://github.com/Samridh212/File_Uploader_goFile
- https://gofile.io/api
- https://github.com/rkwyu/gofile-dl
- https://stackoverflow.com/questions/1131220/get-the-md5-hash-of-big-files-in-python
- https://packaging.python.org/en/latest/tutorials/packaging-projects/
- https://github.com/f-o
- https://stackoverflow.com/questions/66665336/validate-python-typeddict-at-runtime
Raw data
{
"_id": null,
"home_page": null,
"name": "GofileIOUploader",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.9",
"maintainer_email": "Alex Mi <alexmi3.14@gmail.com>",
"keywords": "gofile, upload, storage, parallel",
"author": null,
"author_email": "Alex Mi <alexmi3.14@gmail.com>",
"download_url": "https://files.pythonhosted.org/packages/1a/49/a1b37fa50384da56ec28e7e29baa2fb29f80551308657029a5066530a00a/gofileiouploader-0.13.0.tar.gz",
"platform": null,
"description": "# GofileIO Uploader\n \n\nA python script to upload files or directories to Gofile.io\nBuilt using `asyncio`, `aiohttp`, and `tqdm`\n\n## Supports\n- Gofile accounts\n- Private and public directory uploads\n- Parallel uploads\n- Retries\n- Progress bars\n- Upload logging\n- Skipping duplicate uploads\n- Local configs\n\n## Usage\n1. `pip install GofileIOUploader`\n\n```\nusage: gofile-upload [-h] [-t TOKEN] [-z {na,eu}] [-f FOLDER] [-d]\n [--debug-save-js-locally | --no-debug-save-js-locally]\n [--rename-existing | --no-rename-existing]\n [-c CONNECTIONS] [--timeout TIMEOUT]\n [--public | --no-public] [--save | --no-save]\n [--use-config | --no-use-config]\n [--recurse-directories | --no-recurse-directories]\n [--recurse-max RECURSE_MAX]\n [--exclude-file-types EXCLUDE_FILE_TYPES]\n [--only-file-types ONLY_FILE_TYPES] [-r RETRIES]\n [--hash-pool-size HASH_POOL_SIZE]\n [--log-level {debug,info,warning,error,critical}]\n [--log-file LOG_FILE]\n file\n\nGofile.io Uploader supporting parallel uploads\n\npositional arguments:\n file File or directory to look for files in to upload\n\noptional arguments:\n -h, --help show this help message and exit\n -t TOKEN, --token TOKEN\n API token for your account so that you can upload to a\n specific account/folder. You can also set the\n GOFILE_TOKEN environment variable for this\n -z {na,eu}, --zone {na,eu}\n Server zone to prefer uploading to\n -f FOLDER, --folder FOLDER\n Folder to upload files to overriding the directory\n name if used\n -d, --dry-run Don't create folders or upload files\n --debug-save-js-locally, --no-debug-save-js-locally\n Debug option to save the retrieved js file locally.\n (default: False)\n --rename-existing, --no-rename-existing\n If a file is already found on the remote server but\n the names differ, rename the file to its local name.\n (default: True)\n -c CONNECTIONS, --connections CONNECTIONS\n Maximum parallel uploads to do at once. (default: 6)\n --timeout TIMEOUT Number of seconds before aiohttp times out. If a\n single upload exceed this time it will fail. This will\n depend on internet speed but in best case scenario 5GB\n requires 300s. (default: 600)\n --public, --no-public\n Make all files uploaded public. By default they are\n private and not unsharable. (default: False)\n --save, --no-save Don't save uploaded file urls to a\n \"gofile_upload_<unixtime>.csv\" file. (default: True)\n --use-config, --no-use-config\n Whether to create and use a config file in\n $HOME/.config/gofile_upload/config.json. (default:\n True)\n --recurse-directories, --no-recurse-directories\n Whether to recursively iterate all directories and\n search for files to upload if a directory is given as\n the upload file\n --recurse-max RECURSE_MAX\n Maximum number of files before the program errors out\n when using --recurse-directory feature. Put here as\n safety feature.\n --exclude-file-types EXCLUDE_FILE_TYPES\n Exclude files ending with these extensions from being\n uploaded. Comma separated values. Example: jpg,png\n --only-file-types ONLY_FILE_TYPES\n Only upload files ending with these extensions. Comma\n separated values. Example: jpg,png\n -r RETRIES, --retries RETRIES\n How many times to retry a failed upload. (default: 3)\n --hash-pool-size HASH_POOL_SIZE\n How many md5 hashes to calculate in parallel.\n (default: 4)\n --log-level {debug,info,warning,error,critical}\n Log level. (default: warning)\n --log-file LOG_FILE Additional file to log information to. (default: None)\n\n```\n## Details\n### Duplicate Files\nIf you try to upload a file and it already exists then the upload will be skipped. This comparison is based on MD5 sums,\nThis check is based on the account being used. You can upload the same file twice to an account if different directories were specified.\n\n### History\nConfigs are stored in `$HOME/.config/gofile_upload/config.json` and all successful uploads and md5 sum hashes will be saved in there.\nEach time you complete an upload a `gofile_upload_<timestamp>.csv` will be created with the items uploaded and the following metadata:\n`filePath,filePathMD5,fileNameMD5,uploadSuccess,code,downloadPage,fileId,fileName,guestToken,md5,parentFolder`\n\n### Local Configuration\nA local configuration can be used when you specify the `--use-config` flag\nThis config is saved in `$HOME/.config/gofile-upload/config.json` and contains the following options:\n```json\n{\n \"token\": \"Optional[str]\",\n \"zone\": \"Optional[str]\",\n \"connections\": \"Optional[int]\",\n \"public\": \"Optional[bool]\",\n \"save\": \"Optional[bool]\",\n \"retries\": \"Optional[int]\",\n \"history\": {\n \"md5_sums\": {\n \"<filepath>\": \"<md5sum of file>\"\n },\n \"uploads\": [\n \"<upload response>\"\n ]\n }\n}\n```\nThis config is loaded at runtime and combined with CLI options and defaults to make one config for the program.\nThe precedence for this is:\n1. CLI Defaults\n - connections: 6\n - public: False\n - retries: 3\n - save: True\n - debug_save_js_locally: False\n - use_config: True\n2. Local Config\n3. CLI Options\n\nThe config will be saved when MD5 sums are calculated for files as well as when uploads are completed.\nConfigs that have a value of `None` will be omitted.\n\nIf you specify `--no-use-config` the local file will not be loaded and will not be saved to.\n\n## Examples\nGiven\n```\ndirectory/\n\u251c\u2500\u2500 sample2.mkv\n\u2514\u2500\u2500 sample.mkv\n```\n**Upload single file anonymously** \nThe file will be private\n\n`gofile-upload directory/sample.mkv`\n\n**Upload single file to your account and make it public**\n\n`gofile-upload --token 123 --public directory/sample.mkv`\n\n**Upload single file to directory `foo` in your account and make it public**\n\n`gofile-upload --token 123 --public --folder foo directory/sample.mkv`\n\n**Upload directory to your account and make them public**\n\n`gofile-upload --token 123 --public directory`\n\n**Upload directory to directory `foo` in your account and make them public**\n\n`gofile-upload --token 123 --public --folder foo directory`\n\n# Development\n## Optional Prerequesites\n- tk `sudo pacman -S base-devel tk`\n- [pyenv](https://github.com/pyenv/pyenv?tab=readme-ov-file#set-up-your-shell-environment-for-pyenv)\n- [pyenv-virtualenv](https://github.com/pyenv/pyenv-virtualenv?tab=readme-ov-file#installing-as-a-pyenv-plugin)\n\n```bash\npyenv install 3.9\n```\n\n## Setup\n```bash\npip install -r requirements-dev.txt\npre-commit install\n```\n\n## Packaging\nThis package currently uses [just](https://github.com/casey/just which is a Makefile like utility.\n\nYou must install `just` first and then you can do things like `just build` or `just release` which depend on the `justfile` to take actions.\n\n# Testing\n**WARNING:** Tests will use a gofile account and are destructive (they will delete all created files). \nDo not use your regular account for tests and be careful of running tests in the same environment if a `GOFILE_TOKEN` environment variable exists.\n\nThis packages uses [pytest](https://docs.pydantic.dev/latest/) and [pytest-asyncio](https://pytest-asyncio.readthedocs.io/en/latest/) for testing.\nIn order to omit different pytest async decorators, pytest has its configuration setup in `pyproject.toml` to\n```\n[tool.pytest.ini_options]\nasyncio_mode = \"auto\"\n```\nIn practice this is of little value because pytest-asyncio seems to be [a mess](https://github.com/pytest-dev/pytest-asyncio/issues/706) when working with fixtures at different scopes. \nI've ended up setting all fixtures and tests to session based levels even when it does not make sense but at least this does work.\n\nIt also makes use of [pydantic](https://docs.pytest.org) in order to try and validate that certain server responses match the TypedDicts I've setup.\nIf there is a better way to do this please let me know or draft a PR.\n\nYou should set the gofile token in your environment so that new accounts are not created for each test.\nThis can be done via something like `export GOFILE_TOKEN=123` or your IDE.\n\nTo test you can run the following from the root directory:\n```\n(venv) [alex@xyz gofile-uploader]$ pytest src/gofile_uploader\n============================ test session starts ============================\nplatform linux -- Python 3.9.18, pytest-8.2.2, pluggy-1.5.0\nrootdir: /home/alex/PycharmProjects/gofile-uploader\nconfigfile: pyproject.toml\nplugins: asyncio-0.23.7\nasyncio: mode=auto\ncollected 3 items \n\nsrc/gofile_uploader/tests/test_api_servers.py .. [ 66%]\nsrc/gofile_uploader/tests/test_api_wt.py . [100%]\n\n============================= 3 passed in 1.05s =============================\n```\n\n## Coverage\nTest coverage is also generated using [pytest-cov](https://github.com/pytest-dev/pytest-cov).\nYou should disable this when debugging.\n\n\n# Improvements Wishlist\n- [ ] Paid accounts support, I don't have a paid account so I can't test\n- [ ] Add more tests\n- [ ] Recursive directory upload support\n\n# Thanks\n- https://stackoverflow.com/questions/68690141/how-to-show-progress-on-aiohttp-post-with-both-form-data-and-file\n- https://github.com/londarks/Unofficial-gofile.io-API-Documentation\n- https://github.com/Samridh212/File_Uploader_goFile\n- https://gofile.io/api\n- https://github.com/rkwyu/gofile-dl\n- https://stackoverflow.com/questions/1131220/get-the-md5-hash-of-big-files-in-python\n- https://packaging.python.org/en/latest/tutorials/packaging-projects/\n- https://github.com/f-o\n- https://stackoverflow.com/questions/66665336/validate-python-typeddict-at-runtime\n",
"bugtrack_url": null,
"license": "MIT License",
"summary": "Gofile.io uploader supporting parallel uploads",
"version": "0.13.0",
"project_urls": {
"Gofile API": "https://gofile.io/api",
"Homepage": "https://github.com/alexmi256/gofile-uploader",
"Repository": "https://github.com/alexmi256/gofile-uploader.git",
"Service": "https://gofile.io"
},
"split_keywords": [
"gofile",
" upload",
" storage",
" parallel"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "b9fd5aabdc1a9be5fef3209052b4a73c58552395a059230a1446dd17fdfa74b6",
"md5": "a93e46578802ed00d3b2f548c77159dd",
"sha256": "3ce2e8af6b9942c74775db3d48a1a7ac5c97db10007fbb29d7e26d24bd1dcd61"
},
"downloads": -1,
"filename": "GofileIOUploader-0.13.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "a93e46578802ed00d3b2f548c77159dd",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9",
"size": 28272,
"upload_time": "2024-08-24T15:23:11",
"upload_time_iso_8601": "2024-08-24T15:23:11.749178Z",
"url": "https://files.pythonhosted.org/packages/b9/fd/5aabdc1a9be5fef3209052b4a73c58552395a059230a1446dd17fdfa74b6/GofileIOUploader-0.13.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "1a49a1b37fa50384da56ec28e7e29baa2fb29f80551308657029a5066530a00a",
"md5": "f7b3ab593b846318d905a7e2db62a88e",
"sha256": "2abeac35c1c8870293ea7615061f918378f423f32a4123289acff9726d7cc303"
},
"downloads": -1,
"filename": "gofileiouploader-0.13.0.tar.gz",
"has_sig": false,
"md5_digest": "f7b3ab593b846318d905a7e2db62a88e",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9",
"size": 22323,
"upload_time": "2024-08-24T15:23:12",
"upload_time_iso_8601": "2024-08-24T15:23:12.695567Z",
"url": "https://files.pythonhosted.org/packages/1a/49/a1b37fa50384da56ec28e7e29baa2fb29f80551308657029a5066530a00a/gofileiouploader-0.13.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-08-24 15:23:12",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "alexmi256",
"github_project": "gofile-uploader",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [
{
"name": "aiohttp",
"specs": []
},
{
"name": "tqdm",
"specs": []
},
{
"name": "typing-extensions",
"specs": []
}
],
"lcname": "gofileiouploader"
}
JSON
