| Name | wandb-offline-sync JSON |
| Version |
0.0.14
JSON |
| download |
| home_page | |
| Summary | Continuously sync offline wandb runs |
| upload_time | 2023-10-11 16:51:04 |
| maintainer | |
| docs_url | None |
| author | |
| requires_python | >=3.7 |
| license | Copyright (c) 2023 WandbOfflineSync Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. |
| keywords |
wandb
offline
sync
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# Wandb Offline Sync
Continuously sync offline wandb runs.
## Why?
If you work on computing nodes without internet access, you can use wandb in offline mode to log your runs. Normally you would sync an offline run to wandb at the end of the run, but if you have a long running job, you may want to sync your run during job's execution.
## How it works
This project has two components:
* Farm: a https server that listens for sync requests and syncs the runs to wandb.
* Agent: a python module that you use in your code to request the syncs to the farm.
## Quickstart
* Install this package: `pip install wandb-offline-sync`
* Generate a SSL certificate for the sync farm. You can use the command: `openssl req -newkey rsa:4096 -nodes -keyout key.pem -x509 -days 365 -out cert.pem`. This command will create two files: `cert.pem` and `key.pem`.
* Run `export WANDB_SYNC_FARM_USERNAME=<your_username>; export WANDB_SYNC_FARM_PASSWORD=<your_password>` to set the username and password for the sync farm. You can also put these commands in your `.bashrc` file. Replace `<your_username>` and `<your_password>` with your credentials. If these variables are not set, the farm will use the default credentials `("user", "pass")`
* Run the farm with the command `wandb_sync_farm --cert=<path_to_cert.pem> --key=<path_to_key.pem>` in a node with internet connection. The farm will listen for sync requests. Run `wandb_sync_farm --help` to see all the available options.
* Run `export WANDB_SYNC_FARM_HOST=<sync_farm_ip_address_or_hostname>; export WANDB_SYNC_FARM_PORT=<sync_farm_port>` to set the hostname and port of the sync farm. You can also put these commands in your `.bashrc` file. These variables will be used by the agent.
In the code of your job:
* Import the agent: `from wandb_offline_sync import agent`
* After calling `wandb.init(...)`, initialize the agent with: `agent.init(...)`. You can pass a `frequency` argument to set the minimum time between syncs (in seconds). For example, if you set `frequency=60`, the agent will request a sync at most once per minute. The default value for the frequency is 5 minutes.
Each time `wandb.log(...)` is called, the agent will check if the minimum time interval (given by the frequency) between syncs has passed, and if so, it will request a sync to the farm.
When `wandb.finish()` is called, the agent will wait some seconds (default is 30) to ensure that previous syncs have been completed, and then it will request a final sync to the farm.
## Notes
* If you run the farm with `--verbose`, it may happen that in the first minutes of the run, the output shows the error `.wandb file is empty`, and the run is not synced to the wandb server. Don't worry, after some minutes the data of the run will be available and will be synced to the wandb server.
Raw data
{
"_id": null,
"home_page": "",
"name": "wandb-offline-sync",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.7",
"maintainer_email": "",
"keywords": "wandb,offline,sync",
"author": "",
"author_email": "Martin Menabue <martinmnb@live.it>",
"download_url": "https://files.pythonhosted.org/packages/dc/8d/e666fed17a79b2f770aa14b295ddc5c985fb78bd1964b35682ae1995f7ea/wandb_offline_sync-0.0.14.tar.gz",
"platform": null,
"description": "# Wandb Offline Sync\nContinuously sync offline wandb runs.\n\n## Why?\nIf you work on computing nodes without internet access, you can use wandb in offline mode to log your runs. Normally you would sync an offline run to wandb at the end of the run, but if you have a long running job, you may want to sync your run during job's execution.\n\n## How it works\nThis project has two components:\n* Farm: a https server that listens for sync requests and syncs the runs to wandb.\n* Agent: a python module that you use in your code to request the syncs to the farm.\n\n## Quickstart\n* Install this package: `pip install wandb-offline-sync`\n* Generate a SSL certificate for the sync farm. You can use the command: `openssl req -newkey rsa:4096 -nodes -keyout key.pem -x509 -days 365 -out cert.pem`. This command will create two files: `cert.pem` and `key.pem`.\n* Run `export WANDB_SYNC_FARM_USERNAME=<your_username>; export WANDB_SYNC_FARM_PASSWORD=<your_password>` to set the username and password for the sync farm. You can also put these commands in your `.bashrc` file. Replace `<your_username>` and `<your_password>` with your credentials. If these variables are not set, the farm will use the default credentials `(\"user\", \"pass\")`\n* Run the farm with the command `wandb_sync_farm --cert=<path_to_cert.pem> --key=<path_to_key.pem>` in a node with internet connection. The farm will listen for sync requests. Run `wandb_sync_farm --help` to see all the available options.\n* Run `export WANDB_SYNC_FARM_HOST=<sync_farm_ip_address_or_hostname>; export WANDB_SYNC_FARM_PORT=<sync_farm_port>` to set the hostname and port of the sync farm. You can also put these commands in your `.bashrc` file. These variables will be used by the agent.\n\nIn the code of your job:\n* Import the agent: `from wandb_offline_sync import agent`\n* After calling `wandb.init(...)`, initialize the agent with: `agent.init(...)`. You can pass a `frequency` argument to set the minimum time between syncs (in seconds). For example, if you set `frequency=60`, the agent will request a sync at most once per minute. The default value for the frequency is 5 minutes.\n\nEach time `wandb.log(...)` is called, the agent will check if the minimum time interval (given by the frequency) between syncs has passed, and if so, it will request a sync to the farm.\n\nWhen `wandb.finish()` is called, the agent will wait some seconds (default is 30) to ensure that previous syncs have been completed, and then it will request a final sync to the farm.\n\n## Notes\n* If you run the farm with `--verbose`, it may happen that in the first minutes of the run, the output shows the error `.wandb file is empty`, and the run is not synced to the wandb server. Don't worry, after some minutes the data of the run will be available and will be synced to the wandb server.\n",
"bugtrack_url": null,
"license": "Copyright (c) 2023 WandbOfflineSync Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \"Software\"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.",
"summary": "Continuously sync offline wandb runs",
"version": "0.0.14",
"project_urls": {
"Bug Tracker": "https://github.com/MartinMenabue/WandbOfflineSync/issues",
"Homepage": "https://github.com/MartinMenabue/WandbOfflineSync"
},
"split_keywords": [
"wandb",
"offline",
"sync"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "77b6ab97189a611e35878c03dd2b6381b20480e18465c0398fc3bcdb90e47336",
"md5": "1c5d0bc133f9152faa14b093ca1dcc20",
"sha256": "b856d9960d175ca0bd825d99bbbf54efee02cb22e10a65e8a924dcf02ca6728c"
},
"downloads": -1,
"filename": "wandb_offline_sync-0.0.14-py3-none-any.whl",
"has_sig": false,
"md5_digest": "1c5d0bc133f9152faa14b093ca1dcc20",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7",
"size": 8129,
"upload_time": "2023-10-11T16:51:02",
"upload_time_iso_8601": "2023-10-11T16:51:02.780213Z",
"url": "https://files.pythonhosted.org/packages/77/b6/ab97189a611e35878c03dd2b6381b20480e18465c0398fc3bcdb90e47336/wandb_offline_sync-0.0.14-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "dc8de666fed17a79b2f770aa14b295ddc5c985fb78bd1964b35682ae1995f7ea",
"md5": "fa816c006d4edccfe6581ee6662d717f",
"sha256": "1ee46317d0f88b8f395ba419f45e348bc20080752d01ae1e3a72192a67bb1363"
},
"downloads": -1,
"filename": "wandb_offline_sync-0.0.14.tar.gz",
"has_sig": false,
"md5_digest": "fa816c006d4edccfe6581ee6662d717f",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7",
"size": 8118,
"upload_time": "2023-10-11T16:51:04",
"upload_time_iso_8601": "2023-10-11T16:51:04.347587Z",
"url": "https://files.pythonhosted.org/packages/dc/8d/e666fed17a79b2f770aa14b295ddc5c985fb78bd1964b35682ae1995f7ea/wandb_offline_sync-0.0.14.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-10-11 16:51:04",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "MartinMenabue",
"github_project": "WandbOfflineSync",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"requirements": [],
"lcname": "wandb-offline-sync"
}
