# gvmkit-build
Golem VM Image builder used as companion app for Golem Registry: https://registry.golem.network
## Requirements
Running docker engine is required. Tool supports Linux, Windows and macOS.
Note that when using macOS ARM version use --platform linux/amd64 option for docker builds.
## Installation
You can install gvmkit-build using pip (python3 with pip installer is required)
```
pip install gvmkit-build
```
or install form npm (npm installation is required)
```
npm install -g gvmkit-build
```
or install using cargo (Rust toolchain is required, a bit slow because it compiles from sources)
```
cargo install gvmkit-build
```
or download prebuild from github releases page:
https://github.com/golemfactory/gvmkit-build-rs/releases
or build from sources, you can find binary in target/release/gvmkit-build (or gvmkit-build.exe on Windows)
```
cargo build --release
```
## Images
Golem Network is using gvmi images as base for creating VMs for tasks.
These images are basically squashfs images with some additional metadata.
They can be prepared from docker images using this (gvmkit-build) tool.
## Quick start
1. Make sure your docker service is running and you have gvmkit-build installed
2. Go to the folder with your dockerfile and run
```docker build . -t my_image```
3. Create account on registry portal https://registry.golem.network
Let's assume your user name is golem
4. Create repository on registry portal
Let's assume you created repository named my_example
5. Create and copy personal access token from registry portal
6. Run (you will be asked for login and personal access token)
```gvmkit-build my_image --push-to golem/my_example:latest```
7. Your tag ```golem/my_example:latest``` is ready to use in one of Golem Network APIs
## Naming image
The tool as main argument takes docker image name.
Docker image name can be resolved using ImageId + tag or Repository name + tag.
Repository name can be composed of maximum two parts: ```<username>/<repository>```
Examples:
```python``` resolves to ```python:latest```
```python:3.8``` resolves to ```python:3.8```
```golemfactory/blender``` resolves to ```golemfactory/blender:latest```
You can also use image id instead of name, use ```docker image ls``` to find your image id.
Following command will build image and create *.gvmi file in current directory.
```
gvmkit-build <image_name>
```
If image not exist locally the tool is trying to pull it from docker hub.
To use it in Golem Network, you have to upload it to registry portal.
To successfully add image to registry portal you have to name image accordingly or use
```
gvmkit-build <user_name>/<image_name>:<tag> --push
```
or if your local image name is not compatible use
```
gvmkit-build <docker_image_id> --push-to <user_name>/<image_name>:<tag>
```
## Build process explained a bit
Tool is creating new container and is copying data from given image to new container.
After copying is finished mksquashfs command is used inside tool container to create squashfs image.
After adding metadata *.gvmi file is created and tool container removed.
Note that tool image will stay downloaded on your machine for future use (it is quite small so no worry about disk space)
## [Optional] - building squashfs-tools image
If you want to use your own tool without pulling from dockerhub:
Go to squashfs-tools directory and run
```
docker build . -t my_squash_fs_builder
```
add environment variable to .env in folder where you run gvmkit-build
```
SQUASHFS_IMAGE_NAME=my_squash_fs_builder
```
For managing docker images and containers bollard library is used. https://docs.rs/bollard/latest/bollard/
## Troubleshooting login to registry portal
The tool is using https://registry.golem.network as default registry portal.
You can change this behaviour by setting `REGISTRY_URL` environment variable.
You should create account on registry portal and generate access token for your user.
If you really don't want to create account you can use anonymous upload (see section below about anonymous upload).
The tool will ask for login when --login --push or --push-to option is specified.
For storing login information rpassword library is used: https://docs.rs/crate/rpassword/latest
Only one instance of login/token is kept saved at "gvmkit-build-rs/default". So if you login with new user/token old pair will be forgotten.
This option will ask for login to registry portal
```
gvmkit-build --login
```
You can use command for check if your login information is correct
```
gvmkit-build --login-check
```
If you want to forget your login information you can use, it will clear your login information
```
gvmkit-build --logout
```
Above command are optional, because you will be asked for login automatically if you use --push or --push-to option.
On some systems when no secure store is provided you will be forced to use following method of keeping login/token pair.
You can optionally put them in .env file from convenience in your working directory.
Remember to set proper permissions to the file (chmod 600 .env) if using on shared machine.
```
REGISTRY_USER=<username-in-registry-portal>
REGISTRY_TOKEN=<access-token-generated-in-registry-portal>
```
When REGISTRY_USER/REGISTRY_TOKEN is set in environment variables, secure rpassword storage won't be used.
## Uploading to multiple tags
The tool cannot upload to multiple tags, but you can call it multiple times with different tags.
All steps of operations are cached so no worry about re-uploading same file multiple times.
## Uploading large files
The tool is using chunked upload with default chunk size 10MiB for images greater than 500MiB (changing not recommended).
Four upload workers are created by default (you can increase/decrease number of workers using --upload-workers argument depending on your network conditions).
If you think your upload is stuck you can always stop and run tool again to finish download. Only chunks that were not uploaded
will be uploaded again.
Note: Total limit of chunks is set to 1000 (so around 10GB by default). If you want to upload larger file you have to set greater chunk size accordingly.
## Uploading image without login
You can upload images anonymously. Note that lifetime of such images is limited,
and they can be removed from registry portal after some time without notice
```
cargo run --release -- <image_name> --push --nologin
```
## Uploading image without building part
If you are sure that you have proper *.gvmi file for example my-test.gvmi you can use
```
gvmkit-build --direct-file-upload my-test.gvmi --push-to <user_name>/<image_name>:<tag>
```
or anonymously
```
gvmkit-build --direct-file-upload my-test.gvmi --push --nologin
```
## Changing squashfs options when creating image
You can change compression used in mksquashfs to produce more or less compact images.
Look for help for more information. Note that currently zstd is not supported by Golem Network (you can use xz instead for extra compact images).
```
gvmkit-build --help
```
Raw data
{
"_id": null,
"home_page": "",
"name": "gvmkit-build",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.7",
"maintainer_email": "",
"keywords": "Golem",
"author": "Golem Factory <contact@golem.network>",
"author_email": "Golem Factory <contact@golem.network>",
"download_url": "",
"platform": null,
"description": "# gvmkit-build\n\nGolem VM Image builder used as companion app for Golem Registry: https://registry.golem.network\n\n## Requirements\n\nRunning docker engine is required. Tool supports Linux, Windows and macOS.\nNote that when using macOS ARM version use --platform linux/amd64 option for docker builds.\n\n## Installation\n\nYou can install gvmkit-build using pip (python3 with pip installer is required)\n```\npip install gvmkit-build\n```\nor install form npm (npm installation is required)\n```\nnpm install -g gvmkit-build\n```\nor install using cargo (Rust toolchain is required, a bit slow because it compiles from sources)\n```\ncargo install gvmkit-build\n```\nor download prebuild from github releases page:\n\nhttps://github.com/golemfactory/gvmkit-build-rs/releases\n\nor build from sources, you can find binary in target/release/gvmkit-build (or gvmkit-build.exe on Windows)\n```\ncargo build --release\n```\n\n## Images\n\nGolem Network is using gvmi images as base for creating VMs for tasks.\nThese images are basically squashfs images with some additional metadata.\nThey can be prepared from docker images using this (gvmkit-build) tool.\n\n## Quick start\n\n1. Make sure your docker service is running and you have gvmkit-build installed\n\n2. Go to the folder with your dockerfile and run\n\n```docker build . -t my_image```\n\n3. Create account on registry portal https://registry.golem.network\n\nLet's assume your user name is golem\n\n4. Create repository on registry portal\n\nLet's assume you created repository named my_example\n\n5. Create and copy personal access token from registry portal\n\n6. Run (you will be asked for login and personal access token)\n\n```gvmkit-build my_image --push-to golem/my_example:latest```\n\n7. Your tag ```golem/my_example:latest``` is ready to use in one of Golem Network APIs\n\n## Naming image\n\nThe tool as main argument takes docker image name.\n\nDocker image name can be resolved using ImageId + tag or Repository name + tag.\nRepository name can be composed of maximum two parts: ```<username>/<repository>```\n\nExamples:\n```python``` resolves to ```python:latest```\n```python:3.8``` resolves to ```python:3.8```\n```golemfactory/blender``` resolves to ```golemfactory/blender:latest```\nYou can also use image id instead of name, use ```docker image ls``` to find your image id.\n\nFollowing command will build image and create *.gvmi file in current directory.\n\n```\ngvmkit-build <image_name>\n```\n\nIf image not exist locally the tool is trying to pull it from docker hub.\n\nTo use it in Golem Network, you have to upload it to registry portal.\n\nTo successfully add image to registry portal you have to name image accordingly or use\n```\ngvmkit-build <user_name>/<image_name>:<tag> --push\n```\nor if your local image name is not compatible use\n```\ngvmkit-build <docker_image_id> --push-to <user_name>/<image_name>:<tag>\n```\n\n## Build process explained a bit\n\nTool is creating new container and is copying data from given image to new container.\nAfter copying is finished mksquashfs command is used inside tool container to create squashfs image.\nAfter adding metadata *.gvmi file is created and tool container removed.\nNote that tool image will stay downloaded on your machine for future use (it is quite small so no worry about disk space)\n\n## [Optional] - building squashfs-tools image\n\nIf you want to use your own tool without pulling from dockerhub:\nGo to squashfs-tools directory and run\n```\ndocker build . -t my_squash_fs_builder\n```\nadd environment variable to .env in folder where you run gvmkit-build\n```\nSQUASHFS_IMAGE_NAME=my_squash_fs_builder \n```\n\nFor managing docker images and containers bollard library is used. https://docs.rs/bollard/latest/bollard/\n\n## Troubleshooting login to registry portal\n\nThe tool is using https://registry.golem.network as default registry portal.\nYou can change this behaviour by setting `REGISTRY_URL` environment variable.\nYou should create account on registry portal and generate access token for your user.\nIf you really don't want to create account you can use anonymous upload (see section below about anonymous upload).\n\nThe tool will ask for login when --login --push or --push-to option is specified.\n\nFor storing login information rpassword library is used: https://docs.rs/crate/rpassword/latest\n\nOnly one instance of login/token is kept saved at \"gvmkit-build-rs/default\". So if you login with new user/token old pair will be forgotten.\n\nThis option will ask for login to registry portal\n```\ngvmkit-build --login\n```\nYou can use command for check if your login information is correct\n```\ngvmkit-build --login-check \n```\nIf you want to forget your login information you can use, it will clear your login information\n```\ngvmkit-build --logout \n```\n\nAbove command are optional, because you will be asked for login automatically if you use --push or --push-to option.\n\nOn some systems when no secure store is provided you will be forced to use following method of keeping login/token pair.\nYou can optionally put them in .env file from convenience in your working directory.\nRemember to set proper permissions to the file (chmod 600 .env) if using on shared machine.\n\n```\nREGISTRY_USER=<username-in-registry-portal>\nREGISTRY_TOKEN=<access-token-generated-in-registry-portal>\n```\n\nWhen REGISTRY_USER/REGISTRY_TOKEN is set in environment variables, secure rpassword storage won't be used.\n\n## Uploading to multiple tags\n\nThe tool cannot upload to multiple tags, but you can call it multiple times with different tags.\nAll steps of operations are cached so no worry about re-uploading same file multiple times.\n\n## Uploading large files\n\nThe tool is using chunked upload with default chunk size 10MiB for images greater than 500MiB (changing not recommended).\nFour upload workers are created by default (you can increase/decrease number of workers using --upload-workers argument depending on your network conditions). \nIf you think your upload is stuck you can always stop and run tool again to finish download. Only chunks that were not uploaded\nwill be uploaded again.\n\nNote: Total limit of chunks is set to 1000 (so around 10GB by default). If you want to upload larger file you have to set greater chunk size accordingly.\n\n## Uploading image without login\n\nYou can upload images anonymously. Note that lifetime of such images is limited, \nand they can be removed from registry portal after some time without notice\n\n```\ncargo run --release -- <image_name> --push --nologin\n```\n\n## Uploading image without building part\n\nIf you are sure that you have proper *.gvmi file for example my-test.gvmi you can use \n\n```\ngvmkit-build --direct-file-upload my-test.gvmi --push-to <user_name>/<image_name>:<tag>\n```\nor anonymously\n```\ngvmkit-build --direct-file-upload my-test.gvmi --push --nologin\n```\n\n## Changing squashfs options when creating image\n\nYou can change compression used in mksquashfs to produce more or less compact images. \n\nLook for help for more information. Note that currently zstd is not supported by Golem Network (you can use xz instead for extra compact images).\n```\ngvmkit-build --help\n```\n\n",
"bugtrack_url": null,
"license": "GPL-3.0",
"summary": "Golem VM Image builder.",
"version": "0.3.18",
"project_urls": {
"repository": "https://github.com/golemfactory/gvmkit-build-rs"
},
"split_keywords": [
"golem"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "5653b4f99e1137245c2fa2ef96a9ebd753f74fd93a4d54fa67e43a9bd8e8a386",
"md5": "2b434a49d2770786871c0379d959a967",
"sha256": "9fc83666a954dca1e5d37079cbf9374cfe2691344f7884d69eb0ce2a6278f761"
},
"downloads": -1,
"filename": "gvmkit_build-0.3.18-py3-none-macosx_10_12_x86_64.whl",
"has_sig": false,
"md5_digest": "2b434a49d2770786871c0379d959a967",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7",
"size": 2669583,
"upload_time": "2023-12-11T15:40:58",
"upload_time_iso_8601": "2023-12-11T15:40:58.376764Z",
"url": "https://files.pythonhosted.org/packages/56/53/b4f99e1137245c2fa2ef96a9ebd753f74fd93a4d54fa67e43a9bd8e8a386/gvmkit_build-0.3.18-py3-none-macosx_10_12_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "3745d92ce94b7f2ee863ba6fa931bb49870d81c8881e984fc69d0bf4473fe530",
"md5": "f7487c8ee61566abe499462fc302f744",
"sha256": "cfbbd51e6ae85a080d0e93c8bf36cf1babfd1a634cc8586a0cd41c735b6a72ba"
},
"downloads": -1,
"filename": "gvmkit_build-0.3.18-py3-none-macosx_11_0_arm64.whl",
"has_sig": false,
"md5_digest": "f7487c8ee61566abe499462fc302f744",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7",
"size": 2458000,
"upload_time": "2023-12-11T15:41:00",
"upload_time_iso_8601": "2023-12-11T15:41:00.044075Z",
"url": "https://files.pythonhosted.org/packages/37/45/d92ce94b7f2ee863ba6fa931bb49870d81c8881e984fc69d0bf4473fe530/gvmkit_build-0.3.18-py3-none-macosx_11_0_arm64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "1142294f6d88e1837322265dc36b00a4af857f01a764659463cdee60cfa249e6",
"md5": "563117afc2303b57e195d96d7d0e34c5",
"sha256": "9f82b05422426700e193d80787a8de5b7304d35eab5863e7fa20499f68db7156"
},
"downloads": -1,
"filename": "gvmkit_build-0.3.18-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl",
"has_sig": false,
"md5_digest": "563117afc2303b57e195d96d7d0e34c5",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7",
"size": 3319759,
"upload_time": "2023-12-11T15:41:02",
"upload_time_iso_8601": "2023-12-11T15:41:02.314033Z",
"url": "https://files.pythonhosted.org/packages/11/42/294f6d88e1837322265dc36b00a4af857f01a764659463cdee60cfa249e6/gvmkit_build-0.3.18-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "2e18c0bb7f3e34aafc7129e08f4cffddef172ca4c300f74f8a72763424100728",
"md5": "2543a3238dc6237281479de23cef7843",
"sha256": "3ed509dc6ef834298e29d030711f6f464c4762b9d38d8f2acb2ae26e09332316"
},
"downloads": -1,
"filename": "gvmkit_build-0.3.18-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"has_sig": false,
"md5_digest": "2543a3238dc6237281479de23cef7843",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7",
"size": 3527430,
"upload_time": "2023-12-11T15:41:03",
"upload_time_iso_8601": "2023-12-11T15:41:03.854420Z",
"url": "https://files.pythonhosted.org/packages/2e/18/c0bb7f3e34aafc7129e08f4cffddef172ca4c300f74f8a72763424100728/gvmkit_build-0.3.18-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "6b2aec2129c241533249dfac3fb2ecd13b653024e88857d2cb8f459d57117d16",
"md5": "4ea74f355a423d45c952e1871e511e2e",
"sha256": "720ce44d0460611b0bcc812d900443584528bcaca8a16d6fbc30417bf42138e1"
},
"downloads": -1,
"filename": "gvmkit_build-0.3.18-py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl",
"has_sig": false,
"md5_digest": "4ea74f355a423d45c952e1871e511e2e",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7",
"size": 3612963,
"upload_time": "2023-12-11T15:41:05",
"upload_time_iso_8601": "2023-12-11T15:41:05.866626Z",
"url": "https://files.pythonhosted.org/packages/6b/2a/ec2129c241533249dfac3fb2ecd13b653024e88857d2cb8f459d57117d16/gvmkit_build-0.3.18-py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "3c43379c60632ac0e7cd569d9db52c7827628635ae6d13106b389389764f891d",
"md5": "b8864f9d39cea47b4b3948e2c7edd805",
"sha256": "6b017e50565933f70b3d72e2efb8a77cc118770c3fbb3e6ebd9c65d87162e626"
},
"downloads": -1,
"filename": "gvmkit_build-0.3.18-py3-none-musllinux_1_2_aarch64.whl",
"has_sig": false,
"md5_digest": "b8864f9d39cea47b4b3948e2c7edd805",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7",
"size": 3319745,
"upload_time": "2023-12-11T15:41:07",
"upload_time_iso_8601": "2023-12-11T15:41:07.353268Z",
"url": "https://files.pythonhosted.org/packages/3c/43/379c60632ac0e7cd569d9db52c7827628635ae6d13106b389389764f891d/gvmkit_build-0.3.18-py3-none-musllinux_1_2_aarch64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "3ed9965f1a78ccc1ed24c889d3e28f6b835d6ca1b5b6d82a8837470235e6786b",
"md5": "e67483f035b3ced576e67786e33ae89a",
"sha256": "3c9a60bfa2adf0c2eebfa6dffb0a03d14ea6c2c442070f58125c10255d55dbd5"
},
"downloads": -1,
"filename": "gvmkit_build-0.3.18-py3-none-musllinux_1_2_x86_64.whl",
"has_sig": false,
"md5_digest": "e67483f035b3ced576e67786e33ae89a",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7",
"size": 3612953,
"upload_time": "2023-12-11T15:41:08",
"upload_time_iso_8601": "2023-12-11T15:41:08.963691Z",
"url": "https://files.pythonhosted.org/packages/3e/d9/965f1a78ccc1ed24c889d3e28f6b835d6ca1b5b6d82a8837470235e6786b/gvmkit_build-0.3.18-py3-none-musllinux_1_2_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "50ca2b6a5925e14a1cfd1b7e348454909383a9cbc5c4db627a96cd9249cde5b3",
"md5": "210c6bdcd60dbca73f811b1a57c3f6f9",
"sha256": "beacd6da7b025c8f07ff90e84f4a234f33062c4b70aa7937d02d0516bdc1c129"
},
"downloads": -1,
"filename": "gvmkit_build-0.3.18-py3-none-win_amd64.whl",
"has_sig": false,
"md5_digest": "210c6bdcd60dbca73f811b1a57c3f6f9",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7",
"size": 2648938,
"upload_time": "2023-12-11T15:41:11",
"upload_time_iso_8601": "2023-12-11T15:41:11.087459Z",
"url": "https://files.pythonhosted.org/packages/50/ca/2b6a5925e14a1cfd1b7e348454909383a9cbc5c4db627a96cd9249cde5b3/gvmkit_build-0.3.18-py3-none-win_amd64.whl",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-12-11 15:40:58",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "golemfactory",
"github_project": "gvmkit-build-rs",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "gvmkit-build"
}
JSON
