# Deploy Custom Environments in MAIA
[](https://github.com/MAIA-KTH/Hive_Gate/actions/workflows/build.yaml)
To create custom environments and deploy applications in MAIA (including pods, services and ingresses)
a Helm chart is available: [Hive_Gate](https://maia-kth.github.io/Hive_Gate/).
With the **Hive_Gate** chart it is possible to deploy any *Docker Image* as a Pod, expose the required ports as
services, mount persistent volumes on the specified locations and optionally create Ingress resources to expose the
application to the external traffic using the HTTPS protocol.
To add the chart to Helm, run:
```
helm repo add hive-gate https://maia-kth.github.io/Hive_Gate/
helm repo update
```
## Custom Helm values
A number of custom parameters can be specified for the Helm chart, including the Docker image to deploy, the port to expose, etc.
The custom configuration is set in a JSON configuration file, following the conventions described below.
### General Configuration
#### Namespace [Required]
Specify the Cluster Namespace where to deploy the resources
```json
{
"namespace": "NAMESPACE_NAME"
}
```
#### Chart Name [Required]
Specify the Helm Chart Release name
```json
{
"chart_name": "Helm_Chart_name"
}
```
#### Docker image [Required]
To specify the Docker image to deploy
```json
{
"docker_image": "DOCKER_IMAGE"
}
```
In case of a custom Docker image, provide the docker build context ( the folder path containing the *Dockerfile* and all the required files), and the corresponding custom
image name. The built Docker image is pushed to the private docker registry.
```json
{
"docker_image": {
"context_folder": "/PATH/TO/MY/DOCKER_CONTEXT",
"image_name": "my-custom-image"
}
}
```
#### Requested Resources [Required]
To request resources (RAM,CPU and optionally GPU).
```json
{
"memory_size" : "REQUESTED_RAM_SIZE",
"cpu" : "REQUESTED_CPUs"
}
```
Optionally, to request GPU usage:
```json
{
"gpu_request" : "NUMBER_OF_GPUs"
}
```
#### Services
To specify which ports (and corresponding services) can be reached from outside the pod.
```json
{
"ports" : {
"SERVICE_NAME_1": ["PORT_NUMBER"],
"SERVICE_NAME_2": ["PORT_NUMBER"]
}
}
```
The default *Service Type* is **ClusterIP**. To expose a service as a type **NodePort**:
```json
{
"service_type": "NodePort",
"ports" : {
"SERVICE_NAME_1": ["PORT_NUMBER", "NODE_PORT_NUMBER"],
"SERVICE_NAME_2": ["PORT_NUMBER", "NODE_PORT_NUMBER"]
}
}
```
#### Persistent Volumes
2 different types of persistent volumes are available: **hostPath** (local folder) and **nfs** (shared nfs folder).
For each of these types, it is possible to request a Persistent Volume via a Persistent Volume Claim, or to directly reference the folder in the host/nfs server to mount.
Direct access:
```json
{
"host_volume_mounts":
{
"host-folder_1": ["/host/folder/path_1", "/mount/path_1","readOnly"],
"host-folder_2": ["/host/folder/path_2", "/mount/path_2"]
},
"nfs_server": "NFS_SERVER_IP",
"nfs_volume_mounts":
{
"nfs-folder_1": ["/nfs/folder/path_1", "/mount/path_1","readOnly"],
"nfs-folder_2": ["/nfs/folder/path_2", "/mount/path_2"]
}
}
```
The *"readOnly"* options can be added to specify the mounted folder as read-only.
Request PVC:
```json
{
"persistent_volume" :
[
{
"mountPath": "/mount/path_1",
"size": "VOLUME_SIZE",
"access_mode": "ACCESS_TYPE",
"pvc_type": "STORAGE_CLASS"
},
{
"mountPath": "/mount/path_2",
"size": "VOLUME_SIZE",
"access_mode": "ACCESS_TYPE",
"pvc_type": "STORAGE_CLASS"
}
]
}
```
**"STORAGE_CLASS"** can be any of the storage classes available on the cluster:
```
kubectl get sc
```
#### Existing Persistent Volumes
Previously created pv can be mounted into multiple pods (ONLY if the *access mode* was previously set to **ReadWriteMany
**)
```json
{
"existing_persistent_volume": [
{
"name": "EXISTING_PVC_NAME",
"mountPath": "/mount/path"
}
]
}
```
#### Mounted files
Single files can be mounted inside the Pod. First, a ConfigMap including the file is created, and then it is mounted
into the Pod.
```json
{
"mount_files": {
"file_name": [
"/local/file/path",
"/file/mount/path"
]
}
}
```
#### Node Selection
To optionally select which node in the cluster to use for deploying the application.
```json
{
"node_selector": "NODE_NAME"
}
```
#### GPU Selection
To optionally select which available GPUs in the cluster to request. `type` and `vram` attribute can be specified (only one of them is needed, both can be included).
Example: `type: "RTX-2070-Super"`, `vram: "8G"`
```json
{
"gpu_selector": {
"type": "GPU_TYPE",
"vram": "VRAM_SIZE"
}
}
```
#### Ingress
Used to create an Ingress resources to access the application at the specified port by using an HTTPS address.
IMPORTANT! The specified DNS needs to be active and connected to the cluster DNS (**".k8s-maia.com"**)
IMPORTANT! *oauth_url* should be explicitly specified, since only oauth-based authenticated users can be authorized
through the ingress.
Contact the MAIA admin to retrieve this information.
```json
{
"ingress": {
"host": "SUBDOMAIN.k8s-maia.com",
"port": "SERVICE_PORT",
"oauth_url": "oauth.MY_NAMESPACE"
}
}
```
#### Environment variables
To add environment variables, used during the creation and deployment of the pod (i.e., environment variables to specify for the Docker Image).
```json
{
"env_variables":
{
"KEY_1": "VAL_1",
"KEY_2": "VAL_2"
}
}
```
### Hive Docker Configuration
#### User info
When deploying Hive-based applications, it is possible to create single-multiple user account in the environment.
For each of the users, *username*, *password* *email*, and, optionally, an *ssh public key* are required.
This information is stored inside Secrets:
```
USER_1_SECRET:
user: USER_1
password: pw
email: user@email.com
ssh_publickey [Optional]: "ssh-rsa ..."
```
To provide the user information to the Pod:
```json
{
"user_secret" :
[
"user-1-secret",
"user-2-secret"
],
"user_secret_params" : ["user","password","email", "ssh_publickey"]
}
```
## Configuration File Example
```json
{
"namespace": "machine-learning",
"chart_name": "jupyterlab-1-v1",
"docker_image": "jupyter/scipy-notebook",
"tag" : "latest",
"memory_size" : "4Gi",
"cpu" : "5000m",
"ports" : {
"jupyter": [8888]
},
"persistent_volume" :
[
{
"mountPath": "/home/jovyan",
"size": "100Gi",
"access_mode": "ReadWriteOnce",
"pvc_type": "local-hostpath"
}
]
}
```
## Deploy Charts
To deploy an Helm Hive Chart, first create a config file according to the specific requirements (as described [above](#Custom Helm values)).
Then install the **Hive_Gate** package running:
```
pip install hive-gate
```
Or download the executable file:
[Hive_Gate_deploy_helm_chart (Windows .exe)]
[Hive_Gate_deploy_helm_chart (Ubuntu)](https://gits-15.sys.kth.se/MAIA/Hive_Gate/releases/download/v1.1/Hive_Gate_deploy_helm_chart)
Finally:
```
Hive_Gate_deploy_helm_chart --config-file /PATH/TO/CONFIG/FILE
```
Raw data
{
"_id": null,
"home_page": "https://gits-15.sys.kth.se/MAIA/Hive_Gate.git",
"name": "hive-gate",
"maintainer": "",
"docs_url": null,
"requires_python": "",
"maintainer_email": "",
"keywords": "helm,kubernetes,maia,resource deployment",
"author": "Simone Bendazzoli",
"author_email": "simben@kth.se",
"download_url": "https://files.pythonhosted.org/packages/e0/e3/32ac500825829cb1d11e9eaafe237623d57529940a3cc896defff9387cae/hive-gate-1.3.tar.gz",
"platform": null,
"description": "# Deploy Custom Environments in MAIA\n\n[](https://github.com/MAIA-KTH/Hive_Gate/actions/workflows/build.yaml)\n\nTo create custom environments and deploy applications in MAIA (including pods, services and ingresses)\na Helm chart is available: [Hive_Gate](https://maia-kth.github.io/Hive_Gate/).\n\nWith the **Hive_Gate** chart it is possible to deploy any *Docker Image* as a Pod, expose the required ports as\nservices, mount persistent volumes on the specified locations and optionally create Ingress resources to expose the\napplication to the external traffic using the HTTPS protocol.\n\nTo add the chart to Helm, run:\n```\nhelm repo add hive-gate https://maia-kth.github.io/Hive_Gate/\nhelm repo update\n```\n\n## Custom Helm values\n\nA number of custom parameters can be specified for the Helm chart, including the Docker image to deploy, the port to expose, etc.\n\nThe custom configuration is set in a JSON configuration file, following the conventions described below.\n\n### General Configuration\n\n#### Namespace [Required]\nSpecify the Cluster Namespace where to deploy the resources\n```json\n{\n \"namespace\": \"NAMESPACE_NAME\"\n}\n```\n#### Chart Name [Required]\nSpecify the Helm Chart Release name\n```json\n{\n \"chart_name\": \"Helm_Chart_name\"\n}\n```\n#### Docker image [Required]\nTo specify the Docker image to deploy\n```json\n{\n \"docker_image\": \"DOCKER_IMAGE\"\n}\n```\nIn case of a custom Docker image, provide the docker build context ( the folder path containing the *Dockerfile* and all the required files), and the corresponding custom\n image name. The built Docker image is pushed to the private docker registry.\n```json\n{\n \"docker_image\": {\n \"context_folder\": \"/PATH/TO/MY/DOCKER_CONTEXT\",\n \"image_name\": \"my-custom-image\"\n }\n}\n```\n#### Requested Resources [Required]\nTo request resources (RAM,CPU and optionally GPU).\n```json\n{\n \"memory_size\" : \"REQUESTED_RAM_SIZE\",\n \"cpu\" : \"REQUESTED_CPUs\"\n}\n```\n\nOptionally, to request GPU usage:\n```json\n{\n \"gpu_request\" : \"NUMBER_OF_GPUs\"\n}\n```\n#### Services\nTo specify which ports (and corresponding services) can be reached from outside the pod.\n```json\n{\n \"ports\" : {\n \"SERVICE_NAME_1\": [\"PORT_NUMBER\"],\n \"SERVICE_NAME_2\": [\"PORT_NUMBER\"]\n }\n}\n\n```\nThe default *Service Type* is **ClusterIP**. To expose a service as a type **NodePort**:\n```json\n{\n \"service_type\": \"NodePort\",\n \"ports\" : {\n \"SERVICE_NAME_1\": [\"PORT_NUMBER\", \"NODE_PORT_NUMBER\"],\n \"SERVICE_NAME_2\": [\"PORT_NUMBER\", \"NODE_PORT_NUMBER\"]\n }\n}\n\n```\n#### Persistent Volumes\n2 different types of persistent volumes are available: **hostPath** (local folder) and **nfs** (shared nfs folder).\nFor each of these types, it is possible to request a Persistent Volume via a Persistent Volume Claim, or to directly reference the folder in the host/nfs server to mount.\n\nDirect access:\n```json\n{\n \"host_volume_mounts\": \n {\n \"host-folder_1\": [\"/host/folder/path_1\", \"/mount/path_1\",\"readOnly\"],\n \"host-folder_2\": [\"/host/folder/path_2\", \"/mount/path_2\"]\n },\n \"nfs_server\": \"NFS_SERVER_IP\",\n \"nfs_volume_mounts\": \n {\n \"nfs-folder_1\": [\"/nfs/folder/path_1\", \"/mount/path_1\",\"readOnly\"],\n \"nfs-folder_2\": [\"/nfs/folder/path_2\", \"/mount/path_2\"]\n }\n}\n```\nThe *\"readOnly\"* options can be added to specify the mounted folder as read-only.\n\nRequest PVC:\n\n```json\n{\n \"persistent_volume\" : \n [\n {\n \"mountPath\": \"/mount/path_1\",\n \"size\": \"VOLUME_SIZE\",\n \"access_mode\": \"ACCESS_TYPE\",\n \"pvc_type\": \"STORAGE_CLASS\"\n },\n {\n \"mountPath\": \"/mount/path_2\",\n \"size\": \"VOLUME_SIZE\",\n \"access_mode\": \"ACCESS_TYPE\",\n \"pvc_type\": \"STORAGE_CLASS\"\n }\n ]\n}\n```\n\n**\"STORAGE_CLASS\"** can be any of the storage classes available on the cluster:\n\n```\nkubectl get sc\n```\n\n#### Existing Persistent Volumes\n\nPreviously created pv can be mounted into multiple pods (ONLY if the *access mode* was previously set to **ReadWriteMany\n**)\n\n```json\n{\n \"existing_persistent_volume\": [\n {\n \"name\": \"EXISTING_PVC_NAME\",\n \"mountPath\": \"/mount/path\"\n }\n ]\n}\n```\n\n#### Mounted files\n\nSingle files can be mounted inside the Pod. First, a ConfigMap including the file is created, and then it is mounted\ninto the Pod.\n\n```json\n{\n \"mount_files\": {\n \"file_name\": [\n \"/local/file/path\",\n \"/file/mount/path\"\n ]\n }\n}\n```\n\n#### Node Selection\nTo optionally select which node in the cluster to use for deploying the application.\n```json\n{\n \"node_selector\": \"NODE_NAME\"\n}\n```\n\n#### GPU Selection\nTo optionally select which available GPUs in the cluster to request. `type` and `vram` attribute can be specified (only one of them is needed, both can be included).\nExample: `type: \"RTX-2070-Super\"`, `vram: \"8G\"`\n```json\n{\n \"gpu_selector\": {\n \"type\": \"GPU_TYPE\",\n \"vram\": \"VRAM_SIZE\"\n }\n}\n```\n\n#### Ingress\nUsed to create an Ingress resources to access the application at the specified port by using an HTTPS address.\n\nIMPORTANT! The specified DNS needs to be active and connected to the cluster DNS (**\".k8s-maia.com\"**)\n\nIMPORTANT! *oauth_url* should be explicitly specified, since only oauth-based authenticated users can be authorized\nthrough the ingress.\nContact the MAIA admin to retrieve this information.\n\n```json\n{\n \"ingress\": {\n \"host\": \"SUBDOMAIN.k8s-maia.com\",\n \"port\": \"SERVICE_PORT\",\n \"oauth_url\": \"oauth.MY_NAMESPACE\"\n }\n}\n```\n\n\n#### Environment variables\nTo add environment variables, used during the creation and deployment of the pod (i.e., environment variables to specify for the Docker Image).\n```json\n{\n \"env_variables\": \n {\n \"KEY_1\": \"VAL_1\",\n \"KEY_2\": \"VAL_2\"\n }\n}\n```\n\n### Hive Docker Configuration\n\n#### User info\nWhen deploying Hive-based applications, it is possible to create single-multiple user account in the environment.\nFor each of the users, *username*, *password* *email*, and, optionally, an *ssh public key* are required.\nThis information is stored inside Secrets:\n```\nUSER_1_SECRET:\n user: USER_1\n password: pw\n email: user@email.com\n ssh_publickey [Optional]: \"ssh-rsa ...\" \n```\nTo provide the user information to the Pod:\n```json\n{\n \"user_secret\" : \n [ \n \"user-1-secret\",\n \"user-2-secret\"\n ],\n \"user_secret_params\" : [\"user\",\"password\",\"email\", \"ssh_publickey\"]\n}\n```\n\n## Configuration File Example\n\n```json\n{\n \"namespace\": \"machine-learning\",\n \"chart_name\": \"jupyterlab-1-v1\",\n \"docker_image\": \"jupyter/scipy-notebook\",\n \"tag\" : \"latest\",\n \"memory_size\" : \"4Gi\",\n \"cpu\" : \"5000m\",\n \"ports\" : {\n \"jupyter\": [8888]\n },\n \"persistent_volume\" :\n [\n {\n \"mountPath\": \"/home/jovyan\",\n \"size\": \"100Gi\",\n \"access_mode\": \"ReadWriteOnce\",\n \"pvc_type\": \"local-hostpath\"\n }\n ]\n}\n```\n\n## Deploy Charts\n\nTo deploy an Helm Hive Chart, first create a config file according to the specific requirements (as described [above](#Custom Helm values)).\nThen install the **Hive_Gate** package running:\n```\npip install hive-gate\n```\n\nOr download the executable file:\n\n[Hive_Gate_deploy_helm_chart (Windows .exe)]\n\n[Hive_Gate_deploy_helm_chart (Ubuntu)](https://gits-15.sys.kth.se/MAIA/Hive_Gate/releases/download/v1.1/Hive_Gate_deploy_helm_chart)\n\nFinally:\n```\nHive_Gate_deploy_helm_chart --config-file /PATH/TO/CONFIG/FILE\n```\n",
"bugtrack_url": null,
"license": "GPLv3",
"summary": "Python interface to MAIA. It can be used as interface to any Kubernetes-based platform.",
"version": "1.3",
"split_keywords": [
"helm",
"kubernetes",
"maia",
"resource deployment"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "c9e5b7168bb7e50fd1c31e38d8268ae434f9ac2a8bd3a686939b785bcc84ea05",
"md5": "1b3b32c43cb7060637fd13f57dcac574",
"sha256": "ee913f16ba435284396e8ce789df5a5dee2f092560142b6e988a937e0a58cd77"
},
"downloads": -1,
"filename": "hive_gate-1.3-py3-none-any.whl",
"has_sig": false,
"md5_digest": "1b3b32c43cb7060637fd13f57dcac574",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 16516,
"upload_time": "2023-03-31T08:16:58",
"upload_time_iso_8601": "2023-03-31T08:16:58.675325Z",
"url": "https://files.pythonhosted.org/packages/c9/e5/b7168bb7e50fd1c31e38d8268ae434f9ac2a8bd3a686939b785bcc84ea05/hive_gate-1.3-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "e0e332ac500825829cb1d11e9eaafe237623d57529940a3cc896defff9387cae",
"md5": "09eae11a561699abfdb7c539940f1146",
"sha256": "b20e1f56685efcf75a444a6bba706e01cf982e998298363c62215ad1a1a9c7d6"
},
"downloads": -1,
"filename": "hive-gate-1.3.tar.gz",
"has_sig": false,
"md5_digest": "09eae11a561699abfdb7c539940f1146",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 17334,
"upload_time": "2023-03-31T08:17:01",
"upload_time_iso_8601": "2023-03-31T08:17:01.088677Z",
"url": "https://files.pythonhosted.org/packages/e0/e3/32ac500825829cb1d11e9eaafe237623d57529940a3cc896defff9387cae/hive-gate-1.3.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-03-31 08:17:01",
"github": false,
"gitlab": false,
"bitbucket": false,
"lcname": "hive-gate"
}
JSON