| Name | isolate-controller JSON |
| Version |
0.0.1a0
JSON |
| download |
| home_page | |
| Summary | Managed isolated environments for Python |
| upload_time | 2022-12-06 02:29:39 |
| maintainer | |
| docs_url | None |
| author | Features & Labels |
| requires_python | >=3.7,<4 |
| license | |
| keywords |
|
| VCS |
|
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# fal-isolate-cloud
## Regenerating gRPC definitions
- Ensure you are on Python 3.9 or higher
- From the root directory execute the following commands:
```sh
$ pip install grpcio_tools mypy-protobuf refactor
$ python tools/regen_grpc.py src/isolate_controller/definitions/controller.proto
```
## Building the images
- From the root directory
```sh
$ docker build -f infra/worker/Dockerfile -t isolate-cloud-worker .
$ docker build -f infra/controller/Dockerfile -t isolate-cloud-controller .
```
## Testing the server (locally)
- Assuming you built the image `isolate-cloud-worker`, try the following setup:
```sh
$ docker run --rm -e CONTROLLER_KEY=local_secret -p 50001:50001 isolate-cloud-worker
$ ISOLATE_TEST_MODE=1 python -m isolate_controller.server
# CHANGE THE IP FIRST
$ python test_script.py
```
## Setting kubectl with the GCP cluster
- Install the GKE plugin if you don't have it
- Get the clusters list
```sh
$ gcloud components install gke-gcloud-auth-plugin
$ gcloud container clusters get-credentials <cluster-name> --zone <your-zone>
$ kubectl get pods -A
# should print kube-system pods
```
## Deploying
### Deploying the worker
```sh
$ gcloud builds submit --config=infra/worker/cloudbuild.yml .
```
### Deploying the controller
```sh
$ export PROJECT_ID=$(gcloud config list --format 'value(core.project)' 2>/dev/null)
$ gcloud builds submit --config=infra/controller/cloudbuild.yml .
$ export IMAGE_SHA=$(python tools/update_images.py)
$ envsubst < infra/controller/controller.yml | kubectl apply -f -
# Run this just once to expose the service:
$ kubectl expose deployment fal-isolate-controller --type=LoadBalancer --name=load-balancer
```
### Connect redis
Make sure to do this when you are setting up the first time
https://cloud.google.com/memorystore/docs/redis/connect-redis-instance#connecting-cluster
```sh
$ gcloud redis instances list --region=us-central1
# Take note of the HOST value
$ export REDISHOST_IP=...
```
```sh
$ export REDISHOST_IP=...
$ git clone https://github.com/bowei/k8s-custom-iptables.git
$ cd k8s-custom-iptables/
$ TARGETS=${REDISHOST_IP} ./install.sh
```
```sh
$ export REDISHOST_IP=...
$ kubectl create configmap redishost --from-literal=REDISHOST=${REDISHOST_IP}
```
### Setup NFS cache
Make sure to do this when you are setting up the first time
```sh
$ gcloud filestore instances list
# Take note of the IP_ADDRESS
$ export NFS_SERVER_IP=...
```
```sh
$ export NFS_SERVER_IP=...
$ envsubst < infra/fileserver/nfs-server.yml | kubectl apply -f -
$ kubectl apply -f infra/fileserver/claim.yml
```
### Bind Google Service Account to Kubernetes Service Account
Make sure to do this when you are setting up the first time
This is necessary to [connect Postgres](#connect-postgres)
```sh
$ kubectl annotate serviceaccount default iam.gke.io/gcp-service-account=gke-service-account@${PROJECT_ID}.iam.gserviceaccount.com
```
### Connect Postgres
Make sure to do this when you are setting up the first time
```sh
# xargs to remove extra quotes
$ cd fal_terraform/environments/<environment>
$ export SQITCH_PASSWORD=$(terraform output isolate_cloud_sql_user_server_password | xargs)
$ kubectl create secret generic sqlaccess \
--from-literal=SQL_DBNAME=users \
--from-literal=SQL_USER=gke_service \
--from-literal=SQL_PASSWORD=$SQITCH_PASSWORD
```
#### Apply Postgres migrations
Download CLI tool [sqitch](https://sqitch.org/download/)
Download the [Cloud SQL Auth Proxy](https://cloud.google.com/sql/docs/mysql/connect-admin-proxy#install)
Run the `cloud_sql_proxy` binary to connect to the Cloud SQL instance as a local connection:
```sh
# Notice the PORT is 5433 to differ from the default 5432 of Postgres
$ ./cloud_sql_proxy -instances=${PROJECT_ID}:us-central1:isolate-cloud=tcp:0.0.0.0:5433
```
In another shell, run the migrations
```sh
# xargs to remove extra quotes
$ export SQITCH_PASSWORD=$(terraform output isolate_cloud_sql_user_server_password | xargs)
$ cd fal-isolate-cloud/db
$ sqitch deploy --verify cloud-proxy
# Adding registry tables to cloud-proxy
# Deploying changes to cloud-proxy
# + create_table_jobs .. ok
```
And close (`CTRL-C`) the close_sql_proxy process.
Raw data
{
"_id": null,
"home_page": "",
"name": "isolate-controller",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.7,<4",
"maintainer_email": "",
"keywords": "",
"author": "Features & Labels",
"author_email": "hello@fal.ai",
"download_url": "https://files.pythonhosted.org/packages/42/00/a70a79c2c6cf80fd7ea4940ffccc23868ac4d5ffa39c8d5ce4cfb674ae56/isolate_controller-0.0.1a0.tar.gz",
"platform": null,
"description": "# fal-isolate-cloud\n\n## Regenerating gRPC definitions\n\n- Ensure you are on Python 3.9 or higher\n- From the root directory execute the following commands:\n\n```sh\n$ pip install grpcio_tools mypy-protobuf refactor\n$ python tools/regen_grpc.py src/isolate_controller/definitions/controller.proto\n```\n\n## Building the images\n\n- From the root directory\n\n```sh\n$ docker build -f infra/worker/Dockerfile -t isolate-cloud-worker .\n$ docker build -f infra/controller/Dockerfile -t isolate-cloud-controller .\n```\n\n## Testing the server (locally)\n\n- Assuming you built the image `isolate-cloud-worker`, try the following setup:\n\n```sh\n$ docker run --rm -e CONTROLLER_KEY=local_secret -p 50001:50001 isolate-cloud-worker\n$ ISOLATE_TEST_MODE=1 python -m isolate_controller.server\n# CHANGE THE IP FIRST\n$ python test_script.py\n```\n\n## Setting kubectl with the GCP cluster\n\n- Install the GKE plugin if you don't have it\n- Get the clusters list\n\n```sh\n$ gcloud components install gke-gcloud-auth-plugin\n$ gcloud container clusters get-credentials <cluster-name> --zone <your-zone>\n$ kubectl get pods -A\n# should print kube-system pods\n```\n\n## Deploying\n\n### Deploying the worker\n\n```sh\n$ gcloud builds submit --config=infra/worker/cloudbuild.yml .\n```\n\n### Deploying the controller\n\n```sh\n$ export PROJECT_ID=$(gcloud config list --format 'value(core.project)' 2>/dev/null)\n$ gcloud builds submit --config=infra/controller/cloudbuild.yml .\n$ export IMAGE_SHA=$(python tools/update_images.py)\n$ envsubst < infra/controller/controller.yml | kubectl apply -f -\n# Run this just once to expose the service:\n$ kubectl expose deployment fal-isolate-controller --type=LoadBalancer --name=load-balancer\n```\n\n### Connect redis\n\nMake sure to do this when you are setting up the first time\n\nhttps://cloud.google.com/memorystore/docs/redis/connect-redis-instance#connecting-cluster\n\n```sh\n$ gcloud redis instances list --region=us-central1\n# Take note of the HOST value\n$ export REDISHOST_IP=...\n```\n\n```sh\n$ export REDISHOST_IP=...\n$ git clone https://github.com/bowei/k8s-custom-iptables.git\n$ cd k8s-custom-iptables/\n$ TARGETS=${REDISHOST_IP} ./install.sh\n```\n\n```sh\n$ export REDISHOST_IP=...\n$ kubectl create configmap redishost --from-literal=REDISHOST=${REDISHOST_IP}\n```\n\n### Setup NFS cache\n\nMake sure to do this when you are setting up the first time\n\n```sh\n$ gcloud filestore instances list\n# Take note of the IP_ADDRESS\n$ export NFS_SERVER_IP=...\n```\n\n```sh\n$ export NFS_SERVER_IP=...\n$ envsubst < infra/fileserver/nfs-server.yml | kubectl apply -f -\n$ kubectl apply -f infra/fileserver/claim.yml\n```\n\n### Bind Google Service Account to Kubernetes Service Account\n\nMake sure to do this when you are setting up the first time\n\nThis is necessary to [connect Postgres](#connect-postgres)\n\n```sh\n$ kubectl annotate serviceaccount default iam.gke.io/gcp-service-account=gke-service-account@${PROJECT_ID}.iam.gserviceaccount.com\n```\n\n### Connect Postgres\n\nMake sure to do this when you are setting up the first time\n\n```sh\n# xargs to remove extra quotes\n$ cd fal_terraform/environments/<environment>\n$ export SQITCH_PASSWORD=$(terraform output isolate_cloud_sql_user_server_password | xargs)\n$ kubectl create secret generic sqlaccess \\\n --from-literal=SQL_DBNAME=users \\\n --from-literal=SQL_USER=gke_service \\\n --from-literal=SQL_PASSWORD=$SQITCH_PASSWORD\n```\n\n#### Apply Postgres migrations\n\nDownload CLI tool [sqitch](https://sqitch.org/download/)\n\nDownload the [Cloud SQL Auth Proxy](https://cloud.google.com/sql/docs/mysql/connect-admin-proxy#install)\n\nRun the `cloud_sql_proxy` binary to connect to the Cloud SQL instance as a local connection:\n\n```sh\n# Notice the PORT is 5433 to differ from the default 5432 of Postgres\n$ ./cloud_sql_proxy -instances=${PROJECT_ID}:us-central1:isolate-cloud=tcp:0.0.0.0:5433\n```\n\nIn another shell, run the migrations\n\n```sh\n# xargs to remove extra quotes\n$ export SQITCH_PASSWORD=$(terraform output isolate_cloud_sql_user_server_password | xargs)\n$ cd fal-isolate-cloud/db\n$ sqitch deploy --verify cloud-proxy\n# Adding registry tables to cloud-proxy\n# Deploying changes to cloud-proxy\n# + create_table_jobs .. ok\n```\n\nAnd close (`CTRL-C`) the close_sql_proxy process.\n",
"bugtrack_url": null,
"license": "",
"summary": "Managed isolated environments for Python",
"version": "0.0.1a0",
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"md5": "e44fb491032fe8e193e1a04c8b4fbef4",
"sha256": "1c70af96dac01ccc3c9b7b8ccbcd03a2dab6e9b90ed45794f5824f36ba2f734b"
},
"downloads": -1,
"filename": "isolate_controller-0.0.1a0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "e44fb491032fe8e193e1a04c8b4fbef4",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7,<4",
"size": 25517,
"upload_time": "2022-12-06T02:29:35",
"upload_time_iso_8601": "2022-12-06T02:29:35.936997Z",
"url": "https://files.pythonhosted.org/packages/08/aa/3f1682eefcc68cb591733b05a66a5164872c9e76035c79b9f6ab60d7bd33/isolate_controller-0.0.1a0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"md5": "5bf5bfb1ca632f72f0acc6caa0cc0b99",
"sha256": "f3dc11675b1ee76deb8a8dc1cd93d5a62c1b62000149eed4b737140f42fcef83"
},
"downloads": -1,
"filename": "isolate_controller-0.0.1a0.tar.gz",
"has_sig": false,
"md5_digest": "5bf5bfb1ca632f72f0acc6caa0cc0b99",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7,<4",
"size": 21301,
"upload_time": "2022-12-06T02:29:39",
"upload_time_iso_8601": "2022-12-06T02:29:39.223065Z",
"url": "https://files.pythonhosted.org/packages/42/00/a70a79c2c6cf80fd7ea4940ffccc23868ac4d5ffa39c8d5ce4cfb674ae56/isolate_controller-0.0.1a0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2022-12-06 02:29:39",
"github": false,
"gitlab": false,
"bitbucket": false,
"lcname": "isolate-controller"
}
