# Spark On Kubernetes
Spark on Kubernetes is a python package that makes it easy to submit and manage spark apps on Kubernetes.
It provides a Python client that can be used to submit apps in your API or scheduler of choice, and a CLI
that can be used to submit apps from the command line, instead of using spark-submit.
It also provides an optional REST API with a web UI that can be used to list and manage apps, and access the
spark UI through the reverse proxy.
## Installation
To install the core python package (only the Python client and the helpers), run:
```bash
pip install spark-on-k8s
```
If you want to use the REST API and the web UI, you will also need to install the api package:
```bash
pip install spark-on-k8s[api]
```
You can also install the package from source with pip or poetry:
```bash
# With pip
pip install . # For the core package
pip install ".[api]" # For the API package
# With poetry
poetry install # For the core package
poetry install -E api # For the API package
```
## Usage
### Setup the Kubernetes namespace
When submitting a Spark application to Kubernetes, we only create the driver pod, which is responsible for creating and
managing the executors pods. To give the driver pod the permissions to create the executors pods, we can give it a
service account with the required permissions. To simplify this process, we provide a helper function that creates
a namespace if needed, and a service account with the required permissions:
With Python:
```python
from spark_on_k8s.utils.setup_namespace import SparkOnK8SNamespaceSetup
spark_on_k8s_setuper = SparkOnK8SNamespaceSetup()
spark_on_k8s_setuper.setup_namespace(namespace="<namespace name>")
```
With the CLI:
```bash
spark-on-k8s namespace setup -n <namespace name>
```
### Python Client
The Python client can be used to submit apps from your Python code, instead of using spark-submit:
```python
from spark_on_k8s.client import SparkOnK8S
client = SparkOnK8S()
client.submit_app(
image="my-registry/my-image:latest",
app_path="local:///opt/spark/work-dir/my-app.py",
app_arguments=["arg1", "arg2"],
app_name="my-app",
namespace="spark-namespace",
service_account="spark-service-account",
app_waiter="log",
image_pull_policy="Never",
ui_reverse_proxy=True,
)
```
### CLI
The CLI can be used to submit apps from the command line, instead of using spark-submit, it can also be used to
manage apps submitted with the Python client (list, get, delete, logs, etc.):
Submit a app:
```bash
spark-on-k8s app submit \
--image my-registry/my-image:latest \
--path local:///opt/spark/work-dir/my-app.py \
-n spark \
--name my-app \
--image-pull-policy Never \
--ui-reverse-proxy \
--log \
param1 param2
```
Kill a app:
```bash
spark-on-k8s app kill -n spark-namespace --app-id my-app
```
List apps:
```bash
spark-on-k8s apps list -n spark-namespace
```
You can check the help for more information:
```bash
spark-on-k8s --help
spark-on-k8s app --help
spark-on-k8s apps --help
```
### REST API
The REST API implements some of the same functionality as the CLI but in async way, and also provides a web UI
that can be used to list the apps in the cluster and access the spark UI through a reverse proxy. The UI will be
improved in the future and more functionality will be added to both UI and API.
To run the API, you can use the CLI:
```bash
spark-on-k8s api start \
--host "0.0.0.0" \
--port 8080 \
--workers 4 \
--log-level error \
--limit-concurrency 100
```
To list the apps, you can use the API:
```bash
curl -X 'GET' \
'http://0.0.0.0:8080/apps/list_apps/spark-namespace' \
-H 'accept: application/json'
```
To access the spark UI of the app APP_ID, in the namespace NAMESPACE, you can use the web UI link:
`http://0.0.0.0:8080/webserver/ui/NAMESPACE/APP_ID`, or getting all the application and then clicking
on the button `Open Spark UI` from the link `http://0.0.0.0:8080/webserver/apps?namespace=NAMESPACE`.
### API in production
To deploy the API in production, you can use the project helm chart, that setups all the required resources in the
cluster, including the API deployment, the service, the ingress and the RBAC resources. The API has a configuration
class that loads the configuration from environment variables, so you can use the helm chart `env` values to configure
the API and its Kubernetes client.
To install the helm chart, you can run:
```bash
helm repo add spark-on-k8s http://hussein.awala.fr/spark-on-k8s-chart
helm repo update
helm install spark-on-k8s-release spark-on-k8s/spark-on-k8s --values examples/helm/values.yaml
```
## Configuration
The Python client and the CLI can be configured with environment variables to avoid passing the same arguments every
time if you have a common configuration for all your apps. The environment variables are the same for both the client
and the CLI. Here is a list of the available environment variables:
| Environment Variable | Description | Default |
|-------------------------------------------|-------------------------------------------------------|--------------|
| SPARK_ON_K8S_DOCKER_IMAGE | The docker image to use for the spark pods | |
| SPARK_ON_K8S_APP_PATH | The path to the app file | |
| SPARK_ON_K8S_NAMESPACE | The namespace to use | default |
| SPARK_ON_K8S_SERVICE_ACCOUNT | The service account to use | spark |
| SPARK_ON_K8S_SPARK_CONF | The spark configuration to use | {} |
| SPARK_ON_K8S_CLASS_NAME | The class name to use | |
| SPARK_ON_K8S_APP_ARGUMENTS | The arguments to pass to the app | [] |
| SPARK_ON_K8S_APP_WAITER | The waiter to use to wait for the app to finish | no_wait |
| SPARK_ON_K8S_IMAGE_PULL_POLICY | The image pull policy to use | IfNotPresent |
| SPARK_ON_K8S_UI_REVERSE_PROXY | Whether to use a reverse proxy to access the spark UI | false |
| SPARK_ON_K8S_DRIVER_CPU | The driver CPU | 1 |
| SPARK_ON_K8S_DRIVER_MEMORY | The driver memory | 1024 |
| SPARK_ON_K8S_DRIVER_MEMORY_OVERHEAD | The driver memory overhead | 512 |
| SPARK_ON_K8S_EXECUTOR_CPU | The executor CPU | 1 |
| SPARK_ON_K8S_EXECUTOR_MEMORY | The executor memory | 1024 |
| SPARK_ON_K8S_EXECUTOR_MEMORY_OVERHEAD | The executor memory overhead | 512 |
| SPARK_ON_K8S_EXECUTOR_MIN_INSTANCES | The minimum number of executor instances | |
| SPARK_ON_K8S_EXECUTOR_MAX_INSTANCES | The maximum number of executor instances | |
| SPARK_ON_K8S_EXECUTOR_INITIAL_INSTANCES | The initial number of executor instances | |
| SPARK_ON_K8S_CONFIG_FILE | The path to the config file | |
| SPARK_ON_K8S_CONTEXT | The context to use | |
| SPARK_ON_K8S_CLIENT_CONFIG | The sync Kubernetes client configuration to use | |
| SPARK_ON_K8S_ASYNC_CLIENT_CONFIG | The async Kubernetes client configuration to use | |
| SPARK_ON_K8S_IN_CLUSTER | Whether to use the in cluster Kubernetes config | false |
| SPARK_ON_K8S_API_DEFAULT_NAMESPACE | The default namespace to use for the API | default |
| SPARK_ON_K8S_API_HOST | The host to use for the API | 127.0.0.1 |
| SPARK_ON_K8S_API_PORT | The port to use for the API | 8000 |
| SPARK_ON_K8S_API_WORKERS | The number of workers to use for the API | 4 |
| SPARK_ON_K8S_API_LOG_LEVEL | The log level to use for the API | info |
| SPARK_ON_K8S_API_LIMIT_CONCURRENCY | The limit concurrency to use for the API | 1000 |
| SPARK_ON_K8S_API_SPARK_HISTORY_HOST | The host to use for the spark history server | |
| SPARK_ON_K8S_SPARK_DRIVER_NODE_SELECTOR | The node selector to use for the driver pod | {} |
| SPARK_ON_K8S_SPARK_EXECUTOR_NODE_SELECTOR | The node selector to use for the executor pods | {} |
| SPARK_ON_K8S_SPARK_DRIVER_LABELS | The labels to use for the driver pod | {} |
| SPARK_ON_K8S_SPARK_EXECUTOR_LABELS | The labels to use for the executor pods | {} |
| SPARK_ON_K8S_SPARK_DRIVER_ANNOTATIONS | The annotations to use for the driver pod | {} |
| SPARK_ON_K8S_SPARK_EXECUTOR_ANNOTATIONS | The annotations to use for the executor pods | {} |
| SPARK_ON_K8S_EXECUTOR_POD_TEMPLATE_PATH | The path to the executor pod template | |
## Examples
Here are some examples of how to package and submit spark apps with this package. In the examples, the base image is
built with the spark image tool, as described in the
[spark documentation](https://spark.apache.org/docs/latest/running-on-kubernetes.html#docker-images).
### Python
First, build the docker image and push it to a registry accessible by your cluster,
or load it into your cluster's local registry if you're using minikube or kind:
```bash
docker build -t pyspark-job examples/python
# For minikube
minikube image load pyspark-job
# For kind
kind load docker-image pyspark-job
# For remote clusters, you will need to change the image name to match your registry,
# and then push it to that registry
docker push pyspark-job
```
Then, submit the job:
```bash
python examples/python/submit.py
```
Or via the bash script:
```bash
./examples/python/submit.sh
```
### Java
Same as above, but with the java example:
```bash
docker build -t java-spark-job examples/java
# For minikube
minikube image load java-spark-job
# For kind
kind load docker-image java-spark-job
# For remote clusters, you will need to change the image name to match your registry,
# and then push it to that registry
docker push java-spark-job
```
Then, submit the job:
```bash
python examples/java/submit.py
```
Or via the bash script:
```bash
./examples/java/submit.sh
```
## What's next
You can check the [TODO](https://github.com/hussein-awala/spark-on-k8s/blob/main/TODO.md) list for the things that we
will work on in the future. All contributions are welcome!
Raw data
{
"_id": null,
"home_page": "https://github.com/hussein-awala/spark-on-k8s",
"name": "spark-on-k8s",
"maintainer": null,
"docs_url": null,
"requires_python": "<4.0,>=3.8",
"maintainer_email": null,
"keywords": "spark, kubernetes, k8s, spark-submit, spark-on-k8s",
"author": "Hussein Awala",
"author_email": "hussein@awala.fr",
"download_url": "https://files.pythonhosted.org/packages/57/7d/75432f74a10aa3f613d8051908d3341b3af32c9a0c3e2202a839dfab7928/spark_on_k8s-0.5.1.tar.gz",
"platform": null,
"description": "# Spark On Kubernetes\n\nSpark on Kubernetes is a python package that makes it easy to submit and manage spark apps on Kubernetes.\nIt provides a Python client that can be used to submit apps in your API or scheduler of choice, and a CLI\nthat can be used to submit apps from the command line, instead of using spark-submit.\n\nIt also provides an optional REST API with a web UI that can be used to list and manage apps, and access the\nspark UI through the reverse proxy.\n\n## Installation\nTo install the core python package (only the Python client and the helpers), run:\n```bash\npip install spark-on-k8s\n```\nIf you want to use the REST API and the web UI, you will also need to install the api package:\n```bash\npip install spark-on-k8s[api]\n```\n\nYou can also install the package from source with pip or poetry:\n```bash\n# With pip\npip install . # For the core package\npip install \".[api]\" # For the API package\n\n# With poetry\npoetry install # For the core package\npoetry install -E api # For the API package\n```\n\n## Usage\n\n### Setup the Kubernetes namespace\n\nWhen submitting a Spark application to Kubernetes, we only create the driver pod, which is responsible for creating and\nmanaging the executors pods. To give the driver pod the permissions to create the executors pods, we can give it a\nservice account with the required permissions. To simplify this process, we provide a helper function that creates\na namespace if needed, and a service account with the required permissions:\n\nWith Python:\n```python\nfrom spark_on_k8s.utils.setup_namespace import SparkOnK8SNamespaceSetup\n\nspark_on_k8s_setuper = SparkOnK8SNamespaceSetup()\nspark_on_k8s_setuper.setup_namespace(namespace=\"<namespace name>\")\n```\n\nWith the CLI:\n```bash\nspark-on-k8s namespace setup -n <namespace name>\n```\n\n### Python Client\nThe Python client can be used to submit apps from your Python code, instead of using spark-submit:\n\n```python\nfrom spark_on_k8s.client import SparkOnK8S\n\nclient = SparkOnK8S()\nclient.submit_app(\n image=\"my-registry/my-image:latest\",\n app_path=\"local:///opt/spark/work-dir/my-app.py\",\n app_arguments=[\"arg1\", \"arg2\"],\n app_name=\"my-app\",\n namespace=\"spark-namespace\",\n service_account=\"spark-service-account\",\n app_waiter=\"log\",\n image_pull_policy=\"Never\",\n ui_reverse_proxy=True,\n)\n```\n\n### CLI\nThe CLI can be used to submit apps from the command line, instead of using spark-submit, it can also be used to\nmanage apps submitted with the Python client (list, get, delete, logs, etc.):\n\nSubmit a app:\n```bash\nspark-on-k8s app submit \\\n --image my-registry/my-image:latest \\\n --path local:///opt/spark/work-dir/my-app.py \\\n -n spark \\\n --name my-app \\\n --image-pull-policy Never \\\n --ui-reverse-proxy \\\n --log \\\n param1 param2\n```\nKill a app:\n```bash\nspark-on-k8s app kill -n spark-namespace --app-id my-app\n```\n\nList apps:\n```bash\nspark-on-k8s apps list -n spark-namespace\n```\n\nYou can check the help for more information:\n```bash\nspark-on-k8s --help\nspark-on-k8s app --help\nspark-on-k8s apps --help\n```\n\n### REST API\n\nThe REST API implements some of the same functionality as the CLI but in async way, and also provides a web UI\nthat can be used to list the apps in the cluster and access the spark UI through a reverse proxy. The UI will be\nimproved in the future and more functionality will be added to both UI and API.\n\nTo run the API, you can use the CLI:\n```bash\nspark-on-k8s api start \\\n --host \"0.0.0.0\" \\\n --port 8080 \\\n --workers 4 \\\n --log-level error \\\n --limit-concurrency 100\n```\n\nTo list the apps, you can use the API:\n```bash\ncurl -X 'GET' \\\n 'http://0.0.0.0:8080/apps/list_apps/spark-namespace' \\\n -H 'accept: application/json'\n```\n\nTo access the spark UI of the app APP_ID, in the namespace NAMESPACE, you can use the web UI link:\n`http://0.0.0.0:8080/webserver/ui/NAMESPACE/APP_ID`, or getting all the application and then clicking\non the button `Open Spark UI` from the link `http://0.0.0.0:8080/webserver/apps?namespace=NAMESPACE`.\n\n### API in production\n\nTo deploy the API in production, you can use the project helm chart, that setups all the required resources in the\ncluster, including the API deployment, the service, the ingress and the RBAC resources. The API has a configuration\nclass that loads the configuration from environment variables, so you can use the helm chart `env` values to configure\nthe API and its Kubernetes client.\n\nTo install the helm chart, you can run:\n```bash\nhelm repo add spark-on-k8s http://hussein.awala.fr/spark-on-k8s-chart\nhelm repo update\nhelm install spark-on-k8s-release spark-on-k8s/spark-on-k8s --values examples/helm/values.yaml\n```\n\n## Configuration\n\nThe Python client and the CLI can be configured with environment variables to avoid passing the same arguments every\ntime if you have a common configuration for all your apps. The environment variables are the same for both the client\nand the CLI. Here is a list of the available environment variables:\n\n| Environment Variable | Description | Default |\n|-------------------------------------------|-------------------------------------------------------|--------------|\n| SPARK_ON_K8S_DOCKER_IMAGE | The docker image to use for the spark pods | |\n| SPARK_ON_K8S_APP_PATH | The path to the app file | |\n| SPARK_ON_K8S_NAMESPACE | The namespace to use | default |\n| SPARK_ON_K8S_SERVICE_ACCOUNT | The service account to use | spark |\n| SPARK_ON_K8S_SPARK_CONF | The spark configuration to use | {} |\n| SPARK_ON_K8S_CLASS_NAME | The class name to use | |\n| SPARK_ON_K8S_APP_ARGUMENTS | The arguments to pass to the app | [] |\n| SPARK_ON_K8S_APP_WAITER | The waiter to use to wait for the app to finish | no_wait |\n| SPARK_ON_K8S_IMAGE_PULL_POLICY | The image pull policy to use | IfNotPresent |\n| SPARK_ON_K8S_UI_REVERSE_PROXY | Whether to use a reverse proxy to access the spark UI | false |\n| SPARK_ON_K8S_DRIVER_CPU | The driver CPU | 1 |\n| SPARK_ON_K8S_DRIVER_MEMORY | The driver memory | 1024 |\n| SPARK_ON_K8S_DRIVER_MEMORY_OVERHEAD | The driver memory overhead | 512 |\n| SPARK_ON_K8S_EXECUTOR_CPU | The executor CPU | 1 |\n| SPARK_ON_K8S_EXECUTOR_MEMORY | The executor memory | 1024 |\n| SPARK_ON_K8S_EXECUTOR_MEMORY_OVERHEAD | The executor memory overhead | 512 |\n| SPARK_ON_K8S_EXECUTOR_MIN_INSTANCES | The minimum number of executor instances | |\n| SPARK_ON_K8S_EXECUTOR_MAX_INSTANCES | The maximum number of executor instances | |\n| SPARK_ON_K8S_EXECUTOR_INITIAL_INSTANCES | The initial number of executor instances | |\n| SPARK_ON_K8S_CONFIG_FILE | The path to the config file | |\n| SPARK_ON_K8S_CONTEXT | The context to use | |\n| SPARK_ON_K8S_CLIENT_CONFIG | The sync Kubernetes client configuration to use | |\n| SPARK_ON_K8S_ASYNC_CLIENT_CONFIG | The async Kubernetes client configuration to use | |\n| SPARK_ON_K8S_IN_CLUSTER | Whether to use the in cluster Kubernetes config | false |\n| SPARK_ON_K8S_API_DEFAULT_NAMESPACE | The default namespace to use for the API | default |\n| SPARK_ON_K8S_API_HOST | The host to use for the API | 127.0.0.1 |\n| SPARK_ON_K8S_API_PORT | The port to use for the API | 8000 |\n| SPARK_ON_K8S_API_WORKERS | The number of workers to use for the API | 4 |\n| SPARK_ON_K8S_API_LOG_LEVEL | The log level to use for the API | info |\n| SPARK_ON_K8S_API_LIMIT_CONCURRENCY | The limit concurrency to use for the API | 1000 |\n| SPARK_ON_K8S_API_SPARK_HISTORY_HOST | The host to use for the spark history server | |\n| SPARK_ON_K8S_SPARK_DRIVER_NODE_SELECTOR | The node selector to use for the driver pod | {} |\n| SPARK_ON_K8S_SPARK_EXECUTOR_NODE_SELECTOR | The node selector to use for the executor pods | {} |\n| SPARK_ON_K8S_SPARK_DRIVER_LABELS | The labels to use for the driver pod | {} |\n| SPARK_ON_K8S_SPARK_EXECUTOR_LABELS | The labels to use for the executor pods | {} |\n| SPARK_ON_K8S_SPARK_DRIVER_ANNOTATIONS | The annotations to use for the driver pod | {} |\n| SPARK_ON_K8S_SPARK_EXECUTOR_ANNOTATIONS | The annotations to use for the executor pods | {} |\n| SPARK_ON_K8S_EXECUTOR_POD_TEMPLATE_PATH | The path to the executor pod template | |\n\n\n## Examples\n\nHere are some examples of how to package and submit spark apps with this package. In the examples, the base image is\nbuilt with the spark image tool, as described in the\n[spark documentation](https://spark.apache.org/docs/latest/running-on-kubernetes.html#docker-images).\n\n### Python\nFirst, build the docker image and push it to a registry accessible by your cluster,\nor load it into your cluster's local registry if you're using minikube or kind:\n```bash\ndocker build -t pyspark-job examples/python\n\n# For minikube\nminikube image load pyspark-job\n# For kind\nkind load docker-image pyspark-job\n# For remote clusters, you will need to change the image name to match your registry,\n# and then push it to that registry\ndocker push pyspark-job\n```\nThen, submit the job:\n```bash\npython examples/python/submit.py\n```\nOr via the bash script:\n```bash\n./examples/python/submit.sh\n```\n\n### Java\nSame as above, but with the java example:\n```bash\ndocker build -t java-spark-job examples/java\n\n# For minikube\nminikube image load java-spark-job\n# For kind\nkind load docker-image java-spark-job\n# For remote clusters, you will need to change the image name to match your registry,\n# and then push it to that registry\ndocker push java-spark-job\n```\nThen, submit the job:\n```bash\npython examples/java/submit.py\n```\n\nOr via the bash script:\n```bash\n./examples/java/submit.sh\n```\n\n## What's next\n\nYou can check the [TODO](https://github.com/hussein-awala/spark-on-k8s/blob/main/TODO.md) list for the things that we\nwill work on in the future. All contributions are welcome!\n",
"bugtrack_url": null,
"license": "Apache-2.0",
"summary": "A Python package to submit and manage Apache Spark applications on Kubernetes.",
"version": "0.5.1",
"project_urls": {
"Bug Tracker": "https://github.com/hussein-awala/spark-on-k8s/issues",
"Documentation": "https://github.com/hussein-awala/spark-on-k8s/blob/main/README.md",
"Homepage": "https://github.com/hussein-awala/spark-on-k8s",
"Repository": "https://github.com/hussein-awala/spark-on-k8s"
},
"split_keywords": [
"spark",
" kubernetes",
" k8s",
" spark-submit",
" spark-on-k8s"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "e93cd4716d155bd285faa0853ea3eec3e097b5509e64420ce81f118edcee522a",
"md5": "c54da041bec503c584d42fa2d471eb4a",
"sha256": "866f189b592cf448baed4ebe1ba1fcf9bf4d29fb5475f498c48238f1e7004b8d"
},
"downloads": -1,
"filename": "spark_on_k8s-0.5.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "c54da041bec503c584d42fa2d471eb4a",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4.0,>=3.8",
"size": 45598,
"upload_time": "2024-04-30T22:26:22",
"upload_time_iso_8601": "2024-04-30T22:26:22.047984Z",
"url": "https://files.pythonhosted.org/packages/e9/3c/d4716d155bd285faa0853ea3eec3e097b5509e64420ce81f118edcee522a/spark_on_k8s-0.5.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "577d75432f74a10aa3f613d8051908d3341b3af32c9a0c3e2202a839dfab7928",
"md5": "1a59626f0836490168190420cf96fb6f",
"sha256": "adb8bb0d1f614811f2fac15e31a96db8b68f08956315ece7d6d03ac3bef4b8d7"
},
"downloads": -1,
"filename": "spark_on_k8s-0.5.1.tar.gz",
"has_sig": false,
"md5_digest": "1a59626f0836490168190420cf96fb6f",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4.0,>=3.8",
"size": 35489,
"upload_time": "2024-04-30T22:26:23",
"upload_time_iso_8601": "2024-04-30T22:26:23.367489Z",
"url": "https://files.pythonhosted.org/packages/57/7d/75432f74a10aa3f613d8051908d3341b3af32c9a0c3e2202a839dfab7928/spark_on_k8s-0.5.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-04-30 22:26:23",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "hussein-awala",
"github_project": "spark-on-k8s",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "spark-on-k8s"
}
JSON
