# README for Dagster Integration with Perian
## Introduction
This Dagster integration allows you to easily dockerize your codebase and execute it on the Perian platform, our serverless GPU environment. By leveraging this integration, you can streamline your workflow, enabling efficient code deployment and execution without the need for managing infrastructure.
### Features
- **Dockerization**: Automatically packages your codebase into a Docker container.
- **Serverless Execution**: Runs your Docker containers on Perian’s scalable GPU infrastructure.
- **Ease of Use**: Simple commands to get your project up and running quickly.
## Prerequisites
- Docker installed on your local machine.
- Access to the Perian platform (ensure your account is set up).
- Dagster installed in your project (follow the [Dagster installation guide](https://docs.dagster.io/getting-started/install-dagster)).
## Ops
This integration introduces the following Dagster ops:
1. **`containerize_codebase`**:
- Packages your codebase into a Docker container, including the installation of any necessary dependencies.
- **Parameters**:
- `path`: Directory of the codebase.
- `dependency_file`: Path to a requirements file (optional).
- `python_version`: Version of Python to use in the Docker image.
- `image_name`: Name for the Docker image.
- `tag`: Tag for the Docker image (default is "latest").
- `docker_username`, `docker_password`, `docker_repository`: Credentials for Docker Hub.
- `command`: Command to run in the container.
- `parameters`: Parameters for the command.
2. **`create_perian_job`**:
- Creates a job on the Perian platform using the Docker image built from your codebase.
- **Parameters**:
- `image_name`: Name of the Docker image.
- `tag`: Tag of the Docker image.
- `docker_username`, `docker_password`, `docker_repository`: Credentials for Docker Hub.
- `cores`, `memory`, `accelerators`, `accelerator_type`, `country_code`, `provider`: Configuration for the job instance.
3. **`get_perian_job`**:
- Retrieves the status and logs of a job running on the Perian platform.
- **Input**:
- `job_metadata`: Metadata containing the job ID.
4. **`cancel_perian_job`**:
- Cancels a running job on the Perian platform.
- **Input**:
- `job_metadata`: Metadata containing the job ID.
## Installation
1. Install dagster-perian package:
```bash
pip install dagster-perian
```
or
```bash
poetry add dagster-perian
```
## Example
Here’s an example of how to define a Dagster job that utilizes the containerize_codebase and create_perian_job ops:
```python
from dagster import job
from perian_job.resources import perian_job_resource
from perian_job.operations import create_perian_job, containerize_codebase
@job(resource_defs={"perian_job_manager": perian_job_resource})
def perian_job_flow():
containerize_codebase()
create_perian_job()
```yaml
resources:
perian_job_manager:
config:
api_url: "https://api.perian.cloud"
organization: "<your-perian-organization-name>"
token: "<your-perian-token>"
ops:
create_perian_job:
config:
accelerators: 1
accelerator_type:"A100"
image_name: "<your-image-name>"
tag: "<your-image-tag>"
docker_username: "<your-docker-username>"
docker_password: "<your-docker-password>"
containerize_codebase:
config:
path: "<your-code-base-to-be-dockerized>"
image_name: "<your-image-name>"
tag: "<your-image-tag>"
python_version: "<python-version-to-be-used>"
docker_username: "<your-docker-username>"
docker_password: "<your-docker-password>"
command: "<running-script-or-module>"
parameters: "<command-parameters>"
Raw data
{
"_id": null,
"home_page": "https://perian.io",
"name": "dagster-perian",
"maintainer": null,
"docs_url": null,
"requires_python": "<3.13,>=3.10",
"maintainer_email": null,
"keywords": "perian, sky, sky-computing, sky-platform, cli",
"author": "Perian",
"author_email": "info@perian.io",
"download_url": "https://files.pythonhosted.org/packages/92/77/b37b93ef345513f08640452990e018511d315380c0dc87dff9efed4bc36b/dagster_perian-0.1.1.tar.gz",
"platform": null,
"description": "# README for Dagster Integration with Perian\n\n## Introduction\n\nThis Dagster integration allows you to easily dockerize your codebase and execute it on the Perian platform, our serverless GPU environment. By leveraging this integration, you can streamline your workflow, enabling efficient code deployment and execution without the need for managing infrastructure.\n\n### Features\n\n- **Dockerization**: Automatically packages your codebase into a Docker container.\n- **Serverless Execution**: Runs your Docker containers on Perian\u2019s scalable GPU infrastructure.\n- **Ease of Use**: Simple commands to get your project up and running quickly.\n\n## Prerequisites\n\n- Docker installed on your local machine.\n- Access to the Perian platform (ensure your account is set up).\n- Dagster installed in your project (follow the [Dagster installation guide](https://docs.dagster.io/getting-started/install-dagster)).\n\n\n## Ops\n\nThis integration introduces the following Dagster ops:\n\n1. **`containerize_codebase`**: \n - Packages your codebase into a Docker container, including the installation of any necessary dependencies.\n - **Parameters**:\n - `path`: Directory of the codebase.\n - `dependency_file`: Path to a requirements file (optional).\n - `python_version`: Version of Python to use in the Docker image.\n - `image_name`: Name for the Docker image.\n - `tag`: Tag for the Docker image (default is \"latest\").\n - `docker_username`, `docker_password`, `docker_repository`: Credentials for Docker Hub.\n - `command`: Command to run in the container.\n - `parameters`: Parameters for the command.\n\n2. **`create_perian_job`**: \n - Creates a job on the Perian platform using the Docker image built from your codebase.\n - **Parameters**:\n - `image_name`: Name of the Docker image.\n - `tag`: Tag of the Docker image.\n - `docker_username`, `docker_password`, `docker_repository`: Credentials for Docker Hub.\n - `cores`, `memory`, `accelerators`, `accelerator_type`, `country_code`, `provider`: Configuration for the job instance.\n\n3. **`get_perian_job`**: \n - Retrieves the status and logs of a job running on the Perian platform.\n - **Input**: \n - `job_metadata`: Metadata containing the job ID.\n\n4. **`cancel_perian_job`**: \n - Cancels a running job on the Perian platform.\n - **Input**: \n - `job_metadata`: Metadata containing the job ID.\n\n## Installation\n\n1. Install dagster-perian package:\n\n ```bash\n pip install dagster-perian\n ```\n or\n ```bash\n poetry add dagster-perian\n ```\n\n\n## Example\n\nHere\u2019s an example of how to define a Dagster job that utilizes the containerize_codebase and create_perian_job ops:\n\n\n```python\nfrom dagster import job\nfrom perian_job.resources import perian_job_resource\nfrom perian_job.operations import create_perian_job, containerize_codebase\n\n@job(resource_defs={\"perian_job_manager\": perian_job_resource})\ndef perian_job_flow():\n containerize_codebase()\n create_perian_job()\n\n```yaml\nresources:\n perian_job_manager:\n config:\n api_url: \"https://api.perian.cloud\"\n organization: \"<your-perian-organization-name>\"\n token: \"<your-perian-token>\"\n\nops:\n create_perian_job:\n config:\n accelerators: 1\n accelerator_type:\"A100\"\n image_name: \"<your-image-name>\"\n tag: \"<your-image-tag>\"\n docker_username: \"<your-docker-username>\"\n docker_password: \"<your-docker-password>\"\n\n containerize_codebase:\n config:\n path: \"<your-code-base-to-be-dockerized>\"\n image_name: \"<your-image-name>\"\n tag: \"<your-image-tag>\"\n python_version: \"<python-version-to-be-used>\"\n docker_username: \"<your-docker-username>\"\n docker_password: \"<your-docker-password>\"\n command: \"<running-script-or-module>\"\n parameters: \"<command-parameters>\"\n\n\n\n",
"bugtrack_url": null,
"license": null,
"summary": "A dagster plugin for the Perian Sky Platform",
"version": "0.1.1",
"project_urls": {
"Documentation": "https://perian.io",
"Homepage": "https://perian.io"
},
"split_keywords": [
"perian",
" sky",
" sky-computing",
" sky-platform",
" cli"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "3b73d0df40101eb47e4af2f4abc86be7b69cb895ab5fbf515ac0d7c46372c11f",
"md5": "c9890422e217c860efc8d318c32d09e5",
"sha256": "2dbb7ae8e13fc1fc204463b696941e5dd5760e8707739f5436ee0c30120e48f3"
},
"downloads": -1,
"filename": "dagster_perian-0.1.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "c9890422e217c860efc8d318c32d09e5",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<3.13,>=3.10",
"size": 5719,
"upload_time": "2024-10-21T11:51:50",
"upload_time_iso_8601": "2024-10-21T11:51:50.080461Z",
"url": "https://files.pythonhosted.org/packages/3b/73/d0df40101eb47e4af2f4abc86be7b69cb895ab5fbf515ac0d7c46372c11f/dagster_perian-0.1.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "9277b37b93ef345513f08640452990e018511d315380c0dc87dff9efed4bc36b",
"md5": "8129c9157e0d443fd21b1125929b15e2",
"sha256": "60dc28a77d2960a7990d045933684dd3af7a9461248c3220a6e429c7b52ed029"
},
"downloads": -1,
"filename": "dagster_perian-0.1.1.tar.gz",
"has_sig": false,
"md5_digest": "8129c9157e0d443fd21b1125929b15e2",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<3.13,>=3.10",
"size": 4894,
"upload_time": "2024-10-21T11:51:51",
"upload_time_iso_8601": "2024-10-21T11:51:51.354372Z",
"url": "https://files.pythonhosted.org/packages/92/77/b37b93ef345513f08640452990e018511d315380c0dc87dff9efed4bc36b/dagster_perian-0.1.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-10-21 11:51:51",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "dagster-perian"
}
JSON