| Name | dagster-ray JSON |
| Version |
0.2.0
JSON |
| download |
| home_page | None |
| Summary | Dagster integration library for Ray |
| upload_time | 2025-09-18 11:39:15 |
| maintainer | None |
| docs_url | None |
| author | None |
| requires_python | <3.13,>=3.9 |
| license | Apache-2.0 |
| keywords |
etl
dagster
distributed
ray
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# `dagster-ray`
[](https://pypi.python.org/pypi/dagster-ray)
[](https://pypi.python.org/pypi/dagster-ray)
[](https://pypi.python.org/pypi/dagster-ray)
[](https://github.com/danielgafni/dagster-ray/actions/workflows/CI.yml)
[](https://github.com/pre-commit/pre-commit)
[](https://docs.basedpyright.com)
[](https://github.com/astral-sh/ruff)
---
**Ray integration for Dagster.**
`dagster-ray` enables you to orchestrate distributed Ray compute from Dagster pipelines, providing seamless integration between Dagster's orchestration capabilities and Ray's distributed computing power.
> [!NOTE]
> This project is ready for production use, but some APIs may change between minor releases.
Learn more in the [docs](https://danielgafni.github.io/dagster-ray)
## 🚀 Key Features
- **Run Launchers & Executors**: Submit Dagster runs or individual ops as Ray jobs
- **Ray Resources**: Manage Ray clusters with Kubernetes (KubeRay) or local backends, connect to them in client mode
- **Dagster Pipes**: Execute external Ray scripts with rich logging and metadata
- **Production Ready**: Tested against a matrix of core dependencies and platform versions, integrated with Dagster+
## 📦 Quick Start
### Installation
```shell
pip install dagster-ray
```
### Example
```python
import dagster as dg
from dagster_ray import LocalRay, RayResource, KubeRayInteractiveJob
import ray
@ray.remote
def compute_square(x: int) -> int:
return x**2
@dg.asset
def my_distributed_computation(ray_cluster: RayResource) -> int:
futures = [compute_square.remote(i) for i in range(10)]
return sum(ray.get(futures))
ray_cluster = LocalRay() if not IN_KUBERNETES else KubeRayInteractiveJob()
definitions = dg.Definitions(
assets=[my_distributed_computation],
resources={"ray_cluster": ray_cluster},
)
```
## 📚 Docs
**📖 [Full Documentation](https://danielgafni.github.io/dagster-ray)**
- **[Tutorial](https://danielgafni.github.io/dagster-ray/tutorial/)**: Step-by-step guide with examples
- **[API Reference](https://danielgafni.github.io/dagster-ray/api/)**: Complete API documentation
## 🛠️ Integration Options
| Component | Use Case | Cluster Management | Ray Mode |
|-----------|----------|-------------------|------|
| `RayRunLauncher` | Deployment-wide Ray runtime | External | Job Mode |
| `ray_executor` | Ray runtime scoped to a Code Location | External | Job Mode |
| `PipesRayJobClient` | Submit external scripts as Ray jobs | External | Job Mode |
| `PipesKubeRayJobClient` | Submit an external script as a `RayJob`, forward logs and Dagster metadata | Automatic | Job Mode |
| `KubeRayInteractiveJob` | Create a `RayJob`, connect in Client mode without an external script | Automatic | Client Mode |
## 🤝 Contributing
Contributions are very welcome! To get started:
```bash
git clone https://github.com/danielgafni/dagster-ray.git
cd dagster-ray
uv sync --all-extras
uv run pre-commit install
```
### 🧪 Testing
```bash
uv run pytest
```
Running KubeRay tests requires the following tools to be present:
- `docker`, `kubectl`, `helm`, `minikube`
### Documentation
To build and serve the documentation locally:
```bash
# Serve documentation locally
uv run --group docs mkdocs serve
# Build documentation
uv run--group docs mkdocs build
```
The documentation is automatically deployed to GitHub Pages.
Raw data
{
"_id": null,
"home_page": null,
"name": "dagster-ray",
"maintainer": null,
"docs_url": null,
"requires_python": "<3.13,>=3.9",
"maintainer_email": null,
"keywords": "ETL, dagster, distributed, ray",
"author": null,
"author_email": "Daniel Gafni <danielgafni16@gmail.com>",
"download_url": "https://files.pythonhosted.org/packages/93/50/8dc1e03498af3407d1d06eaafac15bf1cede038fab537c572dba53192175/dagster_ray-0.2.0.tar.gz",
"platform": null,
"description": "# `dagster-ray`\n\n[](https://pypi.python.org/pypi/dagster-ray)\n[](https://pypi.python.org/pypi/dagster-ray)\n[](https://pypi.python.org/pypi/dagster-ray)\n[](https://github.com/danielgafni/dagster-ray/actions/workflows/CI.yml)\n[](https://github.com/pre-commit/pre-commit)\n[](https://docs.basedpyright.com)\n[](https://github.com/astral-sh/ruff)\n\n---\n\n**Ray integration for Dagster.**\n\n`dagster-ray` enables you to orchestrate distributed Ray compute from Dagster pipelines, providing seamless integration between Dagster's orchestration capabilities and Ray's distributed computing power.\n\n> [!NOTE]\n> This project is ready for production use, but some APIs may change between minor releases.\n\nLearn more in the [docs](https://danielgafni.github.io/dagster-ray)\n\n## \ud83d\ude80 Key Features\n\n- **Run Launchers & Executors**: Submit Dagster runs or individual ops as Ray jobs\n- **Ray Resources**: Manage Ray clusters with Kubernetes (KubeRay) or local backends, connect to them in client mode\n- **Dagster Pipes**: Execute external Ray scripts with rich logging and metadata\n- **Production Ready**: Tested against a matrix of core dependencies and platform versions, integrated with Dagster+\n\n## \ud83d\udce6 Quick Start\n\n### Installation\n\n```shell\npip install dagster-ray\n```\n\n### Example\n\n```python\nimport dagster as dg\nfrom dagster_ray import LocalRay, RayResource, KubeRayInteractiveJob\nimport ray\n\n\n@ray.remote\ndef compute_square(x: int) -> int:\n return x**2\n\n\n@dg.asset\ndef my_distributed_computation(ray_cluster: RayResource) -> int:\n futures = [compute_square.remote(i) for i in range(10)]\n return sum(ray.get(futures))\n\n\nray_cluster = LocalRay() if not IN_KUBERNETES else KubeRayInteractiveJob()\n\n\ndefinitions = dg.Definitions(\n assets=[my_distributed_computation],\n resources={\"ray_cluster\": ray_cluster},\n)\n```\n\n## \ud83d\udcda Docs\n\n**\ud83d\udcd6 [Full Documentation](https://danielgafni.github.io/dagster-ray)**\n\n- **[Tutorial](https://danielgafni.github.io/dagster-ray/tutorial/)**: Step-by-step guide with examples\n- **[API Reference](https://danielgafni.github.io/dagster-ray/api/)**: Complete API documentation\n\n## \ud83d\udee0\ufe0f Integration Options\n\n| Component | Use Case | Cluster Management | Ray Mode |\n|-----------|----------|-------------------|------|\n| `RayRunLauncher` | Deployment-wide Ray runtime | External | Job Mode |\n| `ray_executor` | Ray runtime scoped to a Code Location | External | Job Mode |\n| `PipesRayJobClient` | Submit external scripts as Ray jobs | External | Job Mode |\n| `PipesKubeRayJobClient` | Submit an external script as a `RayJob`, forward logs and Dagster metadata | Automatic | Job Mode |\n| `KubeRayInteractiveJob` | Create a `RayJob`, connect in Client mode without an external script | Automatic | Client Mode |\n\n## \ud83e\udd1d Contributing\n\nContributions are very welcome! To get started:\n\n```bash\ngit clone https://github.com/danielgafni/dagster-ray.git\ncd dagster-ray\nuv sync --all-extras\nuv run pre-commit install\n```\n\n### \ud83e\uddea Testing\n\n```bash\nuv run pytest\n```\n\nRunning KubeRay tests requires the following tools to be present:\n- `docker`, `kubectl`, `helm`, `minikube`\n\n### Documentation\n\nTo build and serve the documentation locally:\n\n```bash\n# Serve documentation locally\nuv run --group docs mkdocs serve\n\n# Build documentation\nuv run--group docs mkdocs build\n```\n\nThe documentation is automatically deployed to GitHub Pages.\n",
"bugtrack_url": null,
"license": "Apache-2.0",
"summary": "Dagster integration library for Ray",
"version": "0.2.0",
"project_urls": {
"repository": "https://github.com/danielgafni/dagster-ray"
},
"split_keywords": [
"etl",
" dagster",
" distributed",
" ray"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "cb7053ea99522699ea629044fb936201538f8033dc9f77393b01ad25901e2c8a",
"md5": "dca7928f660532fe66c8d9337bf5ea62",
"sha256": "c88a1c35ef6689a8e429b34d1fe8d76ce551a024f1b347a31f17ed48fc1a9c03"
},
"downloads": -1,
"filename": "dagster_ray-0.2.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "dca7928f660532fe66c8d9337bf5ea62",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<3.13,>=3.9",
"size": 7011,
"upload_time": "2025-09-18T11:39:14",
"upload_time_iso_8601": "2025-09-18T11:39:14.369787Z",
"url": "https://files.pythonhosted.org/packages/cb/70/53ea99522699ea629044fb936201538f8033dc9f77393b01ad25901e2c8a/dagster_ray-0.2.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "93508dc1e03498af3407d1d06eaafac15bf1cede038fab537c572dba53192175",
"md5": "17b748a4707de5bb98aac44ab7ebe717",
"sha256": "343f13e844b945fa3c0284632433e38cbec5f8146e894fb47d7d637465dd4e6a"
},
"downloads": -1,
"filename": "dagster_ray-0.2.0.tar.gz",
"has_sig": false,
"md5_digest": "17b748a4707de5bb98aac44ab7ebe717",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<3.13,>=3.9",
"size": 39870,
"upload_time": "2025-09-18T11:39:15",
"upload_time_iso_8601": "2025-09-18T11:39:15.228703Z",
"url": "https://files.pythonhosted.org/packages/93/50/8dc1e03498af3407d1d06eaafac15bf1cede038fab537c572dba53192175/dagster_ray-0.2.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-09-18 11:39:15",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "danielgafni",
"github_project": "dagster-ray",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "dagster-ray"
}
