# raypack
Raypack will create a package for AWS Glue, Ray.io tasks. This automates
this [documentation page](https://docs.aws.amazon.com/glue/latest/dg/edit-script-ray-env-dependencies.html) that has
some handwavy descriptions of shell commands.
If you are only using included-by-default packages, public packages, pure python packages,
binary wheel packages, you don't have to do this.
AWS Glue can't handle anything without a binary wheel or private package repositories, gcc or other build tools are not
in Glue runtime images.
So you have build on a machine that matches the AWS runtime OS (Fedora-like), create a virtual directory,
and then zip it up and upload it to s3.
Some code generate with ChatGPT (OpenAI)
raypack is not supported by Amazon, AWS, nor Anyscale, Inc the makers of ray.io.
 [](https://pepy.tech/project/raypack/month)
## Installation
You are encouraged to install with pipx so that the CLI tools dependencies do not conflict with your project
dependencies.
```shell
pipx install raypack
```
## Capabilities
- TODO: Warn if not python 3.9 or other glue compatible version
- Calls poetry to create a virtualenv without dev dependencies
- TODO: support pip, pipenv to create virtualenv.
- Finds site-packages
- Zips virtualenv and zips own package
- TODO: support single file modules, eg. mymodule.py
- Skips cruft
- Run as few subprocesses as possible
- config using pyproject.toml or CLI args
- TODO: Uploads to s3
- pipx installable
- works on any OS as well as is possible (can't handle linux binaries on windows for example)
- Remove packages AWS includes
- [AWS's documentation on packaging ray jobs](https://docs.aws.amazon.com/glue/latest/dg/edit-script-ray-env-dependencies.html)
- [ray's documentation on dependencies](https://docs.ray.io/en/latest/ray-core/handling-dependencies.html.
- [AWS's documentation on packaging spark jobs](https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-python-libraries.html)
## Usage
```bash
raypack
```
```bash
python -m raypack
```
Configuration. If none specified, defaults are as below.
```toml
[tool.raypack]
exclude_packaging_cruft = true
outer_folder_name = "venv"
source_venv = ".venv"
venv_tool = "poetry"
```
## How it works
1. Gather info from pyproject.toml or CLI args, but not both.
2. Create a local .venv and .whl using poetry.
3. Create a new zip file with an extra top level folder.
4. Find the site-packages folder and copy to a new zip
5. Find the module contents in .whl and copy to a new zip
6. Upload to s3
7. Use s3 py modules `"--s3-py-modules", "s3://s3bucket/pythonPackage.zip"`
## Contributing
To install and run tests and linting tools.
```bash
poetry install --with dev
make check
```
To see if the app can package up other apps
```bash
poetry build
# exist poetry shell so that pipx can install with the right base python
exit
pipx install /e/github/raypack/dist/raypack-0.1.0-py3-none-any.whl
```
And then in a different project with a `pyproject.toml` file, run
```bash
raypack
```
## Prior Art
[Random scripts in comments](https://github.com/python-poetry/poetry/issues/1937#issuecomment-983754739)
[Some make file script](https://github.com/bhavintandel/py-packager)
Similar to PEX or other venv zip tools, which as far as I know are not AWS aware, or they don't include all the
dependencies, or they are more interested in making the archive file executable or self-extracting.
AWS Lambdas also have to go through a similar ad hoc zip process.
## Documentation
- [TODO](https://github.com/matthewdeanmartin/raypack/blob/main/docs/TODO.md)
## Change Log
- 0.1.0 - Idea and reserve package name.
Raw data
{
"_id": null,
"home_page": "https://github.com/matthewdeanmartin/raypack",
"name": "raypack",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.9,<4.0",
"maintainer_email": "",
"keywords": "awsglue,ray.io,aws glue",
"author": "Matthew Martin",
"author_email": "matthewdeanmartin@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/c7/97/f83a3d40ebd55001b11ffafbb1d9f2a04f5084020370c3420c78470d1326/raypack-0.3.20.tar.gz",
"platform": null,
"description": "# raypack\n\nRaypack will create a package for AWS Glue, Ray.io tasks. This automates\nthis [documentation page](https://docs.aws.amazon.com/glue/latest/dg/edit-script-ray-env-dependencies.html) that has\nsome handwavy descriptions of shell commands.\n\nIf you are only using included-by-default packages, public packages, pure python packages,\nbinary wheel packages, you don't have to do this.\n\nAWS Glue can't handle anything without a binary wheel or private package repositories, gcc or other build tools are not\nin Glue runtime images.\n\nSo you have build on a machine that matches the AWS runtime OS (Fedora-like), create a virtual directory,\nand then zip it up and upload it to s3.\n\nSome code generate with ChatGPT (OpenAI)\n\nraypack is not supported by Amazon, AWS, nor Anyscale, Inc the makers of ray.io.\n\n [](https://pepy.tech/project/raypack/month)\n\n## Installation\n\nYou are encouraged to install with pipx so that the CLI tools dependencies do not conflict with your project\ndependencies.\n\n```shell\npipx install raypack\n```\n\n## Capabilities\n\n- TODO: Warn if not python 3.9 or other glue compatible version\n- Calls poetry to create a virtualenv without dev dependencies\n- TODO: support pip, pipenv to create virtualenv.\n- Finds site-packages\n- Zips virtualenv and zips own package\n- TODO: support single file modules, eg. mymodule.py\n- Skips cruft\n- Run as few subprocesses as possible\n- config using pyproject.toml or CLI args\n- TODO: Uploads to s3\n- pipx installable\n- works on any OS as well as is possible (can't handle linux binaries on windows for example)\n- Remove packages AWS includes\n - [AWS's documentation on packaging ray jobs](https://docs.aws.amazon.com/glue/latest/dg/edit-script-ray-env-dependencies.html) \n - [ray's documentation on dependencies](https://docs.ray.io/en/latest/ray-core/handling-dependencies.html.\n - [AWS's documentation on packaging spark jobs](https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-python-libraries.html) \n \n## Usage\n\n```bash\nraypack\n```\n\n```bash\npython -m raypack\n```\n\nConfiguration. If none specified, defaults are as below.\n```toml\n[tool.raypack]\nexclude_packaging_cruft = true\nouter_folder_name = \"venv\"\nsource_venv = \".venv\"\nvenv_tool = \"poetry\"\n```\n\n## How it works\n\n1. Gather info from pyproject.toml or CLI args, but not both.\n2. Create a local .venv and .whl using poetry.\n3. Create a new zip file with an extra top level folder.\n4. Find the site-packages folder and copy to a new zip\n5. Find the module contents in .whl and copy to a new zip\n6. Upload to s3\n7. Use s3 py modules `\"--s3-py-modules\", \"s3://s3bucket/pythonPackage.zip\"`\n\n## Contributing\n\nTo install and run tests and linting tools.\n```bash\npoetry install --with dev\nmake check\n```\n\nTo see if the app can package up other apps\n```bash\npoetry build\n# exist poetry shell so that pipx can install with the right base python\nexit \npipx install /e/github/raypack/dist/raypack-0.1.0-py3-none-any.whl\n```\nAnd then in a different project with a `pyproject.toml` file, run\n\n```bash\nraypack\n```\n\n## Prior Art\n\n[Random scripts in comments](https://github.com/python-poetry/poetry/issues/1937#issuecomment-983754739)\n\n[Some make file script](https://github.com/bhavintandel/py-packager)\n\nSimilar to PEX or other venv zip tools, which as far as I know are not AWS aware, or they don't include all the\ndependencies, or they are more interested in making the archive file executable or self-extracting.\n\nAWS Lambdas also have to go through a similar ad hoc zip process.\n\n## Documentation\n\n- [TODO](https://github.com/matthewdeanmartin/raypack/blob/main/docs/TODO.md)\n\n## Change Log\n\n- 0.1.0 - Idea and reserve package name.\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Unofficial AWS Glue Ray.io packaging tool. Pure python or cross platform.",
"version": "0.3.20",
"project_urls": {
"Bug Tracker": "https://github.com/matthewdeanmartin/raypack/issues",
"Change Log": "https://github.com/matthewdeanmartin/raypack/blob/main/CHANGES.md",
"Documentation": "https://github.com/matthewdeanmartin/raypack",
"Homepage": "https://github.com/matthewdeanmartin/raypack",
"Repository": "https://github.com/matthewdeanmartin/raypack"
},
"split_keywords": [
"awsglue",
"ray.io",
"aws glue"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "c8c45eb2dce0277683e6a713b4d30a7a9ac95e7a6d917545b3cd7fc47f39726d",
"md5": "3f808fa398add8c3b3be1a375d3b0910",
"sha256": "3730cc7abf776022e2a2d6eff5fe90db7fa8002e278ef4d8350a489984ed2af3"
},
"downloads": -1,
"filename": "raypack-0.3.20-py3-none-any.whl",
"has_sig": false,
"md5_digest": "3f808fa398add8c3b3be1a375d3b0910",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9,<4.0",
"size": 13469,
"upload_time": "2023-11-02T02:30:43",
"upload_time_iso_8601": "2023-11-02T02:30:43.252989Z",
"url": "https://files.pythonhosted.org/packages/c8/c4/5eb2dce0277683e6a713b4d30a7a9ac95e7a6d917545b3cd7fc47f39726d/raypack-0.3.20-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "c797f83a3d40ebd55001b11ffafbb1d9f2a04f5084020370c3420c78470d1326",
"md5": "0c9babee4c534dbfda95f9e5515ca309",
"sha256": "37049ab8365c121137d761b587e3ec3d40cee99f102e913df0782c17ed4cd40b"
},
"downloads": -1,
"filename": "raypack-0.3.20.tar.gz",
"has_sig": false,
"md5_digest": "0c9babee4c534dbfda95f9e5515ca309",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9,<4.0",
"size": 12801,
"upload_time": "2023-11-02T02:30:44",
"upload_time_iso_8601": "2023-11-02T02:30:44.835809Z",
"url": "https://files.pythonhosted.org/packages/c7/97/f83a3d40ebd55001b11ffafbb1d9f2a04f5084020370c3420c78470d1326/raypack-0.3.20.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-11-02 02:30:44",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "matthewdeanmartin",
"github_project": "raypack",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"tox": true,
"lcname": "raypack"
}
JSON
