| Name | venvstacks JSON |
| Version |
0.2.1
JSON |
| download |
| home_page | None |
| Summary | Use layered Python virtual environment stacks to share large dependencies |
| upload_time | 2024-12-05 07:36:58 |
| maintainer | None |
| docs_url | None |
| author | None |
| requires_python | >=3.11 |
| license | MIT |
| keywords |
packaging
dependency
venv
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
Virtual Environment Stacks for Python
=====================================
Machine learning and AI libraries for Python are big. Really big. Nobody wants to download
and install multiple copies of [PyTorch](https://pypi.org/project/torch/) or
[CUDA](https://pypi.org/project/cuda-python/) if they can reasonably avoid it.
``venvstacks`` allows you to package Python applications and all their dependencies into a
portable, deterministic format, *without* needing to include copies of these large Python
frameworks in every application archive.
It achieves this by using Python's `sitecustomize.py` environment setup feature to
chain together three layers of Python virtual environments:
* "Runtime" layers: environment containing the desired version of a specific Python runtime
* "Framework" layers: environments containing desired versions of key Python frameworks
* "Application" layers: environments containing components to be launched directly
Application layer environments may include additional unpackaged Python launch modules or
packages for invocation with `python`'s `-m` switch.
While the layers are archived and published separately, their dependency locking is integrated,
allowing the application layers to share dependencies installed in the framework layers,
and the framework layers to share dependencies installed in the runtime layers.
Refer to the [Project Overview](https://venvstacks.lmstudio.ai/overview/) for an
example of specifying, locking, building, and publishing a set of environment stacks.
Installing
----------
`venvstacks` is available from the [Python Package Index](https://pypi.org/project/venvstacks/),
and can be installed with [pipx](https://pypi.org/project/pipx/):
$ pipx install venvstacks
Alternatively, it can be installed as a user level package (although this may
make future Python version upgrades more irritating):
$ pip install --user venvstacks
Interactions with other packaging tools
---------------------------------------
The base runtime environment layers are installed with `pdm` (with the installed runtimes coming
from the `python-build-standalone` project). `pdm` is also used to manage the development
of the `venvstacks` project itself.
The layered framework and app environments are created with the standard library's `venv` module.
The Python packages in each layer are currently being installed directly with `pip`, but
are expected to eventually move to being installed with `uv` to reduce environment
setup times during builds.
Platform-specific environment locking for each layer is performed using
`uv pip compile`. Refer to `pyproject.toml` for the specific issues preventing
the adoption of `uv` for additional purposes.
`venvstacks` expects precompiled `wheel` archives to be available for all included
Python distribution packages. When this is not the case, other projects like
[`wagon`](https://pypi.org/project/wagon/#files) or
[`fromager`](https://pypi.org/project/fromager/)
may be useful in generating the required input archives.
Caveats and Limitations
-----------------------
* This project does NOT support combining arbitrary virtual environments with each other.
Instead, it allows larger integrated applications to split up their Python dependencies into
distinct layers, without needing to download and install multiple copies of large
dependencies (such as the PyTorch ML/AI framework). The environment stack specification
and build process helps ensure that shared dependencies are kept consistent across layers,
while unshared dependencies are free to vary across the application components that need them.
* The `venvstacks` Python API is *not yet stable*. Any interface not specifically
declared as stable in the documentation may be renamed or relocated without a
deprecation period. API stabilisation (mostly splitting up the overly large
`venvstacks.stacks` namespace) will be the trigger for the 1.0 milestone release.
* While the `venvstacks` CLI is broadly stable, there are still some specific areas
where changes may occur (such as in the handling of relative paths).
* Dynamic library dependencies across layers currently only work on Windows.
There is a [proposal](https://github.com/lmstudio-ai/venvstacks/issues38) in
place for resolving that limitation, but it has not yet been implemented.
* Local exports to filesystems which do not support symlinks (such as `VFAT` and
`FAT32`) are nominally supported (with symlinks being replaced by the files
they refer to), but this support is *not* currently tested.
* To avoid relying on the Python ecosystem's still limited support for cross-platform
component installation, the stack build processes need to be executed on the target
platform (for example, by using an OS matrix in GitHub Actions).
Project History
---------------
The initial (and ongoing) development of the `venvstacks` project is being funded
by [LM Studio](https://lmstudio.ai/), where it serves as the foundation of
LM Studio's support for local execution of Python AI frameworks such as
[Apple's MLX](https://lmstudio.ai/blog/lmstudio-v0.3.4).
The use of "🐸" (frog) and "🦎" (newts are often mistaken for lizards and
vice-versa!) as the Unicode support test characters is a reference to the
internal LM Studio project that initially built and continues to maintain
`venvstacks`: Project Amphibian.
Raw data
{
"_id": null,
"home_page": null,
"name": "venvstacks",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.11",
"maintainer_email": "LM Studio <team@lmstudio.ai>",
"keywords": "packaging, dependency, venv",
"author": null,
"author_email": "Alyssa Coghlan <ncoghlan@gmail.com>",
"download_url": "https://files.pythonhosted.org/packages/a8/fb/f9aae83aa6581e29f88468a98857f29e0504683bac993f9db1a8dde8f685/venvstacks-0.2.1.tar.gz",
"platform": null,
"description": "Virtual Environment Stacks for Python\n=====================================\n\nMachine learning and AI libraries for Python are big. Really big. Nobody wants to download\nand install multiple copies of [PyTorch](https://pypi.org/project/torch/) or\n[CUDA](https://pypi.org/project/cuda-python/) if they can reasonably avoid it.\n\n``venvstacks`` allows you to package Python applications and all their dependencies into a\nportable, deterministic format, *without* needing to include copies of these large Python\nframeworks in every application archive.\n\nIt achieves this by using Python's `sitecustomize.py` environment setup feature to\nchain together three layers of Python virtual environments:\n\n* \"Runtime\" layers: environment containing the desired version of a specific Python runtime\n* \"Framework\" layers: environments containing desired versions of key Python frameworks\n* \"Application\" layers: environments containing components to be launched directly\n\nApplication layer environments may include additional unpackaged Python launch modules or\npackages for invocation with `python`'s `-m` switch.\n\nWhile the layers are archived and published separately, their dependency locking is integrated,\nallowing the application layers to share dependencies installed in the framework layers,\nand the framework layers to share dependencies installed in the runtime layers.\n\nRefer to the [Project Overview](https://venvstacks.lmstudio.ai/overview/) for an\nexample of specifying, locking, building, and publishing a set of environment stacks.\n\n\nInstalling\n----------\n\n`venvstacks` is available from the [Python Package Index](https://pypi.org/project/venvstacks/),\nand can be installed with [pipx](https://pypi.org/project/pipx/):\n\n $ pipx install venvstacks\n\nAlternatively, it can be installed as a user level package (although this may\nmake future Python version upgrades more irritating):\n\n $ pip install --user venvstacks\n\n\nInteractions with other packaging tools\n---------------------------------------\n\nThe base runtime environment layers are installed with `pdm` (with the installed runtimes coming\nfrom the `python-build-standalone` project). `pdm` is also used to manage the development\nof the `venvstacks` project itself.\n\nThe layered framework and app environments are created with the standard library's `venv` module.\n\nThe Python packages in each layer are currently being installed directly with `pip`, but\nare expected to eventually move to being installed with `uv` to reduce environment\nsetup times during builds.\n\nPlatform-specific environment locking for each layer is performed using\n`uv pip compile`. Refer to `pyproject.toml` for the specific issues preventing\nthe adoption of `uv` for additional purposes.\n\n`venvstacks` expects precompiled `wheel` archives to be available for all included\nPython distribution packages. When this is not the case, other projects like\n[`wagon`](https://pypi.org/project/wagon/#files) or\n[`fromager`](https://pypi.org/project/fromager/)\nmay be useful in generating the required input archives.\n\n\nCaveats and Limitations\n-----------------------\n\n* This project does NOT support combining arbitrary virtual environments with each other.\n Instead, it allows larger integrated applications to split up their Python dependencies into\n distinct layers, without needing to download and install multiple copies of large\n dependencies (such as the PyTorch ML/AI framework). The environment stack specification\n and build process helps ensure that shared dependencies are kept consistent across layers,\n while unshared dependencies are free to vary across the application components that need them.\n* The `venvstacks` Python API is *not yet stable*. Any interface not specifically\n declared as stable in the documentation may be renamed or relocated without a\n deprecation period. API stabilisation (mostly splitting up the overly large\n `venvstacks.stacks` namespace) will be the trigger for the 1.0 milestone release.\n* While the `venvstacks` CLI is broadly stable, there are still some specific areas\n where changes may occur (such as in the handling of relative paths).\n* Dynamic library dependencies across layers currently only work on Windows.\n There is a [proposal](https://github.com/lmstudio-ai/venvstacks/issues38) in\n place for resolving that limitation, but it has not yet been implemented.\n* Local exports to filesystems which do not support symlinks (such as `VFAT` and\n `FAT32`) are nominally supported (with symlinks being replaced by the files\n they refer to), but this support is *not* currently tested.\n* To avoid relying on the Python ecosystem's still limited support for cross-platform\n component installation, the stack build processes need to be executed on the target\n platform (for example, by using an OS matrix in GitHub Actions).\n\n\nProject History\n---------------\n\nThe initial (and ongoing) development of the `venvstacks` project is being funded\nby [LM Studio](https://lmstudio.ai/), where it serves as the foundation of\nLM Studio's support for local execution of Python AI frameworks such as\n[Apple's MLX](https://lmstudio.ai/blog/lmstudio-v0.3.4).\n\nThe use of \"\ud83d\udc38\" (frog) and \"\ud83e\udd8e\" (newts are often mistaken for lizards and\nvice-versa!) as the Unicode support test characters is a reference to the\ninternal LM Studio project that initially built and continues to maintain\n`venvstacks`: Project Amphibian.\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Use layered Python virtual environment stacks to share large dependencies",
"version": "0.2.1",
"project_urls": {
"Changelog": "https://venvstacks.lmstudio.ai/changelog/",
"Documentation": "https://venvstacks.lmstudio.ai/",
"Homepage": "https://github.com/lmstudio-ai/venvstacks",
"Issues": "https://github.com/lmstudio-ai/venvstacks/issues"
},
"split_keywords": [
"packaging",
" dependency",
" venv"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "e2e12a2bb2a7d688d8f4712dbfee52f662c7d13fc20f2b85681960b1a377f23e",
"md5": "1df8e69d7015f0b0be00e544e79234aa",
"sha256": "72d256cb6ac4b7550d29e9627cfda56a72905bce56e705925f5902c7042ac4fa"
},
"downloads": -1,
"filename": "venvstacks-0.2.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "1df8e69d7015f0b0be00e544e79234aa",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.11",
"size": 45517,
"upload_time": "2024-12-05T07:36:56",
"upload_time_iso_8601": "2024-12-05T07:36:56.363735Z",
"url": "https://files.pythonhosted.org/packages/e2/e1/2a2bb2a7d688d8f4712dbfee52f662c7d13fc20f2b85681960b1a377f23e/venvstacks-0.2.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "a8fbf9aae83aa6581e29f88468a98857f29e0504683bac993f9db1a8dde8f685",
"md5": "dc8809c103653b5975122598f2bb7ac4",
"sha256": "1b2a490363b62951405e5ae55ec7424e7f93138eedf99ed2d216d12d76490a48"
},
"downloads": -1,
"filename": "venvstacks-0.2.1.tar.gz",
"has_sig": false,
"md5_digest": "dc8809c103653b5975122598f2bb7ac4",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.11",
"size": 93452,
"upload_time": "2024-12-05T07:36:58",
"upload_time_iso_8601": "2024-12-05T07:36:58.117265Z",
"url": "https://files.pythonhosted.org/packages/a8/fb/f9aae83aa6581e29f88468a98857f29e0504683bac993f9db1a8dde8f685/venvstacks-0.2.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-12-05 07:36:58",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "lmstudio-ai",
"github_project": "venvstacks",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"tox": true,
"lcname": "venvstacks"
}
