pyodide-pack


Namepyodide-pack JSON
Version 0.2.0 PyPI version JSON
download
home_pagehttps://github.com/rth/pyodide-bundler
SummaryA bundler for Python packages for the web
upload_time2022-12-14 08:51:31
maintainer
docs_urlNone
authorRoman Yurchak
requires_python>=3.9
license
keywords
VCS
bugtrack_url
requirements No requirements were recorded.
Travis-CI No Travis.
coveralls test coverage No coveralls. 
            # pyodide-pack

[![PyPI Latest Release](https://img.shields.io/pypi/v/pyodide-pack.svg)](https://pypi.org/project/pyodide-pack/)
[![GHA CI](https://github.com/rth/pyodide-pack/actions/workflows/main.yml/badge.svg?branch=main)](https://github.com/rth/pyodide-pack/actions/workflows/main.yml)

Python package bundler for the web

THIS PACKAGE IS STILL VERY EXPERIMENTAL

Pyodide-pack detects used modules in a Python application running in the web with Pyodide, and creates a minimal bundle with them. This allows to significantly reduce the download size of Python applications, provided that the code to execute is known in advance.

## Install

Pyodide-pack requires Python 3.10+ as well as NodeJS,

```bash
pip install pyodide-pack
npm install pyodide@0.20.1-alpha.2
# A hack due to the npm package having issues
wget https://cdn.jsdelivr.net/pyodide/v0.20.0/full/packages.json -O node_modules/pyodide/packages.json
```

## Quickstart

1. Create file with the code of your Python application running in the web. As example we will take,
   `examples/scikit-learn/app.py`

   **app.py**

   ```py
   import pandas as pd  # noqa

   pd.DataFrame(range(10))
   ```

   This application can run with Pyodide, and will need to download around 10.5
   MB of packages, including numpy and pandas in addition to
   ~7MB for CPython with stdlib.

2. Create the package bundle,

   ```bash
   pyodide pack examples/pandas/app.py
   ```
   which would produce the following output

   ```
   Running pyodide-pack on examples/pandas/app.py

   Note: unless otherwise specified all sizes are given for gzip compressed files to
   be representative of CDN compression.

   Loaded requirements from: examples/pandas/requirements.txt
   Running the input code in Node.js to detect used modules..

   [..]

   Done input code execution in 11.1 s

   Detected 8 dependencies with a total size of 10.54 MB  (uncompressed: 40.99 MB)
   In total 425 files and 54 dynamic libraries were accessed.

                                          Packing..
   ┏━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━┓
   ┃ No ┃ Package                        ┃ All files ┃ .so libs ┃   Size (MB) ┃ Reduction ┃
   ┡━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━┩
   │  1 │ distutils.tar                  │   101 → 0 │    0 → 0 │ 0.26 → 0.00 │   100.0 % │
   │  2 │ numpy-1.22.3-cp310-cp310-emsc… │  418 → 94 │  19 → 13 │ 3.63 → 2.49 │    31.4 % │
   │  3 │ pandas-1.4.2-cp310-cp310-emsc… │ 469 → 283 │  42 → 41 │ 5.11 → 4.50 │    12.0 % │
   │  4 │ pyparsing-3.0.7-py3-none-any.… │    17 → 0 │    0 → 0 │ 0.10 → 0.00 │   100.0 % │
   │  5 │ python_dateutil-2.8.2-py2.py3… │   25 → 15 │    0 → 0 │ 0.24 → 0.22 │     9.4 % │
   │  6 │ pytz-2022.1-py2.py3-none-any.… │   612 → 5 │    0 → 0 │ 0.43 → 0.02 │    96.1 % │
   │  7 │ setuptools-62.0.0-py3-none-an… │   213 → 0 │    0 → 0 │ 0.76 → 0.00 │   100.0 % │
   │  8 │ six-1.16.0-py2.py3-none-any.w… │     6 → 1 │    0 → 0 │ 0.01 → 0.01 │    18.5 % │
   └────┴────────────────────────────────┴───────────┴──────────┴─────────────┴───────────┘
   Wrote pyodide-package-bundle.zip with 7.36 MB (30.2% reduction)

   Running the input code in Node.js to validate bundle..

           Validating and benchmarking the output bundle..
   ┏━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┓
   ┃ Step                 ┃ Load time (s) ┃ Fraction of load time ┃
   ┡━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━┩
   │ loadPyodide          │          2.59 │                24.4 % │
   │ fetch_unpack_archive │          0.27 │                 2.5 % │
   │ load_dynamic_libs    │          6.21 │                58.5 % │
   │ import_run_app       │          1.56 │                14.7 % │
   │ TOTAL                │         10.63 │                 100 % │
   └──────────────────────┴───────────────┴───────────────────────┘

   Bundle validation successful.
   ```
3. Load your Python web application with,
   ```js
   let pyodide = await loadPyodide({fullStdLib: false});

   await pyodide.runPythonAsync(`
     from pyodide.http import pyfetch

     response = await pyfetch("<your-server>/pyodide-package-bundle.zip")
     await response.unpack_archive(extract_dir='/')
   `)

   await pyodide.pyimport('pyodide_pack_loader').setup();
   ```

## Implementation

This bundler runs your applications in a Node.js and intercepts,
 - `FS.open` calls in read mode, which includes accessed files in the Emscripten's MEMFS file system opened from Python, C or Javascript.
 - calls to load a dynamic library

Package wheels are then repacked into a single bundle with the accessed files and dynamic libraries.

## License

Pyodide-pack uses the [Mozilla Public License Version 2.0](https://choosealicense.com/licenses/mpl-2.0/).

Raw data

            {
    "_id": null,
    "home_page": "https://github.com/rth/pyodide-bundler",
    "name": "pyodide-pack",
    "maintainer": "",
    "docs_url": null,
    "requires_python": ">=3.9",
    "maintainer_email": "",
    "keywords": "",
    "author": "Roman Yurchak",
    "author_email": "",
    "download_url": "https://files.pythonhosted.org/packages/69/ab/512b98d585a333fee0b94fac6b191ab19d0f19527db4c27c25d27c0ecaa2/pyodide-pack-0.2.0.tar.gz",
    "platform": null,
    "description": "# pyodide-pack\n\n[![PyPI Latest Release](https://img.shields.io/pypi/v/pyodide-pack.svg)](https://pypi.org/project/pyodide-pack/)\n[![GHA CI](https://github.com/rth/pyodide-pack/actions/workflows/main.yml/badge.svg?branch=main)](https://github.com/rth/pyodide-pack/actions/workflows/main.yml)\n\nPython package bundler for the web\n\nTHIS PACKAGE IS STILL VERY EXPERIMENTAL\n\nPyodide-pack detects used modules in a Python application running in the web with Pyodide, and creates a minimal bundle with them. This allows to significantly reduce the download size of Python applications, provided that the code to execute is known in advance.\n\n## Install\n\nPyodide-pack requires Python 3.10+ as well as NodeJS,\n\n```bash\npip install pyodide-pack\nnpm install pyodide@0.20.1-alpha.2\n# A hack due to the npm package having issues\nwget https://cdn.jsdelivr.net/pyodide/v0.20.0/full/packages.json -O node_modules/pyodide/packages.json\n```\n\n## Quickstart\n\n1. Create file with the code of your Python application running in the web. As example we will take,\n   `examples/scikit-learn/app.py`\n\n   **app.py**\n\n   ```py\n   import pandas as pd  # noqa\n\n   pd.DataFrame(range(10))\n   ```\n\n   This application can run with Pyodide, and will need to download around 10.5\n   MB of packages, including numpy and pandas in addition to\n   ~7MB for CPython with stdlib.\n\n2. Create the package bundle,\n\n   ```bash\n   pyodide pack examples/pandas/app.py\n   ```\n   which would produce the following output\n\n   ```\n   Running pyodide-pack on examples/pandas/app.py\n\n   Note: unless otherwise specified all sizes are given for gzip compressed files to\n   be representative of CDN compression.\n\n   Loaded requirements from: examples/pandas/requirements.txt\n   Running the input code in Node.js to detect used modules..\n\n   [..]\n\n   Done input code execution in 11.1 s\n\n   Detected 8 dependencies with a total size of 10.54 MB  (uncompressed: 40.99 MB)\n   In total 425 files and 54 dynamic libraries were accessed.\n\n                                          Packing..\n   \u250f\u2501\u2501\u2501\u2501\u2533\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2533\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2533\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2533\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2533\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2513\n   \u2503 No \u2503 Package                        \u2503 All files \u2503 .so libs \u2503   Size (MB) \u2503 Reduction \u2503\n   \u2521\u2501\u2501\u2501\u2501\u2547\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2547\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2547\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2547\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2547\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2529\n   \u2502  1 \u2502 distutils.tar                  \u2502   101 \u2192 0 \u2502    0 \u2192 0 \u2502 0.26 \u2192 0.00 \u2502   100.0 % \u2502\n   \u2502  2 \u2502 numpy-1.22.3-cp310-cp310-emsc\u2026 \u2502  418 \u2192 94 \u2502  19 \u2192 13 \u2502 3.63 \u2192 2.49 \u2502    31.4 % \u2502\n   \u2502  3 \u2502 pandas-1.4.2-cp310-cp310-emsc\u2026 \u2502 469 \u2192 283 \u2502  42 \u2192 41 \u2502 5.11 \u2192 4.50 \u2502    12.0 % \u2502\n   \u2502  4 \u2502 pyparsing-3.0.7-py3-none-any.\u2026 \u2502    17 \u2192 0 \u2502    0 \u2192 0 \u2502 0.10 \u2192 0.00 \u2502   100.0 % \u2502\n   \u2502  5 \u2502 python_dateutil-2.8.2-py2.py3\u2026 \u2502   25 \u2192 15 \u2502    0 \u2192 0 \u2502 0.24 \u2192 0.22 \u2502     9.4 % \u2502\n   \u2502  6 \u2502 pytz-2022.1-py2.py3-none-any.\u2026 \u2502   612 \u2192 5 \u2502    0 \u2192 0 \u2502 0.43 \u2192 0.02 \u2502    96.1 % \u2502\n   \u2502  7 \u2502 setuptools-62.0.0-py3-none-an\u2026 \u2502   213 \u2192 0 \u2502    0 \u2192 0 \u2502 0.76 \u2192 0.00 \u2502   100.0 % \u2502\n   \u2502  8 \u2502 six-1.16.0-py2.py3-none-any.w\u2026 \u2502     6 \u2192 1 \u2502    0 \u2192 0 \u2502 0.01 \u2192 0.01 \u2502    18.5 % \u2502\n   \u2514\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n   Wrote pyodide-package-bundle.zip with 7.36 MB (30.2% reduction)\n\n   Running the input code in Node.js to validate bundle..\n\n           Validating and benchmarking the output bundle..\n   \u250f\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2533\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2533\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2513\n   \u2503 Step                 \u2503 Load time (s) \u2503 Fraction of load time \u2503\n   \u2521\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2547\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2547\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2529\n   \u2502 loadPyodide          \u2502          2.59 \u2502                24.4 % \u2502\n   \u2502 fetch_unpack_archive \u2502          0.27 \u2502                 2.5 % \u2502\n   \u2502 load_dynamic_libs    \u2502          6.21 \u2502                58.5 % \u2502\n   \u2502 import_run_app       \u2502          1.56 \u2502                14.7 % \u2502\n   \u2502 TOTAL                \u2502         10.63 \u2502                 100 % \u2502\n   \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n\n   Bundle validation successful.\n   ```\n3. Load your Python web application with,\n   ```js\n   let pyodide = await loadPyodide({fullStdLib: false});\n\n   await pyodide.runPythonAsync(`\n     from pyodide.http import pyfetch\n\n     response = await pyfetch(\"<your-server>/pyodide-package-bundle.zip\")\n     await response.unpack_archive(extract_dir='/')\n   `)\n\n   await pyodide.pyimport('pyodide_pack_loader').setup();\n   ```\n\n## Implementation\n\nThis bundler runs your applications in a Node.js and intercepts,\n - `FS.open` calls in read mode, which includes accessed files in the Emscripten's MEMFS file system opened from Python, C or Javascript.\n - calls to load a dynamic library\n\nPackage wheels are then repacked into a single bundle with the accessed files and dynamic libraries.\n\n## License\n\nPyodide-pack uses the [Mozilla Public License Version 2.0](https://choosealicense.com/licenses/mpl-2.0/).\n",
    "bugtrack_url": null,
    "license": "",
    "summary": "A bundler for Python packages for the web",
    "version": "0.2.0",
    "split_keywords": [],
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "bcfdb2fcde502fa2bf42b62247938068",
                "sha256": "98f703abe72e7eda2fd0305726958889f13ecb58fe06d4a433832599459cb1f5"
            },
            "downloads": -1,
            "filename": "pyodide_pack-0.2.0-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "bcfdb2fcde502fa2bf42b62247938068",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": ">=3.9",
            "size": 17420,
            "upload_time": "2022-12-14T08:51:30",
            "upload_time_iso_8601": "2022-12-14T08:51:30.159051Z",
            "url": "https://files.pythonhosted.org/packages/60/47/27971449f919eeba2709d918723e06b24a0d7bf63d6764ec2650409ec8d0/pyodide_pack-0.2.0-py3-none-any.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": "",
            "digests": {
                "md5": "1b1a3abc3bb480028cdca7d37a354d40",
                "sha256": "7329291152c5f5b53cc0a94dcea7b0da549cc0aee25d08c1196039775dbe8538"
            },
            "downloads": -1,
            "filename": "pyodide-pack-0.2.0.tar.gz",
            "has_sig": false,
            "md5_digest": "1b1a3abc3bb480028cdca7d37a354d40",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": ">=3.9",
            "size": 21106,
            "upload_time": "2022-12-14T08:51:31",
            "upload_time_iso_8601": "2022-12-14T08:51:31.691218Z",
            "url": "https://files.pythonhosted.org/packages/69/ab/512b98d585a333fee0b94fac6b191ab19d0f19527db4c27c25d27c0ecaa2/pyodide-pack-0.2.0.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2022-12-14 08:51:31",
    "github": true,
    "gitlab": false,
    "bitbucket": false,
    "github_user": "rth",
    "github_project": "pyodide-bundler",
    "travis_ci": false,
    "coveralls": false,
    "github_actions": true,
    "lcname": "pyodide-pack"
}
Roman Yurchak
Elapsed time: 0.02281s