# Merak
Merak is a package building toolkit.
This project is started as an attempt to implement a tool that builds a single Cython extension from a Python package, based on the discussion on StackOverflow - [Collapse multiple submodules to one Cython extension](https://stackoverflow.com/questions/30157363/collapse-multiple-submodules-to-one-cython-extension). See the **Idea** section below.
More features and functionalities may be added in the future.
## Install
To install the current release:
```sh
$ pip install merak
```
To upgrade Merak to the latest version, add `--upgrade` flag to the above command.
## Usage
Currently, Merak only supports the `cythonize` command for building binary extension from a Python package. More features and functionalities may be added in the future.
To build a binary extension from a Python package:
```sh
$ merak cythonize PACKAGE_PATH OUTPUT_PATH
```
The package built will be placed at `<OUTPUT_PATH>/<PACKAGE_NAME>`. If `-f` is specified, any existing file / directory at this path will be overwritten.
```
usage: merak cythonize [-h] [-v] [-k] [-s SEP] [-f] path output
positional arguments:
path Python package path
output Output directory
optional arguments:
-h, --help show this help message and exit
-v, --verbose Log verbosity level. Default -> WARNING, -v -> INFO, -vv
or above -> DEBUG.
-k, --color Display logging messages in colors.
-s SEP, --sep SEP Module layer separator, must be Python identifier.
Defaults to '_'
-f, --force Force overwrite if target path exists
--py-cmd PY_CMD Python interpreter to be used for building Cython
package. Defaults to value of environment variable
"PYTHON_CMD", or "python" if "PYTHON_CMD" not specified.
```
## Example
An example package `foo` is included in the `examples/` directory. It consists of one subpackage `bar` with a module `baz` containing a function `do()` in it.
To build the `foo` package, run the following command in the project root:
```sh
$ merak cythonize examples/foo foo-build
```
The `foo` binary package can then be found at `foo-build/foo`. Change directory to `foo-build` and use an interactive Python session to try it out:
```
>>> from foo.bar import baz
__main__:1: DeprecationWarning: Deprecated since Python 3.4. Use importlib.util.find_spec() instead.
>>> baz.do()
Running: foo.bar.baz.do()
```
The deprecation warning seems to originate from the import logic in the compiled `__init__` extension by Cython. It should cause no execution problems at all.
The binary package can be built into a Python distribution via `setuptools` by simply adding a `setup.py` in the output directory that includes the cython extension as package data. For this example, add `setup.py` to `foo-build/` with the following content:
```python
import setuptools
setuptools.setup(
name="foo",
version="0.1.0",
packages=["foo"],
include_package_data=True,
package_data={"foo": ["*"]},
)
```
and run
```sh
$ python setup.py bdist_wheel
```
The distribution can be found at `foo-build/dist/`.
## Idea
Based on [this answer](https://stackoverflow.com/a/52714500/14927788), it appears that it is possible to build a single Cython extension with multiple modules included in it.
However, it does NOT work with multi-level packages. Cython builds a C source file for each module with an initializer named `PyInit_xxx`, which depends on the base name of the module. As the function is defined in the global scope, a name collision would happen if the same base name is used for different modules. For instance, the following package would have a name collision for `__init__.py` and `base.py`:
```
foo/
__init__.py
bar/
__init__.py
base.py
baz/
__init__.py
base.py
```
Here, we solve the problem in two steps:
1. **Module Flattening:** We move all modules to the base layer, with name constructed from their original relative path: `path.replace(path_separator, sep)`, where `sep` is a legal Python identifier. For example, `foo/bar/base.py` -> `foo/bar_sep_base.py` if `sep="_sep_"`.
2. **Import Redirection:** We inject a finder inside the main `__init__.py` that redirects dotted-paths to their flattened counterparts. Using the above example, the finder redirects the import `foo.bar.base` to `foo.bar_sep_base`.
The injected finder is based on [this answer](https://stackoverflow.com/a/52729181/14927788) with some modifications. See the [template](./merak/data/__init__.tmpl) for implementation detail.
The result would contain a single `__init__` extension inside the package folder. The package folder is still required for the builtin importer to load it as a package, rather than a module. The above example would result in a `foo/` folder with a single `__init__` Cython extension in it.
## Resources
- [Cython.org](https://cython.org/)
- [Stackoverflow - Collapse multiple submodules to one Cython Extension](https://stackoverflow.com/questions/30157363/collapse-multiple-submodules-to-forone-cython-extension)
## License
[Apache License 2.0](./LICENSE)
Raw data
{
"_id": null,
"home_page": "https://github.com/dave-msk/merak",
"name": "merak",
"maintainer": "",
"docs_url": null,
"requires_python": ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, <4",
"maintainer_email": "",
"keywords": "merak,cython,binary,package,build",
"author": "(David) Siu-Kei Muk",
"author_email": "david.muk@protonmail.com",
"download_url": "https://github.com/dave-msk/merak/archive/v0.3.1.tar.gz",
"platform": null,
"description": "# Merak\n\nMerak is a package building toolkit.\n\nThis project is started as an attempt to implement a tool that builds a single Cython extension from a Python package, based on the discussion on StackOverflow - [Collapse multiple submodules to one Cython extension](https://stackoverflow.com/questions/30157363/collapse-multiple-submodules-to-one-cython-extension). See the **Idea** section below.\n\nMore features and functionalities may be added in the future.\n\n## Install\n\nTo install the current release:\n\n```sh\n$ pip install merak\n```\n\nTo upgrade Merak to the latest version, add `--upgrade` flag to the above command.\n\n## Usage\n\nCurrently, Merak only supports the `cythonize` command for building binary extension from a Python package. More features and functionalities may be added in the future.\n\nTo build a binary extension from a Python package:\n\n```sh\n$ merak cythonize PACKAGE_PATH OUTPUT_PATH\n```\n\nThe package built will be placed at `<OUTPUT_PATH>/<PACKAGE_NAME>`. If `-f` is specified, any existing file / directory at this path will be overwritten.\n\n```\nusage: merak cythonize [-h] [-v] [-k] [-s SEP] [-f] path output\n\npositional arguments:\n path Python package path\n output Output directory\n\noptional arguments:\n -h, --help show this help message and exit\n -v, --verbose Log verbosity level. Default -> WARNING, -v -> INFO, -vv\n or above -> DEBUG.\n -k, --color Display logging messages in colors.\n -s SEP, --sep SEP Module layer separator, must be Python identifier.\n Defaults to '_'\n -f, --force Force overwrite if target path exists\n --py-cmd PY_CMD Python interpreter to be used for building Cython\n package. Defaults to value of environment variable\n \"PYTHON_CMD\", or \"python\" if \"PYTHON_CMD\" not specified.\n```\n\n## Example\n\nAn example package `foo` is included in the `examples/` directory. It consists of one subpackage `bar` with a module `baz` containing a function `do()` in it.\n\nTo build the `foo` package, run the following command in the project root:\n\n```sh\n$ merak cythonize examples/foo foo-build\n```\n\nThe `foo` binary package can then be found at `foo-build/foo`. Change directory to `foo-build` and use an interactive Python session to try it out:\n\n```\n>>> from foo.bar import baz\n__main__:1: DeprecationWarning: Deprecated since Python 3.4. Use importlib.util.find_spec() instead.\n>>> baz.do()\nRunning: foo.bar.baz.do()\n```\n\nThe deprecation warning seems to originate from the import logic in the compiled `__init__` extension by Cython. It should cause no execution problems at all.\n\nThe binary package can be built into a Python distribution via `setuptools` by simply adding a `setup.py` in the output directory that includes the cython extension as package data. For this example, add `setup.py` to `foo-build/` with the following content:\n\n```python\nimport setuptools\n\nsetuptools.setup(\n name=\"foo\",\n version=\"0.1.0\",\n packages=[\"foo\"],\n include_package_data=True,\n package_data={\"foo\": [\"*\"]},\n)\n```\n\nand run\n\n```sh\n$ python setup.py bdist_wheel\n```\n\nThe distribution can be found at `foo-build/dist/`.\n\n## Idea\n\nBased on [this answer](https://stackoverflow.com/a/52714500/14927788), it appears that it is possible to build a single Cython extension with multiple modules included in it.\n\nHowever, it does NOT work with multi-level packages. Cython builds a C source file for each module with an initializer named `PyInit_xxx`, which depends on the base name of the module. As the function is defined in the global scope, a name collision would happen if the same base name is used for different modules. For instance, the following package would have a name collision for `__init__.py` and `base.py`:\n\n```\nfoo/\n __init__.py\n bar/\n __init__.py\n base.py\n baz/\n __init__.py\n base.py\n```\n\nHere, we solve the problem in two steps:\n\n1. **Module Flattening:** We move all modules to the base layer, with name constructed from their original relative path: `path.replace(path_separator, sep)`, where `sep` is a legal Python identifier. For example, `foo/bar/base.py` -> `foo/bar_sep_base.py` if `sep=\"_sep_\"`.\n2. **Import Redirection:** We inject a finder inside the main `__init__.py` that redirects dotted-paths to their flattened counterparts. Using the above example, the finder redirects the import `foo.bar.base` to `foo.bar_sep_base`.\n\nThe injected finder is based on [this answer](https://stackoverflow.com/a/52729181/14927788) with some modifications. See the [template](./merak/data/__init__.tmpl) for implementation detail.\n\nThe result would contain a single `__init__` extension inside the package folder. The package folder is still required for the builtin importer to load it as a package, rather than a module. The above example would result in a `foo/` folder with a single `__init__` Cython extension in it.\n\n## Resources\n\n- [Cython.org](https://cython.org/)\n- [Stackoverflow - Collapse multiple submodules to one Cython Extension](https://stackoverflow.com/questions/30157363/collapse-multiple-submodules-to-forone-cython-extension)\n\n## License\n\n[Apache License 2.0](./LICENSE)\n\n\n",
"bugtrack_url": null,
"license": "Apache 2.0",
"summary": "Python binary package builder (via Cython)",
"version": "0.3.1",
"project_urls": {
"Download": "https://github.com/dave-msk/merak/archive/v0.3.1.tar.gz",
"Homepage": "https://github.com/dave-msk/merak"
},
"split_keywords": [
"merak",
"cython",
"binary",
"package",
"build"
],
"urls": [
{
"comment_text": "/home/david/.pypirc",
"digests": {
"blake2b_256": "f6f57be0eb5914a2ee282772621343201e8165225b20751d1df7839100e605b9",
"md5": "9712374c453d371818957194a607760d",
"sha256": "38fcccb4dcf46d75b8ba379367e03df2670c78365e534bf588a0265ecb6bf31e"
},
"downloads": -1,
"filename": "merak-0.3.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "9712374c453d371818957194a607760d",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, <4",
"size": 21560,
"upload_time": "2023-08-12T10:43:35",
"upload_time_iso_8601": "2023-08-12T10:43:35.385150Z",
"url": "https://files.pythonhosted.org/packages/f6/f5/7be0eb5914a2ee282772621343201e8165225b20751d1df7839100e605b9/merak-0.3.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-08-12 10:43:35",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "dave-msk",
"github_project": "merak",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"requirements": [],
"lcname": "merak"
}
JSON
