# reorder_editable
Naive implementation to reorder my `easy-install.pth` file
It is meant to be used to make sure editable namespace packages in my `easy-install.pth` are in a specific order.
## Editable Namespace Packages?
To expand:
- Editable: A package that is installed in editable mode (like `pip install -e`), i.e., if you make any changes to the code, your changes are reflected immediately. Is useful for packages that you change very often, or while developing. See the [`site` module docs](https://docs.python.org/3.8/library/site.html) for more information on how this modifies `sys.path`
- Namespace Packages: Namespace packages let you split a package across multiple directories on disk, merging any submodules into the parent package. For more info, see [PEP420](https://www.python.org/dev/peps/pep-0420/#dynamic-path-computation)
_Sidenote_: A namespace package is typically installed using `setuptools.find_namespace_packages`, instead of `setuptools.find_packages`
So, an editable, namespace package is multiple directories on disk all installed as a single package. If any changes are made to any of the directories, the package updates immediately.
This is the current strategy [HPI](https://github.com/karlicoss/HPI) uses for extension. I can keep up to date with [upstream](https://github.com/karlicoss/HPI) and manage [my own modules](https://github.com/seanbreckenridge/HPI) by installing both (or more) like:
```
pip install -e /local/clone/of/karlicoss/HPI
pip install -e /local/clone/of/seanbreckenridge/HPI
```
This creates a file in your python installation that looks like this:
```bash
$ cat ~/.local/lib/python3.9/site-packages/easy-install.pth
/home/sean/Repos/karlicoss/HPI
/home/sean/Repos/seanbreckenridge/HPI
```
... to link those installs to the paths you specified.
However, for namespace packages in particular, the order that those directories appear in the `easy-install.pth` matter. Since items in `easy-install.pth` are added to `sys.path` in order, that determines which order python searches for packages in when trying to resolve imports.
For example, given the following structure:
```
.
├── my_HPI
│ ├── package_name
│ │ ├── a.py
│ │ └── b.py
│ └── setup.py
└── upstream_HPI
├── package_name
│ ├── a.py
│ └── c.py
└── setup.py
```
If `easy-install.pth` was ordered:
```
my_HPI
upstream_HPI
```
`import package_name.a` would import `my_HPI/package_name/a.py`, instead of `upstream_HPI/package_name/a.py`, because that's the first item it matched in the `easy-install.pth`. This process is described much more technically in the [PEP](https://www.python.org/dev/peps/pep-0420/#specification)
This is pretty much a native plugin system, as it lets me overlay specific files/modules with personal changes in my directory structure, while keeping up to date with the upstream changes. There's very little overhead since all I'm doing is adding python files to a local directory and the globally installed package immediately updates.
In the example above, I can still import `package_name.b` and `package_name.c` as normal, even though they're in different directory structures.
---
Now - to the problem this aims to solve.
There is no way to manage your `easy-install.pth` file, to make sure packages are in a defined order.
In particular, I want [my repository](https://github.com/seanbreckenridge/HPI) to be above [my fork of the upstream repo](https://github.com/seanbreckenridge/HPI-fork), as that means I'm able to override files from upstream with my own changes, while maintaining two separate directories - which prevents me from running into merge conflicts. While developing, I may end up uninstalling/reinstalling one or more of my local clones of the `HPI` packages, and that leads to it resolving to a file from the upstream repository, when I was expecting my own -- leading to confusing and difficult to debug errors.
The script itself is pretty basic. All `easy-install.pth` is lines of absolute paths pointing to directories, so this just takes the directories you pass as positional arguments and makes sure they're in that order in your `easy-install.pth` file by shuffling it around.
I don't believe this breaks and built-in python/pip behaviour, but please open a PR/Issue if you think there's an issue/this could be improved.
Should be noted that if you've already imported a namespace module [the `__path__` is cached](https://www.python.org/dev/peps/pep-0420/#rationale) (which determines the import order), so this (probably?) won't work if you re-order the `easy-install.pth` file after the module has already been loaded -- at least not for that python process/unless you re-import it by messing with `sys.modules` and re-import the library.
Still - at least this tells me when it breaks, and fixes it for the next time python runs, so I don't have to worry about it/do it manually.
The actual hack that runs in my `HPI` configuration script, so I never have to think about this again:
```python
def repo(name: str) -> str:
return path.join(environ["REPOS"], name)
# https://github.com/seanbreckenridge/reorder_editable
# if my easy-install.pth file was ordered wrong, fix it and exit!
from reorder_editable import Editable
if Editable().reorder([repo("HPI"), repo("HPI-fork")]):
# this is true if we actually reordered the path, else path was already ordered
print(
"easy-install.pth was ordered wrong! It has been reordered, exiting to apply changes...",
file=sys.stderr,
)
sys.exit(0)
```
Note: Either install all editable packages as `--user`, or all at the system level. Otherwise, the lines of text/packages are split across two separate `easy-install.pth` files, and its not possible to reorder.
## Installation
Requires `python3.7+`
To install with pip, run:
python3 -m pip install reorder_editable
Can always be accessed like `python3 -m reorder_editable`, if your python local bin directory isn't on your `$PATH`
## Usage
```
Usage: reorder_editable [OPTIONS] COMMAND [ARGS]...
Manage your editable packages - your easy-install.pth file
Options:
--help Show this message and exit.
Commands:
cat print easy-install.pth contents
check check easy-install.pth
locate print easy-install.pth file location
reorder reorder easy-install.pth
```
To reorder:
```
Usage: reorder_editable reorder [OPTIONS] DIRECTORY...
If the order specified in your easy-install.pth doesn't match the order of
the directories specified as positional arguments, reorder them so that it
does. This always places items you're reordering at the end of your easy-
install.pth so make sure to include all items you care about the order of
Also fails if one of the paths you provide doesn't exist, or it isn't
already in you easy-install.pth
e.g.
reorder_editable reorder ./path/to/repo /another/path/to/repo
If ./path/to/repo was below /another/path/to/repo, this would reorder
items in your config file to fix it so that ./path/to/repo is above
/another/path/to/repo
Options:
-e, --easy-install-location TEXT
Manually provide path to easy-install.pth
--help Show this message and exit.
```
As an example using the descriptions above, If I wanted to make sure `my_HPI` was above `upstream_HPI`, I'd run:
```bash
$ python3 -m reorder_editable reorder ./my_HPI ./upstream_HPI
```
### Tests
```bash
git clone 'https://github.com/seanbreckenridge/reorder_editable'
cd ./reorder_editable
pip install '.[testing]'
mypy ./reorder_editable
pytest
```
Raw data
{
"_id": null,
"home_page": "https://github.com/seanbreckenridge/reorder_editable",
"name": "reorder-editable",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": "",
"keywords": "packaging,pip",
"author": "Sean Breckenridge",
"author_email": "\"seanbrecke@gmail.com\"",
"download_url": "https://files.pythonhosted.org/packages/a0/0d/90a663ba9c9a6f87b7bd97fbf06d0ee7c117275b5fc7b3e59cddd5992864/reorder_editable-0.1.2.tar.gz",
"platform": null,
"description": "# reorder_editable\n\nNaive implementation to reorder my `easy-install.pth` file\n\nIt is meant to be used to make sure editable namespace packages in my `easy-install.pth` are in a specific order.\n\n## Editable Namespace Packages?\n\nTo expand:\n\n- Editable: A package that is installed in editable mode (like `pip install -e`), i.e., if you make any changes to the code, your changes are reflected immediately. Is useful for packages that you change very often, or while developing. See the [`site` module docs](https://docs.python.org/3.8/library/site.html) for more information on how this modifies `sys.path`\n- Namespace Packages: Namespace packages let you split a package across multiple directories on disk, merging any submodules into the parent package. For more info, see [PEP420](https://www.python.org/dev/peps/pep-0420/#dynamic-path-computation)\n\n_Sidenote_: A namespace package is typically installed using `setuptools.find_namespace_packages`, instead of `setuptools.find_packages`\n\nSo, an editable, namespace package is multiple directories on disk all installed as a single package. If any changes are made to any of the directories, the package updates immediately.\n\nThis is the current strategy [HPI](https://github.com/karlicoss/HPI) uses for extension. I can keep up to date with [upstream](https://github.com/karlicoss/HPI) and manage [my own modules](https://github.com/seanbreckenridge/HPI) by installing both (or more) like:\n\n```\npip install -e /local/clone/of/karlicoss/HPI\npip install -e /local/clone/of/seanbreckenridge/HPI\n```\n\nThis creates a file in your python installation that looks like this:\n\n```bash\n$ cat ~/.local/lib/python3.9/site-packages/easy-install.pth\n/home/sean/Repos/karlicoss/HPI\n/home/sean/Repos/seanbreckenridge/HPI\n```\n\n... to link those installs to the paths you specified.\n\nHowever, for namespace packages in particular, the order that those directories appear in the `easy-install.pth` matter. Since items in `easy-install.pth` are added to `sys.path` in order, that determines which order python searches for packages in when trying to resolve imports.\n\nFor example, given the following structure:\n\n```\n.\n\u251c\u2500\u2500 my_HPI\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 package_name\n\u2502\u00a0\u00a0 \u2502\u00a0\u00a0 \u251c\u2500\u2500 a.py\n\u2502\u00a0\u00a0 \u2502\u00a0\u00a0 \u2514\u2500\u2500 b.py\n\u2502\u00a0\u00a0 \u2514\u2500\u2500 setup.py\n\u2514\u2500\u2500 upstream_HPI\n \u251c\u2500\u2500 package_name\n \u2502\u00a0\u00a0 \u251c\u2500\u2500 a.py\n \u2502\u00a0\u00a0 \u2514\u2500\u2500 c.py\n \u2514\u2500\u2500 setup.py\n```\n\nIf `easy-install.pth` was ordered:\n\n```\nmy_HPI\nupstream_HPI\n```\n\n`import package_name.a` would import `my_HPI/package_name/a.py`, instead of `upstream_HPI/package_name/a.py`, because that's the first item it matched in the `easy-install.pth`. This process is described much more technically in the [PEP](https://www.python.org/dev/peps/pep-0420/#specification)\n\nThis is pretty much a native plugin system, as it lets me overlay specific files/modules with personal changes in my directory structure, while keeping up to date with the upstream changes. There's very little overhead since all I'm doing is adding python files to a local directory and the globally installed package immediately updates.\n\nIn the example above, I can still import `package_name.b` and `package_name.c` as normal, even though they're in different directory structures.\n\n---\n\nNow - to the problem this aims to solve.\n\nThere is no way to manage your `easy-install.pth` file, to make sure packages are in a defined order.\n\nIn particular, I want [my repository](https://github.com/seanbreckenridge/HPI) to be above [my fork of the upstream repo](https://github.com/seanbreckenridge/HPI-fork), as that means I'm able to override files from upstream with my own changes, while maintaining two separate directories - which prevents me from running into merge conflicts. While developing, I may end up uninstalling/reinstalling one or more of my local clones of the `HPI` packages, and that leads to it resolving to a file from the upstream repository, when I was expecting my own -- leading to confusing and difficult to debug errors.\n\nThe script itself is pretty basic. All `easy-install.pth` is lines of absolute paths pointing to directories, so this just takes the directories you pass as positional arguments and makes sure they're in that order in your `easy-install.pth` file by shuffling it around.\n\nI don't believe this breaks and built-in python/pip behaviour, but please open a PR/Issue if you think there's an issue/this could be improved.\n\nShould be noted that if you've already imported a namespace module [the `__path__` is cached](https://www.python.org/dev/peps/pep-0420/#rationale) (which determines the import order), so this (probably?) won't work if you re-order the `easy-install.pth` file after the module has already been loaded -- at least not for that python process/unless you re-import it by messing with `sys.modules` and re-import the library.\n\nStill - at least this tells me when it breaks, and fixes it for the next time python runs, so I don't have to worry about it/do it manually.\n\nThe actual hack that runs in my `HPI` configuration script, so I never have to think about this again:\n\n```python\ndef repo(name: str) -> str:\n return path.join(environ[\"REPOS\"], name)\n\n\n# https://github.com/seanbreckenridge/reorder_editable\n# if my easy-install.pth file was ordered wrong, fix it and exit!\nfrom reorder_editable import Editable\n\nif Editable().reorder([repo(\"HPI\"), repo(\"HPI-fork\")]):\n # this is true if we actually reordered the path, else path was already ordered\n print(\n \"easy-install.pth was ordered wrong! It has been reordered, exiting to apply changes...\",\n file=sys.stderr,\n )\n sys.exit(0)\n```\n\nNote: Either install all editable packages as `--user`, or all at the system level. Otherwise, the lines of text/packages are split across two separate `easy-install.pth` files, and its not possible to reorder.\n\n## Installation\n\nRequires `python3.7+`\n\nTo install with pip, run:\n\n python3 -m pip install reorder_editable\n\nCan always be accessed like `python3 -m reorder_editable`, if your python local bin directory isn't on your `$PATH`\n\n## Usage\n\n```\nUsage: reorder_editable [OPTIONS] COMMAND [ARGS]...\n\n Manage your editable packages - your easy-install.pth file\n\nOptions:\n --help Show this message and exit.\n\nCommands:\n cat print easy-install.pth contents\n check check easy-install.pth\n locate print easy-install.pth file location\n reorder reorder easy-install.pth\n```\n\nTo reorder:\n\n```\nUsage: reorder_editable reorder [OPTIONS] DIRECTORY...\n\n If the order specified in your easy-install.pth doesn't match the order of\n the directories specified as positional arguments, reorder them so that it\n does. This always places items you're reordering at the end of your easy-\n install.pth so make sure to include all items you care about the order of\n\n Also fails if one of the paths you provide doesn't exist, or it isn't\n already in you easy-install.pth\n\n e.g.\n reorder_editable reorder ./path/to/repo /another/path/to/repo\n\n If ./path/to/repo was below /another/path/to/repo, this would reorder\n items in your config file to fix it so that ./path/to/repo is above\n /another/path/to/repo\n\nOptions:\n -e, --easy-install-location TEXT\n Manually provide path to easy-install.pth\n --help Show this message and exit.\n```\n\nAs an example using the descriptions above, If I wanted to make sure `my_HPI` was above `upstream_HPI`, I'd run:\n\n```bash\n$ python3 -m reorder_editable reorder ./my_HPI ./upstream_HPI\n```\n\n### Tests\n\n```bash\ngit clone 'https://github.com/seanbreckenridge/reorder_editable'\ncd ./reorder_editable\npip install '.[testing]'\nmypy ./reorder_editable\npytest\n```\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "naive implementation to reorder my easy-install.pth file",
"version": "0.1.2",
"project_urls": {
"Homepage": "https://github.com/seanbreckenridge/reorder_editable"
},
"split_keywords": [
"packaging",
"pip"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "f9879ba439bf6d1e2af356f48302722936f23f0cd3d0213a5a1b7968158a5266",
"md5": "51555dd405a35fcd3b9171ee78f25130",
"sha256": "74916e89801989093d8b30169719e62252ce96a6ba86bdc8ea031c2503264bac"
},
"downloads": -1,
"filename": "reorder_editable-0.1.2-py3-none-any.whl",
"has_sig": false,
"md5_digest": "51555dd405a35fcd3b9171ee78f25130",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 9626,
"upload_time": "2023-09-28T03:19:14",
"upload_time_iso_8601": "2023-09-28T03:19:14.154275Z",
"url": "https://files.pythonhosted.org/packages/f9/87/9ba439bf6d1e2af356f48302722936f23f0cd3d0213a5a1b7968158a5266/reorder_editable-0.1.2-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "a00d90a663ba9c9a6f87b7bd97fbf06d0ee7c117275b5fc7b3e59cddd5992864",
"md5": "9d4d8b74fe418d75fcdabfe3def49818",
"sha256": "0d6b4be68102ffebfbe631b3c4056528760804e3b14cd0f66ae40be568a370b5"
},
"downloads": -1,
"filename": "reorder_editable-0.1.2.tar.gz",
"has_sig": false,
"md5_digest": "9d4d8b74fe418d75fcdabfe3def49818",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 11848,
"upload_time": "2023-09-28T03:19:15",
"upload_time_iso_8601": "2023-09-28T03:19:15.461911Z",
"url": "https://files.pythonhosted.org/packages/a0/0d/90a663ba9c9a6f87b7bd97fbf06d0ee7c117275b5fc7b3e59cddd5992864/reorder_editable-0.1.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-09-28 03:19:15",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "seanbreckenridge",
"github_project": "reorder_editable",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "reorder-editable"
}
JSON
