# httpimport
## *Python's missing feature!*
##### [The feature has been suggested in Python Mailing List](https://lwn.net/Articles/732194/)
_Remote_, _in-memory_ Python _package/module_ `import`ing **through HTTP/S**
[](https://pypi.org/project/httpimport/)
[](https://pypi.python.org/pypi/httpimport)
[](https://github.com/operatorequals/httpimport/actions/workflows/python-package.yml)
A feature that _Python_ **misses** and has become popular in other languages is the **remote loading of packages/modules**.
`httpimport` lets Python packages and modules to be *installed* and *imported* directly in Python interpreter's process memory, through **remote `URIs`**, and *more*...
#### **Python2 support has been discontinued**. Last version that supports Python2 is [`1.1.0`](https://pypi.org/project/httpimport/1.1.0/).
## Basic Usage
### Load package/module accessible through any HTTP/S location
```python
with httpimport.remote_repo('http://my-codes.example.com/python_packages'):
import package1
```
### Load a module from PyPI:
```python
with httpimport.pypi_repo():
import distlib # https://pypi.org/project/distlib/
print(distlib.__version__)
# '0.3.6' <-- currently latest (https://github.com/pypa/distlib/blob/0.3.6/distlib/__init__.py#L9)
```
### Load directly from a GitHub/BitBucket/GitLab repo
```python
with httpimport.github_repo('operatorequals', 'httpimport', ref='master'):
import httpimport as httpimport_upstream
# Also works with 'bitbucket_repo' and 'gitlab_repo'
```
### Load a Python module from a Github Gist (using [this gist](https://gist.github.com/operatorequals/ee5049677e7bbc97af2941d1d3f04ace)):
```python
url = "https://gist.githubusercontent.com/operatorequals/ee5049677e7bbc97af2941d1d3f04ace/raw/e55fa867d3fb350f70b2897bb415f410027dd7e4"
with httpimport.remote_repo(url):
import hello
hello.hello()
# Hello world
```
### Load a package/module directly to a variable
```python
# From HTTP/S URL
http_module = httpimport.load('package1', 'https://my-codes.example.com/python_packages')
print(http_module)
<module 'package1' from 'https://my-codes.example.com/python_packages/package1/__init__.py'>
# From PyPI
pypi_module = httpimport.load('distlib', importer_class=httpimport.PyPIImporter)
print(pypi_module)
<module 'distlib' from 'https://files.pythonhosted.org/packages/76/cb/6bbd2b10170ed991cf64e8c8b85e01f2fb38f95d1bc77617569e0b0b26ac/distlib-0.3.6-py2.py3-none-any.whl#distlib/__init__.py'>
```
### Load Python packages from archives served through HTTP/S
*No file is touching the disk in the process*
```python
# with httpimport.remote_repo('https://example.com/packages.tar'):
# with httpimport.remote_repo('https://example.com/packages.tar.bz2'):
# with httpimport.remote_repo('https://example.com/packages.tar.gz'):
# with httpimport.remote_repo('https://example.com/packages.tar.xz'):
with httpimport.remote_repo('https://example.com/packages.zip'):
import test_package
```
## Serving a package through HTTP/S
Any package can be served for `httpimport` using a simple HTTP/S Server:
```bash
echo 'print("Hello httpimport!")' > module.py
python -m http.server
Serving HTTP on 0.0.0.0 port 8000 ...
```
```python
>>> import httpimport
>>> with httpimport.remote_repo("http://127.0.0.1:8000"):
... import module
...
Hello httpimport!
```
## Profiles
After `v1.0.0` it is possible to set HTTP Authentication, Custom Headers, Proxies and several other things using *URL* and *Named Profiles*!
### URL Profiles
URL Profiles are INI configurations, setting specific per-URL options, as below:
```ini
[http://127.0.0.1:8000]
allow-plaintext: yes ; also 'true' and '1' evaluate to True
[https://example.com]
proxy-url: https://127.0.0.1:8080 ; values must not be in quotes (')
```
Now, requests to `http://127.0.0.1:8000` will be allowed (HTTP URLs do not work by default) and requests to `https://example.com` will be sent to an HTTP Proxy.
```python
with httpimport.remote_repo("https://example.com"): # URL matches the URL profile
import module_accessed_through_proxy
```
### Named Profiles
Named Profiles are like URL profiles but do not specify a URL and need to be explicitly used:
```ini
[github]
headers:
Authorization: token <Github-Token>
```
And the above can be used as follows:
```python
with httpimport.github_repo('operatorequals','httpimport-private-test', profile='github'):
import secret_module
```
##### Github Tokens look like `github_pat_<gibberish>` and can be issued here: https://github.com/settings/tokens/new
### Profiles for PyPI
When importing from PyPI extra options can be used, as described in the profile below:
```ini
[pypi]
# The location of a 'requirements.txt' file
# to use for PyPI project versions
requirements-file: requirements-dev.txt
# Inline 'requirements.txt' syntax appended
requirements:
distlib==0.3.5
sampleproject==3.0.0
# Only version pinning notation is supported ('==')
# with 'requirements' and 'requirements-file' options
# A map that contains 'module': 'PyPI project' tuples
# i.e: 'import sample' --> search 'sample' module at 'sampleproject' PyPI Project:
# https://pypi.org/project/sampleproject/
project-names:
sample: sampleproject
```
The PyPI Profiles can be used exactly like all *Named Profiles*:
```python
with httpimport.pypi_repo(profile='pypi'):
import distlib
import sample
distlib.__version__
# '0.3.5' <-- pinned in the profile 'requirements' option
sample.__url__
# 'https://files.pythonhosted.org/packages/ec/a8/5ec62d18adde798d33a170e7f72930357aa69a60839194c93eb0fb05e59c/sampleproject-3.0.0-py3-none-any.whl#sample/__init__.py' <-- loaded from 'sampleproject'
```
Additionally, all other options cascade to PyPI profiles, such as HTTPS Proxy (HTTP proxies won't work, as PyPI is hosted with HTTPS), `headers`, etc.
##### NOTE: The values in Profiles MUST NOT be quoted (`'`,`"`)
### Profile Creation
Profiles can be provided as INI strings to the `set_profile` function and used in all `httpimport` functions:
```python
httpimport.set_profile("""
[profile1]
proxy-url: https://my-proxy.example.com
headers:
Authorization: Basic ...
X-Hello-From: httpimport
X-Some-Other: HTTP header
""")
with httpimport.remote_repo("https://code.example.com", profile='profile1'):
import module_accessed_through_proxy
```
#### Advanced
Profiles are INI configuration strings parsed using Python [`configparser`](https://docs.python.org/3/library/configparser.html) module.
The `ConfigParser` object for `httpimport` is the global variable `httpimport.CONFIG` and can be used freely:
```python
import httpimport
httpimport.CONFIG.read('github.ini') # Read profiles from a file
with httpimport.github_repo('operatorequals','httpimport-private-test', profile='github'):
import secret_module
```
## Default Profiles
The `httpimport` module automatically loads Profiles found in `$HOME/.httpimport.ini` and under the `$HOME/.httpimport/` directory. Profiles under `$HOME/.httpimport/` override ones found in `$HOME/.httpimport.ini`.
### Profile Options:
#### Supported
HTTP options
* `zip-password` - `v1.0.0`
* `proxy-url` - `v1.0.0`
* `headers` - `v1.0.0`
* `allow-plaintext` - `v1.0.0`
* `ca-verify` - `v1.3.0`
* `ca-file` - `v1.3.0`
PyPI-only options
* `project-names` - `v1.2.0`
* `requirements` - `v1.2.0`
* `requirements-file` - `v1.2.0`
#### Not yet (subject to change)
* `allow-compiled`
* `auth`
* `auth-type`
* `tls-cert`
* `tls-key`
* `tls-passphrase`
## Debugging...
```python
import httpimport
import logging
logging.getLogger('httpimport').setLevel(logging.DEBUG)
```
## Beware: **Huge Security Implications!**
_Using the `httpimport` with plain **HTTP URLs** is highly discouraged_
As HTTP traffic can be read and changed from all intermediate hosts (_unlike HTTPS_), it is possible for a remote adversary to alter the HTTP responses consumed by `httpimport` and add arbitrary _Python_ code to the downloaded _packages/modules_. This directly results in arbitrary _Remote Code Execution_ on your current user's context of your host!
In other words, using plain HTTP through the Internet can compromise your host *without a way to notice it*.
##### You have been warned! Use only **HTTPS URLs** with `httpimport`!
## Contributors
* [ldsink](https://github.com/ldsink) - The `RELOAD` flag and Bug Fixes
* [lavvy](https://github.com/lavvy) - the `load()` function
* [superloach](https://github.com/superloach) - Deprecation of `imp` module in Python3 in favour of `importlib`
* [yanliakos](https://github.com/yanliakos) - Bug Fix
* [rkbennett](https://github.com/rkbennett) - Relative Imports fix, Proxy support
## Donations
In case my work helped you, you can always buy me a beer or a liter of gas [through the Internet](https://www.buymeacoffee.com/operatorequals) or in case you meet me personally.
In the second case we can talk about any video of [Internet Historian](https://www.youtube.com/@InternetHistorian) or [Ordinary Things](https://www.youtube.com/@OrdinaryThings), while listening to a [Lofi Girl Playlist](https://www.youtube.com/watch?v=jfKfPfyJRdk), like the citizens of the Internet that we are.
[](https://www.buymeacoffee.com/operatorequals)
Raw data
{
"_id": null,
"home_page": "https://github.com/operatorequals/httpimport",
"name": "httpimport",
"maintainer": "",
"docs_url": null,
"requires_python": "",
"maintainer_email": "",
"keywords": "import,loader,memory,http,network",
"author": "John Torakis - operatorequals",
"author_email": "john.torakis@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/4d/95/4e3f5586f74e259aa0cc41b41ef340c60e5cd81a681c24e57924e7fbd9be/httpimport-1.3.1.tar.gz",
"platform": null,
"description": "# httpimport\n## *Python's missing feature!*\n##### [The feature has been suggested in Python Mailing List](https://lwn.net/Articles/732194/)\n\n_Remote_, _in-memory_ Python _package/module_ `import`ing **through HTTP/S**\n\n[](https://pypi.org/project/httpimport/)\n\n\n[](https://pypi.python.org/pypi/httpimport)\n[](https://github.com/operatorequals/httpimport/actions/workflows/python-package.yml)\n\n\n\n\nA feature that _Python_ **misses** and has become popular in other languages is the **remote loading of packages/modules**.\n\n`httpimport` lets Python packages and modules to be *installed* and *imported* directly in Python interpreter's process memory, through **remote `URIs`**, and *more*...\n\n#### **Python2 support has been discontinued**. Last version that supports Python2 is [`1.1.0`](https://pypi.org/project/httpimport/1.1.0/).\n\n## Basic Usage\n\n### Load package/module accessible through any HTTP/S location\n```python\nwith httpimport.remote_repo('http://my-codes.example.com/python_packages'):\n import package1\n```\n\n### Load a module from PyPI:\n```python\nwith httpimport.pypi_repo():\n import distlib # https://pypi.org/project/distlib/\n\nprint(distlib.__version__)\n# '0.3.6' <-- currently latest (https://github.com/pypa/distlib/blob/0.3.6/distlib/__init__.py#L9)\n```\n\n### Load directly from a GitHub/BitBucket/GitLab repo\n```python\nwith httpimport.github_repo('operatorequals', 'httpimport', ref='master'):\n import httpimport as httpimport_upstream\n # Also works with 'bitbucket_repo' and 'gitlab_repo'\n```\n\n### Load a Python module from a Github Gist (using [this gist](https://gist.github.com/operatorequals/ee5049677e7bbc97af2941d1d3f04ace)):\n```python\nurl = \"https://gist.githubusercontent.com/operatorequals/ee5049677e7bbc97af2941d1d3f04ace/raw/e55fa867d3fb350f70b2897bb415f410027dd7e4\"\n\nwith httpimport.remote_repo(url):\n import hello\n\nhello.hello()\n# Hello world\n```\n\n### Load a package/module directly to a variable\n```python\n# From HTTP/S URL\nhttp_module = httpimport.load('package1', 'https://my-codes.example.com/python_packages')\nprint(http_module)\n<module 'package1' from 'https://my-codes.example.com/python_packages/package1/__init__.py'>\n\n# From PyPI\npypi_module = httpimport.load('distlib', importer_class=httpimport.PyPIImporter)\nprint(pypi_module)\n<module 'distlib' from 'https://files.pythonhosted.org/packages/76/cb/6bbd2b10170ed991cf64e8c8b85e01f2fb38f95d1bc77617569e0b0b26ac/distlib-0.3.6-py2.py3-none-any.whl#distlib/__init__.py'>\n```\n\n### Load Python packages from archives served through HTTP/S\n*No file is touching the disk in the process*\n```python\n# with httpimport.remote_repo('https://example.com/packages.tar'):\n# with httpimport.remote_repo('https://example.com/packages.tar.bz2'):\n# with httpimport.remote_repo('https://example.com/packages.tar.gz'):\n# with httpimport.remote_repo('https://example.com/packages.tar.xz'):\nwith httpimport.remote_repo('https://example.com/packages.zip'):\n import test_package\n```\n\n## Serving a package through HTTP/S\nAny package can be served for `httpimport` using a simple HTTP/S Server:\n```bash\necho 'print(\"Hello httpimport!\")' > module.py\npython -m http.server\nServing HTTP on 0.0.0.0 port 8000 ...\n\n```\n\n```python\n>>> import httpimport\n>>> with httpimport.remote_repo(\"http://127.0.0.1:8000\"):\n... import module\n...\nHello httpimport!\n```\n\n## Profiles\nAfter `v1.0.0` it is possible to set HTTP Authentication, Custom Headers, Proxies and several other things using *URL* and *Named Profiles*!\n\n### URL Profiles\nURL Profiles are INI configurations, setting specific per-URL options, as below:\n\n```ini\n[http://127.0.0.1:8000]\nallow-plaintext: yes ; also 'true' and '1' evaluate to True\n\n[https://example.com]\nproxy-url: https://127.0.0.1:8080 ; values must not be in quotes (')\n\n```\n\nNow, requests to `http://127.0.0.1:8000` will be allowed (HTTP URLs do not work by default) and requests to `https://example.com` will be sent to an HTTP Proxy.\n\n```python\nwith httpimport.remote_repo(\"https://example.com\"): # URL matches the URL profile\n import module_accessed_through_proxy\n```\n### Named Profiles\nNamed Profiles are like URL profiles but do not specify a URL and need to be explicitly used:\n\n```ini\n[github]\nheaders:\n Authorization: token <Github-Token>\n```\n\nAnd the above can be used as follows:\n\n```python\nwith httpimport.github_repo('operatorequals','httpimport-private-test', profile='github'):\n import secret_module\n```\n\n##### Github Tokens look like `github_pat_<gibberish>` and can be issued here: https://github.com/settings/tokens/new\n\n### Profiles for PyPI\nWhen importing from PyPI extra options can be used, as described in the profile below:\n\n```ini\n[pypi]\n\n# The location of a 'requirements.txt' file\n# to use for PyPI project versions\nrequirements-file: requirements-dev.txt\n\n# Inline 'requirements.txt' syntax appended\nrequirements:\n distlib==0.3.5\n sampleproject==3.0.0\n\n# Only version pinning notation is supported ('==')\n# with 'requirements' and 'requirements-file' options\n\n# A map that contains 'module': 'PyPI project' tuples\n# i.e: 'import sample' --> search 'sample' module at 'sampleproject' PyPI Project:\n# https://pypi.org/project/sampleproject/\nproject-names:\n sample: sampleproject\n```\n\nThe PyPI Profiles can be used exactly like all *Named Profiles*:\n\n```python\nwith httpimport.pypi_repo(profile='pypi'):\n import distlib\n import sample\ndistlib.__version__\n# '0.3.5' <-- pinned in the profile 'requirements' option\nsample.__url__\n# 'https://files.pythonhosted.org/packages/ec/a8/5ec62d18adde798d33a170e7f72930357aa69a60839194c93eb0fb05e59c/sampleproject-3.0.0-py3-none-any.whl#sample/__init__.py' <-- loaded from 'sampleproject'\n```\n\nAdditionally, all other options cascade to PyPI profiles, such as HTTPS Proxy (HTTP proxies won't work, as PyPI is hosted with HTTPS), `headers`, etc.\n\n##### NOTE: The values in Profiles MUST NOT be quoted (`'`,`\"`)\n\n### Profile Creation\nProfiles can be provided as INI strings to the `set_profile` function and used in all `httpimport` functions:\n```python\nhttpimport.set_profile(\"\"\"\n[profile1]\n\nproxy-url: https://my-proxy.example.com\nheaders:\n Authorization: Basic ...\n X-Hello-From: httpimport\n X-Some-Other: HTTP header\n\"\"\")\nwith httpimport.remote_repo(\"https://code.example.com\", profile='profile1'):\n import module_accessed_through_proxy\n```\n\n#### Advanced\nProfiles are INI configuration strings parsed using Python [`configparser`](https://docs.python.org/3/library/configparser.html) module.\n\nThe `ConfigParser` object for `httpimport` is the global variable `httpimport.CONFIG` and can be used freely:\n\n```python\nimport httpimport\nhttpimport.CONFIG.read('github.ini') # Read profiles from a file\n\nwith httpimport.github_repo('operatorequals','httpimport-private-test', profile='github'):\n import secret_module\n```\n\n## Default Profiles\nThe `httpimport` module automatically loads Profiles found in `$HOME/.httpimport.ini` and under the `$HOME/.httpimport/` directory. Profiles under `$HOME/.httpimport/` override ones found in `$HOME/.httpimport.ini`.\n\n### Profile Options:\n#### Supported\nHTTP options\n* `zip-password` - `v1.0.0`\n* `proxy-url` - `v1.0.0`\n* `headers` - `v1.0.0`\n* `allow-plaintext` - `v1.0.0`\n* `ca-verify` - `v1.3.0`\n* `ca-file` - `v1.3.0`\n\nPyPI-only options\n* `project-names` - `v1.2.0`\n* `requirements` - `v1.2.0`\n* `requirements-file` - `v1.2.0`\n\n#### Not yet (subject to change)\n* `allow-compiled`\n\n* `auth`\n* `auth-type`\n\n* `tls-cert`\n* `tls-key`\n* `tls-passphrase`\n\n## Debugging...\n```python\nimport httpimport\nimport logging\n\nlogging.getLogger('httpimport').setLevel(logging.DEBUG)\n```\n\n## Beware: **Huge Security Implications!**\n_Using the `httpimport` with plain **HTTP URLs** is highly discouraged_\n \nAs HTTP traffic can be read and changed from all intermediate hosts (_unlike HTTPS_), it is possible for a remote adversary to alter the HTTP responses consumed by `httpimport` and add arbitrary _Python_ code to the downloaded _packages/modules_. This directly results in arbitrary _Remote Code Execution_ on your current user's context of your host!\n\nIn other words, using plain HTTP through the Internet can compromise your host *without a way to notice it*.\n\n##### You have been warned! Use only **HTTPS URLs** with `httpimport`!\n\n## Contributors\n* [ldsink](https://github.com/ldsink) - The `RELOAD` flag and Bug Fixes\n* [lavvy](https://github.com/lavvy) - the `load()` function\n* [superloach](https://github.com/superloach) - Deprecation of `imp` module in Python3 in favour of `importlib`\n* [yanliakos](https://github.com/yanliakos) - Bug Fix\n* [rkbennett](https://github.com/rkbennett) - Relative Imports fix, Proxy support\n\n## Donations\nIn case my work helped you, you can always buy me a beer or a liter of gas [through the Internet](https://www.buymeacoffee.com/operatorequals) or in case you meet me personally.\n\nIn the second case we can talk about any video of [Internet Historian](https://www.youtube.com/@InternetHistorian) or [Ordinary Things](https://www.youtube.com/@OrdinaryThings), while listening to a [Lofi Girl Playlist](https://www.youtube.com/watch?v=jfKfPfyJRdk), like the citizens of the Internet that we are.\n\n[](https://www.buymeacoffee.com/operatorequals)\n",
"bugtrack_url": null,
"license": "Apache2",
"summary": "Module for remote in-memory Python package/module loading through HTTP",
"version": "1.3.1",
"project_urls": {
"Homepage": "https://github.com/operatorequals/httpimport"
},
"split_keywords": [
"import",
"loader",
"memory",
"http",
"network"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "bb86d9744a971602522d214bc933463d5fddef2716a689129587d29ac9a59f1c",
"md5": "b63c948946387ffd0b38b9e2be47ada8",
"sha256": "b1fdfaaa51892707651dcb895656b6e5274585b0ef524bad8179238c86b14843"
},
"downloads": -1,
"filename": "httpimport-1.3.1-py2.py3-none-any.whl",
"has_sig": false,
"md5_digest": "b63c948946387ffd0b38b9e2be47ada8",
"packagetype": "bdist_wheel",
"python_version": "py2.py3",
"requires_python": null,
"size": 17163,
"upload_time": "2023-07-25T12:15:45",
"upload_time_iso_8601": "2023-07-25T12:15:45.711243Z",
"url": "https://files.pythonhosted.org/packages/bb/86/d9744a971602522d214bc933463d5fddef2716a689129587d29ac9a59f1c/httpimport-1.3.1-py2.py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "4d954e3f5586f74e259aa0cc41b41ef340c60e5cd81a681c24e57924e7fbd9be",
"md5": "fc2dd1582f31991426bbd369c99d20f7",
"sha256": "5b3d448404bee70d4ec7b5d5d65d9777c06f15851ed7a8d96af4a0cd0ef9ebb2"
},
"downloads": -1,
"filename": "httpimport-1.3.1.tar.gz",
"has_sig": false,
"md5_digest": "fc2dd1582f31991426bbd369c99d20f7",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 19394,
"upload_time": "2023-07-25T12:15:46",
"upload_time_iso_8601": "2023-07-25T12:15:46.811885Z",
"url": "https://files.pythonhosted.org/packages/4d/95/4e3f5586f74e259aa0cc41b41ef340c60e5cd81a681c24e57924e7fbd9be/httpimport-1.3.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-07-25 12:15:46",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "operatorequals",
"github_project": "httpimport",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "httpimport"
}
JSON
