<div align="center">
<h1>davos</h1>
<img src="https://user-images.githubusercontent.com/26118297/116332586-0c6ce080-a7a0-11eb-94ad-0502c96cf8ef.png" width=250/>
<br>
<br>
<a href="https://github.com/ContextLab/davos/actions/workflows/ci-tests-jupyter.yml">
<img src="https://github.com/ContextLab/davos/actions/workflows/ci-tests-jupyter.yml/badge.svg?branch=main" alt="CI Tests (Jupyter)">
</a>
<!-- <a href="https://github.com/ContextLab/davos/actions/workflows/ci-tests-colab.yml">
<img src="https://github.com/ContextLab/davos/actions/workflows/ci-tests-colab.yml/badge.svg?branch=main&event=push" alt="CI Tests (Colab)">
</a> -->
<a href="https://pypi.org/project/davos/">
<img src="https://img.shields.io/pypi/pyversions/davos?logo=python&logoColor=white" alt="Python Versions">
</a>
<a href="https://pepy.tech/project/davos">
<img src="https://static.pepy.tech/personalized-badge/davos?period=total&units=international_system&left_color=grey&right_color=blue&left_text=Downloads" alt="PyPI Downloads">
</a>
<br>
<a href="https://arxiv.org/abs/2211.15445">
<img src="https://img.shields.io/badge/arXiv-2211.15445-b31b1b" alt="link to arXiv paper">
</a>
<a href="https://www.codefactor.io/repository/github/contextlab/davos">
<img src="https://img.shields.io/codefactor/grade/github/ContextLab/davos/main?logo=codefactor&logoColor=brightgreen" alt="code quality (CodeFactor)">
</a>
<img src="https://img.shields.io/badge/mypy-type%20checked-blue" alt="mypy: checked">
<br>
<a href="https://github.com/ContextLab/davos/blob/main/LICENSE">
<img src="https://img.shields.io/github/license/ContextLab/davos" alt="License: MIT">
</a>
<br>
<br>
</div>
> _Someone once told me that the night is dark and full of terrors. And tonight I am no knight. Tonight I am Davos the
smuggler again. Would that you were an onion._
<div align="right">
—<a href="https://gameofthrones.fandom.com/wiki/Davos_Seaworth">Ser Davos Seaworth</a>
<br>
<a href="https://en.wikipedia.org/wiki/A_Song_of_Ice_and_Fire"><i>A Clash of Kings</i></a> by
<a href="https://en.wikipedia.org/wiki/George_R._R._Martin">George R. R. Martin</a>
<br>
</div>
# Introduction
The `davos` library provides Python with an additional keyword: **`smuggle`**.
[The `smuggle` statement](#the-smuggle-statement) works just like the built-in [`import` statement](https://docs.python.org/3/reference/import.html), with two major differences:
1. You can `smuggle` a package _without installing it first_
2. You can `smuggle` a _specific version_ of a package
## Why would I want an alternative to `import`?
In many cases, `smuggle` and `import` do the same thing—*if you're running code in the same environment you
developed it in*. But what if you want to share a [Jupyter notebook](https://jupyter.org/) containing your code with
someone else? If the user (i.e., the "someone else" in this example) doesn't have all of the packages your notebook
imports, Python will raise an exception and the code won't run. It's not a huge deal, of course, but it's inconvenient
(e.g., the user might need to `pip`-install the missing packages, restart their kernel, re-run the code up to the point
it crashed, etc.—possibly going through this cycle multiple times until the thing finally runs).
A second (and more subtle) issue arises when the developer (i.e., the person who *wrote* the code) used or assumed
different versions of the imported packages than what the user has installed in their environment. So maybe the
original author was developing and testing their code using `pandas` 1.3.5, but the user hasn't upgraded their `pandas`
installation since 0.25.0. Python will happily "`import pandas`" in both cases, but any changes across those versions
might change what the developer's code actually does in the user's (different) environment—or cause it to fail
altogether.
The problem `davos` tries to solve is similar to the idea motivating virtual environments, containers, and virtual
machines: we want a way of replicating the original developer's environment on the user's machine, to a sufficiently
good approximation that we can be "reasonably confident" that the code will continue to behave as expected.
When you `smuggle` packages instead of importing them, it guarantees (for whatever environment the code is running in)
that the packages are importable, even if they hadn't been installed previously. Under the hood, `davos` figures out
whether the package is available, and if not, it uses `pip` to download and install anything that's missing (including
missing dependencies). From that point, after having automatically handled those sorts of dependency issues, `smuggle`
behaves just like `import`.
The second powerful feature of `davos` comes from another construct, called "[_onion comments_](#the-onion-comment)."
These are like standard Python comments, but they appear on the same line(s) as `smuggle` statements, and they are
formatted in a particular way. Onion comments provide a way of precisely controlling how, when, and where packages are
installed, how (or if) the system checks for existing installations, and so on. A key feature is the ability to specify
exactly which version(s) of each package are imported into the current workspace. When used in this way, `davos`
enables authors to guarantee that the same versions of the packages they developed their code with will also be imported
into the user's workspace at the appropriate times.
## Why not use virtual environments, containers, and/or virtual machines instead?
Psst-- we'll let you in on a little secret: importing `davos` *automatically* creates a virtual environment for your notebook. However,
setting up a virtual environment is usually left to the user, `davos` handles the pesky details for you, without you needing to think about them.
Any packages you `smuggle` via `davos` that aren't available in the notebook's original runtime environment are installed into a new virtual environment. This ensures that `davos` will not change the runtime environment (e.g., by installing new packages, changing existing package versions, etc.).
By default, each notebook's virtual environment is stored in a hidden ".davos" folder inside the current user's home directory. The default
environment name is computed to uniquely identify each notebook, according to its filename and path. However, a notebook's virtual environment may
be customized by setting `davos.project` to any string that can be used as a valid folder name in the user's operating system. This is useful for multi-notebook projects that share dependencies (without needing to duplicate each package installation for each notebook).
If you prefer, you can also disable `davos`'s virtual environment infrastructure by setting `davos.project` to `None`. Doing so will cause
any packages installed by `davos` to affect the notebook's runtime environment. This is generally not recommended, as it can lead to unintended
consequences for other code that shares the runtime environment. That said, `davos` also works great when used inside of (standard) virtual environments, containers, and virtual machines.
There are a few additional specific advantages to `davos` that go beyond more typical virtual environments, containers, and/or virtual machines:
- `davos` is very lightweight—importing `davos` into a notebook-based environment unlocks all of its
functionality without needed to install, set up, and learn how to use additional stuff. There is none of the
typical overhead of setting up a new virtual environment (or container, virtual machine, etc.), installing
third-party tools, writing and sharing configuration files, and so on. All of your code *and its dependencies* may
be contained in a single notebook file.
- using onion comments, `davos` can enable mutliple versions of the same package to be used or specified in different
parts of the same notebook. Want to use some deprecated or removed function in `scikit-learn` in one cell, but then
use one of the latest features in another? You can! Just add onion comments specifying which versions of the
package you want to `smuggle` in which cells of your notebook.
## Okay... so how do I use this thing?
To turn a standard [Jupyter (IPython) notebook](https://jupyter.org/), including a [Google Colaboratory notebook](https://colab.research.google.com), into a `davos`-enhanced notebook, just add two lines to the first cell:
```python
%pip install davos
import davos
```
This will enable the `smuggle` keyword in your notebook environment. Then you can do things like:
```python
# pip-install numpy v1.20.2, if needed
smuggle numpy as np # pip: numpy==1.20.2
# the smuggled package is fully imported and usable
arr = np.arange(15).reshape(3, 5)
# and the onion comment guarantees the desired version!
assert np.__version__ == '1.20.2'
```
Interested? Curious? Intrigued? Check out the table of contents for more details! You may also want to check out our [paper](paper/main.pdf) for more formal descriptions and explanations.
## Table of contents
- [Table of contents](#table-of-contents)
- [Installation](#installation)
- [Latest Stable PyPI Release](#latest-stable-pypi-release)
- [Latest GitHub Update](#latest-github-update)
- [Installing in Colaboratory](#installing-in-colaboratory)
- [Overview](#overview)
- [Smuggling Missing Packages](#smuggling-missing-packages)
- [Smuggling Specific Package Versions](#smuggling-specific-package-versions)
- [Use Cases](#use-cases)
- [Simplify sharing reproducible code & Python environments](#simplify-sharing-reproducible-code--python-environments)
- [Guarantee your code always uses the latest version, release, or revision](#guarantee-your-code-always-uses-the-latest-version-release-or-revision)
- [Compare behavior across package versions](#compare-behavior-across-package-versions)
- [Usage](#usage)
- [The `smuggle` Statement](#the-smuggle-statement)
- [Syntax](#smuggle-statement-syntax)
- [Rules](#smuggle-statement-rules)
- [The Onion Comment](#the-onion-comment)
- [Syntax](#onion-comment-syntax)
- [Rules](#onion-comment-rules)
- [The `davos` Config](#the-davos-config)
- [Reference](#config-reference)
- [Top-level Functions](#top-level-functions)
- [How It Works: The `davos` Parser](#how-it-works-the-davos-parser)
- [Additional Notes](#additional-notes)
## Installation
### Latest Stable PyPI Release
[](https://pypi.org/project/davos/)
[]((https://pypi.org/project/davos/))
[]((https://pypi.org/project/davos/))
```sh
pip install davos
```
### Latest GitHub Update
[](https://github.com/ContextLab/davos/releases)
[](https://github.com/ContextLab/davos/commits/main)
[](https://github.com/ContextLab/davos/releases/latest)
```sh
pip install git+https://github.com/ContextLab/davos.git
```
### Installing in Colaboratory
To use `davos` in [Google Colab](https://colab.research.google.com/), add a cell at the top of your notebook with an
percentage sign (`%`) followed by one of the commands above (e.g., `%pip install davos`). Run the cell to install
`davos` on the runtime virtual machine.
**Note**: restarting the Colab runtime does not affect installed packages. However, if the runtime is "factory reset"
or disconnected due to reaching its idle timeout limit, you'll need to rerun the cell to reinstall `davos` on the fresh
VM instance.
## Overview
The primary way to use `davos` is via [the `smuggle` statement](#the-smuggle-statement), which is made available
simply by running `import davos`. Like
[the built-in `import` statement](https://docs.python.org/3/reference/import.html), the `smuggle` statement is used to
load packages, modules, and other objects into the current namespace. The main difference between the two is in how
they handle missing packages and specific package versions.
### Smuggling Missing Packages
`import` requires that packages be installed _before_ the start of the interpreter session. Trying to `import` a package
that can't be found locally will throw a
[`ModuleNotFoundError`](https://docs.python.org/3/library/exceptions.html#ModuleNotFoundError), and you'll have to
install the package from the command line, restart the Python interpreter to make the new package importable, and rerun
your code in full in order to use it.
**The `smuggle` statement, however, can handle missing packages on the fly**. If you `smuggle` a package that isn't
installed locally, `davos` will install it for you, make its contents available to Python's
[import machinery](https://docs.python.org/3/reference/import.html), and load it into the namespace for immediate use.
You can control _how_ `davos` installs missing packages by adding a special type of inline comment called an
["onion" comment](#the-onion-comment) next to a `smuggle` statement.
### Smuggling Specific Package Versions
One simple but powerful use for [onion comments](#the-onion-comment) is making `smuggle` statements _version-sensitive_.
Python doesn't provide a native, viable way to ensure a third-party package imported at runtime matches a specific
version or satisfies a particular [version constraint](https://www.python.org/dev/peps/pep-0440/#version-specifiers).
Many packages expose their version info via a top-level `__version__` attribute (see
[PEP 396](https://www.python.org/dev/peps/pep-0396/)), and certain tools (such as the standard library's
[`importlib.metadata`](https://docs.python.org/3/library/importlib.metadata.html) and
[`setuptools`](https://setuptools.readthedocs.io/en/latest/index.html)'s
[`pkg_resources`](https://setuptools.readthedocs.io/en/latest/pkg_resources.html)) attempt to parse version info from
installed distributions. However, using these to constrain imported package would require writing extra code to compare
version strings and _still_ manually installing the desired version and restarting the interpreter any time an
invalid version is caught.
Additionally, for packages installed through a version control system (e.g., [git](https://git-scm.com/)), this would be
insensitive to differences between revisions (e.g., commits) within the same semantic version.
**`davos` solves these issues** by allowing you to specify a specific version or set of acceptable versions for each
smuggled package. To do this, simply provide a
[version specifier](https://www.python.org/dev/peps/pep-0440/#version-specifiers) in an
[onion comment](#the-onion-comment) next to the `smuggle` statement:
```python
smuggle numpy as np # pip: numpy==1.20.2
from pandas smuggle DataFrame # pip: pandas>=0.23,<1.0
```
In this example, the first line will load [`numpy`](https://numpy.org/) into the local namespace under the alias "`np`",
just as "`import numpy as np`" would. First, `davos` will check whether `numpy` is installed locally, and if so, whether
the installed version _exactly_ matches `1.20.2`. If `numpy` is not installed, or the installed version is anything
other than `1.20.2`, `davos` will use the specified _installer program_, [`pip`](https://pip.pypa.io/en/stable/), to
install `numpy==1.20.2` before loading the package.
Similarly, the second line will load the "`DataFrame`" object from the [`pandas`](https://pandas.pydata.org/) library,
analogously to "`from pandas import DataFrame`". A local `pandas` version of `0.24.1` would be used, but a local version
of `1.0.2` would cause `davos` to replace it with a valid `pandas` version, as if you had manually run `pip install
pandas>=0.23,<1.0`.
In both cases, the imported versions will fit the constraints specified in their [onion comments](#the-onion-comment),
and the next time `numpy` or `pandas` is smuggled with the same constraints, valid local installations will be found.
You can also force the state of a smuggled packages to match a specific VCS ref (branch, revision, tag, release, etc.).
For example:
```python
smuggle hypertools as hyp # pip: git+https://github.com/ContextLab/hypertools.git@564c1d4
```
will load [`hypertools`](https://hypertools.readthedocs.io/en/latest/) (aliased as "`hyp`"), as the package existed
[on GitHub](https://github.com/ContextLab/hypertools), at commit
[98a3d80](https://github.com/ContextLab/hypertools/tree/98a3d80). The general format for VCS references in
[onion comments](#the-onion-comment) follows that of the
[`pip-install`](https://pip.pypa.io/en/stable/topics/vcs-support) command. See the
[notes on smuggling from VCS](#notes-vcs-smuggle) below for additional info.
And with [a few exceptions](#notes-c-extensions), smuggling a specific package version will work _even if the package
has already been imported_!
**Note**: `davos` v0.1 supports [IPython](https://ipython.readthedocs.io/en/stable/) environments (e.g.,
[Jupyter](https://jupyter.org/) and [Colaboratory](https://colab.research.google.com/) notebooks) only. v0.2 will add
support for "regular" (i.e., non-interactive) Python scripts.
### Use Cases
#### Simplify sharing reproducible code & Python environments
Different versions of the same package can often behave quite differently—bugs are introduced and fixed, features
are implemented and removed, support for Python versions is added and dropped, etc. Because of this, Python code that is
meant to be _reproducible_ (e.g., tutorials, demos, data analyses) is commonly shared alongside a set of fixed versions
for each package used. And since there is no Python-native way to specify package versions at runtime (see
[above](#smuggling-specific-package-versions)), this typically takes the form of a pre-configured development
environment the end user must build themselves (e.g., a [Docker](https://www.docker.com/) container or
[conda](https://docs.conda.io/en/latest/) environment), which can be cumbersome, slow to set up, resource-intensive, and
confusing for newer users, as well as require shipping both additional specification files _and_ setup instructions
along with your code. And even then, a well-intentioned user may alter the environment in a way that affects your
carefully curated set of pinned packages (such as installing additional packages that trigger dependency updates).
Instead, `davos` allows you to share code with one simple instruction: _just `pip install davos`!_ Replace your `import`
statements with `smuggle` statements, pin package versions in onion comments, and let `davos` take care of the rest.
Beyond its simplicity, this approach ensures your predetermined package versions are in place every time your code is
run.
#### Guarantee your code always uses the latest version, release, or revision
If you want to make sure you're always using the most recent release of a certain package, `davos` makes doing so easy:
```python
smuggle mypkg # pip: mypkg --upgrade
```
Or if you have an automation designed to test your most recent commit on GitHub:
```python
smuggle mypkg # pip: git+https://username/reponame.git
```
#### Compare behavior across package versions
The ability to `smuggle` a specific package version even after a different version has been imported makes `davos` a
useful tool for comparing behavior across multiple versions of the same package, within the same interpreter session:
```python
def test_my_func_unchanged():
"""Regression test for `mypkg.my_func()`"""
data = list(range(10))
smuggle mypkg # pip: mypkg==0.1
result1 = mypkg.my_func(data)
smuggle mypkg # pip: mypkg==0.2
result2 = mypkg.my_func(data)
smuggle mypkg # pip: git+https://github.com/MyOrg/mypkg.git
result3 = mypkg.my_func(data)
assert result1 == result2 == result3
```
## Usage
### The `smuggle` Statement
#### <a name="smuggle-statement-syntax"></a>Syntax
The `smuggle` statement is meant to be used in place of
[the built-in `import` statement](https://docs.python.org/3/reference/import.html) and shares
[its full syntactic definition](https://docs.python.org/3/reference/simple_stmts.html#the-import-statement):
```ebnf
smuggle_stmt ::= "smuggle" module ["as" identifier] ("," module ["as" identifier])*
| "from" relative_module "smuggle" identifier ["as" identifier]
("," identifier ["as" identifier])*
| "from" relative_module "smuggle" "(" identifier ["as" identifier]
("," identifier ["as" identifier])* [","] ")"
| "from" module "smuggle" "*"
module ::= (identifier ".")* identifier
relative_module ::= "."* module | "."+
```
<sup>
<i>
NB: uses the modified BNF grammar notation described in
<a href="https://docs.python.org/3/reference">The Python Language Reference</a>,
<a href="https://docs.python.org/3/reference/introduction.html#notation">here</a>; see
<a href="https://docs.python.org/3/reference/lexical_analysis.html#identifiers">here</a> for the lexical definition
of <code>identifier</code>
</i>
</sup>
In simpler terms, **any valid syntax for `import` is also valid for `smuggle`**.
#### <a name="smuggle-statement-rules"></a>Rules
- Like `import` statements, `smuggle` statements are whitespace-insensitive, unless a lack of whitespace between two
tokens would cause them to be interpreted as a different token:
```python
from os.path smuggle dirname, join as opj # valid
from os . path smuggle dirname ,join as opj # also valid
from os.path smuggle dirname, join asopj # invalid ("asopj" != "as opj")
```
- Any context that would cause an `import` statement _not_ to be executed will have the same effect on a `smuggle`
statement:
```python
# smuggle matplotlib.pyplot as plt # not executed
print('smuggle matplotlib.pyplot as plt') # not executed
foo = """
smuggle matplotlib.pyplot as plt""" # not executed
```
- Because the `davos` parser is less complex than the full Python parser, there are two fairly non-disruptive edge
cases where an `import` statement would be syntactically valid but a `smuggle` statement would not:
1. The [exec](https://docs.python.org/3.8/library/functions.html#exec) function
```python
exec('from pathlib import Path') # executed
exec('from pathlib smuggle Path') # raises SyntaxError
```
2. A one-line [compound statement](https://docs.python.org/3.9/reference/compound_stmts.html#compound-statements)
clause:
```python
if True: import random # executed
if True: smuggle random # raises SyntaxError
while True: import math; break # executed
while True: smuggle math; break # raises SyntaxError
for _ in range(1): import json # executed
for _ in range(1): smuggle json # raises SyntaxError
# etc...
```
- In [IPython](https://ipython.readthedocs.io/en/stable/) environments (e.g., [Jupyter](https://jupyter.org/) &
[Colaboratory](https://colab.research.google.com/) notebooks) `smuggle` statements always load names into the global
namespace:
```python
# example.ipynb
import davos
def import_example():
import datetime
def smuggle_example():
smuggle datetime
import_example()
type(datetime) # raises NameError
smuggle_example()
type(datetime) # returns
```
### The Onion Comment
An _onion comment_ is a special type of inline comment placed on a line containing a `smuggle` statement. Onion comments
can be used to control how `davos`:
1. determines whether the smuggled package should be installed
2. installs the smuggled package, if necessary
Onion comments are also useful when smuggling a package whose _distribution name_ (i.e., the name
used when installing it) is different from its _top-level module name_ (i.e., the name used when importing it). Take for
example:
```python
from sklearn.decomposition smuggle pca # pip: scikit-learn
```
The onion comment here (`# pip: scikit-learn`) tells `davos` that if "`sklearn`" does not exist
locally, the "`scikit-learn`" package should be installed.
#### <a name="onion-comment-syntax"></a>Syntax
Onion comments follow a simple but specific syntax, inspired in part by the
[type comment syntax](https://www.python.org/dev/peps/pep-0484/#type-comments) introduced in
[PEP 484](https://www.python.org/dev/peps/pep-0484). The following is a loose (pseudo-)syntactic definition for an onion
comment:
```ebnf
onion_comment ::= "#" installer ":" install_opt* pkg_spec install_opt*
installer ::= ("pip" | "conda")
pkg_spec ::= identifier [version_spec]
```
<sup>
<i>
NB: uses the modified BNF grammar notation described in
<a href="https://docs.python.org/3/reference">The Python Language Reference</a>,
<a href="https://docs.python.org/3/reference/introduction.html#notation">here</a>; see
<a href="https://docs.python.org/3/reference/lexical_analysis.html#identifiers">here</a> for the lexical definition
of <code>identifier</code>
</i>
</sup>
where `installer` is the program used to install the package; `install_opt` is any option accepted by the installer's
"`install`" command; and `version_spec` may be a
[version specifier](https://www.python.org/dev/peps/pep-0440/#version-specifiers) defined by
[PEP 440](https://www.python.org/dev/peps/pep-0440) followed by a
[version string](https://www.python.org/dev/peps/pep-0440/#public-version-identifiers), or an alternative syntax valid
for the given `installer` program. For example, [`pip`](https://pip.pypa.io/en/stable/) uses specific syntaxes for
[local](https://pip.pypa.io/en/stable/cli/pip_install/#local-project-installs),
[editable](https://pip.pypa.io/en/stable/cli/pip_install/#editable-installs), and
[VCS-based](https://pip.pypa.io/en/stable/topics/vcs-support) installation.
Less formally, **an onion comment simply consists of two parts, separated by a colon**:
1. the name of the installer program (e.g., [`pip`](https://pip.pypa.io/en/stable/))
2. arguments passed to the program's "install" command
Thus, you can essentially think of writing an onion comment as taking the full shell command you would run to install
the package, and replacing "_install_" with "_:_". For instance, the command:
```sh
pip install -I --no-cache-dir numpy==1.20.2 -vvv --timeout 30
```
is easily translated into an onion comment as:
```python
smuggle numpy # pip: -I --no-cache-dir numpy==1.20.2 -vvv --timeout 30
```
In practice, onion comments are identified as matches for the
[regular expression](https://en.wikipedia.org/wiki/Regular_expression):
```regex
#+ *(?:pip|conda) *: *[^#\n ].+?(?= +#| *\n| *$)
```
<sup>
<i>
NB: support for installing <code>smuggle</code>d packages via
<a href="https://docs.conda.io/en/latest/"><code>conda</code></a> will be added in v0.2. For v0.1,
"<code>pip</code>" should be used exclusively.
</i>
</sup>
**Note**: support for installing smuggled packages via the [`conda`](https://docs.conda.io/en/latest/) package manager
will be added in v0.2. For v0.1, onion comments should always specify "`pip`" as the `installer` program.
#### <a name="onion-comment-rules"></a>Rules
- An onion comment must be placed on the same line as a `smuggle` statement; otherwise, it is not parsed:
```python
# assuming the dateutil package is not installed...
# pip: python-dateutil # <-- has no effect
smuggle dateutil # raises InstallerError (no "dateutil" package exists)
smuggle dateutil # raises InstallerError (no "dateutil" package exists)
# pip: python-dateutil # <-- has no effect
smuggle dateutil # pip: python-dateutil # installs "python-dateutil" package, if necessary
```
- An onion comment may be followed by unrelated inline comments as long as they are separated by at least one space:
```python
smuggle tqdm # pip: tqdm>=4.46,<4.60 # this comment is ignored
smuggle tqdm # pip: tqdm>=4.46,<4.60 # so is this one
smuggle tqdm # pip: tqdm>=4.46,<4.60# but this comment raises OnionArgumentError
```
- An onion comment must be the first inline comment immediately following a `smuggle` statement; otherwise, it is not
parsed:
```python
smuggle numpy # pip: numpy!=1.19.1 # <-- guarantees smuggled version is *not* v1.19.1
smuggle numpy # has no effect --> # pip: numpy==1.19.1
```
This also allows you to easily "comment out" onion comments:
```python
smuggle numpy ## pip: numpy!=1.19.1 # <-- has no effect
```
- Onion comments are generally whitespace-insensitive, but installer arguments must be separated by at least one space:
```python
from umap smuggle UMAP # pip: umap-learn --user -v --no-clean # valid
from umap smuggle UMAP#pip:umap-learn --user -v --no-clean # also valid
from umap smuggle UMAP # pip: umap-learn --user-v--no-clean # raises OnionArgumentError
```
- Onion comments have no effect on standard library modules:
```python
smuggle threading # pip: threading==9999 # <-- has no effect
```
- When smuggling multiple packages with a _single_ `smuggle` statement, an onion comment may be used to refer to the
**first** package listed:
```python
smuggle nilearn, nibabel, nltools # pip: nilearn==0.7.1
```
- If multiple _separate_ `smuggle` statements are placed on a single line, an onion comment may be used to refer to the
**last** statement:
```python
smuggle gensim; smuggle spacy; smuggle nltk # pip: nltk~=3.5 --pre
```
- For multiline `smuggle` statements, an onion comment may be placed on the first line:
```python
from scipy.interpolate smuggle ( # pip: scipy==1.6.3
interp1d,
interpn as interp_ndgrid,
LinearNDInterpolator,
NearestNDInterpolator,
)
```
... or on the last line:
```python
from scipy.interpolate smuggle (interp1d, # this comment has no effect
interpn as interp_ndgrid,
LinearNDInterpolator,
NearestNDInterpolator) # pip: scipy==1.6.3
```
... though the first line takes priority:
```python
from scipy.interpolate smuggle ( # pip: scipy==1.6.3 # <-- this version is installed
interp1d,
interpn as interp_ndgrid,
LinearNDInterpolator,
NearestNDInterpolator,
) # pip: scipy==1.6.2 # <-- this comment is ignored
```
... and all comments _not_ on the first or last line are ignored:
```python
from scipy.interpolate smuggle (
interp1d, # pip: scipy==1.6.3 # <-- ignored
interpn as interp_ndgrid,
LinearNDInterpolator, # unrelated comment # <-- ignored
NearestNDInterpolator
) # pip: scipy==1.6.2 # <-- parsed
```
- The onion comment is intended to describe how a specific smuggled package should be installed if it is not found
locally, in order to make it available for immediate use. Therefore, installer options that either (A) install
packages other than the smuggled package and its dependencies (e.g., from a specification file), or (B) cause the
smuggled package not to be installed, are disallowed. The options listed below will raise an `OnionArgumentError`:
- `-h`, `--help`
- `-r`, `--requirement`
- `-V`, `--version`
### The `davos` Config
The `davos` config object stores options and data that affect how `davos` behaves. After importing `davos`, the config
instance (a singleton) for the current session is available as `davos.config`, and its various fields are accessible as
attributes. The config object exposes a mixture of writable and read-only fields. Most `davos.config` attributes can be
assigned values to control aspects of `davos` behavior, while others are available for inspection but are set and used
internally. Additionally, certain config fields may be writable in some situations but not others (e.g. only if the
importing environment supports a particular feature). Once set, `davos` config options last for the lifetime of the
interpreter (unless updated); however, they do *not* persist across interpreter sessions. A full list of `davos` config
fields is available [below](#config-reference):
#### <a name="config-reference"></a>Reference
| Field | Description | Type | Default | Writable? |
| :---: | --- | :---: | :---: | :---: |
| `active` | Whether or not the `davos` parser should be run on subsequent input (cells, in Jupyter/Colab notebooks). Setting to `True` activates the `davos` parser, enables the `smuggle` keyword, and injects the `smuggle()` function into the user namespace. Setting to `False` deactivates the `davos` parser, disables the `smuggle` keyword, and removes "`smuggle`" from the user namespace (if it holds a reference to the `smuggle()` function). See [How it Works](#how-it-works) for more info. | `bool` | `True` | ✅ |
| `auto_rerun` | If `True`, when smuggling a previously-imported package that cannot be reloaded (see [Smuggling packages with C-extensions](#notes-c-extensions)), `davos` will automatically restart the interpreter and rerun all code up to (and including) the current `smuggle` statement. Otherwise, issues a warning and prompts the user with buttons to either restart/rerun or continue running. | `bool` | `False` | ✅ (**Jupyter notebooks only**) |
| `confirm_install` | Whether or not `davos` should require user confirmation (`[y/n]` input) before installing a smuggled package | `bool` | `False` | ✅ |
| `environment` | A label describing the environment into which `davos` was running. Checked internally to determine which interchangeable implementation functions are used, whether certain config fields are writable, and various other behaviors | `Literal['Python', 'IPython<7.0', 'IPython>=7.0', 'Colaboratory']` | N/A | ❌ |
| `ipython_shell` | The global IPython interactive shell instance | [`IPython.core`<br>`.interactiveshell`<br>`.InteractiveShell`](https://ipython.readthedocs.io/en/stable/api/generated/IPython.core.interactiveshell.html#IPython.core.interactiveshell.InteractiveShell) | N/A | ❌ |
| `noninteractive` | Set to `True` to run `davos` in non-interactive mode (all user input and confirmation will be disabled). **NB**:<br>1. Setting to `True` disables `confirm_install` if previously enabled <br>2. If `auto_rerun` is `False` in non-interactive mode, `davos` will throw an error if a smuggled package cannot be reloaded | `bool` | `False` | ✅ (**Jupyter notebooks only**) |
| `pip_executable` | The path to the `pip` executable used to install smuggled packages. Must be a path (`str` or [`pathlib.Path`](https://docs.python.org/3/library/pathlib.html#pathlib.Path)) to a real file. Default is programmatically determined from Python environment; falls back to `sys.executable -m pip` if executable can't be found | `str` | `pip` exe path or `sys.executable -m pip` | ✅ |
| `smuggled` | A cache of packages smuggled during the current interpreter session. Formatted as a `dict` whose keys are package names and values are the (`.split()` and `';'.join()`ed) onion comments. Implemented this way so that any non-whitespace change to installer arguments re-installation | `dict[str, str]` | `{}` | ❌ |
| `suppress_stdout` | If `True`, suppress all unnecessary output issued by both `davos` and the installer program. Useful when smuggling packages that need to install many dependencies and therefore generate extensive output. If the installer program throws an error while output is suppressed, both stdout & stderr will be shown with the traceback | `bool` | `False` | ✅ |
#### <a name="top-level-functions"></a>Top-level Functions
`davos` also provides a few convenience for reading/setting config values:
- **`davos.activate()`**
Activate the `davos` parser, enable the `smuggle` keyword, and inject the `smuggle()` function into the namespace.
Equivalent to setting `davos.config.active = True`. See [How it Works](#how-it-works) for more info.
- **`davos.deactivate()`**
Deactivate the `davos` parser, disable the `smuggle` keyword, and remove the name `smuggle` from the namespace if (and
only if) it refers to the `smuggle()` function. If `smuggle` has been overwritten with a different value, the variable
will not be deleted. Equivalent to setting `davos.config.active = False`. See [How it Works](#how-it-works) for more
- info.
- **`davos.is_active()`**
Return the current value of `davos.config.active`.
- **`davos.configure(**kwargs)`**
Set multiple `davos.config` fields at once by passing values as keyword arguments, e.g.:
```python
import davos
davos.configure(active=False, noninteractive=True, pip_executable='/usr/bin/pip3')
```
is equivalent to:
```python
import davos
davos.active = False
davos.noninteractive = True
davos.pip_executable = '/usr/bin/pip3'
```
## How It Works: The `davos` Parser
Functionally, importing `davos` appears to enable a new Python keyword, "_`smuggle`_". However, `davos` doesn't actually
modify the rules or [reserved keywords](https://docs.python.org/3/reference/lexical_analysis.html#keywords) used by
Python's parser and lexical analyzer in order to do so—in fact, modifying the Python grammar is not possible at
runtime and would require rebuilding the interpreter. Instead, in [IPython](https://ipython.readthedocs.io/en/stable/)
enivonments like [Jupyter](https://jupyter.org/) and
[Colaboratory](https://colab.research.google.com/notebooks/intro.ipynb) notebooks, `davos` implements the `smuggle`
keyword via a combination of namespace injections and its own (far simpler) custom parser.
The `smuggle` keyword can be enabled and disabled at will by "activating" and "deactivating" `davos` (see the
[`davos` Config Reference](config-reference) and [Top-level Functions](#top-level-functions), above). When `davos` is
imported, it is automatically activated by default. Activating `davos` triggers two things:
1. The _`smuggle()` function_ is injected into the `IPython` user namespace
2. The _`davos` parser_ is registered as a
[custom input transformer](https://ipython.readthedocs.io/en/stable/config/inputtransforms.html)
IPython preprocesses all executed code as plain text before it is sent to the Python parser in order to handle
special constructs like [`%magic`](https://ipython.readthedocs.io/en/stable/interactive/magics.html) and
[`!shell`](https://ipython.readthedocs.io/en/stable/interactive/reference.html#system-shell-access) commands. `davos`
hooks into this process to transform `smuggle` statements into syntactically valid Python code. The `davos`
parser uses [this regular expression](https://github.com/ContextLab/davos/blob/main/davos/core/regexps.py) to match each
line of code containing a `smuggle` statement (and, optionally, an onion comment), extracts information from its text,
and replaces it with an analogous call to the _`smuggle()` function_. Thus, even though the code visible to the user may
contain `smuggle` statements, e.g.:
```python
smuggle numpy as np # pip: numpy>1.16,<=1.20 -vv
```
the code that is actually executed by the Python interpreter will not:
```python
smuggle(name="numpy", as_="np", installer="pip", args_str="""numpy>1.16,<=1.20 -vv""", installer_kwargs={'editable': False, 'spec': 'numpy>1.16,<=1.20', 'verbosity': 2})
```
The `davos` parser can be deactivated at any time, and doing so triggers the opposite actions of activating it:
1. The name "`smuggle`" is deleted from the `IPython` user namespace, *unless it has been overwritten and no longer
refers to the `smuggle()` function*
2. The `davos` parser input transformer is deregistered.
**Note**: in Jupyter and Colaboratory notebooks, IPython parses and transforms all text in a cell before sending it
to the kernel for execution. This means that importing or activating `davos` will not make the `smuggle` statement
available until the _next_ cell, because all lines in the current cell were transformed before the `davos` parser was
registered. However, _deactivating_ `davos` disables the `smuggle` statement immediately—although the `davos`
parser will have already replaced all `smuggle` statements with `smuggle()` function calls, removing the function from
the namespace causes them to throw `NameError`.
## Additional Notes
- <a name="notes-reimplement-cli"></a>**Reimplementing installer programs' CLI parsers**
The `davos` parser extracts info from onion comments by passing them to a (slightly modified) reimplementation of
their specified installer program's CLI parser. This is somewhat redundant, since the arguments will eventually be
re-parsed by the _actual_ installer program if the package needs to be installed. However, it affords a number of
advantages, such as:
- detecting errors early during the parser phase, before spending any time running code above the line containing the
`smuggle` statement
- preventing shell injections in onion comments—e.g., `#pip: --upgrade numpy && rm -rf /` fails due to the
`OnionParser`, but would otherwise execute successfully.
- allowing certain installer arguments to temporarily influence `davos` behavior while smuggling the current package
(see [Installer options that affect `davos` behavior](#notes-installer-opts) below for specific info)
- <a name="notes-installer-opts"></a>**Installer options that affect `davos` behavior**
Passing certain options to the installer program via an [onion comment](#the-onion-comment) will also affect the
corresponding `smuggle` statement in a predictable way:
- [**`--force-reinstall`**](https://pip.pypa.io/en/stable/cli/pip_install/#cmdoption-force-reinstall) |
[**`-I`, `--ignore-installed`**](https://pip.pypa.io/en/stable/cli/pip_install/#cmdoption-I) |
[**`-U`, `--upgrade`**](https://pip.pypa.io/en/stable/cli/pip_install/#cmdoption-I)
The package will be installed, even if it exists locally
- [**`--no-input`**](https://pip.pypa.io/en/stable/cli/pip/#cmdoption-no-input)
Disables input prompts, analogous to temporarily setting `davos.config.noninteractive` to `True`. Overrides value
of `davos.config.confirm_install`.
- [**`--src <dir>`**](https://pip.pypa.io/en/stable/cli/pip_install/#cmdoption-src) |
[**`-t`, `--target <dir>`**](https://pip.pypa.io/en/stable/cli/pip_install/#cmdoption-t)
Prepends `<dir>` to [`sys.path`](https://docs.python.org/3/library/sys.html#sys.path) if not already present so
the package can be imported.
- <a name="notes-c-extensions"></a>**Smuggling packages with C-extensions**
Some Python packages that rely heavily on custom data types implemented via
[C-extensions](https://docs.python.org/3.9/extending/extending.html) (e.g., `numpy`, `pandas`) dynamically generate
modules defining various C functions and data structures, and link them to the Python interpreter when they are first
imported. Depending on how these objects are initialized, they may not be subject to normal garbage collection, and
persist despite their reference count dropping to zero. This can lead to unexpected errors when reloading the Python
module that creates them, particularly if their dynamically generated source code has been changed (e.g., because the
reloaded package is a newer version).
This can occasionally affect `davos`'s ability to `smuggle` a new version of a package (or dependency) that was
previously imported. To handle this, `davos` first checks each package it installs against
[`sys.modules`](https://docs.python.org/3.9/library/sys.html#sys.modules). If a different version has already been
loaded by the interpreter, `davos` will attempt to replace it with the requested version. If this fails, `davos` will
restore the old package version _in memory_, while replacing it with the new package version _on disk_. This allows
subsequent code that uses the non-reloadable module to still execute in most cases, while dependency checks for other
packages run against the updated version. Then, depending on the value of `davos.config.auto_rerun`, `davos` will
either either automatically restart the interpreter to load the updated package, prompt you to do so, or raise an
exception.
- <a name="notes-from-reload"></a>**_`from` ... `import` ..._ statements and reloading modules**
The Python docs for [`importlib.reload()`](https://docs.python.org/3/library/importlib.html#importlib.reload) include
the following caveat:
> If a module imports objects from another module using
> [`from`](https://docs.python.org/3/reference/simple_stmts.html#from) …
> [`import`](https://docs.python.org/3/reference/simple_stmts.html#import) …, calling
> [`reload()`](https://docs.python.org/3/library/importlib.html#importlib.reload) for the other module does
> not redefine the objects imported from it — one way around this is to re-execute the `from` statement, another is to
> use `import` and qualified names (_module.name_) instead.
The same applies to smuggling packages or modules from which objects have already been loaded. If object _`name`_ from
module _`module`_ was loaded using either _`from module import name`_ or _`from module smuggle name`_, subsequently
running _`smuggle module # pip --upgrade`_ will in fact install and load an upgraded version of _`module`_, but the
the _`name`_ object will still be that of the old version! To fix this, you can simply run _`from module smuggle
name`_ either instead in lieu of or after _`smuggle module`_.
- <a name="notes-vcs-smuggle"></a>**Smuggling packages from version control systems**
The first time during an interpreter session that a given package is installed from a VCS URL, it is assumed not to be
present locally, and is therefore freshly installed. `pip` clones non-editable VCS repositories into a temporary
directory, runs `setup.py install`, and then immediately deletes them. Since no information is retained about the
state of the repository at installation, it is impossible to determine whether an existing package satisfies the state
(i.e., branch, tag, commit hash, etc.) requested for smuggled package.
[comment]: <> (- As with _all_ code, you should use caution when running Python code containing `smuggle` statements that was not written by you or someone you know. )
Raw data
{
"_id": null,
"home_page": "https://github.com/ContextLab/davos",
"name": "davos",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.6",
"maintainer_email": "",
"keywords": "import install package module automatic davos smuggle pip conda",
"author": "Paxton Fitzpatrick, Jeremy Manning",
"author_email": "contextualdynamics@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/91/de/bcc0620b8bcf16eade8af9f8b28cfae11621de1a457d361aacdf799e5ea8/davos-0.2.2.tar.gz",
"platform": null,
"description": "<div align=\"center\">\n <h1>davos</h1>\n <img src=\"https://user-images.githubusercontent.com/26118297/116332586-0c6ce080-a7a0-11eb-94ad-0502c96cf8ef.png\" width=250/>\n <br>\n <br>\n <a href=\"https://github.com/ContextLab/davos/actions/workflows/ci-tests-jupyter.yml\">\n <img src=\"https://github.com/ContextLab/davos/actions/workflows/ci-tests-jupyter.yml/badge.svg?branch=main\" alt=\"CI Tests (Jupyter)\">\n </a>\n <!-- <a href=\"https://github.com/ContextLab/davos/actions/workflows/ci-tests-colab.yml\">\n <img src=\"https://github.com/ContextLab/davos/actions/workflows/ci-tests-colab.yml/badge.svg?branch=main&event=push\" alt=\"CI Tests (Colab)\">\n </a> -->\n <a href=\"https://pypi.org/project/davos/\">\n <img src=\"https://img.shields.io/pypi/pyversions/davos?logo=python&logoColor=white\" alt=\"Python Versions\">\n </a>\n <a href=\"https://pepy.tech/project/davos\">\n <img src=\"https://static.pepy.tech/personalized-badge/davos?period=total&units=international_system&left_color=grey&right_color=blue&left_text=Downloads\" alt=\"PyPI Downloads\">\n </a>\n <br>\n <a href=\"https://arxiv.org/abs/2211.15445\">\n <img src=\"https://img.shields.io/badge/arXiv-2211.15445-b31b1b\" alt=\"link to arXiv paper\">\n </a>\n <a href=\"https://www.codefactor.io/repository/github/contextlab/davos\">\n <img src=\"https://img.shields.io/codefactor/grade/github/ContextLab/davos/main?logo=codefactor&logoColor=brightgreen\" alt=\"code quality (CodeFactor)\">\n </a>\n <img src=\"https://img.shields.io/badge/mypy-type%20checked-blue\" alt=\"mypy: checked\">\n <br>\n <a href=\"https://github.com/ContextLab/davos/blob/main/LICENSE\">\n <img src=\"https://img.shields.io/github/license/ContextLab/davos\" alt=\"License: MIT\">\n </a>\n <br>\n <br>\n</div>\n\n> _Someone once told me that the night is dark and full of terrors. And tonight I am no knight. Tonight I am Davos the\nsmuggler again. Would that you were an onion._\n<div align=\"right\">\n —<a href=\"https://gameofthrones.fandom.com/wiki/Davos_Seaworth\">Ser Davos Seaworth</a>\n <br>\n <a href=\"https://en.wikipedia.org/wiki/A_Song_of_Ice_and_Fire\"><i>A Clash of Kings</i></a> by\n <a href=\"https://en.wikipedia.org/wiki/George_R._R._Martin\">George R. R. Martin</a>\n <br>\n</div>\n\n# Introduction\n\nThe `davos` library provides Python with an additional keyword: **`smuggle`**.\n\n[The `smuggle` statement](#the-smuggle-statement) works just like the built-in [`import` statement](https://docs.python.org/3/reference/import.html), with two major differences: \n1. You can `smuggle` a package _without installing it first_\n2. You can `smuggle` a _specific version_ of a package\n\n## Why would I want an alternative to `import`?\n\nIn many cases, `smuggle` and `import` do the same thing—*if you're running code in the same environment you\ndeveloped it in*. But what if you want to share a [Jupyter notebook](https://jupyter.org/) containing your code with\nsomeone else? If the user (i.e., the \"someone else\" in this example) doesn't have all of the packages your notebook\nimports, Python will raise an exception and the code won't run. It's not a huge deal, of course, but it's inconvenient\n(e.g., the user might need to `pip`-install the missing packages, restart their kernel, re-run the code up to the point\nit crashed, etc.—possibly going through this cycle multiple times until the thing finally runs). \n\nA second (and more subtle) issue arises when the developer (i.e., the person who *wrote* the code) used or assumed\ndifferent versions of the imported packages than what the user has installed in their environment. So maybe the\noriginal author was developing and testing their code using `pandas` 1.3.5, but the user hasn't upgraded their `pandas`\ninstallation since 0.25.0. Python will happily \"`import pandas`\" in both cases, but any changes across those versions\nmight change what the developer's code actually does in the user's (different) environment—or cause it to fail\naltogether.\n\nThe problem `davos` tries to solve is similar to the idea motivating virtual environments, containers, and virtual\nmachines: we want a way of replicating the original developer's environment on the user's machine, to a sufficiently\ngood approximation that we can be \"reasonably confident\" that the code will continue to behave as expected.\n\nWhen you `smuggle` packages instead of importing them, it guarantees (for whatever environment the code is running in)\nthat the packages are importable, even if they hadn't been installed previously. Under the hood, `davos` figures out\nwhether the package is available, and if not, it uses `pip` to download and install anything that's missing (including\nmissing dependencies). From that point, after having automatically handled those sorts of dependency issues, `smuggle`\nbehaves just like `import`.\n\nThe second powerful feature of `davos` comes from another construct, called \"[_onion comments_](#the-onion-comment).\"\nThese are like standard Python comments, but they appear on the same line(s) as `smuggle` statements, and they are\nformatted in a particular way. Onion comments provide a way of precisely controlling how, when, and where packages are\ninstalled, how (or if) the system checks for existing installations, and so on. A key feature is the ability to specify\nexactly which version(s) of each package are imported into the current workspace. When used in this way, `davos`\nenables authors to guarantee that the same versions of the packages they developed their code with will also be imported\ninto the user's workspace at the appropriate times.\n\n## Why not use virtual environments, containers, and/or virtual machines instead?\n\nPsst-- we'll let you in on a little secret: importing `davos` *automatically* creates a virtual environment for your notebook. However,\nsetting up a virtual environment is usually left to the user, `davos` handles the pesky details for you, without you needing to think about them.\nAny packages you `smuggle` via `davos` that aren't available in the notebook's original runtime environment are installed into a new virtual environment. This ensures that `davos` will not change the runtime environment (e.g., by installing new packages, changing existing package versions, etc.).\n\nBy default, each notebook's virtual environment is stored in a hidden \".davos\" folder inside the current user's home directory. The default\nenvironment name is computed to uniquely identify each notebook, according to its filename and path. However, a notebook's virtual environment may\nbe customized by setting `davos.project` to any string that can be used as a valid folder name in the user's operating system. This is useful for multi-notebook projects that share dependencies (without needing to duplicate each package installation for each notebook).\n\nIf you prefer, you can also disable `davos`'s virtual environment infrastructure by setting `davos.project` to `None`. Doing so will cause\nany packages installed by `davos` to affect the notebook's runtime environment. This is generally not recommended, as it can lead to unintended\nconsequences for other code that shares the runtime environment. That said, `davos` also works great when used inside of (standard) virtual environments, containers, and virtual machines.\n\nThere are a few additional specific advantages to `davos` that go beyond more typical virtual environments, containers, and/or virtual machines:\n - `davos` is very lightweight—importing `davos` into a notebook-based environment unlocks all of its\n functionality without needed to install, set up, and learn how to use additional stuff. There is none of the\n typical overhead of setting up a new virtual environment (or container, virtual machine, etc.), installing\n third-party tools, writing and sharing configuration files, and so on. All of your code *and its dependencies* may\n be contained in a single notebook file.\n - using onion comments, `davos` can enable mutliple versions of the same package to be used or specified in different\n parts of the same notebook. Want to use some deprecated or removed function in `scikit-learn` in one cell, but then\n use one of the latest features in another? You can! Just add onion comments specifying which versions of the\n package you want to `smuggle` in which cells of your notebook.\n\n## Okay... so how do I use this thing?\n\nTo turn a standard [Jupyter (IPython) notebook](https://jupyter.org/), including a [Google Colaboratory notebook](https://colab.research.google.com), into a `davos`-enhanced notebook, just add two lines to the first cell:\n```python\n%pip install davos\nimport davos\n```\n\nThis will enable the `smuggle` keyword in your notebook environment. Then you can do things like:\n\n```python\n# pip-install numpy v1.20.2, if needed\nsmuggle numpy as np # pip: numpy==1.20.2\n\n# the smuggled package is fully imported and usable\narr = np.arange(15).reshape(3, 5)\n\n# and the onion comment guarantees the desired version!\nassert np.__version__ == '1.20.2'\n```\n\nInterested? Curious? Intrigued? Check out the table of contents for more details! You may also want to check out our [paper](paper/main.pdf) for more formal descriptions and explanations.\n\n## Table of contents\n- [Table of contents](#table-of-contents)\n- [Installation](#installation)\n - [Latest Stable PyPI Release](#latest-stable-pypi-release)\n - [Latest GitHub Update](#latest-github-update)\n - [Installing in Colaboratory](#installing-in-colaboratory)\n- [Overview](#overview)\n - [Smuggling Missing Packages](#smuggling-missing-packages)\n - [Smuggling Specific Package Versions](#smuggling-specific-package-versions)\n - [Use Cases](#use-cases)\n - [Simplify sharing reproducible code & Python environments](#simplify-sharing-reproducible-code--python-environments)\n - [Guarantee your code always uses the latest version, release, or revision](#guarantee-your-code-always-uses-the-latest-version-release-or-revision)\n - [Compare behavior across package versions](#compare-behavior-across-package-versions)\n- [Usage](#usage)\n - [The `smuggle` Statement](#the-smuggle-statement)\n - [Syntax](#smuggle-statement-syntax)\n - [Rules](#smuggle-statement-rules)\n - [The Onion Comment](#the-onion-comment)\n - [Syntax](#onion-comment-syntax)\n - [Rules](#onion-comment-rules)\n - [The `davos` Config](#the-davos-config)\n - [Reference](#config-reference)\n - [Top-level Functions](#top-level-functions)\n- [How It Works: The `davos` Parser](#how-it-works-the-davos-parser)\n- [Additional Notes](#additional-notes)\n\n\n## Installation\n### Latest Stable PyPI Release\n[](https://pypi.org/project/davos/)\n[]((https://pypi.org/project/davos/))\n[]((https://pypi.org/project/davos/))\n```sh\npip install davos\n```\n\n\n### Latest GitHub Update\n[](https://github.com/ContextLab/davos/releases)\n[](https://github.com/ContextLab/davos/commits/main)\n[](https://github.com/ContextLab/davos/releases/latest)\n\n```sh\npip install git+https://github.com/ContextLab/davos.git\n```\n\n\n### Installing in Colaboratory\nTo use `davos` in [Google Colab](https://colab.research.google.com/), add a cell at the top of your notebook with an\npercentage sign (`%`) followed by one of the commands above (e.g., `%pip install davos`). Run the cell to install\n`davos` on the runtime virtual machine.\n\n**Note**: restarting the Colab runtime does not affect installed packages. However, if the runtime is \"factory reset\"\nor disconnected due to reaching its idle timeout limit, you'll need to rerun the cell to reinstall `davos` on the fresh\nVM instance.\n\n\n## Overview\nThe primary way to use `davos` is via [the `smuggle` statement](#the-smuggle-statement), which is made available\nsimply by running `import davos`. Like\n[the built-in `import` statement](https://docs.python.org/3/reference/import.html), the `smuggle` statement is used to\nload packages, modules, and other objects into the current namespace. The main difference between the two is in how\nthey handle missing packages and specific package versions.\n\n\n### Smuggling Missing Packages\n`import` requires that packages be installed _before_ the start of the interpreter session. Trying to `import` a package\nthat can't be found locally will throw a\n[`ModuleNotFoundError`](https://docs.python.org/3/library/exceptions.html#ModuleNotFoundError), and you'll have to\ninstall the package from the command line, restart the Python interpreter to make the new package importable, and rerun\nyour code in full in order to use it.\n\n**The `smuggle` statement, however, can handle missing packages on the fly**. If you `smuggle` a package that isn't\ninstalled locally, `davos` will install it for you, make its contents available to Python's\n[import machinery](https://docs.python.org/3/reference/import.html), and load it into the namespace for immediate use.\nYou can control _how_ `davos` installs missing packages by adding a special type of inline comment called an\n[\"onion\" comment](#the-onion-comment) next to a `smuggle` statement.\n\n\n### Smuggling Specific Package Versions\nOne simple but powerful use for [onion comments](#the-onion-comment) is making `smuggle` statements _version-sensitive_.\n\nPython doesn't provide a native, viable way to ensure a third-party package imported at runtime matches a specific\nversion or satisfies a particular [version constraint](https://www.python.org/dev/peps/pep-0440/#version-specifiers).\nMany packages expose their version info via a top-level `__version__` attribute (see\n[PEP 396](https://www.python.org/dev/peps/pep-0396/)), and certain tools (such as the standard library's\n[`importlib.metadata`](https://docs.python.org/3/library/importlib.metadata.html) and\n[`setuptools`](https://setuptools.readthedocs.io/en/latest/index.html)'s\n[`pkg_resources`](https://setuptools.readthedocs.io/en/latest/pkg_resources.html)) attempt to parse version info from\ninstalled distributions. However, using these to constrain imported package would require writing extra code to compare\nversion strings and _still_ manually installing the desired version and restarting the interpreter any time an\ninvalid version is caught.\n\nAdditionally, for packages installed through a version control system (e.g., [git](https://git-scm.com/)), this would be\ninsensitive to differences between revisions (e.g., commits) within the same semantic version.\n\n**`davos` solves these issues** by allowing you to specify a specific version or set of acceptable versions for each\nsmuggled package. To do this, simply provide a\n[version specifier](https://www.python.org/dev/peps/pep-0440/#version-specifiers) in an\n[onion comment](#the-onion-comment) next to the `smuggle` statement:\n```python\nsmuggle numpy as np # pip: numpy==1.20.2\nfrom pandas smuggle DataFrame # pip: pandas>=0.23,<1.0\n```\nIn this example, the first line will load [`numpy`](https://numpy.org/) into the local namespace under the alias \"`np`\",\njust as \"`import numpy as np`\" would. First, `davos` will check whether `numpy` is installed locally, and if so, whether\nthe installed version _exactly_ matches `1.20.2`. If `numpy` is not installed, or the installed version is anything\nother than `1.20.2`, `davos` will use the specified _installer program_, [`pip`](https://pip.pypa.io/en/stable/), to\ninstall `numpy==1.20.2` before loading the package.\n\nSimilarly, the second line will load the \"`DataFrame`\" object from the [`pandas`](https://pandas.pydata.org/) library,\nanalogously to \"`from pandas import DataFrame`\". A local `pandas` version of `0.24.1` would be used, but a local version\nof `1.0.2` would cause `davos` to replace it with a valid `pandas` version, as if you had manually run `pip install\npandas>=0.23,<1.0`.\n\nIn both cases, the imported versions will fit the constraints specified in their [onion comments](#the-onion-comment),\nand the next time `numpy` or `pandas` is smuggled with the same constraints, valid local installations will be found.\n\nYou can also force the state of a smuggled packages to match a specific VCS ref (branch, revision, tag, release, etc.).\nFor example:\n```python\nsmuggle hypertools as hyp # pip: git+https://github.com/ContextLab/hypertools.git@564c1d4\n```\nwill load [`hypertools`](https://hypertools.readthedocs.io/en/latest/) (aliased as \"`hyp`\"), as the package existed\n[on GitHub](https://github.com/ContextLab/hypertools), at commit\n[98a3d80](https://github.com/ContextLab/hypertools/tree/98a3d80). The general format for VCS references in\n[onion comments](#the-onion-comment) follows that of the\n[`pip-install`](https://pip.pypa.io/en/stable/topics/vcs-support) command. See the\n[notes on smuggling from VCS](#notes-vcs-smuggle) below for additional info.\n\nAnd with [a few exceptions](#notes-c-extensions), smuggling a specific package version will work _even if the package\nhas already been imported_!\n\n**Note**: `davos` v0.1 supports [IPython](https://ipython.readthedocs.io/en/stable/) environments (e.g.,\n[Jupyter](https://jupyter.org/) and [Colaboratory](https://colab.research.google.com/) notebooks) only. v0.2 will add\nsupport for \"regular\" (i.e., non-interactive) Python scripts.\n\n\n### Use Cases\n#### Simplify sharing reproducible code & Python environments\nDifferent versions of the same package can often behave quite differently—bugs are introduced and fixed, features\nare implemented and removed, support for Python versions is added and dropped, etc. Because of this, Python code that is\nmeant to be _reproducible_ (e.g., tutorials, demos, data analyses) is commonly shared alongside a set of fixed versions\nfor each package used. And since there is no Python-native way to specify package versions at runtime (see\n[above](#smuggling-specific-package-versions)), this typically takes the form of a pre-configured development\nenvironment the end user must build themselves (e.g., a [Docker](https://www.docker.com/) container or\n[conda](https://docs.conda.io/en/latest/) environment), which can be cumbersome, slow to set up, resource-intensive, and\nconfusing for newer users, as well as require shipping both additional specification files _and_ setup instructions\nalong with your code. And even then, a well-intentioned user may alter the environment in a way that affects your\ncarefully curated set of pinned packages (such as installing additional packages that trigger dependency updates).\n\nInstead, `davos` allows you to share code with one simple instruction: _just `pip install davos`!_ Replace your `import`\nstatements with `smuggle` statements, pin package versions in onion comments, and let `davos` take care of the rest.\nBeyond its simplicity, this approach ensures your predetermined package versions are in place every time your code is\nrun.\n\n\n#### Guarantee your code always uses the latest version, release, or revision\nIf you want to make sure you're always using the most recent release of a certain package, `davos` makes doing so easy:\n```python\nsmuggle mypkg # pip: mypkg --upgrade\n```\nOr if you have an automation designed to test your most recent commit on GitHub:\n```python\nsmuggle mypkg # pip: git+https://username/reponame.git\n```\n\n\n#### Compare behavior across package versions\nThe ability to `smuggle` a specific package version even after a different version has been imported makes `davos` a\nuseful tool for comparing behavior across multiple versions of the same package, within the same interpreter session:\n```python\ndef test_my_func_unchanged():\n \"\"\"Regression test for `mypkg.my_func()`\"\"\"\n data = list(range(10))\n\n smuggle mypkg # pip: mypkg==0.1\n result1 = mypkg.my_func(data)\n\n smuggle mypkg # pip: mypkg==0.2\n result2 = mypkg.my_func(data)\n\n smuggle mypkg # pip: git+https://github.com/MyOrg/mypkg.git\n result3 = mypkg.my_func(data)\n\n assert result1 == result2 == result3\n```\n\n\n## Usage\n### The `smuggle` Statement\n#### <a name=\"smuggle-statement-syntax\"></a>Syntax\nThe `smuggle` statement is meant to be used in place of\n[the built-in `import` statement](https://docs.python.org/3/reference/import.html) and shares\n[its full syntactic definition](https://docs.python.org/3/reference/simple_stmts.html#the-import-statement):\n```ebnf\nsmuggle_stmt ::= \"smuggle\" module [\"as\" identifier] (\",\" module [\"as\" identifier])*\n | \"from\" relative_module \"smuggle\" identifier [\"as\" identifier]\n (\",\" identifier [\"as\" identifier])*\n | \"from\" relative_module \"smuggle\" \"(\" identifier [\"as\" identifier]\n (\",\" identifier [\"as\" identifier])* [\",\"] \")\"\n | \"from\" module \"smuggle\" \"*\"\nmodule ::= (identifier \".\")* identifier\nrelative_module ::= \".\"* module | \".\"+\n```\n<sup>\n <i>\n NB: uses the modified BNF grammar notation described in\n <a href=\"https://docs.python.org/3/reference\">The Python Language Reference</a>,\n <a href=\"https://docs.python.org/3/reference/introduction.html#notation\">here</a>; see\n <a href=\"https://docs.python.org/3/reference/lexical_analysis.html#identifiers\">here</a> for the lexical definition\n of <code>identifier</code>\n </i>\n</sup>\n\n\nIn simpler terms, **any valid syntax for `import` is also valid for `smuggle`**.\n\n\n#### <a name=\"smuggle-statement-rules\"></a>Rules\n- Like `import` statements, `smuggle` statements are whitespace-insensitive, unless a lack of whitespace between two\n tokens would cause them to be interpreted as a different token:\n ```python\n from os.path smuggle dirname, join as opj # valid\n from os . path smuggle dirname ,join as opj # also valid\n from os.path smuggle dirname, join asopj # invalid (\"asopj\" != \"as opj\")\n ```\n- Any context that would cause an `import` statement _not_ to be executed will have the same effect on a `smuggle`\n statement:\n ```python\n # smuggle matplotlib.pyplot as plt # not executed\n print('smuggle matplotlib.pyplot as plt') # not executed\n foo = \"\"\"\n smuggle matplotlib.pyplot as plt\"\"\" # not executed\n ```\n- Because the `davos` parser is less complex than the full Python parser, there are two fairly non-disruptive edge\n cases where an `import` statement would be syntactically valid but a `smuggle` statement would not:\n 1. The [exec](https://docs.python.org/3.8/library/functions.html#exec) function\n ```python\n exec('from pathlib import Path') # executed\n exec('from pathlib smuggle Path') # raises SyntaxError\n ```\n 2. A one-line [compound statement](https://docs.python.org/3.9/reference/compound_stmts.html#compound-statements)\n clause:\n ```python\n if True: import random # executed\n if True: smuggle random # raises SyntaxError\n\n while True: import math; break # executed\n while True: smuggle math; break # raises SyntaxError\n\n for _ in range(1): import json # executed\n for _ in range(1): smuggle json # raises SyntaxError\n\n # etc...\n ```\n- In [IPython](https://ipython.readthedocs.io/en/stable/) environments (e.g., [Jupyter](https://jupyter.org/) &\n [Colaboratory](https://colab.research.google.com/) notebooks) `smuggle` statements always load names into the global\n namespace:\n ```python\n # example.ipynb\n import davos\n\n\n def import_example():\n import datetime\n\n\n def smuggle_example():\n smuggle datetime\n\n\n import_example()\n type(datetime) # raises NameError\n\n smuggle_example()\n type(datetime) # returns\n ```\n\n\n### The Onion Comment\nAn _onion comment_ is a special type of inline comment placed on a line containing a `smuggle` statement. Onion comments\ncan be used to control how `davos`:\n1. determines whether the smuggled package should be installed\n2. installs the smuggled package, if necessary\n\nOnion comments are also useful when smuggling a package whose _distribution name_ (i.e., the name\nused when installing it) is different from its _top-level module name_ (i.e., the name used when importing it). Take for\nexample:\n```python\nfrom sklearn.decomposition smuggle pca # pip: scikit-learn\n```\nThe onion comment here (`# pip: scikit-learn`) tells `davos` that if \"`sklearn`\" does not exist\nlocally, the \"`scikit-learn`\" package should be installed.\n\n#### <a name=\"onion-comment-syntax\"></a>Syntax\nOnion comments follow a simple but specific syntax, inspired in part by the\n[type comment syntax](https://www.python.org/dev/peps/pep-0484/#type-comments) introduced in\n[PEP 484](https://www.python.org/dev/peps/pep-0484). The following is a loose (pseudo-)syntactic definition for an onion\ncomment:\n```ebnf\nonion_comment ::= \"#\" installer \":\" install_opt* pkg_spec install_opt*\ninstaller ::= (\"pip\" | \"conda\")\npkg_spec ::= identifier [version_spec]\n```\n<sup>\n <i>\n NB: uses the modified BNF grammar notation described in\n <a href=\"https://docs.python.org/3/reference\">The Python Language Reference</a>,\n <a href=\"https://docs.python.org/3/reference/introduction.html#notation\">here</a>; see\n <a href=\"https://docs.python.org/3/reference/lexical_analysis.html#identifiers\">here</a> for the lexical definition\n of <code>identifier</code>\n </i>\n</sup>\n\nwhere `installer` is the program used to install the package; `install_opt` is any option accepted by the installer's\n\"`install`\" command; and `version_spec` may be a\n[version specifier](https://www.python.org/dev/peps/pep-0440/#version-specifiers) defined by\n[PEP 440](https://www.python.org/dev/peps/pep-0440) followed by a\n[version string](https://www.python.org/dev/peps/pep-0440/#public-version-identifiers), or an alternative syntax valid\nfor the given `installer` program. For example, [`pip`](https://pip.pypa.io/en/stable/) uses specific syntaxes for\n[local](https://pip.pypa.io/en/stable/cli/pip_install/#local-project-installs),\n[editable](https://pip.pypa.io/en/stable/cli/pip_install/#editable-installs), and\n[VCS-based](https://pip.pypa.io/en/stable/topics/vcs-support) installation.\n\nLess formally, **an onion comment simply consists of two parts, separated by a colon**:\n1. the name of the installer program (e.g., [`pip`](https://pip.pypa.io/en/stable/))\n2. arguments passed to the program's \"install\" command\n\nThus, you can essentially think of writing an onion comment as taking the full shell command you would run to install\nthe package, and replacing \"_install_\" with \"_:_\". For instance, the command:\n```sh\npip install -I --no-cache-dir numpy==1.20.2 -vvv --timeout 30\n```\nis easily translated into an onion comment as:\n```python\nsmuggle numpy # pip: -I --no-cache-dir numpy==1.20.2 -vvv --timeout 30\n```\n\nIn practice, onion comments are identified as matches for the\n[regular expression](https://en.wikipedia.org/wiki/Regular_expression):\n```regex\n#+ *(?:pip|conda) *: *[^#\\n ].+?(?= +#| *\\n| *$)\n```\n<sup>\n <i>\n NB: support for installing <code>smuggle</code>d packages via\n <a href=\"https://docs.conda.io/en/latest/\"><code>conda</code></a> will be added in v0.2. For v0.1,\n \"<code>pip</code>\" should be used exclusively.\n </i>\n</sup>\n\n**Note**: support for installing smuggled packages via the [`conda`](https://docs.conda.io/en/latest/) package manager\nwill be added in v0.2. For v0.1, onion comments should always specify \"`pip`\" as the `installer` program.\n\n\n#### <a name=\"onion-comment-rules\"></a>Rules\n- An onion comment must be placed on the same line as a `smuggle` statement; otherwise, it is not parsed:\n ```python\n # assuming the dateutil package is not installed...\n\n # pip: python-dateutil # <-- has no effect\n smuggle dateutil # raises InstallerError (no \"dateutil\" package exists)\n\n smuggle dateutil # raises InstallerError (no \"dateutil\" package exists)\n # pip: python-dateutil # <-- has no effect\n\n smuggle dateutil # pip: python-dateutil # installs \"python-dateutil\" package, if necessary\n ```\n- An onion comment may be followed by unrelated inline comments as long as they are separated by at least one space:\n ```python\n smuggle tqdm # pip: tqdm>=4.46,<4.60 # this comment is ignored\n smuggle tqdm # pip: tqdm>=4.46,<4.60 # so is this one\n smuggle tqdm # pip: tqdm>=4.46,<4.60# but this comment raises OnionArgumentError\n ```\n- An onion comment must be the first inline comment immediately following a `smuggle` statement; otherwise, it is not\n parsed:\n ```python\n smuggle numpy # pip: numpy!=1.19.1 # <-- guarantees smuggled version is *not* v1.19.1\n smuggle numpy # has no effect --> # pip: numpy==1.19.1\n ```\n This also allows you to easily \"comment out\" onion comments:\n ```python\n smuggle numpy ## pip: numpy!=1.19.1 # <-- has no effect\n ```\n- Onion comments are generally whitespace-insensitive, but installer arguments must be separated by at least one space:\n ```python\n from umap smuggle UMAP # pip: umap-learn --user -v --no-clean # valid\n from umap smuggle UMAP#pip:umap-learn --user -v --no-clean # also valid\n from umap smuggle UMAP # pip: umap-learn --user-v--no-clean # raises OnionArgumentError\n ```\n- Onion comments have no effect on standard library modules:\n ```python\n smuggle threading # pip: threading==9999 # <-- has no effect\n ```\n- When smuggling multiple packages with a _single_ `smuggle` statement, an onion comment may be used to refer to the\n **first** package listed:\n ```python\n smuggle nilearn, nibabel, nltools # pip: nilearn==0.7.1\n ```\n- If multiple _separate_ `smuggle` statements are placed on a single line, an onion comment may be used to refer to the\n **last** statement:\n ```python\n smuggle gensim; smuggle spacy; smuggle nltk # pip: nltk~=3.5 --pre\n ```\n- For multiline `smuggle` statements, an onion comment may be placed on the first line:\n ```python\n from scipy.interpolate smuggle ( # pip: scipy==1.6.3\n interp1d,\n interpn as interp_ndgrid,\n LinearNDInterpolator,\n NearestNDInterpolator,\n )\n ```\n ... or on the last line:\n ```python\n from scipy.interpolate smuggle (interp1d, # this comment has no effect\n interpn as interp_ndgrid,\n LinearNDInterpolator,\n NearestNDInterpolator) # pip: scipy==1.6.3\n ```\n ... though the first line takes priority:\n ```python\n from scipy.interpolate smuggle ( # pip: scipy==1.6.3 # <-- this version is installed\n interp1d,\n interpn as interp_ndgrid,\n LinearNDInterpolator,\n NearestNDInterpolator,\n ) # pip: scipy==1.6.2 # <-- this comment is ignored\n ```\n ... and all comments _not_ on the first or last line are ignored:\n ```python\n from scipy.interpolate smuggle (\n interp1d, # pip: scipy==1.6.3 # <-- ignored\n interpn as interp_ndgrid,\n LinearNDInterpolator, # unrelated comment # <-- ignored\n NearestNDInterpolator\n ) # pip: scipy==1.6.2 # <-- parsed\n ```\n- The onion comment is intended to describe how a specific smuggled package should be installed if it is not found\n locally, in order to make it available for immediate use. Therefore, installer options that either (A) install\n packages other than the smuggled package and its dependencies (e.g., from a specification file), or (B) cause the\n smuggled package not to be installed, are disallowed. The options listed below will raise an `OnionArgumentError`:\n - `-h`, `--help`\n - `-r`, `--requirement`\n - `-V`, `--version`\n\n\n### The `davos` Config\nThe `davos` config object stores options and data that affect how `davos` behaves. After importing `davos`, the config\ninstance (a singleton) for the current session is available as `davos.config`, and its various fields are accessible as\nattributes. The config object exposes a mixture of writable and read-only fields. Most `davos.config` attributes can be\nassigned values to control aspects of `davos` behavior, while others are available for inspection but are set and used\ninternally. Additionally, certain config fields may be writable in some situations but not others (e.g. only if the\nimporting environment supports a particular feature). Once set, `davos` config options last for the lifetime of the\ninterpreter (unless updated); however, they do *not* persist across interpreter sessions. A full list of `davos` config\nfields is available [below](#config-reference):\n\n#### <a name=\"config-reference\"></a>Reference\n| Field | Description | Type | Default | Writable? |\n| :---: | --- | :---: | :---: | :---: |\n| `active` | Whether or not the `davos` parser should be run on subsequent input (cells, in Jupyter/Colab notebooks). Setting to `True` activates the `davos` parser, enables the `smuggle` keyword, and injects the `smuggle()` function into the user namespace. Setting to `False` deactivates the `davos` parser, disables the `smuggle` keyword, and removes \"`smuggle`\" from the user namespace (if it holds a reference to the `smuggle()` function). See [How it Works](#how-it-works) for more info. | `bool` | `True` | \u2705 |\n| `auto_rerun` | If `True`, when smuggling a previously-imported package that cannot be reloaded (see [Smuggling packages with C-extensions](#notes-c-extensions)), `davos` will automatically restart the interpreter and rerun all code up to (and including) the current `smuggle` statement. Otherwise, issues a warning and prompts the user with buttons to either restart/rerun or continue running. | `bool` | `False` | \u2705 (**Jupyter notebooks only**) |\n| `confirm_install` | Whether or not `davos` should require user confirmation (`[y/n]` input) before installing a smuggled package | `bool` | `False` | \u2705 |\n| `environment` | A label describing the environment into which `davos` was running. Checked internally to determine which interchangeable implementation functions are used, whether certain config fields are writable, and various other behaviors | `Literal['Python', 'IPython<7.0', 'IPython>=7.0', 'Colaboratory']` | N/A | \u274c |\n| `ipython_shell` | The global IPython interactive shell instance | [`IPython.core`<br>`.interactiveshell`<br>`.InteractiveShell`](https://ipython.readthedocs.io/en/stable/api/generated/IPython.core.interactiveshell.html#IPython.core.interactiveshell.InteractiveShell) | N/A | \u274c |\n| `noninteractive` | Set to `True` to run `davos` in non-interactive mode (all user input and confirmation will be disabled). **NB**:<br>1. Setting to `True` disables `confirm_install` if previously enabled <br>2. If `auto_rerun` is `False` in non-interactive mode, `davos` will throw an error if a smuggled package cannot be reloaded | `bool` | `False` | \u2705 (**Jupyter notebooks only**) |\n| `pip_executable` | The path to the `pip` executable used to install smuggled packages. Must be a path (`str` or [`pathlib.Path`](https://docs.python.org/3/library/pathlib.html#pathlib.Path)) to a real file. Default is programmatically determined from Python environment; falls back to `sys.executable -m pip` if executable can't be found | `str` | `pip` exe path or `sys.executable -m pip` | \u2705 |\n| `smuggled` | A cache of packages smuggled during the current interpreter session. Formatted as a `dict` whose keys are package names and values are the (`.split()` and `';'.join()`ed) onion comments. Implemented this way so that any non-whitespace change to installer arguments re-installation | `dict[str, str]` | `{}` | \u274c |\n| `suppress_stdout` | If `True`, suppress all unnecessary output issued by both `davos` and the installer program. Useful when smuggling packages that need to install many dependencies and therefore generate extensive output. If the installer program throws an error while output is suppressed, both stdout & stderr will be shown with the traceback | `bool` | `False` | \u2705 |\n\n#### <a name=\"top-level-functions\"></a>Top-level Functions\n`davos` also provides a few convenience for reading/setting config values:\n- **`davos.activate()`**\n Activate the `davos` parser, enable the `smuggle` keyword, and inject the `smuggle()` function into the namespace.\n Equivalent to setting `davos.config.active = True`. See [How it Works](#how-it-works) for more info.\n\n- **`davos.deactivate()`**\n Deactivate the `davos` parser, disable the `smuggle` keyword, and remove the name `smuggle` from the namespace if (and\n only if) it refers to the `smuggle()` function. If `smuggle` has been overwritten with a different value, the variable\n will not be deleted. Equivalent to setting `davos.config.active = False`. See [How it Works](#how-it-works) for more\n- info.\n\n- **`davos.is_active()`**\n Return the current value of `davos.config.active`.\n\n- **`davos.configure(**kwargs)`**\n Set multiple `davos.config` fields at once by passing values as keyword arguments, e.g.:\n ```python\n import davos\n davos.configure(active=False, noninteractive=True, pip_executable='/usr/bin/pip3')\n ```\n is equivalent to:\n ```python\n import davos\n davos.active = False\n davos.noninteractive = True\n davos.pip_executable = '/usr/bin/pip3'\n ```\n\n## How It Works: The `davos` Parser\nFunctionally, importing `davos` appears to enable a new Python keyword, \"_`smuggle`_\". However, `davos` doesn't actually\nmodify the rules or [reserved keywords](https://docs.python.org/3/reference/lexical_analysis.html#keywords) used by\nPython's parser and lexical analyzer in order to do so—in fact, modifying the Python grammar is not possible at\nruntime and would require rebuilding the interpreter. Instead, in [IPython](https://ipython.readthedocs.io/en/stable/)\nenivonments like [Jupyter](https://jupyter.org/) and\n[Colaboratory](https://colab.research.google.com/notebooks/intro.ipynb) notebooks, `davos` implements the `smuggle`\nkeyword via a combination of namespace injections and its own (far simpler) custom parser.\n\nThe `smuggle` keyword can be enabled and disabled at will by \"activating\" and \"deactivating\" `davos` (see the\n[`davos` Config Reference](config-reference) and [Top-level Functions](#top-level-functions), above). When `davos` is\nimported, it is automatically activated by default. Activating `davos` triggers two things:\n1. The _`smuggle()` function_ is injected into the `IPython` user namespace\n2. The _`davos` parser_ is registered as a\n[custom input transformer](https://ipython.readthedocs.io/en/stable/config/inputtransforms.html)\n\nIPython preprocesses all executed code as plain text before it is sent to the Python parser in order to handle\nspecial constructs like [`%magic`](https://ipython.readthedocs.io/en/stable/interactive/magics.html) and\n[`!shell`](https://ipython.readthedocs.io/en/stable/interactive/reference.html#system-shell-access) commands. `davos`\nhooks into this process to transform `smuggle` statements into syntactically valid Python code. The `davos`\nparser uses [this regular expression](https://github.com/ContextLab/davos/blob/main/davos/core/regexps.py) to match each\nline of code containing a `smuggle` statement (and, optionally, an onion comment), extracts information from its text,\nand replaces it with an analogous call to the _`smuggle()` function_. Thus, even though the code visible to the user may\ncontain `smuggle` statements, e.g.:\n```python\nsmuggle numpy as np # pip: numpy>1.16,<=1.20 -vv\n```\nthe code that is actually executed by the Python interpreter will not:\n```python\nsmuggle(name=\"numpy\", as_=\"np\", installer=\"pip\", args_str=\"\"\"numpy>1.16,<=1.20 -vv\"\"\", installer_kwargs={'editable': False, 'spec': 'numpy>1.16,<=1.20', 'verbosity': 2})\n```\n\nThe `davos` parser can be deactivated at any time, and doing so triggers the opposite actions of activating it:\n1. The name \"`smuggle`\" is deleted from the `IPython` user namespace, *unless it has been overwritten and no longer\n refers to the `smuggle()` function*\n2. The `davos` parser input transformer is deregistered.\n\n**Note**: in Jupyter and Colaboratory notebooks, IPython parses and transforms all text in a cell before sending it\nto the kernel for execution. This means that importing or activating `davos` will not make the `smuggle` statement\navailable until the _next_ cell, because all lines in the current cell were transformed before the `davos` parser was\nregistered. However, _deactivating_ `davos` disables the `smuggle` statement immediately—although the `davos`\nparser will have already replaced all `smuggle` statements with `smuggle()` function calls, removing the function from\nthe namespace causes them to throw `NameError`.\n\n\n## Additional Notes\n- <a name=\"notes-reimplement-cli\"></a>**Reimplementing installer programs' CLI parsers**\n\n The `davos` parser extracts info from onion comments by passing them to a (slightly modified) reimplementation of\n their specified installer program's CLI parser. This is somewhat redundant, since the arguments will eventually be\n re-parsed by the _actual_ installer program if the package needs to be installed. However, it affords a number of\n advantages, such as:\n - detecting errors early during the parser phase, before spending any time running code above the line containing the\n `smuggle` statement\n - preventing shell injections in onion comments—e.g., `#pip: --upgrade numpy && rm -rf /` fails due to the\n `OnionParser`, but would otherwise execute successfully.\n - allowing certain installer arguments to temporarily influence `davos` behavior while smuggling the current package\n (see [Installer options that affect `davos` behavior](#notes-installer-opts) below for specific info)\n\n- <a name=\"notes-installer-opts\"></a>**Installer options that affect `davos` behavior**\n\n Passing certain options to the installer program via an [onion comment](#the-onion-comment) will also affect the\n corresponding `smuggle` statement in a predictable way:\n\n - [**`--force-reinstall`**](https://pip.pypa.io/en/stable/cli/pip_install/#cmdoption-force-reinstall) |\n [**`-I`, `--ignore-installed`**](https://pip.pypa.io/en/stable/cli/pip_install/#cmdoption-I) |\n [**`-U`, `--upgrade`**](https://pip.pypa.io/en/stable/cli/pip_install/#cmdoption-I)\n\n The package will be installed, even if it exists locally\n\n - [**`--no-input`**](https://pip.pypa.io/en/stable/cli/pip/#cmdoption-no-input)\n\n Disables input prompts, analogous to temporarily setting `davos.config.noninteractive` to `True`. Overrides value\n of `davos.config.confirm_install`.\n\n - [**`--src <dir>`**](https://pip.pypa.io/en/stable/cli/pip_install/#cmdoption-src) |\n [**`-t`, `--target <dir>`**](https://pip.pypa.io/en/stable/cli/pip_install/#cmdoption-t)\n\n Prepends `<dir>` to [`sys.path`](https://docs.python.org/3/library/sys.html#sys.path) if not already present so\n the package can be imported.\n\n- <a name=\"notes-c-extensions\"></a>**Smuggling packages with C-extensions**\n\n Some Python packages that rely heavily on custom data types implemented via\n [C-extensions](https://docs.python.org/3.9/extending/extending.html) (e.g., `numpy`, `pandas`) dynamically generate\n modules defining various C functions and data structures, and link them to the Python interpreter when they are first\n imported. Depending on how these objects are initialized, they may not be subject to normal garbage collection, and\n persist despite their reference count dropping to zero. This can lead to unexpected errors when reloading the Python\n module that creates them, particularly if their dynamically generated source code has been changed (e.g., because the\n reloaded package is a newer version).\n\n This can occasionally affect `davos`'s ability to `smuggle` a new version of a package (or dependency) that was\n previously imported. To handle this, `davos` first checks each package it installs against\n [`sys.modules`](https://docs.python.org/3.9/library/sys.html#sys.modules). If a different version has already been\n loaded by the interpreter, `davos` will attempt to replace it with the requested version. If this fails, `davos` will\n restore the old package version _in memory_, while replacing it with the new package version _on disk_. This allows\n subsequent code that uses the non-reloadable module to still execute in most cases, while dependency checks for other\n packages run against the updated version. Then, depending on the value of `davos.config.auto_rerun`, `davos` will\n either either automatically restart the interpreter to load the updated package, prompt you to do so, or raise an\n exception.\n\n- <a name=\"notes-from-reload\"></a>**_`from` ... `import` ..._ statements and reloading modules**\n\n The Python docs for [`importlib.reload()`](https://docs.python.org/3/library/importlib.html#importlib.reload) include\n the following caveat:\n > If a module imports objects from another module using\n > [`from`](https://docs.python.org/3/reference/simple_stmts.html#from) \u2026\n > [`import`](https://docs.python.org/3/reference/simple_stmts.html#import) \u2026, calling\n > [`reload()`](https://docs.python.org/3/library/importlib.html#importlib.reload) for the other module does\n > not redefine the objects imported from it \u2014 one way around this is to re-execute the `from` statement, another is to\n > use `import` and qualified names (_module.name_) instead.\n\n The same applies to smuggling packages or modules from which objects have already been loaded. If object _`name`_ from\n module _`module`_ was loaded using either _`from module import name`_ or _`from module smuggle name`_, subsequently\n running _`smuggle module # pip --upgrade`_ will in fact install and load an upgraded version of _`module`_, but the\n the _`name`_ object will still be that of the old version! To fix this, you can simply run _`from module smuggle\n name`_ either instead in lieu of or after _`smuggle module`_.\n\n\n- <a name=\"notes-vcs-smuggle\"></a>**Smuggling packages from version control systems**\n\n The first time during an interpreter session that a given package is installed from a VCS URL, it is assumed not to be\n present locally, and is therefore freshly installed. `pip` clones non-editable VCS repositories into a temporary\n directory, runs `setup.py install`, and then immediately deletes them. Since no information is retained about the\n state of the repository at installation, it is impossible to determine whether an existing package satisfies the state\n (i.e., branch, tag, commit hash, etc.) requested for smuggled package.\n\n[comment]: <> (- As with _all_ code, you should use caution when running Python code containing `smuggle` statements that was not written by you or someone you know. )\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Install and manage Python packages at runtime using the \"smuggle\" statement.",
"version": "0.2.2",
"project_urls": {
"Download": "https://github.com/ContextLab/davos",
"Homepage": "https://github.com/ContextLab/davos"
},
"split_keywords": [
"import",
"install",
"package",
"module",
"automatic",
"davos",
"smuggle",
"pip",
"conda"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "a37dfcdec972721b637b3ca162889487e6495d0f8829575af26940c169d63944",
"md5": "bb22c5a3cc189c8ef12d9f46eca7710e",
"sha256": "d30a6f150c2975327e934f12677dec44b63f372039ac9dabe2bec12be35a1fae"
},
"downloads": -1,
"filename": "davos-0.2.2-py3-none-any.whl",
"has_sig": false,
"md5_digest": "bb22c5a3cc189c8ef12d9f46eca7710e",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.6",
"size": 99692,
"upload_time": "2023-08-07T23:16:07",
"upload_time_iso_8601": "2023-08-07T23:16:07.058802Z",
"url": "https://files.pythonhosted.org/packages/a3/7d/fcdec972721b637b3ca162889487e6495d0f8829575af26940c169d63944/davos-0.2.2-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "91debcc0620b8bcf16eade8af9f8b28cfae11621de1a457d361aacdf799e5ea8",
"md5": "5a81114a26fb839097ee9b45e28be9b2",
"sha256": "5459dd82187c4cea51eee78dbf88a2c3837d72510dcd5e7b22b1e984a7e18cf1"
},
"downloads": -1,
"filename": "davos-0.2.2.tar.gz",
"has_sig": false,
"md5_digest": "5a81114a26fb839097ee9b45e28be9b2",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.6",
"size": 116728,
"upload_time": "2023-08-07T23:16:09",
"upload_time_iso_8601": "2023-08-07T23:16:09.010675Z",
"url": "https://files.pythonhosted.org/packages/91/de/bcc0620b8bcf16eade8af9f8b28cfae11621de1a457d361aacdf799e5ea8/davos-0.2.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-08-07 23:16:09",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "ContextLab",
"github_project": "davos",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "davos"
}
JSON
