| Name | igogo JSON |
| Version |
1.1.0
JSON |
| download |
| home_page | |
| Summary | Execute several jupyter cells simultaneously |
| upload_time | 2023-11-10 14:44:51 |
| maintainer | |
| docs_url | None |
| author | |
| requires_python | >=3.8 |
| license | |
| keywords |
execute
ipython
jupyter
jupyterlab
python
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# igogo 🐎🏎️
Execute several jupyter cells at the same time
> Have you ever just sited and watched a long-running jupyter cell?
> **Now, you can continue to work in the same notebook freely**
https://user-images.githubusercontent.com/25539425/227176976-2bdda463-ecc9-4431-afec-6d31fbd4c214.mov
---
## Use Cases
1) **You have a long-running cell, and you need to check something.
You can just start the second cell without interrupting a long-running cell**.
> **Example:** you run a machine learning train loop and want to immediately save the model's weights or check metrics.
> With `igogo` you can do so without interrupting the training.
2) **If you need to compare the score of some function with different parameters, you can run several
functions at the same time and monitor results**.
> **Example:** you have several sets of hyperparameters and want to compare them.
> You can start training two models, monitoring two loss graphs at the same time.
3) **Process data in chunks**. Check processed data for validity
> **Example:** you do data processing in steps. With `igogo` you can execute several steps at the same time
> and process data from the first processing step in the second processing step in chunks.
> Also, you can quickly check that the first step produces the correct results
## Install
Igogo is available through PyPi:
```bash
pip install igogo
```
## Wait, isn't it just a background job? No.
- **No multithreading, no data races, no locks**.
You can freely operate with your notebook variables without the risk of corrupting them.
- **Beautiful output**. When several cells execute in parallel,
all printed data is displayed in the corresponding cell's output. No more twisted and messed out concurrent outputs.
- **Easily cancel jobs, wait for completion, and start the new ones**.
- **Control execution of jobs through widgets**.
## Usage
At the core of igogo is collaborative execution. Jobs need to explicitly allow other jobs to execute through `igogo.yielder()`. Mind that regular cells also represent a job.
Placing `igogo.yielder()` in code that is not executed in igogo job is not a mistake. It will return immediately. So, you don't need to care about keeping `igogo.yielder()` only in igogo jobs. You can place it anywhere
To start an igogo job, you can use `%%igogo` cell magic or function decorator.
```python
import igogo
@igogo.job
def hello_world(name):
for i in range(3):
print("Hello, world from", name)
# allows other jobs to run while asleep
# also can be `igogo.yielder()`
igogo.sleep(1)
return name
```
Call function as usual to start a job:
```python
hello_world('igogo'), hello_world('other igogo');
```
https://user-images.githubusercontent.com/25539425/227186815-6870e348-46e6-4086-a89b-be416c0cc1a7.mov
### Configure Jobs
Decorator `@igogo.job` has several useful parameters.
- `kind`\
Allows to set how to render output. Possible options: `text`, `markdown`, `html` Default: `text`
- `displays`\
As igogo job modify already executed cell, it needs to have spare placeholders for rich output.
This parameter specifies how many spare displays to spawn. Default: `1`
- `name`\
User-friendly name of igogo job.
- `warn_rewrite`\
Should warn rewriting older displays? Default: `True`
- `auto_display_figures`\
Should display pyplot figures created inside igogo automatically? Default: `True`
Markdown example:
https://user-images.githubusercontent.com/25539425/227203729-af94582c-8fe2-40fe-a6f0-6489a374a88f.mov
### Display Additional Data
Pyplot figures will be automatically displayed in igogo cell.
You can also use `igogo.display` inside a job to display any other content or several figures. Mind that displays must be pre-allocated by specifying displays number in `igogo.job(displays=...)`
```python
import numpy as np
import matplotlib.pyplot as plt
import igogo
def experiment(name, f, i):
x = np.linspace(0, i / 10, 100)
fig = plt.figure()
plt.plot(
x,
f(x)
)
plt.gca().set_title(name)
igogo.display(fig)
fig = plt.figure()
plt.scatter(
x,
f(x)
)
plt.gca().set_title(name)
igogo.display(fig)
igogo.sleep(0.05)
```
As noted in "Configure jobs" section, `igogo` jobs have limited number of displays.
If you try to display more objects than job has, warning will be shown and the oldest displays will be overwritten.
### Cell Magic
The same way with `%%igogo`:
```python
%load_ext igogo
```
```python
%%igogo
name = 'igogo'
for i in range(3):
print("Hello, world from", name)
igogo.sleep(1)
```
### Widgets
All executed `igogo` jobs spawn a widget that allows to kill them. Jobs are not affected by `KeyboardInterrupt`
### Killing Jobs
Apart from killing through widgets, `igogo` jobs can be killed programmatically.
- `igogo.stop()` \
Can be called inside `igogo` job to kill itself.
- `igogo.stop_all()`\
Stops all running `igogo` jobs
- `igogo.stop_latest()`\
Stops the latest `igogo` job. Can be executed several times.
- `igogo.stop_by_cell_id(cell_id)`\
Kills all jobs that were launched in cell with `cell_id` (aka [5], cell_id=5).
Also, you can stop jobs of one specific function.
- `hello_world.stop_all()`\
Stops all `igogo` jobs created by `hello_world()`
## Supported Clients
Currently, `igogo` runs fully correct on:
- Jupyter Lab
- Jupyter
Runs but has problems with output from igogo jobs. Jobs are executed, but there could be problems with widgets and output:
- VSCode. For some reason it does not update display data. Therefore, no output is produced.
- DataSpell. It displays `[object Object]` and not output.
- Colab. It does not support updating content of executed cells
## More Examples
[**Check out pretty notebooks**](https://github.com/alexdremov/igogo/tree/main/examples)
---
### Train model and check metrics
https://user-images.githubusercontent.com/25539425/227651626-cba8a317-a986-4971-9639-84cdb388e2d3.mov
Also, you can modify training parameters, freeze/unfreeze layers, switch datasets, etc. All you need is to place `igogo.yielder()` in train loop.
### Process data and montitor execution
```python
import igogo
import numpy as np
from tqdm.auto import tqdm
%load_ext igogo
raw_data = np.random.randn(100000, 100)
result = []
```
```python
def row_processor(row):
return np.mean(row)
```
```python
%%igogo
for i in tqdm(range(len(raw_data))):
result.append(row_processor(raw_data[i]))
igogo.yielder()
```
```python
result[-1]
```
### Process data in chunks
```python
import igogo
import numpy as np
from tqdm.auto import tqdm
%load_ext igogo
raw_data = np.random.randn(5000000, 100)
igogo_yield_freq = 32
igogo_first_step_cache = []
result = []
```
```python
%%igogo
for i in tqdm(range(len(raw_data))):
processed = np.log(raw_data[i] * raw_data[i])
igogo_first_step_cache.append(processed)
if i > 0 and i % igogo_yield_freq == 0:
igogo.yielder() # allow other jobs to execute
```
```python
%%igogo
for i in tqdm(range(len(raw_data))):
while i >= len(igogo_first_step_cache): # wait for producer to process data
igogo.yielder()
result.append(np.mean(igogo_first_step_cache[i]))
```
https://user-images.githubusercontent.com/25539425/227224077-a3ce664c-cb52-4aa2-a3fe-71ac5a03cdeb.mov
Raw data
{
"_id": null,
"home_page": "",
"name": "igogo",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": "",
"keywords": "execute,ipython,jupyter,jupyterlab,python",
"author": "",
"author_email": "Alex Dremov <igogo@alexdremov.me>",
"download_url": "https://files.pythonhosted.org/packages/65/dc/d8129f2e8eb97f3e6bd1fa656c3b00cc7dbda2efe354fdbf2a1ef4302d2e/igogo-1.1.0.tar.gz",
"platform": null,
"description": "# igogo \ud83d\udc0e\ud83c\udfce\ufe0f\n\nExecute several jupyter cells at the same time\n\n> Have you ever just sited and watched a long-running jupyter cell?\n> **Now, you can continue to work in the same notebook freely**\n\nhttps://user-images.githubusercontent.com/25539425/227176976-2bdda463-ecc9-4431-afec-6d31fbd4c214.mov\n\n---\n\n## Use Cases\n1) **You have a long-running cell, and you need to check something.\n You can just start the second cell without interrupting a long-running cell**.\n > **Example:** you run a machine learning train loop and want to immediately save the model's weights or check metrics.\n > With `igogo` you can do so without interrupting the training.\n2) **If you need to compare the score of some function with different parameters, you can run several\n functions at the same time and monitor results**. \n > **Example:** you have several sets of hyperparameters and want to compare them.\n > You can start training two models, monitoring two loss graphs at the same time. \n3) **Process data in chunks**. Check processed data for validity\n > **Example:** you do data processing in steps. With `igogo` you can execute several steps at the same time\n > and process data from the first processing step in the second processing step in chunks.\n > Also, you can quickly check that the first step produces the correct results\n\n## Install\n\nIgogo is available through PyPi:\n\n```bash\npip install igogo\n```\n\n## Wait, isn't it just a background job? No.\n\n- **No multithreading, no data races, no locks**.\nYou can freely operate with your notebook variables without the risk of corrupting them.\n- **Beautiful output**. When several cells execute in parallel,\nall printed data is displayed in the corresponding cell's output. No more twisted and messed out concurrent outputs.\n- **Easily cancel jobs, wait for completion, and start the new ones**.\n- **Control execution of jobs through widgets**.\n\n## Usage\n\nAt the core of igogo is collaborative execution. Jobs need to explicitly allow other jobs to execute through `igogo.yielder()`. Mind that regular cells also represent a job.\n\nPlacing `igogo.yielder()` in code that is not executed in igogo job is not a mistake. It will return immediately. So, you don't need to care about keeping `igogo.yielder()` only in igogo jobs. You can place it anywhere\n\nTo start an igogo job, you can use `%%igogo` cell magic or function decorator. \n\n```python\nimport igogo\n\n@igogo.job\ndef hello_world(name):\n for i in range(3):\n print(\"Hello, world from\", name)\n \n # allows other jobs to run while asleep\n # also can be `igogo.yielder()`\n igogo.sleep(1) \n return name\n```\n\nCall function as usual to start a job:\n\n```python\nhello_world('igogo'), hello_world('other igogo');\n```\n\nhttps://user-images.githubusercontent.com/25539425/227186815-6870e348-46e6-4086-a89b-be416c0cc1a7.mov\n\n### Configure Jobs\n\nDecorator `@igogo.job` has several useful parameters. \n\n- `kind`\\\n Allows to set how to render output. Possible options: `text`, `markdown`, `html` Default: `text`\n- `displays`\\\n As igogo job modify already executed cell, it needs to have spare placeholders for rich output.\n This parameter specifies how many spare displays to spawn. Default: `1`\n- `name`\\\n User-friendly name of igogo job.\n- `warn_rewrite`\\\n Should warn rewriting older displays? Default: `True`\n- `auto_display_figures`\\\n Should display pyplot figures created inside igogo automatically? Default: `True`\n\nMarkdown example:\n\nhttps://user-images.githubusercontent.com/25539425/227203729-af94582c-8fe2-40fe-a6f0-6489a374a88f.mov\n\n### Display Additional Data\n\nPyplot figures will be automatically displayed in igogo cell.\n\nYou can also use `igogo.display` inside a job to display any other content or several figures. Mind that displays must be pre-allocated by specifying displays number in `igogo.job(displays=...)`\n\n```python\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport igogo\n\ndef experiment(name, f, i):\n x = np.linspace(0, i / 10, 100)\n fig = plt.figure()\n plt.plot(\n x,\n f(x)\n )\n plt.gca().set_title(name)\n igogo.display(fig)\n \n fig = plt.figure()\n plt.scatter(\n x,\n f(x)\n )\n plt.gca().set_title(name)\n igogo.display(fig)\n igogo.sleep(0.05)\n```\n\nAs noted in \"Configure jobs\" section, `igogo` jobs have limited number of displays.\nIf you try to display more objects than job has, warning will be shown and the oldest displays will be overwritten.\n\n### Cell Magic\n\nThe same way with `%%igogo`:\n\n```python\n%load_ext igogo\n```\n\n```python\n%%igogo\nname = 'igogo'\nfor i in range(3):\n print(\"Hello, world from\", name)\n igogo.sleep(1)\n```\n\n### Widgets\n\nAll executed `igogo` jobs spawn a widget that allows to kill them. Jobs are not affected by `KeyboardInterrupt`\n\n### Killing Jobs\n\nApart from killing through widgets, `igogo` jobs can be killed programmatically.\n\n- `igogo.stop()` \\\n Can be called inside `igogo` job to kill itself.\n- `igogo.stop_all()`\\\n Stops all running `igogo` jobs\n- `igogo.stop_latest()`\\\n Stops the latest `igogo` job. Can be executed several times.\n- `igogo.stop_by_cell_id(cell_id)`\\\n Kills all jobs that were launched in cell with `cell_id` (aka [5], cell_id=5).\n\nAlso, you can stop jobs of one specific function.\n\n- `hello_world.stop_all()`\\\n Stops all `igogo` jobs created by `hello_world()`\n\n## Supported Clients\n\nCurrently, `igogo` runs fully correct on:\n\n- Jupyter Lab\n- Jupyter\n\nRuns but has problems with output from igogo jobs. Jobs are executed, but there could be problems with widgets and output:\n- VSCode. For some reason it does not update display data. Therefore, no output is produced.\n- DataSpell. It displays `[object Object]` and not output.\n- Colab. It does not support updating content of executed cells\n\n## More Examples\n\n[**Check out pretty notebooks**](https://github.com/alexdremov/igogo/tree/main/examples)\n\n---\n\n### Train model and check metrics \n\nhttps://user-images.githubusercontent.com/25539425/227651626-cba8a317-a986-4971-9639-84cdb388e2d3.mov\n\nAlso, you can modify training parameters, freeze/unfreeze layers, switch datasets, etc. All you need is to place `igogo.yielder()` in train loop.\n\n### Process data and montitor execution\n\n```python\nimport igogo\nimport numpy as np\nfrom tqdm.auto import tqdm\n%load_ext igogo\n\nraw_data = np.random.randn(100000, 100)\nresult = []\n```\n\n```python\ndef row_processor(row):\n return np.mean(row)\n```\n\n```python\n%%igogo\nfor i in tqdm(range(len(raw_data))):\n result.append(row_processor(raw_data[i]))\n igogo.yielder()\n```\n\n```python\nresult[-1]\n```\n\n### Process data in chunks\n\n```python\nimport igogo\nimport numpy as np\nfrom tqdm.auto import tqdm\n%load_ext igogo\n\nraw_data = np.random.randn(5000000, 100)\n\nigogo_yield_freq = 32\nigogo_first_step_cache = []\n\nresult = []\n```\n\n```python\n%%igogo\n\nfor i in tqdm(range(len(raw_data))):\n processed = np.log(raw_data[i] * raw_data[i])\n igogo_first_step_cache.append(processed)\n \n if i > 0 and i % igogo_yield_freq == 0:\n igogo.yielder() # allow other jobs to execute\n```\n\n```python\n%%igogo\n\nfor i in tqdm(range(len(raw_data))):\n while i >= len(igogo_first_step_cache): # wait for producer to process data\n igogo.yielder()\n \n result.append(np.mean(igogo_first_step_cache[i]))\n \n```\n\nhttps://user-images.githubusercontent.com/25539425/227224077-a3ce664c-cb52-4aa2-a3fe-71ac5a03cdeb.mov\n\n\n",
"bugtrack_url": null,
"license": "",
"summary": "Execute several jupyter cells simultaneously",
"version": "1.1.0",
"project_urls": {
"Homepage": "https://github.com/alexdremov/igogo"
},
"split_keywords": [
"execute",
"ipython",
"jupyter",
"jupyterlab",
"python"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "03349b6fed65f2fcc6eef26fa954804556938488433f01d40b82ccae4e9b7d07",
"md5": "f3b48b73496eb0f37b3c7a4eaf4f7d57",
"sha256": "c4150644ae26a1afbb2574426b9ca8333dcc241a505cbb063cfe5e1a38c8a3e9"
},
"downloads": -1,
"filename": "igogo-1.1.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "f3b48b73496eb0f37b3c7a4eaf4f7d57",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 13422,
"upload_time": "2023-11-10T14:44:48",
"upload_time_iso_8601": "2023-11-10T14:44:48.721566Z",
"url": "https://files.pythonhosted.org/packages/03/34/9b6fed65f2fcc6eef26fa954804556938488433f01d40b82ccae4e9b7d07/igogo-1.1.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "65dcd8129f2e8eb97f3e6bd1fa656c3b00cc7dbda2efe354fdbf2a1ef4302d2e",
"md5": "1014dfc8e6b79890586a40ddbba66a45",
"sha256": "159a01ec71c95f9fe9d550e80cca150f52025edaa750c827ffedf44205dab1fc"
},
"downloads": -1,
"filename": "igogo-1.1.0.tar.gz",
"has_sig": false,
"md5_digest": "1014dfc8e6b79890586a40ddbba66a45",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 120171,
"upload_time": "2023-11-10T14:44:51",
"upload_time_iso_8601": "2023-11-10T14:44:51.185338Z",
"url": "https://files.pythonhosted.org/packages/65/dc/d8129f2e8eb97f3e6bd1fa656c3b00cc7dbda2efe354fdbf2a1ef4302d2e/igogo-1.1.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-11-10 14:44:51",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "alexdremov",
"github_project": "igogo",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "igogo"
}
