# MPIDE - MicroPython TUI-REPL powered by `mpremote` and Textual
This project is currently being set up and not guaranteed to be of any use for
anyone but me.
However `mpide` _runs_ on my machine and might as well run on your's, too.
Project goal is a 'smart' REPL for MicroPython devices similar to
[`mpremote`](https://docs.micropython.org/en/latest/reference/mpremote.html)
(currently `mpide` is even based on `mpremote`), but with additional features like
* automatic synchronization of locally modified files with device
* automatic pre-compiling using `mpy-cross` if applicable
* providing a powerful set of pure Python base functionality for rapid development
* providing a web based remote update mechanism
* transparent and intuitive device bootstrapping, logging etc.
## Usage
Install:
```
[python3 -m ]pip install [--user] [--upgrade] mpide
```
Execute:
```
mpide [<PROJECT-PATH=.>]
```
Or add `mpide` as dependency to an existing MicroPython project in order to
synchronize locally modified files while `mpide` is running.
The `mpide` package brings with it `mpremote` and `esptool` as dependency, so installing
`mpide` or making it a dependency (e.g. for a `poetry` based project) provides them
implicitly.
All the following stuff assumes that you've already successfully flashed
your device with a [recent version of MicroPython](https://micropython.org/download/)
Here is a very quick walk-through for a generic ESP32 based device:
```bash
wget https://micropython.org/resources/firmware/ESP32_GENERIC-20240602-v1.23.0.bin
esptool.py -p /dev/ttyACM0 -b 460800 --chip esp32 erase_flash
esptool.py -p /dev/ttyACM0 -b 460800 --chip esp32 write_flash -z 0x1000 ESP32_GENERIC-20240602-v1.23.0.bin
```
You should now be able to run `mpremote` to connect to your device and start a REPL.
Press CTRL-D for a soft-reboot, enter Python commands and see the result and press
CTRL-X to exit `mpremote`.
If that worked you can now use `mpide` instead of `mpremote`. If not you have to
find your way yourself for now..
## `mpide` walkthrough and project integration
While current interaction with a project folder is still quite sparse, you can at
least make `mpide` watch for filesystem changes in the current project directory
and auto-sync them with the device. There is even support for pre-compiling
`.py` files to `.mpy` files before uploding them - more on that later.
In order to let `mpide` watch and sync, you need to create a file called
`mpide_project.py`, containing information used on both the development host and
the MicroPython target.
I hear you asking why that configuration does not go to `pyproject.toml` or
any other fancy modern format instead. The reason is better memory efficency due
to the fact that you don't have to import a parser on your tiny ESP32 device.
Also, compared to a JSON or YAML file you can import additional stuff or even
dynamically evaluate values (which might be considered a security hazard, but so
is the possibility to upload arbitrary Python code..)
The smallest config file enabling for auto-syncing looks like this:
```python
config = {
"static_files": [
"main.py",
],
}
```
With this configuration you tell `mpide` to just sync `main.py` with `/main.py`
on the target device every time it changes on your filesystem (while `mpide` is
running of course) - no precompilation involved.
The term `static_files` implies that files will be written to the filesystem
root, making changes persistent immediately, while also slowly wearing out your
precious flash.
To test your configuration, spin up `mpide`, open `main.py` in your favorite
editor and add some meaningful code, e.g.
```python
def run():
print("hello world")
```
In order to copy files to a volatile directory, you (currently) have to do two
things: declare the file to be `dynamic` and setup a RAM-disk at `/ramdisk`.
To do this, modify the config file like this:
```python
config = {
"ramdisk": {
"mountpoint": "/ramdisk",
"block_size": 512,
"num_blocks": 200,
},
"static_files": [
"mpide_config.py",
],
"dynamic_files": [
"main.py",
],
}
```
(consider '`/ramdisk`' to be hard-coded for now - this will change in the near future)
You changed a couple of details now:
- added a RAM-disk configuration (will NOT be auto-mounted)
- made `main.py` "dynamic", i.e it will now be copied to `/ramdisk/main.py` instead
- added `mpide_config.py` to the tracked files - it will now be synchronized as
well (but to `/mpide_config.py`, i.e. stored permanently)
Now by modifying `main.py` your file will be automatically stored on the ramdisk.
No wait - you just get a badly formatted exception instead, mumbling something about
`OSError: [Errno 2] ENOENT`..
This is because the RAM-disk has been configured but not actually _mounted_ yet.
While this will be done automatically soon, you have to send `:ramdisk` once every
time the device has been rebooted..
Once done, saving `main.py` again you should see a message
`copied <project>/main.py to <device>/ramdisk/main.py` in the REPL, indicating
that you can now `import main` and `main.run()` to execute the code you just
edited.
In order to save time and space on the flash-FS/RAM-disk you can make `mpide`
involve `mpy-cross` from the MicroPython toolchain to precompile `.mpy` files
from your sources and upload them instead.
To do this, you again have to do two things: provide `mpy-cross` to `mpide` and
list `.mpy` files in your `mpide_project.py` file instead of `.py` files.
`mpide` will still watch for changes on `.py` files in your project folder
but do the mapping to `.mpy` files for you conveniently.
I take this as an opportunity to show off an advantage of scripted configuration
compared to a static JSON/YAML file. Imagine you have multiple development
machines you're working on, both with a different location for the `mpy-cross`
executable. You can just add a line
```
config.update(config_local.config)
```
to your `mpide_project.py` file, add a file called `config_local.py` which
you exclude from your project e.g. by listing it in `.gitignore`, and put in
all information which might be different among development machines. That
might be only the path to `mpy-cross` for now, but you can also think of
WiFi credentials, etc.
So here goes the full configuration:
`mpide_config.py`:
```python
import config_local
config = {
"ramdisk": {
"mountpoint": "/ramdisk",
"block_size": 512,
"num_blocks": 200,
},
"static_files": [
"mpide_config.mpy",
"local_config.mpy",
],
"dynamic_files": [
"main.mpy",
],
}
config.update(config_local.config)
```
`config_local.py`:
```python
config = {
"mpy_cross_path": "~/micropython/mpy-cross/build/mpy-cross",
}
```
### Built in commands
**`:cp <SRC> <DST>`**
Copies files from `SRC` on your host machine to `<DST>` on the target device
**`:cat <PATH>`**
Same as Posix `cat` - plots a file (located on the target device) into the REPL
**`:ramdisk`**
Executes the built-in `setup_ramdisk` snippet, which will create a RAM-disk
mounted at `/ramdisk`
**`:snippet <BASENAME> [<EXTRA-SNIPPET>]`**
Executes an arbitrary snippet taken from a Python source file, identified by
the provided basename. The file will be searched for in your project directory
or - if not found - somewhere in the `mpide` package directory.
If provided the optional `EXTRA-SNIPPET` get's attached, so for example
```
:snippet my_snippet print(foo('bar'))
```
.. will load `my_snippet.py` and execute it together with `print(foo('bar'))`.
This comes in handy when you have a file defining a function (here `foo()`),
which you can call with arbitrary parameters without modifying the actual
source file containing the snippet.
Go ahead and give it a try with this `my_snippet.py`:
```python
print("my_snippet")
def foo(text: str, count: int) -> None:
for i in range(count):
print(i, text)
```
Working around all bugs and uncovering undocumented behavior is left for you
as homework.
## Roadmap
This is what I intend to implement in the near future, since it's what I need in daily live:
Version 1.0 (MLP)
- [x] Textual based IDE stub based on `trickkiste/TuiBaseApp`
- [x] involve `mpremote` to automatically connect to attached (ESP32) devices running MicroPython
- [x] REPL command history
- [x] fix multi-line input
- [x] auto-update mpy/py files on change
- [x] make auto-update optional
- [x] provide RAM-FS for temporarily patched files and to avoid wearing out your flash
- [x] auto-unload updated modules on target device
- [ ] allow device / connection speed settings
- [ ] bootstrap (i.e. populate a newly flashed device with helper stuff for remote working)
- [ ] provide `--init` switch for creating example files to bootstrap a project
- [ ] add instructions to build MicroPython
- [ ] add commands: `:help`, `:rm`, `:tree`
- [ ] add passthrough (`!..`)
- [ ] make pre-compilation optional
- [ ] persist temporary files via `:persist`
- [ ] fix dark-mode
- [ ] check micropython / mpy-cross versions
- [ ] cleanup cache folder
Version 1.1
- [ ] Web-REPL
- [ ] Web-based remote updating
- [ ] support terminal colors
- [ ] make web-server optional
## License
For all code contained in this repository the rules of GPLv3 apply unless
otherwise noted. That means that you can do what you want with the source
code as long as you make the files with their original copyright notice
and all modifications available.
See [GNU / GPLv3](https://www.gnu.org/licenses/gpl-3.0.en.html) for details.
This project is not free for machine learning. If you're using any content
of this repository to train any sort of machine learned model (e.g. LLMs),
you agree to make the whole model trained with this repository and all data
needed to train (i.e. reproduce) the model publicly and freely available
(i.e. free of charge and with no obligation to register to any service) and
make sure to inform the author (me, frans.fuerst@protonmail.com) via email
how to get and use that model and any sources needed to train it.
## NAQ
* Q: Why don't you host this on GitHub instead (or at least _`gitlab.com`_)?
_A: I don't like the way Microsoft (or anyone else) is making money from projects they don't
own without asking, providing an option ot opt out or even informing about details_
## Interesting Links
* Phind.com: [with mpy-cross is it possible to pre-compile `@viper` decorated python-functions? how would mpy-cross know which platorm to create code for?](https://www.phind.com/search?cache=hbflpe9qo1caidumt6edsaz2)
* [Esp32-c3 native and viper decorators](https://github.com/orgs/micropython/discussions/9785)
* https://www.analog.com/en/resources/analog-dialogue/articles/introduction-to-spi-interface.html
Raw data
{
"_id": null,
"home_page": "https://projects.om-office.de/frans/mpide.git",
"name": "mpide",
"maintainer": null,
"docs_url": null,
"requires_python": "<4.0.0,>=3.10.4",
"maintainer_email": null,
"keywords": null,
"author": "Frans F\u00fcrst",
"author_email": "frans.fuerst@protonmail.com",
"download_url": "https://files.pythonhosted.org/packages/d7/1a/1a8d593844d5d1c9cd2e3bbbd9dfb46eb2df9c60c16335ea472e3d224bfc/mpide-0.2.0.dev2.tar.gz",
"platform": null,
"description": "# MPIDE - MicroPython TUI-REPL powered by `mpremote` and Textual\n\nThis project is currently being set up and not guaranteed to be of any use for\nanyone but me.\nHowever `mpide` _runs_ on my machine and might as well run on your's, too.\n\nProject goal is a 'smart' REPL for MicroPython devices similar to\n[`mpremote`](https://docs.micropython.org/en/latest/reference/mpremote.html)\n(currently `mpide` is even based on `mpremote`), but with additional features like\n\n* automatic synchronization of locally modified files with device\n* automatic pre-compiling using `mpy-cross` if applicable\n* providing a powerful set of pure Python base functionality for rapid development\n* providing a web based remote update mechanism\n* transparent and intuitive device bootstrapping, logging etc.\n\n![_(you miss a screenshot here)_](screenshot.png?raw=true \"A possibly recent screenshot of mpide in action\")\n\n\n## Usage\n\nInstall:\n```\n[python3 -m ]pip install [--user] [--upgrade] mpide\n```\n\nExecute:\n```\nmpide [<PROJECT-PATH=.>]\n```\n\nOr add `mpide` as dependency to an existing MicroPython project in order to\nsynchronize locally modified files while `mpide` is running.\n\nThe `mpide` package brings with it `mpremote` and `esptool` as dependency, so installing\n`mpide` or making it a dependency (e.g. for a `poetry` based project) provides them\nimplicitly.\n\nAll the following stuff assumes that you've already successfully flashed\nyour device with a [recent version of MicroPython](https://micropython.org/download/)\n\nHere is a very quick walk-through for a generic ESP32 based device:\n```bash\nwget https://micropython.org/resources/firmware/ESP32_GENERIC-20240602-v1.23.0.bin\nesptool.py -p /dev/ttyACM0 -b 460800 --chip esp32 erase_flash\nesptool.py -p /dev/ttyACM0 -b 460800 --chip esp32 write_flash -z 0x1000 ESP32_GENERIC-20240602-v1.23.0.bin\n```\nYou should now be able to run `mpremote` to connect to your device and start a REPL.\nPress CTRL-D for a soft-reboot, enter Python commands and see the result and press\nCTRL-X to exit `mpremote`.\n\nIf that worked you can now use `mpide` instead of `mpremote`. If not you have to\nfind your way yourself for now..\n\n\n## `mpide` walkthrough and project integration\n\nWhile current interaction with a project folder is still quite sparse, you can at\nleast make `mpide` watch for filesystem changes in the current project directory\nand auto-sync them with the device. There is even support for pre-compiling\n`.py` files to `.mpy` files before uploding them - more on that later.\n\nIn order to let `mpide` watch and sync, you need to create a file called\n`mpide_project.py`, containing information used on both the development host and\nthe MicroPython target.\n\nI hear you asking why that configuration does not go to `pyproject.toml` or\nany other fancy modern format instead. The reason is better memory efficency due\nto the fact that you don't have to import a parser on your tiny ESP32 device.\nAlso, compared to a JSON or YAML file you can import additional stuff or even\ndynamically evaluate values (which might be considered a security hazard, but so\nis the possibility to upload arbitrary Python code..)\n\nThe smallest config file enabling for auto-syncing looks like this:\n\n```python\nconfig = {\n \"static_files\": [\n \"main.py\",\n ],\n}\n```\n\nWith this configuration you tell `mpide` to just sync `main.py` with `/main.py`\non the target device every time it changes on your filesystem (while `mpide` is\nrunning of course) - no precompilation involved.\n\nThe term `static_files` implies that files will be written to the filesystem\nroot, making changes persistent immediately, while also slowly wearing out your\nprecious flash.\n\nTo test your configuration, spin up `mpide`, open `main.py` in your favorite\neditor and add some meaningful code, e.g.\n\n```python\ndef run():\n print(\"hello world\")\n```\n\nIn order to copy files to a volatile directory, you (currently) have to do two\nthings: declare the file to be `dynamic` and setup a RAM-disk at `/ramdisk`.\n\nTo do this, modify the config file like this:\n\n```python\nconfig = {\n \"ramdisk\": {\n \"mountpoint\": \"/ramdisk\",\n \"block_size\": 512,\n \"num_blocks\": 200,\n },\n \"static_files\": [\n \"mpide_config.py\",\n ],\n \"dynamic_files\": [\n \"main.py\",\n ],\n}\n```\n(consider '`/ramdisk`' to be hard-coded for now - this will change in the near future)\n\nYou changed a couple of details now:\n- added a RAM-disk configuration (will NOT be auto-mounted)\n- made `main.py` \"dynamic\", i.e it will now be copied to `/ramdisk/main.py` instead\n- added `mpide_config.py` to the tracked files - it will now be synchronized as\n well (but to `/mpide_config.py`, i.e. stored permanently)\n\nNow by modifying `main.py` your file will be automatically stored on the ramdisk.\n\nNo wait - you just get a badly formatted exception instead, mumbling something about\n`OSError: [Errno 2] ENOENT`..\n\nThis is because the RAM-disk has been configured but not actually _mounted_ yet.\nWhile this will be done automatically soon, you have to send `:ramdisk` once every\ntime the device has been rebooted..\n\nOnce done, saving `main.py` again you should see a message\n`copied <project>/main.py to <device>/ramdisk/main.py` in the REPL, indicating\nthat you can now `import main` and `main.run()` to execute the code you just\nedited.\n\nIn order to save time and space on the flash-FS/RAM-disk you can make `mpide`\ninvolve `mpy-cross` from the MicroPython toolchain to precompile `.mpy` files\nfrom your sources and upload them instead.\n\nTo do this, you again have to do two things: provide `mpy-cross` to `mpide` and\nlist `.mpy` files in your `mpide_project.py` file instead of `.py` files.\n`mpide` will still watch for changes on `.py` files in your project folder\nbut do the mapping to `.mpy` files for you conveniently.\n\nI take this as an opportunity to show off an advantage of scripted configuration\ncompared to a static JSON/YAML file. Imagine you have multiple development\nmachines you're working on, both with a different location for the `mpy-cross`\nexecutable. You can just add a line\n```\nconfig.update(config_local.config)\n```\nto your `mpide_project.py` file, add a file called `config_local.py` which\nyou exclude from your project e.g. by listing it in `.gitignore`, and put in\nall information which might be different among development machines. That\nmight be only the path to `mpy-cross` for now, but you can also think of\nWiFi credentials, etc.\n\nSo here goes the full configuration:\n\n`mpide_config.py`:\n```python\nimport config_local\nconfig = {\n \"ramdisk\": {\n \"mountpoint\": \"/ramdisk\",\n \"block_size\": 512,\n \"num_blocks\": 200,\n },\n \"static_files\": [\n \"mpide_config.mpy\",\n \"local_config.mpy\",\n ],\n \"dynamic_files\": [\n \"main.mpy\",\n ],\n}\nconfig.update(config_local.config)\n```\n\n`config_local.py`:\n```python\nconfig = {\n \"mpy_cross_path\": \"~/micropython/mpy-cross/build/mpy-cross\",\n}\n```\n### Built in commands\n\n**`:cp <SRC> <DST>`**\n\nCopies files from `SRC` on your host machine to `<DST>` on the target device\n\n**`:cat <PATH>`**\n\nSame as Posix `cat` - plots a file (located on the target device) into the REPL\n\n**`:ramdisk`**\n\nExecutes the built-in `setup_ramdisk` snippet, which will create a RAM-disk\nmounted at `/ramdisk`\n\n**`:snippet <BASENAME> [<EXTRA-SNIPPET>]`**\n\nExecutes an arbitrary snippet taken from a Python source file, identified by\nthe provided basename. The file will be searched for in your project directory\nor - if not found - somewhere in the `mpide` package directory.\nIf provided the optional `EXTRA-SNIPPET` get's attached, so for example\n```\n:snippet my_snippet print(foo('bar'))\n```\n.. will load `my_snippet.py` and execute it together with `print(foo('bar'))`.\nThis comes in handy when you have a file defining a function (here `foo()`),\nwhich you can call with arbitrary parameters without modifying the actual\nsource file containing the snippet.\n\nGo ahead and give it a try with this `my_snippet.py`:\n\n```python\nprint(\"my_snippet\")\ndef foo(text: str, count: int) -> None:\n for i in range(count):\n print(i, text)\n```\n\nWorking around all bugs and uncovering undocumented behavior is left for you\nas homework.\n\n\n## Roadmap\n\nThis is what I intend to implement in the near future, since it's what I need in daily live:\n\nVersion 1.0 (MLP)\n\n- [x] Textual based IDE stub based on `trickkiste/TuiBaseApp`\n- [x] involve `mpremote` to automatically connect to attached (ESP32) devices running MicroPython\n- [x] REPL command history\n- [x] fix multi-line input\n- [x] auto-update mpy/py files on change\n- [x] make auto-update optional\n- [x] provide RAM-FS for temporarily patched files and to avoid wearing out your flash\n- [x] auto-unload updated modules on target device\n- [ ] allow device / connection speed settings\n- [ ] bootstrap (i.e. populate a newly flashed device with helper stuff for remote working)\n- [ ] provide `--init` switch for creating example files to bootstrap a project\n- [ ] add instructions to build MicroPython\n- [ ] add commands: `:help`, `:rm`, `:tree`\n- [ ] add passthrough (`!..`)\n- [ ] make pre-compilation optional\n- [ ] persist temporary files via `:persist`\n- [ ] fix dark-mode\n- [ ] check micropython / mpy-cross versions\n- [ ] cleanup cache folder\n\nVersion 1.1\n\n- [ ] Web-REPL\n- [ ] Web-based remote updating\n- [ ] support terminal colors\n- [ ] make web-server optional\n\n\n## License\n\nFor all code contained in this repository the rules of GPLv3 apply unless\notherwise noted. That means that you can do what you want with the source\ncode as long as you make the files with their original copyright notice\nand all modifications available.\n\nSee [GNU / GPLv3](https://www.gnu.org/licenses/gpl-3.0.en.html) for details.\n\nThis project is not free for machine learning. If you're using any content\nof this repository to train any sort of machine learned model (e.g. LLMs),\nyou agree to make the whole model trained with this repository and all data\nneeded to train (i.e. reproduce) the model publicly and freely available\n(i.e. free of charge and with no obligation to register to any service) and\nmake sure to inform the author (me, frans.fuerst@protonmail.com) via email\nhow to get and use that model and any sources needed to train it.\n\n\n## NAQ\n\n* Q: Why don't you host this on GitHub instead (or at least _`gitlab.com`_)?\n _A: I don't like the way Microsoft (or anyone else) is making money from projects they don't\n own without asking, providing an option ot opt out or even informing about details_\n\n\n## Interesting Links\n\n* Phind.com: [with mpy-cross is it possible to pre-compile `@viper` decorated python-functions? how would mpy-cross know which platorm to create code for?](https://www.phind.com/search?cache=hbflpe9qo1caidumt6edsaz2)\n* [Esp32-c3 native and viper decorators](https://github.com/orgs/micropython/discussions/9785)\n\n* https://www.analog.com/en/resources/analog-dialogue/articles/introduction-to-spi-interface.html\n",
"bugtrack_url": null,
"license": null,
"summary": "MicroPython Development Environment",
"version": "0.2.0.dev2",
"project_urls": {
"Homepage": "https://projects.om-office.de/frans/mpide.git",
"Repository": "https://projects.om-office.de/frans/mpide.git"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "c5469a85989f915ace237a1c89664457f2aa2647e02ac2483a6a936dd74c935b",
"md5": "ec186220678a61fe38719615a49822e8",
"sha256": "9414398baf124bb8135111d2a33e01946569371d5d5c981d4c659c6c21934a9e"
},
"downloads": -1,
"filename": "mpide-0.2.0.dev2-py3-none-any.whl",
"has_sig": false,
"md5_digest": "ec186220678a61fe38719615a49822e8",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4.0.0,>=3.10.4",
"size": 20294,
"upload_time": "2024-08-08T17:38:03",
"upload_time_iso_8601": "2024-08-08T17:38:03.995983Z",
"url": "https://files.pythonhosted.org/packages/c5/46/9a85989f915ace237a1c89664457f2aa2647e02ac2483a6a936dd74c935b/mpide-0.2.0.dev2-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "d71a1a8d593844d5d1c9cd2e3bbbd9dfb46eb2df9c60c16335ea472e3d224bfc",
"md5": "a9246f99cd72b4fc01953a07fda8a4f6",
"sha256": "25d1f9bed2e330beb51894f5feeeb88b5506ce0a1707da46905575e1b64e5860"
},
"downloads": -1,
"filename": "mpide-0.2.0.dev2.tar.gz",
"has_sig": false,
"md5_digest": "a9246f99cd72b4fc01953a07fda8a4f6",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4.0.0,>=3.10.4",
"size": 20560,
"upload_time": "2024-08-08T17:38:05",
"upload_time_iso_8601": "2024-08-08T17:38:05.629683Z",
"url": "https://files.pythonhosted.org/packages/d7/1a/1a8d593844d5d1c9cd2e3bbbd9dfb46eb2df9c60c16335ea472e3d224bfc/mpide-0.2.0.dev2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-08-08 17:38:05",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "mpide"
}
JSON