# Interactive agent for visually impaired and blind individuals (based on Llama2)
This project uses a tier application to run the Large Language Model and accesses it using an API. **This is a prototype!**
The current tier application is based on Oobabooga's text generation webui. In the future support for Petals, or OpenAI API will be added.
## Installation
You first need to setup the tier application server. Then you can install this server application on a different or on the same machine.
### Installation of the tier application
Here Oobabooga's application is installed on an Ubuntu server equipped with 2 RTX 3090.
For reference, I'm using the [one click installer](https://github.com/oobabooga/text-generation-webui#one-click-installers) with a script to set up the API and a Ngrok tunnel. I installed Ngrok via pip (`python -m pip install pyngrok`).
```bash
#!/usr/bin/env bash
trap_with_arg() { # from https://stackoverflow.com/a/2183063/804678
local func="$1"; shift
for sig in "$@"; do
trap "$func $sig" "$sig"
done
}
stop() {
trap - SIGINT EXIT
printf '\n%s\n' "received $1, killing child processes"
kill -s SIGINT 0
}
trap_with_arg 'stop' EXIT SIGINT SIGTERM SIGHUP
export OOBABOOGA_FLAGS="--extension api --loader exllama_hf --model TheBloke_Llama-2-13B-chat-GPTQ --verbose --listen --chat --max_seq_len 4096 --compress_pos_emb 1"
bash /path/to/oobabooga_linux/start_linux.sh &
# Wait for the application to start
sleep 30s
# Start the reverse proxy
# --basic-auth 'api_token:your_custom_token' is optional, it allows using basic access authentication
# The username is 'api_token' and the password is 'your_custom_token'
ngrok http --domain=my-ngrok-domain.ngrok-free.app 5000 --basic-auth 'api_token:your_custom_token' &
while true; do read; done
```
Then I use `screen` to launch the script in the detached mode:
```bash
# api is the session name, -dm is to start the session in detached mode.
screen -dm -S api bash ~/start_api.sh
```
I can check if the `screen` session is running with:
```bash
screen -ls
```
And I can kill the session using:
```bash
screen -X -S api kill
```
But, I'm still battling with the API on the client-side, trying to generate a working prompt…
### Installation of this application
Note that you might need to install espeak on your computer. You can install this application using pip (and git) via the following command:
```bash
python3 -m pip install --force-reinstall seeingllama2
```
Or you can install it from the git repository:
```bash
python3 -m pip install --force-reinstall git+https://github.com/Vincent-Stragier/c-bot.git
```
#### Configuration
To simplify the configuration, multiple settings can be changed using a YAML file. Then you can start the application using the following command:
```bash
seeingllama2 -c /path/to/config.yaml
```
##### config.yaml
```yaml
---
flask:
# Use the debug functionality of Flask
debug: true
# Listen on port 80
port: 80
# Listen from any interface.
# You can use "127.0.0.1" when using a reverse proxy.
host: "0.0.0.0"
# You can configure Flask to use an SSL certificate.
# To generate this certificate, you need an hostname, and
# a service like Letsencrypt to register your domain.
# ssl_keyfile: "/etc/letsencrypt/live/example.com/fullchain.pem"
# ssl_certfile: "/etc/letsencrypt/live/example.com/privkey.pem"
# Here you can change some details of the application
html_index:
title: C-Bot
chat_title: Your chat
chat_number_of_messages: 0 message
chat_send_button_value: Send
chat_input_placeholder: Write your message here
footer_text: >-
© 2023 - C-Bot, handmade by Vincent Stragier,
University of Mons and les Amis des Aveugles, Ghlin
js_app:
bot_name: C-Bot
username: Vini
interface:
locale: en_US.UTF-8
# Here are the pyttsx4 settings
voice:
id: 2
# In word per minute
rate: 200
# A float between 0 and 1.
volume: 1.0
llm_api:
# The API to use
name: oobabooga
# The URL of the API (or its IP address)
url: "my-ngrok-domain.ngrok-free.app"
port: 443
path: "/api/v1/generate"
http_basic_auth:
username: api_token
password: your_custom_token
```
Raw data
{
"_id": null,
"home_page": "https://www.amisdesaveugles.org/comingsoon/",
"name": "seeingllama2",
"maintainer": "",
"docs_url": null,
"requires_python": "<4,>=3.10",
"maintainer_email": "",
"keywords": "",
"author": "Vincent Stragier",
"author_email": "vincent.stragier@outlook.com",
"download_url": "https://files.pythonhosted.org/packages/31/0f/4adef968a379905fa763df03bf03cb1c3f510c0c7dc898b70af5179e48b4/seeingllama2-0.0.6.tar.gz",
"platform": null,
"description": "# Interactive agent for visually impaired and blind individuals (based on Llama2)\r\n\r\nThis project uses a tier application to run the Large Language Model and accesses it using an API. **This is a prototype!**\r\nThe current tier application is based on Oobabooga's text generation webui. In the future support for Petals, or OpenAI API will be added.\r\n\r\n## Installation\r\n\r\nYou first need to setup the tier application server. Then you can install this server application on a different or on the same machine.\r\n\r\n### Installation of the tier application\r\n\r\nHere Oobabooga's application is installed on an Ubuntu server equipped with 2 RTX 3090.\r\n\r\nFor reference, I'm using the [one click installer](https://github.com/oobabooga/text-generation-webui#one-click-installers) with a script to set up the API and a Ngrok tunnel. I installed Ngrok via pip (`python -m pip install pyngrok`).\r\n\r\n```bash\r\n#!/usr/bin/env bash\r\n\r\ntrap_with_arg() { # from https://stackoverflow.com/a/2183063/804678\r\n local func=\"$1\"; shift\r\n for sig in \"$@\"; do\r\n trap \"$func $sig\" \"$sig\"\r\n done\r\n}\r\n\r\nstop() {\r\n trap - SIGINT EXIT\r\n printf '\\n%s\\n' \"received $1, killing child processes\"\r\n kill -s SIGINT 0\r\n}\r\n\r\ntrap_with_arg 'stop' EXIT SIGINT SIGTERM SIGHUP\r\n\r\nexport OOBABOOGA_FLAGS=\"--extension api --loader exllama_hf --model TheBloke_Llama-2-13B-chat-GPTQ --verbose --listen --chat --max_seq_len 4096 --compress_pos_emb 1\"\r\n\r\nbash /path/to/oobabooga_linux/start_linux.sh &\r\n# Wait for the application to start\r\nsleep 30s\r\n\r\n# Start the reverse proxy\r\n# --basic-auth 'api_token:your_custom_token' is optional, it allows using basic access authentication\r\n# The username is 'api_token' and the password is 'your_custom_token'\r\nngrok http --domain=my-ngrok-domain.ngrok-free.app 5000 --basic-auth 'api_token:your_custom_token' &\r\nwhile true; do read; done\r\n\r\n```\r\n\r\nThen I use `screen` to launch the script in the detached mode:\r\n\r\n```bash\r\n# api is the session name, -dm is to start the session in detached mode.\r\nscreen -dm -S api bash ~/start_api.sh\r\n```\r\n\r\nI can check if the `screen` session is running with:\r\n\r\n```bash\r\nscreen -ls\r\n```\r\n\r\nAnd I can kill the session using:\r\n\r\n```bash\r\nscreen -X -S api kill\r\n```\r\n\r\nBut, I'm still battling with the API on the client-side, trying to generate a working prompt\u2026\r\n\r\n### Installation of this application\r\n\r\nNote that you might need to install espeak on your computer. You can install this application using pip (and git) via the following command:\r\n\r\n```bash\r\npython3 -m pip install --force-reinstall seeingllama2\r\n```\r\n\r\nOr you can install it from the git repository:\r\n\r\n```bash\r\npython3 -m pip install --force-reinstall git+https://github.com/Vincent-Stragier/c-bot.git\r\n```\r\n\r\n#### Configuration\r\n\r\nTo simplify the configuration, multiple settings can be changed using a YAML file. Then you can start the application using the following command:\r\n\r\n```bash\r\nseeingllama2 -c /path/to/config.yaml\r\n```\r\n\r\n##### config.yaml\r\n\r\n```yaml\r\n---\r\nflask:\r\n # Use the debug functionality of Flask\r\n debug: true\r\n # Listen on port 80\r\n port: 80\r\n # Listen from any interface.\r\n # You can use \"127.0.0.1\" when using a reverse proxy.\r\n host: \"0.0.0.0\"\r\n # You can configure Flask to use an SSL certificate.\r\n # To generate this certificate, you need an hostname, and\r\n # a service like Letsencrypt to register your domain.\r\n# ssl_keyfile: \"/etc/letsencrypt/live/example.com/fullchain.pem\"\r\n# ssl_certfile: \"/etc/letsencrypt/live/example.com/privkey.pem\"\r\n\r\n# Here you can change some details of the application\r\nhtml_index:\r\n title: C-Bot\r\n chat_title: Your chat\r\n chat_number_of_messages: 0 message\r\n chat_send_button_value: Send\r\n chat_input_placeholder: Write your message here\r\n footer_text: >-\r\n \u00a9 2023 - C-Bot, handmade by Vincent Stragier,\r\n University of Mons and les Amis des Aveugles, Ghlin\r\n\r\njs_app:\r\n bot_name: C-Bot\r\n username: Vini\r\n\r\ninterface:\r\n locale: en_US.UTF-8\r\n\r\n# Here are the pyttsx4 settings\r\nvoice:\r\n id: 2\r\n # In word per minute\r\n rate: 200\r\n # A float between 0 and 1.\r\n volume: 1.0\r\n\r\nllm_api:\r\n # The API to use\r\n name: oobabooga\r\n # The URL of the API (or its IP address)\r\n url: \"my-ngrok-domain.ngrok-free.app\"\r\n port: 443\r\n\r\n path: \"/api/v1/generate\"\r\n http_basic_auth:\r\n username: api_token\r\n password: your_custom_token\r\n```\r\n",
"bugtrack_url": null,
"license": "",
"summary": "A Python interactive agent for visually impaired and blind individuals.",
"version": "0.0.6",
"project_urls": {
"Bug Tracker": "https://github.com/Vincent-Stragier/c-bot/",
"Homepage": "https://www.amisdesaveugles.org/comingsoon/"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "d9e5fe148f6e5b9469a104ac8d90c5475d0904d733c81ac2c519a4bdd300bcdd",
"md5": "836f70f56ada91556f0fbcf1a746947b",
"sha256": "dbad50f8072401b8d0818029098e223df9faea75c0f69ad958bb4400b0722a3c"
},
"downloads": -1,
"filename": "seeingllama2-0.0.6-py3-none-any.whl",
"has_sig": false,
"md5_digest": "836f70f56ada91556f0fbcf1a746947b",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4,>=3.10",
"size": 359689,
"upload_time": "2023-08-27T13:25:36",
"upload_time_iso_8601": "2023-08-27T13:25:36.014236Z",
"url": "https://files.pythonhosted.org/packages/d9/e5/fe148f6e5b9469a104ac8d90c5475d0904d733c81ac2c519a4bdd300bcdd/seeingllama2-0.0.6-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "310f4adef968a379905fa763df03bf03cb1c3f510c0c7dc898b70af5179e48b4",
"md5": "4ae937fb6ec26895309e2dcdf02f4931",
"sha256": "0edd5cf9476edec321e1a3a736fceeed8ee4b6100c8bdab0c1df51a7f121f517"
},
"downloads": -1,
"filename": "seeingllama2-0.0.6.tar.gz",
"has_sig": false,
"md5_digest": "4ae937fb6ec26895309e2dcdf02f4931",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4,>=3.10",
"size": 353030,
"upload_time": "2023-08-27T13:25:39",
"upload_time_iso_8601": "2023-08-27T13:25:39.454017Z",
"url": "https://files.pythonhosted.org/packages/31/0f/4adef968a379905fa763df03bf03cb1c3f510c0c7dc898b70af5179e48b4/seeingllama2-0.0.6.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-08-27 13:25:39",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "Vincent-Stragier",
"github_project": "c-bot",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [
{
"name": "eventlet",
"specs": [
[
"==",
"0.33.3"
]
]
},
{
"name": "Flask",
"specs": [
[
"==",
"2.2.5"
]
]
},
{
"name": "Flask_SocketIO",
"specs": [
[
"==",
"5.3.1"
]
]
},
{
"name": "pyttsx4",
"specs": [
[
"==",
"3.0.15"
]
]
},
{
"name": "PyYAML",
"specs": [
[
"==",
"6.0.1"
]
]
},
{
"name": "setuptools",
"specs": [
[
">=",
"65.5.1"
]
]
}
],
"lcname": "seeingllama2"
}
JSON
