# Language Pipes (Beta)
**Distribute language models across multiple systems**
[![GitHub license][License-Image]](License-Url)
[![Release][Release-Image]][Release-Url]
[License-Image]: https://img.shields.io/badge/license-MIT-blue.svg
[License-Url]: https://github.com/erinclemmer/language-pipes/blob/main/LICENSE
[Release-Url]: https://github.com/erinclemmer/language-pipes/releases/latest
[Release-Image]: https://img.shields.io/github/v/release/erinclemmer/language-pipes
[PyPiVersion-Url]: https://img.shields.io/pypi/v/language-pipes
[PythonVersion-Url]: https://img.shields.io/pypi/pyversions/language-pipes
Language pipes is a FOSS distributed network application designed to increase access to local language models.
[Join our Discord](https://discord.gg/CPvC78E53a) for any comments or questions!
---
**Disclaimer:** This software is currently in Beta. Please be patient and if you encounter an error, please [fill out a github issue](https://github.com/erinclemmer/language-pipes/issues/new)!
Over the past few years open source language models have become much more powerful yet the most powerful models are still out of reach of the general population because of the extreme amounts of RAM that is needed to host these models. Language Pipes allows multiple computer systems to host the same model and move computation data between them so that no one computer has to hold all of the data for the model.
- Quick Setup
- Peer to peer network
- OpenAI compatible API
- Download and use models by HuggingFace ID
- Encrypted communication between nodes
### What Does it do?
In a basic sense, language models work by passing information through many layers. At each layer, several matrix multiplicatitons between the layer weights and the system state are performed and the data is moved to the next layer. Language pipes works by hosting different layers on different machines to split up the RAM cost across the system.
#### How is this different from Distributed Llama?
[Distributed Llama](https://github.com/b4rtaz/distributed-llama) is built to be a static network and requires individual setup and allocation for each model hosted. Language Pipes meanwhile, has a more flexible setup process that automatically selects which parts of the model to load based on what the network needs and the local systems resources. This allows separate users to collectively host a network together while maintaining trust that one configuration will not break the network. Users can come and go from the network and many different models can be hosted at the same time.
### Installation
Ensure that you have Python 3.10.18 (or any 3.10 version) installed. For an easy to use Python version manager use [pyenv](https://github.com/pyenv/pyenv). This specific version is necessary for the [transformers](https://github.com/huggingface/transformers) library to work properly.
If you need gpu support, first make sure you have the correct pytorch version installed for your GPU's Cuda compatibility using this link:
https://pytorch.org/get-started/locally/
To download the models from Huggingface, ensure that you have [git](https://git-scm.com/) and [git lfs](https://git-lfs.com/) installed.
To start using the application, install the latest version of the package from PyPi.
Using Pip:
```bash
pip install language-pipes
```
# Two Node Example
The following example will show how to create a small network. Firstly, create a network key for the network on the first computer:
```bash
language-pipes create_key network.key
```
Also create a `config.toml` file to tell the program how to operate:
```toml
node_id="node-1"
oai_port=6000 # Hosts an OpenAI compatible server on port 6000
[[hosted_models]]
id="Qwen/Qwen3-1.7B"
device="cpu"
max_memory=1
```
**Note:** Go to the [configuration documentation](/documentation/configuration.md) for more information about how the config properties work.
Once the configuration has been created you can start the server:
```bash
language-pipes run --config config.toml
```
This tells language pipes to download with the ID "Qwen/Qwen3-1.7B" from [huggingface.co](huggingface.co) and host it using 1GB of ram. This will load part of the model but not all of it.
Next, install the package on a separate computer on your home network and create a `config.toml` file like this:
```toml
node_id="node-2"
bootstrap_address="192.168.0.10" # Local ip address of node-1
[[hosted_models]]
id="Qwen/Qwen3-1.7B"
device="cpu"
max_memory=3
```
Copy the `network.key` file to the same directory that the config is in using a usb drive or sftp.
Run the same command again on the computer two:
```bash
language-pipes run --config config.toml
```
Node-2 will connect to node-1 and load the remaining parts of the model. The model is ready for inference using a [standard openai chat API interface](https://platform.openai.com/docs/api-reference/chat/create). An example request to the server is provided below:
```python
import requests
import json
# node-1 IP address here
url = "http://127.0.0.1:6000/v1/chat/completions"
headers = {
"Content-Type": "application/json"
}
payload = {
"model": "Qwen/Qwen3-1.7B",
"max_completion_tokens": 10,
"messages": [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Write a haiku about distributed systems."}
]
}
response = requests.post(url, headers=headers, data=json.dumps(payload))
print("Status Code:", response.status_code)
print("Response JSON:", response.json())
```
### Models Supported
* Llama 2 & Llama 3.X
* Qwen3
* More to come!
### Dependencies
- [pytorch](pytorch.org)
- [transformers](https://huggingface.co/docs/transformers)
- [llm-layer-collector](https://github.com/erinclemmer/llm-layer-collector)
- [distributed-state-network](https://github.com/erinclemmer/distributed_state_network)
### Citation
If you use the project for an academic endeavour please use this citation.
```latex
@software{Clemmer_Language_Pipes_2025,
author = {Erin Clemmer},
title = {Language Pipes},
abstract = {Distribute language models across multiple systems.},
version = {0.0.1},
date = {2025-09-01},
url = {https://github.com/erinclemmer/language-pipes},
keywords = {Large Language Models, Transformers, Distributed Networks},
license = {MIT},
orcid = {0009-0005-0597-6197}
}
```
Raw data
{
"_id": null,
"home_page": null,
"name": "language-pipes",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.10",
"maintainer_email": null,
"keywords": "AI, Language Model, distributed, networking, pipe, pipeline, server",
"author": null,
"author_email": "Erin Clemmer <erin.c.clemmer@gmail.com>",
"download_url": "https://files.pythonhosted.org/packages/49/38/9d541424d482d320901cf4369022fb7e15932c0af25b0f801069221dd822/language_pipes-0.1.0.tar.gz",
"platform": null,
"description": "# Language Pipes (Beta)\n\n**Distribute language models across multiple systems** \n\n[![GitHub license][License-Image]](License-Url)\n[![Release][Release-Image]][Release-Url] \n\n\n\n[License-Image]: https://img.shields.io/badge/license-MIT-blue.svg\n[License-Url]: https://github.com/erinclemmer/language-pipes/blob/main/LICENSE\n\n[Release-Url]: https://github.com/erinclemmer/language-pipes/releases/latest\n[Release-Image]: https://img.shields.io/github/v/release/erinclemmer/language-pipes\n\n[PyPiVersion-Url]: https://img.shields.io/pypi/v/language-pipes\n[PythonVersion-Url]: https://img.shields.io/pypi/pyversions/language-pipes\n\nLanguage pipes is a FOSS distributed network application designed to increase access to local language models. \n\n[Join our Discord](https://discord.gg/CPvC78E53a) for any comments or questions! \n\n--- \n\n**Disclaimer:** This software is currently in Beta. Please be patient and if you encounter an error, please [fill out a github issue](https://github.com/erinclemmer/language-pipes/issues/new)! \n\nOver the past few years open source language models have become much more powerful yet the most powerful models are still out of reach of the general population because of the extreme amounts of RAM that is needed to host these models. Language Pipes allows multiple computer systems to host the same model and move computation data between them so that no one computer has to hold all of the data for the model.\n- Quick Setup\n- Peer to peer network\n- OpenAI compatible API\n- Download and use models by HuggingFace ID\n- Encrypted communication between nodes\n\n### What Does it do?\nIn a basic sense, language models work by passing information through many layers. At each layer, several matrix multiplicatitons between the layer weights and the system state are performed and the data is moved to the next layer. Language pipes works by hosting different layers on different machines to split up the RAM cost across the system.\n\n#### How is this different from Distributed Llama?\n[Distributed Llama](https://github.com/b4rtaz/distributed-llama) is built to be a static network and requires individual setup and allocation for each model hosted. Language Pipes meanwhile, has a more flexible setup process that automatically selects which parts of the model to load based on what the network needs and the local systems resources. This allows separate users to collectively host a network together while maintaining trust that one configuration will not break the network. Users can come and go from the network and many different models can be hosted at the same time.\n\n### Installation\nEnsure that you have Python 3.10.18 (or any 3.10 version) installed. For an easy to use Python version manager use [pyenv](https://github.com/pyenv/pyenv). This specific version is necessary for the [transformers](https://github.com/huggingface/transformers) library to work properly. \n \nIf you need gpu support, first make sure you have the correct pytorch version installed for your GPU's Cuda compatibility using this link: \nhttps://pytorch.org/get-started/locally/\n\nTo download the models from Huggingface, ensure that you have [git](https://git-scm.com/) and [git lfs](https://git-lfs.com/) installed. \n\nTo start using the application, install the latest version of the package from PyPi.\n\nUsing Pip:\n```bash\npip install language-pipes\n```\n\n# Two Node Example\nThe following example will show how to create a small network. Firstly, create a network key for the network on the first computer:\n```bash\nlanguage-pipes create_key network.key\n```\n\nAlso create a `config.toml` file to tell the program how to operate:\n\n```toml\nnode_id=\"node-1\"\noai_port=6000 # Hosts an OpenAI compatible server on port 6000\n\n[[hosted_models]]\nid=\"Qwen/Qwen3-1.7B\"\ndevice=\"cpu\"\nmax_memory=1\n```\n\n**Note:** Go to the [configuration documentation](/documentation/configuration.md) for more information about how the config properties work.\n\nOnce the configuration has been created you can start the server:\n```bash\nlanguage-pipes run --config config.toml\n```\n\nThis tells language pipes to download with the ID \"Qwen/Qwen3-1.7B\" from [huggingface.co](huggingface.co) and host it using 1GB of ram. This will load part of the model but not all of it.\n\nNext, install the package on a separate computer on your home network and create a `config.toml` file like this:\n\n```toml\nnode_id=\"node-2\"\nbootstrap_address=\"192.168.0.10\" # Local ip address of node-1\n\n[[hosted_models]]\nid=\"Qwen/Qwen3-1.7B\"\ndevice=\"cpu\"\nmax_memory=3\n```\n\nCopy the `network.key` file to the same directory that the config is in using a usb drive or sftp. \n\nRun the same command again on the computer two:\n```bash\nlanguage-pipes run --config config.toml\n```\n\nNode-2 will connect to node-1 and load the remaining parts of the model. The model is ready for inference using a [standard openai chat API interface](https://platform.openai.com/docs/api-reference/chat/create). An example request to the server is provided below:\n\n```python\nimport requests\nimport json\n\n# node-1 IP address here\nurl = \"http://127.0.0.1:6000/v1/chat/completions\"\n\nheaders = {\n \"Content-Type\": \"application/json\"\n}\n\npayload = {\n \"model\": \"Qwen/Qwen3-1.7B\",\n \"max_completion_tokens\": 10,\n \"messages\": [\n {\"role\": \"system\", \"content\": \"You are a helpful assistant.\"},\n {\"role\": \"user\", \"content\": \"Write a haiku about distributed systems.\"}\n ]\n}\n\nresponse = requests.post(url, headers=headers, data=json.dumps(payload))\n\nprint(\"Status Code:\", response.status_code)\nprint(\"Response JSON:\", response.json())\n```\n\n### Models Supported\n* Llama 2 & Llama 3.X \n* Qwen3\n* More to come!\n\n### Dependencies\n- [pytorch](pytorch.org)\n- [transformers](https://huggingface.co/docs/transformers) \n- [llm-layer-collector](https://github.com/erinclemmer/llm-layer-collector)\n- [distributed-state-network](https://github.com/erinclemmer/distributed_state_network)\n\n### Citation\nIf you use the project for an academic endeavour please use this citation.\n\n```latex\n@software{Clemmer_Language_Pipes_2025,\n author = {Erin Clemmer},\n title = {Language Pipes},\n abstract = {Distribute language models across multiple systems.},\n version = {0.0.1},\n date = {2025-09-01},\n url = {https://github.com/erinclemmer/language-pipes},\n keywords = {Large Language Models, Transformers, Distributed Networks},\n license = {MIT},\n orcid = {0009-0005-0597-6197}\n}\n```",
"bugtrack_url": null,
"license": null,
"summary": "Easily distribute language models across multiple systems",
"version": "0.1.0",
"project_urls": {
"Homepage": "https://github.com/erinclemmer/language-pipes",
"Issues": "https://github.com/erinclemmer/language-pipes/issues"
},
"split_keywords": [
"ai",
" language model",
" distributed",
" networking",
" pipe",
" pipeline",
" server"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "6916b7e1a2fa7730ec55299f1d4f42a479a760a301730e88e4de472407584139",
"md5": "40eada59bca1bc70e4bd1904d2085e9a",
"sha256": "a04543bae1188bf7bc1a2c2d7eb15162ba87b56ea5246edec5c2f2259e600ba0"
},
"downloads": -1,
"filename": "language_pipes-0.1.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "40eada59bca1bc70e4bd1904d2085e9a",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.10",
"size": 29615,
"upload_time": "2025-09-14T19:47:23",
"upload_time_iso_8601": "2025-09-14T19:47:23.378650Z",
"url": "https://files.pythonhosted.org/packages/69/16/b7e1a2fa7730ec55299f1d4f42a479a760a301730e88e4de472407584139/language_pipes-0.1.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "49389d541424d482d320901cf4369022fb7e15932c0af25b0f801069221dd822",
"md5": "7a263b1f4b0be389c2eda8208f599535",
"sha256": "774ee5d6681f63932304e52badf387cbfa1f548c4349892b421f93bc804f8eb6"
},
"downloads": -1,
"filename": "language_pipes-0.1.0.tar.gz",
"has_sig": false,
"md5_digest": "7a263b1f4b0be389c2eda8208f599535",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.10",
"size": 24197,
"upload_time": "2025-09-14T19:47:24",
"upload_time_iso_8601": "2025-09-14T19:47:24.766432Z",
"url": "https://files.pythonhosted.org/packages/49/38/9d541424d482d320901cf4369022fb7e15932c0af25b0f801069221dd822/language_pipes-0.1.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-09-14 19:47:24",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "erinclemmer",
"github_project": "language-pipes",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "language-pipes"
}
JSON
