# Long Range Models
[](https://badge.fury.io/py/long_range_models)
[](https://github.com/google/flax)
[](/LICENSE)
A collection of simple implementations of long-range sequence models, including [LRU](/long_range_models/lru.py), [S5](/long_range_models/s5.py), and [S4](/long_range_models/s4.py).
More implementations to come.
## Install
```bash
$ pip install long_range_models
```
## Usage
This library offers detailed documentation for every module and layer implemented.
Models are created by composing different pieces together.
Check out the examples below.
### Discrete sequence data
Consider a language model built with an LRU sequence layer and the architecture proposed in the S4 paper:
```py
from functools import partial
import jax.random as jrandom
from long_range_models import SequenceModel, S4Module, LRULayer
rng = jrandom.PRNGKey(0)
model = SequenceModel(
num_tokens=1000,
module=S4Module(
sequence_layer=partial(LRULayer, state_dim=256),
dim=128,
depth=6,
),
)
x = jrandom.randint(rng, (1, 1024), 0, 1000)
variables = model.init(rng, x)
model.apply(variables, x) # (1, 1024, 1000)
```
### Continuous sequence data
For sequences with continuous values, the setup looks as follows:
```py
from functools import partial
import jax.random as jrandom
from long_range_models import ContinuousSequenceModel, S4Module, LRULayer
rng = jrandom.PRNGKey(0)
model = ContinuousSequenceModel(
out_dim=10,
module=S4Module(
sequence_layer=partial(LRULayer, state_dim=256),
dim=128,
depth=6,
),
)
x = jrandom.normal(rng, (1, 1024, 32))
variables = model.init(rng, x)
model.apply(variables, x) # (1, 1024, 10)
```
**Note:** both model types offer several customization options. Make sure to check out [their documentation](/long_range_models/sequence_models.py).
## Upcoming features
- **More implementations:** Extend the library with models like S4D, S4Liquid, BiGS, Hyena, RetNet, SGConv, H3, and others.
- **Customization:** Allow users to better customize currently implemented layers and architectures (e.g., activation functions, initialization, etc.).
- **Sequential API:** Allow recurrent models to run sequentially, allowing for efficient inference.
Raw data
{
"_id": null,
"home_page": "https://github.com/gbrlfaria/long_range_models",
"name": "long-range-models",
"maintainer": "",
"docs_url": null,
"requires_python": "",
"maintainer_email": "",
"keywords": "artificial intelligence,state space models,long range models",
"author": "Gabriel Faria",
"author_email": "gabfaria.cs@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/f0/c9/09c3991abdad63eae9648a07c04bfa9fcf89f223670e9215a95d214cb602/long_range_models-0.0.1.tar.gz",
"platform": null,
"description": "# Long Range Models\n\n[](https://badge.fury.io/py/long_range_models)\n[](https://github.com/google/flax)\n[](/LICENSE)\n\n\nA collection of simple implementations of long-range sequence models, including [LRU](/long_range_models/lru.py), [S5](/long_range_models/s5.py), and [S4](/long_range_models/s4.py).\nMore implementations to come.\n\n## Install\n\n```bash\n$ pip install long_range_models\n```\n\n## Usage\n\nThis library offers detailed documentation for every module and layer implemented.\nModels are created by composing different pieces together.\nCheck out the examples below.\n\n### Discrete sequence data\n\nConsider a language model built with an LRU sequence layer and the architecture proposed in the S4 paper:\n\n```py\nfrom functools import partial\nimport jax.random as jrandom\nfrom long_range_models import SequenceModel, S4Module, LRULayer\n\nrng = jrandom.PRNGKey(0)\n\nmodel = SequenceModel(\n num_tokens=1000,\n module=S4Module(\n sequence_layer=partial(LRULayer, state_dim=256),\n dim=128,\n depth=6,\n ),\n)\n\nx = jrandom.randint(rng, (1, 1024), 0, 1000)\n\nvariables = model.init(rng, x)\nmodel.apply(variables, x) # (1, 1024, 1000)\n\n```\n\n### Continuous sequence data\n\nFor sequences with continuous values, the setup looks as follows:\n\n```py\nfrom functools import partial\nimport jax.random as jrandom\nfrom long_range_models import ContinuousSequenceModel, S4Module, LRULayer\n\nrng = jrandom.PRNGKey(0)\n\nmodel = ContinuousSequenceModel(\n out_dim=10,\n module=S4Module(\n sequence_layer=partial(LRULayer, state_dim=256),\n dim=128,\n depth=6,\n ),\n)\n\nx = jrandom.normal(rng, (1, 1024, 32))\n\nvariables = model.init(rng, x)\nmodel.apply(variables, x) # (1, 1024, 10)\n\n```\n\n**Note:** both model types offer several customization options. Make sure to check out [their documentation](/long_range_models/sequence_models.py).\n\n## Upcoming features\n\n- **More implementations:** Extend the library with models like S4D, S4Liquid, BiGS, Hyena, RetNet, SGConv, H3, and others.\n- **Customization:** Allow users to better customize currently implemented layers and architectures (e.g., activation functions, initialization, etc.).\n- **Sequential API:** Allow recurrent models to run sequentially, allowing for efficient inference.\n",
"bugtrack_url": null,
"license": "",
"summary": "Simple Flax implementations of long-range sequence models",
"version": "0.0.1",
"project_urls": {
"Homepage": "https://github.com/gbrlfaria/long_range_models"
},
"split_keywords": [
"artificial intelligence",
"state space models",
"long range models"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "c2ffd0ab4547344bb3d6fcf90cad0e96a1c820d421ea0feb9b7afd288ff0b493",
"md5": "c90f6b9c4230140ba6a9f18b73a88dfc",
"sha256": "0a181dc51f44b79e6d375e5d758c650ae4660ab90b9300f053d19ebe2f07e0d6"
},
"downloads": -1,
"filename": "long_range_models-0.0.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "c90f6b9c4230140ba6a9f18b73a88dfc",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 10865,
"upload_time": "2023-09-15T18:34:31",
"upload_time_iso_8601": "2023-09-15T18:34:31.343741Z",
"url": "https://files.pythonhosted.org/packages/c2/ff/d0ab4547344bb3d6fcf90cad0e96a1c820d421ea0feb9b7afd288ff0b493/long_range_models-0.0.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "f0c909c3991abdad63eae9648a07c04bfa9fcf89f223670e9215a95d214cb602",
"md5": "3c38eab5b1d81ebca87865c2e4f2d16e",
"sha256": "8e58431831f8c08ea4d6c7f75a6fccd1274cdb046f28568787535ec140904f42"
},
"downloads": -1,
"filename": "long_range_models-0.0.1.tar.gz",
"has_sig": false,
"md5_digest": "3c38eab5b1d81ebca87865c2e4f2d16e",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 9479,
"upload_time": "2023-09-15T18:34:32",
"upload_time_iso_8601": "2023-09-15T18:34:32.634395Z",
"url": "https://files.pythonhosted.org/packages/f0/c9/09c3991abdad63eae9648a07c04bfa9fcf89f223670e9215a95d214cb602/long_range_models-0.0.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-09-15 18:34:32",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "gbrlfaria",
"github_project": "long_range_models",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "long-range-models"
}
JSON
