-----------------
# Kaleidoscope-SDK
[](https://kaleidoscope-sdk.readthedocs.io/en/latest/)
A user toolkit for analyzing and interfacing with Large Language Models (LLMs)
## Overview
``kaleidoscope-sdk`` is a Python module used to interact with large language models
hosted via the Kaleidoscope service available at: https://github.com/VectorInstitute/kaleidoscope.
It provides a simple interface to launch LLMs on an HPC cluster, asking them to
perform basic features like text generation, but also retrieve intermediate
information from inside the model, such as log probabilities and activations.
These features are exposed via a few high-level APIs, namely:
* `model_instances` - Shows a list of all active LLMs instantiated by the model service
* `load_model` - Loads an LLM via the model service
* `generate` - Returns an LLM text generation based on prompt input
* `module_names` - Returns all modules names in the LLM neural network
* `get_activations` - Retrieves all activations for a set of modules
## Getting Started
Requires Python version >= 3.8
### Install
```bash
python3 -m pip install kscope
```
or install from source:
```bash
pip install git+https://github.com/VectorInstitute/kaleidoscope-sdk.git
```
### Authentication
In order to submit generation jobs, a designated Vector Institute cluster account is required. Please contact the
[AI Engineering Team](mailto:ai_engineering@vectorinstitute.ai?subject=[Github]%20Kaleidoscope)
in charge of Kaleidoscope for more information.
### Sample Workflow
The following workflow shows how to load and interact with an OPT-175B model
on the Vector Institute Vaughan cluster.
```python
#!/usr/bin/env python3
import kscope
import time
# Establish a client connection to the Kaleidoscope service
# If you have not previously authenticated with the service, you will be prompted to now
client = kscope.Client(gateway_host="llm.cluster.local", gateway_port=3001)
# See which models are supported
client.models
# See which models are instantiated and available to use
client.model_instances
# Get a handle to a model. If this model is not actively running, it will get launched in the background.
# In this example we want to use the LLama2 7b model
llama2_model = client.load_model("llama2-7b")
# If the model was not actively running, this it could take several minutes to load. Wait for it come online.
while llama2_model.state != "ACTIVE":
time.sleep(1)
# Sample text generation w/ input parameters
text_gen = llama2_model.generate("What is Vector Institute?", {'max_tokens': 5, 'top_k': 4, 'temperature': 0.5})
dir(text_gen) # display methods associated with generated text object
text_gen.generation['sequences'] # display only text
text_gen.generation['logprobs'] # display logprobs
text_gen.generation['tokens'] # display tokens
# Now let's retrieve some activations from the model
# First, show a list of modules in the neural network
print(llama2_model.module_names)
# Setup a request for module acivations for a certain module layer
requested_activations = ['layers.0']
activations = llama2_model.get_activations("What are activations?", requested_activations)
print(activations)
# Next, let's manipulate the activations in the model. First, we need to import a few more modules.
import cloudpickle
import codecs
import torch
from torch import Tensor
from typing import Callable, Dict
# Define a function to manipulate the activations
def replace_with_ones(act: Tensor) -> Tensor:
"""Replace an activation with an activation filled with ones."""
out = torch.ones_like(act, dtype=act.dtype).cuda()
return out
# Now send the edit request
editing_fns: Dict[str, Callable] = {}
editing_fns['layers.0'] = replace_with_ones
edited_activations = llama2_model.edit_activations("What is Vector Institute?", editing_fns)
print(edited_activations)
```
## Documentation
Full documentation and API reference are available at: http://kaleidoscope-sdk.readthedocs.io.
## Contributing
Contributing to kaleidoscope is welcomed. See [Contributing](CONTRIBUTING) for
guidelines.
## License
[MIT](LICENSE)
## Citation
Reference to cite when you use Kaleidoscope in a project or a research paper:
```
Willes, J., Choi, M., Coatsworth, M., Shen, G., & Sivaloganathan, J (2022). Kaleidoscope. http://VectorInstitute.github.io/kaleidoscope. computer software, Vector Institute for Artificial Intelligence. Retrieved from https://github.com/VectorInstitute/kaleidoscope-sdk.git.
```
Raw data
{
"_id": null,
"home_page": "https://github.com/VectorInstitute/kaleidoscope-sdk",
"name": "kscope",
"maintainer": "",
"docs_url": null,
"requires_python": "",
"maintainer_email": "",
"keywords": "python nlp machine-learning deep-learning distributed-computing neural-networks tensor llm",
"author": "['Vector AI Engineering']",
"author_email": "ai_engineering@vectorinstitute.ai",
"download_url": "https://files.pythonhosted.org/packages/6c/bd/90ecff6a24fad8df9e5d46790be16ef8c03bb3817c0299d14787151b406c/kscope-0.10.0.tar.gz",
"platform": null,
"description": "\n-----------------\n# Kaleidoscope-SDK\n\n\n\n\n[](https://kaleidoscope-sdk.readthedocs.io/en/latest/)\n\nA user toolkit for analyzing and interfacing with Large Language Models (LLMs)\n\n\n## Overview\n\n``kaleidoscope-sdk`` is a Python module used to interact with large language models\nhosted via the Kaleidoscope service available at: https://github.com/VectorInstitute/kaleidoscope.\nIt provides a simple interface to launch LLMs on an HPC cluster, asking them to\nperform basic features like text generation, but also retrieve intermediate\ninformation from inside the model, such as log probabilities and activations.\nThese features are exposed via a few high-level APIs, namely:\n\n* `model_instances` - Shows a list of all active LLMs instantiated by the model service\n* `load_model` - Loads an LLM via the model service\n* `generate` - Returns an LLM text generation based on prompt input\n* `module_names` - Returns all modules names in the LLM neural network\n* `get_activations` - Retrieves all activations for a set of modules\n\n\n\n## Getting Started\n\nRequires Python version >= 3.8\n\n### Install\n\n```bash\npython3 -m pip install kscope\n```\nor install from source:\n\n```bash\npip install git+https://github.com/VectorInstitute/kaleidoscope-sdk.git\n```\n\n### Authentication\n\nIn order to submit generation jobs, a designated Vector Institute cluster account is required. Please contact the\n[AI Engineering Team](mailto:ai_engineering@vectorinstitute.ai?subject=[Github]%20Kaleidoscope)\nin charge of Kaleidoscope for more information.\n\n### Sample Workflow\n\nThe following workflow shows how to load and interact with an OPT-175B model\non the Vector Institute Vaughan cluster.\n\n```python\n#!/usr/bin/env python3\nimport kscope\nimport time\n\n# Establish a client connection to the Kaleidoscope service\n# If you have not previously authenticated with the service, you will be prompted to now\nclient = kscope.Client(gateway_host=\"llm.cluster.local\", gateway_port=3001)\n\n# See which models are supported\nclient.models\n\n# See which models are instantiated and available to use\nclient.model_instances\n\n# Get a handle to a model. If this model is not actively running, it will get launched in the background.\n# In this example we want to use the LLama2 7b model\nllama2_model = client.load_model(\"llama2-7b\")\n\n# If the model was not actively running, this it could take several minutes to load. Wait for it come online.\nwhile llama2_model.state != \"ACTIVE\":\n time.sleep(1)\n\n# Sample text generation w/ input parameters\ntext_gen = llama2_model.generate(\"What is Vector Institute?\", {'max_tokens': 5, 'top_k': 4, 'temperature': 0.5})\ndir(text_gen) # display methods associated with generated text object\ntext_gen.generation['sequences'] # display only text\ntext_gen.generation['logprobs'] # display logprobs\ntext_gen.generation['tokens'] # display tokens\n\n# Now let's retrieve some activations from the model\n# First, show a list of modules in the neural network\nprint(llama2_model.module_names)\n\n# Setup a request for module acivations for a certain module layer\nrequested_activations = ['layers.0']\nactivations = llama2_model.get_activations(\"What are activations?\", requested_activations)\nprint(activations)\n\n# Next, let's manipulate the activations in the model. First, we need to import a few more modules.\nimport cloudpickle\nimport codecs\nimport torch\nfrom torch import Tensor\nfrom typing import Callable, Dict\n\n# Define a function to manipulate the activations\ndef replace_with_ones(act: Tensor) -> Tensor:\n \"\"\"Replace an activation with an activation filled with ones.\"\"\"\n out = torch.ones_like(act, dtype=act.dtype).cuda()\n return out\n\n# Now send the edit request\nediting_fns: Dict[str, Callable] = {}\nediting_fns['layers.0'] = replace_with_ones\nedited_activations = llama2_model.edit_activations(\"What is Vector Institute?\", editing_fns)\nprint(edited_activations)\n\n```\n\n## Documentation\nFull documentation and API reference are available at: http://kaleidoscope-sdk.readthedocs.io.\n\n\n## Contributing\nContributing to kaleidoscope is welcomed. See [Contributing](CONTRIBUTING) for\nguidelines.\n\n\n## License\n[MIT](LICENSE)\n\n\n## Citation\nReference to cite when you use Kaleidoscope in a project or a research paper:\n```\nWilles, J., Choi, M., Coatsworth, M., Shen, G., & Sivaloganathan, J (2022). Kaleidoscope. http://VectorInstitute.github.io/kaleidoscope. computer software, Vector Institute for Artificial Intelligence. Retrieved from https://github.com/VectorInstitute/kaleidoscope-sdk.git.\n```\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "A user toolkit for analyzing and interfacing with Large Language Models (LLMs)",
"version": "0.10.0",
"project_urls": {
"Homepage": "https://github.com/VectorInstitute/kaleidoscope-sdk"
},
"split_keywords": [
"python",
"nlp",
"machine-learning",
"deep-learning",
"distributed-computing",
"neural-networks",
"tensor",
"llm"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "ee5c4ef3cb5651fcee70c419ce48ef82b1ea524e6272c9ca87b32c7c1e36eced",
"md5": "6174058bb4dd941ab48e996f50e9422b",
"sha256": "9031dd2a4624145a9441478e7aff2dc59c073cac83a7f834c4b3fda703b32b73"
},
"downloads": -1,
"filename": "kscope-0.10.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "6174058bb4dd941ab48e996f50e9422b",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 8299,
"upload_time": "2023-10-30T19:46:12",
"upload_time_iso_8601": "2023-10-30T19:46:12.292720Z",
"url": "https://files.pythonhosted.org/packages/ee/5c/4ef3cb5651fcee70c419ce48ef82b1ea524e6272c9ca87b32c7c1e36eced/kscope-0.10.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "6cbd90ecff6a24fad8df9e5d46790be16ef8c03bb3817c0299d14787151b406c",
"md5": "df5868b701710ff02aef901f40ddfc4c",
"sha256": "024cb04ff4473235cfeb12665dac32fdd1fe96282cf15d9e4b6597561177ad42"
},
"downloads": -1,
"filename": "kscope-0.10.0.tar.gz",
"has_sig": false,
"md5_digest": "df5868b701710ff02aef901f40ddfc4c",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 8097,
"upload_time": "2023-10-30T19:46:13",
"upload_time_iso_8601": "2023-10-30T19:46:13.821561Z",
"url": "https://files.pythonhosted.org/packages/6c/bd/90ecff6a24fad8df9e5d46790be16ef8c03bb3817c0299d14787151b406c/kscope-0.10.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-10-30 19:46:13",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "VectorInstitute",
"github_project": "kaleidoscope-sdk",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "kscope"
}
JSON
