# Embedme
Embedme is a python module that allows you to easily use embeddings from text fields with OpenAI's Embedding API and store them in a local folder.
It's like a lazy version of pinecone - Numpy is actually pretty fast for embeddings stuff at smaller scale, why overthink stuff? We store the data and vectors as json and build the numpy array before you search (and store it until you add more)
## Installation
To install Embedme, you can use pip:
```sh
pip install embedme
```
## Setup
The only thing you _must_ do before you use `embedme` is setup auth with OpenAI. We use it to embed your items and search queries, so it is required. I don't want to touch **any** of that code - just sign in how they tell you to, either in the script via a file for the key, or an environment variable for your key.
[OpenAI Python Module (With Auth Instructions)](https://github.com/openai/openai-python)
## Usage
Embedme provides a simple interface to use embeddings from text fields with OpenAI's Embedding API and store them in a local folder.
Check out the example notebook for a better example, but useage is something like:
```py
import openai
import nltk
from more_itertools import chunked
from embedme import Embedme
from tqdm import tqdm
# Downloading the NLTK corpus
nltk.download('gutenberg')
# Creating an instance of the Embedme class
embedme = Embedme(data_folder='.embedme', model="text-embedding-ada-002")
# Getting the text
text = nltk.corpus.gutenberg.raw('melville-moby_dick.txt')
# Splitting the text into sentences
sentences = nltk.sent_tokenize(text)
input("Hey this call will cost you money and take a minute. Like, a few cents probably, but wanted to warn you.")
for i, chunk in enumerate(tqdm(chunked(sentences, 20))):
data = {'name': f'moby_dick_chunk_{i}', 'text': ' '.join(chunk)}
embedme.add(data, save=False)
embedme.save()
```
And to search:
```py
embedme.search("lessons")
```
You can do anything you would want to with `.vectors` after you call `.prepare_search()` (or... search something, it's automatic mostly), like plot clusters, etc.
## Follow Us
Some friends and I are writing about large language model stuff at [SensibleDefaults.io](https://sensibledefaults.io), honest to god free. Follow us (or star this repo!) if this helps you!
## Note
Embedme uses OpenAI's Embedding API to get embeddings for text fields, so an API key is required to use it. You can get one from https://beta.openai.com/signup/
The token limit today is about 8k, so... you're probably fine
Raw data
{
"_id": null,
"home_page": "https://github.com/morganpartee/embedme",
"name": "embedme",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.10,<4.0",
"maintainer_email": "",
"keywords": "nlp,embeddings,search,openai,gpt3,AI,machine learning",
"author": "John Partee",
"author_email": "",
"download_url": "https://files.pythonhosted.org/packages/76/06/fa9ed815bd7325b42d038f2e1629643e007eabde9a0289129f29823a50bb/embedme-0.1.1.tar.gz",
"platform": null,
"description": "# Embedme\n\nEmbedme is a python module that allows you to easily use embeddings from text fields with OpenAI's Embedding API and store them in a local folder.\n\nIt's like a lazy version of pinecone - Numpy is actually pretty fast for embeddings stuff at smaller scale, why overthink stuff? We store the data and vectors as json and build the numpy array before you search (and store it until you add more)\n\n## Installation\n\nTo install Embedme, you can use pip:\n\n```sh\npip install embedme\n```\n\n## Setup\n\nThe only thing you _must_ do before you use `embedme` is setup auth with OpenAI. We use it to embed your items and search queries, so it is required. I don't want to touch **any** of that code - just sign in how they tell you to, either in the script via a file for the key, or an environment variable for your key.\n\n[OpenAI Python Module (With Auth Instructions)](https://github.com/openai/openai-python)\n\n## Usage\n\nEmbedme provides a simple interface to use embeddings from text fields with OpenAI's Embedding API and store them in a local folder.\n\nCheck out the example notebook for a better example, but useage is something like:\n\n```py\nimport openai\nimport nltk\nfrom more_itertools import chunked\nfrom embedme import Embedme\nfrom tqdm import tqdm\n\n# Downloading the NLTK corpus\nnltk.download('gutenberg')\n\n# Creating an instance of the Embedme class\nembedme = Embedme(data_folder='.embedme', model=\"text-embedding-ada-002\")\n\n# Getting the text\ntext = nltk.corpus.gutenberg.raw('melville-moby_dick.txt')\n\n# Splitting the text into sentences\nsentences = nltk.sent_tokenize(text)\n\ninput(\"Hey this call will cost you money and take a minute. Like, a few cents probably, but wanted to warn you.\")\n\nfor i, chunk in enumerate(tqdm(chunked(sentences, 20))):\n data = {'name': f'moby_dick_chunk_{i}', 'text': ' '.join(chunk)}\n embedme.add(data, save=False)\n\nembedme.save()\n```\n\nAnd to search:\n\n```py\nembedme.search(\"lessons\")\n```\n\nYou can do anything you would want to with `.vectors` after you call `.prepare_search()` (or... search something, it's automatic mostly), like plot clusters, etc.\n\n## Follow Us\n\nSome friends and I are writing about large language model stuff at [SensibleDefaults.io](https://sensibledefaults.io), honest to god free. Follow us (or star this repo!) if this helps you!\n\n## Note\n\nEmbedme uses OpenAI's Embedding API to get embeddings for text fields, so an API key is required to use it. You can get one from https://beta.openai.com/signup/\n\nThe token limit today is about 8k, so... you're probably fine\n",
"bugtrack_url": null,
"license": "GPL-3.0-or-later",
"summary": "Easily create and search text embeddings using OpenAI's API using json for local storage. Just add dicts of info and search! Built for rapid prototyping.",
"version": "0.1.1",
"split_keywords": [
"nlp",
"embeddings",
"search",
"openai",
"gpt3",
"ai",
"machine learning"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "43a92e802e4f4d5222dd8ab82fa750fbb931c20fb682af4aa1a99abe4b373e64",
"md5": "06cd6f6718d51a74d5eed9d8964e78a7",
"sha256": "9911dd37b129ecae9b5bcd702d5408948645335b07bc4352d5bab65e4c36e973"
},
"downloads": -1,
"filename": "embedme-0.1.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "06cd6f6718d51a74d5eed9d8964e78a7",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.10,<4.0",
"size": 4539,
"upload_time": "2023-01-22T19:57:10",
"upload_time_iso_8601": "2023-01-22T19:57:10.983109Z",
"url": "https://files.pythonhosted.org/packages/43/a9/2e802e4f4d5222dd8ab82fa750fbb931c20fb682af4aa1a99abe4b373e64/embedme-0.1.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "7606fa9ed815bd7325b42d038f2e1629643e007eabde9a0289129f29823a50bb",
"md5": "9ca5a6c62add31dd0bb0710b727dddce",
"sha256": "d8b0632520f1e073e97316e24ea7d0a7c907e8659af8285d31e88192a83bc9a0"
},
"downloads": -1,
"filename": "embedme-0.1.1.tar.gz",
"has_sig": false,
"md5_digest": "9ca5a6c62add31dd0bb0710b727dddce",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.10,<4.0",
"size": 4623,
"upload_time": "2023-01-22T19:57:12",
"upload_time_iso_8601": "2023-01-22T19:57:12.857380Z",
"url": "https://files.pythonhosted.org/packages/76/06/fa9ed815bd7325b42d038f2e1629643e007eabde9a0289129f29823a50bb/embedme-0.1.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-01-22 19:57:12",
"github": true,
"gitlab": false,
"bitbucket": false,
"github_user": "morganpartee",
"github_project": "embedme",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "embedme"
}
JSON
