gpt-index


Namegpt-index JSON
Version 0.8.41 PyPI version JSON
download
home_pagehttps://github.com/jerryjliu/llama_index
SummaryInterface between LLMs and your data
upload_time2023-10-07 22:10:59
maintainer
docs_urlNone
authorJerry Liu
requires_python
licenseMIT
keywords
VCS
bugtrack_url
requirements No requirements were recorded.
Travis-CI No Travis.
coveralls test coverage No coveralls.
            # 🗂️ LlamaIndex 🦙
[![PyPI - Downloads](https://img.shields.io/pypi/dm/llama-index)](https://pypi.org/project/llama-index/)
[![GitHub contributors](https://img.shields.io/github/contributors/jerryjliu/llama_index)](https://github.com/jerryjliu/llama_index/graphs/contributors)
[![Discord](https://img.shields.io/discord/1059199217496772688)](https://discord.gg/dGcwcsnxhU)


LlamaIndex (GPT Index) is a data framework for your LLM application.

PyPI:
- LlamaIndex: https://pypi.org/project/llama-index/.
- GPT Index (duplicate): https://pypi.org/project/gpt-index/.

LlamaIndex.TS (Typescript/Javascript): https://github.com/run-llama/LlamaIndexTS.

Documentation: https://gpt-index.readthedocs.io/.

Twitter: https://twitter.com/llama_index.

Discord: https://discord.gg/dGcwcsnxhU.

### Ecosystem

- LlamaHub (community library of data loaders): https://llamahub.ai
- LlamaLab (cutting-edge AGI projects using LlamaIndex): https://github.com/run-llama/llama-lab


## 🚀 Overview

**NOTE**: This README is not updated as frequently as the documentation. Please check out the documentation above for the latest updates!

### Context
- LLMs are a phenomenal piece of technology for knowledge generation and reasoning. They are pre-trained on large amounts of publicly available data.
- How do we best augment LLMs with our own private data?

We need a comprehensive toolkit to help perform this data augmentation for LLMs.

### Proposed Solution

That's where **LlamaIndex** comes in. LlamaIndex is a "data framework" to help you build LLM apps. It provides the following tools:

- Offers **data connectors** to ingest your existing data sources and data formats (APIs, PDFs, docs, SQL, etc.)
- Provides ways to **structure your data** (indices, graphs) so that this data can be easily used with LLMs.
- Provides an **advanced retrieval/query interface over your data**: Feed in any LLM input prompt, get back retrieved context and knowledge-augmented output.
- Allows easy integrations with your outer application framework (e.g. with LangChain, Flask, Docker, ChatGPT, anything else).

LlamaIndex provides tools for both beginner users and advanced users. Our high-level API allows beginner users to use LlamaIndex to ingest and query their data in
5 lines of code. Our lower-level APIs allow advanced users to customize and extend any module (data connectors, indices, retrievers, query engines, reranking modules),
to fit their needs.


## 💡 Contributing

Interested in contributing? See our [Contribution Guide](CONTRIBUTING.md) for more details.

## 📄 Documentation

Full documentation can be found here: https://gpt-index.readthedocs.io/en/latest/.

Please check it out for the most up-to-date tutorials, how-to guides, references, and other resources!


## 💻 Example Usage

```
pip install llama-index
```

Examples are in the `examples` folder. Indices are in the `indices` folder (see list of indices below).

To build a simple vector store index:
```python
import os
os.environ["OPENAI_API_KEY"] = 'YOUR_OPENAI_API_KEY'

from llama_index import VectorStoreIndex, SimpleDirectoryReader
documents = SimpleDirectoryReader('data').load_data()
index = VectorStoreIndex.from_documents(documents)
```


To query:
```python
query_engine = index.as_query_engine()
query_engine.query("<question_text>?")
```


By default, data is stored in-memory.
To persist to disk (under `./storage`):

```python
index.storage_context.persist()
```

To reload from disk:
```python
from llama_index import StorageContext, load_index_from_storage

# rebuild storage context
storage_context = StorageContext.from_defaults(persist_dir='./storage')
# load index
index = load_index_from_storage(storage_context)
```


## 🔧 Dependencies

The main third-party package requirements are `tiktoken`, `openai`, and `langchain`.

All requirements should be contained within the `setup.py` file. To run the package locally without building the wheel, simply run `pip install -r requirements.txt`.


## 📖 Citation

Reference to cite if you use LlamaIndex in a paper:

```
@software{Liu_LlamaIndex_2022,
author = {Liu, Jerry},
doi = {10.5281/zenodo.1234},
month = {11},
title = {{LlamaIndex}},
url = {https://github.com/jerryjliu/llama_index},
year = {2022}
}
```

            

Raw data

            {
    "_id": null,
    "home_page": "https://github.com/jerryjliu/llama_index",
    "name": "gpt-index",
    "maintainer": "",
    "docs_url": null,
    "requires_python": "",
    "maintainer_email": "",
    "keywords": "",
    "author": "Jerry Liu",
    "author_email": "",
    "download_url": "https://files.pythonhosted.org/packages/35/fb/83176f8ba0860d731af84be8ea4613e0e763fb0c386902237ef81c8c650f/gpt_index-0.8.41.tar.gz",
    "platform": null,
    "description": "# \ud83d\uddc2\ufe0f LlamaIndex \ud83e\udd99\n[![PyPI - Downloads](https://img.shields.io/pypi/dm/llama-index)](https://pypi.org/project/llama-index/)\n[![GitHub contributors](https://img.shields.io/github/contributors/jerryjliu/llama_index)](https://github.com/jerryjliu/llama_index/graphs/contributors)\n[![Discord](https://img.shields.io/discord/1059199217496772688)](https://discord.gg/dGcwcsnxhU)\n\n\nLlamaIndex (GPT Index) is a data framework for your LLM application.\n\nPyPI:\n- LlamaIndex: https://pypi.org/project/llama-index/.\n- GPT Index (duplicate): https://pypi.org/project/gpt-index/.\n\nLlamaIndex.TS (Typescript/Javascript): https://github.com/run-llama/LlamaIndexTS.\n\nDocumentation: https://gpt-index.readthedocs.io/.\n\nTwitter: https://twitter.com/llama_index.\n\nDiscord: https://discord.gg/dGcwcsnxhU.\n\n### Ecosystem\n\n- LlamaHub (community library of data loaders): https://llamahub.ai\n- LlamaLab (cutting-edge AGI projects using LlamaIndex): https://github.com/run-llama/llama-lab\n\n\n## \ud83d\ude80 Overview\n\n**NOTE**: This README is not updated as frequently as the documentation. Please check out the documentation above for the latest updates!\n\n### Context\n- LLMs are a phenomenal piece of technology for knowledge generation and reasoning. They are pre-trained on large amounts of publicly available data.\n- How do we best augment LLMs with our own private data?\n\nWe need a comprehensive toolkit to help perform this data augmentation for LLMs.\n\n### Proposed Solution\n\nThat's where **LlamaIndex** comes in. LlamaIndex is a \"data framework\" to help you build LLM apps. It provides the following tools:\n\n- Offers **data connectors** to ingest your existing data sources and data formats (APIs, PDFs, docs, SQL, etc.)\n- Provides ways to **structure your data** (indices, graphs) so that this data can be easily used with LLMs.\n- Provides an **advanced retrieval/query interface over your data**: Feed in any LLM input prompt, get back retrieved context and knowledge-augmented output.\n- Allows easy integrations with your outer application framework (e.g. with LangChain, Flask, Docker, ChatGPT, anything else).\n\nLlamaIndex provides tools for both beginner users and advanced users. Our high-level API allows beginner users to use LlamaIndex to ingest and query their data in\n5 lines of code. Our lower-level APIs allow advanced users to customize and extend any module (data connectors, indices, retrievers, query engines, reranking modules),\nto fit their needs.\n\n\n## \ud83d\udca1 Contributing\n\nInterested in contributing? See our [Contribution Guide](CONTRIBUTING.md) for more details.\n\n## \ud83d\udcc4 Documentation\n\nFull documentation can be found here: https://gpt-index.readthedocs.io/en/latest/.\n\nPlease check it out for the most up-to-date tutorials, how-to guides, references, and other resources!\n\n\n## \ud83d\udcbb Example Usage\n\n```\npip install llama-index\n```\n\nExamples are in the `examples` folder. Indices are in the `indices` folder (see list of indices below).\n\nTo build a simple vector store index:\n```python\nimport os\nos.environ[\"OPENAI_API_KEY\"] = 'YOUR_OPENAI_API_KEY'\n\nfrom llama_index import VectorStoreIndex, SimpleDirectoryReader\ndocuments = SimpleDirectoryReader('data').load_data()\nindex = VectorStoreIndex.from_documents(documents)\n```\n\n\nTo query:\n```python\nquery_engine = index.as_query_engine()\nquery_engine.query(\"<question_text>?\")\n```\n\n\nBy default, data is stored in-memory.\nTo persist to disk (under `./storage`):\n\n```python\nindex.storage_context.persist()\n```\n\nTo reload from disk:\n```python\nfrom llama_index import StorageContext, load_index_from_storage\n\n# rebuild storage context\nstorage_context = StorageContext.from_defaults(persist_dir='./storage')\n# load index\nindex = load_index_from_storage(storage_context)\n```\n\n\n## \ud83d\udd27 Dependencies\n\nThe main third-party package requirements are `tiktoken`, `openai`, and `langchain`.\n\nAll requirements should be contained within the `setup.py` file. To run the package locally without building the wheel, simply run `pip install -r requirements.txt`.\n\n\n## \ud83d\udcd6 Citation\n\nReference to cite if you use LlamaIndex in a paper:\n\n```\n@software{Liu_LlamaIndex_2022,\nauthor = {Liu, Jerry},\ndoi = {10.5281/zenodo.1234},\nmonth = {11},\ntitle = {{LlamaIndex}},\nurl = {https://github.com/jerryjliu/llama_index},\nyear = {2022}\n}\n```\n",
    "bugtrack_url": null,
    "license": "MIT",
    "summary": "Interface between LLMs and your data",
    "version": "0.8.41",
    "project_urls": {
        "Homepage": "https://github.com/jerryjliu/llama_index"
    },
    "split_keywords": [],
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "c2cc6fe07b3327d1532f6529ec85bcf4e78f559e19b305c1eddde39154178f15",
                "md5": "63f99f1a05b673298aae961bbe513300",
                "sha256": "7ce0af3d9a12b906113da90902ce4227b0e99a5ce3c546043776542eb7e24b37"
            },
            "downloads": -1,
            "filename": "gpt_index-0.8.41-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "63f99f1a05b673298aae961bbe513300",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": null,
            "size": 868065,
            "upload_time": "2023-10-07T22:10:57",
            "upload_time_iso_8601": "2023-10-07T22:10:57.568520Z",
            "url": "https://files.pythonhosted.org/packages/c2/cc/6fe07b3327d1532f6529ec85bcf4e78f559e19b305c1eddde39154178f15/gpt_index-0.8.41-py3-none-any.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "35fb83176f8ba0860d731af84be8ea4613e0e763fb0c386902237ef81c8c650f",
                "md5": "02e8c7421e8c18a3496d467ca04b3a39",
                "sha256": "ee371a01ab38368f7c66389fe154f25ea19131b56b973f255dadd3112b1637bf"
            },
            "downloads": -1,
            "filename": "gpt_index-0.8.41.tar.gz",
            "has_sig": false,
            "md5_digest": "02e8c7421e8c18a3496d467ca04b3a39",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 559554,
            "upload_time": "2023-10-07T22:10:59",
            "upload_time_iso_8601": "2023-10-07T22:10:59.573707Z",
            "url": "https://files.pythonhosted.org/packages/35/fb/83176f8ba0860d731af84be8ea4613e0e763fb0c386902237ef81c8c650f/gpt_index-0.8.41.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2023-10-07 22:10:59",
    "github": true,
    "gitlab": false,
    "bitbucket": false,
    "codeberg": false,
    "github_user": "jerryjliu",
    "github_project": "llama_index",
    "travis_ci": false,
    "coveralls": false,
    "github_actions": true,
    "requirements": [],
    "lcname": "gpt-index"
}
        
Elapsed time: 0.12591s