<!---
Copyright (c) 2022-present, FriendliAI Inc. All rights reserved.
-->
<p align="center">
<img src="https://docs.friendli.ai/img/logo.svg" width="30%" alt="Friendli Logo">
</p>
<h2><p align="center">Supercharge Generative AI Serving with Friendli đ</p></h2>
<p align="center">
<a href="https://github.com/friendliai/friendli-client/actions/workflows/ci.yaml">
<img alt="CI Status" src="https://github.com/friendliai/friendli-client/actions/workflows/ci.yaml/badge.svg">
</a>
<a href="https://pypi.org/project/friendli-client/">
<img alt="Python Version" src="https://img.shields.io/pypi/pyversions/friendli-client?logo=Python&logoColor=white">
</a>
<a href="https://pypi.org/project/friendli-client/">
<img alt="PyPi Package Version" src="https://img.shields.io/pypi/v/friendli-client?logo=PyPI&logoColor=white">
</a>
<a href="https://docs.friendli.ai/">
<img alt="Documentation" src="https://img.shields.io/badge/read-doc-blue?logo=ReadMe&logoColor=white">
</a>
<a href="https://github.com/friendliai/friendli-client/blob/main/LICENSE">
<img alt="License" src="https://img.shields.io/badge/License-Apache%202.0-green.svg?logo=Apache">
</a>
</p>
Welcome to Friendli Suite, the ultimate solution for serving generative AI models. We offer three distinct options to cater to your specific needs, each designed to provide superior performance, cost-effectiveness, and ease of use.
# Friendli Suite
## 1. Friendli Serverless Endpoints
Imagine a playground for your AI dreams.
Friendli Serverless Endpoint is just that - a simple, click-and-play interface that lets you access popular open-source models like Llama-2 and Stable Diffusion without any heavy lifting.
Choose your model, enter your prompt or upload an image, and marvel at the generated text, code, or image outputs.
With pay-per-token billing, this is ideal for exploration and experimentation.
You can think of it as an AI sampler.
## 2. Friendli Dedicated Endpoints
Ready to take the reins and unleash the full potential of your own models?
Friendli Dedicated Endpoint is for you.
This service provides dedicated GPU resources in the cloud platform of your choice (AWS, GCP, Azure), letting you upload and run your custom generative AI models.
Reserve the exact GPU you need (A10, A100 40G, A100 80G, etc.) and enjoy fine-grained control over your model settings.
Pay-per-second billing makes it perfect for regular or resource-intensive workloads.
## 3. Friendli Container
Do you prefer the comfort and security of your own data center?
Friendli Container is the solution.
We provide the Friendli Engine within Docker containers that can be installed on your on-premise GPUs so your data stays within your own secure cluster.
This option offers maximum control and security, ideal for advanced users or those with specific data privacy requirements.
> [!NOTE]
>
> ## The Friendli Engine: The Powerhouse Behind the Suite
>
> At the heart of each Friendli Suite offering lies the Friendli Engine, a patented, GPU-optimized serving engine.
> This technological marvel is what enables Friendli Suite's superior performance and cost-effectiveness, featuring innovations like continuous batching (iteration batching) that significantly improve resource utilization compared to traditional LLM serving solutions.
# đšī¸ Friendli Client
## Installation
```sh
pip install friendli-client
```
> [!NOTE]
> If you have a Hugging Face checkpoint and want to convert it to a Friendli-compatible format with applying quantization, you need to install the package with the necessary machine learing library (`mllib`) dependencies.
> In this case, install the package with the following command:
>
> ```sh
> pip install "friendli-client[mllib]"
> ```
## Python SDK Examples
> [!IMPORTANT]
> You must set `FRIENDLI_TOKEN` environment variable before initializing the client instance with `client = Friendli()`.
> Alternatively, you can provide the value of your personal access token as the `token` argument when creating the client, like so:
>
> ```python
> from friendli import Friendli
>
> client = Friendli(token="YOUR PERSONAL ACCESS TOKEN")
> ```
### Default
```python
from friendli import Friendli
client = Friendli()
chat_completion = client.chat.completions.create(
model="llama-2-13b-chat",
messages=[
{
"role": "user",
"content": "Tell me how to make a delicious pancake"
}
],
stream=False,
)
print(chat_completion.choices[0].message.content)
```
### Streaming
```python
from friendli import Friendli
client = Friendli()
stream = client.chat.completions.create(
model="llama-2-13b-chat",
messages=[
{
"role": "user",
"content": "Tell me how to make a delicious pancake"
}
]
stream=True,
)
for chunk in stream:
print(chunk.choices[0].delta.content or "", end="")
```
### Async
```python
import asyncio
from friendi import AsyncFriendli
client = AsyncFriendli()
async def main() -> None:
chat_completion = await client.chat.completions.create(
model="llama-2-13b-chat",
messages=[
{
"role": "user",
"content": "Tell me how to make a delicious pancake"
}
]
stream=False,
)
print(chat_completion.choices[0].message.content)
asyncio.run(main())
```
### Streaming (Async)
```python
import asyncio
from friendi import AsyncFriendli
client = AsyncFriendli()
async def main() -> None:
stream = await client.chat.completions.create(
model="llama-2-13b-chat",
messages=[
{
"role": "user",
"content": "Tell me how to make a delicious pancake"
}
]
stream=True,
)
async for chunk in stream:
print(chunk.choices[0].delta.content or "", end="")
asyncio.run(main())
```
## CLI Examples
You can also call the generation APIs directly with CLI.
```sh
friendli api chat-completions create \
-g "user Tell me how to make a delicious pancake" \
-m llama-2-13b-chat
```
For further information about the `friendli` command, run `friendli --help` in your terminal shell.
This will provide you with a detailed list of available options and usage instructions.
> [!TIP]
> **Check out our [official documentations](https://docs.friendli.ai/) to learn more!**
Raw data
{
"_id": null,
"home_page": "https://friendli.ai/",
"name": "friendli-client",
"maintainer": null,
"docs_url": null,
"requires_python": "<4.0.0,>=3.8.1",
"maintainer_email": null,
"keywords": "generative-ai, serving, llm, inference, finetuning",
"author": "FriendliAI teams",
"author_email": "eng@friendli.ai",
"download_url": "https://files.pythonhosted.org/packages/9d/23/d7a9c3e595054e11b4ad8791e600f558fe6abd08d3f6ae139f72186dbfe6/friendli_client-1.3.4.tar.gz",
"platform": null,
"description": "<!---\nCopyright (c) 2022-present, FriendliAI Inc. All rights reserved.\n-->\n\n<p align=\"center\">\n <img src=\"https://docs.friendli.ai/img/logo.svg\" width=\"30%\" alt=\"Friendli Logo\">\n</p>\n\n<h2><p align=\"center\">Supercharge Generative AI Serving with Friendli \ud83d\ude80</p></h2>\n\n<p align=\"center\">\n <a href=\"https://github.com/friendliai/friendli-client/actions/workflows/ci.yaml\">\n <img alt=\"CI Status\" src=\"https://github.com/friendliai/friendli-client/actions/workflows/ci.yaml/badge.svg\">\n </a>\n <a href=\"https://pypi.org/project/friendli-client/\">\n <img alt=\"Python Version\" src=\"https://img.shields.io/pypi/pyversions/friendli-client?logo=Python&logoColor=white\">\n </a>\n <a href=\"https://pypi.org/project/friendli-client/\">\n <img alt=\"PyPi Package Version\" src=\"https://img.shields.io/pypi/v/friendli-client?logo=PyPI&logoColor=white\">\n </a>\n <a href=\"https://docs.friendli.ai/\">\n <img alt=\"Documentation\" src=\"https://img.shields.io/badge/read-doc-blue?logo=ReadMe&logoColor=white\">\n </a>\n <a href=\"https://github.com/friendliai/friendli-client/blob/main/LICENSE\">\n <img alt=\"License\" src=\"https://img.shields.io/badge/License-Apache%202.0-green.svg?logo=Apache\">\n </a>\n</p>\n\nWelcome to Friendli Suite, the ultimate solution for serving generative AI models. We offer three distinct options to cater to your specific needs, each designed to provide superior performance, cost-effectiveness, and ease of use.\n\n# Friendli Suite\n\n## 1. Friendli Serverless Endpoints\n\nImagine a playground for your AI dreams.\nFriendli Serverless Endpoint is just that - a simple, click-and-play interface that lets you access popular open-source models like Llama-2 and Stable Diffusion without any heavy lifting.\nChoose your model, enter your prompt or upload an image, and marvel at the generated text, code, or image outputs.\nWith pay-per-token billing, this is ideal for exploration and experimentation.\nYou can think of it as an AI sampler.\n\n## 2. Friendli Dedicated Endpoints\n\nReady to take the reins and unleash the full potential of your own models?\nFriendli Dedicated Endpoint is for you.\nThis service provides dedicated GPU resources in the cloud platform of your choice (AWS, GCP, Azure), letting you upload and run your custom generative AI models.\nReserve the exact GPU you need (A10, A100 40G, A100 80G, etc.) and enjoy fine-grained control over your model settings.\nPay-per-second billing makes it perfect for regular or resource-intensive workloads.\n\n## 3. Friendli Container\n\nDo you prefer the comfort and security of your own data center?\nFriendli Container is the solution.\nWe provide the Friendli Engine within Docker containers that can be installed on your on-premise GPUs so your data stays within your own secure cluster.\nThis option offers maximum control and security, ideal for advanced users or those with specific data privacy requirements.\n\n> [!NOTE]\n>\n> ## The Friendli Engine: The Powerhouse Behind the Suite\n>\n> At the heart of each Friendli Suite offering lies the Friendli Engine, a patented, GPU-optimized serving engine.\n> This technological marvel is what enables Friendli Suite's superior performance and cost-effectiveness, featuring innovations like continuous batching (iteration batching) that significantly improve resource utilization compared to traditional LLM serving solutions.\n\n# \ud83d\udd79\ufe0f Friendli Client\n\n## Installation\n\n```sh\npip install friendli-client\n```\n\n> [!NOTE]\n> If you have a Hugging Face checkpoint and want to convert it to a Friendli-compatible format with applying quantization, you need to install the package with the necessary machine learing library (`mllib`) dependencies.\n> In this case, install the package with the following command:\n>\n> ```sh\n> pip install \"friendli-client[mllib]\"\n> ```\n\n## Python SDK Examples\n\n> [!IMPORTANT]\n> You must set `FRIENDLI_TOKEN` environment variable before initializing the client instance with `client = Friendli()`.\n> Alternatively, you can provide the value of your personal access token as the `token` argument when creating the client, like so:\n>\n> ```python\n> from friendli import Friendli\n> \n> client = Friendli(token=\"YOUR PERSONAL ACCESS TOKEN\")\n> ```\n\n### Default\n\n```python\nfrom friendli import Friendli\n\nclient = Friendli()\n\nchat_completion = client.chat.completions.create(\n model=\"llama-2-13b-chat\",\n messages=[\n {\n \"role\": \"user\",\n \"content\": \"Tell me how to make a delicious pancake\"\n }\n ],\n stream=False,\n)\nprint(chat_completion.choices[0].message.content)\n```\n\n### Streaming\n\n```python\nfrom friendli import Friendli\n\nclient = Friendli()\n\nstream = client.chat.completions.create(\n model=\"llama-2-13b-chat\",\n messages=[\n {\n \"role\": \"user\",\n \"content\": \"Tell me how to make a delicious pancake\"\n }\n ]\n stream=True,\n)\nfor chunk in stream:\n print(chunk.choices[0].delta.content or \"\", end=\"\")\n```\n\n### Async\n\n```python\nimport asyncio\nfrom friendi import AsyncFriendli\n\nclient = AsyncFriendli()\n\n\nasync def main() -> None:\n chat_completion = await client.chat.completions.create(\n model=\"llama-2-13b-chat\",\n messages=[\n {\n \"role\": \"user\",\n \"content\": \"Tell me how to make a delicious pancake\"\n }\n ]\n stream=False,\n )\n print(chat_completion.choices[0].message.content)\n\n\nasyncio.run(main())\n```\n\n### Streaming (Async)\n\n```python\nimport asyncio\nfrom friendi import AsyncFriendli\n\nclient = AsyncFriendli()\n\n\nasync def main() -> None:\n stream = await client.chat.completions.create(\n model=\"llama-2-13b-chat\",\n messages=[\n {\n \"role\": \"user\",\n \"content\": \"Tell me how to make a delicious pancake\"\n }\n ]\n stream=True,\n )\n async for chunk in stream:\n print(chunk.choices[0].delta.content or \"\", end=\"\")\n\n\nasyncio.run(main())\n```\n\n## CLI Examples\n\nYou can also call the generation APIs directly with CLI.\n\n```sh\nfriendli api chat-completions create \\\n -g \"user Tell me how to make a delicious pancake\" \\\n -m llama-2-13b-chat\n```\n\nFor further information about the `friendli` command, run `friendli --help` in your terminal shell.\nThis will provide you with a detailed list of available options and usage instructions.\n\n> [!TIP]\n> **Check out our [official documentations](https://docs.friendli.ai/) to learn more!**\n",
"bugtrack_url": null,
"license": "Apache-2.0",
"summary": "Client of Friendli Suite.",
"version": "1.3.4",
"project_urls": {
"Documentation": "https://docs.friendli.ai/",
"Homepage": "https://friendli.ai/",
"Repository": "https://github.com/friendliai/friendli-client"
},
"split_keywords": [
"generative-ai",
" serving",
" llm",
" inference",
" finetuning"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "d9173f7948ad972d9fdab1fee80ec53e4d3686881edba14a925fa39ba3927c12",
"md5": "8c2f35e05f3517ddb84d5f27443a9247",
"sha256": "6bb4d8e5bd67605f935d13cb103ec671f95c2aa03ee6d906b397900c40163455"
},
"downloads": -1,
"filename": "friendli_client-1.3.4-py3-none-any.whl",
"has_sig": false,
"md5_digest": "8c2f35e05f3517ddb84d5f27443a9247",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4.0.0,>=3.8.1",
"size": 197697,
"upload_time": "2024-04-02T04:03:35",
"upload_time_iso_8601": "2024-04-02T04:03:35.193760Z",
"url": "https://files.pythonhosted.org/packages/d9/17/3f7948ad972d9fdab1fee80ec53e4d3686881edba14a925fa39ba3927c12/friendli_client-1.3.4-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "9d23d7a9c3e595054e11b4ad8791e600f558fe6abd08d3f6ae139f72186dbfe6",
"md5": "529a90389260886a74992a836f121cba",
"sha256": "44b96366b7d9426ad4adbc4bac00bae0c76f3a1b9150508309bf56d22cdc2a6f"
},
"downloads": -1,
"filename": "friendli_client-1.3.4.tar.gz",
"has_sig": false,
"md5_digest": "529a90389260886a74992a836f121cba",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4.0.0,>=3.8.1",
"size": 119504,
"upload_time": "2024-04-02T04:03:37",
"upload_time_iso_8601": "2024-04-02T04:03:37.643475Z",
"url": "https://files.pythonhosted.org/packages/9d/23/d7a9c3e595054e11b4ad8791e600f558fe6abd08d3f6ae139f72186dbfe6/friendli_client-1.3.4.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-04-02 04:03:37",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "friendliai",
"github_project": "friendli-client",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"tox": true,
"lcname": "friendli-client"
}
JSON
