<!-- PROJECT LOGO -->
<div align="center">
<img src="images/logo.png" alt="Scrappy, the SCRAPI mascot!" width="200">
<h3 align="center">Python Dialogflow CX Scripting API (SCRAPI)</h3>
<p align="center">
A high level scripting API for bot builders, developers, and maintainers.<br>
</p>
</div>
<!-- TABLE OF CONTENTS -->
<details open="open">
<summary>Table of Contents</summary>
<ol>
<li>
<a href="#introduction">Introduction</a>
<ul>
<li><a href="#what-can-i-do-with-dfcx-scrapi">What Can I Do with SCRAPI?</a></li>
<li><a href="#built-with">Built With</a></li>
</ul>
</li>
<li>
<a href="#getting-started">Getting Started</a>
<ul>
<li><a href="#environment-setup">Environment Setup</a></li>
<li><a href="#authentication">Authentication</a></li>
</ul>
</li>
<li><a href="#usage">Usage</a>
<li>
<a href="#library-composition">Library Composition</a>
<ul>
<li><a href="#core">Core</a></li>
<li><a href="#tools">Tools</a></li>
</ul>
</li>
<li><a href="#contributing">Contributing</a></li>
<li><a href="#license">License</a></li>
<li><a href="#contact">Contact</a></li>
<li><a href="#acknowledgements">Acknowledgements</a></li>
</ol>
</details>
<!-- INTRODUCTION -->
# Introduction
The Python Dialogflow CX Scripting API (DFCX SCRAPI) is a high level API that extends the official Google [Python Client for Dialogflow CX](https://github.com/googleapis/python-dialogflow-cx). SCRAPI makes using DFCX easier, more friendly, and more pythonic for bot builders, developers, and maintainers.
```
SCRAPI --> Python Dialogflow CX
as
Keras --> Tensorflow
```
## What Can I Do With DFCX SCRAPI?
With DFCX SCRAPI you can perform many bot building and maintenance actions at scale including, but not limited to:
- Create, Update, Delete, Get, and List for all CX resources types (i.e. Intents, Entity Types, Pages, Flows, etc.)
- Convert commonly accessed CX Resources to Pandas Dataframes
- Have fully automated conversations with a CX agent (powerful for regression testing!)
- Extract Validation information
- Extract Change History information
- Search across all Flows/Pages/Routes to find a specific parameter or utterance using Search Util functions
- Quickly move CX resources between agents using Copy Util functions!
- Build the fundamental protobuf objects that CX uses for each resource type using Builder methods
- ...and much, much more!
## Built With
* Python 3.8+
<!-- AUTHENTICATION -->
# Authentication
Authentication can vary depending on how and where you are interacting with SCRAPI.
## Google Colab
If you're using SCRAPI with a [Google Colab](https://colab.research.google.com/) notebook, you can add the following to the top of your notebook for easy authentication:
```py
project_id = '<YOUR_GCP_PROJECT_ID>'
# this will launch an interactive prompt that allows you to auth with GCP in a browser
!gcloud auth application-default login --no-launch-browser
# this will set your active project to the `project_id` above
!gcloud auth application-default set-quota-project $project_id
```
After running the above, Colab will pick up your credentials from the environment and pass them to SCRAPI directly. No need to use Service Account keys!
You can then use SCRAPI simply like this:
```py
from dfcx_scrapi.core.intents import Intents
agent_id = '<YOUR_AGENT_ID>'
i = Intents() # <-- Creds will be automatically picked up from the environment
intents_map = i.get_intents_map(agent_id)
```
---
## Cloud Functions / Cloud Run
If you're using SCRAPI with [Cloud Functions](https://cloud.google.com/functions) or [Cloud Run](https://cloud.google.com/run), SCRAPI can pick up on the default environment creds used by these services without any additional configuration!
1. Add `dfcx-scrapi` to your `requirements.txt` file
2. Ensure the Cloud Function / Cloud Run service account has the appropriate Dialogflow IAM Role
Once you are setup with the above, your function code can be used easily like this:
```py
from dfcx_scrapi.core.intents import Intents
agent_id = '<YOUR_AGENT_ID>'
i = Intents() # <-- Creds will be automatically picked up from the environment
intents_map = i.get_intents_map(agent_id)
```
---
## Local Python Environment
Similar to Cloud Functions / Cloud Run, SCRAPI can pick up on your local authentication creds _if you are using the gcloud CLI._
1. Install [gcloud CLI](https://cloud.google.com/sdk/docs/install).
2. Run `gcloud init`.
3. Run `gcloud auth login`
4. Run `gcloud auth list` to ensure your principal account is active.
This will authenticate your principal GCP account with the gcloud CLI, and SCRAPI can pick up the creds from here.
---
## Exceptions and Misc.
There are some classes in SCRAPI which still rely on Service Account Keys, notably the [DataframeFunctions](https://github.com/GoogleCloudPlatform/dfcx-scrapi/blob/main/src/dfcx_scrapi/tools/dataframe_functions.py) class due to how it authenticates with Google Sheets.
In order to use these functions, you will need a Service Account that has appropriate access to your GCP project.
For more information and to view the official documentation for service accounts go to [Creating and Managing GCP Service Accounts](https://cloud.google.com/iam/docs/creating-managing-service-accounts).
Once you've obtained a Service Account Key with appropriate permissions, you can use it as follows:
```py
from dfcx_scrapi.core.intents import Intents
from dfcx_scrapi.tools.dataframe_functions import DataframeFunctions
agent_id = '<YOUR_AGENT_ID>'
creds_path = '<PATH_TO_YOUR_SERVICE_ACCOUNT_JSON_FILE>'
i = Intents(creds_path=creds_path)
dffx = DataframeFunctions(creds_path=creds_path)
df = i.bulk_intent_to_df(agent_id)
dffx.dataframe_to_sheets('GOOGLE_SHEET_NAME', 'TAB_NAME', df)
```
<!-- GETTING STARTED -->
# Getting Started
## Environment Setup
Set up Google Cloud Platform credentials and install dependencies.
```sh
gcloud auth login
gcloud auth application-default login
gcloud config set project <project name>
```
```sh
python3 -m venv venv
source ./venv/bin/activate
pip install -r requirements.txt
```
## Usage
To run a simple bit of code you can do the following:
- Import a Class from `dfcx_scrapi.core`
- Assign your Service Account to a local variable
```python
from dfcx_scrapi.core.intents import Intents
creds_path = '<PATH_TO_YOUR_SERVICE_ACCOUNT_JSON_FILE>'
agent_path = '<FULL_DFCX_AGENT_ID_PATH>'
# DFCX Agent ID paths are in this format:
# 'projects/<project_id>/locations/<location_id>/agents/<agent_id>'
# Instantiate your class object and pass in your credentials
i = Intents(creds_path, agent_id=agent_path)
# Retrieve all Intents and Training Phrases from an Agent and push to a Pandas DataFrame
df = i.bulk_intent_to_df()
```
# Library Composition
Here is a brief overview of the SCRAPI library's structure and the motivation behind that structure.
## Core
The [Core](/src/dfcx_scrapi/core) folder is synonymous with the core Resource types in the DFCX Agents (agents, intents, flows, etc.)
* This folder contains the high level building blocks of SCRAPI
* These classes and methods can be used to build higher level methods or custom tools and applications
## Tools
The [Tools](/src/dfcx_scrapi/tools) folder contains various customized toolkits that allow you to do more complex bot management tasks, such as
- Manipulate Agent Resource types into various DataFrame structures
- Copy Agent Resources between Agents and GCP Projects on a resource by resource level
- Move data to and from DFCX and other GCP Services like BigQuery, Sheets, etc.
- Create customized search queries inside of your agent resources
## Builders
The [Builders](/src/dfcx_scrapi/builders) folder contains simple methods for constructing the underlying protos in Dialogflow CX
- Proto objects are the fundamental building blocks of Dialogflow CX
- Builder classes allow the user to construct Dialogflow CX resource _offline_ without any API calls
- Once the resource components are constructed, they can then be pushed to a live Dialogflow CX agent via API
<!-- CONTRIBUTING -->
# Contributing
We welcome any contributions or feature requests you would like to submit!
1. Fork the Project
2. Create your Feature Branch (git checkout -b feature/AmazingFeature)
3. Commit your Changes (git commit -m 'Add some AmazingFeature')
4. Push to the Branch (git push origin feature/AmazingFeature)
5. Open a Pull Request
<!-- LICENSE -->
# License
Distributed under the Apache 2.0 License. See [LICENSE](LICENSE.txt) for more information.
<!-- CONTACT -->
# Contact
Patrick Marlow - pmarlow@google.com - [@kmaphoenix](https://github.com/kmaphoenix)
Milad Tabrizi - miladt@google.com - [@MRyderOC](https://github.com/MRyderOC)
Project Link: [https://github.com/GoogleCloudPlatform/dfcx-scrapi](https://github.com/GoogleCloudPlatform/dfcx-scrapi)
<!-- ACKNOWLEDGEMENTS -->
# Acknowledgements
[Dialogflow CX Python Client Library](https://github.com/googleapis/python-dialogflow-cx)
[Hugging Face - Pegasus Paraphrase](https://huggingface.co/tuner007/pegasus_paraphrase)
<!-- MARKDOWN LINKS & IMAGES -->
<!-- https://www.markdownguide.org/basic-syntax/#reference-style-links -->
Raw data
{
"_id": null,
"home_page": "https://github.com/GoogleCloudPlatform/dfcx-scrapi",
"name": "dfcx-scrapi",
"maintainer": null,
"docs_url": null,
"requires_python": "<4,>=3.6",
"maintainer_email": null,
"keywords": "dialogflow, cx, google, bot, chatbot, intent, dfcx, entity",
"author": "Patrick Marlow",
"author_email": "pmarlow@google.com",
"download_url": "https://files.pythonhosted.org/packages/45/3b/af391279698a8ed3fff0ef478499c7c7012a2d7b59dad9238c402749b1df/dfcx_scrapi-1.12.5.tar.gz",
"platform": null,
"description": "<!-- PROJECT LOGO -->\n<div align=\"center\">\n <img src=\"images/logo.png\" alt=\"Scrappy, the SCRAPI mascot!\" width=\"200\">\n\n <h3 align=\"center\">Python Dialogflow CX Scripting API (SCRAPI)</h3>\n <p align=\"center\">\n A high level scripting API for bot builders, developers, and maintainers.<br>\n </p>\n</div>\n\n<!-- TABLE OF CONTENTS -->\n<details open=\"open\">\n <summary>Table of Contents</summary>\n <ol>\n <li>\n <a href=\"#introduction\">Introduction</a>\n <ul>\n <li><a href=\"#what-can-i-do-with-dfcx-scrapi\">What Can I Do with SCRAPI?</a></li>\n <li><a href=\"#built-with\">Built With</a></li>\n </ul>\n </li>\n <li>\n <a href=\"#getting-started\">Getting Started</a>\n <ul>\n <li><a href=\"#environment-setup\">Environment Setup</a></li>\n <li><a href=\"#authentication\">Authentication</a></li>\n </ul>\n </li>\n <li><a href=\"#usage\">Usage</a>\n <li>\n <a href=\"#library-composition\">Library Composition</a>\n <ul>\n <li><a href=\"#core\">Core</a></li>\n <li><a href=\"#tools\">Tools</a></li>\n </ul>\n </li>\n <li><a href=\"#contributing\">Contributing</a></li>\n <li><a href=\"#license\">License</a></li>\n <li><a href=\"#contact\">Contact</a></li>\n <li><a href=\"#acknowledgements\">Acknowledgements</a></li>\n </ol>\n</details>\n\n<!-- INTRODUCTION -->\n# Introduction\n\nThe Python Dialogflow CX Scripting API (DFCX SCRAPI) is a high level API that extends the official Google [Python Client for Dialogflow CX](https://github.com/googleapis/python-dialogflow-cx). SCRAPI makes using DFCX easier, more friendly, and more pythonic for bot builders, developers, and maintainers.\n\n```\nSCRAPI --> Python Dialogflow CX\nas\nKeras --> Tensorflow\n```\n\n## What Can I Do With DFCX SCRAPI?\nWith DFCX SCRAPI you can perform many bot building and maintenance actions at scale including, but not limited to:\n- Create, Update, Delete, Get, and List for all CX resources types (i.e. Intents, Entity Types, Pages, Flows, etc.)\n- Convert commonly accessed CX Resources to Pandas Dataframes\n- Have fully automated conversations with a CX agent (powerful for regression testing!)\n- Extract Validation information\n- Extract Change History information\n- Search across all Flows/Pages/Routes to find a specific parameter or utterance using Search Util functions\n- Quickly move CX resources between agents using Copy Util functions!\n- Build the fundamental protobuf objects that CX uses for each resource type using Builder methods\n- ...and much, much more!\n\n## Built With\n* Python 3.8+\n\n\n<!-- AUTHENTICATION -->\n# Authentication \nAuthentication can vary depending on how and where you are interacting with SCRAPI.\n\n## Google Colab\nIf you're using SCRAPI with a [Google Colab](https://colab.research.google.com/) notebook, you can add the following to the top of your notebook for easy authentication:\n```py\nproject_id = '<YOUR_GCP_PROJECT_ID>'\n\n# this will launch an interactive prompt that allows you to auth with GCP in a browser\n!gcloud auth application-default login --no-launch-browser\n\n# this will set your active project to the `project_id` above\n!gcloud auth application-default set-quota-project $project_id\n```\n\nAfter running the above, Colab will pick up your credentials from the environment and pass them to SCRAPI directly. No need to use Service Account keys!\nYou can then use SCRAPI simply like this:\n```py\nfrom dfcx_scrapi.core.intents import Intents\n\nagent_id = '<YOUR_AGENT_ID>'\ni = Intents() # <-- Creds will be automatically picked up from the environment\nintents_map = i.get_intents_map(agent_id)\n```\n---\n## Cloud Functions / Cloud Run\nIf you're using SCRAPI with [Cloud Functions](https://cloud.google.com/functions) or [Cloud Run](https://cloud.google.com/run), SCRAPI can pick up on the default environment creds used by these services without any additional configuration! \n\n1. Add `dfcx-scrapi` to your `requirements.txt` file\n2. Ensure the Cloud Function / Cloud Run service account has the appropriate Dialogflow IAM Role\n\nOnce you are setup with the above, your function code can be used easily like this:\n```py\nfrom dfcx_scrapi.core.intents import Intents\n\nagent_id = '<YOUR_AGENT_ID>'\ni = Intents() # <-- Creds will be automatically picked up from the environment\nintents_map = i.get_intents_map(agent_id)\n```\n\n---\n## Local Python Environment\nSimilar to Cloud Functions / Cloud Run, SCRAPI can pick up on your local authentication creds _if you are using the gcloud CLI._\n\n1. Install [gcloud CLI](https://cloud.google.com/sdk/docs/install).\n2. Run `gcloud init`.\n3. Run `gcloud auth login`\n4. Run `gcloud auth list` to ensure your principal account is active.\n\nThis will authenticate your principal GCP account with the gcloud CLI, and SCRAPI can pick up the creds from here. \n\n---\n## Exceptions and Misc.\nThere are some classes in SCRAPI which still rely on Service Account Keys, notably the [DataframeFunctions](https://github.com/GoogleCloudPlatform/dfcx-scrapi/blob/main/src/dfcx_scrapi/tools/dataframe_functions.py) class due to how it authenticates with Google Sheets.\n\nIn order to use these functions, you will need a Service Account that has appropriate access to your GCP project. \nFor more information and to view the official documentation for service accounts go to [Creating and Managing GCP Service Accounts](https://cloud.google.com/iam/docs/creating-managing-service-accounts).\n\nOnce you've obtained a Service Account Key with appropriate permissions, you can use it as follows:\n```py\nfrom dfcx_scrapi.core.intents import Intents\nfrom dfcx_scrapi.tools.dataframe_functions import DataframeFunctions\n\nagent_id = '<YOUR_AGENT_ID>'\ncreds_path = '<PATH_TO_YOUR_SERVICE_ACCOUNT_JSON_FILE>'\n\ni = Intents(creds_path=creds_path)\ndffx = DataframeFunctions(creds_path=creds_path)\n\ndf = i.bulk_intent_to_df(agent_id)\ndffx.dataframe_to_sheets('GOOGLE_SHEET_NAME', 'TAB_NAME', df)\n```\n\n<!-- GETTING STARTED -->\n# Getting Started\n## Environment Setup\nSet up Google Cloud Platform credentials and install dependencies.\n```sh\ngcloud auth login\ngcloud auth application-default login\ngcloud config set project <project name>\n```\n```sh\npython3 -m venv venv\nsource ./venv/bin/activate\npip install -r requirements.txt\n```\n\n## Usage\nTo run a simple bit of code you can do the following:\n- Import a Class from `dfcx_scrapi.core`\n- Assign your Service Account to a local variable\n\n```python\nfrom dfcx_scrapi.core.intents import Intents\n\ncreds_path = '<PATH_TO_YOUR_SERVICE_ACCOUNT_JSON_FILE>'\nagent_path = '<FULL_DFCX_AGENT_ID_PATH>'\n\n# DFCX Agent ID paths are in this format:\n# 'projects/<project_id>/locations/<location_id>/agents/<agent_id>'\n\n# Instantiate your class object and pass in your credentials\ni = Intents(creds_path, agent_id=agent_path)\n\n# Retrieve all Intents and Training Phrases from an Agent and push to a Pandas DataFrame\ndf = i.bulk_intent_to_df()\n```\n\n# Library Composition\nHere is a brief overview of the SCRAPI library's structure and the motivation behind that structure.\n\n## Core \nThe [Core](/src/dfcx_scrapi/core) folder is synonymous with the core Resource types in the DFCX Agents (agents, intents, flows, etc.)\n* This folder contains the high level building blocks of SCRAPI\n* These classes and methods can be used to build higher level methods or custom tools and applications\n\n## Tools\nThe [Tools](/src/dfcx_scrapi/tools) folder contains various customized toolkits that allow you to do more complex bot management tasks, such as\n- Manipulate Agent Resource types into various DataFrame structures\n- Copy Agent Resources between Agents and GCP Projects on a resource by resource level\n- Move data to and from DFCX and other GCP Services like BigQuery, Sheets, etc.\n- Create customized search queries inside of your agent resources\n\n## Builders\nThe [Builders](/src/dfcx_scrapi/builders) folder contains simple methods for constructing the underlying protos in Dialogflow CX\n- Proto objects are the fundamental building blocks of Dialogflow CX\n- Builder classes allow the user to construct Dialogflow CX resource _offline_ without any API calls\n- Once the resource components are constructed, they can then be pushed to a live Dialogflow CX agent via API\n\n<!-- CONTRIBUTING -->\n# Contributing\nWe welcome any contributions or feature requests you would like to submit!\n\n1. Fork the Project\n2. Create your Feature Branch (git checkout -b feature/AmazingFeature)\n3. Commit your Changes (git commit -m 'Add some AmazingFeature')\n4. Push to the Branch (git push origin feature/AmazingFeature)\n5. Open a Pull Request\n\n<!-- LICENSE -->\n# License\nDistributed under the Apache 2.0 License. See [LICENSE](LICENSE.txt) for more information.\n\n<!-- CONTACT -->\n# Contact\nPatrick Marlow - pmarlow@google.com - [@kmaphoenix](https://github.com/kmaphoenix) \nMilad Tabrizi - miladt@google.com - [@MRyderOC](https://github.com/MRyderOC)\n\nProject Link: [https://github.com/GoogleCloudPlatform/dfcx-scrapi](https://github.com/GoogleCloudPlatform/dfcx-scrapi)\n\n<!-- ACKNOWLEDGEMENTS -->\n# Acknowledgements\n[Dialogflow CX Python Client Library](https://github.com/googleapis/python-dialogflow-cx) \n[Hugging Face - Pegasus Paraphrase](https://huggingface.co/tuner007/pegasus_paraphrase)\n\n\n\n<!-- MARKDOWN LINKS & IMAGES -->\n<!-- https://www.markdownguide.org/basic-syntax/#reference-style-links -->\n",
"bugtrack_url": null,
"license": null,
"summary": "A high level scripting API for bot builders, developers, and maintainers.",
"version": "1.12.5",
"project_urls": {
"Homepage": "https://github.com/GoogleCloudPlatform/dfcx-scrapi"
},
"split_keywords": [
"dialogflow",
" cx",
" google",
" bot",
" chatbot",
" intent",
" dfcx",
" entity"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "c28c5c7c5a621e9904949b2ac3bdff4573f875d6058c49d9f6f02177b1711add",
"md5": "2af3ffd44a80c23475f16e866c5eaa56",
"sha256": "bcbbc8050b68ace8b84f3a9edfab1ede6216f9a8dc83c04dd5fcc421a814fdb4"
},
"downloads": -1,
"filename": "dfcx_scrapi-1.12.5-py3-none-any.whl",
"has_sig": false,
"md5_digest": "2af3ffd44a80c23475f16e866c5eaa56",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4,>=3.6",
"size": 252523,
"upload_time": "2024-10-29T16:57:37",
"upload_time_iso_8601": "2024-10-29T16:57:37.318053Z",
"url": "https://files.pythonhosted.org/packages/c2/8c/5c7c5a621e9904949b2ac3bdff4573f875d6058c49d9f6f02177b1711add/dfcx_scrapi-1.12.5-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "453baf391279698a8ed3fff0ef478499c7c7012a2d7b59dad9238c402749b1df",
"md5": "796e1f2a023f237efb5f434ad73b874f",
"sha256": "ea3d1a720f78d682c0804a7a975b8b8ff4c1c077c77d35d72f0df728ec2a9ad5"
},
"downloads": -1,
"filename": "dfcx_scrapi-1.12.5.tar.gz",
"has_sig": false,
"md5_digest": "796e1f2a023f237efb5f434ad73b874f",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4,>=3.6",
"size": 1206991,
"upload_time": "2024-10-29T16:57:39",
"upload_time_iso_8601": "2024-10-29T16:57:39.417796Z",
"url": "https://files.pythonhosted.org/packages/45/3b/af391279698a8ed3fff0ef478499c7c7012a2d7b59dad9238c402749b1df/dfcx_scrapi-1.12.5.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-10-29 16:57:39",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "GoogleCloudPlatform",
"github_project": "dfcx-scrapi",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [
{
"name": "google-cloud-aiplatform",
"specs": [
[
">=",
"1.39.0"
]
]
},
{
"name": "google-cloud-dialogflow-cx",
"specs": [
[
">=",
"1.34.0"
]
]
},
{
"name": "google-cloud-discoveryengine",
"specs": [
[
">=",
"0.11.10"
]
]
},
{
"name": "google-auth",
"specs": [
[
">=",
"2.27.0"
]
]
},
{
"name": "google-oauth",
"specs": []
},
{
"name": "oauth2client",
"specs": []
},
{
"name": "pandas",
"specs": []
},
{
"name": "tabulate",
"specs": []
},
{
"name": "gspread",
"specs": [
[
"==",
"5.10.0"
]
]
},
{
"name": "gspread_dataframe",
"specs": []
},
{
"name": "numpy",
"specs": []
},
{
"name": "requests",
"specs": []
},
{
"name": "pylint",
"specs": [
[
"==",
"2.8.3"
]
]
},
{
"name": "pytest",
"specs": [
[
"==",
"6.0.2"
]
]
},
{
"name": "pytest-cov",
"specs": [
[
"==",
"2.11.1"
]
]
},
{
"name": "pytest-xdist",
"specs": [
[
"==",
"2.1.0"
]
]
},
{
"name": "pyyaml",
"specs": []
},
{
"name": "rouge-score",
"specs": []
},
{
"name": "torch",
"specs": []
},
{
"name": "transformers",
"specs": []
},
{
"name": "tqdm",
"specs": []
}
],
"lcname": "dfcx-scrapi"
}
JSON
