<p align="center">
<img src="https://github.com/permitio/opal/assets/4082578/4e21f85f-30ab-43e2-92de-b82f78888c71" height=170 alt="opal" border="0" />
</p>
<h2 align="center">
OPAL Fetcher for MySQL
</h2>
[Check out OPAL main repo here.](https://github.com/permitio/opal)
### What's in this repo?
An OPAL [custom fetch provider](https://docs.opal.ac/tutorials/write_your_own_fetch_provider) to bring authorization state from [MySQL](https://dev.mysql.com/).
This fetcher is both:
- **A fully functional fetch-provider for MySQL:** can be used by OPAL to fetch data from MySQL DB.
- **Serving as an example** how to write custom fetch providers for OPAL and how to publish them as pip packages.
### How to try this custom fetcher in one command? (Example docker-compose configuration)
You can test this fetcher with the example docker compose file in this repository root. Clone this repo, `cd` into the cloned repo, and then run:
```
docker compose up
```
this docker compose configuration already correctly configures OPAL to load the MySQL Fetch Provider, and correctly configures `OPAL_DATA_CONFIG_SOURCES` to include an entry that uses this fetcher.
### ✏️ How to use this fetcher in your OPAL Setup
#### 1) Build a custom opal-client `Dockerfile`
The official docker image only contains the built-in fetch providers. You need to create your own `Dockerfile` (that is based on the official docker image), that includes this fetcher's pip package.
Your `Dockerfile` should look like this:
```
FROM permitio/opal-client:latest
WORKDIR /app/
COPY . ./
RUN python setup.py install --user
```
#### 2) Build your custom opal-client container
Say your special Dockerfile from step one is called `custom_client.Dockerfile`.
You must build a customized OPAL container from this Dockerfile, like so:
```
docker build -t yourcompany/opal-client -f custom_client.Dockerfile .
```
#### 3) When running OPAL, set `OPAL_FETCH_PROVIDER_MODULES`
Pass the following environment variable to the OPAL client docker container (comma-separated provider modules):
```
OPAL_FETCH_PROVIDER_MODULES=opal_common.fetcher.providers,opal_fetcher_mysql.provider
```
Notice that OPAL receives a list from where to search for fetch providers.
The list in our case includes the built-in providers (`opal_common.fetcher.providers`) and our custom mysql provider.
#### 4) Using the custom provider in your DataSourceEntry objects
Your DataSourceEntry objects (either in `OPAL_DATA_CONFIG_SOURCES` or in dynamic updates sent via the OPAL publish API) can now include this fetcher's config.
Example value of `OPAL_DATA_CONFIG_SOURCES` (formatted nicely, but in env var you should pack this to one-line and no-spaces):
```json
{
"config": {
"entries": [
{
"url": "mysql://root:mysql@example_db:3306/test?password=mysql",
"config": {
"fetcher": "MySQLFetchProvider",
"query": "SELECT * from country;",
"connection_params": {
"host": "example_db",
"port": "3306",
"db": "test",
"user": "root",
"password": "mysql"
}
},
"topics": ["mysql"],
"dst_path": "countries"
}
]
}
}
```
Notice how `config` is an instance of `MySQLFetcherConfig` (code is in `opal_fetcher_mysql/provider.py`).
Values for this fetcher config:
- The `url` is actually a mysql dsn. You can set the mysql password in the dsn itself if you want however this implementation uses `connection_params` so you must include them.
- Your `config` must include the `fetcher` key to indicate to OPAL that you use a custom fetcher.
- Your `config` must include the `query` key to indicate what query to run against mysql.
### 🚩 Possible User Issues
While trying to send requests to a mysql data source, you may encounter that the request fails. This can be caused by the format of the config entry URL for which the standard is:
`mysql://<user>:<password>@<host>/<db>`
It might be most common that this request fails due to the password field being incorrectly parsed by the underlying library called `aiomysql`, which is one of the required libraries used within our OPAL custom data fetcher.
In order to solve the issue, you need to change the data source config entry URL to the format shown below:
`mysql://<host>/<db>?user=<user>&password=<password>`
### 📖 About OPAL (Open Policy Administration Layer)
[OPAL](https://github.com/permitio/opal) is an administration layer for Open Policy Agent (OPA), detecting changes to both policy and policy data in realtime and pushing live updates to your agents.
OPAL brings open-policy up to the speed needed by live applications. As your application state changes (whether it's via your APIs, DBs, git, S3 or 3rd-party SaaS services), OPAL will make sure your services are always in sync with the authorization data and policy they need (and only those they need).
Check out OPAL's main site at [OPAL.ac](https://opal.ac).
<img src="https://i.ibb.co/CvmX8rR/simplified-diagram-highlight.png" alt="simplified" border="0">
Raw data
{
"_id": null,
"home_page": null,
"name": "opal-fetcher-mysql",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.7",
"maintainer_email": null,
"keywords": "Open Policy Agent, OPA, OPAL, Open Policy Administration Layer, MySQL, Permit.io",
"author": null,
"author_email": "Bhimesh Agrawal <bhimeshagrawalggc@gmail.com>",
"download_url": "https://files.pythonhosted.org/packages/f8/9b/f9ae638f70f083e7a2c9fab28f97782bfbf570df300c1aa8521d16522302/opal_fetcher_mysql-1.0.3.tar.gz",
"platform": null,
"description": "<p align=\"center\">\n <img src=\"https://github.com/permitio/opal/assets/4082578/4e21f85f-30ab-43e2-92de-b82f78888c71\" height=170 alt=\"opal\" border=\"0\" />\n</p>\n<h2 align=\"center\">\nOPAL Fetcher for MySQL\n</h2>\n\n[Check out OPAL main repo here.](https://github.com/permitio/opal)\n\n### What's in this repo?\n\nAn OPAL [custom fetch provider](https://docs.opal.ac/tutorials/write_your_own_fetch_provider) to bring authorization state from [MySQL](https://dev.mysql.com/).\n\nThis fetcher is both:\n\n- **A fully functional fetch-provider for MySQL:** can be used by OPAL to fetch data from MySQL DB.\n- **Serving as an example** how to write custom fetch providers for OPAL and how to publish them as pip packages.\n\n### How to try this custom fetcher in one command? (Example docker-compose configuration)\n\nYou can test this fetcher with the example docker compose file in this repository root. Clone this repo, `cd` into the cloned repo, and then run:\n\n```\ndocker compose up\n```\n\nthis docker compose configuration already correctly configures OPAL to load the MySQL Fetch Provider, and correctly configures `OPAL_DATA_CONFIG_SOURCES` to include an entry that uses this fetcher.\n\n### \u270f\ufe0f How to use this fetcher in your OPAL Setup\n\n#### 1) Build a custom opal-client `Dockerfile`\n\nThe official docker image only contains the built-in fetch providers. You need to create your own `Dockerfile` (that is based on the official docker image), that includes this fetcher's pip package.\n\nYour `Dockerfile` should look like this:\n\n```\nFROM permitio/opal-client:latest\nWORKDIR /app/\nCOPY . ./\nRUN python setup.py install --user\n```\n\n#### 2) Build your custom opal-client container\n\nSay your special Dockerfile from step one is called `custom_client.Dockerfile`.\n\nYou must build a customized OPAL container from this Dockerfile, like so:\n\n```\ndocker build -t yourcompany/opal-client -f custom_client.Dockerfile .\n```\n\n#### 3) When running OPAL, set `OPAL_FETCH_PROVIDER_MODULES`\n\nPass the following environment variable to the OPAL client docker container (comma-separated provider modules):\n\n```\nOPAL_FETCH_PROVIDER_MODULES=opal_common.fetcher.providers,opal_fetcher_mysql.provider\n```\n\nNotice that OPAL receives a list from where to search for fetch providers.\nThe list in our case includes the built-in providers (`opal_common.fetcher.providers`) and our custom mysql provider.\n\n#### 4) Using the custom provider in your DataSourceEntry objects\n\nYour DataSourceEntry objects (either in `OPAL_DATA_CONFIG_SOURCES` or in dynamic updates sent via the OPAL publish API) can now include this fetcher's config.\n\nExample value of `OPAL_DATA_CONFIG_SOURCES` (formatted nicely, but in env var you should pack this to one-line and no-spaces):\n\n```json\n{\n \"config\": {\n \"entries\": [\n {\n \"url\": \"mysql://root:mysql@example_db:3306/test?password=mysql\",\n \"config\": {\n \"fetcher\": \"MySQLFetchProvider\",\n \"query\": \"SELECT * from country;\",\n \"connection_params\": {\n \"host\": \"example_db\",\n \"port\": \"3306\",\n \"db\": \"test\",\n \"user\": \"root\",\n \"password\": \"mysql\"\n }\n },\n \"topics\": [\"mysql\"],\n \"dst_path\": \"countries\"\n }\n ]\n }\n}\n```\n\nNotice how `config` is an instance of `MySQLFetcherConfig` (code is in `opal_fetcher_mysql/provider.py`).\n\nValues for this fetcher config:\n\n- The `url` is actually a mysql dsn. You can set the mysql password in the dsn itself if you want however this implementation uses `connection_params` so you must include them.\n- Your `config` must include the `fetcher` key to indicate to OPAL that you use a custom fetcher.\n- Your `config` must include the `query` key to indicate what query to run against mysql.\n\n### \ud83d\udea9 Possible User Issues\n\nWhile trying to send requests to a mysql data source, you may encounter that the request fails. This can be caused by the format of the config entry URL for which the standard is:\n\n`mysql://<user>:<password>@<host>/<db>`\n\nIt might be most common that this request fails due to the password field being incorrectly parsed by the underlying library called `aiomysql`, which is one of the required libraries used within our OPAL custom data fetcher.\n\nIn order to solve the issue, you need to change the data source config entry URL to the format shown below:\n\n`mysql://<host>/<db>?user=<user>&password=<password>`\n\n### \ud83d\udcd6 About OPAL (Open Policy Administration Layer)\n\n[OPAL](https://github.com/permitio/opal) is an administration layer for Open Policy Agent (OPA), detecting changes to both policy and policy data in realtime and pushing live updates to your agents.\n\nOPAL brings open-policy up to the speed needed by live applications. As your application state changes (whether it's via your APIs, DBs, git, S3 or 3rd-party SaaS services), OPAL will make sure your services are always in sync with the authorization data and policy they need (and only those they need).\n\nCheck out OPAL's main site at [OPAL.ac](https://opal.ac).\n\n<img src=\"https://i.ibb.co/CvmX8rR/simplified-diagram-highlight.png\" alt=\"simplified\" border=\"0\">\n",
"bugtrack_url": null,
"license": "Apache-2.0",
"summary": "An OPAL fetch provider to bring authorization state from MySQL DB.",
"version": "1.0.3",
"project_urls": {
"Bug Tracker": "https://github.com/bhimeshagrawal/opal-fetcher-mysql/issues",
"Source": "https://github.com/bhimeshagrawal/opal-fetcher-mysql"
},
"split_keywords": [
"open policy agent",
" opa",
" opal",
" open policy administration layer",
" mysql",
" permit.io"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "a159788c9cc809abd2c1336ad84bb4add521140da92c3cc723042015af2a1d0a",
"md5": "db92db73d6201c978cd825fe75fc4197",
"sha256": "94488835d8fcb4ec148231eccfad79190b9feae27daeae8629fc2b68b9c65ded"
},
"downloads": -1,
"filename": "opal_fetcher_mysql-1.0.3-py3-none-any.whl",
"has_sig": false,
"md5_digest": "db92db73d6201c978cd825fe75fc4197",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7",
"size": 9587,
"upload_time": "2024-06-16T10:25:19",
"upload_time_iso_8601": "2024-06-16T10:25:19.430418Z",
"url": "https://files.pythonhosted.org/packages/a1/59/788c9cc809abd2c1336ad84bb4add521140da92c3cc723042015af2a1d0a/opal_fetcher_mysql-1.0.3-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "f89bf9ae638f70f083e7a2c9fab28f97782bfbf570df300c1aa8521d16522302",
"md5": "6bc266f6399e950fb0b4419c2cdb4d3a",
"sha256": "402eae1591d0d7dfcd10e93c8da6a0299693873a58a724edc3a9d8a2c0b0aaa8"
},
"downloads": -1,
"filename": "opal_fetcher_mysql-1.0.3.tar.gz",
"has_sig": false,
"md5_digest": "6bc266f6399e950fb0b4419c2cdb4d3a",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7",
"size": 9548,
"upload_time": "2024-06-16T10:25:20",
"upload_time_iso_8601": "2024-06-16T10:25:20.576717Z",
"url": "https://files.pythonhosted.org/packages/f8/9b/f9ae638f70f083e7a2c9fab28f97782bfbf570df300c1aa8521d16522302/opal_fetcher_mysql-1.0.3.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-06-16 10:25:20",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "bhimeshagrawal",
"github_project": "opal-fetcher-mysql",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "opal-fetcher-mysql"
}
JSON
