# Connector Development Kit \(Python\)
The Airbyte Python CDK is a framework for rapidly developing production-grade Airbyte connectors. The CDK currently offers helpers specific for creating Airbyte source connectors for:
* HTTP APIs \(REST APIs, GraphQL, etc..\)
* Singer Taps
* Generic Python sources \(anything not covered by the above\)
The CDK provides an improved developer experience by providing basic implementation structure and abstracting away low-level glue boilerplate.
This document is a general introduction to the CDK. Readers should have basic familiarity with the [Airbyte Specification](https://docs.airbyte.com/understanding-airbyte/airbyte-protocol/) before proceeding.
## Getting Started
Generate an empty connector using the code generator. First clone the Airbyte repository then from the repository root run
```text
cd airbyte-integrations/connector-templates/generator
./generate.sh
```
then follow the interactive prompt. Next, find all `TODO`s in the generated project directory -- they're accompanied by lots of comments explaining what you'll need to do in order to implement your connector. Upon completing all TODOs properly, you should have a functioning connector.
Additionally, you can follow [this tutorial](https://docs.airbyte.io/connector-development/tutorials/cdk-tutorial-python-http) for a complete walkthrough of creating an HTTP connector using the Airbyte CDK.
### Concepts & Documentation
See the [concepts docs](docs/concepts/) for a tour through what the API offers.
### Example Connectors
**HTTP Connectors**:
* [Exchangerates API](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-exchange-rates/source_exchange_rates/source.py)
* [Stripe](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-stripe/source_stripe/source.py)
* [Slack](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-slack/source_slack/source.py)
**Singer connectors**:
* [Salesforce](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-salesforce-singer/source_salesforce_singer/source.py)
* [Github](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-github-singer/source_github_singer/source.py)
**Simple Python connectors using the barebones `Source` abstraction**:
* [Google Sheets](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-google-sheets/google_sheets_source/google_sheets_source.py)
* [Mailchimp](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-mailchimp/source_mailchimp/source.py)
## Contributing
### First time setup
We assume `python` points to python >=3.8.
Setup a virtual env:
```text
python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]" # [dev] installs development-only dependencies
```
#### Iteration
* Iterate on the code locally
* Run tests via `python -m pytest -s unit_tests`
* Perform static type checks using `mypy airbyte_cdk`. `MyPy` configuration is in `.mypy.ini`.
* The `type_check_and_test.sh` script bundles both type checking and testing in one convenient command. Feel free to use it!
##### Autogenerated files
If the iteration you are working on includes changes to the models, you might want to regenerate them. In order to do that, you can run:
```commandline
SUB_BUILD=CDK ./gradlew format
```
This will generate the files based on the schemas, add the license information and format the code. If you want to only do the former and rely on
pre-commit to the others, you can run the appropriate generation command i.e. `./gradlew generateComponentManifestClassFiles`.
#### Testing
All tests are located in the `unit_tests` directory. Run `python -m pytest --cov=airbyte_cdk unit_tests/` to run them. This also presents a test coverage report.
#### Building and testing a connector with your local CDK
When developing a new feature in the CDK, you may find it helpful to run a connector that uses that new feature. You can test this in one of two ways:
* Running a connector locally
* Building and running a source via Docker
##### Installing your local CDK into a local Python connector
In order to get a local Python connector running your local CDK, do the following.
First, make sure you have your connector's virtual environment active:
```bash
# from the `airbyte/airbyte-integrations/connectors/<connector-directory>` directory
source .venv/bin/activate
# if you haven't installed dependencies for your connector already
pip install -e .
```
Then, navigate to the CDK and install it in editable mode:
```bash
cd ../../../airbyte-cdk/python
pip install -e .
```
You should see that `pip` has uninstalled the version of `airbyte-cdk` defined by your connector's `setup.py` and installed your local CDK. Any changes you make will be immediately reflected in your editor, so long as your editor's interpreter is set to your connector's virtual environment.
##### Building a Python connector in Docker with your local CDK installed
You can build your connector image with the local CDK using
```bash
# from the airbytehq/airbyte base directory
CONNECTOR_TAG=<TAG_NAME> CONNECTOR_NAME=<CONNECTOR_NAME> sh airbyte-integrations/scripts/build-connector-image-with-local-cdk.sh
```
Note that the local CDK is injected at build time, so if you make changes, you will have to run the build command again to see them reflected.
##### Running Connector Acceptance Tests for a single connector in Docker with your local CDK installed
To run acceptance tests for a single connectors using the local CDK, from the connector directory, run
```bash
LOCAL_CDK=1 sh acceptance-test-docker.sh
```
To additionally fetch secrets required by CATs, set the `FETCH_SECRETS` environment variable. This requires you to have a Google Service Account, and the GCP_GSM_CREDENTIALS environment variable to be set, per the instructions [here](https://github.com/airbytehq/airbyte/tree/b03653a24ef16be641333380f3a4d178271df0ee/tools/ci_credentials).
##### Running Connector Acceptance Tests for multiple connectors in Docker with your local CDK installed
To run acceptance tests for multiple connectors using the local CDK, from the root of the `airbyte` repo, run
```bash
./airbyte-cdk/python/bin/run-cats-with-local-cdk.sh -c <connector1>,<connector2>,...
```
#### Publishing a new version to PyPi
1. Open a PR
2. Once it is approved and **merged**, an Airbyte member must run the `Publish CDK Manually` workflow from master using `release-type=major|manor|patch` and setting the changelog message.
## Coming Soon
* Full OAuth 2.0 support \(including refresh token issuing flow via UI or CLI\)
* Airbyte Java HTTP CDK
* CDK for Async HTTP endpoints \(request-poll-wait style endpoints\)
* CDK for other protocols
* Don't see a feature you need? [Create an issue and let us know how we can help!](https://github.com/airbytehq/airbyte/issues/new?assignees=&labels=type%2Fenhancement&template=feature-request.md&title=)
Raw data
{
"_id": null,
"home_page": "https://github.com/airbytehq/airbyte",
"name": "valmi-airbyte-cdk",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": "",
"keywords": "airbyte connector-development-kit cdk",
"author": "Airbyte",
"author_email": "contact@airbyte.io",
"download_url": "https://files.pythonhosted.org/packages/b0/0c/5c55df2f84fd9ab64bd18a629264fcfe93949cdb1fd1ceee8d5a81e61692/valmi-airbyte-cdk-0.30.3.tar.gz",
"platform": null,
"description": "# Connector Development Kit \\(Python\\)\n\nThe Airbyte Python CDK is a framework for rapidly developing production-grade Airbyte connectors. The CDK currently offers helpers specific for creating Airbyte source connectors for:\n\n* HTTP APIs \\(REST APIs, GraphQL, etc..\\)\n* Singer Taps\n* Generic Python sources \\(anything not covered by the above\\)\n\nThe CDK provides an improved developer experience by providing basic implementation structure and abstracting away low-level glue boilerplate.\n\nThis document is a general introduction to the CDK. Readers should have basic familiarity with the [Airbyte Specification](https://docs.airbyte.com/understanding-airbyte/airbyte-protocol/) before proceeding.\n\n## Getting Started\n\nGenerate an empty connector using the code generator. First clone the Airbyte repository then from the repository root run\n\n```text\ncd airbyte-integrations/connector-templates/generator\n./generate.sh\n```\n\nthen follow the interactive prompt. Next, find all `TODO`s in the generated project directory -- they're accompanied by lots of comments explaining what you'll need to do in order to implement your connector. Upon completing all TODOs properly, you should have a functioning connector.\n\nAdditionally, you can follow [this tutorial](https://docs.airbyte.io/connector-development/tutorials/cdk-tutorial-python-http) for a complete walkthrough of creating an HTTP connector using the Airbyte CDK.\n\n### Concepts & Documentation\n\nSee the [concepts docs](docs/concepts/) for a tour through what the API offers.\n\n### Example Connectors\n\n**HTTP Connectors**:\n\n* [Exchangerates API](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-exchange-rates/source_exchange_rates/source.py)\n* [Stripe](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-stripe/source_stripe/source.py)\n* [Slack](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-slack/source_slack/source.py)\n\n**Singer connectors**:\n\n* [Salesforce](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-salesforce-singer/source_salesforce_singer/source.py)\n* [Github](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-github-singer/source_github_singer/source.py)\n\n**Simple Python connectors using the barebones `Source` abstraction**:\n\n* [Google Sheets](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-google-sheets/google_sheets_source/google_sheets_source.py)\n* [Mailchimp](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-mailchimp/source_mailchimp/source.py)\n\n## Contributing\n\n### First time setup\n\nWe assume `python` points to python >=3.8.\n\nSetup a virtual env:\n\n```text\npython -m venv .venv\nsource .venv/bin/activate\npip install -e \".[dev]\" # [dev] installs development-only dependencies\n```\n\n#### Iteration\n\n* Iterate on the code locally\n* Run tests via `python -m pytest -s unit_tests`\n* Perform static type checks using `mypy airbyte_cdk`. `MyPy` configuration is in `.mypy.ini`.\n* The `type_check_and_test.sh` script bundles both type checking and testing in one convenient command. Feel free to use it!\n\n##### Autogenerated files\nIf the iteration you are working on includes changes to the models, you might want to regenerate them. In order to do that, you can run:\n```commandline\nSUB_BUILD=CDK ./gradlew format\n```\nThis will generate the files based on the schemas, add the license information and format the code. If you want to only do the former and rely on\npre-commit to the others, you can run the appropriate generation command i.e. `./gradlew generateComponentManifestClassFiles`.\n\n#### Testing\n\nAll tests are located in the `unit_tests` directory. Run `python -m pytest --cov=airbyte_cdk unit_tests/` to run them. This also presents a test coverage report.\n\n#### Building and testing a connector with your local CDK\n\nWhen developing a new feature in the CDK, you may find it helpful to run a connector that uses that new feature. You can test this in one of two ways:\n* Running a connector locally\n* Building and running a source via Docker\n\n##### Installing your local CDK into a local Python connector\n\nIn order to get a local Python connector running your local CDK, do the following.\n\nFirst, make sure you have your connector's virtual environment active:\n```bash\n# from the `airbyte/airbyte-integrations/connectors/<connector-directory>` directory\nsource .venv/bin/activate\n\n# if you haven't installed dependencies for your connector already\npip install -e .\n```\n\nThen, navigate to the CDK and install it in editable mode:\n```bash\ncd ../../../airbyte-cdk/python\npip install -e .\n```\n\nYou should see that `pip` has uninstalled the version of `airbyte-cdk` defined by your connector's `setup.py` and installed your local CDK. Any changes you make will be immediately reflected in your editor, so long as your editor's interpreter is set to your connector's virtual environment.\n\n##### Building a Python connector in Docker with your local CDK installed\n\nYou can build your connector image with the local CDK using\n```bash\n# from the airbytehq/airbyte base directory\nCONNECTOR_TAG=<TAG_NAME> CONNECTOR_NAME=<CONNECTOR_NAME> sh airbyte-integrations/scripts/build-connector-image-with-local-cdk.sh\n```\nNote that the local CDK is injected at build time, so if you make changes, you will have to run the build command again to see them reflected.\n\n##### Running Connector Acceptance Tests for a single connector in Docker with your local CDK installed\n\nTo run acceptance tests for a single connectors using the local CDK, from the connector directory, run\n```bash\nLOCAL_CDK=1 sh acceptance-test-docker.sh\n```\nTo additionally fetch secrets required by CATs, set the `FETCH_SECRETS` environment variable. This requires you to have a Google Service Account, and the GCP_GSM_CREDENTIALS environment variable to be set, per the instructions [here](https://github.com/airbytehq/airbyte/tree/b03653a24ef16be641333380f3a4d178271df0ee/tools/ci_credentials).\n\n##### Running Connector Acceptance Tests for multiple connectors in Docker with your local CDK installed\n\nTo run acceptance tests for multiple connectors using the local CDK, from the root of the `airbyte` repo, run\n```bash\n./airbyte-cdk/python/bin/run-cats-with-local-cdk.sh -c <connector1>,<connector2>,...\n```\n\n#### Publishing a new version to PyPi\n\n1. Open a PR\n2. Once it is approved and **merged**, an Airbyte member must run the `Publish CDK Manually` workflow from master using `release-type=major|manor|patch` and setting the changelog message.\n\n## Coming Soon\n\n* Full OAuth 2.0 support \\(including refresh token issuing flow via UI or CLI\\) \n* Airbyte Java HTTP CDK\n* CDK for Async HTTP endpoints \\(request-poll-wait style endpoints\\)\n* CDK for other protocols\n* Don't see a feature you need? [Create an issue and let us know how we can help!](https://github.com/airbytehq/airbyte/issues/new?assignees=&labels=type%2Fenhancement&template=feature-request.md&title=)\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "A framework for writing Airbyte Connectors.",
"version": "0.30.3",
"split_keywords": [
"airbyte",
"connector-development-kit",
"cdk"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "084a2e09e3f5e1a5a4ad0262ec6c6af8d10554ebc8b173bf96f9c48c672c9c2c",
"md5": "ae7bbd5ba2a3dd8ef017f6b3370f3435",
"sha256": "fc9972897f7929caaa902a94a5b92054c55d710604e88fbcc40249d483929048"
},
"downloads": -1,
"filename": "valmi_airbyte_cdk-0.30.3-py3-none-any.whl",
"has_sig": false,
"md5_digest": "ae7bbd5ba2a3dd8ef017f6b3370f3435",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 193153,
"upload_time": "2023-04-17T11:11:46",
"upload_time_iso_8601": "2023-04-17T11:11:46.640871Z",
"url": "https://files.pythonhosted.org/packages/08/4a/2e09e3f5e1a5a4ad0262ec6c6af8d10554ebc8b173bf96f9c48c672c9c2c/valmi_airbyte_cdk-0.30.3-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "b00c5c55df2f84fd9ab64bd18a629264fcfe93949cdb1fd1ceee8d5a81e61692",
"md5": "28a34d76a1cd5bb36c0a6a908799360d",
"sha256": "590e5d9f322deb6ef45712a6c44ec8a994653d5921e4d527fbe0264c0dcfe077"
},
"downloads": -1,
"filename": "valmi-airbyte-cdk-0.30.3.tar.gz",
"has_sig": false,
"md5_digest": "28a34d76a1cd5bb36c0a6a908799360d",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 196299,
"upload_time": "2023-04-17T11:11:50",
"upload_time_iso_8601": "2023-04-17T11:11:50.010698Z",
"url": "https://files.pythonhosted.org/packages/b0/0c/5c55df2f84fd9ab64bd18a629264fcfe93949cdb1fd1ceee8d5a81e61692/valmi-airbyte-cdk-0.30.3.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-04-17 11:11:50",
"github": true,
"gitlab": false,
"bitbucket": false,
"github_user": "airbytehq",
"github_project": "airbyte",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "valmi-airbyte-cdk"
}
JSON
