| Name | airbyte-lib JSON |
| Version |
0.1.0
JSON |
| download |
| home_page | |
| Summary | AirbyteLib |
| upload_time | 2024-02-12 16:51:18 |
| maintainer | |
| docs_url | None |
| author | Airbyte |
| requires_python | >=3.9,<4.0 |
| license | |
| keywords |
|
| VCS |
|
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# airbyte-lib
airbyte-lib is a library that allows to run Airbyte syncs embedded into any Python application, without the need to run Airbyte server.
## Development
- Make sure [Poetry is installed](https://python-poetry.org/docs/#).
- Run `poetry install`
- For examples, check out the `examples` folder. They can be run via `poetry run python examples/<example file>`
- Unit tests and type checks can be run via `poetry run pytest`
## Release
- In your PR:
- Bump the version in `pyproject.toml`
- Add a changelog entry to the table below
- Once the PR is merged, go to Github and trigger the `Publish AirbyteLib Manually` workflow. This will publish the new version to PyPI.
## Secrets Management
AirbyteLib can auto-import secrets from the following sources:
1. Environment variables.
2. [Google Colab secrets](https://medium.com/@parthdasawant/how-to-use-secrets-in-google-colab-450c38e3ec75).
3. Manual entry via [`getpass`](https://docs.python.org/3.9/library/getpass.html).
_Note: Additional secret store options may be supported in the future. [More info here.](https://github.com/airbytehq/airbyte-lib-private-beta/discussions/5)_
### Retrieving Secrets
```python
from airbyte_lib import get_secret, SecretSource
source = get_connection("source-github")
source.set_config(
"credentials": {
"personal_access_token": get_secret("GITHUB_PERSONAL_ACCESS_TOKEN"),
}
)
```
The `get_secret()` function accepts an optional `source` argument of enum type `SecretSource`. If omitted or set to `SecretSource.ANY`, AirbyteLib will search all available secrets sources. If `source` is set to a specific source, then only that source will be checked. If a list of `SecretSource` entries is passed, then the sources will be checked using the provided ordering.
By default, AirbyteLib will prompt the user for any requested secrets that are not provided via other secret managers. You can disable this prompt by passing `prompt=False` to `get_secret()`.
### Versioning
Versioning follows [Semantic Versioning](https://semver.org/). For new features, bump the minor version. For bug fixes, bump the patch version. For pre-releases, append `dev.N` to the version. For example, `0.1.0dev.1` is the first pre-release of the `0.1.0` version.
## Documentation
Regular documentation lives in the `/docs` folder. Based on the doc strings of public methods, we generate API documentation using [pdoc](https://pdoc.dev). To generate the documentation, run `poetry run generate-docs`. The documentation will be generated in the `docs/generate` folder. This needs to be done manually when changing the public interface of the library.
A unit test validates the documentation is up to date.
## Connector compatibility
To make a connector compatible with airbyte-lib, the following requirements must be met:
* The connector must be a Python package, with a `pyproject.toml` or a `setup.py` file.
* In the package, there must be a `run.py` file that contains a `run` method. This method should read arguments from the command line, and run the connector with them, outputting messages to stdout.
* The `pyproject.toml` or `setup.py` file must specify a command line entry point for the `run` method called `source-<connector name>`. This is usually done by adding a `console_scripts` section to the `pyproject.toml` file, or a `entry_points` section to the `setup.py` file. For example:
```toml
[tool.poetry.scripts]
source-my-connector = "my_connector.run:run"
```
```python
setup(
...
entry_points={
'console_scripts': [
'source-my-connector = my_connector.run:run',
],
},
...
)
```
To publish a connector to PyPI, specify the `pypi` section in the `metadata.yaml` file. For example:
```yaml
data:
# ...
remoteRegistries:
pypi:
enabled: true
packageName: "airbyte-source-my-connector"
```
## Validating source connectors
To validate a source connector for compliance, the `airbyte-lib-validate-source` script can be used. It can be used like this:
```bash
airbyte-lib-validate-source —connector-dir . -—sample-config secrets/config.json
```
The script will install the python package in the provided directory, and run the connector against the provided config. The config should be a valid JSON file, with the same structure as the one that would be provided to the connector in Airbyte. The script will exit with a non-zero exit code if the connector fails to run.
For a more lightweight check, the `--validate-install-only` flag can be used. This will only check that the connector can be installed and returns a spec, no sample config required.
## Changelog
| Version | PR | Description |
| ----------- | ---------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------- |
| 0.1.0 | [#35184](https://github.com/airbytehq/airbyte/pull/35184) | Beta Release 0.1.0 |
| 0.1.0dev.2 | [#34111](https://github.com/airbytehq/airbyte/pull/34111) | Initial publish - add publish workflow |
Raw data
{
"_id": null,
"home_page": "",
"name": "airbyte-lib",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.9,<4.0",
"maintainer_email": "",
"keywords": "",
"author": "Airbyte",
"author_email": "contact@airbyte.io",
"download_url": "https://files.pythonhosted.org/packages/60/39/c337d510c8df129fa6680f9f967bf3d18029d6ed73447f2ff44f20b8fcc4/airbyte_lib-0.1.0.tar.gz",
"platform": null,
"description": "# airbyte-lib\n\nairbyte-lib is a library that allows to run Airbyte syncs embedded into any Python application, without the need to run Airbyte server.\n\n## Development\n\n- Make sure [Poetry is installed](https://python-poetry.org/docs/#).\n- Run `poetry install`\n- For examples, check out the `examples` folder. They can be run via `poetry run python examples/<example file>`\n- Unit tests and type checks can be run via `poetry run pytest`\n\n## Release\n\n- In your PR:\n - Bump the version in `pyproject.toml`\n - Add a changelog entry to the table below\n- Once the PR is merged, go to Github and trigger the `Publish AirbyteLib Manually` workflow. This will publish the new version to PyPI.\n\n## Secrets Management\n\nAirbyteLib can auto-import secrets from the following sources:\n\n1. Environment variables.\n2. [Google Colab secrets](https://medium.com/@parthdasawant/how-to-use-secrets-in-google-colab-450c38e3ec75).\n3. Manual entry via [`getpass`](https://docs.python.org/3.9/library/getpass.html).\n\n_Note: Additional secret store options may be supported in the future. [More info here.](https://github.com/airbytehq/airbyte-lib-private-beta/discussions/5)_\n\n### Retrieving Secrets\n\n```python\nfrom airbyte_lib import get_secret, SecretSource\n\nsource = get_connection(\"source-github\")\nsource.set_config(\n \"credentials\": {\n \"personal_access_token\": get_secret(\"GITHUB_PERSONAL_ACCESS_TOKEN\"),\n }\n)\n```\n\nThe `get_secret()` function accepts an optional `source` argument of enum type `SecretSource`. If omitted or set to `SecretSource.ANY`, AirbyteLib will search all available secrets sources. If `source` is set to a specific source, then only that source will be checked. If a list of `SecretSource` entries is passed, then the sources will be checked using the provided ordering.\n\nBy default, AirbyteLib will prompt the user for any requested secrets that are not provided via other secret managers. You can disable this prompt by passing `prompt=False` to `get_secret()`.\n\n### Versioning\n\nVersioning follows [Semantic Versioning](https://semver.org/). For new features, bump the minor version. For bug fixes, bump the patch version. For pre-releases, append `dev.N` to the version. For example, `0.1.0dev.1` is the first pre-release of the `0.1.0` version.\n\n## Documentation\n\nRegular documentation lives in the `/docs` folder. Based on the doc strings of public methods, we generate API documentation using [pdoc](https://pdoc.dev). To generate the documentation, run `poetry run generate-docs`. The documentation will be generated in the `docs/generate` folder. This needs to be done manually when changing the public interface of the library.\n\nA unit test validates the documentation is up to date.\n\n## Connector compatibility\n\nTo make a connector compatible with airbyte-lib, the following requirements must be met:\n* The connector must be a Python package, with a `pyproject.toml` or a `setup.py` file.\n* In the package, there must be a `run.py` file that contains a `run` method. This method should read arguments from the command line, and run the connector with them, outputting messages to stdout.\n* The `pyproject.toml` or `setup.py` file must specify a command line entry point for the `run` method called `source-<connector name>`. This is usually done by adding a `console_scripts` section to the `pyproject.toml` file, or a `entry_points` section to the `setup.py` file. For example:\n\n```toml\n[tool.poetry.scripts]\nsource-my-connector = \"my_connector.run:run\"\n```\n\n```python\nsetup(\n ...\n entry_points={\n 'console_scripts': [\n 'source-my-connector = my_connector.run:run',\n ],\n },\n ...\n)\n```\n\nTo publish a connector to PyPI, specify the `pypi` section in the `metadata.yaml` file. For example:\n\n```yaml\ndata:\n # ...\n remoteRegistries:\n pypi:\n enabled: true\n packageName: \"airbyte-source-my-connector\"\n```\n\n## Validating source connectors\n\nTo validate a source connector for compliance, the `airbyte-lib-validate-source` script can be used. It can be used like this:\n\n```bash\nairbyte-lib-validate-source \u2014connector-dir . -\u2014sample-config secrets/config.json\n```\n\nThe script will install the python package in the provided directory, and run the connector against the provided config. The config should be a valid JSON file, with the same structure as the one that would be provided to the connector in Airbyte. The script will exit with a non-zero exit code if the connector fails to run.\n\nFor a more lightweight check, the `--validate-install-only` flag can be used. This will only check that the connector can be installed and returns a spec, no sample config required.\n\n## Changelog\n\n| Version | PR | Description |\n| ----------- | ---------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------- |\n| 0.1.0 | [#35184](https://github.com/airbytehq/airbyte/pull/35184) | Beta Release 0.1.0 |\n| 0.1.0dev.2 | [#34111](https://github.com/airbytehq/airbyte/pull/34111) | Initial publish - add publish workflow |\n",
"bugtrack_url": null,
"license": "",
"summary": "AirbyteLib",
"version": "0.1.0",
"project_urls": null,
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "1ed9b68a45bceb9ff1717e3b279058950dfeb6d58dda9445213196835933146a",
"md5": "dc2f1151f592ae098431cf3603ebc447",
"sha256": "02b8918556a4f49461dcc652d012b4a45e662d475b0e90d8daac5772f00adc02"
},
"downloads": -1,
"filename": "airbyte_lib-0.1.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "dc2f1151f592ae098431cf3603ebc447",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9,<4.0",
"size": 60703,
"upload_time": "2024-02-12T16:51:17",
"upload_time_iso_8601": "2024-02-12T16:51:17.111654Z",
"url": "https://files.pythonhosted.org/packages/1e/d9/b68a45bceb9ff1717e3b279058950dfeb6d58dda9445213196835933146a/airbyte_lib-0.1.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "6039c337d510c8df129fa6680f9f967bf3d18029d6ed73447f2ff44f20b8fcc4",
"md5": "90f53ed43d4ac7f81113ce34bd35c5ca",
"sha256": "25535fed4e2dadfaf350dc6345b923e7d6551b693579ca7f8888a218fc2f0031"
},
"downloads": -1,
"filename": "airbyte_lib-0.1.0.tar.gz",
"has_sig": false,
"md5_digest": "90f53ed43d4ac7f81113ce34bd35c5ca",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9,<4.0",
"size": 47881,
"upload_time": "2024-02-12T16:51:18",
"upload_time_iso_8601": "2024-02-12T16:51:18.950444Z",
"url": "https://files.pythonhosted.org/packages/60/39/c337d510c8df129fa6680f9f967bf3d18029d6ed73447f2ff44f20b8fcc4/airbyte_lib-0.1.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-02-12 16:51:18",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "airbyte-lib"
}
