# `tap-stackexchange`
[](https://github.com/MeltanoLabs/tap-stackexchange/actions/workflows/test.yml)
[](https://sonarcloud.io/summary/new_code?id=MeltanoLabs_tap-stackexchange)
[](https://sonarcloud.io/summary/new_code?id=MeltanoLabs_tap-stackexchange)
[](https://sonarcloud.io/summary/new_code?id=MeltanoLabs_tap-stackexchange)
Singer tap for the StackExchange API.
Built with the [Meltano SDK](https://sdk.meltano.com) for Singer Taps and Targets.
## Capabilities
* `sync`
* `catalog`
* `state`
* `discover`
## Settings
| Setting | Required | Default | Description |
|:----------|:--------:|:-------:|:------------|
| key | False | None | Pass this to receive a higher request quota |
| site | False | stackoverflow.com | StackExchange site |
| tags | False | None | Question tags |
| start_date| False | None | The earliest record date to sync |
A full list of supported settings and capabilities is available by running: `tap-stackexchange --about`
## Custom filter
The StackExchange API supports a number of [custom filters](https://api.stackexchange.com/docs/filters) that can be used to
include or exclude certain fields from the response objects. This application has a baked-in filter with the following
parameters:
- `include`
| Parameter | Description |
|:-------------------------|:-----------------------------------------|
| `question.comment_count` | The number of comments on the question |
| `tag.last_activity_date` | The date of the last activity on the tag |
- `unsafe=false`
### Update the baked-in filter
To update the baked-in filter, edit the `FILTER_ID` constant in `tap_stackexchange/tap.py`.
To generate a new filter, use the _Try It_ button on the [StackExchange API documentation](https://api.stackexchange.com/docs/create-filter), using the baked-in filter as the `base` parameter.
### Use a custom filter
To use a custom filter, set the `filter` setting to the filter ID. Note that if you use a custom filter, you will need to
use a custom catalog that includes the fields you want to sync. That is you will need to
1. Write the default catalog to a file: `tap-stackexchange --discover > catalog.json`
2. Edit the catalog file to include the fields not included by the default API filter
3. Run the tap with the custom catalog: `tap-stackexchange --config config.json --catalog catalog.json`
## Installation
```bash
pipx install git+https://github.com/edgarrmondragon/tap-stackexchange.git
```
### Source Authentication and Authorization
Register a [new application on Stack Apps](https://stackapps.com/apps/oauth/register) and copy the generated `key`.
## Usage
You can easily run `tap-stackexchange` by itself or in a pipeline using [Meltano](https://meltano.com/).
### Executing the Tap Directly
```bash
tap-stackexchange --version
tap-stackexchange --help
tap-stackexchange --config CONFIG --discover > ./catalog.json
```
### Initialize your Development Environment
```bash
pipx install poetry
poetry install
```
<!--
### Create and Run Tests
Create tests within the `tap_stackexchange/tests` subfolder and
then run:
```bash
poetry run pytest
```
You can also test the `tap-stackexchange` CLI interface directly using `poetry run`:
```bash
poetry run tap-stackexchange --help
```
-->
### Testing with [Meltano](https://www.meltano.com)
_**Note:** This tap will work in any Singer environment and does not require Meltano.
Examples here are for convenience and to streamline end-to-end orchestration scenarios._
Your project comes with a custom `meltano.yml` project file already created. Open the `meltano.yml` and follow any _"TODO"_ items listed in
the file.
Next, install Meltano (if you haven't already) and any needed plugins:
```bash
# Install meltano
pipx install meltano
# Initialize meltano within this directory
cd tap-stackexchange
meltano install
```
Now you can test and orchestrate using Meltano:
```bash
# Test invocation:
meltano invoke tap-stackexchange --version
# OR run a test `elt` pipeline:
meltano elt tap-stackexchange target-sqlite --job_id=stackexchange-sqlite
# Runtime configuration
TAP_STACKEXCHANGE__LOAD_SCHEMA=dragon_ball_gt \
TAP_STACKEXCHANGE_SITE=anime \
TAP_STACKEXCHANGE_TAGS='["dragon-ball-gt"]' \
meltano elt tap-stackexchange target-sqlite
```
### SDK Dev Guide
See the [dev guide](https://sdk.meltano.com/en/latest/dev_guide.html) for more instructions on how to use the SDK to
develop your own taps and targets.
Raw data
{
"_id": null,
"home_page": "https://github.com/MeltanoLabs/tap-stackexchange",
"name": "tap-stackexchange",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.9",
"maintainer_email": null,
"keywords": "ELT, meltano, singer-io, StackExchange",
"author": "Edgar Ram\u00edrez-Mondrag\u00f3n",
"author_email": "edgarrm358@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/37/c8/3585f84442a292cf5647753cc436eb8a704e1175ea7ca183e613b0746d30/tap_stackexchange-0.0.1.tar.gz",
"platform": null,
"description": "# `tap-stackexchange`\n\n[](https://github.com/MeltanoLabs/tap-stackexchange/actions/workflows/test.yml)\n[](https://sonarcloud.io/summary/new_code?id=MeltanoLabs_tap-stackexchange)\n[](https://sonarcloud.io/summary/new_code?id=MeltanoLabs_tap-stackexchange)\n[](https://sonarcloud.io/summary/new_code?id=MeltanoLabs_tap-stackexchange)\n\nSinger tap for the StackExchange API.\n\nBuilt with the [Meltano SDK](https://sdk.meltano.com) for Singer Taps and Targets.\n\n## Capabilities\n\n* `sync`\n* `catalog`\n* `state`\n* `discover`\n\n## Settings\n\n| Setting | Required | Default | Description |\n|:----------|:--------:|:-------:|:------------|\n| key | False | None | Pass this to receive a higher request quota |\n| site | False | stackoverflow.com | StackExchange site |\n| tags | False | None | Question tags |\n| start_date| False | None | The earliest record date to sync |\n\nA full list of supported settings and capabilities is available by running: `tap-stackexchange --about`\n\n## Custom filter\n\nThe StackExchange API supports a number of [custom filters](https://api.stackexchange.com/docs/filters) that can be used to\ninclude or exclude certain fields from the response objects. This application has a baked-in filter with the following\nparameters:\n\n- `include`\n\n | Parameter | Description |\n |:-------------------------|:-----------------------------------------|\n | `question.comment_count` | The number of comments on the question |\n | `tag.last_activity_date` | The date of the last activity on the tag |\n\n- `unsafe=false`\n\n### Update the baked-in filter\n\nTo update the baked-in filter, edit the `FILTER_ID` constant in `tap_stackexchange/tap.py`.\n\nTo generate a new filter, use the _Try It_ button on the [StackExchange API documentation](https://api.stackexchange.com/docs/create-filter), using the baked-in filter as the `base` parameter.\n\n### Use a custom filter\n\nTo use a custom filter, set the `filter` setting to the filter ID. Note that if you use a custom filter, you will need to\nuse a custom catalog that includes the fields you want to sync. That is you will need to\n\n1. Write the default catalog to a file: `tap-stackexchange --discover > catalog.json`\n2. Edit the catalog file to include the fields not included by the default API filter\n3. Run the tap with the custom catalog: `tap-stackexchange --config config.json --catalog catalog.json`\n\n## Installation\n\n```bash\npipx install git+https://github.com/edgarrmondragon/tap-stackexchange.git\n```\n\n### Source Authentication and Authorization\n\nRegister a [new application on Stack Apps](https://stackapps.com/apps/oauth/register) and copy the generated `key`.\n\n## Usage\n\nYou can easily run `tap-stackexchange` by itself or in a pipeline using [Meltano](https://meltano.com/).\n\n### Executing the Tap Directly\n\n```bash\ntap-stackexchange --version\ntap-stackexchange --help\ntap-stackexchange --config CONFIG --discover > ./catalog.json\n```\n\n### Initialize your Development Environment\n\n```bash\npipx install poetry\npoetry install\n```\n\n<!--\n### Create and Run Tests\n\nCreate tests within the `tap_stackexchange/tests` subfolder and\n then run:\n\n```bash\npoetry run pytest\n```\n\nYou can also test the `tap-stackexchange` CLI interface directly using `poetry run`:\n\n```bash\npoetry run tap-stackexchange --help\n```\n-->\n\n### Testing with [Meltano](https://www.meltano.com)\n\n_**Note:** This tap will work in any Singer environment and does not require Meltano.\nExamples here are for convenience and to streamline end-to-end orchestration scenarios._\n\nYour project comes with a custom `meltano.yml` project file already created. Open the `meltano.yml` and follow any _\"TODO\"_ items listed in\nthe file.\n\nNext, install Meltano (if you haven't already) and any needed plugins:\n\n```bash\n# Install meltano\npipx install meltano\n# Initialize meltano within this directory\ncd tap-stackexchange\nmeltano install\n```\n\nNow you can test and orchestrate using Meltano:\n\n```bash\n# Test invocation:\nmeltano invoke tap-stackexchange --version\n\n# OR run a test `elt` pipeline:\nmeltano elt tap-stackexchange target-sqlite --job_id=stackexchange-sqlite\n\n# Runtime configuration\nTAP_STACKEXCHANGE__LOAD_SCHEMA=dragon_ball_gt \\\nTAP_STACKEXCHANGE_SITE=anime \\\nTAP_STACKEXCHANGE_TAGS='[\"dragon-ball-gt\"]' \\\nmeltano elt tap-stackexchange target-sqlite\n```\n\n### SDK Dev Guide\n\nSee the [dev guide](https://sdk.meltano.com/en/latest/dev_guide.html) for more instructions on how to use the SDK to\ndevelop your own taps and targets.\n\n",
"bugtrack_url": null,
"license": "Apache-2.0",
"summary": "Singer tap for StackExchange, built with the Meltano SDK for Singer Taps.",
"version": "0.0.1",
"project_urls": {
"Documentation": "https://github.com/MeltanoLabs/tap-stackexchange/blob/main/README.md",
"Homepage": "https://github.com/MeltanoLabs/tap-stackexchange",
"Repository": "https://github.com/MeltanoLabs/tap-stackexchange"
},
"split_keywords": [
"elt",
" meltano",
" singer-io",
" stackexchange"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "8b4cfb195a8e934b09e86c798e8908967dee99b811a616a6c629b35e73d921ed",
"md5": "62ad0930066b6b6ea5ab739b2287482b",
"sha256": "49888cd48ba8cfb2a9c0d5ce7b430985fab7f65353027e6ab3baa71ceb781cbb"
},
"downloads": -1,
"filename": "tap_stackexchange-0.0.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "62ad0930066b6b6ea5ab739b2287482b",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9",
"size": 13097,
"upload_time": "2024-11-12T02:05:48",
"upload_time_iso_8601": "2024-11-12T02:05:48.878379Z",
"url": "https://files.pythonhosted.org/packages/8b/4c/fb195a8e934b09e86c798e8908967dee99b811a616a6c629b35e73d921ed/tap_stackexchange-0.0.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "37c83585f84442a292cf5647753cc436eb8a704e1175ea7ca183e613b0746d30",
"md5": "a619ad832dc37c72aee7cde29941a7ee",
"sha256": "98cc621ed55f6969f655e3adc5ecf7d61d55a5c2ade86a35351fe661dd778037"
},
"downloads": -1,
"filename": "tap_stackexchange-0.0.1.tar.gz",
"has_sig": false,
"md5_digest": "a619ad832dc37c72aee7cde29941a7ee",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9",
"size": 12882,
"upload_time": "2024-11-12T02:05:50",
"upload_time_iso_8601": "2024-11-12T02:05:50.180330Z",
"url": "https://files.pythonhosted.org/packages/37/c8/3585f84442a292cf5647753cc436eb8a704e1175ea7ca183e613b0746d30/tap_stackexchange-0.0.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-11-12 02:05:50",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "MeltanoLabs",
"github_project": "tap-stackexchange",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"tox": true,
"lcname": "tap-stackexchange"
}
JSON
