# `tap-stackexchange`
[](https://github.com/MeltanoLabs/tap-stackexchange/actions/workflows/test.yml)
[](https://sonarcloud.io/summary/new_code?id=MeltanoLabs_tap-stackexchange)
[](https://sonarcloud.io/summary/new_code?id=MeltanoLabs_tap-stackexchange)
[](https://sonarcloud.io/summary/new_code?id=MeltanoLabs_tap-stackexchange)
Singer tap for the StackExchange API.
Built with the [Meltano SDK](https://sdk.meltano.com) for Singer Taps and Targets.
## Capabilities
* `sync`
* `catalog`
* `state`
* `discover`
## Settings
| Setting | Required | Default | Description |
|:----------|:--------:|:-------:|:------------|
| key | False | None | Pass this to receive a higher request quota |
| site | False | stackoverflow.com | StackExchange site |
| tags | False | None | Question tags |
| start_date| False | None | The earliest record date to sync |
A full list of supported settings and capabilities is available by running: `tap-stackexchange --about`
## Custom filter
The StackExchange API supports a number of [custom filters](https://api.stackexchange.com/docs/filters) that can be used to
include or exclude certain fields from the response objects. This application has a baked-in filter with the following
parameters:
- `include`
| Parameter | Description |
|:-------------------------|:-----------------------------------------|
| `question.comment_count` | The number of comments on the question |
| `tag.last_activity_date` | The date of the last activity on the tag |
- `unsafe=false`
### Update the baked-in filter
To update the baked-in filter, edit the `FILTER_ID` constant in `tap_stackexchange/tap.py`.
To generate a new filter, use the _Try It_ button on the [StackExchange API documentation](https://api.stackexchange.com/docs/create-filter), using the baked-in filter as the `base` parameter.
### Use a custom filter
To use a custom filter, set the `filter` setting to the filter ID. Note that if you use a custom filter, you will need to
use a custom catalog that includes the fields you want to sync. That is you will need to
1. Write the default catalog to a file: `tap-stackexchange --discover > catalog.json`
2. Edit the catalog file to include the fields not included by the default API filter
3. Run the tap with the custom catalog: `tap-stackexchange --config config.json --catalog catalog.json`
## Installation
```bash
pipx install git+https://github.com/edgarrmondragon/tap-stackexchange.git
```
### Source Authentication and Authorization
Register a [new application on Stack Apps](https://stackapps.com/apps/oauth/register) and copy the generated `key`.
## Usage
You can easily run `tap-stackexchange` by itself or in a pipeline using [Meltano](https://meltano.com/).
### Executing the Tap Directly
```bash
tap-stackexchange --version
tap-stackexchange --help
tap-stackexchange --config CONFIG --discover > ./catalog.json
```
### Initialize your Development Environment
```bash
pipx install poetry
poetry install
```
<!--
### Create and Run Tests
Create tests within the `tap_stackexchange/tests` subfolder and
then run:
```bash
poetry run pytest
```
You can also test the `tap-stackexchange` CLI interface directly using `poetry run`:
```bash
poetry run tap-stackexchange --help
```
-->
### Testing with [Meltano](https://www.meltano.com)
_**Note:** This tap will work in any Singer environment and does not require Meltano.
Examples here are for convenience and to streamline end-to-end orchestration scenarios._
Your project comes with a custom `meltano.yml` project file already created. Open the `meltano.yml` and follow any _"TODO"_ items listed in
the file.
Next, install Meltano (if you haven't already) and any needed plugins:
```bash
# Install meltano
pipx install meltano
# Initialize meltano within this directory
cd tap-stackexchange
meltano install
```
Now you can test and orchestrate using Meltano:
```bash
# Test invocation:
meltano invoke tap-stackexchange --version
# OR run a test `elt` pipeline:
meltano elt tap-stackexchange target-sqlite --job_id=stackexchange-sqlite
# Runtime configuration
TAP_STACKEXCHANGE__LOAD_SCHEMA=dragon_ball_gt \
TAP_STACKEXCHANGE_SITE=anime \
TAP_STACKEXCHANGE_TAGS='["dragon-ball-gt"]' \
meltano elt tap-stackexchange target-sqlite
```
### SDK Dev Guide
See the [dev guide](https://sdk.meltano.com/en/latest/dev_guide.html) for more instructions on how to use the SDK to
develop your own taps and targets.
Raw data
{
"_id": null,
"home_page": "",
"name": "tap-stackexchange",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": "",
"keywords": "ELT,meltano,singer-io,StackExchange",
"author": "Edgar Ram\u00edrez-Mondrag\u00f3n",
"author_email": "edgarrm358@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/d0/61/3031da741bea564496ca083c878e856466f7006a12e8ae4079bf3cece467/tap_stackexchange-0.0.1a3.tar.gz",
"platform": null,
"description": "# `tap-stackexchange`\n\n[](https://github.com/MeltanoLabs/tap-stackexchange/actions/workflows/test.yml)\n[](https://sonarcloud.io/summary/new_code?id=MeltanoLabs_tap-stackexchange)\n[](https://sonarcloud.io/summary/new_code?id=MeltanoLabs_tap-stackexchange)\n[](https://sonarcloud.io/summary/new_code?id=MeltanoLabs_tap-stackexchange)\n\nSinger tap for the StackExchange API.\n\nBuilt with the [Meltano SDK](https://sdk.meltano.com) for Singer Taps and Targets.\n\n## Capabilities\n\n* `sync`\n* `catalog`\n* `state`\n* `discover`\n\n## Settings\n\n| Setting | Required | Default | Description |\n|:----------|:--------:|:-------:|:------------|\n| key | False | None | Pass this to receive a higher request quota |\n| site | False | stackoverflow.com | StackExchange site |\n| tags | False | None | Question tags |\n| start_date| False | None | The earliest record date to sync |\n\nA full list of supported settings and capabilities is available by running: `tap-stackexchange --about`\n\n## Custom filter\n\nThe StackExchange API supports a number of [custom filters](https://api.stackexchange.com/docs/filters) that can be used to\ninclude or exclude certain fields from the response objects. This application has a baked-in filter with the following\nparameters:\n\n- `include`\n\n | Parameter | Description |\n |:-------------------------|:-----------------------------------------|\n | `question.comment_count` | The number of comments on the question |\n | `tag.last_activity_date` | The date of the last activity on the tag |\n\n- `unsafe=false`\n\n### Update the baked-in filter\n\nTo update the baked-in filter, edit the `FILTER_ID` constant in `tap_stackexchange/tap.py`.\n\nTo generate a new filter, use the _Try It_ button on the [StackExchange API documentation](https://api.stackexchange.com/docs/create-filter), using the baked-in filter as the `base` parameter.\n\n### Use a custom filter\n\nTo use a custom filter, set the `filter` setting to the filter ID. Note that if you use a custom filter, you will need to\nuse a custom catalog that includes the fields you want to sync. That is you will need to\n\n1. Write the default catalog to a file: `tap-stackexchange --discover > catalog.json`\n2. Edit the catalog file to include the fields not included by the default API filter\n3. Run the tap with the custom catalog: `tap-stackexchange --config config.json --catalog catalog.json`\n\n## Installation\n\n```bash\npipx install git+https://github.com/edgarrmondragon/tap-stackexchange.git\n```\n\n### Source Authentication and Authorization\n\nRegister a [new application on Stack Apps](https://stackapps.com/apps/oauth/register) and copy the generated `key`.\n\n## Usage\n\nYou can easily run `tap-stackexchange` by itself or in a pipeline using [Meltano](https://meltano.com/).\n\n### Executing the Tap Directly\n\n```bash\ntap-stackexchange --version\ntap-stackexchange --help\ntap-stackexchange --config CONFIG --discover > ./catalog.json\n```\n\n### Initialize your Development Environment\n\n```bash\npipx install poetry\npoetry install\n```\n\n<!--\n### Create and Run Tests\n\nCreate tests within the `tap_stackexchange/tests` subfolder and\n then run:\n\n```bash\npoetry run pytest\n```\n\nYou can also test the `tap-stackexchange` CLI interface directly using `poetry run`:\n\n```bash\npoetry run tap-stackexchange --help\n```\n-->\n\n### Testing with [Meltano](https://www.meltano.com)\n\n_**Note:** This tap will work in any Singer environment and does not require Meltano.\nExamples here are for convenience and to streamline end-to-end orchestration scenarios._\n\nYour project comes with a custom `meltano.yml` project file already created. Open the `meltano.yml` and follow any _\"TODO\"_ items listed in\nthe file.\n\nNext, install Meltano (if you haven't already) and any needed plugins:\n\n```bash\n# Install meltano\npipx install meltano\n# Initialize meltano within this directory\ncd tap-stackexchange\nmeltano install\n```\n\nNow you can test and orchestrate using Meltano:\n\n```bash\n# Test invocation:\nmeltano invoke tap-stackexchange --version\n\n# OR run a test `elt` pipeline:\nmeltano elt tap-stackexchange target-sqlite --job_id=stackexchange-sqlite\n\n# Runtime configuration\nTAP_STACKEXCHANGE__LOAD_SCHEMA=dragon_ball_gt \\\nTAP_STACKEXCHANGE_SITE=anime \\\nTAP_STACKEXCHANGE_TAGS='[\"dragon-ball-gt\"]' \\\nmeltano elt tap-stackexchange target-sqlite\n```\n\n### SDK Dev Guide\n\nSee the [dev guide](https://sdk.meltano.com/en/latest/dev_guide.html) for more instructions on how to use the SDK to\ndevelop your own taps and targets.\n\n",
"bugtrack_url": null,
"license": "Apache-2.0",
"summary": "`tap-stackexchange` is a Singer tap for StackExchange, built with the Meltano SDK for Singer Taps.",
"version": "0.0.1a3",
"project_urls": null,
"split_keywords": [
"elt",
"meltano",
"singer-io",
"stackexchange"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "e83b29cd6947e7c4f9dfee55842a37c1cd282838f289044b31c9ca826d169c92",
"md5": "ca5ed94ec2858f0d7a9dcaa1d76dc56a",
"sha256": "2d03d950ef5560f504a128db40547bffa3c23ce3d269d10b5f28524c2e69813d"
},
"downloads": -1,
"filename": "tap_stackexchange-0.0.1a3-py3-none-any.whl",
"has_sig": false,
"md5_digest": "ca5ed94ec2858f0d7a9dcaa1d76dc56a",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 13041,
"upload_time": "2024-03-05T15:21:16",
"upload_time_iso_8601": "2024-03-05T15:21:16.282850Z",
"url": "https://files.pythonhosted.org/packages/e8/3b/29cd6947e7c4f9dfee55842a37c1cd282838f289044b31c9ca826d169c92/tap_stackexchange-0.0.1a3-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "d0613031da741bea564496ca083c878e856466f7006a12e8ae4079bf3cece467",
"md5": "89fe8e84a1b5f50d736d644f1a98f357",
"sha256": "1d93f2a19f7b622426e1643a2cab070c69d8d5afe7202592d52d995c3a7e3268"
},
"downloads": -1,
"filename": "tap_stackexchange-0.0.1a3.tar.gz",
"has_sig": false,
"md5_digest": "89fe8e84a1b5f50d736d644f1a98f357",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 12782,
"upload_time": "2024-03-05T15:21:17",
"upload_time_iso_8601": "2024-03-05T15:21:17.867475Z",
"url": "https://files.pythonhosted.org/packages/d0/61/3031da741bea564496ca083c878e856466f7006a12e8ae4079bf3cece467/tap_stackexchange-0.0.1a3.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-03-05 15:21:17",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "tap-stackexchange"
}
JSON