| Name | tap-typo JSON |
| Version |
0.2.1
JSON |
| download |
| home_page | https://www.typo.ai/ |
| Summary | Typo is the intelligent data quality barrier for enterprise information systems. The Typo tap retrieves results and data from the Typo platform. |
| upload_time | 2024-01-23 15:52:12 |
| maintainer | |
| docs_url | None |
| author | Typo |
| requires_python | |
| license | |
| keywords |
typo.ai
data
quality
singer
tap
|
| VCS |
|
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# tap-typo
[Singer](https://singer.io) tap that extracts data from the [Typo](https://www.typo.ai?utm_source=github&utm_medium=tap-typo) platform. The tap produces JSON-formatted data output
following the [Singer spec](https://github.com/singer-io/getting-started/blob/master/docs/SPEC.md).
- [Usage](#usage)
- [Installation](#installation)
- [Create a configuration file](#create-a-configuration-file)
- [Discovery mode](#discovery-mode)
- [Sync mode](#sync-mode)
- [Saving state and resuming](#saving-state-and-resuming)
- [Saving state messages](#saving-state-messages)
- [Creating a State file](#creating-a-state-file)
- [Resuming with a State file](#resuming-with-a-state-file)
- [Catalog file](#catalog-file)
- [Typo registration and setup](#typo-registration-and-setup)
- [Development](#development)
- [Support](#support)
## Usage
This section describes the basic usage of **tap-typo** through an example data extraction from a Typo dataset. It assumes that you already have a Typo account, with an existing repository and a dataset. If you do not meet these prerequisites, please go to [Typo Registration and Setup](#typo-registration-and-setup).
### Installation
Python 3 is required. It is recommended to create a separate virtual environment for each tap or target as their may be incompatibilities between dependency versions.
```bash
> pip install tap-typo
```
### Create a configuration file
The config file (usually config.json) is a JSON file describing the tap's settings.
The following sample configuration can be used as a starting point:
```json
{
"api_key": "my_apikey",
"api_secret": "my_apisecret",
"cluster_api_endpoint": "https://cluster.typo.ai/management/api/v1",
"repository": "my_repository",
"dataset": "my_dataset",
"audit_id": "audit_id",
"output_rfc3339_datetime": false
}
```
**Please note: the `dataset` and `audit_id` parameters are optional. When not specified the tap-typo will automatically run in sync mode using the selected datasets from the catalog.**
- **api_key**, **api_secret** and **cluster_api_endpoint** can be obtained by logging into the [Typo Console](https://console.typo.ai/?utm_source=github&utm_medium=tap-typo), clicking on your username, and then on **My Account**.
- **repository** and **dataset** correspond to their respective names and **audit_id** is optional and should be only provided when syncing data from an audit.
- Additionally, a **records_per_page** parameter can be provided to override the number of records requested at once, and a **record_limit** parameter can indicate the maximum number of records that will be obtained when the tap is executed.
### Discovery mode
In discovery mode, **tap-typo** will infer the Singer Catalog from the config file and data in Typo. The output can be redirected to a file in order to be modified and used as input to **tap-typo** (see [Catalog file section](#catalog-file)).
```bash
> tap-typo -c config.json -d > catalog.json
```
### Sync mode
Sync mode will fetch data from Typo and output to stdout. Each record has two additional fields: `__typo_result`, that can have a value of `Error` or `OK` and `__typo_record_id`, which indicates the record's internal ID in Typo. Before starting the sync, unless a custom Catalog file is provided, Typo will run discovery and build the catalog.
```bash
> tap-typo -c config.json
```
### Saving state and resuming
#### Saving state messages
When **tap-typo** runs in Sync mode it will emit one [STATE message](https://github.com/singer-io/getting-started/blob/master/docs/SPEC.md#state-message) for every RECORD message emitted. STATE messages contain a value JSON property with the state information.
A Singer target should output the contents of the value JSON property in a STATE message. By redirecting this target output to a file, the value property of each STATE message will be stored per line.
```bash
> tap-typo -c config.json | target-google-bigquery > state-history.txt
```
#### Creating a State file
To resume from a failed or terminated transfer, you will need create a STATE file from the last line in the redirected output (state-history.txt in our example). Below is an example command that performs the step to create a STATE file, state.json, from state-history.txt. You may edit this STATE file as necessary. The STATE file can be used as input to **tap-typo** to resume.
```bash
tail -n 1 state-history.txt > state.json
```
Example STATE file:
```json
{
"bookmarks": {
"tap-typo-repository-repo1-dataset-dataset1": {
"__typo_record_id": 26
}
}
}
```
#### Resuming with a State file
To resume by providing a State file, **tap-typo** can be started with a -s parameter and providing a path to a STATE file. **tap-typo** searches the bookmarks property for a key that matches the stream name. If found, **tap-typo** will try to resume from the location defined in the bookmark.
```bash
> tap-typo -c config.json -s state.json | target-google-bigquery > state-history.txt
```
### Catalog file
A catalog file can be provided by adding the --catalog parameter with a file path. This will prevent the discovery process and use the catalog provided in the file path.
```bash
> tap-typo -c config.json --catalog catalog.json | target-google-bigquery > state-history.txt
```
## Typo registration and setup
In order to create a Typo account, visit [https://www.typo.ai/signup](https://www.typo.ai/signup?utm_source=github&utm_medium=tap-typo) and follow the instructions.
Once registered you can log in to the Typo Console ([https://console.typo.ai/](https://console.typo.ai/?utm_source=github&utm_medium=tap-typo)) and go to the Repositories section to create a new Repository.
Next, you can start uploading data by using [target-typo](https://github.com/typo-ai/target-typo). A new dataset will be created automatically when data is submitted.
## Development
To work on development of tap-typo, clone the repository, create and activate a new virtual environment, go into the cloned folder and install tap-typo in editable mode.
```bash
git clone https://github.com/typo-ai/tap-typo.git
cd tap-typo
python3 -m venv .venv
source .venv/bin/activate
pip install -e .
```
## Support
You may reach Typo Support at the email address support@ followed by the typo domain or see the full contact information at [https://www.typo.ai](https://www.typo.ai?utm_source=github&utm_medium=tap-typo).
---
Copyright © 2020 Stitch
Raw data
{
"_id": null,
"home_page": "https://www.typo.ai/",
"name": "tap-typo",
"maintainer": "",
"docs_url": null,
"requires_python": "",
"maintainer_email": "",
"keywords": "typo.ai data quality singer tap",
"author": "Typo",
"author_email": "",
"download_url": "https://files.pythonhosted.org/packages/bd/5d/71b760c8e89f63069e7d33bc6642d0d9c52d21affdf68d53d2bb6c43a2fd/tap-typo-0.2.1.tar.gz",
"platform": null,
"description": "# tap-typo\n\n[Singer](https://singer.io) tap that extracts data from the [Typo](https://www.typo.ai?utm_source=github&utm_medium=tap-typo) platform. The tap produces JSON-formatted data output\nfollowing the [Singer spec](https://github.com/singer-io/getting-started/blob/master/docs/SPEC.md).\n\n- [Usage](#usage)\n - [Installation](#installation)\n - [Create a configuration file](#create-a-configuration-file)\n - [Discovery mode](#discovery-mode)\n - [Sync mode](#sync-mode)\n - [Saving state and resuming](#saving-state-and-resuming)\n - [Saving state messages](#saving-state-messages)\n - [Creating a State file](#creating-a-state-file)\n - [Resuming with a State file](#resuming-with-a-state-file)\n - [Catalog file](#catalog-file)\n- [Typo registration and setup](#typo-registration-and-setup)\n- [Development](#development)\n- [Support](#support)\n\n\n\n## Usage\n\nThis section describes the basic usage of **tap-typo** through an example data extraction from a Typo dataset. It assumes that you already have a Typo account, with an existing repository and a dataset. If you do not meet these prerequisites, please go to [Typo Registration and Setup](#typo-registration-and-setup).\n\n\n\n### Installation\n\nPython 3 is required. It is recommended to create a separate virtual environment for each tap or target as their may be incompatibilities between dependency versions.\n\n```bash\n> pip install tap-typo\n```\n\n\n\n### Create a configuration file\n\nThe config file (usually config.json) is a JSON file describing the tap's settings.\n\nThe following sample configuration can be used as a starting point:\n\n\n```json\n{\n \"api_key\": \"my_apikey\",\n \"api_secret\": \"my_apisecret\",\n \"cluster_api_endpoint\": \"https://cluster.typo.ai/management/api/v1\",\n \"repository\": \"my_repository\",\n \"dataset\": \"my_dataset\",\n \"audit_id\": \"audit_id\",\n \"output_rfc3339_datetime\": false\n}\n```\n\n**Please note: the `dataset` and `audit_id` parameters are optional. When not specified the tap-typo will automatically run in sync mode using the selected datasets from the catalog.**\n\n- **api_key**, **api_secret** and **cluster_api_endpoint** can be obtained by logging into the [Typo Console](https://console.typo.ai/?utm_source=github&utm_medium=tap-typo), clicking on your username, and then on **My Account**.\n- **repository** and **dataset** correspond to their respective names and **audit_id** is optional and should be only provided when syncing data from an audit.\n- Additionally, a **records_per_page** parameter can be provided to override the number of records requested at once, and a **record_limit** parameter can indicate the maximum number of records that will be obtained when the tap is executed.\n\n\n\n### Discovery mode\n\nIn discovery mode, **tap-typo** will infer the Singer Catalog from the config file and data in Typo. The output can be redirected to a file in order to be modified and used as input to **tap-typo** (see [Catalog file section](#catalog-file)).\n\n```bash\n> tap-typo -c config.json -d > catalog.json\n```\n\n\n\n### Sync mode\n\nSync mode will fetch data from Typo and output to stdout. Each record has two additional fields: `__typo_result`, that can have a value of `Error` or `OK` and `__typo_record_id`, which indicates the record's internal ID in Typo. Before starting the sync, unless a custom Catalog file is provided, Typo will run discovery and build the catalog.\n\n\n```bash\n> tap-typo -c config.json\n```\n\n\n\n### Saving state and resuming\n\n#### Saving state messages\n\nWhen **tap-typo** runs in Sync mode it will emit one [STATE message](https://github.com/singer-io/getting-started/blob/master/docs/SPEC.md#state-message) for every RECORD message emitted. STATE messages contain a value JSON property with the state information.\n\nA Singer target should output the contents of the value JSON property in a STATE message. By redirecting this target output to a file, the value property of each STATE message will be stored per line.\n\n```bash\n> tap-typo -c config.json | target-google-bigquery > state-history.txt\n```\n\n\n\n#### Creating a State file\n\nTo resume from a failed or terminated transfer, you will need create a STATE file from the last line in the redirected output (state-history.txt in our example). Below is an example command that performs the step to create a STATE file, state.json, from state-history.txt. You may edit this STATE file as necessary. The STATE file can be used as input to **tap-typo** to resume.\n\n```bash\ntail -n 1 state-history.txt > state.json\n```\n\nExample STATE file:\n\n```json\n{\n\t\"bookmarks\": {\n\t\t\"tap-typo-repository-repo1-dataset-dataset1\": {\n\t\t\t\"__typo_record_id\": 26\n\t\t}\n\t}\n}\n```\n\n\n\n#### Resuming with a State file\n\nTo resume by providing a State file, **tap-typo** can be started with a -s parameter and providing a path to a STATE file. **tap-typo** searches the bookmarks property for a key that matches the stream name. If found, **tap-typo** will try to resume from the location defined in the bookmark.\n\n```bash\n> tap-typo -c config.json -s state.json | target-google-bigquery > state-history.txt\n```\n\n\n\n### Catalog file\n\nA catalog file can be provided by adding the --catalog parameter with a file path. This will prevent the discovery process and use the catalog provided in the file path.\n\n```bash\n> tap-typo -c config.json --catalog catalog.json | target-google-bigquery > state-history.txt\n```\n\n\n\n## Typo registration and setup\n\nIn order to create a Typo account, visit [https://www.typo.ai/signup](https://www.typo.ai/signup?utm_source=github&utm_medium=tap-typo) and follow the instructions.\n\nOnce registered you can log in to the Typo Console ([https://console.typo.ai/](https://console.typo.ai/?utm_source=github&utm_medium=tap-typo)) and go to the Repositories section to create a new Repository.\n\nNext, you can start uploading data by using [target-typo](https://github.com/typo-ai/target-typo). A new dataset will be created automatically when data is submitted.\n\n\n\n## Development\n\nTo work on development of tap-typo, clone the repository, create and activate a new virtual environment, go into the cloned folder and install tap-typo in editable mode.\n\n```bash\ngit clone https://github.com/typo-ai/tap-typo.git\ncd tap-typo\npython3 -m venv .venv\nsource .venv/bin/activate\npip install -e .\n```\n\n\n\n## Support\n\nYou may reach Typo Support at the email address support@ followed by the typo domain or see the full contact information at [https://www.typo.ai](https://www.typo.ai?utm_source=github&utm_medium=tap-typo).\n\n\n\n---\n\nCopyright © 2020 Stitch\n",
"bugtrack_url": null,
"license": "",
"summary": "Typo is the intelligent data quality barrier for enterprise information systems. The Typo tap retrieves results and data from the Typo platform.",
"version": "0.2.1",
"project_urls": {
"Homepage": "https://www.typo.ai/"
},
"split_keywords": [
"typo.ai",
"data",
"quality",
"singer",
"tap"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "bd5d71b760c8e89f63069e7d33bc6642d0d9c52d21affdf68d53d2bb6c43a2fd",
"md5": "6b8cfe30e5224d38c861565c8863dc5e",
"sha256": "cced5604b113582e72f1b1b16c80b26d0b202ee57b37b200881c2ece4e9ac09e"
},
"downloads": -1,
"filename": "tap-typo-0.2.1.tar.gz",
"has_sig": false,
"md5_digest": "6b8cfe30e5224d38c861565c8863dc5e",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 28214,
"upload_time": "2024-01-23T15:52:12",
"upload_time_iso_8601": "2024-01-23T15:52:12.791063Z",
"url": "https://files.pythonhosted.org/packages/bd/5d/71b760c8e89f63069e7d33bc6642d0d9c52d21affdf68d53d2bb6c43a2fd/tap-typo-0.2.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-01-23 15:52:12",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "tap-typo"
}
