# target-miso
This package is a [Singer](https://singer.io) target that sends data to Miso's [Data API](https://api.askmiso.com). Being a singer target, you can integrate it into your data pipeline using your favorite DataOps framework, for instance, [Meltano](https://meltano.com/).
<p>
<a href="https://pypi.org/project/target-miso/"><img alt="PyPI" src="https://img.shields.io/pypi/v/target-miso"></a>
<img alt="PyPI - Python Version: 3" src="https://img.shields.io/pypi/pyversions/target-miso">
<a href="https://github.com/MisoAI/target-miso/tree/main/LICENSE"><img alt="PyPI - License: MIT" src="https://img.shields.io/pypi/l/target-miso"></a>
</p>
# Use with Meltano
Follow the steps below to setup `target-miso` in your Meltano project. Or see the most essential project example [here](https://github.com/MisoAI/target-miso/tree/main/examples/csv).
## Setup
Setup `meltano.yml` like this:
```yml
# ...
plugins:
# ...
loaders:
- name: target-miso
namespace: target_miso
pip_url: target-miso
executable: target-miso
config:
template_folder: template
api_key: your_miso_api_key
```
Make sure to run `meltano install` to install the dependency.
## Configuration
The config object accepts the following properties:
| name | required | default | explanation |
| --- | --- | --- | --- |
| api_server | no | https://api.askmiso.com | The Miso API server host. |
| api_key | yes | | Your Miso API key. |
| template_folder | yes | | Where you keep the template files. The path is relative to Meltano project directory. |
| use_async | no | False | Whether to send request in asynchronous mode. |
| dry_run | no | False | Whether to send request in dry-run mode. |
| write_record_limit | no | 100 | The maximum number of records to be written. |
## Replication methods
Currently, this target supports `FULL_TABLE` and `INCREMENTAL` replication methods. `LOG_BASED` is not yet supported.
## Template
To specify how records are transformed into payloads of Miso API, for each stream from the tap, put a corresponding [jinja2](https://jinja.palletsprojects.com/en/3.1.x/) template file in your template folder. For example, given a stream `product`, put a template file `product.jinja` like this:
```nunjucks
{
"product_id": "{{ data.uuid }}",
"created_at": "{{ data.created_at | datetime_format }}",
"title": "{{ data.title }}",
"description": "{{ data.description }}",
"categories": "{{ data.category | convert_categories }}",
"custom_attributes": {
"vender": "{{ data.vender if data.vender }}"
}
}
```
### Rules on output data types
Miso takes 3 kinds of data records: `user`, `product`, and `interaction`. A record is classified into one of these type by the following rules:
* If the payload contains the `type` field, it is an interaction record.
* If the payload contains the `user_id` field, it is a user record.
* If the payload contains the `product_id` field, it is a product record.
### Built-in filters
Target Miso comes with a few built-in filters that can be used in template expressions:
#### `datetime_format`
Takes a string in any format compatible with [dateparser](https://dateparser.readthedocs.io/en/latest/) and output in ISO format, which is desired by Miso API.
#### `list_of_str`
Wrap a string to a singleton list of string. For example, `"apple"` to `["apple"]`.
#### `convert_categories`
Wrap a string to a singleton double-layered list of string. For example, `"apple"` to `[["apple"]]`.
#### `remove_symbol`
1. Convert an int to string.
1. Strip off some special characters from input string, including: `"`, `\`, `\\N`, `“`, `\r\n`, `\n`, `\r`.
#### `split`
Split a string into a list by comma.
#### `fix_url`
Encode (as URL component) the path component of a URL string.
#### `jsonify`
Serialize an obj to a JSON string.
----
Copyright © 2021 Miso Corp.
Raw data
{
"_id": null,
"home_page": "https://github.com/MisoAI/target-miso",
"name": "target-miso",
"maintainer": "",
"docs_url": null,
"requires_python": "",
"maintainer_email": "",
"keywords": "singer,singer.io,target,ETL,ELT,MisoAI,Miso,Meltano",
"author": "Hash Lin",
"author_email": "hashlin@askmiso.com",
"download_url": "https://files.pythonhosted.org/packages/14/71/e9954b0db020a23cb853262c0210fbe422c33bdfa35ccab378be98cf4641/target-miso-0.9.5.tar.gz",
"platform": null,
"description": "# target-miso\nThis package is a [Singer](https://singer.io) target that sends data to Miso's [Data API](https://api.askmiso.com). Being a singer target, you can integrate it into your data pipeline using your favorite DataOps framework, for instance, [Meltano](https://meltano.com/).\n\n<p>\n <a href=\"https://pypi.org/project/target-miso/\"><img alt=\"PyPI\" src=\"https://img.shields.io/pypi/v/target-miso\"></a>\n <img alt=\"PyPI - Python Version: 3\" src=\"https://img.shields.io/pypi/pyversions/target-miso\">\n <a href=\"https://github.com/MisoAI/target-miso/tree/main/LICENSE\"><img alt=\"PyPI - License: MIT\" src=\"https://img.shields.io/pypi/l/target-miso\"></a>\n</p>\n\n# Use with Meltano\n\nFollow the steps below to setup `target-miso` in your Meltano project. Or see the most essential project example [here](https://github.com/MisoAI/target-miso/tree/main/examples/csv).\n\n## Setup\nSetup `meltano.yml` like this:\n\n```yml\n# ...\nplugins:\n # ...\n loaders:\n - name: target-miso\n namespace: target_miso\n pip_url: target-miso\n executable: target-miso\n config:\n template_folder: template\n api_key: your_miso_api_key\n```\n\nMake sure to run `meltano install` to install the dependency.\n\n## Configuration\n\nThe config object accepts the following properties:\n\n| name | required | default | explanation |\n| --- | --- | --- | --- |\n| api_server | no | https://api.askmiso.com | The Miso API server host. |\n| api_key | yes | | Your Miso API key. |\n| template_folder | yes | | Where you keep the template files. The path is relative to Meltano project directory. |\n| use_async | no | False | Whether to send request in asynchronous mode. |\n| dry_run | no | False | Whether to send request in dry-run mode. |\n| write_record_limit | no | 100 | The maximum number of records to be written. |\n\n## Replication methods\n\nCurrently, this target supports `FULL_TABLE` and `INCREMENTAL` replication methods. `LOG_BASED` is not yet supported.\n\n## Template\n\nTo specify how records are transformed into payloads of Miso API, for each stream from the tap, put a corresponding [jinja2](https://jinja.palletsprojects.com/en/3.1.x/) template file in your template folder. For example, given a stream `product`, put a template file `product.jinja` like this:\n\n```nunjucks\n{\n \"product_id\": \"{{ data.uuid }}\",\n \"created_at\": \"{{ data.created_at | datetime_format }}\",\n \"title\": \"{{ data.title }}\",\n \"description\": \"{{ data.description }}\",\n \"categories\": \"{{ data.category | convert_categories }}\",\n \"custom_attributes\": {\n \"vender\": \"{{ data.vender if data.vender }}\"\n }\n}\n```\n\n### Rules on output data types\n\nMiso takes 3 kinds of data records: `user`, `product`, and `interaction`. A record is classified into one of these type by the following rules:\n* If the payload contains the `type` field, it is an interaction record.\n* If the payload contains the `user_id` field, it is a user record.\n* If the payload contains the `product_id` field, it is a product record.\n\n### Built-in filters\n\nTarget Miso comes with a few built-in filters that can be used in template expressions:\n\n#### `datetime_format`\n\nTakes a string in any format compatible with [dateparser](https://dateparser.readthedocs.io/en/latest/) and output in ISO format, which is desired by Miso API.\n\n#### `list_of_str`\n\nWrap a string to a singleton list of string. For example, `\"apple\"` to `[\"apple\"]`.\n\n#### `convert_categories`\n\nWrap a string to a singleton double-layered list of string. For example, `\"apple\"` to `[[\"apple\"]]`.\n\n#### `remove_symbol`\n\n1. Convert an int to string.\n1. Strip off some special characters from input string, including: `\"`, `\\`, `\\\\N`, `\u201c`, `\\r\\n`, `\\n`, `\\r`.\n\n#### `split`\n\nSplit a string into a list by comma.\n\n#### `fix_url`\n\nEncode (as URL component) the path component of a URL string.\n\n#### `jsonify`\n\nSerialize an obj to a JSON string.\n\n----\n\nCopyright © 2021 Miso Corp.\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "A Singer target for writing data to Miso API",
"version": "0.9.5",
"project_urls": {
"Homepage": "https://github.com/MisoAI/target-miso"
},
"split_keywords": [
"singer",
"singer.io",
"target",
"etl",
"elt",
"misoai",
"miso",
"meltano"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "dff59600691d54e65553d6531e0a883223e87d3d7444b3d0fad26f3016d405cf",
"md5": "50f87eb814d2446e24c8d8e4e14dce50",
"sha256": "cbb083b4364740c4e6aa92dd7900e6bac2dd2115bfb576719723ba6cdee91020"
},
"downloads": -1,
"filename": "target_miso-0.9.5-py3-none-any.whl",
"has_sig": false,
"md5_digest": "50f87eb814d2446e24c8d8e4e14dce50",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 10051,
"upload_time": "2023-07-13T15:04:54",
"upload_time_iso_8601": "2023-07-13T15:04:54.057348Z",
"url": "https://files.pythonhosted.org/packages/df/f5/9600691d54e65553d6531e0a883223e87d3d7444b3d0fad26f3016d405cf/target_miso-0.9.5-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "1471e9954b0db020a23cb853262c0210fbe422c33bdfa35ccab378be98cf4641",
"md5": "e31cd78eac3cdd1e5101f28f362304fa",
"sha256": "ad68fcf1f7cb904cd2755180c29ed156fe635531b72e32c69119a76d9f00e6b6"
},
"downloads": -1,
"filename": "target-miso-0.9.5.tar.gz",
"has_sig": false,
"md5_digest": "e31cd78eac3cdd1e5101f28f362304fa",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 9663,
"upload_time": "2023-07-13T15:04:56",
"upload_time_iso_8601": "2023-07-13T15:04:56.264021Z",
"url": "https://files.pythonhosted.org/packages/14/71/e9954b0db020a23cb853262c0210fbe422c33bdfa35ccab378be98cf4641/target-miso-0.9.5.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-07-13 15:04:56",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "MisoAI",
"github_project": "target-miso",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "target-miso"
}
JSON
