realit-singer-encodings
===================
[](https://badge.fury.io/py/realit-singer-encodings)
[](https://pypi.org/project/realit-singer-encodings/)
[](https://opensource.org/licenses/Apache-2.0)
Writes the Singer format from Python.
This is a fork of [Singer's singer-encodings](https://github.com/singer-io/singer-encodings) made for [PipelineWise](https://transferwise.github.io/pipelinewise) and [Meltano](https://meltano.com/).
Usage
---
### Setup environment
This library depends on python3. We recommend using a `virtualenv` like this:
```bash
python3 -m venv ~/.virtualenvs/realit-singer-encodings
```
### Installation
Next, install this library:
```bash
source ~/.virtualenvs/realit-singer-encodings/bin/activate
git clone http://github.com/s7clarke10/realit-singer-encodings
cd realit-singer-encodings
pip install .
```
### Usage example
This package is used by tap-s3-csv. One of the key configurable items is the `table` field.
The `table` field consists of one or more objects, that describe how to find files and emit records. A more detailed (and unescaped) example below:
```
[
{
"search_prefix": "exports"
"search_pattern": "my_table\\/.*\\.csv",
"table_name": "my_table",
"key_properties": ["id"],
"date_overrides": ["created_at"],
"delimiter": ",",
"guess_types": true,
"string_overrides": ["field_1","field_2"],
"remove_character": "\""
},
...
]
```
The following specific settings influence the behaviour of the singer-encodings package.
- **string_overrides**: Specifies field names in the files that should be parsed as a string regardless of what was discovered.
- **guess_types**: (default `true`) By default, column data types will be determined via scanning the first file in a table_spec. Set this to `false` to disable this and set all columns to `string`.
- **remove_character**: Specifies a character which can be removed from each line in the the file e.g. `"\""` will remove all double-quotes.
- **encoding**: The encoding to use to read these files from [codecs -> Standard Encodings](https://docs.python.org/3/library/codecs.html#standard-encodings)
## Developer Resources
### Initialize your Development Environment
```bash
pip install poetry
poetry install
```
### Create and Run Tests
Create tests within the `tests/` directory and
then run:
```bash
poetry run pytest
```
or
```bash
poetry run coverage run --parallel -m pytest
```
### Continuous Integration
Run through the full suite of tests and linters by running
```bash
poetry run tox
```
These must pass in order for PR's to be merged.
License
-------
Distributed under the Apache License Version 2.0
Raw data
{
"_id": null,
"home_page": "https://github.com/s7clarke10/realit-singer-encodings",
"name": "realit-singer-encodings",
"maintainer": null,
"docs_url": null,
"requires_python": "<3.13,>=3.8",
"maintainer_email": null,
"keywords": "singer, meltano, pipelinewise, framework",
"author": null,
"author_email": null,
"download_url": "https://files.pythonhosted.org/packages/9f/b8/3dae8dc8715619846e7dd233a940da4bf4ccc1eab04c427c07caba461593/realit_singer_encodings-2.1.1.tar.gz",
"platform": null,
"description": "realit-singer-encodings\n===================\n[](https://badge.fury.io/py/realit-singer-encodings)\n[](https://pypi.org/project/realit-singer-encodings/)\n[](https://opensource.org/licenses/Apache-2.0)\n\nWrites the Singer format from Python.\n\nThis is a fork of [Singer's singer-encodings](https://github.com/singer-io/singer-encodings) made for [PipelineWise](https://transferwise.github.io/pipelinewise) and [Meltano](https://meltano.com/).\n\nUsage\n---\n\n### Setup environment\nThis library depends on python3. We recommend using a `virtualenv` like this:\n\n```bash\npython3 -m venv ~/.virtualenvs/realit-singer-encodings\n```\n\n### Installation\nNext, install this library:\n\n```bash\nsource ~/.virtualenvs/realit-singer-encodings/bin/activate\ngit clone http://github.com/s7clarke10/realit-singer-encodings\ncd realit-singer-encodings\npip install .\n```\n\n### Usage example\n\nThis package is used by tap-s3-csv. One of the key configurable items is the `table` field.\nThe `table` field consists of one or more objects, that describe how to find files and emit records. A more detailed (and unescaped) example below:\n\n```\n[\n {\n \"search_prefix\": \"exports\"\n \"search_pattern\": \"my_table\\\\/.*\\\\.csv\",\n \"table_name\": \"my_table\",\n \"key_properties\": [\"id\"],\n \"date_overrides\": [\"created_at\"],\n \"delimiter\": \",\",\n \"guess_types\": true,\n \"string_overrides\": [\"field_1\",\"field_2\"],\n \"remove_character\": \"\\\"\"\n },\n ...\n]\n```\n\nThe following specific settings influence the behaviour of the singer-encodings package.\n\n- **string_overrides**: Specifies field names in the files that should be parsed as a string regardless of what was discovered.\n- **guess_types**: (default `true`) By default, column data types will be determined via scanning the first file in a table_spec. Set this to `false` to disable this and set all columns to `string`.\n- **remove_character**: Specifies a character which can be removed from each line in the the file e.g. `\"\\\"\"` will remove all double-quotes.\n- **encoding**: The encoding to use to read these files from [codecs -> Standard Encodings](https://docs.python.org/3/library/codecs.html#standard-encodings)\n\n## Developer Resources\n\n### Initialize your Development Environment\n\n```bash\npip install poetry\npoetry install\n```\n\n### Create and Run Tests\n\nCreate tests within the `tests/` directory and\nthen run:\n\n```bash\npoetry run pytest\n```\n\nor \n\n```bash\npoetry run coverage run --parallel -m pytest\n```\n\n### Continuous Integration\nRun through the full suite of tests and linters by running\n\n```bash\npoetry run tox\n```\n\nThese must pass in order for PR's to be merged.\n\nLicense\n-------\n\nDistributed under the Apache License Version 2.0\n\n",
"bugtrack_url": null,
"license": "Apache 2.0",
"summary": "Singer.io encodings library - PipelineWise and Meltano compatible",
"version": "2.1.1",
"project_urls": {
"Homepage": "https://github.com/s7clarke10/realit-singer-encodings",
"Repository": "https://github.com/s7clarke10/realit-singer-encodings"
},
"split_keywords": [
"singer",
" meltano",
" pipelinewise",
" framework"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "c2040d9a50de0a9213d08cf3c46c62421a42434faa2ee6e11205ecab33b6fe6d",
"md5": "2f46e93e45428d6227252ba2df697113",
"sha256": "eb813537b4660a59de0e3f9ad8086759c72e1a8bd4e69cafef8289f510e90e44"
},
"downloads": -1,
"filename": "realit_singer_encodings-2.1.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "2f46e93e45428d6227252ba2df697113",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<3.13,>=3.8",
"size": 12896,
"upload_time": "2024-08-18T23:59:13",
"upload_time_iso_8601": "2024-08-18T23:59:13.267143Z",
"url": "https://files.pythonhosted.org/packages/c2/04/0d9a50de0a9213d08cf3c46c62421a42434faa2ee6e11205ecab33b6fe6d/realit_singer_encodings-2.1.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "9fb83dae8dc8715619846e7dd233a940da4bf4ccc1eab04c427c07caba461593",
"md5": "936d1cb6f8cd9a9d07a10f7434ed0efc",
"sha256": "c564f2403e2bc74c7df83f805faa6cb508ca620b41f787c62c6f0473e313974c"
},
"downloads": -1,
"filename": "realit_singer_encodings-2.1.1.tar.gz",
"has_sig": false,
"md5_digest": "936d1cb6f8cd9a9d07a10f7434ed0efc",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<3.13,>=3.8",
"size": 12036,
"upload_time": "2024-08-18T23:59:14",
"upload_time_iso_8601": "2024-08-18T23:59:14.632702Z",
"url": "https://files.pythonhosted.org/packages/9f/b8/3dae8dc8715619846e7dd233a940da4bf4ccc1eab04c427c07caba461593/realit_singer_encodings-2.1.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-08-18 23:59:14",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "s7clarke10",
"github_project": "realit-singer-encodings",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"circle": true,
"tox": true,
"lcname": "realit-singer-encodings"
}
JSON
