| Name | fs2elastic JSON |
| Version |
0.1.1
JSON |
| download |
| home_page | None |
| Summary | Python based CLI for Syncing File System based Data (CSV file etc.) to Elasticsearch. |
| upload_time | 2024-03-23 18:21:34 |
| maintainer | None |
| docs_url | None |
| author | None |
| requires_python | >=3.10 |
| license | MIT License Copyright (c) 2024 Pankaj Jackson Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. |
| keywords |
csv
sync
elasticsearch
data
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# FS2Elastic
Python based CLI for Syncing File System based Data (CSV file etc.) to Elasticsearch.
### Features
- Real time csv file to Elasticsearch dataset syncing
- Configurable with custom config file
### Installation
```bash
pip install fs2elastic
```
### Configuration
configuration file present at `~/.fs2elastic/fs2elastic.conf`.
- please create configuration file if not present.
- replace values with correct values
- remove configs that does not require
#### default configuration:
```bash
[AppConfig]
app_home = "/home/john/.fs2elastic"
app_config_file_path = "/home/john/.fs2elastic/fs2elastic.conf"
[DatasetConfig]
dataset_source_dir = "/home/john/csv_data_set"
dataset_supported_file_extensions = [ "csv", "xlsx", "xls", "json"]
dataset_max_workers = 1
dataset_threads_per_worker = 10
dataset_chunk_size = 200
[ESConfig]
es_hosts = [ "https://localhost:9200",]
es_username = "elastic"
es_password = ""
es_timeout = 300
es_index_prefix = "fs2es-"
es_ssl_ca = ""
es_verify_certs = false
[LogConfig]
log_file_path = "/home/john/.fs2elastic/fs2elastic.log"
log_max_size = 10485760
log_backup_count = 5
```
### Usage
#### with Default config file
```bash
fs2elastic
```
#### with Custom config file
```bash
fs2elastic -c <config file path>
eg: fs2elastic -c ~/Documents/fs2elastic_custom_config.conf
```
##### Help
```bash
fs2elastic -h
```
##### Version
```bash
fs2elastic -v
```
#### Use as a system service
`NOTE: Required sudo permission`
```bash
# install Package
sudo pip install fs2elastic
```
```bash
# Create fs2elastic home directory somewhere. for eg /tmp/fs2elastic/
mkdir /tmp/fs2elastic/
```
```bash
# create config file at /tmp/fs2elastic/fs2elastic.conf inside fs2elastic home directory created above
[AppConfig]
app_home = "/home/john/.fs2elastic"
app_config_file_path = "/home/john/.fs2elastic/fs2elastic.conf"
[DatasetConfig]
dataset_source_dir = "/home/john/csv_data_set"
dataset_supported_file_extensions = [ "csv", "xlsx", "xls", "json"]
dataset_max_workers = 1
dataset_threads_per_worker = 10
dataset_chunk_size = 200
[ESConfig]
es_hosts = [ "https://localhost:9200",]
es_username = "elastic"
es_password = ""
es_timeout = 300
es_index_prefix = "fs2es-"
es_ssl_ca = ""
es_verify_certs = false
[LogConfig]
log_file_path = "/home/john/.fs2elastic/fs2elastic.log"
log_max_size = 10485760
log_backup_count = 5
# replace with correct values
```
```bash
# create service file at /etc/systemd/system/fs2elastic.service with following content
[Unit]
Description=FS2Elastic Service
After=network.target
[Service]
User=root
Group=root
ExecStart=/usr/bin/fs2elastic -c /tmp/fs2elastic/fs2elastic.conf
Restart=always
[Install]
WantedBy=multi-user.target
```
```bash
# Reloading Change to Systemd Unit Files
sudo systemctl daemon-reload
```
```bash
# Start service
sudo systemctl start fs2elastic
```
```bash
# Enable service to start on boot
sudo systemctl enable fs2elastic
```
```bash
# check status
sudo systemctl status fs2elastic
```
```bash
# check logs
sudo tail -f /var/log/fs2elastic.log
```
```bash
# To stop service
sudo systemctl stop fs2elastic
```
```bash
# To remove service
sudo systemctl disable fs2elastic
sudo rm /etc/systemd/system/fs2elastic.service
```
### Uninstall
```bash
pip uninstall fs2elastic
```
### Who do I talk to?
- Repo owner or admin
- Other community or team contact
Raw data
{
"_id": null,
"home_page": null,
"name": "fs2elastic",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.10",
"maintainer_email": null,
"keywords": "csv, sync, elasticsearch, data",
"author": null,
"author_email": "Pankaj Jackson <pankajackson@live.co.uk>",
"download_url": "https://files.pythonhosted.org/packages/b8/cc/ac867a1b88a838ee3295ef298d36ca64a210d86e8e3973428578881db2fb/fs2elastic-0.1.1.tar.gz",
"platform": null,
"description": "# FS2Elastic\n\nPython based CLI for Syncing File System based Data (CSV file etc.) to Elasticsearch.\n\n\n\n### Features\n\n- Real time csv file to Elasticsearch dataset syncing\n- Configurable with custom config file\n\n### Installation\n\n```bash\npip install fs2elastic\n```\n\n### Configuration\n\nconfiguration file present at `~/.fs2elastic/fs2elastic.conf`.\n\n- please create configuration file if not present.\n- replace values with correct values\n- remove configs that does not require\n\n#### default configuration:\n\n```bash\n[AppConfig]\napp_home = \"/home/john/.fs2elastic\"\napp_config_file_path = \"/home/john/.fs2elastic/fs2elastic.conf\"\n\n[DatasetConfig]\ndataset_source_dir = \"/home/john/csv_data_set\"\ndataset_supported_file_extensions = [ \"csv\", \"xlsx\", \"xls\", \"json\"]\ndataset_max_workers = 1\ndataset_threads_per_worker = 10\ndataset_chunk_size = 200\n\n[ESConfig]\nes_hosts = [ \"https://localhost:9200\",]\nes_username = \"elastic\"\nes_password = \"\"\nes_timeout = 300\nes_index_prefix = \"fs2es-\"\nes_ssl_ca = \"\"\nes_verify_certs = false\n\n[LogConfig]\nlog_file_path = \"/home/john/.fs2elastic/fs2elastic.log\"\nlog_max_size = 10485760\nlog_backup_count = 5\n```\n\n### Usage\n\n#### with Default config file\n\n```bash\nfs2elastic\n```\n\n#### with Custom config file\n\n```bash\nfs2elastic -c <config file path>\neg: fs2elastic -c ~/Documents/fs2elastic_custom_config.conf\n```\n\n##### Help\n\n```bash\nfs2elastic -h\n```\n\n##### Version\n\n```bash\nfs2elastic -v\n```\n\n#### Use as a system service\n\n`NOTE: Required sudo permission`\n\n```bash\n# install Package\nsudo pip install fs2elastic\n```\n\n```bash\n# Create fs2elastic home directory somewhere. for eg /tmp/fs2elastic/\nmkdir /tmp/fs2elastic/\n```\n\n```bash\n# create config file at /tmp/fs2elastic/fs2elastic.conf inside fs2elastic home directory created above\n[AppConfig]\napp_home = \"/home/john/.fs2elastic\"\napp_config_file_path = \"/home/john/.fs2elastic/fs2elastic.conf\"\n\n[DatasetConfig]\ndataset_source_dir = \"/home/john/csv_data_set\"\ndataset_supported_file_extensions = [ \"csv\", \"xlsx\", \"xls\", \"json\"]\ndataset_max_workers = 1\ndataset_threads_per_worker = 10\ndataset_chunk_size = 200\n\n[ESConfig]\nes_hosts = [ \"https://localhost:9200\",]\nes_username = \"elastic\"\nes_password = \"\"\nes_timeout = 300\nes_index_prefix = \"fs2es-\"\nes_ssl_ca = \"\"\nes_verify_certs = false\n\n[LogConfig]\nlog_file_path = \"/home/john/.fs2elastic/fs2elastic.log\"\nlog_max_size = 10485760\nlog_backup_count = 5\n\n# replace with correct values\n```\n\n```bash\n# create service file at /etc/systemd/system/fs2elastic.service with following content\n\n[Unit]\nDescription=FS2Elastic Service\nAfter=network.target\n\n[Service]\nUser=root\nGroup=root\nExecStart=/usr/bin/fs2elastic -c /tmp/fs2elastic/fs2elastic.conf\nRestart=always\n\n[Install]\nWantedBy=multi-user.target\n```\n\n```bash\n# Reloading Change to Systemd Unit Files\nsudo systemctl daemon-reload\n```\n\n```bash\n# Start service\nsudo systemctl start fs2elastic\n```\n\n```bash\n# Enable service to start on boot\nsudo systemctl enable fs2elastic\n```\n\n```bash\n# check status\nsudo systemctl status fs2elastic\n```\n\n```bash\n# check logs\nsudo tail -f /var/log/fs2elastic.log\n```\n\n```bash\n# To stop service\nsudo systemctl stop fs2elastic\n```\n\n```bash\n# To remove service\nsudo systemctl disable fs2elastic\nsudo rm /etc/systemd/system/fs2elastic.service\n\n```\n\n### Uninstall\n\n```bash\npip uninstall fs2elastic\n```\n\n### Who do I talk to?\n\n- Repo owner or admin\n- Other community or team contact\n",
"bugtrack_url": null,
"license": "MIT License Copyright (c) 2024 Pankaj Jackson Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \"Software\"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ",
"summary": "Python based CLI for Syncing File System based Data (CSV file etc.) to Elasticsearch.",
"version": "0.1.1",
"project_urls": {
"Bug Tracker": "https://github.com/pankajackson/FS2Elastic/issues",
"Homepage": "https://github.com/pankajackson/FS2Elastic",
"Source": "https://github.com/pankajackson/FS2Elastic"
},
"split_keywords": [
"csv",
" sync",
" elasticsearch",
" data"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "360bf31777edd59ff8cbfd86fe2d2b6c9de9da14f2992b6f5502167ef4d331b4",
"md5": "4cc958357306925c60dc18b2f52d7c44",
"sha256": "72ec094f4f2e78a358f4df7fb7944ba1acccbbe83b8a968be2a724b973886f8c"
},
"downloads": -1,
"filename": "fs2elastic-0.1.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "4cc958357306925c60dc18b2f52d7c44",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.10",
"size": 15301,
"upload_time": "2024-03-23T18:21:32",
"upload_time_iso_8601": "2024-03-23T18:21:32.208448Z",
"url": "https://files.pythonhosted.org/packages/36/0b/f31777edd59ff8cbfd86fe2d2b6c9de9da14f2992b6f5502167ef4d331b4/fs2elastic-0.1.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "b8ccac867a1b88a838ee3295ef298d36ca64a210d86e8e3973428578881db2fb",
"md5": "b4cae188ba1ad4708b28199aa65c5bdf",
"sha256": "8c52003b382ec93b20a49b41e1af1369e65705f4467926e9c5eb54df46418528"
},
"downloads": -1,
"filename": "fs2elastic-0.1.1.tar.gz",
"has_sig": false,
"md5_digest": "b4cae188ba1ad4708b28199aa65c5bdf",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.10",
"size": 14468,
"upload_time": "2024-03-23T18:21:34",
"upload_time_iso_8601": "2024-03-23T18:21:34.409361Z",
"url": "https://files.pythonhosted.org/packages/b8/cc/ac867a1b88a838ee3295ef298d36ca64a210d86e8e3973428578881db2fb/fs2elastic-0.1.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-03-23 18:21:34",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "pankajackson",
"github_project": "FS2Elastic",
"github_not_found": true,
"lcname": "fs2elastic"
}
