# A Cli tool for Grepsr Developers
## Installation
```
$ pip install grepsr-cli
```
## Usage
### passing parameters to `amazon_com` service.
```bash
gcli crawler test -s amazon_com -p '{"urls":["https://amazon.com/VVUH4HJ","https://amazon.com/FV4434"]}'
```
### if JSON is complex, use file instead
```
# contents of /tmp/amazon_params.json
{"urls": ["https://amazon.com/VV%20UH4HJ"], "strip": ["'", "\"", "\\"]}
gcli crawler test -s amazon_com --params-file '/tmp/amazon_params.json'
```
#### Hacks Used.
> If the json parameter has a space, it might break parameter parsing.
> If the json parameter has a dash `-` and any character after it has a space, it will break parameter parsing.
Cause: no double quoting around $@ in `run_service.php:5:49` [here](https://bitbucket.org/grepsr/vortex-backend/src/09c263fb0bb538003db01e1d6742a43ae6ebc61a/deploy/vortex-backend/scripts/run_service.sh#lines-5)
> This is fixed hackily by replacing string with its unicode \u0020 sequence. This works beacause $@ does not split on \u0020.
### inject custom command.
Say, for example you wanted to a inject a php function so that it could be called from inside you service_code when testing locally.
Note: All these files should only be created inside `~/.grepsr/tmp`. Creating it outside will not work.
1. Create a file called `inject.php` inside `~/.grepsr/tmp/`
2. Implement your function inside `~/.grepsr/tmp/inject.php`
```php
function addRowLocal($arr) {
...
...
}
```
3. Create a file called `inject.sh` inside `~/.grepsr/tmp/`
4. inside inject.sh add:
```
alias php='php -d auto_prepend_file=/tmp/inject.php'
```
Note: the file location is `/tmp/inject.php` instead of `~/.grepsr/tmp/inject.php`.
This is because, the local path `~/.grepsr/tmp` gets mapped to `/tmp` in the docker container.
And `inject.sh` runs inside docker, instead of the local filesystem.
5. Add an entry in `~/.grepst/config.yml` like so:
```yml
php:
...
sdk_image: ...
pre_entry_run_file: inject.sh # relative and limited to the tmp/ dir
```
6. Now you can use `addRowLocal()` in your any of your files.
```php
public function main($params) {
...
$arr = $this->dataSet->getEmptyRow();
addRowLocal($arr); // won't throw error
...
}
```
## Development
> Be sure to uninstall gcli first, with
`pip uninstall grepsr-cli`
```bash
git clone git@bitbucket.org:zznixt07/gcli.git grepsrcli
cd grepsrcli
pip install -e .
```
## Features Added
- drop stash after pushed successully. Before this, all stashes were always kept.
- run a custom shell file before running your crawler. This allows possiblity like always injecting a php function in all your crawlers.
- auto add `Dependencies: ...` that your crawler class extends (dependecies that are not extended by crawler classes but used elsewhere is upcoming)
# TODO:
- Experiment with git rebase on deploy fail. `git rebase origin/master --autostash && git push`
- Handle Prioritization of same plugin name across multiple repo more deterministically. (maybe prioritize cwd path?)
- node only run crawler is npm install is successfull. (add && between npm install and npm start)
- run `tsc` before deploying `vortex-ts-registry` packages
- add option to force update dependecies to latest version for all/specific `vortex-ts-registry` dependencies
- handle ctrl+c during node package install on docker. (currently it continues running in BG)
- add new baseclass typescript package and do not include SOP, (do not normalize - to _) change `npm start` to `tsc` and test runs. Also generate .d.ts file in tsconfig.json
Raw data
{
"_id": null,
"home_page": "https://bitbucket.org/grepsr/grepsr-cli/",
"name": "grepsr-cli",
"maintainer": null,
"docs_url": null,
"requires_python": null,
"maintainer_email": null,
"keywords": null,
"author": "grepsr",
"author_email": "dev@grepsr.com",
"download_url": "https://files.pythonhosted.org/packages/39/9f/71c2d3c7f7b2f13099c2c887c89c07c8eb994e3f16e5702973502d4dc557/grepsr_cli-0.9.18.tar.gz",
"platform": null,
"description": "# A Cli tool for Grepsr Developers\n\n## Installation\n```\n$ pip install grepsr-cli\n```\n\n\n## Usage\n### passing parameters to `amazon_com` service.\n```bash\ngcli crawler test -s amazon_com -p '{\"urls\":[\"https://amazon.com/VVUH4HJ\",\"https://amazon.com/FV4434\"]}'\n```\n\n### if JSON is complex, use file instead\n```\n# contents of /tmp/amazon_params.json\n{\"urls\": [\"https://amazon.com/VV%20UH4HJ\"], \"strip\": [\"'\", \"\\\"\", \"\\\\\"]}\n\ngcli crawler test -s amazon_com --params-file '/tmp/amazon_params.json'\n```\n\n#### Hacks Used.\n> If the json parameter has a space, it might break parameter parsing.\n> If the json parameter has a dash `-` and any character after it has a space, it will break parameter parsing.\nCause: no double quoting around $@ in `run_service.php:5:49` [here](https://bitbucket.org/grepsr/vortex-backend/src/09c263fb0bb538003db01e1d6742a43ae6ebc61a/deploy/vortex-backend/scripts/run_service.sh#lines-5)\n> This is fixed hackily by replacing string with its unicode \\u0020 sequence. This works beacause $@ does not split on \\u0020.\n\n### inject custom command.\nSay, for example you wanted to a inject a php function so that it could be called from inside you service_code when testing locally.\nNote: All these files should only be created inside `~/.grepsr/tmp`. Creating it outside will not work.\n\n1. Create a file called `inject.php` inside `~/.grepsr/tmp/`\n2. Implement your function inside `~/.grepsr/tmp/inject.php`\n```php\nfunction addRowLocal($arr) {\n ...\n ...\n}\n```\n3. Create a file called `inject.sh` inside `~/.grepsr/tmp/`\n4. inside inject.sh add:\n```\nalias php='php -d auto_prepend_file=/tmp/inject.php'\n```\nNote: the file location is `/tmp/inject.php` instead of `~/.grepsr/tmp/inject.php`.\nThis is because, the local path `~/.grepsr/tmp` gets mapped to `/tmp` in the docker container.\nAnd `inject.sh` runs inside docker, instead of the local filesystem.\n5. Add an entry in `~/.grepst/config.yml` like so:\n```yml\n php:\n ...\n sdk_image: ...\n pre_entry_run_file: inject.sh # relative and limited to the tmp/ dir\n```\n6. Now you can use `addRowLocal()` in your any of your files.\n```php\npublic function main($params) {\n ...\n $arr = $this->dataSet->getEmptyRow();\n addRowLocal($arr); // won't throw error\n ...\n}\n```\n## Development\n> Be sure to uninstall gcli first, with\n`pip uninstall grepsr-cli`\n\n```bash\ngit clone git@bitbucket.org:zznixt07/gcli.git grepsrcli\ncd grepsrcli\npip install -e .\n```\n\n## Features Added\n- drop stash after pushed successully. Before this, all stashes were always kept.\n- run a custom shell file before running your crawler. This allows possiblity like always injecting a php function in all your crawlers.\n- auto add `Dependencies: ...` that your crawler class extends (dependecies that are not extended by crawler classes but used elsewhere is upcoming)\n\n\n# TODO:\n- Experiment with git rebase on deploy fail. `git rebase origin/master --autostash && git push`\n- Handle Prioritization of same plugin name across multiple repo more deterministically. (maybe prioritize cwd path?)\n- node only run crawler is npm install is successfull. (add && between npm install and npm start)\n- run `tsc` before deploying `vortex-ts-registry` packages\n- add option to force update dependecies to latest version for all/specific `vortex-ts-registry` dependencies\n- handle ctrl+c during node package install on docker. (currently it continues running in BG)\n- add new baseclass typescript package and do not include SOP, (do not normalize - to _) change `npm start` to `tsc` and test runs. Also generate .d.ts file in tsconfig.json\n",
"bugtrack_url": null,
"license": "unlicensed",
"summary": "A Cli tool for Grepsr Developers",
"version": "0.9.18",
"project_urls": {
"Homepage": "https://bitbucket.org/grepsr/grepsr-cli/"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "e850d79f13f68c2ae0dfd85e54e37e99644e8a828229e3ef5e0f16c5642eb72f",
"md5": "413d9db2c5a47d72addcc8821bf0029d",
"sha256": "3ba94632c52bcb107cbbc69f1058efba16e3ecab858dce41b287ddb004b95217"
},
"downloads": -1,
"filename": "grepsr_cli-0.9.18-py3-none-any.whl",
"has_sig": false,
"md5_digest": "413d9db2c5a47d72addcc8821bf0029d",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 378065,
"upload_time": "2024-11-26T11:34:50",
"upload_time_iso_8601": "2024-11-26T11:34:50.513210Z",
"url": "https://files.pythonhosted.org/packages/e8/50/d79f13f68c2ae0dfd85e54e37e99644e8a828229e3ef5e0f16c5642eb72f/grepsr_cli-0.9.18-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "399f71c2d3c7f7b2f13099c2c887c89c07c8eb994e3f16e5702973502d4dc557",
"md5": "47409acab1a05db140713bd237725f71",
"sha256": "1af507ed42e913b7390c5da7bfeab27c2bd314f842339b6531fb907666348ff6"
},
"downloads": -1,
"filename": "grepsr_cli-0.9.18.tar.gz",
"has_sig": false,
"md5_digest": "47409acab1a05db140713bd237725f71",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 31397,
"upload_time": "2024-11-26T11:34:53",
"upload_time_iso_8601": "2024-11-26T11:34:53.071336Z",
"url": "https://files.pythonhosted.org/packages/39/9f/71c2d3c7f7b2f13099c2c887c89c07c8eb994e3f16e5702973502d4dc557/grepsr_cli-0.9.18.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-11-26 11:34:53",
"github": false,
"gitlab": false,
"bitbucket": true,
"codeberg": false,
"bitbucket_user": "grepsr",
"bitbucket_project": "grepsr-cli",
"lcname": "grepsr-cli"
}
JSON