grepsr-cli


Namegrepsr-cli JSON
Version 0.9.18 PyPI version JSON
download
home_pagehttps://bitbucket.org/grepsr/grepsr-cli/
SummaryA Cli tool for Grepsr Developers
upload_time2024-11-26 11:34:53
maintainerNone
docs_urlNone
authorgrepsr
requires_pythonNone
licenseunlicensed
keywords
VCS
bugtrack_url
requirements No requirements were recorded.
Travis-CI No Travis.
coveralls test coverage No coveralls.
            # A Cli tool for Grepsr Developers

## Installation
```
$ pip install grepsr-cli
```


## Usage
### passing parameters to `amazon_com` service.
```bash
gcli crawler test -s amazon_com -p '{"urls":["https://amazon.com/VVUH4HJ","https://amazon.com/FV4434"]}'
```

### if JSON is complex, use file instead
```
# contents of /tmp/amazon_params.json
{"urls": ["https://amazon.com/VV%20UH4HJ"], "strip": ["'", "\"", "\\"]}

gcli crawler test -s amazon_com --params-file '/tmp/amazon_params.json'
```

#### Hacks Used.
> If the json parameter has a space, it might break parameter parsing.
> If the json parameter has a dash `-` and any character after it has a space, it will break parameter parsing.
Cause: no double quoting around $@ in `run_service.php:5:49` [here](https://bitbucket.org/grepsr/vortex-backend/src/09c263fb0bb538003db01e1d6742a43ae6ebc61a/deploy/vortex-backend/scripts/run_service.sh#lines-5)
> This is fixed hackily by replacing string with its unicode \u0020 sequence. This works beacause $@ does not split on \u0020.

### inject custom command.
Say, for example you wanted to a inject a php function so that it could be called from inside you service_code when testing locally.
Note: All these files should only be created inside `~/.grepsr/tmp`. Creating it outside will not work.

1. Create a file called `inject.php` inside `~/.grepsr/tmp/`
2. Implement your function inside `~/.grepsr/tmp/inject.php`
```php
function addRowLocal($arr) {
    ...
    ...
}
```
3. Create a file called `inject.sh` inside `~/.grepsr/tmp/`
4. inside inject.sh add:
```
alias php='php -d auto_prepend_file=/tmp/inject.php'
```
Note: the file location is `/tmp/inject.php` instead of `~/.grepsr/tmp/inject.php`.
This is because, the local path `~/.grepsr/tmp` gets mapped to `/tmp` in the docker container.
And `inject.sh` runs inside docker, instead of the local filesystem.
5. Add an entry in `~/.grepst/config.yml` like so:
```yml
    php:
        ...
        sdk_image: ...
        pre_entry_run_file: inject.sh      # relative and limited to the tmp/ dir
```
6. Now you can use `addRowLocal()` in your any of your files.
```php
public function main($params) {
    ...
    $arr = $this->dataSet->getEmptyRow();
    addRowLocal($arr); // won't throw error
    ...
}
```
## Development
> Be sure to uninstall gcli first, with
`pip uninstall grepsr-cli`

```bash
git clone git@bitbucket.org:zznixt07/gcli.git grepsrcli
cd grepsrcli
pip install -e .
```

## Features Added
- drop stash after pushed successully. Before this, all stashes were always kept.
- run a custom shell file before running your crawler. This allows possiblity like always injecting a php function in all your crawlers.
- auto add `Dependencies: ...` that your crawler class extends (dependecies that are not extended by crawler classes but used elsewhere is upcoming)


# TODO:
- Experiment with git rebase on deploy fail. `git rebase origin/master --autostash && git push`
- Handle Prioritization of same plugin name across multiple repo more deterministically. (maybe prioritize cwd path?)
- node only run crawler is npm install is successfull. (add && between npm install and npm start)
- run `tsc` before deploying `vortex-ts-registry` packages
- add option to force update dependecies to latest version for all/specific `vortex-ts-registry` dependencies
- handle ctrl+c during node package install on docker. (currently it continues running in BG)
- add new baseclass typescript package and do not include SOP, (do not normalize - to _) change `npm start` to `tsc` and test runs. Also generate .d.ts file in tsconfig.json

            

Raw data

            {
    "_id": null,
    "home_page": "https://bitbucket.org/grepsr/grepsr-cli/",
    "name": "grepsr-cli",
    "maintainer": null,
    "docs_url": null,
    "requires_python": null,
    "maintainer_email": null,
    "keywords": null,
    "author": "grepsr",
    "author_email": "dev@grepsr.com",
    "download_url": "https://files.pythonhosted.org/packages/39/9f/71c2d3c7f7b2f13099c2c887c89c07c8eb994e3f16e5702973502d4dc557/grepsr_cli-0.9.18.tar.gz",
    "platform": null,
    "description": "# A Cli tool for Grepsr Developers\n\n## Installation\n```\n$ pip install grepsr-cli\n```\n\n\n## Usage\n### passing parameters to `amazon_com` service.\n```bash\ngcli crawler test -s amazon_com -p '{\"urls\":[\"https://amazon.com/VVUH4HJ\",\"https://amazon.com/FV4434\"]}'\n```\n\n### if JSON is complex, use file instead\n```\n# contents of /tmp/amazon_params.json\n{\"urls\": [\"https://amazon.com/VV%20UH4HJ\"], \"strip\": [\"'\", \"\\\"\", \"\\\\\"]}\n\ngcli crawler test -s amazon_com --params-file '/tmp/amazon_params.json'\n```\n\n#### Hacks Used.\n> If the json parameter has a space, it might break parameter parsing.\n> If the json parameter has a dash `-` and any character after it has a space, it will break parameter parsing.\nCause: no double quoting around $@ in `run_service.php:5:49` [here](https://bitbucket.org/grepsr/vortex-backend/src/09c263fb0bb538003db01e1d6742a43ae6ebc61a/deploy/vortex-backend/scripts/run_service.sh#lines-5)\n> This is fixed hackily by replacing string with its unicode \\u0020 sequence. This works beacause $@ does not split on \\u0020.\n\n### inject custom command.\nSay, for example you wanted to a inject a php function so that it could be called from inside you service_code when testing locally.\nNote: All these files should only be created inside `~/.grepsr/tmp`. Creating it outside will not work.\n\n1. Create a file called `inject.php` inside `~/.grepsr/tmp/`\n2. Implement your function inside `~/.grepsr/tmp/inject.php`\n```php\nfunction addRowLocal($arr) {\n    ...\n    ...\n}\n```\n3. Create a file called `inject.sh` inside `~/.grepsr/tmp/`\n4. inside inject.sh add:\n```\nalias php='php -d auto_prepend_file=/tmp/inject.php'\n```\nNote: the file location is `/tmp/inject.php` instead of `~/.grepsr/tmp/inject.php`.\nThis is because, the local path `~/.grepsr/tmp` gets mapped to `/tmp` in the docker container.\nAnd `inject.sh` runs inside docker, instead of the local filesystem.\n5. Add an entry in `~/.grepst/config.yml` like so:\n```yml\n    php:\n        ...\n        sdk_image: ...\n        pre_entry_run_file: inject.sh      # relative and limited to the tmp/ dir\n```\n6. Now you can use `addRowLocal()` in your any of your files.\n```php\npublic function main($params) {\n    ...\n    $arr = $this->dataSet->getEmptyRow();\n    addRowLocal($arr); // won't throw error\n    ...\n}\n```\n## Development\n> Be sure to uninstall gcli first, with\n`pip uninstall grepsr-cli`\n\n```bash\ngit clone git@bitbucket.org:zznixt07/gcli.git grepsrcli\ncd grepsrcli\npip install -e .\n```\n\n## Features Added\n- drop stash after pushed successully. Before this, all stashes were always kept.\n- run a custom shell file before running your crawler. This allows possiblity like always injecting a php function in all your crawlers.\n- auto add `Dependencies: ...` that your crawler class extends (dependecies that are not extended by crawler classes but used elsewhere is upcoming)\n\n\n# TODO:\n- Experiment with git rebase on deploy fail. `git rebase origin/master --autostash && git push`\n- Handle Prioritization of same plugin name across multiple repo more deterministically. (maybe prioritize cwd path?)\n- node only run crawler is npm install is successfull. (add && between npm install and npm start)\n- run `tsc` before deploying `vortex-ts-registry` packages\n- add option to force update dependecies to latest version for all/specific `vortex-ts-registry` dependencies\n- handle ctrl+c during node package install on docker. (currently it continues running in BG)\n- add new baseclass typescript package and do not include SOP, (do not normalize - to _) change `npm start` to `tsc` and test runs. Also generate .d.ts file in tsconfig.json\n",
    "bugtrack_url": null,
    "license": "unlicensed",
    "summary": "A Cli tool for Grepsr Developers",
    "version": "0.9.18",
    "project_urls": {
        "Homepage": "https://bitbucket.org/grepsr/grepsr-cli/"
    },
    "split_keywords": [],
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "e850d79f13f68c2ae0dfd85e54e37e99644e8a828229e3ef5e0f16c5642eb72f",
                "md5": "413d9db2c5a47d72addcc8821bf0029d",
                "sha256": "3ba94632c52bcb107cbbc69f1058efba16e3ecab858dce41b287ddb004b95217"
            },
            "downloads": -1,
            "filename": "grepsr_cli-0.9.18-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "413d9db2c5a47d72addcc8821bf0029d",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": null,
            "size": 378065,
            "upload_time": "2024-11-26T11:34:50",
            "upload_time_iso_8601": "2024-11-26T11:34:50.513210Z",
            "url": "https://files.pythonhosted.org/packages/e8/50/d79f13f68c2ae0dfd85e54e37e99644e8a828229e3ef5e0f16c5642eb72f/grepsr_cli-0.9.18-py3-none-any.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "399f71c2d3c7f7b2f13099c2c887c89c07c8eb994e3f16e5702973502d4dc557",
                "md5": "47409acab1a05db140713bd237725f71",
                "sha256": "1af507ed42e913b7390c5da7bfeab27c2bd314f842339b6531fb907666348ff6"
            },
            "downloads": -1,
            "filename": "grepsr_cli-0.9.18.tar.gz",
            "has_sig": false,
            "md5_digest": "47409acab1a05db140713bd237725f71",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 31397,
            "upload_time": "2024-11-26T11:34:53",
            "upload_time_iso_8601": "2024-11-26T11:34:53.071336Z",
            "url": "https://files.pythonhosted.org/packages/39/9f/71c2d3c7f7b2f13099c2c887c89c07c8eb994e3f16e5702973502d4dc557/grepsr_cli-0.9.18.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2024-11-26 11:34:53",
    "github": false,
    "gitlab": false,
    "bitbucket": true,
    "codeberg": false,
    "bitbucket_user": "grepsr",
    "bitbucket_project": "grepsr-cli",
    "lcname": "grepsr-cli"
}
        
Elapsed time: 0.40422s