| Name | vectorgrep JSON |
| Version |
1.1.0
JSON |
| download |
| home_page | |
| Summary | Utilities for rapid text file processing using Vectorscan/Hyperscan in Python |
| upload_time | 2024-03-17 17:33:57 |
| maintainer | |
| docs_url | None |
| author | David Fritz |
| requires_python | >=3.10 |
| license | MIT |
| keywords |
regex
logs
hyperscan
vectorscan
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# VectorGrep
[](https://docs.python.org/3.10/)
[](https://devguide.python.org/versions)
[](https://google.github.io/styleguide/pyguide.html)
[](https://github.com/PyCQA/isort)
[](https://github.com/psf/black)
[](https://github.com/PyCQA/pycodestyle)
[](https://github.com/PyCQA/pydocstyle)
[](https://github.com/python/mypy)
[](https://github.com/PyCQA/pylint)
[](https://github.com/pytest-dev/pytest)
[](https://github.com/PyCQA/bandit)
[](LICENSE)
VectorGrep is a high-performance (Vectorized) Global Regular Expression "Processing" library for Python. It uses
Vectorscan (a portable fork of Intel Hyperscan) to maximize performance, and can be used with multithreaded
or multiprocessed applications. VectorGrep is also home to `vectorgrep` (Vectorized Global Regular Expression Printer),
a multithreaded/multi-file "grep" command to search many files in parallel. It can often be used as a drop in
replacement for `grep/egrep/zgrep` etc.
While a standard "grep" is designed to "print", VectorGrep is designed to allow full control over "processing".
It supports scanning compressed, or uncompressed, text files for regular expressions, and customizing the action
to take when a match is found. For full information about the performance of Vectorscan (and Hyperscan), refer to:
[VectorCamp: Vectorscan](https://github.com/VectorCamp/vectorscan)
[Intel: Hyperscan](https://github.com/intel/hyperscan)
VectorGrep also is the successor to [HyperGrep](https://github.com/pyranha-labs/hypergrep). It is designed to be
a drop in replacement for the original during initial releases. Refer to the [FAQ](#faq) for more information
about this change.
## Table Of Contents
* [Key Features](#key-features)
* [Compatibility](#compatibility)
* [Getting Started](#getting-started)
* [Installation](#installation)
* [Examples](#examples)
* [Contribute](#contribute)
* [Advanced Guides](#advanced-guides)
* [FAQ](#faq)
## Key Features
- **Simplicity**
- No experience with Vectorscan/Hyperscan required. Provides "grep" styled interfaces.
- No external dependencies, and no building required (on natively supported platforms).
- Built in support for compressed and uncompressed files.
- **Speed**
- Uses Vectorscan/Hyperscan, a high-performance multiple regex matching library.
- Performs read and regex operations outside Python.
- Batches results for Python, reducing overhead (customizable).
- **Parallelism**
- Bypasses GIL (Global Interpreter Lock) during read and regex operations to allow proper multithreading.
- Python consumer threads (callbacks) are able to handle many producer threads (readers).
## Compatibility
- Supports Python 3.10+
- Supports Linux systems with x86_64 architecture
- Ubuntu Focal (20.04), Debian Bullseye (11), and above out of the box
- Ubuntu Trusty (14.04) and above with gcc-9/g++-9 installed
- Other Operating System configurations may work, but are not tested/guaranteed
- Linux distros other than Debian/Ubuntu should work, assuming GLIBC is high enough
- May be able to be built on Windows/OSX manually
- More platforms are planned to be supported (natively) in the future
- Some regex constructs are not supported by Vectorscan/Hyperscan in order to guarantee stable performance
- For more information refer to: [Unsupported Constructs](https://intel.github.io/hyperscan/dev-reference/compilation.html#unsupported-constructs)
## Getting Started
### Installation
- Install VectorGrep via pip:
```shell
pip install vectorgrep
```
- Or via git clone:
```shell
git clone <path to fork>
cd vectorgrep
pip install .
```
- Or build and install from wheel:
```shell
# Build locally.
git clone <path to fork>
cd vectorgrep
make wheel
# Push dist/vectorgrep*.tar.gz to environment where it will be installed.
pip install dist/vectorgrep*.tar.gz
```
### Examples
- Read one file with the example single threaded command:
```shell
# vectorgrep/scanner.py <regex> <file>
vectorgrep/scanner.py pattern ./vectorgrep/scanner.py
```
- Read multiple files with the multithreaded command (drop in replacement for `grep` where patterns are compatible):
```shell
# From install:
# vectorgrep <regex> <file(s)>
vectorgrep pattern ./vectorgrep/scanner.py
# From package:
# vectorgrep/multiscanner.py <regex> <file>
vectorgrep/multiscanner.py pattern ./vectorgrep/scanner.py
```
- Collect all matches from a file, similar to grep, and perform a custom operation on results:
```python
import vectorgrep
file = "./vectorgrep/scanner.py"
pattern = 'pattern'
results, return_code = vectorgrep.grep(file, [pattern])
for index, line in results:
print(f'{index}: {line}')
```
- Manually scan a file and perform a custom operation on match:
```python
import vectorgrep
file = "./vectorgrep/scanner.py"
pattern = 'pattern'
def on_match(matches: list, count: int) -> None:
for index in range(count):
match = matches[index]
line = match.line.decode(errors='ignore')
print(f'Custom print: {line.rstrip()}')
vectorgrep.scan(file, [pattern], on_match)
```
- Override the `libhs` and/or `libzstd` libraries to use files outside the package.
Must be called before any other usage of `vectorgrep`:
```python
import vectorgrep
vectorgrep.configure_libraries(
libhs='/home/myuser/libhs.so.mybuild',
libzstd='/home/myuser/libzstd.so.mybuild',
)
```
### Contributing
Refer to the [Contributing Guide](CONTRIBUTING.md) for information on how to contribute to this project.
### Advanced Guides
Refer to [How Tos](docs/HOW_TO.md) for more advanced topics, such as building the shared library objects.
## FAQ
#### Q: How does VectorGrep compare to other Vectorscan/Hyperscan python libraries?
**A:** VectorGrep has a specific goal: provide a high performance "grep" like interface in python,
but with more control. It is not intended to be a full set of bindings to Vectorscan/Hyperscan. If you need
full control over the low level backend, there are other python libraries intended for that use case. Here are
a few of the reasons for the focused goal of this library:
- Simplify developer integration.
- No experience with Vectorscan/Hyperscan required.
- Familiarity with `grep` variants beneficial, but not required.
- Avoid messy subprocess chains common in "parallel grep" implementations.
- Commands like `zgrep` are actually a `zcat` + `grep`. This can lead to 3+ processes per file read.
- Subprocessing is messy in general, best to minimize its use as much as possible.
- Optimize performance.
- Reduce callbacks to/from python to reduce overhead.
- Allow true multithreading during read and regex matching.
- Provide the pattern matched in multi-regex searches, without having to repeat the search in Python.
When it comes to performance, here is an example of the benefit of this design. Due to the performance of
Vectorscan/Hyperscan, it is also often faster than native `grep` variants, even while using python. Scenario setup:
- 2.10GHz Intel x86_64 Processor
- ~17M line file (~300M gzip compressed, ~3G uncompressed).
- ~800 PCRE patterns.
- Counting only, no extra processing of lines.
- Each job run 5 times and averaged (lower is better).
| | Scenario (Uncompressed timings in parenthesis) | VectorGrep | Full bindings | zgrep (grep) |
|---|------------------------------------------------|---------------|-------------------|---------------|
| 1 | ~90K matches, 1 pattern | 8.2s (2.5s) | 22.8s (15.5s) | 12.5s (5.2s) |
| 2 | ~900K matches, 10 patterns | 9.7s (3.8s) | 25.7s (16.8s) | 19.8s (17.3s) |
| 3 | ~15M matches, ~800 patterns | 44.2s (38.1s) | 73.5s (57.7s) | * |
| 4 | Scenario #3 (x4 files), 1 process (4 threads) | 49.6s (46.8s) | 1432.6s (1302.2s) | * |
* GNU grep does not allow multiple PCRE patterns natively, and concatenation via "or" failed.
#### Q: How do I make a custom build for a system other than Linux x86_64?
**A**: Refer to [How To: Build the libraries for different architectures](docs/HOW_TO.md#build-the-libraries-for-different-architectures)
#### Q: I only have an ARM CPU, can I build/run the x86_64 libraries?
**A**: Depends. The current production build supports native x86_64 CPUs, as well as virtualized (in most scenarios).
For example, if you are on a Mac M1/M2/etc., you can use Docker and a supported image in x64 mode with
`--platform linux/amd64`. Performance may vary however, as the code is running through virtual machine emulation.
This process can also be used to build new libraries if your system is set up properly for emulation.
Refer to [How To: Build the libraries for different architectures](docs/HOW_TO.md#build-the-libraries-for-different-architectures)
for more information about supporting additional environments (natively or through emulation) besides Linux x86_64.
#### Q: Why was Vectorscan forked from Hyperscan?
**A:** Vectorscan was originally created to provide a portable fork of Hyperscan, and allow running on other
architectures such as ARM. Intel changed the license of Hyperscan from BSD to IPL (Intel Proprietary License)
starting in 5.5, while Vectorscan continues to provide updates and remain fully open source. For more information:
[Vectorscan: Why was there a need for a fork?](https://github.com/VectorCamp/vectorscan#why-was-there-a-need-for-a-fork)
[Vectorscan: Hyperscan license change](https://github.com/VectorCamp/vectorscan#hyperscan-license-change-after-54)
#### Q: Why is VectorGrep not a fork of HyperGrep?
**A:** HyperGrep receives maintenance updates, but over time it will become a different solution from
VectorGrep, and eventually become no longer updated, due to the licensing changes made by Intel to Hyperscan. In order
to keep the responsibilities of each clearly separated, and avoid any confusion about backports or feature requests,
it was decided to make a "clean cut" of HyperGrep, instead of using a "fork". There are no plans to backport any
features from VectorGrep to HyperGrep. VectorGrep starts from HyperGrep commit 9c6f2b2. The original commit
history can be found in [HyperGrep History](docs/HYPERGREP_HISTORY)
Raw data
{
"_id": null,
"home_page": "",
"name": "vectorgrep",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.10",
"maintainer_email": "",
"keywords": "regex,logs,hyperscan,vectorscan",
"author": "David Fritz",
"author_email": "",
"download_url": "https://files.pythonhosted.org/packages/ac/c2/5833bd1b41e1bc40a3fc286f67db19bc54dc7f4873f43eec0ba951763a37/vectorgrep-1.1.0.tar.gz",
"platform": "Linux",
"description": "# VectorGrep\n\n[](https://docs.python.org/3.10/)\n[](https://devguide.python.org/versions)\n[](https://google.github.io/styleguide/pyguide.html)\n[](https://github.com/PyCQA/isort)\n[](https://github.com/psf/black)\n[](https://github.com/PyCQA/pycodestyle)\n[](https://github.com/PyCQA/pydocstyle)\n[](https://github.com/python/mypy)\n[](https://github.com/PyCQA/pylint)\n[](https://github.com/pytest-dev/pytest)\n[](https://github.com/PyCQA/bandit)\n[](LICENSE)\n\nVectorGrep is a high-performance (Vectorized) Global Regular Expression \"Processing\" library for Python. It uses\nVectorscan (a portable fork of Intel Hyperscan) to maximize performance, and can be used with multithreaded\nor multiprocessed applications. VectorGrep is also home to `vectorgrep` (Vectorized Global Regular Expression Printer),\na multithreaded/multi-file \"grep\" command to search many files in parallel. It can often be used as a drop in\nreplacement for `grep/egrep/zgrep` etc.\n\nWhile a standard \"grep\" is designed to \"print\", VectorGrep is designed to allow full control over \"processing\".\nIt supports scanning compressed, or uncompressed, text files for regular expressions, and customizing the action\nto take when a match is found. For full information about the performance of Vectorscan (and Hyperscan), refer to: \n[VectorCamp: Vectorscan](https://github.com/VectorCamp/vectorscan) \n[Intel: Hyperscan](https://github.com/intel/hyperscan)\n\nVectorGrep also is the successor to [HyperGrep](https://github.com/pyranha-labs/hypergrep). It is designed to be\na drop in replacement for the original during initial releases. Refer to the [FAQ](#faq) for more information\nabout this change.\n\n\n## Table Of Contents\n\n * [Key Features](#key-features)\n * [Compatibility](#compatibility)\n * [Getting Started](#getting-started)\n * [Installation](#installation)\n * [Examples](#examples)\n * [Contribute](#contribute)\n * [Advanced Guides](#advanced-guides)\n * [FAQ](#faq)\n\n\n## Key Features\n\n- **Simplicity**\n - No experience with Vectorscan/Hyperscan required. Provides \"grep\" styled interfaces.\n - No external dependencies, and no building required (on natively supported platforms).\n - Built in support for compressed and uncompressed files.\n- **Speed**\n - Uses Vectorscan/Hyperscan, a high-performance multiple regex matching library.\n - Performs read and regex operations outside Python.\n - Batches results for Python, reducing overhead (customizable).\n- **Parallelism**\n - Bypasses GIL (Global Interpreter Lock) during read and regex operations to allow proper multithreading.\n - Python consumer threads (callbacks) are able to handle many producer threads (readers).\n\n\n## Compatibility\n\n- Supports Python 3.10+\n- Supports Linux systems with x86_64 architecture\n - Ubuntu Focal (20.04), Debian Bullseye (11), and above out of the box\n - Ubuntu Trusty (14.04) and above with gcc-9/g++-9 installed\n - Other Operating System configurations may work, but are not tested/guaranteed\n - Linux distros other than Debian/Ubuntu should work, assuming GLIBC is high enough\n - May be able to be built on Windows/OSX manually\n - More platforms are planned to be supported (natively) in the future\n- Some regex constructs are not supported by Vectorscan/Hyperscan in order to guarantee stable performance\n - For more information refer to: [Unsupported Constructs](https://intel.github.io/hyperscan/dev-reference/compilation.html#unsupported-constructs)\n\n\n## Getting Started\n\n### Installation\n\n- Install VectorGrep via pip:\n ```shell\n pip install vectorgrep\n ```\n\n- Or via git clone:\n ```shell\n git clone <path to fork>\n cd vectorgrep\n pip install .\n ```\n\n- Or build and install from wheel:\n ```shell\n # Build locally.\n git clone <path to fork>\n cd vectorgrep\n make wheel\n \n # Push dist/vectorgrep*.tar.gz to environment where it will be installed.\n pip install dist/vectorgrep*.tar.gz\n ```\n\n### Examples\n\n- Read one file with the example single threaded command:\n ```shell\n # vectorgrep/scanner.py <regex> <file>\n vectorgrep/scanner.py pattern ./vectorgrep/scanner.py\n ```\n\n- Read multiple files with the multithreaded command (drop in replacement for `grep` where patterns are compatible):\n ```shell\n # From install:\n # vectorgrep <regex> <file(s)>\n vectorgrep pattern ./vectorgrep/scanner.py\n\n # From package:\n # vectorgrep/multiscanner.py <regex> <file>\n vectorgrep/multiscanner.py pattern ./vectorgrep/scanner.py\n ```\n\n- Collect all matches from a file, similar to grep, and perform a custom operation on results:\n ```python\n import vectorgrep\n \n file = \"./vectorgrep/scanner.py\"\n pattern = 'pattern'\n \n results, return_code = vectorgrep.grep(file, [pattern])\n for index, line in results:\n print(f'{index}: {line}')\n ```\n\n- Manually scan a file and perform a custom operation on match:\n ```python\n import vectorgrep\n \n file = \"./vectorgrep/scanner.py\"\n pattern = 'pattern'\n\n def on_match(matches: list, count: int) -> None:\n for index in range(count):\n match = matches[index]\n line = match.line.decode(errors='ignore')\n print(f'Custom print: {line.rstrip()}')\n \n vectorgrep.scan(file, [pattern], on_match)\n ```\n\n- Override the `libhs` and/or `libzstd` libraries to use files outside the package.\nMust be called before any other usage of `vectorgrep`:\n ```python\n import vectorgrep\n\n vectorgrep.configure_libraries(\n libhs='/home/myuser/libhs.so.mybuild',\n libzstd='/home/myuser/libzstd.so.mybuild',\n )\n ```\n\n### Contributing\n\nRefer to the [Contributing Guide](CONTRIBUTING.md) for information on how to contribute to this project.\n\n### Advanced Guides\n\nRefer to [How Tos](docs/HOW_TO.md) for more advanced topics, such as building the shared library objects.\n\n\n## FAQ\n\n#### Q: How does VectorGrep compare to other Vectorscan/Hyperscan python libraries?\n\n**A:** VectorGrep has a specific goal: provide a high performance \"grep\" like interface in python,\nbut with more control. It is not intended to be a full set of bindings to Vectorscan/Hyperscan. If you need\nfull control over the low level backend, there are other python libraries intended for that use case. Here are\na few of the reasons for the focused goal of this library:\n\n- Simplify developer integration.\n - No experience with Vectorscan/Hyperscan required.\n - Familiarity with `grep` variants beneficial, but not required.\n- Avoid messy subprocess chains common in \"parallel grep\" implementations.\n - Commands like `zgrep` are actually a `zcat` + `grep`. This can lead to 3+ processes per file read.\n - Subprocessing is messy in general, best to minimize its use as much as possible.\n- Optimize performance.\n - Reduce callbacks to/from python to reduce overhead.\n - Allow true multithreading during read and regex matching.\n - Provide the pattern matched in multi-regex searches, without having to repeat the search in Python.\n\nWhen it comes to performance, here is an example of the benefit of this design. Due to the performance of\nVectorscan/Hyperscan, it is also often faster than native `grep` variants, even while using python. Scenario setup:\n- 2.10GHz Intel x86_64 Processor\n- ~17M line file (~300M gzip compressed, ~3G uncompressed).\n- ~800 PCRE patterns.\n- Counting only, no extra processing of lines.\n- Each job run 5 times and averaged (lower is better).\n\n| | Scenario (Uncompressed timings in parenthesis) | VectorGrep | Full bindings | zgrep (grep) |\n|---|------------------------------------------------|---------------|-------------------|---------------|\n| 1 | ~90K matches, 1 pattern | 8.2s (2.5s) | 22.8s (15.5s) | 12.5s (5.2s) |\n| 2 | ~900K matches, 10 patterns | 9.7s (3.8s) | 25.7s (16.8s) | 19.8s (17.3s) |\n| 3 | ~15M matches, ~800 patterns | 44.2s (38.1s) | 73.5s (57.7s) | * |\n| 4 | Scenario #3 (x4 files), 1 process (4 threads) | 49.6s (46.8s) | 1432.6s (1302.2s) | * |\n\n* GNU grep does not allow multiple PCRE patterns natively, and concatenation via \"or\" failed.\n\n#### Q: How do I make a custom build for a system other than Linux x86_64?\n\n**A**: Refer to [How To: Build the libraries for different architectures](docs/HOW_TO.md#build-the-libraries-for-different-architectures)\n\n#### Q: I only have an ARM CPU, can I build/run the x86_64 libraries?\n\n**A**: Depends. The current production build supports native x86_64 CPUs, as well as virtualized (in most scenarios).\nFor example, if you are on a Mac M1/M2/etc., you can use Docker and a supported image in x64 mode with\n`--platform linux/amd64`. Performance may vary however, as the code is running through virtual machine emulation.\nThis process can also be used to build new libraries if your system is set up properly for emulation.\nRefer to [How To: Build the libraries for different architectures](docs/HOW_TO.md#build-the-libraries-for-different-architectures)\nfor more information about supporting additional environments (natively or through emulation) besides Linux x86_64.\n\n#### Q: Why was Vectorscan forked from Hyperscan?\n\n**A:** Vectorscan was originally created to provide a portable fork of Hyperscan, and allow running on other\narchitectures such as ARM. Intel changed the license of Hyperscan from BSD to IPL (Intel Proprietary License)\nstarting in 5.5, while Vectorscan continues to provide updates and remain fully open source. For more information: \n[Vectorscan: Why was there a need for a fork?](https://github.com/VectorCamp/vectorscan#why-was-there-a-need-for-a-fork) \n[Vectorscan: Hyperscan license change](https://github.com/VectorCamp/vectorscan#hyperscan-license-change-after-54)\n\n#### Q: Why is VectorGrep not a fork of HyperGrep?\n\n**A:** HyperGrep receives maintenance updates, but over time it will become a different solution from\nVectorGrep, and eventually become no longer updated, due to the licensing changes made by Intel to Hyperscan. In order\nto keep the responsibilities of each clearly separated, and avoid any confusion about backports or feature requests,\nit was decided to make a \"clean cut\" of HyperGrep, instead of using a \"fork\". There are no plans to backport any\nfeatures from VectorGrep to HyperGrep. VectorGrep starts from HyperGrep commit 9c6f2b2. The original commit\nhistory can be found in [HyperGrep History](docs/HYPERGREP_HISTORY)\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Utilities for rapid text file processing using Vectorscan/Hyperscan in Python",
"version": "1.1.0",
"project_urls": {
"Changelog": "https://github.com/pyranha-labs/vectorgrep/releases",
"Home": "https://github.com/pyranha-labs/vectorgrep",
"Issues": "https://github.com/pyranha-labs/vectorgrep/issues"
},
"split_keywords": [
"regex",
"logs",
"hyperscan",
"vectorscan"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "8d7c0e1a134fe09665db3eef2c7e90a76198281cb4c38691e86f050248f4687f",
"md5": "03e55d8fb60dc8e80d393897ec971908",
"sha256": "fb9d5ae893e9d1dcc214802e99462e6f8d2f80c7168f5296aeb4b1effbce2083"
},
"downloads": -1,
"filename": "vectorgrep-1.1.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "03e55d8fb60dc8e80d393897ec971908",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.10",
"size": 2942537,
"upload_time": "2024-03-17T17:33:55",
"upload_time_iso_8601": "2024-03-17T17:33:55.746405Z",
"url": "https://files.pythonhosted.org/packages/8d/7c/0e1a134fe09665db3eef2c7e90a76198281cb4c38691e86f050248f4687f/vectorgrep-1.1.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "acc25833bd1b41e1bc40a3fc286f67db19bc54dc7f4873f43eec0ba951763a37",
"md5": "5c6dfa5b4b3f51ab96a5dd81f3e25d69",
"sha256": "2ba69469819b66d250d736c99e45958f99671fc0817ae49f64dda246c6a6c550"
},
"downloads": -1,
"filename": "vectorgrep-1.1.0.tar.gz",
"has_sig": false,
"md5_digest": "5c6dfa5b4b3f51ab96a5dd81f3e25d69",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.10",
"size": 2930989,
"upload_time": "2024-03-17T17:33:57",
"upload_time_iso_8601": "2024-03-17T17:33:57.964354Z",
"url": "https://files.pythonhosted.org/packages/ac/c2/5833bd1b41e1bc40a3fc286f67db19bc54dc7f4873f43eec0ba951763a37/vectorgrep-1.1.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-03-17 17:33:57",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "pyranha-labs",
"github_project": "vectorgrep",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "vectorgrep"
}
