| Name | ocsf-lib JSON |
| Version |
0.8.2
JSON |
| download |
| home_page | None |
| Summary | Tools for working with the OCSF schema |
| upload_time | 2024-08-30 22:21:14 |
| maintainer | None |
| docs_url | None |
| author | Jeremy Fisher |
| requires_python | <4.0,>=3.11 |
| license | Apache-2.0 |
| keywords |
|
| VCS |
|
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# OCSF Library for Python
Tools for building Python scripts and applications leveraging the OCSF.
## Quick Start
If you just want to use this library as a CLI tool, install it with `pip` or
`poetry` and try the following commands:
```sh
python -m ocsf.compile path/to/ocsf-schema
python -m ocsf.compare my-schema-export.json path/to/ocsf-schema
python -m ocsf.schema 1.2.0
python -m ocsf.validate.compatibility path/to/ocsf-schema 1.2.0
```
## About
This project began with two goals:
1. Provide the OCSF community with a validator that tests for breaking changes
in `ocsf-schema` PRs.
2. Begin to provide the OCSF community with more composable tools and libraries,
as well as approachable reference implementations of OCSF related functions, in
order to make OCSF more "hackable."
The scope of this project may grow to include things like a reference
implementation OCSF schema compiler.
The project targets Python 3.11 for a balance of capability and availability.
The root level package, `ocsf`, is a namespace package so that other
repositories and artifacts can also use the `ocsf` namespace.
This library is divided into several discrete packages.
### ocsf.util: The utilities package
The `ocsf.util` package provides the `get_schema` function. This function
leverages the functionality in the `ocsf.schema` and `ocsf.api` packages (below)
to easily build an OCSF schema from a file on disk, a working copy of an OCSF
repository, or from the API.
```python
schema = get_schema("1.1.0")
schema = get_schema("./1.3.0-dev.json")
schema = get_schema("path/to/ocsf-schema")
```
### ocsf.schema: The Schema Package
The `ocsf.schema` package contains Python data classes that represent an
OCSF schema as represented from the OCSF server's API endpoints. See the
`ocsf.schema.model` module for the data model definitions.
It also includes utilities to parse the schema from a JSON string or file.
### ocsf.repository: The Repository Package
The `ocsf.repository` package contains a typed Python representation of a
working copy of an OCSF schema repository. Said another way, it represents the
OCSF metaschema and repository contents in Python.
It also includes the `read_repo` function to read a repository from disk.
### ocsf.compile: An OCSF Compiler
The `ocsf.compile` package "compiles" the OCSF schema from a repository just as
the OCSF server does (with very few exceptions). It is meant to provide:
1. An easy to use CLI tool to compile a repository into a single JSON schema
file.
2. A reference implementation for others looking to better understand OCSF
compilation or to create their own compiler.
### ocsf.api: The API Package
The `ocsf.api` package exports an `OcsfApiClient`, which is a lightweight HTTP
client that can retrieve a version of the schema over HTTP and cache it on the
local filesystem. It uses thes `export/schema`, `api/versions`, `api/profiles`,
and `api/extensions` endpoints of the OCSF server.
### ocsf.compare: The Compare Package
The `ocsf_tools.compare` package compares two versions of the OCSF schema and
generates a type safe difference. Its aim is to make schema comparisons easy to
work with.
This package grew out of a library used internally at [Query](https://query.ai).
The original is used extensively to manage upgrading Query's data model to newer
versions of OCSF, as well as to build adapters between different OCSF flavors
(like AWS Security Lake on rc2 and Query on 1.1).
There is a very simple `__main__` implementation to demonstrate the comparison.
You can use it as follows:
```sh
$ poetry run python -m ocsf_tools.compare 1.0.0 1.2.0
```
The comparison API is straightforward. Want to look for removed events?
```python
diff = compare(get_schema("1.0.0", "1.1.0"))
for name, event in diff.classes.items():
if isinstance(event, Removal):
print(f"Oh no, we've lost {name}!")
```
Or changed data types?
```python
diff = compare(get_schema("1.0.0", "1.1.0"))
for name, event in diff.classes.items():
if isinstance(event, ChangedEvent):
for attr_name, attr in event.attributes.items():
if isinstance(attr, ChangedAttr):
if isinstance(attr.type, Change):
print(f"Who changed this data type? {name}.{attr_name}")
```
Or new objects?
```python
diff = compare(get_schema("1.0.0", "1.1.0"))
for name, obj in diff.objects.items():
if isinstance(obj, Addition):
print(f"A new object {name} has been discovered!")
```
### ocsf.validate.framework: The Validation Framework Package
The `ocsf.validate.framework` package provides a lightweight framework for
validators. It was inspired by the needs of `ocsf-validator`, which may be
ported to this framework in the future.
### ocsf.validate.compatibility: The Backwards Compatibility Validator
The `ocsf.validate.compatibility` provides a backwards compatibility validator
for OCSF schema. This compares the changes between two OCSF schemata and reports
any breaking changes between the old and new version.
## Getting Started
### PyPI
The easiest way to install `ocsf-lib` is from PyPI using `pip` or `poetry`:
```sh
$ pip install ocsf-lib
```
### From Source
If you want to work with the source, the recommended installation is with `asdf`
and `poetry`.
```sh
$ asdf install
$ poetry install
```
## Contributing
This project uses `ruff` for formatting and linting, `pyright` for type
checking, and `pytest` as its test runner.
Before submitting a PR, make sure you've run following:
```sh
$ poetry run ruff format
$ poetry run ruff check
$ poetry run pyright
$ poetry run pytest
```
### Type Checking
With great effort, this library passes pyright's strict mode type checking. Keep
it that way! The OCSF schema is big, and even the metaschema is a lot to hold in
your head. Having the type checker identify mistakes for you can be very
helpful.
There is one cast used from the concrete `ChangedModel` types (`ChangedSchema`,
`ChangedAttr`, etc.) in the compare package to the generic type. For the life of
me, I can't figure it out. I blame pyright but it's probably my own fault.
### Tests
Running unit tests:
```sh
$ poetry run pytest -m "not integration"
```
Running integration tests:
```sh
$ poetry run pytest -m integration
```
**NOTE**: Some of the integration tests require an OCSF server instance, and are
using the public instance at [https://schema.ocsf.io](https://schema.ocsf.io).
This should probably use a local instance of the OCSF server instead.
Raw data
{
"_id": null,
"home_page": null,
"name": "ocsf-lib",
"maintainer": null,
"docs_url": null,
"requires_python": "<4.0,>=3.11",
"maintainer_email": null,
"keywords": null,
"author": "Jeremy Fisher",
"author_email": "jeremy@query.ai",
"download_url": "https://files.pythonhosted.org/packages/a3/a2/2199f062f4bb08ff7aa18b13b43afaf10c503e60851cbe2b28cc20f52ba3/ocsf_lib-0.8.2.tar.gz",
"platform": null,
"description": "# OCSF Library for Python\n\nTools for building Python scripts and applications leveraging the OCSF.\n\n## Quick Start\n\nIf you just want to use this library as a CLI tool, install it with `pip` or\n`poetry` and try the following commands:\n\n```sh\npython -m ocsf.compile path/to/ocsf-schema\npython -m ocsf.compare my-schema-export.json path/to/ocsf-schema\npython -m ocsf.schema 1.2.0\npython -m ocsf.validate.compatibility path/to/ocsf-schema 1.2.0\n```\n\n## About\n\nThis project began with two goals:\n\n1. Provide the OCSF community with a validator that tests for breaking changes\n in `ocsf-schema` PRs.\n2. Begin to provide the OCSF community with more composable tools and libraries,\n as well as approachable reference implementations of OCSF related functions, in\n order to make OCSF more \"hackable.\"\n\nThe scope of this project may grow to include things like a reference\nimplementation OCSF schema compiler.\n\nThe project targets Python 3.11 for a balance of capability and availability.\nThe root level package, `ocsf`, is a namespace package so that other\nrepositories and artifacts can also use the `ocsf` namespace.\n\nThis library is divided into several discrete packages.\n\n### ocsf.util: The utilities package\n\nThe `ocsf.util` package provides the `get_schema` function. This function\nleverages the functionality in the `ocsf.schema` and `ocsf.api` packages (below)\nto easily build an OCSF schema from a file on disk, a working copy of an OCSF\nrepository, or from the API.\n\n```python\nschema = get_schema(\"1.1.0\")\nschema = get_schema(\"./1.3.0-dev.json\")\nschema = get_schema(\"path/to/ocsf-schema\")\n```\n\n### ocsf.schema: The Schema Package\n\nThe `ocsf.schema` package contains Python data classes that represent an\nOCSF schema as represented from the OCSF server's API endpoints. See the\n`ocsf.schema.model` module for the data model definitions.\n\nIt also includes utilities to parse the schema from a JSON string or file.\n\n### ocsf.repository: The Repository Package\n\nThe `ocsf.repository` package contains a typed Python representation of a\nworking copy of an OCSF schema repository. Said another way, it represents the\nOCSF metaschema and repository contents in Python.\n\nIt also includes the `read_repo` function to read a repository from disk.\n\n### ocsf.compile: An OCSF Compiler\n\nThe `ocsf.compile` package \"compiles\" the OCSF schema from a repository just as\nthe OCSF server does (with very few exceptions). It is meant to provide:\n\n 1. An easy to use CLI tool to compile a repository into a single JSON schema\n file.\n 2. A reference implementation for others looking to better understand OCSF\n compilation or to create their own compiler.\n\n### ocsf.api: The API Package\n\nThe `ocsf.api` package exports an `OcsfApiClient`, which is a lightweight HTTP\nclient that can retrieve a version of the schema over HTTP and cache it on the\nlocal filesystem. It uses thes `export/schema`, `api/versions`, `api/profiles`,\nand `api/extensions` endpoints of the OCSF server.\n\n### ocsf.compare: The Compare Package\n\nThe `ocsf_tools.compare` package compares two versions of the OCSF schema and\ngenerates a type safe difference. Its aim is to make schema comparisons easy to\nwork with.\n\nThis package grew out of a library used internally at [Query](https://query.ai).\nThe original is used extensively to manage upgrading Query's data model to newer\nversions of OCSF, as well as to build adapters between different OCSF flavors\n(like AWS Security Lake on rc2 and Query on 1.1).\n\nThere is a very simple `__main__` implementation to demonstrate the comparison.\nYou can use it as follows:\n\n```sh\n$ poetry run python -m ocsf_tools.compare 1.0.0 1.2.0\n```\n\nThe comparison API is straightforward. Want to look for removed events?\n\n```python\ndiff = compare(get_schema(\"1.0.0\", \"1.1.0\"))\nfor name, event in diff.classes.items():\n if isinstance(event, Removal):\n print(f\"Oh no, we've lost {name}!\")\n```\n\nOr changed data types?\n\n```python\ndiff = compare(get_schema(\"1.0.0\", \"1.1.0\"))\nfor name, event in diff.classes.items():\n if isinstance(event, ChangedEvent):\n for attr_name, attr in event.attributes.items():\n if isinstance(attr, ChangedAttr):\n if isinstance(attr.type, Change):\n print(f\"Who changed this data type? {name}.{attr_name}\")\n```\n\nOr new objects?\n\n```python\ndiff = compare(get_schema(\"1.0.0\", \"1.1.0\"))\nfor name, obj in diff.objects.items():\n if isinstance(obj, Addition):\n print(f\"A new object {name} has been discovered!\")\n```\n\n\n### ocsf.validate.framework: The Validation Framework Package \n\nThe `ocsf.validate.framework` package provides a lightweight framework for\nvalidators. It was inspired by the needs of `ocsf-validator`, which may be\nported to this framework in the future.\n\n### ocsf.validate.compatibility: The Backwards Compatibility Validator\n\nThe `ocsf.validate.compatibility` provides a backwards compatibility validator\nfor OCSF schema. This compares the changes between two OCSF schemata and reports\nany breaking changes between the old and new version.\n\n## Getting Started\n\n### PyPI\n\nThe easiest way to install `ocsf-lib` is from PyPI using `pip` or `poetry`:\n\n```sh\n$ pip install ocsf-lib\n```\n\n\n### From Source\n\nIf you want to work with the source, the recommended installation is with `asdf`\nand `poetry`.\n\n```sh\n$ asdf install\n$ poetry install\n```\n\n## Contributing\n\nThis project uses `ruff` for formatting and linting, `pyright` for type\nchecking, and `pytest` as its test runner.\n\nBefore submitting a PR, make sure you've run following:\n\n```sh\n$ poetry run ruff format\n$ poetry run ruff check\n$ poetry run pyright\n$ poetry run pytest\n```\n\n### Type Checking\n\nWith great effort, this library passes pyright's strict mode type checking. Keep\nit that way! The OCSF schema is big, and even the metaschema is a lot to hold in\nyour head. Having the type checker identify mistakes for you can be very\nhelpful.\n\nThere is one cast used from the concrete `ChangedModel` types (`ChangedSchema`,\n`ChangedAttr`, etc.) in the compare package to the generic type. For the life of\nme, I can't figure it out. I blame pyright but it's probably my own fault.\n\n### Tests\n\nRunning unit tests:\n\n```sh\n$ poetry run pytest -m \"not integration\"\n```\n\nRunning integration tests:\n\n```sh\n$ poetry run pytest -m integration\n```\n\n**NOTE**: Some of the integration tests require an OCSF server instance, and are\nusing the public instance at [https://schema.ocsf.io](https://schema.ocsf.io).\nThis should probably use a local instance of the OCSF server instead.\n",
"bugtrack_url": null,
"license": "Apache-2.0",
"summary": "Tools for working with the OCSF schema",
"version": "0.8.2",
"project_urls": null,
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "641e67c0bafd3f881baf6a8247d66a8a0cae6e227243f6e790f29a792cc2110a",
"md5": "ad87f84511778872525152d5410821d6",
"sha256": "ca43aa86baa389761c97263dc3556d1af7210767faf013e5afd3cc92ab759c1f"
},
"downloads": -1,
"filename": "ocsf_lib-0.8.2-py3-none-any.whl",
"has_sig": false,
"md5_digest": "ad87f84511778872525152d5410821d6",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4.0,>=3.11",
"size": 74186,
"upload_time": "2024-08-30T22:21:13",
"upload_time_iso_8601": "2024-08-30T22:21:13.578662Z",
"url": "https://files.pythonhosted.org/packages/64/1e/67c0bafd3f881baf6a8247d66a8a0cae6e227243f6e790f29a792cc2110a/ocsf_lib-0.8.2-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "a3a22199f062f4bb08ff7aa18b13b43afaf10c503e60851cbe2b28cc20f52ba3",
"md5": "122aba7f674827adf0f2f3e7c2ba61d9",
"sha256": "61333c0cd2d70f61fa4b7c7d32af353cccdb8fafc682440b4b9ee2c1f8ac49f6"
},
"downloads": -1,
"filename": "ocsf_lib-0.8.2.tar.gz",
"has_sig": false,
"md5_digest": "122aba7f674827adf0f2f3e7c2ba61d9",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4.0,>=3.11",
"size": 52929,
"upload_time": "2024-08-30T22:21:14",
"upload_time_iso_8601": "2024-08-30T22:21:14.717772Z",
"url": "https://files.pythonhosted.org/packages/a3/a2/2199f062f4bb08ff7aa18b13b43afaf10c503e60851cbe2b28cc20f52ba3/ocsf_lib-0.8.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-08-30 22:21:14",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "ocsf-lib"
}
