# resource-policy-evaluation-library
rpe-lib is made up of `Policy Engines (rpe.engines.Engine)`, `Resources (rpe.resources.Resource)`, and `Extractors (rpe.resources.Extractor)`.
`Resources` produce details about the current state of a given type of resource. A resource can be just about anything, but the initial goal of the project was to support Google Cloud Platform (GCP) resources. The implemention is intentionally generic to allow support for other resource types
`Policy Engines` evaluate `Resources` against their configured policies. Given a resource, they can return a list of `Evaluations (rpe.policy.Evaluation)` indicating the name of each policy that applies to the resource (by resource type), and whether or not the resource is compliant. `Resource` also define a `remediate()` function, that should be able to take a dictionary description of how to manipulate a resource to make it compliant.
`Extractors` parse structured data and return a list of resources, and optionally metadata about the data source.
As an example, for GCP resources, the `remediate()` function expects details about what method to call in the Google REST API for the given resource, and what parameters to pass. So for a Google Cloud Storage Bucket, a policy that enforces bucket versioning could define remediation as a call to the buckets `patch()` method with the appropriate arguments to enable versioning.
[](https://github.com/cleardataeng/resource-policy-evaluation-library/actions/workflows/build.yml)
[](https://badge.fury.io/py/rpe-lib)
[](https://github.com/psf/black)
---
## Policy Engines
There are currently 2 `Policy Engines` included with rpe-lib:
### Open Policy Agent (opa) Engine
The Open Policy Agent engine uses the Open Policy Agent REST API to evalute policy for a given resource. In order to use this engine, you'll need to run the opa server with the rpe-lib base policies, and whatever policies you'd like to evaluate and optionally enforce. The opa engine passes the result of `Resource.get()` to the opa server as input. All policies need to be defined in the `rpe.policy` namespace, include a few specific rules, and operate on the `input` document:
* _applies\_to_: a list of resource types the policy applies to
* _description_: a human-readable description of the policy
* _compliant_: returns true if the resource defined in the `input` document adheres to the policy
* _excluded_: returns true if there is a reason to exclude the resource from evaluation, for GCP we define resource labels that can mark a resource as excluded
* _remediate (optional)_: returns a JSON object explaining how to remediate the resource for the given policy
Example policy:
```
# This is a real policy included with rpe-lib though slightly simplified
package rpe.policy.storage_buckets_require_object_versioning
description = "Require object versioning for storage buckets"
applies_to = ["storage.googleapis.com/Bucket"]
default compliant = false
default excluded = false
compliant {
input.resource.versioning.enabled = true
}
excluded {
input.resource.labels["forseti-enforcer"] = "disabled"
}
remediate = {
"_remediation_spec": "v2",
"steps": [
"method": "patch",
"params": {
"bucket": input.resource.name,
"body": {"versioning": {"enabled": true}},
}
]
}
```
### Python Engine
The OPA engine is very powerful, but is limited by what OPA is capable of. For use-cases that are more complicated, you might want to use the python engine. This engine expects the path to a python package that contains classes that perform the evaluation, and optionally remediation.
As mentioned, the python policies are actually classes. Here is an example python policy class:
```
# Stubbed out example of the above OPA engine policy as a python engine policy
class GCPBucketVersioningPolicy:
description = 'Require object versioning for storage buckets'
applies_to = ['cloudresourcemanager.googleapis.com/Project']
@classmethod
def compliant(cls, resource):
return resource.get()['resource']['versioning']['enabled'] == True
@classmethod
def excluded(cls, resource):
return resource.get()['resource']['labels']['forseti-enforcer'] == 'disabled'
@classmethod
def remediate(cls, resource):
# execute cloudfuntion to enable versioning,
# or shell execute gsutil,
# or anything you can do with python
pass
```
When you configure your RPE object, you provide the path to your python package. That package doesn't need to be installed, it will be loaded dynamically. The path you provide should contain an `__init__.py` file with every policy class you wish to use. These can be defined directly in that file, or imported. Since RPElib takes care of importing the package, you will be able to use relative imports in your python files. For example your directory structure might look like this:
```
/etc/rpe/policies-python
* __init__.py
* policy1.py
* policy2.py
```
And your `__init__.py` might look like this:
```python
# __init__.py
from .policy1 import MyFirstPythonPolicy
from .policy2 import AnotherPythonPolicy, YetAnotherPolicy
```
## Resources
`Resources` must define the following functions
* `get()`: returns metadata describing the current state of the resource
* `remediate(remediation_spec)`: applies the remediation (the spec is specific to the resource type/implementation)
* `type()`: returns the type of resource, used by policy engines to determine which policies apply to a given resource
## Examples
#### Using the OPA engine
This assumes you have the `opa` binary in your path
```
# First, start opa with our policies
opa run --server ./policy/
```
Now we need to create an RPE instance with the opa engine configured to use the local OPA server:
```python
from rpe import RPE
from rpe.resources import GoogleAPIResource
config = {
'policy_engines': [
{
'type': 'opa',
'url': 'http://localhost:8181/v1/data'
}
]
}
rpe = RPE(config)
# Create a resource object for the resource we want to evaluate
res = GoogleAPIResource.from_cai_data(
'//storage.googleapis.com/my-test-bucket',
'storage.googleapis.com/Bucket',
project_id='my-test-project',
)
evals = rpe.evaluate(res)
for e in evals:
print(f'Policy: {e.policy_id}, Compliant: {e.compliant}')
if not e.compliant and e.remediable:
e.remediate()
```
#### Using the Python engine
Using the Python policy engine is similar to the above:
```python
from rpe import RPE
from rpe.resources import GoogleAPIResource
config = {
'policy_engines': [
{
'type': 'python',
'path': '/etc/rpe/policies-python'
}
]
}
rpe = RPE(config)
# Create resource objects, and evaluate as needed
```
And your policies may look like this:
/etc/rpe/policies-python/__init__.py
```python
from .gcs_policy import GCPBucketVersioningPolicy
```
/etc/rpe/policies-python/gcs_policy.py
```python
class GCPBucketVersioningPolicy:
description = 'Require object versioning for storage buckets'
applies_to = ['cloudresourcemanager.googleapis.com/Project']
@classmethod
def compliant(cls, resource):
# Return true/false if the resource is compliant
@classmethod
def excluded(cls, resource):
# Return true/false if the resource is excluded
@classmethod
def remediate(cls, resource):
# Enable versioning
```
# Applications using rpe-lib
* [Forseti Real-time Enforcer](https://github.com/forseti-security/real-time-enforcer) - The Forseti Real-time enforcer uses rpe-lib for the evaluation and enforcement of policy for Google Cloud resources. It uses a Stackdriver log export to a Pub/Sub topic to trigger enforcement.
Raw data
{
"_id": null,
"home_page": "https://github.com/cleardataeng/resource-policy-evaluation-library",
"name": "rpe-lib",
"maintainer": null,
"docs_url": null,
"requires_python": null,
"maintainer_email": null,
"keywords": "gcp policy enforcement",
"author": "Joe Ceresini",
"author_email": null,
"download_url": "https://files.pythonhosted.org/packages/21/4e/2be5173fcdc156248358edced96a668e779d90803de58b5db80576dbfd06/rpe-lib-3.2.1.tar.gz",
"platform": null,
"description": "# resource-policy-evaluation-library\n\nrpe-lib is made up of `Policy Engines (rpe.engines.Engine)`, `Resources (rpe.resources.Resource)`, and `Extractors (rpe.resources.Extractor)`.\n\n`Resources` produce details about the current state of a given type of resource. A resource can be just about anything, but the initial goal of the project was to support Google Cloud Platform (GCP) resources. The implemention is intentionally generic to allow support for other resource types\n\n`Policy Engines` evaluate `Resources` against their configured policies. Given a resource, they can return a list of `Evaluations (rpe.policy.Evaluation)` indicating the name of each policy that applies to the resource (by resource type), and whether or not the resource is compliant. `Resource` also define a `remediate()` function, that should be able to take a dictionary description of how to manipulate a resource to make it compliant.\n\n`Extractors` parse structured data and return a list of resources, and optionally metadata about the data source.\n\nAs an example, for GCP resources, the `remediate()` function expects details about what method to call in the Google REST API for the given resource, and what parameters to pass. So for a Google Cloud Storage Bucket, a policy that enforces bucket versioning could define remediation as a call to the buckets `patch()` method with the appropriate arguments to enable versioning.\n\n[](https://github.com/cleardataeng/resource-policy-evaluation-library/actions/workflows/build.yml)\n[](https://badge.fury.io/py/rpe-lib)\n[](https://github.com/psf/black)\n\n---\n\n## Policy Engines\n\nThere are currently 2 `Policy Engines` included with rpe-lib:\n\n### Open Policy Agent (opa) Engine\n\nThe Open Policy Agent engine uses the Open Policy Agent REST API to evalute policy for a given resource. In order to use this engine, you'll need to run the opa server with the rpe-lib base policies, and whatever policies you'd like to evaluate and optionally enforce. The opa engine passes the result of `Resource.get()` to the opa server as input. All policies need to be defined in the `rpe.policy` namespace, include a few specific rules, and operate on the `input` document:\n\n* _applies\\_to_: a list of resource types the policy applies to\n* _description_: a human-readable description of the policy\n* _compliant_: returns true if the resource defined in the `input` document adheres to the policy\n* _excluded_: returns true if there is a reason to exclude the resource from evaluation, for GCP we define resource labels that can mark a resource as excluded\n* _remediate (optional)_: returns a JSON object explaining how to remediate the resource for the given policy\n\nExample policy:\n\n```\n# This is a real policy included with rpe-lib though slightly simplified\npackage rpe.policy.storage_buckets_require_object_versioning\n\ndescription = \"Require object versioning for storage buckets\"\n\napplies_to = [\"storage.googleapis.com/Bucket\"]\n\ndefault compliant = false\n\ndefault excluded = false\n\ncompliant {\n input.resource.versioning.enabled = true\n}\n\nexcluded {\n input.resource.labels[\"forseti-enforcer\"] = \"disabled\"\n}\n\nremediate = {\n \"_remediation_spec\": \"v2\",\n \"steps\": [\n \"method\": \"patch\",\n \"params\": {\n \"bucket\": input.resource.name,\n \"body\": {\"versioning\": {\"enabled\": true}},\n }\n ]\n}\n```\n\n### Python Engine\nThe OPA engine is very powerful, but is limited by what OPA is capable of. For use-cases that are more complicated, you might want to use the python engine. This engine expects the path to a python package that contains classes that perform the evaluation, and optionally remediation.\n\nAs mentioned, the python policies are actually classes. Here is an example python policy class:\n\n```\n# Stubbed out example of the above OPA engine policy as a python engine policy\nclass GCPBucketVersioningPolicy:\n description = 'Require object versioning for storage buckets'\n applies_to = ['cloudresourcemanager.googleapis.com/Project']\n\n @classmethod\n def compliant(cls, resource):\n return resource.get()['resource']['versioning']['enabled'] == True\n\n @classmethod\n def excluded(cls, resource):\n return resource.get()['resource']['labels']['forseti-enforcer'] == 'disabled'\n\n @classmethod\n def remediate(cls, resource):\n # execute cloudfuntion to enable versioning,\n # or shell execute gsutil,\n # or anything you can do with python\n pass\n```\n\nWhen you configure your RPE object, you provide the path to your python package. That package doesn't need to be installed, it will be loaded dynamically. The path you provide should contain an `__init__.py` file with every policy class you wish to use. These can be defined directly in that file, or imported. Since RPElib takes care of importing the package, you will be able to use relative imports in your python files. For example your directory structure might look like this:\n\n```\n/etc/rpe/policies-python\n* __init__.py\n* policy1.py\n* policy2.py\n```\n\nAnd your `__init__.py` might look like this:\n\n```python\n# __init__.py\nfrom .policy1 import MyFirstPythonPolicy\nfrom .policy2 import AnotherPythonPolicy, YetAnotherPolicy\n```\n\n## Resources\n\n`Resources` must define the following functions\n\n* `get()`: returns metadata describing the current state of the resource\n* `remediate(remediation_spec)`: applies the remediation (the spec is specific to the resource type/implementation)\n* `type()`: returns the type of resource, used by policy engines to determine which policies apply to a given resource\n\n## Examples\n\n#### Using the OPA engine\n\nThis assumes you have the `opa` binary in your path\n\n```\n# First, start opa with our policies\nopa run --server ./policy/\n```\n\nNow we need to create an RPE instance with the opa engine configured to use the local OPA server:\n\n```python\nfrom rpe import RPE\nfrom rpe.resources import GoogleAPIResource\n\nconfig = {\n 'policy_engines': [\n {\n 'type': 'opa',\n 'url': 'http://localhost:8181/v1/data'\n }\n ]\n}\n\nrpe = RPE(config)\n\n# Create a resource object for the resource we want to evaluate\nres = GoogleAPIResource.from_cai_data(\n '//storage.googleapis.com/my-test-bucket',\n 'storage.googleapis.com/Bucket',\n project_id='my-test-project',\n)\n\nevals = rpe.evaluate(res)\n\nfor e in evals:\n print(f'Policy: {e.policy_id}, Compliant: {e.compliant}')\n\n if not e.compliant and e.remediable:\n e.remediate()\n```\n\n#### Using the Python engine\n\nUsing the Python policy engine is similar to the above:\n\n```python\nfrom rpe import RPE\nfrom rpe.resources import GoogleAPIResource\n\nconfig = {\n 'policy_engines': [\n {\n 'type': 'python',\n 'path': '/etc/rpe/policies-python'\n }\n ]\n}\n\nrpe = RPE(config)\n\n# Create resource objects, and evaluate as needed\n```\n\nAnd your policies may look like this:\n\n/etc/rpe/policies-python/__init__.py\n```python\nfrom .gcs_policy import GCPBucketVersioningPolicy\n```\n\n/etc/rpe/policies-python/gcs_policy.py\n```python\nclass GCPBucketVersioningPolicy:\n\n description = 'Require object versioning for storage buckets'\n applies_to = ['cloudresourcemanager.googleapis.com/Project']\n\n @classmethod\n def compliant(cls, resource):\n # Return true/false if the resource is compliant\n\n @classmethod\n def excluded(cls, resource):\n # Return true/false if the resource is excluded\n\n @classmethod\n def remediate(cls, resource):\n # Enable versioning\n```\n\n# Applications using rpe-lib\n\n* [Forseti Real-time Enforcer](https://github.com/forseti-security/real-time-enforcer) - The Forseti Real-time enforcer uses rpe-lib for the evaluation and enforcement of policy for Google Cloud resources. It uses a Stackdriver log export to a Pub/Sub topic to trigger enforcement.\n\n\n",
"bugtrack_url": null,
"license": "Apache 2.0",
"summary": "A resource policy evaluation library",
"version": "3.2.1",
"project_urls": {
"Homepage": "https://github.com/cleardataeng/resource-policy-evaluation-library"
},
"split_keywords": [
"gcp",
"policy",
"enforcement"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "02407343038a15bf8720768852502d8e91dad8ac98f7905c016daceac775e37b",
"md5": "5cfe3a8f54f48935385515bd17365a8b",
"sha256": "a3490488089886cd3f95f444ce570cefb5d3990cb2e2347606b6e10b9344ee79"
},
"downloads": -1,
"filename": "rpe_lib-3.2.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "5cfe3a8f54f48935385515bd17365a8b",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 31127,
"upload_time": "2024-12-19T17:56:55",
"upload_time_iso_8601": "2024-12-19T17:56:55.603139Z",
"url": "https://files.pythonhosted.org/packages/02/40/7343038a15bf8720768852502d8e91dad8ac98f7905c016daceac775e37b/rpe_lib-3.2.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "214e2be5173fcdc156248358edced96a668e779d90803de58b5db80576dbfd06",
"md5": "36dcc0f093789894e5221fd462610156",
"sha256": "0501e30c4290a659634b3a0b48946b8f95b7f405457f25344cfd20354734b8bf"
},
"downloads": -1,
"filename": "rpe-lib-3.2.1.tar.gz",
"has_sig": false,
"md5_digest": "36dcc0f093789894e5221fd462610156",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 63260,
"upload_time": "2024-12-19T17:56:57",
"upload_time_iso_8601": "2024-12-19T17:56:57.046699Z",
"url": "https://files.pythonhosted.org/packages/21/4e/2be5173fcdc156248358edced96a668e779d90803de58b5db80576dbfd06/rpe-lib-3.2.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-12-19 17:56:57",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "cleardataeng",
"github_project": "resource-policy-evaluation-library",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [
{
"name": "google-api-python-client-helpers",
"specs": [
[
">=",
"1.2.6"
]
]
},
{
"name": "google-api-python-client",
"specs": [
[
"~=",
"2.126.0"
]
]
},
{
"name": "jmespath",
"specs": []
},
{
"name": "tenacity",
"specs": []
},
{
"name": "python-dateutil",
"specs": []
},
{
"name": "mock",
"specs": []
},
{
"name": "google-cloud-pubsub",
"specs": []
},
{
"name": "pydantic",
"specs": []
}
],
"tox": true,
"lcname": "rpe-lib"
}
JSON
